# D1.2 Roadmap and pilot requirements 2nd version

Christophe Guillotel-Nothmann, Jacopo de Berardinis, Thomas Bottini, Philippe Cathé, Enrico Daga, Marilena Daquino, Achille Davy-Rigaux, Marco Gurrieri, Peter Van Kranenburg, Eleonora Marzi, et al.

## ► To cite this version:

Christophe Guillotel-Nothmann, Jacopo de Berardinis, Thomas Bottini, Philippe Cathé, Enrico Daga, et al. D1.2 Roadmap and pilot requirements 2nd version. Institut de recherche en musicologie (IReMus). 2022. halshs-04079523

HAL Id: halshs-04079523

https://shs.hal.science/halshs-04079523

Submitted on 28 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
D1.2 Roadmap and pilot requirements 2nd version

<table><thead><tr><td colspan="2">Deliverable information</td></tr></thead><tbody><tr><td>WP</td><td>WP1 Pilots and Web portal</td></tr><tr><td>Document dissemination level</td><td>○ PU Public</td></tr><tr><td>Deliverable type</td><td>○ R Document, report</td></tr><tr><td>Lead beneficiary</td><td>CNRS</td></tr><tr><td>Contributors</td><td>UNIBO</td></tr><tr><td>Document status</td><td>Final</td></tr><tr><td>Document version</td><td>V1.0</td></tr><tr><td>Date</td><td>30/06/2022</td></tr><tr><td>Authors</td><td>Christophe Guillotel-Nothmann (CNRS-IReMus) [Task leader]<br>Jacopo de Berardinis (KCL)<br>Thomas Bottini (CNRS-IReMus)<br>Philippe Cathé (Sorbonne-IReMus)<br>Enrico Daga (OU)<br>Marilena Daquino (UNIBO)<br>Achille Davy-Rigaux (CNRS-IReMus)<br>Marco Gurrieri (CNRS-IReMus)<br>Peter van Kranenburg (KNAW)<br>Eleonora Marzi (UNIBO)<br>James McDermott (NUIG)<br>Albert Meroño Peñuela (KCL)<br>Paul Mulholland (OU)<br>Andrea Scharnhorst (KNAW)<br>Rocco Tripodi (UNIBO)</td></tr></tbody></table>

<figure><img src="image_2.png" /></figure>
**Project information**

**Project Start Date:** 1st of January 2021
**Project Duration:** 40 months

**Project website:** http://www.polifonia-project.eu

## Project contacts

### Project Coordinator

Valentina Presutti

ALMA MATER STUDIORUM - UNIVERSITÀ DI
BOLOGNA

Department of Language, Literature and
Modern Cultures (LILEC)

E-mail: valentina-presutti@unibo.it

### Project Manager

Marta Clementi

ALMA MATER STUDIORUM - UNIVERSITÀ DI
BOLOGNA

Research division

E-mail: marta.clementi3@unibo.it

## POLIFONIA consortium

<table><thead><tr><td>No.</td><td>Short name</td><td>Institution name</td><td>Country</td></tr></thead><tbody><tr><td>1</td><td>UNIBO</td><td>ALMA MATER STUDIORUM - UNIVERSITÀ DI BOLOGNA</td><td>Italy</td></tr><tr><td>2</td><td>OU</td><td>THE OPEN UNIVERSITY</td><td>United Kingdom</td></tr><tr><td>3</td><td>KCL</td><td>KING'S COLLEGE LONDON</td><td>United Kingdom</td></tr><tr><td>4</td><td>NUI GALWAY</td><td>NATIONAL UNIVERSITY OF IRELAND GALWAY</td><td>Ireland</td></tr><tr><td>5</td><td>MiC</td><td>MINISTERO DELLA CULTURA</td><td>Italy</td></tr><tr><td>6</td><td>CNRS</td><td>CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE CNRS</td><td>France</td></tr><tr><td></td><td>SORBONNE</td><td>SORBONNE UNIVERSITE (LinkedTP)</td><td>France</td></tr><tr><td>7</td><td>CNAM</td><td>CONSERVATOIRE NATIONAL DES ARTS ET METIERS</td><td>France</td></tr><tr><td>8</td><td>NISV</td><td>STICHTING NEDERLANDS INSTITUUT VOORBEELD EN GELUID</td><td>Netherlands</td></tr><tr><td>9</td><td>KNAW</td><td>KONINKLIJKE NEDERLANDSE AKADEMIE VAN WETENSCHAPPEN</td><td>Netherlands</td></tr><tr><td>10</td><td>DP</td><td>DIGITAL PATHS SRL</td><td>Italy</td></tr></tbody></table>

## Project Summary

European musical heritage is a dynamic historical flow of experiences, leaving heterogeneous traces that are difficult to capture, connect, access, interpret, and valorise. Computing technologies have the potential to shed a light on this wealth of resources by extracting, materialising and linking new knowledge from heterogeneous sources, hence revealing facts and experiences from hidden voices of the past. Polifonia makes this happen by building novel ways of inspecting, representing, and interacting with digital content. Memory institutions, scholars, and citizens will be able to
navigate, explore, and discover multiple perspectives and Stories about European Musical Heritage.

*Polifonia* focuses on European Musical Heritage, intended as musical contents and artefacts - or music objects - (tunes, scores, melodies, notations, etc.) along with relevant knowledge about them such as: their links to tangible objects (theatres, conservatoires, churches, etc.), their cultural and historical contexts, opinions and Stories told by people having diverse social and artistic roles (scholars, writers, students, intellectuals, musicians, politicians, journalists, etc.), and facts expressed in different styles and disciplines (memoirs, reportages, news, biographies, reviews), different languages (English, Italian, French, Spanish, and German), and across centuries.

The overall goal of the project is to realise achieve an Ecosystem of computational methods and tools supporting discovery, extraction, encoding, interlinking, classification, exploration of, and access to, musical heritage knowledge on the Web. An equally important objective is to demonstrate that these tools improve the state of the art of Social Science and Humanities (SSH) methodologies. Hence their development is guided by, and continuously intertwined with, experiments and validations performed in real-world settings, identified by musical heritage stakeholders (both belonging to the Consortium and external supporters) such as cultural institutes and collection owners, historians of music, anthropologists and ethnomusicologists, linguists, etc.
# Executive summary

Polifonia is driven by ten pilot use cases which provide both a validation context and the input requirements to the other research and development work packages. The pilots are heterogeneous in terms of knowledge domains, e.g. bells heritage, popular Irish music, history of music in Bologna, music influence on children<fnref n="1" />. They involve interdisciplinary teams that bring different experience and methodological practices.

This report describes the efforts made in relation with the Technical Board (TB) (see deliverable 1.3²); the Data Management Plan (DMP) (see deliverables 7.1 and 7.4³); the working group (WG<fnref n="4" />) focused on the Web portal (see deliverable 1.9⁵); work package 2 (WP 2)<fnref n="6" /> (see deliverables 2.1, 2.4 and 2.7⁷), WP 5 and the groups gathering the pilots and the work packages for the definition of a socio-technical Roadmap. This Roadmap aims to provide a methodological framework for the delivery of the 10 pilots. The initial version of the Roadmap (D1.1. produced at M6⁸) described the efforts made to gather the requirements and needs of the pilots on the basis of three tools: a) the Stories, b) the Maninpasta, and c) the Survey (SMS tools). This second version provides a definition of the methods, tools and strategies that regulate the socio-technical interactions and lead to an overall Validation framework.

After outlining the background to this deliverable and its status in Polifonia, the following four sections successively address the four main objectives of the Roadmap: 1. Coordinating collection of resources for use in the pilots, 2. Gathering requirements from internal and external adopters/stakeholders, 3. Setting objectives and challenges and mapping them from specific domains to the technology provider work packages, and 4. Monitoring validation within the pilot tasks. Six short appendixes make this report self-contained, including a glossary of terms, a description of the ten pilots, a description of Polifonia work packages' organisation and an updated version of the Survey.

# Document History

<footnote n="1">¹ The pilots are described in detail in appendix 2.</footnote>

<footnote n="2">² Enrico Daga et al. D1.3. Pilots development – collaborative methodology and tools, Polifonia, 2021.</footnote>

<footnote n="3">³ Andrea Scharnhorst et al. D7.1. First Data Management Plan, Polifonia, 2021; Scharnhorst, Andrea et al. D7.4. DMP webinar on dynamic data management, Polifonia, 2021.</footnote>

<footnote n="4">⁴ Appendix 1 contains a list of acronyms with their definitions.</footnote>

<footnote n="5">⁵ Marilena Daquino et al., D1.9. Polifonia Web portal – 1st version, Polifonia, 2022.</footnote>

<footnote n="6">⁶ For a description of Polifonia's 8 work packages, see appendix 4.</footnote>

<footnote n="7">⁷ Albert Meroño-Peñuela et al. D.2.1. Ontologies and knowledge graphs of music objects, patterns, and software package – 1st version, Polifonia 2021; Albert Meroño-Peñuela et al. D.2.4. Methods for interlinking knowledge graphs – 1st version, Polifonia 2022; Albert Meroño-Peñuela et al. D.2.7. Ontology testing and evaluation report; knowledge graph documentation and access interfaces – 1st version, Polifonia, 2022.</footnote>

<footnote n="8">⁸ Christophe Guillotel-Nothmann et al. D1.1. Roadmap and pilot requirements 1st version, Polifonia, 2021.</footnote>
<table><thead><tr><td>Version</td><td>Release date</td><td>Summary of changes</td><td>Author(s) - Institution</td></tr></thead><tbody><tr><td>V0.1</td><td>30/04/2022</td><td>First draft released</td><td>CNRS</td></tr><tr><td>V0.2</td><td>20/05/2022</td><td>Initial version submitted to internal reviewers</td><td>CNRS</td></tr><tr><td>V0.3</td><td>19/06/2022</td><td>Version updated according to the reviewers' remarks</td><td>CNRS</td></tr><tr><td>V1.0</td><td>30/06/2022</td><td>Final Version submitted to EU</td><td>UNIBO</td></tr></tbody></table>
## Table of contents

<table><tr><td>Project information</td><td>2</td></tr><tr><td>Project contacts</td><td>3</td></tr><tr><td>POLIFONIA consortium</td><td>3</td></tr><tr><td>Executive summary</td><td>5</td></tr><tr><td>Document History</td><td>5</td></tr><tr><td>Introduction</td><td>8</td></tr><tr><td>1. Coordinating the collection of resources, methods and tools used in the pilots</td><td>9</td></tr><tr><td>1.1 Goals, methods and strategies</td><td>9</td></tr><tr><td>1.2 Results</td><td>11</td></tr><tr><td>1.3 Lessons learned</td><td>20</td></tr><tr><td>2. Centralizing needs and requirements</td><td>21</td></tr><tr><td>2.1 Methods and strategies</td><td>22</td></tr><tr><td>2.2 Results</td><td>24</td></tr><tr><td>2.3 Lessons learned</td><td>28</td></tr><tr><td>3. Mapping objectives and challenges from pilot specific domains to the technology provider WPs</td><td>28</td></tr><tr><td>3.1 Goals, methods and strategies</td><td>28</td></tr><tr><td>3.2 Results</td><td>34</td></tr><tr><td>3.3 Lessons learned</td><td>35</td></tr><tr><td>4. Monitoring pilot validation</td><td>37</td></tr><tr><td>4.1 Goals, methods and strategies</td><td>37</td></tr><tr><td>4.2 Results</td><td>38</td></tr><tr><td>4.3 Lessons learned</td><td>42</td></tr><tr><td>FAIR protocol</td><td>43</td></tr><tr><td>Conclusion</td><td>44</td></tr><tr><td>References</td><td>46</td></tr><tr><td>Appendices</td><td>48</td></tr><tr><td>Appendix 1: Glossary</td><td>48</td></tr><tr><td>Appendix 2: List of the pilots</td><td>51</td></tr><tr><td>Appendix 3: WP1 participants, efforts, objectives, tasks and deliverables</td><td>53</td></tr><tr><td>Appendix 4: Work packages in Polifonia project</td><td>55</td></tr><tr><td>Appendix 5: Stories - updated template</td><td>58</td></tr><tr><td>Appendix 6: Updated Survey template</td><td>59</td></tr></table>
# Introduction

Polifonia is driven by 10 pilot use cases addressing preservation, management, studying musical heritage (MH), and interacting with it. Being real-world use cases – for example the Italian bell heritage, organs in the Netherlands, access to musical heritage for deaf people or the identification of recurring melodic patterns –, these pilots provide a means to validate Polifonia's computational solutions, to strengthen the project's outreach, and to integrate new scenarios, early-adopters and stakeholders. They are developed by interdisciplinary teams of researchers from the humanities and social sciences, and from computer science.

This deliverable presents the second version of a "socio-technical Roadmap" which aims at providing a common framework to all pilots for:

1. Coordinating collection of resources for use in the pilots,

2. Gathering requirements from internal and external adopters/stakeholders,

3. Setting objectives and challenges and mapping them from specific domains to the technology provider work packages, and

4. Monitoring validation within the pilot tasks.

While the first version of the Roadmap, D1.1 released at M6⁹, adopted a bottom-up approach, the current version proposes a generalization of methods to share, interlink, articulate and validate resources, tools and results within a decentralized yet interconnected Ecosystem.

This generalization is based on an interdisciplinary dialogue between a) the musicologists, linguists, historians and curators working at the pilots and b) the computer scientists, designers and researchers in cognitive sciences involved in the project. The outcome is the result of a continuous interaction between WP1¹⁰ – the driving force of Polifonia and place where the pilots happen – and the technology provider work packages (WP2-5).

An important vector of this dialogue are the "SMS-tools" – Stories, Maninpasta, and Survey –, which were presented in detail in the first version of the Roadmap. These tools have helped a) to constantly keep and update a centralized view of the information collected (Survey), b) to strengthen synergies through practical and concrete work between researchers from different disciplines (Maninpasta), and c) to collect the requests, needs and expectations of the pilots (Stories). As central pillars of the Roadmap, they have allowed both to identify and to promote common principles, methods and approaches among the pilots while preserving their specific needs and orientations.

The four sections of this deliverable relate to the four objectives outlined above and critically present how the tool's current state of development, methods and strategies contribute to the Roadmap. While the deliverable is organised according to the four objectives, the tools presented do not focus exclusively on one of them but cover several of them: the Survey, for example, contributes to both resource collection and validation monitoring. For methodological reasons, however, the tools will be presented in detail in the section in which they play a key role. Beyond the presentation of a set of tools, principles and methods that can be applied to the pilots, that regulate their interaction and

<footnote n="9">⁹ See Christophe Guillotel-Nothmann et al. *D1.1. Roadmap and pilot requirements 1st version*, Polifonia, 2021.</footnote>

<footnote n="10">¹⁰ For a detailed description of WP1, see appendix 3.</footnote>
that can be transferred to other use cases, the objective of this document is also to describe how
the collaborative work took place and how the framework was shaped.

To facilitate the reading of this deliverable, a glossary (which provides a resolution of acronyms, short definitions and links) is included in Appendix 1. Appendix 2 gives a presentation of the pilots and their objectives. Appendix 3 summarizes the essential data of WP1, to which the Roadmap is attached and which constitutes the driving force of the project. Appendix 4 provides key data on the other Polifonia work packages. The last two Appendices 5 and 6 give an update of the pilot's Stories and of the Survey.

# 1. Coordinating the collection of resources, methods and tools used in the pilots

Three main devices are implemented to identify, organize and articulate the resources, methods and tools used by the pilots: a) the Ecosystem with its Rulebook, b) the Web portal and c) the Survey.

## 1.1 Goals, methods and strategies

### 1.1.1 Ecosystem with Rulebook.

The Polifonia Ecosystem<fnref n="11" /> – which is the collection of data, software, and documentation related to the project – allows the identification of all resources and tools created in Polifonia. The TB supervises the definition of the guidelines for collaborative development, encoded in a Rulebook, targeting consortium members as well as external stakeholders, following the *open science* principles. The TB reflects on the organization principles of the Ecosystem. This includes the provision of a set of descriptors through which developers can describe the components, allowing to support exploratory interfaces to navigate between them, tracing their evolution, and documenting the decisions made at both the technical and methodological level. As such, the TB actively participates in the effort to identify, organize and articulate resources, methods and tools that lie at the heart of the socio-technical Roadmap.

### 1.1.2 Web portal.

The Web portal aims at collecting highly curated data produced by pilots and serving them, along with third-party data, via user-friendly interfaces. To produce a coherent and meaningful resource, an important task is the harmonization of content produced by pilot sources. Since the scope, the completeness and the granularity of data sources produced by data providers in the consortium may significantly vary, the primary goal of the Web portal is to define a sound narrative to connect music-related domains, expertise and research requirements, and to define the target of our expected impact.

On the one hand, a comprehensive strategy should be provided to target the broad audience of music passionate and professionals. On the other hand, preserving the diversity of approaches, methods and research interests should also be encouraged. Therefore, the design of the Web portal

<footnote n="11">¹¹ Browsable site: https://polifonia-project.github.io/ecosystem/pages/pilots/, Github repository: https://github.com/polifonia-project/ecosystem.</footnote>
is articulated into two main research areas: a) design interfaces for lay people (i.e. without music skills and/or without any research task in mind) to enjoy music heritage collections via interactive Stories; b) design task-driven interfaces which are based on data-driven storytelling strategies, so as to accommodate scholars' research tasks.

To foster co-creation and co-design of the Web portal, we refined the methodological tools offered by the SMS tools, the Rulebook, the DMP and WP5 user interface (UI) / user experience (UX) design strategies<fnref n="12" />. In particular, we systematically analyzed Stories, Competency Questions (CQ)<fnref n="13" /> and ontology requirements, to identify knowledge discovery patterns shared by pilots and organize content of the Web portal accordingly. Secondly, we had focus groups with scholars in the consortium and lay people (questionnaires with around 150 participants were collected) to iteratively collect expectations and potential frustration elements, as well as to validate our design choices. Lastly, we classified cognitive tasks underlying CQ and expectations, and we co-designed flexible, advanced data-driven interfaces (i.e. dashboards) to allow current and future (unexpected) research questions to be addressed via the Web portal interfaces without requiring us to revisit our work.

### 1.1.3 Survey.

The Survey<fnref n="14" /> was developed in order to a) systematically and iteratively collect information about the pilots, b) allow an adjustment of the pilots through the examination of the answers given, and c) push their design through the questions asked. It consists of three main sections: the first one relates to the pilot's specific domain, the second provides technical information, and the third deals with validation and engagement issues<fnref n="15" />. The Survey's main objective is to create a map of the resources, methods and tools in relation with these three domains to articulate these components, to foster synergies and to identify common perspectives and particularities. The objective is not only to follow the evolution of the pilots through the different iterations<fnref n="16" />, but also to allow for progressive adjustments between them and to adapt the Survey to the problems encountered and the questions that arise at different moments of the project.

Compared to the first iteration of the Survey, described in deliverable 1.1, several adjustments have been made. In the domain-specific part, questions have been added about the pilots' objectives, the results obtained and the progress made towards the full achievement of the objectives. In the technical part, questions were added about the expectations of the pilots towards the technology providers (WP2-5), the collaborative space of WP1, and towards the Web portal (presentation of resources, type of knowledge made available, types of interactions, link with the pilot-specific demonstrator, added value of the Web portal). The "engagement and validation" part (former "socio-pedagogical part") has been extensively modified, too. Questions were added about the identification of the pilot's partner institutions (beyond the coordinating institution) and an inventory

<footnote n="12">¹² For details, see Marilena Daquino et al., D1.9. Polifonia Web portal – 1st version, Polifonia, 2022.</footnote>

<footnote n="13">¹³ See section 2 below about these concepts.</footnote>

<footnote n="14">¹⁴ The Survey is available on github: https://github.com/polifonia-project/survey.</footnote>

<footnote n="15">¹⁵ The questions from the Survey, in its current version, are reproduced in Appendix 6. The answers given by the pilots are available on the Polifonia Github: https://github.com/polifonia-project/Survey/blob/main/Survey.xlsx.</footnote>

<footnote n="16">¹⁶ Iterations took place at the following dates: 07.05.2021, 14.05.2021, 15.11.2021 and 13.06.2022. See above in this document, section 1, table 1.</footnote>
of how the feedback from users/experts has been considered has been requested. The pilots are
also asked about the demonstrators that will be made available in the coming months. Finally,
questions related to pilot monitoring and validation were added/merged.

The Survey was originally an Excel spreadsheet that was progressively updated by the pilot leaders.
Our goal for the remainder of the project is to transform the Survey into a knowledge graph about
the pilots: the pilot knowledge graph (PiKG). The motivation behind this is to identify both the
components that are shared between the pilots and the pilot-specific concepts. (e.g. formats,
standards, collections, licenses, methods, epistemological positions, scientific questions, validation
devices, etc.). As a complement to the Polifonia Ontology Network (PON), which aims at
interconnecting the pilots through the entities and properties they use, the interconnection made
by the Survey is at a meta-level. The Survey provides a framework to identify and document
connections – methodological choices, scientific assumptions, ways of engagement, validation
protocols – that cannot be deduced by analyzing common entities/properties. The different
iterations of the Survey also show how the pilot evolved over time, how methodological
convergences appeared, how they refocused, etc.

## 1.2 Results

### 1.2.1 Ecosystem.

The methodology for the collaborative development that led to the design of the Ecosystem was
already presented in D1.3¹⁷. Since then, the TB designed a set of guidelines, recommendations and
standards to regulate the way consortium members (and stakeholders) can contribute to the
Ecosystem. These results are presented in the GitHub Polifonia Organisation, specifically, in the
repository Rulebook<fnref n="18" />.

<figure><img src="image_3.png" /><figcaption>**Example 1: Ecosystem, Rulebook.**</figcaption></figure>

<footnote n="17">¹⁷ Enrico Daga et al. D1.3. Pilots development – collaborative methodology and tools, Polifonia, 2021.</footnote>

<footnote n="18">¹⁸ https://github.com/polifonia-project/Rulebook.</footnote>
The Rulebook includes an annotation schema for Ecosystem components that allows to annotate each component with a rich metadata set (example 1). It defines when to create a repository (i.e. for any component of the Ecosystem), how to associate it with a "champion" (i.e. the repository leader), how to structure it and how to enrich it with metadata that: identifies the component within the Ecosystem, defines it by keywords, links it to the different WPs and pilots, assigns it an order of priority, and classifies it according to a typology. As an example, the Amy#2_OrganBuilders component (example 2) is a "Story" component (i.e. a narrative that sets out a certain problem to be explored)<fnref n="19" />, related to WP2 and to the Organs pilot. This Story's Persona (i.e. the protagonist of this Story) is "Amy" and is about "durch organs", "instruments", traditions".

<figure><img src="image_4.png" /></figure>

## Example 2: Amy2_OrganBuilders as an Ecosystem component.

Such methodological practices contribute to make the whole set of components explorable through an Ecosystem website. Based on the metadata provided, the components are indexed in the Ecosystem and can be navigated according to the structural relationships defined. This approach allows the publication of components in real time, as Polifonia progresses and is an important methodological outcome that could be published itself with a view to being made reusable and adaptable to other projects. The first version of the Polifonia Ecosystem was released in April 2022 (Milestone 2) - v1.0²⁰. New releases are expected in line with the software deliverables and the other milestones of the Polifonia Project.

<footnote n="19">¹⁹ The "Story" concept will be explained in detail in section 2 of this document.</footnote>

<footnote n="20">²⁰ https://polifonia-project.github.io/ecosystem/.</footnote>
### 1.2.2 Web portal.

While the design of interfaces for lay people had limited appeal to scholars in the consortium (and will therefore be subject of further study in the next phase of the project), the design of data-driven dashboard interfaces led us to grasp a comprehensive outlook on research methods (i.e. qualitative and quantitative) and expectations of scholars with different interests and levels of expertise.

In particular, we were able to classify research methodologies into three schemata, in turn represented by three dashboard templates. Templates address different (incrementally difficult) types of analysis and interaction patterns, namely: statistics-based and qualitative argumentation, geographic-based interactive exploration, charts manipulation for datasets integration and temporal analysis. These templates have been developed on the basis of the Meetups, Musicbo, Bells, Child, Tunes and Organs pilots whose only interface will be the Web portal, while the pilots with an independent interface (Interlink, Facets, Tonalities) have not been taken into account at this stage<fnref n="21" />. In the case of Meetups, which explores the encounters between musicians between 1800 and 1945, the templates provide general information on the number of people, performances and countries considered (example 3a). They allow geographical visualization of the data based on filters (example 3b), give general information on individual data points (example 3c) and provide tools for granular exploratory visualization of data points, for example on instruments used, listeners and socio-cultural environments (example 3d).

<figure><img src="image_5.png" /><figcaption>Example 3: Web portal, MEETUPS prototype, geographic-based interactive exploration<fnref n="22" />.</figcaption></figure>

<footnote n="21">²¹ The Access pilot has not been considered either.</footnote>

<footnote n="22">²² See Marilena Daquino et al., D1.9. Polifonia Web portal – 1st version, Polifonia, 2022, section 4.4.1.</footnote>
While more interaction patterns and types of analysis could be relevant in the future, we believe that we achieved a reasonable coverage of requirements, which will allow us to share meaningful insights from pilot datasets that accommodate most desiderata<fnref n="23" />.

### 1.2.3 Survey.

The Survey provides detailed insights into the pilots' objectives, resources, methods, tools and progress. This tool is a central entry point to the components related to the pilots (including the Personas, Stories, and Competency questions (see below section 2)) and does not only allow to explore the individual pilots but also to identify the interconnections between them. As such, the Survey has proven to be a valuable tool for the coordination of the pilots and for their articulation within Polifonia.

The information was collected in four iterations of the Survey, listed in bold in table 1. The first iteration was carried out during the first weeks of Polifonia and aimed both at obtaining initial data and adapting/refining the questions. Iterations 2 (17/05/2021) and 4 (13/11/2021) are situated in the direct context of the two versions of the Roadmap which are central milestones in the update of pilot information. Iteration 3 (15/11/2021) follows on from the 2nd Project Meeting (Bologna, 13-15/10/2021) which allowed the pilots to refine their objectives and methodological choices. Independently of the completion of the Roadmap, it is planned to continue updating the Survey by correlating it with other important future moments for the pilots: the fifth Project Meeting (Paris, 22-25/09/2022) the pilot's mid-term evaluation (M24) and their final evaluation at M40.

<table><caption>**Table 1: Survey commits on GitHub.**</caption><thead><tr><td>Commit</td><td>Date</td></tr></thead><tbody><tr><td>Survey initial publication (March 2021)</td><td>07/05/2021</td></tr><tr><td>Update Survey after 2nd iteration (14 05 2021)</td><td>17/05/2021</td></tr><tr><td>Reorganized formats and standards in technical part</td><td>17/06/2021</td></tr><tr><td>Merged with DMP Annex 1, col F and G</td><td>17/06/2021</td></tr><tr><td>Minor format update</td><td>17/06/2021</td></tr><tr><td>New iteration: validation protocols and general update</td><td>15/11/2021</td></tr><tr><td>New iteration: validation protocols and general update</td><td>19/05/2022</td></tr><tr><td>Objective questions and validation questions updated</td><td>13/06/2022</td></tr><tr><td>Survey 4th iteration : validation and stakeholders</td><td>13/06/2022</td></tr></tbody></table>

a) **Domain-specific part.** The Survey's first part provides granular information on the pilot's objectives, results, potential innovations, input data, working hypotheses, knowledge produced, tools, concepts, controlled vocabularies (created or used), disciplines involved, etc. Since the first version of the Roadmap, the pilots have not only documented their objectives, main innovations and contribution to the state of the art, they have also detailed their achievements until now and estimated the progress made towards the realization of their overall objectives (see table 2). The goal of this part is thus to position the pilots in the current scientific landscape, to identify their ability to answer questions of central importance, and to assess their interest for the community of

<footnote n="23">²³ As confirmed by the evaluation with scholars described in D1.9. See Marilena Daquino et al., *D1.9. Polifonia Web portal – 1st version*, Polifonia, 2022.</footnote>
researchers, students, professionals, industries and amateurs who are interested in cultural heritage. Furthermore, the information collected allows us to monitor the pilots' progress: the pilots have already completed some important tasks – the constitution of their corpora (Musicbo), the creation of resources for the interconnection of collections (Interlink), the design of a theoretical model for the indexing of scores (Facets), etc. – and are all planning in line with the project's duration.

<table><thead><tr><td>Contribution to the state of art</td><td>What are the pilot's main innovations? / How do the methods and results contribute to the state of the art?</td></tr></thead><tbody><tr><td>MUSICBO</td><td>The main innovations of the pilot are: 1) creation of a corpus dedicated to the role that music has played for the city of Bologna; 2) extraction of new knowledge relating to the links between music and the history of the city has remained latent until now</td></tr><tr><td>TONALITIES</td><td>Tonalities aims to compare different (sometimes controversial) music analyses that MUST be referred to specific theoretical frameworks. In this way, the pilot innovates by contextualising theoretical and analytical information. The open linked data paradigm allows to articulate different large digital collections of notated music and to explore them through a quantitative-qualitative approach. By modelling different theories and applying them to musical works in a dedicated interface combining machine learning and human annotations, we can analyse larger score collections and deepen our understanding of them.</td></tr><tr><td>CHILD</td><td>Cultural studies in music: a systematic study of music and childhood in the reference period is still to be done. State of art knowledge extraction methods can partly be used (eg, named entity recognition) but new research is needed to support factual information extraction such as event extraction and characterization for the domain of interest.</td></tr><tr><td>MEETUPS</td><td>Contributes new tools and methods for building a KG of historical meetups. The resources contributes to historical musicology research.</td></tr><tr><td>ACCESS</td><td>By contributing new tools, frameworks, perspectives and an understanding of their effects.</td></tr><tr><td>BELLS</td><td>Building a formalized knowledge about the phenomenon, which is non available at the moment</td></tr><tr><td>INTERLINK</td><td>Largest collection available of MIDI files harvested from the Web, cleaned and aligned with the Million Song Dataset; Largest collection of open and high-quality chord annotations in different formats and notations with timing information. Tools for dataset enrichment and linking.</td></tr><tr><td>FACETS</td><td>The SotA on search-engine for musical scores is rudimentary, and was mostly started by our team. We aim at extending previous preliminary theoretical work and produce working implementations.</td></tr><tr><td>ORGANS</td><td>In the first place, the results will facilitate the research that is needed for restoration and maintenance of the organs. Results on trends and longitudinal developments will contribute to better understanding of the individual instruments, and provide a better understanding of history of pipe organ building.</td></tr><tr><td>TUNES</td><td>Methods for analysing music (melodies) at a higher abstraction level than single notes are lacking in Music Information Retrieval. We will contribute to that.</td></tr></tbody></table>

<table><caption>Table 2: Survey, domain specific part, main innovations, progress towards the pilot's objectives.</caption><thead><tr><td>Goals and objectives</td><td>What is the progress towards the pilot's objectives (please indicate %)</td></tr></thead><tbody><tr><td>MUSICBO</td><td>*Design and collection of corpus 100%<br/>*Modeling extraction of textual information 20%</td></tr><tr><td>TONALITIES</td><td>*Classify musical works into modal-tonal categories on the basis of an in-depth understanding of their inner organisation 10%<br/>*Develop and apply at least 5 different theoretical models to identify how the work's modal-tonal interpretation changes through the lense of a given theoretical framework; 40%<br/>*Compare modal-tonal attributions of a musical work; 10%<br/>*Look "inside" modal-tonal theories and compare them; 100%<br/>*Provide an argued, documented and authored modal-tonal analysis on the basis of the theoretical and analytical observations made.<br/>*Develop a semantic model for generic score collaborative annotation; 20%<br/>*Design & develop a Web interface and a Web service for collaborative score annotation; 20%</td></tr><tr><td>CHILD</td><td>*Develop a music history study: 30%<br/>*Discovery and collection of resources from books: 50%<br/>*Automatic curation of documentary evidence: 30%<br/>*UI for visualisation, exploration, and sense-making: 0%<br/>*Knowledge Graph of Music and Childhood: 20%</td></tr><tr><td>MEETUPS</td><td>User story: 100%<br/>Requirements: 50% (UI requirements missing)<br/>Ontology: 50% (first version, to be further developed)<br/>Knowledge Graph: 30% (first version focused on 1000 bios)<br/>Study in the music domain: 20% (focus on Edward Elgar)</td></tr><tr><td>ACCESS</td><td></td></tr><tr><td>BELLS</td><td></td></tr><tr><td>INTERLINK</td><td>Creation of new resources to facilitate and enable the linking operations (80%); Discovery of links at the entity level (10%); Discovery of links at the musical content level (33%);</td></tr><tr><td>FACETS</td><td>*Theoretical model for indexing scores efficiently: 90%<br/>*Features of music to extract: 50%<br/>*Faceted search: 0%<br/>*Software distribution: 30%</td></tr><tr><td>ORGANS</td><td></td></tr><tr><td>TUNES</td><td></td></tr></tbody></table>
Another field of work concerns the licenses and copyright of the sources on which the pilots are based. If the cultural heritage is in the public domain, the facsimiles, transcriptions, electronic editions – for example an edition of the works by Palestrina or a madrigal by Tasso – on which the pilots might focus, may be under copyright. The Survey allowed to collaboratively contribute to the identification of the licenses and copyrights of the collections taken into account and to design a contact scheme according to the different solutions encountered. The aim is to identify, in close connection with the DMP, the challenges that could arise in order to achieve the objective of making reusable the pilot’s outputs, including for commercial purposes by industries.

**b) Technical part.** The technical section reviews the formats and standards of the data and metadata used. It identifies the controlled vocabularies, the thesauri and ontologies, the interoperability/alignments envisioned, and the tools and algorithms developed. This part also highlights the (envisioned) overlaps between the pilots' expectations of the pilots towards the technology providers, and the needs addressed at the WP1 level. Finally, this part questions the pilots on the application they will release and on their expectations regarding the Web portal.

Since the first version of the Roadmap, the pilots have clarified the areas of shared work, for example between Musicbo, Meetups and Child; or Tunes, Facets, Interlink and Tonalities (see table 3). These interconnections crystallizing through the Survey and being exploited by the pilots – Interlink and Tunes have an active and ongoing collaboration, for example – play a central role in coordinating resources. They contribute to the goal not to juxtapose Polifonia's studies but to articulate them in a rich network that reconciles complementary perspectives.

In the past months, the pilots have also clarified their expectations towards the technology provider WPs (i.e. WP2-5) (see table 3). The expectations towards WP2 are mainly related to ontology design (Tonalities), methods and tools for data extraction from knowledge graphs (Meetups) or conversely the conversion of pre-existing data (textual and/or musical) into linked data (Interlink, Organs, Tunes). The pilots also aim to build on the expertise of WP2 for the extension and harmonization of metadata schemas (Bells), the design of algorithms for linking musical scores and metadata (Interlink) and the representation of knowledge graphs (Organs). The collaborations with WP3 are mainly focused on the extraction of musical patterns (Tunes), partly based on neural networks (Tonalities), as well as on the definition of musical patterns and contents (Facets). Conversely, WP4 is also involved in knowledge extraction from textual data (Musicbo, Child, Meetups, Organs). Finally, WP5 is expected to contribute to the design of interfaces (Tonalities, Child, Meetups) and to the design of general dashboards (Tunes). The relationships between pilots and technology providers, uncovered by the Survey, have played an important role in a) the collection of resources used in the pilots, b) the centralization of expectations (see section 2 below) and c) the mapping of disciplinary needs to technologies. In particular, the associations between pilots and WPs have led to weekly working groups which are now a driving force for the project (see section 3 below).
<table><thead><tr><th></th><th>Are there planned overlaps (already defined or envisioned) between the pilot's resources and other resources used within Polifonia?</th></tr></thead><tbody><tr><td>MUSICBO</td><td>*The sources of MusicBo and MEETUPS are complementary and can mutually enrich the two pilots.</td></tr><tr><td>TONALITIES</td><td>*Overlaps with TUNES, FACETS ?</td></tr><tr><td>CHILD</td><td>*The Polifonia Corpus (probably)</td></tr><tr><td>MEETUPS</td><td>*Sources from MusicBo may contribute data about Bologna and the CHILD pilot may include texts that could be used to extract encounters as well.<br/>*Shared interest in the theme of meetings between musicians: therefore it is possible to share a portion of the corpus that deals with the Meetups that took place in Bologna</td></tr><tr><td>ACCESS</td><td>*Possibly in WP8</td></tr><tr><td>BELLS</td><td>*Part of the ontology shared with organs (instruments, locations).</td></tr><tr><td>INTERLINK</td><td>*Yes (e.g. MIDI Linked Data cloud)</td></tr><tr><td>FACETS</td><td>*Yes, definitely with TONALITIES</td></tr><tr><td>ORGANS</td><td>Part of the ontology shared with Bells (instruments, locations).</td></tr><tr><td>TUNES</td><td>*There is envisioned overlap with TONALITIES and INTERLINK.<br/>*Also with FACETS.</td></tr></tbody></table>

<table><caption>**Table 3: Survey, technical part, planned overlaps between the pilots and expectations towards the technology providing WPs (WP2-5)**</caption><thead><tr><th></th><th>What do you expect in terms of technologies from the technical provider WPs?</th></tr></thead><tbody><tr><td>MUSICBO</td><td>WP4 - Knowledge extraction components; WP5 - UI design</td></tr><tr><td>TONALITIES</td><td>*WP2: Help in ontology design and alignment. Strategies for ontology-based reasoning<br/>*WP3: Help in the design/training of a neural network for rate analysis<br/>*WP5: Contribution to the design of the collaborative annotation interface.</td></tr><tr><td>CHILD</td><td>*Knowledge extraction research performed in WP4<br/>* UI research developed in WP5 for the exploration/curation of accounts<br/>*UI design</td></tr><tr><td>MEETUPS</td><td>*WP2 to provide methods and tools for extracting content from existing knowledge graphs (Wikidata/DBpedia/...) publishing linked data<br/>*WP4 to provide methods to extract encounters from Wikipedia and digitized texts<br/>*WP5 - UI design</td></tr><tr><td>ACCESS</td><td>We are our own technical provider, though we will collaborate with others within and outside the project, as applicable.</td></tr><tr><td>BELLS</td><td>*Data enrichment<br/>*Possible extension and harmonisation of metadata schemes<br/>*Data sources hosted by content service providers;<br/>*Map hosted by Polifonia portal<br/>*Publish/enrich LOD</td></tr><tr><td>INTERLINK</td><td>*Linked Data conversion of MIDI and other notations to RDF,<br/>*Ontologies for these notations<br/>*Linking algorithms between notations and metadata (e.g. related texts, events)</td></tr><tr><td>FACETS</td><td>*Guidelines so that we may easily integrate in the Polifonia ecosystem.<br/>*WP3: pattern definitions / dimensions of musical content relevant for search</td></tr><tr><td>ORGANS</td><td>*NLP: tailored named entity recognition, techniques to infer relations between the entities from the text.<br/>*KG: Support for extracting a knowledge graph from the data.<br/>WP2: knowledge graph representation<br/>*WP4: Natural Language Processing</td></tr><tr><td>TUNES</td><td>*WP3: technology to extract patterns from the musical contents.<br/>*WP2: technology to extract a knowledge graph from these extracted patterns.<br/>*Generic dashboard;<br/>*Convert dataset(s) to linked data; RDM: enriching existing database (?) similar to Bells; application possible in Portal (?); see BELLS on SLA's</td></tr></tbody></table>

A new field compared to the first version of the Roadmap concerns the description of the application that will be published by the pilots and their relationship to the Web portal (see table 4). The objective is to lead the pilots to sketch as soon as possible what will be pilot specific and part of their individual output and. A complementary goal is to exploit the canalizing potential of the Web portal as a shared environment to aggregate, articulate and interconnect the individual resources. The Survey thus aims to reinforce the transversal dimension of the project and to generate added value through the identification/documentation of inter-team collaboration.

The answers given show that while most pilots suppose an independent application (Musicbo, Tonalities, Access, Bells, Organs, Tunes, Facets) some of them do not exclude relying on the Web
portal only (Child, Meetups, Interlink). Independently, all the pilots agree on the need for a joined (yet not exclusive) presentation through the Web portal. The portal thus not only contributes to the coordination of resources. It also has a channeling effect on the technical and even methodological level, through the presentation of the pilots' resources.

<table><thead><tr><td></td><td>What is the expected application to be released for this pilot?</td></tr></thead><tbody><tr><td>MUSICBO</td><td>A knowledge map of Bologna, which shows semantic relations about music in some specific points of the city, as places, public or private building, churches and so on.</td></tr><tr><td>TONALITIES</td><td>Provide an interface for annotation and analytical argumentation. This interface would<br/>- Classify musical works into modal-tonal categories on the basis of an in-depth understanding of their inner organisation;<br/>- Apply different theoretical models to identify how the work's modal-tonal interpretation changes through the lens of a given theoretical framework;<br/>- Confront modal-tonal attributions of a musical work;<br/>- Look "inside" modal-tonal theories and compare them;<br/>- Provide an argued, documented and authored modal-tonal analysis on the basis of the theoretical and analytical observations made.</td></tr><tr><td>CHILD</td><td>A] A knowledge acquisition UI for listening experiences, linked to the LED<br/>B] A knowledge graph of listening experiences (LED+updates focusing on CHILD)<br/>C] A UI for thematic exploration of LEs, focusing on Childhood (->WP5)</td></tr><tr><td>MEETUPS</td><td>Website to allow the exploration of meetups through timeline and map<br/>Knowledge graph of musical meetups</td></tr><tr><td>ACCESS</td><td>UI for Poly query<br/>UI for Poly Browse<br/>UI for Harmony Browse</td></tr><tr><td>BELLS</td><td>- Map, to locate the bells<br/>- Facet research<br/>- Time period in layers<br/>- Customize the criteria/facets<br/>- Natural language question answering over knowledge graphs - queries in natural language translated in SPARQL<br/>- issue: can the data be enriched by users ?<br/>- issue: can we guide the user ?</td></tr><tr><td>INTERLINK</td><td>identify similarities in heterogeneous corpora</td></tr><tr><td>FACETS</td><td>Provide a faceted search engine for large collections of musical documents</td></tr><tr><td>ORGANS</td><td>The primary goal is to populate an existing database, developed by the NIVO (National Institute for Organ Art) with data extracted from the raw text of the organ encyclopedia.<br/>The NIVO already prepared a web-interface to the database: pipeorgan.nl<br/>Additionally a dashboard which allows exploration of the knowledge graph that is extracted from the full text of the encyclopedia. Various visualizations and querying methods could be included.</td></tr><tr><td>TUNES</td><td>- collection of tools<br/>- issue: who will build this application ?<br/>- issue: how do we represent melodic contents as KG ?</td></tr></tbody></table>

<table><caption>Table 4: Survey, technical part, expected applications and types of interactions supported by the Web portal.</caption><thead><tr><td></td><td>What type of interaction do you expect should be supported by the Web portal?</td></tr></thead><tbody><tr><td>MUSICBO</td><td>Browsing and querying of the Pilot corpus</td></tr><tr><td>TONALITIES</td><td>Exploration of the information on the works, navigation in the corpus, discovery of the available analyses, navigation towards the specific display resources of TONALITIES.</td></tr><tr><td>CHILD</td><td>Exploration of the information on the works, navigation in the corpus, discovery of the available analyses, navigation towards the specific display resources of CHILD.</td></tr><tr><td>MEETUPS</td><td>Exploration of the information on the works, navigation in the corpus, discovery of the available analyses, navigation towards the specific display resources of MEETUPS.</td></tr><tr><td>ACCESS</td><td>For this particular pilot, the portal will not be a major focus of interaction.</td></tr><tr><td>BELLS</td><td></td></tr><tr><td>INTERLINK</td><td>Note-by-note navigation, score representation, playable on demand, browse and search specific measures</td></tr><tr><td>FACETS</td><td>Search engine, with demo queries provided so the user can grasp the power of the tool.</td></tr><tr><td>ORGANS</td><td>No clear vision yet</td></tr><tr><td>TUNES</td><td>Maybe some sort of exploration tool.</td></tr></tbody></table>

c) Engagement and validation. The "engagement and validation" part has been subject to the most extensive adaptations. The answers related to user feedback show that the pilots are aware of the need to involve, from an early stage, external experts (see below section 2.2). The other area with the most significant extensions in this section concerns validation<fnref n="24" />. The pilots were asked to specify their success criteria, to identify validation methods, to define a validation setup (see table 5), and to establish KPIs/metrics. The Survey shows that the pilots see success criteria in the quality of the data (Meetups), the networking of information from different sources and/or formats (Tonalities, Interlink, Organs, Tunes) and the speed and accuracy of task execution (e.g. counting

<footnote n="24">²⁴ While this section gives an overview of the information on validation provided by the pilots, the actual Validation framework will be discussed in section 4.</footnote>
musical patterns in large corpora or identifying textual resources) (Tonalities, Child, Interlink, Facets, Organs). At the same time, the pilots emphasize the need to create innovative and ergonomic tools (Musicbo, Tonalities, Bells, Facets, Organs), made available to a wide audience and involving new forms of engagement and accessibility (Access).

The validation criteria that emerge from the Survey are strongly oriented towards validation by experts and/or target audiences. This concerns in particular the pilot's objectives, the input data, interfaces and results. The KPIs reflect the importance given to the target users satisfaction (in terms of objectives, methods, interfaces and results), the number of scholarly activities supported, and the use of the tools in educational situations or by cultural heritage institutions. These indicators are complemented by quantitative and statistical approaches that focus on the measurement of responses based on f-scores, the number of items processed, the size of KGs or the speed of query processing.

<table><caption>**Table 5: Survey, Engagement and validation part, validation setup.**</caption><thead><tr><th></th><th>What will be the validation setup?</th></tr></thead><tbody><tr><td>MUSICBO</td><td>Validation of a case study (one buildings/place) by a musicologist.</td></tr><tr><td>TONALITIES</td><td><ul><li>Component 1: Goals and objectives: Do the objectives of the pilot correspond to real musicological needs? Do they take into account suggestions made by musicological colleagues?</li><li>Component 2: Analytical results: Does the theoretical model allow for the correct inference of modal attributions? - Use a test corpus (c. 150 works) and check whether the attributions made by the theorists can be inferred from the models. Use the justification for the modal inference produced by the reasoner as metrics?</li><li>Component 3: Data processing: Is the analytical information provided by user(s) or computed by algorithms correctly stored and retrievable? - Use a test corpus of multi-layered analyses (c. 15 works which have more than 1 analysis) and check if the information layers are stored correctly</li><li>Component 4: User engagement / ergonomics: Does the music annotation environment meet usability requirements? Can de CQ be answered with the interface and tools provided?</li></ul></td></tr><tr><td>CHILD</td><td><ul><li>Quantitative: ability of the system to produce accurate recommendations to curators</li><li>User study on usability of thematic explorer</li><li>User study on assisted formulation of research hypotheses</li></ul></td></tr><tr><td>MEETUPS</td><td><ul><li>User study with domain experts on how the tool can support scholarship</li><li>User study on the quality of the meetups KB</li></ul></td></tr><tr><td>ACCESS</td><td><ul><li>Usability, accuracy, user feedback</li><li>Quantitative: ability of the UIs to pass accurate well-formed query to back ends</li></ul></td></tr><tr><td>BELLS</td><td><ul><li>Validate the methodology: 10 case-studies to implement a methodology and good practices to build a complex knowledge system (ex: links between bells and sounds, bells and bell tower, performing events, archival documentation, playing techniques).</li><li>N19-interviews for validating the interface</li><li>Check that the corpus is cleaned, consistent through experts</li></ul></td></tr><tr><td>INTERLINK</td><td><ul><li>Quantitative evaluation of links according to clustering metrics and ground-truth data;</li><li>Quantitative evaluation of music similarities based on MIR benchmark tasks;</li><li>Qualitative evaluation of both, through the involvement of collection curators/experts and musicologists respectively.</li></ul></td></tr><tr><td>FACETS</td><td><ul><li>Technical aspects: Information Retrieval metrics (precision, recall, accuracy), computational time to execute queries. Identification of hard/specific score problems for retrieval.</li><li>Evaluation setup with a survey from experts, for the UI part.</li></ul></td></tr><tr><td>ORGANS</td><td><ul><li>Component 1. Measurement how much of the information from the encyclopedia has been transferred correctly to the NIVO database. Component 2. We will perform a user test, comparing the time certain queries take manually versus with the dashboard.</li></ul></td></tr><tr><td>TUNES</td><td><ul><li>Output: Validate the output by using the existing manual annotation of tune origins as ground truth.</li><li>Interface: User study with domain experts on how the tool can support scholarship.</li></ul></td></tr></tbody></table>

An important challenge for the pilots is to establish individual validation protocols based on the identified success criteria, methods and KPIs. At this stage, these protocols combine between one and five components that cover the areas of objectives, methods, data processing, usability, and
results. All this information on validation has been decisive in the implementation of a global Validation framework for Polifonia which will be described in section 4.

## 1.3 Lessons learned

### 1.3.1 Ecosystem.

The first release of the Polifonia Ecosystem was the first time the consortium experimented with the collaborative methodology and, therefore, collected feedback from project partners. Such feedback was informally collected during TB meetings. The general approach has proven effective in that it allowed to decouple the effort required in developing the software output of the specific research activities without enforcing strong dependencies between developers and their related workflow. In addition, the common annotation schema seemed to add a sustainable overhead to the effort required to prepare the Ecosystem release. The majority of the components released in the first version were successfully composed into the Ecosystem Website and required minimal effort in metadata curation. The success of this methodology brought to the attention new opportunities. Specifically, we plan to extend the annotation schema to integrate metadata relevant to describe the pilot Validation framework (see below section 4) and formally describe the licensing and terms of use of the components. Finally, we plan to enrich the description of *how* the components interact with each other.

### 1.3.2 Web portal.

The co-design of interfaces and methods for promoting results of pilot-driven research has proven to be effective in a) preventing the lack of clear goals in the production of data sources, highlighting the importance of final users, and b) defining impact-driven application studies, such as designing actual data stories that exemplify the added value of produced data. Moreover, the continuous interaction with partners on the preliminary analysis of Stories and CQ allowed us to increase self-awareness of the pilots' status, revising expectations and objectives.

### 1.3.3 Survey.

The Survey has proven to be effective in coordinating the resources, methods and tools used in the pilots. In conjunction with the Web portal and the Ecosystem, it has been valuable for setting up a methodological framework within which the pilots can evolve. Its role is not restricted to collecting information, but also to giving impulses and steering methodological work. The Survey stimulates the pilots by asking them to specify their methodology, their objectives and the way they intend to involve experts. It also inspired the DMP, which worked on its elaboration and, based on this, identified more precisely some of the pilots' needs (e.g. fine-tuned analysis of licenses and copyright of external resources). Indirectly the Survey also stimulated the development of complementary tools - i.e. the Stories - to capture, from a discursive and narrative angle, the individual orientations, needs and requests of the pilots (see below section 2). The Web portal both stimulated and benefits from the responses obtained for the identification of the pilots' needs and can potentially use the information collected for the publication of pilot "identity cards". Finally, the Survey contributes to the metadata of the Ecosystem in the context of the Validation framework (see above in this section and below section 4). As such, this tool contributes to stimulating the coordination of resources, methods and tools.
Another contribution lies in the ability of the Survey to network the answers provided and thus to create an interconnected map of methods, tools and resources. The ongoing conversion of the Survey into a knowledge graph will make this map even more browsable. In particular, it will allow for a more systematic exploration of the conceptual interconnections that arise between the pilots while providing access to the pilot's DNA by delivering highly granular information about them.

The Survey could in principle be completed with this final version of the Roadmap. However, it is planned to continue feeding and adapting it until the end of the project. On the one hand, our plan is to document the ongoing updating and validation process of the pilots (at M24 and M40). On the other hand, the Survey, once transformed in a knowledge graph, will provide an interconnected and permanent entry point to all the information and resources relating to the pilots. This entry point will allow to check the status of the pilots and their related resources (data, KGs, ontologies, software bricks, etc.).

Beyond, the questions and answers of the Survey will be explored and updated in the context of other WPs and deliverables. The second version of the "Methods for interlinking knowledge graphs" (D2.5 M36)<fnref n="25" /> may exploit the information available to discover links between music objects and contexts in Polifonia. The second version of the "Plan for Exploitation, Dissemination and Communication of Results"<fnref n="26" /> will be able to use the Survey to produce both a quantitative and qualitative report on Polifonia's pilot-related results achieved compared to the objectives set. Finally, the technical questions of the Survey will be taken up in the next two iterations of the DMP (D7.2²⁷ and D7.3²⁸) which will focus on FAIR principles. In this context, the Survey will be an important cornerstone for documenting the pilot's individual relation to FAIRness.

Despite all these contributions and outcomes, it must be kept in mind that the Survey is based on a mainly static interaction with the pilots: they are invited to answer questions that have been formulated beforehand and that are only adapted during new iterations. While the tool is therefore suitable for the methodological generalization that this deliverable aims to produce, it has to be supplemented by other tools that are more adaptable to the individual pilots and their narratives.

## 2. Centralizing needs and requirements

The centralization of needs and requirements happens at two levels within the pilots. On the one hand, a central issue is to actively involve external stakeholders as early as possible in the design and validation process. The objective here is to ensure that the pilots reflect real expectations of the communities related to cultural heritage, that they are used by these communities, that they can be adapted to similar problems, and that they will survive the project itself. On the other hand, at

<footnote n="25">²⁵ D2.4 aims at providing a technical description of the method to discover links between music objects and contexts in Polifonia and external datasets, at the schema and data levels.</footnote>

<footnote n="26">²⁶ D6.2 contains a first quantitative and qualitative report on planned and completed communication and dissemination activities, success measures against the stated goals, and recommended modifications to the original strategy. Also, the initial Exploitation and Business Plan outlines the exploitation activities planned for during the project and an initial exploitation and business plan based on initial feedback from industry experts.</footnote>

<footnote n="27">²⁷ D7.2 consists of an updated version of a Data Management Plan (DMP) to support the management of the outputs of the project during the development of the project, and after the project has ended.</footnote>

<footnote n="28">²⁸ D7.3 is the Final version of a Data Management Plan (DMP). Its objective is to support the management of the outputs of the project during the development of the project, and after the project has ended.</footnote>
the internal level, the objective is to identify the technological needs of the pilots. More precisely, this involves a) defining the structure of the Polifonia Ontology Network (PON) based on the needs of the pilots and b) identifying and defining the queries that the pilots would like to address to the Polifonia Knowledge Graph (PKN). The Survey helps to identify some of the pilot requirements and expectations. However, a more discursive and interactive approach has been designed to enter into dialogue with the pilots and to identify as closely as possible the scientific questions and the technical needs and requirements that they imply.

## 2.1 Methods and strategies

Polifonia's overall approach for centralizing needs and requirements is based on agile methodologies, eXtreme Design (XD) in particular<fnref n="29" />, and focuses on the definition of Personas, Stories and CQ. While Personas are user profiles (researcher, music lover, physically impaired person, etc.) characterized by their occupations, knowledge and interests, Stories describe the general objectives and the particular scenario of the research undertaken by this person<fnref n="30" />. The CQ are the queries that the Persona seeks to answer in the Story<fnref n="31" />.

A central concern of this methodology lies in its ability to create an interface between the disciplinary and computational dimensions of modelling: Queries are first formulated through Stories in natural language by domain experts (musicologists, historians, linguists, etc.) before being modelled by the OD team. Disciplinary technicality and computer technicality are thus distributed without being isolated. The formulation of queries in natural language by the domain experts ensures that the modelling is really driven by disciplinary problems, questions and needs. Conversely, the transposition of the queries into computer language by the ontology design (OD) team ensures that the modelling complies with current computer practices and standards. Since the approach was described in detail in D1.1³², the focus here will be on the methodological changes, the results obtained and the lessons learned since then.

To facilitate the OD activities, increase the usability of their outputs by the pilots, and articulate even more effectively the disciplinary and technological dimensions of the modelling process, the OD workflow has been extended during the last months. The new framework removes some complexity from the Story creators, while leveraging their domain expertise/experience to conveniently contextualize the specific technical scope of a Story. In particular, when defining (or refining) a Story, the creator should only focus on explaining the scope and goals of the Story itself – which task the Persona intends to achieve – and most importantly, provide practical examples and/or data to comprehensively explain the goals, needs, and artefacts of the Story. On the other hand, the Story creator is freed from defining any CQ – a responsibility which is entirely up to the OD team. Therefore, the resulting methodology is example- and/or data- driven. This has clear advantages,

<footnote n="29">²⁹ Kent L. Beck and Cynthia Andres. *Extreme programming explained - embrace change*, The XP series, Addison-Wesley, 2005²; Russ Unger & Carolyn Chandler. *A Project Guide to UX Design: For user experience designers in the field or in the making*, New Riders, 2012.</footnote>

<footnote n="30">³⁰ The current Story template used in Polifonia is available in appendix 5.</footnote>

<footnote n="31">³¹ On the status and role of QC in the design process, see Michael Grüninger & Mark S. Fox, "The role of competency questions in enterprise engineering", in *Benchmarking - Theory and practice*, ed. by A. Rolstadås, Springer, 1995, pp. 22-31.</footnote>

<footnote n="32">³² See Christophe Guillotel-Nothmann et al. *D1.1. Roadmap and pilot requirements 1st version*, Polifonia, 2021, section 1.1.</footnote>
as it: a) does not bias the definition of the CQ, b) promotes their consistency and reuse in other/related Stories (not necessarily from the same Persona), c) allows for the definition of objective and realistic ontological requirements, and d) makes the resulting ontologies more usable, as they are aligned *apriori* with the specific ontological needs of the pilot. On a larger scale, the consistent integration of CQ across pilots also contributes to the modular reuse of the ontology network and naturally achieves a level of interlinking among the pilot-specific modules.

To initiate the new workflow, the INTERLINK pilot has introduced a new Persona, Linka<fnref n="33" />, and a new Story<fnref n="34" /> in collaboration with the OD team. As can be seen from example 4, reporting an excerpt of the new Story (*Linka#1_MusicKnowledge*), a new section "Example Data" provides both examples of relationships pertaining to the Story (e.g. "Penny Lane is a musical work by The Beatles") as well as relevant links to online data (e.g. MusicBrainz, Wikidata). The "Cataloguing information" section provides a number of basic assertions in natural language: "Penny Lane" is a musical work by "The Betales", "Penny Lane" was composed by "John Lennon" and "Paul McCartney", etc. These assertions constitute the starting point from which more detailed discographic explorations are undertaken: The album "Strawberry Fields Forever" has been originally released in the UK on 1967-02-17, The background and lead vocals of "Penny Lane" are by "Paul McCartney", etc. Based on the example data, the OD team produces de QCs that are the point of departure for the actual modelling. The modelling itself is then submitted to a testing tool and protocol that identifies inconsistencies and ensures that the formalization actually meets the requirements<fnref n="35" />.

<footnote n="33">³³ https://github.com/polifonia-project/stories/tree/main/Linka:%20Computer%20Scientist.</footnote>

<footnote n="34">³⁴ https://github.com/polifonia-project/stories/blob/main/Linka:%20Computer%20Scientist/Linka%231_MusicKnowledge.md.</footnote>

<footnote n="35">³⁵ The protocol and testing tool are extensively described in Meroño-Peñuela, Albert et al. D.2.7. Ontology testing and evaluation report; knowledge graph documentation and access interfaces – 1st version, Polifonia, 2022.</footnote>
## Example Data

For simplicity, music-related data can be categorised as: cataloguing information, musical features, musical facts, and musical relationships.

### Cataloguing information

Information needed to identify a musical work as well as its relationships with the recordings that are derived from it. For example:

* "Penny Lane" is a musical work by "The Beatles",

* "Penny Lane" was composed by "John Lennon" and "Paul McCartney";

* The first recording of "Penny Lane" was released in February 1967 as a double A-side single with "Strawberry Fields Forever".

Linka is interested in tracing the basic discographic information associated to the recordings of a certain musical work, including: the year of release, the name of the release, the place of recording, the instruments/performers relationships, and the vocals. Example data to address this category can be found from MusicBrainz and Wikidata.

* The specific track of "Penny Lane" featured in the album "Strawberry Fields Forever" was recorded in "Abbey Road Studios: Studio 2" in "St John's Wood, Westminster, London (UK)" on 1966-12-29, 1967-01-05, 1967-01-06, 1967-01-09, and 1967-01-17.

* The album "Strawberry Fields Forever" has been originally released in the UK on 1967-02-17.

* The background and lead vocals of "Penny Lane" are by "Paul McCartney".

* The "tambourine" is played by "Ringo Starr".

## Example 4: Excerpt of the Story Linka#1_MusicKnowledge.

The involvement of external stakeholders in the design of the pilots is promoted at two levels – a) the Survey, b) the Validation framework – through the tools set up in the Roadmap.

a) The Survey has put in place questions that aim not only to identify the communities that are relevant to the pilot, but also the (potential) partner institutions and their involvement, the way user feedback is considered, and how users are to be recruited/solicited. The questions clarify the relationship between the pilots and the stakeholder networks, address the nature of the public demonstrators that may be made available at an early stage, and ask the pilots to identify stakeholders from different institutional backgrounds and countries<fnref n="36" />.

b) The Validation framework that the pilots will apply for the mid-term (M 24) and final (M 40) validation places user feedback at its centre. Evaluation by cultural heritage communities – both amateurs and experts (researchers, teachers, students, curators, interpreters) – plays a key role for almost all pilots for the validation of a) the tools and means implemented and b) the pilot's objectives and results (see below section 4.2). The Validation framework is thus a key tool for collecting feedback from stakeholders and actively involving them in the design of the pilot and the assessment of the results produced.

## 2.2 Results

In all, the Polifonia pilots have produced 19 Personas, 28 Stories, and more than 190 CQ. Example 5 makes explicit the links that exist between a) the pilots, b) the Stories and c) the Personas. The Personas with their related Stories focus on varied problems that the pilots aim at addressing such as organ building (Amy#2_OrganBuilders<fnref n="37" />), experiencing live music with the help

<footnote n="36">³⁶ See also above section 1.</footnote>

<footnote n="37">³⁷ https://github.com/polifonia-project/stories/blob/main/Amy:%20Organologist/Amy%232_OrganBuilders.md.</footnote>
of haptic technology (Anna#1_HearingMusic<fnref n="38" />), finding information about a composer related to the city of Bologna (Carolina#1_SourcesCrossAnalysis<fnref n="39" />), the socio-cultural context of musical performances (David#1_MusicHistorian<fnref n="40" />), compiling music programs for festivals (Keith#1_MusicConnections), the relationship of Dutch popular music to French opera (Mark#1_FolkMusic<fnref n="41" />), etc.

While the pilot with its particular scientific problem is the main entry point for the creation of a Persona with its specific Story and QC, it is common for these to acquire a higher level of generality later on. Thus, the Persona of Sethus, Music Theorist is used by both Tonalities and Facets, the Story Ortenz#1_MusicAndChildhood is used by both Meetups and Child, etc. This pooling is becoming even more systematic at the level of individual QC. From a methodological point of view, the approach thus allows for the sharing and reuse of requirements.

<figure><img src="image_6.png" /><figcaption>Example 5: Polifonia's pilots in relation to the Persona and Stories.</figcaption></figure>

<footnote n="38">³⁸ https://github.com/polifonia-project/stories/blob/main/Anna:%20Hearing-impaired/Anna%231_HearingMusic.md.</footnote>

<footnote n="39">³⁹ https://github.com/polifonia-project/stories/blob/main/Carolina:%20Music%20Historian/Carolina%231_SourcesCrossAnalysis.md.</footnote>

<footnote n="40">⁴⁰ https://github.com/polifonia-project/stories/blob/main/David:%20Music%20Historian/David%231_MusicHistorian.md.</footnote>

<footnote n="41">⁴¹ https://github.com/polifonia-project/stories/blob/main/Mark:%20Computational%20Musicologist/Mark%231_FolkMusic.md.</footnote>
Besides their utility per se, the new Persona and Story introduced by the INTERLINK pilot were intended to provide an example following the new data-/example- driven methodology. All the other pilots were thus invited to transition accordingly, by revising their current Stories based on this example. This was also done during the WP2 Knowledge graph pilot clinics - a forum organized by WP2 where the OD team interacts with a specific pilot every week (see section 3, below).

During the initial phase of the project, the collection and updating of the Stories and their requirements was done asynchronously and according to the rhythm of each individual pilot. As the project progressed, a number of guiding elements were introduced to regulate the pace of requirements updating, such as the Maninpasta, pilot clinics and weekly cross-WP working groups (see section 3 below).

It is worth remarking that the use case of Linka already intersects common ontological needs from all the other pilots, and this has currently brought the OD team to focus on such requirements. However, out of all 19 Personas, Linka is still the only one that fully aligns with the new methodology. This suggests that the transition remains difficult, as the other pilots are still working on addressing this change.

To make the transition more fluid, a number of measures have been adopted in recent weeks. In particular, a Maninpasta organized on 16 June 2022 has contributed to refine the new workflow and to specify its four objectives: 1. making sure that ontologies check all the quality boxes, 2. making sure that pilots follow the XD methodology, 3. ensuring that pilot ontologies are integrated within PON, and 4. improving efficiency in the OD. According to these objectives, the foundations have been laid for the publication of an OD Rulebook that will provide guidelines for implementing the workflow and publishing ontologies. It seems reasonable to expect that the OD Rulebook together with the ongoing weekly working sessions (see below section 3) will foster the transition and lead to the intermediate (M24) and final (M40) validation of the pilots, which will definitively endorse the collection and validation of requirements.

Regarding the centralization of external needs and expectations, the pilots have clarified their strategies for involving external collaborators through their responses to the Survey (see table 6). This involvement is done through the validation of user scenarios (Tonalities, Child, Meetups), interfaces and/or applications (Tonalities, Child, Meetups, Bells, Interlink, Facets, Organs and Tunes) or the results produced (Interlink, Facets). It also becomes clear from the answers that the pilots have the ambition to have the Stories validated by experts (Tonalities, Child Meetups). Finally, some pilots wish to involve experts in the creation of controlled vocabularies (Bells), the identification of musical research patterns (Facets) or the identification of analytical criteria to take into account (Tonalities). To collect user feedback, adapt the tools and bring them in line with the real needs of the targeted communities, the pilots were encouraged to make demonstrators available at an early stage. The pilots thus plan beta versions of applications (Musicbo), interfaces with reduced functionalities (Tonalities, Meetups, Bells, Organs), dashboards (Organs and Tunes), case studies (Interlink) or docker versions of the tools created (Facets). These demonstrators will be made available in the coming months, some of them before summer 2022, the other during the 5th project meeting in September in Paris or during the European Research Night in September.
<table><thead><tr><th colspan="2">Beyond the lead institution, who are the partners of the pilot and what is their involvement?</th></tr></thead><tbody><tr><td>MUSICBO</td><td></td></tr><tr><td>TONALITIES</td><td>CNAM: NEUMA score library made available and adapted to include copyright/licence information.<br/>KCL: Providing support for the modelling of theories.<br/>OU: Draft collaborative annotation interface mock-up;<br/>Identification of possible representation types/modes to be adopted; Critical discussions on theoretical models.</td></tr><tr><td>CHILD</td><td></td></tr><tr><td>MEETUPS</td><td></td></tr><tr><td>ACCESS</td><td></td></tr><tr><td>BELLS</td><td>*local superintendences of Ministry of Culture (Liguria, Emilia Romagna)<br/>*Association of bells ringers (Associazione Campanari Liguri, Genova Carillons).<br/>*Other agreements are under preparation with: other local superintendences (Molise) foundries (Pontificia Fonderia Marinelli); association of experts in campanology (Associazione Italiana di Campanologia)</td></tr><tr><td>INTERLINK</td><td>*NISV at the moment, and eventually CNAM for NEUMA as well as KNAW for the TUNES collections</td></tr><tr><td>FACETS</td><td>*Iremus-CNRS (already in the Neuma platform),<br/>*NUIG<br/>*BNF.<br/>*Comment: The pilot is still in a very early development stage and the involvement of other partners is currently limited. They will be solicited in the next months.</td></tr><tr><td>ORGANS</td><td>*Nationalea Instituut voor de Orgelkunst (National Institute for Organ Art): Data provider. Maintainer of target database. Publisher of the Organ Encyclopaedia.</td></tr><tr><td>TUNES</td><td>No direct partners</td></tr></tbody></table>

<table><thead><tr><th colspan="2">How is user/expert feedback to be taken into account in the pilot?</th></tr></thead><tbody><tr><td>MUSICBO</td><td>We plan a validation of the QC and the mock-up by external experts</td></tr><tr><td>TONALITIES</td><td>We plan a validation of the QC and the mock-up by external experts, specialists in music analysis and theory. We actively involve experts in the identification of analytical patterns/criteria to be implemented)</td></tr><tr><td>CHILD</td><td>We plan a validation of the QC and the mock-up by external experts</td></tr><tr><td>MEETUPS</td><td>We plan a validation of the QC and the mock-up by external experts</td></tr><tr><td>ACCESS</td><td></td></tr><tr><td>BELLS</td><td>We are working on vocabularies with experts and practitioners; we plan a validation of the pilot (vocabularies, mockup, application) by experts and practitioners.</td></tr><tr><td>INTERLINK</td><td>Quantitative and qualitative evaluation is planned for assessing the quality of links established.</td></tr><tr><td>FACETS</td><td>We plan a validation of the pilot (UI, search quality) by external experts. We aim to involve Polifonia experts (through WP3) in selecting relevant similarity measures and patterns for the search engine.</td></tr><tr><td>ORGANS</td><td>We are building a network of organ advisors and organists, who we will invite to evaluate interfaces. We also plan to do a broader user study, recruiting organ enthusiasts through organ new sites and organ fora and mailing lists.</td></tr><tr><td>TUNES</td><td>We plan evaluation of the dashboard with expert users (musicologists).</td></tr></tbody></table>

<table><caption>**Table 6: Survey, engagement and validation part, partner institutions, user/expert feedback and public demonstrators.**</caption><thead><tr><th colspan="2">What kind of public demonstrators of the pilot could be made available in the coming months?</th></tr></thead><tbody><tr><td>MUSICBO</td><td>We plan a first demonstration of the Pilot - through a beta version of the app - during the European researchers' night in September</td></tr><tr><td>TONALITIES</td><td>An alpha version of the interface with limited annotation features.</td></tr><tr><td>CHILD</td><td>None, the UI is the Listening Experience Database</td></tr><tr><td>MEETUPS</td><td>An alpha version of the interface with limited features.</td></tr><tr><td>ACCESS</td><td></td></tr><tr><td>BELLS</td><td>A limited number of complete examples (relation between bell towers/bells/sound recordings/performing practices) through the production of catalogue sheets: A first version of the interface and with limited data extracted.</td></tr><tr><td>INTERLINK</td><td>Tracing potential source of influence of The Beatles through music similarity. Experiments in progress.</td></tr><tr><td>FACETS</td><td>A standalone docker version of a preliminary version of the search engine will be released before Summer 22. It will serve as the basis of future releases (e.g., with faceting).</td></tr><tr><td>ORGANS</td><td>Data extraction is mostly done. We can provide a preliminary interface to explore the extracted knowledge graph. Also a prototype of the dashboard can be presented.</td></tr><tr><td>TUNES</td><td>Prototype of the dashboard.</td></tr></tbody></table>

The pilots were also encouraged to define their relationship to the stakeholder networks. Through the Survey, they were invited to explain how their work relates to the network and to add/confirm the stakeholders with whom they (plan to) collaborate (table 7). The answers given show that the pilots target collaborations with an important network of institutions (IRISA, EPFL), infrastructures (Clarin) and individual actors in the academic world. Public institutions in the field of culture (The Stables), cultural heritage institutions (BnF, Soprintendenza Archeologia Belle Arti e Paesaggio di Bologna, Museo internazionale e biblioteca della musica di Bologna, etc.) and/or institutions focusing on inclusion (Drake music) are also at the heart of the network. Finally, the pilots, and especially Interlink, cultivate fruitful relationships with industrial stakeholders (Muziekweb, Discogs, Songfacts, WhoSampled). Through the Survey (and the Validation framework, see section 4 below), the Roadmap thus contributes to coordinating and intensifying the interactions between Polifonia and the outside world.

<caption>**Table 7: Relations to the stakeholder network and stakeholder involvement in the definition/validation of the pilot**</caption>

It is not possible, at this stage, to comment on the success of the involvement of experts in the validation of the pilots. However, we can observe from the answers given that the pilots have all planned user validation, and that they have established and put into practice strategies for the recruitment and involvement of stakeholders.
## 2.3 Lessons learned

As components of the Roadmap, the Survey and the Validation framework stimulate the involvement of stakeholders and the centralization of external feedback. Independently of the finalization of the Roadmap with this deliverable, WP1 will continue, through the Survey and the fine-tuning of the Validation framework (see below section 4 and conclusion) to foster this involvement and to bring it into practice. To this end, we plan from September of this year to organize working sessions gathering the pilots and the stakeholders around questions dedicated to the pilot's review and validation.

The transition to the new methodological workflow for OD still requires efforts from all pilots to enable a more fruitful collaboration with the OD team, which, besides the advantages highlighted in Section 2.1, is fundamental to guarantee that the XD principles and methodology^42 relying on pattern-based design^43 are consistently followed. Without such level of involvement, the integration of all the pilot-specific ontologies within the PON, and the resulting creation/inter-linking of the Polifonia KG are expected to be daunting, and potentially complex. Therefore, additional efforts will be made to implement the desired transition through additional Maninpasta sessions, where pilots sharing similar ontological needs (which were identified after the first series of the WP2 pilot KG clinics) will be invited in a dedicated forum to advance the status of their Stories. These sessions will also be an opportunity to specify common guidelines for updating ontologies and pushing them towards the expected standards.

# 3. Mapping objectives and challenges from pilot specific domains to the technology provider WPs

## 3.1 Goals, methods and strategies

In Polifonia's overall architecture, as stated in the Grant Agreement, the 10 pilots are divided into four groups^44. Through their associated task and deliverables, these groups constitute a driving force for the realization of four major main objectives in relation to musical heritage: 1. preserve, 2. manage, 3. study musical heritage and 4. interact with. Additionally to the work within these groups, a tool called Maninpasta (which literally means hands in the dough in Italian) was developed at an early stage of the project to move from theoretical to practical work and to foster interdisciplinary synergy during half day events. These Maninpasta events involve researchers and developers that engage in collaborative task specific activities. They are organized on a periodical basis in the manner of hackathons with the important difference that they provide a forum not only to solve technical problems but also to discuss disciplinary issues. As such the Maninpasta contributes to articulating the disciplinary and technical components of the modelling tasks by creating a space for interdisciplinary dialogue and creation.

<footnote>^42 Eva Blomqvist et al. "Experimenting with eXtreme Design", in *Proceedings of EKAW 2010*, ed. by P. Cimiano, H.S. Pinto, Springer, 2010, pp. 120-134.</footnote>

<footnote>^43 Eva Blomqvist et al. "Engineering Ontologies with Patterns - The eXtreme Design Methodology", in *Ontology Engineering with Ontology Design Patterns, vol. 25. Studies on the Semantic Web*, ed. by P. Hitzler, A. Gangemi, K. Janowicz, A. Krisnadhi, V. Presutti, IOS Press, 2016, pp. 23-50; Valentina Anita Carriero et al. "Pattern-based design applied to cultural heritage knowledge graphs", «Semantic Web», 12/2 (2021), pp. 313-357.</footnote>

<footnote>^44 See appendix 3.</footnote>
In this decisive phase of the project, where technologies are tested, results become concrete and validation protocols are formalized, the pilots need to interact almost daily with the technology provider WP (i.e. WP 2-5). There is also a need to foster collaboration between pilots whose areas of exploration, and therefore technological needs/solutions, have affinities. This is why the 4 pilot groups and the Maninpasta have been complemented by weekly cross-pilot-WP WGs (table 8). These groups bring together pilots and different WP to promote regular technological development and methodological reflection, tailored to the pilots, and driven by their needs. The pilots focusing on music data (Tunes, Facets, Tonalities, Interlink<sup>45</sup>) interact weekly with WP3 and WP2 while the pilots exploring textual data (Musicbo, Bells, Meetups, Child) collaborate weekly with WP4 and WP2. In addition, two "generalist" groups have been created. The first, "pilot clinics", is dedicated to the needs and challenges encountered in the creation of individual knowledge graphs. The issues raised by the pilots (2 or 3 pilots during each weekly session) can be either methodological (e.g. modelling, relevant vocabularies and ontologies, etc.) or technological (e.g. how to use tools, infrastructures, etc.). The second "generalist" group brings together WP5, WP1 and all the pilots around issues related to the Web portal. This group plays a key role in reconciling the specific objectives and challenges of individual pilots with a common presentation and exploration environment. The group thus helps to make the pilots' objectives and challenges explicit and to identify corresponding technological solutions.

<table><caption>**Table 8: Cross pilot-wp-groups.**</caption><thead><tr><td>Work package</td><td>Pilot(s)</td></tr></thead><tbody><tr><td>WP2</td><td>All pilots ("Pilot clinics")</td></tr><tr><td>WP2 + WP3</td><td>Music-related pilots</td></tr><tr><td>WP2 + WP4</td><td>Text-related pilots</td></tr><tr><td>WP5</td><td>All pilots</td></tr></tbody></table>

The following sections show how these four groups, which constitute an important methodological innovation compared to the previous version of the Roadmap, contribute with the Maninpasta to translating objectives and challenges from pilot specific domains to the technology provider WPs.

**WP2 Knowledge graph pilot clinics.** One of our major goals is the creation of an ontology network and a knowledge graph that build on the domain knowledge and datasets coming from the pilots. An appropriate methodology for constructing knowledge graphs has been, however, subject to much study, and various different approaches exist<sup>46</sup> to create knowledge graphs from existing data. Size and complexity are key factors to consider: for small and regular KGs off-the-shelf tools

<footnote n="45"><sup>45</sup> While the Interlink pilot, focusing on music data, relates to this group, its objective of interconnecting collections grants it a more general status. For this reason, the pilot is dealt with mainly in the pilot clinics (see below).</footnote>

<footnote n="46"><sup>46</sup> Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d'Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane et al. "Knowledge graphs." *Synthesis Lectures on Data, Semantics, and Knowledge* 12, no. 2 (2021): 1-257.</footnote>
or custom code can be sufficient,<fnref n="47" /> while for large-scale (millions of items, thousands of contributors), multi-domain KGs, such as Wikidata, a distributed platform with a purposely defined user interface is necessary⁴⁸. The purpose of the WP2 KG pilot clinics is to understand pilot requirements and support the decision-making processes behind tool selection and customization for the creation of musical KGs from pilot data. In these clinics, experts in the area of knowledge graph creation visit the pilots cyclically (see table 9), diving into their datasets and data challenges, and suggest methodologies, techniques, and tools to support their uplifting as knowledge graphs; as well as the relevant resources of the PON (classes, properties) to annotate them.

<table><caption>**Table 9: First (partial) cycle of pilot clinics.**</caption><thead><tr><td>Date</td><td>Pilot(s)</td></tr></thead><tbody><tr><td>10/02/2022</td><td>Bells</td></tr><tr><td>17/02/2022</td><td>Facets</td></tr><tr><td>24/02/2022</td><td>Organs, Tunes</td></tr><tr><td>10/03/2022</td><td>Tonalities</td></tr><tr><td>31/03/2022</td><td>Child/Meetups</td></tr><tr><td>28/04/2022</td><td>Tunes</td></tr><tr><td>05/05/2022</td><td>Interlink</td></tr></tbody></table>

**WP2+WP3+music-related pilots.** Here, our disciplinary goals include: extracting patterns from pieces of music and from corpora, distinguishing musicologically important ones, finding relationships between pieces of music based on patterns, querying/searching corpora for patterns. We have chosen to start with a bottom-up strategy. That is, for several different genres of music and different applications, multiple teams within the group (roughly corresponding to the pilots Tunes, Facets, Tonalities and Interlink) have each defined “pattern” concepts in their own way. Each team has presented preliminary work to the WP3 group for feedback, leading to common understanding of methods and challenges. Each team then implemented their own algorithms and applications (see table 10).

<table><thead><tr><td>Date</td><td>Agenda</td></tr></thead><tbody><tr><td>04/02/2022</td><td>How to organise and use this WG</td></tr><tr><td>11/02/2022</td><td>How to organise and use this WG<br/>List of music ontologies<br/>Create a specific plan</td></tr></tbody></table>

<footnote n="47">⁴⁷ Enrico Daga, Luigi Asprino, Paul Mulholland, and Aldo Gangemi. "Facade-X: an opinionated approach to SPARQL anything." *Studies on the Semantic Web* 53 (2021): 58-73.</footnote>
<table><caption>Table 10: WP2+WP3+music-related pilots WG agenda.</caption><tr><td>18/02/2022</td><td>Taxonomy of musical pattern discovery<br/>Set of "pattern dimensions"<br/>WP2 Clinic and relation to this WG</td></tr><tr><td>08/04/2022</td><td>Tonalities requests for algorithms/expertise in machine learning</td></tr><tr><td>15/04/2022</td><td>Knowledge graphs, lexicons, ontologies of patterns. NUIG will present something concrete for discussion.</td></tr><tr><td>29/04/2022</td><td>Items from GA/SC meeting<br/>Deliverable 3.3 status update<br/>KG discussion</td></tr><tr><td>06/05/2022</td><td>Requirements for pattern knowledge graphs (see initial examples below).<br/>D3.3 status update ("mining patterns" versus D3.1 "pattern extraction").<br/>Reporting items for upcoming plenary and quarterly report.</td></tr><tr><td>13/05/2022</td><td>Plenary reporting:<br/>D3.3 reporting<br/>Internal gathering of Pattern KG requirements (see below)<br/>How to gather further requirements for Pattern KG? Do we need a Survey of likely users, musicologists, ...?</td></tr></table>

**WP2+WP4+text-related pilots.** Our disciplinary goals include extracting information from the text-based pilots Musicbo, Meetups, Child, Bells and Organs. First of all, we have collected the pilot's corpora and have enriched them with metadata (author, title, language of the document, date of publication). We have also collected for each pilot a sample of 1000 sentences, representative of the variety of texts encountered. We then provided the corpora with the following annotations: tokenization, lemmatization, part of speech identification, entity linking, word disambiguation and annotation of musical terms. These features do not only provide us with statistics on the linguistic composition of the Corpus, but also enable an extraction of information from the pilot's text corpora at a general level. Finally, we used the pilot samples for abstract meaning representation (AMR). i.e. a semantic parsing which consists of converting a natural language text into a logical form and representing the meaning of a sentences as a semantic graph (see table 11).

<table><thead><tr><td>Date</td><td>Agenda</td></tr></thead><tbody><tr><td>20/01/2022</td><td>Check/Complete/Comment the WP4 annual report</td></tr><tr><td>27/01/2022</td><td>What provenance information do we want/need to extract?<br/>Facts from the pilots: identify a possible list of types of facts, evaluate feasibility, identify overlaps.<br/>T4.2 Artifact extraction: is there any useful resource from the corpus we can use?</td></tr></tbody></table>

<table><thead><tr><td>Date</td><td>Agenda</td></tr></thead><tbody><tr><td>10/03/2022</td><td>Organization of the presentation for the first Polifonia review meeting</td></tr><tr><td>24/03/2022</td><td>ORGANS: updates<br/>Semantic Parsing updates<br/>Corpus Sampling updates</td></tr></tbody></table>
<table><caption>Table 11: WP2+WP4+text-related pilots WG agenda.</caption><tr><td></td><td>(15')<br>Benchmarks for WP4-related pilots (5')</td><td></td><td></td></tr><tr><td>02/02/2022</td><td>Present the work on the annotation of the corpus (10' + 5' discussion)<br>Present the work on selecting listening experience (10' + 5' discussion)<br>Examples from the pilots (10' + 5' discussion)<br>Discussion: how to combine tools and requirements (15')</td><td>31/03/2022</td><td>AMR validation: updates<br>Corpus Sampling updates<br>Knowledge extraction<br>Deliverables</td></tr><tr><td>10/02/2022</td><td>Present AMR annotations<br>Discuss examples and CQ from the pilots for knowledge extraction supported by the annotation provided<br>Organs: status of play and link to bells (just to make sure this work links to WP2)</td><td>06/04/2022</td><td>Deliverables<br>Periodicals corpus: cleaning, sentence splitting, AMR annotation (Rocco)<br>Pilots: status updates</td></tr><tr><td>17/0272022</td><td>To present results of the inspection/validation of sentence-pairs and definition of validation criteria<br>Discuss examples and CQ from the pilots for knowledge extraction supported by the annotation provided<br>Discuss clustering approaches to select sentences relevant to the Pilots<br>Organs: status of play and link to bells (just to make sure this work links to WP2)</td><td>14/04/2022</td><td>D4.2 corpus organization<br>D4.2 metadata<br>D4.2 annotation - data format<br>WP4 Knowledge Extraction Pilots clinics</td></tr><tr><td>24/02/2022</td><td>Discuss AMR (abstract meaning representation) validation criteria: motif and threshold<br>Selection of features for the fuzzy clustering: CQ and concepts selection<br>Fuzzy clustering of CQ/sentences<br>Status of ORGANS</td><td>28/04/2022</td><td>D4.2: Pilot corpora<br>D4.2: corpus annotation<br>D4.3: status<br>D4.5: AMR annotation of the Musicbo sample</td></tr><tr><td>03/03/2022</td><td>Definition of the contribution for the deliverables (M18)<br>Discuss co-reference resolution results<br>Discuss AMR validation criteria: motif and threshold</td><td>05/05/2022</td><td>D4.2: Pilot corpora<br>D4.2: corpus annotation<br>D4.3: status<br>D4.5: AMR annotation of the Musicbo sample</td></tr><tr><td></td><td></td><td>12/05/2022</td><td>[D4.2] Rocco: corpus annotation<br>[D4.2] Rocco: corpus interrogation<br>[D4.2] Arianna: status of the report<br>[D4.2] Eleonora: status of the corpus<br>[D1.2] Describe the work conducted with the pilots</td></tr></table>
**WP5+WP1+all pilots.** There are two overall objectives of the cross pilot-WP1-WP5 WG meetings. First, they aim to inform WP5 of the challenges within the pilots in terms of incremental interface design, implementation and evaluation<fnref n="49" /> so that WP5 can provide more targeted support. Second, they are intended to raise awareness among the WP1 pilots of Human Computer Interaction (HCI) tools, techniques and methods being applied in WP5 research which may be of assistance to the pilots. The meetings are held weekly. The agenda alternates between being set by WP1 or WP5. Topics covered in the meetings have included validation, UX methods<fnref n="50" />, WP1 Web portal, the MEETUPS, MUSICBO and BELLS pilots, focus groups and WP4 interfaces (table 12).

<table><caption>**Table 12: WP1+WP5 + all pilots, WG agenda.**</caption><thead><tr><td>Date</td><td>Agenda</td></tr></thead><tbody><tr><td>28/01/2022</td><td>Web portal: Competitive analysis for the Web portal design, Dashboard templates and data Stories<br>Meetups template</td></tr><tr><td>08/02/2022</td><td>UX Methodology<br>Web portal: Update: templates and dashboard<br>Sonar++</td></tr><tr><td>15/02/2022</td><td>Dashboard app<br>Meetups + MUSICBO + BELLS template</td></tr><tr><td>22/02/2022</td><td>Validation criteria</td></tr><tr><td>01/03/2022</td><td>SONAR+ - WP 5 presentation</td></tr><tr><td>08/03/2022</td><td>Web portal and data Story (MEETUPS) prototype</td></tr><tr><td>22/03/2022</td><td>Presentation on generous interfaces</td></tr><tr><td>29/03/2022</td><td>Questionnaire with students on habits, expectations, frustrations in music websites UX</td></tr><tr><td>05/04/2022</td><td>D5.2 deliverable planning</td></tr><tr><td>12/04/2022</td><td>Web portal mockup</td></tr><tr><td>26/04/2022</td><td>D1.9 Deliverable, Polifonia Text corpus</td></tr><tr><td>10/05/2022</td><td>WP4 interfaces</td></tr><tr><td>17/05/2022</td><td>Deliverable review before Polifonia general meeting</td></tr></tbody></table>

<footnote n="49">⁴⁹ This process of incrementally developing and testing designs is described by Gaver as Research Through Design: each design object (e.g. mock-up or prototype) commits to a set of design choices. See William Gaver. "What should we expect from research through design?", in *Proceedings of the SIGCHI conference on human factors in computing systems*, ed. by J.A. Konstan, Association for Computing Machinery, 2012, pp. 937-946.</footnote>

<footnote n="50">⁵⁰ A number of guides have been written describing the motivations and practice of UX Design including Rex Hartson and Pardha S. Pyla. *The UX book: Agile UX design for a quality user experience*, Morgan Kaufmann, 2018; Russ Unger and Carolyn Chandler. *A Project Guide to UX Design: For user experience designers in the field or in the making*, New Riders, 2012.</footnote>
## 3.2 Results

As a whole, the groups significantly stimulate collaboration between the technical and disciplinary actors of the project: they help to specify methods and workflows for the implementation of disciplinary requirements (WP5+WP1+all pilots), and lead to a weekly collaboration of a) the text-related and b) the music-related pilots with the OD team to combine exploration and modelling tasks. Finally, the ontology clinics are a central space where the OD team collaborates with the pilots to produce, adapt and test ontologies against the initial requirements and objectives.

**WP2 Knowledge graph pilot clinics.** Since datasets from pilots display a great deal of heterogeneity, the results regarding methodology, techniques and tools for them to use in the construction of their knowledge graphs is varied. For example, INTERLINK relies on a specific framework to homogenize chord data into the JAMS<fnref n="51" /> data structure, and then creates knowledge graphs using SPARQL templates and SPARQL Anything<fnref n="52" />. For the creation of links, string similarity algorithms (such as Levenshtein, sequence matching, etc.) are used; albeit the resolution of ambiguous composers and works remains a challenge. TONALITIES follows an ontology-based approach in which specific data-files (e.g. in MusicXML and MEI) are directly loaded as ontology instances and used for reasoning – the results can be serialised as RDF, therefore creating Knowledge Graphs. Other pilots follow a more data-centric approach in which the data is transformed from its original format directly to RDF: FACETS uses an API providing data in MEI and JSON; TUNES a database in JSON; BELLS an Excel spreadsheet; etc. For these, either custom code, JSON-LD contexts, SPARQL Anything or Open Refine<fnref n="53" /> are viable options. ORGANS is primarily a textual database, and hence a specific approach will be needed. Lastly, WP1 has proposed that all these knowledge graphs shall be connected through a metadata abstraction layer, the Pilots Knowledge Graph (see above section 1.1.3), documenting their contents and providing a useful resource for the Ecosystem and the Web portal.

**WP1+WP3+music-related pilots.** The results include the following. 1. A complete tool for pattern extraction, identification of important patterns, and identification of relationships between pieces in the context of Irish traditional music. Informal evaluation suggests that it correctly discovers related pieces. 2. A music representation suitable for querying/searching large corpora, specifically the NEUMA corpus. It is based on a system of music content descriptors leading to features suitable for standard search methods. 3. A tool for measuring harmonic similarity between pieces, based on local harmonic patterns. It uses harmonic reduction, chord decomposition, and pitch simultaneities to arrive at a representation in which harmonic similarity can be measured. The result is a network of pieces (Isophonics for pop music, the JAAH jazz harmony dataset, and Schubert Winterreise for classical music) linked by similarity, including links within corpora and between corpora.

**WP1+WP4+text-related pilots.** Results include an annotated corpus following the CoNLL-U format where each word/token is reported in tab-separated columns on one line, and sentence boundaries are indicated by blank lines. This allows to automatically extrapolate the implicit morphosyntactic, semantic and domain-specific information. The corpus has been annotated

<footnote n="51">⁵¹ https://jams.readthedocs.io/en/stable/.</footnote>

<footnote n="52">⁵² https://github.com/SPARQL-Anything/sparql.anything.</footnote>

<footnote n="53">⁵³ https://openrefine.org/.</footnote>
through tokenization, lemmatization, part-of-speech tagging, word sense disambiguation, named entity recognition, entity linking. We used EWISER for all languages of the corpus, ExTenD for English and mGENRE for languages other than English. The Musicbo pilot sample (1000 sentences) has been annotated with AMR.

**WP5+WP1+all pilots.** The meetings have assisted in promoting knowledge sharing and collaboration across WP1 and WP5. For example, the meetings directly led to WP5/WP1 collaboration on the nature of the focus groups to guide design of the dashboards and the Survey of music consumption in order to inform the design of the portal. Other 1-to-1 meetings have also been held or scheduled between WP5 and the pilots as a result of discussions in the WG.

## 3.3 Lessons learned

**WP2 Knowledge graph pilot clinics.** The most important lesson learned in the clinics is that the data Ecosystem of Polifonia's pilots is highly heterogeneous, and hence a unique, uniform approach for the construction of knowledge graphs, and their linking to the existing Ontology Network, is highly unlikely and reasonably discouraged. Disciplinary goals and challenges are varied among the pilots, and naturally are translated into different specific technical resources and tools. This variety comes in different shapes: size (30M triples in chord datasets in INTERLINK; to a few sheets in BELLS) is a critical dimension, but so are Personas and requirements to link with ontology classes and properties, and data formats (JSON, JSON-LD, MEI, MusicXML, text, Excel, JAMS). For the formats supported, SPARQL Anything and SPARQL query templates were successfully used to batch-convert millions of RDF triples (e.g. INTERLINK, ChoCo dataset). String similarity remains the baseline for basic entity linking, but more sophisticated methods for disambiguation, especially in music catalogs, are needed (e.g. INTERLINK, Discogs and GTAA from NISV). Various data-insights of the clinics, e.g. in BELLS, were critical to revise versions of the ontology network and correct the way resources were modelled or named.

**WP2+WP3+music-related pilots.** We have found that the bottom-up strategy already mentioned has allowed fast progress in parallel across the pilots, but a remaining challenge is to unify the multiple pattern technologies provided by WP3 to the pilots. One avenue for this unification is in current discussions of ontologies and knowledge graphs for representation of patterns. A second methodological challenge is in the evaluation of results, as we lack "gold standard" data sets labelled with "pattern importance" or "piece relationship", and moreover there is no musicological consensus on such issues. Some workarounds are being used, such as the cover-song detection task: a piece-relationship tool can be evaluated for its ability to detect cover songs or (in the case of oral musical heritage), tune variants. Moreover, such a tool based on pattern extraction can indirectly validate the appropriateness of that pattern definition. Research continues in this area.

**WP2+WP4+text-related pilots.** Some technical and methodological difficulties have been important to improve our approaches. The first concerns the annotation method: the annotation process is very time consuming, so we used textual samples representative of the linguistic variety of the pilots. This allowed us to make a first iteration of annotation in a shorter time however considering the completeness of the variety of the data. The second relates to the importance to align the semantic categories related to the analysis of musical discourse with AMR, so as to be able to maximize the extraction of information from the text following the research needs of the pilots.
**WP5+WP1+all pilots.** The meetings are providing an effective focal point for collaboration and are set to continue. One issue we may wish to consider in the future is to ensure that collaborations are appropriately captured elsewhere in the data Ecosystem, for example, capturing changes in how the users are understood as updates within the Polifonia Stories.
# 4. Monitoring pilot validation

The SMS tools in connection with the other methodological devices key to WP1 (Ecosystem, Web portal) and WP5 (XD-UX-design methodology), led to the implementation of an overall Validation framework. This section only describes the Validation framework as such and the methodological considerations behind it. The actual validation of the pilots through individual validation protocols based on the framework will be done in later deliverables at M. 24 (D1.4-7) and M. 40 (D1.8).

## 4.1 Goals, methods and strategies

The main objective of this part of the Roadmap is to design a global Validation framework that confirms that:

* O1: The pilot objectives reflect concrete needs expressed by the target communities and contribute to the state of the art;

* O2: the methods and approaches are sound;

* O3: the data on which the pilots are based are of sufficient quality to achieve the desired objectives;

* O4: the objectives and results expressed through the Stories and the Survey are achieved;

* O5: the resources, methods and tools provided are made explicit and integrated into the Polifonia Ecosystem, interconnected through the PON and conveyed through the Web portal<fnref n="54" />

The overall methodology was targeted towards a bottom-up approach that took the diversity of the pilots as a starting point and relied on the Survey, the cross-pilot-WP WGs and the Stories. This allowed common approaches and validation paths to emerge while respecting the specificities of the different pilots. As a complement to the SMS-tools, other canalizing elements including the Ecosystem with its guidelines, the XD-UX-design methodology defined within WP5, and the Web portal have been integrated in the framework. The strategy was thus not to impose homogeneity at the level of the validation protocols which will be applied by the pilots, but to create a Validation environment in which Polifonia's driving methodological principles converge.

The architecture of the Validation framework was built through a combination of Polifonia's general methodological guidelines and an in-depth analysis of the validation methods, protocols, success criteria and KPIs reported by the individual pilots in the Survey (see above section 1). On this basis, a task force composed of Polifonia members developed a general scheme that is flexible enough to accommodate the current and future needs of the pilots and binding enough to meet the objectives listed above.

<footnote n="54">⁵⁴ Only some of the pilots' data are conveyed through the webportal. See the Survey, technical part, questions "Is it to be released through the Polifonia Web portal or as an independent application?", "How should the pilot's resources be presented, linked, or stored on the website dedicated to the pilot?", "What type of knowledge can contribute to the Polifonia Web portal?", "What type of interaction do you expect should be supported by the Web portal?", "In which ways the Web portal shall link to the (separate) pilot demonstrator (if any)?" and "What kind of 'augmentation/enrichment' do you expect to get out of being displayed via the portal?". See also Marilena Daquino et al., *D1.9. Polifonia Web portal – 1st version*, Polifonia, 2022.</footnote>
The Validation framework itself is a Github repository in the Polifonia Ecosystem<fnref n="55" /> and will become an Ecosystem component. The framework can thus be documented, adapted and monitored until its actual application in the dedicated deliverables at M24 and 40.

## 4.2 Results

The Validation framework is modular in nature. It includes core criteria with which all pilots must comply: a) the use of a methodological approach combining XD and UX design, b) the inclusion of resources in an Ecosystem governed by specific guidelines, c) the presentation of the pilots and (part of) their resources via the Web portal; and d) the regular monitoring via the Survey. In addition to these criteria, the framework consists of specific validation methods for individual pilots or pilot groups (table 13).

<table><caption>Table 13: Validation framework, core validation criteria and pilot-specific methods.</caption><thead><tr><th rowspan="4">Core validation criteria</th><th colspan="10">Survey (with success criteria, KPIs and validation setup)</th></tr><tr><th colspan="10">XD and UX Design</th></tr><tr><th colspan="10">Ecosystem + Rulebook</th></tr><tr><th colspan="10">Web Portal</th></tr></thead><tbody><tr><td rowspan="10">Pilot specific methods</td><td rowspan="3">Input validation</td><td>Validation of mechanically produced input data by ground truth</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td></tr><tr><td>Quantitative validation of input data</td><td></td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td></tr><tr><td>Validation of input data quality by experts</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>✓</td><td>✓</td></tr><tr><td rowspan="3">Validation of tools/means/software</td><td>Validation of interfaces by users/experts</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Performance validation</td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td></td></tr><tr><td>Validation of performance, requests/information passed between front and back end</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td>✓</td></tr><tr><td rowspan="4">Output/goal validation</td><td>Compliance of the data produced with standards</td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Validation of results by ground truth</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td>✓</td></tr><tr><td>Validation of results by experts</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td></td></tr><tr><td>Validation from a sub-corpus/case study and transfer to a larger corpus</td><td></td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td></td></tr><tr><td></td><td>Validation of objectives through experts</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td>✓</td></tr><tr><td colspan="2">ACCESS</td><td>BELS</td><td>CHILD</td><td>FACETS</td><td>INTERLINK</td><td>MEETUPS</td><td>MUSICBO</td><td>ORGANS</td><td>TONALITIES</td><td>TUNES</td></tr><tr><td colspan="2"></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

### 4.2.1 Core validation criteria

The compliance of the pilots to a set of common core tools, principles and guidelines grants a methodological and conceptual coherence to the whole while leaving enough space for individual pilot specific validation protocols.

a) **XD and UX design:** The strategy consists of applying a mix of eXtreme Design and UX design through user interviews, Personas, Stories, CQ, mock-ups, and interface prototypes. This approach, which is part of design-centered research (RTD) helps unify the design process of the pilots and ensures that their objectives are in line with real needs (O1). Furthermore, the inclusion of the Stories as key elements in the pilot validation process enables us to monitor the extent to which the requests and needs formulated through them are actually implemented (O5).

b) **Ecosystem and Rulebook:** The conformity of the pilots (and of their components) to the Ecosystem and its guidelines plays a leading role in the validation process. This criterion ensures that all components that are discussed, presented and published are correctly referenced (O5). The

<footnote n="55">⁵⁵ https://github.com/polifonia-project/socio-technical_roadmap/tree/main/validation_framework.</footnote>
Ecosystem also makes sure that the decisions, modifications and progress related to the pilots are documented. This provides central perspectives for the maintenance of the pilots and for assessing that the Stories are considered in the pilot implementation and validation process. Finally, this validation criterion helps to interconnect the pilot components and to provide them with metadata (type, WP, pilot, keywords).

c) Web portal: As a place for disseminating, enhancing and interconnecting the pilots, the Web portal helps to harmonize the way Polifonia's resources can be presented, explored and searched by both amateur and expert users. It also favors co-creation processes and the mutual adjustment of the individual pilots. Methodologically, the portal helps to steer the pilots towards a story-telling, data-, and hypothesis-driven approach. The design principles based on QC analysis, focused groups to collect users' expectations and frustrations, and prototyping (based on the information collected and feedback from Polifonia researchers) lead to identify and apply shared presentation and interaction patterns. Thus, the Web portal is an important validation criterium because it contributes to applying sound methods and approaches (O2) and to conveying Polifonia's resources to a dedicated and shared place (O5).

d) Survey: The key position of the Survey is due to its ability to both reveal the DNA of the pilots and to make conclusions on the architecture of the Validation framework. This is because the Survey provides meticulous documentation on the choices, orientations, resources and objectives of each pilot: musical heritage collections explored, copyright and licensing issues, technical specificities, application to be developed, relationship to the Web portal, involvement of experts, consideration of user feedback, collaborations etc. The Survey is also the main source of information for the pilot's individual validation strategies. It is the place where the monitoring of the pilots has been carried out so far and where detailed information on the validation methods, the validation protocol and the KPIs used is collected<fnref n="56" />. The Survey thus has a double role. On the one hand, it is a central tool for designing the Validation framework presented here. On the other hand, it provides us with a fingerprint of the pilots scientific, technical and socio-pedagogical specificities. This fingerprint can be used not only internally (to specify and analyze individual approaches) but also externally (for example on the Web portal, on the pilots' individual applications, or in publications). The Survey thus gives a view of the level of implication of the scientific communities (O1). It also allows us to reconstruct the pilot's methodological principles (O2), and to monitor the extent to which the objectives are achieved (O4).

Although only the results of the pilots and of the overall project can demonstrate the effectiveness of the core tools implemented (XD and UX-design, Ecosystem and Rulebook, Web portal and Survey), these tools themselves should be evaluated and monitored, particularly with a view to adapting and transferring them to other projects. The use of user Surveys as a qualitative approach to validate these components is methodologically difficult to implement as it cannot be entrusted to external experts. It is planned instead to methodologically focus the validation of the framework's components on KPIs according to the metrics outlined in example 6. As regards the XD-UX-design methodology our strategy consists not only of demonstrating what the methodologies produce (i.e. number of components, Stories, Personas, competency questions, see above section 2.2) but also to monitor the success of the tools that have been designed on the basis of this methodology

<footnote n="56">⁵⁶ See section 1.1 above.</footnote>
through the number of people testing, using and adopting them. The indicators of success currently used, which stem from the Grant Agreement<fnref n="57" />, may be extended to include additional criteria, such as the number of stars in repositories, number of external developers and peer-reviewed publications.

<figure><img src="image_7.png" /><figcaption>Example 6: KPI for the Web portal, Ecosystem, XD-UX-design methodology and Survey</figcaption></figure>

### 4.2.2. Pilot specific methods

The pilot specific methods identified are the result of an in-depth analysis of the answers given by the pilots during several iterations of the Survey<fnref n="58" />. First, the pilots freely described their validation methods. The answers were then reformulated in order to identify common methods and to group them into homogeneous categories. These categories were then submitted to the pilots for adjustment and validation. The validation methods identified on this basis cover a) the “input data” of the pilots, i.e. the data that serve as starting points for the case studies; b) the tools, means and software bricks by which new knowledge and/or results are produced; and c) the validation of objectives and results (see table 13).

The following paragraphs describe these validation methods in general and illustrate them with individual examples. A systematic and detailed description of the methods applied to each pilot is provided in the deliverables for the mid-term and final evaluation.

a) **Input validation.** The goal of this first validation group is to assess the quality of the data (musical scores, historical texts) with regard to pilot specific needs (O3). Another objective is to leverage the quality control of the input data to identify what changes have been made to the dataset to suit the pilot's requirements. Validation on the basis of ground truth of mechanically produced input data concerns the scanning quality of printed documents (Organs) and the harvesting of information collected on the Internet (Meetups). Organs for example will compare the automatic conversion with manual input to provide a coefficient of correctly converted sources. In addition to this approach, two pilots (Bells and Musicbo) explicitly consider the quantity of input data as a validation criterion. For instance, Bells will collect at least 100 complete case studies and will digitize at least 300 records. The criterion most frequently used by the pilots is the qualitative validation of data by

<footnote n="57">⁵⁷ The KPIs currently considered are based on the Grant Agreement, part B, p. 11, table 1.1.</footnote>

<footnote n="58">⁵⁸ https://github.com/polifonia-project/Survey. Iterations took place at the following dates 07.05.2021, 14.05.2021, 17.06.2021 and 19.05.2022.</footnote>
experts, for example the information provided by bell ringers (Bells) or the quality of the digital scores explored (Facets).

**b) Validation of tools/means/software.** The objective here is to verify both the robustness and the ergonomics of the technical solutions implemented (O2). It is highly significant that the validation of the tools/media/software bricks focuses on the validation of interfaces by amateur and/or expert users. The almost complete consensus on this criterion reflects the pilots' adherence to the XD-UX-design methodology used as a core criterion. Tonalities for instance plans to run a usability survey complemented by interviews for the interface and will consider the resource to be successful when the user survey will achieve 90% positive evaluation. Facets on the other hand will take into account the number of partners or external institution/people using its tool (or collaborating to it) by measuring Github forks (collaboration), Docker image downloads, and the number of verified installations. Apart from this method, the pilots focus on the accuracy and reliability of the requests passed between the front and back end (Access, Facets, Tonalities) and on the performance of the computer solutions to be developed (Facets). To this end, Facets measures the accuracy and the average time the search engine (i.e. elastic search) needs to answer a query compared to other search strategies (example 7).

<figure><img src="image_8.png" /><figcaption>Example 7: Comparison of execution times of regex-based searches and elastic search (ES) servers</figcaption></figure>

**c) Output/result validation.** The last subgroup covers the validation of both the objectives and the results of the pilots (O4). One part of the pilots validates its results based on the ground truth previously created. This validation method concerns the music-related pilots Interlink, Tonalites and Tunes. Tonalities for example aims at providing 90% correct modal-tonal identifications in a corpus with at least 150 works with one analysis. The strategy adopted in the case of Bells and Musicbo, both focusing on text, is to validate a larger set of results from narrow case studies: information gathered about a small set of bells or a specific aspect of Bologna's musical heritage. Two pilots (Bells and Interlink) consider respectively the conformity of the data to standards and the validation by experts of the results obtained (i.e. the interconnections made between different musical works). Interlink for instance plans to rank the similarities detected between musical pieces based on known categories (performances of the same composition, covers songs, etc.) according to f-score and to
compare these results with human performances. The validation of the objectives by experts is,
however, here again, the most important criterion: Access for instance considers the views of music
educators and music venues involved while Child takes into account the declared interest of the
Music Department in Open University in tools and methods defined.

## 4.3 Lessons learned

An important effort has been made to build a coherent validation infrastructure. This effort is both
individual and collective: The pilots have defined on their own their validation strategies according
to what is best suited to their needs. At the same time, the cross-pilot-WP WGs and the task force
activities related to the pilot validation have contributed to designing an overall Validation
framework combining core validation criteria and pilot specific methods organized into three
subgroups.

In all subgroups, validation by users is a central criterion. This general focus is in line with and fosters the project's overall commitment to obtaining early feedback from musical heritage communities, to avoid technology driven development and to foster user-centered approaches. By doing so, we intend to create a virtuous circle between the users' feedback and the subsequent adaptation/refocusing of the pilots. The validation process has a central role to play here since its application will lead to a substantial increase of the feedback obtained. At the same time, the information collected from the Survey suggests that there is a need to map user categories more systematically to the pilot's general goals and, more specifically, to the validation tasks associated with them: preserve (Bells, Organs), manage (Facets, Interlink), study music heritage (Tonalities, Tunes, Child, Musicbo), and interact with it (Meetups and Access).

The methods that relate to the validation of outputs and results identify some affinities between Interlink, Tonalities and Tunes on the one hand and Musicbo and Bells on the other. However, it is not possible at this stage to identify pilot sets that would share homogeneous validation methods (e.g., text-related pilots and music-related pilots). This may be because the responses are not yet systematic enough and/or because the pilots, depending on their scientific perspectives and objectives, grant unequal importance to the different validation methods. At least one new iteration of the Survey before the mid-term validation (M24) is scheduled to update the information. Furthermore, it is planned to integrate the different WGs more actively in the process of identifying shared methods in the coming months.

Finally, it appears necessary to generalize the interconnections between the CQ and their effective integration and implementation in the pilots. The bi-directional link between CQ and pilot updates is taken into account in the current validation infrastructure through the XD-UX-design methodology at the level of the "core criteria". However, this link still needs to be concretely implemented in the Ecosystem in the coming months before the intermediate validation.
FAIR protocol

The following components have been released as a direct or indirect result of the work described
in this deliverable:

1. Ecosystem

2. Web portal

3. XD-UX methodology related data

4. Survey

5. Validation framework

Via Github, all components – or in the case of the XD-UX-design methodology their outcomes (i.e the Personas, Stories and CQ) – have a unique and persistent identifier and are registered in a searchable resource. The Survey, in its current version, has also been deposited in Zenodo<fnref n="59" />. All the components have been described with and/or contribute to produce rich metadata: The Ecosystem is targeted towards producing metadata through its annotation schema; all resources of the Web portal are provided with metadata (scope, audience, type of data served, data access points)<fnref n="60" />. Finally, as components of the Ecosystem, the Survey and the Validation framework are described with metadata defining their type, contents, and relation to the WPs. This also applies to the outcomes of the XD-UX-design methodology. The storage of all resources on Github and on Zenodo (in the case of the Survey) guarantees their persistence beyond the project.

Since all components and their related data benefit from the Ecosystem annotation schema (and
are registered in Zenodo in the case of the Survey), they are all retrievable by their identifier using
standardised communications protocols. These protocols are open, free, and universally
implementable. Furthermore, the Ecosystem infrastructure with its annotation schema is
independent of its individual components. Therefore, the metadata remains accessible, even if the
data is no longer available.

All data and metadata produced use a formal, accessible, shared, and broadly applicable language for knowledge representation. The resources disseminated by the Web portal are made available in the form of RDF triples from the PKG. The Survey data will also eventually be transformed into an RDF knowledge graph (the PiKG). The resources created by the XD-UX design methodology through the Persona, Stories and CQ are all open linked data of the PON. As for the Ecosystem and Validation framework, their organisation is made explicit by the Ecosystem annotation scheme.

Finally, the Ecosystem, the Survey, the Stories which are derived from the XD-UX-design
methodology, and the Validation framework are released with a CC-BY 4.0 licence that allows all
resources to be copied and redistributed in any medium or format. The material may also be
remixed, transformed, and build upon for any purpose, even commercially. The different resources
of the Web portal are released under CC0 and ISC licences.

<footnote n="59">⁵⁹ Guillotel-Nothmann, Christophe, Gurrieri, Marco, van Kranenburg, Peter, Musumeci, Elena, Veninata, Chiara, Berardinis, Jacopo de, Fournier-S'niehotta, Raphaël, Marzi, Eleonora, Barlow, Helen, Daga, Enrico, & Holland, Simon. (2022). Polifonia Survey (Version 19062022) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.6664200.</footnote>

<footnote n="60">⁶⁰ Furthermore, it is planned to develope a dedicated pipeline for data enrichment for case the metadata
would be incomplete.</footnote>
Conclusion

This deliverable describes the work done over the last 12 months to set up the second version of a socio-technical Roadmap. This Roadmap consists of a common methodological framework on which the pilots, as the driving force behind Polifonia, can build. In the first version of the Roadmap, our efforts focused on coordinating the collection of the pilot's expectations and needs through a) the Stories, b) the Maninpasta, and c) the Survey (SMS) in a bottom-up approach. Based on an in-depth analysis of the information provided through the Survey and considering the methodological approaches that crystallized through the Stories and the Maninpasta/cross-pilot-WP WGs, this second version of the Roadmap presents a generalization of the methods, standards and procedures which culminates in the Validation framework. To this end, the collection of the pilot's expectations, which continued during this year, has been integrated into a broader set of objectives:

1. Coordinating resources, methods and tools used in the pilots

2. Centralizing needs and requirements of internal and external stakeholders

3. Mapping objectives and challenges from pilot specific domains to the technology provider WPs

4. Monitoring pilot validation

1. The coordination of the resources used by the pilots through a) the Ecosystem and its guidelines, b) the Web portal and c) the Survey has led to structured, homogeneous and highly granular information on the pilots' resources, methods, tools, principles and guidelines (including the Personas, Stories and CQ relating to the pilots). The tools have helped to uncover interconnections between the pilots and to identify the resulting opportunities for resource sharing. The Survey, once transformed in the PiKG has the ambition to be the focal entry point to understand the pilots in their domain-specific, technical and socio-pedagogical dimension.

2. Stories have proven to be a suitable tool for centralizing the pilot's requirements and needs. A systematic analysis of the 19 Personas, 28 Stories and 192 CQ currently available has allowed to define the PON, to design each module and its expected inter-relationships, and to implement the basic features of each module. The centralization of external feedback, especially from stakeholders, will be provided through the Validation framework, which integrates the XD-UX-design methodology as a core criterium and focuses on user feedback through the individual validation methods.

3. Maninpasta and, at a later stage, the cross-pilot-WP WGs have proven successful in mapping challenges from pilot specific domains to the technology provider WPs. They have led to common tools and methods adapted to the exploration of textual and musical data and have contributed, through the pilot clinics, to build a direct link between the pilot's specific domains of expertise and the modeling of these domains in the PON.

4. The detailed analysis of the Survey and the work carried out in the cross-pilot-WP WGs allowed for the development of a Validation framework combining core validation criteria and pilot specific validation methods. This framework is the cornerstone of the pilot-driven work conducted in WP1 in conjunction with the technology provider WPs and leads to a harmonization of methods, practices and approaches.
While the socio-technical Roadmap is completed with this deliverable, various actions directly or indirectly related to the Roadmap are planned for the coming months. We will continue to populate the Survey and eventually will transform it into a KG to make it an interconnected, browsable and explorable component for both internal and external purposes. Another field of activity – taking place in WP2 but directly connected to the objective of centralizing needs and requirements – consists of reformulating the Stories and carrying out their systematic analysis. We will actively engage the network of stakeholders in the validation process by collecting their feedback through user studies, Surveys, reports and dedicated WGs. Finally, in conjunction with the pilots and the cross-pilot-WP WGs, the fine-tuning of the Validation framework will be pursued to apply it effectively in the intermediate (M24) and final (M40) pilot evaluations.
## References

Beck, Kent L., & Andres, Cynthia. Extreme programming explained - embrace change, Addison-Wesley, 2005².

Blomqvist, Eva et al. "Experimenting with eXtreme Design", in *Proceedings of EKAW 2010*, ed. by P. Cimiano, H.S. Pinto, Springer, 2010, pp. 120-134.

Blomqvist, Eva et al. "Engineering Ontologies with Patterns - The eXtreme Design Methodology", in *Ontology Engineering with Ontology Design Patterns, vol. 25. Studies on the Semantic Web*, ed. by P. Hitzler, A. Gangemi, K. Janowicz, A. Krisnadhi, V. Presutti, IOS Press, 2016, pp. 23-50.

Carriero, Valentina Anita et al. "Pattern-based design applied to cultural heritage knowledge graphs", «Semantic Web», 12/2 (2021), pp. 313-357.

Daga, Enrico et al. D1.3. *Pilots development – collaborative methodology and tools*, Polifonia, 2021.

Daga, Enrico, Luigi Asprino, Paul Mulholland, and Aldo Gangemi. "Faculty-X: an opinionated approach to SPARQL anything." *Studies on the Semantic Web* 53 (2021): 58-73.

Daquino, Marilena et al., D1.9. *Polifonia Web portal – 1st version*, Polifonia, 2022.

Gaver, William. "What should we expect from research through design?", in *Proceedings of the SIGCHI conference on human factors in computing systems*, ed. by J.A. Konstan, Association for Computing Machinery, 2012, pp. 937-946.

Grüninger, Michael & Fox, Mark S., "The role of competency questions in enterprise engineering", in *Benchmarking - Theory and practice*, ed. by A. Rolstadås, Springer, 1995, pp. 22-31.

Guillotel-Nothmann, Christophe et al. D1.1. *Roadmap and pilot requirements 1st version*, Polifonia, 2021.

Hartson, Rex, & Pyla, Pardha S. *The UX book: Agile UX design for a quality user experience*, Morgan Kaufmann, 2018.

Hogan, Aidan, Eva Blomqvist, Michael Cochez, Claudia d'Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane et al. "Knowledge graphs." *Synthesis Lectures on Data, Semantics, and Knowledge* 12, no. 2 (2021): 1-257.

Meroño-Peñuela, Albert et al. D.2.1. *Ontologies and knowledge graphs of music objects, patterns, and software package – 1st version*, Polifonia 2021.

Meroño-Peñuela, Albert et al. D.2.4. *Methods for interlinking knowledge graphs – 1st version*, Polifonia 2022.

Meroño-Peñuela, Albert et al. D.2.7. *Ontology testing and evaluation report; knowledge graph documentation and access interfaces – 1st version*, Polifonia, 2022.

Scharnhorst, Andrea et al. D7.1. *First Data Management Plan*, Polifonia, 2021.
Scharnhorst, Andrea et al. D7.4. DMP webinar on dynamic data management, Polifonia, 2021.

Unger, Russ, & Chandler, Carolyn. *A Project Guide to UX Design: For user experience designers in the field or in the making*, New Riders, 2012.

Vrandečić, Denny, and Markus Krötzsch. "Wikidata: a free collaborative knowledgebase." *Communications of the ACM* 57, no. 10 (2014): 78-85.
# Appendices

## Appendix 1: Glossary

<table><thead><tr><td>Term</td><td>Definition/description</td></tr></thead><tbody><tr><td>AMR</td><td>Abstract Meaning Representation</td></tr><tr><td>ArCo</td><td>Architettura della Conoscenza. A project for developing the Italian cultural heritage knowledge graph.</td></tr><tr><td>CIDOC CRM</td><td>CIDOC Conceptual Reference Model (CRM) is a theoretical and practical tool for information integration in the field of cultural heritage.</td></tr><tr><td>CoNLL</td><td>An annotation format, see https://universaldependencies.org/format.html</td></tr><tr><td>CQ</td><td>Competency Question: a question that an ontology would need to answer.</td></tr><tr><td>CRMinf</td><td>CRMinf: the Argumentation Model. An Extension of CIDOC-CRM to support argumentation.</td></tr><tr><td>D</td><td>Deliverable.</td></tr><tr><td>Discord</td><td>Instant messaging and digital distribution platform designed for creating communities.</td></tr><tr><td>DMP</td><td>Data Management Plan.</td></tr><tr><td>Domain specific part/aspect</td><td>Any scientific, epistemological and methodological considerations concerning one or more pilots.</td></tr><tr><td>EWISER</td><td>Enhanced WSD Integrating Synset Embeddings and Relations.</td></tr><tr><td>ExTenD</td><td>Extractive Entity Disambiguation.</td></tr><tr><td>eXtreme Design (XD)</td><td>Pattern-based ontology engineering methodology, which collects requirements in the form of Competency Questions.</td></tr><tr><td>Github</td><td>Git is an open-source software for managing versions of documents. The documents typically managed in Git repositories are computer source code files, but it is in fact perfectly suitable for text data in general, including humanities data. The development of Git is closely related to that of GitHub a service for hosting Git repositories.</td></tr><tr><td>Goal</td><td>A short textual description of the goal(s) that the Persona needs to be addressed in the Story.</td></tr><tr><td>H</td><td>Human(s).</td></tr></tbody></table>
<table><tr><td>HCI</td><td>Human Computer Interaction.</td></tr><tr><td>IReMus</td><td>Institut de Recherche en Musicologie, UMR 8223. Joint research unit composed of the CNRS, the BnF, Sorbonne University and the French Ministry of Higher Education and Research.</td></tr><tr><td>KG</td><td>Knowledge graph.</td></tr><tr><td>Knowledge unit/type</td><td>New knowledge produced by the pilot through the analysis (by algorithms, machine learning and/or human annotation) of the input data. This unit may correspond for example to a cadence in a score, to the identification of a previously anonymous author of a source, to the type of a bell, etc. (see diagram 2. Survey – Knowledge units investigated by the pilots).</td></tr><tr><td>mGENRE</td><td>A sequence-to- sequence system for the Multilingual Entity Linking.</td></tr><tr><td>M[1-40]</td><td>Month counting from the start of Polifonia on 1 January 2021: M6 (= June 2021), M18 (= June 2022), etc.</td></tr><tr><td>MEI</td><td>Music Encoding Initiative.</td></tr><tr><td>MH</td><td>Musical heritage.</td></tr><tr><td>NLP</td><td>Natural language processing.</td></tr><tr><td>Observation unit</td><td>A portion of input data on which the pilot relies to provide new units of knowledge through algorithms, machine learning and/or human annotation. These observation units may correspond for example to a sentence in a text, a score fragment (for example a cadence) or the dedicatee engraved on a bell.</td></tr><tr><td>OD</td><td>Ontology design.</td></tr><tr><td>ODP</td><td>Ontology design patterns which correspond to small ontologies that work as reusable solutions to recurrent modelling problems.</td></tr><tr><td>OEI</td><td>Office of ethics and integrity.</td></tr><tr><td>OU</td><td>The Open University.</td></tr><tr><td>OWL</td><td>Web Ontology Language (OWL) is a Semantic Web language designed to represent rich and complex knowledge about things, groups of things, and relations between things.</td></tr><tr><td>Persona</td><td>A research-based description of a typical user. This description contains attributes such as name, age, occupation, and relevant characteristics such as knowledge and skills.</td></tr></table>
<table><tr><td>PiKG</td><td>Pilot Knowledge Graph.</td></tr><tr><td>PKG</td><td>Polifonia Knowledge Graph.</td></tr><tr><td>PON</td><td>Polifonia Ontology Network.</td></tr><tr><td>POPD</td><td>Protection of personal data.</td></tr><tr><td>RTD</td><td>Research through design.</td></tr><tr><td>Scenario</td><td>A description of how in a Story the Persona's task/need/problem is solved before, during and after interaction with the resource/software/service being developed.</td></tr><tr><td>SMS tools</td><td>The three tools at the core of this deliverable: Story, <i>Maninpasta</i> and Survey.</td></tr><tr><td>Socio-pedagogical aspect</td><td>An item linked to a holistic and relationship-centred way of working in care and educational settings with the possible final users of <i>Polifonia</i>.</td></tr><tr><td>Source</td><td>Within a dataset, it is meant to define the origin of the input data, in opposition to the output data, that are the results collected by the pilot.</td></tr><tr><td>Story</td><td>A Story is a template for collecting requirements.</td></tr><tr><td>TB</td><td>The Technical Board, within the WP1, is in charge of the technical development, making sure that synergies are identified in a timely manner and issues around compatibility and technical interoperability are addressed harmonically during the development phase. The TB defines the common methodological approach and identifies the supporting collaborative tools to be used.</td></tr><tr><td>UI</td><td>User interface.</td></tr><tr><td>UNIBO</td><td>Alma mater studiorum – Università di Bologna.</td></tr><tr><td>URI</td><td>A Uniform Resource Identifier is a unique sequence of characters that identifies a logical or physical resource used by web technologies.</td></tr><tr><td>UX</td><td>User Experience Design.</td></tr><tr><td>WG</td><td>Working group.</td></tr><tr><td>WP</td><td>Work package.</td></tr><tr><td>XD</td><td>Extreme design, a framework for pattern based ontology design.</td></tr></table>
## Appendix 2: List of the pilots

<table><thead><tr><td>Polifonia's Pilot</td><td>Action</td></tr></thead><tbody><tr><td>BELLS</td><td>Bell structures are widespread both in urban and rural areas. They contribute to the distinctive shape of a landscape, to defining its soundscape and play as markers of daily, festive and ritual times. Bell heritage is complex and fascinating and influences our perception of the places we live daily. Both its tangible and intangible assets, and their dependencies are hardly encoded explicitly: most of this heritage is transmitted orally. This pilot intends to encode this valuable information in a knowledge graph, which will be publicly available and particularly relevant for scholars and cultural institutes.</td></tr><tr><td>ORGANS</td><td>The history of pipe organs is rich and diverse, and highly interrelated to economic, religious and artistic contexts. Currently, the information about building practices and characteristics of ~2000 Dutch pipe organs is only retrievable by manually paging a 15 volumes (4,500+ pages) encyclopaedia: the Orgelencyclopedie (1997-2010). This pilot will build a knowledge graph out of the text of Orgelencyclopedie, which will provide digital (and quick) access to such huge and detailed knowledge, including connection to data about aspects of their wider historic contexts.</td></tr><tr><td>FACETS</td><td>Music libraries currently lacks well-founded information retrieval tools. This pilot will design a faceted search engine (FSE) for music score collections, supporting explorations and discovery of scores of interest in large collections, based on features such as melodic, harmonic or rhythmic patterns, style, structure, instrumentation, metadata leveraged at the collection level.</td></tr><tr><td>INTERLINK</td><td>In order to answer research questions, musical heritage scholars need to combine diverse datasets (music scores, audiovisual materials, metadata) from European digital music libraries and audiovisual archives. They need to identify common entities and concepts implicitly present in the data, across different collections in different institutions. This process is mainly conducted manually in isolation and the diverse results are rarely connected. This pilot will focus on revealing and make compatible the entities and concepts hidden in digital music libraries and audiovisual archives.</td></tr><tr><td>CHILD</td><td>This pilot will build a knowledge graph of the historical experience of music in childhood, using life writing (letters, diaries, memoirs, travel writing) and other historical texts as sources for adult reflections on music heard in childhood, third-party observations on children's engagement with music, and children's own first-hand accounts. The resulting knowledge graphs will inform an interface enabling the exploration and analysis through the dimensions of themes, time, and space.</td></tr></tbody></table>
<table><tr><td>MUSICBO</td><td>Music has always played a central role in the city of Bologna. Nevertheless, its musical heritage is only partly known and enjoyed compared to its full potential. This pilot will create multilingual (English, French, Spanish and German) digital corpora, containing the testimonies of scholars, journalists, travellers, writers and students from medieval to modern times through published documents showing diverse discourse styles such as Stories, letters, reports, news, reportage, etc. These corpora will be the base to build a knowledge graph available to researchers, cultural institutes and public administrations for reuse.</td></tr><tr><td>TUNES</td><td>The digital music collection of the Meertens Instituut (Amsterdam) includes thousands of melodies from Dutch popular culture, spanning a period of more than five centuries. To trace possible international origins of Dutch early popular music culture, this pilot will interlink the entire melody collection of the Meertens Institute with a large number of other European collections. The linked melodic data sets will be highly valuable for musicologists and music historians interested in cultural evolution of musical style and in oral transmission and variation.</td></tr><tr><td>TONALITIES</td><td>The modal-tonal organisation of Western music is decisive for its inner coherence, its dramatic plot and, ultimately, for its artistic meaning. This pilot develops tools for the modal-tonal identification, exploration and classification of monophonic and polyphonic notated music from the Renaissance to the 20th century.</td></tr><tr><td>ACCESS</td><td>The aim of this use case is to co-design, develop and evaluate wearable haptic technology to enable people who are Deaf or hearing impaired to engage as audience members in live performances.</td></tr><tr><td>MEETUPS</td><td>This pilot focuses on supporting music historians and teachers by providing a Web tool that enables the exploration and visualisation of encounters between people in the musical world in Europe from c.1800 to c.1945, relying on information extracted from public domain books such as biographies, memoirs and travel writing, and open-access databases. These encounters will be explored in a timeline and map interface and may reveal unexpected connections and relationships that cast new light on aspects of European music history. The tool will provide persistent, citable identifiers in order to support referencing in scholarship outputs.</td></tr></table>
# Appendix 3: WP1 participants, efforts, objectives, tasks and deliverables

<table><tr><td>Title</td><td colspan="11">Web portal and pilots</td></tr><tr><td rowspan="2">Participant</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td><td>9</td><td>10</td></tr><tr><td>BO</td><td>OU</td><td>KCL</td><td>NUIG</td><td>ICCD</td><td>ICBSA</td><td>IREMUS</td><td>CNAM</td><td>NISV</td><td>KNAW</td></tr><tr><td>Person months</td><td>8</td><td>10</td><td>10</td><td>6</td><td>6</td><td>5</td><td>12</td><td>11</td><td>4</td><td>11</td></tr><tr><td>Start month</td><td colspan="3">1</td><td colspan="5">End month</td><td colspan="2">40</td></tr></table>

## Objectives

WP1 is Polifonia's validation WP, it contributes to achieve O3 (Tailor), and O5 (Share & Engage). As such it drives the whole development. It also delivers a registry of all resources and materials retrieved, used and produced in the project, in the form of a Web portal. Its goal is twofold: 1) to demonstrate that the methods and tools developed in the technology provider work packages (WP2-5) are effective in facilitating management of large musical heritage collections and supporting enhanced understanding, preservation of, and interaction with, musical heritage, 2) to contribute to push the state of the art in relevant, though specific, musical heritage use cases. It includes seven tasks: T1 delivers a socio-technical Roadmap to all partners ensuring supervision for the co-creation process; T2 provides a Technical Board supervising the development; T3 delivers a Web portal / registry and associated services; T4-7 develop ten pilots. Each task represents a theme relevant to the scope and challenges of the call: Preserving MH (T3), Managing MH (T4), Studying and understanding MH (T5), Interacting with MH (T6). Notice that ten "external" early adopters have already expressed their interest in participating in seven pilots, as explicitly indicated in the pilot descriptions (see also attached support letters).

## Tasks

### Task 1: Socio-Technical Roadmap (Leader: IREMUS - Participants: ALL)

This task provides a common framework to all pilots (T4-7) for: 1) coordinating collection of resources for use in the pilots, 2) gathering requirements from internal and external adopters/stakeholders, 3) setting objectives and challenges and mapping them from specific domains to the technology provider work packages (WP2-5), 4) monitoring validation within the pilot tasks. The goal is to maximise synergy and methodological soundness.

### Task 2: Technical Coordination (Leader: OU - Participants: ALL)

This task will provide the pilots and the technology provider WPs with a reference Technical Board (TB) which, according to the Roadmap defined in T1, will supervise the technical development making sure that synergies are identified in a timely manner and issues around compatibility and technical interoperability are addressed harmonically during the development phase. The TB will define the common methodological approach and identify the supporting collaborative tools to be used. It will interact with WP6 to make sure that all the developed components are released
according to the FAIR protocols defined in T7 of WP6. This task will also collect and report validation results from the pilots.

**Task 3: Polifonia Web portal: an aggregator of digital musical heritage collections (Leader: UNIBO - Participants: OU, NUIG, KCL)**

By extending musoW, a catalogue of Musical Data on the Web [http://musow.kmi.open.ac.uk/](http://musow.kmi.open.ac.uk/) (Daquino et al. 2018), this task will develop the *Polifonia Web portal*. It will include: 1) a reference registry of MH resources, including but not limited to all collections used and produced in the project, 2) methods for continuous indexing of MH resources, 3) methods for automatic and semiautomatic generation of metadata according to the *Polifonia ontologies*, 4) methods for searching, querying, browsing MH resources. The task will also provide input to WP6 about recommendations for the sustainability of the Web portal, beyond the lifespan of the project.

**Task 4: Preserving musical heritage through knowledge graphs (Leader: KNAW - Participants: UNIBO, OU, KCL, NUIG, ICCD, ICBSA, Nationaal Instituut voor de Orgelkunst (NiVO) (external), Soprintendenza Archeologica Bologna (external), Soprintendenza Archeologica Genova (external))**

This task includes two pilots focusing on building knowledge graphs for musical heritage that currently is hardly or not accessible, e.g. orally transmitted practices, embedded in texts, etc.

**Task 5: Managing musical heritage collections through knowledge graphs (Leader: KCL - Participants: ALL (except DP) + External adopters: BNF, CLARIN, DARIAH, CLARIAH, Europeana)**

This task includes two pilots focusing on building knowledge graphs for supporting curators, owners, etc. in managing their large collections

**Task 6: Studying musical heritage through (interlinked) knowledge graphs (Leader: IREMUS - Participants: UNIBO, OU, KCL, NUIG, KNAW, Biblioteca Universitaria di Bologna (external))**

This task includes pilots for demonstrating how *Polifonia* technologies can be leveraged to demonstrate, discover, document, support etc. research hypotheses/theories.

**Task 7: Interacting with musical heritage knowledge graphs (Leader: OU - Participants: UNIBO, ICBSA, The Stables Theatre, Milton Keynes, UK (external))**

This task includes pilots demonstrating novel interaction solutions. The goal is to show how musical heritage knowledge graphs can be leveraged for supporting accessible and improved user experience.

## Deliverables

* D1.1: Roadmap and pilot requirements - 1st version (M6)

* D1.2: Roadmap and pilot requirements - 2nd version (M18)

* D1.3: Pilots development: collaborative methodology and tools (M6)

* D1.4: Intermediate validation reports for pilots: ORGANS and BELLS (M24)

* D1.5: Intermediate validation reports for pilots: INTERLINK and FACETS (M24)

* D1.6: Intermediate validation reports for pilots: TONALITIES, TUNES, MUSICBO and CHILD (M24)

* D1.7: Intermediate validation reports for pilots: MEETUPS and ACCESS (M24)

* D1.8: Final ten-pilots validation report and lessons learned (M40)

* D1.9: *Polifonia* Web portal - 1st version (M18)

* D1.10: *Polifonia* Web portal - 2nd version (M36)
Appendix 4: Work packages in Polifonia project

<table><thead><tr><td>Acronym</td><td>Title</td><td>Tasks</td></tr></thead><tbody><tr><td>WP1</td><td>Web portal and pilots</td><td>1. Socio-Technical Roadmap<br>2. Technical coordination<br>3. Polifonia Web portal: an aggregator of digital musical heritage collections<br>4. Preserving musical heritage collections through knowledge graphs<br>5. Managing musical heritage collections through knowledge graphs<br>6. Studying musical heritage through (interlinked) knowledge graphs<br>7. Interacting with musical heritage knowledge graphs</td></tr><tr><td>WP2</td><td>Musical heritage knowledge graphs</td><td>1. Ontology-based knowledge graphs for music objects<br>2. Ontology-based knowledge graphs for music objects context<br>3. Interlinking knowledge graphs<br>4. Licences, ownerships, and conditions of use</td></tr><tr><td>WP3</td><td>Mining musical patterns</td><td>1. Pattern extraction<br>2. Pattern recognition and definition in monodic and polyphonic music<br>3. Network of musical patterns<br>4. Music classification from musical patterns</td></tr><tr><td>WP4</td><td>Extracting musical heritage knowledge from text</td><td>1. Building and evaluating multilingual text corpora on musical heritage. Themes, reception, and...<br>2. Automatic extraction of time, space, events, people and musical artifacts from text<br>3. Automatic extraction of socio-cultural and historical context of musical heritage<br>4. Evaluation of automatic knowledge extraction methods</td></tr></tbody></table>
<table><tr><td>WP5</td><td>Human music interaction and engagement</td><td>1. Evaluation of interaction components<br>2. Multidimensional searching, browsing and exploration of musical assets and knowledge<br>3. Haptic and gestural interaction with music<br>4. Pattern exploration, composition and visualisation<br>5. Publishing, accessing and reusing musical scholarly objects</td></tr><tr><td>WP6</td><td>Dissemination, Communication and Exploitation</td><td>1. Plan for exploitation and dissemination of results (PEDR)<br>2. Web presence and Polifonia image<br>3. Dissemination of project results<br>4. Innovation drive: stakeholder network<br>5. Enhanced accessibility and inclusion strategies<br>6. Musical heritage promotion: the Polifonia open source artistic digital installation</td></tr><tr><td>WP7</td><td>Project coordination and Management</td><td>1. Overall coordination of project activities, monitoring, quality control, reporting<br>2. Overall legal and contractual management<br>3. Consortium agreement, management of the knowledge generated by the project and IPRs<br>4. Ethical compliance<br>5. Financial and administrative management and reporting<br>6. Project governance and partnership communication<br>7. Data Management Plan</td></tr><tr><td>WP8</td><td>Ethics requirements</td><td>1. OEI – Requirement No. 1<br>2. H - Requirement No. 2<br>3. H - Requirement No. 6<br>4. H - Requirement No. 7</td></tr></table>
<table><tr><td></td><td>5. H - Requirement No. 8</td></tr><tr><td></td><td>6. POPD - Requirement No. 9</td></tr><tr><td></td><td>7. POPD - Requirement No. 10</td></tr></table>
# Appendix 5: Stories – updated template

## 1. Persona

* It is a research-based description of a typical user.

* It contains attributes such as name, age, occupation (if the Persona has more than one role, indicate which one is their primary role and which one(s) the secondary role(s)), and relevant characteristics of the person such as their knowledge and skills and their interests.

## 2. Goal

* It is a short textual description of the goal(s) that the Persona needs to be addressed in the Story.

* maximum number of characters: 1200.

* The goal(s) is(are) also represented by a short (maximum 5) list of keywords.

## 3. Scenario

* It is a Story describing how the Persona's task/need/problem is solved before, during and after interaction with the resource/software/service being developed.

* maximum number of characters: 1200.

## 4. Example Data

* This section provides examples of relationships pertaining to the Story as well as relevant links to online data

## 5. Data Resources

* List of resources (with references/links) where it is expected or known that the Persona can find what she's looking for.
# Appendix 6: Updated Survey template

## 1. DOMAIN SPECIFIC PART.

1. What are the pilot's goals?

2. What are the pilot's results and achievements so far?

3. What is the progress towards the pilot's objectives (please indicate %)

4. Are the sources digitized?

5. If so, are the sources born digital?

6. If your sources are online, can you provide links to the sources in a (partial) list?

7. Where is the data stored?

8. How scattered is musical knowledge data over different sources?

9. What is the nature of the sources?

10. Under what licences are the sources provided?

11. Who has the rights to the used datasets?

12. Research outcomes

13. What are your scientific assumptions and/or initial hypotheses?

14. What kind of scientific knowledge do you intend to produce (for example factual knowledge, conceptual knowledge, procedural knowledge, or metacognitive knowledge)?

15. Describe the knowledge units produced (melodic patterns, intertextual units, networks of composers and compositions, ...)

16. Is the new knowledge anchored to a specific fragment in a source?

17. Is the new knowledge purely descriptive? (e.g. a diplomatic transcription)

18. Can the new knowledge be the subject of dissensus?

19. Does the new knowledge involve an interpretation? And should the interpretative context make explicit through linked data?

20. Could the knowledge be inferred by algorithms, or can it only proceed from human interpretation?

21. Should it be possible to represent hypotheses and/or statements with varying degrees of certainty?

22. Could the knowledge proceed from collaborative scientific practices?

23. Does the new knowledge rely on a controlled vocabulary?

24. Does the new knowledge involve the creation of new controlled vocabularies to reflect specific concepts?

25. Could the links to the socio-cultural context be made explicit?

26. What are the pilot's main innovations? How do the methods and results contribute to the state of the art?

27. To which discipline/subdiscipline/specialty do you aim to contribute knowledge?
28. Identification/creation of analytical categories

29. Identification/creation of analytical and conceptual tools

30. Please provide links to the Competency Questions that your pilot aims to answer?

31. What are the pilot's representative Personas?

32. Please provide a specific contact person for questions related to the pilot's scientific dimension (pilot coordinator)?

33. What else has to be reported that is not covered by this part? Do you have any remarks?

2. TECHNICAL PART. The technical part specifies the actual implementation of the conceptual model:

1. Metadata: What formats are used?

2. Metadata: What standards are used?

3. Datasets: What formats are used?

4. Datasets: What standards are used?

5. Please list the controlled vocabularies, taxonomies, thesaurus, ontologies, etc. that you will use in your pilot

6. Please indicate if you plan to expand existing knowledge organisation systems or if you plan to create a specific one

7. Do you plan to support interoperability towards third parties?

8. Do you have APIs on your resources?

9. What algorithms will be developed for knowledge extraction or alignment?

10. Are there planned overlaps (already defined or envisioned) between the pilot's resources and other resources used within *Polifonia*?

11. What do you expect in terms of technologies from the technical provider WPs?

12. What are your needs from WP1?

13. What is the expected application to be released for this pilot?

14. Is it to be released through the Polifonia Web portal or as an independent application?

15. How should the pilot's resources be presented, linked, or stored on the website dedicated to the pilot?

16. What type of knowledge you expect can contribute to the *Polifonia Web* portal?

17. What type of interaction do you expect should be supported by the Web portal?

18. In which ways the Web portal shall link to the (separate) pilot demonstrator (if any)?

19. What kind of 'augmentation/enrichment' do you expect to get out of being displayed via the portal?
20. Can you provide a specific contact person for questions related to the pilot's technical aspects?

21. What else has to be reported that is not covered by this part? Do you have any remarks?

3. SOCIO-PEDAGOGICAL PART. The socio-pedagogical part aims to identify the target groups and their conditions

1. What is the target group of your pilot?

2. If academics, specify. Other?

3. Beyond the lead institution, who are the partners of the pilot and what is their involvement?

4. How is user/expert feedback to be taken into account in the pilot?

5. What kind of public demonstrators of the pilot could be made available in the coming months?

6. How do you intend to recruit users?

7. In order for the pilot to be successful, what criteria would the new tools and methods need to satisfy (e.g. certain tasks carried out more quickly or accurately, the amount or type of data that can be used in the task, number of people who are able to use the tools, the attractiveness of the interface, new tools being easy to learn, etc.)?

8. What will be the validation setup?

9. What are the pilot's validation methods?

10. What KPIs or metrics will be used for validating the pilot?

11. What values for KPIs/metrics shall be achieved for claiming success?

12. Can you name at least three stakeholders from different institutional background and countries who could contribute to the definition/validation of the pilot?

13. What facilities, if any, does the pilot make available for people with disabilities?

14. Can you provide a specific contact person for questions related to the pilot's socio-pedagogical implications?

15. What else has to be reported that is not covered by this part? Do you have any remarks?

4. Do you have any general comments and remarks?