# A cathedral of spatialised annotations portraying the multidisciplinary study of Notre Dame de Paris

Roxane Roussel, Livio De Luca, Anaïs Guillem, Florent Comte

► To cite this version:

Roxane Roussel, Livio De Luca, Anaïs Guillem, Florent Comte. A cathedral of spatialised annotations portraying the multidisciplinary study of Notre Dame de Paris. DIGITAL HERITAGE (2025), S. Campana, D. Ferdani, H. Graf, G. Guidi, Z. Hegarty, S. Pescarin, and F. Remondino (Editors), Sep 2025, Siena, Italy. 10.2312/dh.20253126. hal-05272937

HAL Id: hal-05272937

https://hal.science/hal-05272937v1

Submitted on 22 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# A cathedral of spatialised annotations portraying the multidisciplinary study of Notre Dame de Paris

Roxane Roussel<fnref n="1" />, Livio De Luca<fnref n="1" />, Anaïs Guillem<fnref n="1" />,<fnref n="2" />, Florent Comte<fnref n="1" />

## Abstract

This article examines annotation in the documentation field as more than a technical feature, framing it as a structured trace of expert activity embedded in spatial, temporal, and semantic contexts. Using the large scale, multidisciplinary worksite established after the 2019-fire at Notre-Dame de Paris, as a case study, it explores how annotations function as epistemic, multiscalar, and semantically rich knowledge objects that mediate observation, interpretation, and analysis. The interdisciplinary scientific worksite provides a unique setting to test large-scale annotation practices, with hundreds of scientists from diverse disciplines converging around a shared object of study, and address challenges in tool integration, terminology, and workflows. The study focuses on semantic annotation work conducted via the aïoli platform, a web-based 3D annotation tool, analyzing a corpus of 14,000 annotations linked to over 135,000 spatialized images.

## 1.Introduction

### 1.1. Operational context

On the night of April 15, 2019, a devastating fire broke out in the roof of Notre-Dame de Paris, causing the collapse of the spire and major damage to the vaults, timber framework, and upper parts of the cathedral. Beyond the immediate emergency response, the French Ministry of Culture launched a scientific worksite ("chantier scientifique") involving over 175 researchers and professionals from diverse disciplines. This unprecedented initiative was organized into thematic working groups (WG) addressing specific aspects of restoration and study, such as Wood, Metal, Decor, Acoustics, and Emotion Mobilization, Stone, each contributing domain-specific methods, expertise, and documentation practices. The Digital Data WG was given a transversal mission: to design and implement a digital ecosystem capable of integrating, structuring, analyzing, and preserving the vast and heterogeneous datasets produced by the project.

The challenges were manifold. These datasets ranged from historical sources to high-resolution photogrammetry, from stratigraphic paint samples to acoustic recordings, from 3D scans of charred timber to spreadsheets of handwritten site notes. Each reflected its own methodology, terminology, and objectives, mirroring the diversity of disciplines involved. The Digital Data WG therefore adopted a flexible, modular, and collaborative approach, developing tools and protocols so that each data type could find a place—operational and meaningful—within a coherent, searchable, and reusable digital ecosystem [NPTR*24]. Central to this effort is the aïoli platform, a web-based application

developed by the MAP laboratory for reality-based annotation and semantic enrichment of heritage data. In aïoli, annotations are more than digital labels: they are structured, spatially anchored, and semantically enriched entities linking observation, interpretation, and knowledge organization.

### 1.2. Annotations as traces of scientific or professional activity

Each annotation reflects a moment of interpretation: the recognition of a relevant feature, the formulation of a hypothesis, or the need to preserve a memory of material evidence. Annotations are epistemic traces, capturing not only what is observed, but how and why it is observed. The structure of an annotation—its textual content, categorization, metadata, and spatial anchoring—encodes expert decision-making: what to highlight, how to describe it, and how to relate it to other features or phenomena. A condition report marking a crack, for example, is not merely a factual note; it is the trace of a diagnostic process, an entry point into broader questions about material degradation, structural stability, or historical construction techniques. This perspective enables the retrospective study of annotations to reconstruct reasoning chains, compare disciplinary practices, or trace the evolution of knowledge and hypotheses across time and teams. Annotations are not only representations of the heritage object, but also expressions of the scientific and professional practices mobilized to study, understand, and restore it.

### 1.3. Purpose and structure of the article

This article contributes to the field of digital documentation by examining annotation not simply as a technical feature or

<footnote n="1">¹MAP UPR 2002 CNRS, campus CNRS, 31 chemin Joseph Aiguier, 13009 Marseille</footnote>

<footnote n="2">²Ludwig Maximilian University of Munich, Geschwister-Scholl-Platz 1, 80539 München, Germany</footnote>
interface element, but as a structured record of expert activity, embedded in spatial, temporal, and semantic contexts. Drawing on the large-scale, multidisciplinary worksite of Notre-Dame de Paris, it investigates how annotations can be created, enriched, queried, visualized, and reused as knowledge objects that mediate between observation, interpretation, and analysis.

The article follows the progressive development of this model: Section 1 presents the operational and institutional context; Section 2 outlines the challenges of integrating diverse documentation sources into a unified digital ecosystem; Section 3 introduces the tools supporting this integration, notably the aïoli platform. Section 4 describes the core annotation corpus, covering architectural datasets and remains, while Section 5 details the collaborative workflow for structuring and preparing data. Section 6 focuses on semantic enrichment, cross-referencing, and the transfer of annotations to a graph database (Neo4j) to enable querying and spatial-semantic navigation. Section 7 explores applications beyond documentation, including modeling, AI training datasets, and interoperable interfaces across heterogeneous systems. The article concludes with a reflection on the evolving epistemic role of annotations in the digital humanities and heritage research.

# 1.4. State of the art

## 1.4.1 From Documentation to Meta-Documentation

The documentation of cultural heritage has traditionally focused on recording observable characteristics of artifacts, sites, and environments to support conservation, interpretation, or scholarly inquiry. With the advent of digital technologies, this documentation has expanded to include 3D models, structured databases, and annotation systems. More recently, the field has seen growing interest in meta-documentation—the effort not only to record heritage objects, but to trace and structure the processes by which documentation itself is produced. This shift reflects broader developments in digital humanities and heritage studies that emphasize reflexive practices, where the epistemic status of data is interrogated alongside its content [Dal16][Ter22]. In this context, annotations are no longer seen merely as metadata markers or visual pointers, but as inscriptions of reasoning and interpretation—what some have described as epistemic artefacts embedded within research infrastructures [Dru11][RVdh14]. These reflections align with emerging perspectives in the theory of cultural techniques, where annotation is understood as a fundamental practice that shapes the production of knowledge in the digital domain [Krä23].

## 1.4.2 Annotation in Heritage Science: Roles and Limitations

In current heritage documentation platforms such as Arches, Recogito, and Omeka S, annotations are commonly used to tag or comment on specific textual, visual, or spatial regions of interest. While these systems support basic metadata enrichment and user interaction, they often fall short on semantic complexity, spatial anchoring, and the representation of longitudinal reasoning chains—critical for large-scale heritage research. The Arches platform [MAG16], for example, offers a robust geospatial and semantic data infrastructure using CIDOC CRM and controlled

vocabularies for heritage inventories, but in its default form focuses on cataloguing and requires customization to support fine-grained, process-based annotation workflows or cross-referenced analytical structures.

Recogito 2 [SBID17] provides a lightweight, web-based environment for semantic annotation of texts, maps, and images, emphasizing manual annotation and editorial control with features like entity linking to the Pelagios Gazetteer. However, its relatively flat annotation model limits the expression of nested or multiscale reasoning chains common in heritage diagnostics. In Omeka S, the Annotate module implements W3C Web Annotation standards for structured comments and highlights [Ber], enabling interoperable annotations but focusing on user-driven content management and lacking advanced semantic inference or ontology alignment by default. Together, these platforms illustrate the growing role of annotation in heritage documentation, while highlighting their limits in representing the rich epistemic, spatial, and temporal dimensions of heritage research—particularly in collaborative, cross-project, or interdisciplinary contexts.

## 1.4.3 Semantic and Spatial Anchoring of Annotations

Recent advances in semantic web technologies, graph databases, and knowledge modeling have enabled more expressive forms of annotation, especially when combined with spatial representations such as 3D models, point clouds, or maps. Several initiatives have explored how annotations can be linked to ontological structures, reinforcing their function as interoperable and durable data units. For instance, [PsSRS12] developed an interactive semantic enrichment tool for 3D cultural heritage collections, facilitating the annotation of 3D models based on the CIDOC Conceptual Reference Model (https://cidoc-crm.org/). The 3DSA project similarly offers semantic annotation services for 3D heritage models, integrating them into a semantic framework [Chi14]. In spatial humanities, geospatial references have been incorporated into historical annotation, extending to 3D heritage contexts. Projects like WissKI use the CIDOC CRM to provide an extensible ontology for cultural heritage documentation, improving access to museum-related information [BCG17]. Additionally, ResearchSpace has operationalized semantic technologies to enable scholarly research across museum datasets, linking annotations to ontological structures and visual interfaces to support both epistemic and user-driven exploration [OT18].

## 1.4.4 Towards Analytical, Multiscale and Multitemporal Annotation Corpora

Transforming annotations into reusable and analyzable corpora requires rethinking their internal structure, lifecycle, and positional logic within broader spatial and temporal frameworks. Cultural heritage investigations operate across multiple scales—from micro-level observations of tool marks to the architectural diagnosis of entire vaults—and multiple temporalities, comparing pre- and post-intervention states, tracking degradation over time, or aligning contemporary findings with archival records. Supporting this complexity demands annotation models that are: scale-agnostic, able to
reference fragments, objects, or building sectors within a unified conceptual space; time-aware, encoding the date of observation, its relation to other temporal layers (e.g., restoration phases, historical periods), and enabling cross-version comparisons; and interoperable, anchoring semantic descriptors to shared vocabularies and linking observations across diverse representations (images, models, documents).

This challenge aligns with initiatives in semantic versioning and provenance tracking consistent with FAIR data principles [GARC*25], and with emerging perspectives in digital hermeneutics and AI-driven sciences [DIAGR24], where annotations act as epistemic interfaces bridging spatial, temporal, and conceptual dimensions. Annotation systems become multiscale, multitemporal knowledge infrastructures supporting documentation, interdisciplinary analysis, comparative reasoning, and long-term monitoring of heritage assets.

## 1.5. Issues

The growing recognition of annotations as epistemic, multiscalar, and semantically structured documentation units calls for practical frameworks in which such models can be tested, refined, and applied at scale. The scientific worksite established after the 2019 Notre-Dame de Paris fire offers an exceptional context for this challenge, bringing together diverse expertise, documentary practices, and data types—all focused on a shared object of study but differing in tools, terminologies, and workflows. This diversity makes it both a site of urgent heritage intervention and a real-world laboratory for exploring how digital annotations can bridge observation and interpretation, scales and temporalities, and distinct epistemic cultures. Through semantic structuring, spatial referencing, and graph-based querying, the project not only applies annotation practices but also tests their limits, redefines their role, and opens new perspectives for interoperability and scientific collaboration.

In this unique environment—characterized by numerous stakeholders, diverse methodologies, institutions, expertise, and data types—the digital data WG proposed a data integration strategy using the aïoli annotation platform. Collaborating with various teams and grounded in their documentation and knowledge production, distinct themes and goals emerged, along with specific challenges. To faithfully represent the original material and meet the teams’ querying needs, it became essential to reassess data integration methods.

Thus, this led us to ask the following questions: How can we adapt to the specificities of each working situation and each objective pursued? In the absence of a process that could be universally applied across projects, how can we create a link to the source data, to ultimately produce a digital version on aïoli ? How can we structure the data to facilitate its query-ability while staying as faithful as possible to the provided source data? It became clear that data integration was not merely about replicating information in 3D space, but about designing collaborative processes with stakeholders, acting as translators of their data into the digital tools developed. The goal was not a one-size-fits-all method, but tailored workflows—from data collection and structuring, to identifying key elements for

integration, co-designing annotations, groups, layers, description sheets, and attachments within the aïoli platform. This also required appropriate 3D supports, prompting the processing and integration of several photogrammetric campaigns into the platform as a project collection. These steps will be detailed in this article and illustrated through case studies. The first focused was on analyses from the post-fire condition report by the architects leading the reconstruction [VFP19], alongside other expert contributions :

* Radar inspection of walls and vaults impacted by the fire (LERM laboratory, Digital Data WG)

* Coring analysis of collapsed stones (LRMH)

* Painted decor stratigraphic analysis (Parant Andaloro)

* Stained glass panel condition report (multiple actors)

* Deformation of vaults (Digital Data WG + Bestrema).

This work was continued with the topics of the WG, including:

* The catalog of lapidary signs throughout the cathedral (Emotion Mobilization WG)

* The inventory of the upper stained glass windows' lead sealing, staples, and gutter walls (Metal WG)

* Stratigraphic analysis of the western façade and analysis of stone samples (Decor WG)

* Condition report and dendrochronological analysis of timber remains (Wood WG)

* Cartographies of tool traces on the stones of the central nave's interior elevations (Stone WG)

* Spatialization of sound recordings (Acoustics WG)

## 2. Tools and source materials

### 2.1. The aïoli platform

The annotation corpus was established on the experimental platform [AMPC*23], developed at the MAP laboratory with support from the French Ministry of Culture. This web application enables collaborative annotation and semantic enrichment of heritage objects. Projects can be created either by importing a photogrammetric acquisition set—automatically processed through an integrated Micmac pipeline [PdDIR11]—or by uploading a zip file generated from a Metashape project via a conversion script to a Micmac workflow (https://page.hn/s9anv5) [PRHC*22]. To manage spatial referencing across the cathedral’s datasets, a feature was developed to align photogrammetry projects to topographic reference frames. This ensures metric consistency in aïoli projects and compatibility with other tools in the project’s digital ecosystem, including web viewers built with ThreeJS and PotreeJS. Once created, projects use the point cloud as a shared geometric framework for distributing and correlating 2D annotations across the entire image set. A 2D region drawn on one image undergoes a 2Dto3Dto2D projection process—implemented as a Docker-based web service—that propagates the annotation to the full photogrammetric set (2D) and the point cloud (3D). This operation can be run serially in the background. Annotations can be organized in groups and layers (Fig. 1) and enriched with semantic descriptors (text, dates, numerical values, lists, URLs) and supplementary resources (documents, images, videos, audio). Computed 3D descriptors add morphological information such as geometry, color, normals, roughness, and ambient occlusion.
<figure><img src="image_2.png" /><figcaption>**Figure 1:** System of annotations in aïoli, and generation of semantic, morphological and spatio-temporal layers</figcaption></figure>

## 2.2. Acquisitions and models

As the foundation of spatialized annotations, image-based digitization is central to linking on-site realities, methods, analyses, and interpretations—through both the final 3D model and the entire photogrammetric scene. In the project, two main types of digitizations were conducted: architectural scans of the cathedral and scans of remains recovered after the fire or during archaeological excavations, each with specific technical and application constraints. The dataset comprises a heterogeneous set of images from campaigns between 2002 and 2024, with varying coverage, conditions, and sources. Pre-fire data came from donors such as Professor Andrew Tallon (Vassar College, New York). Post-fire, during the clearing and securing phases, digitizations were carried out by AGP, Bestrema, Mercurio, and research labs including MAP and LRMH. AGP produced extensive terrestrial and aerial photogrammetry, while MAP used a cable-cam system to capture the vault extrados and fallen remains in the central nave. In 2023–2024, as restoration progressed, our team documented the nave and choir chapels, transept crossing, restored oculus, and galleries. Limited access and the need to avoid disrupting restoration led us to favor lightweight equipment and combine photogrammetry with lidar. These scans, though constrained, ensured spatial compatibility with the aïoli platform. For the corpus of remains, which includes hundreds of items, we were faced with a number of constraints in order to have digitizations that were compatible with the ecosystem : each artefact has its own morphological and material characteristics ; some collections (wood, stone) have been contaminated by lead, and must be digitised in approved areas; most collections have hundreds of objects to be digitised (250 woods, 150 keystones and stones of the oculus, 1100 fragments of the rood screen); some items require particular colour calibration for polychrome fragments or contrast work for items that are too dark, such as burnt wood. To meet these challenges, custom photogrammetric supports were developed and automated processing pipelines implemented, enabling over 1,200 scans (Jan 2023–June 2025) to be processed and exported in aïoli-compatible formats [CG24].

# 3. Projects on the aïoli platform

## 3.1. Corpus of architecture

Given the diversity and scope of the photogrammetric corpus, the objective was not to produce a single unified model of the

cathedral, but rather a set of projects reflecting its different temporal, spatial, and typological states. Cross-referencing and querying across these projects was ensured by aligning all datasets to common topographic reference frames, allowing consistent scale, spatial arrangement, and orientation. A spatial division (Fig. 2), based on the architects’ nomenclature and sub-zones [VFP19], distinguished two main sub-families: interior and exterior. Exterior areas were divided by level and bay for the nave and choir, and by levels and space types for the transept and western massif. The interior was divided by level and typology (chapels, nave, galleries, vaults, etc.). While some divisions matched the initial acquisition perimeters, others required subdividing larger datasets. A temporal division organized projects according to acquisition periods, since the same space could be documented multiple times (e.g., the vault extrados, scanned after the fire by drone, six times by cable-cam, and again after the timber framework reconstruction). On the aïoli platform, these temporal states are grouped into two heritage assets: pre-restoration and post-restoration. The corpus comprises 293 architectural projects (5–930 images each, average 130), totaling 38,000 spatialized images. Of these, 242 projects represent the pre-restoration state and 51 the post-restoration state, with 137 interior and 156 exterior projects.

<figure><img src="image_3.png" /><figcaption>**Figure 2:** Spatial division of the cathedral into aïoli projects, by level or span (exterior), and area of interest (interior)</figcaption></figure>

## 3.2. Corpus of remains

In addition to the architectural corpus, the remains have been integrated into aïoli, each one as a specific project, without a geographical reference frame, but with a consistent scale and colorimetry within the corpus. The corpus currently includes 183 projects, with 138 wooden remains of the timber structures, 16 metal staples from the gable walls, 5 statues, as well as 24 acquisitions of remains on-site before sorting and removal. The corpus of voussoirs from the double arch of the vault, the nave, and the rood screen are in the process of integration. In total, 31 000 images of the corpus of remains are on the platform, with an additional 73 000 images of the rood screen currently uploaded.

## 3.3. Project data structuring and storage

With a dataset of 476 projects, the need arose to establish a standardized nomenclature that would support the organization and querying of the collection. This system was designed to capture, in a concise title, essential information about each project—namely, its inclusion in the corpus of architecture or remains, its temporal status, its spatial location within the cathedral (interior or exterior), its typology (e.g., a vault, a chapel), and any further identification, such as a reference number. This nomenclature was primarily based on reference frameworks established by the architects, which attributed short
identifiers—often two-letter codes—to the cathedral’s subspaces (e.g., NF for Nave). These identifiers were also integrated into the Opentheso platform. By combining these codes, each project’s scale, typology, and location were encoded. Additional metadata (unique IDs, authorship, number of images, creation methods, annotations, etc.) were compiled in a centralized spreadsheet, enabling fast and efficient access to key project information.

# 4. The source data

## 4.1. The restoration worksite and its condition report

The first dataset integrated—both chronologically within the project and in terms of source volume—was the condition report compiled by architects Philippe Villeneuve, Rémi Fromont, Pascal Prunet, and their teams between 2019 and 2021 [VFP19]. Produced as multi-volume reports, it documented post-fire studies on the cathedral’s exterior and interior. Each volume comprised written documents, graphical materials, and appendices, covering historical information, architectural descriptions, condition assessments, and preliminary intervention proposals. The graphical component, based on orthophotographs, annotated plans, and detailed photographs, located and clarified information. Encompassing varied topics, areas, and representations, this corpus was the first integrated into Aïoli and remains the largest annotated dataset [RDI23], with 10,836 annotations (78% of the total). Its structure—general information (painted decors, furnishings, etc.) and diagnostics with recommended interventions—mirrored the graphical documents.

The appendices compiled third-party studies informing the architects’ recommendations: structural analyses (geometry, wind load, collapsed arches, vault consolidation) by Bestrema and Studio Comes; radar scans by LERM; material studies (stone decay, mechanical properties, salts, moisture, mortars) by BPE and LRMH; a report on collapsed voussoirs by the Stone WG; and a panel-by-panel stained glass diagnosis by master glaziers. Annotating these analyses bridged the condition assessment phase and the collaborative scientific worksite, as their written format matched the former while their analytical depth aligned with the latter.

Additional datasets included: radar inspection of damaged walls and vaults (LERM, Digital Data WG) – 30 annotations; coring analysis of collapsed stones (LRMH); stratigraphic study of painted decor (Parant Andaloro Restoration) – 19 annotations; stained glass condition report – 905 annotations; and post-fire vault deformation analysis (Digital Data WG, Bestrema) – 263 annotations.

## 4.2. The scientific worksite and its multiple areas of study

The annotation of data produced by the scientific project's various members and working groups evolved through ongoing collaboration rather than relying solely on written reports. To ensure long-term preservation of the project's vast, heterogeneous datasets—varying in subject, scale, and temporality—a range of materials has been progressively integrated.

<figure><img src="image_4.png" /><figcaption>Figure 3: Timber remains on the extrados of the vaults in aïoli</figcaption></figure>

The Wood WG’s post-fire study of the timber framework remains involved a two-scale approach, posing challenges for multi-scalar and multi-format integration. At the first scale, a global survey mapped the framework using photographs to document wooden remains still in place on the vault. At the second scale, individual remains were digitized, sectioned for dendrochronological analysis [PGHM*24], and described in summary sheets recording condition data (Fig. 3). In aïoli, annotations were organized into distinct project environments: large-scale annotations included a “general information” layer (provenance, dating, infestation, etc.) and links to digitized models, bridging the two temporalities of study. At the finer scale, two layers located dendrochronological slices and condition data. In total, 305 elements were annotated at the large scale. Of the 138 digitized fragments, 96 have so far been studied and annotated.

The stratigraphic analyses of the portal decorations and the Galerie des Rois on the western façade, by the Decor WG posed challenges related to multi-temporal data integration. Reports from the 1960s to today documented a hundred samples, detailing location, dating, and polychromy layers. These were integrated into 4 aïoli projects, with 114 samples annotated. A key challenge was reconciling data collected at different times and in varying formats across ten separate reports within a unified environment.

Across other study areas, despite varied subjects, a common issue emerged: the need to create digital cartographies of observed elements. This applied to the catalog of lapidary signs (Emotion Mobilization WG), the inventory of lead sealings, staples, and gutter walls in the upper stained-glass windows (Metal WG), and the mapping of tool marks on the nave's interior elevations (Stone WG). For each, spatial localization was the first step, enabling the addition of analytical data. For the 84 stone mark annotations, this meant identifying the tool type, symbol, or inventory number; for the 643 lead sealing annotations, it involved recording their locations to show distribution across bays, followed by inventory or chemical data. The integration of further datasets is ongoing and introduces new challenges. A notable example is the acoustic recordings by the Acoustics WG [DmBMV*19], which raise the issue of localizing non-visual data. In this case, photographs in the 2D/3D environment serve not to represent the object of study, but the location from which the study was conducted.
# 5. Project data structuring

## 5.1. The matter of standardization in projects

The diversity of disciplines and research topics in the scientific project resulted in varied working methods, document types, formats, and data recording approaches, ranging from hand-drawn or digital illustrations to summary sheets, spreadsheets, architectural plans, and textual documents (digital or handwritten) (Fig. 4). This heterogeneity, combined with long-term, multi-contributor research, meant that key information often came from different sources and used varying terminologies. It was therefore necessary to reorganize the data and establish a coherent structure before preserving and integrating it into the aïoli platform. The aim was not to alter the original data or terminology—each reflecting its disciplinary context—nor to impose a unified nomenclature. Instead, the process involved collecting all available data, designing a structure based on categories from the original materials while ensuring overall consistency, and later aligning terminology via the Opentheso platform in collaboration with the original data producers. Finally, organizing the information in spreadsheets, with categories as columns, enabled its conversion into structured description records within aïoli.

<figure><img src="image_5.png" /><figcaption>**Figure 4:** Source data from the scientific worksite: Identification of seals for high bays (Metal WG, A. Azéma), summary sheets for wood analysis (Wood WG, C. Penagos, M. Lafarge), location of lapidary signs (D. Syvilay et S. Hardy), stratigraphic analyses of the western facade (Decor WG, S. Duchêne)</figcaption></figure>

## 5.2. Semantic enrichment

To complement the visual information conveyed by the reprojected annotation, a set of descriptive fields was defined and implemented within each project. These fields could take various formats—text, note, decimal, thesaurus term, hyperlink, boolean, or date—with their type, number, and label determined at the layer level, each layer grouping observations around shared themes. Users populated these fields for each individual annotation (Fig. 5). In addition, supplementary files (e.g., PDFs, images, audio) could be attached to further contextualize the visual data and textual descriptions. Collectively, these elements formed structured description sheets, enhancing the semantic depth of both the annotations and the overall project. The standardized structure and content of these sheets, and the coordination with the original data providers ensured both the

reliability of the integrated information and the interoperability of the dataset, facilitating subsequent data queries and visualization.

<figure><img src="image_6.png" /><figcaption>**Figure 5:** Collection of various description sheets and joined files from the aïoli projets created for the scientific worksite</figcaption></figure>

## 5.3. An iterative process

The entire process—data collection, structuring, and integration—was carried out iteratively. As noted, these stages were closely coordinated with the original data producers to ensure accuracy and align with expectations regarding data localization, querying, and visualization within the digital ecosystem. Joint working sessions, often via videoconference, were held throughout to monitor the integration process. Semantic enrichment and annotation tasks within the aïoli platform were initially handled by the digital data WG, with a subsequent validation workflow involving the collaborative group to verify methodological soundness and accuracy. This iterative approach allowed testing of different data integration strategies, particularly in organizing annotations within thematic layers, aiding informed decision-making. The integration timeline mirrored the scientific project's pace. Some pre-analyzed datasets, like the stratigraphic analysis from the Decor WG, were integrated directly. In other cases, such as with the Wood, Metal, or Emotion Mobilization WG, structuring data was the first step, with integration into aïoli progressing as data were processed and delivered. The platform also allowed for iterative updates, enabling revisions or corrections as necessary.

## 5.4. The collaborative annotation corpus

The various stakeholders in the scientific and restoration worksites are represented in the platform by distinct user accounts, enabling project sharing and collaborative annotation. After processing in Metashape, projects were imported under "parent" accounts. These projects were then shared with specific users to integrate their data. As a result, aïoli projects corresponding to areas with diverse scientific investigations contain groups and thematic layers linked to the respective data producers. The entire corpus comprises 13,788 annotations, distributed across nine accounts and thematic domains (Fig. 6), excluding datasets still under experimental integration—such as those from the Acoustics and Stone WG. The volume of annotations varies significantly by subject area, ranging from 19 to 894 for the scientific worksite studies, and up to 10,836 for condition report annotations. The diversity and scale of this corpus, alongside the structuring efforts undertaken, provided a rich foundation for the first experiments in data visualization, cross-referencing, and large-scale querying of the corpus.
<figure><img src="image_7.png" /><figcaption>**Figure 6:** Number of annotations and subjects, by aïoli account</figcaption></figure>

## 6. Cross-referencing and large-scale querying of the annotation corpus

The evolution of annotation practices on the aïoli platform within the project has progressively transformed annotations from localized observations (related to cathedral parts) into fully integrated data objects, embedded within a semantic framework and capable of being cross-referenced across an extensive and heterogeneous corpus. Central to this transformation is the implementation of a semantic enrichment pipeline and the transfer of annotations from the original document-oriented database (CouchDB) to a native graph database (Neo4j), which enables powerful, context-aware queries at scale.

### 6.1 From descriptive fields to semantic signatures

Each annotation created on aïoli contains a variety of descriptive elements (project, group and layer hierarchy, name, description, custom description fields, associated documents...). While these descriptors are input manually and vary in form and terminology, they reflect a valuable layer of situated knowledge, grounded in the working practices of each contributor. To unlock the analytical potential of this content, a process was developed to extract and normalize this raw descriptive matter. The extraction pipeline reads all relevant information from the annotation database—titles, group and layer names, annotation fields and values, and any structured or semi-structured descriptors provided by users. By using a dedicated NLP based process, these terms are automatically categorized by origin, resulting in a sort of ‘semantic signature’ for each annotation, based on collections of ‘tags’. For instance, the term “stratigraphy” may appear in a layer name, while “pigment” could be mentioned in a free-text description field, and “LRMH” as the actor responsible for the annotation. All of these are retained, but tagged with their source (group, field, actor), preserving their context of use. This approach ensures that each annotation is associated with a structured and interpretable semantic footprint, which can then be reused for navigation, indexing, and querying.

### 6.2 Indexing in a graph: enabling semantic-based querying

Once the semantic signatures have been generated, they are imported into a Neo4j graph database. Each annotation becomes a node in the graph, linked to other nodes representing concepts, actors, spatial zones, materials, and projects. The terms extracted

from the annotation are mapped to nodes in a hierarchical concept taxonomy, making it possible to establish complex connections across the entire corpus (Fig. 7).

<figure><img src="image_8.png" /><figcaption>**Figure 7:** Integrated visualization of a set of spatialized annotations and the graph of associated semantic relationships.</figcaption></figure>

This structure allows users to formulate complex, multi-criteria queries that go far beyond what is possible within a project-by-project interface. For example, one might retrieve and spatialize all annotations related to specific diagnostic phenomena allowing for their visualization within a unified interface for the interactive visualisation or 2D-3D semantically-enriched content (Fig. 8). Each of these annotations is located both in the 3D space of the associated point clouds and in the 2D coordinates of the original photographic documentation. This dual anchoring enables the construction of a visual 2D catalog of observations while maintaining their correspondence with their spatial positioning in 3D, thus bridging image-based interpretation with geometric context.

<figure><img src="image_9.png" /><figcaption>**Figure 8:** Example of 4 queries across the annotation corpus, in a unified interface, targeting distinct diagnostic phenomena: (1)</figcaption></figure>
stone deterioration, (2) surface deformation, (3) efflorescence, and (4) flaking. Queries can span multiple projects and annotation campaigns, breaking down the silos imposed by spatial divisions or working groups. Since every node retains information about its provenance and relationships, it is possible to filter results by contributor, timeframe, annotation layer, etc.

## 6.3 Bridging the semantic interface and the 3D viewer

To link semantic exploration with spatial understanding, each annotation remains anchored to its original 2D+3D context. When a user selects an annotation our interface can automatically focus the camera within a 3D viewer (based on a dedicated Javascript viewer built on Three.js, Potree.js and Nexus [Abe21]) on the corresponding location in the point cloud or 3D model. This interaction is made possible by the precise spatial referencing of annotations during their creation in aïoli, as well as the alignment of photogrammetric datasets with a shared reference frame. It ensures that the act of annotation remains not only a textual or conceptual gesture, but also a spatial one. In addition to spatial navigation, a timeline interface is under development to display annotations along a temporal axis. This will allow researchers to visualize how annotations evolve across different stages of the restoration project, compare multiple observation points over time, or synchronize spatial data with external events (e.g., intervention milestones, structural measurements, environmental recordings).

## 6.4 Querying across object collections

Beyond architectural datasets anchored to the cathedral's topographic reference frame, the Aïoli annotation model supports corpus-level queries across collections of mobile or fragmented objects. A notable example is the timber remains corpus. To enable structured querying, annotations are processed through the same pipeline as architectural data: descriptive elements are extracted, tagged by origin, and injected into the Neo4j graph. Instead of being placed in a unified architectural space, the objects are spatially arranged in the 3D viewer according to analytical dimensions, creating a virtual spatialization of heterogeneous data. This allows comparative queries across the corpus—identifying similar tool marks, selecting fragments with dendrochronological matches, or isolating objects with comparable burn patterns—visualized directly in the 3D interface through analytical layouts designed for cross-comparison rather than architectural reconstruction (Fig. 9).

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /><figcaption>**Figure 9:** Interactive visualization of a query targeting annotations related to dowel on the corpus of timber remains. Top: spatial distribution of the annotations across the 3D layout of fragments, allowing comparative analysis. Bottom: detailed view of a selected annotation showing its dual anchoring—on the 3D surface of the point cloud and on the original 2D image—as well as associated user-generated descriptors and automatically computed geometric and visual metrics.</figcaption></figure>

This enables scale shifts in analysis, from the local (individual object or architectural feature) to the global (systemic patterns across materials or conservation states), reinforcing the integrative nature of the platform across types of data, levels of observation, and scientific approaches.

## 7. (Conclusion) Beyond documentation: the annotation as a research object

While annotations are often considered as end products—marks made to document an observed state, a condition, or a feature—our work with the aïoli platform and the Notre-Dame corpus has shown that annotations can be regarded as research objects in their own right. They are not merely auxiliary elements of documentation; they can be considered as expressions of scientific activities, embedded in both material context and epistemic intent.

### 7.1. From annotation to modeling

Annotations often serve as entry points for advanced modeling processes. Their structured nature allows translation into mathematical, statistical, or procedural forms, bridging qualitative observation and quantitative analysis. More than simple markers, they mediate between empirical data and formalized inquiry. For instance, in the structural diagnosis of Notre-Dame de Paris's buttressing system [GDIDV*25], annotations of alterations (e.g., mortar cracks, block dislocations) were extracted, categorized via controlled vocabularies (e.g., ICOMOS glossary on stone degradation), and embedded into a knowledge graph. This enabled correlation with mechanical models using the discrete element method (DEM), allowing geometric and semantic comparisons between observed features and predicted collapse scenarios. Thus, observed annotations become comparable with predicted phenomena, supporting a feedback loop between qualitative observation and quantitative modeling.
## 7.2 Annotations as training data for AI

The richness and structuring of the annotation corpus within aïoli make it a strong foundation for training machine learning and AI models in cultural heritage preservation [RGDI23]. Each annotation combines spatial and visual coordinates—2D image regions and their 3D projections onto point clouds—with semantic descriptors (keywords, categories, user-defined properties), contextual metadata (source documents, user roles, project affiliations), and high-resolution visual references (dense point clouds and calibrated textures). Together, these form a valuable dataset for supervised learning. They allow the generation of large, labeled corpora that capture complex degradation phenomena and contextual information about the architectural fabric of Notre-Dame. As shown in [WRMGb*24], a subset of 180 annotations was used to create a dataset for detecting and segmenting stone degradation patterns, converted into COCO format to support state-of-the-art computer vision pipelines while preserving heritage specificity. It covers nine degradation categories (e.g., cracks, lacunae, scaling, erosion), manually aligned with conservation glossaries such as the ICOMOS Illustrated Glossary on Stone Deterioration Patterns.

Crucially, the semi-automated propagation system of annotation allows a single annotation to be projected across multiple views and onto the 3D model, multiplying its representational utility. This mechanism significantly increases the size and variability of the training data while maintaining semantic consistency. It also enables multimodal learning, where annotations can be correlated with both geometric and visual features.

## 7.3 Reuse and interoperability

Because annotations in aïoli are stored in a semantically rich, structured, and exportable format, they are not confined to a single platform or project. They can be reused across analytical contexts, disciplinary domains, and phases of research. A single set of annotations describing tool marks in the nave, for example, may inform archaeological hypotheses about construction techniques, feed structural analyses of load distribution, and later contribute to interpretation panels or educational materials for public outreach. Similarly, annotations originally created to support restoration planning may be revisited in longitudinal studies to assess material aging over time or to compare interventions across different restoration campaigns. Their reuse should be made possible by the alignment of annotations with shared reference frameworks—spatial [GRADI23] (e.g., topographic information and coordinate systems), semantic (via concept hierarchies and domain-specific vocabularies), and temporal (restoration phases, historical layers). This alignment ensures interoperability across tools and systems, enabling annotations to be queried and visualized not only within aïoli, but also within external applications and platforms.

To support this, we are currently developing a RESTful API, documented through the OpenAPI standard, which will soon provide programmatic access to the entire corpus of annotations. This service will allow researchers and developers to query, filter, and retrieve annotations based on spatial areas, concept tags, contributors, or linked resources—opening the way for advanced

data exploration, integration with third-party tools, or even automated processing pipelines. However, the availability of such a rich and extensive annotation corpus raises critical questions about data formats and standardization. At present, there is no established standard for hybrid annotations that simultaneously cover 2D and 3D representations, while maintaining the semantic depth and contextual richness necessary for research use. The annotations produced in aïoli occupy precisely this hybrid space: they are spatialized both in image space and in 3D geometry, yet their core function is to mediate between observation and semantic interpretation. In this sense, annotations are not merely interoperable data points—they are dynamic interfaces between representations, able to bridge spatial and temporal data, conceptual frameworks, and evolving research interpretations across systems and disciplines.

## Acknowledgments

This work has been funded since 2019 by the CNRS and the French Ministry of Culture within the national scientific action Notre-Dame de Paris, and, since 2022, by the European Research Council (ERC Advanced Grant nDame_Heritage: n-Dimensional analysis and memorization ecosystem for building cathedrals of knowledge in Heritage Science, Project ID: 101055423). The authors acknowledge the collaboration of the chief architects of historical monuments—Philippe Villeneuve, Pascal Prunet and Rémi Fromont—the Établissement public chargé de la conservation et de la restauration de la cathédrale Notre-Dame de Paris (RNDP), and the heritage conservators. They also thank the numerous scientific partners and collaborators, with special recognition to the scientific worksite working groups, and in particular to colleagues at Archéovision, MIS, MOM, LRMH, LCE, ArScAn, IRAMAT, Chrono-environnement, and the Héritages: Culture/s, Patrimoine/s, Création/s laboratory. Finally, the authors express their gratitude to AGP, Vassar College New York, Mercurio, A-Bime, and BESTREMA for sharing their data.

## References

[AMPC*23] ABERGEL V., MANUEL A., PAMART A., CAO L., DE LUCA L.: Aïoli: A reality-based 3D annotation cloud platform for the collaborative documentation of cultural heritage artefacts. Digital Applications in Archaeology and Cultural Heritage, 2023, 30, pp.e00285. https://doi.org/10.1016/j.daach.2023.e00285

[Abe21] ABERGEL V.: A viewer for Notre-Dame de Paris multidimensional data, https://[X].fr:42000/[online]

[Ber] BERTHEREAU, D.: Annotate – Omeka S Module, (n.d.) https://omeka.org/s/modules/Annotate/

[BCG17] BRUSEKER G., CARBONI N., GUILLEM A.: “Cultural Heritage Data Management: The Role of Formal Ontology and CIDOC CRM.” In Heritage and Archaeology in the Digital Age, 93–131. Springer, (2017).

[Chi14] CHIH-HAO Y.: Semantic annotation services for 3D models of cultural heritage artefacts. PhD Thesis, School of Information Technology and Electrical Engineering, The University of Queensland, (2014). https://doi.org/10.14264/uql.2014.617
[CG24] COMTE F., GATTET E., De l'acquisition à l'archivage : solutions matérielles et logicielles intégrées pour la numérisation 3D du patrimoine de Notre-Dame. n-Dame 2024 : une cathédrale de données numériques et connaissances pluridisciplinaires, CNRS; Ministère de la Culture; MAP - UPR CNRS 2002, 2024, Marseille, France.

[Dal16] DALLAS C.: Curating Archaeological Knowledge in the Digital Continuum: From Practice to Infrastructure. Open Archaeology, 2(1), (2016), 232–247.

[DIGR24] DE LUCA L., GUILLEM A., RÉBY K.: “Intelligences humaines, collective et artificielle dans le chantier scientifique et de restauration de Notre-Dame de Paris.” In Recherche et intelligence artificielle, 8–12. Culture et Recherche. Ministère de la Culture, (2024).

[DmBMV*19] DE MUYNKE J., BALTAZAR M., MONFERRAN M., VOISENAT C., KATZ B.: Ears of the Past, an Inquiry into the Sonic Memory of the Acoustics of Notre-Dame before the Fire of 2019. Journal of Cultural Heritage 65, (2024). https://doi.org/10.1016/j.culher.2022.09.006.

[Dru11] DRUCKER J.: Humanities Approaches to Graphical Display. Digital Humanities Quarterly, (2011).

[GARC*25] GUILLEM A., ABERGEL V., ROUSSEL R., COMTE F., PAMART A., DE LUCA L.: “Provenance as the Missing Link in Digital Heritage Data from 3D Digitization to Semantic Enrichment (under Review).” Heritage, no. Advanced Data Environment in Current Cultural Heritage 3D Digitization Practices, (2025).

[GRADI23] GUILLEM A., GROS A., RÉBY K., ABERGEL V., DE LUCA L.: “RCC8 for CIDOC CRM: Semantic Modeling of Mereological and Topological Spatial Relations in Notre-Dame de Paris.” In . Athens, Greece, (2023).

[GDIDV*25] GROS A., DE LUCA L., DUBOIS F., VÉRON P., JACQUOT K.: From surveys to simulations: Integrating Notre-Dame de Paris’ buttressing system diagnosis with knowledge graphs. *Automation in Construction* 170, 105927 (2025).

[Krä23] KRÄMER, S.: Should We Really ‘Hermeneutise’ the Digital Humanities? A Plea for the Epistemic Productivity of a ‘Cultural Technique of Flattening’ in the Humanities. Journal of Cultural Analytics, (2023).

[MAG16] MYERS D., AVRAMIDES I., GANT M.: The Arches heritage inventory and management system: A platform for the heritage field. Journal of Cultural Heritage Management and Sustainable Development, (2016), 213–224. https://doi.org/10.1108/JCHMSD-03-2016-0012

[NPTR*24] NÉROULIDIS A., POUYET T., TOURNON S., ROUSSET M., CALLIERI M., MANUEL A., et al.: A Digital Platform for the Centralization and Long-Term Preservation of Multidisciplinary Scientific Data Belonging to the Notre Dame de Paris Scientific Action ». Journal of Cultural Heritage 65 (janvier 2024): 210-20. https://doi.org/10.1016/j.culher.2023.09.016.

[OT18] OLDMAN D., TANASE D.: Reshaping the Knowledge Graph by Connecting Researchers, Data and Practices in ResearchSpace. In The Semantic Web – ISWC 2018: 17th International Semantic Web Conference, Monterey, CA,

USA, October 8–12, 2018, Proceedings, Part II. Springer-Verlag, Berlin, Heidelberg, 325–340. https://doi.org/10.1007/978-3-030-00668-6_20

[PRHC*22] PAMART A., ROUSSEL R., HUBERT E., COLOMBINI A., SALÉRI R.: A semantically enriched multimodal imaging approach dedicated to conservation and restoration studies. *Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.* XLVI-2/W1-2022, 415–420 (2022).

[PGHM*24] PENAGOS C., GIRARCLOS O., HUNOT J. Y., MARTIN C., JACQUOT K., et al : Naming, Relocating and Dating the Woods of Notre-Dame “Forest”, First Results Based on Collated Data and Archaeological Surveys of the Remains. Journal of Cultural Heritage 65 (2024): 21-31. https://doi.org/10.1016/j.culher.2022.09.004.

[PsSRS12] PENA SERNA S., SCHMEDT H., RITZ M., STORK A.: *Interactive Semantic Enrichment of 3D Cultural Heritage Collections*. In D. Arnold, J. Kaminski, F. Niccolucci, & A. Stork (Eds.), *VAST: International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage* (pp. 33–40), (2012). Eurographics Association.

[PdDIR11] PIERROT-DESEILLIGNY M., DE LUCA L., REMONDINO F., 2011. Automated Image-Based Procedures for Accurate Artifacts 3D Modeling and Orthoimage Generation. Geoinformatics FCE CTU 6, 291–299. https://doi.org/10.14311/gi.6.36

[RGDI23] RÉBY K., GUILLEM A., DE LUCA L.: “Semantic Segmentation Using Foundation Models for Cultural Heritage: An Experimental Study on Notre-Dame de Paris.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1689–97, (2023).

[RVdh14] ROORDA D., VAN DEN HEUVEL C.: Annotation as a New Paradigm in Research Archiving (2014). arXiv:1412.6069.

[RDl23] ROUSSEL R., DE LUCA L. An approach to build a complete digital report of the notre dame cathedral after the fire, using the aioli platform. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLVIII-M-2-2023 (26 juin 2023): 1359-65. https://doi.org/10.5194/isprs-archives-XLVIII-M-2-2023-1359-2023

[SBID17] SIMON R., BARKER E., ISAKSEN L., DE SOTO CAÑAMARES P.: Linked Data Annotation Without the Pointy Brackets: Introducing Recogito 2. Journal of Map & Geography Libraries, 13(1), (2017), 111–132. https://doi.org/10.1080/15420353.2017.1307303

[Ter22] TERRAS M.: Digital Humanities and Digitised Cultural Heritage. In A New Companion to Digital Humanities, Wiley-Blackwell, (2022).

[VFP19] VILLENEUVE P., FROMONT R., PRUNET P.: 2019. Étude de diagnostic à la suite de l'incendie du 15 avril 2019.

[WRMGb*24] WILLOT L., REBY K., MANUEL A., GOUET-BRUNET V., VODISLAV D., DE LUCA L.; Creating a Dataset for the Detection and Segmentation of Degradation Phenomena in Notre-Dame de Paris. in Proceedings of the 6th workshop on the analySis, Understanding and proMotion of heritAge Contents 5–12 (ACM, Melbourne VIC Australia, 2024). https://doi.org/10.1145/3689094.3689473