# Spatialization, fusion and enrichment of Multimodal Imaging for Interdisciplinary Digital Heritage Studies

Anthony Pamart, Livio De Luca, Philippe Véron

► To cite this version:

Anthony Pamart, Livio De Luca, Philippe Véron. Spatialization, fusion and enrichment of Multimodal Imaging for Interdisciplinary Digital Heritage Studies. Digital Heritage International Congress 2025, Sep 2025, Sienne, Italy. pp.Digital Heritage International Congress 2025, 10.2312/dh.20253104. halshs-05262909

HAL Id: halshs-05262909

https://shs.hal.science/halshs-05262909v1

Submitted on 16 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Spatialization, fusion and enrichment of Multimodal Imaging for Interdisciplinary Digital Heritage Studies

A. Pamart¹, L. De Luca¹ and P. Veron²

¹ Modèles et Simulations pour l'Architecture et le Patrimoine, UPR 2002 CNRS, 13009 Marseilles, France

²LISPEN (EA 7515), Arts et Métiers Institute of Technology, 13617 Aix-en-Provence, France

## Abstract

Digital Heritage (DH) has emerged as a dynamic and crucial concept for the future of Heritage Science (HS), at the crossroads of interdisciplinary fields. Nowadays, methods, tools and technologies for capturing reality play a predominant role in Cultural Heritage (CH) documentation framework. Indeed, the 2D/3D imaging and digitization techniques are massively employed today to survey, document, and study heritage artifacts. They offer numerous perspectives and avenues for investigation to enhance shared knowledge around heritage conservation and restoration. However, as the need for digital expertise expends in both scope and number, these various methods employed face the challenge of multimodality. This concept is at the core of this research, it is understood and justified by the growing need to cooperate with various digital resources (originating from multiple sensors, scales of observation, spectral or temporal layers) in order to feed and cross expertise. This paper explores 2D and 3D digitization strategies to enhance the potential of multimodal studies. The main contributions develop data-driven methods improving Spatialization, Fusion and Enrichment stages. Different types of approaches are proposed, aiming to better articulate the instrumental, computational and analytical phases in order to increase the informative potential of multi-source 2D/3D modelling. The strategy is demonstrated by a series of works and experiments enabling to construct and explore enriched multimodal data sets. The proposal offers reasoned and pragmatic solutions while the discussion anticipates emerging challenges such as semantic technologies, open science, artificial intelligence, and digital sobriety.

## 1. Introduction

### 1.1. Digital(ized) Heritage(s) and multimodality

While Digital Heritage (DH) [AGS18] is now an understood framework for the scientific communities studying Cultural Heritage (CH) [Fer21], a delimitation and a discussion articulating it with the (fuzzy) key concept of multimodality is required. This research is focused on tangible objects of Cultural Heritage and question the specific role of imaging techniques for Heritage Science [Str18]. The Digital Heritage framework peculiarly accepts a twofold definition of data ; it could refer to “data as Heritage” and at the same time “data about Heritage”. The whole data lifestyle is analyzed by the prism of the multimodality, which is the core of the study. A proposal is made to comprehend the multimodal imaging in CH domain as the stratification of several layers, including sensors, scales, spectra, temporalities and actors fostering multidisciplinary studies (see Fig. 1). Several case studies and experimental works are used to demonstrate the problem raised by current multimodal digitization practices, whose require renewing conventional and simplistic workflows; acquisition, processing and interpretation. The statement is made that multimodality is increasing the entropy of the digital-based CH documentation system leading nowadays to what is called a *Digital Bulk* (i.e., a sparse and unstructured doc-

umentation framework reducing its potential as an informational source) [Cha21]. A suggestion is made to update the methodology of digitization practices, with new components aligned with the arising challenges of Spatialization, Fusion and Enrichment, after defined in section 2.

<figure><img src="image_2.png" /><figcaption>**Figure 1: Diagram of multidisciplinarity in Digital Heritage.**</figcaption></figure>
## 1.2. Context of the research

The research presented is this article is a longitudinal study focusing on the construction of multimodal 2D/3D CH imaging strategies, substantiated by a series of experiments published between 2016 and 2024. Therefore, any deeper or complementary information is accessible within those cited resources (results, discussion, tools, limitations, in depth state of the art). An extended version of this work has been published in the form of a PhD dissertation [Pam24] (in French). This article is a digested and enriched version, aiming to summarize the main contributions developed and lessons learned from this doctoral journey within the form of an article-based PhD.

# 2. State of the art of CH oriented 2D/3D multimodal digitization system

## 2.1. Extended definition of multimodality for CH application

The first part of the state of the art is dedicated to a proposal for an extended definition of multimodality in the framework of CH studies. Multimodality is widely used in DH domain without a precise definition in terms of scope and application. Etymologically, this concept is inherited from Linguistic then Psychology before being used to describe interactions in Human Machine-Interaction domain (HMI). In this Informatics related domain, multimodality is entangled with fusion processes and semantic driven approaches. Indeed, a multimodal device or system is supposed to have a higher sense-making purpose than unimodal ones [NC93] (see Fig. 2). Multimodality is therefore defined as cooperation between different modalities or processes. That being said, it widens the comprehension of this concept for CH studies; hence the necessity of analyzing the whole documentation system and its components through the prism of multimodality.

In CH imaging and HS instrumentation, multimodal in composed of multiple modal layers, as follows :

**Multi-sensor** is the use of different types of sensors during an

<figure><img src="image_3.png" /><figcaption>**Figure 2:** Diagram of a multimodal system from an HMI point of view, derived from Nigay and Coutaz definition.</figcaption></figure>

imaging campaign, whether active (e.g., laser), passive (e.g., photography), or hybrid. Beyond the technological differences in sensing and measurements, this also implies variability in resolution, precision, and uncertainty, and may affect the format and digital encoding of the data in terms of volume and complexity. Multisensor approaches generally imply scattered and heterogeneous data [SDRDM*13, Cer22].

**Multi-scale** refers to the management of dimensional variations inherent to the native resolution of sensors or acquisition modalities, as well as the scales of representation or observation enabled by each acquisition technique. In CH and HS applications, scales typically range from millimeter to meter, defined as mesoscopic (observable without instrumentation), but may also extend toward microscopic (down to nanometers) or macroscopic (beyond the kilometer). Multiscalar approaches could be zoom-in or zoom-out and lead to the management of sparse data in terms of dimensionality [KCC17].

**Multi-spectral** involves the integration of radiometric data captured beyond the visible spectrum. This enables the analysis of phenomena in either multispectral contexts (discrete spectral bands) or hyperspectral contexts (continuous spectral coverage). In HS, imaging typically spans from UV-A to IR-A, with extensions possible toward X-ray radiography or terahertz spectroscopy, depending on the technology. Multi-spectral approaches has impact on data visualization and structuration for enabling multi-band exploration [SCMMB13, WFB18].

**Multi-temporal** relates to the inclusion of multiple temporalities in the data acquisition process, through repetition, completion, or consolidation, to monitor changes or deterioration over time. In geospatial fields like remote sensing or astrophysics, the term multi-epoch is often preferred. Diachronic analysis of temporal layers varies with the heritage object, which is subject to either natural or artificial ageing, requiring the management of past, contemporary, and future data and resources [GRY*19].

These aspects stem from technical and material considerations, which often lead to them being grouped under terms such as multi-layered, multi-overlay, or multi-faceted applications. Additionally, the interdisciplinary nature of DH studies requires taking into account an additional layer :

**Multi-actor** arise from multidisciplinary practices, this aspect involves acknowledging the inherently collaborative nature of studies conducted on heritage objects—especially when they involve digital tools, platforms, or technologies. It implies a dialogue and synergy between diverse areas of expertise, often from fields that differ significantly in terms of technical approaches, methodologies, and terminologies, especially in open-science and open-data prospects [DH21]. This creates an opportunity to gather knowledge, expertise, and observations around a single object or an entire collection.
The concatenation of the proposed definition of layered-multimodality above defined implies a major change in CH context. Multimodality could be employed for any approaches fostering the cooperation between sensings or measurements, dimensions (geometric and colorimetric scales, times) and expertises all along the data cycle, while multimodal fusion remains an underestimated open challenge [LAJ15]. More importantly, multimodality is not only a data-acquisition matter, but a condition impacting multiple CH practices and digital documentation as a whole (see Fig. 3).

<figure><img src="image_4.png" /><figcaption>**Figure 3:** *Diagram of the multimodal layers in CH and HS imaging.*</figcaption></figure>

## 2.2. From acquisition to spatialization

A detailed literature study has been conducted to analyze multi-modal layers of protocols, instruments, tools, and other spatialization systems in CH imaging. It reveals that current approaches are rather bimodal than truly multimodal. Indeed, two modalities are in the better case cooperating, but most of the time one is just used to enrich the other, without reciprocity. It means that the cooperation is often unilateral or has a directionality [AR21]. Another outcome concerns the intentionality, unanticipated multimodality is perceived as source of issues while intentional multimodal imaging allows extending the cooperation between the modalities. The instrumentation techniques and protocols for real-based modeling or measuring in CH and HS are numerous and fast-evolving (see Fig. 4). The multimodality increase exponentially the cardinality between various possible combinations while place at the center the spatialized resources, and more precisely the anchorage offered by 3D digital models. The spatialization of data, information and knowledge being the core feature and challenge of the so-called Heritage Digital Twin (HDT) [JH20, NMT*23] which slightly differs from other Digital Twin accepted meaning out of our application domain [JSN*20, VM21, CHB*23]. The spatialization is therefore defined by the ways and the means enabling to reference sensors, data, information or semantic resources in spatio-temporal framework.

<figure><img src="image_5.png" /><figcaption>**Figure 4:** *Taxonomy of spatial instrumentation in CH/HS application (non-exhaustive, only the ones encompass during the study at the laboratory level) within MEMORIA IS.*</figcaption></figure>

## 2.3. From processing to data fusion

The study of literature from variable domains unveils an underestimated issue when sliding the cursor from data processing to dedicated fusion procedures [RR15], even more when dealing with non-homogenous multi-source data (covering most of DH scenarios). It also remains a terminological imprecision, requiring to remind that Data Registration cannot be assimilated to Data Fusion (DF). The literature in DF highlights a list of potential issues to evaluate the Degree of Confidence (DoC) from data input and the Measure or Performance (MoP). As a matter of fact, most of sub-optimal features of data for fusion purpose are characteristic of CH data (lack of ground truth, heterogeneity, incompleteness, sparsity, uncertainty, etc). In addition, multimodality could be tackled by many levels (raw sources, feature dimensionality, purposes) at various stage of the data cycle (from early to late stage) [KKKR13]. To this end, a taxonomic description in proposed in Fig. 5. In DF domain, even ad-hoc methods are unwisely efficient [HS01]. Finally, an open-challenge is related to the fusion of soft data (generated by human operator, mostly qualitative and possibly subjective) versus hard data (generated by physical sensor, mostly quantitative and mostly objective) [KKKR13, HMLM08].

In this research, we understand DF, as "the study of effective methods for transforming variable sources into useful representations in order to enhance their significance." (definition derived from [BAB*05, Wal02]).

<figure><img src="image_6.png" /><figcaption>**Figure 5:** *Taxonomy of Data Fusion applied to CH digitization within MEMORIA IS.*</figcaption></figure>
## 2.4. From interpretation to enrichment

The methods and framework for interpreting and enriching her-
itage data were discussed in-depth through up-to-date bibliographic
prospects [AR19]. It browses a wide and shaded panel of ap-
proaches from top-down, craft or manual and human driven anal-
ysis to massive and almost fully automated IA driven ecosystems
considered as bottom-up that encompass various aspects like meta-
data schema, semantic annotation or mapping and ontological ap-
proaches [SK15] (see Fig. 6). Despite the fragmented practices,
it testifies the richness and the vitality of CH as application do-
main from numerous research field. At the same time the scattered
approaches are the main obstacle for stabilizing, - if ever possi-
ble – a unified framework [SGSH23]. The variability of CH ori-
ented studies encompass in depth case-study experiments, nowa-
days facing scalability and interoperability issue to be deployed as
it on broader corpus or collection. Current methods are divided be-
tween upstream massive AI driven approaches and a data prove-
nance weaknesses downstream [DB23], both leading to expend the
data lineage gap. Nowadays, the massive digitization trends are not
opposed but possibly misaligned with the open-science challenges
(LoD, FAIR, JUST) [HR18]. Indeed, the current practices are more
representative of digital bulk and infobesity concepts than digital
pragmatism and sobriety.

For this study, the most suitable definition of Data enrichment is
as a "general term that refers to the processes used to strengthen,
refine, or improve raw data in various ways".

<figure><img src="image_7.png" /><figcaption>**Figure 6:** Overview of the enrichment and semantic driven work-
flow in DH practices. In red, exponential learning curve and cost
with using bottom-up approach. In green, opposed top-down ap-
proach advocating early enrichment.</figcaption></figure>

3. Research aims toward multimodal approaches

On the methodological side, current scattered and unstructured
practices lead to an entropic effect that could not be solved by a
generic and dogmatic solution. Instead, one has to consider, analyze
and comprehend the digitization in CH studies a part of a complex
system (from a systemic point of view [Cam08]). Systemic splitting
is used to decompose a complex mechanism into manageable sub-
systems and sublevel issues. This systemic approach is completed

by a strategic one, aiming to review and potentially renew the co-
ordination of key actions to construct meaningful digital models.
However, developing theoretical proposals with high levels of ab-
straction is usually misaligned with practical and real-case appli-
cations (see Fig. 7). For this purpose, two operative approaches are
derived from two concepts borrowed in the field of ergonomics (i.e.,
the scientific study of people and their working conditions) from
the study of Pierre Rabardel's research [Rab95, RB03, Bou07].

The first one is called instrumental and proposes to focus on the
use of tools and techniques to solve a specific problem. It leads to
the concept of “pivot instrument” understood as an intermediary
tool helping and improving the interaction between the other parts
of the system and supposedly increase our capacity for adaptation
and learning. Pivot instruments also have a transitional role to facil-
itate the development of cognitive schemas and skills of a user. The
second approach concentrates on methods and specific steps used
in problem-solving regardless of the tools and is therefore called
as procedural. In Ergonomics, a “schema” is a cognitive structure
of coordinating task management using tools in a specific work-
ing environment. Schemata (plural form) represents dynamic and
evolving mental frameworks forged by experience in any activity.
It is similar to a workflow with a higher abstraction level. For exam-
ple, the schema of taking pictures is applied differently to scientific
imaging, photographic art or tourism.

The combination of instrumental and procedural approaches is
used to operate targeted contributions, and the concepts are de-
veloped to raise our understanding of CH studies based on mul-
timodal imaging. Ideally, the aim is to develop on top of that a syn-
ergic approach taking advantage of them all in a virtuous circle to
build an improved multimodal documentation framework in Her-
itage Science. A graphical abstract is proposed plan announcement
and methodological outline of the research conducted and exposed
in this article (see Fig. 8).

<figure><img src="image_8.png" /><figcaption>**Figure 7:** Graphical representation of the current workflow and practice to handle multimodal imaging in CH field.</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>**Figure 8:** Graphical abstract and methodological structure of the research. The colors symbolize the systemic approach. The arrows depict the strategic approach and the actions required toward a cyclic system. The instrumental (A1-B1-C1) and procedural approaches (A2-B2-C2 in bold) are located for each key interactions. The synergic approach is the synthesis of all.</figcaption></figure>

## 4. Challenge of Spatialization

This section set the first part of the contribution and is devoted to the Spatialization topic. It is based on several photogrammetric-based approaches trying to deploy multimodal spatialization strategies.

The first experiments A1 develops the instrumental approach and refers to a couple of articles [PGVDL16, PGF*17] dealing with CH oriented multimodal photogrammetry experiments. The first one exposes the methodological approach and the prerequisites for implementing data acquisition protocols in multi-scale, multi-temporal, multi-spectral and multi-sensor scenario. The experiments presented settle photogrammetric technique as a candidate for a pivot instrument in CH documentation system. The second experiment pursues this idea, proposing an instrumental set-up for generalizing and expanding the benefits of multimodal photogrammetry for large scale or massive digitization purpose [CPRDL24]. A robotized rig for multispectral capture of wall painting (or any quasi-planar vertical surfaces) is used to generate high-resolution multiband orthophotomosaics. These initial works enable to distinguish two divergent schemata of practice. One by extension, in which multimodality is problematic because a consequence of un-structured documentation strategies. Another one, by intention in which multimodality is anticipated and has the potential to enrich the documentation.

The second experiment develops the procedural approach A2 aiming to spatialize complex and massive multimodal photogrammetric datasets [PMDLV20]. Current image-based modelling practices are no longer one shot and linear application but rather a more or less structured surveys aggregated overtime, and leading to a photogrammetric bulk. This issue is addressed by developing

an incremental photogrammetric registration routine to support a wide range multimodal scenario. It takes the form of an algorithm called Totally Automated Co-registration and Orientation of image sets (TACO). It uses the open-source package (MicMac) to enhance the accuracy and the robustness of photogrammetric based spatialization and reconstruction from multi-source acquisition. The fully automated mode of the algorithm has been implemented as the photogrammetric engine of AIOLI [AMP*23], into a web service devoted to collaborative 2D/3D semantic annotation.

This first part of the contributions exposes potential issues in current image-based strategies (see Fig. 9). It shows how the pivotal image-based modelling could reinforce the multimodal data continuum while encompassing a wide variety of CH digitization approaches and how the schemata of capture impacts the whole operative chain.

## 5. Challenge of Fusion

This second section devoted to contribution is focused on the topic of Data Fusion widely underrated in CH and HS domains. Through a serie of experiments based on genuine CH oriented practices some fusion approaches are presented.

A first experiment states as instrumental approach B1; it describes an attempt to combine two of the most used image-based modelling in CH applications, namely RTI (multi-light) and photogrammetry (multi-view) by the means of a multimodal visualization system [PPA*19]. Those two techniques are complementary on the purpose, one modelling the appearance better, while the other models the geometry. But they are antagonists regarding their capture schemata, in which Digital Cameras act as pivot instruments. Learnings for the past section 4 are applied (ad hoc capture protocol, photogrammetric based spatial registration with TACO) until a 3D scene containing both imaging modalities is set and ready for a spatial semantic annotation process through AIOLI the web-platform. It enables to transfer an observation from one modality to the other, or reversely and therefore revealing the potential for cooperation between modalities. This experiment is also used to discuss the limit of an incomplete data fusion method (i.e., strictly different to data registration or visual overlays) and the necessity to consider also the *soft data* understood as qualitative and use-defined datum (i.e., opposed *hard data* as purely objective or sensor-based datum).

The second experiment B2 aims to tackle the issue of multi-source 3D reconstruction to reinforce the dialogue between imaging modalities using a procedural approach [PAfV23]. The case study is a chapel documented through multimodal acquisition campaigns for the past decade, ending with a 3D model containing 250 millions of spatial coordinates compiled from 28 difference sources. It uses the raw output of Real Based Modelling (e.g., the 3D point cloud) to generate a fusion index revealing as a heat map the density and the redundancy of information contained in multi-source 3D models. The computation is powered by Octree structure, geometric descriptors (e.g., neighbouring based density) and the algebraic features of CloudCompare to compute this Multimodal Enhancement Fusion Index (MEFI). The procedure is given as a pseudocode and formula for replication purposes and is complete with an experimental viewer enabling to retrieve all intrinsic
<figure><img src="image_10.png" /><figcaption>**Figure 9:** Case-study of multimodal imaging in HS (left) co-registered with A2 (TACO). Diagram of multimodal spatialization practices (right) and potential issues: (a) compatibility or redundancy with previous data, (b) obsolescence or updating of processing procedures, (c) the complexity of registration and fusion methods (d) accumulation and disparity of results.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>**Figure 10:** Advanced exploration of 3D point cloud (hard-data) enhanced with Multimodal Fusion Index resulting of B2 (MEFI) correlated with metadata description (soft-data).</figcaption></figure>
documenting (i.e., soft and hard metadata) each source to better
comprehend the multimodal layers of the 3D model (see Fig. 10).

This second part of contributions discusses the role of visual-
ization tools and modelling paradigms to enhance representation
and the comprehension of complex and multimodal datasets. At the
methodological level, the existence of systems of pivot instruments
has been revealed, embedding visualization and annotation pro-
cesses within RBM documentation framework. At the same time,
it also underlines our current difficulty in capturing and tracing the
schemata of multimodal imaging practices (i.e., the intention or in-
tuition motivating the cooperation between several modalities).

6. Challenge of Enrichment

This last section of development is dedicated to the method of data
enrichment, understood as the various ways to augment the seman-
tic value of digitized heritage, as a whole. Beyond the bottom-up
(from data to semantic exploration) or top-down (semantic knowl-
edge fitted to data) approaches explored in the state of the art, a
crucial gap remains in tracking the operative tool chain. Nowadays,
data enrichment is considered only in the late stage of the digiti-
zation framework (ingestion, sharing, archival or harvest), while
workflows and research protocols remain mainly undocumented
since the capture and all along the processes.

In a first experiment C1, propose an instrumental approach C1
aiming to construct semantically enriched digital representation for
conservation and restoration purposes [PRH*22]. A global frame-
work presents how tailored digitization strategies can lead to 3D
digital models enriched by semantic spatialized annotations. De-
spite the added value of condition reports linked to fused multi-
modal imaging fully accessible through 3D exploration, the digiti-
zation schemata remain latent and raising traceability issues.

The second experiment C2 introduces a Metadata Enriched Mul-
timodal dOcumentation System (MEMoS) to contribute to the issue
of data lineage concerning instrumental data [PLV22]. It is based
on the W7 conceptual model used in various fields to reinforce data
provenance [LR17]. Instead of altering metadata schemes (or im-
posing a new one), this procedural approach invites adopting a ver-
satile metadata structure. The W7 enables integrating useful de-
scriptions of any imaging modality into explicit sections (WHAT,
WHO, WHEN, WHERE, HOW, WHICH, WHY) with prior onto-
logical mappings. The MEMoS system mimics the modus operandi
of the cinematographic clapping board to keep trace of the capture
schema intended to be useful for the following stages. The system
works in three steps:

* Describe the modality using user-friendly formats (JSON, YAML, CSV).

* Encode the metadata into a 2D barcode (QR code, JAB) or into a unique and perennial identifier (UUID, URI, DOI).

* Expose the metadata code and or its identification onto the digitization scene or into digital resources, consequently leading to better exposure level from data-science meaning.

This method creates a solid anchorage of data, metadata [Fur20]
and paradata [BCD*21], which has been tested in various scenar-
ios, from one shot digitization, to massive collection management

and complex multimodal digitization campaigns (e.g., Notre Dame
de Paris scientific worksite [GAR*25]). The MEMoS experiment
continues today with the development of an open hybrid system in-
cluding software, hardware and semantic components (see Fig. 11).
It already demonstrates two major advances for CH-oriented digital
resource long-term management. Firstly, enabling automatic data
ingestion in an Open Repository (Data Lake or Warehouse) without
struggling with time-consuming and tedious metadata retrieval and
completion. Secondly, generating from the capture web-semantic
friendly or compliant data leverage tedious data curation.

This last third of contributions unveils the possibility and ne-
cessity of early-stage data enrichment and provides a lightweight
and user-friendly solution for Heritage scientists. The approaches
described revalorize the act of survey, which tends to be under
evaluated nowadays, and emphasize its role in Heritage data
lifecycle.

<figure><img src="image_12.png" /><figcaption>Figure 11: Overview of the MEMoS based documentation frame-
work currently implemented in a web-platform and IoT device
within ANAMNESIS project.</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>**Figure 12:** *Graphical synthesis of the multimodal imaging strategies designed and implemented in this study.*</figcaption></figure>

## 7. Toward intermodality

This research introduces a new and extended definition of multimodality involving imaging and digital documentation for CH and HS applications. It proposes a comprehensive and critical overview of CH imaging system, where multimodality stands nowadays as a paradigm shift. A renewed digitization methodology has been developed leveraging current challenges by addressing new issues like Spatialization, Fusion and Enrichment in cooperation with conventional workflows sequenced from acquisition, to processing and interpretation. Several experiments were conducted to verify the hypotheses and efficiency of our so-called instrumental and procedural approaches. The demonstration is illustrated by various case studies showing their relevancy on a wide spectrum of current CH practices. The proposed overall strategy enhances the FAIR readiness of data from their genesis. In addition, it is worth mentioning that open-science principles were applied all along the experiments conducted. Resources like publications, source-code, data are all available in open-access (cf. Appendix of [Pam24]). This research helps to reveal photogrammetry (in digitization-driven practices only) as a pivot instrument and the image-based modelling framework as a system of pivot instruments encompassing advanced tools such as multimodal visualization and multi-support annotations. It opens up the possibility of finding out other pivot instruments and systems to raise the performance of CH studies. Some schemata could emerge from ad-hoc multimodal strategies if the CH/HS scientific community raises its capacity for data provenance management and data linage through documented protocols and workflows. This works remains limited to a panel of imaging in the field of 2D/3D real-based digitization even though the extension to other modalities, including perceptive ones (e.g. acoustic measurements [BDP*22]), has been experimented.

This research concludes by following the idea that scattered in-

terdisciplinary practices of CH and HS studies are not only a testimony of an emerging and fruitful research field, neither a problem but rather a part of the solution if solutions are proposed to embrace this diversity. However, this prospect of learning and transferring experiences from current digital studies is only feasible by strengthening our capabilities in three key topics: data spatialization, information fusion, and knowledge enrichment through a streamlined operational chain that anticipates the data lifecycle (cf 12). This studies also revealed the key role of instrumentation, visualization and annotation frameworks in up-to-date CH digital ecosystem. This study demonstrates that a sort of collective and distributed cognitive gain could be achievable at minimal cost by interacting at key moments of the digitization process and our mainstream workflows. In this manner, the digital cultural object itself could be mutilated into a frontier object (in the definition of Leigh Star [LS10]) with a potential impact above disciplinary frontiers and territories. If multimodality is a trigger to multidisciplinary practices, strategic approaches of imaging in CH fields could lead to intermodal prospects fostering interdisciplinary research [VdBH*01].

## Acknowledgements

The main author would like to deeply acknowledge all its co-authors participating in the experiments cited in this present article for their precious and esteemed contributions.

## Roles and contributions

Contributors: A. Pamart (AP), L. De Luca (LDL), P. Veron (PV). CRediT (https://credit.niso.org/): Conceptualization: AP; Data curation: AP; Formal analysis: AP; Funding acquisition: LDL; Investigation: AP; Methodology: AP, LDL, PV; Project administration: LDL; Resources: AP; Software: AP; Supervision: LDL, PV; Validation: AP; Visualization: AP; Writing – original draft: AP; Writing – review & editing: AP.
## References

[AGS18] ALBUERNE A., GRAU-BOVE J., STRLIC M.: The Role of Heritage Data Science in Digital Heritage. In *Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection*, vol. 11196. Springer International Publishing, Cham, 2018, pp. 616–622. doi:10.1007/978-3-030-01762-0-54. 1

[AMP*23] ABERGEL V., MANUEL A., PAMART A., CAO I., DE LUCA L.: Aioli: A reality-based 3D annotation cloud platform for the collaborative documentation of cultural heritage artefacts. e00285. URL: https://linkinghub. elsevier.com/retrieve/pii/S2212054823000309, doi:10.1016/j.daach.2023.e00285. 5

[AR19] ADAMOPOULOS E., RINAUDO F.: 3D Interpretation and Fusion of Multidisciplinary Data for Heritage Science: A Review. *ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLII-2/W15* (Aug. 2019), 17–24. doi:10.5194/isprs-archives-XLII-2-W15-17-2019. 4

[AR21] ADAMOPOULOS E., RINAUDO F.: Close-Range Sensing and Data Fusion for Built Heritage Inspection and Monitoring—A Review. *Remote Sensing* 13, 19 (Oct. 2021), 3936. doi:10.3390/rs13193936. 3

[BAB*05] BOSTRÓM H., ANDLER S. F., BROHEDE M., JOHANSSON R., KARLSSON A., VAN LAERE J., NIKLASSON L., NILSSON M., PERSSON A., ZIEMKE T.: On the Definition of Information Fusion as a Field of Research. 8. 3

[BCD*21] BRUNEL V., COLIN P., DANCIU A., GALLIS Q., GROSHENS E., MARIE L., MICHAUD G., ROUXEL B., VILLETTE T.: Des méta-données et paradonnées pour mieux exploiter des données d'enquêtes pour la recherche. In *Mate la Science ouverte* (2021). 7

[BDP*22] BLAISE J.-Y., DUDEK I., PAMART A., BERGEROT L., VIDAL A., FARGEOT S., ARAMAKI M., YSTAD S., KRONLAND-MARTINET R.: Acquisition & integration of spatial and acoustic features: A workflow tailored to small-scale heritage architecture. pp. 1. URL: https://shs.hal.science/halshs-03483210, doi: 10/document. 8

[Bou07] BOURMAUD G.: L'organisation systémique des instruments : méthodes d'analyse, propriétés et perspectives de conception ouvertes. In *Colloque de l'Association pour la Recherche Cognitive - ARCo'07 : Cognition - Complexité - Collectif* (Nov. 2007), p. 61. 4

[Cam08] CAMBIEN A.: Une introduction à l'approche systémique : appréhender la complexité. report, Centre d'études sur les réseaux, les transports, l'urbanisme et les constructions publiques (CERTU), 2008. 4

[Cer22] CERA V.: Multisensor Data Fusion for Culture Heritage Assets Monitoring and Preventive Conservation. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLVI-2-W1-2022* (Feb. 2022), 151–157. doi:10.5194/isprs-archives-XLVI-2-W1-2022-151-2022. 2

[Cha21] CHABIN M.-A.: La recherche en archivistique à l'épreuve de la société numérique. *Archives* 49, 1-2 (Mar. 2021), 141–160. doi: 10.7202/1075710ar. 1

[CHB*23] CHARRIER Q., HAKAM N., BENFRIHA K., MEYRUEIS V., LIOTARD C., AOUSSAT A.: A roadmap for monitoring cyber-physical systems through the digital twin using an iot sensing architecture. *Available at SSRN* 4406361 (2023). 3

[CPRDL24] COMTE F., PAMART A., RÉBY K., DE LUCA L.: Strategies and Experiments for Massive 3D Digitalization of the Remains After the Notre Dame de Paris' Fire. 127–134. URL: https://isprs-archives.copernicus.org/articles/XLVIII-2-W4-2024/127/2024/, doi:10.5194/isprs-archives-XLVIII-2-W4-2024-127-2024. 5

[DB23] DUDEK I., BLAISE J.-Y.: Research workflows, paradata, and information visualisation: feedback on an exploratory integration of issues and practices - MEMORIA IS. *Peer Community In Archaeology* (2023). URL: https://shs.hal.science/halshs-04091200, doi: 10.5281/zenodo.8311129. 4

[DH21] DAVIS E., HERAVI B.: Linked Data and Cultural Heritage: A Systematic Review of Participation, Collaboration, and Motivation. *Journal on Computing and Cultural Heritage 14*, 2 (June 2021), 1–18. doi:10.1145/3429458. 2

[Fer21] FERRAZZI S.: The Notion of “Cultural Heritage” in the International Field: Behind Origin and Evolution of a Concept. *International Journal for the Semiotics of Law - Revue internationale de Sémiotique juridique* 34, 3 (July 2021), 743–768. doi:10.1007/s11196-020-09739-0. 1

[Fur20] FURNER J.: Definitions of “Metadata”: A Brief Survey of International Standards. *Journal of the Association for Information Science and Technology* 71, 6 (2020), E33–E42. doi:10.1002/asi.24295. 7

[GAR*25] GUILLEM A., ABERGEL V., ROUSSEL R., COMTE F., PAMART A., DE LUCA L.: A bridging the provenance knowledge gap between 3d digitization and semantic interpretation. doi:InPress. 7

[GRY*19] GHAMISI P., RASTI B., YOKOYA N., WANG Q., HOFLE B., BRUZZONE L., BOVOLO F., CHI M., ANDERS K., GLOAGUEN R., ATKINSON P. M., BENEDIKTSSON J. A.: Multisource and Multitemporal Data Fusion in Remote Sensing: A Comprehensive Review of the State of the Art. *IEEE Geoscience and Remote Sensing Magazine* 7, 1 (Mar. 2019), 6–39. doi:10.1109/MGRS.2018.2890023. 2

[HMLM08] HALL D. L., MCNEESE M., LLINAS J., MULLEN T.: A framework for dynamic hard/soft fusion. In *2008 11th International Conference on Information Fusion* (2008), IEEE, pp. 1–8. 3

[HR18] HASNAIN A., REBHOLZ-SCHUHMAN D.: Assessing FAIR Data Principles Against the 5-Star Open Data Principles: ESWC 2018 Satellite Events, Heraklion, Crete, Greece, June 3-7, 2018, Revised Selected Papers. Aug. 2018, pp. 469–477. doi:10.1007/978-3-319-98192-5_60. 4

[HS01] HALL D. L., STEINBERG A.: Dirty secrets in multisensor data fusion. *Handbook of Multisensor Data Fusion* (2001), 211–212. 3

[JH20] JOUAN P., HALLOT P.: Digital Twin: Research Framework to Support Preventive Conservation Policies. *ISPRS International Journal of Geo-Information* 9, 4 (Apr. 2020), 228. doi:10.3390/ijgi9040228. 3

[JSN*20] JONES D., SNIDER C., NASSEHI A., YON J., HICKS B.: Characterising the digital twin: A systematic literature review. *CIRP journal of manufacturing science and technology* 29 (2020), 36–52. 3

[KCC17] KAR B., CHOW E., CHOW E.: Fusion of Multiscaled Spatial and Temporal Data: Techniques and Issues. https://www.taylorfrancis.com/, Jan. 2017. doi:10.1201/9781315373720-14. 2

[KKKR13] KHALEGHI B., KHAMIS A., KARRAY F. O., RAZAVI S. N.: Multisensor data fusion: A review of the state-of-the-art. *Information Fusion 14*, 1 (Jan. 2013), 28–44. doi:10.1016/j.inffus.2011.08.001. 3

[LAJ15] LAHAT D., ADALI T., JUTTEN C.: Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects. *Proceedings of the IEEE 103*, 9 (Sept. 2015), 1449–1477. doi:10.1109/JPROC. 2015.2460697. 3

[LR17] LIU J., RAM S.: Improving the Domain Independence of Data Provenance Ontologies: A Demonstration Using Conceptual Graphs and the W7 Model. *Journal of Database Management* 28, 1 (Jan. 2017), 43–62. doi:10.4018/JDM.2017010104. 7

[LS10] LEIGH STAR S.: This is Not a Boundary Object: Reflections on the Origin of a Concept. *Science, Technology, & Human Values* 35, 5 (Sept. 2010), 601–617. doi:10.1177/0162243910377624. 8

[NC93] NIGAY L., COUTAZ J.: A design space for multimodal systems: Concurrent processing and data fusion. In *Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems* (New York, NY, USA, May 1993), CHI '93, Association for Computing Machinery, pp. 172–178. doi:10.1145/1690509.169143. 2
[NMT*23] NICCOLUCCI F., MARKHOFF B., THEODORIDOU M., FE-LICETTI A., HERMON S.: The Heritage Digital Twin: A bicycle made for two. The integration of digital methodologies into cultural heritage research. *arXiv preprint arXiv:2302.07138 (2023). doi:2302.07138.3*

[PAfV23] PAMART A., ABERGEL V., FAMILY=LUCA GIVEN=LIVIO P. U., VERON P.: Toward a Data Fusion Index for the Assessment and Enhancement of 3D Multimodal Reconstruction of Built Cultural Heritage. 2408. URL: https://www.mdpi.com/2072-4292/15/9/2408, doi:10.3390/rs15092408.5*

[Pam24] PAMART A.: Spatialisation, fusion et enrichissement d'imagerie multimodale pour les sciences du patrimoine, 2024. Ph.D. thesis, HESAM University 2024, Paris. URL: http://www.theses.fr/2024HESAE001.2, 8*

[PGF*17] PAMART A., GUILLON O., FARACI S., GATTET E., GENEVOIS M., VALLET J.-M., DE LUCA L.: Multispectral photogrammetric data acquisition and processing for wall paintings studies. In *ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences* (2017), vol. XLII-2-W3, Copernicus GmbH, pp. 559-566. URL: https://hal.archives-ouvertes.fr/hal-01480562, doi:10.5194/isprs-archives-XLII-2-W3-559-2017.5*

[PGVDL16] PAMART A., GUILLON O., VALLET J.-M., DE LUCA L.: Toward a Multimodal Photogrammetric Acquisition and Processing Methodology for Monitoring Conservation and Restoration Studies. In *Proceedings of the 14th Eurographics Workshop on Graphics and Cultural Heritage* (2016), GCH '16, Eurographics Association, pp. 207-210. URL: https://doi.org/10.2312/gch.20161409, doi:10.2312/gch.20161409.5*

[PLV22] PAMART A., LUCA L. D., VÉRON P.: A metadata enriched system for the documentation of multi-modal digital imaging surveys. 1-24. URL: https://scholarworks.iu.edu/journals/index.php/sdh/article/view/33767, doi:10.14434/sdh.v6i1.33767.7*

[PMDLV20] PAMART A., MORLET F., DE LUCA L., VERON P.: A Robust and Versatile Pipeline for Automatic Photogrammetric-Based Registration of Multimodal Cultural Heritage Documentation. 2051. URL: https://www.mdpi.com/2072-4292/12/12/2051, doi:10.3390/rs12122051.5*

[PPA*19] PAMART A., PONCHIO F., ABERGEL V., ALAQUI M'DARHRI A., CORSINI M., DELLEPIANE M., MORLET F., SCOPIGNO R., LUCA L. D.: A Complete Framework Operating Spatially-Oriented RTI in a 3D/2D Cultural Heritage Documentation and Analysis Tool. In *ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences* (2019), vol. XLII-2-W9, Copernicus GmbH, pp. 573-580. URL: https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLII-2-W9/573/2019/, doi:10.5194/isprs-archives-XLII-2-W9-573-2019.5*

[PRH*22] PAMART A., ROUSSEL R., HUBERT E., COLOMBINI A., SALERI R., MOUADDIB E. M., CASTRO Y., LE GOÏC G., MANSOURI A.: A Semantically Enriched Multimodal Imaging Approach Dedicated to Conservation and Restoration Studies. 415-420. URL: https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLVI-2-W1-2022/415/2022/, doi:10.5194/isprs-archives-XLVI-2-W1-2022-415-2022.7*

[Rab95] RABARDEL P.: *Les hommes et les technologies; approche cognitive des instruments contemporains*. Armand Colin, 1995. 4*

[RB03] RABARDEL P., BOURMAUD G.: From computer to instrument system: A developmental perspective. *Interacting with Computers* 15, 5 (Oct. 2003), 665-691. doi:10.1016/S0953-5438(03)00058-4.4*

[RR15] RAMOS M. M., REMONDINO F.: Data fusion in Cultural Heritage - A Review. *ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences* XL-5/W7 (Aug. 2015), 359-363. doi:10.5194/isprs-archives-XL-5-W7-359-2015.3*

[SCMMB13] SIMON CHANE C., MANSOURI A., MARZANI F. S., BOOCHS F.: Integration of 3D and multispectral data for cultural heritage applications: Survey and perspectives. *Image and Vision Computing* 31, 1 (Jan. 2013), 91-102. doi:10.1016/j.imavis.2012.10.006.2*

[SDRDM*13] STAL C., DE ROO B., DE MAEYER P., NUTTENS T., DE WULF A.: Considerations on the fusion of multi-sensor spatial data for cultural heritage. In *Geographic Information Science, 16th AGILE Conference, Proceedings* (2013). 2*

[SGSH23] STOREIDE M. S. B., GEORGE S., SOLE A., HARDEBERG J. Y.: Standardization of digitized heritage: A review of implementations of 3D in cultural heritage. *Heritage Science* 11, 1 (Nov. 2023), 249. doi:10.1186/s40494-023-01079-z.4*

[SK15] STEINER E., KOCH C.: A Digital Archive of Cultural Heritage Objects: Standardized Metadata and Annotation Categories. *New Review of Information Networking* 20, 1-2 (July 2015), 255-260. doi:10.1080/13614576.2015.1112171.4*

[Str18] STRLIČ M.: Heritage Science: A Future-Oriented Cross-Disciplinary Field. *Angewandte Chemie International Edition* 57, 25 (2018), 7260-7261. doi:10.1002/anie.201804246.1*

[VdBH*01] VAN DEN BESSELAAR P., HEIMERIKS G., ET AL.: Disciplinary, multidisciplinary, interdisciplinary: Concepts and indicators. In *ISSI* (2001), University of New South Wales, pp. 705-716. 8*

[VM21] VANDERHORN E., MAHADEVAN S.: Digital twin: Generalization, characterization and implementation. *Decision support systems* 145 (2021), 113524. 3*

[Wal02] WALD L.: *Data Fusion: Definitions and Architectures; Fusion of Images of Different Spatial Resolutions*. Les Presses de l'École des Mines, Paris, 2002. 3*

[WFB18] WILSON M., FRANCE F., BOLSER C.: Multispectral Imaging for Scientific Analysis and Preservation of Cultural Heritage Materials. In *Archiving Conference* (2018), vol. 2018, Society for Imaging Science and Technology, pp. 147-150. 2*