ARCHERIX, Archaeology and Cultural Heritage
through XR

Selma Rizvic, Valérie Gouranton, Holger Graf, Théophane Nicolas, Ronan Gaugne

► To cite this version:

Selma Rizvic, Valérie Gouranton, Holger Graf, Théophane Nicolas, Ronan Gaugne. ARCHERIX, Archaeology and Cultural Heritage through XR. ARCHERIX 2025 - 1st IEEE VR Workshop on Archaeology and Cultural Heritage through XR, Mar 2025, Saint-Malo, France. pp.1-43, 2025. hal-05267712

HAL Id: hal-05267712

https://inria.hal.science/hal-05267712v1

Submitted on 22 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

IEEE VR

SAINT-MALO, FRANCE
March 8-12, 2025

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

ARCHERIX

IEEE VR

SAINT-MALO, FRANCE

# 1st Workshop on
Archaeology and Cultural Heritage through XR

March 8, 2025

Saint-Malo, France
# 1st IEEE VR workshop on Archaeology and Cultural Heritage through XR
ARCHERIX

Selma Rizvić*

Faculty of Electrical
Engineering, University of
Sarajevo
Sarajevo, Bosnia and
Herzegovina

Valérie Gouranton†

Univ Rennes, INSA Rennes,
Inria, CNRS, IRISA
Rennes, France

Holger Graf‡

Fraunhofer IGD
Darmstadt, Germany

Théophane Nicolas§

Inrap, UMR 8215
Trajectoires
Rennes, France

Ronan Gaugne¶

Univ Rennes, Inria, CNRS,
IRISA
Rennes, France

## ABSTRACT

The goal of this interdisciplinary workshop is to bring together in one place researchers specialized in XR and Cultural Heritage users such as archaeologists, restorers and curators, who until now have often expressed themselves collectively but separately through a number of scientific works. As each participant will contribute his or her own expertise, experience and point of view, the workshop should serve to strengthen cooperation and partnerships between disciplines.

## 1 ORGANIZERS

Selma Rizvić is a Professor of Computer Graphics at the Faculty of Electrical Engineering, University of Sarajevo. She has been teaching there Bachelor, Master, and Ph.D. study programs. In 2004 she founded the Laboratory for Computer Graphics - Sarajevo Graphics Group (SGG), a research group that specializes in the use of IT technologies for the presentation of cultural heritage. The SGG research group is multidisciplinary, it includes computer graphics experts and works with archaeologists, historians, visual artists, writers, and film professionals, in order to design and implement virtual cultural heritage applications that have educational values and are engaging, entertaining, and attractive at the same time. In 2010 they founded the Association for Cultural Heritage Digitization DIGI.BA.

Valérie Gouranton is a senior lecturer since 1999 and her research is attached to IRISA (Institut de Recherche en Informatique et Systèmes Aléatoires, UMR 6074) and Inria at the University of Rennes, while her teaching is attached to INSA (Institut National des Sciences Appliquées) Rennes. Her research focuses on extended reality (XR), and more specifically on models and tools for producing XR applications. Her main areas of interest are health, cultural heritage and the digital arts, but she is also interested in applications in construction and sport. She is in charge of the Arts, Culture and Heritage cross-disciplinary area at IRISA, the local co-ordinator for INSA Rennes of DemoES AIR (Augmenting Interactions in Rennes) and a member of its executive committee. She is a member of the executive committee of EquipEx+ Continuum, a collaborative research infrastructure of 30 platforms located throughout France, designed to advance interdisciplinary research between

computer science and the humanities and social sciences. She is in several Program Committees, such IEEE VR.

Holger Graf is head of the VR and AR department at the Fraunhofer Institute for Computer Graphics Research, Darmstadt Germany. He has a longstanding record in national and international EU-funded projects especially in the digital cultural heritage domain in which he was responsible for the administrative and scientific coordination of research on XR applied to Cultural Heritage, defining strategic agendas for the digitisation and the deployment of XR to the CH domain and applied XR solutions within the whole metaverse of the CH domain. He is co-organiser of several scientific events of applied computer graphics to cultural heritage such as the Eurographics (EG) Workshop on Computer Graphics for Cultural Heritage (GCH) series 2014, 2023, 2024 as well as the IEEE/EG World Congress on Digital Heritage in 2013, 2015 and 2025.

Théophane Nicolas is an archaeologist, ceramologist specialist in the Bronze Age of Western Europe at French national Institute for Preventive Archaeological Research (Inrap), in Trajectoires research laboratory, France. His research focuses on developing new approaches and tools for archaeology based on medical imaging (CT-scan) and Extended Realities.

Ronan Gaugne is a research engineer at University of Rennes since 2010, and his research is attached to IRISA (Institut de Recherche en Informatique et Systèmes Aléatoires, UMR 6074) and Inria at the University of RennesFrance. He received a PhD degree in computer science from Rennes University, France, in 1997. He is the technical director of the Immersia research platform in virtual reality, member of the French National Research Infrastructure Continuum. His research interests are applications of XR to art, culture and heritage.

## 2 WORKSHOP MANIFESTO

In the same way as 3D, innovative uses for XR are beginning to emerge in the field of historical sciences, on the initiative of archaeologists, anthropological biologists, paleopathologists and conservator-restorers. These technologies, which are constantly progressing, are opening up a new era in the knowledge, conservation and dissemination of archaeological and more generally cultural heritage data. Fundamentally interdisciplinary, they are revolutionizing practices and raising new concerns and issues. XR based methods provide powerful interactive and non-destructive access to fragile historical material and immediate information. Furthermore, new approaches that combine AI methods to XR might benefit the cultural heritage domain, providing high visual quality solutions that allow for digital restoration, prediction of future (de)generative natural processes, scientific exploration and communication to a broader audience. In addition, XR provides a powerful opportunity for the preservation and dissemination of intangible heritage. With the increasing development of virtual reconstruc-

<footnote>*e-mail: srizvic@etf.unsa.ba</footnote>

<footnote>†e-mail: valerie.gouranton@irisa.fr</footnote>

<footnote>‡e-mail: holger.graf@igd.fraunhofer.de</footnote>

<footnote>§theophane.nicolas@inrap.fr</footnote>

<footnote>¶ronan.gaugne@irisa.fr</footnote>
tions in the domain of Cultural Heritage, and thanks to advances in the production and 3D animation of virtual humans, immersive and interactive simulations have emerged to preserve human activity associated with traditional culture and folklore.

However, the production of heritage related environments in XR raises a number of questions of both a methodological and a technical nature. Indeed, the nature and construction of such interactive 3D digital environments combine real elements, or at least those obtained by digitizing the real thing, with interpreted elements. The latter may be linked to a previous complete environment to represent missing elements, or to data created by the user during virtual work sessions to enrich the environment with the help of tools offered to him. It is important to define a methodology for controlling and representing these different levels of data in the XR environment, and to assess the cognitive relationship between the expert user and these representations.

### 3 PROGRAM OVERVIEW

The ARCHERIX workshop is structured around two high-quality keynote presentations, two sessions of selected presentations, one titled "Methodology and innovative approach", and one titled "Case study", and one panel session.

The first keynote presentation, given by Sofia Pescarin, is entitled "Universal concepts as triggers in the design of VR/MR applications for cultural heritage", and focuses on the challenges and perspective of designing interactive and immersive applications for cultural heritage based on cognitive and emotional goals. Sofia Pescarin, PhD in History and Computing, is currently a Senior Researcher at CNR ISPC. Her research interests cover a wide domain that includes Virtual Archaeology, Digital Heritage, Virtual Museums, Embodied Museology, Applied Games, Interactive Media Design. She is Chief Editor since 2017 of the Elsevier Journal "Digital Application in Archaeology and Cultural Heritage" (DAACH). Since 2018, she is a professor at the University of Bologna, where she teaches Intangible Heritage and Multimedia. She is currently coordinating the european project PERCEIVE, WP3 of the PNRR project CHANGES / Spoke 4 (Virtual Technologies) and the Task protitpying platforms for virtual exhibitions within the project PNRR H2IOSC.

The second keynote presentation, given by Maurizio Forte, is entitled "Neurophysiology and human perception: VR-AI experiments in archaeology". Maurizio Forte, PhD, is William and Sue Gross Distinguished Professor of Classical Studies Art, Art History, and Visual Studies at Duke University. He was a pioneer in digital archaeology and the first archaeologist working in a Supercomputing Center (CINECA) in the 80-90s. He was recently awarded the Saint Francis Prize in Techno-Humanities 2024 for his work in the field of Digital Humanities. This Prize is to be presented to individuals or institutions in recognition of their outstanding contributions to the development of technologies in the humanities for the well-being of humankind.

The session "Methodology and innovative approach" gathers four papers, one research paper, one technical paper, and two position papers. These papers were selected for their exploration of heritage research that include XR and the paradigm shift that 3D brings to archaeologists: digitisation, hypothesis formulation, knowledge organization and information visualization, and interaction tools.

The session "Case study" gathers four papers, two research papers, one technical paper, and one "work-in-progress paper. They illustrate widely differing aspects of the integration of XR into research, dissemination, and mediation in archaeology and cultural heritage.

The closing panel session aims to create a valuable cross-community discussion about the opportunities for collaboration between XR and the archaeology and cultural heritage domains, the issues and challenges of this interdisciplinary collaboration, and the

potential for innovation brought by XR in relation to AI for archaeology and cultural heritage.

### 4 PROGRAM COMMITTE

An international program committee, made up of nine experts in the fields of XR, archaeology, and digital heritage, was responsible for reviewing the papers submitted:

* Maria Roussou, University of Athens, Greece

* Karina Rodriguez Echavarria, University of Brighton, United Kingdom

* Mai Otsuki, AIST, Japan

* Gabriele Guidi, Indiana University, USA

* François Giligny, Université Paris 1-Panthéon Sorbonne, France

* Bernd Froehlich, Bauhaus-Universität Weimar, Germany

* Nicolo Dell'Unto, Lund University, Sweden

* Marc Macé, CNRS, France

* Jean-Baptiste Barreau, CNRS, France

### ACKNOWLEDGMENTS

The workshop was funded in part by the CNRS MITI Digital Twin "AJAX" project, and by the 3DHN consortium of Huma-Num. The organizers would like to thank the ACM journal JOCCH for inviting the workshop's best paper for publication. Finally, the organizers would like to thank the two keynote speakers, the members of the program committee, and all the authors and participants in this first ARCHERIX workshop.
# Table of content

## Session Methodology and innovative approach

Immersive Autochrome: Enhanced Digitized Mono-Autochrome Viewing through Deep Learning and Virtual Reality 6

Embodied and Tangible Interaction in Collaborative Room-Scale Mixed Reality: Insights from a Visitor Experience in a Cultural Heritage Context 12

Cognitive Archaeology in Virtual Environment 19

For a Perception of Monumentality in Eastern Arabia from the Neolithic to the Bronze Age: 3D Reconstruction and Multidimensional Simulations of Monuments and Landscapes 23

## Session Case Study

Usability Testing of VR Reconstructions for Museums and Heritage Sites: A Case Study from 14th Century Chester (UK) 28

Exploring Written Artefacts in Virtual Reality -Potential and Challenges for Creating Immersive Tools and Applications 32

Virtual Immersion in Underground Quarries: An Innovative Exploration of Historical and Cultural Heritage 36

Gladiator training in XR 40
<figure><img src="image_6.png" /></figure>

Session Methodology
and innovative approach
# ImmersiveAutochrome: Enhanced Digitized Mono-Autochrome Viewing through Deep Learning and Virtual Reality

<figure><img src="image_7.png" /><figcaption>Figure 1: The pipeline utilizes deep learning techniques to transform a digitized mono-autochrome into an immersive 3D autochrome (layered depth image with corresponding mesh), enabling visualization through a Virtual Reality (VR) headset.</figcaption></figure>

## ABSTRACT

Resulting from the earliest impactful color photography process developed by the Lumière Brothers, autochromes are invaluable historical artifacts with a high sensitivity and susceptibility to degradation over time. We investigate the potential of deep learning to enhance the experience of historical autochromes through 3D photography and Virtual Reality (VR). Our proposed pipeline utilizes single-image depth estimation and depth-aware inpainting to transform digitized mono-autochromes into layered depth images and corresponding 3D meshes, enabling immersive 3D visualization via a WebXR application. Our user study shows the effectiveness of VR-based experience of autochromes compared to traditional visualization methods, revealing that participants found the VR experience as pleasant as analog exploration despite the latter's authenticity. Thereby, our findings suggest that 3D VR technology could play a crucial role in digitally preserving and revitalizing these culturally significant artifacts and foster investigations with more samples and more types of user interfaces to mature these insights.

**Index Terms:** 3D photography, single-image-based depth estimation, inpainting, virtual reality

## 1 INTRODUCTION

Color photography has become a vital aspect of contemporary society. The Lumière brothers' invention of the autochromes [12] is an early method of color photography from the beginning of the 20th century and served as an essential step in the development of

color photography. Launched commercially in 1907, autochromes are considered the first successful color photography process and maintained their dominance until the introduction of Kodachrome films in the 1930s, which also makes them represent historical artifacts and documents. Autochrome plates, coated with microscopic red, green, and violet/blue potato starch grains, create a full-color image of the original subject by allowing light to pass through these color filters to the photographic emulsion, which is then processed to produce a positive transparency. Autochromes were typically viewed privately by merely holding them against a light source. However, for a more enhanced viewing experience, they were typically placed on specially designed stands known as diascopes, which contained a mirror. This setup not only brightened the image but also facilitated simultaneous viewing by multiple people. For larger public exhibitions, magic lanterns were employed to project the autochromes. The glass plates used in autochrome photographs are highly sensitive, making damage-free conservation challenging, and various types of defects have developed in these collections over time. Hence, digitized copies of these autochromes are commonly shown to the public compared to the analog autochromes, as the latter are susceptible to degradation. The underlying challenge is to preserve the original look of the autochrome in the best possible way and to maintain the associated cultural and historical context.

3D photography, which involves capturing the world through a camera and employing image-based rendering techniques for novel view synthesis, offers a significantly more immersive experience than traditional 2D photography, approaching lifelike realism in Virtual Reality (VR) and providing enhanced depth perception on flat displays through parallax. In this paper, we exploit deep learning technologies to convert a digitized mono-autochrome to a 3D photo [11, 24] and then visualize them using Virtual Reality headsets [11]. Based on leveraging deep learning technologies like single-image depth estimation and depth-aware inpainting, we enable to view digitized autochromes in modern VR devices which provides a new and immersive perspective to these historical artifacts, enhancing their cultural and historical significance by bringing them to life in a three-dimensional space. Our main contribution is to

<footnote>*e-mail: saptarshi.neil.sinha@igd.fraunhofer.de</footnote>

<footnote>†e-mail: robin.horst@igd.de</footnote>

<footnote>‡e-mail: irina-mihaela ciortan@ntnu.no</footnote>

<footnote>§e-mail: holger.graf@igd.fraunhofer.de</footnote>

<footnote>||e-mail: arjan.kuijper@igd.fraunhofer.de</footnote>

<footnote>|||e-mail: M.Weinmann@tudelft.nl</footnote>
<figure><img src="image_8.png" /><figcaption>Figure 2: Single image depth estimation of different autochrome scenes using Boosting Monocular Depth Estimation [16]</figcaption></figure>

tions include:

* A pipeline that utilizes deep learning technologies, specifically single-image depth estimation and depth-aware inpainting, to convert a mono-autochrome into a layered depth image [22] and corresponding mesh, followed by the creation of a WebXR application for visualization in virtual reality (VR), as shown in Fig. 1.
* A user study conducted to evaluate the comparative effectiveness and impact of viewing autochromes through VR devices versus traditional visualization methods using a back-light board to display analog autochromes, as shown in Fig. 6. The 3D autochromes generated from selected digitized autochromes of the Harold Taylor collection [26] are now available on GitHub to aid in research progress.

## 2 RELATED WORKS

### 2.1 Single-image depth estimation

Monocular depth estimation (MDE) is an advanced task that involves interpreting monocular depth cues, including occlusions, perspective, and the relative sizes of objects. Initial studies [21, 9, 15] mainly depended on handcrafted features and conventional computer vision methods, which were restricted by their reliance on explicit depth cues and difficulties in addressing complex scenes with occlusions and areas lacking texture. Deep learning approaches have transformed monocular depth estimation by efficiently learning depth representations from carefully annotated datasets. In particular, recent deep neural network approaches [5, 34, 28, 31, 23] have been demonstrated to directly learn depth-related priors from training data, resulting in remarkable advancements in performance due to the availability of large-scale depth datasets [6, 25]. To facilitate effective multi-dataset joint training, the groundbreaking work in zero-shot depth estimation achieved based on the MiDaS approach [20] employs an affine-invariant loss that disregards the potentially differing depth scales and shifts present in various datasets, thereby providing relative depth information. In our study, we utilize an improved version of the MiDaS model [3] in conjunction with the approach presented in [16] for depth estimation. While recent advancements in zero-shot depth estimation models [2, 33] have emerged, we did not evaluate these models in our experiments. Investigating the compatibility of 2D autochromes with these models will be an avenue for future research.

### 2.2 Single-image novel view synthesis

Novel view synthesis allows for the generation of images from novel camera viewpoints not considered during the capture pro-

cess by leveraging 2D images and their related camera poses. Recently, both implicit [1, 18] and explicit [10] learning-based scene representations have demonstrated to offer an impressive potential in generating photorealistic novel views and represent the current state-of-the-art novel view synthesis. Nonetheless, these approaches typically require the availability of many dense views as input, which are not always accessible in many situations and also in the case of 2D mono autochromes. Additionally, they prioritize interpolation using multiple views instead of extrapolation. Thus, we focus on approaches designed to handle single inputs. Several works [27, 7, 14, 32] aim to infer the 3D structure of scenes by learning to predict a scene representation from a single image. While these methods are typically trained end-to-end, they struggle to generalize to real-world photographs. The approaches most pertinent for generating 3D photographs of autochromes include those that leverage depth estimation followed by inpainting of occluded regions [24, 30, 11, 33, 20, 2]. 3D Photo [24] estimates monocular depth maps and employs layered depth images (LDIs) [22], where context-aware color and depth inpainting is used. We adopt this method [24] as it treats the scene as static, contrasting with approaches like 3D cinematography [13], which are inspired by space-time view synthesis techniques. This static representation is essential for autochromes since they are historical artifacts, and our objective is to avoid animating the scene's objects to maintain authenticity.

## 3 METHODOLOGY

To extend the perception of autochromes from a 2D experience to an immersive 3D experience, we use a pipeline built on single-image monocular depth estimation, layered depth image generation, layered depth inpainting, and the production of a mesh (.glb file) for visualization in VR, thereby following the principle proposed by Kopf et al. [11]. In the following, we provide more details regarding the involved processing steps.

### 3.1 Depth estimation

Monocular or single-image depth estimation aims to extract the scene's structure from a single image, relying on high-level monocular depth cues like occlusion boundaries and perspective, in contrast to settings where raw depth information is available from depth sensors or multi-view data with geometric constraints. We used MergeNet [16] along with the zero-shot depth estimator MiDaS [3] to estimate the depth of the 2D mono-autochromes and some of the examples are shown in Fig. 2.

### 3.2 Generation of layered depth image (LDI)

A layered depth image (LDI) [22] is a flexible 4-connected image that can store multiple pixels at each position, with each pixel con-
<figure><img src="image_9.png" /><figcaption>Figure 3: Example stereo view of 3D autochromes from a particular camera perspective</figcaption></figure>

taining a color and depth value, and it explicitly represents local connectivity by storing pointers to at most one neighbor in each cardinal direction, allowing for connectivity in smooth regions while excluding neighbors across depth discontinuities. As proposed by Shih et. al. [24], we use the estimated depth to create a LDI. Given an input RGB-D image (depth estimated using single-image depth estimation as above), the method initializes a trivial LDI and detects major depth discontinuities, grouping them into connected depth edges that serve as the foundation for the inpainting process.

<figure><img src="image_10.png" /><figcaption>Figure 4: Example scenes used for conducting the user study.</figcaption></figure>

## 3.3 Depth-aware inpainting

The inpainting algorithm [24] processes one depth edge at a time to synthesize new color and depth content in adjacent occluded regions, starting by disconnecting LDI pixels across the discontinuity, creating silhouette pixels. It generates a synthesis region using an iterative flood-fill algorithm, expanding from silhouette pixels to cover the occluded area while ensuring that the synthesis region remains strictly within this area and does not backtrack across silhouettes. Additionally, a context region is defined that only includes LDI pixels directly connected to the synthesis region, allowing the inpainting networks to focus solely on relevant surrounding content while addressing misalignments due to depth estimation by dilating the synthesis region near the depth edge.

## 3.4 Mesh generation

In the final stage of the pipeline, the depth-inpainted LDI is converted to a mesh [11]. The mesh is then simplified and converted to a Graphics Library Transmission Format (GLTF) to ensure web compatibility and facilitate faster transmission over the network. Due to the LDI's self-overlapping parts and complex topology, it cannot be mapped to a single contiguous texture image, necessitating the partitioning into flat charts that can be packed into an atlas image for texturing. The mesh simplification algorithm begins by converting the outline of each chart into a detailed 2D polygon and simplifying it using the Douglas-Peucker algorithm [4], ensuring shared boundaries between charts are simplified identically. Next, the chart interiors are triangulated by generating strips of vertical

polylines with evenly spaced interior vertices, followed by a fast plane-sweep algorithm for triangulation. Finally, the 2D triangulation is lifted to 3D by projecting each vertex along its corresponding ray according to its depth, resulting in a 3D triangle mesh.

## 3.5 Visualization in VR

In virtual reality (VR), two slightly different images are created for each eye to enhance depth perception and immersion. 3D autochromes fully utilize the six degrees of freedom available in exploration based on virtual reality technology, allowing for rich interaction. The process begins with rendering a 3D scene using a mesh generated from an LDI (as described in Sec. 3.4) through ThreeJS in aARBGL context. WebXR connects this rendered scene to a VR headset, producing stereo images (as shown in Fig. 3) tailored to the device's specifications — one for each eye. Additionally, the movement of the user's head directly translates to the virtual camera's motion, with constraints on viewing angles, further enhancing the immersive experience.

# 4 EVALUATION

We evaluated our technology by comparing the perception of 3D autochrome representations shown in VR to the traditional 2D perception of autochromes in their physical, analog form. For the latter, we utilized a backlight board, as shown in Fig. 6, that was taken during our study.

## 4.1 Implementation and dataset

The 3D autochromes (layered depth images and the corresponding meshes) were generated from selected digitized mono-autochrome images of the Harold Taylor collection [26] utilizing a machine equipped with an Nvidia RTX 3090 graphics card. The resulting WebXR application was developed using Three.js, and the user study was conducted with the Meta Quest 3 headset. Some of the scenes used for conducting the user study are presented in Fig. 4.

## 4.2 Study design

In a comparative AB within-subject design, each participant engaged with both conditions (A and B). In our study, condition A involves viewing analog autochrome images, while condition B entails experiencing 3D autochrome images through a VR headset. This approach facilitates a direct comparison between the two experiences while effectively controlling for individual differences. The study was conducted in a controlled laboratory environment (see Fig. 6 and Fig. 7) to minimize external influences and ensure the integrity of the results. It involved 12 unpaid and voluntary participants (8 male, 1 female, 1 diverse, and 2 who would not provide this information) between 21 and 57 years with an average age of 33.7 and standard deviation of 11.9 years. Their experience in
<figure><img src="image_11.png" /><figcaption>Figure 5: Boxplot visualization for Q1-Q8 for both conditions: Qx-A for analog/traditional experience and Qx-3D for our 3D VR experience.</figcaption></figure>

general photography was asked using a four-point scale (1=no experience, 4=expert). With an average of 1.3 and standard deviation of 1.0, we classify our participant sample group as beginners but with high deviation. The user tests included two task conditions (*T-analog* and *T-3D*) to be accomplished by our participants using our software prototype. For T-analog, we asked them to explore a set of two physical autochromes with the backlight, whereas for T-VR, we asked to explore a set of five digitized 3D autochromes within our VR applications (randomized from a set of 17). We randomized the order that our participants performed the task conditions in, as well as the order and sub-set of autochromes within the tasks. For example, for our VR app, we had a total set of 17 3D autochromes prepared.

<figure><img src="image_12.png" /><figcaption>Figure 6: Example image taken during our user study where a participant explores physical autochrome photographs by means of a backlight board.</figcaption></figure>

We asked our participants to fill out a questionnaire at four parts of our study: 1) at the beginning (introduction and demographic questions), 2&3) after each task, where the questionnaires incorporated the same questions, and 4) a post-study questionnaire at the end.

Questionnaires 2&3 incorporated the abbreviated version of the standardized AttrakDiff questionnaire [8], measuring the experience character by means of both pragmatic (PQ, e.g., usability) and hedonic qualities (HQ). In these questionnaires, we also asked eight custom questions on a 7-point semantic differential scale, where low numbers indicate a negatively connoted rating of the experience. Questions 1 to 5 captured aspects about the overall perception and aesthetic aspects of the autochrome experience, whereas questions 6 to 8 captured emotional aspects.

Q1 How do you perceive the colours of the pictures? (very unnatural – very natural)

Q2 How would you rate the image quality? (low – high)

Q3 Did the pictures have a nostalgic or picturesque effect on you? (not at all – very much)

Q4 How authentic did you find the pictures you looked at? (not authentic at all – very authentic)

Q5 How detailed were you able to perceive the pictures? (not at all detailed – very detailed)

Q6 How strongly did you feel immersed in the scene of the pictures? (not at all - very much)

Q7 How strongly do you feel connected to the scene depicted in the picture? (not at all connected – very strongly connected)

Q8 How strong were the emotions that the picture triggered in you? (no emotions – very strong emotions)

The post-study questionnaire incorporated further demographic questions and optional free-text questions about negative and positive aspects of the entire experience and suggestions for improvements, as well as the mandatory question *Q9: Which medium do you prefer for displaying an autochrome image? (analog or VR).*

<figure><img src="image_13.png" /><figcaption>Figure 7: Illustration of our study environment.</figcaption></figure>
## 4.3 Study Results

<figure><img src="image_14.png" /><figcaption>Figure 8: Portfolio-presentation of the AttrakDiff values.</figcaption></figure>

The results of the AttrakDiff questionnaire are visualized in the portfolio presentation in Fig. 8. It shows T-3D was assessed with higher hedonic than pragmatic quality whereas T-analog was assessed as rather balanced in this regard. The visualization places T-3D in the 'self-oriented' area and T-analog between 'task-oriented' and 'desired'. Both confidence rectangles intersect with other areas of the continuum. Overall, our software used for T-3D obtained a higher average rating for HQ (4.2), but lower for PQ (3.3) than the traditional autochrome exploration (HQ 4.0, PQ 4.3).

The description of the word-pairs (Fig. 9) shows the mean value for each item of the AttrakDiff individually. Both experiences were rated positively, since only one item lies below the neutral mark of 3, which is 'confusing—clearly structured' for our 3D VR experience. The graph also shows that T-analog was perceived more pragmatic in all items except for 'practical—impractical' which is tied between both conditions. Regarding hedonic qualities, T-analog was rated more stylish and premium than our VR experience, which we can associate with the visual quality of our prototypical user interface implementation.

Figure 5 illustrates the results of Q1-Q8 as boxplots, split into the two test conditions, respectively. For Q1, Q2, Q3 and Q4 that relate to image quality and authenticity, the perception of the 3D autochrome mode is slightly inferior to the perception of the original 2D autochrome. However, for Q5, Q6, Q7 and Q8 that relate to the level of immersion, the perception of details and the triggering of connections and emotions, the potential of the 3D autochrome visualization gets apparent. Furthermore, our question asking which of the two experiences was preferred led to a result of 7 to 5 in favor of the analog exploration. We conducted Wilcoxon signed-rank tests [29] for the individual questions Q1-Q8 between the test conditions to analyze how significantly our experiences differ. With a threshold for statistical significance of 5%, no questions except for Q4 (p=0.01) show significant differences. We evaluated our participants' free-text comments by identifying different themes and aspects, utilizing affinity diagrams as proposed by Preece, Rogers, and Sharp [19]. We could point out four aspects our participants

<figure><img src="image_15.png" /><figcaption>Figure 9: Word-pair visualization of the mean values of the AttrakDiff questionnaire.</figcaption></figure>

mentioned. Firstly, the depth estimation we utilized was criticized, leading to 'unnatural' perceptions of the shown scene, specifically when humans were shown in the images. Secondly, our participants noted positively that the ability of having a large representation of the image did add to the overall experience and also to how detailed the pictures could be explored. The third aspect was a critique where the participants mentioned that the point of view of the 3D scene did look good only from one position, however, there was one participant extensively walking around and shifting the head position up and down, mentioning that some things would never have been explored without this feature. Lastly, our participants mentioned that such an VR application should make use of the interactivity that this medium can offer ('Hardly any interaction 'beyond' watching'). Suggestions that were made are 1) highlighting 'cool' spots or spots 'characteristic for autochrome photographs' within the 3D imagery, 2) an option to view a 2D digital copy in the hands during 3D VR exploration, and 3) leveraging advantages of Augmented Reality or eXtended Reality technologies to offer exploring a 3D autochrome while also being able to interact with the physical one. Overall, our study results indicate that besides the authenticity (in favor of the analog experience), our participants found the 3D VR software for exploring autochrome photographs similarly pleasant (taken into account the AttrakDiff results) as the analog exploration. This underlines the potential that a 3D VR exploration can have in the future on digitally preserving the slowly decaying physical autochrome pictures. However, larger subject numbers and a more sophisticated user interface will be needed in order to get definitive insights.

## 5 LIMITATIONS

Despite the promising results of this study, several limitations must be acknowledged. Firstly, the depth estimation algorithms employed in our 3D reconstruction process could benefit from further refinement [33, 17]. Current algorithms may not accurately capture the complexities of the original images, leading to perceptions of unnatural depth in some scenes, particularly those featuring human subjects. Additionally, the visual quality of the autochromes in the virtual reality environment can be enhanced. Improvements in contrast and color fidelity are necessary to provide a more authentic representation of the original photographs.

Since autochromes are susceptible to degradation and defects including emulsion loss, dye bleeding, and glass cracking, utilizing artificial intelligence to remove these imperfections in 2D autochromes can enhance their quality and provide a more authentic viewing experience of 3D autochromes, thereby elevating the im-
mersive experience and better conveying their historical and cultural significance. Addressing these limitations through advanced depth estimation algorithms, restoration of defects, and improved image contrast will maximize the effectiveness of the 3D VR experience, ensuring a more engaging interaction with these valuable historical artifacts.

# 6 CONCLUSION

This study demonstrates the promising integration of deep learning technologies and 3D photography in reimagining the viewing experience of historical artifacts, specifically autochromes. By transforming digitized mono-autochromes into immersive three-dimensional representations, we can significantly enhance their cultural and historical significance, offering a richer engagement through virtual reality. User feedback highlights both the strengths and limitations of the current VR implementation. While many participants appreciated the immersive nature and the ability to explore images in greater detail, concerns were raised regarding the depth perception in certain scenes. These insights suggest that while the 3D VR medium can deliver a compelling alternative to traditional viewing methods, there remains room for improvement. To maximize the effectiveness of our approach, future work should focus on refining depth estimation, enhancing interactivity, advancing mesh simplification, increasing participant numbers, and utilizing a greater variety of analog autochromes for more robust studies. Addressing these areas will enrich the exploration experience of fragile historical artifacts like autochromes, ensuring their old is preserved and appreciated in a modern context.

**Acknowledgement** The work in this paper was partially funded by the European Commission for the PERCEIVE project (grant agreement 101061157).

# REFERENCES

[1] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields, 2021. 2

[2] S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. *arXiv preprint arXiv:2302.12288*, 2023. 2

[3] R. Birkl, D. Wofk, and M. Müller. Midas v3.1 – a model zoo for robust monocular relative depth estimation. *arXiv preprint arXiv:2307.14460*, 2023. 2

[4] D. Douglas and T. Peucker. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. *Cartographica: The International Journal for Geographic Information and Geovisualization*, 1973. 3

[5] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. In *Proceedings of the Neural Information Processing Systems (NeurIPS)*, 2014. 2

[6] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. *International Journal of Robotics Research*, 2013. 2

[7] Y. Han, R. Wang, and J. Yang. Single-view view synthesis in the wild with learned adaptive multiplane images. In *ACM SIGGRAPH 2022 Conference Proceedings*, 2022. 2

[8] M. Hassenzahl, M. Burmester, and F. Koller. Attrakdiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. In *Mensch & computer 2003*, pp. 187–196. Springer, 2003. 4

[9] D. Hoiem, A. A. Efros, and M. Hebert. Recovering surface layout from an image. *International Journal of Computer Vision*, 2007. 2

[10] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. *ACM Transactions on Graphics*, 42(4), July 2023. 2

[11] J. Kopf, K. Matzen, S. Alsisan, O. Quigley, F. Ge, Y. Chong, J. Patterson, J.-M. Frahm, S. Wu, M. Yu, P. Zhang, Z. He, P. Vajda, A. Saraf, and M. Cohen. One shot 3d photography. In *ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH)*, 2020. 1, 2, 3

[12] B. Lavedrine, J.-P. Gandolfo, and C. Capdero. *The Lumiere Autochrome: History, Technology, and Preservation*. Getty Publications, english edition ed., 2013. 1

[13] X. Li, Z. Cao, H. Sun, J. Zhang, K. Xian, and G. Lin. 3d cinematography from a single image. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2023. 2

[14] X. Li, C. Hong, Y. Wang, Z. Cao, K. Xian, and G. Lin. Symmnerf: Learning to explore symmetry prior for single-view view synthesis. In *Proceedings of the Asian Conference on Computer Vision (ACCV)*, 2022. 2

[15] C. Liu, J. Yuen, A. Torralba, J. Sivic, and W. T. Freeman. Sift flow: Dense correspondence across different scenes. In *Proceedings of the European Conference on Computer Vision (ECCV)*. Springer, 2008. 2

[16] S. M. H. Miangoleh, S. Dille, L. Mai, S. Paris, and Y. Aksoy. Boosting monocular depth estimation models to high-resolution via content-adaptive multi-resolution merging. In *Proc. CVPR*, 2021.2

[17] S. M. H. Miangoleh, M. Reddy, and Y. Aksoy. Scale-invariant monocular depth estimation via ssi depth. In *Proc. SIGGRAPH*, 2024. 5

[18] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoor-thi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In *ECCV*, 2020. 2

[19] J. Preece, Y. Rogers, and H. Sharp. *Interaction design: beyond human-computer interaction*. John Wiley & Sons, 2015. 5

[20] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. *arXiv:1907.01341*, 2019. 2

[21] A. Saxena, M. Sun, and A. Y. Ng. Make3d: Depth perception from a single still image. In *Proceedings of the AAAI Conference on Artificial Intelligence*. AAAI, 2008. 2

[22] J. Shade, S. Gortler, L.-w. He, and R. Szeliski. Layered depth images. In *Association for Computing Machinery, Inc.*, 1998. 2

[23] S. Shao, Z. Pei, W. Chen, X. Wu, and Z. Li. Nddepth: Normal-distance assisted monocular depth estimation. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*. IEEE, 2023. 2

[24] M.-L. Shih, S.-Y. Su, J. Kopf, and J.-B. Huang. 3d photography using context-aware layered depth inpainting. In *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2020. 1, 2, 3

[25] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. In *Proceedings of the European Conference on Computer Vision (ECCV)*. Springer, 2012. 2

[26] H. A. Taylor. Harold taylor slide collection. California Revealed, 1896. Visited on 11/26/2024. 2, 3

[27] R. Tucker and N. Snavely. Single-view view synthesis with multiplane images. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2020. 2

[28] L. Wang, J. Zhang, Y. Wang, H. Lu, and X. Ruan. Cliffnet for monocular depth estimation with hierarchical embedding loss. In *Proceedings of the European Conference on Computer Vision (ECCV)*. Springer, 2020. 2

[29] F. Wilcoxon and R. A. Wilcox. Some rapid approximate statistical procedures. Lederle Laboratories, 1964. 5

[30] O. Wiles, G. Gkioxari, R. Szeliski, and J. Johnson. Synsin: End-to-end view synthesis from a single image. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2020. 2

[31] A. Wong and S. Soatto. Bilateral cyclic constraint and adaptive regularization for unsupervised monocular depth prediction. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. IEEE, 2019. 2

[32] D. Xu, Y. Jiang, P. Wang, Z. Fan, H. Shi, and Z. Wang. Sinnerf: Training neural radiance fields on complex scenes from a single image. In *Proceedings of the European Conference on Computer Vision (ECCV)*, 2022. 2

[33] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In *CVPR*, 2024. 2, 5

[34] C. Zheng, T.-J. Cham, and J. Cai. T2net: Synthetic-to-realistic translation for solving single-image depth estimation tasks. In *Proceedings of the European Conference on Computer Vision (ECCV)*. Springer, 2018. 2
# Embodied and Tangible Interaction in Collaborative Room-Scale Mixed Reality: Insights from a Visitor Experience in a Cultural Heritage Context

Maria Roussou, Dimitris Nastos*
University of Athens

Sophia Bakogianni†
Panteion University

Dimitris Christopoulos, George Sofianopoulos‡
Foundation of the Hellenic World

Christian Schönauer§
TU Wien

<figure><img src="image_16.png" /><figcaption>Figure 1: Visitors interacting with tangible replicas in a multi-user Mixed Reality cultural heritage experience.</figcaption></figure>

## ABSTRACT

The rapid expansion of eXtended Reality (XR) technologies, particularly collaborative room-scale Mixed Reality (MR), has created opportunities for shared virtual experiences that blend real and virtual environments, allowing multiple users to interact and engage collaboratively. This paper presents an evaluation of a cultural heritage MR application, focusing on two key aspects: embodied interaction and tangibility. Embodied interaction was enabled through free movement and tactile engagement with co-present participants. Tangibility was facilitated by allowing visitors to handle replicas of ancient artifacts and interact with physical elements aligned with their virtual counterparts. Drawing on insights from a study involving

73 visitors, the paper highlights how these elements contributed to the overall experience, fostering immersion, collaboration, and deeper engagement with cultural content. By focusing on visitor perspectives and presenting qualitative findings, this study aims to advance understanding of the impact of tangibility and embodiment in collaborative MR experiences within cultural heritage contexts.

Index Terms: mixed reality, tangibility, embodiment, room scale, cultural heritage.

## 1 INTRODUCTION

eXtended Reality technologies are increasingly recognized as powerful tools for enhancing user experiences across diverse domains, including cultural and recreational contexts. Immersive Mixed Reality, a subset of XR, blends real and virtual environments to enable users to interact with digital content while engaging with their physical surroundings. Included within the broader definition of MR are capabilities such as free-form movement in larger physical spaces, natural gesture-based interactions, multi-user functionality, and multi-modal sensory experiences, including audio, haptics, vibrotactile feedback, and even unconventional modalities like temperature, wind, taste,

<footnote>*e-mail: mroussou@di.uoa.gr, dnastos@makebelieve.gr</footnote>

<footnote>†e-mail:sophia.bakogianni@gmail.com</footnote>

<footnote>‡e-mail:christop@fhw.gr, gsofian@fhw.gr</footnote>

<footnote>§e-mail:christian.schoenauer@tuwien.ac.at</footnote>
and smell [19].

Room-scale MR, characterized by its support for physical movement and natural interactions, has seen some application in gaming (with location-based entertainment attractions like The Void<fnref n="1" />, training, simulation, architecture, healthcare, and collaborative work. Although still nascent in cultural heritage and archaeology, MR holds significant promise for enhancing visitor engagement and learning [13]. Cultural institutions are beginning to explore how immersive and interactive experiences can expand their appeal to broader demographics, with a growing emphasis on multimodal and multisensory engagement to attract visitors of all ages and backgrounds.

This paper examines two critical aspects of MR—embodied interaction and tangible engagement with physical spaces and objects—within the informal learning context of a cultural heritage center or museum. The study draws on the evaluation of a custom MR installation deployed in a cultural space designed to support immersive experiences through the use of headsets, simultaneous participation for up to five co-located visitors, and device-less interaction enabled by hand and motion tracking. Visitors could interact with tangible physical objects while experiencing multi-sensory stimuli, including smell and wind. Tailored for institutions open to a broader public, the system offers a low-cost, adaptable solution suitable for spaces of varying sizes. Deployed in a prominent cultural institution in a European capital city, this installation has been actively used by the general public, engaging visitors of all ages. The findings presented in this paper focus on visitor experience, highlighting the specific contributions of embodiment and tangibility to engagement and learning within this cultural heritage context.

# 2 BACKGROUND AND RELATED WORK

## 2.1 Mixed Reality and Cultural Heritage

Mixed Reality applications in cultural heritage are still emerging but have shown great potential for transforming visitor experiences by blending the physical and digital realms. Bekele et al. [1] developed a cloud-based collaborative and multi-modal MR application for virtual heritage, emphasizing the integration of multiple sensory modalities to create immersive and interactive learning environments. Their work highlights the importance of collaboration and co-located experiences in fostering deeper cultural engagement. Building on this, Bekele et al. [2] examined the influence of collaborative MR on cultural learning, revealing that multi-user interaction significantly enhances understanding and retention of cultural knowledge. Their findings emphasize the role of MR in promoting social interaction and collective discovery within heritage contexts. Broader surveys offer reviews of the state of AR, VR, and MR in cultural heritage [3] and wearable MR technology in exhibitions [23], identifying trends and challenges in adopting these technologies. Key insights include the importance of designing user-centered experiences that balance technical complexity with accessibility and relevance to diverse audiences.

Other notable examples include Plecher et al. [15], who explored MR as a tool for cultural heritage storytelling, demonstrating its potential for creating immersive and narrative-driven experiences. Dima et al. [5] investigated MR's use in decolonizing heritage sites, leveraging its ability to present multiple perspectives and engage visitors emotionally and intellectually. Vosinakis et al. [21] employed MR to co-design playful installations, showcasing its versatility in creating educational and entertaining heritage experiences.

Schönauer et al. [16, 17] introduced the ImmersiveDeck, an XR platform designed for a variety of purposes. The platform's ability to support multi-sensory and embodied interactions has implications for cultural heritage applications, particularly in fostering engaging

and practical learning environments. The experience presented in this paper is based on the ImmersiveDeck and was developed as part of an innovation action aiming to deploy such installations in informal learning and first responder training contexts [8].

A similarly rich location-based installation embedded in museum settings is "Thresholds" by Tennent et al. [20], demonstrating how immersive technologies can seamlessly integrate with traditional exhibition spaces. This project highlights the potential for VR to enhance visitor engagement by providing contextually rich and interactive storytelling experiences within cultural institutions.

## 2.2 Tangible Interaction in Cultural Heritage

Tangible interaction has become a key focus in the design of engaging cultural heritage experiences, where physical objects and sensory feedback are leveraged to deepen visitor immersion and connection to the past. Duranti et al. [7] provide a comprehensive survey of tangible and embodied interactions in museums, highlighting the integration of smart objects and replicas that allow visitors to handle physical artifacts while receiving augmented digital content. This approach bridges the gap between the intangible and the physical, fostering an enriched understanding of cultural heritage [12].

The significance of touch in enhancing user experiences is well-documented. Gallace and Spence [9] emphasize how tactile interactions engage both sensory and cognitive processes, making learning more immersive and memorable. Dima et al. [6] explored haptic augmented reality (AR) as a means to engage museum visitors with historical artifacts, demonstrating how touch-based interactions can significantly enhance the sensory and cognitive dimensions of learning. Similarly, Petrelli et al. [14] introduced a tangible interaction toolkit for designing heritage installations, enabling rapid prototyping of experiences that blend physical objects with interactive technology to create meaningful and contextually relevant visitor engagements. Projects such as Ioannidis et al.'s "We Dare You" [10] have further demonstrated the potential of tactile and substitutional reality systems in museum spaces, where physical replicas and real-world interactions are embedded into immersive virtual environments. These studies underscore the importance of tangible interaction in fostering active participation and a sense of presence among museum-goers.

# 3 MR PLATFORM DESCRIPTION

The MR platform presented in this study combines established XR elements with novel multi-sensory and collaborative features, delivering an immersive experience beyond conventional multi-user VR setups. Participants, represented by avatars, can freely walk through a virtual environment, converse in a shared physical space, and interact with virtual objects and avatars—including touching virtual representations of other participants' hands. Each user wears an HP Reverb G2 HMD tethered to an HPZ VR backpack G2 PC for local rendering, minimizing latency. Hand and finger tracking is achieved through an Ultraleap stereo camera attached to the HMD, enabling intuitive, gesture-based interactions (Fig. 2).

The installation is site-specific, with the physical room precisely mirrored in the virtual environment on a 1:1 scale. A digital twin of the space was generated using photogrammetry and Lidar scanning, serving both for virtual world development and for placing lightweight cardboard ceiling markers required for accurate tracking. Real-world structures, including walls and columns, align perfectly with their virtual counterparts, enhancing spatial awareness, tangibility, and the exploratory nature of the experience.

To achieve this real-virtual alignment, the platform employs a combination of off-the-shelf hardware and custom-developed software. A proprietary real-time tracking system ensures accurate synchronization of users' movements and interactions [16]. Tracking combines optical and inertial head and body tracking, with ceiling-mounted markers detected by wide-angle fisheye cameras

<footnote n="1">¹https://en.wikipedia.org/wiki/The%22oid(virtual,eality)</footnote>
<figure><img src="image_17.png" /><figcaption>Figure 2: Above: Users' virtual hand (left) and real hand (right) are tracked. Below: The object-tracked 3D printed vase handled by visitors in the MR environment (left) and its virtual counterpart (right).</figcaption></figure>

on the HMD. These markers, in conjunction with Simultaneous Localization and Mapping (SLAM) data, provide accurate absolute positioning and high frame-rate relative tracking [18].

A key distinguishing feature of this platform is its emphasis on multi-sensory engagement. While visual and auditory stimuli are standard in most XR applications, this installation uses tactile interaction to address a known gap in XR research [9, 22]. Users physically interact with real objects integrated into the virtual environment, such as movable pottery equipped with autonomous inside-out tracking systems. These systems use Linux-based Odroid N2 mini-PCs and Intel RealSense stereo cameras for real-time localization.

Small Ultraleap devices mounted on the HMD also enable precise hand and finger tracking for non-verbal communication and object manipulation. Users can engage in realistic interactions, such as handshakes, and navigate the virtual environment by physically walking within the tracked space [11]. Standard 3D object interactions, such as grabbing, attaching objects, or opening drawers and doors, are available throughout the experience.

Unlike traditional VR systems, where the virtual world remains entirely detached from the physical space, this platform seamlessly bridges the two realms, enhancing realism and presence. This integration is particularly valuable for scenarios requiring high spatial fidelity, such as cultural heritage experiences, where tangible interaction enriches engagement and learning outcomes.

The platform's software is primarily based on the Unity game engine and editor, with user and object tracking software encapsulated as separate processes and assemblies implemented in C++ and C#.

The MR experience featured a guided storytelling scenario set in Classical Athens, designed to immerse visitors in the daily life of an Athenian family. Participants assumed the roles of guests invited to the home of a middle-class pottery merchant. The group entered a specially constructed physical environment that corresponded to the layout of the virtual house. Guided by virtual hosts, visitors could walk through different rooms of the house, ascend a staircase to the upper level, and gather around the hearth to participate in household activities. These activities included preparing a meal, offering aromatic incense to the gods (which visitors could smell), and interacting with virtual and tangible objects using natural hand gestures (Fig. 3).

# 4 EVALUATION

The site-specific nature of the offered experience creates the need for testing on-site and evaluating the site-specific requirements and their impact on the users' overall embodied experience. For this reason,

<figure><img src="image_18.png" /><figcaption>Figure 3: Participants in the real space (above) and their avatars in the virtual world (below) interacting in MR.</figcaption></figure>

the evaluation of the MR experience employed various methods to cover the whole design continuum, from evaluating the development stages of tools, methodologies, and experiences (formative evaluation) to final prototypes (summative evaluation). To capture the impact of the experience and its multifaceted influence in the informal learning and edutainment context of the cultural institution in which it was deployed, the evaluation employed a holistic, triangulated, multi-method, and multilevel approach, covering not only the usability but also the achievement of pedagogical and experiential goals, meeting the operational, and technical objectives, too [23]. Furthermore, the developed evaluation framework considers all the general and typical requirements or risks of a collaborative XR experience to guarantee users' safety and hygiene.

## 4.1 Methodology

### 4.1.1 Evaluation Framework

All evaluation activity was broadly guided by six axes established early in the project's lifetime, during the requirements analysis phase (see Table 1). These axes serve as the basis for developing a cross-context evaluation framework tailored to assess immersive experiences in collaborative room-scale MR environments. The evaluation framework was applied in evaluation studies in two diverse application domains, adding depth and practical relevance to its effectiveness and adaptability. In this paper, we present its application in the cultural heritage domain, focusing on the pedagogical (PE) and experiential (EX) aspects of embodiment and tangibility.

### 4.1.2 Participants

An open call was created and sent out via e-mail to our networks of colleagues, friends, and acquaintances approximately two weeks before the start of the evaluation sessions. A total of 73 visitors were recruited, forming 19 groups with an average of 4 participants each. We had no influence over the selection and (gender, age, experience) balancing of participants. As visitors were recruited through a semi-open call, it was impossible for us to control who would finally participate. Thus, female participants outnumbered males. Two participants identified as non-binary.
<figure><img src="image_19.png" /><figcaption>Figure 4: Visitors using their (virtual) hands to interact.</figcaption></figure>

### 4.1.3 Procedure

The following procedure was applied to all participants who arrived at the venue to take part in the MR experience: 1) Group formation, 2) Researchers' introduction to the experience and consent, 3) Pre-experience questionnaire, 4) Observation of the groups during their experience, 5) Post-experience questionnaire, 6) Post-experience focus group. Specifically, participants were asked to complete pre- and post-experience questionnaires. The pre-experience questionnaire (Pre-X) included 10 single and multiple-choice questions and questions using Likert scales. The post-experience (Post-X) questionnaire comprised different sections containing 18 questions. Most questions were either multiple choice or Likert ranking, while one was open-ended, asking for a free text response. None of the questions were mandatory.

# 5 FINDINGS

This section presents a selection of the aggregated qualitative data gathered from the Pre-X and Post-X questionnaires, the observation of the users' experience, and the verbal feedback from the focus group interviews. The first level of analysis was conducted by three researchers (R1-R3), who summarised patterns across the collected data. As a next step, we are currently analyzing these findings to yield insights and guidelines for the design of such MR experiences in cultural contexts.

Specifically, the observation of participants during their experience was guided by the pre-identified axes. Then, short video snippets were created, grouping the various sections of the MR experience and activities of interest. Finally, all 19 focus group interviews recording visitors' opinions and feelings about the experience were transcribed verbatim by R1, who acted as a moderator and thus could distinguish participants' voices. R2 and R3 incorporated their written notes into the transcribed interviews. This resulted in approximately 120 pages of text. Every transcribed segment was assigned to one or more themes using a thematic analysis approach [4].

<figure><img src="image_20.png" /><figcaption>Figure 5: Moving around and interacting with others by touch.</figcaption></figure>

## 5.1 Room-Scale Movement and Embodiment

### 5.1.1 Embodiment and Virtual Hands

Participants consistently reported that using their virtual bodies to navigate and perform tasks in the MR environment felt natural and intuitive. Notably, their attention was frequently drawn to their (virtual) hands, which they used extensively during the experience (Fig. 4). Observing their virtual hands, moving them, and using them to interact with objects reinforced their sense of embodiment. Quotes from participants highlight this connection:

> "I am excited by the accuracy between my body and the space, both the physical and the virtual one. I had never worn such a headset and equipment" (g1-03, male participant).

> "I observed my VR body and my hands. I was moving my hands, I touched objects, and I interacted with the environment. My virtual hands had two moles, and I wondered to whom they belonged. I felt like I was in a parallel universe" (g7-01, male participant).

> "I was impressed by the fact that I could coordinate my hand movements in the virtual space with my real hands, even though the virtual hands were a bit ugly. I had the feeling of my hands. We counted, we gave five, and we did a handshake like in the real world. It was really something!" (g1-03, male participant).

### 5.1.2 Movement and Spatial Alignment

The ability to freely move around the virtual environment enhanced participants' awareness of their physical and virtual bodies [11]. Many noted how the alignment between the physical and virtual spaces contributed to a heightened sense of immersion. Participant reflections include:

> "I was impressed by the matching of the real space with the VR space and the fact that we can move around" (g3-01, female participant).

> "I liked very much that I could move around the home. It was so real..." (g3-02, female participant).

> "Compared to other VR experiences, which are static, this is more immersive because you can walk" (g6-05, female participant).

> "The fact that I could walk around was very interesting, and this is different from other computer games..." (g13-04, male participant).

### 5.1.3 Interaction with Others

Sharing the experience with other participants, represented as avatars, added a social dimension to the experience, reinforcing the sense of immersion and reducing feelings of isolation. Physical interactions,
such as sensing others' proximity or being touched, were particularly impactful (Fig. 5). Participants shared their thoughts:

> "I felt the other bodies near me, compared to the virtual characters. Being with others helped me feel immersed in the experience. I felt the existence of g12-01 near to me in space—not just seeing her avatar. I couldn't see her real body in the virtual environment, but I sensed her presence" (g12-02, male participant).

> "I felt someone touch me when I was in the corridor" (g5-05, female participant).

> "When we were walking around, someone could touch you, and you had the sense that you were not alone there. This was amazing..." (g7-01, female participant).

## 5.2 Tangible Elements

The MR experience employed a multi-sensory approach designed to motivate and engage participants through hearing, sight, touch, and smell stimulation. Engaging multiple senses was identified as one of the most positively received aspects of the experience. These findings align with those of Tennent et al. [20], who emphasized that the "devil is in the details" when evaluating a similar room-scale tactile XR experience, suggesting that multi-sensory details significantly enhance the overall impact. The tactile dimension, in particular, was a standout feature for many participants. The ability to touch physical objects, interact with the environment, and make physical contact with avatars was a pleasant surprise for most visitors. Participants shared their impressions:

> "I liked very much the fact that we could touch each other." (g4-03, male participant).

> "I also liked that I could touch Ippias, my virtual son... this was very nice!" (g16-03, female participant).

> "I also liked that we could touch other participants and grab things into our hands. This part was the most interesting." (g18-03, female participant).

A particularly compelling element was the inclusion of a 3D-printed tangible object—a vase—within the virtual environment (see Fig. 1). For many participants, this tangible addition was an unexpected and memorable aspect of the experience, as reflected in the following comments:

> "I would also like it if there were more tangible objects. The tangible object was a surprise." (g13-02, female participant).

> "When I got the vase into my hands, I said, 'Is this real?' I didn't expect it." (g1-03, male participant).

> "I wanted to touch everything. I also grasped the vase before the others. I had the desire to feel the whole space." (g4-03, male participant).

However, some participants reported mixed feelings about the tangible vase. For a few, the experience of touching the vase was stressful:

> "The tangible vase made me wonder what it was that I touched. It stressed me a bit and surprised me." (g12-01, female participant). Additionally, technical glitches with the tangible object were noted, such as poor tracking or inconsistent synchronization with virtual interactions. While some found these issues distracting, most felt they did not significantly detract from the overall experience:

> "The tangible vase was trembling from the beginning. Probably something did not function properly." (g11-03, male participant).

> "When I passed the vase to someone else, her hands seemed to be away, and the vase was hanging in the air. That was weird." (g16-02, female participant).

> "The tangible vase was very promising from a technological point of view. However, the bad tracking of hands made it less effective as an experience. Still, it was amazing!" (g11-02, male participant).

> "I am not very familiar with VR, so the trembling of the vase did not affect my experience. I don't have high expectations." (g11-04, male participant).

Despite these issues, the integration of multisensory and tangible elements was viewed mainly as a strong positive, enriching the immersive nature of the XR experience.

# 6 DISCUSSION

Embodiment and tangibility in location-specific, multimodal MR experiences remain relatively underexplored in cultural heritage contexts. This study offers a contribution to this area by examining how these elements influence visitor engagement, immersion, and informal learning in such settings. The findings suggest that while embodied and tangible interactions enrich the experience, they also present specific challenges related to storytelling and narrative coherence.

On the positive side, the evaluation revealed that both embodiment and tangibility contributed significantly to the immersive and engaging nature of the MR experience. The sense of embodiment—facilitated by free movement, natural hand-based interactions, and tactile engagement with the environment—created a strong feeling of presence for participants. Visitors actively explored the reconstructed historical setting rather than passively consuming information, making the experience more dynamic and engaging. Tangibility, primarily through physical interaction with a 3D-printed object (the tangible vase), was consistently highlighted as one of the most memorable and novel elements. Participants were surprised and delighted by the unexpected physicality of the object, which bridged the gap between the real and virtual worlds, enhancing their overall sense of immersion.

Social interaction, a key aspect of embodiment, also emerged as an essential contributor to engagement. The shared experience allowed visitors to interact with one another physically and virtually, fostering collaboration and playful exploration. Many participants found this element highly enjoyable, noting that it added a unique, interactive dimension to their visit. This social component not only enhanced immersion but also encouraged shared reflection and discussion—valuable outcomes for informal learning environments such as museums.

However, this very strength of embodied and tangible interaction posed a significant challenge to the storytelling aspect of the experience. The narrative, which aimed to guide visitors through a coherent plot in the reconstructed historical setting, was often overshadowed by the playful nature of physical and social interactions. Many participants became so absorbed in interacting with the tangible vase, exploring the virtual environment, and engaging with other visitors that they lost track of the plot. This tension between interaction and narrative coherence reflects a known challenge in the design of XR experiences, particularly in cultural contexts where both learning and storytelling are key objectives. While multisensory engagement and social interaction enhance user immersion, they can also distract from following the intended storyline, diluting the impact of the narrative.

Another area for improvement concerns the pacing and structure of the experience. Several participants expressed a desire for more time to explore and reflect on the content, as the fast pace led to a sense of being rushed and reduced opportunities for contemplation. Additionally, while the multi-sensory design was generally appreciated, some participants found that the abundance of simultaneous activities created sensory overload, making it difficult to focus on key elements of the plot. Participants also expressed a desire for greater agency within the experience, as some felt that their actions did not meaningfully influence the unfolding events. This highlights the need for balancing rich, interactive features with clear narrative progression and moments of reflection to enhance both engagement and learning outcomes.

Despite these challenges, the findings underline the educational potential of embodied and tangible MR experiences in cultural contexts. Visitors reported acquiring new factual knowledge and engag-
ing deeply with the reconstructed environment. Notably, the sense of embodiment enabled a hands-on approach to learning, making cultural content more accessible and relatable. Moreover, the experience fostered historical empathy, particularly in scenarios that explored interpersonal relationships in antiquity, helping participants form emotional connections with the past.

Future efforts in designing MR experiences for cultural heritage should focus on resolving the tension between interaction and narrative coherence. One potential approach is to integrate moments of guided reflection or structured narrative breaks to help participants process their experience without disrupting the flow of interaction. Additionally, improving the reliability of tangible elements, such as addressing tracking glitches, could further enhance user engagement without detracting from the plot. By carefully balancing embodiment, tangibility, and storytelling, future MR applications can create more cohesive and impactful experiences that blend learning, play, and narrative in meaningful ways.

## 7 CONCLUSION

The findings of this evaluation carry implications for the design and evaluation of XR technology and experiences across various contexts. Moving forward, our aim is to further refine and expand our analysis to yield deeper insights and ultimately propose comprehensive guidelines for designing MR experiences employing tangible and embodied aspects in diverse settings. These guidelines will serve the broader XR community by offering considerations related to operational efficiency, site-specific implementation across different contexts, technical enhancements, experiential qualities, and pedagogical effectiveness. By providing guidance on these crucial layers of XR experiences, we hope to contribute to the continuous improvement of processes and the creation of more impactful and engaging XR encounters.

## ACKNOWLEDGMENTS

This research is part of the BRIDGES project https://www.bridgeshorizon.eu/, which has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 952043. We thank all partners for their contribution. Special appreciation goes to Julien Rüggeberg, Jim Rüggeberg, Sakis Rogkas, and Lucas Katsikaris for their support in carrying out the evaluation sessins.

## REFERENCES

[1] M. K. Bekele. Clouds-Based Collaborative and Multi-Modal Mixed Reality for Virtual Heritage. *Heritage*, 4(3):1447–1459, July 2021. doi: 10.3390/heritage4030080

[2] M. K. Bekele, E. Champion, D. A. McMeekin, and H. Rahaman. The Influence of Collaborative and Multi-Modal Mixed Reality: Cultural Learning in Virtual Heritage. *Multimodal Technologies and Interaction*, 5(12):79, Dec. 2021. doi: 10.3390/mti5120079

[3] M. K. Bekele, R. Pierdicca, E. Frontoni, E. S. Malinverni, and J. Gain. A Survey of Augmented, Virtual, and Mixed Reality for Cultural Heritage. *Journal on Computing and Cultural Heritage*, 11(2):1–36, Mar. 2018. doi: 10.1145/3145534

[4] V. Braun and V. Clarke. *Thematic Analysis: A Practical Guide*. SAGE Publications Ltd, 1st ed., 2021.

[5] M. Dima, D. Daylamani-Zad, and V. Lympouridis. Mixed Reality Heritage Performance As a Decolonising Tool for Heritage Sites, 2024. Version Number: 1. doi: 10.48550/ARXIV.2404.07348

[6] M. Dima, L. Hurcombe, and M. Wright. Touching the Past: Haptic Augmented Reality for Museum Artefacts. In *Proceedings of the 6th International Conference on Virtual, Augmented and Mixed Reality. Applications of Virtual and Augmented Reality - Volume 8526*, pp. 3–14. Springer-Verlag New York, Inc., New York, New York, USA, 2014. doi: 10.1007/978-3-319-07464-1_1

[7] D. Duranti, D. Spallazzo, and D. Petrelli. Smart Objects and Replicas: A Survey of Tangible and Embodied Interactions in Museums and

Cultural Heritage Sites. *Journal on Computing and Cultural Heritage*, 17(1):1–32, Feb. 2024. doi: 10.1145/3631132

[8] K. El Raheb, A. Soulis, D. Nastos, C. Lougiakis, M. Roussou, D. Christopoulos, G. Sofianopoulos, S. Papagiannis, J. Rüggeberg, L. Katsikaris, and J. Rüggeberg. Eliciting requirements for a multisensory eXtended Reality platform for training and informal learning. In *CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter*, pp. 1–8. ACM, New York, NY, USA, Nov. 2021. doi: 10.1145/3489410.3489428

[9] A. Gallace and C. Spence. *In Touch with the Future: The sense of touch from cognitive neuroscience to virtual reality*. Oxford University Press, Jan. 2014. doi: 10.1093/acprof:oso/9780199644469.001.0001

[10] P. Ioannidis, L. Eklund, and A. S. Løvlie. We Dare You: A Lifecycle Study of a Substitutional Reality Installation in a Museum Space. *Journal on Computing and Cultural Heritage*, 14(3):1–21, July 2021. doi: 10.1145/3439862

[11] N. C. Nilsson, S. Serafin, F. Steinicke, and R. Nordahl. Natural Walking in Virtual Reality. *Computers in Entertainment*, 16(2):1–22, Apr. 2018. doi: 10.1145/3180658

[12] E. Nofal, R. Reffat, V. Boschloos, H. Hameeuw, and A. Moere. *The Role of Tangible Interaction to Communicate Tacit Knowledge of Built Heritage*, vol. 1. Dec. 2018. Publication Title: Heritage Issue: 2 ISSN: 2571-9408. doi: 10.3390/heritage1020028

[13] G. Papagiannakis, E. Geronikolakis, M. Pateraki, V. M. López-Menchero, M. Tsioumas, S. Sylaiou, F. Liarokapis, A. Grammatikopoulou, K. Dimitropoulos, N. Grammalidis, N. Partarakis, G. Margetis, G. Drossis, M. Vassiliadi, A. Chalmers, C. Stephaniidis, and N. Magnenat-Thalmann. Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums. In N. Lee, ed., *Encyclopedia of Computer Graphics and Games*, pp. 1150–1162. Springer International Publishing, Cham, 2024. doi: 10.1007/978-3-031-23161-2_249

[14] D. Petrelli, L. Ciolfi, and G. Avram. Envisioning, designing, and rapid prototyping heritage installations with a tangible interaction toolkit. *Human-Computer Interaction*, pp. 1–41, Sept. 2021. doi: 10.1080/07370024.2021.1946398

[15] D. A. Plecher, M. Wandinger, and G. Klinker. Mixed Reality for Cultural Heritage. In *2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*, pp. 1618–1622. IEEE, Mar. 2019. doi: 10.1109/VR.2019.8797846

[16] I. Podkosova, K. Vasylevska, C. Schoenauer, E. Vonach, P. Fikar, E. Bronederk, and H. Kaufmann. Immersivedeck: a large-scale wireless VR system for multiple users. In *2016 IEEE 9th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS)*, pp. 1–7. IEEE, Mar. 2016. doi: 10.1109/SEARIS.2016.7551581

[17] C. Schönauer, H. Kaufmann, M. Roussou, J. Rüggeberg, J. Rüggeberg, L. Katsikaris, S. Rogkas, and D. Christopoulos. Creating Informal Learning and First Responder Training XR Experiences with the ImmersiveDeck. In *2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)*, pp. 53–60. IEEE, Mar. 2023. doi: 10.1109/VRW58643.2023.00016

[18] N. Shibata and S. Yamamoto. GPGPU-Assisted Subpixel Tracking Method for Fiducial Markers. *Journal of Information Processing*, 22(1):19–28, 2014. doi: 10.2197/ipsjip.22.19

[19] M. Speicher, B. D. Hall, and M. Nebeling. What is Mixed Reality? In *Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems*, CHI ’19, pp. 1–15. Association for Computing Machinery, New York, NY, USA, May 2019. doi: 10.1145/3290605.3300767

[20] P. Tennent, S. Martindale, S. Benford, D. Darzentas, P. Brundell, and M. Collishaw. Thresholds: Embedding Virtual Reality in the Museum. *Journal on Computing and Cultural Heritage*, 13(2):1–35, 2020. doi: 10.1145/3369394

[21] S. Vosinakis, V. Nikolakopoulou, M. Stavrakis, L. Fragkedis, P. Chatzigrigoriou, and P. Koutsabasis. Co-Design of a Playful Mixed Reality Installation: An Interactive Crane in the Museum of Marble Crafts. *Heritage*, 3(4):1496–1519, Dec. 2020. doi: 10.3390/heritage3040083

[22] N. Yannier, K. R. Koedinger, and S. E. Hudson. Tangible collaborative learning with a mixed-reality game: Earthshake. In *Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial
*Intelligence and Lecture Notes in Bioinformatics*, vol. 7926 LNAI, pp. 131–140, 2013. ISSN: 03029743. doi: 10.1007/978-3-642-39112-5-14

[23] J. H. Yi and H. S. Kim. User Experience Research, Experience Design, and Evaluation Methods for Museum Mixed Reality Experience. *Journal on Computing and Cultural Heritage*, 14(4):1–28, Dec. 2021. doi: 10.1145/3462645
Cognitive Archaeology in Virtual Environment

Maxime Dumonteil *

Univ. Rennes, Inria, CNRS,
IRISA, France

Valérie Gouranton †

Univ. Rennes, INSA Rennes,
Inria, CNRS, IRISA, France

Marc Macé ‡

IRISA, CNRS, Univ. Rennes,
Inria, France

Théophane Nicolas §

Inrap, UMR Trajectoires,
France

Ronan Gaugne ¶

Univ. Rennes, Inria, CNRS,
IRISA, France

<figure><img src="image_21.png" /><figcaption>Figure 1: Comparing the behavior during an archaeological task between different modalities</figcaption></figure>

**ABSTRACT**

The integration of Virtual Reality (VR) and Extended Reality (XR) has significantly transformed the working methods of archaeologists, enabling experts to explore artifacts in a controlled virtual environment without altering them. However, it may differ when XR technologies are used compared to real-world experiences. This paper introduces a workflow that generates standardized eye-tracking and behavioral data, allowing for a comparison between real and virtual modalities. A user study of visual exploration of an artifact based on this workflow is presented to identify biases in the use of VR tools for archaeology. The paper also outlines ongoing work to explore virtual interactions, such as representing missing or hidden parts of archaeological materials, to study their influence on user behavior. XR offers promising opportunities for enhancing the understanding and interactions with cultural heritage, but further research is needed to extend and refine these methods to ensure they provide a relevant archaeological experience.

**Index Terms:** Virtual Reality, Archaeology, HMD, Immersive room, Behavior.

# 1 INTRODUCTION

Virtual reality (VR) has become increasingly employed in the fields of archaeology and cultural heritage [5, 7, 14]. It facilitates interaction with historical or archaeological material, artifacts and sites from the past without altering them, in a safer and more comprehensive way. For archaeologists and specialists, the integration of virtual reality-based workflows is gradually transforming techniques, offering new methods of interaction, interpretation and exploration. The observation of archaeological material is central to the archaeologist's work. Virtual reality makes it easy to visualize digital copies of such material. In this sense, techniques based on virtual reality, and more generally extended reality (XR), can address

a wide range of observational activities [12].

However, exposure to virtual archaeological representations may induce different knowledge, information and observational behaviors than direct confrontation with the real thing, depending on the use case [9, 15, 21, 10]. This question is becoming particularly critical as recent studies link behavioral studies of the way people look at artifacts with archaeological interpretations of the organization of the society that produced them [1, 2] or how they were used [20].

For this reason, we propose to design an experimental workflow to assess the extent to which the use of virtual reality can adequately reproduce a real archaeological experience. To this end, it is important to identify the main parameters that need to be observed to ensure that the simulation is as close as possible to reality and that the numerical exploration deviates as little as possible from the real experience.

The studies carried out with this workflow will then be used to identify cognitive and behavioral biases introduced by XR environments in the execution of archaeological tasks. Cognitive biases relate to differences in knowledge information deduced from observation tasks. Behavioral biases relate to differences in user activity during these tasks. These two types of biases can lead to differences in interpretation.

In this article, we present a brief overview of the workflow developed in previous work, followed by some results from a user study we conducted on an archaeological use case with this workflow, and we introduce the ongoing work we are pursuing to further explore the field of cognitive archaeology.

# 2 STUDYING ARCHAEOLOGICAL BEHAVIOR IN XR

The approach that we propose to develop considers user behavior through their gaze activity and interactions with the environment. Gaze activity is analyzed using eye-tracking data such as eye fixations and gaze path. The analysis of the user's interaction with the environment is based on metrics about the position or the distance between the observed object and the user's head. In order to be able to compare the behavior of a user performing an archaeological task in real and virtual environments, we established a generic methodology for extracting standardized and comparable behavioral data across different modalities. Subsequently, this workflow was implemented to conduct a user study aimed at identifying the potential impact of the virtual modality on user behavior.

<footnote>*0009-0004-4267-4289</footnote>

<footnote>†0000-0002-9351-2747</footnote>

<footnote>‡0000-0003-4124-8697</footnote>

<footnote>§0000-0003-0098-9456</footnote>

<footnote>¶0000-0002-4762-4342</footnote>
<figure><img src="image_22.png" /><figcaption>Figure 2: A workflow to compare gaze-related behavior across real and virtual modalities (Figure from [4])</figcaption></figure>

## 2.1 Comparison Workflow

Analyzing user behavior can be a complex task, especially when attempting to do so across different modalities and therefore different devices. In addition, comparing 3D gaze data across modalities presents unique challenges. For example, user movement tracking devices in real and virtual environments produce different data logs in different frames of reference, whether for eye tracking or body movement. Standardized methods of data collection and analysis are required to perform comparable analyses. We have therefore developed a comprehensive data preparation and analysis workflow based on four main steps (Fig. 2):

* **Experimental settings:** this step is related to the definition of the protocol, the constraints and the rules necessary to establish a comparison,

* **Acquisition:** this step is related to the description of the required data for processing and analysis and how to generate them,

* **Processing:** this core step is related to the description of the processes to transform the raw acquired data into the standardized format of 3D gaze data, and behavior data,

* **Interpretation:** this step is related to the analytical techniques to be applied to these data sets in order to produce metrics and graphics.

The complete workflow is detailed in [4]. It should be noted that the design of this methodology is sufficiently generic to be able to apply the process of comparing visual and interactive behavior between real and virtual to other application domains.

## 2.2 User Study Across Modalities

We applied this methodology to an archaeological task of observing artifacts. To this end, we set up a user study experiment in which a corpus of archaeological artifacts was analyzed in three modalities (Fig. 1): a real environment, a virtual environment using a head-mounted VR device (HMD), and a virtual environment using a large CAVE-like immersive room. The aim of this experiment was to compare participants’ behavior (head movements, gaze patterns and task performance) between the three modalities in order to define the potential interpretation biases introduced by the use of VR tools in this domain.

It was essential to clearly define the rules for participant interactions to ensure the experiment remained consistent across different modalities. Similarly, the virtual environments replicate the real environment as faithfully as possible, using for example photogrammetry to digitize the real artifacts.

The results, detailed in [3], highlight several issues that need to be considered when using VR solutions in archaeology for analysis

and interpretation. Most importantly, they also indicate a coherence in user behavior and performance across modalities in performing the task under consideration. Participants’ visual exploration is consistent across modalities, with the same amount of information being obtained about them, despite the variability in the characteristics of the artifact within the corpus selected for the experiment. Statistical discrepancies in the user’s movements were identified, but these can also be mitigated by their small amplitude. The analysis of the eye-tracking data did not reveal any differences in the global gaze pattern, as determined by the classification of the exploration of the different regions of interest on the observed objects. However, differences were found in specific oculomotor behaviors, but they still did not affect the global gaze pattern on the object.

In summary, the results suggest that VR tools can be used without bias in the archaeological context of artifact observation tasks, provided that the virtual features are accurately rendered with the correct geometry and scale of the virtual model.

## 3 GOING FURTHER WITH COGNITIVE ARCHAEOLOGY IN XR

The two contributions presented provide a foundation for studying user behavior when carrying out archaeological tasks with XR devices. At the moment we are interested in the reproduction of real tasks in XR. However, XR environments make it possible to go beyond the simple reproduction of reality and perform tasks that would be difficult to perform in real life. To this end, we are now focusing on two specific use cases: the representation of missing parts of artifacts and the correct visualization of hidden elements in archaeological material.

### 3.1 Representing the missing parts of artifacts

Archaeological objects are often incomplete or damaged. When physical copies of these objects are made, the missing parts are often reconstructed in order to restore a complete element and thus more faithfully represent its actual function or appearance for an exploration and manipulation study. However, how this reconstruction is carried out can be a subject of debate: on the one hand, to remain as faithful as possible, even if this means inducing false representations, or, on the other hand, to move away from fidelity so that there is no confusion between the original elements and those that have been added.

This choice is illustrated by the concept of anastylosis (Fig. 3), which corresponds to the reassembly of an existing artifact (or monument) using fragmentary parts added in a recognizable way in order to ensure that conservation is as accurate as possible [16, 13]. While XR environments facilitate the representation of complete reconstructions of artifacts or monuments, they also allow different visualizations of the reconstructed elements to be proposed, as well as the representation of different reconstruction hypotheses. It is particularly interesting because of its ability to encompass all types of artifacts, regardless of their scale or the number of missing
elements. For example, its application can be proposed in the representation of a complete ruin, filling in the missing or empty sections of the building, thus creating the presumed complete structure [22].

The reconstruction process for a representation in an XR environment can be approached in different ways, with greater implications due to the multiple possibilities offered by 3D modeling. For example, it is possible to hide missing parts or, on the contrary, make reconstructed areas more visible. In addition, the visibility of these added elements can be controlled and adapted by the user

[11, 18]. Virtual anastylosis effects represent a valuable tool for experts, as they facilitate the straightforward application and evaluation of diverse reconstruction hypotheses, while also providing a quantitative representation of each hypothesis's reliability.

Existing work already recommends some guidelines for an effective anastylosis effect in an architectural context [16], emphasizing the importance of visually representing the added elements in a way that is clearly distinct from the original parts. In addition, this work points out that it is also important that the modifications made to the digital representation of the original artifacts should maintain a coherence of aesthetic dominants and keep them at the center of the observer's attention. This work also shows that it is preferable to add new elements to the missing parts than to leave the segment empty.

The impact of virtual restoration is influenced by many aspect and choices, such as the luminance values of added parts [16] but also the textures and virtual material, color changes, choice of background or highlight the area. Design choices for the graphical representation of missing parts can thus have a significant impact. We believe that a behavioral user study should provide interesting guidelines for identifying relevant metaphors for virtual restoration. Therefore, this topic will be examined in a future work through this approach. First, we plan to conduct an experiment using reconstructed pottery with anastylosis and the same one in a complete and unaltered version, both in a real and a virtual context. The objective is to better define the impact of the anastylosis effects to establish guidelines for their use in XR applications, thereby expanding the advantages of XR in archaeology. We aimed to apply various types of anasylosis effects in an XR environment, including transparency, border highlighting, color changes, or texture modifications. Our goal is to determine which effect is preferred and why, based on behavioral analysis.

<figure><img src="image_23.png" /><figcaption>Figure 3: Three types of anasylosis used in the experiment: (A) complete; (B) with an empty cavity; (C) with a cavity filled wither lighter stone (Figure from [16])</figcaption></figure>

## 3.2 Representing the hidden parts of artifacts

Another interesting aspect made possible by XR environments is the ability to see through to the inside of archaeological material

to access parts inside an artifact, or to visualize the techniques employed in the fabrication of these objects. To access this hidden content, the most common solution is to use medical imaging techniques, in particular CT visualization (Fig. 4) [6, 8], which is a relevant method due to its efficiency and non-destructive nature in capturing the interior of an object. However, the visualization tools associated with medical imaging require certain skills to be handled correctly, and offer mainly 2D rendering, resulting in a lack of depth perception.

This raises the question of how best to represent these hidden elements in order to avoid confusion with others. Special consideration should be given to depth perception, with the aim of making the exploration of artifacts as efficient and natural as possible. The challenge for archaeologists is to visualize and access new information directly, to improve their interpretation and to prepare an excavation in the best possible way.

For example, simple solutions to clearly examine the inside of the object could be to visualize the object in a semi-transparent way, filtering the value from the scan to show only a part of the whole object, or using a virtual window inside and displaying only the inner elements through this window [19, 17]. Another study presents an experiment evaluating the estimation of the position of some elements inside the archaeological material [6]. The aim is to design and evaluate representations of these hidden elements, and their perception so that they can coexist with directly visible parts with a new effect. This study also underlines that it is important to see both the external and internal structure of the artifact in order to understand the production process or to spatially locate the various elements inside. In addition, according to the study, experts confirmed the effectiveness of the application's spatial perception capabilities, as demonstrated by the ability to accurately identify different depth maps, up to eight elements in a row while judging the relative distances between them.

Consequently, a more precise study of the representation of the internal structure of archaeological material, and the resulting quality of depth perception, is a subject that will benefit from the approach based on the study of user behavior in future work. Analysis of the user's oculomotor and interactive behavior should provide a better understanding of the cognitive process used by the expert in such environments. This approach will be used to improve visualization techniques and facilitate the exploration and non-destructive analysis of archaeological material. It is important to highlight a particular scientific issue linked to this type of representation: medical imaging visualization is based on volume rendering, which is rarely used in XR and even less so in conjunction with eye tracking. We therefore first need to extend our methodology to address this problem, by proposing tools for analyzing gaze in a 3D volumetric environment.

<figure><img src="image_24.png" /><figcaption>Figure 4: Mix of representation of a cremation urn; Top Left: Cremation urn. Top, middle and right: slice views from CT scan. Bottom: Volume rendering of the content of the cremation (Figure from [6])</figcaption></figure>
# 4 CONCLUSION

In summary, the XR has become a relevant tool for archaeologists and cultural heritage experts, facilitating new exploration and manipulation techniques that ensure safer and more comprehensive methods, as well as new forms of interaction and access to information about artifacts. The studies presented focus on the improvement and analysis of specific archaeological use cases involving a virtual environment. In order to validate the relevance of using XR environments for archaeology, we proposed a methodology to discern the differences between real and virtual exploration and the parameters that elucidate these dissimilarities. This methodology, which falls within the field of research known as cognitive archaeology, is based on the study of user behavior when carrying out archaeological tasks. We conducted a user study using this methodology in order to identify potential behavioral biases. The initial results encourage us to take this cognitive archaeology approach further in an XR environment. Moreover, we plan to study two specific use cases: the correct representation of a missing part of an incomplete artifact and the representation of hidden parts of an artifact. The workflow will also be implemented to support the analysis of cognitive behavior and to propose visualization and interaction solutions relevant to these two use cases. This will allow to characterize the effectiveness of anastylosis variants and gain a better understanding of the depth perception in a scanned representation, with a subsequent goal of improving the efficacy of XR tools for archaeology. Both would contribute to a more complete and comprehensive analysis of archaeologists' activities in XR applications.

## FIGURE CREDITS

Figure 2: Workflow Components schema, structured in four blocks: Experiment Settings (purple), Acquisition (green), Processing (blue) and Interpretation (red). Figure from [4].

Figure 3: Three types of anastylosis used in the experiment: (A) complete; (B) with an empty cavity; (C) with a cavity filled with lighter stone. Figure from [16].

Figure 4: Top Left: Cremation urn. Top, middle and right: slice views from CT scan. Bottom: Volume rendering of the content of the cremation. Figure from [6].

## ACKNOWLEDGEMENTS

This work was partially funded by Equipex+ Continuum ANR-21-ESRE-0030 and CNRS MITI AAP Digital Twin AJAX.

## REFERENCES

[1] F. Criado-Boado, D. Alonso-Pablos, M. J. Blanco, Y. Porto, A. Rodríguez-Paz, E. Cabrejas, E. del Barrio-Álvarez, and L. M. Martínez. Coevolution of visual behaviour, the material world and social complexity, depicted by the eye-tracking of archaeological objects in humans. *Scientific Reports*, 9(1):3985, mar 2019. Number: 1 Publisher: Nature Publishing Group. doi: 10.1038/s41598-019-39661-9

[2] F. Criado-Boado, L. M. Martínez, M. J. Blanco, D. Alonso-Pablos, Y. Porto, and E. d. Barrio-Álvarez. Gazed Pottery: an Archaeometric-Cognitive Approach to Material Culture Visuality. *Journal of Archaeological Science*, 154:105770, jun 2023. doi: 10.1016/j.jas.2023.105770

[3] M. Dumonteil, V. Gouranton, M. J.-M. Macé, T. Nicolas, and R. Gaugne. Do we study an archaeological artifact differently in vr and in reality? In *ICAT-EGVE 2024-joint 34th International Conference on Artificial Reality and Telexistence & the 29th Eurographics Symposium on Virtual Environments*, pp. 1–9, 2024.

[4] M. Dumonteil, M. J.-M. Macé, V. Gouranton, T. Nicolas, and R. Gaugne. A method for standardizing eye-tracking and behavioral data in real and virtual environments. In *Proceedings of VISIGRAPP (1): GRAPP*, 2025.

[5] R. Gaugne, J.-B. Barreau, F. Lécuyer, T. Nicolas, J.-M. Normand, and V. Gouranton. Extended reality for cultural heritage. In *Handbook of Cultural Heritage Analysis*, pp. 1405–1437. Springer, 2022.

[6] R. Gaugne, T. Nicolas, Q. Petit, M. Otsuki, and V. Gouranton. Evaluation of a mixed reality based method for archaeological excavation support. In *ICAT-EGVE 2019*, pp. 1–8, 2019.

[7] M. Hostettler, A. Buhlke, C. Drummer, L. Emmenegger, J. Reich, and C. Stäheli. Digital archaeology between hype and reality: The results of a survey on the use of 3d technologies in archaeology. In *The 3 Dimensions of Digitalised Archaeology: State-of-the-Art, Data Management and Current Challenges in Archaeological 3D-Documentation*, pp. 181–203. Springer International Publishing Cham, 2024.

[8] L. M. McKnight, J. E. Adams, A. Chamberlain, S. D. Atherton-Woolham, and R. Bibb. Application of clinical imaging and 3D printing to the identification of anomalies in an ancient Egyptian animal mummy. *Journal of Archaeological Science: Reports*, 3:328–332, Sept. 2015. doi: 10.1016/j.jasrep.2015.06.028

[9] M. A. Mirzaei, J.-R. Chardonnet, and F. Merienne. Comparing real and virtual object manipulation by physiological signals analysis: A first study. 2018.

[10] B. Molloy and M. Milić. Wonderful things? a consideration of 3d modelling of objects in material culture research. *Open Archaeology*, 4(1):97–113, 2018.

[11] T. Nicolas, R. Gaugne, C. Tavernier, E. Millet, R. Bernadet, and V. Gouranton. Lift the veil of the block samples from the warcq chariot burial with 3d digital technologies. In *International Conference on Virtual Systems & Multimedia (VSMM 2018)*, pp. 1–8. IEEE, 2018.

[12] L. Opgenhaffen. Visualizing archaeologists: a reflexive history of visualization practice in archaeology. *Open Archaeology*, 7(1):353–377, 2021.

[13] E. Pietroni and D. Ferdani. Virtual Restoration and Virtual Reconstruction in Cultural Heritage: Terminology, Methodologies, Visual Representation Techniques and Cognitive Models. *Information*, 12(4):167, Apr. 2021. Number: 4 Publisher: Multidisciplinary Digital Publishing Institute. doi: 10.3390/info12040167

[14] D. Psarros, M. I. Stamatopoulos, and C.-N. Anagnostopoulos. Information technology and archaeological excavations: A brief overview. *Scientific Culture*, 8(2), 2022.

[15] M. Rusnak. 2d and 3d representation of objects in architectural and heritage studies: in search of gaze pattern similarities. *Heritage Science*, 10(1):1–18, 2022.

[16] M. Rusnak, A. Brzozowska-Jawornicka, and Z. Koszewicz. How to enhance perception of reassembled but incomplete works of ancient art? eye-tracking study of virtual anastylosis. In *Arts*, vol. 12, p. 15. MDPI, 2023.

[17] G. Schall, E. Mendez, E. Kruijff, E. Veas, S. Junghanns, B. Reitinger, and D. Schmalstieg. Handheld augmented reality for underground infrastructure visualization. *Personal and ubiquitous computing*, 13:281–291, 2009.

[18] S. Shakya et al. Virtual restoration of damaged archeological artifacts obtained from expeditions using 3d visualization. *Journal of Innovative Image Processing (JIIP)*, 1(02):102–110, 2019.

[19] T. Sielhorst, C. Bichlmeier, S. M. Heining, and N. Navab. Depth perception—a major issue in medical ar: evaluation study by twenty surgeons. In *Medical Image Computing and Computer-Assisted Intervention—MICCAI 2006: 9th International Conference, Copenhagen, Denmark, October 1-6, 2006. Proceedings, Part I*, pp. 364–372. Springer, 2006.

[20] M. Silva-Gago, F. Ioannidou, A. Fedato, T. Hodgson, and E. Bruner. Visual attention and cognitive archaeology: an eye-tracking study of palaeolithic stone tools. *Perception*, 51(1):3–24, 2022.

[21] N. Takahashi, T. Inamura, Y. Mizuchi, and Y. Choi. Evaluation of the difference of human behavior between vr and real environments in searching and manipulating objects in a domestic environment. In *2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)*, pp. 454–460. IEEE, 2021.

[22] E. Torun and S. Ercan. Two decades of anastylosis experience at sagalassos. *Exempli Gratia. Sagalassos, Marc Waelkens and interdisciplinary archaeology*, Leuven, pp. 27–41, 2013.
# For a Perception of Monumentality in Eastern Arabia from the Neolithic to the Bronze Age: 3D Reconstruction and Multidimensional Simulations of Monuments and Landscapes

Tiffany Aires Da Cruz*

PhD Student

University of Paris 1 Panthéon-Sorbonne

UMR 8215 Trajectoires. De la sédentarisation à l'Etat

Olivia Munoz*

Researcher (Arabian archaeology, funerary archaeology, bioanthropology)

CNRS

UMR 8215 Trajectoires. De la sédentarisation à l'Etat

François Giligny*

Professor of Archaeological Methodology

University of Paris 1 Panthéon-Sorbonne

UMR 8215 Trajectoires. De la sédentarisation à l'Etat

Valérie Gouranton*

Researcher (Virtual Reality)

Institut national des sciences appliquées de Rennes (INSA Rennes)

UMR 6074 IRISA / Inria

<figure><img src="image_25.png" /><figcaption>Figure 1: Dhabtiyah archaeological site, Saudi Arabia</figcaption></figure>

## ABSTRACT

The monumentality of Neolithic and Early Bronze Age (6th to 3rd millennium BC) structures in the Arabian Peninsula has never been approached through a comprehensive approach of simulations and reconstructions. As a result, its perception remains understudied. By combining archaeological and paleoenvironmental data, 3D reconstruction, 4D simulations, virtual reality and generative AI, this PhD research project proposes to analyse the perception of monuments, exploring their spatial, visual and temporal dimensions, in order to answer to the following question: how can we reconstruct and analyse the perception of monumentality in Eastern Arabia through 4D simulations, and how can the study of this perception influence our understanding of monumentality and territories?

This article presents a work in progress, after three months of research of which one month on the Dhabtiyah archaeological site (Saudi Arabia, Eastern Province).

**Index terms:** Monumentality, Virtual Reality, 3D reconstruction, 4D simulations, Eastern Arabia

## 1 INTRODUCTION

Neolithic and Bronze Age were marked in the Arabian Peninsula by the emergence of monumentalism, a crucial stage in social changes. These large structures seem to have played a role in territorial organisation and spatial orientation, perhaps marking routes of circulation, resources and gathering places [9]. However, in the absence of written documents, studying monumentality and territories requires considering the visual links between monuments, their locations, architecture, lighting and, more generally, their perception within their environmental context.

The 4D simulation of archaeological sites and the use of generative Artificial Intelligence (AI) open up new perspectives for understanding this monumentality. This project focuses on the perception of monumental structures, particularly in Saudi Arabia, through their 3D reconstruction and that of their environments, using archaeological data and digital tools. The multi-dimensional

<footnote>*tiffany.airesdacruz@gmail.com</footnote>

<footnote>*olivia.munoz@cnrs.fr</footnote>

<footnote>*francois.giligny@univ-paris1.fr</footnote>

<footnote>*valerie.gouranton@irisa.fr</footnote>
space also incorporates the fourth dimension, time, through the
observation of environmental changes, light exposure and the
simulation of human activities associated with these monuments.
This involves the development of an innovative methodology that
can be applied to different environmental and cultural contexts.

In order to achieve these goals, this PhD research project aims
to fulfil several objectives:

- Create 3D representations of monumental structures within their reconstructed hypothetical environments, based on excavation and paleoenvironmental data.

- Examine monument architecture to study their impact, especially the influence of materials, on the perception.

- Simulate varying vegetation covers to document their influence on the perception of structures.

- Conduct lighting simulations to understand the role of light in perception.

- Simulate hypotheses of ceremonial processions or deposits when archaeological data are sufficient.

- Study general perception of structures and analyse relationships between visibility, monumentality, and territory through virtual reality immersion.

- Propose an enhancement of archaeological heritage through 3D reconstructions and 4D simulations, via *in situ* (museums) or online supports.

## 2 CONTEXT AND STATE OF ART

The Neolithic and Bronze Age (6th-3rd millennium BCE) are characterized by a wide variety of megalithic structures in both northern and southern Arabia. Interpreted as cenotaphs, sanctuaries, or ceremonial gathering places, these structures reflect the impact of communities on their environment and a strong territorial marking [8]. These structures include long alignments of stones (such as *mustatils* or pendants [12]), trapezoidal or rectangular platforms [7, 9], dolmens [11, 13], and steles and stone circles.

From the Bronze Age (3rd millennium BCE) onwards, tens of thousands of monumental megalithic tombs, shaped like towers or igloos, mark the entire Arabian Peninsula. Often grouped in necropolises, these tombs are located on high points of the landscape or form true "funerary avenues" [2, 8]. They exhibit various morphological types and are sometimes associated with alignments of standing stones (trains), triangular structures, circular enclosures, or stacked stone structures (caissons) [8].

These dry-stone monuments, known but understudied until the 1970s, have only recently become the focus of extensive research, particularly through aerial and satellite surveys. They reflect the increasing complexity of societies and their practices, seemingly indicating a desire to mark territory and legitimize the ancestral authority of communities. To shed light on these and other aspects, attention has now turned to their perception within landscapes, opening new research avenues. This perception will be analysed using 3D reconstruction and 4D simulation methods, integrated in virtual reality, enabling a multidimensional study.

Virtual Reality (VR), as defined by Philippe Fuchs in 2006, provides a sensorimotor and cognitive experience within a virtual environment [4]. It relies on technical and computational devices that simulate and immerse the user in real-time. This definition is based on two key and intrinsically linked concepts: immersion and interaction. Immersion stimulates and isolates the user's senses, particularly vision and hearing, while interaction is based on the user's ability to act and react within the environment. Thus, the use of virtual reality allows for an enhanced perception experience by immersing users within 4D simulations.

However, before it can be fully utilized, it is necessary to begin with an essential step: the 3D modeling of virtual environments. The use of 3D in archaeology dates back to the 1990s. During this

period, it primarily concerned the 3D digitization of
archaeological sites and artifacts. Notable examples include the
3D model of the Lascaux Cave (Périgord) and others projects
such as *Rome Reborn* [3]. While these 3D reconstructions were
often carried out with conservation or promotion goals, research
interest has developed in recent decades, particularly with the
integration of 4D. For instance, the restitution of the awning of the
Theatre of Pompey, which allows for solar simulations, or various
projects involving the study of objects in action, such as the
operation of a Roman ballista, are noteworthy. Nevertheless, 4D
is still used marginally in archaeological research, just like
generative artificial intelligence. In this regard, the work of
Maurizio Forte, which enables the reconstruction of Etruscan
tombs and their environments in images and videos [5], is a
notable example.

VR is used in a differentiated manner, either for promotion,
research, or both. Several projects have been carried out with a
concrete research goal, such as those by Valérie Gouranton and
Jean-Baptiste Barreau, who have also highlighted the value of VR
for archaeological and historical research by focusing on the
professional applications that could be made of it [6, 10]. For
example, in the context of a study on the 18th-century ship Le
Boullongne, virtual reality was used to create a 1:1 scale
immersive reconstruction of this East India Company ship based
on historical data. This approach enabled a better understanding of
life on board and stimulated new reflections [1].

In this PhD research project, the integration of monuments and their environments in virtual reality appears as the most suitable means of studying the perception of monumentality, defined as the way in which monuments were seen, understood, and used by past societies. This involves, for example, the impact of their architectural design, the materials used, lighting, the environment, or their positioning within the landscape on these elements. Perception is thus partly based on what our senses perceive and how it is interpreted by the individual. It is therefore subjective. However, by allowing the researcher to immerse themselves in reconstructed past environments, while isolating them from the real world, virtual reality enables the exploration and analysis of how these structures may have been perceived in an immersive and interactive way, allowing the researcher to get as close as possible to the sensorimotor experience that individuals from the relevant periods may have experienced.

This research is based on methodological foundations drawn
from various studies, particularly in the field of heritage.
However, it stands out due to the combination of different
approaches, notably 4D simulations in virtual reality and
generative artificial intelligence, as well as its object of study:
monumental structures' perception. It also aims to propose a
methodological framework that can be reproduced by the
archaeological community, enabling the exploration of this
complex and subjective concept and opening new perspectives.

## 3 METHODOLOGY

The methodology primarily relies on the analysis of paleoenvironmental and architectural data obtained from archaeological excavations, combined with the use of digital tools, some of which are still relatively underutilized in archaeology. Bibliographic research, discussions with archaeologists and geoarchaeologists, and data collection in the field constitute an essential first step help establish the specifications for the 3D modeling of structures and the environmental reconstructions to be carried out.

Subsequently, the monumental structures will be modeled in 3D
using Blender software, allowing for a more detailed study of
their architecture. With the help of generative AI tools such as
Stable Diffusion, images and videos that inspire improvements to
the 3D models can be generated. This will enable an assessment of the limitations and advantages of this tool for the study of archaeological sites.

Next, several projects will be developed using the Unreal Engine 5 game engine to create virtual simulations. The terrain in these simulations will be based on a Digital Terrain Model (DTM) for each relevant area, ensuring that subsequent geographic information is accurately incorporated. Multiple environmental hypotheses will be reconstructed after discussions with paleoenvironmental specialists. For example, hydrography or vegetation cover may vary depending on the hypotheses and periods. This is crucial for better understanding the impact of changing vegetation cover in Eastern Arabia on the environment and perception. Furthermore, it will help address the lack of paleoenvironmental data for certain sites by proposing several possibilities. Generative AI tools will again be used to inspire plausible environmental reconstructions. At this stage, the 3D models of the structures will be positioned in their locations. By positioning themselves within these reconstructed environments, spatial studies, especially visibility analyses, can be conducted, taking into account the effects of materials (colors), vegetation, and lighting on perception.

The temporal (4D) dimension will be integrated at multiple timescales (short, medium, and long) through simulations of natural light and internal lighting, the day/night cycle, weather conditions, and ceremonial activities (including the integration of virtual humans). For the long timescale, the study will focus on the evolution of necropolises, including the addition of new monuments and changes in vegetation cover over time.

Immersion in virtual reality within these environments will enable an in-depth analysis of perception and the various axes discussed. Perception, inherently subjective, is influenced by complex factors such as vegetation, architecture, employed materials, lighting, and spatial interactions. By providing researchers with the ability to explore a 4D virtual environment, virtual reality allows for a dynamic study of the influences of these elements, thereby opening new avenues for research and interpretation. The system used will be a virtual reality headset. This choice was made considering the low cost and practicality of this device, which can be easily used in archaeology laboratories, universities, and museums, facilitating the sharing of the simulation with others.

The interpretation of the results will be guided by an analytical framework based on existing studies on the concept of monumentality and the perception of monumental structures according to socio-cultural contexts. Spatial relationships, visual perception, or temporal evolution will serve as key analytical criteria. This framework will be refined throughout the research, based on the initial results obtained.

## 4 DHABTIYAH (SAUDI ARABIA): A MULTIMILLENNIAL MONUMENTAL SITE

The Dhabtiyah site is an exceptional case study for furthering our understanding of long-term monumentality in Saudi Arabia. Located in the Eastern Province of Saudi Arabia, near Hanidh, it is a diachronic site with remains dating from the 6th millennium BCE to the 1st millennium CE. Positioned on rocky terraces, it overlooks a *sebkha*. Various types of structures have been observed, including many presumed tombs identified through satellite, aerial, and ground surveys in 2021 during the Franco-Saudi mission at Thaj.

The site is now being explored as part of a French-Saudi cooperation between the CNRS and the Saudi Heritage Commission, through the Dhabtiyah Archaeological Mission of the French Ministry for Europe and Foreign Affairs. It is also

integrated into the ANR Arabiancairns project (#ANR-22-CE27-0001), which focuses on monumental protohistoric tombs in the Arabian Peninsula. Currently, over a thousand dry-stone structures of various types have been identified, including:

- *Mustatils*: These rectangular structures are formed by two long, more or less parallel walls, connected by two shorter walls. They feature a platform at one end, which may include niches where deposits have been observed.

- Circular cairns/tower tombs: Various types and sizes of cairns are found in large numbers on the site. They may be associated with circular, triangular, trapezoidal, or rectangular enclosures or platforms. They feature a corbelled roof chamber surrounded by several structured concentric walls and an external facing.

- Several elongated “cuneiform” structures are also present on the site.

While the *mustatils* are dated to the Neolithic, most of the other structures are attributed to the 3rd and 2nd millennia BCE, with possible re-uses until the modern times.

The 3D modelling of these structures and their repositioning in their environmental context, reconstituted thanks to the geoarchaeological data collected during the mission, will enable us to analyse their perception, in particular by modelling several hypotheses for environments and lighting and by simulating various scenarios of use.

The first excavation campaign at Dhabtiyah, from November to December 2024, allowed for the excavation of four structures (two circular cairns, one *mustatil* and one cairn with sub trapezoidal enclosure). Two tombs and one cuneiform structure had also been excavated in 2021. Additionally, sampling for ¹⁴C dating has been conducted on all the excavated structures and on several surveyed tombs.

## 5 MODELING OF STRUCTURES THROUGH PHOTOGRAMMETRY

15 structures were modelled in 3D using photogrammetry scanning, some excavated and others not. The acquisition was carried out using a drone (for the excavated structures) or a camera mounted on a pole. The 3D models created have various benefits for this project.

They allow for 3D documentation of all the observed types of structures, especially cairns of varying sizes, more or less collapsed, some looted ones, and sometimes featuring slabs still in place.

Scaling the models will enable measurements of the stones and provide dimensions of the modules used. This will facilitate the modelling of typical stones that can be added to the 3D models created via photogrammetry to restore architectural elements, as well as aid in the full modelling of certain structures, supported by data from the excavations. Several modules of slabs were also modelled by photogrammetry. These can serve as models or bases for the modelling of the slabs of circular cairns, with textures that can be extracted.

The external appearance of these structures, now recorded, which will help determine the textures to use to best represent the materials employed, as well as how they may have appeared following natural or anthropogenic degradation (looting, sand accumulation, collapse). Indeed, since the necropolis spans millennia, newly built structures must have coexisted with older and more or less degraded or looted ones.

Thus, by cross-referencing excavation data of this site and similar sites, photogrammetry models, and the information they provide (dimensions, texture, modules), we will be able to model structures of different types and varying degrees of degradation and add them to the 3D and 4D reconstruction of the necropolis. While hypothetical, these reconstructions will be informed by
multiple data sources to reduce the margin of error as much as
possible.

Plants observed on-site were also digitized. A palaeobotanical
study will determine whether their integration into the
reconstructed environment is relevant.

We are still considering whether to directly integrate the
photogrammetry 3D models into the hypothetical reconstructed
environments. From a technical perspective, we still need to
determine whether it is more practical to remodel the structures
directly using the information gathered, allowing for better control
over mesh optimization and texture consistency, or to integrate
them as they are.

All the pictures taken, whether for photogrammetry modelling
or not, will be used to feed the image database provided to the
generative AI, Stable Diffusion, to create images offering
reconstructions of the structures and environments.

# 6 FUTURE DIRECTIONS AND DEVELOPMENTS OF THE RESEARCH

The next steps of this PhD research project will be to model in 3D
several structures of the Dhabtiyah site from the photogrammetry
3D models (enabling to obtain measurements, textures, examples
of degradation, etc.). These models will be able to be integrated
into a hypothetical environment created in Unreal Engine 5, using
geoarchaeological data and a DEM.

Afterward, a first trial in virtual reality can be conducted. Early
testing of the 3D environment in virtual reality will ensure its
proper functionality within the first year of research, allowing for
the optimization of its various components, and adjustments
where necessary. By incorporating temporal and environmental
changes and enabling movement and interactions, 4D and VR go
beyond the limitations of static approaches. They will allow for
dynamic and interactive analyses of monumental structures,
providing a unique way to study key aspects of the monuments
that could influence their perception within landscapes (e.g., light,
materials, etc.). This experimental method, through immersion,
should further engage the researcher in the virtual environment
and in simulations of the structures' use. This will allow us to
determine whether the use of VR enhances the perception
experience and thus facilitates its study.

# 7 CONCLUSION

This research aims to provide a new understanding of
monumentality in Eastern Arabia from the Neolithic to the Bronze
Age, through 3D reconstructions and 4D simulations integrated in
virtual reality. The results will shed light the role of these
structures as territorial markers, particularly through visual
relationships and the impact of materials, architectures, and
environments on visibility. They will also reveal how individuals
perceived these monuments and the potential objectives behind
certain choices (e.g., architecture, positioning). The creation of
multiple environmental reconstructions will allow to observe how
environmental changes influenced the visual perception of
monuments. Simulations of ceremonial processions and deposits
will add another layer to the study of how individuals perceived
monuments within ritual contexts at the time of their use.

Thus, this PhD research project relies on an innovative
methodology to study monumentality and its perception within
landscapes, offering immersive virtual reality reconstructions that
are unprecedented in the study of the Neolithic and Bronze Age,
allowing for the overcoming of the limitations of traditional
methods of studying monumentality.

FIGURE CREDITS

Figure 1 image credit: Dhabtiyah Archaeological Mission / Kaïna Rointru, 2024.

ACKNOWLEDGMENTS

The French Archaeological Mission to Dhabtiyah received
support from the Excavations Commission of the French Ministry
of Foreign Affairs, the Heritage Commision of the Saudi
Ministry of Culture, the French Embassy in Riyadh, the French
Center for Research on the Arabian Peninsula (CEFREPA), the
CNRS (UMR 8215 Trajectoires), the Louvre Museum
(Department of Near Eastern Antiquities), and the French
National Agency for Research (ANR, Arabiancairns project,
#ANR-22-CE27-0001). Heartfelt thanks are extended to the
Heritage Commission of the Saudi Ministry of Culture for their
steadfast logistical and institutional support.

We would also like to thank the Jaussen & Savignac program
for its support, which has been invaluable in conducting the
doctoral research underlying this article.

REFERENCES

[1] J.-B. Barreau, F. Nouviale, R. Gaugne, Y. Bernard, S. Linares, and V. Gouranton. An Immersive Virtual Sailing on the 18th-Century Ship Le Boullongne. Presence: Teleoperators and Virtual Environments, 24(3): 201-219, 2015.

[2] M. Dalton, J. McMahon, M.A. Kennedy, R. Repper, S.E. Al Shilali, Y. AlBalawi, D.D. Boyer, and H. Thomas. The Middle Holocene 'funerary avenues' of north-west Arabia. The Holocene, 32(3): 183-199, 2022.

[3] Flyover Zone. Rome Reborn: Flight Over Rome [online resource], 2024. Available at: https://www.flyoverzone.com/rome-reborn-flight-over-rome/.

[4] P. Fuchs, G. Morceau, and B. Arnaldi. Le traité de la réalité virtuelle Volume 4 – Les applications de la réalité virtuelle. Mathématique et informatique. Presse des Mines, 3, 2006.

[5] M. Forte. Digital AI for AI: Artificial Intelligence for Interpretative Archaeology. In A. Guidazzoli and M. C. Liguori (dir.). AI and Cultural Heritage. Between Research and Creativity, pp. 43-56. Bologne, Cineca, 2023. doi: 10.1388/workshop-AICH-01.

[6] R. Gaugne, J.-B. Barreau, F. Cousseau, and V. Gouranton. Large-scale immersive reconstitution of a Neolithic corbel dome. International Symposium on Virtual Reality, Archaeology and Cultural Heritage, pp. 1-8. Brighton, United Kingdom, Nov. 2012.

[7] J. McCorriston, M. Harrower, L. Martin, and E. Oches. Cattle Cults of the Arabian Neolithic and Early Territorial Societies. American Anthropologist, 114(1): 45-63, 2012.

[8] O. Munoz. Protohistoric cairns and tower tombs in South-Eastern Arabia (end of the 4th - beginning of the 3rd millennium BCE). In L. Laporte, J.-M. Large, L. Nespoulous, C. Scarre, et T. Steimer-Herbet (dir.). Megaliths of the world, pp. 905-959. Archaeopress, Oxford, United Kingdom, 2022.

[9] O. Munoz, M. Cotty, G. Charloux, C. Bouchaud, H. Monchot, C. Marquaire, A. Zazzo, R. Crassard, O. Brunet, V. Boschloos, and T. al-Malki. Marking the sacral landscape of a north Arabian oasis: a sixth-millennium BC monumental stone platform and surrounding burials. Antiquity, 94(375): 601-621, 2020.

[10] M.-A. Paradis, T. Nicolas, R. Gaugne, J.-B. Barreau, R. Auger, and V. Gouranton. Making virtual archaeology great again (without scientific compromise). 27th CIPA International Symposium "Documenting the past for a better future", pp. 879-886. Ávila, Spain, Sep. 2019.

[11] T. Steimer-Herbet (dir.). Standing stones, anthropomorphic stelae and dolmens. Oxford, Archaeopress (BAR international series 2317), 210 p., 2011.

[12] H. Thomas, M. A. Kennedy, M. Dalton, J. McMahon, D. Boyer, and R. Repper. The mustatils: cult and monumentality in Neolithic north-western Arabia. Antiquity, 95(381): 605-626, 2021. doi: 10.15184/aqy.2021.51.

[13] J. Zarins, M. Ibrahim, D.T. Potts, and C. Edens. Saudi Arabian Archaeological Reconnaissance 1978. The Preliminary Report on the Third Phase of the Comprehensive Archaeological Survey Program - The Central Province. Atlas, 3: 9-42, 1979.
<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

IEEE VR

SAINT-MALO, FRANCE

March 8-12, 2025

<figure><img src="image_28.png" /></figure>

Session Case study

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>
# Usability Testing of VR Reconstructions for Museums and Heritage Sites:
A Case Study from 14th Century Chester (UK)

Dr Helen V Southall *

University of Chester, UK

<figure><img src="image_33.png" /><figcaption>Figure 1: VR representation of the interior of St. John's Church, Chester, in the 14th Century, with pilgrim figure</figcaption></figure>

## ABSTRACT

This paper reports research on the usability of a 3D Virtual Reality (VR) model of the interior of St. John's Church, Chester, as it probably appeared in the 14th Century. A VR visualization was created in Unity, based on archive data and historical records. This was adapted for use with Oculus Quest 2 VR headsets. Participants took part in usability tests of the experience, providing both qualitative and quantitative usability data. Although created with modest time and financial resources, the experience received a good overall usability rating, and numerous positive comments, including from novice VR users. Negative comments mainly related to the experience of wearing a VR headset. This paper concludes by suggesting further work, with thoughts on highly immersive VR in heritage contexts, especially combined with recent developments in generative artificial intelligence.

**Index Terms:** Usability, virtual reality, digital heritage, museum collections.

## 1 INTRODUCTION

This project originated in an AHRC-funded network project on The Mobility of Objects Across Boundaries: 1000 – 1700 (MOB), which brought items from West Cheshire Museums’ collection of medieval artefacts to a wider audience, culminating in public exhibitions. [6, 29, 30, 4] A goal of the both the MOB project and this

work was to explore how digital technologies such as virtual reality (VR) might be exploited by small and regional museums and heritage sites, especially those with restricted resources.

For the MOB project, 3D scans were made of selected 14th century artefacts from the physical exhibition, such as a church floor tile and a pilgrim brooch. These scans were incorporated into VR visualisations of two key locations in 14th century Chester: the medieval port, and St. John's Church (which at that time was the city's cathedral). The church has changed in appearance substantially since that time (e.g., sections have collapsed and been rebuilt, and the interior decoration has changed), so the model was based on local archives and other historical sources.[15, 13, 3] These visualisations (e.g. Fig. 1), were shown as non-interactive video elements of the MOB exhibitions at the Grosvenor Museum and Chester Cathedral, and also made available online [5, 8].

The 3D model of St. John's Church has been reworked for this project, making it suitable for display on an untethered VR headset. The new version was used for usability testing sessions, to establish what aspects of the VR experience were usable and enjoyable, and identify aspects causing concerns or difficulties for users.

## 2 BACKGROUND

### 2.1 XR in Digital Heritage

Museums, heritage attractions and researchers have made use of digital extended reality (XR), including VR, for several decades [16, 31, 26, 25, 19, 12, 10, 14]. Initially this tended to be restricted to specialist applications and/or larger institutions. Smaller heritage sites and regional museums often hold unique collections, but resource constraints limit their ability to use XR to bring these collections to wider audiences [27].

<footnote>*h.southall@chester.ac.uk</footnote>
## 2.2 Applying Usability Testing to VR

### 2.2.1 Development of Usability Testing Methodologies

Usability testing has roots in industrial design, e.g., the design of safety critical user interfaces in aircraft cockpits and nuclear power stations [24, 21]. The purpose of usability testing is not to check that a device or application is technically capable of doing its job, but to check that human users will be able to use it easily and effectively. Most usability testing methods emphasise prototype testing at an early stage, to find usability problems before committing resources to a final version [20, 17]. This is very important where resources for the project are limited. A badly designed interface can cause users to interact with the application reluctantly and with difficulty, if at all. An application whose poor usability acts as a barrier, preventing some or all of the target audience from accessing its content, is clearly undesirable [28].

### 2.2.2 The System Usability Scale

Much of the data generated by usability testing is qualitative, concerning the thoughts and feelings of participants. However, it is useful to have a quantitative measure as well, to allow comparisons. The System Usability Scale [9] was developed for this purpose in 1986, and has been widely used ever since. It gives a comparative score based on a tested methodology with a long history of use in industry. For this study, the SUS has been augmented by qualitative free text answers, to give a holistic view of the usability of the prototype system.

## 3 METHOD

### 3.1 VR Model of St. John's Church, Chester in the 14th Century

With modest extra funding, the original 3D model of the medieval St. John's church created for the MOB project was adapted for interactive use with VR headsets. (This saved development time, but led to some problems, as described in Results, below.) The model's file size was reduced, due to VR headset storage constraints, and interactive functions were introduced, such as the ability to move to different locations, and items (such as a goblet and a bible) which could be 'picked up' using hand controllers. Interaction design was based on standard VR game controller conventions. Human characters were retained from the original model, including a 'pilgrim' wearing items scanned from the Grosvenor Museum's collection. Due to time constraints, the pilgrim is not an interactive character. (There is no technical reason why interactivity could not be added in the future.)

#### 3.1.1 Hardware and software

Both versions of the VR model were created using Unity video game development software. The model was edited to run independently as an interactive app on the Oculus Quest 2 VR headset, and specific sound effects (such as the pilgrim character's footsteps) were added to the soundtrack. The use of hardware to add smells linked to specific locations in the model [2] was explored, but could not be made to work reliably in the development time covered by project funds, so was not included in the usability testing session.

#### 3.1.2 Participants

41 participants tested the VR experience. 21 were invited to take part in a session at the Grosvenor Museum. The other 20 were members of the public who attended a public testing session at Chester Cathedral. Backgrounds (where known) included museum curators and volunteers, staff at Chester Cathedral, computer science specialists, tour guides, local history enthusiasts, school teachers, and retail and hospitality workers. Participants were not asked for detailed demographic information, but from observation the age range ran from college age to older retirees. This was a pilot study,

so the results are not intended or expected to give a representative view of a particular population. Nonetheless, the number and range of participants involved suggests that sufficient data has been collected able to identify the most serious usability problems with this prototype experience [7, 23].

#### 3.1.3 Testing procedure

The VR experience was loaded onto the headsets by the experimenters before the headsets were given to participants. Each participant started by reading and completing the necessary paperwork. They were given a headset, and told that it was running the VR St. John's Church experience. They were also given a pair of hand controllers, shown how to hold and use them, and told that these would be needed to interact with the experience. They were then invited to put on the headset and explore the experience. Detailed instructions were not given, as the goal of the test was to find out how usable the experience would be without an instructor present. The researchers were on hand to help if required (e.g. rebooting the headset, or providing a chair).

#### 3.1.4 Data collection and analysis

Once each participant had finished testing the VR experience (which took between 5 and 20 minutes), they returned the VR equipment, and were given a questionnaire to complete. This had two sections. The first was the System Usability Scale questionnaire, for a quantitative measure of usability. The second section asked qualitative questions about the user's experience of the interactive VR model. The SUS data was processed using the normal formula for the questionnaire [18]. Themes were extracted from the free text answers by looking for the most frequently mentioned topics, as well as unexpected or unusual ideas.

## 4 RESULTS

### 4.1 System Usability Scale Results

The average SUS score for this system was 76.2, pushing it firmly in the 'quite good, but with room for improvement' range. This is what we were expecting, given that the VR experience was a high quality visualisation of St. John's Church, but was not originally intended for a VR headset environment. It was also very encouraging, given that participants received minimal instruction on how to interact with the system.

### 4.2 Qualitative Feedback - General

Response rates to the free text questions were good, with almost all participants answering at least some of the available questions. There was strong consensus on some topics, and more of a variety of opinions on others. There was agreement that VR experiences such as this are novel, enjoyable and engaging, and particularly suitable for persuading students of all ages to engage with key topics, aiding their learning. Participants felt that the experience would give visitors and students a good visual understanding of what life and buildings were like in earlier times, bringing the past into view. Some participants liked how VR allows exploration of historic or physically inaccessible structures, while others commented on opportunities for heritage interpretation (including enhanced live tours), and the potential for researchers to discover new meanings from new perspectives. Positive comments included: 'A great initiative', and 'Just make it bigger! Loved it!'. There was more diversity of opinion regarding possible improvements to the system. The issues that arose can broadly be divided into two categories: the design and implementation of this VR experience, and issues relating to VR headsets and controllers more generally.

### 4.3 Design & Implementation Issues

As mentioned above, the VR model used for these tests was adapted from an earlier version. Changes required included reducing the file
size, by concentrating only on the interior of the church, and adding
interactive elements, such as the ability for the user to move within
the model, and artefacts which could be picked up and inspected.
Also, boundaries had to be added to the model to stop users
moving through solid objects; it became clear that not all were
complete, as some users found places where they were able to
'walk through walls' and out of the cathedral. More common
suggestions related to how much information users would need
before using the system. These comments were very varied,
ranging from 'a member of staff would be needed to assist users at
all times', to saying that the controls were 'surprisingly simple
and easy to navigate', and 'with clear instructions and basic
guidance, most people would be able to use [the system]'. Other
suggestions related to ideas for future embellishments, such as
the ability to call up information on a specific object or area, an in-
vision reminder of the date / period, the addition of a narrative to
encourage engagement, a view of the church as it was compared
with how it is now, or the ability to move outside and see the
surroundings. A small number of participants commented on the
non-player character of the 'pilgrim', who is inside the church at
the same time as the player, but in the current version does not
interact with them. One participant asked what users would expect
in terms of interactions with such characters. An- other stated
that they expected the character to acknowledge them, or maybe
even talk to them.

## 4.4 VR Technology Issues

There were numerous comments on the controls for location and
movement, many relating to known issues with VR headset expe-
riences, such as motion sickness caused by the VR environment
moving when the head is still. We chose to require the users to
move their heads to look around the church. Despite this, several
commented that they felt 'wobbly' standing up with the headset
on, and others said they would prefer not to have to move their
heads. (Seats were available for those who wanted them.) One
participant commented that the VR experience was 'slightly
jumpy', and several mentioned that they would have liked to have
been able to see things (such as wall paintings) in greater detail,
or that they would have preferred higher definition, or weren't sure
how clear the images were supposed to be. These issues probably
relate to the relatively limited visual display quality of the
untethered Quest 2 headsets, which store the entire VR model in
the headset, as well as carrying out all the necessary processing on
board. More serious issues mentioned by some participants relate
to things which severely limited their enjoyment of the VR
experience, or made it difficult for them to participate at all. One
was that some participants found the headsets uncomfortably hot
to wear: one mentioned that they stopped earlier than they would
have done because of this. Both testing sessions were done
during the summer, but neither room was uncomfortably hot
without a headset on. The other issue was wearing spectacles
while using the headset. One person commented that she needed
her glasses, but feared the headset would break them. Several
found the headsets difficult to use with glasses, even with the
adapters supplied.

## 4.5 Practical Issues

As well as the data derived from the questionnaires, lessons were
also learned from the process of running the tests. For instance, it
was clear that many users worked out how to navigate the headset
quickly on their own, but some needed extra help. More instruc-
tions within the VR world might have helped with this issue, but
not with problems with the headset itself (e.g wearing with specta-
cles). There were also issues with the length of time required for
the headsets to reboot between uses, possibly due to the file size of
the VR model.

# 5 DISCUSSION

Overall, usability testing proved to be very worthwhile, providing
feedback both on the current usability of the system, and on ways
in which it might be improved. These will be considered when
planning future developments in detail. (For instance, more time
would be required if integrating new technologies such as the Cilia
Scent kit, and care would need to be given to testing 'recycled' VR
models for issues such as gaps in walls.) The feedback also gives
clues as to what situations VR headsets might be best suited for, and
where other implementations of VR - such as web-based or room-
sized CAVE facilities - might be more appropriate [11]. Generally,
users with no prior experience of VR headsets, games controllers
or games control conventions are likely to need more help interact-
ing with a VR experience than people who are familiar with these.
Help might be provided by human helpers, or (less expensively) by
instructions in the VR experience. However, if instructions are in-
cluded in the system, experienced users are likely to want to bypass
them. Where VR headsets are used, care needs to be taken that they
are made as accessible as possible. In particular, spectacle wear-
ers and people with balance or mobility issues, or who are prone
to motion sickness [22], are more likely to have problems experi-
encing VR via a headset. Room-sized CAVE VR has advantages in
not requiring numerous expensive and relatively delicate headsets,
and avoiding safety issues (e.g. headset-wearers bumping into each
other; motion sickness). It also mitigates issues such as overheat-
ing and claustrophobia, and no special arrangements are needed for
spectacle-wearers. This may be why they are already in use by both
heritage and entertainment venues wanting to make use of 3D VR
with minimal inconvenience. However, such implementations are
often aimed at groups of visitors rather than individuals, and have
relatively low levels of interactivity. Future projects will therefore
investigate an optimal balance between individualised interactivity,
accessibility, cost, and comfort.

## 6 CONCLUSIONS

This project has demonstrated that it is possible to use computer
games development technology to produce a 3D VR model of a her-
itage site, and run productive usability testing sessions, with modest
funding and staff resources. Such a model can be used as the basis
for both a static museum exhibit, and as an interactive VR expe-
rience. Usability tests of the VR headset experience showed good
overall usability, and provided useful qualitative and quantitative
feedback. Based on these results, further work is being planned,
including collaboration with larger heritage sites in the Chester re-
gion whose staff are interested in adding VR experiences to their
public offerings. In addition, the development and testing for this
project has sparked thinking on the following related topics: the
role of 'real' and imagined data in heritage XR, and the potential
applications and impact of artificial intelligence (AI).

### 6.1 Real & Imagined Data in Heritage XR

3D VR models can be extremely realistic and immersive, and there-
fore highly convincing. While this can be an advantage, in heritage
applications there is also a risk that users could take what they see in
VR as a 'true' representation of how things were, even where there
is really uncertainty. It is therefore important to make clear which
parts of the model are effectively 'artist's impressions', based on
interpretation and interpolation, rather than on solid artefacts and
scholarship.

### 6.2 Potential Applications & Impact of AI

Generative Artificial Intelligence (Gen AI) can produce images
based on text and image prompts with minimal user input. AI im-
ages which look superficially convincing, but contain mistakes no
knowledgeable human illustrator would make, are already a cliché.
This is a result of how generative AI works, by extrapolating from
existing available data. While this does not mean that AI should not be used at all in heritage XR, it does suggest that great care must be taken, especially in academic and educational settings, and even more so when combined with the convincing, immersive technologies of extended reality.

## FIGURE CREDITS

Figure 1 is a screenshot from the St. John's Church VR video created for the MOB project [8, 1].

## ACKNOWLEDGEMENTS

The author wishes to thank Dr. Katherine Wilson, Dr. Elizabeth Montgomery, and Martin Moss. This work was supported in part by a grant from the Arts Humanities Research Council.

## REFERENCES

[1] Dextra visual. https://www.dextravisual.com. Accessed 29th December 2024.

[2] Haptics solution. https://hapticsol.com/cilia/cilia. Accessed 29th December 2024.

[3] Historic England Listing. https://historicengland.org.uk/listing/the-list/list-entry/1375977?section=official-list-entry. Accessed 29th December 2024.

[4] Mobility of objects across boundaries. https://mob.chester.ac.uk. Accessed 29th December 2024.

[5] Place objects in past surroundings using augmented reality. https://mob.chester.ac.uk/activities/place-objects-in-past-surroundings-using-augmented-reality/. Accessed 29th December 2024.

[6] West Cheshire Museums Collections. https://westcheshiremuseums.co.uk/collections/. Accessed 29th December 2024.

[7] Why you only need to test with 5 users. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/. Accessed 24th July 2018.

[8] YouTube video on mobility of objects. https://youtu.be/R8bNbSrotgw?si=BS-KRSKDEjCw58G&t=119. Accessed 29th December 2024.

[9] J. Brooke. SUS - a quick and dirty usability scale. In P. Jordan, B. Thomas, B. Weerdmeester, and I. McClelland, eds., Usability Evaluation in Industry. Taylor & Francis, 1996.

[10] D. Buhalis. eTourism: Information technology for strategic tourism management. Prentice Hall, 2003.

[11] B. Cassidy, G. Sim, D. Robinson, and D. Gandy. A virtual reality platform for analysing remote archaeological sites. Interacting With Computers, 31(2):167–176, 2019.

[12] E. Ch'ng. Experiential archaeology: Is virtual time travel possible? Journal of Cultural Heritage, 10:458–470, 2009.

[13] CJH. St. John's Church Ruins, Chester. Technical report, Ramboll, 2020.

[14] G. Clough. Best of Both Worlds: Museums, Libraries and Archives in the Digital Age. Smithsonian Institution, 2013.

[15] J. Coley. University of Chester. n.d.

[16] B. De Carolis, C. Gena, T. Kuflik, and J. Lanir. Editorial: Special issue on advanced interfaces for cultural heritage. International Journal of Human-Computer Studies, 114:1–2, 2018.

[17] G. Fitzgerald and N. Russo. The turnaround of the London ambulance service computer-aided despatch system (lascad). European Journal of Information Systems, 14(3):244–257, 2005.

[18] J. Guerci. Easily calculate SUS score. https://medium.com/@guerci/sus-score-calculator. Accessed 29th Dec. 2024.

[19] D. Guttentag. Virtual reality: Applications and implications for tourism. Tourism Management, 31:637–651, 2010.

[20] J. Knapp, J. Zeratsky, and B. Kowitz. Sprint: How To Solve Big Problems and Test New Ideas in Just Five Days. Bantam Press, 2016.

[21] S. Krug. Don't Make Me Think, Revisited. Pearson Education, 2013.

[22] B. Mason. Virtual reality has a motion sickness problem. Society for Science and the Public, 2017.

[23] J. Nielsen and T. Landauer. A mathematical model of the finding of usability problems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 206–213. ACM, 1993.

[24] D. Norman. The Design of Everyday Things: Revised and Expanded Edition. Basic Books, 2013.

[25] M. Portman, D. Natapov, and D. Fisher-Gewirtzman. To go where no man has gone before: Virtual reality in architecture, landscape architecture and environmental planning. Computers, Environment and Urban Systems, 54:376–384, 2015.

[26] C. Power, A. Lewis, H. Petrie, K. Green, J. Richards, M. Eramian, B. Chan, E. Walia, I. Sijaranamual, and M. de Rijke. Improving archaeologists' online archive experiences through user-centered design. Journal on Computing and Cultural Heritage, 10(1):1–20, 2017.

[27] J. Russick and J. Ludden. Digital transformation: It's a process and you can start now. In G. Anderson, ed., Reinventing the Museum: Relevance, Inclusion, and Global Responsibilities, pp. 338–348. 2023.

[28] H. Southall, M. Marmion, and A. Davies. Adapting Jake Knapp's design sprint approach for AR/VR applications in digital heritage. In Tom Diek and T. Jung, eds., Augmented Reality and Virtual Reality: The Power of AR and VR for Business, pp. 59–70. Springer Nature, 2019.

[29] K. Wilson, M. Bird, and D. Egan-Simon. Handling objects in primary and secondary educational settings facilitating educational processes and challenging heteronormative gender constructs: The case study of mobility of objects across boundaries 1000–1700. In C. Antenhofer and U. Leitner, eds., Geschlecht und Materialität Historische Perspekven auf Erziehung, Bildung und Sozialisation von der Anske bis zur Gegenwart, pp. 33–50. Transcript Verlag, 2024.

[30] K. Wilson and L. Clark. Mobility of objects across boundaries 1000–1700. In Mobility of Objects Across Boundaries 1000–1700. Liverpool University Press, 1st ed., 2022.

[31] G. Younes, R. Kahil, M. Jallad, D. Asmar, I. Elhajj, G. Turkiyyah, and H. Al-Harthy. Virtual and augmented reality for rich interaction with cultural heritage sites: A case study from the Roman Theater at Byblos. Digital Applications in Archaeology and Cultural Heritage, 5:1–9, 2017.
# Exploring Written Artefacts in Virtual Reality –
Potential and Challenges for Creating Immersive Tools and Applications

Jenny Gabel*

University of Hamburg

Ann Lauren Osthof†

University of Hamburg

Christof Berns‡

University of Hamburg

Jost Eickmeyer§

State Library Eutin

Kaja Harter-Uibopuu¶

University of Hamburg

Johann Anselm Steiger /¹

University of Hamburg

Frank Steinicke**

University of Hamburg

<figure><img src="image_34.png" /><figcaption>Figure 1: Our interactive VR applications with markers indicating the location of all Written Artefacts. Left: scenario (a) ancient theatre of Miletus, right: interior of the church in Lucklum</figcaption></figure>

## Abstract

In this work, we introduce two interactive virtual reality (VR) applications, so-called Immersive Inscribed Spaces, for researching Written Artefacts in our interdisciplinary projects. We discuss the potential of using VR to access historical cultural heritage sites and how the immersive experience facilitates individual perception of Written Artefacts. For this novel context, we explore potential use cases and present our implemented features for the VR applications. Furthermore, we examine two major challenges: (1) Spatial interaction and (2) visualization of spatial information. Finally, we describe how we integrated improvements into our applications, based on our research results.

**Index Terms:** Human-computer interaction, classical archaeology, ancient history, epigraphy, theology, inscriptions, cultural heritage, virtual reality, interactive systems

## 1 Introduction

Modern standalone head-mounted displays (HMDs) greatly facilitate the setup and usage of HMDs for a wider audience and in new areas of research. One of which is

the research on so-called *Written Artefacts* – a cover term for natural or artificial objects bearing visual signs applied or formed by humans, regardless of their material composition (materiality). Written Artefacts are an essential part of our collective cultural heritage and include, among others, written manuscripts, inscriptions, pictorials (drawings/illustrations), and ancient graffiti [1, 15]. In our previous work, we applied Virtual Reality (VR) technology for research in two interdisciplinary projects involving researchers from classic archaeology, ancient history, theology, literary and art history, and human-computer interaction [5, 6]. Both projects were carried out within the Cluster of Excellence "Understanding Written Artefacts" at the University of Hamburg. We created realistic, true-to-scale 3D VR representations with high-resolution textures of two real-world locations with Written Artefacts. These *Immersive Inscribed Spaces* spaces can be freely explored and their Written Artefacts including additional information can be viewed in their spatial context. We also added 3D reconstructions of former structures and developed various features which encourage users to gain new perspectives and raise new research questions on the Written Artefacts and their spatial constructions. Within our interdisciplinary collaboration, we defined use cases and explored innovative use cases for VR in the context of researching Written Artefacts that we will discuss in this paper. However, these new application scenarios come with their specific requirements and issues for interaction in VR. During our work, we identified two major interaction challenges for our *Immersive Inscribed Spaces* that we addressed and investigated in several user studies. This paper presents how we integrated the findings into our *Immersive Inscribed Spaces*.

<footnote>*e-mail: jenny.gabel@uni-hamburg.de</footnote>

<footnote>†e-mail: ann.lauren.osthof@uni-hamburg.de</footnote>

<footnote>‡e-mail: christof.berns@uni-hamburg.de</footnote>

<footnote>§e-mail: j.eickmeyer@lb-eutin.de</footnote>

<footnote>¶e-mail: kaja.harter@uni-hamburg.de</footnote>

<footnote>e-mail: anselm.steiger@uni-hamburg.de</footnote>

<footnote>**e-mail: frank.steinicke@uni-hamburg.de</footnote>
## 2 Immersive Inscribed Spaces

In our projects we worked on two inscribed spaces with contrasting properties as pictured in Figure 1:

(a) The ancient Greco(Greek)-Roman theatre of Miletus (Turkey). It is one of the largest preserved ancient theatres, which is a large-scale open space with over 300 (Ancient Greek) Written Artefacts carved in marble stone.

(b) The interior of the early modern European church in Lucklum (Germany) is an enclosed interior space with 209 Latin inscriptions and 156 emblematic paintings (monochromatic paintings using the European *Grisaille* technique) representing biblical themes.

To summarise, both spaces contain a large amount of Written Artefacts from entirely different time periods and with different materiality. Regardless of these differences, both scenarios can benefit equally from the immersive experience of the space and the Written Artefacts. The use of VR for investigating Written Artefacts is independent of the geography, materiality, time period, and architecture of the inscribed space [2]. The *Immersive Inscribed Spaces* that we created allow the humanities researchers in our projects to investigate diverse, multifaceted questions. They support the researchers in gaining historical insights and investigating new questions based on the immersive, spatial perception of the Written Artefacts that are inaccessible in the real-world locations. Our previous evaluations showed that usability and sense of spatial presence were generally rated high [6, 8]. We will describe the core features and benefits of the *Immersive Inscribed Spaces* in the following subsections. Further details on the user-centred design and development process and the implemented features can be found in our previous work [6, 8].

### 2.1 Providing Spatial Context and Restoring Visibility

A key question for the humanities researchers is how people used to structure and interact with a space, and how they occupied and represented themselves in it. The *Immersive Inscribed Spaces* can display additional temporal layers, allowing researchers to view former conditions that no longer exist, e.g. due to decay of structures in scenario (a), as shown in Figure 2 or overpainting in scenario (b). For scenario (a), the 3D reconstructions of former structures visualise multiple time periods of the ancient theatre. Moreover, they provide a non-invasive alternative to anastylosis, the erection of non-preserved architectural elements in the real world. With anastylosis, only one time period can be restored, and the process often results in damage to ancient structures. Combined with a time of day simulation, the reconstructions reveal how much shadow was cast onto inscriptions. In scenario (b), virtual overlays of non-invasive X-ray Fluorescence (XRF)-scans uncover hidden layers in an emblematic painting.

In addition, the 3D VR representations of both cultural heritage sites preserve their current state in case of further destruction or decay.

### 2.2 Gaining New Perspectives

Another important aspect is the ability to view inaccessible locations and engage with new or impossible perspectives of these spaces in VR. We implemented various overview spots, including a bird's-eye perspective for scenario (a) that offers a complete top down overview of the ancient theatre and a close-up view of Written Artefacts on the ceiling of the church for scenario (b).

<figure><img src="image_35.png" /><figcaption>Figure 2: 3D reconstructions of former structures of the ancient theatre, including the stage building and seating tiers.</figcaption></figure>

### 2.3 Visualising and Filtering Written Artefacts

We added spatial markers that provide an overview of the distribution and clustering of all the Written Artefacts in their inscribed spaces. These markers can be selected to open up spatial information panels for each Written Artefact. These markers can be displayed in a filtered view based on their properties. For scenario (a) the markers indicate different groups of people (e.g. Jews), time periods, and functions of the Written Artefacts as displayed in Figure 3. Colour coding was used to distinguish between the different types, consisting of inscriptions, pictorial elements, game boards, and architectural marks. Filtering was used in scenario (b), as seen in Figure 4, to visualise the distribution of different themes and religious iconography of the Written Artefacts within the church.

<figure><img src="image_36.png" /><figcaption>Figure 3: Menu for scenario (a) showing a filtered selection of Written Artefacts of the type *inscription*.</figcaption></figure>

### 2.4 Reference, Information, Communication

Due to the large amount of Written Artefacts, the spatial information panels with their numbering are important for providing reference to the researchers. However, they are not only used as reference guides by researchers. They also contain proper, easier to read transcriptions and translations of the Written Artefacts, which makes them more accessible for other researchers, students, or interested users in general. The illustrations and photographs on the panels offer different perspectives and zoomed in detail views of Written Artefacts and the commentary text explains their social, cultural, and historical context.
<figure><img src="image_37.png" /><figcaption>Figure 4: Menu for scenario (b) with a filtered view of based on the selected iconographic category.</figcaption></figure>

## 3 Interaction Challenges in VR

We identified two major challenges for interacting with our *Immersive Inscribed Spaces* in VR that we studied in our research:

1. Spatial Interaction – Interacting with Written Artefacts: The large amount of Written Artefacts are distributed throughout the *Immersive Inscribed Spaces*. Their 3D position markers form clusters and overlap. This makes precise and distant spatial selections difficult.

2. Spatial Information – Presenting Written Artefacts Information: As detailed explanations, texts, illustrations, and photographs are integrated in VR, it's challenging to present the amount of content in a spatial context.

We explored and implemented possible solutions for both challenges and evaluated them in multiple user studies. The results and findings were integrated into our *Immersive Inscribed Spaces*.

### 3.1 Interacting with Written Artefacts

Raycasting is one of the most frequently used spatial interaction techniques in modern HMDs. It is easy to understand and use, as it is based on the pointer metaphor. However, selection performance significantly deteriorates with decreasing target size and increasing target distance [14, 13]. Our *Immersive Inscribed Spaces* require flexible selection of close as well as distant Written Artefacts markers that are clustered or close together. We proposed and evaluated redirected raycasting for facilitating spatial selection interaction [9, 10]. This assistive technique uses gain-based movement interpolation, similar to the already established redirected walking technique for VR locomotion, to subtly redirect the raycast towards the closest target. When the raycast enters a specified redirection zone, it is gradually pulled towards the redirection centre, which counters movement and tracking jitter to facilitate and stabilise the selection. We developed and evaluated three different raycast visualizations with different degrees of ray curvature and a gaze-assisted redirection technique. The results of our two user studies suggest that raycast redirection significantly improves selection performance and decreases perceived workload compared to classic raycasting (Baseline). Different visualizations did not affect the performance, but they might

influence the perception of the redirection. In both studies, redirection did not negatively affect the sense of agency of the users. Based on these promising findings, we implemented raycast redirection into our application for scenario (a), the Ancient Theatre of Miletus, as pictured in Figure 5. This large open space with scattered and clustered Written Artefacts makes raycast selection of the 3D markers especially challenging. With the improved assistive raycasting, we could also increase the maximum ray interaction distance to 20 meters in VR.

<figure><img src="image_38.png" /><figcaption>Figure 5: Illustration of visualized actual tracked raycast position (right ray) and the redirected ray pointer (left ray) shown to the user.</figcaption></figure>

### 3.2 Presenting Written Artefacts Information

For our specific use case involving Written Artefacts, it is essential to include various texts and descriptions in our spatial information panels. This requires users, especially for scenario (b) the interior of the church in Lucklum, to read long texts in VR. Although the increased display resolution of modern HMDs makes reading long texts feasible, there are still many challenges for visualizing textual information that have not been examined in detail. While there are some studies on text colours and text parameters such as font size [3, 4, 11, 12] there are no insights on layouts and user interface (UI) controls for accessing and reading long texts in VR. Thus, our previous work focused on performance and UX aspects for presenting and reading long texts in VR [7]. We developed four different text panels based on familiar desktop UI patterns, two discrete variants using pagination and two continuous variants with a scrollbar only, and scrollbar and scroll-buttons. Our user study revealed that the different panels did not influence the recall of content. This indicates that established desktop UI patterns can be applied to spatial UI. While some users liked using buttons, others preferred navigating with the scrollbar. However, the scrollbar only variant was the least preferred. Therefore, we adjusted our spatial information panels for scenario (b) and used the combination of scrollbar and scroll-buttons as UI controls for the long description texts as seen in Figure 6a. Since a lot of different information as well as images and photos need to be displayed, we created an adaptive spatial panel design based on input and feedback from the humanitites researchers. As pictured in Figure 6b, the panel is set up in a default, tabbed mode that contains different pages but can be switched to an alternative, side-by-side view with all pages arranged as a curved display to allow individual access for reading and comparing information.
<figure><img src="image_39.png" /><figcaption>Figure 6: Spatial information panel in default tabbed view mode (left) and alternative side-by-side mode (right).</figcaption></figure>

## 4 Conclusion

Through our interdisciplinary work on *Immersive Inscribed Spaces* we identified and implemented various features that demonstrate the potential of VR for research on Written Artefacts: It gives researchers access to impossible perspectives, provides them with a systematic overview of large amounts of Written Artefacts, and unlocks temporal context by restoring visibility of hidden layers and former, non-preserved structures. VR enables the immersive experience of historical cultural heritage sites and facilitates individual perception of Written Artefacts in their spatial context. However, we also showed that the use of immersive technologies in the new context for researching Written Artefacts brings specific, new challenges for the XR community regarding technical implementation, interaction techniques, and UI visualizations. We believe that interdisciplinary collaboration will be crucial for addressing these issues and for creating purposeful XR-based tools and applications for cultural heritage. We believe that these novel tools and applications will not only benefit researchers, but could also be used in science communication and for raising public awareness of the importance of cultural heritage and the research field of Written Artefacts.

## Acknowledgments

The research for this paper was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2176 ‘Understanding Written Artefacts: Material, Interaction and Transmission in Manuscript Cultures’, project no. 390893796. The research was conducted within the scope of the Centre for the Study of Manuscript Cultures (CSMC) at Universität Hamburg.

## References

[1] A. Bausi, D. Bondarev, N. Bregler, S. Chiarini, G. Ciotti, J. Droese, E. Dal Sasso, M. Friedrich, A. Helman-Ważny, M. Kohs, L. Mascia, A. L. Osthof, M. Ratzke, O. Škrabal, S. Sövegjártó, T. Staack, and H. Wimmer. Definition of ‘written artefact’. This paper is an occasional paper. Its results are based on exchanges, reflections, and discussions of the Theory and Terminology working group., Nov. 2023. doi: 10.25592/uhhfdm.13836 1

[2] C. Berns and A. Osthof. *Streets as contested spaces in ancient Miletus*, pp. 131–141. 2024. 2

[3] T. Dingler, K. Kunze, and B. Outram. Vr reading uis: Assessing text parameters for reading in vr. In *Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems*, CHI EA ’18, p. 1–6. Association for

Computing Machinery, New York, NY, USA, 2018. doi: 10.1145/3170427.3188695 3

[4] A. Erickson, K. Kim, G. Bruder, and G. F. Welch. Effects of dark mode graphics on visual acuity and fatigue with virtual reality head-mounted displays. In *2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*, pp. 434–442. IEEE, Manhattan, NY, USA, 2020. doi: 10.1109/VR46266.2020.00064 3

[5] J. Gabel, C. Berns, S. Bosch, J. Eickmeyer, K. Harter-Uibopuu, N. Martin, A. L. Osthof, J. A. Steiger, and F. Steinicke. User-Centered Design of Immersive Research Applications for Understanding Written Artefacts. In *J. Orlowski, D. Reiners, and B. Weyers, eds., ICAT-EGVE 2021 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments*. The Eurographics Association, 2021. doi: 10.2312/egve.20211325 1

[6] J. Gabel, C. Berns, S. Bosch, J. Eickmeyer, K. Harter-Uibopuu, N. Martin, A. L. Osthof, J. A. Steiger, and F. Steinicke. Immersive inscribed spaces – bringing virtuality to written artefacts for humanities. *i-com*, 21(1):7–18, 2022. doi: 10.1515/icom-2022-0012 1, 2

[7] J. Gabel, M. Ludwig, and F. Steinicke. Immersive reading: Comparison of performance and user experience for reading long texts in virtual reality. In *Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems*, CHI EA ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 10.1145/3544549.3585895 3

[8] J. Gabel, S. Rings, and F. Steinicke. Ancient theatre of miletus: A virtual reality research tool for studying written artefacts. In *Proceedings of the 2023 ACM Symposium on Spatial User Interaction*, SUI ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 10.1145/3607822.3618021 2

[9] J. Gabel, S. Schmidt, O. Ariza, and F. Steinicke. Redirecting rays: Evaluation of assistive raycasting techniques in virtual reality. In *Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology*, VRST ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 10.1145/3611659.3615716 3

[10] J. Gabel, S. Schmidt, K. Pfeuffer, and F. Steinicke. Guiding handrays in virtual reality: Comparison of gaze- and object-based assistive raycast redirection. In *Proceedings of the 2024 ACM Symposium on Spatial User Interaction*, SUI ’24. Association for Computing Machinery, New York, NY, USA, 2024. doi: 10.1145/3677386.3682080 3

[11] S. Kobayashi, K. Kanari, and M. Sato. An examination of view-settings for long texts in vr reading. In *ACM SIGGRAPH 2021 Posters*, SIGGRAPH ’21. Association for Computing Machinery, New York, NY, USA, 2021. doi: 10.1145/3450618.3469164 3

[12] T. Kojić, D. Ali, R. Greinacher, S. Möller, and J.-N. Voigt-Antons. User experience of reading in virtual reality — finding values for text distance, size and contrast. In *2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX)*, pp. 1–6. IEEE, Manhattan, NY, USA, 2020. doi: 10.1109/QoMEX48832.2020.9123091 3

[13] J. J. LaViola, E. Kruijff, R. P. McMahan, D. A. Bowman, and I. Poupyrev. *3D User Interfaces - Theory and Practice*. Addison-Wesley, Redwood City, CAUnited States, 2nd ed., 2017. 3

[14] I. Poupyrev, T. Ichikawa, S. Weghorst, and M. Billinghurst. Egocentric Object Manipulation in Virtual Environments: Empirical Evaluation of Interaction Techniques. *Computer Graphics Forum*, Volume17(Issue3):41–52, 1998. doi: 10.1111/1467-8659.00252 3

[15] O. Škrabal, L. Mascia, A. Osthof, and M. Ratzke, eds. *Grafiti Scratched, Scrawled, Sprayed: Towards a Cross-Cultural Understanding*. De Gruyter, Berlin, Boston, 2023. doi: 10.1515/9783111326306 1
# Virtual Immersion in Underground Quarries: An Innovative Exploration of Historical and Cultural Heritage

Nicolas Bremard*
CRISTAL, CNRS

Michel Dubois†
LGCgE, Université de Lille

Arnaud Gauthier‡
LGCgE, Université de Lille

Florent Berthaut§
CRISTAL, Université de Lille

<figure><img src="image_40.png" /><figcaption>Figure 1: Views from the VR application : (from left to right) animated quarry worker, the Blue Lake, and interactive quarry inspector</figcaption></figure>

## ABSTRACT

Underground quarries, as essential elements of cultural heritage, present unique challenges in terms of mediation, both from a safety and accessibility perspective. The use of virtual reality (VR) in this field allows for immersion in complex underground environments, thus providing an innovative solution to showcase them to the general public. This document presents a virtual reality installation dedicated to underground quarries, explaining its purpose, objectives, and possibilities.

**Index Terms:** VR, underground, quarry, accessibility, mediation

## 1 INTRODUCTION

Underground quarries represent a valuable cultural heritage, showcasing human ingenuity in the extraction of natural resources throughout history. These sites, often forgotten or difficult to access, contain important remnants related to industry, geology, and history, but they also present unique challenges in terms of conservation and mediation. Indeed, the safety of visitors and the accessibility of these complex spaces require innovative solutions to enable true immersion while ensuring the preservation of the sites.

The integration of virtual reality (VR) technologies in cultural heritage has proven useful in many other contexts [1]. By offering the ability to recreate detailed and interactive underground environments, VR might allow for overcoming the physical limitations of exploring these quarries. This technology might thus democratise access to these spaces while offering an interactive and immersive experience with added information, capable of attracting a wide audience.

This article aims to present such a VR installation dedicated to underground quarries in the city of Lezennes (France), designed by PIRVI (Pôle de compétences Interactions Réalité Virtuelle et Images / Skill Center for Interactions, Virtual Reality and Images, CRISTAL, Université de Lille) and LGCgE (Laboratoire de Génie Civil et géo-Environnement / Civil Engineering and Geo-Environment Laboratory).

The goal of this project was to provide a virtual version of the quarries for conservation and mediation purposes. On the mediation side, this virtual reconstruction is to be presented in regional exhibitions for schools and the general public, but it may also inform the design of similar experiences for cities around Lezennes, in the north of France and Belgium.

In this paper, we first present the Lezennes quarries. We then analyse the objectives of this installation, focusing on its content, the possibilities offered by VR, and its implications in terms of accessibility.

## 2 DIGITISING LEZENNES UNDERGROUND QUARRIES

The European Metropolis of Lille (comprising 95 municipalities, including 12 with quarries) has nearly 160 registered quarries, which represent 4.5 million cubic meters of void over an area of 181 hectares. These quarries constitute a significant cultural heritage, with geological and historical value as well as architectural and social, yet they remain largely under-exploited. For instance, numerous inscriptions can be found, ranging from their beginnings (12th century) to the present day.

In order to highlight this heritage, PIRVI, an engineering platform specialised in VR, has partnered with geologists of LGCgE, as well as the city council of Lezennes, on several projects related to underground quarries: degradation monitoring, training, and mediation. One of these projects, supported by the Call for Expressions of Interest (AMI) *Sciences Infusent* from the University of Lille, led to the creation of an exhibition on underground quarries, featuring a VR application. The goal of this exhibition was to immerse participants, who cannot visit the quarries for safety or accessibility reasons, in a virtual twin and introduce them to the history, dangers,

<footnote>*e-mail: nicolas.bremard@univ-lille.fr</footnote>

<footnote>†e-mail: michel.dubois@univ-lille.fr</footnote>

<footnote>‡e-mail: arnaud.gauthier@univ-lille.fr</footnote>

<footnote>§e-mail: florent.berthaut@univ-lille.fr</footnote>
jobs and uses of the quarries.

## 2.1 Lezennes Quarries

The village of Lezennes (about 3.500 inhabitants) has the most extended area of underground quarries (about 70 ha) of the European Metropolis of Lille. Extraction probably began during the 11th century and ended in the middle of the 19th century [2, 3]. The rock mined was a white chalk of Coniacian to Santonian age (end of the Cretaceous period, about 90 My). The material was mainly used for the construction of many buildings of the Lille metropolitan area (houses, farms as well as prestigious buildings of the City of Lille such as the medieval city fortifications, the XVIIth century citadel, several palaces, churches, ...). Additionally the carbonaceous rock was exploited during the 19th and 20th centuries to produce lime in the so-called "catiches" consisting of very particular bottle-shaped mining rooms. The peculiarity of the quarries is that they were used for several other purposes later on. Indeed, they were exploited from the middle of 19th century to the middle of the 20th century for vegetable productions such as mushrooms and wild chicory ("barbe de capucin"). Moreover the quarries served as shelters during some conflicts, particularly during the two world wars both for soldiers and civilians of the village [2, 3].

Since 1999 the Municipality of Lezennes planned to promote this heritage in collaboration with the University of Lille and the local history association C.R.H.L. ("Cercle de Recherche Historique Lezennois"). Indeed, since 1999, the quarries are open to public during the European Heritage Days week-end representing an influx of 400 to 500 people during two days. Moreover, the municipality has launched several actions around the theme of subterranean quarries, for instance the creation of a processional giant that symbolises the village (Isidore), a statue dedicated to quarry works, a monument at the town entrance (white wall), a stone festival ...

Whereas very attractive to the public, the underground heritage is difficult to access. The annual visit is limited to a single gallery representing about 40 meters length. Inferred risks include going down stairs, walking in a narrow gallery, gallery height not high enough, air quality (low oxygen rate), roof and wall instabilities. Additionally, access for young children and persons with reduced mobility is impossible. Thus, access to very interesting sectors, such as the famous "Blue lake", is impossible (and forbidden by decision of the mayor) except for specialists and quarry inspectors. The restriction of access has led to the investigation of other solutions to satisfy the public curiosity and VR represents the approach we chose to enhance historical heritage.

## 2.2 Digitising the quarries

3D scanning of underground quarries offers numerous benefits, both in terms of preservation and mediation. By creating an accurate and detailed digital replica of these sites, scanning allows for a faithful record of their current condition, which is essential for their long-term protection. Indeed, underground quarries are often exposed to risks of degradation, whether due to rocky material fatigue, water saturation variations in chalk, or surface human activities [5]. Through digitisation, it becomes possible to monitor the evolution of these sites and identify areas that require conservation interventions, while offering a solution to preserve them virtually without risking physical deterioration [4].

Transforming these scanned quarries into virtual environments opens up new possibilities for cultural mediation and heritage discovery. By enabling full immersion in these often difficult-to-access environments, VR democratises access to these unique sites, whether they are located underground or in prohibited areas. This approach allows for reaching a broad audience, from researchers and students to history and geology enthusiasts, while providing an interactive experience that enhances the understanding of geological, historical, and technical processes. Moreover, VR enables the

exploration of these quarries in a safe manner, without the physical constraints associated with direct exploration, through various 3D navigation techniques.

That is why we conducted numerous visits to carry out more than a hundred 3D scans (using a FARO Focus, see Fig. 2a). This allowed us to create VR versions of a dozen locations, despite difficult access and fragile equipment. Each digitised site was chosen based on its geometry (large enough to navigate in the virtual environment) and its content (presence of inscriptions, geological features, tool marks, traces of various activities, etc.). For instance, Figure 2b shows the map of the *Blue Lake* area and Figure 2c depicts the resulting reconstructed mesh of this area.

## 3 VIRTUAL REALITY PUBLIC PRESENTATION

Once the chosen locations in the quarries were digitised, we developed a VR application in order to offer an unprecedented immersion into these often overlooked and difficult-to-access spaces. To make this experience as enriching as possible, the 3D environments representing each location are carefully arranged by themes, allowing users to explore the quarries from various perspectives: historical, geological, technical, and even patrimonial. Each thematic area provides specific information, enhancing the experience with interactive elements such as animations (characters or interactive events, such as the collapse of a gallery rook to illustrate a major risk) and texts. This setup enables a personalised exploration, where each user can focus on the aspects they are interested in, while facilitating an educational and enjoyable approach for various types of audiences.

The application has also been designed with the needs of people with reduced mobility in mind, ensuring an experience that is accessible to everyone.

### 3.1 Selection of themes and interactions

Over time, underground quarries have been used for different reasons. We decided to present the stroll between the locations as a journey through history: from their first use, the chalk extraction activity, to their present state. Thus the visitor enters the quarry and gradually reaches different locations. The themes covered are:

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /><figcaption>(a) 3D Scanning of the quar- (b) Top view of the Blue Lake
 quarries area</figcaption></figure>

<figure><img src="image_43.png" /><figcaption>(c) Outside view of the reconstructed mesh for the Blue Lake area</figcaption></figure>

<caption>Figure 2: Digitising process of the underground quarries</caption>
1. Quarries in the Middle Ages;

2. The quarries in the 19th century;

3. Resting place for soldiers during the First World War;

4. Shelter for civilians during the Second World War;

5. Mushroom cultivation;

6. The Blue Lake (overdigging down to the water table), shown in Figure 1 centre;

7. The Scout Altar room, a peculiar worship room attributed to scout group activity;

8. Geology and paleontology;

9. Metrology and site monitoring.

Each theme is explained through text and animations. For example, in the quarry themes, the visitor can see a quarryman working, as shown in Figure 1 left, and even choose the successive movements and postures of the quarry worker; regarding scenes relating to wars, the visitor can virtually meet characters (a soldier and a female refugee) whose portraits and costumes have been designed from vintage photos.

For each location, users can point at various elements using a ray casting metaphor in order to retrieve information on the thematic, as depicted in Figure 1, right.

The transition from one theme to another is performed via a menu. The themes are organised in chronological order. We therefore made the choice not to allow the possibility of moving completely freely between them: from a theme, it is only possible to go to the previous or next theme. The menu appears when the visitor moves to the center of the physical space and appears in front of them, at the same height as their head. The menu contains a list of other themes, each represented by a name and a picture of the most iconic element in it, which get highlighted when visitors point at it.

In addition, to recreated the immersion in real quarries, we gave the visitor the opportunity to play with the light, positioned above their head like a headlamp, as can be seen in Figure 1 centre. By pressing a button, this light can be switched between different modes: off, red, low, medium and high intensity. This choice was made because many headlamps are adjustable in this way. We also added a circular flashlight pattern to replicate the way a headlamp illuminates.

## 3.2 Accessibility

In order to ensure accessibility for as many visitors as possible, we implemented 2 interaction modes: a seated mode and a standing mode.

The seated mode is designed for persons with reduced mobility (PRM) to move in the VR environment. This mode allows a visitor to move while seated; the person can use joystick to turn and go around(up and down to go forward/backward; and right and left for rotating). This mode permits visitors in wheelchair, people that cannot stand for long, or people that have difficulties in VR (like balance problems) to stay a long time in VR environment. However, this mode of navigation could cause motion-sickness and techniques to avoid it should be investigated.

The standing mode enables a person to walk around. The area from the real space is mapped to a space of the same size in VR, where the visitor can move. This technique allows the visitor to move freely without using any controllers.

We integrated a video tutorial in order to let visitors use the VR application without presence of a facilitator or a guide. It allows anyone to quickly grasp the interaction mechanics in just a few minutes and configure its use. The videos are accompanied by very short texts and appear only one or two at a time.

## 3.3 Public presentation

The application was showcased in combination with an exhibition related to the underground quarries. The VR experience was more

focused on contemplation and contains little text, whereas the exhibition included numerous posters (covering the same themes), display cases with objects, and educational games.

This exhibition began during the 2024 Heritage Days and remained open to the public for over a month (from 20/09 to 27/10). During this time, two booklets were produced: an explanatory one (explaining each theme in more detail), and a playful one (more targeted for young children).

The event was a resounding success, attracting 560 visitors in total (357 visitors on opening hours, including 10 school groups, not counting visitors during 2 special "Heritage Days"), highlighting the growing interest in immersive technologies for exploring heritage We believe that the application not only captivated participants with its interactive nature but also helped strengthen cultural mediation by offering an innovative way to understand and preserve local history. By integrating educational tools suited to different audiences, the exhibition created a bridge between the past and the present, while sparking curiosity and enthusiasm among visitors.

## 4 CONCLUSION

This VR application dedicated to the exploration of underground quarries represents a contribution in the promotion of historical and cultural heritage. By offering an interactive immersion, it allows visitors to discover, in a pleasant and enriching way, places that are often inaccessible. The application not only helps to understand the history of the quarries but also enables users to grasp their significance in the development of past societies and their environmental impact. By relying on innovative digital tools, this initiative contributes to modern cultural mediation, accessible to all and particularly suited to new generations, showing curiosity for it, and persons with reduced mobility (PRM). It demonstrates that immersive technologies can play a key role in preserving and spreading collective memory, while meeting the expectations of an increasingly curious public sensitive to heritage issues.

The exhibition, accompanied by the virtual reality application, will be moved to other locations and events, allowing for reaching a broad audience. After its launch during the Heritage Days, the exhibition can be set up in other cities or historical sites, thus providing increased visibility to these underground quarries and their heritage. Moreover, the software base of the application has been designed to be modular, allowing for easy adaptation to other heritage sites or themes, ensuring its flexible reuse for other cultural mediation projects. This approach supports the application sustainability and expansion, while diversifying the experiences offered to visitors and enriching the cultural heritage offering at both the national and international levels.

## ACKNOWLEDGMENTS

The authors would like to express their sincere thanks to the University of Lille for its valuable support throughout this project. Special thanks go to the Sciences Infusent, whose dedication and professionalism have been essential in highlighting our initiative. Thanks to their expertise and support, we were able to reach a wide audience and ensure the visibility of the exhibition and the virtual reality application. Their contribution was indispensable to the success of this event, and we are deeply grateful for their ongoing collaboration and support.

We would also like to extend our sincere gratitude to the Service Commun des Carrières of the City of Lille for their invaluable assistance and professionalism. Their expertise and timely support have been essential for our work, and we are deeply appreciative of their commitment to maintaining the highest standards in quarry management. The collaboration has been seamless, and we look forward to continuing to work together to achieve mutual success. Thank you once again for your dedication and support.
Finally we thank the city council of Lezennes for everything it
has done for facilitate the work (access to the quarry, providing a
media library space, inauguration ceremony, reception of the pub-
lic).

REFERENCES

[1] M. K. Bekele, R. Pierdicca, E. Frontoni, E. S. Malinverni, and J. Gain. A Survey of Augmented, Virtual, and Mixed Reality for Cultural Heritage. Journal on Computing and Cultural Heritage, 11(2):1–36, 2018. doi: 10.1145/3145534 1

[2] B. Bivert. *Les souterrains du Nord-Pas-de-Calais*. Ed. Conseil Général Département du Nord, 1988. 2

[3] Cercle de Recherche Historique Lezennois. *Voyage au cœur de Lezennes, chronique des carrières souterraines*. Ed. Cercle de Recherche Historique Lezennois, 2009. 2

[4] M.-P. Hanesse and T. Richard. Guide de surveillance des cavités souterraines d'origine anthropique. Rapport Ineris DRS-16-156834-00810B, 2016. 2

[5] G. Ramadan. *La stabilité et l'évolution des cavités souterraines: sur l'exemple des carrières souterraines de la Métropole lilloise*. PhD thesis, Université de Lille 1, 2016. 2