# X-Reality Museums: Unifying the Virtual and Real World Towards Realistic Virtual Museums
## 2 Related Work
### True Mediated Reality  
Virtual characters play a fundamental role for attaining a high level of believability in XR environments [54]. In this respect, we introduce the term \"true mediated reality\" to define the need for delivering realistic virtual characters, by means of (i) photo-realistic reconstruction of 3D models from the real world, (ii) real-time mixed-reality virtual character rendering and animation, and (iii) realistic interactive virtual characters. In the context of virtual museums, true mediated reality is a key element for achieving high quality user experiences, but also for transferring knowledge to museum visitors. With regard to realistic reconstruction of 3D models from the real world, significant technical progress has been achieved in capturing, creating, archiving, preserving, and visualizing 3D digital items [55]. However, for the realization of the proposed unified XR experiences in realistic virtual museums, state of the art 3D reconstruction techniques should be further enhanced, in order to produce realistic 3D reconstructions that will be appropriate for deployment in real-time XR environments. In addition, in the context of modelling cultural heritage artefacts, a challenge that needs to be overcome is that antiquities, and especially statues, are often incomplete (e.g., broken, or partly recovered), thus rendering typical 3D reconstruction techniques ineffective. Such a restoration would be highly specialized and resource demanding, requiring the involvement of domain experts (e.g., archaeologists and curators) with sophisticated 3D modelling skills to manually extend a partially reconstructed model. To resolve this, future research should explore semi-automated approaches, through interdisciplinary joined efforts involving the fields of archaeology, museology and curation, as well as computer science. Rendering deformable objects and characters in XR, attaining a high level of believability and realism of real-time registration between real scenes and virtual augmentations requires two main aspects for consistent matching: geometry and illumination [55]. Super-imposing in real-time virtual objects and dynamic-deformable virtual scenes onto an image of a real scene is still an open research field [56; 57]. In addition, a demanding undertaking in order to achieve realistic representation is the real-time animation of deformable virtual agents through real-time model skinning. Given the proliferation of low CPU power devices (e.g., untethered headsets), a prominent technological challenge is the need for realistic, high fidelity and high refresh rate rendering of the deformable 3D characters in such devices. No matter how efficient model deformation algorithms for the rendering of humanoid 3D models are, several aspects that imitate the human behavior should also be considered for creating convincing virtual agents. Realistic interactive virtual characters should feature a reliable and consistent motion and dialog behavior, but also nonverbal communication and affective components. Modelling the \"mind\" and creating intelligent communication behavior on the encoding side, is an active field of research in AI [58]. On the other hand, the visual representation of a character including its perceivable behavior, from a decoding perspective, such as facial expressions and gestures, belongs to the domain of computer graphics and implicates many open issues concerning natural communication [59]. Other non-verbal communication attributes that should be addressed by embodied agents in order to further enhance their naturalness include body posture, which provides emotional cues, as well as gaze direction, which is important not only for conveying emotion but also for semantic cues during a conversation.