## 4 Multimodal Interaction
### Interaction Based on the Visual Channel  
3D models can be digitally defined when the purpose is to extend the experience with augmented and virtual reality. Virtual and mixed (augmented) reality technologies [2; 9] allow the user to enrich his/her fruition experience or visit sites that are not physically accessible [39]. The virtual immersion usually involves sight and has a minor impact on other senses. Music, sounds, narration and interactive speech could be elements easily included in the immersive user's experience and could give a practical, emotional impact. Tactile [40], smell [41] perceptions require more complex devices and technologies that are expensive and difficult to include in a consumer system. Generally, AR and VR systems that interact are better appreciated if they do not require user adaptation and only employ intuitive and natural human communication channels. Ideally, a multimodal experience could assure a compelling realism of an augmented or of a virtual environment and a realistic simulation of the user's presence in it. On the other hand, the user's multimodal perception has to take into account its affective impact [42]. For this reason, the monitoring, and understanding of the human emotional reaction [43] should be another aspect to consider to build dynamic and adaptable systems. The potential offered by AR and VR systems can be significantly increased through the Internet of Things (IoT) [44], it is considered a key enabling technology for making environments smarter and more interactive [45; 46; 47; 48]. Internet of Things (IoT) can enhance the fruition experience. Museum installations can be devised by using smart autonomous objects, that are objects augmented with sensing, processing, storing, and actuating capabilities and that can also exchange information with each other [49; 50]. It is of interest to consider how AR can give an intuitive way to communicate with IoT objects. Main issues are related to three aspect regarding data management, viewer and display devices and interfaces with their interaction methods. The main problems related to data management consist of building a virtual objects dataset and giving information on the surrounding environment for AR whereas, for IoT, there is a need to guarantee access to information available from sensors and a good quality of service. Viewer and display devices should avoid latency and registration errors for AR and less-intuitive context information for IoT. Finally, the interfaces are expected to give a fixed interaction method for AR and intuitively communication devices with a limited response time for IoT. However, the use of both AR and IoT results advantageous compared to a ubiquitous computer and a more straightforward AR approach. As a matter of fact, even if computers are always present in different forms, the user is forced to interact with different devices; differently, the augmented interaction allows using a unique computer connected to the systems to be controlled. Besides, by using both approaches, the user can interact intuitively for example, by means of a helmet with hand gestures; it results in a more natural interaction with surrounding world and allows interaction with devices even if they are not physically present or with objects that cannot be touched or handled, such as valuables or cultural goods. This approach also permits additional information to be proposed to the user depending on his/her preferences. To access the surrounding objects, the user receives the collected AR attributes by a server in which information are shared; then, the user by the AR browser program can connect all of the things exploiting prebuilt relationship mapping [44]. This approach in which things are available under a unified Web framework (also known as \"webization of things\") can be usefully exploited in cultural heritage framework with the development of AR devices, assuring the user a good experience.