###### Abstract  
In building-scale VR, where the entire interior of a large-scale building is a virtual space that users can walk around in, it is very important to handle movable objects that actually exist in the real world and not in the virtual space. We propose a mechanism to dynamically detect such objects (that are not embedded in the virtual space) in advance, and then generate a sound when one is hit with a virtual stick. Moreover, in a large indoor virtual environment, there may be multiple users at the same time, and their presence may be perceived by hearing, as well as by sight, e.g., by hearing sounds such as footsteps. We, therefore, use a GAN deep learning generation system to generate the impact sound from any object. First, in order to visually display a real-world object in virtual space, its 3D data is generated using an RGB-D camera and saved, along with its position information. At the same time, we take the image of the object and break it down into parts, estimate its material, generate the sound, and associate the sound with that part. When a VR user hits the object virtually (e.g., hits it with a virtual stick), a sound is generated. We demonstrate that users can judge the material from the sound, thus confirming the effectiveness of the proposed method. building-scale VR; impact sound generation; GAN +
Footnote †: journal:  
area. Various applications are conceivable. For example, it could be used to help simulate a situation in which reality is difficult to reproduce. One such situation is the occurrence of a disaster, for which building-scale VR could be used to give many people the simulated experience of a disaster, in a realistic manner. Several technologies are used for creating building-scale VR, as listed below. Due to space limitations, we do not discuss the following technologies in great depth, but please see [1] for details. 1. Automatic generation of indoor 3D maps--technology for creating a 3D map that faithfully reproduces the actual 3D shapes in a building. 2. Enlargement of human tracking area by deep learning--technology for tracking the pose (position and orientation) of users moving around the indoor area of a building. 3. Tracking of user's hands by visual-inertial odometry--technology for manipulating objects with both hands in virtual space. 4. Modification of building-scale indoor scene--technology for modifying VR space in accordance with its intended usage by segmenting or replacing 3D point clouds. 5. Representation of people in the area--technology in which people are displayed in a virtual space and all of them share the same information as in real life. 6. Simulation of behaviors and phenomena--technology to simulate human behaviors and physical phenomena in virtual space. We have previously developed an application for building-scale VR, in which we conduct virtual training in disaster situations [1]. This training is not only for evacuation scenarios, in cases of fire, but also for firefighting activities using a fire extinguisher. We have also conducted behavioral simulations during floods. Other applications include support for presentations and discussions in virtual meeting rooms. Specifically, we developed an application that creates a 3D map of a meeting room in the real world with presentation slides and other materials that are displayed on a virtual screen to enable a meeting to be held, in the same way as in reality. A key feature here is that we can display a 3D object and have a meeting while referring to the object. This is a meeting that can only be performed in a space that extends the real world. In building-scale VR, it is necessary to reconstruct the real world, in both visual and auditory terms. Since it is impossible to reproduce, in real time, the dynamic environment of the real world in virtual space, we create a virtual environment that has almost no change, in advance. Then, the changing environment (mainly a movable object) is reconstructed in the virtual space in real time.