# Blending Real and Virtual Worlds Using Self-reflection and Fiducials

Martin Fischbach, Dennis Wiebusch, Marc Erich Latoschik, Gerd Bruder, Frank Steinicke

► To cite this version:

Martin Fischbach, Dennis Wiebusch, Marc Erich Latoschik, Gerd Bruder, Frank Steinicke. Blending Real and Virtual Worlds Using Self-reflection and Fiducials. 11th International Conference on Entertainment Computing (ICEC), Sep 2012, Bremen, Germany. pp.465-468, 10.1007/978-3-642-33542-6_54. hal-01556142

HAL Id: hal-01556142

https://inria.hal.science/hal-01556142

Submitted on 4 Jul 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Blending Real and Virtual Worlds using Self-Reflection and Fiducials

Martin Fischbach<fnref n="1" />, Dennis Wiebusch<fnref n="1" />, Marc Erich Latoschik<fnref n="1" />
Gerd Bruder<fnref n="2" />, Frank Steinicke<fnref n="2" />

**Abstract.** This paper presents an enhanced version of a portable out-of-the-box platform for semi-immersive interactive applications. The enhanced version combines stereoscopic visualization, marker-less user tracking, and multi-touch with self-reflection of users and tangible object interaction. A virtual fish tank simulation demonstrates how real and virtual worlds are seamlessly blended by providing a multi-modal interaction experience that utilizes a user-centric projection, body, and object tracking, as well as a consistent integration of physical and virtual properties like appearance and causality into a mixed real/virtual world.

**Keywords:** Mixed Reality, Self-Reflection, Fiducials, Fish Tank Virtual Reality, Interactive Virtual Art, Multi-Touch

## 1 Introduction

Milgram and Kishimo define Mixed Reality (MR) to be located on a continuum between Augmented Reality (AR) and Augmented Virtuality (AV) [1]. While AR superimposes the real world by computer-generated images, AV enhances virtual worlds using real-world data. The general idea blends real and virtual objects into a combined real/virtual space which shares certain perceptive attributes like consistency of geometry, time, or illumination [2].

Merging real and virtual worlds using computer technology is challenging. One of the most important tasks concerns placements of superimposed objects to be consistent with the real world. Likewise, lighting effects like reflections and shadows have to match to obtain a consistent perception of a shared illumination space. In addition, movements of real/virtual objects have to be consistent and follow a plausible simulation of cause and effect between both worlds.

The combined synthetic and real illumination of objects to produce blended high quality images works only under specific limitations. Such limitations include renouncement of real-time performance, requirement of a geometric computer model of the real scene, light sources to be static [3], or special hardware usually not accessible in most MR system environments [2].

<footnote n="1">¹ Human-Computer Interaction</footnote>

<footnote n="2">² Immersive Media Group</footnote>

<footnote>University of Würzburg</footnote>

<footnote>martin.fischbach@uni-wuerzburg.de,</footnote>

<footnote>WWW home page: {hci|img}.uni-wuerzburg.de</footnote>
This paper presents an enhanced version of the smARTbox [4, 5], a portable out-of-the-box platform for semi-immersive interactive applications in promotional, artistic, or educational areas. The enhanced version blends real and virtual worlds using self-reflection of users and fiducial physical interaction objects to manipulate the virtual content. In addition to the stereoscopic visualization, marker-less user tracking and direct interscopic touch input of the smARTbox, the enhanced version provides two new key features demonstrated using a virtual fish tank simulation: (1) Interacting users perceive a self-reflection of their physical presence in the virtual world, i.e., on the simulated liquid surface of the fish tank and (2) real physical objects modify the simulation behavior, i.e., the virtual fishes react to objects placed in vicinity of the fish tank.

## 2 System Setup

The smARTbox's [4] metal frame provides the skeleton for a 112 × 90 × 63 cm ($w \times h \times d$) wooden box. The top side is a back-projection surface fed with stereoscopic 3D images at 720p by an Optoma GT720 DLP-projector via a wide-angle converter lens and a mirror mounted at the bottom of the box at a 45° angle. The demonstration uses an Intel Core i7 @3.40GHz processor with 8GB of main memory and an nVidia Quadro 4000 graphics card. One Microsoft Kinect multi sensor system is utilized in combination with a set of IR-lights and an IR-camera in order to track the user's body and the physical objects, respectively. The hardware setup (without the computer) was constructed at a total cost of less than €2,500 during a students project. The smARTbox is driven by Simulator X [6], a Scala-based software framework for intelligent interactive applications.

## 3 Touch & Fiducial Marker Tracking

In recent years, multi-touch interaction in combination with stereoscopic display has received considerable attention [7–9]. Hence, we constructed the smARTbox in such a way that the screen serves as a multi-touch-enabled surface using the Rear-DI principle [10]. Therefore, six clusters of IR-LEDs illuminate the screen from the inside with infrared light, which is reflected by objects in contact with the surface. The contact points are sensed by a Point Grey Dragonfly®2 digital video camera with a wide-angle lens and infrared band-pass filter. Touch positions and gestures are analyzed by a modified version of the NUI Group's CCV software [11]. In addition to tracking multiple touch points, CCV supports tracking of small fiducials, which can be pre-registered with corresponding real-world objects. For instance, a fiducial marker can be placed at a can for fish feed (see Figure 1 right). When the can (respectively the fiducial) is in contact with the touch surface, it gets recognized and its touch area serves as attractor for the fish. Other fiducials may be associated to objects, which serve as detractors to the virtual fish such as predators.
## 4 Self-Reflection

The augmentation of the virtual scene with real world information additionally enhances the sense of presence for mixed reality environments [12]. Continuing this approach, the smARTbox setup utilizes 2.5D models of the real scene obtained from the Kinect to realize virtual self-reflections of the user:

A triangulated 2.5D mesh with corresponding normals, faces, and texture coordinates is processed using the nestk library [13]. The related texture, a 640×480 pixel color image, is captured from the Kinect's RGB-camera. The communication between the native library-based image processing and the Scala-based Simulator X is handled by the Scala Native Access (SNA) framework. Simulator X' built-in renderer is used to appropriately position the textured mesh data in the virtual scene. The mesh contains the necessary spatial (geometric) data which is used to calculate a reflection at a virtual surface aligned with the water surface. The result is blended by a custom water shader with the interactive water surface (see Figure 1 left).

<figure><img src="image_2.png" /><figcaption>Fig. 1. Reflections on water surface (left) and interaction with a fiducial marker (right).</figcaption></figure>

## 5 Conclusion

The smARTbox has great potential for presentations and exhibitions, since it is less expensive, easier to set up and transport than typical virtual reality installations. For this demo we have enhanced the smARTbox by an advanced blending of virtual and mixed reality concepts, which considerably increases realism of virtual objects. The lack of obtrusive instrumentation in combination with presence-enhancing visual and tangible interaction improves the engagement with users, e. g., in exhibition scenarios. The utilized software platform
Simulator X provides a state-of-the-art simulation engine with an integrated AI-
core and several multimodal interaction components. Its scalable architecture
provides an ideal platform, e. g., to experiment with alternate sensors, possibly
enhancing accuracy or precision. An evaluation that we plan during the demo
session will show in how far the usage of virtual reflections and tangible object
interaction increases the users sense of presence and improves object interaction.

*Acknowledgements*
The authors want to thank Christian Treffs, Alexander Strehler
and Anke Giebler-Schubert for supporting the implementation and realization
of the presented work.

References

1. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems E77-D(12) (1994)

2. Naemura, T., Nitta, T., Mimura, A., Harashima, H.: Virtual Shadows in Mixed Reality Environment Using Flashlight-like Devices. Transactions of Virtual Reality Society of Japan 7(2) (2002) 227–237

3. Jacobs, K., Loscos, C.: Classification of illumination methods for mixed reality.
Comput. Graph. Forum 25(1) (2006) 29–51

4. Fischbach, M., Latoschik, M., Bruder, G., Steinicke, F.: smARTbox: Out-of-the-
Box Technologies for Interactive Art and Exhibition. In: Virtual Reality Interna-
tional Conference (VRIC), ACM (2012)

5. Fischbach, M., Wiebusch, D., Latoschik, M.E., Bruder, G., Steinicke, F.: smartbox: A portable setup for intelligent interactive applications. In: Mensch und Computer Demo Papers. (2012)

6. Latoschik, M.E., Tramberend, H.: Simulator X: A Scalable and Concurrent Software Platform for Intelligent Realtime Interactive Systems. In: Proceedings of the IEEE VR 2011. (2011)

7. Steinicke, F., Benko, H., Daiber, F., Keefe, D., de la Rivière, J.B.: Touching the 3rd dimension (T3D). In: Proceedings of the Conference extended Abstracts on Human Factors in Computing Systems, ACM (2011) 161–164

8. Valkov, D., Steinicke, F., Bruder, G., Hinrichs, K., Schöning, J., Daiber, F., Krüger, A.: Touching floating objects in projection-based virtual reality environments. In: Proceedings of Joint Virtual Reality Conference. (2010) 17–24

9. Valkov, D., Steinicke, F., Bruder, G., Hinrichs, K.: 2D touching of 3D stereoscopic objects. In: Proceedings of the Conference on Human Factors in Computing Systems, ACM (2011) 1353–1362

10. Schöning, J., Hook, J., Bartindale, T., Schmidt, D., Oliver, P., Echtler, F., Mottamedi, N., Brandl, P., Zadow, U.: Building interactive multi-touch surfaces. Tabletops-Horizontal Interactive Displays (2010) 27–49

11. Cetin, G., Bedi, R., Sandler, S.: Multi-touch technologies. Natural User Interface (NUI) Group (2009)

12. Steinicke, F., Hinrichs, K.H., Ropinski, T.: Virtual reflections and virtual shadows in mixed reality environments. In: Proceedings of the International Conference on Human-Computer Interaction (INTERACT). (2005) 1018–1021

13. Burrus, N.: NESTK Library. "http://nicolas.burrus.name/index.php/Research/-KinectUseNestk" (2012) last accessed: 30/05/2012.