# Tangible Multi-card Projector-Based Interaction with Physics

Songxue Wang, Youquan Liu, Junxiu Guo

► To cite this version:

Songxue Wang, Youquan Liu, Junxiu Guo. Tangible Multi-card Projector-Based Interaction with Physics. 19th International Conference on Entertainment Computing (ICEC), Nov 2020, Xi'an, China. pp.248-254, 10.1007/978-3-030-65736-9_21. hal-03686032

HAL Id: hal-03686032

https://inria.hal.science/hal-03686032

Submitted on 2 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Tangible Multi-Card Projector-based Interaction
With Physics

Songxue Wang¹[0000-0001-8881-8680], Youquan Liu¹, and Junxiu Guo¹[0000-0001-6869-2830]

School of Information Engineering, Chang'an University, Xi'an, China
{2018124053,youquan,2018124055}@chd.edu.cn

**Abstract.** We propose a projector-based interaction method and system with fusion of virtuality and reality, supporting participants to interact with real objects like cards physically on the desk or table. An infrared camera is mounted on the micro-projector to detect the hidden markers embedded on cards, while the projector is used to project the generated virtual objects on the surface of cards. When participants move or rotate the tangible cards, the velocity of markers is calculated from the pose change of cards. Especially both the pose and the velocity are used to control the virtual characters which enables participants to interact with virtual objects realistically with the help of physics simulation. Meanwhile, by tracking the multiple cards and orientation projecting, we design a configurable dynamic storyboard system. A variety of examples of interactive scenarios are provided, which verify our method's validity and extensibility.

**Keywords:** Projector-based interaction · Physical interaction · Fusion of virtuality and reality · Tangible cards.

# 1 Introduction

Many intelligent projectors are integrated with interaction through fingers, gestures or even tangible objects, such as Sony's Xperia Torch [8]. Such kind of projector-based interactions are often used in education, entertainment and other fields.

In this paper, we propose a novel projection system supporting tangible interaction with cards, with temporal-spatial information used to control the virtual characters. The projected content is manipulated by the participants through the cards, which is similar to the traditional Chinese shadow play. The difference is that the participants manipulates cards with infrared (IR) hidden markers [6], and the generated virtual content is projected directly on tabletop and surfaces of tangible cards according to the pose of target markers.

In this paper, our contributions can be exhibited at two different aspects:

- We obtain the pose and velocity information of cards by tracking the IR hidden markers, and combine physical simulation to give physical properties to the projected virtual objects.
- We present a configurable dynamic storyboard system through multi-target tracking and orientation projection onto multiple cards, while designing all kinds of cards makes the storyboard expandable.

## 2 Related Work

As the pioneer work, DigitalDesk [10] presented a way to combine the physical document with the digital one through projection. Willson designed a desktop system named PlayAnywhere [12], which used oblique projection to display virtual information, shadows to detect touch, optical flow to estimate motion to support translation, scaling and rotation of virtual objects, and Hough transform with the Sobel edge to track pages. However only images or videos are displayed and interactions are only available in these transforms. With the popularity of projectors, interactive electronic whiteboard is widely used. This kind of interaction mainly uses fingers or pen to play with the content, such as Xperia Touch [8] which is based on IR sensor to perceive the movement of fingers on the table, and DreamBoard [3] which is based on handwriting recognition. These systems lack a sense of control over the projected objects.

Marco et al designed a vision based desktop system named NIKVision [5], in which users manipulate the toys with marker attached at the bottom to control the story character. Since a camera is needed to mount under the desktop to read the markers, it imposes a limitation on user's operation space. Villar et al presented a Near Field Communication (NFC) based system named Zanzibar [9], in which toys with NFC label attached are used to control the story character. A mat integrated with NFC scanner and capacitor induction is used to track the toys. Users can manipulate the toys on the mat to create story, and the story scenario is displayed on the screen. However users have to balance the operation of tangible objects with the screen display.

Wu et al proposed MagicPAPER [13] based on multi-sensors, such as Air-Bar, Kinect, LeapMotion and camera, using tangible objects like toothbrushes and wooden blocks to interact on kraft paper. As paper markers also belong to tangible objects and can be well applied in reality, some existing work, such as [7,1,11], explored tangible interaction based on tracking paper markers without many sensors. In particular, Willis et al [11] proposed mobile interaction with tangible objects and surfaces by detecting IR hidden markers. However just position information is extracted from the tracked markers to drive the projected content.

## 3 Method

Our system is composed of 4 parts, as demonstrated in Fig. 1.

Here OpenCV library is used to calibrate the camera [14] and the projector [2] to obtain internal parameters, external parameters and distortion coefficients. Our hardware design ensures that the camera-projector system only needs to be calibrated once, and there is no need to re-calibrate when moving the device.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Overview of our system</figcaption></figure>

The IR hidden markers printed on tangible cards are tracked through the visual tracking algorithm to obtain pose of the cards, and then the velocity is calculated from the card pose change. participants control the virtual character by manipulating the corresponding card on the desktop. Different cards play different roles in a virtual scenario with different physical effect with not only pose used but also velocity used.

In the following sections we will describe the key points of our system, including hardware, tracking and locating, aligned projection, physical simulation and storyboard.

## 3.1 Hardware

Our customized hardware consists of a projector with resolution of 1024 x 768, an IR camera used to capture 640 x 480 images and IR light sources. The camera-projector system is mounted on a stand. Since the card is used as both tracking input and projection output, we use IR-absorbing ink to print the markers on the card to avoid the interference between projected content and marker patterns.

## 3.2 Tracking and Locating of Cards

To ensure the robustness of tracking and fluency of interaction, fiducial markers are used to locate the cards. Specifically, we perform adaptive threshold segmentation on the original image by OpenCV, and then detect markers by using the ARToolkit library [4].

Here, the marker coordinate frame $O_M X_M Y_M Z_M$ and camera coordinate frame $O_C X_C Y_C Z_C$ are established to describe the marker locating process. The displacement of each marker corner in marker coordinates $p_M$ is preset combined with marker size, and its displacement in the camera coordinate frame $p_C$ is then calculated after the marker is detected. Suppose $\tilde{p}_C = (p_C, 1)^T$, $\tilde{p}_M = (p_M, 1)^T$, the transformation matrix $M_{MC}$ can be calculated by

$$ \tilde{p}_C = M_{MC} \tilde{p}_M. \qquad (1) $$

Since the marker center is the origin of $O_M X_M Y_M Z_M$, let $\tilde{p}_M = (0, 0, 0, 1)^T$, the position of marker is estimated.
## 3.3 Aligned Projection

In order to make the observed character consistent with the tangible object, we project it on the surface of card directly. Therefore, alignment of the virtual object to the card needs to be considered.

From Section 3.2, the displacement of marker in camera coordinate frame $p_C$ is calculated. Suppose its displacement in world coordinate frame is $p_W$, and the displacement of virtual object in world coordinate frame is also $p_W$. Let the displacement of virtual object in projector coordinate frame be $p_P$, the displacement in pixel coordinate frame be $p_i$, and the displacement in virtual scene coordinate frame be $p_v$.

Define $\tilde{p}_W = (p_W, 1)^T$, which can be obtained from the following equation,

$$ \tilde{p}_W = M_{WC}^{-1}\tilde{p}_C, \qquad (2) $$

where $M_{WC}$ is the transformation matrix from $O_W X_W Y_W Z_W$ to $O_C X_C Y_C Z_C$, that is, the external parameter matrix of the camera.

Define $\tilde{p}_P = (p_P, 1)^T$, which can be calculated by the following equation,

$$ \tilde{p}_P = M_{WP}\tilde{p}_W, \qquad (3) $$

where $M_{WP}$ is the transformation matrix from $O_W X_W Y_W Z_W$ to $O_P X_P Y_P Z_P$, that is, the external parameter matrix of the projector.

Considering distortion, let $p_P = (x_P, y_P, z_P)$, $\tilde{p}_i = (p_i, 1)^T$. From the following equation we get $\tilde{p}_i$,

$$ \tilde{p}_i = M_P \begin{bmatrix} (1 + k_1 r^2 + k_2 r^4 + k_3 r^6)x + 2p_1 xy + p_2(r^2 + 2x^2) \\ (1 + k_1 r^2 + k_2 r^4 + k_3 r^6)y + p_1(r^2 + 2y^2) + 2p_2 xy \\ 1 \end{bmatrix}, \qquad (4) \\ \text{with } x = \frac{x_P}{z_P}, y = \frac{y_P}{z_P}, r = \sqrt{x^2 + y^2}. $$

Here, $M_P$ is the internal parameter matrix of the projector, and $k_1, k_2, k_3, p_1, p_2$ are the distortion coefficients of the projector.

Define $\tilde{p}_v = (p_v, 1)^T$, which can be calculated by the following equation,

$$ \tilde{p}_v = M_{iv}\tilde{p}_i, \qquad (5) $$

where $M_{iv}$ is the transformation matrix from $O_i X_i Y_i$ to $O_v X_v Y_v$ calculated by the following equation,

$$ M_{iv} = s \begin{bmatrix} 2/C & 0 & -C/R \\ 0 & 2/R & 1 \\ 0 & 0 & 1 \end{bmatrix}. \qquad (6) $$

Here, $C \times R$ is the resolution of the projector, and $s$ is the scale factor of virtual orthogonal camera.
## 3.4 Physical Simulation

To be more interesting, the motion state of virtual characters can change from walking to running, and vice versa. The state updates are controlled by the velocity information of the tracked target with motion graph. During interaction, participants control the virtual object by moving the card. At a lower speed, the character will be in walking state, at a higher speed, the character will be in running state. If the card is static, then the character will be in idle state. Meanwhile, there are physical interactions among virtual characters, such as collisions. We utilize the physics engine to simulate the physical interaction among virtual objects.

## 3.5 Storyboard

In our work, four types of scene elements are supported from the perspective of interaction. The story background is the initial setting and the scene elements include the direct scene elements, indirect scene elements, and virtual characters. The direct scene elements mainly include the sun, trees and barriers, which are directly driven by corresponding cards to update their positions. The indirect scene elements are computer generated effects, such as the ball controlled by the character, or artificial visual effects, which are not controlled by the user, but driven by the storyline. The virtual character is directly controlled by card. Its position is bound with card's position, while the motion state is driven by card's velocity.

# 4 Experiment

Our experiment platform is equipped with Intel Core i5 1.6GHz CPU, 8GB memory, 64-bit Windows 10 operating system, and the development environment is Unity 2018.1 and Visual Studio 2013.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Shooting game</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Fig. 3. Passing game</figcaption></figure>

**4.1 Shooting Game**

To reflect more scene elements, we design a shooting game (see Fig. 2), which contains direct scene elements, the barriers, indirect scene elements, an arrow representing initial velocity and a ball. The user moves the cards in a certain time, to give the ball an initial velocity to shoot. The trajectory of the ball, which can be changed by barriers controlled by user, is presented through physical simulation.

**4.2 Passing Game**

Considering multi-user interaction scenario, we design a passing game based on storyboard (see Fig. 3). Two participants manipulate the cards to drive the virtual characters move on the tabletop. Character's motion state can switch smoothly from idle to walk or from walk to run in accordance with participants manipulation. When the character meets other objects in the scene, the event trigger mechanism will respond accordingly. For example, when touching a ball on the ground, the character will pick up the ball, then walk with the ball, and finally pass the ball. The speed of character determines the force applied to the ball when it is thrown. The player can also control a board to catch the ball. When the card is moved or rotated, the pose of the board will also change.

**5 Conclusion**

We propose a projector-based interaction method with fusion of virtuality and reality, supporting participants to interact with real objects like cards on the desk or table. Cards are interaction command input, containing both the pose and velocity information. Compared with existing methods, our work makes full use of the pose and velocity information to provide virtual objects physical properties, allowing interaction have more physical effects.

The limitation is that it is hard to detect when the card is moved too fast. Therefore, we plan to improve the performance of tracking to interact with the cards more naturally.
References

1. Bang, C.y., Lee, J., Jung, H.w., Choi, O.y., Park, J.I.: Tangible interactive art using marker tracking in front projection environment: the face cube. In: International Conference on Entertainment Computing. pp. 397–404. Springer (2010)

2. Falcao, G., Hurtos, N., Massich, J.: Plane-based calibration of a projector-camera system. VIBOT master 9(1), 1–12 (2008)

3. Hatanaka, T., Hayashi, T., Suzuki, K., Sawano, H., Tuchiya, T., Koyanagi, K.: Dream board: a visualization system by handwriting recognition. In: SIGGRAPH Asia 2013 Posters, pp. 1–1 (2013)

4. Kato, H., Billinghurst, M.: Marker tracking and hmd calibration for a video-based augmented reality conferencing system. In: Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99). pp. 85–94. IEEE (1999)

5. Marco, J., Cerezo, E., Baldasarri, S., Mazzone, E., Read, J.C.: User-oriented design and tangible interaction for kindergarten children. In: Proceedings of the 8th International Conference on Interaction Design and Children. pp. 190–193 (2009)

6. Park, H., Park, J.I.: Invisible marker tracking for ar. In: Third IEEE and ACM International Symposium on Mixed and Augmented Reality. pp. 272–273. IEEE (2004)

7. Rekimoto, J., Ayatsuka, Y.: Cybercode: designing augmented reality environments with visual tags. In: Proceedings of DARE 2000 on Designing augmented reality environments. pp. 1–10 (2000)

8. Sony: Xperia torch homepage, https://developer.sony.com/develop/xperia-touch/. Last accessed 30 Apr 2020

9. Villar, N., Cletheroe, D., Saul, G., Holz, C., Regan, T., Salandin, O., Sra, M., Yeo, H.S., Field, W., Zhang, H.: Project zanzibar: A portable and flexible tangible interaction platform. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. pp. 1–13 (2018)

10. Wellner, P.: Interacting with paper on the digitaldesk. Communications of the ACM 36(7), 87–96 (1993)

11. Willis, K.D., Shiratori, T., Mahler, M.: Hideout: mobile projector interaction with tangible objects and surfaces. In: Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction. pp. 331–338 (2013)

12. Wilson, A.D.: Playanywhere: a compact interactive tabletop projection-vision system. In: Proceedings of the 18th annual ACM symposium on User interface software and technology. pp. 83–92 (2005)

13. Wu, Q., Wang, J., Wang, S., Su, T., Yu, C.: Magicpaper: tabletop interactive projection device based on tangible interaction. In: ACM SIGGRAPH 2019 Posters, pp. 1–2 (2019)

14. Zhang, Z.: A flexible new technique for camera calibration. IEEE Transactions on pattern analysis and machine intelligence 22(11), 1330–1334 (2000)