<figure><img src="image_1.png" /></figure>

HAL
open science

# A Tabletop System Using Infrared Image Recognition for Multi-user Identification

Shota Suto, Susumu Shibusawa

► To cite this version:

Shota Suto, Susumu Shibusawa. A Tabletop System Using Infrared Image Recognition for Multi-user Identification. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. pp.55-62, 10.1007/978-3-642-40480-1_4 . hal-01501783

HAL Id: hal-01501783

https://inria.hal.science/hal-01501783

Submitted on 4 Apr 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Tabletop System Using Infrared Image Recognition for Multi-user Identification

Shota Suto<fnref n="1" /> and Susumu Shibusawa <fnref n="2" />

**Abstract.** Many tabletop systems have been developed to facilitate face-to-face collaboration and work at small meetings. These systems often require users to attach sensors to their bodies to identify their positions, but attaching a sensor to one's body can be bothersome and annoying, and user position and posture may be restricted depending on where the sensor is attached. We have proposed a technique for estimating user position in a tabletop system by image recognition and implemented a tabletop system having a user position identification function incorporating the proposed technique. This technique first obtains touch points and hand-area information from touch operations performed by the user, and establishes an association between the touch points and hand from those positional relationships. Since the direction in which a hand is extended can be derived from that hand's touch information, the position of the user of the touch points belonging to that hand can be estimated. As part of this study, we also implemented a photo-object manipulation application, which has a function for orienting a photo object to face the user based on the results of the above user-position estimation technique. We performed an experiment to evaluate the position identification rate, and found that the proposed technique could identify user position with high accuracy.

**Keywords:** Tabletop system, Image recognition, FTIR, Multi-touch, User position identification, Area extraction

# 1 Introduction

A tabletop system facilitates face-to-face collaboration at small meetings or any setting where people come together to work as a group. It enables all users to manipulate or examine displayed information and enables a variety of input operations to be executed by touching displayed information and performing gestures. Research surrounding tabletop systems with these features has been active including studies of interactive techniques [1], [2], development of information display technologies [3], [4], and the use of tabletop systems for supporting collaborative work [5], [6], [7].

<footnote n="1">¹Ibaraki University, Graduate School of Science and Engineering, Nakanarusawa 4-12-1, Hita-
chi, Ibaraki 316-8511, Japan</footnote>

<footnote>12nd304x@hcs.ibaraki.ac.jp</footnote>

<footnote n="2">²Ibaraki University, Dept. of Computer and Information Sciences, Nakanarusawa 4-12-1, Hita-
chi, Ibaraki 316-8511, Japan</footnote>

<footnote>sibusawa@mx.ibaraki.ac.jp</footnote>
In a tabletop system, objects can be oriented in various ways by touch gestures performed by multiple users, which means that a user may find it difficult to understand the text or photo of an object that is currently not facing in the user's direction. In response to this problem, several techniques have been developed to identify user positions by attaching sensors to chairs or the users themselves [8], [9], [10], and these technique are used to automatically modify the orientation of an objects according to the position of the user manipulating that objects. For example, Diamond Touch [9], registers beforehand each user and the user's position by having each user sit on a conductive sheet for user-identification purposes, and uses this information to determine the position of the user whenever the user is identified. However, identifying user position by attaching sensors to chairs or people can be troublesome, and time must be devoted to learning how to use sensor equipment. User posture may also be restricted depending on where the sensor is attached.

On the other hand, research has been performed on an interactive system that extracts images of body extremities using image recognition technology so that physical movements performed by the user can be used as input operations [11]. This kind of interactive system using image recognition negates the need for wearing a sensor thereby enabling users to use the system in a free and natural manner.

We propose a technique for estimating user position by image recognition in a tabletop system and construct a tabletop system incorporating this technique. This technique negates the need for wearing a sensor and removes restrictions depending on where the sensor is attached. This system uses a frustrated total internal reflection (FTIR) multi-touch panel and obtains touch points and hand-area information from user touch operations using image recognition. It establishes an association between the touch points and hand from those positional relationships. Since the direction in which a hand is extended can be derived from that hand's area information, the position of the user of the touch points belonging to that hand can be estimated. This study includes the implementation of a photo-object manipulation application that enables users to manipulate photo objects on the tabletop system by touch operations. The application has a function for orientating a photo object to face the user according to the results of estimating user position from touch gestures.

# 2 Proposed System

## 2.1 System Configuration

For this study, we designed a tabletop system with a multi-touch panel capable of touch-point recognition by the FTIR method [12]. This system features an infrared Web camera installed beneath the table to capture infrared images on the tabletop by picking up the acrylic panel on the table via a mirror. The system also includes a projector connected to a personal computer (PC) to display photo objects to users by projecting images onto the acrylic panel from underneath the table. In addition, tracing paper is pasted onto the acrylic panel to act as a screen and users are presented with information by having the projector project images onto this tracing paper. System configuration is shown in Figure 1.
Finally, to obtain information on hand area, an infrared light is installed on the ceiling above the table. Since a hand on the tabletop will block infrared beams emitted from this light, an infrared shadow corresponding to the hand will form. The Web camera is used to pick up this shadow effect.

<figure><img src="image_3.png" /><figcaption>Fig. 1. System configuration</figcaption></figure>

## 2.2 Overview of User-position Estimation Technique

In a tabletop system, a user typically extends a hand from the edge of the table to manipulate some object. Accordingly, if it were possible to determine the direction from which the hand including touch points is being extended, the position of the user generating those touch points could be estimated.

In FTIR-based touch-point recognition as used here, the system picks up touch points as white light owing to the diffused reflection of infrared light. It also picks up the area occupied by a hand as a shadow generated by the blocking of light emitted from the infrared light installed on the ceiling. At this time, the system uses brightness values in the captured image to differentiate from the background image in two ways. Specifically, it extracts touch areas having a higher brightness value than the background and an area having a change in brightness value as a hand area. This extracted hand area includes the touch areas as a subset. Thus, when focusing on certain touch points, a hand area that includes those touch areas as a subset certainly exists, which means that the direction from which that hand is being extended can be determined from that hand-area information. As a result, the position of the user associated with certain touch points can be estimated since touch points and user position can be indirectly connected through the direction of extension of the hand to which those touch points belong. Extraction of these key areas and estimation of user position are outlined in Figure 2.
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Extraction of key areas and estimation of user position</figcaption></figure>

## 2.3 User-position Estimation Model

The FTIR touch-point recognition system picks up touch points as white light and the hand area as a shadow of infrared light. Establishing a difference with the background image enables the touch areas and the hand area that includes those touch areas to be extracted. An image of a touch area and that of a hand area extracted by background differentiation are shown in Figures 3 (a) and (b), respectively.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Extraction of touch-point and hand areas and their positional relationship</figcaption></figure>

The positional relationship among touch area (TA), touch point (TP), and hand shadow (HS) is shown in Figure 3 (c), where the outer circle and inner circle at the fingertip of HS corresponds to TA and TP, respectively. This technique determines TP by calculating the center of gravity of TA. When we define hand-area *Hand* as the union of TA and HS, TP is an element of *Hand*.

The proposed technique first investigates the attribution relation between *TP* and *Hand*. It next determines from which edge in the image the *Hand* to which *TP* belongs to is extending. It finally estimates user position by drawing a correspondence between that direction and the touch point.
## 2.4 User-position Estimation Technique

More than one hand area may exist on the tabletop at any one time. To recognize individual hand areas, the technique labels areas having connected pixels and assigns the label *L* to each hand area.

Then, when focusing on a certain touch point *TP*, that a hand area *Hand* that includes the coordinates of *TP* as an element exists. The results of labeling can therefore be used to extract label *L[TP(x, y)]* corresponding to the coordinates of *TP*. Here, *L[TP(x, y)]* is equivalent to label *L[Hand]* including *TP*. The hand area corresponding to the targeted touch point can therefore be identified by referring to *L[TP(x, y)]*.

Referring to Figure 3(b), it can be seen that *Hand* consists of a continuous area connected to a certain edge of the image. Accordingly, if that edge can be determined, the direction from which *Hand* is being extended can likewise be determined.

## 3 System Implementation

Our prototype tabletop system has a height of 70 cm, a panel 100 cm × 90 cm in size, and a display manipulation range of 60 cm × 50 cm. The tabletop is shown in Figure 4.

We also implemented an application for manipulating photo objects by touch gestures. This application reads in image data as photo objects and displays them on the tabletop system. It treats touch gestures made by the user as input and generates results in response to those gestures. Touch gestures are listed and described in Table 1. Change direction is an operation which makes the object face the user when touching it with three fingers. An example of the change-direction gesture is shown in Figure 5.

<figure><img src="image_6.png" /><figcaption>Fig. 4. View of tabletop</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 5. Change-direction gesture</figcaption></figure>

<table><caption>Table 1. Touch gestures</caption><thead><tr><td>Operation</td><td>No. of Touches</td><td>Description</td></tr></thead><tbody><tr><td>Move</td><td>1</td><td>Move object</td></tr><tr><td>Zoom in/out</td><td rowspan="2">2</td><td>Change object size</td></tr><tr><td>Rotate</td><td>Rotate object</td></tr><tr><td>Change direction</td><td>3</td><td>Change object's direction to face user</td></tr></tbody></table>
# 4 Evaluation Experiment

## 4.1 User-identification Evaluation Experiment

To assess the accuracy of the user-position estimation technique presented in section 3.4, we conducted a subjective experiment using the 3-finger change-direction gesture. After briefing four subjects on how to perform this gesture, we asked each to perform the gesture ten times on the tabletop system in each of the up, down, left, and right directions. At this time, we recorded actual user position and system-estimated position in order and determined position identification rate by comparing the values obtained. Specifically, denoting the number of times this gesture was performed as $d_{act}$ and the number of times that actual user position agreed with system-estimated position as $d_{correct}$, we computed position identification rate by Eq. (1).

$$ \text{position identification rate} = \frac{d_{correct}}{d_{act}} \times 100 \ [\%]. \qquad (1) $$

The experiment was performed in the evening after sunset considering the possible effects of infrared light contained in sunlight. The prototype tabletop described in section 4.1 was installed in the center of a room and two infrared lights were installed on the ceiling above the table. Each light was 90-cm long incorporating six equally spaced infrared LEDs and the two lights were set 70 cm apart. The distance from the ceiling to the tabletop panel was 185 cm.

## 4.2 Results and Discussion

Average identification rate for the change-direction gesture by four subjects for each of the four tabletop directions and overall average for all directions are shown in Figure 6.

<figure><img src="image_8.png" /><figcaption>Fig. 6. Identification rate for change-direction gesture</figcaption></figure>

Average identification rate for the change-direction gesture was found to be about 96%. Examining these results, it can be seen that hand extension from the up and down tabletop directions could be accurately identified. There were times, however, when hand extension from the left and right tabletop directions could not be accurately
identified. Examples of incorrect hand-extension identification from the left and right directions are shown in Figure 7. Images for a change-direction gesture from the left are shown in Figures 7 (a) and (b) and those for a change-direction gesture from the right are shown in Figures 7 (c) and (d).

<figure><img src="image_9.png" /><figcaption>**Fig. 7.** Change-direction gesture from left and right directions</figcaption></figure>

In the examples of Figure 7, the number of pixels in the hand shadow area crossing an edge are higher in the up direction than in the left and right directions. As a result, the system erroneously judges those tabletop operations to be those of a user positioned at the top of the tabletop (in the up direction). This is because the touch-point coordinates on the captured image cannot necessarily be used to correctly obtain the touch-point coordinates on the acrylic panel.

## 5 Conclusion

We proposed a technique for identifying user position in a tabletop system through image recognition and introduced a prototype tabletop system incorporating this technique. We also implemented a photo-object manipulation application for running on the tabletop and conducted a subjective experiment to evaluate the accuracy of the proposed technique in estimating user position. In this experiment, we recorded actual user position and system-estimated position when subjects made change-direction gestures and calculated the position identification rate by determining the rate of agreement between actual and estimated positions. We found that an average position identification rate of about 96% could be achieved in the case of change-direction gestures, which demonstrates that user position can be estimated by the proposed image
recognition technique. The technique is a simple technique for identifying the user
position and negates the need for wearing a sensor.

Errors in identifying user position occurred in the system as a result of camera image
compensation. An offset between the manipulation area and display area on the tab-
letop prevented the system from correctly recognizing how a hand shadow area was
crossing an edge making it easy for erroneous judgments to occur. We considered that
using a wide-angle camera would be able to capture the entire tabletop panel head-on,
which should eliminate complicated image-compensation processing by the system and
suppress this phenomenon. In future research, we plan to evaluate the position identi-
fication rate during simultaneous operations by multiple users and to add gesture
functions making full use of user position identification. We also look to apply the
proposed technique to face-to-face collaborative work systems and to evaluate its
usability in such an application.

References

1. Clifton, P., Mazalek, P., Sanford, J., Rébola, C., Lee, S., Powell, N.: SketchTop: Design Collaboration on a Multi-touch Tabletop, TEI'11, pp. 333-336 (2011).

2. Haller, M., Brandl, P., Leithinger, D., Leitner, J., Seifried, T., Billinghurst, M.: Shared Design Space: Sketching Ideas Using Digital Pens and a Large Augmented Tabletop Setup, Advances in Artificial Reality and Tele-Existence, pp. 185-196 (2006).

3. Yamaguchi, T., N.acenta, M., Sakurai, S., Itoh, Y., Kitamura, Y., Subramanian, S., Gutwin, C., Kishino, F.: An Evaluation of Perspective Correction Regarding Locations of a User and Displays, IEICE Trans on Information and System, Vol. J91-D, No. 12, pp. 2746-2754 (2008).

4. Sakurai, S., Kitamura, Y., Subramanian, S.: A Tabletop Display with Visibility Control Using Revolving Polarizer, IPSJ Journal, Vol. 50, No. 1, pp. 332-343 (2009).

5. Funamoto, S. and Sugiyama K.: Support and Evaluation for Group Discussions in Seminar Type Lectures Using Table Top Interface, Proc of the 6th Knowledge Creation Support Systems Symposium, pp. 119-126 (2009).

6. Ohashi, M. Itou, J., Munemori, J., Matsushita, M., Matsuda M.: Development and Applica-
tion of Idea Generation Support System Using Table-Top Interface, IPSJ Journal, Vol. 49,
No. 1, pp. 105-115 (2008).

7. Goh, W.-B., Shou, W., Tan, J., Jackson Lum, G.T.: Interaction Design Patterns for Mul-
ti-touch Tabletop Collaborative Games, CHI EA '12, pp. 141-150 (2012)

8. Watanabe K., Takeuchi T., Inoue T., Okada K.: Face-to-Face Collaboration System that
Identifies Individual User, IEICE Transactions, Vol. J91-D, No. 12, pp. 2755-2764 (2008).

9. Dietz, P., Leigh, D.: DiamondTouch: A Multiuser Touch Technology, UIST '01, pp. 219-226 (2001)

10. Annett, M., Grossman, T., Wigdor, D., Fitzmaurice, G.: Medusa: a Proximity-ware Multi-touch Tabletop, UIST '11, pp. 337-346 (2011)

11. Mamiya, H., Sato, T. Fukuchi K., Koike H.: Tabletop Entertainment System with a Finger Opening and Closing Gesture, WISS2007 (2007)

12. Han, J. Y.: Low-cost Multi-touch Sensing Through Frustrated Total Internal Reflection, UIST '05, pp. 115-118 (2005)