Ubi-Finger: Gesture Input Device for Mobile Use
Koji Tsukadaa and Michiaki YasumurabaKeio University Graduate School of Media and Governance, 5322 Endo Fujisawa, Kanagawa 252-8520, Japanb
Keio University Faculty of Environmental Information
tsuka@sfc.keio.ac.jp

ABSTRACT
We propose a novel interface in mobile environment called “Ubi-Finger” that realizes sensuous operations for PDA andinformation appliances by gestures of fingers. Since gestures are usual methods of non-verbal communications andenables sensuous operations for users, many researches on them carried out especially in the field of Virtual Reality.
However most of those existing gesture-input systems are either very expensive or large, and have not been used in
mobile environment. In contrast, Ubi-Finger is a gesture-input device, which is simple, compact, and optimized formobile use. We develop prototype systems, and evaluate how effective our approach works.
Keywords: Gesture Recognition, Ubiquitous Computing, Wearable Computing, Information Appliances

1. INTRODUCTION
As the computers became more downsized and powerful recently, the uses of them have extended to the daily life. With
that, the concern with new human interfaces which focus on the various uses in real world has been growing.
As the popularity of personal digital assistants (PDAs) and cellular phones has increased rapidly in recent years, more
and more computers are used in mobile environment. In such situations, the user interfaces should be compact and easy
to use, compared to conventional computers in office or school environments. Many researchers have developed input
devices in such mobile-computing environment3, 5, 17, 18. But many of them focused on more efficient methods of text
inputs, and have not proposed more natural methods with another modality. In the meanwhile, Mark Weiser proposed avision of a future computing environment called “Ubiquitous Computing” in which many various computers work inharmony19. Now, his vision is going to be realized as the diffusion of information appliances. More and more computersand networks are expected to spread into households in near future, and various kinds of home electronics become
information appliances connected to the networks. Although many appliances become more high-performance andmultifunctional in such an environment, control methods of appliances tend to become more complicated and increase
the difficulty for users to master enormous control methods of various appliances. Therefore, new interfaces that cancontrol various information appliances with simple and friendly operation are desirable10.
This paper focuses on human gestures with fingers to solve the above problems of mobile computing and information
appliances. Generally, in the communication of human emotions or wills, the nonverbal means are more important thanverbal means11. Human gestures are typical examples of nonverbal communication, and help people communicatesmoothly8. Human gestures are used not only in the communication between foreigners or language-disabled, but also
actively used in the usual communication. In this way, human gestures are very useful communication means, andnaturally used by everyone. For these reasons we have focused on the human gestures. Although there are many
researches about gesture input systems especially for Artificial Reality9, 20, most of those existing systems are very
expensive and large, and have not been used in mobile environment. Our method is unique in its approach to apply thegesture input method to control mobile computers and information appliances.
We propose a new wearable interface in mobile environment called “Ubi-Finger” that realizes sensuous operations
for PDA and information appliances with natural gestures of fingers. We also developed prototype systems, and evaluate
how effective our approach works.

2. UBI-FINGER
Ubi-Finger is a compact gesture input device attached on fingers. It realizes sensuous operations of various devices inreal world with gestures of fingers. The main concepts of Ubi-Finger are as follows.
•

Sensuous operations with gestures of fingers.

•

Wearable interface optimized for mobile use.

•

Multiple uses with a single interface.

We describe details of these concepts, discuss the problems, and propose the approach of development.

2.1. Control of Real-world Devices with Gestures
Ubi-Finger realizes operations of real-world devices with gestures of fingers. The benefits of applying the gesture input
method to operations of real-world devices are: (1) easy-to-understand mappings of operations with existing metaphor,(2) active use of corporeality. The former benefit means that we proposed easy-to-understand control techniques byusing existing methods to control devices in real world. Real-world devices naturally offer their properties by providing
their own affordance4. For example a volume knob of an audio amplifier makes people feel like "turning" it (Fig. 1).
Since many users are familiar with these control methods, they should easily control various appliances by using these
metaphor. The latter benefit means that the gesture input method should be one of the most suited methods to makegood uses of corporeality. Since corporeality is considered as a significant factor in Ubiquitous Computing13, it will be
useful for the operations of real- learned by moving one's body is more memorable than the method learned by pushing
many buttons in order.

Figure 2. Selection of devices by pointing.
Figure 1. Affordance of the real-world devices
Upper: “Push” a switch of a room light
Lower: “Turn” a volume knob of an audio amplifier

2.2 Wearable Interface
Fukumoto2 said that there are three factors required for wearable interfaces: “Portability”, “Operationality”, and“Constancy”. That is, wearable interfaces must be compact and light weight, easy to use despite of downsizing, andalways ready to interact with users. However, most of the existing approaches of gesture input systems have focused onvarieties of gestures. Since users must attach sensors to all fingers in those systems, they failed to satisfy the factors ofwearable interfaces such as “Portability” and “Operationality”.
2.2.1. Trade-offs between Varieties of Gestures and User's load on Wearing
To increase kinds of recognizable gestures, we must detect (1) bending of top joint and second joint of each finger, (2)angles of joints between fingers, and (3) tilt angles of a wrist. To satisfy all of the above requirements, users must attachsensors to all fingers and their loads on wearing increased severely. Thus, these approaches don't suit for wearable
interfaces used in the daily life from the aspects of practicability. Moreover, since we mainly aim at common and simple
operations of information appliances, we less likely need to recognize complicated gestures, compared with the case of
the researches on sign languages12, 16.
2.2.2. Reduction of unexpected inputs
Since wearable interfaces should be always ready to use, the methods to differ intended user inputs from unexpected
ones are very important. However, most of the existing gesture input systems haven't cared about the above problem, anddon't provide features to reduce unexpected inputs. In Ubi-Finger, we have attached touch sensors on the side of an index
finger, and proposed a new method to start and stop the gesture recognition by pushing touch sensors with a thumb. By
providing the clear trigger mechanism, we could succeed at reducing unexpected gestures effectively. In the meanwhile,
there are trade-offs between the trigger mechanism and natural operations, we should have more discussions in future.

2.3. Multiple Uses with a Single Interface
Ubi-Finger realizes the operation techniques for multiple devices in real world with a single interface. Although users
can control multiple devices with learnable remote controls, there are significant problems that the correspondence
between functions and buttons becomes more complicated in such existing systems as the number of target devices
increase, and users’ loads on learning gets increased severely.
By using Ubi-Finger, users can control various devices in real world simply and sensuously. The brief steps ofcontrols are as follows.
Select a target device with pointing
A user can select a target device by pointing it with her/his index finger, and send her/his ID via infrared (Fig. 2).
Control the device with gestures
A user can control the target device with gestures of fingers. Since she/he has selected the target device previously, thecontrol methods don't become complicated regardless of increase of target devices.

3. DEVELOPMENT
3.1. Device Architecture
In this section, we describe the device architecture of Ubi-Finger. As shown in Fig. 3, Ubi-Finger consists of three
sensors (a bending sensor, an acceleration sensor, and a touch sensor) to detect gestures of fingers, an infrared transmitterto select a target device in real world and a microcomputer to control these sensors and communicate with a hostcomputer. Each sensor generates the information of motions as follows: (1) a bending degree of an index finger, (2) tilt
angles of a wrist, (3) operations of touch sensors by a thumb. We use (1) and (2) for recognition of gestures, and use (3)for the trigger mechanism to start and stop gesture recognition.

3.2. System Architecture
We describe the system architecture to control real-world devices with Ubi-Finger. As shown in Fig. 4, the system
consists of four main factors: Ubi-Finger device, Ubi-Appliance (an information appliance with an infrared receiver,
LEDs, and a network connection), Ubi-Host (a host computer of Ubi-Finger device), and Ubi-Server (a server to control
Ubi-Appliance).

Figure 3. Basic Concept of Ubi-Finger.
Figure 4. System Architecture of Ubi-Finger.

We explain the system flow briefly. First, a user points at an information appliance (Ubi-Appliance) with Ubi-Finger,
and transmits her/his unique ID via infrared signal. When the Ubi-Appliance receives the signal, it transmits both its own
ID and the user's ID to a server (Ubi-Server), and the Ubi-Server associates these IDs together to get the correspondenceof the user and the Ubi-Appliance. Then, the Ubi-Server transmits a feedback signal to the Ubi-Appliance, and the UbiAppliance shows its state of selection with LEDs. After the user confirms the state of the target Ubi-Appliance, she/he
performs a gesture with Ubi-Finger. A host computer (Ubi-Host) recognizes the gesture, converts it to a specific ID, andtransmits it to the Ubi-Server with the user's ID. The Ubi-Server identifies the target Ubi-Appliance with the user's ID,
converts the gesture's ID to an appropriate command for the Ubi-Appliance, and controls it via network.

3.3. Prototypes
We have developed prototype systems of Ubi-Finger based on the above concepts. We describe the architectures,
technologies, and properties of the prototype systems.
3.3.1. Ubi-Finger Hardware
We explain the prototype of Ubi-Finger hardware (Fig. 5). The
size of the prototype device is as compact as a fingertip. The
device architectures of the prototype are as follows. We attached
a bending sensor (BendMini by Infusion Systems*) on an index
finger and two compact switches on the side of an index finger. In
addition, we have set a 2-axis acceleration sensor (ADXL202E by
Analog Devices), an infrared transmitter, and two LEDs on the
portion of an index finger. We used a microcomputer (Tiny-H8
to convert the information
by Akizuki Denshi Tsusho†)generated by the sensors to numerical values, and transmit them
to a laptop computer (OS: Windows 2000 Professional). Then,the Ubi-Host software on the laptop computer recognizes gesturesby analyzing the input value in real time, and communicates with
Ubi-Server via wireless network.

Figure 5. Prototype of Ubi-Finger Hardware

3.3.2. Ubi-Appliance and Attachable Controller
While the necessary factors of Ubi-Appliance such as an infrared receiver, LEDs, and a network connection will besatisfied by many information appliances in near future, it would take a long time that these information appliances
become popular. Therefore, as shown in Fig. 6, we have developed “Attachable Controller” which can add the necessary
factors of Ubi-Appliance to existing electronic appliances by only attaching it to the appliances. As shown in Fig. 7,
Attachable Controller consists of three subsystems: “Infrared receiver system”, “Network connection system”, and“Device control system”.
Infrared receiver system
The function of this subsystem is to receive infrared signals from Ubi-Finger and show the selection state of UbiAppliance. This system consists of an infrared receiver, two LEDs, a microcomputer (PIC16F84A by Microchip
Technology), and other surrounding circuits.
Network connection system
The function of this subsystem is to communicate with Ubi-Server via network. This system is mainly based on a
product called “PICNIC” released by Tristate‡. PICNIC is a microcomputer which has a LAN interface and functions
just like a web server.
Device control system
The function of this subsystem is to control electronic appliances via infrared or other ways. This system consists ofinfrared transmitter, a convertible EEPROM (Electronically Erasable and Programmable Read Only Memory), a micro*

http://www.infusionsystems.com/
http://www.akizuki.ne.jp/‡
http://www.tristate.ne.jp/†

computer (PIC16F84A by Microchip Technology§), and other surrounding circuits.

Figure 6. Basic Concept of Attachable Controller.

Figure 7. System Architecture of Attachable Controller

4. APPLICATIONSIn this section, we describe three applications of Ubi-Finger.

4.1 Control of Information Appliances.
We have developed an application to control information appliances with Ubi-Finger. When users control electronic
appliances in real world with existing remote controls, they must use multiple remote controls as many as target devices
and learn different control methods with each devices. Although users can control multiple devices with learnableremote controls, the correspondence between functions and buttons becomes more complicated in those approaches, andusers' loads on learning increase more severely. In contrast, Ubi-Finger doesn't have such problems, since users can
simply select target appliances with “pointing”, and control them naturally with gestures by using Ubi-Finger.
We have developed some examples to control electronic appliances such as lights, TVs, component stereos, andvideo recorders with Ubi-Finger and Attachable Controller. We concerned here with the example to control a TV with
Ubi-Finger. We have provided six functions to control a TV, that is, “Turn on/off a TV”, “Turn up the volume”, “Turndown the volume”, “Change to next channel”, “Change to previous channel”, “Mute the volume”. We have proposed
easy-to-understand correspondence between these functions and gestures by using existing metaphors. For example,
since the “turn up the volume” function makes users think of a motion to “turn right a volume knob”, we have allocatedthat function to a gesture to “turn right a wrist”. We show the examples of correspondence between functions andgestures in Fig. 8.

4.2 Help of Efficient Window Scrolling of Computers
We have developed an application to scroll a window efficiently with Ubi-Finger during inputting texts. When users
input texts or code programs with a text editor, it's a popular situation to scroll the window in order to refer other parts ofthe texts. There have been some existing methods to achieve the task such as mouse wheels or touch-pads, however,those approaches are not so efficient since users must leave their fingers from keyboards. By using Ubi-Finger, users canefficiently achieve the task by only moving their index fingers slightly, without leaving their fingers from keyboards (Fig.
9).

4.3 Dynamic Presentation
We have developed an application to give dynamic presentations with gestures. When users give presentations withconventional ways, they had to control presentation software in front of laptop computers. But such control methods are
sometimes bothering, especially when users give presentations in front of large crowds. By using Ubi-Finger, users
hardly need to mind controlling computers, and give presentations more smoothly.
§

http://www.microchip.com/

Figure 9. Help efficient window scrolling of computers
Figure 8. Examples of Operations using Ubi-Finger

5. EVALUATION
We have evaluated the effectiveness of Ubi-Finger's features such as the selection method of target devices with pointing
and the control method of the devices with gestures. We have performed experiments that users control multiple
appliances with the prototype of Ubi-Finger. We have selected 10 test subjects who had never used Ubi-Finger. Theirages are between 23 and 47.

5.1. Evaluation Methods
We have prepared a component stereo and a table light, and gave test subjects the task to control both devices
continuously. We set the distance between target devices and test subjects two meters, and set the distance between eachdevice one meter. We focused on commonly used functions of these appliances in the experiment, that is, “Play”, “Stop”,“Turn the volume up”, and “Turn the volume down” for the component stereo and “Turn on”, “Turn off”, “Brighten thelight”, and “Dim the light” for the table light. Then, we have made questionnaire survey to get their subjective feedbacks,
and asked them oral questions about problems of the prototypes and so on.

5.2. Evaluation Results
First, we round up the typical result of the questionnaire survey. We asked the test subjects “Are the operation techniquesof real-world devices with gestures easy to understand?”, and got answers with four options as follows: “Very easy tounderstand”, “Easy to understand”, “Difficult to understand”, and “Very difficult to understand”. In result, 70% of thetest subjects chose the option of “Very easy to understand”. Along with the users who chose the option of “Easy tounderstand”, 90% of them affirmed our control techniques of real-world devices with gestures (Fig. 10).
Moreover, we asked the test subjects “Do you think the selection techniques of real-world devices by pointing with
fingers attractive?”, and got answers with four options as follows: “Very attractive”, “Attractive”, and “Not very
attractive”, “Never Attractive”. In result, 80% of the test subjects chose the option of “Very attractive”. Furthermore, we
asked them “Are the selection techniques of real-world devices by pointing easy to understand?”, and got answers withfour options. In result, that all of them chose the options of “Very easy to understand” or “Easy to understand”, andaffirmed our selection techniques of real-world devices with pointing (Fig. 11).
In this way, almost all users valued the basic concepts of Ubi-Finger such as selection techniques with pointing andcontrol techniques with gestures. These results clearly show the effectiveness of our approach. In the meanwhile, someproblems of the prototype systems became clear by the oral questions, such as “Cables are troublesome “, “Size of the
Ubi-Finger device should be adjustable”, “Users should be able to add or modify gestures”, “Control of trigger
mechanism is a little difficult”. In addition, our experience shows another problem of “Difficulty in selection of close-inmultiple devices”. We will make efforts to solve these problems.

Figure 10. Evaluation of Operations with Gestures

Figure 11. Evaluation of Operations with Pointing

6. RELATED WORKS
1

5DT DataGlove is a glove-type gesture input device which consists of five bending sensors, an acceleration sensor andso on. 5DT Data Glove mainly purpose to control software in computers, such as Virtual Reality and 3D modeling tool.
Our approach is unique in mainly aiming to control devices in real world such as information appliances.
There are some researches on controlling information appliances with particular remote controls such as FieldMouse
and Air-Real. FieldMouse10 consists of a barcode reader to select a target device and motion-sensing devices such as amouse or an acceleration sensor. FieldMouse realizes controls of target appliances by some motions such as tilting. Our
approach can realize more sensuous controls of appliances by providing easy-to-understand correspondence between
functions of appliances and gestures with existing metaphors. Air-Real6 is a system which consists of remote controls
with laser-pointers, a camera to track motions of the laser-pointers, and a projector to show user interfaces near target
appliances. Air-Real realizes controls of target devices by pointing with the laser-pointer. While Air-Real needs verylarge-scale systems, our approach only needs compact systems. We can apply Ubi-Finger to various appliances only byattaching Attachable Controller on existing appliances.
There are some researches on wearable remote controls such as “Gesture Pendant”. Gesture Pendant15 is a pendanttype wearable device which consists of infrared LEDs and a camera. Gesture Pendant realizes controls of appliances byrecognizing gestures with the camera. Our approach is different at recognizing gestures with wearable sensors. The
advantages by using wearable sensors are that users can perform gestures without caring about views of cameras orexternal noise. In addition, our approach is unique at smoothly selecting and control ling target appliances with pointing
and gestures.
Attachable Computer7 proposed a concept to add existing appliances new functions of showing additional
information by attaching microcomputers and sensors to the appliances. While Attachable Computer focuses on adding
appliances output functions, our Attachable Controller mainly focuses on adding appliances input and control functions.
Gesture Wrist14 proposed a simple gesture input technique, by using an acceleration sensor and sensor electrodes on awrist. Gesture Wrist can recognize several variations of gestures only by wearing a compact device on a wrist. However,since Gesture Wrist doesn't provide the function of clear trigger mechanism to start and stop gesture recognition, it had
still some problems on unexpected recognitions.

7. CONCLUSION
We have proposed a wearable interface in mobile environment called “Ubi-Finger” that realizes sensuous controls ofmobile computers and information appliances with gestures of fingers. We have also developed prototype systems, andevaluated the effectiveness of our approach.

By using Ubi-Finger, users can select target appliances with “pointing”, and control the appliances with gestures offingers. In our approach, users can control multiple devices without bothering about complicate control methods.
Moreover, they can sensuously control various appliances by using existing metaphors and corporeality. Almost allusers valued our approach in the evaluation, and we have confirmed the effectiveness of our approach.
When we solve several existing problems of current prototype systems, we believe that Ubi-Finger will change theinteraction between users and various appliances in Ubiquitous Computing more naturally.

REFERENCES
1. 5DT Data Gloves:
http://www.5dt.com/hardware.html\#glove.
2. Fukumoto, M.: Can You Input on 24-hours!? - Wearable Interfaces -, IPSJ MagazineVol.41, No.2, pp. 123--126(2000). (in Japanese)
3. Fukumoto, M. and Tonomura, Y.: Body coupled FingeRing: Wireless wearable keyboard, Proceedings of the
ACM Conference on Human Factors in Computing Systems(CHI'97), Addison-Wesley, pp. 147--154 (1997).
4. Gibson, J.: The ecological approach to visual perception, Houghton Mifflin Company (1979).
5. HalfKeyboard:
http://www.halfkeyboard.com/index.html.
6. Hoshino, T., Maruyama, Y., Katayama, A., Shibata, Y., and Yoshimaru, T.: Air-Real: Object-Oriented User
Interface for Home Network System, Proceedings of Workshop on Interactive Systems and Software 2001
(WISS’2001), pp. 113--118 (2001). (in Japanese)
7. Iga, S., Itoh, E., Higuchi, F. and Yasumura, M.: Attachable Computer: Augmentation of Electric Household
Appliances by Fit-up Computer, Proceedings of Asia Pacific Computer Human Interaction (APCHI'98), pp. 51-56 (1998).
8. Kita, S.: Why do people gesture?, Cognitive Studies, Vol.7, No.1, pp. 9--21 (2000). (in Japanese)
9. Krueger, M.: Artificial Reality II, Addison-Wesley (1990).
10. Masui, T. and Siio, I.: Real-World Graphical User Interfaces, Proceedings of The International Symposium on
Handheld and Ubiquitous
Computing (HUC'2000), pp. 72--84 (2000).
11. Mehrabian, A.: Silent messages: Implicit communication of emotions and attitudes, Wadsworth, 2nd edition(1981).
12. Ohki, M., Sagawa, H., Sakiyama, T., Oohira, E., Ikeda, H. and Fujisawa, H.: Pattern recognition and synthesis forsign language translation system, Proceedings of the first annual ACM conference on Assistive technologies,
ACM Press, pp. 1--8 (1994).
13. Rekimoto, J.: “The World through the Computer: Computer Augmented Interaction with Real World
Environments”, in Designing Communication and Collaboration Support Systems, Gordon and Breach Science
Publishers, 1999.
14. Rekimoto, J.: GestureWrist and GesturePad: Unobtrusive Wearable Interaction Devices, Proceedings of 5thInternational Symposium on Wearable Computers (ISWC'2001) (2001).
15. Starner, T., Auxier, J., Ashbrook, D. and Gandy, M.: The Gesture Pendant: A Self-illuminating, Wearable,
Infrared Computer Vision System for Home Automation Control and Medical Monitoring, Proceedings of 4thInternational Symposium on Wearable Computers (ISWC'2000) (2000).
16. Starner, T., Weaver, J. and Pentland, A.: A Wearable Computer Based American Sign Language Recognizer,
Proceedings of First International Symposium on Wearable Computers (ISWC'97) (1997).
17. Sugimoto, M. and Takahashi, K.: SHK: Single Hand Key Card for Mobile Devices, Proceedings of the ACM
Conference Companion on Human Factors in Computing Systems(CHI'96), Addison-Wesley, pp.7--8 (1996).
18. Twiddler2:
http://www.handykey.com/site/twiddler2.html.
19. Weiser, M.: The Computer for the 21st Century, Scientific American (International Edition), Vol.265, No.3,
pp.66--75 (1991).
20. Zimmerman, T. and Lanier, J.: A Hand Gesture Interface Device, Proceedings of the ACM Conference on Human
Factors in Computing Systems (CHI'87), Addison-Wesley, pp. 235--240 (1987).