The 18th IEEE International Symposium on
Robot and Human Interactive Communication
Toyama, Japan, Sept. 27-Oct. 2, 2009

WeC2.1

Rebo: A Remote Control with Strokes
Kazuki Kobayashi, Yutaro Nakagawa, Seiji Yamada,Shinobu Nakagawa, and Yasunori Saito

Abstract— This paper describes a new remote control operable with stroking its surface. There are lots of remote controls in
our houses such as TV remote controls, air conditioner remotecontrols, and so on. However, when we use a remote control, we
need to look at both the fingers and an appliance that we wantto control. It is not highly problematic for young people, butelderly people have a difficulty in operating remote controls. It
will be comfortable for people to use a remote control withoutlooking at the fingers. In this study, we propose a remotecontrol, Rebo, operable with stroking its surface and apply
to a TV interaction system. The developed remote control has
three advantages; (1) Familiarity, (2) Function awareness, and(3) Stroke operation. Those enable users to have familiarity
with it, to easily notice its functions, and to use it withoutlooking at the fingers. The feature of Rebo in comparison withconventional button-based remote controls is the tolerance for
mistakes because it enables unfamiliar users to home electric
appliances to use it casually without fear of mistakes andunexpected behavior.

I. I NTRODUCTION
There are various remote controls in our home. We have
TV remote controls, air conditioner remote controls, room
light remote controls, and so on. Home electric remotecontrols are commonplace devices. However, there are someproblems in relation to remote controls. For example, many
remote controls confuse us when we use home electric
appliances. Universal remote[1] that aggregates functions ofvarious remote controls is one of technical solutions for thisproblem. On the other hand, users need to search a button
that they want and to correctly move the fingers to push it.
It is not highly problematic for young people, but elderly
people have a difficulty in operating remote controls. It will
be comfortable to use a remote control without looking attheir hands. Such a remote control is also comfortable toyoung people familiar to home electric appliances.
In this paper, we propose a remote control, Rebo, likea pet animal. Users stroke its surface with the fingers to
Kazuki Kobayashi is with Graduate School of Science and Technology, Shinshu University, 4-17-1 Wakasato, Nagano City, 380-8553 Japankkobayashi@cs.shinshu-u.ac.jp
Yutaro Nakagawa is with Design Department, Osaka University of Arts,469 Higashiyama, Kanan-cho, Minami Kawachi-gun Osaka, 585-8555 Japan

yutarou19870307@yahoo.co.jp
Seiji Yamada is with National Institute of Informatics, 2-1-2 Hitotsuba
shi, Chiyoda, Tokyo 101-8430, Japan and The Graduate University for
Advanced Studies, Shonan Village, Hayama, Kanagawa, 240-0193 Japanseiji@nii.ac.jp
Shinobu Nakagawa is with Design Department, Osaka University of Arts,469 Higashiyama, Kanan-cho, Minami Kawachi-gun Osaka, 585-8555 Japanshinobu@osaka-geidai.ac.jp
Yasunori
University,

Saito is with Faculty of Engineering, Shinshu
4-17-1 Wakasato, Nagano City, 380-8553 Japansaitoh@cs.shinshu-u.ac.jp

978-1-4244-5081-7/09/$26.00 ©2009 IEEE

control various home electric appliances far from them. Rebo
has three advantages which conventional remote controls
have never had; (1) Familiarity, (2) Function awareness,and (3) Stroke operation. Its life-like appearance and facial
expressions make it familiar with users. As for Function
awareness, we adopt Action sloping[2], [3] that enables themto notice its functions. Peripheral visual feedback provides a
stroke operation of it. Therefore, users do not have to seek
a button that they want and use it easily.
As an example of the proposed remote control, we develop
an interaction system with TV. We confirmed that we caneasily use it with stroking its surface.
The rest of this paper is presented as follows. Related work
is described in the next section, and the concept and thebasic architecture are explained in Section III. A TV remote
agent as an implementation of a remote control agent, anddiscussion about its advantage of the TV remote agent andthe interaction system are described in Section IV and V,
respectively. Finally, we conclude our work in Section VI.
II. R ELATED W ORK
Rebo is a device which a user can controls various
appliances by stroking with the fingers, and also a kind oftouch-based remote control. In human-computer interaction
and user interface, such touch-based remote controls havebeen developed so far. Since Rebo utilizes a user’s strokewith the fingers, he or she does not need to visually recognize
and understand any icon, signal, and acknowledgment froma remote control. The user can concentrate on the feedback
from appliances, but remote controls. The advantage of ourapproach partially resembles that of “eyes-free” approach formany people including visually-impaired persons and elder
people.
Shengdong et al. developed a potable audio player having
eyes-free menu selection which utilizes touch input andreactive audio feedback[4]. Vincent et al. also applied atouch-based operation to control headphones[5]. Stephen
et al. designed eyes-free multimodal interaction for Wearable devices[6]. Particularly for using mobile and wearable
computers, input or output problems occur due to limited
screen space and interaction techniques. They overcame theproblems by developing a 3D audio radial pie menu and asonically enhanced 2D gesture recognition system on a beltmounted PDA.
Micire et. al proposed a multi-touch remote control forhuman-robot interaction[7]. They tried to apply a multi-touch
table to control a physical robot agent and described analysis751

Fig. 1.

Remote Control Agent: Reboon users’ interaction styles in complicated functions of themulti-touch interface.
These previous studies attempted just to enrich usability of
the user interface by touch-based devices like touch panels.
In the view of developing touch-based interface devices,
Rebo is very similar to the previous studies. However, weconsider Rebo’s touch-based approach is not only for usability of interface, but also for Partial execution for Function
awareness as discussed later. Thus Rebo provides novel advantage of touch-based interface in a human-adaptive system.
There are more concrete control methods related to Rebo.
For example, stroking alphabets and numbers, by Graffitilike unistrokes, on a touch-pad device has been developed[8].
This kind of function can be implemented on Rebo to changea TV channel directly.
There are also studies on multi-modal remote control ofappliances using speech, gesture, and so on[9]. Balchandran
et al. have presented a novel prototype system that implements a multi-modal interface to control the television[10].
The system combines the standard TV remote control with a
dialog management based natural language speech interface
to allow users to efficiently interact with the TV. Unfortunately, in contrast with Rebo, these systems are unstable due
to complicated speech recognition although natural languagebased interface is provided.
Another view point of Rebo is a universal remote control by which a user can control various home electric
appliances. One of the most advanced universal remotecontrols is Personal Universal Controllers (PUC). PUC isan approach for improving the interfaces to complex appliances by introducing an intermediary graphical or speechinterface[11], [12]. A PUC executes two-way communication
with everyday appliances; first downloading a specificationof the appliance’s functions, and then automatically creating
an interface. At the final stage of our work, this kind offunction such as an intelligent universal remote control will
be implemented to Rebo.
Partial execution, one of Rebo’s advantages, is an exten-

Fig. 2.

System Architecturesion of our previous work on Action sloping for making
a user to notice a robot’s function[2], [3]. In our previous
work, Partial execution was restricted in robot’s physicalfunctions like robot’s motions. In contrast with it, in thiswork, Partial execution is generalized to partially execute
various unphysical functions including feedback expression
such as an animation on a two-dimensional surface.
III. AGENT AS A R EMOTE C ONTROL
Figure 1 shows the developed remote control agent, Rebo.
Rebo is an agent of remote controls of home electric appliances. It has a user-friendly appearance; a smooth surface
for stroking and a back side fit for users’ thighs. Rebo isdifferent from conventional remote controls in being used. It
is 249 mm long, 146 mm wide, and 96 mm high.
A. System Architecture
Fig. 2 shows the architecture of our system. In the current
stage, a main processing device (PC) is not embedded in
Rebo. In the future, all of devices including batteries andPC will be embedded in the body of Rebo.
Rebo has a display for facial expressions, full-color LED
modules, and three touch devices (iPod touches) are embedded in the body. Two iPods are used as touch sensors andthe remaining one is used for displaying information. The
iPods used as touch sensors are covered with a soft andsmooth cloth. Touch positions are sent to a host PC throughWiFi from iPod’s browser, Safari. We adopted Open Sound
Control (OSC)[13] as a communication protocol. When auser touches the browser on an iPod, a JavaScript programis called and then a PHP function sends OSC packets to aconnected PC.

752

One of the embedded iPods is used for the facial expression display. An OSC server on the PC deals with OSC packets and rewrites a local XML file. A JavaScript program on
the iPod for facial expression periodically accesses the XML
file using Ajax. According to the XML file description, theprogram on the iPod changes gif animations. The concrete
facial expressions are described in Section IV.
Three full-color LED modules embedded in Rebo provide
a user with some light-based feedback when he or she operates Rebo. The user can grasp the result of their operationfrom their peripheral vision. The LED modules are controlled
by the PC through USB connection. We use a programming
language, Processing to deal with OSC packets, the LED
modules, and XML file writing/reading. It can communicate
with home electric appliances through infrared signals.
In the following paragraphs, we explain three advantagesof Rebo: Familiarity, Function awareness, and Stroke operation.
B. Familiarity
Rebo has a life-like appearance (Fig. 1) and facial expressions to acquire familiarity with a user. We adopt theconcept of “intermediate entity between artifacts and animate
beings”. Rebo is not only a tool but also a partner. The
body of Rebo is covered with soft and bouncy cloth andit is pleasant to the touch. When the user strokes Rebo,
it changes its facial expressions to inform him or her ofvarious emotional states of Rebo. We consider this makes
the user more comfortable with Rebo. Therefore, interaction
with Rebo can be the purpose of the user as well as operationof home electric appliances. We believe that this concept
plays an important role for establishing familiarity between
users and Rebo.
C. Function Awareness
Function awareness is awareness that which operation
causes which function of a machine. For example, whena user purchases a new home electric appliance, he or she
reads the owner’s manual before using it to understand its
functions. However, searching for the desired functions andstudying how to operate the remote control often can becomplicated and difficult. Therefore, having a way for theuser to comprehend a machine’s functions without reading
the manuals would be useful.
In this study, we adopt Action sloping[2], [3] that enables
a user to notice its functions to achieve Function awareness. Action sloping makes machines provide feedback that
gradually changes in intensity as the user carries out given
actions. As for intensity of feedback behavior, it is assumed
that the volume, frequency and quality of representation arechanged. Some patterns of feedback behavior that can beused are changing lights displayed, sounds emitted, or timingof movements.
Figure 3(a) shows the conventional method for exhibiting
feedback behavior. When a user operates a machine or
remote control with performing triggering action, it executes

Fig. 3.

Intensity of Feedback Behavior

a function. The user observes the machine’s behavior as feedback behavior. However, the machine provides no feedback
if the user operates it without triggering action. In contrast,with Action sloping, shown in Fig. 3(b), the machine exhibits
feedback behavior even when the user takes actions otherthan the triggering action. Changing the intensity of feedback
behavior in response to the user’s action, the robot willencourage the user to perform the triggering action.
For example, we can consider a case of using a TV remote
as follows. When a power button is pushed (triggering action
is performed), the TV turn off (feedback behavior). A user
receives feedback of his or her operation when the TV turns
off. In contrast, Action sloping constantly provides feedback
before the TV turns off. Before the user touches the button,
the TV decreases the image dimensions to express “the TV is
likely to turn off”. This is a just example and needs a sensor
that can measure the distance between the fingers and the
button.
In this study, a user interacts with Rebo by stroking it.
This kind of interaction manner bears a continuous actionand easily achieves Action sloping. We utilize this featureand propose “Partial execution” as a way for the user tonotice functions of the machines. Partial execution is afeedback expression method that provides a meaning ofusers’ operation by showing a part of an executing function.
For example, when the user strokes Rebo for a short time,
Rebo decrease the image dimensions of the TV for a shorttime. If the user strokes Rebo for a long time, it decreases
the image dimensions and turned off the TV power.
Feedback expressions of the machine strongly depend on
its functions and are designed for each application.
D. Stroke Operation
Rebo has no button and a user strokes its surface to control
devices. In this kind of operation, he or she does not needto move the fingers correctly and not seek the button that heor she wants. It is more comfortable for the user to gaze at
a machine that he or she wants to control than to gaze at the
remote control, because the feedback from the machine ismore important than that from the remote control. However,
the operation by stroking is different from the operation bypushing a button in a physical feeling. We then use LED
lighting as an acknowledgment of the operation.

753

Fig. 4.

Fig. 5.

Channel Switching

Volume of Sound Changing

The advantage of the LED lighting has less interference
against auditory and visual information. If a remote machinesuch as a TV provides auditory information, it is inadequatethat a remote control provides auditory feedback. In contrast,
expressions by simple LED lighting can be grasped by users
from their peripheral vision. It is comfortable for them to use
Rebo without looking at it. We call this method “Peripheral
feedback”. Peripheral feedback has less interference against
their central visual field. Peripheral feedback is similar toperipheral display[14], [15], and its concept is based on ourview that “the central player is the remote machine and thebackseat player is the remote control”.
IV. TV R EMOTE AGENT
We developed a TV remote agent based on the proposed
remote control agent to implement all of the three advantages in the previous section. In addition, we also designed
feedback behavior of the TV and constructed an interaction
system.
A. Stroke Operation
Functions we implemented in the system are channel
switch, volume of sound change, and power on/off. The TV
screen was implemented on a PC. In this section, operation
methods of Rebo and feedback behavior of the TV are
described.

1) Channel Switch: Figure 4 shows a series of screenshots
in which the positions of two video pictures are changed asa user strokes the surface of Rebo right and left. When theuser widely strokes the surface of Rebo with her fingers, avideo picture goes the outside of the TV frame and anothervideo picture comes into the frame. When the user strokes
Rebo for a short time, a part of another video picture comes
into the frame and then goes out of the frame automatically.
In this implementation, we used recorded TV movies anddid not use TV tuners. It is technically possible to capture
TV movies and show them in the monitor.
2) Volume of Sound Change: When a user strokes Reboup and down, the volume indicator (a vertical bar) is shownon the video picture (Fig. 5). The length of the indicator ischanged as the movement distance of the fingers is changed.
3) Turn On and Off: A series of screenshots in the
top of Fig. 6 shows a video picture being reduced in animage dimensions. When a user touches Rebo for more than
one second, the image dimensions of the video picture are
reduced and finally disappeared. If the user stops touching
Rebo before the picture is disappeared, the video picture isenlarged and goes to an ordinary size. On the other hand, aseries of screenshots in the bottom of Fig. 6 shows a video
picture being enlarged. When the TV has been turned off, thepicture is gradually enlarged while the user touches Rebo.

754

Fig. 6.

Turn On and Off

Fig. 7.

Facial Expressions

When the user stops touching before the picture becomes toa maximum size, it is automatically going down and thendisappeared.
B. Facial Expressions and LED lighting
Figure 7 shows implemented facial expressions of Rebo.
It goes a sleep mode when its surface is not touched for 10
seconds. In the sleep mode, Rebo closes the eyes and slowly
blinks the LED in a blue color. When a user touches Reboin the sleep mode, it wakes up with rapidly blinking theeyes and the LED in yellow shown in the upper 3rd picturein Fig. 6. When the user strokes Rebo right and left in thechannel switching operation, it lights the LED in red with asmile. When the user strokes it up and down in the volume
of sound change operation, it lights the LED in purple witha smile as well. If Rebo is not being touched, it lights the
LED in green and sometimes blinks the eyes.
C. Partial Execution for Function Awareness
By the operation of Rebo described above, users easily
operate it without looking at their fingers and searching
buttons. They can concentrate on the feedback from the TV
monitor, but Rebo. Animations of the video picture such as
sliding and zooming are implemented as Partial execution for

Function awareness. They can understand the meaning of theanimation based feedback from the TV before the function iscompletely executed. Therefore, they can immediately notice
the function of the TV by stroking Rebo.
V. D ISCUSSIONSIn this paper, we developed a TV based interaction system
as an example of Rebo. In this section, we mainly discussa difference between Rebo and conventional button-based
remote control in an operation method.
Let us consider to implement a button-based remote control with functions equivalent to Rebo in a TV interaction.
The button-based remote control needs to have two buttons
for switching channels, another two buttons for changing
volume of sound, and a button for turning on and off. It
may be possible to develop such a useful button-based remotecontrol because of a small number of buttons. We can design
the buttons with different in colors and shape, and that will
enables users to easily distinguish them by visual and haptic
feelings.
The feature of Rebo in comparison with such button-based
remote control is the tolerance for mistakes. In our proposedsystem, a user concentrates on the feedback from a TV
because he or she can operate Rebo with strokes without755

looking at it. The user can cancel the function before it iscompletely executed because he or she understands whichfunction is executed by observing the feedback based on
Partial execution from the TV. For example, when the user
wants to change the volume and operates Rebo, he or shetries other ways in stroking by observing the video picture
moving left and right. However, in the button-based remotecontrol, when the user pushes a button, the channel waschanged immediately. The user has to search a button thattunes the previous channel and push it, and then search abutton for changing the volume.
The tolerance for mistakes in Rebo enables unfamiliar
users to home electric appliances to use it casually withouta fear of making mistakes and unexpected behavior. We
have a plan to investigate the effect of this tolerance in anexperiment with elderly people.
As for functionality of the TV system with Rebo, it canbe improved. If it would provide a function that a user writesa channel number on its surface to switch channels, it will
provide functions equivalent to popular TV remote controls.
Also Rebo’s familiarity and Function awareness are obviously advantages because conventional remote controls
have never had them. However, these advantages should be
evaluated through quantitative and qualitative experimentswith adequate participants. Thus we are planning to conduct
such experiments.
We consider that the concept of Rebo is applicable to manyother remote controls besides a TV. It is our future work todevelop such remote controls and functions for a TV.
VI. C ONCLUSION
In this paper, we propose an agent as a remote control forhome electric appliances, Rebo and develop a TV interaction
system. Rebo has three advantages; (1) Familiarity, (2) Function awareness, and (3) Stroke operation. Life-like appearance and facial expression makes it familiar with users. We
implemented five facial expressions as a life-like behavior
and video picture animations as feedback from the TV toenable users to easily notice its functions. LED lighting
provided Peripheral feedback for users to concentrate on the
feedback from appliances, but remote controls. The featureof Rebo in comparison with a button-based remote control
is the tolerance for mistakes. This enables unfamiliar users
to home electric appliances to use it casually without a fearof making mistakes and unexpected behavior.
In the next stage of our study, we have a plan to apply
Rebo to various home electric appliances and enable users
to seamlessly control them depending on the context.

[4] S. Zhao, P. Dragicevic, M. Chignell, R. Balakrishnan, and P. Baudisch,“Earpod: eyes-free menu selection using touch input and reactive audio
feedback,” in Proc. of the SIGCHI conference on Human factors incomputing systems (CHI’07), 2007, pp. 1395–1404.
[5] V. Buil, G. Hollemans, and S. van de Wijdeven, “Headphones withtouch control,” in Proc. of the 7th international conference on Humancomputer interaction with mobile devices & services, 2005, pp. 377–
378.
[6] S. Brewster, J. Lumsden, M. Bell, M. Hall, and S. Tasker, “Multimodal’eyes-free’ interaction techniques for wearable devices,” in Proc. of the
SIGCHI conference on Human factors in computing systems, 2003, pp.
473–480.
[7] M. Micire, J. L. Drury, B. Keyes, and H. A. Yanco, “Multi-touch
interaction for robot control,” in Proc. of the 13th international
conference on Intelligent user interfaces (IUI’09), 2009, pp. 425–428.
[8] N. R. N. Enns and I. S. MacKenzie, “Touchpad-based remote control
devices,” in Proc. of the SIGCHI conference on Human factors incomputing systems, 1998, pp. 229–230.
[9] J. Nichols, “Informing automatic generation of remote control interfaces with human designs,” in CHI’02 extended abstracts on Human
factors in computing systems, 2002, pp. 864–865.
[10] R. Balchandran, M. E. Epstein, G. Potamianos, and L. Seredi, “A
multi-modal spoken dialog system for interactive tv,” in Proc. of the10th international conference on Multimodal interfaces, 2008, pp.
191–192.
[11] J. Nichols, B. A. Myers, M. Higgins, J. Hughes, T. K. Harris,
R. Rosenfeld, and M. Pignol, “Generating remote control interfaces
for complex appliances,” in Proc. of the 15th annual ACM symposium
on User interface software and technology, 2002, pp. 161–170.
[12] J. Nichols, D. H. Chau, and B. A. Myers, “Demonstrating the viabilityof automatically generated user interfaces,” in Proc. of the SIGCHI
conference on Human Factors in computing systems, 2007, pp. 1283–
1292.
[13] M. Wright and A. Freed, “Open Sound Control: A New Protocol for
Communicating with Sound Synthesizers,” in Proc. of the International Computer Music Conference, 1997, pp. 101–104.
[14] B. MacIntyre, E. D. Mynatt, S. Voida, K. M. Hansen, J. Tullio, andG. M. Corso, “Support for multitasking and background awarenessusing interactive peripheral displays,” in Proc. of the 14th annual ACM
symposium on User interface software and technology, 2001, pp. 41–
50.
[15] G. Hsieh, K. Wood, and A. Sellen, “Peripheral display of digital
handwritten notes,” in Proc. of the SIGCHI conference on Human
Factors in computing systems, 2006, pp. 285–288.

R EFERENCES[1] B. LaPlant, S. Trewin, G. Zimmermann, and G. Vanderheiden, “Theuniversal remote console: A universal access bus for pervasive computing,” IEEE Pervasive Computing, pp. 76–80, 2004.
[2] K. Kobayashi, Y. Kitamura, and S. Yamada, “Action sloping as a way
for users to notice a robot’s function,” in Proc. of the 16th International
Workshop on Robot and Human Interactive Communication, 2007, pp.
445–450.
[3] K. Kobayashi, S. Yamada, and Y. Kitamura, “Action sloping forincreasing awareness of robot’s function,” Transactions of Human
Interface Society, vol. 10, no. 1, pp. 37–46, 2008.

756