Silhouette Interactions– Using theHand Shadow As Interaction
Modality
Eria Chita

Kai Kunze

Graduate School of Media Design Graduate School of Media Design
Keio University, Japan
Keio University, Japan
kai@kmd.keio.ac.jp
eriachita@gmail.com

Yuta Sugiura

Masahiko Inami

Graduate School of Media Design
National Institute of Advanced
Industrial Science and Technology Keio University, Japaninami@kmd.keio.ac.jp
y-sugiura@kmd.keio.ac.jp

Sunao Hashimoto

Masa Ogata

Meiji University, Japanhashimoto@kougaku-navi.net

Keio University, Japanogata@ayu.ics.keio.ac.jp

Abstract
We present the concept of Silhouette Interactions, using
the shadow of our hand as an extension of our body to interact with our physical environment. We apply SilhouetteInteractions to the application case of home appliance control, show 2 user studies to identify interesting appliances,
actions and shadow gestures to be used. Informed by thestudies, we also implement an initial prototype system forfurther evaluation. We discuss best practices and lessons
learned for Silhouette Interactions.

Author Keywords
Shadow Interaction; LEAP Motion; Home Appliances; HomeAutomation

ACM Classification Keywords
H.5.2 [User Interfaces]: Input devices and strategies.

Introduction

Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for third-party
components of this work must be honored. For all other uses, contact theOwner/Author. Copyright is held by the owner/author(s).
Ubicomp/ISWC’15 Adjunct , September 07-11, 2015, Osaka, Japan
ACM 978-1-4503-3575-1/15/09.
http://dx.doi.org/10.1145/2800835.2800870

Our shadow is a ”natural” extension to our body, it’s often used for play and entertainment (e.g. shadow puppets). There is substantial work in researcher applying
shadow interactions to projections (mobile and stationaryprojectors)[2]. To our knowledge, there are very few works
that explore hand shadow manipulate real world objects.
However, we believe it’s an intuitive way to interact insmart environments. The hands are our most importantLights"

tools and the silhouette of our hands can be seen as a easy
to understand extension.

15"TV"

10"

Loudspeakers"

8"

Blinds/Curtains"

8"

Aicrondi5oning"

4"

Radio"

2"

Phone"

2"
0"

2"

4"

6"

8"

10"

12"

14"

16"

Figure 1: The user preferences
Appliances
LightTV
Speakers
BlindsFan
Cleaning robot

Actionsturn on/off, control brightness
turn on/off, switch channels
turn on/off, adjust volume
move up/down, adjust angle
turn on/off, adjust intensity
turn on/off, tell where to clean

Table 1: Appliances andcorresponding interactions for theexperiments.

This paper explores the design space of hand shadow based
interactions with a focus on the home appliance scenario.
We believe home appliances are the right application case,as we get more and more computer-based devices in our
house and tools like a universal remote are hard to setup,
not even mentioning the task to remember the functionalities of the buttons.
In this case natural gestures based on shadow projection
could be the intuitive solution. Although we focus on home
appliances, we believe that most findings can also be applied to other use cases.
The main contributions of this paper are (1) we present
the concept of Silhouette Interactions: interacting with thephysical world using your shadow as a natural extension, (2)we apply the concept to the control of home appliances,
exploring in user studies the most interesting appliances,evaluating gestures and use cases.

Related Work
There is a lot of work using hand and body shadows tointeract with projections or large displays[3]. We neglect
the area of mobile projection as it is only faintly related
and cannot be described in detail in this paper. Shoemaker et al. show how to use an adjustable spotlight toreach interact even with large size projections[6]. In contrast we use shadow projections to interact with real world
objects. Some systems have a hybrid approach overlaying
information on objects using projection[4]. However thehand shadow interactions envisioned are often very basic
and not explored (e.g. only one finger tap or just focusing
on the which part gets occluded). The closest to our work,
Cowan et al. focus on exploring shadow gestures for colFigure 2: Initial prototype setup.

laboration on mobile projectors [2]. Yet, the emphasis on
this work is on the group aspects and picking easy understandable gestures for a couple of people to work together.
Okahara et al. explore the psychological effect of using a
projected hand as an extension to manipulate objects [5].
Also there is a lot of work on mobile hand gestures without projection [1], we see them as complementary to ourresearch. So far we are unaware of any work that explores
hand shadow gestures as a means to manipulate real world
objects in particular in regard to the home automation scenario.

Concept:Silhouette Interactions
We propose to use the projected shadow of the user’s handto interact with physical, smart objects around the userextending and augmenting the user’s body.
For an example in the teaser figure on the first page, a usersits on the couch and controls the blinds by projecting theshadow or silhouette of his hand on the object performing
an intuitive shadow gesture to move them up.

What Devices do people want to control withshadows?
We performed structured interviews with 20 users (12 male
8 female, average age 25, std 8) focusing on SilhouetteInteractions. We introduced them to the concept showing
them an LED light and the shadow projection. 11 alreadyused flash lights as a kid to play with their shadow. All 20
found the Silhouette Interactionsconcept useful and would
use it during everyday life. 4 had reservations depending on
the size of the device and would not want to wear a lighton the top of their arm/torso all day.
15 were familiar with hand gesture recognition using Kinect
and/or Leap Motion. 13 of the 15 found the shadow more

suitable to interact with real world objects. 5 users think theinteractions are easier to remember if they see their handshadow. We aimed the remainder of this study to get ideason the design and promising use cases. We asked the users
to explore what kind of (home) appliances they could thinkof controlling with shadows and in what circumstances theusers wanted to control sth in sight from a distance. Theinterviews were conducted in different environments so thatthe users are not primed to certain interaction ideas (7 atoffice, 7 at living room, 6 at restaurant).
All users want to control lights, interestingly most of themwished to not only turn them on or off but change theirbrightness 1. The next appliances the users want to interact
with are TV, speakers, curtains/blinds and air-conditioning/fans. 4 users also wanted to use the radio (controlling channels and volume) as well as taking phone calls using a freespeaker. We also got several unique answers: controlling acleaning robot, radiator, heated bathtub and washing machine. One user mentioned they want to use the shadow to
get haptic feedback from devices (feel how warm/cold theradiator is etc.).
Figure 3: Sketches of thegestures derived from the user
study and used in the initial
prototype: tap to turn on adevice, circle,pinch,swipe, move
up/down and flip hand to control
a device.

User Study: What shadow gestures to use
for which interaction for which appliance?
After evaluating the best appliances and action types, wepicked the most interesting use cases (shown in Table 1)for this study with the aim to find suitable shadow gestures
for the given actions. 12 volunteers (5 female, mean age26, std 7) participated.

Experimental Design and Setup
We discussed with 15 users (10 male 5 female, age 22-60)
the idea of Silhouette Interactions. We introduced them tothe concept showing them an LED light and the shadow
projection. 12 found the Silhouette Interactionsconcept

useful and would use it during everyday. 3 had reservations depending on the size of the device and would notwant to wear a light on the top of their arm/torso all day.
We discussed which appliances the users would want tocontrol.
In a follow-up study with the aim to find suitable shadow
gestures for a home automation scenario. 8 volunteers (3female, age 24-36) participated.
The independent variables for our study were the appliances. We performed the study in a living room scenario.
We provided a stand light, fan, speakers, TV, cleaning
robot and blinds. We explained the Silhouette Interactionsconcept to the participants as given above. We equippedeach participant with an LED We let them also try different places to mount the light. All users mounted it on theleft/right side of the torso (depending on their handednessand preference) and we let them perform some free-form
hand shadows on the wall, to get used to the shadow. Afterwards we asked them to come up with gestures to perform actions they associated with an appliance using theirshadow. The order of the appliances was determined bylatin square design. At the end of an experimental run,
we interviewed them about how much they liked the interaction idea (positive and negative thoughts), if they could
imagine using it and where they see other application areasof Silhouette Interactions. The complete user study wasvideotaped for later reference. One interviewer conducted
the study and one observer took notes during each run.

Results and Discussion
Figures 3 and 4 give an overview over most of the gestures
the study participants came up with. For the light, mostusers performed a tap gesture (Figure 4a) on top of the lightfor turning it on/off, one user swiped and one knocked on
the light.

To control brightness most users performed the up/down
movement along the light(Figure 3e), one used a pinch gesture (open to close for decrease, close to open for increase),
two used the circle gesture (right turn for increase, left turnfor decrease). Most participants performed a tap press on
the on button of the TV to turn it on/off, 2 used tap on theTV top, swipe left/right for changing channels. One turned
the hand palm left right for changing. For the speakers, allparticipants taped the on/off switch, and used the move
up/down gesture to control the volume, except 2 used who
used the circle motion. To open/close the blinds, 4 users
moved the hand up/down (Figure 3e) to indicate the position, 2 users taped at the position they wanted the blinds
to be, 2 users swiped upwards/downwards.
For interaction with the fan, most users performed also tap
for turning it off and on. Yet, the location of the tap diverged, most taped on the top again, yet 2 users tapped on
the black start button. To increase and decrease the intensity all users used the circle with the index finger (right turnfor increase, left turn for decrease), one user used a slow
circular motion for decrease and fast motion for decreasing intensity. We implemented a prototype system basedon the gestures using Leap Motion an LED and a smart
phone, see Figure 2 for details.

Conclusion and Future Work
We present Silhouette Interactions, an intuitive way to interact with the physical world. From the user studies wesaw that all users liked the interaction style and we could
come up with a concise interaction vocabulary (shadow gestures) to use for the application case. In future work, wewant to evaluate the prototype in more detail overcoming
the current limitation (using the smartphone orientation for
Figure 4: Sample pictures of thegestures performing actions: tap
to turn on a device,
circle,pinch,swipe, move up/down
and flip hand to control a device.

device selection and the limited distance for the gestures related to the leap motion skeleton tracking range).

Acknowledgements
This work is supported in part by the CREST project fromJST.

References[1] Bannach, D., Amft, O., Kunze, K. S., Heinz, E. A.,
Troster, G., and Lukowicz, P. Waving real hand gestures
recorded by wearable motion sensors to a virtual car
and driver in a mixed-reality parking game. In Computational Intelligence and Games, 2007. CIG 2007. IEEE
Symposium on, IEEE (2007), 32–39.
[2] Cowan, L. G., and Li, K. A. Shadowpuppets: supporting collocated interaction with mobile projector phones
using hand shadows. In Proceedings of the CHI, ACM(2011), 2707–2716.
[3] Krueger, M. W., Gionfriddo, T., and Hinrichsen, K.
Videoplacean artificial reality. In ACM SIGCHI Bulletin,vol. 16, ACM (1985), 35–40.
[4] Mistry, P., and Maes, P. Sixthsense: a wearable gesturalinterface. In ACM SIGGRAPH ASIA 2009 Sketches,ACM (2009), 11.
[5] Okahara, K., Ogawa, S., Shinmei, T., Iwai, D., andSato, K. A study on projection representation of extended hand for augmented body interface. JapaneseUser Study Journal 19, 3 (2014), 349–355.
[6] Shoemaker, G., Tang, A., and Booth, K. S. Shadow
reaching: a new perspective on interaction for large
displays. In Proceedings of the 20th annual ACM UIST,ACM (2007), 53–56.