A Gesture-Based American Sign Language
Game for Deaf ChildrenSeungyon Lee, Valerie Henderson, Harley Hamilton , Thad Starner, and Helene Brashear
Atlanta Area School for the Deaf
College of Computing
Georgia Institute of Technology
Clarkston, GA 30021
Atlanta, GA 30332
HHamilto@doe.k12.ga.us
sylee, vlh, thad, brashear @cc.gatech.edu
ABSTRACTWe present a system designed to facilitate language development in deaf children. The children interact with a computer
game using American Sign Language (ASL). The system
consists of three parts: an ASL (gesture) recognition engine;an interactive, game-based interface; and an evaluation
system. Using interactive, user-centered design and theresults of two Wizard-of-Oz studies at Atlanta Area Schoolfor the Deaf, we present some unique insights into the spatialorganization of interfaces for deaf children.
Author Keywords

Deaf; children; ASL; design; Wizard-of-Oz method; computer games; computer aided language learning
ACM Classification Keywords

H.5.2 [Information interfaces and presentation]: User Interface - User-centered Design; K.8.0 [Personal Computing]:
Games
INTRODUCTION

to that of hearing subjects [1, 4]. This limited memory
processing ability may also be responsible for the slower
acquisition of language due to an inability to receive andprocess longer utterances efficiently. Our goal was to expose
the children not only to individual vocabulary signs, butalso to encourage them to formulate those signs into longer
concepts and phrases.
Proposed Solution

Our project involves the development of an American Sign
Language (ASL) game which uses gesture recognition technology to develop ASL skills in young children. The system
is an interactive game with tutoring video (demonstrating
the correct signs), live video (providing input to the gesture recognition system and feedback to the child via theinterface), and an animated character executing the child’s
instructions. The system concentrates on the practice andcorrect repetition of ASL phrases and allows the child tocommunicate with the computer via ASL. We target children
attending the Atlanta Area School for the Deaf (AASD),
ages 6–8, who are behind in their language development.

Motivation

Ninety percent of deaf children are born to hearing parents
who do not know sign language [3]. Often these children’s
only exposure to language is from signing at school. Early
childhood is a critical period for language acquisition, andexposure to language is key for linguistic development [7,8].
By twenty-four months of age, hearing children learning a
spoken language are combining words in spoken communication [11]. By age eighteen months, deaf children ofdeaf parents are combining signs to communicate. A thirdgroup, deaf children of hearing parents, develop language
at a much slower pace, attributable to lack of exposure tolanguage and incomplete language models [5, 9]. The shortterm memory for sequential linguistic information in deaf
children of hearing parents also appears limited compared

Copyright is held by the author/owner(s). CHI 2005, April 2–7,2005, Portland, Oregon, USA.
ACM 1-59593-002-7/05/0004.

Figure 1. Screenshot of Game Interface. a) Tutor Videob) Live Camera Feed c) Attention Button d) AnimatedCharacter and Environment e) Action Buttons
The game involves Iris the cat who can interact with her
environment in a number of ways based on what the childsigns to her. With the assistance of educators at AASD, wedeveloped a list of eight age-appropriate ASL phrases (seeTable 1).

When beginning the game, Iris is asleep. The child can pressan Action Button (Fig. 1e) and watch a short video clip
showing the correct ASL command in the Tutor Window(Fig. 1a). Next, the child clicks on the Attention Button(Fig. 1c), and Iris wakes up. The child then signs thecommand to Iris. A live video feed allows the child to seeherself (Fig. 1b). If the phrase is correct, Iris carries out thecommand or answers the child’s question with a pictorial
thought bubble in the window (Fig. 1d). If the child does
not sign correctly, a thought bubble with a question markappears above Iris’s head.

Subject &Gender
1-F
2-M
3-F
4-M
5-M

Playing Time(min:sec)
14:06
4:55
9:42
18:10
12:22

NumberAttempts
18
11
21
50
28

Self-Ended?
yes
yes
yes
nonoTable 2. Subjects’ Pilot Test Data.

Eventually, a gesture recognition system will control thecomputer’s response to the child’s signing. However, while
the gesture recognition system is under development, webegan the initial game design via Wizard of Oz (WOz)
methods. The goal of this phase of the project was to design
a prototype for the game which the children would findengaging and interesting enough to practice their signing.
By observing how the children use the system, we will
enhance usability and enjoyment.

The system is equipped with a desktop computer, twomonitors, a camera, and a digital video (DV) recording
device. For input, the mouse from the computer is given
to the child while the keyboard is given to the wizard tocontrol the interface actions. This configuration allows forthe use of a single desktop machine with two monitors, thus
eliminating the need for networking and synchronization
across computers. The DV device records the interface as it
is shown to both the participant and the wizard. Additionallythe computer creates a log file of concrete events (mouseclicks, button presses, key clicks, wizard actions, etc.).

Task Development and Method

First Pilot Test

Due to a very limited subject population, it was importantto remove as many errors from the interface and interaction
sequence as possible before attempting to run a longitudinal
study to evaluate language acquisition using the game. We
decided to use children slightly older than our targeted age(ages 9–11) for short pilot studies because they were capableof giving us more feedback about the interface. We tested
student in groups of 2 or 3, making subtle changes based on
the previous groups’ opinions and feedback.

The first pilot test of our system had three concrete goals.
First, we wanted to evaluate whether the game was engaging to the children. Second, we needed to confirm
the interchangeability of interpreters and ascertain whether
the interpreters were consistent in their evaluation of thechildren’s signing. Third, we were concerned with the“push-to-sign” mechanism and wanted to confirm that thechildren would remember to press the button before and after
they signed.

In addition to giving us crucial user feedback and allowing
us to improve our design, the WOz study provided an ideal
method to gather data for our gesture recognition system.
This diverse sample set allows for testing and developmentof our gesture recognition software with a variety of signers,
signing techniques, skin tones, and physical attributes. Thisdata set is invaluable for training of the mathematical models
and performance testing of the gesture recognition system.

The test consisted of two girls and one boy from the sameclass. All children watched a brief video in ASL tellingthem how to play the game. We informed all participants
they could stop playing the game whenever they wanted.
Our facilitator demonstrated the game. The children thenattempted to play the game on their own. Table 2 (subjects
1–3) shows the amount of time the children spent playing,
the total number of attempted phrases, and whether they
stopped playing of their own accord or were asked to stop.

Glossed ASLq(YOU LIKE MOUSE)q(YOU HUNGRY NOW)YOU GO PLAY BALLOONI RIS GO - SLEEP NOWYOU GO CATCH BUTTERFLY
whq(WHO BEST FRIENDWHO )LOOK - THERE IRIS
MOUSE OVER - THERE
YOU MAKE FLOWERSGROW GO - ON

English Translation
Do you like mice?
Are you hungry now?
Go play with the balloon.
Go to sleep now, Iris.
Go catch the butterfly.
Who is your best friend?
Look, Iris! A
mouse, over there!
Go make the flowers
grow.

Table 1. Glossed ASL Phrases and English Translations
WIZARD OF OZ EXPERIMENTS

To date, we have performed two pilot studies with five
participants at AASD.

Apparatus and Conditions

An ASL interpreter from AASD evaluated the child’s signs
from behind a partition, and the wizard keyed the appropriate
response. After finishing the test, we asked participants
several questions related to preference, ease of use, andsatisfaction to evaluate the game and test setting.
Observations and Conclusions

We observed that the interpreters had very different criteria
for what they considered “correct” signing. They often based
their assessment on their personal knowledge of the child’sASL skills. We considered this unacceptable and changed
the experimental setup for the next pilot test.
The push-to-sign mechanism was not a problem for thechildren. After having it demonstrated once or twice, theyrarely forgot to use it.

Each child played until he/she indicated that he/she wasfinished. Our goal was to engage the child for 20 minutes,
yet none of the children met that goal. We decided thegame required a smoother flow to encourage the children toattempt the phrases again after an incorrect response.
Second Pilot Test

Our previous work on gesture recognition [2] had determined that a combination of sensors provided better accuracy
than only a computer vision approach. Thus, we wanted
to perform the tests with children wearing small colored
gloves (to enhance the computer vision algorithms) withsmall wireless accelerometers mounted on the wrists. We
were unsure how the children would react to the gloves andif the gloves would impair their signing.
To stimulate longer interactions, we changed the flow of
the game slightly. Upon receiving the indication that theirsigning was not correct, we automatically played a video clipof the phrase with the explicit instruction, such as, “You tell
Iris, ‘you go play balloon’.” Previously, when Iris did notunderstand a phrase, the child had to press the same action
button to see the video again.
For this test, our facilitator from AASD acted as the onlyinterpreter, and eliminated the variability of interpretation.
We also encouraged him to be consistent and strict for eachparticipant. The facilitator watched the live video and cued
the wizard to have Iris respond or look puzzled.

we report several interesting observations that will heavily
influence our future work.
Spatial Aspects of ASL

ASL is a spatial language with rich directional expression. InASL each signer has a “signing space” which is maintainedin front of the signer. By setting up subsections of thesigning space and indexing the subsections (by referencing
them in conversation), signers indicate interactions between
concepts, people, times, etc. For a full discussion of thespatial construction of ASL, see [6] (Ch. 2–3) and [10].
For example, verbs can be directional and imply subjectand object from the way they move through space. For
example in the phrase “Bob, tell me the answer,” the signfor ‘tell’ moves from Bob to me. If instead the phrase was“I’ll tell you the answer” the sign moves from me to Bob.
During the second test, we used phrases such as, “You tell
Iris, ‘go play balloon.’ ” We thought the children would
understand that the imperative “you tell Iris” was directed
at them and the second half (‘go play balloon’) would be
the phrase needed to activate Iris. However, we discoveredthat both children included the directive ’you tell Iris.’ After
reviewing the video clips that the children were shown, both
we and the ASL educators at AASD believe the interface
layout and expression of the signing video were not spatiallycompatible.

The subjects this time were two boys from the same class as
the previous subjects. (Participants 4–5 in Table 2.)
Apparatus and Conditions

During the prior test, the children sat behind a child-sized
desk. Many signs were blocked as the children’s hands
tended to slip behind the desk. We changed the arrangementof the furniture and moved the desk to the participant’s rightside (both children were right handed), leaving the front
camera view unobstructed.
Conclusions and Observations

Again, Table 2 gives the times the children spent playing
the game. However, both of these children played until westopped them (due to class time limitations) and appeared
highly engaged in the game, offering commentary such as,“That’s weird!” when the system did not recognize what
they thought they had signed correctly.
One boy indicated that the accelerometers and gloves were“fine.” The other boy indicated they bothered him a “littlebit,” but he would still use them if they were necessary toplay the game.
OBSERVATIONS AND CONCLUSIONS

These experiments were not meant to yield statistical results,
and we chose not to concentrate on the accuracy of the children’s signing due to the wide variability of interpreters andthe confounding issues of ASL’s spatial grammar. Instead

Figure 2. Interface and Tutor’s Signing Space
We believe the children were treating the interface as an ASL
construct rather than a 2D interface. Thus, when the tutor
signs “you tell Iris,” she should sign directly to the child(Figure 2a). When signing the command “go play balloon”
she should turn 90 degrees and sign toward the animated cat(Figure 2b). Thus, she would maintain proper ASL spatial
grammar. Since the video was shot without this concern, thechild was confused.
Effect of Repetitive Tutoring

Anecdotally, some teachers at AASD report that many of
the children have difficulty generating complete sentences insign. Instead, the children rely on context to fill in the blanks
for them when communicating with others. For example,
they may sign ”balloon”, instead of the more complete
request, Can I have the balloon?”When creating this game,a primary goal was to motivate the students to practicegenerating complete sentences for clearer communicationand stronger language skills.

In our experiments, children sometimes violated proper ASL
grammar or omitted several vocabulary words while signing
the phrases. For example, the phrase “you go play balloon”
was sometimes shortened to “you balloon,” and the wizard
evaluated the partial phrase as incorrect. However, when
Iris did not understand, the children tended to watch thetutoring video again. They then tried the phrase again andcorrected their sign. We observed that the stricter evaluation
criterion led to more attempts by the children, and they
signed more clearly and distinctly with each attempt. Unlikea teacher who can easily guess unclear signs, the animatedcharacter and computer didn’t allow the child to insert ordelete vocabulary from the phrases. This led the children
to practice the process of problem solving through multiple
attempts instead of reverting to their previous linguistic
habits. They tried to “think” and came to “know” what
the computer would not accept with their signing thoughvarious trials. This is a good starting point for motivating
and accelerating proper language skills.
A longitudinal study will need to be conducted to measure
whether language improvement occurs or if the children are
merely duplicating what they see.

Users’ Preferences

The phrases in the game contain actions which direct thecat to interact with a butterfly, a flower, mice, etc. In
general the contents of the game seemed to engage the girls
more than the boys. Girls tended to play again even after
finishing all the actions with clear signs. Boys indicated theywould prefer races or more competitive games. Developing
a gender neutral narrative will provide greater motivation tochildren and is key to our next design iteration.
Among the eight phrases, three are questions whose answers appear in Iris’s pictorial thought bubble, and five arecommands directing Iris to a particular action. Children
preferred animations with kinetic action rather than static
thought bubbles. This has lead to a revision of the phrases
for the next pilot tests.

FUTURE WORKWe will continue to use small pilot studies with older,
articulate students to refine our interface and game design.
We will then undertake a larger WOz study with children
ages 6–8. The children will play two times a week forfour weeks to determine how their language skills improve.
Both before and after the experiments, the teachers of theparticipants will be asked to answer surveys on the children’slanguage skills. These surveys will be used to evaluate thesystem.
Additionally, camera feeds and log data can be used tocompile a “workbook” that is viewable in chronologicalformat. The workbook will be designed to help teachers
target problem areas for an entire class or an individual
student. This evaluation package will enable our system tofunction as a complete educational tool.

To achieve our goal of longer interaction time with the game,we decided that the game needed a more elaborate structure
and interactional goals. We believe incorporating various
rewards (such as moving on to a higher level or achieving ahigher score) will allow the user to have a richer experienceand may enhance the user’s motivation. By making eachtask or level progressively more difficult, we expect that thegame will be more challenging to students and help them todevelop better signing skills and linguistic memory.
ACKNOWLEDGMENTS:

We would particularly like to thank the children, staff, andadministration at AASD for their enthusiasm and supportof this project. This work is funded in part by NSF Career
Grant #0093291 and the NIDRR Wireless RERC.
REFERENCES

1. J.M. Bebko. Learning, language, memory and reading:
The role of language automatization and its impact oncomplex cognitive activities. Journal of Deaf Studies
and Deaf Education, 3(1):4–14, 1998.

2. H. Brashear, T. Starner, P. Lukowicz, and H. Junker.
Using multiple sensors for mobile sign language
recognition. In Proc. of the 7th IEEE Intl. Symposium
on Wearable Computers, pages 45–52, Oct 2003.
3. Gallaudet University GRI. Regional and nationalsummary, 1999-2000 annual survey of deaf and hard ofhearing children and youth. Technical report,
Washington, D. C., January 2001.
4. H. Hamilton and T. Holzman. Linguistic encoding inshort-term memory as a function of stimulus type.
Memory and Cognition, 17(5):541–50, Sept 1989.
5. H. Hamilton and D. Lillo-Martin. Imitative productionof verbs of movement and location: A comparativestudy. Sign Language Studies, 50:29–57, 1986.
6. E. Klima and U. Bellugi. The Signs of Language.
Harvard University Press, Cambridge, MA, 1979.
7. R.I. Mayberry and E.B. Eichen. The long-lasting
advantage of learning sign language in childhood:
Another look at the critical period for language
acquisition. Journal of Memory and Language,
30:486–498, 1991.
8. E. L. Newport. Maturational constraints on language
learning. Cognitive Science, 14:11–28, 1990.
9. P. Spencer and A. Lederberg. Communication andLanguage: Discoveries from Atypical Development,chapter Different modes, different models:Communication and language of young deaf children
and their mothers, pages 203–30. Harvard University
Press, 1997.
10. C. Valli and C. Lucas. Linguistics of American Sign
Language: An Introduction. Gallaudet University
Press, Washington DC, 1992.
11. G. Wells. Language Development in Preschool Years.
Harvard University Press, Cambridge, MA, 1985.