<figure><img src="image_1.png" /></figure>

HAL
open science

# Play and Learn with an Intelligent Robot: Enhancing the Therapy of Hearing-Impaired Children

Andri Ioannou, Anna Andreva

► To cite this version:

Andri Ioannou, Anna Andreva. Play and Learn with an Intelligent Robot: Enhancing the Therapy of Hearing-Impaired Children. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.436-452, 10.1007/978-3-030-29384-0_27. hal-02544587

HAL Id: hal-02544587

https://inria.hal.science/hal-02544587

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Play and learn with an intelligent robot: Enhancing the therapy of hearing-impaired children?

Andri Ioannou<fnref n="1" />,<fnref n="2" /> [0000-0002-3570-6578] and Anna Andreva<fnref n="3" />

**Abstract.** This study suggests an innovative way of using an intelligent robot to support speech therapy for hearing impaired children through play. Although medical technology (e.g., hearing aid, cochlear implant) for children with hearing impairment has advanced significantly, the amplification itself does not provide optimal development of hearing and speaking; it must be combined with specialized therapy. The present study focuses on the use of the humanoid robot NAO in auditory-verbal therapy, an approach to the development of auditory and verbal skills, which does not allow lipreading or other non-verbal cues to facilitate communication. NAO does not have a human mouth and therefore children with hearing impairment cannot do lipreading; this unique characteristic of the technology has been successfully used in the study to create playful and engaging auditory-verbal therapy sessions for six kindergarten hearing impaired children, allowing them to improve their ability to follow instructions using the hearing aid/ cochlear implant rather than visual clues in the environment. Our results, although preliminary, seem to encourage further research in supporting hearing-impaired children via play with intelligent robots.

**Keywords:** NAO, Intelligent Robot, Humanoid Robot, Hearing Impairment, Deaf, Speech Therapy, Auditory-Verbal Therapy, Special Education, Playful Learning, Technology-Enhanced Learning.

## 1 Introduction

Children with hearing impairment are at high risk for language, academic and social difficulties [1]. Fortunately, the advancement of medical technology (e.g., hearing aid, cochlear implant) has significantly changed the prospects of children with hearing impairment. As Levy (2012) explained, with reference to the invention of the cochlear implant, there is now a device that allows a large proportion of deaf people to hear; this has dramatically changed the deaf community and deaf education [2]. Because of the advancement of medical technology, the education of children with hearing impairment has evolved into a change in speech therapy, educational audiology, and

<footnote n="1">¹ Cyprus Interaction Lab, Cyprus University of Technology, Limassol, Cyprus</footnote>

<footnote n="2">² Research center on Interactive media, Smart systems and Emerging technologies (RISE), Nicosia, Cyprus</footnote>

<footnote n="3">³Department of Logopedics, Faculty of Public Health, Health Care and Sport, South-West University “Neofit Rilski”, Blagoevgrad, Bulgaria
andri@cyprusinteractionlab.com</footnote>
special education [3]. Educational programs that serve hearing impaired children have changed and keep changing, especially since most of these children are now placed in mainstream schools rather than schools for the deaf in the developed countries [4]. Regional school administration and councils have stopped training programs for children with hearing impairment, on the idea that these students, supported by medical technology, should not be educated separately; instead, they are included to general/mainstream classroom whilst they receive additional therapy and support from speech therapists and educational audiologists [3]. Despite the medical technology advancement, amplification itself does not provide optimal development of hearing and speaking; it must be combined with a specialized program for early intervention [5].

The focus of this study is on the integration of an intelligent robot, NAO, as a supportive tool for learning through play in this early intervention, particularly in audito- ry-verbal therapy for hearing impaired children, an approach to the development of auditory and verbal skills that does not allow lipreading or other non-verbal cues to facilitate communication. In general, play is crucial for the communication, cognitive, physical, social, and emotional development of young children [6]. While initiating and taking part in playful activities children are motivated to communicate and prac- tice their language [7]. For this reason, play is extensively used in therapy and educa- tion. There are many studies exploring play and language learning in groups of chil- dren with hearing disorders. Some studies are related to play and deaf children in special schools [8]; other studies compare the ability of children with hearing impair- ment to play with their hearing peers [9 – 13]. NAO is well documented as a tool that can be used successfully in learning and therapy through play (e.g., [14 – 18], alt- hough not with hearing impaired children.

Furthermore, NAO has a unique characteristic and a great affordance in the context of the present study. The robot does not have a human mouth and therefore children with hearing impairment cannot do lipreading. The unique feature of the technology has been identified by the authors of this work (an educational technologist and a speech therapist - researcher and practitioner) and has been used in this study to create a unique environment for auditory-verbal therapy sessions for children with hearing impairment. This study was inspired by a true need for engaging hearing-impaired children in auditory-verbal therapy sessions, as identified during previous work of the authors [19] as well as practical experience of the therapists involved. The present manuscript focuses on:

1. What are the children's gains in auditory skills, name the ability of the hearing-impaired child to detect sounds and follow instructions?

2. What is the perceived value of the robot-enhanced environment for auditory-verbal therapy?

The rest of this manuscript begins with an overview of deaf studies and technology. Then, we elaborate on the methods and procedures of the study, including details on the robot-enhanced activities. We finally present our findings followed by a discussion of implications for future research and practice.
# 2 State of the Art

## 2.1 Therapy of Children with Hearing Disorder

Humans use speech to share information, to predict and explain behavior such as desires, beliefs, and feelings [20]. Young children learn to participate in communication situations through listening to the speech of their parents and caregivers. The first years of life are crucial for language development, while about 85% of neural development happens till the third year of life. Early access to the auditory brain is of importance for developing spoken language adequate to the norm for certain age [21]. If mild or partial hearing loss occur in this sensitive period, it could result in delay of language development.

Today, there are about 34 million children with hearing loss worldwide [22]. Various factors affect the communication skills of hearing impaired children, ranging from the degree of hearing loss, age of onset of hearing loss, age of identifying the child's hearing disorder, etiology of the disorder, adequacy of intervention, the presence of other disabilities, age of receiving intervention, type of intervention program, family and environmental influences, consistency of use of medical technology (e.g., hearing aid, cochlear implant), and the attitudes of the child or parents [7, 9 – 13].

One of the main issues the family of a deaf child has is whether to use manual or verbal communication with their child [23]. Initially the parents of children with hearing impairment and later the children themselves can choose a communication approach to learn speech and language. Parents seek information from the professionals (medical doctors, audiologist, speech and language pathologist, and teacher) about the communication method. Those who choose the oral communication method expect from their child an ability to fit in with typically developing children [24]. According to Gallaudet Research Institute report (2008), around 96% of hearing impaired children have hearing parents [25]. Consequently, hearing parents want to give the opportunity to their child with hearing impairment to communicate verbally. Unfortunately, there is no data that clearly illustrates the best communication approach [20, 24].

According to Harris (2014), the degree of a child's hearing loss has been a primary factor used by parents to decide which communication modality is applicable for their child. Usually the parents of children with mild or moderate hearing loss choose oral mode (i.e., listening and spoken language) whereas with more severe losses, parents opt for a manual form of communication such as Sign Language and finger spelling. Socialization, academic achievement and self-esteem may also influence the decision of the parents regarding the communication modality [11]. Researchers in Australia [26] have investigated the way of decision making about the communication mode in families of deaf children. They found out that most of the children with hearing impairment use oral communication; only one third have experience using a type of non-verbal communication mode. The caregivers choose the communication mode depending on source of information – professionals, family and friends, own research by parents; child individual needs and preferences; accessibility of communication – languages and communication modality used in family and community; access to intervention; audiological characteristics; child's future life [26, 27]. The aim of all
communication approaches is to give the opportunity to the child with hearing impairment to develop sufficient skills to be able to communicate.

## 2.2 Medical Technology and Therapy of Hearing Impaired

The advancement of medical technology has significantly improved the prospects of children with hearing impairment. Amplification via electronic devices such as hearing aids or cochlear implants can provide a sense of sound to hearing impaired people, although they do not restore hearing loss. In any case, speech therapy and early intervention upon use of an appropriate aid (hearing aid or cochlear implant) is a major factor for development of the child's communication skills.

Hearing aid amplifies sounds and makes them louder. The microphones of the device pick up the sounds and after the amplification, they are sent into the ear canal, through the middle ear to the cochlear where the hair cells are activated and provide sound signals to the brain. Hearing aid is suitable for people with mild to severe hearing loss. Cochlear implant is surgically implanted electronic medical device. Cochlear implants are suitable for people with severe-to-profound sensorineural hearing loss, who cannot benefit from hearing aid. It has two main parts – internal (receiver/ simulator) and external (speech processor). The internal part is implanted by surgery under the skin behind the ear. Electrodes are placed by medical professionals into the cochlear to stimulate the auditory nerve. Cochlear implant substitutes the function of the damaged hair cells in the inner ear and directly stimulates the auditory nerve that provides the signals to the brain. The external part is behind the ear. It is attached via transmitter cord to the transmitter coil, which magnetically attaches itself to the internal receiver [20, 28]. Children with successful surgery could hear the whole range of speech sounds and they have the possibility to learn and use the verbal speech; however, it is difficult to predict whether the cochlear implant(s) surgery will be successful or not [29]. Then, there are situations when the surgery is successful but the child is not using the implant due to other communication preferences, lack of family support, psychological weakness to accept the implant, the school that the child is attending, the difficulty to get used to the signal of the implant [11]. Resent research has shown that cochlear implant children trained without the use of sign language have shown significant advantage in their narratives, range of vocabulary and use of expressions, and in the complexity of the syntax used in their language [30].

Assuming an appropriate aid (hearing aid, cochlear implants) is used, support from speech therapists and educational audiologists in the early years of age is critical. This puts strong emphasis on the options of auditory-oral and auditory-verbal therapy approaches to development of verbal communication skills for the child who uses amplification [31]. In the auditory-oral approach, the child listens to others while s/he pays attention to lipreading (i.e. receiving visual information from the lips) and body language. Instead, the auditory-verbal method is a very strict approach that does not allow the use of lipreading or other non-verbal cues (e.g. gestures) to facilitate communication [20]. Auditory-verbal approach is child-directed, based on the child's interests and using play. It includes strategies that encourage joint attention, turn-taking, thus facilitates the development of listening skills, speech and language [32]. In this case, during therapy sessions, speech therapists stay in position which does not allow lipreading to force the child use his/her amplification device (hearing aid or
cochlear implant) to hear the speech sounds. This enables children to make auditory images of what they hear in their brains and to create neural pathways for speech and language development [30, 33].

Overall, research of Cole and Flexer (2015) has shown that learning through listening is the most effective way of developing spoken language and cognition [30]. Thus, the auditory-verbal method is a very attractive in early intervention and therapy of children with all kind of amplification (e.g., hearing aid, cochlear implant). A technique that has been associated with auditory-verbal therapy is covering the therapist's mouth with his/her hand to eliminate lipreading. In their recent work, Estabrooks et al. (2016) encourage therapists to act more naturally during auditory-verbal therapy. They argue that covering the mouth might disrupt sensorimotor input during infancy and may have negative implications for the development of speech motor control; it can also provoke stress in young children which, in turn, negatively affects speech perception. The listening skill should be stimulated, but therapists should find ways to eliminate lipreading without having to cover their mouth, a rather atypical act in social interaction [32]. The humanoid robot NAO has a unique characteristic in this case - it does not have a human mouth and therefore children with hearing impairment cannot do lipreading. This unique feature of the technology has been identified by the researchers and has been used in this work to create a unique environment for audito-ry-verbal therapy sessions that can be playful and effective.

## 2.3 Play and Learning with Robots

Play has an adaptive function when children explore new situation, new environment, like games with a humanoid robot. It could support the development of sensorimotor skills and cognitive development – in our case stimulating the listening skills and the language[44].

The published studies about use of intelligent robots in interaction with children with and without disabilities are rapidly increasing during the past decade. Some of them explore the human-robot interaction in play scenarios with typically developing children (e.g., [16, 34]). Others investigate play and learning with intelligent robots in the therapy of children with disabilities, e.g., diagnosed with cerebral palsy [35] or Autism Spectrum Disorder (ASD) [14, 17, 18, 36 – 38]. One example is the program known as ASKNAO (Autism Solution for Kids) by Aldebaran Robotics, which uses NAO as a tool to help ASD children; the program has supported ASD children in learning about the social and communicative skills they lacked [39]. Intelligent humanoid robots show effectiveness with ASD children because they are simple and predictable with basic conversational function; they can be used as mediators to interact with the professional in therapy sessions, with the parents at home or with their peers at school [18]. Overall, research on the use of robots for play and learning shows that children (with disabilities or not) tend to interact with the intelligent robot smoothly, accept it as a peer in play, and are willing to initiate communication with the robot.

Studies on using intelligent robots with deaf children are scarce. Sawada et al. (2008) used a robotic voice simulator for speech therapy with hearing impaired children; the invented robot consisted of motor-controlled vocal organs (vocal cords), a
vocal tract and a nasal cavity to generate a natural voice imitating a human vocaliza-
tion. The robot reproduced the vocalization of the deaf children and taught them how
to generate clearer speech through repeating the correct vocalization [40.] Moreover,
NAO has been used in play activities for the assessment of listening and speaking
skills of seven hearing-impaired students who use cochlear implant(s) deaf children
[19]. A couple of other studies used intelligent robots with deaf children for manual
communication. One such study involved NAO to assist a story telling by showing
signs from the Turkish sign language, therefore facilitating non-verbal communica-
tion [41]. Then, unlike NAO that has only three fingers and thus, limited ability to
present all signs used in manual communication, [42] used a robotic platform with
five fingers (Robovie R3) for teaching sign language to deaf participants. There are
virtually no studies focused on the use of intelligent robots in therapy of children with
hearing impairment aiming specifically at the development of auditory and verbal
communication skills.

3 Methodology

This was a mixed-method study which made use of quantitative and qualitative data to realize gains in the ability of the hearing-impaired child to detect sounds and follow instructions as well as the value of the robot-enhanced experience based on the observations of the participating speech therapist and special teacher.

**3.1 Participants and setting**

The study took place in a mainstream public kindergarten-preschool in an Eastern
Mediterranean country. This kindergarten-preschool (2-5 years old children) has the
largest percentage of children with hearing impairment of the country. It is located
next to and shares facilities and staff with the single school for the deaf in the country.
Upon completion of the kindergarten, the parents of hearing impaired children have to
decide to mainstream their children (i.e., put them in a general school) or keep them at
the school for the deaf, with only 3% choosing the latter, as of today's data (personal
communication with school principal). Children with hearing impairment in the kin-
dergarten are supported by special teachers and speech therapy experts who focus on
the child's development of auditory and communication skills.

Six kindergarten children, 3-4 years old, participated in the study. Detailed charac-
teristics of the children (e.g., current speech and language skills, degree of hearing
loss, age at identification of hearing loss, age at intervention for hearing loss) were
unknown to the researchers, as requested by the school principal for confidentiality
purposes. Despite the unfortunate fact, the researchers decided that the study was still
worth pursuing in order to detect and document, even preliminary, the added value of
an intelligent robot in the particular context. All children were supported by either a
cochlear implant or a hearing aid or both, as seen in Table 1. These children attended
regular kindergarten lessons together with eight hearing children in the same class.
They also attended personal speech therapy sessions at the school's special unit.
Participants were also the school therapist (audiology and speech therapy expert) and the special teacher of the school (also speech therapy expert). The school therapist worked with the researchers in the co-design of the activities for the study; she was also active during the implementation of the study and data collection as described below. The special teacher of the school was an observer in most (80%) of the therapy sessions. During the implementation and data collection the researchers (educational technologist and a speech therapist) were observers of the experience.

## 3.2 Activities and co-design

The activities were co-designed by the researchers (an educational technologist and a speech therapist - researcher and practitioner) and the school therapist, based on previous experiences in research and practice, although such activities are considered general practice in speech therapy. Based on the school therapist, she had used similar activities with her students in the past, but there was no structured way of doing these activities (how often, what duration, what kinds of animals, what kinds of sounds). Also, in applying the auditory-verbal therapy method she was concerned about covering her mouth and acting unnaturally in any way that could disturb the child.

A total of four activities were designed and deployed on NAO. In accordance to the auditory-verbal method, all activities involved listening, followed by spoken instructions by NAO in order to assess children's ability to correctly respond to the spoken sounds using their cochlear implant and/or hearing aid, rather than other cues in the environment. For every correct answer, NAO praised the child and NAO's eyes turned green; for every mistake, NAO said "try again" and his eyes turned red. The visual clue of "green" and "red" eyes was the only visual clue in the environment. These four activities are briefly described below (see also Figure 1).

1. *Ling Sounds Story*. An object for each of the six Ling Sounds is placed on the desk (i.e., /m/: ice cream, /i/: mouse, /h/a/: Plane, /sh/j/: baby who sleeps, /s/: Snake, /u/: ghost). These six sounds represent speech in different frequencies (low, mid and high). They are used to check how well children are hearing with their cochlear implant and / or hearing aid. This check is used in order to detect changes in child's quality of hearing. NAO tells a story and when the above sounds are spoken, the child picks the appropriate object to give to NAO e.g., "Maria spent a wonderful weekend riding with her family in the mountain. As they walked she heard a sound /iii/, what can it be? [child picks up the mouse]. However, her dad saw something terrible /sss/ [child picks up the snake]. "Dad, this snake is very big !"...". The story changes from session to session so that the Ling Sounds are spoken in different order.
<table><caption>**Table 1. Participants Profile**</caption><thead><tr><td>Child (Gender)</td><td>Cochlear Implant</td><td>Hearing Aid</td></tr></thead><tbody><tr><td>P1 (Girl)</td><td>X</td><td></td></tr><tr><td>P2 (Girl)</td><td>X</td><td></td></tr><tr><td>P3 (Boy)</td><td>X</td><td></td></tr><tr><td>P4 (Boy)</td><td>X</td><td>X</td></tr><tr><td>P5 (Girl)</td><td></td><td>X</td></tr><tr><td>P6 (Girl)</td><td></td><td>X</td></tr></tbody></table>

2. Music Density. The child holds a musical toy-instrument of his/her selection e.g., a drum (child selects from a box of a few toy-instruments). As NAO signs a rhyme in gentle intensity, the child beats the drum gently. When the intensity goes stronger at unexpended time, the child needs to immediately respond by beating the drum louder, and vice versa. In a round of play, after a medium starting density of the song, there are three unexpended changes in sound intensity (louder or lower). The game is used in order to develop child's ability of discrimination and identification of sound intensity.

3. Farm animals. Various animals (farm and jungle animals) and a farm built of blocks are placed on the desk. NAO asks the child to place in and out of the farm one animal at a time, based on the animal's sound e.g., "Take [cow audio sound] and place it in the farm", "Take [dog audio sound] and place it out of the farm". The seven farm animals are cow, sheep, dog, cat, rooster, pig, horse and duck; instructions play in random order for animals in and out of the farm. In a round of play, there are six statements to be executed by the child. With this game children develop their ability to discriminate and identify animal sounds which are with different frequency (e.g. low frequency – cow sound, high frequency – cat sound).

4. Vegetables. Vegetable-toys and a basket are placed on the desk. The child listens to NAO's simple statements, such as "Put x in the basket", "Take x and y from the basket". The five vegetables are: potato, tomato, lettuce, cucumber and carrot; instructions play in random order. In a round of play, there are five statements to be executed by the child. The purpose of the game is discrimination and identification of vegetables.

## 3.3 Procedures

The study took place during six consecutive weeks of speech therapy sessions using strictly the auditory-verbal therapy method, which does not allow lipreading or other non-verbal cues to facilitate communication. Therapies were done daily, during the morning, in the special unit of the school. Some sessions were lost due to schools events, other obligations of the school therapist, or absence of the child from the school. In six weeks, children completed between 16 and 22 sessions with NAO. Children arrived at the special unity in groups of three (i.e., 2 groups of 3 children) and the session lasted 45-minutes. Within the session, the children performed in the
activities one by one, while the other two children waited for their turn. That is, each child typically had a 15-minutes slot to participate in the four activities (Ling sounds story, Music density, Farm Animals, Vegetables). Some children managed to complete all four activities in their 15-minutes slot. Some others completed fewer activities. By week 3 (around 10 sessions) all children were fast enough to get through all four activities within their 15 minutes slot in each session.

The week before the investigation begun, all spoken sounds in the activities were pilot-tested with all participating children. For each child, NAO's pitch tone and speed of spoken words was adjusted and noted for subsequent session, based on the school therapist's input from observing the child's response to the sounds. NAO's speak volume was set nearly to maximum for all children, which according to the therapist approximated how loud a teacher spoke in a typical lesson in the classroom. NAO's language was set to the national language of the country. We used NAO's robotic voice only for the instructions e.g., "take [audio sound for cow] and place it in the farm." However for the sounds in the activities, such as the "cow "sound in the farm game or "/i:/" for mouse or "h/a/" for the plane in the Ling sounds game, the sounds were downloaded as audio files in high quality and per screening of the school therapist for quality and appropriateness for the activity.

During the study, NAO was sitting without any interaction with the child other than verbal. We used strictly the auditory-verbal therapy method, which does not allow lipreading or other non-verbal cues to facilitate communication. Any cues from NAO or in the environments could have jeopardize the philosophy of the auditory-verbal therapy method. Moreover, NAO was semi-autonomous. NAO could play the activities and randomize the order of sounds within each activity. The therapist could touch NAO's head (head-sensor on NAO) for NAO to act a positive response whilst a touch on his foot (foot-sensor on NAO) triggered a negative response (e.g., switching between activities, activating positive feedback or "try again" via touching as relevant). Although the experience could be manageable by the school therapist only, due to the exploratory nature of the present study, a student programmer was also in close proximity to the robot to ensure everything would work as planned. The special teacher of the school as well as the two researchers (authors of this work) observed the study from a seated position and without interfering.

<figure><img src="image_3.png" /></figure>
<figure><img src="image_4.png" /><figcaption>Fig. 1. Implementations of Farm Animals (top), Ling Sounds Story (bottom)</figcaption></figure>

In terms of data collection, the school therapist recorded each child's performance in every activity completed (Ling sounds story, Music density, Farm Animals, Vegetables) in every session. Although NAO gave a "try again" option for the sake of play and learn, for data collection purposes the therapist marked a correct answer only on child's first trial and only for correctly detected sounds. Table 2 presents an example of data recording for the "Ling Sounds Story." Similar tables were used for data recording of other activities in every session and for each participating child. At the end of the activity the therapist together with the special teacher who observed the study participated in a group-interview with the researchers (authors of this work). The interview aimed to elicit details about the overall experience and perceived added value of the technology.

<table><caption>Table 2. Sample data recording and interpretation table.</caption><thead><tr><th colspan="7">Week X - Child name - Ling Sounds Story</th></tr><tr><th></th><th>/m/:<br>ice<br>cream</th><th>/i/:<br>mouse</th><th>/a/:<br>plane</th><th>Sh/ʃ/:<br>baby who<br>sleeps</th><th>/s/:<br>snake</th><th>/u/:<br>ghost</th></tr></thead><tbody><tr><td>Monday</td><td>√</td><td>√</td><td>√</td><td>√</td><td>-</td><td>√</td></tr><tr><td>Tuesday</td><td>√</td><td>√</td><td>√</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Wednesday</td><td>√</td><td>√</td><td>√</td><td>-</td><td>-</td><td>√</td></tr><tr><td>Thursday</td><td>√</td><td>√</td><td>√</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Friday</td><td>√</td><td>√</td><td>√</td><td>√</td><td>-</td><td>-</td></tr></tbody></table>

Note: Interpreting the above results:

- /ah/, /m/, and /i/ were all detected consistently and correctly

- /sh/ and /u/ are not being detected consistently

- /s/ is not being detected at all.

# 4 Findings

## 4.1 Gains in detecting sounds and following instructions

Using the detailed data recording as of Table 2, a percent success score was computed for each child per activity per week; see Table 3 for the average percent success
scores across children. Plotting of participants' success scores across weeks allowed us to realize the progress of each participating child's gains in the ability to understand tasks presented through NAO over time. Some children mastered the detection of sounds earlier than others (e.g., by week 3; around 10 daily sessions), while all of them presented a consistent record of detecting sounds correctly by the last week of the study (i.e., week 6; 16-22 daily sessions). Figure 2 illustrates the progress charts of two of a participating child, as example. All participating children had steadily progress on all activities until a consistent correct pattern of detecting sounds occurred towards the last two weeks of the study.

<table><caption>**Table 3.** Average percent (%) success score across participants, on weekly activities</caption><thead><tr><th>Weeks (up to 5 sessions per week)</th><th>Vegetables (child follows instruction correctly)</th><th>Farm animals (child follows instruction correctly)</th><th>Music density (child beats the drum louder or lower immediately after the density changes)</th><th>Ling sounds story (child picks the correct object)</th></tr></thead><tbody><tr><td>Week 1</td><td>19%</td><td>20%</td><td>14%</td><td>12%</td></tr><tr><td>Week 2</td><td>28%</td><td>30%</td><td>14%</td><td>12%</td></tr><tr><td>Week 3</td><td>28%</td><td>50%</td><td>24%</td><td>22%</td></tr><tr><td>Week 4</td><td>55%</td><td>70%</td><td>70%</td><td>42%</td></tr><tr><td>Week 5</td><td>90%</td><td>88%</td><td>100%</td><td>82%</td></tr><tr><td>Week 6</td><td>100%</td><td>90%</td><td>100%</td><td>95%</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** An example of child's progress chart in terms of % correct answers across weeks (percent of correct answers on y-axes)</figcaption></figure>

With respect to observed gains in detecting sounds and following instructions, the perspectives of the school therapist and special teacher were consistent with the quantitative outcomes. Both argued that the auditory-verbal therapy with NAO seemed to improve the children's auditory skills, including their ability to pay attention to NAO
and to use their amplifier technology rather than relying on external cues. They also noted their observation of increased self-confidence for the participating children, as they performed better and better from session to session and as NAO positively reinforced their responses (“good job” and green eyes). As the therapist explained:

> "NAO kept children's attention, which is very important in developing verbal communication skills. NAO is interesting and cute enough to keep the child playing and completing the tasks while s/he is forced to listen. In early sessions, I noticed that all of them looked at me again and again seeking for clues. Because I directed their attention to NAO, they learnt to rely less on other cues, and use their amplifiers more, to play with NAO. I think the use of the robot in this sense is brilliant! [...]. I also noticed gains in self-confidence from session to session and as children started learning to listen and to recognize the sounds that NAO spoke, for example [child's name] was very shy at the beginning, but gradually she became confident and enthusiastic in playing with NAO."

## 4.2 Playful and engaging therapy sessions

The therapist elaborated on how the children responded positively and enthusiastically to the activities during the auditory-verbal therapy. As she explained,

> "...the auditory-verbal therapy is not always pleasant for the child with hearing impairment. The child seeks for cues and s/he is confused and annoyed when I cover my mouth. I often hold a picture in front of my mouth, to distract him/her from the fact that I am covering my mouth on purpose."

Yet, both therapist and special teacher thought that NAO robot made auditory-verbal therapy fun for the child with hearing impairment, engaging them in a playful learning experience. In the teacher's own words:

> "Children were happy playing with NAO and I saw their enthusiasm, in completing the tasks and working with NAO. Their joy lasted for the duration of the activity, and even after therapy, for the duration of the school day."

In addition to NAO's great prospects for speech therapy, a rather unforeseen idea in teachers' feedback was the potential of play and learn with NAO in inclusive education settings. In their view, the experience could be beneficial for both hearing impaired students and their hearing peers (i.e., the activities could serve all students in class). For example, as the special teacher explained, the Music Density game could involve more density variations, from sharp & steep to smooth & gradual changes; this could make it challenging for hearing children of this age, in a fun, class-wide, music activity. Also, the special teacher thought that 3-4-year old hearing children do not necessarily know the sounds of farm animals and with some more customization (e.g., to include less familiar animal sounds) the activity could easily challenge the hearing child. This way all children could enjoy sessions of play and learn with NAO. Furthermore, in the special teachers' view, the use of NAO in typical lessons both
learning and children with hearing impairment could expand the auditory-verbal therapy time for these children who unavoidably do much lipreading and use other non-verbal cues as the teachers interacts with the whole class. As stated by the special teacher:

> "I think it is a given that NAO can be used successfully in auditory-verbal therapy sessions. But he could work equally well in the inclusive classroom. Some of our hearing 3-year old children do not necessarily know the farm animals yet, neither the vegetables, so there could be value for them in playing with NAO [...]. I just think the possibilities of this technology are beyond individual speech therapy sessions. Also, consider that in class, my students with hearing impairment continue to do lipreading, which is unavoidable. NAO could help me expand the auditory-verbal therapy time for them without taking anything away from my hearing children [...]."

# 5 Discussion

This study aimed to examine an innovative way of using an intelligent robot to support speech therapy for hearing impaired children through play. The study presented a unique environment for auditory-verbal therapy for hearing impaired children. NAO does not have a human mouth and therefore it is not supporting lipreading during auditory-verbal therapy; this unique characteristic of the technology was successfully used in this study to create playful therapy sessions for six kindergarten children with hearing impairment over the course of six weeks. The co-designed activities and implementation procedures were detailed in this work. The authors believe that the innovative idea can be successfully transferred and replicated with success in similar circumstances and learning/therapy contexts.

Are there gains in the ability of the hearing-impaired child to detect sounds and follow instructions? The study demonstrated quantitative gains in hearing impaired children's ability to respond correctly to sounds and to follow directions. All participating children had steadily progress on all activities until a consistent correct pattern of detecting sounds occurred towards the last two weeks of the study. We need to acknowledge however, that because no baseline measurements were provided as to how well the children could complete the tasks in non-NAO contexts, or data about their level of development in speech, language, or listening (please see discussion of confidentiality earlier), the results presented can only describe children's gains in the ability to understand tasks presented through NAO over time, which may or may not, reflect a change in their level of skill in listening and language. Future work should aim to address these important limitations of the present work.

What is the perceived value of the robot-enhanced environment for auditory-verbal therapy? The added value of the intelligent robot during the auditory-verbal therapy was documented by the speech therapist and special teacher-observer in a couple of ways. The views of the researchers are also consistent with these reported observations. First, they noted observed gains in children's ability to pay attention to NAO and to use their amplifier technology rather than relying on external cues. In other
words, the hypothesized affordance of the intelligent robot was well materialized in the implementation. Second, they noted the playfulness of the learning experience and the children's overall positive attitude towards the lack of non-verbal cues. This finding replicates previous work on children learning through play with intelligent robots, especially involving ASD children (e.g., [14, 18, 36]). Overall, the study might be suggesting that NAO can make auditory-verbal therapy fun and engaging for children with hearing impairment, compared to an experience in which the therapist must hide all non-verbal cues (i.e., covering mouth). Of course, we need to acknowledge that the input for the therapist may be biased because of her involvement in the design of the activities and implementation with NAO. We were bounded by this limitation in the context of this study; despite the speech therapy (research and practice) and educational technology expertise of the researchers, the direct involvement of the school therapist was instrumental for the investigation in an authentic school-therapy setting and practice. Overall however, the findings suggest an innovative way of using an intelligent robot in this context and justify further investigation.

Furthermore, the technology and experience were perceived as a promising path-
way to inclusive education, an unforeseen finding, which however, the authors, con-
sider a significant reflection. A complete suite of NAO applications with customiza-
ble content and difficulty levels can support the inclusive (kindergarten) classroom
and benefit both hearing and students with hearing impairment. How intelligent ro-
bots can support the educator in adopting an inclusive education approach is a very
promising direction for future research, considering the rise of children with hearing
impairment in mainstream schools vis-à-vis the underutilization of novel technologies
[43]. Yet, while the above-mentioned idea presents an exciting opportunity for future
research in inclusive education settings, the present study does not yet provide evi-
dence that NAO would be appropriate for or beneficial for use with children without
hearing loss. Also, in a mainstream classroom (i.e., an inclusive setting) background
noise in the classroom could make the task impossible for the children with hearing
loss. These ideas coupled with plausible difficulties merit investigation in future re-
search.

All in all, this study aimed to explore the humanoid robot's potential for assisting therapists auditory-verbal exercises. Our conclusions are provisional and at this stage, we can only claim that NAO has potential in this context. A potential is only a possibility; we cannot offer any certainties, based on this initial work, but we do share our positive initial impression. The study remains a subject of future work and replication. We have offered our lessons learned as well as initial evidence that NAO could be an important means of assisting therapy of children with hearing impairment. We further demonstrated that NAO is an attractive tool in this therapy context because of the lack of visual clues (has no mouth) while at the same time it is fun and friendly [16]. We believe that a laptop or tablet could not have had the same effect, although future work could address this hypothesis via an experiment design with a control/comparison group.
# 6 Conclusion

The presented study focused on the use of the humanoid NAO robot in auditory-verbal therapy, an approach to the development of auditory and verbal skills, which does not allow lipreading or other non-verbal cues to facilitate communication. This is, in our view, innovative and is worthy of further exploration while aiming to address the limitations of the present study.

Play with intelligent robots is now a reality. The question of interest is how to best utilize such technology-mediated experiences for the sake of learning, including therapy. The study contributes to the technology-enhanced learning, speech therapy and special education communities by presenting a case of practical utility of humanoid robots with real world impact. The findings of this study are encouraging and warrant further investigation to fully exploit the possibilities of intelligent robots in the context of education and therapy for children with hearing impairment.

# Acknowledgements

This work has been partly supported by the project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 739578 (RISE - Call: H2020-WIDESPREAD-01-2016-2017-TeamingPhase2) and the Government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.

# References

1. Spencer, P. E., & Marschark, M.: Evidence-based practice in educating deaf and hard-of-hearing students. Oxford University Press (2010).

2. Lee, C.: Deafness and cochlear implants: a deaf scholar's perspective. Child Neurology of the Journal, 27 (6), 821-823 (2012). https://doi.org/10.1177/0883073812441248

3. Beal-Alvarez, J., Cannon, J. E.: Technology intervention research with deaf and hard of hearing learners: Levels of evidence. American Annals of the Deaf, 158(5), 486 – 505 (2014).

4. Kelman, C. A., Branco, A. U.: (Meta) Communication strategies in inclusive classes for deaf students. American annals of the deaf, 154(4), 371–381(2009).

5. Wilkins, M., Ertmer, D. J.: Introducing Young Children Who Are Deaf or Hard of Hearing to Spoken Language: Child's Voice, an Oral School. Language, Speech, and Hearing Services in Schools, 33(3), 196-204 (2002).

6. Ginsburg, K. R.: The importance of play in promoting healthy child development and maintaining strong parent-child bonds. Pediatrics, 119(1), 182-191 (2007).

7. Mills, P. E., Beecher, C. C., Dale, P. S., Cole, K. N., Jenkins, J. R.: Language of children with disabilities to peers at play: Impact of ecology. Journal of Early Intervention, 36(2), 111-130 (2014).

8. Qayyum, A., Khan, A. Z., Rais, R. A.: Exploring Play of Children with Sensory Impairments in Special Schools at Karachi, Pakistan. The Qualitative Report, 20(2), 1-17 (2015).
9. Bobzien, J., Richels, C., Raver, S. A., Hester, P., Browning, E., Morin, L.: An observational study of social communication skills in eight preschoolers with and without hearing loss during cooperative play. Early Childhood Education Journal, 41(5), 339–346 (2013).

10. Cejas, I., Barker, D. H., Quittner, A. L., Niparko, J. K.: Development of joint engagement in young deaf and hearing children: effects of chronological age and language skills. Journal of Speech, Language, and Hearing Research, 57(5), 1831–1841 (2014).

11. Harris, L. G.: Social-Emotional Development in Children with Hearing Loss. Theses and Dissertations - Communication Sciences and Disorders. 4. (2014).
http://uknowledge.uky.edu/commdisorders_etds/4/, last accessed 2017/08/01.

12. Sininger, Y. S., Grimes, A., Christensen, E.: Auditory development in early amplified children: Factors influencing auditory-based communication outcomes in children with hearing loss. Ear and hearing, 31(2), 166–185 (2010).

13. Yuhan, X.: Peer Interaction of Children with Hearing Impairment. International Journal of Psychological Studies, 5(4), 17–25 (2013).

14. Arendsen, J., Janssen, J. B., Begeer, S., Stekelenburg, F. C.: The use of robots in social behavior tutoring for children with ASD. In 28th Proceedings on Annual European Conference on Cognitive Ergonomics pp. 371–372. ACM. (2010).

15. Fridin, M.: Storytelling by a kindergarten social assistive robot: A tool for constructive learning in preschool education. Computers & education, 70, 53–64 (2014).

16. Ioannou, A., Andreou, E., Christofi, M.: Preschoolers’ interest and caring behaviour around a humanoid robot. TechTrends, 59 (2), 23–26 (2015).

17. Kartapanis, I., Ioannou, A., Zaphiris, P.: NAO as an assistant in ASD therapy sessions: The case of Joe. INTED2015 Proceedings. (2015).

18. Lee, H., Hyun, E.: The intelligent robot contents for children with speech-language disorder. Journal of Educational Technology & Society, 18(3) (2015).

19. Polycarpou, P., Andreeva, A., Ioannou, A., Zaphiris, P.: Don't Read My Lips: Assessing Listening and Speaking Skills through Play with a Humanoid Robot. In: Proceedings of Human-Computer Interaction International 2016, pp. 2559–260. Toronto, Canada (2016).

20. Welling, D. R., Ukstins, C. A.: Fundamentals of Audiology for the Speech-language Pathologist. Jones & Bartlett Learning. (2017).

21. Shonkoff, J.P., Phillips, D.A., ed.: From Neurons to Neighborhoods: The Science of Early Childhood Development. Washington (DC): National Academies Press (US); (2000).
Available from: https://www.ncbi.nlm.nih.gov/books/NBK225557/ doi: 10.17226/9824

22. World Health Organization, https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss, last accessed 2018/08/28

23. Marshark, M., Hauser, P. C.: How Deaf Children Learn: What Parents and Teachers Need to Know. Oxford University Press (2012).

24. Decker, K.B.: Influences on Parental Decisions Regarding Communications Options for Children Identified with Hearing Loss. Human Development and Family Studies, pp. 1409. Michigan: Michigan State University (2010).

25. Gallaudet Research Institute: Regional and national summary report of data from the 2007-08 annual survey of deaf and hard of hearing children and youth (2008).

26. Crowe, K., McLeod, S., McKinnon, D. H., Ching, T. Y. C.: Speech, sign, or multilingualism for children with hearing loss: Quantitative insights into caregivers' decision-making. Language, Speech, and Hearing Services in Schools, 45, 234–247 (2014).

27. Crowe, K., Fordham, L. A., McLeod, S., & Ching, T. Y. C.: Part of our world: Influences on caregiver decisions about communication choices for children with hearing loss. Deafness and Education International, 16(2), 61-85. (2014).
28. Paul, P.V., Whitelaw, G. M.: Hearing and Deafness an Introduction for Health and Education Professionals. Jones & Bartlett Publishers (2011)

29. Cooper, H., Craddock, L.: Cochlear Implants: A Practical Guide, 2nd Edition. Whurr Publishers London and Philadelphia (2006)

30. Cole, E. B., Flexer, C.: Children with Hearing Loss: Developing Listening and Talking Birth to Six. San Diego, CA: Plural Publishing. (2015).

31. Ling, D.: Speech and the hearing impaired child: Theory and practice; Second edition, Washington, DC: Alexander Graham Bell Association for the Deaf (2002)

32. Estabrooks, W., Maclver-Lux, K., Rhoades, E.A.: Auditory-verbal therapy: For young children with hearing loss and their families, and the practitioners who guide them. San Diego, CA: Plural Publishing, p.286 (2016).

33. MuSiEk, F. E.: Neurobiology, Cognitive Science, and Intervention. Handbook of Central Auditory Processing Disorder, Volume II: Comprehensive Intervention, 2, 1, Plural Publishing (2013).

34. Keren, G., Ben-David, A., Fridin, M.: Kindergarten Assistive Robotics (KAR) as a Tool for Spatial Cognition Development in Pre-school Education. International Conference on Intelligent Robots and Systems October 7-12, 2012. Vilamoura, Algarve, Portugal (2012).

35. Malik, N. A., Yussof, H., Hanapiah, F. A.: Development of Imitation Learning Through Physical Therapy Using a Humanoid Robot. International Conference on Robot PRIDE 2013-2014 - Medical and Rehabilitation Robotics and Instrumentation, ConfPRIDE 2013-2014, Procedia Computer Science, vol. 42, p. 191-197, Elsevier (2014)

36. Hamzah, M. S. J., Shamsuddin, S., Miskam, M. A., Yussof, H., Hashim, K. S.: Development of Interaction Scenarios Based on pre-school Curriculum in Robotic Intervention for Children with Autism. International Conference on Robot PRIDE 2013-2014 - Medical and Rehabilitation Robotics and Instrumentation, ConfPRIDE 2013-2014, Procedia Computer Science, vol. 42, p. 214-221, Elsevier (2014)

37. Robins, B., Dautenhahn, K., Boekhorst, R., Billard, A.: Robotic assistants in therapy and education of children with autism: can a small humanoid robot help encourage social interaction skills? In: Universal Access in the Information Society, vol.4, pp. 105-120. Springer, Heidelberg (2005)

38. Srinivasan, S. M., Park, I. K., Neelly, L. B., Bhat, A. N.: A comparison of the effects of rhythm and robotic Interventions on Repetitive Behaviors and affective States of children with Autism Spectrum Disorder (ASD). Research in Autism Spectrum Disorders, vol. 18, p. 51-63, Elsevier (2015)

39. Salleh, M. H. K., Yussof, H., Ainuddin, H. A., Muda, M. Z., Shamsuddin, S., Miskam, M. A., Omar, A. R.: Experimental Framework for the Categorization of Special Education Programs of ASKNAO, 76 OP-I, 480 (2015).

40. Sawada, H., Kitani, M., Hayashi, Y.: A robotic voice simulator and the interactive training for hearing-impaired people. BioMed Research International, 2008 (2008).

41. Kose-Bagci, H., Yorganci, R.: Tale of a robot: Humanoid Robot Assisted Sign Language Tutoring. In: 11th IEEE-RAS International Conference on Humanoid Robots, pp. 105-111. IEEE, Bled Slovenia (2011)

42. Uluer, P., Akalin, N., Kose, H.: A New Robotic Platform for Sign Language Tutoring Humanoid Robots as Assistive Game Companions for Teaching Sign Language. In: International Journal of Social Robotics, vol.7, pp.571-585. Springer, Netherlands (2015)

43. Constantinou, V., Ioannou, A., Klironomos, I., Antona, M., Stephanidis, C.: Technology support for the inclusion of deaf students in mainstream schools: A summary of research from 2007 to 2017. Universal Access in the Information Society, 1-6 (2018).
44. Besio, S. (2017). The Need for Play for the Sake of Play. In: Besio, S., Bulgarelli, D., Stancheva-Popkostadinova, VaskaPlay development in children with disabilities