# Sign Language GIFs Exchange Communication System: A PECS-Based Computer-Mediated Communication Tool for the Deaf

Celeste Zhilla, Giulio Galesi, Barbara Leporini

► To cite this version:

Celeste Zhilla, Giulio Galesi, Barbara Leporini. Sign Language GIFs Exchange Communication System: A PECS-Based Computer-Mediated Communication Tool for the Deaf. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.490-494, 10.1007/978-3-030-85607-6_64. hal-04291194

HAL Id: hal-04291194

https://inria.hal.science/hal-04291194v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Sign Language GIFs Exchange Communication System:
A PECS-Based Computer-Mediated Communication Tool
for the Deaf

Celeste Zhilla, Giulio Galesi, Barbara Leporini

ISTI-CNR, Pisa, Italy

zhillaceleste@gmail.com {giulio.galesi, barbara.leporini}

@isti.cnr.it

**Abstract.** Thanks to technological advances, Sign Language (SL), which is used by most deaf people, has gradually been freed from the need for face-to-face interaction. Deaf people used to communicating via SL may experience many problems in writing and reading text contents. Considering those difficulties, we propose a messaging system that integrates a Graphics Interchange Format (GIF) gallery representing phrases and words in SL to promote written communication closer to the needs of this user category.

**Keywords:** Deaf, sign language, CMC, GIF, chat

## 1 Introduction

According to the World Health Organization (WHO), around 466 million people worldwide currently have disabling hearing loss, and around 95% of deaf/hard of hearing children are born to hearing-impaired parents. Based on the estimation of WHO, 900 million people will be suffering from hearing loss in 2050¹.

Many hearing-impaired people use Sign Language (SL) for communication. SL is a linguistic system, consisting of a structured and organized set of signs, with a precise grammar, syntax, and morphology of its own. However, SL communication is not universal. Differences exist between countries and there are even regional variations of SL [1].

SL is the preferred choice for people who have totally lost their hearing capability. SL users rarely use hearing aids, write notes or lip read. So, SL communication leads to several issues for deaf people when interacting with non-deaf people that cannot understand SL. The development of methods that provide some types of ICT solutions for deaf-hearing and hearing-deaf communication has increased. In order to ease their communication, there is considerable research activity in the development of possible supporting tools. These tools have a wide range from systems of Speech to Text (which translates verbal speech into written words and vice versa with Text to Speech), Display System, Haptic or Visual Feedback (e.g., applications focused on multimodal

<footnote n="1">¹ https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss</footnote>
approach, able to convert speech to visual contexts and vibrations, and similarly, the contexts and vibrations can be converted to speech [2]) up to Mobile support [3].

There are also wearable technologies but, that often means that the deaf person must wear numerous very visible devices (e.g., gloves with sensors, helmet with infra-red filters camera). This could potentially represent a stigma for the hearing-impaired person.

Computer-mediated SL communication involves new ways to manipulate language structure and performance, including experimentation of new proposed language forms. Thanks to the developments of new tools, for the first time, hearing-impaired people can communicate using manual visual language, which in many cases is their native language, across space and time zones [4]. Also, the animation capabilities of modern screens offer an exciting opportunity to create a new communication system for people who use SL to communicate. So, SL communication needs to be further investigated to support it through tools and applications.

This work is focused on a component proposed in a project aimed at developing a remote video application to support SL interpreting. A messaging system prototype that makes use of Augmentative Alternative Communication (AAC) methodology is presented. SL is a movement-based language without a standard written form; because of this, deaf people communicating via SL may face several difficulties, including the use of the written form [5]. In order to support chat communication between SL users, in our prototype, we propose the integration of a gallery of Graphics Interchange Format (GIF) representing phrases and words in SL format to promote written communication closer to the needs of the deaf or hard-hearing people.

## 2 Design approach

Communication is one of the most important aspects affecting the social interaction for the deaf. Additional methods such as the use of AAC may be used to overcome communication problems and barriers for deaf people [6]. AAC is an area of clinical practice that seeks to reduce, moderate, and compensate for the temporary or permanent difficulties of people with severe communication disorders both on the expressive and receptive fronts and help them in their social needs of life [7]. For deaf people, one of the possible used AAC-based practices are representational systems [8]. The best known is the Picture Exchange Communication System (PECS), which is a communication strategy based on the exchange of several PECS images that make up a sentence. PECS is used to remedy the mismatch in the communication methods of people with deafness and one or more intellectual disabilities (mild or moderate) who communicate through the SL and those who do not use it. In this way, studies - proposing illustrated dictionaries to facilitate message writing intended for their communication partners - have shown that, with adequate preparation regarding the rules of use, the subjects were able to use these tools with success and they have had the role of facilitating writing [9].

Following the representational assisted AAC methodology, in this work, we propose the use of short and silent moving images, the GIFs with the aim to facilitate text-based
computer-mediated communication for the deaf. The idea is proposing a chat compo-
nent based on the AAC communication system. To prepare the aforementioned GIFs,
videos provided by the Spread the Sign's website were converted through a special
app.

3 Chat Prototype

The chat component is part of a remote communication tool designed for those deaf people who use SL interpretation. Similar characteristics and tasks offered by typical popular chat systems for the communication between users are provided by the proposed chat component (see Figure 1). To support chat communication for the deaf user, in addition to the traditional text editing via keyboard, a series of GIFs is shown when the specific button "GIF SL" is hit. Those GIFs representing sentences and words expressed in SL can be selected by the user to edit the message to be sent via chat.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Chat prototype window.</figcaption></figure>

The GIFs set developed for the chat prototype contains a limited number of GIFs
images, but it was enough to test the proposed approach. Therefore, the main limitation
is that the user may not find the appropriate SL GIF needed to compose and send a
certain word/phrase. On the other hand, the main benefit of GIFs is that they can auto-
matically play short, non-audio "looping" video clips [10]. Thanks to these characteris-
tics, the GIF format is well suited as a representation system for assisted AAC. GIFs,
in fact, as non-audio contents, are a very suitable medium for the SL rendering, be-
cause they can show SL characteristics like: (1) the area where the sign is made, show-
ing part of the signer's body; (2) the configuration of the hands in executing the sign,
that is the shape it assumes by positioning the fingers; (3) the orientation of the palm of
the hand in making the sign concerning the signer's body; (4) the movement of the
hand or hands in carrying out the sign [11]. The GIF format can play animations in an
endless silent loop, so it is suitable for learning new signs.

The GIFs gallery layout is in line with the common grid outline used for the AAC
systems, in which the symbols are in specific positions within a grid (see Figure 2). As
each symbol is isolated, it supports the recall of the individual contents of each cell
[12]. The use of GIFs representing SL movements grouped in a gallery is an interesting
PECS-based methodology that can be great benefit to the Deaf community.

<figure><img src="image_4.png" /><figcaption>Fig. 2. A subset of a SL GIFs gallery</figcaption></figure>

4 Conclusions and Future Work

This paper presents a potential assistive technology to support chat communication by the deaf. The chat prototype was designed to be used by deaf people who use SL to communicate and have difficulty with written production. The proposed system differs from existing solutions in (1) no advanced digital skills are required and (2) only a few clicks are required to write SL messages.

The current chat prototype has the sign language limitation which needs to be consid-
ered in the future work. In fact, our proposed system has been designed for the Italian
Sign Language (LIS), and so it needs to be adapted to the various existing sign lan-
guages. So, further future development will have to consider the interface localization
so that the gif gallery will be shown in the selected language.

In addition, future studies will further develop the chat tool so that the system provides automatic translation from text messages to GIFs (i.e. to SL) and vice versa, thus also allowing communication between a person who uses SL to communicate and one who does not.
References

1. Tilano-Vega, L.M. Cardenas-Torres, A.M., Betancur-Caro, I.C., Rivera-Muñoz, L.M., Gómez-Gómez, B.L., Arango-Restrepo, J.P., Moreno-Asprilla, Y., Jaramillo-Velásquez, A.: Tools Facilitating Communication for the Deaf. Educación y Educadores. 17. 468-480. (2014)

2. Sobhan, M., Chowdhury, M.Z., Ahsan, I., Hasan, M., Kamrul, H.: A Communication Aid System for Deaf and Mute using Vibrotactile and Visual Feedback. 184-190. 10.1109/ISEMANTIC.2019.8884323. (2019)

3. Dhanjal, A.S., Singh, W.: Tools and Techniques of Assistive Technology for Hearing Impaired People. International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), pp. 205-210 (Faridabad, India). (2019)

4. Keating, E., Mirus, G.: American Sign Language in Virtual Space: Interactions between Deaf Users of Computer-Mediated Video Communication and the Impact of Technology on Language Practices. Language in Society, vol. 32, no. 5, pp. 693-714. (2003)

5. Marschark, M., Mouradian, V., Halas, M.: Discourse Rules in the Language Productions of Deaf and Hearing Children, Journal of Experimental Child Psychology, Volume 57, Issue 1, Pages 89-107, ISSN 0022-0965. (1994)

6. Davis, T.N., Barnard-Brak, L., Dacus, S., Pond, A.: Aided AAC systems among individuals with hearing loss and disabilities. Journal of Developmental and Physical Disabilities, 22, 241-256 (2010)

7. Koul, R.K., Lloyd, L.L.: Survey of Professional Preparation in Augmentative and Alternative Communication (AAC) in Speech-Language Pathology and Special Education Programs. American Journal of Speech-Language Pathology, 3(3), 13-22 (1994).

8. Malandraki G.A., Okalidou A.: The Application of PECS in a Deaf Child With Autism: A Case Study. Focus on Autism and Other Developmental Disabilities. 22(1):23-32. (2007)

9. Allgood, M., Kathryn, H., Easterbrooks, S., Fredrick, L.: Use of Picture Dictionaries to Promote Functional Communication in Students with Deafness and Intellectual Disabilities. Communication Disorders Quarterly, 31, 56-64 (2009)

10. Morris, S., Vollmer, T.: A comparison of picture and GIF-based preference assessments for social interaction. Journal of applied behavior analysis. 53. 10.1002/jaba.680. (2020)

11. Goldin-Meadow, S., Brentari, D.: Gesture, sign, and language: The coming of age of sign language and gesture studies. The Behavioral and brain sciences, 40, e46. (2017)

12. Cooper, A., Ireland D.: Designing a Chat-Bot for Non-Verbal Children on the Autism Spectrum. Studies in health technology and informatics vol. 252: 63-68. (2018)