See-Through Captions: Real-Time Captioning on Transparent
Display for Deaf and Hard-of-Hearing People
Kenta Yamamoto∗1 , Ippei Suzuki∗1 , Akihisa Shitara∗1 , and Yoichi Ochiai1
1

University of Tsukuba

Hearing Person

DHH Person

Figure 1: Proposed real-time captioning system.
Abstract
Real-time captioning is a useful technique for deaf and hard-of-hearing (DHH) people to talk to hearing
people. With the improvement in device performance and the accuracy of automatic speech recognition(ASR), real-time captioning is becoming an important tool for helping DHH people in their daily lives.
To realize higher-quality communication and overcome the limitations of mobile and augmented-reality
devices, real-time captioning that can be used comfortably while maintaining nonverbal communication andpreventing incorrect recognition is required. Therefore, we propose a real-time captioning system that uses
a transparent display. In this system, the captions are presented on both sides of the display to address
the problem of incorrect ASR, and the highly transparent display makes it possible to see both the bodylanguage and the captions.

1IntroductionDeaf and hard of hearing (DHH) people need to be more accessible to sound information in their daily lives.
The importance of spoken language in interpersonal communication, education, and business situations has
necessitated its visualization. Visualization methods include real-time translation by an interpreter using sign
language [1] and real-time spoken-language transcription by a supporter [2]; however, they are expensive and thus
infeasible for casual daily use. In recent years, real-time captioning via automatic speech recognition (ASR)
has attracted attention because of the advances in the processing performance of mobile devices, internetcommunication speed, and speech-recognition accuracy. This technology enables the provision of support toDHH people in various situations.
ASR has long been expected to serve as a universal method of accessing voice information [3]. Significant
effort has been devoted to introduce such a technology in the field of education [4] and implement the use ofthis technology by DHH people [5, 6, 7]. The main method used in the educational setting is the capturing of∗ Three authors contributed equally to this research.

1Table 1: Comparison with previous methods of real-time captioning.
Mobile Device

AR DeviceTransparent Display

YesNo

YesNo

Yes

Yes

Multiple People

Multiple People

One-to-One

One-to-One(Smartphone, Smartwatch)
Confirmation of ASR result
by hearing person
Seeing body language ofhearing person
Situation of communication

One-to-One

teachers’ voice information via ASR. After studies in a static context such as lectures in a classroom, research on
the free utilization of speech recognition by DHH people in various situations has been conducted. In methods
adopted in previous studies, the sound information to be recognized by DHH people was sent to translators
via the internet, and the text that they verbalized was sent to mobile devices [8, 9]. With the developmentof high-performance mobile devices, such as smartphones and smartwatches, research on text conversion via
ASR on mobile devices has increased [10]. Furthermore, with the progress made in the augmented reality (AR)technology, research on the use of AR to realize caption displays is being actively conducted [10, 11, 12, 13].
The method of realizing real-time captioning via ASR using such mobile and AR devices has some limitations.
For example, when using a mobile device, the body language of the partner cannot be confirmed because themobile device must be viewed to see the ASR result; when using an AR device, DHH people can see the voicerecognized text while observing the partner’s body language. Previous studies have favored this advantage ofusing an AR device [14]. However, when an AR device is used, the speaker who does not wear it cannot confirm
whether the speech has been correctly recognized, which may lead to errors in communication.
In this study, to address these problems, we developed a real-time captioning system that utilizes a transparent display and allows people to check the speech-recognition results while seeing their partner during theconversation (Fig. 1). Table 1 summarizes the characteristics of the proposed system and existing methods (mobile devices and AR devices). Although the proposed system is limited to one-on-one communication, we expect
it to help in improving the quality of communication because it can prevent incorrect ASR and overlooking ofthe body language of the partner. In addition, the installation of this system in common places where voice
conversation is expected, such as cash registers in supermarkets and reception desks at government offices, canpossibly help DHH people in their daily lives.

2

Proposed System

2.1

System Configuration

The proposed system presents the result of real-time captioning via ASR on a transparent display, and hearing
and DHH people can communicate while checking the speech-recognition result. The two main functions of thissystem are ASR and caption display. First, a directional microphone is used as a speech-input device for ASR,
and only the speech of the hearing person is used as the input speech. The input speech is converted into text
via ASR. The system uses SpeechRecognition of the Web Speech API from Google Chrome (ver 87.0.4280.88)
for ASR. Next, the text of the speech-recognition result is presented on the transparent display in real time. In
this prototype, a transparent display from Japan Display Inc.1 is used, and the entire processing from ASR tocaption display is performed on the browser.

2.2

Communication Process

The proposed system is expected to be used in one-to-one and face-to-face communication. The conversation
between two people across a transparent display is illustrated in Fig. 2. A hearing person speaks into a speechinput device, and the recognized speech is converted into text and presented on a transparent display. The
captions are primarily meant for the DHH person to read. However, to enable the hearing person to confirm
that there is no significant difference between their speech and the ASR result, the same text as the caption
for the DHH person is displayed with the left and right reversed. If the ASR result is incorrect, the hearing
person indicates through gestures that there was a recognition error and then repeats their speech such that it
is correctly recognized. The DHH person understands the spoken-language content by reading the captions on
the transparent display and responding through their voice. The hearing person listens to the voice of the DHH
1 https://www.j-display.com/technology/jdinew/clear_display.html [Last Accessed: 01/11/2021]

2

for DHH Person

for Hearing Person

Figure 2: (left) Photograph of DHH person and hearing person using the proposed system. They talk using
their voice while seeing the captions displayed on the transparent display. (right) Captions presented on thedisplay. A small caption with left and right flips is displayed for the hearing person.
person and inputs the voice to the microphone again. The conversation continues with a repetition of theseexchanges. The characteristics of the transparent display allow for the displaying of captions on both sides ofa single display.

2.3

Caption Design

The caption design has a significant effect on readability. Previous studies have investigated the captionappearance preferences [15] and designs based on the reliability of speech recognition [16]. The proposedsystem provides an interface that allows the user to change the caption-design parameters such as character
size, character color, character transparency, and font. There are some design parameters that have not been
examined; these include the character-display speed, number of lines, and the caption-disappearance speed.
Therefore, further improvements are required in the future.

3

Design Space of See-Through Caption

3.1

How to Design Optimal Real-Time Captioning

What is the optimal real-time captioning? We consider three factors to be important for superior real-time
captioning, as shown in Fig. 3.
3.1.1

Assisting in understanding of spoken language

Enabling accurate and smooth access of audio information by DHH people is the most important role of realtime captioning. Therefore, it is necessary to improve not only the speech-recognition accuracy but also theoverall quality of captioning, and we must thus consider the design of characters that make up captions andhow they can be displayed.
3.1.2

Not interfering nonverbal communication

Communication between people is not limited to spoken language. A previous study emphasized the importanceof eye contact during a conversation [14]. Captioning tools should be designed for nonverbal communication
as well. Through the use of a transparent display, the proposed system avoids any obstruction to nonverbal
communication, similar to captioning by AR devices. In the future, it will be necessary to further consider how
captions can be positioned relative to the speaker and how they can displayed such that they do not obstructnonverbal communication.
3.1.3

Not spoiling interior design

Finally, we should consider how the device for real-time captioning contributes to the designing of an indoorspace. As mentioned earlier, the proposed system is expected to be used in daily life (e.g., at cash registersand receptions). However, in such situations, captioning is only used when necessary. Therefore, it is also
important to integrate the display at the installation location. Because mobile and AR devices are carried by
a DHH person, the location design need not be considered. Owing to the transparency of the display, visual
impediments are less than those in the case of a normal display; however, a detailed discussion will be required.

3Environment
Non-verbal Communication

CaptionFigure 3: Three factors for superior real-time captioning system with transparent displays. This picture wastaken from the viewpoint of a DHH person when using this system.

3.2

Future Work

We have developed and demonstrated a see-through type real-time captioning system that utilizes a transparent
display. Shitara, who is a deaf person and one of the authors of this manuscript, was involved in the development
and designing of the system. However, more user studies are required to achieve a better design. For example,
it is necessary to further clarify the merits and demerits of this system from the user studies and carefullycompare them with those of mobile and AR devices. Furthermore, it is necessary to investigate the relationshipbetween the understanding of verbal and nonverbal communication in real-time captioning by performing eyegaze analysis during conversations. We believe that this system is useful for supporting the communication
with DHH people, and we will conduct appropriate user studies to achieve a better system.

References[1] Brenda Schick, Kevin Williams, and Haggai Kupermintz. Look Who’s Being Left Behind: EducationalInterpreters and Access to Education for Deaf and Hard-of-Hearing Students. The Journal of Deaf Studies
and Deaf Education, 11(1):3–20, 09 2005.
[2] Walter Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato Borrello, Raja Kushalnagar, and Jeffrey Bigham. Real-time captioning by groups of non-experts. In Proceedings of the 25th Annual
ACM Symposium on User Interface Software and Technology, UIST ’12, page 23–34, New York, NY, USA,2012. Association for Computing Machinery.
[3] Mike Wald and Keith Bain. Universal access to communication and learning: the role of automatic speech
recognition. Universal Access in the Information Society, 6(4):435–447, 2008.
[4] Keith Bain, Sara H. Basson, and Mike Wald. Speech recognition in university classrooms: Liberated
learning project. In Proceedings of the Fifth International ACM Conference on Assistive Technologies,
Assets ’02, page 192–196, New York, NY, USA, 2002. Association for Computing Machinery.
[5] Richard Kheir and Thomas Way. Inclusion of deaf students in computer science classes using real-time
speech transcription. In Proceedings of the 12th Annual SIGCSE Conference on Innovation and Technology
in Computer Science Education, ITiCSE ’07, page 261–265, New York, NY, USA, 2007. Association forComputing Machinery.
[6] Raja S. Kushalnagar, Walter S. Lasecki, and Jeffrey P. Bigham. Accessibility evaluation of classroomcaptions. ACM Trans. Access. Comput., 5(3), January 2014.
4

[7] Saba Kawas, George Karalis, Tzu Wen, and Richard E. Ladner. Improving real-time captioning experiences
for deaf and hard of hearing students. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’16, page 15–23, New York, NY, USA, 2016. Association
for Computing Machinery.
[8] Tara Matthews, Scott Carter, Carol Pai, Janette Fong, and Jennifer Mankoff. Scribe4me: Evaluating amobile sound transcription tool for the deaf. In Paul Dourish and Adrian Friday, editors, UbiComp 2006:
Ubiquitous Computing, pages 159–176, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
[9] Samuel White. Audiowiz: Nearly real-time audio transcriptions. In Proceedings of the 12th International
ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’10, page 307–308, New York,NY, USA, 2010. Association for Computing Machinery.
[10] Dhruv Jain, Rachel Franz, Leah Findlater, Jackson Cannon, Raja Kushalnagar, and Jon Froehlich. Towards
accessible conversations in a mobile context for people who are deaf and hard of hearing. In Proceedings ofthe 20th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’18, page
81–92, New York, NY, USA, 2018. Association for Computing Machinery.
[11] Dhruv Jain, Bonnie Chinh, Leah Findlater, Raja Kushalnagar, and Jon Froehlich. Exploring augmented
reality approaches to real-time captioning: A preliminary autoethnographic study. In Proceedings of the2018 ACM Conference Companion Publication on Designing Interactive Systems, DIS ’18 Companion, page
7–11, New York, NY, USA, 2018. Association for Computing Machinery.
[12] Yi-Hao Peng, Ming-Wei Hsi, Paul Taele, Ting-Yu Lin, Po-En Lai, Leon Hsu, Tzu-chuan Chen, Te-Yen Wu,
Yu-An Chen, Hsien-Hui Tang, and Mike Y. Chen. Speechbubbles: Enhancing captioning experiences fordeaf and hard-of-hearing people in group conversations. In Proceedings of the 2018 CHI Conference on
Human Factors in Computing Systems, CHI ’18, page 1–10, New York, NY, USA, 2018. Association for
Computing Machinery.
[13] Alex Olwal, Kevin Balke, Dmitrii Votintcev, Thad Starner, Paula Conn, Bonnie Chinh, and Benoit Corda.
Wearable subtitles: Augmenting spoken communication with lightweight eyewear for all-day captioning.
In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, UIST ’20,page 1108–1120, New York, NY, USA, 2020. Association for Computing Machinery.
[14] Leah Findlater, Bonnie Chinh, Dhruv Jain, Jon Froehlich, Raja Kushalnagar, and Angela Carey Lin. Deafand hard-of-hearing individuals’ preferences for wearable and mobile sound awareness technologies. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, page 1–13,New York, NY, USA, 2019. Association for Computing Machinery.
[15] Larwan Berke, Khaled Albusays, Matthew Seita, and Matt Huenerfauth. Preferred appearance of captions
generated by automatic speech recognition for deaf and hard-of-hearing viewers. In Extended Abstracts ofthe 2019 CHI Conference on Human Factors in Computing Systems, CHI EA ’19, page 1–6, New York,NY, USA, 2019. Association for Computing Machinery.
[16] Larwan Berke, Christopher Caulfield, and Matt Huenerfauth. Deaf and hard-of-hearing perspectives on
imperfect automatic speech recognition for captioning one-on-one meetings. In Proceedings of the 19thInternational ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’17, page 155–164,
New York, NY, USA, 2017. Association for Computing Machinery.

5