Building a Spoken Dialogue System for Supporting Blind People in Accessing Mathematical Expressions
Authors: Pier Felice Balestrucci, Alessandro Mazzei, Luca Anselma, Cristian Bernareggi (Università di Torino, 2023)
Topic: Chatbots and Dialogue Systems

Abstract
Mathematical expressions are complex hierarchical structures of symbols that are usually accessed by visual inspection. These expressions are seldom rendered with natural language since users are not usually required to read them aloud. People with a visual impairment generally use LATEX with screen readers to acquire mathematical expressions. However, LATEX can be verbose, slow to listen, and difficult to learn. This work proposes a way to make mathematical expressions easier to be accessed by people with disabilities by exploiting their hierarchical structures. We describe and evaluate a dialogue system to vocally navigate mathematical expressions in English. In contrast with standard screen readers, the vocal interaction allows people to query the system about sub-parts of the expressions.

Introduction
In the last few years, improvements in speech technologies have enabled the use of Natural Language Processing in various application domains. Among these, the field of assistive technologies stands out as one with the most potential. Specifically, spoken dialogue systems (SDSs) can enhance the quality of life for numerous users with special needs. In this paper we study the task of accessing mathematics for blind people. Traditionally, people with a visual impairment, in order to acquire mathematics, use LATEX with screen readers, that are dedicated text-to-speech software. However, LATEX is verbose, slow to listen, and not well known in young people. Indeed, LATEX is designed as a typographical language, that is a technical language to provide typographical details rather than a compact description of a mathematical content. Moreover, mathematical expressions can range over multiple dimensions (e.g. fractions are arranged on two levels) and the linear encoding of LATEX can produce a very long sequence with distant relations among symbols closely connected. To tackle these issues, in this paper we propose a SDS based on the main idea to use standard natural language to interact with mathematical contents. For instance by using a SDS people with a visual impairment do not need to use a keyboard and a mouse to use their device. They can activate vocally the SDS to interact directly. Furthermore, this approach can also address other disabilities, including motor impairments. The SDS uses mathematical sentences, that are natural language sentences containing the semantics of a mathematical expression [1, 2]. There are many advantages in using mathematical sentences with respect to LATEX. First, mathematical sentences are shorter. Second, they are not semantically ambiguous as some LATEX expressions (e.g. f(x) could also stand for f ·(x)). Third, they do not require knowledge of LATEX, so that they can be used by a wider class of users (e.g. children). In particular, the SDS converts the LATEX encoding of the mathematical expression in a semantic language, called Content Math Markup Language (CMML)1. From CMML, English mathematical sentences are generated according to good practices for spoken mathematics [3], through a standard Natural Language Generation (NLG) architecture, i.e. a sentence planner and a realizer. The interaction design of the SDS is quite straightforward: the SDS pronounces the mathematical sentences and the user can interrupt it to “navigate” the expression (e.g. ask for repetition of its parts). So, the dialogue manager component coordinates the recognition of a repetition command with the generation of (a subpart of) the mathematical sentence. We performed a user-based evaluation on the effectiveness and the usability of the SDS. The system implementation and evaluation were conducted with the involvement of visually impaired experts. The results show that the developed system has a good impact both on the comprehension of mathematical expressions and on the user experience, constituting a promising approach for helping people with visual impairments. The paper is structured as follows: In Section 2, we report related work. In Section 3, we describe the main components of the SDS. In Section 4, we describe a human-based evaluation of the SDS and in Section 5, we closes the paper with some consideration and future work. Finally, Section 6 discusses some limitations of our SDS.

Conclusion
In this paper we described a SDS designed for allowing visually impaired people to access mathematical expressions. In Experiment 1 we focused on the understanding of the mathematical sentences generator for English (i.e. using EM and SPICE measures), replicating the good results obtained for Italian in Mazzei et al. [1, 2]. In Experiment 2, we tested the complete SDS allowing user to ask for repetition. With respect to expressions understanding, the results of this experimentation are less encouraging than Experiment 1, but we speculate that this is a consequence of the complexity of the experimental setting due to the necessity of online interaction. However, the UEQ showed that the users really appreciated the interaction with the SDS. In the future we want to improve the SDS by adding new intents. Moreover, we want to define a new SDS designed for diagrams and other visual structures, creating accurate descriptions and making them navigable.