Visualization: The Missing Factor in Simultaneous Speech Translation
Authors: Sara Papi, Matteo Negri, Marco Turchi (Fondazione Bruno Kessler, Università di Trento, 2021)
Topic: Multimodal

Abstract
Simultaneous speech translation (SimulST) is the task in which output generation has to be performed on partial, incremental speech input.In recent years, SimulST has become popular due to the spread of multilingual application scenarios, like international live conferences and streaming lectures, in which on-the-fly speech translation can facilitate users' access to audio-visual content. In this paper, we analyze the characteristics of the SimulST systems developed so far,discussing their strengths and weaknesses.We then concentrate on the evaluation framework required to properly assess systems' effectiveness.To this end, we raise the need for a broader performance analysis, also including the user experience standpoint. We argue that SimulST systems, indeed, shouldbe evaluated not only in terms of quality/latency measures, but also via task-oriented metrics accounting, for instance, for the visualization strategy adopted.In light of this, we highlight which are the goals achieved by the communityand what is still missing.

Introduction
Simultaneous speech translation (SimulST) is the task in which the translation of a source language speech has to be performed on partial, incremental input. This is a key feature to achieve low latency in scenarios like streaming conferences and lectures, where the text has to be displayed following as much as possible the pace of the speech. SimulST is indeed a complex task in which the difficulties of performing speech recognition from partial inputs are exacerbated by the problem to project meaning across languages. Despite the increasing demand for such a system, the problem is still far from being solved. So far, research efforts mainly focused on the quality/latency trade-off, i.e. producing high quality outputs in the shortest possible time, balancing the need for a good translation with the necessity of a rapid text generation. Previous studies, however, disregard how the translation is displayed and, consequently, how it is actually perceived by the end users. After a concise survey of the state of the art in the field, in this paper we posit that, from the users' experience standpoint, output visualization is at least as important as having a good translation in a short time. This raises the need for a broader, task-oriented and human-centered analysis of SimulST systems' performance, also accounting for this third crucial factor.

Conclusion
SimulST systems have become increasingly popular in recent years and many efforts have been madeto build robust and efficient models.Despite the difficulties introduced by the online framework, these models have rapidly improved, achieving comparable results to the offline systems.However, many research directions have not been explored enough (e.g., the adoption of dynamic or fixed segmentation, the offline or the online training).First among all, the visualization strategy that is adopted to display the output of the simultaneous systems is an important and largely under-analyzed aspect of the simultaneous experience. We posit that the presence of application-oriented metrics (e.g., reading speed), which are strongly related to the visualization and, as an ultimate goal, to the user experience, is the factor that misses in the actual evaluation environment.Indeed, thispaper points out that BLEU and Average Lagging are not the only metrics that matter to effectively evaluate a SimulST model, even if they are fundamental to judge a correct and real-timed translation. We hope that this will inspire the community to work on this critical aspect in the future.