# Enabling the replicability of speech synthesis perceptual evaluations

Sébastien Le Maguer, Gwénolé Lecorvé, Damien Lolive, Naomi Harte, Juraj Šimko

► To cite this version:

Sébastien Le Maguer, Gwénolé Lecorvé, Damien Lolive, Naomi Harte, Juraj Šimko. Enabling the replicability of speech synthesis perceptual evaluations. Interspeech 2025 - 26th edition of the Interspeech Conference, Aug 2025, Rotterdam, Netherlands. pp.2545-2549, 10.21437/Interspeech.2025-401. hal-05477590

HAL Id: hal-05477590

https://inria.hal.science/hal-05477590v1

Submitted on 26 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

# Enabling the replicability of speech synthesis perceptual evaluations

Sébastien Le Maguer¹, Gwénolé Lecorvé², Damien Lolive³, Naomi Harte⁴, Juraj Šimko¹

¹University of Helsinki, Finland

²Orange Research, France

³IRISA, Université Bretagne Sud, France

⁴Sigmedia Group, School of Engineering, Trinity College Dublin, Ireland

sebastien.lemaguer@helsinki.fi

## Abstract

How speech synthesis is evaluated is nowadays questioned. Not only have conventional listening tests as a whole been proven a poor match for modern synthesis, but more fundamentally, important information (e.g., the question asked to the listener) is frequently missing in the report of the outcome of the evaluation despite the impact on the interpretation of the test results. This can lead to uncertainty about the validity of these evaluations. To address this issue, we propose standardising the structure of any evaluation report. To facilitate this standardisation, our contribution is twofold: an open-source subjective evaluation platform; and a set of reporting guidelines. The platform is designed to enable the development of easily shareable evaluation recipes. The set of guidelines complements the platform to support researchers in reporting their evaluation choices and analysis in more detail while relying on the recipe to describe the actual evaluation process.

**Index Terms:** speech synthesis evaluation, replicability, open science, evaluation platform, evaluation guidelines

## 1. Introduction

Standard speech synthesis evaluation methodologies have shown their limitations. Although recent models are capable of producing synthetic speech with unprecedented human likeness, several studies have demonstrated and emphasised the limits of standard protocols such as Mean Opinion Score (MOS) [1, 2, 3] and Multiple Stimuli with Hidden Reference and Anchors (MUSHRA) [4]. The research community is now well aware of this issue and has been investigating alternative protocols such as [5]. However, a more fundamental problem with evaluation must be addressed.

In 2015, Wester et al. [6] demonstrated that researchers do not report sufficient details about their evaluation. Almost ten years after these conclusions, Kirkland et al. reported that this trend remains predominant in the field [1]. These two studies emphasise that important details are missing from the description of the evaluation protocol. For example, Kirkland et al. [1] noted that researchers omit to provide the precise question asked to the listener. They also demonstrated that the phrasing of this question has a strong impact on the ratings, and therefore show the danger of omitting such critical information. Edlund et al. [7] pushed this reflection further by proving that the framing of an evaluation will have a strong impact on its outcome. In conclusion, these studies pose the fundamental question of the scientific foundations of the reporting of our evaluation outcomes.

In 2019, the US academies of Sciences, Engineering and Medicine were requested to provide a set of recommendations about *reproducibility* and *replicability* in Science overall. The

resulting report [8] defines these two concepts as follows:

**Reproducibility** is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis.

**Replicability** is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.

From these definitions, two key observations emerge:

**Point 1** Reproducibility of a subjective evaluation is not achievable due to the impact of human participants on the evaluation: Human participants can't take a subjective evaluation twice, as they will be influenced by the first run during the second one; and new human participants are different by nature; Hence, the present paper will focus on replicability, not reproducibility;

**Point 2** A less obvious aspect of these definitions is that the evaluation should align with the hypothesis posed behind the evaluated proposition. Overall evaluation may show an improvement of the synthesis due to the methodology, but it is necessary to design an evaluation to ensure that this improvement is indeed the result of addressing the identified problem.

Some propositions have emerged to improve the replicability of the evaluation. The most notable is the TTS arena [9], aiming to provide a unique framework to evaluate the naturalness of synthetic speech under general conditions using a preference test. While this is a notable step forward, we think we need to go beyond this proposition: modern synthetic speech cannot be evaluated in a one-for-all setup, and the target task will have a strong impact on the evaluation outcome [7]. Furthermore, evaluation protocols will evolve, and new protocols will be designed. This evolution will be necessary as the quality of modern synthesisers has now saturated the evaluation capabilities of standard protocols [3], and we are in crucial need to address Point 2. Therefore, we need tools and practices which are flexible enough to preempt this evolution.

This is why, to enable the replicability of speech synthesis perceptual evaluations and following the recommendations of [1, 3, 6], we propose to standardise the structure of any evaluation report into three complementary parts: 1. the evaluation protocol itself, which has to be referenced, presented and justified; 2. the evaluation recipe encompassing the evaluation procedure and the anonymised data of the conducted evaluation; 3. the critical analysis and report of the evaluation outcome. Based on this decomposition, the evaluation protocol and the analysis of its outcome should be extensively presented in the publication, while the recipe can be shared as supplementary material.
The contribution of this paper is to propose a first step toward the implementation of this new standard. Firstly, we present *replikant*<fnref n="1" /> (a refactoring of FlexEval [10]), an open-source platform designed to share recipes for any researcher to replicate an evaluation campaign. Secondly, we propose a set of high-level guidelines for conference reviewers so they can assess the validity of the evaluation report.

## 2. Why replikant: a survey of existing evaluation tools

To conduct a speech synthesis subjective evaluation campaign, researchers can rely on a diverse set of tools. First, an open-source platform has been developed for the Blizzard Challenge [11]. Although being open and very stable, the script constituents of this platform are difficult to extend and deploy. A second solution is WebMUSHRA [12]. In contrast to the scripts of the Blizzard challenge, WebMUSHRA is a fully integrated platform. The maintainers of WebMUSHRA also provide full support for the standard protocols (MOS, MUSHRA, AB(X)-test). Despite its qualities, introducing a new test is cumbersome as it requires developers to deeply modify the server code in addition to designing the user interface. Easy-to-use alternatives are survey-based tools such as SurveyMonkey [13], Alchemer [14] (previously known as SurveyGizmo) or Gorilla [15]. These solutions face several drawbacks. Not only are they generally proprietary and require a subscription, they also do not provide an easy way to share a dedicated evaluation. The main exception is Gorilla. Gorilla supports an easy design and sharing process of conducted evaluation, but it still requires a substantial fee. Finally, researchers can also consider the use of annotation tools. This can be justified as the current set of evaluation methodologies consists of annotating samples with a score (for MOS or MUSHRA) or implicitly with a flag (preferred / not preferred for the AB-test). The most complete representative of the annotation tools is Label Studio [16]. However, this solution is proprietary and designed as a fully integrated annotation solution. In other words, adapting Label Studio for the purpose of subjective evaluation is a complex process. Finally, the closest solution to our needs is the open source implementation of the P.808 [17], but it is not designed to support easily shareable recipes. To summarize, researchers do not have an adequate tool to ensure an easy way to properly share their evaluation resources.

## 3. Overview of replikant

As part of their discussion, Kirkland et al. [1] suggested “to maintain shared tools and platforms for TTS evaluation”. In this section, we describe our contribution to address this suggestion as the goal of *replikant* is to enable researchers to share evaluation recipes in order to ensure the replicability of subjective evaluations. To present this platform, we consider the points of view of the three main actors: the designer, the maintainer and the user. Our aim is that this presentation serves as an entry point for the platform. First, from the designer's perspective, compromises had to be made and have conditioned some development choices. These choices are presented in Section 3.1. As a maintainer, it is necessary to understand the core architecture of a platform. Section 3.2 is dedicated to the presentation of an overview of the core architecture of *replikant*. While more technical, we consider

this section to be necessary as we hope that the community will help support *replikant*. in Section 3.2, we present an overview of the core architecture of *replikant*. Finally, a researcher who wants to use the platform will mostly be interested in Section 3.3, which describes the composition of an evaluation recipe.

### 3.1. Development choices

In order to develop *replikant*, we followed some design principles resulting from realistic choices and compromises.

The first and foremost principle is *openness*. The goal of *replikant* is to facilitate the design and implementation of a subjective evaluation recipe easily shareable with the community. Promoting open-science, shareable recipes have been popularised in speech technology by Kaldi [18] and are now common in several standard toolkits [19, 20]. The goal of the platform is to provide all the underlying mechanics to run such a recipe.

The second principle is *stability*. As we aim to develop a standard evaluation platform, we need to ensure its longevity and maintainability. To facilitate this process, we rely on well-maintained standard tools and libraries such as Python, SQLite, Flask and Jinja2. This allows us to reduce the workforce required to maintain the core of the platform. For the same reasons, and in contrast to more complex tools such as Gorilla or Label Studio, we do not provide a Graphical User Interface (GUI) frontend to design a test. While it is possible to develop one, providing a GUI would require an important development and maintenance effort on our side. This would jeopardise the maintenance of the tool and go against the stability principle. As a result, a minimal set of coding skills is required to implement an evaluation campaign. These coding skills mainly involve standard Web technologies (HTML, CSS and JavaScript) as well as beginner-level skills in Python and Jinja2. We aim to keep this minimal by providing a set of standard recipes already used in published evaluation campaigns.

Finally, the last principle is *flexibility*. This is achieved by ensuring that *replikant* is modular on two fronts. First, the core of the platform is designed around *activities*. Let's consider an evaluation campaign such as the ones conducted for the Blizzard Challenge [11]. The evaluation is composed of different sections (collecting participant information, similarity test, naturalness test, ...) An activity is the software equivalent of the type of operation required to run one section of the evaluation. The current implementation already provides a set of activities (e.g., authentication compatible with the online platform Professional [21]). This set can easily be extended by any developers by ensuring that a new type of activity respects some conventions. The second front is the structure of an evaluation recipe. Mainly, a recipe is organised into steps (e.g., registration of user information, explanations, listening to a sample, etc.), and each step is associated with an HTML template. By heavily relying on the possibilities offered by Jinja2, a template can be freely decomposed into fundamental elements, such as different types of audio and video players, different sets of scales and instructions or MUSHRA sliders. These elements can easily be reused, adapted or extended. To facilitate backwards compatibility, we purposefully impose a minimal set of conventions about a recipe encoded in the platform. This set will be explained in the next sections.

<footnote n="1">¹https://github.com/seblemaguer/replikant</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: Simplified class diagram presenting the organization of *replikant*. As Python does not provide interfaces, the realization arrow is used to indicate that the "realization class" corresponds to flask implementation package. The classes *Scope*, *ParticipantScope*, *AdminScope* and *Activity* are all abstract classes.</figcaption></figure>

## 3.2. Software architecture

Figure 1 presents a simplified class diagram summarizing the architecture of *replikant*.

The core of *replikant* is centred around the campaign instance (class `Campaign`). The campaign instance acts as the coordinator. It is in charge of preparing and executing an evaluation. To do so, it relies on the recipe's main configuration (instantiated using class `Config`) to instantiate the required activities (`Activity`), create and then start the evaluation.

A fundamental concept of *replikant* is the scope (*Scope*). From a software perspective, its purpose is to instantiate the templates to obtain the User Interface (UI), prepare the assets (e.g., audio, images or videos) and ensure that the authentication is validated (if required). We currently distinguish two scopes: the *AdminScope* is the base for all the administration interactions (such as the export of the database *ExportDB*), the *ParticipantScope* for all the interactions of the standard participants during the evaluation. Each *Activity* is associated with a *ParticipantScope*. The campaign instance uses the configuration of the recipe to instantiate the required activities. It then creates the execution graph of activities (instance of *ActivityGraph*). This execution graph corresponds to the evaluation as presented to the participant.

In the current implementation, we distinguish four types of activities: a simple page (class `Page`) without any information to be collected, a form (class `Form`) which corresponds to fixed information to be collected, an authentication page (`Auth`), and an actual evaluation task (class `Task`). The implementation of an activity follows the Model-View-Controller (MVC) pattern. It relies on the combination `Jinja2` templates and an instance of `ParticipantScope` for the view and on an instance of a `Activity` acting as the controller. Furthermore, we implemented the controller and the model to support dynamic data creation. By doing so, and relying on the functionalities provided by `Jinja2`, we allow the user to control the form and the simple page by only implementing the templates in the recipe.

Finally, the class `Task` is the software implementation of an evaluation step. This implementation is as generic as possible. This is necessary to not only facilitate the integration of new protocols, but also to allow the use of multiple protocols in one evaluation. Consequently, we impose a generic structure to store the data collected during an evaluation: the timestamp, the index of the step in the test, the sample identifier, the nature of the information to be stored and its value. This structure

also enables more dynamic data to be collected (e.g., number of clicks in the case of a behavioural evaluation). We also dedicate to the platform to manage the selection, randomisation, and filename obfuscation of the samples. Combining these two constraints with the use of `Jinja2` template to design the UI and to define the information to be stored, we do not require the user to modify the platform to introduce a new protocol.

## 3.3. The shareable evaluation recipe

Having presented the architecture of *replikant*, we can now focus on the reason for proposing this platform: the recipe to be shared. A recipe has to be self-contained and therefore composed of three parts: the evaluated material (what is evaluated), the UI and the configuration (how the evaluation is conducted), and the database (the raw outcome of the evaluation campaign).

The structure of a recipe is partially imposed as it should contain the following files and directories:

**config.yaml** the entry-point of a recipe (it is the input configuration parsed by `Config` to feed the campaign instance). It is composed of three sections: the *overall section* defining the global variables as well as the entry point of the evaluation, the *administration configuration* including the password required to download the data of the evaluation, the *activity graph* describing the different phases of the evaluation. As noted previously, each phase can either be a simple page shown to the participant, a form for the participant to fill out, or an evaluation task. For the configuration of an evaluation task, it is also necessary to reference which systems will be evaluated;

**systems** the directory containing all the evaluated material. This is composed of two parts: a set of CSV files (one per system) referencing the material to be evaluated and the actual material (i.e., text, audio, video or images);

**templates** the directory containing the user templates. This is the place where the designer of the test can introduce new templates (referenced in the root configuration) or override a default template. `Jinja2` facilitates the implementation of this functionality: to override a template, the designer of the test simply needs to create a template in this directory using the exact same name as the original;

**assets** the directory containing the assets aside from the evaluated material. This includes logos, material used as reference (e.g. for similarity tests) or any files which need to be used as is.
Finally, the database which maintains the current state of a campaign once it has been started (e.g., progress of participant, number of evaluations for each sample, each system) is handled by the platform. This includes its creation, which is based on the provided recipe configuration. However, when the campaign is over, it is the responsibility of the designer of the test to include the database as part of the recipe to ensure the replicability of the evaluation campaign.

Recipes for the speech synthesis evaluation standard protocols are already implemented (MOS, MUSHRA, AB-test, Rapid Prosody Transcription (RPT) [22, 23]). Recipes for evaluation purposes beyond speech synthesis, such as video summarisation, are also implemented. These recipes are shared as part of the platform, and some have already been presented in published work [3, 24, 25].

# 4. Reporting guidelines

As we have seen in the introduction, the primary purpose of an evaluation is to prove/disprove a hypothesis. To assess the validity of the evaluation, we need to ensure that it is fully replicable. replikant aims to enable researcher to improve the replicability of their evaluation. However, even without using this platform, researchers still have to share their evaluations and document their choices. To facilitate this documentation, we propose a set of guidelines primarily addressed to the conference reviewers. Researchers should consider these guidelines as a checklist to comply with. We keep these guidelines fairly high-level as we do not aim to burden the reviewers. These guidelines are not meant to be a rigid canvas. The final say remains with the reviewers, and they should feel free to deviate from these guidelines when they fail to capture the peculiarities of a presented evaluation. Due to space constraints, we will not develop the reasoning behind each item but this list is compiled based on the requirements from standard recommendations such as the P.800 [26] and the P.85 [27] as well as meta-analyses such as Kirkland et al. [1] and Wester et al. [6]. We refer the readers to these publications for more details.

## 4.1. The evaluation campaign

The foundation of an evaluation campaign is the protocol(s) it relies on. Researchers should carefully refer to the publication describing the chosen protocol, justifying why this protocol has been chosen and ensure the readers understand its implications. To ensure this, reviewers should assess the following items:

☐ Is the reference of the protocol provided? or, if it is a new protocol, is it fully described?

☐ If an existing protocol, is it correctly implemented?

☐ Is the choice of the protocol justified in relation to the contribution of the submission?

☐ Are the modifications or the design of the protocol justified?

☐ Are the limitations of the protocol in relation to the contribution of the submission analysed?

Assuming the protocol is properly described and justified, the second main source of bias is the material used. To assess the material, the reviewer should assess the following items:

☐ Is the set of compared systems open-sourced? If not, is there enough description to replicate the implementation?

☐ Are the selected systems relevant for the comparison?

☐ Is the dataset used for the evaluation described and justified in relation to the goal of the evaluation?

☐ Are the potential biases of the dataset analysed?

As we are dealing with human evaluation, it is also critical to describe and analyse the selected cohort of participants. While using many listeners can lead to statistically significant results, it is also critical that researchers analyse how a cohort of participants may approach the test. We propose that the reviewer assesses the following items:

☐ Is the number of participants appropriate for the protocol?

☐ Is the description of participants sufficiently detailed for the goal of the analysis?

☐ Are the listeners qualified to evaluate the tested hypothesis (e.g., are they native or non-native)?

☐ Is the cohort of listeners well-balanced or is it justified?

## 4.2. The analysis of the outcome

Having tackled the description of a recipe and the description of the evaluation design, the last point to address is the analysis of the results of the evaluation. To do so, we propose that the reviewers assess the following items in order to ensure that researchers do not overclaim the impact of their proposition and do not miss any potential bias:

☐ Do the conclusions address the evaluated hypothesis?

☐ Is the statistical test adequate?

☐ Do the conclusions consider the limitations of the protocol?

☐ Is there any potential unexpected effect besides the main target captured by the protocol?

# 5. Conclusion

In this paper, we have proposed a standardised structure of an evaluation report in order to improve the replicability of speech synthesis evaluations. This structure decomposes the report into two complementary elements: the actual report and the shared recipe containing all data required to replicate the evaluation as supplementary material. To promote the development of such recipes, we designed and presented a dedicated open-source platform. The guidelines are independent of the platform, as researchers need to produce replicable subjective evaluations independently of the tools used during the campaign. This is a starting point. While we will ensure the development of the platform is compatible with future standards, we, as a scientific community, must also ensure that these standards are adequately respected and constantly questioned. In other words, the presented guidelines will have to be continuously adapted to meet the future requirements of research in speech synthesis.
# 6. Acknowledgements

The authors would like to thank the members of the team EXPRESSION who impulsed the development of FlexEval. The authors also would like to thank E. Cooper, C. Valentini-Botinao, O. Watts, S. King, P. Oplustil-Gallegos, D. Wells for the discussions which fueled the development of the platform and the methodologies. S. Le Maguer is funded by the Academy of Finland project #357262 - Predictive Processing Approach to Modelling Prosodic Hierarchy for Speech Synthesis. This work was also partially conducted with the financial support of Irish Research Council (IRC) under Grant Agreement No. 208222/15425 and at the ADAPT SFI under Grant No. 13/RC/2106_P2.

# 7. References

[1] A. Kirkland, S. Mehta, H. Lameris, G. E. Henter, E. Szekely, and J. Gustafson, "Stuck in the mos pit: A critical analysis of mos test methodology in tts evaluation," in *12th ISCA Speech Synthesis Workshop (SSW2023)*, 2023, pp. 41–47. doi: 10.21437/SSW.2023-7

[2] E. Cooper, W.-C. Huang, Y. Tsao, H.-M. Wang, T. Toda, and J. Yagagishi, "A review on subjective and objective evaluation of synthetic speech," *Acoustical Science and Technology*, p. e24.12, Apr. 2024. doi: 10.1250/ast.e24.12

[3] S. Le Maguer, S. King, and N. Harte, "The limits of the Mean Opinion Score for speech synthesis evaluation," *Computer Speech & Language*, vol. 84, p. 101577, Mar. 2024. doi: 10.1016/j.csl.2023.101577

[4] S. Zieliński, P. Hardisty, C. Hummersone, and F. Rumsey, "Potential biases in mushra listening tests," in *Audio Engineering Society Convention 123*. Audio Engineering Society, 2007.

[5] D. Wells, A. L. Aldana Blanco, C. Valentini, E. Cooper, A. Pine, J. Yamagishi, and K. Richmond, "Experimental evaluation of mos, ab and bws listening test designs," in *Interspeech 2024*, 2024. ISSN 2958-1796 pp. 2695–2699. doi: 10.21437/Interspeech.2024-1778

[6] M. Wester, C. Valentini-Botinhao, and G. E. Henter, "Are we using enough listeners? no! — an empirically-supported critique of interspeech 2014 tts evaluations," in *Interspeech 2015*, 2015. ISSN 2958-1796 pp. 3476–3480. doi: 10.21437/Interspeech.2015-689

[7] J. Edlund, C. Tännander, S. Le Maguer, and P. Wagner, "Assessing the impact of contextual framing on subjective tts quality," in *Interspeech 2024*, 2024. ISSN 2958-1796 pp. 1205–1209. doi: 10.21437/Interspeech.2024-781

[8] National Academies of Sciences, Engineering, and Medicine, Policy and Global Affairs, Committee on Science, Engineering, Medicine, and Public Policy, Board on Research Data and Information, Division on Engineering and Physical Sciences, Committee on Applied and Theoretical Statistics, Board on Mathematical Sciences and Analytics, Division on Earth and Life Studies, Nuclear and Radiation Studies Board, and Division of Behavioral and Social Sciences and Education, *Reproducibility and Replicability in Science*. Washington, D.C., DC: National Academies Press, Oct. 2019.

[9] V. Srivastav, C. Fourrier, L. Pouget, Y. Lacombe, main, and S. Gandhi, "Text to speech arena," https://huggingface.co/spaces/TTS-AGI/TTS-Arena, 2024.

[10] C. Fayet, A. Blond, G. Coulombel, C. Simon, D. Lolive, G. Lecorvé, J. Chevelu, and S. Le Maguer, "Flexeval, création de sites web légers pour des campagnes de tests perceptifs multimédias," in *6e conférence conjointe Journées d'Etudes sur la Parole (JEP, 33e édition)*. ATALA; AFCP, 2020, pp. 22–25.

[11] S. King and V. Karaiskos, "The blizzard challenge 2012," in *The Blizzard Challenge 2012*, 2012, pp. 1–11. doi: 10.21437/Blizzard.2012-1

[12] M. Schoeffler, S. Bartoschek, F.-R. Stöter, M. Roess, S. Westphal, B. Edler, and J. Herre, "webmushra — a comprehensive framework for web-based listening tests," *Journal of Open Research Software*, vol. 6, no. 1, p. 8, Feb. 2018. doi: 10.5334/jors.187

[13] "SurveyMonkey: The World's Most Popular Survey Platform," Dec. 2024, [Online; accessed 2. Dec. 2024]. [Online]. Available: https://www.surveymonkey.com

[14] "Alchemer: Enterprise Online Survey Software & Tools," Nov. 2024, [Online; accessed 2. Dec. 2024]. [Online]. Available: https://www.alchemer.com

[15] A. L. Anwyl-Irvine, J. Massonnié, A. Flitton, N. Kirkham, and J. K. Evershed, "Gorilla in our midst: An online behavioral experiment builder," *Behav. Res. Methods*, vol. 52, no. 1, pp. 388–407, Feb. 2020. doi: 10.3758/s13428-019-01237-x

[16] M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov, "Label Studio: Data labeling software," 2020-2024, [Online; accessed 2. Dec. 2024]. [Online]. Available: https://github.com/HumanSignal/label-studio

[17] B. Naderi and R. Cutler, "An open source implementation of itu-t recommendation p.808 with validation," in *Interspeech 2020*, 2020. ISSN 2958-1796 pp. 2862–2866. doi: 10.21437/Interspeech.2020-2665

[18] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz et al., "The kaldi speech recognition toolkit," in *IEEE 2011 workshop on automatic speech recognition and understanding*. IEEE Signal Processing Society, 2011.

[19] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, "ESPnet: End-to-end speech processing toolkit," in *Proceedings of Interspeech*, 2018, pp. 2207–2211. doi: 10.21437/Interspeech.2018-1456

[20] M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Subakan, P. Plantinga, Y. Wang, P. Mousavi, L. D. Libera, A. Ploujnikov, F. Paissan, D. Borra, S. Zaiem, Z. Zhao, S. Zhang, G. Karakasidis, S.-L. Yeh, P. Champion, A. Rouhe, R. Braun, F. Mai, J. Zuluaga-Gomez, S. M. Mousavi, A. Nautsch, H. Nguyen, X. Liu, S. Sagar, J. Duret, S. Mdhaffar, G. Laperrière, M. Rouvier, R. D. Mori, and Y. Estève, "Open-source conversational ai with speechbrain 1.0," *Journal of Machine Learning Research*, vol. 25, no. 333, 2024. [Online]. Available: http://jmlr.org/papers/v25/24-0991.html

[21] "Prolific | Quickly find research participants you can trust," Dec. 2024, [Online; accessed 2. Dec. 2024]. [Online]. Available: https://www.prolific.com

[22] E. Gutierrez, P. Oplustil-Gallegos, and C. Lai, "Location, Location: Enhancing the Evaluation of Text-to-Speech synthesis using the Rapid Prosody Transcription Paradigm," in *Proc. 11th ISCA Speech Synthesis Workshop (SSW 11)*, 2021, pp. 25–30. doi: 10.21437/SSW.2021-5

[23] J. Cole and S. Shattuck-Hufnagel, "New methods for prosodic transcription: Capturing variability as a source of information," *Laboratory Phonology*, vol. 7, no. 1, p. 8, Jun. 2016. doi: 10.5334/labphon.29

[24] K. Hiovain-Asikainen, A. Suni, and S. Le Maguer, "Neural text-to-speech for north sámi: Development and evaluation," in *Saami Linguistics Symposium (SAALS 6)*. Helsinki, Finland: University of Helsinki, Nov. 2024.

[25] S. Kotey, "Multimodal Summarization in Natural Language Conversations," Ph.D. dissertation, Trinity College Dublin. School of Computer Science & Statistics. Discipline of Computer Science, Dublin, Ireland, 2025. [Online]. Available: https://www.tara.tcd.ie/handle/2262/111819?show=full

[26] ITU, "Methods for subjective determination of transmission quality," International Telecommunication Union (ITU), Geneva, ITU-T Recommendation P.800, 1996.

[27] ITU-T, "A method for subjective performance assessment of the quality of speech voice output devices," International Telecommunication Union (ITU-R), Tech. Rep. P.85, 1994.