# Evaluating Speech Synthesis in a Nonstandardized, Multidialectal Context: A Teochew Case Study

Agathe Wallet, Ilaine Wang, Emmett Strickland, Pierre Magistry

► To cite this version:

Agathe Wallet, Ilaine Wang, Emmett Strickland, Pierre Magistry. Evaluating Speech Synthesis in a Nonstandardized, Multidialectal Context: A Teochew Case Study. Speech Synthesis Workshop, Aug 2025, Leeuwarden, Netherlands. hal-05460279

HAL Id: hal-05460279

https://hal.science/hal-05460279v1

Submitted on 15 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Evaluating Speech Synthesis in a Nonstandardized, Multidialectal Context: A
Teochew Case Study

*Agathe Wallet*<fnref n="1" />, *Ilaine Wang*<fnref n="1" />, *Emmett Strickland*<fnref n="1" />,², *Pierre Magistry*<fnref n="1" />

¹ERTIM, Inalco, France

²MoDyCo, Paris Nanterre University, France

*{agatheima wallet, ilaine wang, pierre magistry}@inalco.fr,*
*emmett.strickland@parisnanterre.fr*

Abstract

In this paper, we implement an experiment to determine
how the segmental content of synthesized utterances impact hu-
man and ASR evaluations of synthesized speech. Our exper-
iment involves a VITS-based system for Teochew, an unstan-
dardized Sinitic language with several varieties spoken within
the Chinese diaspora. Heritage language speakers of the lan-
guages are asked to evaluate the acceptability of synthesized
words for an online dictionary. Human evaluations are com-
pared against rankings produced by an ASR model in order to
determine whether human and machine evaluations align, and
the extent to which language variety impacts perceived accept-
ability. The best stimuli as determined by each method is also
transcribed in IPA using an automatic transcription system. This
allows us to compare the stimuli based on articulatory features
and compute a Feature Error Rate.

**Index Terms:** speech synthesis, TTS, evaluation, low-resource languages

1. Introduction

Traditional methods of evaluating text-to-speech platforms of-
ten neglect to provide useful metrics for measuring performance
on specific linguistic features like phoneme categories. Another
factor rarely considered in evaluation methodologies is socio-
dialectal background: both that of the speaker(s) in the training
data, and of the evaluators responsible for rating the synthesized
speech. Overlooking this factor can be understandable in the
context of most TTS systems, which usually involve languages
with a widely recognized standard or prestige variety, almost
invariably the one used in the training data.

But as TTS technologies expand to lower-resource lan-
guages, evaluators will need to consider an entirely different
sociolinguistic reality for many of the world's language com-
munities. Minority languages often lack a strong written tradi-
tion or institutional support, existing instead on a dialect contin-
uum with no proverbial center. Some such languages are spread
across national boundaries and thus influenced by different ad-
ministrative languages on each side of the border.

In this article, we attempt to resolve these questions through
the lens of one such language, Teochew. Teochew is a Sinitic
language which emerged in Southeastern China, though suc-
cessive migrations beginning in the 18th century also estab-
lished large communities across Southeast Asia, particularly in
Thailand and Cambodia. In both countries, Teochew became
the most widely spoken language among ethnic Chinese, while
smaller but still significant populations were also established
in Vietnam, Indonesia, Malaysia, Laos, and the Philippines.
From there, a more recent migratory wave was precipitated by
that region's geopolitically tumultuous 1960s and 70s, which

saw a dispersal of Teochew speakers towards the United States,
France, Australia, and other highly developed economies. This
exodus brought with it distinct varieties of Teochew, steeped for
centuries in different lands and languages, yet which now coex-
ist in the Chinatowns of Paris, San Francisco, and other global
metropolitan centers. In this paper, we devise a novel experi-
mental methodology to evaluate synthesized utterances using a
Teochew speech synthesis system originally presented by [1].
All code used for our evaluation process, including word lists,
are available online².

2. Methodology

Our experimental methodology involves a multi-step pipeline
involving both human annotators and several automatic evalu-
ation tools. In this section, we describe step-by-step each tool
and how it fits within the wider experimental paradigm. A cor-
nerstone of our approach is synthesizing a pool of acoustically
different utterances from the same set of texts. From this pool,
a single "best" version of each synthesized utterance is selected
from automatic ASR methods on the one hand, and human eval-
uators on the other. We then analyze the articulatory differences
between the human-preferred and ASR-preferred utterances to
better understand the ways in which these differ. Each step in
our pipeline will be described in the following subsections.

2.1. The Teochew TTS system

The evaluation methods described in this paper are applied to
the Teochew speech synthesis system introduced by [1].

This TTS platform is based on the Coqui TTS implementa-
tion of the Variational Inference Text-to-Speech (VITS) archi-
tecture [2, 3]. Like other VITS-based systems, this stochastic
model relies on probabilistic sampling from a latent space dur-
ing inference, meaning that synthesizing the same input multi-
ple times can result in several acoustically distinct outputs. The
system was trained on individual lexical items recorded for a
mobile dictionary application³, and is intended to synthesize in-
dividual words and expressions rather than lengthy sentences.
As input, the system uses texts written in the Peng'im romaniza-
tion scheme, with tones specified using numbers after each syl-
lable. The output is the best sample selected by an ASR model,
which was not evaluated against human judgments. One of the
objectives of the present study is to provide some insights on
this matter.

<footnote n="1">¹Code available at https://gitlab.com/dilsi/
roboty-podman
²https://gitlab.com/dilsi/ssw2025
³Training data available at https://zenodo.org/records/
11085624.</footnote>
## 2.2. Synthesizing our dataset

The first step in our pipeline involves synthesizing a set of 40 Teochew lexical items ranging from one to four syllables in length. Care was taken to train the model with none of these 40 items in the training data, but using the parameters as in [1]. The combined set of items covered the Teochew's full phonemic inventory. We re-synthesized each of the 40 lexical items to produce three variants of each, resulting in a total of 120 samples. The leftover recordings of these items would later be used as ground truth files in future evaluations of the synthesized speech.

We also trained a Wav2Vec 2.0 CTC ASR model on the same dataset as the TTS model. It functions as an end-to-end recognition model to produce Peng'im romanizations from the speech signal and allows us to score the samples produced by the TTS system. The score is obtained by comparing the intended Peng'im text (TTS input) with the recognized Peng'im (ASR output). We use the ratio of correctly matched graphemes to the length of the sequences, normalized to produce a score between 0 and 1.<fnref n="4" />

Each sample was passed to this ASR validator, and we rejected any utterances with an accuracy score below 0.6. Utterances were re-synthesized as many times as needed to obtain three variants of acceptable quality. These 120 utterances were then used in a set of human and machine evaluation tasks to select the human-preferred and ASR-preferred variants synthesized from the 40 items.

## 2.3. ASR ranking

For each of the 40 lexical items, we used the accuracy scores assigned by the ASR validator to select the best of the three synthesized recordings. The 40 ASR-preferred files identified during this process were then used in the relevant analyses and comparisons described in later sections.

## 2.4. Human ranking

In addition to identifying an ASR-preferred sample for each item, we also identified a single human-preferred sample. For this, a group of 6 layperson evaluators was recruited from an international online community of second-generation Teochew immigrants. All six participants were heritage speakers living in France (3), the United States (2), or Singapore (1).

The human evaluators took part in a pairwise comparison task in which each sample is seen twice, in two duels against the two other samples of the same lexical item. Each pair of recordings was accompanied by a romanization of the word, and its translation in English or French according to the speaker's place of residence (see the interface for English speakers in ??).

The positions of each sample inside the duels are randomized only once, meaning that these positions are fixed for all participants. However, the duels are presented in a random order different for each participant.

This allows us to infer a ranking using the outcomes of the duels. The full task requires at most 30 minutes from each participant. The task was implemented via a web-based experimental interface created using the annotation software LabelStudio [4].

Considering that some samples are barely distinguishable, inconsistencies in the pairwise classification are inevitable. This is the case when speakers contradict themselves and create a

<figure><img src="image_2.png" /><figcaption>Figure 1: *Example of forced choice task on LabelStudio, where the second option (highlighted in green) was chosen by the user.*</figcaption></figure>

paradoxical ranking (A > B, B > C but C > A). A quick analysis of the resulting rankings does reveal the occasional inconsistency, especially for five lexical items for which at least two speakers have been inconsistent. For four of these lexical items, the ASR validator gave the same score to all three productions. For the fifth, the scores were 1, 1 and 0.9.

## 2.5. Phonetization and metrics

For a fine-grained evaluation of our synthesis model, we need to determine which sounds were produced in comparison to those that were expected. The expected sounds can be identified either by using a canonical IPA phonetization of the Peng'im text, or by using an IPA transcription of a native speaker's pronunciation of the utterance in question. To achieve this, we used a phonetic model for the canonical transcription and an automatic transcription model for the human and synthesized samples.

For each of the 40 lexical items used as input for the TTS model, we used a FOMA transducer [5] to convert each Peng'im romanization into an IPA phonetization corresponding to the language variety used to train the TTS model.

Separately, the speech transcription model MultIPA [6] was applied to both the human-preferred and ASR-preferred sample of each utterance. MultIPA was also applied to the 40 ground-truth recordings. For each of the 40 utterances, we calculated error rates between the two categories of preferred recordings, and the IPA phonetization of the corresponding ground truth recording. Our methods involve two separate error metrics provided by the Panphon library [7]. The phone error rate (PER) takes into account the differences between two phonetizations at the phone level. However, it does not take into account the various articulatory features of each phone, such as voicing, place, or manner of articulation. As an illustration, the phonetizations [pan] and [gak] would both have a PER of 66.66% when calculated against the ground truth [bam], even though the first phonetization is featurally and perceptually closer to the ground truth than the second.

To account for potential distortions of this type, we utilized Panphon's feature error rate (FER), which calculates the error rate based on 24 articulatory features. Therefore, when comparing the phonetizations [pan] and [gak] with the ground truth [bam], the resulting FER would be 4.86% for the first and 15.28% for the second. This reflects the greater similarity of the first item to the ground truth.

For a more detailed error analysis, we also compared each articulatory feature, using bags of features, between the different datasets:

<footnote n="4">⁴We use the implementation of such ratio from the DIFFLIB module from Python3's standard library</footnote>
* Theory: the 40 IPA phonetizations generated with FOMA from the Peng'im transcriptions

* Teochew Speaker: the ground-truth audio files of these 40 utterances, transcribed with MultIPA

* Synthesis: two sets of 40 synthesized samples, those preferred by the ASR and those preferred by the Teochew speakers

We then employed the Student's t-test to assess whether the presence, absence, over-representation or under-representation of a specific feature in a dataset is statistically significant. To maximize the information available for comparison, we separated consonants, vowels and semivowels into distinct categories. However, tones were not considered, as MultIPA is not trained on tone languages and thus cannot transcribe them. Given the model's good performance in representing tones in [1] and the relative complexity of automatically evaluating this aspect of synthesized speech, we chose to limit this study to segmental aspects of the model output.

# 3. Results

## 3.1. Distribution Analysis of the ASR Evaluations

The results of the Student's t-tests we employed are presented in three separate tables, one for each category: vowels, consonants and semi-vowels.

For each, three comparisons were made: the transcriptions of the Teochew Speaker's recordings (Teochew Speaker) against the phonetizations of the canonical Peng'im transcriptions (Theory), the transcriptions of the synthesized samples (Synthesis) against the Theory, and finally the Synthesis against the Teochew Speaker. A significant positive value shows an overrepresentation of the feature in the gold standard (designated to the left of each dash in the table header), while a significant negative value shows an overrepresentation of the feature in the prediction (to the right).

The evaluated synthesized files are those preferred by the ASR model among the three options presented to the speakers for each lexical item. The significant differences are highlighted.

### 3.1.1. Vowels

<table><caption>Table 1: t-statistics for articulatory features of vowels. Based on ASR-preferred samples.</caption><thead><tr><th rowspan="2">Features</th><th colspan="3">Teochew Speaker</th><th colspan="3">Synthesis</th><th colspan="3">Synthesis</th></tr><tr><th>+</th><th>0</th><th>-</th><th>+</th><th>0</th><th>-</th><th>+</th><th>0</th><th>-</th></tr></thead><tbody><tr><td>anterior</td><td>nan</td><td>0.451</td><td>nan</td><td>nan</td><td>0.956</td><td>nan</td><td>nan</td><td>0.517</td><td>nan</td></tr><tr><td>back</td><td>1.235</td><td>nan</td><td>-0.645</td><td>1.324</td><td>nan</td><td>-0.234</td><td>0.122</td><td>nan</td><td>0.418</td></tr><tr><td>continuant</td><td>0.451</td><td>nan</td><td>nan</td><td>0.956</td><td>nan</td><td>nan</td><td>0.517</td><td>nan</td><td>nan</td></tr><tr><td>high</td><td>2.079</td><td>nan</td><td>-1.842</td><td>2.233</td><td>nan</td><td>-1.231</td><td>0.0</td><td>nan</td><td>0.631</td></tr><tr><td>low</td><td>-1.676</td><td>nan</td><td>1.62</td><td>-2.228</td><td>nan</td><td>2.653</td><td>-0.461</td><td>nan</td><td>0.848</td></tr><tr><td>long</td><td>-3.911</td><td>nan</td><td>1.626</td><td>-4.284</td><td>nan</td><td>2.295</td><td>-0.6</td><td>nan</td><td>0.755</td></tr><tr><td>nasal</td><td>2.003</td><td>nan</td><td>0.0</td><td>2.623</td><td>nan</td><td>0.376</td><td>1.0</td><td>nan</td><td>0.406</td></tr><tr><td>round</td><td>2.349</td><td>nan</td><td>-1.172</td><td>2.572</td><td>nan</td><td>-0.793</td><td>0.158</td><td>nan</td><td>0.411</td></tr><tr><td>syllabic</td><td>0.451</td><td>nan</td><td>nan</td><td>0.956</td><td>nan</td><td>nan</td><td>0.517</td><td>nan</td><td>nan</td></tr><tr><td>tense</td><td>-3.464</td><td>nan</td><td>7.063</td><td>-3.232</td><td>nan</td><td>7.063</td><td>0.466</td><td>nan</td><td>0.0</td></tr></tbody></table>

The t-scores calculated from the articulatory features of vowels are represented in Table 1. One can observe no statistically significant difference between the synthesized utterances and ground truth recording. However, evaluating the human and synthesized recordings against the phonetizations based on the Peng'im transcriptions (Theory) reveals several significant differences among certain articulatory features. Interestingly,

these differences are almost identical between the human and synthesized recordings, suggesting that the speech synthesis system effectively reproduces the vowel qualities of the speaker.

Among the individual features, one observes that high, nasal, and rounded vowels are overrepresented in the theoretical IPA phonetization as compared to the human and synthesized samples. Meanwhile, long and tense vowels are overrepresented among the human and synthesized speech. Synthesized speech also exhibits a greater number of low vowels, though this difference is not deemed significant for the equivalent human speech.

Some of those observations are related to the symbols we chose to use for the theoretical phonetization. Indeed, while the [i] and [u] sounds in diphthongs and triphthongs are transcribed by our FOMA model using the lax vowels [i] and [u] respectively, MultIPA sometimes uses the semi-vowels [j] and [w] (i.e., 'strange' /kʰi.kuaɪ/ vs. [ki:kwä:i]). This could explain the overrepresentation of high and rounded in the phonetization, as well as the overrepresentation of tense in MultIPA's transcriptions.

Conversely, nasal vowels provide an interesting case study. The decline of presence of nasal vowels in the younger generation is often mentioned among the diaspora<fnref n="5" />. However, this is not the case for our speaker. The overrepresentation of nasal vowels in theory may be because MultIPA does not detect or render them accurately. In our dataset, none of its transcriptions contain a nasal vowel; the word 'mango' /suaɪ/ for example, is transcribed [suäi] by MultIPA. Interestingly, it may also be that the nasality is weak, but MultIPA correctly perceives the extended length of nasal vowels<fnref n="6" />. Indeed, the model frequently represents them as long vowels ([säɪn] instead of /sã/ 'three') even though we did not take vowel length into account in our phonetizations. This difference may explain the overrepresentation of long in MultIPA's transcriptions.

### 3.1.2. Consonants

<table><caption>Table 2: t-statistics for articulatory features of consonants. Based on ASR-preferred samples.</caption><thead><tr><th rowspan="2">Features</th><th colspan="3">Teochew Speaker</th><th colspan="3">Synthesis</th><th colspan="3">Synthesis</th></tr><tr><th>+</th><th>0</th><th>-</th><th>+</th><th>0</th><th>-</th><th>+</th><th>0</th><th>-</th></tr></thead><tbody><tr><td>anterior</td><td>-0.876</td><td>nan</td><td>2.07</td><td>-1.661</td><td>nan</td><td>2.637</td><td>-0.724</td><td>nan</td><td>0.833</td></tr><tr><td>back</td><td>1.757</td><td>nan</td><td>-0.355</td><td>2.76</td><td>nan</td><td>-1.233</td><td>1.131</td><td>nan</td><td>-0.782</td></tr><tr><td>constricted glottis</td><td>nan</td><td>nan</td><td>0.694</td><td>nan</td><td>nan</td><td>0.595</td><td>nan</td><td>nan</td><td>-0.17</td></tr><tr><td>consonantal</td><td>0.694</td><td>nan</td><td>nan</td><td>0.595</td><td>nan</td><td>nan</td><td>-0.17</td><td>nan</td><td>nan</td></tr><tr><td>continuant</td><td>0.188</td><td>nan</td><td>0.675</td><td>0.377</td><td>nan</td><td>0.457</td><td>0.189</td><td>nan</td><td>-0.273</td></tr><tr><td>coronal</td><td>-0.67</td><td>nan</td><td>1.629</td><td>-2.2</td><td>nan</td><td>3.378</td><td>-1.549</td><td>nan</td><td>1.804</td></tr><tr><td>affricate</td><td>1.253</td><td>nan</td><td>1.243</td><td>0.784</td><td>nan</td><td>0.266</td><td>-0.48</td><td>nan</td><td>0.0</td></tr><tr><td>distributed</td><td>0.608</td><td>1.629</td><td>-1.205</td><td>0.185</td><td>3.378</td><td>-2.504</td><td>-0.426</td><td>1.804</td><td>-1.481</td></tr><tr><td>high</td><td>2.344</td><td>nan</td><td>-0.844</td><td>2.743</td><td>nan</td><td>-1.488</td><td>0.609</td><td>nan</td><td>-0.524</td></tr><tr><td>labial</td><td>0.509</td><td>nan</td><td>0.527</td><td>1.605</td><td>nan</td><td>-0.171</td><td>1.105</td><td>nan</td><td>-0.72</td></tr><tr><td>lateral</td><td>0.842</td><td>nan</td><td>0.545</td><td>0.391</td><td>nan</td><td>0.521</td><td>-0.457</td><td>nan</td><td>-0.088</td></tr><tr><td>low</td><td>nan</td><td>nan</td><td>0.694</td><td>nan</td><td>nan</td><td>0.595</td><td>nan</td><td>nan</td><td>-0.17</td></tr><tr><td>long</td><td>-1.433</td><td>nan</td><td>0.861</td><td>-1.433</td><td>nan</td><td>0.768</td><td>0.0</td><td>nan</td><td>-0.174</td></tr><tr><td>nasal</td><td>-0.581</td><td>nan</td><td>1.28</td><td>-0.626</td><td>nan</td><td>1.184</td><td>0.0</td><td>nan</td><td>-0.217</td></tr><tr><td>spread glottis</td><td>4.892</td><td>nan</td><td>-0.619</td><td>4.892</td><td>nan</td><td>-0.854</td><td>nan</td><td>nan</td><td>-0.17</td></tr><tr><td>sonorant</td><td>-0.106</td><td>nan</td><td>0.958</td><td>-0.112</td><td>nan</td><td>0.831</td><td>0.0</td><td>nan</td><td>-0.233</td></tr><tr><td>strident</td><td>0.878</td><td>nan</td><td>0.244</td><td>0.572</td><td>nan</td><td>0.26</td><td>-0.321</td><td>nan</td><td>0.0</td></tr><tr><td>velar</td><td>nan</td><td>nan</td><td>0.694</td><td>nan</td><td>nan</td><td>0.595</td><td>nan</td><td>nan</td><td>-0.17</td></tr><tr><td>voice</td><td>-0.735</td><td>nan</td><td>1.553</td><td>-0.571</td><td>nan</td><td>1.251</td><td>0.209</td><td>nan</td><td>-0.503</td></tr></tbody></table>

Consonants, represented in Table 2, tell a similar story. No statistically significant differences were found between the hu-

<footnote n="5">⁵One of the authors teaches Peng'im transcription within the diaspora community and has encountered difficulties with nasal vowels. For certain words, heritage speakers would claim that they do not pronounce them. This topic is also often discussed on a Discord server used by the community.</footnote>

<footnote n="6">⁶See [8, 9] for works on the connection between nasality and vowel duration.</footnote>
man and the synthesized speech, suggesting once again that the TTS model succeeds in replicating the features of the original speaker. However, the differences between the phonetized Peng'im romanization and the human/synthesized speech do not mirror each other to the same degree as for vowels. In fact, only three features yielded a statistically significant difference between the Peng'im transcription and the human pronunciation, compared to five for the synthesized speech.

Comparing the theoretical transcription to the human speaker, one observes an overrepresentation of high and *SpreadGlottis* consonants in the theoretical transcription. Meanwhile, anterior consonants appear underrepresented. These trends also hold true for the synthesized recordings, but these also have fewer back consonants, and more coronal and +/0 distributed consonants.

Similarly to nasal vowels, our MultiPA model might not be trained on languages where aspiration is significant. The over-representation of *SpreadGlottis* consonants in our phonization results from the complete absence of aspirated consonants in MultiPA's transcriptions.

As for the audio transcriptions containing more consonants pronounced with the front of the tongue (*coronal*, anterior, -distributed), we can observe that bilabials are often replaced by alveolars and the velar [ɪ] is often simply realized as [n]. To illustrate, 'to applaud' /pʰaʔ.pʰɔk/ becomes [tætɔ] for the synthesized sample, and 'girlfriend' /nuŋ.pʰɛŋ.iɔ/ becomes [näɪntɛni].

### 3.1.3. Semi-vowels

<table><caption>Table 3: *t-statistics for articulatory features of semivowels.*
Based on ASR-preferred samples.</caption><thead><tr><th rowspan="2">Features</th><th colspan="2">Theory - Teochew Speaker</th><th colspan="2">Theory - Synthesis</th><th colspan="2">Teochew Speaker - Synthesis</th></tr><tr><th>+</th><th>0</th><th>+</th><th>0</th><th>+</th><th>0</th></tr></thead><tbody><tr><td>anterior</td><td>nan</td><td>nan</td><td>0.0</td><td>nan</td><td>-2.377</td><td>nan</td></tr><tr><td>back</td><td>1.317</td><td>nan</td><td>-2.082</td><td>0.604</td><td>nan</td><td>-4.333</td></tr><tr><td>constricted glottis</td><td>2.223</td><td>nan</td><td>-2.623</td><td>1.737</td><td>nan</td><td>-4.284</td></tr><tr><td>continuant</td><td>-2.623</td><td>nan</td><td>2.223</td><td>-4.284</td><td>nan</td><td>1.737</td></tr><tr><td>coronal</td><td>nan</td><td>nan</td><td>0.0</td><td>nan</td><td>-2.377</td><td>nan</td></tr><tr><td>distr</td><td>nan</td><td>0.0</td><td>nan</td><td>-2.377</td><td>nan</td><td>-2.5</td></tr><tr><td>high</td><td>-2.623</td><td>nan</td><td>2.223</td><td>-4.284</td><td>nan</td><td>1.737</td></tr><tr><td>labial</td><td>-1.433</td><td>nan</td><td>0.604</td><td>-1.778</td><td>nan</td><td>-1.963</td></tr><tr><td>sonorant</td><td>0.0</td><td>nan</td><td>nan</td><td>-2.377</td><td>nan</td><td>-2.5</td></tr><tr><td>tense</td><td>nan</td><td>0.0</td><td>nan</td><td>nan</td><td>-2.377</td><td>nan</td></tr><tr><td>velar</td><td>nan</td><td>nan</td><td>0.0</td><td>nan</td><td>-2.377</td><td>nan</td></tr><tr><td>voice</td><td>-2.623</td><td>nan</td><td>2.223</td><td>-4.284</td><td>nan</td><td>1.737</td></tr></tbody></table>

The error scores calculated for semivowels, represented in Table 3, show statistically significant differences between the human and synthesized speech on every feature. This suggests that while the speech synthesis system successfully replicates the consonants and vowels of the original speaker, its production of semivowels is significantly less accurate.

The human speaker's production of semivowels also diverges from the phonetization of the Peng'im transliteration, which one could attribute to differences in pronunciation from the phonetization strategy we used. The differences here are more numerous than for those of consonants and vowels, though one might suppose that the greater articulatory complexity of semivowels could magnify differences attributable to the speaker's variety of the language. It is also telling that the Theory-Synthesis differences mirror the Human-Synthesis differences almost exactly, which suggests that the ASR system judges the human utterances as being much closer to the canonical pronunciation used than the speech synthesis system. This further supports the argument that the TTS system performs badly on semivowels in particular.

One must also consider the possibility that the ASR system itself is responsible for certain errors, and is unable to detect semivowels as accurately as other classes. However, that the Theory-Human test does not exhibit the same radical departures as those involving the speech synthesis system suggests that the problem lies with the TTS system itself. Another, more nuanced analysis is that the ASR system struggles to detect semivowels in synthesized speech, even if the acoustic differences that generate these errors may not be perceptively salient. Given the numerous divergences on nearly every relevant feature, we have chosen not to list them individually in our analyses.

## 3.2. Error Rates Analysis

Looking at pairwise error rates within each phoneme category in Table 8, we can observe the extremely high error rates among semivowels. Both the phoneme error rate (PER) and feature error rates (FER) approach 100% in all tests. Feature error rates tend to be only slightly lower than the phoneme error rates. For both consonants and vowels, PER scores fall at around 60% or above, though feature error rates are at most half as high, and typically much lower. Vowels tend to generate higher phoneme error rates, but lower feature error rates than consonants. One can therefore surmise that while phoneme mismatches are common among vowels, the mismatched phonemes tend to be very close acoustically. Consonant mismatches are rarer, but the feature differences of those mismatches are somewhat more serious than among vowels.

## 3.3. Human Evaluation

<table><caption>Table 4: *t-statistics for articulatory features of vowels.*
Based on human-preferred samples.</caption><thead><tr><th rowspan="2">Features</th><th colspan="2">Theory - Teochew Speaker</th><th colspan="2">Theory - Synthesis</th><th colspan="2">Teochew Speaker - Synthesis</th></tr><tr><th>+</th><th>0</th><th>+</th><th>0</th><th>+</th><th>0</th></tr></thead><tbody><tr><td>anterior</td><td>nan</td><td>nan</td><td>0.451</td><td>nan</td><td>nan</td><td>0.812</td></tr><tr><td>back</td><td>1.235</td><td>nan</td><td>-0.645</td><td>1.393</td><td>nan</td><td>0.0</td></tr><tr><td>continuant</td><td>0.451</td><td>nan</td><td>nan</td><td>1.222</td><td>nan</td><td>0.812</td></tr><tr><td>high</td><td>2.079</td><td>nan</td><td>-1.842</td><td>2.436</td><td>nan</td><td>-1.071</td></tr><tr><td>low</td><td>-1.676</td><td>nan</td><td>1.62</td><td>-1.581</td><td>nan</td><td>2.592</td></tr><tr><td>long</td><td>-3.911</td><td>nan</td><td>1.626</td><td>-4.284</td><td>nan</td><td>2.678</td></tr><tr><td>relax</td><td>2.003</td><td>nan</td><td>0.0</td><td>2.003</td><td>nan</td><td>0.735</td></tr><tr><td>round</td><td>2.349</td><td>nan</td><td>-1.172</td><td>2.321</td><td>nan</td><td>-0.311</td></tr><tr><td>syllabic</td><td>0.451</td><td>nan</td><td>nan</td><td>1.222</td><td>nan</td><td>0.161</td></tr><tr><td>tense</td><td>-3.464</td><td>nan</td><td>7.063</td><td>-2.623</td><td>nan</td><td>0.812</td></tr><tr><td></td><td></td><td></td><td></td><td></td><td>6.374</td><td>0.881</td></tr></tbody></table>

<table><caption>Table 5: *t-statistics for articulatory features of semivowels.*
Based on human-preferred samples.</caption><thead><tr><th rowspan="2">Features</th><th colspan="2">Theory - Teochew Speaker</th><th colspan="2">Theory - Synthesis</th><th colspan="2">Teochew Speaker - Synthesis</th></tr><tr><th>+</th><th>0</th><th>+</th><th>0</th><th>+</th><th>0</th></tr></thead><tbody><tr><td>anterior</td><td>nan</td><td>nan</td><td>0.0</td><td>nan</td><td>-2.072</td><td>nan</td></tr><tr><td>back</td><td>1.317</td><td>nan</td><td>-2.082</td><td>0.943</td><td>-3.591</td><td>-0.457</td></tr><tr><td>constricted glottis</td><td>2.223</td><td>nan</td><td>-2.623</td><td>2.223</td><td>-4.0</td><td>nan</td></tr><tr><td>continuant</td><td>-2.623</td><td>nan</td><td>2.223</td><td>-4.0</td><td>2.223</td><td>-2.17</td></tr><tr><td>coronal</td><td>nan</td><td>nan</td><td>0.0</td><td>nan</td><td>-2.072</td><td>nan</td></tr><tr><td>distr</td><td>nan</td><td>0.0</td><td>nan</td><td>-2.072</td><td>nan</td><td>-2.17</td></tr><tr><td>high</td><td>-2.623</td><td>nan</td><td>2.223</td><td>-4.0</td><td>2.223</td><td>-2.17</td></tr><tr><td>labial</td><td>-1.433</td><td>nan</td><td>0.604</td><td>-1.778</td><td>nan</td><td>-0.457</td></tr><tr><td>sonorant</td><td>0.0</td><td>nan</td><td>nan</td><td>-2.072</td><td>nan</td><td>-2.17</td></tr><tr><td>tense</td><td>nan</td><td>0.0</td><td>nan</td><td>nan</td><td>-2.072</td><td>nan</td></tr><tr><td>velar</td><td>nan</td><td>nan</td><td>0.0</td><td>nan</td><td>nan</td><td>-2.17</td></tr><tr><td>voice</td><td>-2.623</td><td>nan</td><td>2.223</td><td>-4.0</td><td>2.223</td><td>-2.17</td></tr></tbody></table>

When comparing the transcriptions of the Teochew speaker's recordings and the phonetizations of the Peng'im canonical transcriptions against the human-preferred synthesized samples, we observe similar trends, particularly for vowels (Table 4) and semivowels (Table 5). This is confirmed by the
results presented in Table 7, which shows no significant differences between the features detected in the ASR-preferred samples and the human-preferred ones.

<table><caption>**Table 6: t-statistics for articulatory features of consonants.**
Based on human-preferred samples.</caption><thead><tr><th rowspan="2">Features</th><th colspan="3">Teochew Speaker</th><th colspan="3">Synthesis</th><th colspan="3">Synthesis</th></tr><tr><th>+</th><th>0</th><th>-</th><th>+</th><th>0</th><th>-</th><th>+</th><th>0</th><th>-</th></tr></thead><tbody><tr><td>anterior</td><td>-0.876</td><td>nan</td><td>2.07</td><td>-0.894</td><td>nan</td><td>2.042</td><td>0.0</td><td>nan</td><td>0.134</td></tr><tr><td>back</td><td>1.757</td><td>nan</td><td>-0.355</td><td>1.952</td><td>nan</td><td>-0.472</td><td>0.299</td><td>nan</td><td>-0.086</td></tr><tr><td>constricted</td><td>nan</td><td>nan</td><td>0.694</td><td>nan</td><td>nan</td><td>0.83</td><td>nan</td><td>nan</td><td>0.083</td></tr><tr><td>consonantal glottis</td><td>0.694</td><td>nan</td><td>nan</td><td>0.83</td><td>nan</td><td>nan</td><td>0.083</td><td>nan</td><td>nan</td></tr><tr><td>continuant</td><td>0.188</td><td>nan</td><td>0.675</td><td>0.567</td><td>nan</td><td>0.615</td><td>0.379</td><td>nan</td><td>-0.087</td></tr><tr><td>coronal</td><td>-0.67</td><td>nan</td><td>1.629</td><td>-1.27</td><td>nan</td><td>2.436</td><td>-0.613</td><td>nan</td><td>0.848</td></tr><tr><td>affricate</td><td>1.253</td><td>nan</td><td>0.243</td><td>0.784</td><td>nan</td><td>0.521</td><td>-0.48</td><td>nan</td><td>0.253</td></tr><tr><td>distributed</td><td>0.608</td><td>1.629</td><td>-1.205</td><td>0.372</td><td>2.436</td><td>-1.655</td><td>-0.215</td><td>0.848</td><td>-0.572</td></tr><tr><td>high</td><td>2.344</td><td>nan</td><td>-0.844</td><td>2.05</td><td>nan</td><td>-0.702</td><td>-0.145</td><td>nan</td><td>0.176</td></tr><tr><td>labial</td><td>0.509</td><td>nan</td><td>0.527</td><td>1.198</td><td>nan</td><td>0.26</td><td>0.705</td><td>nan</td><td>-0.274</td></tr><tr><td>lateral</td><td>0.842</td><td>nan</td><td>0.545</td><td>0.842</td><td>nan</td><td>0.672</td><td>0.0</td><td>nan</td><td>0.085</td></tr><tr><td>low</td><td>nan</td><td>nan</td><td>0.694</td><td>nan</td><td>nan</td><td>0.83</td><td>nan</td><td>nan</td><td>0.083</td></tr><tr><td>long</td><td>-1.433</td><td>nan</td><td>0.861</td><td>-1.778</td><td>nan</td><td>1.078</td><td>-0.457</td><td>nan</td><td>0.169</td></tr><tr><td>nasal</td><td>-0.581</td><td>nan</td><td>1.28</td><td>-0.374</td><td>nan</td><td>1.294</td><td>0.24</td><td>nan</td><td>-0.109</td></tr><tr><td>spread glottis</td><td>4.892</td><td>nan</td><td>-0.619</td><td>4.892</td><td>nan</td><td>-0.584</td><td>nan</td><td>nan</td><td>0.083</td></tr><tr><td>sonorant</td><td>-0.106</td><td>nan</td><td>0.958</td><td>0.224</td><td>nan</td><td>0.831</td><td>0.341</td><td>nan</td><td>-0.233</td></tr><tr><td>strident</td><td>0.878</td><td>nan</td><td>0.244</td><td>0.572</td><td>nan</td><td>0.51</td><td>-0.321</td><td>nan</td><td>0.247</td></tr><tr><td>velar</td><td>nan</td><td>nan</td><td>0.694</td><td>nan</td><td>nan</td><td>0.83</td><td>nan</td><td>nan</td><td>0.083</td></tr><tr><td>voice</td><td>-0.735</td><td>nan</td><td>1.553</td><td>-0.114</td><td>nan</td><td>1.108</td><td>0.626</td><td>nan</td><td>-0.598</td></tr></tbody></table>

However, the t-statistics for consonants are slightly higher than those for vowels and semivowels. Additionally, fewer features are significantly different when comparing the evaluator-preferred samples to the theory than when comparing the ASR-chosen samples to it (Table 6 and Table 2).

Upon examining pairwise error rates (Table 8), we observe that the evaluator-selected files tend to be closer to the theoretical model and farther from the Teochew speaker's recording than those preferred by the ASR. For instance, when analyzing vowels, the evaluator-preferred synthesis exhibits a 16.86% FER compared to the theory, while the ASR-selected files show a higher 19.66% FER. Conversely, when compared to the Teochew speaker's recording, the evaluator-chosen files have a 15.88% FER, whereas the ASR-selected files have a 13.27% FER. Incidentally, the FER scores between the Teochew speaker's recordings and the evaluator-preferred samples are equivalent when compared to the theory.

## 4. Conclusion

To revisit our question on whether human and ASR evaluations strongly align, our results suggest that the preferences of the human evaluators align more strongly with the canonical pro-

<table><caption>**Table 7: t-statistics for articulatory features of ASR-preferred and speakers-preferred audio files.**</caption><thead><tr><th rowspan="2">Features</th><th colspan="3">Vowels</th><th colspan="3">Consonants</th><th colspan="3">Semivowels</th></tr><tr><th>+</th><th>0</th><th>-</th><th>+</th><th>0</th><th>-</th><th>+</th><th>0</th><th>-</th></tr></thead><tbody><tr><td>anterior</td><td>nan</td><td>-0.327</td><td>nan</td><td>-0.737</td><td>nan</td><td>0.631</td><td>nan</td><td>nan</td><td>-0.182</td></tr><tr><td>back</td><td>-0.115</td><td>nan</td><td>-0.245</td><td>0.766</td><td>nan</td><td>-0.737</td><td>-0.391</td><td>nan</td><td>0.0</td></tr><tr><td>constricted glottis</td><td>x</td><td>x</td><td>x</td><td>nan</td><td>nan</td><td>-0.28</td><td>-1.0</td><td>nan</td><td>0.0</td></tr><tr><td>consonantal</td><td>x</td><td>x</td><td>x</td><td>-0.28</td><td>nan</td><td>nan</td><td>x</td><td>x</td><td>x</td></tr><tr><td>continuant</td><td>-0.327</td><td>nan</td><td>nan</td><td>-0.19</td><td>nan</td><td>-0.191</td><td>0.0</td><td>nan</td><td>-1.0</td></tr><tr><td>coronal</td><td>x</td><td>x</td><td>x</td><td>-0.93</td><td>nan</td><td>-0.927</td><td>nan</td><td>nan</td><td>-0.182</td></tr><tr><td>affricate</td><td>x</td><td>x</td><td>x</td><td>0.0</td><td>nan</td><td>-0.279</td><td>x</td><td>x</td><td>x</td></tr><tr><td>distributed</td><td>x</td><td>x</td><td>x</td><td>-0.194</td><td>0.927</td><td>-0.878</td><td>nan</td><td>-0.182</td><td>nan</td></tr><tr><td>high</td><td>-0.24</td><td>nan</td><td>-0.122</td><td>0.687</td><td>nan</td><td>-0.74</td><td>0.0</td><td>nan</td><td>-1.0</td></tr><tr><td>labial</td><td>x</td><td>x</td><td>x</td><td>0.372</td><td>nan</td><td>-0.443</td><td>0.0</td><td>nan</td><td>-0.211</td></tr><tr><td>lateral</td><td>x</td><td>x</td><td>x</td><td>-0.457</td><td>nan</td><td>-0.194</td><td>x</td><td>x</td><td>x</td></tr><tr><td>low</td><td>-0.636</td><td>nan</td><td>0.124</td><td>nan</td><td>nan</td><td>-0.28</td><td>x</td><td>x</td><td>x</td></tr><tr><td>long</td><td>0.0</td><td>nan</td><td>-0.29</td><td>0.457</td><td>nan</td><td>-0.374</td><td>x</td><td>x</td><td>x</td></tr><tr><td>nasal</td><td>1.0</td><td>nan</td><td>-0.434</td><td>-0.26</td><td>nan</td><td>-0.122</td><td>x</td><td>x</td><td>x</td></tr><tr><td>round</td><td>0.338</td><td>nan</td><td>-0.548</td><td>x</td><td>x</td><td>x</td><td>x</td><td>x</td><td>x</td></tr><tr><td>spread glottis</td><td>x</td><td>x</td><td>x</td><td>nan</td><td>nan</td><td>-0.28</td><td>x</td><td>x</td><td>x</td></tr><tr><td>sonorant</td><td>x</td><td>x</td><td>x</td><td>-0.361</td><td>nan</td><td>0.0</td><td>-0.182</td><td>nan</td><td>nan</td></tr><tr><td>strident</td><td>x</td><td>x</td><td>x</td><td>0.0</td><td>nan</td><td>-0.265</td><td>x</td><td>x</td><td>x</td></tr><tr><td>syllabic</td><td>-0.327</td><td>nan</td><td>nan</td><td>x</td><td>x</td><td>x</td><td>x</td><td>x</td><td>x</td></tr><tr><td>tense</td><td>-0.471</td><td>nan</td><td>0.397</td><td>x</td><td>x</td><td>x</td><td>x</td><td>x</td><td>x</td></tr><tr><td>velar</td><td>x</td><td>x</td><td>x</td><td>nan</td><td>nan</td><td>-0.28</td><td>nan</td><td>nan</td><td>-0.182</td></tr><tr><td>voice</td><td>x</td><td>x</td><td>x</td><td>-0.454</td><td>nan</td><td>0.134</td><td>0.0</td><td>nan</td><td>-1.0</td></tr></tbody></table>

<table><caption>**Table 8: PER and FER (in %) for the different datasets.**</caption><thead><tr><th rowspan="2"></th><th colspan="2">Vowels</th><th colspan="2">Consonants</th><th colspan="2">Semivowels</th></tr><tr><th>PER</th><th>FER</th><th>PER</th><th>FER</th><th>PER</th><th>FER</th></tr></thead><tbody><tr><td>Theory – Teochew Speaker</td><td>76.04</td><td>16.81</td><td>59.93</td><td>22.90</td><td>100.0</td><td>96.21</td></tr><tr><td>Theory – Human-preferred synthesis</td><td>74.24</td><td>16.86</td><td>64.14</td><td>22.68</td><td>100.0</td><td>95.47</td></tr><tr><td>Theory – ASR-preferred synthesis</td><td>79.86</td><td>19.66</td><td>68.47</td><td>25.10</td><td>100.0</td><td>95.94</td></tr><tr><td>Teochew – Human-preferred Speaker synthesis</td><td>56.80</td><td>15.88</td><td>61.07</td><td>28.95</td><td>100.0</td><td>92.40</td></tr><tr><td>Teochew – ASR-preferred Speaker synthesis</td><td>59.64</td><td>13.27</td><td>60.22</td><td>28.13</td><td>94.44</td><td>87.73</td></tr><tr><td>Human-preferred – ASR-preferred synthesis synthesis</td><td>30.92</td><td>9.69</td><td>19.43</td><td>7.12</td><td>41.18</td><td>37.38</td></tr></tbody></table>

nunciations, while the ASR model is more biased towards the the speaker's variety of the language. While the latter is not surprising as the speaker's variety is the only variety that the model was exposed to, the former was not necessarily expected. Interestingly, the feature error rates (FER) calculated between the human recordings and the theoretical pronunciations are similar to those calculated between the human-selected synthesis samples and the theoretical pronunciations. This may indicate that the human annotators have a certain bias against the variety used in the training data, and preferred synthesized speech samples that were closer to the variety used in our theoretical phonetization. It also suggests the possibility that the exposure to the transliteration system influences their judgments towards a written standard.

One factor that prevents a deeper analysis within the scope of this paper is the difference in vocabulary in the different phonetization strategies employed. The FOMA-generated IPA transcriptions were not always equivalent to those generated by MultIPA, which treated long vowels and diphthongs differently. Future studies should implement strategies to mitigate this issue.

Most of the evidence generated in this study suggests that the model performs well at replicating the articulatory features of consonants and vowels of the original speaker, even if human evaluators may have a certain bias against the samples that sound closest to that speaker. However, this is not necessarily the case with semivowels, which show statistically significant differences between the synthesized recordings and the ground truth. Further investigation is needed to identify whether it is the ASR system which performs badly at recognizing these, or if the synthesis system performs badly at generating them.

## 5. References

[1] P. Magistry, I. Wang, and E. T. Lim, “Experiments on Speech Synthesis for Teochew, Can Taiwanese Help?” in *Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)*, 2024, pp. 6849–6854.

[2] G. Eren, “Coqui TTS,” The Coqui TTS Team, 2021. [Online]. Available: https://www.coqui.ai

[3] J. Kim, J. Kong, and J. Son, “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech,” in *International Conference on Machine Learning*. PMLR, 2021, pp. 5530–5540.

[4] M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov, “Label Studio: Data Labeling Software,” 2020-2022, open source software available from https://github.com/heartexlabs/label-studio. [Online]. Available: https://github.com/heartexlabs/label-studio
[5] M. Hulden, "Foma: a finite-state compiler and library," in *Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics*. Association for Computational Linguistics, 2009, pp. 29–32.

[6] C. Taguchi, Y. Sakai, P. Haghani, and D. Chiang, "Universal Automatic Phonetic Transcription into the International Phonetic Alphabet," *arXiv preprint arXiv:2308.03917*, 2023.

[7] D. R. Mortensen, P. Littell, A. Bharadwaj, K. Goyal, C. Dyer, and L. Levin, "PanPhon: A Resource for Mapping IPA Segments to Articulatory Feature Vectors," in *Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers*, 2016, pp. 3475–3484.

[8] D. H. Whalen and P. S. Beddor, "Connections between nasality and vowel duration and height: Elucidation of the eastern algonquian intrusive nasal," *Language*, pp. 457–486, 1989.

[9] P. S. Beddor, "The perception of nasal vowels," in *Nasals, nasalization, and the velum*. Elsevier, 1993, pp. 171–196.