<figure><img src="image_1.png" /></figure>

HAL
open science

# Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization

Natalia Tomashenko, Emmanuel Vincent, Marc Tommasi

► To cite this version:

Natalia Tomashenko, Emmanuel Vincent, Marc Tommasi. Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization. 2025 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2025), Apr 2025, Hyderabad, India. 10.1109/ICASSP49660.2025.10887896. hal-04853872

HAL Id: hal-04853872

https://hal.science/hal-04853872v1

Submitted on 22 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization

Natalia Tomashenko

Université de Lorraine, CNRS, Inria
Loria, F-54000
Nancy, France

natalia.tomashenko@inria.fr

Emmanuel Vincent

Université de Lorraine, CNRS, Inria
Loria, F-54000
Nancy, France

emmanuel.vincent@inria.fr

Marc Tommasi

Université de Lille, CNRS, Inria
Centrale Lille, UMR 9189 - CRISTAL
Lille, France

marc.tommasi@inria.fr

**Abstract**—In this paper, we investigate the impact of speech temporal dynamics in application to automatic speaker verification and speaker voice anonymization tasks. We propose several metrics to perform automatic speaker verification based only on phoneme durations. Experimental results demonstrate that phoneme durations leak some speaker information and can reveal speaker identity from both original and anonymized speech. Thus, this work emphasizes the importance of taking into account the speaker's speech rate and, more importantly, the speaker's phonetic duration characteristics, as well as the need to modify them in order to develop anonymization systems with strong privacy protection capacity.

***Index Terms***—Speech temporal dynamics, speech rate, anonymization, automatic speaker verification, phoneme duration characteristics.

## I. INTRODUCTION

Speech data carries personal or sensitive information via speaker traits (e.g., identity, gender, age, ethnicity, accent), and sometimes via linguistic content (e.g., name, address) and paralinguistic content (e.g., emotion). Most voice-based human-computer interaction technologies today rely on cloud-based machine learning systems trained on speech data collected from the users. This poses serious privacy risks and requires implementation of privacy-enhancing technologies to protect users' sensitive and private information.

One common approach to privacy protection of speech data is voice anonymization, which aims to suppress personally identifiable speaker traits, leaving linguistic and paralinguistic content intact [1]. Voice anonymization methods can be broadly classified into two categories. Signal processing based methods rely on simple signal transformations such as spectral warping using the McAdams coefficient [2], pitch shifting based on time-scale modification [3], and others [4], [5]. By contrast, neural voice conversion based methods [6]–[9] rely on disentangling attributes such as content, speaker, pitch, emotion, etc., anonymizing the selected attributes, and generating the anonymized speech signal using a speech synthesis model. Most state-of-the-art voice conversion based anonymization

methods use large-scale pre-trained models for extracting specific attributes and provide better content and privacy preservation than signal processing based methods. The diversity of approaches is illustrated by the VoicePrivacy 2024 Challenge [10], which provided six baseline anonymization systems, namely anonymization using x-vectors and a neural source-filter model [6], [11], signal processing based anonymization using the McAdams coefficient [2], anonymization using phonetic transcription and generation of artificial pseudo-speaker embeddings by a generative adversarial network (GAN) [12] anonymization using neural audio codec (NAC) language modeling [13], and anonymization using acoustic vector quantization bottleneck (VQ-BN) features from an automatic speech recognition (ASR) acoustic model.

While specific studies have been dedicated to speaker information carried by pitch [5], [6], [8], the impact of speech temporal dynamics on speaker verification and re-identification has been overlooked. As a result, all the baseline systems of the VoicePrivacy 2024 Challenge modify various characteristics of the input (original) speech signal linked with speaker identity but keep speech rate and phoneme durations unchanged. Most other state-of-the-art anonymization systems also do not modify phoneme durations [7], [9], [14].

Among the rare exceptions are cascaded ASR and text-to-speech (TTS) systems where word-level [15] or phoneme-level transcripts obtained by an ASR system are provided to a TTS system for synthesis of the given linguistic content with a new target voice. It can be assumed that these systems do not retain any information about speaker identity, however they fail to preserve any paralinguistic attributes, which are required in real-life voice anonymization scenarios. The most relevant work for our study is [16] where speed perturbation with a constant factor was used as an anonymization method either alone or in combination with anonymization based on a cycle consistent generative adversarial network (CycleGAN). The authors showed that speech rate perturbation with a constant factor degrades the performance of the automatic speaker verification systems associated with *ignorant* and *lazy-informed* attackers [17], but they did not consider the stronger *semi-informed* attack model that is today's standard [18].

Speed perturbation alone cannot be considered as a strong privacy protection method since a lot of speaker information remains in the speech signal after it. Yet we show that it is an essential ingredient in

<footnote>This work was supported by the French National Research Agency under project Speech Privacy and project IPoP of the Cybersecurity PEPR. Experiments were carried out using the Grid'5000 testbed.</footnote>
suppressing speaker information and must be taken into account in state-of-the-art anonymization systems. Indeed, while state-of-the art automatic speaker verification (ASV) systems do not explicitly rely on speaker temporal dynamics [19], a few past studies have shown the applicability of durational characteristics for this task [20]–[22]. Works [21], [22] propose to use speech rhythm-based emebeddings for speech synthesis. We also show that more information about the speaker is contained in the temporal dynamics and the duration of phonemes than in the speech rate.

Our contributions build upon [20]–[22] and include: phoneme duration features and distance metrics for ASV based on phoneme durations (Section II); and experimental evaluation and analysis of the resulting ASV performance on original data and data anonymized using two state-of-the-art anonymization systems with and without temporal dynamics modification (Section III). To our best knowledge, this is the first work that performs such analysis and evaluation of the impact of speaker temporal dynamics on the anonymization task and demonstrates its importance for the design of voice anonymization systems.

## II. SPEAKER VERIFICATION USING PHONEME DURATION DYNAMICS

### A. Metrics

We define two metrics to quantify the distance between speakers’ temporal dynamics in the context of speaker verification. Let us denote by $N$ the number of phoneme classes $ph_1, \dots, ph_N$, and for two speakers $s_i$ and $s_j$ in the dataset by $u_i^1 \dots u_i^{M_i}$ the utterances of speaker $s_i$ and $u_j^1 \dots u_j^{M_j}$ the utterances of speaker $s_j$.

The first metric is based on the cosine distance between two vectors of mean phoneme durations:

$$ \rho_1(s_i, s_j) = 1 - \cos(\mu_i, \mu_j), \quad (1) $$

where $\mu_i$, $\mu_j$ are $N$-dimensional vectors composed of the average lengths of phonemes $ph_1, \dots, ph_N$ computed over utterances $u_i^1 \dots u_i^M$ (for $\mu_i = [\mu_i^{(1)}, \dots, \mu_i^{(N)}]$) and $u_j^1 \dots u_j^M$ (for $\mu_j = [\mu_j^{(1)}, \dots, \mu_j^{(N)}]$), respectively. In case phoneme $ph_k$ is missing in the considered utterances or has less than a given number of instances (considered as a threshold parameter), its mean values in $\mu_i$ or $\mu_j$ are replaced by the global mean duration of all phonemes in the considered utterances for a given speaker. Before computing metric $\rho_1$, mean normalization is applied to all $\mu_i$ and $\mu_j$.

We also propose a second metric that is defined as follows:

$$ \rho_2(s_i, s_j) = 1 - \frac{1}{N} \sum_{k=1}^{N} \min \left\{ \frac{\mu_i^{(k)}}{\mu_j^{(k)}}, \frac{\mu_j^{(k)}}{\mu_i^{(k)}} \right\}. \quad (2) $$

Based on the proposed metrics we can perform ASV and compute an equal error rate (EER). Such ASV systems can also be considered as attackers for the anonymization task.

### B. Phoneme sets

We experimented with two sets of phonetic classes: (1) $N = 39$ phonemes based on the ARPAbet symbol set corresponding to the Carnegie Mellon University pronunciation dictionary<fnref n="1" />, not counting variations due to lexical stress; and (2) $N = 336$ phoneme classes that take into account position in the word and stress.

## III. EXPERIMENTAL RESULTS

### A. Data

Experiments were conducted on the LibriSpeech<fnref n="2" /> [23] corpus of read English audiobooks, which was used in all VoicePrivacy Challenge editions. It contains approximately 1,000 hours of speech from 2,484 speakers sampled at 16 kHz. We conducted the first series of experiments and analyses on the full LibriSpeech-train-960 dataset that contains data from 2,338 speakers. In the second series of experiments, we used the LibriSpeech-train-clean-360 subset anonymized by the two different speaker voice anonymization systems denoted SAS-1 and SAS-2 described in Section III-B. To perform phoneme segmentation, four triphone Gaussian mixture model - hidden Markov model (GMM-HMM) acoustic models were trained using the Kaldi speech recognition toolkit [24] on the following training data: (1) original LibriSpeech-train-960; (2) original LibriSpeech-train-clean-360; (3) LibriSpeech-train-clean-360 anonymized by SAS-1; and (4) LibriSpeech-train-clean-360 anonymized by SAS-2. Statistics for the number of trials used in ASV evaluation are given in Table I.

<table><caption>TABLE I STATISTICS FOR TRIALS.</caption><thead><tr><th rowspan="2">Average # utter.<br>per trial</th><th colspan="2">LibriSpeech-train-960</th><th colspan="2">LibriSpeech-train-clean-360</th></tr><tr><th>Same speaker</th><th>Different speaker</th><th>Same speaker</th><th>Different speaker</th></tr></thead><tbody><tr><td>1</td><td>17,527,076</td><td></td><td>5,944,163</td><td></td></tr><tr><td>3</td><td>1,816,610</td><td></td><td>716,384</td><td></td></tr><tr><td>5</td><td>644,790</td><td></td><td>253,935</td><td></td></tr><tr><td>10</td><td>154,308</td><td>233,800</td><td>60,786</td><td></td></tr><tr><td>20</td><td>35,070</td><td></td><td>13,815</td><td></td></tr><tr><td>40</td><td>7,014</td><td></td><td>2,763</td><td></td></tr><tr><td>60</td><td>2,338</td><td></td><td>921</td><td></td></tr></tbody></table>

### B. Anonymization systems

To investigate the impact of speaker voice anonymization, we consider two different state-of-the-art speaker voice anonymization systems (SASs):

* SAS-1 keeps the original temporal phoneme dynamics, but changes the other speaker characteristics (speaker identity and some prosodical characteristics such as pitch and energy).

* SAS-2 is a cascaded ASR-TTS system that changes phoneme durations.

Below we briefly describe these systems.

1) SAS-1: SAS-1, proposed in [12] and used as baseline B3 in the VoicePrivacy 2024 Challenge [10], is a system based on anonymization using phonetic transcription and a GAN that generates artificial pseudo-speaker embeddings. Anonymization is performed in three steps: (1) extraction of the speaker embedding, phonetic transcription, pitch, energy, and phone duration from the original audio waveform; (2) speaker embedding anonymization, pitch and energy modification; and (3) synthesis of an anonymized speech waveform from the anonymized speaker embedding, modified pitch and energy features, original phonetic transcripts and original phone durations.

<footnote n="1">¹http://www.speech.cs.cmu.edu/cgi-bin/cmudict</footnote>

<footnote n="2">²LibriSpeech: http://www.openslr.org/12</footnote>
2) SAS-2: SAS-2, proposed in [25], is one of the best systems developed by the VoicePrivacy 2024 Challenge participants in terms of linguistic content and privacy preservation. It is a cascaded ASR-TTS system, where first the text transcripts are obtained from the source audio and then a TTS system is used to generate corresponding anonymized speech from the obtained transcripts with a new anonymized speaker voice. The ASR model is the medium English Whisper model [26]. The TTS model is VITS (variational inference with adversarial learning for end-to-end text-to-speech, [27]), trained on the LibriTTS dataset [28].

3) Automatic speaker verification and speech recognition results for SAS systems: The ASV results in terms of equal error rate (EER) and the automatic speech recognition (ASR) results in terms of word error rate (WER) are shown in Table II on the LibriSpeech test set for original and anonymized data. The trial lists for ASV evaluation in the LibriSpeech test data are taken from the VoicePrivacy 2024 Challenge [10] setup. ASV evaluation for original and anonymized data was performed with the same ASV model architecture and training setup as proposed in the VoicePrivacy 2024 Challenge. For anonymized data, the strongest semi-informed attack models, trained on the utterance-level anonymized data were used in evaluation.

<table><caption>TABLE II
EER (%) AND WER (%) ON ORIGINAL AND ANONYMIZED DATA FROM THE LibriSpeech test DATASET.</caption><thead><tr><th>System</th><th>EER, % female</th><th>EER, % male</th><th>WER, %</th></tr></thead><tbody><tr><td>Original</td><td>8.8</td><td>0.4</td><td>1.85</td></tr><tr><td>SAS-1</td><td>27.9</td><td>26.7</td><td>4.35</td></tr><tr><td>SAS-2</td><td>47.5</td><td>48.8</td><td>3.76</td></tr></tbody></table>

## C. Results

To analyse speech temporal dynamics we performed several series of experiments dedicated to (1) impact of the speaker's phoneme durations on the ASV performance and metric comparison for ASV; (2) impact of the phoneme set on the ASV performance with the proposed models; (3) impact of the speech rate on the ASV performance of the proposed attack models and the effect of normalizing all speakers to the same speech rate; and (4) effect of different anonymization strategies on the ASV performance of the proposed systems.

1) Speaker verification using phoneme durations and metric choice: The ASV results obtained using metric $ρ_1$ and 38 phonemes are given in Table III in terms of EER. Each line corresponds to the EERs obtained when the average number of utterances per speaker used to compute metric $ρ_1$ equals to the value in the first column ("Average # utter. per trial"). Different columns (1,3,..., 20 - minimum number of phoneme instances for averaging) correspond to different values of the threshold parameter as defined in Section II-A. Increasing the number of utterances used to compute the speaker similarity metric allows us to significantly reduce the EER down to 9.2% with 60 utterances per speaker.

The results reported in Table IV for metric $ρ_2$ on the same data show similar trends. Also we can see that this metric is more efficient than $ρ_1$ when more utterances are used. Thus we use $ρ_2$ in the following experiments.

2) Selecting a set of acoustic units: Table V reports results with the increased number of phoneme classes: $N = 336$. Comparing

<table><caption>TABLE III
EER (%) ON ORIGINAL DATA FROM THE LibriSpeech-train-960 DATASET OBTAINED USING METRIC $ρ_1$ AND $N = 38$ PHONEMES.</caption><thead><tr><th rowspan="2">Average # utter.<br>per trial</th><th colspan="5">Minimum # phoneme instances for aver.</th></tr><tr><th>1</th><th>3</th><th>5</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>1</td><td>39.9</td><td>38.5</td><td>38.7</td><td>39.8</td><td>40.3</td></tr><tr><td>3</td><td>34.9</td><td>32.4</td><td>32.0</td><td>32.2</td><td>34.9</td></tr><tr><td>5</td><td>31.9</td><td>28.4</td><td>27.8</td><td>27.9</td><td>29.3</td></tr><tr><td>10</td><td>28.2</td><td>23.1</td><td>22.3</td><td>22.3</td><td>23.4</td></tr><tr><td>20</td><td>22.4</td><td>23.4</td><td>18.1</td><td>16.7</td><td>17.1</td></tr><tr><td>40</td><td>16.0</td><td>17.7</td><td>21.0</td><td>12.8</td><td>12.3</td></tr><tr><td>60</td><td>12.8</td><td>13.2</td><td>16.8</td><td>15.4</td><td>9.2</td></tr></tbody></table>

<table><caption>TABLE IV
EER (%) ON ORIGINAL DATA FROM THE LibriSpeech-train-960 DATASET OBTAINED USING METRIC $ρ_2$ AND $N = 38$ PHONEMES.</caption><thead><tr><th rowspan="2">Average # utter.<br>per trial</th><th colspan="5">Minimum # phoneme instances for aver.</th></tr><tr><th>1</th><th>3</th><th>5</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>1</td><td>40.3</td><td>38.9</td><td>39.4</td><td>39.9</td><td>39.9</td></tr><tr><td>3</td><td>33.0</td><td>29.6</td><td>29.8</td><td>31.8</td><td>33.4</td></tr><tr><td>5</td><td>26.8</td><td>23.8</td><td>22.6</td><td>23.8</td><td>27.3</td></tr><tr><td>10</td><td>17.6</td><td>16.5</td><td>15.1</td><td>13.6</td><td>14.9</td></tr><tr><td>20</td><td>10.2</td><td>9.1</td><td>9.3</td><td>7.7</td><td>6.9</td></tr><tr><td>40</td><td>5.1</td><td>4.7</td><td>4.3</td><td>4.2</td><td>3.2</td></tr><tr><td>60</td><td>3.3</td><td>3.1</td><td>2.7</td><td>2.7</td><td>2.5</td></tr></tbody></table>

them with Table IV, we can see that increasing the number of phoneme classes does not provide improvement in EER.

<table><caption>TABLE V
EER (%) ON ORIGINAL DATA FROM THE LibriSpeech-train-960 DATASET OBTAINED USING METRIC $ρ_2$ AND $N = 336$ PHONEME CLASSES.</caption><thead><tr><th rowspan="2">Average # utter.<br>per trial</th><th colspan="5">Minimum # phoneme instances for aver.</th></tr><tr><th>1</th><th>3</th><th>5</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>1</td><td>39.3</td><td>39.7</td><td>39.9</td><td>39.8</td><td>39.9</td></tr><tr><td>3</td><td>31.8</td><td>32.5</td><td>32.9</td><td>33.1</td><td>33.4</td></tr><tr><td>5</td><td>26.3</td><td>27.0</td><td>27.6</td><td>28.3</td><td>28.8</td></tr><tr><td>10</td><td>18.7</td><td>19.5</td><td>20.0</td><td>21.1</td><td>22.3</td></tr><tr><td>20</td><td>12.0</td><td>12.5</td><td>13.1</td><td>14.0</td><td>15.5</td></tr><tr><td>40</td><td>6.5</td><td>6.6</td><td>6.5</td><td>7.7</td><td>8.7</td></tr><tr><td>60</td><td>4.4</td><td>3.6</td><td>4.0</td><td>4.5</td><td>5.6</td></tr></tbody></table>

3) Speech rate as a discriminative feature and normalization of speech temporal dynamics to speech rate: Table VI shows ASV performance based on speaker's speech rate. The speech rate was calculated as $\frac{\sum_{k=1}^{K} \bar{l}_k}{\sum_{k=1}^{K} l_k}$, where K is the number of phones in the utterance, $l_k$ is the actual duration of phone k in the utterance, $\bar{l}_k$ is the expected mean duration of the corresponding phoneme k estimated from the training corpus. We can see that speech rate allows us to successfully perform ASV although, when the average number of utterances is larger than 3, the EER is higher in comparison with the cases when we use phoneme-based temporal characteristics.

Tables VII and VIII show the ASV results after performing global speech rate normalization. In these experiments, we first computed phoneme duration statistics over the full LibriSpeech-train-960 corpus and then the speech rate of each utterance was adjusted with a constant factor to match the average speech rate. As expected, such normalization degrades the performance of the ASV systems compared to the results without normalization in Tables IV and V in most cases. However, interestingly, normalization achieves lower
<table><caption>**TABLE VI**
EER (%) ON ORIGINAL DATA FROM THE LibriSpeech-train-960 DATASET OBTAINED USING METRIC $\rho_2$ AND SPEECH RATE.</caption><thead><tr><th>Average # utter.<br>per trial</th><th></th></tr></thead><tbody><tr><td>1</td><td>38.6</td></tr><tr><td>3</td><td>31.9</td></tr><tr><td>5</td><td>27.4</td></tr><tr><td>10</td><td>22.1</td></tr><tr><td>20</td><td>17.8</td></tr><tr><td>40</td><td>13.9</td></tr><tr><td>60</td><td>11.8</td></tr></tbody></table>

EER results (2%) when using a large number of utterances (60) and $N = 336$.

<table><caption>**TABLE VII**
EER (%) ON ORIGINAL DATA FROM THE LibriSpeech-train-960 DATASET OBTAINED USING METRIC $\rho_2$ WITH GLOBAL SPEECH RATE NORMALIZATION AND $N = 38$ PHONEMES.</caption><thead><tr><th rowspan="2">Average # utter.<br>per trial</th><th colspan="5">Minimum # phoneme instances for aver.</th></tr><tr><th>1</th><th>3</th><th>5</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>1</td><td>45.1</td><td>45.5</td><td>46.7</td><td>48.3</td><td>49.0</td></tr><tr><td>3</td><td>38.8</td><td>36.5</td><td>37.6</td><td>41.0</td><td>44.0</td></tr><tr><td>5</td><td>33.3</td><td>30.6</td><td>29.8</td><td>32.0</td><td>37.2</td></tr><tr><td>10</td><td>24.0</td><td>22.8</td><td>21.0</td><td>19.9</td><td>21.8</td></tr><tr><td>20</td><td>14.6</td><td>13.7</td><td>13.6</td><td>11.5</td><td>10.9</td></tr><tr><td>40</td><td>7.3</td><td>7.2</td><td>6.7</td><td>6.5</td><td>4.8</td></tr><tr><td>60</td><td>4.7</td><td>4.4</td><td>4.7</td><td>4.2</td><td>3.8</td></tr></tbody></table>

<table><caption>**TABLE VIII**
EER (%) ON ORIGINAL DATA FROM THE LibriSpeech-train-960 DATASET OBTAINED USING METRIC $\rho_2$ WITH GLOBAL SPEECH RATE NORMALIZATION AND $N = 336$ PHONEME CLASSES.</caption><thead><tr><th rowspan="2">Average # utter.<br>per trial</th><th colspan="5">Minimum # phoneme instances for aver.</th></tr><tr><th>1</th><th>3</th><th>5</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>1</td><td>46.7</td><td>47.8</td><td>48.0</td><td>48.1</td><td>48.2</td></tr><tr><td>3</td><td>40.2</td><td>42.6</td><td>44.0</td><td>45.1</td><td>45.7</td></tr><tr><td>5</td><td>33.7</td><td>35.7</td><td>38.0</td><td>41.0</td><td>42.6</td></tr><tr><td>10</td><td>23.9</td><td>22.9</td><td>24.1</td><td>28.7</td><td>34.2</td></tr><tr><td>20</td><td>15.4</td><td>12.4</td><td>12.3</td><td>13.3</td><td>18.1</td></tr><tr><td>40</td><td>8.6</td><td>5.2</td><td>4.9</td><td>4.8</td><td>5.4</td></tr><tr><td>60</td><td>5.6</td><td>3.1</td><td>2.4</td><td>2.5</td><td>2.0</td></tr></tbody></table>

<table><caption>**TABLE IX**
EER (%) ON ORIGINAL DATA FROM THE LibriSpeech-train-clean-360 DATASET OBTAINED USING METRIC $\rho_2$ AND $N = 38$ PHONEMES.</caption><thead><tr><th rowspan="2">Average # utter.<br>per trial</th><th colspan="5">Minimum # phoneme instances for aver.</th></tr><tr><th>1</th><th>3</th><th>5</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>1</td><td>40.4</td><td>39.2</td><td>39.6</td><td>40.1</td><td>40.1</td></tr><tr><td>3</td><td>34.7</td><td>31.7</td><td>31.7</td><td>33.4</td><td>34.3</td></tr><tr><td>5</td><td>28.1</td><td>25.6</td><td>24.7</td><td>26.0</td><td>28.9</td></tr><tr><td>10</td><td>18.5</td><td>17.6</td><td>16.1</td><td>15.5</td><td>16.9</td></tr><tr><td>20</td><td>10.4</td><td>9.6</td><td>9.7</td><td>8.2</td><td>8.0</td></tr><tr><td>40</td><td>4.9</td><td>4.6</td><td>4.1</td><td>3.9</td><td>3.7</td></tr><tr><td>60</td><td>2.7</td><td>2.9</td><td>3.0</td><td>2.5</td><td>2.5</td></tr></tbody></table>

4) Experiments on anonymized data: Experiments on anonymized LibriSpeech-train-clean-360 data for the two anonymization systems SAS-1 and SAS-2 are reported in Tables X and XI, respectively. For comparison purposes, we also added results on the original

data for the same dataset in Table IX. SAS-1 does not change phoneme durations and we can see despite some degradation of results (in Table X vs. IX) that the preserved speech dynamics still allow us to retrieve speaker information (the lowest EER is 7%). SAS-2 changes phoneme durations and as expected provides much higher privacy protection (Table XI). However, surprisingly, for a large number of utterances (60), the EER is still low (26.3%). One possible explanation might be that, in read speech, the book content may impact the speaking style and thus temporal dynamic statistics.

<table><caption>**TABLE X**
EER (%) ON DATA ANONYMIZED BY SAS-1 FROM THE LibriSpeech-train-clean-360 DATASET OBTAINED USING METRIC $\rho_2$ AND $N = 38$ PHONEMES.</caption><thead><tr><th rowspan="2">Average # utter.<br>per trial</th><th colspan="5">Minimum # phoneme instances for aver.</th></tr><tr><th>1</th><th>3</th><th>5</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>1</td><td>42.3</td><td>40.2</td><td>40.2</td><td>40.4</td><td>40.3</td></tr><tr><td>3</td><td>37.9</td><td>35.1</td><td>34.2</td><td>34.3</td><td>34.7</td></tr><tr><td>5</td><td>33.2</td><td>30.5</td><td>29.3</td><td>28.9</td><td>29.8</td></tr><tr><td>10</td><td>25.4</td><td>24.1</td><td>23.1</td><td>21.3</td><td>21.3</td></tr><tr><td>20</td><td>17.7</td><td>16.7</td><td>16.6</td><td>15.4</td><td>13.9</td></tr><tr><td>40</td><td>10.4</td><td>10.2</td><td>9.8</td><td>9.6</td><td>8.5</td></tr><tr><td>60</td><td>7.4</td><td>7.3</td><td>7.2</td><td>7.0</td><td>7.1</td></tr></tbody></table>

<table><caption>**TABLE XI**
EER (%) ON DATA ANONYMIZED BY SAS-2 FROM THE LibriSpeech-train-clean-360 DATASET OBTAINED USING METRIC $\rho_2$ AND $N = 38$ PHONEMES.</caption><thead><tr><th rowspan="2">Average # utter.<br>per trial</th><th colspan="5">Minimum # phoneme instances for aver.</th></tr><tr><th>1</th><th>3</th><th>5</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>1</td><td>49.0</td><td>49.4</td><td>49.4</td><td>49.3</td><td>49.7</td></tr><tr><td>3</td><td>47.7</td><td>47.4</td><td>47.5</td><td>48.4</td><td>48.7</td></tr><tr><td>5</td><td>46.3</td><td>45.6</td><td>45.6</td><td>46.0</td><td>47.8</td></tr><tr><td>10</td><td>43.4</td><td>43.1</td><td>42.1</td><td>41.8</td><td>41.9</td></tr><tr><td>20</td><td>39.1</td><td>38.6</td><td>38.7</td><td>36.2</td><td>36.8</td></tr><tr><td>40</td><td>32.1</td><td>32.0</td><td>31.4</td><td>31.5</td><td>28.0</td></tr><tr><td>60</td><td>27.6</td><td>27.0</td><td>26.3</td><td>27.5</td><td>26.3</td></tr></tbody></table>

## IV. CONCLUSIONS

In this study, we demonstrated the importance of speech temporal dynamics analysis which has been under-explored in voice anonymization research to date. Using the proposed metrics and sufficient amount of data per speaker, we achieve an EER as low as 7% on the anonymized data obtained by a speaker voice anonymization system that does not modify phoneme durations. In future work, we plan to verify the observed phenomena on other types of speech data, in particular on spontaneous speech, and to improve state-of-the-art anonymization techniques by integrating temporal dynamics normalization. The proposed simple approach to analyze temporal dynamics shows the potential for more advanced analysis by means of machine learning (ML) models that will allow integrating multiple discovered discriminative factors into ML models and performing more fine-grained and efficient analysis, e.g., using attention mechanisms.

## REFERENCES

[1] Natalia Tomashenko, Brij Mohan Lal Srivastava, Xin Wang, Emmanuel Vincent, Andreas Nautsch, Junichi Yamagishi, Nicholas Evans, Jose Patino, Jean-François Bonastre, Paul-Gauthier Noé, and Massimiliano Todisco, "Introducing the VoicePrivacy Initiative," in Interspeech, 2020, pp. 1693–1697.
[2] Jose Patino, Natalia Tomashenko, Massimiliano Todisco, Andreas Nautsch, and Nicholas Evans, “Speaker anonymisation using the McAdams coefficient,” in *Interspeech*, 2021, pp. 1099–1103.

[3] Candy Olivia Mawalim, Shogo Okada, and Masashi Unoki, “Speaker anonymization by pitch shifting based on time-scale modification,” in *2nd Symposium on Security and Privacy in Speech Communication*, 2022, pp. 35–42.

[4] Priyanka Gupta, Gauri P. Prajapati, Shrishti Singh, Madhu R. Kamble, and Hemant A. Patil, “Design of voice privacy system using linear prediction,” in *2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)*, 2020, pp. 543–549.

[5] Lauri Tavi, Tomi Kinnunen, and Rosa González Hautamäki, “Improving speaker de-identification with functional data analysis of f0 trajectories,” *Speech Communication*, vol. 140, pp. 1–10, 2022.

[6] Brij Mohan Lal Srivastava, Mohamed Maouche, Md Sahidullah, Emmanuel Vincent, Aurélien Bellet, Marc Tommasi, Natalia Tomashenko, Xin Wang, and Junichi Yamagishi, “Privacy and utility of x-vector based speaker anonymization,” *IEEE/ACM Transactions on Audio, Speech and Language Processing*, vol. 30, pp. 2383–2395, 2022.

[7] Xiaoxiao Miao, Xin Wang, Erica Cooper, Junichi Yamagishi, and Natalia Tomashenko, “Speaker anonymization using orthogonal Householder neural network,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 31, pp. 3681–3695, 2023.

[8] Pierre Champion, *Anonymizing speech: evaluating and designing speaker anonymization techniques*, Ph.D. thesis, Université de Lorraine, 2023.

[9] Jixun Yao, Qing Wang, Pengcheng Guo, Ziqian Ning, Yuguang Yang, Yu Pan, and Lei Xie, “MUSA: Multi-lingual speaker anonymization via serial disentanglement,” *arXiv preprint arXiv:2407.11629*, 2024.

[10] Natalia Tomashenko, Xiaoxiao Miao, Pierre Champion, Sarina Meyer, Xin Wang, Emmanuel Vincent, et al., “The VoicePrivacy 2024 challenge evaluation plan,” *arXiv preprint arXiv:2404.02677*, 2024.

[11] Fuming Fang, Xin Wang, Junichi Yamagishi, Isao Echizen, Massimiliano Todisco, Nicholas Evans, and Jean-Francois Bonastre, “Speaker anonymization using x-vector and neural waveform models,” in *Speech Synthesis Workshop*, 2019, pp. 155–160.

[12] Sarina Meyer, Florian Lux, Julia Koch, Pavel Denisov, Pascal Tilli, and Ngoc Thang Vu, “Prosody is not identity: A speaker anonymization approach using prosody cloning,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2023, pp. 1–5.

[13] Michele Panariello, Francesco Nespoli, Massimiliano Todisco, and Nicholas Evans, “Speaker anonymization using neural audio codec language models,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2024, pp. 4725–4729.

[14] Xiaoxiao Miao, Xin Wang, Erica Cooper, Junichi Yamagishi, and N Tomashenko, “Language-independent speaker anonymization approach using self-supervised pre-trained models,” *arXiv preprint arXiv:2202.13097*, 2022.

[15] Yamini Sinha, Jan Hintz, Matthias Busch, Tim Polzehl, Matthias Haase, Andreas Wendemuth, and Ingo Siegert, “Why Eli Roth should not use TTS-systems for anonymization,” in *Proceedings of the 2nd Symposium on Security and Privacy in Speech Communication*, 2022, pp. 17–22.

[16] Gauri P. Prajapati, Dipesh K. Singh, Preet P. Amin, and Hemant A. Patil, “Voice privacy using CycleGAN and time-scale modification,” *Computer Speech & Language*, vol. 74, 2022.

[17] Natalia Tomashenko, Xin Wang, Emmanuel Vincent, Jose Patino, Brij Mohan Lal Srivastava, Paul-Gauthier Noé, Andreas Nautsch, Nicholas Evans, Junichi Yamagishi, et al., “The VoicePrivacy 2020 Challenge: Results and findings,” *Computer Speech and Language*, vol. 74, pp. 101362, 2022.

[18] Natalia Tomashenko, Xiaoxiao Miao, Emmanuel Vincent, and Junichi Yamagishi, “The first VoicePrivacy Attacker Challenge evaluation plan,” *arXiv preprint arXiv:2410.07428*, 2024.

[19] Brecht Desplanques, Jenthe Thienpontd, and Kris Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in *Interspeech*, 2020, pp. 3830–3834.

[20] Elena Bulgakova, Aleksei Sholohov, Natalia Tomashenko, and Yuri Matveev, “Speaker verification using spectral and durational segmental characteristics,” in *17th International Conference on Speech and Computer*, 2015, pp. 397–404.

[21] Kenichi Fujita, Atsushi Ando, and Yusuke Ijima, “Phoneme duration modeling using speech rhythm-based speaker embeddings for multi-speaker speech synthesis,” in *Interspeech*, 2021, pp. 3141–3145.

[22] Kenichi Fujita, Atsushi Ando, and Yusuke Ijima, “Speech rhythm-based speaker embeddings extraction from phonemes and phoneme duration for multi-speaker speech synthesis,” *IEICE Transcations on Information and Systems*, vol. 107, no. 1, pp. 93–104, 2024.

[23] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, “LibriSpeech: an ASR corpus based on public domain audio books,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2015, pp. 5206–5210.

[24] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlíček, et al., “The Kaldi speech recognition toolkit,” in *IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)*, 2011.

[25] Henry Li Xinyuan, Zexin Cai, Ashi Garg, Kevin Duh, Leibny Paola García-Perera, Sanjeev Khudanpur, Nicholas Andrews, and Matthew Wiesn, “HLTCOE JHU submission to the Voice Privacy challenge,” in *4th Symposium on Security and Privacy in Speech Communication*, 2024.

[26] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” in *International Conference on Machine Learning (ICML)*, 2023.

[27] Jaehyeon Kim, Jungil Kong, and Juhee Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” in *International Conference on Machine Learning (ICML)*, 2021, pp. 5530–5540.

[28] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu, “LibriTTS: A corpus derived from LibriSpeech for text-to-speech,” in *Interspeech*, 2019, pp. 1526–1530.