<figure><img src="image_1.png" /></figure>

HAL
open science

# On temporal constraints for deep neural voice alignment

Yann Teytaut

► To cite this version:

Yann Teytaut. On temporal constraints for deep neural voice alignment. Sound [cs.SD]. Sorbonne Université, 2023. English. NNT: 2023SORUS196. tel-04229423

HAL Id: tel-04229423

https://theses.hal.science/tel-04229423

Submitted on 5 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
DOCTORAL THESIS FROM SORBONNE UNIVERSITÉ

Spécialité **Informatique**

École Doctorale Informatique, Télécommunications et Électronique (Paris)

Sciences et Technologies de la Musique et du Son (STMS UMR 9912)

Institut de Recherche et Coordination Acoustique/Musique (IRCAM)

Équipe Analyse et Synthèse des Sons

Subject of the thesis:

ON TEMPORAL CONSTRAINTS FOR
DEEP NEURAL VOICE ALIGNMENT

Author:
Yann TEYTAUT

Supervisor:
Dr. Axel ROEBEL

Reviewers

Pr. Gaël RICHARD
Professor, Télécom Paris

Dr. Emmanouil BENETOS
Reader, Queen Mary University of London (QMUL)

Examiners

Pr. Jean-Pierre BRIOT
Research Director, LIP6 (CNRS/SU)

Dr. Emmanuel VINCENT – *Jury President*
Research Director, Inria Nancy - Grand Est

Dr. Rachel BITTNER
Research Manager, Spotify Inc.

Dr. Romain HENNEQUIN
Head of Research, Deezer

Dr. Chitralekha GUPTA
Research Fellow, National University of Singapore (NUS)
À Jeannette DAT
née Berthe-Dominiquette LADRIX

Celle qui a tant donné
que des mots ne sauraient suffisamment l'exprimer.

To Jeannette DAT
born Berthe-Dominiquette LADRIX

The one who has given so much
that no words can express it enough.

“Es pas l’ôme que ganha es lo **temps**.”

“Ce n’est pas l’homme qui gagne, c’est le **temps**.”

“It is not the human who wins, it is the **time**.”

Proverbe Occitan – Occitan Saying
# Preface

The research presented in this thesis results from a long history of passion for sciences and music, and a strong desire to work in their interdisciplinary field. In high school, I already knew that I wanted to enroll in the ATIAM<fnref n="1" /> (*Acoustique, Traitement du signal et Informatique Appliqués à la Musique*, i.e., sciences applied to music) Master’s program coordinated and housed at IRCAM<fnref n="2" />, the Institute for Research and Coordination Acoustics/Music, which I did upon graduation from the IOGS<fnref n="3" /> (*Institut d'Optique Graduate School*).

This is during the course of the ATIAM mandatory internship that I first discovered and got familiar with general music processing and the specific audio alignment task as Arshia CONT and Philippe CUVILLIER gave me the opportunity to work on their score following system, *Antescofo*<fnref n="4" />. Ever since, the open challenges and the numerous and various applications offered by the alignment task have kept captivating me.

As I was willing to experience a journey within the academic field, I applied for the voice-related PhD position proposed by Axel ROEBEL from IRCAM’s Analysis/Synthesis team, in the context of a new ANR (*Agence Nationale de la Recherche*) project. Entitled **ARS<fnref n="5" />**, this project deals with the **Analysis** and **tRansformation** of singing **Style** and, to this aim, the development of alignment algorithms dedicated to voice – and not music as I used to at Antescofo – were necessary.

This is the story the present document covers. This journey began on January 20th, 2020 and shall end on June, 30th 2023. The thesis has been defended on July, 7th 2023 at Ircam<fnref n="6" />.

This work has been fully funded by the French National Research Agency (ANR) project **ARS (ANR-19-CE38-0001-01)** and has also benefited from IRCAM’s **UPI “ISiS Voices”**.

<footnote n="1">¹https://www.atiam.ircam.fr/</footnote>

<footnote n="2">²https://www.ircam.fr/</footnote>

<footnote n="3">³https://www.institutoptique.fr/</footnote>

<footnote n="4">⁴https://www.metronautapp.com/fr</footnote>

<footnote n="5">⁵https://ars.ircam.fr/</footnote>

<footnote n="6">⁶https://www.youtube.com/watch?v=05RWU1_vZ9M&ab_channel=Ircam</footnote>
Abstract

To listen, to respond, to make coincide, to coordinate, to adjust, to follow, to adapt, to be
in unison, to synchronize, to align... The rich vocabulary dedicated to the correspondence of
human activities shows the importance of their temporal organization. Human communication,
multi-modal by nature, is fully concerned by this problematic since there exists a semantic
gap between oral locutions and their symbolic sequences: how to interpret a written message
without the vocal intonation? what performative style beyond a fixed musical score? This
thesis proposes to uncover the complex underlying relationships between the audio and symbolic
domains in order to reduce this gap through the fine study of the inherent temporality contained
in voice recordings. The voice alignment task lies at the core of this objective, as it aims
to determine the temporal occurrence of symbols that are assumed to be present in a voice
signal. This work notably focuses on the development of an acoustic model, ADAGIO, capable
of estimating such time-symbol links. Recent progress in deep learning have led to implement
ADAGIO as a deep neural network in a powerful generic formalism: the "Connectionist Temporal
Classification" (CTC). However, the great flexibility offered by CTC is undermined by its
intrinsic lack of guarantees for temporally accurate predictions. Therefore, the key contributions
of this research consist in reinforcing CTC with additional temporal constraints to improve the
quality of the inferred alignments. To do so, three ancillary tasks of (1) spectral content
reconstruction; (2) audio structure propagation; and (3) guided monotony are introduced and
induce a positive impact on the alignment between voices, texts, and notes. Then, ADAGIO
contributes to many practical applications via collaborations such as concatenative speech
synthesis or the study of expressive production strategies at play for both social attitudes
in speech and singing style in musical performances.
# Résumé

S'écouter, se répondre, faire se coïncider, se coordonner, s'accorder, se suivre, s'adapter, être à l'unisson, se synchroniser, s'aligner... Le riche vocabulaire dédié à la mise en correspondance dans le temps des activités humaines montre l'importance que revêt leur organisation temporelle. La communication humaine, multi-modale par nature, est pleinement concernée par cette problématique puisqu'il existe un écart sémantique entre les locutions orales et leurs séquences symboliques : comment bien interpréter un message écrit sans l'intonation vocale ? quel style performatif au delà d'une partition musicale figée ? Cette thèse se propose de révéler et expliquer les complexes relations entre les domaines audio et symbolique afin de réduire cet écart grâce à l'étude fine de l'inhérente temporalité contenue dans les enregistrements vocaux. Au coeur de cet objectif, se trouve la tâche d'alignement de voix qui vise à déterminer l'occurrence temporelle de symboles supposés présents dans un signal vocal. Ces travaux s'intéressent tout particulièrement au développement d'un modèle acoustique, ADAGIO, capable d'estimer de tels liens temps-symboles. Les récents progrès en apprentissage profond amènent à implémenter ADAGIO sous la forme d'un réseau de neurones profond dans un puissant formalisme générique : la "Classification Temporelle Connectioniste" (CTC). Cependant, la grande flexibilité offerte par la CTC est mise en défaut par son absence intrinsèque de garanties de prédictions temporellement précises. Les contributions clefs de cette recherche visent à renforcer la CTC par des contraintes temporelles supplémentaires pour améliorer la qualité des alignements déduits. Pour cela, trois tâches annexes de (1) reconstruction du contenu spectral, (2) propagation de la structure audio, et (3) monotonie guidée sont introduites et induisent un impact positif sur l'alignement entre voix, textes, et notes. Dès lors, ADAGIO contribue à de nombreuses applications pratiques au travers de collaborations telles que la synthèse vocale concaténative ou l'étude des stratégies de production expressives en jeu tant pour les attitudes sociales dans la parole que pour le style de chant dans des performances musicales.
Acknowledgments

One cannot write the story of a thesis alone. It is written in the ink of progressive learning, scientific achievements, and the human relationships that made it possible and that developed during its course. In this respect, it would be impossible to begin this document without thanking all the people who have counted for and contributed to this story.

My first thoughts go to my family, whose love and benevolence have never failed me, and have been the foundation of my personal and professional development. Thank you to my mother and father, Annie and Jean-Claude, to all my brothers and sisters — Loïc, Hélène, Marie-Émilie, Julie and Laura — and to Brigitte, Didier and Françoise. I owe you everything.

Secondly, I'd like to express my deepest gratitude to my thesis supervisor, Axel ROEBEL, who placed his trust in me for this project, was always available, offered invaluable advice and provided inspiring understanding. I'd also like to thank Guillaume DORAS, my "shadow" supervisor, who taught me so much while working with him. I'll never forget that. Thanks also to Arshia CONT and Philippe CUVILLIER for this year of research in their company, where I knew I had found my field. It was a pleasure to do this thesis in the Sound Analysis-Synthesis team and to meet so many passionate and exciting people at Ircam during these years of thesis or Master ATIAM.

Thanks to Clément, my research brother, for all those moments, discussions, coffees, escapes and laughter. It was an honor, and a real inspiration, to live this adventure with you. Thank you to our entire doctoral team for all the memories, all too many BMs, "All night— all night long", top chef debriefs, Tata Burgers, cultural and not-so-cultural outings and mutual support. In particular, Constance for that ATIAM night writing, all those laughs and moments of non-verbal communication that have given me so much joy over the years, Victor for his patience, his imitations, his calls and voicemails and his scandals that I wouldn't change for anything in the world, Nadia for her natural and luminous goodness, the Korea-Japan visios, and the many pieces of chocolate I ate from her, Claire for her words and her sensitivity, the "cups of water" and her tent that stayed in the office for months, Baptiste for the maroilles, the "easy" piano solos and his "bip bap boup" experiments, Paul for his infectious dynamism and the start of
my cinema career, *Valérian* for one of the rarest sincerity and kindness I have ever met and
for being the best ATIAM neighbor, *Vincent* for being an amazing cultural calendar and opera
companion, as evidenced by these numerous concerts together, and finally *Pablo* for his many
personal and professional advice.

I have a special place in my heart for all my office colleagues: *Gabriel*, *Hadrien*, *Lenny*, *Alice* and *Mathilde*. My timing couldn't have been better. I'd like to thank all of you, in your own way, for enabling me to move forward. A special thank you to *Lenny*, the quiet force, for our discussions and his commitment. I am looking forward to our next Pizza. Thanks to *Caio*, *Théo* and *Lucas* for doing their internships/graduate projects with me. Thanks to the students I taught. Thanks to all the Ircam staff, whose kindness I will always remember: *Deborah*, *Brigitte*, *Éric*, *Bruno*, the *PDS team* for all their hospitality, and to all the people I can't mention here for lack of space but not of heart.

An immediate thought goes to all the people I met at/thanks to the Institut d'Optique Graduate School (IOGS) and today dear friends. Thanks to *Fahim* and *Alice* for your harmonies, to *Perrine* and *Éric* for Rennes and Haute-Savoie, to *Bathilde*, *François* and *Yvan* for always being there at major events, to *Arianna* and *Lambert* for our increasingly unexpected exchanges of stories, to *François-Marie* and *Léna* for simply being themselves and *Jean-Baptiste* for always being such a great friend. Thanks to all the BrBi's and VK's — I can't name you all, but my heart's in it. Thanks to *Juliette* for my fabulous first years in Paris and this road trip my thoughts often go back to.

Thank you *Salomé*. You have done so much that it would be impossible to simply sum it all up. I will never forget the year we turned 26.

I'm deeply touched to see that, despite the years, relationships can disrupt and remain strong. Thanks to *Adeline* for her constant support over the years and Disneyland, to *Fanny* for her sincerity and sweetness. Thanks to my prépa friends — the whole team, *Élodie*, *Zakaria*, *Victor*, *Léa* and *Marie* — for so many memorable moments that I often think back on. “Teytaut” thanks you sincerely. Thank you to the friendships from primary school, junior high and high school that have remained intact — *Sophie* for all those stories to retrace, from Breaking Benjamin to “My Liege” and visits during lockdowns, *Estelle* for her infectious joy, those “very chill” moments and our bursts of laughter, *Caroline* for being the best partner in S, the TPE, the INSA stays and so much more, and *Anna*, very humbly, for the music and the piano.

These acknowledgements end with the person to whom this thesis is dedicated: *Berthe-Dominiquette* or *Jeanette*. Not all heroines are household names. I am eternally grateful for everything she has given me and passed on to me. The memory that comes to mind is reciting the multiplication tables with her, which undoubtedly enabled me to take my first step towards scientific research.
Remerciements

L'histoire d'une thèse ne s'écrit pas seul. Elle s'écrit à l'encre des apprentissages progressifs, des acquis scientifiques, et des relations humaines qui l'ont permise et qui se sont nouées pendant son déroulement. À ce titre, il serait impossible de commencer ce document sans remercier toutes les personnes qui ont compté pour et contribué à cette histoire.

Mes premières pensées vont à ma famille dont l'amour et la bienveillance, qui n'ont jamais fait défaut, ont été les fondements de mon épanouissement personnel et professionnel. Merci à mère et à mon père, Annie et Jean-Claude, à mes frères et mes soeurs de sang et de cœur – Loïc, Hélène, Marie-Émilie, Julie et Laura – ainsi qu'à Brigitte, Didier et Françoise. Je vous dois tout.

Ensuite, j'adresse ma plus grande reconnaissance à mon directeur de thèse, Axel ROEBEL, qui m'a fait confiance pour ce projet, a toujours été disponible, de conseils précieux, et d'une compréhension inspirante. Merci aussi à Guillaume DORAS, l'encadrant de l'ombre, qui m'a énormément apporté et appris. Je ne l'oublierai pas. Merci également à Arshia CONT et Philippe CUVILLIER pour cette année de recherche dans leur entreprise, où j'ai su que j'avais trouvé mon domaine. C'était un plaisir d'effectuer cette thèse dans l'équipe Analyse-Synthèse des Sons et de rencontrer tant de gens passionnés et passionnant à l'Ircam au cours de ces années de thèse ou de Master ATIAM.

Merci à Clément, mon frère de recherche, pour tous ces moments, discussions, cafés, évasions et rires. C'était un honneur, et une réelle inspiration, de vivre cette aventure avec toi. Merci à toute notre team doctorale pour les souvenirs gravés, les trop nombreux BM, les "All night- all night long", les debriefs top chef, les Tata Burgers, les sorties culturelles et moins culturelles et le soutien mutuel. Notamment, Constance pour cette nuit blanche rédactionnelle ATIAM, tous ces rires et moments de communication non-verbale qui m'ont apporté tant de joie au fil des années, Victor pour sa patience, ses imitations, ses appels et messages vocaux et ses scandales que pour rien au monde je ne changerais, Nadia pour sa bonté naturelle et lumineuse, les visios Corée-Japon, et les nombreux morceaux de chocolats que je lui ai mangés, Claire pour ses mots et sa sensibilité, les "cups of water" et sa tente restée des mois dans le bureau, Baptiste pour
les maroilles, les solos de piano jugés faciles et ses "bip bap boup", *Paul* pour son dynamisme communicatif et le début de ma carrière cinéma, *Valérian* pour l'une des plus rares sincérité et gentillesse que j'ai pu rencontrer et pour avoir été le meilleur voisin ATIAM, *Vincent* pour avoir été le meilleur agenda et compagnon culturel comme en attestent nos nombreux opéra, et enfin *Pablo* pour ses nombreux conseils personnels et professionnels.

Je pense et garde une émotion toute particulière pour tous mes collègues de bureau : *Gabriel*, *Hadrien*, *Lenny*, *Alice* et *Mathilde*. Je n'aurais pas pu mieux tomber. Je vous remercie, toutes et tous à votre façon, d'avoir permis mon avancée. Un merci tout particulier à *Lenny*, la force tranquille, pour nos discussions et son investissement. J'ai hâte de notre prochaine Pizza. Merci à *Caio*, *Théo* et *Lucas* d'avoir effectué leurs stages/projets de fin d'étude avec moi. Merci aux étudiants à qui j'ai pu enseigner. Merci, enfin, à tout le personnel Ircam dont la gentillesse ne sera pas oubliée : *Deborah*, *Brigitte*, *Éric*, *Bruno* et toute l'équipe PDS pour leur hospitalité, et toutes les personnes manquantes à cette liste faute de place mais pas de cœur.

Une pensée immédiate va à toutes les personnes rencontrées à ou grâce à l'Institut d'Optique Graduate School (IOGS) et aujourd'hui amis chers. Merci à *Fahim* à *Alice* pour vos harmonies, à *Perrine* et *Éric* pour Rennes et la Haute-Savoie, à *Bathilde*, *François* et *Yvan* pour leur présence sans faille aux grands évènements, à *Arianna* et *Lambert* pour nos échanges d'histoires toujours plus inattendues, à *François-Marie* et *Léna* d'être purement et simplement eux-mêmes et à *Jean-Baptiste* pour toujours avoir été un merveilleux ami. Merci à toutes les BrBi et VK – je ne peux pas tou-tes vous citer mais le cœur y est. Merci à *Juliette* pour mes premières années parisiennes qui ont été fabuleuses et ce *road trip* auquel souvent je repense.

Merci à *Salomé*. Tu as tant fait qu'il serait impossible de le résumer. Jamais je n'oublierai nos 26 ans.

Je suis profondément touché de voir que, malgré les années, les relations peuvent perturber et rester fortes. Merci à *Adeline* pour la constance de son soutien au fil des années et pour Disneyland, à *Fanny* pour sa sincérité et douceur. Merci à mes ami-es de prépa – toute l'équipe Élodie, Zakaria, Victor, Léa et Marie – pour tant de moments marquants auxquels souvent je repense. « Teytaut » vous remercie sincèrement. Merci aux amitiés de primaire, collège et lycée qui sont restées entières – Sophie pour tous ces récits à retracer, de Breaking Benjamin à « My Liege » en passant par les « motifs familiaux impérieux », Estelle pour sa joie communicative, ces moments « très chills » et nos éclats de rire, Caroline pour avoir été la meilleure binôme de S, le TPE, les séjours INSA et tant encore, et Anna, très humblement, pour la musique et le piano.

Ces remerciements se concluent par la personne la personne à qui cette thèse est dédiée : *Berthe-Dominiquette* ou *Jeanette*. Toutes les héroïnes ne sont pas connues du grand public. Pour tout ce qu'elle a su m'apporter et me transmettre, ma reconnaissance est éternelle. Le souvenir qui me vient concerne la récitation avec elle des tables de multiplication, me permettant d'effectuer, sans doute, mon premier pas vers la recherche scientifique.
# Table of Contents

<table><tr><td>Preface</td><td>3</td></tr><tr><td>Abstract</td><td>4</td></tr><tr><td>Résumé</td><td>5</td></tr><tr><td>Acknowledgments/Remerciements</td><td>6</td></tr><tr><td>1 Introduction</td><td>14</td></tr><tr><td>1.1 Thesis research scope</td><td>16</td></tr><tr><td>1.2 Summary of contributions</td><td>18</td></tr><tr><td>1.3 Dissertation outline</td><td>19</td></tr><tr><td>1.4 List of publications</td><td>21</td></tr><tr><td>I Voice alignment: context &amp; background</td><td>23</td></tr><tr><td>2 Context and applicative motivations</td><td>24</td></tr><tr><td>2.1 Human communication</td><td>25</td></tr><tr><td>2.1.1 Communication fundamentals</td><td>25</td></tr><tr><td>2.1.2 Writing and symbolic modality</td><td>27</td></tr><tr><td>2.1.3 Voice production and oral modality</td><td>30</td></tr><tr><td>2.1.4 Expressivity and interpretative style</td><td>32</td></tr><tr><td>2.2 Voice signal processing</td><td>33</td></tr><tr><td>2.2.1 Digital audio basics</td><td>33</td></tr><tr><td>2.2.2 Voice features</td><td>37</td></tr><tr><td>2.2.3 Speech vs singing comparison</td><td>40</td></tr><tr><td>2.2.4 Voice analysis and synthesis</td><td>41</td></tr></table>
TABLE OF CONTENTS

11

<table><tr><td>2.3</td><td>Transcription and alignment tasks</td><td>43</td></tr><tr><td>2.3.1</td><td>Definitions</td><td>43</td></tr><tr><td>2.3.2</td><td>Transcription algorithms</td><td>47</td></tr><tr><td>2.3.3</td><td>Alignment algorithms</td><td>49</td></tr><tr><td>2.4</td><td>Applicative motivations of voice alignment</td><td>51</td></tr><tr><td>2.5</td><td>Context and applicative motivations in a nutshell</td><td>53</td></tr><tr><td>3</td><td>Scientific background for voice alignment</td><td>54</td></tr><tr><td>3.1</td><td>Deep learning background</td><td>55</td></tr><tr><td>3.1.1</td><td>Data-driven approach</td><td>55</td></tr><tr><td>3.1.2</td><td>Essentials</td><td>55</td></tr><tr><td>3.1.3</td><td>Architecture design</td><td>60</td></tr><tr><td>3.2</td><td>Voice alignment problem statement</td><td>63</td></tr><tr><td>3.2.1</td><td>Formalization</td><td>63</td></tr><tr><td>3.2.2</td><td>Voice data</td><td>64</td></tr><tr><td>3.3</td><td>Traditional approaches to voice alignment</td><td>65</td></tr><tr><td>3.3.1</td><td>Dynamic Time Warping (DTW)</td><td>65</td></tr><tr><td>3.3.2</td><td>Pioneer acoustic modeling</td><td>67</td></tr><tr><td>3.3.3</td><td>Hidden Markov Models (HMM)</td><td>67</td></tr><tr><td>3.4</td><td>Neural approaches to voice alignment</td><td>69</td></tr><tr><td>3.4.1</td><td>Frame-wise classification</td><td>70</td></tr><tr><td>3.4.2</td><td>Attention mechanism</td><td>70</td></tr><tr><td>3.4.3</td><td>Connectionist Temporal Classification (CTC)</td><td>71</td></tr><tr><td>3.5</td><td>CTC-based voice alignment</td><td>72</td></tr><tr><td>3.5.1</td><td>CTC-based acoustic modeling</td><td>72</td></tr><tr><td>3.5.2</td><td>CTC-based decoding module</td><td>76</td></tr><tr><td>3.6</td><td>Scientific background for voice alignment in a nutshell</td><td>77</td></tr></table>

## II Contributions: time-constrained neural voice alignment

78

## 4 ADAGIO: An acoustic model for temporal voice alignment

79

<table><tr><td>4.1</td><td>Model history</td><td>80</td></tr><tr><td>4.1.1</td><td>Baselines</td><td>80</td></tr><tr><td>4.1.2</td><td>Early explorations and learnings</td><td>83</td></tr><tr><td>4.1.3</td><td>Acoustic model requirements</td><td>84</td></tr><tr><td>4.2</td><td>ADAGIO: Automatic Deep AliGnment of vOIce</td><td>85</td></tr><tr><td>4.3</td><td>Acoustic model for temporal voice alignment in a nutshell</td><td>88</td></tr></table>
5 Temporal constraints for alignment enhancement 89

5.1 The need for additional constraints 90

5.2 Temporal constraints for reinforcing alignment 91

5.2.1 Spectral reconstruction 92

5.2.2 Temporal structure propagation 94

5.2.3 Guided audio-symbol monotony 97

5.3 Multi-objective training 101

5.3.1 Worst-case scenario studies 101

5.3.2 Scaling the losses 103

5.4 Time-constrained acoustic modeling 104

5.5 Temporal constraints for alignment enhancement in a nutshell 104

III Outcomes: experiments & applications 107

6 Evaluations of deep voice alignment 108

6.1 Voice corpora 109

6.1.1 Speech datasets 109

6.1.2 Singing datasets 111

6.2 Evaluation procedure 112

6.2.1 Assessment metrics 112

6.2.2 Implementation details 114

6.3 Ablation study 116

6.4 Temporal alignment results 120

6.4.1 Voice-to-text alignment 120

6.4.2 Voice-to-note alignment 122

6.4.3 Robustness to transcription errors 123

6.5 Evaluations of deep voice alignment in a nutshell 128

7 Applications and collaborations 129

7.1 Concatenative singing synthesis (ISiS) 130

7.2 Temporal production strategies of vocal attitudes 133

7.3 A musicological pipeline for singing style analysis 135

7.3.1 Musicological context 136

7.3.2 Pipeline introduction 136

7.3.3 Case study – Taylor SWIFT’s “Blank Space” 140

7.4 Perspectives 144

7.5 Applications and collaborations in a nutshell 146
<table><tr><td>TABLE OF CONTENTS</td><td>13</td></tr></table>

<table><tr><td>8 Conclusion</td><td>147</td></tr></table>

<table><tr><td>8.1 Manuscript summary</td><td>148</td></tr></table>

<table><tr><td>8.2 Coming full circle: back to initial questions</td><td>149</td></tr></table>

<table><tr><td>List of Figures</td><td>152</td></tr></table>

<table><tr><td>List of Tables</td><td>156</td></tr></table>

<table><tr><td>Bibliography</td><td>157</td></tr></table>
Chapter 1

Introduction

"If you knew **Time** as well as I do," said the Hatter, "you wouldn't talk about wasting it."

The Hatter in Alice's Adventures in Wonderland

— Lewis Carroll

<figure><img src="image_2.png" /></figure>

*Time*.

A name, a concept, a notion so common and yet so disconcerting.

Not only has time obsessed artists, philosophers, or even the common, ordinary human sensitive to, concerned by, or simply aware of the finitude of the human condition, but it has also captivated scientists of various disciplines.

*Representing time* — whether objective (“physical”) or subjective (“lived”) — has especially stimulated many minds. It is a recurrent source of inspiration in arts, maybe as a necessity to express its omnipresence and the lack of means to control it. Because of its absence of any concrete *form*, time has been portrayed through common related objects (e.g., hourglass or clocks) by painters or embodied in characters by writers, as shown in Figure 1.1, often dealing with the underlying anguish of time flying and of the fleeting nature of life and memory.

The question of time representations is also of highest importance for scientists eager to study the evolution of physical phenomena — as most physical properties are not static but rather dynamic — and for general *ordering* between events. Such objectives have led to define a reference for time *measurements*, the *second*, and incited to gather *knowledge* about time itself.
<figure><img src="image_3.png" /><figcaption>Figure 1.1: Time representations in arts.</figcaption></figure>

The probably never-ending quest for the deep understanding of time has repeatedly revealed its special nature. Tracing back its history, questioning its origin in astrophysics, trying to model its evolution or report on its impacts on physical observations with mathematical laws — time was shown to be, and still remains, an intriguing concept full of secrets.

Indeed, among the four dimensions humans are familiar with — three spatial coordinates, usually denoted with the Cartesian $(x, y, z)$ or Polar $(r, \theta, \varphi)$ coordinate systems, and time $t$, time is the only dimension that cannot be “touched”, crossed forward *and* backward, but rather that humans inexorably experience. The noticeable theories of Special Relativity and General Relativity, focusing on time-space relationships, demonstrate that time measurements are not even universal but are subject to the quite counter-intuitive *time dilation* effect such that time passes at different rates for two clocks (observers) in relative motion or submitted to different gravitational potentials — perhaps as if the very essence of time could not be universally determined but should rather maintain some level of inaccessibility.

This manuscript, far from considering time *per se* as a core topic of study, rather proposes to investigate some of its implications, which are ubiquitous in one's daily life. Many signals, as observed quantities in the surrounding world, are non-stationary such that their observations vary with the passing of time. Among the wide variety of existing signals with this property, this thesis clearly focuses on characterizing audio observations — taking various practical shapes between sounds, music, speech and singing, each with their intrinsic and fascinating diversity.

Notably, it is undeniable that time plays a crucial role in the way humans interact with one another — and particularly through audio signals. The very vocabulary of verbal communication but also of musical exchanges, is highly correlated with temporality: *listening to*, *responding*, *following*, *coordinating*, *adapting*, *synchronizing*, *aligning*... A deeper look at this temporal organization may thus reveal some of the characteristics at the heart of human communication.
The means of communication are, for humans, fundamentally *multi-modal* in the sense that the *same* idea can be expressed by *several media* and be represented in *different formats*. For example, a primarily way of expression is the *voice* which allows one to talk or sing by emitting sounds. But these locutions are very often associated with some underlying representation(s) of *symbolic* nature. Reciting a poem, reading a story out loud, giving a speech or a lecture, usually imply the existence of *written texts*. Similarly, performing a musical piece, acting in a musical or singing in an opera may be dependent on *music scores* and music theory.

Yet, albeit originally characterizing the same *message* to communicate, transitions between these diverse modalities are far from being straightforward. There are, indeed, for a single symbolic text, many ways of expressing it through the voice which can lead, for example, to humorous situations or misunderstandings when only a written message is available but the voice, intonation and *temporality* are missing. In the same way, a music score, although fixed, allows for multiple interpretations so that each concert and performance turns out to be unique.

The *semantic gap* precisely relates to these differences between two descriptions of a message or concept in different representations – e.g., sounds, letters or symbols. In the very words of Andreas HEIN (2010), the semantic gap corresponds to “*the difference in meaning between constructs formed within different representation systems*”.

In the context of the **ARS** project – **A**nalysis and **t**ransformation of **S**inging style, ANR-19-CE38-0001-03 (http://ars.ircam.fr) –, which has fully funded this thesis, one of the core objective was to develop means that allow describing the singing style in popular music. Given the multi-modal nature of music and especially recordings featuring singing voice, as mentioned, the idea to investigate algorithms capable of jointly and comparatively manipulating symbolic sound descriptions with arbitrary alphabets and audio recordings, saw the light of day.

By keeping these algorithms as generic and general as possible, they would not only allow studying singing performances but also, in a way broader scope, allow bridging the semantic gap – notably by exploiting the inherent temporality contained in voice signals.

## 1.1 Thesis research scope

Therefore, the scope of this thesis is to uncover the complex underlying temporal relationships between several representations conveying communication messages, with a focus on *voice* signals which, by nature, feature strong variations over time. The main research problem at the core of this challenge is *temporal voice alignment* or *synchronization*. It is studied as a goal for designing systems capable of revealing temporal information from voice data, and as a means for the analysis and synthesis of voice signals and their intrinsic expressivity.

Concretely, the *alignment* task aims to determine the precise time positions of a symbolic sequence, whose symbols are known to constitute the content of a given audio signal. The problem requires an audio file and a symbolic sequence as inputs. Note that the alignment is distinct from *recognition*, which consists in finding the symbols present in an audio signal.
Deep Neural Networks (DNN) are worthy candidates to address the alignment problem as they allow learning complex mapping functions between different domains (here, audio and symbolic) entirely informed by data. To fully benefit from this *data-driven* approach, no task-specific contextual information is provided besides the ones that can be directly inferred from the inputs at disposal (i.e., voice signals and symbols).

A central objective is to develop an algorithm working for arbitrary symbol sets such as characters, phonemes, or even music notes, that does not need any specific information about the problem at stake. This way, the system can handle different languages, speakers, audio and voice qualities (speech and singing), and ideally arbitrary audio length. The main idea in this thesis is that the integration of general (i.e., non-task specific) *temporal constraints* directly in the design and data modeling stage of DNNs must result in a better alignment accuracy, given the primary importance of time in the processes discussed.

Addressing such a research problem requires to raise and tackle practical and/or research questions. Three of them (and their corresponding sub-questions) are answered in this document.

**Question 1 (Q1).** *Temporal voice alignment – what?*

* What is the temporal alignment task in general and for voice?

* What kind of representations can be used to align voice data?

**Question 2 (Q2).** *Temporal voice alignment – how?*

* How to develop a system for the temporal alignment of voice?

* How to extract suitable temporal information from audio to reinforce alignment accuracy?

**Question 3 (Q3).** *Temporal voice alignment – why?*

* Why is temporal alignment of interest in various research communities?

* Why does temporal voice alignment lead to numerous research applications?

**Q1** – *what?* is a practical question necessary to get familiar with the task addressed in this thesis. **Q2** – *how?* is the research question that aims at investigating ways to achieve temporal voice alignment. **Q3** – *why?* is another practical question thought to better understand the need for temporal alignment, all in all giving motivation based on examples demonstrating the potential of such algorithms.
## 1.2 Summary of contributions

Subsequent to these interrogations, this thesis has proposed solutions and led various studies for their exploration and answering. This section provides a summary of these contributions and applications.

### ADAGIO – A system for the Automatic Deep Alignment of vOIce

The core proposal of this work is the development of **ADAGIO** – a system dedicated to the Automatic Deep Alignment of vOIce. It is an acoustic model, i.e., a model capable to temporally estimate and represent the symbolic content associated with an audio recording. Built upon recent advances in deep learning for voice analysis research, ADAGIO is implemented as an end-to-end Deep Neural Network (DNN) in a Connectionist Temporal Classification (CTC) training scenario. Its generic nature allows aligning voice signals with symbolic sequences as long as they describe successive events occurring in the audio.

### Temporal constraints for acoustic model training

The second main proposal of this thesis is the definition of additional temporal constraints integrated during the training phase of the acoustic model, with the aim to better capture the temporal relationships and lead to an overall higher alignment quality. These constraints are:

* **Spectral content reconstruction** with the claim that a precise temporal reconstruction of the relevant spectral information (i.e., spectral envelope or excitation) is heavily dependent on the voice alignment;

* **Temporal structure preservation** with the claim that shared similarity patterns, informing on the local temporal structure, are expected to be found in the original voice signals and in the alignment predictions;

* **Guided time-symbol monotony** with the claim that a pertinent alignment must highlight a clear monotonic path between the voice data sequences.

### Validation on voice-to-text and voice-to-notes alignments

As a third contribution, experiments are conducted to evaluate ADAGIO enhanced with the above-mentioned temporal constraints and validate its generic nature and potential. Voice-to-text alignment is tackled at various granularities. Word-level and syllable-level alignment are obtained by aligning *graphemes*, and phonetic alignment through the alignment of *phonemes*.
Two evaluation datasets, one for speech (20 mn audio with manual word-level annotations) and one for singing voice, are proposed to the community. The robustness of the model to music instrumentals and even imperfect transcripts is demonstrated through a series of quantitative evaluations. Then, the voice-to-*notes* alignment problem is also tackled and the feasibility of aligning voice music scores with audio (despite background music) with ADAGIO is confirmed. However, this note alignment task can still be improved at the time of writing.

*Collaborative applications to voice analysis and synthesis*

Last but not least, ADAGIO has been employed in many practical collaborations in diverse research contexts such as automatic speech analysis, singing voice synthesis and musicological studies. The different applicative contexts in which ADAGIO has already been used are listed below:

* **Concatenative singing voice synthesis** with the integration of new voices for ISiS (Ircam Singing Synthesizer) thanks to their phonetic alignment with ADAGIO;

* **Production strategies of social attitudes** with a core component, offered by the phonetic alignment of expressive speech with ADAGIO, dedicated to study the temporal aspects involved when speakers aim at conveying precise social intentions;

* **Musicological singing style analysis** with the development and future online release of a complete pipeline allowing musicologists to study expressivity in sung performances notably thanks to *both* syllables *and* notes alignments made possible by ADAGIO;

* **Very long audio alignment** for which ADAGIO, as an acoustic model, is coupled with a linear memory decoding module, to synchronize very long recordings (i.e., several hours, e.g., entire music playlists or audiobooks) with their wide text transcripts.

1.3 Dissertation outline

The Figure 1.2 gives a global overview over the organization of the document. It also shows how the different parts and chapters relate to the raised practical and research questions. Concretely, besides this introduction (Chapter 1) and an overall conclusion (Chapter 8), this work is divided into three parts, each composed of two chapters.

* Part I — Voice alignment: context & background

In this part, all the notions necessary for a complete reading of the whole manuscript are introduced by getting familiar with the task of temporal alignment of vocal signals. In this perspective, the practical motivations of a research work focused on temporal voice alignment and an extensive literature review are presented. More precisely:
<figure><img src="image_4.png" /><figcaption>Figure 1.2: Outline of the dissertation and its relationships with the practical and research questions raised by temporal voice alignment.</figcaption></figure>

— Chapter 2: *Context and applicative motivations* explores the notions related to human communication, and in particular vocal communication, to introduce audio time-frequency representations and symbolic sequences manipulated in this work (Q1 – *what?*). Following their presentation, the task of their temporal alignment is defined (Q1 – *what?*), as well as the opportunities offered by such synchronizations in the voice community (Q3 – *why?*), thus making the link between the scientific disciplines mentioned and the technical tools to be presented in the next chapter.

— Chapter 3: *Scientific background for voice alignment* proposes a systematic review of the state of the art associated with voice alignment by presenting mathematical concepts and models from the literature. The specific task of voice alignment is formalized (Q1 – *what?*) and then the various strategies used for its resolution are presented (Q2 – *how?*). Recent approaches, including this thesis, are based on artificial intelligence, which motivates a summary of the key notions of deep learning. This work will concretely adopt a Connectionist Temporal Classification (CTC) training strategy whose specificities are detailed.
*   Part II — Contributions: time-constrained neural voice alignment

This part introduces a deep neural voice alignment algorithm, denoted ADAGIO, which, together with multiple losses imposing temporal constraints, is the main contribution of the thesis. This part, consequently, fully addresses the research question **Q2 – how?** More precisely:

-   Chapter 4: ADAGIO, an acoustic model for temporal voice alignment exposes the acoustic model at the heart of this work – ADAGIO, a system dedicated to the “Automatic Deep Alignment of Voice”. Its convolutional architecture and comparisons with other existing and previous systems are made.

-   Chapter 5: Temporal constraints for alignment enhancement aims at improving ADAGIO through the introduction of additional temporal information. Supplementary modeling objectives are proposed to reinforce the temporal coherency of the predictions and alignment quality. These contributions include a temporal audio reconstruction, propagation of the temporal structure, and temporal-sequential monotony assurance.

*   Part III — Outcomes: experiments & applications

This part, finally, focuses on the results of this thesis through the evaluations of the proposed models in comparison to relevant baselines, demonstrating the interest of the contributions, and by putting ADAGIO into practice. More precisely:

-   Chapter 6: Evaluations of deep voice alignment conducts a set of experiments evaluating the accuracy of the ADAGIO algorithm, an essential step for determining the relevance of any synchronizer (**Q2 – how?**). This evaluation procedure is applied on several cases of practical interest (**Q3 – why?**). Performance comparisons with relevant baseline algorithms demonstrate the robustness of ADAGIO and the proposals defended in this thesis.

-   Chapter 7: Applications and collaborations is dedicated to the practical and scientific contributions made possible by ADAGIO, by means of describing applications done in collaboration with other researchers (**Q3 – why?**).

Chapter 8 features an overall conclusion to this work by summarizing the content of the manuscript and associated contributions as well as eventually answering to the three practical and research questions.

## 1.4 List of publications

The list of research papers that have been written during this PhD and accepted for publication in peer-reviewed national/international conferences and journal is presented below.
<table><tr><td>(TEYTAUT and ROEBEL, 2021)</td><td><b>Y. TEYTAUT</b> and A. ROEBEL. Phoneme-to-audio alignment with recurrent neural networks for speaking and singing voice. In Proceedings of Interspeech 2021, pages 61–65. International Speech Communication Association; ISCA, 2021.</td></tr><tr><td>(LOISEAU et al., 2022)†</td><td>R. LOISEAU, B. BOUVIER, <b>Y. TEYTAUT</b>, E. VINCENT, M. AUBRY, and L. LANDRIEU. A model you can hear: Audio identification with playable prototypes. In 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), 2022.</td></tr><tr><td>(SALAIS et al., 2022)</td><td>L. SALAIS, P. ARIAS, C. LE MOINE, V. ROSI, <b>Y. TEYTAUT</b>, N. OBIN, and A. ROEBEL. Production strategies of vocal attitudes. In Interspeech 2022, pages 4985–4989. ISCA, 2022.</td></tr><tr><td>(TEYTAUT et al., 2022)</td><td><b>Y. TEYTAUT</b>, B. BOUVIER, and A. ROEBEL. A study on constraining connectionist temporal classification for temporal audio alignment. In Interspeech 2022, pages 5015–5019. ISCA, 2022.</td></tr><tr><td>(DORAS et al., 2023)</td><td>G. DORAS, <b>Y. TEYTAUT</b>, and A. ROEBEL. A linear memory ctc-based algorithm for text-to-voice alignment of very long audio recordings. Applied Sciences, 13(3):1854, 2023.</td></tr><tr><td>(TEYTAUT et al., 2023)</td><td><b>Y. TEYTAUT</b>, A. PETIT, C. CHABOT-CANET, and A. ROEBEL. A musicological pipeline for singing voice style analysis with neural voice processing and alignment. In Journées d’Informatique Musicale (JIM 2023), Saint-Denis, 2023.</td></tr></table>

Remark: the publication marked with †, as a collaborative and fruitful side project between vision and audio communities, is *not* concerned with temporal voice alignment – although it was interested in defining relevant temporal sound transformations to address speaker/voice identification. Its content, therefore, will *not* be discussed in this manuscript.
# Part I

## Voice alignment: context & background
Chapter 2

Context and applicative motivations

"Music is nothing else but wild sounds civilized into *Time*
and tune."

— Thomas FULLER

This chapter presents the general context associated with this thesis, inscribing this research
into larger and more specific fields of study, and the main practical motivations behind the
development of temporal alignment algorithms for voice processing.

The starting point of this thesis is the *human communication* theory, and in particular vocal communication, as exposed in section 2.1. It allows introducing two major modalities of messages that human exchange when interacting, that are in line with this research: symbolic sequences and voice signals. Notably, representing the temporal evolution of the parameters contained in such signals is a mandatory step to further focus on temporal aspects. This requires *voice signal processing* background, given in section 2.2. Then, the two *transcription and alignment* tasks, at the heart of the manuscript as they link symbolic and audio representations, are deeply introduced in section 2.3. Finally, section 2.4 proposes an overview of the *practical applications* offered by the temporal synchronization between voice and symbolic information, motivating a research work in this domain. The chapter is summarized in section 2.5.

<figure><img src="image_5.png" /></figure>
### 2.1 Human communication

*Communication* is an essential aspect of human life and has been crucial in the evolution of humans as a species. From hearing the voice of their mother before they are born to daily life implications – even up to summarizing years of research in PhD manuscripts –, humans communicate to connect and maintain relationships with one another, share information and knowledge. It is by communicating that humans bring about a common interpretation of the world in which they live, their environment, which makes possible their collaboration in the broadest sense. As a result, the study of human communication is of highest interest.

To this aim, a mathematical formalization of the concept of communication, introduced by SHANNON (1948) in the form of the *communication theory*, is first presented. It allows defining the generic and fundamental notion of *message* necessary in all future investigations. A message can contain information of numerous natures, yet this thesis is particularly interested in *symbolic* and *oral* messages whose brief writing history and involved physical production mechanisms, respectively, are further exposed. In addition to this core part, humans are able to induce and interpret many variations when communicating a message – a short overview on this *expressivity* is proposed.

#### 2.1.1 Communication fundamentals

*SHANNON's theory of communication*

A theoretical approach to understand communication can be traced back to 1948, when Claude SHANNON introduced the concept of the communication chain – a model for comprehending the process of transmitting and receiving information, also known as the SHANNON *model of communication* (SHANNON, 1948).

According to this theory, the communication process involves six elements, as depicted in Figure 2.1. A sender emits a message *m* which is encoded by an encoder *E*, transmitted over a communication channel and eventually decoded by a decoder *D* and intercepted by a receiver. If communication has been successfully achieved, the decoded message (*D* ◦ *E*)(*m*) must be exactly the original message *m*.

However, this cannot be systematically guaranteed as there are sources of *noise* along the transmission chain, blurring information throughout the propagation of the message, and altering the effectiveness of the communication process. Noise can take many forms from physical (e.g., interferences on a telephone line) to psychological (e.g., misunderstandings or distractions preventing the receiver from fully comprehending the message).

The notion of *noise* is not static and highly depends on the message that is transmitted. For example, in different contexts, musical background can be considered as noise (e.g., hearing someone talking in a concert) or as the key information to decode (e.g., focusing on the construction of a music piece).
<figure><img src="image_6.png" /><figcaption>Figure 2.1: SHANNON’s communication chain.</figcaption></figure>

*Communication message*

The message *m* is a core element of the communication chain as it contains the information one wants to transmit. Figure 2.2 proposes a succinct overview of what messages are made of. It can be summarized as follows:

* On the one hand, there is the central part of the message, its *content* and its *denotation*, which precisely relates to its strict meaning, *i.e.*, the sense and ideas that are to be communicated. This implies the existence of an underlying system — a *language* — defining a set of rules, admitted and shared by a group of people that allow to state understandable and meaningful messages. Human linguistics (COOK, 2003) and music theory (BESSON and SCHÖN, 2001) are examples of languages of interest in this work.

* On the other hand, there is always *meta*-information in a message, which is divided into two main categories: (1) the *para*-linguistic which *connotes* the strict meaning through, *e.g.*, expressivity and emotions; and (2) the *extra*-linguistic which places the speech in a context (*e.g.*, geo-socio origin, age, communication style). These *extra-* and *para-linguistic* aspects of communication will have their importance for applicative motivations — they are therefore highlighted in *section 2.1.4*.

*Message modalities in voice research*

Much like the information it contains, the nature of a message is highly variable. For a given language, indeed, a message can be expressed in diverse *modalities*.

In this thesis with a focus on the relationships between audio and texts, two well-known human communication modalities are ubiquitous: *oral* and *symbolic* modalities.
<figure><img src="image_7.png" /><figcaption>Figure 2.2: Content of communication messages. Illustrations from https://openmoji.org/</figcaption></figure>

*Symbolic* messages are dependent on the existence of a *writing system* and are the focus of section [2.1.2]. *Oral* messages, such as speech and singing, are produced by humans thanks to their voice via physical body mechanisms that are presented in section [2.1.3].

For the sake of completeness, even though beyond the focus of this thesis, it is worth mentioning that other types of modalities are studied in their associated literature — e.g., facial motion or body gesture (FARES et al., 2021).

### 2.1.2 Writing and symbolic modality

*Writing* is a representation of a language through the use of visual *symbols* such as pictograms, figures, letters or characters. It is a system that allows people to communicate messages and transmit ideas in a *symbolic* form — through written documents. The ensemble of all symbols constituting a writing system is known as an *alphabet* $\mathcal{A}$.

#### *A brief history of writing*

The history of writing (FISCHER, 2003) dates back to ancient civilizations, where it was a means of recording, preserving as well as transmitting information. Thanks to writing, humans were able to share notions beyond time and space, enabling the exchange of knowledge and the growth of civilizations.
<figure><img src="image_8.png" /></figure>

time

English

時間

Mandarin

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

وقت

Arabic

χρόνος

Greek

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

час

Ukrainian

시각

Korean

<figure><img src="image_13.png" /><figcaption>Figure 2.3: Writing systems of some modern languages. The meaning is “*time*” for all words.</figcaption></figure>

One of the earliest known forms of writing is *cuneiform*, which involved pressing a reed stylus into clay tablets to create symbols representing objects, words, and ideas. It was used by the Sumerians in ancient Mesopotamia (4000 BCE) to keep track of a variety of content (e.g., religious texts, legal codes, etc.). In ancient Egypt (3000 BCE), writing was also an important tool of communication and record-keeping. The ancient Egyptians relied on *hieroglyphics* — a system of writing based on pictorial symbols representing words and ideas. Hieroglyphics were primarily inscribed on stone or other durable materials, such as tombs and temple walls.

With the passing of time, writing systems also evolved and adapted to the needs of different languages (SCHMANDT-BESSERAT, 2014). For example, the Phoenicians (2000 BCE) developed an alphabet leading to a simpler and more efficient system of writing: individual symbols were meant to represent *sounds* rather than whole *words* or *concepts*. The Phoenician alphabet served as the basis for the Greek alphabet and other upcoming writing systems.

Throughout history, writing played a crucial role in the development and spread of language and culture – and music notation fully inscribes itself in these considerations (STRAYER, 2013). Writing, though, is not an inherent part of a language: most spoken languages, actually, have no standardized written forms or no written system at all. For illustration purposes, Figure 2.3 displays some contemporary writing systems.

*Symbolic modality of communication*

The research presented in this manuscript deals with languages that do have a well-defined
*alphabet A* to write documents based on linguistic or grammar rules. This includes recent
versions of unsigned living languages (e.g., English, French, Greek, etc.) and classical musical
notation and notes from music theory.
<figure><img src="image_14.png" /><figcaption>Figure 2.4: Singing music sheet features both text and note as symbolic information. Taylor SWIFT’s “Blank Space”, 1989 (2014), measures 7–10 (15:30–25:30). Transcription done by Antoine PETIT. Courtesy of Antoine PETIT (PETIT, 2022).</figcaption></figure>

In a *textual* context — A *sequence* (or *text*) is defined as an ensemble of one or more of written message(s). Characters from the alphabet $\mathcal{A}$, which are referred to as *graphemes*, allow to constitute *syllables* or *words*, that compose *sentences*, leading to *paragraphs*. The *Latin* alphabet $\mathcal{A}_l$ will be primarily used as common to many spoken languagues. It contains all the basic latin characters/graphemes (a, b, c, etc.) and is augmented with the digits (0, 1, 2, etc.) as well as a *space* ($\emptyset$) to separate words. All in all, it is defined as a set of 37 symbols, that is

$$ \mathcal{A}_l = \{a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z, \\ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, \emptyset\}. \qquad (2.1) $$

In a *musical* context — A *sequence* (or *melody* or *melodic line* or *sheet*) is similarly defined as a succession of *notes* from the alphabet. Musical words (*e.g.*, *chords*), sentences and paragraphs can be analogically defined, but will not be necessary in this work. The *note alphabet* $\mathcal{A}_n$ used this manuscript will be based on the piano, hence including the natural notes and their altered versions, *i.e.*, 12 semi-tones per octave — thus resulting in 88 classes from $A_0$ to $C_8$. It is also augmented with a silence token denoted 0.

Interestingly, these two symbolic natures (*e.g.*, text and notes) can be entangled as shown in Figure 2.4. Indeed, this sheet music features a text and singing melody that are expected to “happen” simultaneously in such a way that each part of a word (*i.e.*, a *syllable*) is associated with a note or even hold on several notes in the case of *melisma* — *e.g.*, see “*play?*” on Figure 2.4. It becomes a perspective of research to choose whether to study the text, the notes, or both, which justifies their previous mutual introduction. From a general point of view, both these symbolic information are contained in the sound that a singer would produce. This calls for more expertise on another modality: *orality*.
### 2.1.3 Voice production and oral modality

Besides symbolic modalities, humans are able to exchange communication messages by generating sounds with their voice and listening to voice sounds from others. The voice is intrinsically speaker-dependent but the production mechanisms of these *oral* messages are shared by all humans. They are therefore briefly detailed in the following.

#### Acoustic phonetics

The range of ways to articulate sounds when speaking a language is determined by the anatomy of the human body and cannot be expanded upon. This means that all humans have the same potential for making sounds, and sounds from the world's languages rely on similar body configurations. As a result, there exist categories of sounds that can be distinguished based on the way they are produced (*articulation*) or their characteristics (*acoustics*). Each language can use a combination of these sounds to form its own inventory of *phonemes* (STEVENS, 2000).

Phonemes are the building blocks of any spoken language as they are defined as the smallest units of sound that can alter the meaning of a word when changed or swapped out, e.g.,

$$ 
\begin{array}{lclcl}
\text{time} & \mapsto & /t/ & /a/ & /i/ & /m/ \\
\text{dime} & \mapsto & /d/ & /a/ & /i/ & /m/
\end{array}
 $$

Understanding how phonemes can be *voiced* — and become sounds — requires further information on the vocal apparatus.

#### Production mechanisms

The *vocal apparatus* is the physical system humans are born with that is capable to turn air flow into sounds. As summarized in Figure 2.5, it can be described by two main elements:

* First, the *voice generator* whose vocal folds, in the case of harmonic sounds, turns air flow from the lungs into periodically spaced pulses, resulting in an excitation with a given tone — or Fundamental Frequency (F0);

* Second, the *vocal tract* composed of the tongue, nasal cavity and lips, which is responsible for creating the “color” of the voice — its *timbre*.

Phonemes are classified in two upper categories, namely *vowels* and *consonants*. Consonants (e.g., /t/, /m/) are articulated with complete or partial closure of the vocal tract. Vowels (e.g., /a/, /e/, /i/, /o/, /u/, /y/) are generated without any restriction of the vocal tract, resulting in periodic air pulses leading systematically to harmonic sounds. That is why vowels have a special role in speech and singing analysis: they carry the tone or the melody. In singing, the notes are held on the vowels of each syllable (SUNDBERG and ROSSING, 1990).
<figure><img src="image_15.png" /><figcaption>Figure 2.5: Voice production mechanisms. Courtesy of Léane SALAIS (SALAIS et al., 2022).</figcaption></figure>

The form of the vocal tract changes with respect to time as its constituent elements are dynamically in motion. Therefore, in order to capture the time-dependent fluctuations of the phonemes, one must rely on Time-Frequency Representations (TFR) — they will be extensively presented in the next *section* [2.2].

### Formants

The *formants* are the main resonances of the vocal tract. If these resonances are sufficiently dense, formants may be observed on TFR, in regions surrounding the frequency peaks (*partials*) — *i.e.*, maxima of magnitude (energy). Depending on the phonetic context, there can be one or several formants at a given time as depicted in Figure 2.6.

In the case of vowels, two formants are usually detected and analysed. The first formant $F_1$ is linked to the aperture of the vocal tract while the second formant $F_2$ is related to the position of the tongue. Singing voice (especially in opera) also has a high third formant $F_3$ referred to as the *singing formant* and located around 3kHz (SUNDBERG, 1974).

In the case of unvoiced excitation (or unvoiced consonant), only one formant $F_1$ can be expected as there are no vibrations of the vocal cords generating harmonic structure.

Phoneme identification can be based on formant analysis — *e.g.*, the two first formants $F_1$ and $F_2$ have been used to discriminate between vowels by defining the vowel diagram (HELLWAG, 1886). However, formants alone are not sufficient to fully describe voiced phonemes. There are many aspects, characterizing one's expression or interpretation, that can shape the sound of a spoken or sung phoneme. These are the focus of upcoming paragraphs.
<figure><img src="image_16.png" /><figcaption>Figure 2.6: Visualization of phonemes and *F*₁/*F*₂ formants on Time-Frequency Representation. Courtesy of Léane SALAIS (SALAIS et al., 2022).</figcaption></figure>

### 2.1.4 Expressivity and interpretative style

Humans convey emotions when they talk; singers integrate feelings during a performance. This *expressivity* is precisely what makes a voice sound *natural* and the reason voice goes beyond a simple succession of phonemes and formants. The very simple fact to rise or lower the tone of a voice creates an *intonation* (COLLIER and HART, 1975) that shares a state of a mind.

Many studies have been dedicated to emphasize the outstanding ability humans have to communicate social signals when speaking — from *emotions* and their identification (LE MOINE and OBIN, 2020) to social *attitudes* (WICHMANN, 2000)<fnref n="1" /> conveyed in speech utterances (LE MOINE et al., 2021). Interestingly, analogous phenomena have been highlighted between musicians (JUSLIN, 2019) for, e.g., humor in music (RODRIGUEZ et al., 2021).

Vocal expressivity is known to induce significant changes in speech characteristics. For instance, the smiling process leads to higher formants *F*₁ and *F*₂ during both smiled speech production (ARIAS et al., 2018b) and perception (PONSOT et al., 2018) to the extent that auditory smiles communicated through formant frequencies can trigger emotional reaction in listeners (ARIAS et al., 2018a). Facial expression of disgust also modifies the formants (CHONG et al., 2018).

In a similar manner, singers carry their very own personal *singing* or *interpretative style* that have driven musicological research in various genres, e.g., French chanson (CHABOT-CANET, 2020b; CHABOT-CANET et al., 2020) or Rap (MIGLIORE and OBIN, 2018). Key aspects, like the *vibrato*, will be mentioned when comparing speech and singing in section 2.2.3.

Like recent collaborative work (ARDAILLON et al., 2016), an applicative motivation of this thesis is to uncover some temporal facets of (1) speech production and (2) singing expressivity in a musicological context, and throw them into relief. This will be detailed in section 2.4.

<footnote n="1">¹Attitudes are concerned with a speaker's social intention while emotions refer to a speaker's affective state.</footnote>
### Section summary – Human communication

Humans exchange information with one another through communication messages that, for a given language, have two main modalities: (1) symbolic messages that are dependent on a writing system and an alphabet of symbols $\mathcal{A}$; and (2) oral messages that are based on production mechanisms – from the vocal folds generating periodic tones to the vocal tract creating the formants (timbre of voiced phonemes). In addition to these physical aspects, voice has a wide range of expressivity that comes into play.

## 2.2 Voice signal processing

The study of a physical phenomenon can be described as one or several quantities, i.e., physical variable(s) depending on time and/or space. Extracting information from the observations at our disposal often requires to look closely at these temporal or spatial aspects. It is common to refer to a temporal evolution as signal and to a spatial evolution as image.

In this context, signal processing is a wide scientific field (RABINER and GOLD, 1975) dedicated to extract, manipulate and transform information contained in signals. While there exist many natures of signals (e.g., electric measurements, temperature values, weather maps, etc.), the research in this thesis is exclusively focused on sounds. Indeed, voice, audio and music are sound signals and, as such, can be manipulated with signal processing tools (MÜLLER, 2015).

This section naturally exposes the main notions necessary to study voice and audio from a signal processing perspective. Once the basics from digital audio are presented, the main characteristics – or features – one can extract from voice signals are defined. Since voice can result in speech or singing signals that share similarities but also differences, a comparison between speaking and singing voices is drawn. Finally, two major signal-oriented research fields highly tied to each other are presented: analysis and synthesis of voice signals.

### 2.2.1 Digital audio basics

The starting point of (sound) signal processing is the signal itself. When a sound is produced in a fluid, the surrounding molecules are set in motion by the sound waves, creating pressure variations step by step in both space and time.

A sound signal $f$ is therefore a function varying continuously in time that maps an instant $t_c \in \mathbb{R}$ to a vibration amplitude $f(t_c)$, that is

$$ \begin{array}{c} f : \mathbb{R} \to \mathbb{R} \\ t_c \mapsto f(t_c). \end{array} \qquad (2.2) $$
## From continuous-time to discrete-time signals

In practice, sound signals are captured and often emitted by systems such as microphones and loudspeakers that cannot deal with continuous values for time and amplitudes as it would require an infinite storage capacity. Therefore, in the context of digital signal processing, audio signals are discretized by means of quantization and sampling.

Quantization consists in limiting the amplitude range of $f$ to a finite number of allowed values $Q_b$, depending on the number of digital bits $N_b$ implied in the process, such that each amplitude $f(t_c)$ becomes the closest permitted value $f_q(t_c) \in Q_b$, i.e.,

$$f_q(t_c) = \min_{q \in Q_b} |f(t_c) - q|. \qquad (2.3)$$

Sampling consists in measuring the signal amplitude at only discrete instants $t_n$ with $n \in \mathbb{N}$. In the most classical sampling setup for audio, known as equidistant sampling, two successive discrete instants are separated by a constant, fixed sampling period $T_s$ so that $t_n = nT_s$.

This leads to define the discrete-time signal $x$ from the continuous-time signal $f$ as follows:

$$x : \mathbb{N} \rightarrow Q_b \\ n \mapsto x[n] = f_q(t_n) = \min_{q \in Q_b} |f(nT_s) - q|. \qquad (2.4)$$

## Frequency and harmonic analysis

A signal is referred to as periodic if there exists a minimal quantity called period $T_0 \in \mathbb{R}_+^*$ such that $\forall t \in \mathbb{R}, f(t) = f(t + T_0)$. The (fundamental) frequency $F_0$ is the reciprocal of the period:

$$F_0 = 1/T_0. \qquad (2.5)$$

If the period (in sec.) defines the duration before the signal repeats itself, the frequency (in HERTZ, Hz) measures the number of repetitions in an unit time interval. Integer multiples of the fundamental frequency are known as harmonics.

The harmonic decomposition of signals, which is the basis of all FOURIER'S analysis theory (HIGGINS et al., 1996), has revealed that any signal (even non-periodic ones) can actually be represented by a summation/integral of periodic signals. Hence, a signal contains information at many frequencies — from zero to an upper limit $F_{max}$ above which no information is left.

It was shown that discretizing a signal involves a sampling period $T_s$, one can thus define similarly the sampling frequency:

$$F_s = 1/T_s. \qquad (2.6)$$

One major result of digital signal processing is the NYQUIST-SHANNON-KOTELNIKOV theorem stipulating that a signal whose maximum frequency is $F_{max}$ can be sampled without loss of information if and only if the sampling frequency verifies the SHANNON'S criterion:

$$F_s > \frac{F_{max}}{2}. \qquad (2.7)$$
As far as humans are concerned, their hearing system has a frequency upper bound around 20kHz and voice components, whether voiced or unvoiced, are rather small for frequencies above 8kHz. As a result, given Eq. (2.7) usual values for the sampling frequency $F_s$ are 44, 1Hz or 48kHz for the recording industry and 16kHz for voice processing technologies. One exception to mention is the reduced bandwidth in telephony (less than 4kHz), for which a sampling frequency of 8kHz is enough.

Due to the mentioned harmonic decomposition, it is extremely common to work in the frequency domain rather than the time domain as many operations, such as filtering (i.e., alteration of the content of a signal), are easier to perform in the former one. Therefore, the most relevant frequency representations of audio signals are further introduced.

### Discrete Fourier Transform (DFT)

The Discrete Fourier Transform (DFT) is a mathematical operator allowing to decompose a discrete-time signal into its constituent frequencies, i.e., its spectrum or spectral representation. The DFT of the discrete-time signal $x$ of length $N$ – denoted $\text{DFT}[x]$ – is defined as follows:

$$ \text{DFT}[x] : \mathbb{N} \rightarrow \mathbb{C} \\ f \mapsto \text{DFT}[x](f) = \sum_{n=0}^{N-1} x[n] \exp \left( -\frac{2j\pi fn}{N} \right) \quad (2.8) $$

where $j = \sqrt{-1}$ is the imaginary unit and $f \in \{0, \dots, F-1\}$ is the $f$th frequency. The FOURIER coefficient $\text{DFT}[x](f) \in \mathbb{C}$ contains the magnitude and the phase of the sinusoidal component of the signal with frequency $fF_s/N$.

Note that a common variant of the Discrete Fourier Transform (DFT) is the Discrete Cosine Transform (DCT) that makes use of a cosine function instead of a complex exponential in Eq. (2.8). In the case of DCT, FOURIER coefficients are real numbers.

### Short-Time Fourier Transform (STFT)

The spectral content of voice or audio recordings is evolving with respect to time, e.g., people do not talk permanently with the exact same “tone” and chords/notes usually vary in songs. Thus, sound signals are intrinsically non-stationary as their statistical properties are time-dependent.

A single Discrete Fourier Transform (DFT) is not sufficient to study such signals as it would result in a representation that would not capture the temporal evolution of the spectral properties. Time-Frequency Representations (TFR) are much more relevant features to manipulate in this regard.

The Short-Time Fourier Transform (STFT) is specifically designed for the analysis of non-stationary signals as it considers them as successive short signals on which local DFTs are computed and concatenated.
This is done by sliding a temporal window $w$ of fixed sample size $W$, step by step with overlapping. Each resulting sub-signal is referred to as a *frame* and indexed through time by $t \in \mathbb{N}$. The sliding and overlapping parameter is known as *hop size* $H$ and typically defines the temporal precision $\delta t = H/F_s$. All in all, the STFT $\mathbf{x}$ of the signal $x$ is a function of both time frame $t$ and frequency $f$:

$$ 
\begin{array}{l}
\mathbf{x} : \mathbb{N} \times \mathbb{N} \rightarrow \mathbb{C} \\
[t, f] \mapsto \mathbf{x}[t, f] = \sum_{n=0}^{N-1} w[n] x[n + tH] \exp \left( -\frac{2j\pi fn}{N} \right).
\end{array}
\qquad (2.9) $$

Windowing the signal might prevent edge effects like the GIBBS phenomenon (GOTTLIEB and SHU, 1997) from appearing but distort the samples' weight, which is compensated by overlapping. STFT computations classically rely on the HANN (or HANNING) window:

$$ w[n] = \begin{cases} \frac{1}{2}\left(1 - \cos\left(2\pi\frac{n}{W}\right)\right) & \text{if } n \in \{0, \dots, W-1\} \\ 0 & \text{elsewhere.} \end{cases} \qquad (2.10) $$

### Spectrogram

The spectrogram $|\mathbf{x}|$ is the magnitude of the STFT, that is

$$ 
\begin{array}{l}
|\mathbf{x}| : \mathbb{N} \times \mathbb{N} \rightarrow \mathbb{R} \\
[t, f] \mapsto |\mathbf{x}|[t, f] = \left| \sum_{n=0}^{N-1} w[n] x[n + tH] \exp \left( -\frac{2j\pi fn}{N} \right) \right|.
\end{array}
\qquad (2.11) $$

It is one of the most used representations for audio and music processing.

### Convolution & signal filtering

Convolution, denoted $*$, is an extremely popular operation as it mathematically embodies linear filtering, i.e., the modification of some inputs by application of a kernel $\mathcal{K}$. It concretely performs a smooth weighted average between the kernel and the inputs at each point.

In the case of a 1-D convolution – inputs are typically time series like a raw signal $x$. The kernel $\mathcal{K}$ is also one-dimensional and the resulting filtering is expressed as:

$$ (x * \mathcal{K})[n'] = \sum_{n} x[n] \mathcal{K}[n' - n]. \qquad (2.12) $$

In the case of a 2-D convolution – inputs are typically images or, for audio, any TFR like the spectrogram $|\mathbf{x}|$. The kernel $\mathcal{K}$ is therefore two-dimensional and the resulting filtering is expressed as:

$$ (|\mathbf{x}| * \mathcal{K})[t', f'] = \sum_{t'} \sum_{f'} |\mathbf{x}|[t', f'] \mathcal{K}[t' - t, f' - f]. \qquad (2.13) $$
### 2.2.2 Voice features

The general notions and frequency representations of sound signals have been introduced, allowing one to specifically define the main characteristics — or *features* — related to voice signals. In order to clearly emphasize the relationships among these features, their Time-Frequency Representations (TFR) are summarized in Figure 2.7.

#### Pitch — Fundamental Frequency (F0)

The *pitch* of a voice, similar to the pitch of a note, is related to its perceived highness or lowness. The common auditory sensation is that the greater the pitch, the higher the voice. It allows to naturally distinct voice tones from one another and order them in a gradually increasing scale from low to high.

In practice, when a given pitch is perceived, it can be mapped to the frequency of a pure sound (sine wave). Therefore, throughout this manuscript, the Fundamental Frequency (F0) will be the main feature to characterize a pitch and both terms may be used interchangeably, although there is more to pitch than a single F0 value, e.g., (RÉVÉSZ, 1954; SHEPARD, 1982).

#### Log-Mel-spectrogram

Psychoacoustic studies dedicated to human auditory system revealed that human hearing perception scales up logarithmically in frequency and intensity (FASTL and ZWICKER, 2006). The ear, indeed, is able to distinguish (1) very calm from very loud sounds; and (2) lower from higher pitches with a better discriminative pitch perception in low frequencies. These considerations are not taken into account in a simple spectrogram as defined in Eq. (2.11).

Regarding the ear's loudness perception, it is more relevant to compute log-scaled magnitudes than linear ones. By means of the operation $|\mathbf{x}| \mapsto 20 \log_{10}(\epsilon + |\mathbf{x}|)$, with $\epsilon$ a small value, the spectrogram is turned into a *log-spectrogram*.

Regarding the ear's frequency perception, STEVENS et al. (1937) introduced a perceptive "mel"ody scale — the *Mel scale* — specifically designed to represent frequencies in line with human audition:

$$M(f) = 2595 \times \log_{10} \left( 1 + \frac{f}{700} \right). \qquad (2.14)$$

Mapping the spectrogram linear frequencies to the Mel scale is usually done with *F* overlapping triangular filters referred to as *Mel filterbank*. In the end, the *mel-spectrogram* is obtained.

Coupling both loudness and frequency aspects, one can compute the *log-mel-spectrogram*, which contains all perceptual features, making it a robust default representation of voice and audio signals.
## Spectral envelope

A classical audio production model is the source-filter model, which assumes that a signal $f$ results from an original excitation (the *source*) $e$ modified by a resonator (the *filter*) $r$. Remembering section 2.1.3, the voice correlates are the vocal cords for the source and the vocal tract for the filter. For each time $t_c \in \mathbb{R}$, the signal is therefore expressed as a temporal convolution:

$$f(t_c) = (e * r)(t_c). \qquad (2.15)$$

In the frequency domain, denoting $\mathcal{F}$ a general FOURIER transform, the spectrum are multiplied,

$$\mathcal{F}[f](f) = \mathcal{F}[e](f) \times \mathcal{F}[r](f). \qquad (2.16)$$

Taking the logarithm of Eq. (2.16), one can isolate spectral components, respectively the pitch (from the *source*) and the harmonic content or *timbre* (resonances/attenuations from the *filter*). Summing up, the *cepstrum* is defined as:

$$\log(\mathcal{F}[f](f)) = \underbrace{\log(\mathcal{F}[e](f))}_{\text{fast variations in } f} + \underbrace{\log(\mathcal{F}[r](f))}_{\text{slow variations in } f}. \qquad (2.17)$$

The (log-) *spectral envelope* precisely corresponds to the resonator term $\log(\mathcal{F}[r])$. It is supposed to be a smooth and F0-free representation carrying only information from the vocal tracts (voice formants and timbre). The problem is $r$ is concretely unknown. In practice, a Discrete Cosine Transform (DCT) of the whole *cepstrum* is performed to separate high and low *quefrencies* and isolate the spectral envelope from the excitation structure. See (RÖBEL and RODET, 2005) for another method for spectral envelope estimation.

## Mel-Frequency Cepstral Coefficients (MFCCs)

The Mel-Frequency Cepstral Coefficients (MFCCs) have been one of the widespread features in the audio and voice literature, allowing a compact representation of above-mentioned spectral envelope (DAVIS and MERMELSTEIN, 1980).

The MFCCs are the low order coefficients (i.e., usually up to 13) from the Discrete Cosine Transform (DCT) calculated on the log-mel-spectrogram. According to previous source-filter modeling, MFCCs are expected to globally follow the spectral envelope and be pitch-independent to a certain extent.

## Phonetic characterization

As discussed in section 2.1.3, formants are induced by the vocal tract. Therefore, as previously exposed, spectral envelope and/or MFCCs features – given their strong links – seem particularly well-suited for representing formants and phonetic information. Concretely, a given spectral envelope should be enough to recognize a precise phoneme and reciprocally.
<figure><img src="image_17.png" /><figcaption>Figure 2.7: Time-Frequency Representations (TFR) in voice signal processing. Example is sung vocals “I ain’t got the *time*” from Amy WHINEHOUSE’s Rehab, HANSEN (2012)’s dataset.</figcaption></figure>
### 2.2.3 Speech vs singing comparison

The main representations used to study and process voice signals have been presented. While these features can be exploited for both speech and singing, it is undeniable that — despite common characteristics — speaking and singing voices cannot be classified as equals. This section provides insights on these differences that are worth mentioning. A visual example highlighting these variations can be found in Figure 2.8.

Either specialized on singing (SUNDBERG and ROSSING, 1990) or speech (KENT and READ, 2002), or clearly addressing their comparison (LIVINGSTONE et al., 2013; OHISHI et al., 2005), even up to the recording, release and analysis of a specific dataset (DUAN et al., 2013), there is a long research history dedicated to such questions. Based on these references, five main differences between speaking and singing voices are listed. They are either related to characteristics of the speaker's vocal apparatus (see section 2.1.3), or prosodic gestures and expressivity (see section 2.1.4). A summary of these speech/singing discrepancies is proposed:

*   **Vocal tract shape** — During the generation of voice signals, the shape of the vocal tract (i.e., mouth, tongue, and throat) varies to produce the several phonemes and sounds. A first difference between speech and singing voice is that the vocal tract changes much faster in speech than in singing.

*   **Intonation** — Then, intonation, is an important way to convey meaning and emotion through voice by, e.g., slightly adapting F0 curves. There are generally faster variations in speech but they may purposely be exaggerated in a singing context, while rather subtle in spoken utterances.

*   **Vibrato** — The vibrato may be the prime example of speech-singing differences as this effect is not observed for spoken voice. Created by a fine control of the muscles of the vocal apparatus, the vibrato manifests through local and rapid fluctuations in the pitch of a note. It has been highly analyzed by musicologists to understand a singer's style (CHABOT-CANET et al., 2020) as it is a major piece of expressivity and is ubiquitous in, e.g., French chanson or popular music.

*   **Fundamental Frequency (F0)** — The fundamental frequency of spoken voice typically ranges between 80Hz and 200Hz. The singing voice has a much wider tessitura as sung pitches can reach high frequencies, up to 1kHz and more.

*   **Phoneme duration** — In speech, phoneme duration is around 20-200ms (as measured on (ZUE et al., 1990)). In singing, for which a pitched note is associated with a vowel, phonemes can be hold much longer as singers tend to sustain vowels for longer periods. In doing so, they can add expressivity to their performances, duration being in itself an expressive means, via intensity variations over a syllable or vocal techniques mastering.
<figure><img src="image_18.png" /><figcaption>Figure 2.8: Illustration of differences between speech and singing: vowel duration, F0, vibrato. Example is the same sentence read and sung by KENN_17 from DUAN et al. (2013)'s dataset.</figcaption></figure>

Moving forward, having covered the diversity encountered in voice signals – both in terms of their features and their quality (speech *vs* singing) – the core tasks and challenges involved in current voice research can be exposed: analysis and synthesis.

### 2.2.4 Voice analysis and synthesis

Scientists in voice-related research have extensively worked on three main fields – voice analysis, synthesis and transformation. The later field, voice transformation, consists in modifying one or some feature(s) of a given voice, e.g., changing the attitude in a recording (TAO et al., 2006) or convert several vocal attributes (SISMAN et al., 2020), but it is out of the scope of this thesis. On the contrary, the two former, synthesis and analysis, have an important role in this work.

#### *Synthesis of voice signals*

Voice synthesis is the field aiming at producing artificial, yet plausible, voices. This includes synthesis of both speech and singing signals that should be perceived as natural as possible. As in many other voice-oriented tasks, early research was focused on speech and the same procedures were later extended and adapted to singing.

The first attempt to synthesize speech dates back to 1791 with the “speaking machine” (VON KEMPELEN, 1791). This mechanical system, and others that followed (DUDLEY et al., 1939), was designed to mimic the physical mechanisms of speech production that were introduced in section 2.1.3. Far from this original system that required manual intervention, speech synthesis has benefited from the progress in computer sciences towards the rise of automatic methods.

The modern era of speech synthesizer has been marked with the development of Text-to-Speech (TTS) systems, that are capable to directly synthesize acoustic signals with relevant speech parameters for a given text to be voiced. Most current systems rely on approaches based on artificial intelligence for both speech (NING et al., 2019) and singing (CHO et al., 2021) synthesis. Related deep learning notions for voice will be covered in section 3.1.
Today, voice synthesis is ubiquitous in daily life technologies and has found relevance in telecommunications (e.g., vocal assistants (PAL et al., 2019)), medicine (BRUNOW and CULLEN, 2021; RÉPOVÁ et al., 2021) or artistic creations (e.g., video games (FARNER et al., 2008)), etc.

While this thesis is not dedicated to voice synthesis, there exists a synthesis method that resonates with this work. It is *concatenative synthesis* that achieves TTS by creating smooth transitions between phonemes — or groups of successive phonemes (*biphones*, *triphones*) — from pre-existing voice recordings (ARDAILLON, 2017). This approach intrinsically requires a strong correspondence between audio and text, which is in line with this research. Thus, voice synthesis will remain an interesting application that will be discussed in section 2.4.

## Analysis of voice signals

Voice analysis is the field aiming at extracting all kind of information from voice signals, which covers a wide scope of possible objectives. Among numerous examples, typical voice analysis tasks are concerned with pitch estimation (ARDAILLON and ROEBEL, 2019), language and speaker identification (TIRUMALA et al., 2017), keyword spotting (LÓPEZ-ESPEJO et al., 2021), query by singing and humming (LIANG et al., 2021a), or vocals extraction (COHEN-HADRIA et al., 2019). There is also an active research field — Music Information Retrieval (MIR) — specialized on similar considerations for music (MÜLLER, 2007b; SCHEDL et al., 2014).

Early research was based on *hand-crafted* methods, i.e., deriving systematic rules upon computation of audio descriptors (PEETERS et al., 2011). With the emergence of large collections of data and higher computational power over the years, recent research rather relies on *data-driven* methods (e.g., machine learning and deep learning) that are able to construct relevant prediction patterns from data themselves (PAPAKOSTAS et al., 2017), or a coupling of both. The required background to manipulate data-driven systems is introduced in the next chapter.

This thesis inscribes itself in voice analysis as it aims to understand and uncover the underlying relationships between voice and symbolic representations. Hence, the upcoming section is dedicated to the tasks intimately related to this research: transcription and alignment.

## Section summary – Voice signal processing

Voice signal processing a wide scientific field allowing to define, represent and transform information in *digital voice signals*. These signals result from a temporal sampling with a usual *sampling frequency* of $F_s = 16$kHz. Among the numerous existing *voice features*, the *log-mel-spectrogram* and *spectral envelope* are two Time-Frequency Representations (TFR) of highest interest. The later captures the formants and phonemes while the former integrates all human perception aspects. Based on TFR, active research on voice *analysis* and *synthesis* has been conducted. In practice, despite the known speech-singing differences (e.g., vowels duration, etc.), speech approaches tend to be adapted to singing.
## 2.3 Transcription and alignment tasks

Transcription and alignment are two prominent analysis tasks in many fields. They have gained tremendous interest and found major and diverse implications in the audio domain, and especially in the voice community, and beyond. The research presented in this manuscript is dedicated to voice-to-symbols synchronization, yet, given the high proximity between alignment and transcription in terms of concepts, formalization, modeling approaches as well as general multi-modal problems, both of them will be studied and referred to throughout the entire document.

### 2.3.1 Definitions

The common definitions and notations that will used for both tasks are introduced first. These concepts are, more generally, in line with all literature processing data sequences — a sequence being a finite succession of elements, with a notion of order through an indexing via indices.

#### Sequence modeling & decoding

Let $\mathcal{X}$, $\mathcal{Y}$ and $\mathcal{H}$ denote three feature spaces. Let $\mathcal{X}^*$, $\mathcal{Y}^*$ and $\mathcal{H}^*$ be the sets of all sequences over $\mathcal{X}$, $\mathcal{Y}$ and $\mathcal{H}$, respectively.

Let $m$ denote a message emitted in two different forms with, e.g., different modalities and/or expressivity. Let $\mathbf{x} \in \mathcal{X}^*$ denote the first emission and $\mathbf{y} \in \mathcal{Y}^*$ denote the second emission. Let $T$ be the length of $\mathbf{x}$ such that $\mathbf{x} = \{\mathbf{x}_0, \dots, \mathbf{x}_{T-1}\}$ with $\forall t \in \{0, \dots, T-1\}, \mathbf{x}_t \in \mathcal{X}$. Let $M$ be the length of $\mathbf{y}$ such that $\mathbf{y} = \{\mathbf{y}_0, \dots, \mathbf{y}_{M-1}\}$ with $\forall m \in \{0, \dots, M-1\}, \mathbf{y}_m \in \mathcal{Y}$.

Transcription and alignment tasks can be seen as twofold procedures composed of

* A similar modeling function $\mathcal{M}$, such that

$$ \mathcal{M}: \begin{array}{ccc} \mathcal{X}^* & \rightarrow & \mathcal{H}^* \\ \mathbf{x} & \mapsto & \mathcal{M}(\mathbf{x}). \end{array} \qquad (2.18) $$

Applied on the first emission $\mathbf{x}$, this function returns the quantity $\mathcal{M}(\mathbf{x})$, which has a different designation (e.g., saliency map, posteriorgram) according to the task objective. With no loss of generality, it can be called *latent code*, *hidden code* or *hidden representation*.

* A specialized decoding module $\mathcal{D}$, which differs between transcription and alignment tasks, but somehow connects the two emissions $\mathbf{x}$ and $\mathbf{y}$.

The existing and chosen approaches for designing the modeling and decoding functions are at the core of this research and will be mathematically exposed in the following chapters.
## Sequence similarity & alignment

The notion of alignment comes from the long research history dedicated to the relationships and similarities between sequences, which manipulates specific tools that are introduced hereinafter.

To define a notion of correspondence between two sequences $x \in \mathcal{X}^*$ and $y \in \mathcal{Y}^*$, a sequence $\pi$ of length $K = \max(M, T)$ of ordered tuples over the indices of $x$ and $y$ is defined as:

$$ \pi = \{\pi_k\} \quad \text{with} \quad \forall k \in \{0, \dots, K-1\}, \quad \pi_k = (t_k, m_k) \in \{0, \dots, T-1\} \times \{0, \dots, M-1\} \qquad (2.19) $$

This sequence goes by many names including a *path* or *pathway* (NEEDLEMAN and WUNSCH, 1970), *warping* or *warping function* (ITAKURA, 1975), or *alignment path*. The latter option will be chosen in this work. The ensemble of all paths between elements of $\mathcal{X}^*$ and $\mathcal{Y}^*$ is denoted $\Pi(\mathcal{X}^*, \mathcal{Y}^*)$. There exist several properties – or alignment constraints – that a given path $\pi \in \Pi(\mathcal{X}^*, \mathcal{Y}^*)$ is expected to satisfy. Namely,

* Boundary conditions such that the alignment starts (resp. ends) at the first (resp. last) indices of the two sequences:

$$ \pi_0 = (0, 0) \quad \text{and} \quad \pi_{K-1} = (T-1, M-1). \qquad (2.20) $$

* Monotony, which stipulates that the path can only move forward, such that successive indices verify:

$$ \forall k \in \{0, \dots, K-2\}, \quad t_k \le t_{k+1} \quad \text{and} \quad m_k \le m_{k+1}. \qquad (2.21) $$

* Limited progression of the path, such that only some transitions are permitted between successive states $\pi_k$ and $\pi_{k+1}$ according to a set of allowed gaps $\mathcal{G}$, i.e.,

$$ \forall k \in \{0, \dots, K-2\}, \quad \pi_{k+1} - \pi_k = (t_{k+1} - t_k, m_{k+1} - m_k) \in \mathcal{G}. \qquad (2.22) $$

The Figure 2.9 illustrates an alignment verifying these constraints between audio and text (e.g., $x$ represents $T$ spectral frames and $y$ represents $M$ characters) when $\mathcal{G} = \{(1,0), (1,1)\}$. Finally, let $\mathcal{S}: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ be a similarity measure between the elements of $\mathcal{X}$ and $\mathcal{Y}$. The alignment between the sequences $x \in \mathcal{X}^*$ and $y \in \mathcal{Y}^*$ corresponds to the optimal path $\pi^*$, that is the path maximizing the accumulative similarity measures (i.e., for all tuples of indices). It reads, with $x_t \in \mathcal{X}$ and $y_m \in \mathcal{Y}$ the $t$th and $m$th elements of $x$ and $y$, respectively,

$$ \pi^* = \underset{\pi}{\operatorname{argmax}} \sum_{(t,m) \in \pi} \mathcal{S}(x_t, y_m). \qquad (2.23) $$

The resulting alignment score $\mathbf{A}^* \in \mathbb{R}$ between the sequences is the full similarity measure of this optimal path:

$$ \mathbf{A}^* = \sum_{(t,m) \in \pi^*} \mathcal{S}(\mathbf{x}_t, \mathbf{y}_m). \qquad (2.24) $$

Solving Eq. (2.23) is a typical optimization problem. Techniques for efficiently computing this path via Dynamic Programming (DP) will be seen in the next chapter – in section 3.3.1.
<figure><img src="image_19.png" /><figcaption>Figure 2.9: An alignment path $\pi$ between sequences $\mathbf{x}$ (voice log-mel-spectrogram) and $\mathbf{y}$ (text).</figcaption></figure>

## Transcription/Recognition

Transcription – or recognition – is the task that aims at predicting a symbolic sequence of a message from some input data representing its emission. Concretely, from the message emission in a given modality $\mathbf{x} \in \mathcal{X}^*$, it is expected to retrieve $\mathbf{y} \in \mathcal{Y}^*$ – where in practice, $\mathcal{Y}$ is a finite alphabet of symbols $\mathcal{A}$. The Figure 2.10 illustrates the philosophy of this task.

Although there might be a little distinction between transcription and recognition terms<fnref n="2" />, they will be both used interchangeably from now on, very much as done in the literature.

In a transcription context associated with a modeling $\mathcal{M}$, the decoding module $\mathcal{D} \equiv \mathcal{D}_T$ becomes

$$ \mathcal{D}_T : \mathcal{H}^* \rightarrow \mathcal{Y}^* \\ \mathcal{M}(\mathbf{x}) \mapsto (\mathcal{D}_T \circ \mathcal{M})(\mathbf{x}) = \hat{\mathbf{y}} \qquad (2.25) $$

and therefore estimates a sequence $\hat{\mathbf{y}} \in \mathcal{Y}^*$ as close as possible to the real sequence $\mathbf{y}$.

Classical options for the decoding module $\mathcal{D}_T$ are greedy search (GERMANN, 2003) or beam search decoding (FREITAG and AL-ONAIZAN, 2017) that are designed to estimate the most probable sequence given an hidden representation like $\mathcal{M}(\mathbf{x})$.

<footnote n="2">²Actually, recognition is a specific and constrained case of transcription that aims to predict one-to-one correct symbol while transcription, more generally, aims to reflect a correct content (as a semantic concept), which is more flexible than per-symbol prediction.</footnote>
<figure><img src="image_20.png" /><figcaption>Figure 2.10: Transcription task overview: an emission **x** of a message *m* is processed by a model **M** prior to the estimation **ŷ** of a second and symbolic emission **y** thanks to a decoder **D**<sub>T</sub>.</figcaption></figure>

Alignment/Synchronization

Alignment – or synchronization – is the task that consists in finding the precise correspondence between several emissions of a message. The events to tie, hence, share a common underlying information but differ in their respective emissions due to different modalities and/or expressivity. Concretely, from two emissions **x** ∈ 𝒳* and **y** ∈ 𝒴* of the same message *m*, it is expected to retrieve the optimal path **π*** ∈ Π(𝒳*, 𝒴*) (and its similarity score **A*** ∈ ℝ) between them. The Figure 2.11 illustrates the philosophy of this task.

According to its Latin etymology *linea* – meaning *line* or *string* – the word “alignment” intrinsically carries the idea to draw a line between objects or concepts and, by doing so, mapping them out. *Synchronization* is the specific case of alignment for which *time* is involved in the process, *i.e.*, when linking the different events requires and is done with a temporal reference – a * timeline *. For such cases, one can refer synonymously to both *alignment* and *synchronization* terms, as it will be done in the rest of the writing.

In this thesis, more specifically, *forced alignment* problems are tackled, as finding the best path ensures that none of the events to align is missed. Indeed, the above-mentioned constraints, *i.e.*, Eq. (2.20), Eq. (2.21) and Eq. (2.22), make sure that each event in the sequence **y** has at least one corresponding event in the other sequence **x** (admitting that *M* ≤ *T* otherwise some symbol from **y** would be left out). Many alignment issues actually deal with forced alignment.

In such a context associated with a modeling $\mathcal{M}$, the decoding module $\mathcal{D} \equiv \mathcal{D}_{\pi}$ becomes

$$
\begin{equation}
\begin{aligned}
\mathcal{D}_{\pi} : \mathcal{H}^* \times \mathcal{Y}^* & \rightarrow \Pi(\mathcal{X}^*, \mathcal{Y}^*) \\
(\mathcal{M}(\mathbf{x}), \mathbf{y}) & \mapsto \mathcal{D}_{\pi}(\mathcal{M}(\mathbf{x}), \mathbf{y}) = \hat{\pi}^*
\end{aligned}
\tag{2.26}
\end{equation}
$$

and therefore estimates an alignment path $\hat{\pi}^*$ as close as possible to the true optimal path $\pi^*$.

As a reminder, the modeling function $\mathcal{M}$ shares a common role between transcription and alignment: to encode the input data $\mathbf{x}$ into a more relevant and exploitable hidden representation before decoding. This function has been implemented with multiple approaches over the years that will be exposed in the upcoming chapter covering mathematical essentials.
<figure><img src="image_21.png" /><figcaption>Figure 2.11: Alignment task overview: an alignment path $\hat{\pi}^*$ between two emissions x and y of a message m is estimated through a modeling function $\mathcal{M}$ and a mixed decoder $\mathcal{D}_{\pi}$.</figcaption></figure>

### 2.3.2 Transcription algorithms

In this section, the wide variety of transcription algorithms are presented starting from general then audio-related transcriptions and with a final emphasis on the voice transcription tasks. Some of the mentioned examples are shown in Figure 2.12.

#### General transcription

There are many applications of transcription algorithms in various fields such as Computer Vision (CV) or Natural Language Processing (NLP).

Concrete examples are the optical recognition of characters (SRIVASTAVA et al., 2022) or handwriting digits (LECUN et al., 1989) in images, or address numbers from street views (GOODFELLOW et al., 2014).

From videos, one can attempt to transcribe sign language (BANTUPALLI and XIE, 2018) or identify movements in sports (CUST et al., 2019), which are special cases of automatic gesture recognition (WU and HUANG, 1999).

Through their application in studies of texts and languages, transcription algorithms also have their impacts in NLP (DISTER et al., 2009). One core domain is Machine Translation (MT) which proceeds to text-to-text predictions by automatically translating content from a source language to a target language (STAHLBERG, 2020).

#### Audio transcription

The audio domain, closer to this work, also requires transcription algorithms for diverse tasks. Besides speech/singing recognition, which are discussed below, a main research objective is to recover symbolic music (e.g., music sheet) from audio recordings.
<figure><img src="image_22.png" /><figcaption>Figure 2.12: Examples of transcription tasks: (a) handwritten digits recognition (LeCun et al., 1989); (b) Optical Music Recognition (OMR) applied to the first music score scan published by Prerau (1972); and (c) a typical voice recognition scenario.</figcaption></figure>

This includes notes/main melody estimation for singing voice (RIGAUD and RADENEN, 2016), instrument (ABESSER and MÜLLER, 2021) or over the global piece (BITTNER et al., 2015). Others topics are chord identification (JIANG et al., 2019), drums transcription (JACQUES and ROEBEL, 2019) or even several of these objectives at the same time (RYYNÄNEN and KLAPURI, 2008).

In the field of Optical Music Recognition (OMR), music scores are to be recognized inside images from high quality to highly deteriorated, such as pictures or scans of formatted scores (CALVO-ZARAGOZA et al., 2017) or handwritten ones (BARÓ et al., 2019).

## Voice recognition

The voice community has heavily relied on recognition algorithms, and this thesis inherits from this research that can be traced back to the 1970s (VELICHKO and ZAGORUYKO, 1970) and remains today an active field of investigations (LIU et al., 2023). In the long quest to adapt daily experiences and knowledge on human-human interaction to human-machine communication, speech and singing recognition systems have played a crucial role in the development of, e.g., vocal assistants (MICHAELY et al., 2017).

Automatic Speech Recognition (ASR) is the task dedicated to retrieve the *spoken* content in a *speech* recording. Automatic Lyrics Transcription (ALT) is the task dedicated to retrieve the *sung* content in a *singing* recording. As they are by essence extremely close, ASR algorithms serve as basis and are usually adapted to ALT. However, due to the more challenging nature of singing voice (cf. section 2.2.3), ALT performances are far below the ones reported on ASR.
<figure><img src="image_23.png" /><figcaption>Figure 2.13: Examples of alignment tasks: (a) biological sequences alignment based on NEEDLEMAN and WUNSCH (1970)'s algorithm; and (b) a typical voice alignment scenario.</figcaption></figure>

At the core of the scientific content defended in this thesis, the speech and singing recognition algorithms, especially relying on deep learning techniques, are explained in the next chapter.

### 2.3.3 Alignment algorithms

Similar to voice transcription, the sequence alignment algorithms are not unique to speech or singing processing. In fact, several research communities in the 1970s were facing the same need to find the optimal alignment path between sequences of several types and, as a result of their independent yet close investigations, similar algorithms were (re)discovered almost simultaneously. Some of the mentioned examples are shown in Figure 2.13.

#### General alignment

Telecommunications were the first field to benefit from a solution to sequence alignment when VITERBI (1967) introduced the fundamentals of his forward-backward decoding algorithm, which has become since then a core reference on all alignment systems and beyond (RAMASSO et al., 2007). Many authors still refer to the “VITERBI’s algorithm” or “VITERBI’s decoding”.
In bioinformatics, alignment systems have been widely used to arrange sequential data such as sequences of DNA, RNA, protein, amino acid, or equivalent through the work of NEEDLEMAN and WUNSCH (1970). The alignment of biological sequences have allowed to study their homology and identify regions of (dis)similarity between them. With such precise information, it has been possible to study gene expression (AACH and CHURCH, 2001) and enhance shared structural patterns highlighting, e.g., consequences of evolution, existence of common ancestor or effects of mutations (NG and HENIKOFF, 2001).

In the broader scope of *time series* manipulation (FOLGADO et al., 2018), such alignments algorithms have been used to compare such series leading to human activity analysis (MACHADO et al., 2015) or on-line secure validation of handwritten signature (XIA et al., 2018).

Sequence alignments are also found in Natural Language Processing (NLP) with the example of bitext word alignment that consists in retrieving, given a text and its translation in another language, the translation relationships among the words (BROWN et al., 1993).

## Audio alignment

Alignment challenges were also studied in the audio literature. Synchronization and performance comparisons between musicians (or singers) can be achieved via audio-to-audio alignment (KIRCHHOFF and LERCH, 2011).

Audio-to-score alignment, which deals with mapping a music score and an associated audio recording, is also a well-known research problem (SIMONETTA et al., 2021), and can be used to analyse how a given performer (musician/singer) enacts a symbolic score. Its real-time variant, referred to as score following (CONT, 2009), can lead to on-the-fly page turning during a live performance (ARZT et al., 2008) or be the starting point towards adaptive and automatic accompaniment (CONT et al., 2012; RAPHAEL, 2001).

## Voice alignment

This thesis is primarily concerned with the alignment between audio recordings featuring voice and symbolic sequences. Automatic Speech Alignment (ASA) is the task dedicated to align a symbolic sequence with an associated *speech* recording. Automatic Lyrics Alignment (ALA) is the task dedicated to align a symbolic sequence with an associated *singing* recording.

ASA (resp. ALA) systems have been systematically built upon the recent advances on ASR (resp. ALT) — therefore, ASA and ALA are also based on similar models and ALA still faces more challenges than ASA because of the complexity of singing voice.

The scope of these voice-related paragraphs remain voluntarily limited to a simple introduction of voice recognition/alignment as the next chapter will extensively detail the technical background and design choices for their practical implementations — especially *section 3.3* for legacy approaches and *section 3.4* for deep learning-based strategies.
### Section summary – Transcription and alignment tasks

Given a message *m* characterized by two emissions **x** and **y** with different modalities and/or expressivity,

* *Transcription* is the task aiming at predicting **ŷ** – an estimate of **y** – from **x**;

* *Alignment* is the task aiming at mapping at least one dimension of each sequences **x** and **y** through $\hat{\pi}^*$ – an estimate of the optimal path between them.

Sequence transcription and alignment problems arose in many literature and have been concerned with various objectives in audio and voice processing. They follow a similar two-stage architecture made of a modeling function $\mathcal{M}$ applied on **x** and a decoding module $\mathcal{D}$ specific to each task allowing the final predictions.

## 2.4 Applicative motivations of voice alignment

The central text-to-voice alignment problem, which historically emerged in the late 1970s, originated from a bottleneck in the speech recognition community: the need to automatically segment voice recordings into small labelled excerpts (i.e., pairs of audio and their associated text), with the aim to build larger corpora than those available at that time. Ever since, the synchronization of a voice signal with its corresponding text has known many practical applications for the entertainment industry or the general public such as audio indexing via text, automatic subtitling or karaoke generation (FUJIHARA and GOTO, 2012).

In this short section, the applicative motivations presented in this thesis are briefly mentioned to give a global overview on the launched initiatives and scientific and technological cooperations. The Chapter 7 will be dedicated to their practical accomplishment.

* *Concatenative phonetic synthesis*

As mentioned, phonemes are the building blocks of voiced languages, from syllables to words and sentences etc. In the context of concatenative synthesis – a specific strategy for Text-to-Speech (TTS) – a target text is decomposed into its corresponding phonemes and audio is generated by selecting in an available dataset the most relevant voice samples associated with each phoneme and their surrounding context (usually, succession of two phonemes, i.e., biphones, are used). Voice-to-phoneme alignment is prominent as the reference voice recordings needs to be aligned very precisely so that chosen regions for a given biphone make sense. The alignment algorithms of this thesis are applied to the alignment of new voices for ISiS<fnref n="3" />. See section 7.1 for further details.

<footnote n="3">³Ircam Singing Synthesis: https://forum.ircam.fr/projects/detail/isis/.</footnote>
* *Production strategies of speech attitudes*

Humans are clearly able to communicate social intentions when interacting with one another. In the context of such expressive speech, conveying noticeable social attitudes, the synchronization of audio with phoneme sequences allows group statistics between speakers regarding their speech prosody modulations and phonetic structure changes over time — a major step forward uncovering the interactional properties beyond speech prosody. See section 7.2 for further details.

* *Musicological analyses of singing voice style*

Singers, very much like speakers, convey emotions and use deliberate effects to integrate expressivity in their performances. The study of singing style — *i.e.*, these production strategies at play in a singer's performances defining their very own artistic identity — is of great interest for musicologists eager to understand the artistic choices made by a singer. The automatic voice alignment algorithms developed in this thesis — *both* for syllables *and* notes — fully inscribe themselves in such musicological research as they highly simplify tedious tasks traditionally done manually and allow exploring in details, *e.g.*, rythmic, articulation, or multi-modal gesture like melisma. To this aim, a musicological pipeline for singing voice style analysis based on neural voice processing and alignment has been developed in the **ARS** project and benefit from this thesis research. See section 7.3 for further details.

* *Perspectives*

Finally, some ongoing/future applications are mentioned in section 7.4, two of which are

- Automatic pre-segmentation of large corpora like turning an audiobook into small coherent pairs of audio and text, which is possible thanks to a collaborative work dedicated to a linear memory decoding algorithm allowing to align very long audio recordings, e.g., full audiobooks or lengthy music playlists and their transcripts;

- Joint singing voice separation and alignment so that both tasks can help each other as suggested by recent literature.

## Section summary – Applicative motivations of voice alignment

Several applications of voice alignment motivates the research in this thesis. Phoneme-level alignment paves the way for singing concatenative synthesis and understanding the production strategies involved in speech attitudes. Word-level alignment, which is the standard case study, is also of high interest, especially when applied on very long audio recordings. Coupled with note alignment, it greatly simplifies tedious and time-consuming processes required in musicological analyses.
## 2.5 Context and applicative motivations in a nutshell

### Chapter summary – Context and applicative motivations

In this chapter, I exposed the key concepts to carry out voice analysis research, with a focus on voice alignment and its applications. Based on human communication theory, I introduced the two modalities at the heart of my research: *symbolic sequences* and *voice*. Sequences inherit from a long history of writing languages and require the existence of an alphabet of characters *A* — like the Latin alphabet *A*<sub>l</sub> or note alphabet *A*<sub>n</sub>. From a signal processing perspective, I presented two relevant representations to analyse and synthesize voice: (1) the *log-mel-spectrogram* — capturing all perceptual features; and (2) the *spectral envelope* — characterizing well formants and phonetic content. Finally, I defined the closely related *transcription* and *alignment* tasks and mentioned the thrilling applications offered by synchronizing voice with symbols, including the ones this thesis is concerned with, e.g., musicological studies, production of vocal attitudes, notes alignment, etc.
Chapter 3

Scientific background for voice alignment

> "Nothing in life is to be feared, it is only to be understood.
> Now is the **Time** to understand more, so that we may fear
> less."

> — Marie SKŁODOWSKA-CURIE

This chapter aims to provide a comprehensive review of the mathematical concepts, technical tools and scientific strategies for designing voice-to-symbol synchronization systems and the practical implementations of their associated acoustic models and decoding modules.

Upon delving into the essentials of *deep learning* in section 3.1, which form the foundation for the rest of this thesis, and providing a formalization of the *voice alignment problem* in section 3.2, the various options found in the literature for the acoustic modeling and the decoding module are presented. To this aim, the section 3.3 is dedicated to the *traditional approaches* and notably introduces the Dynamic Time Warping (DTW) algorithm as a reference decoding module and retraces the emergence of probabilistic acoustic models. Following recent and promising trends, the section 3.4 focuses on the latest *deep learning techniques* applied to voice alignment with an exploration in greater depth, in section 3.5, of the *Connectionist Temporal Classification (CTC)* which, given its flexibility, has been chosen as the main strategy to perform voice-to-symbol alignment in this thesis. This chapter is summarized in section 3.6.

<figure><img src="image_24.png" /></figure>
### 3.1 Deep learning background

The emergence of deep learning approaches, and especially deep neural networks, has initiated a breakthrough in many scientific fields. For a given task at hand, a deep learning-based model can specialize “itself” by automatically targeting and exploiting the most relevant task-related information in the data, thus outperforming legacy methods that were rather dependent on fixed prior knowledge. Speech and singing voice communities are no exception as recent analysis and synthesis systems have been built upon deep learning techniques, increasingly replacing former approaches, while denoting major progress in comparison to previous literature.

This section reviews the motivations and the mathematical notions necessary to carry out research using deep learning as a tool for voice processing tasks (i.e., voice transcription and alignment). A brief contextualization on data-driven systems serves as basis to further introduce general machine/deep learning concepts, notably deep neural networks and their architecture design. For an extensive explanation of deep learning, one can refer to (GOODFELLOW et al., 2016), the pointing reference of this section.

#### 3.1.1 Data-driven approach

*Deep learning* is the computer science field dedicated to the development of algorithms capable to *learn* specific and relevant patterns from data according to some objective. These algorithms are known as as *data-driven* systems as they exploit the observation of a wide quantity of data.

In opposition to *classic* systems — in the denomination of HUMPHREY et al. (2012) — deep learning models do not require expert prior knowledge for defining *hand-crafted* 1) features and/or 2) prediction rules from them. Instead, and from the data themselves, such a system can manipulate complex abstractions, automatically extract relevant features and make predictions. Frequently, instead of the raw data, compact and interpretable hand-crafted representations, *e.g.*, Time-Frequency Representations (TFR) for audio, are computed at the very beginning of the system — but are eventually turned into learned features by the model.

Benefiting from greater computational power and the latest improvements in hardware and software infrastructures, deep learning-based systems have shown impressive generalization properties that have lead to significant advances in many states of the art, *e.g.*, computer vision (LOISEAU et al., 2021; VOULODIMOS et al., 2018), natural language processing (OTTER et al., 2020) and audio domain (BRIOT, 2022; DOUWES et al., 2023; PURWINS et al., 2019).

#### 3.1.2 Essentials

In this thesis, feature extraction and task-oriented prediction rely on the training of Deep Neural Networks (DNN) in a supervised learning scenario, obtaining their parameters by optimization of a loss function via mini-batch gradient descent. These core concepts are gradually developed.
## Artificial neuron

A whole category of learning models have been vaguely inspired from the human brain and specifically from a simplified version of its functioning centered around the notion of neuron. Let $\mathcal{X}^*$ and $\mathcal{W}$ denote some feature space definition ensemble with no a priori.

In imitation of a biological neuron, an artificial neuron receives an input excitation $\mathbf{x} \in \mathcal{X}^*$ to which it is more or less sensitive. The reactivity of the neuron to the excitation is embodied by weighting parameters $\mathbf{w} \in \mathcal{W}$ and a response function $f: \mathcal{W} \times \mathcal{X}^* \to \mathbb{R}$ that applies some transformation to $\mathbf{x}$ according to $\mathbf{w}$. By electrical analogy, if the resulting “potential” $f(\mathbf{w}, \mathbf{x}) \in \mathbb{R}$ is sufficient, e.g., higher than a threshold $b \in \mathbb{R}$, then the neuron is said to be activated and can transmit information to the next neuron, and so on. This neural activation, which is intrinsically non-linear, is simulated by an activation function $\sigma: \mathbb{R} \to \mathbb{R}$.

Summing up, an artificial neuron predicts an output $\hat{y} \in \mathbb{R}$ from inputs $\mathbf{x} \in \mathcal{X}^*$ following

$$ \hat{y} = \sigma(f(\mathbf{w}, \mathbf{x}) + b) \qquad (3.1) $$

and weights $\mathbf{w}$ and bias $b$ are referred to as parameters of the neuron.

## Non-linear activation

The non-linearity introduced in a neural response is of utmost importance in deep learning as it is the core reason a neuron can represent (and learn) complex relationships between its outputs and inputs.

There is a wide range of options for choosing the activation function $\sigma$, three of which will be used in this manuscript:

* The Rectified Linear Unit (ReLU) introduced by MAAS et al. (2013) and defined $\forall x \in \mathbb{R}$ as

$$ \text{ReLU}(x) = \max(0, x) \qquad (3.2) $$

* The hyperbolic tangent, smoother than ReLU and with a bounded output range $[-1, 1]$, such that $\forall x \in \mathbb{R}$,

$$ \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} \qquad (3.3) $$

* The softmax function, which returns a probability distribution (hence a bounded output range $[0, 1]$) from the inputs it receives, i.e., $\forall \mathbf{x} \in \mathbb{R}^T$, $\text{softmax}(\mathbf{x}) \in \mathbb{R}^T$ with

$$ \text{softmax}(\mathbf{x})_t = \frac{\mathbf{x}_t}{\sum_{t=0}^{T-1} \mathbf{x}_t} \qquad (3.4) $$

where the $t \in \{0, \dots, T-1\}$ subscript denotes the $t$th element of its associated vector.
Deep Neural Networks (DNN)

Eq. (3.1) represents the response of a neuron to an excitation. In practice, there is hardly ever a single neuron but rather a *layer* of neurons simultaneously excited. For a given layer $\ell$ composed of $n \in \mathbb{N}$ neurons, the previous formalism remains identical when considering the weights of the entire layer $\mathbf{W}_\ell \in \mathcal{W}^n$ as the ensemble of all neurons' weights, their associated response function $\mathbf{f}$, and the outputs $\hat{\mathbf{y}}_\ell \in \mathbb{R}^n$ and bias $\mathbf{b}_\ell \in \mathbb{R}^n$ as vectors instead of scalars:

$$ \hat{\mathbf{y}}_\ell = \mathbf{f}_\ell(\mathbf{x}) = \sigma(\mathbf{f}(\mathbf{W}_\ell, \mathbf{x}) + \mathbf{b}_\ell). \qquad (3.5) $$

The weights $\mathbf{W}_\ell$ and bias $\mathbf{b}_\ell$ are the parameters of the neural layer and can be changed to orientate the outputs $\hat{\mathbf{y}}_\ell$ towards certain values, as it will be further detailed.

In the human brain, layers of neurons are inter-connected to one another, entangled in a intricate *network*, and propagate information step by step up to specialized regions of the brain. Artificial Neural Networks (ANN) follow the same approach as they are defined by a succession of layers, each composed of several neurons. Deep Neural Networks (DNN) are artificial networks sufficiently *deep*, that is composed of “many” layers – although there is no concrete consensus on how much “many” should be.

Layer after layer, more and more parameters and nonlinearities allow increasingly complex modeling of the data $\mathbf{x}$ towards the final prediction $\hat{\mathbf{y}}$. For $\ell = 0, \dots, l-1$ indexing the layers, the prediction of the DNN, which is the output last layer, is expressed as:

$$ \hat{\mathbf{y}} = \underbrace{(\mathbf{f}_{l-1} \circ \dots \circ \mathbf{f}_{\ell} \circ \dots \circ \mathbf{f}_0)}_{=f_{\Theta}}(\mathbf{x}) = \mathbf{f}_{\Theta}(\mathbf{x}). \qquad (3.6) $$

where $\Theta$ refer to all trainable parameters $\{\mathbf{W}_\ell, \mathbf{b}_\ell\}_{\ell \in [0, l-1]}$ – often simply called *weights* through misuse of language. The neural prediction can then be supervised, i.e., compared to a reference.

Supervised learning

*Supervised learning* is a paradigm in which data are annotated and these ground truths are used as references to evaluate and update the weights and bias of deep neural networks. Given a dataset $\mathcal{D}$ of labeled input-output pair $(\mathbf{x}, \mathbf{y})$, the label $\mathbf{y}$ associated with representation $\mathbf{x}$ is what is expected to be predicted by the final output $\hat{\mathbf{y}} = \mathbf{f}_{\Theta}(\mathbf{x})$ of a neural model.

It is desired that $\mathbf{f}_{\Theta}$ perfectly fits an unknown function $\mathbf{f}^*$ that fully reports on the connection between the labels $\mathbf{y}$ and the representations $\mathbf{x}$, i.e., $\mathbf{f}^*(\mathbf{x}) = \mathbf{y}$. This is barely possible in practice as the design choices for $\mathbf{f}_{\Theta}$ typically induce *inductive bias* preventing from *exactly matching* $\mathbf{f}^*$. The *universal approximation theorem*, though, states that any neural network properly configured can represent a whole family of functions with only few hypotheses on the activation function of each neuron (verified by, e.g., ReLU, tanh and softmax). Deep neural networks are therefore good candidates for *approximating* via $\mathbf{f}_{\Theta}$ the underlying function $\mathbf{f}^*$.
This research is exclusively based on supervised learning but systems can also rely on semi-supervised learning (ZHU, 2005) when only some part of the data are annotated or even unsupervised learning (BARLOW, 1989) when no annotation is accessible. Supervised learning is heavily dependent on the quality of the annotations as they are, by nature, considered as references to predict. Wrongly labeled pair can therefore induce severe errors. Annotation checking is in itself an activate research field in deep learning (MESEGUER-BROCAL et al., 2020a; RIDZUAN and ZAINON, 2019). In this work, however, all annotations are considered correct even if manual inspections or automatic checking of the labels are not systemically done.

## Loss function

In order to determine to which extent the model prediction $f_{\Theta}$ is a relevant approximation of $f^{*}$, and tune its parameters towards a better fit if need be, a quantitative measure of the “proximity” between $f^{*}$ and $f_{\Theta}$ is essential.

This is done by defining a *loss function* $\mathcal{L}(\Theta)$ – or *cost function*, *objective function* or *error function* – which compares the neural prediction to the ground truth for each pair $(x, y)$ contained in the dataset $\mathcal{D}$ given some distance $d$ (not necessarily a *distance* in the mathematical sense). It reads:

$$ \mathcal{L}(\Theta) = \sum_{(x,y) \in \mathcal{D}} d[\hat{y} || y] = \sum_{(x,y) \in \mathcal{D}} d[f_{\Theta}(x) || y]. \qquad (3.7) $$

The choice of the cost/distance $d$ depends on the task at hand. Several distances will be manipulated in this manuscript, as a core part of this research is precisely dedicated to emphasize a correct combination of cost functions. These will be introduced whenever necessary.

## Optimization

The goal of a deep learning algorithm is to minimize the value of its associated loss function $\mathcal{L}(\Theta)$ as the smaller the loss, the better $f_{\Theta}$ fits the true function $f^{*}$ to uncover. Concretely, one is looking for the theoretical optimal parameters $\Theta^{*}$ such that

$$ \Theta^* = \underset{\Theta}{\operatorname{argmin}} \mathcal{L}(\Theta). \qquad (3.8) $$

This is an *optimization* problem which can be solved thanks to numerical methods allowing to progressively update the parameters $\Theta$ while diminishing the loss function – with the aim to eventually reach sufficiently adequate weights and bias (i.e., finding a *local* minimum close enough to the *global* minimum).
Back-propagation & gradient descent

Back-propagation algorithms (RUMELHART et al., 1986) are commonly used to perform the optimization of a model. Their philosophy is to “learn from made mistakes”.

To do so, given the current weights and bias $\Theta$, the loss value $\mathcal{L}(\Theta)$ is computed. Then, its partial derivate or gradient with respect to the model’s parameters is calculated, i.e.,

$$ \nabla_{\Theta} \mathcal{L}(\Theta) = \frac{\partial \mathcal{L}}{\partial \Theta}(\Theta). \qquad (3.9) $$

As a positive gradient indicates the direction augmenting the error for parameters $\Theta$, the opposite of the gradient is the direction to follow for decreasing the loss.

The deep learning community has extensively relied on gradient descent to update step by step the parameters $\Theta$ in this direction. The $k$th modification of the weights and bias is

$$ \Theta_k \leftarrow \Theta_{k-1} - \lambda \nabla_{\Theta} \mathcal{L}(\Theta_{k-1}). \qquad (3.10) $$

This is known as a training step in which $\lambda$ – the learning rate – regulates the strength of the update. Training deep neural networks consists in successively applying this procedure.

More sophisticated and adaptive gradient descent algorithms have been proposed in order to both speed up convergence and avoid local minima. An efficient approach is to not only consider the gradient, but also its momentum (e.g., mean and variance) in the update rule. This is the strategy employed by the ADAM optimizer (KINGMA and BA, 2014). Ubiquitous in the literature, it will be systemically used for training in this work.

Minibatch gradient descent

In practice, the datasets $\mathcal{D} = \{(\mathbf{x}^{(i)}, \mathbf{y}^{(i)})\}_i$ are too large to allow computation of the loss value as expected in Eq. (3.7), given the limited resources and available memory on modern machines.

An alternative is to rely on minibatch gradient descent to compute an approximate of the loss and its gradient. Instead of considering the whole database $\mathcal{D}$, a few numbers of samples (i.e., a minibatch) are randomly selected and the loss function is calculated only on this subset of examples. The gradient can be computed and the weights and bias updated following above-mentioned procedure. The size of the minibatch – the batch size – is a key hyperparameter for training a model, very much like the learning rate.

Model generalization

Measuring the loss value during the training procedure does not indicate the generalization properties of the model, i.e., its capacity to predict coherent results when applied to new data outside of the dataset $\mathcal{D}$.
Indeed, the network may become too specialized on the data seen during training and unable to deal with data outside of this scope. In order to prevent such an *over-fitting*, the dataset *D* is generally split into two subsets referred to as the *training set* and the *validation set*.

The training set is used to update the model parameters via minibatch gradient descent. The validation set, containing different data, ensures that the model still generalizes by monitoring that the validation loss (or a validation metric) keeps decreasing alongside the training loss. The training procedure can be interrupted as soon as over-fitting is detected: this is known as *early stopping*.

Another prevalent option to counter the appearance of over-fitting is *dropout* (SRIVASTAVA et al., 2014), which consists in randomly deactivating several neurons between two layers $ℓ$ and $ℓ + 1$. This simple strategy allows independent and efficient joint learning of the many sub-network composing the whole network.

### 3.1.3 Architecture design

Previous section presented Deep Neural Networks (DNN) and their training to optimize the parameters $\Theta$ defining an approximative, yet robust, predictive function $f_{\Theta}$. According to Eq. (3.6), this learned function can be decomposed into many successive layers of neurons, $f_{\ell}$ with $\ell \in \{0, \dots, l - 1\}$. Among the available options to design the architecture of a neural layer, each associated with a different $f_{\ell}$, the ones relevant to this thesis are exposed.

#### Dense

A *dense* layer – or fully connected layer – connects all neurons to the input representations with a simple multiplicative weighting:

$$ \hat{\mathbf{y}}_{\ell} = \mathbf{f}_{\ell}(\mathbf{x}) = \sigma(\mathbf{W}_{\ell}\mathbf{x} + \mathbf{b}_{\ell}). \qquad (3.11) $$

Dense layers are the building block of Feed-Forward Neural Networks and remain often used as final layer in many practical cases.

#### Convolutional Neural Networks (CNN)

Convolutional Neural Networks (CNN) (LECUN et al., 1995) exploit convolution operation – as presented in Eq. (2.12) or Eq. (2.13) according to data dimension –, to extract abstractions from filtering the data. The learnable weights $\mathbf{W}_{\ell}$ constitute the convolution kernel, such that

$$ \hat{\mathbf{y}}_{\ell} = \mathbf{f}_{\ell}(\mathbf{x}) = \sigma(\mathbf{W}_{\ell} * \mathbf{x} + \mathbf{b}_{\ell}). \qquad (3.12) $$

Frequently, a single convolutional layer $\ell$ is *multi-channel*, meaning that multiple kernels are learned and applied to the same input $\mathbf{x}$, thus resulting in several filtering of the data.
Due to the nature of the convolution operation, the dimensions on which the kernel(s) is(are) slid across are susceptible to change. Padding of the input data can be done before the convolution to keep the(se) dimension(s) identical. This is systematically done in this thesis.

## Recurrent Neural Networks (RNN)

Recurrent Neural Networks (RNN) (RUMELHART et al., 1986) are specifically designed to process sequential data by introducing internal memory and dependence between components of the prediction. Denoting $\mathbf{x} = \{\mathbf{x}^{(0)}, \dots, \mathbf{x}^{(t)}, \dots, \mathbf{x}^{(T-1)}\}$ and $\hat{\mathbf{y}}_{\ell} = \{\hat{\mathbf{y}}_{\ell}^{(0)}, \dots, \hat{\mathbf{y}}_{\ell}^{(t)}, \dots, \hat{\mathbf{y}}_{\ell}^{(T-1)}\}$, a typical recurrent layer computes:

$$ \forall t \in \{0, \dots, T-1\}, \quad \hat{\mathbf{y}}_{\ell}^{(t)} = \sigma \left( \mathbf{W}_{\ell}^{y} \hat{\mathbf{y}}_{\ell}^{(t-1)} + \mathbf{W}_{\ell}^{x} \mathbf{x}^{(t)} + \mathbf{b}_{\ell} \right). \quad (3.13) $$

The learnable weights are thus separated into recurrent weights processing both current input vector $\mathbf{W}_{\ell}^{x}$ and previous internal output state $\mathbf{W}_{\ell}^{y}$. If causaliy is not a key requirement for the foreseen application, one can rely on bidirectional recurrent layer (SCHUSTER and PALIWAL, 1997) by considering Eq. (3.13) for the reserved input sequences $\mathbf{x}_{r} = \{\mathbf{x}^{(T-1)}, \dots, \mathbf{x}^{(0)}\}$ and concatenate causal and anti-causal predictions.

Well-known drawbacks of basic RNNs are the vanishing and exploding gradients problems induced by their recurrences (BENGIO et al., 1993). As a result, RNN cells with internal memory reset were introduced to specifically address these limitations. The Long Short-Term Memory (LSTM) (HOCHREITER and SCHMIDHUBER, 1997) is one of the widely used option.

## Attention mechanism

Attention mechanisms allow to learn the relative importance between the components of two sequences and integrate this information into a third sequence. They were introduced by BAHDANAU et al. (2014) for the machine translation task, a problem for which the structure between input and output sequences may differ and a general context (e.g., at the sentence level) can be highly beneficial to learn the structural changes.

A BAHDANAU attention (also known as additive attention) layer, therefore, learns a weighting between a query vector $\mathbf{q} \in \mathbb{R}^{T \times E}$ and a key vector $\mathbf{k} \in \mathbb{R}^{M \times E}$ and generates an attention context from a value vector $\mathbf{v} \in \mathbb{R}^{M \times E'}$. It reads, with T the transpose operator,

$$ \hat{\mathbf{y}}_{\ell} = \mathbf{f}_{\ell}(\mathbf{k}, \mathbf{q}, \mathbf{v}) = \mathrm{softmax} \left( \mathbf{q} \mathbf{W}_{\ell} \mathbf{k}^{\mathrm{T}} + \mathbf{b}_{\ell} \right) \mathbf{v}. \quad (3.14) $$

Note that the query and the key must share their last dimension $E$ and that the key and the value must share their first dimension $M$. In practice, the key and the value are often equal $\mathbf{k} = \mathbf{v}$. Self-attentions are also considered when $\mathbf{k} = \mathbf{q} = \mathbf{v}$. Architectures relying partially – or even solely – on (self-)attention mechanisms, e.g., Transformer (VASWANI et al., 2017), have become popular design choices for DNN.
<figure><img src="image_25.png" /><figcaption>Figure 3.1: A generic deep neural network. Inputs **x** are turned into predictions **ŷ** thanks to the modeling function **f**_Θ whose parameters Θ are updated via gradient descent on the loss function **L** measuring the proximity between **ŷ** and ground truth **y**.</figcaption></figure>

Batch Normalization (BN)

Batch Normalization (BN) is a method proposed by IOFFE and SZEGEDY (2015) to fasten the training of Artificial Neural Networks (ANN) and make this procedure more stable. It consists in normalizing the inputs received by a layer by re-centering and re-scaling of its values.

Section summary – Deep learning background

With the emergence of large annotated datasets $\mathcal{D} = \{(\mathbf{x}^{(i)}, \mathbf{y}^{(i)})\}_i$ and progress in computer infrastructures, Deep Neural Networks (DNN) were shown to be worthy candidates to estimate the underlying function describing the relationships between representation-label pairs $\{(\mathbf{x}^{(i)}, \mathbf{y}^{(i)})\}_i$ with a non-linear, $\Theta$-parameterized function $\mathbf{f}_{\Theta}$. This function can be implemented via a succession of convolutional, recurrent, attention and/or dense layers connected with non-linear activations. The weights and bias $\Theta$ of the model are typically obtained through the progressive optimization, i.e., minimization, of a loss function $\mathcal{L}(\Theta)$ by minibatch gradient descent under ADAM update rule. The Figure 3.1 proposes a visual summary of all of these notions.
<figure><img src="image_26.png" /><figcaption>Figure 3.2: The voice alignment problem – finding the best time-symbol correspondence between two message emissions from audio (log-mel-spectrogram) and symbolic (texts, notes) domains.</figcaption></figure>

# 3.2 Voice alignment problem statement

In this short section, the voice alignment problem is clearly stated, with an emphasis on the preparation of the voice data.

## 3.2.1 Formalization

The Figure 3.2 depicts how the generic sequence-to-sequence alignment can be applied to the specific voice alignment problem.

In that case, data to synchronize represent an audio recording featuring voice with a temporal dimension of interest, and a symbolic sequence of characters (text alignment) or notes (melody alignment). The goal, in aligning these sequences, is to map each label (or symbol) to an instant and a duration or, which is equivalent, a start and end time markers.

The modeling function $\mathcal{M}$ aims to generate a timestamped encoding of the audio, i.e., it creates a hidden representation which conserves a temporal axis, essential for temporal alignment. As this model processes audio data only and aims to extract relevant voice-related parameters, it is often referred to as acoustic model by the speech community (HINTON et al., 2012). The resulting latent code, or timestamped encoding $\mathcal{M}(\mathbf{x})$, usually goes by the name of posteriorgram or saliency map so that notation $\mathcal{P} \equiv \mathcal{M}(\mathbf{x})$ will be chosen for such hidden encoding.
The decoding module, $D_\pi$ is generally based on Dynamic Time Warping (DTW) or equivalent. This work, however, will rely on a specific decoding module as it uses a specific framework, known as Connectionist Temporal Classification (CTC) presented in a later section.

### 3.2.2 Voice data

#### Voice representations

In this work, the inputs $x$ of all neural network are 2-D Time-Frequency Representations (TFR) such as (log-mel-) spectrograms or spectral envelope as introduced in section 2.2. They are accounting for the oral modality of a message communicated through voice (see section 2.1.3).

Their temporal axis is indexed by $t \in \{0, \dots, T-1\}$ where $T$ is the number of computed frames. Their frequency axis is indexed by $f \in \{0, \dots, F-1\}$ where $F$ is the number of bins.

#### Symbolic representations

The supervised labels $y$ a neural network aims to predict are sequences of symbols belonging to an alphabet $A$ of size $L$. In order to manipulate and process sequences of numbers, each element of $A$ is mapped to a unique figure of the ensemble $A' = \{0, \dots, L-1\}$.

An one-hot encoding of the labels is also possible to turn them into a 2-D representation: for a symbol $s \in A'$ a vector $[\delta_s^0, \dots, \delta_s^{L-1}]$ is generated with $\delta_s^k$ being the KRONECKER delta returning 1 when $s = k$ and 0 otherwise, and such vectors can be concatenated for each label.

The following examples illustrate these mappings, i.e., the sequence-to-number conversion and the final one-hot encoding:

$$ y = b a \emptyset a b \equiv 1 0 L-1 0 1 \equiv \begin{bmatrix} 0 & 1 & 0 & 1 & 0 \\ 1 & 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & 1 & 0 & 0 \end{bmatrix}. $$

The labels account for the symbolic modalities of a message communicated in a voice signal. As defined in section 2.1.2, the Latin alphabet $A_\ell$ will be used for characters and the note alphabet $A_\flat$ for melodic lines.

Their sequential axis is indexed by $m \in \{0, \dots, M-1\}$ where $M$ is the length of the label sequence. Their encoding axis, if present, is indexed by $l \in \{0, \dots, L-1\}$ with $L$ the alphabet size.
Section summary – Voice alignment problem statement

Voice alignment systems are based on a two-step principle made of:

* an acoustic model $\mathcal{M}$ generating a timestamped encoding $\mathcal{P} \equiv \mathcal{M}(\mathbf{x})$ of the audio voice recordings $\mathbf{x}$;

* and a decoding module $\mathcal{D}_\pi$ mapping the timing information from $\mathcal{P}$ to the target symbols to synchronize $\mathbf{y}$.

Concretely, inputs $\mathbf{x} \in \mathbb{R}^{T \times F}$ are Time-Frequency Representations (TFR) and labels $\mathbf{y} \in \{0, \dots, L-1\}^{M \times 1}$ or $\{0, 1\}^{M \times L}$ are symbolic sequences such as texts (lyrics, syllables, phonemes) or notes according to an alphabet $\mathcal{A}$ of size $L$.

## 3.3 Traditional approaches to voice alignment

Over the years, the forced alignment problem has been implemented with multiple strategies. In this section, an historical literature review presents the traditional approaches that have been used to tackle voice alignment. The decoding module $\mathcal{D}_\pi$ involved in an alignment system has been quite systematically based on Dynamic Time Warping (DTW) inheriting from sequence-to-sequence similarity research. The acoustic modeling $\mathcal{M}$, however, has been built upon several strategies from hand-crafted systems to the noteworthy emergence of stochastic approaches and notably Hidden Markov Models (HMM).

### 3.3.1 Dynamic Time Warping (DTW)

A naive determination of the alignment, as defined by the optimal path from Eq. (2.23) and illustrated in Figure 2.9, based on path exploration is usually intractable as there are too many possible paths between the two sequences $\mathbf{x} \in \mathcal{X}^*$ and $\mathbf{y} \in \mathcal{Y}^*$. Indeed, as reported by GARREAU et al. (2014), the set $\Pi(\mathcal{X}^*, \mathcal{Y}^*)$ contains all paths on a rectangular grid starting from the northwest (0,0) and ending at the southeast $(T-1, M-1)$ corners. Its cardinality defines the DELANNOY numbers (BANDERIER and SCHWER, 2005). In the case of an infinitely large square grid (i.e., $T = M$ and $T \to +\infty$), TORRES et al. (2003) showed that:

$$ \#\Pi(\mathcal{X}^*, \mathcal{Y}^*) \approx \frac{(3 + 2\sqrt{2})^T}{\sqrt{\pi T}\sqrt{3\sqrt{2} - 4}}. \qquad (3.15) $$

Despite being finite values, examples for $T = 50$ (10³⁷ paths) and for $T = 100$ (10⁷⁵ paths) clearly points out the impossibility to calculate each path independently and keep the best one.
In the early 1970s, researchers from various communities (NEEDLEMAN and WUNSCH, 1970; VELICHKO and ZAGORUYKO, 1970; VINTSYUK, 1968; VITERBI, 1967) needed to solve the sequence alignment problem but were confronted to this specific issue.

Interestingly, they all proposed a similar algorithm noticing that the best path could be computed indirectly via Dynamic Programming (DP) techniques. As a result, this algorithm was named Dynamic Time Warping (DTW) by speech specialists (VINTSYUK, 1968).

The core principle of DTW is that the optimal similarity measure between two sequences can actually be obtained recursively, considering the sequence prefixes at intermediate indexes $t$ and $m$, that is $\mathbf{x}_{0:t} = \{\mathbf{x}_0, \dots, \mathbf{x}_t\}$ and $\mathbf{y}_{0:m} = \{\mathbf{y}_0, \dots, \mathbf{y}_m\}$, respectively.

To this aim, an accumulative score matrix $\alpha \in \mathbb{R}^{T \times M}$ is defined so that $\alpha[t, m]$ represents the optimal alignment similarity between the prefixes $\mathbf{x}_{0:t}$ and $\mathbf{y}_{0:m}$. The computation of $\alpha$ involves the similarity measure $\mathcal{S}$ from Eq. (2.23), and the set of permitted gaps $\mathcal{G}$ between successive states introduced in Eq. (2.22).

The classical textbook DTW algorithm (ITAKURA, 1975) allows insertions, deletions and substitutions between the two sequences such that $\mathcal{G} = \{(1, 1), (0, 1), (1, 0)\}$. The initialisation steps and recursion rules for matrix $\alpha$ are therefore:

$$ 
\begin{align} 
t = 0 \quad \forall m \quad \alpha[0, m] &= \sum_{k=0}^{m} \mathcal{S}(\mathbf{x}_0, \mathbf{y}_k) \nonumber \\ 
\forall t \quad m = 0 \quad \alpha[t, 0] &= \sum_{k=0}^{t} \mathcal{S}(\mathbf{x}_k, \mathbf{y}_0) \tag{3.16} \\ 
\forall t > 0 \quad \forall m > 0 \quad \alpha[t, m] &= \mathcal{S}(\mathbf{x}_t, \mathbf{y}_m) + \max \left\{ \begin{array}{l} \alpha[t, m-1] \\ \alpha[t-1, m] \\ \alpha[t-1, m-1] \end{array} \right\} \nonumber 
\end{align} 
$$

In the context of temporal alignment (i.e., $t$ represents a time quantity), the matching between the sequences may involve a strict temporal causality such that $\alpha[t, m]$ cannot depend on $\alpha[t, m']$, $\forall m' \in \{1, \dots, m-1\}$. In that case, the permitted transitions are reduced to $\mathcal{G} = \{(1, 0), (1, 1)\}$. It comes:

$$ 
\begin{align} 
t = 0 \quad \forall m \quad \alpha[0, m] &= \sum_{k=0}^{m} \mathcal{S}(\mathbf{x}_0, \mathbf{y}_k) \nonumber \\ 
\forall t \quad m = 0 \quad \alpha[t, 0] &= \sum_{k=0}^{t} \mathcal{S}(\mathbf{x}_k, \mathbf{y}_0) \tag{3.17} \\ 
\forall t > 0 \quad \forall m > 0 \quad \alpha[t, m] &= \mathcal{S}(\mathbf{x}_t, \mathbf{y}_m) + \max \left\{ \begin{array}{l} \alpha[t-1, m] \\ \alpha[t-1, m-1] \end{array} \right\} \nonumber 
\end{align} 
$$

At each iteration, local scores are accumulated and only the best prefix of the optimal path is kept — this is known as the *forward pass*.
The global optimal path $\hat{\pi}^*$ between the sequences can be retrieved by backtracking the retained transitions that occurred during the forward pass — this is known as the backward pass. In the end, once all $\alpha$ s have been computed, the optimal alignment similarity is simply:

$$A^* = \alpha[T-1, M-1]. \qquad (3.18)$$

The DTW algorithm has found numerous applications in Music Information Retrieval (MIR) tasks (MÜLLER, 2007a) and audio alignment including melody search (MONGEAU and SANKOFF, 1990), music tracking (ARZT, 2016), score following (DIXON, 2005), audio matching (MÜLLER et al., 2005), beat tracking (ELLIS, 2007) or version identification (SERRA and GÓMEZ, 2007). In practice, the decoding module $D_\pi$ of a forced alignment system generally relies on DTW or one of its variant.

### 3.3.2 Pioneer acoustic modeling

In the first stages of voice alignment research, in the 1980s, pioneering works were exploiting hand-crafted strategies in order to force-align sequences to audio.

The synchronization of phonetic utterances to audio typically relied on (1) features extracted from the audio thanks to signal processing techniques; and (2) domain knowledge rules for the forced alignment step per se (LEUNG and ZUE, 1984; WAGNER, 1981).

Similarly, in the early appearances of score following in 1984 (DANNENBERG, 1984; VERCOE, 1984), the alignment challenge was actually dealt with string matching techniques in real-time instead of direct use of the audio stream or spectrograms, due to the limited computational capabilities at that time.

### 3.3.3 Hidden Markov Models (HMM)

A key noteworthy progress in Automatic Speech Recognition (ASR) and, consequently, audio alignment literature was the emergence of approaches based on probability theory during the 1990s (GRUBB and DANNENBERG, 1994; LJOLJE and RILEY, 1991; PLACEWAY and LAFFERTY, 1996). The main motivation for these was the growing necessity to handle all different kinds of uncertainties or unpredictable events in audio performances such as mispronunciation or mistakes or skips, e.g., (GUPTA et al., 2017; NAKAMURA et al., 2015).

Probabilistic approaches, therefore, tend to have better applicative flexibility than previous pioneering works. Most of them have been based on generative probabilistic models whose main assumption is that the audio recording has been generated by the events reported in the sequences to align, e.g., the music score or the list of phonemes.

Maybe the major breakthrough in alignment research was the application of Hidden Markov Models (HMM) to this task (BRUGNARA et al., 1993; RAPHAEL, 1999; VOGEL et al., 1996), leveraging the progress made in ASR systems with these models.
In the HMM framework, the events to align $\mathbf{y} = \{\mathbf{y}_0, \dots, \mathbf{y}_m, \dots, \mathbf{y}_{M-1}\}$ are manipulated via probabilistic *hidden states* $\mathbf{h} = \{\mathbf{h}_0, \dots, \mathbf{h}_t, \dots, \mathbf{h}_{T-1}\}$ such that $\forall t, \mathbf{h}_t \in \mathbf{y}$, that are not directly observable, but are expected to explain the audio frames $\mathbf{x} = \{\mathbf{x}_0, \dots, \mathbf{x}_t, \dots, \mathbf{x}_{T-1}\}$.

Determining the alignment between $\mathbf{x}$ and $\mathbf{y}$ through $\mathbf{h}$, requires to solve an inference problem, that is finding the most likely succession of events characterizing the audio. The formulation of the inference mechanism appears as a comprise between two important quantities: (1) a *prior evolution of hidden states* specifying, e.g., their temporal evolution; and (2) an *observation model*, which measures a likelihood between audio observations $\mathbf{x}$ and the labels from the sequence $\mathbf{y}$ through the hidden states $\mathbf{h}$. The inference rule is expressed thanks to BAYES's theorem on conditional probabilities,

$$ \underbrace{\mathbb{P}(\mathbf{h}|\mathbf{x})}_{\text{inference computation}} \propto \underbrace{\mathbb{P}(\mathbf{x}|\mathbf{h})}_{\text{observation model}} \underbrace{\mathbb{P}(\mathbf{h})}_{\text{prior evolution}} \qquad (3.19) $$

Note that since the quantity $\mathbb{P}(\mathbf{x})$, which should be part of the inference formula, is numerically independent of the hidden states to retrieve, it is usually dropped out of the computations.

As their name suggests, HMM assume the MARKOVIAN hypothesis, which stipulates that the audio at frame $t$ (i.e., $\mathbf{x}_t$) depends exclusively on its associated state (i.e., $\mathbf{h}_t$) and, more drastically, that the sequence of states $\mathbf{h}$ is memoryless as well, such that

$$ \mathbb{P}(\mathbf{x}|\mathbf{h}) = \mathbb{P}(\mathbf{x}_0, \dots, \mathbf{x}_{T-1}|\mathbf{h}_0, \dots, \mathbf{h}_{T-1}) = \prod_{t=0}^{T-1} \mathbb{P}(\mathbf{x}_t|\mathbf{h}_t). \qquad (3.20) $$

This assumption is highly disputable but is widely accepted in practice as it allows the inference mechanism to remain tractable all along.

The inferred values precisely correspond to the timestamped encoding of the audio such that, according to Figure 3.2, $\mathcal{M}(\mathbf{x}) = \mathbb{P}(\mathbf{h}|\mathbf{x})$, and a decoding algorithm $\mathcal{D}_\pi$ (like VITERBI's) can be applied on it to force-align the true labels $\mathbf{y}$ to the audio $\mathbf{x}$.

Extensive literature can be found on methods for designing coherent prior evolution of hidden states $\mathbb{P}(\mathbf{h})$, e.g., in score following (CUVILLIER, 2016; CUVILLIER and CONT, 2014), as well as defining robust observation models $\mathbb{P}(\mathbf{x}|\mathbf{h})$. The later emission probabilities are typically based on Gaussian Mixture Models (GMM) to compute a distance between prototypical and true audio features (CONT, 2009; LJOLJE and RILEY, 1991; PLACEWAY and LAFFERTY, 1996; RAPHAEL, 2006).

Such HMM-GMM aligners are ubiquitous in the voice alignment literature – e.g., (DUAN et al., 2013; GONG et al., 2015; GORMAN et al., 2011; HOSOM, 2009; MCAULIFFE et al., 2017; ROSENFELDER et al., 2017).
### Section summary – Traditional approaches to voice alignment

Dynamic Time Warping (DTW) – or VITERBI’s decoding – is a general sequence-to-sequence alignment algorithm which constructs, recursively, an accumulative cost matrix based on local distances and permitted transitions between the sequences (*forward pass*) and retraces their alignment path from it (*backward pass*). In the light of its success, DTW (or a close variant) is a common choice for the decoding module $\mathcal{D}_{\pi}$. The acoustic modeling $\mathcal{M}$, on the contrary, has known various implementations, notably through Hidden Markov Models (HMM) coupled with Gaussian Mixture Models (GMM) that, by their probabilistic nature, could handle uncertainties when modeling the sequential structure of voice signals, despite assuming the disputable MARKOVIAN hypothesis.

## 3.4 Neural approaches to voice alignment

With the advent and success of deep learning in many fields – their data-driven approach outperforming various states of the art –, an increasing number of audio and voice-related tasks started adopting Deep Neural Networks (DNN) in their design. This is notably the case of the Automatic Speech Recognition (ASR) community in the 2010s (MOHAMED, 2014).

At first, DNN were integrated in hybrid systems in which they were coupled with Hidden Markov Models (HMM). In these dual DNN-HMM architecture, DNN had the role to overcome the limitations of Gaussian Mixture Models (GMM) that were struggling to model complex data with high dimensions (MOHAMED et al., 2011). Then, end-to-end architectures were introduced so that HMM were completely removed from the ASR systems. In doing so, voice acoustic modeling was freed from the MARKOV assumption (GRAVES et al., 2013) and could benefit from the DNN capacity to capture long-term relationships in the voice signals. In opposition to earlier HMM-based models, end-to-end DNN-based systems are completely free from specialized and often complex expert knowledge (HANNUN et al., 2014). As a result, their data processing pipeline, training and inference usage are much simpler to handle than traditional or specialized approaches, yet not consistently better: the state of the art in lyrics transcription/alignment does integrate expert knowledge (GAO et al., 2021; GUPTA et al., 2020).

Given the proximity between voice recognition and voice alignment problems, the success of DNN reported in the ASR literature motivated their progressive adoption for voice-to-text synchronization. In such frameworks, the acoustic modeling function $\mathcal{M} \equiv \mathcal{M}_{\Theta}$ is parametrized by neural weights $\Theta$, and the resulting timestamped encoding, or posteriorgram, $\mathcal{P} = \mathcal{M}_{\Theta}(\mathbf{x})$ can be fed to the decoding module $\mathcal{D}_{\pi}$, always referring to Figure 3.2.

As explained in section 3.1, the design of a training procedure implies the choice of loss function $\mathcal{L}(\Theta)$. Several approaches, based on different losses, for achieving forced alignment with DNN have been proposed in complement to standard methods and are thus presented.
### 3.4.1 Frame-wise classification

Early attempts in the deep learning-based voice alignment direction relied on an intuitive strategy consisting in letting the model predict one label per time step and ensuring that the correct symbol is effectively emitted at each instant. From this point of view, the alignment challenge is a frame-wise classification problem.

In this regard, by predicting and supervising symbol emission at the frame level, both the model outputs $\hat{y} = M_{\Theta}(x)$ and the ground truth reference $y$ must be of the same length as the audio (i.e., $M = T$), by means of duplicating all elements of the target sequence such that the duplicated elements align with the audio. The loss function, as generally presented in Eq. (3.7), to minimize for such a multi-category classification task, is based on a distance $d$ known as the Categorical Cross-Entropy (CCE) such that

$$ \mathcal{L}(\Theta) = \sum_{(x,y) \in \mathcal{D}} d[\hat{y} || y] = \sum_{(x,y) \in \mathcal{D}} -y \log(\hat{y}) = \sum_{(x,y) \in \mathcal{D}} \left[ -\sum_{t=0}^{T-1} y_t \log(\hat{y}_t) \right]. \quad (3.21) $$

Although great performances are to be denoted with this strategy (BACKSTROM et al., 2019; KELLEY and TUCKER, 2018), such training procedure calls for data in the form of paired audio and already precisely time-aligned text. This is a major shortcoming since only few accessible datasets provide that level of annotations, thus the data diversity and contexts seen by the model is intrinsically limited – as the process of manual labelling is tedious and extremely time-consuming.

Therefore, different approaches that could be trained with more widespread data, such as paired audio and text without precise timing information – that are available for, e.g., Text-to-Speech (TTS) algorithms –, were investigated.

### 3.4.2 Attention mechanism

SCHULZE-FORSTER et al. (2020) precisely developed a system permitting the alignment between audios and their respective phonetic transcripts without requiring timestamps.

With the objective to isolate clean speech even in very noisy conditions, the authors proposed an encoder-decoder architecture based on Bidirectional LSTM (Bi-LSTM) and integrating a key attention mechanism (see section 3.1.3).

By taking both the audio $x$ of length $T$ and non-aligned sequences $y$ of length $M$ as inputs, the learnable weights involved in the attention mechanism $\alpha \in [0, 1]^{T \times M}$ turned out to resemble the accumulative score matrix intervening in the forward pass of the DTW algorithm (see section 3.3.1). An alignment path could thus directly be derived from $\alpha$ through a backward pass.

The similar procedure have been applied to singing voice, however, due to its more challenging nature than speech (cf. section 2.2.3), the attention weights $\alpha$ needed to be constrained to follow a monotonic path prior to any backward pass.
The authors proposed to directly impose a DTW-alike progression in the attention weights, thus introducing a new “DTW attention mechanism” (SCHULZE-FORSTER et al., 2021).

Summing up, voice-to-text alignment has been obtained as a positive side effect of voice separation, minimizing during training a simple loss function $\mathcal{L}(\Theta)$ measuring that the extracted vocals were close to the true vocals by means of a classical L1 distance choice for $\mathcal{d}$.

### 3.4.3 Connectionist Temporal Classification (CTC)

Another approach for achieving voice alignment without the need of hard-labelled audio-symbol pairs was inspired from advances in end-to-end Automatic Speech Recognition (ASR) systems with a specialized cost function known as Connectionist Temporal Classification (CTC).

The CTC loss was initially introduced to train Recurrent Neural Networks (RNN) on un-segmented data (GRAVES et al., 2006) and became, ever since, tremendously popular in the speech recognition literature (COLLOBERT et al., 2016; GRAVES and JAITLY, 2014; HORI et al., 2017; KIM et al., 2017; WATANABE et al., 2017; ZHANG et al., 2017).

Given some audio inputs $\mathbf{x}$, a CTC-trained neural network $\mathcal{M}_{\Theta}$ generates a posteriorgram $\mathcal{P} = \mathcal{M}_{\Theta}(\mathbf{x})$ which takes the form of per-frame discrete probability distributions over a finite alphabet of labels (e.g., graphemes, phonemes, notes). In a speech recognition context, the temporal information contained in the posteriorgram is usually discarded as only an estimate $\hat{\mathbf{y}}$ of the target sequence $\mathbf{y}$ must be predicted but, yet again, this timestamping can explicitly be leveraged to force-align a ground-truth sequence $\mathbf{y}$ to the audio via a DTW-alike decoding algorithm $\mathcal{D}_{\pi}$. Concretely, CTC-based architectures have paved the way for the development of new, end-to-end alignment systems.

Regarding the alignment of *singing* utterances, and to the best of the author's knowledge, STOLLER et al. (2019) were the first to explore such a CTC strategy and exploit the temporal axis of posteriorgrams to align raw audio waveforms, with background music, to their text transcripts. The acoustic modeling of the audio samples was a Wav-U-Net (STOLLER et al., 2018) trained on a private dataset of more than 40k songs. Pursuing along the same trend, VAGLIO et al. (2020a) proved that aligning singing voice with CTC could be extended to a multi-lingual context, even for languages with almost no training data, along with the robustness of CTC acoustic modeling for various tasks (RENAULT et al., 2021; VAGLIO et al., 2020b, 2021).

Regarding the alignment of *speech* utterances, KÜRZINGER et al. (2020) showed that a CTC-based model, originally trained for ASR, could outperform legacy HMM-based methods for the automatic segmentation of large speech corpora into small audio excerpts and sentence-level texts.

In the light of its flexibility, the alignment systems proposed in this work are built upon a CTC strategy. Such a framework, though, has its own mathematical specificities and a decoding algorithm close but different to the classical DTW. The next section is dedicated to these notions.
Section summary – Neural approaches to voice alignment

Deep learning-based voice alignment consists in approximating the acoustic modeling function with a neural network $M_\Theta$, parameterized by $\Theta$, capable of extracting high-level and powerful features from the audio. Among the existing approaches found in the literature, i.e., frame-wise classification (requiring hard labelling, i.e., time-stamped ground truth sequences) and attention mechanisms (requiring an auxiliary task, e.g., voice separation), the Connectionist Temporal Classification (CTC) is particularly appealing as it has shown great potential for end-to-end voice-to-sequence alignment with soft labelling. As a result, CTC is at the core of the acoustic models developed in this thesis.

## 3.5 CTC-based voice alignment

In the applicative context of end-to-end voice-to-sequence alignment, acoustic models trained to minimize the Connectionist Temporal Classification (CTC) loss are a pertinent choice. Proposed by GRAVES et al. (2006) in order to train a neural network for labelling unsegmented data, CTC introduced its specific formalism whose main definitions and original ideas relevant to this research are summarized in this section.

This background requires, on one hand, to detail the original CTC modeling $M_\Theta$ for sequence transcription and, on the other hand, to present the CTC adaptation of a classic, DTW-based decoding module $D_\pi$ for deriving alignment from the timing information of CTC predictions.

### 3.5.1 CTC-based acoustic modeling

Let $\mathcal{A}$ be a finite alphabet of symbols of size $L$ associated with the set of sequences $\mathcal{A}^*$ such that

$$y = \{y_m\}_{m \in \{0, \dots, M-1\}} \in \mathcal{A}^* \iff \forall m \in \{0, \dots, M-1\}, y_m \in \mathcal{A}.$$

Elements of $\mathcal{A}^*$ are referred to as *labellings*. Let $\mathcal{A}_\varepsilon$ denote the alphabet $\mathcal{A}$ extended with a blank label $\varepsilon$, and $\mathcal{A}_\varepsilon^*$ be the set of all sequences made of labels from $\mathcal{A}$ and blanks $\varepsilon$. Elements of $\mathcal{A}_\varepsilon^*$ are referred to as (labelling) extensions.

A neural network $M_\Theta$ supervised by CTC typically receives as input a sequence $\mathbf{x} \in \mathbb{R}^{T \times F}$ from which it generates a posteriorsgram $\mathcal{P} \in [0, 1]^{T \times (L+1)}$. The posteriorsgram results from the application of a softmax activation, according to Eq. (3.11), on the last dimension, such that each of its frames $\mathcal{P}[t], t \in \{0, \dots, T-1\}$ represents the emission probability distribution over the alphabet $\mathcal{A}_\varepsilon$, i.e., over $L+1$ symbols including the blank label $\varepsilon$.

Consequently, $\mathcal{P}[t, l] \in [0, 1]$ can be interpreted as the probability that the label $l \in \mathcal{A}_\varepsilon$ is emitted at the time instant $t$ in the input sequence $\mathbf{x}$.
The probability of observing a labelling extension $\mathbf{l} \in \mathcal{A}_{\varepsilon}^{*}$ of length $T$, $\mathbf{l} = \{\mathbf{l}_{0}, \dots, \mathbf{l}_{T-1}\}$, conditionally on the inputs $\mathbf{x}$ associated with posteriorgram $\mathcal{P}$, is expressed as

$$ \mathbb{P}(\mathbf{l}|\mathbf{x}) = \prod_{t=0}^{T-1} \mathcal{P}[t, \mathbf{l}_t]. \qquad (3.22) $$

Let $\mathcal{B}: \mathcal{A}_{\varepsilon}^{*} \to \mathcal{A}^{*}$ denote the CTC mapping operator which, for any extension $\mathbf{l} \in \mathcal{A}_{\varepsilon}^{*}$, first merges all successively repeated symbols into one and then discards the blank labels $\varepsilon$, e.g.,

<table><thead><tr><th>1 ∈ A<sub>ε</sub><sup>*</sup> for T = 15</th><th>merge repetitions</th><th>discard blanks</th><th>B(1) ∈ A<sup>*</sup></th></tr></thead><tbody><tr><td>hεeeeeεεllεlεoo</td><td>hεeεlεlεo</td><td>hello</td><td>hello</td></tr><tr><td>hhhhεεεεelεlloo</td><td>hεelεlo</td><td>hello</td><td>hello</td></tr><tr><td>εεhheeεllεlεoεε</td><td>εheεlεlεoε</td><td>hello</td><td>hello</td></tr></tbody></table>

The previous examples illustrate the importance of the blank label $\varepsilon$ is the CTC framework. Not only does it allow to separate the labels, even handling successive identical ones (e.g., “ll” in “hello”), but also does not appear in the final predictions such that $\varepsilon$ offers flexibility while constructing the posteriorgram $\mathcal{P}$. Due to its non-informative nature, embodying the choice not to specify any symbol from $\mathcal{A}$ at a given frame, predicting high probabilities for the blank label is an easy option for the network during its learning phase to explore without hard penalization (because blanks are not in the final output in opposition to a wrongly recognized symbol).

The operator $\mathcal{B}$ thus performs a many-to-one mapping as the same labelling $\mathbf{y} \in \mathcal{A}^{*}$ of size $M$ can be obtained from several extensions $\mathbf{l} \in \mathcal{B}^{-1}(\mathbf{y})$ of size $T \ge M$. The conditional probability of observing the labelling $\mathbf{y}$ given the audio $\mathbf{x}$ is therefore the sum of all its associated extensions, i.e.,

$$ \mathbb{P}(\mathbf{y}|\mathbf{x}) = \sum_{\mathbf{l} \in \mathcal{B}^{-1}(\mathbf{y})} \mathbb{P}(\mathbf{l}|\mathbf{x}). \qquad (3.23) $$

A CTC-trained neural network aims to predict the labelling $\hat{\mathbf{y}} \in \mathcal{A}^{*}$ maximizing Eq. (3.23),

$$ \hat{\mathbf{y}} = \underset{\mathbf{y} \in \mathcal{A}^{*}}{\operatorname{argmax}} \mathbb{P}(\mathbf{y}|\mathbf{x}) \qquad (3.24) $$

or, which is equivalent, mimizing the associated negative-log-likelihood, which defines the CTC training criterion:

$$ \hat{\mathbf{y}} = \underset{\mathbf{y} \in \mathcal{A}^{*}}{\operatorname{argmin}} -\log \mathbb{P}(\mathbf{y}|\mathbf{x}). \qquad (3.25) $$

Such an optimization algorithm, very much alike DTW (see section 3.3.1), is intractable if approached naively – the cardinal of $\mathcal{B}^{-1}(\mathbf{y})$ being huge for a given labelling $\mathbf{y}$. Here again, Dynamic Programming (DP) techniques must be used to compute the CTC loss function, which is

$$ \mathcal{L}_{\text{CTC}}(\Theta) = \sum_{(\mathbf{x}, \mathbf{y}) \in \mathcal{D}} -\log \mathbb{P}(\mathbf{y}|\mathbf{x}; \Theta) = \sum_{(\mathbf{x}, \mathbf{y}) \in \mathcal{D}} -\log \left[ \sum_{\mathbf{l} \in \mathcal{B}^{-1}(\mathbf{y})} \mathbb{P}(\mathbf{l}|\mathbf{x}; \Theta) \right]. \qquad (3.26) $$
<figure><img src="image_27.png" /><figcaption>Figure 3.3: Transitions allowed in the CTC framework between labels and blanks, for the toy example target sequence “ab” with interleaved blanks $\epsilon$.</figcaption></figure>

In the specific CTC framework, however, the recursion rule for efficiently computing the probabilities is a bit different than DTW due to the introduction and major role of the blank label $\epsilon$. Indeed, the possible value of the $t$th element $l_t$ of an extension $l \in A_\epsilon^*$ (associated with a labelling $y \in A^*$) depends on the previous element $l_{t-1}$:

* When coming from a blank $\epsilon$, an extension can either stay on the same blank $\epsilon$ or transition to a label $y_m$, i.e.,

$$l_{t-1} = \epsilon \implies l_t \in \{\epsilon, y_m\}. \qquad (3.27)$$

* When coming from a label $y_{m-1}$, an extension can either stay on the same label $y_{m-1}$, transition to a blank label $\epsilon$ or jump directly to the next label $y_m$, i.e.,

$$l_{t-1} = y_{m-1} \implies l_t \in \{y_{m-1}, \epsilon, y_m\}. \qquad (3.28)$$

In order to take into account these transitions between blanks and non-blank labels, illustrated in Figure 3.3, GRAVES et al. (2006) defined a new sequence from $y$, by adding a blank at the beginning and the end of it and interleaving blanks between every label. This sequence, denoted $\tilde{y} = \{\tilde{y}_s\}_{s \in \{0,...,S-1\}}$, has therefore a length $S = 2M + 1$, and

$$\tilde{y} = \{\epsilon, y_0, \epsilon, \dots, \epsilon, y_m, \epsilon, \dots, \epsilon, y_{M-1}, \epsilon\}. \qquad (3.29)$$
<figure><img src="image_28.png" /><figcaption>Figure 3.4: CTC loss computation via Dynamic Programming (DP) for the toy example “ab”. Adaptation of the figure proposed by HANNUN (2017).</figcaption></figure>

This extended sequence is used to compute $\mathbb{P}(\mathbf{l}|\mathbf{x})$ dynamically with a reasoning resembling DTW - i.e., through an accumulative score matrix $\alpha$ between the (prefixes of the) sequences $\tilde{\mathbf{y}}$ and $\mathbf{x}$. Given the permitted transitions from Eq. (3.27) and Eq. (3.28), and as illustrated in Figure 3.4, $\alpha \in \mathbb{R}^{T \times S}$ is computed following

$$ \forall t \in \{1, \dots, T-1\}, \forall s \in \{2, \dots, S-1\} \\ \alpha[t, s] = \mathbb{P}[t, \tilde{\mathbf{y}}_s] \times \begin{cases} \alpha[t-1, s] + \alpha[t-1, s-1] & \text{if } \tilde{\mathbf{y}}_s \in \{\epsilon, \tilde{\mathbf{y}}_{s-2}\} \\ \alpha[t-1, s] + \alpha[t-1, s-1] + \alpha[t-1, s-2] & \text{otherwise} \end{cases} \tag{3.30} $$

with the initialization

$$ \begin{aligned} \alpha[t, 0] &= 0 \quad \forall t \ge 0 \\ \alpha[0, s] &= 0 \quad \forall s \ge 0. \end{aligned} \tag{3.31} $$

Finally, considering that a given extension $\mathbf{l}$ can terminate either with the final blank (index $s = S - 1$) or with the last label (index $s = S - 2$), the probability of observing $\mathbf{l}$ conditionally on $\mathbf{x}$ becomes:

$$ \mathbb{P}(\mathbf{l}|\mathbf{x}) = \alpha(T-1, S-1) + \alpha(T-1, S-2). \tag{3.32} $$

As $\mathbb{P}(\mathbf{l}|\mathbf{x})$ depends on the posterior gram $\mathcal{P}$, which is learned, this quantity is thus differentiable with respect to the model's learnable weights $\Theta$ and back-propagation algorithms can be used for training (GRAVES et al., 2006) and minimizing the loss $\mathcal{L}_{CTC}(\Theta)$ for, notably, ASR purposes.
<figure><img src="image_29.png" /><figcaption>Figure 3.5: Alignment retrieval from the best decoded extension via the exploitation of transitions between two symbols or between symbol and blank.</figcaption></figure>

### 3.5.2 CTC-based decoding module

The above-mentioned matrix $\alpha$ is fundamental to compute the CTC loss and allow model training. Naturally, in this objective, $\alpha$ produces a marginalization over all possible alignment paths – i.e., all extensions $l \in \mathcal{B}^{-1}(\mathbf{y})$ associated with the labelling $\mathbf{y}$ –, which is embodied by the sum operation of Eq. (3.30).

However, by considering the maximum operation instead of a summation, hence maximizing the probability $P(\mathbf{l}|\mathbf{x})$, one can come back to a forced alignment scenario close to VITERBI's decoding. The initialization from Eq. (3.31) is unchanged, but the recursion rules become:

$$ \forall t \in \{1, \dots, T-1\}, \forall s \in \{2, \dots, S-1\} \\ \alpha[t, s] = P[t, \tilde{y}_s] \times \begin{cases} \max \left\{ \begin{array}{l} \alpha[t-1, s] \\ \alpha[t-1, s-1] \end{array} \right\} & \text{if } \tilde{y}_s \in \{\varepsilon, \tilde{y}_{s-2}\} \\ \max \left\{ \begin{array}{l} \alpha[t-1, s] \\ \alpha[t-1, s-1] \\ \alpha[t-1, s-2] \end{array} \right\} & \text{otherwise.} \end{cases} \tag{3.33} $$

This corresponds to the forward pass. Then, through the backward pass, one can retrieve the optimal extension $\mathbf{l}^*$, expressed as

$$ \mathbf{l}^* = \underset{\mathbf{l} \in \mathcal{B}^{-1}(\mathbf{y})}{\operatorname{argmax}} P(\mathbf{l}|\mathbf{x}) \tag{3.34} $$

by keeping trace of successive transitions retained during the forward pass.

Finally, the alignment path $\pi^*$ between $\mathbf{x}$ and $\mathbf{y}$ is obtained by removing the blanks from $\mathbf{l}^*$ according to the visualizations in Figure 3.5, which gives insights on how start and end times for each symbol from $\mathbf{y}$ (i.e., without blanks) are retrieved. This defines the CTC variant of the forced alignment decoding module $\mathcal{D}_\pi$ that will be used in this work.
### Section summary – CTC-based neural alignment

Connectionist Temporal Classification (CTC) is a great option for end-to-end, soft-labelled neural voice alignment between an audio $\mathbf{x} \in \mathbb{R}^{T \times F}$ and a sequence $\mathbf{y} \in \mathcal{A}^*$ defined via an alphabet $\mathcal{A}$ of $L$ symbols. The first step is a CTC-based acoustic modeling with a neural network $\mathcal{M}_{\Theta}$, which generates a posteriorgram $\mathcal{P} = \mathcal{M}_{\Theta}(\mathbf{x}) \in [0, 1]^{T \times (L+1)}$ representing per-frame emission probabilities over the alphabet $\mathcal{A}$ and a blank label $\varepsilon$. This extra token, $\varepsilon$, is responsible for the flexibility of a CTC approach, as the model can choose not to specify any symbol from $\mathcal{A}$ but must eventually recognize the target sequence $\mathbf{y}$ from $\mathcal{P}$. The second step is the decoding module $\mathcal{D}_{\pi}$ exploiting the temporal information in $\mathcal{P}$ towards final alignment retrieval. This module is implemented as a variant of the classical VITERBI's algorithm in a CTC context, i.e., accounting for the additional transitions and subtleties induced by the introduction of the blank $\varepsilon$.

## 3.6 Scientific background for voice alignment in a nutshell

### Chapter summary – Scientific background for voice alignment

In this chapter, I introduced the definitions and mathematical tools necessary to tackle the voice alignment problem between an audio $\mathbf{x} \in \mathbb{R}^{T \times F}$ and a sequence $\mathbf{y} \in \mathcal{A}^*$ whose symbols belong to a finite alphabet $\mathcal{A}$. I showed that any synchronization system is based on an acoustic model $\mathcal{M}$ and a decoding module $\mathcal{D}_{\pi}$. In this thesis, exploiting recent advances in deep learning for voice processing and their robustness through their ability to learn from data themselves, the acoustic models are implemented as Deep Neural Networks (DNN) parameterized by $\Theta$, i.e., $\mathcal{M} \equiv \mathcal{M}_{\Theta}$. I motivated the choice of a Connectionist Temporal Classification (CTC) framework for the acoustic model $\mathcal{M}_{\Theta}$ due to its great practical flexibility, enabling model training with only soft labelling and the design of end-to-end architectures. Following such an approach, the alignment between an audio $\mathbf{x}$ and a sequence $\mathbf{y} \in \mathcal{A}^*$ relies on (1) a posteriorgram $\mathcal{P} = \mathcal{M}_{\Theta}(\mathbf{x})$ representing per-frame probabilities over the alphabet $\mathcal{A}$ and the CTC non-informative blank label $\varepsilon$; and (2) a variant of the DTW algorithm, accounting precisely for the blank $\varepsilon$, which allows the forced alignment *per se*.
# Part II

Contributions: time-constrained neural voice alignment
Chapter 4

# ADAGIO: An acoustic model for temporal voice alignment

> "Scientific research involves going beyond the well-trodden and well-tested ideas and theories that form the core of scientific knowledge. During the *Time* scientists are working things out, some results will be right, and others will be wrong. Over *Time*, the right results will emerge."

> — Lisa RANDALL

This chapter aims to present **ADAGIO** — the acoustic model at the heart of this thesis work which serves as a base for the defended contributions. Beyond the acronym standing for **Automatic Deep Alignment** of vOIce, ADAGIO is an end-to-end model thought for voice-to-symbol alignment and developed within the CTC framework.

As that is the case for many advances in research, ADAGIO is based on several inspirations from the literature. Therefore, the *section 4.1* exposes the neural architectures used for the acoustic modeling of recent alignment systems mentioned in the previous chapter and retraces the early explorations that rose the fundamental requirements that an acoustic model should, in the current context, satisfy. In light of these considerations and thought process, the proper design and (fixed) parameters of ADAGIO are detailed in *section 4.2*. A summary of this chapter is given in *section 4.3*.

<figure><img src="image_30.png" /></figure>
<figure><img src="image_31.png" /><figcaption>Figure 4.1: The Wav-U-Net architecture for CTC-based voice alignment (STOLLER et al., 2019).</figcaption></figure>

# 4.1 Model history

Before introducing the ADAGIO system itself, this section elaborates the stages of its design, notably by tracing the inspirations from the models in the literature and the first experimental tests carried out. In this perspective, the neural baselines — as anchors and comparison points — are presented and discussed. The first developmental drafts, run in the context of phonetic alignment with clean signals, are specified. ADAGIO being born from their extension, and especially from the overcoming of their intrinsic limits, these are exposed.

## 4.1.1 Baselines

First and foremost, the existing systems for addressing voice alignment are presented as they constitute inspirations (clarified in the next section [4.1.2]) and baselines for this work.

### Wav-U-Net

STOLLER et al. (2019) implemented the acoustic model as a CTC-trained Wav-U-Net whose architecture is shown in Figure 4.1.
<figure><img src="image_32.png" /><figcaption>Figure 4.2: The recurrent-attention (ARNN) architecture for separation-based voice alignment (SCHULZE-FORSTER et al., 2021).</figcaption></figure>

It processes raw audio recordings, i.e., 1-D signals, through a series of 1D convolution and downsampling layers allowing to capture low-level and higher level features from the audio at multiple time resolution. To generate the posteriorgram, to be optimized with respect to the CTC criterion, upsampling and more 1D convolutions are used. The core of any U-Net architecture is to propagate a multi-level modeling via concatenations between the downsampling and upsampling blocks.

### Attentional Recurrent Neural Network (ARNN)

SCHULZE-FORSTER et al. (2021) implemented the acoustic model as an Attentional Recurrent Neural Network (ARNN) whose architecture is shown in Figure 4.2.

It processes *both* text and audio with recurrent layers and an attention mechanism. The model supervises the reconstruction of the (denoised/clean) voice inputs from the output of the attention layer. The alignment can be derived from the same attention mechanism, as it exploits *attention weights* measuring the “weighting” of each symbol in each audio frame, which is in line with a typical Dynamic Time Warping (DTW) accumulative score matrix.
<figure><img src="image_33.png" /><figcaption>Figure 4.3: The convolutional-recurrent (CRNN) architecture for CTC-based voice alignment (VAGLIO et al., 2020a).</figcaption></figure>

*Convolutional Recurrent Neural Network (CRNN)*

VAGLIO et al. (2020a) implemented the acoustic model as a plain Convolutional Recurrent Neural Network (CRNN) whose architecture is shown in Figure 4.3.

It processes log-mel-spectrograms first with two convolutional layers, extracting more abstract features from the audio representation, then with three recurrent layers (Bi-LSTM) with dropout in-between, and a final (time-distributed) dense layer which generates the desired posteriorgram. The training procedure is supervised by the CTC loss.
### 4.1.2 Early explorations and learnings

#### First attempt

The first proposal made in this thesis was highly inspired from the architecture of SCHULZE-FORSTER et al. (2021) where the separation loss was replaced by the CTC training criterion. Hence, the defined acoustic model was exploiting both non-aligned transcripts **y** and audios **x** as inputs of the acoustic model and was benefiting from an attention mechanism to generate relevant posteriorgram $P$. This model was well suited for phonetic alignment, i.e., alignment of an audio recording with its associated sequence of pronounced phonemes. In the case of clean speech and solo singing voice, the feasibility to develop an end-to-end CTC-based voice aligner was confirmed, even with high temporal precision (as required by the phonetic granularity). This architecture was the starting point of the publication (TEYTAUT and ROEBEL, 2021).

#### Second attempt

However, there was room for improvements:

* On one hand, it was desired to get free from the symbols **y** as second input of the model for two reasons:

- First, because the attention mechanism involves a softmax that generates, for each of the *T* frames, a probability distribution over the *M* symbols, i.e., $\alpha \in [0, 1]^{T \times M}$ in Figure 4.2. During training, only short audio-text pairs are fed to the network due to memory constraints — hence, the attention weights $\alpha$ are specialized on small sequences. But in concrete inference usage, e.g., aligning a song with its lyrics, the softmax operation would be over a much greater number *M* of symbols than during the training phase, resulting in blurred and unexploitable attention weights $\alpha$. A solution would be to pre-segment the whole recording into shorter audio-symbol pairs, inducing circular dependency, which is neither straightforward nor convenient. This approach was, therefore, not suitable for a real world inference scenario.

- Second, because having the symbols as inputs is fundamentally preventing the model from being used for transcription. Although it is not the purpose of this work, predicting and evaluating transcriptions from audio remains an interesting option.

* On the other hand, Recurrent Neural Networks (RNN) are computationally intensive and may be tricky to train. As an associated trend in the ASR literature, it was aimed to replace such layers. The increasingly popular Transformer architecture (VASWANI et al., 2017), and its application to speech recognition with CTC (MIAO et al., 2020), highlighted that similar performances could be obtained without recurrent layers (and less parameters) thanks to simpler dense layers coupled with self-attention mechanisms.
Following these observations, the Bi-LSTM were replaced by Conv 1D layers, which capture a wider temporal context than pure dense layers (COLLOBERT et al., 2016). The attention mechanism became a multi-head self-attention (VASWANI et al., 2017) and, as now independent from the target sequence, the overall model could be used for aligning whole recordings – on condition of applying it successively on small audio portions and concatenate the outputs. This second design was the starting point of the publication (TEYTAUT et al., 2022).

### Limitations

This second approach, though, also remained limited and needed to be improved. Although applicable on entire audio recordings, the attention mechanism was still imposing a segmentation stage (of the audio only). For convenience, the acoustic model should be usable in one-shot on voice signals. Second, and more importantly, this model was small and focused on the phonetic alignment tasks with only short and clean speech and singing voice databases – without noises or background music –, which were, once again, not representative of real world inference scenario.

### 4.1.3 Acoustic model requirements

Given these explorations, insights were gained on the several requirement that any robust and practical acoustic model should satisfy. There are summarized in this section.

At training time,

- PARSIMONY is desirable in terms of number of parameters for the network and data neediness for the training procedure *per se*;

- NON-RECURRENT architectures are preferred to Recurrent Neural Networks (RNN) and their variants (and even more so for CTC-based alignment, as detailed later).

At inference time,

- AUDIO-ONLY models, in terms of inputs, are straightforward to apply and therefore recommended;

- POLYPHONIC usage is essential as the less pre-processing of the audio, the better. The system must be robust to musical accompaniment and usable on real recordings directly, without, e.g., relying on a source separator to isolate vocals.

None of the existing literature proposals satisfy all of these criteria. Indeed, the data-intensive nature of the Wav-U-Net has required its training on a private dataset of more than 40k songs, much more than any publicly available dataset, and its large architecture that cannot fit on limited hardware infrastructure. The ARNN and CRNN both exploit recurrent layers.
<table><caption>Table 4.1: Advisable criteria for a robust voice-to-symbol alignment system.</caption><thead><tr><th></th><th></th><th>Wav-U-Net</th><th>CRNN</th><th>ARNN</th><th>ADAGIO</th></tr></thead><tbody><tr><td rowspan="2">Training</td><td>PARSIMONY</td><td></td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>NON-RECURRENT</td><td>✓</td><td></td><td></td><td>✓</td></tr><tr><td rowspan="2">Inference</td><td>AUDIO-ONLY</td><td>✓</td><td>✓</td><td></td><td>✓</td></tr><tr><td>POLYPHONIC</td><td>✓</td><td></td><td>✓</td><td>✓</td></tr></tbody></table>

The CRNN uses Spleeter (HENNEQUIN et al., 2020) as a pre-processing step to extract vocals for both training and inference, while the ARNN needs the target sequence as second input.

The Table 4.1 summarizes these observations. The core proposal of this research is to develop a new system in line with all of the identified criteria – ADAGIO.

### Section summary – Model history

The early explorations of the current research, which has been based on and inspired from the existing models of the literature – Wav-U-Net (STOLLER et al., 2019), CRNN (VAGLIO et al., 2020a) and ARNN (SCHULZE-FORSTER et al., 2021) –, have allowed to identify criteria that any robust voice-to-symbol alignment system should satisfy but were not systemically fulfilled. These are: parsimony both in terms of data neediness and architecture size, absence of recurrent layers, audio-based inference only and robustness to noise/music accompaniment for real world application on, e.g., polyphonic music. This thesis precisely aims to develop a system in line with these objectives – ADAGIO.

## 4.2 ADAGIO: Automatic Deep Alignment of vOIce

This section is dedicated to the presentation of ADAGIO, the acoustic model at the center of this research. Based on convolutional architecture and the Connectionist Temporal Classification (CTC) framework, ADAGIO allows generating posteriorgrams characterizing the temporal evolution of the symbolic information contained in voice signals. The core architecture of the acoustic model is depicted in Figure 4.4.

### Data pre-processing

The network takes as inputs normalized log-mel-spectrograms $x \in [0, 1]^{T \times F \times 1}$ that are derived from the audio. For numerical stability and uniformity over the whole dataset, the Time-Frequency Representations (TFR) are scaled between 0 and 1 based on a 80dB threshold, as commonly done in the community – e.g., this is the default audio reading setting of the *librosa* audio library for Python (MCFEE et al., 2015).
<figure><img src="image_34.png" /><figcaption>Figure 4.4: ADAGIO – Neural architecture.</figcaption></figure>

The ground-truth transcripts **y** serve for supervision.

During the on-the-fly creation of the data batches, the TFR and symbolic sequences are padded if necessary so that all audios (resp. sequences) have the same number of frames *T* (resp. number of symbols *M*), i.e., the greatest one. Zero-padding is used on raw audios and a dedicated padding token is used for the labels.

## Model design

ADAGIO's neural layers are organized into two successive subnets:

- an *Audio Modeling Net*, in charge of capturing deep learned features from the audio;

- and a *Posteriorgram Net*, which simply generates the posteriorgram from the learned feature map.
<table><caption>Table 4.2: ADAGIO – summary of neural layers and input-output shapes.</caption><thead><tr><th>Net</th><th>Stage</th><th>Layer</th><th>Input size</th><th>Output size</th></tr></thead><tbody><tr><td rowspan="8">Audio Modeling</td><td rowspan="2">n = 1</td><td>Conv. Block 1</td><td>T × F × 1</td><td>T × F × C</td></tr><tr><td>Conv. Block 2</td><td>T × F × C</td><td>T × F/2 × C</td></tr><tr><td rowspan="2">n > 1</td><td>Conv. Block 1</td><td>T × (F/2<sup>n-1</sup>) × (2<sup>n-2</sup> · C)</td><td>T × (F/2<sup>n-1</sup>) × (2<sup>n-1</sup> · C)</td></tr><tr><td>Conv. Block 2</td><td>T × (F/2<sup>n-1</sup>) × (2<sup>n-1</sup> · C)</td><td>T × (F/2<sup>n</sup>) × (2<sup>n-1</sup> · C)</td></tr><tr><td>⋮</td><td>⋮</td><td></td><td></td></tr><tr><td rowspan="2">n = 8</td><td>Conv. Block 1</td><td>T × 1 × (2<sup>n-2</sup> · C)</td><td>T × 1 × E</td></tr><tr><td>Conv. Block 2</td><td>T × 1 × E</td><td>T × 1 × E</td></tr><tr><td rowspan="4">Posteriorgram</td><td rowspan="4"></td><td>Batch Norm</td><td>T × 1 × E</td><td>T × 1 × E</td></tr><tr><td>Conv 2D</td><td>T × 1 × E</td><td>T × 1 × (L + 1)</td></tr><tr><td>Batch Norm</td><td>T × 1 × (L + 1)</td><td>T × 1 × (L + 1)</td></tr><tr><td>Softmax</td><td>T × 1 × (L + 1)</td><td>T × 1 × (L + 1)</td></tr></tbody></table>

The *Audio Modeling Net* is a fully convolutional network composed of 8 successive stages of 2 convolution blocks. A convolutional block is constructed with the following layers: batch normalization, 2D-convolution, batch normalization, ReLU activation, and dropout. No time pooling is performed – the number of frames *T* is thus completely determined by the signal processing setup. At each stage, the first block uses a 1 × 1 stride but the second one uses a 1 × 2 stride for the 2D convolution, hence halving the number of Mel frequency bins, and the number of convolutional filters (common to both blocks) is increased by two times. By changing the number of frequency bins and channels every other layer, more convolutional blocks can be integrated, increasing the modeling power and the *receptive field*.

The *Posteriorgram Net* is composed of a final succession of batch normalization, a last 2D convolution, batch normalization and a softmax activation to obtain the per-frame probability distributions over the labels. The number of channels of the last convolution is the desirable dimension for the posteriorgram, i.e., the size *L* of the symbol alphabet *A* plus the blank token. Predictions can be derived from the posteriorgram $P \in [0, 1]^{T \times (L+1)}$.

For each layer, the shapes of the input-output tensors are shown in Table 4.2 and the network hyperparameters are specified in Table 4.3. Note that, with this setup, the global receptive field of the 16 convolutional layers is 1024ms. Therefore, the model can be trained on audio excerpts of several seconds (typically 10–20s), but can be used in inference on audio with much longer durations thanks to its fully convolutional architecture.
<table><caption>Table 4.3: ADAGIO – hyperparameter setup.</caption><thead><tr><th>Parameter</th><th>Notation</th><th>Value</th><th>Unit</th><th>Remark</th></tr></thead><tbody><tr><td colspan="5" style="text-align: center; font-style: italic;">Signal processing</td></tr><tr><td>Sampling rate</td><td>F<sub>S</sub></td><td>16000</td><td>Hz</td><td rowspan="2">HANN window – see Eq. (2.10)</td></tr><tr><td>Window nature</td><td>w[n]</td><td>1024</td><td>samples</td></tr><tr><td>Window length</td><td></td><td>1024</td><td></td><td></td></tr><tr><td>FFT size</td><td></td><td>512</td><td>samples</td><td rowspan="3">= 1000 × H/F<sub>S</sub><br>Audio normalization threshold</td></tr><tr><td>Hop size</td><td>H</td><td>32</td><td>ms</td></tr><tr><td>Frame duration</td><td>δt</td><td>80</td><td>dB</td></tr><tr><td>Top dB value</td><td></td><td>128</td><td></td><td></td></tr><tr><td colspan="5" style="text-align: center; font-style: italic;">Neural architecture</td></tr><tr><td>Filters (init.)</td><td>C</td><td>16</td><td></td><td rowspan="4">Number of stages of the Audio Modeling Net<br>= 2 × D × (K − 1) × δt</td></tr><tr><td>Filters (max.)</td><td>E</td><td>512</td><td></td></tr><tr><td>Kernel size</td><td>K</td><td>3 × 3</td><td></td></tr><tr><td>Deep depth</td><td>D</td><td>8</td><td>ms</td></tr><tr><td>Receptive field</td><td></td><td>1024</td><td></td><td></td></tr></tbody></table>

<table><thead><tr><th colspan="5" style="text-align: center; font-weight: bold;">Section summary – ADAGIO: Automatic Deep Alignment of vOIce</th></tr></thead><tbody><tr><td colspan="5">ADAGIO is an end-to-end, fully convolutional network which processes normalized audio log-mel-spectrograms x ∈ [0, 1]<sup>T×F×1</sup>. Its neural architecture consists of 16 convolutional blocks proceeding to an audio modeling with a receptive field around 1s, and a final convolutional block generating the posteriorgram P ∈ [0, 1]<sup>T×(L+1)</sup> over the L labels and the CTC blank ε.</td></tr></tbody></table>

## 4.3 Acoustic model for temporal voice alignment in a nutshell

<table><thead><tr><th colspan="2" style="text-align: center; font-weight: bold;">Chapter summary – An acoustic model for temporal voice alignment</th></tr></thead><tbody><tr><td colspan="2">In this chapter, I presented ADAGIO, the acoustic model at the core of my thesis, which is dedicated to the Automatic Deep Alignment of vOIce. Retracing the early explorations of my research, I shed light on relevance and convenience criteria that a robust voice aligner should satisfy. For the sake of flexibility, and according to these criteria, ADAGIO has been implemented as an end-to-end, fully convolutional network trainable with a reasonable (i.e., publicly available) amount of data and directly applicable on voice signals in the presence of music accompaniment.</td></tr></tbody></table>
Chapter 5

Temporal constraints for alignment
enhancement

"One must work with *Time* and not against it."

— Ursula KROEBER LE GUIN

It has been seen that a temporal voice alignment system depends on an acoustic model and a decoding module. The previous chapter, exploiting the technical tools of deep learning, was dedicated to the introduction of ADAGIO – a convolutional neural network predicting time-symbol posteriorgrams from audio that can be exploited to force-align a target sequence. The quality of the temporal information contained in the posteriorgram is thus essential and crucial for the relevance of the estimated alignments. The purpose of this chapter precisely is to reinforce the temporality of the neural predictions with ADAGIO via temporal constraints.

The *section* [5.1] first exposes why ADAGIO, due to the Connectionist Temporal Classification (CTC) formalism itself, carries some *limitations for alignment*, which motivates a search for a better robustness. The fundamental *section* [5.2] introduces proposals to this aim. Concretely, temporal (1) spectral reconstruction, (2) structure propagation and (3) guided monotony are thought as *additive temporal constraints*. These ideas are practically implemented as loss functions to be minimized on the vocals in addition to CTC, so that a *multi-objective training* phase is specified in *section* [5.3]. In the end, this new *time-constrained* version of ADAGIO is fully summarized in *section* [5.4] prior to a chapter recap in *section* [5.5].

<figure><img src="image_35.png" /></figure>
<figure><img src="image_36.png" /><figcaption>Figure 5.1: The CTC alignment limitation: Two posteriorgrams both relevant for transcription of the text “ab” but leading to fundamentally different alignments.</figcaption></figure>

## 5.1 The need for additional constraints

Previous Chapter 4 introduced ADAGIO as an acoustic model built upon the Connectionist Temporal Classification (CTC) algorithm. In section 3.5, the formalism of CTC was thoroughly presented. Its founding principle is an one-to-many prediction framework based on the existence of a blank label $ε$ that allows, by its non-informative nature, to decode several acceptable sequences from a given input (GRAVES et al., 2006; HANNUN, 2017). This is achieved by inserting blanks $ε$ between the symbols of the target sequence $y$ — creating the so-called extensions 1 of $y$. By marginalizing over all possible extensions 1 from $y$ and interleaved blanks, a CTC-trained neural network eventually estimates a conditional probability of $y$ given $x$ — i.e., a posteriorgram $\mathcal{P}$ that can be further used to force-align a sequence of symbols.

However, this approach also means that CTC cannot guarantee accurate temporal alignments between the input and output sequences, by the very nature of this one-to-many mapping it exploits. The blank symbols can occur at any time step in the output sequence, which makes it difficult to know precisely when each audio frame is aligned with each symbol.

The Figure 5.1 illustrates this on a toy example where two different CTC-trained models produce different yet relevant posteriorgrams as both of them (1) perfectly *recognize* the target text “ab”; but (2) lead to drastically different temporal alignments. Such time shifts were often empirically observed during this thesis and were naturally dependent on the neural architecture at play (e.g., recurrences or convolutional receptive field). This can be problematic for some applications, such as automatic captioning of videos, where the timing of the output text is crucial and, in a wider scope, for temporal audio and voice alignments in general.

This is the reason why speech specialists started introducing attention mechanisms as Eq. (3.14) in joint CTC-attention models. Attention allows handling relevant dependencies between audio and symbols over time, which can benefit downstream tasks as speech recognition (KIM et al., 2017; PARK et al., 2022; WATANABE et al., 2017). Extending such approaches for designing voice synchronizers precisely resembles SCHULZE-FORSTER et al. (2021)'s proposal, at the cost of (1) increasing computation complexity; and (2) being dependent on the target sequence for the acoustic modeling.