Some approaches have tried regularization on the CTC loss to better understand the role and utmost importance of its blank label (BLUCHE et al., 2015), prevent peaky probability distribution (LIU et al., 2018), or improve its scalability with Cross-Entropy (CE) through sampling (VARIANI et al., 2018). But, all in all, and to the best of the author's knowledge, no existing work has been dedicated to explicitly address and ensure the emergence of temporal alignment directly in CTC probabilities.

This thesis has focused on this point since its very beginning, alongside the search of the neural architecture for ADAGIO, as seen in the two publications (TEYTAUT and ROEBEL, 2021; TEYTAUT et al., 2022) on which this chapter is based on. The final proposals inheriting from these explorations are detailed in the next sections to temporally constraint CTC posteriors.

## Section summary – The need for additional constraints

Connectionist Temporal Classification (CTC) has launched a new trend in the voice alignment literature with the great benefit of not requiring aligned data for training deep acoustic models. Yet, alignment remains intrinsically difficult to couple with CTC. Indeed, by nature, CTC measures a transcription cost and can therefore be minimized without guaranteeing precise alignment properties. There thus exists a need to define additional constraints to reinforce the temporality in the CTC predictions.

## 5.2 Temporal constraints for reinforcing alignment

This section details the temporal constraints proposed to enhance the emergence of voice alignment properties in CTC posteriors, which has been one of the key contributions of this thesis work — in addition to the basic neural architecture of ADAGIO, which serves as a starting point hereinafter.

The expression “temporal constraints” convey two essential pieces of information. First, the term “*contraints*” is used to mean that the ramifications added to the initial network take the form of additional *cost functions* that will also be minimized during neural training alongside the CTC. Second, the term “*temporal*” clearly states that these introduced losses are concerned with the temporal axis of involved tensors, in order to precisely bring in temporal knowledge during the optimization phase.

In this context, three angles to define new temporal training objective have been identified. The first one is to ensure an accurate temporal reconstruction of the spectral content. The second goal is to preserve the structural organization from the audio recordings. The third angle aims to guarantee the occurrence of a precise monotonicity between the predictions of the acoustic model and the symbolic sequence to be decoded through a classical accumulative score. This section explores each of these research paths. All details about the related network structures can be found in Table 5.1 at the end of this chapter.
<figure><img src="image_37.png" /><figcaption>Figure 5.2: Illustration of the proposed spectral reconstruction content.</figcaption></figure>

5.2.1 Spectral reconstruction

The first approach proposed to help the model predicting the symbols at their accurate and relevant position is a supplementary constraint that consists in reconstructing the audio spectral information from the CTC posteriogram, which can be seen as a compressed representation of the audio. The reconstruction is performed by a small neural network — the *Reconstruction Net*. It is also of convolutional nature — this way, the spectral reconstruction at a given frame *t* depends only on a small temporal context around *t* in the posteriogram. The inputs to the *Reconstruction Net* are the probabilities of all symbols without the blank as it is precisely desired to limit the importance of the blank in the overall prediction as it is mostly responsible for the CTC intrinsic alignment limitation (see section 5.1). During training, the supervision is done with a L1 distance loss between the estimated and real spectral content. The Figure 5.2 illustrates this additional reconstruction constraint in the case of voice-to-text alignment.

Preliminary research

In the first study (TEYTAUT and ROEBEL, 2021), the reconstruction of the entire input
spectrogram x ∈ [0, 1]^{T×F} was supervised. In the second study (TEYTAUT et al., 2022), it
was stated that such a systematic global reconstruction might not be pertinent. For instance,
the propagation of the F0 and harmonics is irrelevant for the alignment of phonemes — or any
textual modality of communication more generally — as phonemes are associated with formants.
Thus, the network would have to dedicate some of its modeling capacity to produce an F0 estimation that would not be pertinent for the alignment task. In this second study, the spectral reconstruction was supervised on the Mel-Frequency Cepstral Coefficients (MFCCs). In this manuscript, it is proposed to directly rely on the spectral envelope that fully characterizes formants and is a F0-free Time-Frequency Representations (TFR) — see section 2.2.2.

## Vocals separation

However, considering the spectral envelope of the entire audio mixture is no interest as the envelope associated with musical accompaniment conveys no information on the phonetic content. Therefore, the spectral reconstruction must only be based on the *vocals* contained in the audio recordings. This implies the usage of a source separation algorithm, a common need in the voice community (VINCENT et al., 2018), and here specifically a singing voice extractor (JANSSON et al., 2017).

A re-implementation of CHOI et al. (2019)'s neural voice separator was at disposal upon training with data augmentation (COHEN-HADRIA et al., 2019; LANCASTER and SOUVIRAÀ-LABASTIE, 2020). It achieves high quality extraction of the voice and these estimated vocals were judged clean enough to be reliable for constraining the network. More information can be found on a dedicated section 7.3.2 where the voice separation algorithm is detailed, evaluated (VINCENT et al., 2006) and integrated in a musicological pipeline.

Note that the isolated vocals solely serve during the training phase. They are *not* required for inference: only the audio mixture shall be fed to the network.

## Final proposal

Given an estimation $\hat{e} \in [0, 1]^{T \times F}$ of the “true” spectral envelope $e \in [0, 1]^{T \times F}$ extracted from solo vocals upon voice separation, the spectral reconstruction loss $\mathcal{L}_{\text{REC}}(\Theta)$ is defined as the following L1 distance:

$$ \mathcal{L}_{\text{REC}}(\Theta) = \|\hat{e} - e\|_1 = \sum_{t=0}^{T-1} \sum_{f=0}^{F-1} |\hat{e}[t, f] - e[t, f]|. \qquad (5.1) $$

Figure 5.3 shows the ramification of ADAGIO associated with this spectral reconstruction constraint.

It is worth mentioning that this proposal remains valid for the case of *note* alignment although one should precisely consider spectral *excitation* — i.e., Fundamental Frequency (F0) and harmonics or everything but the spectral envelope. Remembering the source-filter modeling in Eq. (2.15) and Figure 2.7, the information of the varying pitches will be well represented in the spectral excitation, which is what should be supervised in that context.
<figure><img src="image_38.png" /><figcaption>Figure 5.3: Spectral reconstruction constraint (here, for voice-to-text alignment).</figcaption></figure>

### 5.2.2 Temporal structure propagation

The second approach proposes to study the *temporal structure* with the claim that shared similarity patterns, informing on the local temporal structure, are to be found in the original voice signals and in the alignment predictions. To do this, Self-Distance Matrix (SDM) are computed on the input audio — as features representing the structural content — and are to be recovered from the CTC posteriors. This is performed by a small neural network — the *Structural Net*. It is also of convolutional in nature — this way, the estimated structure at a given frame *t* depends only on a small temporal context around *t* in the posteriors. The inputs to the *Structural Net* are again the probabilities of all symbols *without* the blank. During training, the supervision is done with a L1 distance loss between the estimated and real SDM. The Figure 5.4 illustrates this additional reconstruction constraint in the case of voice-to-text alignment.

#### *Self-Distance Matrix (SDM)*

Self-Distance Matrix (SDM) are well-known representations for capturing the structure of pieces of music (COHEN-HADRIA and PEETERS, 2017; FELL et al., 2022).
<figure><img src="image_39.png" /><figcaption>Figure 5.4: Illustration of the proposed temporal structure propagation.</figcaption></figure>

The SDM $S \in [0, 1]^{T \times T}$ of the log-mel-spectrogram $\mathbf{x} \in [0, 1]^{T \times F}$ measures a proximity between every spectral frame. The value $S[t, t'] \in [0, 1]$ is the dissimilarity between the frame $t$ and $t'$. The so-called dissimilarity implies the choice of a distance, whose nature depends on the studied context, calculated between the frame contents.

The lack of contrast in SDM is known for often limiting their discriminative power (PEETERS and ANGULO, 2022), which was commonly observed when relying on a cosine distance measure. For data whose value range is $[0,1]$, a L1 distance based on their L2 norms coupled with a simple normalization factor has empirically appeared as a sufficiently discriminative choice.

Therefore, in this work, the non-resemblance between frames is defined as a frequency-based scoring based on L2 norms, i.e.,

$$ \forall t, t' \quad S[t, t'] = \frac{1}{F_p} \sum_{f=0}^{F-1} |\mathbf{x}[t, f]^2 - \mathbf{x}[t', f]^2|. \qquad (5.2) $$

where $p = 1/\sqrt{2}$ has been fixed upon empirical observations.
For the same reasons as exposed before for spectral reconstruction, the structure of the musical background does not carry relevant information for voice alignment. Hence, the ground-truth structural information is derived from the spectral envelope (for text synchronization) or excitation (for notes synchronization) of the solo vocals estimated thanks to a voice separator.

Finally, as (1) one is looking for local patterns but not up to the frame level; and (2) all SSM are (T × T)-shaped hence memory demanding, a (4 × 4)-average pooling operation with stride (2 × 2) is used both to smooth local structural singularities and reduce memory storage.

## Comparisons with previous work

The structural loss was introduced in the publication (TEYTAUT et al., 2022), although Self-Similarity Matrices were considered in that paper (with no loss of purpose). They were computed using the cosine similarity on the complete log-mel-spectrogram, hence including both spectral envelope and source (F0 and harmonics), which was not an optimal strategy. The data already were clean solo speech and singing signals so the extension to real world recordings was limited.

Here, the reference structure is derived from the spectral envelope of the solo vocals estimated, once again, via a voice separator. This separation is only required for training and not inference. The other main difference between previous proposal and this manuscript regarding temporal structure is that the SDMs were computed directly on the posteriorgrams $\mathcal{P}$ in (TEYTAUT et al., 2022) while they are derived from an extra encoding from $\mathcal{P}$ here. This choice was motivated by realizing that the structural constraint alone was not helping the alignment procedure, and ad hoc analyses revealed that posteriorgrams could be trickily shaped to have correct structures, yet without predicting the full duration of each label. In order to prevent this negative side effect from happening, while continuing to propagate the temporal structure information, it is here proposed to generate a new feature map from $\mathcal{P}$ and compute the SDM of this one.

## Final proposal

Given an estimation $\hat{\mathcal{S}} \in [0, 1]^{\frac{T}{2} \times \frac{T}{2}}$ of the temporal structure $\mathcal{S} \in [0, 1]^{\frac{T}{2} \times \frac{T}{2}}$ extracted from solo vocals envelope upon voice separation and after the strided average pooling (hence T/2 frames), the structural propagation loss $\mathcal{L}_{STR}(\Theta)$ is defined as the following L1 distance:

$$ \mathcal{L}_{STR}(\Theta) = \left\| \hat{\mathcal{S}} - \mathcal{S} \right\|_{1} = \sum_{t=0}^{\frac{T}{2}-1} \sum_{t'=0}^{\frac{T}{2}-1} \left| \hat{\mathcal{S}}[t, t'] - \mathcal{S}[t, t'] \right| \quad (5.3) $$

The Figure 5.5 shows the ramification of ADAGIO associated with this structural propagation constraint.
<figure><img src="image_40.png" /><figcaption>Figure 5.5: Structural propagation constraint (here, for voice-to-text alignment).</figcaption></figure>

### 5.2.3 Guided audio-symbol monotony

The third and final approach proposes to directly measure and quantify a sequence-to-sequence accumulative score, typical derived from the Dynamic Time Warping (DTW) algorithm. Speech and singing signals and symbolic transcripts are ordered sequences. Remembering the very definition of alignment between general sequences (see section 2.3.1), synchronizing audio with such symbols implies uncovering a path with *monotonic* properties and relevant transitions between the symbols. This is performed by a small external module — the *Monotonic Net*. Although it is not a neural network *per se* (no extra parameters), it does define a new gradient to be propagated through the posteriors. The inputs to the *Monotonic Net* are the CTC symbol probabilities (without blank) and the target sequence. The symbolic sequence is thus used during training to supervise CTC and this new constraint but it is still not needed at inference time. A DTW-alike score is obtained and becomes the loss aimed at being minimized.
<figure><img src="image_41.png" /><figcaption>Figure 5.6: Illustration of the proposed time-symbol guided monotony.</figcaption></figure>

The Figure 5.6 illustrates this additional audio-symbol monotonic constraint in the case of voice-to-text alignment.

## Audio-symbol likelihood

The starting point of this additional constraint is the definition of a proximity measure between each of the *T* frames and each of the *M* symbols. Such a matrix is easy to obtain from the posteriorgram without blank $P \in [0, 1]^{T \times L}$ and the one-hot encoded symbols $y \in \{0, 1\}^{M \times L}$. From these two tensors, one can compute the audio-symbol log-likelihood $D \in [0, 1]^{T \times M}$ by taking the opposite of the logarithm of the direct matrix multiplication between $P$ and $y$. The probabilities are set to a minimum threshold of $1e^{-9}$ so that these log-likelihoods have an upper bound of 9Np (Neper), which allows renormalization of the measures between 0 and 1. In short, with *T* the transpose operator,

$$D = -\log (P y^T). \qquad (5.4)$$

The existence of a monotonic alignment implies the emergence of a pseudo-diagonal in matrix $D$ (hence its denomination), which can actually be seen in Figure 5.6. This diagonal is supposedly associated with the best alignment path to uncover.
## Accumulative score matrix

The next step, highly similar to Dynamic Time Warping (DTW) from Eq. (3.17), consists in computing an accumulative score matrix with only two permitted transitions $G = \{(1, 1), (1, 0)\}$ – i.e., coming from the same label or the previous label at the previous frame. This accumulative score $S \in \mathbb{R}^{T \times M}$ uses $D$ as reference for distances. It follows:

$$ 
\begin{array}{l}
t = 0 \quad m = 0 \quad \mathbf{S}[t, 0] = \mathbf{D}[t, 0] \\
t = 0 \quad \forall m > 0 \quad \mathbf{S}[0, m] = \infty \\
\forall t > 0 \quad \forall m > 0 \quad \mathbf{S}[t, m] = \mathbf{D}[t, m] + \min \left\{ \begin{array}{l} \mathbf{S}[t-1, m] \\ \mathbf{S}[t-1, m-1] \end{array} \right\}.
\end{array} 
\quad (5.5) 
$$

Note that, as opposed to the DTW, which usually maximises a resemblance, the min operator is used here to define a training criterion to minimize.

## Guided audio-symbol monotony

Following Eq. (3.18), the final accumulative value is a direct measure of the (mis)alignment between the two sequences, i.e., between time and symbols, e.g., between voice signals and text or notes. This leads to introduce the monotonic, DTW-alike loss $\mathcal{L}_{\text{DTW}}(\Theta)$ as:

$$ \mathcal{L}_{\text{DTW}}(\Theta) = \mathbf{S}[T-1, M-1]. \quad (5.6) $$

By minimizing this score, the network is guided towards the best alignment path. This can happen only if CTC systematically highlights the full duration of each label and not only their onset. The Figure 5.7 shows the ramification of ADAGIO associated with this guided audio-symbol monotony constraint.

## Comparison with previous work

The idea to exploit the sequence $y$, which is available during training, to measure a time-label proximity based on the posteriorgram $P$ has been introduced in (TEYTAUT et al., 2022). The matrix $D$ was similarly defined and it was precisely desired to ensure the existence of a diagonal in it. For this, inspired from guided attention literature (TACHIBANA et al., 2018), a Gaussian-decreasing matrix was used to extract a fixed diagonal out of $D$ and derive a loss value from it. However, this strategy carried a prior that all labels should have similar duration, which was intrinsically not true. The continuation of this work was rather to couple $D$ with a better alignment quality metric – Dynamic Time Warping (DTW), hence the current proposal.
<figure><img src="image_42.png" /><figcaption>Figure 5.7: Guided monotony constraint (here, for voice-to-text alignment).</figcaption></figure>

## Section summary – Temporal constraints for reinforcing alignment

In order to reinforce both onset and duration detection of non-blank labels in CTC posteriors for a given CTC sequence, with the ultimate goal to guarantee the emergence of alignment properties directly in the probabilities, three temporal constraints were defined, namely:

* **Spectral reconstruction** aiming at reproducing the relevant spectral information (e.g., *envelope* for text or *excitation* for note alignment) from the label probabilities;

* **Temporal structure propagation** ensuring that the same structural information as the input spectral content can be temporally retrieved from the posteriors;

* **Guided audio-symbol monotony** measuring an accumulative (mis)alignment score derived from both the labels and the posteriors.

Vocals extraction is performed prior to the supervision of the two former cases as musical accompaniment is an inherent obstacle to learn the reconstruction and structure of *voice*.
## 5.3 Multi-objective training

The present research aims to couple various losses together — the CTC and three additional constraints — thus raising the question of how to combine them and to determine their respective trade-off, which is a concern for all multi-task learning problems (KIM et al., 2017; LIANG et al., 2021b).

Since all the losses are duration-dependent, it is proposed to ensure that they all scale similarly, linearly with the time length $T$, which is the dimension of interest for temporal audio alignment. Prior to the scaling process itself, an idea of the relative order of magnitude of the different losses is necessary. They are estimated from worst-case scenario studies, and allow, in the end, to build a global loss function to minimize during training.

### 5.3.1 Worst-case scenario studies

This section presents the estimation of cost functions from critical and theoretical scenarios, which tend to represent the early stages of training where the weights are random and the network has not yet had time to specialize. For each loss, its associated formula is remembered and further investigated. It is reminded that $x \in [0, 1]^{T \times F}$ denotes the input log-mel-spectrogram and $y \in \mathcal{A}^M$ or $\{0, 1\}^{M \times L}$ represents the target symbols as a linear sequence or one-hot encoding. The model's learnable parameters are denoted $\Theta$.

#### Connectionist Temporal Classification (CTC)

Retracing the definition of the Connectionist Temporal Classification (CTC) training criteria from Eq. (3.25), the CTC loss can be expressed as

$$ \mathcal{L}_{\text{CTC}}(\Theta) = -\log \left( \sum_{\ell \in \mathcal{B}^{-1}(\mathbf{y})} \prod_{t=0}^{T-1} \mathcal{P}[t, \ell] \right). \qquad (5.7) $$

The number of alignment paths considered in the sum of the CTC computation is the cardinal of $\mathcal{B}^{-1}(\mathbf{y})$, which contains all (labelling) extensions $l$ that reduce to $\mathbf{y}$ after application of the operator $\mathcal{B}$ merging repeated character and removing blanks. HANNUN (2017) and MAO (2019) revealed that there were $\binom{T+M}{T-M}$ possibilities. In the most uninformative way possible, the CTC posteriorgram would be uniform over the $L$ labels and blank. An estimate of the loss value then becomes:

$$ \mathcal{L}_{\text{CTC}}(\Theta) \sim -\log \left[ \binom{T+M}{T-M} \left( \frac{1}{L+1} \right)^T \right]. \qquad (5.8) $$
Going further, it can be assumed that there are much more time frames T than symbols to synchronize M, i.e., $T \gg M$, leading to:

$$ \mathcal{L}_{\text{CTC}}(\Theta) \sim \log(L+1)T. \qquad (5.9) $$

Note that the CTC algorithm necessarily needs that $T \ge M$. (One cannot map each of the M symbols to an audio frame if there are less than M frames.)

## Spectral reconstruction

As seen in Eq. (5.3), the spectral reconstruction directly compares an estimate of the spectral content, e.g., envelope $\hat{e} \in [0, 1]^{T \times F}$ to a reference one $e \in [0, 1]^{T \times F}$ by minimizing:

$$ \mathcal{L}_{\text{REC}}(\Theta) = \|\hat{e} - e\|_1 = \sum_{t=0}^{T-1} \sum_{f=0}^{F-1} |\hat{e}[t, f] - e[t, f]|. \qquad (5.10) $$

Given the value range and shapes associated with both tensors, the maximum difference one could theoretically measure is:

$$ \mathcal{L}_{\text{REC}}(\Theta) \sim \text{FT}. \qquad (5.11) $$

## Temporal structure propagation

Similarly, the temporal structure loss minimizes the difference between the input Self-Distance Matrix (SDM) $S \in [0, 1]^{\frac{T}{2} \times \frac{T}{2}}$ and an estimated one from the posteriorgram $\hat{S} \in [0, 1]^{\frac{T}{2} \times \frac{T}{2}}$. It comes:

$$ \mathcal{L}_{\text{STR}}(\Theta) = \|\hat{S} - S\|_1 = \sum_{t=0}^{\frac{T}{2}-1} \sum_{t'=0}^{\frac{T}{2}-1} |\hat{S}[t, t'] - S[t, t']|. \qquad (5.12) $$

Given the value range and sizes associated with both tensors, the maximum difference one could theoretically measure is:

$$ \mathcal{L}_{\text{STR}}(\Theta) \sim \frac{1}{4}T^2. \qquad (5.13) $$

## Guided audio-symbol monotony

Finally, the monotony constraint is based on a per-frame accumulative penalty score, see Eq. (5.5). It is maximized when CTC does not recognize any label correctly, and even less their full duration. In this case, the greatest distance possible (which is 1) is added at each of the T frames. In the end:

$$ \mathcal{L}_{\text{DTW}}(\Theta) \sim T. \qquad (5.14) $$
### 5.3.2 Scaling the losses

It has been shown that chosen objective criteria do not result in similar variations when the audio length, i.e., number of frames $T$, changes. This is a major issue since different elements in the training set will not end up inducing comparable updates for the gradients and weights themselves.

Consequently, from previous worst-case scenario studies, highlighting *estimates* of the loss value, it is proposed to scale each loss so that they all have identical, linear dependency on $T$, which is reasonable because time segments in sequences will always have the same impact independently of the phrase they are found in.

The scaled losses are defined as:

$$ 
\begin{align*}
\mathcal{L}_{\text{CTC}}^{n}(\Theta) &\leftarrow \frac{1}{\log(L+1)} \mathcal{L}_{\text{CTC}}(\Theta) \\
\mathcal{L}_{\text{REC}}^{n}(\Theta) &\leftarrow \frac{1}{F} \mathcal{L}_{\text{REC}}(\Theta) \\
\mathcal{L}_{\text{STR}}^{n}(\Theta) &\leftarrow \frac{4}{T} \mathcal{L}_{\text{STR}}(\Theta) \\
\mathcal{L}_{\text{DTW}}^{n}(\Theta) &\leftarrow \mathcal{L}_{\text{DTW}}(\Theta)
\end{align*}
\tag{5.15} $$

All in all, the global loss to be minimized during training is:

$$ \mathcal{L}(\Theta) = \mathcal{L}_{\text{CTC}}^{n}(\Theta) + \frac{1}{3} \sum_{i} s_{i} \mathcal{L}_{i}^{n}(\Theta) \tag{5.16} $$

with $i$ an index over the above-mentioned constraints and $s_i \in \{0, 1\}$ whether the constraint $i$ is used or not. The factor $\frac{1}{3}$ aims at preventing even all joint constraints from dominating the CTC, which remains the core objective as it does generate the posteriors. This results in 8 possible configurations to be evaluated, which is precisely the core of the next Part III – to evaluate and apply ADAGIO as an acoustic modeling.

## Section summary – Multi-objective training

The proposals to integrate more temporal information during the training of the acoustic model ADAGIO suffer from not having the same dependency on the number of audio frames, which is a problem for their direct combination. To cope with this, based on worst-case scenario studies, a scaling factor was estimated to normalize each of them, ensuring that the losses all have an identical, linear variability with respect to audio length.
## 5.4 Time-constrained acoustic modeling

All in all, a *time-constrained* version of ADAGIO has been proposed and implemented to reinforce alignment quality. The Table 5.1 depicts the succession of layers and tensor shapes for ADAGIO and these extensions. It notably sheds light on the number of convolutional filters used for spectral reconstruction and structure propagation. These new convolutions use a kernel size of 1 × 1. The other hyperparameters are the same as the ones shared in Table 4.3.

### Section summary – Time-constrained acoustic modeling

Summing up, three additional temporal constraints were proposed with the aim to reinforce ADAGIO by assuring the emergence of alignment properties directly in the posteriors. Each of these supplementary constraints is implemented in practice by adding a subnetwork that is connected to the output of ADAGIO, i.e., the CTC posteriors for *without* blank. The respective *Reconstruction Net*, *Structural Net* and *Monotony Net* are therefore integrated to complement the original neural architecture. This *time-constrained* extension of ADAGIO is fully summarized in Figure 5.8.

## 5.5 Temporal constraints for alignment enhancement in a nutshell

### Chapter summary – Temporal constraints for alignment enhancement

In this chapter, I presented a *time-constrained* extension of my acoustic model ADAGIO. I started by explaining why the Connectionist Temporal Classification (CTC) formalism, notably due to the blank label $\varepsilon$, cannot guarantee precise temporal alignment. I proposed to explicitly address this problem by introducing three additional temporal constraints aimed at forcing the posteriors to capture deep alignment properties by integrating time-dependent information. These contributions took the form of three auxiliary tasks of spectral reconstruction, temporal structure propagation, and guided time-sequence monotony. Each of them relied on a similar idea: to generate a new representation heavily dependent on coherently aligned emissions in the posteriors. They required completing the neural architecture and design a multi-objective training scenario by normalizing the diverse losses at play. The final upcoming chapters evaluate the relevance of these proposals and present their concrete applications to voice research.
<table><caption>Table 5.1: Time-constrained ADAGIO – summary of neural layers and input-output shapes.</caption><thead><tr><th>Net</th><th>Stage</th><th>Layer</th><th>Input size</th><th>Output size</th></tr></thead><tbody><tr><td rowspan="8">Audio Modeling</td><td rowspan="2">n = 1</td><td>Conv. Block 1</td><td>T × F × 1</td><td>T × F × C</td></tr><tr><td>Conv. Block 2</td><td>T × F × C</td><td>T × F/2 × C</td></tr><tr><td rowspan="2">n > 1</td><td>Conv. Block 1</td><td>T × (F/2<sup>n-1</sup>) × (2<sup>n-2</sup> · C)</td><td>T × (F/2<sup>n-1</sup>) × (2<sup>n-1</sup> · C)</td></tr><tr><td>Conv. Block 2</td><td>T × (F/2<sup>n-1</sup>) × (2<sup>n-1</sup> · C)</td><td>T × (F/2<sup>n</sup>) × (2<sup>n-1</sup> · C)</td></tr><tr><td>⋮</td><td>⋮</td><td></td><td></td></tr><tr><td rowspan="2">n = 8</td><td>Conv. Block 1</td><td>T × 1 × (2<sup>n-2</sup> · C)</td><td>T × 1 × E</td></tr><tr><td>Conv. Block 2</td><td>T × 1 × E</td><td>T × 1 × E</td></tr><tr><td rowspan="4">Posteriorgram</td><td rowspan="4"></td><td>Batch Norm</td><td>T × 1 × E</td><td>T × 1 × E</td></tr><tr><td>Conv 2D</td><td>T × 1 × E</td><td>T × 1 × (L + 1)</td></tr><tr><td>Batch Norm</td><td>T × 1 × (L + 1)</td><td>T × 1 × (L + 1)</td></tr><tr><td>Softmax</td><td>T × 1 × (L + 1)</td><td>T × 1 × (L + 1)</td></tr><tr><td rowspan="3">Reconstruction</td><td rowspan="3"></td><td>Conv. Block 1</td><td>T × 1 × L</td><td>T × 1 × E</td></tr><tr><td>Conv. Block 2</td><td>T × 1 × E</td><td>T × 1 × F</td></tr><tr><td>Tanh</td><td>T × 1 × F</td><td>T × 1 × F</td></tr><tr><td rowspan="3">Structural</td><td rowspan="3"></td><td>Conv. Block 1</td><td>T × 1 × L</td><td>T × 1 × E</td></tr><tr><td>Conv. Block 2</td><td>T × 1 × E</td><td>T × 1 × F</td></tr><tr><td>Tanh</td><td>T × 1 × F</td><td>T × 1 × F</td></tr><tr><td rowspan="4">Monotony</td><td rowspan="4"></td><td>MatMul</td><td>❶ T × 1 × L ≡ T × L</td><td>T × M</td></tr><tr><td>Log-likelihood</td><td>❷ M × L</td><td>T × M</td></tr><tr><td rowspan="2">DTW</td><td>T × M</td><td>T × M</td></tr><tr><td>T × M</td><td>T × M</td></tr></tbody></table>
<figure><img src="image_43.png" /><figcaption>Figure 5.8: ADAGIO enhanced with temporal constraints of spectral reconstruction, structure propagation and guided monotony.</figcaption></figure>
# Part III

## Outcomes: experiments & applications
Chapter 6

Evaluations of deep voice alignment

"The history is important because science is a discipline
deeply immersed in history. In other words, every **Time** you
perform an experiment in science or in medicine, what
you're actually doing is you're answering someone,
answering a question raised by someone in the past."

— Siddhartha MUKHERJEE

This chapter focuses on the evaluation of deep learning-based temporal voice alignment
systems, with a particular emphasis on the contributions introduced in this research. Through
a series of carefully designed experiments, the effectiveness and accuracy of these proposals are
quantified and provide valuable insights into their capabilities and limitations.

As developing and evaluating the models require data featuring speaking and singing voice,
the many voice corpora used in this manuscript are first presented in section 6.1. The core of
the evaluation procedure, including alignment retrieval by neural inference and decoding to the
classical quantitative metrics, is then detailed in section 6.2. Then, in section 6.3, an ablation
study is carried out to evaluate the impact of the proposed contributions. Finally, results from
various voice-to-symbols alignments scenarios are shared, compared to relevant baselines, and
commented in section 6.4. The main outcomes obtained in this chapter are summarized in
section 6.5.

<figure><img src="image_44.png" /></figure>
<table><caption>Table 6.1: Overview of all voice corpora used in experiments.</caption><thead><tr><th rowspan="3">Dataset</th><th rowspan="3">Language(s)</th><th rowspan="3">Duration [h]</th><th colspan="2">Voice</th><th colspan="6">Content</th></tr><tr><th rowspan="2">Speech</th><th rowspan="2">Singing</th><th colspan="2">Transcripts</th><th colspan="3">Alignments</th><th rowspan="2">Background music</th></tr><tr><th>Text</th><th>Phonemes</th><th>Words</th><th>Phones</th><th>Notes</th></tr></thead><tbody><tr><td>LibriSpeech</td><td>English</td><td>1,000</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>TIMIT</td><td>English</td><td>5</td><td>✓</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td></tr><tr><td>Philos 10</td><td>English</td><td>0.35</td><td>✓</td><td></td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td></tr><tr><td>Att-HACK DALI</td><td>French</td><td>30</td><td>✓</td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td></tr><tr><td>English</td><td>English</td><td>220.1</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Playlist50</td><td>English</td><td>2.6</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Hansen</td><td>English</td><td>0.6</td><td></td><td>✓</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td></tr><tr><td>Jamendo</td><td>English</td><td>1.2</td><td></td><td>✓</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td></tr><tr><td>Chanter RT</td><td>French</td><td>1.5</td><td></td><td>✓</td><td></td><td>✓</td><td>✓</td><td>✓</td><td></td><td>✓</td></tr></tbody></table>

# 6.1 Voice corpora

This section describes the various speech and singing corpora used for training and evaluating the voice aligners. The origin and role of each dataset in the undertaken studies are specified as well as the nature of their annotation (manual vs automatically generated). For the sake of conciseness, Table 6.1 summarizes their main characteristics.

## 6.1.1 Speech datasets

### LibriSpeech

LibriSpeech is a corpus featuring 1,000h of spoken voice from audiobook recordings sampled at 16kHz and read by various speakers. It has been prepared in the context of the LibriVox project and has manually been segmented into small audio excerpts and their associated transcripts (PANAYOTOV et al., 2015). Train, test and development sets for clean and more challenging speech are shared. LibriSpeech has been widely used in the speech recognition literature and, although no text alignment is provided, the audio-text pair allows CTC training.

### TIMIT

TIMIT (ZUE et al., 1990) is another extremely popular dataset in the speech recognition literature. TIMIT is a multi-speaker dataset of 5 hours of speech, organized in 6,300 sentences (4,000 for training, 1,300 for testing), with available word and phonetic transcripts. Although TIMIT is much smaller than LibriSpeech, respective word- and phone-level manual alignments to the audio are available, making it a relevant test set for both word and phonetic alignments.
*Philos 10 (The Problems of Philosophy, B. RUSSELL, Chapter 10)*

TIMIT and LibriSpeech feature naturally short or pre-segmented audio excerpts that are necessary to train models due to memory limitation on GPU. Yet, evaluations on these datasets may not be really relevant for real-world applications. Indeed, in practice, one rarely needs to align 5-10s of audio with its text, but rather longer context. This is the problem of (very) long audio alignment, which is known as a limitation for most alignment algorithms and that will be briefly mention in section [7.4]. There is thus a need for a long audio-text evaluation dataset.

Instead of artificially creating longer test audio-text pairs by concatenating existing small pieces, it was decided to manually annotate an entire chapter from a publicly available audiobook. Given the extremely time-consuming nature of manual annotations, the annotation is limited to word-level alignments. To report fair evaluations, neither the chosen audiobook nor its reader could be part of LibriSpeech. It has been found that “*The Problems of Philosophy*” by Bertrand RUSSELL (2001) fulfilled these criteria. The 10th chapter of this book, which contains exactly 100 sentences (i.e., 2672 words) and has an audio duration of 21min, has been chosen and collectively aligned. This dataset is referred to as *Philos 10* and is shared with the community<fnref n="1" />.

The annotation process, as detailed in the associated publication (DORAS et al., 2023), has been done as follows: once the Librivox audio preamble (title and copyright information) was discarded, ADAGIO was used to roughly synchronize the onset of each word with the audio. Given these alignments, temporal markers were extracted and manually corrected in some audio visualization software, e.g., Audacity² or Partiels³. A spectral display was used to better capture the phonetic evolution and mark the word onsets. Some inconsistencies between audio and text were heard, so the text was modified/corrected accordingly to match the audio perfectly.

*Att-HACK*

The Att-HACK dataset (LE MOINE and OBIN, 2020) complements the above-mentioned ones as it exposes (French) expressive speech, i.e., speech acted by professional actors that were asked to sound as natural as possible given some instructions.

Concretely, 20 native French speakers (9 men and 11 women) had to portray four social attitudes — friendly, seductive, dominant and distant — over 100 isolated sentences. To account for the individual variations when it comes to produce a vocal attitude, each utterance has been recorded multiple times (three to five versions), resulting in a total of 30h of expressive speech. All the audio excerpts are provided with their orthographic text transcription.

<footnote n="1">¹https://ircam-anasynth.github.io/papers/2023/a-linear-memory-ctc-based-algorithm-for-text-to-voice-alignment-of-very-long-audio-recordings
²https://github.com/audacity/audacity
³https://forum.ircam.fr/projects/detail/partiels/</footnote>
## 6.1.2 Singing datasets

### DALI

DALI (Dataset of synchronised Audio, LyrIcs and notes)<fnref n="4" /> is the first publicly available, large-scale dataset of singing voice with various annotations (MESEGUER-BROCAL et al., 2018). It contains 5,358 songs, each with note alignments and lyrics alignments at the word, sentence and paragraph levels as well as additional meta information (language, genre, artist, year, etc.). The dataset is coloured in terms of western genres (e.g., pop, rock, rap, etc.) and languages (e.g., English, French, Spanish, Italian, Polish, etc.).

It has been created using the machine learning teacher-student paradigm (MESEGUER-BROCAL et al., 2020b) from online manual annotations considered as reliable. Its release has permitted significant improvement and many success in various Music Information Retrieval (MIR) tasks involving singing, e.g., (DEMIREL et al., 2021; MESEGUER-BROCAL and PEETERS, 2020; RENAULT et al., 2021), since DALI offers sufficient amount of data to train deep models.

### *Playlist 50*

In order to have a unique dataset allowing the evaluations of both text and notes alignment for singing voice with polyphonic music, 50 songs (i.e., about 3 hours) and their available annotations were selected to constitute a test set — the so-called *Playlist 50*. It is larger than the classical evaluations sets (see below) and has originally been used to assess very long alignment by DORAS et al. (2023). The DALI IDs and annotations are accessible to the community<fnref n="5" />.

### Hansen (a capella)

Hansen's dataset (HANSEN, 2012) is one of the most famous singing datasets for evaluating and benchmarking systems on automatic lyrics transcription and alignment tasks. It is composed of 9 entire pop music songs in English with manual annotations of word onsets and offsets. The version at disposal is the *a cappella* one, hence featuring solo singing voice only.

### Jamendo

Jamendo (STOLLER et al., 2019) is also a very popular dataset to assess singing transcription and alignment. It features 20 entire music songs (from 10 different Western genres) with manual annotations of word onsets. All songs have instrumental accompaniment allowing to quantify the performances of a model on real-world data and its robustness to background music.

<footnote n="4">⁴https://github.com/gabolsgabs/DALI/</footnote>

<footnote n="5">⁵https://ircam-anasynth.github.io/papers/2023/a-linear-memory-ctc-based-algorithm-for-text-to-voice-alignment-of-very-long-audio-recordings</footnote>
## Chanter RT

The 1.5-hour French Chanter RT dataset of solo singing was recorded to build a singing system based on phonetic concatenation as detailed in (ARDAILLON, 2017). As a result, the singer (RT) was asked to sing slowly and hold long vowels to facilitate this synthesis objective. The temporal distribution of the sung phonemes is bi-modal with a lobe around 1s for the vowels and another around 200ms for the other phonemes. It features 36 French phonemes plus a silence token. This dataset will allow testing whether ADAGIO alignments are relevant for concatenative synthesis, which is one of the target applications of this work — see section 7.1.

## Section summary – Voice corpora

The diverse voice corpora used in this thesis — for experiments (training phase), evaluations (inference phase) and for voice studies — have been presented. Two of them were proposed in co-authored (DORAS et al., 2023) and further shared with the community: (1) *Philos 10* for a manually word-level aligned audiobook chapter; and (2) *Playlist 50* for a larger singing evaluation dataset with word-level notes and lyrics alignments.

## 6.2 Evaluation procedure

In this section, the complete evaluation procedure is thoroughly detailed from the computed quantitative metrics to implementation details for ADAGIO and baselines.

### 6.2.1 Assessment metrics

The quantitative evaluation of voice alignment requires the definition of assessment metrics. There already exist standard evaluation procedure and metrics for the alignment task (CONT et al., 2007). In this work, benefiting from the existence of the (in their own words) “transparent, standardized, and straightforward” library *mir_eval*<sup>6</sup> (RAFFEL et al., 2014), the metrics they proposed will be used and briefly introduced. Note that, in all computations, only the symbol onsets are considered as decision boundaries. Taking end positions into account would penalize a system detecting perfectly each start of utterance without predicting its full duration. This is commonly done in the literature, e.g., in the competitive challenge MIREX<sup>7</sup>.

The starting point of any alignment evaluation is the alignment errors between the predicted timestamps and the truth timestamps associated with each event of the symbolic sequence.

<footnote n="6"><sup>6</sup>https://craffel.github.io/mir_eval/
<sup>7</sup>https://www.music-ir.org/mirex/wiki/2020:Lyrics_Transcription</footnote>
### Average Absolute Error (AAE)

The Average Absolute Error (AAE) reports the *average* over all events of the absolute difference between estimated and reference timestamps. It is the metric most used to evaluate alignment. The lower, the better.

### Median Absolute Error (MAE)

The Median Absolute Error (MAE) reports the *median* of the absolute difference between estimated and reference timestamps over all events. In opposition to the AAE, this metric is insensitive to outliers. The lower, the better.

### Percentage of Correct Onsets (PCO)

The Percentage of Correct Onsets (PCO) is a metric measuring the percentage of predicted onset timestamps that can be considered correctly aligned. A threshold of 300ms is commonly admitted and chosen by the community for the misaligned/well-aligned binary decision (CONT et al., 2007; VAGLIO et al., 2020a), and so do these evaluations. The higher, the better.

### Karaoke Perceptual Metric (KPM)

Finally, the Karaoke Perceptual Metric (KPM), introduced as “Perc-PCO” in MASCLEF et al. (2021), takes the human synchronicity perception into account and reflects how alignments would be judged synchronous by users in a Karaoke-like scenario. The higher, the better.

The acoustic model is also often evaluated for transcription, see Eq. (2.25), via two metrics.

### Character Error Rate (CER)

The Character Error Rate (CER) indicates the percentage (%) of elements (e.g., graphemes, phonemes, notes, etc.) that were incorrectly predicted by a model in comparison to a ground-truth transcription. It is computed with the JiWER toolkit<fnref n="8" />. The lower, the better.

### Word Error Rate (WER)

The Word Error Rate (WER) indicates the percentage (%) of groups of elements (e.g., words) that were incorrectly predicted by a model in comparison to a ground-truth transcription. A single error in the group classifies the group as false. This metric is therefore much more drastic than CER. The JiWER toolkit is also used. The lower, the better.

<footnote n="8">⁸https://github.com/jitsi/jiwer</footnote>
### 6.2.2 Implementation details

In this section the practical implementations of the deep neural trainings, alignment retrieval, and baseline comparisons are mentioned.

#### Data & alphabet

For singing voice, the models are trained on the English part of the DALI dataset which excludes (1) some validation data with unique singers that are *not* part of the training data; and (2) songs from evaluation set *Playlist 50* – none of their different 50 singers are part of the training or validations sets. DALI songs were segmented into small audio excerpts between 10 and 20s (with their transcripts) by making sure not to truncate the last annotated word. For speaking voice, the models are trained on the complete LibriSpeech dataset. The validation set is TIMIT.

Depending on the experiments different alphabets $\mathcal{A}$ will be used. For phonetic alignment, a custom alphabet covering all the phonemes encountered in the associated database is constituted. For text alignment, the augmented Latin alphabet $\mathcal{A}_\ell$ from Eq. (2.1), which includes graphemes, digits and a space token $\emptyset$ with a size of $L = 37$, is considered. For note alignment, a note alphabet $\mathcal{A}_j$ is defined. To do so, the F0 annotations in DALI were retrieved and converted into notes. The resulting notes ranged from $C_1$ to $C_7$, which is particularly large for the human voice, but a manual inspection of outliers has not been pursued. The alphabet $\mathcal{A}_j$ contains all 12 semitones per octave and a silence token for long pauses (i.e., silences longer than 500ms according to DALI annotations), hence a size of $L = 73$.

#### Training setup

Unless clearly stated otherwise, all experiments rely on the same training procedure.

Training is performed on Graphical Processing Units (GPU) – either with one GeForce GTX 1080 Ti with 11Go or NVIDIA TITAN Xp COLLECTORS EDITION with 12Go – to benefit from their computational power and parallelization strategies. A number of 10 epochs is typically chosen. Each epoch processes the entire database by means of 16-sample (for DALI) or 8-sample (for LibriSpeech) batches. The loss(es) function(s) is(are) minimized with default ADAM optimizer and an initial learning rate set to $\lambda = 1e^{-4}$.

At the end of each epoch, the Average Absolute Error (AAE) is computed over the associated validation set. The learning rate is reduced by a factor 0.8 each time the alignment error has not decreased on the evaluation set for 2 consecutive epochs. The training time for one configuration of ADAGIO on DALI/Librispeech in the given setup ranges between 18h (no constraints) to 29h (all constraints).

Codes are written in Python/Tensorflow and are supported, at the time of writing, in Tensorflow 2.8. The original inspiration was the CTCModel implementation as proposed and shared by SOULLARD et al. (2019).
### Alignment retrieval

Remembering the general overview of any voice aligner (see Figure 3.2), an acoustic model $M_\Theta$ generates an encoding of the audio **x** and, second, a dedicated decoding module $D_\pi$ is used to force-align the target sequence **y** to the temporal axis of the encoded representation $M_\Theta(\mathbf{x})$.

In this context, ADAGIO acts as the acoustic model and generates a CTC posteriorgram $\mathcal{P}$ from the voice recordings, as explained in section 3.5.1, such that $\mathcal{P} = M_\Theta(\mathbf{x})$. It is the role of the specific CTC decoder, which is the focus of section 3.5.2, to finally retrieve $\hat{\pi}^*$ – the estimated alignment between **x** and **y**.

### Baselines & state of the art

The performances of ADAGIO are then compared to those of other alignment systems.

**Baselines** – In terms of end-to-end neural architectures, CRNN (VAGLIO et al., 2020a) and ARNN (SCHULZE-FORSTER et al., 2020) have been re-implemented and will be used for polyphonic music evaluation (CRNN) and for phonetic alignment (ARNN) – see section 4.1.1.

**State of the art** – Results from the latest Music Information Retrieval Evaluation eXchange (MIREX)<fnref n="9" /> with lyrics alignment task are also reported as the work of GAO et al. (2021) remains, to the best of the author’s knowledge, current state of the art in Automatic Lyrics Alignment (ALA) on classical evaluation datasets. It is based on the system of GUPTA et al. (2020), which exploits (1) an acoustic model made of time-dilated neural layers trained with Kaldi (POVEY et al., 2011) on the English subset of DALI; (2) an extended pronunciation lexicon addressing the long hold on vowels in singing (GUPTA et al., 2018); (3) additional genre-informed modeling of phonemes and silences (GUPTA et al., 2020); and (4) a tri-gram word language model created from the available lyrics. It is directly usable on polyphonic music but, as seen, connects several specialized modules, as opposed to end-to-end models. This model is referred to as SOTA.

## Section summary – Evaluation procedure

ADAGIO, upon training on a single GPU with publicly available dataset for speech and singing, will be used in inference to estimate voice-to-symbol alignments (predictions) that will be compared to annotations at disposal (ground-truth). The temporal differences will allow to assess the model with four metrics for alignment – i.e., Average Absolute Error (AAE), Median Absolute Error (MAE), Percentage of Correct Onsets (PCO) and Karaoke Perceptual Metric (KPM) –, and two metrics for transcription – i.e., Character Error Rate (CER) and Word Error Rate (WER). Comparisons with existing baselines are performed via the re-implementation of CRNN and ARNN models and latest MIREX challenge results to relate to the state of the art (SOTA) held by GAO et al. (2021).

<footnote n="9">⁹https://www.music-ir.org/mirex/wiki/2020:Lyrics_Transcription</footnote>
<table><caption>Table 6.2: Model configurations for ADAGIO according to the loss(es) to minimize.</caption><thead><tr><th>Configuration</th><th><i>L</i><sub>CTC</sub>(Θ)</th><th><i>L</i><sub>REC</sub>(Θ)</th><th><i>L</i><sub>STR</sub>(Θ)</th><th><i>L</i><sub>DTW</sub>(Θ)</th></tr></thead><tbody><tr><td>C</td><td>×</td><td></td><td></td><td></td></tr><tr><td>CR</td><td>×</td><td>×</td><td></td><td></td></tr><tr><td>CS</td><td>×</td><td></td><td>×</td><td></td></tr><tr><td>CD</td><td>×</td><td></td><td></td><td>×</td></tr><tr><td>CRS</td><td>×</td><td>×</td><td>×</td><td></td></tr><tr><td>CRD</td><td>×</td><td>×</td><td></td><td>×</td></tr><tr><td>CSD</td><td>×</td><td></td><td>×</td><td>×</td></tr><tr><td>CRSD</td><td>×</td><td>×</td><td>×</td><td>×</td></tr></tbody></table>

<table><caption>Table 6.3: Impact of additional temporal constraints (with scaling and supervised on vocals) on voice-to-word alignment. For metrics, ↑ means higher is better, ↓ means lower is better, and 95% confidence intervals on the mean are shown.</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Configuration</th><th colspan="5">Alignment metrics</th><th colspan="2">Transcription</th></tr><tr><th>AAE [ms]↓</th><th>MAE [ms]↓</th><th>PCO [%]↑</th><th>KPM [%]↑</th><th>CER [%]↓</th><th>WER [%]↓</th></tr></thead><tbody><tr><td rowspan="7">Playlist50</td><td>C</td><td>124.2 (± 29.2)</td><td>39.2 (± 3.7)</td><td>95.2 (± 1.1)</td><td>88.6 (± 1.4)</td><td>53.7 (± 1.3)</td><td>94.5 (± 1.7)</td></tr><tr><td>CR</td><td>96.2 (± 19.2)</td><td>39.6 (± 4.7)</td><td>96.5 (± 0.8)</td><td>87.9 (± 1.3)</td><td>49.4 (± 1.6)</td><td>91.3 (± 2.2)</td></tr><tr><td>CS</td><td>103.4 (± 27.6)</td><td>39.7 (± 4.7)</td><td>96.4 (± 1.0)</td><td>87.9 (± 1.3)</td><td>50.1 (± 1.6)</td><td><b>89.1</b> (± 1.7)</td></tr><tr><td>CD</td><td>98.8 (± 17.3)</td><td>38.9 (± 3.0)</td><td>96.3 (± 0.8)</td><td><b>90.4</b> (± 1.1)</td><td>51.3 (± 1.6)</td><td>91.2 (± 1.6)</td></tr><tr><td>CRS</td><td>102.6 (± 18.5)</td><td>39.6 (± 4.3)</td><td>95.7 (± 0.9)</td><td>87.7 (± 1.3)</td><td>51.3 (± 1.5)</td><td>91.4 (± 1.5)</td></tr><tr><td>CRD</td><td>96.1 (± 13.2)</td><td>41.5 (± 3.5)</td><td>96.3 (± 0.8)</td><td>50.2 (± 1.6)</td><td>50.2 (± 1.6)</td><td>89.9 (± 1.7)</td></tr><tr><td>CSD</td><td>98.5 (± 38.5)</td><td>38.5 (± 4.4)</td><td>96.6 (± 0.8)</td><td>88.5 (± 1.2)</td><td><b>49.0</b> (± 1.6)</td><td>90.1 (± 2.1)</td></tr><tr><td></td><td>CRSD</td><td><b>93.1</b> (± 16.4)</td><td><b>38.0</b> (± 4.0)</td><td><b>96.8</b> (± 0.8)</td><td>89.2 (± 1.2)</td><td>49.4 (± 1.7)</td><td>89.6 (± 2.2)</td></tr><tr><td rowspan="2">Philos 10</td><td>C</td><td>51.2 (± 19.1)</td><td>46.2 (± 2.6)</td><td><b>100.0</b> (± 0.0)</td><td>95.6 (± 2.1)</td><td><b>15.1</b> (± 1.1)</td><td><b>47.9</b> (± 2.6)</td></tr><tr><td>CRSD</td><td><b>45.7</b> (± 20.1)</td><td><b>39.9</b> (± 2.1)</td><td><b>100.0</b> (± 0.0)</td><td><b>96.7</b> (± 1.8)</td><td>17.9 (± 2.5)</td><td>55.0 (± 2.9)</td></tr></tbody></table>

## 6.3 Ablation study

Before comparing the ADAGIO system to reference aligners, its best version must be determined. In view of the proposals introduced in the previous Part II, comparative tests must be conducted on the various versions of ADAGIO based on the possible combinations of additional temporal constraints. This section specifically addresses such ablation studies. The denomination of the model's configurations with respect to their supervision strategy is exposed in Table 6.2.

### *Impacts of the temporal constraints*

The Table 6.3 reports the evaluations of ADAGIO on voice-to-word alignment for its different configurations (these are also depicted on the left side of Figure 6.1 and Figure 6.2).

As singing voice in a musical context is much more challenging to align than clean speech, the observed effects are stronger for singing, hence easier to showcase – that is why, despite similar conclusions for speech and singing, all configurations are shown for *Playlist 50* and only the two extremes (CTC alone vs fully constrained CTC) for *Philos 10*.
First, one can see that *the proposed additional temporal constraints are indeed beneficial to the alignment task*. The very fact to add one supplementary loss already improves the AAE. Interestingly, the MAE remains quite stable among the configuration and around 40ms for both speech and singing voice. The configuration with best potential for *karaoke* application is CD with the best KPM – its alignment would globally be perceived more synchronous in a karaoke-like display. It is also confirmed, as stated in *section 5.1*, that better performances in *transcription* with CTC do not necessarily result in better *alignment*.

For speech, the AAE is typically half of that observed for singing. This indicates that there are fewer outliers for speech and that these outliers are less severe, which is in line with the absence of a high “corruption” of the voice through the musical accompaniment.

The full combination of constraints, *i.e.*, the final configuration CRSD, again slightly improves the overall alignment accuracy and stands out in AAE, MAE and PCO metrics. Thus, from now on, any mention of ADAGIO will systematically refer to the CRSD configuration.

*Impact of scaling the losses*

The Figure 6.1 shows, for each configuration, the Average Absolute Error (AAE) measured whether the temporal constraints are scaled or not. One can see the utmost importance of the scaling process to obtain relevant alignment performances.

In the case of the *reconstruction* loss $\mathcal{L}_{\text{REC}}(\Theta)$ – without scaling, the reconstruction dominates the CTC cost such that the model struggles to converge and denegerates into some simple “spectral auto-encoder” that does not perform alignment. This holds true for the structural loss $\mathcal{L}_{\text{STR}}(\Theta)$ with an even stronger degradation as it scales quadratically in audio length.

Interestingly, as the monotony loss $\mathcal{L}_{\text{DTW}}(\Theta)$ does not require any scaling – see Eq. (5.15), the configuration CD is less sensitive to the scaling process. Yet, no scaling leads to a CTC loss dominating DTW, which turns out to have a negative impact on the alignment.

*Impact of voice supervision*

The Figure 6.2 shows, for each configuration, the Average Absolute Error (AAE) measured whether the temporal constraints are supervised on *mixes* or *vocals* (estimated, as explained in *section 5.2.1*), via a voice separator based on the work of CHOI et al. (2019)).

It is straightforward to conclude that supervision on the vocals is the right option, as the mixes contain music instrumental that does not inform about voice properties and, if considered as ground-truth for the temporal constraints, necessarily confuses the network. Here again, as the monotony loss $\mathcal{L}_{\text{DTW}}(\Theta)$ is independent from any references derived from audio features for its computation – but only requires the available posteriorgrams and transcripts –, the configuration CD is not affected by this study.
<figure><img src="image_45.png" /><figcaption>Figure 6.1: Ablation study – impact of scaling the losses. Errors bars correspond to 95% confidence intervals on the mean. AAEs greater than 500ms are masked for readability.</figcaption></figure>

<figure><img src="image_46.png" /><figcaption>Figure 6.2: Ablation study – impact of the temporal constraints and the nature of their supervision (mixes vs vocals). Errors bars correspond to 95% confidence intervals on the mean. AAEs greater than 200ms are masked for readability.</figcaption></figure>
<figure><img src="image_47.png" /><figcaption>Figure 6.3: Impact of the several temporal constraints during the training phase. The sample used to create the figure was selected from the *validation* set at the end of the first, an intermediate and the final epoch. Example is a clean singing voice dataset. Throughout training, ADAGIO progressively learns not only to recognize the correct phonemes but also to solve the additional constraints leading to better alignments (see evaluations).</figcaption></figure>

## Visualizations during training

Finally, for illustrations purposes, the Figure 6.3 shows the evolution of the different losses during training.

At the beginning of the training, no information is captured so that neither recognition nor alignment are actually performed. A little bit at the time, symbols (here, phonemes) are recognized and their selection serves to reconstruct an estimate of the spectral envelope, temporal structure and construct an alignment score. By the end of the training procedure, phonetic content is well recognized and predicted with high temporal relevance so that the additional temporal constraints are also satisfied.
## Section summary – Ablation study

Through ablation studies, the relevance of the additional temporal constraints proposed in this thesis was confirmed. The best configuration, in terms of AAE, MAE and PCO, is associated with the full combination of these supplementary objectives and will be systematically implied hereinafter when mentioning ADAGIO. Also, the necessity to scale the various losses and to supervise them on *vocals* – estimated via a source separation algorithm – instead of direct mixes was highlighted.

# 6.4 Temporal alignment results

This section is dedicated to the evaluation of the optimal version of ADAGIO, according to above-mentioned ablation studies, on various voice databases and its comparison with other algorithms from the literature. On the one hand, voice-to-*text* alignment is evaluated at different granularities – word-level and phone-level text synchronizations. On the other hand, voice-to-*note* alignment is also assessed. These three concrete usages will have their importance in the next *Chapter 7* dedicated to the exploitations of such time-symbol mappings.

## 6.4.1 Voice-to-text alignment

First, the task of synchronizing an audio recording with its text transcript is evaluated. Depending on the chosen text representations – ultimately engraved in the used alphabet $\mathcal{A}$ which can be composed of graphemes or phonemes – two subtasks can be investigated, i.e., alignment of word and alignment of phonemes.

### *High level alignment – word granularity*

The Table 6.4 sets the results for *word*-level alignment out for speech and singing voice. From this table, one can conclude that ADAGIO produces relevant alignment performances that compare favorably to the Convolutional Recurrent Neural Network (CRNN) baseline, even outperforming it on most alignment metrics. The CRNN model, though, remains almost systematically better for pure transcription, which can be explained by its recurrent architecture.

However, SOTA performances measured in Average Absolute Error (AAE) are not reached by ADAGIO on the classical evaluations datasets (Jamendo and Hansen). *Ad hoc* error repartition revealed two strong outliers for 2 out of the 20 songs in Jamendo, and especially an extreme one (>4s) for a song that, interestingly, all MIREX submissions are struggling to align<fnref n="10" />. This naturally degrades the AAE, all the more so given the small number of track samples.

<footnote n="10">¹⁰*Pure_Mids_-_The_Leader_*</footnote>
<table><caption>Table 6.4: Results on voice-to-word alignment. For metrics: ↑ means higher is better, ↓ means lower is better, and 95% confidence intervals on the mean are shown. Results for SOTA are from the latest MIREX challenge (https://www.music-ir.org/mirex/wiki/2020: Automatic_Lyrics-to-Audio_Alignment_Results) and only share AAE and MAE metrics for the two classical evaluations sets Jamendo and Hansen.</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Model</th><th colspan="4">Alignment metrics</th><th colspan="2">Transcription</th></tr><tr><th>AAE [ms]↓</th><th>MAE [ms]↓</th><th>PCO [%]↑</th><th>KPM [%]↑</th><th>CER [%]↓</th><th>WER [%]↓</th></tr></thead><tbody><tr><td rowspan="2">Philos10</td><td>ADAGIO</td><td><b>45.7</b> (± 20.1)</td><td><b>39.9</b> (± 2.1)</td><td><b>100.0</b> (± 0.0)</td><td><b>96.7</b> (± 1.8)</td><td><b>17.9</b> (± 2.5)</td><td>55.0 (± 2.9)</td></tr><tr><td>CRNN</td><td>79.2 (± 30.2)</td><td>56.9 (± 6.9)</td><td><b>100.0</b> (± 0.0)</td><td>94.9 (± 3.3)</td><td>18.9 (± 2.1)</td><td><b>53.9</b> (± 3.2)</td></tr><tr><td rowspan="2">Playlist50</td><td>ADAGIO</td><td><b>93.1</b> (± 16.4)</td><td><b>38.0</b> (± 4.0)</td><td><b>96.8</b> (± 0.8)</td><td><b>89.2</b> (± 1.2)</td><td>49.4 (± 1.7)</td><td>89.6 (± 2.2)</td></tr><tr><td>CRNN</td><td>135.4 (± 13.6)</td><td>42.5 (± 4.9)</td><td>93.5 (± 0.7)</td><td>88.4 (± 1.2)</td><td><b>42.8</b> (± 2.3)</td><td><b>81.7</b> (± 2.2)</td></tr><tr><td rowspan="3">Hansen</td><td>ADAGIO</td><td>115.5 (± 104.1)</td><td>47.3 (± 11.9)</td><td>97.4 (± 2.6)</td><td>84.0 (± 2.9)</td><td><b>36.8</b> (± 3.8)</td><td>75.6 (± 4.2)</td></tr><tr><td>CRNN</td><td>146.5 (± 57.0)</td><td>51.3 (± 10.0)</td><td>96.2 (± 2.2)</td><td>82.2 (± 3.1)</td><td>37.4 (± 4.3)</td><td><b>72.1</b> (± 4.2)</td></tr><tr><td>SOTA</td><td><b>86.7</b> (± 65.6)</td><td><b>31.6</b> (± 7.5)</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td rowspan="3">Jamendo</td><td>ADAGIO</td><td>284.8 (± 212.3)</td><td>47.7 (± 12.5)</td><td><b>94.5</b> (± 2.7)</td><td>84.0 (± 3.5)</td><td>49.2 (± 4.0)</td><td>87.4 (± 4.1)</td></tr><tr><td>CRNN</td><td>323.8 (± 89.3)</td><td>55.7 (± 4.5)</td><td>93.2 (± 2.0)</td><td><b>84.2</b> (± 2.2)</td><td><b>46.4</b> (± 3.2)</td><td><b>83.1</b> (± 3.8)</td></tr><tr><td>SOTA</td><td><b>217.0</b> (± 127.8)</td><td><b>46.1</b> (± 6.1)</td><td>—</td><td>—</td><td>—</td><td>—</td></tr></tbody></table>

Yet, the Median Absolute Error (MAE) values produced by ADAGIO – 40-50ms on all test sets – are in line with the ones reported for SOTA. Therefore, it can be stated that these assessments are very much acceptable remembering that SOTA involves acoustic, language, genre and pronunciation models while ADAGIO relies on a simpler, fully data-driven architecture.

## Low level alignment – phoneme granularity

The Table 6.5 sets the results for phoneme-level alignment out. In this table, the Phoneme Error Rate (PER) metric is defined by analogy with the Character Error Rate (CER).

For speech, TIMIT is used as it also features phonetic alignment. Data are already divided into train and test sets. While all three aligners can precisely align phonemes on clean speech, ADAGIO stands out for its alignment quality together with the other criteria that motivated its construction: absence of recurrent layers (as opposed to CRNN & ARNN), and only processing audio (as opposed to ARNN). Again, CRNN remains a better transcriber. In comparison to the first thesis proposal in (TEYTAUT and ROEBEL, 2021) with proper similar configuration, AAE (44.1ms) and MAE (27.3ms) have significantly improved with the ADAGIO architecture.

A second evaluation is done on Chanter RT, which also comes with phonetic alignment ground-truths. Yet, just like the specific nature of this dataset (single pitch point, vowels voluntarily held very long, less than 2h of audio in total), a special alignment procedure is proposed here. Instead of dividing the dataset into train and test sets, it is entirely used for training and test stages. 100 training epochs processing the whole dataset are used.
<table><caption>Table 6.5: Results on voice-to-*phoneme* alignment. For metrics: ↑ means higher is better, ↓ means lower is better, and 95% confidence intervals on the mean are shown. “n/a”: non-applicable when overfitting (texts are known) and for ARNN (phonemes are model inputs).</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Model</th><th colspan="4">Alignment metrics</th><th>Transcription</th></tr><tr><th>AAE [ms]↓</th><th>MAE [ms]↓</th><th>PCO [%]↑</th><th>KPM [%]↑</th><th>PER [%]↓</th></tr></thead><tbody><tr><td rowspan="3">TIMIT</td><td>ADAGIO</td><td><b>18.0</b> (± 0.7)</td><td>14.3 (± 0.5)</td><td><b>100.0</b> (± 0.0)</td><td><b>100.0</b> (± 0.0)</td><td>35.2 (± 4.7)</td></tr><tr><td>CRNN</td><td>20.5 (± 1.2)</td><td>13.3 (± 0.5)</td><td><b>100.0</b> (± 0.0)</td><td><b>100.0</b> (± 0.0)</td><td><b>29.3</b> (± 3.3)</td></tr><tr><td>ARNN</td><td>22.5 (± 1.5)</td><td><b>12.2</b> (± 0.6)</td><td><b>100.0</b> (± 0.0)</td><td><b>100.0</b> (± 0.0)</td><td>n/a</td></tr><tr><td rowspan="3">Chanter RT<br>(overfitting)</td><td>ADAGIO</td><td><b>39.3</b> (± 1.8)</td><td><b>28.0</b> (± 0.9)</td><td><b>99.3</b> (± 0.2)</td><td>94.7 (± 0.2)</td><td>n/a</td></tr><tr><td>CRNN</td><td>52.2 (± 1.8)</td><td>35.2 (± 3.3)</td><td>99.0 (± 0.2)</td><td>93.9 (± 0.2)</td><td>n/a</td></tr><tr><td>ARNN</td><td>44.5 (± 3.4)</td><td>34.0 (± 1.8)</td><td><b>99.3</b> (± 0.2)</td><td><b>96.3</b> (± 0.2)</td><td>n/a</td></tr></tbody></table>

In doing so, *overfitting* is deliberately intended. Such an uncommon approach in deep learning, as one is usually looking for models with high generalization capabilities and avoid specializations on training data, actually has its interest for aligning small datasets with specific phonetic alphabet (theoretically any custom alphabet) whose small amount of data would prevent the learning of an inference algorithm anyway. However, since the CTC only requires weak labeling for its supervision, the symbol timestamps are new information gained throughout the training procedure. The reported results highlights the capability of ADAGIO to competitively produce phonetic alignments in such a context. The overall process (training and alignment retrieval) took about ~2.5h. Concrete applications of this strategy will be detailed in section 7.1 and section 7.2.

## 6.4.2 Voice-to-note alignment

The Table 6.6 sets the results for note alignment out. In this table, the Note Error Rate (NER) metric is defined by analogy with the Word Error Rate (WER) as notes, similar to words for a text, are here represented through a succession of characters, e.g., D4# or 0 for silences, separated by a space.

In the same vein as the previous evaluations, one can see that the recurrent architecture (CRNN) achieves better recognition than the purely the convolutional one (ADAGIO), but the later clearly outperforms the baseline when aligning a singing melody to an audio performance.

This note aligner has been successfully integrated in a musicological study, see section 7.3.3, thus demonstrating the practical feasibility of aligning singing notes with CTC. To the best of the author's knowledge, it is the first time that an end-to-end, CTC-based model addresses note alignment – while note transcription was recently tackled (WEISS and PEETERS, 2021). Yet, as all the metrics reported, and notably the Average Absolute Error (AAE) and Median Absolute Error (MAE), remain quite high, these results call for modesty and shall be seen as a first step towards future investigations – see perspectives in section 7.4.
<table><caption>Table 6.6: Results on voice-to-*note* alignment task. For metrics: ↑ means higher is better, ↓ means lower is better, and 95% confidence intervals on the mean are shown.</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Model</th><th colspan="4">Alignment metrics</th><th>Transcription</th></tr><tr><th>AAE [ms]↓</th><th>MAE [ms]↓</th><th>PCO [%]↑</th><th>KPM [%]↑</th><th>NER [%]↓</th></tr></thead><tbody><tr><td rowspan="2">Playlist50</td><td>ADAGIO</td><td><b>231.8</b> (± 73.1)</td><td><b>105.1</b> (± 47.3)</td><td><b>83.6</b> (± 4.4)</td><td><b>75.3</b> (± 4.0)</td><td>66.7 (± 5.7)</td></tr><tr><td>CRNN</td><td>334.1 (± 100.8)</td><td>139.3 (± 39.9)</td><td>79.7 (± 3.8)</td><td>70.0 (± 3.2)</td><td><b>59.7</b> (± 3.5)</td></tr></tbody></table>

In terms of cross-modality comparisons, two statements can be made. First, regarding *recognition*, it appears than musical transcription is easier than textual transcription – although reported performances are far from being exploitable for, e.g., automatic melody transcription. Second, with an AAE and MAE more than twice as large for note alignment than for word alignment, it is clear that the voice-to-note (V2N) alignment problem is more challenging than its voice-to-word (V2W) counterpart.

Three reasons come to mind: (1) for V2N, *each* note is individually aligned (with background music, as opposed to phonetic alignment) whereas for V2W, *several characters* are in the end merged into a single timestamp, which may forgive some local mistakes; (2) the F0 ground-truth in DALI, which is used to obtain the note sequences, is not as reliable as lyrics annotation<fnref n="11" />; and (3) the musical background is in the *same* modality as the symbolic sequence to uncover in the case of V2N so that the network must learn to distinguish two types of musical information (*i.e.*, singing voice pitches and chords), often coherently and harmonically mixed, which may be more complex task than “fully” ignoring the musical context to focus exclusively on voice production mechanism.

For the sake of illustration, the Figure 6.4 displays an example of inference with this model.

### 6.4.3 Robustness to transcription errors

In practice, the symbolic transcriptions associated with an audio are error prone and can typically contain mistakes. For instance, a text composed of words can contain typos – e.g., one or several characters might have been added, removed, or altered (wrongly spelled for instance), or entire words could be inserted or deleted. In singing, lyrics also carry their own level of uncertainties as onomatopoeia, words or even lines or paragraphs (e.g., spoken interlude in a song) that can actually be heard in the recording might be missing in the transcript. This holds true for music scores which, depending on the context and the singer's actual interpretation of the score, might not represent all notes really sung in a given performance.

<footnote n="11">¹¹This is even explicitly stated on the official DALI webpage: https://github.com/gabolsgabs/DALI (accessed 04/04/2023)</footnote>
<figure><img src="image_48.png" /><figcaption>Figure 6.4: An example of inference for voice-to-note alignment with ADAGIO. Note that in this context *e* represents an excitation signal (all but spectral envelope). The posteriorgram, temporally constrained during training, predicts per-frame probability of note occurrences.</figcaption></figure>

In this section, the robustness of ADAGIO to these kinds of transcription errors is evaluated for voice-to-word alignment. To this aim, the alignment accuracy is measured on *Philos 10* and *Playlist 50* with modified transcripts. Similar studies were presented in the publication (DORAS et al., 2023) yet, although close conclusions are made, these are distinct evaluations.

### *Effect of replaced characters*

Figure 6.5 first shows the impact of character substitution. For these investigations, *p*% of the characters are randomly replaced in both *Philos 10* and *Playlist 50*. The space characters *ø* are not modified so that the number of words is not altered.

The new characters are sampled uniformly at random in the letters (a, ..., z). The audio posteriorgram *P* is not altered by these transcriptions as the acoustic model *M*<sub>θ</sub> exclusively processes the audio. The difference is that the decoding module *D*<sub>π</sub> here tries to force-align *P* to the altered transcript.

For *speech* — one can see that the alignment error remains remarkably stable around the baseline (i.e., true transcript) up to 50% of replaced characters. The error even remains under a 200ms threshold up to 80% of substitutions, but dramatically rises after that. The system is thus able to compensate for transcription errors, as long as it obtains enough correct anchorage characters, which is guaranteed by the space token *ø*.
<figure><img src="image_49.png" /><figcaption>Figure 6.5: Average Absolute Error (AAE) for different percentages of substituted characters for speaking and singing voices with ADAGIO.</figcaption></figure>

For *singing* — the error increases much faster than for speech, even for a small percentage of replaced characters. Due to the presence of musical accompaniment, the posterior program exhibits less clear-cut character probabilities, which causes the correct anchor characters to be more easily confused with wrong characters and the optimal alignment path to diverge faster during the decoding step.

### *Effect of added/removed characters*

Figure 6.6 then shows the impact of character insertion and deletion. For these investigations, *p*% of the characters are randomly added to or removed from both *Philos 10* and *Playlist 50*. The space characters Ø are not modified so that the number of words is not altered. When *p* decreases towards -100%, it becomes likely that several characters will be removed per word — it is thus ensured that at least one character remains for each word. The alignment path is estimated between the posterior program and the new transcript.

For *speech* — the alignment accuracy remain extremely stable around the reference (true transcript) up to 50% of the added/removed characters. The error increases steadily when adding extra characters, as expected. More surprisingly, alignment becomes a bit better when removing 0–30% of the characters. An interpretation for this effect is that removing some characters could be beneficial for the CTC decoding should the model fail to recognize them properly. When more than 50% of the characters are removed, the error increases again.
<figure><img src="image_50.png" /><figcaption>Figure 6.6: Average Absolute Error (AAE) for different percentages of character insertion and deletion for speaking and singing voices with ADAGIO.</figcaption></figure>

Interestingly, the Average Absolute Error (AAE) always remains under 200ms, even after each word contains only one single character. This indicates and confirms that the space character `Ø` is a very powerful anchor, and plays a crucial role in word-level speech alignment.

For *singing* — once again, the model is way more sensitive to these alterations. This can be similarly explained by the fact that anchor characters are more ambiguous for singing than for speech, which makes the optimal path decoding more difficult and more subject to divergence.

## *Effect of added/removed words*

Figure 6.7 finally shows the impact of *word* insertion and deletion. For these investigations, *p%* of the words are randomly added to or removed from *Philos 10* and *Playlist 50*. For each added word, a length *l* is randomly drawn from a normal distribution with a mean of six characters, and then *l* characters are sampled uniformly at random in the letters (a, ..., z). In these evaluations, the AAE is computed only on the original words, for which true timestamps exist.

For *speech* — the alignment error remains under 200ms error even when every other word is removed or added. This indicates that the system is very robust to noise as long as more than half of the true words remain untouched. However, at some point, there are not enough words to keep the alignment stable, dramatically increasing the AAE.
<figure><img src="image_51.png" /><figcaption>Figure 6.7: Average Absolute Error (AAE) for different percentages of word insertion and deletion for speaking and singing voices with ADAGIO.</figcaption></figure>

For *singing* — similarly to the previous experiments, one can see that singing voice alignment is much more sensitive to a full word addition or deletion. This can be interpreted by the fact that the posteriorgram does not exhibit sufficiently salient probabilities for correcting characters in order to provide steady anchorage compensating for transcription errors.

### Section summary – Temporal alignment results

According to various experiments with data with reference alignments, ADAGIO was demonstrated to produce relevant voice-to-symbol alignments in line with recent baselines. First, text alignment has been studied at the word level (by classical inference) and at the phoneme level (by overfitting strategy) and has shown relevant performances with a Median Absolute Error (MAE) below 50ms for both types of voice. The robustness to imperfect transcripts was also investigated and revealed that the system can keep aligning as long as a minimal amount of correct characters serve as anchors for the decoding. Although inferior to the state of the art on lyrics alignment, ADAGIO stands out by its simplicity and ability to be easily transferred from one domain to another thanks to its end-to-end architecture. Second, albeit to a lesser extent, note alignment with ADAGIO was also tackled as a first step towards further development.
## 6.5 Evaluations of deep voice alignment in a nutshell

### Chapter summary – Evaluations of deep voice alignment

In this chapter, I quantitatively showed that my proposed contributions – ADAGIO and its enhancement via temporal constraints – were well-suited for the synchronization between voice and symbols. Based on various *voice corpora*, two of which introduced in this research as evaluation sets and shared with the community, I conducted key experiments to evaluate my acoustic model ADAGIO. First, through an *ablation study*, I demonstrated that all additional temporal constraints ultimately have a positive effect on the alignment accuracy. Then, I compared the best version of ADAGIO, fully time-constrained, to recent baselines and state of the art for text and note alignments. I highlighted the various associated *temporal alignment results* to gather insights from these assessments. Concretely, ADAGIO can perform word-level and phone-level alignment with a median error below 50ms for both speaking and singing voices. Although state-of-the-art performances are not reached for lyrics alignment, ADAGIO stands out by an end-to-end architecture, allowing great flexibility. Finally, note alignment was shown to be feasible and promising but, at the time of writing, still requires further investigations.
Chapter 7

Applications and collaborations

"Hier encore, j'avais vingt ans, je gaspillais le *Temps* en croyant l'arrêter, et pour le retenir, même le devancer, je n'ai fait que courir, et me suis essoufflé."

Hier Encore (1964)

— Charles AZNAVOUR

The whole manuscript, up to this point, has consisted in the exhaustive presentation of the alignment task between voice and symbolic data as well as its practical implementation through AGADIO – an acoustic neural model reinforced by temporal constraints – whose relevance has been validated by quantitative evaluations. It is now *time* for practice. This chapter, therefore, presents various concrete applications of the ADAGIO algorithm. **These results are from collaborations with other researchers and must be seen as proofs of concept of the ADAGIO aligner, rather than original research work done solely by the author.**

In *section 7.1*, the task of *singing voice synthesis* based on the concatenation of a phonetically aligned singing dataset is addressed. In the following *section 7.2*, speech phonetic alignment is used to identify some *temporal production strategies* involved in the expression of social attitudes. Continuing the study of vocal expressivity, the next *section 7.3* focuses on the *musicological characterization* of singing style, which is at the heart of artistic choices in vocal performances, through syllabic and note alignments. Finally, some *ongoing research and future works* are mentioned in *section 7.4*. A summary of the chapter is proposed in *section 7.5*.

<figure><img src="image_52.png" /></figure>
<figure><img src="image_53.png" /><figcaption>Figure 7.1: Global view of a concatenative synthesis system. To generate an utterance, (di)phonetic units are selected from a pre-aligned database; units are successively concatenated and transformed according to some target parameters. Phonetic alignments of the reference dataset are thus required beforehand. Highly inspired from Figure 3.1 of ARDAILLON (2017).</figcaption></figure>

## 7.1 Concatenative singing synthesis (ISiS)

As first concrete application, temporal voice alignment is used as a means to enrich a singing voice synthesis algorithm based on a concatenative approach which, by nature, requires precise knowledge of the phonetic regions in a reference pre-aligned database. The core of the singing synthesizer – Ircam Singing Synthesis (ISiS)<fnref n="1" /> – has been developed in ARDAILLON (2017)'s thesis and has not been modified in this work. The application of ADAGIO is an upstream contribution, independent from the synthesis motor *per se* yet necessary to its functioning.

### Context

An overview of the synthesis strategy can be found on Figure 7.1 and is briefly summarized. An user asks to synthesize a *target utterance* (*e.g.*, “C’est une chanson”) associated with given *control parameters* (*e.g.*, note pitches and durations, singing style, etc.). The desired text is firstly turned into a phonetic sequence (*e.g.*, “s E t y n S a s o _”) with a dedicated algorithm. The resulting phonemes are grouped into *diphones*, *e.g.*, succession of two phonemes as [sE] [Et]. The theory of diphones stipulates that transitions between diphones are smoother than between raw phonemes. Then, relevant sound units are selected from an existing singing database that (1) covers all diphones of the language; and (2) *has been temporally aligned*. For each diphone, the unit corresponding best to its surrounding context is chosen. Units are successively concatenated resulting in a first version of the synthesis. The final step consists in transforming the pitches, durations and transitions between the units with signal processing tools to fit some control parameters, *e.g.*, music score or singing style (ARDAILLON et al., 2016).

<footnote n="1">¹https://forum.ircam.fr/projects/detail/isis/</footnote>
<figure><img src="image_54.png" /><figcaption>Figure 7.2: Example of a sound sample found in Chanter RT, a reference corpus for ISiS, with phonetic alignment of the French locution “Rythmë” (*rhythmic*) phonemized as “R i t m e”.</figcaption></figure>

Unit selection, as a fundamental part of this system, is heavily dependent on the phonetic alignment quality for the reference corpus of sounds. At the time of ISiS creation, an industrial partner with proprietary algorithms was in charge of aligning the corpus. In order to diverse the potential of ISiS, e.g., adding new voices and languages, a voice-to-phoneme aligner is necessary. (The Ircam Analysis-Synthesis team voice aligner, “ircamAlign” (LANCHANTIN et al., 2008), is not adapted to ISiS corpora as its statistical models were trained for speech.) This is where ADAGIO comes into play. In the context of **Ircam’s UPI 2022 “ISiS Voices”**, new singers were recorded for ISiS. The ground-truth phonetic sequences are known but are not aligned – which opens an interesting opportunity for ADAGIO.

*Alignment strategy*

Currently, only French language is supported in ISiS but extension to other languages is highly desirable. The recorded datasets are small (less than 2h of audio) and composed solely of short audio excerpts (around 3s) in which vowels are purposely held long to facilitate “diphonic” unit selection. An example is depicted in Figure 7.2. Theoretically, any phonetic alphabet could be used as each language would rely on different phonemes. The International Phonetic Alphabet (IPA) being quite large and not fully covered in all languages does not seem as interesting as having a specific and dedicated phonetic catalog per language.

All in all, training ADAGIO as an acoustic model capable of inferring phonetic posteriograms for *all* languages seems vain, both due to the lack of available data and potentially infinite diversity of phonetic representations encountered in practice.
<figure><img src="image_55.png" /><figcaption>Figure 7.3: Phonetic alignment produced on a newly recorded voice with ADAGIO overfitting.</figcaption></figure>

For these reasons, an *overfitting* approach was chosen. The training of ADAGIO does not aim at achieving a generic acoustic model (e.g., capable of generalization) but at specializing on the target database. Inference, indeed, is made on the same data as during training. The predicted posteriorgrams are then used to predict a forced phonetic alignment.

While everything is usually done to prevent a system from overfitting, temporal information has been gained thanks to this procedure: phonetic sequences, which were not aligned, now are associated with timestamps.

## Results

The relevance of this procedure was confirmed in the evaluations conducted in section 6.4 for the Chanter RT dataset — which is the default sound corpus used in ISiS (ARDAILLON et al., 2016). Given all of these considerations, a new voice for ISiS has been recorded and aligned by ADAGIO. A result is shown in Figure 7.3. Recording of other singers are on the way. Again, one of the key potential offered by ADAGIO through such usage — intrinsically due to the generic framework of the Connectionist Temporal Classification (CTC) —, is the ability to align any phonetic alphabet in the future, thus extending ISiS to other languages.

Audio examples of voice synthesis with this procedure are accessible online<fnref n="2" />.

## Section summary – Concatenative singing synthesis

Ircam Singing Synthesis (ISiS) relies on a concatenative strategy to generate sung utterances from pre-aligned dataset at the phoneme level. ADAGIO has been successively applied to align a new French voice thanks to an *overfitting* strategy. This method was validated in previous evaluations for a reference dataset for ISiS with gold ground truth. In the future, ADAGIO will allow the integration of potentially any (phonetic) alphabet in this system and thus, notably, its extension and usage beyond French.

<footnote n="2">²https://www.youtube.com/watch?v=U1VZ9M&si=cm2w1qmvC3hjP5ti&t=2128 (35:27)</footnote>
## 7.2 Temporal production strategies of vocal attitudes

Depending on the outcome humans expect from an interaction and the nature of the relationship between the individuals – e.g., strangers meeting for the first time, longstanding friends, hierarchy in job interview, etc. –, voice and its expressivity are modulated to convey different social attitudes. The temporality involved in these communicative strategies has not been a core focus of previous research. Benefiting from phonetic alignment, this collaborative work has precisely aimed at uncovering some of the mechanisms at the heart of such expressive speech based on these temporal aspects. This section showcases a summary of associated findings – further details can be found in the complete publication (SALAIS et al., 2022).

### Context

Human interactions rely on communicating social attitudes to other individuals (MCALEER et al., 2014). These attitudes, that are distinct from emotions, indicate an individual's social intentions such as being friendly or dominant (WICHMANN, 2000). Despite their fundamental role in spoken interactions, the way these attitudes are conveyed through vocal communication has been barely explored. This collaborative study, therefore, has aimed at developing an acoustic evaluation based on anatomic considerations derived from phoneme-to-audio alignment, to uncover the production strategies at play when speakers communicate social attitudes.

To do this, an analysis procedure has been conducted on a 20-speaker subset of the Att-HACK dataset (LE MOINE and OBIN, 2020), introduced in section 6.1, which features acted expressive speech in four attitudes – dominant, friendly, seductive and distant. The texts were converted into phoneme sequences with the phonemizer of BERNARD and TITEUX (2021).

### Alignment strategy

In order to investigate both articulatory and phonetic information in these vocal strategies, speech signals needed to be segmented so that each phonetic region was clearly identifiable.

In this specific use case, one is not interested in developing a system for inference on new, unseen data, but rather to fully align a fixed dataset of paired audios and unaligned phonetic transcripts. This is possible, once again, with the CTC framework in an *overfitting* scenario – instead of the common seek for *generalization* properties. Therefore, the acoustic model is trained to overfit on the entire Att-HACK subset, and forced phonetic alignments are derived from the posteriorgrams.

### Exploitations

The temporal alignment information was then leveled to study both articulatory strategies and phonetic structure.
<figure><img src="image_56.png" /><figcaption>Figure 7.4: Computation and analysis of vocal tract actuation and phonetic structure extracted from voice temporal alignments for friendly, dominant, distant and seductive vocal attitudes. ‘★’: statistically significant difference (p<0.05). Error bars represent 95% confidence intervals on the mean.</figcaption></figure>

Regarding the *articulatory strategies* — as switching between articulatory modes corresponds to transitions between formants (PISANSKI et al., 2022), the Vowel Space Density (VSD) was examined as proposed by STORY and BUNTON (2017). It represents the space formed by the two formants $F_1$ and $F_2$ for all vowels, whose time frames can be known thanks to the alignment, and by only keeping the highest density regions per attitude.

Regarding the *phonetic structure* — revealing the speech prosodic stress and rhythm —, the phonetic timestamps were used to compute (1) the mean phoneme duration, (2) the Speech Rate (SR) as the mean numbers of phoneme per second in each utterance, (3) the Rhythmic Irregularity Measure (RIM) as the mean duration difference between all segments in a sentence, and (4) the Rhythm Ratio (RR) as the mean duration difference between contiguous speech segments. For each attitude, the mean value of these descriptors (over all sentences) was computed upon speaker normalization and zero-centering.

## Results

The obtained results are depicted in Figure 7.4.

In terms of *vocal tract articulation*, the Vowel Space Density shows that Distance has a significantly lower $F_1$ in comparison to Dominance that has a high $F_1$. This suggests that speakers wanting to establish distance from their audience *shorten* their vocal tract using mostly *closed* mouth, as if being understood by other(s) was not important. Conversely, people desired to be dominant rather *elongate* their vocal tract and *open* their mouth, as a necessity to be understood. More broadly, clusters for each attitude — albeit associated with mostly marginal differences — are identifiable, indeed pointing to an articulatory dimension in speech intention communication, which is a novel perspective.
In terms of *phonetic structure*, significant inter-attitude variations are found for the Speech Rate (SR) and the Rhythmic Irregularity Measure (RIM) but not for Rhythm Ratio (RR). This indicates that attitudes impact *global* rhythmic communication schemes but not *local* ones.

In view of these results, it appears that humans express Seductiveness with a specific temporality, clearly distinct from the other three attitudes studied. As a result, a prototype of “seductiveness” featuring these *temporal* strategies can be highlighted. Specifically, seductive utterances are significantly associated with small speech rate (in line with longer vowel duration) and high RIM, such that they are produced slowly and with irregular rhythm, as if speakers were purposely taking the time to express themselves and hint at their intentions.

## Section summary – Temporal production strategies of vocal attitudes

Phonetic alignment has permitted a deep look into the temporal aspects involved when communicating social intentions. It has been shown that French speakers use shared production strategies to express vocal attitudes such as friendliness, dominance, distance or seductiveness. From a phonetic structure perspective, the later attitude (*seductiveness*) is significantly different from the others as associated with slower speech rate and more irregular rhythmics. Besides, and to the best of the authors’ knowledge, this study has revealed for the first time that social attitudes are also conveyed via articulatory modes. Insights from significant observed effects are that people might shorten their vocal tract to impose *distance* with their interlocutor and, conversely, elongate it to be *dominant*.

## 7.3 A musicological pipeline for singing style analysis

In the continuation of the analysis of voice expressivity, this section focuses on the production strategies at play in singing performances, referred to as *singing voice style*.

The study of singing voice style is of great interest both for expressive vocal synthesis and for the musicological analysis of vocal performances and incites to a fruitful convergence between signal processing and musicology. A previous research in that direction (ARDAILLON et al., 2016) has precisely been integrated in Ircam Singing Synthesis (ISiS) for expressive control of voice rendering, see section 7.1, and might call for more findings. However, for musicologists, these studies often come up against the absence of semi-automatic analysis tools for voices recorded in a musical context, imposing long and tedious manual annotation work.

This collaboration introduces a complete processing chain in support of musicological analysis, notably providing musicologists with powerful tools for the automatic analysis of singing voices. ADAGIO is a core element of this pipeline, as it is in charge of producing both voice-to-*syllable* and voice-to-*notes* alignments necessary for the fine-grained exploration of artistic choices made by a singer during their performances. This section is based on the collaborative publication (TEYTAUT et al., 2023) and was made possible by the **ARS** project (https://ars.ircam.fr).
### 7.3.1 Musicological context

The singing voice, as a vector of communication, appropriates many of the codes of the spoken voice to convey emotions or social attitudes (LACHERET-DUJOUR and BEAUGENDRE, 1999). As a result, in the 2010s, some musicologists initiated research in vocal performance analysis (CHABOT-CANET, 2008; LACASSE, 2010) based on, e.g., paralinguistics, phonostylistics or psycholinguistics (FÓNAGY, 1983; LÉON, 1993; POYATOS, 2002).

Singers can exploit a rich *palette* of singing techniques to shape their interpretations. These techniques can either relate to *general* musical conventions or “trends” – e.g., belting in musicals (HENRICH BERNARDONI, 2020) or singing format in classical singing (SUNDBERG, 1974) – or rather a *personal* style via particular vibrato, timbre, phrasing, intonation and more.

The very essence of musical interpretation implies that there may possibly be a large, if not infinite, number of performances corresponding to one piece of music or song. However, an artistic identity necessarily means a certain amount of coherence between the performances – which is ultimately and fortunately fixed, to a certain extent, in some reference recording. Such recordings allow to study, in the most neutral, objective and reproducible way possible, the production strategies and the *palette* of effects deliberately chosen by the singer to be “engraved” in the studio – i.e., the singer’s very own *singing voice style* (CHABOT-CANET, 2020a).

For years, musicologists interested in singing performance were facing the absence of tools for automatic acoustic analyses of voices recorded in a musical context. Consequently, they had to manually transcribe and synchronize the audio by ear or via visual spectral representations. While expert listening of musicologists remains essential for data supervision and analyses, a purely manual approach has its own limits. It is very time-consuming, imposing one to limit oneself to small corpora, can be error-prone and, in some cases, overly subjective. Thus, relying on automatic systems for transcription (GAO et al., 2022) and alignment (DORAS et al., 2023) may lead to a considerable gain in time, and help in setting a common base for the musicological community.

This is the context in which a new musicological pipeline is introduced, with the aim to highly simplify tedious steps previously carried out by hand and opens new perspectives through automation.

### 7.3.2 Pipeline introduction

This section presents the proposed pipeline for the musicological analysis of singing voice style. As shown in Figure 7.5, it is composed of four main categories, with independent modules (hence a flexible workflow) that are connected, namely *voice characterization*, *musicological expertise*, *voice alignment*, and *musicological exploitations*, that are further detailed.
<figure><img src="image_57.png" /><figcaption>Figure 7.5: Overview of the complete analysis pipeline involving musicological expertise, deep learning models for the automation of voice characterization and alignment in order to help musicologists studying singing voice style.</figcaption></figure>

## Voice characterization

(This research has been conducted by co-author Axel ROEBEL. It is detailed as the voice separator was used to estimate the vocals supervising the temporal constraints in section 5.2.)

The core of any singing or singer analysis system is the voice itself. As musicologists are interested in commercial recordings, the background music must be removed as it is not compatible with parameter estimation algorithms and can hinder a precise description of the singing intonation. The first step, therefore, is to isolate the vocals. To do so, CHOI et al. (2019)'s model was re-implemented as it achieves state-of-the-art singing voice extraction quality with a comparatively small number of parameters. This model, denoted $M_V$ in Figure 7.5, has been trained using the publicly available MUSDB18 (RAFII et al., 2017) and CCMixter (LIUTKUS et al., 2015) datasets, and a collection of internal data featuring solo singing voices and instrumental music with notably instruments not well covered in the public datasets. During training, the voice and music samples were randomly mixed and pitch-shifted following (COHEN-HADRIA et al., 2019; LANCASTER and SOUVIRAÀ-LABASTIE, 2020). In line with CHOI et al.'s results, extracted vocals are of very satisfying quality — i.e., SDR metric (VINCENT et al., 2006) of 9.2dB for the vocals separated from the HQ-MUSDB test set. In inference, separation is faster than real-time even when running on a CPU on a small laptop.
The isolated vocals are then used to extract the F0 with another neural network, denoted $M_{F0}$ in Figure 7.5 and detailed in (ARDAILLON and ROEBEL, 2019).

## Musicological expert knowledge

A typical objective is to correlate the voice features with other relevant information and emphasize their relationships via temporal alignment. In the context of singing performances, both lyrics and singing melody (notes) are of high interest. Musicologists have the role to collect such symbolic data and adapt them if necessary to match their research angle.

Regarding the *lyrics* — lyrics are easy to gather for most commercial music today, either online or via an album booklet, so that manual transcription is rarely required. In practice, a succession of *syllables* is more musicologically relevant than *words* for the text as singing notes are held on the vowels of each syllable (SUNDBERG, 1987). Plus, one can note that the lyrics rarely exactly match the singing content due to additive onomatopoeia (e.g., “yeah”, “hm”) or unpronounced utterances. Fortunately, it was shown in section 6.4.3 that ADAGIO can handle a decent amount of such irregularities and aligns despite missing or additional syllables. The role of the musicologists in lyrics retrieval is thus twofold: (1) to ensure that the text is coherent and correctly written; and (2) to explicitly adapt, whenever necessary, repeated syllables, missing entries, or onomatopoeia judged pertinent, i.e., conveying meaningful interpretative aspects.

Regarding the *melody* — one is looking for the note sequences performed by the singers (as in, e.g., a music score). When the music score is not accessible, transcriptions remain often done by musicologists — this way they can annotate any precise gesture made by the artist.

## Voice-to-symbols synchronization

This is the core integration of this thesis into this collaboration since ADAGIO is used to temporally align both syllables and notes.

First, voice-to-syllable alignment is tackled with ADAGIO trained for the acoustic modeling of words — it is denoted $M_{V2T}$ in Figure 7.5. Albeit trained for word-level alignment, syllables can be synchronized as well. Indeed, as words and syllables share the same alphabet, the acoustic modeling does not require any adaptation. The only difference lies in the decoding step, as there are more spaces (label $\emptyset$) to synchronize between syllables than between words.

Second, voice-to-note alignment is tackled with ADAGIO trained for the acoustic modeling of notes — it is denoted $M_{V2N}$ in Figure 7.5. A major novel perspective offered by this dual approach is the capability to fully align melisma, i.e., multiple notes held on the same syllable (shown in musical notation by slurs). Musicologists can thus concretely rely on note-level alignment to complement the syllable-level alignment in such case as portrayed in Figure 7.6. To the best of the author’s knowledge, it is the first option proposed to musicologists for dealing with the automatic analysis of melisma.
<figure><img src="image_58.png" /><figcaption>Figure 7.6: Automatic analysis of a melisma: syllable-level alignment only predicts the word “play” on the full duration of this excerpt, without taking pitch variation into account — note-level alignment allows a deeper look into this multi-modal gesture. Score transcription done by Antoine PETIT. See Taylor SWIFT case study — section 7.3.3.</figcaption></figure>

## Musicological exploitations

Finally, further musicological studies can be carried out benefiting from the flexibility of the proposed pipeline. The alignment time markers can be read into visualization software (e.g., Sonic Visualiser<fnref n="3" /> or RX<fnref n="4" />) for unavoidable manual corrections — being still much less tedious than starting from scratch. All in all, these timestamps are available for qualitative and/or computational analyses. In the rest of this section, a demonstration of the whole musicological protocol is proposed via a case study on Taylor SWIFT. The temporal data from the alignments have allowed performing fine-grained rhythmic analyses and to investigate the structural role played by articulation and micro-rhythm. This is interestingly in line with the previous speech studies (section 7.2) proving interconnections between all voice research angles.

Note that the publication (TEYTAUT et al., 2023) also features a case study on Charles AZNAVOUR, made by co-author Céline CHABOT-CANET, which is focused on vocal phrasing and rhetorical effects involving intonation. However, both for the sake of conciseness and as it mostly exploited the F0 estimation, with textual alignment being only used as a qualitative support to instantaneously relate the effects heard and spectral observations, as shown in Figure 7.7, this analysis is voluntary left out of the current manuscript.

<footnote n="3">³https://www.sonicvisualiser.org</footnote>

<footnote n="4">⁴https://www.izotope.com/en/products/rx.html</footnote>
<figure><img src="image_59.png" /><figcaption>Figure 7.7: Verse 1, line 1 from Charles AZNAVOUR’s “La Bohème”. Score transcription, sonogram, F0 and text alignment. ‘a’: fast flow, intonational instability, and absence of vibrato; ‘b’: sustained note with vibrato on the last syllable of the phrase.</figcaption></figure>

### 7.3.3 Case study – Taylor SWIFT’s “Blank Space”

(This research has been conducted by co-author Antoine PETIT. It is detailed as it demonstrates strong exploitations of alignment time markers and their evaluations via manual corrections.)

This study examines Taylor SWIFT’s 2014 hit single “Blank Space”<fnref n="5" /> which was selected because (1) at the time of writing, it is SWIFT’s second-best charting song and its analysis may uncover what makes a successful song; (2) SWIFT’s songs have not yet been the object of much, if any, musicological attention with the recent exception of (PETIT, 2022); and (3) beyond a standard form, its organization reveals lower-level patterns in verses and chorus linked to vocal delivery. Overall, it presents a prime example of analytical work afforded by the pipeline.

The musicologist has focused on the first half of the first verse, *i.e.*, eight bars from 5:30 to 25:30, a score transcription of which is given in Figure 7.8. It exhibits a **srdc**<fnref n="6" /> structure (EVERETT, 2009) and can thus be divided into four parts – two fairly similar segments (**sr**) and a contrasting passage leading to a concluding gesture (**dc**). Following HANNINEN (2012), one may wish to understand which criteria elicit such a quite self-evident segmentation. The claim of this study is that articulation and micro-rhythm play a key role in shaping the form of the excerpt. This can be investigated with the proposed pipeline and its temporal alignments.

#### *Application of the pipeline*

Syllables retrieved from a first score transcription were synchronized with the audio with ADAGIO. For each syllable not followed by a rest, the P-center (*perceptual* center, *i.e.*, the moment *heard* as beginning) is approximated by the mean between the start and end times of the syllable. These were then manually checked.

<footnote n="5">⁵Words and music by Taylor SWIFT, Max MARTIN and SHELLBACK. Reference recording: 1989, Big Machine, 2014.</footnote>

<footnote n="6">⁶Statement, *restatement/response*, *departure*, *closure*.</footnote>
<figure><img src="image_60.png" /><figcaption>Figure 7.8: Taylor SWIFT, “Blank Space”, first verse, measures 3–10. Transcription done by Antoine PETIT.</figcaption></figure>

Next, pitches were transcribed and refined thanks to the visualization of both F0 curves and syllables alignment. The resulting MIDI notes were synchronized to the audio with ADAGIO afterwards. Only the notes associated with the five identified melisma (four in measure 4 and one on “*play*?” in measures 9-10) are kept and their time markers were also manually corrected.

One can then compare the time differences between the corrected markers and the predicted markers leading to an Average Absolute Error (AAE) from a musicological expertise. This is reported on Table 7.1. Interestingly, one can note that the manual syllable corrections are below the theoretical precision $\delta t = 16$ms of ADAGIO in its original setup (which is half of the frame size and given the signal processing setup from Table 4.3). Once again, note alignment appears as less stable but remains acceptable and exploitable.

<table><caption>Table 7.1: Manual alignment corrections for the “Blank Space” case study.</caption><thead><tr><th>Alignments</th><th>Syllables (all)</th><th>Notes (on melisma)</th></tr></thead><tbody><tr><td>“Blank Space”<br>(manual correction)<br>∠ AAE (ms)</td><td>13.5 ± 40.9</td><td>22.5 ± 63.0</td></tr></tbody></table>
<table><caption>Table 7.2: Non-legato notes by subsection in Taylor SWIFT’s “Blank Space”, measures 3–10.</caption><thead><tr><th></th><th>s</th><th>r</th><th>d</th><th>c</th></tr></thead><tbody><tr><td>Non-<i>legato</i> notes</td><td></td><td></td><td></td><td></td></tr><tr><td>Number</td><td>4</td><td>6</td><td>10</td><td>2</td></tr><tr><td>Proportion (%)</td><td>20</td><td>46</td><td>59</td><td>22</td></tr><tr><td>Mean duration (ms)</td><td>213</td><td>162</td><td>161</td><td>180</td></tr><tr><td>std. dev. (ms)</td><td>(14.1)</td><td>(71.5)</td><td>(84.8)</td><td>(175)</td></tr></tbody></table>

## Musicological analyses

This timing data at hand — onset, P-center and end of every note in the excerpt — allow analyzing how Taylor SWIFT uses articulation and micro-rhythm to structure her vocals.

Legato articulation, or lack thereof, can be computed by subtracting the onset time of the $n+1^{th}$ note with the end time of the $n^{th}$ note. The Table 7.2 displays the number, proportion, and mean duration of non-legato notes by subsection, painting a vivid picture of form organized through articulation. The excerpt begins with mostly legato singing, interspersed with a few very homogeneous silences. SWIFT’s vocals then gradually become more jagged — mostly non-legato, with many overall shorter, but also much more heterogeneous, silences — before returning to the initial legato articulation in the concluding melisma, which is split in two by the longest silence in the excerpt. This arch-like progression seems fairly obvious upon listening (especially when it has been explicitly pointed out beforehand), but may has been missed without the ability to gather accurate timing data.

But articulation is only part of the story. Figure 7.9 maps the duration of every note (i.e., difference between two successive P-centers to which is subtracted the length of the intervening silence, if any) to its P-center. Legato articulation is shown with connecting lines, the internal notes of each melisma (aligned with ADAGIO) is shown as unfilled dots. The vertical dashed lines correspond to the beginning of the four subsections, and the horizontal ones to the projected duration of eighth, sixteenth, and thirty-second notes (i.e., the three most frequent symbolic durations in the transcription) at 96BPM.

Not all notes last for their projected duration. In particular, many sixteenth notes appear “uneven”, with the on-beat one being longer than the off-beat one. Long/Short subdivision (CAPORALETTI, 2014) (i.e., swing) is endemic to *s* and *r*, where it affects almost all sixteenth notes, but is absent in *d*, which prioritizes straight eighth notes.

Thus, there exists a subtle interplay between two contrasting local vocal styles: (1) mostly legato, with L/S (swung) sixteenth notes and step-wise motion (*sr* and *c*); and (2) mostly non-legato, with straight rhythms and large leaps filled with gliding intonations (*d*). The more jagged articulation of *r* allows SWIFT to smoothly transition from style (1) to style (2), while the leap of a major sixth coupled with L/S sixteenth notes on “wanna” at the beginning of *c* enables the reverse.
<figure><img src="image_61.png" /><figcaption>Figure 7.9: Note durations (log-scale) in Taylor SWIFT’s “Blank Space”, measures 3–10.</figcaption></figure>

<table><caption>Table 7.3: Micro-rhythmic displacements by subsection in Taylor SWIFT’s “Blank Space”, measures 3–10.</caption><thead><tr><th></th><th>s</th><th>r</th><th>d</th><th>c</th></tr></thead><tbody><tr><td>Mean displacement (ms)</td><td>32.6</td><td>38.6</td><td>27.2</td><td>30.5</td></tr><tr><td>std. dev. (ms)</td><td>(38.4)</td><td>(27.7)</td><td>(9.82)</td><td>(29.8)</td></tr></tbody></table>

These local styles also share a number of characteristics, among which propulsive tendencies (CAPORALETTI, 2014) in melismatic passages (i.e., the notes are shorter than projected)<fnref n="7" /> and lengthened notes when followed by a silence (both of these can be observed on Figure 7.9), a lack of vibrato (this can be observed on the automatically-extracted F0), and micro-rhythmic displacements of most notes (computed by subtracting their P-center with their projected onset at 96BPM), which consistently appear about 30ms later than projected, as shown in Table 7.3.

Thanks to the pipeline streamlining the annotation process, this analysis can easily be expanded upon, so that it encompasses the whole verse/verse-chorus unit/song, etc., up to (at least) the level of the album — thus shedding light on SWIFT’s multifaceted vocal style. The many strategies discovered during the analysis can then be compared with other artists’.

<footnote n="7">⁷Because such propulsive tendencies are independent from the underlying pulse, which does *not* change, the last note of a melisma must last longer than projected, as “compensation”.</footnote>
## Section summary – Musicological pipeline for singing style analysis

A complete pipeline for the musicological analysis of singing voice style has been proposed. It notably exploits deep learning models for singing voice extraction from background music, voice parameter (F0) estimation and — as far as this thesis is concerned — ADAGIO for robust automatic alignment of both syllables *and* notes to the audio. Not only does this pipeline greatly simplify the tedious tasks traditionally done manually by musicologists, but it also offers practical flexibility, as demonstrated in two concrete musicological studies: (1) text and note alignments allowed investigating articulation and micro-rhythm in a Taylor SWIFT song (American pop, 2014); and (2) although not detailed in this manuscript, F0 curves and vowel regions were exploited to highlight vocal phrasing and rhetorical effects involving intonation in a Charles AZNAVOUR song (French chanson, 1966). More generally, this work is meant for future and strong collaborations between musicologists and deep learning researchers sharing a common interest in singing voice. The tools presented will be made available to the community in a web interface hosted at https://passagesxx-xxi.univ-lyon2.fr/activites/projets-anr/projet-ars-analyse-et-transformation-du-style-de-chant-1.

## 7.4 Perspectives

This thesis paves the way for further investigations, concrete applications and collaborations. This section thus exposes some ongoing projects that cannot be considered as achieved at the time of writing but that are in line with the main scope of this thesis, as well as burgeoning ideas for future research.

### Automatic corpora segmentation

The alignment of very long audio recordings has been a well-known limitation to many aligners (BORDEL et al., 2015; KATSAMANIS et al., 2011). This is due to classical DTW- or CTC-based decoding modules $D_\pi$, which compute the forced alignment *per se*, as they require the computation and storage of an alignment path that scales *quadratically* in number of frames and symbols to synchronize (cf. section 2.3.1). This quickly becomes prohibitive for long audio and symbolic sequences — exceeding the memory storage available on modern machines.

This thesis was involved in a project on this issue (DORAS et al., 2023) in which ADAGIO was the acoustic model $M_\Theta$ used in experiments and was revealed for the first time. However, as the main contribution of the paper was the design a new decoding module $D_\pi$ supporting very long sequences, and was implemented by the paper’s first author, this section is voluntarily kept short. The core idea was to adapt the CTC decoding with an exact reformulation of the DTW algorithm, which scales linearly in memory, introduced by TRALIE and DEMPSEY (2020).
This aligner was shown to be capable of synchronizing several hours of audio and text even for different languages and despite some errors in transcripts (as the evaluations presented in section 6.4.3). In the future, it is planned to use this system for the automatic segmentation of large corpora like audiobooks into small excerpts of paired audio and labels for enriching and completing existing datasets for downstream tasks, such as voice recognition, conversion or synthesis (KÜRZINGER et al., 2020; ROEBEL and BOUS, 2022).

*Joint singing voice alignment and separation*

An open research problem is to investigate to what extent both separation and alignment can be addressed simultaneously. These tasks are traditionally studied independently, but recent research shed light on their potential complementarity.

Indeed, STOLLER et al. used the same neural architecture to perform each task individually (STOLLER et al., 2018, 2019); SCHULZE-FORSTER et al. (2021) derived alignment from an attention mechanism informing voice separation; MESEGUER-BROCAL and PEETERS (2020) exploited pre-aligned phonemes to help vocal extraction; VAGLIO et al. (2020b) built their acoustic model on pre-isolated vocals; and this thesis improved voice alignment thanks to an auxiliary constraint of spectral reconstruction of estimated vocals.

These results tend to indicate that alignment and separation are beneficial to each other. Yet, to the best of the author's knowledge, no approach has tried a joint training procedure so far. One potentially interesting direction to start with would be the work of CHOI et al. (2019) that proposed a relatively light U-Net architecture for singing voice extraction in the spectral domain. (It was the voice separator used for the temporal constraints in section 5.2 and the musicological pipeline from section 7.3.) The continuation of this work (CHOI et al., 2021) introduced a signal-based conditioning that might be adapted for alignment data.

*Note alignment for voice and music*

Further experiments are required to enhance and complete the study on the synchronization between notes and singing voice as already mentioned in section 6.4.2. The author believes there is way for improvement in terms of alignment accuracy and, also, in exploitation of the posteriorgram for singing note transcription based on *empirical* observations of the saliency maps – see Figure 6.4. In a similar vein, the problem of note alignment can go beyond voice signals such that the alignment of monophonic instruments could probably be handled with a CTC-based aligner like ADAGIO. To go even further, one could tackle the simultaneous alignment of multiple notes for polyphonic instruments by taking inspiration, e.g., from the work of WEISS and PEETERS (2021) who introduced a multi-label CTC for polyphonic *transcription*. A final objective would be to integrate such CTC-based acoustic modeling into a real-time audio-to-score alignment (LAJUGIE et al., 2016) for score following applications (CONT, 2011).
Improving the acoustic modeling & transcription

This work, as commonly done in the alignment literature, has extensively relied on acoustic modeling approaches originally thought for voice transcription to decode and synchronize symbols to audio from some hidden representations. As a result, it is reasonable to assume that better acoustic modeling will lead to performance improvements. Two options in that direction come to mind. Either to keep a flexible and general end-to-end architecture and informing it with other types of information e.g., pitch detection as recently proposed by HUANG et al. (2022), or to add additional modules, like language models (YCART et al., 2019), to complement the acoustic model in the decoding step — this is the strategy supported by GUPTA et al. (2020). It is worth mentioning that this thesis was concerned with the former strategy, i.e., additional temporal constraints. The later strategy would contradict the thesis initial objective, which was to be able to align without domain knowledge. More generally, there is an aim for universality such that a theoretically optimal system could proceed to both audio/voice transcription and alignment — even without requiring the ground-truth texts (ZHU et al., 2022).

Section summary – Perspectives

There are many upcoming investigations in the continuation of this work as voice
alignment algorithms can (1) serve to automatically segment large corpora; (2) be coupled
with the voice separation task; (3) be improved and extended for notes and melody
synchronization; and (4) keep motivating the search of an “universal” transcription-
alignment system.

7.5 Applications and collaborations in a nutshell

Chapter summary – Applications and collaborations

In this chapter, the practical applications of my thesis were presented by means of
conducted collaborations. First, I investigated singing voice synthesis in a concatenative
strategy for which ADAGIO was trained to voluntary specialize (overfit) on a new
recorded dataset and produce phonetic alignments. Then, I detailed collaborative studies
on voice expressivity focused on production strategies of both spoken social attitudes and
singing style in musical performances. The former (speech) revealed for the first time that
social attitudes are also conveyed through articulation. The later (singing) introduced a
complete pipeline dedicated to the analysis of singing style with neural voice processing
and alignment of both syllables and notes, that has already been exploited by musicology
partners. Lastly, I mentioned recent ongoing works and perspectives for future research
– notably one following a project for the alignment of very long audio recordings.
Chapter 8

Conclusion

“Never is an awfully long *Time*.”

Peter Pan

— James Matthew BARRIE

<figure><img src="image_62.png" /></figure>

*Time*.

An essential, and yet so perplexing, aspect of the universe and human daily life. Many
physical laws are governed by time so that it determines the evolution of various observations
and surrounding signals. Audio data are no exception: music and voice are fundamentally
non-stationary signals whose temporal study can reveal many properties.

Humans can interact with one another through messages that they can commonly express with their voices, as *orality* is a primary modality of human communication. These vocal locutions are very often associated with underlying representations of *symbolic* nature, which translate another modality of communication, typically depending on a textual language (e.g., stories, tales, poems, legends, etc.) and/or music theory (e.g., songs, opera, musical scores).

In the context of such communicative approaches, the notion of temporality is of utmost
importance and has greatly motivated research studies dedicated to its analysis. This thesis
has precisely proposed to automatically tackle the problem of temporal alignment between voice
signals and their related symbolic sequences, to uncover some key temporal aspects involved in
human communication. In doing so, it was aimed to bridge, to some extent, the semantic gap
between diverse communication modalities.
## 8.1 Manuscript summary

The content of this thesis is briefly summarized in this section.

(Chapter 2.) First and foremost, I exposed the overall context of my work: to better understand some expressive strategies in human communication by revealing the temporal relationships between two voice modalities – oral and symbolic. To characterize the orality, I introduced signal processing basics allowing to highlight relevant voice features capturing the temporal evolution of the spectral content. The representation of symbolic sequences was highly correlated with the history of writing of both spoken languages and music theory. At the heart of the interaction between voice signals and symbols, lie two prominent analysis tasks: transcription and alignment. My thesis has been primarily interested in temporal voice alignment, or synchronization, which aims at associating each symbol with a timestamp of appearance in the audio recording, and offers thrilling applications for the general public (automatic closed-captioning, karaoke) and research community (voice analysis and synthesis).

(Chapter 3.) Through a systemic review of the literature, I showed that any system for temporal voice alignment is composed of (1) an acoustic model, inferring a time-symbol representation from the audio, and (2) a decoding module, using this representation for the forced alignment of a ground-truth sequence. While the decoding module is typically based on a reference dynamic programming algorithm, the acoustic model has seen many implementations. Inspired by recent advances in deep learning for voice processing, I motivated the choice to design the acoustic model as a deep neural network. In particular, I presented the Connectionist Temporal Classification (CTC) framework, which is particularly convenient and appealing for end-to-end voice alignment and was the chosen approach for my thesis.

(Chapter 4.) Then, I presented my preliminary proposals of CTC-based aligners which allowed me to develop a series of criteria that a robust and practical aligner should respect. These criteria were not systematically validated either by existing systems or by my first attempts, which prompted the development of a new acoustic model in accordance with these identified needs. This is ADAGIO – for Automatic Deep Alignment of Voice –, the central node of my thesis research. Trained on the CTC criterion, ADAGIO predicts, from an audio, a posterior program estimating the temporal evolution of the underlying symbolic content.

(Chapter 5.) However, since the CTC is originally a transcription measure, it can be minimized without aligning as precisely as desired. Faced with this intrinsic limitation, I proposed to introduce additional temporal constraints when training the networks to ensure the emergence of alignment properties directly in the posterior programs. This temporal knowledge took the form of ancillary tasks of spectral reconstruction, temporal structure propagation, and time-sequence monotonicity. This defined a version of ADAGIO enhanced with temporal information learning. It is worth highlighting that these enhancements were purposely thought generic, i.e. not based on any domain/expert knowledge (e.g., specific language or music genre).
(Chapter 6.) Objective evaluations based on classical metrics were conducted to quantify, and ultimately confirm, the suitability of ADAGIO for voice-to-text and (to a lesser extent) voice-to-note alignments, and the beneficial impacts of my auxiliary temporal constraints when supervised on isolated voices. Two word-level aligned datasets were derived from existing works: *Philos 10* for speech and *Playlist 50* for lyrics and notes. Although state of the art is not reached in the classical evaluation case, ADAGIO stands out for its simplicity and diversity of use.

(Chapter 7.) Finally, ADAGIO has been put into practice through various collaborations in which I performed the alignments and analyses were dominantly done by respective end users. These works have focused on the concatenative synthesis of singing voice via phonetic alignment, and the study of expressive production strategies for both spoken social attitudes and singing style in musical performances – bridging, to some extent, the semantic gap between symbols and audio. Lastly, ongoing research and perspectives were mentioned.

## 8.2 Coming full circle: back to initial questions

In the course of this thesis, I have *deeply learned* to investigate and answer to practical and research questions – and notably the ones raised by temporal voice alignment in section 1.1. To give a greater overview to this research, and essentially conclude this manuscript, this last section sums up the answers I propose to these initial questions from accumulated knowledge.

**Question 1 (Q1).** *Temporal voice alignment – what?*

* *What* is the temporal alignment task in general and for voice?

A task consisting in predicting a precise timestamping for each event reported in a sequence of ordered elements expressed in an audio recording. This thesis specializes on voice alignment, which links oral and symbolic modalities of human communication.

* *What* kind of representations can be used to align voice data?

Time-Frequency Representations (TFR) are relevant features to manipulate voice signals as they capture the variations of the spectral content over time. Symbolic sequences are defined through an associated finite alphabet of labels (*e.g.*, letters, digits, notes, etc.).

**Question 2 (Q2).** *Temporal voice alignment – how?*

* *How* to develop a system for the temporal alignment of voice?

Most voice aligners in the literature are composed of an acoustic model and a decoding module. The decoding module often relies on a Dynamic Time Warping (DTW) algorithm. Recent acoustic models exploit deep neural networks learning robust features from audio. In this thesis, an acoustic model, ADAGIO, was proposed as an end-to-end, convolutional network trained in the highly flexible Connectionist Temporal Classifical (CTC) framework.
* *How* to extract suitable temporal information from audio to reinforce alignment accuracy?

The main claim of this research is that incorporating additional temporal information in the acoustic modeling must result in higher alignment accuracy. This appeared as a natural extension for ADAGIO, as the flexibility offered by the CTC came with no intrinsic guarantee of precise alignments. In the end, temporal spectral reconstruction, temporal structure propagation, and audio-symbolic monotonicity were introduced as new supplementary and general (i.e. without domain knowledge) objectives that were integrated in the training phase. These constraints, when supervised on solo vocals estimated with voice separation, were shown to have a beneficial impact on voice alignment.

**Question 3 (Q3).** *Temporal voice alignment – why?*

* *Why* is temporal alignment of interest in various research communities?

General alignment algorithms emerged in many fields requiring similarity measures between
sequences such as telecommunications, bioinformatics and the audio domain. The specific
case of voice synchronization appears as a means to bridge, to some extent, the semantic
gap between different modalities involved in human communication by uncovering and
looking closely at the temporal relationships between these diverse representations. Voice
aligners have already found well-known mainstream applications for the general public
such as text-based audio indexing, automatic closed captioning, or karaoke generation.

* *Why* does temporal voice alignment lead to numerous research applications?

Beyond the above-mentioned use cases, temporal voice alignment is of great interest
in research dedicated to voice analysis and synthesis. Concatenative singing synthesis,
indeed, exploits aligned phonetic content to generate sung utterances. The automatic
segmentation of audiobooks, often needed by speech specialists, is possible via long audio
alignment. Synchronizing voice data also allows fine-grained analysis of voice expressivity
and thus a better understanding of, e.g., the production strategies at play to convey social
attitudes or singing interpretative style according to musicological studies.

Naturally, one could wonder whether asking such questions in the first place was of interest.
In the very words of Luciano BERIO (2006), “I think that the search for a universal answer to
the questions raised by musical experience will never be completely fulfilled; but we know that a
question raised is often more significant than the answer received. Only a reckless spirit, today,
would try to give a total explanation of music, but anyone who would never pose the problem
is even more reckless.” Perhaps the very essence of questioning music and voice, as essential
expressive and communicative vectors, would be a springboard to better apprehend *time* itself.

And even beyond music, voice and time — the very act of raising questions, of all kinds and
at all levels, seems more essential than ever, today, to stimulate the minds and the intellect.
So many more questions have to be raised to, hopefully, face the challenges of the century such
as global warming, modern wars and human condition — this sure will require some *time*.
END OF THE THESIS

"*Time* isn't the main thing. It's the only thing."

Miles DAVIS
List of Figures

<table>
  <tbody>
    <tr>
      <td>1.1</td>
      <td>Time representations in arts.</td>
      <td>15</td>
    </tr>
    <tr>
      <td>1.2</td>
      <td>Outline of the dissertation and its relationships with the practical and research questions raised by temporal voice alignment.</td>
      <td>20</td>
    </tr>
    <tr>
      <td>2.1</td>
      <td>SHANNON's communication chain.</td>
      <td>26</td>
    </tr>
    <tr>
      <td>2.2</td>
      <td>Content of communication messages. Illustrations from https://openmoji.org/</td>
      <td>27</td>
    </tr>
    <tr>
      <td>2.3</td>
      <td>Writing systems of some modern languages. The meaning is “<i>time</i>” for all words.</td>
      <td>28</td>
    </tr>
    <tr>
      <td>2.4</td>
      <td>Singing music sheet features both text and note as symbolic information. Taylor SWIFT's “Blank Space”, 1989 (2014), measures 7–10 (15:30–25:30). Transcription done by Antoine PETIT. Courtesy of Antoine PETIT (PETIT, 2022).</td>
      <td>29</td>
    </tr>
    <tr>
      <td>2.5</td>
      <td>Voice production mechanisms. Courtesy of Léane SALAIS (SALAIS et al., 2022).</td>
      <td>31</td>
    </tr>
    <tr>
      <td>2.6</td>
      <td>Visualization of phonemes and F<sub>1</sub>/F<sub>2</sub> formants on Time-Frequency Representation. Courtesy of Léane SALAIS (SALAIS et al., 2022).</td>
      <td>32</td>
    </tr>
    <tr>
      <td>2.7</td>
      <td>Time-Frequency Representations (TFR) in voice signal processing. Example is sung vocals “I ain't got the <i>time</i>” from Amy WHINEHOUSE's Rehab, HANSEN (2012)'s dataset.</td>
      <td>39</td>
    </tr>
    <tr>
      <td>2.8</td>
      <td>Illustration of differences between speech and singing: vowel duration, F0, vibrato. Example is the same sentence read and sung by KENN_17 from DUAN et al. (2013)'s dataset.</td>
      <td>41</td>
    </tr>
    <tr>
      <td>2.9</td>
      <td>An alignment path <i>&pi;</i> between sequences <b>x</b> (voice log-mel-spectrogram) and <b>y</b> (text).</td>
      <td>45</td>
    </tr>
    <tr>
      <td>2.10</td>
      <td>Transcription task overview: an emission <b>x</b> of a message <i>m</i> is processed by a model <i>M</i> prior to the estimation <b>y</b> of a second and symbolic emission <b>y</b> thanks to a decoder <i>D</i><sub>T</sub>.</td>
      <td>46</td>
    </tr>
    <tr>
      <td>2.11</td>
      <td>Alignment task overview: an alignment path <i>&pi;</i>* between two emissions <b>x</b> and <b>y</b> of a message <i>m</i> is estimated through a modeling function <i>M</i> and a mixed decoder <i>D</i><sub>&pi;</sub>.</td>
      <td>47</td>
    </tr>
  </tbody>
</table>
6.1 Ablation study – impact of scaling the losses. Errors bars correspond to 95% confidence intervals on the mean. AAEs greater than 500ms are masked for readability. 118

6.2 Ablation study – impact of the temporal constraints and the nature of their supervision (mixes vs vocals). Errors bars correspond to 95% confidence intervals on the mean. AAEs greater than 200ms are masked for readability. 118

6.3 Impact of the several temporal constraints during the training phase. The sample used to create the figure was selected from the *validation* set at the end of the first, an intermediate and the final epoch. Example is a clean singing voice dataset. Throughout training, ADAGIO progressively learns not only to recognize the correct phonemes but also to solve the additional constraints leading to better alignments (see evaluations). 119

6.4 An example of inference for voice-to-note alignment with ADAGIO. Note that in this context *e* represents an excitation signal (all but spectral envelope). The posteriorgram, temporally constrained during training, predicts per-frame probability of note occurrences. 124

6.5 Average Absolute Error (AAE) for different percentages of substituted characters for speaking and singing voices with ADAGIO. 125

6.6 Average Absolute Error (AAE) for different percentages of character insertion and deletion for speaking and singing voices with ADAGIO. 126

6.7 Average Absolute Error (AAE) for different percentages of word insertion and deletion for speaking and singing voices with ADAGIO. 127

7.1 Global view of a concatenative synthesis system. To generate an utterance, (di)phonetic units are selected from a pre-aligned database; units are successively concatenated and transformed according to some target parameters. Phonetic alignments of the reference dataset are thus required beforehand. Highly inspired from Figure 3.1 of ARDAILLON (2017). 130

7.2 Example of a sound sample found in Chanter RT, a reference corpus for ISiS, with phonetic alignment of the French locution “Rythmé” (*rhythmic*) phonemized as “R i t m e”. 131

7.3 Phonetic alignment produced on a newly recorded voice with ADAGIO overfitting. 132

7.4 Computation and analysis of vocal tract actuation and phonetic structure extracted from voice temporal alignments for friendly, dominant, distant and seductive vocal attitudes. ‘*: statistically significant difference (p<0.05). Error bars represent 95% confidence intervals on the mean. 134

7.5 Overview of the complete analysis pipeline involving musicological expertise, deep learning models for the automation of voice characterization and alignment in order to help musicologists studying singing voice style. 137
List of Tables

<table><tbody><tr><td>4.1</td><td>Advisable criteria for a robust voice-to-symbol alignment system.</td><td>85</td></tr><tr><td>4.2</td><td>ADAGIO – summary of neural layers and input-output shapes.</td><td>87</td></tr><tr><td>4.3</td><td>ADAGIO – hyperparameter setup.</td><td>88</td></tr><tr><td>5.1</td><td>Time-constrained ADAGIO – summary of neural layers and input-output shapes.</td><td>105</td></tr><tr><td>6.1</td><td>Overview of all voice corpora used in experiments.</td><td>109</td></tr><tr><td>6.2</td><td>Model configurations for ADAGIO according to the loss(es) to minimize.</td><td>116</td></tr><tr><td>6.3</td><td>Impact of additional temporal constraints (with scaling and supervised on vocals) on voice-to-word alignment. For metrics, ↑ means higher is better, ↓ means lower is better, and 95% confidence intervals on the mean are shown.</td><td>116</td></tr><tr><td>6.4</td><td>Results on voice-to-word alignment. For metrics: ↑ means higher is better, ↓ means lower is better, and 95% confidence intervals on the mean are shown. Results for SOTA are from the latest MIREX challenge (https://www.musicir.org/mirex/wiki/2020:Automatic_Lyrics-to-Audio_Alignment_Results) and only share AAE and MAE metrics for the two classical evaluations sets Jamendo and Hansen.</td><td>121</td></tr><tr><td>6.5</td><td>Results on voice-to-phoneme alignment. For metrics: ↑ means higher is better, ↓ means lower is better, and 95% confidence intervals on the mean are shown. “n/a”: non-applicable when overfitting (texts are known) and for ARNN (phonemes are model inputs).</td><td>122</td></tr><tr><td>6.6</td><td>Results on voice-to-note alignment task. For metrics: ↑ means higher is better, ↓ means lower is better, and 95% confidence intervals on the mean are shown.</td><td>123</td></tr><tr><td>7.1</td><td>Manual alignment corrections for the “Blank Space” case study.</td><td>141</td></tr><tr><td>7.2</td><td>Non-legato notes by subsection in Taylor SWIFT’s “Blank Space”, measures 3–10.</td><td>142</td></tr><tr><td>7.3</td><td>Micro-rhythmic displacements by subsection in Taylor SWIFT’s “Blank Space”, measures 3–10.</td><td>143</td></tr></tbody></table>
Bibliography

J. AACH and G. M. CHURCH. Aligning gene expression time series with time warping algorithms. *Bioinformatics*, 17(6):495-508, 2001. (Cited on page 50.)

J. ABESSER and M. MÜLLER. Jazz bass transcription using a u-net architecture. *Electronics*, 10(6):670, 2021. (Cited on page 48.)

L. ARDAILLON. *Synthesis and expressive transformation of singing voice (Chapter 3)*. PhD thesis, Paris VI, 2017. (Cited on pages 42, 112, 130, and 154.)

L. ARDAILLON and A. ROEBEL. Fully-convolutional network for pitch estimation of speech signals. In *Interspeech 2019*, 2019. (Cited on pages 42 and 138.)

L. ARDAILLON, C. CHABOT-CANET, and A. ROEBEL. Expressive control of singing voice synthesis using musical contexts and a parametric f0 model. In *Interspeech 2016*, volume 2016, pages 1250-1254, 2016. (Cited on pages 32, 130, 132, and 135.)

P. ARIAS, P. BELIN, and J.-J. AUCOUTURIER. Auditory smiles trigger unconscious facial imitation. *Current Biology*, 28(14):R782-R783, 2018a. (Cited on page 32.)

P. ARIAS, C. SOLADIE, O. BOUAFIF, A. ROEBEL, R. SEGUIER, and J.-J. AUCOUTURIER. Realistic transformation of facial and vocal smiles in real-time audiovisual streams. *IEEE Transactions on Affective Computing*, 11(3):507-518, 2018b. (Cited on page 32.)

A. ARZT. *Flexible and robust music tracking*. PhD thesis, Johannes Kepler University, Linz, 2016. (Cited on page 67.)

A. ARZT, G. WIDMER, and S. DIXON. Automatic page turning for musicians via real-time machine listening. In *ECAI 2008*, pages 241-245. IOS Press, 2008. (Cited on page 50.)

D. A. BACKSTROM, B. V. TUCKER, and M. C. KELLEY. Forced-alignment of the sung acoustic signal using deep neural nets. *Canadian Acoustics*, 47(3):98-99, 2019. (Cited on page 70.)
D. BAHDANAU, K. CHO, and Y. BENGIO. Neural machine translation by jointly learning to align and translate. *arXiv preprint arXiv:1409.0473*, 2014. (Cited on page 61.)

C. BANDERIER and S. SCHWER. Why delannoy numbers? *Journal of statistical planning and inference*, 135(1):40–54, 2005. (Cited on page 65.)

K. BANTUPALLI and Y. XIE. American sign language recognition using deep learning and computer vision. In *2018 IEEE International Conference on Big Data (Big Data)*, pages 4896–4899. IEEE, 2018. (Cited on page 47.)

H. B. BARLOW. Unsupervised learning. *Neural computation*, 1(3):295–311, 1989. (Cited on page 58.)

A. BARÓ, P. RIBA, J. CALVO-ZARAGOZA, and A. FORNÉS. From optical music recognition to handwritten music recognition: a baseline. *Pattern Recognition Letters*, 123:1–8, 2019. (Cited on page 48.)

Y. BENGIO, P. FRASCONI, and P. SIMARD. The problem of learning long-term dependencies in recurrent networks. In *IEEE international conference on neural networks*, pages 1183–1188. IEEE, 1993. (Cited on page 61.)

L. BERIO. *Remembering the future*, volume 52. Harvard University Press, 2006. (Cited on page 150.)

M. BERNARD and H. TITEUX. Phonemizer: Text to phones transcription for multiple languages in python. *Journal of Open Source Software*, 6(68):3958, 2021. (Cited on page 133.)

M. BESSON and D. SCHÖN. Comparison between language and music. *Annals of the New York Academy of Sciences*, 930(1):232–258, 2001. (Cited on page 26.)

R. M. BITTNER, J. SALAMON, S. ESSID, and J. P. BELLO. Melody extraction by contour classification. In *International Conference on Music Information Retrieval (ISMIR)*, 2015. (Cited on page 48.)

T. BLUCHE, H. NEY, J. LOURADOUR, and C. KERMORVANT. Framewise and ctc training of neural networks for handwriting recognition. *13th International Conference on Document Analysis and Recognition (ICDAR)*, pages 81–85, 2015. (Cited on page 91.)

G. BORDEL, M. PENAGARIKANO, L. J. RODRÍGUEZ-FUENTES, A. ÁLVAREZ, and A. VARONA. Probabilistic kernels for improved text-to-speech alignment in long audio tracks. *IEEE Signal Processing Letters*, 23(1):126–129, 2015. (Cited on page 144.)

J.-P. BRIOT. Apprentissage profond et génération de musique. *ActuIA-Le magazine de l'intelligence artificielle*, (7), 46:2682–1265, 2022. (Cited on page 55.)
P. F. BROWN, S. A. DELLA Pietra, V. J. DELLA Pietra, R. L. MERCER, et al. The mathematics of statistical machine translation: Parameter estimation. 1993. (Cited on page 50.)

F. BRUGNARA, D. FALAVIGNA, and M. OMOLOGO. Automatic segmentation and labeling of speech based on hidden markov models. *Speech Communication*, 12(4):357-370, 1993. (Cited on page 67.)

D. A. BRUNOW and T. A. CULLEN. Effect of text-to-speech and human reader on listening comprehension for students with learning disabilities. *Computers in the Schools*, 38(3):214-231, 2021. (Cited on page 42.)

J. CALVO-ZARAGOZA, J. J. VALERO-MAS, and A. PERTUSA. End-to-end optical music recognition using neural networks. In *Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR*, pages 23-27, 2017. (Cited on page 48.)

V. CAPORALETTI. *Swing e Groove: sui fondamenti estetici delle musiche audiotattili*. Libreria Italiana Musicale, 2014. (Cited on pages 142 and 143.)

C. CHABOT-CANET. *Léo Ferré: une voix et un phrasé emblématiques*. L'Harmattan, 2008. (Cited on page 136.)

C. CHABOT-CANET. L'analyse spectrale au fondement d'une rhétorique des styles interprétatifs dans la chanson française. *Volume!*, 16(2)/17(1):29-47, 2020a. (Cited on page 136.)

C. CHABOT-CANET. Spectrum analysis and the rhetorics of performance styles in french chanson. *Volume!*, 16(2)/17(1):29-47, 2020b. (Cited on page 32.)

C. CHABOT-CANET, L. ARDAILLON, and A. ROEBEL. Modeling vocal styles, synthesizing expressive singing. the example of édith piaf. *Volume!*, 16(2)/17(1):63-85, 2020. (Cited on pages 32 and 40.)

Y.-P. CHO, F.-R. YANG, Y.-C. CHANG, C.-T. CHENG, X.-H. WANG, and Y.-W. LIU. A survey on recent deep learning-driven singing voice synthesis systems. In *2021 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)*, pages 319-323. IEEE, 2021. (Cited on page 41.)

W. CHOI, M. KIM, J. CHUNG, D. LEE, and S. JUNG. Investigating U-Nets with various intermediate blocks for spectrogram-based singing voice separation. *arXiv preprint arXiv:1912.02591*, 2019. (Cited on pages 93, 117, 137, and 145.)

W. CHOI, M. KIM, J. CHUNG, and S. JUNG. LaSAFT: Latent source attentive frequency transformation for conditioned source separation. In *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 171-175. IEEE, 2021. (Cited on page 145.)
C. S. CHONG, J. KIM, and C. DAVIS. Disgust expressive speech: the acoustic consequences of the facial expression of emotion. *Speech Communication*, 98:68–72, 2018. (Cited on page 32.)

A. COHEN-HADRIA and G. PEETERS. Music structure boundaries estimation using multiple self-similarity matrices as input depth of convolutional neural networks. In *Audio Engineering Society Conference: 2017 AES International Conference on Semantic Audio. Audio Engineering Society*, 2017. (Cited on page 94.)

A. COHEN-HADRIA, A. ROEBEL, and G. PEETERS. Improving singing voice separation using deep u-net and wave-u-net with data augmentation. In *2019 27th European Signal Processing Conference (EUSIPCO)*, pages 1–5. IEEE, 2019. (Cited on pages 42, 93, and 137.)

R. COLLIER and J. HART. The role of intonation in speech perception. In *Structure and process in speech perception*, pages 107–123. Springer, 1975. (Cited on page 32.)

R. COLLOBERT, C. PUHRSCH, and G. SYNNAEVE. Wav2letter: an end-to-end convnet-based speech recognition system. *arXiv preprint arXiv:1609.03193*, 2016. (Cited on pages 71 and 84.)

A. CONT. A coupled duration-focused architecture for real-time music-to-score alignment. *IEEE transactions on pattern analysis and machine intelligence*, 32(6):974–987, 2009. (Cited on pages 50 and 68.)

A. CONT. On the creative use of score following and its impact on research. In *SMC 2011: 8th Sound and Music Computing conference*, 2011. (Cited on page 145.)

A. CONT, D. SCHWARZ, N. SCHNELL, and C. RAPHAEL. Evaluation of real-time audio-to-score alignment. In *International Society for Music Information Retrieval (ISMIR)*, 2007. (Cited on pages 112 and 113.)

A. CONT, J. ECHEVESTE, J.-L. GIAVITTO, and F. JACQUEMARD. Correct automatic accompaniment despite machine listening or human errors in antescofo. In *ICMC 2012-International Computer Music Conference*, 2012. (Cited on page 50.)

G. COOK. *Applied linguistics*. Oxford University Press, 2003. (Cited on page 26.)

E. E. CUST, A. J. SWEETING, K. BALL, and S. ROBERTSON. Machine and deep learning for sport-specific movement recognition: a systematic review of model development and performance. *Journal of sports sciences*, 37(5):568–600, 2019. (Cited on page 47.)

P. CUVILLIER. *On temporal coherency of probabilistic models for audio-to-score alignment*. PhD thesis, Université Pierre et Marie Curie-Paris VI, 2016. (Cited on page 68.)
P. CUVILLIER and A. CONT. Coherent time modeling of semi-markov models with application to real-time audio-to-score alignment. In *2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP)*, pages 1–6. IEEE, 2014. (Cited on page 68.)

R. B. DANNENBERG. An on-line algorithm for real-time accompaniment. In *ICMC*, volume 84, pages 193–198, 1984. (Cited on page 67.)

S. DAVIS and P. MERMELSTEIN. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. *IEEE transactions on acoustics, speech, and signal processing*, 28(4):357–366, 1980. (Cited on page 38.)

E. DEMIREL, S. AHLBÄCK, and S. DIXON. Mstre-net: Multisampling acoustic modeling for automatic lyrics transcription. *International Society for Music Information Retrieval (ISMIR)*, 2021. (Cited on page 111.)

A. DISTER, M. CONSTANT, and G. PURNELLE. Normalizing speech transcriptions for natural language processing. In *3rd International Conference on Spoken Communication (GSCP'09)*, 2009. (Cited on page 47.)

S. DIXON. An on-line time warping algorithm for tracking musical performances. In *IJCAI*, pages 1727–1728, 2005. (Cited on page 67.)

G. DORAS, Y. TEYTAUT, and A. ROEBEL. A linear memory ctc-based algorithm for text-to-voice alignment of very long audio recordings. *Applied Sciences*, 13(3):1854, 2023. (Cited on pages 22, 110, 111, 112, 124, 136, and 144.)

C. DOUWES, G. BINDI, A. CAILLON, P. ESLING, and J.-P. BRIOT. Is quality enough? integrating energy consumption in a large-scale evaluation of neural audio synthesis models. In *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2023. (Cited on page 55.)

Z. DUAN, H. FANG, B. LI, K. C. SIM, and Y. WANG. The nus sung and spoken lyrics corpus: A quantitative comparison of singing and speech. In *2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference*, pages 1–9. IEEE, 2013. (Cited on pages 40, 41, 68, and 152.)

H. DUDLEY, R. R. RIESZ, and S. S. WATKINS. A synthetic speaker. *Journal of the Franklin Institute*, 227(6):739–764, 1939. (Cited on page 41.)

D. P. ELLIS. Beat tracking by dynamic programming. *Journal of New Music Research*, 36(1): 51–60, 2007. (Cited on page 67.)

W. EVERETT. *The Foundations of Rock: From “Blue Suede Shoes” to “Suite: Judy Blue Eyes”*. Oxford University Press, 2009. (Cited on page 140.)
M. FARES, C. PELACHAUD, and N. OBIN. Multimodal generation of upper-facial and head gestures with a transformer network using speech and text. 2021. (Cited on page 27.)

S. FARNER, A. ROEBEL, C. VEAUX, G. BELLER, X. RODET, and L. ACH. Voice transformation and speech synthesis for video games. In *Paris Game Developers Conference, Paris, France*, 2008. (Cited on page 42.)

H. FASTL and E. ZWICKER. *Psychoacoustics: facts and models*, volume 22. Springer Science & Business Media, 2006. (Cited on page 37.)

M. FELL, Y. NECHAEV, G. MESEGUER-BROCAL, E. CABRIO, F. GANDON, and G. PEETERS. Lyrics segmentation via bimodal text-audio representation. *Natural Language Engineering*, 28(3):317-336, 2022. (Cited on page 94.)

S. R. FISCHER. *History of writing*. Reaktion books, 2003. (Cited on page 27.)

D. FOLGADO, M. BARANDAS, R. MATIAS, R. MARTINS, M. CARVALHO, and H. GAMBOA. Time alignment measurement for time series. *Pattern Recognition*, 81:268-279, 2018. (Cited on page 50.)

I. FÓNAGY. *La Vive Voix: essais de psycho-phonétique*. Payot, 1983. (Cited on page 136.)

M. FREITAG and Y. AL-ONAIZAN. Beam search strategies for neural machine translation. *ACL 2017*, page 56, 2017. (Cited on page 45.)

H. FUJIHARA and M. GOTO. Lyrics-to-audio alignment and its application. In *Dagstuhl Follow-Ups*, volume 3. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2012. (Cited on page 51.)

X. GAO, C. GUPTA, and H. LI. Lyrics transcription and lyrics-to-audio alignment with music-informed acoustic models. *MIREX*, 2021. (Cited on pages 69 and 115.)

X. GAO, C. GUPTA, and H. LI. Automatic lyrics transcription of polyphonic music with lyrics-chord multi-task learning. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 30:2280-2294, 2022. (Cited on page 136.)

D. GARREAU, R. LAJUGIE, S. ARLOT, and F. BACH. Metric learning for temporal sequence alignment. *Advances in neural information processing systems*, 27, 2014. (Cited on page 65.)

U. GERMANN. Greedy decoding for statistical machine translation in almost linear time. In *Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics*, pages 72-79, 2003. (Cited on page 45.)
R. GONG, P. CUVILLIER, N. OBIN, and A. CONT. Real-time audio-to-score alignment of singing voice based on melody and lyric information. In *Interspeech*, 2015. (Cited on page 68.)

I. GOODFELLOW, Y. BENGIO, and A. COURVILLE. *Deep learning*. MIT press, 2016. (Cited on page 55.)

I. J. GOODFELLOW, Y. BULATOV, J. IBARZ, S. ARNOUD, and V. SHET. Multi-digit number recognition from street view imagery using deep convolutional neural networks. *International Conference on Learning Representations*, 2014. (Cited on page 47.)

K. GORMAN, J. HOWELL, and M. WAGNER. Prosodylab-aligner: A tool for forced alignment of laboratory speech. *Canadian Acoustics*, 39(3):192–193, 2011. (Cited on page 68.)

D. GOTTLIEB and C.-W. SHU. On the gibbs phenomenon and its resolution. *SIAM review*, 39 (4):644–668, 1997. (Cited on page 36.)

A. GRAVES and N. JAITLY. Towards end-to-end speech recognition with recurrent neural networks. In *International conference on machine learning*, pages 1764–1772. PMLR, 2014. (Cited on page 71.)

A. GRAVES, S. FERNÁNDEZ, F. GOMEZ, and J. SCHMIDHUBER. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In *Proceedings of the 23rd international conference on Machine learning*, pages 369–376, 2006. (Cited on pages 71, 72, 74, 75, and 90.)

A. GRAVES, A.-r. MOHAMED, and G. HINTON. Speech recognition with deep recurrent neural networks. In *2013 IEEE international conference on acoustics, speech and signal processing*, pages 6645–6649. Ieee, 2013. (Cited on page 69.)

L. GRUBB and R. B. DANNENBERG. Automated accompaniment of musical ensembles. In *Proceedings of the 12th National Conference on Artificial Intelligence (AAAI)*, pages 94–99, 1994. (Cited on page 67.)

C. GUPTA, D. GRUNBERG, P. RAO, and Y. WANG. Towards automatic mispronunciation detection in singing. In *ISMIR*, pages 390–396, 2017. (Cited on page 67.)

C. GUPTA, H. LI, and Y. WANG. Automatic pronunciation evaluation of singing. In *Interspeech*, pages 1507–1511, 2018. (Cited on page 115.)

C. GUPTA, E. YILMAZ, and H. LI. Automatic lyrics alignment and transcription in polyphonic music: Does background music help? In *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 496–500. IEEE, 2020. (Cited on pages 69, 115, and 146.)
D. A. HANNINEN. *A Theory of Music Analysis: On Segmentation and Associative Organization*. University of Rochester Press, 2012. (Cited on page 140.)

A. HANNUN. Sequence modeling with ctc. *Distill*, 2017. doi: 10.23915/distill.00008.
https://distill.pub/2017/ctc. (Cited on pages 75, 90, 101, and 153.)

A. HANNUN, C. CASE, J. CASPER, B. CATANZARO, G. DIAMOS, E. ELSEN, R. PRENGER, S. SATHEESH, SENGUPTA Shubho, A. COATES, et al. Deep speech: Scaling up end-to-end speech recognition. *arXiv preprint arXiv:1412.5567*, 2014. (Cited on page 69.)

J. K. HANSEN. Recognition of phonemes in a-cappella recordings using temporal patterns and mel frequency cepstral coefficients. In *9th Sound and Music Computing Conference (SMC)*, pages 494–499, 2012. (Cited on pages 39, 111, and 152.)

A. M. HEIN. Identification and bridging of semantic gaps in the context of multi-domain engineering. In *Forum on Philosophy, Engineering & Technology*, pages 58–57, 2010. (Cited on page 16.)

C. F. HELLWAG. *Dissertatio de Formatione Loquelae* (1781). Verlag von Gebr. Henninger, 1886. (Cited on page 31.)

R. HENNEQUIN, A. KHLIF, F. VOITURET, and M. MOUSSALLAM. Spleeter: a fast and efficient music source separation tool with pre-trained models. *Journal of Open Source Software*, 5 (50):2154, 2020. (Cited on page 85.)

N. HENRICH BERNARDONI. La voix timbrée dans les chansons: considérations physiologiques et acoustiques. *Volume!*, 16(2)/17(1):49–61, 2020. (Cited on page 136.)

J. R. HIGGINS et al. *Sampling theory in Fourier and signal analysis: foundations*. Oxford University Press on Demand, 1996. (Cited on page 34.)

G. HINTON, L. DENG, D. YU, G. E. DAHL, A.-r. MOHAMED, N. JAITLY, A. SENIOR, V. VANHOUCKE, P. NGUYEN, T. N. SAINATH, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. *IEEE Signal processing magazine*, 29(6):82–97, 2012. (Cited on page 63.)

S. HOCHREITER and J. SCHMIDHUBER. Long short-term memory. *Neural computation*, 9(8): 1735–1780, 1997. (Cited on page 61.)

T. HORI, S. WATANABE, Y. ZHANG, and W. CHAN. Advances in joint ctc-attention based end-to-end speech recognition with a deep cnn encoder and rnn-lm. *Proc. Interspeech 2017*, pages 949–953, 2017. (Cited on page 71.)
J.-P. HOSOM. Speaker-independent phoneme alignment using transition-dependent states. *Speech communication*, 51(4):352–368, 2009. (Cited on page 68.)

J. HUANG, E. BENETOS, and S. EWERT. Improving lyrics alignment through joint pitch detection. In *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 451–455. IEEE, 2022. (Cited on page 146.)

E. J. HUMPHREY, J. P. BELLO, and Y. LECUN. Moving beyond feature design: Deep architectures and automatic feature learning in music informatics. In *International Society for Music Information Retrieval (ISMIR)*, pages 403–408, 2012. (Cited on page 55.)

S. IOFFE and C. SZEGEDY. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In *International conference on machine learning*, pages 448–456. pmlr, 2015. (Cited on page 62.)

F. ITAKURA. Minimum prediction residual principle applied to speech recognition. *IEEE Transactions on acoustics, speech, and signal processing*, 23(1):67–72, 1975. (Cited on pages 44 and 66.)

C. JACQUES and A. ROEBEL. Data augmentation for drum transcription with convolutional neural networks. In *2019 27th European Signal Processing Conference (EUSIPCO)*, pages 1–5. IEEE, 2019. (Cited on page 48.)

A. JANSSON, E. HUMPHREY, N. MONTECCHIO, R. BITTNER, A. KUMAR, and T. WEYDE. Singing voice separation with deep u-net convolutional networks. 2017. (Cited on page 93.)

J. JIANG, K. CHEN, W. LI, and G. XIA. Large-vocabulary chord transcription via chord structure decomposition. In *International Society for Music Information Retrieval (ISMIR)*, pages 644–651, 2019. (Cited on page 48.)

P. N. JUSLIN. *Musical emotions explained: Unlocking the secrets of musical affect*. Oxford University Press, USA, 2019. (Cited on page 32.)

A. KATSAMANIS, M. BLACK, P. G. GEORGIOU, L. GOLDSTEIN, and S. NARAYANAN. Sailalign: Robust long speech-text alignment. In *Proceedings of workshop on new tools and methods for very-large scale phonetics research*, 2011. (Cited on page 144.)

M. C. KELLEY and B. V. TUCKER. A comparison of input types to a deep neural network-based forced aligner. 2018. (Cited on page 70.)

R. D. KENT and C. READ. *The acoustic analysis of speech*. San Diego, California: Singular, 2nd edn, 2002. (Cited on page 40.)
S. KIM, T. HORI, and S. WATANABE. Joint ctc-attention based end-to-end speech recognition using multi-task learning. In *2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)*, pages 4835–4839. IEEE, 2017. (Cited on pages 71, 90, and 101.)

D. P. KINGMA and J. BA. Adam: A method for stochastic optimization. *arXiv preprint arXiv:1412.6980*, 2014. (Cited on page 59.)

H. KIRCHHOFF and A. LERCH. Evaluation of features for audio-to-audio alignment. *Journal of New Music Research*, 40(1):27–41, 2011. (Cited on page 50.)

L. KÜRZINGER, D. WINKELBAUER, L. LI, T. WATZEL, and G. RIGOLL. Ctc-segmentation of large corpora for german end-to-end speech recognition. In *Speech and Computer: 22nd International Conference, SPECOM 2020, St. Petersburg, Russia, October 7–9, 2020, Proceedings 22*, pages 267–278. Springer, 2020. (Cited on pages 71 and 145.)

S. LACASSE. The phonographic voice: paralinguistic features and phonographic staging in popular music singing. In A. Bayley, editor, *Recorded Music: Performance, Culture and Technology*, pages 225–251. Cambridge University Press, 2010. (Cited on page 136.)

A. LACHERET-DUJOUR and F. BEAUGENDRE. *La Prosodie du français*. CNRS, 1999. (Cited on page 136.)

R. LAJUGIE, P. BOJANOWSKI, P. CUVILLIER, S. ARLOT, and F. BACH. A weakly-supervised discriminative model for audio-to-score alignment. In *2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 2484–2488. IEEE, 2016. (Cited on page 145.)

E. P. LANCASTER and N. SOUVIRAÀ-LABASTIE. A frugal approach to music source separation, 2020. URL https://hal.science/hal-02986241v1/document. (Cited on pages 93 and 137.)

P. LANCHANTIN, A. C. MORRIS, X. RODET, and C. VEAUX. Automatic phoneme segmentation with relaxed textual constraints. In *LREC*, 2008. (Cited on page 131.)

C. LE MOINE and N. OBIN. Att-hack: An expressive speech database with social attitudes. *Speech Prosody*, Tokyo, Japan, May, 2020. (Cited on pages 32, 110, and 133.)

C. LE MOINE, N. OBIN, and A. ROEBEL. Speaker attentive speech emotion recognition. In *Interspeech 2021*, pages 2866–2870. ISCA, 2021. (Cited on page 32.)

Y. LECUN, B. BOSER, J. DENKER, D. HENDERSON, R. HOWARD, W. HUBBARD, and L. JACKEL. Handwritten digit recognition with a back-propagation network. *Advances in neural information processing systems*, 2, 1989. (Cited on pages 47, 48, and 153.)
Y. LECUN, Y. BENGIO, et al. Convolutional networks for images, speech, and time series. *The handbook of brain theory and neural networks*, 3361(10):1995, 1995. (Cited on page 60.)

H. LEUNG and V. ZUE. A procedure for automatic alignment of phonetic transcriptions with continuous speech. In *ICASSP'84. IEEE International Conference on Acoustics, Speech, and Signal Processing*, volume 9, pages 73–76. IEEE, 1984. (Cited on page 67.)

K.-W. LIANG, H.-C. LEE, Y.-T. LAI, and P.-C. CHANG. Query by singing and humming system based on combined dtw and linear scaling. In *2021 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW)*, pages 1–2. IEEE, 2021a. (Cited on page 42.)

S. LIANG, C. DENG, and Y. ZHANG. A simple approach to balance task loss in multi-task learning. In *2021 IEEE International Conference on Big Data (Big Data)*, pages 812–823. IEEE, 2021b. (Cited on page 101.)

A. H. LIU, W.-N. HSU, M. AULI, and A. BAEVSKI. Towards end-to-end unsupervised speech recognition. In *2022 IEEE Spoken Language Technology Workshop (SLT)*, pages 221–228. IEEE, 2023. (Cited on page 48.)

H. LIU, S. JIN, and C. ZHANG. Connectionist temporal classification with maximum entropy regularization. *Advances in Neural Information Processing Systems*, 31:831–841, 2018. (Cited on page 91.)

A. LIUTKUS, D. FITZGERALD, and Z. RAFII. Scalable audio separation with light kernel additive modelling. In *2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 76–80. IEEE, 2015. (Cited on page 137.)

S. R. LIVINGSTONE, K. PECK, and F. A. RUSSO. Acoustic differences in the speaking and singing voice. In *Proceedings of Meetings on Acoustics ICA2013*, volume 19, page 035080. Acoustical Society of America, 2013. (Cited on page 40.)

A. LJOLJE and M. RILEY. Automatic segmentation and labeling of speech. In *Acoustics, Speech, and Signal Processing, IEEE International Conference on*, pages 473–476. IEEE Computer Society, 1991. (Cited on pages 67 and 68.)

R. LOISEAU, T. MONNIER, M. AUBRY, and L. LANDRIEU. Representing shape collections with alignment-aware linear models. In *2021 International Conference on 3D Vision (3DV)*, pages 1044–1053. IEEE, 2021. (Cited on page 55.)

R. LOISEAU, B. BOUVIER, Y. TEYTAUT, E. VINCENT, M. AUBRY, and L. LANDRIEU. A model you can hear: Audio identification with playable prototypes. In *23rd International Society for Music Information Retrieval Conference (ISMIR 2022)*, 2022. (Cited on page 22.)
I. LÓPEZ-ESPEJO, Z.-H. TAN, J. H. HANSEN, and J. JENSEN. Deep spoken keyword spotting: An overview. *IEEE Access*, 10:4169–4199, 2021. (Cited on page 42.)

P. LÉON. *Précis de phonostylistique: parole et expressivité*. Nathan Université, 1993. (Cited on page 136.)

A. L. MAAS, A. Y. HANNUN, A. Y. NG, et al. Rectifier nonlinearities improve neural network acoustic models. In *Proceedings of the International Conference on Machine Learning (ICML)*, volume 30, page 3. Atlanta, Georgia, USA, 2013. (Cited on page 56.)

I. P. MACHADO, A. L. GOMES, H. GAMBOA, V. PAIXÃO, and R. M. COSTA. Human activity data discovery from triaxial accelerometer sensor: Non-supervised learning sensitivity to feature extraction parametrization. *Information Processing & Management*, 51(2):204–214, 2015. (Cited on page 50.)

L. MAO. Number of alignments in connectionist temporal classification (ctc). web resource, https://leimao.github.io/blog/CTC-Alignment-Combinations/, 2019. accessed 18 March 2023. (Cited on page 101.)

N. L. MASCLEF, A. VAGLIO, and M. MOUSSALLAM. User-centered evaluation of lyrics-to-audio alignment. In *International Society for Music Information Retrieval (ISMIR)*, pages 420–427, 2021. (Cited on page 113.)

P. McALEER, A. TODOROV, and P. BELIN. How do you say ‘hello’? personality impressions from brief novel voices. *PloS one*, 9(3):e90779, 2014. (Cited on page 133.)

M. McAULIFFE, M. SOCOLOF, S. MIHUC, M. WAGNER, and M. SONDREGGER. Montreal forced aligner: Trainable text-speech alignment using kaldi. In *Interspeech*, volume 2017, pages 498–502, 2017. (Cited on page 68.)

B. MCFEE, C. RAFFEL, D. LIANG, D. P. ELLIS, M. MCVICAR, E. BATTENBERG, and O. NIETO. librosa: Audio and music signal analysis in python. In *Proceedings of the 14th python in science conference*, volume 8, pages 18–25, 2015. (Cited on page 85.)

G. MESEGUER-BROCAL and G. PEETERS. Content based singing voice source separation via strong conditioning using aligned phonemes. In *21st International Society for Music Information Retrieval (ISMIR)*, 2020. (Cited on pages 111 and 145.)

G. MESEGUER-BROCAL, A. COHEN-HADRIA, and G. PEETERS. Dali: A large dataset of synchronized audio, lyrics and notes, automatically created using teacher-student machine learning paradigm. *International Society for Music Information Retrieval (ISMIR)*, Paris, France, 2018. (Cited on page 111.)
G. MESEGUER-BROCAL, R. BITTNER, S. DURAND, and B. BROST. Data cleansing with contrastive learning for vocal note event annotations. In *Proceedings of the 21st International Society for Music Information Retrieval Conference*, 2020a. (Cited on page 58.)

G. MESEGUER-BROCAL, A. COHEN-HADRIA, and G. PEETERS. Creating dali, a large dataset of synchronized audio, lyrics, and notes. *Transactions of the International Society for Music Information Retrieval (ISMIR)*, 3(1), 2020b. (Cited on page 111.)

H. MIAO, G. CHENG, C. GAO, P. ZHANG, and Y. YAN. Transformer-based online ctc/attention end-to-end speech recognition architecture. In *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 6084–6088. IEEE, 2020. (Cited on page 83.)

A. H. MICHAELY, X. ZHANG, G. SIMKO, C. PARADA, and P. ALEKSIC. Keyword spotting for google assistant using contextual speech recognition. In *2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)*, pages 272–278. IEEE, 2017. (Cited on page 48.)

O. MIGLIORE and N. OBIN. At the interface of speech and music: A study of prosody and musical prosody in rap music. In *Speech Prosody*, 2018. (Cited on page 32.)

A.-r. MOHAMED. *Deep Neural Network Acoustic Models for ASR*. PhD thesis, University of Toronto, 2014. (Cited on page 69.)

A.-r. MOHAMED, G. E. DAHL, and G. HINTON. Acoustic modeling using deep belief networks. *IEEE transactions on audio, speech, and language processing*, 20(1):14–22, 2011. (Cited on page 69.)

M. MONGEAU and D. SANKOFF. Comparison of musical sequences. *Computers and the Humanities*, 24:161–175, 1990. (Cited on page 67.)

M. MÜLLER. Dynamic time warping. *Information retrieval for music and motion*, pages 69–84, 2007a. (Cited on page 67.)

M. MÜLLER. *Information retrieval for music and motion*, volume 2. Springer, 2007b. (Cited on page 42.)

M. MÜLLER. *Fundamentals of music processing: Audio, analysis, algorithms, applications*, volume 5. Springer, 2015. (Cited on page 33.)

M. MÜLLER, F. KURTH, and M. CLAUSEN. Audio matching via chroma-based statistical features. In *International Society for Music Information Retrieval (ISMIR)*, volume 2005, page 6. Citeseer, 2005. (Cited on page 67.)
T. NAKAMURA, E. NAKAMURA, and S. SAGAYAMA. Real-time audio-to-score alignment of music performances containing errors and arbitrary repeats and skips. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 24(2):329–339, 2015. (Cited on page 67.)

S. B. NEEDLEMAN and C. D. WUNSCH. A general method applicable to the search for similarities in the amino acid sequence of two proteins. *Journal of molecular biology*, 48 (3):443–453, 1970. (Cited on pages 44, 49, 50, 66, and 153.)

P. C. NG and S. HENIKOFF. Predicting deleterious amino acid substitutions. *Genome research*, 11(5):863–874, 2001. (Cited on page 50.)

Y. NING, S. HE, Z. WU, C. XING, and L.-J. ZHANG. A review of deep learning based speech synthesis. *Applied Sciences*, 9(19):4050, 2019. (Cited on page 41.)

Y. OHISHI, M. GOTO, K. ITOU, and K. TAKEDA. Discrimination between singing and speaking voices. In *Ninth European Conference on Speech Communication and Technology*, 2005. (Cited on page 40.)

D. W. OTTER, J. R. MEDINA, and J. K. KALITA. A survey of the usages of deep learning for natural language processing. *IEEE transactions on neural networks and learning systems*, 32 (2):604–624, 2020. (Cited on page 55.)

D. PAL, C. ARPNIKANONDT, S. FUNILKUL, and V. VARADARAJAN. User experience with smart voice assistants: the accent perspective. In *2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT)*, pages 1–6. IEEE, 2019. (Cited on page 42.)

V. PANAYOTOV, G. CHEN, D. POVEY, and S. KHUDANPUR. Librispeech: an asr corpus based on public domain audio books. In *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 5206–5210. IEEE, 2015. (Cited on page 109.)

M. PAPAKOSTAS, E. SPYROU, T. GIANNAKOPOULOS, G. SIANTIKOS, D. SGOUROPOULOS, P. MYLONAS, and F. MAKEDON. Deep visual attributes vs. hand-crafted audio features on multidomain speech emotion recognition. *Computation*, 5(2):26, 2017. (Cited on page 42.)

H. PARK, C. KIM, H. SON, S. SEO, and J.-H. KIM. Hybrid ctc-attention network-based end-to-end speech recognition system for korean language. *Journal of Web Engineering*, pages 265–284, 2022. (Cited on page 90.)

G. PEETERS and F. ANGULO. Ssm-net: feature learning for music structure analysis using a self-similarity-matrix based loss. *arXiv preprint arXiv:2211.08141*, 2022. (Cited on page 95.)
G. PEETERS, B. L. GIORDANO, P. SUSINI, N. MISDARIIS, and S. MCADAMS. The timbre toolbox: Extracting audio descriptors from musical signals. *The Journal of the Acoustical Society of America*, 130(5):2902–2916, 2011. (Cited on page 42.)

A. PETIT. Le travail de la pop : écouter 1989 (2014) de Taylor Swift en musicologue. Colloque *Le Silence du mainstream*, Strasbourg, 2022. https://www.canalc2.tv/video/16316 (accessed February, 26th 2022). (Cited on pages 29, 140, and 152.)

K. PISANSKI, A. ANIKIN, and D. REBY. Vocal size exaggeration may have contributed to the origins of vocalic complexity. *Philosophical Transactions of the Royal Society B*, 377(1841): 20200401, 2022. (Cited on page 134.)

P. PLACEWAY and J. LAFFERTY. Cheating with imperfect transcripts. In *Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP'96*, volume 4, pages 2115–2118. IEEE, 1996. (Cited on pages 67 and 68.)

E. PONSOT, P. ARIAS, and J.-J. AUCOUTURIER. Uncovering mental representations of smiled speech using reverse correlation. *The Journal of the Acoustical Society of America*, 143(1): EL19–EL24, 2018. (Cited on page 32.)

D. POVEY, A. GHOSHAL, G. BOULIANNE, L. BURGET, O. GLEMBEK, N. GOEL, M. HANNEMANN, P. MOTLICEK, Y. QIAN, P. SCHWARZ, et al. The kaldi speech recognition toolkit. In *IEEE 2011 workshop on automatic speech recognition and understanding*, number CONF. IEEE Signal Processing Society, 2011. (Cited on page 115.)

F. POYATOS. *Nonverbal Communication across Disciplines*. John Benjamins Publishing, 2002. (Cited on page 136.)

D. S. PRERAU. Computer pattern recognition of printed music. In *Proceedings of the November 16-18, 1971, fall joint computer conference*, pages 153–162, 1972. (Cited on pages 48 and 153.)

H. PURWINS, B. LI, T. VIRTANEN, J. SCHLÜTER, S.-Y. CHANG, and T. SAINATH. Deep learning for audio signal processing. *IEEE Journal of Selected Topics in Signal Processing*, 13(2):206–219, 2019. (Cited on page 55.)

L. R. RABINER and B. GOLD. Theory and application of digital signal processing. *Englewood Cliffs: Prentice-Hall*, 1975. (Cited on page 33.)

C. RAFFEL, B. MCFEE, E. J. HUMPHREY, J. SALAMON, O. NIETO, D. LIANG, D. P. ELLIS, and C. C. RAFFEL. Mir_eval: A transparent implementation of common mir metrics. In *International Society for Music Information Retrieval (ISMIR)*, pages 367–372, 2014. (Cited on page 112.)
Z. RAFII, A. LIUTKUS, F.-R. STÖTER, S. I. MIMILAKIS, and R. BITTNER. The musdb18 corpus for music separation, 2017. (Cited on page 137.)

E. RAMASSO, M. ROMBAUT, and D. PELLERIN. Forward-backward-viterbi procedures in the transferable belief model for state sequence analysis using belief functions. In *Symbolic and Quantitative Approaches to Reasoning with Uncertainty: 9th European Conference, ECSQARU 2007, Hammamet, Tunisia, October 31-November 2, 2007. Proceedings 9*, pages 405–417. Springer, 2007. (Cited on page 49.)

C. RAPHAEL. Automatic segmentation of acoustic musical signals using hidden markov models. *IEEE transactions on pattern analysis and machine intelligence*, 21(4):360–370, 1999. (Cited on page 67.)

C. RAPHAEL. Music plus one: A system for flexible and expressive musical accompaniment. *Proc. of the ICMC*, 2001, 2001. (Cited on page 50.)

C. RAPHAEL. Aligning music audio with symbolic scores using a hybrid graphical model. *Machine learning*, 65(2-3):389–409, 2006. (Cited on page 68.)

L. RENAULT, A. VAGLIO, and R. HENNEQUIN. Singing language identification using a deep phonotactic approach. In *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 271–275. IEEE, 2021. (Cited on pages 71 and 111.)

G. RÉVÉSZ. *Introduction to the psychology of music*. University of Oklahoma Press, 1954. (Cited on page 37.)

F. RIDZUAN and W. M. N. W. ZAINON. A review on data cleansing methods for big data. *Procedia Computer Science*, 161:731–738, 2019. (Cited on page 58.)

F. RIGAUD and M. RADENEN. Singing voice melody transcription using deep neural networks. In *International Society for Music Information Retrieval (ISMIR)*, pages 737–743, 2016. (Cited on page 48.)

A. RÖBEL and X. RODET. Efficient spectral envelope estimation and its application to pitch shifting and envelope preservation. In *International Conference on Digital Audio Effects*, pages 30–35, 2005. (Cited on page 38.)

H. RODRIGUEZ, P. ARIAS, and C. CANONNE. Investigating the pragmatic effects of musical syntax through musical humor. In *15th International Symposium of Cognition, Logic, and Communication*, 2021. (Cited on page 32.)

A. ROEBEL and F. BOUS. Neural vocoding for singing and speaking voices with the multi-band excited wavenet. *Information*, 13(3):103, 2022. (Cited on page 145.)
I. ROSENFELDER, J. FRUEHWALD, K. EVANINI, S. SEYFARTH, K. GORMAN, H. PRICHARD, and J. YUAN. Fave (forced alignment and vowel extraction) 1.1.3. web resource, http://dx.doi.org/10.5281/zenodo.9846, 2017. accessed 22 February 2023. (Cited on page 68.)

D. E. RUMELHART, G. E. HINTON, and R. J. WILLIAMS. Learning representations by back-propagating errors. *Nature*, 323(6088):533–536, 1986. (Cited on pages 59 and 61.)

B. RUSSELL. *The problems of philosophy (Chapter 10)*. OUP Oxford, 2001. (Cited on page 110.)

M. P. RYYNÄNEN and A. P. KLAPURI. Automatic transcription of melody, bass line, and chords in polyphonic music. *Computer Music Journal*, 32(3):72–86, 2008. (Cited on page 48.)

L. SALAIS, P. ARIAS, C. LE MOINE, V. ROSI, Y. TEYTAUT, N. OBIN, and A. ROEBEL. Production strategies of vocal attitudes. In *Interspeech 2022*, pages 4985–4989. ISCA, 2022. (Cited on pages 22, 31, 32, 133, and 152.)

M. SCHEDL, E. GÓMEZ, J. URBANO, et al. Music information retrieval: Recent developments and applications. *Foundations and Trends in Information Retrieval*, 8(2-3):127–261, 2014. (Cited on page 42.)

D. SCHMANDT-BESSERAT. The evolution of writing. *International Encyclopedia of Social and Behavioral Sciences*, pages 1–15, 2014. (Cited on page 28.)

K. SCHULZE-FORSTER, C. S. DOIRE, G. RICHARD, and R. BADEAU. Joint phoneme alignment and text-informed speech separation on highly corrupted speech. In *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 7274–7278. IEEE, 2020. (Cited on pages 70 and 115.)

K. SCHULZE-FORSTER, C. S. DOIRE, G. RICHARD, and R. BADEAU. Phoneme level lyrics alignment and text-informed singing voice separation. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 29:2382–2395, 2021. (Cited on pages 71, 81, 83, 85, 90, 145, and 153.)

M. SCHUSTER and K. K. PALIWAL. Bidirectional recurrent neural networks. *IEEE transactions on Signal Processing*, 45(11):2673–2681, 1997. (Cited on page 61.)

J. SERRA and E. GÓMEZ. A cover song identification system based on sequences of tonal descriptors. *Music Information Retrieval Evaluation eXchange (MIREX)*, 46, 2007. (Cited on page 67.)

C. E. SHANNON. A mathematical theory of communication. *The Bell system technical journal*, 27(3):379–423, 1948. (Cited on page 25.)
R. N. SHEPARD. Geometrical approximations to the structure of musical pitch. *Psychological review*, 89(4):305, 1982. (Cited on page 37.)

F. SIMONETTA, S. NTALAMPIRAS, and F. AVANZINI. Audio-to-score alignment using deep automatic music transcription. In *2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP)*, pages 1–6. IEEE, 2021. (Cited on page 50.)

B. SISMAN, J. YAMAGISHI, S. KING, and H. LI. An overview of voice conversion and its challenges: From statistical modeling to deep learning. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 29:132–157, 2020. (Cited on page 41.)

Y. SOULLARD, C. RUFFINO, and T. PAQUET. Ctcmodel: a keras model for connectionist temporal classification. Research report, Université de Rouen Normandie, 2019. hal-02420358. (Cited on page 114.)

N. SRIVASTAVA, G. HINTON, A. KRIZHEVSKY, I. SUTSKEVER, and R. SALAKHUTDINOV. Dropout: a simple way to prevent neural networks from overfitting. *The journal of machine learning research*, 15(1):1929–1958, 2014. (Cited on page 60.)

S. SRIVASTAVA, A. VERMA, and S. SHARMA. Optical character recognition techniques: A review. In *2022 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS)*, pages 1–6. IEEE, 2022. (Cited on page 47.)

F. STAHLBERG. Neural machine translation: A review. *Journal of Artificial Intelligence Research*, 69:343–418, 2020. (Cited on page 47.)

K. N. STEVENS. *Acoustic phonetics*, volume 30. MIT press, 2000. (Cited on page 30.)

S. S. STEVENS, J. VOLKMANN, and E. B. NEWMAN. A scale for the measurement of the psychological magnitude pitch. *The journal of the acoustical society of America*, 8(3):185–190, 1937. (Cited on page 37.)

D. STOLLER, S. EWERT, and S. DIXON. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. *International Society for Music Information Retrieval (ISMIR)*, 2018. (Cited on pages 71 and 145.)

D. STOLLER, S. DURAND, and S. EWERT. End-to-end lyrics alignment for polyphonic music using an audio-to-character recognition model. In *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 181–185. IEEE, 2019. (Cited on pages 71, 80, 85, 111, 145, and 153.)

B. H. STORY and K. BUNTON. Vowel space density as an indicator of speech performance. *The Journal of the Acoustical Society of America*, 141(5):EL458–EL464, 2017. (Cited on page 134.)
H. R. STRAYER. From neumes to notes: The evolution of music notation. 2013. (Cited on page 28.)

J. SUNDBERG. Articulatory interpretation of the “singing formant”. *The Journal of the Acoustical Society of America*, 55(4):838–844, 1974. (Cited on pages 31 and 136.)

J. SUNDBERG. *The Science of the Singing Voice*. Northern Illinois University Press, 1987. (Cited on page 138.)

J. SUNDBERG and T. D. ROSSING. The science of singing voice, 1990. (Cited on pages 30 and 40.)

H. TACHIBANA, K. UENOYAMA, and S. AIHARA. Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention. In *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 4784–4788. IEEE, 2018. (Cited on page 99.)

J. TAO, Y. KANG, and A. Li. Prosody conversion from neutral speech to emotional speech. *IEEE transactions on Audio, Speech, and Language processing*, 14(4):1145–1154, 2006. (Cited on page 41.)

Y. TEYTAUT and A. ROEBEL. Phoneme-to-audio alignment with recurrent neural networks for speaking and singing voice. In *Proceedings of Interspeech 2021*, pages 61–65. International Speech Communication Association; ISCA, 2021. (Cited on pages 22, 83, 91, 92, and 121.)

Y. TEYTAUT, B. BOUVIER, and A. ROEBEL. A study on constraining connectionist temporal classification for temporal audio alignment. In *Interspeech 2022*, pages 5015–5019. ISCA, 2022. (Cited on pages 22, 84, 91, 92, 96, and 99.)

Y. TEYTAUT, A. PETIT, C. CHABOT-CANET, and A. ROEBEL. A musicological pipeline for singing voice style analysis with neural voice processing and alignment. In *Journées d’Informatique Musicale (JIM 2023)*, Saint-Denis, 2023. (Cited on pages 22, 135, and 139.)

S. S. TIRUMALA, S. R. SHAHAMIRI, and R. GARHWAL, Abhimanyu Singh and WANG. Speaker identification features extraction methods: A systematic review. *Expert Systems with Applications*, 90:250–271, 2017. (Cited on page 42.)

A. TORRES, A. CABADA, and J. J. NIETO. An exact formula for the number of alignments between two dna sequences. *DNA Sequence*, 14(6):427–430, 2003. (Cited on page 65.)

C. TRALIE and E. DEMPSEY. Exact, parallelizable dynamic time warping alignment with linear memory. *International Society for Music Information Retrieval (ISMIR)*, 2020. (Cited on page 144.)
A. VAGLIO, R. HENNEQUIN, M. MOUSSALLAM, G. RICHARD, and F. D'ALCHÉ-BUC. Multilingual lyrics-to-audio alignment. In *International Society for Music Information Retrieval (ISMIR)*, 2020a. (Cited on pages 71, 82, 85, 113, 115, and 153.)

A. VAGLIO, R. HENNEQUIN, M. MOUSSALLAM, G. RICHARD, and F. D'ALCHÉ-BUC. Audio-based detection of explicit content in music. In *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 526–530. IEEE, 2020b. (Cited on pages 71 and 145.)

A. VAGLIO, R. HENNEQUIN, M. MOUSSALLAM, and G. RICHARD. The words remain the same: Cover detection with lyrics transcription. In *22nd International Society for Music Information Retrieval Conference ISMIR 2021*, 2021. (Cited on page 71.)

E. VARIANI, MCDERMOTT Erik, LAHOUEL Kamel, BACCHIANI Michiel, and BAGBY Tom. Sampled connectionist temporal classification. In *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 4959–4963, 2018. (Cited on page 91.)

A. VASWANI, N. SHAZEER, N. PARMAR, J. USZKOREIT, L. JONES, A. N. GOMEZ, Ł. KAISER, and I. POLOSUKHIN. Attention is all you need. *Advances in neural information processing systems*, 30, 2017. (Cited on pages 61, 83, and 84.)

V. VELICHKO and N. ZAGORUYKO. Automatic recognition of 200 words. *International Journal of Man-Machine Studies*, 2(3):223–234, 1970. (Cited on pages 48 and 66.)

B. VERCOE. The synthetic performer in the context of live performance. In *Proceedings of International Computer Music Conference*, pages 199–200, 1984. (Cited on page 67.)

E. VINCENT, R. GRIBONVAL, and C. FÉVOTTE. Performance measurement in blind audio source separation. *IEEE transactions on audio, speech, and language processing*, 14(4):1462–1469, 2006. (Cited on pages 93 and 137.)

E. VINCENT, T. VIRTANEN, and S. GANNOT. *Audio source separation and speech enhancement*. John Wiley & Sons, 2018. (Cited on page 93.)

T. K. VINTSYUK. Speech discrimination by dynamic programming. *Cybernetics*, 4(1):52–57, 1968. (Cited on page 66.)

A. VITERBI. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. *IEEE transactions on Information Theory*, 13(2):260–269, 1967. (Cited on pages 49 and 66.)

S. VOGEL, H. NEY, and C. TILLMANN. Hmm-based word alignment in statistical translation. In *COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics*, 1996. (Cited on page 67.)
W. VON KEMPELEN. *Mechanismus der menschlichen Sprache nebst der Beschreibung seiner sprechenden Maschine*. Bei JV Degen, Wien, 1791. (Cited on page 41.)

A. VOULODIMOS, N. DOULAMIS, A. DOULAMIS, E. PROTOPAPADAKIS, et al. Deep learning for computer vision: A brief review. *Computational intelligence and neuroscience*, 2018, 2018. (Cited on page 55.)

B. ŘEPOVÁ, M. ZÁBRODSKÝ, J. PLZAK, D. KALFERT, J. MATOUŠEK, and J. BETKA. Text-to-speech synthesis as an alternative communication means after total laryngectomy. 2021. (Cited on page 42.)

M. WAGNER. Automatic labelling of continuous speech with a given phonetic transcription using dynamic programming algorithms. In *ICASSP '81. IEEE International Conference on Acoustics, Speech, and Signal Processing*, volume 6, pages 1156-1159. IEEE, 1981. (Cited on page 67.)

S. WATANABE, T. HORI, S. KIM, J. R. HERSHEY, and T. HAYASHI. Hybrid ctc/attention architecture for end-to-end speech recognition. *IEEE Journal of Selected Topics in Signal Processing*, 11(8):1240-1253, 2017. (Cited on pages 71 and 90.)

C. WEISS and G. PEETERS. Training deep pitch-class representations with a multi-label ctc loss. In *International Society for Music Information Retrieval (ISMIR)*, 2021. (Cited on pages 122 and 145.)

A. WICHMANN. The attitudinal effects of prosody, and how they relate to emotion. In *ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion*, 2000. (Cited on pages 32 and 133.)

Y. WU and T. S. HUANG. Vision-based gesture recognition: A review. *Gesture-Based Communication in Human-Computer Interaction: International Gesture Workshop, GW'99 Gif-sur-Yvette, France, March 17-19, 1999 Proceedings*, pages 103-115, 1999. (Cited on page 47.)

X. XIA, X. SONG, F. LUAN, J. ZHENG, Z. CHEN, and X. MA. Discriminative feature selection for on-line signature verification. *Pattern Recognition*, 74:422-433, 2018. (Cited on page 50.)

A. YCART, A. MCLEOD, E. BENETOS, K. YOSHII, et al. Blending acoustic and language model predictions for automatic music transcription. 2019. (Cited on page 146.)

Y. ZHANG, M. PEZESHKI, P. BRAKEL, S. ZHANG, C. L. Y. BENGIO, and A. COURVILLE. Towards end-to-end speech recognition with deep convolutional neural networks. *arXiv preprint arXiv:1701.02720*, 2017. (Cited on page 71.)
J. ZHU, C. ZHANG, and D. JURGENS. Phone-to-audio alignment without text: A semi-supervised approach. In *International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 8167–8171. IEEE, 2022. (Cited on page 146.)

X. J. ZHU. Semi-supervised learning literature survey. 2005. (Cited on page 58.)

V. ZUE, S. SENEFF, and J. GLASS. Speech database development at mit: Timit and beyond. *Speech communication*, 9(4):351–356, 1990. (Cited on pages 40 and 109.)