# A Linear Memory CTC-Based Algorithm for Text-to-Voice Alignment of Very Long Audio Recordings
## 5 Conclusions  
Popular applications such as automatic closed-captioning or karaoke generally come to mind when mentioning text-to-voice synchronization. However, it also plays an increasingly important role in the academic field by generating an automatically large corpora of paired voice and text for downstream tasks, such as voice synthesis or speaker identity change. Text-to-voice synchronization algorithms typically consist of two steps: an encoding step, where a time series of symbols is inferred from the audio, and a forced-alignment step, where these symbol timestamps are assigned to the ground-truth text. We presented in this work a novel algorithm, where the encoding module is implemented as a fully CTC-based convolutional network, and where the forced-alignment module is implemented as a CTC decoder with linear memory complexity. We demonstrated that this approach is able to align the text to several hours of audio with a mean alignment error of 50 ms for speech, and 120 ms for the singing voice. We also demonstrated that the system is robust for increasing audio durations, as well as severe ground-truth text alterations, such as character and word insertions or deletions. We interpreted these performances by the fact that the system only requires a minimal amount of correct characters that serve as anchors to align the entire text correctly. This also explains that the system, albeit trained solely in English text, obtains a similar accuracy with other languages for speech and, to a lesser extent, with the singing voice. In the future, we plan to speed up the algorithm by adapting the forced-alignment step for GPU parallelization. We also plan to generate and publicly release a large corpora of synchronized text and voice for future research. \begin{table}
\begin{tabular}{l c c c c} \hline \multicolumn{5}{c}{**[PERSON] (_a Cappella_)**} & \multicolumn{2}{c}{**[PERSON]**} \\ \hline
**System** & **MAAE [ms]** & **Q50\% [ms]** & **MAAE [ms]** & **Q50\% [ms]** \\ \hline [PERSON] et al. [66]\({}^{\ddagger}\) & **87** & **32** & **217** & **46** \\ [PERSON] et al. [67]\({}^{\ddagger}\) & 110 & **32** & 610 & 60 \\ [PERSON] et al. [68] & 930 & 946 & 500 & 85 \\ \hline Ours & 117 & 44 & 310 & **46** \\ \hline \end{tabular}
\end{table} 
Table 6: Lyrics’ alignment MAAE (ms) on the [PERSON] (a cappella variant) and Jamendo datasets and comparison to the latest alignment systems (MIREX 2020). \({}^{\ddagger}\) We indicate best performing version on each dataset, as reported on MIREX 2020 results ([[https://www.music-ir.org/mirex/wiki/2020:Automatic_Lyrics-to-Audio_Alignment_Results](https://www.music-ir.org/mirex/wiki/2020:Automatic_Lyrics-to-Audio_Alignment_Results)]([https://www.music-ir.org/mirex/wiki/2020:Automatic_Lyrics-to-Audio_Alignment_Results](https://www.music-ir.org/mirex/wiki/2020:Automatic_Lyrics-to-Audio_Alignment_Results)) (accessed on 31 January 2023)). **Author Contributions:** Conceptualization, G.D., Y.T. and A.R. ; methodology, G.D.; software, G.D. and Y.T. ; validation, G.D., Y.T. and A.R. ; formal analysis, G.D.; investigation, G.D.; resources, A.R. ; data curation, G.D. and Y.T. ; writing--original draft preparation, G.D. and Y.T. ; writing--review and editing, G.D., Y.T. and A.R. ; visualization, G.D. and Y.T. ; supervision, A.R. ; project administration, A.R. ; funding acquisition, A.R. All authors have read and agreed to the published version of the manuscript. **Funding:** This research was partly funded by the French National Research Agency (Agence Nationale de la Recherche--ANR) as part of the **ARS** project (ANR-19-CE38-0001-01). **Institutional Review Board Statement:** Not applicable. **Informed Consent Statement:** Not applicable. **Data Availability Statement:** The Librispeech dataset can be found at [[https://www.openslr.org/12](https://www.openslr.org/12)]([https://www.openslr.org/12](https://www.openslr.org/12)) (accessed on 31 January 2023).