Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis

Authors: taras  kucherenko
rajmund  nagy
youngwoo  yoon
jieyeon  woo
teodor  nikolov
mihail  tsakov
pieter  wolfert
teodor  viegas
mihail  nikolov
gustav  tsakov
  eje
matthew  le
apoorv  vyas
bowen  shi
brian  karrer
  leda
rashel  sari
mary  moritz
vimal  williamson
  manohar
naihan  li
shujie  liu
yanqing  liu
sheng  zhao
yaron  lipman
ricky t q chen
heli  ben-hamu
alexander h liu
jeff  cheng-i
wei-ning  lai
  hsu
zhengqing  li
you  zhou
elif  bozkurt
bo  zheng
Venue: thecvf_com_CVPR_2024_Workshop_HuMoGen
Published: 2024-03-24 02:45:05

Abstract: Although humans engaged in face-to-face conversation simultaneously communicate both verbally and non-verbally, methods for joint and unified synthesis of speech audio and co-speech 3D gesture motion from text are a new and emerging field. These technologies hold great promise for more human-like, efficient, expressive, and robust synthetic communication, but are currently held back by the lack of suitably large datasets, as existing methods are trained on parallel data from all constituent modalities. Inspired by student-teacher methods, we propose a straightforward solution to the data shortage, by simply synthesising additional training material. Specifically, we use unimodal synthesis models trained on large datasets to create multimodal (but synthetic) parallel training data, and then pre-train a joint synthesis model on that material. In addition, we propose a new synthesis architecture that adds better and more controllable prosody modelling to the state-of-the-art method in the field. Our results confirm that pre-training on large amounts of synthetic data improves the quality of both the speech and the motion synthesised by the multimodal model, with the proposed architecture yielding further benefits when pre-trained on the synthetic data.

Main paper: 
 introduction: 
 Human beings are embodied, and we use a wide gamut of the expressions afforded by our bodies to communicate. In concert with the lexical and non-lexical (prosodic) components of speech, humans also leverage gestures realised by face, head, arm, finger, and body motion -all driven by a shared, underlying communicative intent [58] -to improve face-to-face communication [30, 66] .Research into automatically recreating different kinds of human communicative behaviour, whether it be speech audio from text [85] , or gesture motion from speech [92] , have a long history, as these are key enabling technologies for, e.g., virtual agents, game characters, and social robots Fine-tuned LLMs allow generating of diverse text samples for many domains through prompting the model, i.e., providing a written text prompt at runtime describing the output to generate. Prompting has been useful for many tasks including creating synthetic dialogue datasets [1] and selecting appropriate gestures based on verbal utterances [28] . We use this ability to create an arbitrarily large material of conversational text sentences in the style of a given speaker/corpus as a basis for our synthetic-data creation. 
speech synthesis: 
 Recent advancements in deep generative modelling have significantly improved text-to-speech (TTS) [83] , achieving levels of naturalness that rival recorded human speech [80, 84] . TTS approaches are primarily divided into two broad classes: autoregressive (AR) and non-autoregressive (NAR)architectures. AR architectures produce acoustic outputs sequentially, using mechanisms such as neural cross-attention [10, 15, 50, 79, 110] or neural transducers [59, 60, 101] to connect inputs symbols to the outputs. Conversely, nonautoregressive models [25, 36, 37, 48, 63, 69, 75, 112] generate the entire utterance in parallel. The NAR approach is typically faster, especially on GPUs, but AR methods (which invest more computation into synthesis) often have the edge in synthesis quality.Recently, there has been a trend [10, 12, 15, 46, 93] to quantise audio waveforms into discrete tokens [16, 46] ,and then adapt an LLM-like autoregressive approach (e.g., with GPTs) to learn to model these audio tokens on large datasets. Synthesised token sequences can subsequently be converted back to audio [81] . Speaker and style adapta- In NAR TTS, it has been found that conditioning the TTS on the output of a model of prosodic properties, e.g., perphone pitch and energy, can benefit synthesis [67, 75, 112] .This furthermore affords control over speech output by replacing or manipulating the prosodic features prior to synthesis. Especially important for convincing prosody are the durations of the synthesised speech sounds. It has been oped for read-aloud speech, whereas co-speech gesturing is more closely associated with conversational settings.Joint synthesis of speech and motion was first considered by [78] . The first neural model was DurIAN [106] ,which simultaneously generated speech audio and 3D facial expressions, albeit for speech read aloud.[6] trained separate deep-learning TTS and speech-to-gesture systems to synthesise speech and 3D motion for the same speaker and the same (spontaneous) speaking style. This was followed by [94] , which investigated adapting and extending AR [79] and NAR [36] neural TTS models to perform joint multimodal synthesis. put at all.) The results in [62] show that, e.g., the synthetic speech falls short of human-level naturalness, and the quality we find from systems trained on very large datasets. Accordingly, we propose to circumvent the data limitation by using strong unimodal synthesisers to create a large synthetic training corpus for our joint model.The idea of training deep neural models on the output of other such models has an extensive history. This was originally proposed for classifiers [29] , but has subsequently been adapted to generative models, e.g., for TTS [89] . Synthesis (and synthetic data) is also appealing in scenarios where real data is scarce or difficult to obtain, as demonstrated in applications to human poses and motion [90, 108] .It also allows for the creation of diverse and controlled datasets that can enable more accurate and versatile models [35] . We here propose to generalise such approaches by chaining together multiple unimodal synthesisers, to enable training multimodal speech-and-gesture models. We provide more detail in the following subsections. We utilised the above procedure to generate a total of 600 phrases, each approximately 250 characters in length. We found that limiting the length of the prompt helps prevent issues with the subsequent speech synthesis, which shows a tendency to produce unintelligible or confabulated output when processing overly long utterances. The 600 generated phrases will be shared in future revisions of the paper. 
speech generation: 
 The next step was to synthesise speech audio from the 600 LLM-generated phrases. For this, we considered multiple TTS systems capable of multi-speaker and spontaneous speech synthesis, including Bark In order to obtain more diverse data containing multiple speakers, each of the 600 phrases was synthesised 16 times, once in each of 16 different voices. These voices were selected as a gender-balanced set (8 male and 8 female speakers) from the VCTK corpus [97] , and elicited from XTTS by seeding the synthesis of each individual utterance with the audio of longest VCTK utterance spoken by the relevant speaker as an acoustic prompt. These prompting utterances tended to be around 9 seconds long. In total, we thus synthesised 16 ⇥ 600 = 9600 audio utterances.Interestingly, despite the spontaneous nature of the input phrases, we found that false starts and fillers explicitly present in the input were sometimes omitted in the XTTS output. This could be partly due to the choice of temperature parameter at synthesis time (the default, 0.65), which favours more consistent and likely output, and partly due to the public English-language training datasets cover read rather than spontaneous speech. Since XTTS furthermore was prompted using a snippet of read-aloud speech audio Since this paper is focused on multimodal synthesis from data where no interlocutor is present or recorded (i.e., not back-and-forth conversations), interlocutor-related inputs were removed from the architecture. The input is thus an audio track with time-aligned text transcripts. We used the pre-trained weights from [17] for the CSMP module and re- Compared to conventional conditioning approaches where audio is represented using mel-spectrograms, the speakerindependent data2vec embeddings in the CSMP module are expected to better handle the differences between natural and synthetic voices during synthesis, thus making it feasible to generate large amounts of gesture data based on synthetic speech without undue degradations due to domain mismatch. This data was used to train the different multimodal synthesis systems considered in our experiments.The current state of the art in joint speech-and-gesture synthesis is Match-TTSG [62] , a non-autoregressive model which uses conditional flow matching (OT-CFM) [51] to learn Ordinary Differential Equations (ODEs) with more linear vector fields than continuous-time diffusion models[82] create. Such simpler vector fields offer advantages for easier learning and faster synthesis.We extend the Match-TTSG framework in three ways:1. Probabilistic instead of deterministic duration modelling, which can benefit deep generative NAR TTS [37] .2. Additional prosody-prediction modules, which are widely used in NAR TTS [75, 112] . Finally, we added a speaker embedding for multispeaker synthesis. Specifically, we used a one-hot speaker vector to represent the 16 different speakers in the synthetic training data. This vector was concatenated to other inputs at multiple stages of the synthesis process, including the text encoder, prosody predictors and decoder. The idea with this was to minimise information loss and ensure coherent output across different speaker identities. Since the concatenated vectors only have 16 elements, the impact on model parameter count is small (an increase of a few thousand). 
experiments: 
 This section experimentally compares our proposed training method and architecture with the previous state-of-the-art method Match-TTSG [62] . Since this is a synthesis work, the gold standard approach to evaluation -and thus the focus of our experimental validation -is subjective user studies. The experiments closely follows those in previous joint synthesis works [61, 62] , which in turn follows established practices in speech [32] and gesture evaluation [44].To test the effectiveness of our method we carried out 3 different subjective evaluations with systems trained on Trinity Speech-Gesture Dataset II (TSGD2) [22] , a dataset containing 6 hours of multimodal data: recordings of time-aligned 44.1 kHz audio coupled with 120 FPS marker-based 3D motion capture, in which a male native speaker of Hiberno-English discusses a variety of topics whilst gesturing freely.The same train-test split of the data was used as in [61] , with around 4.5 hours of training data -much less than the 38 hours of synthetic multimodal data we created.We trained Match-TTSG (MAT) containing 30.2M parameters, and MAGI (MAGI) containing 31.6M parameters for 300k steps on only the TSGD2 data, we refer to these conditions MAT-T and MAGI-T respectively. We also took proach, drawing inspiration from the Blizzard Challenge for text-to-speech systems [70] . Participants were asked, "How natural does the synthesized speech sound?", rating their responses on a scale from 1 to 5, where 1 represented "Completely unnatural" and 5 indicated "Completely natural." The intermediary values of 2 to 4 were provided without textual descriptions. Each participant evaluated 15 stimuli per system and 4 attention checks resulting in a total of 525 responses per condition by 35 participants. Fine-tuning with synthetic data led to performance enhancements for both MAGI and MAT, reducing the WER from 13.28% in MAGI-T to 9.29% in MAGI-FT, and from 12.26% in MAT-T to 8.35% in MAT-FT.We evaluate motion quality using video stimuli that only visualised motion, without any audio, in order to have an independent assessment of motion quality. This ensures that ratings are not affected by speech and follows the practice of recent evaluations of gesture quality [33, 74] . Similarly to the speech evaluation, participants were asked "How natural and humanlike the gesture motion appear?", and gave responses on a scale of 1 ("Completely unnatural") to 5("Completely natural"). The number of stimuli and attention checks were identical to the speech-only evaluation.We finally evaluated how appropriate the generated speech and motion were for each other, whilst controlling for the effect of their individual quality following [33, 45, 62, 74, 105] . For each speech segment and condition, we created two video stimuli: one with the original video and sound, and the other combining the original speech audio with motion from a different video clip, adjusting the motion speed to align with the audio duration. Both videos feature comparable motion quality and characteristics from the same condition, but only one video's motion is synchronised with the audio track, without indicating which video is which.The test inquired which character's motion most accurately matched the speech in rhythm, intonation, and meaning. Participant ability to identify the correctly synchronised video indicates a strong rhythmic and/or semantic link between generated motion and speech. Following [61] we opted for five response choices instead of the typical three for better resolution. Options were "Left is much better", "Left is slightly better","Both are equal", "Right is slightly better", "Right is much better". For the purposes of analysis, codes in the range of 2 to 2 were assigned to each response, as in [61] , with 2 representing the participant's preference for the mismatched stimulus and 2 the matched stimulus. Participants reviewed motions from 14 of the 15 segments, displayed as 7 screens of pairs of videos, plus and the comparison against a robust baseline without alterations that directly influence cross-modal synthesis aspects.Lastly, the accuracy of capturing cross-modal aspects might be least represented in synthetic datasets created from unimodal synthesizers trained on non-cohesive data. 
pitch and energy control: 
 As stated, the proposed multi-stage architecture with separate prosody predictors allows for modifying or substituting the pitch and energy contours before synthesis. This enables direct control of prosodic properties of the speech, with the synthesis process having the option to adjust the gestures to match. On our anonymous webpage cvprhumo-gen24.github.io/MAGI we provide example videos showing the effect that modifying (scaling) the pitch and energy contours returned by the predictors has on the synthesised output. One can observe that reducing the pitch seems to promote creaky voice, which makes sense from a speechproduction perspective and fits earlier findings from autoregressive TTS on spontaneous-speech data [47]. 
conclusion and future work: 
 We have described improvements to the joint and simultaneous multimodal synthesis of speech audio and 3D ges-  
 
 Appendix: https://github.com/suno-ai/barkhttps://elevenlabs.io/ 


Decision: The paper addresses the task of multimodal synthesis of speech and gesture from text.

Pros:
* Well written
* Simple yet effective method
* Quantitative and Qualitative results outperform current work

Cons:
* Limited novelty

**Guidance to authors:** Resolve concerns raised by the authors, particularly the ones related to language.

Reviews: review: title: title: Elegantly increasing the amount of labeled data to obtain improved results, but the novelty is limited.
main_review: main_review: This paper uses unimodal synthesis models to create multimodal synthetic parallel training data. Then it trains a joint synthesis model, that outperforms prior art.

Pros:

- Mitigates the shortage in labeled data.
- The authors present a simple and elegant pipeline. Simplicity IS an advantage.
- Experiments demonstrate both the effect of scaling data size and the effect of enhancements in Match-TTSG [62].
- Results outperform prior works, both quantitatively and qualitatively.
- Manuscript is written in a clear and well organized way.

Cons:

- I am afraid the novelty is Incremental: The authors use two existing works to create the synthetic data, with minor changes only. Creating synthetic data is also not a novel idea. The final network that predicts multimodal output is an improved version of Match-TTSG [62], where the improved parts are based on existing ideas..
- Language should be somewhat improved. For example, in several places, the word ‘train’ is more appropriate than the word ‘pretrain’. If this work is accepted, please review the manuscript for any linguistic errors and make necessary corrections.
 

Altogether, although this work uses existing building blocks, it combines them in an elegant way that induces improved results. Hence, I am inclined for a weak acceptance.
title: title: An interesting exploration of alleviating Data Shortage in speech-and-gesture synthesis through synthetic data generation. But more convincing experiments for the synthetic data's effectiveness are needed. Tending to a weak acceptance.
main_review: main_review: Data Shortage is a common challenge in speech-and-gesture synthesis. This paper attempts to construct a synthetic dataset and explores its utilization to enhance an existing pre-trained speech-and-gesture model. The clarity and structure of this paper facilitate ease of understanding. The authors employ a subjective evaluation mechanism, which is a standard method in this field.

**Pros**
* Combining multiple large-scale models (GPT-4, XTTS, and Whisper) with a pre-trained speech-and-gesture model to generate synthetic data.
* Adapting concepts from the audio model field to the design of motion models, potentially inspiring future research.

**Cons**
* The evaluation of the synthetic data's effectiveness is not convincing. The model used for synthetic motion generation is trained on the TalkingWithHands (TWH) dataset, implying that the motion distribution is similar to that of the TWH dataset. The MAT-FT/MAGI-FT setup—pre-trained on synthetic data and then fine-tuned using TSGD2—is intuitively equivalent to training on a combination of TWH and TSGD2 datasets. It is straightforward that the performance of this setting is better than MAT-T/MAGI-T that trained on only the TSGD2. But it is hard to prove the effectiveness of the synthetic dataset. More fair and further experiments need to be conducted, such as:
    * Establishing a baseline trained on both TWH and TSGD2 datasets.
    * Utilizing synthetic data to fine-tune the pre-trained model described in Section 3.1.4. If the performance of the model is improved, it indicates that synthetic data works and perhaps enhances the diversity of the original data. This setting is common in NLP.

**Typo**
* Lines 1231-1251: The first letter of each bullet point varies in case.

This paper examines the increasingly important issue of Data Shortage in the data-driven domain of speech-and-gesture synthesis. While the discussion is intriguing, it needs more convincing experiments for support. Consequently, my inclination is towards a weak acceptance.