Fugatto 1: Foundational Generative Audio Transformer Opus 1
Authors: Rafael Valle
Rohan Badlani
Zhifeng Kong
Sang-gil Lee
Arushi Goel
Sungwon Kim
Joao Felipe Santos
Shuqi Dai
Siddharth Gururani
Aya Aljafari
Alexander H. Liu
Kevin J. Shih
Ryan Prenger
Wei Ping
Chao-Han Huck Yang
Bryan Catanzaro
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Models
Audio
Foundation Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Fugatto: Foundational Generative Audio Transformer Opus 1 with ComposableART - an inference-time technique that extends classifier-free guidance into compositional guidance.

Abstract: Fugatto is a versatile audio synthesis and transformation model capable of following free-form text instructions with optional audio inputs. While large language models (LLMs) trained with text on a simple next-token prediction objective can learn to infer instructions directly from the data, models trained solely on audio data lack this capacity. This is because audio data does not inherently contain the instructions that were used to generate it. To overcome this challenge, we introduce a specialized dataset generation approach optimized for producing a wide range of audio generation and transformation tasks, ensuring the data reveals meaningful relationships between audio and language. Another challenge lies in achieving compositional abilities -- such as combining, interpolating between, or negating instructions -- using data alone. To address it, we propose ComposableART, an inference-time technique that extends classifier-free guidance to compositional guidance. It enables the seamless and flexible composition of instructions, leading to highly customizable audio outputs outside the training distribution. Our evaluations across a diverse set of tasks demonstrate that Fugatto performs competitively with specialized models, while ComposableART enhances its sonic palette and control over synthesis. Most notably, we highlight our framework's ability to execute emergent sounds and tasks -- sonic phenomena that transcend conventional audio generation -- unlocking new creative possibilities. \href{https://fugatto.github.io/}{Demo Website.}

Published as a conference paper at ICLR 2025
Fugatto 1
Foundational Generative Audio Transformer Opus
NVIDIA
Rafael Valle, Rohan Badlani, Zhifeng Kong, Sang-gil Lee, Arushi Goel, Sungwon Kim, João Felipe Santos, Shuqi Dai, Siddharth Gururani, Aya AlJa fari, Alexander H. Liu, Kevin Shih, Ryan Prenger, Wei Ping, Chao-Han Huck Yang. Bryan Catanzaro ra
ABSTRACT
Fugatto is versatile audio synthesis and transformation model capable of follow- ing free-form text instructions with optional audio inputs. While large language models (LLMs) trained with text on simple next-token prediction objective learn to infer instructions directly from the data, models trained solely on audio data lack this capacity. This is because audio data does not inherently contain the instructions that were used to generate it. To overcome this challenge, we introduce specialized dataset generation approach optimized for producing wide range of audio generation and transformation tasks, ensuring the data reveals meaningful relationships between audio and language. Another challenge lies in achieving compositional abilities such as combining, interpolating between, or negating instructions using data alone. To address it, we propose ComposableART, an inference-time technique that extends slassffirr-free guidance to compositional guidance. enables the seamless and flexible composition of instructions, leading to highly customizable audio outputs outside the training distribution. Our evalua- tions across diverse set tasks demonstrate that Fugalto performs competitively with specialized models, while ComposableART enhances its sonic palette and control over synthesis. Most notably, we highlight emergent tasks and properties that surface in our framework's s sonic phenomena that transcend conventional audio generation unlocking new creative possibilities. Demo Website.
INTRODUCTION
Recent research has sparked a strong debate between specialist and generalist models. While specialist models excel at specific tasks, they tend to be brittle, often struggling with changes in data distribution or task requirements. In contrast, generalist models eliminate the need for task-specific designs, can process diverse data, and scale effectively with increased compute and data. They also demonstrate emergent capabilities, enabling unsupervised task learning by leveraging broader datasets (Radford et al., 2019) and demonstrations (Brown et al., 2020). In this paper, we propose strategy for developing a generalist audio synthesis and transformation model, called Fugatto, and an inference method for composing instructions through latent space manipulation, including from different models, called Composable Audio Representation Transformation (ComposableART))
Large language models (LLMs) have demonstrated impressive unsupervised multitask learning capabilities in the text domain (Radford et al., 2019), where instructions can be inferred from the data itself. However, such instructions are typically absent the audio domain, making difficult to generalize to unseen tasks without explicit guidance. Although models such (Wang al., 2024; Yang et al., 2023; Vyas et al., 2023) exist, they have several limitations enumerated in Table In this panorama, dataset and instruction generation is necessary.
We employ a multifaceted data and instruction generation strategy that considerably expands the range of tasks of audio generation models. First, we use LLMs to generate and augment instructions and captions, providing Fugatto with instructions that are closer to free-form instructions (Goel et al., 2024; Doh et al., 2023). Second, we develop instructions that be either absolute (e.g..

---

Published as a conference paper at ICLR 2025
"synthesize a happy voice") relative (e.g., "increase the happiness of this voice"), enabling Fugatto to handle a wide array of dynamic tasks (OpenAI, 2024). Third, we leverage audio understanding models (Kong al., 2024a: Gong et al., 2023) to create descriptions and synthetic captions for audio clips, enriching the dataset where annotations are sparse, allowing for better generalization and more accurate performance (Kong al., 2024b). Fourth, we transmute existing datasets to uncover new relationships, enabling the creation of entirely new tasks without requiring additional raw data. Finally, we use audio processing tools to create new connections between text, audio, and their corresponding transformations.
By combining these approaches, we ensure that Fugatto has access diverse and enriched datasets, allowing it to learn from various audio domains and contexts. This strategy enhances the model's 's task diversity and lays the groundwork for unsupervised multitask learning at scale, unlocking emergent abilities such synthesizing entirely new sounds, such as "a saxophone barking".
While data important, the ability to compose, interpolate between, and negate instructions is generally difficult to obtain through data alone. Negation data, for instance, is typically unavailable, and generating outputs that represent the composition or interpolation of instructions equally challenging. Though this has been explored in the vision domain using Energy Based Models (EBMs) (Du et al., 2020) and EBMs in latent space (Nie et al., 2021). EBMs require training a new model per attribute, which can be cumbersome and impractical for large number of attributes.
To overcome this limitation, we propose an inference-time technique called ComposableART, which is based classifier-free guidance (CFG) (Ho & Salimans, 2022) and further expands the dual classifier-free guidance in (Yang et al., 2024; Lee et al., 2024). We propose a generalized framework that leverages the weighted combination of vector fields between instructions, frame indices and models. This approach enables Fugatto to handle complex instruction-based operations, such as smoothly interpolating between instructions or negating specific instructions exclude undesired effects. In contrast, models like (Vyas et al., 2023; Yang et al., 2023) rely on more rigid methods, often requiring external classifiers or manual intervention achieve similar results.
In this paper, we present a detailed exploration of Fugatto's dataset and instruction generation, training strategies. implementation improvements. and performance across a wide range of tasks. Through extensive evaluations and comparisons with specialist and generalist models, we establish Fugatto as a new benchmark for foundation models for audio synthesis and transformation. Similarly, we establish ComposableART as a highly desirable framework for compositional guidance that unlocks the full potential of score-based generative models. Our contributions include:
We offer a comprehensive strategy for building a foundation model for audio generation and transformation given text and audio inputs, delivering strong performance across wide range of tasks and providing a robust framework for both research and practical applications.
We demonstrate how to enhance and create contextually rich audio and text datasets while generating flexible instructions with LLMs, enabling our community to replicate and adapt these techniques for their own models.
We demonstrate how to perform composition, interpolation, and negation of instructions by extending CFG compositional guidance, enabling better control over the model's 's outputs.
Table 1: Comparison of our proposed model Fugatto with other models.
Fugatto
AudioBox
NEXT-GPT
UniAudio
AUDIT
Vnicel DMM
Emergent properties
Large-scale data
/
/
X
Supports numerous tasks
/
x
/
x
X
Free-Form Instructions
?
Open-ended generation
/
x
x
/
x
X
Compositionality
/
X
/
Mudii-Modal Inputs
2

---

Published as a conference paper at ICLR 2025
2 APPROACH
Our approach to audio generation given text and optional audio is similar to recent approaches in LLMs, focusing on large scale compute and datasets, followed by pre-training and fine-tume tie-ttning stages. Our approach differs in two aspects: first, our dataset generation mechanism requires going beyond unsupervised next token prediciton (Section 2.1); second, we propose inference time techniques to control audio generation (Section 2.4). Appendix A.1 provides list of tasks and instructions.
2.1 DATASET GENERATION
We aim to build large and diverse dataset capture demonstrations of a wide range of audio tasks and contexts. We emphasize that our ultimate goal is not to just excel on such tasks, but to drive, as a community, towards future where unsupervised multitask learning emerges from data and model scale. Towards this goal, we propose a dataset generation strategy built on the five pillars below, and provide in Appendix A.1.1 thorough description of each pillar.
Our strategy consists of prompting an LLM to create specifii instruction generators, similar to Kocielník al. (2023). We prompt an LLM to create python methods that generate instructions of different lengths, and for different personas (standard, young-crowd, somethss,, professional), given task-specific inputs including audio description, language, and others. Each persona has its own set of parts of speech.
I Generating Free-Form Instructions with LLMs:
II - Generating Absolute and Relative Instructions: to perform a relative change in speech, we aim to obtain audio generation model that is able to follow instructions that are absolute or relative, such as "synthesize a happy voice" or "increase the happiness in this voice.". Given that such data is normally not available, existing datasets and leveraging audio processing tools. Thus, we can select one sample and create an absolute instruction for it, select two samples and create a relative instruction. Similar to absolute instructions, relative instructions are produced by an LLM, that produces a python method that creates an instruction given the task, the attribute, and its modification.
Following GPT4-o's (OpenAI, 2024) ability
create it by transmuting
III - Generating Audio Descriptions with Audio Understanding Models: We use audio under- standing and classification models (Kong et al., 2024a) to produce synthetic captions for audio in the wild, following recent research (Leng et al., 2023; Kong et al., 2024b) showing that it is possible to drastically expand or improve text-to-audia models with synthetic captions. As such, we expand the strategies described in (Leng et al., 2023; Kong et al., 2024b) to generate high quality synthetic captions. For speech data, we implemented a prompt generation pipeline that automates the creation of natural language descriptions for voices. The pipeline converts speech attributes predicted by models - such as "gender", emotion, and speech quality - into detailed natural language descriptions using LLM-generated templates. These templates describe voices in various ways based on the speaker attributes, enhancing diversity by generating descriptions in multiple formats.
IV Creating New Tasks and Datasets by Transmuting Datasets: We leverage implicit relation- ships between samples in a dataset to enable new tasks. Generally speaking, we look for datasets where one factor is held constant while other factors change. For example, we leverage emotional speech synthesis datasets with different renditions of the same text (Livingstone & Russo, 2018) by the same speaker to define speech transformation task. Similarly, we leverage instrument synthesis datasets with different renditions of the same note (Engel et al., 2017) to define an instrument trans- formation task. We also leverage datasets that provide the individual parts of sounds mixture (Rafii et al., 2017) to support tasks such as source separation, and audio generation conditioned on audio context and captions, possibly synthetic.
V - Creating New Tasks and Datasets by Leveraging Audio Processing Tools: We create synthetic paired data for speech and audio by using Praat (Boersma & Van Heuven, 2001) and Pedalboard (Spotify, 2024) to manipulate several speech and audio factors. For each factor, we apply controlled modifications, allowing us to generate speech and audio samples with specific alterations. With this strategy, we create speech-audio pairs for transformation tasks, like adjusting speech factors e.g.. "increase the FO variance slightly and decrease the speech rate." or "add moderate reverb to this audio file". For each factor, we determine practical range of adjustments and define increments that correspond to varying degrees of change, such as "slightly", "moderate", and "significant".
3

---

Published as a conference paper at ICLR 2025
With these pillars established and leveraging open source datasets, able build large text and audio dataset with at least 20 million rows, not including on-the-fly modifications to captions, instructions and audio. Assuming each row refers to 10 seconds of audio, our dataset is comprised of at least 50,000 hours of audio. We provide a full list of datasets, tasks, and instructions in Appendices A.1.2, A.1.3 and A.1.4 respectively.
2.2 INSTRUCTION GENERATION
Our approach supports template-based and free-form instructions.
Template-basee based Instructions: In these instructions, the task is explicitly provided, followed by task-specifie attributes, always in the same order, and with each attribute wrapped between start and end of attribute markers. We dynamically construct template-basee and the set of factors. Each factor is specified by its name and corresponding value, using the format given {name}: }: : (value) followed by a closing HTML-like tag < </[factor]> The instruction always starts with the task, followed by its factors, and ends with output: :. Additionally, given ven context clause is appended at different locations, determined by the task at hand, when audio contexts are present. The structure of a template-basee based instruction is:
instructions based on the task
input: : ( (takk) given |factor): (value /atllee////faae htt///aatoorr// | context: <audio>output:
where [task} refers to the specific task, [factor] and (value) refers to the different factors and their respective values, and <audi o> refers to the audio context provided. We provide examples of task and dataset specific instructions in Appendix A.1.4.
Free-form Instructions: We dynamically construct free-form instructions by using instruction
generators introduced in Section 2.1. Unlike template-based instructions, where we know beforehand which part of the text refers to each audio context, free-form text makes it less straightforward to determine which words correspond to the audio. As such, in free-form instructions we simply append to the instruction given context.k: <audio>> for each audio context, resulting this structure:
input: ( (instruction) [given context: <audio -otttoo output:
In order to promote simplicity and agility during development, we decided to use raw text instead of learnable tokens for given factor and </factor>, Following LLM practice, the full instruction and each audio are wrapped with learnable <start of> and <end of> tokens.
2.3 MODEL AND TRAINING
In this section we describe the text and audio representations used in our model, as well as the training objective and architecture. We provide graphical depiction and details for hyperparameters, objective function, training stages and oversampling in A.2. In consonance with recent work in LLMs, we aim for a general model with weaker modeling assumptions, or inductive priors, versus model with several hand-crafted or modality specific assumptions.
The text representation is obtained by encoding the previously described instructions with a pre-trained language model held frozen during training. In this Opus, we use the byT5 tokenizer free language model (Xue et al., 2022), which supports large set of characters, including IPA. For now, the audio representation is a 22khz mel-spectrogram with 80 bins, which is subsumed by a relatively shallow learnable transformer encoder. The mel spectrogram is scaled to have approximately 0 mean and 1 standard deviation.
Text and Audio representation:
We train our model with the Optimal Transport Conditional Flow Matching (OT-CFM) objective (Lipman et al., 2022; Tong et al., 2023), and use T5-based (Raf- fel et al., 2020) Transformer (Vaswani, 2017) with Adaptive Layer Norm (Xu et al., 2019) as the parameterization of the vector field estimator. We replace the Transformer MLPs with kernel size 3 convolutions. After independently projecting the encoded text and audio to a shared embedding space, time-wise concatenate the embedded audio and text. The model cross-attends to this representation, applying Adaptive Layer Norm (Xu et al., 2019) to them on every layer.
Training Objective and Architecture:
We observed that certain implementation choices yielded better training curves. Specifically, adaptive layer norm is completely computed in FP32, GELU uses approximate tanh, and the final layers are initialized outputs zeros, which is approximately the mean of our scaled mel-distribution.
4

---

Published as a conference paper at ICLR 2025
Training Stages: Fugatto training follows curriculum learning' (Bengio et al., 2009). We start with complte-basedd instructions and a subset of tasks. Eventually, once we informally establish, by observing validation scores and listening to samples, that the model is able to follow template- based instructions, we proceed with an equal mixture of template-based and free-form instructions. Given that some tasks underrepresented in the data, find that oversampling leads to better results. Empirically, we find that starting sampling from multinomial distribution with upsampling parameter 0.25, similar to Le al., 2024, sufficient. As training progresses, we adjust each dataset's ss weight according to validation scores on target tasks.
2.4 COMPOSABLE AUDIO REPRESENTATION TRANSFORMATION (ComposableART)
We extend Classifier Free Guidance(CFG) (Ho & Salimans, 2022) to support compositional guidance. Compositional guidance provides an OT-CFM model with the ability to independently control and generate (unseen) combinations of instructions and tasks, including with vector fields from different models (Karras et al., 2024). Compositional generation has been explored in Diffusion Models (Liu et al., 2023; Yang et al., 2024; Lee al., 2024), for images and audio. To the best of our knowledge, we are the first to showcase novel ways of applying compositional guidance, expanding it to not just attributes, but also tasks, models and temporal composition of attributes.
Compositional Guidance Method Classifier Free Guidance(CFG), generally applied to diffusion models, combines the conditional and unconditional score estimates to obtain samples of higher quality and diversity pertaining to the condition. The following equation summarizes the application of CFG, where Eo represents the score estimate and y represents the gradient scaling factor:
c/ Ez(. c) co c) + (ce((z, c) ca(2)))
(1)
We extend the Classifier Free Guidance framework to support the combination of vector fields across multiple instructions (c2), multiple mel-frame indices (f) and multiple models (6m.). Let uir((k 6m) be the vector field produced at flow-step by model m, parameterized by 6m. given condition & Ø, for mel-frame f. Additionally, let Wkrm be the flow-step invariant and user-determined scalar weight associated with (9, 0m). The equation for compositional guidance across instructions, frames, and models defined as:
0r. Σ w.r.m W.r.m(er 0m) + (Vrs(er, 0m) vir(@0, 0m)))
where y refers the gradient scale parameter from CFG, and ö.r is the resulting composed vector field for flow step t and frame f. This follows similar conditional independence assumptions in (Nie et al., 2021; Liu al., 2023) to support compositional guidance. We apply this compositional guidance at every step of the flow-matching inference procedure.
Attribute//vent Composition: An attribute is simple input prompt belonging particular task, such as speech synthesis and audio event generation. A task can have multiple prompts attributes as input. With compositional guidance, we can support unique unseen combinations of attributes. This gives the users an ability to create artistic combinations like simulating scene with multiple-audio events, e.g. by composing 'thunder', 'rain' and 'wind' storm can be achieved.
The model has been trained on many individual tasks, but it has not encountered combinations of tasks, such as speech synthesis alongside audio event generation. Using composi- tional guidance, we can enable the synthesis of unique, unseen task combinations, like generating speech with specific audio event in the background.
Task Composition:
The same technique can be extended to integrate distinct models. This is particularly useful when training domain-specific versions of Fugatto, each with its own parameters and datasets. This allows users to synthesize "mixture of experts" sample that combines models trained on independent domains such as speech and audio events. Following (Karras et al., 2024), use the velocities for each independent model, defined by parameters 9m.
Model Composition:
It simplifies development and supports incremental research, though strictly necessary.
5

---

Published as a conference paper at ICLR 2025
Temporal Composition:
Instead of using the same scalar for every frame f, we assign unique weight Watm to each frame. This enables users to control the compositional output with arbitrary temporal curves (e.g., sigmoid linear increase decrease), while retaining the advantages of combining attributes, tasks, and models.
EXPERIMENTS
We present a comprehensive evaluation of Fugatto across multiple tasks to demonstrate its effective- ness and versatility. We begin with an ablation study, examining the impact of various design choices. Next, we evaluate Fugatto's performance in audio synthesis and transformation tasks in speech, music and general sounds. Finally, we explore Fugatto's emergent capabilities, and perform a thorough evaluation of our ComposableART method. Unless otherwise specified, we use template-based instructions, 2nd order Heun solver with 50 function evaluations, and task specific gradient scale 7.
3.1 ABLATIONS
We first analyze the impact of different I-sampling strategies OT-CFM, comparing the traditional uniform sampling with others. Then, we examine the effect of model size on both loss metrics and emergent capabilities, including the ability to synthesize novel sounds not found the training data-such as a "saxophone meowing" or "a person speaking while barking."
i-sampling strategy: In the OT-CFM framework, the timestep is typically sampled from a uni- form distribution, / ~ U(0,) 1). However, recent discussions within the community have introduced conflicting strategies for this sampling process. Notably, Stable Audio's ' GitHub repository proposes coamoodd-raansfomme
sampling from trating samples around f = 0.5. On the other hand, (Lovelace et al., 2023) advocate for increased sampling from values of t closer
normal distribution, t sigmoid(A(0) 1)). thereby concen-
In our experiments, we observe that although Stable Audio's ' strategy provides a marginal improve- ment on TTA tasks, it renders the model unable effectively attend to the transcript text-to-speech (TTS) tasks a critical requirement. We provide an explanation for this phenomenon in Appendix A.3.
Training with f ~ 21(0, 1) is effective across all tasks. Training with sigmoid 1)) significantly degrades TTS performance.
Model capacity: We evaluate how the learnable parameter count influences loss curves and emergent capabilities. We consider models with 0.8 B, 1.4 B params, and 2.5 B parameters. Under a fixed data composition and sampling weights, we observe that increasing the parameter count from the smallest to the largest model not only improves validation losses but also delays overfitting. In Appendix A.2, we provide task specific validation loss plots that showcase the expected decrease in validation loss as parameter count increases. Informally and consistent with findings in (Radford et al., 2019), we observe that the smaller model does not exhibit emergent abilities comparable the larger models, particularly in their ability to synthesize novel sounds absent from the training data, such as "saxophone barking". We invite readers to evaluate samples in our supplementary materials.
Emerging capabilities surface with sufficient model capacity and training data.
3.2 AUDIO SYNTHESTS
Text-To- Voice Synthesis: We evaluate in in-context text-to-speech synthesis (TTS) and singing voice synthesis (SVS). For TTS, we follow the evaluation in Wang al. (2023), using the same transcripts as Eskimez et al. (2024), to evaluate our model's ability to perform speech synthesis given a transcript and a speech sample from an unseen speaker. Following our training strategy, during evaluation we always provide the speaker's previous sentence when possible, otherwise random sample different from the target. For SVS, we evaluate Fugatto's 's ability to generate singing voice samples from instructions describing the desired lyries and musical style without backing track, for example: "Showcases a female singer with an interesting sound, conveys the message through american english lyrics, and infuses country influences throughout." The full list is available in Appendix A.4

---

Published as a conference paper at ICLR 2025
Our zero-shot TTS results in Table 2a show that Fugatto has word error rates similar to ground truth, and is competitive with expert and generalist (Omni) models in terms of speaker similarity. Our SVS results in Table 2b shows high cosine similarity between CLAP embeddings (Wu et al., 2023) of the synthetic samples and the captions used to create it. We note that the higher WHR in SVS likely comes from the higher difficulty for the generative model and the speech transcription model.
Table 2: Fugaito is comparable to generalist models and expert models on in-context TTS henchmarks.
On SVS, synthesizes samples with high similaiiyl and low WER relative to the task hand.
(a) TTS on the LibriSpeoch Test Clean benchmark in Wang ct al. (2023)
(b) SVS: WER and CLAP-Scores a set of
Model
Omni
WER +
SIM-o 1
SIM-r 1
10 music styles and 13 lyrics snippets from famous songs.
Ground Truth
2.20
Vall-E (24khz)
5.90
0.58
Model
WER J.
CLAP1
Natural Speech (16khz)
1.81
0.67
0.76
AudioBox (16khz)
4.80
0.73
Fugatto 7
71.90
0.49
UniAudio (16khz)
2.00
0.71
Fugatto 7
19.54
0.45
Fugatto 2
2.66
0.60
0.61
Fugatto
2.44
0.61
0.62
Text- To-Audi (TTA): We showcase Fugatto's performance on traditional TTA benchmarks that measure model's ability synthesize general sounds (AudioCAPS) and music (MusicCAPS) that follow instructions provided in text. We use the metrics (FD, FAD, and IS) and data splits (train, test) used in Kong et al. (2024b). Results in Table 3a and Table 3b shows that our model achieves strictly better scores than existing generalist models, while occasionally outperforming expert models.
Table Fugatto outperforms generalists models and occasionally outperforms specialist models in TTA benchmarks on AudioCaps and MusicCaps.
(a) TTA on AudioCaps benchmark in Kong et al., 2024b
(b) TTA on the MusicCaps benchmark Kong et al., 2024b
Model
Omni
FD |
FAD
IS
Model
Omni
FAD 1
VoiceLDM-Muune
x
2.50
MusicGen (medium)
X
35.52
5.02
1.94
AudioBox (16khz)
10.14
1.10
11.90
AudioLDM 2M-2--aare
16.12
2.74
2.30
NEXT-GPT
1.68
Tango-AF&AC AF&AC--T--CC
21.84
2.21
UniAudio
3.12
UniAudio*
3.65
Fugatto 7
16.73
1.36
9.72
Fugatto 7
11.52
1.43
2.73
Fugutto 7
20.20
2.21
10.21
Fugatto 7
13.18
1.93
2.97
3.3 AUDIO TRANSFORMATIONS
Speech Enhancement: We evaluate speech denoising and bandwidth extension tasks. Speech denoising evaluates the ability to extract speech from an additive mixture comprised of speech and noise. Bandwidth extension (sometimes referred to "upsampling") evaluates the ability to recreate missing content from audio that low passed filtered and downsampled not to include frequencies above certain threshold frequency².. For speech denoising, traditional metries PESQ and STOT described in Kong et al., 2023. For bandwidth extension, we use the VCTK benchmark and the traditional metric LSD described in Liu et al., 2024. In Fugatto, this task interpreted as source separation, in contrast with the enhancement task that modifies the acoustic qualities of the target audio. Though Fugatto is comparable to specialist models in bandwidth extension, work remains to be done to close the gap in denoising.
use the DNS-Nois benchmark and
Speech Modulation: In this task evaluate our model's ability transform person's 's emotion in speech into another emotion, while preserving their speaker identity and the transcript. For this purpose, construct a train and test set based on the ESD dataset. We use open-source models to report emotion classification and correlation with Valence, Arousal and Dominance (VAD). We establish upper bounds by also computing scores on ground truth data. Our results in Table Sa show that our model is able to properly transform the emotion as well as the ground truth, it needs improvement on speaker similarity and word error rates.
We use librosa after observing that torch.audio leaks frequencies above the threshold frequency.
7

---

Published as a conference paper at ICLR 2025
Table Fugatto is comparable to specialist models for speech denoising and upsampling.
(4) Speech Denoising on the benchmark Kong et al.,
(b) Upsampling on the VCTK benchmark et al.
Model
Omni
PESQwB
PESQNB
STOI
Model
Omni
LSD 4khz
LSD 8khz
Noisy dataset
1.59
2.16
91.60
Unprocessed (to 22khz)
2.74
1.84
FollSubNet
X
2.90
3.37
96.40
NuWave (to 22khz)
X
1.37
0.88
BAIl-Denuiser
3.23
96.60
NVSR (to 22klz)
1.49
CleanUNet 2
X
3.26
3.66
97.70
AudioSR (to 22khz)
X
1.25
1.08
0.1 Fugatlo
Fugatto
/
2.77
3.32
96.70
Fugatto ? 0.1
1.29
1.25
/
2.73
3.31
95,90
Fugatto
/
1.38
We evaluate our model's ability convert midi to audio with the 250 simple 2-bar monophonic melodies from Pati et al., 2020. Note that Fugatto has never seen monophonic melodies during training, with the average number of stems present training being 8. We use the error in pitch estimation on the ground truth MIDI notes as the upper-bound quality. We evaluate the I.1 distance of the FO contours extracted from the input rendered MIDI and the generated audio. During evaluation, we transpose the pitch contours to common key. We provide examples in Appendix A.5.
MIDIZAudio:
Table 5: Fugatto performs well on novel tasks such as emotion conversion and MIDI2Aaudio.
(u) Speech modulation (Emotion Conversion): remarkably high Top-2 accuracy
(b) MIDI2Audio: surprising zero-shot abilities monophonic melody to audio.
and pearson correlation between ground truth and synthetic samples on VAD. Low speaker similarity.
Model
WER
SIM-o
PVAD
Top-2 Acc.
Model
L1
Ground Truth
7.42
0.62
FO estimation error
0.21
Fugatto 7
24.17
0.19
0.68, 0.77, 0.77
0.59
Fugatto ?
1.74
Fugatto y = 3 21.96
0.21
0.68, 0.77, 0.77
0.62
Fugatto 2
1.00
3.4 EMERGENT CAPABILITIES AND SOUND GALLERY
We consider an ability be emergent if is absent in smaller models but appears larger ones (Wei et al., 2022; Radford et al., 2019), and Fugatto demonstrates emergent sounds and emergent tasks. In this section, we provide qualitative results through our demo page highlight the model's 's emergent abilities and invite readers to a guided tour through our sound gallery, which showcases compelling examples of Fugatto's artistic potential and emergent capabilities.
Emergent Sounds: Fugatto exhibits the ability to generate outputs that are not present in the training data itself. For instance, can synthesize cello that shouts with anger or person that speaks and barks.
Emergent Tasks:
Beyond generating novel sounds, Fugatto demonstrates the ability to perform tasks it was not explicitly trained for by combining tasks seen during training. For example, it can perform speech prompt conditioned singing voice synthesis or convert monophonic MIDI to singing voice.
3.5 COMPOSITIONALITY
Compositionality enables users to combine attributes and tasks to generate novel input combinations not found in the training data, providing artistic and fine-grained control the desired output. Since such novel combination rarely exists in the training data or the natural world, evaluating these samples typically challenging. Furthermore, there are no established basclines and metries for such sounds. Despite these challenges, we aim to provide qualitative and quantitative evaluation of our proposed approach and invite readers listen compositional samples on our website.
3.5.1 ATTRIBUTE/EVENT COMPOSITION
Control intensity of each instruction (Weighted Combination): Compositional synthesis with instructions gives users knob control the intensity of each instruction. In order to evaluate this
8

---

Published as a conference paper at ICLR 2025
abilityy, create (2)) pairs of instructions leveraging 10 event labels provided in Appendix A.6.
Given pair of events we can generate composite instructions through language or ComposableART:
Baseline linguistic composition example input:
input: synthesize Event 1 and Event
Proposed ComposableART composition example inputs:
wl.v (input: synthesize Event 1) w2 (input: synthesize Event 2)
For each pair of events, first generate samples using ComposableAR7 with different convex combination of weights for each instruction, and using the linguistic baseline". Then, for each method and generated audio, compute the cosine similarity between the audio and text CLAP embeddings for each event in the event pair used to produce the audio, shown as Instruction and in Figure I.
Figure shows that we increase the weight Instruction the occurrence of the event in the generated clip increases as evidenced by the CLAP scores, Reciprocally, as we decrease the weight on Instruction 2, the occurrence of the event in the generated clip decreases. In the linguistic baseline, we cannot control the weights of each attribute and their independent existence in synthesized samples lower than the ComposubleART samples at higher weights.
touRbe Sockk
Telsate 2 Luncee
2000
AdRCCIn pengle
Linguistic baseline
ComposableART (proposed).
Figure 1: Comparison of CLAP scores between the Linguistic baseline and ComposableART's composition of attributes with Fugatto. Instruction is equivalent to event.
Linguistic Baseline
ComposableART (proposed)
Figure 2: Comparison Baseline and ComposableART temporal guidance for instruction sequences.
Negation of attributes: With approach, users assign negative weights attributes, producing negative velocity that steers the model away from the attribute. We use the previous (10) event pairs to evaluate our approach alongside linguistic negation with the keyword 'NOT":
Baseline Linguistic Negation example input:
input: synthesian ze Event 1 and NOT Event
ComposableART Negation example input:
v (input: synthesize Event 1) v (input:
synthesize Event 2)
Table 6 shows CLAP cosine similarity of the linguistic baseline against the proposed ComposableART method. It can be observed that while the positive event remains similar to the baseline, the negative event has considerably lower cosine similarity than the baseline, indicating the effectiveness in
IIn Figure the diagonal fit, different weights, and the lack of samples in the linguistic baseline are an implementation and compute timeout hyproduct given that the linguistic baseline does not support weights.

---

Published as a conference paper at ICLR 2025
removal of the audio event using ComposableART as compared to the linguistic approach. Qualita- tively, we also observe that this can be immensely useful in steering towards negative emotions in speech synthesis or swapping gender, task which is not trivial.
Interpolation of attributes: ComposableART also supports interpolation of attributes by having the ability to independently control one attribute while keeping others. We evaluate our ability to control "pitch" as the interpolation variable by changing the weights on pitch and keeping "text" and "language" attributes fixed. Figure showcases the expected decrease in fundamental frequency as we increase the weight on the "low pitch" attribute.
160
Table 6: CLAP Cosine Similarity of Attributes
FD mccan
140
Instruction Type
CLAP Score
120
Linguistic baseline ve Instruction
0.02 + 0.02
-2
Linguistic baseline ve Instruction
0.17 + 0.02
Weight
ComposableART vee Instruction ComposableART -ve Instruction
0.06 + 0.01
Figure 3: FO mean given weight on the "low pitch" condition
-..04 +: 0.02
3.5.2 TEMPORAL COMPOSITION
To evaluate the effectiivneess of temporal guidance, we combine the previous set of audio events pairs over time. For ComposableART, we use sigmoid-like curve to increase and decrease over time the weights on event and event 2 respectively. For the equivalent linguistic baseline, we create the instruction "Event 2 followed by Event l". Figure showcases time-windowed CLAP scores, y-axis, for each approach and event across time, x-axis. We observe that the baseline is unable to establish temporal trend as well as CompoxableART, where we observe, as expected, a consistent increase in event and consistent decrease in event 2.
3.5.3 TASK AND MODEL COMPOSITION
Task Composition: In our website, we provide qualitative samples where we compose set of tasks involving "electronic music", "birds chirping". "dog barking". and "TTS". Our results show that the output conforms to the composition of such tasks using the proposed method.
Model Composition: our website, we provide samples where we consider different Fugatto models, one trained on speech datasets and the other trained on general sounds and audio events. We perform model composition to synthesize samples that contain both speech as well as audio events. This can be immensely useful in the future, where each domain specific model is expert model and a combination of such high-quality experts can he used to synthesize compositional outputs without the need train monolithic large generative model.
DISCUSSION AND LIMITATIONS
We work towards future where unsupervised multitask learning in audio synthesis and transformation emerges from data and model scale. Our proposed framework ComposableART Fugatto establishes first step towards this direction, and in this step we become aware of our current limitations and challenges. For example, optimizing dataset sampling weights to drive performance on multiple benchmarks is hereulean task, and generative models for audio and video would certainly benefit from research similar to (Albalak et al., 2023; Chung et al., 2023; Xie et al., 2024). Along straighter paths, we plan to replace mels with latent representation that better supports low frequencies and stereo. We believe this modification should be rather straight forward. Further work is necessary to establish the impact of data and free-form instructions on Fugatto's emergent abilities, especially using language to combine tasks not jointly seen during training. Finally, ComposableART requires more analysis on the choice of weights, and how the norm of the vector field can be used for easier and more stable control.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We plan to release our synthetic captions, instructions and code to facilitate reproducible research.
REFERENCES
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In
Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009,
Mathieu Bernard and Hadrien Titeux. Phonemizer: Text to phones transcription for multiple languages in python. Journal of Open Source Software, 6(68))3958, 2021.
Paul Boersma and Vincent Van Heuven. Speak and unspeak with praat. Glot International, 5(9/10):
341-347, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems, 13:1877-1901, 2020.
Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. arXiv preprint arXiv:2304.09151/ 2023.
Shuqi Dai, Ming- Yu Liu, Rafael Valle, and Siddharth Gururani. Expressivesinger: Multilingual and multi-style score- based singing voice synthesis with expressive performance control. In ACM Multimedia 2024, 2024.
SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. Lp-musiccapp Lim-based pscudo music captioning. arXiv preprint ar arXiv:2307.16372, 2023.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 6637-6647. Curran Asso- ciates, Inc., 2020. URL BBL.attpss/ppprceeinn..e..ere ccpppss..pee.ri...ppppp...eee files/paper/ 20t00///00//5555aaaaaaa111111110
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference Machine Learning, pp. 1068-1077 PMLR, 2017.
Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et E2 uts: Embarrassingly easy fully non- autoregressive zero-shot tts. arXiv preprint arXiv:2406. 18009, 2024.
Arushi Goel, Zhifeng Kong, Rafael Valle, and Bryan Catanzaro. Audio dialogues: Dialogues dataset for audio and music understanding. arXiv preprint arXiv:2404.0 07616, 2024.
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint 2.0.: 2023.
Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. Prompttts: Controllable text-to- speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint Stti:22277.125999 2022.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself, 2024. URL https: / arxiv.oro. /206 02507.
11

---

Published as a conference paper at ICLR 2025
Rafal Kocielnik, Shrimai Prabhumoye, Vivian Zhang, R Michael Alvarez, and Anima Anandkumar. Autobiastest: Controllable sentence generation for automated and open-ended social bias testing in language models. arXiv preprint arXt1:2302.03371, 2023.
Zhifeng Kong, Wei Ping, Ambrish Dantrey, and Bryan Catanzaro. Cleanunet 2: A hybrid speech denoising model waveform and spectrogram. In Proc. Interspeech 2023, pp. 790-794, 2023.
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In International Conference on Machine Learning, pp. 25125-25148, PMLR, 2024a.
Zhifeng Kong, Sang-gil Lee, Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Rafael Valle, Soujanya Poria, and Bryan Catanzaro. Improving text-to-audio models with synthetic captions. arXiv preprint arXiv:2406. 110.0006.54499 15487, 2024b.
Matthew Le, Apoory Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024.
Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. In The Eleventh International Conference on Learning Representations, 2023. URL https:
Yeonghyeon Lee, Inmo Yeon, Juhan Nam, and Joon Son Chung. Voiceldm: Text-to-speech with environmental context. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pp. 12576- IEEE, 2024.
Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, al. Prompttts 2: Describing and generating voices with text prompt. arXiv preprint 2rXiv:2300.00288. 2023.
Yaron Lipman, Ricky TQ Chen, Heli for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Maximilian Nickel, and Matt Le. Flow matching
Haohe Liu, Ke Chen, Qiao Tian, Wenwu Wang, and Mark D Plumbley. Audiosr: super-resolution at scale. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1076-1080. IEEE, 2024.
Versatile audio
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusson models, 2023. URL ttttps://arxiv. http://xxxi...//aa/////aaaa 01714.
Steven R Livingstone and Frank Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)c001663)))) 2018
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- net formm?id
ence on Learning Representations, 2019. URL Bkg6RicqY7.
Justin Lovelace, Soham Ray, Kwangyoun Kim, Kilian Q Weinberger, and Felix Wu. Simple-tts: End-to-end text-to-speech synthesis with latent diffusion. 2023.
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space based models, 2021. URL
OpenAl. Gpt-4o: A powerful multimodal language model. hello-gpt-4e, 2024. Accessed: 2024-09-21.
Ashis Pati, Siddharth Kumar Gururani, and Alexander Lerch. dmelodies: A music dataset for disentanglement learning. In Proceedings of the 21th International Society for Music Information Retrieval Conference, ISMIR 2020, Montreal, Canada, October 11-16. 2020, pp. 125-133, 2020. URL http:://archhh/ ves. sami net / smi p0mmm///pppppe///000000000
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020... 2020.
Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos loannis Mimilakis, and Rachel Bittner. The musdb 18 corpus for music separation. 2017.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 4779447777333 IEEE, 2018
Spotify. Pedalboard documentation. http:://ggo......tttoo..eeeeeela html, 2024. Accessed: 2024-09-23.
Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Conditional flow matching: Simulation dynamic optimal transport. arXiv preprint arXiv:2302.00482, 2023.
Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis. arXiv preprint arXiv:2005.05957, 2020.
Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et Audiobox: Unified audio generation with natural language prompts. arXiv preprint 2rX1: 2023.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301. 02111121,1,,,. 2023.
Xiaofei Wang. Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as a versatile speech transformer. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07633 2022.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kinn-KKikkatrik, and Shlomo Dubnov. Large-scale contrastive language-aude pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in neural information processing systems, 32, 2019.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2022.
Dongchao Yang. Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu. Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint 10000:2310000704, 2023.
13

---

Published as a conference paper at ICLR 2025
Jinhyeok Yang, Junhyeok Lee, Hyeong- -eok Choi, Seunghun Ji, Hyeongju Kim, and Juheon Lee. Dualspeech: Enhancing speaker-fidelie and text-intelligiiility through dual classifier free guidance. arXiv preprint arXiv:2408. 14423, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DATASETS
In this section we provide information about our dataset generation strategy, tasks, instruction generators, and full list of datasets created and used during training, including their sampling weights and task probabilities.
A.1.1 APPROACH DATASET GENERATION
In this subsection provide further descriptions of the procedures in each of the five pillars described in Section 2.1. Overall, the captions and instructions generation process is centered around leveraging LLMs transform tags into richer descriptions, always being careful that the description relates to the audio content, and to create what we call instruction generators, i.e. python methods that create free-form instructions based on the synthetic descriptions and task hand.
I- The python code snippet below is an LLM generated script that outputs free-form instructions for the task reverse sound, for example. Once such script exists, it can be used to prompt an LLM to generate scripts for other tasks with increasing levels of complexity.
Reve:sekudioinat.uue
audia, reforrereeeeeeee
"standazd""
"vezbsss ('aeeersa", "play backward', "iavert'],
"gerands"s ("reeersing ngg "playing backward', 'inverting")
"ountexiaf= P'audiu,, "suund", "swcosding', felig', (Lrack)), "aska" : Can you', "please', "Could you', 'I naed you to 'stylea's If (foomlleee eegllsselyy "preci soly', "acmurately" 'mentions' P'I pouvided', sentf, allashedf]
Bstwt homethod
del genwzabe wiste uni fon sunssu ao standard"))
re- www.eeeoeeeterreeoeeeeeeee audio references
templates
"lask): (verbi the (context 1.",
*Iverb] the [cpobext [ssylel.",
*lask) (verb)
[context] inentioni.",
"Your taak ia to [verb] the joontext 1.
*Me need the (contex)) fon) to be igecund).",
"Igerund) iconzexti the goal.",
(gerund) "The objective is (garund] the [ccstext (mentisnl.",
complate randon. choice (semplatcc)
ruct ion terplate- forman {
rendranoon choice (ree (faks'
verb-randon,co chosce "retl" gerunderandon cheige
gerunds" 1],
context-roni randon, cassinncccccceeoo izet contexts' 11,
style=randon. chol tree lsttlee' ])
nontion candon.chaaee. [scf('ncciions" menntionn"]
nome
ReversebacioInsttrn audio wtw.eeoeeeeeeeeeeeeeeeeeoeeeeeee reterences..eyy print (e".eepersoosaaaaaeeeeeeee cap.talizoo instructions:")
15

---

Published as a conference paper at ICLR 2025
II The python code snippet below is LLM generated script that outputs free-form instructions for the task speech modulation, for example.. Note that the instructions refer to relative changes such increase and decrease with different magnitudes such small or large changes. Once such script exists, it can be used to prompt an L.M to generate scripts for other tasks with increasing levels of complexity.
claxe SpeechModu lat Tn ructor:
Insbruction
"scale_ formant's
increase's
"saall's
add a touch nore resonance",
sliggtll wchanoe the Formant frequenciess
Clazee's
hrsssivly boost the formant Frequencies",
decrease"; {
"amall";
"Lon down the resomance just lit:l=" "elightly
treguencies",
b.
Clarge';
"dramaticslly zeduce the resenance", "rassively lover the tornant trequencies",
@stati onethod
pet_innttuction inodulation, direstion, inzensity):
return zanson, zanson.choicn choices (Specchtecoleetttottssrrroo int.ruuttitttttiiio imodulstion rdirection [intensityl) [0]
snctti
wambiie_insttruttiin )
parts !!
foz madulation_i in modelations:
modulation modelation_.t l'mduliaaiiiiio
direction modulasion p
instruction_Fe /t (nitonsstty"] parts..peaastnnsrrettt instruction
inteacity modulstion 1f
Specntbadiiattttonnsssseeee httperrrtttttteeteeetttttiiiiooo (modulation, cirection, inteneity)
if pasta:
1 Combinc parts into single iastruction
combined_inacruttion ", ".join (prrta((a) parta[-i]
clsc:
consines_insteueete parts[U]
"Let's combined
return iastroction
12 name main ("main")
16

---

Published as a conference paper at ICLR 2025
III Figure 4 provides visual depiction of our speech captioning, dubbed Prompt-2- Voice (P2V) pipeline. We applied this approach to several open- socech datasets. Additionally, we incorporated existing speaker descriptions from datasets like PromptSpeech (Guo et al., 2023), which provide details on gender, pitch, volume, and speaking rate, and enriched them by adding emotion and quality information. This automation not only streamlines the process but also allows efficiently label in-the-wild datasets, significantly scaling the available training data wigh high quality captions.
Gendar
Fangtratig
Emeilion Classiter
Acceet
Estiraiinn
Ago
LLM
Dripiig
Notaal Descirion
Sucaking Ral
Cloodieer
Figure 4: Synthetic caption generation pipeline for Prompt-to-Vont Voice (P2V).
V. We use audio effects libraries to create synthetic paired data where some factors are held constant while others change. We use Pedalboard (Spotify, 2024), a Python library for audio effects, to apply different audio effects to sounds believed not to have effects. For each effect, there are number of parameters that can be altered. We generate on the fly audio segments with the same effect but different levels of single parameter while keeping the others fixed, so that we can not only create instructions with synthetic pairs of uneffected vs. effected data, but also synthetic pairs where we ask the model increase decrease the intensity of given parameter (e.g. "increase the compression rate little bit" or "reduce the room size for the reverb moderately") For each parameter, we determined reasonable range for the parameter, and increments that would correspond to "a little". "moderate", and "a lot" of that parameter.
17

---

Published as a conference paper at ICLR 2025
A.1.2 LIST OF DATASETS
18

---

Published as conference paper at ICLR 2025
Open Source Vocal Datasets
Table 7: Open source vocal datasets. tts is Text-To-Speech and tta is Text-To-Audi (Audio from Caption)
Dataset
Sampling Weights
Task and Probabilities
AISHELL--
2.46
tts
CML-Dutch
0.769
CML-French
0.334
tts
CMI--Geman
1.762
CML-Italian
0.158
tts
CML-Polish
0.051
CML-Portuguese
0.086
tts
CML-Spanish
0,512
tts
0.131
tta
0.074
tta
DAPS-Ennaancement
0.21
enhancement paired 0.99
deenhancement paired 0.01
DNS-Challenge-2022
14.247
source separation
amoo-db-EmooionClaasffiiiion
0.068
tta
JEMOCAPPEnoiooCCaasscctiiii
0.059
jiccopu--Enotioonlaaificctiin
0.024
tta
LibriTTSS-Clean-1000
1.23
LibriTTS--Clean-366
4.31
tts
Librr/7SS-Other-5000
0.643
LibriVox-Englise
52.54
tts
LibriVox-French
0,909
tts
Geemann
1.147
LibriVox-Italiaa
0.114
tts
LibriVox-Portuguese
0.12
LibriVox-Spanish
0.276
tts
LIMMITS2024.*
0.768
0.80
inpainting 0.10
upsampling 0.10
0.451
tta
0.063
OMGEEmotion-Emoiionllsiiiiiitoo
0.017
tta
ravdess-EmotionClaasiiiiiionnn
0.014
tta
SongDessiiber-AuutiCaapiionioo
0.077
tta
SONYYCUUST-EeemCllasiiccaooo
0,279
tta
enssEEmmttonnCllasfffcttioon
0.028
tta
VCTIK-VaiceConvarsion
2.89
voice conversion paired vetk
VCTK-TTS
0.137
VocalSound-
VocaSound--YoalCllssicccallll VocalClassification
0.155
tta
19

---

Published as a conference paper at ICLR 2025
Open Source Non-Vocal Datasets
Table 8: Open Source Non-Vocal Datasets. its is Text-To-Specch and tta is Text-To-Audin (tta)
Dataset
Sampling Weights
Task and Probabilities
audiocess-AudiiCCptiinnggg
5.21
Ita 0.90
inpainting 0.10
0.15
tta
chine-hom--EvenClassiiieee
0.05
tta
Mooth-AAOA-EveetCllasiiieeee
0.01
tta
Clotho-AQA Seggleaaaee--eeeeClasifiiicii
0.07
tta
Clotho-2-AuuddiCaptinnn
0.19
tta
0.61
Ita
Epidemic sound-Aude AudiioCCaattiiinnngggg
0.41
tta
ESC-50
1.12 inpainting 0.10
tta 0.90
FMA-GGeneCClasfficatioo
1.04
Ita
FSD5O0-FFvennClassficciin
0.41
tta
GTZZAN-GGeneCCaassffcttion
0.01
Ita
LP-MusscCaas-MC--AuullCcppoinng JP-MusiCCaps-MSDDo
0.07
Ita
8.56
tta 0.90
inpainting 0.10
JP-MusscCapp-MSDD-1
8.62
tta 0.90
inpainting 0.10 tta
11.82
J--MusiCCassMMTTAAuuuiCCaaiiii
0.47
tta
MACS-AudioCaptioning
0.17
tta
Maestro
0.08
tta 0.90
upsampling 0.10
Medley-solos-DB
0.19
Ita 0.90 upsampling 0.
MSD
12.38
tta 0.80
upsampling 0.10
inpainting 0.10
MTG-Jamendo
2.16
tta 0.90
inpainting 1. tta
musdbba--nstrChassifaatiin
0.10
MusiiCaps-AAdiioCCppiinngg
0.54
tta 0.90
inpainting 0.10 tta
NunSpeca17-EvvenCCsssffccccn
0.06
NSynth-MIR
2.31
tta
SongDeesribar--uutiCCpppnnnn
0.08
tta
SONNCCUUSTHHivenClllaiicccaio
0.28
Ita
SoundDesss-AudiiDDeerriiooon
0.23
Ita
SoundVE-Caps
37.00
Ita
0.09
tta
WawText5k-AAdioCatttonnn
0.04
tta
W
0.02
tta
20

---

Published as a conference paper at ICLR 2025
New Datasets generated from Open Source Data
Table 9: New datasets generated from open source data. tts is Text-To Text-ToSpeeec, tta is Text-To- Au-iooo (Audio from Caption), and P2V refers to prompt to voice, in which LLMs were used to create full form captions given speech attributes. The -AF suffix indicates synthetie captions generated with Audio Flamingo.
Datasel
Sampling Weights
Task and Prubabilities
Mudnneap--AuuliCCqpiinnn---
5.79
0.90
ippainting 0.10
AudioSet-AF
37.24
0.80
inpainting 0.05
inpuinting random mask 0.05
upsampling 0.09
downsampling 0.01
AISHELI-3 SAHHELL--AAAdRRmovee SaaeeeSSonnneeeee
add sound effects 0.50
remove sound eflects 0.50
AISHELI ASSHELL--SSnundFffeccht --SuundFficcssMMbblationn
3.84
sound effects modulation
MSSIEELL--SSeecccMouaaie
3.84
speech modulation praut tta
CLAP freesound-AF
2.82
CREMA-D-P2V
0.62
inpainting 0.10
upsampling 0.10
SGEFxSet-AddSSumdiiieee
0.00
add sound effects paired
RomoveSoundEfficts emov-db-- ooo---mmttionClssiiiaattto OSD-ChaageElmotion ESD-ENGLISSH-P2
0.00
remove sound effects paired
0.07
1.90
specch modulation paired
0.93
0.80
inpainting 0.10
upsampling 0.10
ESD-MANDARIN-PPN
0.63
0.80 inpainting 0.10
upsampling 0.10
SAMOOCAP-EmacionCClaiiiicc
0.06
tta
Axcarpus-Fmotiionlllsss
0.02
LibriTT-Cllamn 100-Add
(0-AAdd-Reooee Soond-Sofeeeee
7.85
add sound effects 0.50
remove sound effects 0,50
LibriTTS-Cles 1att--000--p 1000 SomnEfffcctsMccuuattio
0.06
sound effects modulation
LibriTS--Cees Caars---Ccan---- Seeaaan-0---- Saae--00----ppepp N00-SpcccMModuatioonPPPe
9.34
speech modulation praat
LibriTT--Clcan- 1aeaann------- 1ee--000-E-nnan Enhanncemmnn
0.85
enhancement paired 0.99
decnhancement paired 0.01
LibriTS--Ceen Ceeee-Cleann3--naaaaaa 660-ahhaneemmnt
enhancement paired 00.99
deenhancement paired 0.01
SoniTTS-Other500 JL-CORPUS-P2V
6.11
enhancement paired
0.62
0.80
inpainting 0.10
upsampling 0.10 midi2audio
LMD-Aligne
1.17
musdbhy- Instgp-nnnrrrranssnieeee InstCCaasification
0.10
musdbby-addd-sd
3.81
add sound 0.50 add sound mixture 0.50
musdhhq-singing
0.86
singing voice synthesis
0.86
singing voice synthesis
3.81
scparation 0.50
remove sound from mixture 0.50
NonSpeech -iveeeeh/t--eeeCasssicttiiio
WSynth-MIl
2.31
OMGEmetion- EmottionClssifiiationn
ravdess- FinotionClassiiiattinn
01.1
RAVDESS Create Variation
0.02
speech modulation paired
RAVDESS- Changelntensity
SWNDESS-Changelntnnss
0.15
speech modulation paired
RAVDESS ChangeErottin
0.22
speech modulation paired
RAVDESS-P2V
0.80
inpainting 0.10
upsampling 0.10 singing voice synthesis nolanguage
FxpressiveSinger-PP P2V tess- EmootionCassiiieaatio
2.84
0.03
tta
VGG-AF
0.92
YODAS [English] (subset additions)
27.23
It 0.80
inpainting inpainting random mask 0.05
upsampling 0.04
reverse sound 0.04
downsampling 0.01
WavCaps-AF
7.83
21

---

Published as a conference paper at ICLR 2025
Below we provide descriptions for the suffixes associated with our generated datasets presented in Table 7. Table 8, and Table 2.
-AF; Refers to generating synthetic captions with the strategy described in (Kong et al., 2024b). In summary, the strategy consists of using an audio understanding model, here Audio Flamingo Chat, to caption sounds in the wild, and then filtering out synthetic captions based on the CLAP cosine similarity between the audio and the synthetic caption. In this paper, we used this strategy to create synthetic captions for AudioCaps, AudioSet, CLAP Freesound, and WavCaps.
AAddd, Remove] Sound Effects: These refer to applying. on the fly, audio effects modifications to existing audio data create paired data that describe tasks related to adding and removing sound effects. We focus on speech, which we assume has the least amount of audio effects applied to it, especially when compared to music data. In this iteration, the audio modulations are performed with (Spotify, 2024) and include a large list of effects such as chorus, reverb, distortion, amongst others.
[Add,, Remove]: This refers to splitting an audio mixture into separate audio stems and creating manifests that add one track given another track. In this opus, we have not explored creating artificial mixtures by adding random waveforms but imagine this strategy can yield good results.
-P2V: P2V, or prompt-to-voice, a task that enables control of speech synthesis through textual prompts, allowing for the description of speaker characteristics when an appropriate audio prompt that matches the desired persona is unavailable. The strategy has several components. First, we extract and curate all possible tags from existing metadata. For example, the expressive singer (Dai et al., 2024) dataset includes, implicitly and explicitly, information about vocal range, accent, language, style and others. Then, following the strategy in Section 2.1 we first prompt LLMs to produce long form descriptions of each attribute, then we prompt an LLM to create a instruction generator that combines the modified attributes in different ways.
-iigingg refers to leveraging music stems dataset with vocal tracks to create singing datasets. As usual, we leverage all the metadata available, combined with transcriptions obtained from speech transcription models and song captions obtained by prompting LLMs. In cases where an accompa- niment or backing track is available, we associate the timestamps on each lyrics snippet with the respective accompaniment. Once the metadata types are available, we prompt an LLM to create python method that produces instructions given the metadata for singing voice synthesis task with or without accompaniment.
adds to -Singing the combination of adjacent sentences, given criteria such as max gap between sentences and max sentence length.
Singing-Aggregateel
Source Separation and Add To Mixture: refers leveraging existing In this iteration, assuming doing such would produce samples undesirable oft-o--distiibution samples, we do not create mixtures by combining random samples together. Instead, we leverage mixtures that are already exist.
-Speech Modulation: refers to applying speech modulations to existing speech data to create paired data that describe a task in which a relative change is being applied to an attribute of speech, e.g. "Increase the speaking rate in this sample". In this iteration, the speech modulations are performed with Praat (Boersma& & Van Heuven, 2001) and we focus on transformations such as formant scaling, FO mean scaling (equivalent to transposition in music), FO variance scaling (equivalent to flattening or expanding the FO contour), and speaking rate scaling (equivalent to making a person speak faster or slower). Due artifacts that can be introduced in the audio result of the modulation, we limit the scaling values to ranges that we find acceptable in terms of audio quality.
This refers to leveraging available paired data to create new tasks. For example, an emotional speech dataset with paired data can be used to enable an emotion transformation tasks. The procedure straight forward and consists of first grouping samples by speaker and transcript, then creating pairs that establish relationships between two samples. For example, given a speaker and transcript, two samples with different emotion can be used define a "convert emotion task", two samples with the same emotion and intensity can be used to create "create a variation of this speech",, and two samples with the same emotion but different intensity can be used to define a "increase the intensity in the emotion task". Once the pairs and new tasks are defined, we prompt an LLM to create a script that takes in the attributes and tasks to generate task-specific instructions.
-Speech Modulation Paired:
22

---

Published as a conference paper at ICLR 2025
-Sound Effeets Modulation: refers to applying sound effects modulations to existing data to create paired data for relative change in audio effects in a file. In this iteration, the speech modulations are performed with Praat (Boersma & Van Heuven, 2001) and focus on formant scaling, FO mean scaling, FO variance scaling and speaking rate scaling. We limit the values to the range below
A.1.3 TASKS
Below we provide the list of tasks that are known to be supported by our model, and the respective minimal instruction for that task. Please note that we provide minimal instructions due to layout reasons, not due to model limitations.
Task
Minimal Instruction
Add Sounds Instrument Track
using audio example
add drums like this <audio> to this guitar track <audic> add rock drums to this guitar track saucio>
using text description
Add Sounds to Sound Mixture
using audio example
add guitar like this <audios oz to this backing track <audic> add guitar to this backing track <audio>
using text description
Apply Sound Effects
add long reverb to this audio <audlo> copy this <a saudios downsample this to TokHz caudios
Copy Audio
Downsample Audio
Enbance Audio
cabance this sound <audio>
Generate Audio From Captious
generate saxophone barking
Generate Music From Captions
generate calm sound track
Generate Speech From Captions
generate an angry voice
Inpaint Audio (Continuation)
inpaint this sound <aućio>
MIDI2Audio
using audio example
turn this MIDI track <audio> into natural audio. turn this MIDI track caudio> into heavy metal track.
using text description
Remove Sound Effects
remove sound effects <audio>
Remove Sound from Sound Mixture
from audio examples
remove this <aud! n> from these sounds remove the piano from this music track <audio> reverse this sound
from texi captions
Reverse Sound
Singing- Voice Synthesis
given Speech Captions
sing this 'At last my love' with male voice sing last my English sing last my
given Language
given Accent
English an Italian accent
Sepurate Audio into Sources
given audio examples
give this caudio> from this music track caudics give the piano from this music track <audtos
given text description
Sound Effects Modulation
(Increase and Decrease Attribures)
Chorus
increase the chorus a bit <audic>
Compressor
decrcase the compressor threshold <audio>
Delay
decrease the delay time <oudio>
Distortion
increase the distortion <audio>
Limiter
make the limiter less strong <audio>
Phaser
increase the phaser speed <audio> increase the reverb room size <audio>
Reverb
Speech Modulation (Paired)
Voice Conversion
convert from coudio> this coudio> couvert from British English American English make this calm sample <audio> sound angry give different take this voice <audic>
Accent Conversion
Emotion Conversion
Speech Variation
Speech Modulation (Pruat)
Scale Speaking Rate
increase the speaking rate <audio> increase the average pitch <audi o> increase the variance in pitch <audio> scale the formants here <audio>
Scale FD Mean
Scale FD Variance
Scale Formants
Texi-To-Speech
from Speech Prompt
say this "May the force!' given this voice <audio> say this 'May the force!' say 'May force!' English say "May the foree!' English with an Italian accent upsample sound caudios
from Speech Captions
male voice
from Language
from Accent
Upsample Audio
23

---

Published as a conference paper at ICLR 2025
A.1.4 INSTRUCTIONS
After an informal evaluation, we concluded that, for our purpose. Claude Sonnet better than GPT4-o at producing instructions and following prompts. Below we provide examples of template-based and free-form instructions for handful of tasks and datasets. For clarity, we remove the redundant 'input:' and 'output:" parts from all instructions.
ALSHELL-3-AdRRemov Seenovv-Sooun-EErr Sound-Effects tradicating the audio from the provided audio material with the Distortion, and Phaser effect your task. Focus on eradicating it systematically.
We need to minimize the delay time. </caption>
SSSHELLL--SSeechhMotulttttn
Let's subtly enhance the pitch for brighter sound, and dramatically slow down the speech for a very relaxed delivery.<ccapioo>
audiocaps-AudiiCCatiinnnng synthesize A consistent, loud mechanical motor</caption>
synthesize This is a sound of Speech</caption>
AudioSet-AF
Yo, mind fill in the missing bits in this tune? Please do this smoothly.]
CREMA-D-P2V
Mani fest a voice reproduction in American English verbalizing "The airplane is almost full.", with the timbre is often monotone and lacks energy, and its like middle-aged who is not Hispanic.
LibriTS--Clan-1000
Stitch up spiel in en declaring "BBu there was a passenger dropped eff for you-a little girl.", with female speaker with moderate pitch and intonation delivers her words quite rapidly a confined, clear acoustic environment.
WWNDESS-Changeeimee
modulate want to switch from angry to fearful. </capionn given example:
24

---

Published as a conference paper at ICLR 2025
A.2 MODEL AND TRAINING
Model visualization: Figure provides a visual description of Fugalto's 's architecture and input handling and Algorithm provides pseudo-algorithm for the optimal-transport conditional flow matching loss. Essentially, it minimizes the mean squared error between the estimator's prediction and linear interpolation between the data and the Gaussian noise sample used condition the model on It. scaled by (1 σ), where a is small enough value.
Language
"Can you turn this caln sample
T = Text Encoder
= Audio Encoder (Toutsoornee tnised
Audio Ourput
into extrese anger?
yee-mined)
Examples of instructions with interleaved modalities
T(enhance this (sspechh))
T(separate vocals fros this)5(music).
T(create druss for (keybos track)
X Arm Blocks
Trasformer Blocks
T(synthesize dogs birds chirping electronic music)
T(synthesize "Ah, So, given It voice:)5(voice example).
T(Can you make this sarple)S(cale spench)((Into extrere anger?)
Noise
Figure A description Fugatto's 's architecture and input handling.
Algorithm Optimal Transport Conditional Flow Matching Loss Pseudo- Algorithm
1: Sample X1 ~ X, where is the data distribution
2: 0.01
3: Xy f- randn _x)exx))
4: x; (1- σ) () Xo 1. X1
5: V. & estimalory (xt (it
6: u, X1 X0 (1 σ)
7: loss + mse(v,, uz)
Optimal Transport Conditional Flow Matching Pseudo-Algoree Aggorith::
25

---

Published as a conference paper at ICLR 2025
Model hyperparameters:
We provide list of Fugatto's hyperparameters in Table 10.
Table 10: Model Hyperparameters for the main Fugallo evaluated in this paper.
Hyperparameter
Value
t.schedule
uniform
nchelechan
80
n_hidden
1536
sigma
0.01
config.name
google/byt5-larg
text.ennod@r.configg.cclle
1.0
cex._enooder.coffig.nhhiieeee
1536
mel.encoding.strateey
separate
meLLenccoerr.confgg...........
false
mal.encco@e..cca1....pp........
rope
cel basse
16384
mel
o attention
true
rel.encooer..ccniii...eeetttttt.
false
wel.enccoder.connnig......yyss
3
0.1
0.0
heleen:oodr..coniii....bbbaa
16
mel_enocoee..conri...aaaattttt
false
false
pre
3
true
0.1
0.95
neleencode..coafig..d.oee.
1536
decooder.configg.dd::iee..
128
false
name
rope
16384
attention
true
false
24
0.1
16
true
3
hrt.r xattn.n heads
16
1536
decoder..comfig.ddddtt.
80
decoder.configg.dmmdee
1536
Training and Inference During the first phase, Fugatto trained on least 32 NVIDIA A100 GPU for approximately 1M iterations with template- based instructions and subset of tasks. During the second phase, we restart the optimizer and train for approximately 250k iterations, sampling from template-based and free-form instructions uniformly and adding all tasks. We use the AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate le-4, annealing the learning rate to 1e-6 during the second phase. A G2P model (Bernard & Titeux, 2021) pre-processee the text into the International Phonetic Alphabet (IPA) format. During inference, we generate mel-spectrograms using 50 function evaluations, 100 in practice, with Heun Solver and task-specific guidance scale y. Mel-spectrogram to waveform conversion is performed using the pre-trained universal Big VGAN V2 vocoder, available in the BigVGAN (Lee et al., 2023) repository
*Big http:///ihhhh..omhiiiaiiyyyyy
26

---

Published as a conference paper at ICLR 2025
A.3 ABLATIONS
I-sampling We draw inspiration from (Shen et al., 2018: Valle et al., 2020), where, due optimization issues, the model would be stuck in local minima and not make use of the text conditioning variable, rendering the model useless during inference. We believe the same is true with t sigmoid(N(0, 1)), and that with such distribution the model is stuck in a local minima and does not leverage the text to minimize the loss.
Figure 6: description of the image.
Model capacity: Exponentially smoothed validation loss per task at different t-values from smaller models with 0.8 B, 1.4 B params, to larger models with 2.5 B parameters.
Volidtion LoGs
3.703
snal
31175
31990
31525
31500
1107%
11800
31825
0..0
mna
0.30
mal
mml
0.500
0:132
03.20
6.5%
0.020
0330
0035
0.39
Sumey Ancher
Tumnn Bmukat
0.100
ena
mal
37873
mal
3.022
13923
6.77
8.50
6120
A2775
0275
0.24
0.22
Ancker
6100
0.70
0.300
3.25
61.0
3.25
0 3.22
6.20
3.20
8.0
Tueing Aneilan
Turnay Amviian
Figure 7: Validation scores for different Fugatto sizes on 3 benchmarks at different f-values.
27

---

Published as a conference paper at ICLR 2025
A.4 SINGING VOICE SYNTHESIS MUSIC STYLES AND LYRICS
The Voice-Synthesis experiments in Section 3.2 evaluates all combinations between the the singing-voice-
13 lyrics snippets and 10 music styles below. For each combination, synthesis instruction generator to create instructions such as: "Showcases female singer with an interesting sound, conveys the message through american english lyrics, and infuses country influences throughout."
This is the of 13 lyries snippets used during evaluation:
" "Is this the real life?\nIs this just fantasy?" "As I walk through the valley of the shadow of death" "Somebody once told me\nThe world is gonna roll me." "Look\nIf you had\nOne shotAnOr one opportunity."
"Joy to the world\nThe lord is come."
"Carry on, my wayward son non'nnhere' 11 be peace when you are done." "Please allow me to introduce myself." "
"At first was afraid, was petrified."
"The world was on fire and no one could save me but you."
"Josie's s on vacation far away."
"She's s got a smile that it seems to me."
"She was fast machine, she kept her motor clean."
"Do you have the time to listen to me whine."
This is the of 10 music styles used during evaluation:
"Country", "Electronic", "Hard Rock", "Hip-Hop", "Latin Rock", "Metal",
"Opera",
"Pop",
"RaB",
"Singee=Son
28

---

Published as a conference paper at ICLR 2025
A.5 MIDIZAUDIO FO CONTOURS
Figure shows comparisons of 25 monophonic melodies from the test set. As evident from the plot, the model manages to follow the provided MIDI notes and timing well, while inserting nuances, and varying timbres to the performance.
77.3
75-3
000
30.0
615
m1.0
97.5-
025
Melcty
Melody
Helody 0
Melody
Melody 10
17.5
12.5
22.5
7000
70.0
635
15.0
61.5
0..0
60.0
62.5
02.
00.0
Molody 12
Melody 12
Moledy 13
Nelody 14
Meloddy 25
19.2
72.5
32.5
70.0
70.0
05.0
00.3
Melldd
75.0
20.0
19.5
11
15.0-
10.01
0733
65.9
23.5
63.5
65.5
00.0
60.0
Melody 21
Melady 22
Weledl 23
Melody 24
Melody 25
32.5
30.0
72.5
72.0
72-
62.5
65.9
03.0
Figure 8: FO contours of input MIDI (cyan), and generated melodies (magenta). X-axis denotes time seconds, and Y-axis denotes MIDI pitch.
A.6 COMPOSITIONALITY
Attribute Composition (Audio Events): This is the list of audio events used during Attribute/Fseent composition in 3.5:
Violin, fiddle; Accelerating, revving, vroom; Water; Acoustic
guitar; Afrobeat; Whistle; Air conditioning) Air horn, truck horn; Aircraft; Wind
29


Reviews:
[{"review_id": "bgWXOmcZXt", "reviewer": "Reviewer_xcth", "summary": "The paper presents Fugatto, a generalist audio synthesis model capable of handling various audio generation and transformation tasks using text prompts. It introduces a dataset generation method that broadens Fugatto’s capabilities. In addition, ComposableART is proposed which extends classifier-free guidance to allow compositional control over audio generation. The model is evaluated on tasks across different audio domains, showcasing its versatility and competitive performance.", "questions": "1. How does ComposableART impact model performance quantitatively on specific tasks? It is recommended that the authors include more subjective and objective comparisons of the proposed system against task-specific models, such as LASS-Net and AudioSep for text-based audio removal.\n\n2. Could the authors clarify which benchmarks or metrics were used to evaluate compositional tasks?\n\n3. As a scientific conference submission, I would personally advise against the use of excessive fancy fonts and varied colors within the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Fugatto handles multiple tasks with text and audio inputs.\n2. The approach to synthetic instruction generation and data enrichment is well-structured, supporting the model’s generalization across diverse tasks\n3. ComposableART enables flexible audio composition, interpolation, and negation of prompts, adding control over generated outputs.", "weaknesses": "1. Techniques such as using LLMs for synthetic instruction generation lack novelty, which may challenge the paper's originality and scientific contribution.\n\n3. The comparison with state-of-the-art specialist models is limited for certain tasks, and the overall impact of ComposableART on performance remains unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "juNm1paHIS", "reviewer": "Reviewer_sTa1", "summary": "The paper proposes Fugatto 1, a versatile audio synthesis and transformation model, has capable of following free-form text instruction with optimal audio inputs. To address the problem that audio data does not inherently contain the instructions that were used to generate it, the paper introduces a specialized dataset generation approach to enhance the data reveals meaningful relationships between audio and language. Additionally, the paper proposes ComposableART with CFG to enhance the compositional abilities of the model.", "questions": "1. If I understand correctly, the paper provides an incremental method for building a large foundational audio generation model. What is the technical contribution of this paper?\n\n2. Have you verified the quality of the synthesized new dataset? How do you ensure the synthesized data is high-quality and strongly corresponds to the prompt instruction?\n\n3. Will you release the codes, the new datasets and the pretrained model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The paper proposes Fugatto, a large foundational audio synthesis and transformation model. This paper attempts to address the problem of audio data does not highly correspond to the text-instruction by a data generation strategy and extend CFG to support compositional guidance. The paper provides extensive experiments across different tasks in the audio domain.", "weaknesses": "The paper is a bit difficult to follow, and the proposed methods are trivial incremental. The main contribution of this paper is to train a generalist model with additional enhanced diverse synthesized datasets. I admit that the motivation of exploring a generalist model to benefit downstream tasks is good; however, the paper methodology lacks insights in the domain of audio synthesis and generation. The contribution of this paper for audio domain research is limited. In my opinion, this paper is more suitable for a technical report. \n\nI feel lost when reading the experiment section, the paper should provide at least a brief introduction for evaluation metrics used in the experiments, experimental details for adapting the proposed method in each single-task, and insights about why to adapt the method to each different single-task.\n\nEven though the paper showcased its applicability in various audio related tasks, however, it is not convincing to me the advantage of the proposed generalist model compared to other specialist models for each single-task. For example, in the TTS experiment, only speech similarity and intelligibility been evaluated, a further analysis, such as MOS study or F0 visualization compared to other methods and GT, is necessary for evaluating speech naturalness and quality to showcase the method can provide more natural speech than existing methods; No model comparison for the SVS experiment; Table 3(b) compares its performance with other specialist models, however, MusicGen and AudioLDM2 have different focuses (music v.s., multi-modality audio generation). The experiment is not a fair comparison and not convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XFoL6fuASs", "reviewer": "Reviewer_FfHR", "summary": "This paper introduces Fugatto, a versatile audio synthesis and transformation model capable of following free-form text instructions with optional audio inputs. The paper focuses on enabling generalist audio models with emergent capabilities, similar to large language models (LLMs), but optimized for audio. Fugatto can handle tasks such as audio generation, transformation, and compositional tasks by using a new dataset generation approach and a technique called Composable Audio Representation Transformation (ComposableART).  This method enhances classifier-free guidance, enabling complex operations like combining, interpolating, and negating instructions.", "questions": "Refer to weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) Generalist Audio Model: Fugatto offers a broad range of audio generation and transformation capabilities, filling the gap between specialist models and generalist models.\n\n(2) ComposableART: The novel technique extends classifier-free guidance to handle compositional tasks, allowing the model to compose instructions in ways that were not seen during training.\n\n(3) Dataset Generation: The paper provides a strategy for generating diverse and dynamic datasets, using LLMs for instruction creation and data augmentation. The authors claims they will release these the dataset and instruction generation code, which is useful for the research community.", "weaknesses": "At this stage, I believe the presentation of this paper is not strong enough. I still have the following questions:\n\na. How should we understand \"unsupervised multitask learning\"? In your training tasks, each task has both inputs and labels, such as TTS and TTA. I am not clear on how this relates to unsupervised multitask learning. UniAudio and AudioBox are closely related to your work, and I believe both Fugatto and UniAudio are trained on multiple tasks. Generally, we refer to 'unsupervised learning' when there are no labels, such as in the pre-training of LLMs.\n\nb. Your main framework is based on Flow Matching (FM). How do you control the duration? Since FM is a non-autoregressive model, this could be a challenge. For example, AudioBox or VoiceBox use a phoneme duration predictor. Are you using a duration predictor for TTS? Similarly, for tasks such as singing generation, how is the duration controlled?\n\nc. In Table 1, what qualifies as \"large-scale data\" in terms of hours? It seems UniAudio uses more than 100,000 hours of audio data. It would be helpful if you could list the number of hours for each model, as this would provide readers with a clearer understanding of what constitutes 'large-scale.'\n\nd. The definition of \"Emergent properties\": In the context of LLMs, we describe 'emergent properties' as the ability to solve unseen tasks and perform reasoning. However, in this paper, the examples given for emergent properties seem to focus on generating sounds that don’t exist in the real world or the training data. From my perspective, this doesn’t fit the definition of 'emergent properties,' as the model has already learned to understand sound types based on text descriptions. I strongly recommend discussing this point with other reviewers, as it currently seems like a bit of an overstatement.\n\ne. From a high-level perspective, Fugatto follows the multi-task training paradigm used in UniAudio and AudioBox. The authors need to explicitly highlight the advantages Fugatto has over AudioBox. That said, I do agree that the model's performance is impressive.\n\nIn conclusion, I agree that building a generalist model is a valuable topic, and this paper demonstrates good performance. However, the authors need to improve the presentation to help readers better understand the contributions.\n\nI am happy to improve my score during the rebuttal stage if the authors solve my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)