In-Context Prompt Editing For Conditional Audio Generation
Authors: name: Ernie Chang, name: Pin-Jie Lin, name: Yang Li, name: Sidd Srinivasan, name: Gael Le Lan, name: David Kant, name: Yangyang Shi, name: Forrest Iandola, name: Vikas Chandra

Distributional shift is a central challenge in the deployment of machine
learning models as they can be ill-equipped for real-world data. This is
particularly evident in text-to-audio generation where the encoded
representations are easily undermined by unseen prompts, which leads to the
degradation of generated audio -- the limited set of the text-audio pairs
remains inadequate for conditional audio generation in the wild as user prompts
are under-specified. In particular, we observe a consistent audio quality
degradation in generated audio samples with user prompts, as opposed to
training set prompts. To this end, we present a retrieval-based in-context
prompt editing framework that leverages the training captions as demonstrative
exemplars to revisit the user prompts. We show that the framework enhanced the
audio quality across the set of collected user prompts, which were edited with
reference to the training captions as exemplars.