Improving Inference-Time Optimisation for Vocal Effects Style Transfer
  with a Gaussian Prior
Authors: name: Chin-Yun Yu, name: Marco A. Martínez-Ramírez, name: Junghyun Koo, name: Wei-Hsiang Liao, name: Yuki Mitsufuji, name: György Fazekas
Organization: Sony

Style Transfer with Inference-Time Optimisation (ST-ITO) is a recent approach
for transferring the applied effects of a reference audio to a raw audio track.
It optimises the effect parameters to minimise the distance between the style
embeddings of the processed audio and the reference. However, this method
treats all possible configurations equally and relies solely on the embedding
space, which can lead to unrealistic or biased results. We address this pitfall
by introducing a Gaussian prior derived from a vocal preset dataset, DiffVox,
over the parameter space. The resulting optimisation is equivalent to
maximum-a-posteriori estimation. Evaluations on vocal effects transfer on the
MedleyDB dataset show significant improvements across metrics compared to
baselines, including a blind audio effects estimator, nearest-neighbour
approaches, and uncalibrated ST-ITO. The proposed calibration reduces parameter
mean squared error by up to 33% and matches the reference style better.
Subjective evaluations with 16 participants confirm our method's superiority,
especially in limited data regimes. This work demonstrates how incorporating
prior knowledge in inference time enhances audio effects transfer, paving the
way for more effective and realistic audio processing systems.