<figure><img src="image_1.png" /></figure>

HAL
open science

# Time-Varying Audio Effect Modeling by End-to-End
Adversarial Training

Yann Bourdin, Pierrick Legrand, Fanny Roche

<figure><img src="image_2.png" /></figure>

## To cite this version:

Yann Bourdin, Pierrick Legrand, Fanny Roche. Time-Varying Audio Effect Modeling by End-to-End Adversarial Training. 2025. hal-05413743

HAL Id: hal-05413743

https://inria.hal.science/hal-05413743v1

Preprint submitted on 15 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
TIME-VARYING AUDIO EFFECT MODELING BY
END-TO-END ADVERSARIAL TRAINING

SUBMITTED TO THE JOURNAL OF THE AUDIO ENGINEERING SOCIETY, DECEMBER 2025

Yann Bourdin ID 1.2*, Pierrick Legrand2,3, and Fanny Roche1

¹Arturia, F-38330 Montbonnot-Saint-Martin, France

²Inria center at the University of Bordeaux, Astral Inria Team, F-33405 Talence, France

³IMS, UMR CNRS 5218, ENSC, Bordeaux INP, F-33405 Talence, France

ABSTRACT

Deep learning has become a standard approach for the modeling of audio effects, yet strictly black-box modeling
remains problematic for time-varying systems. Unlike time-invariant effects, training models on devices with
internal modulation typically requires the recording or extraction of control signals to ensure the time-alignment
required by standard loss functions. This paper introduces a Generative Adversarial Network (GAN) framework
to model such effects using only input-output audio recordings, removing the need for modulation signal
extraction. We propose a convolutional-recurrent architecture trained via a two-stage strategy: an initial
adversarial phase allows the model to learn the distribution of the modulation behavior without strict phase
constraints, followed by a supervised fine-tuning phase where a State Prediction Network (SPN) estimates the
initial internal states required to synchronize the model with the target. Additionally, a new objective metric
based on chirp-train signals is developed to quantify modulation accuracy. Experiments modeling a vintage
hardware phaser demonstrate the method's ability to capture time-varying dynamics in a fully black-box context.

# 1 INTRODUCTION

Audio effects are fundamental tools widely used by musicians,
producers, and engineers to shape sound timbre, dynamics, and
spatial characteristics. With the widespread adoption of digital
workflows, the need for the digital emulation of analog hardware
has intensified, aiming to preserve the unique, often non-linear
sonic characteristics of classic gear in a modern software en-
vironment. This field, known as Virtual Analog, distinguishes
three method categories, depending on their reliance on target
circuit knowledge or input-output data. White-box techniques
require extensive knowledge of the target effect's topology in
order to emulate it via numerical simulation or digital signal
processing; they require minimal data. In contrast, black-box
techniques rely primarily on measured input-output signals. Fi-
nally, grey-box methods combine hardware knowledge and data,
typically by extracting information from signal measurements
to improve white-box models. Machine learning has gained
significant attention for facilitating this extraction process and
enabling new black-box methods capable of learning complex,
non-linear mappings directly from input-output recordings. A
large body of research in grey-box and black-box methods has
successfully modeled effects with limited time-dependency, in-
cluding linear effects (e.g., equalizers [1]) and non-linear ones
(e.g., distortions and tube amplifiers [2]). Fewer studies have
tackled time-dependent effects such as compressors or specific
distortions [3, 4, 5, 6]. A less explored subject is modeling
time-varying audio effects, where the internal parameters of the
effect change as a function of time. Examples include chorus,
flanger, and phaser effects driven by a Low-Frequency Oscillator
(LFO). Several prior studies investigated the use of data-driven
methods for modeling phasers. In [7] the authors proposed a
grey-box model by analyzing a specific test signal. In [8], an
autoencoder-like architecture was proposed, employing bidirec-

tional Long Short-Term Memory (LSTM) networks in the latent
space. The authors of [9] and [10] utilize neural networks but
within a frame-based spectral processing approach. Other works
first extract the LFO signal, either manually [11] or with a neural
network [12], to use it as an additional input to an LSTM.

In the literature, the distinction between time-dependent and
time-varying effects can be ambiguous [3, 4]. Time-dependent
effects are time-invariant systems, viewed as functions mapping
a number of past input values to output samples. The time
dependency duration of a time-dependent effect and, as such,
the number of past values affecting an output sample, varies
with the effect type. It tends to be short for effects like distor-
tions (10-100 ms) but can be long for effects such as compres-
sion or reverberation (several seconds). Deep learning models
addressing such effects can either be stateless or stateful. In
stateless models, each output sample results from an operation
with a finite number of past input values, corresponding to the
model's receptive field [3]. In contrast, stateful models have a
theoretically infinite time dependency by employing an internal
state updated recursively, often through recurrent layers such as
LSTMs [8, 10, 11, 12]. This allows reducing the receptive field
but requires proper initialization of the internal state. It typically
consists in filling the state with zeros and running the model
for a number of timesteps, without gradient backpropagation, to
'warm up' the states. However, long-range time dependencies
may require large warm-up durations, making this inefficient. In
[5, 6], a State Prediction Network (SPN) uses the past values
of both the input and target output signals to predict proper ini-
tial states, replacing the use of warm-up samples with a more
efficient computation.

Time-varying effects are time-variant systems, viewed as func-
tions of both input samples and time. They typically involve
one or more of the effect's internal parameters (e.g., delay time,
gain, filter cutoff frequency) modified over time by a modulation signal commonly called a Low-Frequency Oscillator (LFO). In stateful models, the internal state can naturally represent the influence of time or modulation; thus recurrent models can handle the time-varying nature of an effect. However, the internal state now embeds information relative to the time when a signal is processed. In the supervised learning approach, input-output data are recorded at a specific time. For an LFO-based effect, it implies that each recording features a different starting LFO phase. However, supervised loss functions typically expect time-alignment of the compared features; therefore, the same signal processed by the same effect but with different LFO initial phases will yield a large error when compared. Thus, without proper care, training on a dataset with varying initial LFO phases results, at best, in a model averaging the behavior of the effect over the whole LFO period. Other works attempting end-to-end modeling of LFO-based effects either limit their dataset and adapt the training [9, 10], train over a dataset with a fixed LFO signal [8], or pre-train an LFO extraction network [12]. In our work, we extend the state prediction approach – originally designed for time-dependent effects – to time-varying ones, and propose an adversarial approach enabling training on large datasets with fewer constraints and without depending on LFO extraction. Additionally, our method is designed with the intent to apply to any time-varying effect, not just those operating with an LFO.

The paper is structured as follows. Section 2 reviews Generative Adversarial Networks. Section 3 details the proposed method, describing the generator and discriminator architectures, the State Prediction Network, and the training strategy and novel modulation metric. Section 4 outlines the experimental setup, including the target effect and dataset creation. Section 5 presents results regarding state initialization, mode seeking, and the two-phase training efficacy. Section 6 concludes.

## 2 GENERATIVE ADVERSARIAL NETWORKS

Generative Adversarial Networks (GAN) are generative models obtained through the concurrent optimization of two models called the generator and discriminator (or critic). The discriminator is a binary classifier distinguishing between 'true' samples from the dataset and 'fake' samples from the generator. The generator aims to generate samples resembling the target data, fooling the discriminator. The GAN framework can be formulated as a two-player minimax game [13]:

$$ \min_G \max_D \mathbb{E}_{x \sim p_d(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))] \quad (1) $$

where $p_d$ represents the distribution over the training data, and $p_z$ the distribution over the generator's input variables. In practice, this optimization is performed separately by alternating minimization steps between the generator and the discriminator:

$$ \min_D \mathbb{E}_{x \sim p_d(x)}[\log D(x)] - \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))] \quad (2) $$

$$ \min_G \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))] \quad (3) $$

However, following this original formulation, numerous improvements have been proposed, notably regarding the nature of the adversarial losses, incorporating regularization terms [14, 15], as well as normalization methods [16]. Though GANs were initially proposed for image domain tasks, they have seen

extensive success in audio tasks such as speech synthesis and neural vocoding [17, 18], as well as neural compression [19]. Key architectural advancements in GANs for audio include specialized discriminators such as the Multi-Scale Discriminator (MSD) [17] and multi-resolution STFT-based discriminators [19], handling audio at different resolutions, as well as the Multi-Period Discriminator (MPD) [18] to better capture the periodic structures inherent in audio signals. With such discriminators, GAN models are able to learn data-driven loss functions which can improve the details and perceptual quality of the generated audio [19, 20]. In audio effect modeling, GANs have been applied to guitar amplifier modeling [20, 21], especially when paired input-output data is unavailable.

In the context of this study, input-output pairs are available. Our dataset contains tuples $(x, y, \varphi)$ of input $x$ and output $y$ audio slices and the corresponding vector of user controls $\varphi$. The generator processes batches of tuples $(x, \varphi, h^0)$, where $h^0$ corresponds to initial states, to produce output signals $\hat{y}$. The discriminator evaluates 'true' tuples $D_{x,\varphi}(y)$ and generated tuples $D_{x,\varphi}(\hat{y})$.

## 3 METHOD

### 3.1 Generator

Although our framework is compatible with any supervised audio effect modeling architecture, we employ the Series-Parallel Time-Variant Modulation (SPTVMod) network, depicted in Figure 1. Derived from the SPTVMod architecture proposed in [5, 6], it features a modulation path and an audio path, the former computing modulation signals applied to the latter. Compared to SPTVMod, SPTVMod introduces processing blocks into the audio path, called FXBlocks; and the modulation path's first block no longer receives the audio input but a zero-filled signal.

Conditioning by user controls can be performed using Feature-Wise Linear Modulation (FiLM) [3, 22], as in SPTVMod. This involves processing user controls through a neural network to obtain channel-wise gains and biases modulating some activations in the model. While the architecture is thus suited for full modeling with user controls, we limit this study to 'snapshot' datasets, containing a single set of user controls, resulting in FiLM being bypassed.

The modulation blocks, ModBlocks, detailed in Figure 2 (left), are the same as in [6]. Each block begins with a convolutional layer and PReLU activation, feeding a recurrent branch that undergoes FiLM conditioning (if active) and contains an LSTM layer. To handle timesteps at a lower sample rate, the LSTM is surrounded by Average Pooling (downsampling) and Linear Interpolation (upsampling) layers. The branch terminates with two parallel $1 \times 1$ layers (convolutional layers with a kernel size of one). One generates the modulation tensors $\mu_j$ for the audio path, and the other produces a residual added to the block's main branch to form $s_j$, the input for the next ModBlock.

The FXBlocks, detailed in Figure 2 (upper right), consist of a residual connection around a sequence of convolutional layers, whose output is modulated by the tensor $\mu_j$ received from the adjacent ModBlock via a FiLM operation, followed by a PReLU activation. We apply an exponentially increasing dilation factor to these convolutions. While typically used to expand the
<figure><img src="image_4.png" /><figcaption>Figure 1: Block diagram of the architecture of SPTVMod. The generator has an orange background, and the discriminator a yellow one.</figcaption></figure>

receptive field [3], our motivation lies in signal processing: here dilation extends the filters' impulse response length, improving spectral accuracy and low-frequency processing. To keep the parameter count and computational cost low, we employ depth-wise convolutions for all layers following the first, i.e. each input channel undergoes a single convolution.

The LSTM layers are initialized with the states $h_j^0$. In our GAN framework, the $h_j^0$ introduce the necessary stochasticity for generation; they are either sampled from a probability distribution or predicted by a State Prediction Network (SPN), to be introduced in section 3.3.

The cached padding and cropping layers account for the reduction in temporal length caused by operations like convolutions and pooling. To avoid artifacts associated with zero-padding, we compute the precise input length needed so that the model outputs the target length, as done in [5, 6]. However, the architecture retains a cached padding mechanism [23] to facilitate streaming inference and potentially using Truncated Backpropagation Through Time [6].

## 3.2 Discriminator

Prior to the adoption of the adversarial framework, we investigated the use of a SPN [5, 6] for a fully-supervised setup. Given its structure as a convolutional classifier, the architecture is naturally suited to serve as a discriminator's backbone. We adapt this idea by constructing our discriminator from a series

<figure><img src="image_5.png" /><figcaption>Figure 2: Composition of the processing blocks: ModBlock (left), FXBlock (upper right) and FeatBlock (lower right).</figcaption></figure>

of feature extraction blocks, named FeatBlocks and depicted in Figure 2 (lower right). It consists of a convolution, PReLU activation, FiLM conditioning (if active) and average pooling, extracting hierarchical features $z_j$ (for real data) or $\tilde{z}_j$ (for generated data). The pooling layers progressively reduce the time length of the features. Subsequently, as depicted in Figure 1b, each feature $z_j$ (or $\tilde{z}_j$) is processed by a $1 \times 1$ convolution and a global average pooling layer (averaging over the whole duration). These processed tensors are summed and finally passed through a fully connected network with PReLU activations to yield the discriminator's scalar output.

## 3.3 State Prediction

We use a State Prediction Network (SPN) to initialize the LSTM states $h_j^0$, adapted here for time-varying effects, and detailed in Figure 1b. Unlike previous work where the SPN processed past values preceding the target [5], our SPN analyzes the current processing window $(x, y, \varphi)$, comprising both input and output signals. We leverage the discriminator's architecture for feature extraction. Since the discriminator already processes $(x, y, \varphi)$ tuples, we re-use its learned features to drive state prediction. Our preliminary results indicated this was more efficient than instantiating a duplicate architecture. Furthermore, this acts as a form of transfer learning, facilitating the SPN's training.

Therefore, the SPN comprises $1 \times 1$ layers and global average pooling layers (similarly to the discriminator) that process the discriminator's features, followed by a neural network determining the initial states $h_j^0$. The parameters of these layers are updated by the generator's optimizer. When using an SPN in this setup, the discriminator's parameters should be frozen to ensure that its features remain stable; otherwise, our preliminary experiments showed that the SPN would fail to learn properly.
## 3.4 Training Method

### 3.4.1 Loss Functions

The generator and the discriminator are trained using the hinge loss objective, given its success in prior work [17, 19, 20, 21]. The generator minimizes:

$$ \mathcal{L}_{\text{adv}}^{G} = \max(0, 1 - D_{x,\varphi}(\hat{y})) \quad (4) $$

while the discriminator minimizes:

$$ \mathcal{L}_{\text{adv}}^{D} = \max(0, 1 - D_{x,\varphi}(y)) + \max(0, 1 + D_{x,\varphi}(\hat{y})) \quad (5) $$

Previous studies demonstrate that including supervised losses, over y and ŷ, and feature-matching losses, over the true and fake features of the discriminator ($z_j$ and $\bar{z}_j$), can facilitate the training of GANs [17, 19]. We include a Multi-Resolution Short-Term Fourier Transform (MR-STFT) loss to the generator, to be referred to as spectral loss. It is a mean of multiple $\mathcal{L}_{\text{STFT}}^G$ losses, defined thereafter, computed using different STFT window sizes (512, 1024 and 2048 samples):

$$ \mathcal{L}_{\text{STFT}}^{G} = \frac{\|\|Y| - |\hat{Y}\|\|_{F}}{\|\|Y\|\|_{F}} + \frac{1}{L} \|\log|Y| - \log|\hat{Y}|\|_{1} \quad (6) $$

where L is the STFT window size, Y and ŷ are the STFTs of y and ŷ, and $\|\cdot\|_F$ is the Frobenius norm. This loss function is widely used in audio processing literature for its discrimination of time-frequency features, resembling human perception, and is robust to phase mismatches between signals. However, using a feature-matching loss was not considered in this study as our preliminary experiments indicated that it negatively impacted the training in our context.

In order to enhance the stability of the adversarial training, since the gradient of $\mathcal{L}_{\text{adv}}^G$ can vary over several orders of magnitude, we employ the adaptive loss balancing technique proposed in [19]. It consists in rescaling the gradients of each term of the generator's loss according to:

$$ g'_{i} = \frac{\lambda_{i}}{\sum_{j} \lambda_{j}} \cdot \frac{g_{i}}{\langle \|g_{i}\|_{2} \rangle} \quad (7) $$

where $g'_i$ is the corrected gradient of the i-th generator loss, associated to a weight $\lambda_i$ and the original gradient $g_i$, and $\langle \cdot \rangle$ is an exponential averaging with a rate of 0.99.

### 3.4.2 Training Strategy

Preliminary experiments revealed that a purely adversarial training faced significant stability issues. We observed recurring imbalances between the generator and the discriminator, alongside signs of catastrophic forgetting [15]: despite initially capturing the modulation characteristics, the generator would occasionally undergo sudden performance degradation, losing the learned modulation behavior and requiring numerous iterations to recover. Furthermore, the SPN failed to converge in the adversarial context, possibly due to the instable learning of modulation patterns. With LFO-driven effects, the learned LFO frequency keeps varying during training, making the prediction of its initial phase difficult. To mitigate these issues, we propose a two-phase training strategy:

**Phase I: Adversarial Training.** The first phase focuses on training the model using primarily adversarial losses. State prediction is disabled; instead, the internal states $h_j^0$ are initialized stochastically. This forces the model to learn the time-varying effect's general behavior without relying on a specific LFO phase. We found that retaining a spectral loss with a very low weighting was beneficial for stability.

**Phase II: Supervised Fine-tuning.** The second phase involves removing the adversarial objective, rendering discrimination unnecessary, and optimizing the model using the spectral loss combined with the SPN. Here, the SPN learns to synchronize the model's internal modulation with the target data. This approach stabilizes the training by providing a perceptually motivated objective, subsequently serving as the stopping criterion and evaluation metric.

### 3.4.3 Mode Seeking

Our framework is that of Conditional GANs [24], where the generation process is conditioned on the input audio signal and potentially user control parameters. Despite their success in various applications, conditional GANs are notably susceptible to mode collapse, a phenomenon where the generator yields a limited subset of outputs, failing to capture the full diversity of the target data. This issue is common in conditional settings: because the high-dimensional conditioning signals provide strong structural priors, the generator tends to disregard the lower-dimensional stochastic input vector intended to induce variation, here embodied by the initial states $h_j^0$. In the specific context of our work, this phenomenon manifests when the generator's internal states fail to influence the time-varying behavior of the model, yielding a fixed initial modulation phase regardless of the stochastic input. However, it seems essential that the modulation remains sensitive to the initial states; otherwise the SPN would be unable to synchronize the modulation phase with that of the training data during the subsequent fine-tuning stage.

To mitigate mode collapse, various techniques have been proposed, including Wasserstein-GAN regularization [14, 15] and spectral normalization [16] methods. Despite our efforts to utilize these techniques, they introduced difficulties in training without mitigating mode collapse. For our application, we adopt a strategy based on the mode seeking method introduced in [25]. They proposed adding a regularization term to the generator's objective to explicitly penalize mode collapse. This term maximizes the ratio of the distance between generated samples to the distance between their corresponding latent codes. We adapt this approach by minimizing the inverse of the MR-STFT distance between signals generated from the same input but with different initial states. Rather than normalizing by the distance in the latent space, we employ a sensitivity parameter $\epsilon$. The resulting mode seeking loss is defined as:

$$ \mathcal{L}_{\text{ms}}^{G} = \frac{\epsilon}{\epsilon + \mathcal{L}_{\text{MRSTFT}}(G(x, h_i^0), G(x, h_i^{0'})')} \quad (8) $$

where $x_i$ represents a slice of the input signal batch, and $h_i^0$ and $h_i^{0'}$ denote two distinct samplings of the corresponding initial states. During the initial adversarial phase, both are sampled randomly. However, in the presence of the SPN (during the fine-tuning phase), $h_i^{0'}$ is obtained by permuting the batch of initial states $h_i^0$ predicted by the SPN.
This loss function is particularly well-suited for LFO-based effects, as signals processed with different initial LFO phases inherently exhibit large spectral errors, resulting in a low mode seeking loss. Consequently, this objective not only prevents mode collapse but also encourages the SPN to diversify its predicted initial states.

## 3.5 Modulation Metric

In the case of an LFO-driven effect, the adversarial training enables the model to learn the LFO shape without the need to synchronize its LFO phase with that of the target. However, this limits the relevance of the MR-STFT metric, which cannot be used to determine when the adversarial training should stop. As to be discussed in the results (Section 5), a low MR-STFT value in this phase may be linked to an absence of modulation in the model's outputs, rather than a properly learned time-variation. Moreover, while the adversarial loss values can indicate whether there is a balance between the generator and the discriminator, they do not constitute an objective metric, although the models tend to have interesting behaviors when the generator and discriminator are close to a Nash equilibrium.

We propose an objective metric designed to be sensitive to the presence of a periodic modulation in a time-varying effect. It relies on a chirp-train signal, a test signal commonly used to analyze time-varying effects, which consists in the repetition of a sine wave with a logarithmic frequency sweep, generated by the formula:

$$x[n] = \sin\left(2\pi f_0 \frac{k \frac{\pi}{f_s} - 1}{\log k}\right) \quad \text{where } k = \left(\frac{f_1}{f_0}\right)^{\frac{f_s}{N}} \quad (9)$$

where N is the number of signal samples, $f_s$ the sampling rate, $f_0$ the starting frequency, and $f_1$ the final frequency. This test signal enables us to sample the time-varying transfer function of the target effect, at a rate equal to the inverse of the chirp duration. For an effect such as an LFO-modulated phaser, the time-frequency representation of its output should exhibit periodic patterns. We can compute a spectrum over each chirp to produce a chirp-aligned spectrogram (Figure 3a). Each frequency in the amplitude spectrogram is a signal itself which, if affected by the LFO, should evolve at the LFO rate. Therefore, we apply a Fourier transform to each frequency in the amplitude spectrogram to obtain a frequency-frequency representation (Figure 3b). In the presence of an LFO, we expect the amplitudes of many audio frequencies to have a large peak at the same modulation frequency in their modulation spectrum. On the frequency-frequency representation, this corresponds to columns of intensity localized at the LFO frequency and its harmonics. Finally, we sum over the audio frequency dimension to obtain a modulation spectrum, as shown in Figure 3c, exhibiting a large peak centered around the LFO frequency. Figure 4 summarizes our procedure. We formulate our metric through the following criteria based on the modulation spectrums $M(y)$ and $M(\hat{y})$:

$$\mathcal{L}_{\text{mod,p}} = \frac{|\sum_m M(y)[m] - \sum_m M(\hat{y})[m]|}{\sum_m M(y)[m]} \quad (10)$$

$$\mathcal{L}_{\text{mod,w}} = \sum_{i} \left| F_{M(y)}^{-1}(r_i) - F_{M(\hat{y})}^{-1}(r_i) \right| (r_i - r_{i-1}) \quad (11)$$

where $F_{M(y)}^{-1}$ is the inverse of the cumulative distribution function of $M(y)$, $r$ is the ordered set of quantiles of the modulation spectrums. The $\mathcal{L}_{\text{mod,p}}$ term evaluates the presence of modulation by comparing the total energies in the modulation spectrums; while $\mathcal{L}_{\text{mod,w}}$ aims to evaluate the horizontal shift between the modulation spectrums, and is an approximation to the one-dimensional 1-Wasserstein distance. This function is derived from Optimal Transport in [26], and we use the implementation shared by the authors. In order to combine both criteria into a single metric, we first generated a number of outputs through several models at multiple epochs. We then observed their spectrograms and modulation spectrums, and ordered each example according to our preferences. We finally combined the terms by adjusting a coefficient, so that the ordering of the metric values would preserve our own ordering by preference. This resulted in the following modulation metric:

$$\mathcal{L}_{\text{mod}} = \mathcal{L}_{\text{mod,p}} + 1.5\mathcal{L}_{\text{mod,w}} \quad (12)$$

Though this metric is differentiable, we cannot use it as a training objective because of its reliance on the specific train-chirp signal, and that it requires signal windows larger than several LFO periods. This metric could however be adapted for training in grey-box approaches.

# 4 EXPERIMENTAL SETUP

## 4.1 Target Effect

We focus on the phaser effect of the Ensoniq DP/4 digital multi-effect unit. It is controlled by the LFO rate, width, and center frequency, as well as the Notch filter depth and feedback amount. In this article we only investigate snapshot modeling, i.e. modeling a single user control configuration per model. We thus fixed the LFO width to 50%, LFO center to 50%, filter depth to 100%, and feedback amount to 0. Our experiments are done over two datasets using an LFO rate set to either 50% (corresponding to a 1.3 s period) or 80% (0.3 s), respectively referred to as Slow-LFO and Fast-LFO datasets for the rest of the paper.

## 4.2 Dataset Recording

We prepared two input audio files, reserved for training and for validation. Each file contains the 4-second-long chirp-train signal, described by (9) and necessary for the modulation metric, and a concatenation of 4-second music excerpts taken from the Free Music Archive dataset [27]. The chirps have a frequency ranging from 20 Hz to Nyquist and a duration of 1/33 of a second. The training audio is 620 seconds long, whereas the validation audio is 144 seconds long. A gain envelope evolving from -20 to 0 dB is applied to the chirp-train signal, as well as on the whole musical section. Both datasets' input-output pairs were recorded using the same input audio files, at a sample rate of 44.1 kHz. Since the hardware exhibits an aliasing frequency of 17.8 kHz, the input files were pre-filtered using a low-pass filter before reaching the device. Similarly, the output recordings were filtered by a Finite Impulse Response (FIR) filter with 1024 taps, designed to achieve a -100 dB attenuation above 17.8 kHz. This exact FIR filter was also added at the end of SPTVMod.
<figure><img src="image_6.png" /><figcaption>Figure 3: Steps producing a modulation spectrum. The reference, shown in (a-c), and outputs A and B correspond respectively to the spectrograms shown further in Fig. 7i, 7j and 7l.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 4: Process for extracting a modulation spectrum.</figcaption></figure>

## 4.3 Training and Evaluation Setup

SPTVMod is implemented and trained using the PyTorch deep learning library. Its source code and complementary material to this article are available on the companion website<fnref n="2" />. The training batches are formed from windows of the training data, with a window size of 32768 and a hop length of 4096. We use a batch size of 16 and the Adam optimizer [28] for both the generator and discriminator, with respective learning rates of 5·10⁻⁴ and 1·10⁻³. The validation loss is evaluated every 2085 steps (5 epochs) at which moment a 'checkpoint', a file capturing the state of the model's parameters and optimizers, is saved.

SPTVMod's audio processing path comprises three blocks containing sequences of four convolutional layers having 16 channels and a kernel size of 16. We chose a dilation base of four for the convolutions, i.e. the dilation factor of the *k*-th convolution in a block is equal to 4*¹⁻¹. The resulting impulse response length is 1357 samples (31 ms at a sample rate of 44.1 kHz). In the modulation path, the ModBlocks' convolutions have 16 channels and a kernel size of 16, without dilation. Their pooling size is 64, and the hidden size of the LSTM is 32.

The discriminator has six FeatBlocks with a pooling size of 4 and convolutions using 24 channels. The three first convolutions have a kernel size of 8, the fourth has 12, and the last two have 16. These values were chosen with the aim of reducing the information as efficiently as possible, resulting in a final length of 3. The fully-connected feature-processing networks of both the discriminator and the SPN contain two layers of 24 neurons.

The frequency range evaluated by the MR-STFT loss was limited to the aliasing limit of the device (17.8 kHz). The MR-STFT metric is evaluated in a Windowed Target (WT) fashion: like during training, the validation dataset is split into windows of length 32768 (743 ms), the SPN processes all windows and the MR-STFT loss is computed over every window then averaged. Streamed Target (ST) differs from WT in that the audio files are processed in consecutive windows without state reset, with the SPN initializing states for the first window only. Though ST represents our final goal when modeling audio effects, mirroring inference, and that previous work addresses the WT-ST discrepancy [6], we only consider WT in this study as a first step towards modeling time-varying effects.

# 5 EXPERIMENTS AND RESULTS

## 5.1 Experiment 1: State Initialization Methods and Mode Seeking

Since the SPN is absent in the adversarial phase, the initial states must be generated stochastically. We propose and compare two approaches in this first experiment. In both cases, a stochastically generated batched vector *u* is processed by a trainable linear layer to produce the states $h_j^0$. In the first method, which we call normal initialization, the elements of *u* are sampled from a standard normal distribution. In the second method, which we call angle initialization, a batch of angles $\theta$ is sampled from a uniform distribution $U[0, 2\pi]$, and *u* is formed by $u = [\cos \theta \ \sin \theta]^T$.

Our first experiment assesses the impact of the normal and angle initialization methods, as well as the absence of mode seeking or its presence with $\epsilon = 0.01$ and $\lambda_{ms} = 0.01$. Specifically, as depicted in Figure 5, we observe their impact on the adversarial training phase in terms of the modulation loss. The four combinations are trained three times, for each dataset, using different random seeds. Figures 5a and 5b display, for both datasets, the running minimum of the modulation loss across iterations (an iteration is considered to be a step of the generator's optimizer). On the Slow-LFO dataset (Figure 5a) with mode seeking disabled, convergence was more difficult for both initialization methods. When mode seeking was enabled, over this dataset, convergence seemed faster and the two methods performed similarly, but a model using noise initialization failed

<footnote n="2">²https://ybourdin.github.io/sptvmod</footnote>
<figure><img src="image_8.png" /><figcaption>Figure 5: Evolution of the modulation loss during the adversarial phase. The presence of mode seeking, as well as the noise and angle state initialization methods are compared.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 6: Minimum modulation loss over the validation subset obtained after the adversarial phase (log-scaled axis).</figcaption></figure>

to converge. On the Fast-LFO dataset however (Figure 5b), all models converged without significant differences between the categories.

We noticed that during training the modulation loss tends to significantly increase after it reaches a new minimum, indicating the instability of the adversarial training. Given that this behavior is obscured by the running minimum operation, Figure 5c presents the mean difference between the modulation loss and its running minimum. The gaps between the categories do not seem significant, suggesting that mode seeking neither particularly improves training stability nor does the state initialization method.

## 5.2 Experiment 2: Mode Seeking and Fine-Tuning

Our second and larger experiment applies the two-step training process detailed in Section 3.4.2, and assesses the impact of the presence of mode seeking as well as two variations in the fine-tuning phase, to be detailed in Section 5.2.2. For mode seeking, we consider a grid of its hyperparameters $\lambda_{\text{ms}}$ (regularization weight) and $\varepsilon$ (appearing in Equation 8), with $\lambda_{\text{ms}} \in \{0.01, 0.1, 1\}$ and $\varepsilon \in \{0.001, 0.01, 0.1\}$. For each $(\lambda_{\text{ms}}, \varepsilon)$ pair, three instances

of the models were trained independently using different random seeds, for each dataset. For a fairer assessment, the case where there is no mode seeking ($\lambda_{\text{ms}} = 0$) is evaluated with nine instances per dataset.

### 5.2.1 Adversarial Phase

First, each model is trained in the adversarial mode, with the MR-STFT loss weight set to 0.005. We chose to initialize the states using the angle initialization. We stop training after 400k iterations and select the checkpoint achieving the lowest modulation loss in validation as the resulting model of this phase and starting point for the next one.

Figure 6 depicts the modulation loss reached by all trainings in the adversarial phase. It is seen that the Slow-LFO phaser effect is harder to train than the Fast-LFO one. The impact of mode seeking is mixed: it improves modulation loss in the context of a slow LFO, but it seems detrimental for a fast LFO as the values are spread further to the top. We note that, empirically, values of the modulation loss below 0.2 tend to mean that a periodic modulation close to the one of the target LFO is present, while values above 0.9 tend to represent an absence of modulation. Over the Fast-LFO dataset, all the models obtained a modulation loss below 0.4, indicating they all display a form of modulation.

Figure 7 shows the spectrograms<fnref n="3" /> of the outputs of some models using the chirp-train signal (and used for the evaluation of the modulation loss). They illustrate the potential models one can obtain through adversarial training, and they also highlight the limits of our modulation metric. This metric is not directly sensitive to the modulation shape, as shown in Fig. 7b and 7l achieving a low modulation error despite the unusual shape of the resulting modulation. The metric is sensitive to the duration for which the modulation is sustained. While this may appear desirable, it can result in favoring long-duration, unexpected modulation patterns (Fig. 7e) over preferable but non-persisting shapes (Fig. 7f). Lastly, as can be expected, this metric does not evaluate audio quality or spectral accuracy, and thus a number

<footnote n="3">³The spectrograms use boxcar chirp-aligned windows, i.e. the window and hop sizes are equal to the chirp length (1336 samples); the frequency axis ranges from 0.04 to 17 kHz in log-scale.</footnote>
<figure><img src="image_10.png" /><figcaption>Figure 7: Output spectrograms of some models (including best and worst) trained through the adversarial phase on the Slow-LFO (top) and Fast-LFO (bottom) datasets, using the 4-second-long chirp-train input signal, ordered by modulation loss.</figcaption></figure>

of undesirable effects are observed such as static attenuations at certain frequencies (as in Fig. 7j and 7m), artefacts (Fig. 7k and 7n), and poor high-frequency processing globally. Consequently, continuing training using a spectral loss should improve modeling quality.

### 5.2.2 Fine-Tuning Phase

Following the adversarial phase, we proceed to spectral fine-tuning. An SPN is instantiated, and we resume training from the best checkpoint. The discriminator – except for its feature extraction layers – and the adversarial losses are disabled, and the FeatBlocks are frozen. The loss function used is the MR-STFT loss, including mode seeking regularization where applicable.

Prior experiments indicated that the sudden addition of a just-initialized SPN could degrade the model. Because the SPN cannot initially predict correct states, the model tends to remove modulation effects entirely, as this is the easiest way to minimize the MR-STFT loss. Therefore, we investigate the addition of an intermediary phase to pre-train the SPN. In this phase, resuming from the adversarial stage, the SPN is introduced while all other model parameters are frozen. The SPN is then trained for 50k iterations. At the end of this stage, we select the checkpoint with the lowest validation MR-STFT loss and unfreeze the generator parameters (keeping FeatBlocks frozen). We compare the results with and without this SPN pre-training step, obtained after 400k fine-tuning iterations in total.

Figure 8 shows output spectrograms ordered by spectral loss, generated using the chirp-train signal and models obtained during the fine-tuning phase, not necessarily chosen at their checkpoint minimizing the loss. While they depict a large improvement in quality in many cases, they illustrate the difficulty of training a time-varying effect and evaluating it objectively. The quality of the phasing effect appears very similar between Fig. 8b and 8j; however, due to a difference in initial phase, the MR-STFT error of the latter is substantially higher than that of the unmodulated signal shown in Fig. 8g. Even a slight LFO phase shift strongly penalizes the spectral error (Fig. 8h).

The observation of these spectrograms clarifies the objective results presented in Figure 9, which depicts the WT MR-STFT loss computed over the validation set for all models and datasets. For each model, the checkpoint retained is the one minimizing the validation MR-STFT subject to a modulation loss below 0.9. This ensures that the models are compared on realistic checkpoints suitable for our modeling, where modulation occurs, avoiding cases where low MR-STFT errors result from the absence of modulation. Mode seeking appears detrimental to spectral loss for both datasets, despite the improved modulation loss observed in the adversarial stage for the Slow-LFO dataset. Surprisingly, though we observed more occurrences of mode collapse when mode seeking was disabled in the adversarial stage, we did not find a link between mode collapse and any failure of later fine-tuning. Therefore, this suggests that mode seeking is not necessary when training in multiple phases, and that it should be disabled in the fine-tuning phase. A different
<figure><img src="image_11.png" /><figcaption>Figure 8: Output spectrograms of some models during spectral fine-tuning, on the Slow-LFO dataset, using the chirp-train signal restricted to a single training window (743 ms), ordered by MR-STFT value over this specific window.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 9: Minimum Validation Spectral (MR-STFT) loss obtained after the fine-tuning phase.</figcaption></figure>

formulation of the mode seeking objective or a distinct choice of its hyperparameters might potentially lead to more promising results. Nonetheless, we observe that the SPN pre-training strategy tends to improve the final spectral loss, especially on the Fast-LFO dataset, and is thus recommended.

As highlighted by Fig. 8, the MR-STFT is highly sensitive to phase and frequency discrepancies in the reproduced modulation, which relies on correct predictions of the SPN. Therefore, the perceptual information given by this metric is mixed with the SPN's generalization capabilities and has limited interpretation when the modulation is not perfectly learned. Notably, we noticed that our approach was prone to overfitting because we used to observe the training MR-STFT loss decrease while the validation loss did not. That is why we introduced an overlap in the division of the dataset into batches (as explained in Section 4.3), given the short duration of the datasets.

A drawback of our fine-tuning approach is that it only considers minimizing the WT error, i.e. obtaining accurate modulation for the duration of the training window. Therefore, in Streamed Target, the modulation tends to collapse shortly after the duration

of a single window, as observed in Fig. 7d, 7f, 7o and 7p. This occurs regularly when modeling the Slow-LFO effect, given that its LFO period is greater than the window duration. However, this is rare for the Fast-LFO dataset, presenting about two LFO periods in a training window. More examples are available on the accompanying website.

## 6 CONCLUSION

In this paper, we presented a novel framework for black-box modeling of time-varying audio effects, addressing the challenge of training without known or aligned modulation signals. We proposed the SPTVMod architecture, a convolutional-recurrent generative adversarial network, and introduced a robust two-phase training strategy. This begins with an adversarial phase which encourages the model to learn the distribution of the modulation behavior without strict phase constraints; followed by a supervised fine-tuning phase where a State Prediction Network (SPN) adapts the model's internal state to synchronize its modulation with the target data. To support this process, we introduced a specialized modulation metric based on chirp-train signals and optimal transport. This metric enables the objective assessment of LFO frequency and presence, while also serving as a stopping criterion for the adversarial phase. Our experiments on a hardware phaser effect demonstrated that this approach effectively yields models capable of reproducing time-varying behaviors, while black-box supervised methods typically require the knowledge or extraction of modulation signals.

Future work will focus on several key areas. Originally, in the context of dynamic range compression [5], the SPN's tendency to overfit was a desirable training mechanism that did not impact inference, which could rely on states initialized with zeros. However, for time-varying effects, reliable state prediction becomes essential for model evaluation, requiring the SPN to present generalization capabilities on unseen data. Investigating the underlying structure of these learned states could allow for manual initialization during inference, thereby removing the dependency on the SPN for evaluation and inference. Furthermore, to address the issue of modulation collapse observed in streamed processing, future architectures could enforce LFO periodicity
by employing differentiable oscillators instead of LSTM layers. Alternatively, analyzing the dynamics of the LSTM layers could guide us towards manual control over the modulation. Beyond LFO-driven units, this framework should be evaluated on the modeling of non-periodic time-varying effects, such as magnetic tape recorders which may present stochastic variations. A key priority is to investigate the conditioning of SPTVMod by user controls, and to integrate streamed target enhancements as proposed in [6], especially to enable the training of slowly evolving modulation effects. Concurrently, training stability could be improved by investigating new discriminator and SPN architectures. Finally, the validation of these models requires more rigorous assessment. Our modulation metric could be enhanced by incorporating psychoacoustic models of modulation frequency perception, similar to the modulation spectrum distance proposed in [8], and complemented by formal perceptual listening tests to correlate objective loss values with subjective audio quality.

# 7 ACKNOWLEDGMENT

This work is part of a Cifre PhD project funded by ANRT. It benefited from access to the computing resources of the ‘CALI 3’ cluster, operated by the University of Limoges and part of the HPC network in the Nouvelle Aquitaine region, financed by the State and the Region.

# REFERENCES

[1] Marco Martínez-Ramírez and Joshua Reiss. End-to-end equalization with convolutional neural networks. In *Int. Conf. on Digital Audio Effects (DAFx)*, 2018.

[2] Tara Vanhatalo, Pierrick Legrand, et al. A review of neural network-based emulation of guitar amplifiers. *Applied Sciences*, 12(12):5894, 2022. ISSN 2076-3417.

[3] C. Steinmetz and J. Reiss. Efficient neural networks for real-time analog audio effect modeling. In *Audio Engineering Society Convention*, 2022.

[4] M. Comunità, C. Steinmetz, et al. Modelling black-box audio effects with time-varying feature modulation. In *IEEE Int. Conf. on Acoustics, Speech and Signal Proc. (ICASSP)*, 2023.

[5] Yann Bourdin, Pierrick Legrand, and Fanny Roche. Tackling long-range dependencies in dynamic range compression modeling via deep learning. In *16th Int. Conf. Evolution Artificielle (EA)*, Bordeaux, France, 2024.

[6] Yann Bourdin, Pierrick Legrand, and Fanny Roche. Empirical results for adjusting truncated backpropagation through time while training neural audio effects. In *Int. Conf. on Digital Audio Effects (DAFx)*, 2025.

[7] Roope Kiiski, Fabián Esqueda, and Vesa Välimäki. Time-variant gray-box modeling of a phaser pedal. In *Int. Conf. on Digital Audio Effects (DAFx)*, Brno (Czech Republic), 2016.

[8] Marco Martínez-Ramírez, Emmanouil Benetos, and Joshua Reiss. A general-purpose deep learning approach to model time-varying audio effects. In *Int. Conf. on Digital Audio Effects (DAFx)*, 2019.

[9] Alistair Carson, Cassia Valentini-Botinhao, et al. Differentiable grey-box modelling of phaser effects using frame-based spectral processing. In *Int. Conf. on Digital Audio Effects (DAFx)*, Copenhagen (Denmark), 2023.

[10] Gyubin Lee, Hounsu Kim, Junwon Lee, et al. CONMOD: Controllable neural frame-based modulation effects. In *Int. Conf. on Digital Audio Effects (DAFx)*, Guildford, United Kingdom, 2024.

[11] Alec Wright and Vesa Välimäki. Neural modeling of phaser and flanging effects. *Journal of the Audio Engineering Society*, 69 (7/8):517–529, November 2021. ISSN 15494950.

[12] Christopher Mitcheltree, Christian Steinmetz, Marco Comunità, et al. Modulation extraction for LFO-driven audio effects. In *Int. Conf. on Digital Audio Effects (DAFx)*, 2023.

[13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. Generative adversarial nets. *Advances in Neural Information Processing Systems (NIPS)*, 27, 2014.

[14] Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In *Int. Conf. on Learning Rep. (ICLR)*, 2017.

[15] Hoang Thanh-Tung and Truyen Tran. Catastrophic forgetting and mode collapse in GANs. In *Int. Joint Conf. on Neural Networks (IJCNN)*, pages 1–10. IEEE, 2020.

[16] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, et al. Spectral normalization for generative adversarial networks. In *Int. Conf. on Learning Rep. (ICLR)*, 2018.

[17] Kundan Kumar, Rithesh Kumar, et al. MelGAN: Generative adversarial networks for conditional waveform synthesis. *Advances in Neural Information Processing Systems (NIPS)*, 32, 2019.

[18] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. *Advances in Neural Information Processing Systems (NIPS)*, 33, 2020.

[19] Alexandre Défossez, Jade Copet, et al. High fidelity neural audio compression. *arXiv preprint arXiv:2210.13438*, 2022.

[20] Yu-Hua Chen, Woosung Choi, et al. Improving unsupervised clean-to-rendered guitar tone transformation using GANs and integrated unaligned clean data. In *Int. Conf. on Digital Audio Effects (DAFx)*, Guildford, United Kingdom, 2024.

[21] Alec Wright, Vesa Välimäki, and Lauri Juvela. Adversarial guitar amplifier modelling with unpaired data. In *IEEE Int. Conf. on Acoustics, Speech and Signal Proc. (ICASSP)*. IEEE, 2023.

[22] Ethan Perez, Florian Strub, Harm de Vries, et al. FiLM: Visual reasoning with a general conditioning layer. In *AAAI Conf. on Artificial Intelligence*, 2018.

[23] Antoine Caillon and Philippe Esling. Streamable neural audio synthesis with non-causal convolutions. In *Int. Conf. on Digital Audio Effects (DAFx)*, 2022.

[24] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. *arXiv preprint arXiv:1411.1784*, 2014.

[25] Qi Mao, Hsin-Ying Lee, et al. Mode seeking generative adversarial networks for diverse image synthesis. In *IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)*, 2019.

[26] Bernardo Torres, Geoffroy Peeters, and Gaël Richard. Unsupervised harmonic parameter estimation using differentiable DSP and spectral optimal transport. *IEEE Int. Conf. on Acoustics, Speech and Signal Proc. (ICASSP)*, 2024.

[27] M. Defferrard, K. Benzi, et al. FMA: A dataset for music analysis. In *Int. Society for Music Information Retrieval Conference (ISMIR)*, 2017.

[28] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. *arXiv preprint arXiv:1412.6980*, 2014.