Latent Fourier Transform
Authors: Mason Long Wang
Cheng-Zhi Anna Huang
Venue: ICLR 2026 Oral (2026)
Keywords: Music Generation
Signal Processing
Diffusion Models
Audio
Music
Audio Generation
Controllable Generation
Fourier Transform
Diffusion Autoencoders
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce novel frequency-domain controls for generative music models by applying the Fourier transform to the latent space of a diffusion autoencoder.

Abstract: We introduce the Latent Fourier Transform (LatentFT), a framework that provides novel frequency-domain controls for generative music models. LatentFT combines a diffusion autoencoder with a latent-space Fourier transform to separate musical patterns by timescale. By masking latents in the frequency domain during training, our method yields representations that can be manipulated coherently at inference. This allows us to generate musical variations and blends from reference examples while preserving characteristics at desired timescales, which are specified as frequencies in the latent space. LatentFT parallels the role of the equalizer in music production: while traditional equalizers operates on audible frequencies to shape timbre, LatentFT operates on latent-space frequencies to shape musical structure. Experiments and listening tests show that LatentFT improves condition adherence and quality compared to baselines. We also present a technique for hearing frequencies in the latent space in isolation, and show different musical attributes reside in different regions of the latent spectrum. Our results show how frequency-domain control in latent space provides an intuitive, continuous frequency axis for conditioning and blending, advancing us toward more interpretable and interactive generative music models.

LATENT FOURIER TRANSFORM
Mason L. Wang
Cheng-Zhi Anna Huang
CSAIL
CSAIL
Massachusetts Institute Technology ycda .mit edu
Massachusetts Institute Technology
huangcza@mit.enn
ABSTRACT
We introduce the Latent Fourier Transform (LATENTFT), a framework that pro- vides novel frequency-domain controls for generative music models. LATENTFT combines a diffusion autoencoder with latent-space Fourier transform to sepa- rate musical patterns by timescale. By masking latents in the frequency domain during training, our method yields representations that can be manipulated coher- ently inference. This allows us generate musical variations and blends from reference examples while preserving characteristics at desired timescales, which are specified as frequencies in the latent space. LATENTFT parallels the role of the equalizer in music production: while traditional equalizers operates on audible frequencies to shape timbre, LATENTFT operates on latent-space frequencies to shape musical structure. Experiments and listening tests show that LATENTFT improves condition adherence and quality compared to baselines. We also present technique for hearing frequencies in the latent space in isolation, and show dif- ferent musical attributes reside in different regions of the latent spectrum. Our results show how frequency-domaie control in latent space provides an intuitive, continuous frequency axis for conditioning and blending, advancing us toward more interpretable and interactive generative music models.
INTRODUCTION
Modern audio generation models often operate in coarse-to-fine manner, generating progressively finer representations of the output signal a conditional chain. In diffusion models (Kong et al,, 2020 Liu et al. 2023 Huang et al. (202)). higher noise levels provide coarser representations, while lower noise levels provide finer representations. In autoregressive models like AudioLM (Borsos) al. 2023a) and MusicL. (Agostinellii et al. (023).
encoding stage represents the input signal as a hierarchy of coarse-to-fine tokens, and a generative model attempts to predict fine tokens from coarser ones. This is also the for masked token models (Garcia et al. 2023), discrete diffusion Yang et al. 2023), and next-scale prediction (Qiu et al. 2024).
Since the generative process involves conditioning on coarse representations, is natural generate new samples using the coarse representations of reference example. This type of conditioning has been used for ttroke-based image editing and image translation (Meng al. 2021 Choi al. [021] I). However, conditioning on small- or mid-scale features is harder, since the representations used by the generative model rarely capture these features in isolation. For instance, discrete representations define fine tokens relative to coarse ones via residual vector quantization (RVQ) (Zeghidour et al.)
2021 Kumar al., 2023), preventing them from being interpreted independently.
Conditioning arbitrary timescales from reference example would be useful in music, which con- tains slow-moving patterns (like chord progressions) and fast-moving patterns (like trills). Patterns occurring at different timescales may be desirable starting points for generating musical variations, but are difficult to specify precisely using text. Existing reference-basee controls (Villa- Meltera al. 2025 Garcia et al. 2025) target attributes like pitch, loudness, and instrumentation. which distributed across multiple timescales. While these methods provide control over various semantic axes, none directly expose the 'timescale' axis.
To address this, we explore the use the Fourier transform, which provides a decomposition of a signal into oscillations at different frequencies. High frequencies capture the most rapid variations in

---

the signal ('small-scale characteristics), while low frequencies capture slow variations in the signal ('large-scal characteristics). This representation has two benefits:
First, frequency components are orthogonal, meaning that changing the signal's representa- tion one frequency does not affect the signal's representation at other frequencies. Thus, the Fourier transform provides an inductive bias for separating information across timescales.
Second, the frequency axis provides an intuitive, continuous axis for specifying timescales precisely. The user can select for patterns based on the timescales in Hz which they occur, instead of relying heuristic approaches for timescale specification.
diffusion Our approach merges the Fourier transform with deep representation learning: Fourier trans- autoencoder (Preechakul et al. 2022) to capture musical patterns, and a latent-space form to separate them by scale. To achieve synergy between these components, propose a simple end-to-end training framework: an encoder transforms audio into time series of latent vectors, which is randomly masked in the Fourier domain. Then, decoder attempts to use this frequency-masked latent sequence to reconstruct the audio with a diffusion based objective.
After training, we can encode sser-selected music into a sequence of latent vectors. Then, we can apply Fourier transform to this latent sequence, creating a latent spectrum. The latent spectrum maps different musical patterns to different frequencies in it, which we refer to as latent frequencies. These latent frequencies correspond to the timescales at which the musical patterns occur. The user can hear different parts of the latent spectrum in isolation, or generate variations while conditioning on patterns desired timescales, which are specified as latent frequencies. Separation between timescales also allows us to blend two musical examples together, retaining features at user-selected timescales from each. In short, we introduce novel frequency-basee based controls for generative models.
To explain these controls and their effects, we draw parallels between our framework and the equal- izer (EQ), an essential tool in audio signal processing. The equalizer manipulates the audible spec- frum, or the frequencies in the audio waveform within the limits of human hearing (20 20,000 Hz). This shapes sonic characteristics like "warmth," " "brightness," "'larity," and "shine," which re- late to different frequency ranges (Izhaki 2017, pp. 223-232). The equalizer is particularly crucial for mixing multiple musical elements together coherently, by highlighting frequencies from each element and ensuring that elements do not "clash" over similar frequency ranges (Owsinski 2017] Pp. 14, 160-161). Since the equalizer operates on audio waveform frequencies, is unable change musical or structural patterns (like notes or chords). These are more complex than waveform oscilla- tions, and unfold on temporal scales below 20 Hz, where such oscillations are inaudible. Still, these structural patterns are also vital to combining multiple musical elements together in a coherent way.
By operating on the latent spectrum instead of the audible spectrum, our framework provides a complement to the traditional equalizer that operates on musical patterns instead of sonic qualities. For instance, we can blend sounds together in musically coherent ways, while preserving patterns from each sound user-specified latent frequencies. This is akin to the way traditional EQs are used to mix sounds together in musically pleasant ways, by choosing which audible frequencies of each sound to highlight. We dub framework LATENTFT, and show several applications:
LATENTFT can generate musically coherent variations of given song, while preserving pat- terns at desired timescales. These timescales are specified as a mask over the latent frequency spectrum. (Sec. 4.2)
LATENTFT can blend two songs, preserving patterns from each desired timescales. These timescales are specified as masks over the latent frequency spectrum. (Sec. 4.3)
3. We can 'zoom-in' on parts of the latent spectrum, allowing to hear musical patterns at desired timescales, which are specified as latent frequencies (Sec. 4.5).
We can interpret the latent spectrum of a song, and show where various musical characteris- tics like genre, tempo, and pitch reside on the latent spectrum (Sec. 4.6).
We demonstrate these applications through quantitative metrics (Table 1.. listening tests (Sec. 4.4
and qualitative examples, which can be found on our websitt
https::/masse https//ss//eeennnoooooooo/o/totr
2

---

2 RELATED WORK
Audio Generation. Recent years have witnessed a great expansion in audio-domain generative models, which operate in continuous domain or by generating discrete tokens. Diffusion mod- els (Sohl-Dickstein et al. 2013
Ho et al. 2020 Song et al. (020) generate samples by iteratively denoising pure Gaussian noise. Other approaches to audio generation rely discrete audio codec
tokens (Zeghidour et al. 2021 Kumar et al., 2023), which compress audio into a multi-layer se-
quence of tokens, with successive layers capturing increasingly fine details. Token generation can proceed in an autoregressive (Borsos et al. 2223a| Copet et al. [023] AAgostinelli et al. (023) or non- autoregressive (Gareia et al.) 2023 Borsos et al. 2023b) manner, but in both cases, coarse tokens typically condition the generation of liner ones. We propose Fourier-based representations that let us condition on features at arbitrary scales. We compare our method to conditioning on intermediate
or fine tokens in our Masked Token Model baseline
Controls for Audio and Music Generation. Current audio generation methods offer global con-
trols like text (Forsgren & Martiros [022 Huang et al. 2023 Liu et al.] [223]] Copet et al.) 2023 [henn controls based on musical attributes like pitch and loudness curves (Wu et al. 2024 García et al.) 2025) or stems (Parker et al., 2024 Villa-Renteri et al. 2025). Different tme-varying signals allow for control along different semantic axes, but not along the 'timescale" axis. These works mostly condition on the entire control signal, not selected frequency components. The exception is Sketch2Sound (Garefa et al. 2025), which optionally smooths the pitch or loudness-based con- trol signal using median filtering. Still, this type of filtering is heuristic, applies only to preserving large-scale features, and operates on hand-extracted features instead of latent ones. Guidance (Levy et al. 2023) and initial noise optimization (Novack et al., 2024) have also been used to control music generation using differentiable objectives. We use guidance for our tasks in our Guidance baseline
Agostinelli et al. [023 [hen et al. 2024 Schneider et al. 2024 Evans et | [2025), or time-varying
Image Editing Frameworks. The coarse-to-fine paradigm lends itself to image editing frame- works that generate variations of input examples based on their low-frequency features. SDEdit (Meng et al. (021) enables stroke-based image generation and editing by adding white noise to a given reference (which acts like a heuristic low-pass filter), and running the denoising process. Similarly, Iterative Latent Variable Refinement (ILVR) (Choi et al. 2021) can generate variations of images while preserving large-scale structure. During the denoising process, ILVR continually replaces the low-frequency components of the noisy sample with the low-frequency components of
a (noised) reference, enabling image translation and stroke-based editing. ILVR does not condition on high-frequency or mid fid-frequency components, but we attempt this in our ILVR baseline
Fourier-Barr Based Deep Learning. While we apply the Fourier transform to latent vectors, many works use frequency- domain representations of the input or output space. These include works in vision (Lee et al. 2018 Yang & Soatto 2020: Atzzmn et al. 2024) and audio (San Roman al. 2023: Moliner al. 2024). Similar to method (Sec. 3). Zheng al. (2024b) propose frequency-maske autoencoder that extends the masked image modeling paradigm (He et al. 2022] Xie et al. 2022) to the frequency domain. AudioMAE (Huang et al.J (2022) applies masked image modeling to audio spectrograms, randomly masking time- free-freyeency bins in the audio spectrogram domain. However, our method masks latent-space frequencies.
Other works do apply the Fourier transform to hidden states, but do so as part of black- architec- tural units, and focus on downstream tasks instead of directly using the latent spectra. This use of the Fourier transform has been shown to improve learning in language (Lee-Thorg et al. [021 He
et al. 2023) and vision (Rao et al. 2021: Chi et al. 2020 Guibas et al. 2021 Lin et al. 2023).
Finally, some works apply the Fourier transform post-hoc to latent states of pretrained models, choosing and interpreting latent-space frequencies. PRISM (Tamkin et al., 2020) shows that differ- ent firqueeccy bands of language model embedding sequences are useful for different downstream tasks. In vision, Khan et al. (2017) shows that the spectra of intermediate activations in a pretrained CNN can be used to categorize scenes. These works focus analysis, while we focus synthe- sis: we can isolate frequencies in the latent representation, but also invert them and observe their realizations in the input domain. Applying frequency-doman domain manipulations post-hoc pretrained representations fails to synthesize coherent audio, which we show in the [AC and [RAVE] baselines and our ablations (Appendix B.I). This shortcoming motivates our frequency-masking strategy dur- ing training, which deliberately encourages our latents to be manipulable in the frequency domain.
3

---

Blending. LATENTFT can blend two examples together while choosing timescales from each (by selecting latent frequencies from each example). This is like style transfer in images (Ashikhmin,
2003 Gatys et al. 2016 Johnson al. 2016 Huang & Belongie 2017 Deng et al. 2022 Efros & Freeman 2023), which merges "content" from one image with the "style" from another. Applying these methods music is challenging due to the multiscale nature of musical style, style can refer to "high-level compositional features" or "low-level acoustic features" " (Dai et al. 2018). We ameliorate this ambiguity by introducing fuuroddcinggrequeeyyy-ubuu frequency-based controls, which provide a continuous axis for specifying which timescales we want from each input. In contrast, existing works in musical style transfer focus on specific aspects of music like timbre (Huang al.) 2018 Li al., 2024]
Wang et al. (2024), musical arrangement (Cifka et al. 2022)), or composition (SE, 2016). Traditional techniques are also used to blend sounds, as done in the Cross Synthesis baseline (Smith 2011).
METHOD
3.1
BACKGROUND
The discrete Fourier transform (DFT) cor- relates input signal E C. with N complex sinusoidal signals, giving its spectral representation X E CN The kth DFT coefficient is given by:
Discrete Fourier Transform.
[
Wk:
(DFT)
where (.) denotes the complex dot product, and [ = (2nk/N)) de- notes the leth complex sinusoid. The complex sinusoids w1,--, wN form an orthogonal basis for C.,, allowing the DFT to be inverted:
WN
--1
X X w
(IDFT)
N
The inverse DFT is also called the "synthesis" equation, since it expresses as weighted sum of complex sinusoids. To provide more concrete intu- ition, if æ is real-valued, we can express the sum of real sinusoids with various frequencies & N. amplitudes Ag. and phase shifts ou:
[N/2]
x mmn]
Σ k=O
A cos
2T k N" + on
(1)
h 8
Where A& and OR are both derived from the coefficient X(k). as shown in Appendix [.I] In words, the DFT can decompose real signal into sum of real sinusoids of different frequencies, all of which are mutually orthogonal. We show this decomposition for an example signal in Fig. [
Figure 1: æ e R
decomposed via Eq
Diffusion Autoencoders. The diffusion autoencoder was proposed by (Preechakul et al. 2022) to harness the power of diffusion models for representation learning. During training, encoder maps an image Zg into a non-spatial semantic vector zsem- Then, a diffusion model (which acts the decoder) tries reconstruct æp from Zsem and a noisy version of the image æ,. Diffusion autoencoders are typically trained with a MSE loss that determines how well æ; is denoised, (or cquivalently, how well æo reconstructed). During inference, Zsem can be used to condition generative diffusion process and produce an image.
We have three motivations for using a diffusion autoencoder. First, the decoder harnesses the gener- ative power of diffusion model, allowing it to generate high-quality music even when information has been removed (masked) from the latent conditioning vector. Second, since the generative pro- cess is random, one can generate multiple variations for the same input condition. Third, diffusion autoencoders have been shown to yield latent representations žeem that are semantically meaningful and linear, supporting interpolation between images and attribute manipulation. In fact, recent work shows the applicability diffusion autoencoders to music representation learning (Pasini et al., 2024 Bindi & Esling 2024).
We present simplified notation for the DFT for clarity and brevity. Note that we is typically presented as the complex conjugate of what have here.
4

---

Input
Variation
Mask
Encoder
Decoder
Rand.
SatentSpe
Masked Spec.
Add Noise
Reconstruction
Figure 2: Latent Fourier Transform (LATENTFT), We encode audio (which may be represented as waveform or spectrogram) into series of latent vectors and compute latent spectrum. During training (red), this spectrum is masked randomly and used to reconstruct the input. During inference (blue), the user specifies a spectral mask, which selects features from the input at specific latent frequencies and conditions generative process.
3.2 МЕTHОО OVERVIEW
Our goal two-fold. First, we want to map an audio waveform or spectrogram By into a time series of latent vectors, whose spectrum encodes semantic patterns. We refer to the DFT spectrum of this latent time series as the latent spectrum. It is important to distinguish the latent spectrum from the audible spectrum: The audible spectrum refers to the DFT spectrum of the audio waveform, and cap- tures variations in the waveform occurring at different frequencies. In contrast, the latent spectrum captures variations in the latent time series occurring different frequencies, which correspond to musical patterns occurring at different timescales. Second, we should be able isolate features at selected latent frequencies and use them to generate variations, blend them with other audio clips, or hear them in isolation. These goals motivate an end-to-end encoder-decoder architecture that encodes music into latent spectra, and decodes latent spectra into music. We apply latent Fourier transform and requenncymaskinn during truining, shown in Alg. 1 and Fig. 2
3.3 ENCODING THE LATENT SPECTRUM
Encoder. An encoder maps input music æo E RCT to a time series of latent vectors E RC XT',
Enca (xo)
Here, C' and C' are the number of input and la- tent channels, while T and T" are the number of input and latent timesteps. Although 7 and 7" do not have to be equal, x must have a linear tempo- ral axis in order produce latent spectrum. This favors convolutional architectures or networks like the U-Net (Ronneberger et al. 2015), whose skip connections promote input-output alignment. We define as the latent frame rate in Hz, the num-
Algorithm I Training.
Input: Audio Waveform or Spectrogram ang I: x + Enc, (2x)
3: 2: 2 + DFT(2)
1)
Sample threshold
4: NOO, E) Saampl froquency bin scores
5: M 1azn
D Get Mask
6: Z nadded
20 M
7: 2 masked + IDFT (muniod)
ber of latent vectors (frames) needed to represent one second of audio.
8: Sample noise level p(7)
Forward (æo, T) Add noise 10: do + Docs (=umdded xr,7) 7) & Reconstruct Zo
9:
11: f + L (āoo #0)
Latent Fourier Transform. The latent spec- trum rrfeers to the DFT of the latent timeseries z, applied to each channel in the latent timeseries:
12: Update parameters 6, using Va.et
Z DFT(2)) Z CO'
Applying the DFT along the time axis of our latent sequence represents each latent channel sum of K = [ - sinusoids (see Eq. [].. The sinusoids have K diffeerntt linearly-spaced frequencies, which capture variations in each Tatent channel at different temporal rates. The kth sinusoid completes & cycles in T" latent timeframes (see Fig. I.. For instance, the Oth sinusoid is constant, the 1st sinusoid has a period of T" latent frames, and the 2nd sinusoid has a period T latent frames. The sinusoids are also orthogonal from one another, creating an inductive bias for separating information across timescales.
"The DFT is different from the Short-time Fourier Transform, which yields a spectrogram, not spectrum.

---

Specifically, D=T(=) stores K complex coefficients indicating the amplitude and phase of each sinusoid along a length- K frequency axis. We refer to the frequency-axis of DFT(=) as the latens frequency axis, and we call points along this axis latent frequencies. Like audible frequencies, latent frequencies are described in Hz. However, Hz on the latent spectrum corresponds to oscillations in the latent sequence occurring at 1 cycle per second, instead of oscillations in the audio waveform. The kth sinusoid has a period of T /k latent frames or T" (kfrr seconds, and thus a latent frequency of fü kff/TT Hz.
Increasing Spectral Granularity. In practice, we zero-pad x at its end, expanding its temporal length by factor of L. This increases the number of frequency bins by a factor of zs L, allowing for more spectral granularity via spectral interpolation (Smith) 2007). This is especially useful for capturing very low-frequency patterns (below cycle per timeframes). We let F LTT' /2 + be the number of spectral bins (sinusoids) after zero padding z.
3.4 FREQUENCY MASKING
At inference, we want to select specific frequencies from the latent spectrum to generate variations from them or hear them isolation ('zoom-in' on them). This is accomplished by applying latent spectral mask M E {0. 1). taking maakeed = Z M. During inference, this mask is chosen by the user. During training, this mask is randomizedl: First, we sample random scalar threshold ? ~ (0, 1), which helps decide the proportion of bins to be masked. Second, we sample 8 ~ (OFF Σ), where E R assigns scores to each frequency bin. Third, we set the mask to keep bins whose score is greater than the threshold, setting M = 1827
Random Threshold. Using a random threshold ensures uniform distribution over the number of masked bins. contrast, independently masking each frequency bin with probability p corresponds to setting fixed threshold and E = T. This results in binomial distribution over the number of masked bins, which does not reflect the inference-time distribution of user-specified masks.
Correlating Bins. Instead of masking each frequency bin independently, we create a "soft group- ing" between nearby frequency bins by correlating their scores. This is done by multiplying uncor- related scores u ~ NO0p. I) with a radial basis function matrix K:
K,, = c, exp (-
|0: 10-04 ag
K KER
(4)
20F
where a; = log(Ji + <) is the frequency of bin i mapped to a logarithmic axis, p.a, and are hyperparameters, and & normalizes each row of K to have unit l2 norm. Multiplying a = Ku results in correlated scores between frequency bins, where the amount of correlation between two frequency bins is determined by their distance on a logarithmic axis. The covariance matrix of is Σ KKT. Ablations (Appendix B.1 show that correlating bin scores is key our method's 's per- formance. Intuitively, masking frequency bins independently forms speckled masks where masked bins are often adjacent to unmasked ones. The unmasked bins provide strong local cues about nearby masked ones, reducing the model's ability to fill in contiguous regions of the latent spectrum dur- ing inference. In contrast, correlated bin scores form masks with larger contiguous regions, which combats the effect of spectral leakage and better reflects inference-time. time, user-specified masks.
Logarithmically scaling the frequency-axis is also key performance (Appendix B.I This is com- mon in audio, exemplified by the Mel scale (Stevens et al.) 1937). Constant Q- Transforrm (Brown) 1991). and others. More generally, structured signals from images (San Roman et al., (023) to coastlines and mountains (Bak et al.) 1987) have spectra that follow 1/ curve. Segmenting such spectra into groups of equal width along log-frequency axis yields groups of roughly equal energy. This motivates our logarithmic scaling, where higher frequencies are more likely to form larger groups. Lastly, normalizing the rows of K ensures equal marginal variance between every bin score 8g. so that all bins have the same marginal probability of being masked for any given threshold.
3.5 DECODING THE LATENT SPECTRUM
We transform Z masked back into the time domain by applying the inverse DFT, obtaining a frequency- masked latent sequence masked IDFF Z masked) The decoder then uses masked reconstruct the input 2o from a noisy version of it (training), or to condition a diffusion process (inference).

---

Training. During training, we obtain noisy version 2, of the input Zy through forward diffu- sion process. This process samples a diffusion time T ~ p(T) from a predetermined distribution and adds a r-dependent amount of noise to zo. We supply 2 masked and x, to the decoder, which gives an estimate of the clean input in!
Lo f- Deco (zmasked, [r, T)
(5)
Then, we compute a reconstruction loss l L (d0,, Tg), which is used to update the parameters é, f of both the encoder and decoder. This procedure effectively trains a diffusion model, which can generate new outputs conditioned on masked While we do not require a particular diffusion framework, in practice we follow the ODE formulation in Karras et al. (2022). This framework preconditions the model inputs and outputs, uses approximately linear diffusion trajectories, and applies a second-order correction at each sampling step (omitted in Algs. and for clarity).
Conditional Generation. Our conditional gener- ation task attempts to generate a variation of ref- erence song y that preserves characteristics at user- specified latent frequencies. The reference y is en- coded and masked in the latent frequency domain to obtain masked The mask is user-speciffed, and typ- ically selects low frequencies, high frequencies, or a band of intermediate frequencies. We use masked to condition a reverse diffusion process, which iteratively denoises pure Gaussian noise to yield a new variation.
Algorithm 2 Conditional Generation
Input: x manked J decreasing
I: x ~ NN0 o
2: for i E {0, ..... N do
Lo Decs (2 musie
2. r.)
4: d + (xr ao) /o: Deriv of Noise Traj.
5: x + x + (T++1 - Tz)
return x
Algorithm 3 Blending
Blending. Our blending task attempts to combine two musical references y/, Y2 into new song that pre- serves characteristics from each at user-specified la- tent frequencies. Like before, 21, 22 are obtained and masked in the latent frequency domain to get condi- tions maakeed and 2 Here, the user specifies two masks specifying which latent frequencies to retain from each input. We obtain our blend by simulating the reverse diffusion process, at each step interpolat- ing the derivatives induced by each condition (Alg. 3).
Input: #1 munked musked (Ti) weights a, 3
#2
1: x ~ NíO, m
2: for E (O...., N do
#b < Decs (2T (2nuddad a. Ti)
4:
ab + Deng (z) (zmasted n. Tc)
5:
dj + (x a 1//0
da t (a a. 221
7:
d t ad, + 8d2
8:
x + x + (fi+1 7:) d
return x
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets and Training. We train three versions of LATENTFT with three different encoders. We use UNet and MLP encoders with a mel-spectrogram frontend, as well as raw audio encoder that utilizes a Descript Audio Codec (DAC) Kumarr et al. 2023) frontend. These encoders are described further and compared in Appendix I Each model generates mel-spectrograms, which are inverted using the Big BGAN neural vocoder (Lee et al. 2022). We train our model on MTG-Jamendo (Bog- danov al. 2 large-scale collection of over 55,000 songs spanning diverse musical genres (described more in Appendix (A5)) segmented into 5,9-second musical clips. Hyperparameters for the decoders and training are Appendix A.2 and A.3 respectively.
Baselines. We compare LATENTFT to various traditional and learned methods of generating or representing audio. First, we try several generation baselines, adapting them to our task:
Masked Token Model (Garcia et al. 2023). We use the Vampnet masked token model, which generates discrete acoustic tokens from coarse-to-fine. Vampnet is trained to predict all acous- tic tokens given a random subset of them, and supports supplying arbitrary token masks during inference. For conditional generation, condition on, and for the blending task, we select a different layer to take from each reference.
select different contiguous subsets of RVQ layers to
Guidance (Levy et al. 2023). We generate mel-spectrograms with an unconditional diffusion model. At each denoising step, we compute the DFT along the time axis of the reference
7

---

spectrogram(s) and the current reconstruction ëy. We compute the loss between these DFT spectra within the selected frequency bins, using it to update the intermediate output.
ILVR (Choi et al. 2021). We generate speestrets from an unconditional diffusion model. At each denoising step, we compute DFT spectra of the intermediate output and the reference(s) set to the current noise level. We replace selected DFT frequencies of the inter- mediate output with the corresponding DFT frequencies of the noisy reference(s).
Cross Synthesis Smith 2011). Cross synthesis blends two sounds by replacing the spectral envelope of one sound with that of the other. We follow the implementation in Smith (201T)
In the Guidance and ILVR baselines, note that we use the spectrum of the mel-spectrogram to steer the diffusion process instead of the latent spectrum. We also attempt post-hoc frequency-domain filtering of existing representations of audio for our tasks, similar to Tamkin et al. (2020):
DAC (Kumar al., 2023). We encode our reference(s) using Descript Audio Codec, a popular deep neural audio codec. We frequency-mask the latent states post-quantization, and feed the filtered latent sequence to the decoder to produce audio.
RAVE (Caillon & Esling 2021 offers another latent representation of the audio signal, which is often manipulated in the latent space and used to generate audio (Nabi et al. [024] iZhen (et al., 2024a). Similar to DAC, we frequency-mase the latent states obtained from the RAVE encoder, then provide them to the decoder.
Spectrogram. We filter the input mel-spectrogram representation(s) directly, by computing the DFT of the mel-speetroorram()) along the time axis, then masking the DFT(s). We convert the filtered mel-spectrograms to audio with BigVGAN (Lee et al. 2022).
In each case, blend by taking selected frequency components from two latent representations derived from two inputs, by adding the two frequency-masker latents together before decoding.
4.2 CONDITIONAL GENERATION
We show that LATENTFT can generate variations of given song while preserving patterns at user- specified timescales. We take 1024 random 5.9-second clips from the MTG-Jamendo test set, en- suring each clip originates from a unique song (results on more datasets in Appendix B.2]. We then generate variations of each clip, conditioning on 14 different latent frequency bands of varying widths and locations (see Appendix A.6)) Good variations should adhere to the condition, preserv- ing input characteristics at the specified timescales, and have musically coherent, high-quality audio.
Metrics. To measure adherence, we extract time-series descriptor signals (e.g., loudness curves) from both the input and generated audio. We bandpass these descriptor signals to the selected frequency band, and measure their similarity or eиror with standard metrics. We select four percep- tually relevant time series descriptors. First, we extract loudness curves following Morrison et al. (2024), and quantify their similarity using their correlation coefficient (Kosta et al.) (016)). Second, we quantify rhythmic preservation by computing onset strength envelopes (Böck & Widmer 2013) and measuring their beat-spectral cosine similarity (Foote et al., 2002). Third, to quantify timbral preservation, we extract Mel-Frequency Cepstral Coefficients and compute Mel-Cepstral Distortion (Kominek et al.) 2008). Fourth, we measure harmonic characteristics (relating to chords and music notes) by computing tonal centroid features, and quantify error using Tonnetz distance (Milne & Holland 2016). We measure audio quality by computing the Frechet Audio Distance (Kilgour al., 2018) between the set of generated music and the MTG-Jamendo validation set.
Results and Analysis. We recommend listening to the qualitative results, which are available on the website and show our variations are diverse and musically interesting. Quantitative results are in Table I Our model outperforms all basclines in terms of adherence, indicating that the latent spectrum captures and reproduces variations in loudness, rhythm, timbre, and harmony occurring at selected timescales. We also surpass all baselines in terms of quality. Our metrics confirm that (1) previous audio generation models cannot condition on features from arbitrary timescales, and (2) previous representations of audio are not robust to post-hoc spectral modifications.
8

---

Conditional Generation
Blending
Adherence
Quality
Adherence Both Inputs
Quality
Loud.
Rhyth.
Timb. 4 Harm.
4
Rhyth. f
Timb
Harm.
FAD
Masked Tok.
4.317
6.033
Guidance
0.529
0.813
1.430
0.099
1.061
0.557
0.832
1.607
0.114
1.466
ILVR
0,575
0.839
0.781
0.100
1.537
0.624
0.858
0.825
0.112
2.696
DAC
0.661
0.838
4.064
0.209
7,016
0,550
0.792
3.980
0.236
6.257
RAVE
-0.016
0,718
3,836
0,180
4,695
-0.006
0.697
4,439
0.171
4.478
Spectrogram
0.366
0.858
2,104
0.139
7.608
0.272
0.824
2,975
0.128
7.021
Cross Synth.
2.447
LATENTFT-MLA
0.815
0.963
0.376
0.079
0.337
0.686
0.873
1.021
0.108
1.387
LATENTFT-UNet 0.834
0.966
0.391
0.079
0.348
0.686
0.878
1.118
0.109
1.357
LATENTFT DAC
0.878
0.922
1.390
0.915
0.131
1.364
Table 1: Results on Conditional Generation and Blending on the MTG-Jamendo Test set. Mel- Cepstral Distortion (Timbre) is divided by 100. Compared to baselines, LATENTFT variants achieve superior adherence and audio quality. The Masked Token Model and Cross Synthesis baselines do not offer frequency-base based controls, so we do not compute adherence. Cross Synthesis also only applies to the blending task.
4.3 BLENDING
Setup. We show that LATENTFT can blend two songs together, while preserving patterns from each at user-specified latent frequencies. This application is motivated by the traditional equal- izer, whose primary use is to promote coherence between tracks by emphasizing different audible frequencies from each of them. The experimental setup is similar to the conditional generation ex- periment. However, instead of selecting single latent frequency band from a single song, select two non-overiapping bands from two songs (details in Appendix (A.6). We then measure the blended song's 's adherence each song with respect its selected subband, and average the two. To ensure that the blending is successful and musically coherent, we also report the FAD.
Analysis. We provide examples of blending on the website and quantitative results are shown in T The blending task requires adherence-quality tradeoff, since adhering to both conditions perfectly may not result in pleasant audio. Since ILVR iteratively replaces frequency components of the output with those of the conditions, it has slightly better adherence score on the timbre metric, while being worse in terms of quality. ILVR also loses to LATENTFT in user studies by a substantial margin (Fig- 3 in terms of both audio quality and ability blend. In general, LATENTFT can better adhere to two conditions simultaneously compared to baselines, and generates higher-quality audio. The ability to adhere to disjoint frequuncy components from two reference examples also indicates that the latent spectrum separates information by timescale to some extent.
4.4 LISTENING STUDY
To validate our method against human preferences, we conduct a listening study comparing LA- TENTFT and three other systems on the blending task. We choose a discrete method (the Masked Token Model baseline), diffusion-based method (ILVR), and traditional method (Cross Syn- thesis) to compare with LATENTFT. We recruited 29 musicians to complete a 12-question survey comparing every ordered pair of systems. For each question, participants first heard two randomly- selected music clips from the MTG-Jamendo test set. They then heard two blendings of the music clips, each produced by a different system. Participants rated which blending they preferred in terms of (i) audio quality and (ii) how well the clips were merged, using two separate 5-point Likert scales. Fig. 3 shows that our model outperforms the baselines both metrics. Additional details about the listening study and statistical analyses of the results can be found in Appendix A.7
4.5 HEARING LATENT FREQUENCIES ISOLATION
LATENTFT can 'zoom in' 'boost' patterns at specific latent frequencies, analogous to how audio engineers boost various audible frequencies to identify interesting or problematic regions (Izhaki) 2017 p. 265). We show this in Fig. 4 The first spectrogram shows electronic music clip, con- taining patterns at various timescales. The second spectrogram boosts latent frequencies between and 1 Hz, which removes rapid drum patterns (vertical lines the top of the spectrograms) and

---

MТM
Reference
8.5
8192
Crass.
4096
V H Accemmmared
25
50
75
100
125
Audio Quality Num. Wins
2048
MTM
ILVR
Cross.
512
Ours
25
50
75
100
Ability to Blend Num. Wins
Time (5)
Bass
Time (s)
Bass Reduced
Time (s)
Figure Listening study Figure 4: Isolating frequencies from an electronic music clip. We with pairwise comparisons. show three audio spectrograms. The second spectrogram smooths the We achieve the most head-to- reference spectrogram, and the third accentuates patterns occurring at head wins on both criteria. 8 Hz while removing lower-frequency patterns, like the bass.
bass patterns (near the bottom), and makes the spectrogram notably smoother along the horizontal (time) axis. The third spectrogram boosts latent frequencies between 7.5 and 8.5 Hz. This accen- tuates a pattern the original song occurring at 8 Hz, seen by comparing the vertical lines in the third spectrogram with those in the first. Also, the third spectrogram does retain the rhythmic patterns of the bass, which occur below 7.5 Hz. This can be seen by comparing the lower regions of spectrograms one and three. LATENTFT allows for performing low-pass and high-pas figh-pass oper- ations on music representations while retaining musical coherence. Low-passing or high-passing spectrograms directly along their time axes cannot do this (Table I We achieve isolation using self-blending procedure described in Appendix A.8
4.6 INTERPRETING THE LATENT SPECTRUM
Musical concepts like genre, tempo, pitch, and chord changes
Rock Song
distributed across different regions of a song's latent spec-
Preservation
trum, analogous to how different sonic characteristics occupy distinct ranges of the audible spectrum. Given song, generate many variants while performing a sweep through the frequencies condition preservation of genre (using classifier). chord progression, predominant pitch, and tempo, with respect the original song. We plot how well the variation preserves these traits against the frequency we condition on, applying smoothing. Fig. shows these traits are distributed across the latent spec- trum differently, Genre is a more global feature; chords change at latent frequencies below Hz; and predominant pitch and tempo reside at higher frequencies, tending to multiples the GTZAN Tzanetakis & Cook (2002) dataset, since contains ground- truth genre labels. More details about how these preservation curves computed are Appendix [A.9 Also, we interpret the latent spectra of more songs of various styles in Appendix B.3
For each variant, measure
Wich
0.0
Jazz Song
1.0
Preservation
0.0
1.0
2.0
8.0
Frequency (Hz)
of the song's 's BPM. For this experiment,
Figure 5: Preservation curves in- dicating where tempo, pitch, genre reside in in the latent spectra two reference songs.
CONCLUSION
In this work, introduced the Latent Fourier Transform, which provides novel frequency-based controls for generative models. We showed applications in conditional generation and blending in the domain of music. Future work should include enabling real-time interactivity, or disentangling the latent spectrum along semantic axes, combining both timescale-based and semantic controls.
10

---

KEPRODUCIBILITY STATEMENT
To promote reprodueibility, we include code for training, generating, and blending examples from LATENTFT. We also include all baseline implementations and our experiment pipeline for the conditional generation task, the blending task, code for generating sweeps for the interpretability experiment (Sec. (.6)) and code for isolating frequency components (Sec. 4.5). We also include all our model urchitectures, training configurations and hyperparameters (Appendix A). and code for replicating the model architecture. Finally, we include code for preprocessing the datasets we used.
REFERENCES
Andrea Agostinelli. Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023.
Ashikhmin. Fast texture transfer. IEEE computer Gruphics and Applications, 23(4):38-43, 2003.
Yuval Atzmon, Maciej Bala, Yogesh Balaji, Tiffany Cai, Yin Cui, Jiaojiao Fan, Yunhao Ge, Sid- dharth Gururani, Jacob Huffman, Ronald Isaac, et al. Edify image: High-quality image generation with pixel space laplacian diffusion models. arXiv preprint arXiv:2411.07126 2024.
Philip Bachman. An architecture for deep. hierarchical generative models. Advances in Neural Information Processing Systems, 29, 2016.
Per Bak, Chao Tang, and Kurt Wiesenfeld. Self-organized criticality: An explanation of the 1/f noise. Physical review letters, 59(4)3381, 1987.
Giovanni Bindi and Philippe Esling. Unsupervised composable representations for audio. arXiv 2024.
preprint arXiv:2408.098
Sebastian Böck and Gerhard Widmer. Maximum filter vibrato suppression for onset detection. In Proc. of the 16th Int. Conf. on Digital Audio Effects (DAFx). Maynooth, Ireland (Sept 2013), volume 7, pp. 4. Citeseer, 2013.
Dmitry Bogdanov, Nicolas Wack, Emilia Gómez, Sankalp Gulati, Perfecto Herrera, Osear Mayor, Gerard Roma, Justin Salamon, José Zapata, and Xavier Serra. Essentia: an open-source library for sound and music analysis. In Proceedings of the 21st ACM international conference on Mul- timedia, pp. 855-858, 2013.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The jateedd dataset for automatic music tagging. ICML, 2019.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. IEEF/ACM transactions on audio, speech, and language processing, 31:2523-2553 2023a.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.096. 2023b.
Judith C Brown. Calculation of constant q spectral transform. The Journal of the Acoustical Society of America, (9(1):425-434 1991.
Antoine Caillon and Philippe Esling. Rave: A variational autoencoder for fast and high-quality
neural audio synthesis, 2021.
Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor
and Shlomo Dub- nov. Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pp. 1206-1210 IEEE, 2024.

---

Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. Advances in Neural Information Processing Systems, 33:4479 4488, 2020.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. arXiv preprint Sit::2000.009338, 2021.
Ondiej Cifka, Umut Şimşekli, and Gaël Richard. Groove2groove: One-shot music style transfer with supervision from synthetic data. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2638-2666 2020.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36:47704-44737 2023.
Shuqi Dai, Zheng Zhang, and Gus G Xia. Music style transfer: A position paper. arXiv preprint arXiv: 06841, 2018.
Yingying Deng, Fan Tang, Weiming Dong. Chongyang Ma, Xingjia Pan, Lei Wang. and Changsheng Xu. Stytr2: Image style transfer with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11326-11336, 2022.
Caiwen Ding, Siyu Liao, Yanzhi Wang, Zhe Li, Ning Liu, Youwei Zhuo, Chao Wang, Xuehai Qian, Yu Bai, Geng Yuan, et al. Cirenn: accelerating and compressing deep neural networks using block-circulant weight matrices. In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, pp. 395-408, 2017.
Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In Seminal graphies papers: pushing the boundaries, volume pp. 571-576. 2023.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Jonathan Foote, Matthew Cooper, and Unjung Nam. Audio retrieval by rhythmic similarity. In ISMIR, 2002.
Seth Forsgren and Hayk Martiros. Riffusion Stable diffusion for real-time music generation, 2022.
URL http:hhtt:///.ttt.. fuus www.iono......o wwt.ccom/aaaaa /boutt
Hugo Flores Garcia, Prem Seetharaman, Rithesh Kumar, and Bryan Pardo. Vampnet: Music gener- ation via masked acoustic token modeling. arXiv preprint arXiv:2307.04666, 2023.
Hugo Flores Gareía, Oriol Nieto, Justin Salamon, Bryan Pardo, and Prem Seetharaman. Sketch2sound: Controllable audio generation via time-varying signals and sonic imitations. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 2414-2423, 2016.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catan- zaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587, 2021.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. arXiv preprint arXiv:1611.0501. 2016.
Takuya Hasumi, Tatsuya Komatsu, and Yusuke Fujita. Music tagging with classifier group chains. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEFE, 2025.
12

---

Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel. and Douglas Eck. Enabling factorized piano music modeling and generation with the maestro dataset. arXiv preprint 1r10v:1810.1222 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Ziwei Hc, Meng Yang. Minwei Feng, Jingcheng Yin. Xinbing Wang, Jingwen Leng, and Zhouhan Lin. Fourier transformer: Fast long range modeling by removing sequence redundancy with fit operator. arXiv preprint arXiv:2305. 15099, 2023.
Hendrycks. Gaussian error linear units (gelus). arXiv preprint arXiv: /606:6606011111 2016.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif Saurous, Bryan Seybold, al. Cnn architectures for large-scale audio classification. In 2017 ieee international conference acoustics, speech and signal processing (icassp), pp. 131-135. IEEE, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Po- Yao Huang, Hu Xu, Juncheng Li, Alexci Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Infor- mation Processing Systems, 35:28708 04228788-28720, 2022.
Qingqing Huang, Daniel S Park, Tao Wang, Timo Denk, Andy Ly. Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03911 2023.
Sicong Huang, Qiyang Li, Cem Anil, Xuchan Bao, Sageev Oore, and Roger B Grosse Tim- bretron: A wavenet (cyclegan (eqt (audio))) pipeline for musical timbre transfer. arXiv preprint arXiv:1811.09620 2018
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. Proceedings of the IEEE international conference on computer vision, pp. 1501-1510, 2017.
Roey Izhaki. Mixing audio: concepts, practices, and tools. Routledge, 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neurul information processing systems, 35:26365-26577 2022.
Salman H Khan, Munawar Hayat, and Fatih Porikli. Scene categorization with spectral features. In Proceedings of the IEEE international conference computer vision, pp. 5638-5648, 2017.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr', 'echet audio distance: A metric for evaluating music cnhancement algorithms. arXiv preprint 1rXiv:1812.08466 2018.
John Kominek, Tanja Schultz, and Alan W Black. Synthesizer voice quality new languages calibrated with mean mel cepstral distortion. In SLTUU, pp. 63-68, 2008.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09661, 2020.
13

---

Katerina Kosta, Rafael Ramírez, Oscar F Bandtlow, and Elaine Chew. Mapping between dynamic markings and performed loudness: a machine learning approach. Journal of Mathematics and Music, 10(2):149- 172, 2016.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36:27980-279927 27993, 2023.
J Richard Landis and Gary Koch. The measurement of observer agreement for categorical data. biometrics, pp. 174, 1977.
Luca A Lanzendörfer, Florian Grötschla, Michael Ungersböck, and Roger Wattenhofer. High- fidelity music vocoder using neural audio codecs. In ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Jae-Han Lee, Minhyeok Heo, Kyung-Rae Kim, and Chang-Su Kim. Single-image depth estimation based on fourier domain analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 330-339, 2018.
Sang-gil Lee, Wei Ping. Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.0400 2022.
James Lee-Thorp, Joshua Ainslie, Ilya Bckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint 105.84210 2021.
Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, and Tom Nickson. Con- trollable music production with diffusion models and guidance gradients. arXiv preprint DxXiv:2311.00611: 2023.
Sifei Li, Yuxin Zhang, Fan Tang, Chongyang Ma, Weiming Dong, and Changsheng Xu. Music style transfer with time-varying inversion of diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 547-555, 2024.
Shiqi Lin, Zhizheng Zhang, Zhipeng Huang, Yan Lu, Cuiling Lan, Peng Chu, Quanzeng You, Jiang Wang, Zicheng Liu, Amey Parulkar, et al. Deep frequency filtering for domain generalization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
11797-11807, 2023.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 2023.
Michael Mathicu, Mikael Henaff, and Yann LeCun. Fast training of convolutional networks through fits. arXiv preprint arXiv:1312.585// 2013.
Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. SciPy, 2015:18-24, 2015.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint 01100 2021.
Andrew J Milne and Simon Holland. Empirically testing tonnetz, voice-leading, and spectral models of perceived triadic distance. Journal of Mathematics and Music, 10( 10(1):59-85, 2016.
Eloi Moliner, Maija Turunen, Filip Elvander, and Vesa Välimäki. A diffusion-based generative equalizer for music restoration. arXiv preprint arXiv:2403. 18636, 2024.
Max Morrison, Cameron Churchwell, Nathan Pruyne, and Bryan Pardo. Fine-grained and inter- pretable neural speech editing. arXiv preprint arXiv:2407.05477, 2024.
Sarah Nabi, Philippe Esling, Geoffroy Peeters, and Frédéric Bevilacqua. Embodied exploration of deep latent spaces in interactive dance-music performance. Proceedings of the 9th International Conference Movement and Computing, pp. 1-9, 2024.
14

---

Zachary Novack, Julian McAuley, Taylor Berg- Kirkpatrick, and Nicholas Bryan. Ditto: Diffusion inference-tinee serence-timeett t-optimization for music generation. arXiv preprint arXiv:2401.12179, 2024.
Bobby Owsinski. The mixing engineer's handbook. BOMG Publishing Burbank, CA, 2017.
Julian Parker, Janne Spijkervet, Katerina Kosta, Furkan Yesiler, Boris Kuznetsov, Ju-Chiang Wang, Matt Avent, Jitong Chen, and Duc Le. Stemgen: A music generation model that listens. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1116-1120. IEEE, 2024.
Marco Pasini, Stefan Lattner, and George Fazekas. Music2latent: Consistency autoencoders for latent audio compression. arXiv preprint arXiv:2408.06500 2024.
Pascal Passigan and Vayd Ramkumar. Analyzing the effect of k-space features in mri classification models. arXiv preprint arXhv:2409.13588 2024.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakorn. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. (0619-1062)) 2022.
Kai Qiu, Xiang Li, Hao Chen, Jie Sun, Jinglu Wang, Zhe Lin, Marios Savvides, and Bhiksha Raj. Ef- ficient autoregressive audio modeling via next-scale prediction. arXiv preprint arXiv:2408. 09027, 2024.
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advances neural information processing systems, 34:980-993, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Justin Salamon, Emilia Gómez, Daniel PW Ellis, and Gaël Richard. Melody extraction from poly- phonic music signals: Approaches, applications, and challenges. IEEE Signal Processing Maga- zine, (1))): 10:1188-144 2014.
Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Défossez. From discrete tokens to high-fidelity audio using multi-band diffusion. Advances in Neural Information Processing Systems, 36: 1526-1538, 2023.
Flavio Schneider, Ojasv Kamal, Zhijing Jin, and Bernhard Schölkopf. Moñsai: Efficient text-to- music diffusion models. In Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguisties (Volume Long Papers), pp. 8050-8068, 2024.
SAP SE. Musical style modification as an optimization problem. In Proceedings of the International Computer Music Conference, pp. 206, 2016.
Julius O Smith. Mathematics of the discrete Fourier transform (DFT): with audio applications.
Julius Smith, 2007.
Julius O Smith. Spectral audio signal processing. Julius Smith, 2011.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learn- ing. pp. 2256-2265 pmlr, 2015.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456. 2020.
Stanley Smith Stevens, John Volkmann, and Edwin Broomell Newman. A scale for the measurement of the psychological magnitude pitch. The journal of the acoustical society of america, 8(3):185- 190, 1937.
15

---

Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. arXiv preprint arXiv:1806.03168 2018.
Alex Tamkin, Dan Jurafsky, and Noah Goodman. Language through prism: A spectral approach for multiscale language representations. Advances in Neural Information Processing Systems, 33: 5492-5504, 2020.
George Tzanetakis and Perry Cook. Musical genre classification of audio signals. IEEE Transactions on speech and audio processing, 10(5):293-300 302, 2002.
Ivan
Mason Long Wang, Zachary Shah, Zhe Li, Soohyun Kim, Neelesh Ramachan- dran, and Mert Pilanci. Subtractive training for music stem insertion using latent diffusion models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 1-5. IEEE, 2025.
Hechwan Wang, Joonwoo Kwon, Sooyoung Kim, Shinjae Yoo. Yuewei Lin, and Jiook Cha. A training-free approach for music style transfer with latent diffusion models. arXiv preprint arXiv:2411.15913. 2024.
Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and Nicholas J Bryan. Music controlnet: Multiple time-varying controls for music generation. IEEE/ACM Transactions Audio, Speech, and Language Processing, 32:2692-2703. 2024.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp- 3-19, 2018
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9653-9663, 2022.
Donghhao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions Audio, Speech, and Language Processing. 31:1720-1733 2023.
Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4085- 4095, 2020.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from generative 2017.
models. arXiv preprint
Shuoyang Zheng, Anna Xambó Sedó, and Nick Bryan-Kinns A mapping strategy for interacting with latent audio synthesis using artistic materials. arXiv preprint MXXi:22407.04379 2024a.
Tianyi Zheng, Bo Li, Shuang Wu, Ben Wan, Guodong Mu, Shice Liu, Shouhong Ding, and Jia Wang. Mfae: Masked frequency autoencoders for domain generalization face anti-spoofing. IEEE transactions on information forensics and security, 19:4058-4066 2024b.
16

---

APPENDIX TABLE OF CONTENTS
Experimental Details
18
A.I
Encoders
18
A.1.1
MLP Encoder Architecture
18
A.1.2
ID U-Net Encoder Architecture
19
A.1.3
DAC Encoder Architecture
19
A.2
Decoders/Diffusion Model Architecture
19
A.3
Training Details
20
A.4
Other Hyperparameters
................................
20
A.5 Datasets
20
A.6 Conditional Generation and Blending Experiments
21
A.7 Listening Study Details and Analysis
A.8 Isolation Experiments
22
A.9 Interpreting the Latent Spectrum.
22
Additional Experiments
24
B.1 Ablations
24
B.2 Results More Datasets
27
B.3 More Interpretability Results
28
B.4 Removing the Latent DFT
29
B.5 Per-Band Error
C Additional Related Work
30
D Extended Background
31
D.I DFT
31
E LLM Usage
32
17

---

A EXPERIMENTAL DETAILS
Below, we describe our experiments more detail. We provide code for training and evaluating LATENTFT our publicc github repository
A.1 ENCODERS
We experiment with three encoders:
1. MLP Encoder. The audio is converted into 80 512 mel-spectrogrram. Each (80 1) timeframe passed through an MLP to obtain an 80 x 512 latent sequence. Since each timeframe is processed independently, this encoder enforces input-output alignment, and results no leakage between timeframes.
2. ID U-Net Encoder. The audio first converted into an x 512 mel-spectrogram.. This processed by ID U-Net (Ronneberger et al., 2015 Stoller et al. 2018) with convolutions along the temporal axis to obtain an 80 x 512 latent sequence. While this encoder does not entirely prevent leakage between frames, the U-Net's cxip-connections promote input- output alignment, allowing the encoding to be interpreted as a temporal sequence.
3. DAC Encoder. We use the encoder from the pretrained Descript Audio Codec (Kumar et al. 2023) model to extract 1024 X 512 embeddings from the raw audio waveform. We then pass these embeddings to a ID U-Net encoder that identical to the one described above, except for the first convolutional layer, which is expanded to have 1024 input chan- nels instead of 80 to accommodate the number of latent channels DAC.
We find qualitatively that the U-Net encoder produces more pleasant-sounding audio when listening latent frequencies in isolation. We also find that the U-Net encoder is better for blending, while the MLP Encoder is better for conditional generation. The DAC encoder's waveform frontend requires significantly more GPU memory, which required reducing the batch size from 256 to 64 during training. We observe in Table 1 that it is better at preserving loudness curves. Below, we provide more hyperparameters for each of our encoders.
A.1.1 MLP ENCODER ARCHITECTURE
Our MLP encoder takes in 80 512 mel-spectrogram, but processes each of the 512 latent time- frames independently, operating only on the channel axis. It can also be thought of as convolutional network, where the convolutions apply to the time axis and have kernel size of 1. consists of series of linear layers with SiLU activations (Hendrycks) 2016), group normalization layers (Wu & He 2018), and residual connections (He al. 2016). The hyperparameters for our MI.P encoder are listed in Table 2
Attribute
Value
Input
80 512 mel-spectrogram 512 latent sequence
Output
Architecture
Frame-wise MLP
Hidden Dim.
512
Num. Hidden Layers
16
Table MLP Encoder Architecture
18

---

A.1.2 ID U-NET ENCODER ARCHITECTURE
Our ID U-Net encoder is ID version of the encoder used Karras et al. (2022). The convolutions occur along the temporal axis. The U-Net consists of several blocks that process information at different resolutions, which are listed below Table In addition, we add self-attentin layers to blocks at particular resolutions, which are also listed below in Table
Attribute
Value
Input
80 X 512 speestecr
Output Architecture
80 X 512 latent sequence
ID U-Net
Kernel Size
Resolutions
[512, 256, 128, 64, 32, 16] [512, 512, 512. 768, 768, 1024] [64, 32, 16]
Channels Per Resolution
Resolutions with Attention
Table 3: ID U-Net Encoder Hyperparameters
A.1.3 DAC ENCODER ARCHITEECTREE
The DAC encoder takes in a raw audio waveform which is resampled to 44.1 kHz. First, it creates a 1024 512 sequence of continuous embeddings using the encoder of Descript Audio Codec (Kumar et al. 2023). This sequence is passed to a ID U-Net identical to the one in Table 3 except for the first convolutional layer, which has 1024 input channels instead of 80, to match the latent dimension of DAC. Table provides details.
Attribute
Value
Architecture
DAC + ID U-Net
DAC Encoder Input
1 2621 44 audio waveform
DAC Encoder Output
1024 512 DAC embedding
ID-UNet Input
1024 512 DAC embedding
ID-UNet Output
80 x 512 latent sequence
Table 4: DAC Encoder Hyperparameters
A.2 DECODERS/DIFFUSION MODEL ARCHITECTURE
Our decoder (diffusion models) are ID-U-Nets that combine convolutional layers with self-attention layers. The decoder is very similar to the ID-UNet encoder described in Appendix [A.122] The main difference is the decoder's input is noisy mel-spectrogram #; well the masked latent Zmusked- These two inputs are concatenated channel-wise before heing fed to the U-Net. The U-Net predicts a linear combination of the added noise and the clean input wo. as described in Karras al. (2022). Again, we follow the architectures in Karras et al. (2022) and provide our code in our github repository. Details are shown in Table
Attribute
Value
Input
80 512 noisy mel-spectrogram
Input 2 Output
80 x 512 frequency-masee latent
80 x 512 clean mpe-sppectogrra
Architecture
ID U-Net
Kernel Size
3 [512, 256, 128, 62, 16] [512, 512, 512, 768, 768, 1024] [64, 32, 16]
Resolutions
Channels Per Resolution
Resolutions with Attention
Table 5: Decoder Hyperparameters
19

---

A.3 TRAINING DETAILS
For the experiments in the main paper, we train for 700k iterations on L40S GPUs with a logical batch size of 1024. We use linear warmup for the first 4,000 training steps, and apply cosine annealing to the learning rate after 350k iterations. In addition, following Karras et al. (2022), we
store an exponential moving average of the model weights, which we use for inference. Hyperpa- rameters for this are shown in Table 6 For the ablation experiments shown in Appendix B.1 we train for 350k iterations, and not perform annealing.
Attribute
Value
Num. Total Iters
700k
Num. Warmup Iters
Training Schedule
Num. Decay Iters
350k
Decay Schedule
Cosine
Optimizer
Adam
Learning Rate
le
Optimizer
81
0.9
32
0.999
Batch Size (Logical)
1024
Batching
Batch Size (Per-GPU)
256
Distribution Strategy
DDP
Precision
Mixed FP32 + BF16
Other
Grad Clip Value
1.0
EMA Decay
0.999
Table 6: Training Hyperparameters
A.4 OTHER HYPERPARAMETERS
Here, we list the values of other hyperparameters mentioned in the Methods section (Sec. 3).
Attribute
Value
L
2
0.5
DFT / Frequency Mask
p
Ic-6
max
80
Diffusion
0.5
B
0.5
Table 7: Other Hyperparameters, Full descriptions can be found in the Methods section (Sec.
A.5 DATASETS
Our experiments in the main paper use two datasets. All clips are resampled sampling rate of 22050 Hz.
MTG-Jamendo. MTG-Jamendo (Bogdanov et al. 2019) a large-scale collection of over 55,000 spanning diverse genres, like classical, electronic, pop, and rock music. The dataset is publicly available, and is popular in tasks like neural audio compression, vocod- ing (Lanzendörfer et al. 2025), and music-tagging (Hasumi al. (025).. We train our mod- els on a dataset of 2.5 million 5.9-second clips from the MTG-Jamendo training split. The MTG-Jamendo dataset is used in the conditional generation (Sec. 4.2p. blending (Sec. 4.3
listening study (Sec. [4.4). and isolation experiments (.5).
2. GTZAN. GTZAN (Tzanetakis & Cook 2002) is a standard benchmark for genre classifi- cation, containing 1,000 30-second audio clips evenly distributed across 10 genres (blues,
20

---

classical, country, disco, hip-hop, jazz, metal, pop, reggae, and rock). We use GTZAN for the interpretability experiment (Sec. 4.6), since we require high-quality genre labels.
We show results on more datasets in Appendix B.2 where we perform the conditional generation and blending experiments on GTZAN and the Maestro dataset (Hawthorne et al. 2018).
A.6 CONDITIONAL GENERATION AND BLENDING EXPERIMENTS
For these experiments, we partition the latent spectrum into 2 bands, bands, and 8 bands. In each of the three partitionings, bands are equal-width on a logarithmic axis. For the conditional generation task, we condition each song on all 14 bands one-at-a-tinn a-t-me averaging results. For the blending task, we take two examples and condition on every possible unordered pair of bands inside the 4-band partition, for total of six possible conditions.
A.7 LISTENING STUDY DETAILS AND ANALYSIS
We used Prolific to recruit high-quality participants for our survey. All respondents self-identified as musicians, and all respondents reside in the United States. The respondents ranged in age from 20-73 years old, with an average age of 43.4 years old, and median age of 41.5 years old.
The survey consists of 12 questions, each comparing two ordered pairs of systems the blend- ing task. Each question presents two reference recordings, and then presents two blendings of the reference clips from two different systems, for total of four clips. The users are asked which recording they prefer both in terms of audio quality, and how well the clips were "blended" together. screenshot a question from our survey is shown in Fig. 6
Example 3
Lisaan the following audio cippc
Inputs
DO01020
$000000
The next two cllps attempt to blend the two inout class topodher. Please dosermine which "blend" you profer in terms zudio quality, and which "blend" you think best margas the may roplay any audio ciips making your Blendlings
Blend
Blend 2
Which beed quilt??
blend batar onagle the suo Inou??
Figure 6: A question from our listening study survey, participant will compare each ordered pair of systems in the study once.
The order of all questions is randomized. In addition, we include one attention check question for each survey participant. In the attention check, all recordings are silent, and the participant instructed to select '2' and '4' for their two Likert scale ratings. The total duration of all the audio recordings in our survey was minutes and 9 seconds. However, the median survey response time was 10 minutes and 25 seconds.
Statistical Significance. We performed a Kruskal- Wallis test, which confirmed that there are statistically significant pairs among the permutations (p = 6.4 x 10-83), We also perform post- hoc analysis using the Wilcoxon signed rank test. We apply Bonferroni correction, which corrects the significant threshold p 0.05/6. According to this test, all pairs of systems have statistically significant differences in audio quality, except for the cross-synthesis and ILVR baselines. This means that LATENTFT outperforms all baselines in terms of audio quality according to study, to a statistically significant extent. Another Wilcoxon signed rank test indicates that all pairs of systems have statistically significant differences in "ability blend", except for LATENTFT and the cross synthesis baseline. Pairwise significance test results are shown in Table
21

---

System
System
p-value, Audio Quality
p- Value, Ability to Blend
LATENTFT
Cross Synthesis
1.59 x 10-3
9.54 x 10-2+
LATENTFT
ILVR
3.83 10-4
8.84 x 10-7
LATENTFT
VampNet
7.02 x 10
1.64 x 10
Cross Synthesis
ILVR
9.51 x 10-2+
6.62 x 10-4
Cross Synthesis
VampNet
1.91 10-6
8.09 x 10-10
ILVR
VampNet
1.55 10
1.69 x 10
Table 8: Results from Kruskal-Wallis H test performed on listening study results. All pairs of systems have statistically significant differences in audio quality, except for ILVR and Cross Syn- thesis. All pairs of systems have statistically significant differences in "Ability to Blend" besides LATENTFT and Cross Synthesis. These pairs are indicated with an asterisk (*).
Inter-rater Agreement. To compute inter-rater agreement between our 29 participants, we cal- culate Fleiss's ' Kappa, which measures the degree of agreement beyond chance for multiple raters. We report к 0.0654 for our question about audio quality, and 0.0914 14 for our question about "ability to blend".. Both values fall in the "slight agreement" range (Landis & Koch 1977), indicat- ing substantial subjective variation in perceptual judgments. This level of agreement possible due to individual preferences and perceptual differences, which naturally lead to varied responses.
A.8 ISOLATION EXPERIMENTS
We accomplish isolation by taking music clip x: and obtaining the full-spectrum latent sequence, and h0, a version of the latent sequence z bandpassed to the selected frequency range. We then guide the diffusion process with both and hh (see Alg. 3 with blend weights a, B, resulting in an output that emphasizes the selected band while suppressing content outside of it. The ratio of B and a determines the amount of boosting that occurs, with B >> a resulting in isolating the selected band almost completely.
A.9 INTERPRETING THE LATENT SPECTRUM.
In the interpretability experiment (Sec. 4.6), we analyze the latent spectrum of individual songs, and associate different frequencies song ss latent spectrum with musical attributes like genre, chords, tempo, and pitch. We select one song time to analyze. An input song is chosen from vali- dation split of GTZAN (Tzanetakis & Cook 2002). We generate hundreds of variations of the input song, while conditioning sweep over the latent frequency axis, conditioning on every 10-bin range of the latent DFT spec- trum. We measure each generation's adherence the input song along several axes, determine how the latent frequency that we condition on affects which attributes are preserved.
different parts of its latent spectrum. We do this by performing linear
First, we classify the genre of the generated variations. We train classifier GTZAN, which is linear probe on VGGish embeddings (Hershey et al. (017). and obtains 81.8% accuracy on the validation set. Then, we apply our classifier to the generated variations, determining if the classifier's ' prediction of the generated variation matches the ground truth genre of the input song. For each frequency bin listed along the x-axis of our plot, we compute the accuracy across every variation whose condition included that bin. We normalize the curve to be between 0 and 1, so that can be plotted alongside the other curves.
training split of
Second, we measure the Tonnetz correlation between the variations and the reference. This provides a proxy to measuring changes in chords, since tonal centroid features are used to identify and com- pute the distance between chords (Milne & Holland 2016). Again, we plot the Tonnetz correlation between input and variation against which frequency bins we condition on. We normalize this curve to be between 0 and for the sake of plotting.
Third, we measure the pitch error using the Essentia (Bogdanov et al. 2013) package. First, we use Essentia's algorithm to predict the predominant pitch (the pitch of the melody) in song. Then, compute the "overall accuracy" metric described in Salamon et al.) (2014). to measure how well the pitches of the generated variation match with the reference. Again, we plot the pitch accuracy against
22

---

the latent frequencies that we condition on. We flip this curve vertically, that 'up' corresponds to higher preservation instead of higher error, and normalize the curve to have minimum and maximum one.
Fourth, we estimate the BPM of the variations and the reference using Librosa (McFee et al. 2015). We compute the absolute tempo error between the reference and variants, again orienting the curve so that 'up' corresponds to higher preservation, and normalizing the curve to be between 0 and 1.
We achieve the plots by applying Gaussian smoothing to all four curves. Note that unlike the blend- ing and conditional generation experiments, we measure characteristics of the entire generation versus the entire reference, instead of bandpassing descriptor signals.
23

---

B ADDITIONAL EXPERIMENTS
B. ABLATIONS
We ablate several components from LATENTFT-MLP to demonstrate the necessity of each compo- nent. In these experiments, we train LATENTFT-MLP and each of its variants for 350k iterations, skipping the annealing phase. Quantitative results for the conditional generation task are shown in T Quantitative results for the blending task are shown in Table TQ We also show example spectrograms for conditional generation Fig. and example spectrograms for blending Fig. TO
Adherence
Quality
Loudness t
Rhythm
Timbre +
Harmony
|
FAD |
LATENTFT-MLP
0.800
0.961
0.397
0.081
0.349
w/o Freq. Masking
0,476
0.907
2.675
0.121
5.341
w/o Correlation
0.694
0.932
1.284
0.109
2.744
w/o Log. Scale
0.512
0.838
1.322
0.097
1.196
w/o Encoder
0.028
0.565
3.569
0.130
0.846
w/ Bandpass Augmentation
0.861
0.953
0.562
0.084
1.511
Table 9: Ablation results the Conditional Generation Task. Mel-Cepstral Distortion (Timbre) divided by 100. Ablating any component of the model generally leads to worse audio quality and adherence.
Adherence
Quality
Loudness t
Rhythm f
Timbre
Harmony +
FAD
LATENTFT-MLP
0.678
0.875
1.030
0.109
1.371
w/o Freq. Masking
0.597
0.902
1.152
0.127
4.789
w/o Correlation
0.635
0.885
1.167
0.115
2.534
w/o Log. Scale
0.535
0.827
1.382
0.111
2.119
w/o Encoder
0.030
0.539
4.026
0.147
0.854
w/ Bandpass Augmentation
0.664
0.885
1.636
0.117
2.586
Table 10: Ablation results on the Blending Task. Mel-Cepstral Distortion (Timbre) is divided by
100. Ablating any component of the model generally leads either significantly worse audio qual- ity, or significantly worse adherence.
Ablating Frequeney Masking During Training. First, ablate frequency masking during train- ing, applying only the inference-tinet time user-specified mask. Previous methods apply frequency- masking post-hoc, analyze pretrained model's latent space (Tamkin al., (020). In Tables y TO) ("w/o Freq. Masking"), we see that removing frequency-masking during training results in a substantial degradation in audio quality. Without masking during training, the decoder does not leam how to reconstruct music from frequency masked latents, and fails generate high-quality maskeed masked latents during inference. This ablation also shows post-hoc hoc masking insufficient for coherent audio synthesis, which requires incorporating masking during training.
audio from frequency
Ablating Correlations between Bins. Next, we ablate correlations between frequency bins. As explained in Sec. 3.4] we use locally correlated scores mask frequency bins. If we mask each bin independently, we will end up with speckled, erratic masks, where unmasked bins and masked bins next each other. This is shown in Fig. 7 Unmasked bins provide strong local cues to nearby masked bins, making the reconstruction/denoisii from locally correlated scores shown in Fig. 8 Our strategy of correlating scores results in large, contiguous regions of unmasked and masked bins, which makes the learning task more difficult, and better reflects inference-time user-specified masks. Tables and I0 (""wo Correlation") verify that using an uncorrelated mask results in substantial degradations to audio quality.
task easier. In contrast, masks generated
24

---

Uncorrelated Masking
Our Masking
0
1
Example
Example
3
4,
5
6
7
20
40
60
80
100
120
0
40 60
80
100
120
Latent Frequency Bin
Latent Frequency Bin
Figure 7: Example masks where there is no correlation between the scores associated with each frequency bin. The masks are speckled and erratic.
Figure 8: Example masks from our mask- ing strategy, where bin scores are locally cor- related after being mapped logarithmic axis. The mask forms contiguous regions.
Ablating Logarithmic Scaling of Latent Frequency Axis. We also ablate the logarithmic scaling of the frequency axis, discussed in Sec. Sec.[3.4 Our intuition for using logaritmic scaling is as follows: Most structured signals have a 1//-sppetrum, meaning that the energy at high frequencies much lower than the energy at low frequencies. Thus, "group" of low-frequenc bins will contain much more energy than a "group" high-frequency bins of equal width. To counterbalance this effect, encourage high-frequency "groups" to be wider, by mapping the frequency bins to a logarithmic scale before computing correlations between bins. This reflects the fact that 1/ fspeccra have equal energy per-octave. Indeed, removing logarithmic scaling reduces both quality and adherence in both the conditional generation and blending tasks, shown in Tables and 10 ("w/o Log. Scale").
to the audio wave- form directly, to show that our model's representations capture things the waveform cannot. Ablat- ing the encoder results in poor adherence, but better audio quality in the of blending (Tables and TO ("wwo Encoder"). Note that allowing for poor adherence improves audio quality, since the generation is less constrained. Since there is very little information in the waveform for frequencies we condition on (0-43 Hz), removing the encoder is almost like running an unconditional model.
Ablating Encoder. We also ablate the encoder, applying frequency-masken
Random Bandpass Augmentation. Lastly, would like to test the necessity of using DFT- based mask as our latent augmentation, instead another frequency- awaare latent-space augmenta- tion. Thus, instead of applying DFT Mask the latent space during training and inference, apply a randomized bandpass filter to the latent space during training, and user-specified one dur- ing inference. We found that this resulted in some training instability, requiring several restarts. We believe the orthogonality of the DFT is helpful for training stability: From a theoretical perspective, DFT-masking in the forwards pass results in applying the same DFT mask the upstream gradient in the backwards pass. In the backwards pass, the DFT mask can thus be interpreted masking out orthogonal components of the upstream gradient, while leaving the unmasked components of the full-band gradient intact.
Example Spectrograms. Fig. 9 shows example spectrograms for the ablations on conditional gen- eration, and Fig. [00 shows example spectrograms for blending. The figures show that many of the baselines fail to generate coherent audio. The ablation without the encoder generates coherent audio, but fails to follow the condition(s). Please refer to the figure captions for more details.
25

---

Reference
LatentFT
w/o Freq. Masking
8192
1096
2048
1024
512
2
Time (s)
Time (s)
Time
w/o Correlation
w/o Log, Scale
w/o Encoder
8192
4096
2048
1024
3
5
2
4
Time (s)
Time (s)
Time
Figure 9: conditional generation example, where we take 0.68-2.70 Hz. from the latent spectrum of the reference (top left). LATENTFT generates variation capturing the rhythmic pattern near 2
Hz. The fraquency-masking, correlation, and log-scaling ablations also have a pattern near 2 Hz, but the audio quality is much worse. The encoder ablation does not follow the conditioning.
Reference
Reference 2
LatentFT
w/o Freq, Masking
8392-
4048-
2048
1024
Time (5)
Time (s)
Time (s)
Time
w/o Correlation
w/o Log, Scale
wio Encoder
B142
2045
512
Tiime (5)
Time (6)
Figure 10: A blending example, where we take 0-0.68 Hz from the first reference, and 10.78-4. Hz from the second reference. LATENTFT generates a variation that contains characteristics from both examples. For instance, the rapid rhythmic patterns of Reference are retained, well the horizontal line from Reference The correlation and log-scaling ablations retain some of these characteristics, while the encoder and frequency masking ablations ignore the references.
26

---

B.2 RESULTS ON MORE DATASETS
To demonstrate generality, we also use LATENTFT to perform conditional generation and blending on two other datasets: GTZAN and Maestro. The GTZAN dataset was previously used for the interpretability experiment in Sec. 4.6] and is described in Appendix A.5 The Maestro dataset (Hawthorne et al. 2018) is large collection of over 200 hours of aligned piano performance audio and MIDI from the International Sfano-eCCoppttiin
Taking our LATENTFT-MLP model trained on the MTG-Jamendo training set, we evaluate the model on 1024 5.9-second clips from both the GTZAN and Maestro datasets. The results for GTZAN are show in Table II and the results for Maestro are shown in Table [2] Although LA- TENTFT performs worse in terms of audio quality compared to our evaluations on MTG-Jamendo, we find that outperforms our baselines on both GTZAN and Maestro. This indicates that LA- TENTFT can work recordings that are only piano, datasets with diverse set of genres.
Conditional Generation
Blending
Adherence
Quality
Adherence to Both Inputs
Quality
Loud.
Rhyth. f
Timb. 1 Harm.
FAD
Loud.
Rhyth. 1
Timb. L
Harm.
FAD 1
Vampnet
5.748
7.173
Guidunce Gradients
0.585
0.825
1,470
0.094
1.368
0.611
0,850
1.643
0.105
1,961
ILR
0.628
0.852
0.730
0.088
1,873
0.672
0.877
0,744
0.097
3,137
DAC
0.723
0.845
4.045
0.191
8,810
0.610
0,794
4.115
0.212
7.162
Spectrogram
0.503
0.876
1.873
0.128
8.734
0.402
0.840
2,972
0.111
8,397
Cross Synthesis
2.884
LATENIFT-MLL
0.840
0,965
0.356
0.073
0,844
0,721
0.885
0.970
0.095
1.987
LATENTFT-UNct
0.855
0.967
0.377
0.073
0.905
0.714
0.891
1.056
0.095
1.926
Table 11: Results Conditional Generation and Blending the GTZAN dataset. Compared to baselines, LATENTFT achieves superior adherence and audio quality, demonstrating the general- ity of LATENTFT when it comes to new datasets with multiple genres. Mel-Cepstral Distortion (Timbre) is divided by 100. The Masked Token Model and Cross Synthesis baselines do not offer frequency- -based controls, do not compute adherence. Cross Synthesis also only applies the blending task.
Conditional Generation
Blending
Adherence
Quality
Adherence Both Inputs
Quality
Loud.
Rhyth. 1 Timb. Harm.
FAD
Loud.
Rhyth.
Timb. L Harm.
FAD
Vampaet
11.914
14.387
Guidunce Gradients
0.530
0.795
1.483
0.116
8,588
0.557
0.824
1.606
0.133
6.221
ILVR
0.580
0.817
0.976
0.118
9.923
0.627
0.857
1.007
0.131
10.018
DAC
0.729
0.835
4.088
0.243
11.745
0.639
0.776
3.720
0.297
11.614
Spectrogram
0.413
0.853
1.981
0.152
14,208
0.330
0.817
2.640
0.157
14.131
Cross Synthesis
3,139
LATENTFT-MLP
0.809
0.967
0.553
0.085
0.667
0.689
0.892
0.886
0.121
2.767
LATENIFT-UNel
0.830
0.968
0.590
0.085
0.865
0.710
0.899
0.943
0.124
2.708
Table 12: Results Conditional Generation and Blending on the Maestro dataset. Even though the Maestro dataset is only piano recordings, LATENTFT demonstrates super audio quality and adherence compared to basclines. Mel-Cepstral Distortion (Timbre) divided by 100. The Masked Token Model and Cross Synthesis basclines do not offer frequency-based controls, so we do not compute adherence. Cross Synthesis also only applies to the blending task.
27

---

B.3 MORE INTERPRETABILIT RESULTS
Our interpretability experiment, introduced in Sec. 4.6 attributes parts of a particular song's la- tent spectrum with musical characteristics like genre, chords, tempo, and pitch. In this section, we present more examples where we analyze individual songs, and plot how well conditioning on various latent frequencies in the song preserve genre, chords, tempo, and pitch. These extra plots are show in Fig. [I] Across several musical styles, we see the trend that genre tends to lie in the frequency range around Hz, indicating that it is global characteristic. Chord changes also oceur at low frequencies, with peak preservation between 0.25-2 Hz. Tempo and pitch occur at higher la- tent frequencies, since prominent rhythmic and melodic patterns are typically more rapid than chord changes. Please refer to Appendix A.9 for implementation details.
Rock Song
Hip-hop Song
Preservation
Chards
Chords
0.2
lempo
0.2
Tempo
Ritch
Pitch
0.
0.000
Latent Frequency (Hz)
Latent Frequency (Hz)
Blues Song
Blues Song
Preserval Preservation
0.2
0.5
Chards
Chords
Tempo
Ritch
Pitch
0.50
0.00
Latent Frequency (Hz)
Latent Frequency (Hz)
Hip-hap Song
Pop Song
1.2
1.0
Preservation
0.6
0.6
0.1
Chards
Chords
Tempo
Tempo
Ritch
05
30
2.0
10
6.0
0.00
05
1.0
3.
43
6.0
Latent Frequency (Hz)
Latent Frequency (Hz)
Jazz Song
Reggae Song
1.0
1.0
Preservation
0.4
Chards
Chords
Tampo
Tempo
20g
03
:0
2.0
40
5. 00
03
1.
25
45
8.0
Latent Frequency (Hz)
Latent Frequency (Hz)
Figure 11: More Sweep Examples. Songs taken from the GTZAN dataset. Generally, genre tends to be a global characteristic, lying around Hz. Chord changes also lie in the low end of the latent spectrum, while tempo and pitch are associated with higher latent frequencies. Please refer to Sec. for our motivations behind this experiment, and Appendix A.9 for implementation details.
28

---

B.4 REMOVING THE LATENT DFT
In this experiment, we remove the Latent DFT entirely from both training and inference. During training, the model tries reconstruct the input from the full latent sequence z. During inference, the full latent sequence z remains unmasked. This is similar to the original Diffusion Autoencoder from Preechakul al. (022). We find that without frequency masking, the decoder reconstructs in the input without generating any interesting variations, as show in Fig. 12 refer the website under "Removing DFT Masking".
For audio examples,
Reference
Generation
8192
4096
2048
1024
512
1
3
5
1
5
Time (s)
Time (s)
Reference
Generation
8192
4096
2048
1024
512
1
3
5
1
5
Time (s)
Time (5)
Reference
Generation
8192
4096
2048
1024
1
3
5
5
Time (s)
Time (s)
Figure 12: Mel-spectss where we remove the DFT during both training and inference. During inference, we condition the diffusion process on the full latent sequence z derived from reference (left). This reconstructs the input without creating variation (right).
29

---

B.5 PER-BAND ERROR
We show in Fig. that conditioning on mid-scale or fine-scale RVQ levels leads to rapid degrada- tion in audio quality. On the left, we generate audio using the Masked Token Model baseline (Garcia et al. 2023). which contains 14 RVQ levels in total. We condition on each of the levels individually, and observe a degradation in quality as we condition on finer and finer tokens. On the right, we show a comparison with our model, conditioning on different latent frequency bands instead of different RVQ layers. As we condition on higher and higher frequencies (smaller timescales), the audio qual- ity does not degrade. The metrics shown are averaged across 1024 songs from the J-GJedddddo test set.
RVQ-Based Generation
LatentFT UNet
FAD (Audio Quality)
RVQ Level
Latent Frequency (Hz)
Conditioning on various RVQ layers in the Vampnet Model (left) and on various latent frequencies in our model (right). Our model maintains generation quality even when conditioning on finer-scale features.
Figure 13:
C ADDITIONAL RELATED WORK
Separating Info by Scale. Our work relates learned multiscale representations that attempt to separate information by scale. Hierarchical VAEs attempt to model the data distribution using a (of- ten multiscale) stack of latent variables. However, Zhao et (2017) show theoretically and experi- mentally that most hierarchical VAEs (Sønderby et al. 2016; Gulrajani et al. 2016 Bachman (0166) have difficulty separating information between levels. They propose an alternative multi nulti-network ar- chitecture under the assumption that deeper networks encode more abstract features, while shallower ones will encode simpler ones. Using these networks, they show they can vary features an image across few (c.g. 4) scales independently. Still, the exact scale that each network corresponds to depends on the data distribution. We extend this by 1) providing a continuous scale axis and 2) providing intuitive, non-heuristic way of specifying scales via Hz.
Generative Audio Equalizer. Similar to our work, Moliner et al. 2024) introduce a diffusion- based generative audio equalizer. While this work generates content selected audible frequencies, we generate content at latent frequencies.
Other Uses of the Fourier Transform in Deep Learning. The Fourier transform has also been used in CNNs to accelerate convolutions (Mathieu et al. 20133] Ding et al. (017). Audio signals are also ubiquitously represented in the frequency domain, are MRI images (Passigan & Ramkumar 2024
AudioMAF. Another work in audio that uses masking strategy during training is AudioMAE (Huang et al. (022). This work builds off of masked autoencoders for images (He et el. 2022). Here, neural network tries to reconstruct been masked. This task allows the network to learn representations that are useful for classification, event detection, and retrieval. While AudioMAE masks random time- frequeeyyncyy bins, LATENIFT masks random bins in the latent spectrum.
audio spectrogram after many time-frequency patches have
30

---

D EXTENDED BACKGROUND
D.I DFT
Here, derive Eq.
X 2k
N
X (22k//N)
æ
A=
We consider the case where is odd. By the periodicity of complex sinusoids:
[N/2]
1
Σ
X
N
LN/21
We can combine the k < 0 terms with the k >> 0 terms in the sum like so:
N//2
1
x
[kkjee(2nk/N X[-kle [(2ak/NN))]
X
N
N
Where we take the k = 0 out of the sum. The DFT of real-valued signal is Hermitian, meaning
that X [ X [A]]
LN/2
x (22k/12) + [k]e ((2kk/N))) +
X
N
has the same magnitude as X [k]. but the opposite phase. We can split X [ into its magnitude X and phase dk. and do the same for X [k]:
As its complex conjugate, X [K] x
X X e
X [k]
X [k:] Joox
Plugging these into formula:
N/2J
X
x [K|kee Ja2k//N
[kklee job
R=
N//21
x
[ ((((k//N)))))) + (=5(((2//N)mm00)) +
x
N
N
Using Euler's Formula:
[N/2]
1
2T
X
x
Σ
[||
cos
kn + 1x))
N
N
N
k=1
This can be expressed as:
[N/2]
2n k n + dx N
x [n]
A cos
Which is the form that we desire. Observe that the constant term outside of the re-enters the sum as a constant cosine (k: = 0). The case where is even is quite similar, but includes another term (the Nyquist term), which is always real cosine.
31

---

E LLM USAGE
We used LLMs to help us improve the writing of our paper, for instance, by finding synonyms for certain words or for finding more concise ways phrase particular ideas. We also used LLMs as a search tool to help us find related work, but relied on our own interpretation of the work after references were provided.
32


Reviews:
[{"review_id": "NamAEkhAln", "reviewer": "Reviewer_jWrm", "summary": "The paper introduces a generative audio framework that enables a novel frequency-domain control by manipulating musical patterns in the latent space according to their timescales. Their method:\n- combines a diffusion autoencoder with a latent-space Fourier transform, trained end-to-end to decompose and control musical structures -- timescale directly in the latent space.\n- Allows user-specified timescale conditioning, analogous to how equalizers shape timbre in audible frequencies. They use their model for generating music variation, blending, and separating patterns by timescale directly in the latent space with musically coherent results.\n- Introduces new evaluation baselines tailored to assess this novel control task.", "questions": "- Line 352: The author state that in the Guidance and ILVR baselines, the diffusion process is guided using Mel spectrograms rather than the latent spectrum. Why was this approach chosen, and wouldn’t applying the DFT directly to the latent space make for a more comparable baseline?\n- Could the authors clarify the influence of framerate T’ on the results? How should it be chosen to effectively capture musical patterns?\n- Minor note: in line 830, there appears to be an extra space that should be corrected.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors present a new method for guiding music generation by adjusting timescales in the latent space, offering an alternative to global, time varying, or token-based conditioning approaches.\n- In the absence of prior studies on this type of control, they design thoughtful and relevant baselines, setting the stage for future developments in the field.\n- The proposed method consistently outperforms existing baselines, demonstrating stronger conditioning behavior and clearer interpretability in blending and timescale separation tasks. Their analysis of the latent spectrum further reveals meaningful correlations with musical traits such as tempo, and pitch.\n- The audio demonstrations show that their method is promising, and the upcoming code release will make the work easily reproducible and enhanced.\n- Altogether, the paper lays a strong foundation for a new direction in generative audio research, opening the door to further refinements and creative extensions.", "weaknesses": "- The notion of timescale conditioning offers an interesting angle, but it provides a somewhat narrow view of musical structure. Many musical genres do not rely on clear, repeating patterns, which may limit the method’s generality beyond pattern-based music like pop or rock.\n- It remains unclear whether the model can control acoustic attributes such as timbre or warmth, which reside in the actual frequency domain rather than in the latent frequency space. For instance, the diffusion model seems to receive no extra guidance, such as text or acoustic features, to generate the missing frequency content, resulting in more random variations and reduced controllability.\n- Although the model can isolate latent frequencies, it struggles to disentangle higher-level musical concepts such as instruments or timbral components. This raises concerns about scalability, for instance, how would it handle evolving patterns or instruments in a full-length piece, given that training and evaluation rely on short (≈5.9 s) audio clips?\n- The audio quality of the generated samples is still limited, with noticeable metallic artifacts that affect realism and listening comfort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fXQ5Xj6ycS", "reviewer": "Reviewer_YTEd", "summary": "This authors introduce LATENTFT, which applies Fourier transforms to latent time series from a diffusion autoencoder and trains with random, correlated log-frequency masking to enable frequency-domain control over music generation. The approach allows users to manipulate latent frequency bands at inference to generate variations preserving specific timescales or to blend references by combining disjoint frequency regions. Experiments on MTG-Jamendo demonstrate improvements over baselines in adherence metrics (loudness, rhythm, timbre, harmony) and quality (FAD), supported by a 29-musician listening study. The method offers interpretable control via \"isolation\" experiments linking latent Hz to musical attributes.", "questions": "(1) Can the authors provide failure case analysis? When does latent frequency control break down (e.g., very low-Hz bands dominating, cross-band dependencies)?\n\n(2) How does the method perform with longer contexts (30+ seconds, ideally ~3 mins) where hierarchical structure becomes important? Does the flat latent time series model capture phrase/section boundaries?\n\n(3) What happens with alternative encoders (e.g., raw-audio encoders like Encodec)? The appendix hints that encoder architecture affects behavior; to support the broader general claim, this deserves fuller investigation.\n\n(4) Could you add a latent-augmentation baseline mentioned above without DFT to clarify what the Fourier structure specifically contributes versus generic frequency-aware training?\n\n(5) What are the computational costs of the increased spectral resolution (zero-padding factor L) and correlated masking during training/inference?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors' decision to operate in latent frequency space rather than audio frequency is well-motivated, and the correlated log-frequency masking scheme aligns naturally with musical structure and audio engineering intuitions. The proposed algorithms are clearly specified, the training procedure is straightforward, and the mask-conditioning interface provides an interpretable control mechanism. Quantitative improvements over multiple baselines and positive user study results support the core claims for the tested domain. Preservation curves and isolation experiments provide insight into what different latent frequency bands capture, and can potentially offer practitioners guidance for control.", "weaknesses": "(1) All experiments use 5.9-second clips from a single dataset (MTG-Jamendo) with mel-spectrogram front-ends. This raises serious questions about generalization to: (a) longer musical forms where phrase/section structure matters, (b) other audio domains (speech, non-Western music, sound design), and (c) different representation choices (raw waveform, neural codec latents). The paper's claims about \"music generation\" are stronger than what this limited scope can support.\n\n(2) The most competitive baselines (ILVR, codec/spectrogram filtering) operate on spectrogram rather than latent frequency, creating an uneven comparison. I suggest the authors to train the same architecture with random band-drop augmentation in latent time but without explicit DFT structure as a baseline/ablation. This would isolate whether gains come from Fourier properties specifically or just from any structured frequency-aware regularization.\n\n(3) While Table 3 shows that correlated/log masking matters for quality, the ablations don't clearly separate: (a) the value of training with any latent-space augmentation, (b) the specific benefit of DFT orthogonality, and (c) the role of the log-frequency parameterization. The large FAD changes suggest the masking strategy is critical, but mechanistic understanding remains limited.\n\n(4) With only 29 participants and no reported effect sizes, confidence intervals, or inter-rater reliability, it's difficult to assess the robustness of the preference results. The study design (direct comparison on 20 pairs) is appropriate but underpowered for such strong claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1iZNyMHfFD", "reviewer": "Reviewer_1GMu", "summary": "The paper proposes controllable music generation using a Diffusion Autoencoder framework. The method involves applying a DFT over time to encoded latents (interpreted as “timescales” or “latent frequency”), random masking of selected bands, inverting back to latents via an inverse DFT, and decoding with a diffusion model. The core idea of “latent frequency” is promising and appears to offer control over musical structure including song sections, harmony, all the way down to short transient content. \nThe paper, however, has weaknesses in multiple fronts. The writing is unclear on core concepts and methods (e.g., what “frequency” means in latent space, what masking actually teaches the network), and lacks explanation on how to interpret fundamental quantities like latent space sampling rate. Several claims lack experimental validation and theoretical justification, and are not evaluated against adequate baselines. Although this idea is worth exploring, in its current state I unfortunately cannot recommend this paper for acceptance.", "questions": "The authors provide an analogy of EQing, however I believe this can be misleading. I encourage the authors to consider other examples that illustrate “latent frequencies” from a more compositional angle i.e, emphasizing structural as opposed to spectral attributes. \nHow well do different latent spectrum regions map to musical structures?\n\nPlease provide experiments (beat/downbeat F-scores vs. latent frequency bands, chroma change rate vs. latent frequency bands, onset density vs. bands, ground truth chord changes) and show cross-track consistency.\n\nWhat is masking teaching the network? Explain the training signal and why random contiguous band masking encourages the encoder/decoder to organize information by temporal rate, and how this yields controllable edits at test time. Can you provide experiments that show the effect of masking vs no-masking? \n\nCan you include more audio demos on other styles of music? It is hard to guage how successful the method is given only one style of music.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The model is open-sourced.\n\n2. The engineering effort is huge.\n\n3. Dual-NTP and SPC all considered music prior into generative modeling, solid contributions.\nThe model performance is competitive, considering the data they can use.", "weaknesses": "Timescales vs. audio frequencies: The method manipulates “frequency” in latent space, and not audible frequency bins in the waveform domain. This distinction (lines 84-89) is crucial but is not emphasized clearly enough in the methods section. Please clearly define latent frequency, latent frame rate (f_r), the mapping (f_k = k f_r/T'), and how “Hz” must be interpreted in latent space. Consider moving the (f_r) explanation from the appendix into Section 3. \n\nIt is unclear as to how well latent frequency bands align with musical structure. Please provide analyses or experiments showing whether latent bands correlate with sectional changes (say chorus/verse), beat changes, chord changes etc. In addition, it would help to show whether this mapping is consistent across diverse tracks from different styles of music. Otherwise the claims are perhaps somewhat vague.\n\nSection 4 does not include a basic experiment that removes latent DFT (e.g., by conditioning simply on raw latents) to show why latent DFT along with random masking is necessary or what fundamental gains it offers. \n\nMissing model architecture. Please provide the encoder/decoder architectures and key hyperparameters in the appendix. In addition, consider reducing the section on DFTs and instead add more explanations on Diffusion Autoencoders, how they are trained, loss functions and the basic motivation for choosing this architecture over say, Latent Diffusion. \n\nAblations section in (Appendix B.1): The text references ablations (without masking/correlation/log-scale/encoder”), but only a table is provided. No qualitative plots or thorough analyses. The authors may have inadvertently forgotten to add plots and results in this sedition. Kindly add figures and more metrics to support the takeaways.\n\nDataset is too narrow: All core claims are validated on short Jamendo-style clips. Add at least one other domain e.g., other genres of music, piano (MAESTRO dataset) to highlight the “latent frequency” concept and demonstrate generality. It would be interesting to see the idea extended to speech signals. \n\nTerminology inconsistency: If authors prefer to use “timescales,” kindly define once and stick to either “timescales (latent Hz)” or just “latent frequencies.” Please do not alternate between terms as it leads to confusion.\n\nMINOR:\n\nIn Fig. 4, please mark the vertical axis as frequency in Hz instead of bin-index. If you are logarithmically scaling the latent frequency axis, please show this clearly as well. For clarity, add a section on how to convert log-spaced spectrum back to the temporal domain via an IDFT. \n\nInput representation ambiguity: Fig. 2 suggests STFT/mel input, while text says waveform input. Please clarify the exact input representation and update the figure accordingly..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6BeMKjYIK1", "reviewer": "Reviewer_eE9f", "summary": "This paper introduces Latent Fourier Transform (LATENTFT), a framework for controllable music generation that has a capability of latent frequency‑domain controls. LATENTFT is based on a diffusion autoencoder with a latent‑space discrete Fourier transform (DFT), and uses randomized frequency masking during training so that latent spectra can be manipulated coherently at inference. \nLATENTFT uniquely supports conditional generation and music blending by conditioning on user‑selected latent frequency filtering.", "questions": "- Would it be possible to benchmark RAVE and/or DDSP with the evaluation setups conducted on this paper? Alternatively, if these models are considered unsuitable for benchmarking, could the authors please elaborate on the reasons?\n\n[Comments]\n- The reviewer noice that there would be a high-level conceptual connection between the training strategy on LATENTFT and AudioMAE [3] (which is done in Mel-spectrogram domain). This would suggest that learned latent space might have strong semantic meanings not only for controllable music generation but also for audio classification and understanding tasks. Further investigation of the latent representation on such downstream tasks could provide interesting insights and would further strengthen the controbution.\n\n[3] Huang, Po-Yao, et al. \"Masked autoencoders that listen.\" NeurIPS 2020.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Unique controllabilty along with timescale and frequency on latent representation.\nLATENTFT offers a continuous, interpretable frequency axis in latent space, enabling users to select which timescales to preserve or blend. This is a crisp and distinctive contribution to controllable music generation and muisc blending.\n\n- The overall design choice is simple yet effective and well-motivated based on classical audio signal processing, e.g., DFT/iDFT with masking strategy on the basis of insights on frequency characteristics of audio.\n- Empirical evidence that latent‑space frequency filtering has interesting characteristic that could pave the way to new control axis on music generation. Spesifically, int the demo page, the authors demonstrate clear difference of dynamics between singal-based and latent-frequencey-based filtering, which makes the model possible to conduct novel music blending.", "weaknesses": "Although other representative autoencoder-based audio syhtesis methods such as RAVE [1] and DDSP [2] do not perform explicit latent frequency filtering, the lack of disucssion and comparison with these methods remains a minor weakness. Clarifying how these approaches relate to and are positioned against LATENTFT in the main body or the appendix would make the paper stronger.\n\n[1] Caillon, Antoine, et al. \"RAVE: A variational autoencoder for fast and high-quality neural audio synthesis.\" arXiv preprint arXiv:2111.05011 (2021).\n\n[2] Engel, Jesse, et al. \"DDSP: Differentiable digital signal processing.\" ICLR 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)