Differentiable Short-Time Fourier Transform:
A Time-Frequency Layer with Learnable Parameters

Maxime Leiber 1 Yosra Marnissi 2 Axel Barrau 3

Abstract

Similarly, the hop or overlap length (window shift between
frames) controls the trade-off between smooth frequency
evolution and computational cost. It also affects frame positioning relative to signal components. Misalignment canlead to energy leakage in the spectrogram, highlighting theimportance of aligning frames with signal components tocapture the signal’s temporal dynamics. These parameters
are crucial and influence the accuracy and interpretabilityof the STFT representation [9], [10].

We present a differentiable version of the shorttime Fourier transform (STFT), enabling gradientbased optimization of its parameters. This approach integrates with neural networks, allowing
joint learning of both STFT and network parameters. Tests on simulated and real data demonstrate
an improved time-frequency representation andenhanced performance on downstream tasks, illustrating the potential of our method as a standard
for setting spectrogram parameters automatically.

Contributions: This paper gathers and extends previous
works [11]–[15] by proposing a gradient-based optimization
approach for tuning STFT parameters, paving the way fora standard approach in neural networks (STFT as a learnable layer). Our gradient-based approach avoid exhaustive
search of traditional discrete approaches. In addition, continuous parameters allow for more precise optimization oftime-frequency resolution and window positioning. Our key
contribution lies in modifying the STFT definition to makewindow length and hop size continuous and the STFT representation differentiable with respect to these parameters.
The key ideas are to decompose the window length L into an
integer numerical window support N and a continuous time
resolution θ, and to use the continuity of the tapering function to differentiate with respect to the temporal position of
the frames.

1. Introduction
Context: The short-time Fourier transform (STFT) is a
widely used technique for analyzing non-stationary signals
in a variety of fields, including audio processing [1] andmedical applications [2]. STFT-based representations suchas spectrograms are popular tools for visualizing and processing signals. They are often used to visualize the frequency content of music [3] or speech [4]. They can also
be combined with other processing methods such as neuralnetworks to perform advanced tasks. They have been extensively utilized in various tasks, including speech recognition[5], [6], music detection [7], and data augmentation [8] etc.
The combination of neural networks and spectrograms has
shown remarkable success in these applications.

Outline: The paper is organized as follows. In Section 2 we
present the STFT, introduce the differentiable STFT withrespect to both the window and hop lengths. In Section 3 we
propose two optimization approaches for the STFT parameters: representation learning and end-to-end learning and
we demonstrate the effectiveness of our approach throughseveral applications.

Motivation: Nevertheless, accurate STFT-based representations require careful parameter selection. The window
length determines the trade-off between temporal and frequency resolution: shorter windows offer better time resolution but suffer in frequency resolution, and vice versa.
The selection of an appropriate window length depends on
the specific characteristics of the signal and the application
requirements. (e.g., transient events favor shorter windows
and slowly varying components favor longer windows).

2. Differentiable short-time Fourier transform
2.1. Short-time Fourier transform
The STFT is a two-step operation that consists of windowing an input signal into segments and computing the discrete
Fourier Transform (DFT) on each of these chunks, resulting
in a two-dimensional complex matrix SωL [i, f ], where i is
an integer expressing the starting frame index and f the associated frequency. The concept involves sliding a window

1

Safran Tech, Châteaufort, France 2 Capital Fund Management,
Paris, France 3 Offroad, Alfortville, France. Correspondence to:
Maxime Leiber <maxime.leiber0@gmail.com>.
Published at the 2 nd Differentiable Almost Everything Workshop
at the 41 st International Conference on Machine Learning, Vienna, Austria. July 2024. Copyright 2024 by the author(s).

1Differentiable Short Time Fourier Transformover the signal s[t] to be analyzed, selecting a specific timeinterval within the signal. At each position of the window, aFourier transform is computed, representing the frequency
content within that particular time span. It is common tomultiply these signal slices by an analysis window of unit
norm, which is a smooth function called tapering function.
Let ωL denote the tapering function, the STFT of s[t] canbe then explicitly written as:support of the analysis window. The next step is to introduce
a tapering function that is differentiable with respect to thewindow lengths and temporal positions of the windows.
Let ω(x, θ) : R×]0, +∞[→ R+ be a non-negative functionthat is continuous and differentiable with respect to both of
its parameters (x, θ) and with compact support such that:∀x ∈/ [0, θ],SωL [i, f ] = F (ωL s [ti : ti + L − 1]) [f ]=ti +L−1Xk=ti

ωL [k − ti ]s[k]e−2jπkfL

ω(x, θ) = 0(2)

An example of such a function is the common Hann function
given by (see Figure 1):



2πx
1
1 − cos
10≤x≤θ
(3)
ω(x, θ) =2
θ

(1)

In (1), each column SωL [i, :] is the DFT F(.)[f ] of a tapered chunk ωL s [ti : ti + L − 1] of length L of the signals, starting from an index ti to an index ti + L − 1. The time
indices ti of signal intervals on which spectra are computed
are usually equally spaced, so we only have to set the firstindex t0 and the spacing H = ti+1 − ti , known as the hoplength, between ti and ti+1 .

ω(x, θ)θ

2.2. Differentiable STFT with respect to window andhop lengths

x

Figure 1. An example of tapering function : the Hann window

Another common example is the Gaussian function centredon θ/2 and with standard deviation θ/3:
9(x − θ/2)23
exp −10≤x≤θ (4)ω(x, θ) = √
2θ2
2πθ2

Defining a differentiable STFT (DSTFT) with respect to its
parameters (i.e., the window length and window temporal
position) involves modifying the definition of the STFT operator to obtain a formula similar to Equation (1), where
L and ti are continuous parameters and SωL [i, f ] is differentiable with respect to L and ti . Note that differentiating
the STFT with respect to the frame temporal position (orwindow temporal position) is equivalent to differentiatingSωL [i, f ] with respect to the hop length, which is the difference between two consecutive frames.

In the latter expression of the Gaussian window, we set thestandard deviation to be equal to θ/3 because we assumed
that the Gaussian function is approximately zero outside of
the interval centered around the mean and with a half-lengthof three times the standard deviation i.e, 3 × θ/3 = θ. Then,θ can be interpreted as the length of the window function.

There are some challenges to overcome in order to makethe STFT differentiable with respect to these parameters. In
fact, L appears in the formula (1) in the lower bound of thesum and as denominator in the exponential. In both cases, L
must be an integer as we can only sum over integers along
the length of the tapering window and the values ranging
from ti to ti + L − 1 must be uniformly distributed on theunit circle in the complex exponential.

Let N > 0 be a positive integer and define the translated
version of ω on the first variable x denoted by:


1
(5)
ωN (x, θ) = ω x + (θ − N + 1), θ
2From Equation (2), we obtain:
1
1∀θ ∈]0, N − 1[,∀x ∈]/ (N − 1 − θ), (N − 1 + θ)[,22
ωN (x, θ) = 0(6)

To overcome these challenges, i.e., window length upperbound of the sum and denominator of the complex exponential, we decompose L into two parameters:

We can then rewrite the STFT in Equation (1) as follows:• An integer window support N (upper bound),• A continuous time resolution θ (learnable parameter).

SωN [i, f ] =The window support will be fixed and serve to define an
upper bound of the tapering window length. We can now
have a sum over integers namely over an integer valued

X

k∈Z

2jπ

ωN (k − ti , θi,f )s[k]e− N kf

(7)

where θi,f and ti are continuous window length and temporal positions variables respectively.
2

s
×

↶

Differentiable Short Time Fourier Transform

(j)

(j+1)

ωN,θif(j+1)

ωN,θif(j)

ωN,θi+1f

ωN,θi+1f

(j)

(j)

θif

θi+1f
(j+1)

(j+1)

θif
(j)

ti

(j+1)

ti

θi+1f

N

(j+1)

ti+1

(j)

Hi+1

(j)

ti+1

N

(j+1)

Hi+1

Figure 2. Differentiable STFT: The window support N of the subsignal on which DFT is computed is fixed, while on other hand, thetemporal resolution θif of the tapering function ωN (k − {ti }, θτ f ), which actually determines time resolution is allowed to vary.
Additionally, the position of the tapering windows can smoothly shift along the time axis, while the window supports start at the integer
part of the temporal position of the tapering windows. The exponent j denotes the iteration in the gradient descent optimizer.

Proposition 2.1. The modified STFT defined in Equation(7) is differentiable with respect to both the window and hop
lengths (or window temporal positions).

cies have their own window length value, and all frames
have their own hop length. It is however possible to sharesame values of parameters around one or both axes inthe time-frequency plane. For instance, one can assume
frequency-only varying window θf as for the S-transform,time-only varying window θi or time-and-frequency varying
θ as for the Gabor transform or in the classical STFT case.
The same remark is valid for the window temporal position.
Note that the classical STFT defined in (1) can be obtainedby setting θi,f = L and Hi = H ∈ N.

Proof. The modified STFT in Equation (7) is a finite sum
since function x → ωN (x, θ) has support contained in] 12 (N − 1 − θ), 21 (N − 1 + θ)[⊂ [0, N − 1]. We canthus differentiate SωN with respect to θi,f and ti as all
terms involved are differentiable, namely tapering function
is differentiable with respect to its first and second parameter, and exponential is differentiable too. Since the hoplengths Hi are defined as the difference between the starting points of two consecutive window temporal positions
Hi = ti − ti−1 , the STFT is also differentiable with respect
to the hop lengths.

In Appendix A we provide all the gradient and backpropagation formulas. In Appendix B we discuss computational
complexity and specify numerical implementation.

3. Applications

The time resolution parameter θi,f has a meaning similar
to that of L: it sets the time length of the signal chunk on
which a local spectrum is computed. The difference withthe classical STFT is that the slice is filled with zeros inorder to always be of size N . As a result, the maximum
frequency resolution (number of frequency bins) of the local
spectra is no longer controlled by θi,f but by the second
parameter N > θ. This frequency resolution is unrealistic
because the signal was padded with zeros, which amounts
to interpolating in the frequency domain. Increasing N
does not bring more information. In fact, N should only beconsidered as an upper bound on the temporal resolution
θi,f that allows differentiation (see Figure 2).

Our proposed DSTFT can be of particular interest in two
types of applications.
3.1. Representation Learning
The objective is to find optimal STFT parameters (windowand hop lengths) for a specific signal, in order to adapt
STFT parameters to each different signal. To define optimality, we establish criteria translating desired representation
properties into objective functions. Common criteria fromliterature include energy concentration i.e. sparsity suchas norm ratios ℓp -over-ℓq [16] and the Renyi entropy [17].
In Appendix C we performed two experiments for optimal
representation. The first experiment optimized a time- andIn our differentiable STFT definition, all frames and frequen3

Differentiable Short Time Fourier Transform

frequency-varying window length. The second experiment
optimized a time-varying window and hop lengths.

Table 1. Training, validation and testing losses of CNN per fixedwindow length.

window length
10
20
30
40
50

3.2. End-to-End Learning
A promising advantage of DSTFT is the capability to optimize window and hop lengths within neural networks.
Unlike traditional methods with fixed parameters, our approach treats the DSTFT as a learnable layer with trainableweights representing these parameters. This allows for joint
optimization of all parameters alongside the network fortasks like classification or regression. Notably, this is applicable beyond neural networks to any learning algorithm
using STFT-based representations. It is worth to note thatthe optimal representation for learning tasks might differ
from what’s ideal for visualization. In the following, we
will investigate the joint optimisation of the window lengthwith the weights of a convolutional neural network for a
spoken digit classification task.
Joint optimization with a neural network: In this example, we show how DSTFT can be easily integrated into
existing neural networks. We train a convolutional neuralnetwork to classify spoken digits from the Free Spoken
Digit Dataset (FSDD), also called the audio MNIST. The
goal is to find the best window size for the spectrogram andthe best network weights together in a joint optimisation
by minimizing the cross-entropy with the ground truth. To
evaluate the impact of window size on network performance,
we trained a 2-layer convolutional neural network (CNN)with 16 filters per layer. Depthwise separable convolutions
were employed to achieve a balance between efficiency andaccuracy. The network utilized ReLU (REctified Linear
Unit) activation functions for hidden layers, dropout for regularization to prevent overfitting, and the Adam optimizer
for efficient training. We see in Table 1, that the window
size parameter is very important because the accuracy ofthe same trained network for different values of the window size varies strongly. Also, whatever the initial value oftime resolution is, during a joint optimization with network
weights, the time resolution window length parameter converges to an optimal value as shown in the first row of Figure
3. Indeed, losses decrease by jointly optimizing the weightsof the neural network and the time resolution continuous
parameter θ that converged to an optimal value: at the endof the gradient descent, the window length reached the value
34.9 whereas we started the training with a window size of200. The second row of Figure 3 shows the spectrogram ofa sample at the end of the convergence. We note that theobtained spectrogram is not the best for visualization but
the best representation for the neural network.

train loss
0.36720.03040.00890.00640.0061

val loss
1.14420.44430.24080.25500.4859test loss
1.03910.31770.0306
0.12490.2868Figure 3. Training loss, validation loss and window length perepoch (left) and Spectrogram of a sample obtained at the end of
the training (right)

ology applicable to any existing signal processing algorithmor neural network involving spectrograms: replace the spectrogram computation step by the computation of our differentiable spectrogram, then optimize the window length bygradient descent based on the backpropagation formulas wehave introduced.

4. Conclusion
We have defined a differentiable version of the STFT, which
is differentiable with respect to its parameters, enabling
to optimize by gradient descent the window lengths andthe window temporal positions. This capability has thepotential to establish this approach as a standard method
for parameter selection in STFT and STFT-based representations. This contribution aligns with the ongoing efforts
within the machine learning community to provide differentiable relaxations of various traditional tools for efficient
optimization with gradient-based algorithms.

This simple simulation proves the effectiveness of our backpropagation procedure based on a differentiable version of
STFT. It illustrates a general window length tuning method4

Differentiable Short Time Fourier Transform

References
[1]

[2]

[3]
[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

K. M. Stafford, C. G. Fox, and D. S. Clark, “Long-range
acoustic detection and localization of blue whale calls in
the northeast pacific ocean,” The Journal of the AcousticalSociety of America, vol. 104, no. 6, pp. 3616–3625, 1998.
J. Huang, B. Chen, B. Yao, and W. He, “Ecg arrhythmia
classification using stft-based spectrogram and convolutional neural network,” IEEE access, vol. 7, pp. 92 871–
92 880, 2019.
A. Klapuri and M. Davy, “Signal processing methods formusic transcription,” 2007.
B. E. Kingsbury, N. Morgan, and S. Greenberg, “Robust
speech recognition using the modulation spectrogram,”Speech communication, vol. 25, no. 1-3, pp. 117–132, 1998.
Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” arXiv preprint arXiv:2104.01778,
2021.
A. M. Badshah, J. Ahmad, N. Rahim, and S. W. Baik,“Speech emotion recognition from spectrograms with deep
convolutional neural network,” in 2017 international conference on platform technology and service (PlatCon),
IEEE, 2017, pp. 1–5.
J. Schlüter and S. Böck, “Improved musical onset detection with convolutional neural networks,” in 2014 ieeeinternational conference on acoustics, speech and signal
processing (icassp), IEEE, 2014, pp. 6979–6983.
D. S. Park, W. Chan, Y. Zhang, et al., “Specaugment: A
simple data augmentation method for automatic speech
recognition,” arXiv preprint arXiv:1904.08779, 2019.
L. R. Rabiner, R. W. Schafer, et al., “Introduction to digitalspeech processing,” Foundations and Trends® in Signal
Processing, vol. 1, no. 1–2, pp. 1–194, 2007.
C. Garcı́a Ruı́z, J. M. Martı́n Doñas, Á. M. Gómez Garcı́a,
et al., “The role of window length and shift in complexdomain dnn-based speech enhancement,” 2022.
M. Leiber, A. Barrau, Y. Marnissi, and D. Abboud, “Adifferentiable short-time fourier transform with respect tothe window length,” in European Signal Processing Conference (EUSIPCO), 2022, pp. 1392–1396.
M. Leiber, Y. Marnissi, A. Barrau, and M. El Badaoui, “Differentiable adaptive short-time fourier transform with respect to the window length,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
2023.
M. Leiber, Y. Marnissi, A. Barrau, and M. El Badaoui,“Differentiable short-time fourier transform with respect tothe hop length,” in IEEE Workshop on Statistical Signal
Processing (SSP), 2023.
M. Leiber, A. Barrau, Y. Marnissi, D. Abboud, and M.
El Badaoui, “Optimisation de la longueur de fenêtre duspectrogramme au sein d’un réseau de neurones,” 2022.
A. Zhao, K. Subramani, and P. Smaragdis, “Optimizing
short-time fourier transform parameters via gradient descent,” in IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), 2021, pp. 736–
740.
S. Pei and S. Huang, “STFT with adaptive window width
based on the chirp rate,” IEEE Transactions on Signal
Processing, vol. 60, no. 8, pp. 4065–4080, 2012.

[18]

[19]

[20]

[21]
[22]

5

S. Meignen, M. Colominas, and D. Pham, “On the useof rényi entropy for optimal window size computation in
the short-time fourier transform,” in IEEE International
Conference on Acoustics, Speech and Signal Processing(ICASSP), 2020, pp. 5830–5834.
P. J. Werbos, “Backpropagation through time: What it does
and how to do it,” Proceedings of the IEEE, vol. 78, no. 10,
pp. 1550–1560, 1990.
A. Griewank and A. Walther, Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM,
2008.
A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M.
Siskind, “Automatic differentiation in machine learning: A
survey,” Journal of Marchine Learning Research, vol. 18,
pp. 1–43, 2018.
E. Stevens, L. Antiga, and T. Viehmann, Deep learningwith PyTorch. Manning Publications, 2020.
G. Gilboa and S. Osher, “Nonlocal operators with applications to image processing,” Multiscale Modeling & Simulation, vol. 7, no. 3, pp. 1005–1028, 2009.

Differentiable Short Time Fourier TransformA. Gradients and backpropagation formulas
We introduced the DSTFT that admit continuous parameters namely the window length and the frame position (orequivalently the hop length). We showed that the DSTFT is differentiable with respect to its parameters. This means that∂SωN [i,f ]∂S N [i,f ]and ω∂texist and are finite. In this section, we start by giving the analytical expression of these derivatives∂θi,fi
and show how these formulas can be integrated in more general applications. Finally, we specify numerical implementation.
A.1. Analytical formulas for DSTFT gradients
In the following, we will denote by ∂x ωN and ∂θ ωN the derivatives of ωN in Equation 5 with respect to the first and second
variables, respectively. Let us compute the differential of our proposed DSTFT with respect to the window length θi,f , frame
temporal position ti , and hop length Hi . SωN [i, f ] being complex, we apply the term-by-term differentiation by considering
complex numbers as vectors with two real components, in particular exp (jx) ≡ [cos(x), sin(x)].
A.1.1. G RADIENT WITH RESPECT TO WINDOW LENGTH
Time-and-frequency varying window length First, we consider the case of a time-and-frequency varying window length
θi,f . By compute the partial derivative of each value in the time-frequency plane with respect to the window length θi,f , weobtain the Jacobian of size (2, 1) as follows :2jπkf∂SωN [i, f ] X ∂ωN (k − ti , θi,f )s [k] e− N
=
∂θi,f
∂θi,f

(8)

k∈Z

where we recognize the STFT of s where the tapering function ωN (k, θi,f ) is replaced by the functionω̄N (k, θi,f ) = ∂θ ωN (k, θi,f )(9)∂SωN [η, f ]= Sω̄N [i, f ]1i (η)1f (f )
∂θi,f

(10)

Then, we can write:

where 1a (.) is the Dirac function in a.
Shared window length
writesIn the case of shared window length, the derivative of the STFT with respect to window length θ∂SωN [η, f ]= Sω̄N [η, f ]
∂θ

(11)

A.1.2. G RADIENT WITH RESPECT TO WINDOW POSITION AND HOP LENGTH
The derivative of the STFT with respect to window position writes
2jπkf∂SωN [i, f ] X=
ω̃N (k − ti , θi,f )s [k] e− N
∂ti

(12)

ω̃N (k, θi,f ) = ∂x ωN (k, θi,f )(13)∂SωN [η, f ]= Sω̃N [i, f ]1i (η).
∂ti

(14)

k∈Z

where
This results in:

Time-varying hop length Using the formula of a time-varying hop length ti =derivatives for the hop length are given by:∂S[η, f ]= Sω̃N [η, f ]1[0,η] (τ )∂Hi
where 1[0,η] is the indicator function defined on the interval [0, η].
6Pi

t=0 Ht , we can deduce that the(15)

Differentiable Short Time Fourier Transform

Shared hop length Using the formula of a shared hop length ti = (i + 1)H, we can deduce that the derivatives for the
hop length are given by:∂S[η, f ](16)
= (i + 1)Sω̃N [η, f ]∂H
Remark A.1. One interesting feature of our DSTFT is that all derivative expressions share the same form as the forward
pass. In other words, they can always be represented as a DSTFT transform step, by using modified tapering functions.
A.2. Backpropagation formulas
The above derivative expressions enable us to optimize any almost everywhere smooth differentiable scalar loss function
with respect to DSTFT parameters. This is fulfilled by employing gradient descent and the backpropagation algorithm [18]and automatic differentiation (AD) [19], [20] two fundamental tools in machine learning.
Let L be the loss to be optimized. With the STFT and automatic differentiation tools, such as PyTorch [21], which allow
for the automatic calculation of gradients, we can optimize a loss the DSTFT parameters using gradient descent. In thissection, we provide the analytical expressions for backpropagation through the STFT and then we compare them to the ones
computed numerically using PyTorch.
A.2.1. A NALYTICAL EXPRESSIONS
Let L be the loss function to be differentiated with respect to our DSTFT parameters.
Variable parameters
obtain:In the general case of time-and-frequency varying window length and time-varying hop-length, we∂L∂L∂L∂SωN [i, f ]==Sω̄ [i, f ]∂θi,f∂SωN [i, f ] ∂θi,f∂SωN [i, f ] N

(17)

N −1

X
∂L
∂SωN [i, f ]∂L=∂ti∂SωN [i, f ]
∂ti
f =0

=

N
−1
X
f =0

∂L
Sω̃ [i, f ]∂SωN [i, f ] N

(18)

−1
T
−1 N
X
X
∂L
∂SωN [η, f ]
∂L
=
∂Hi
∂S
[η,
f
]
∂Hi
ω
N
η=0
f =0

=

−1
τ N
X
X

η=0 f =0∂LSω̃ [η, f ]∂SωN [η, f ] N(19)

where ∂Sω∂L(i,f ) is the vector of derivatives with respect to real and imaginary parts of size (1, 2) and S is understood as aN
vector of size (2,1).
Shared parameters

In the classical STFT case of window and hop length sharing, we obtain:


−1
T
−1 N
X
X
∂L
∂L
∂SωN [i, f ]∂L, Sω̄ ,==
∂θ∂SωN [i, f ]
∂θ
∂SωN
i=0

(20)



T
−1 N
−1
X
X
∂L
∂L
∂SωN [i, f ]
∂L
, Sω̃
=
=
∂H
∂SωN [i, f ]
∂H
∂SωN
i=0

(21)

f =0

f =0

P
where ⟨·, ·⟩ denotes the Frobenius scalar product of two matrices : ⟨A, B⟩ = i,j Re(Aj,i )Re(Bi,j ) + Im(Aj,i )Im(Bi,j )with Re(X) and Im(X) the real part and imaginary part of X, respectively.
Remark A.2. These analytical expressions are highly flexible and enable differentiation of any scalar function L of the
STFT’s outputs with respect to its tuning parameters. In fact, all backpropagation computation can be performed efficiently
7

Differentiable Short Time Fourier Transformusing matrix multiplications, where the matrices involved have the same dimension as the forward propagation, sinceindividual derivatives are just DSTFT operations. In the following section, we will show that this approach reducescomputational complexity over automatic differentiation tools, is conducive to faster gradient calculations, and use exact
gradient values.

B. Computational complexity and numerical implementation
B.1. Computational complexity
Our proposed approach (SωN ) shares a computational space and time complexity of O(N 2 ) with the classical STFT using
the DFT. However, by leveraging the FFT computation, our approach can achieve a reduced complexity of O(N log N ).
This improved complexity is attainable when using only a shared window length θi,f = θ or a time-varying window
length θi,f = θi . This efficient computational complexity makes the time-varying window length approach particularly
advantageous for real-time applications where fast processing is essential. On the other hand, for applications that do nothave memory or time constraints but require higher precision with respect to the temporal positions of windows or thelengths of windows in frequency, the complexity is O(N 2 ).
Regarding the gradient computation in Appendix A, the backward pass has a computational complexity equivalent to theforward pass, i.e., O(N log N) when using the FFT, otherwise, the complexity is O(N 2 ).
B.2. Numerical implementation
To implement the DSTFT transformation, we need to modify the infinite sum to a finite one because the tapering function
has limited support. The tapering function is nonzero only within the support interval [0, N − 1], so we perform the sum only
over this interval. To achieve this, we make a variable transformation, so that k is an integer within the interval [0, N − 1].
More specifically, we execute the following transformation: k → k + ⌊ti ⌋, taking advantage of the fact that ti = ⌊ti ⌋ + {ti }.
Here, {ti } represents the fractional part of ti , and ⌊ti ⌋ is the integer part of ti . Consequently, we obtain the following
expression:SωN [i, f ] =
=

X

k∈Z

ωN (k − ti , θi,f )s[k]e−

N
−1
X
k=0

= e−

2jπkf
N

2jπ

ωN (k − {ti }, θi,f )s [⌊ti ⌋ + k] e− N (k+⌊ti ⌋)f2jπ⌊ti ⌋f
N

N
−1
X
k=0

ωN (k − {ti }, θi,f )s [⌊ti ⌋ + k] e−

(22)

2jπkf
N

Therefore, to overcome the constraint of starting on integer values, we start the window at ⌊ti ⌋ and compensate for the effectof the integer part by the combination of a small shift, {ti } = ti − ⌊ti ⌋, in the argument of ωN and a factor e−2jπ⌊ti ⌋f /N in
the exponential. Note that the above equation, we set the lower bound of the sum to be 0 and the upper bound to be N − 1
because 0 ≤ {ti } < 1, and the tapering function ωN is zero at the boundaries of its support.

C. Additional experiments
C.1. Representation-driven optimization
In this section, we will demonstrate, through simulated and real signals, the immediate interest and significance of DSTFT,
highlighting its potential for finding a good representation. The optimal settings for STFT, such as the choice of the best
window and hop lengths, should be adapted to the specific input signal and its unique features. The DSTFT allows thewindow lengths to flexibly adapt locally to the signal’s frequency content. This improves the ability to capture both timeand frequency information accurately, enhancing the overall time-frequency resolution of the transformation. Second, it
fine-tunes the hop lengths, which determine the temporal positions of the analysis windows. By aligning these positions
with the frequency components of the input signal, energy leakage can be then minimized and short-time signals can be well
localized in the spectrogram. That is why, the DSTFT can be seen as an adaptive time-frequency transform, automaticallyadjusting its parameters to track the unique characteristics of the input signal. To establish an optimal representation.
8

Differentiable Short Time Fourier Transform

Figure 5. Spectrograms (left) and Zoomed-in spectrograms (right)
with respectively from top to bottom small window of length 100,long window of length 1000, single-window DSTFT with a timeand-frequency-varying window length without (middle) and with(bottom) regularisation. Spectrograms (left) and distributions of
window lengths (right).

Figure 4. Spectrograms (left) and Zoomed-in spectrograms (right)
with respectively from top to bottom small window of length 100,long window of length 1000, single-window DSTFT with a timeand-frequency-varying window length without (middle) and with(bottom) regularisation. Spectrograms (left) and distributions of
window lengths (right).

9Differentiable Short Time Fourier Transform

C.1.1. T IME AND FREQUENCY VARYING WINDOW LENGTH
We consider a simulated signal composed of 3 components: a main non-stationary component, a close and stationary
component, and a transient signal. In this example, we focus on optimizing the window length. A small window length
gives fine temporal resolution to localize the transient event in time but a coarse frequency resolution to distinguish nearby
frequencies as displayed in the first row of Figure 4. In contrast, a long window length (in second row of Figure 4) enables todistinguish the main component from the stationary frequency but a poor temporal resolution of thePtransient event. We now
optimize DSTFT with a single window length by minimizing the entropy loss H(SωN ; θ) = − i,f pif log(pif ) where∥S[i,f ]∥

N
pif = P ω∥S. At the end of the optimization, we obtain a spectrogram, displayed in the third row of Figure 4 that isωN [k,l]∥k,loptimal given the criterion. Our optimization is done in a number of iterations much lower than the grid-search case. In thisspecific case, for grid-search, we have tested all integer values between 100 and 1000, which is a fixed computational cost
proportional to 900 forward steps. In the case of DSTFT, the number of iterations depends on the choice of hyperparameters,
including the gradient descent optimizer hyperparameters (e.g. learning rate, stopping condition). It is therefore possible
to perform the optimization between 10 and 100 iterations. Knowing that one iteration consists of a forward pass anda backward pass, and that a backward pass is exactly equivalent to a forward pass in terms of computational cost as we
saw in the previous section, we obtain a variable computational cost between 20 and 200 iterations. This is much lower
than traditional grid-search. We will see that the gap is even greater in the case of a window length that varies in time andfrequency. In this case of single window length, DSTFT gives an interesting compromise between time and frequency
resolution. But since we use only one window length for the whole spectrogram, we are not able to localize precisely in timetransient event, and in frequency close frequencies. As we can see in this specific simulated example, it is difficult to find an
optimal unique window length for all different components in the spectrogram. We need for different window lengths, more
precisely a time-and-frequency-varying window length, to adapt to the time-varying spectral structure of the signal.

We now optimize DSTFT using time-and-frequency-varying window length by minimizing the following criteria:
L(Θ) = H(SωN ; Θ) + λR(Θ)(23)

where R(Θ) a regularization term that constrains nearby windows in the time-frequency plane to have close values, whichsmooths the distribution of window lengths and improves robustness to noiseand λ is an hyperparameter controlling
P qP
2
trade-off between these two terms. Particularly, we consider R(Θ) = i,f(j,k)∈Vi,f γ(j,k);(i,f ) (θi,f − θj,k ) where

Vi,f is a neighboring index set for the bin (i, f ) and γj,k is a weight given to the neighbor θj,k to measure the similarity
between the frequency content of the regions (i, f ) and (j, k). These weights can be equal or set from a previously computed
spectrogram with a constant window length. The above regularization is related to the non-local total variation penalization
used in image processing [22]. Results with DSTFT using time-and-frequency-varying window length are shown in fourthand fifth row of Figure 4 for spectrograms and window length distribution respectively. We see that we can localize precisely
all the components of our signal both in time and in frequency, thanks to time-and-frequency-varying window length. In fact,
the latter adapts to each component as expected e.g. small windows for transient events and long windows for stationary
components.
C.1.2. T IME - VARYING WINDOW AND HOP LENGTHSIn this section, we will show on a simulated signal that differentiable STFT with respect to both window length and hop
length (or frame temporal position) can be of immediate interest and deserves more attention. We consider a signal thatsimulates shocks of different frequencies and durations. We also added noise to the signal as shown in the first row of
Figure 5. Classical STFT uses frames that are uniformly spread along the signal which may not be the optimal positioning
to localize frequency changes as frequencies have a variable length. In fact, using small, medium, or long window size,
where the frames are uniformly spaced along the signal, produce a spectrogram with a lot of energy leakage ad shownin the the second, third and fourth rows of Figure 5 respectively. We then consider optimizing the window positions andlengths by gradient descent using our DSTFT. More specifically, our goal is to find windows that are well-distributed over
the overall signal, ensuring that there are no gaps and loss of information, and that allow for minimal energy leakage whileachieving a better concentration of energy in the time-frequency plane. To promote energy concentration in each frame, we
PT −1 Ef [∥SωN [i,f ]∥4 ]1
maximize the kurtosis of frame spectrum which is defined as K(SωN ; Ω) = PT −1
i=0 λi Ef [∥SωN [i,f ]∥2 ]2 where λi is
i=0 λia weight used to minimize the contribution of windows that share the same segment of the signal. More specifically, we set
i−1
λ0 = t1 − t0 , λT −1 = tT −1 − tT −2 and λi = ti+1 −t
for every i ∈ {1, . . . , T − 2}. To avoid information loss, we also210

Differentiable Short Time Fourier Transform



PT −1
1
min
θ
,
H̃
want to maximize the spectrogram coverage defined as C(Ω) = Mii+1 with H̃T = +∞ and M is thei=0signal length. Let us recall that in cases where the window’s support extends beyond the signal, we zero-pad the signal forthe calculation of the STFT. However, it is important to note that the coverage metric only takes into account the part of thewindow that overlaps with the signal. While we observe that the spectrogram obtained at the end of the optimization has abetter concentration of energy due to reduced spectral leakage resulting from the proper positioning of frames, which are nolonger straddling two different frequencies, as shown in the fifth row of Figure 5. However, caution must be exercised whenreading the time-axis of the resulting time-frequency representation as the distribution of frames is no longer uniform.

11