Residual Hybrid Filterbanks

Vincent Lostanlen, Xiran Zhang, Daniel Haider, Mathieu Lagrange, Martin

Ehler, Peter Balazs

► To cite this version:

Vincent Lostanlen, Xiran Zhang, Daniel Haider, Mathieu Lagrange, Martin Ehler, et al. Residual Hybrid Filterbanks. IEEE Statistical Signal Processing Workshop (SSP), Jun 2025, Edinburgh, United Kingdom. hal-05032079

HAL Id: hal-05032079

https://hal.science/hal-05032079v1

Submitted on 13 Apr 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Residual Hybrid Filterbanks

Vincent Lostanlen<fnref n="1" />, Xiran Zhang<fnref n="1" />, Daniel Haider<fnref n="2" />, Mathieu Lagrange<fnref n="1" />, Martin Ehler<fnref n="3" />, Peter Balazs<fnref n="2" />

**Abstract**—A hybrid filterbanks is a convolutional neural network (convnet) whose learnable filters operate over the subbands of a non-learnable filterbank, which is designed from domain knowledge. While hybrid filterbanks have found successful applications in speech enhancement, our paper shows that they remain susceptible to large deviations of the energy response due to randomness of convnet weights at initialization. Against this issue, we propose a variant of hybrid filterbanks, by inspiration from residual neural networks (ResNets). The key idea is to introduce a shortcut connection at the output of each non-learnable filter, bypassing the convnet. We prove that the shortcut connection in a residual hybrid filterbank lowers the relative standard deviation of the energy response while the pairwise cosine distances between non-learnable filters contributes to preventing duplicate features.

**Index Terms**—audio and speech processing, convolutional neural networks, filterbank analysis, hybrid deep learning, random matrix theory.

## I. INTRODUCTION

Hybrid deep learning [1], also known as model-based deep learning [2], has a key role to play in speech and music technology. By integrating domain knowledge within deep neural networks, hybrid systems can enhance explainability, controllability and resource efficiency [3]. In this context, hybrid auditory filterbanks have recently been proposed for speech enhancement [4]. Formally:

**Definition 1.1.** A hybrid filterbank composes non-learnable filters $\psi_1...\psi_J \in \mathbb{C}^N$ with learnable filters $w_1...w_J \in \mathbb{R}^T$, $T \le N$. Its response to $x \in \mathbb{R}^N$ is a double convolution: $(w_j * \psi_j * x)$.

While the $\psi_j$'s are designed by expert knowledge and kept fixed, the $w_j$'s are typically initialized at random and then iteratively updated by gradient descent. Yet, convnets for raw audio are known to be susceptible to numerical instabilities, particularly for $T > 2^J$ and for $x$ having strong correlations at lags up to $T$ [5].

In this article, we present theoretical results for a specific setting of hybrid filterbanks, where we reduce numerical instabilities while retaining their capability for gradient-based optimization. Our problem is that the non-learned filter $\psi_j$ may introduce long-range correlations into $(\psi_j * x)$, causing random fluctuations of the output energy $\|w_j * \psi_j * x\|_2^2$. Of course, these random fluctuations could be canceled by initializing the $w_j$'s with zeros; but such a design choice would cause the hybrid layer to predict a constant, which is detrimental to gradient-based optimization [6, Chapter 8.4]. Formally:

**Definition 1.2.** A residual hybrid filterbank (RHF) is a hybrid filterbank in which the filters $w_1...w_J$ are initialized as i.i.d. random Gaussian filters: $w_j \sim N(\mu\delta, \sigma^2\mathbf{I})$ of length $T$ where $\delta$ is the Kronecker symbol, i.e., $\delta[t] = 1$ if $t = 0$ and zero otherwise.

The work of V. Lostanlen and X. Zhang is supported by ANR project MuReNN (ANR-23-CE23-0007-01). D. Haider is recipient of a DOC Fellowship of the Austrian Academy of Sciences at the Acoustics Research Institute (A 26355). The work of P. Balazs is supported by the FWF projects LoFT (P 34624), NoMASP (P 34922) and Voice Prints (P 36446). To reproduce our numerical simulations, please visit: https://github.com/lostanlen/lostanlen2025ssp

<figure><img src="image_2.png" /><figcaption>Fig. 1. Random samples of residual hybrid filters ($w * \psi$) where $w \sim N(\mu\delta, \sigma^2\mathbf{I})$: see Definition 1.2. Columns correspond to different designs of $\psi$ while rows correspond to different values of the residual connection parameter $\mu$. Each x-axis tick denotes 512 time samples. Note the change in y-axis scaling across rows. $N = 2048$, $T = 1024$.</figcaption></figure>

The term “residual” is chosen in reference to residual networks (ResNets) [7]. Indeed, the nonzero expected value of $w_j$ may be interpreted as an identity mapping which is weighted by the parameter $\mu$: $(\mu\delta + w) * \psi = (\mu\psi + w * \psi)$. However, an important difference is that ResNet blocks typically contain one or several nonlinearities before the identity mapping, whereas the response of an RHF is linear with respect to the input signal $x$.

Figure 1 illustrates the effect of the residual connection onto random samples of a hybrid filter ($w * \psi$), for $\psi$ being a Dirac impulse (left); a low-pass filter (center); and a band-pass filter (right). We observe that larger values of $\mu$ improve the temporal localization of ($w * \psi$) and reduce the relative effect of random initialization.

Section II gives exact formulae for the expected value and variance of $\|w_j * \psi_j\|_2^2$, and derives bounds for its relative standard deviation. Section III gives exact formulae for the expected area of the parallelogram with sides ($w_j * \psi_j$) and ($w_j' * \psi_j'$) so as to measure feature diversity. Section IV supports our main findings with a numerical simulation. We conclude the article in Section V and defer the proofs to an appendix (Section VI).

<footnote n="1">¹Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France</footnote>

<footnote n="2">²Acoustics Research Institute, Austrian Academy of Sciences, A-1010 Vienna, Austria</footnote>

<footnote n="3">³Faculty of Mathematics, University of Vienna, Vienna, Austria</footnote>
## II. MOMENTS OF THE ENERGY RESPONSE

**Definition II.1.** The circular cross-correlation between $y$ and $y'$ is

$$ \mathbf{R}_{(y,y')}[\tau] = \sum_{n=0}^{N-1} \overline{y[n]} y'[n+\tau] = \sum_{n=0}^{N-1} \overline{y[n-\tau]} y'[n], \quad (1) $$

where $y, y' \in \mathbb{C}^N$, the variable $\tau$ is an integer lag, the overline denotes complex conjugation, and indexing is understood modulo $N$. The circular autocorrelation of $y$ is $\mathbf{R}_y = \mathbf{R}_{(y,y)}$.

**Proposition II.2.** Given $y \in \mathbb{C}^N$ and $w \sim \mathcal{N}(\mu\delta, \sigma^2\mathbf{I})$ of length $T$:

$$ \mathbb{E}[\|w * y\|_2^2] = (\mu^2 + \sigma^2 T) \|y\|^2. \quad (2) $$

**Proposition II.3.** Given $y \in \mathbb{C}^N$ and $w \sim \mathcal{N}(\mu\delta, \sigma^2\mathbf{I})$ of length $T$:

$$ \mathbb{V}[\|w * y\|_2^2] = 4\mu^2\sigma^2 \sum_{\tau=0}^{T-1} |\mathbf{R}_y[\tau]|^2 \\ + 2\sigma^4 \sum_{\tau=-T}^{T} (T - |\tau|) |\mathbf{R}_y[\tau]|^2. \quad (3) $$

*Note.* The proposition above generalizes [5, Proposition II. 1] to complex values of $y$ and to nonzero values of $\mu$.

**Theorem II.4.** Given a residual hybrid filterbank with $w_1...w_J \sim \mathcal{N}(\mu\delta, \sigma^2\mathbf{I})$ i.i.d. and nonzero $\psi_1...\psi_J$, for all $j$:

$$ \frac{2}{T} \left( 1 - \frac{\mu^4}{(\mu^2 + \sigma^2 T)^2} \right) \le \frac{\mathbb{V}[\|w_j * \psi_j\|_2^2]}{\mathbb{E}[\|w_j * \psi_j\|_2^2]^2} \le 2 \left( 1 - \frac{\mu^4}{(\mu^2 + \sigma^2 T)^2} \right). \quad (4) $$

Furthermore, the lower bound is reached if and only if there exist $n_0 \in \mathbb{Z}_N$ and $c \ne 0$ such that $\psi[n_0] = c$ and $\psi[n] = 0$ for all $n \ne n_0$.

## III. EXPECTED BIVECTOR MAGNITUDES

**Definition III.1.** Given $y, y' \in \mathbb{C}^N$, their bivector magnitude is

$$ |y \wedge y'| = \sqrt{\|y\|_2^2 \|y'\|_2^2 - |\langle y | y' \rangle|^2}, \quad (5) $$

i.e., the square root of the determinant of the Gram matrix associated to the indexed family $(y, y')$.

*Note.* Geometrically, $|y \wedge y'|$ is the area under the parallelogram with sides $y$ and $y'$. After normalization by $\mathbb{E}[\|y\|^2]\mathbb{E}[\|y'\|^2]$, it can be interpreted as a quantitative measure of feature diversity.

**Proposition III.2.** Given $y, y' \in \mathbb{C}^N$ and $w, w' \sim \mathcal{N}(\mu\delta, \sigma^2\mathbf{I})$ i.i.d.:

$$ \mathbb{E}[\langle w * y | w' * y' \rangle] = \mu^2 \langle y | y' \rangle \quad (6) $$

where the bracket notation is the Hermitian inner product over $\mathbb{C}^N$.

**Proposition III.3.** Given $y, y' \in \mathbb{C}^N$ and $w, w' \sim \mathcal{N}(\mu\delta, \sigma^2\mathbf{I})$ i.i.d.:

$$ \mathbb{V}[\langle w * y | w' * y' \rangle] = 2\mu^2\sigma^2 \sum_{\tau=0}^{T-1} |\mathbf{R}_{(y',y)}[\tau]|^2 \\ + \sigma^4 \sum_{\tau=-T}^{T} (T - |\tau|) |\mathbf{R}_{(y',y)}[\tau]|^2 \quad (7) $$

**Proposition III.4.** Given a residual hybrid filterbank with $w_1...w_J \sim \mathcal{N}(\mu\delta, \sigma^2\mathbf{I})$ i.i.d. and $\psi_1...\psi_J$, for all $j \ne j'$:

$$ \begin{aligned} \mathbb{E}[|(w_j * \psi_j) \wedge (w_{j'} * \psi_{j'})|^2] = & \mu^4 (\|\psi_j\|_2^2 \|\psi_{j'}\|_2^2 - |\langle \psi_j | \psi_{j'} \rangle|^2) \\ & + 2\mu^2\sigma^2 \sum_{\tau=0}^{T-1} (\|\psi_j\|_2^2 \|\psi_{j'}\|_2^2 - |\mathbf{R}_{(\psi_j, \psi_{j'})}[\tau]|^2) \\ & + \sigma^4 \sum_{|\tau|<T} (T - |\tau|) (\|\psi_j\|_2^2 \|\psi_{j'}\|_2^2 - |\mathbf{R}_{(\psi_j, \psi_{j'})}[\tau]|^2) \end{aligned} \quad (8) $$

<figure><img src="image_3.png" /><figcaption>Fig. 2. Statistics of residual hybrid filterbanks with $J=2$ filters as functions of the parameter $\mu$. Blue, lower curves: $\psi_1$ and $\psi_2$ are Dirac impulses, as in a plain “Conv1D” layer. Orange, upper curves: $\psi_1$ and $\psi_2$ are orthogonal low-pass and band-pass filters. Left: relative standard deviation of energy (lower is better): see Theorem II.4. Right: average cosine distance between the filters (higher is better): see Theorem III.5. $N=32$, $T=16$, 100 i.i.d. trials.</figcaption></figure>

**Theorem III.5.** Given a residual hybrid filterbank with $w_1...w_J \sim \mathcal{N}(\mu\delta, \sigma^2\mathbf{I})$ i.i.d. and nonzero $\psi_1...\psi_J$, for all $j \ne j'$:

$$ \lim_{\mu/\sigma \to +\infty} \frac{\mathbb{E}[|(w_j * \psi_j) \wedge (w_{j'} * \psi_{j'})|^2]}{\mathbb{E}[\|w_j * \psi_j\|_2^2 \|w_{j'} * \psi_{j'}\|_2^2]} = 1 - \frac{|\langle \psi_j | \psi_{j'} \rangle|^2}{\|\psi_j\|_2^2 \|\psi_{j'}\|_2^2}. \quad (9) $$

*Note.* The right-hand side in the equation above is the cosine distance between vectors $\psi_j$ and $\psi_{j'}$. This distance is minimal if and only if $\psi_j = \psi_{j'}$ and maximal if and only if $\psi_j$ and $\psi_{j'}$ are orthogonal.

## IV. NUMERICAL SIMULATION

For simplicity, we consider a residual hybrid filterbank with only $J=2$ filters: $\psi_1$ is a low-pass filter and $\psi_2$ is a band-pass filter, as seen in the center and right columns of Figure 1. As a baseline, we also construct a plain convolutional layer or “Conv1D” for short. The Conv1D layer is obtained by setting $\psi_1 = \psi_2 = \delta$. For both filterbanks, we measure this relative standard deviation by estimating the expected value and variance of energy over 100 i.i.d. trials.

In Figure 2 (left), we observe that, in a non-residual setting, making the non-learned filters $\psi_j$'s autocorrelated (i.e., non-Dirac) increases the typical random fluctuations of energy. This is in accordance with Theorem II.4, which states that, for $\mu=0$, Conv1D is optimal in terms of residual standard deviation of energy. However, we may compensate for it via residual connections (see Theorem II.4): with our filterbank design, setting $\mu=2.5$ leads to a relative standard deviation that is on par with Conv1D with $\mu=0$. Crucially, this shift in expected value does not come at the cost of feature diversity: as seen in Figure 2 (right), on average over 100 i.i.d. trials, the cosine distance between $(w_1 * \psi_1)$ and $(w_2 * \psi_2)$ goes up with $\mu$ for our low-pass/band-pass pair, even so it goes down with $\mu$ for Conv1D. This is in accordance with Theorem III.5.

## V. CONCLUSION

We have presented a simple solution against excessive random fluctuations of energy in hybrid filterbanks. On the theoretical side, it remains to be seen how our characterization of feature diversity scales beyond $J=2$ filters: the distribution of the condition number of residual hybrid filters with random Gaussian weights remain an open question [8]. On the practical side, residual connections could be paired with dilated convolutions with $w_j$'s, as in multiresolution neural networks (MuReNN) [9].
**Lemma VI.1.** Given $y, y' \in \mathbb{C}^N$ and $w, w' \in \mathbb{R}^T$:

$$ \langle w * y | w' * y' \rangle = w^\mathrm{T} \cdot Q_T(y, y') \cdot w' \quad (10) $$

where $Q_T(y, y') [t, t'] = R_{(y',y)} [t' - t]$ for every $0 \le t, t' < T$.

Note. In the circular cross-correlation (Definition II.1), we conjugate the first argument to enable its interpretation as inverse Fourier transform of cross-power spectral density. However, in the Hermitian dot product, we conjugate the second term so as to follow the convention of sesquilinear forms. Such a mismatch explains why Lemma VI.1 permutes indices $j$ and $j'$ when defining $Q_T$.

*Proof.* We write the circular convolution $(w*y)$ as the matrix-vector product $C_T(y) \cdot w$ where

$$ C_T(y) = \begin{pmatrix} y[0] & y[N-1] & \cdots & y[N-T+1] \\ y[1] & y[0] & \cdots & y[N-T+2] \\ \vdots & \vdots & & \vdots \\ y[N-2] & y[N-3] & \cdots & y[N-T-1] \\ y[N-1] & y[N-2] & \cdots & y[N-T] \end{pmatrix} $$

has entries $C_T(y)[n,t] = y[n-t]$ for $0 \le n < N$ and $0 \le t < T$. Likewise: $(w'*x') = C_T(y') \cdot w'$. Hence the bilinear form:

$$ \begin{aligned} \langle w * y | w' * y' \rangle &= (w' * y')^\mathrm{T} \cdot (w * y) \\ &= (C_T(y') \cdot w')^\mathrm{T} \cdot (C_T(y) \cdot w) \\ &= (w')^\mathrm{T} \cdot (C_T(y')^\mathrm{T} \cdot C_T(y)) \cdot w \\ &= w^\mathrm{T} \cdot (C_T(y')^\mathrm{T} \cdot C_T(y))^\mathrm{T} \cdot w', \end{aligned} \quad (11) $$

where $T$ (resp. $H$) denote transpose (resp. conjugate transpose). We recognize the definition of circular cross-correlation:

$$ \begin{aligned} (C_T(y')^\mathrm{T} \cdot C_T(y))^\mathrm{T} [t, t'] &= \sum_{n=0}^{N-1} \overline{C_T(y') [t', n]} C_T(y) [n, t] \\ &= \sum_{n=0}^{N-1} \overline{y' [n-t']} y [n-t] \\ &= \sum_{m=0}^{N-1} \overline{y' [m-(t'-t)]} y [m] \\ &= \mathbf{R}_{(y',y)} [t'-t]. \end{aligned} \quad (12) $$

Defining the above as $Q_T(y, y') [t, t']$ concludes the proof. ■

**Proof of Proposition II.2.** By the cyclic property of the trace, Lemma VI.1 with $w = w'$ and $y = y'$ yields

$$ \begin{aligned} \|w * y\|_2^2 &= w^\mathrm{T} \cdot Q_T(y, y) \cdot w \\ &= \mathrm{Tr}(w^\mathrm{T} \cdot Q_T(y, y) \cdot w) \\ &= \mathrm{Tr}(Q_T(y, y) \cdot w \cdot w^\mathrm{T}). \end{aligned} \quad (13) $$

Thus, by linearity of the expected value and by definition of $Q_T(y, y)$:

$$ \begin{aligned} \mathbb{E}[\|w * y\|_2^2] &= \mathrm{Tr}(Q_T(y, y) \cdot \mathbb{E}[w_j \cdot w_j^\mathrm{T}]) \\ &= \mathrm{Tr}(Q_T(y, y) \cdot (\mu^2 \delta \delta^\mathrm{T} + \sigma^2 \mathbf{I})) \\ &= \mu^2 \mathrm{Tr}(\delta^\mathrm{T} \cdot Q_T(y, y) \cdot \delta) + \sigma^2 \mathrm{Tr}(Q_T(y, y)) \\ &= (\mu^2 + \sigma^2 T) \mathbf{R}_{(y,y)} [0]. \end{aligned} \quad (14) $$

Replacing $\mathbf{R}_{(y,y)}[0]$ by $\|y\|_2^2$ concludes the proof. ■

**Proof of Proposition II.3.** By Proposition II.2:

$$ \begin{aligned} \mathbb{E}[\|w * y\|_2^2] &= \mathbb{E}[(\|w * y\|_2^2)^2] - \mathbb{E}[\|w * y\|_2^2]^2 \\ &= \mathbb{E}[\|w * y\|_2^4] - (\mu^2 + T\sigma^2)^2 \|y\|_2^4. \end{aligned} \quad (15) $$

By applying Lemma VI.1 with $w' = w$ and $y' = y$, and by noting that $w$ is real-valued, the term $\|w * y\|_2^4$ can be expanded as:

$$ \begin{aligned} \|w * y\|_2^4 &= (\|w * y\|_2^2) (\|w * y\|_2^2) \\ &= \left( \sum_{0 \le t, t' < T} \mathbf{R}_y[t' - t] w[t] w[t'] \right) \overline{\left( \sum_{0 \le u, u' < T} \mathbf{R}_y[u' - u] w[u] w[u'] \right)} \\ &= \left( \sum_{0 \le t, t' < T} \mathbf{R}_y[t' - t] w[t] w[t'] \right) \overline{\left( \sum_{0 \le u, u' < T} \overline{\mathbf{R}}_y[u' - u] w[u] w[u'] \right)} \\ &= \sum_{t, t', u, u'} \mathbf{R}_y[t' - t] \overline{\mathbf{R}}_y[u' - u] w[t] w[t'] w[u] w[u'], \end{aligned} \quad (16) $$

where the indices $t, t', u, u'$ range between zero and $(T-1)$. Hence:

$$ \mathbb{E}[\|w * y\|_2^4] = \sum_{t, t', u, u'} \mathbf{R}_y[t' - t] \overline{\mathbf{R}}_y[u' - u] \mathbb{E}[w[t] w[t'] w[u] w[u']]. \quad (17) $$

We distinguish 12 cases in the sum above:

(i) if $t = t' = u = u' = 0$, $\mathbb{E}[w[t]^4]$ is the kurtosis of $w[0]$; i.e., $\mu^4 + 6\mu^2\sigma^2 + 3\sigma^4$. This term is weighted by $|\mathbf{R}_y[0]|^2$.

(ii) if $t = t' = u = u' \neq 0$, $\mathbb{E}[w[t]^4]$ is the kurtosis of $w[t]$; i.e., $3\sigma^4$. There are $(T-1)$ such terms, weighted by $|\mathbf{R}_y[0]|^2$.

(iii) if $0 = t = t' \neq u = u'$, $\mathbb{E}[w[t]^2] = \mu^2 + \sigma^2$ and $\mathbb{E}[w[u]^2] = \sigma^2$. There are $(T-1)$ such terms, weighted by $|\mathbf{R}_y[0]|^2$.

(iv) if $0 = u = u' \neq t = t'$, we refer back to (iii).

(v) if $0 \neq u = u' \neq t = t' \neq 0$, $\mathbb{E}[w[t]^2] = \mathbb{E}[w[u]^2] = \sigma^2$. There are $(T-1)(T-2)$ such terms, weighted by $|\mathbf{R}_y[0]|^2$.

(vi) if $0 = t = u \neq t' = u'$, $\mathbb{E}[w[t]^2] = \mu^2 + \sigma^2$ and $\mathbb{E}[w[t']^2] = \sigma^2$. These terms are weighted by $|\mathbf{R}_y[t']|^2$ for $0 \le t' < T$.

(vii) if $0 = t' = u' \neq t = u$, $\mathbb{E}[w[t]^2] = \sigma^2$ and $\mathbb{E}[w[t']^2] = \mu^2 + \sigma^2$. These terms are weighted by $|\mathbf{R}_y[-t]|^2$ for $0 < t < T$.

(viii) if $0 = t = u' \neq t' = u$, we refer back to (vi).

(ix) if $0 = t' = u \neq t = u'$, we refer back to (vii).

(x) if $0 \neq t = u \neq t' = u' \neq 0$, $\mathbb{E}[w[t]^2] = \mathbb{E}[w[u]^2] = \sigma^2$. These terms are weighted by $|\mathbf{R}_y[t' - t]|^2$ for $0 < t \neq t' < T$.

(xi) if $0 \neq t = u' \neq t' = u \neq 0$, we refer back to (x).

(xii) otherwise, the term equals zero.

The autocorrelation has Hermitian symmetry: $\mathbf{R}_y[-\tau] = \overline{\mathbf{R}}_y[\tau]$. Hence: $|\mathbf{R}_y[-\tau]|^2 = |\mathbf{R}_y[\tau]|^2$. Equation (17) becomes:

$$ \begin{aligned} \mathbb{E}[\|w * y\|_2^4] &= (\mu^4 + 6\mu^2\sigma^2 + 3\sigma^4) |\mathbf{R}_y[0]|^2 && (i) \\ &+ 3(T-1)\sigma^4 |\mathbf{R}_y[0]|^2 && (ii) \\ &+ 2(T-1)(\mu^2\sigma^2 + \sigma^4) |\mathbf{R}_y[0]|^2 && (iii) \text{ and } (iv) \\ &+ (T-1)(T-2)\sigma^4 |\mathbf{R}_y[0]|^2 && (v) \\ &+ 4(\mu^2\sigma^2 + \sigma^4) \sum_{0<\tau<T} |\mathbf{R}_y[\tau]|^2 && (vi) \text{ to } (ix) \\ &+ 2\sigma^4 \sum_{0<t \neq t' < T} |\mathbf{R}_y[t'-t]|^2 && (x) \text{ and } (xi) \\ &= \alpha\mu^4 + \beta\mu^2\sigma^2 + \gamma\sigma^4 \end{aligned} \quad (18) $$

where $\alpha = |\mathbf{R}_y[0]|^2 = \|y\|_2^4$,

$$ \begin{aligned} \beta &= 6|\mathbf{R}_y[0]|^2 + 2(T-1)|\mathbf{R}_y[0]|^2 + 4 \sum_{0<\tau<T} |\mathbf{R}_y[\tau]|^2 \\ &= 2T \|y\|_2^4 + 4 \sum_{\tau=0}^{T-1} |\mathbf{R}_y[\tau]|^2, \end{aligned} \quad (19) $$
$$
\begin{align}
\gamma &= ((3+3(T-1)+2(T-1)+(T-1)(T-2))|\mathbf{R}_y[0]|^2 \nonumber \\
&\quad +4 \sum_{0<\tau<T} |\mathbf{R}_y[\tau]|^2 +2 \sum_{0<t\neq t'<T} |\mathbf{R}_y[t'-t]|^2 \nonumber \\
&= T^2 \|\mathbf{y}\|_2^4 +2 \sum_{t=0}^{T-1} \sum_{t'=0}^{T-1} |\mathbf{R}_y[t'-t]|^2. \tag{20}
\end{align}
$$

Given an integer $\tau$ such that $|\tau| < T$, there are $(T - |\tau|)$ pairs $(t, t')$
in the double sum above such that $t' - t = \tau$. Thus:

$$
\gamma = T^2 \sigma^4 \|\mathbf{y}\|_2^4 + 2\sigma^4 \sum_{\tau=-T}^{T} (T - |\tau|) |\mathbf{R}_y[\tau]|^2 \quad (21)
$$

Using Equation (18) and Proposition II.2, we rewrite Equation (15)
as:

$$
\mathcal{V}[\|\mathbf{w} * \mathbf{y}\|_2^2] = \alpha \mu^4 + \beta \mu^2 + \gamma - (\mu^2 + T \sigma^2)^2 \|\mathbf{y}\|_2^4. \quad (22)
$$

After having replaced $\alpha$, $\beta$, and $\gamma$ by their definitions, we cancel the
term $(\mu^2 + T\sigma^2)^2 \|\mathbf{y}\|_2^4$, which completes the proof. ■

**Lemma VI.2.** Given $\mathbf{y} \in \mathbb{C}^N$ and $\tau \in \mathbb{Z}$, $|\mathbf{R}_y[\tau]| \le \mathbf{R}_y[0]$.

Proof. Given $\tau$, we express $\mathbf{R}_y[\tau]$ as an inverse discrete Fourier
transform and apply the triangular inequality, yielding:

$$
|\mathbf{R}_y[\tau]| \le \frac{1}{N} \sum_{\omega=0}^{N-1} |\hat{\mathbf{y}}[\omega]|^2 e^{2\pi i \omega \tau / N} = \frac{1}{N} \sum_{\omega=0}^{N-1} |\hat{\mathbf{y}}[\omega]|^2. \quad (23)
$$

On the right hand side, we recognize the inverse discrete Fourier
transform of $|\hat{\mathbf{y}}|^2$ at lag zero, which concludes the proof. ■

**Proof of Theorem II.4.** Given $j$, we apply Proposition III.3 with $\mathbf{w} = \mathbf{w}_j$ and $\mathbf{y} = \psi_j$. On one hand, for every $\tau > 0$, $|\mathbf{R}_{\psi_j}[\tau]|^2 \ge 0$. Thus:

$$
\mathcal{V}[\|\mathbf{w}_j * \psi_j\|_2^2] \ge 4\mu^2 \sigma^2 |\mathbf{R}_{\psi_j}[0]|^2 + 2\sigma^4 T |\mathbf{R}_{\psi_j}[0]|^2. \quad (24)
$$

We observe that $2\mu^2\sigma^2 + \sigma^4 T^2 = (\mu^2 + \sigma^2 T)^2 - \mu^4$ and replace
$|\mathbf{R}_{\psi_j}[0]|^2$ by $\|\psi_j\|_2^4$. Hence, after dividing by $T/2$:

$$
\mathcal{V}[\|\mathbf{w}_j * \psi_j\|_2^2] \ge \frac{2}{T} ((\mu^2 + \sigma^2 T)^2 - \mu^4) \|\psi_j\|_2^4. \quad (25)
$$

By applying Proposition II.2 with $\mathbf{w} = \mathbf{w}_j$ and $\mathbf{y} = \psi_j$, we recognize
the squared expected value: $\mathbb{E}[\|\mathbf{w}_j * \psi_j\|_2^2] = (\mu^2 + \sigma^2 T)^2 \|\psi_j\|_2^4$,
yielding the lower bound. On the other hand, Lemma VI.2 yields:

$$
\mathcal{V}[\|\mathbf{w}_j * \psi_j\|^2] \le 4\mu^2 \sigma^4 \sum_{\tau=0}^{T-1} |\mathbf{R}_{\psi_j}[0]|^2 + 2\sigma^4 \sum_{\tau=-T}^{T} (T-|\tau|) |\mathbf{R}_{\psi_j}[0]|^2. \quad (26)
$$

The first sum has $T$ equal terms. For the second sum, we compute:

$$
\sum_{\tau=-T}^{T} (T - |\tau|) = T + 2 \sum_{\tau=1}^{T} (T - \tau) = T + 2 \frac{T(T-1)}{2} = T^2. \quad (27)
$$

Thus, by replacing $|\mathbf{R}_{\psi_j}[0]|^2$ by $\|\psi_j\|_2^4$, and similarly to Equation (25):

$$
\begin{align}
\mathcal{V}[\|\mathbf{w}_j * \psi_j\|_2^2] &\le (4\mu^2\sigma^2 T + 2\sigma^4 T^2)\|\psi_j\|_2^4 \nonumber \\
&\le 2((\mu^2 + \sigma^2 T)^2 - \mu^4)\|\psi_j\|_2^4. \tag{28}
\end{align}
$$

Dividing by $\mathbb{E}[\|\mathbf{w}_j * \psi_j\|_2^2]$ concludes the proof.

Moreover, for $\psi_j : n \mapsto c\delta[n-n_0]$, we compute $\mathbf{R}_{\psi_j} = \tau$ and check the lower bound via Equations (24) and (25). Conversely, the existence of multiple nonzero elements in $\psi_j$ implies that $|\psi_j|^2$ is nonconstant, and thus (sim. Lemma VI.2) that $\mathbf{R}_{\psi_j}$ has at least one nonzero coefficient for some nonzero lag $\tau$, leading to a contradiction. ■

**Proof of Proposition III.2.** Lemma VI.1 yields:

$$
\langle \mathbf{w} * \mathbf{y} | \mathbf{w}' * \mathbf{y}' \rangle = \mathbf{w}_j^\mathrm{T} \cdot \mathbf{Q}_T(\mathbf{y}, \mathbf{y}') \cdot \mathbf{w}_{j'} \quad (29)
$$

By the independence of the Gaussian vectors $\mathbf{w}, \mathbf{w}'$:

$$
\begin{align}
\mathbb{E}[\langle \mathbf{w} * \mathbf{y} | \mathbf{w}' * \mathbf{y}' \rangle] &= \mathbb{E}[\mathbf{w}_{j'}^\mathrm{T} \cdot \mathbf{Q}_T(\mathbf{y}, \mathbf{y}') \cdot \mathbb{E}[\mathbf{w}_j]] \nonumber \\
&= \mu^2 \delta^\mathrm{T} \cdot \mathbf{Q}_T(\mathbf{y}, \mathbf{y}') \cdot \delta \nonumber \\
&= \mu^2 \mathbf{Q}_T(\mathbf{y}, \mathbf{y}')[0, 0]. \tag{30}
\end{align}
$$

Recalling that $\mathbf{Q}_T(\mathbf{y}, \mathbf{y}')[0,0] = \langle \mathbf{y} | \mathbf{y}' \rangle$ concludes the proof. ■

**Proof of Proposition III.3.** By Lemma VI.1:

$$
\begin{align}
\mathbb{E}[|\langle \mathbf{w} * \mathbf{y} | \mathbf{w}' * \mathbf{y}' \rangle|^2] &= \mathbb{E}[\langle \mathbf{w} * \mathbf{y} | \mathbf{w}' * \mathbf{y}' \rangle \langle \mathbf{w} * \mathbf{y}' | \mathbf{w}' * \mathbf{y}' \rangle] \nonumber \\
&= \sum_{t, t', u, u'} \mathbf{R}_{(\mathbf{y}, \mathbf{y})}[t' - t] \mathbf{R}_{(\mathbf{y}', \mathbf{y})}[u' - u] \mathbb{E}[\mathbf{w}[t] \mathbf{w}'[t'] \mathbf{w}[u] \mathbf{w}'[u']], \tag{31}
\end{align}
$$

where the indices $t, t', u, u'$ range between zero and $(T-1)$. We
distinguish six cases in the sum above:

1) if $t = t' = u = u' = 0$, $\mathbb{E}[|\mathbf{w}[t]|^2] = \mathbb{E}[|\mathbf{w}'[t']|^2] = \mu^2 + \sigma^2$. This term is weighted by $|\mathbf{R}_{(\mathbf{y}, \mathbf{y})}[0]|^2$.

2) if $t = t' = u = u' \neq 0$, $\mathbb{E}[|\mathbf{w}[t]|^2] = \mathbb{E}[|\mathbf{w}'[t']|^2] = \sigma^2$. There are $(T-1)$ such terms, weighted by $|\mathbf{R}_{(\mathbf{y}, \mathbf{y})}[0]|^2$.

3) if $0 = t = u \neq t' = u'$, $\mathbb{E}[|\mathbf{w}[t]|^2] = \mu^2 + \sigma^2$ and $\mathbb{E}[|\mathbf{w}'[t]|^2] = \sigma^2$. These terms are weighted by $|\mathbf{R}_{(\mathbf{y}, \mathbf{y})}[t']|^2$ for $0 < t' < T$.

4) if $t = u \neq t' = u' = 0$, $\mathbb{E}[|\mathbf{w}[t]|^2] = \sigma^2$ and $\mathbb{E}[|\mathbf{w}'[t]|^2] = \mu^2 + \sigma^2$. These terms are weighted by $|\mathbf{R}_{(\mathbf{y}, \mathbf{y})}[t]|^2$ for $0 < t < T$.

5) if $0 \neq t = u \neq t' = u' \neq 0$, $\mathbb{E}[|\mathbf{w}[t]|^2] = \sigma^2$ and $\mathbb{E}[|\mathbf{w}'[t]|^2] = \sigma^2$. These terms are weighted by $|\mathbf{R}_{(\mathbf{y}, \mathbf{y})}[t']|^2$ for $0 < t < t' < T$.

6) otherwise, the term equals zero.

We obtain: $\mathbb{E}[|\langle \mathbf{w} * \mathbf{y} | \mathbf{w}' * \mathbf{y}' \rangle|^2] = \alpha\mu^4 + \beta\mu^2\sigma^2 + \gamma\sigma^4$ with
$$
\alpha = |\mathbf{R}_{(\mathbf{y}, \mathbf{y})}[0]|^2 = |\langle \mathbf{y} | \mathbf{y}' \rangle|^2, \quad (32)
$$

$$
\begin{align}
\beta &= 2|\mathbf{R}_{(\mathbf{y}, \mathbf{y})}[0]|^2 + \sum_{\tau=1}^{T-1} (|\mathbf{R}_{(\mathbf{y}', \mathbf{y})}[\tau]|^2 + |\mathbf{R}_{(\mathbf{y}', \mathbf{y})}[-\tau]|^2) \nonumber \\
&= 2 \sum_{\tau=0}^{T-1} |\mathbf{R}_{(\mathbf{y}', \mathbf{y})}[\tau]|^2, \tag{33}
\end{align}
$$

$$
\begin{align}
\gamma &= T |\mathbf{R}_{(\mathbf{y}, \mathbf{y})}[0]|^2 + 2 \sum_{\tau=1}^{T-1} |\mathbf{R}_{(\mathbf{y}', \mathbf{y})}[\tau]|^2 + \sum_{0<t \neq t' < T} |\mathbf{R}_{(\mathbf{y}', \mathbf{y})}[t'-t]|^2 \nonumber \\
&= \sum_{t=0}^{T-1} \sum_{t'=0}^{T-1} |\mathbf{R}_{(\mathbf{y}', \mathbf{y})}[t'-t]|^2 = \sum_{\tau=-T}^{T} (T-|\tau|) |\mathbf{R}_{(\mathbf{y}', \mathbf{y})}[\tau]|^2. \tag{34}
\end{align}
$$

**Proposition III.2 yields:**

$$
\begin{align}
\mathcal{V}[\langle \mathbf{w} * \mathbf{y} | \mathbf{w}' * \mathbf{y}' \rangle] &= \mathbb{E}[|\langle \mathbf{w} * \mathbf{y} | \mathbf{w}' * \mathbf{y}' \rangle|^2] - \mathbb{E}[\langle \mathbf{w} * \mathbf{y} | \mathbf{w}' * \mathbf{y}' \rangle]^2 \nonumber \\
&= (\alpha - |\langle \mathbf{y} | \mathbf{y}' \rangle|^2)\mu^4 + \beta\mu^2\sigma^2 + \gamma\sigma^4. \tag{35}
\end{align}
$$

After having replaced $\alpha$, $\beta$, and $\gamma$ by their definitions, we cancel the
term $(\alpha - |\langle \mathbf{y} | \mathbf{y}' \rangle|^2)$, which completes the proof. ■

**Proof of Proposition III.4.** By Definition III.1, by independence of $\mathbf{w}_j$ and $\mathbf{w}_{j'}$, and by property of the Hermitian dot product in $\mathbb{C}^N$:

$$
\begin{align}
\mathbb{E}[|(\mathbf{w}_j * \psi_j) \wedge (\mathbf{w}_{j'} * \psi_{j'})|^2] & \nonumber \\
&= \mathbb{E}[\|\mathbf{w}_j * \psi_j\|_2^2 \|\mathbf{w}_{j'} * \psi_{j'}\|_2^2 - |\langle \mathbf{w}_j * \psi_j | \mathbf{w}_{j'} * \psi_{j'} \rangle|^2] \nonumber \\
&= \mathbb{E}[\|\mathbf{w}_j * \psi_j\|^2] \mathbb{E}[\|\mathbf{w}_{j'} * \psi_{j'}\|^2] - \mathbb{E}[|\langle \mathbf{w}_j * \psi_j | \mathbf{w}_{j'} * \psi_{j'} \rangle|^2]. \tag{36}
\end{align}
$$

Applying Propositions II.2, III.2, and III.3 concludes the proof. ■

**Proof of Theorem III.5.** By Propositions II.2 and III.4. ■
REFERENCES

[1] G. Richard, P. Chouteau, and B. Torres, "A fully differentiable model for unsupervised singing voice separation," in *Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2024, pp. 946-950.

[2] N. Shlezinger, J. Whang, Y. C. Eldar, and A. G. Dimakis, "Model-based deep learning," *Proceedings of the IEEE*, vol. 111, no. 5, pp. 465-499, 2023.

[3] G. Richard, V. Lostanlen, Y.-H. Yang, and M. Müller, "Model-based deep learning for music information research: Leveraging diverse knowledge sources to enhance explainability, controllability, and resource efficiency," *IEEE Signal Processing Magazine*, vol. 41, no. 6, pp. 51-59, 2025.

[4] D. Haider, F. Perfler, V. Lostanlen, M. Ehler, and P. Balazs, "Hold me tight: Stable encoder-decoder design for speech enhancement," in *Proceedings of the International Speech Communication Association Conference (INTERSPEECH)*, ISCA, 2024.

[5] D. Haider, V. Lostanlen, M. Ehler, and P. Balazs, "Instabilities in convnets for raw audio," *IEEE Signal Processing Letters*, vol. 31, pp. 1084-1088, 2024.

[6] Y. Bengio, I. Goodfellow, and A. Courville, *Deep learning*. MIT Press, 2017.

[7] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*. IEEE, 2016, pp. 770-778.

[8] P. Balazs, D. Haider, V. Lostanlen, and F. Perfler, "Trainable signal encoders that are robust against noise," in *Proceedings of the International Congress and Exposition on Noise Control Engineering (INTER-NOISE)*, vol. 270, no. 10, 2024, pp. 1836-1844.

[9] V. Lostanlen, D. Haider, H. Han, M. Lagrange, P. Balazs, and M. Ehler, "Fitting auditory filterbanks with multiresolution neural networks," in *Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*. IEEE, 2023.