Wavelet Networks: Scale-Translation Equivariant Learning From Raw Time-Series
Authors: David W. Romero
Erik J Bekkers
Jakub M. Tomczak
Mark Hoogendoorn
Venue: Accepted by TMLR ()

Abstract: Leveraging the symmetries inherent to specific data domains for the construction of equivariant neural networks has lead to remarkable improvements in terms of data efficiency and generalization. However, most existing research focuses on symmetries arising from planar and volumetric data, leaving a crucial data source largely underexplored: *time-series*. In this work, we fill this gap by leveraging the symmetries inherent to time-series for the construction of equivariant neural network. We identify two core symmetries: *scale and translation*, and construct scale-translation equivariant neural networks for time-series learning. Intriguingly, we find that scale-translation equivariant mappings share strong resemblance with the *wavelet transform*. Inspired by this resemblance, we term our networks *Wavelet Networks*, and show that they perform nested non-linear wavelet-like time-frequency transforms. Empirical results show that Wavelet Networks outperform conventional CNNs on raw waveforms, and match strongly engineered spectrogram techniques across several tasks and time-series types, including audio, environmental sounds, and electrical signals. Our code is publicly available at https://github.com/dwromero/wavelet_networks.

Published in Transactions on Machine Learning Research (12/2023)
Wavelet Networks: Learning From Raw Time-Series
Equivariant
David W. Romero'
duromero@nvidia.com
NVIDIA Research
Erik J. Bekkers
e.j.bekkerssuww..d
Universitedl van Amsterdam
Jakub M. Tomczak"
j. i.m.tomcakk@tue.l
Technische Universiteit Eindhoven
Mark Hoogendoorn
m.hoogendoomm@v...
Vrije Universiteit Amsterdam
Reviewed on OpenReview: https: // openreview. net/ forun? id- ga5SNul Yet
Abstract
Leveraging the symmetries inherent to specific data domains for the construction of equiv- ariant neural networks has lead to remarkable improvements in terms of data efficiency and generalization. However, most existing research focuses on symmetries arising from planar and volumetric data, leaving a crucial data source largely underexplored: time-series. In this work, we fill this gap by leveraging the symmetrics inherent timo-scries for the con- struction of equivariant neural network. We identify two core symmetries: scale and trans- lation, and construct scale-trae translation equivariant neural networks for time-series learning. Intriguingly, we find that scale-translation equivariant mappings share strong resemblance with the wavelet transform. Inspired by this resemblance, we term our networks Wavelet Networks, and show that they perform nested non-linear wavelet-like time-frequency trans- forms. Empirical results show that Wavelet Networks outperform conventional CNNs on raw waveforms, and match strongly engineered spectrogram tochniques across several tasks and time-series types, including audio, environmental sounds, and electrical signals. Our code is publicly available at https:/ http//gttbb////////////////...tt thub. com/ http///dworrrrwwoeeeee www.eaeeeeeeeeeeeeeeeeeeeee
1 Introduction
Leveraging the symmetrics inherent specific data domains for the construction of statistical models, such as neural networks, has proven highly advantageous, by restricting the model to the family of functions that accurately describes the data. A prime example this principle is Convolutional Neural Networks (CNNs) (LeCun et al., 1989). CNNs embrace the translation symmetries in visual data by restricting their mappings to convolutional structure. Convolutions possess a distinctive property called translation equivariance: if the input is translated, the output undergoes an equal translation. This property endows CNNs with better data efficiency and generalization than unconstrained models like multi-layered perceptrons.
Group equivariant convolutional neural networks (G-CNNs) (Cohen & Welling, 2016) extend equivariance to more general symmetry groups through the use of group convolutions. Group convolutions are group equivariant: if the input is transformed by the symmetries described by the group, e.g., scaling, the output undergoes equal transformation. Equivariance to larger symmetry groups endows G-CNNs with increased data efficiency and generalization on data exhibiting these symmetries. Existing group equivariance research primarily focuses on symmetrics found in visual data, c.g,, planar rotations, planar scaling (Weiler et al., 2018; Worrall Welling, 2019; Sosnovik al., 2020), and more recently, on 3D symmetries, e.g., for spherical
* Work done while the Vrije Universiteit Amsterdam

---

Published in Transactions on Machine Learning Research (12/2023)
Equinetion Map
w/pp
Mm
fomeiies Map
Mp
Tratakation Sealing
Tranolation Seiling
Transaatio Scalleg
Trnen Mag
Equinarias Map
(a)
(b)
(c)
Figure 1: Equivariance, invariance and their impact on the hierarchical representations. a group equivariant map- ping, when the input is transformed by group transformation, its output undergoes equivalent transformation (Fig. la). In contrast, in group invariant maps, the output remains unchanged for all group transformations of the input (Fig. 1b). This distinction holds significant implications in the construction of hierarchical feature representa- tions. For example, a face recognition system built upon invariant eye, nose and mouth detectors would be unable to set the portraits in Fig. ic apart. However, by leveraging equívariant mappings, information about the imput transformations can be used to distinguish these portraits effectively. In essence, in contrast to equivariant maps, invariant maps permit senseless pattern combinations resulting for overly restraining constraints in their design.
and molccular data (Thomas ct al., 2018; Fuchs et al., 2020; Satorras et al., 2021). Yet, an important category remains underexplored, which also exhibits symmetries: time-series. Notably, their translation symmetry is a cornerstone in signal processing and system analysis, e.g., Linear Time-Invarian, (LTT) systems.
In this work, we bridge this gap by constructing neural networks that embrace the symmetrics inherent to time-series. We begin by asking: "What symmetries are inherently present in time-series?" We identify two fundamental symmetries -scale and translation-, whose combination elucidate several phenomena observed in time-series, e.g., temporal translations, phase shifts, temporal scaling, resolution changes, pitch shifts, seasonal occurrences, etc. By leveraging group convolutions equivariant to the scale-translation group, we construct neural architectures such that when the input undergoes translation, scaling or a combination of the two, all intermediate layers will undergo an equal transformation in a hicrarchical manner, akin to the methods proposed by Sosnovik et al. (2020); Zhu el al. (2022) for visual data. Interestingly, we observe that constructing convolutional layers equivariant to scale and translation results in layers that closely resemble the wavelet transform. However, we find that in order to preserve these symmetrics consistently across the whole network, the output of each layer must be processed by a layer that also behaves like the wavelet transform. This approach substantially deviates from common approaches that rely on spectro-temporal representations, e.g., the wavelet transform, which compute temporal representations once and pass their response to a 2D CNN for further processing.
Inspired by the resemblance of scale-translation group equivariant convolutions with the wavelet transform, we term our scale-translation equivariant networks for time-series processing Wavelet Networks. Extensive empirical results show that Wavelet Networks consistently outperform conventional CNNs operating on raw waveforms, and match strongly engineered spectogram-based approaches, e-g,, on Mel-spectrograms, across several tasks and time-series types, c.g., audio, environmental sounds, electrical signals. To our best knowledge, we are first to propose scale-translation equivariant neural networks for time-series processing.
2 Related Work
Learning from raw time-series. Several end-to-end learning approaches for time-series exist (Dieleman & Schrauwen, 2014; Dieleman et al., 2016; Dai et al., 2017; Rethage et al., 2018; Stoller et al., 2018). Given the considerable high-dimensionalityy of time-series, existing works focus on devising techniques with parameter- and compute--fficient large memory horizons (Romero et al., 2021; Goel et al., 2022). Due to small effective memory horizons and long training times, Recurrent Neural Networks (RNNs) (Rumelhart et al., 1985) have gradually been overshadowed by CNN backbones (Bai et al., 2018).
While CNNs are equivariant to translations, they do not inherently incorporate a distinct notion of scale. Although mothods involving layer-wise multi-scale representations have been proposed, e.g., Zhu et al. (2016); Lu et al. (2019); von Platen et al. (2019); Guizzo et al. (2020), these layers are not scale equivariant. As a result, networks incorporating them struggle to maintain consistent scale information across layers.
2

---

Published in Transactions on Machine Learning Research (12/2023)
Group-invariant time-series learning. Learning invariant representations from raw speech and sound has been extensively studied in past. Scattering operators (Mallat, 2012; Bruna & Mallat, 2013) construct group invariant feature representations that can be used to construct neural architectures invariant to scale and translation (Andén & Mallat, 2014; Poddinti et al., 2014; Salamon & Bello, 2015). In contrast to the invariant fealure representations developed by these works, Wavelet networks construct equivarianl feature representations. Since group equivariance is a generalization of group invariance (Fig. 1b, Sec. 3.1), Wavelet Networks accommodate a broader functional family than previous works, while still upholding scale and translation preservation. Notably, equivariant methods shown superior performance compared to invariant methods across several tasks, even for intrinsically invariant tasks like classification (Cohen & Welling, 2016; Zaheer et al., 2017; Maron et al., 2018). This phenomenon stems from the hierarchical form in which neural networks extract features. Enforcing invariance early in the feature extraction process imposes an overly restrictive constraint in the resulting models (Fig. 1c).
Group-cquivariant time-scrics learning. To our best knowledge, Zhang et al. (2015) the only approach that proposes equivariant learning for time-series data. They propose to learn feature representations equiv- ariant to vocal tract length changes inherent symmetry of speech. However, vocal tract length changes do not conform to the mathematical definition of a group, making this equivariance only an approximate es- timation. Interestingly, vocal tract length changes can be characterized by specific (scale, translation) tuples. Consequently, considering equivariance to the scale-translation group implicitly describes vocal tract length changes as well as many other symmetries encountered in audio, speech and other time-series modalities.
3
Background
This work assumes a basic familiarity with the concepts of group; subgroup and group action. For those who may not be acquainted with these terms, we introduce these terms in Appx. A.
3.1 Group equivariance, group invariance and symmetry preservation
Group equivariance. Group equivariance is the property of a mapping to respect the transformations in a group. We say that a map is oquivariant to a group transformation of the input by elements of the group leads an equivalent transformation of the output (Fig. 1a). Formally, for a group G with elements 9 G acting on a set X,, and mapping & . X, we say that ø is equivariant to G if:
(1) For example, the convolution of a signal f: : - IR and a kernel R RR is equinariant to the group of translations -or translation equivariant- because when the input is translated, its convolutional descriptors are equivalently translated, i.e., (vb f f), with Le a translation operator by L: [=f())=[(x-l),
<(gx) = gō(x), E E
Group invariance. Group invariance is a special case of group equivariance in which the output of the map is equal for all transformations of the input (Fig. 1b). Formally, for group G with elements g G XX + X, we say that ø is invariant to G if:
acting on a set X,, and a mapping & :...
xxgz) d(x), Vr e E
(2)
Relation to symmetry preservation. A symmetry-preseerinn mapping preserves the symmetries of the input. That is, if the input has certain symmetries, c.g., translation, rotation, scale, these symmetrics will also be present in the output. Since symmetries are mathematically described as groups, it follows that group equivariant mappings preserve the symmetries the group to which the mapping is equivariant. In contrast, invariant mappings do not preserve symmetry, as they remove all symmetric information from the input.
3.2 gymmetry-preserving mappings: The group and the lifting convolution
When talking about (linear) cemmettry-preservve mappings, are obliged to talk about the group convolu- tion. Previous work has shown that group convolutions are the most general class of group equivariant linear maps (Cohen et al., 2019). Hence, it holds that any linear equivariant map is in fact a group convolution.
Group convolution. Let f: G + R and vi G + R be a scalar-valued signal and convolutional kernel defined on a group G. The group convolution (xe) between f and sb is given by:
(f (g) =/ ./ f())L,())))) duc(7) = =f f(r) (g duc(?).
(3)
3

---

Published in Transactions on Machine Learning Research (12/2023)
Figure 2: Locality of visual and auditory objects. Whereas visual objects are local (left), auditory objects are not. The latter often cover large parts of the frequeney axis in a sparse manner (right).
where 9.7 E G, (z (g (g=\>) is the action of the group G on the kernel v, and jgg (7) is the (invariant) Haar measure of the group & for 7, Notably, the group convolution generalizes the translation equivariance of convolutions to general groups. The group convolution is equivariant in the sense that for all 7.9 E G.
Lf #e ()(()) = (Lif ao (7), with f ()
(4)
The lifting convolution. In practice, the input signals f might not be readily defined on the group of interest G, but on a sub-domain thereof X, i.e., ʃ X + R. For example, time-series are defined on R although we might want to consider larger groups such as the ccale-translation group. Hence, we require a symmetry-preserving mapping from X to & that lifts the input signal G to use group convolutions. This operation is called a lifting convolution. Formally, with f : X R and v : X I scalar-valued signal and convolutional kernel defined on I, and X' sub-group of G, the lifting convolution (+g+) is mapping from functions on X' to functions G defined as:
= -f [()) dpp(x)= = / ((x)(1) 22) d
(f eot (g) -
(5)
Note that, the lifting convolution is also group equivariant mapping. That is, Lg *ct () *ot v).
4 The problem of learning 2D convolutional kernels on the time-frequency plane
CNNs have been major breakthrough in computer vision, yielding startling results in countless applications. Due to their success, several works have proposed to treat ppectro-teppore representations representations on the time-frequency plane- as 2D images and learn 2D CNNs on top. In this section, we delve into the dif- ferences between visual and spectro-lempordl representations and assess the suitability of training 2D CNNs directly on top of ppectro-temporal representations. Our analysis suggest that treating opectro-tempone rep- resentations as images and learning 2D CNNs on top might not be adequate for effective timo-scrics learning.
To enhance clarity, we define spectro-temporal representations in separate gray boxes throughout the section to avoid interrupting the reading flow. Those already familiar with these concepts may skip these boxes.
Spectro-temporal representations. Let f(t) E (2(R) be square integrable function on R. An opectro-temporal representation w[f](tww) R C of f is constructed by means of a linear time- frequency transform tb that correlates the signal f with a dictionary D of localized fime-frequency atoms (0 JeERWERR 6ew R + C of finite energy and unitary norm, i.e., 6e.0 E LIR),, 1l =1, VI E R.RWER R. The resulting spectro-temporal representation ion Φ[ʃ] is given by:
(1) = (J, br..) = J[ (T)
with 0² the complex conjugate of Φ, and (: the dot product of its arguments. Using different time- frequency components &r.w, spectro-temporal representations with different properties can be obtained.
4.1 Fundamental differences between visual representations and spectro-temporal representations
There exist two fundamental distinctions between visual data and spectro-temporal representations, which are universal to all spectro-temporal representations: (i) locality and (ii transparency. Unlike visual data, auditory signals exhibit strong non-local characteristics. Auditory signals consist of auditory objects, e,g., spoken words, which contain components resonating at multiple non-local frequencies known as the har- monics of the signal. Consequently, the spectro-temporal representations of auditory objects often occupy a significant, portion of the time-frequency plane -prticularly along the frequency axis (w)- in sparse manner
4

---

Published in Transactions on Machine Learning Research (12/2023)
MV WVV
Figure 3: Occlusion and superposition. Visual objects occlude each other when they appear simultaneously at given position (left). Auditory objects, instead, superpose at all shared positions (right).
(Fig. 2). Furthermore, when considering auditory signals comprising multiple auditory objects, these objects exhibit phenomenon known superposilion. This properly is notably different from visual data, where visual objects in the same location occlude one another, resulting in only the object closest to the camera being visible (Fig. 3). This inherent property of sound is colloquially referred to as transparency.
4.2 The problem of learning 2D kernels on short-time Fourier spectro-temporal representations
The short-time Fourier transform constructs representation in which a signal is decomposed in terms of its correlation with time-frequency atoms of constant time and frequency resolution. As result, it is effective as long as the signal f does not exhibit transient behavior -components that evolve quickly over time- with some waveform structures being very localized in time and others very localized in froquency.
The short-time Fourier transform. The short-time Fourier transform S -also called windowed Fourier transform is linear time-frequency transform that uses dictionary of time-frequency atoms (tw(T)=w(T - t)e -tut teR, WeR, constructed with symmetric window w(r) of local support shifted by t and modulated by the frequency w. The spectro-temporal representation 8] 8[f] is given by:
S[f(ttw)) (f. f(T))) - f(T)w(( t)e--in -er dr.
(7)
Intuitively, the short-time Fourier transform divides the time- fine-freyyeccy plane in tiles of equal resolu- tion, whose value is given by the correlation between f and the time-frequency atom Φr.w (Fig. 4a).
Nevertheless, decades of research in psychology and neuroscience have shown that humans largely rely in the transient behavior of auditory signals to distinguish auditory objects (Cherry, 1953; van Noorden et al., 1975; Moore & Gockel, 2012). In addition, has been shown that the human auditory system has high spectral resolution al low-frequencies and high temporal resolution at higher frequencies (Stevens et al., 1937; Santoro al., 2014; Bidelman & Khaja, 2014). For example, semitone at the bottom of the piano scale (~30Hz) is of about 1.5Hz, while at the top of the musical scale (~5kHzz) it is of about 200Hz. These properties of the human auditory signal largely contrast both with (f) the inability of the short-time Fourier transform to detect transient signals, as well as with (w) its constant spectro-temporal resolution.
To account for these differences, improved spectro-temporal representations on top of the short-time Fourier transform have been proposed such as log-Mel spectrograms (Stevens al., 1937; Furui, 1986). These devel- opments revolve around transforming the frequency axis of the short-time Fourier transform in logarithmic scale, thereby compressing the frequency axis and better aligning with the spectro-temporal resolution of
Figure Tiling of the time-frequency plane for the short-time Fourier trans- form (Fig. 4a) and the wavelet transform (Fig. 4b). The short-time Fourier trans- form divides the time- lime-frequency plane in tiles of equal resolution. This makes it adequate for signals without tran- sient behaviour. The Wavelet. trans- form, on the other hand, divides the time-frequency plane on tiless of chang- ing spectro-temporal resolution. This allows it to represent detect highly local- ized events both on time and frequency.
Maci
(a)
(b)
5

---

Published in Transactions on Machine Learning Research (12/2023)
the human auditory system. Consequently, this adjustment enables local structures, e.g., 2D convolutional kernels, to better capture non-local relationships (Ulrrich et al., 2014; Choi et al., 2016; Xu et al., 2018). However, despite their improved learning characteristics, these spectro-temporal representations remain in- complete due to their inability to modify the constant temporal resolution of the short-time Fourier transform.
4.3 The problem of learning 2D kernels on Wavelet spectro-temporal representations
In contrast to the short-time Fourier transform, the Wavelet transform constructs a opectro-temporal repre- sentation in terms of correlations with time-frequency atoms, whose time and frequency resolution change. As a result, the resulting decomposition of the time-frequency plane allows the wavelet transform to correctly describe signals with transient behaviour with localized components both on time and frequency (Fig. 4b).
The wavelet transform. The wavelet transform ( is a linear frequeryy transform that uses a dictionary of time-frequency atoms (T) - (-)) E R, w E Rgo- The function the, is called a Wavelet and satisfies the properties of having zero mean, i..., ʃ ( (r)dr=0,s and being unitary, i... 11= for any & E RR, w E RRzo- The resulting spectro-temporal representation (0[s] is given by:
([fl(t,w) = (f. = -
/ f(()) {
T dr.
(8)
Intuitively, the Wavelet transform divides the time-frequency plane in tiles of different resolutions, with high frequency resolution and low spatial resolution at low frequencies, and low frequency resolution and high spatial resolution for high frequencies (Fig. 4b)."
"Importantly, is not possible to have high frequency and spatial resolution at the same time due to the uncertainty principle (Gabor, 1946). It states that the joint time-frequency resolution of spectro-temporal representations is limited by minimum surface aww 2 2. with o Few the spread of the time-froquency atom & on time and froquency.
Interestingly, the modus operandi of the wavelet transform perfectly aligns with the spectro-temporal reso- lution used by the human auditory system for the processing of auditory signals. Nevertheless, despite this resemblance, training local 2D structures, e.g., convolutional kernels, directly on the wavelet transform's 's output still falls short in addressing the non-local, transparent characteristics inherent in auditory signals. Consequently, researchers have devised several strategies to overcome these challenges, e.g., by defining sep- arable kernels that span large memory horizons along the frequency and time axis independently (Pons & Serra, 2019) or by prioritizing learning along the harmonics of a given frequency (Zhang et al., 2020).
As shown in the next section, a better alternative arises from considering the symmetries appearing in time- series data. Starting from this perspective, we are led to scale-translation equivariant mappings and find striking relationships between these family of mappings and the wavelet transform. Nevertheless, our anal- ysis indicates that all layers within neural network should be symmetry preserving -a condition not met by the methods depicted in this section. By doing so, we devise neural archilectures, whose convolutional layers process the output of previous layers in a manner akin to the wavelet transform. As a result, each con- volutional layer performs spectro-temporal decompositions of the input in terms of localized time- frequecyy atoms able to process global and localized patterns both on time and frequency.
5 Wavelet networks: Scale-translation equivariant learning from raw waveforms
We are interested in mappings that. preserve the scale and translation symmetries of lime-series. In this section, we start by tailoring lifting and group convolutions to the scale-translation group. Next, we outline the general form of Wavelet Networks and make concrete practical considerations for their implementation. At the end of this section, we formalize the relationship between Wavelet Networks and the wavelet transform, and provide thorough analysis on the equivariance properties of common spectro-temporal transforms.
5.1 Ecale-translation preserving mappings: group convolutions on the talee-translation group
We are interested in mappings that preserve scale and translation. By imposing equivariance to the scale- trunslation group, we guarantee that if input patterns are scaled, translated, or both, their feature represen- tations will transform accordingly, but not be modified.
The scale-translation group. From a mathematical perspective scale and translational symmetries are described by the affine senle-translation group G=R x R2o, which emerges from the semi-direct. product of
6

---

Published in Transactions on Machine Learning Research (12/2023)
Figure 5: The action of unimodular and non- unimodular groups. Most unimodular groups, e.g., rotation, mirroring, keep the volume of the objects they act upon intact. In contrast, non-unimodular groups, e.g,, scaling, change it through their action.
Bip
totate
scale
the translation group J=(R, +) and the scale group S=(R20. x) acting on R. As a result, we have that the resulting group product is given by g. s) (r, (t ST, 8),, with L, T E R and 8,5 R2z- In addition, by solving g g= we obtain that the inverse of group element g-(t, is given by (--,,)
Semi-direct product and affine groups. When treating data defined on Rd, one is mainly interested in the analysis of groups the form G=R² x H resulting from the semi- product (x) between the translation group (R+,+) +) and an arbitrary (Lie) group H acting on F. e.g., rotation, scale, etc. This kind of groups are called affine groups and their group product defined as:
91 92 (x1, (222 h2) = (x1 + Ah, (22), h1 h2); with (11 h2) E G. ø1, 22 E Rd and h1, h2 E H. A denotes the action of H on Rd.
(9)
Unimodular and non-unimodular groups. Unimodular groups, such rotation, translation and mir- roring, are groups whose action keeps the volume of the objects on which they act intact (Fig. 5). Recall that group convolution performs an integral over the whole group (Eq. 3). Hence, for its result to be invariant over different group actions, it is required for the Haar measure to be equal for all elements of the group -therefore the name invariant Haar measure. Since the action of (most) unimodular groups does not alter the size of the objects on which they act, their action on infinitesimal objects keeps their size unchanged. As consequence, for (most) unimodular groups, the Haar measure is equal to the Lebesgue measure, i.e., d (7)= Vyy E G, and therefore, it is often omitted in literature, c.g., in Cohen & Welling (2016).
In contrast, non-unimodular groups, such as the scale group and the scale-translation group, do modily the size of objects on which they act (Fig. 5 right). Consequently, their action on infinitesimal objects changes their size. As a result, the Haar mcasure must be treated carefully in order to obtain equivariance to non- unimodular groups (Bekkers, 2020). The Haar measure guarantees that d (gr), A g.7 E G. For the scale-translation group, is obtained as:
1
(p)(7)= = dug (17) = duell + ST, 85) = dug + sT)duu((s) = dr ds,
(10)
[8]
[8]
T E RR, 8,5 E Rso; dr, ds are the Lebesgue measure of the respective spaces; and |s| depicts the determinant of the matrix representation of the group element." Intuitively, the Haar measure counteracts the growth of infinitesimal elements resulting from the action of on RxR50-
where (), LTE
Scale-translation group convolutions. The general formulation of the group convolution is given in Eq. 3. Interestingly, the scale-translation group has additional properties with which this formulation can be simplified. In particular, by taking advantage the fact that the scale-translation group is an offine group G-R x 8, with (= x), as well as of the definition of the Haar measure for the scale-translation group in Eq. 10 can reformulate the group convolution for the scale-translation group as:
(f (())) 1/ ((1)0(1011))))))))))))) dup(T)
= [f00 ([t,s)
ds ds= = / f(T,s) Φ (5- -t,)) dr ds [s] dr [s]
= [ - - t,s) dr ds= / (fxx (odo (t,s) ds
1
(11)
'A member the scale group Rz0 acting on N-dimensional space represented matrix diag(s,...,)) Since, its determinant. NN depends the value of the group element s, the factor 1 f Eq. 10 cannot omitted.
7

---

Published in Transactions on Machine Learning Research (12/2023)
Translation Lifting Convolution:
if not v306.e)
(ir)
Taans-atatitiion Group Convolusion:
((ss))
(Fig 1900.0
Figure 6: Scale- translaiann lifting and group convolution. The lifting convolution can be seen a set 1D convolutions with bank of scaled convolutional kernels L.45,, and the group convolution can be seen set of ID convolutions with a bank scaled convolutional kernels daat, followed by an integral over scales ş E R. Their main difference is the scale-translation group
that, for group convolutions, the input f and the convolutional kernel &a are functions whereas for lifting convolutions these are functions on R. Lifting and group convolutions can be seen as spectro- temporal decompositions with large values of relating to coarse features and small values to finer features.
where g= h), >=(r,s) G, f,T E R. and 5 E Bso: and L,b((T))= (s,)) is the (left) action of the scale group 8 on a convolutional kernel vi R x Rşo + R defined on the scale-translation group. In other words, for the scale-translation group, the group convolution can be seen as a set of 1D convolutions with a bank of scaled convolutional kernels (ss., followed by an integral over scales ç E R (Fig. 6, bottom).
seale-translation lifting convolution. Like the group convolution, the lifting convolution can also be simplified by considering the properties of the scale-translation group. In particular, we can rewrite it as:
(f *ot () - J f(x))(( 2x) du)())) / ((r)((--1))sf
dr (f (t, / (T))(((t 7r) [s| / f(T) - (s (T dr
(t)
(12)
where g=(t, (ttt,)),-=(111 ( E G. t E. and S. Ryo: and L. (.)(F.))=u (5 is the (left) action of the scale group 8 on a 1D convolutional kernel sb : RR. In other words, for the seale-translation group, the lift- ing convolution (Fig, 6, top). Note that the Haar measure imposes normalization factor of 1 for group convolutions and of I for the lifting convolution. This is because space on which the group convolutionn is performed (IR x R>o) has an additional dimension relative to the space on which the lifting convolution is performed (TR).
be seen as a set of ID convolutions with a bank of scaled convolutional kernels (
5.2 Wavelet Networks: architecture and practical implementation
The general architecture of our proposed Wavelet networks is shown in Fig. 7. Wavelet networks consist of several stacked layers that respect scale and transla- tion. They consist of a lifting group convolution layer that lifts input time-series to the scale-translation group, followed by arbitrarily many group convolutional layers. At the end of the network, a global pooling layer is used to produce scale-translation invariant representations. Due to their construction, Wavelet networks make sure that common neural operations, e.g., point-wise nonlincar- ities, do not disrupt scale and translation equivariance. This in turn, makes them broadly applicable and easily extendable to other existing neural archi-
Lidd Langorr
Soow
MasPoolRd
GesepSoar
x|.
MaaGuol34
DrajOut
Linwer
ClobekProling
Figure Wavelet networks.
8

---

Published in Transactions on Machine Learning Research (12/2023)
Figure Convolutional kernels on dis- crete and continuous bases. In red the canonical basis used for the construction of the convolutional kernel is shown: delta Dirac for the discrete case, and a B² -p-liie for the continuous case. Pos- sible resulting kernels are shown in blue.
0.5
0.5
(a) Discrete bases (Dirac deltas)
(b) Continnous bases (B?-splines)
(a) linear grid
(b) exponential grid
(c) log-plot exponential grid
Figure 9: Riemann intogration of functions on Ry0 using lincar (9a) and exponential grids (9b, 9c).
tectures, e.g. ResNets (He et al., 2016), U-Nets (Ronneberger et al., 2015).
5.2.1 Group convolutional kernels on continuous bases
Although our previous derivations build upon continuous functions, in practice, computations are performed on discretized versions of these functions. Continuous bases have proven advantageous for the construction of group convolutions as the action of relevant groups often impose transformations not well-defined for discrete bases (Weiler et al., 2018; Bekkers et al., 2018; Weiler & Cesa, 2019). For instance, in the context of scale-translations, scaling kernel [W1, W2, wy] by a factor of two results in a filter [w1, W1.5, W22 W2.5, Wy] wherein the introduced values [w1.5, W2.5) do not exist in the original basis (Fig. 8a).
The most adopted solution to address this problem is interpolation, i.e., deriving the value of [w1.5, W2.5] based on the neighbouring known pixels. However, interpolation introduces spurious artifacts which are particularly severe for small kernels. Instead, we adopt an alternative approach: we define convolutional kernels directly on a continuous basis (Fig. 8b). Drawing from the resemblance of gammatone filters strongly motivated by the physiology of the human auditory system for the processing and recognition of auditory signals (Johannesma, 1972; Hewitt & Meddis, 1994; Lindeberg & Friberg, 2015a)- to BP-splines, we parameterize our filters within -sspiieee basis as in Bekkers (2020). As result, our convolutional filters are parameterized as linear combination of shifted B2-splines (=))= EN w, -T), rather than the commonly used shifted Dirac delta's basis (r))=>>> w;o(r TT).
5.2.2 Constructing discrete scale grid
From the response of the lifting layers onward, the feature representations of wavelet networks possess an additional axis s E Rzo- Just like the spatial axis, this axis must be discretized in order to perform computational operations. That is, we must approximate the scale axis Ryo by finite set of discrete scales {s} ( 1 Smin Inspired by previous work, we approximate the scale axis with a dyadic set {2i (Mallat, 1999; Lindeberg & Friberg, 2015b; Worrall & Welling, 2019). Dyadic sets resemble the spectro-temporal resolution of the human auditory system, and are widely used for discrete versions of the wavelet transform.
Integrating on exponential grids. A subtlety ariscs with respect to integrating over the scale axis when implementing the contin theory in discrete setting that is suitable for numerical compulations. The group convolutions include scale correction factors part of the Haar measure, which makes the integration invariant to actions along the scale axis. That is, the integral of a signal f(s) over scale is the same as that of the same signal f(> -s), whose scale changed by a factor 2 E Ryo:
/
/ f(= 1 / f(s)
{ f(z f(z-1s) 8-22
(13)
JR20
R20
9

---

Published in Transactions on Machine Learning Research (12/2023)
We can translate the scale integration to the discrete setting via Riemann integrals, where we sample the function on a grid and take the weighted sum of these values with weights given by the bin-width:
J f(s) = E(o)
(14)
R20
When the scale grid is linear, the bin-widths A are constant, as depicted in Fig. 9a. When the scale grid is exponential, e-g., 8;=b'= with some base factor, the bin widths are proportional to the scale values at the grid points, i.e,, A x s; (Fig. 9b). In this setting, the factor 1 &, cancels out (up to some constant) with the bin width A,, and integration is simply done by summing the values sampled on the scale grid. Consequently, when working with an exponential grid along the scale axis, the factor in the group convolutions (Eq. 11) becomes ! instcad of 1
s². It is worth mentioning that using an exponential grid is the natural thing to do when dealing with the scale group. The scale group is a multiplicative group with natural distance wetween group elements& 2, s E Ryo defined by || log== s||. Consequently, on an exponential grid, the grid points are spaced uniformly with respect to this distance, as illustrated Fig. 9c.
Defining the discrete scale grid. In practice, Wavelet networks must define the number of scales N₂ to be considered in the dyadic set as well as its limits Smin: 9max Fortunately, it turns out that these values are related to the spatial dimension of the input f itself, and thus, we can use it to determine these values.
Let us consider a signal I and a convolutional kernel & sampled on discrete grids [1, [1, C Z of sizes N,, and Ng, respectively. When we re-scale the convolutional kernel 1h, we are restricted (i) at the bottom of the scale axis by the Nyquist criterion, and (i) at the top of the scale by the scale for which the filter becomes constant in an interval of N, samples. The Nyquist criterion is required to avoid aliasing and intuitively restricts us to compression factor on a such that it becomes as big as 2 grid samples. On the other hand, by having & re-scaled to an extreme to which it is constant in the support of the input signal f. the kernel will only be able to perform average operations.
Considerations regarding computational complezity. Note that the computational cost of Wavelet networks increases lincarly with the number of scales considered. Hence, it is desirable to reduce the number of scales used as much as possible. To this end, we reason that using scales for which the sampled support of vi is smaller than No is unnecessary as the functions that can be described at those scales can also be described -and learned- at the unscaled resolution of the kernel s=1. Therefore, we define the minimum scale Smin=1. Furthermore, we reason that using scales for which the support of the filter overpasses that of the input, i.e, N, < Np. is also suboptimal, as the values outside of the region [1, are unknown. Therefore, we consider the set of sensible scales to be given by the interval [1, NrJ. In terms of a dyadic set {2) (mm this corresponds to the j-values given by the interval [0, 2, .... Jmax s.t. N₂ 27 NNyl.
No
Effect of downsampling on the scale grids used. Neural architectures utilize pooling operations e.g., max- pooling, to reduce the spatial dimension of the input as function of depth. Following the rationale outlined in the previous paragraph, we take advantage of these reductions to reduce the number of scales that representations at given depth should use. Specifically, we use the factor of downsampling as a proxy for the number of scales that can be disregarded. For example, if we use a pooling of 8 at a given layer, subsequent layers should reduce the number of scales considered by the same factor, i.e., 23, For a set of dyadic scales before a pooling layer given by {2) f2ije pj and pooling layer of factor 2", the set of dyadic -p
scales considered after pooling will be given by (2) j=mmin
5.2.3 Imposing wavelet structure to the learned convolutional kernels
In classical spectro-tempore analysis, wavelets are designed to have unit norm ||0||==1 and zero mean I (T) dr=0, These constraints useful for both theoretical and practical reasons including energy preser- vation, numerical stability and the ability to act as band-pass filters (Mallat, 1999). Since Wavelet networks construct time-frequency aeprsennaations of the input, we experiment with an additional regularization loss that encourages the learned convolutional kernels to behave like wavelets. First, we note that lifting and group convolutions inherently incorporate a normalization term -1, .' - in their definitions. Therefore, the normalization criterion is inherently satisfied. To encourage the learned kernels to have zero mean, we for- mulate regularization term that promote this behaviour. Denoting vha as the convolutional kernel at the
10

---

Published in Transactions on Machine Learning Research (12/2023)
d-th layer of neural network with D convolutional layers, the regularization term wavelet is defined as:
wavelet (mmcan(bu))1*
(15)
Interestingly, we observe that enforcing wavelet structure in the learned convolutional kernels consistently yields improved performance across all tasks considered (Sec. 6). This result underscores the potential value of integrating insights from classical signal processing, e-g., teeporall analysis (Scharf, 1991; Mallat, 1999; Daubechies, 2006), in the design of deep learning architectures.
5.3 Wavelet networks perform nested non-linear time-frequency transforms
Interestingly, we can use spectro-temporal analysis to understand the modus operandi of wavelet networks. Our analysis reveals that wavelet networks perform nested time-frequency transforms interleaved with point- wise nonlinearities. In this process, each fime-freuennc transform emerges linear combination of parallel wavelet-like transformations of the input computed with learnable convolutional kernels tb.
The relation between scale-translatio equivariant mappings and the wavelet transform. The wavelet transform shows many similarities to the ccale-translation group and lifting convolutions (Grossmann et al., 1985). In fact, by analyzing the definition of the wavelet transform (Eq. 8), we obtain that the Wavelet transform is equivalent a lifting group convolution (Eq. 12 with w=s) up to a normalization factor >
1 w[f[(t,w)= = [ ((T) Vam (T=t) dr = - S(T)
1
/w
(w (r t)) dr
VW
((T) f 1) dr -1+ -1-
(1) va (s *gt v)) (1,w).
(16)
Furthermore, if we let the imput f be a function defined on the scale-translation group, and let w act on this group according the group structure of the scalo-translation group, we have that the scale-translation group convolution is equivalent to Wavelet. transform whose input has been obtained by previously applied Wavelet transform, up to normalization factor wv=
W[A(),w) = Sro { f(T,s) 1 Vw
1,5) dr ds L o (w- -))s) dr ds ( [ foo
-
R20
=fe (f*2 S Lur (t,5) ds wyw (f xo 2) (t,w) R20
(17)
In other words, lifting and group convolutions on the ccale-translation group can be interpreted as linear time-freee feme-freyuency transforms that adopt time-freeee fine-freuenccy plane tiling akin wavelet transform (Fig. 4b), for which the group convolution accepts wavelet-like spectro-temporal representations as input.
Equivariance propertics of common time-frequency transforms. For completeness, also analyze the equivariance properties of common time-frequency transforms and their normalized reprosentations, e.g., spectrogram. Careful interpretations and proofs are provided in Appx. B.
Let Li,f = f(t - to) and f() = f( (t) to E R, So E Rso, be translation and scaling operators. The Fourier, short-time Fourier and Wavelet transform of Lis and Leaf, f E T,2(R), are given by:
Fourier Transform:
F(wro/((w) = -iwto F ([](ww)
(18)
F[Lm,][[w)
sodl : iI (WW
[(w)) |s0 1 -f ((W))
(19)
Short-Time Fourier Transform:
f(t,,w w) iwto ((
+ [s[[i..f(t,w]] |L ((f(ttw)
(20)
f(t,, so so 8 sow)
|s11 Spw)) (+)
(21)
11

---

Published in Transactions on Machine Learning Research (12/2023)
Wavelet Transform:
([fti[ff(t; w)) Li, w[f((t,) f(t,,w = 550 Le w) - f(t, = | (f (f](t
[ = 14,, ([)
(22)
(23)
(+) Eq. 21 only approximately holds for large windows (see Appx. B.2 for a detailed explanation).
In other words, the Wavelet transform and the scalogram |W|+| are the only time freeuennyy repre- sentations that cxhibit both translation and scaling equivariance in practical way.
Wavelet networks apply parallel time-frequency transforms with learned bases at every layer. So far, our analysis has been defined for scalar-valued imput and convolutional kernels. However, in practice, convolutional layers perform operations between inputs f R + RN and convolutional kernels v R + N X to produce outputs (f wh) R + RNow as the linear combination along the Nu dimension of convolutions with several lcarned convolutional kernels computed in parallel:
( * 1b)))) 2 (J: * vh)), o 21. Nont].
(24)
In practice, both lifting and group convolutional layers adhere the same structure. In a dilation-translation convolutional layer with Nout output channels, Nout independent convolutional kernels, each consisting of Nin channels, are learned. During the forward pass, the input is group-convolved with each of these kernels in parallel. The Nout output channels are then formed by linearly combining the outcomes of the Nin channels. In other words, lifting and group convolutional layers produce lincar combinations of distinct time-frequency decompositions of the input computed in parallel at each layer.
Wavelet networks are forms. Just like in conventional neural architectures, the outputs of lifting and group convolutional layers are interleaved with point-wise nonlinearities. Therefore, wavelet networks compute nonlinear ceae-translation equivariant feature representations that resemble nested nonlinear time-frequency transforms of the input.
equivariant nested non-linear time- frequenny trans-
6 Experiments
In this section, we empirically evaluate wavelet networks. To this end, we take existing neural architectures designed to process raw signals and construct equivalent wavelet works (W-Nets). We then compare the performance of W-Nets and the corresponding baselines on tasks defined on raw environmental sounds, raw audio and raw electric signals. We replicate as close as possible the training regime of the corresponding baselines and utilize their implementation as a baseline whenever possible. Detailed descriptions of the specific architectures as well as the hyperparameters used for each experiment are provided in Appx. C.
6.1 Classification of environmental sounds
First, we consider the task of classifying environmental sounds the UrbanSound8K (USSK) dataset (Salamon et al., 2014). The US&K dataset consists of 8732 audio clips uniformly drawn from 10 environmental sounds, c.g., siren, jackhammer, etc, of 4 seconds or less, with total of 9.7 hours of audio.
Experimental setup. We compare the Mn-Nets of Dai et al. (2017) and the IDCNNs of Abdoli et al. (2019) with equivalent W-Nets in terms of number of layers and parameters. Contrarily to Dai et al. (2017) we sample the audio files at 22. 22.05kHz as opposed to 8kHz. This results from preliminary studies of the data, which indicated that some classes become indistinguishable for the human ear at such low sampling rates.? 2 For the comparison with the 1DCNN of Abdoli et al. (2019), we select the 50999-1DCNN as baseline, is the network type that requires the less human engineering. We note, however, that we were unable to replicate the results reported in Abdoli et al. (2019). In contrast to the 83=1,3% reported, we were only able to obtain a final accuracy of 62.0+6.779.. This inconsistency is further detailed in Appx. C.I.
it
To compare to models other than Mn-nets and IDCNNs, e.g., Pons et al. (2017a); Tokozume & Harada (2017), we also provide 10-fold cross-validation results. This is done by taking 8 of the 10 official subsets for
See https: http://tww....eeeeeeeeeeeeeeeeee
http::/c....tttoate.e.e..eeteenn www.ssanaaaassssaa ipynb.
12

---

Published in Transactions on Machine Learning Research (12/2023)
Table Experimental results on UrbanSoundSK.
UrbanSound8K
10 FOLD
CROSS VAL.
MODEL
Acc. (%)
Acc. (%)
PARAMS.
M3-NET
54.48
220.67k
W3-NET
61.05
W3-NET-WL
63.08
219.45K
M5-NET
69.89
558.084
W5-NET
72.28
W3-NET-WL
74.55
558.03K
MI1-NET
74.43
1.7841M
WII-NET
79.33
66.9 96.97 + 5.178
WI1-NET-WI
80.41
68.47 + 4.914
1.806M
MI8-NET
69.65
3. 3,680M
W1S-NET
75.87
64.02 + 4.645
3.759M
W18 NET-WL WL
78.26
65.01
+: 5.431
M31-NET
75.15
3.978M
W34-NET
76.22
65.69 + 5.780
W34 NET- WL
78.38
66.77 +: 4.771
4.021M
IDCNN
62. + 6.791
453 42K
W. WDCNN
62.47 +: 4.925
W W-IDCNN-WL
62.64 + 4.979
458.61K
Comparison With Other Approaches
CROSS- VALL. Acc. (%)
MODEL
TYPE
# PARAMS.
WI1 WIT-NET-WL
Raw
68.47 + 4.914
1.806M
PICZARCNN PIeZAK (2015)
73.7
26M
MEL SPECTROCRAM
VGG Poss & SERRA (2019)
70.74
77M
ENVNET-v2 ToKozUMe & HARADA (2017)
Raw (BaggIng)
78
training, one for validation and one for test. We consistently select the mod10 subset for validation when lesting on the n-th subset. We note that this training regime might be different from those used in other works, as previous works often do not disclose which subsets are used for validation.
Results. Our results (Tab. 1) show that wavelet networks consistently outperform CNNs on raw waveforms. In addition, they are competilive speetrograme approaches, while using significantly fewer parameters and bypassing the need for preprocessing. Furthermore, we observe that encouraging wavelet structure to the convolutional kernels denoted by the WL suffix consistently leads to improved accuracy.
6.2 Automatic music tagging
Next, consider the task of automatic music tagging on the MagnaTaggATine TagATune (MTAT) dataset (Law et al., 2009). The MTAT dataset consists of 25879 audio clips with a total of 170 hours of audio, along with several per-song tags. The goal of the task to provide the right tags to each of the songs in the datasct.
Experimental setup. Following Lee al. (2017), we extract the most frequently used 50 tags and trim the audios 29.1 seconds at sample-rate of 22.05kHzz. Following the convention in literature, we use ROC- curve (AUC) and mean average precision (MAP) performance metries. We compare the best performing model of Lee et al. (2017), the 3e-Net with corresponding wavelet network denoted W39-Net
Results. Our results (Tab. 2) show that wavelet networks consistently outperform CNNs on raw waveforms and perform competitively to rogra-bggeed approaches in this dataset as well. In addition, we observe that encouraging the learning of wavelet-like kernels consistently results increased accuracy as well.
6.3 Bearing fault detection
Finally, we also validate Wavelet networks for the task of condition monitoring in induction motors. To this end, we classify healthy and faulty bearings from raw data provided by Samotics. The datasct consists of 246 clips of 15 seconds sampled 20kHz. The dataset is slightly unbalanced containing 155 healthy and 91 faulty recordings [155,91 91). The dataset is previously split into training set of [85, 52) and test
13

---

Published in Transactions on Machine Learning Research (12/2023)
Table 2: Experimental results on MTAT.
Magna Tune
AVERAGE AUC
MAP
MODEL
# PARAMS.
PER-CLASS
PER-CLIP
PER-CLASS
PER-CLIP
3"-NET
0.893
0.936
0.385
0.700
2.394M
W3'. -NTT
0.895
0.941
0.397
0.719
-3T-NET-WL
0.899
0.943
0.404
0.723
2.404M
Comparison With Other Approaches
AVERAGE AUC
MAP
MODEL
PARAMS.
PER-CLASS
PER-CLIP
PER-CLASS
PER-CLIP
PCNN LIU ET AL. (2016)
0.9013
0.9365
0.4267
0.6902
CNN PONS ET AL. (2017A)*
0.8905
0.3492
11.8M
(RAw)
CNN PONS ET AL. (2017A)*
(SPECT.)
0.9040
0.3811
5M
CNN PONS ET AL. (2017B)
(SPRCT.) Reported results are obtained in a more difficult version of this dataset.
0.893
191K
Table 3: Experimental results on bearing fault detection.
MODEL
Acc. (%)
PARAMS.
MI1-NET 65.1376
1.806M
WI1-NET
68.8073
1.823M
W11-NET-WT.
70.207
set of [70, 39] samples, respectively. These splits are provided ensuring that measurements from the same motor are not included both in the train and the test set. We utilize 20% of the training set for validation. Each clip is composed of 6 channcls measuring both current and voltage the 3 poles of the motor.
Experimental setupp We take the best performing networks on the US8K dataset: the M-11 and W-11 networks, and utilize variants of these architectures for our experiments on this dataset.
Results. Once again we observe that Wavelet networks outperform CNNs on raw waveforms and encouraging the learning of wavelet-like kernels consistently improves accuracy (Tab. 3).
6.4 Discussion
Our empirical results results firmly establish wavelet networks as a promising avenue for learning from raw time-series data. Notably, these results highlight that considering the symmetries inherent to time-series data namely translation and scale- for the development of neural networks consistently leads to improved outcomos. Furthermore, we observe that the benefits of wavelet networks extend beyond sound and audio domains. This result advocates for the use of wavelet networks and scale-translation equivariance for learning on time- scries data from diffcrent sources, c.g., financial data, sensory data. Finally, we also note that promoting the learning of wavelet-like convolutional kernels consistently leads to improved outcomes. We posit that this discovery may hold broader implications for group equivariant networks in general.
Relation to scale-equivariant models of images and 2D signals. In the past, multiple scale-equivariant
models have been proposed for the processing of images and 2D signals (Worrall & Welling, 2019; Sosnovik et al., 2020; 2021). Interestingly, we find that the difference in the lengths of the inputs received by image and time-series models leads very different insights per modality. For comparison, Sosnovik et al. (2020) considers images up to 96x96 pixels, whereas audio files in the US8K dataset are 32.000 samples long. We find that this difference in input lengths has crucial implications for how scale interactions within scale- equivariant models function. Sosnovik et al. (2020) mentions that using inter-scale interactions introduces additional equivariance errors due to the truncation of the set S. Therefore, their networks are built with either no scale interaction or interactions of maximum 2 scales. This strongly contrasts with time-series where incorporating inter-scale interactions consistently leads to performance improvements. In our case, the number of scales and inter-scale interactions rather constrained by the size and computational cost of convolutional kernels (Sec. 5.2.2) rather than their potential negative impact on the model's accuracy.
14

---

Published in Transactions on Machine Learning Research (12/2023)
7 Limitations and future work
Memory and time consumption grows proportionally to the number of scales considered. The biggest limitation of approach is the increase in memory and time demands the number of scales considered grows. One potential avenue to mitigate this could involve adopting Monte-Carlo approximations for the computation of group convolutions (Finzi et al., 2020). This stralegy might not only establish equivariance to the continuous scale group -in expectation-, but also dramatically reduce the number of scales considcred in each forward pass. Another intriguing direction lies in the extension of partial cquivariance (Romero & Lohit, 2022) to the scale group. This extension would enable learning the subset scales to which the model is equivariant, which in turn could lead faster execution and enhanced adaptability, Lastly, the adaptation of separable group convolutions (Knigge et al., 2022) offers a means reduce the computational and memory requirements of wavelet networks.
Convolutions with large convolutional kernels: parameterization and efficiency. The foundation of our approach hinges on computing convolutions with banks of dilated convolutional kernels (Eq. 12, 11). Consequently, considering how these kernels are parameterized as well as how these convolutions are com- puted can unveil avenues for future improvement. Recently, Romero et al. (2021) introduced an expressive continuous parameterization for (large) convolutional kernels that has proven advantageous for complex lasks such large language modelling (Poli et al., 2023) and processing DNA chains (Nguyen et al., 2023). Exploring the use of this parameterization for wavelet networks could lead to valuable insights and improve- ments, potentially surpassing the current utilization of B²-spline bases. Furthermore, convolutional networks that rely on convolutions with very large convolutional kernels, e.g., Romero et al. (2021); Poli et al. (2023); Nguyen et al. (2023), leverage the Fourier transform to compute convolutions in the frequency domain. In the context of wavelet networks, dynamically selecting between spatial and Fourier convolutions based on the size of convolutional kernels has the potential to significantly improve their efficiency.
8 Conclusion
In conclusion, this study introduces Wavelet Networks, a new class of neural networks for raw time-series processing that. harness the symmetries inherent. to time-series data. -caae and translation- for the construc- tion of neural architectures that respect them. We observe clear connection between the wavelet transform and translttannltioon group convolutions, establishing a profound link between our approach and classical ppectro-temporc analysis. In contrast to the usual approach, which uses ppectro-temporal representations as a frontend for the subsequent use of 2D CNNs, wavelet networks consistently preserve these symmetries across the whole network through the use of convolutional layers that resemble the wavelet transform. Our analysis reveals that wavelet nelworks combine the benefits of wavelet-like free-eeyy decompositions with the adaptability and non-linearity of neural networks.
Our empirical results demonstrate the superiority of Wavelet Networks over conventional CNNs raw time-series data, achieving comparable performance to approaches that rely on engineered spectrogram- based methods, e.g., log-Mel spectrograms, with reduced parameters and no need for preprocessing.
This work pioncers the concept of scale-translation equivariant neural networks for time-series analysis, opening new avenues for time-series processing.
References
Sajjad Abdoli, Patrick Cardinal, and Alessandro Lameiras Koerich. End-to-end environmental sound classi- fication using 1d convolutional neural network. Expert Systems with Applications, 136:252-263, 2019.
Joakim Andén and Stéphane Mallat. Deep scattering spectrum. IEEE Trunsactions on Signal Processing, 62(16))4114-4422 2014.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arxiv:1803.0 01211, 2018
Erik J Bekkers. B-spline {enn]s on lie groups. In International Conference on Learning Representations, 2020. URL stttp::/opennrviee..eett net:forrumidd-HHgga.aaeee
15

---

Published in Transactions on Machine Learning Research (12/2023)
Erik J Bekkers, Maxime W Lafarge, Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, and Remco Duits. Roto-translation covariant convolutional networks for medical image analysis. In International Conference on Medical Image Computing and Computer-Assisterr Intervention, pp. 440 448. Springer, 2018
Gavin M Bidelman and Ameenuddin Syed Khaja. Spectrotemporal resolution tradeoff in auditory processing revealed by human auditory brainstem responses and psychophysical indices. Neuroscience letters, 572: 53-57, 2014.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions pattern analysis and machine intelligence, 35(8):1872 1886, 2013.
E Colin Cherry. Some experiments on the recognition of speech, with one and with two ears. The Journal of the acoustical society of America, (5(5):975-9773 1953.
Keunwoo Choi, George Fazekas, and Mark Sandler. Automatic tagging using deep convolutional neural networks. Xrviv preprint Xiv::660 00298, 2016.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 1990-2990 2016.
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. In Advances in Neural Information Processing Systems, pp. 9142-9153, 2019.
Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das. Very deep convolutional neural networks for raw waveforms. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 421-425. IEEE, 2017.
Ingrid Daubechies. Fundamental papers in wavelet theory. Princeton University Press, 2006.
Sander Dieleman and Benjamin Schrauwen. End-to-end learning for music audio. In 2014 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6964-6968. IEEE, 2014.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convolutional neural networks. ar preprint arxvv:1602. 02660, 2016.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. arXiv preprint arXiv:2002.12880 2020.
Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling, Se t3)-transformers: 3d roto-translation equivariant attention networks. Xi preprind arXio:2006. 10503, 2020.
Sadaoki Furui. Speaker-indeppendett isolated word recognition based on emphasized spectral dynamics. In ICASSP'80 86. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 11, pp. 1991 1994. IEEE, 1986.
Dennis Gabor. Theory of communication, part 1: The analysis of information. Journal of the Institution of Electrical Engineers-Pau III: Radio and Communication Engineering, 93(26):429-444 1946.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. In International Conference on Machine Learning, pp. 7616-7633. PMLR, 2022.
Alex Grossmann, Jean Morlet, and T Paul. Transforms associated to square integrable group representations. i. general results. Journal of Mathematical Physics, 26(10):2473 2479, 1985.
Eric Guizzo, Tilmman Weyde, and Jack Barnett Leveson. Multi-time-scale convolution for emotion recognition from speech audio signals. In ICASSP 2020-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6489 6493. IEEE, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770 770-778, 2016.
16

---

Published in Transactions on Machine Learning Research (12/2023)
Michael Hewitt and Ray Meddis. A computer model of amplitude-mocuultiin sensitivity of single units in the inferior colliculus. The Journal of the Acoustical Sociely of America, 95(4):2145-2153) 1994.
PLM Johannesma. The pre-response stimulus ensemble of neurons the cochlear nucleus. Symposium on Hearing Theory, 1972. IPO, 1972.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprint
arXiv:1422 6980, 2014.
David M Knigge, David W Romero, and Erik J Bekkers. Exploiting rodundancy: Separable group convolu- tional networks on lie groups. In International Conference Machine Learning, pp. 11359 11386. PMLR,
2022.
Edith Law, Kris West, Michael I Mandel, Mert Bay, and J Stephen Downie. Evaluation algorithms using games: The case of music tagging. In ISMIR, pp. 137-392, 2009.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1 (4):541 551, 1989.
Jongpil Lee, Jiyoung Park, Keunhyoung Luke Kim, and Juhan Nam. Sample-level deep convolutional neural networks for music aulo-lagging using raw waveforms. arXiv preprinl arXiv: 17:31700.01 789, 2017.
Tony Lindeberg and Anders Friberg. Idealized computational models for auditory receptive fields. PLoS one, 10(3), 2015a.
Tony Lindeberg and Anders Friberg. Scale-space theory for auditory signals. In International Conference on Scale Space and Variational Methods in Computer Vision, pp. 3-15. Springer, 2015b.
Jen-Yu Liu, Shyh-Kang Jeng, and Yi-Hsuan Yang. Applying topological persistence in convolutional neural network for music audio signals. arXiv preprint wxXiv16608.07373 2016.
Xugang Lu, Peng Shen, Sheng Li, Yu Tsao, and Hisashi Kawai. Deep progressive multi-scale attention for acoustic event classification. arXiv preprinl 1Xix:1912.10011 2019.
Stéphane Mallat, A wavelet tour of signal processing. Elsevier, 1999.
Stéphane Mallat. Group invariant scattering. Communications on Pure and Applied Mathematics, 65())): 1331-1398, 2012.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint ar Xro: 1812 09902, 2018.
Brian CJ Moore and Hedwig E Gockel. Properties of auditory stream formation. Philosophioal Transactions of the Royal Society B: Biological Sciences, 367(1991):919 931, 2012.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. arXiv preprint 2rXiv:2306.111 15794, 2023.
Vijayaditya Peddinti, TaraN Sainath, Shay Maymon, Bhuvana Ramabhadran, David Nahamoo, and Vaib- hava Goel. Deep scattering spectrum with deep neural networks. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 210 214. IEEE, 2014.
Karol Piczak. Environmental sound classification with convolutional neural networks. In 2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2015.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint Xic:23302 10866, 2023.
17

---

Published in Transactions on Machine Learning Research (12/2023)
Jordi Pons and Xavier Serra. Randomly weighted cnns for (music) audio classification. In ICASSP 2019- 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 336-340. LEEE, 2019.
Jordi Pons, Oriol Nicto, Matthow Prockup, Erik Schmidt, Andreas Ehmann, and Xavier Serra. End-to-end learning for music audio tagging at scale. arXiv preprint arXiv:1711.02522 2017a.
Jordi Pons, Olga Slizovskaia, Rong Gong, Emilia Gómez, and Xavier Serra. Timbre analysis of music audio signals with convolutional neural networks. In 2017 25th European Signal Processing Conference (EUSIPCO), pp. 2744-2748. IEEE, 2017b.
Dario Rethage, Jordi Pons, and Xavier Serra. A wavenet for speech denoising. In 2018 IEEE International Conference Acouslics, Speech and Signal Processing (TCASSP), pp.
IEEE, 2018.
David W Romero and Suhas Lohit. Learning partial equivariances from data. Advances in Newral Information Processing Systems, 35:36466-364777 2022.
David W Romero, Erik J Bekkors, Jakub M Tomczak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. arXiv preprinl arxi:22022 20::0000,,,830, 2020.
David W Romero, Anna Kuzina, Erik Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102 10::2102002211 1, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Compuling and Compuler-Assisted International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part 1II 18, pp. 234-222222 241. Springer, 2015.
Assisted Intervention MICCAI 2015: 18th
David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. Learning internal reprosentations by error propagation, 1985.
Justin Salamon and Juan Pablo Bello. Feature learning with deep scattering for urban sound analysis. In 2015 23rd European Signal Processing Conference (EUSIPCO), pp. 724 724-728. TEFE, 2015.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference Multimedia, pp. 1041 1044, 2014.
Roberta Santoro, Michelle Moerel, Federico De Martino, Rainer Goebel, Kamil Ugurbil, Essa Yacoub, and Flia Formisano. Encoding of natural sounds at multiple spectral and temporal resolutions in the human auditory cortex. PLoS computational biology, 10(1), 2014.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. International conference on machine learning, pp. 9323 93322 PMLR, 2021.
Louis Scharf. Statistical signal processing, volume 98. Addison Addison-Wesley Reading, MA, 1991.
Ivan Sosnovik, Michat Szmaja, and Arnold Smeulders. Scale-equivariant steerable networks. In International Conference Learning Representations, 2020. URL
Ivan Sosnovik, Artem Moskalev, and Arnold WM Smoulders. Scale equivariance improves siamese tracking. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2765-2774 2021.
Stanley Smith Stevens, John Volkmann, and Edwin Newman. A scale for the measurement of the psycho- logical magnitude pitch. The Journal of the Acoustical Society of America, 8(3):185-190, 1937.
Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for and-to-end audio source separation. arXiv preprint arXio:1800 806803186.....0 2018.
18

---

Published in Transactions on Machine Learning Research (12/2023)
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor Field Networks: Rotation-and Conusattio-Rquivarinnn Neural Networks for 3D Point Clouds. arXiv preprint 102.00898, 2018.
Yuji Tokozume and Tatsuya Harada. Learning environmental sounds with end-to-end convolutional neural network. In 2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 2721 2725. IEEE, 2017.
Karen Ulrrihh, Jan Schlüter, and Thomas Grill. Boundary detoction in music structure analysis using convolutional neural networks. In ISMIR, pp. 417-422, 2014.
Leo Paulus Antonie Servatius van Noorden et al. Temporal coherence in the perceplion of lone sequences, volume 3. Institute for Perceptual Research Eindhoven, the Netherlands, 1975.
Patrick von Platen, Chao Zhang, and Philip Woodland. Multi-span acoustic modelling using raw waveform signals. arXiv preprint 1xXv:1906.11047; 2019.
Maurice Weiler and Gabriele Cesa. General e equiiivrar steerable cnns. In Advances in Neural Infor- mation Processing Systems, pp. 14334-14345, 2019.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnms. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849 858, 2018.
Daniel E Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. arXio preprinl arXiv:1905. 11697, 2019.
Yong Xu, Qiuqiang Kong, Wenwu Wang, and Mark D Plumbley, Large-scale weakly supervised audio clas- sification using gated convolutional neural network. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 121 125. IEEE, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexan- der J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXie preprint arXiv:1212.570) 2012.
Chiyuan Zhang, Stephen Voinca, Georgios Evangelopoulos, Lorenzo Rosasco, and Tomaso Poggio. Discrimi- native template learning in group-convolutional networks for invariant speech representations. In Sixteenth Annual Conference of the International Speech. Communication Association, 2015.
Zhoutong Zhang, Yunyun Wang, Chuang Gan, Jiajun Wu, Joshua B. Tenenbaum, Antonio Torralba, and William T. Freeman. Deep audio priors emerge from harmonic convolutional networks. In International Conference on Learning Representations, 2020. URL
Wei Zhu, Qiang Qiu, Robert Calderbank, Guillermo Sapiro, and Xiuyuan Cheng, Beeling-translatiig equivariant networks with decomposed convolutional filters. The Journal of Machine Learning Research, 23(1):2958-33 -008,, 2022.
Zhenyao Zhu, Jesse H Engel, and Awni Hannun, Learning multiscale features directly from waveforms. arxiv preprint Xiv:::1603 09509, 2016.
19

---

Published in Transactions on Machine Learning Research (12/2023)
Appendix
A Group and group action
Group. A group is an ordered pair (G. where G is a set and :: G + is binary operalion on G, such that (i) the set is closed under this operation, (it) the operation is associative, ie., (g1 92) 93 = 91 (g2 ga). 91, 92. E G, (íit) there cxists an identity clement e E G such that Vg E & we have e g for each g G, there exists an inverse y such that g 9 ----
= g e = 9, and (iv)
Subgroup. Given group (), say that subset H is subgroup of G if the tuple (H, also complies to the group axioms. For example, the set of rotations by 90", H= 90", 2,00,,700 is subgroup of the continuous rotation group as it also complies to the group axioms.
Group action. Let G be a group and I'' a set. The (left) group action of G on X' is a function
x +X, AZ:I-''
(25)
such that for any 91, 92 E G, 9- o Agg In other words, the action of G on X describes how the elements in the set x E X are transformed by elements E G. For brevily, Ag (x) is written as ge.
Equivariance properties of common time- feeeefrreuency transforms
B.1 The Fourier transform
The Fourier transform represents a function with finite energy L2( as sum of complex sinusoidal waves c'wt= coot=coouu wt sinwt.
f(t) 2x [ f(w) ciwe dw.
where, f(w) depicts the amplitude of each component but in f. The Fourier transform I is defined as:
([f](w) = f(w) = (J, ( =f f(t) e iwt dt.
In other words, the Fourier transform encodes f into a Lime-frequency dictionary (wt JUER. Input translation. Let Li, (f))==((( 1o) be translated version of f. Its Fourier transform is given by:
(=
{(f(((ll(w)= - f(t - to) e eiv dt i-t - to; dé=dt
- [(i) e -i((tte) di = twta { f(i) e iwi di = -iwta (f)(
(26)
In other words, a translation of to corresponds to phase modulation of e -twto in the frequency domain. Input scaling. Let de,lf((t==(s5) i),, 80 E R>o, be scaled version of f. Its Fourier transform equals:
F(fr)
1(5 e iut di i= di=
0
f(i) e im(()) d(30t) so
f(1) (((ow) dē
[)( = 1f
(27)
In other words, we observe that dilation on the time domain produces a compression in the Fourier domain times the inverse of the dilation.
Simultaneous input translation and scaling. Following the same derivation procedure, we can show the behavior of the Fourier transform to simultaneous translations and dilations of the input:
{( = spe twta (f)(sw
= - tuto (F11(())
(28)
This corresponds to the superposition of the previously exhibited behaviours.
20

---

Published in Transactions on Machine Learning Research (12/2023)
Effect of input transformations the spectral density. The spectral density of function f E (R) is given by (.)). Input translations and dilations produce the following transformations:
| [+(+) = =
(29)
(30)
Equivariance and invariance propertics of the Fourier transform. From Eq. 26 we that the
Fourier transform is translation equivariant it encodes translations of the input as phase modulation of the output. In addition, it is also scale equivariant (Eq. 27), as it encodes dilations of the imput as a modulation of the frequency components in the output. We can prove that the Fourier transform is dilation and translation equivariant by showing that the output transformations -ittt and soL. are group representations of the translation and scaling group in the Fourier space.
Group representation. Let G be group and f be a function on given functional space (xC). The (left) regular reprosentation of G is a linear transformation L G x Lo(X) + Lv(XC) which extends group actions to functions on (xC) by:
Lg f (()x(zz) = f(x) + f'(x) f(g f2)
such (hat for any 92 E G, foan= Lg-- In other words, the group representation describes how a function on a functional space f E Lv(X) is modified by the effect of group elements g E G.
We can show that the combination of imput translations to, t E R or dilations so. $1 Ry0 produces a trans- formation on the Fourier domain that preserves the group structure. In other words, that the transformations previously outlined are group representations. Specifically, for L', [f]] and L',, (f [f]] it holds:
3 (fio (w) -tttte wwwtee-wtty twto I [[][ -w((t)+ F(f((w) = L Fourler (9F [f] Fourier [f]]][w)
F [L. [L [f]]] (w) = 3I [ (9([]) (w) = [ spo) = 1
with L. Fourier [ [[] tw (/([( the representation of the Fourier transform for the translation group, the representation of the Fourier transform for the dilation group.
and L Fourter [3[5](w)=
Unfortumatcly, the resulting group reprosentations rapidly bocome cumbersome specially the presence of several input components. In addition, although the calculation of the spectral density leaves the scale equivariance property of the transformation unaffected, Eq. 29 shows that it reduces translation equivariance of the Fourier transform to translation invariance. This is why the Fourier transform is commonly considered not to carry positional information.
B.2
The short-time Fourier transform
The short-time Fourier transform of signal L2(RR) is given by:
8[f((t,w) - f(T)W(T - f) e -
iw dr.
In other words, encodes the input. into a time-frequency dictionary D D=(dr,), diw--((7 e-iwr Input translation. Let Lio(f((r==((rtt)) be translated version of f. Its short-time Fourier transform is given by:
/ f(r - to) t) - iwr dr i=r- to; dř=dī
8 (fii,[
f(])w(t + tt) -ie(Et1)) dř / { ((0)(T (t -t-))) c iiw dē
(31) In other words, a translation by to in the time domain, corresponds to shift by to on the time axis of the short-time Fourier transform, and an additional phase modulation of eiEo similar that of the Fourier transform (Eq. 26).
-itb twto 8[S] - w) w) = e wwbo Lt (.((8(()((,
21

---

Published in Transactions on Machine Learning Research (12/2023)
100 58 ooe
no coe Ba- co
((t) con 2
a con
nm 3 c.e
(0
(F
M.nd
(F
Flng.
1 Kengy
Fing, Dec.
Fing Dev.
Scale equivariance of the short-time Fourier transform. Consider a function f(t)= coswit + cos composed of two frequencies 31 and wz=7, and window function w(t), with which the short-time Fourier transform is calculated. For relatively high frequencies (left column), the dot- dot-product of f and (f, w), is able to capture sufficient spectral information from f to correctly extract the frequencies w1, 42 from it. However, for dilated versions of the same signal f (right columns) obtained by reducing the frequency of the spectral components wi, of f; the capacity of the dol-product (f. to capture the spectral information eventually, is entirely lost. Cousequently, scale equivariance holds (approximately) for scales for which all of the spectral components the signal f lie within the range of the window w.
Figure 10:
imput gradually degrades and,
Input scaling. Let Lxg ((+5)) 80 E Rso, be scaled version of f. Its short-time Fourier transform is given by: (655t)) -iir dr i-s0 rT di di-sp
8 8lLn,(/1((,w)
f(])w(sot t) iwlood d(sot) s0 1 f(l)w(sot - -t) e ei(cow) dē t=sh sot
f(f)w(so(t (f e ((sow))E di w(ST) w(T)
Su
80 1())o(T- 100vo(T-5551 21 e (( di 2 80 811) ),ي)
(32)
In other words, dilation in the time domain produces a compression in the frequency domain analogous to the Fourier transform (Eq. 27). However, it is important to note that we rely on the approximate w(z)<<w(s)) to arrive to the final expression. Nevertheless, it is important to note that this approximate does not generally holds in practice This approximation implics that the window function w invariant to scaling, which holds only for increasing window sizes, i.e., when the short-term Fourier transform starts to approximate the (global) Fourier transform.
Simultancous input translation and scaling. Following the same derivation procedure, we show the behavior of the short-time Fourier transform to simultaneous translations and scaling. We have that:
( spe -iwta 8 [[]] lo), snw)
(33)
22

---

Published in Transactions on Machine Learning Research (12/2023)
Effect of imput transformations on the spectrogram. The spectrogram of function f E L2(R) is given by (8]S Input translations and dilations produce the following transformations:
[[+io = 1 [3f]I(([]) [ | sow)]
(34)
(35)
Equivariance and invariance properties of the short-time Fourier transform. The short-time Fourier transform is approzimately translation and scale equivariance in a manner similar to that of the Fourier Transform. In contrast to the Fourier transform, however, it decomposes input translations into a translation t- to and a phase shift e -wtto in the output (Eq. 31). This decomposition can be interpreted rough estimate t to signalizing the position in which the window w is localized, and fine grained localization within that window given by the phase shift e ritte indicating the relative position of the pattern within the window fu)(t)/w](7-
Equivariance to dilations is analogous to the Fourier transform up to the fact that time and frequency are now jointly described. However, since the window itself does not scale with the sampled frequency -as the case in wavelet transforms-, exact equivariance is not obtained. Note that equivariance to dilations is only approximate, and is restricted to the set of scales that can be detected with the width of the window used (see Fig. 10 for a visual explanation). Since this is not generally the case, the short-time Fourier transform is not scale equivariant.
The calculation of the spectrogram leaves the scale equivariance property of the transformation unaffected and is equivalent in a join manner to the scale equivariance property of the Fourier transform (Eq. 32). Differently however, Eq. 34 shows that translation equivariance is partially preserved and only information about the phase shift within the window is lost. This why the short-time Fourier transform said to carry positional information, i.e,, to be (approximalely) translation equivariant.
B.3
The Wavelet Transform
The wavelet transform of a signal f E (R) given by:
w[[[(,,)) = (J, vr.)) 1 f(T)
1/ (T-1) dr,
V8
and is equivalent to encoding f into time-fryyuency dictionary D JUER, (T)
Vo 1/* (E=T).
Input translation. Let L. f() - to) be a translated version of f. Its wavelet transform is given by:
((f1o1/1(,s)) / S(T ((T- lovo w- (T (s- 1)) dr [= i lo: dā=dr
1 10N/ - 10,1 = = (1, 8)
1)) di / 110/='''0s50 (t-))) dē
(36)
In other words, a translation of the input produces an equivalent translation in the wavelet domain.
Input scaling. Let (z, t) be scaled version of f. The corresponding wavelet transform is:
()
4s (T - t)) dr i=sū'n T; dē=sū dr SOOS (5 1)) d(sul) |=5 spt
f(O/=' 5000(s so(E so 1)) dē s0= v s0 so
V50 [(1) to 15" 1- (.55t>) )
-1 (ī ((-=551) .5) di
W
t, t, s) vso L'a, v (t),sss
(37)
23

---

Published in Transactions on Machine Learning Research (12/2023)
In other words, dilation so in the imput domain produces equivalent dilation in the wavelet domain both components (t, multiplied by factor v That is, the wavelet transform is translation equivariant.
Simultaneous input translation and scaling. Following the same procedure, we can show the behavior of the wavelet transform to simultaneous translations and dilations of the imput:
W(s(6 (o)]((,)) = v W[[[(5 lo), k5*²) v Ltg L. (1][(1
(38)
We observe that the Wavelet transform is the only time- frequecyyy transform that respects equivariance with equivalent group representations in the input and output domain.
Effect of input transformations on the scalogram. The scalogram of a function / E (IR is given by [w]/],, s) Input translations and dilations produce the following transformations on the scalogram:
|0 If s)|² = (ft, (w] s) <. s)I² s. [W[f]((,,s))
(39)
(40)
In other words the scalogram is exactly equivariant to both translations and dilations.
Equivariance and invariance properties of the wavelet transform. From Eq. 36, we can see that the wavelet transform is ezactly equivariant to tronslations and the group representation of the output space equals that of the input space. Furthermore, translation equivariance is preserved in the scalogram as well (Eq. 39). Similarly, scale equivariance is preserved on the wavelet transform up to a multiplicative factor (Eq. 37). However, the scalogram preserves both translation and dilation equivariance exactly (Eq. 40).
We emphasize that the group representation on the output space resembles that of the input space. This behavior leads to much more straightforward group representations than that exhibited by the Fourier transform and the short-lime Fourier transform. Additionally, exact scale equivariance is only obtained on the scalogram (Eq. 40), whilst for the wavelet transform it is retained up to multiplicative factor (Eq. 37). This behavior clucidates the fact that time-frequency transforms have been optimized for energy density representations rather than for the frequeccyy representations themselves.
C Experimental details
Whenever possible, we use existing code for the baselines of our wavelet networks as a starting point for the general infrastructure of our model. Specifically, we utilize the PyTorch implementation provided in https: //tptsu.///ttt...../// kyynguunlee/sampleeMN pappaaepeennn-eeecoehhhaaa as baseline for the US8K experiments and the MTAT experiments Lee et al. (2017), respectively. By doing so, we aim to preserve the reproducibility of the experiments in the baseline papers during our own experiments, as some important training factors are not specified in the baseline papers, e.g., the learning rale used in Dai et al. (2017). Unfortunately, Abdoli el al. (2019) do not provide code and we were forced to interpret some of the ambiguities in the paper, e.g., the pooling type utilized in the pooling layers and the loss metric used.
and https: com/
Any omitted parameters can safely be considered to be the default values in PyTorch 1.5.0. Our experi- ments are carried out in Nvidia TITAN RIX GPU,
C.1 UrbanSound8K
Wn-Nets. We use a sampling rate of 22.05kHz as opposed to the 8kHz used in Dai et al. (2017). An early study that indicated that some classes were indistinguishable for the human car at this sampling rate. We zero-pad signals shorter than 4 seconds Following the implementation of Dai et al. (2017), we utilize the Adam optimizer (Kingma & Ba, 2014) with 1r=1e-2 and weight_ deaett_dccy=1ee4,, and perform training on the official first 9 folds and test on the 10th fold. We noticed that reducing the learning rate from 1e-2 to 1e-3 increased the performance of our Wetets. The reported results of the W-Net variants are obtained with this learning rate.
that all input signals have constant length of 80200 samples.
We utilize batches of size 16 and perform training for 400 epochs. The learning rale is reduced by half after 20 epochs of no improvement in validation loss. The Wn-nets used are specified in Table 4. See Dai et al. (2017, Tab. 1) for comparison.
See http://ttwttttt......../../e//aa
ipynb.
24

---

Published in Transactions on Machine Learning Research (12/2023)
Table 4: Wn-networks. W3-Net (0.219M) denotes 3-layer network with 0.219M parameters. (79/4, 150, 3] denotes a group convolutional layer with nominal kernel size of 79 samples, 150 filters and 3 scales, with a stride of 4. Stride is omitted for stride (e.g., [3, 150, 3] has stride 1). Each convolutional layer uses batch normalization right after the convolution, after which ReLU is applied. Following the findings of Romero et (Romero et al., 2020, Appx. C) on the influence of stride the equivariance the network, we replace strided convolutions by normal convolutions, followed by spatial pooling. [...] k denotes stacked layers and double layers in brackets denote residual blocks defined in (Dai et al., 2017, Fig. 1b). In each of the levels of convolutional layers and residual blocks, the first convolution of the first block has scale 3 and the remaining convolutional layers at that level has scale 1.
W3 NET (0.219M)
W3-NET (0.558M)
WIT-NRT (1.806M)
W18-NE NET (3.759M)
W34-NeT (4.021M)
INPUT: 80200x TIME DOMAIN WAVEFORM
LIFTNNG LAYER (99 SCALES)
779/4, 150]
[79/4,74]
[79/4,51]
[79/4,57]
[79/4,45]
MAXPOOL: 4x1 (OUTPUT: 80200X 9X 1)
3,445
(3, 150,3]
31.
[3,51, x
[3,57. 4
3,45
MAXPOOL: 4x1 (OUTPUT: 80200x 7x 1)
3, 109
[3, 148,3]
3. 102, 3) x
[3, 114,3) x
3, 90
MANPOOL: 4x1 (OUTPUT: 80200x 5x
3. 180]
[3, 296,3]
3,204, 3| x
[3,228, 3) x
x
3, 180
MAXPOOL: 4x1 (OUTPUT: 80200x 3x 1) 3. 408. x 2
3.
[3. 3] x
3,0
GLOBAL AVERAGE POOLING (OUTPUT: 1 X N)
SOFTMAX [110] (OUTPET: N)
25

---

Published in Transactions on Machine Learning Research (12/2023)
Table 5: Wavelet network variant of the 50999-IDCN, (Abdoli et al., 2019). [31/ denotes a group convolutional layer with a nominal kernel size of 31 samples, 24 filters and 3 scales, with a stride of 2. FC: 196,48, 48] denotes a fully-connected layer with 96 imput channels and 48 output channels. Each convolutional layer uses batch normalization right after the convolution followed by ReLU. All fully comnected layers expect for the last one use dropout of 0.25 and ReLU. Following the findings of Romero et al. (2020, Appx. C) on the influence of stride in the equivariance of the network, we replace strided convolutions with normal convolutions followed by spatial pooling. We note that the input size of our network is (presumably) different from that in Abdoli et al. (2019). Consoquently, the last pooling layer utilizes a region of 5, in contrast to 4 as used in Abdoli et al. (2019). However, as it is not clear how the input dimension is reduced from 64000 to 50999 in Abdoli et al. (2019) and we stick to their original sampling procedure. We interpret their poling layers as max-pooling ones.
W-IDONN (0.519M)
INPUT: 64000X
LIFTING LAVER (9 SCALES)
163/2, 12]
MAXPOOL: 8x1
[31/2,24, 24,3)
MAXPOOL: 8x1
[15/2, 48.
[7/2, 96, 8]
[3/2, 408, 3)
MAXPOOL: 5x1
FLATTEN 196 x 6 1152
FC: [1152, 96]
FC: [96, 48
FC: [18, 10
SOFTMAX
W-IDCNN. Following Abdoli et al. (2019), we utilize a sampling rate of 16kHz during our experiments. We zero-pad signals shorter than 4 seconds so that all input signals have constant length of 64000 samples. Following the experimental description of the paper, we utilize the AdaDelta optimizer (Zeiler, 2012) with 1r=1.0 and perform training in a 10-fold cross validation setting as described in Sec. 6. We use batches of size 100 and perform training for 100 epochs. We utilize the 50999-1DCNN variant of Abdoli et al. (2019), as it is the variant that requires the less human engineering.
Unfortunately, we were not able to replicate the results reported in Abdoli et al. (2019) (83+13%% in our experiments. Our replication of Abdoli et al. (2019) lead to a 10-cross fold accuracy of 62+1.3%, which is 21 accuracy points less relative to the results reported. We experiment with our interpretation of the mean squared logarithmic error (MSLE) loss defined in (Abdoli et al., 2019, Eq. 4). However, we find that the conventional cross-entropy loss leads to better results. Consequently, all our reported results are based on training with this loss.5 5
The description of the Wavelet 50999-1DCNN Abdoli et al. (2019) is provided in Table 5 (see (Abdoli el al., 2019, Tab. 1) for comparison).
"The remaining architectures partition the input signal into overlapping windows after which the predictions of each windows is the only variant that truly
are summarized via a voting mechanism. Consequently, one could argue that the receives the raw waveform sigual. Nevertheless it is not clear from the paper how the input signal of 64000 samples is reduced to 50999 samples, which is the input dimeusion of the raw signal for this architecture type.
oThee MSLE loss in Abdoli et al. (2019, Eq. 4) is defined as N E log a,+1 : where Pt, a, and N are the predicted class, the actual class, and the number of samples respectively. Note, however, that obtaining the predicted class pi, iee. f(=a) where f(xz) C gd is the output of the network for classification problem with classes and imput x,, is non-differentiable function. Cousequently, it is not possible to train the network based on the formulation provided there.
P:+12
In order to train our model with this loss, we re-formulate the MSLE loss as 3 EN' Eo= log Poot12 where 10 (0 is a one-hot encoded version of the label aq. That is, we measure the difference between the one-hot encoded label and the output.
aj,o
26

---

Published in Transactions on Machine Learning Research (12/2023)
Table 6: -ew-network. [3/1, 90, 3] denotes a group convolutional layer with a nominal kernel size of 3 samples, filters and scales, with a stride of MP:3x1 denotes a max-pooling layer of size FC: [360, 50] denotes fully-connected layer with 360 input channels and 50 output channels. Each convolutional layer uses batch normalization after the convolution followed by ReLU. Dropout of 0.5 used after the 6 and 11 layer. Following the findings of Romero et al. (2020, Appx. C) on the influence of stride in the network equivariances, we replace strided convolutions by normal convolutions followed by spatial pooling.
W-3 NET (2.404M)
INPUT: 59049X
LIFTING LAVER (9 SCALES)
[3/3,90 90
3/1,90,3) MP:3X1
[3/1,90, 1]. MP:3x1
3/11, 180, 1). MP:3x1
[3/1, 3|, MP:3X1
13/1, 180, 1), MP:3x1
3/11, 180, 1), MP:3X1
3/1,, 180, 3,, MP:3X1
3/1, 180, 11, MP:3x1
3/1,, 360, 1/. MP:3X1
(601,360,
FC: [360, 50)
SIGMOI
C.2 Magna TagATune
W3 work.. For the experiments in the MTAT dataset, we utilize the PyTorch code provided by Lee et al. Lee et al. (2017). We use the data and tag preprocessing used in Lee et al. (2017). We utilize the SGD and nesterov=True. We use batches of size 23 and perform
optimizer with 1r=1e-2, geight_decay=ee training for 100 epochs. The learning rate is reduced by 5 after 3 epochs of improvement in the validation loss. Early stopping is used if the learning rate drops under 1e-7.
We were unable to replicate the per-class AUC results reported in Lee et al. (2017). Our experiments indicated a per-class AUC of 0.893 instead of the 0.905 reported in Lee et al. (2017). Details of the W39-Net used are given in Table (see (Lee et al., 2017, Tab. 1) for comparison).
27


Reviews:
[{"review_id": "KRVTAT2uP0", "reviewer": "Reviewer_WoBG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. Barring a few minor typos, the paper is well written, and the text and illustrations are generally easy to follow.\n\n   I particularly like the discussion in section 4 about the limitations of using convolutional neural networks over spectro-temporal data. Inspired by the success of CNNs on images, many prior works convert time series into 2D domain by computing spectrogram (or some variant) and pass them to a 2D CNN. The authors highlight why this is not the best strategy since unlike images, spectrograms of time series are often highly non-local.\n\n2. The noted improvement in performance obtained by imposing a ‘wavelet-like’ structure on the filters is interesting.\n\n**Weaknesses:**\n\nWhile I like the way this paper is written and the experimental results are also nice, I currently have one major concern that I would like the authors to clarify.\n\nIt is not clear to me what the key technical contribution is, in this paper. The authors mention that the main motivation is to build scale-translation equivariant networks for 1D data. However, is there any particular challenge in the 1D case that makes it non-trivial to directly extend the prior 2D scale-translation equivariant approaches to it? It appears to me that the proposed approach is quite similar to the 2D case of Sosnovik et al. [2]", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In 2D and 3D data domain, prior works have proposed to exploit equivariances to different symmetry groups to improve generalization performance, and parameter and data efficiency. However, in the 1D domain, this has not been explored much beyond the usual translation group exploited by conventional CNNs.\n\nThe paper proposes Wavelet networks to learn from time series data by exploiting equivariance to the scale-translation group. The convolutional layers for this group are constructed following the general formulation of group convolution proposed in [1] and seems somewhat similar to the prior scale-translation equivariance work [2] proposed for the 2D domain.\n\nTo avoid interpolation while lifting to different scales, convolutions are performed in the continuous domain by parameterizing the convolutional kernels using splines and the scale grid is suitably discretized to approximate the convolution integrals.\n\nThe paper also highlights the similarities between the scale-translation group convolution and the definition of wavelet transform. To promote a wavelet-like structure in the learnt convolutional kernels, they are regularized during training to have zero mean. The authors show that this interestingly provides a consistent improvement in performance.\n\nExperimental results show that the proposed wavelet networks improve performance over baseline models on different learning tasks. \n\n[1] Cohen, Taco, and Max Welling. \"Group equivariant convolutional networks.\" International conference on machine learning. PMLR, 2016.\n\n[2] Sosnovik, Ivan, Michał Szmaja, and Arnold Smeulders. \"Scale-Equivariant Steerable Networks.\" International Conference on Learning Representations. 2019.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please refer to my comments in the Strengths And Weaknesses section", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YzLr4qS9vG", "reviewer": "Reviewer_A2P8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "--------------- Strength:\n\nThe paper devotes many introductory efforts to distinguishing visual and auditory data, where the latter can be represented in the 2D time-frequency plane. This contradictory exposure highlights the difficulty of learning with auditory data and elucidates the insufficiency of standard CNNs.\n\nDerivations and definitions regarding groups and group convolutions are clearly stated with many graphical illustrations. This provides an easier understanding towards the complicated math expressions.\n\nExperimental results support the efficiency of the wavelet network, as it uses less parameters with good performance.\n\n--------------- Weakness:\n\nThere is no obvious weakness, but please refer to the requested changes for a detailed discussion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a wavelet network for scale-translation equivariant learning. The primary application area is time-series data, and the empirical verifications focus on sound signals. To my best knowledge, the group convolution and lifting convolution in the wavelet network are novel and serve the purpose of encoding scale-translation equivariance. The major contribution is to develop implementation details of the group convolution and lifting convolution, as well as testify their performance on real data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Section 4 might need a different highlighting style. Currently, the definition of Fourier transform and wavelet transform are highlighted in a grey box. However, from my understanding, the purpose of this section is to show the unique challenge and the insufficiency of existing ideas for handling auditory data. It could be better to articulate these problems in a more outstanding manner.\n\nSection 5 looks very complicated. It begins with definitions of group convolutions, then talks about the wavelet network architecture, and lastly provides implementation details. I feel like a better presentation structure can be used, such as starting with the network architecture and then introduce building blocks in the network. The implementation details may deserve a separated section.\n\nI am not an expert on auditory data processing. After reading the experiments, my questions are\n\n- it is good to compare with baseline methods, but are these baseline methods the state-of-the-art? \n\n- There is extensive study on the UrbanSound8K data. How does the performance of wavelet network compare to a broader range of methods beyond the CNN backbone? In particular, transformers are powerful network architectures for sequential data processing; are wavelet networks comparable to transformers?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "22MpdZEo2F", "reviewer": "Reviewer_47NA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper tackles an important problem by exploring equivariant neural networks w.r.t. to scale and translation for time series data. Given the prevalence of time-series data across various applications and the demonstrated benefits of equivariant models in enhancing sample efficiency and generalization, this research is a valuable contribution.\n- Instead of solely focusing on their methodologies, the authors offer an in-depth comparison with established techniques in the field, such as the wavelet transform and short-time Fourier transform. This comparative analysis sheds light on the reasons behind the observed performance enhancements. The discussions, including those on the differences between visual and spectro-temporal representations and between the short-term Fourier transform from the wavelet transform, are particularly interesting.\n- The paper also delves into the practical aspects of their research, emphasizing considerations like discretization and computational complexity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper identifies two important symmetries inherent to time-serious data: scale and translation. Based on this,  the paper proposes scale-translation equivariant neural networks (named Wavelet Networks) for time-series data, which is largely underexplored in the literature on geometric deep learning.\n- The authors discuss the relationship between the construction of scale-translation equivariant mappings in this paper with the well-known wavelet transform, thereby shedding some light on the merit of the proposed methods.\n- Their empirical results demonstrate that the proposed Wavelet Networks outperform baselines such as conventional CNNs on a variety of tasks and time-serious types, e.g. audio, environmental sounds, and electrical signals.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In fact, I think this work is ready for publication and I don't have any essential requested changes.", "broader_impact_concerns": "I have no concerns over the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fKYmaUBRZ4", "reviewer": "Reviewer_R4e9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Weakness**\n1. The proposed model is essentially a special case of G-CNN (with regular representations) for the scaling-translation group acting on 1d signals. More specifically, the first layer is the lifting group convolution, and all subsequent layers are the scaling-translation group convolutions. This model structure is not novel; it has been extensively explored and documented in prior literature. Consequently, it appears that the authors' contribution in this domain does not introduce any new concepts.\n\n2. References [1, 2] have proposed a remarkably similar architecture targeting this group, albeit for 2D signals. Regrettably, there is an apparent omission in the current manuscript, as these two prior works have not been acknowledged.\n\n3. The proposition of using continuous bases is neither groundbreaking nor original. Both references [1] and [2] have previously used this approach to mitigate interpolation artifacts.\n\n4. The numerical experiments are conducted to compare only with non-equivariant models. A more compelling evaluation would involve comparisons against, for instance, the model in [1] when adapted for 1D signals.\n\n**Strength**\n1. The paper is well-written. The regularization on \"wavelet structure\" is interesting. However, it is hard to understand why this regularization helps the overall generalization performance.\n\n[1] Sosnovik, Ivan, Michał Szmaja, and Arnold Smeulders. \"Scale-Equivariant Steerable Networks.\" International Conference on Learning Representations. 2019.\n[2] Zhu, Wei, et al. \"Scaling-translation-equivariant networks with decomposed convolutional filters.\" The Journal of Machine Learning Research 23.1 (2022): 2958-3002.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the wavelet network, an equivariant network tailored for 1D signals that ensures scaling and translation equivariance. The authors explained the connection between the (lifting) group convolution and the wavelet transform. Numerical experiments are presented to demonstrate the improved performance of the proposed model compared to conventional models, including CNNs.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please see the previous section", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on my comments above, I am recommending this paper is accepted as is. I would recommend the authors do a complete proof-read of the paper to prepare their camera-ready version (there are still some minor typos, see e.g. sentence prior to Eq.17).
Decision: Accept (poster)