Implicit Neural Representations and the Algebra of Complex Wavelets
Authors: T Mitchell Roddenberry
Vishwanath Saragadam
Maarten V. de Hoop
Richard Baraniuk
Venue: ICLR 2024 poster (2024)
Keywords: implicit neural representations
algebra
multilayer perceptrons
wavelet
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We discuss the algebra of implicit neural representations from an applied harmonic analysis perspective

Abstract: Implicit neural representations (INRs) have arisen as useful methods for representing signals on Euclidean domains. By parameterizing an image as a multilayer perceptron (MLP) on Euclidean space, INRs effectively couple spatial and spectral features of the represented signal in a way that is not obvious in the usual discrete representation. Although INRs using sinusoidal activation functions have been studied in terms of Fourier theory, recent works have shown the advantage of using wavelets instead of sinusoids as activation functions, due to their ability to simultaneously localize in both frequency and space. In this work, we approach such INRs and demonstrate how they resolve high-frequency features of signals from coarse approximations performed in the first layer of the MLP. This leads to multiple prescriptions for the design of INR architectures, including the use of progressive wavelets, decoupling of low and high-pass approximations, and initialization schemes based on the singularities of the target signal.

Published as a conference paper at ICLR 2024
IMPLICIT NEURAL REPRESENTATIONS AND THE ALGEBRA OF COMPLEX WAVELETS
T. Mitchell Roddenberry, Vishwanath Saragadam; Maarten de Hoop, Richard G. Baraniuk
Rice University
Houston, TX, USA
(mitch, mvd2, ri chb) @rice. edu, sut.rraaat..araagaammmueuuuuu .ddu
ABSTRACT
Implicit neural representations (INRs) have arisen as useful methods for repre- senting signals on Euclidean domains. By parameterizing an image as multi- layer perceptron (MLP) on Euclidean space, INRs effectively couple spatial and spectral features of the represented signal in way that is not obvious in the usual discrete representation. Although INRs using sinusoidal activation functions have been studied in terms of Fourier theory, recent works have shown the advantage of using wavelets instead of sinusoids as activation functions, due to their ability simultaneously localize in both frequency and space. In this work. approach such INRs and demonstrate how they resolve high-frequency features of signals from coarse approximations performed
the first layer of the MLP. This leads to multiple prescriptions for the design of INR architectures, including the use of progressive wavelets, decoupling of low and high-pass approximations, and ini- tialization schemes based on the singularities of the target signal.
INTRODUCTION
Implicit neural representations (INRs) are powerful set of neural architectures for representing and processing signals on low-dimensional spaces. By learning a continuous interpolant of a set of sampled points, INRs have enabled and advanced state-of-the-an methods in signal processing (Xu et al., 2022) and computer vision (Mildenhall et al., 2020).
Typical INRs specially designed multilayer perceptrons (MLPs), where the activation functions are chosen in such a way to yield a desirable signal representation. Although INRs often can be easily understood the first layer due the simplicity of plotting the function associated each neuron based on its weights and biases, the behavior of the network in the second layer and beyond more opaque, apart from some theoretical developments the particular case of a sinusoidal first layer (Yüce et al., 2022).
This work develops broader theoretical understanding of INR architectures with wider class of activation functions, followed by practical prescriptions rooted in time-frequency analysis. In particular, we
Characterize the function class of INRs in terms of Fourier convolutions of the neurons in the first layer (Lemma 1)
2. Demonstrate how INRs that use complex wavelet functions preserve useful properties of the wavelet, even after the application of the nonlinearities (Corollary 4)
3. Suggest split architecture approximating signals that decouples the smooth and nons- mooth parts into linear and nonlinear INRs, respectively (Section 4.3)
4. Leverage connections with wavelet theory to propose efficient initialization schemes for wavelet INRs based on the wavelet modulus maxima for capturing singularities the target function (Section 5).
Now affiliated with UC Riverside.

---

Published as a conference paper at ICLR 2024
Following a brief survey of INR methods, the class of architectures we study is defined in Section 2. The main result bounding the function class represented by these architectures is stated in Section 3, which is then related to the algebra of complex wavelets in Section 4. The use of the wavelet modulus maxima for initialization of wavelet INRs is described and demonstrated in Section 5, before concluding in Section 6.
2 IMPLICIT NEURAL REPRESENTATIONS
Wavelets as activation functions in MLPs have been shown yield good function approxima- tors (Zhang& & Benveniste, 1992; Marar et al., 1996). These works have leveraged the sparse repre- sentation functions by wavelet dictionaries in order to construct simple neural architectures and training algorithms for effective signal representation. Indeed, an approximation of a signal by finite linear combination of ridgelets (Candès, 1998) can viewed as one such MLP using wavelet activation functions. Additionally, wavelets have been used to study the expressivity of deep neural networks and their approximation capacity for functions on manifolds (Shaham et al., 2018), for instance.
Recently, sinusoidal activation functions in the first layer (Tancik et al., 2020) and beyond (Sitzmann et al., 2020; Fathony al., 2020) have been shown to yield good function approximators, coupled with a harmonic analysis-type bound on the function class represented by these networks (Yüce al., 2022). Similar to the Fourier embedding of the coordinate space that is done by methods such as SIREN (Sitzmann et al., 2020), eigenvectors of graph operators have been used to define INRs for signals more general spaces (Grattarola & Vandergheynst, 2022). Other methods have used activation functions that, unlike sinusoids, are localized in space, such as gaussians (Ramasinghe & Lucey, 2021) or Gabor wavelets (Saragadam et al., 2023).
Following the formulation of (Yúce et al., 2022), we define an INR to be map fo R + C defined in terms of function v Rdd C, followed by an MLP with analytic activation functions p(() C for layers f I,...,I.
z (r) 2 ( r b(o)
2 (r) = p¹ (W (f-1) (x) (x) bb))
fo(r) = w (
(r) (r)+bbu +b())
where O denotes the set of parameters dictating the tensor W E Bffixxxx matrices ( Fe 1xF: and (0) E R Fixdd xd and vectors (0) E C for l = I,...,,, with fixed integers F satisfying FL+1 = 1. The function sb Fdd + C is understood to act on W + (() row-wise, ie.. as a map & R + C We will henceforth refer as the template function of the INR.
Owing to the use of Gabor wavelets by Saragadam et al. (2023), we will refer to functions of the form (1) as WIRE INRs, although (1) also captures architectures that do not use wavelets, such as SIREN (Sitzmann et al., 2020).
3 EXPRESSIVITY OF INRS
For the application of INRs to practical problems, it is important to understand the function class that an INR architecture can represent. We will demonstrate how the function parameterized by an INR can be understood via time-frequee frequeeyy analysis, ultimately motivating the of wavelets as template functions.
Noting that polynomials of sinusoids generate linear combinations of integer harmonics of said sinu- soids, Yüice et al. (2022) bounded the expressivity of SIREN (Sitzmann et al., 2020) and related ar- chitectures (Fathony et al., 2020). These results essentially followed from identities relating products of trigonometric functions. For template functions that are not sinusoids, such as wavelets (Sara- gadam et al., 2023), these identities do not hold. The following result offers bound the class of functions represented by an INR.
That is, entire C.
2

---

Published as a conference paper at ICLR 2024
Complex Simusoid
Gaussian
Gabor Wavelet
Complex Meyer Wavelet
1.0
0.4
1.0
Scaling
0.3
0.3
2
3
0.1
0.1
0.0
2.0
4.0
6.0
5.0
2200
0.0
2.0
99.0
2.0
4.0
6.0
8.0
0.00
5.0
10.0
15.0
20.0
Figure 1: Fourier transforms of template functions vé and their powers vhnh
Lemma 1. Let So TRd - C be WIRE INR. Assume that each of the activation functions p()) is polynomial of degree most K, and that the Fourier transform of the template function to exists." Let ( r = [ II...., W W | for W1,... WF, E gddd each having full rank, and also let ((O) bp. for bi...., by, E Rd. For 2 O, denote by (FF1, k) the set of ordered [M...... mF, such that Σ me =k.
Fi Fi-tuples of nonnegative integers
Let point ro E Rd be given. Then, there exists open neighborhood U 3 ro such that for all Co (0)
Kh-1
Fi * t=1
Φ. fo(())
Σ R=O m))(F).))
E
Bm (.
(5),
(2)
for coefficients Bm E C independent of the choice of U, where (.) denotes m-fold con- volution' of the argument with itself with respect to &, and C (U) denotes the set of all infinitely differentiable functions with compact support contained in U. Furthermore, the coefficients 3m are only nonzero when each [1...., Fi] such that m, # also satisfies W iro W b, supp(v).
The proof, a simple application of the convolution theorem, is left Appendix A. Lemma 1 illus- trates two things. First, the output of an INR has a Fourier transform determined by convolutions of the Fourier transforms of the atoms in the first layer with themselves. serving to generate "integer harmonics" of the initial atoms determined by scaled, shifted copies of the template function vb. No- tably, this recovers (Yüce al., 2022, Theorem 1). Second, the support of these scaled and shifted atoms is preserved, so that the output given coordinate is dependent only upon the atoms in the first layer whose support contains r.
Remark The assumptions behind Lemma I can be relaxed capture broader class of architec- tures. By imposing continuity conditions on the template function v/, the activation functions can be reasonably extended to analytic functions. These extensions are discussed in Appendix B.
4 THE ALGEBRA OF COMPLEX WAVELETS
Of the INR architectures surveyed in Section 2, the only one to use a complex wavelet template function is WIRE (Saragadam al., 2023), where Gabor wavelet is used. Gabor wavelets essentially band-pass filters and are necessarily complex-valued due to their lack of conjugate sym- metry in the Fourier domain. We now consider the advantages using complex wavelets, more precisely progressive wavelets, as template functions for INRs by examining their structure as an algebra of functions.
EEen if only sense of tempered distributions.
'0-fold convolution is defined by convention to yield the Dirac delta. We also use the symbol * the convolution of several functions, in this indexed by 1,..., Fi.
to denote

---

Published as a conference paper at ICLR 2024
4.1 PROGRESSIVE TEMPLATE FUNCTIONS
For the sake of discussion, suppose that d = 1. so that the INR represents a ID function. The template function vh I + C is said be progressive' if it has no negative frequency components, i.e, for 5 < O. we have (5) = O (Mallat, 1999). No nonzero real-valued function is progressive. It is obvious that progressive functions remain progressive under scalar multiplication, shifts, and positive scaling. That is, for arbitrary 8 > O, u E E, E C. if (()) is progressive, then the function z
:= (((r s) is also progressive. Moreover, progressive functions are closed under multiplication, so that if 1/1 and 1/ are progressive, then (/)(2) == 4/2 (x))b((2)) is also progressive, i.e., progressive functions constitute
algebra over C.
Example (Complex Sinusoid), For any w > O, the complex sinusoid (); - progressive function, as its Fourier transform is Dirac delta centered at w. As pictured in Fig. 1 (a), the exponents 2² (+:w) are themselves complex sinusoids, where v (x:w) = (x;
Example (Gaussian). The gaussian function, defined for some o 0 as ();;)) = exp(-2² (2or)), is not progressive function, as its Fourier transform is symmetric and cen- tered about zero. Moreover, as pictured in Fig. 1 (b), the exponents are also gaussian functions (); vb(x;; alz;;///), which also have Fourier transform centered at zero. Unlike the complex sinusoid, the powers of the gaussian are all low-pass, but with increasingly wide passhand.
Example 3 (Gabor Wavelet). For any w. > 0, the Gabor wavelet defined as (z o) exp(-2² /2ar²) i2Tww) is not a progressive function, as its Fourier transform is gaussian cen- tered at w with standard deviation 1/0. However, the Fourier transform of the exponents 1/th for integers n > 0 are gaussians centered at niw with standard deviation vn/o, as pictured in Fig. (c). So, as n grows sufficiently large, the effective support of tnth will be contained in the positive reals, so that the Gabor wavelet can be considered as a progressive function for the purposes of studying INRs.
A progressive function on E has Fourier support contained in the nonnegative real numbers. Of course, there is not an obvious notion of nonnegativity that generalizes to Ed for d > 1. Noting that the nonnegative reals form a convex conic subset of R, we define the notion of a progressive function with respect to some conic subset of Rdd
Definition 3. Let T C Fed be a convex conic set, i.e., for all 71,72 E T and 01,02 02 > 0, we have
that a171 02722 E F.6 A function vi Fd C is said to be I'-progressive if supp ()) C r. The function ab is said to be locally T-proresssive at rg E IRdd if there exists some --progressive function Vrr, : Rd C so that for all smooth functions Φ C. (RR) with support in sufficiently small neighborhood of r, we have
+ δ Vir..
Curvelets (Candès & Donoho, 2004), for instance, are typically defined in a way to make them r- progressive for some conic set I that indicates the oscillatory direction of curvelet atom. Observe that if I is a conic set, then for any matrix W. the set TT is also conic. Thus, for a function v) that is I' ir-progressive, the function ti( (Wx) is W WTIL I'pprogressive.
Observe further that for two -rrogrrsssive functions 1h,, ths, their product is also I-progressive. The closure of progressive functions under multiplication implies that an analytic function applied point- wise to progressive function is progressive. For INRs as defined in (1), this yields the following corollary to Lemma 1.
Corollary 4. Let P C Tedd be conic, and let 1/o Rdd - C be given, with Fourier support denoted
To == supp (z). Let w = ir,,.... W rF for W WI...., WF, E pdxd each having full rank. Assume that for each t = = Fi, we have W.T. CT. Then, the WIRE INR fe Rdd C defined by (1) is a I T-progressive function.
Moreover, we fix some ro IRd, and if the assumption W.T.o To holds that Wer + b, E supp(w)) then fo is locally T-progressive at ro-
holds for the indices t such
'More commonly known as analytic signal, use this terminology (following Grossmann et al. (1990)) avoid confusion with the analytic activation functions used in the INR.
"This is a simple consequence of the convolution theorem. We henceforth refer such sets as simply "conic."
4

---

Published as a conference paper at ICLR 2024
a/2
(a)
(b)
(e)
F
9
T2
/
m/2
Figure 2: (a) A conic set 11 C. IR? and a weakly conic set L2 C I² (both truncated for illustration purposes). (b) Modulus of function given by the sum of four template atoms. (c) Fourier transform of fo(r) = ((g())), where to the respectively highlighted parts of the function g. Effective Fourier supports of the template atoms constituting g are enclosed by rectangles, and approximate centers of Frequency support for each atom and product of atoms marked by colored circles.
+ 23 The blue and orange cones correspond
The proof is left to Appendix C; essentially, the property of --progressive functions constitut- ing an algebra over C combined with the polynomial structure of the INR is shown to preserve the -pprogressvve property. Thus, any advantages/immiations of approximating functions using T- progressive template functions are maintained.
Remark 5. One may notice that a --progressive function will always incur a large error when ap- proximating real-valued function, real-valued functions have conjugate symmetric Fourier trans- forms (apart from the case r ) For fitting real-valued functions, is effective to simply fit the real part of the INR output to the function, as taking the real part of function symmetrizes it in the Fourier domain. In the particular case = 1, fitting the real part progressive INR to function is equivalent to fitting the INR to that function's 's Hilbert transform.
4.2 BAND-PASS PROGRESSIVE WAVELETS
Corollary 4 holds for conic sets I, but is also true for larger class of sets. If some set I C Rd is conic, it by definition closed under all sums with nonnegative coefficients. Alternatively, consider the following weaker property:
Definition 6. Let C dd T is said to be weakly conic all 72 and a1, 02 2 1, we have that 71 + 0272 E r, and that 0 E т. A function vh Rd + C is said to be T-progressive if (1 Т. The function z is said be locally T-progressive ro E Rd if there exists some I'-progressive function ur IRd + C so that for all smooth functions & E (RR( with support in sufficiently small? neighborhood of r, we have
. Φx 2'ro
The notion of a weakly conic set is illustrated in Fig. 2 (a). Just as in the case of progressive functions for a conic set, the of I --progressive functions for a weakly conic set I C Idd constitutes an algebra over C. One can check, then, that Corollary 4 holds for weakly conic sets as well. Putting this into context, consider template function té such that & vanishes in some neighborhood of the origin. Assume furthermore that supp(1)) is contained in some weakly conic set T.
Example 4 (Complex Meyer Wavelet). The complex Meyer wavelet is most easily defined in terms of its Fourier transform. Define
sin 3
T/2) T/2)
E E /2m/33 4m /3]
((E)
cos
{ E /47/3, 8T/3] otherwise.
0
AAaai,, not necessarily compact.
5

---

Published as a conference paper at ICLR 2024
Target (a)
Split, Complex, L (b)
scaling network, Real (c)
No scaling network. Complex (d)
Split, Real (e)
Split, Complex
-15
0.5
0.5 1.5
1.5 -1
0.5 0.5
Figure 3: (a) Target signal, sampled at n = 512 uniformly spaced points in the interval [-2, 2 2]. Wavelet modulus maxima are marked in red. (b) Split complex wavelet INR with hidden layers (MSE = 0.0016). (c) "Real" wavelet INR with no scaling network (MSE = 0.0096). (d) Complex wavelet INR with no scaling network (MSE 0.0606). (e) Split "real" wavelet INR (MSE 0.0061). (f) Split complex wavelet INR (MSE 0. 0.0011).
The complex Meyer wavelet and its exponents are pictured in Fig. (d). Observe that these functions are not only progressive, but are also I'pprogeessive for the weakly conic set = [2x/3, oo). The Meyer scaling function, pictured by the dashed line in Fig. (d), has Fourier support that only overlaps that of the complex Meyer wavelet, but none of its powers.
Applying this extension of Corollary 4, we that if the atoms in the first layer of INR using such a function v have vanishing Fourier transform some neighborhood of the origin, then the output of the INR has Fourier support that also vanishes in that neighborhood. We illustrate this in I22 using a template function e : 2 + C where vi is the tensor product of a gaussian and complex Meyer wavelet. Using this template function, we construct an INR with Fi 4 in the first layer, and single polynomial activation function. The modulus of the sum of the template functions before applying the activation function is shown in Fig. (b). We then plot the modulus of the Fourier transform of so in Fig. 2 (c). First, observe that since the effective supports of the transformed template functions are supported by two disjoint sets, the Fourier transform of So can be separated into two cones, each corresponding to region in R2, Second, since the complex Meyer wavelet vanishes in a neighborhood of the origin, these cones are weakly conic, so that the Fourier transform of fe vanishes in neighborhood of the origin as well, by Corollary 4 applied to weakly conic sets.
Remark The weakly conic sets pictured in Fig. 2 (c) only approximation bounds of the true Fourier support of the constituent atoms. We see that Corollary 4 still holds in an approximate sense, the bulk of the Fourier support the atoms is contained in each of the pictured cones.
4.3 A SPLIT ARCHITECTURE FOR INRs
Based on this property of INRs preserving the band-pass properties progressive template func- tions, it is well-motivated approximate functions using sum of two INRs: one to handle the low-pass components using a scaling function, and the other to handle the high-pass components using wavelet. We illustrate this in Fig. 3, where fit classic test signal R (Donoho & Johnstone, 1994), pictured in Fig. (a).
The first INR uses a gaussian template function (xx)) exp(-(=x)*/6)) with L 1, and the constraint that the weights (0)) are all equal to one, i.e., the template atoms only vary in their

---

Published as a conference paper at ICLR 2024
abscissa. Such network is essentially single-layer perceptron (Zhang & Benveniste, 1992) for representing smooth signals. We refer to this network as the "scaling network."
The second INR uses Gabor template function (x()) exp(-(mz)) /6) exp(-i2x.x) with L = 3, where we initialize the weights in the first layer to be positive, thus satisfying the condition of W. C r in Corollary 4 for r R Although db is not progressive, its Fourier transform has fast decay, so we consider it to be essentially progressive, and thus approximately fulfilling the conditions of Corollary 4. We refer this network as the "wavelet network," it is the WIRE architecture (Saragadam et al., 2023) for signals on R.
Denoting the scaling and wavelet networks by Sa.s: Jo, R C, respectively, we consider their sum fe = fe.. + fe w as a model, and approximate the target signal by taking fe R + R.
The reason for modeling a signal as the sum of a linear scaling INR and nonlinear INR with a progressive wavelet is apparent in Fig. 1 (d), where the scaling function and powers of complex Meyer wavelet are pictured. Observe that the portions of the Fourier spectrum covered by the gaus- sian scaling function and the high powers of the Gabor wavelet (as in an INR, by Lemma 1) are essentially disjoint. To approximate the low-frequency components of a signal using an INR with progressive wavelet would require large dilations of the atoms in the first layer, in order to force the center frequency of the wavelet towards zero. Moreover, if the progressive wavelet has Fourier transform that vanishes in neighborhood of the origin, such dilations will never properly cover small neighborhood of the origin. This does not arise when using real-valued template functions, as powers of such functions generate wwwfrrquency components. However, this phenomena is not al- parts of a signal become highly correlated
ways desirable, as the fow-frequency and in this regime, property that is not necessarily true for natural signals.
The idea behind the "split" architecture, then, is to use a simple network to approximate the smooth parts of the target signal, and then a more complicated nonlinear network approximate the nons- mooth parts of the signal.
We fit an array of INR architectures the target signal: split INR architecture with complex Gabor wavelets but no hidden layers (L = 1) in the MLP, one with no scaling network and real wavelets, one with no scaling network and complex wavelets, one with a scaling network and real wavelets, and finally our proposed split architecture with complex Gabor wavelets and hidden layers in the MLP. All wavelet networks apart from the first one have two hidden layers (L 3). In the architectures that use real wavelets, we use a gaussian multiplied by sinusoid, rather than a complex exponential as in the complex Gabor wavelet.
The results of training these architectures are respectively shown in Fig. 3 (b-f). Hyperparameters for each architecture are described in Appendix E. We observe slightly better performance, measured in mean squared error (MSE), from the proposed architecture than the split network with no hidden layers, which in turn outperforms the split architecture using real wavelets. This illustrates both the advantages of using complex wavelets over real wavelets, due to their ability to decouple low and high-frean gregufryy parts of a signal, as well as the advantage of the hidden layers that couple wavelet coefficients across scales, as stated by Lemma 1. Indeed, the nonlinear activation functions generate "wavelets" from the template function atoms, rather than needing to form a long list of wavelet atoms and their abscissa in a way that does not capture how wavelet coefficients near singularities are correlated across scales.
The two INR architectures without scaling networks fared the worst in this experiment; however, we note that the INR using real wavelets outperformed the one using complex wavelets. This is because powers of real-valued wavelets can generate uwwfrrquency signal content.
To see the role of the nonlincarities in the wavelet INR, we freeze the weights and biases in the first layer of the split complex wavelet INR, and take an optimal linear combination of the resulting template atoms to fit the signal, thus yielding an INR with no hidden layers (Zhang & Benveniste, 1992). We compare the Fourier transforms of the original wavelet network to this "linearized" one in Fig. 4 (b-c), where we see that the nonlinear wavelet network is able to resolve much more high- frequency signal content than the linear one. This reflects how the activation functions resolve high- frequency features from low-free www-fregencyy approximations, as illustrated initially in Fig. 2. Moreover, the fact that both wavelet networks have the bulk of their Fourier support only on positive frequencies
7

---

Published as a conference paper at ICLR 2024
Re()(x))
Scaling Wivelet
(a)
10a
Wiwelet
Scaling
(b)
10
10-4
10a
Wowelet
(d)
10'
10"
1.5
0.5
0.5
1.5
-//2
0
n/2
Figure 4: (a) Split complex wavelet INR, separated into scaling and wavelet networks. (b) Fourier transform of scaling and wavelet networks. (c) Linearized complex wavelet INR. (d) Fourier trans- form of lincarized wavelet network.
illustrates the how the algebraic closure of progressive wavelets under multiplication applies to INR architectures, as in Corollary
5 RESOLUTION OF SINGULARITIES
useful model for studying sparse representations of images the cartoon-like image, which is smooth function IR² apart from singularities along twice-differentiable curve (Candès & Donoho, 2004; Wakin et al., 2006).
The smooth part of image be handled by the scaling function associated wavelet trans- form, while the singular parts are best captured by the wavelet function. In the context of the proposed split INR architecture, the scaling INR yields a smooth approximation the signal, and the wavelet INR resolves the remaining singularities. Inspired by this, we now consider how the wavelet INR can be initialized with the resolution of isolated singularities in mind.
5.1 INITIALIZATINN WITH THE WAVELET MODULUS MAXIMA
As demonstrated by Lemma 1, the function s in the first layer of INR determines the expres- sivity of the network. Many such networks satisfy a universal approximation property (Zhang & Benveniste, 1992), but their value in practice comes from their implicit bias (Yüce al., 2022; Saragadam et al., 2023) in representing particular class of functions. For instance, using a wavelet in the first layer results in sharp resolution of edges with spatially compact error (Saragadam al., 2023). In the remainder of this section, we demonstrate how an understanding of singular points in terms of the wavelet transform can used to bolster INR architectures and initialization schemes.
Roughly speaking, isolated singularities in a signal are points where the signal is nonsmooth, but smooth in a punctured neighborhood around that point. Such singularities generate "wavelet modulus maxima" (WMM) curves in the continuous wavelet transform (Mallat, 1999), which have slow decay in the Fourier domain. With Lemma mind, we see that INRs can use a collection of low- fow-frequency template atoms and generate collection of coupled high-frequency atoms, while also preserving the spatial locality the template atoms.
The combination of these insights suggests method for the initialization of INRs. In particular, for a given number of template atoms F1 in an INR, the network weights ( and abscissa ()) should be initialized a way that facilitates effective training of the INR via optimization methods. We empirically demonstrate the difference in performance for INRs initialized at random and INRs initialized accordance with the singularities in the target signal. Once again, we fit the sum of scaling network and wavelet network to the target signal in Fig. 3. In Fig. 5 (a), plot the mean squared error (MSE) for this setting after 1000 training steps for both randomly initialized and strategically initialized INRs, for F Km, where K E (1,3,5, 7) and is the number of
8

---

Published as a conference paper at ICLR 2024
(a) 10-2
- Random WMM
Setting
PSNR (dB)
Gabor, Random Init. (fit)
30.081
MSE
Gabor, WMM Init. (fit)
30.835
(b) Gaussian (fit)
26.950
10-4
Real Gabor (fit)
30.826
Sine (fit)
18.001
Gabor, Random Init. (denoise)
29.236
Gabor, WMM Init. (denoise) 29.385
K
Figure 5: (a) Mean and standard deviation of the MSE over 10 trials the ID test signal. (b) PSNR of different architectures and initialization schemes on the Kodak dataset (kod, 1999), for image fitting and denoising tasks. In denoising task, noisy images had average PSNR of 17.35 dB.
WMM points as determined by an estimate of the continuous wavelet transform of the target signal. The randomly initialized INRs have abscissa distributed uniformly at random over the domain of the signal. The strategically initialized INRs place K template atoms at each WMM point (so, a deterministic set of abscissa points). Both initialization schemes randomly distribute the scale weights uniformly in the interval [1, K]. We observe that for all K, the MSE of the strategically initialized INR is approximately an order magnitude less than that the randomly initialized INR.
When d = 2, e.g., images, the WMM can be approximated by the gradients of the target signal to obtain initial set of weights and biases for the wavelet INR. We evaluate this empirically on the Kodak Lossless True Color Image Suite (kod, 1999). We approximate the target images using the proposed split INR architecture. For the WMM-based initialization, we apply a Canny edge detector (Canny, 1986) to encode the positions and directions of the edges. Further details can be found in Appendix D. The architectures used are described in Appendix E.2.
We record the results of this experiment for a variety of template functions and initialization schemes in Fig. (b). Experiments are done for two tasks: image representation, where the network is fit to the ground truth image, and denoising, where the network is fit to noisy image and the error relative the ground truth is measured. The latter experiment demonstrates how the implicit bias of INRs is such that they fit natural images better than they fit noise. We observe that over the whole dataset, initializing the network using the WMM yields higher PSNR for both tasks, demonstrating the utility of smart initialization of INRs. Results are aggregated over the whole dataset; see Appendix F for results on the individual images.
CONCLUSIONS
We have offered a time-frequency analysis of INRs that leverages polynomial approximations of the nonlinear behavior of MLPs beyond the first layer. By noting that progressive functions form an algebra over the complex numbers, we demonstrated that this analysis yields insights into the behavior of INRs using complex wavelets, such as WIRE (Saragadam et al., 2023). This leads to split architecture for approximating signals, which decouples the low-pass and high-pass parts of signal using two INRs, roughly corresponding to the scaling and wavelet functions of a wavelet transform. Furthermore, the connection with the theory of wavelets yields natural initialization scheme for the weights of an INR based on the singularities of signal.
INR architectures built using wavelet activation functions offer useful advantages for function ap- proximation that balance locality in space and frequency. The structure of complex wavelets as an algebra of functions with conic Fourier support, combined with the application of INRs for interpo- lating sampled functions, suggests connection with microlocal and semiclassical analysis (Monard & Stefanov, 2023). This could potentially be understood and improved by incorporating ideas from shearlet and curvelet systems (Labate al., 2005; Candès & Donoho, 2004). We also foresee the decoupling of the smooth and singular parts of a signal by the split INR architecture having useful properties for solving inverse problems.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by NSF grants CCF-1911094 I1S-1838177, and IIS-1730574; ONR grants
N00014-18-1-277, N00014-20-1-2553 and MURI N00014-20-1-2787; AFOSR grant FA9550- 22-1-0060; and a Vannevar Bush Faculty Fellowship. ONR grant N00014-18-1-2044 Maarten de Hoop gratefully acknowledges support from the Department of Energy under grant DE-SC0020345, the Simons Foundation under the MATH+X program, and the corporate members of the Geo- Mathematical Imaging Group Rice University.
REFERENCES
Kodak lossless true color image suite. 2022-11-09
cttp:apppiiss/ooaa//////akk 1999. Accessed:
Emmanuel Jean Candès. Ridgelets: theory and applications. PhD thesis, Stanford University, 1998.
Emmanuel Jean Candès and David Leigh Donoho. New tight frames of curvelets and optimal rep- resentations objects with piecewise C2 singularities. Communications on Pure and Applied Mathematics, 57(2):219-266, 2004.
John Canny. A computational approach edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, (6):679-698, 1986.
David Leigh Donoho and lain M Johnstone. Ideal spatial adaptation by wavelet shrinkage. Biometrika, 8133):22-4555 1994.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. In International Conference on Learning Representations, 2020.
Daniele Grattarola and Pierre Vandergheynst. Generalised implicit neural representations. Advances in Neural Information Processing Systems, 35:30446-30458, 2022.
Alexandre Grossmann, Richard Kronland-Martinn and J Morlet. Reading and understanding con- tinuous wavelet transforms. In Wavelets: Time- Time-Freques Frequenyy Methods and Phase Space Proceedings of the International Conference, Marseille, France, December 14-18, 1987, pp. 2-20. Springer, 1990.
Demetrio Labate, Wang-Q Lim, Gitta Kutyniok, and Guido Weiss. Sparse multidimensional repre- sentation using shearlets. In Wavelets XI, pp. 254 262. SPIE, 2005.
Stéphane Mallat. A wavelet tour of signal processing. Elsevier, 1999.
Joao Fernando Marar, Edson CB Carvalho Filho, and Germano C Vasconcelos. Function approxi- mation by polynomial wavelets generated from powers of sigmoids. In Wavelet Applications III, volume 2762. pp. 365-374. SPIE, 1996.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In IEEE Euro- pean Conference on Computer Vision, 2020.
François Monard and Plamen Stefanov. Sampling the X-ray transform on simple surfaces. SIAM
Journal on Mathematical Analysis, 55(3):1707-1735 2023.
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for activations in coordinate-MLP In IEEE European Conference on Computer Vision, 2021.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018. URL. ht..tttps///// /ttp:///ppeeeiieeeeeeeeee htt://orummicc iiniiiiuddddurrru
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. WIRE: Wavelet implicit neural representations. In IEEE/CVF Computer Vision and Pattern Recognition Conference, pp. 18507-18516, 2023.
10

---

Published as a conference paper at ICLR 2024
Uri Shaham, Alexander Cloninger, and Ronald R Coifman. Provable approximation properties for deep neural networks. Applied and Computational Harmonic Analysis, 44(3):537-557, 2018.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 2020.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. Advances in Neural Informa- fion Processing Systems, 2020.
Michael В Wakin, Justin K Romberg, Hyeokho Choi, and Richard G Baraniuk. Wavelet-domasd approximation and compression of piecewise smooth images. IEEE Transactions on Signal Pro- cessing, 15(5); 1071-10777 2006.
Dejia Xu, Peihao Wang, Yifan Jiang, Zhiwen Fan, and Zhangyang Wang. Signal processing for implicit neural representations. Advances in Neural Information Processing Systems, 35, 2022.
Gizem Yüce, Guillermo Ortiz-Jiménez, Beril Besbinar, and Pascal Frossard. A structured dictio- nary perspective on implicit neural representations. In /EEE/CVF Computer Vision and Pattern Recognition Conference, 2022.
Qinghua Zhang and Albert Benveniste. Wavelet networks. IEEE Transactions Neural Networks, 3(6))889-899) -989 1992.
A PROOF OF LEMMA
Let Po be given, and let S [1.....Fi] Fi] be the set of indices such that Waro + b, E supp(1)) vt). Then, there exists an open neighborhood U such that for all r E U, (WWr 1rr+ b,) 0 for any $ I. Thus, for any r E U, the assumptions on the activation functions imply that fo(r) is expressible as complex multivariate polynomial of (*(W b.)) with degree at most K L-1 i.e.,
KL
fo(r) = Σ Σ
III (Wr++b)
K=
REA(T) .R)
SE/
for some set of complex coefficients 3e. The desired result follows immediately from the convolu- tion theorem.
B RELAXED CONDITIONS ON THE INR ARCHITECTURE
Lemma assumes that the template function v Rd C has Fourier transform that exists in the sense of tempered distributions, and that the activation functions are polynomials, which is a stronger condition than merely assuming they complex analytic. Here, diseuss ways in which this can be relaxed to include more general activation functions, as well as how template functions on Euclidean spaces other than Rd can be used.
B. RELAXING THE CLASS ACTIVATION FUNCTIONS
In Lemma 1, two assumptions made. The first is that the template function has Fourier trans- form that exists, possibly in the sense of tempered distributions. The second is that the activation functions polynomials of finite degree. However, given that Lemma is a local result, mild sumptions on the template function allow for reasonable extension of this result to analytic activation functions.
Indeed, if db is assumed to be continuous, then one can take V C U to be a compact subset of the neighborhood guaranteed by Lemma 1. follows, then, that the functions (d(Wrr b,) are bounded over V. Then, if the activation functions merely assumed to be analytic, then the INR

---

Published as a conference paper at ICLR 2024
can be approximated uniformly well over V by finite polynomials. Without repeating the details of the proof, this yields an "infinite-derres version of Lemma 1, where for any & E Co (V), we have
( g.b.) (W, T)) oRe )) (5).
o fo(E)
Be
O
* 1=1
K- REA(FF)
This condition is not strong enough handle general continuous activation functions on C, since the theorem for approximating complex continuous functions requires polynomials terms to include conjugates of the arguments. One could conceivably extend Lemma in this way by including conjugation, but this would not be compatible with the algebra of I --progressive functions, since I -rrrogressive functions are not generally closed under complex conjugation. Under these conditions, then, Corollary does not hold for general continuous activation functions.
the preservation of conic support via the wavelets being --progressivv is not an issue, we can note further that when real-valued wavelets are being used, the Stone Weierstrasss theorem offers way to understand uniform approximations of an INR architecture with continuous activation functions by INRs that satisfy the conditions of Lemma 1. For instance, ReLU activation functions not polynomials, nor are they analytic, but uniform approximations by polynomials do exist over finite domains.
B.2 TEMPLATE FUNCTIONS FROM ANOTHER DIMENSION
It is possible to consider cases where v is defined as map from space of different dimension, say _ C. If g < d, then one can construct map & Rg - C so that (z T.g g .a) ( .Ty)- This, for instance, is the case with SIREN (Sitzmann et al., 2020) and the 1D variant of WIRE (Saragadam et al., 2023), where 1b R + R is used for functions on higher-dimensine gigees-dimensional spaces. In this case, the Fourier transform of 1 only exists in the sense of distributions. If q d, then one can apply the results in this paper by treating the INR as a map from RR. - C followed by restriction that "zeroes out" the excess coordinates by restricting the domain of fe to the subspace of points x such that x = (I1,..., Ld, O, .....,... 0).
C PROOF OF COROLLARY 4
We will show that the property of fe being locally I' r-progressive at a point holds, as the result for being "globally" T- T-progressive follows.
Let To be given, and let I S {1..... Fi} be the set of indices such that W + b, ( By Lemma 1, there exists an open neighborhood U ro such that for all E U, (x( takes the form of complex multivariate polynomial of {ıb( (WW + b.)her- Denoting this polynomial by P(r), we have that for any & E (U),
Φ P- 0. fo.
Under the given assumptions, each of the terms (W(Wrr + bi)lier is a I'-progressive function. Since --progressive functions constitute an algebra over C, any polynomial of them will yield I- T-progressive function. Letting U be the "sufficiently small neighborhood of ro," this implies that So locally T-progressive at ro, as desired.
Noting that --progressive functions also constitute algebra over C when I is weakly conic, this result also applies that case.
D WAVELET MODULUS MAXIMA INITIALIZATION IN TWO DIMENSIONS
For 2D images, we approximate WMM location as edges of the image. Fig. 6 shows the flowchart for WMM-type initialization. We first start with the image be represented and perform Canny edge detection on it. We then use the binary edge map as a proxy for WMM locations. We then use the edge locations as bias values for each neuron in the Gabor INR. The weights initialized so that each neuron generates a radially symmetric complex Gabor wavelet, oriented such that the oscillatory direction is perpendicular to the detected edges.
12

---

Published as a conference paper at ICLR 2024
(a) Image be represented
(b) Canny edge detection
WMM-type bias initialization shown as red dots
Figure 6: WMM initialization two dimensions. (a) shows an image to represented, while (b) shows the output of Canny edge detector. (c) shows WMM-type bias initialization with the red dots showing some of locations used as bias values.
Table I: Architectural descriptions for wavelet networks in Section 4
Architccture
(x))
F
L
F2.....
Fr
Parameters
(wavelet+scaling-toaal
Split, Complex, 1
exp
x=
+ 12
2220
1
n/a
6661 257 = 6918
No scaling, Real
exp
x²u²
sin s2n
37
3
641
6731 + 0 6731
2=22
No scaling, Complex
exp
12
37
3
64
6731 + 6731
Split, Real
exp
x
2
sin (T
37
3
64
6731
257 = 6988
Split, Complex
exp
22
37
3
64
6731
257 6988
EXPERIMENTAL CONDITIONS
E. ARCHITECTURAL DESCRIPTIONS FROM SECTION 4.3
We describe here both the architectures and training setups for the networks used in the experiments in Section 4.3. All scaling networks use gaussian template functions and no hidden layers beyond the one containing the template functions, i.e., for all scaling networks.
x
(xxx = exp
6
Fi 128,
where the 128 template atoms are initialized to be uniformly spaced over the signal domain [-2,
Apart from the scaling networks shared by three of the five architectures used Section 4.3, the wavelet networks considered have parameters listed in Table 1. All hidden layers use the activation function p(z) sin(z), for E C, where sin(2) understood to be the analytic continuation of the real-valued sine the complex plane. They are initialized according to the wavelet modulus maxima scheme described in Section 5. placing K atoms each maxima point, where K is chosen for architectures with hidden layers, and chosen in the L = case to achieve similar number of parameters to the other networks. The scales ())) are initialized uniformly at random the interval [1, K], so that the network with I, has many scales wavelets fit wide range of frequencies, while guaranteeing that the atoms progressive at initialization.
All architectures are trained for a total of 4000 epochs using AMSgrad (Reddi et al., 2018) to min- imize the mean-squared error between the real part of the INR and the target signal uniformly spaced points in the interval [-2, 2]. Split architectures are trained by first fitting the scaling network to the target signal for 2000 epochs, then fitting the scaling and wavelet networks simultaneously for 2000 additional epochs. Architectures that do not use scaling network are trained for 4000 epochs.
512
13

---

Published as a conference paper at ICLR 2024
E.2 ARCHITECTURAL DESCRIPTIONS FROM SECTION 5.1
We describe here both the architectures and training setups for the networks used in the experiments in Section 5.1. For the examples fitting the 1D test signal, the split complex Gabor architecture from Section 4.3 was used, as described in Appendix E.
For the examples fitting test images, we use a WIRE INR (Saragadam al., 2023) with L hidden layers for the wavelet network, with each Fi, F2, F's 128, and the same hidden nonlinear- ities as described in (Saragadam al., 2023). For the scaling network, we use INR with L and F1 128, with gaussian template function.
The network labeled "Gaussian" uses ID gaussian as template function (see Appendix B.2), the network labeled "Real Gabor" uses the imaginary part of complex Gabor wavelet as template function, and the network labeled "Sine" uses a sinusoid as a template function (Sitzmann et al., 2020).
F RESULTS ON KODAK IMAGE DATASET
To demonstrate some of the qualitative properties of INRs using complex template functions, we show here the individual representations of subset of the images from the image fitting experiments in Section 5.1. Each image is pictured in Fig. 7 with its PSNR in dB relative to the ground truth. We exclude the representations by the real-valued Gabor network, due to similar observed behavior to the complex-valued version, as well as the representations using sinusoidal template functions, due to poor overall performance.
We first notice that the gaussian template function is outperformed by the Gabor wavelet for all pictured images, no matter what initialization scheme is used. The randomly initialized Gabor wavelet network occasionally outperforms the using the WMM scheme, but this is by more than 0.39dB for the pictured images. The network initialized using the WMM scheme, on the other hand, has an advantage of up 1.85dB over the random initialization, indicating the strong advantage of using this scheme for images that are "cartoon-lit like," in the sense that they are smooth apart from isolated singularities such as edges.
14

---

Published as a conference paper at ICLR 2024
Ground Truth
Gabor (Random)
Gabor (WMM)
Gaussian 25.088
27.915
28.062
34.722
34.558
29.992
35.753
35.363
32.594
25.127
26.776
22.817
29.075
30.119
25.334
32.192
32.693
27.552
26. 26.077
27.928
23.621
Figure 7: INRs fit to subset of Kodak Lossless True Color Image Suite (kod, 1999). Images are labeled with their PSNR dB relative to ground truth image, with the highest PSNR for given image bolded and the second highest italicized.
15


Reviews:
[{"review_id": "2OlTgsCftV", "reviewer": "Reviewer_ksCd", "summary": "This paper studies a type of Implicit Neural Representation based on a wavelet nonlinearity.\nAn implicit neural representation is a parameterization of a (usually) scalar function of a (low-dimensional variable), e.g. a function f: R^d->R using a neural network. In this setting, an image is a function R^2->R and the input to the function are pixel locations. This setup requires some reflection about the role of in particular the first non-linearity. Using ReLU will lead to lots of ramps, requiring their combination in intricate ways to represent usual real-world signals.\nHere the use of wavelets as first non-linearity is analyzed. The use of wavelets comes out of a line of research leading via sinusoidal components (e.g. SIREN) and presents itself as a clear follow-up. The setting studied here is a first layer with wavelet nonlinearity followed by several layers of pointwise mixing layers with polynomial nonlinearities.\n\nIn this context, using the Fourier convolutional theorem, the functions expressible by this architecture are characterized in Fourier space as essentially a collection of higher-order self-convolutions of affine-transformed versions of the base wavelets.\n\nIntroducing the notion of \"progressive wavelet\" which have their Fourier support on a cone (or a \"weak cone\" that is closed for factors >= 1), it is shown that the wavelet self-convolutions above never leave their cone, leading to a neat characterization of what functions can be matched using a particular wavelet.\n\nTo accommodate low-pass signal, a split of the INR is proposed into the sum of two INRs, one using a wavelet, the other using a low-pass filter. This decomposition is shown to be beneficial in fitting a signal.\n\nFurther, wavelet modulus maxima points are proposed to initialize the representation. It is shown that for certain signals the use of WMM points as initialization leads to a better fit.", "questions": "The term \"progressive wavelet\" was new to me. However, as far as I can tell, its definition fits exactly what I know as \"analytic wavelets\". Could the authors confirm this is the same and in that case justify the use of a new term, or explain in what way these concepts differ?\n\nThere are two more direct questions listed in the weaknesses section.\n\nRegarding the low-pass + progressive wavelet decomposition for images: Is it possible to show the smooth parts and the wavelet parts? A conjecture for this, if both INRs are allowed to output in RGB space, is that the smooth parts contain much more color than the wavelet parts. Would be interesting to see if that is the case for certain natural images.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is very clearly written and presents its contributions is a highly succinct way. It is a pleasure to read - in fact it reads a bit like an advanced chapter of a wavelet applications textbook. The illustrations contribute to the ease of understanding.\n\nThe paper gives a concise characterization of the function space spanned by the wavelet nonlinearity followed by layers of pointwise polynomial mixing. From this characterization it clearly identifies, using progressive wavelets, that a split into low-pass and wavelets is highly useful.\n\nIn short, it leads to a complete understanding of the particular setting introduced. The hope is that this understanding can extend to adjacent settings.", "weaknesses": "The provable statements in the paper are not in any way non-obvious. Theorem 1 is a direct consequence of the Fourier convolution theorem.\n\nThe setting in which these proofs work are highly impoverished with respect to the setting of actual interest, which is that of non-polynomial nonlinearities, such as the ReLU, for the pointwise mixing layers. The shifting of frequencies along a cone does not hold for these, and complicated ringing processes emerge that can also define sharp boundaries by the rectifier suppressing negative values. (The effects of ReLU wrt wavelets are partially analyzed e.g. here https://arxiv.org/abs/1810.12136 but it looks intractable to integrate this in the current analysis).\n\nGiven the nature of the expressed function space, what is an obvious advantage of this particular implicit neural representation over a sparse continuous wavelet transform with a sufficiently expressive bank of filters (e.g. some base filters and some of their polynomial powers)? In particular, if one needs to use wavelet modulus maxima to initialize the representation. Could these advantages be concisely stated? (e.g. I can imagine that potentially the representation requires fewer parameters)\n\nIn general, could some numerical comparisons be done to place the analyzed method within context of other INRs? We are not looking for state of the art here, but to have an idea of whether the proposed setup is close or far from that. If it is close, this can also partially alleviate the expressivity concern mentioned above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gcB0IOoWEx", "reviewer": "Reviewer_3UQT", "summary": "The authors use Fourier analysis to study the properties and expressive power of implicit neural representations (INR). In particular, they analyze multi-layer INRs whose first-layer activations (which they call the template function) map from Euclidean space into the complex numbers and whose activations on all other layers are polynomials. \n\nThe authors show that when the template function is essentially a band-pass filter, the class of INRs they study also act essentially as band-pass filters. Based on this observation, the authors propose to model signals as a sum of two INRs instead, one that acts as a low-pass filter and one that acts as a high-pass filter.\n\nThe authors also suggest initializing INRs with wavelet template functions placed on estimate singular (i.e., non-smooth) points in the domain of the signal (e.g., edges in an image). They show that this careful initialization results in significantly improved performance compared to random initialization.", "questions": "n/a", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Analyzing the expressivity of multi-layer INRs, not just one-hidden-layer ones, is a very relevant problem; hence, any progress in the area is nice. I found the low-pass-high-pass decomposition idea and the suggestion to initialize the template function's biases to the signal's singular points interesting.", "weaknesses": "As someone whose primary area of research or background is not in classical signal processing, I found the paper quite confusing. In particular, the authors provide little to no interpretation of their results and observations. This led me to feel that I was constantly expected to be able to interpret and understand them, which I failed to do in many cases. Moreover, it is unclear what mathematical level the authors expect of the reader. They state some quite elementary results rigorously while handwaving others (e.g., effective support, Minkowski sums). Finally, several technical signal processing terms (such as atoms, band-pass filters, and WIRE) are undefined, which makes reading the paper quite challenging.\n\nFor example, the authors showcase Thm 1 as one of their main results. However, Eq (2) is just the Fourier transform of a slightly rearranged definition of the INR architecture they are studying. On the other hand, some of the assumptions and both conclusions are unclear:\n- Why is restricting our focus to polynomial activations beyond the first layer interesting to study?\n- Why is the Fourier transform stated by considering multiplication by a smooth function $\\phi$?\n- How does the fact that the INR is composed of the sum of the template function's \"integer harmonics\" illustrate the expressivity of the INR?\n- \"Second, the support of these scaled and shifted atoms is preserved so that the output at a given coordinate $r$ is dependent only upon the atoms in the first layer whose support contains $r$.\" - How is this not a trivial statement? What is the relevance of this observation?\n\nSimilarly, what is the purpose of section 3.2? While I follow and agree with the argument, the whole discussion is informal, so it technically contains no actual results. More importantly, though, it's unclear what the insight is.\n\nHowever, Section 4 is perhaps the most confusing part of the paper. In the first paragraph, the authors state that in this section, they \"consider the advantages of using complex wavelets, or more precisely progressive wavelets.\" This sentence has two issues: first, it shows that the paper title is a misnomer because the authors actually only consider the algebra of progressive wavelets. Second, they don't demonstrate any advantages of using progressive wavelets. Perhaps most confusingly, even though the concept is featured in the paper title, the authors never actually make use of the fact that progressive wavelets form an algebra.\n\nEssentially, the authors spend the first two subsections defining a multivariate notion of a band-pass filter using weakly conic sets. Then, in the third section, the authors claim in Section 4.3: \"Based on this property of INRs preserving the band-pass properties of progressive template functions, it is well-motivated to approximate functions using a sum of two INRs: one to handle the low-pass components using a scaling function, and the other to handle the high-pass components using a wavelet.\" I do not follow this argument. I'm not saying it is not a good idea, but I do not see why it is well-motivated based on the multivariate notion of band-pass filters to decompose signals into a sum of a high-pass INR and a low-pass INR.\n\nFinally, the experimental section of the work is severely lacking. The authors only conducted one ablation study for their suggested initialization technique vs. random initialization by reconstructing three different images. Hence, it is unclear how the suggested signal decomposition into two INRs or the initialization technique helps with the usual tasks INRs solve in practice compared to other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sl6YcARtWa", "reviewer": "Reviewer_ewcV", "summary": "The study conducts a time-frequency analysis of INRs by leveraging polynomial approximations to delve into the behaviors of MLPs beyond the first layer. By decomposing a signal into its low-pass and high-pass segments using two INRs, motivated from the scaling and wavelet functions of the wavelet transform. This approach bridges the structure of complex wavelets and the application of INRs, which is a novel concept.", "questions": "The reviewer is curious if the proposed method and its theoretic analysis/intuition is valid, when the method is applied to practical applications.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper presents a novel perspective on behaviors of INR models, including the decomposition of low and band-pass approximations, along with specific initialization methods, and enhances both the depth and practical relevance of the study.", "weaknesses": "Although the performance of the proposed method was supported by several tests and analysis in the paper, it is suggested to include some practical applications such as regression tasks on images or other high-dimensional signals to justify its practicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9g5xwUxPSD", "reviewer": "Reviewer_rBV3", "summary": "The article studies implicit neural representation models which typically uses sinusoidal activation functions and wavelet-based activation functions, in order to represent signals. This work develops theoretical understanding of such architectures and also a practical split architecture to capture singularities in target functions.", "questions": "-\tYour definition of the model in eq 1 seems to have some issue to me. Psi is a function of R^d -> C but somehow W^0 r and b^0 is in R^{F_1 x d}. Thus I do not understand the model. \n-\tThe notation inf Theorem 1 is not clear. What is the definition of the Fourier transform for functions in C_0^inf (U) in eq 2? What does this stand for C_0^inf (U)? What is the product_t from t=1 to t=F_1? Does hat beta_l depends on W^{l} and b^{l}? Is * a convolution, over which domain? As a consequence, the argument in Section 3.2 regarding the support of the product_t term in eq. 2 is not clear. Is W^T a typo of W_l^T? \n-\tIn Section 4.3, what does it mean to model a signal as a sum of a linear scaling INR and a nonlinear INR ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The theoretical analysis seems to be novel. By using Fourier analysis, theorem 1 gives an ide a about the effective time-frequency support of implicit neural representations. Numerical results such as in Fig 4 further support the theory.", "weaknesses": "The overall clarity of the presentation still needs to be improved to make results accurate. See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SYQKPyDhZC", "reviewer": "Reviewer_ZxVa", "summary": "This work studies the task of Implicit Neural Representations using neural networks with wavelet activation functions. Paper provides a theoretical framework for the approximation capabilities of such networks, and demonstrates how the theory can be helpful in designing the INR architectures. Experiments are provided on a few images.", "questions": "How can we interpret the result for the medical image in the bottom row of figure 6? To me, it seems that the error is more for the WMM case. Is that correct? If that is the case, the description in Appendix 5 may not be as accurate when it says “WMM based initialization has limited advantages …”. This would be a disadvantage for WMM and not an advantage?\n\nIntuitively, how does the definition of progressive wavelets affect the approximation capability of a model? What would happen if we use a non-progressive wavelet to model the same images that authors present in Figures 1, 6, 7? We will lose the guarantees from the theorems, but how would that affect the learned representations? How would that empirically affect the error.\n\nIn Figure 6, how many parameters do the models have?\n\nHow could the theory and the method be used for image compression?\n\nPlease also see questions under weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The theory presented in the paper is a solid contribution for research and practice.\n\nPaper is well written. Arguments are clear.", "weaknesses": "In my view, the weakness of this paper is its small set of experiments. I also think more details can be provided to interpret the experiments and to discuss the theory.\n\nI think providing more details and more discussions would make the paper more approachable for a broader audience.\n\n---------\n\nPossible typo in the abstract: band-pass -> high pass. In my understanding, the method decouples the low-pass parts from the high-pass...\n\n----------\n\nI would have liked to see experiments on more images. I only saw three examples. Specifically, I'd be interested to see more examples of how the method works on more images, its errors, and its possible failure modes.\n\nIt may be useful if authors present cases where one might encounter troubles in training -- and cases where the learned representation may be flawed. The first image in Figure 6 seems to be more of a challenge from the training convergence perspective. Are there images where the reduction of the training loss would be even more challenging?\n\nIn the right column of Figure 6, the curves seem to still have a positive slope even towards the end of the horizontal axis. Is that correct? If yes, how would the curve proceed if training is continued further?\n\nAuthors can consider presenting the likes of figure 6 for earlier stages of training, e.g., for 5, 10, 50 epochs. How would those images look like? The convergence curve (right column in Figure 6) seems to be overly compact, so it is not easy to see the particulars for the early epochs of training. It seems that for the parrot picture, the WMM initialization lags behind the random initialization at the early epochs.\n\nIn Figure 6, the WMM result for the parrot has error at the left corners of the image. It may be useful if authors interpret those errors. Specifically, the patterns at the top left corner seem to appear in that general area of the image. Why does that magnitude of error only happen at that top left corner and not in that whole green area?\n\nOverall, demonstrating results on more images would give a better understanding to a reader.\n\n---------\n\nI think the discussion in the context of wavelet literature could have been broader. For example, I did not see any discussions on the topic of Daubechies wavelets. Are Daubechies wavelets also progressive by the authors’ definition?\n\n------\n\nIf authors think Shearlets may have any potential here, providing a discussion might be useful. Most of the approximation error for the pictures in Figure 6 appear to be at locations where the colors change in a small neighborhood. Could a shear matrix be potentially helpful in reducing the error because of its ability to extract anisotropic features?\n\n------\n\nA relevant prior work that authors may consider citing if they see fit:\n\n-- Grattarola, D. and Vandergheynst, P., 2022. Generalised implicit neural representations. Advances in Neural Information Processing Systems, 35, pp.30446-30458.\n\n-------\n\nUsing wavelets to study approximation properties of neural networks, and leveraging that to design the architecture of the network, is studied in the past, but I did not see a mention of that in the paper.\n\n-- Shaham, U., Cloninger, A. and Coifman, R.R., 2018. Provable approximation properties for deep neural networks. Applied and Computational Harmonic Analysis, 44(3), pp.537-557.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)