SPDER: Semiperiodic Damping-Enabled Object Representation
Authors: Kathan Shah
Chawin Sitawarin
Venue: ICLR 2024 poster (2024)
Keywords: Implicit neural representations
spectral bias
computer vision
neural network architectures
activations
image representation
edge detection
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A new activation function that we claim overcomes spectral bias in neural networks and be used to represent images, audio, video, etc.

Abstract: We present a neural network architecture designed to naturally learn a positional embedding and overcome the spectral bias towards lower frequencies faced by conventional implicit neural representation networks. Our proposed architecture, SPDER, is a simple MLP that uses an activation function composed of a sinusoidal multiplied by a sublinear function, called the damping function. The sinusoidal enables the network to automatically learn the positional embedding of an input coordinate while the damping passes on the actual coordinate value by preventing it from being projected down to within a finite range of values. Our results indicate that SPDERs speed up training by 10 times and converge to losses 1,500 to 50,000 times lower than that of the state-of-the-art for image representation. SPDER is also state-of-the-art in audio representation. The superior representation capability allows SPDER to also excel on multiple downstream tasks such as image super-resolution and video frame interpolation. We provide intuition as to why SPDER significantly improves fitting compared to that of other INR methods while requiring no hyperparameter tuning or preprocessing. See code at https://github.com/katop1234/SPDER.

Published as a conference paper at ICLR 2024
SPDER: SEMIPERIODIC DAMPING-ENABLED OBJECT REPRESENTATION
Kathan Shah UC Berkeley ka whwn@berkeley.c.mm edu
Chawin Sitawarin UC Berkeley Shhawins@beerkeee..eee
ABSTRACT
We present a neural network architecture designed to overcome the spectral bias towards lower frequencies faced by conventional implicit neural representation networks. Our proposed architecture, SPDER, is a simple ML.P that uses an activation function composed of a sinusoidal multiplied by a sublinear function, called the damping function. The sinusoidal enables the network to automatically learn the positional encoding of input coordinate while the damping passes the actual coordinate value by preventing it from being projected down to within a finite range of values. Our results indicate that SPDERs speed up training by 10x and converge to losses for image representation. SPDER is also state-of-the-art in audio representation.
lower than that of the state-of-the-art 10000-10,,0000
The superior representation capability allows SPDER to also excel on multiple downstream tasks such image super-resolution and video frame interpolation. We provide intuition to why SPDER significantly improves fitting compared to that of other INR methods while requiring no hyperparameter tuning or preprocessing. See code at htttss//gihhbbb.oo//koo///2244//
INTRODUCTION
Implicit neural representations (INRs) have demonstrated remarkable potential in applications such as novel view synthesis, image impainting, super-resolution, and video frame interpolation (Sitzmann
2020
Mildenhall al. 2020 Chen al. 2021 Simeonov al.) 2021 Park et al. [2019] Barron
et al. 2021 Zhuang et al. 2022). Such representations, also known as coordinate- based multi-layer
perceptrons (MLPs), can be modeled as
do (x) (x)
where de is neural network that learns a continuous mapping from coordinate grid x to the ground truth signal vú of an object. Here, x E x can represent a pixel coordinate in an image and (xr) the pixel value
These networks have attracted significant research interest for a number of reasons. Conventional methods for representing coordinate-basee objects (c.g.. images, videos, 3D objects, and audio) are restricted to discrete input grid (Wallace, 1992 Painter and Spanias 2000; WWeegnd et al., 2003) while INRs can interpolate to continuous input values and thereby generate arbitrarily high resolution. They also encode an entire object solely within their weights, which is particularly useful due to their relative compactness (Strümpler et al. 2021 Yang et al. 2022 Ballé et al.) 2016 Li et al. (017).
However, coordinate-based MIPs, and neural networks collectively, have struggled with spectral bias towards low-frequency modes and an underemphasis on center- and high-frequencies (Rahaman) et al. 2018). They target lower frequencies, which tend to be more robust to perturbations, at the expense of higher ones encoding the finer structure of the signal. Consequently, they suffer from a greater loss than is ideal. Note that bias towards lower frequency modes improves the generalization, whereas one towards higher frequencies improves task-specific performance (Cao et al.) 2020 Xu et al. 2018).
Frequencies are fundamentally based on patterns in values over given positions. Consider an ideal INR: should be capable of identifying that pixel P is located in the upper right corner of image

---

Published as a conference paper at ICLR 2024
SPDER Reconstruction
SIREN Reconstruction
Log training lesses
SPDRR
5PEE
10:2
10-3
10"
10
10"7
Figure 1: SPDERs learn neural representations of image magnitudes better than other MLP techniques The 256x256 ground truth was trained on by SPDER (left) and SIREN (center), our baseline, for the same number of steps. By simply applying a damping factor the sine nonlinearity, the loss goes down by nearly x2,000x, and training speeds up by 10x.
(position) and that P represents part of the sky (value). Therefore, maintaining awareness of both positional and value information for inputs is crucial to overcoming spectral bias and also achieving good performance. This is important not only from research standpoint, but also for applications such as reconstruction, compression, and edge detection. Our goal is to devise an INR architecture that maximizes
where is function that returns similarity score between two vectors and F is the Fourier transform (Morin 2021). Recall that the Fourier transform provides an equivalent representation of an object over the frequency domain.
Conventional methods rely on a single MI.P to represent an object. Here, each neuron technically has the receptive field of the entire object, which makes representing its complexity very challenging. Consequently, researchers map inputs through sinusoidals of certain frequencies during preprocessing fit higher frequencies. However, this requires hardcoding inductive biases and does not necessarily capture the true spectrum (Ke et al.) 2020: Yúce et al. 2021). Another technique, known as SIREN Sitzmann al. 2020p, directly uses sine as activation function the MLP to help learn high frequencies. However, the output from single neuron is "lossy" because of its many-to-one mapping. Hierarchical methods (Müller al. 2022]] Hao et al. 2022) break down an image into smaller chunks, for which less complex representations final output. Likewise, most neurons have much smaller receptive fields, which makes encoding complexity easier. These methods tend to have significant hardware requirements, which have prevented widespread adoption.
be learned and then aggregated for the
We propose SPDER to address all of the above. SPDER involves single MLP with activation function composed of a sinusoidal multiplied by sublinear function, which we refer to as the damping function. We show that SPDER learns near-perfect mapping from coordinate grid to any high-fidelity signal including images, audio, and videos. This is because the sinusoidal enables learning the positional information of the input, while the damping helps pass its value between layers. Note this allows the network to naturally learn a positional embedding.
We summarize the contributions of this paper below:
We propose a simple and elegant architecture that captures the frequencies of different types signals exceptionally well, achieving state-of-the-art in both image and audio representation. Specifically, it outperforms state-of-the-ar image INRs by several orders of magnitude. Compared to other hierrriraal methods like itself. it speeds up training by 10x and converges to a 15,000 lower loss.
We demonstrate how our novel activation function helps a neural network overcome spectral bias and learn a positional embedding without any hyperparameter tuning or hardeoded inductive biases by having each neuron balance encoding both the position and value information of signal.
We highlight its generalizability through downstream applications such as image super-resolution, edge detection, video frame interpolation, and more.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Sitzmann et al. (2020) popularized general-use INRs by showing their ability to represent a myriad of objects such images, audio, and 3D) objects, well as solve physics equations. SIREN, the architecture they proposed, is a simple MI.P with sinusoidal activation functions and demonstrated previous architectures. It is the main baseline compare our
significant improvement architecture to.
Regarding balancing the position and value information of inputs, notably, Vaswani al. (2017) found that hardcoding sinusoidals of logarithmically-saaee frequencies during preprocessing allowed model use the relative position of tokens in sequence. This method is known as positional encod- ing (PE). Several INR architectures use a similar approach to represent higher frequencies (Mildenhall
jt al. 2020, Chen et al., 2021; Barron et al. 2021). However, these fail to successfully represent common media like images and audio without extra hyperparameter-tuning ("Tnciik et al.) [2020] Ke
ct al. 2020 Luo et al. 2022).
The spectral bias of neural networks is also an interesting property we wish to highlight for this paper. Such bias occurs because neural networks fit lower frequencies to generalize better and improve validation performance. However, they end up excluding the detailed, high- high-frequency structure of an input which leads to worse fitting (Xu et al. 2018 Rahaaman et al.] 2018). PE has been proposed to solve this, but interestingly, MLPs using PE have been shown overfit the integer harmonics of the PE frequencies (Yüce et al. 2021). Moreover, they include hardcoding inductive biases.
Video-specific INR techniques were explored by (Chen al. 2023 2022 2021). NeRF networks, used for novel view synthesis, typically use ReLU activations with sinusoidal positional encoding (Eq. (7) Other methods, such Fourier features, include appending the input with frequencies generated from a Gaussian distribution to overcome spectral bias (Tancik el al. 2020). These networks leverage neural tangent kernel theory (Jacot al., 2020) and are widely used in computer vision. In general. sinusoidals have extensively been used in INR architectures because of their periodic nature and ability to map uniformly spaced inputs to dense output signal.
Interestingly, Ramasinghe and Lucey (2021 examined several nonlinearities and concluded that necessary condition for a neural implicit representation is having nonlinearity with high Lipschitz smoothness. For example, (Chng et al. (2022) demonstrated that Gaussian activation functions work
under limited capacity.
Hierarchical methods have also shown great performance (Hao et al. 2022, Mildenhall et al. 2020). Notably, Instant-NGP (Müller et al. 2022) uses a hierarchy for an object over several resolutions, then queries select few parameters generate output. It speeds up training for high-information media such as neural radiance fields, gigapixel images, and SDFs by several magnitudes. To the best of our understanding, Instant-NGP, in particular, learns piecewise-linest approximation of object and is only reasonable to use for extremely high-information objects like the ones listed prior. For example, in our experiments, we tried running Instant-NGP several different machines, but all but a single high-end machine (costing over $20,000) failed due to lacking system requirements. Our method has no such restrictions and can even run on personal laptops.
3 APPROACH
3.1 MOTIVATION
Recall that for INRs, an input is point from a coordinate grid, and the goal is to map it onto signal (i.e. pixels of an image or samples of audio). An ideal network would correctly represent the frequencies between points in all directions of the grid while also keeping track of the actual coordinate value. This is because points along a direction may share similar frequencies/have dependencies, but have vastly different local features.
In SIREN, neurons massage points along the same frequencies onto the same outputs. Although they learn positional information automatically, they lose any sense of the original value of the coordinate point due to their mapping a wide domain onto a small, cyclical output range. For example, for a sinusoidal activation of period 0.25, sample points {0, 0.25, 0.5} with possibly vastly different local
3

---

Published as a conference paper at ICLR 2024
Ourve sintx!
sinki aarctanidl
Curve for sinte) sartijit
1.0
0.5
-1.0
-15
-0
Curve for sintx] loo(pC)
Curve for sinDo) *.
Curvi for sincio xx"22
1050.
70
-40 -20
20
-20 20
40
Figure 2: Six possible SPDER nonlinearities. Note for 8(x) arctan(2) (top center), the curve converges + sin(x) for sufficiently far from O. For 5(x) VIØl (top right), the magnitude of the curve grows relatively slowly, and resembles two parabolas oriented sideways. For d(x) (bottom center) and 8(x) x² (bottom right), we emphasize how the y-axis significantly stretched out and the modulations drown out the periodicity. For these reasons, sublinear 8 is desirable.
features would be mapped to the exact same output. To remember an input value, the network would therehy have to coordinate several neurons, reducing effectiveness.
3.2 DAMPING
SPDER is neural network architecture that semiperiodic activation function. Such functions are characterized as
sin(2)
5(x:)
where is a sublinear function of x, also called the damping function (Fig. 2 We use the term semiperiodic to describe function if it oscillates across the x-axis with predictable pattern.
By modulating the magnitude of the sinusoidal through we embed knowledge the raw value of input directly into the activations while still reaping the benefits of its positional information through the periodic sinusoidal. This acts as a substitute for artificial positional encoding but with no additional preprocessing.
For activation functions described by Eq. 3 the local Lipschitz constant is bounded by ((ō(a) V8(a)) for an input (proof in A.1). In plain English, this means the rate of change of the neuron's 's output with respect to small changes in its input is proportional to the damping function and its gradient evaluated that input. Consequently, for linear (and superlinear) poor performance as the activation function exhibited unstable oscillatory behavior (Fig. 2). This meant the effective periods were far too irregular encode meaningful frequencies. Restricting to be sublinear lowers the Lipschitz constant enough to ensure smoothness across all neurons and allows the network to balance both the input's positional and value information.
empirically observed
Unless stated otherwise, 8(z) will refer the function /|z|, which is just the square root of extended to the negative numbers. Other promising damping functions that fared well on our tasks included log([ær); arctan(x), and /Z)..U(:). We chose V because it performed exceptionally on our image dataset and has mostly well-defined gradients, but note the others may be valuable for different settings. For example, SPDER with 8 arctan represented audio the best. We also a constant is equivalent to SIREN which markedly underperforms against
note the case where sublinear ?(1) like the ones listed above. We found SPDERs perform optimally in settings where they had to learn a mapping from coordinate grid to a signal. For example, they are less successful novel view synthesis, where the input direction is polar.
4

---

Published as a conference paper at ICLR 2024
The jagged curve beyond step 150 in Fig. indicates the complexity of the loss landscape that SPDER navigates. When optimizing past a certain loss threshold, SPDER must traverse a highly non-convex landscape, leading to fluctuations. Unlike other networks whose performance plateaus, SPDER's gradients can keep navigating the complex landscape towards the global minimum. In fact, SPDER's reconstruction often has 0 pixel deviations on 8-bit scale. We note these loss spikes do not translate into visible defects in the resulting representations, and SPDER consistently outperforms all other architectures. certainly possible (but not necessary) to select the minimal loss representation up to certain step or extend the training until the loss no longer spikes.
3.3 FREQUENCIES
In general, an implicit neural representation de is fed in a single pair of input x and ground truth output (xx throughout training. The training loss is simply mean-squared error (MSE), i.e.
L = (x) (x)]1
MSE, albeit a convenient objective for backpropagation, can be a misleading metric to humans for discerning the successful reconstruction of an object. In the case of images, for example, it is marginally affected by blurriness but is very sensitive to small shifts. Naturally, studying similarity in the frequency domain would mitigate these issues (Amir and Weiss 2021 Zhang al. 2018). Through Parseval's ' theorem, however, can be shown that minimizing the MSE of the frequency domain of the represented image and ground truth is tantamount to minimizing the MSE between the pixels themselves [Wronski] (2021).
This necessitates finding another metric validate the frequency structure of representation. Referencing Eq. (2). we seek to maximize the cosine similarity between the amplitude spectrum of the neural representation and the ground truth. We formulate this maximizing
ENHO An V VINI AR EN-i G2
PAG
(5)
where each G, is zero-centered magnitude from the Fourier series of the ground truth signal we wish to represent, each A,, is the corresponding zero-centered magnitude from the neurally represented signal, and PAG is the cosine similarity between A (Ap. A1, A1...., AN-1) and G (Go, G1,
GN-1)) Note in Eq. (5). we omit any notion of phase shifts because the representation is learned from the same basis that of the true signal, and hence, phase shifts should not impact the overall similarity. Therefore, for some vector s encoding the signal of interest, the æith term in A and G comes from the general formula for the Fourier amplitude spectrum of discrete-time signal [
N
SPe
12nnkNN
(6)
N
K=
We demonstrate how SPDER not only has incredibly low loss but also maximizes PAG and, therefore, captures the exact frequency structure of an image (Table [.. Moreover, it can achieve this in compact and efficient architecture, with no hyperparameter- or fine-tuning. We believe this because each neuron in SPDER is designed to rapidly learn the frequency domain. Note we chose not to optimize for this metric during training as it's far more complicated to compute and had no practical advantage. We present it simply to lend insight into SPDER's overcoming of spectral bias.
We wish to highlight this as one of the most exciting features of our architecture, as significant research has been conducted on learning the frequency distribution of objects through positional encoding, hierarchies, increasing layer width, etc. These certainly help, but they introduce more inductive biases and most have yet to overcome the spectral bias known to pervade neural networks (Rahaman et al. 2018: Lieskovská et al., 2021 Bai et al. 2018). We find SPDER's ' performance to be extremely conceptually intriguing because succeeds when each neuron has the receptive field of the entire object. Therefore, we hope our findings can advance research done in fitting the frequency manifold of inputs.
5

---

Published as a conference paper at ICLR 2024
Activation Distribution for bix)
Activation Distribution for 6(x) scrt([x])
5000
4000
4000
Count
Count
2000
2000
1000
1000
Activation
Activation
Figure 3: For 5(x) (left), the activation values distinctly peak at local extrema of sin(x) at #l. For ((x) / |a| (right), they peak local extrema of sin(x) // #1.31, =2.18, =2.81, =3.11 etc. Using solely (i)( as a nonlinearity discards information about the magnitude of the input, unlike conventional SPDER network.
3.4 ACTIVATIONS
Experimentally, we observed that the distribution of activation values across trained SPDERs peak at values of æ for which (sin(a:) 6(z)) 0 (Fig. 3 Fig. 8). Activated neurons apparently try to their output by seeking out stationary points. This demonstrates feature
maximize the magnitude selection means neurons conveying information have their signals amplified while others' outputs get minimized (similar to how a high-pass filter works).
Due to the damping, neurons with inputs of larger magnitude will, average, be mapped to outputs of larger magnitude for 8 E 22(1). Moreover, given their semiperiodic nature, inputs spaced equally far apart will still be mapped to roughly equal outputs. Through maintaining this delicate balance, the network is able to relay value information from one layer to the next while automatically learning a positional embedding. Further examples in A.3
RESULTS
4.1 BASELINES
ReLU: Standard activation for most MLPs. Neurons map an input to max(0, i).
ReLU with positional encoding: The input layer is appended with:
(sin( cos(wnt) ... sin( cos(wr,11))
(7)
where is the frequency of the k-th sinusoid, and L is the number of frequency bands used for the encoding.
ReLU with Fourier Feature Networks: Similar to positional encoding, but frequencies are generated from a Gaussian distribution. We append 20 of such encodings the input of a ReL.U network from standard Gaussian.
Instant-NGP: We show how although Instant-NGP converges very rapidly, underperforms SPDER in the low-resolution regime. Further comments in A.8
SIREN: Neurons map inputs to sin(w))), where wi is hyperparameter which 30 unless stated otherwise.
SPDER:
We demonstrate this to have a 10,000 to 100,000-fold improvement in image representation loss compared to SIREN. Neurons map inputs to sin(wpi) 5(i) for sublinear ō.
4.2 IMAGE REPRESENTATION
Results are shown in Table [ SPDER consistently outperformed the other architectures across all metrics all training steps. Specifically, we demonstrate how SPDERs excel at representing
This easily be evaluated through NumPy's fCc module

---

Published as a conference paper at ICLR 2024
Ground Truth
SPDER Reconstruction
Log training losses Image
10.4
10
10-5
WWW VVV
10"
50
100
350
200
250
300
350
Figure 4: (Left) SPDER's 's reconstruction of the image from sk skimage data cat () after training for 500 steps (taking ~55 seconds). Notice has no aliases. (Right) The single-channel log-loss curves for various activation functions on the same image are shown. By the end, SPDER has more than 100,000x improvement compared to SIREN. Note that no configuration of positional encoding, network size, hierarchy, etc. from literature has been shown to match simple SPDER on this task.
Architecture
Metric
Step
ReLU
ReLU w/ PE
ReLU w/ FFN
SIREN
Instant-NGP
SPDER
25
0. 0.006
0.1204::0.05
0.2360+00.09 0.1325+0.06
0.0334+0.02
6.0142:46e-3
0.0079=3e-3
Loss (J)
100
0.0850=:0.05
0.0400=00002
0.0100+55e33
0.0012+7c-4
7.0c-5=5o-5
500
0.0498.10.03
0.0052:13e-3
0.1313:10.06
0.0010.1.e-4
0.0001119-55
6.7e-8-1e-7
25
15.4+2.2
15.74:2.1
12.58:11.7
21.5=2.5
25.0:22.3
27.5+22.0
PSNR (t)
100
17.4
20.711..6
15.29112.2
26.91229
36.2.12.9
48.6.1..1
500
19.9+2.9
29.7+:2.8
15.34+2222
36.6+2.33
46.6#2.9
85.7=11077
25
0.7860::011
0.7823:10.12
0.5327:0.21
0.9522:40.03
0.9936=3e-3
PAG (t)
100
0.8543:10.1 10
0.9443:10.04
0.7208 14
0. 0.9866119e-3
0,99980 4e-4
500
0.918510..66
0.992716e-3
0.730010.14
0.9988199.44
0.99995 2
Table 1: DIV2K 256 256 image representation loss, peak signal-to-noise ratio (PSNR), and frequency similarity (PAG) with standard deviations over various training steps. Sample sizes: ReLU (N=801), ReLU w/ PE (N=800), ReLU w/ FFN (N=800), SIREN (N=801), Instant-NGP (N=800), and SPDER (N=702). Note how by 500 steps in, SPDER's representation is virtually lossless 13. The PSNRs are also significantly above that of SIREN's 's and Instant-NGP's, which both have decent quality (A.14). We emphasize how within merely 25 steps, SPDER's s loss is low enough such that the representation is visually pleasing to any viewer. SPDER has near near-perfect cosine similarity with the amplitude spectrum of the images, which means it faces no difficulty overcoming the spectral bias known to pervade conventional neural networks.
images, with 10x improvement in training steps to achieve similar loss to baselines and 10,000-100,000x reduction in loss compared to the others. The amplitude correlation of SPDER's s representation with the ground truth is greater than 0.9999, indicating almost perfect representation of the frequencies. A PSNR of 45 is generally considered excellent quality, and SPDER achieves significantly beyond that. We observed that in most cases. SPDER's 's representation exactly matched the ground truth after 500 steps (3.13). It is the only known method in literature to achieve lossless neural image representation (which we claim be more proof it overcomes spectral bias for images).
We also wish to reinforce the sheer simplicity of the model. Being solely 5-layer MLP, one could initialize in PyTorch in less than 20 lines of code and achieve these results.
4.2.1 IMAGE GRADIENT REPRESENTATION
We observed that SPDER is able to learn a near-perfect representation of the gradients of an image (. SPDER models supervised solely the pixel values of an image, and the gradients are determined from their reconstruction of the image. Consequently, the model could only succeed in this task if its representation accurately included the structure of the edges.
7

---

Published as a conference paper at ICLR 2024
Original image
Gradienss mage though Gradeenss of image throu Gradients of image ttrough
Gradients of mage thoough
dracients of imge through
Figure 5: 5-layer networks were trained on the image from sk skimage data. data.textt () (Ist from left). The gradients of their reconstructions are shown. Note how SPDER represents the edges well within merely 10 training steps (2nd), especially w.r.t SIREN and ReLU with positional encoding (4th, 6th). After 25 steps. SPDER has captured the exact edges of the image (3rd).
Amolitude Spectrum of 256x256 Ground Truth
Amalitude Spectrum of 256x256 SPOER Representation
Arpiltude
Superreso
1024x1074 SPDER
Weettel Sppettet meep aerrrryy
hortzonnal Spauell Fraquency
Horizental Spa0ial Fraquency
Sporial Frequency
Figure 6: 2D amplitude spectrums of single channel of the 256x256 ground truth image (left) from skimage, data ast ronaut () and of SPDER's representation While interpolating resolution 1024x 1024 (right). SPDER is able to preserve the structure of the original signal, hence the defined horizontal and vertical bands near the center, which represent lower frequencies. Note the solid, pale blue regions around the corners of the 1024x 1024 resolution spectrum, which indicate SPDER introduces minimal high-frequency content/noise into the super- resolved image.
it (center) are shown.
Properties of the Fourier transform dictate that determining F(I) and the frequency coordinate along one axis uniquely determines F of the gradient of along that axis. This is formulated as:
F(V,1) = = wiy F(T)
(8)
where wy is the frequency in the y-direction and i is the imaginary unit (proof in A.2 since SPDER learns the correct pixel frequency structure, it is also able to recognize the edges properly. Further examples in A.4
Therefore,
4.3 SUPER-RFSOLLUTION
While super-resolving images, SPDER preserves the frequency structure of the training image while introducing only the minimal necessary higher frequencies required for smooth interpolation (Fig. 6). One might believe that chaotic function like sin(x) / |æ/ sin(x). +log(==)) oscillates far too irregularly to interpolate meaningfully between points it has been trained on; however, we'r ve empirically determined this is not the case (Fig. 7). Further details and examples in A.5
4.4
AUDIO REPRESENTATION AND INTERPOLATION
To the best of our knowledge, SPDER's s performance is state-of-the-art for neural audio representation. Interestingly, for our experiments on audio, we found the optimal 8 to be arctan. Our findings are in 2 We also test SPDER's performance in audio interpolation. Further details in A.6
4.5 VIDEO REPRESENTATION AND FRAME INTERPOLATION
We show how SPDER can represent video as well, achieving PSNRs ~30, which indicate good quality reconstructions. We then asked our overfit SPDER network to interpolate to frames it hadn't directly been trained on, and it could do so with plausible results. This is exciting because indicates that SPDER is learning the true frequency structure of the video. For example, if it were biased towards lower frequencies, the interpolations would be blurry or blocky. If the bias was towards higher frequencies, we would grainy frames. Further details and examples in A.7
8

---

Published as a conference paper at ICLR 2024
SPDER Super-resolution 8192x8192
256x256 Ground Truth
GT
SPDER SIREN
Figure skimage data retina 256x256 ground truth (left) and SPDER super-resolution of it to 8192x8192 (center). Note this has 1024 pixels, and yet SPDER does not introduce any additional noise. Moreover, the output image has 201 million parameters, while the SPDER has less than 330,000 parameters (a 99. 99.84% reduction). The detail around blood vessel is shown for the ground truth (right above). SIREN and SPDER are trained for the same number of steps on the ground truth and then asked to super-resolve to 8192x8 192 (right center and below). Without any priors, SPDER still visibly clearer and creates a plausible reconstruction.
Architecture
Metric
Step
ReLU
ReLU w/ FFN
ReLU w/ PE
SIREN
SPDER
25
0.3718100.56
0.369410.56
0.0298110.44
0.0172:10.03
0.0135:10.02
100
0.0294110.04
0.0294:10004
0.029110.04
0.0084:10.02
0.0052-10001
Loss (L)
500
0.0292:10.04
0.0292:10004
0.028410.04
0.002119e-3
0.000913e-3
1000
0.029110.04
0.009
0.0279:10.0
0.000711e-3
0.0003-22-3
0.0378++0.09
0.0388+0.10
0.0947+-.11
0.5740+0.30
0.7276-0.25
100
0.0928++0.11
0.0934+-0.12
0.1144+0.11
0.7744+0.27
0.8843-0.19
PAG (t)
500
0.1026110.2
0.1032:1.12
0.194410.12
0.928810.17
0.9699=0.00
1000
0.103111.12
0.103710.11
0.2676.10.13
0.964710.12
0.9863-0.06
Table 2: ESC-50 audio average representation loss and firequency similarity (pAG) with standard deviations over various training steps. Sample sizes: ReLU (N=779), ReLU w/ FFN (N=787), ReLU w/ PE (N=747), SIREN (N=795), and SPDER (N=637). We note that the majority of correla- tions we recorded for SPDER by step 1000 were above 0.99, indicating near-perfect reconstruction of the audio (see A.6.2..
CONCLUSION AND FUTURE WORK
In this work, we presented simple and intuitive MLP architecture that can be used to represent images with 10,000-100,0000 lower loss than current state-of-the-art methods without the need for priors or augmentations. We show its generalizability in representing audio, video, gradients of images, etc. We also believe our architecture's 's automatic balancing of the positional and value information of inputs can offer new insights into learned positional embeddings. Likewise, studying how SPDER overcomes spectral bias may help learn how to improve neural network performance generally. Through our interpolation experiments, we also show that SPDER preserves the frequency spectrum for points it hasn't 't seen and, in sense, has low spectral "variance" well.
The most reasonable next steps include combining hierarchical methods with SPDER, where the network learns each small block extremely accurately, then aggregates them for near- near-perfect representation efficiently. Eventually, as neural hardware becomes more commonplace and advanced, the use of neural networks to compress and represent media will become increasingly attractive. The ability for INRs to capture frequency structure efficiently will lead a whole host of applications where information is stored in the weights of network rather than in bytecode.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The second author of this paper was supported in part by funds provided by the National Science Foundation (under grant 2229876), the KACST-UCB Center for Secure Computing, the Department of Homeland Security, IBM, the Noyce Foundation, Google, and Open Philanthropy. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
D. Amir and Y. Weiss. Understanding and simplifying perceptual distances. In 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12221-12230, 2021. doi: 10.1 1991CC/44444 2021.01205
Bai, J. Z. Kolter, and V. Koltun. An empirical evaluation of generic convolutional and recurent networks for sequence modeling, 2018. URL httpt:tttaaaatttit org. http//bbs/18...00 01221
J. Ballé, V. Laparra, and E. P. Simoncelli. End-to-end optimized image compression. Computer Vision and Puttern Recognition, 2016. doi: 100550/ARXIV.111.0011
01704
J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. MarrinnBruulla, and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. CoRR, abs/2103. 13415, 2021. URL
https://arxivy sttts://aax....orrra//////1/
Cao, Fang. Y. Wu, D.-X. Zhou, and Q Gu. Towards understanding the spectral bias of deep learning, 2020.
H. Chen, B. He H. Wang, Y. Ren, S. Lim, and A. Shrivastava. Nerv: Neural representations for videos. CoRR, abs/2110.133033, 2021. URLhttps:://
13903
H. Chen, M. Gwilliam, S.-N. Lim, and A. Shrivastava. Hnerv: hybrid neural representation for videos, 2023.
Z. Chen, Y. Chen, J. Liu, X. Xu, V. Goel, Z. Wang, H. Shi, and X. Wang. Vidcoinr: Learning vidco implicit noural representation for continuous space-time super-resolution, 2022.
S.-F. Chng, Ramasinghe, J. Sherrah, and S. Lucey. Garf: Gaussian activated radiance fields for high fidelity reconstruction and pose estimation, 2022. URL /tt////tsss///aaar...oo/////////
H.J. Gonzalez. Fourier transform of derivative, wttph:tpptttttttcttt http:s///maahheee (30588
com/ https//..cco/eesetionooooo questions/
2013. Accessed: 2023-05-15.
Hao, A. Mallya, Belongie, and M.-Y. Y. Liu. Implicit neural representations with levels-of-experts.
Oh, A. Agarwal, Belgrave, and Cho, editors, Advances Neural Information Processing Systems, 2022. net fom 55l
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks, 2020.
G. Kc, D. He, and T. Liu. Rethinking positional encoding in language pre-training. CoRR, httw/2006.15599 2020. (RLt/t::t:/ .wgg abs/ abs/a/as//.2000 15595
M. Li, W. Zuo. S. Gu, D. Zhao, and D. Zhang. Learning convolutional networks for content -neihtt-weightdd image compression, 2017. URL httpt::///atttt /amps://aaxiin http://xxvvvoor... httpss///rrxi..orr//bbb//// 703. 10553
E. Lieskovská, M. Jakubec, R. Jarina, and M. Chmulík. A review on speech emotion recognition using deep learning and attention mechanism. Electronics, 10(10), 2021. ISSN 2079-9292. doi: 10.3390/ clectronies10101116 URL URL hhtpt:://////
wwwp. wtt.c//m////// 00107999992/0010/1100
S. Luo. S. Li. S. Zheng, T.-Y. 2022. URL btttp:://axxi... awwivv ozg/ abs https//2....1311
Y. Liu. L. Wang, and D. He. Your transformer may be powerful you expect.
Mildenhall, Srinivasan, M. Tancik, T. Barron, Ramamoorthi, and R. Ng. Nerf Representing scenes as neural radiance fields for view synthesis. CoRR, abx/2003.08934, 2020. URL
D. Morin. Waves and fourier transform: Lecture notes, 2021. URL hwt.:htt/ssshhhoo/eateto http://sooolar..aaaad..
files welee//wavs_forrrr fourier .f
Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash
encoding. ACM Transactions on Graphics, 41(4): 15, jul 2022. doi: 10.1145/55282233333111 URL
https://aco wtt.://.....1123522222222222
3530127
T. Painter and A. Spanias. Perceptual coding of digital audio. Proceedings of the IEEE, 88(4):551-51,, 2000. doi:
J. Park, P. Florence, J. Strauh, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation, 2019. URL h0t0:////.....p.//rrrvoooor////a
11

---

Published as a conference paper at ICLR 2024
N. Rahaman, Baratin, D. Arpit, F. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio. and A. Courville. On spectral bias of neural networks. Machine Learning, 2018. doi: 10.48550ARXXIV1106... 080.34334. URL
Ramasinghe Lucey. Beyond periodicity: Towards unifying framework for activations coordinate- mlps, 2021. URL https: wwtp://www.w.aaaaaaaaooooo 1
Simeonov, Du, A. Tagliasacchi, Tenenbaum, Rodriguez, Agrawal, and V. Sitzmann. Neural descriptor fields: -quivvaqui object representations for manipulation. CoRR, abs/2112.00112. 2021. BRLhhttps://azxx
httgg/bb//230 05124
Sitzmann, J. N. P. Martel, A. Bergman, D. B. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions, CoRR, abs/2006. 09661, 2020. htpp:://///// 2006.09661 09661
Y. Strümpler, J. Postels, R. Yang, L. van Gool, and F. Tombari. Implicit neural representations for image compression, 2021. URL attps::/arxlv.oor/abbb. / 22122211111. 04267
M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich- Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains, 2020.
httggg//b//200000 10739
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkorcit, L. Jones, A. N. Gomez, L. Kaiser, and Polosukhin. Attention all you need. CoRR, abs/1706.03762, 2017. URL http:://axxi..or//aaaa
03762
G. Wallace. The jpeg still picture compression standard. IEEE Transactions Consumer Electronics, 38(1): xviii-xxxiv, 1992. doi: 10.1 1009/0. 25072.
T. Wiegand, G. Sullivan, G. Bjontegaard, and A. Luthra. Overview of the h.264/ave video coding standard. JEEE Transactions on Circuits and Systems for Video Technology. 13(7):560-55) -76 2003, doi: 10.1109/TCSVT.2003. 815165.
B. Wronski. Comparing images in frequency domain: Spectral loss does it make sense?
https:
bartwronski. com 200220202100/// 2021. Accessed: February 28, 2023.
Z.Q. Xu, Y. Zhang, and Xiao, Training behavior deep neural network frequency domain, 2018. URL
/ttps::/arxxv..org// awtpr//////////// abs/1807 31251
Yang, S. Mandt, and L. Theis An introduction neural data compression, 2022. URL UtLbsttp:://an
org/abs/ abaa/bb2/222 06533
Yüce, G. Ortiz-Jiménez, B. Besbinar, Frossard. structured dictionary perspective on implicit neural representations, 2021. URL httppp/s///.....rr/////////222..
Zhang, Isola, A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586 595, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society, doi: 10. 100/CCPP P21P 20182001800000 0006.00 htttttss:: //cp:::://////io/// iesecomputersoci wtw.soiciitt........ orw.1/..rr00000........ http://pppppprroooo.o...oo......
20 60068
W. Zhuang, T. Hascoet, R. Takashima, and T. Takiguchi. Optical flow regularization of implicit neural reprosentations for video frame interpolation, 2022. hwr./ittpps//aaxx...or//////a/
10886
12

---

Published as a conference paper at ICLR 2024
A SUPPLEMENTARY MATERIAL
A.1 PROOF OF LIPSCHITZ BOUND
As stated in the main paper, for activation functions described by the expression in Eq. (3) the local Lipschitz constant is bounded by ó is differentiable. We demonstrate this by deriving the lower and upper bounds of the derivative of sin(z) ((x) and showing equality between the two. For the proof, we will assume that is differentiable across all points barring measure zero set, which means in practice, we are not concerned with singularities in Vć since they are highly unlikely to occur.
in the neighborhood around input a E. when
Proof. Let [(a) be the local Lipschitz constant of nonlinearity around input point We interested in L evaluated within a neighborhood N of a as it is good indicator if the function oscillates too much for any given input. Ideally, L would grow slowly. Note that bounding L is lantamount to determining the magnitude of the derivative of the nonlinearity.
L(a)=
max (()
(ssin(>) 6(x))
(9)
max (())
(cos(x) - ((x) + sin(x) Vo(x)]
(10)
max (()(a)
(cos(x)) 5(x) + |sin V6(z))
(11)
Since both cos(x) and sin(x) are bounded by [-1, 1J. we can write
I(a)= S max xEN(a)
10(z) Vo(x))
(12)
Note that the absolute value operator simply applies constant factor an input and that the derivative of a differentiable function is bounded by constant factor over sufficiently small N(a).
Therefore, we conclude that L(a) E ((((a) + Vo(a)).
Likewise, we wish to determine the lower bound. Since both cos(x) and sin(x) are bounded by [-1, 1J. we can assign them to the constant values and c22 respectively.
((a)=
max |cos(x) -8(x) + sin(z) v5(z)]
(13)
xEN(a)
max |c| ..(x(())))))) + vō(=) *EN(a)
(14)
In the worst-case scenario where either ea C2 is 0, have that
I(a) 2 max (le, 5(a)), le2 Vō(a)))
(15)
We note that event that this occurs with a probability measure of 0, but lower bounds all other scenarios. Note that max(a, 6) E O(a+b), b). Therefore, it implies L(a) E (28(()) + Vo(a)). Because both the lower and upper bounds are equal, the Lipschitz constant L evaluated near a point is in ((((a) + vō(a)).
A.2 PROOF OF RELATION BETWEEN FOURIER TRANSFORM OF AN IMAGE AND ITS GRADIENT
The proof is adapted from Gonzalez (2033)
Proof. Let ftt) be signal with Fourier transform F(w). In the frequency domain, the signal f(t) can be represented as follows:
~00
((() = 2 -00
F(w)eeuttde
(16)
13

---

Published as a conference paper at ICLR 2024
Applying the gradient operator to ftt), obtain:
2n [
vif(t)= - V. 2 F(w)et"rduy
dw
(17)
Therefore, the Fourier transform of f() (w), establishing the desired relationship between the Fourier transform of signal and its gradient. This implies that knowing the frequency rep- resentation of signal enough to deduce the frequency representation of the gradient of that signal.
A.3 ACTIVATIONS
As can seen in Fig. 8 the outputs of the linear layers are normally distributed. This phenonemon originally observed occur for SIREN, where (itzmann et al. (2020) claim that this proves how the architecture does not suffer from vanishing or exploding gradients.
The outputs the activations in each layer cluster around the stationary points of sin(x) | Note how in the earlier layers of the network, the activations spread over wide domain, but by the later layers, they are concentrated in a few stationary points near x = O. We speculate that this demonstrates "eearning"/feattii selection, where the network first takes in wide range of frequencies and then aggressively distills it into the most important components.
A.3.1 ACTIVATION DISTRIBUTIONS FOR OTHER S'S
We include the distribution of activations for ((x) = log([=[)) (Fig. 9 and 6(x) = arctan(a:) (Fig. 10p When the has large span, these activations peak many values, and therefore be somewhat representative of the magnitude of the input that went into it. For example, in Fig. [ an input into neuron has many possible "peaks" choose from, so if it has higher input magnitude, it will likely be mapped to higher output magnitude. In Fig. 10 it's mostly going to be either 0 or 1.04, and the exact input that went into would be very hard to recover. This is because sin(x) arctan(z) has distinct stationary points near the origin, which then all converge to constant values sufficiently far from However, the stationary points for sin (a) log([z[) keep growing without bound.
14

---

Published as a conference paper at ICLR 2024
Actettion Detrituion
Actination
Activattinn Eeprrbutte ofneet)
Activation Jatriction net 2inner
Activation Ditribution of not
Acthation Datitution ret
Activation Dutribtten ofnst
Actiation Jutristion net A Annar
Aciivtion Ustr butten nft4
Acthvation Ditiftution of ret Sinter
Arthationn Ditritutonn dfnet.
Astrnation Dblt bulkon of maLb
Figure 8: Distribution of activations of each layer of a 5-layer SPDER with 8(z) Va trained on skimage, data. namera (). The output of the linear layers are on the left, and the outputs from the neurons in each layer are on the right, which peak at the stationary points sin(z) V| (notice their resemblance to spider, the architecture's namesake). The activations of the final output shown at the bottom.
15

---

Published as a conference paper at ICLR 2024
Activation Distribution for 6(x) (xg(
4000
3000
Count
2000
1000
0
-2
2
Activation value
Figure 9: The activation values distinctly peak at local extrema of sin(x) Iog([x|) at #0.36, 00.64, 1.56 +2.06, etc.
Activation Distribution for 6(x) arctan(x)
3500
3000
2500
Count
2000
1500
1000
500
-2
-1
2
Activation value
The activation visibly peaks at and 1.04 here, although the stationary points of arc sin(x: technically approach - and in either limit. This is because the stationary points nearest = O, where most of the inputs to neurons are, occur at and 1.04. Beyond that, they're distributed extremely close to - and _. See Fig. 2 for visual of the activation.
Figure 10:
16

---

Published as a conference paper at ICLR 2024
Gradients of SIREN Representation at Step 1
Gradients of SPDER Representation at Step 1
Gradients SIREN Representation Step 50
Gradients SPDER Representation Step 50
The gradients of the representations after training on the image from skimage data. camera () for single step shown (top). Interestingly, using only sine activation (top left) yields few, thick gradients the beginning. Adding a damping factor (top right) enables the representation to start off with hundreds of small strands which presumably align around detail at any frequency much more accurately. We believe this is because of the semiperiodic nature of the activation function, where a continuum of frequencies can be encoded by single neuron, whereas sinusoidals encode some number of discrete frequencies.
Figure 11:
A.4 IMAGE GRADIENT REPRESENTATION
Visuals are in Fig. and Fig. 12 The intensity of each pixel in the resulting gradient image corre- sponds how much small change in each input coordinate value would affect the corresponding pixel prediction. Areas such as edges and other high-frequency components exhibit high gradient values.
17

---

Published as a conference paper at ICLR 2024
Ground Truth
Gradients SPDER Representation Step
Gradients SPDER Representation Step
Gradients of SPDER Representation Step
Gradients of SPDER Representation at Step 50
Gradients of Ground Truth
Figure 12: Gradients of SPDER representation the image from skimage data bbrick () over various training steps. Note how SPDER can encode straight edges (low frequencies) incredibly well.
18

---

Published as a conference paper at ICLR 2024
Super-resolation Factor (SRF)
Metric
Dataset
2x
4x
8x
SIREN
SPDER
SIREN
SPDER
SIREN
SPDER
DIV2K
2e-2:18e33
6e-3:13e-3
2e-2:196-3
76-3:3e-3
2e-2:19-3
86-3d3e-3
flhe0
96-3-4o-3
30-3d:le33
8e-3d33-3
36-3:te-33
9e-313e-3
3e-3die-3
MSE (J)
Ahyl
9e.3+t-5c3
4e:34-3e-3
Se-34-Se-3
3e-3++3e-3
8e-3+-5es3
4e:34:3e-3
Mhq2
1e-2488-3
4e-34:2e-3
1e-248e-3
3e-3:t
1e-248e-3
3e-3::2e33
Flickr2K
4o-243o-3
1e-249e-3
4o-2:33-2
20-2:tlo22
4o-2-33-3
2o-2dle-2
DIV2K
23.2:t:199
29.4::2.6
22.8:2000
28.0+:2.4
22.8:42.00
27.74:2.3
flho0
26.71:1.
31642.0
27.0:1:.6
32,242.0
27.01:1.6
318119
PSNR (1)
Abyl
27,34:2.3
31,242
27.84:2,44
31.9:22555
27.7::2.4
31,44:266
fhqq
25.3::2.7
30.5422.1
25,8:12.7
31,32199
25.74:2.7
31.02.00
Mickr2K
23.2--2.0
25.6++3.0
20.8++2.7
25.1+-2.8
20.8+22.7
24.94+2888
DIV2K
0.99165-:6e-3
0.9975143e-3
0.99110-H6e-3
0.99712++3-3
0.99108-+6e-3
0.99702-43e-3
0.99799-+1e-
0.99942444e44
0.99823+11:5
0.99956+33e44
0.99819+1es3
3.99953+-3e44
PaG (t)
fhqgl
0,99835:18-4
0.99949:42e-4
0.99856:77-4
0.99962422-4
0,99852:7c-4
0.99958:2e-4
fthq2
0,99775=1e-3
0.99920:15e-4
0.99812-t1e-3
0.99945=3e-4
0.99807+1e-3
0.9994113e-4
Flickr2K
0.96332:33e2
0.9889111e-2
0.96324:14e-2
0.98825:11e22
0.96320-14e22
0.9879711e-2
Table 3: Super-resolution metries with standard deviations over SRFs of 2x, 4x, and 8x. The base resolution was 512x512, and the super-resolutions were at 1024x10244 2048x2048, and 4196x4196, respectively. We chose three different subsets of the Flickr-Faces-HC dataset (each corresponding to a different category and, therefore, distribution) denoted as ffhq0, ffhql, and fthq2. As stated before, by simply applying damping factor, the super-resolution quality goes up significantly. Note how the PSNRs are ~ points higher compared to SIREN in each subcategory. For implementation details, see A.9.4
A.5
In this section, we wish to highlight one application of SPDER's success in capturing the frequency structure: generating higher resolution detail without the need for priors meta-learning.
We noticed that when interpolating from base resolution 512x512 or higher, no noise visible to the naked eye is introduced into any segment of the super-resolved reconstruction at any super- resolution factor. At base resolutions below that, some noise visible when interpolating by 4x more, presumably because the SPDER has too few points meaningfully learn the frequency distribution. We note this still impressive, given that SPDER networks have no outside prior on pixel distributions.
A.5.1 RESULTS
We include quantitative results of the comparison between SPDER and SIREN in Table 3 Inter- estingly, SPDER can superresolve to factor of 8x (64x pixels) and still have high reconstruction quality consistently. Most of the PSNRs of SPDER's super-resolutions quality. We once again highlight how simply applying a damping factor to nonlinearity boosts performance noticeably.
above 30, indicating good
A.5.2 EXAMPLES
Visuals in Fig. B and Fig.
19

---

Published as a conference paper at ICLR 2024
512x512 GT
4096x4096 SPDER
Super-resolution
GT
Bicubic
SPDER
Figure 13: Image from skimage. astronaut () at 512x512 ground truth (top) and SPDER super-resolution
it to 4096x4096 (center). (Bottom, left to right) The detail around the eye is shown at the 512x512 ground truth, bicubic interpolation 4096x4096. and SPDER's super-resolution to 4096x4096, respectively. Note how compared to bicubic, SPDER is visually sharper and traces out the eyelids better.
20

---

Published as a conference paper at ICLR 2024
256x256 GT
1024x1024 SPDER Super-resolution
GT
SPDER
Figure 14: Image from skimage. coffee () at ground truth resolution 256x256 (top) and SPDER super-resolution to 1024x1024 (center). Edge of cup for ground truth (bottom left) and SPDER super-resolution (bottom right) are also shown.
21

---

Published as a conference paper at ICLR 2024
Architecture
Metric
UF
SIREN
SPDER
2x
0.00719=0.014
0.00522+0.012
MSE (L)
4x
0.00959=00017
0.00755++00015
8x
0.00959=0.017
0.0087410.017
Table 4: Audio interpolation results: mean-squared error over various audio upsampling factors (UFs) with standard deviations over UFs of 2x, 4x, and 8x. Across all upsampling factors, SPDER's mean reconstruction error lower than that of SIREN's.
Distribution SPDER audio representation corelaaiins below 0.98 after 1000 steps
Distribution of SIREN audic representation correlations below 0.98 after 1000 steps
2.5
2.0-
Dersity
2.0
1.5
1.0
0.5
2.0
0.0
0.3
0.6
0.7
0.8
0.9
1.0
0.0
0.2
0.4
0.8
10
Correlation (pue)
Correlation (pui)
Figure 15: Distribution of Pas 's below 0.98 for SPDER and SIREN audio representations. We drop the top out of 100 bins for both SPDER and SIREN (dropping 89.4% and 81.4% of the total count, respectively) to make these bars visible. Note how SPDER has a noticeably sparser left tail, meaning can represent many challenging examples SIREN cannot.
A.6 AUDIO REPRESENTATION AND INTERPOLATION
A.6.1 INTERPOLATION
Here, we show SPDER's metrics on reconstructing audio clips even after being trained on a small fraction of the entire clip. This is promising because it demonstrates that applying damping indeed preserves the frequency structure not just for 2D images but ID audio better than without having it.
We train an INR on an 8x downsampled version of entire ground truth audio clip. The model then has to predict the values at the upsampling factors (UFs) of 2x, 4x, and 8x, where the UF is how many times larger the inference sample is than the training one. For example, the 8x upsampled version simply the ground truth (GT), 4x UF corresponds to GT downsampled by 2x, and 2x UF is GT downsampled by 4x. See further implementation details in A.9.6
Quantitative results are in Table 4
A.6.2 AUDIO DISCUSSION
We observed that in both SIREN and SPDER, the vast majority of reconstructions had high correlation with the ground truth, with the peak in the 0.99-1.00 bin. Both architectures also had handful of pathological samples they were not able to reconstruct accurately. However, the distribution of PAGG for SIREN has noticeably fatter left tail, meaning it has much more difficulty representing some samples that SPDER has no problem with (Fig. 15).
In general, the performance audio was strongly impacted by small percentage of pathological (highly noisy) outliers in our dataset. We chose not to exclude these from our results to keep the data consistent. However, in the real world, assuming the audio clips intelligible and not full of noise,
22

---

Published as a conference paper at ICLR 2024
Log training losses bunny video
treining Iosoes on siees video
20
18-9
155
250
150
250
192
Steps
PSNR video
PSNR bikes
27.5
5935
23.0
22.5
31.5
125
12.9
Stezs
Figure 16: Log training loss and PSNR curves for two videos for SPDER and SIREN. Note how SPDER's PSNRs are consistently ~9 points above that of SIREN's (bottom left and right). This illustrates how the simple idea of damping can succeed on non-spatial features (i.e. video frames). There is performance difference between the two examples because the bunny video has 20% more frames than the bikes video.
SPDER would perform even better than what we present in these results. Training for more steps or using a larger network would also likely lessen the impact of these outliers.
We acknowledge that, ideally, SPDER should not have problems with any samples (however anoma- lous). Studying these challenging samples may be the basis for future work.
A.7 VIDEO REPRESENTATION AND FRAME INTERPOLATION
First, we trained a SPDER on a portion of Big Buck Bunny, a standard video used in computer vision research. SPDER achieved a peak PSNR of 29.74 within 450 steps, indicating good quality reconstruction. Note how SPDER's 's representation also serves a lossy compression scheme because the network has 12 million parameters, but the original video has ~0 million. Second, we trained a 256x256 resolution clip from sk mage. data a b kes () and it achieved a PSNR of 30.9 within 450 steps. The performances plotted in Fig. 16
For the frame interpolation experiments, we trained on the bikes video at 30 FPS, and then asked the model predict what it would look it if had twice as many frames (i.e. 60 FPS).
A.7.1 VIDEO REPRESENTATION EXAMPLES
The examples, from skvideo datasets bikes (), are in Fig.
A.7.2 FRAME INTERPOLATION EXAMPLES
Examples are in Fig. 18 and Fig. 19
23

---

Published as a conference paper at ICLR 2024
Ground Truth of Frame 25
Ground Truth of Frame 120
SPDER Reconstruction Frame
SPDER Reconstruction of Frame 120
Figure 17: This 250-frame video from skvideo. www.daa.at.iiii. () is fairly chaotic with many moving parts, yet SPDER still able capture without any artifacts. Here, we chose frames with overall low and high frequencies (left and right, respectively) to contrast. SPDER evidently has no bias and can encode both accurately.
24

---

Published as a conference paper at ICLR 2024
Ground Truth of Frame 199
Ground Truth of Frame 200
C
SPDER Reconstruction of Frame 199.5
Figure 18: We selected two consecutive frames where movement would be perceptible possible. We can see a person's 's leg move partially behind a pole in the ground truth frames (top left and top right). SPDER's interpolation for the intermediate frame is indeed plausible and doesn't 't include any visual distortions not already present in the ground truth.
25

---

Published as a conference paper at ICLR 2024
Ground Truth of Frame 150
Ground Truth of Frame 151
SPDER Reconstruction of Frame 150.5
Figure 19: We selected two relatively still consecutive frames and demonstrate how SPDER's inter- polation does not seem visually grainier blurrier (i.e., no higher or lower frequencies introduced) than in frames from the ground truth distribution.
26

---

Published as a conference paper at ICLR 2024
A.8 COMMENTS ON INSTANT-NGP
Given the hierarchical setup and CUDA C++-tailored implementation, Instant-NGP has more com- plexity and a significantly large overhead compared SPDER. We tried running Instant-NGP on several machines, but due to insufficient system requirements, all but a machine with 8 Quadro RTX 6000's (costing over $20,000) failed to run it. Although it offers an impressive speedup compared to other architectures, we found the system requirements to be quite prohibitive compared to using simple model like SPDER.
That being said, it is also much more efficient and extremely useful for most high-information media. For example, it was much faster than SPDER during training, taking about ~55 seconds for 500 steps on single 256x256 image, whereas SPDER took 15 seconds on the same machine. Likewise, it allows for training on objects such as gigapixel images within minutes, which would otherwise take a few hours with the current iteration of SPDER.
To the best of our understanding, Instant-NGP learns a piccewise-lineal approximation of the signal and is prone to pseudo reeudorrandom hash collisions (Müller et al. 2022), so although it achieves very low loss, it technically cannot model the high-frequency detail as well as SPDER. We were unable to retrieve its reconstructed image and therefore could not calculate paG or use it for super-resolution, so we emphasize that any claims we make regarding Instant-NGP's frequency alignment are purely based on our theoretical understanding of it.
We believe that Instant-NGP attempts to overcome spectral bias by having many learned parametric encodings which are responsible for very small receptive field in the image. Each field has relatively low complexity, which means they can get away with using augmentations like ReLU and bilinear interpolation to approximate the signal precewise-linearje However, in SPDER, each neuron has the receptive field of the entire image, which means in order to represent it well, it is forced to learn the true frequency structure. To the best of our knowledge, our architecture is the only one that naturally learns the true frequency structure of an image and overcomes spectral bias- without hardcoded inductive biases like hierarchies. That being said, we believe combining a hicrarchical approach with SPDER is the next reasonable step.
A.9 REPRODUCIBILLTY
By default, we use 5-layer network with 256 neurons in each layer and learning rate of 1 x 10-4, Input and output values are scaled to be in [-1, 1]. As we wish to completely "overfit" to the training data, use full batch gradient descent (i.e., do not want the noise from SGD) on each object. We discovered that clamping the input of sin(x:) / [ to a tiny value (less than 1 10-30) avoided division by zero errors on the GPU while maintaining overall functionality. For any resizing operations, we use Py PyTorch's built-in torchvision transforms Resize module which uses bilinear interpolation under the hood.
We used two GeForce GTX 1080 Ti's with 12 GB of memory each. We note that all SPDER experiments can be run on personal laptops and require no tailored hardware.
Please refer the full source code for reproducing the experiments in the supplementary material.
A.9.1 WEIGHT INITIALIZATION
We followed the weight initialization scheme from Sitzmann et al. (2020). Weights are initialized fan_in where fan in is the number of inputs to a layer. They claimed this
from 24 fan in
leads to the input of each activation being normally distributed and keeps gradients stable, which we indeed observed in SPDER (Fig.
Also consistent with their implementation, we multiply the input into each activation by a predeter- mined Wo. which is always 30.
A.9.2 IMAGE REPRESENTATION
Recall that for INRs, the training data are the pixels from a single image; the model's goal is to overfit it. The training process can be viewed as "encoding" step, where the data from an image is loaded
27

---

Published as a conference paper at ICLR 2024
onto the weights of the model. The "decoding" step is simply single forward pass of the model, where the information from the weights is mapped onto a prediction at each pixel coordinate.
For these experiments, sampled images from DIV2K, high-quality super-resolution dataset. Each image was resized to a resolution of 256x256 and fed into an untrained network, which would then train it for fixed number of steps. For simplicity, only one channel of each image was included, but note that representing all three is trivial. The loss, peak signal-to-noiss ratio (PSNR), and PAG between the overfit model's i representation after training and ground truth were recorded for training steps 25, 100, and 500. Loss is the mean-squared error between the model's prediction compared the ground truth over each pixel. Recall that the values come from a scale of 1-1, 1] (A.13). PSNR is calculated through Eq. (23). PAG comes from Eq. ( and is simply the cosine similarity between the amplitude spectrum of the model's 's reconstruction and the ground truth image.
For Instant-NGP. we had to use much more high-end machine (A.8) to collect data. Note that for ReLU with FFN, the method requires manual tuning of scale hyperparameter, so perhaps a configuration better than a standard Gaussian exists, but it was not apparent to even after several attempts. SPDER, on the other hand, requires no such manual tuning for different datasets.
A.9.3 GRADIENT REPRESENTATION
For gradient representation, we overfit a network to an image using the same hyperparameters as before. The gradient of the reconstructed image is then computed. Specifically, we use the PyTorrh function torch. aut ograd. grad to calculate the gradient of the output of the model (i.e., the reconstructed pixels) with respect to the coordinate inputs.
A.9.4 SUPER-RESOLLTIOO
For these experiments, we utilized three datasets DIV2K, Flickr2K, and Flickr-Paces-HQ. DIV2K, standard benchmark for image super-resolution, comprises 2K resolution RGB images. Flickr2K is a diverse dataset of high-resolution images sourced from Flickr, widely used for training super- resolution models. Lastly, Flickr-Faces (FFHQ) is dataset of high-quality PNG images at 1024x 1024 resolution, employed to test model performance on detailed, structured imagery like human faces. We test images from each dataset on three super-resolution factors (SRFs) 2, 4, and 8. For each combination of SRF, dataset, and architecture, we use N = 12 images from each dataset.
For training on each dataset, we first sample image, select single channel, and then resize it down to 512x512 (the base resolution). We overfit the model (either SIREN or SPDER) to the resized image by training it for 100 steps (by when both models' reconstructions reasonably converge). Using the trained model overfit to the base resolution version of the image, we then ask the model to predict what the image would look like at a resolution that 2x, 4x, or 3x that of the base resolution. Recall this is possible because INRs learn continuous representation for an object, and can therefore be queried at any input coordinate value within a reasonable domain. To compute scores on our metrics, we take the original image along the single channel and (if needed) resize it at the given super-resolution to get the ground truth. We calculate the metric between the model's 's prediction and the ground truth at the given super-resolution, using the same procedure as for that of image representation.
A.9.5 AUDIO REPRESENTATION
Consistent with the implementation by Sitzmann et al. 2020). we 1) scale the input grid to have bounds 100, 100] instead of [-1, 1]. which is tantamount to scaling the weights of the input layer by 100, and 2) use learning rate of x 10-5
Each training sample was cropped to the first 7 seconds of clips from ESC-50, a labeled collection of 2000 environmental audio recordings, and then trained on for 1000 steps.
A.9.6 AUDIO INTERPOLATION
For each ground truth, sample a clip from ESC-50 and crop it to the first 4 seconds. We train the network (either SIREN or SPDER) on an 8x downsampled version of this ground truth clip for 250 steps. For inference, then take the original ground truth clip and downsample by 4x, 2x, and
28

---

Published as a conference paper at ICLR 2024
1x Note the overfit network will have seen only half of the first set of values, a quarter of the second, and an eighth of the third during training. The model then outputs a prediction for the ground truth audio clips across these resolutions, and the MSE is recorded. For each combination of architecture and resolution, we use = 30.
Unless stated otherwise, the rest of the implementation is the same that of for audio representation.
A.9.7 VIDEO REPRESENTATION
We used 12-layer 1024-neuron wide network with a learning rate of 5 x 10-6 for our video experiments. Each architecture (SIREN and SPDER) was trained for at least 350 steps, and the loss was recorded at each step. To speed up training, we selected single channel from each video and center-cropped then resized to 256x Training on a single video for SPDER took ~~6 hours on setup with 24 GB of compute.
The first video we trained on was from skvideo. dat aseasets s bi gbuckbunny O a standard video used in computer vision research. The first 300 frames were kept. The second video was from skvideo. datasets bikes () . The first 250 out of 300 total frames were kept.
A.9.8 VIDEO FRAME INTERPOLATION
We used the SPDER trained on the 250-frame video from skvideo. datasets bikes () from the representation experiments. While the original video was shot in 30 FPS, we asked the SPDER to predict what it would look like 60 FPS by inputting the same coordinate grid but 2x denser along the frame dimension. We then visualized the results.
Unless stated otherwise, the rest of the implementation is the same as that of for video representation.
A.10 LIMITATIONS
SPDER shows the most success on tasks where the input comes from coordinate grid. For instance, converges slower than NeRFs for novel view synthesis, presumably due to the non-affine nature of the view direction input. Future work explore preprocessing approaches to aptly project such imputs onto a coordinate space to potentially see the same success as for images.
When compared to SIREN, SPDER's 's training is ~1.3x slower since it must calculate the damping factor for each input. While storing the gradients does consume 11,5 more memory, this issue can be circumvented through batch processing.
Determining the optimal 5 for media types other than image, audio, and video may require some experimentation.
A.11 IMAGE FFT FORMULA
In the main paper, we included the general formula for the Fourier amplitude spectrum of a discrete- time signal for vector input. However, the Fourier transform can also apply to tensors, which relevant in the case for images. Here, we include the formula for the Fourier transform of a 2D tensor:
M
N
Flu,
1
Σ
Σ
f(x,
12 (2 *
(18)
MN
Z=0
==0
which we used when evaluating the frequency similarity of images. In this formula, F(u, u) represents the Fourier transform of the 2D array f(z y) with respect to the spatial frequencies u and v. The size of the array is M by N. The summation is taken over all possible values of x and y in the array. The exponential term represents the phase shift each point in the array. Note that the method numpy. fft f can be used to calculate this efficiently, so we included this just for reference.
A.12 ABLATION ON VARIOUS 8's
We tested various forms on the image from skimage, , data. camera () over 250 steps in Fig. 20 The log-loss results are shown. Using 8(x) of ReI.U(z)) or just performs extremely poorly,
29

---

Published as a conference paper at ICLR 2024
Log training losses on image
10-1
10-2
Masa
Loss
10-3
Vmmm
m
ReLU(x)
10-4
log([x()
arctan(x)
10-5
sqrt([x()
50
100
200
250
Steps
Figure 20: Log training losses on the cameraman image for various 0's
and the loss curves are cropped off from the top because the losses are so high. The performance of the other 8's in order from best to worst is vlzl. log([z:), arctan(z). and then 1 (which simply sinusoidal, used in SIREN).
The loss curves in Fig. 20 for SPDER and SIREN look different from Fig. because we trained on a CPU with clamping here, but the overall trend remains.
A.13 CORRESPONDENCE TO 8-BIT SCALE
In our training setup, inputs and outputs are scaled to be in [-1, The mean-squared error (MSE) loss calculated on this scale well. To interpret this on an 8-bit scale (where pixel values are integers, with p E {0, 1.... 255}), we need to apply the following transformation on output value y E 1-111;
(y) + 1)- 255
1:y+P
f(y)
(19)
2
In practice, however, the bounds of the prediction values slightly diverge from [-1, 1], so the most reliable way to reconstruct images is through normalization:
(y) Vmii)) 255
f:yーp
f(v)
(20)
(ymaes Ymin)
If we want to calculate the MSE the 8-bit pixel scale, apply this transformation:
f MSE MSE f(MSE,) MSE, 255
(21)
where assume Dmae Ymnén In Table I demonstrate how SPDER has average training loss of 6.7 x 10 which corresponds to an MSE of 0,001 pixels? on an 8-bit pixel scale, so incorrectly predicted bits are extremely rare. No reconstruction at this resolution had any artifacts. according to our observations.
A.14 PSNR CALCULATION
The formula for peak signal-to-noise ratio (dB) is given by:
MAX?
PSNR 10 log10
(22)
MSE
30

---

Published as a conference paper at ICLR 2024
Note that outputs are scaled from [-1, 1] and have range of 2. Using the same MSE in training, can calculate PSNR as follows:
(1 (1-(-1))²
4 MSE
10 1og10
10 log10
(23)
MSE
Therefore, PSNR can be directly determined from the training loss (MSE) on each sample. We only include because of its interpretability.
31


Reviews:
[{"review_id": "4b9mSqsc07", "reviewer": "Reviewer_V9Ce", "summary": "This paper proposed a simple architecture to overcome the spectral bias towards lower frequencies in traditional neural network.\nIt formulated a so called damping function implemented by MLP. It contains an activation function composed of a sinusoidal multiplied by a sublinear function.\nThe sinusoidal enables the network to automatically learn the positional encoding of an input coordinate while the damping passes on the actual coordinate value by preventing it from being projected down to within a finite range of values.\nFurther experiments demonstrated good performance on multiple downstream tasks such as image super-resolution and video frame interpolation.", "questions": "Can you evaluate the proposed method on more complex images? Consider to perform formal evaluations on popular benchmarks, such as CelebA, UHDSR4K and Vimeo-90k to demonstrate its superiority.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The proposed method is novel.\n+ The introduced damping function is interesting and theoretical sound.\n+ The paper shows cases of multiple potential applications using proposed method.\n+ Experiments demonstrated that it can largely improve the performance and training efficiency in dummy setups.", "weaknesses": "- Do not prove the derivatives of different variants of proposed method.\n- It lacks experiments on more realistic datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rAuCGvofM5", "reviewer": "Reviewer_9hcy", "summary": "This paper proposes a novel family of activation functions to enable an MLP to represent an input signal (image, audio) to very high fidelity. Both the loss and the architecture are modified to enable this. A number of experiments are provided to show the benefits of the resulting model.", "questions": "There is a statement buried in page 4 that says: \"For example, they are less successful at novel view synthesis, where the input direction is polar.\". However, no examples of performance on novel view synthesis was provided. Can the authors comment on this?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is simply written and easy to follow (although slightly prone to hyperbolic language; pls see suggestion below). The provided experiments, while not comprehensive, paint a solid picture of the abilities of the model to represent signals in near-lossless manner. The method itself seems very simple to code (although no code was provided, and no training details given). Overall the paper has the strengths of simplicity and should be very easy to use in practice.", "weaknesses": "1. I am concerned that this method tends to overfit heavily and easily to a given particular input image/signal, and evidence for generalization was very limited to (as far as I can tell) a single video example. How sensitive is the model to noise in the input, for example? This can be important for real-world applications.\n\n2. I am not an expert in this space, but one question I have is whether the baselines are strong. SIREN is the most commonly used baseline. Maybe the authors can provide some perspective in this regard.\n\nLastly, I would recommend the authors, in the spirit of academic inquiry, to tone down the language a little bit. An example is a statement of the form: \"We demonstrate how SPDER not only has incredibly low loss\", can be restated as \"We demonstrate how SPDER not only has significantly lower loss\" etc. This is a minor point, admittedly my personal taste.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qall78493Y", "reviewer": "Reviewer_TKxh", "summary": "The paper proposes a novel architecture for coordinate-based networks used for signal representation. The key contribution of this architecture is that the activation function is changed to a sinusoidal function multiplied by a damping function. This result leads to improvement over existing activations which either use ReLU, and have a spectral bias towards low-frequency functions, or use sinusoidal activations, which do not have a notion of locality due to the lack of damping factor and thus lead to poorer reconstructions. This contribution is justified empirically by showing that it leads to significantly better signal fits on image signals.", "questions": "I do not have any additional questions on the paper, as it is described very clearly. Overall, I think that the contribution of the paper is strong due to its simplicity and demonstrated improvement on image-signal overfitting tasks. However, I think that the evaluations are limited - there are other coordinate-based network architectures which lead to improvement over the baselines which are compared to here, and most importantly, the method is not compared on radiance fields, which is the most impactful application of coordinate-based networks. I think that addressing these two weaknesses would make the paper significantly stronger and in that case I would be happy to raise my score.\n\n**Update after author response**\n\nAfter reading the author response, I am not inclined to change my score. While I appreciate the additional comparisons, and I believe the paper does a convincing job showing that overfitting image signals can be done extremely well with SPDER, I think the paper weakness is justifying *why* this is a relevant problem. The method seems to not work on radiance fields, has not been shown to work on SDFs, nor am I sure it offers compression benefits in overfitting image signals. Thus, I'm not sure what application it would be used in. On the other hand, there is no theoretical proof that it is able to encode a wider frequency band of signals, only empirical results on a relatively narrow range of experiments. With further work, I believe the paper could make a meaningful contribution, but in this state I remain borderline.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "In my opinion, the strengths of the paper are:\n1. The paper is presented exceptionally clearly. The activation function and possible choices are explained very well, and the reasoning for the chosen function is justified intuitively, and there are theoretical guarantees on the Lipschitz constant of the function represented by the network. I view this as very valuable as the paper can have more impact if researchers understand the reason for the increase in performance, as this is under-explored area of activation functions in coordinate-based networks.\n2. The comparisons show that the proposed SPDER activation function leads to significant improvement over positional encoding, SIREN, and ReLU networks in signal memorization tasks. This shows that the relatively simple contribution does lead to an improvement in what is claimed, and has the potential to be impactful.", "weaknesses": "In my opinion, the weaknesses of the paper are:\n1. I think that additional experiments could be used as there are other coordinate-based network architectures which claim improvements over positional encoding and SIREN architectures. I'm not sure why Instant-NGP is chosen here, as it is a hybrid implicit-explicit representation and is very different from the other coordinate-based networks. I believe a better fit would be [1] or [2], which show that simple networks with different activations, structures, or positional embeddings can improve on the FF/SIREN architectures. \n2. Following the additional experiments thread, I'm not sure why comparisons on radiance field fitting are not included. While interesting as a toy problem, image signal representation is not very useful in many computer vision tasks, besides maybe super-resolution. However, coordinate-based networks have been very useful in computer graphics, where they can be used in solving inverse problems and reconstructing 3D from 2D. Not including an experiment on radiance field, which is the most impactful field where an improvement in coordinate-based network fitting would be utilized, leads me to believe that the impact of the method is severely limited due to some unknown reason, perhaps speed or memory constraints in the architecture?\n3. Even simpler than radiance field fitting, I don't understand why there is no experiment on overfitting 3D shapes, such as SDFs? This is where coordinate-based networks originated, and could be used as a compressive representation. The paper mentions compressive signal representations in the conclusion, but there is no study on how effective SPDER is for this - no study on memory consumption, or size of networks needed to reach an acceptable level of quality. This is usually done when trying to represent 3D objects with SDFs, so this leads me to believe the method may have some limitations here.\n\n[1] https://openreview.net/forum?id=OmtmcPkkhT\n\n[2] https://arxiv.org/abs/2106.01553", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)