FreSh: Frequency Shifting for Accelerated Neural Representation Learning
Authors: Adam Kania
Marko Mihajlovic
Sergey Prokudin
Jacek Tabor
Przemysław Spurek
Venue: ICLR 2025 Poster (2025)
Keywords: spectral bias
automatic hyperparameter selection
implicit neural representation
discrete fourier transform
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Implicit Neural Representations (INRs) have recently gained attention as a powerful approach for continuously representing signals such as images, videos, and 3D shapes using multilayer perceptrons (MLPs). However, MLPs are known to exhibit a low-frequency bias, limiting their ability to capture high-frequency details accurately. This limitation is typically addressed by incorporating high-frequency input embeddings or specialized activation layers. In this work, we demonstrate that these embeddings and activations are often configured with hyperparameters that perform well on average but are suboptimal for specific input signals under consideration, necessitating a costly grid search to identify optimal settings. Our key observation is that the initial frequency spectrum of an untrained model's output correlates strongly with the model's eventual performance on a given target signal. Leveraging this insight, we propose frequency shifting (or FreSh), a method that selects embedding hyperparameters to align the frequency spectrum of the model’s initial output with that of the target signal. We show that this simple initialization technique improves performance across various neural representation methods and tasks, achieving results comparable to extensive hyperparameter sweeps but with only marginal computational overhead compared to training a single model with default hyperparameters.

Published as a conference paper at ICLR 2025
FRESH: FREQUENCY SHIFTING FOR ACCELERATED NEURAL REPRESENTATION LEARNING
Adam Kania', Marko Mihajlovic", Sergey Prokudin 2.3, Jacek Tabor', Przemyslaw Spurek Faculty of Mathematics and Computer Science, Jagiellonian University, Krakow, Poland"; ETH Zurich²; Balgrist University Hospital IDEAS NCBR
ABSTRACT
Implicit Neural Representations (INRs) have recently gained attention power- ful approach for continuously representing signals such as images, videos, and 3D shapes using multilayer perceptrons (MLPs). However, MLPs are known to ex- hibit low-frequency bias, limiting their ability to capture high-frequency details accurately. This limitation is typically addressed by incorporating fighffrequency input embeddings or specialized activation layers. In this work, we demonstrate that these embeddings and activations are often configured with hyperparameters that perform well on average but are suboptimal for specific input signals under consideration, necessitating costly grid search to identify optimal settings. Our key observation is that the initial frequency spectrum of an untrained model's s output correlates strongly with the model's eventual performance on given tar- get signal. Leveraging this insight, we propose frequency shifting (or FreSh), a method that selects embedding hyperparameters to align the frequency spec- trum of the model's initial output with that of the target signal. We show that this simple initialization technique improves performance across various neural representation methods and tasks, achieving results comparable to extensive hy- perparameter sweeps but with only marginal computational overhead compared to training a single model with default hyperparameters. The code is available at: https://qithun com/ qwwmmmmuuaaeeeeee FFeshh
INTRODUCTION
Implicit Neural Representations (INRs) advancing computer graphics research by integrating classical algorithms with continuous signal representations. They have been successfully applied in signal representation and inverse problems, with notable applications in neural rendering, compres- sion, and 2D and 3D signal reconstruction (Xie et al. 2022).
INRs primarily rely multilayer perceptrons (MLPs), making them susceptible spectral bias, which refers to the slower convergence of MLPs when approximating high-frequency components of the target signal (Rahaman et al.) 2019). Although spectral bias benefit generalization (Ronen et al. 2019), it can also hinder performance GGorii et al. 2023 Rahaman et al. 2019), especially in scenarios that require high precision of signal reconstruction, such as the training of INRs. This led to the development of numerous architectures aimed at overcoming spectral bias and its resulting capacity constraints by increasing the frequencies present in the input signal at the first layer of the model (embedding layer), e.g., through pequency-changing activation functions (Sitzmann et al.)
2020b Liu et al. 2024 Tancik et al., 2020) or auxiliary data structures (Chan et al.) 2022 Müller et al. 2022).
The frequency of such embedding layers is typically controlled by hyperparameters whose con- figuration significantly affect performance (see Figure I]. In section we show that default hyperparameter values can hinder performance and lead to blurry reconstructions. Improving per- formance is possible by using parameters sweeps, however, optimizing embedding parameters by training multiple models introduces significant overhead and is not feasible in practice.
The high computational cost of performing parameter sweep comes from training each model. Instead of training, we approximate model performance using the Fourier transform and the Wasser- stein distance (Wasserstein 1969), which was approximately 20 times faster than grid search in

---

Published as a conference paper at ICLR 2025
Siren +Grid Search (win = 90)
PSNR: 28.1
1422
PSNR: 1422
PSNR:
142
Ground Truth
Siren +FreSh (wip = (10)
PSNR: 28.5
1428
PSNR: 29.8
1433
PSNR: 33.5
Final PSNR
32
PSNR: 25.5
PSNR: 27.1
MIN
PSNR: 31.3
Siren (wl = 30)
50
100 150
Wo
Step O
Step 500
Step 1000
Step 5000
Figure 1: The configuration of embeddings is crucial for the convergence speed. We train Siren with various embedding configurations (wo E [10, 200)) for 5k steps on a Kodak image (top-left). The best grid-search found model (wi = 90). the FreSh configuration (wo 110) and the baseline (wo = 30) are marked with diamonds (bottom-left). The optimal and FreSh configurations (top and middle rows) lead sharper details, such as the number on the sail, compared with the baseline (bottom row). Even though Siren uses frequency embedding, the baseline is blurry due to low frequency bias. Note how the sizes of uniformly colored areas in the output at step indicate the size of image features the network can easily learn this observation is pivotal for FreSh.
experiments (see Table 4). Our method, dubbed FreSh (Frequency Shifting for Accelerated Neural Representation Learning). selects the embedding configuration where the frequency distribution of the model's 's output is close to that of the target signal. This shift in the model's 's frequency distri- bution results in better signal modeling (see Figure Пp. We validate demonstrating improved quality in representation tasks such as image and video overfitting and in an inverse problem, specifically 3D shape modeling with NeRF. We show that these improvements result from a more accurate approximation of all frequencies (see Figure 6).
approach experimentally,
In existing INR studies (Sitzmann al., 2020b Tancik et al. 2020 Saragadam al. 2023
et al. 2024), new architectures are often introduced with minimal attention to simplifying the costly process hyperparameter selection, even though it is crucial for achieving the best performance. Our framework is the first to address this gap by leveraging frequency information to guide the configuration process for embeddings of existing INR models, with the potential to easily extend to future architectures. Our key contributions are:
We develop a technique for comparing frequency contents of images based on the Discrete Fourier Transform and the Wasserstein distance.
We introduce FreSh (Frequency Shifting for Accelerated Neural Representation Learning) model agnostic method for configuring coordinate embeddings for better performance, that can be easily applied using provided script. FreSh works by adjusting the capacity of an INR model all target signal's frequencies.
We achieve state-of-the-art results in image and video approximation, well as 3D shape reconstruction (NeRF), using a fraction of the compute of a conventional grid search.
RELATED WORK
INRs are neural models used for signal representation that received considerable attention in re-
search (Tewari et al. 2022) and have been applied in various domains, including representation of images (Klocek et al. 2019), videos (Chen et al. (022b), and 3D shapes (Park et al. 2019). Notable applications include 3D shape reconstruction Mildenhall et al. (222)) robotics (Wang et al. 202 lb
Lin et al.] 202I). and compression (Lu et al.
2021| Takikawa et al. 2021). INR architectures are often simple, consisting of single MLP, with improvements focusing embedding layers (Chen
et al. 2022a 2022a|Müller Müller et al., 2022), activations (Sitzmann et al. 2020b), rendering techniques (Barron
al.
2021 2023), and regularization methods (Yang et al. 2023)
2

---

Published as a conference paper at ICLR 2025
Spectral bias is a phenomenon observed in MLPs describing their preference for learning low- frequency functions and ignoring high-frequency noise (Rahaman et al. 2019 Ronen et al. 2019
Xu 2018), which helps explain the remarkable generalization properties of deep models. Never- theless, this low-frequency bias hurts model performance when high-frequency components of the signal are informative (Gorji et al.; 2023). A common way of overcoming this bias in INRs is by introducing high-frequency embeddings (Sitzmann et al., 2020b; Mildenhall et al. 2021; Müller al., 2022; Liu al. (024) that change the space over which the MLP operates, thus modifying the frequencies of the target function. However, the effectiveness of such approaches is limited as optimal hyperparameter configurations have to be found by trial and error for each target signal.
Positional encodings are a broad class of functions that map coordinates into a high dimensional space through a function of adjustable frequency. Their particularly prominent use in Transform- ers (Vaswani et al., 2017), but they are also crucial for INRs. One of their first applications was NeRF (Mildenhall et al. (02T), a model for 3D scene reconstruction from set of posed 2D images. NeRF embeddings are not stable in sparse settings due to its usage of very high frequencies (Yang) ct al.
2023), and its axis-alignment makes reconstruction quality rotation-dependent (Tancik et al.)
2020 Despite its drawbacks, it is frequently used (Barron al. 2021 Pumarola et al. [02 | Barron 2023). which could be due to its low sensitivity to hyperparameters compared to alternatives.
et al.
These alternatives include various activation functions (Sitzmann al. [020b] /aargadaam et al.)
2023)
and a inrection-invarian version of NeRF using Fourier features (Tancik et al. 20200).
Activation functions are popular method of improving the capacity of INRs, where the first layer acts as a positional embedding. A well-known example is Siren (Sitzmann et al.) (020b), which uses a sine activation to increase signal frequencies in the model's first layer. Architectures that generalize this approach utilize the Gabor wavelet (Saragadam el al., (023)) non-periodic functions [Ramasinghe & Lucey] 2022). and variable- -erboo- functions (Liu et al. 2024). In this work, we particularly focus on Siren due to its popularity, but we also address other activations.
Auxiliary data structures are used in neural scene representation to associate fragments of the scene with a feature vector of trainable parameters, trading a larger memory footprint for smaller computational costs. However, due to the extremely small MLPs used, these approaches can lose some of global reasoning and implicit regularization (Neyshabur 2016) capabilities of neural models. As directly storing a fine grid of features would be prohibitively expensive, practical approaches use low-rank approximations (Chen et al. 2022a), 2D feature maps (Chan et al., 2022) and hash tables (Müller et al.) 2022) to reduce the memory cost. In such settings spectral bias is not avoided, and the resolution of the voxel grid needs be tuned for each scene. Although our main goal is to improve pure neural network-base solutions, we also verify on the recent architecture from Müller et al. (2022) that FreSh is applicable grid-based approaches.
al. 2014 Goodfellow et al.)
ResFields is a novel framework for INRs that improves their capacity for representing complex signals by incorporating temporal residual layers into MLPs (Mihajlovic et al. 2024). By modifying network weights with a time-dependent component represented as a factorized matrix, it increases the performance with only small impact on parameter count and inference speed. We use ResFields to improve current state-of sate-of-the-aae fe-thhe-rt art results on video representation.
Initialization schemes for INRs have been extensively studied to accelerate training convergence. IGR (Gropp et al. 2020) introduced an implicit geometric initialization to speed up learning 3D shapes, while others (Rajeswaran et al. [2019] [Sitzmann et al.] [2020a] Wang et al.) [021aa] Tancik eet al. (2021) leveraged data-driven meta-learning approaches (Finn et al. 2017, Nichol] (2118) for learning implicit fields. In contrast to these methods, which rely on computationally expensive pre- training or hand-crafted priors, our approach avoids such requirements. offering more efficient alternative. Moreover, novel initialization methods such as the ones proposed by Saratchandran et al. (2024) are largely orthogonal with our method.
Regularization strategies can be applied to stabilize the training of -bRed models, especially in sparse settings Yang et al., 2023). As our interest is in hyperparameter selection, we do not test the effects of regularization on final results. Moreover, some regularization methods (Yang et all 2023] were developed only for NeRF-like embeddings and would have to be generalized for fair comparison. is also worth noting that the Wasserstein distance has been already applied to improve INRs (Ramasinghe et al. 2024) through regularization, which is different from our application.
3

---

Published as a conference paper at ICLR 2025
Table 1: Comparison between the default, optimal, and FreSh configurations of Siren during 15,000- step training on various datasets (Chest X-Ray, FFHQ-1024, FFHQ-wild, Kodak, Wiki Art), using one image per dataset. We provide the PSNR scores and the total training time, which includes wa selection. The default value of leads to suboptimal results, but improvements can be achieved by using an optimal configuration found through a hyperparameter sweep (wo E {30, 10,... 140)).
This process involves training multiple models, making fixing wo common strategy. FreSh achieves similar improvements to grid search while being an order of magnitude faster and comparable in time with plain Siren. Results are averaged over three seeds. The grid search performed once, and the best configuration was re-evaluated using different seeds.
PSNR f
Step
Training
500
1000
5000
15000
time (h) +
Siren (whp = 30)
25.26 10.00
26.10
10.04
29.13 10.06
31.18 10.02
2.661120.01
Siren + Grid Search
27.36 10.06
28.39
10.03
30.51 10.03
32.02 10.02
32.69
Siren FreSh
27.11 +0.03
28.16 #0.02
30.32 40.03
31.81 =0.01
2.82+0.01
MOTIVATION
In this section, we show that the training of an INR highly depends on configuring the embedding in way that aligns its frequencies with the target signal. This leaves practitioners with two options, ei- ther using a suboptimal, default embedding configuration or finding a well performing configuration through a costly parameter sweep.
We illustrate the impact of proper hyperparameter selection in an image representation task in Fig- ure [ using the Siren model. which uses an input-scaling parameter wi to increase embedding fre- quencies. We compare a default, unaligned with the target signal configuration of Siren (wo = 30) an aligned configuration (wp = 90) selected through a parameter sweep over Wg {30,... .... 140). The aligned configuration speeds up training by employing frequencies three times greater than the baseline, achieving sharp details while the baseline is blurry. Note how the sizes of uniformly col- ored areas in the output at step O indicate the size of features the network can easily learn this observation pivotal for FreSh. We perform a similar investigation on images, each from a dif- ferent dataset (see section 5). reporting the results in Table [ We find that optimizing wo always improves the baseline results, with specific best values of wi depending on the target signal. The fail- of SGD to optimize the embedding layer (see Appendix A necessitates hyperparameter sweeps, as a one-size-fits-all solution will inevitably be suboptimal for some target signals. Our goal is to perform this search and enhance INR performance while avoiding the high computational cost of repeated model re-training.
4 FRESH
In this section, we discuss how to compare the frequency contents of images and introduce FreSh, a computationally efficient method for initializing frequency embeddings that biases the model to- wards the frequencies present in the target signal.
4.1 PRELIMINARIES
This section discusses important theoretical concepts relevant to our study. INR architectures discussed in the Appendix, with the exception of the Siren model (Sitzmann et al. 20205), which high-level example to illustrate how similar approaches work. We provide list of embedding hyperparameters from each model used in our study in Table
Siren (Sitzmann et al. 2020b) addresses spectral bias by mapping its inputs, x IRd, through high-frequency embedding, given as
(s()) = San Wx + b),
where W ~ -1 4] are the weights of the layer and is bias. The scaling parameter Wy controls the frequency magnitudes of this embedding and the authors (Sitzmann et al. 2020b) recommend
4

---

Published as a conference paper at ICLR 2025
Bikes
sum
Magnitude
0.06
Bikes
Stren
0.04
0.02
Initiaization oulput
Frequency
ue E (10, 20, ................................ 200)
Siren output (initialization)
sump
Training with wo 120
Repeit for
WiBikss,
DFT
Afeer 500 steps
60 120
180
33 61
Figure 2: Example workflow of FreSh when applied to Siren and a high-frequeen Kodak image. The image and outputs from various model configurations undergo Discrete Fourier Transform (left). The Fourier coefficients of the same degree are then summed to produce the image spec- trum (top-right). The model spectra are compared with the dataset spectrum using the Wasserstein distance W (bottom-middle), with only the configuration at the global minimum, highlighted by diamond, used for training (bottom-right).. Note that the Wasserstein distance follows smooth trend with a distinet global minimum, indicating stable and predictable behavior. For additional examples of model spectra, see Figure 8
setting Wo 30, but adjustments are needed to reach optimal performance (Saragadam al. 2023) Siren was specifically designed so that the first layer is the sole contributor to the frequency increase inside the model, which results in the frequency distribution (spectrum) of this model being very concise.
Other models adopt similar approach Siren and control embedding frequencies through hyperpa- rameter, with higher values corresponding to higher frequencies. Notable models that employ this strat- egy include NeRF (Mildenhall et al., (021), Fourier (Tancik et al., 2020). Finer (Liu al.) 2024). and Hashgrid (Mülee et al., 2022). While most models embedding frequencies that similar to what is present in the dataset and not excessively high, NeRF employs frequencies that often surpass those found in the dataset, which makes it incompatible with method. Similarly, Wire (Sara- gaddm et al. (2023) uses very high frequencies and in- creases frequencies at each hidden layer, not just at the embedding layer. This makes it incompatible with FreSh. As such, we provide results for NeRF and Wire as a reference, but do not optimize them with FreSh. The approach of Finer differs from other models by employing two hyperparameters, though they largely serve the same purpose, rais- ing questions about the necessity of both. We explore two scenarios for Finer: each scenario, we optimize only one of the parameters.
Table 2: Embedding hyperparameters of architectures used in our study. In all mod- els increasing the hyperparameter value re- sults in higher embedding frequencies.
Frequency
Model
Parameter
Description
Fourier
Weight variance
Siren
Wo
Scaling parameter
Finer
w
Scaling parameters
our comparison-ber
Finer
Bias range
Hashgrid Nmax
Grid resolution
Discrete Fourier Transform (DFT) of an image A E C describes its frequencies based on both their magnitude and direction. For example, sin(xa) and sin(x1) represent the same frequency magnitude but different directions. We denote the DFT of the c-th channel of as F(Ac) E N where the element (Ac) is defined as
F3,((A)) Σ
N
12x to Acm,.
Since DFT depends direction, it is not invariant transposition, i.e., F(A,) + F(AT).
Direction dependence makes DFT different from embeddings used in INR models, which treat all inputs symmetrically. Consequently, DFT has an unnecessarily complex structure (a matrix instead
5

---

Published as a conference paper at ICLR 2025
of vector), capturing differences, such as image transposition, that irrelevant to application. To make the DFT more suitable for our purposes, we reduce it to a spectrum vector S(A) = (S(A, .,. S(A, N 1)) which removes direction dependence of DFT (S(A) S(A")), for examples, see Figure 2 We achieve this by summing together elements of DFT that represent the same frequency, but not direction, meaning the elements where DFT indices number. This is well illustrated by transposing an image, as Fig is the same as (Ac). Specifically, the elements of the spectrum vector are created by summing along DFT diagonals:
the same
S(A,d)
Σ
E
(F,((A,))
DE (O....C-11] (O.)) N-I) i4j=d
We additionally denote the first n entries of ((A) as
(A) (S(A, ..... S(A,
(4)
We omit the term S(A, 0) corresponding to constant signal, as changes to this component can be fully captured by the bias of the output layer of neural network. As such, S(A, 0) is easy model and is not affected by changes to the embedding.
The term spectrum is sometimes used to describe the 2-dimensional DFT of an image. However, to the best of our knowledge, it has not been widely applied in the context of INRs, and no prior work has introduced the spectrum as a vector.
Wasserstein distance is a distance function between probability distributions. For distributions P and Q over E, and cost function c(z,y) 1 yl. is defined as
inf / clz, y) dr (x. =PI(P.Q) RxR
Q(P,Q)
(5)
where I(P, Q) is the set of all joint distributions on IR x R with marginals P and Q. Although complex to estimate in multiple dimensions, in our 1-dimensional, discrete case the Wasserstein distance is the L1 norm of the difference between cumulative distribution functions: |[CDF(Q)-
CDF(P))l (Panaretos Zemel 2019).
Table 3: Average PSNR on image representation tasks. FreSh outperforms matches the bascline
performance without introducing significant computational costs (see Table 4 Results for Wire are
provided for reference only, as is incompatible with FreSh (see section 4. The best results in
each section are bolded. Results are averaged over 3 seeds.
PSNR f
Average
Chest X-Ray
FFHQ-1024
FFHQ-wild
Kodak
Wiki Art
Siren
33.85 10.01
37.35=>0.01
37.54 10.04
34.32 10.01
31.60 10.03
28.45 10.01
+FreSh
34.62=0.01
37.99
39.1 =0.01
35.400..01
31.7840.02
28.80=0.01
Fourier
32.12 10.01
36.96 10.03
35.01
10.04
10.01 32.65
28.84 10.04
27.15 10.01
+FreSh
33.45 02
37.77:0.01
36.81 00.08
34.6220.01 01
30.06:0.01
28.01
Finer
38.6331000
40.45=0.01
36.4820.00
31.40
28.57=0.01
+FreSh
35.03 10.01
38.51 20.04
40.31 10.07
36.48=0.01
31.31 20.03
28.54 #0.01
Finerx=o
34.81 40.011
38, 18.4440.00
39.91
40.02
35.96
+0.01
$1.43++.00
28.31 40.01
+FreSh
34.8820.03
38.43 20.03
40.12
12.0.0
36.2882.02
31.16 #0.01
28. 20.05
Wire
33.54 40.02
37.96 #0.02
38.13 4..04
35.04
40.01
28.95
#0.06
27.62 4.0.11
4.2 FRESH
FreSh performs a parameter sweep in which the Discrete Fourier Transform and the Wasserstein distance are used to approximate model performance, instead of the costly model training required for grid search.
Method description. Our goal is to seleet an embedding configuration 0 from a set of M configura- tions {0:)se(......) that would maximize performance when fitting a target image Y E RRCxxXX We propose to use the configuration 0; where the associated output of the model at initialization, ÿ
ini has a similar frequency distribution the target image, in other words SY) s (Yii..

---

Published as a conference paper at ICLR 2025
Ground Truth
Fourier
Fourier + FreSh
Siren
Siren + FreSh
Ground Truth
Fourier
Siren
Hashgrid
Finer
Fourier+FreSh
Siren+FreSh
Hashgrid+FreSh
Finer+FreSh
Figure 3: Example model outputs for image modeling (top) and NeRF (bottom). FreSh representa- tions are better at modeling high-frequency details such text ropes. For additional examples, see appendix D
Siren
Fourier
0.9
PSNR
SSIM
Baseline FreSh
0.85
322
0.8
30
0.75
10 Steps ('000)
15
0
5
10
15
0
10
15
5
10
15
Steps ('000)
Steps ('000)
Steps ('000)
Figure Mean PSNR and SSIM values during training on 50 images (averaged over seeds). FreSh improves the final performance and speeds up convergence. Dotted lines indicate the final results of the bascline model. Shaded area is the 99% confidence interval.
We note that the spectrum (,) is absolutely homoge- neous (S,, (a.A) |o (A)), which implies that scaling does not affect the relative presence of different frequen- cies. Additionally,
FFHQ
Wiki Art
equivalent scaling the original signal, a common data pre-processing step. As such, the spectrum can be interpreted as probability distribution, making the Wasserstein distance natural choice for sim- ilarity measure. This requires only that we use the normal-
W
Ficus
Hotdog
40
80
120
400
ized spectrum, defined as (A) = (A)]E- With this, we define the FreSh configuration fJ as the one that minimizes
Sa(A)
Figure 5: Wasserstein distance for selected image and NeRF datasets across different Siren configurations. It follows smooth trend with dis- tinct global minimum, indicating sta- (6) ble and predictable behavior. Shaded area represents the 95% confidence (7) interval.
Wasserstein distance between the target signal and the model, meaning
arg min (Y),,
= arg min| min|| CDF(SS (Yr
In settings where multiple target images are available (video approximation and NeRF), we select one at random calculate the Wasserstein distance. We visualize the entire selection process for images in Figure 2 and provide a full algorithm of FreSh in the Appendix (see algorithm [
Measurement noise. Due to the randomness of the image used as the target signal, Y. video approximation and NeRF tasks and the randomness of the model output y arising from random network weights, the measurement of the Wasserstein distance is noisy. To prevent this from affect- ing the selection process, we measure the Wasserstein distance 10 times and use its mean to select

---

Published as a conference paper at ICLR 2025
the optimal configuration. This is partieularly important on video approximation and NeRF tasks as there both Y and Y/nr are sources of noise. This results in higher measurement variance in those tasks (see Figure(5) (5) The FreSh process could be further optimized to use less compute by decreas- ing the number of measurements, especially low-noise image representation tasks, but we do not investigate it in detail as FreSh already requires only negligible amount of compute.
Spectrum size. The reason for using a cropped spectrum (equation 4. instead of the full spectrum (equation B is due to the noise present in real-world signals. Synthetic signals such as Yini have predominately low-frequeney components (see Fi which makes them inherently mismatched with real-worle signals in the high-frequency range. As such, using equation (or increasing the spectrum size n) would shift FreSh embeddings towards higher frequencies. On the other hand, low values of vi can remove high-frequency (not noise) components of the target signal, leading to worse performance. We found setting to 64 achieves good results, with additional improvements possible through adjusting n, which we further investigate in an ablation study in the Appendix. Although method introduces new hyperparameter, it is not sensitive to the target signal, making it easy to configure. Additionally, the spectrum size could be used as an implicit regularizer, as lowering leads to lower frequencies being selected.
Table 4: Total time (h) needed to train all mod- els (Siren, Fourier, Finer, =inerk=o) for the image fitting task with grid search and FreSh, assuming that 20 hyperparameter values are tested (except Finer with 31 values). Time of grid search was estimated based on training time of single con- figuration. FreSh is an order of magnitude faster. Measurements are averaged over 3 seeds.
5 EXPERIMENTS
FreSh increases results quality and learning speed by tailoring the embedding to the tar- get signal, which improves modeling across all frequencies (see Figure 6 without enor- mous computational costs of conventional grid searches (see Table 4.. We show performance improvements on signal representation tasks and on an inverse problem the form of estimating radiance fields (Mildenhall et al.] 202I]. In all experiments, we use FreSh to select one embedding configuration from e {1, 20]. Wp E {10, 20, ,00)). w E (10,20, ... 200} and i 10.0, 0.1, 3.0)
Time (h) 4
Baseline
Grid Search
FreSh
Siren
29.1 10.3
584.2 26.6
30.3 20.4
Fourier
23.5 #0.5
470.2 #0.4
24.5 =0.4
Finer
33.0 10.1
(022.712.2
36.0 10.3
Finer
31.5 10.1
629.6 12.5
34.6 20.3
(parameters are described in table 2 Unless stated otherwise, we set the spectrum size hyperpa- rameter, n. to 64. All experiments are implemented in PyTorch (Paszke al. 2019) and use the Adam optimizer (Kingma & Ba 2014).
Calculating both the Fourier transform and Wasserstein distance is computationally cheap relative to the cost of training. This is crucial advantage of our method over a trial and error approach, as it makes feasible test multiple embedding configurations. We measured the highest relative cost of running FreSh on a high-resolution FFHQ-wild image, where the time required by FreSh reached 50 seconds per tested configuration, which is equivalent 1% of the training time.
5.1 SIGNAL REPRESENTATION
We evaluate on image and video overfitting using the first 10 images from FFHQ (Karras et al. 2019) (both the "in the wild" and cropped images at a resolution of 1024x 1 0224). Wiki Art (Saleh & Elgammal| 2015), Chest X-Ray (Kermany et 2018), and Kodak (Franzen 2024 datasets. For videos, we use the "bikes" and "cat" videos from (Sitzmann et al., 2020b). Image heights and widths for the image fitting task range from 380 to 6720. video overfitting, we use the current state-of- the-art solution in the form of ResFields (Mihajlovic et al. 2024). which improves the capacity of the MLP by making the weights time-dependent.
We report the average PSNR across all image datasets after full training (15k steps) in Table 3 example outputs in Figure 3 and training curves in Figure 4 FreSh consistently achieves results that are either better than comparable to the respective baselines. achieving similar performance about times faster in the case of Siren and about times faster in the case of Fourier features, while being order of magnitude less costly execute than conventional grid search (see Table 4
8

---

Published as a conference paper at ICLR 2025
Modifying the hyperparameters of Finer did not lead to any noticeable improvements. Nonetheless, FreSh performs comparably well, being especially helpful when the bias is eliminated (k = 0).
In this situation, FreSh successfully reproduces the performance of the original model with bias, raising questions about the need for both parameters. The advantage of FreSh is especially visible in terms of perceived quality, as measured by SSIM. These im- provements are achieved through better approximation of the target signal across all frequency components (see Figure thanks to the increased frequency mag- nitudes in the embedding layer. Similarly to optimal configurations selected by FreSh are highly varied, highlighting the subopti- mality of any constant configuration (sce Appendix E).
1.5
Relative error
Siren FreSh
Fourier FreSh
1
0.5
150
300
450
Frequency
configurations from Table [
Figure 6: The mean spectrum of residuals for the image representation task expressed as a percentage of the baseline configura- tion performance, as measured on Kodak images. The dotted line indicates the base- line performance. FreSh improves model- ing across all frequencies, lowering the ror by about 30%. The shaded area repre- sents the 95% confidence interval.
To show the versatility of FreSh, we successfully ap- ply it to videos, which contain temporal changes not directly measured by our method. We measure PSNR and SSIM and report the results in Table with full table available the Appendix (see Table 9. We found that to improve the model's frequency model- ing, we needed to remove time from the input coor- dinates, making an indirect input only through the weight modifications of ResFields. This suggests that commonly used embedding strategies are ineffective for signals with qualitatively different directions (see Figure
5.2 NEURAL RADIANCE FIELDS
Neural radiance fields (Mildenhall et al. 202[) are used for synthesizing novel views of a 3D object based on a limited number of measurements (images) of the object. Unlike in the image fitting task, the target sig- nal is unknown and spectra calculations required by FreSh are performed on the available images. To ren- der views Yi from the model, we take single sam- ple along each ray in the middle of the scene, where we assume the volume density maximal. We per- form experiments using torch implementation (Tang 2022 Tang et al., 2022) ) of InstantNGP (uüller et al. 2022 and synthetic NeRF data. Due to the complex- ity of this task, we adjusted the spectrum size for each model, and use 128 for Hashgrid, 64 for Fourier and Finer, and 32 for the spectrally-concte Siren.
Table 5: Mean PSNR video representa- tion. FreSh outperforms baseline embed- ding configurations and the NeRF-like em- bedding (Positional Encoding). Each con- figuration was tested with and without time as an input coordinate. The model benefits from embeddings reconfigured with FreSh only when time is not input, indicating that different frequency magnitudes are re- quired for spatial and temporal directions. Results for Positional Encoding are pro- vided for reference only, it is incompat- ible with FreSh (see section 4.1] Results averaged over seeds.
FreSh achieves similar or better reconstruction qual- ity than baseline configurations and positional encod- ing, with the exception of axis-aligned datasets (lego, materials) where positional encodings have an advan- tage due to their frequencies also being axis-aligned (see Table In the case of the Hashgrid embedding, we found that the recommended range for the reso- lution parameter Nma. E [512, 524288 induced fre- quencies that are too high, with FreSh selecting resolu- tions outside this range (Nv E {64, 128, 256)). This lowering of embedding frequencies leads to higher re- construction quality, highlighting the applicability of FreSh in various settings and detecting models with both too low and too high frequencies.
Time Input
Cat PSNR
Bikes PSNR f
Siren
38.72 +0.05
41.12 10.03
+FreSh
36.84 40.0s
40.28 40.01
Siren
X
39.84 40.03
40.39 40.02
+FreSh
40.61 20.04
41 01.
Fourier
38.14 4.0007
40.92 4.007
+FreSh
J 37.38 40.04
10.29 20.03
Fourier
38.82 40.03
40.61 40.03
+FreSh
X 39.68.0.04
41.13
Pos. enc.
37.39
39.78 $9.78 +0.04
Pos. enc.
X
37.13 10.04
39.57 20.05
Hashgrid
34.71 40.15
37.13 420.01

---

Published as a conference paper at ICLR 2025
Table 6: Average PSNR achieved with Positional Encoding (also known as NeRF), Siren, Fourier, Finer and Hashgrid embeddings on synthetic NeRF datasets. FreSh improves performance in many instances. Results for positional encoding are provided for reference only, as is incompatible with FreSh (see section 4.1
PSNR
Averuge
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Sinen
31.2120.00
34, 20:0.11
26.03:20.13
30.2540.06
85.8120.18 30 30.28:0.14 14 29.6720.11 34. 34.2540.08
28.60=0.18
#FreSh
31.43=0.08
71.111..11
15:0040.14
31 .1.0300006
35.70 13 30.81 15 28.40=0.12
34.41=0.07
28. 7...0.00
Fourier
92.0420.00
74.70:10.13
26.42:2012
31.19:0.00
35.9720.14 14 31.58::.116
31.67::0.10
34.38:0009
30.07=0.12
+FreSh
32 32.63=0.00
36. 36.8220.11
26. 0.1422:0.14
32.3810.05
36.2620.11
33.41 17 30.99=0.12
34 34.43=0.00 09 30. 50.5810011
Finer
31.64=0.09 34.64#0.111
26.12#0.13
00000
20.70##.114
31.69+0011
29.71#0.11
34.35=0.08 29.60
+PreSh
31.6210.09
34.8610.11
26.1020.13
31.1310.06
35.77=0.14 14 31. 80:00.1 13 29.51=0.11
34.2710.07
29.47:0.17
Finerawo
31.3940.00 34.38#0.111
26.02#0.13
30.80#0.006
86.72=0.14 31.08#0.13
29.68=0.111
34.18+D.08 0.194
+IreSh
81.62=0.09 84.84#0.11
26.02$0.13
35.60#0.14 2360
06.60111 29.47=0.11
34.37=0.08
13
Hushgrid
31.09#0.00 34.04#0.13
25.77=0.09
30.65#0.05
34.8220.18
31.4240.13
28.67=0.02 34.34=0.08
29.06
+PreSh
81.22=0.09
33.5520.14
26.17=0.11
41.06
25.22-0.14
#2.6620.14
28.9240.09
34.0940.07
28.71=0.20
Pos. enc.
32.2240.00 36.46#0.11
26.0420.12 30.77#0.06
35.7620.14
14 33.7740.16
31.78=0.11
35.15#0.08
29.72
LIMITATIONS
While this work represents an initial step toward automating embedding configuration for INRs, FreSh has several limitations inherited from existing embedding methods. Notably, FreSh does not account for direction-dependent frequency magnitudes, potentially impacting performance in sce- narios where directionality is important, such video approximation. Although the spectrum size hyperparameter is easier to configure than traditional embedding parameters due to its independence from the target signal, still can require manual tuning. Moreover, FreSh performance inherently limited by the constraints of the embeddings it configures and cannot achieve quality better than conventional grid search. Specific to FreSh, not applicable to models that utilize excessively high embedding frequencies, such NeRF or Wire.
7 CONCLUSION
Hyperparameter selection is crucial for INR performance, yet there is limited research in this area. This makes evaluating new architectures costly due to the need for extensive grid searches, or un- fair when suboptimal hyperparameter values are used. We address these challenges by introducing FreSh, a model-agnostic method for configuring coordinate embeddings that significantly reduces the cost of finding effective configurations compared to parameter sweeps. FreSh leverages fre- quency information to select the configuration that best aligns with the target signal, effectively biasing the model to fit all relevant frequencies. While FreSh such as Wire, it proves highly effective when applicable, facilitating the use of improved embed- dings like Fourier features. This particularly relevant in the context of Neural Radiance Fields (NeRF), where the adoption of this embeddings has been limited by high hyperparameter sensitiv- ity. Although FreSh introduces a new hyperparameter, requires little to none adjustment. By utilizing ResFields, we have observed that frequency mag- nitudes in certain tasks are significantly direction-dependent. This suggests that new embeddings may be needed to account for this dependence, and expanding FreSh to also consider directional dependencies is a promising research direction that could further enhance its effectiveness.
compatible with certain models,
not sensitive to the target signal and
ACKNOWLEDGMENTS
The research of A. Kania was funded by the program Excellence Initiative Research University the Jagiellonian University in Kraków. The work of P. Spurek was supported by the National Centre of Science (Poland) Grant No. 2023/50FE/ST6/0006.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo PrrtinnBruuall,, and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855-5864, 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 19697-19705, 2023.
Zhicheng Cai, Hao Zhu, Qiu Shen, Xinran Wang, and Xun Cao. Batch normalization alleviates the spectral bias in coordinate networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 60-171111,,, 2024.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-ave -waare 3d generative adversarial networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16123-16133, 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, pp. 333-350. Springer, 2022a.
Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Videoinr: Learning video implicit neural representation for continuous space-time super-resolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2047-2057 2022b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Richard W. Franzen. True Color Kodak Images, June 2024. URL. https://rOk.
graahiics/koda ca/oodak [Online; accessed 11. Jun. 2024].
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5501-5510, 2022.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT Press, 2016.
Ali Gorji, Andisheh Amrollahi, and Andreas Krause. A scalable walsh-hadamard regularizer to overcome the low-degree spectral bias of neural networks. arXiv preprint arXiv:2305.09779, 2023.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regu- larization for learning shapes. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3789-3799 PMLR, 13-18 Jul 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. Identifying medical diag- noses and treatable diseases by image-based deep learning. cell, 172(5):1122-1133 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.691412.6988 2014.
Sylwester Klocek, Lukasz Maziarka, Maciej Wolezyk, Jacek Tabor, Jakub Nowak, and Marek Smieja. Hypernetoork functional image representation. International Conference on Arti- ficial Neural Networks, pp. 496-510. Springer, 2019.

---

Published as a conference paper at ICLR 2025
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5741-5115 2021.
Zhen Liu, Hao Zhu, Qi Zhang. Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexible spectral-bias tuning implicit neural representation by variable-periodic activa- tion functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2713-7722,22 2024.
Yuzhe Lu, Kairong Jiang. Joshua A Levine, and Matthew Berger. Compressive neural representa- tions of volumetric scalar fields. In Computer Graphics Forum, volume 40, pp. 135-146. Wiley Online Library, 2021.
Marko Mihajlovic, Sergey Prokudin, Mare Pollefeys, and Siyu Tang. ResFields: Residual neural fields for spatiotemporal signals. In International Conference on Learning Representations, 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(:::1-15, 2022.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint 1412v:1412.660 2014.
A Nichol. On first-order meta-learning algorithms. arXiv preprint urXiv:/803.omrm /803.0299,, 2018.
Victor M Panaretos and Yoav Zemel. Statistical aspects of wasserstein distances. Annual review of statistics and its application, 6:405-431, 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.
Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10318-10327, 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pp. 5301-5310. PMLR, 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with im- plicit gradients. Advances in neural information processing systems, 32, 2019.
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for acti- vations in coordinate-mlps. In European Conference on Computer Vision, pp. 142-158. Springer, 2022.
Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, Hisham Husain, and Anton den Hen- gel. Improving the convergence of dynamic neRFs via optimal transport. In International Con- ference Learning Representations, 2024.
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural net- works for learned functions of different frequencies. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint arXiv:1505.00885, 2015.
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18507-18516, 2023.
Hemanth Saratchandran, Sameera Ramasinghe, and Simon Lucey. From activation to initialization: Scaling insights for optimizing neural fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 413-422, 2024.
Kexuan Shi, Xingyu Zhou, and Shuhang Gu. Improved implicit neural representation with fourier reparameterized training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25985-25999 2024.
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-leaming signed distance functions. Advances in Neural Information Processing Systems, 13:10136-10147, 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7475. 2020b.
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11358-11367, 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-bel based neural representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2846-2855, 2021.
Jiaxiang Tang. http://ggiilb..coommaaaawv/veeee
Torch-ngp:
pytorch
implementation of instant-ngp. 2022.
Jiaxiang Tang, Xiaokang Chen, Jingbo Wang, and Gang Zeng. Compressible-compen nerf via rank-residual decomposition. Advances in Neurul Information Processing Systems, 35:14798- 14809, 2022.
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, al. Ad- vances in neural rendering. In Computer Graphics Forum, volume 41, pp. 703-735. Wiley Online Library, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan Barron, and Pratul P Srini- vasan. Ref-nerf: Structured view- new-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5481-5490 IEEE, 2022.
Shaofei Wang, Marko Mihajlovic, Qianli Ma, Andreas Geiger, and Siyu Tang. Metaavatar: Learn- ing animatable clothed human models from few depth images. Advances in Neural Information Processing Systems, 34:2810-2822 10-2822, 2021a.
Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural b021b.
radiance fields without known camera parameters. arXiv preprint
13

---

Published as a conference paper at ICLR 2025
Leonid Nisonovich Wasserstein. Markov processes over denumerable products of spaces describing large systems of automata. Problems of Information Transmission, 5:47-52, 1969.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari. James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Graphics Forum, volume 41, pp. 641-676. Wiley Online Library, 2022.
Zhiqin John Xu. Understanding training and generalization in deep learning by fourier analysis. arXiv preprint arXiv: 10X.:0/8080042995 2018
Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8254 8263, 2023.
14

---

Published as a conference paper at ICLR 2025
A OPTIMIZATION OF EMBEDDING WEIGHTS
The embedding layer of an INR requires careful tun-
Histogram of Embedding Frequencies
ing of its hyperparameters (as discussed in section 3). even though its parameters should be optimized by gradient descent. To explain this phenomenon, we
0.3
Initialised
Fraction of Total
Trained
investigate the embedding of Siren and observe pat- tern similar to that noted by/Taccik et al. (2020) about the Fourier model: gradient descent fails to effec- tively optimize the embedding layer. This likely caused by the periodic nature of activation function used in the embedding.
0.2
Optimal
0.1
Measuring frequency of an embedding neuron Let's 's recall that the Siren embedding is given as
20
60
Freq. Mangitude (wo (W:l2)
embedding(x)) sin(wD Wx + b),
(8)
Figure 7: Frequency magnitudes of the em- bedding layer of Siren for baseline configu- rations (at initialization and after training) and FreSh configuration (at initialization) during training on 5 images from the exper- iment in section nitudes of the bascline model fail to in- crease a scale comparable with the op- timal configurations, highlighting that em- bedding frequencies are not effectively op- timized by gradient descent. As frequency magnitudes remain largely constant during training, they must be configured as hy- perparameter. Magnitudes were clipped to range [0, 80].
where W are the embedding weights. Denot- ing the i-th row of W as wi. we analyze the frequency of each embedding neuron indepen- dently, ignoring the bias term b, as it only affects phase, not frequency magnitude. Specifically, we consider the frequency of sin (wio x), which maximal along the direction specified by W;- Given that W; x [|welle||x [x|] cos( (os(<(w,,))) x)), can simplify the analysis to the fre- quency of sin (w// /w/l//wall sin (wol w: |
During training, mag-
(ww/ll/lxx |X||2 cos( ((w,, x)))
Frequency is defined as the number of full periods a function completes over a unit distance. By choosing 2x as the unit distance, the frequency of sin( (in(()/|2) is
simply 1. The frequency of sin(wol] ww/////lll ||||||/]]]] (2) scales proportionally and is given by wol wol/w:/ll.
Experiment We investigate how frequency magnitudes change during training on 5 images (we use the same images as in section 3) by comparing the distributions of frequency magnitudes between models at initialization and after training (see Figure 7). We additionally provide frequency magni- tudes of embeddings configured using grid search (see Table[]] as a reference of optimal magnitudes. We use the same experimental setup in section 5 only changing whether the embedding weights are optimized or not.
During training the magnitudes change only slightly, but this increase is negligible when compared to magnitudes induced by optimal embeddings, whose size reflects the multiple-fold increase of we observed in Table [ Since gradient descent fails notably increase embedding frequencies. the magnitudes must be adjusted as a hyperparameter, significantly increasing the cost of finding optimal models.
B ARCHITECTURES FROM RELATED WORKS
This section covers architectures that were left out of the main text.
NeRF MMildenhall et al. 2021) maps 5D coordinates -spatial location (x, y. 2) and viewing direction (0, () volume density and view-dependent emitted radiance, which are then used render novel views of a scene. It employs positional embedding /Vaswani et al. 2017), which is a multiresolution sequence of L frequencies:
(x(x) = [sin x)), (20(22xx) sin xx) cos(22-1x))
TP helps overcome the spectral bias of MLPs but is biased toward axis aligned directions, which can result in performance loss depending on the rotation of the target object (Tancik et al.] (020). This embedding works well with frequencies which are higher than the main components of the target
15

---

Published as a conference paper at ICLR 2025
signal (e.g.. L 16). When training NeRF models, embed the spatial coordinates using Yp and the viewing direction with the spherical harmonics basis (Fridovich- Keil et al. 2022 Verbin et al. 2022). following the approach of Müller et al. (2022).
Fourier features (Tancik et al. 2020 is a
alternative to eq. 9 which densely
samples Fourier basis functions. Although such sampling is not feasible in realistic settings, it can
be well approximated through random sampling (Rahimi & Recht 2007). resulting in the mapping
Y/ (x) [sin(2* (2 Wx)].
(10)
where weights are sampled from an isotropic frequency distribution, such as Gaussian W ~ (00, or). Scale of this distribution, a. controls frequency magnitudes and it was found to be an important factor in the final performance of a model (Tancik et al. (020). Even though the perfor- mance of Fourier features shown to be better than that of eq. (2) it has not been widely adopted, possibly due to the high sensitivity of this embedding to the value of a. We refer to this embedding as Fourier.
Finer (Liu et al. 2024) is a variation of the Siren model with a broader supported frequency set. It extends the frequency set of Siren by utilizing a suriable-periodie activation, (p(x) sin(([=|+1))) in its embedding
[[(==[++)aa
(r(x) ( WW + b)),
(11)
where W E Rmxd is the matrix of weights and b E IRm is bias. This model controls frequencies
through the scaling parameter w and the width of bias distribution & (b ~ Uniform(-ke k)) D
The authors of Finer suggest using bias as the main method for increasing the model capacity to approximate high-frequency signals. However, a similar effect can also be achieved by adjusting w. which raises questions about the necessity of using bias. We consider two scenarios: fixed at 30 and is optimized, and another where bias is removed (k- = 0) and w is optimized. We find that bias can be removed without significantly affecting performance (see Table 3). We denote the model with no bias as Finerk-o-
where is
Multiresolution hash encoding (Hashgrid) divides the space into increasingly finer grids (e.g., 16), assigning each voxel to random feature vector via hash table (Müller et al. (022). The authors recommend adjusting specific hyperparameters for each dataset, particularly the size of the hash table, which impacts the memory footprint, and the resolution of the finest level, Nuax which affects the size of details that can be easily modelled. As a baseline, we use a resolution of 2048. At low resolutions, the number of grid vertices is of comparable size to the number of hash table entries, effectively limiting the total parameter count of low resolution models. To prevent different parameter counts from affecting the performance, we set a relatively low hash table size of 8192 entries.
Wire uses continuous complex Gabor wavelet activation function
(12) where we controls frequency and 8 controls width of the wavelet, with typically used values of wi 20 and so = 10. This activation makes model design problematic, as it increases frequencies at every layer of the model, making frequencies of the model depth-dependent. Moreover, we note that, similarly NeRF, Wire uses extremely high frequencies, which are incompatible with FreSh. As such, we report results of Wire only for context.
(xr) - - |=0=||
C ADDITIONAL SPECTRUM EXAMPLES
In this section we consider untrained models, showing their outputs and comparing their spectra to the Kodak "bikes" image (see Figure 2 As illustrated in Figure g the Wasserstein distance effectively highlights that using high- frequency embedding configurations would be beneficial for Siren and Fourier, as their spectra are closer to the spectrum of the signal. We note that the feature sizes in most model outputs align with those typically found in natural images. However, models such as Wire produce extremely small, high-frequency features, implying that they perform well despite using high frequencies, possibly by taking advantage of some additional mechanism.
Saragadam al. (003) denote as wp- We removed the index to make hyperparameters of Finer and Siren casier differentiate.
16

---

Published as a conference paper at ICLR 2025
D PSEUDOCODE FOR FRESH
We present the pseudocode for FreSh in Algorithm [ The algorithm is slightly different between image and video/NeRR tasks, due to multiple images being available for the latter. This is reflected in the definition of Y sample which consist of multiple images one for each measurement of the Wasserstein distance.
constructed by sampling 10 images for NeRF and video approximation tasks, while for the image approximation task, Y sample consists of the same image repeated 10 times. Even though the target signal is not random for the image approximation task, it also measured multiple times due to the randomness of the model output.
Algorithm 1:
FreSh
Input: of embedding configurations,
Data: sample a sample of images, X input coordinates
Output: Obest embedding configuration with the lowest Wasserstein distance
doest 00;
Obest + None;
for 0 E θ do
distances + II:
for Y in Ssmmlec do
Surget + full full_spectrum(Y)) nl]
get_random_model
Smodel + full-spectrum, (moecel (f, Φ, [: n]:
wasserstein distance mSnodal Starget;
distances.aie (apprnneess.ppedd(()))))..
amean f- mean mean(distanue (distances);
if dmeean dbest then
doest f dmean'
Obest 0;
return Obest:
E ADDITIONAL SIGNAL REPRESENTATION AND RECONSTRUCTION RESULTS
Table 7: Ablation study spectrum size on image approximation task. All runs in the table were conducted with fixed seed. The optimal spectrum size depends primarily the base architecture, with the dataset having minimal impact. The best results in each section are bolded.
71
Average
Chest X-Ray
FFHQ-1024
FFHQ-wild
Kodak
Wiki Art
Siren FreSh
32
34.28
37.49
38.13
35.00
32.13
28.67
Siren + FreSh
34.59
38.
39.09
35.37
31.72
28.78
Siren FreSh
128
34.15
38.37
38.66
34.93
30.41
28.37
Fourier FreSh
32
33.00
37.33
36.03
34.06
29.93
27.68
Fourier + FreSh
64
33.47
37.80
36.83
34.64
30.04
28.05
Fourier + FreSh
128
33.55
38.10
37.13
34.89
29.58
28.03
Finer FreSh
34.80
38.43
39.87
35.96
31.44
28.30
Finer FreSh
64
35.06
38.59
40.44
36.50
31.23
28.56
Finer FreSh
34.97
38.76
40.09
36.38
31.17
28.46
Finers + FreSh
32
34.44
38.07
39.16
35.33
31.42
28.19
Finers-o FreSh
64
34.95
38.50
40.27
36.32
31.13
28.53
Finers-o + FreSh
128
34.79
38.68
40.10
36.35
30.61
28.24
Spectrum size Spectrum size, n, controls the number of frequencies included in the spectrum (eq. (4). The size the spectrum has to limited due to the inherent mismatch between natural and synthetic signals (as discussed in section 4.. Given that the architecture primarily determines
17

---

Published as a conference paper at ICLR 2025
the characteristies of the spectrum (e.g.. Siren was designed to have concise spectrum), the specific target signal is not expected to play a significant role selecting the optimal spectrum size. Our ablation study in Table / supports this, showing that the source of the dataset has minimal impact on the optimal spectrum size. Since the optimal spectrum size does not depend on the target signal, fine-tuning this value is significantly simplified, as it can be selected using a small subset of target signals. After determining the optimal spectrum size for a given architecture, no further tuning should be needed.
In general, we find that FreSh performs well when the spectrum size n. is set to 64, although perfor- mance can be further improved through fine-tuning (see Table 7). Interestingly, Siren- based models gain no benefit from increasing the spectrum size beyond 64. This is likely because Siren is designed with hidden layers that have minimal impact on frequencies (Sitzmann et al., 2020b), leading to a concise spectrum, which is well described even with shorter spectrum. In contrast, the Fourier model benefits from larger spectra, likely because no special considerations were made in its design to keep the spectrum concise.
Selected hyperparameter values We report configurations selected by FreSh on the image ap- proximation task in Figure Similarly to the optimal configurations (see T. observe high variability in model configurations selected by FreSh. Even when narrowed to a single dataset (e.g. Kodak), the configurations are varied, highlighting the need for adjusting the embedding frequencies for each target signal.
Decreasing the default embedding frequency In almost all experiments in section 5 models benefit from embedding configurations that induce higher frequencies than the baseline configura- tion. However, the Hashgrid model is an exception, where FreSh improves performance by selecting configurations that induce lower frequencies. In this section, we present another such example by testing Siven on a low-frequency synthetic dataset.
We generate the target signal as a sum of sinusoids with up to 5 periods and train Siren using the same setting as described in section 5 Since the target signal is relatively simple, we reduce the training time by factor of 10. We found that with the default learning rate training is not stable and PSNR can decrease by us much as 40% in 100 steps, which is why we additionally lower the learning rate by factor of 10. For this dataset, FreSh selects configuration of wo 10, resulting in frequencies three times lower than those in the baseline model. This low-frequency embedding improves the results, as shown in Figure IO This shows that FreSh improves quality by aligning frequencies and not by simply increasing them.
Learning speed Re-configuring a model with FreSh increases the learning speed, with image approximations being sharp after as little as 1000 iterations. We include example outputs from all image datasets in Figures to 14
Video approximation We report PSNR and SSIM scores on the video approximation task in Ta- bl Similarly to PSNR, SSIM is highest with models configured using FreSh and without a time input, making it an indirect input only through the weight modifications ResFields. This could be explained by different characteristics of the video signal along temporal and spatial directions (see Figure (6).
Object reconstruction We provide additional examples in Figure 15
INR Improvement Methods Recent advances have introduced methods to mitigate the spectral bias in INRs, such Batch Normalization (Cai al. 2024 and Fourier Reparameterization (Shi et al. 2024)). Meanwhile, Saratchandaan et al. 2024) propose the From Activation to Initialization (FAI) method, which optimizes initialization based on the activation function but does not directly address spectral bias. We evaluate FreSh alongside most of these approaches to highlight their complementary nature (see Th. We used the same experimental setup as section [3] We don't. report results using Batch Normalization, as using it yielded results worse than baseline Siren.
18

---

Published as a conference paper at ICLR 2025
Table FreSh compared with other INR improvement methods. Other than FreSh, we evaluate improved weight initialization (From Activation to Inirialization or FAI) and Fourier Reparameteri- zation (FR). FreSh improves the performance of other methods.
PSNR
Average
Chest X-Ray
FFHQ-1124
FFHQ-wild
Kodak
Wiki Art
Siren
33.85 10.01
37.35 +>0.01
37.54 10.04
34.32 20.01
31.60 40.03
28.45 10.01
+FreSh
34.62 #0.01
37.99 +0.01
39.11 +0.01
35.40 #0.01
31.78 +0.02
38.80#a.0r
FAT
34.32 #0.01
10.01 37.86
38.33 10.01
34.84 #0.01
31.97=0 os
28.61 10.01
+ FAI + FreSh
34.87:0.01
38. 30.03
39.66=0.01
35.77=0.01
31.79 #0.03
28.80
FR
33.79 #0.01
37.50 +50.01
37.51 +0.04
33.98 #0.01
31 65 40.00
28.32 +0.02
FR
FreSh
34.281200011
37.80=0.01
38.66=2.01
34.8220.02
31.54 20.03
28.57 20.01
Sirew 30, W 14.6)
Fourier (a 1. W 15.2)
Finer (wp 30, W 6.1)
Wine (W = 8.6)
Siren (wop 900
Fowrier
Finer (ulp W 6.4)
Spectrum
Spectrum
Spectrum
Spectrum
0.08
Biles
Buscline
Magnitude
0.06
1ligh
0.04
0.02
20
60
20
40
60
0
20
60
Frequency
Frequency
Frequency
Frequency
Figure 8: Example outputs from untrained models (top two rows) and their corresponding spectra (bottom row). For comparison, the spectrum of the Kodak "bikes" image also provided, along with the Wasserstein distances between the model configurations and the target image. We consider baseline (top row) and figh-freeuency (middle row) configuration for each model other than Wire, where the baseline is already high-frequency. The Wasserstein distance effectively shows that Siren and Fourier baseline configurations are too low-frequency for the highly detailed "bikes" image. Notably, the uniformly colored regions in outputs from models other than Wire (and the high-frreqency Finer) tend to be comparable in size or larger than typical features in natural images. In contrast, the output from Wire exhibits extremely fine, high- frequeryyy features.
19

---

Published as a conference paper at ICLR 2025
Siren
Fourier
Finer
Finerk=0
30
FFHQ-1024
Count
20
FFHQ-wild
Chest X-Ray
10
Kodak
Wiki Art
60
80 Wp
100
120
3
5
6
30
50
70
k
Figure 9: Embedding configurations selected by FreSh for the image representation task. Hyper- parameter values are highly varied even within datasets, highlighting the need for fine-tuning the embedding layer for each target signal.
50
PSNR
45
40
Siren Siren + FreSh
35
500
1,000
1,500
(a) Target signal.
Step
(b) Mean PSNR during training.
low frequency image. (a) Target image created as of sinusoids of at most periods. (b) PSNR values averaged over 10 seeds. Siren using the default configuration (wo 30) is slower in fitting the signal than a low-frequency Siren configured with FreSh (WD 10). This example demonstrates that reducing the model frequency can sometimes be advantageous, and FreSh capable of identifying such situations.
Figure 10: Training of Siren
Table Video representation results for NeRF, Siren and Fourier embeddings. FreSh outperforms baseline embedding configurations and NeRF. Each configuration was tested with and without time as an input coordinate. The model benefits from embeddings reconfigured with increased frequen- cies only when time is not input, indicating that different frequency magnitudes are required for spatial and temporal directions. Results are averaged over seeds.
Use time
Cat
Bikes
PSNR
SSIM
PSNR
SSIM
Siren
38.72
0.9518 10.0004
41.12 +0.03
0.9677
10.0001
+FreSh
36.84 10.03
0.9444 40.0002
40.28 10.01
0.9684
+.00000
Siren
39.84 +0.03
0.9553 +0.0001
40.39 +0.02
0.9657
#0.0001
+FreSh
40.61+0.01
1.9579+0.000 0002
41.62++001
0.9708=0.00 0001
Fourier
38.14 #0.07
0.948I 0000000
40.92 40.07
0.9678
+0.0000
+ FreSh
37.38 #0.04
0.9483 +0.0002
40.29
+0.03
0.9696
+50000001
Fourier
38.82 40.03
0.9514
+0.0003
40.61 40.03
0.9667
10.0001
+FreSh
39.68+0.01
0.9555+10
41.13
0.9700=0.u0uzz
NeRF
37.39
0.9471
0000002
39.78++001
0.9675=#0.000
NeRF
37.13 40.04
0.9462 +0.0002
39.57 +0.05
0.9664 +0.0004
20

---

Published as a conference paper at ICLR 2025
Fourier
Siren
Finer
Ground Truth
Fourier + FreSh
Siren + FreSh
Finer + FreSh
Fourier
Siren
Finer
Ground Truth
Fourier FreSh
Siren FreSh
Finer + FreSh
Figure 11: Model comparison Kodak images after 1000 iterations of training.
Fourier
Siren
Finer
Ground Truth
Fourier FreSh
Siren + FreSh
Finer + FreSh
Figure 12: Model comparison on a Chest X-Ray image after 1000 iterations of training.
21

---

Published as a conference paper at ICLR 2025
Fourier
Siren
Finer
Ground Truth
KIM
KIM
KIM
Fourier FreSh
Siren + FreSh
Finer + FreSh
KIm
KIm
Fourier
Siren
Finer
Ground Truth
Fourier FreSh
Siren FreSh
Finer + FreSh
Figure 13: Model comparison images from FFHQ-wild and FFHQ-1024 after 1000 iterations of training.
22

---

Published as a conference paper at ICLR 2025
Fourier
Siren
Finer
Ground Truth
Fourier FreSh
Siren
Finer + FreSh
Fourier
Siren
Finer
Ground Truth
D
Fourier
+ FreSh
Figure 14: Model comparison Wiki Art images after 1000 iterations of training.
23

---

Published as a conference paper at ICLR 2025
Ground Truth
Fourier
Siren
Hashgrid
Finer
Fousher+FreSh
Siren+FreSh
Hashgrid+FreSSh
FinertFreSh
Ground Truth
Fourier
Siren
Hashgrid
Finer
Fourier+FreSh
Siren+FreSh
Hashgrid+FreSh
FinertFreSh
Figure 15: Example model outputs for the object modeling task after approximately 20% of the training.
Bikes
Cat
Bikes
Cat
10
Mean Pixel Value
Magnitude
H
0.5
2
10
20
0.5
0.5 Pixel Location
Frequency
Frequency
Pixel Location
Figure 16: Axis-aligned analysis of video signals. We compute the average DFT of the video signal over 1000 randomly selected axis-aligned vectors and measure the mean pixel values along three directions: Time (T), Width (W), and Height (H). Signal along the time direction rapidly changes, such as during scene transitions, leading to spiky frequency distributions. This qualitative difference between temporal and spatial coordinates suggests that they should be treated differently an embedding layer. Pixel locations and values were normalized to the range [0, 1] to account for variations in video sizes and pixel value ranges.
24


Reviews:
[{"review_id": "AgqepeTnUN", "reviewer": "Reviewer_vNDz", "summary": "The authors propose a method for selecting the frequency hyperparameter involved in using periodic activations in implicit neural representations. Selection of the frequency hyperparameter is critical to INR performance, and authors often either select arbitrarily e.g. $\\omega=30$, or employ a costly grid-search. The authors propose selecting this hyperparameter based on the target signal frequencies, by minimising the Wasserstein distance between the target signal spectrum and the untrained network output spectrum. This is conducted by searching across candidate frequencies. The authors demonstrate performance improvements across images, neural radiance fields, and videos. Overall this is a nicely written paper which introduces a simple method with interest to the INR community.", "questions": "- The description of a vector spectrum by summing the DFT appears a little non-standard (L285-7). Could the authors provide a reference for this approach (potentially outside the INR literature?)\n- In Appendix C it is mentioned that the for image regression the same image is resampled 10 times due to randomness of the model output. The same network should be deterministic in output - does the randomness occur due to re-sampling the network with reinitialisation? \n- The paper mentions that other activations are addressed L131. Have experiments on non-periodic functions (e.g. Gaussian) been performed? \n- Could the authors provide details on the compute required for Fresh on more complex signals like NeRF (L414-6 notes that this is negligible for images). \n- Table 5 records video results. It mentions 'Results for NeRF are provided for reference only as it is incompatible with FreSh'. Could the authors clarify this?\n - Could the authors clarify what is meant by 'direction-dependent frequency magnitudes' and its importance for video (L510)? \n\nMinor Presentation: \n- The presentation of Figure 2 could be improved (visually the figure is difficult to interpret without the caption / main text). Tweaking this figure (perhaps additional labeling) would greatly improve the clarity of the method and extend its impact.\n- The font in figures 12, 13, 14 is inconsistent \n- The text size in Table 6 is inconsistent / appears to have been scaled \n- Figure 5 - suggest changing 'Thousands to steps' to 'Steps ('000)' \n- Figure 1 could be improved (the PSNR labels are difficult to distinguish)\n- Left-hand quotation makes can be done in Latex using the ` character", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There are a number of strengths in this paper: \n- The issue is well-motivated: selection of frequency is a common frustration when using SIREN implicit neural representations. A method for a-priori selection of frequency rather than needing to grid-search across trained networks will be useful for the community. \n- The method is conceptually simple and should be easy to implement / include in INR pipelines meaning it may be broadly applicable.\n- Experimental results extensively tested with results averaged across seeds for multiple modalities including image datasets (WikiArt, FFHQ, Chest X-Ray, etc), video, and neural radiance fields. \n- The authors have incorporated their method in multiple pipelines (Siren, Fourier, Hashgrid embeddings, Finer). Results broadly indicate that the method leads to a performance improvements within these pipelines. \n- The experiments are well ablated, with key components (e.g. spectrum size n) evaluated with multiple configurations and datasets\n- The paper is well written, logically structured, and clearly presented. Existing literature is reviewed well.", "weaknesses": "There are a few minor weaknesses in the paper. These principally relate to small aspects of presentation (e.g. Figure 2). I've listed these in the Questions below as they are largely minor issues rather than critical weaknesses. \n\nIn terms of technical weaknesses, while the method avoids a costly grid search across trained networks it still requires a grid search across candidate frequencies on untrained networks (this cost will be negligible in comparison). In addition, the method introduces an additional hyperparameter (selection of the spectrum cut-off). If I'm reading it correctly, this hyperparameter would still need a trained network to be evaluated (even though the authors note it transfers across signals). This could reduce some of the benefit of the method, especially if the frequency hyperparameter needs to be searched in combination with the frequency cut-off. Could the authors clarify this issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hilubymSiE", "reviewer": "Reviewer_cowA", "summary": "Hyperparameter optimization is a tricky but important problem for implicit neural representations (INRs), since different hyperparameters may be optimal for different signals, but hyperparameter sweeps can be expensive. The authors introduce a fast proxy for determining if an INR’s hyperparameters are well-suited to a particular training signal. Instead of directly optimizing the post-training loss, the authors propose to minimize the Wasserstein distance between frequency distributions of the training signal and the newly initialized INR’s output. This skips the step of actually training each INR, dramatically reducing the time needed to perform a hyperparameter sweep. This proxy appears to perform quite well, conferring most of the performance advantage of a traditional hyperparameter grid search. The authors evaluate their method across a range of image datasets, videos and NeRFs.", "questions": "> Similarly, Wire (Saragadam et al., 2023) uses very high frequencies and increases frequencies at each hidden layer, not just at the embedding layer. This makes it incompatible with FreSh\n\nI am curious if you can say more about these models (NeRF and Wire) which use very high frequencies. What purpose do these high frequency components serve if they are not reflected in the training data? Why don’t these models perform poorly as FreSh would predict?\n\nYou mention that your FreSh measurements are noisy due to random initialization, so you average across 10 initializations. [1] find that SIREN’s random initialization of the first layer has a significant effect on the final PSNR (Appendix A.6). I would be interested to see if your method could be further improved by using FreSh to select not only the hyperparameters, but also the random initialization to start from.\n\nIn the appendix you say that there is high variability in the model configurations selected by FreSh, even within a single dataset. But how much of this variability is random vs. actually important for performance? I would be curious to see how per-image FreSh compares to a dataset-wide FreSh: how much of the advantage of FreSh comes from optimizing for each individual image, and how much can be achieved just by making an appropriate hyperparameter choice for each dataset. \n\n[1] https://openreview.net/pdf?id=iKPC7N85Pf", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Clear, simple, effective solution to a practical problem. Thorough demonstrations that it works. The paper is well-written and well-presented, easy to read and understand. I can imagine this technique being widely adopted for INR hyperparameter optimization.", "weaknesses": "Section 5 contains a few typos:\n\nSection 5, paragraph 2, “a trail and error approach”\n\nSection 5.1: You fell victim to LaTeX’s backwards quotation marks, one of the classic blunders\n\nOtherwise the paper is quite solid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xoi09gTUdw", "reviewer": "Reviewer_swg8", "summary": "This paper proposes a method for selecting the hyperparameters for current MLP-based INR models to improve the suboptimal representation performance of models with default hyperparameters. This method measures the Wasserstein distance between frequency distributions of initial-state models and the target signal and selects the hyperparameters that minimize the Wasserstein distance.  This paper validates this method on several current INR models via 2D image approximation, 2D video fitting and neural radiance fields. Experimental results demonstrate that this method is effective to some extent.", "questions": "Typically, spectral bias means that neural networks tend to fit the high-frequency component of the target signal. Therefore, spectral bias can be reflected as different learning speeds for different frequency components. Although the advanced activation functions such as Sine improve the representation performance, MLPs with Sine still first fits the low-frequency parts as shown in some paper such as “Improved implicit neural representation with Fourier reparameterzied Training”. Inspired by this phenomenon, these methods might not change the bias; rather, it broadens the range of frequencies the model can represent. Under this perspective, Fresh might shift some special spectrum distribution that enjoys the fastest learning speeds of MLPs towards the target signals. So the detailed description of this special spectrum distribution or the validation of the existence of this special spectrum might be an interesting problem. This point will not affect my scoring. If authors could try to find this special spectrum and do more exploration about this problem, I think that it will further deepen our understanding of deep learning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The method of this paper is fairly easy to understand and flow. This is the first hyperparameter selection method for implicit neural representation models based on the idea of frequency alignment. \n2. The computation cost of this method can be ignored. \n3. Through relatively extensive experiments and the experiments on \"decreasing the default embedding frequency\" in the supplementary materials, this paper well demonstrates that this method could adjust part hyperparameters to adjust the “preferred spectrum” of MLPs, achieve better representation of target signals.", "weaknesses": "1. The related work of this paper seems overly lengthy and overlooks some of the latest methods for spectral bias. For example, spectral bias can be overcame by special initialization and training dynamics adjustment, such as reparameterization and batch normalization. Moreover, as stated in the paper that Fresh is a simple initialization technique in line 024, Fresh should compare to the previous initialization method like “From Activation to Initialization: Scaling Insights for Optimizing Neural Fields”. \n\n2. There lacks a reliable theoretical or experimental link between the key observation (line 019) and Fresh (line 241，Table 2).  More concretely, there need an explanation that the adjustment of embedding layer hyperparameters could affect the whole model's spectrum. A more quantitative expression would be better.  The illustration in Fig. 1 seems to be vague. Clarification of this point would certainly raise my opinion of the paper.\n\n3. Some experimental results show that Fresh could not provide a better result even worse than baseline models, such as results with time input in Table 5 and in Table 6. Intuitively, Fresh should not obtain worse results. It might show that there are several unclear relationships. Solving the weakness 2. might help this issue.\n\n4. The experiment in Appendix A seems for demonstrating that weights of embedding layer could not be trained sufficiently by gradient descent. However, why is the optimization algorithm SGD? Meanwhile, is the index L2 norm （Eq. 8） used by the paper (720) meaningful? This experiment should be illustrated more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kFW2n3BhzA", "reviewer": "Reviewer_TDK4", "summary": "The authors notice that the initial frequency spectrum of an untrained model’s output correlates strongly with the model’s eventual performance on a given target signal. They want to “pretrain” the untrained model output with the end signal through a hyperparameter search over initialization parameters.", "questions": "- Why not just have these parameters be learned rather than grid searched (as I believe omega_0 can be learnable at least, I am not sure about the others)? \n- 15,000 steps seems very excessive to fit a single image, how much wall clock does this take on a gpu for good results? Other methods like instant-ngp and spder (above) are much faster.\n- In 5.2 in experiments why is NeRF not used as a baseline also as it is mentioned in the paper? I am not sure which one it is in Table 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well written, and introduces a novel method of initialization by fitting the frequency spectrum of the untrained model with the desired downstream task. The logic is sound, and this concept may apply to other fields.", "weaknesses": "Evidence\n- The experimental evidence is extremely weak in my opinion and does not demonstrate a significant increase that would warrant other researchers’ usage of this method. Most results show improvements of less than 1dB, which is essentially identical to the naked eye.\n- The results in Table 1 are concerning. After 15,000 steps the difference between the Base Siren model and using Fresh have virtually no difference (PSNRs of 31.18 and 31.81). \n- Table 3 - Why does FreSH have no improvement on Finer and Finer with k=0 and most samples? If it is only applicable to the SIREN and Fourier features model it would be way too limited to warrant an accept as these are several years old baselines.\n- Moreover in Table 6, overall the results are not impressive as most PSNRs increase by < 1dB (i.e. Finer has virtually no change)\n- Table 8, why does Fresh hurt the results when you include time? It seems as expected input for video.\n\nBaselines\n- Paper seems to have a lot of overlap with SPDER https://arxiv.org/pdf/2306.15242 (especially in theory, and that the untrained model output aligns with end reconstruction) and should compare the baseline against it.\n- Other baselines should be (Ramasinghe & Lucey, 2022) which suggests Gaussian activations. It is mentioned in the paper but not reported on to the best of my knowledge.\n- Instant NGP (HashGrid?) may also be used in the image baseline, as it excels in high-resolution images but is not included here for that.\n- Overall, SIREN seems to be a very weak baseline model. I understand that tuning the omega_0 parameter can improve results, but I don’t believe this to be as significant of a contribution on top of the existing model.\n\nComplexity\n- Although computationally cheap (utilizing random initialization and DFT), it adds much more complexity to the setup\n- For example, on the Wasserstein measurement setup, “To prevent this from affecting the selection process, we measure the Wasserstein distance 10 times and use its mean to select.” I feel this is too much of a hack to be reliably used by other researchers, but I may yield to what other reviewers say.\n- Authors could clarify what the parameter values represent in the first section of Experiments for ease of reading.\n\nImpact\n- It is stated the method is incompatible for “extremely high frequencies” (i.e. NeRF and Wire), which makes me doubt its generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)