Predicting the Encoding Error of SIRENs
Authors: Jeremy Vonderfecht
Feng Liu
Venue: Accepted by TMLR ()

Abstract: Implicit Neural Representations (INRs), which encode signals such as images, videos, and 3D shapes in the weights of neural networks, are becoming increasingly popular. Among their many applications is signal compression, for which there is great interest in achieving the highest possible fidelity to the original signal subject to constraints such as neural network size, training (encoding) and inference (decoding) time. But training INRs can be a computationally expensive process, making it challenging to determine the best possible tradeoff under such constraints. Towards this goal, we propose a novel problem: predicting the encoding error (i.e. training loss) that an INR will reach on a given training signal. We present a method which predicts the encoding error that a popular INR network (SIREN) will reach, given its network hyperparameters and the signal to encode. This method is trained on a unique dataset of 300,000 SIRENs, trained across a variety of images and hyperparameters. Our predictive method demonstrates the feasibility of this regression problem, and allows users to anticipate the encoding error that a SIREN network will reach in milliseconds instead of minutes or longer. We also provide insights into the behavior of SIREN networks, such as why narrow SIRENs can have very high random variation in encoding error, and how the performance of SIRENs relates to JPEG compression.

Published in Transactions on Machine Learning Research (05/2024)
Predicting the Encoding Error of SIRENs
Jeremy Vonderfecht
vonder2 edu
Department of Computer Science Portland State University
Feng Liu
fliu@pde. edu
Department Computer Science Portland State University
Reviewed on OpenReview: htips: // openreview. net/ met/foruue forum? id- 3KPC7N85PJ
Abstract
Implicit Neural Representations (INRs), which encode signals such as images, videos, and 3D shapes in the weights of neural networks, are becoming increasingly popular. Among their many applications is signal compression, for which there is great interest in achiev- ing the highest possible fidelity to the original signal subject to constraints such as neural network size, training (encoding) and inference (decoding) time. But training INRs can be a computationally expensive process, making challenging to determine the best possible tradeoff under such constraints. Towards this goal, present method which predicts the encoding error that popular INR network (SIREN) will reach, given its network hyperpa- rameters and the signal to encode. This method is trained unique dalaset of 300,000 SIRENs, trained across a variety of images and hyperparameters D Our predictive method demonstrates the fcasibility of this regression problem, and allows users to anticipate the encoding error that a SIREN network will reach in milliseconds instead of minutes or longer. We also provide insights into the behavior of SIREN networks, such as why narrow STRENs can have very high random variation in encoding error, and how the performance of SIRENs relates to JPEG compression.
1 Introduction
Since the introduction of neural radiance fields in 2020, there has been swell of interest in neural networks representations of data. Such networks, commonly called implicit newral representations (INRs), coordinate networks, are trained to fit relatively low-dimensional signals, such 2D images, 3D shapes (typically as occupancy functions or signed distance functions), or neural radiance fields (NcRFs). Implicit neural representations have proven useful for diverse range of tasks, including super-resolution (Chen et al.
2021b), novel view synthesis (Park et al. 2021), video manipulation PPumarola et al. 2021 Gao et al. 2021
Mai Lin 2022 Li et al. 2021 Xian
2021 Tretschk et al. 2021), and compression.
There is great interest. in achieving the highest possible fidelity to the encoded signal subject constraints such network size, training and inference time. In the context of signal compression, researchers wish to minimize the network's size while maintaining low encoding error.
In this paper, we study SIREN networks trained to fit. natural images. Multilayer perceptrons with periodic activation functions, or SIRENs, have many advantages. They are simple to implement and quick to train.
They train stably even when 10+ layers deep, and have well-behaved derivatives (Sitzmann et al. 2020
Therefore, they have become a baseline architecture for INRs, for example, see Dupont et al. (2021): Martel
et al.) 2021 y; Lindell et al. (2022): ; Chen et al. (2021a). Mcanwhile, natural images have become standard
training target for the evaluation of coordinate network architectures, as seen in papers such as Sitzmann
et al. 2020), Tancik el al. 2020 and Dupont et al. 2021). )
PSNR Dataset available here: hugghhggace..co//laaaeettesrrree Satp CONN-collectonn

---

Published in Transactions on Machine Learning Research (05/2024)
In the context of images, by far the most common metric by which SIRENs are judged is mean squared error (MSE), often formulated as peak signal-to-noise ratio (PSNR). Practitioners often seek to maximise their INRs' PSNR subject to constraints such as training time (for signal compression, this is encoding time), number of trainable parameters (for compression), or forward-pass computation time (for decoding efficiency). However, training SIRENs can be time-consuming. For example, training SIREN to compress a 512x768 pixel image 0.3 bits per pixel with the method from Dupont et al. (2021) takes 50 minutes on Titan X GPU. Suppose that we wish to compress an image using the smallest SIREN that will achieve PSNR of at least 30 dB. Without a predictive model of SIREN performance, we will have to perform time-consuming hyperparameter sweep to determine the best STREN architecture for our task. A predictive model of what encoding error a given SIREN will reach will allow us to make quick, informed choices across a broad range of applications.
To address this issue, we develop a predictive model of how well a SIREN will encode a given image. Our model takes in of SIREN hyperparameters and the target image, and predicts the encoding crror, measured PSNR, that the SIREN will achieve. Within the training domain, our model makes accurate predictions of final training loss in just milliseconds, opposed to minutes of training.
To train this model, we develop a dataset of 300,000 SIRENs, trained across 100,000 photographs and a range of STREN hyperparameters. This dataset contains the hyperparameters, training loss curves, and final weights each SIREN, Initial weights for each SIREN can be reconstructed using our source code. We have made this dataset publicly available.
Like all neural networks, our predictive model of SIREN encoding error is somewhat of black box. Therefore, also present collection of experiments which offer more human-friendly insights into the relationship between the SIREN hyperparameters, the image to encode, and the SIREN's encoding error.
Our contributions are as follows:
novel dataset of 300,000 SIRENs trained to encode RGB images. (Section 2.2
neural-network--bees predictive model of SIREN encoding error. (Section 3.7
A series of studies on SIRENs, which show that:
Deep, narrow SIRENs like those used by Dupont et al. (2021 have significant random variation
in their encoding error, largely explained by the initialization of the first layer. (Section 3.1
SIRENs follow scaling power-laws as described in Bahri et al. (2021). (Section 3.2
There is a strong correlation between JPEG and SIREN compression. (Section 3.4
Unlike some other neural networks, SIREN training dynamics not well-approximated by their empirical neural tangent kernels. (Section 3.5)
low bitrate SIRENs visibly compress the color space the images they encode to a more restricted color palette. (Appendix A.7
2 Methods
2.1 A brief overview of the SIREN Architecture
SIRENs, introduced by Sitzmann al. (220)). are simply multilayer perceptrons with sinusoidal activations. A SIREN (2) has the following form:
(x(x) Wzhz-1 bb
(x) sin(Wah by) for 1 I,..., L-1
(1)
ho hz(z)= war
2

---

Published in Transactions on Machine Learning Research (05/2024)
SIREN Parameters
SIREN Reconstruction
Width - 10
Depth
train doin_siren.py
minutes
Waj 16.5
Image
PSNR Predictor
PSNR: 23.2
MSE Loss
Training Image
PE
MLP
m70
PSNR:
24.5 24.5->
CNN
Figure Overview of our many-architecture encoding error predictor. First, we train STRENs on many images with random sampling of hyperparamcters. Then, our many-architccture predictor takes the same inputs as the SIREN training script, and predicts the SIRENs PSNR. The training image is passed through nanvolational-neewrrk-bbbbo
feature extractor (CNN), while the SIREN hyperparameters fed through positional encoding (PE). Then both are concatenated and passed through fully-connected network (MLP) which predicts what PSNR the training script will reach.
where L is the number of layers, 0 = (Wo, bo, Wi,bi..... Wr, are the network's ' trainable weight matrices and bias vectors, and wp is scalar hyperparameter which controls the spatial frequency of activations in the first hidden layer hi (2), and hence the spectral bias of the network.
SIRENs are commonly used as compressive representations of images. In this setup, a STREN learns to represent single image by learning function from (x, y) pixel coordinates to (r, g, b) colors. Inputs and outputs are both normalized to [-1, 1]. The SIREN trained with L2 loss between predicted and true colors. By storing the learned weights at low precision, can use the network compressed representation of the image. The encoding error of this representation, as measured in PSNR, is equivalent to the best L2 loss
achieved during training.
2.2 Datasets
In this paper, make use of two image datasets: Kodak 1991] and MSCOCO (Lin et al. 2014). The Kodak dataset is popular reference dataset in image compression literature, and we use it for our small-scale exploratory studies in Sections 3.1 3.2 and 3.4. But the Kodak dataset consists of only 24 images, and our predictive models of SIREN encoding error require larger datasets for training and evaluation. Therefore, we train 300,000 SIRENs on a set of 100,000 images from the the MSCOCO dataset, all downsampled and center-cropped to 512x512 pixcls.
For each of these 300,000 SIRENs, we record their hyperparameters, training loss curves, and BigghsttPPNnn weights after 20,000 steps of training. The primary dependent variable of interest throughout this paper is the maximum PSNR value achieved during training. These 300,000 SIRENs are divided into two datascts. 100,000 of these SIRENs all share the same architecture, and only vary by the image that they encode. We call this collection of SIRENs the single-architecture dataset. The remaining 200,000 comprise the many- architecture dataset, and they are trained across a range of architectural hyperparameters. In total, these 300,000 SIRENs took roughly 12,000 GPU hours to train. Sections 2.3 and 2.4 describe these two of SIRENs in more detail.
2.3 The Single-Architecture Dataset
We train 100,000 SIRENs on the 100,000 images in our random sampling of the MSCOCO datasct. Each image downsampled 224x224 pixels and then used train SIREN with 8 layers, 32 hidden units per layer, and wp of 15. Tf store the SIREN parameters at 16-bit precision, this corresponds to 0.9
3

---

lests splits are done such that no Published in Transactions on Machine Learning Research (05/2024)
bpp representation of each image. The PSNRs of the resulting images have mean of 30.53 cB, standard deviation of 4.41 dB, and are Gamma-distributed.
2.4 The Many-Architecture Dataset
We also train collection of SIRENs over range of hyperparameter values. For this group of SIRENs, we select 200,000 random samples of training parameters from the following range of options:
Target image All images were chosen from our subset of 100,000 photographs randomly sampled from the MSCOCO dataset. The image appears in two splits.
Image size- We train our SIRENs on square images between 112 and 512 pixels in width.
Network Depth Following Dupont et al. 2021 sweep through networks between 2 and 12 layers deep. We notice dropoff in performance at both ends of this distribution, suggesting we have swept the useful range.
Network width Each SIREN we train has a uniform "width," " Le. a uniform number of hidden units per layer. We sample the number of hidden units such that the SIRENs' compression ratios follow log-uniform distribution between 0.5 and 9 bits per pixel (bpp).
Wo This is an important SIREN hyperparameter which determines the spatial frequency of the activations which come of the first layer. Sitzmann et al. (2020) found that wh of 30 worked well for their applications. Consistent with findings from previous works (Yüce et al. 2022 Benbarka et al. 2022 Zhu et al.) 2024, we find that the optinal choice of wo varies with both the image size and image content. In general, larger images with more high frequency detail benefit from a higher wh
wo. Therefore, we randomly sample a value 7 image size from a log-uniform distribution between 0.02 and 0.12, and set wi = 7 x image size. Like the number of layers, this distribution appears to span the optimal range of we values.
Two important parameters which we do not vary in this paper are 1) number of training steps and 2) learning rate. We find that across all of our networks, a learning rate of 0.001 is near-optimal. We also find that our SIREN networks show diminishing returns in PSNR after around 20,000 training steps. Indeed, during the last 1,000 steps, the median STREN in our dataset improved by only 0.0016 dB PSNR, and the 99% of networks improved by less than 0.070 dB. Therefore, we fix the number of training steps to 20,000 for all our networks.
Because this random sampling includes many suboptimal architecture choices, the PSNR of the many- architecture dataset trends lower than in the single-architecture dataset. The average PSNR of this group is 28.12 dB, the standard deviation is 4.86 dB. Figure 12 in the Appendix shows the gamna-distributed histograms of PSNR values for the single- and mnch-trccitecture datasets.
2.5 Metrics
The main objective of our paper is to predict the maximum PSNR that SIREN will obtain on the image it encodes, given the initial training conditions. Our analysis will use a few basic metrics:
Maximum PSNR:
When training SIRENs, it is common practice to use the version of the model that achieved the best training loss, instead of the final model. We follow this practice and focus on the maximum PSNR seen during training the dependent variable of interest. Unless otherwise stated, "PSNR" or "encoding error" refers to this maximum PSNR value.
RMSE / R² Coefficient / Explained Variance: We also report the accuracy of our predictive models in terms of Root Mean Squared error (RSME), and explained variance (EV). Explained variance is defined as:
4

---

Published in Transactions on Machine Learning Research (05/2024)
EV = VarlY V] Var[Y]
(2)
This metric has a maximum value of 1 and is often expressed as a percentage. the case of lincar regression, explained variance is equivalent to the coefficient of determination, R². In such cases we may refer to the two concepts interchangeably. For example, when we say that model explains 95% of the variance in encoding error, this means that the R²² coefficient between the predicted and actual encoding error is 0.95.
Our predictive models do not account for all factors which affect the final SIREN PSNR. For example, the final PSNR reached depends on the random seed used during training, which not an input to our models. Therefore perfect prediction impossible. When trying to predict variable Y (here, PSNR) from variable X, the irreducible error e, i.e. the minimum possible RMSE that any predictive model can reach is:
Irreducible Error:
VarlY|X ø]
(3)
xm
In our setting, X represents the SIREN's training inputs: width, depth, wo and the target image.
To estimate this irreducible error, sample 10,000 training imputs F2, ................................
.N EN X from each of our SIREN datasets. For each input Ij, sample two PSNRs V1,1, Mh.2 P(Y = =4). We then estimate the irreducible error using the following formula (see Appendix A.6.1 for derivation):
N
Σ
U1,1 y1,2)2
(4)
2NV
3 Experiments
In this project, seek to understand how SIREN encoding error is affected by variety of experimental variables. In addition to training and evaluating our encoding error prediction models, we perform variety of controlled experiments, meant to isolate those experimental variables. This section is organized as follows:
Sections
through
3.4 provide context for understanding the SIREN encoding error landscape.
Scction 3.1
explores the irreducible error of our prediction problem due random variation
Siren PSNR.
Scction 3.2
offers useful rule of thumb for how encoding error varies with model size.
Sections and 3.4 present simple 1-variable linear regression models for predicting encoding error. We will use these models as baseline comparison our more sophisticated encoding error prediction networks.
In Section 3.6 we use Gaussian process regression to predict encoding error from the SIREN hyper- parameters and image features.
Section
3.7 reports the accuracy of our predictive models.
Finally, Section 3.8
demonstrates a practical application of these predictive models.
3.1 Random Variation in SIREN Encoding Error
We find that there significant random variation in what encoding error a given SIREN will reach, basod solely on its random initialization. We quantify this random variation with an estimate of the standard deviation in PSNR among SIR which have been trained identically except for their random initialization.

---

Published in Transactions on Machine Learning Research (05/2024)
For the 10-layer-deep, 28-neuron-wide SIRENs trained on the Kodak dataset from COIN (upont et al, 2021), the random variation is approximately + 0.18 dB.
The magnitude of this random variation decreases with network width, and can be largely attributed to the random initialization of the first layer. The consequence is that very narrow, deep SIRENs like the ones used by COIN, are the most. affected, but this effect can be ameliorated by holding the first layer fixed, or by using meta-learned initialization. (See Appendix A.6 for further explanation.)
This random variation is important to keep in mind when comparing the performance of different SIREN architectures and training schemes. For example, Dupont et al. (2021) conducted small-scale hyperparam- eter sweep suggesting that 10 layers was an optimal depth for their networks.. This result was obtained by training one network al each depth, and selecting the network with the best PSNR. We retrain these networks 10 times each with different random seeds, and then bootstrap a random sample of possible experi- mental outcomes which could have occurrod. We found that 95% of the time, their procodure selects optimal dept.ss between and 13 layers deep. This demonstrates that random variation can play an important role in the outcome of such small-scale experiments.
3.2 SIREN Scaling Follows Power Laws
PSNR vs. model size for KODAK images
As rule of thumb, find that each doubling of the network size improves PSNR by about 2 dB. The literature on scaling laws posits a theoretical basis for this: Bahri et al. (2021) present a simple model of neural networks trained with large amounts of data and limited parameters, which they refer to as the resolution-limited regime. According to their model, MSE loss L should decrease with the number of parameters P according to power law L(P) 2(P-2/d), where Ω represents a lower bound and d is the implicit dimension of the data manifold.
PSNR
30-
25
2.0
8.0
32.0
128.0
512.0
# params (thousands)
Figure 2: Scaling curves for SIRENs trained on the first six images in the Kodak dataset. Each line rep- resents the scaling curve for different image.
Since PSNR is proportional to the log of MSE, this law implics that the PSNR should grow lincarly with log(P). Figure shows PSNR vs. log(P) for SIRENs trained 6 of the 24 Kodak images, for networks with ten layers, and widths from 4 to 128 neurons per layer. For a limited range of network widths, this linear relationship holds up quite well.
However, the exponent of this power law appears to be contingent on our experimental setup. Much smaller or larger STRENs, or STRENs trained on very different data, will have different scaling curves. For example, notice that the image represented by the green line follows a steeper power law exponent than the other five images. And as we zoom out to consider networks which are orders of magnitude larger or smaller than the original COIN network, the relationship between PSNR and log(P) becomes nonlinear.
These power laws allow to make local predictions about how the encoding error will change with the SIREN width. But to predict encoding error across other variables, such as the target image content, we will need a model which depends on those inputs as well.
3.3 Extrapolating from Past PSNR to Future PSNR
Perhaps one of the simplest methods to predict the encoding error SIREN will reach after n training steps is to train the SIREN for m < n steps, observe its encoding error, and extrapolate what the error will be at training step n, using linear regression. For example, the COIN networks are trained on the Kodak dataset for 50,000 steps each. The PSNR of each SIREN after 10,000 training steps explains 99.7% of the variance in PSNR after 50,000 steps. Table [ shows the explained variance (i.e. R² coefficient) you can reach with this prediction method for each of our datasets.
6

---

Published in Transactions on Machine Learning Research (05/2024)
Table I: Explained variance (EV) in PSNR after n training steps from the PSNR after m training steps, as calculated by R² score. For example, the PSNR after 1,000 steps explains 95.8% of the variance in PSNR after 50,000 steps for the SIRENs trained with the images and architecture from COIN (Dupont et al. 2021
EV (%) after m steps f
Dataset
n steps
100
1,000
10,000
COIN
50,000
2.5
95.8
99.7
single-arch. 20,000
25.8
93.0
99.8
multi-arch. 20,000
20.7
90.9
99.9
We would like to predict the encoding error to a high accuracy in orders of magnitude less time than it takes to train the network. Table I shows that this simple extrapolation method is inadequate to achieve this goal. By step 10,000, these STRENs are approaching their asymptotic behavior of sharply diminishing returns in PSNR per training iteration. In this regime, there is strong correlation with the final PSNR, but only because the PSNR doesn't change very much in the remaining training steps: From step 10,000 to step 50,000, the COIN networks only improve by an average of 0.4 PSNR. We would be better off predicting the final PSNR using an equivalent JPEG compression, as explained in Section 3.4 In the next section, we will see that we can do better by solving this regression problem using neural nets.
3.4 Comparing SIREN and JPEG Compression
By storing trainable weights at 16-bit precision, Dupont et al. (2021) use SIRENs as compressed image representation competitive with JPEG2000 at low bitrates. We find that there is strong correlation between how well JPEG and SIREN compression can fit given image. This suggests a simple technique for predicting SIREN encoding error: find the encoding error of a SIREN architecture on a few images, and then use a standard zeroth-order numerical optimizer (e.g. scipy.optimize) to find the corresponding JPEG compression ratio which maximises the correlation between JPEG and SIREN encoding error.
Using the Kodak dataset and SIREN hyperparameters from Dupont et al. (2021). ) we train a SIREN on each image to obtaín 0.3 bits-per-pixel (bpp) compressed reprosentation of each image. Using JPEG2000 and compressing to 0.34 bpp, we find that the JPEG's PSNR explains 98% of the variance in SIREN's PSNR
(See Figure 3a
This correlation is not quite as strong for our single-architocture SIREN datasct. Comprossing these images to 0.9 bpp using JPEG2000 explains 96% of the variance in SIREN PSNR. JPEG PSNR predicts the SIRENs' PSNR to within 1.0 dB RMSE. See Figure 3h
Comparing JPEG and SIREN compression for many-architocture dataset is more complex. For the Kodak and single-architecture sets of SIRENs, we treated JPEG compression as one- one-parameter predictive model of SIREN PSNR. But with the multiple-architeeture dataset, each SIREN architecture is best predicted by a different JPEG compression ratio. Therefore, we predict the encoding errors using a Gaussian Process (GP) regression model using the following features: JPEG PSNR on the target image at compression ratios of 7, 25, and 100, and the three STREN hyperparameters: width, depth, and wOi. Coincidentally, this regression model predicts the SIREN's PSNR within 1.1 dB RMSE. For more details, soe Scction 3.6
Additionally, for each trained SIREN network in our dataset, we find the JPEG compression ratio that reaches the same PSNR, and then analyze the relationship between SIREN bpp and equivalent JPEG bpp. For each image size, we obscrve a lincar relationship between the compression ratio of our hyhimal-yyperparamttrr SIREN networks (as measured in bits per pixel) and the equivalent JPEG compression ratio, as can be seen in Figure Bc Our single-architecture dataset mostly contains SIRENs at bpps which are not competitive with JPEG, but notice that extrapolating the lines to the left indicates that our SIRENs should outperform JPEG at between 0.1 and 0.3 bpp. This is consistent with Figure of (Dupont et al. 2021). which shows SIRENs outperforming JPEG compression below 0.3 bpp.

---

Unfortunately, we did not find success in applying this approximation our networks. Following the method from Zancato al. (2020), approximate the empirical NTK of the network at initialization and used this extrapolate the network's learning curve. For problem, these NTK-basod extrapolations saturate Published in Transactions on Machine Learning Research (05/2024)
SIREN bpp vs. equivalent JPEG bpp across image Sizes
Kodak Dataset (R-00.811)
MSCOCO Dataset (R'=0.959)
50
2,00
212 312px
2.4 BPP SIREN PSNR
45
equivalent Jpeg hpp
1.75
312 412px
3.50
X-y
1.00
0.75
30
0.50
25-
20
0.5
1.5
2.5
0.34 BPP JPEG PSNR (a)
0,9 BPP JPEG PSNR
SIREN bpp (c)
(b)
Figure SIREN JPEG representations. In (a) and (b), each data point represents image, and the PSNR to which it can be compressed by JPEG and COIN respectively. In (c) we show which JPEG compression ratio reaches the same PSNR as COIN for given image size and COIN compression ratio.
3.5 Extrapolating training curves using the empirical NTK
In settings, gradient descent of the empirical, finitewwitth NTK is known to be good and efficient approximation for optimizing the neural network itself (Mohamadi et al. 2023 Zancato et al. (2020) used this approxination to successfully predict the training times on fine-tuning tasks to within 20% accuracy, With full-batch training, the MSE loss L of the NTK approximation after t steps of training with a learning rate develops according to the following equation (Arora et al. 2019):
[()) -[(I- -0)
where is the NTK matrix, yo is the network's output at initialization, and y is the target output.
very early, predicting that the werw will essentially learn flat-colored image where each pixel just takes on the average color of the target image.
Others have found that even when the NTK at initializa- tion a poor proxy for nelwork convergence, the empiri- cal NTK can become significantly better approximation after just few steps of training (Kopitkov & Indelman 2020). To test if this held for problem, took snap- 1,2,4, 4..., 215
SIREN training curve vs. empirical NTK extrapolations
20
shots of our SIRENs' weights after training steps, and used the NTK approximation from Zancato et al. (2020) to extrapolate the learning curves forward from that point onward.
16
PSNR
14
12-
10
Figure shows the true training curve vs. empirical- NTK-extrrapllate training curves starting from 1,2, 4, 2..,, training steps. It is clear that the SIRENs and their NTKs bchave very differently. At first, the NTKs produced from snapshots of the network early in training follow the exact same dynamic described above: they learn only the DC component of the target function, and then saturate. Around 128 training steps, the learn- ing rate of n 0.002 leads to divergent behavior on the NTK model. This is due to the increasing eigenvalues
Curve
6-
NTK extrapalations
100
200
500
600
Step
Figure 4: A typical example of a SIREN learning curve (thick blue line) vs. NTK-based extrapo- lations of that learning curve.
8

---

Published in Transactions on Machine Learning Research (05/2024)
image features: 2.268
MLP acts 0.670
acts 0.579
acts 3: 0.563
JPEG proxy: 1.061
50
40
gredicted
prodicted
prodictto
presiices
predicted
30
10
10
20
30
40
50
10
20
30
50
30 actual
40
50
10
20
30
40
50
10
20
30
40
50
actual
adtual
actual
actual
Figure 5: Actual vs. predicted PSNR for GP regression models trained on several different input features. From left to right- "full image features": 512-dimensional features extracted from each image using the CNN-component of our PSNR prediction network, concatenated with the SIREN hyperparameters, then 'MLP acts': three different 256-dimensional features from the activations after each layer of the 3-layer MLP regression head, then 'JEEG proxy': JPEG2000 proxy PSNRs derived by compressing the image to encode using compression ratios of 1:7, 1:25, and 1:100, concatenated with the SIREN hyperparameters.
of the kernel matrix θ according to Equation 5 SGD will begin to diverge when the largest eigenvalue of θ exceeds J. Instead of decreasing, the loss increases exponentially with each training step.
As mentioned in Scction 4.3 NTK theory good approximation for the behavior of overparmetteiied networks. Our experiments here simply confirm that NTK-based predictions break down in the underpa- rameterized regime.
3.6 Predicting Encoding Error using Gaussian Process Regression
In Neural Architecture Search and hyperparameter optimization problems, it is common to model the net- work's performance across different hyperparameters using Gaussian Process (GP) Regression (Snoek et al. 2012). Our problem very similar a traditional neural architecture search problem, except that ours is also highly dependent on the image to encode. To apply GP regression successfully, we must extract appropriate image features.
Building our findings from Section 3.4. we calculate the PSNR of JPEG2000 compression of our images at three different compression ratios: 7x, 25x, and 100x. We then fit a GP regression model on these PSNRs, plus our SIREN hyperparametors, to predict final encoding error. Duc to the higher time complexity of GP regression compared to gradient descent, we cannot. feasibly fit this regression model on the hundreds of thousands of samples in our training set. Therefore, our GP regression models are fit on just 1,000 random samples from the training set. This GP regression model obtains an RMSE of about 1 dB PSNR, as compared to the 0.55 dB PSNR of our best prediction network. However, the GP model is more sample efficient, and the neuraa-network-bbssel approach only out ourperfomms it when given at least 5,000 training samples.
Additionally, we try fitting a GP model using features extracted using our pre-trained PSNR prediction network. We try extracting features from 1) the convolutional network which outputs a 512-dimensional image feature, and 2) layers 1, 2, and 3 from the multilayer perceptron (MLP) regression head, each of which is a 256-dimensional feature. When using the features from layer 3, we are essentially replacing the lincar regression head in the last layer of our neural network with GP regression. In this case, average RMSE remains about the same, whether using linear regression GP regression.
For further implementation details, see Section A.1
3.7 Encoding Error Prediction Networks
Given an image encode and SIREN hyperparameterrs, can we predict the encoding error the SIREN will attain? To answer this question, we trained neural networks on this regression problem for both our single- and many-architecture datasets. Our models optimized minimize the MSE of their PSNR predictions.
9

---

Published in Transactions on Machine Learning Research (05/2024)
For the single-architecture dataset, all aspects of encoding besides the target image are held constant. We train convolutional network to take in target image, and predict what PSNR our SIREN will reach on that image (see Appendix A.2 for details). We chose a variant of Normalization-Free Network from [rock ct al. h021 as our backbonn CNN architecture, due to its high accuracy on the validation set. Appendix A.3 describes our backbone architecture selection procedure in more detail.
For the multiple-architecture dataset, we have several independent variables from which to predict PSNR: the training image, Wa, network width, and network depth. We feed all of these variables into a neural network whose architecture is depicted in Figure !
To train these models, we freeze the pretrained weights of the backbone image classifier and optimize the randomly-initiaaizee "regression head" on the network for 10 epochs. Then we unfreeze the entire network for an additional 10 epochs. We use an 80/10/10 train/vatt /eaaivvvaviittteeeee split, and select the version of the network which obtained the best validation set accuracy during training-
As performance baseline, Section 3.4 showed that we can predict SIREN PSNR from JPEG compression PSNR with over 94% explained variance. Section 3.3 explores a simple method for predicting SIREN encoding error by only training the SIREN for small number of iterations, and then extrapolating.
Our encoding error prediction networks outperform these simple baselines. On the single-architecture dataset, our model predicts the SIRENs' PSNR to within 0.30 dB RMSE, with an Ri score of 0.996. The irreducible error of this prediction problem is approximately 0.21 dB PSNR. The many-architecture predictor is accurate to within 0.55 dB RMSE, to an R² score of 0.987. A portion of the higher error comes from higher írreducible error of 0.27 dB RMSE, due the inclusion of some suboptimal SIREN architectures which have very high random variation in PSNR.
For the single-architctury dataset, the prediction error residuals are very nearly normally distributed. In- terestingly, this is not the case for the many-architecture dataset, where the prediction error magnitudes are roughly exponentially distributed, with a median (i.e. "half-life") of 0.185 dB PSNR.
A key limitation our models is how much training data they require. The costliest step of producing these models is generating their training data, which requires training tens of thousands of SIRENs. Figure t reports the model's test accuracy for successive halvings of the training set size, compared to two far more sample-efficient baseline predictive models discussed above. The scaling curve is remarkably uniform, and suggests that even larger training sets would continue to improve test accuracy.
With a forward-pan time of 70 ms on a Titan X GPU, as opposed to an original encoding time of 2.5 minutes per network, this allows us predict PSNR 2,000x faster than we could by training the SIREN. For the man--architecture predictor, the speedup is even more significant: If we wish to evaluate multiple SIREN architectures against single image, can feed that image through the CNN-based feature extractor once, and then run only the small MLP regression head for each new model architecture. On our system, this regression head can process tens of thousands of inputs per second. In the next section, we will demonstrate how to take advantage of this speedup to perform an accelerated hyperparameter search.
3.8 Accelerated Hyperparameter Search
By using our encoding error predictor as a fast proxy for SIREN training, we can accelerate any hyperpa- rameter search method by more than 10,000x, as long as 1) the hyperparameters stay within the domain of our training data and 2) the objective function we are trying to optimize is a function of the variables of our predictive model. In this section we explore two toy applications of our PSNR. prediction network to hyperparameter search.
First, given a SIREN size limit and an image to encode, we optimized the SIREN's width, depth, and wo to maximise the expected PSNR. However, we found that this approach did not confer a significant PSNR improvement over using a single SIREN architecture, selected for having the highest average PSNR across a test set of 100 images. This result held across many SIREN sizes and image sizes.
10

---

Published in Transactions on Machine Learning Research (05/2024)
Single Architecture
Single-arch. PSNR
3.0
Prediction net
x
2.5
95th percentile prediction error
PSNR RMSE
2.0
JPEG proxy
Extrapolated from step 1,000
Predicted
40
1.5
Irreducible Error
30
1.0
0.5
1250
2500
5000
10000
20000
40000
80000
20
30
40
50
# Training Samples
(a)
(b)
Many Architecture
Many-arch. PSNR
3.0
50
PSNR RMSE
2.5
2.0
Predicted
40
1.5
30
1.0
0.5
1250
2500
5000
10000
20000
40000
80000
Actual
Training Sam samples
(c)
(d)
Figure Number of training samples vs. test set error for the (a) single and (c) multiple architecture datasets. '95th percentile prediction error" is for the distribution of prediction net errors. Horizontal lines represent 1) the RMSE we get by linearly extrapolating the PSNR from training step 1,000, as described in
Section 3.3 2) RMSE from predicting PSNR with "JPFG proxy model" depicted in Figure 3 and
the irreducible error of the regression problem, as described in Section 3.1 (b) and (d) are scatterplots of
predicted v8. actual PSNR on the test sets, using the full training sets.
11

---

Published in Transactions on Machine Learning Research (05/2024)
PSNR 95% Cls vs. model size
Second, chose different optimization problem: given target image, find the smallest STREN net- work which will achieve at least 30 dB PSNR. How- ever, since our PSNR predictions are approximate; we cannot guaranlee that the resulting PSNR will exceed 30. Assuming that the predictive model's er- ror is normally distributed, we should aim for a pre- dicted PSNR. 2 standard deviations in error above 30 dB to be 95% confident we will meet the require- ment. In this context, the standard deviation of the error, or RMSE of our predictive model, plays a de- cisive role in determining how small of a SIREN we can use.
34
JPEG proxy
32
our method
PSWR
30
28
26
24
3
model size (bpp)
Figure 7: PSNR 95% confidence intervals, predicted by our predictive network (our method) and the JPEG proxy. The two vertical lines indicate the model sizes prescribed by these two predictive models to be 95% sure the PSNR exceeds 30.
As comparison to this hyperparameter search, build baseline using the JPEG-based PSNR pre- dictor from Scction 3.4 We find sequence of 30 SIREN architectures of increasing size on the Pareto frontier of the network size / average PSNR tradeoff. Then we estimate the 'effective JPEG compression ratio' for each of these SIREN architectures, as described in Section 3.4 Then, we use JPEG encoding error as a fast prediction of SIREN encoding error, and select the first model which we are 95% confident will exceed 30 PSNR. For more details, see Appendix A.5
As noted in section 3.7 the JPEG proxy method is less accurate than our prediction network. In the context of this problem, the superior accuracy of our neworrk-bssed predictive model translates directly into more efficient SIREN selection. Figure 7 illustrates this process. The two predictive models, JPEG compression and the PSNR predictor, provide different 95% confidence intervals in the PSNR that the optimal modol of a given size will reach on a given image. A lighter confidence interval allows us to select a smaller model closer that threshold, In our case, using the PSNR predictor allows us select models which are on average 12% smaller than those selected using the JPEG proxy method.
4 Related Work
4.1 Implicit Neural Representations
The field of implicit neural representations, INRs, is very broad, and in this section will mention just few of the many alternative coordinate network architectures. Tannik et al. (2020) showed that using random Fourier features as a positional encoding significantly improved the accuracy of implicit neural representations. SIRENs (Sitzmann et al. 2020) extend this idea by using wave activations, effectively turning the first layer of the network into trainable positional encoding layer. Since then, many other positional encodings and activation functions for INRs have been proposed: [Ramasinghe & Lucey (2022) present unified framework for reasoning about the influence of different activation functions. and show that Gaussian activations are more stable to random initialization than SIRENs, (Saragadan et al. (2023) propose wavelet- based activation function which further improves the accuracy of implicit neural representations. FINER (Liu et al. 2024) uses a variable-frequeng activation function to further increase the performance of INRs. There have been many more proposed schemes for INRs, such as hash-based mothods (Müller et al. Zhu et al., 2024), or multiplicative filter networks (aathony el al. (202)), but SIRENs remain the most commonly used baseline, and this is why we choose them as our subject of study.
2022
4.2
Compression with Implicit Neural Representations
INR-basod data compression is the application most directly relevant our rescarch. Since the introduction of the SIREN network by Sitzmann et (2020), researchers have sought to use SIRENs as compressed repre- sentations for images. In 2021 Dupont al. (2021) showed that SIRENs putperform JPEG compression
12

---

Published in Transactions on Machine Learning Research (05/2024)
at low bitrates. We use their COIN (COmpression with Implicit Neural representations) architecture and training setup as a baseline for our experiments.
Since then, several schemes have been devised to improve a SIREN's compression capabilities. These schemes
employ a combination of meta-learning (Dupont et al. 2022b Schwarz & Teh 2022 Lee et al. 2021 Strüm-
pler et al.3 2022), weight pruning (Lee et al. 2021 Ramirez & Gallego- Paragg--- 2022 and quantization
(Gordon et al. (023) to improve the rate-distortion curves of SIREN-based image compression. Guo et al. (2023) extend this compression paradigm to Bayesian neural networks. All of these projects attempt to obtain the best-possible rate-distortion curves for some set of images, although the size and content of the imagos studied varies across papers. Gao et al. (2023) propose a new loss term to enforce structural con- sistency between the reconstructed and ground truth images based on their segmentation maps. Almost all of these works build upon the SIREN architecture; therefore we believe that our observations about STREN-based image compression are broadly relevant to this whole body of work.
Dupont et al. (2022b) show that INRs can be used to compress many other data types, including sound, MRI, and weather data. Video is an especially common and data-intensive modality which has received
special attention:
while Sitzmann et al. (2020) and subsequent papers such as Mehta et al. (2021 applied
SIRENs to relatively small (e.g. 7x224x224) videos, more specialized INR architectures such as NeRV (Chen
let al.
2021a), F-NERV Li et al.] (0022, NVP Kim et. al.] (2022), and many others (Rho et al. 2022 Zhang
le al.
2022 He et al. 2023 Chen et al. 2022, Maiya et al. 2023 have reached performance comparable
with conventional video compression methods such as H.264 (Wiegand et al.) 2003) and HEVC (Sullivan
et al.
2012). While outside the scope of the current work, our encoding error prediction method could be extended to these hehe modalities.
4.3 Theoretical Models of Neural Networks
Our primary goal is to develop a predictive model of how well SIREN can encode a target image. While our methods are data-driven and empirical, others works have developed relevant theoretical models of network training dynamics, which may be predictive of SIREN performance. In particular, Neural Tangent Kernel (NTK) theory (Jacot et al., 2018) offers one possible tool for modelling the performance of SIRENs. As networks become sufficienlly overparameterized, they enter lazy training regime, where their training is well-approximated by Taylor series expansion around their initialization {Atanasov et al. 2023). Even outside of this lazy training regime, researchers have successfully applied NTK theory to explain certain properties of coordinate networks. For example, Tancik et al. (2020) use NTK theory to explain the advantage conferred by using a random Fourier feature (RFF) positional encoding for coordinate networks. As they show, applying NTK theory to neural networks with an RFF positional encoding naturally leads to a frequency-basec perspective of the network's 's behavior.
There are many works based upon or inspired by a frequency-based perspective on implicit neural represen- tations. Ronen et al. (2019) perform more extensive analysis of the convergence rate of neural networks to functions of different. frequencies, and confirm their theory matches observat ions for a few circumstances. Yúce et al. 2022) employ different mathematical approach to reach similar conclusions about the spectral bias of the fourier-feature networks and SIRENs, Ramasinghe & Lucey (0022) analyze the choice of activa-
tion functions in implicit neural representations Benbarka et al. (2022) and Zhu et al. (2024) also introduce methods motivated by Fourier analysis of implicit neural representations.
However, none of these works precisely predict the loss curves of underparamellrrzeed networks, which lack enough parameters to fully fit their training data. While some general arguments from NTK theory still apply, the approximation is too rough to make the kind of fino-grained PSNR predictions we seek.
For example, in Predicting Training Time Wilhoul Training, Zancato et al. (022)) use neural tangent kernels to predict the fine-tuning loss curves of pretrained image classification networks. On the surface, their problem extremely similar ours. But their networks operate in the lazy training regime, while ours do not. Due to this difference, we found that their method does not transfer well to our setting. See Appendix 3.5 for more details.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Bahri et al. 2021 propose a theoretical model which is more applicable to setting. They identify four regimes of neural network training dynamics, including resolulion-limital regime, where the number of training samples far exceeds the number of parameters. They argue that the model's loss will decrease as a power law of the number of trainable paramcters. In Section 3.2 we show that this model is in closer accordance with our observations. However, this power law only tells us how the encoding error will vary with model size, with everything else held constant. We are specifically interested in how PSNR varies with the training data, and here their model offers no guidance.
4.4 Meta-Learning Implicit Neural Representations
Meta-learning, such as MAML (Finn al. 2017). optimizes the initial weights of a neural network converge quickly to any of set of target functions. Mcta-lcarning dramatically reduce the amount of time needed to fit SIREN to given signal (Tancik et al. 2021 Dupont et al. 2022a Lee el al. 2021). Because meta-learning optimizes the initial weights of the network, it implicitly holds the network's hyperparameters, such as its width and depth, fixed. However, our encoding error prediction model works over a continuous range of possible hyperparameters.
4.5 Neural Architecture Search
Our problem is very similar to the problem of neural architecture search (NAS). In both cases, the challenge is to predict the performance given neural network architecture before training it. NAS a mature field with a great variety of established solutions. Here, we connect our work to just a few sample papers from this broad field.
We wish to predict how the network's performance varies with both architecture and training task, i.c the image to encode. However, the name implies, NAS focuses primarily variations in architecture, while task-awere NAS methods are rarer. SMAC (Hutter et al., 2011) can predict model performance from a mixture of both hyperparameters and task meta-features. However, this method is based on a bayesian optimization algorithm which becomes slow for large sample sizes, and not well-suited for highly complex, high-dimensional features such as images. Istrate et al. (2019) introduce TAPAS, a predictive model of neural network accuracy which also considers the difficulty of the training data using a method from [Scheidegger et al. 22021). Kokiopoulou et al. (2020) learn embedding space for image classification datasets, which are then used to make task-aware predictions of neural architecture performance. At high level, our problem and approach are very similar to theirs. The key difference lics in the choice of domain: we study SIRENs trained on single images, as opposed to classification nelworks trained on labeled image datasets
4.6 Predicting NeRF Parameters
While we train a neural network to predict a coordinate network's encoding error on a target signal, other works go a step further, training "hypernetworks" which generate the coordinate network to model the signal. For example, Lorraine et. al.) (2023) and Xie et al. (2024) produce NeRFs from textual descriptions. Hong et al. (2023) Pavilo et (2023 generate Signed Distance Functions (SDFs) and NeRFs from monocular views of an object.
We place "hypernetworks" in quotes here because these networks do not directly output the NeRF's param- eters, as is common for true hypernetworks (Ha et al. 2016); instcad, their output modulates the activity of a pretrained NeRF somehow, either by directly modulating the activations (Lorraine el. al.] 2023), or by generating triplanes which are used as inputs to the NeRF NeRE/SDF (Hong et al., 2023 Pavllo et al. 2023). As for neural architecture search, there are many more works on this topic than we have cited; we are simply pointing out few sample projects. Our dataset, which contains the trained weights of 300,000 SIRENs, may be of interest to those who study coordinate hypernetworks.
14

---

Published in Transactions on Machine Learning Research (05/2024)
5 Conclusions
We have observed several important trends in SIREN encoding error. First, SIRENs have significant random variation in performance between random initializations, which is mostly explained by the first layer. This effect gets more pronounced as the networks become narrower. Pre-selecting a high-quality positional encod- ing layer instead of sampling one randomly appears to be an easy win for improving the encoding error of a narrow SIREN. We also find that SIREN PSNR correlates strongly with JPEG PSNR. SIRENs outperform JPEGs when the representation is very small: representing small image
very low bpp.
We have shown that relatively standard deep-learning architecture can solve novel regression problem: predicting the encoding error SIREN will reach on a given image. Our encoding error prediction networks compare favorably to three alternative approaches: 1) using JPEG compression loss as a proxy for STREN loss, 2) linearly extrapolating future PSNR from a few steps of gradient descent and 3) NTK- approxi- mations of SIRENs in Section 3.5 We show that this works for a single SIREN architecture, and also across range of possible architectures We can leverage such predictive models to simulate hyperparameter sweeps over the the SIREN architecture search space 10,000x faster than would otherwise be possible.
6 Discussion
The "holy grail" of this line of research is a thcoretically well-founded model which can predict the training loss (here, encoding error) of underparancterized neural networks with high accuracy. Towards this aim, we suspect. that our dataset of 300,000 SIRENs contains a lot of scientific value beyond the scope of this work. SIRENs trained to fit photographs using mean sauun-suaree-errro loss are a promising "model" deep learning problem, the same way that fruit flies are a common model organism. This model has several advantages:
Architecturally Simple: SIRENs are a simple variation on multi layer perceptrons, which are one of the most ubiquitous architectural building blocks in deep learning.
Computationally Cheap: In the context of our coordinate networks, a single photograph repre- sents an entire training dataset. Both our SIRENs and their training data (photographs) have a low memory footprint, less than megabyte each. Indeed it is this quality which has allowed us publish dataset of 300,000 of them.
Understandable:
Perhaps no naturalyy-ocourring kind of dataset is more amenable to visualization than a photograph. The training samples, image pixels, are positioned on regular grid, making them uniquely amenable to mathematical analyses such the discrete Fourier transform. The loss function used to train our models, La loss, is extremely well-studied and mathematically well- behaved.
Our 300,000 trained SIRENs provide lot of empirical data about how simple neural networks learn to approximate functions. We hope that rescarchers who are interested in modeling the behavior of neural networks, especially networks in the resolulion-limited regime (Bahri et al. 2021), can evaluate their theories against our data.
References
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322 332. PMLR, 2019.
Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, and Cengiz Pehlevan. The onset of variance- limited behavior for networks in the lazy and rich regimes. In The Eleventh International Conference on Learning Representations, 2023.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Xio preprint (xxx:2102.007000 2021.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Nuri Benbarka, Timon Höfer, Andreas Zell, et al. Seeing implicit neural representations fourier series. In Proceedings of the IEFE/CVF Winter Conference on Applications of Computer Vision, pp. 2041-2050, 2022.
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recog- nition without normalization. In International Conference Machine Learning, pp. 1079-1071. PMLR, 2021.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. Aduances in Newral Information Processing Systems, 1-211577-21688, 2021a.
Hao Chen, Matt Gwilliam, Bo He, Ser-Nam Lim, and Abhinav Shrivastava. Cnerv: Content-adaptive neural representation for visual data. arXiv preprint arXiv:2211.10421 2022.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628-8638, 8638, 2021b.
Emilien Dupont, Adam Golinski, Milad Alizadoh, Yee Whye Tch, and Arnaud Doucet. COIN: with implicit neural representations. In Neural Compression: From Information Theory to Applications Workshop @ ICLR 2021, 2021.
COmpression
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5694 5725. PMLR, 17 23 Jul 2022a.
Emilien Dupont, Hrushikesh Loya, Milad Alizadch, Adam Golinski, Yee Whye Tch, and Arnaud Doucet. COIN++: Neural compression across modalities. Transactions on Machine Learning Research, 2022b. ISSN 2835-8856.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. In International Conference on Learning Representations, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocu- lar video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5712 5721, 2021.
Harry Gao, Weijie Gan, Zhixin Sun, and Ulugbek S Kamilov. Sinco: A novel structural regularizer for image compression using implicit neural representations. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Cameron Gordon, Shin-Fang Chng, Lachlan MacDonald, and Simon Luccy. On quantizing implicit neural representations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 350, 2023.
Zongyu Guo, Gergely Flamich, Jiajun He, Zhibo Chen, and José Miguel Hernández-Lobato. Compression with bayesian implicit neural representations. arXiv preprint arXiv:2305. 211.:23055.9185, 2023.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. International Conference on Learning Represen- tations, 2016.
Bo Hc, Xitong Yang, Hanyu Wang, Zuxuan Wu, Hao Chen, Shuaiyi Huang, Yixuan Ren, Ser-Nam Lim, and Abhinav Shrivastava. Towards scalable neural representation for diverse videos. In Proceedings of the IFFF/CVF Conference on Computer Vision and Pattern Recognation, pp. 6132 6142, 2023.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Trm: Large reconstruction model for single image to 3d. ar preprinl arXiv:2311.044000 2023.
Frank Hutter, Holger И Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5, pp. 507-523. Springer, 2011.
Roxana Istrate, Florian Scheidegger, Giovanni Mariani, Dimitrios Nikolopoulos, Constantine Bekas, and Adelmo Cristiano Innocenza Malossi. Tapas: Train-less accuracy predictor for architecture search. Tn Proceedings of the AAAI conference on artificial intelligence, pp. 3927 3934, 2019.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Subin Kim, Sihyun Yu, Jaeho Lee, and Jinwoo Shin. Scalable neural video representations with learnable positional features. Advances in Neural Information Processing Systems, 11112718-1273,1 2022.
Kodak. Kodak dataset, 1991. URL http: https///.....g/rraaa.cs///oo.o
Efi Kokiopoulou, Anja Hauth, Luciano Sbaiz, Andrea Gesmundo, Gábor Bartók, and Jesse Berent. Task- aware performance prediction for elficient architecture search. In ECAT 2020, pp. 1238-1245. IOS Press, 2020.
Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In Artificial Neu- ral Networks and Machine Learning ICANN 2020: 29th International Conference on Artificial Newral Networks, Bratislava, Slovakia, September 15-18, 2020, Proceedings, Part II 29, pp. 168-179. Springer, 2020.
Jaeho Lee, Jihoon Tack, Namhoon Lee, and Jinwoo Shin. Meta-learning sparse implicit neural representa- tions. Advonces in Neural Information Processing Systems, 14:11769-11777 2021.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6498 6498-6508, 6508, June 2021.
Zizhang Li, Mengmeng Wang, Huaijin Pi, Kechun Xu, Jianbiao Mei, and Yong Liu. E-nerv: Expedite neural video representation with disentangled temtalltemporal context. In European Conference Computer Vision, pp. 267-284. Springer, 2022.
Tsung-Y: Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Swilzerland, Seplember 6-12, 2014, Proceedings, Parl V 13, pp. 740 755. Springer, 2014.
David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVF conference on computer vision and pallern recogniition, pp. 16252 16262, 2022.
Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexible spectral-bias tuning in implicit neural representation by variable-periodic activation functions, 2024.
Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung- Yi Lin, Ming- Yu Liu, Sanja Fidler, and James Lucas. Att3d: Amortized text-to-3d object synthesis. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 17946-17956, 2023.
Long Mai and Feng Liu. Motion-adjustable neural implicit video representation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Patlern Recognation, pp. 10738-10744 2022.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Shishira R. Maiya, Sharath Girish, Max Ehrlich, Hanyu Wang, Kwot Sin Lee, Patrick Poirson, Pengxiang Wu, Chen Wang, and Abhinav Shrivastava. Nirvana: Neural implicit representations of videos with adaptive networks and autoregressive patch-wise modeling. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14378-14387, June 2023.
Julien NP Martel, David B Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Weizstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint orXiv:2105.022 027888 2021.
Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chan draker. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14214-14223, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1))99-106, 2021.
Mohamad Amin Mohamadi, Wonho Bae, and Danica J Sutherland. A fast, well-founded approximation to the empirical neural tangent kernel. In International Conference on Machine Learning, pp. 25061-25088 PMLR, 2023.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1-15, 2022.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021.
Dario Pavllo, David Joseph Tan, Marie-Julie Rakotosaona, and Federico Tombari. Shape, pose, and ap- pearance from a single image via bootstrapped radiance field inversion. In IEEE/CVE Conference on Computer Vision and Pattern Recognétion (CVPR), 2023.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards unifying framework for activations in coordinate-mlps. In European Conference Computer Vision, pp. 142 158. Springer, 2022.
Juan Ramirez and Jose Gallego-Posada. L 0 onie: Compressing coins with 1 0-constraints. arXiv preprint arXiv:2207.044. 2022.
Daniel Rho, Junwoo Cho, Jong Hwan Ko, and Eunbyung Park. Neural residual flow fields for elficient video representations. In Proceedings of the Asian Conference on Computer Vision,
2022.
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems, 32, 2019.
Vishwanath Saragadan, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G. Baraniuk. Wire: Wavelet implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18507 18516, June 2023.
Florian Scheidegger, Roxana Istrate, Giovanni Mariani, Luca Benini, Costas Bekas, and Cristiano Malossi. Efficient image dataset classification difficulty estimation for predicting deep-learning accuracy. The Visual Computer, (6):1)59 1610, 2021.
Jonathan Schwarz and Yee Whye Teh. Meta-learning sparse compression networks. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL. http:///ggaaeeeeeeeeeeeeeee/eees
18

---

Published in Transactions on Machine Learning Research (05/2024)
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representation ions with periodic activation functions. Advances in Newral Information Processing Systems. 33:7462 7473, 2020.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Yannick Strümpler, Janis Postels, Ren Yang, Lue Van Gool, and Federico Tombari. Implicit neural repre- sentations for image compression. In Computer Vision ECCoV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pp. 74-91. Springer, 2022.
Gary J Sullivan, Jens-Raincs Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions on circuits and systems for video technology, 22(12): 1649-1668, 2012.
Matthew Taneik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537 75477, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt,, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-basod neural representations. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 2846 2855, June 2021.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular vidco. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22959-12977 2021.
Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology, 13(7):560-57)) 2003.
Ross Wightman. Pytorch image models. http:://ss......ooo./.rraaaaaaas
2019.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2021.
Recognition, pp.
Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, and Xiaohui Zeng. Latte3d: Large-scale amortized text-to-enhanced3d synthesis. arXiv preprint arXiv:2403. 15385, 2024.
Gizem Yice, Guillermo Ortiz-Jiménez, Beril Besbinar, and Pascal Frossard. A structured dictionary per- spective on implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion (CVPR), pp. 19228-19238, June 2022.
Luca Zancato, Alessandro Achille, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Predicting training time without training. Advances in Neural Information Processing Systems, 68:6136-6146, 2020.
Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022. URL
https://oppenevia net/ fe///foormmiirrrgeeee2dd---
Hao Zhu, Shaowen Xie, Zhen Liu, Fongyi Liu, Qi Zhang, You Zhou, Yi Lin, Zhan Ma, and Xun Cao. Disorder-invariant implicit neural representation. IEEE Trensactions on Pallern Analysis and Machine Intelligence, 2024.
19

---

Published in Transactions on Machine Learning Research (05/2024)
A Appendix
A.1 Gaussian Process Regression Implementation Details
All input features for our GP regression models the neural activation vectors, JPEG2000 PSNR scores, and SIREN hyperparameters- were normalized to have a standard deviation 1. Each GP Regression model was fitted on 1,000 images from the training set, and then evaluated using 10,000 samples from our test. set.
All of our GP regression models use a composite kernel consisting of a constant kernel multiplied by a Radial Basis Function (RBF) kernel, plus a white kernel. The white kernel is intended to model the irreducible error in our data, Section 3.1 We used standard GP Regression via Learn's Gaussian Regresss class.
A.2 Encoding Error Predictor Implementation Details
A.2.1
Single Architecture Implementation Details
To train our single-rcchitecture encoding error predictor, we use the ADAM optimizer with a learning rate of 0.0001 and a batch size of 8. PSNR values are normalized to have a mean of 0 and a standard deviation of for the training data. We start from backbone pretrained classification network. (See Section A.3 for details.) We remove the classification head, and replace it with "regression head", fully-connected layer which outputs a single value: the PSNR prediction. We freeze all weights in the network except the regression head and perform 10 epochs of training, after which we unfreeze the entire network for an additional 10 epochs of training. Finally, we select network checkpoint from the epoch with the best validation set accuracy.
A.2.2 Many Architecture Implementation Details
The many-architecture encoding error predictor is slightly more complex, and has more implementation details to consider.
First is the positional encoding, which encodes the SIREN hyperparamotors (width, depth, Wo, and image size), into a vector in R We use the positional encoding scheme from Mildenhall et al. (2021):
(p) (sin(2 (osp) mp), sin(" Tp), cos(22" xp))
(6)
First, the hyperparameters are normalized to lie in the range (0,1). Log-uniformly distributed parameters w and Wi are re-scaled by taking their logarithm first, before normalization. For normalized width w, depth d, image size 8, and wo, the full positional encoding of the hyperparameters is:
T(w, d, s, wp) = (y(w), (s)) (w)))
(7)
This positional encoding and the feature vector from the last layer of the classification network are concate- nated and fed into the MLP regression head. In our experiments, we found that the performance of the multi-architecture classifier was not too sensitive to the width and depth of this MLP head, so we keep the network relatively small for performance reasons: 4 layers deep and 128 hidden units wide.
To train the multi-architecture encoding error predictor, we use the same optimizer, learning rate, and batch size as for the single-architecutre predictor. We start by freezing the weights of the pretrained classifier network, and training just the MLP regression head for 10 epochs. Then we unfreeze the classifier and train the entire network for 10 epochs, seleeting the checkpoint with the epoch with the best validation accuracy.
A.3 Choosing the PSNR Predictor Backbone Architecture
Our single-architccture encoding error predictor simply a fine-tuned image classification network. We used the PyTorch Image models library (TIMM) (Wightman) 2019) to quickly sweep through hundreds of candidate classification networks to fine-tune on our task. Table 8 shows the R² scores the validation
20

---

Published in Transactions on Machine Learning Research (05/2024)
data from our initial sweep over many of the models in the TIMM library. Note that some details of training are not the same in our final model, for example, these models were trained for fewer epochs. Table 2 shows a more thorough evaluation on some of the best models identified in Table 3 Note that for this second sweep, we excluded some of the best-performing models from Table 3 due to their slow training times.
the end, this sweep led us choose ECA NFnet. LO, a TIMM-specific variant the NENET architecture from Brock et al. (2021) as our final model architecture.
Model
Validation R²
Test R
oca nfnet LO nfnet LO
0.9956
0.9956
0.9954
0.9955
nfnet L1
0.9952
0.9953
coat lite mini nf regnet b1 convnextv2 large coat lite small
0.9933
0.9936
0.9932
0.9933
0.9923
0.9923
0.9922
0.9924
Table 2: 7 final architectures we tested before making a final decision use nfnet for the rest of our experiments.
Table 3: 128 backbone classification architectures we tried from the Torch Image Models (TIMM) library [Wightman 2019), on an carly, smaller version of the single-architecture dataset. Ordered by the R score they obtained on the validation set.
model eca nfnet 11 convnext convnext_large large nf [[_rrgee__] bl coat lite lite__iiil mini convnext xlarge in22ft1k coat mini
R²2
model
R²
0.987
eca nfnet 12
0.986
0.984
convnext base
0.984
0.983
convnext base in22k
0.981
coat_lite_______
0.980
convnext tiny
0.980
0.979
nfnet 10 convnext small
0.978
0.977
0.976
nfnet_ 10 coat tiny deit. patch16 224
0.975
convnext base 384 in22ft
0.974
0.973
convnext base in22fclk
0.971
0.967
crossvit tiny 240
0.965
crossvit 15 dagger 408
0.964
coat lite small
0.963
crossvit small 240
0.962
crossvit 15 240
_40
0.962
crossvit 9 240
0.961
deit. base patch16 384
0.958
pit 224
0.956
crossvit 18 dagger 240
0.956
crossvit 9 dagger 240
0.956
mixer b16 224 miil
0.954
pit b 224
0.953
crossvit base_ _40
0.952
jx_nest base
0.952
jx_nest nest small resnet50
0.951
cait xxs36 224
0.950
0.950
convnext xlarge in22k
0.946
crossvit 15 dagger 240
0.946
crossvit 18 240
0.945
crossvit 18 dagger 408
0.944
coat lite tiny
0.943
resmlp_ 12 distilled 224
0.939
pit li 224 convnext large in22k
0.930
deit base patch16 224
0.929
0.928
resnetv2 50x1 bit distilled
0.927
resnet50 gn fift_xxs24441 224
0.927
gluon seresnext 32x4d
0.926
0.922
resmlp_ 1itp__3__dsssilllede distillldd 224
0.921
regnety 016
0.921
regnety 006
0.918
jx nest tiny convnext large in22ft1k
0.916
res2net50 14w 14w 8s
0.914
0.913
ret__eeeseeelt___________111111e 32x4d
0.912
regnetz_cli c16 _
0.909
regnety_00
0.908
0.907
choon_senet154
0.905
21

---

Published in Transactions on Machine Learning Research (05/2024)
model
R²a
model
eca botnex(26ts 256
0.904
ese vovnet39b
0.904
resnest50d 4s2x40d
0.903
regnety_ 008
0.902
regnety_00e 040
0.902
resnetv2 50x1 bitm
0.901
ccaresnet26t
0.899
gernet
0.899
resnetrs101
0.899
halo2botnet50ts
256
0.899
regnetz_d8
0.898
regnetz_es
0.898
gluon seresnext101 64x4d
0.898
deit small patch16 224
0.897
regnetx_000
0.892
resnetrs200
0.892
resnetrs50
0.890
convit base
0.889
repvgg b0
0.889
eca halonex(26ts
0.888
regnetz d32
0.888
resnet.331s
0.888
resnext26ss
0.887
regnety_ 004
0.887
scresnct152d
0.885
gcresncxt26ts
0.885
resmet26
0.885
regnetx 032
0.884
resnet152d
0.883
seresnext26d_ 32x4d
0.883
pit xs 224
0.883
dpn68b
0.881
regnetx 008
0.881
regnetx 016
0.879
gluon resmet50 vlc
0.877
resnet101d
0.874
resnetv2 50x1 bitm in21k
0.874
resnet50d
0.873
resnetblu550 resnet26t
0.872
cait _ait_s$4____ $24__24
0.872
0.872
regnetx 120
0.871
resnetv2_1011 111e2__00111___________ bitm
0.871
mixer_ b16__11___00 224 miil in21k
0.870
gcresnext-50ts
0.869
lambda resnet26t
0.868
densenetblur12dd
0.867
regnetx_ 006
0.866
gluon_resmett vlc
0.866
mixer 116 224
0.865
regnetx_003
0.865
resnest50d. 1s4x24d
0.865
gluon resmet101 vld
0.865
resnet26d
0.865
lambda_resnet50e resnet50ts
0.865
gmlp_ s16 224
0.864
aiton_resnctt0_vv
0.864
rat_rosncxt26ts
0.863
gernet_
0.861
lamhalobotnet50ts_ 256
0.861
repvgg
0.860
repvgg blg4
0.860
gluon resnext50 32x4d
0.858
regnety 032
0.857
geresnet50t regnetx 080
0.857
resnext26ts
0.856
0.855
gluon resnet vlc
0.852
rognety_(ee 080
0.850
resnet33ts
0.848
22

---

Published in Transactions on Machine Learning Research (05/2024)
A.4 Input Ablation Study
Removed Input
Error +
R f
No inputs removed
0.55
0.987
Which inputs to our model are imporiant for accurately dicting PSNR? Table 4 shows the test error of our multiple- architecture PSNR predictor with various combinations of in- puts to the network removed. We see that removing any of the input features significantly degrades the accuracy of the predic- tion network, indicating that the final PSNR varies predictably with each of these inputs.
Image Size
0.71
0.980
Wg
1.25
0.939
Number of Layers
2.27
0.794
Layer Size All Hyperparamoters Image
2.52
0.746
2.90
0.668
4.11
0.317
Table 4: Many-architocture prodiction error with various removed inpuls.
23

---

Published in Transactions on Machine Learning Research (05/2024)
A.5 Accelerated Hyperparameter Search Implementation Details
In Section 3.8 we look at the problem of selecting which of 30 SIREN architectures of progressively increasing size we should use to encode an image to at least 30 dB PSNR. We compared two methods, one based on the JPEG proxy model from section 3.4 and one based on our PSNR prediction network from Section 3.7
For the JPEG proxy model, need an "effective JPEG compression ratio' for each of our 30 SIREN architectures, which are sized compress the encoded image between 0.5 and BPP. We estimate these ratios by selecting 5 SIREN architectures that span the range of sizes (sizes which compress the image to 0.5, 1, 2, 4, and 8 bpp), and train these architectures on 30 images each. Then for each architecture, we use those 30 images to estimate an effective JPEG compression ratio, as described in Section 3.4 We also estimate the RMSE of our JPEG proxy models. The models' predicted PSNRs, and our estimated RMSEs allow us to estimate confidence intervals for the PSNR value, assuming normality. To get effective JPEG compression ratios and RMSEs for the remaining 25 models, we linearly interpolate those values from the 5 sample models.
We reuse the PSNRs of those same 5005=150 SIRENs to estimate the RMSE of our PSNR prediction network for each SIREN architecture, again, using lincar interpolation.
24

---

Published in Transactions on Machine Learning Research (05/2024)
Random PSNR variation network width Random PSNR variation network depth
random PSNR variation
1.0
0.8
0.6
0.4
0.2
10
20
30
40
50
60
10
12
width (# hidden units layer)
Depth (# layers)
Figure 9: Effect of SIREN width and depth on the random variation in PSNR. Both sweeps begin from a baseline architecture of 10 layers, 32 hidden units per layer. In the width sweep, the depth is held constant, and vice-versa. For each choice of width/height, 10 SIRENs are trained on the same image, and the standard deviation in PSNR is plotted. Notice how the random variation in PSNR increases dramatically for narrow networks.
PSNR: 23.23
A.6 Random Variation in SIREN PSNR (contd.)
Despite comprising less than 1% of the network's learnable pa- rameters, the first layer is responsible for roughly 66.0% of COIN's variance in final PSNR due to random initialization. By initializing the first layer in the same way each time instead of randomly, we reduce the standard deviation in PSNR among SIRENs using Dupont et al's method from 0.18 to 0.10 PSNR.
PSNR: 18.45
This random variation is inversely correlated with network width (See Figure D). When SIRENs are wide, they take larger random samples of the space random Fourier features, and these large random samples converge similar distribution. But for very narrow networks, such the ones used by COIN, the positional encoding consists of only a few randomly sam- pled sine waves, so there is significant variation in the quality of the positional encoding. Figure S iillssrates this principle for very small SIRENs with a width of just 3 hidden units in the first layer. In this extreme case, the effect of a poorly initialized PE layer is especially clear.
Random Positional Encoding
Learned Output
Figure 8: Output of two small SIRENs with randomly-nnitialized 3-channel posi- tional encodings.
We can improve SIREN performance by selecting a good first layer, instead choosing one at random. We train 100 randomly initialized COIN networks on each of the 24 Kodak images, for total of 2,400 trained networks. For each of the 24 images, train 24 new SIRENs, one for each image in the Kodak dataset, which are initialized with this layer. This gives us collection of 24 x 24 = 576 SIRENs whose first layers are not randomly initialized, but have been selected as the best out of 100 randomly initialized layers. In the interest of training time, we reduce the number of training steps for each of the 2400 + 576 SIRENs in this experiment from 50,000 to 20,000.
take the first layer from the SIREN with the best final PSNR, and
When retraining on the same image using the positional encoding with the best PSNR, our SIRENs perform 0.23 PSNR better on average than with random PEs. When applying the best. PE from one image to another image, average improvement in PSNR is 0.14 PSNR. This suggests that good positional encodings are somewhat transferable from image to image. Figure 10 shows the results of our experiment in detail.
25

---

Published in Transactions on Machine Learning Research (05/2024)
A.6.1 Derivation of Equation 4
We would like to estimate the irreducible error our encoding error prediction problem. we are trying predict variable from a variable X, the minimum possible RMSE that any regression model can reach
RMSE- |E
In our setting, Y is the encoding error of a SIREN as measured in PSNR, and X represents the SIREN's training inputs: width, depth, Wo and the target image. Y|X = a is the distribution of encoding errors a given SIREN setup & will reach across different random seeds.
To estimate E arl [X]]| we randomly sample N SIREN training inputs "1,..., IN X from our dataset, and estimate the variance S = X zı| for each. This gives us the following irreducible error formula:
1 N
RMSE
Σ S
We can estimate the variance S of a population from m samples using the unbiased sample variance estimator:
8²
Σ i=1
(E
v2)²
(9)
To estimate Var[Y]X = 23), we draw two samples Vi,1, Vi.2 p(y(zt)) So m 2, and Ely] 31.1 By
2
substituting and simplifying, Equation g becomes:
S = 2 - V1,2)
(10)
Finally, substitute Equation 10 into Equation
to get Equation
26

---

Published in Transactions on Machine Learning Research (05/2024)
253
150
12.5
100
6a
00
23R
21.
262
280
2EA
25-
2000
24-
15.0
1600
42-
5-
5:0
26.5
27.3
15.4
15.5
35.
25.2
25.4
291
28.4
22.4
25.0
250
35.
366
301
23:0
20.3
20-
15-
12.5
10-
153
20.3
2044
20.5
23.8
24.3
204
250
252
234
273
28.
23.5
27.3
26.6
23.7
23
230
241
15-
Figure 10: Histograms of the PSNR which the default COIN network reaches on each of the 24 images in the Kodak dataset across 100 different random seeds. For each image, we pick the randomized run with the best PSNR, extract its first layer, and reuse that first layer in an otherwise newly randomized network on each image. Each vertical line represents the PSNR reached by one of these retrained networks that used the best first layer out of 100. The blue vertical lines indicate that the first layer was selected from a network trained different image, the red vertical lines indicate the models that were retrained on the same image.
A.7 SIRENs compress the Global Color Space
Figure illustrates the significant qualitative differences in the information lost by JPEG and COIN com- pression. At the same compression ratio, COIN better maintains the sharpness of high-contrast odges. Artifacts and distortions COINs appear more structurally complex. Most interestingly, COINS appear to globally compress an image's color space; reducing variation in color to reduce the amount of information per pixel. We posit that this global color compression accounts for much of COIN's advantage over JPEG at low bitrates.
If our hypothesis is correct, then our JPFG proxy models for SIREN encoding error (from Figures 3a and 3b) could be further improved by accounting for the discrepancy in how COINs and JPEGs compress color space. But this remains a topic for future work.
27

---

Published in Transactions on Machine Learning Research (05/2024)
Ground Truth
JPEG2000 PSNR 22.18
COIN PSNR 22.74
200
200
R
R
150
200 200
200 200
200 200
Figure 11: Images and their pixel-color scatter plots in RGB space. Note how the SIREN sacrifices color fidelity for spatial fidelity to achieve higher net PSNR than JPEG2000 at 0.3 bpp.
single-arch. many-arch.
0.08
frequency
0.06
0.04
0.02
0.00
15
20
25
30
35
40
45
50
55
PSNR
Figure 12: PSNR. distribution for the single- and many-architecture SIREN datasets.
28


Reviews:
[{"review_id": "TrM89baV1D", "reviewer": "Reviewer_qpjz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros,\n+ It is a novel and interesting task. Most existing papers focus on improving the accuracy or eliminating the spectral bias of INR; however, few works discuss parameter selection or related topics since they are dirty work and cannot be summarized as high-quality papers.\n+ It is novel and interesting to discuss the relationship between SIREN-based and JPEG-based image compression. Several papers have noticed that SIREN or Fourier encoding-based INR could be viewed as Fourier expansion, which is also the base of JPEG-based compression; however, none of the previous papers discuss them directly. This paper may inspire researchers to investigate INR-based compression.\n\nCons,\nThe title is too big. Although there are several types of INR, such as the SIREN, Fourier feature INR, WIRE[1], Gaussian-INR[2], and FINER[3], only the SIREN is analyzed throughout the paper. It is suggested that the title be revised and that other INRs be cited.\n\n\nSome statements are imprecise. Sec. 2.4, line 15, 'We find, perhaps unsurprisingly, that the optimal choices of \\omega_0 vary with both the image size and content'. Many papers have observed that expressive power of SIREN could be viewed as Fourier expansian[4,5,6], where the parameter \\omega_0 is related to the used frequency base functions in Fourier expansian, as a result, it is intuitively to observe and conclude the phenomenon mentioned above. This phenomenon was observed and mentioned in several previous papers, such as [4,5,6]. Consequently, it is suggested that this sentence and related statements be revised.\n\n\nAlthough the analysis of the relationship between SIREN-based and JPEG-based image compression is interesting, it is unclear what the logical relationship between this topic and the main task (i.e., predicting the fitting error of SIREN) is. It is suggested that the paper be revised.\n\n[1] Vishwanath et al. WIRE: Wavelet Implicit Neural Representations. IEEE CVPR 2023.\n[2] Sameera et al. Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs. Springer ECCV 2022.\n[3] Zhen et al. FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions. IEEE CVPR 2024.\n[4] Gizem et al. A structured dictionary perspective on implicit neural representations. IEEE CVPR 2022.\n[5] Nuri et al. Seeing Implicit Neural Representations as Fourier Series. IEEE WACV 2022.\n[6] Hao et al. Disorder-invariant implicit neural representation. IEEE TPAMI 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper centers on a novel and intriguing task: predicting the fitting error of Implicit Neural Representations (INRs) using fixed image content and network parameters. To accomplish this, a dataset is initially constructed, comprising 300,000 SIRENs trained on a diverse range of images and distinct network configurations. A uniquely designed hybrid network, combining Convolutional Neural Networks (CNNs) with Multi-Layer Perceptrons (MLPs), is utilized to forecast the INR's fitting error. In addition to predicting SIREN's fitting errors, this paper also makes several original observations. These encompass the discovery of scaling power-law patterns in SIRENs and the exploration of the relationship between SIREN-based image compression and the conventional JPEG method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See cons above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fFcqiTL8mf", "reviewer": "Reviewer_r9k3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- large scale experimentations \n- solid empirical analysis \n- valuable insights to INR practitioners \n\n**Weaknesses**\n- the field is somewhat narrow, and the findings are more into engineering than solving research questions", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work trains a network that predicts the encoding error of the SEREN network (of INR class). The paper provides an analysis of tehse results and their correlation with other metrics (e.g. the jpg compression rate)", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I find the work overall complete and well-written.\n\n(nitpick) I would consider rearranging sections. For example, Sec 4.2, which contains negative results about NTK, could be deferred further.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8xVcQpxIQm", "reviewer": "Reviewer_j9KN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Predicting encoding errors is an important area of research, especially for hyper-parameter search.  The paper is well-motivated by the observation that for every image, an implicit neural representation that has some hyper-parameters needs to be trained. This paper proposes that understanding the connection between the hyper-parameters is useful for quickly fitting SIREN models to data\n2. The data setup of images/hyper-parameters to errors is reasonable.\n\n**Weaknesses**\n1.  While the experimental setup is well-motivated, predicting the error of another model seems extremely challenging. While Fig 5 shows that indeed models can be fit to the error, I do not think the accelerated hyper-parameter search in S4.7 was sufficiently explored as there is not a clear improvement for architecture search.\n2. A lot of the content of the paper analyzing the errors on the datasets is somewhat interesting, but does not seem relevant to the main point of predicting errors. E.g., Figure 4 shows the relationship between trained SIREN and JPEG representation and seems disconnected from the idea of predicting the errors.\n\n**Other related work**\n1. [ATT3D: Amortized Text-to-3D Object Synthesis](https://openaccess.thecvf.com/content/ICCV2023/papers/Lorraine_ATT3D_Amortized_Text-to-3D_Object_Synthesis_ICCV_2023_paper.pdf) and [LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis](https://arxiv.org/abs/2403.15385) are interesting relevant papers here. They have a similar setting where they have text-conditional 3d implicit representations and learn the mapping directly from the text to the optimal parameters of a NeRF representing the object. The connection here is also that the optimal parameters of the NeRF for the object also needs to be re-computed/re-learned for every instance in their training dataset, and they do this by predicting it (conditional on the text prompt)\n2. It also may be worth connecting into the other architecture/hyper-parameter search code that predicts errors, such as [TAPAS: Train-Less Accuracy Predictor for Architecture Search](https://arxiv.org/abs/1806.00250) and the citation graph around it. My understanding is the submitted paper focuses more on image-conditioning and high-level hyper-parameters while TAPAS focuses on predicting the error from the architecture", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to train a model to predict the encoding error of the SIREN architecture implicit neural representations. S 3.3 and 3.4 describe how they ran the SIREN training process on the Kodak and MSCOCO datasets to generate a dataset of learned SIREN models with their parameters and errors, where the main hyper-parameter choices are the image size, network depth, width, and frequency parameter. The experimental results investigate how well the NTK extrapolates the training performance, motivating their deficiency in S4.2, then they continue to note the scaling laws and performance of SIREN in S4.3 and S4.4. S4.5 compares the SIREN parameters vs JPEG for compressing the images. S4.6 gets to predicting the errors of the representations and S4.7 combines it with hyper-parameter search methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the paper presents a reasonable experimental study of predicting SIREN errors. While predicting errors is a very general and challenging topic for ML, this paper presents a study of image and hyper-parameter conditional error predictions for the SIREN model. I do not have many changes to request, other than possibly connecting into other work predicting parameters of NeRFs and other architecture search work with error predictions", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PzzPBpbAoj", "reviewer": "Reviewer_FHQY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe authors make several valuable contributions that will no doubt be interesting to the TMLR community:\n - The collection of two large datasets consisting of triples of INR input data, hyperparameter settings, and the best achievable PSNR on the data using those settings: one dataset with a fixed architecture and varying input images and one dataset where both the hyperparameters and the input data vary.\n - The exploratory analysis of the dataset, the conclusions drawn from it, and the testing and verification of different theories on the scaling behavior of INRs.\n - Demonstrating that we can successfully utilize deep learning to predict the best achievable PSNR performance of INRs from their hyperparameter settings and input data.\n\nFurthermore, the paper is generally well-written.\n\n## Weaknesses\nI have one major issue with the paper: there is no mention of other methods to obtain surrogate models for the INRs' performance, let alone a comparison with them! The most obvious method to compare against would be using Gaussian process regression to predict the PSNR, which can then be used to perform Bayesian optimization (BO) to find optimal hyperparameters for our INRs, see e.g. [2]. My guess is that BO would be a significantly more sample-efficient procedure (i.e., it would require us to train many fewer INRs) than the one given in Section 4.7. Could the authors comment on why they omitted such experiments?\n\nIn a similar vein, I think the sentence \"Towards this goal, we propose a novel problem: predicting the encoding error (i.e. training loss) that an INR will reach on a given training signal.\" in the abstract should be clarified as predicting the train/test error of neural networks is certainly not a new problem, see again [2] for example.\n\nOn a final related note, while I am happy with most of the authors' methodology, I found Section 4.7 on hyperparameter optimization questionable. If the authors intended to showcase a potential dummy application of their PSNR predictor network, then I don't mind the content, but they should clarify that this is only a toy example. Otherwise,  they need to compare their approach with BO.\n\nOtherwise, there are a couple of more minor issues, which I outline in the requested changes section below.\n\n[2] Snoek, J., Larochelle, H., & Adams, R. P. Practical Bayesian optimization of machine learning algorithms. NeurIPS 2012\n\n## Questions\n - Is there any explanation for why the encoding errors should follow a gamma distribution?\n - Section 4.7: \"to maximise the expected PSNR\" / \"However, since our PSNR predictions are probabilistic\" - I am not sure how the authors' method makes probabilistic predictions; as I understand their network predicts a single PSNR value, not a distribution over it. Could the authors please explain what I'm missing?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors conduct a thorough empirical study of how various hyperparameter settings and inputs affect the fit of implicit neural representations (INRs) with sine activations. In particular, the authors study the peak signal-to-noise ratio (PSNR) achieved by various INRs trained on a large dataset of images.\n\nThe authors make several contributions. First, they create a large dataset of INRs fitted to various images. Then, they conduct some exploratory data analysis on their dataset and test whether various theoretical scaling laws can be used to predict the INRs' performance. Concretely, they find that 1) INRs with few units on their hidden layers are more sensitive to the random initialization of the networks, 2) this sensitivity is primarily due to the randomness on the first layer, 3) NTK theory does not make the correct predictions for INR performance due to the violation of its necessary assumptions and 4) in a certain local sense the INRs' performance exhibits power-law scaling in the model size, as predicted by  [1].\n\nThen, the authors use the dataset they collected to predict the best PSNR a given INR will achieve using three different techniques: 1) a simple linear model, 2) using the codelength of JPEG2000 as a proxy, and 3) a deep learning-based approach.\n\nFinally, the authors use their deep learning-based PSNR prediction approach to perform hyperparameter search for INRs using different objectives.\n\n[1] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Major:\n - While Figure 5 is a good illustration of the average performance of the model, the authors should add figures similar to Figures 4a and 4b that show how well the predicted PSNRs correlate with the actual PSNRs on a per-image basis. This addition would be useful to demonstrate that the model doesn't make gross mispredictions.\n\n## Minor:\n - The authors make many nice contributions, so I think they should outline them in a bullet-point list at the end of the introduction section.\n - I would prefer to have the related works section after the experiments.\n - The future work section is currently more of a discussion section, as there is little mention of future directions. Hence, please rename the section or add more appropriate content.\n - Figure 2 is difficult to read; please add a legend indicating what the different curves mean. Furthermore, I suggest that the authors use a gradient color palette for the NTK curves, where the gradient would indicate the training step at which the curve was extrapolated.\n - Figure 5 is missing a legend indicating that the blue curve is the authors' deep learning-based predictor.\n - The x-axis label of Figure 6 is cropped; please adjust it.\n - Please add error bars to Figures 5a and 5b\n - Section 4.2: \"when the largest eigenvalue of K exceeds\" - should be $\\Theta$.\n - The sentence below eq (9): \"we draw two samples...\" at the end, it should be $y_1, y_2 \\sim p(y | x)$", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for predicting the encoding error of a popular Implicit Neural Representation (INR) network, SIREN, given its hyperparameters and the signal to encode. The authors constructed a dataset of 300,000 SIRENs, trained across a variety of images and hyperparameters. They used a hybrid network combining Convolutional Neural Networks (CNNs) and Multi-Layer Perceptrons (MLPs) to predict the SIREN's fitting error. This method provides the encoding error of SIRENs in milliseconds, significantly improving efficiency compared to traditional methods. The paper also provides deep insights into the behavior of SIREN networks and explores their relationship with JPEG compression. 

Reviewers mainly provided suggestions for improvement regarding the clarity, the precision, and the comparison with existing methods. The authors have revised the paper according to these suggestions and enhanced the quality of the paper.
Decision: Accept (poster)