Learning Transferable Features for Implicit Neural Representations
Authors: Kushal Vyas
Ahmed Imtiaz Humayun
Aniket Dashpute
Richard Baraniuk
Ashok Veeraraghavan
Guha Balakrishnan
Venue: NeurIPS 2024 poster (2024)
Keywords: implicit neural representations
inverse problems
transfer learning
Primary area: machine_vision

Abstract: Implicit neural representations (INRs) have demonstrated success in a variety of applications, including inverse problems and neural rendering. An INR is typically trained to capture one signal of interest, resulting in learned neural features that are highly attuned to that signal. Assumed to be less generalizable, we explore the aspect of transferability of such learned neural features for fitting similar signals. We introduce a new INR training framework, STRAINER that learns transferable features for fitting INRs to new signals from a given distribution, faster and with better reconstruction quality. Owing to the sequential layer-wise affine operations in an INR, we propose to learn transferable representations by sharing initial encoder layers across multiple INRs with independent decoder layers. At test time, the learned encoder representations are transferred as initialization for an otherwise randomly initialized INR. We find STRAINER to yield extremely powerful initialization for fitting images from the same domain and allow for a ≈ +10dB gain in signal quality early on compared to an untrained INR itself. STRAINER also provides a simple way to encode data-driven priors in INRs. We evaluate STRAINER on multiple in-domain and out-of-domain signal fitting tasks and inverse problems and further provide detailed analysis and discussion on the transferability of STRAINER’s features.

Learning Transferable Features for Implicit Neural Representations
Kushal Vyas kushal. vyas@rice edu
Ahmed Imtiaz Humayun imtiazOrice.co edu
Aniket Dashpute aniket dashpute@rice edu
Richard G. Baraniuk richbBrice edu
Ashok Veeraraghavan vashokOrice.co
Guha Balakrishnan guha@rice. edu
Rice University
Abstract
Implicit neural representations (INRs) have demonstrated success in a variety applications, including inverse problems and neural rendering. An INR is typically trained to capture one signal interest, resulting in learned neural features that are highly attuned to that signal. Assumed to be less generalizable, aspect transferability of such learned neural features for fitting similar signals. We introduce new INR training framework, STRAINER that learns transferrable features for fitting INRs to new signals from a given distribution, faster and with better reconstruction quality. Owing the sequential layer-wise affine operations in an INR, we propose to learn transferable representations by sharing initial encoder layers across multiple INRs with independent decoder layers. At test time, the learned encoder representations are transferred as initialization for an otherwise randomly initialized INR. We find STRAINER yield extremely powerful initialization for fitting images from the same domain and allow for a RJ +10dB gain in signal quality early on compared an untrained INR itself. STRAINER also provides simple way to encode data-driven priors in INRs. We evaluate STRAINER on multiple in-domain and off-minn signal fitting tasks and inverse problems and further provide detailed analysis and discussion on the transferability of STRAINER features. Our demo can be accessed here.
explore the
Introduction
Implicit neural representations (INRs) powerful family of continuous learned function ap- proximators for signal data that are implemented using multilayer perceptron (MLP) deep neural networks. An INR fø Rm + R maps coordinates lying in m-dimeesional space value in a n--iimensional output space, where represents the MLP's tunable parameters. For example, typical INR for natural image would use input space in R (consisting of the and pixel coor- dinates), and an output space in R (representing the RGB value of a pixel). INRs have demonstrated several useful properties including capturing details all spatial frequencies (39 36), providing
powerful priors for natural signals [36] [3). and facilitating compression [12 27 For these reasons,
in the past 5 years, INRs have found important uses in image and signal processing including shape
representation 17 [6], novel view synthesis [3 34 42), material rendering (24), computational
imaging [5] 30 medical imaging [49] linear inverse problems
virtual reality [ and
compression 12 43 51].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Shared Encoder, fo
Decoders, db
(c) Reconstructed output
STRAINER
SIREN
Ground Truth
(a) Training
(b) Fitting
new image
36.35 dB
27.94 dB
PSNR at 200 iterations
Figure 1: STRAINER Learning Transferable Features for Implicit Neural Representations. During training time (a), STRAINER divides an INR into encoder and decoder layers. STRAINER fits similar signals while sharing the encoder layers, capturing rich set of transferrable features. At test- time, STRAINER serves as powerful initialization for fitting a new signal (b). An INR initialized with STRAINER'S learned encoder features achieves (c) faster convergence and better quality reconstruction compared to baseline SIREN models.
A key difference between training INRs compared to other neural architectures like CNNs and Transformers is that a single INR is trained on a single signal. The features learned by an INR, therefore, are finely tuned the morphology of just the one signal represents. SplineCAM [19] shows that INRs learn to finely partition the input coordinate space by essentially overfitting to the spatial gradients (edges) of the signal. While this allows an INR represent its signal with high fidelity, its features can not "transfer" in any way to represent second signal, even with similar content. If INRs could exhibit elements of transfer learning, as the case with CNNs and Transformers, their potential would dramatically increase, such as by encoding data distribution priors for inverse imaging problems.
In this work, we take a closer look at INRs and transferable features, and demonstrate that the first several layers of an INR can be readily transferred from one signal to another from domain when trained in a shared setting. We propose STRAINER, a simple INR training framework to do so (see Figure D. STRAINER separates an INR into two parts: an "encoder" that maps coordinates to features, and a "decoder" that maps those features to output values. We fit the encoder over a number of training signals (1 to 10 our experiments) from domain, e.g., face images, with separate decoders for each signal. At test time, we initialize new INR for the test signal consisting of the trained encoder and randomly initialized decoder. This INR may then be further optimized according the application of interest. STRAINER offers simple and general means of encoding data-driven priors into an INR's 's parameter initialization.
We empirically evaluate STRAINER in several ways. First, we test STRAINER on image fitting across several datasets including faces (celeb-HH)) and medical images (OASIS-MR) and show (Figure 2 that STRAINER'S 's learned features are indeed transferrable resulting in a RS+ 10dB gain in reconstruction quality compared vanilla SIREN model. We further assess the data-driven prior captured by STRAINER by evaluating it on inverse problems such as denoising and super resolution. Lastly, we provide detailed exploration of how STRAINER learns transferable features by exploring INR training dynamics. We conclude by discussing consequences of our results for the new area of INR feature transfer.
2 Background
Implicit neural representations. We define so(p) an implicit neural representation (INR) 31 39 29 where fo a multi-layer perceptron (MIP) with randomly initialized weights 0 and p is the m-dimensional coordinates for the signal. Each layer in the MLP is an affine operation followed

---

by nonlinearity such as ReLU B1 or sine [39. Given an n-dimensional signal I(p). the INR learns mapping f: Tm + R" The INR is iteratively trained fit the signal by minimizing a loss function such as 1/2 loss between the signal T(p) and its estimate [o(:):
0" = arg min
Σ Afo(p:) T(p:) 11
8
where p, 's E R' span the given coordinates, 6" are the optimal weights that represent the signal, and A is differentiable forward model operator such identity for signal representation and downsampling operation for inverse problems such as super-resolution.
Representation capacity of INRs. The repre- sentation capacity of an INR can be described the quality of signal the INR can repre-
sent within some number of iterations. ReLUU
based INRs suffer from spectral bias during
Mank-S00000 to Netsa Hatio(PSNR)
training (6), preferentially learning low fre-
50-
quency details in signal and thus leading to
a blurry reconstruction of the represented signal.
40-
Fourier positional encodings 31 46) or si-
nusoidal activation functions (SIREN
39 help
30
better capture high frequency information.
20
Recent works increase the representation capac- ity of INRs with activations flexible in the fre- quency domain. WIRE B6 uses a continuous Gabor wavelet-based nonlinearity, and demon- strates impressive results for a range of forward and inverse problems. FINER (26) splits the sine nonlinearity to have flexible frequency coverage, and DINER uses hash map to non-uniformly map the input coordinates to feature vector, effectively re-arranging the spa- tial arrangement of frequencies and leading to faster and better reconstruction quality.
SIREN
SIREN finetuned
IPG T10
Trand w/TO
10-
STHAINZIL decsdest
Kataiesreed
Transthit TTO
500
1000
1500
2000
Iterations
Figure 2: STRAINER learns faster. We show the reconstruction quality (PSNR) of different initial- ization schemes for in-domain image fitting on CelebA-HQ 22 We compare SIREN
33 model
initialized by (1) random weights (SIREN), (2) fit- ting on another face image (SIREN finetuned), (3) STRAINER -1 (trained using one face image), and (4) STRAINER- 1 (trained using ten face images). We also evaluate against multiple baselines such as Meta-Learned 5K 455. TransINR 9], and [PC
Weight initialization for INRs. Previous work has shown that smart initialization of INR weights allows for faster convergence. As shown in the SIREN study (39, hyper-networks are pro- posed to capture a prior over the space of implicit functions, mapping a random code to the weights of SIREN model. Further, TransINR also shows Transformer-hypor heaffrrmer-hypemeeworrss as powerful met- alearners for INR weight initialization. MetaSDF [38] and Light Field Networks (LFN) 40) use meta-learning-bast based initialization schemes fit signed distance functions (SDFs) and light fields.
Neural Compression algorithms [4 43 use weights obtained from meta-learning optimization as reference to store MLP weights, leading better compression than naively storing MLP weights. Meta-Learning (MAML) [5] and Reptile 32] for coordinate based neural representations. While these meta-learning approaches yield powerful initialization, they often require long computation time (over 1 150K steps (45) and ample numbers of training data, and are unstable to train [48). Further, meta-learning initial modulations for an INR which are later optimized to fit data within few gradient updates has been shown to be an effective and scalable[6| strategy for smoothly representing data(sets) as approach, Implicit Pattern Composers (IPC) proposes to keep
Tancik et al. 45 propose meta-leame seett-leeaming--aeee based approaches such as Model-Agnont
functa(sets) [3. Contrary the second layer of an INR instance-specific, while sharing the remaining layers and use a transformer hypernetwork to learn the modulations for the INR.
Prior informed INRs. Recent work has also explored embedding a learned prior in INRs for tasks such as audio compression, noise removal, and better CT reconstructions. Siamese Siren [25] uses similar approach where they propose compact siamese INR whose initial layers are shared followed by 2 siamese decoders. Since 2 randomly initialized decoders will yield slightly different reconstructions, this difference leveraged for better noise estimation in audio signal. NERP (37 learns an internal INR prior for medical imaging by first fitting high quality MRI or CT data. Weights
3

---

of this learned INR are used as an initialization for reconstructing new MRI or CT undersampled data. While this paper shows a method to learn an implicit prior, their prior embedding learned from single MRI or CT scan of the same subject whereas our work explores learning a prior for INRs by constraining it to learn an underlying implicit representation across multiple different images. PINER [4] introduces a test-time INR adaptation framework for sparse-view CT reconstruction with unknown noise.
Methods
We introduce STRAINER, We first explain our motivation to share initial layers in an INR Section 3.1 In Section (32 we describe the training phase of STRAINER where we learn transferrable features for INRs by sharing the initial layers of N INRs being fit independently to N images. Section 3.3 details how our captured basis is used to fit an unseen image. In subsequent sections, we seek to understand what our shared basis captures and how to expand it to other problems such super resolution. For simplicity, we build upon the SIREN 39 model as our base model.
3.1 Why share the initial INR layers?
A recent method called SplineCAM 19] provides lens with which to visualize neural network partition geometries. SplineCAM interprets an INR as function that progressively warps the input space and fits a given signal through layerwise affine transforms and non- non-linearities [19]. For continuous piecewise affine activation functions, we use an approximation to visualize (see Figure the deep network's 's partition geometry for different pre-activation level sets 20).
An INR fit signal highly adapts the underlying structure of the data in layer-wise fashion. Furthermore, by approximating the spatial position of the pre-activation zero level sets, we see that initial layers showcase coarse, less-partitioned structure while deeper layers induce dense partitioning collocated with sharp changes in the image. Since natural signals tend to be similar in their lower frequencies, we hypothesize that initial layers of multiple INRs are better suited for transferability, We therefore design STRAINER to share the initial encoder layers, effectively giving rise to an input space partition that can generalize well across different similar signals.
3.2 Learning transferable features from N images
Consider a SIREN 39 model hp)) with L layers. Let K out of L layers correspond to an encoder sub- fo The remaining layers correspond to the decoder sub-network represented as as seen in Figure [a). For given input coordinates p, we express the SIREN model h (p) as a composition (o) of our encoder-decoder sub-networks.
network represented
hp) = 96 fo(p).
(2)
In a typical case, given the task of fitting N signals, each of the N signals is independently fit to an INR, thus not leveraging any similarity across these images. Since want to learn a shared representation transferrable across multiple similar images, our method shares the encoder across all N INRs while maintaining set of individual signal-specific decoders 9 ... 9 .Our overall architecture is shown in Figure [ We call this STRAINER'S 's training phase Figure I a). We start with randomly initialized layers and optimize the weights fit N signals in parallel. For each signal 4(p), we use L2 loss between (p) and its corresponding learned estimate h2sa (p) and sum the loss over all the N signals. Iteratively, we learn a set of weights θ that minimizes the following objective:
N
arg min
Σ
ll oe o folp) 1,(1) 112
1=1
where θ 10, 0² IM represents the full set of weights of the shared encoder (0) and the N different decoders (1) ... g and O" represents the resulting optimized weights.
3.3 Fitting unseen signal with STRAINER
After sufficient iterations during STRAINER'S 's training phase, we get optimized encoder weights fo- which corresponds to the rich shared representation learned over signals of the same category. To fit

---

(a) PCA of learned features
(b) PSNR after 0, 50, 100, 2000 iters respectively
#100
8.32
29.74 dB
34.64 dB
55.02
SIREN STRAINER
0.0
8.88 dB
26.17 dB
27.59 dB
38.59 dB
-1.0
-2.0
Figure 3: Visualization of learned features in STRAINER and baseline SIREN model. We visualize (a) the first principal component of the learned encoder features for STRAINER and corresponding layer for SIREN At iteration O, STRAINER ' features already capture a low dimensional structure allowing quickly adapt the image. High frequency detail emerges in STRAINER's learned features by iteration 50, whereas SIREN is lacking at iteration 100. The inset showing the power spectrum of the reconstructed image further confirms that STRAINER learns high frequency faster. We also show the (b) reconstructed images and remark that STRAINER fits high frequencies faster.
novel signal T we initialize the STRAINER model with the learned shared encoder weights so=o- and randomly initialize decoder 9 weights solve for:
$0.0" arg min || S 4+
(4)
fo=o- serves learned initial encoder features. Our formulation is equivalent to a initial set of learned encoder features followed by set of random projections. While fitting unseen signal, we iteratively update all the weights of the STRAINER model, similar any INR.
3.4 Learning intermediate partition space in the shared encoder Se-
During the training phase, explicitly sharing layers in STRAINER allows us to learn a set of INR features which exhibits common partition space shared across multiple images. Since deep networks perform layer wise subdivision of the input space, sharing the encoder enforces the layer to find the partition that can be further subdivided into multiple coarse partitions corresponding to the tasks being trained. In Figure 6 ai), while pre-training an INR using the STRAINER framework on CelebA-HQ dataset, we see emergence of face-like structure captured in our STRAINER encoder fo-. We expect our STRAINER encoder weights fø- to be used as transferrable features and be used as initialization for fitting unseen in-domain samples.
In comparison, meta learning methods to learn initialization for INRs 455 exhibit a partitioning of the input space that is closer random. As in Figure (( there is faint image structure captured by the the learned initialization. This is an indication that the initial subdivision of the input space, found by the meta learned pre-training layers, captures less of the domain specific information therefore is a worse initialization compared to STRAINER We further explain our findings in Section 5 and discuss STRAINER'S learned features being more transferrable and lead to better quality reconstruction.
4 Experiments
In all experiments, used the SIREN 39
MLP with 6 layers and sinusoid nonlinearities. We
considered two versions of STRAINER (i) STRAINER (1 decoder), where the encoder layers are initialized using our shared encoder trained on a single image, and (ii) STRAINER- 10 (10 Decoders), where the encoder layers are initialized using our shared encoder trained on 10 images. We considered following baselines: (i) a vanilla SIREN model with random uniform initialization [39], (ii) fine-tuned SIREN model initialized using the weights from another STREN fit to an image from the same domain, (iii) a SIREN model initialized using Meta-learned 5K
45 (iv) transformer-based
metalcarning models such as TransINR 9 and IPC 23 We ensured an equal number of learnable
5

---

Encoder(1)
Encoder(5)
Decoder
SIREN STRAINER-10
SIREN STRAINER- 10
SIREN STRAINER-10
10-3
10-3
Gradient
Gradieat
Gradient
10 0
10 5
10-7
10-7
10-7
0
200
400 lterations
600
800
200
400
600
800
0
200
400
600
800
ltertions
Iterations
Figure 4: STRAINER converges to low and high frequencies fast. We plot the histogram of absolute gradients of layers 1,5 and last over 1000 iterations while fitting an unseen signal. At test time, STRAINER's initialization quickly learns low frequency, receiving large gradients update the start in its initial layers and reaching convergence. The Decoder layer in STRAINER also fits high frequency faster. Large gradients from corresponding SIREN layers show it learning significant features late as 1000 iterations.
parameters (ncurons) for all models. We normalized all images between (0-1), 1), and input coordinates between (-1,1). We used the Adam optimizer with a learning rate of 10 for STRAINER'S training and test-time evaluation, unless mentioned otherwise. Further implementation details are provided in Supplementary.
4.1 Datasets
We mainly used the CelebA-HQ 22) Animal Faces-HQ (AFHQ) [TO], and OASIS-MRI [T8] 28 images for experiments. We randomly divided CelebA-HQ into 10 train images and 550 test images. For AFHQ, we used only the cat data, and used ten images for training and 368 images for testing. For OASIS-MRI, we used 10 of the (template-alignel) 2D raw MRI slices for training, and 144 for testing. We also used Stanford Cars and Flowers [ to further validate out of domain generalization and Kodak (3) true images for demonstrating high-resolution image fitting.
4.2 Training STRAINER'S shared encoder
We first trained separate shared encoder layers of STRAINER on 10 train images from each dataset. We share five layers, and train separate decoder for each training image. For each dataset, we trained the shared encoder for 5000 iterations until the model acheives PSNR Rs 30dB for all training images. We use the resulting encoder parameters as initialization for test signals in the following experiments. For comparison, we also trained the Meta-learned 5K baseline using the implementation provided by Tancik et.al. (45) with 5000 outer loop iterations. We also use the implementation provided by IPC[2]] as our baselines for TransINR[9] and IPC and train them with 14,000 images from CelebA-HQ We report a comparison of number of training images and parameters, gradient updates, and learning time in Table
4.3 Image fitting (in-domain)
We first evaluated STRAINER on the task of in-domain image fitting. We cropped and resized all images to 178 178 and ran test-time optimization on all models for 2000 steps.
At test-time, both STRAINER and STRAINER- 10 use only I decoder, resulting in the same number of parameters as a SIREN INR. Table average image metrics for in-domain image fitting reported with std. deviation. Instead of naively fine tuning using another INR, STRAINER'S 's design of sharing initial layers allows for learning highly effective features which transfer well across images in the same domain, resulting in high quality reconstruction across CelebA-HQ and AFHQ and comparable to Meta-learned 5K for OASIS-MRI images. Table CelebA-HH ID) also shows that STRAING initialization results in better quality reconstruction, when optimized at test-time, compared to more recent transformer-basee INR approaches such as TransINR and IPC as well.

---

Table I: In-domain image fitting evaluation. STRAINER'S ' learned features yield powerful initializa- tion test-time resulting in high quality in-domain image fitting
CelehA-HQ
AFHQ
OASIS-MRI
Method
PSNR1
SSIMI
PSNR
SSIM
PSNR1
SSIM1
SIREN
44,91 +: 2.13
0.991 + 0.007
45.11 + 3.13
0,991 = 0.005
53.03 +: 1.72
0.999 :: 0.0002
SIREN fine-tuned
51.11 + 3,16
0.997 0.013
53.07
3.47
0,997 0.001
58.86 +: 4.12
0.999 0.0012
Meta-leamndd 5K
53.08 + 3.36
0.994 +: 0.053
2.52
0.996 0.044
67.02
-: 2.27
0,999 0.0000
STRAINER (1 decoder)
50.34
+ 2.81
0.997
1 0.001
51.27
+: 2.94
0.997
- 0.001
57.76
+: 2.19
0,999
+: 0.0001
STRAINER-10
57.80
+ 3.46
0.999
+ 0.001
58.06 1 3.75
0.999 = 0.001
62.80
+: 3.17
0,999 + 0.0003
SIREN
STRAINER
Meta-I.carned 5K
STRAINER-10
Ground truth
Figure 5: Fitting MRI images from OASIS-MRI dataset. At just 100 iterations, STRAINER is able to represent medical images with high quality, STRAINER'S initialization allows for fast recovery for sparse and delicate structures, showing applicability in low-resource medical domains as well.
Table 2:
Out of domain image fitting evaluation, when trained CelebA-HQ and tested AFHQ and OASIS-MRI. STRAINER's 's learned features are a surprisingly good prior for fitting
images of its training domain.
AFHQ
OASIS-MRI
Method
PSNR
SSIMI
PSNR1
SSIM
Meta-Icamed 5K
52.40 + 4.21 57.46 + 3.39
0.991 0.077
65.06 = 1.04
0,999 0.00001
STRAINER- 10
0.999 + 0.0003
72.21 = 8.73
0.999 + 0.0001
STRAINER 10 (Gray)
74.61 = 9.96
0.999 0.0003
Table Baseline evaluation for image-fitting for in-domain(ID) and out-of-domain(OD) data. STRAINER learns more transferable features resulting in better performance across the board. Models trained on CelebA-HQ unless mentioned otherwise. TTO = Test time optimization.
CelebA-HQ (ID)
AFHQ (OOD)
OASIS MRI (OOD)
Method
PSNR1
PSNR1
PSNR1
Meta-learned 5K
$3.08
$2.40
55.86
Trans INR w/o TTO
31.59
28.63
31.97
Trans INR TTO
51.86
49,01
55.45
IPC(ReL.U + Pos Enc.) w/o TTO
33.27
29.96
33.96
IPC(ReLUU + Pos Enc.) w TTO
49.72
47.19
51.35
STRAINER-10
$7.80
$7.46
59.50
STRAINER- trained on Flowers
56.98
58.52
STRAINER- 10 trained StanfordCars
56.88
59.66
4.4 Image fitting (out-of-domain)
To test out-of-domain transferability of learned STRAINER features, we used STRAINER-ID 10 's encoder trained on CelebA-HQ as initialization for fitting images from AFHQ (cats) and OASIS- datasets (see Table 2). Since OASIS-MRI are single channel images, we trained Meta-learned 5K and STRAINER- T0 (GRAY) on the green channel only of CelebA-HQ images. To our surprise, we see STRAINER - 10 and STRAINEE 10 (GRAY) clearly outperform not only Meta-learned 5K but also
STRAINER- 10 (in-domain). To further validate out of domain performance of STRAINER, we train
7

---

Table 4: Out-of-domain image fitting on Kodak Images 3
STRAINER (trained on CelebA-HQ )
allows better convergence comparable high capacity SIREN models as indicated by PSNR metric.
Parrot
Airplane SSIM
Statue
Method
Width
PSNR
SSIM
LPIPS
PSNR
LPIPS.
PSNR
SSIM
LPIPS|
256
36.77
0.94
0.13
31.89
0.87
0.19
34.68
0.94
0.093
STRAINER-10 10
256
39.55
0.96
0.087
35.03
0.92
0.09
37.84
0.96
0.037
SIREN
512
40.18
0.96
0.11
34.23
0.90
0.14
38.80
0.97
0.051
STRAINER-10 10
512
44.38
0.97
0.021
38.96
0.96
0.023
43.92
0.98
0.008
Table 5: Training time and compute complexity. We train all the methods for 5000 steps. STRAINER instantly learns a powerful initialization with minimal data and significantly fewer gradient updates.
Method
training images
# Iearnable params
Gradient updates / itteaation
Time (Nvidia A100)
SIREN
N/A
264.707
N/A
N/A
STRAINER (1 decoder)
264,707
264,707
1188s
STRAINER- (10 decoders)
10
271.646
271,646
24.54s
Meta-leamedd
10
264.707
794.121 (=e 3xmore)
1432.3s 23.8 min
[ra
14,000
s 40M
= 40.M
day
IPC 23 w TTO
14,000
8 40M
= 40M
R day
Table 6: STRAINER accelerates recovery of latent images in inverse problems. STRAINER captures an implicit prior over the training data allowing to recover clean latent image of comparable quality 3x faster making it useful for inverse problems.
Super Resolution (Fast)
Super-Resolution (HQ)
Denoising
Method
PSNR
# iterations
PSNR
# iterations
PSNR
# iterations
SIREN
32.10
3329
32.10 10
3329
26.75 + 1.67
203 + 66
STRAINER -10
31.56
1102
faster)
32.43
3045
26.41 + 1.39
76 + 27
STRAINER- 10 's shared encoder on simply 10 images from Flowers and Stanford Cars 2 datasets which have different spatial distribution of color and high frequencies than AFHQ and OASIS-MRI. For fair comparison, all models in Table RobleBBoon OOD) were with 3-channel RGB or concatenated gray images in case of OASIS-MRI. As shown in Table OOD), STRAINER- 10 provides superior out of domain performance for AFHQ trained on CelebA-HQ followed by Flowers and Stanford Cars. For OASIS-MRI, we see STRAINER- 10 having best performance when trained with StanfordCars. This result suggests that STRAINER capable of capturing transferable features that generalize well across natural images.
STRAINER also fits high resolution Kodak images well and is comparable SIREN networks with twice the network width.
4.5 Inverse problems: super-resolution and denoising
STRAINER provides a simple way to encode data-driven priors, which can accelerate convergence on inverse problems such as super-resolution and denoising. We sampled 100 images from CelebA-HQ at 178 x 178 and added 2dB of Poisson random noise. We report mean values of PSNR achieved by STRAINER and SIREN models along with the iterations required achieve the values. For super- resolution, we demonstrate results on one image from DIV2K 4 47). downscaled to 256 x 256 for a low resolution input. We used the formulation shown in Equation (I) with A set downsampling operation. To embed a prior relevant for clean images, we trained the shared encoder of STRAINER with high quality images of resolution same as the latent recovered image. At test time, we fit the STRAINER model to the corrupted image, following Equation (T) and recovered the latent image during the iteration. We report STRAINER'S 's ability to recover latent images fast as well as with high quality in Section 4.5
4x
8

---

(a) Initialization
(b) Fitting new image
M.Maa--.aarredd SK
i. STRAINER
iii. SIREN
iv. SIREN
v. Meta-LLamedd 5K
STRAINER
Generated
GT image
E
E
E
Layer 4
Layer
Layer Layer 3
Layer
Layer 2
Figure 6: Visualizing density of partitions in input space of learned models. We use the method introduced in [20] to approximate the input space partition of the INR. We present the input space partitions for layers 2.3,4 across (a) Meta-learned 5K and STRAINER initialization and (b) at test time optimization. STRAINER learns an input space partitioning which is more attuned to the prior of the dataset, compared to meta learned which is comparatively more random. We also observe that SIREN (iii) learns an input space partitioning highly specific to the image leading to inefficient transferability for fitting new image (iv) with significantly different underlying partitioned input space This explains the better in-domain performance of STRAINER compared to Meta-learned 5K the shallower layers after pre-training provide better input space subdivision the deeper layers to further subdivide.
Discussion and Conclusion
Results in Table demonstrate that STRAINER can learn a set of transferable features across an image distribution to precisely fit unseen signals at test time. STRAINER- 10 clearly achieves the best reconstruction quality in terms of PSNR and SSIM on CelebA-HQ and AFHQ datasets, and is comparable with Meta-learned 5K OASIS-MRI images. STRAINER- 10 also fits images fast and achieves highest reconstruction quality than all baselines as shown in Figure 2 (1 decoder) with fine-tuned STREN seems that the representation learned on one image is not sufficiently powerful. However, as little as 10 images result in a rich and transferable set of INR features allowing STRAINER- 10 achieve 27-B higher reconstruction quality than SIREN and SIREN fine-tuned.
Comparing STR AINER
As in Table surprising.
2 (ODD STRAINER also performs well out-of-domain tasks, which is quite
STRAINER's transferable representations are capable of recovering small and delicate structures as early as 100 iterations as shown in Figure aJnd do not let the scale features from the training phase affect its reconstruction ability. Another interesting finding is that STRAINER- 10 achieves far better generalization for DASIS-MRI (Table 2 when pretrained on CelebA-HQ Further, STRAINER generalizes well to out-of-domain high-resolution images, demonstrated by our experiments of training STRAINER on CelcbA-HQ and testing on the Kodak data (see Table 4.
STRAINER is fast and cheap to run. 5a a 6 layered MLP INR for STRAINER Meta-learned 5K and transformer-basee methods such as TNRnsINR and IPC. At 5000 iterations, STRAINER learns transferable representation in just 24.54 seconds. Meta-learned 5K, in comparison, uses MAML [5) which is far more computationally intensive and results in 20x slower runtime when exact number of gradient updates are matched. Further, STRAINER training setup is an elegant deviation from recent methods such as TransINR and IPC, requiring large datasets and complex training routines.
the time for learning the initialization for

---

5.1 Limitations
Due to the encoder layers of STRAINER being tuned on data and the later layers being randomly initialized, we have observed occasional instability when fitting to test signal in the form of PSNR "drops." However, we observe that STRAINER usually quickly recovers, and the speedup provided by STRAINER outweighs this issue. While work demonstrates that INR parameters may be transferred across signals, it is not fully clear what features are being transferred, how they change for different image distributions, and how they compare to the transfer learning of CNNs and Transformers. Further work is needed to characterize these.
5.2 Further analysis of STRAINER
To further understand how STRAINER'S ' initialized encoder enables fast learning of signals at test time, we explored the evolution of STRAINER's 's hidden features over iterations in Figure 3 In Fa), we visualize the first principal component of learned INR features of the STRAINER encoder and corresponding hidden layer for SIREN across iterations and observe that STRAINER captures high frequencies faster than SIREN. This is corroborated by the power spectrum inset plots of the reconstructed images. We also visualize histogram of gradient updates in Figure 4 and observe that STRAINER receives large gradients in its encoder layers early on during training, suggesting that the encoder rapidly learns of low-frequency details.
Next, we visualize the input space partitions induced by STRAINER and the adaptability of STRAINER's initialization for fitting new signals. We use the local complexity(LC) measure proposed by Humayun et.al. [2 to approximate different pre-activation level sets of the INR neurons. For ReLU networks, the zero level sets correspond to the spatial location of the non-linearities of the network. For periodic activations, there can be multiple non-linearities affecting the input domain. provide the t 2 shifted level sets. Darker regions in the figure indicate high LC, ie.. higher local non- non-linearity. FigureGaalso presents partitions for the baseline models.
In Figure 6 we present the zero level sets of the network, and in Supplementary
SIREN models tend to overfit, with partitions strongly adapting to image details. Since the sensitivity to higher frequencies is mapped to specific input partitions, when finetuning with SIREN, the network has to unlearn partitions of the pretrained image resulting in sub optimal reconstruction quality. When comparing Meta-learned 5K with STRAINER, more attuned to the prior the dataset, compared to Meta-learned 5K which is comparatively more random. While both partitions imply learning high-frequenc, details, STRAINER'S 's partitions are better adapted to facial geometry, justifying its better in-domain performance.
see that STRAINER learns an input space partitioning
6 Broader Impacts
STRAINER introduces how to learn transferable features for INRs resulting in faster convergence and higher reconstruction quality. We show with little data, we can learn powerful features initialization for INRs to fit signals at test-time. Our method allows the use of INRs to become ubiquitous in data-hungry areas such as patient specific medical imaging, personalized speech and video recordings. as well as real-time domains such as video streaming and robotics. However, our method is for training INRs to represent signals in general, which can adopted regardless of underlying positive or negative intent.
Acknowledgments and Disclosure of Funding
This work is partially supported by NIH DeepDOF R01DE032051-001 OneDegree CNS-1956297, IARPA WRIVA 140D0423C0076 and NSF grants CCF191 1094, I1S-1838177, and II$-1730574; ONR grants N00014- 18-12571, N00014-20-1-220 1-2534, and MURI N00014-20-1-2788 AFOSR grant FA0550-22-1-0060; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047.
References
[1] Flower recognition.
10

---

[2] Stanford cars dataset. http:s/www...eec.ccccmm////////d
[3] Kodak lossless true color image suite. https::/p///t us/ http:///w....ggaaaacc//oooooaaaa 1999. [Accessed 2024-05-22J.
[4] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
[5] Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O' O'Toole. Törf: Time-of-fiige of-ighht radiance fields for dynamic scene view synthesis. Advances in neural information processing systems. 34:26289-26300 2021.
[6] Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Richard Schwarz, and Hyunjik Kim. Spatial functa: 2023.
Scaling functa to imagenet classification and generation,
[7] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li. Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information Richh 3D Model Repository. Technical Report arXiv: 151..03012 [cs.GR], Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
2015.
[8] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In CVPR, 2021.
(9] Yinbo Chen and Xiaolong Wang. Transformers as meta-learners for implicit neural representa- tions. In European Conference on Computer Vision, 2022.
[10] Yunjey Choi, Youngjung Uh, Jacjun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains, 2020.
[11] Nianchen Deng, Zhenyi He. Jiannan Ye. Budmonde Duinkharjav, Pranceth Chakravarthula, Xubo Yang, and Qi Sun. Fov-nerf: Foveated neural radiance fields for virtual reality. IEEE Transactions on Visualization and Computer Graphics, 28(1 (8(11):38544388 -3644,33442422 2022.
[12] Emilien Dupont, Adam Goliński, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Compression with implicit neural representations. arXiv preprint arXiv:2103.031
2021. [13] Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo J. Rezende, and Dan Rosenbaum From data to functa: Your data point is a function and you should treat it like one. CoRR, abs/2201. 12204, 2022.
[14] Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Yee Whye Teh, and Arnaud Doucet. Coin++: Neural compression across modalities. Transactions on Machine Learning Research, 2022.
[15] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR, 06-11 Aug 2017.
[16] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1857-4866, 2020.
[17] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Samna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7154-7164, 2019.
[18] Andrew Hoopes, Malte Hoffmann, Douglas N. Greve, Bruce Fischl, John Guttag, and Adrian V. Dalca. Learning the effect of registration hyperparameters with hypermorph. Machine Learning for Biomedical Imaging, 1(PMI 2021):1-30 April 2022.
[19] Ahmed Imtiaz Humayun, Randall Balestriero. Guha Balakrishnan, and Richard G. Baraniuk. Splinecam: Exact visualization and characterization of deep network geometry and decision boundaries. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3789-3798, June 2023.
[20] Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Deep networks always grok and here is why, 2024.

---

[21] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[22] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CoRR, 0tbs/1710.10111 2017.
[23] Chiheon Kim, Doyup Lee, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Generalizable implicit neural representations via instance pattern composers. arXiv preprint 1rX1v:2211.13223, 2022.
[24] Alexandr Kuznetsov. Neumip: Multi-resolution neural materials. ACM Transactions Graphics (TOG), 40(4), 2021.
[25] Luca A Lanzendörfer and Roger Wattenhofer. Siamese siren: Audio compression with implicit neural representations. arXiv preprint arXiv:2306. (2t.::22......757, 2023.
[26] Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weihing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexible spectral-bias tuning in implicit neural representation by variable-periodic activation functions, 2023.
[27] Shishira R Maiya, Sharath Girish, Max Ehrlich, Hanyu Wang, Kwot Sin Lee, Patrick Poirson, Pengxiang Wu, Chen Wang, and Abhinav Shrivastava. Nirvana: Neural implicit representations of videos with adaptive networks and autoregressive patch-wise modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14378-14387, 2023.
[28] Daniel S. Marcus, Tracy H. Wang, Jamie Parker, John G. Csernansky, John C. Morris, and Randy I.. Buckner. Open Access Series of Imaging Studies (OASIS): Cross-sectional MRI Data in Young, Middle Aged, Nondemented, and Demented Older Adults. Journal of Cognitive Neuroscience, 1499::148-1507, 09 2007.
[29] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[30] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 16190-16199, 2022.
[31] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[32] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 1803. 029999 2018
[33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library, 2019.
[34] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 523-540. Springer, 2020.
[35] Stanford 3D Scans Repository. Thai statue.
[36] Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18507-18516. 2023.
[37] Liyue Shen, John Pauly, and Lei Xing. Nerp: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction. IEEE Transactions on Neural Networks and Learning Systems, 2022.
12

---

[38] Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 33:10136-10144 2020.
[39] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
[40] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313-19925, 2021.
[41] Bowen Song, Liyue Shen, and Lei Xing. Piner: Prior-informed implicit neural representation learning for test-time adaptation in sparse-view ct reconstruction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1928-1938, 2023.
[42] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7495-7504 2021.
[43] Yannick Strümpler, Janis Postels, Ren Yang, Luc Van Gool, and Federico Tombari. Implicit neural representations for image compression. In European Conference on Computer Vision, pages 74-91. Springer, 2022.
[44] Yu Sun, Jiaming Liu, Mingyang Xie, Brendt Wohlberg, and Ulugbek S Kamilov. Coil: Coordinate-basee internal learning for tomographic imaging. IEEE Transactions Com- putational Imaging, 7: 7:1400-1412, 1410, 2021.
[45] Matthew Tancik, Ben Mildenhall, Terrance Wang. Divi Schmidt, Pratul Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural repre- sentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2846-2855, 2021.
[46] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
[47] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
[48] Anna Vettoruzzo, Mohamed- Rafik Bouguelia, Joaquin Vanschoren, Thorsteinn Rögnvaldsson, and KC Santosh. Advances and challenges in meta-learning: A technical review, 2023.
[49] Yuehao Wang, Yonghao Long, Siu Hin Fan, and Qi Dou. Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery. In Inil Conf. Medical Image Computing and Computer-Assisted Intervention, 2022.
[50] Shaowen Xie, Hao Zhu, Zhen Liu, Qi Zhang, You Zhou, Xun Cao, and Zhan Ma. Diner: Disorder-invariant implicit neural representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[51] Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression. In ICLR Workshop Deep Generative Models for Highly Structured Data, 2022.
13

---

Suplementary Material A
In our work, increase the representation capacity of the INR by leveraging the similarity across natural images (of given class). Since each layer of an INR MLP is an affine transformation followed by linearity, space to fit the given signal, in our case the signal being an image. Similar images when independently fit to their respective INRs capture similar low-frequency detail such shape, geometry, etc. whereas high-frequency information such as edges and texture are unique to each INR. We propose that these low-frequency features from the initial layers of learned INR are highly transferable and can be used as basis and initialization while fitting an unseen signal. To that end, we introduce novel method of learning our basis by sharing a set of initial layers across INRs fitting their respective images.
interpret the INR
function that progressively warps the input coordinate
Our implementation be found on Google Colab
Understanding the effect of sharing encoder layers
We further investigate how the number of initial layers shared affects the quality of reconstructed image. We start by sharing K" layer the encoder, and N K layers in each decoder and vary K from 1 to N 1. K N is equivalent to simply fine tuning the INR based on all weights from fellow model. We tabulate our results for image quality (PSNR) in fixed runtime of 1000 iterations. We find that sharing all but the last layer results the most effective capturing of our shared basis leading to higher reconstruction quality as seen in fig, 7 This also suggests that the last decoder of the INR mainly responsible for very localized features. Further our work motivates further interest to sutdy the nature of the decoder layers itself.
We show the effect of sharing layers and resulting reconstruction quality. We use a layered Siren model for this experiment. We fit a vanilla Siren model to an image and report its PSNR in fig. 7 Further, we train our shared encoder by sharing K layers and on until we share K N layers.
We that the reconstruction quality progressively increases by sharing layers.
60
50
Peak signal (PSNR)
40
30
20
Siren finetuned
Strainer (Shared layers
Strainer (Shared layers
10
Strainer (Shared layers
Strainer (Shared layers 5)
250
500
1000
1250
1500
1750
2000
Iterations
Figure 7: Sharing different number of layers in STRAINER'S 's encoder. We see that by increasing the number of shared layers, STRAINER'S 's ability recover the signal also improves.
Reporting std. deviation STRAINER for image fitting on CelebA-HQ
We also report the PSNR within std. deviation while comparing STRAINER -10 with SIREN, SIREN -fnetuned, STRAINER (1-decoder), and Meta-learned 5K in Figure
14

---

[a)
(b)
OreP OrICIAN
(c) Flip
(d) Wrap
2010
Figure 8: Fitting STRAINER shifted, rotated and flipped versions of face image. We that despite the transformations done on face image: (a) roll (wrapping the image both vertically and horizontally), (b) rotate, (c) flip vertically, and (d) wrap vertically, STRAINER fits equally well on all of them.
Signal to Noise Ratio(PSNR)
50
40
30
Peak-Signal
SIREN
20
SIREN finetuned
Meta-Learned 5K
10
STRAINER (1 decoder)
STRAINER-10 (10 decoders)
500
1000
1500
2000
Iterations
Figure 9: STRAINER learns fast. We show limited baseline comparison of STRAINER with SIREN STREN -finetuned and Meta-learned 5K methods for the task of image-fitting on CelebA-HQ dataset and note that STRAINER achieves superior reconstruction quality.
Effect of orientation of the input image
We wanted further assess whether the INR is overfitting to one particular aligned face arrangement. To further test this, we take test image and apply various augmentations such as flip, rotate, and roll and study how STRAINER fits it (see F). We find that the initialization learned by strainer is invariant, at test time, to the input signals orientation and can successfully capture the high frequency details fast.
Measuring time for pretraining STRAINER and Meta-learned 5K
Our implementation is written PyTorch 33 whereas Meta-learned implemented by Tancik et.al 45 is a JAX implementation. For measuring runtime, we use the python's 's time package and very conservatively time the step where the forward pass and gradient updates occur in both methods. Further, the code on Nvidia A100 GPU and report the time after averaging 3 such runs
15

---

for each method. There may be system level differences, however, to the best of our knowledge and observation, our timing estimates if not accurate are atleast indicative of the speedup provided by STRAINER
Training details for Kodak high resolution images
To further demonstrate that STRAINER'S can adapted to high resolution images, we evaluated our method on high quality Kodak 3 images with resolution 512 x 768 (see Tables and 7 We present the reconstruction quality attained by STRAINER -10, SIREN model, and a STREN model initialized using Meta-learned 5K with widths of 256, 512. For this experiment, we train our STRAINER encoder using CelebA- HQ Images which are resized the same resolution to Kodak images. Further, we follow all steps previously described for test-image evaluation of Kodak images. Here is another results from the Kodak high resolution images experiment.
Table 7: STRAINER allow better convergence comparable to high capacity Siren models, and meta- learned initializations, as indicated by PSNR metric. Tested on high quality Kodak Images. ID = In domain, OD= Out of domain.
Parmt (OD)
Airplane (OD)
Statue (OD)
Painted Face(TD)
PSNR1
LPIPS
PSNR
SSIM)
T.PIPS]
PSNR1
SSIM
LPIPS]
PSNR1
SSIM;
I.PPS.
SIREN
36.77
0.94
0.13
31.89
0.87
0.19
34.68
0.94
0.093
12,03
0.85
0.26
STRAINER-10
0,087
0.09
Meta-learned 5K
37.07
0.94
0.06
33.92
0.89
0.12
34.32
0.93
0.07
32.96
0.89
0.11
Width=512
LPIPS)
PSNR
SSIMT
LPIPS]
PSNR1
SSIM
LPIPS]
LPIPS.
SIREN
40.18
0.96
0.11
34,23
0.90
0.14
38.80
0.97
0.051
34.45
0.90
0.17
STRAINUR-10
44.38
0.021
0.96
0.023
43.92
0.98
0
41.37
0.006
Meta-learned SK
41.60
0.97
0.02
39.33
0.96
0.02
39.18
0.97
0.02
37.90
0.96
0.03
Results for Inverse problems SSpper Resolution
We discuss how STRAINER provides a useful prior for inverse problems such as super resolution, For the results reported in section 4.5 we attach supplementary plots shown in fig, [10] STRAINER-ID 10 (Fast) a STRAINER- 10 model with 5 shared encoder layers out of 6 total layers. STRAINER 10 (HQ) is a high quality STRAINER model with 3 shared encoder layers. Unlike forward fitting, more degree of randomness in the decoder helps recover better detail for inverse problems. We also showease the effectiveness of STRAINER for in domain super resolution shown in fig. 11
Siren
23
PSNR
20
Strainer
Strainer-10(HO)
10
Strainer 10(Fast)
Siren
1000
2000
3000
4000
5000
Iterations
Figure 10: Super Resolution using STRAINER We show the reconstructed results (a) the left using SIREN and STRAINER. We also plot (b) the trajectory of PSNR with iterations. STRAINER- 10 (Fast) achieves comparable PSNR to SIREN in approximately third of the runtime.
STRAINER for Occupancy fitting
STRAINER a general purpose transfer learning framework which can used initialize INRs for regressing 3D data like occupancy maps, radiance fields or video. To demonstrate the effectiveness
16

---

40
35
30
PSNR
25
15
10
SIREN STRAINER
1000
2000
3000
4000
5000
Iterations
Figure 11: In domain 4x super resolution using STRAINER We see that STRAINER allows for faster convergence for in-domain super resolution making it useful especially for low time budgets. Max value achieved by STRAINER 40.43dB while SIREN achieves 39.75dB. Within 500 iterations STRAINER achieves > 30dB PSNR
of STRAINER on 3D data, we have performed the following OOD generalization experiment. We pre-train STRAINER on 10 randomly selected 'Chair' objects from the ShapeNet[7] dataset. At test time, we fit the 'Thai Statue" 3D object (35]. STRAINER achieves a 12.3 relative improvement in IOU compared random initialization for SIREN architecture in 150 iterations STRAINER- 10 obtains an IOU of 0.91 compared to an IOU of 0.81 without STRAINER 10 initialization. We present visualizations of the reconstructed Thai Statue in Figure T2] Upon qualitative evaluation, we see that STRAINER- 10 is able to capture ridges and edges better and faster than compared to SIREN.
17

---

Ground Truth
STRAINER
SIREN
Figure 12: We use ten shapes from the chair category of ShapeNet to train STRAINER. and that initialization to fit a much more complex volume (the Thai statue We compare the intermediate outputs for both STRAINER and SIREN for 150 iterations to highlight STRAINER ability to learn ridges and high frequency information faster.
Offsetting Pre-activations
18

---

Siren Layers
Offiet=0
Offsct-pi/8
Offsctpii66
8
Offict=pi/4
Offset-pi/3
Offset=pi/2
Figure 13: Siren offset
19

---

Strainer Layers
3
4
5
Offiet-O
Offiet=pi/8
Offset=pi/6
Offsct=pi/4
Offict=pi/3
Offset-pi/2
Figure 14: Strainer offset
20

---

Metalcarned Layers
2
3
4
5
Offset=0
Offsct=pi/8
Offiet-pi/66
Offset-pi/4
Offsct=pi/3
Offistt pi/2
Figure 15: Meta-learned 5K offset
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our goals and contributed are clearly stated in abstract and introduction. Our method addresses the goals and we include results to support them.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations our work in section 5.1 and section
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, all information about our models, training hyperparameters, dataset, and number of iterations are provided our report. Further we also provide the hardware used and report figures such as train time to ensure further corroboration for successful reproducibility. We discuss how choice of data also plays a role in our method's performance.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes. We release the base code of our paper as a Google Colab notebook. We also provide our code files in supplementary material. All data used in this paper belongs to existing open source datasets and have been correctly cited to ensure reproduction. Upon acceptance, we plan to clean and release our code base and share it on GitHub.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Careful treatment to our training method has been present in section
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide our metrics with std. deviation where needed. Our plots also include 1 std. deviation to indicate the variability of results accumulated.
8. Experiments Compute Resources
22

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiment compute resources are mentioned in Section 4 as well as supple- mentary material.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http:///www.ppp.c//p///////aiaaa
Answer: [Yes]
Justification: We conform the NeurIPS code of ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed in section in main submission.
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our methods do not entail a specific released model as it is applicable for fitting any arbitrary signal. We express caution for potential misuse in section
Licenses existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: cite all used resources such as implementations of basclines and data. We release our work with CC-By 4.0 license.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details of dataset are provided in the Section with correct citations. We also provide code Google Colab notebook well python file for the central idea of our paper. While the python script allows to reproduce our intuition experiments, for detailed reproduction the paper, please follow steps mentioned in Section Please refer to attached zip file for assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable for our work. Paper does not involve crowdsourcing nor research with human subjects.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
23

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable for work. Paper does not involve crowdsourcing research with human subjects.
24


Reviews:
[{"review_id": "n5VAgmiFsq", "reviewer": "Reviewer_HDgr", "summary": "The research focuses on learning transferable features using a SIREN model, where encoder and decoder sub-networks are utilized to optimize weights for encoding and decoding. The paper evaluates the main claims made in the abstract and introduction, confirming that the goals are clearly stated and the method effectively addresses them, supported by results. The study discusses fitting unseen signals using optimized encoder weights and highlights the fast convergence of STRAINER to low and high frequencies during training. Experiments involve using the SIREN MLP model with sinusoid nonlinearities, along with different versions of STRAINER compared to baselines, all with equal parameters for fair comparison. Datasets such as CelebA-HQ, AFHQ, and OASIS-MRI are used for experimentation, with the shared encoder of STRAINER trained on 10 images from each dataset for shared initialization in subsequent tests. The research also explores image fitting tasks in-domain and out-of-domain, demonstrating the effectiveness of STRAINER compared to other models, specifically showcasing superior performance in various scenarios like super-resolution and denoising, emphasizing the importance of learned transferable features.", "questions": "- Besides the improvement in PSNR, is the generalization ability expected to help with completion of the signal (i.e. inpainting)? \n\n- It is mentioned that the encoder trained on CelebA-HQ works even better than in-domain pretraining on AFHQ (cats) and OASIS-MRI datasets. Have you tried out-of-distribution generalization trained on other datasets as source domain?", "limitations": "Limitations are discussed in Section 5.1.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces the concept of STRAINER, which demonstrates fast learning of signals at test time by capturing high frequencies efficiently.\n\n- It highlights the adaptability of STRAINER's initialization for fitting new signals, showcasing better performance compared to Meta-learned 5K in learning input space partitioning.\n\n- Improved in-domain performance with STRAINER compared to meta-learning methods for INRs, providing better input space subdivision that aids in faster convergence on inverse problems like super-resolution and denoising.", "weaknesses": "- The paper missed mentioning a couple of important papers on meta-learning INRs, say Functa [A] and Spatial Functa [B]. Especially the latter achieves good PSNR as well as classification accuracy, despite that the methods and the focus are different.\n\n[A] E Dupont et al. Your data point is a function and you can treat it like one.\n[B] M Bauer et al. Scaling Functa to ImageNet Classification and Generation.\n\n- The method is only tested on image datasets but not 3D shapes or radiance fields, where INRs are more applied to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful and thorough evaluation of our work. We appreciate the kind comments on STRAINER being able to learn fast at test time, efficient recovery of high frequency components, and its ability to learn more transferable input space subdivision. We address the comments and suggestions below.\n\n\n__Citing additional papers on meta-learning INR.__ We thank the reviewer for the additional references. We find them relevant to our literature review and will include them in *Section 2, Background*.\n\n\n__Evaluating STRAINER on 3D signals.__ STRAINER is a general purpose transfer learning framework which can be used to initialize INRs for regressing 3D data like occupancy maps, radiance fields or video. To demonstrate the effectiveness of STRAINER on 3D data, we have performed the following OOD generalization experiment. We pre-train STRAINER on 10 randomly selected ‘Chair’ objects from the ShapeNet[44] dataset. At test time, we fit the ‘Thai Statue’ 3D object[52]. STRAINER achieves a 12.3% relative improvement in IOU compared to random initialization for a SIREN architecture – in 150 iterations STRAINER-10 obtains an IOU of 0.91 compared to an IOU of 0.81 without STRAINER-10 initialization. We present visualizations of the reconstructed Thai Statue in RT Figure 1(c). Upon qualitative evaluation, we see that STRAINER-10 is able to capture ridges and edges better and faster than compared to SIREN. We will add the results in the main text in Sec 4. and compare with baselines LearnIt[37], and IPC[48].\n\n\nFitting signals (2D, 3D shapes, video) to STRAINER is straightforward and intuitive. However, extending STRAINER to inverse problems in 3D such as learning Neural Radiance Fields from limited observations and 3D Gaussian Splatting is non-trivial. We consider this an exciting future direction for us and thank the reviewer for their kind suggestions. \n\n\n__New Out-of-Distribution generalization experiments.__ Following the recommendations by the reviewer, we have trained STRAINER’s shared encoder on Flowers [45] and Stanford Cars [46] datasets and report reconstruction quality (PSNR) when testing on AFHQ [5] dataset in RT Table 1. We specifically chose Flowers [45] and Stanford Cars [46] as they have different spatial distribution of color and high frequencies compared to AFHQ and OASIS-MRI which would allow us to evaluate out of domain image fitting more thoroughly. We find that STRAINER is able to learn transferable features and achieves comparable PSNR to the shared encoder trained on CelebA-HQ[16]. As shown in RT Table 1, in-domain fitting for AFHQ has the highest reconstruction PSNR, followed by STRAINER trained on CelebA-HQ, Flowers, and Stanford Cars (in decreasing order, presented in RT Table 1). The results suggest that STRAINER is capable of capturing transferable representations that generalize well across natural images. We propose this as a direction for future work.\n\n\nWe also emphasize in RT Table 2 (and MP Table 3 ) where we train on CelebA-HQ but evaluate out of domain on high resolution Kodak Images[1] on categories of airplane and statue where STRAINER outperforms baseline models substantially or is very comparable. Further, we  also compare STRAINER with recent work on generalizable INRs [47, 48].  We show in RT Table 1,  that STRAINER outperforms meta-learning as well as transformer baselines such as TransINR[47] and IPC[48], for out of domain fitting for AFHQ and OASIS-MRI datasets even when STRAINER is trained on unrelated source domains such as stanford cars.\n\n\nDue to limited time, our experiments have been carefully chosen to address the concerns of all reviewers and are indicative of STRAINER learning highly transferable features for tasks such as image representation, with minimum training time and as little as 10 images. A comparison of training statistics and model footprints is mentioned in RT Table 3. We are happy to engage more and provide more findings during the discussion period.\n\n\n__Generalization ability of STRAINER for Inpainting.__ We thank the reviewer for raising this important point. Successful recovery for inverse problems such as single-image inpainting relies on the inductive bias of the model. STRAINER’s rich representation encoded in the shared initial layers of an INR captures a prior over the data allowing it to converge rapidly fast. However, since it’s based on SIREN, its inductive bias is the same as siren - which may result in similar performance as Siren. We expect results similar to denoising and super-resolution as reported in MP Table 5 depending on the fidelity of the available signal. Further, for a non-trivial problem such as inpainting, conventional INRs may require support from added regularization terms if lacking strong inductive biases that favor solving inpainting, as shown by INRR [49].  Additionally, convolutional architectures such as the Deep Image Prior [50] leverage self-similarity and are successful at inpainting. While INRs do not exhibit locality bias, models such as Wire[51] have strong inductive bias due to the Gabor Wavelet which favors solving inverse problems.  We’re curious to assess if STRAINER can capture any favorable properties which can benefit the task of inpainting and will explore it as future efforts to understand STRAINER deeply.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AHVJ2g8rg6", "reviewer": "Reviewer_FYQ8", "summary": "This paper proposes STRAINER, where implicit neural representations (INRs) are trained for a class of similar objects, with the INRs for all objects having a shared, instance-agnostic encoder and instance-specific decoders. The paper also examines INR training dynamics. STRAINER is evaluated on 2D image regression and fitting MRI images.", "questions": "1. While in a different setting, the IPC paper finds that the optimal instance-specific layer is the second layer of an INR. What accounts for the discrepancy between this paper's findings (the instance-specific layers should be the last few layers) and the findings of the IPC paper?\n2. Why is the LearnIt baseline not used for the Kodak task?", "limitations": "Limitations are discussed in the paper. Negative societal impact does not seem to be a concern for this paper.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "Proposes the novel idea of sharing the first $K$ layers of the INR among all INR instances for a class of objects, while having instance-specific decoders for each instance, with some improvements in performance over SIREN/meta-learning baselines. \n\nThe authors include many experimental details so the experiments should be easy to reproduce.\n\nThe paper is well-written and easy to understand.", "weaknesses": "The paper only compares against one meta-learning baseline, ignoring the large amount of related literature (Trans-INR [1], Instance Pattern Composer (IPC) [2], PONP [3], Locality-aware generalizable implicit neural representation [4], etc.) that also looks at generalizable INRs. In particular, the results of this paper seem incongruent with the results of Instance Pattern Composers, which shows that modulation is most effective in the second layer of an INR (i.e., the second layer should be instance-specific while all other layers should be instance-agnostic). \n\nThe paper has limited experimental evaluation, limited only to image reconstruction and MRI reconstruction. The paper did not investigate the more complicated tasks that were also investigated by the baselines (e.g. LearnIt), such as novel view synthesis or CT reconstruction. Results on some tasks are also only comparable to the baseline (e.g. superresolution), and some tasks do not include all the baselines (e.g. the Kodak task does not have the LearnIt baseline). \n\nNo ablation study is done on some key hyperparameters, such as the number of shared encoder layers $N$. \n\nIn light of other works on generalizable INR, the contribution of this work is limited since it does not compare to many of the other works. \n\nReferences\n1. Chen, Yinbo, and Xiaolong Wang. \"Transformers as meta-learners for implicit neural representations.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n2. Kim, Chiheon, et al. \"Generalizable implicit neural representations via instance pattern composers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n3. Gu, Jeffrey, Kuan-Chieh Wang, and Serena Yeung. \"Generalizable Neural Fields as Partially Observed Neural Processes.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n4. Lee, Doyup, et al. \"Locality-aware generalizable implicit neural representation.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for careful and thorough evaluation of our work. We appreciate the kind comments made on the novelty of our method and performance over Siren/meta-learning baselines. We welcome the reviewers suggestion on comparing with more baselines, and are thankful for the references provided and we will update them in Section 2, Background. \n\n__Comparison of baselines:__ Prompted by the primary concern of comparisons to baselines, we further conduct thorough image fitting comparison of STRAINER with IPC[48] and TransINR[47] with and without test time optimization as detailed below. As seen in RT Table 1, STRAINER significantly outperforms IPC[48], TransINR[47] and other baselines by  __~ 3-5+ db__ in PSNR. We attribute the superior performance of STRAINER to better input space partitioning learned by STRAINER as a consequence of sharing the initial layers, coupled with the instance specific decoder which provides an added degree of freedom allowing STRAINER to fit an unseen image with different morphology better. We see that pretrained TransINR and IPC yield good PSNR at iteration 0, STRAINER catches up within little time and then consistently outperforms up until the end of 2000 iterations. \n\nWe also note that STRAINER features are more transferable for out of domain image fitting as shown for AFHQ and OASIS-MRI, even when the STRAINER encoder has been pre-trained on domains such as Flowers[45] and Stanford Cars[46]. In-domain fitting for AFHQ has the best reconstruction quality, followed by STRAINER trained on CelebA-HQ, Flowers, and Stanford Cars (in decreasing order, presented in RT Table 1.).\n\n__More details on our baselines.__ We use the publicly available implementation of IPC[48] which also includes the baselines for TransINR, both of which use FFNets[38]. While STRAINER uses Siren[31], we remark that FFNets and Siren models offer comparable representation capabilities for image fitting. We train TransINR[47] and IPC[48] baselines on 14,000 images from CelebA-HQ for 300 epochs[48]. To verify successful training of the model, we ensure that the PSNR on the test set is in near agreement with that reported in their respective papers barring changes due to the number of data samples.  We also ensure that all baselines and STRAINER MLPs have 6 layers and the same number of parameters. All models are evaluated on our test images for 2000 iterations. Lastly, for the OASIS-MRI dataset, we run all baselines using 3 channel MRI images and report metrics in RT Table 1.  \n\n__STRAINER and IPC.__ In MP Figure 6.iv, we visualize the layerwise subdivisions / partitions in the input space of INRs. We observe that partitions evolve from coarse to fine as we go deeper in the INR. Initial layers provide for more global (low-frequency) transferable features. The deeper layers of the INR result in more complex subdivision of the input space giving rise to more local features and pertaining to high frequency detail in the image. As mentioned in MP Sec. 3.1, we motivate sharing layers in STRAINER to learn representations that give rise to partitions that generalize well across samples. IPC[48] proposes that the second layer in an INR be instance-specific while keeping all the remaining layers instance-agnostic. Modulating the second layer's weights would result in non-local changes resulting in suboptimal reconstruction. Further, IPC’s second layer weights obtained from a Transformer model may also affect the location of the optimal instance-specific layer.  \n\nWe further adapt IPC’s design of an instance specific second layer in STRAINER and compare it to other STRAINER models. As shown in RT Figure 1(b), we see the best reconstruction given by STRAINER-10(ours), then STRAINER (instance-specific second layer) and then SIREN - which validates our above intuition on input space division for STRAINER and IPC. \n\n\n__Metalearning baseline for Kodak:__ STRAINER learns higher quality reconstruction as indicated by image metrics (PSNR, SSIM[54], and LPIPS[53]) compared to Learnit baseline as shown in RT Table 2. We train all models on CelebA-HQ images of the same resolution as Kodak[1] images.  STRAINER performs better than SIREN and Meta-Learned 5K[37] with +2db in PSNR for network width of 256, and ~+3db for network width of 512. STRAINER(width=256) performs similar to SIREN(width=512), further attesting to the representation power of STRAINER’s learned features. We also remark that training time for STRAINER’s shared encoder on high resolution is negligible compared to meta learning models for high resolution image fitting - highlighting that STRAINER can be easily adapted to high resolution and out of domain images.\n\n__Ablation study on key hyperparameters:__\nWe refer the reviewer to SP Figure 7. We show an ablation study on the number of layers shared in the STRAINER encoder and its ability to reconstruct the signal. We observe progressive increase in reconstruction quality (PSNR) as we increase the number of layers shared.\n\n__Evaluating STRAINER on complex (3D) signals.__ Please refer to our response to Reviewer HDgr, “Evaluating STRAINER on 3D signals”\n\nDue to limited time, our experiments have been carefully chosen to address the concerns of all reviewers and are indicative of STRAINER learning highly transferable features for tasks such as image representation, with minimum training time and as little as 10 images. A comparison of training statistics and model footprints is mentioned in RT Table 3. We are happy to engage more and provide more findings during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vBJSIddAJ2", "reviewer": "Reviewer_GtRf", "summary": "This paper introduces STRAINER, a novel training framework for Implicit Neural Representations (INRs). Unlike traditional INRs, which are trained on a single signal, STRAINER aims to learn transferable features that can be effectively utilized for fitting new signals from a similar distribution. The method involves sharing initial encoder layers across multiple INRs while using independent decoder layers. This setup enables STRAINER to achieve faster convergence and better reconstruction quality compared to baseline models. The paper evaluates STRAINER on various tasks, including image fitting, super-resolution, and denoising, demonstrating its effectiveness in both in-domain and out-of-domain scenarios. Detailed analyses are provided to understand the transferability of the learned features and their implications for INR training dynamics.", "questions": "1.Can the authors provide more theoretical insights or an intuitive explanation for why sharing initial encoder layers results in better feature transferability?\n2.How does STRAINER perform when the distribution of training and test signals significantly differs? Are there specific characteristics of the signals that influence the transferability of features?\n3.Could the authors elaborate on the stability issues observed during test signal fitting? Are there specific conditions or types of signals where these issues are more pronounced?\n4.How does STRAINER compare with other transfer learning techniques that use CNNs or Transformers?", "limitations": "The authors have adequately addressed the limitations of their work, discussing the occasional instability observed during test signal fitting and the need for further characterization of the transferred features.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality:\nThe paper presents a novel approach to improving the transferability of INRs by sharing initial encoder layers. This is a significant departure from traditional methods that train INRs on a single signal without focusing on feature transferability.\n\nQuality:\nThe empirical evaluations are thorough, covering multiple datasets and tasks such as image fitting, super-resolution, and denoising. The experiments convincingly demonstrate the advantages of STRAINER in terms of reconstruction quality and convergence speed.\nDetailed analysis of training dynamics and feature transferability adds depth to the understanding of STRAINER's performance.\n\nClarity:\nThe paper is well-structured, with clear explanations of the methodology, experimental setup, and results. The inclusion of visualizations and tables enhances the readability and comprehension of the findings.\n\nSignificance:\nSTRAINER addresses a critical limitation in the current INR landscape by enabling feature transferability, which has broad implications for various applications, including medical imaging and inverse problems. This makes the work highly significant for the field.", "weaknesses": "Limited Theoretical Insights:\nWhile the empirical results are strong, the paper could benefit from more theoretical insights into why the shared encoder layers facilitate better feature transferability. A deeper theoretical framework could strengthen the contributions.\n\nStability Issues:\nThe paper mentions occasional instability in the form of PSNR drops during test signal fitting. Although STRAINER recovers quickly, addressing this issue more comprehensively would improve the robustness of the method.\n\nComparison with Other Models:\nThe comparison with other advanced models, such as those using CNNs and Transformers for transfer learning, is not fully explored. Including more comparisons could provide a clearer picture of STRAINER's relative performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for careful and thorough evaluation of our work. We appreciate the reviewer’s kind comments on STRAINER’s originality in learning transferable representations for INRs, extensive evaluation of STRAINER on datasets, and the detailed analysis provided by STRAINER on feature transferability and its training dynamics. We address the comments and suggestions below.\n\n__Theoretical insights: Why the shared encoder layers facilitate better learning.__\n\nSuppose we have an INR consisting of $L$ fully connected layers and non-linearity $\\sigma$ after each layer. Assume that $\\sigma$ is a piecewise-linear approximation of the sine function. For any piecewise linear non-linearity, the INR can be considered a continuous piecewise-affine operator, i.e., the INR subdivides the input space into pieces or regions $\\omega \\in \\Omega$, where $\\Omega_L$ is the input space partition due to all $L$ layers. Each region/piece of the function is mapped to the output in an affine manner. \n\nThe subdivision $\\Omega_L$, happens in a layer-wise fashion, e.g., layer $\\ell$ contributes $\\partial \\Omega_\\ell$ to the overall subdivision [13]. An important outcome of this layerwise subdivision is that shallower layers learn coarser partitions of the input domain, therefore resulting in shallower layers learning lower frequency features [55]. Deeper layers however can be more localized therefore, learn higher frequency or less symmetric features. \n\nThe motivation behind INR is to learn a joint partition of the shallower layers, that can be further subdivided by the deeper layers into an optimal input space partition, resulting in a low error continuous piecewise-affine operator.\n\n\n__Unstable test time optimization for STRAINER.__\nWe attribute the instability of STRAINER at test time fitting to the underlying SIREN model. In SP Figure 7 of the supplementary, reconstruction quality (PSNR) plots of Siren and STRAINER models plotted across time reflect the periodic instability. However, this can be fixed by using a learning rate (LR) scheduler.  As shown in RT Figure1(b), we apply an exponential multiplicative LR scheduler that decays the learning rate by 0.4 resulting in a smooth and stable learning for STRAINER-10 (shown in green). \n\n__Training distribution of images.__\nWe conduct experiments in in-domain (ID) and out of domain (OD) image fitting where the shared encoder is trained on CelebA-HQ[16] and tested on CelebA-HQ, AFHQ[5], and OASIS-MRI[12,21]. We notice the superior performance of STRAINER compared to baselines (see RT Table 1) for in-domain image fitting. We also find that STRAINER captures highly transferable features. For OD image fitting on AFHQ, we find that STRAINER is able to reconstruct OD AFHQ almost comparable to ID fitting on AFHQ reconstructions, differing by a single digit in PSNR (MP Table 1,2). While for OASIS-MRI, we find STRAINER trained on CelebA-HQ surprisingly outperform even ID fitting on OASIS-MRI. To further assess the effect of training distribution on STRAINER, we train the shared encoder on Flowers [45] and Stanford Cars [46] as they have different spatial distribution of color and high frequencies compared to AFHQ and OASIS-MRI. We find that STRAINER is able to learn transferable features and achieves comparable PSNR to the shared encoder trained on CelebA-HQ[16]. As shown in RT Table 1, in-domain fitting for AFHQ[5] has the highest reconstruction PSNR, followed by STRAINER trained on CelebA-HQ, Flowers, and Stanford Cars (in decreasing order, presented in RT Table 1). The results suggest that STRAINER trained on natural images yields highly transferable representations, that inturn generalize well across natural images.  We propose this as a direction for future work. \n\n__Comparison with other transfer learning techniques.__ We compare STRAINER with other metalearning and transformer baseline models such as Meta-Learned 5K[37], TransINR[47] and IPC[48] (with and without test time optimization)  for image fitting. We evaluate STRAINER and baselines on CelebA-HQ (in-domain) , AFHQ(out of domain) and OASIS-MRI(out of domain). As seen in RT Table 1, STRAINER significantly outperforms IPC[48], TransINR[47] and other baselines by  ~ 3-5+ db in PSNR. We see that pretrained TransINR and IPC yield good PSNR at iteration 0, STRAINER catches up within little time and then consistently outperforms up until the end of 2000 iterations. \n\nWe refer the reviewer to a more detailed summary of baseline results in our response to Reviewer FYQ8, “Comparison of baselines”, and “Metalearning baseline for Kodak”.\n\nAdditionally, please also refer to our response to Reviewer HDgr, “New Out-of-Distribution generalization experiments”  for detailed evaluation of out of domain generalizability.\n\nDue to limited time, our experiments have been carefully chosen to address the concerns of all reviewers and are indicative of STRAINER learning highly transferable features for tasks such as image representation, with minimum training time and as little as 10 images. A comparison of training statistics and model footprints is mentioned in RT Table 3. We are happy to engage more and provide more findings during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are thankful to the reviewers for their careful and thorough evaluation of our work. We first provide key contributions of our work and then provide additional experiments and results as requested by reviewers.  

__Summary of paper:__ 

STRAINER provides a framework to learn powerful and transferable features for implicit neural representations (INRs) by sharing a set of initial encoder layers of multiple INRs while being trained on a small set of prototypical signals (e.g. images, 3D occupancy). At test-time, for fitting an unseen signal, a STRAINER INR is initialized with the learned encoder and a randomly initialized signal-specific decoder. Our work addresses feature transferability in INRs - enabling INRs to fit unseen in-domain and out-of-domain signals faster and with better quality.

We empirically show that STRAINER learns a powerful representation from just 10 images and 24 seconds of training time (measured on Nvidia A100) and significantly outperforms models such as Siren and Meta-Learning based learned initializations[37] for image fitting - making it highly efficient for learning transferable features in data-scarce scenarios. Current data-driven learned initialization methods such as meta learning or hypernetworks rely on large models, big datasets, and long and potentially unstable training regimes[40].

Further, we provide detailed visualization of STRAINER’s learned representations, training dynamics and input space partitioning to further validate that STRAINER learns high frequency details faster and leads to better convergence. Using an approximation of SplineCAM[13], we visualize the input space partitioning which evidences STRAINER to learn more transferable features.

We also explore the nature of prior capture by STRAINER which leads to rapid or better convergence for inverse problems such as super resolution and denoising, and showcase its applicability for enabling INRs to learn powerful representations in medical imaging as well.


__Summary of Reviews.__
We thank the reviewers for their careful and thorough evaluation of our work, and for providing relevant references. Our reviewers have requested to conduct 
- Detailed evaluation of baselines with recent INR literature such as TransINR[47], and IPC[48]. 
- Showcase STRAINER on various tasks beyond image fitting, such as 3D data and inpainting.
- Offer theoretical insights in how STRAINER learns transferable features and address the difference in STRAINER and IPC for generalization.
- address distribution of training data, out of domain generalizability of STRAINER, and stability of STRAINER while fitting a new signal


__Summary of additional experiments and analysis__

- We run multiple baselines with recent transformer based models such as TransINR[47] and IPC[48] for in-domain(ID) and out-of-domain(OD) image fitting and show that STRAINER outperforms across all baselines and tasks.
- We run OD experiments on 3 channel OASIS MRI images (compared to previously reported single channel MRI images) to facilitate fair comparison across newer baselines of TransINR and IPC.
- We show additional experiments where the STRAINER encoder is trained on different source datasets Flowers[45] and StanfordCars[46] and showcase the out of domain generalizability of STRAINER.
- We provide more detailed theoretical understanding on transferable features learned by STRAINER and address the difference in STRAINER and IPC for generalization.
- We expand STRAINER beyond just 2D images, inverse problems, and apply it on 3D occupancy maps showing STRAINER to learn 12.3% better in a limited number of iterations.

For conciseness and reading clarity, we define following acronyms to refer to figures and tables from their respective sources.

- RT : Refers to the Rebuttal PDF present in the author rebuttal. 
- MP : Main paper
- SP : Supplementary material / Appendix attached to Main paper.


__References__

[44] A Chang et.al., ShapeNet: An Information-Rich 3D Model Repository, CORR 2015

[45] Kaggle, Flowers Dataset. 

[46] Kaggle, Stanford Cars Dataset. 

[47] Yinbo Chen et.al, Transformers as Meta-Learners for Implicit Neural Representations, ECCV 2022

[48] Chiheon Kim, Doyup Lee et.al, Generalizable Implicit Neural Representations via Instance Pattern Composers, CVPR 2023.

[49] Zhemin Li, et.al. Regularize implicit neural representation by itself, CVPR 2023.

[50] Dmitry Ulyanov et.al., Deep Image Prior, CVPR 2018

[51] Vishwanath Sargadam et.al, Wire : Wavelet Implicit Neural Representations, CVPR 2023

[52] Stanford 3D Scans Repository, Thai Statue.

[53] Richard Zhang et.al, The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, CVPR 2018.

[54] Zhou Wang et.al., Image Quality Assessment: From Error Visibility to Structural Similarity, IEEE Trans. on Image Processing 2004.

[55] Randall Balestriero et.al., A Spline Theory of Deep Learning, PMLR 2018.

Meta-review: All reviewer recommend acceptance as the paper proposes a novel idea for the transferability of INRs. After rebuttal reviewers were happy with the evaluations. Authors are encouraged to add the additional info in the final copy.
Decision: Accept (poster)