Learning Transferable Features for Implicit Neural Representations

Authors: kushal  vyas
ahmed imtiaz humayun
richard g baraniuk
guha  balakrishnan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 23:18:53

Abstract: Implicit neural representations (INRs) have demonstrated success in a variety of applications, including inverse problems and neural rendering. An INR is typically trained to capture one signal of interest, resulting in learned neural features that are highly attuned to that signal. Assumed to be less generalizable, we explore the aspect of transferability of such learned neural features for fitting similar signals. We introduce a new INR training framework, STRAINER that learns transferable features for fitting INRs to new signals from a given distribution, faster and with better reconstruction quality. Owing to the sequential layer-wise affine operations in an INR, we propose to learn transferable representations by sharing initial encoder layers across multiple INRs with independent decoder layers. At test time, the learned encoder representations are transferred as initialization for an otherwise randomly initialized INR. We find STRAINER to yield extremely powerful initialization for fitting images from the same domain and allow for a ≈ +10dB gain in signal quality early on compared to an untrained INR itself. STRAINER also provides a simple way to encode data-driven priors in INRs. We evaluate STRAINER on multiple in-domain and out-of-domain signal fitting tasks and inverse problems and further provide detailed analysis and discussion on the transferability of STRAINER’s features.

Main paper: 
 introduction: 
 Implicit neural representations (INRs) are a powerful family of continuous learned function approximators for signal data that are implemented using multilayer perceptron (MLP) deep neural networks. An INR f θ : R m → R n maps coordinates lying in a m-dimensional space to a value in a n-dimensional output space, where θ represents the MLP's tunable parameters. For example, a typical INR for a natural image would use an input space in R 2 (consisting of the x and y pixel coordinates), and an output space in R 3 (representing the RGB value of a pixel). INRs have demonstrated several useful properties including capturing details at all spatial frequencies [39, 36] , providing powerful priors for natural signals [36, 39] , and facilitating compression [12, 27] . For these reasons, in the past 5 years, INRs have found important uses in image and signal processing including shape representation [17, 16] , novel view synthesis [31, 34, 42] , material rendering [24] , computational imaging [5, 30] , medical imaging [49] , linear inverse problems [8, 44] , virtual reality [11] and compression [12, 27, 43, 51] . A key difference between training INRs compared to other neural architectures like CNNs and Transformers is that a single INR is trained on a single signal. The features learned by an INR, therefore, are finely tuned to the morphology of just the one signal it represents. SplineCAM [19] shows that INRs learn to finely partition the input coordinate space by essentially overfitting to the spatial gradients (edges) of the signal. While this allows an INR to represent its signal with high fidelity, its features can not "transfer" in any way to represent a second signal, even with similar content. If INRs could exhibit elements of transfer learning, as is the case with CNNs and Transformers, their potential would dramatically increase, such as by encoding data distribution priors for inverse imaging problems.In this work, we take a closer look at INRs and transferable features, and demonstrate that the first several layers of an INR can be readily transferred from one signal to another from a domain when trained in a shared setting. We propose STRAINER , a simple INR training framework to do so (see Figure 1 ). STRAINER separates an INR into two parts: an "encoder" that maps coordinates to features, and a "decoder" that maps those features to output values. We fit the encoder over a number of training signals (1 to 10 in our experiments) from a domain, e.g., face images, with separate decoders for each signal. At test time, we initialize a new INR for the test signal consisting of the trained encoder and a randomly initialized decoder. This INR may then be further optimized according to the application of interest. STRAINER offers a simple and general means of encoding data-driven priors into an INR's parameter initialization.We empirically evaluate STRAINER in several ways. First, we test STRAINER on image fitting across several datasets including faces (CelebA-HQ) and medical images (OASIS-MRI) and show (Figure 2 ) that STRAINER's learned features are indeed transferrable resulting in a ≈+10dB gain in reconstruction quality compared to a vanilla SIREN model . We further assess the data-driven prior captured by STRAINER by evaluating it on inverse problems such as denoising and super resolution. Lastly, we provide a detailed exploration of how STRAINER learns transferable features by exploring INR training dynamics. We conclude by discussing consequences of our results for the new area of INR feature transfer. 
background: 
 Implicit neural representations. We define f θ (p) as an implicit neural representation (INR) [31, 39, 29] where f θ is a multi-layer perceptron (MLP) with randomly initialized weights θ and p is the m-dimensional coordinates for the signal. Each layer in the MLP is an affine operation followed by a nonlinearity such as ReLU [31] , or sine [39] . Given an n-dimensional signal I(p), the INR learns a mapping f : R m → R n . The INR is iteratively trained to fit the signal by minimizing a loss function such as L 2 loss between the signal I(p) and its estimate f θ (p):EQUATIONwhere p i 's ∈ R m span the given coordinates, θ * are the optimal weights that represent the signal, and A is a differentiable forward model operator such as identity for signal representation and a downsampling operation for inverse problems such as super-resolution. [22] . We compare SIREN [39] model initialized by (1) random weights (SIREN), (2) fitting on another face image (SIREN finetuned), (3) STRAINER -1 (trained using one face image), and (4) STRAINER-10 (trained using ten face images). We also evaluate against multiple baselines such as Meta-Learned 5K [45] , TransINR [9] , and IPC [23] Representation capacity of INRs. The representation capacity of an INR can be described as the quality of signal the INR can represent within some number of iterations. ReLUbased INRs suffer from spectral bias during training [26] , preferentially learning low frequency details in a signal and thus leading to a blurry reconstruction of the represented signal. Fourier positional encodings [21, 31, 46] or sinusoidal activation functions (SIREN ) [39] help better capture high frequency information.Recent works increase the representation capacity of INRs with activations flexible in the frequency domain. WIRE [36] uses a continuous Gabor wavelet-based nonlinearity, and demonstrates impressive results for a range of forward and inverse problems. FINER [26] splits the sine nonlinearity to have a flexible frequency coverage, and DINER [50] uses a hash map to non-uniformly map the input coordinates to a feature vector, effectively re-arranging the spatial arrangement of frequencies and leading to faster and better reconstruction quality.Previous work has shown that smart initialization of INR weights allows for faster convergence. As shown in the SIREN study [39] , hyper-networks are proposed to capture a prior over the space of implicit functions, mapping a random code to the weights of a SIREN model. Further, TransINR [9] also shows Transformer-hypernetworks as powerful metalearners for INR weight initialization. MetaSDF [38] and Light Field Networks (LFN) [40] use meta-learning-based initialization schemes to fit signed distance functions (SDFs) and light fields. Neural Compression algorithms [14, 43] use weights obtained from meta-learning optimization as a reference to store MLP weights, leading to better compression than naively storing MLP weights. Tancik et al. [45] propose meta-learning-based approaches such as Model-Agnostic Meta-Learning (MAML) [15] and Reptile [32] for coordinate based neural representations. While these meta-learning approaches yield powerful initialization, they often require long computation time (over 150K steps [45] ) and ample numbers of training data, and are unstable to train [48] . Further, meta-learning initial modulations for an INR which are later optimized to fit data within few gradient updates has been shown to be an effective and scalable [6] strategy for smoothly representing data(sets) as functa(sets) [13] . Contrary to our approach, Implicit Pattern Composers (IPC) [23] proposes to keep the second layer of an INR instance-specific, while sharing the remaining layers and use a transformer hypernetwork to learn the modulations for the INR.Prior informed INRs. Recent work has also explored embedding a learned prior in INRs for tasks such as audio compression, noise removal, and better CT reconstructions. Siamese Siren [25] uses a similar approach where they propose a compact siamese INR whose initial layers are shared followed by 2 siamese decoders. Since 2 randomly initialized decoders will yield slightly different reconstructions, this difference is leveraged for better noise estimation in audio signal. NERP [37] learns an internal INR prior for medical imaging by first fitting high quality MRI or CT data. Weights of this learned INR are used as an initialization for reconstructing new MRI or CT undersampled data. While this paper shows a method to learn an implicit prior, their prior embedding is learned from a single MRI or CT scan of the same subject whereas our work explores learning a prior for INRs by constraining it to learn an underlying implicit representation across multiple different images. PINER [41] introduces a test-time INR adaptation framework for sparse-view CT reconstruction with unknown noise. 
methods: 
 We introduce STRAINER . We first explain our motivation to share initial layers in an INR Section 3.1. In Section 3.2 we describe the training phase of STRAINER where we learn transferrable features for INRs by sharing the initial layers of N INRs being fit independently to N images. Section 3.3, details how our captured basis is used to fit an unseen image. In subsequent sections, we seek to understand what our shared basis captures and how to expand it to other problems such as super resolution. For simplicity, we build upon the SIREN [39] model as our base model.A recent method called SplineCAM [19] provides a lens with which to visualize neural network partition geometries. SplineCAM interprets an INR as a function that progressively warps the input space and fits a given signal through layerwise affine transforms and non-linearities [19] . For continuous piecewise affine activation functions, we use an approximation to visualize (see Figure 6 ) the deep network's partition geometry for different pre-activation level sets [20] .An INR fit to a signal highly adapts to the underlying structure of the data in a layer-wise fashion. Furthermore, by approximating the spatial position of the pre-activation zero level sets, we see that initial layers showcase a coarse, less-partitioned structure while deeper layers induce dense partitioning collocated with sharp changes in the image. Since natural signals tend to be similar in their lower frequencies, we hypothesize that initial layers of multiple INRs are better suited for transferability. We therefore design STRAINER to share the initial encoder layers, effectively giving rise to an input space partition that can generalize well across different similar signals.Consider a SIREN [39] model h(p) with L layers. Let K out of L layers correspond to an encoder subnetwork represented as f θ The remaining layers correspond to the decoder sub-network represented as g ϕ as seen in Figure 1 (a). For given input coordinates p, we express the SIREN model h ϕ,θ (p) as a composition (•) of our encoder-decoder sub-networks.EQUATIONIn a typical case, given the task of fitting N signals, each of the N signals is independently fit to an INR, thus not leveraging any similarity across these images. Since we want to learn a shared representation transferrable across multiple similar images, our method shares the encoderf θ across all N INRs while maintaining a set of individual signal-specific decoders g 1 ϕ . . . g N ϕ .Our overall architecture is shown in Figure 1 . We call this STRAINER's training phase -Figure 1(a) . We start with randomly initialized layers and optimize the weights to fit N signals in parallel. For each signal I i (p), we use a L 2 loss between I i (p) and its corresponding learned estimate h i ϕ,θ (p) and sum the loss over all the N signals. Iteratively, we learn a set of weights Θ that minimizes the following objective:EQUATIONwhere Θ = [θ, ϕ 1 . . . ϕ N ] represents the full set of weights of the shared encoder (θ) and the N different decoders (g 1 ϕ . . . g N ϕ ) and Θ * represents the resulting optimized weights.After sufficient iterations during STRAINER's training phase, we get optimized encoder weights f θ * which corresponds to the rich shared representation learned over signals of the same category. To fit a We also show the (b) reconstructed images and remark that STRAINER fits high frequencies faster.novel signal I ψ (p) we initialize the STRAINER model with the learned shared encoder weights f θ=θ * and randomly initialize decoder g ψ ϕ weights to solve for:EQUATIONf θ=θ * serves as a learned initial encoder features. Our formulation is equivalent to a initial set of learned encoder features followed by a set of random projections. While fitting an unseen signal, we iteratively update all the weights of the STRAINER model, similar to any INR.During the training phase, explicitly sharing layers in STRAINER allows us to learn a set of INR features which exhibits a common partition space shared across multiple images. Since deep networks perform layer wise subdivision of the input space, sharing the encoder enforces the layer to find the partition that can be further subdivided into multiple coarse partitions corresponding to the tasks being trained. In Figure 6 (a.ii), while pre-training an INR using the STRAINER framework on CelebA-HQ dataset, we see emergence of a face-like structure captured in our STRAINER encoder f θ * . We expect our STRAINER encoder weights f θ * to be used as transferrable features and be used as initialization for fitting unseen in-domain samples.In comparison, meta learning methods to learn initialization for INRs [45] exhibit a partitioning of the input space that is closer to random. As seen in Figure 6 (a.i) there is a faint image structure captured by the the learned initialization. This is an indication that the initial subdivision of the input space, found by the meta learned pre-training layers, captures less of the domain specific information therefore is a worse initialization compared to STRAINER . We further explain our findings in Section 5 and discuss STRAINER's learned features being more transferrable and lead to better quality reconstruction. 
experiments: 
 In all experiments, we used the SIREN [39] MLP with 6 layers and sinusoid nonlinearities. We considered two versions of STRAINER : (i) STRAINER (1 decoder), where the encoder layers are initialized using our shared encoder trained on a single image, and (ii) STRAINER-10 (10 Decoders), where the encoder layers are initialized using our shared encoder trained on 10 images. We considered the following baselines: (i) a vanilla SIREN model with random uniform initialization [39] , (ii) a fine-tuned SIREN model initialized using the weights from another SIREN fit to an image from the same domain, (iii) a SIREN model initialized using Meta-learned 5K [45] , (iv) transformer-based metalearning models such as TransINR [9] and IPC [23] . We ensured an equal number of learnable Figure 4 : STRAINER converges to low and high frequencies fast. We plot the histogram of absolute gradients of layers 1,5 and last over 1000 iterations while fitting an unseen signal. At test time, STRAINER's initialization quickly learns low frequency, receiving large gradients update at the start in its initial layers and reaching convergence. The Decoder layer in STRAINER also fits high frequency faster. Large gradients from corresponding SIREN layers show it learning significant features as late as 1000 iterations.parameters (neurons) for all models. We normalized all images between (0-1), and input coordinates between (-1,1). We used the Adam optimizer with a learning rate of 10 -4 for STRAINER's training and test-time evaluation, unless mentioned otherwise. Further implementation details are provided in Supplementary.We mainly used the CelebA-HQ [22] , Animal Faces-HQ (AFHQ) [10] , and OASIS-MRI [18, 28] images for our experiments. We randomly divided CelebA-HQ into 10 train images and 550 test images. For AFHQ, we used only the cat data, and used ten images for training and 368 images for testing. For OASIS-MRI, we used 10 of the (template-aligned) 2D raw MRI slices for training, and 144 for testing. We also used Stanford Cars [2] and Flowers[1] to further validate out of domain generalization and Kodak [3] true images for demonstrating high-resolution image fitting.We first trained separate shared encoder layers of STRAINER on 10 train images from each dataset. We share five layers, and train a separate decoder for each training image. For each dataset, we trained the shared encoder for 5000 iterations until the model acheives PSNR ≈ 30dB for all training images. We use the resulting encoder parameters as initialization for test signals in the following experiments.For comparison, we also trained the Meta-learned 5K baseline using the implementation provided by Tancik et.al. [45] with 5000 outer loop iterations. We also use the implementation provided by IPC [23] as our baselines for TransINR [9] and IPC [23] and train them with 14,000 images from CelebA-HQ . We report a comparison of number of training images and parameters, gradient updates, and learning time in Table 5 .We first evaluated STRAINER on the task of in-domain image fitting. We cropped and resized all images to 178 × 178 and ran test-time optimization on all models for 2000 steps. To test out-of-domain transferability of learned STRAINER features, we used STRAINER-10 's encoder trained on CelebA-HQ as initialization for fitting images from AFHQ (cats) and OASIS-MRI datasets (see Table 2 ). Since OASIS-MRI are single channel images, we trained Meta-learned 5K and STRAINER-10 (GRAY) on the green channel only of CelebA-HQ images. To our surprise, we see STRAINER-10 and STRAINER-10 (GRAY) clearly outperform not only Meta-learned 5K , but also STRAINER-10 (in-domain). To further validate out of domain performance of STRAINER , we train STRAINER also fits high resolution Kodak [3] images well and is comparable to SIREN networks with twice the network width.STRAINER provides a simple way to encode data-driven priors, which can accelerate convergence on inverse problems such as super-resolution and denoising. We sampled 100 images from CelebA-HQ at 178 × 178 and added 2dB of Poisson random noise. We report mean values of PSNR achieved by STRAINER and SIREN models along with the iterations required to achieve the values. For superresolution, we demonstrate results on one image from DIV2K [4, 47] , downscaled to 256 × 256 for a low resolution input. We used the formulation shown in Equation (1), with A set to a 4× downsampling operation. To embed a prior relevant for clean images, we trained the shared encoder of STRAINER with high quality images of resolution same as the latent recovered image. At test time, we fit the STRAINER model to the corrupted image, following Equation (1) and recovered the latent image during the iteration. We report STRAINER's ability to recover latent images fast as well as with high quality in Section 4.5 STRAINER learns an input space partitioning which is more attuned to the prior of the dataset, compared to meta learned which is comparatively more random. We also observe that SIREN (iii) learns an input space partitioning highly specific to the image leading to inefficient transferability for fitting a new image (iv) with significantly different underlying partitioned input space This explains the better in-domain performance of STRAINER compared to Meta-learned 5K , as the shallower layers after pre-training provide a better input space subdivision to the deeper layers to further subdivide. 
discussion and conclusion: 
 Results in Table 1 , 3 demonstrate that STRAINER can learn a set of transferable features across an image distribution to precisely fit unseen signals at test time. STRAINER-10 clearly achieves the best reconstruction quality in terms of PSNR and SSIM on CelebA-HQ and AFHQ datasets, and is comparable with Meta-learned 5K on OASIS-MRI images. STRAINER-10 also fits images fast and achieves highest reconstruction quality than all baselines as shown in Figure 2 . Comparing STRAINER (1 decoder) with a fine-tuned SIREN , it seems that the representation learned on one image is not sufficiently powerful. However, as little as 10 images result in a rich and transferable set of INR features allowing STRAINER-10 to achieve ≈7-10dB higher reconstruction quality than SIREN and SIREN fine-tuned.As seen in Table 2 , 3(OOD) STRAINER also performs well on out-of-domain tasks, which is quite surprising.STRAINER's transferable representations are capable of recovering small and delicate structures as early as 100 iterations as shown in Figure 5 and do not let the scale of features from the training phase affect its reconstruction ability. Another interesting finding is that STRAINER-10 achieves far better generalization for OASIS-MRI (Table 2 ) when pretrained on CelebA-HQ . Further, STRAINER generalizes well to out-of-domain high-resolution images, as demonstrated by our experiments of training STRAINER on CelebA-HQ and testing on the Kodak data (see Table 4 ).STRAINER is fast and cheap to run. Table 5 summarizes the time for learning the initialization for a 6 layered MLP INR for STRAINER , Meta-learned 5K and transformer-based methods such as TransINR and IPC. At 5000 iterations, STRAINER learns a transferable representation in just 24.54 seconds. Meta-learned 5K , in comparison, uses MAML [15] which is far more computationally intensive and results in 20× slower runtime when exact number of gradient updates are matched. Further, STRAINER 's training setup is an elegant deviation from recent methods such as TransINR and IPC, requiring large datasets and complex training routines.Due to the encoder layers of STRAINER being tuned on data and the later layers being randomly initialized, we have observed occasional instability when fitting to a test signal in the form of PSNR "drops." However, we observe that STRAINER usually quickly recovers, and the speedup provided by STRAINER outweighs this issue. While our work demonstrates that INR parameters may be transferred across signals, it is not fully clear what features are being transferred, how they change for different image distributions, and how they compare to the transfer learning of CNNs and Transformers. Further work is needed to characterize these.To further understand how STRAINER's initialized encoder enables fast learning of signals at test time, we explored the evolution of STRAINER's hidden features over iterations in Figure 3 . In Figure 3 (a), we visualize the first principal component of learned INR features of the STRAINER encoder and corresponding hidden layer for SIREN across iterations and observe that STRAINER captures high frequencies faster than SIREN. This is corroborated by the power spectrum inset plots of the reconstructed images. We also visualize a histogram of gradient updates in Figure 4 , and observe that STRAINER receives large gradients in its encoder layers early on during training, suggesting that the encoder rapidly learns of low-frequency details.Next, we visualize the input space partitions induced by STRAINER and the adaptability of STRAINER's initialization for fitting new signals. We use the local complexity(LC) measure proposed by Humayun et.al. [20] to approximate different pre-activation level sets of the INR neurons. For ReLU networks, the zero level sets correspond to the spatial location of the non-linearities of the network. For periodic activations, there can be multiple non-linearities affecting the input domain.In Figure 6 we present the zero level sets of the network, and in Supplementary we provide the ±π/2 shifted level sets. Darker regions in the figure indicate high LC, i.e., higher local non-linearity. Figure 6 also presents partitions for the baseline models.SIREN models tend to overfit, with partitions strongly adapting to image details. Since the sensitivity to higher frequencies is mapped to specific input partitions, when finetuning with SIREN , the network has to unlearn partitions of the pretrained image resulting in sub optimal reconstruction quality. When comparing Meta-learned 5K with STRAINER , we see that STRAINER learns an input space partitioning more attuned to the prior of the dataset, compared to Meta-learned 5K which is comparatively more random. While both partitions imply learning high-frequency details, STRAINER's partitions are better adapted to facial geometry, justifying its better in-domain performance.  
 
 Appendix: In our work, we increase the representation capacity of the INR by leveraging the similarity across natural images (of a given class). Since each layer of an INR MLP is an affine transformation followed by a non linearity, we interpret the INR as a function that progressively warps the input coordinate space to fit the given signal, in our case the signal being an image. Similar images when independently fit to their respective INRs capture similar low-frequency detail such as shape, geometry, etc. whereas high-frequency information such as edges and texture are unique to each INR. We propose that these low-frequency features from the initial layers of a learned INR are highly transferable and can be used as a basis and initialization while fitting an unseen signal. To that end, we introduce a novel method of learning our basis by sharing a set of initial layers across INRs fitting their respective images.Our implementation can be found on1 Google Colab.We further investigate how the number of initial layers shared affects the quality of reconstructed image. We start by sharing K = 1 layer as the encoder, and N -K layers in each decoder and vary K from 1 to N -1. K = N is equivalent to simply fine tuning the INR based on all weights from a fellow model. We tabulate our results for image quality (PSNR) in a fixed runtime of 1000 iterations. We find that sharing all but the last layer results in the most effective capturing of our shared basis leading to higher reconstruction quality as seen in fig. 7 . This also suggests that the last decoder of the INR is mainly responsible for very localized features. Further our work motivates further interest to sutdy the nature of the decoder layers itself.We show the effect of sharing layers and resulting reconstruction quality. We use a 5 layered Siren model for this experiment. We fit a vanilla Siren model to an image and report its PSNR in fig. 7 . Further, we train our shared encoder by sharing K = 1 layers and so on , until we share K = N -1 layers.We see that the reconstruction quality progressively increases by sharing layers. Figure 7 : Sharing different number of layers in STRAINER's encoder. We see that by increasing the number of shared layers, STRAINER's ability to recover the signal also improves.We also report the PSNR within 1 std. deviation while comparing STRAINER -10 with SIREN , SIREN -finetuned, STRAINER (1-decoder), and Meta-learned 5K in Figure 9 . We wanted to further assess whether the INR is overfitting to one particular aligned face arrangement.To further test this, we take a test image and apply various augmentations such as flip, rotate, and roll and study how STRAINER fits it (see Figure 8 ). We find that the initialization learned by strainer is invariant, at test time, to the input signals orientation and can successfully capture the high frequency details fast.Our implementation is written in PyTorch [33] whereas Meta-learned 5K implemented by Tancik et.al [45] is a JAX implementation. For measuring runtime, we use the python's time package and very conservatively time the step where the forward pass and gradient updates occur in both methods. Further, we run the code on an Nvidia A100 GPU and report the time after averaging 3 such runs for each method. There may be system level differences, however, to the best of our knowledge and observation, our timing estimates if not accurate are atleast indicative of the speedup provided by STRAINER .To further demonstrate that STRAINER's can be adapted to high resolution images, we evaluated our method on high quality Kodak [3] images with resolution 512 × 768 (see Tables 4 and 7 ). We present the reconstruction quality attained by STRAINER -10, SIREN model, and a SIREN model initialized using Meta-learned 5K , with widths of 256, 512. For this experiment, we train our STRAINER encoder using CelebA-HQ Images which are resized to the same resolution to Kodak images. Further, we follow all steps as previously described for test-image evaluation of Kodak images. Here is another results from the Kodak high resolution images experiment. We discuss how STRAINER provides a useful prior for inverse problems such as super resolution. For the results reported in section 4. of STRAINER on 3D data, we have performed the following OOD generalization experiment. We pre-train STRAINER on 10 randomly selected 'Chair' objects from the ShapeNet [7] dataset. At test time, we fit the 'Thai Statue' 3D object [35] . STRAINER achieves a 12.3 relative improvement in IOU compared to random initialization for a SIREN architecture -in 150 iterations STRAINER-10 obtains an IOU of 0.91 compared to an IOU of 0.81 without STRAINER-10 initialization. We present visualizations of the reconstructed Thai Statue in Figure 12 . Upon qualitative evaluation, we see that STRAINER-10 is able to capture ridges and edges better and faster than compared to SIREN.Figure 12 : We use ten shapes from the chair category of ShapeNet [7] to train STRAINER , and use that initialization to fit a much more complex volume (the Thai statue [35] ). We compare the intermediate outputs for both STRAINER and SIREN for 150 iterations to highlight STRAINER 's ability to learn ridges and high frequency information faster.Offsetting Pre-activations https://colab.research.google.com/drive/1fBZAwqE8C_lrRPAe-hQZJTWrMJuAKtG2?usp=sharing 


Decision: All reviewer recommend acceptance as the paper proposes a novel idea for the transferability of INRs. After rebuttal reviewers were happy with the evaluations. Authors are encouraged to add the additional info in the final copy.

Reviews: review: paper_summary: paper_summary: The research focuses on learning transferable features using a SIREN model, where encoder and decoder sub-networks are utilized to optimize weights for encoding and decoding. The paper evaluates the main claims made in the abstract and introduction, confirming that the goals are clearly stated and the method effectively addresses them, supported by results. The study discusses fitting unseen signals using optimized encoder weights and highlights the fast convergence of STRAINER to low and high frequencies during training. Experiments involve using the SIREN MLP model with sinusoid nonlinearities, along with different versions of STRAINER compared to baselines, all with equal parameters for fair comparison. Datasets such as CelebA-HQ, AFHQ, and OASIS-MRI are used for experimentation, with the shared encoder of STRAINER trained on 10 images from each dataset for shared initialization in subsequent tests. The research also explores image fitting tasks in-domain and out-of-domain, demonstrating the effectiveness of STRAINER compared to other models, specifically showcasing superior performance in various scenarios like super-resolution and denoising, emphasizing the importance of learned transferable features.
strength_weakness: strength_weakness: - The paper missed mentioning a couple of important papers on meta-learning INRs, say Functa [A] and Spatial Functa [B]. Especially the latter achieves good PSNR as well as classification accuracy, despite that the methods and the focus are different.

[A] E Dupont et al. Your data point is a function and you can treat it like one.
[B] M Bauer et al. Scaling Functa to ImageNet Classification and Generation.

- The method is only tested on image datasets but not 3D shapes or radiance fields, where INRs are more applied to.
questions: questions: - Besides the improvement in PSNR, is the generalization ability expected to help with completion of the signal (i.e. inpainting)? 

- It is mentioned that the encoder trained on CelebA-HQ works even better than in-domain pretraining on AFHQ (cats) and OASIS-MRI datasets. Have you tried out-of-distribution generalization trained on other datasets as source domain?
limitations: limitations: Limitations are discussed in Section 5.1.
paper_summary: paper_summary: This paper proposes STRAINER, where implicit neural representations (INRs) are trained for a class of similar objects, with the INRs for all objects having a shared, instance-agnostic encoder and instance-specific decoders. The paper also examines INR training dynamics. STRAINER is evaluated on 2D image regression and fitting MRI images.
strength_weakness: strength_weakness: The paper only compares against one meta-learning baseline, ignoring the large amount of related literature (Trans-INR [1], Instance Pattern Composer (IPC) [2], PONP [3], Locality-aware generalizable implicit neural representation [4], etc.) that also looks at generalizable INRs. In particular, the results of this paper seem incongruent with the results of Instance Pattern Composers, which shows that modulation is most effective in the second layer of an INR (i.e., the second layer should be instance-specific while all other layers should be instance-agnostic). 

The paper has limited experimental evaluation, limited only to image reconstruction and MRI reconstruction. The paper did not investigate the more complicated tasks that were also investigated by the baselines (e.g. LearnIt), such as novel view synthesis or CT reconstruction. Results on some tasks are also only comparable to the baseline (e.g. superresolution), and some tasks do not include all the baselines (e.g. the Kodak task does not have the LearnIt baseline). 

No ablation study is done on some key hyperparameters, such as the number of shared encoder layers $N$. 

In light of other works on generalizable INR, the contribution of this work is limited since it does not compare to many of the other works. 

References
1. Chen, Yinbo, and Xiaolong Wang. "Transformers as meta-learners for implicit neural representations." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
2. Kim, Chiheon, et al. "Generalizable implicit neural representations via instance pattern composers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
3. Gu, Jeffrey, Kuan-Chieh Wang, and Serena Yeung. "Generalizable Neural Fields as Partially Observed Neural Processes." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
4. Lee, Doyup, et al. "Locality-aware generalizable implicit neural representation." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: 1. While in a different setting, the IPC paper finds that the optimal instance-specific layer is the second layer of an INR. What accounts for the discrepancy between this paper's findings (the instance-specific layers should be the last few layers) and the findings of the IPC paper?
2. Why is the LearnIt baseline not used for the Kodak task?
limitations: limitations: Limitations are discussed in the paper. Negative societal impact does not seem to be a concern for this paper.
paper_summary: paper_summary: This paper introduces STRAINER, a novel training framework for Implicit Neural Representations (INRs). Unlike traditional INRs, which are trained on a single signal, STRAINER aims to learn transferable features that can be effectively utilized for fitting new signals from a similar distribution. The method involves sharing initial encoder layers across multiple INRs while using independent decoder layers. This setup enables STRAINER to achieve faster convergence and better reconstruction quality compared to baseline models. The paper evaluates STRAINER on various tasks, including image fitting, super-resolution, and denoising, demonstrating its effectiveness in both in-domain and out-of-domain scenarios. Detailed analyses are provided to understand the transferability of the learned features and their implications for INR training dynamics.
strength_weakness: strength_weakness: Limited Theoretical Insights:
While the empirical results are strong, the paper could benefit from more theoretical insights into why the shared encoder layers facilitate better feature transferability. A deeper theoretical framework could strengthen the contributions.

Stability Issues:
The paper mentions occasional instability in the form of PSNR drops during test signal fitting. Although STRAINER recovers quickly, addressing this issue more comprehensively would improve the robustness of the method.

Comparison with Other Models:
The comparison with other advanced models, such as those using CNNs and Transformers for transfer learning, is not fully explored. Including more comparisons could provide a clearer picture of STRAINER's relative performance.
questions: questions: 1.Can the authors provide more theoretical insights or an intuitive explanation for why sharing initial encoder layers results in better feature transferability?
2.How does STRAINER perform when the distribution of training and test signals significantly differs? Are there specific characteristics of the signals that influence the transferability of features?
3.Could the authors elaborate on the stability issues observed during test signal fitting? Are there specific conditions or types of signals where these issues are more pronounced?
4.How does STRAINER compare with other transfer learning techniques that use CNNs or Transformers?
limitations: limitations: The authors have adequately addressed the limitations of their work, discussing the occasional instability observed during test signal fitting and the need for further characterization of the transferred features.

Comments: comment: We sincerely thank the reviewer for their time, effort and their positive feedback in evaluating our work! We request the reviewer to kindly update their scores if our rebuttal experiments and analysis have satisfactorily addressed their comments and questions.
We sincerely thank the reviewer for their time and effort in evaluating our work. Indeed adapting STRAINER for 3D applications such as NERF is an interesting future direction for us. We greatly appreciate the positive assessment and feedback on our work!
We sincerely thank the reviewer for their time and effort in evaluating our work. We greatly appreciate the positive assessment of our work!
I think the author has addressed the issues I raised.
Thanks for the response. I still have some concerns about how the performance is compared with Spatial Functa, although the focus is slightly different. Spatial Functa should have good PSNRs, but I see that the code is not publicly available for a direct comparison. I will raise my score.
I appreciate the authors’ detailed response to the points raised by the reviewers. Most of my main concerns, such as comparisons to baselines such as IPC have been addressed by the authors, except the limitation that the method is mostly limited to signal fitting and is either inapplicable (e.g. NeRF) or has mixed performance (e.g. super-resolution, denoising) on inverse problems. Therefore, I would like to raise my rating to ‘borderline accept’.
Functa[A] proposes to use neural fields (e.g. INRs) as data points itself for deep learning tasks such as classification and generation, but is limited in its ability to represent complex datasets such as ImageNet. Spatial Functa[B] addresses the limitation by using a spatially arranged latent representation of neural fields. At the core, both these papers use a separate _model_ to modulate the weights of an INR that represents an instance of the data, which is later used for downstream deep learning tasks. STRAINER (our work) provides a framework for learning powerful and transferable features for INRs, capturing a shared low-frequency across images by sharing initial _encoder_ layers of an INR and having signal-specific _decoder_ layers. Our work addresses feature transferability in INRs - enabling INRs to fit unseen in-domain and out-of-domain signals faster and with better quality. As you had previously mentioned and we agree, that Functa[A] and Spatial Functa[B] have different focus from our work (STRAINER). Further, since the code for Spatial Functa is not publicly available and due to limited available time in the discussion period, we find it unlikely to run additional baselines of Spatial Functa.

Similar to Spatial-Functa, we find TransINR[47] and IPC[48] as reasonable comparisons since they too have separate _models_(transformer networks) that modulate the weights of the INR. We report our comparison in the rebuttal PDF RT Table 1. We show that while TransINR[47] and IPC[48] are trained using 14000 images for 1 day, STRAINER (our work) outperforms them significantly $\approx$ 5-7+db for in-domain and out of domain image fitting with just 10 training images and 24 seconds of training time. We also show a detailed comparison of training data size, training time and number of learned parameters in RT Table 3, and note that STRAINER uses orders of magnitude lesser training data and faster training time. 

We show extensive comparisons with other baselines in RT Table 1, and emphasize on STRAINER’s superior performance. We also show STRAINER on other modalities such as 3D occupancy maps.  

We request the reviewers to update their scores if our rebuttal experiments have addressed their questions and comments satisfactorily.
Thanks for the response.

I wonder if Spatial Functa should be added as a baseline in the experiments in rebuttal PDF Table 1. In terms of PSNR this method is also competitive.
As we approach the end of the discussion period, we kindly urge the reviewers to update their scores if our rebuttal experiments have satisfactorily addressed their questions.

We sincerely appreciate the time and effort put in the careful evaluation of our work.