Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models
Authors: Jeffrey Gu
Serena Yeung-Levy
Venue: ICLR 2025 Poster (2025)
Keywords: neural fields
neural implicit representations
foundation models
hypernetworks
generalizable INR
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Vision foundation models improve hypernetworks despite the modality difference

Abstract: Large pre-trained models, or foundation models, have shown impressive performance when adapted to a variety of downstream tasks, often out-performing specialized models. Hypernetworks, neural networks that generate some or all of the parameters of another neural network, have become an increasingly important technique for conditioning and generalizing implicit neural representations (INRs), which represent signals or objects such as audio or 3D shapes using a neural network. However, despite the potential benefits of incorporating foundation models in hypernetwork methods, this research direction has not been investigated, likely due to the dissimilarity of the weight generation task with other visual tasks. To address this gap, we (1) show how foundation models can improve hypernetworks with Transformer-based architectures, (2) provide an empirical analysis of the benefits of foundation models for hypernetworks through the lens of the generalizable INR task, showing that leveraging foundation models improves performance, generalizability, and data efficiency across a variety of algorithms and modalities. We also provide further analysis in examining the design space of foundation model-based hypernetworks, including examining the choice of foundation models, algorithms, and the effect of scaling foundation models.

Published as a conference paper at ICLR 2025
FOUNDATION MODELS SECRETLY UNDERSTAND NEU- RAL NETWORK WEIGHTS: ENHANCING HYPERNET- WORK ARCHITECTURES WITH FOUNDATION MODELS
Jeffrey Gu, Serena Yeung-Levy
Institute for Computational and Mathematical Engineering (ICME), Department of Biomedical Data Science Stanford University Stanford, CA 94305, USA
{jeffgu, syyeung) (yyeunng)sttanfodd.ede
ABSTRACT
Large pre-trained models, or foundation models, have shown impressive perfor- mance when adapted to variety of downstream tasks, often out-performing specialized models. Hypernetworks, neural networks that generate some or all the parameters of another neural network, have become increasingly important technique for conditioning and generalizing implicit neural representations (INRs), which represent signals objects such as audio 3D shapes using neural net- work. However, despite the potential benefits of incorporating foundation models hypemetwork methods, this research direction has not been investigated, likely due to the dissimilarity of the weight generation task with other visual tasks. To address this gap, we (1) show how foundation models can improve hypernetworks with Transformer- based architectures, (2) provide an empirical analysis of the benefits of foundation models for hypernetworks through the lens of the generalizable INR task, showing that leveraging foundation models improves performance, generaliz- ability, and data efficiency across variety of algorithms and modalities. We also provide further analysis in examining the design space of foundation model- -ased hypernetworks, including examining the choice of foundation models, algorithms, and the effect of scaling foundation models.
INTRODUCTION
Foundation models, models that are pre-trained using self-supervision diverse large-scale datasets and are readily adaptable to a wide variety of downstream tasks (Bommasani et al. 2021). have revolutionized Al these models have formed the backbone for state-of-the-an models many tasks across a wide range of modalities, such as zero-shot image classification (Radford et al. 202 I] and segmentation (Kirillov et al. 2023). However, the use of foundation models has not been investigated for many downstream tasks where they may be useful.
Hypernetworks, which neural networks that produce adapt some all of the weights of another
neural network, have been investigated as a way to create adaptive layers (Ha et al. 2016: Ba et al.
2016 Goyal et al. 2019), perform neural architecture search (Brock al. 2017] Zhang all (018a). meta- lernnng (Andrychowicz et al., 2016 Zhao et al. 2020) and multi multi-task learning (Tay et al.
2020). continual learning (Von Oswald et 2019), and more. One major area of hypernetwork research is using hypernetworks as means of conditioning or creating generalizable implicit neural representations (INRs) (Mescheder al. 2019 Sitzmann et al., 2019 20206 Chen & Wang 2022
Gu et al. 2023 Kim et al. 2023] Lee et al. 2024). INRs, also known coordinate-basee neural networks or neural fields, represent signals or objects using a neural network and have emerged as a continuous and memory-efficient alternative to traditional discrete representations (Sitzmann et al. 2020b). Typically, INRs are trained to represent a single object from many partial sensor observations of that object. Generalizable INRs improve this training framework by leveraging additional data to improve INR quality (Tancik et al. (202T: Chen & Wang 2022] Gu et al. 2023 Kim et al., 2023 Lee
al. 2024.. training efficiency (Tancik al. 2021) and speed (Hong et 2023 well allow

---

Published as a conference paper at ICLR 2025
the generation of INRs for objects that would otherwise have insufficient partial observations (Hong et al. 2023).
Despite the success of foundation models for other tasks, there has been little investigation into adapting foundation models to improve hypemetworks and generalizable INRs, as none of the state-
of-the-art methods (Chen & Wang 2022 Gu et al.. 2023 Kim al.) 2023 Lee et al. 2024 leverage foundation models. We believe that this is due to modality gap between neural network weights, the output of the hypernetwork task, which differs significantly from the outputs of more typical tasks such as image classification. We address this gap in the existing literature by answering the following questions: Do foundation models improve hypernetwrk performance on the generalizable INR task? In which ways do they improve hypernetworks? And if they do. how should one design a hypernetwork from a foundation model? To answer these questions, we first augment Transformer- based generalizable INR architectures (Chen & Wang 2022, Gu et al. 2023, Kim et al. 2023) with foundation models and show that adaptation via fine-tuning improves downstream task performance, generalization to unseen classes, and data efficiency. We also show that the foundation model approach can outperform existing approaches even when the foundation model features are frozen and only linear heads and extra tokens trained top of these frozen features produce the weights of each layer. In addition, we provide an amalysis of the design space of adapting foundation models to hypernetworks through targeted experiments exploring 1) the choice of foundation model, 2) the choice of algorithm, and 3) scaling properties. Finally, we show that the performance is robust by examining different modalities.
The contributions of our paper are as follows: first, using a framework based on existing Transformer- based hypernetworks, we show that foundation models improve hypernetwork performance on the generalizable INR task on different modalities. Second, we perform additional experiments to analyze many different facets of performance, examining generalization to unseen data, data efficiency, and parameter efficient approaches. We also provide additional analysis on the design space of adapting foundation models to hypernetworks, the choice of algorithm, examining the choice of foundation model, how performance scales with the number of foundation model parameters.
2 BACKGROUND AND SETUP
In this section, we first describe how hypernetoorks can be used to generate generalizable INRs. We then describe the architecture of the foundation model-based hypernetwork we use, which closely based on the Trans-INR (Chen & Wang 2022). monsformer-basel hypernetwork architecture that produces an INR in one forward pass of the hypernetwork. We then discuss the design space of hypernetworks and describe how parameter-efficient fine-tuning can be done using prompt tuning (Jia et al. 2022).
Hypernetworks for INRs to a space of quantities Y. An implicit neural representation (INR) represents the signal I by parameterizing it with neural network fo with weights 6: [(x:) fo(x), Vz. INRs are typically trained with only partial observations v of the signal I using a forward map F that maps the outputs y the partial observations v', and are supervised with reconstruction loss between v, v'. For example, in Neural Radiance Fields (NeRF) (Mildenhall et al. 202T) the signal parameterized is the radiance field of 3D scene (I) and the partial observations v are 2D views of the scene. The forward map F is volume rendering, an operation which takes the radiance field and produces the predicted partial 2D view v'. The INR then supervised with a reconstruction loss between and v', which the mean-squared error (MSE) in this example.
Define signal / X + Y as a function that maps coordinates X
hypernetwork is neural network that produces the weights of another neural network. In our case, a hypernetwork Is produces the weights of an INR So given some partial observations v of the signal I: (v) = 0. The hypernetwork model is then supervised by reconstruction loss between the v and the predicted partial observation o' F(fgator (z)), where are the coordinates corresponding to and F is the forward map. Only the hyperparameter weights e are optimized using backpropagation.
Foundation Model Framework for Hypernetworks We base the foundation model framework for analysis (Figure [ on the Trans- INR architecture (Chen & Wang 2022), a Transformer based hypernetwork architecture for generalizable INR. It consists of four main components: (1) a pre-
2

---

Published as a conference paper at ICLR 2025
Image
Transformer borr-be--ddddd basedaaaddde. Foundation Model
Weight Tokens
Head
Implicit Neural Representation
Head
Head
Head
Base Params
Figure 1: An overview of the hypernetwork-foundatiii model framework. First, an image is tokenized and concatenated with learnable weight tokens. Second, all tokens are encoded by a pre-trained foundation model encoder (Eq. IT. Tokens then grouped, transformed using linear heads Head,, and multiplied element-wise @ with the base parameter BaseParam. (Eq 2 and normalized (not shown). The resulting masked weights are then used instantiate an implicit neural representation (INR). The INR can then be trained as usual.
trained Transformer foundation model consisting of an embedding layer Embed and d-dimensional Transformer encoder Enc consisting of attention blocks [B. (2) extra learnable input tokens {w]) 1 1- (3) an INR f. generally a ReLU MLP with positional encoding composed of layers Tak and (4) a learnable linear head Head, for each layer L, of the INR, and (5)
of learnable base parameters for the INR, which will be modulated by the output of each linear head Head,- This improves training compared to directly producing the weights (Ortiz al. 2023). The main difference between our framework and Trans-INR (Chen & Wang 2022) is that our framework uses a pre-trained Transformer- based foundation model, whereas Trans -NR uses a Transformer encoder trained from scratch. Given input data instance, such as a 2D view in the example above, it is discretized and embedded into R2d by the embedding layer Embed to get data tokens [49:.... (%.) Superscripts indicate the number of attention blocks B3 a token has passed through. For simplicity, unlike Trans-INR we do not use Embed to tokenize task- or modality-speciff auxiliary data such camera pose information in the novel view synthesis task. The extra weight tokens are then concatenated to the data tokens and fed through the transformer encoder:
[t,,..., 'n wt, ...... wn = (B( d- w - w/''))) S
(I)
Finally, only the output tokens corresponding to weight tokens used to generate the weights of each layer:
L = Norm(teeady(w w.. BaseParamy)
(2)
where Norm is operation that normalizes the weights to have unit La norm. For computational efficiency, we keep the weight grouping scheme of Trans-INR, where each token only helps to generate the weights of a single layer, with the number of tokens being the total number of parameters in the layer divided by some hyperparameter g. More details can be found in Chen & Wang 2002). The model trained end-to-end by generating the weights of the INR using the encoder (Embed d and Enc), using the INR to predict the data instance (see the previous section and Figure and supervising the training with reconstruction loss (not shown in Figure
Prompt Tuning Transformer-baitt based Hypernetworks As our framework is based on the Trans- former architecture, parameter-efficien fine-tuning (PEFT) methods for Transformers such as prompt tuning (Ja et al. 2022) can be used almost directly. Prompt tuning is particularly simple because our framework already has learnable prompt tokens the form of the weight tokens (the red tokens in Figure D. so (shallow) prompt tuning can be achieved by just freezing the weights of the pre-trained foundation model encoder, consisting of the embedding layer Embed and Transformer encoder Enc, and fine-tuning the remaining weights, which consist of the learnable weight tokens w.. INR weight-producee linear heads Headk, and base parameters BaseP a am. Unlike prompt tuning, the token input to the linear heads corresponds the learnable prompt tokens and not the data tokens.
3

---

Published as a conference paper at ICLR 2025
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Pre-trained Backbones We experiment with the following large pre-trained models: supervised ViT (Dosovitskiy et al. 2020) trained on ImageNet-2IK (Deng et al. 2009). DeiT (Touvron et al. 2021 a supervvsed model trained on ImageNet-IK lk (Deng et al. 2009) using distillation, DINO (Caron et al.
2021), a supersupe model pre-trained with self-distillation, DINO v2 (Oquab al. 2023), which improves on DINO with additional curated training data and other improvements, CLIP (Radford ct al. 202T), ), large vision- -aiiuul model trained using an image-text contrastive loss, and MAE (He et al. (022), which is pre-trained with masked image modeling. For audio, we use Whisper (Radford et al. 2023), , an encoder-decoder model that is self-supervised variety of speech tasks. For the Whisper model, we only use the encoder.
Baselines addition to our base framework, which is based on Trans-INR (Chen Wang 2022),
also examine two state-of-the-ar extensions which are easily adapted to our framework by replacing their Transform backbone with pre-trained foundation models. PONP (Gu et al. 2023). representative of neural process- wwwpsssss-bse/dd//eee broae//robabillisti methods, improves
INN by adapting the neural process (NP) (Garnelo et al., 20185) meta-learning algorithm for generalizable INR learning. PONP learns probabilistic INR instead of the deterministic one used by Trans-INR, with the output layer of the INR producing mean and variance predictions instead of point predictions. Instead of using MSE a reconstruction loss, PONP uses the maximum- -iielliooioeoiodd loss of conditional NPs (Garnelo et al. 2018a). Instance Pattern Composers (IPC) (Kim et al. 2023). representative of weight-sharing methods, improves on Trans-INR by using low-rank weight modulation modulate just one weight matrix of one layer of the INR to instance-specific parameters while sharing all other weights of the INR among all data instances.
Training Training is done in of three settings: 1) randomly initialized, where all the weights of the model are randomly initialized and then trained, 2) fine-tuned, foundation model or FM, where the o
Transformer encoder in our framework is initialized with the weights of a foundation model and the whole model is fine-tuned, and 3) frozen or prompt tuned, which corresponds to our prompt tuning approach where the Transformer encoder is initialized with foundation model weights and frozen.
3.2 TASKS
We experiment on the following datasets and tasks:
Novel view synthesis We use the novel view synthesis (NVS) dataset of Learnit (Tancik et al. 202I). which consists of 50 rendered views of shapes in the cars, chairs, and lamps categories of the ShapeNet (Chang et al. 2015 3D object dataset. Given a set of views of an object and a new viewing direction, the objective is to generate a view that best matches the ground truth view in that viewing direction. To fairly compare among different pre-trained models, unless otherwise stated we only examine models using the ViT-B/16 architecture. We restrict to the case where a single input view of the object is given. Unlike previous works (Taneik et al.) 202T Chen & Wang 2022 Guo al. 2023) Gu et al. 2023 Kim et al. 2023). unless otherwise stated we train on all categories at once instead of training a separate model for each category, a much harder task, and also evaluate using the average performance on each category. We numerically evaluate all methods with four metrics that cover different aspects of image similarity: peak signal-to-non ratio (PSNR), SSIM (Wang et al.
2004. LPIPS (Zhang et al., 2018b), and FID (Heusel et al. 2017).
Audio reconstruction For audio reconstruction, following IPC Kim et al. 2023 we use the LibriSpeech-clean audio dataset (Panayotov al. 2015). The framework is trained on randomly cropped audio, while test audio is trimmed to Is for evaluation (Kim et al. 2023) which is done with PSNR. For this method, we only benchmark the IPC algorithm.
3.3
MAIN RESULTS
Our main results can be found in Tables 1
We find that:
4

---

Published as a conference paper at ICLR 2025
Table Comparison of different foundation models using the ViT-B/16 architecture as backbones on the NVS task. All backbones examined outperform random initalization except for MAE (He et al. 2022..
which we hypothesize is due the lack of global image representation learning.
Backbone
PSNR (1)
SSIM (t)
LPIPS ())
FID (4)
Randomly initialized
20.862
0.8357
0.1511
0.2751
MAE (He et al.
2022)
20.701
0.8312
0.1753
0.2866
Supervised (Dosovitskiy et al. 2020
21.324
0.8501
0.0966
0.1516
DeiT (Tovrron et al. (2021)
21.587
0,8530
0.1125
0.1860
DINO Caron
(021)
21.737
0.8555
0.1101
0.1810
CLIP Radford et al.] T1.21
21.770
0,8556
0.1126
0.1834
DINOv2 (Oquab al. 2023
22.095
0.8609
0.1063
0.1705
1. In general, hypernetworks with large pre-trained models as backbones outperform hyp hypernetworks with the same architecture trained from scratch, but the choice of pre-trained model matters (3.4). Initializing hypernetwork weights from large pre- trained model improves performance in general, although not all foundation models lead improvements due differences in pre-training strategy. In particular, learning good global image representation seems to be crucial.
2. Foundation models improve generalization to classes unseen during training ( 33.5 full fine-tuning may cause some forgetting of generalizable foundation model features.
3. Hypernetworks with frozen foundation model backbones have at least comparable performance to hypernetworks with the same architecture trained from seratch ($3.7 while using significantly fewer learnable parameters (100K vs 87M parameters).
4. Foundation model-based hypernetworks scale ($36 $3.8 Hypernetworks augmented with foundation models are both more data efficient ($3.6 5.6 and perform better with larger foundation models ($3.8).
5. The effects of foundation models are robust over different algorithms and different modalities ($3.9) We find that hypernetwork performance increases across different al- gorithms, including neural process- baapesss-basddprobbbbbiie weight-sharing methods (Kim et al., 2023), as well as over different modalities (3D objects and audio).
methods (Gu et al. 2023) and
3.4 FOUNDATION MODELS INCREASE HYPPRNETTWOKK PERFORMANCE
Table shows that fine-tuning our foundation model framework for hypernetworks outperforms training from random initialization for almost all of the investigated foundation models, with the exception of MAE (He et al. 2022). We hypothesize that the poor performance of MAE is due to its masked image modeling self self-supervision, which learns good mid-level interaction between image
patches (Li et al. 2022a), but fails to learn good global features Liang et al. al.](222)) (2022). We find that the three best foundation models are CLIP (Radford et al.) 2021 DINO (Caron al. (021). and DINOv2 (Oquab et al. 2023). which we hypothesize is due to these methods learning strong global image representations during pre-training. The contrastive pre-training objective of CLIP promotes the learning of global image representations (Li et al. 2024), whereas the DINO self-distillation objective (Caron et al. 202T) encourages the [CLS] ] token both DINO and DINOv2 to learn global image representation (Li et al. 2024). We hypothesize that learning good global image representations is crucial for the NVS and generalizable INR tasks.
Qualitative results can be found in Figure 4 We find that the foundation model approach is better at learning the shape of objects, such as the curved back chair.
3.5 FOUNDATION MODELS IMPROVE HYPERNETWORK GENERALIZABILITY TO UNSEEN CLASSES
Due to their large pre- trainii datasets, we hypothesize that fine-tuning from foundation model features should also lead to better zero-shot generalization unseen classes. To test this, we train
5

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of hypernetwork generalizability classes unseen during training using random initialization, fine-tuning from DINOv2 (Quub et al. (023). and prompt tuning with frozen DINOv2.
Each method was trained with only two of the classes the ShapeNet NVS dataset and evaluated on the third, unseen class. The best metrics are highlighted in bold.
Method
Training - Test
PSNR (1)
SSIM (T)
LPIPS (1)
FID (L)
Randomly initialized
cars, chairs + lamps 17.377
0.7959
0.1898
0.1179
Frozen
cars, chairs + lamps 18.346
0.7972
0.2941
0.3033
Fine-tuned
cars, chairs + lamps 17.474
0.7977
0.1903
0.0956
Randomly initialized
cars, lamps chairs
13.163
0.6212
0.3751
1.0199
Frozen
cars, lamps + chairs 13,536
0,6112
0,4279
1.0017
Fine-tuned
cars, lamps + chairs
13.322
0.6238
0.3845
0.9680
Randomly initialized
chairs, lamps - cars
15.431
0.7521
0.2987
0.3276
Frozen
chairs, lamps cars 15.503
0.7465
0.3607
0.3623
Fine-tuned
chairs, lamps + cars
15.382
0.7692
0.2310
0.1548
two of the three classes in the ShapeNet NVS dataset and evaluate the third class, which is unseen during training, using the random and fine-tune strategies. Furthermore, to see if full fine-tuning is degrading the generalizability of foundation model features through catastrophic forgetting, we train additional models where the foundation model is frozen. In Table 2 we find that the full fine-tuning of foundation models improves chot generalization unseen classes over training from scratch, and that reconstructions (as measured by PSNR) can be improved even further if the foundation model frozen instead of fine-tuned, indicating that some of the generalizability of the pre-trained features is lost during full fine-tuning.
PSNR
SSIM
0.75
Kakki
lundendonn
lindensanoe (Pianes
Percent total data
Percent of tota data
(a) PSNR (f)
SSIM (t)
Remter Dwn
X12
415
Percent of total data
Percent of total data
(c) LPIPS (L)
(d) FID (L)
Figure 2: Plots showing performance vs the amount of training data for both the randomly initialized (Random) and foundation model (FM) strategies.

---

Published as a conference paper at ICLR 2025
Table Comparison of the three different training strategies using DINO (Caron al. 2021 on the NVS task. We find that models with a frozen DINO encoder perform better than the same model randomly initialized on PSNR, while remaining close on the other metrics with a fraction of the parameters. Full fine-tuning results in significant increase in performance for all metrics.
Method
Trainable Parameters
PSNR (1)
SSIM (t)
LPIPS (L)
FID (4)
Randomly initialized Frozen
87.3M (100%)
20.862
0,8357
0.1511
0.2751
100K (0.11%)
21.035
0.8335
0.1767
0.3212
Fine-tuned
87.3M (100%)
21.737
0.8555
0.1101
0.1810
Table 4: Comparison of the effectiveness of foundation models for different hypernetwork algorithms on the novel view synthesis task. We find that regardless of the algorithm type. using foundation model significantly improves performance. The best performing models are bolded.
Method
PSNR (t)
SSIM (1)
LPIPS (L)
FID (L)
Trans-INR (Chen & Wang 2022)
20.850
0.8346
0.1586
0.2680
Fine-tuned Trans-INK
22.095
0.8609
0.1063
0.1705
PONP (uu et al. 2023)
20.878
0.8357
0.1584
0.2795
Fine-tuned PONP
21.993
0.8591
0.1083
0.1765
Instance Pattern Composers (Kim et al.
2023
20.102
0,8344
0,1811
0.2655
Fine-tuned Instance Pattern Composers
20.672
0,8324
0.1450
0.1971
3.6
FOUNDATION MODELS IMPROVE HYPERNETWORK DATA EFFICIENCY
In Figure 2 compare the random initialization and foundation model strategies when trained on 1 1%, 10%, 20%, and 50% of the data. We find that for PSNR, SSIM, and LPIPS, the foundation model approach significantly outperforms random initialization on every category, and that these metrics closely correlated. We observe that, unlike the other metrics, FID seems to plateau quickly and may even increase slightly with more data. One possible explanation is that FID may not detect gradual improvements in image quality and may instead incorrectly indicate quality degradation @Jyasumana al., 2024), which may be happening here as the image quality gradually improves due to the increasing amount of training data. This indicates that foundation model-based hypernetworks will be better able leverage the increasingly larger datasets being published, such as Objaverse (Deitke)
al. 2023
3.7 FROZEN FOUNDATION MODELS ENABLE PARAMETER EFFICIENT HYPERNETWORKS
In Table we find that even if the Transformer encoder weights frozen, the model's performance can perform on par or even exceed that of the same model randomly initialized, despite using only a fraction of the learnable parameters (100K vs 87M). This means that even if there are computational considerations, training a hypernetwork using the simple formula of extra input tokens, frozen pre-trained backbone, and MLP heads is promising approach. Surprisingly, despite the modality difference between image features and the weights of an INR, prompt tuning can succeed with only a linear head producing the weights of each layer.
3.8 SCALING LAWS FOR HYPERNETWORKS
As shown Figure S we find that increasing the number of parameters of the foundation model generally increases performance on all metrics. This suggests that being able to scale foundation models to more parameters would directly lead to an increase hypernetwork performance. All foundation models investigated showed improved performance with more data with the exception of the supervised ViT (Dosovitskiy et al. 2020), where PSNR increased but all other metrics decreased, with the caveat that many models, including the supervised VIT. only had two model sizes tested. It has been observed before that for ViTs trained on image classification, better upstream performance
does not necessarily result in better performance downstream tasks (Zhai al. 2022

---

Published as a conference paper at ICLR 2025
646
bel
24-
6450
218
0455
PSNR
SSIM
22.
30'
Ie
Parameters
Parameters
(a) PSNR (1)
(b) SSIM (1)
0.7
0.8
D10 CLP
318
033
015
57
LPIP55
0.4
022
A12
016
Parameters
Parameters
(c) LPIPS (1)
(d) FID (4)
Figure 3: Plots of NVS performance vs number of Transformer encoder parameters, as measured by the four metries, the NVS task using the Trans-INR algorithm. We find that increasing model size generally leads to increased performance, with supervised ViTss (Dosovitskiy et al. 2020) being a clear outlier.
3.9 ROBUSTNESS BETWEEN ALGORITIMS AND MODALITIES
Since our hypernetwork framework is based on Trans- (Cen Wang. 2022). follow-up improve- ments (see Sec. 3.1 to this framework can also be enhanced with foundation models. In Table 4 we find that the improvement provided by using foundation model backbones persists regardless of the type of algorithm used (e.g. probabilistic (Gu al. 2023) or weight-sharing (Kim et al.) 2023
We note that while past work showed that weight-sharing approaches (Kim et al. [023] Lee et al. when training separate model per category, they perform much worse
2024) were than competing algorithms when training is done across categories. We hypothesize that this is due to these methods sharing many of the INR parameters among all data instances, limiting the expressivity of the model and resulting in underfitting. This drop in performance holds with the addition of foundation models. In contrast, PONP continues to perform slightly better than Trans- INR in this setting, but with the addition of foundation models, fine-tuned Trans-INR performs slightly better than PONP.
Table showss that this effect extends across different modalities to audio, indicating the robustness of the benefits of foundation models to hypernetworks. Notably, we see that parameter-efffiieet prompt tuning performs slightly better than model with random initialization.
4 RELATED WORKS
Implicit Neural Representations (INRs) INRs represent complex data such 3D objects, scenes, and audio by parameterizing them using a neural network. Architectures for INRs include using Fourier features Mildenhall et al. 2021 Tancik et al.) 2020) and sinusoidal activation functions
(Sitzmann et al.) 2020b). The flexibility of the INR framework has led to applications in a wide variety of domains, including 3D shape and scene reconstruction (Mildenhall et al. 2021 Sitzmann
8

---

Published as a conference paper at ICLR 2025
Table 5: Audio reconstruction on the LibriSpeech dataset using the weight-sharing hypernetwork
approach of Instance Pattern Composers (Kim et al. 2023). The best performing model is bolded.
Method
Params
PSNR (1)
Randomly initialized
72.7M (100%)
24.431
Prompt-tuned Whisper-base (Rafford al. 2023
100K (0.1%)
24.432
Fine-tuned Whisper-tiny (Radlord et al. 2021)
37.9M (52.1
24.431
Fine-tuned Whisper-base
Radtord et al. 2023
72.7M (100%)
24.434
Input view
GT
Rand.
FM
Input view
GT
Rand.
FM
Figure 4: Comparison of qualitative results between the best foundation model-based hypernetwork and hypernetworss trained from scratch. Novel views generated with the hypernetwork approach (FM) are more faithful the groundtruth than the baseline (Random). For example, the lamp in the middle row is better reconstructed at both the top of the lamp and on its stem, while for the two chairs the FM approach better captures their curved backs. You may need zoom in to see the differences.
et al. 2019 2020b a, generative models (Poole et al. 2022 Liu et al. 2023). robotics (Li et al. 20226 and more.
Generalizable INR The problem of learning a generalizable INR is usually formulated as a meta-
learning task, where learning INR for each signal is separate task (Tancik et al.) 2021 Chen
& Wang 2022] Gu et al. 2023). Early methods used auto-decoding (Mescheder et al. 2019] Park
al. 2019), where latent vector is optimized per-instance and concatenated with the input the
INR. The current major approaches to this problem are gradient-based meta-learning. hypernetworss,
and neural processes. The gradient-based weta-learning approach (Sitzmann et al. 2020a Tancik
ct al.
2021 Lee et al. 202I) uses algorithms such as MAML (Finn et al. (017)) 7) or Reptile NNchhol al.
2018) learn good INR initialization that can be quickly finetuned, but has the disadvantage
of requiring additional test-time optimization. Hypernetwork approaches (Mescheder et al., 2019
Sitzmann
2020b 2019 2021 Chen Wang (022) use a separate shared encoder that generates the weights of an INR, and have fast inference, as an INR can be generated in one forward pass of
encoder. Neural process approaches (Guo al. 2023 Gu et al. 2023) use the neural process
meta-learning framework (Gamelo et al.) 2018b which use neural networks to parameterize
stochastic process. This approach may be combined with hypernetwork approaches (Gu et al. 2023). Other approaches generalizable INR follow the strategy of improving INRs by distillation from foundation models (Wang et al. 2022 Ye et al. 2023 Liao et al. 2024). In FeatureNeRF (Ye et al. 2023). foundation model features are distilled by training non-hhyernetwork) generalizable INRs to jointly predict foundation model features along with the reconstruction. Unlike these works, our model focuses only on improving hypernetwork architectures with foundation models.
Hypernetworks Hypernetworks are neural networks that produce or modify the parameters of another network. In this paper, we focus on hypernetworks that generate implicit neural representa- tions. Hypernetworks are used as a means of conditioning implicit neural representation generation and also create generalizable implicit neural representations. Early works using hypernetworks

---

Published as a conference paper at ICLR 2025
to generate implicit neural representations (Mescheder al. 2019 Sitzmann et al. 2020b
2019
2021b. Early hypernetworks methods for generating INRs used simpler MLP SSizzann et al. 2019
2024
convolutional (Mescheder et al. 2019 Sitzmann et al. 2020b) architectures for the hyper- network. Trans-INR (Chen & Wang 2022) proposed using the more powerful vision transformer
(ViT) (Dosovitskiy al. 2020) the base architecture for hypernetworks, and this method has been improved upon by incorporating neural processes (Gu et al. 2023) or using weight modulations to learn only some of the layers of the INR while sharing the rest of the parameters (Kim al. 2023
Lee et al. 2024). Our work examines the impact of foundation models on hypemntwwrks using Transformer-based architectures, which to the best of our knowledge has not been examined before.
CONCLUSION
We present rigorous investigation of using foundation models improve hypernetworks for generalizable INR tasks, providing key insights for designing future hypernetwork models. We demonstrate that foundation models improve hypernttwrk performance on both seen and unseen classes, and show that this effect is robust. We also provide a parameter-efficien way to create hypernetwork models based on prompt tuning. We also further analyze the effect of using foundation models, looking at the choice of foundation as well as scaling with data and parameters. We hope that our investigation serves as a starting point for investigating foundation models for hypernetwork architectures.
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. Advances in neural information processing systems, 29, 2016.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07255. 2021.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXiv: 1708 05344, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint /tt:::/512.0001222 2015.
Yinbo Chen and Xiaolong Wang. Transformers as meta-leamers for implicit neural representations. In European Conference on Computer Vision, pp. 170-187. Springer, 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
10

---

Published as a conference paper at ICLR 2025
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning. pp. 1135-1135. PMLR, 2017.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes, In International conference machine learning, pp. 1704-1715 PMLR, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and 2018b.
Yee Whye Teh. Neural processes. arXiv preprint arXiv:
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909. 10893, 2019.
Jeffrey Gu, Kuan-Chieh Wang, and Serena Yeung. Generalizable neural fields as partially observed neural processes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5330-5339, 2023.
Zongyu Guo, Cuiling Lan, Zhizheng Zhang, Yan Lu, and Zhibo Chen. Versatile neural processes for learning implicit neural representations. arXiv preprint arXiv2230.08888 2023.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworss. arXiv preprint arXiv: 1609.09100 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.044 04.4004440, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 9307-9315, 2024.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Chiheon Kim, Doyup Lee, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Generalizable implicit neural representations via instance pattern composers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11808-11817, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023.
Doyup Lee, Chiheon Kim, Minsu Cho, and WOOK SHIN HAN. Locality-aware generalizable implicit neural representation. Advances in Neural Information Processing Systems, 36, 2024.
Jaeho Lee, Jihoon Tack, Namhoon Lee, and Jinwoo Shin. Meta-learning sparse implicit neural representations. Advances in Neural Information Processing Systems, 34: 14:11769-11780, 2021.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang. Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-pune purpose assistants. Foundations and Trends@ in Computer Graphics and Vision, (6(1-2):1-2) 2024.
11

---

Published as a conference paper at ICLR 2025
Siyuan Li, Di Wu, Fang Wu, Zelin Zang, Stan Li, al. sachitectur-agnnosie masked image modeling from vit back to cnn. arXiv preprint arXiv:2205.1394. 2022a.
Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pp. 112-123, PMLR, 2022b.
Feng Liang, Yangguang Li, and Diana Marculescu. Supmae: Supervised masked autoencoders efficient vision learners. arXiv preprint arXiv:2205. 14540, 2022.
Guibiao Liao, Kaichen Zhou, Zhenyu Bao, Kanglin Liu, and Qing Li. Ov-nerf: Open-vocabulary neural radiance fields with vision and language foundation models for semantic understanding. arXiv preprint arXiv:2402.04648, 2024.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-ro-1 Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9298 9309, 2023.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1660-477,, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02000 2018.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/93, 2023.
Jose Javier Gonzalez Ortiz, John Guttag, and Adrian Dalca. Magnitude invariant parametrizations improve hypernetwork learning. arXiv preprint arXiv:2304.07645. 2023.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5220. IEEE, 2015.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove, Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d
diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp. 28492-28518. PMLR, 2023.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-avd aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 33: 10136-10147, 2020a.
12

---

Published as a conference paper at ICLR 2025
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances neural information processing systems, 33:7462-7473, 2020b.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34: 14:19313-19325, 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp.
1846-2855,, 2021.
Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, and Da-Cheng Juan. Hypergrid transformers: Towards a single model for multiple tasks. In International conference on learning representations, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Johannes Von Oswald, Christian Henning, Benjamin F Grewe, and João Sacramento. Continual learning with hypernetworks. arXiv preprint arXiv:1906. 00695, 2019.
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3835-3844, 2022.
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEFE transactions on image processing, 13(4):600-612, 2004.
Jianglong Ye, Naiyan Wang, and Xiaolong Wang. Featurenerf: Learning generalizable nerfs by distilling foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9962-8973, 2023.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 12104-12113, 2022.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv: 1xXv::8810005449 2018a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018b.
Dominic Zhao, Seijin Kobayashi. João Sacramento, and Johannes von Oswald. Meta-learning via hypernetworks. In 4th Workshop on Meta-Learning at NewrIPS 2020 (MetaLearn 2020). NeurIPS, 2020.
13


Reviews:
[{"review_id": "pyT6I49sLp", "reviewer": "Reviewer_s6Rw", "summary": "The authors propose a novel method of using visual foundation models as part of a hyper-network to prediction the weights of an MLP to perform a task. The authors evaluate their method on a novel viewpoint reconstruction task and an audio reconstruction tasks using three methods: random initialization, fine-tuning, and promp-tuning. They demonstrate a small performance improvement using a foundation model and then fine-tuning or prompt-tuning over random initialization.", "questions": "- What is the exact architecture and number of parameters in the INR network?\n\n- Why did the authors chose generative tasks where the available evaluation metrics such as PSNR, SSIM, LPIPS, and FID are a poor proxy measurement for model performance on the task? Why not use image classification, segmentation, depth prediction, or many other tasks that are easier to evaluate and have more model baselines to compare against?\n\n- How does the performance achieved by the predicted INR weights from this method compare to distilling a foundation model fine-tuned for these tasks into the INR network?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors have selected and interesting question to investigate and have clearly describe their approach (aside from lacking INR model details)\n- This work demonstrates a performance improvement of using foundation model weights over a random baseline.\n- The overall architecture proposed is simple and generalizable to any foundation model or task.", "weaknesses": "- The overall performance improvement is small.\n- The authors chose tasks that are very difficult to evaluate and only two tasks were evaluated.\n- The paper lacks details about the exact architecture and scale of the INR network which seems like an import parameter that would be interesting to vary.\n- The evaluation lacks other baseline methods of training the INR network such as distillation. Although not the goal of this paper, evaluation of other training methods seems important for contextualizing the performance of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fNXQ5N4HNQ", "reviewer": "Reviewer_JGgX", "summary": "The paper proposes to leverage pretrained foundation models for enhancing hypernetworks which are trained to generate weights of implicit neural networks, i.e. networks whose parameters are a representation of a single data sample.  The approach leverages transformer based architectures to learn a (modulated) set of weights vectors that will correspond to the implicit representation of each sample. The method is evaluated on mainly on image data, with an additional experiment on audio data, respectively in a novel image synthesis task and audio reconstruction. The experimental section tests varies aspects and benefits of incorporating foundation models from performance, to sample efficiency, generalization to unseen class and parameter efficiency.", "questions": "- Integration of FM into existing methods :\n    -  How is the architecture and training strategy related to the method proposed? Is the model still trained with meta-learning? \n    -  For the experiments in Table 4, again how is the FM included in the methods specifically? Is there any adaptation to the architecture or training strategy? \n\n- Do the authors have an intuition on why the FID metric seems to grow for foundation models based models in Figure 2?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is simple and effective. Leveraging information stored in pretend foudnation models for enhancing implicit neural representations networks is mostly a novel idea to the best of my knowledge and it may benefits future research in the area. \n\n- The experimental section is quite broad  and comprehensive: it shows the benefits of incorporating foundation models hyper networks to learn implicit representations from the point of view of:  performance, sample complexity, parameter efficiency and multi modality. This seems to suggest that in general pretrained vision models should be incorporated  in learning INRs and pave the way to new applications in this direction. \n\n- In particular, the results by just finetuning the heads is quite impressive, pointing at the fact that the connection between weights and features should be further inspected in future works. \n\n- The paper writing is overall clear.", "weaknesses": "- The architecture and training procedure of the model although it builds on previous work could be explained better in the paper, for See related questions in the question section \n\n- On the experiment on audio data the model doesn't seem to benefit much form the foundation model. Do the authors have an intuition of why? Could it be related with the complexity of the task or to the fact that audio FM are less expressive in general than Vision ones?\n\n- It would be interesting to see some results on a different task/ dataset on image data. \n\n- The following work should be included and discussed in the related works section, as it also explores the relation between features extracted and performance of implicit neural networks: \n\n     - Ye, J., Wang, N., & Wang, X. (2023). Featurenerf: Learning generalizable nerfs by distilling foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8962-8973\n\nSimilarly, yet fare as line of works the following works which distill information in CLIP embeddings to learn better Implicit representations should be discussed: \n\n- Wang, Can, et al. \"Clip-nerf: Text-and-image driven manipulation of neural radiance fields.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022\n\n- Liao, Guibiao, et al. \"Ov-nerf: Open-vocabulary neural radiance fields with vision and language foundation models for 3d semantic understanding.\" arXiv,  2024\n\n\n\n*Minor*\n\n- Visualization quality of Figure 4 could be improved, for example by putting box to zoom on details.\n\n- I spotted the following typos: \n\n    - Figure 3 caption: \"leads generally leads\" -> \"generally leads\"\n    - Figure 4 caption: \"baselin\" -> \"baseline\" \n    - Line 174: \"using of\" -> \"of using\"\n    - Lin e 408: \"descreased\" -> \"decreased\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G6mbHTl5MR", "reviewer": "Reviewer_QLSE", "summary": "There are many applications/tasks where large pre-trained (foundation) models can be potentially helpful, but the possibility has not been explored. The authors identify one of these being generalizable INRs using hypernetworks (hypernets).\n\nThe paper proposes to use transformer-based foundation models within an architecture based on Trans-INR. The authors show that the model can be trained either through full fine-tuning or through a prompt-tuning-based approach that keeps the foundation model encoder weights frozen. In addition to the input patches as input to the transformer, the authors add learnable weight tokens corresponding to the network weights to be generated at the output.\n\nThe authors show that hypernets with foundation model backbones show improved generalization to seen and unseen classes, and improved data and parameter efficiency, compared to hypernets trained from scratch. They note that foundation model choice is important, claiming that models that have learned a good global representation (as opposed to local representations like with MAE) perform better. The authors also show that the scaling benefits of larger foundation models transfer to the generalizable INR task through their incorporation in hypernets. Finally, the authors demonstrate that their idea is robust across two other hypernet algorithms based on Trans-INR and between two modalities (vision and audio).", "questions": "Have you considered deviations from the Trans-INR architecture? Are there any components that become unnecessary or even detrimental with the addition of foundation models (as a motivating example that is not directly related, strong regularization that helps at smaller scales can limit performance gains at larger scales)?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The idea is simple and new to the specific task of generalizable INRs using hypernets.\n- The authors conduct an extensive empirical evaluation to support their claims and show that the gains are reproduced across algorithms and modalities.\n- Most of the paper is easy to follow.", "weaknesses": "Primary:\n- The technical contribution of this paper is replacing a transformer component that has previously been trained from scratch with a pre-trained foundation model, which is not original and is now a fairly standard recipe. Please see the questions section for a possible actionable avenue.\n\nSecondary:\n- It is confusing that in Table 2, random initialization outperforms foundation models on LPIPS in 2/3 settings, but fine-tuning performs much better on the remaining setting. The paper does not provide an explanation.\n- It is not clear what \"positional encoding\" refers to in Figure 1 and the related discussion. In the context of transformers, this term denotes adding position information to the transformer inputs, which is not the case here. It is also hard to see if the linear heads underneath are labeled positional encodings or the $\\gamma(\\mathbf{v})$ block to its right. Most likely, this refers to the MLP's output, but there is some ambiguity.\n- There is no discussion of training hyperparameters used for the experiments. Are all of these the same as what the base frameworks used?\n\nMinor (no impact on score):\n- It is unclear what type of norm is considered around line 140 (context: \"normalizes the weights to have norm 1\"). It is likely L2-norm based on Trans-INR, but it should be specified here.\n- Section 3.1: In the first line, \"we experiments\" -> \"we experiment\".\n- Around line 175: \"Instead using of MSE\" -> \"Instead using MSE\".\n- Last sentence in the abstract is too long (the entire point 2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VY3RSLg1gF", "reviewer": "Reviewer_xDrF", "summary": "This paper investigates the utility of foundation models as backbones for hypernetworks, that is, neural networks which generate the weights for INRs. The authors show that fine-tuning (or freezing and prompt-tuning) foundation models such as DINO and CLIP can indeed lead to hypernetworks that outperform the same architectures trained from scratch.", "questions": "## Questions\n* Is there really enough data in NVS to justify the training of models with 86M parameters from scratch? \n\n## Additional Feedback\n* Figure 1 could match the text a bit better: In the figure, the linear layers labelled as “Positional Encoding” are called BaseParam in the text (I think? Or are these the heads?) and the Embed layer and Enc are not labelled in the figure. It is still understandable, but would remove any uncertainty to label the elements in the figure better.\n* It would be good to motivate and differentiate the four metrics (PSNR, SSIM, LPIPS, FID). \n* Line 141f (“For computational efficiency, each token only helps to generate the weights of a single layer, with the number of tokens r being the total number of parameters in the layer divided by some hyperparameter g.”) is unclear to me, is BaseParam_k a sparse matrix?\n* I think that “Enhancing Hypernetwork architectures with foundation models” would have been a fine title on its own, but that might just be preference (and will not affect my rating of the paper)\n\n## Other thoughts \n(going beyond what would be reasonable to do within the rebuttal period)\n\nIn addition to the three training settings you investigate (starting in line 180), have you considered training LoRA adapters as a middle ground between freezing the FM and fine-tuning it completely? Maybe this could help with the issue of catastrophic forgetting mentioned in line 230.\n\n## Nitpicks\n* Line 323 \"seems to plateau\"\n* Line 407 \"decreased\"\n* Line 416 \"is persists\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper investigates a logical idea, is clearly written and offers a very thorough analysis, in the sense that many questions one might have about the proposed approach are investigated. For example, the authors investigate different foundation models as backbones, compare different fine-tuning methods and hypernetwork algorithms, as well as different tasks. I can imagine that using foundation models as starting points for training INRs will become the standard procedure in certain visual domains.", "weaknesses": "* As somebody unfamiliar with the NVS task, it would have been nice to not only compare to the (potentially weak?) baseline of training from random initialisation, but also to see the performance of the current state-of-the-art method, as a point of reference. I can see that using the FM backbone is better than the baseline, but is it really good in absolute terms? Looking into Tancik et al 2021, it seems like the average PSNR of their best method is 21.333, while that of Chen and Wang 2022 achieves 22.07 on average. I realize that the numbers are not directly comparable, but why did you choose to train one model on all three tasks instead of training individual models?\n* I wonder about the variance of the performance values in Table 1: Ideally, one would train each model multiple times and give an estimate of the standard deviation of performance. I’m not yet convinced that the model differences are really stable. The same holds for table 2.\n* Finally, I would have liked to see an explicit discussion of limitations and shortcomings of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)