Set-based Neural Network Encoding Without Weight Tying
Authors: Bruno Andreis
Bedionita Soro
Philip Torr
Sung Ju Hwang
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Network Encoding
Primary area: deep_learning_architectures

Abstract: We propose a neural network weight encoding method for network property prediction that utilizes set-to-set and set-to-vector functions
to efficiently encode neural network parameters. Our approach is capable of encoding neural networks in a model zoo of mixed architecture and different parameter sizes as opposed to previous approaches that require custom encoding models for different architectures. Furthermore, our \textbf{S}et-based \textbf{N}eural network \textbf{E}ncoder (SNE) takes into consideration the hierarchical computational structure of neural networks. To respect symmetries inherent in network weight space, we utilize Logit Invariance to learn the required minimal invariance properties. Additionally, we introduce a \textit{pad-chunk-encode} pipeline to efficiently encode neural network layers that is adjustable to computational and memory constraints. We also introduce two new tasks for neural network property prediction: cross-dataset and cross-architecture. In cross-dataset property prediction, we evaluate how well property predictors generalize across model zoos trained on different datasets but of the same architecture. In cross-architecture property prediction, we evaluate how well property predictors transfer to model zoos of different architecture not seen during training. We show that SNE outperforms the relevant baselines on standard benchmarks.

Set-based Neural Network Encoding Without Weight Tying
Bruno Andreis', Soro Bedionita', Philip H.S. Torr?, Sung Ju Hwang
KAIST 1, South Korea
University of Oxford, United Kingdom 2 DeepAuto. ai, South Korea
fandries, sorobedio, s [1lwang82]kkaii wtaaat.ac.............o......... ar.
ox ac.uk
Abstract
We propose neural network weight encoding method for network property pre- diction that utilizes set-to-set and set-to-vector functions to efficiently encode neural network parameters. Our approach is capable of encoding neural networks in a model zoo mixed architecture and different parameter sizes as opposed previous approaches that require custom encoding models for different archi- tectures. Furthermore, our Set-based Neural network Encoder (SNE) takes into consideration the hierarchical computational structure of neural networks. To respect symmetries inherent in network weight space, we utilize Logit Invariance to learn the required minimal invariance properties. Additionally, a pud-chunk-encoul pipeline to efficiently encode neural network layers that is adjustable computational and memory constraints. We also introduce new tasks for neural network property prediction: cross-dataset and cross-architecture In cross-dataset property prediction, we evaluate how well property predictors gen- eralize across model zoos trained on different datasets but of the same architecture. In crosttcccite property prediction, evaluate how well property predictors transfer to model zoos of different architecture not seen during training. We show that SNE outperforms the relevant baselines on standard benchmarks.
introduce
Introduction
Recently, deep learning methods have been applied to wide range of fields and problems. With this broad range of applications, huge volumes of datasets are continually being made available in the public domain together with neural networks trained these datasets. Given this abundance of trained neural network models, the following curiosity arises: what can we deduce about these networks with access only to the parameter values? More generally, these networks such as generalization performance on a testset( without access to the test data), the dataset which the model training epochs, choice of model initialization etc. through an analysis of the model parameters? The ability infer such fundamental properties of trained neural networks using only the parameter values has the potential to open up new application and research paradigms [De Luigi et al., 2023, Zhou al., 2023a,b, Navon al., 2023] such learning in the latent space of neural network weights for tasks such as weight generation [Schürholt et al., 2021, Soro et al., 2024], latent space transfer of weights across datasets allowing for transferring weights from one dataset to another as was recently demonstrated in Soro et al. [2024] and latent space optimization using gradient descent where optimization is performed on the weight embeddings [Rusu al., 2018].
predict properties of
trained, the choice of optimizer and learning rate, the number of
We tackle two specific versions of this problem: predicting a) frequencies of Implicit Neural Rep- resentations [Sitzmann et al., 2020] (INRs), and b) the performance of CNNs and Transformers,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

PF
0..9
Figure I: Legend: Padding. Set-to-Set Function, Veetoor Function, PE: Layer-Level & Pe: Layer-Type Encoder. Concept: (left) Given layer weights, SNE begins by padding and chunking the weights into chunksizes. Each chunk goes through a series of set-to-set and eectorr functions to obtain the chunk representation vector. Layer level and rype positional encodings are used to inject structural information of the network at each stage of the chunk encoding process. All chunk encoding vectors are encoded together to obtain the layer encoding. (right) All layer encodings in the neural network encoded to obtain the neural network encoding vector again using as series of sct-to-set and set-to-vector functions. This vector is then used predict the ncural network property of interest.
given access only to the network weights. The first approach to solving this problem, proposed by Unterthiner et al. [2020], involves computing statistics such as the mean, standard deviation and quantiles, of each layer in the network, concatenating them to single vector that represents the neural network encoding, and using this vector to predict the desired network property. Another approach, also proposed baseline in Unterthiner et al. [2020], involves flattening all the parameter values of the network into a single vector which is then fed as input to layers of multilayer perceptrons (MLPs) predict the property of interest. An immediate consequence of this approach is that it is practical only for moderately sized neural network architectures. Additionally, this approach ignores the hierarchical computational structure of neural networks through the weight vectorization process. The second, and most recent approach to this problem, proposed by Zhou et al. [2023a,b], takes a geometric approach the problem by building neural network weight encoding functions, termed neural functionals, that respect symmetric properties of permutation invariance and equivariance of the hidden layers of MLPs under the action of an appropriately applied permutation group. While this approach respect these fundamental properties in the parameter space, it's application is restricted, strictly, MLPs. Also, even when relaxations are made to extend this method to convolutional networks and combinations of convolutional layers and MLPs, these only work under strict conditions of equivalence in the channel size the last convolutional layer and the first linear layer. Hence is clear that while the methods of Zhou et al. [2023a,b] and Navon et al. [2023] enjoy nice theoretical properties through weight rying, their application is limited to only a small subset of carefully chosen architectures.
Moreover, these approaches [Unterthiner et al., 2020, Zhou al., 2023a,b, Navon al., 2023] have a fundamental limitation: their encoding methods are applicable only to a single fixed, pre chosen neural network architecture. Once the performance predictor is trained, in the case of Unterthiner et al. [2020], and the neural network encoder of Zhou et al. [2023a] and Navon et al. [2023] are defined, they cannot be used to predict the performance of neural networks of different architecture These issues are partly addressed by the graph based approaches of Kofinas et al. [2024] and Lim al. [2023]. Consequently, evaluating these models on diverse architectures is impossible without training an new performance predictor for each architecture.
To this end, we propose a Set-based Neural Network Encoder (SNE) for property prediction of neural networks given only the model parameters that is agnostic to the network architecture without weight tying. Specifically, we treat the neural network encoding problem from a set encoding perspective by utilizing compositions of set-to-set and set-to-vector functions. However, the parameters of neural networks are ordered and hierarchical. To retain this order information, we utilize positional encoding [Vaswani et al., 2017] at various stages in our model. Also, our model incorporates the hierarchical computational structure of neural networks in the encoder design by encoding independently, layer-wise, culminating in a final encoding stage that compresses all the layer-wise information into a single encoding vector used to predict the network property of interest. To handle the issue of large and variable parameter sizes efficiently, we incorporate a pud-chunk-encode pipeline that parallelizable and can be used to iteratively encode layer parameters. To learn the correct permutations of MLP weights, we employ the Logit Invariance Regularizer of Moskalev et al. [2023]

---

instead of weight tying. In terms of evaluation, we introduce two new tasks: cross-dataset neural network property prediction and cross-architecttrr neural network property prediction. In cross- dataset neural network performance prediction, we fix the network architecture used to generate the training data and evaluate how well the performance predictors transfer to the same architecture trained on different datasets. For cross-architecture neural network performance prediction, we fix only the architecture for generating the training data and evaluate the performance of the predictor on architectures unseen during training. These tasks are important since in the real setting, access to model zoos is scare, making transferability desirable.
Our contributions are as follows:
We develop a Set-based Neural Network Encoder (SNE, see Figure 1) for network property prediction given access only to parameter values that can encode neural networks of arbitrary architecture, taking into account the hierarchical computational structure of neural networks.
We introduce the cross-dataset property prediction task where we evaluate how well predic- tors transfer across neural networks trained on different datasets.
We introduce the cross-achhitecture property prediction task where we evaluate how well property predictors trained on a specific architecture transfer to unseen architectures.
We benchmark SNE against the relevant baselines [Unterthiner et al., 2020, Navon et al., 2023, Zhou et al., 2023a,b] on the cross-dataset task and show significant improvement over the baselines.
We provide the first set of results on the cross-architecture task, a task for which most of the baselines, with the exception of Zaheer et al. [2017] and Kofinas et al. [2024] under special conditions (see remark in Section 4.2), cannot be used.
2 Related Work
Neural networks that operate on set (un)structured data have recently been used in many applications ranging from point cloud classification to set generation [Kim et al., 202 I]. Set functions required to respect symmetric properties such as permutation invariance and cquivariance. In DeepSets [Zaheer et al., 2017]. a set of oum-dccomposable functions are introduced that are equivariant in the Set-to-Set applications and invariant in the Set-to- Vector applications. In Set Transformers [Lee et al., 2019], a class of attention based Set-to-Set and Set-to- Vector functions are introduced that more expressive and capable of modeling pairwise and higher order interactions between set elements. Recent works such as Bruno et al. [2021] and Willette et al. [2023] deal with the problem of processing sets of large cardinality in the the limited memory/computational budget regime. In this work, we utilize the class of set functions developed in Lee et al. [2019] to develop a neural network encoder for performance prediction that is agnostic to specific architectural choices. Our set- based formulation allows us to build such an encoder, capable of handling neural networks weights of arbitrary parameter sizes. This is deviation from recent approaches to neural network encoding for property prediction that can encode only parameters of a single fixed architecture.
Set Functions:
Predicting the properties of neural networks given access only to the trained parameters is a relatively new topic of research introduced by Unterthiner al. [2020]. In Unterthiner al. [2020], two methods are proposed for predicting the generalization performance of neural networks: the first involves flattening the weights of the network into a single vector, processing it using multiple layers of MLPs obtain an encoding vector which is then used to predict the performance. The second involves computing the statistics of each layer in the network, such as mean, variance, quantiles etc., concatenating them into single vector that is then used for predicting the performance of the network. The most recent approach that we are aware of, Navon al. [2023] and Zhou et al. [2023a,b], proposes a neural network weight encoder that is invariant or equivariant, depending on the application, to an appropriately applied permutation group to the hidden layers of MLPs. Two variants of their model provided: one which operates only on the hidden layers, and conforms strictly to the theory of permuting MLP hidden neurons [Hecht- Nielsen, 1990], and a relaxation that assumes that the neurons of both the input and output layers of MLPs are permutable. Additionally, extensions are provided for convolutional layers. Kofinas et al. [2024] and Lim et al. [2023] represent weights using graph neural networks. Our approach, SNE, is directly comparable to these methods on the neural network property prediction task. However,
Neural Network Property Prediction From Weights:
3

---

unlike the methods of Unterthiner et al. [2020], Navon et al. [2023] and Zhou et al. [2023a] which operate only on neural networks of fixed architecture, and consequently fixed number of encodable parameters, SNE is capable of encoding networks of arbitrary architecture. Moreover, SNE utilizes the hierarchical computation structure of neural networks by encoding, iteratively or in parallel, from the input to the output layers. Additionally, we go further than the experimental evaluation in Unterthiner et al. [2020]. Navon et al. [2023] and Zhou et al. [2023a.b] by introducing two new tasks: cross-dataset and cross-architecture neural network property prediction. Unterthiner et al. [2020], Navon et al. [2023] and Zhou et al. [2023a,b] can only be benchmarked on the cross-dataset task where all networks in the model zoos are of the same architecture. Their restriction to a single fixed architecture makes cross- arosstcrciee evaluation impossible. Our method SNE, and those Kofinas et al. [2024], Lim et al. [2023], on the other hand can be used for both tasks.
3 Set-hased Neural Network Encoding Without Weight Tying
3.1 Preliminaries
We have access to dataset D = (21 y1)..... (Zu: Vn) where for each (24, (1) pair, F, represents the weights of a neural network architecture a, sampled from a set of architectures A and y: corresponds to some property of x; after it has been trained a specific dataset d. ys can be properties such as generalization gap, training loss, the learning rate used to train Ii, or even the number of epochs, choice of weight initialization, and optimizer used to train 2i.- Henceforth,
refer to D as a model z0o, For each H; E D, x = i = where w represents the weights (parameters) of the jth layer of the neural network T2, and |a;| is the total number of layers in rq. Consequently, 1 and w!!-.l are the input and output layers of x; respectively. Additionally, we introduce the Flat: : c; + Rd. operation, that takes as input the weights of neural network and returns the flattened weights and d, is the total number of parameter is 24.
The neural network encoding problem is defined such that, we seek compress I; Rd a compact representation 2x, E R² such that &x, can be used to predict the properties y; of x, with & d;. In what follows, we present the details of our Set-based Neural Network Encoding (SNE) method capable of encoding the weights of neural networks of arbitrary architecture that takes into account the hierarchical computational structure of the given architecture and with efficient methods for processing weights of high dimension.
3.2 Handling Large Layer Weights via Chunking
For a given layer / E I;, the dimension of w [w/]] can be very large. For instance, when considering linear layers, flattening the weights can results in tensor that can require large compute memory to be processable by another neural network. To resolve this issue, we resort to chunking. Specifically, for all layers /i E Ig, we perform the following operations:
ii CCunn((aa(F)aat()))))))
(w. "u."""..,,"""" ,."...,""",,
where for any w E wi, |w R Here, is the chunksize, fixed for all layer types in the neural network and t E [o.....g] q]. The padding operation Pad(w: c) appends zeros, if required, to extend wi and make its dimension a multiple of the chunksize c. To distinguish padded values from actual weight values, each element of أ² has corresponding set of masks ni! {m=ç ..... m }. Note that with this padding and subsequent chunking operation, each element in úi is now small enough, for an appropriately chosen chunksize to be processed. Moreover, all the elements in wi can be processed in parallel. Importantly, chunksizes are chosen ensure that weights from the same neurons are grouped together. This allows for principled learning of specific symmetries which we discuss later Section 3.6. An ablation on the effect of chunksize is provided in Appendix E.
The model zoos we consider in the experimental section are populated by neural networks with stacks of convolutional and linear layers. For each such layer, apply the padding and chunking operation differently. For a linear layer w! E R outxin where out and in are the input and output dimensions respectively, we apply the flattening operation on both dimensions followed by padding and chunking. However for convolutional layer w! R outx inxkxk we do not apply the flattening, padding, and

---

chunking operations to the kernel dimensions k and operate only on the input and output dimensions since the kernels are small enough to be encoded together. Finally we note that for layers with bias values, we apply the procedure detailed above independently to both the weights and biases.
3.3 Independent Chunk Encoding
The next stage in our Set-based Neural Network encoding pipeline is the individual encoding of each chunk of weight in a = {w ..... w J. For each w E u we treat the elements as members of a set. However, it is clear that w has order in its sequence, i.e., an ordered set. We remedy this by providing this order information via positional encoding. Concretely, for given w Rcx1, we first model the pairwise relations between all elements using a sei-to-set function D8, obtain:
to (w ) E Rexh
That is, Øg, captures pair-wise correlations in wa! and projects all elements (weight values) to a new dimension h.
Given un E Rexh, we inject two kinds of positionally encoded information. The first encodes the layer type in list of layers, ie., linear or convolution for the model zoos we experiment with, to obtain:
PosEnc
Type Loyer (wx²") R
Here we abuse notation and assign the output of PosEnc(.) to uhtt to convey the fact that ait'' are modified in place and to simplify the notation. Also, all PosEnc(-)s are variants of the positional encoding method introduced in Vaswani et al. [2017]. Next we inject the layer level information. Since neural networks are computationally hierarchical, starting from the input to the output layer, we include this information to distinguish chunks, wf's s from different layers. Specifically, we compute:
n PosEne Level Layer (w E R.xxA
where the input to PosEnc Lager (.) is the output of Equation 3. We note that this approach is different from previous neural network encoding methods [Unterthiner et al., 2020] that loose the layer/type information by directly encoding the entire flattened weights hence disregarding the hierarchical computational structure of neural networks. Experimentally, we find that injecting such positionally encoded information improves the models performance (Ablation E).
We further model pairwise correlations in uي now infused with layer/type information, using another set-to-set function &ø.:
a to (w²") E Rexh
(5)
The final step in the chunk encoding pipeline involves compressing all elements in ai compact representation. For this, encoding layer computes the following function:
set-to-vector function I'e RRexh R In summary, the chunk
Tupe f. = ree ( Eeeyen Tayer (PosEnc Layer (Bo, (w)),
(6)
where wi! E R RIXH Note that for each chunked layer wf = { (w ..... w.' }. the chunk encoder, da Equation 6, produces new set w/ Concatenate (( ..... पū [] Raxh, which represents all the encodings all chunks layer.
Remark Our usage of set functions e Do, and Ie. allows us to process layers of arbitrary sizes, which allows us to process neural networks of arbitrary architecture using a single model, a property lacking in previous approaches to neural network encoding [Zhou et al., 2023a,b, Unterthiner et al., 2020, Navon et al., 2023].
3.4 Layer Encoding
At this point, we have encoded all the chunked parameters of given layer obtain we Encoding layer, w² then involves defining a function T'pp" Roxh R xx for arbitrary g. In practice, this is done by computing:
w! Loo PPosEnc
Lovel
Level Layer (Des (wF) R 1xb
5

---

Again we have injected the layer level information, via positional encoding, into the encoding processed by the set-to-set function Po.. We then collect all the layer level encodings of the neural network zy:
i: Concatenate (
W Imch
R R
(8)
3.5 Neural Network Encoding
With all layers in Ty encoded, we compute the neural network encoding vector 22, as follows:
Pan I', 19, (PosEncfayet Level (w:))] R
2r, compresses all the layer-wise information into compact representation for the downstream task. Since Te., is agmostic to the number of layers |z;] of network z;, the encoding mechanism can handle networks of arbitrary layers and by extension architecture. Similar the layer encoding pipeline, we again re-inject the layer-level information through positional encoding before compressing with Te,
Henceforth, we refer to the entire encoding pipeline detailed so far as (2;e for a network z,, where è encapsulates the encoder parameters, 2o.-4: a'a I'2 and Ty
3.6 On Minimal Equivariance Without Weight Tying
Given an MLP, there exists permutations of the weights such that the networks are functionally equivalent [Hecht-Nielsen, 1990]. Since not all permutations are functionally correct, the encoder needs to learn the correct functional equivalence. To achieve this, we utilize the concept of Logit Invariance Regularization [Moskalev et al., 2023] where we constrain the output of the non-equivariant SNE, (due to the positional encoding of input and output vectors) to respect the restricted functionally correct permutation group. This results in the following optimization problem:
minimize (D( + (D,
(10)
where G' is the group of functionally equivariant permutations in the weight space, is the training dataset and v balances the task loss (D) and the Logit Invariance Regularization term f(D,C) Proposition 3.1 of Moskalev et al. [2023] guarantees that the resulting SNE model will have low sensitivity to functionally incorrect permutations of the weights. In practice, E (D, G) is the Le distance between functionally equivalent permutations the same weight. This approach differs from previous works [Zhou et al., 2023a,b, Navon et al., 2023] which instead result to weight tying to achieve minimal equivariance.
Remark While we use a regularization approach to achieve the required approximate minimal cquivariance in weight-space, our usage of the Logit Invariance Regularizer [Moskalev et al., 2023] theoretically guarantees that we indeed learn the correct invariance property similar to the weight- typing approaches. Additionally, our formulation is what allows us to deal with arbitrary architectures using single model (see Section 4), as opposed to previous works, since strict enforcement of weight-space equivariance by design requires crafting a new model for different architectures. In this sense, our approach provides general encoder which in principle is applicable to any architecture, resolving the limitations of purely weight-tying approaches.
Theoretical discussions on the adopted regularization based approach to minimal equivariance versus weight tying provided in Appendix C.
3.7 Choice of Set-to-Set and Set-to-Vector Functions
We specify the choice of Set-to-Set and Set-to- functions encapsulated by 2o Fas I's and
T, used to implement SNE. Let X E Rx xd and Y E R xd be arbitrary sets where n = |X] ny = |Y] and d (note the abuse of notation from Section 3.1 where d is dataset) is the dimension of an element in both X and Y. The MultiHead Attention Block (MAB) with parameter w is given by:
MAB (X, Y:w) = LayerNorm(HH H + rFF( where
(11)
H LayenNorm(X* (X + MultiHead Y, YYY;)).
(12)
Here, LayerNorm and rFF are Layer Normalization [Ba al., 2016] and row-wise feedforward layers respectively. MultiHead(XX. Y, YY w)) is the multihead attention layer of Vaswani al. [2017].

---

The Set Attention Block [Lee et al., 2019], SAB, is given by:
SAB(X) == MAB(X,
(13)
That is, SAB computes attention between set elements and models pairwise interactions and hence is a Set-to-Se Set function. Finally, the Pooling MultiHead Attention Block [Lee et al., 2019]. PMA is given by:
(M)
(FAB(S)FF(X)))
where
(14)
e Rexd and X E R"xxd, The & elements S are termed seed vectors and when k as is in all our experiments, PMA; pools set of size nx to single vector making it a Set-to- function.
All parameters encapsulated by Øø are implemented as a stack of two SAB modules: SAB(SAB(X)), Stacking SAB modules enables us not only to model pairwise interactions but also higher order interactions between set elements. Finally, all of T,, L' and r are implemented as single PMA module with k =.
3.8 Downstream Task
Given (.), we train predictor fe(2z, estimate properties of interest of the network z;. In this work, we focus solely on the task of predicting the generalization performance of r;, where yı is the performance on the test set of the dataset used to train æ; for CNNs and frequencies for INRs. The parameters of the predictor fo and all the parameters in the neural network encoding pipeline, O are jointly optimized. In particular, we minimize the error between fo (Zz,) and y:- For a model zoo, the objective is given as:
minimize 0.0
Σ
(15)
for appropriately chosen loss function E(-). In experiments, ((-) is the binary cross entropy or mean squared error loss. The entire SNE pipeline is shown in Figure 1.
4 Experiments
We present experimental results INRs, and the standard CNN benchmark model zoos used in Un- terthiner et al. [2020] et al. [2023a] et al. [2023b], and Navon et al. [2023]. Experimental settings, hyperparameters. model specification. ablation of SNE and discussions on applying SNE to architectures with branches (e.g. ResNets) in Appendix D.
Baselines: We compare SNE with the following basclines: a) MLP: This model flattens the entire weight of the network and encodes it using a stack of MLP layers. b) DeepSets: [Zaheer et al., 2017] This model treats the weights model learns a generative model of the flattened weight vector. d) STATNN [Unterthiner et al., 2020]: This model computes the statistics of each layer such as the mean, variance and quantiles, and concatenates them to obtain the neural network encoding vector. e) DWSNet [Navon et al., 2023]: minimally equivariant model using weight tying developed mainly for MLPs. Various modules are provided for encoding biases, weights and combinations of these two. f) NFN NEN and NFT [Zhou et al., 2023a,b]: These models, termed Neural Functionals(NFF, are developed mainly for MLPs and utilize weight tying to achieve minimal equivariance. HNP, hidden neural permutation, applied only to the hidden layers of each network since the output and input layers MLPs are not invariant/equivariant to the action of permutation group on the neurons. NP, neural permutation, makes a strong assumption that both the input and output layers are also invariant/equivarian under the action of permutation group. NET is similar to both models and utilizes attention layers. g) NeuralGraph: [Kofinas et al., 2024] This method represents the weights of a network as a graph and uses graph pooling techniques to obtain the network representation vector. We note that we are unable to benchmark against Graph Metanetworks [Lim et al., 2023], which uses graph approach similar to NeuralGraph, since no code data is publicly available.
set with ordering. e) HyperRep: [Schürholt et al., 2021] This
7

---

In all Tables, the best methods are shown in red and the second in blue. Additionally an extensive ablation of all the components of SNE is provided in Appendix E. All experiments are performed with single GeForce GTX 1080 TI GPU with 11GB of memory.
Table I: Predicting Frequencies of Im- plicit Neural Representations (INRs).
4.1 Encoding Implicit Neural Representations
Dataset and Network Architecture: We utilize the model zoo of Navon et al. [2023] consisting of INRs [Sitzmann et al., 2020] fit to sine waves on [-x, with frequencies sampled from (0.5, 10). INRs are neural parameterizations of signals such images using multi-layer perceptrons.
Model
#Params
MSR
MLP
14K
1.917:0221
Decpsets
99K
2.674
STATNN
44K
0,937.10.76
NEÑ
2.0M
0
NFN
2.8M
0.998+0.382
Task: The goal is to predict the frequency of a given INR. Each
NET DWSNet
6M
0.401
INR is encoded to a 32 dimensional vector which is then fed to a classifier with two linear layers of dimension 512.
1.5M
0.20910.028
SNE(Ours)
358K
0.09810.002
Results: As can be seen in Table 1, SNE significantly outperforms the baselines on this task. Given that INRs are MLPss, minimal equivariance is particularly important for this task and shows that SNE learns the correct minimal equivariance required to solve the task using the logit invariance approach. Additionally, compared to the minimal equivariance constrained models [Zhou et al., 2023a,b, Navon et al., 2023], SNE is parameter efficient as show in Table 1. We note that increasing the parameter counts of the MLP, DeepSets and STATNN baselines results in overfitting and poor performance.
4.2 Cross-Architecture Performance Prediction
For this task, we train the encoder on 3 ho- mogeneous model zoos of the same architec- ture and test on 3 homogeneous model zoos of
Table 2: Cross-Architecture Performance Prediction.
Arch: - Archa
DoopSets
NeuralGraph
SNE(Ours)
a different architecture unseen during training.
MNIST MNIST
0.460=00.0
0.473 20.00
0.490=1.007
MNIST CIFARIO
0.508=0000
0.528 stu06
0.586=2000
The cross- arosttrccite task demonstrates the en- coder's 's agnosticism to particular architectural choices since training and testing are done on model zoos of different architectures, iee. we perform nut-of-distribution evaluation.
MNIST-> SVHN
0.546=00000
0.502s0:10
0.535
IIFARIO--CIAAT
00007-0000
0.46
0.660-20016
0.459
0.352
0.558 aoro7
CIFARIO-SSVHN
0.534
0.581 =h00s
SVHN-ISVHY
0.553
0.573
0.609 5..0m
-INIST
0.480..mm
0.448
20004
0.531 20000
Datasets and Neural Network Architectures:
SVHN-ACTFAR
0.529
0
We utilize model zoos trained on MNIST, CIFARIO and SVHN datasets. We generate a model zoo
for these dataset with an architecture consisting of 3 convolutional layers followed by two linear layers and term the resulting model zoo Arch Exact architectural specifications are detailed in Appendix G. We generate the model zoos of Archy following the routine described in Appendix A.2 of Unterthiner et al. [2020]. We refer to the model zoos of Unterthiner et al. [2020] as Arche. All model zoos of Arch; are used for training and those of Archy are used for testing and are not seen during training.
Task: Here, we seek explore the following question: Do neural network performance predictors trained on model zoos of Arch, transfer or generalize to the out-o--distributtin model zoos of Arch,? Additionally, we perform cross-dataset evaluation on this task where cross evaluation is with respect to model zoos of Arch,, i.e,, we also check for out-of-distribution transfer across datasets.
Baselines: We compare SNE with DeepSets [Zaheer et al., 2017] and NeuralGraph [Kofinas et al., 2024] for this task. None of the other baselines, MLP, STATNN [Unterthiner et al., 2020], NEN [Zhou et al., 2023a], NFNHNe [Zhou et al., 2023a], NFT [Zhou et al., 2023b] and DWS- Net [Navon et al., 2023] can be used for this task since they impose architectural homogeneity and hence cannot be used for out-of-distribution architectures by design.
We report the quantitative evaluation on the cross-architecture task in Table and report Kendall's 7 [Kendall, 1938]. The first column, Arch, Arch, shows the direction of transfer, where we train using model zoos of Arch, and test on model zoos of Arch2- Additionally, A-B, e.g. MNIST->CIFARII 10 shows the cross-dataset transfer. From Table 2, it can be seen that SNE transfers best across
Results:
architectures and datasets outperforming the DeepSets and NeuralGraph baselines significantly. Interestingly, the DeepSets model, which treats the entire weight
8

---

as set with no ordering performs better than the Neu- ralGraph model on average for this task. In conclu- sion, SNE shows strong transfer across architecture and datasets in the out-of-diitribution benchmark.
Table 3: Pross-Archieecture Performance Predic- tion on Schürholt al. [2022]'s model zoo.
Dataset
HyperRep
SNE(Ours)
SVHN + SVHN
0.45
0.67
Remark: In Table 2, we performed Arch; - Arch2 evaluation specifically to allow benchmarking against NeuralGraph [Kofinas et al., 2024]. Specifically, the convolutional filters of Archı are larger than those of Archg. NeuralGraph requires specifying this maximum filter size before training and can only be used to transfer acToss architectures with filter sizes equal or smaller than the predefined filter size (this can be verified from the official source code') ), i.e., NeuralGraph is not truely agnostic to architectural choices. SNE on the other hand does not have this limitation. To demonstrate this, we use the SNE model trained in Table and test on the SVHN model zoo of Schürholt et al. [2022] which is much larger architecture with larger filter sizes than those of Arch1. We compare with HyperRep [Schürholt et al., 2021] which is trained fully on the testing model zoo. We emphasize that SNE is not trained on the training set of this model zoo. Results for HyperRep are taken from Schürholt et al. [2022] and a single SNE model is evaluated to match the setting of Schürholt et al. [2022]. From Table 3, SNE significantly outperforms HyperRep by very large margins without being trained on the training set of Schürholt et al. [2022] as HyperRep and demonstrates true agnosticism to architectural choices compared to NeuralGraph. We report Kendall's ' r [Kendall, 1938] in Table 3.
SVHN MNIST
0.15
0.61
SVHN CIFARIO
0.10
0.68
Evaluation on Transformers: We generate model 700 of transformer using Pytorch [2024] and test the transfer from Archı to the transformer model 700. For this task, we are unable to benchmark against NeuralGraph as was done in Table 2, since the model cannot process transformer weights when trained on Arch Hence we benchmark against the DeepSets baseline as in Table 2. From Table 4 SNE generalizes better to the unseen transformer architecture at test time than the baselines showing strong architectural transfer. Additionally, here, the model encodes an architecture with about 5 times the number of paramcters in Arch, demonstrating the scalability of our approach. The DeepSets baseline fails to generalize on this task.
Table 4: Cross-Architecturg Performance on Trans- formers. We report Kendall's T.
Archi - Transformer
DeepSets
SNE(Ours)
MNIST -> MNIST
0.1975 + 0.000
0.4625 + 0:06
CIFARIO MNIST
0.1970 + 0.000
0.3278 +
SVHN - MNIST
0.1906 + 0.000
0.3735 + 0.00
4.3 Cross-Datasee Performance Prediction
Table 5: Cross-Datased Prediction. We report Kendall's T.
For this task, we train neural network performance predictors 4 homo- geneous model zoos, of the same ar- chitecture, with each model zoo re- stricted to a single dataset.
Dataset
MLP
STATNN
NFNar
NFN
SNE(Ours)
MNIST
0.618=o
0.788
0.780
0.775an
=.807
FashionMNIST
0.613=a17
0.696
0.727 0.768
0.765
CIFARIO
0.576
0.743:0117
0.
0.680
0.743
SVHN
0.604=0
0.709
0.638.0.
Datasets and Network Architecture: Each model zoo is trained on one of the following datasets: MNIST [Deng, 2012], FashionMNIST [Xiao et al., 2017], CIFARIO [Krizhevsky, 2009] and
SVHN [Netzer et al., 2018]. We use the model zoos of Unterthiner et al. [2020].
thorough description of the model zoo generation process can be found in Appendix A.2 of Un- terthiner et al. [2020].
We train network encoders on single model zoo, e-g. MNIST, and evaluate it the test set of all four datasets and report the averaged performance. This results in in-distribution testing with respect to the dataset used for training and out-of-distrution testing with respect to the other three datasets, e.g. a model trained on MNIST is tested on MNIST, FashionMNIST, CIFARIO and SVHN,
Task:
Baselines: We benchmark against MLP, STATNN [Unterthiner et al., 2020J, NENHp [Zhou et al., 2023a], and the NEN [Zhou et al., 2023a].
Results: We present the results for this task in Table 5, Here we see that SNE again performs better than the competing methods significantly demonstrating strong transfer across different datasets for the same architecture
9

---

(e) SNE(Ourr)
(a) MLP
(b) STATNN
(c) NENsP
(d) NENHNP
Figure 2: TSNE Visualization of Neural Network Encodings. We train neural network performance prediction methods combination of the MNIST, FashionMNIST, CIFARIO and SVHN modelzoos of Unterthiner al. [2020]. We present views of resulting 3-D plots showing how neural networks from each modelzoo are embedded/cnccoeed by the corresponding models. Larger versions these figures are provided in Appendix Zoom in for better viewing.
To understand how SNE transfers well across model zoos, we generate TSNE [Van der Maaten and Hinton, 2008] plots for the neural network encodings of all benchmarked methods on all four homogeneous model zoos in Figure 2. We provide different views of each models embeddings to better illustrate the encoding pattern. In Figures 2c and 2d, we observe that NFNNP and NENDNP have very clear separation boundaries between the networks from each model zoo. In Figures 2a and 2b, MLP and STATNN, respectively show similar patterns with small continuous strings model zoo specific groupings. However, these separations are not as defined as those of NFNNP and NFNHNP. The embedding pattern of SNE the other hand is completely different. In Figure 2e, all networks from the model zoos are embedded almost uniformly close to each other. This may suggest why SNE performs much better on the cross-dataset performance prediction task since it is much easier to interpolate between the neural network encodings generated by SNE across model zoos.
Qualitative Analysis:
5 Conclusion
In this work, we tackled the problem of encoding neural networks for property prediction given access only trained parameter values. We presented Set-based Neural Network Encoder (SNE) that reformulates the neural network encoding problem as set encoding problem. Using sequence of set-to-set and set-to-vector functions, SNE utilizes a wad-chunk-encole pipeline encode each network layer independently; sequence of operations that is parallelizable across chunked layer parameter values. SNE also utilizes the computational structure of neural networks by injecting positionally encoder layer type/level information in the encoding pipeline. As result, SNE is capable of encoding neural networks of different architectures opposed previous methods that only work on fixed architecture. To learn the correct minimal equivariance for MLP weight permutations, we utilized Logit Invariance Regularization as opposed weight tying used previous methods. Experimentally, we introduced the cross-dataset and cross-architecture neural network property prediction tasks. We demonstrated SNE's ability to transfer well across model zoos of the same architecture but with networks trained on different datasets the cross-dataset task. On the mooss-architecture task, we demonstrated SNE's s agnosticism architectural choices and provided the first set of experimental results for this task that demonstrates transferability across architectures.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by Institute for Information communications Technology PIoMotion(IITP) grant funded by the Korea governmen((MSIP)
Artificial Intelligence Graduate School Program(KAIs)) (AIST)) by Samsung Research Funding Center of Samsung Electronics (No. 10201210-08006-011 Institute of Information & communications Technology Planning & Evaluation (IITP) under Open RAN Education and Training Program WITT--2024RRS- RR-2024-004220883 grant funded by the Korea goVernment(MSTT), by Google Research Grant and by the UKRI grant: Turing AI Fellowship EP/W002981/1. We would also like to thank the Royal Academy of Engineering and
FiveAI.

---

References
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Andreis Jeffrry for scalable set encoding. Advances in Neural Information Processing Systems, 34:21365-21374, 2021.
Jeffrey Willette, Juho Lee, and Sung Ju Hwang. Mini-batch consistent slot set encoder
Edo Cohen-Karlik, Avichai Ben David, and Amir Globerson. Regularizing towards permutation invariance in recurrent models. Advances in Neural Information Processing Systems, 33:18364- 18374, 2020.
Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning implicit neural representations of shapes. arXiv preprint arXiv:2302.05438, 2023.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-442 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pages 129-135. Elsevier, 1990.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
Hyunsu Kim, Hyungi Lee, Hongseok Yang, and Juho Lee. Regularizing towards soft equivariance under mixed symmetries. arXiv preprint arXiv:2306.00355. 2023.
Jinwoo Kim, Jaehoon Yoo, Juho Lee, and Seunghoon Hong- Setvae: Learning hierarchical composi- tion for generative modeling of set-structured data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15059-15068, 2021.
Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan Burghouts, Efstratios Gavves, Cees GM Snoek, and David W Zhang. Graph neural networks for learning equivariant representa- tions of neural networks. arXiv preprint 24ti::2403.224433 2024.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: framework for attention based cermutation-invarie neural networks. In International conference on machine learning, pages 3744 3753.. PMLR, 2019.
Derek Lim, Haggai Maron, Marc T Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for processing diverse neural architectures. arXiv preprint arXiv:2312.04500 2023.
Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. Advances in Neural Information Processing Systems, 35:768-781 2022.
Artem Moskalev, Anna Sepliarskaia, Erik J Bekkers, and Arnold WM Smeulders. On genuine invariance learning without weight-tying. In Topological, Algebraic and Geometric Learning Workshops 2023, pages 218-227. PMLR, 2023.
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equiv- ariant architectures for learning in deep weight spaces. arXiv preprint arXiv:2301. 127.0 2023.
Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and A Ng. The street view house numbers (svhn) dataset. Technical report, Technical report, Accessed [016-08-01.(Onlmmee] 2018.
Samuel E Otto, Nicholas Zolman, J Nathan Kutz, and Steven I. Brunton. A unified framework to enforce, discover, and promote symmetry in machine learning. arXiv preprint 1rXiv:2311.0022 2023.
12

---

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. 7achary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, segh-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
Pytorch VIT.
hiiprttcee-tsst----rromnmeeeer- 10 2024. URL https://gitthb. com/ 8- -isirrhhh-caattttttiiioooo-o-o
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprini 1eXivv/807.05000 2018.
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems, $4:16481-16499 2021.
Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Model 200s: A dataset of diverse populations of neural network models. Advances in Neural Information Processing Systems, 35:381 34-38148, 2022.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
Bedionita Soro, Bruno Andreis, Hayeon Lee, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion-based neural network weights generation. arXiv preprint 18X1522402.88153, 2024.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002.1 / /448, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Jeffrey Willette, Seanie Lee, Bruno Andreis, Kenji Kawaguchi, Juho Lee, and Sung Ju Hwang. Scalable set encoding with universal mini-batch consistency and unbiased full set gradient approxi- mation, 2023.
Han Xiao, Kashif Rasul, and Roland Vollgraf. mashion-mnist: novel image dataset for benchmarking 2017.
machine learning algorithms. arXiv preprint
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Allan Zhou, Kaien Yang, Kaylee Burns, Yiding Jiang, Samuel Sokota, Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. arXiv preprint arXiv:2302. 14040, 2023a.
Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. arXiv preprint 135.5 2023b.
13

---

A Organization
In Section B we provide some limitations and future directions for the neural network encoding task. In Section C, we discuss weight tying versus regularization approaches to achieving minimal equivariance for MLPs together with some theoretical considerations. how SNE can be applied to architectures with branches, specifically ResNets [He et al., 2016]. In Section E. we provide an ablation on the various components of the proposed method. In Section F, provide additional experimental results on the crossedataset and cross-arces arsssacccitecttre tasks. In Section G, we specify all the model architectures used for generating the model zoos of Arch, and Archa used for the cross-dataset and cross-architecture tasks. In Section H provide details of the train/validation/ten splits. In Section I, we detail all the hyperparameters used for all experiments. We provide implementation details for SNE in Section J and enlarged versions of Figure in Section K.
Section D, we discuss
B Limitations Future Work
In this work, we focused solely on the task of predicting the properties, specifically generalization performance, of neural networks given access only to the model parameters. While the task of encoding neural network weights is relatively new topic of research with very few baselines, we anticipate new applications/research directions where the neural network encoding vector is used for tasks such as neural network generation or neural network retrieval. These tasks, potential applications, and consideration of more complicated neural network architectures (see discussion in Section D) are out of the scope this paper and we leave for future works.
Additionally, we have not included results on larger architectures such as Transformers for the following reasons: Firstly, we are unaware of any such extensive model zoos for such architectures. In Lim et al. [2023], such model zoo is evaluated, however, this is not publicly available hence we are unable to benchmark against such architectures. Lastly, given that all experiments in this paper were carried out on single GPU with 1 J1GB of memory, we are unable to generate the extensive Transformer model zoos used in Lim et al. [2023]. However in Section D, we discuss how these architectures can be encoded with our method when such model zoos become available.
C Weight Tying vs Regularization and Theoretical Considerations
With regards achieving minimal equivariance: respecting only subset of permutations in the weight space that are functionally equivalent, there are two approaches to satisfying this property.
Weight Tying: This approach is adopted by Zhou et al. [2023a,b], Navon et al. [2023] where minimal equivariance baked into the neural network encoder, i.e., weight tying. However, this has couple of drawbacks:
It requires a specification of the neural network to be encoded beforehand. This results in encoders that cannot be applied to different architectures where at test time, we evaluate SNE on architectures not seen during training.
demonstrated in Section 4.2
It requires processing all layers of the network together as opposed to the layer-wise encoding scheme that we adopt. Note that for large networks, Zhou et al. [2023a,b], Navon al. [2023] will require correspondingly large memory but the memory requirements of SNE remains constant since we share the same encoder across all layers, allowing our model to process arbitrary network architectures efficiently.
Regularization: Recent works [Moskalev et al., 2023, Cohen- Karlik et al., 2020, Kim et al., 2023, Miyato et al., 2022, Otto et al., 2023] have shown that symmetric constraints such permutation invariance/cquiivrianee of a non-symmetric model can be achieved through regularization during training. In this work, we take this approach since offers the following useful properties:
The level of symmetric constraints, i.e., minimal equivariance, can be learned directly from data without architectural constraints.
It allows us to process layers independently, resulting in a model that general and disen- tangled from architectural choices demonstrated in the cross architecture experiments.
14

---

In the Ablation study E, we show that the regularization approach indeed helps to learn model that generalizes well and without the model struggles to learn the correct minimal equivariance.
Theoretical Considerations A natural question that arises is how well the regularization approach learns the correct symmetric properties of interest. We first define the Logit Invariance, from which we obtain the Logit Invariance Regularization loss of Moskalev et al. [2023].
Definition C.1 (Logit Invariance) [Moskalev et al., 2023] For given group G, dataset D and non-symmerric(with respect to G) model f. the Logit Invariance Loss of f is given by:
Σ 2 (1f(x) - 2~D G~D
LDDG))
(16)
Logit Invariance measures the sensitivity of the action of G on data samples in D.
The following proposition Moskalev et al. [2023] characterizes the performance decay of the Logit Invariance Regularizer under some linearity assumptions.
Proposition C.1 (Invariance-Indueed by Spectra Decay) [Moskalev et al., 2023] Logit invari- ance error minimization implies man (W (t) S (O)) when -> 00.
proof: See Section 3.3 of Moskalev et al. [2023].
Here, W are model parameters, optimization steps and maa (W) = ||W||2- Proposition C.I, through analysis of the training dynamics of asserts that f will be insensitive actions of G on data instances thereby learning the correct invariance property when the Logit Invariance Regularizer minimized.
In summary, the SNE model constrained by the Logit Invariance Regularizer, learns the correct minimal equivariance required for processing MLP weights without having to resort to weight tying as the methods of Navon al. [2023] and Zhou et al. [2023a,b].
D Considering Architectures with Branches
Here, outline how SNE may be applied model zoo of architectures such ResNets where branches exist the computational graph. Given that residual blocks are composed of convolutional and linear layers, each of these can be encoded independently residual connections, we propose to introduce special tokens (just as was done for layer types) when encode the entire block. Additionally, any symmetries inherent in such blocks can be enforced using the invariance regularization term introduced. Given that Transformers are composed of linear layers, the same pipeline be applied encode transformers and logit invariance regularization can be used to respect their inherent symmetries.
already do. To account for
While we do not see any impediment applying SNE to such architectures, the unavailability of such model zoos makes it impossible to experimentally verify these and hence we leave as future work when such model zoos become publicly available.
E
Ablation
Table 6: Ablation on SNE Components
Model
MSE
W/O Layer Level Encoder
2.306+11303
W/O Layer Type Encoder
0.128:00.07
W/O Set Functions
1.9
W/O Positional & Hierarchical Encoding
7.452+0.799
W/O Invariance Regularization
0.156+0.011
SNE
0.098=0.002
15

---

1.0
NFT
DWSNet
0.8
SNE(ours)
0.6
Error
0.4
0.2
0.0
20
40
60
80
100
Percentage of Training Data
Figure 3: Ablation: We compare the performance of models in limited training data setting using the experiments of Table As shown, SNE is more data efficient than the baseline models when the amount of training data is constrained.
E.1 Components of SNE
We investigate the impact of the invariant regularization loss, the hierarchical and positional encoding modules, the usage of set functions, layer level and layer type encoders on the performance SNE using the INR dataset. In Table 6, it can be seen that the hierarchical and positional encoding modules play an important role the performance of SNE, Removing this module results significant performance degradation from 0.098 to 7,452. Without positional encoding, SNE is fully permutation equivariant even to functionally incorrect permutations of the weights, making difficult for the model to learn the correct restricted set of permutations. Secondly, removing the invariance regularization term results a degradation in performance, from 0.098 to 0.160, as also makes the task learning the correct minimal equivariance subgroup difficult. We find this only to have huge impact for MLPs such as INRs where weight permutations are well defined. Additionally, removing the set functions and the layer level and type encoders from the model results in performance drop.
E.2 Effect of Chunksize
Table 7: Effect of Chunksize.
Chunksize
MSE
0.095 #0.012
8
0.090 #0.023
16 0.118 +0.019
32
0.056 +0.020
We provide ablation on the effect of chunksize in the table below for the INR experiment presented in Table I of the main paper. From Table 7 we that performance stays almost the same until the largest chunksize. Hence the chunksize is selected to suit the memory requirements.
E.3 How Well Is The Logit Invariance Regularization Term Minimized?
compute the logit invariance loss for the experiments for functionally equivalent permutations of INR weights and obtain a loss of approximately 10-15 This low loss implies that the SNE learns the correct level of invariance which the desired property of the non-weight tying approach that we take.
16

---

Table 8: Cross-Dataset Neural Network Performance Prediction. We benchmark how well each method transfers across multiple datasets. In the first column, A + B implies that model trained homogencous model zoo of dataset A is evaluated on a homogeneous model zoo of dataset B. In the last row, we report the averaged performance of all methods across the cross-datased task. For each row, the best model is shown in red and the second best in blue. Models are evaluated in terms of Kendall's s rank correlation measure.
MLP
STATNN
NFNNp
NFN
SNE(ours)
MNIST + MNIST
0.878:00.001
0.926=0.000
0
0,942=00.011
0.941=0.000
MNIST > FashionMNIST
0,486.10.019
0.756.1.008
0.72610..00 005
0690.10.008
0.773-0.000
MNIST + CIFARIO
0.562+0.024
0.773+0.005
0.756+0.010
0.758+0.000
0.792-0.008
MNIST SVHN
0.544::0.005
0.698=0.005
0.702+400005
0.710:10.010
0.721=0.001
FashionMNIST MNIST
0.874=+0.001
0.915=0.000
0.922++0.00
0.935=00.000
0.928=0.001
FashionMNIST MNIST
0.507:10.007
0.667
0.755
0.617:0.012
0.722=00005
FashionMNIST CIFARIO
0.515::0.007
0.698=0.029
0.733 007
0.695:1..032
0.745=0.008
FashionMMIST SVHN
0.554:10.006
0.502.100043
0.663-10.014
1662.1.0003
0.66410.003
CIFARIO-> CIFARIO
0.880110.000
0.91211.001
0.924110002
0,93110.0000
0.927=0.000
CIFAR 10-> MNIST
0.552+0.0003
0.656+0.005
0.674+0.018
0.000+00025
0.648-0.000
CIFARIO-3 FashionMNIST
0.514:20.003
0.677=0.004
0.629=40.031
0.526:20.038
0.643=0.006
CIFARIO SVHN
0.578 005
0.72810.004
0.6971000006
0.662.10.004 004
0.753
SVHN > SVHN
0.809110.003
0.84420.000
0.8551000011
0.862+0.002
0,858=00003
SVHN MNIST
0.005
0.630#0.009
0.674:t0.008
0.64
0.647=00001
SVHN- FashionMNIST
0.523 026
0.616=0.007
0,567=
0,494=000025 023
0.655=0.003
SVHN > CIFARIO
0.540.10.027
0.74610.002
0.725-10.007
0.547:10.039
0.760=0.006
Posss-Dttsett Task
0.616::0.433
0.73420.115
0.746:0.106
0.705=0.140
0.761=00101
E.4 Measuring the Importance of Training Set Size
We provide a plot of training set size versus error in Figure using the INR experiment presented in Table 1. From this it can be seen that SNE is more data efficient compared to the baselines [Navon et al., 2023, Zhou et al., 2023b] across all percentages of the full training data demonstrating that the proposed method learns good embedding even in the limited data setting.
F Additional Experimental Results
In this section we provide additional experimental results on the cross-dataset and cross-architecture tasks.
F1
Cross-Dataset Evaluation
For this task, we train neural network performance predictors on 4 homogeneous model zoos, of the same architecture, with each model zoo restricted a single dataset.
Each model zoo is trained on one of the following datasets: MNIST [Deng, 2012], FashionMNIST [Xiao et al., 2017], CIFARIO [Krizhevsky, 2009] and SVHN [Netzer et al., 2018). We use the model zoos provided by Unterthiner et al. [2020]. To create each model zoo, 30K different hyperparameter configurations were sampled. The hyperpa- rameters include the learning rate, regularization coefficient, dropout rate, the variance and choice of initialization, activation functions etc. A thorough description of the model zoo generation process can be found in Appendix A.2 of Unterthiner et al. [2020]. Architectural descriptions for the model 200S are outlined in Appendix G. Each model zoo is split into training, testing and validation splits.
Datasets and Network Architecture:
Task: In this task we consider cross-dataset neural network performance prediction where we evaluate the prediction performance the testset of the model zoo which the predictors were trained on. Additionally, we evaluate how well each predictor transfers to the other model zoos. We evaluate all models using Kendall's [Kendall, 1938].
We present the results of the cross-dataset performance prediction task in Table 8. For each row in Table 8, the first column shows the cross-datased evaluation direction. For instance, MNIST-CCIFAR 10 implies that model trained on an MNIST model zoo is cross evaluated on a model 700 populated by neural networks trained on CIFAR10. 10. We note that the A-A setting, e.g. MNIST~MNIST corresponds to the evaluation settings of Unterthiner et al. [2020] and Zhou et al. [2023a]] Also in Table 8 we show the best model in red and the second best model in blue.
Results:
17

---

Table Cross-Archilecturee NN Performance Prediction. We show how SNE transfers architectures and report Kendall's T.
Archa - Arch:
SNE
MNIST > MINIST
0,452
MNIST CIFARIO
0.478=0.010
MNIST SVHN
0.082::0011
CIFAR 10-+ CIFARIO
0.511=0.020
CIFARIO MNIST
0.467=0.020
CIFARIO- SVHN
0.594=0.029
SVHN - SVHN
0.621=0.013
SVHN- MNIST
0.096
SVIIN CIFARIO
0..01
Table 10: Arch, for MNIST, MNShiOnMNIST, CIFAR1O and SVHN.
Output Size
Layers
1 32 32
Input Image
16 30 30
out_channels--6, Kernel_size=]] ReLU
16
x 28
Conv2d(in__ (i__chaneeee=-16 cut_channels=-6, kernel_size-3) ReLU
16 x 26 x 26
Conv2d (ttpiiinnnannnn66..mm out cotnneennn,,,,, kernel_size=2 ReLU
16 x 1
1))
16
Flatten
10
Linear (ineariin_features-t fet_uuaaur)-=10)
As show in Table 8, SNE is always either the best model or second best model the cross-dataset task. SNE is particularly good in the A-BB performance prediction task compared to the next competitive baselines, NFN and NFNHNP- The MLP baseline, as expected, performs the worse since concatenating all weight values in single vector looses information such the network structure. STATNN [Unterthiner et al., 2020] performs relatively better than the MI.P baseline suggesting that the statistics of each layer indeed captures enough information do moderately well on the neural network performance prediction task. NFNNP and NNNN perform much better than STATNN and MLP and NFN in particular shows good results in the A-+A setting, Interestingly, NFNNp is much better cross-dataset performance prediction model than NENANP. However, across the entire cross dotseet neural network performance prediction task, SNE outperforms all the baselines as shown in the last row of Table 8.
F.2 Cross-Architecture Evaluation
Next we reverse the transfer direction from Arch, to Arch; in Section 4.2 and provide the results for SNE Table 9. Note that for this task NeuralGraph [Kofinas et al., 2024] cannot be used since we transfer from smaller architecture to larger one. As can be seen from Table 9, SNE transfers well across architectures further validating the results in Section 4.2.
G Architectures for Generating model zoos
We specify the architectures for generating the model zoos of Arch; and Archz. For the model zoos of Arch, in Table 10, all datasets with with 3 channel images (CIFAR10 and SVHN) are converted grayscale. This in accordance with the setups of Unterthiner et al. [2020], Navon et al. [2023] and Zhou et al. [2023a,b] and allows us to evaluate both methods in the cross-dataset task for this of homogeneous model zoos. For model zoos of Archy in Tables 11 12, we maintain the original channels of the datasets. The cross- pross-architecture transfer task is from Arch; to Archy. Note also that for the same dataset, i.e., CIFARIA 10, the vross-architecture evaluation is also from models trained grayscale to RGB images. All model zoos were generated using the procedure outlined in the Appendix of Unterthiner al. [2020]. The architecture for the INR experiments outlined in Table 16.
18

---

Table 11: Archa for MNIST.
Output Size
Layers
x 28
Input Image
x
24
x 24
Conv2d(in, channels=1 out, channelln,,, kernel_sizess
(onham(in_cammmels-1
x 12 x 12
MaxPoo12d (kernell fhernel_siie=__ stride=2), , ReLU
6 x 8 8
Conv2d chanmels=8 (oar2(((n__cammeees5 out_channels=6, kermel_size=5)
MaxPool2d SaaaPooldkkeree__sss [kernel_siiee2, stride=2), ReLU
4x3x3
Conv2d(in channelsa out cut_channels=4, kernel_size=2] ReLU
36
Flatten
20
Linear (in_ out aut_features=20), ReLU
Linear finttarrr_e out_features=100
Table 12: Arch2 for CIFARIO and SVHN.
Output Size
Layers
x 28
Input Image
x
24
x 24
Conv2d(in, (in_chamells=3 out out_channees=-- kernel_size=5)
8 x 12 x 12
MaxPoo12d (kerrelsdkkrrressieeee stride=2), ReLU
x
x
Conv2d (in_chnnells=8 out_chaanels=6, kernel_size=5)
x
(kerrdeeseeeeeiieee22 stride=2), ReLU
4 x
3
3
Conv2d (hn22l(in_hhamees55 out_ cut_channels=4, kernel_size-2) ReLU
36
Flatten
20
Linear (in_arrin_eetuuee--33333 out out_features=20), ReLU
Linear fin_ttrrret, fut__featues=10)
H
Dataset Details
Dataset splits for model zoos of Arch, is given in Table 13. For the cross-architecture task, we generate model zoos of with 750 neural networks of Archa for testing.
Table 13: Dataset splits for model of Archi.
Model zoo
Train set
Validation set
Test set
MNIST
11998
3000
14999
FNSST
12000
3000
15000
CIFARIO
12000
3000
15000
SVHN
11995
2999
14994
I Hyperparameters
We elaborate all the hyperparameters used for all experiments in Table 14.
Table 14: Hyperparameters for CNN experiments.
Hyperparameter
Value
LR
Optimizer
Adam
Scheduler
Multistep
Batchsize
64
Epochs
300
Metric
Binary Cross Entropy
SAB Hidden Size
64
PMA Seed Size
64
# SAB Blocks
chunksize
SAB LayerNorm
False
19

---

Table 15: Generalization Performance Predictor.
Output Size
Layers
Linear (in_ features==022, out_ out_ffeatures=1000) ReLU
1000
Linear (in_ffetuues=1000, out out_featuree=1000) ReLU
Linear fin_ffaauure=1000, fettureet Sigmoid
Table 16: INR Architecture. Activations are Sinusoidal
Output Size
Layers
32
Linear (inearrii_feeee fin__reaures=1, out ftt_feaauues=32), Sine Linear featurere, out fut__feature=33)) Sine Linear fin_ out_feaatues=:)
32
Implementation Details
SNE is implemented using Pytorch [Paszke et al., 2019]. The SNE model consists of 4 sub-modules:
Layer Chunk Encoder:
This consists of two SAB modules where each SAB module is as stack of two SAB layers, followed by single PMA layer. The layer chunk encoder encodes all the chunks of a given layer independently.
Layer Encoder: This module encodes all the encoded chunks of layer and consists of two SAB modules and a single PMA layer.
Separated Layer Encoder: This module encodes all the encodings of a layer, for instance the weights and biases, into single layer encoding vector. It also consists of two SAB modules and a single PMA layer.
NN Encoding Layer: This module takes as input all the layer encodings and compresses them to obtain the neural network encoding which is used for the downstream task of predicting the neural network generalization performance. also consists of two SAB modules and a single PMA layer.
In addition to the sub-modules above, the layer/level positional encoders are applied to each sub- module when required (sce Section 3). The neural network performance predictor, which takes as input the neural network encoding vector from SNE and predicts the performance is detailed in Table 15.
K Miscellanea
In Figures 4, and 6 we provide enlarged versions of the figures in Figure 2 for better viewing.
20

---

MNIST
PashionMNiST
CIFAR10
SVHN
(a) MLP
(b) STATNN
NFNNP
NFNNNP
SNE(OOur))
Figure 4: TSNE Visualization Neural Network Encoding. We train neural network performance prediction methods on a combination of the MNIST, FashionMNIST, CIFARIO and SVHN modelzoos of Unterthiner et al. [2020]. Zoom in for better viewing.
21

---

MNIST
PashionMMNSS
CIFAR10
SVHN
(a) MLP
(b) STATNN
NFNNP
NFNNNF
SNE(OOur))
Figure 5: TSNE Visualization Neural Network Encoding, We train neural network performance prediction methods on a combination of the MNIST, FashionMNIST, CIFARIO and SVHN modelzoos of Unterthiner et al. [2020]. Zoom in for better viewing.
22

---

MNIST
PashionMNiST
CIFAR10
SVHN
(a) MLP
(b) STATNN
NFNNP
NFN
SNE(OOur))
Figure 6: TSNE Visualization Neural Network Encoding. We train neural network performance prediction methods on a combination of the MNIST, FashionMNIST, CIFARIO and SVHN modelzoos of Unterthiner et al. [2020]. Zoom in for better viewing.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We experimentally demonstrate strong empirical results on various benchmarks to back the claims made in the abstract and introduction. Additionally, we provide theoretical backing for our method in Appendix C.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations of our method Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer: [Yes]
Justification:
Theoretical considerations are provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all experimental settings well descriptions of model zoos used in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The appendix provides details generating the model used for periments in this paper. Additionally, a reference implementation will be made publicly available.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/ cublic/guides/Cee aublic//uudee//ddeeettniissss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wwtppblic/guuieeeeecooooooopcppd httpa///ggiiee//eeeuuuiiieeeee www.eesubbisiiooool.i.eo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
All such details are provided in the experimental section and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results have error bars with the exception of Table where single model is evaluated in accordance with the experimental setting described.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computational resources used all experiments is stated Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tppa//////r/..../.// http:///cwrpppcpppppcii/c/cc///a
Answer:
[Yes]
Justification: All NeurIPS code of Ethics observed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential impact and extensions of our method is provided in Appendix B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets, code, etc are appropriately cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode. thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "zz1u20BwsS", "reviewer": "Reviewer_bv1s", "summary": "The study introduces Set-based Neural Network Encoding (SNE) without weight tying, allowing for encoding network information into a compact form. SNE utilizes Logit Invariance Regularization to ensure the correct functional equivalence of network permutations, departing from traditional weight tying methods. The choice of Set-to-Set and Set-to-Vector functions, including attention mechanisms and pooling layers, are crucial in implementing the SNE framework. Experimental results demonstrate the effectiveness of SNE in predicting generalization performance for Convolutional Neural Networks (CNNs) and Incremental Neural Networks (INRs). SNE is compared with various baselines such as MLP, Deepsets, HyperRep, and others, showcasing its superiority in certain aspects.", "questions": "- In table 1 NeuralGraph is not reported. Is it because the method is not applicable there?\n\n- Does the method need more optimization steps than baseline methods that ensure weight-space symmetry by design? I suppose optimization would be harder because it requires to optimize for the logit invariance.", "limitations": "The limitations of the method are discussed in Appendix B", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- New evaluation benchmarks of cross-dataset and cross-architecture evaluation.\n\n- Solid empirical performance. SNE significantly outperforms HyperRep by very large margins in the cross-architecture task. Also, SNE demonstrates true agnosticism to architectural choices compared to NeuralGraph, and shows robust performance without being trained on the training set of a specific model zoo.", "weaknesses": "- The proposed method did not ensure weight-space equivariance by design but use a way that is more like an augmentation plus a consistency loss. The theoretic contribution is therefore a bit weak.\n\n- The padding and chunking designs look tricky and it is not clear for me how they help with the performance.\n\n- Fig 1 is not really clear. (1) the legend is not sufficient to understand the computation diagram presented in the figure. (2) the input is shown as neurons but from the method I think it should be the weight matrix that connecting neurons in adjacent layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for taking the time to offer constructive feedback for improving the paper. We respond to the questions raised below. \n\n\n**The proposed method did not ensure weight-space equivariance by design but use a way that is more like an augmentation plus a consistency loss. The theoretic contribution is therefore a bit weak.**\n\n\nWhile we use a regularization approach to achieve approximate minimal equivariance in weight-space, our usage of the logit invariance regularizer [1] theoretically guarantees that we indeed learn the correct invariance property similar to the weight-typing approaches used by the baselines. Additionally, our formulation is what allows us to deal with arbitrary architectures using a single model, as opposed to the baselines, since strict enforcement of weight-space equivariance by design requires crafting a new model for every distinct architecture. In this sense, our approach provides a general encoder which in principle is applicable to any architecture, resolving the limitations of purely weight-tying approaches. \n\n**The padding and chunking designs look tricky and it is not clear for me how they help with the performance.**\n\n\nThe padding and chunking operations are introduced in our pipeline to offer computational efficiency in cases where the parameters are large and hence infeasible to process all together. In Table 6 of the appendix we provide an ablation on the effect of the chunk size, which we reproduce below on the frequency prediction task:\n\n| ChunkSize | MSE                 |\n|-----------|---------------------|\n| 4         | $0.095 \\pm 0.012$ |\n| 8         | $0.090 \\pm 0.023$ |\n| 16        | $0.118 \\pm 0.019$ |\n| 32        | $0.056 \\pm 0.020$ |\n\nHere we see that performance stays almost the same until the largest chunksize (which in this case is set to the largest size hence requiring no chunk/pad operations). The chunksize is selected to suit the gpu memory requirements allowing us to train our method even with limited gpu memory which in our case is a single GTX 1080 gpu with 11GB of memory. \n\n\n**Fig 1 is not really clear. (1) the legend is not sufficient to understand the computation diagram presented in the figure. (2) the input is shown as neurons but from the method I think it should be the weight matrix that connecting neurons in adjacent layers.**\n\n\n\nWe have modified the figure to show the weight matrix instead of the layer in compressed form as shown in the figure in the updated paper. We thank the Reviewer for pointing this out as this improves the clarity of the presented pipeline. Additionally, we link the relevant approach subsections to each stage in the figure to aid clarification of the computation diagram.\n\n**In table 1 NeuralGraph is not reported. Is it because the method is not applicable there?**\n\nYes. We were unable to apply NeuralGraph for this task. Before submission, correspondences with the authors of NeuralGraph could not resolve the issue we encountered when applying the method for this task and that of Table 4.\n\n\n**Does the method need more optimization steps than baseline methods that ensure weight-space symmetry by design?**\n\nNo. In all experiments, we train our method for the same number of epochs as all the baselines. The addition of the logit invariance regularizer introduces no further optimization steps in all our experiments and we encountered no difficulties in minimizing the general objective together with the logit invariance loss. \n\n**References**\n\n[1] Artem Moskalev, Anna Sepliarskaia, Erik J Bekkers, and Arnold WM Smeulders. On genuine invariance learning without weight-tying. In Topological, Algebraic and Geometric Learning Workshops 2023, pages 218–227. PMLR, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UE8tugoMPT", "reviewer": "Reviewer_nYWS", "summary": "This work tackles an original and interesting challenge: predicting neural networks properties from their trained weight values. To do so, the authors propose to leverage set to set and set to vector transformations in order to encode weight values. Furthermore, they propose to account for the deep neural network architecture and operation order through the addition of positional encoding at several stages of the property predicting model. Ultimately, they anticipate the problem of various model sizes with a padding/chinking mechanism.\nThe proposed method is then evaluated on small scale datasets in a cross-dataset fashion against method of the field, published in recent years.", "questions": "I summarize the previous points:\n1. could the authors provide results with a predictor trained on CNNs from arch1 and predict the performance of transformers?\n2. could the author measure the importance of the training set size (pairs of DNNs and accuracy) w.r.t. the predictor's performance, in order to estimate the cost of working on ImageNet-like tasks?", "limitations": "The authors made a relevant attempt at highlighting the limitations of their work", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Such research studies can have a significant in other fields. For instance, a good accuracy estimator from model weight values can be leveraged as a proxy objective in DNN quantization or weight pruning.\nThe approach is well motivated and the solutions to the highlighted challenges are sound.\nThe paper is well written and simple to follow.\nThe authors provide a thorough evaluation and ablation study of the proposed method and its components.", "weaknesses": "From my understanding of this work, I identified two major drawbacks:\n1. the authors insist on the generalization across different model architectures and tasks. From my perspective, the former is the most important one. However, as it stands, there is little variation between Arch1 and Arch2 as presented in this study: both are simple CNNs. I wonder if the authors could provide numbers w.r.t transformer architectures against CNNs.\n2. Since these experiments can be expensive (require multiple training of DNNs to create the predictor training set), it would be interesting to measure the impact of the dataset size w.r.t the predictor's performance, in order to estimate how far the community is from being to reasonably work on large scale datasets such as ImageNet.\n\nAs it stands, I believe this work aims at providing empirical results which are not fully convincing yet. I look forward to the authors' response and other reviews and will update my rating accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for taking the time to offer constructive feedback for improving the paper. We respond to the questions raised below.\n\n**could the authors provide results with a predictor trained on CNNs from arch1 and predict the performance of transformers?**\n\nWe provide the requested results below. We generate a model zoo of transformer classifiers based on the [1] and test the transfer from arch1 to the transformer model zoo as requested.\nFor this task, we are unable to benchmark against NeuralGraph[2] as was done in Table 2, since the model cannot process transformer weights when trained on Arch1. Hence we benchmark against the DeepSets baseline as in Table 2. We present the results in the table below:\n\n| Arch1 $\\rightarrow$ Transfer       \t| DeepSets | SNE(Ours)  |\n|----------------------------|----------|------|\n| MNIST $\\rightarrow$ MNIST\t| 0.1975 $\\pm$ 0.000 \t| **0.4625** $\\pm$ 0.006 |\n| CIFAR10 $\\rightarrow$ MNIST   | 0.1970 $\\pm$ 0.000 \t| **0.3278** $\\pm$ 0.029 |\n| SVHN $\\rightarrow$ MNIST | 0.1906 $\\pm$ 0.000 \t| **0.3735** $\\pm$ 0.009 |\n\nFrom the results, the proposed SNE generalizes better to the unseen transformer architecture at test time than the baselines showing strong architectural transfer. Additionally, here, the model encodes an architecture with about 5 times the number of parameters in Arch1 demonstrating the scalability of our approach. The DeepSets baseline fails to generalize on this task.\n\n**could the author measure the importance of the training set size w.r.t. the predictor's performance, in order to estimate the cost of working on ImageNet-like tasks?**\n\nWe provide the requested results in Figure 1 of the attached pdf. From this it can be seen that SNE is more data efficient compared to the baselines (DWSNet[2] and NFT[3]) across all percentages of the full training data demonstrating that the proposed method learns a good embedding even in the limited data setting. We thank the Reviewer for suggesting this ablation as it further demonstrates the strength of SNE.\n\n\n**References**\n\n[1] https://github.com/s-chh/PyTorch-Scratch-Vision-Transformer-ViT\n\n[2] Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. arXiv preprint arXiv:2301.12780, 2023.\n\n[3] Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. arXiv preprint arXiv:2305.13546, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l0UAZmdyeJ", "reviewer": "Reviewer_EgRw", "summary": "This work introduces a set-based neural network encoding that processes each chunk of an input model independently to predict network properties. The proposed model, SNE, is trained using Logit Invariance instead of weight tying to maintain generalizability to unseen architectures. Unlike previous approaches, SNE can be trained and tested on different architectures and it demonstrated superior performance in out-of-distribution experiments involving both datasets and architectures.", "questions": "The writing is generally good but could be improved in certain areas. For example, the abstract contains a redundant sentence: “by utilizing a layer-wise encoding scheme that culminates to encoding all layer-wise encodings to obtain the neural network encoding vector,” which should be revised.", "limitations": "Limitations were properly addressed in appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "SNE is more general than its predecessors and can be applied to any input architecture. Results indicate that it outperforms other encoding methods.", "weaknesses": "While SNE shows improvements in novel tasks such as cross-architecture and cross-dataset scenarios, its comparison on traditional tasks is limited to predicting INR frequencies. Expanding Table 1 to include SNE's performance on additional outputs, such as accuracy, would enhance the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for taking the time to offer constructive feedback for improving the paper. We respond to the questions raised below.\n\n**The writing is generally good but could be improved in certain areas. For example, the abstract contains a redundant sentence: “by utilizing a layer-wise encoding scheme that culminates to encoding all layer-wise encodings to obtain the neural network encoding vector,” which should be revised.**\n\nWe have removed the sentence from the abstract and replaced it with ‘Furthermore, our $\\textbf{S}$et-based $\\textbf{N}$eural network $\\textbf{E}$ncoder (SNE) takes into consideration the hierarchical computational structure of neural networks.’ We hope this resolves the redundancy pointed out.\n\n**Expanding Table 1 to include SNE's performance on additional outputs**\n\nOur request to [1] for the dataset of INRs for the ShapeNet dataset is currently pending and this task will be added to Table 1 when the dataset is received. However we note that fundamentally, there is no difference between the accuracy prediction task for INRs and the frequency prediction task that we present in Table 1 since these are all evaluations on MLPs.\n\nWe would also like to draw the Reviewers attention to the additional results we provide on generalization on vision transformers which we provide in our general response. We hope these additional evaluations further demonstrate the wide range applicability of our method.\n\n**References**\n\n[1] De Luigi, Luca, et al. \"Deep learning on implicit neural representations of shapes.\" arXiv preprint arXiv:2302.05438 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F8XYqEkqbn", "reviewer": "Reviewer_uB2K", "summary": "This paper proposes SNE, or Set-based Network Encoding, a method of encoding the weights of arbitrary neural networks in order to predict properties such as performance, generalization gap, training loss, number of epochs, etc.  Specifically, using a chunking mechanism as well as layer-wise and block-wise positional encodings to generalize across different types of neural networks. Transferability experiments are conducted where a predictor is trained on one network model zoo, then evaluate on another, demonstrating an ability few predictors have. Additional experiments and ablations are performed, producing convincing results. Finally, the paper is decently well-written, but has a few fixable presentation flaws.", "questions": "I find the idea of predicting end-to-end neural network attributes from the weight values to be quite fascinating. It is very similar to the concept of a neural predictor [5] in Neural Architecture Search [6] but quite orthogonal in the approach, relying on the weights instead of the graph structure. Similarly, most neural predictors suffer from a lack of generalizability/transferability [7, 8], like most of the baselines in this paper. Can the authors provide some commentary/comparison between these two umbrella fields? \n\nSimilarly, is it possible to leverage the encoding mechanism (Eqs. 1-9) to derive which types of weights/layers/values are responsible for certain end-to-end attributes, e.g., performance of epochs, e.g., as [9] do?\n\nRefs:\n\n[1] \"Equivariant Architectures for Learning in Deep Weight Spaces\" - ICML'23.\n\n[2] https://modelzoo.co/\n\n[3] https://github.com/onnx/models\n\n[4] https://pytorch.org/serve/model_zoo.html\n\n[5] \"How Powerful are Performance Predictors in Neural Architecture Search?\" - NeurIPS'21.\n\n[6] \"Efficient Neural Architecture Design via Capturing Architecture-Performance Joint Distribution\" - AISTATS'24.\n\n[7] \"GENNAPE: Towards Generalized Neural Architecture Performance Estimators\" - AAAI-23.\n\n[8] \"Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor\" - NeurIPS'22.\n\n[9] \"Building Optimal Neural Architectures using Interpretable Knowledge\" - CVPR'24.", "limitations": "Yes, limitations and future work is discussed in App. B. The authors explain that their experiments are on simpler/smaller/older networks as they have limited computational resources. I find this to be an acceptable justification.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The contribution of the paper, using a network's weights to determine attributes about it, are quite novel. \n- The Methodology section is quite clear and easy to follow. Specifically, the aggregation mechanism in Eq. 9 is similarly analogous to graph aggregation in GNNs. \n- The experimental results are generally convincing.\n- Some visualization of the network representations from different methods are provided.\n- There are ablation results included in the appendix.\n- A limitations section is provided, highlighting reasonable restrictions on the experiments.", "weaknesses": "- First, the motivation of this paper, specifically lines 29-30, is quite weak and fails to really provide examples of the application of this research other than asserting that they must exist.\n- Overall, the presentation is a weak spot, and should be revised. Some examples: \n\t- E.g., in line 31, the authors mention \"Implicit Neural Representations (INR)\" - what are these? No citation is provided. By contrast, \"b) the performance of CNNs\" is very easy to understand.\n\t- \"modelzoo\", this should be \"model-zoo\" [1] or \"model zoo\" [2-4].\n\t- Citations in the paper have issues. For NeurIPS its generally just [num], or if you want to be fancier, \"Author et al., Year [num]\" but not \"Author et al. [num]\" without the year.\n\t- Float captions need work, e.g., Tabs 2/3 need longer captions to fully explain the performance metric Kendall's Tau, etc. Should be longer than 1 sentence.\n\t- Figure 2 needs work. The plots should be bigger and legend is clipped.\n\t- NeurIPS checklist should be placed after the references and supplementary.\n- The evaluation is done on simpler datasets/networks, however, the limitations section highlights why.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for taking the time to offer constructive feedback for improving the paper. We respond to the questions raised below.\n\n**Provide examples of the application of this research other than asserting that they must exist.**\n\nWe add the following to elaborate on the potential applications of this research to complement lines 29-30:\n‘The ability to infer such fundamental properties of trained neural networks using only the parameter values has the potential to open up new application and research paradigms such as  learning in the latent space of neural network weights for tasks such as weight generation [1, 2], latent space transfer of weights across datasets allowing for transferring weights from one dataset to another as was recently demonstrated in [3] and latent space optimization using gradient descent where optimization is performed on the weight embeddings [4].’\nWe hope this addition enhances the motivation for this line of research.\n\n**Overall, the presentation is a weak spot, and should be revised. Some examples:**\n- We have added a citation to Implicit Neural Representations (INR) [1] and added the following description, “INRs are neural parameterizations of signals such as images using multi-layer perceptrons”.\n- All occurrences of ‘modelzoo’ have been replaced with ‘model zoo’.\n- We have changed the citation format to ‘Author et al., Year[num]’.\n- We’ve included the following to captions on Tables 2/3 “Models are evaluated using Kendall’s Tau[5], a rank correlation metric.’ with a citation to the paper that introduces the metric.\n- We have reduced the number of plots in Figure 2 to one per model and increased the size (see Figure 2 in the attached pdf). The remaining figures are moved to the appendix in enlarged form and the clipped legend fixed. \n- The checklist has been placed after the references and supplementary as pointed out.\n\n**The evaluation is done on simpler datasets/networks, however, the limitations section highlights why.**\n\nBased on the suggestion of Reviewer nYWS we provide additional results on more complicated networks, vision transformers, in our response to nYWS and in the general response. We hope that these additional results on more complicated architecture demonstrates the utility of the proposed method.\n\n**Can the authors provide some commentary/comparison between these two umbrella fields (NAS and weight encoding)?**\n\nIn NAS, the predictors mostly rely on the computation graph of generated architectures. This is quite different from the problem we consider where we require access to the parameter values themselves, optimized for some number of epochs. However, the general underlying problem has commonality between the two fields: namely predicting some properties of the networks in consideration. Perhaps a new line of exploration would consider combining both fields. For instance, a neural network encoder, like ours, could be trained to conditionally take as input either the weights, the computational graph or both. Such a pre-trained predictor could be applied to NAS where in test mode, only the computational graph is given to estimate the performance of newly generated architectures. This would require the encoder to be agnostic to architectural choices like our model. The discovered architectures when trained could then be used to update the property predictor using both weights and computation graphs. Such a combination would find application in both fields. To the best of our knowledge, we are unaware of any works that attempt such a combination and we leave this exploration as future work and will draw attention to such a merger of both fields in our discussion and limitation sections.  \n\n**Similarly, is it possible to leverage the encoding mechanism (Eqs. 1-9) to derive which types of weights/layers/values are responsible for certain end-to-end attributes, e.g., performance of epochs, e.g., as [9] do?**\n\nTechnically, such an analysis should be possible. For instance, we could train SNE to predict multiple properties for a given network and at test time, we can remove the encodings of different layers (in Eq 9) and see how it affects the predictions compared to the full network. This will provide a mechanism for determining which layers/weights are responsible for certain attributes. Currently we are unable to do such an analysis since all the model zoos used in this paper only come with a single attribute in the meta-data. Such analysis will require regenerating the model zoos, a task which we are unable to undertake during the short rebuttal period. However, we are inclined to agree with the Reviewer that such an analysis will be very useful.\n\n**References**\n\n[1] Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights, 2022\n\n[2] Wang, Kai, et al. \"Neural Network Diffusion.\" arXiv preprint arXiv:2402.13144 (2024).\n\n[3] Soro, Bedionita, et al. \"Diffusion-based Neural Network Weights Generation.\" arXiv preprint arXiv:2402.18153 (2024).\n\n[4] Rusu, Andrei A., et al. \"Meta-learning with latent embedding optimization.\" arXiv preprint arXiv:1807.05960 (2018).\n\n[5] Kendall, M. G. A new measure of rank correlation. Biometrika, 30(1/2):81–93, 1938.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response**

We thank all Reviewers for taking the time to offer constructive feedback for improving the paper. Based on the suggestions of Reviewers nYWS, we provide the following additional results. 


**Evaluation on Transformers**

We generate a model zoo of transformer classifiers based on the [1] and test the transfer from arch1 to the transformer model zoo as requested.
For this task, we are unable to benchmark against NeuralGraph[2] as was done in Table 2, since the model cannot process transformer weights when trained on Arch1. Hence we benchmark against the DeepSets baseline as in Table 2. We present the results in the table below:

| Arch1 $\rightarrow$ Transfer           | DeepSets | SNE(Ours)  |
|----------------------------|----------|------|
| MNIST $\rightarrow$ MNIST    | 0.1975 $\pm$ 0.000     | **0.4625** $\pm$ 0.006 |
| CIFAR10 $\rightarrow$ MNIST   | 0.1970 $\pm$ 0.000     | **0.3278** $\pm$ 0.029 |
| SVHN $\rightarrow$ MNIST | 0.1906 $\pm$ 0.000     | **0.3735** $\pm$ 0.009 |



From the results, the proposed SNE generalizes better to the unseen transformer architecture at test time than the baselines showing strong architectural transfer. Additionally, here, the model encodes an architecture with about 5 times the number of parameters in Arch1 demonstrating the scalability of our approach. The DeepSets baseline fails to generalize on this task.

**Measuring the Importance of training set size.**

We provide a plot of training set size versus error in Figure 1 of the attached pdf. From this it can be seen that SNE is more data efficient compared to the baselines (DWSNet[2] and NFT[3]) across all percentages of the full training data demonstrating that the proposed method learns a good embedding even in the limited data setting. We thank the Reviewer for suggesting this ablation as it further demonstrates the strength of SNE.

We hope that these additional results further emphasizes the benefits of the proposed method.

**References**

[1] https://github.com/s-chh/PyTorch-Scratch-Vision-Transformer-ViT

[2] Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. arXiv preprint arXiv:2301.12780, 2023.

[3] Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. arXiv preprint arXiv:2305.13546, 2023.

Meta-review: The paper proposes to predict neural network training properties based on set-based representations of neural network parameters. There exist mixed reviews, but reviewers agree that the approach has novelty. The main complaint is comparisons with priors works including NeuralGraphs, as well as the findings of Papa et al., How to Train Neural Field Representations: A Comprehensive Study and Benchmark. In CVPR 2024. The authors make a fair argument that they are expecting a response from the authors before including the final results. I suggest the paper to be accepted under the condition that the include the more detailed comparisons with NeuralGraph that they currently miss, and discuss their findings compared to Papa et al.
Decision: Accept (poster)