Set-based Neural Network Encoding Without Weight Tying

Authors: bruno  andreis
soro  bedionita
philip h s torr
sung ju hwang
  kaist
south  korea
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:46:37

Abstract: We propose a neural network weight encoding method for network property prediction that utilizes set-to-set and set-to-vector functions
to efficiently encode neural network parameters. Our approach is capable of encoding neural networks in a model zoo of mixed architecture and different parameter sizes as opposed to previous approaches that require custom encoding models for different architectures. Furthermore, our \textbf{S}et-based \textbf{N}eural network \textbf{E}ncoder (SNE) takes into consideration the hierarchical computational structure of neural networks. To respect symmetries inherent in network weight space, we utilize Logit Invariance to learn the required minimal invariance properties. Additionally, we introduce a \textit{pad-chunk-encode} pipeline to efficiently encode neural network layers that is adjustable to computational and memory constraints. We also introduce two new tasks for neural network property prediction: cross-dataset and cross-architecture. In cross-dataset property prediction, we evaluate how well property predictors generalize across model zoos trained on different datasets but of the same architecture. In cross-architecture property prediction, we evaluate how well property predictors transfer to model zoos of different architecture not seen during training. We show that SNE outperforms the relevant baselines on standard benchmarks.

Main paper: 
 introduction: 
 Recently, deep learning methods have been applied to a wide range of fields and problems. With this broad range of applications, huge volumes of datasets are continually being made available in the public domain together with neural networks trained on these datasets. Given this abundance of trained neural network models, the following curiosity arises: what can we deduce about these networks with access only to the parameter values? More generally, can we predict properties of these networks such as generalization performance on a testset(without access to the test data), the dataset on which the model was trained, the choice of optimizer and learning rate, the number of training epochs, choice of model initialization etc. through an analysis of the model parameters?The ability to infer such fundamental properties of trained neural networks using only the parameter values has the potential to open up new application and research paradigms [De Luigi et al., 2023 , Zhou et al., 2023a ,b, Navon et al., 2023] such as learning in the latent space of neural network weights for tasks such as weight generation [Schürholt et al., 2021 , Soro et al., 2024] , latent space transfer of weights across datasets allowing for transferring weights from one dataset to another as was recently demonstrated in Soro et al. [2024] and latent space optimization using gradient descent where optimization is performed on the weight embeddings [Rusu et al., 2018] .We tackle two specific versions of this problem: predicting a) frequencies of Implicit Neural Representations [Sitzmann et al., 2020] (INRs), and b) the performance of CNNs and Transformers, Layer-Type Encoder. Concept: (left) Given layer weights, SNE begins by padding and chunking the weights into chunksizes. Each chunk goes through a series of set-to-set and set-to-vector functions to obtain the chunk representation vector. Layer level and type positional encodings are used to inject structural information of the network at each stage of the chunk encoding process. All chunk encoding vectors are encoded together to obtain the layer encoding. (right) All layer encodings in the neural network are encoded to obtain the neural network encoding vector again using as series of set-to-set and set-to-vector functions. This vector is then used to predict the neural network property of interest.given access only to the network weights. The first approach to solving this problem, proposed by Unterthiner et al. [2020] , involves computing statistics such as the mean, standard deviation and quantiles, of each layer in the network, concatenating them to a single vector that represents the neural network encoding, and using this vector to predict the desired network property. Another approach, also proposed as a baseline in Unterthiner et al. [2020] , involves flattening all the parameter values of the network into a single vector which is then fed as input to layers of multilayer perceptrons (MLPs) to predict the property of interest. An immediate consequence of this approach is that it is practical only for moderately sized neural network architectures. Additionally, this approach ignores the hierarchical computational structure of neural networks through the weight vectorization process. The second, and most recent approach to this problem, proposed by Zhou et al. [2023a,b] , takes a geometric approach to the problem by building neural network weight encoding functions, termed neural functionals, that respect symmetric properties of permutation invariance and equivariance of the hidden layers of MLPs under the action of an appropriately applied permutation group. While this approach respect these fundamental properties in the parameter space, it's application is restricted, strictly, to MLPs. Also, even when relaxations are made to extend this method to convolutional networks and combinations of convolutional layers and MLPs, these only work under strict conditions of equivalence in the channel size in the last convolutional layer and the first linear layer. Hence it is clear that while the methods of Zhou et al. [2023a,b] and Navon et al. [2023] enjoy nice theoretical properties through weight tying, their application is limited to only a small subset of carefully chosen architectures.Moreover, these approaches [Unterthiner et al., 2020 , Zhou et al., 2023a ,b, Navon et al., 2023] have a fundamental limitation: their encoding methods are applicable only to a single fixed, pre chosen neural network architecture. Once the performance predictor is trained, in the case of Unterthiner et al. [2020] , and the neural network encoder of Zhou et al. [2023a] and Navon et al. [2023] are defined, they cannot be used to predict the performance of neural networks of different architecture. These issues are partly addressed by the graph based approaches of Kofinas et al. [2024] and Lim et al. [2023] . Consequently, evaluating these models on diverse architectures is impossible without training an new performance predictor for each architecture.To this end, we propose a Set-based Neural Network Encoder (SNE) for property prediction of neural networks given only the model parameters that is agnostic to the network architecture without weight tying. Specifically, we treat the neural network encoding problem from a set encoding perspective by utilizing compositions of set-to-set and set-to-vector functions. However, the parameters of neural networks are ordered and hierarchical. To retain this order information, we utilize positional encoding [Vaswani et al., 2017] at various stages in our model. Also, our model incorporates the hierarchical computational structure of neural networks in the encoder design by encoding independently, layer-wise, culminating in a final encoding stage that compresses all the layer-wise information into a single encoding vector used to predict the network property of interest. To handle the issue of large and variable parameter sizes efficiently, we incorporate a pad-chunk-encode pipeline that is parallelizable and can be used to iteratively encode layer parameters. To learn the correct permutations of MLP weights, we employ the Logit Invariance Regularizer of Moskalev et al. [2023] instead of weight tying. In terms of evaluation, we introduce two new tasks: cross-dataset neural network property prediction and cross-architecture neural network property prediction. In crossdataset neural network performance prediction, we fix the network architecture used to generate the training data and evaluate how well the performance predictors transfer to the same architecture trained on different datasets. For cross-architecture neural network performance prediction, we fix only the architecture for generating the training data and evaluate the performance of the predictor on architectures unseen during training. These tasks are important since in the real setting, access to model zoos is scare, making transferability desirable.Our contributions are as follows:• We develop a Set-based Neural Network Encoder (SNE, see Figure 1 ) for network property prediction given access only to parameter values that can encode neural networks of arbitrary architecture, taking into account the hierarchical computational structure of neural networks.• We introduce the cross-dataset property prediction task where we evaluate how well predictors transfer across neural networks trained on different datasets.• We introduce the cross-architecture property prediction task where we evaluate how well property predictors trained on a specific architecture transfer to unseen architectures.• We benchmark SNE against the relevant baselines [ Unterthiner et al., 2020 , Navon et al., 2023 , Zhou et al., 2023a,b] on the cross-dataset task and show significant improvement over the baselines.• We provide the first set of results on the cross-architecture task, a task for which most of the baselines, with the exception of Zaheer et al. [2017] and Kofinas et al. [2024] under special conditions (see remark in Section 4.2), cannot be used.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We experimentally demonstrate strong empirical results on various benchmarks to back the claims made in the abstract and introduction. Additionally, we provide theoretical backing for our method in Appendix C.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Set Functions: Neural networks that operate on set (un)structured data have recently been used in many applications ranging from point cloud classification to set generation [Kim et al., 2021] . Set functions are required to respect symmetric properties such as permutation invariance and equivariance. In DeepSets [Zaheer et al., 2017] , a set of sum-decomposable functions are introduced that are equivariant in the Set-to-Set applications and invariant in the Set-to-Vector applications. In Set Transformers [Lee et al., 2019] , a class of attention based Set-to-Set and Set-to-Vector functions are introduced that are more expressive and capable of modeling pairwise and higher order interactions between set elements. Recent works such as Bruno et al. [2021] and Willette et al. [2023] deal with the problem of processing sets of large cardinality in the the limited memory/computational budget regime. In this work, we utilize the class of set functions developed in Lee et al. [2019] to develop a neural network encoder for performance prediction that is agnostic to specific architectural choices.Our set-based formulation allows us to build such an encoder, capable of handling neural networks weights of arbitrary parameter sizes. This is a deviation from recent approaches to neural network encoding for property prediction that can encode only parameters of a single fixed architecture.Neural Network Property Prediction From Weights: Predicting the properties of neural networks given access only to the trained parameters is a relatively new topic of research introduced by Unterthiner et al. [2020] . In Unterthiner et al. [2020] , two methods are proposed for predicting the generalization performance of neural networks: the first involves flattening the weights of the network into a single vector, processing it using multiple layers of MLPs to obtain an encoding vector which is then used to predict the performance. The second involves computing the statistics of each layer in the network, such as mean, variance, quantiles etc., concatenating them into a single vector that is then used for predicting the performance of the network. The most recent approach that we are aware of, Navon et al. [2023] and Zhou et al. [2023a,b] , proposes a neural network weight encoder that is invariant or equivariant, depending on the application, to an appropriately applied permutation group to the hidden layers of MLPs. Two variants of their model is provided: one which operates only on the hidden layers, and conforms strictly to the theory of permuting MLP hidden neurons [Hecht-Nielsen, 1990] , and a relaxation that assumes that the neurons of both the input and output layers of MLPs are permutable. Additionally, extensions are provided for convolutional layers. Kofinas et al. [2024] and Lim et al. [2023] represent weights using graph neural networks. Our approach, SNE, is directly comparable to these methods on the neural network property prediction task. However, unlike the methods of Unterthiner et al. [2020] , Navon et al. [2023] and Zhou et al. [2023a] which operate only on neural networks of fixed architecture, and consequently fixed number of encodable parameters, SNE is capable of encoding networks of arbitrary architecture. Moreover, SNE utilizes the hierarchical computation structure of neural networks by encoding, iteratively or in parallel, from the input to the output layers. Additionally, we go further than the experimental evaluation in Unterthiner et al. [2020] , Navon et al. [2023] and Zhou et al. [2023a,b] by introducing two new tasks: cross-dataset and cross-architecture neural network property prediction. Unterthiner et al. [2020] , Navon et al. [2023] and Zhou et al. [2023a,b] can only be benchmarked on the cross-dataset task where all networks in the model zoos are of the same architecture. Their restriction to a single fixed architecture makes cross-architecture evaluation impossible. Our method SNE, and those of Kofinas et al. [2024] , Lim et al. [2023] , on the other hand can be used for both tasks.3 Set-based Neural Network Encoding Without Weight TyingQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss limitations of our method in Appendix B.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 We have access to a dataset D = {(x 1 , y 2 ), . . . , (x n , y n )} where for each (x i , y i ) pair, x i represents the weights of a neural network architecture a, sampled from a set of architectures A and y i corresponds to some property of x i after it has been trained on a specific dataset d. are the input and output layers of x i respectively. Additionally, we introduce the Flat : x i → R di operation, that takes as input the weights of a neural network and returns the flattened weights and d i is the total number of parameter is x i .The neural network encoding problem is defined such that, we seek to compress x i ∈ R di to a compact representation z xi ∈ R h such that z xi can be used to predict the properties y i of x i with h ≪ d i . In what follows, we present the details of our Set-based Neural Network Encoding (SNE) method capable of encoding the weights of neural networks of arbitrary architecture that takes into account the hierarchical computational structure of the given architecture and with efficient methods for processing weights of high dimension.For a given layer w j i ∈ x i , the dimension of w j i , |w j i | can be very large. For instance, when considering linear layers, flattening the weights can results in a tensor that can require large compute memory to be processable by another neural network. To resolve this issue, we resort to chunking. Specifically, for all layers w j i ∈ x i , we perform the following operations:EQUATIONwhere for anyw jt i ∈ ŵj i , |w jt i | ∈ R c .Here, c is the chunksize, fixed for all layer types in the neural network and t ∈ [0, . . . , q]. The padding operation Pad(w j i , c) appends zeros, if required, to extend w j i and make its dimension a multiple of the chunksize c. To distinguish padded values from actual weight values, each element of ŵj i has a corresponding set of masks mj i = {m j0 i , . . . , m jq i }. Note that with this padding and subsequent chunking operation, each element in ŵj i is now small enough, for an appropriately chosen chunksize c, to be processed. Moreover, all the elements in ŵj i can be processed in parallel. Importantly, chunksizes are chosen to ensure that weights from the same neurons are grouped together. This allows for principled learning of specific symmetries which we discuss later in Section 3.6. An ablation on the effect of chunksize is provided in Appendix E.The model zoos we consider in the experimental section are populated by neural networks with stacks of convolutional and linear layers. For each such layer, we apply the padding and chunking operation differently. For a linear layer w j i ∈ R out×in , where out and in are the input and output dimensions respectively, we apply the flattening operation on both dimensions followed by padding and chunking. However for a convolutional layer w j i ∈ R out×in×k×k , we do not apply the flattening, padding, and chunking operations to the kernel dimensions k and operate only on the input and output dimensions since the kernels are small enough to be encoded together. Finally we note that for layers with bias values, we apply the procedure detailed above independently to both the weights and biases.The next stage in our Set-based Neural Network encoding pipeline is the individual encoding of each chunk of weight in ŵj i = {w j0 i , . . . , w jt i }. For each w jt i ∈ ŵj i , we treat the c elements as members of a set. However, it is clear that w jt i has order in its sequence, i.e., an ordered set. We remedy this by providing this order information via positional encoding. Concretely, for a given w jt i ∈ R c×1 , we first model the pairwise relations between all c elements using a set-to-set function Φ θ1 to obtain:EQUATIONThat is, Φ θ1 captures pair-wise correlations in w jt i and projects all elements (weight values) to a new dimension h.i ∈ R c×h , we inject two kinds of positionally encoded information. The first encodes the layer type in a list of layers, i.e., linear or convolution for the model zoos we experiment with, to obtain:EQUATIONHere we abuse notation and assign the output of PosEnc(•) to ŵit i to convey the fact that ŵjt i 's are modified in place and to simplify the notation. Also, all PosEnc(•)s are variants of the positional encoding method introduced in Vaswani et al. [2017] . Next we inject the layer level information. Since neural networks are computationally hierarchical, starting from the input to the output layer, we include this information to distinguish chunks, w jt i s from different layers. Specifically, we compute:EQUATIONwhere the input to PosEnc Level Layer (•) is the output of Equation 3. We note that this approach is different from previous neural network encoding methods [Unterthiner et al., 2020] that loose the layer/type information by directly encoding the entire flattened weights hence disregarding the hierarchical computational structure of neural networks. Experimentally, we find that injecting such positionally encoded information improves the models performance (Ablation E).i , now infused with layer/type information, using another set-to-set function Φ θ2 :EQUATIONThe final step in the chunk encoding pipeline involves compressing all c elements in ŵjt i to a compact representation. For this, we use a set-to-vector function Γ θα : R c×h → R h . In summary, the chunk encoding layer computes the following function:EQUATIONwhere wjt i ∈ R 1×H . Note that for each chunked layer ŵj i = {w j0 i , . . . , w jq i }, the chunk encoder, Equation 6, produces a new set wj i = Concatenate[{ wj0 i , . . . , wjq i }] ∈ R q×h , which represents all the encodings of all chunks in a layer.Remark Our usage of set functions Φ θ1 , Φ θ2 and Γ θα allows us to process layers of arbitrary sizes, which allows us to process neural networks of arbitrary architecture using a single model, a property lacking in previous approaches to neural network encoding [Zhou et al., 2023a ,b, Unterthiner et al., 2020 , Navon et al., 2023] .At this point, we have encoded all the chunked parameters of a given layer to obtain wj i . Encoding a layer, w j i , then involves defining a function Γ θ β : R q×h → R 1×h for arbitrary q. In practice, this is done by computing:EQUATIONAgain we have injected the layer level information, via positional encoding, into the encoding processed by the set-to-set function Φ θ3 . We then collect all the layer level encodings of the neural networkx i : wi = Concatenate[w 0 i , . . . , w |xi| i ] ∈ R |xi|×h . (8)With all layers in x i encoded, we compute the neural network encoding vector z xi as follows:EQUATIONz xi compresses all the layer-wise information into a compact representation for the downstream task.Since Γ θγ is agnostic to the number of layers |x i | of network x i , the encoding mechanism can handle networks of arbitrary layers and by extension architecture. Similar to the layer encoding pipeline, we again re-inject the layer-level information through positional encoding before compressing with Γ θγ .Henceforth, we refer to the entire encoding pipeline detailed so far as SNE Θ (x i ) for a network x i , where Θ encapsulates the encoder parameters, Φ θ1-4 , Γ α , Γ β and Γ γ .Given an MLP, there exists permutations of the weights such that the two networks are functionally equivalent [Hecht-Nielsen, 1990] . Since not all permutations are functionally correct, the encoder needs to learn the correct functional equivalence. To achieve this, we utilize the concept of Logit Invariance Regularization [Moskalev et al., 2023] where we constrain the output of the non-equivariant SNE Θ (due to the positional encoding of input and output vectors) to respect the restricted functionally correct permutation group. This results in the following optimization problem:EQUATIONwhere G is the group of functionally equivariant permutations in the weight space, D is the training dataset and v balances the task loss ℓ f (D) and the Logit Invariance Regularization term ℓ f (D,G) .Proposition 3.1 of Moskalev et al. [2023] guarantees that the resulting SNE model will have low sensitivity to functionally incorrect permutations of the weights. In practice, ℓ f (D, G) is the L 2 distance between functionally equivalent permutations of the same weight. This approach differs from previous works [Zhou et al., 2023a ,b, Navon et al., 2023] which instead result to weight tying to achieve minimal equivariance.Remark While we use a regularization approach to achieve the required approximate minimal equivariance in weight-space, our usage of the Logit Invariance Regularizer [Moskalev et al., 2023] theoretically guarantees that we indeed learn the correct invariance property similar to the weighttyping approaches. Additionally, our formulation is what allows us to deal with arbitrary architectures using a single model (see Section 4), as opposed to previous works, since strict enforcement of weight-space equivariance by design requires crafting a new model for different architectures. In this sense, our approach provides a general encoder which in principle is applicable to any architecture, resolving the limitations of purely weight-tying approaches.Theoretical discussions on the adopted regularization based approach to minimal equivariance versus weight tying is provided in Appendix C.We specify the choice of Set-to-Set and Set-to-Vector functions encapsulated by Φ θ1-4 , Γ α , Γ β and Γ γ used to implement SNE. Let X ∈ R n X ×d and Y ∈ R n Y ×d be arbitrary sets where n X = |X|, n Y = |Y | and d (note the abuse of notation from Section 3.1 where d is a dataset) is the dimension of an element in both X and Y . The MultiHead Attention Block (MAB) with parameter ω is given by:EQUATIONEQUATION) Here, LayerNorm and rFF are Layer Normalization [Ba et al., 2016] and row-wise feedforward layers respectively. MultiHead(X, Y, Y ; ω) is the multihead attention layer of Vaswani et al. [2017] .The Set Attention Block [Lee et al., 2019] , SAB, is given by: SAB(X) := MAB(X, X).(13)That is, SAB computes attention between set elements and models pairwise interactions and hence is a Set-to-Set function. Finally, the Pooling MultiHead Attention Block [Lee et al., 2019] , PMA k , is given by: PMA k (X) = MAB(S, rFF(X)), whereS ∈ R k×d and X ∈ R n X ×d . The k elements of S are termed seed vectors and when k = 1, as is in all our experiments, PMA k pools a set of size n X to a single vector making it a Set-to-Vector function.All parameters encapsulated by Φ θ1-4 are implemented as a stack of two SAB modules: SAB(SAB(X)). Stacking SAB modules enables us not only to model pairwise interactions but also higher order interactions between set elements. Finally, all of Γ α , Γ β and Γ γ are implemented as a single PMA module with k = 1.Given (z xi , y i ), we train a predictor f θ (z xi ) to estimate properties of interest of the network x i . In this work, we focus solely on the task of predicting the generalization performance of x i , where y i is the performance on the test set of the dataset used to train x i for CNNs and frequencies for INRs.The parameters of the predictor f θ and all the parameters in the neural network encoding pipeline, Θ, are jointly optimized. In particular, we minimize the error between f θ (z xi ) and y i . For a model zoo, the objective is given as:EQUATIONfor an appropriately chosen loss function ℓ(•). In our experiments, ℓ(•) is the binary cross entropy or mean squared error loss. The entire SNE pipeline is shown in Figure 1 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The appendix provides details on generating the model zoos used for experiments in this paper. Additionally, a reference implementation will be made publicly available.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We Baselines: We compare SNE with the following baselines: a) MLP: This model flattens the entire weight of the network and encodes it using a stack of MLP layers. b) DeepSets: [Zaheer et al., 2017] This model treats the weights as a set with no ordering. c) HyperRep: [Schürholt et al., 2021] This model learns a generative model of the flattened weight vector. d) STATNN [Unterthiner et al., 2020] : This model computes the statistics of each layer such as the mean, variance and quantiles, and concatenates them to obtain the neural network encoding vector. e) DWSNet [Navon et al., 2023] : is a minimally equivariant model using weight tying developed mainly for MLPs. Various modules are provided for encoding biases, weights and combinations of these two. f) NFN HNP , NFN NP and NFT [Zhou et al., 2023a,b] : These models, termed Neural Functionals(NF), are developed mainly for MLPs and utilize weight tying to achieve minimal equivariance. HNP, hidden neural permutation, is applied only to the hidden layers of each network since the output and input layers of MLPs are not invariant/equivariant to the action of a permutation group on the neurons. NP, neural permutation, makes a strong assumption that both the input and output layers are also invariant/equivariant under the action of a permutation group. NFT is similar to both models and utilizes attention layers. g) NeuralGraph: [Kofinas et al., 2024] This method represents the weights of a network as a graph and uses graph pooling techniques to obtain the network representation vector. We note that we are unable to benchmark against Graph Metanetworks [Lim et al., 2023] , which uses a graph approach similar to NeuralGraph, since no code or data is publicly available.In all Tables, the best methods are shown in red and the second in blue. Additionally an extensive ablation of all the components of SNE is provided in Appendix E. All experiments are performed with a single GeForce GTX 1080 TI GPU with 11GB of memory. Results: As can be seen in Table 1 , SNE significantly outperforms the baselines on this task. Given that INRs are MLPs, minimal equivariance is particularly important for this task and shows that SNE learns the correct minimal equivariance required to solve the task using the logit invariance approach. Additionally, compared to the minimal equivariance constrained models [Zhou et al., 2023a ,b, Navon et al., 2023] , SNE is parameter efficient as show in Table 1 . We note that increasing the parameter counts of the MLP, DeepSets and STATNN baselines results in overfitting and poor performance. We utilize model zoos trained on MNIST, CIFAR10 and SVHN datasets. We generate a model zoo for these dataset with an architecture consisting of 3 convolutional layers followed by two linear layers and term the resulting model zoo Arch 1 . Exact architectural specifications are detailed in Appendix G. We generate the model zoos of Arch 2 following the routine described in Appendix A.2 of Unterthiner et al. [2020] . We refer to the model zoos of Unterthiner et al. [2020] as Arch 2 . All model zoos of Arch 1 are used for training and those of Arch 2 are used for testing and are not seen during training.Task: Here, we seek to explore the following question: Do neural network performance predictors trained on model zoos of Arch 1 transfer or generalize to the out-of-distribution model zoos of Arch 2 ? Additionally, we perform cross-dataset evaluation on this task where cross evaluation is with respect to model zoos of Arch 2 , i.e., we also check for out-of-distribution transfer across datasets.Baselines: We compare SNE with DeepSets [Zaheer et al., 2017] and NeuralGraph [Kofinas et al., 2024] for this task. None of the other baselines, MLP, STATNN [Unterthiner et al., 2020] , NFN NP [Zhou et al., 2023a] , NFN HNP [Zhou et al., 2023a] , NFT [Zhou et al., 2023b] and DWS-Net [Navon et al., 2023] can be used for this task since they impose architectural homogeneity and hence cannot be used for out-of-distribution architectures by design.We report the quantitative evaluation on the cross-architecture task in Kendall, 1938] in Table 3 . Additionally, here, the model encodes an architecture with about 5 times the number of parameters in Arch 1 demonstrating the scalability of our approach. The DeepSets baseline fails to generalize on this task. For this task, we train neural network performance predictors on 4 homogeneous model zoos, of the same architecture, with each model zoo restricted to a single dataset.Datasets and Network Architecture: Each model zoo is trained on one of the following datasets: MNIST [Deng, 2012] , FashionMNIST [Xiao et al., 2017] , CIFAR10 [Krizhevsky, 2009] and SVHN [Netzer et al., 2018] . We use the model zoos of Unterthiner et al. [2020] .A thorough description of the model zoo generation process can be found in Appendix A.2 of Unterthiner et al. [2020] .Task: We train network encoders on a single model zoo, e.g. MNIST, and evaluate it on the test set of all four datasets and report the averaged performance. This results in in-distribution testing with respect to the dataset used for training and out-of-distrution testing with respect to the other three datasets, e.g. a model trained on MNIST is tested on MNIST, FashionMNIST, CIFAR10 and SVHN.Baselines: We benchmark against MLP, STATNN [ Unterthiner et al., 2020] , NFN HP [Zhou et al., 2023a] , and the NFN HNP [Zhou et al., 2023a] .Results: We present the results for this task in Table 5 . Here we see that SNE again performs better than the competing methods significantly demonstrating strong transfer across different datasets for the same architecture. Qualitative Analysis: To understand how SNE transfers well across model zoos, we generate TSNE [ Van der Maaten and Hinton, 2008] plots for the neural network encodings of all benchmarked methods on all four homogeneous model zoos in Figure 2 . We provide 3 different views of each models embeddings to better illustrate the encoding pattern. In Figures 2c and 2d , we observe that NFN NP and NFN HNP have very clear separation boundaries between the networks from each model zoo. In Figures 2a and 2b , MLP and STATNN, respectively show similar patterns with small continuous strings of model zoo specific groupings. However, these separations are not as defined as those of NFN NP and NFN HNP . The embedding pattern of SNE on the other hand is completely different. In Figure 2e , all networks from all the model zoos are embedded almost uniformly close to each other. This may suggest why SNE performs much better on the cross-dataset performance prediction task since it is much easier to interpolate between the neural network encodings generated by SNE across model zoos. 
conclusion: 
 In this work, we tackled the problem of encoding neural networks for property prediction given access only to trained parameter values. We presented a Set-based Neural Network Encoder (SNE) that reformulates the neural network encoding problem as a set encoding problem. Using a sequence of set-to-set and set-to-vector functions, SNE utilizes a pad-chunk-encode pipeline to encode each network layer independently; a sequence of operations that is parallelizable across chunked layer parameter values. SNE also utilizes the computational structure of neural networks by injecting positionally encoder layer type/level information in the encoding pipeline. As a result, SNE is capable of encoding neural networks of different architectures as opposed to previous methods that only work on a fixed architecture. To learn the correct minimal equivariance for MLP weight permutations, we utilized Logit Invariance Regularization as opposed to weight tying used in previous methods. Experimentally, we introduced the cross-dataset and cross-architecture neural network property prediction tasks. We demonstrated SNE's ability to transfer well across model zoos of the same architecture but with networks trained on different datasets on the cross-dataset task. On the cross-architecture task, we demonstrated SNE's agnosticism to architectural choices and provided the first set of experimental results for this task that demonstrates transferability across architectures. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: All such details are provided in the experimental section and the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: All results have error bars with the exception of Table 3 where a single model is evaluated in accordance with the experimental setting described.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All datasets, code, etc are appropriately cited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In Section B we provide some limitations and future directions for the neural network encoding task. In Section C, we discuss weight tying versus regularization approaches to achieving minimal equivariance for MLPs together with some theoretical considerations. In Section D, we discuss how SNE can be applied to architectures with branches, specifically ResNets [He et al., 2016] . In Section E, we provide an ablation on the various components of the proposed method. In Section F, we provide additional experimental results on the cross-dataset and cross-architecture tasks. In Section G, we specify all the model architectures used for generating the model zoos of Arch 1 and Arch 2 used for the cross-dataset and cross-architecture tasks. In Section H we provide details of the train/validation/test splits. In Section I, we detail all the hyperparameters used for all experiments. We provide implementation details for SNE in Section J and enlarged versions of Figure 2 in Section K.In this work, we focused solely on the task of predicting the properties, specifically generalization performance, of neural networks given access only to the model parameters. While the task of encoding neural network weights is a relatively new topic of research with very few baselines, we anticipate new applications/research directions where the neural network encoding vector is used for tasks such as neural network generation or neural network retrieval. These tasks, potential applications, and consideration of more complicated neural network architectures (see discussion in Section D) are out of the scope of this paper and we leave it for future works.Additionally, we have not included results on larger architectures such as Transformers for the following reasons: Firstly, we are unaware of any such extensive model zoos for such architectures.In Lim et al. [2023] , such a model zoo is evaluated, however, this is not publicly available hence we are unable to benchmark against such architectures. Lastly, given that all experiments in this paper were carried out on a single GPU with 11GB of memory, we are unable to generate the extensive Transformer model zoos used in Lim et al. [2023] . However in Section D, we discuss how these architectures can be encoded with our method when such model zoos become available.With regards to achieving minimal equivariance: respecting only a subset of permutations in the weight space that are functionally equivalent, there are two approaches to satisfying this property.Weight Tying: This approach is adopted by Zhou et al. [2023a,b] , Navon et al. [2023] where minimal equivariance is baked into the neural network encoder, i.e., weight tying. However, this has a couple of drawbacks:• It requires a specification of the neural network to be encoded beforehand. This results in encoders that cannot be applied to different architectures as we demonstrated in Section 4.2 where at test time, we evaluate SNE on architectures not seen during training.• It requires processing all layers of the network together as opposed to the layer-wise encoding scheme that we adopt. Note that for large networks, Zhou et al. [2023a,b] , Navon et al.[2023] will require correspondingly large memory but the memory requirements of SNE remains constant since we share the same encoder across all layers, allowing our model to process arbitrary network architectures efficiently.Regularization: Recent works [Moskalev et al., 2023 , Cohen-Karlik et al., 2020 , Kim et al., 2023 , Miyato et al., 2022 , Otto et al., 2023] have shown that symmetric constraints such as permutation invariance/equivariance of a non-symmetric model can be achieved through regularization during training. In this work, we take this approach since it offers the following useful properties:• The level of symmetric constraints, i.e., minimal equivariance, can be learned directly from data without architectural constraints.• It allows us to process layers independently, resulting in a model that is general and disentangled from architectural choices as we demonstrated in the cross architecture experiments.In the Ablation study E, we show that the regularization approach indeed helps to learn a model that generalizes well and without it, the model struggles to learn the correct minimal equivariance.Theoretical Considerations A natural question that arises is how well the regularization approach learns the correct symmetric properties of interest. We first define the Logit Invariance, from which we obtain the Logit Invariance Regularization loss of Moskalev et al. [2023] .Definition C.1 (Logit Invariance) [Moskalev et al., 2023] For a given group G, dataset D and non-symmetric(with respect to G) model f , the Logit Invariance Loss of f is given by:EQUATION)Logit Invariance measures the sensitivity of f to the action of G on data samples in D.The following proposition of Moskalev et al. [2023] characterizes the performance decay of the Logit Invariance Regularizer under some linearity assumptions.Proposition C.1 (Invariance-Induced by Spectra Decay) [Moskalev et al., 2023] Logit invariance error minimization implies σ max (W (t)) ≤ σ(W (0)) when t → ∞.proof: See Section 3.3 of Moskalev et al. [2023] .Here, W are model parameters, t optimization steps and σ max (W ) = ||W || 2 . Proposition C.1, through an analysis of the training dynamics of f asserts that f will be insensitive to actions of G on data instances thereby learning the correct invariance property when the Logit Invariance Regularizer is minimized.In summary, the SNE model constrained by the Logit Invariance Regularizer, learns the correct minimal equivariance required for processing MLP weights without having to resort to weight tying as the methods of Navon et al. [2023] and Zhou et al. [2023a,b] .Here, we outline how SNE may be applied to a model zoo of architectures such as ResNets where branches exist in the computational graph. Given that residual blocks are composed of convolutional and linear layers, each of these can be encoded independently as we already do. To account for residual connections, we propose to introduce special tokens (just as was done for layer types) when we encode the entire block. Additionally, any symmetries inherent in such blocks can be enforced using the invariance regularization term introduced. Given that Transformers are composed of linear layers, the same pipeline can be applied to encode transformers and logit invariance regularization can be used to respect their inherent symmetries.While we do not see any impediment to applying SNE to such architectures, the unavailability of such model zoos makes it impossible to experimentally verify these and hence we leave it as future work when such model zoos become publicly available. 1 . As shown, SNE is more data efficient than the baseline models when the amount of training data is constrained.We investigate the impact of the invariant regularization loss, the hierarchical and positional encoding modules, the usage of set functions, layer level and layer type encoders on the performance of SNE using the INR dataset. In Table 6 , it can be seen that the hierarchical and positional encoding modules play an important role in the performance of SNE. Removing this module results in significant performance degradation from 0.098 to 7.452. Without positional encoding, SNE is fully permutation equivariant even to functionally incorrect permutations of the weights, making it difficult for the model to learn the correct restricted set of permutations. Secondly, removing the invariance regularization term results in a degradation in performance, from 0.098 to 0.160, as it also makes the task learning the correct minimal equivariance subgroup difficult. We find this only to have a huge impact for MLPs such as INRs where weight permutations are well defined. Additionally, removing the set functions and the layer level and type encoders from the model results in performance drop. We provide ablation on the effect of chunksize in the table below for the INR experiment presented in Table 1 of the main paper. From Table 7 we see that performance stays almost the same until the largest chunksize. Hence the chunksize is selected to suit the memory requirements.We compute the logit invariance loss for the experiments for 5 functionally equivalent permutations of INR weights and obtain a loss of approximately 10 -15 . This low loss implies that the SNE learns the correct level of invariance which is the desired property of the non-weight tying approach that we take. We provide a plot of training set size versus error in Figure 3 using the INR experiment presented in Table 1 . From this it can be seen that SNE is more data efficient compared to the baselines [Navon et al., 2023 , Zhou et al., 2023b ] across all percentages of the full training data demonstrating that the proposed method learns a good embedding even in the limited data setting.In this section we provide additional experimental results on the cross-dataset and cross-architecture tasks.For this task, we train neural network performance predictors on 4 homogeneous model zoos, of the same architecture, with each model zoo restricted to a single dataset.Datasets and Network Architecture: Each model zoo is trained on one of the following datasets: MNIST [Deng, 2012] , FashionMNIST [Xiao et al., 2017] , CIFAR10 [Krizhevsky, 2009] and SVHN [Netzer et al., 2018] . We use the model zoos provided by Unterthiner et al. [2020] . To create each model zoo, 30K different hyperparameter configurations were sampled. The hyperparameters include the learning rate, regularization coefficient, dropout rate, the variance and choice of initialization, activation functions etc. A thorough description of the model zoo generation process can be found in Appendix A.2 of Unterthiner et al. [2020] . Architectural descriptions for the model zoos are outlined in Appendix G. Each model zoo is split into a training, testing and validation splits.Task: In this task, we consider cross-dataset neural network performance prediction where we evaluate the prediction performance on the testset of the model zoo on which the predictors were trained on. Additionally, we evaluate how well each predictor transfers to the other model zoos. We evaluate all models using Kendall's τ [Kendall, 1938] .Results: We present the results of the cross-dataset performance prediction task in Table 8 . For each row in Table 8 , the first column shows the cross-dataset evaluation direction. For instance, MNIST→CIFAR10 implies that a model trained on an MNIST model zoo is cross evaluated on a model zoo populated by neural networks trained on CIFAR10. We note that the A→A setting, e.g. MNIST→MNIST, corresponds to the evaluation settings of Unterthiner et al. [2020] and Zhou et al. [2023a] . Also in Table 8 we show the best model in red and the second best model in blue. As show in Table 8 , SNE is always either the best model or the second best model in the cross-dataset task. SNE is particularly good in the A→B performance prediction task compared to the next competitive baselines, NFN NP and NFN HNP . The MLP baseline, as expected, performs the worse since concatenating all weight values in a single vector looses information such as the network structure. STATNN [Unterthiner et al., 2020] performs relatively better than the MLP baseline suggesting that the statistics of each layer indeed captures enough information to do moderately well on the neural network performance prediction task. NFN NP and NFN HNP perform much better than STATNN and MLP and NFN HNP in particular shows good results in the A→A setting. Interestingly, NFN NP is a much better cross-dataset performance prediction model than NFN HNP . However, across the entire cross-dataset neural network performance prediction task, SNE outperforms all the baselines as shown in the last row of Table 8 .Next we reverse the transfer direction from Arch 2 to Arch 1 in Section 4.2 and provide the results for SNE in Table 9 . Note that for this task NeuralGraph [Kofinas et al., 2024] cannot be used since we transfer from a smaller architecture to a larger one. As can be seen from Table 9 , SNE transfers well across architectures further validating the results in Section 4.2.We specify the architectures for generating the model zoos of Arch 1 and Arch 2 . For the model zoos of Arch 1 in Table 10 , all datasets with with 3 channel images (CIFAR10 and SVHN) are converted to grayscale. This is in accordance with the setups of Unterthiner et al. [2020] , Navon et al. [2023] and Zhou et al. [2023a,b] and allows us to evaluate both methods in the cross-dataset task for this set of homogeneous model zoos. For model zoos of Arch 2 in Tables 11 & 12 , we maintain the original channels of the datasets. The cross-architecture transfer task is from Arch 1 to Arch 2 . Note also that for the same dataset, i.e., CIFAR10, the cross-architecture evaluation is also from models trained grayscale to RGB images. All model zoos were generated using the procedure outlined in the Appendix of Unterthiner et al. [2020] . The architecture for the INR experiments is outlined in Table 16 . J Implementation Details SNE is implemented using Pytorch [Paszke et al., 2019] . The SNE model consists of 4 sub-modules:• Layer Chunk Encoder: This consists of two SAB modules where each SAB module is as stack of two SAB layers, followed by a single PMA layer. The layer chunk encoder encodes all the chunks of a given layer independently. • Layer Encoder: This module encodes all the encoded chunks of a layer and consists of two SAB modules and a single PMA layer. • Separated Layer Encoder: This module encodes all the encodings of a layer, for instance the weights and biases, into a single layer encoding vector. It also consists of two SAB modules and a single PMA layer. • NN Encoding Layer: This module takes as input all the layer encodings and compresses them to obtain the neural network encoding which is used for the downstream task of predicting the neural network generalization performance. It also consists of two SAB modules and a single PMA layer.In addition to the sub-modules above, the layer/level positional encoders are applied to each submodule when required (see Section 3). The neural network performance predictor, which takes as input the neural network encoding vector from SNE and predicts the performance is detailed in Table 15 .In Figures 4, 5 and 6 we provide enlarged versions of the figures in Figure 2 for better viewing. NeurIPS Paper Checklisthttps://github.com/mkofinas/neural-graphs 


Decision: The paper proposes to predict neural network training properties based on set-based representations of neural network parameters. There exist mixed reviews, but reviewers agree that the approach has novelty. The main complaint is comparisons with priors works including NeuralGraphs, as well as the findings of Papa et al., How to Train Neural Field Representations: A Comprehensive Study and Benchmark. In CVPR 2024. The authors make a fair argument that they are expecting a response from the authors before including the final results. I suggest the paper to be accepted under the condition that the include the more detailed comparisons with NeuralGraph that they currently miss, and discuss their findings compared to Papa et al.

Reviews: review: paper_summary: paper_summary: The study introduces Set-based Neural Network Encoding (SNE) without weight tying, allowing for encoding network information into a compact form. SNE utilizes Logit Invariance Regularization to ensure the correct functional equivalence of network permutations, departing from traditional weight tying methods. The choice of Set-to-Set and Set-to-Vector functions, including attention mechanisms and pooling layers, are crucial in implementing the SNE framework. Experimental results demonstrate the effectiveness of SNE in predicting generalization performance for Convolutional Neural Networks (CNNs) and Incremental Neural Networks (INRs). SNE is compared with various baselines such as MLP, Deepsets, HyperRep, and others, showcasing its superiority in certain aspects.
strength_weakness: strength_weakness: - The proposed method did not ensure weight-space equivariance by design but use a way that is more like an augmentation plus a consistency loss. The theoretic contribution is therefore a bit weak.

- The padding and chunking designs look tricky and it is not clear for me how they help with the performance.

- Fig 1 is not really clear. (1) the legend is not sufficient to understand the computation diagram presented in the figure. (2) the input is shown as neurons but from the method I think it should be the weight matrix that connecting neurons in adjacent layers.
questions: questions: - In table 1 NeuralGraph is not reported. Is it because the method is not applicable there?

- Does the method need more optimization steps than baseline methods that ensure weight-space symmetry by design? I suppose optimization would be harder because it requires to optimize for the logit invariance.
limitations: limitations: The limitations of the method are discussed in Appendix B
paper_summary: paper_summary: This work tackles an original and interesting challenge: predicting neural networks properties from their trained weight values. To do so, the authors propose to leverage set to set and set to vector transformations in order to encode weight values. Furthermore, they propose to account for the deep neural network architecture and operation order through the addition of positional encoding at several stages of the property predicting model. Ultimately, they anticipate the problem of various model sizes with a padding/chinking mechanism.
The proposed method is then evaluated on small scale datasets in a cross-dataset fashion against method of the field, published in recent years.
strength_weakness: strength_weakness: From my understanding of this work, I identified two major drawbacks:
1. the authors insist on the generalization across different model architectures and tasks. From my perspective, the former is the most important one. However, as it stands, there is little variation between Arch1 and Arch2 as presented in this study: both are simple CNNs. I wonder if the authors could provide numbers w.r.t transformer architectures against CNNs.
2. Since these experiments can be expensive (require multiple training of DNNs to create the predictor training set), it would be interesting to measure the impact of the dataset size w.r.t the predictor's performance, in order to estimate how far the community is from being to reasonably work on large scale datasets such as ImageNet.

As it stands, I believe this work aims at providing empirical results which are not fully convincing yet. I look forward to the authors' response and other reviews and will update my rating accordingly.
questions: questions: I summarize the previous points:
1. could the authors provide results with a predictor trained on CNNs from arch1 and predict the performance of transformers?
2. could the author measure the importance of the training set size (pairs of DNNs and accuracy) w.r.t. the predictor's performance, in order to estimate the cost of working on ImageNet-like tasks?
limitations: limitations: The authors made a relevant attempt at highlighting the limitations of their work
paper_summary: paper_summary: This work introduces a set-based neural network encoding that processes each chunk of an input model independently to predict network properties. The proposed model, SNE, is trained using Logit Invariance instead of weight tying to maintain generalizability to unseen architectures. Unlike previous approaches, SNE can be trained and tested on different architectures and it demonstrated superior performance in out-of-distribution experiments involving both datasets and architectures.
strength_weakness: strength_weakness: While SNE shows improvements in novel tasks such as cross-architecture and cross-dataset scenarios, its comparison on traditional tasks is limited to predicting INR frequencies. Expanding Table 1 to include SNE's performance on additional outputs, such as accuracy, would enhance the paper.
questions: questions: The writing is generally good but could be improved in certain areas. For example, the abstract contains a redundant sentence: “by utilizing a layer-wise encoding scheme that culminates to encoding all layer-wise encodings to obtain the neural network encoding vector,” which should be revised.
limitations: limitations: Limitations were properly addressed in appendix.
paper_summary: paper_summary: This paper proposes SNE, or Set-based Network Encoding, a method of encoding the weights of arbitrary neural networks in order to predict properties such as performance, generalization gap, training loss, number of epochs, etc.  Specifically, using a chunking mechanism as well as layer-wise and block-wise positional encodings to generalize across different types of neural networks. Transferability experiments are conducted where a predictor is trained on one network model zoo, then evaluate on another, demonstrating an ability few predictors have. Additional experiments and ablations are performed, producing convincing results. Finally, the paper is decently well-written, but has a few fixable presentation flaws.
strength_weakness: strength_weakness: - First, the motivation of this paper, specifically lines 29-30, is quite weak and fails to really provide examples of the application of this research other than asserting that they must exist.
- Overall, the presentation is a weak spot, and should be revised. Some examples: 
	- E.g., in line 31, the authors mention "Implicit Neural Representations (INR)" - what are these? No citation is provided. By contrast, "b) the performance of CNNs" is very easy to understand.
	- "modelzoo", this should be "model-zoo" [1] or "model zoo" [2-4].
	- Citations in the paper have issues. For NeurIPS its generally just [num], or if you want to be fancier, "Author et al., Year [num]" but not "Author et al. [num]" without the year.
	- Float captions need work, e.g., Tabs 2/3 need longer captions to fully explain the performance metric Kendall's Tau, etc. Should be longer than 1 sentence.
	- Figure 2 needs work. The plots should be bigger and legend is clipped.
	- NeurIPS checklist should be placed after the references and supplementary.
- The evaluation is done on simpler datasets/networks, however, the limitations section highlights why.
questions: questions: I find the idea of predicting end-to-end neural network attributes from the weight values to be quite fascinating. It is very similar to the concept of a neural predictor [5] in Neural Architecture Search [6] but quite orthogonal in the approach, relying on the weights instead of the graph structure. Similarly, most neural predictors suffer from a lack of generalizability/transferability [7, 8], like most of the baselines in this paper. Can the authors provide some commentary/comparison between these two umbrella fields? 

Similarly, is it possible to leverage the encoding mechanism (Eqs. 1-9) to derive which types of weights/layers/values are responsible for certain end-to-end attributes, e.g., performance of epochs, e.g., as [9] do?

Refs:

[1] "Equivariant Architectures for Learning in Deep Weight Spaces" - ICML'23.

[2] https://modelzoo.co/

[3] https://github.com/onnx/models

[4] https://pytorch.org/serve/model_zoo.html

[5] "How Powerful are Performance Predictors in Neural Architecture Search?" - NeurIPS'21.

[6] "Efficient Neural Architecture Design via Capturing Architecture-Performance Joint Distribution" - AISTATS'24.

[7] "GENNAPE: Towards Generalized Neural Architecture Performance Estimators" - AAAI-23.

[8] "Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor" - NeurIPS'22.

[9] "Building Optimal Neural Architectures using Interpretable Knowledge" - CVPR'24.
limitations: limitations: Yes, limitations and future work is discussed in App. B. The authors explain that their experiments are on simpler/smaller/older networks as they have limited computational resources. I find this to be an acceptable justification.

Comments: title: Discussion
Discussion
5->6
Response to rebuttal
comment: We thank the Reviewer for the feedback.
Thanks for the response. 

I still have concerns about not directly conducting the experiments on the most typical benchmark (MNIST and CIFAR image INRs) established in many prior papers in this field. From my own experience (and may also be suggested by observations from [A]), if the training of the INRs is harder (takes longer steps of optimization to converge), then it is likely more challenging for downstream weight-space tasks as well. Sine wave INRs are not sufficient as a replacement in my view, because they look too simple and toy-ish. Without checking the performance on the most common benchmark, I am not sure if the evaluation is convincing enough.

I see the values of this work including the new benchmark. However aggregating the experimental evaluation and the innovations introduced in the method, I tend to keep a borderline score.

[A] Papa et al. How to Train Neural Field Representations: A Comprehensive Study and Benchmark. In CVPR 2024
Dear Reviewer,
We thank you for taking the time to provide feedback. As the discussion period draws to an end, we would like to know if there are any further queries you'd like as to respond to.

Thank you for your time.
We would like to thank the Reviewer again and respond to the new queries below:

**I wonder what the issue was with NeuralGraph on the experiments in Table 1 and Table 4? If the NNs for the INRs are MLP architecture, I do not expect any issue there since NeuralGraph is able to work on classification of MNIST / CIFAR in INRs.**


- Firstly, we utilize the official implementation of NeuralGraphs provided at [1].
In Table 1 we are unable to get NeuralGraphs to converge on this task using multiple hyper-parameter runs.  
Secondly, the experiments of Table 4 are on ConvNets (the architecture is outlined in Tables 10 and 11 of the appendix. In Table 4, we evaluate in terms of cross-dataset performance prediction and the issue with NeuralGraph, under this evaluation setting, lies in the following: we are unable to obtain positive correlation coefficients when we evaluate it cross-dataset. As we elaborated in our response to Reviewer bv1s,  in our correspondences with the authors of NeuralGraph, we were unable to find and resolve the issues with negative correlation values in Table 4. Interestingly, we do not observe these issues in our evaluation of NeuralGraph in Table 2.
These explain the absence of NeuralGraphs in Tables 1 and 4.


**This should be a more high-level and sophisticated task than sine wave frequencies, and direct comparison with performance reported by prior work would be more convincing.**

- As we elaborated in our response to Reviewer EgRw, there is no fundamental difference between the task of INR frequency prediction and INR classification as these require the same guarantees in MLP weight space. Our request to [1] for the dataset of INRs for the ShapeNet dataset is currently pending and this task will be added to Table 1 when the dataset is received together with INR classification (a task which we are currently undertaking but do not expect to be completed during this discussion period. This is partly due to the large datasets required for for training INR classification for MNIST, FashionMNIST and CIFAR: all of DWSNet, NFT, NeuralGraph generate up to a million datapoints as opposed to the 60000 samples in the original MNIST classification task, resulting in long training iterations).
Importantly, we point the Reviewers attention to the following: that all the experiments presented in Table 2, 3,4 and 7  of the appendix together with the newly introduced Transformer evaluation task suggested by Reviewer NYWS are much more difficult tasks compared to classifying INRs since these  require respecting symmetric properties in both CNN (with MLP heads) and Transformer weight spaces. We hope this provides some clarity. 

**What is the NN architecture tested for the chunksize experiments?**

- We utilize the architecture outlined in Table 15 of the appendix which consists of 3 linear layers each with hidden size of 32 and the first two followed by sinusoidal activation function. This is the architecture for generating the model zoo used in the experiments of Table 1.


Finally, we would like to thank the Reviewer for the continued engagement as both the discussions and our response to the original review concerns as well as the questions we’ve presently addressed greatly improves clarity of the paper and these will be incorporated in the paper.

**References**

[1]https://github.com/mkofinas/neural-graphs/tree/1f2b671ab4988ef212469363005a5b99eec16580

[2] De Luigi, Luca, et al. "Deep learning on implicit neural representations of shapes." arXiv preprint arXiv:2302.05438 (2023).
We would like to thank the Reviewer for the continued engagement. As requested, the additional experimental results as well as the revised related work, limitations and discussions of potential applications and similarities to NAS research will be incorporated in the paper. Thank you.
I thank the authors for their detailed rebuttal, which satisfactorily addressed my concerns and questions. After reading the other reviewer comments and corresponding rebuttals I find that the reasons to accept outweigh concerns to reject, and thus raise my score. If accepted, I encourage the authors to integrate rebuttal experimental results as well as revise related work and limitations to reflect these past discussions.
Thanks for the response.

- I wonder what the issue was with NeuralGraph on the experiments in Table 1 and Table 4? If the NNs for the INRs are MLP architecture, I do not expect any issue there since NeuralGraph is able to work on classification of MNIST / CIFAR in INRs.   

- The experiments on INRs now are on frequency predictions of INRs fitting simple sine waves, which is pretty low-level and toy-ish. I have the concerns about not following the conventions in previous weight space symmetry methods (DWSNet, NFT, NeuralGraphs, etc.) to test INR classification tasks on MNIST, FashionMNIST and CIFAR. This should be a more high-level and sophisticated task than sine wave frequencies, and direct comparison with performance reported by prior work would be more convincing.

- What is the NN architecture tested for the chunksize experiments?
I would like to thank the authors for their response and the new experimental results they provided for this rebuttal.
Overall, I consider that my concerns have been addressed. Thus, I am likely to increase my rating for this work after the end of the discussion period.