Title: Set-based Neural Network Encoding Without Weight Tying
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Network Encoding
Abstract: We propose a neural network weight encoding method for network property prediction that utilizes set-to-set and set-to-vector functions
to efficiently encode neural network parameters. Our approach is capable of encoding neural networks in a model zoo of mixed architecture and different parameter sizes as opposed to previous approaches that require custom encoding models for different architectures. Furthermore, our \textbf{S}et-based \textbf{N}eural network \textbf{E}ncoder (SNE) takes into consideration the hierarchical computational structure of neural networks. To respect symmetries inherent in network weight space, we utilize Logit Invariance to learn the required minimal invariance properties. Additionally, we introduce a \textit{pad-chunk-encode} pipeline to efficiently encode neural network layers that is adjustable to computational and memory constraints. We also introduce two new tasks for neural network property prediction: cross-dataset and cross-architecture. In cross-dataset property prediction, we evaluate how well property predictors generalize across model zoos trained on different datasets but of the same architecture. In cross-architecture property prediction, we evaluate how well property predictors transfer to model zoos of different architecture not seen during training. We show that SNE outperforms the relevant baselines on standard benchmarks.

Set-based Neural Network Encoding Without Weight Tying

Abstract

We propose a neural network weight encoding method for network property prediction that utilizes set-to-set and set-to-vector functions to efficiently encode neural network parameters. Our approach is capable of encoding neural networks in a model zoo of mixed architecture and different parameter sizes as opposed to previous approaches that require custom encoding models for different architectures. Furthermore, our S et-based N eural network E ncoder (SNE) takes into consideration the hierarchical computational structure of neural networks. To respect symmetries inherent in network weight space, we utilize Logit Invariance to learn the required minimal invariance properties. Additionally, we introduce a pad-chunk-encode pipeline to efficiently encode neural network layers that is adjustable to computational and memory constraints. We also introduce two new tasks for neural network property prediction: cross-dataset and cross-architecture. In cross-dataset property prediction, we evaluate how well property predictors generalize across model zoos trained on different datasets but of the same architecture. In cross-architecture property prediction, we evaluate how well property predictors transfer to model zoos of different architecture not seen during training. We show that SNE outperforms the relevant baselines on standard benchmarks.

1 Introduction

Recently, deep learning methods have been applied to a wide range of fields and problems. With this broad range of applications, huge volumes of datasets are continually being made available in the public domain together with neural networks trained on these datasets. Given this abundance of trained neural network models, the following curiosity arises: what can we deduce about these networks with access only to the parameter values? More generally, can we predict properties of these networks such as generalization performance on a testset(without access to the test data), the dataset on which the model was trained, the choice of optimizer and learning rate, the number of training epochs, choice of model initialization etc. through an analysis of the model parameters? The ability to infer such fundamental properties of trained neural networks using only the parameter values has the potential to open up new application and research paradigms [De Luigi et al., 2023, Zhou et al., 2023a,b, Navon et al., 2023] such as learning in the latent space of neural network weights for tasks such as weight generation [Schürholt et al., 2021, Soro et al., 2024], latent space transfer of weights across datasets allowing for transferring weights from one dataset to another as was recently demonstrated in Soro et al. [2024] and latent space optimization using gradient descent where optimization is performed on the weight embeddings [Rusu et al., 2018].

We tackle two specific versions of this problem: predicting a) frequencies of Implicit Neural Representations [Sitzmann et al., 2020] (INRs), and b) the performance of CNNs and Transformers,

PE

PE

>0.92

PE

PE

X

given access only to the network weights. The first approach to solving this problem, proposed by Unterthiner et al. [2020], involves computing statistics such as the mean, standard deviation and quantiles, of each layer in the network, concatenating them to a single vector that represents the neural network encoding, and using this vector to predict the desired network property. Another approach, also proposed as a baseline in Unterthiner et al. [2020], involves flattening all the parameter values of the network into a single vector which is then fed as input to layers of multilayer perceptrons (MLPs) to predict the property of interest. An immediate consequence of this approach is that it is practical only for moderately sized neural network architectures. Additionally, this approach ignores the hierarchical computational structure of neural networks through the weight vectorization process. The second, and most recent approach to this problem, proposed by Zhou et al. [2023a,b], takes a geometric approach to the problem by building neural network weight encoding functions, termed neural functionals, that respect symmetric properties of permutation invariance and equivariance of the hidden layers of MLPs under the action of an appropriately applied permutation group. While this approach respect these fundamental properties in the parameter space, it's application is restricted, strictly, to MLPs. Also, even when relaxations are made to extend this method to convolutional networks and combinations of convolutional layers and MLPs, these only work under strict conditions of equivalence in the channel size in the last convolutional layer and the first linear layer. Hence it is clear that while the methods of Zhou et al. [2023a,b] and Navon et al. [2023] enjoy nice theoretical properties through weight tying, their application is limited to only a small subset of carefully chosen architectures.

Moreover, these approaches [Unterthiner et al., 2020, Zhou et al., 2023a,b, Navon et al., 2023] have a fundamental limitation: their encoding methods are applicable only to a single fixed, pre chosen neural network architecture. Once the performance predictor is trained, in the case of Unterthiner et al. [2020], and the neural network encoder of Zhou et al. [2023a] and Navon et al. [2023] are defined, they cannot be used to predict the performance of neural networks of different architecture. These issues are partly addressed by the graph based approaches of Kofinas et al. [2024] and Lim et al. [2023]. Consequently, evaluating these models on diverse architectures is impossible without training an new performance predictor for each architecture.

To this end, we propose a Set-based Neural Network Encoder (SNE) for property prediction of neural networks given only the model parameters that is agnostic to the network architecture without weight tying. Specifically, we treat the neural network encoding problem from a set encoding perspective by utilizing compositions of set-to-set and set-to-vector functions. However, the parameters of neural networks are ordered and hierarchical. To retain this order information, we utilize positional encoding [Vaswani et al., 2017] at various stages in our model. Also, our model incorporates the hierarchical computational structure of neural networks in the encoder design by encoding independently, layer-wise, culminating in a final encoding stage that compresses all the layer-wise information into a single encoding vector used to predict the network property of interest. To handle the issue of large and variable parameter sizes efficiently, we incorporate a pad-chunk-encode pipeline that is parallelizable and can be used to iteratively encode layer parameters. To learn the correct permutations of MLP weights, we employ the Logit Invariance Regularizer of Moskalev et al. [2023]

instead of weight tying. In terms of evaluation, we introduce two new tasks: cross-dataset neural network property prediction and cross-architecture neural network property prediction. In crossdataset neural network performance prediction, we fix the network architecture used to generate the training data and evaluate how well the performance predictors transfer to the same architecture trained on different datasets. For cross-architecture neural network performance prediction, we fix only the architecture for generating the training data and evaluate the performance of the predictor on architectures unseen during training. These tasks are important since in the real setting, access to model zoos is scare, making transferability desirable.

Our contributions are as follows:

We develop a Set-based Neural Network Encoder (SNE, see Figure 1) for network property prediction given access only to parameter values that can encode neural networks of arbitrary architecture, taking into account the hierarchical computational structure of neural networks.

We introduce the cross-dataset property prediction task where we evaluate how well predictors transfer across neural networks trained on different datasets.

We introduce the cross-architecture property prediction task where we evaluate how well property predictors trained on a specific architecture transfer to unseen architectures.

We benchmark SNE against the relevant baselines [Unterthiner et al., 2020, Navon et al., 2023, Zhou et al., 2023a,b] on the cross-dataset task and show significant improvement over the baselines.

We provide the first set of results on the cross-architecture task, a task for which most of the baselines, with the exception of Zaheer et al. [2017] and Kofinas et al. [2024] under special conditions (see remark in Section 4.2), cannot be used.

2 Related Work

Set Functions: Neural networks that operate on set (un)structured data have recently been used in many applications ranging from point cloud classification to set generation [Kim et al., 2021]. Set functions are required to respect symmetric properties such as permutation invariance and equivariance. In DeepSets [Zaheer et al., 2017], a set of sum-decomposable functions are introduced that are equivariant in the Set-to-Set applications and invariant in the Set-to-Vector applications. In Set Transformers [Lee et al., 2019], a class of attention based Set-to-Set and Set-to-Vector functions are introduced that are more expressive and capable of modeling pairwise and higher order interactions between set elements. Recent works such as Bruno et al. [2021] and Willette et al. [2023] deal with the problem of processing sets of large cardinality in the the limited memory/computational budget regime. In this work, we utilize the class of set functions developed in Lee et al. [2019] to develop a neural network encoder for performance prediction that is agnostic to specific architectural choices. Our set-based formulation allows us to build such an encoder, capable of handling neural networks weights of arbitrary parameter sizes. This is a deviation from recent approaches to neural network encoding for property prediction that can encode only parameters of a single fixed architecture.

Neural Network Property Prediction From Weights: Predicting the properties of neural networks given access only to the trained parameters is a relatively new topic of research introduced by Unterthiner et al. [2020]. In Unterthiner et al. [2020], two methods are proposed for predicting the generalization performance of neural networks: the first involves flattening the weights of the network into a single vector, processing it using multiple layers of MLPs to obtain an encoding vector which is then used to predict the performance. The second involves computing the statistics of each layer in the network, such as mean, variance, quantiles etc., concatenating them into a single vector that is then used for predicting the performance of the network. The most recent approach that we are aware of, Navon et al. [2023] and Zhou et al. [2023a,b], proposes a neural network weight encoder that is invariant or equivariant, depending on the application, to an appropriately applied permutation group to the hidden layers of MLPs. Two variants of their model is provided: one which operates only on the hidden layers, and conforms strictly to the theory of permuting MLP hidden neurons [HechtNielsen, 1990], and a relaxation that assumes that the neurons of both the input and output layers of MLPs are permutable. Additionally, extensions are provided for convolutional layers. Kofinas et al. [2024] and Lim et al. [2023] represent weights using graph neural networks. Our approach, SNE, is directly comparable to these methods on the neural network property prediction task. However,

unlike the methods of Unterthiner et al. [2020], Navon et al. [2023] and Zhou et al. [2023a] which operate only on neural networks of fixed architecture, and consequently fixed number of encodable parameters, SNE is capable of encoding networks of arbitrary architecture. Moreover, SNE utilizes the hierarchical computation structure of neural networks by encoding, iteratively or in parallel, from the input to the output layers. Additionally, we go further than the experimental evaluation in Unterthiner et al. [2020], Navon et al. [2023] and Zhou et al. [2023a,b] by introducing two new tasks: cross-dataset and cross-architecture neural network property prediction. Unterthiner et al. [2020], Navon et al. [2023] and Zhou et al. [2023a,b] can only be benchmarked on the cross-dataset task where all networks in the model zoos are of the same architecture. Their restriction to a single fixed architecture makes cross-architecture evaluation impossible. Our method SNE, and those of Kofinas et al. [2024], Lim et al. [2023], on the other hand can be used for both tasks.

3 Set-based Neural Network Encoding Without Weight Tying

3.1 Preliminaries

We have access to a dataset D = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } where for each ( x i , y i ) pair, x i represents the weights of a neural network architecture a , sampled from a set of architectures A and y i corresponds to some property of x i after it has been trained on a specific dataset d . y i can be properties such as generalization gap, training loss, the learning rate used to train x i , or even the number of epochs, choice of weight initialization, and optimizer used to train x i . Henceforth, we refer to D as a model zoo . For each x i ∈ D , x i = [ w 0 i , . . . , w | x i | i ] where w j i represents the weights (parameters) of the jth layer of the neural network x i , and | x i | is the total number of layers in x i . Consequently, w 0 i and w | x i | i are the input and output layers of x i respectively. Additionally, we introduce the Flat : x i → R d i operation, that takes as input the weights of a neural network and returns the flattened weights and d i is the total number of parameter is x i .

The neural network encoding problem is defined such that, we seek to compress x i ∈ R d i to a compact representation z x i ∈ R h such that z x i can be used to predict the properties y i of x i with h ≪ d i . In what follows, we present the details of our Set-based Neural Network Encoding (SNE) method capable of encoding the weights of neural networks of arbitrary architecture that takes into account the hierarchical computational structure of the given architecture and with efficient methods for processing weights of high dimension.

3.2 Handling Large Layer Weights via Chunking

For a given layer w j i ∈ x i , the dimension of w j i , | w j i | can be very large. For instance, when considering linear layers, flattening the weights can results in a tensor that can require large compute memory to be processable by another neural network. To resolve this issue, we resort to chunking . Specifically, for all layers w j i ∈ x i , we perform the following operations:

where for any w j t i ∈ ˆ w j i , | w j t i | ∈ R c . Here, c is the chunksize , fixed for all layer types in the neural network and t ∈ [0 , . . . , q ] . The padding operation Pad ( w j i , c ) appends zeros, if required, to extend w j i and make its dimension a multiple of the chunksize c . To distinguish padded values from actual weight values, each element of ˆ w j i has a corresponding set of masks ˆ m j i = { m j 0 i , . . . , m j q i } . Note that with this padding and subsequent chunking operation, each element in ˆ w j i is now small enough, for an appropriately chosen chunksize c , to be processed. Moreover, all the elements in ˆ w j i can be processed in parallel. Importantly, chunksizes are chosen to ensure that weights from the same neurons are grouped together. This allows for principled learning of specific symmetries which we discuss later in Section 3.6. An ablation on the effect of chunksize is provided in Appendix E.

The model zoos we consider in the experimental section are populated by neural networks with stacks of convolutional and linear layers. For each such layer, we apply the padding and chunking operation differently. For a linear layer w j i ∈ R out × in , where out and in are the input and output dimensions respectively, we apply the flattening operation on both dimensions followed by padding and chunking. However for a convolutional layer w j i ∈ R out × in × k × k , we do not apply the flattening, padding, and

chunking operations to the kernel dimensions k and operate only on the input and output dimensions since the kernels are small enough to be encoded together. Finally we note that for layers with bias values, we apply the procedure detailed above independently to both the weights and biases.

3.3 Independent Chunk Encoding

The next stage in our Set-based Neural Network encoding pipeline is the individual encoding of each chunk of weight in ˆ w j i = { w j 0 i , . . . , w j t i } . For each w j t i ∈ ˆ w j i , we treat the c elements as members of a set. However, it is clear that w j t i has order in its sequence, i . e ., an ordered set. We remedy this by providing this order information via positional encoding. Concretely, for a given w j t i ∈ R c × 1 , we first model the pairwise relations between all c elements using a set-to-set function Φ θ 1 to obtain:

That is, Φ θ 1 captures pair-wise correlations in w j t i and projects all elements (weight values) to a new dimension h .

Given ˆ w j t i ∈ R c × h , we inject two kinds of positionally encoded information. The first encodes the layer type in a list of layers, i . e ., linear or convolution for the model zoos we experiment with, to obtain:

Here we abuse notation and assign the output of PosEnc ( · ) to ˆ w i t i to convey the fact that ˆ w j t i 's are modified in place and to simplify the notation. Also, all PosEnc ( · ) s are variants of the positional encoding method introduced in Vaswani et al. [2017]. Next we inject the layer level information. Since neural networks are computationally hierarchical, starting from the input to the output layer, we include this information to distinguish chunks, w j t i s from different layers. Specifically, we compute:

where the input to PosEnc Level Layer ( · ) is the output of Equation 3. We note that this approach is different from previous neural network encoding methods [Unterthiner et al., 2020] that loose the layer/type information by directly encoding the entire flattened weights hence disregarding the hierarchical computational structure of neural networks. Experimentally, we find that injecting such positionally encoded information improves the models performance (Ablation E).

We further model pairwise correlations in ˆ w j t i , now infused with layer/type information, using another set-to-set function Φ θ 2 :

The final step in the chunk encoding pipeline involves compressing all c elements in ˆ w j t i to a compact representation. For this, we use a set-to-vector function Γ θ α : R c × h → R h . In summary, the chunk encoding layer computes the following function:

where ˜ w j t i ∈ R 1 × H . Note that for each chunked layer ˆ w j i = { w j 0 i , . . . , w j q i } , the chunk encoder, Equation 6, produces a new set ˜ w j i = Concatenate [ { ˜ w j 0 i , . . . , ˜ w j q i } ] ∈ R q × h , which represents all the encodings of all chunks in a layer.

Remark Our usage of set functions Φ θ 1 , Φ θ 2 and Γ θ α allows us to process layers of arbitrary sizes, which allows us to process neural networks of arbitrary architecture using a single model, a property lacking in previous approaches to neural network encoding [Zhou et al., 2023a,b, Unterthiner et al., 2020, Navon et al., 2023].

3.4 Layer Encoding

At this point, we have encoded all the chunked parameters of a given layer to obtain ˜ w j i . Encoding a layer, w j i , then involves defining a function Γ θ β : R q × h → R 1 × h for arbitrary q . In practice, this is done by computing:

Again we have injected the layer level information, via positional encoding, into the encoding processed by the set-to-set function Φ θ 3 . We then collect all the layer level encodings of the neural network x i :

3.5 Neural Network Encoding

With all layers in x i encoded, we compute the neural network encoding vector z x i as follows:

z x i compresses all the layer-wise information into a compact representation for the downstream task. Since Γ θ γ is agnostic to the number of layers | x i | of network x i , the encoding mechanism can handle networks of arbitrary layers and by extension architecture. Similar to the layer encoding pipeline, we again re-inject the layer-level information through positional encoding before compressing with Γ θ γ .

Henceforth, we refer to the entire encoding pipeline detailed so far as SNE Θ ( x i ) for a network x i , where Θ encapsulates the encoder parameters, Φ θ 1 -4 , Γ α , Γ β and Γ γ .

3.6 On Minimal Equivariance Without Weight Tying

Given an MLP, there exists permutations of the weights such that the networks are functionally equivalent [Hecht-Nielsen, 1990]. Since not all permutations are functionally correct, the encoder needs to learn the correct functional equivalence. To achieve this, we utilize the concept of Logit Invariance Regularization [Moskalev et al., 2023] where we constrain the output of the non-equivariant SNE Θ (due to the positional encoding of input and output vectors) to respect the restricted functionally correct permutation group. This results in the following optimization problem:

where G is the group of functionally equivariant permutations in the weight space, D is the training dataset and v balances the task loss ℓ f ( D ) and the Logit Invariance Regularization term ℓ f ( D,G ) . Proposition 3.1 of Moskalev et al. [2023] guarantees that the resulting SNE model will have low sensitivity to functionally incorrect permutations of the weights. In practice, ℓ f ( D,G ) is the L 2 distance between functionally equivalent permutations of the same weight. This approach differs from previous works [Zhou et al., 2023a,b, Navon et al., 2023] which instead result to weight tying to achieve minimal equivariance.

Remark While we use a regularization approach to achieve the required approximate minimal equivariance in weight-space, our usage of the Logit Invariance Regularizer [Moskalev et al., 2023] theoretically guarantees that we indeed learn the correct invariance property similar to the weighttyping approaches. Additionally, our formulation is what allows us to deal with arbitrary architectures using a single model (see Section 4), as opposed to previous works, since strict enforcement of weight-space equivariance by design requires crafting a new model for different architectures. In this sense, our approach provides a general encoder which in principle is applicable to any architecture, resolving the limitations of purely weight-tying approaches.

Theoretical discussions on the adopted regularization based approach to minimal equivariance versus weight tying is provided in Appendix C.

3.7 Choice of Set-to-Set and Set-to-Vector Functions

We specify the choice of Set-to-Set and Set-to-Vector functions encapsulated by Φ θ 1 -4 , Γ α , Γ β and Γ γ used to implement SNE. Let X ∈ R n X × d and Y ∈ R n Y × d be arbitrary sets where n X = | X | , n Y = | Y | and d (note the abuse of notation from Section 3.1 where d is a dataset) is the dimension of an element in both X and Y . The MultiHead Attention Block (MAB) with parameter ω is given by:

Here, LayerNorm and rFF are Layer Normalization [Ba et al., 2016] and row-wise feedforward layers respectively. MultiHead ( X,Y,Y ; ω ) is the multihead attention layer of Vaswani et al. [2017].

The Set Attention Block [Lee et al., 2019], SAB, is given by:

That is, SAB computes attention between set elements and models pairwise interactions and hence is a Set-to-Set function. Finally, the Pooling MultiHead Attention Block [Lee et al., 2019], PMA k , is given by:

S ∈ R k × d and X ∈ R n X × d . The k elements of S are termed seed vectors and when k = 1 , as is in all our experiments, PMA k pools a set of size n X to a single vector making it a Set-to-Vector function.

All parameters encapsulated by Φ θ 1 -4 are implemented as a stack of two SAB modules: SAB ( SAB ( X )) . Stacking SAB modules enables us not only to model pairwise interactions but also higher order interactions between set elements. Finally, all of Γ α , Γ β and Γ γ are implemented as a single PMA module with k = 1 .

3.8 Downstream Task

Given ( z x i , y i ) , we train a predictor f θ ( z x i ) to estimate properties of interest of the network x i . In this work, we focus solely on the task of predicting the generalization performance of x i , where y i is the performance on the test set of the dataset used to train x i for CNNs and frequencies for INRs. The parameters of the predictor f θ and all the parameters in the neural network encoding pipeline, Θ , are jointly optimized. In particular, we minimize the error between f θ ( z x i ) and y i . For a model zoo, the objective is given as:

for an appropriately chosen loss function ℓ ( · ) . In our experiments, ℓ ( · ) is the binary cross entropy or mean squared error loss. The entire SNE pipeline is shown in Figure 1.

4 Experiments

We present experimental results on INRs, and the standard CNN benchmark model zoos used in Unterthiner et al. [2020],Zhou et al. [2023a],Zhou et al. [2023b], and Navon et al. [2023]. Experimental settings, hyperparameters, model specification, ablation of SNE and discussions on applying SNE to architectures with branches ( e . g . ResNets) in Appendix D.

Baselines: We compare SNE with the following baselines: a) MLP: This model flattens the entire weight of the network and encodes it using a stack of MLP layers. b) DeepSets: [Zaheer et al., 2017] This model treats the weights as a set with no ordering. c) HyperRep: [Schürholt et al., 2021] This model learns a generative model of the flattened weight vector. d) STATNN [Unterthiner et al., 2020] : This model computes the statistics of each layer such as the mean, variance and quantiles, and concatenates them to obtain the neural network encoding vector. e) DWSNet [Navon et al., 2023] : is a minimally equivariant model using weight tying developed mainly for MLPs. Various modules are provided for encoding biases, weights and combinations of these two. f) NFNHNP, NFNNP and NFT [Zhou et al., 2023a,b] : These models, termed Neural Functionals(NF), are developed mainly for MLPs and utilize weight tying to achieve minimal equivariance. HNP, hidden neural permutation, is applied only to the hidden layers of each network since the output and input layers of MLPs are not invariant/equivariant to the action of a permutation group on the neurons. NP, neural permutation, makes a strong assumption that both the input and output layers are also invariant/equivariant under the action of a permutation group. NFT is similar to both models and utilizes attention layers. g) NeuralGraph: [Kofinas et al., 2024] This method represents the weights of a network as a graph and uses graph pooling techniques to obtain the network representation vector. We note that we are unable to benchmark against Graph Metanetworks [Lim et al., 2023], which uses a graph approach similar to NeuralGraph, since no code or data is publicly available.

In all Tables, the best methods are shown in red and the second in blue. Additionally an extensive ablation of all the components of SNE is provided in Appendix E. All experiments are performed with a single GeForce GTX 1080 TI GPU with 11GB of memory.

4.1 Encoding Implicit Neural Representations

Dataset and Network Architecture: We utilize the model zoo of Navon et al. [2023] consisting of INRs [Sitzmann et al., 2020] fit to sine waves on [ -π, π ] with frequencies sampled from U (0 . 5 , 10) . INRs are neural parameterizations of signals such as images using multi-layer perceptrons.

Task: The goal is to predict the frequency of a given INR. Each INR is encoded to a 32 dimensional vector which is then fed to a classifier with two linear layers of dimension 512.

Results: As can be seen in Table 1, SNE significantly outperforms the baselines on this task. Given that INRs are MLPs, minimal equivariance is particularly important for this task and shows that SNE learns the correct minimal equivariance required to solve the task using the logit invariance approach. Additionally, compared to the minimal equivariance constrained models [Zhou et al., 2023a,b, Navon et al., 2023], SNE is parameter efficient as show in Table 1. We note that increasing the parameter counts of the MLP, DeepSets and STATNN baselines results in overfitting and poor performance.

4.2 Cross-Architecture Performance Prediction

For this task, we train the encoder on 3 homogeneous model zoos of the same architecture and test on 3 homogeneous model zoos of a different architecture unseen during training. The cross-architecture task demonstrates the encoder's agnosticism to particular architectural choices since training and testing are done on model zoos of different architectures, i . e ., we perform out-of-distribution evaluation.

Datasets and Neural Network Architectures:

We utilize model zoos trained on MNIST, CIFAR10 and SVHN datasets. We generate a model zoo for these dataset with an architecture consisting of 3 convolutional layers followed by two linear layers and term the resulting model zoo Arch 1 . Exact architectural specifications are detailed in Appendix G. We generate the model zoos of Arch 2 following the routine described in Appendix A.2 of Unterthiner et al. [2020]. We refer to the model zoos of Unterthiner et al. [2020] as Arch 2 . All model zoos of Arch 1 are used for training and those of Arch 2 are used for testing and are not seen during training.

Task: Here, we seek to explore the following question: Do neural network performance predictors trained on model zoos of Arch 1 transfer or generalize to the out-of-distribution model zoos of Arch 2 ? Additionally, we perform cross-dataset evaluation on this task where cross evaluation is with respect to model zoos of Arch 2 , i . e ., we also check for out-of-distribution transfer across datasets.

Baselines: We compare SNE with DeepSets [Zaheer et al., 2017] and NeuralGraph [Kofinas et al., 2024] for this task. None of the other baselines, MLP, STATNN [Unterthiner et al., 2020], NFNNP [Zhou et al., 2023a], NFNHNP [Zhou et al., 2023a], NFT [Zhou et al., 2023b] and DWSNet [Navon et al., 2023] can be used for this task since they impose architectural homogeneity and hence cannot be used for out-of-distribution architectures by design.

Results: We report the quantitative evaluation on the cross-architecture task in Table 2 and report Kendall's τ [Kendall, 1938]. The first column, Arch 1 → Arch 2 shows the direction of transfer, where we train using model zoos of Arch 1 and test on model zoos of Arch 2 . Additionally, A → B, e . g . MNIST → CIFAR10 shows the cross-dataset transfer. From Table 2, it can be seen that SNE transfers best across out-of-distribution architectures and datasets outperforming the DeepSets and NeuralGraph baselines significantly. Interestingly, the DeepSets model, which treats the entire weight

as set with no ordering performs better than the NeuralGraph model on average for this task. In conclusion, SNE shows strong transfer across architecture and datasets in the out-of-distribution benchmark.

Remark: In Table 2, we performed Arch 1 → Arch 2 evaluation specifically to allow benchmarking against NeuralGraph [Kofinas et al., 2024]. Specifically, the

convolutional filters of Arch 1 are larger than those of Arch 2 . NeuralGraph requires specifying this maximum filter size before training and can only be used to transfer across architectures with filter sizes equal or smaller than the predefined filter size (this can be verified from the official source code 1 ), i . e ., NeuralGraph is not truely agnostic to architectural choices. SNE on the other hand does not have this limitation. To demonstrate this, we use the SNE model trained in Table 2 and test it on the SVHN model zoo of Schürholt et al. [2022] which is a much larger architecture with larger filter sizes than those of Arch 1 . We compare with HyperRep [Schürholt et al., 2021] which is trained fully on the testing model zoo. We emphasize that SNE is not trained on the training set of this model zoo. Results for HyperRep are taken from Schürholt et al. [2022] and a single SNE model is evaluated to match the setting of Schürholt et al. [2022]. From Table 3, SNE significantly outperforms HyperRep by very large margins without being trained on the training set of Schürholt et al. [2022] as HyperRep and demonstrates true agnosticism to architectural choices compared to NeuralGraph. We report Kendall's τ [Kendall, 1938] in Table 3.

Evaluation on Transformers: We generate a model zoo of transformer using PytorchViT [2024] and test the transfer from Arch 1 to the transformer model zoo. For this task, we are unable to benchmark against NeuralGraph as was done in Table 2, since the model cannot process transformer weights when trained on Arch 1 . Hence we benchmark against the DeepSets

baseline as in Table 2. From Table 4 SNE generalizes better to the unseen transformer architecture at test time than the baselines showing strong architectural transfer. Additionally, here, the model encodes an architecture with about 5 times the number of parameters in Arch 1 demonstrating the scalability of our approach. The DeepSets baseline fails to generalize on this task.

4.3 Cross-Dataset Performance Prediction

For this task, we train neural network performance predictors on 4 homogeneous model zoos, of the same architecture, with each model zoo restricted to a single dataset.

Datasets and Network Architecture: Each model zoo is trained on one of the following datasets: MNIST [Deng, 2012], FashionMNIST [Xiao et al., 2017], CIFAR10 [Krizhevsky, 2009] and SVHN [Netzer et al., 2018]. We use the model zoos of Unterthiner et al. [2020].

A thorough description of the model zoo generation process can be found in Appendix A.2 of Unterthiner et al. [2020].

Task: We train network encoders on a single model zoo, e . g . MNIST, and evaluate it on the test set of all four datasets and report the averaged performance. This results in in-distribution testing with respect to the dataset used for training and out-of-distrution testing with respect to the other three datasets, e . g . a model trained on MNIST is tested on MNIST, FashionMNIST, CIFAR10 and SVHN.

Baselines: We benchmark against MLP, STATNN [Unterthiner et al., 2020], NFNHP [Zhou et al., 2023a], and the NFNHNP [Zhou et al., 2023a].

Results: We present the results for this task in Table 5. Here we see that SNE again performs better than the competing methods significantly demonstrating strong transfer across different datasets for the same architecture.

MNIST

FashionMNIST

CIFAR10

SVHN

(a) MLP

(b) STATNN

(c) NFNNP

(d) NFNHNP

(e)

SNE(Ours)

Qualitative Analysis: To understand how SNE transfers well across model zoos, we generate TSNE [Van der Maaten and Hinton, 2008] plots for the neural network encodings of all benchmarked methods on all four homogeneous model zoos in Figure 2. We provide 3 different views of each models embeddings to better illustrate the encoding pattern. In Figures 2c and 2d, we observe that NFNNP and NFNHNP have very clear separation boundaries between the networks from each model zoo. In Figures 2a and 2b, MLP and STATNN, respectively show similar patterns with small continuous strings of model zoo specific groupings. However, these separations are not as defined as those of NFNNP and NFNHNP. The embedding pattern of SNE on the other hand is completely different. In Figure 2e, all networks from all the model zoos are embedded almost uniformly close to each other. This may suggest why SNE performs much better on the cross-dataset performance prediction task since it is much easier to interpolate between the neural network encodings generated by SNE across model zoos.

5 Conclusion

In this work, we tackled the problem of encoding neural networks for property prediction given access only to trained parameter values. We presented a Set-based Neural Network Encoder (SNE) that reformulates the neural network encoding problem as a set encoding problem. Using a sequence of set-to-set and set-to-vector functions, SNE utilizes a pad-chunk-encode pipeline to encode each network layer independently; a sequence of operations that is parallelizable across chunked layer parameter values. SNE also utilizes the computational structure of neural networks by injecting positionally encoder layer type/level information in the encoding pipeline. As a result, SNE is capable of encoding neural networks of different architectures as opposed to previous methods that only work on a fixed architecture. To learn the correct minimal equivariance for MLP weight permutations, we utilized Logit Invariance Regularization as opposed to weight tying used in previous methods. Experimentally, we introduced the cross-dataset and cross-architecture neural network property prediction tasks. We demonstrated SNE's ability to transfer well across model zoos of the same architecture but with networks trained on different datasets on the cross-dataset task. On the cross-architecture task, we demonstrated SNE's agnosticism to architectural choices and provided the first set of experimental results for this task that demonstrates transferability across architectures.

Acknowledgments and Disclosure of Funding

This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIP) (No.2019-0-00075 Artificial Intelligence Graduate School Program(KAIST)), by Samsung Research Funding Center of Samsung Electronics (No. IO201210-08006-01), Institute of Information & communications Technology Planning & Evaluation (IITP) under Open RAN Education and Training Program (IITP-2024-RS-2024-00429088) grant funded by the Korea government(MSIT), by Google Research Grant and by the UKRI grant: Turing AI Fellowship EP/W002981/1. We would also like to thank the Royal Academy of Engineering and FiveAI.

References

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Andreis Bruno, Jeffrey Willette, Juho Lee, and Sung Ju Hwang. Mini-batch consistent slot set encoder for scalable set encoding. Advances in Neural Information Processing Systems , 34:21365-21374, 2021.

Edo Cohen-Karlik, Avichai Ben David, and Amir Globerson. Regularizing towards permutation invariance in recurrent models. Advances in Neural Information Processing Systems , 33:1836418374, 2020.

Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning on implicit neural representations of shapes. arXiv preprint arXiv:2302.05438 , 2023.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers , pages 129-135. Elsevier, 1990.

Maurice G Kendall. A new measure of rank correlation. Biometrika , 30(1/2):81-93, 1938.

Hyunsu Kim, Hyungi Lee, Hongseok Yang, and Juho Lee. Regularizing towards soft equivariance under mixed symmetries. arXiv preprint arXiv:2306.00356 , 2023.

Jinwoo Kim, Jaehoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hierarchical composition for generative modeling of set-structured data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15059-15068, 2021.

Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J Burghouts, Efstratios Gavves, Cees GM Snoek, and David W Zhang. Graph neural networks for learning equivariant representations of neural networks. arXiv preprint arXiv:2403.12143 , 2024.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning , pages 3744-3753. PMLR, 2019.

Derek Lim, Haggai Maron, Marc T Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for processing diverse neural architectures. arXiv preprint arXiv:2312.04501 , 2023.

Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. Advances in Neural Information Processing Systems , 35:768-781, 2022.

Artem Moskalev, Anna Sepliarskaia, Erik J Bekkers, and Arnold WM Smeulders. On genuine invariance learning without weight-tying. In Topological, Algebraic and Geometric Learning Workshops 2023 , pages 218-227. PMLR, 2023.

Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. arXiv preprint arXiv:2301.12780 , 2023.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and A Ng. The street view house numbers (svhn) dataset. Technical report, Technical report, Accessed 2016-08-01.[Online], 2018.

Samuel E Otto, Nicholas Zolman, J Nathan Kutz, and Steven L Brunton. A unified framework to enforce, discover, and promote symmetry in machine learning. arXiv preprint arXiv:2311.00212 , 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

PytorchViT. PyTorch-Scratch-Vision-Transformer-ViT . 10 2024. URL https://github.com/ s-chh/PyTorch-Scratch-Vision-Transformer-ViT .

Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 , 2018.

Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems , 34:16481-16493, 2021.

Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Model zoos: A dataset of diverse populations of neural network models. Advances in Neural Information Processing Systems , 35:38134-38148, 2022.

Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems , 33:7462-7473, 2020.

Bedionita Soro, Bruno Andreis, Hayeon Lee, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion-based neural network weights generation. arXiv preprint arXiv:2402.18153 , 2024.

Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002.11448 , 2020.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Jeffrey Willette, Seanie Lee, Bruno Andreis, Kenji Kawaguchi, Juho Lee, and Sung Ju Hwang. Scalable set encoding with universal mini-batch consistency and unbiased full set gradient approximation, 2023.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems , 30, 2017.

Allan Zhou, Kaien Yang, Kaylee Burns, Yiding Jiang, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. arXiv preprint arXiv:2302.14040 , 2023a.

Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. arXiv preprint arXiv:2305.13546 , 2023b.

A Organization

In Section B we provide some limitations and future directions for the neural network encoding task. In Section C, we discuss weight tying versus regularization approaches to achieving minimal equivariance for MLPs together with some theoretical considerations. In Section D, we discuss how SNE can be applied to architectures with branches, specifically ResNets [He et al., 2016]. In Section E, we provide an ablation on the various components of the proposed method. In Section F, we provide additional experimental results on the cross-dataset and cross-architecture tasks. In Section G, we specify all the model architectures used for generating the model zoos of Arch 1 and Arch 2 used for the cross-dataset and cross-architecture tasks. In Section H we provide details of the train/validation/test splits. In Section I, we detail all the hyperparameters used for all experiments. We provide implementation details for SNE in Section J and enlarged versions of Figure 2 in Section K.

B Limitations & Future Work

In this work, we focused solely on the task of predicting the properties, specifically generalization performance, of neural networks given access only to the model parameters. While the task of encoding neural network weights is a relatively new topic of research with very few baselines, we anticipate new applications/research directions where the neural network encoding vector is used for tasks such as neural network generation or neural network retrieval. These tasks, potential applications, and consideration of more complicated neural network architectures (see discussion in Section D) are out of the scope of this paper and we leave it for future works.

Additionally, we have not included results on larger architectures such as Transformers for the following reasons: Firstly, we are unaware of any such extensive model zoos for such architectures. In Lim et al. [2023], such a model zoo is evaluated, however, this is not publicly available hence we are unable to benchmark against such architectures. Lastly, given that all experiments in this paper were carried out on a single GPU with 11GB of memory, we are unable to generate the extensive Transformer model zoos used in Lim et al. [2023]. However in Section D, we discuss how these architectures can be encoded with our method when such model zoos become available.

C Weight Tying vs Regularization and Theoretical Considerations

With regards to achieving minimal equivariance: respecting only a subset of permutations in the weight space that are functionally equivalent, there are two approaches to satisfying this property.

Weight Tying: This approach is adopted by Zhou et al. [2023a,b], Navon et al. [2023] where minimal equivariance is baked into the neural network encoder, i . e ., weight tying. However, this has a couple of drawbacks:

It requires a specification of the neural network to be encoded beforehand. This results in encoders that cannot be applied to different architectures as we demonstrated in Section 4.2 where at test time, we evaluate SNE on architectures not seen during training.

It requires processing all layers of the network together as opposed to the layer-wise encoding scheme that we adopt. Note that for large networks, Zhou et al. [2023a,b], Navon et al. [2023] will require correspondingly large memory but the memory requirements of SNE remains constant since we share the same encoder across all layers, allowing our model to process arbitrary network architectures efficiently.

Regularization: Recent works [Moskalev et al., 2023, Cohen-Karlik et al., 2020, Kim et al., 2023, Miyato et al., 2022, Otto et al., 2023] have shown that symmetric constraints such as permutation invariance/equivariance of a non-symmetric model can be achieved through regularization during training. In this work, we take this approach since it offers the following useful properties:

The level of symmetric constraints, i . e ., minimal equivariance, can be learned directly from data without architectural constraints.

It allows us to process layers independently, resulting in a model that is general and disentangled from architectural choices as we demonstrated in the cross architecture experiments.

In the Ablation study E, we show that the regularization approach indeed helps to learn a model that generalizes well and without it, the model struggles to learn the correct minimal equivariance.

Theoretical Considerations A natural question that arises is how well the regularization approach learns the correct symmetric properties of interest. We first define the Logit Invariance, from which we obtain the Logit Invariance Regularization loss of Moskalev et al. [2023].

Definition C.1 (Logit Invariance) [Moskalev et al., 2023] For a given group G , dataset D and non-symmetric(with respect to G ) model f , the Logit Invariance Loss of f is given by:

Logit Invariance measures the sensitivity of f to the action of G on data samples in D .

The following proposition of Moskalev et al. [2023] characterizes the performance decay of the Logit Invariance Regularizer under some linearity assumptions.

Proposition C.1 (Invariance-Induced by Spectra Decay) [Moskalev et al., 2023] Logit invariance error minimization implies σ max ( W ( t )) ≤ σ ( W (0)) when t →∞ .

proof: See Section 3.3 of Moskalev et al. [2023].

Here, W are model parameters, t optimization steps and σ max ( W ) = || W || 2 . Proposition C.1, through an analysis of the training dynamics of f asserts that f will be insensitive to actions of G on data instances thereby learning the correct invariance property when the Logit Invariance Regularizer is minimized.

In summary, the SNE model constrained by the Logit Invariance Regularizer, learns the correct minimal equivariance required for processing MLP weights without having to resort to weight tying as the methods of Navon et al. [2023] and Zhou et al. [2023a,b].

D Considering Architectures with Branches

Here, we outline how SNE may be applied to a model zoo of architectures such as ResNets where branches exist in the computational graph. Given that residual blocks are composed of convolutional and linear layers, each of these can be encoded independently as we already do. To account for residual connections, we propose to introduce special tokens (just as was done for layer types) when we encode the entire block. Additionally, any symmetries inherent in such blocks can be enforced using the invariance regularization term introduced. Given that Transformers are composed of linear layers, the same pipeline can be applied to encode transformers and logit invariance regularization can be used to respect their inherent symmetries.

While we do not see any impediment to applying SNE to such architectures, the unavailability of such model zoos makes it impossible to experimentally verify these and hence we leave it as future work when such model zoos become publicly available.

E Ablation

20

40

60

80

100

Percentage of Training Data

1.0

0.8

0.6

0.4

0.2

0.0

Error

NFT

DWSNet

E.1 Components of SNE

We investigate the impact of the invariant regularization loss, the hierarchical and positional encoding modules, the usage of set functions, layer level and layer type encoders on the performance of SNE using the INR dataset. In Table 6, it can be seen that the hierarchical and positional encoding modules play an important role in the performance of SNE. Removing this module results in significant performance degradation from 0.098 to 7.452. Without positional encoding, SNE is fully permutation equivariant even to functionally incorrect permutations of the weights, making it difficult for the model to learn the correct restricted set of permutations. Secondly, removing the invariance regularization term results in a degradation in performance, from 0.098 to 0.160, as it also makes the task learning the correct minimal equivariance subgroup difficult. We find this only to have a huge impact for MLPs such as INRs where weight permutations are well defined. Additionally, removing the set functions and the layer level and type encoders from the model results in performance drop.

E.2 Effect of Chunksize

We provide ablation on the effect of chunksize in the table below for the INR experiment presented in Table 1 of the main paper. From Table 7 we see that performance stays almost the same until the largest chunksize. Hence the chunksize is selected to suit the memory requirements.

E.3 How Well Is The Logit Invariance Regularization Term Minimized?

We compute the logit invariance loss for the experiments for 5 functionally equivalent permutations of INR weights and obtain a loss of approximately 10 -15 . This low loss implies that the SNE learns the correct level of invariance which is the desired property of the non-weight tying approach that we take.

E.4 Measuring the Importance of Training Set Size

We provide a plot of training set size versus error in Figure 3 using the INR experiment presented in Table 1. From this it can be seen that SNE is more data efficient compared to the baselines [Navon et al., 2023, Zhou et al., 2023b] across all percentages of the full training data demonstrating that the proposed method learns a good embedding even in the limited data setting.

F Additional Experimental Results

In this section we provide additional experimental results on the cross-dataset and cross-architecture tasks.

F.1 Cross-Dataset Evaluation

For this task, we train neural network performance predictors on 4 homogeneous model zoos, of the same architecture, with each model zoo restricted to a single dataset.

Datasets and Network Architecture: Each model zoo is trained on one of the following datasets: MNIST [Deng, 2012], FashionMNIST [Xiao et al., 2017], CIFAR10 [Krizhevsky, 2009] and SVHN [Netzer et al., 2018]. We use the model zoos provided by Unterthiner et al. [2020]. To create each model zoo, 30K different hyperparameter configurations were sampled. The hyperparameters include the learning rate, regularization coefficient, dropout rate, the variance and choice of initialization, activation functions etc. A thorough description of the model zoo generation process can be found in Appendix A.2 of Unterthiner et al. [2020]. Architectural descriptions for the model zoos are outlined in Appendix G. Each model zoo is split into a training, testing and validation splits.

Task: In this task, we consider cross-dataset neural network performance prediction where we evaluate the prediction performance on the testset of the model zoo on which the predictors were trained on. Additionally, we evaluate how well each predictor transfers to the other model zoos. We evaluate all models using Kendall's τ [Kendall, 1938].

Results: We present the results of the cross-dataset performance prediction task in Table 8. For each row in Table 8, the first column shows the cross-dataset evaluation direction. For instance, MNIST → CIFAR10 implies that a model trained on an MNIST model zoo is cross evaluated on a model zoo populated by neural networks trained on CIFAR10. We note that the A → A setting, e . g . MNIST → MNIST, corresponds to the evaluation settings of Unterthiner et al. [2020] and Zhou et al. [2023a]. Also in Table 8 we show the best model in red and the second best model in blue.

As show in Table 8, SNE is always either the best model or the second best model in the cross-dataset task. SNE is particularly good in the A → B performance prediction task compared to the next competitive baselines, NFNNP and NFNHNP. The MLP baseline, as expected, performs the worse since concatenating all weight values in a single vector looses information such as the network structure. STATNN [Unterthiner et al., 2020] performs relatively better than the MLP baseline suggesting that the statistics of each layer indeed captures enough information to do moderately well on the neural network performance prediction task. NFNNP and NFNHNP perform much better than STATNN and MLP and NFNHNP in particular shows good results in the A → A setting. Interestingly, NFNNP is a much better cross-dataset performance prediction model than NFNHNP. However, across the entire cross-dataset neural network performance prediction task, SNE outperforms all the baselines as shown in the last row of Table 8.

F.2 Cross-Architecture Evaluation

Next we reverse the transfer direction from Arch 2 to Arch 1 in Section 4.2 and provide the results for SNE in Table 9. Note that for this task NeuralGraph [Kofinas et al., 2024] cannot be used since we transfer from a smaller architecture to a larger one. As can be seen from Table 9, SNE transfers well across architectures further validating the results in Section 4.2.

G Architectures for Generating model zoos

We specify the architectures for generating the model zoos of Arch 1 and Arch 2 . For the model zoos of Arch 1 in Table 10, all datasets with with 3 channel images (CIFAR10 and SVHN) are converted to grayscale. This is in accordance with the setups of Unterthiner et al. [2020], Navon et al. [2023] and Zhou et al. [2023a,b] and allows us to evaluate both methods in the cross-dataset task for this set of homogeneous model zoos. For model zoos of Arch 2 in Tables 11 & 12, we maintain the original channels of the datasets. The cross-architecture transfer task is from Arch 1 to Arch 2 . Note also that for the same dataset, i . e ., CIFAR10, the cross-architecture evaluation is also from models trained grayscale to RGB images. All model zoos were generated using the procedure outlined in the Appendix of Unterthiner et al. [2020]. The architecture for the INR experiments is outlined in Table 16.

H Dataset Details

Dataset splits for model zoos of Arch 1 is given in Table 13. For the cross-architecture task, we generate model zoos of with 750 neural networks of Arch 2 for testing.

I Hyperparameters

We elaborate all the hyperparameters used for all experiments in Table 14.

J Implementation Details

SNE is implemented using Pytorch [Paszke et al., 2019]. The SNE model consists of 4 sub-modules:

Layer Chunk Encoder: This consists of two SAB modules where each SAB module is as stack of two SAB layers, followed by a single PMA layer. The layer chunk encoder encodes all the chunks of a given layer independently.

Layer Encoder: This module encodes all the encoded chunks of a layer and consists of two SAB modules and a single PMA layer.

Separated Layer Encoder: This module encodes all the encodings of a layer, for instance the weights and biases, into a single layer encoding vector. It also consists of two SAB modules and a single PMA layer.

NN Encoding Layer: This module takes as input all the layer encodings and compresses them to obtain the neural network encoding which is used for the downstream task of predicting the neural network generalization performance. It also consists of two SAB modules and a single PMA layer.

In addition to the sub-modules above, the layer/level positional encoders are applied to each submodule when required (see Section 3). The neural network performance predictor, which takes as input the neural network encoding vector from SNE and predicts the performance is detailed in Table 15.

K Miscellanea

In Figures 4, 5 and 6 we provide enlarged versions of the figures in Figure 2 for better viewing.

MNIST

FashionMNIST

CIFAR10

SVHN

(a) MLP

(b) STATNN

(c) NFNNP

(d) NFNHNP

Figure 4: TSNE Visualization of Neural Network Encoding. We train neural network performance prediction methods on a combination of the MNIST, FashionMNIST, CIFAR10 and SVHN modelzoos of Unterthiner et al. [2020]. Zoom in for better viewing.

MNIST

FashionMNIST

CIFAR10

SVHN

(a) MLP

(b) STATNN

(c) NFNNP

(d) NFNHNP

MNIST

FashionMNIST

CIFAR10

SVHN

(a) MLP

(b) STATNN

(c) NFNNP

(d) NFNHNP

Figure 6: TSNE Visualization of Neural Network Encoding. We train neural network performance prediction methods on a combination of the MNIST, FashionMNIST, CIFAR10 and SVHN modelzoos of Unterthiner et al. [2020]. Zoom in for better viewing.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We experimentally demonstrate strong empirical results on various benchmarks to back the claims made in the abstract and introduction. Additionally, we provide theoretical backing for our method in Appendix C.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations of our method in Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theoretical considerations are provided in Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all experimental settings as well as descriptions of model zoos used in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The appendix provides details on generating the model zoos used for experiments in this paper. Additionally, a reference implementation will be made publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All such details are provided in the experimental section and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All results have error bars with the exception of Table 3 where a single model is evaluated in accordance with the experimental setting described.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Computational resources used of all experiments is stated in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All NeurIPS code of Ethics are observed.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Potential impact and extensions of our method is provided in Appendix B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets, code, etc are appropriately cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SNE, or Set-based Network Encoding, a method of encoding the weights of arbitrary neural networks in order to predict properties such as performance, generalization gap, training loss, number of epochs, etc.  Specifically, using a chunking mechanism as well as layer-wise and block-wise positional encodings to generalize across different types of neural networks. Transferability experiments are conducted where a predictor is trained on one network model zoo, then evaluate on another, demonstrating an ability few predictors have. Additional experiments and ablations are performed, producing convincing results. Finally, the paper is decently well-written, but has a few fixable presentation flaws.\n\nSTRENGTHS:\n- The contribution of the paper, using a network's weights to determine attributes about it, are quite novel. \n- The Methodology section is quite clear and easy to follow. Specifically, the aggregation mechanism in Eq. 9 is similarly analogous to graph aggregation in GNNs. \n- The experimental results are generally convincing.\n- Some visualization of the network representations from different methods are provided.\n- There are ablation results included in the appendix.\n- A limitations section is provided, highlighting reasonable restrictions on the experiments.\n\nWEAKNESSES:\n- First, the motivation of this paper, specifically lines 29-30, is quite weak and fails to really provide examples of the application of this research other than asserting that they must exist.\n- Overall, the presentation is a weak spot, and should be revised. Some examples: \n\t- E.g., in line 31, the authors mention \"Implicit Neural Representations (INR)\" - what are these? No citation is provided. By contrast, \"b) the performance of CNNs\" is very easy to understand.\n\t- \"modelzoo\", this should be \"model-zoo\" [1] or \"model zoo\" [2-4].\n\t- Citations in the paper have issues. For NeurIPS its generally just [num], or if you want to be fancier, \"Author et al., Year [num]\" but not \"Author et al. [num]\" without the year.\n\t- Float captions need work, e.g., Tabs 2/3 need longer captions to fully explain the performance metric Kendall's Tau, etc. Should be longer than 1 sentence.\n\t- Figure 2 needs work. The plots should be bigger and legend is clipped.\n\t- NeurIPS checklist should be placed after the references and supplementary.\n- The evaluation is done on simpler datasets/networks, however, the limitations section highlights why.\n\nQUESTIONS:\nI find the idea of predicting end-to-end neural network attributes from the weight values to be quite fascinating. It is very similar to the concept of a neural predictor [5] in Neural Architecture Search [6] but quite orthogonal in the approach, relying on the weights instead of the graph structure. Similarly, most neural predictors suffer from a lack of generalizability/transferability [7, 8], like most of the baselines in this paper. Can the authors provide some commentary/comparison between these two umbrella fields? \n\nSimilarly, is it possible to leverage the encoding mechanism (Eqs. 1-9) to derive which types of weights/layers/values are responsible for certain end-to-end attributes, e.g., performance of epochs, e.g., as [9] do?\n\nRefs:\n\n[1] \"Equivariant Architectures for Learning in Deep Weight Spaces\" - ICML'23.\n\n[2] https://modelzoo.co/\n\n[3] https://github.com/onnx/models\n\n[4] https://pytorch.org/serve/model_zoo.html\n\n[5] \"How Powerful are Performance Predictors in Neural Architecture Search?\" - NeurIPS'21.\n\n[6] \"Efficient Neural Architecture Design via Capturing Architecture-Performance Joint Distribution\" - AISTATS'24.\n\n[7] \"GENNAPE: Towards Generalized Neural Architecture Performance Estimators\" - AAAI-23.\n\n[8] \"Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor\" - NeurIPS'22.\n\n[9] \"Building Optimal Neural Architectures using Interpretable Knowledge\" - CVPR'24.\n\nLIMITATIONS:\nYes, limitations and future work is discussed in App. B. The authors explain that their experiments are on simpler/smaller/older networks as they have limited computational resources. I find this to be an acceptable justification.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-13"}

comments: SUMMARY:
This work introduces a set-based neural network encoding that processes each chunk of an input model independently to predict network properties. The proposed model, SNE, is trained using Logit Invariance instead of weight tying to maintain generalizability to unseen architectures. Unlike previous approaches, SNE can be trained and tested on different architectures and it demonstrated superior performance in out-of-distribution experiments involving both datasets and architectures.

STRENGTHS:
SNE is more general than its predecessors and can be applied to any input architecture. Results indicate that it outperforms other encoding methods.

WEAKNESSES:
While SNE shows improvements in novel tasks such as cross-architecture and cross-dataset scenarios, its comparison on traditional tasks is limited to predicting INR frequencies. Expanding Table 1 to include SNE's performance on additional outputs, such as accuracy, would enhance the paper.

QUESTIONS:
The writing is generally good but could be improved in certain areas. For example, the abstract contains a redundant sentence: “by utilizing a layer-wise encoding scheme that culminates to encoding all layer-wise encodings to obtain the neural network encoding vector,” which should be revised.

LIMITATIONS:
Limitations were properly addressed in appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles an original and interesting challenge: predicting neural networks properties from their trained weight values. To do so, the authors propose to leverage set to set and set to vector transformations in order to encode weight values. Furthermore, they propose to account for the deep neural network architecture and operation order through the addition of positional encoding at several stages of the property predicting model. Ultimately, they anticipate the problem of various model sizes with a padding/chinking mechanism.\nThe proposed method is then evaluated on small scale datasets in a cross-dataset fashion against method of the field, published in recent years.\n\nSTRENGTHS:\nSuch research studies can have a significant in other fields. For instance, a good accuracy estimator from model weight values can be leveraged as a proxy objective in DNN quantization or weight pruning.\nThe approach is well motivated and the solutions to the highlighted challenges are sound.\nThe paper is well written and simple to follow.\nThe authors provide a thorough evaluation and ablation study of the proposed method and its components.\n\nWEAKNESSES:\nFrom my understanding of this work, I identified two major drawbacks:\n1. the authors insist on the generalization across different model architectures and tasks. From my perspective, the former is the most important one. However, as it stands, there is little variation between Arch1 and Arch2 as presented in this study: both are simple CNNs. I wonder if the authors could provide numbers w.r.t transformer architectures against CNNs.\n2. Since these experiments can be expensive (require multiple training of DNNs to create the predictor training set), it would be interesting to measure the impact of the dataset size w.r.t the predictor's performance, in order to estimate how far the community is from being to reasonably work on large scale datasets such as ImageNet.\n\nAs it stands, I believe this work aims at providing empirical results which are not fully convincing yet. I look forward to the authors' response and other reviews and will update my rating accordingly.\n\nQUESTIONS:\nI summarize the previous points:\n1. could the authors provide results with a predictor trained on CNNs from arch1 and predict the performance of transformers?\n2. could the author measure the importance of the training set size (pairs of DNNs and accuracy) w.r.t. the predictor's performance, in order to estimate the cost of working on ImageNet-like tasks?\n\nLIMITATIONS:\nThe authors made a relevant attempt at highlighting the limitations of their work", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe study introduces Set-based Neural Network Encoding (SNE) without weight tying, allowing for encoding network information into a compact form. SNE utilizes Logit Invariance Regularization to ensure the correct functional equivalence of network permutations, departing from traditional weight tying methods. The choice of Set-to-Set and Set-to-Vector functions, including attention mechanisms and pooling layers, are crucial in implementing the SNE framework. Experimental results demonstrate the effectiveness of SNE in predicting generalization performance for Convolutional Neural Networks (CNNs) and Incremental Neural Networks (INRs). SNE is compared with various baselines such as MLP, Deepsets, HyperRep, and others, showcasing its superiority in certain aspects.\n\nSTRENGTHS:\n- New evaluation benchmarks of cross-dataset and cross-architecture evaluation.\n\n- Solid empirical performance. SNE significantly outperforms HyperRep by very large margins in the cross-architecture task. Also, SNE demonstrates true agnosticism to architectural choices compared to NeuralGraph, and shows robust performance without being trained on the training set of a specific model zoo.\n\nWEAKNESSES:\n- The proposed method did not ensure weight-space equivariance by design but use a way that is more like an augmentation plus a consistency loss. The theoretic contribution is therefore a bit weak.\n\n- The padding and chunking designs look tricky and it is not clear for me how they help with the performance.\n\n- Fig 1 is not really clear. (1) the legend is not sufficient to understand the computation diagram presented in the figure. (2) the input is shown as neurons but from the method I think it should be the weight matrix that connecting neurons in adjacent layers.\n\nQUESTIONS:\n- In table 1 NeuralGraph is not reported. Is it because the method is not applicable there?\n\n- Does the method need more optimization steps than baseline methods that ensure weight-space symmetry by design? I suppose optimization would be harder because it requires to optimize for the logit invariance.\n\nLIMITATIONS:\nThe limitations of the method are discussed in Appendix B", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}