Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction
Authors: Konstantin Schürholt
Dimche Kostadinov
Damian Borth
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representation Learning
Self-Supervised Learning
Weight Space
Parameter Space
Augmentation
Model Zoos

Abstract: Self-Supervised Learning (SSL) has been shown to learn useful and information-preserving representations. Neural Networks (NNs) are widely applied, yet their weight space is still not fully understood. Therefore, we propose to use SSL to learn hyper-representations of the weights of populations of NNs. To that end, we introduce domain specific data augmentations and an adapted attention architecture.  Our empirical evaluation demonstrates that self-supervised representation learning in this domain is able to recover diverse NN model characteristics. Further, we show that the proposed learned representations outperform prior work for predicting hyper-parameters, test accuracy, and generalization gap as well as transfer to out-of-distribution settings.

Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction
Konstantin Schürholt
konstantin. schuerhol temool@@inigg.. ch
AIML Lab, School of Computer Science University of St.Gallen
Dimche Kostadinov
Damian Borth
dimche. kostadinov@unisg ch
damian borthhunisg. ch
AIML Lab, School of Computer Science University of St.Gallen
AIML Lab, School of Computer Science University of St.Gallen
Abstract
Self-Supervised Learning (SSL.) has been shown to learn useful and information- preserving representations. Neural Networks (NNs) widely applied, yet their weight space is still not fully understood. Therefore, we propose to use SSL to learn repessttorsssnnaatiinn of the weights of populations of NNs. To that end, we introduce domain specific data augmentations and an adapted attention architecture. Our empirical evaluation demonstrates that self-supervised representation learning in this domain able to recover diverse NN model characteristics. Further, we show that the proposed learned representations outperform prior work for predicting hyper-parameters, test accuracy, and generalization gap as well as transfer to out-of-distribution settings. Code and datasets are publicly available
Introduction
This work investigates populations of Neural Network (NN) models and aims to learn representations of them using Self-Supervised Learning. Within NN populations, not all model training is successful, i.e. some overfit and others generalize. This may due to the non-convexity of the loss surface during optimization [Goodfellow et al. [2155) the high dimensionality of the optimization space, or the sensitivity hyperparameters [Hanin and Rolnick] 2018), which causes models to converge to different regions in weight space. What is still not yet fully understood, is how different regions in weight space are related to model characteristics.
Previous work has made progress investigating characteristics of NN models, e.g by visualizing learned features [Zeiler and Fergus, 2014, Karpathy et 2015). Another line of work compares the 2
activations of pairs of NN models [Raghu et al. [017 Morcos et al.) 2018 Kornblith et al. 2019).
Both approaches rely on the expressiveness of the data, and are. the latter case, limited two models at a time. Other approaches predict model properties, such accuracy, generalization gap, or hyperparameters from the margin distribution (Yak et al.) 2019 Jiang al., (209)) graph topology features Correaau et al.) 2020] or eigenvalue decomposition of the weight matrices [Martin and Mahoney 2019]. In a similar direction, other publications propose to investigate populations of models and to predict properties directly from their weights weight statistics in a supervised way [Unterthiner et al. 2020 Dilertsen et al. 2020). However, these manually designed features may not fully capture the latent model characteristies embedded in the weight space.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

I. Model Zoo Generation
II. Representation Learning Approach
III. Down. Tasks
hyper repressttttnnnon
data set:
permutation
hyper parameters:
hypar- initialization
accuracy
fully-connetedd
attentloonbbeed
class-wise
(W....Ww
accuracy
architecturee A
reconstrcction
reconstruction 1 contrast
contrast
ganeralization
epoch
gep
Figure I: An overview of the proposed self-supervisee representation learning approach. I. Popula- tions of trained NNs form model zoos, each model transformed in a vectorized form of its weights. II. reeeeeeettssssnnaaioonn are learned from the model zoos using different augmentations, architec- tures, and Self Self-Suuer I Learning tasks. M. Hrpee-represennattonn are evaluated on downstream tasks which predict model characteristics.
Therefore, our goal is to learn task-agnostic representations from populations of NN models able to reveal such characteristics. Self- Self-Suuervised Learning (SSL) is able to reveal latent structure in complex data without the need of labels, e.g., by compressing and reconstructing data [Goodfellow
ct al. learning has gained popularity [Misra and van der Maaten 2019, Chen et al. 2020a b) Grill et al.)
2016 Kingma and Welling 2014]. Recently, a specific approach to SSL called contrastive
2020 Contrastive learning leverages inherent symmetries and equivariances in the data, allows to encode inductive biases and thus structure the learned representations.
In this paper, we propose a novel approach to apply SSL to learn representations of the weights of NN populations. We learn representations using reconstruction, contrast, and a combination of both. To that end, we adapt a transformer architecture to NN weights. Further, we propose three novel data augmentations for the domain of NN weights. We introduce structure preserving permutations of NN weights as augmentations, which make use of the structural symmetries within the NN weights that we find necessary for learning generalizing representations. We also adapt erasing (Zhong et al. 2020) and noise [Goodfellow al., 2016] augmentations for NN weights. We evaluate the learned representations by linearpprobing for the generating factors and characteristics of the models. An overview for our learning approach is given in Figure [
To validate our approach, we perform extensive numerical experiments over different populations of trained NN models. We find that renpeerrepreeetttioonn can be learned and reveal the characteristics of the model zoo. We show that ryper-representations have high utility on tasks such as the prediction of syper-parameters, test accuracy, and generalization gap. This empirically confirms our hypothesis on meaningful structures formed by NN populations. Furthermore, we demonstrate improved performance compared to the state-of-the-art for model characteristic prediction and outlay the advantages in diitr--diitiitttion predictions. For our experiments, we use publicly available NN model zoos and introduce new model zoos. In contrast to [Unterthiner et al. 2020 Eilertsen et al. 2020), our zoos contain models with different initialization points and diverse configurations, and include densely sampled model versions during training. Our ablation study confirms that the various factors for generating a population of trained NNs play a vital role in how and which properties are recoverable for trained NNs. In addition, the relation between generating factors and model zoo diversity reveals that seed variation for the trained NNs in the zoos is beneficial and adds another perspective when recovering NNs' properties.
2 Model Zoos and Augmentations
Model Zoo We denote as D a data set that contains data samples with their corresponding lahels. We denote À the set of rate, weight initialization, batch-size, epochs). We define as A the specific NN architecture. Training under different prescribed configurations {D, X, A} results in a population of NNs which we refer to as model zoo. We convert the weights and biases of all NNs of each model into a vectorized form. In the resulting model zoo W = [W1,- ..... Was }. w; denotes the flattened vector of dimension N, representing the weights and biases for one trained NN model.
used for training, (e.g., loss function, optimizer, learning
By repeerrepresnntatioon we refer to a leamed representation from a population of NNs. i.e., model zoo in analogy HyperNetworks (Ha al. 2016, which trained to generate weights for larger NN models.

---

Augmentations. Data augmentation generally helps to learn robust models, both by increasing the number of training samples and preventing overfitting of strong features [Shorten and Khoshgoftaar 2019]. In contrastive learning, augmentations can be used to exploit domain-specific inductive biases, e.g., know symmetries or equivariances [Chen and Lij 2020]. To the best of our knowledge, augmentations for NN weights do not yet exist. In order to enable self-supervised representation learning, we propose three methods to augment individual instances of our model zoos.
Neurons in dense layers can change position without changing the overall mapping of the network, if in in-going and out-going connections are changed accordingly [Bishop 2006]. The relation between equivalent versions of the same network translates to permutations of incoming weights with matrix P and transposed permutation P" of the outgoing weights (P"P I). Considering the output at layer 1 + 1, with weight matrices W, biases b, activations a and activation function 0. we have
z w w (W'a + b') + bb-- = W W (W''' + +b')+b'I
(1)
where W (P'), W W''W' and 6' P'b' are the permuted weight matrices and bias vector, respectively. The eguivalences hold not only for the forward pass, but also for the backward pass and weight update The permutation can be extended to kernels of convolution layers. The permutation augmentation differs significantly from existing augmentation techniques. As analogy, flips along the axis of images are similar, but specific instances from the set of possible permutations in the image domain. Each permutable layer with dimension Ni, has Ni! different permutation matrices, and in total there are II Ni! distinct, but equivalent versions of the same NN. While new data can be created by training new models, the generation is computationally expensive. The permutation augmentation, however. allows to compute valid NN samples at almost no computational cost. Empirically, we found the permutation augmentation crucial for our learning approach.
In computer vision and natural language processing, masking parts of the input has proven to be helpful for generalization (Devlin et al. [019]. We adapt the approach of random erasing of sections in the vectorized forms of trained NN weights. As in [Zhong et al.] [2020]. we apply the erasing augmentation with a probability p to an area that is randomly chosen with a lower and upper bounds brow and bup. In our experiments, we set p = 0.5, biow 0.03, gup = 0.3 and erase with zeros. Adding noise augmentation is another way of altering the exact values of NN weights without overly affecting their mapping, and has long been used in other domains [Goodfellow et al. 2016).
3 Hyper-Representatiin Learning
With this work, we propose to learn representations of structures in weight space formed by popu- lations of NNs. We evaluate the representations by predicting model characteristics. A supervised learning approach has been demonstrated [Unterthiner et al. 2020 Eilertsen et al. 2220)). [Unterthiner al. 2020) find that statistics of the weights (mean, var and quintiles) are superior to the weights to predict test accuracy, which we empirically confirm in our results. However, we intend to learn representations of the weights, that contain rich information beyond statistics. 'Labels' for NN models can be obtained relatively simply, yet they can only describe predefined characteristics of a model instance (e.g., accuracy) and so supervised learning may overfit few features, as [Unterthiner et al. 2020) show. Self-supervised approaches, on the other hand, are designed to learn task-agnostic representations, that contain rich and diverse information and are exploitable for multiple downstream tasks [LeCun and Misra 2021]. Below, we present the used architectures and losses for the proposed self-supervised representation learning.
Architectures and Self-Supervised Losses We apply variations of an encoder-decoder architecture. We denote the encoder as go (w,). its parameters as 9, and the heper-repressnntaionn with dimension L as z, = ge (w.). We denote the decoder as h.g (z;), its parameters as s), and the reconstructed NN weights as w; = (z,
hig (ge (w:)). As common in CL, we apply a projection head p., (z.).. with parameters 7, and denote the projected embeddings as 7; = p, (z:) = Py (w:)). In all of the architectures, we embed the hyper-representation z; in a low dimensional space, < N. We employ two common SSL strategies: reconstruction and contrastive learning. Autoencoders (AEs) with reconstruction loss are commonly used to learn low-dimensional representations [Goodfellow
Details, formal statements and proofs can be found in Appendix A.

---

All
All
Figure Trained NN weights are the input sequence to transformer. Left. Each element in the sequence represents the weight that connects two neurons at two different layers. Right, Each element in the sequence represents the set of weights related to neuron.
Multi-heas attention based encoder. Left. Regular sequence-to-sequene translation, each element the output sequence is used. Right. An additional compression token is added to the sequence. From the output sequence, only the compression token is taken.
Figure 3:
et al. 2016]- As AEs aim to minimize the reconstruction error, the representations attempt to fully encode samples. Further, contrastive learning is an elegant method to leverage inductive biases of symmetries and inductive biases [Bronstein et al., 2021].
Reconstruction (ED): For reconstruction, we minimize the MSE LMSE = M
Σ
Iw;
-=
hy (ge ( 1)) We denote the encoder-decoder with reconstruction loss as ED. Contrast (E,): For contrastive learning, we use the common NT NT_Xent loss [Chen et al. 2020al as
Le. For a batch of MB model weights, each sample is randomly augmented twice to form the two
the loss is given as views i and j. With the cosine similarity sim(Z,, 2j) प 2 /z, [أj]|,
L Σ log exp(sim(Z:: 2,, 2,)
(i.5) Eier exp(iimmsim Z4, 2,)/T'
where Lezt is if k * i and 0 otherwise, and T is the temperature parameter. We denote the encoder with a contrastive loss as E...
Reconstruction Contrast (E.D): Further combine reconstruction and contrast via L 3LMSE + (1 C. in order to achieve good quality compression via reconstruction and well- structured representations via the contrast. We denote this architecture with its loss as E.D.
Reconstruction + Positive Contrast (E.,, In contrastive learning, many methods prevented mode collapse by using negative samples. The combined loss contains reconstruction term LMSE. which can be seen as regularizer that prevents mode collapse. Therefore, we also experiment with replacing Ce in our loss with modified contrastive term without negative samples:
Ce+
Σ
log (exp(sim( E //T
Σ sim(z] zτ) log(T))
(3)
2021] explore similar simplifications without reconstruction. We denote the encoder-decoded with the loss L = 3LMSE + (1 (1- B)+ as E.,D.
[Chen and He 2021 Schwarzer et al.
Attention Module. Our encoder and decoder pairs are symmetrical and of the same type. As there is no intuition on good inductive biases the weight space, we apply fully connected feed-forward networks (FFN) as baselines. Further, wee use multi-head self-attention modules (Att) (Vaswani et al. 2017] as an architecture with very little inductive bias. In the multi-head self-attention module, we apply learned position encodings preserve structural information [Dosovitskiy et al., 2020 The explicit combination of value and position makes attention modules ideal candidates to resolve the permutation symmetries of NN weight spaces. We propose two methods to encode the weights into sequence (Figure In the first method, we encode each weight as token in the input sequence. In the second method, we linearly transform the weights of one neuron kernel and use it token. Further, we apply two variants to compress representations in the latent space (Figure (3). In the first variant, we aggregate the output sequence of the transformer and linearly compress to a hyper-represennation 2; In the second variant, similarly to [Devlin et al. 2019 Zhong et al.) 2020). we add learned token to the input sequence that dub compression token. After passing the imput sequence trough the transformer, only the compression token from the output sequence is linearly compressed to a syper-representation z;. Without the compression token, the information is distributed across the output sequence. In contrast, the compression token is learned as an effective query aggregate the relevant information from the other tokens, similar to DJegle et al.) (02TT) The capacity of the compression token can be an information bottleneck. Its dimensionality is directly tied the dimension of the value tokens and its capacity affects the overall memory consumption.

---

TETRIS-
TETRIS-
Rec.
Eph
Acc
Fl
Fle1
Fl
Fl
Rec.
Eph
Acc
Flco
Fla1
Fl
Flcs
Ec
96.7
90.8
67.7
72.0
74.4
85.8
FF
0.0
80.0
85.3
57.3
53.9
64.5
80.1
ED
96.1
88.3
68.9
47.8
57.2
33.0
58.1
Attw
6.8
95.3
71.1
47.9
69.0
49.9
61.3
E.D
84.1
97.0
90.2
70.7
75.9
69.4
86.6
All
74.1
95.4
88.6
65.6
69.8
69.9
86,8
D
95.9
94.0
69.9
48.9
58.1
32.5
58.8
Allx
89.4
97.1
88.4
71.4
80.8
69.1
82.3
A
84.1
97.0
90.2
70.7
75.9
69.4
86.6
Table 1: Ablation results over sulf-supe learning losses. All models implemented with attention-based reference architecture. All val- ues are given in %.
Table 2: Ablation results in % under E,D setup. We use ferd-forward FF and attention-based vari- ants with weight and neuron encoding Allw and Alty each with +/ and without compress. token.
Downstream Tasks. We use linear probing Grill et al. 2020 as a proxy to evaluate the utility of the learned hyper-representationss. As downstream tasks we use accuracy prediction (Acc), generalization gap (GGap), epoch prediction (Eph) as proxy to model versioning, F1-Score prediction (FIc), learning rate (LR). Ez-regularizatio (fz-reg), dropout (Drop) and training data fraction (TF). Using such targets, we solve regression problem and measure the R² score (Wright, 1921). We also evaluate for pyppe-parameters prediction tasks, like the activation function (Act), optimizer (Opt), initialization method (Init). Here, we train linear perceptron by minimizing cross entropy loss GGoddfellow et al. 2016) and measure the prediction accuracy.
4 Empirical Evaluation
4.1
Model Zoos
Publicly Available Model Zoos. [Unterthiner et al. 2020] introduced model zoos CNNs with 4970 parameters trained on MNIST [LeCun and Cortes] Fashion-MNIST (Xiao et al. 2017). CIFARIO Krizhevsky et al.) and SVHN (Netzer et al.] [011]] and made them available under CC BY 4.0. We refer to these as MNIST-HYP, FASHION HYP, CIFAR1O-HYP and SVHN-HYP. We categorize these 70os as large due to their number of parameters. In their model zoo creation, the CNN architecture and seed were fixed, while the activation function, initialization method, optimizer, learning rate, l regularization, dropout and the train data fraction were varied between the models.
Our Model Zoos. We hypothesize that using only one fixed seed may limit the variation in characteristics of a 7oo. To address that, we train zoos where we also vary the seed and perform an ablation study below. As toy example to test architectures, SSL tasks and augmentations, we first create a 4x4 grey-scaled image data set that we call tetris by using four tetris shapes.
CUTRIS-SEER
P
N
P.E.
P.N
E,N
P.E.N
ED
88.1
95.5
89.8
88.8
96.1
95.6
89.7
96.1
E.D
49.2
72.9
67.3
59.4
84.1
81.9
65.4
84.1
E.D
86.3
95.6
88.5
87.6
95.9
95.8
89.0
96.0
Table Ablation results for different augmentations and rep- resentation learning tasks. We use: permutation (P), erasing (E) and noise (N) augmentation and report test-split recon- struction R² scores in %.
We introduce two zoos, which we call TETRIS- PETRIS-SEED and TETRIS-HYP, which we group under small. Both zoos contain FFN with two layers and have total number of 100 learnable parameters. In the TETRIS-SEER 70o, we fix all typee-parameters and vary only the seed to cover a broad range of the weight space. The RCTRIS-SEED zoo contains 1000 models that are trained for 75 epochs. To enrich the diversity of the models, the TETRIS-HVE zoo contains FFNs, which vary in activation function [tanh, relu], the initialization method [uniform. normal, kaiming normal, kaiming uniform, xavier normal, xavier uniform] and the learning rate [le-3, 1e-4, 1e-5]. In addition, each combination is trained with seeds 1-100. Out of the 3600 models in total, we have successfully trained 2900 for 75 epochs the remainders crashed and are disregarded. Similarly to rETRIS-SEED further create zoos of CNN models with 2464 parameters, each using the MNIST and Fashion- MNIST data sets, called MNIST- WWIST-SEED and FASHION AASHION-SEED and grouped them as medium. To maximize the coverage of the weight space, again initialize models with seeds 1-1000.
FFull details on the generation of the zoos be found in the Appendix Section C
5

---

MNIST-SEED
MNIST-HYP NYIST RAND SEED
MNIST S-ED
MNISTHYPP RAND SEED
kerning uniform
uniform
nommal
kaiming normal
Figure 4: UMAP dimensionality reduction for NN model 2oos created with different generating factors. Colors represent initialization methods. Compare numerical results in Table
MNIST-HYP-
MNIST-HYP.
MNIST HYP- 5-PIX- SEED
MNIST-HY' HYP-
1 1-FIX-SEBS
1 1-RAND- SEED
5-RAND-
VAR .234
.155
.152
.091
.092
VARe
.234
.101
.164
.094
.100
Rer
73.4
91.3
80.0
81.3
65.6
Rees
59.0
72.9
37.2
70.0
38.6
W
s(W)
D
S(W)
E,D
s(W)
E..
W
S(W)
E,D
S(W) E,D
EPH
84.5
97.7
97.3
07.2
06.2
11.6
-34
00.5
04.1
-2.7
7.4
10.2
-14
09.6
07.2
ACC
91.3
98.7
98.9
73.6
72.9
89.5
-06
60.1
85.7
57.7
75.1
79.4
-13
74.5
82.5
GGAP
56.9
66.2
66.7
55.3
42.1
61.9
-36
31.1
54.9
37.8
36.7
57.7
-3.2
46.1
60.5
INIT
87.7
63.3
75.4
38.3
56.1
48.2
80.6
54.5
51.4
35.5
47.2
40.4
Table 4: R² score in %. Results the impact of the generating factors for the model zoos. Top: VAR the variance of the weights. VAR, denotes the mean the variances for groups of samples with shared initialization method and activation function. RG, and Ries are the reconstruction R of train and test split on a reference D architecture trained for 500 epochs. Bottom: Downstream task performance compared to baselines predicting epochs, accuracy, generalization gap and initialization.
Model Zoo Generating Factors. Prior work discusses the impact of random seeds on properties of model zoos. While Yaak al. 2019 use multiple random seeds for the same hyper-parameter configuration, [Unterthiner et al. 2020 explicitly argue against that to prevent information leakage between samples. To disentangle the generating factors (seeds and properties, we have created five zoos with approximately the same number of CNN models trained on MNIST MNIST-SEED varies only the random seed (1-1000), MNIST-HYP-1-PIX SEED varies the hyper-parameters with one fixed seed per configuration (similarly to [Unterthiner et al. 2020)). To decouple the hyper-parameter configuration from one specific seed, MNIST -TEST-HYP-1RRAT --AN---A---------- SEED draws random seeds for each hyper-parameter configuration. To investigate the influence of repeated configurations, MWIST-HYP-5- -Y----FII--------- -EED and parameter configurations we add 5 models with different seeds, either 5 fixed seeds or randomly drawn seeds.
and model
for each hyper-
4.2 Training and Testing Setup
Architectures. We evaluate our approach with different types of architectures, including Eg, ED, E.D and E,,,D as detailed in Section 3 The encoder E and decoders D in the FFN baseline are symmetrical 10 -FC- each and linearly reduce dimensionality for the input to the latent space z. and TETRIS-HYP zoos, we used attention blocks with attention head each, token dimensions of 128 and FC layers in the attention module of dimension 512. On the larger zoos, we up to 4 attention heads in attention blocks, token dimensions of up to size 800 and FC layers in the attention module of dimension 1000. For the combined losses, we evaluate different 3 E [0.05, 0.85).
Considering the attention-based encoder and decoder, on the TETRIS-SEER
Hyper-rrpressnntation Learning and Downstream Tasks. We apply the proposed data augmen- tation methods for representation learning (see Section 2).. We run our representation learning algorithms for up to 2500 epochs, using the adam optimizer [Kingma and Ba 2014), a learning rate of le-4, weight decay of le-9, dropout of 0.1 percent and batch-sizes of 500. In all of our experiments, we use 70% of the model zoos for training, 15% for validation and 15% for testing. We use checkpoints of all epochs, but ensure that samples from the same models are either in the

---

TETRIS-SERS
W
PCA|
PCA. PCA, Um
s(W)
E,D
W
PCA,
PCA,
PCA,
Um
s(W)
E&D
Eph
55.4
46.9
77.8
93.5
0.00
96.4
97.0
Eph
02.8
02.7
0.04
13.1
0.03
16.3
16.7
Ace
49.1
23,9
74.7
74.9
0.01
86,9
90.2
Acc
11.0
14.0
0.81
73.8
0.7
80.2
83.7
Ggap
43.9
28.3
72.9
75.7
0.01
81.0
81.9
GGap
12.9
12.9
0.31
76.0
0.3
79.2
81.6
Fco
26.6
0.07
52.8
49.1
0.02
62.5
70.7
LR
-477
-3.2
-1.3
-0.4
-1.4
53,4
51.1
Fei
41.4
30.2
48.6
58.6
0.01
63.1
75.9
Act
74.7
72.7
45.1
74.2
45.9
73.6
86.9
Foz
41.5
0.06
38.3
38.5
0.01
60.9
69.4
Init
38.5
36.4
32.7
35.4
33.6
46.6
47.1
Foa
53.9
44.6
73.8
72.2
0.00
75.2
86.6
Table 5: R² given in %. Left. Reconstruction, epoch, accuracy, generalization gap and class-wiss F-scores prediction. Right. Epoch, accuracy, generalization gap, learning rate, seed, activation function and initialization prediction.
train, validation or the test split of the zoo. As quality metric for the byper-repressentatinn learning, track the reconstruction Rº the validation split of the and report the reconstruction R² on the test split. As a proxy for how much useful information is contained in the hyper-represennation, evaluate downstream tasks as described in Section 3] To ensure numerical stability of the solution to the linear probing, we apply Tikhonov regularization (Tikhonov and Arsenin, 1977] with regularization parameter a in the range [le-5, 1e3] (we choose by cross-validating over the 12 score of the validation split of the zoo) and report the 1o2 score of the test split of the zoo. To minimize the cross entropy loss for the categorical ryper-parameter prediction, we use the adam optimizer with learning rate of 1e-4 and weight-decay of 1e-6. The linear probing is applied to the same train-valideen wat----aaaddinttsssee splits as it is in our representation learning setup.
Experiments. We follow a setup for outto--distributiin experiments similar to (Unterthiner et al. 2020). We investigate how well the linear probing estimator computed over hyper- representations generalize to yet unseen data. Therefore, we use the zoos MNIST-HYP, FASHION-HY? CIFAR10-HYP and SVHN-HYL On each zoo, we apply our self-supervised approach learn their corresponding ayper-representations and fit linear probing estimator to each of them (in-distribution). We then apply both the hyper-representation mapper and the línear probing estimator of one zoo on the weights of the other zoos (out-of-distrite -iii-o--dditrbbution).. The target ranges and distributions vary between the zoos. Linear probe prediction may preserve the relation between predictions, but include bias. Therefore, we use Kendall's ' 7 coefficient as performance metric, which is a measure of rank correlation. It measures the ordinal association between two measured quantities [Kendall] 1938).
Baselines, Computing Infrastructure and Run Time. As baseline, we use the model weights (W). In addition, also consider PCA with linear (PCA)), cosine (PCA,), and radial basis kernel (PCA,), as well as UMAP (Um) [McInnes et al. 2018). We further compare to layer-wise weight-statistics (mean, var, quintiles) s(W) in [Unterthiner et al.) 2020 Bilertsen al., 2020]. As computing hardware, we use half of the available resources from NVIDIA DGX2 station with 3.3GHz CPU and 1.5TB RAM memory, that has total of 16 1.75GHz GPUs, each with 32GB memory. To create one small and medium zoo, it takes I to 2 days and 10 to 12 days. respectively. For one experiment over the small zoo it takes around hours learn the rnperrrppresentation on single GPU and evaluate on the downstream tasks. It takes approximately 1 day for the medium zoos and 2 to 3 days for the large scale zoos for the same experiment. The representation learning model size ranges from 225k on the small zoos to 65M parameters on the largest zoos. We use ray.tune [Liaw et al.) 2018 for hyperparameter optimization and track experiments with W&B [Biewald 2020
4.3 Results
Augmentation Ablation. To evaluate the impact of the proposed augmentations (Section ( for our representation learning method, we present an ablation analysis the TETRIS-SERS 200, in which we measure R22 for ED, E,D and E.. DJ We use 120 permutations, a probability 0.5 for erasing the weights, and zero-mean noise with standard deviation 0.05 (see T)) We find the permutation augmentation to necessary for generalization particularly under higher compression ratios. The additional samples generated with the permutation appear to effectively prevent overfitting
We leave out E, it does use reconstruction loss
7

---

MNIST-HYP
FASHION-HI -YPP
CIFARIO -EFPRIO-HYP
HYYP
SVHN-HYP
W
S(W)
E&D
W
S(W)
E.D
W
S(W)
E.D
W
s(W)
E.D
EPH
25.8
33.2
50.0
26.6
34.6
51.3
25.7
30.3
53.3
22.8
37.8
52.6
Acc
74.7
81.5
94.9
70.9
78.5
96.2
76.4
82.9
92.7
80.5
82.1
91.1
GGAP
23.4
24.4
27.4
48.1
41.1
49.0
37.7
37.4
40.4
38.7
42.2
44.2
LR
29.3
34.3
37.1
33.5
35.6
42.4
27.4
32.3
44.7
24.5
33.4
49.1
22-RBG
12.5
16.5
20.1
11.9
16.3
25.0
08.7
13.8
28.0
09.0
13.6
28.0
DROP
28.5
19.2
35.8
26.7
21.3
38.3
16.7
16.5
33.8
09.0
14.6
23.3
TF
03.8
07.8
15.9
08.
08.2
22.1
08.4
06.9
35.4
03.2
08.8
21.4
ACT
88.6
81.1
88.7
89.8
82.4
90.1
88.3
80.3
90.0
86.9
78.8
87.2
INIT
94.6
72.0
80.6
95.7
76.5
86.7
93.5
73.3
82.6
91.0
73.0
82.8
OPT
76.7
65.4
66.4
79.9
67.4
73.0
74.0
65.5
71.0
72.5
68.2
72.3
Table 6: Top 7 Rows. R2 score in % for Eph, Acc, GGap, LR fg-reg, Drop and TF prediction.
Bottom 3 Rows. Accuracy score for Act, Init and Opt prediction.
of the training set. Without the permutation augmentation, the test performance diverges after few training epochs. Frasing further improves test performance and allows for extended training without overfitting. The addition of noise yields inconsistent results and is difficult to tune, so, we omit it in further experiments.
Architecture Ablation. The different architectures are compared Table I The results show that within the set of used NN architectures for hyper-represennttion learning, the attention-based architectures learn considerably faster, yield lower reconstruction error and have the highest per- formance the downstream tasks compared to the FFN-based architectures. We attribute this to the attention modules, which are able to reliably capture long-range relations on complex data due the global field of visibility in each layer. While tokenizing each weight individually (Attw) is able to learn, the computational load is significant, even for a small zoo, due to the large number of tokens in the sequences. The memory load prevents the application of that encoding zoos. We find Att embedding all weights of one neuron (or convolutional kernel) to one token in combination with compression tokens shows the overall best performance and scales larger architectures. Compression tokens achieve higher performance, which we confirm on larger and more complex datasets. The dedicated token gathers information from all other tokens of the sequence in several attention layers. This appears to enable the typer-representation to grasp more relevant information than linearly compressing the entire sequence. On the other hand, compression tokens are only an advantage, if their capacity is high enough, in particular higher than the bottleneck.
larger
evaluate the usefulness of the superviud learning tasks (Section 3 The application of purely contrastive loss in Eg, learns very useful representations for the downstream tasks. However, E, heavily depends on expressive projection heads and by design cannot reconstruct samples to further investigate the representation. Pure reconstruction ED results in embeddings with low reconstruction loss, but comparably low performance on downstream tasks. Among our losses, the combination of reconstruction with contrastive loss as in E,D, provides hyper-reeressentaiooss
superviper Learning Ablation. In Table
that have the best overall performance. The variation E.,.D. D. is closer to ED in general performance, but still outperforms it on the downstream tasks. The addition of contrastive loss with projection head helps to pronounce distinctiveness, so that the nyper-representations are good at reconstructing the NN weights, and in revealing properties of the NNs through the encoder. Empirically, we found that on some of the larger zoos, Eet D performed better than E, D. On these zoos, it appears that E D is most suitable for homogeneous zoos without distinct clusters, while E.D is suitable for zoos with more sub-structures, compare Figure Table
4
We therefore applied both learning strategies on all zoos and report the more performant one.
Zoo Generating Factors Ablation. Figure 4 visualizes the weights of the zoos, which contain models of all @ [6] In Table we report numerical properties. Only changing the seeds appears to result in homogeneous development with very high correlation between s s(W) and the properties of the samples in the zoo, as previous work already indicated (Schürbolt and Borth 2021]. Varying the hyper- perrmaaresssrss reduces the correlation. With fixed seeds, we observe clusters of models with shared initialization method and activation function. Quantitatively we obtain lower VAR, and high
Further visualizations be found in the Appendix Section C
8

---

MNIST-HY
FASHION- -HYP
SVHN-HYE HYPP
CIFAR1O --HPP
W
S(W)
De..D
W
s(W)
E D
W
S(W)
E D
W
5(W)
Eet D
MNIST-HYP
.36
.29
.36
.21
.14
.27
.26
.12
.23
-.01
-.04
.02
FASHION-HYP
-.02
.08
.02
.54
.48
.56
.06
.14
.01
.07
.10
.27
SVHN-HY*
.05
.15
-004
-.02
.27
.10
.44
.34
.45
-00
.08
.10
CIFAR1O-HYP .11
.09
.06
.38
.36
.39
.14
.14
.15
.41
.28
.35
Table 7: Kendall's r score for the generalization gap (GGap) prediction. We train estimators for each zoo (rows) and evaluate on all zoos (columns). The block diagonal elements contain the in-distribution prediction values. The remaining values are for out-of-distributt prediction.
predictive value of W for the initialization method. That seems a plausible outcome, given that the architecture and activation function determines the shape of the loss surface, while the seed and initialization method decide the starting point. Such clustering appear to facilitate the prediction of categorical characteristics from the weights. We observe similar properties in the zoos of [Unterthiner et al. 2020], see Appendix Section C. Initializing models with random seeds disperses the clusters, compare Figure and Table While VAR between 1 fixed and I random seed comparable, VAR, is considerably smaller with fixed seed, the predictive value of W for the initialization methods drops significantly. Random seeds also appear to make both the reconstruction as well as NN property prediction more difficult. The repetition of configurations with five seeds hinders shortcuts in the weight space, make reconstruction and the prediction of characteristics harder. Thus, we conclude that changing only the seeds results in models with very similar evolution during learning. In these zoos, statistics of the weights perform best. In contrast, using one seed shared across models might create shortcuts in the weight space. Zoos that vary both appear to be most diverse and hardest for learning and NN property prediction. Across all zoos with hyperparameter changes, learned hyper-representatioos significantly outperform the baselines.
Downstream Tasks. We learn and evaluate our hyper-repressenatioos on 11 different zoos: TETRIS-SEED, TETRIS-HYP, 5 variants of MNIST, MNIST-HYP, FASHION-HYP. CIFAR10-HYP and SVPN-HYP. We compare to multiple baselines and s (W). The results are shown in Tables and 6 On all model zoos, ryperrrepresennttiion learn usefuu features for the downstream tasks, which outperform the actual NN weights and biases, all of the baseline dimensionality reduction methods as well as s(W) [Unterthiner et al., 2020]. On the TETRIS-SEED and MNIST-SEED model zoos, s(W) achieves high R scores on all downstream tasks (see Table 5 left). As discussed above, these zoos contain strong correlation between s(W) and sample properties. Nonetheless, learned hyper- representations achieve higher R² scores on all characteristics, with the exception of ANIST-SEED Eph, where E is competitive to s(W). On TETRIS-HYP. the overall performance of all methods is lower compared to TETRIS-SEED (see Table right), making it the more challenging zoo. Here, too, hyper-repressnnatiinss have the highest R score all downstream tasks, except for the LR prediction. On MNIST-HYP FASHION-HYP. CIFAR10-HYP and SVHN-HYP, hyper-represennatiios outperform s(W) on all downstream tasks and achieve higher Ri² score compared to W on the pre- diction of continuous hyper-parameters initialization method and optimizer, the weight space achieves the highest It2 scores (see b). We explain this with the small amount of variation in the model zoo (see Section which allows to separate these properties in weight space.
and activation prediction. On the remaining categorical
Outto--Distribution Prediction. Table 7 shows the out-of-distribution results for generalization gap prediction which is very challenging task. The task is rendered more difficult by the fact that many of the samples which have to be ordered by their performance are very close in performance. As the results show, hyper-representations are able to recover the order of samples by performance to significant degree. Further, hyper-representations outperform the baselines in Kendall's r measure in the majority of the results. This verifies that our approach indeed preserves the distinctive information about trained NNs while compactly relating to their common properties, including the characteristics of the training data. Presumably, learning representations on zoos with multiple different datasets would improve the out-off-distribttio capabilities even further.
In Appendix Section D. also give results for other tasks, including epoch id and test accuracy prediction

---

5
Related Work
There is ample research evaluating the structures of NNs by visualizing activations, e.g. Zeiler and Fergus 2014 4 Karpathy et al. 2013 Yosinski et al. 2015], which allow some insights in the patterns of, e.g., the kernels of CNNs. Other research evaluated networks by computing degree of similarity between networks. [Lakkso and Cottrell 2000) compared the activations of NNs by a measure of "sameness"
[Li et al., 2013] computed correlations between the activations of different nets. [Wang] et al. 2018 tried to match the subspaces of the activation spaces in different networks [Johnson
2019 which showed to be unreliable. (Raghu al., 2017 Morcos al. 2018 Kornblith al.
2019] applied correlation metrics to NN activations in order to study the learning dynamics and
compare NNs. [Dinh al. 2017 link model properties to characteristics of the loss surface around the minimum solution. In contrast comparing models with similarity metrics, other efforts map models to an absolute representation. (Jia ct al. 2019] approximated the space of DNN activations with a convex hull. (Jiang et al. 2019) also used activations to approximate the margin distribution and predict the generalization gap. [Corneanu et al., 2020] proposed persistent homology by using a connectivity patterns in the NN activation, and compute topological summaries.
While previously mentioned related work studied measures defined on the activations for insights about the NN characteristics, the methods applied on populations of NN weights have not received much attention for the same purpose. [Martin and Mahoney| 2019 relate the empirical spectral density of the weight matrices to accuracy. indicating that the weight space alone contains relevant information about the model. [Eilertsen et al., [220) evalutaed a classifier for hyper-parameter prediction directly from the weights. In contrast, we learn a general purpose ryperrrepresenntations in self-supervised fashion. [Unterthiner et al. 2020 proposed layer-wise statistics derived from the NN models to predict the test accuracy of NN model. In [Schürholt and Borth [021]. subspaces of the weights space of populations of NNs are investigated for model uniqueness and epoch order.
In the proposed work, we model associations between the different weights in NN using an attention- based module. This helps us learn representations that compactly extract the relevant and meaning- ful information while considering the correlations between the weights in an NN.
Conclusions
In this work, we present a novel approach to learn Supper-rpressenttionnn from the weights of neural networks. To that end, we proposed novel augmentations, self-supervised learning losses and adapted multi-head attention-based architectures with suitable weight encoding for this domain. Further, we introduced several new model zoos and investigated their properties. We showed that not only learned neural networks but also their hyper-represennations contain the latent footprint of their training data. We demonstrated high performance on downstream tasks, exceeding existing methods in hyper-parametee test accuracy, and generalization gap prediction and showing the potential in an out-of-distribution setting.
Acknowledgements Leading up to this paper. there were many discussions with colleagues, which we would like acknowledge. We are particularly grateful Marco Schreyer, Xavier Giró-i-Nieto. Pol Caselles Rico and Diyar Taskiran.
Funding Disclosure
This work is supported by the University of St.aalen Basic Research Fund,
References
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: / /www. wandb comm Software available from wandb.com.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. arXiv:2104.13: 13478 [cs, stat), May 2021. URL
13478 arXiv: 2104.13478.
Ting Chen and Lala Li. Intriguing Properties of Contrastive Losses. arXiv:201 1.02803 [es, stat), November 2020. URL httt:://arxxt wwwpaawwwwwww..w./aa.aaaa/a abtp//a/......//. 02803 arXiv: 20.100..
10

---

Ting Chen, Simon Kormblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. arXiv:2002.05700 [cs, stat], June 2020a. URL http://arxiv..r org httgg/bs/22000 05709 arXiv: 2002.05709.
Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. CVPR, page 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Momentum Contrastive Learning. arXiv:2003.0m 04297 [es], March 2020b. (RLtttt:://rrt org/ abs/ 2003. 04297
arXiv: 2003.04297.
Ciprian A. Corneanu, Sergio Escalera, and Aleix M. Martinez. Computing the Testing Error Without Testing Set. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2674-2682, Seattle, WA, USA, June 2020. IEEE. ISBN 978- 1778---72817-1688-1 doi: 10. 109/ CYPR42600.2020.002) URL httphhhts////tt
/ttp::tt////eeeeelloorr /iepexeeeeeepeeeeeeeeeee ieee. wtt///ocueet//115ee66
Jacob Devlin, Ming-Wai-ee Weig Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Compulational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp Minima Can Generalize For Deep Nets. arXiv: [cs], May 2017. URL http::////aaxii org/abs////////eooeo/..ooeee Xrvi:: 1703.04933.
woggabbs1703... 1101. 04933
A. Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. Dehghami, Matthias Minderer, Georg Heigold, S. Gelly, Jakob Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.
Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski. Jonas Unger, and Anders Ynnerman. Classifying the classifier: dissecting the weight space of neural networks. Siti:::002.00662 [cs], February arxiv: 2002.05688.
2020. Jttpp///://arxi org/ abs/2002. 05688
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Ian J. Goodfellow, Oriol Vinyals, and Andrew M. Saxe. Qualitatively characterizing neural network optimization problems, 2015.
Jcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach self-supervised Learning. arXiv:2006.077 07733 [cs, stat], September 2020. URL arXiv: 2006.07733.
btttp//arxiv..or www.oogg//////a.ooooooooooe htsr/bbb/20005 2006. 07733
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetorrss. CoRR, aabs//609.0910/ 2016. URL
http://arxiv http://wxxvvvvgg.e//////
org/abs.////.ooooo. /b/////////// 1609/1669.... 000.6999.01.66... 09106
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture, 2018.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General Perception with Iterative Attention. arXiv:2103.032000 [cs, eess], March 2021. arXiv: 2103.03206.
URL httpt:::/aarti..og//b//////////
03206
Yuting Jia, Haiwen Wang, Shuo Shao, Huan Long, Yunsong Zhou, and Xinbing Wang. On Geometric Structure of Activation Spaces in Neural Networks. 1rX.vv/904.0/333 [cs, stat), April 2019. URL arXiv: 1904.01399.
battp://arxiv... hasg://b////
abs//904..111..... 1904.994..00 01399
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the Generalization Gap in Deep Networks with Margin Distributions. arXiv: 1810.00/13 [cs, stat], June 2019. URI http:// /tappv///axii.../ org/ htte//b//1/0 1010/ 001100 arXiv: 1810.00113.
11

---

Jeremiah Johnson. Subspace Match Probably Does Not Accurately Assess the Similarity of Learned Representations. arXiv: 0w1199910088... [cs, math, stat], January 2019. URL http:////aa org/ abs/1901 00884
arXiv: 1901 00088.4
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and Understanding Recurrent Net- works.
[cs], June 2015. URL
02000 arXiv:
1506.02078.
M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
Diederik Kingma and Jimmy Ba. abs/1412.6980, 2014.
Adam: A method for stochastic optimization. CoRR,
Diederik Kingma and Max Welling, Auto-Encoding Variational Bayes. arXiv: [cs, stat], May 2014. URL http://arxiv.org// wrgg/bs///1/// 111.111126611 arXiv: 1312.6114.
Simon Korblithh Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural
Network Representations Revisited. arXiv: 1905, .0414 [es, q-bio, stat], May 2019. htth
//arxiv..ogg aaaaaab//1955 aaxxiv.org/abs/1905..
00414
arXiv: 1905.00414.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research). URL http::////// /ttp::///www.. cs. toronto edu/"kriz/ cifar html
Aarre Laakso and Garrison Cottrell. Content and cluster analysis: Assessing representational similarity in neural systems. Philosophical Psychology. 13(1):47-76, March 2000. ISSN 0951- 5089, 1465-394X. doi: 10. 10010000000555000000000000000000 URL http:://t:/ii tandf Saandooliing ine com/
Yann LeCun and Corinna Cortes. The MNIST database of handwritten digits. hRthttp:///ann
lecun. cwm/coom/cii/eee exdb//mnst//
Yann LeCun and Ishan Misra.
Self-supervised
learning: URL http::://aa...ccbbbooo com/blom
The dark mat-
ter of intelligence, April 2021. self self-supervised- wewpeiiiiidd-eeaannn--ee the-
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent Learning: Do different neural networks learn the same representations? arXiv:/511 1.07543 [cs], November 2015. URL
http/////11111.0.5.. 07544 arXiv: 1511.07543.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint 2018.
Charles H. Martin and Michael W. Mahoney. Traditional and Heavy-Tailed Self Regularization in Neural Network Models, arXiv: 0990118876 [cs, stat], January 2019. URL http::/arxxi..rrg abs/1901 08276, arXiv: 1901.08276.
Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation fao2:0302033 [cs, stat], December 2018. URL
and Projection for Dimension Reduction. 1802. 1802.03426
hctp://arxiv.ort org/ 2at.//2/s//000....
arXiv: 1802.03426.
Ishan Misra and Laurens van der Maaten. Self Supe-Supe Learning of Pretext-Invarian Represen- tations. ArXiv:1912.000 01999/ [cs], December 2019. URL
wwt.s//rrrabbbbb/bbbe// 01991
arXiv: 1912.01991.
Ari S. Morcos. Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. arXiv: 1800180.... 05759 [cs, stat], June 2018. URL. http:// arXiv: 1806.05759.
arxiv. wwt.v...rg/bb.... 1806. 05759
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
12

---

Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. lar Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. arXiv: 170.00500066 [cs, Stat], June 2017. http:tt::... httttp:/aarxi..oor wwtp://.ggga/bb/t/e//e/oo /1/// 1706. 05806 1706.05806.
SVCCA: Singu-
arXiv:
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, and Philip Bachman. Effta--ffic Reinforcement Learning with URL http::/arxivv org/ httggabb/2000../ 05929
Representations, ICLR, May 2021. arXiv: 2007.05922
Konstantin Schürholt and Damian Borth. An investigation of the weight space to monitor the training progress of neural networks, 2021. htt.pttts://aaxivv.ooo absp//bb//200... 10424
Connor Shorten and Taghi M. Khoshgoftaar. A survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6(1):60, December 2019. ISSN 2196-1115. doi: 10.1
s40537-019-0197-0. / htt.://w...........o.ooooooooooa cww con/articles/10 1116/8406337009--1999
Andrey N. Tikhonov and Vasiliy Y. Arsenin. Solutions of ill-posed problems. V. H. Winston & Sons, 1977. Translated from the Russian, Preface by translation editor Fritz John, Scripta Series in Mathematics.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting Neural Network Accuracy from Weights. arXiv:2002. //48 [cs, stat], February 2020. URL arXiv: 2002. 14448
httt::/arxiv..o wtw.a///02... 2042.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30, pages 5998 6008. Curran Associates, Inc., 2017.
L Liwei Wang. Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hopcroft. Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation. 1xXXv:8810.11733 [cs, stat], October 2018. URL httt:://taiiiii org/ aabs 11750, arXiv: 1810.11750.
Sewall Wright. Correlation and causation. J. agric. Res., 20:557-580, 1921.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
Scott Yak, Javier Gonzalvo, and Hanna Mazzawi. Towards Task and Architecture-Indeeendent Generalization Gap Predictors. arXiv: 10X0009906001500 [cs, stat], June 2019. URL http::/arxx arXiv: 1906.01550.
org/abs/1906. 01550
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding Neural Networks Through Deep Visualization. 15X.::950600657 [cs), June 2015. http://///aaxi arXiv: 1506.06579.
org/abs/1506. 06579
Matthew D. Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision
ECCV 2014, volume 8689, pages 318-833. Springer International Publishing, Cham, 2014. ISBN 978-3-319-10589-5 978-3-319-1055999 //link. laonk.springger htmpgeerrooommoommom 10. 0000000097788--31111050------000 --990-105000--53
doi: 10. 1111100097778831-1-0000000
URL http
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta-
tion. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
13


Reviews:
[{"review_id": "ui9zeJzKAzA", "reviewer": "Reviewer_Te8F", "summary": "The authors propose three data augmentations and one attention module to learn a representation of the network weights to predict the network accuracy.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The authors propose three data augmentations. The only novel one seems to be the permutation augmentation. The other two (erasing, noising) seems to be proposed for general self-supervised learning before. The attention modules is also not new. But I think applying self-supervised learning to neural network accuracy prediction itself is new.\n\nQuality: \n1. The motivation is still not very clear. If predicting neural network accuracy from the weights can help understand fundamental characteristics that make NN successful, how does using a self-supervised representation help more on the understanding of the fundamental characteristics. Maybe the authors can give some examples on which understanding the authors gain more on the fundamental characteristics.\n2. I am not sure why we need to use self-supervised representations. I guess for a different random seed, we may get a different model. And so, the data is unlimited.\n3. The experiment results seems to be okay. Self-supervised representation seems to be get better performance on predicting the network accuracy. \n4. The equation under line 96 does not look correct. I do not think matrix multiplication and sigmoid function is exchangeable. Also, I am not very sure about why we want this permutation augmentation.(but this does not affect my score)\n\nclarity:\n1. The paper is clearly written.\n\nSignificance:\n1. It's not clear to me \"predicting neural network accuracy from the weights can help understand fundamental characteristics that make NN successful\". If the authors can give some examples, it will make the contribution more significant.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors described the limitations of the work. And as they mentioned in the checklist, potential negative societal impact does not apply to a single method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "gEcLHmpNh1K", "reviewer": "Reviewer_VBuX", "summary": "The authors propose self-supervised algorithms to predict the model characteristics from the neural network weights. Authors experimented with different self-supervised objectives based on autoencoder, contrastive learning (NT_Xent), and modified contrastive loss. They utilize different architectures based on Fully-Connected Feed-Forward Networks (FF) and transformer (Att). \n\nExtensive experiments show that neural representation learned with self-supervision recovers latent structure from the neural network's weights. The learned representations outperform approaches based on features from weights, different modifications of PCA, UMAP, and layer-wise weight statistics (Unterthiner, 2020) in multiple downstream tasks and out-of-distribution generalization.\n\nAdditionally motivated by the unresolved question: whether model zoo should contain multiple seeds or not due to possible leakage between samples, - they generate new model zoos based on MNIST and show that seed variation for each set of the hyperparameters is beneficial to learn better representation to recover NN's properties.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- This work is more about a toy benchmark for learning from the NN's weights and applying the self-supervision to the specific domain. The experiments and ablations are extensive.\n- The experiments clarified the question about using seeds in model zoos.\n\nWeaknesses:\n\n- There is no supervised baseline, and I fail to see a motivation for self-supervision. The labels and data can be easily generated and synthesized. Most of the SOTA results are still achieved with Supervised models. One could take the encoder from your architectures then add linear layers to train for regression and classification in a multi-task fashion. Then one can take representation before linear classification layers and perform linear evaluation or visualization with UMAP. \n\n- The experiments show that self-supervision works for this domain. However, I do not see them particularly useful for real case scenarios and insights: \n  - It will be interesting to see a different evaluation. For example, the researcher has already trained many models. Then the researcher would use the proposed characteristic prediction model to evaluate new models. Finally, let's ask how close the prediction of the characteristic model will be to the actual performance of the fully converged model given the new weights at the initialization step or after a couple of epochs. The main goal is to save compute and time while trying to find a better model.\n  - The experiments do not uncover new insights about how neural networks work. \n\nMinor:\n\n- Lines 298-299: \" that changing only the seeds results in models with very similar evolution during learning,\" Could we consider seed just as a data augmentation?\n\n- Line 84: It is unclear what the learning procedure for the model is and how it is different from hyper-parameters.\n\n- Line 124: The equation for the modified contrastive loss, it seems, has an only part for positive pairs but not a negative part. I am not sure if it can be called contrastive, and it looks like a cosine similarity that has been used in at least two works as SimSiam [1] and SPR [2]. \n  - [1] Chen, Xinlei, and Kaiming He. \"Exploring simple siamese representation learning.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2021.\n  - [2] Schwarzer, Max, et al. \"Data-Efficient Reinforcement Learning with Self-Predictive Representations.\" *International Conference on Learning Representations*. 2020.\n  - The equation has missing parathesis for the exponent.\n\n- Line 216: Given the text, it seems that the model selection was performed on the test set, which is supposed to be a hold-out set but not as a validation set.\n\n- Line 258: I do not purely agree that vanilla transformer does well with long-range relations, as the community is still trying to solve these questions (e.g., Transformer-XL [3], XLNet [4], Feedback Transformer [5]). The authors should clarify this.\n  - [3] Dai, Zihang, et al. \"Transformer-XL: Attentive Language Models beyond a Fixed-Length Context.\" *ACL (1)*. 2019.\n  - [4] Yang, Zhilin, et al. \"Xlnet: Generalized autoregressive pretraining for language understanding.\" *Advances in neural information processing systems* 32 (2019).\n  - [5] Fan, Angela, et al. \"Addressing some limitations of transformers with feedback memory.\" *arXiv preprint arXiv:2002.09402* (2020).\n\n### Originality:\n\n- To my knowledge, no one has applied self-supervision for this particular domain.\n\n### Quality: \n\n- Overall, it is a good paper. However, there are some issues I mentioned as a Minor.\n\n### Clarity: \n\n- Overall, the paper is well written. However, the location of tables and figures in the document is quite distant from the text.\n\n### Significance:\n\n- I do not find this work a significant contribution due to the lack of new insight or phenomena on how neural networks work compared to related literature (e.g., [6]).\n  - Nguyen, Thao, Maithra Raghu, and Simon Kornblith. \"Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth.\" International Conference on Learning Representations. 2021. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "FCqXFGerQJW", "reviewer": "Reviewer_Rn7M", "summary": "This paper proposes a self-supervised representation learning approach to uncover properties from trained neural network (NN) weights and biases. Representations are learnt through encoder-decoder architectures on the weights, using various losses. Attention-based modules are used to find interactional structure between different weights of a NN, resulting in compressed representations in different manners. To this end, data augmentation based on permutation augmentation is proposed; random erasing and noise augmentation are also explored. Empirics are performed on trained models from MNIST, Fashion-MNIST, SVHN, CIFAR-10 as well as synthetic datasets. Results explore data augmentation, architecture ablation, out-of-distribution prediction and various downstream tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and clear. The topic is relevant to the literature and its goal relevant. The methodology is well described and seems sound. The experimental setting is detailed and experiments are extensive. The addition of a few details would improve the quality of the paper:\n- the amount of compute needed is mentioned. Could you also report model sizes?\n- references to the experiments developed in the appendix could be added (eg. on the impact of the compression ratio)\n\nI recommend an accept for this paper.\n\nTypos:\n- line 57: \", so\" instead of \". So\"\n- line 114: \"a self-supervised fashion\"\n- citing types are often mixed up (eg lines 113, 120)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impacts are discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "CwGqJtODUa", "reviewer": "Reviewer_6PDp", "summary": "This is a very interesting paper. I need to be sure that I understand the main concept and review again carefully.\n\nMy review rating is really a placeholder, please **ignore my review rating**. I will give a true rating after the authors response to my queries.\n\nLet me know if my understanding of this work is accurate. In a nutshell:\n\n1. Train a zoo of M Neural Net, each with weights w_i, i=1,. . M.\n2. All of the w_i has the same dimension N\n3. Consider one of w_i as one 'data point' and devise auto encoder kind of network to extract the latent space of w_i\n4. let latent space of w_i be z_i, z_i=g_\\theta(w_i)\n5. analyse these z_i using a linear regressor on downstream tasks -- line 174 in appendix\n6. downstream tasks are accuracy prediction, generalisation gap etc\n\nessentially if the latent variables z_i of the weights correlates well with (line 174 in appendix) downstream tasks, e.g. good R^2, we can say that the NN zoo (statistically) picks up good information about the tasks at hand.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I see this paper as more of a discovery kind of paper than a paper for an immediate real world application. However I wish to ask if the author has some eventual real world applications in mind that inspire this work.\n\nThe motivation of this work was discussed in lines 46-63. Would be good to strengthen this part of the paper further.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NIL", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a robust discussion between the reviewers and the authors and some discussion amongst the reviewers themselves. The post-rebuttal consensus is that we should accept this work. From the discussion with reviewers, some of the constructive feedback for a final version would be to show more evidence of usefulness in real case scenarios. It would also be useful to explore the directions suggested by reviewer 6PDp in their conversation with the authors.
Decision: Accept (Poster)