Analyzing Populations of Neural Networks via Dynamical Model Embedding
Authors: Jordan Cotler
Kai Sheng Tai
Felipe Hernandez
Blake Elias
David Sussillo
Venue: ICLR 2022 Submitted (2022)
Keywords: dynamics
RNNs
model averaging
model clustering
CNNs
semi-supervised learning

Abstract: A core challenge in the interpretation of deep neural networks is identifying commonalities between the underlying algorithms implemented by distinct networks trained for the same task.  Motivated by this problem, we introduce \textsc{Dynamo}, an algorithm that constructs low-dimensional manifolds where each point corresponds to a neural network model, and two points are nearby if the corresponding neural networks enact similar high-level computational processes. \textsc{Dynamo} takes as input a collection of pre-trained neural networks and outputs a \emph{meta-model} that emulates the dynamics of the hidden states as well as the outputs of any model in the collection.  The specific model to be emulated is determined by a \emph{model embedding vector} that the meta-model takes as input; these model embedding vectors constitute a manifold corresponding to the given population of models. We apply \textsc{Dynamo} to both RNNs and CNNs, and find that the resulting model embedding manifolds enable novel applications: clustering of neural networks on the basis of their high-level computational processes in a manner that is less sensitive to reparameterization; model averaging of several neural networks trained on the same task to arrive at a new, operable neural network with similar task performance; and semi-supervised learning via optimization on the model embedding manifold.  Using a fixed-point analysis of meta-models trained on populations of RNNs, we gain new insights into how similarities of the topology of RNN dynamics correspond to similarities of their high-level computational processes.

Under review as a conference paper at ICLR 2022
ANALYZING POPULATIONS OF NEURAL NETWORKS VIA DYNAMICAL MODEL EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
A core challenge in the interpretation of deep neural networks is identifying com- monalities between the underlying algorithms implemented by distinct networks trained for the same task. Motivated by this problem, we introduce DYNAMO, an algorithm that constructs low-dimensional manifolds where each point corre- sponds to neural network model, and two points nearby if the corresponding neural networks enact similar high-level computational processes. DYNAMO takes input collection of pre- pre-trained neural networks and outputs meta-mode that emulates the dynamics of the hidden states as well as the outputs of any model the collection. The specific model to be emulated is determined by model em- bedding vector that the meta-model takes as input; these model embedding vectors constitute manifold corresponding the given population of models. We apply DYNAMO to both RNNs and CNNs, and find that the resulting model embedding spaces enable novel applications: clustering of neural networks on the basis of their high-level computational processes in a manner that is less sensitive to repa- rameterization; model averaging of several neural networks trained on the same task to arrive at new, operable neural network with similar task performance; and semi-supervised learning via optimization on the model embedding space. Using fixed-point analysis of meta-models trained on populations of RNNs, we gain new insights into how similarities of the topology of RNN dynamics correspond to similarities of their high-level computational processes.
INTRODUCTION
A crucial feature of neural networks with fixed network architecture is that they form a manifold by virtue of their continuously tunable weights, which underlies their capability to trained by gradient descent. However, this conception of the space of neural networks is inadequate for un- derstanding the computational processes they perform. For example, two neural networks trained to perform the same task may have vastly different weights, and yet implement the same high-level algorithms and computational processes (Maheswaranathan et al. 2019b).
In this paper, we construct algorithm which provides alternative parametrizations of the space of RNNs and CNNs with the goal of endowing geometric structure that is more compatible with the high-level computational processes performed by neural networks. In particular, given set of neural networks with possibly different architectures (and possibly trained on different tasks), we find a parametrization of low-dimensional submanifold of neural networks which approximately interpolates between these chosen "base models", as well as extrapolates beyond them. We can use such model embedding spaces to cluster neural networks and even compute model averages of neural networks. A key feature is that two points in model embedding space are nearby if they correspond to neural networks which implement similar high-level computational processes, manner to be described later. In this way, two neural networks may correspond to nearby points in model embedding space even if those neural networks have distinct weights or even architectures.
The model embedding space is parametrized by low-dimensional parameter 0 E Rd and each base model is assigned a value of 8 in the space. This allows us to apply traditional ideas from clustering and interpolation to the space of neural networks. Moreover, each model embedding space has an associated meta-model which, upon being given a 8, rendered into an operable neural network. base model is mapped to some 0, then the meta-model, upon being given that 0, will emulate the

---

Under review as a conference paper at ICLR 2022
Model
Model Embedding Space
Meta-model((h)
2
Model
1
Model 2
(eta-model(Oa)
Model 2
Model N
ON
Meta-model(Ew)
2 Model
conceptual diagram of the DYNAMO algorithm and model embedding space. set of neural networks labelled 1, 1,.... N called the base models mapped corresponding points .1.... ON in the model embedding space. Two points in the model embedding space are nearby if they correspond neural networks with similar dynamies for their hidden and output states. There is an associated neural network called the meta-model which, when given a A; becomes neural network that emulates the hidden and output dynamics of the corresponding base model. The meta- model produces viable neural network for any given any value of 0, including points in the model embedding space that do not correspond any base model.
Figure 1:
he
Fn
he+1
he
F
M-1
n
2
2
Ghi
(ha
((ha..41)
2
@lee..))
((haaat1)
how
F
ha
hg
F
haw.+11
(O.. 22)
(0.. 2.)
The hidden states h2 of Fn are close to the hidden states ha of F after the transformation map V is applied. Right: The visible states G(ht) of Fn close the visible states Ghh....)
Figure 2: Left:
corresponding base model. See Figure I for a diagrammatic depiction. An interesting application that given two models assigned parameters 01 and 02. we can consider the averaged model corresponding to the value (01 + 02)/2. We find that this averaged model performs similarly on the task for which the base models the average were trained. We also use the model embedding space to extrapolate outside the space of base models, and find cases in which the model embedding manifold specifies models that perform better on task than any trained base model. Later on we form of semi-supervised learning.
will explain how this can be regarded
The rest of the paper is organized as follows. We first provide the mathematical setup for our gorithmic construction of model embedding spaces. After reviewing related work, we then present results of numerical experiments which implement the algorithm and explore clustering, model av- eraging, and semi-supervised learning the model embedding space. We further examine how topological features of the dynamics of RNNs model embedding spaces are reflective of classes of level computation processes. Finally, conclude with discussion.
DYNAMICAL MODEL EMBEDDING
2.1 MATHEMATICALL SETUP
In this section, we provide the mathematical setup for construction of model embedding spaces. We treat this in the RNN setting, and relegate the CNN setting Section of the Appendix.
2

---

Under review as a conference paper at ICLR 2022
Algorithm 1 DYNAMO algorithm
Input: Base models {(Pa, Gn)) loss functions {(Cn, L outputt ) datasets (Da I== output loss weight À 2 0
Output: Meta-model (F, G),, state maps [VoJ model embeddings (0, J Initialize networks F, G, (V Initialize model embeddings 8₂ =
for = =....... Tmax do
Sample base model (Fi, G;) uniformly at random
Sample input {2re} } from dataset D;
Compute base model hidden states } and outputs {G,(h,}}
Compute meta-model hidden states {ha ] and outputs (G(ha,x)
Compute mapped meta-model states (Vo(he,,))
Compute loss = L hiddeen + A outputt Update F, G, Vi, 0g using VĆ end for
return (F,G), (Vr). (0..)
Notation. Let RNN be denoted by F(x., h)) where x: is the input and h is the hidden state. We fur- ther denote the hidden-to-output map by Gh)). We consider a collection of N RNNs {FF, Gn)) } we call the base models which may each have distinct dimensions for their hidden states, but all have the same dimension for their inputs as well as the same dimension for their outputs. A sequence of inputs is notated as {2+) ] which induces sequence of hidden states by h++1 F(z,, h2) where the initial hidden state ho is given. A collection of sequences of inputs, possibly each with different maximum lengths T, is denoted by which we call an input data set. We suppose that each base model RNN has an associated input data set.
2.2 MOTA-SODELS FOR RNNS
Given a collection base model RNNs, we would like construct meta-model which emulates the behavior of each of the base models. In this case, the meta-model is itself an RNN with one additional imput O E IRd and a corresponding map F(O, x. h) whose output is the next hidden state. Given sequence of input states {ar)/=o- we have a corresponding sequence of output states hg,141 = F(8, æı, he.r) starting from an initial hidden state hg (which we suppose does not depend on 0). The meta-model also includes a hidden-to-outpar map G(h) that is independent of A.
For the meta-model (F, G) emulate particular base model (Fn, G,)) with respect its corre- sponding data set D,,, we consider the following criteria: there is some 0n, for which
G(ho...) x Gh)) for all and all input sequences in the data set
2. Va (ha,s) by for all 0 and all input sequences in the data set,
where V, transformation of a meta-model's 's hidden activity. We emphasize that 0,, and V. depend on the particular base model under consideration. The first criterion means that at some particular O,, the outputs of the meta-model RNN dynamics are close to the outputs of the base model RNN dynamics. The second criterion means that at the same O,, there is time-independent transformation V, (i.e., V,, does not depend on t) such the transformed hidden state dynamics of the meta-model are close to the hidden state dynamics of the base model. See Figure 2 for visualiza- tion. As depicted the figure, it is convenient to regard the meta-mode RNN as having input states (0n, As such, sequence of inputs (x:) is appended by 0,, to become ((0,, (t))==1
3

---

Under review as a conference paper at ICLR 2022
The desired properties of the meta- meta-model are enforeed in the loss function. Defining the functions
1
Couppur G,G,On] =
d (G Gr(h)))
T
1
Lhidden [F, F,, Vn]
(ho hell2
(2)
T
where d is some suitable distance or divergence, we can construct the loss function
[ (Caisseal [Fa 0ng Vxr] + X Couppll [F.G. 0.]]
(3)
where we average over the choice of sequence { coming from the input data set Dn.. Above, is a hyperparameter. Our aim is to minimize the above over suitable class of F, P,G. Vn. as well as On: this can be implemented computationally via the DYNAMO algorithm (see Algo- rithm As a side remark, it appears that a suitable alternative choice to equation 2 would be =t=1 ||ho... Wn, (ht,) where here Wn, is map from the hidden states of the base model to hidden states the meta-model. However, this would problematic since minimization may pressure W, to be the the zero map (or otherwise have outputs which small in norm) and ac- cordingly pressure the dynamics the meta-model to be the trivial (or have small norm). As such, opt to formulate Luidden as it is written in equation 2
Suppose we want the meta-model to be able to emulate an entire collection of base models {(Fn. G,)l. In particular, the meta-model will attempt to assign to the nth base model 6, and Vn so that the two criteria listed above are satisfies for that base model. These desiderata can be implemented by minimizing the loss function
N
C[F, 10, (V.)
1
Σ
E (o.)aD [ (Cn.iiiteenl 0n Vul + X An, [F, G, enl
N
(4)
In some circumstances, we may want to consider base models with distinct dimensions for their output states. For instance, suppose half of the base models perform a task with outputs in Rd and the other half perform a task without outputs in Rds To accommodate for this, we can have two hidden-to-outpur maps G1, G2 for the meta-model, where the maps have outputs in Rd and R& respectively. The loss function is slightly modified so that we use G₁ where compare the meta- model to the first kind of base model, and G2 when compare the meta-model to the second kind of base model. This construction generalizes to the setting where the base models can divided up into l groups with distinct output dimensions; this would necessitate l hidden-to-outpye functions G1,..., GR for the meta-model.
RELATED WORK
There is substantial body of work on interpreting the computational processes implemented neural networks by studying their intermediate representations. Such analyses can be performed either on individual models (Simonyan al. 2013 Zeiler & Fergus 2014; Lene & Vedaldi 2015) or on col- lections of models (Li et al.) 2016 Raghu et al.) Morcoss
Morcos et al. 2018 Kornblith et al. 2019). Prior work in this latter category has focused on pairwise comparisons between models. For exam- ple, SVCCA (Raghu et al. 2017) uses canonical correlation analysis to measure the representational similarity between pairs of models. While these methods can also be used derive model repre- sentations by embedding the pairwise distance matrix, our approach does not require the ((N2) computational cost of comparing all pairs base models. Moreover, DYNAMO yields an executable metal that can be called with model embedding vectors other than those corresponding to the base models.
There is related body of work in the field of computational neuroscience. CCA based techniques and representational geometry are standard for comparing neural networks to the neural activations of animals performing vision tasks (Yamins & DiCarlo 2016) as well motor tasks. In example
4

---

Under review as a conference paper at ICLR 2022
of the latter, the authors of (Sussillo al. 2015) used CCA techniques compare brain recordings to those of neural networks trained to reproduce the reaching behaviors of animals, while the authors of MMahssaaannahhan et al. 2019b) used fixed point analyses of RNNs consider network similarity from a topological point of view.
DYNAMO can be viewed
form of knowledge distillation (Hinton et al. 2015). ) since the outputs
of the base models serve as targets in the optimization of the meta-model. However, unlike typical instances of knowledge distillation involving an ensemble of teacher networks (Hinton et al. 2015 Fukuda et al. 2017) where individual model predictions are averaged to provide more accurate target labels for the student, our approach instead aims to preserve the dynamics and outputs of each individual base model. FitNets (Romero (015) employ form a knowledge distillation using maps between hidden representations to guide learning; this similar to our use of hidden state maps in training the meta-model.
Our treatment of the model embedding vectors {0,} as learnable parameters similar to the ap- proach used in Generative Latent Optimization (Bojanowski et al. 2018). which jointly optimizes the image generator network and the latent vectors corresponding to each image in the training set. Bojanowski et al. (2018) find that the principal components of the image representation space found by GLO are semantically meaningful. We likewise find that the principal components of the model embeddings found by DYNAMO discriminative between subsets of models.
Unlike methods such as hypernetworks (Ha et al. 2017) and LEO (Rusu et al. 2019) that use a model to generate parameters for separate network, approach does not attempt to reproduce the parameters of the base models in the collection. Instead, the meta-model aims to reproduce only the hidden states and outputs of base model when conditioned on the corresponding embedding vector.
The core focus of this work also differs from that of the meta-learning literature (Santoro et al., 2016 Ravi & Larochelle, [2017] Finn et al.) 20177] Munkhdalai & Yul (017), which primarily con- cerns the problem of few-shot adaptation when presented with data from a new task. Our empirical study centers on the post-hoc analysis of a given collection of models, which may or may not have been trained on different tasks. However, we remark that our exploration of optimization in low- dimensional model embedding space is related to LEO (Rusu et al., 2019), where compressed model representation leveraged for efficient meta-learning.
4 EMPIRICAL RESULTS
In this section, describe the results of our empirical study of meta-models trained using DYNAMO on collections of RNNs trained on NLP tasks, and collections of CNNs trained for image classifica- GRU where the model embedding
tion. For RNN base models, we parameterize the meta-model vector 0 is presented as an additional input in each time step. For CNN base models, we use ResNet meta-model where is an additional input for each ResNet block. In all our experiments, we hold out half the available training data for use as unlabeled data for training the meta- meta-modes model; the base By default, we set the
models were trained on the remaining training data (or fraction thereof) [ output loss hyperparameter X to 1. We defer further details on model architectures and training to the Appendix.
4.1 VISUALIZING MODEL SIMILARITY IN EMBEDDING SPACE
N The base model embeddings 1 1 can be used for cluster analysis to evaluate the similarity structure of a collection of models. We illustrate this use case of DYNAMO via series of example applications on NLP and vision tasks. F shows model embeddings learned by DYNAMO on collections of RNNs. In these plots, we observe clear separation of these networks according the size of the available training data, the RNN model architecture used, and the specific NLP task used for training. By computing the eigenvalues of the covariance matrix corresponding to the N model embeddings, we obtain measure of the intrinsic dimensionality of the corresponding collections of
For example, our IMDB sentiment base models trained on 100% of the availahle training data were trained on 12,500 examples, with the remaining 12,500 examples used unlabeled data for training the meta-model.
5

---

Under review as a conference paper at ICLR 2022
Training
Training trajectories
Model architecture
Training
pr nanippaal component
50%
100%
0.3
9c%
IC0%
GRU mNN
1908 Achewo
25%
29%
9.1
0.0
-0.4
-0.2
2-2
-0.4
0.0
0.2
-0.2
0.0
-0.
0.0
0.5
principal companent
principal component
principal component
principel component
variance
0.5
0.0
componintt
component index
32
component index
16
component
15
Figure 3: PCA plots of DYNAMO model embeddings collections of RNNs. From left right:
(1) The model embedding space for GRUs trained on IMDB sentiment classification dataset (Maas al. 2011) with varying training set sizes (100%. 50% and 25% of the training data): (2) training trajectories of sentiment classification GRUs over 20 epochs, with each point corresponding to an epoch of training (low-opacity points indicate networks early in training);
RNN architectures trained for IMDB sentiment classification (GRUs and vanilla RNNs); (4) GRUs trained on two NLP tasks: IMDB sentiment classification and AG News classification (Zhang et al. (015). The second row shows the spectrum for each set of embeddings. with dotted line indicating the number of components needed to explain 95% of variance.
Data augmentation A=
Training set size 1)
Data augmentation (
Training size (A
0.6
principal component 2
100%
sot,
25%
0.0
wwat arong
6.0
100%
weak strong
50%
-0.3
-0.3
-0.4
22%
-0.2
0.0
0.2
0.4
0.50
0.00
0.2
-0.50
0.00
principal component
principal companent
principal component
principal component
variance
50
0.1
0.00223
0.00
0.00000
comgonent index
component index
component index
component incex
Figure 4: PCA plots of DYNAMO model embeddings on collections of ResNet-34s trained on CIFAR-100. Left two panels: Model embeddings cluster according the size of the training dataset and the data augmentation policy used for training. Right two panels: When trained only by comparing hidden representations (i.e., with output loss weight identify a clustering effect when varying the training dataset size. the case of differing data aug- mentation policies, there weak clustering effect that suggests consistent difference in feature representation.
0), DYNAMO does not

---

Under review as a conference paper at ICLR 2022
IMDB sentiment 50% 25% train
IMDB sentiment classification News classification
10%
25%
1
NGB NGNeNS
INGB
pricibm component
principal component
principal componinnt 1
principal componentt
0.80
1.00
1.08
0.96
1.00
1.04
0.90
3,000
1015
(IMD6)
relative
News)
Figure 6: Meta-model test accuracies over model embedding space. We plot the relative test accuracies (normalized by the maximal test accuracy the base models) realized by meta-models for GRUs trained on IMDB sentiment classification with varying training set size (left), and for GRUs trained on IMDB and on AG News classification (right). In these examples, the embedding vectors that maximize test accuracy (marked with an X) do not correspond any single base model, suggesting that meta-models are capable of generalizing beyond the base models used for training.
models. In Figure 3 we find that to 6 components are sufficient to explain 95% of the variance in the model embeddings.
By tuning the output loss hyperparameter in adjust the degree of empha- sis placed on reproducing the hidden dynamics of the base networks versus their outputs. We
equation 4
Training set size
Model architecture
MDS coordinate
demonstrate this effect in Figure 4 with À = 1, we observe clear separation of ResNet-34 mod- els trained on CIFAR-100 with different data augmentation policies ("weak", with shifts and horizontal flips vs. "strong", with RandAug- ment (Cubuk et al. (0200), and with different training set sizes. In contrast, with À O, we observe a weak clustering effect with differing data augmentation, and detectable separa- tion with different training set sizes. We infer that the change in data augmentation policy sults a larger difference in the learned feature representations than the change in training set size. In Appendix we illustrate the effect of setting
MDS coordinate 1
MDS coordinate
Figure 5: 2D multidimensional scaling (MDS) embeddings the SVCCA pairwise representa- tional distances between RNN models trained on the IMDB sentiment dataset.
for RNN models.
We additionally compare the embeddings obtained using DYNAMO to those derived from SVCCA (Raghu al. (017). pairwise comparison technique that aligns the representations pro- duced by pair of networks using canonical correlation analysis (CCA). In Figure we plot the 2D embeddings obtained using multidimensional scaling (MDS) on the pairwise distance matrix com- puted using SVCCA (modified output I,2 distances instead of correlations). Unlike the principal components of the DYNAMO model embeddings plotted in Figure 3 the MDS coordinates are not semantically interpretable. Additionally, the cluster structure of the collection of GRUs trained with varying training set sizes is less apparent in this representation.
Lastly, note that DYNAMO allows for flexibility in defining the metric used compare the hidden states and outputs of the base models with those of the meta-model. In Appendix B.3] we demonstrate the benefit of using the L' distance for clustering CNN representations.
4.2 EXTRAPOLATION BEYOND BASE MODEL EMBEDDINGS
We study the model embedding space corresponding to trained meta-model by conditioning it model embedding vectors other than those assigned to the set of base models. Figure 6 visualizes the landscape of test accuracies for meta-models: (i) meta-model for 10 GRUs trained with

---

Under review as a conference paper at ICLR 2022
50% of the IMDB training data, and 10 GRUs trained with 25% of the data, and (ii) meta-model for 10 IMDB sentiment GRUs and 10 AG News classification GRUs.
We note two particularly salient properties of these plots. First, the test accuracy varies smoothly when interpolating between pairs of base model embeddings this property when interpolating the parameters of the base GRUs directly, since they were trained with different random initializations and orderings of the training examples. Second, we observe that the embedding vector that realizes the highest test accuracy lies outside the convex hull of the base model embeddings. This is perhaps surprising since typical training and inference protocols involve the use of convex combinations of various objects: for instance, averaging of predictions in model ensembles. and averaging of model parameters during training (e.g., using exponential moving averages, or with Stochastic Weight Averaging (Izmailov et al., 2018)) This extrapolatory phenomenon suggests that DYNAMO is able to derive low-dimensional manifold of models that generalizes beyond the behavior of the base models used for training the meta-model.
would in general not observe
4.3 SEMI-SUPERVISED LEARNING IN MODEL EMBEDDING SPACE
The existence model embeddings that im- prove on the accuracy of the base models sug-
1.08
gests a natural semi-supervised learning (SSL) procedure involving trained meta-model. In particular, we minimize the loss incurred by the meta-model on small set of additional labeled
50%
0.5
25%
relative test accuracy (IMDB)
principal component
1.00
examples by optimizing the value of 0. This done by backpropagating gradients through the meta-model, with the meta-model parame- ters held fixed. Figure shows the result of this procedure on an IMDB sentiment meta-model (previously depicted in Figure () with a set of additional labeled examples (disjoint from the test set) of size equal to 1% of the full training set. This procedure successfully finds a that improves on the test accuracy of the best base model by 6% (86.4% 80.3%).
0.0
Oreal
-1.0
-1.0
0.0
0.5
principal component
Figure 7: Semi-supervised learning with low- dimensional model embeddings. The red line shows the trajectory of 100 SGD iterates in model embedding space, starting from fntt = 0 and ter- minating at final White and orange points indi- cate the base model embeddings.
We observe that this SSL procedure achieves lower accuracy when we train the meta-model using fewer base models. In particular, a meta- model only on the 10 GRUs trained with 50% of the training data yields a test accuracy of 85.4%, and meta- model on only a single GRU out of these 10 yields 81.6%. This result sug- gests that a diversity of base models helps improve the accuracy achievable by the meta-model.
DYNAMICS OF MEDA-MODELS FOR RNNS
In this section we perform an analysis of the dynamical features of the dynamics generated by the meta-models trained on base models that perform the sentiment classification task. The senti- ment classification task has a well-understood dynamical structure (Sussillo & Barak 2013 Mah- eswaranathan et al., 2019a b) Aitken et al.) 2020) that we can use as basis for understanding the behavior of a corresponding meta-model. To first approximation, the sentiment analysis task can be solved by a simple integrator that accumulates sentiment corresponding to each word (positive words such 'good' 'fantastic" adding positive sentiment, and negative words such 'bad' or 'terrible" adding negative sentiment). It has been shown that simple sentiment analysis models ap- proximate this integrator by constructing line attractor in the space of hidden states. For instance, for the zero input x² Ö. it has been observed that the dynamical system generated by the map Fz. (h) = F(x", h) has a tubular region with very little movement, in the sense that F,, (h) - hl]2 very small for hidden states h in this region.
8

---

Under review as a conference paper at ICLR 2022
100% 50% Optimal
0
8.
Figure 8: Model embedding space as a space of line attractors. We plot the model embeddings of base models trained on the IMDB sentiment classification task (left), along with the centroids of each cluster. The green cluster corresponds to models trained on 100% of the data and the blue cluster corresponds models trained on fixed 50% fraction of the training data. A model having better test accuracy than any trained base model is also plotted (marked with an X). For several of the points of interest, we (right) find the structure of line attractor in the hidden state space by computing approximate fixed points of the map h F(x", h). The line attractors are shown for the point marked with an X, the two centroids, and two interpolated values the model embedding space. We also chose two values from each cluster compare the line attractor the centroid to the models in the structure, corresponding to the top left and bottom right model in each cluster.
To investigate the dynamical behavior of the meta-model space, we trained meta-model on a set of 20 base models which were themselves trained on the IMDB sentiment analysis task. Of these 20 base models, 10 were trained with 50% of the available training data and the remaining 10 were trained on 100% of the training data. The 0 points corresponding to these base models clustered the model embedding space according to the amount of training data. In Figure g we perform a fixed-point analysis of several models corresponding to points of interest in the model embedding space.
The fixed-poin analysis was according to the procedure described in (Golub & Sussillo 2018). First we selected a set of candidate hidden states h2 by running the model on a typical batch of inputs. For each hidden state h, obtained in this way, we used gradient descent on the loss [| h) h| to find the nearest approximate fixed point.
interesting finding is that the meta-model found line attractor structures that were very geomet- rically similar for models within a cluster. An interpretation of this result pertaining to topological conjugacy in dynamical systems theory is discussed Section of the Supplementary Materials. Moreover, we find that the meta-model finds continuous interpolation between line attractors that are relatively fat and thin (corresponding to models trained on 50% of the data), and models that are elongated (corresponding models trained on 100% of the data).
6 DISCUSSION
We have introduced the algorithm DYNAMO, which maps a of neural network base models to low dimensional feature space. Our results show that the model embeddings provided by DYNAMO capture relevant computational features of the base models. Moreover, the model embedding spaces produced by DYNAMO are sufficiently smooth that model averaging can be performed, and even model extrapolation to reach new models with better performance than any base model. Indeed, in our experiments where the base models were trained on the sentiment analysis task, the model embedding space describes a space of line attractors which vary smoothly the parameter 0.
We have demonstrated that DYNAMO can be applied broadly to neural networks that have dynam- ical structure; for example, we used the demarcation of layers of convolutional neural networks as proxy for dynamical time variable. This also suggests possible scientific applications of Dy- NAMO to dynamical systems arising in nature. A present limitation of DYNAMO is the need for all base models have the same input structure. For example, one cannot presently utilize DYNAMO

---

Under review as a conference paper at ICLR 2022
to compare language models trained with different encodings Scharacter-based vs. word-based, for example).
REFERENCES
Kyle Aitken, Vinay Ramasesh, Ankush Garg, Yuan Cao, David Sussillo, and Niru Ma- heswaranathan. The geometry of integration in text classification mns. arXiv preprint 10t::200001511144 2020.
Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam. Optimizing the latent space of generative networks. In International Conference Machine Learning, 2018
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. 2017.
Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jía Cui, and Bhuvana Ramab- hadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, 2017.
Matthew D Golub and David Sussillo. Fixedpointfinde:: tensorflow toolbox for identifying and characterizing fixed points in recurrent neural networks. Journal of Open Source Software, 2018.
David Ha, Andrew Dai, and Quoe V. Le. HyperNetworks. In International Conference on Learning Representations, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and T Garipov. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, 2018.
Anatole Katok and Boris Hasselblatt. Introduction to the modern theory of dynamical systems. Number 54. Cambridge University Press, 1997.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, 2019.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivari- ance and equivalence. In IEEE Conference on Computer Vision and Pattern Recognition, 2015,
Yixuan Li. Jason Yosinski, Jeff Clune, Hod Lipson, and John E Hopcroft. Convergent learning: Do different neural networks learn the same representations? In International Conference Learning Representations, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Andrew Maas, Raymond Daly, Peter Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Annual Meeting of the Association for Compu- tational Linguisties, 2011.
Niru Maheswaranathan, Alex H Williams, Matthew D Golub, Surya Ganguli, and David Sussillo. Reverse engineering recurrent networks for sentiment classification reveals line attractor dynam- ics. Advances in Neural Information Processing Systems, 2019a.
Niru Maheswaranathan, Alex H Williams, Matthew D Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. Advances in Neural Information Processing Systems, 2019b.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 2018.
10

---

Under review as a conference paper at ICLR 2022
Tsendsuren Munkhdalai and Hong Yu. Meta Networks. In International Conference Machine Learning. 2017.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. In Interna- tional Conference on Neural Information Processing Systems, 2017.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Representations, 2017.
Adriana Romero. Polytechnique montréal. y. bengio, université de montréal, adriana romero, nicolas ballas, samira ebrahimi kahou, antoine chassang, carlo gatta, and yoshua bengio. fitnets: Hints for thin deep nets. In International Conference on Learning Representations, 2015.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-Learning with Latent Embedding Optimization. In International Conference on Learning Representations, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. In International Conference on Machine Learning. 2016.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: networs. Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/3/2.6033, 2013.
David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural Computation, 2013.
David Sussillo, Mark Churchland, Matthew T Kaufman, and Krishna Shenoy. A neural network that finds a naturalistic solution for the production of muscle activity. Nature Neuroscience, 2015.
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 2016.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, 2014.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in Neural Information Processing Systems, 2015.

---

Under review as a conference paper at ICLR 2022
A META-MODELS FOR CNNS
Our setup in Section 2.2 above can be readily adapted to CNNs, or feedforward neural networks more broadly. For instance, in the case of ResNets, there is single input øy followed by sequence of blocks which operate on different numbers channels. We let the meta-model likewise be a ResNet with the same block structure. Moreover,
consider the output of the fth block in place
of the Gn (ha)) )'s in equation [ and take the hidden states between the and (++ blocks to be the h in equation 2 Since each block of ResNet is distinct, we consider family of maps Vn,, which depend on the base model n and the block layer t. Suppose B is the total number of blocks. Then can more explicitly write
B
CNNN [F,0== ourpat
=
d(bo,.,, br)
(5)
B
1
CNN hidden
[F, On: (Vo,
Σ
hell2
(6)
B
1=1
Then the total loss function accounting for all N of the ResNet base models is
CCNN [0n]
([
IN
N
Σ
EzOND,
CCNN Cn.iildem (F,0,, 0n, [Vr,te= + X n. [F, 0nl]
(7)
n=1
where we note that in the expectation value we are only sampling over zo 's in D,, since xy 's are the only form input data.
B ADDITIONAL EXPERIMENTAL DETAILS
In this section, we provide further details experiments well additional empirical results.
B.I META-MODEL ARCHITECTURES
RNN architecture. We parameterize the meta-model for RNNs as a GRU that takes the model embedding vector as additional input each time step. Specifically, the meta-model GRU takes as input the vector [ 0) at each time step, where 2; is an input token embedding and [- denotes concatenation. The model embedding vector therefore serves as inneinndeenddntt bias for the meta-model.
CNN architecture. We parameterize the meta-model for CNNs with a modified ResNet architec- ture. In each residual block, we use a linear transformation W to map the model embedding vector I the corresponding channel dimension of convolutional layer. We then add the vector W0 to the channels at each spatial location of the feature map. This design emulates the approach used in our parametrization of the RNN meta-model, with the model embedding serving as bias term in each residual block. We reuse the weight matrix W for all residual blocks with the same channel dimension.
Algorithm Meta-Model Residual Block
Input: features x, model embedding vector
MatchhoommConv(m)
ReL.U<++We
BatchNorm (ooNoooomCCon()
ReLU(x +
return
The standard ResNet architecture consists of a sequence of four layers, with each layer consisting of sequence of residual blocks. To compute the hidden state loss Chidden in DYNAMO, compute distances between the output representations each these four layers, averaging over the number of channels and spatial locations in each set of features.
12

---

Under review as a conference paper at ICLR 2022
B.2 TRAINING DETAILS
Table lists the hyperparameters used for training our RNN base models and meta-models, and Table lists the hyperparameters used for our CNN base models and meta-models.. By default, use a model embedding dimension of 16. In the case of visualizing the training trajectories of IMDB sentiment GRUs (Figure B second column), we use model embedding dimension of 32 due the relatively larger number of base models.
Hyperparameter
Base Model
Meta-Model
optimizer
Adam Loshchilov & Hutter 2018 10 3 (TO for vanilla RNN)
AdamW 10 3
learning rate
learning rate annealing
cosine with freq. 7/32
cosine with freq. 7/32
-B
(0.9, 0.999)
(0.9, 0.999)
10
10
weight decay
x 10
1 10
number of training epochs
20 (50 for vanilla RNN) 128
100
batch size
128
input token embedding dimension hidden dimension
256
256
256
512
Table 1: Hyperparameters used for RNN base models and meta-models.
Hyperparameter
Base Model
Meta-Model
optimizer
SGD with Nesterov momentum
SGD with Nesterov momentum
learning rate
0.03
0.03
learning rate annealing
cosine with freq. 7/32
cosine with freq. 7/32
momentum
0.9
0.9
weight decay
5 x 10
5 x 10
number of training batches
216
216
batch size
512
512
Table 2: Hyperparameters used for CNN base models and meta-models.
B.3 SUPPLEMENTARY EMPIRICAL RESULTS
Effeet of output loss weight. Figure 9 shows the effect of setting the output loss weight À = 0 for meta-models on RNNs. These plots illustrate that model clustering can be performed on the basis of comparing hidden state dynamics alone. We note that the change in the À hyperparameter results in qualitative changes in the resulting clustering. In particular, the model embeddings for GRUs trained on AG News classification (rightmost column of Figure 9 are much more tightly coupled relative to the GRUs trained on the IMDB dataset when X 0. This indicates that the dynamics implemented by the AG News GRUs are much more similar than those implemented by the IMDB sentiment GRUS.
Clustering with other loss functions. As noted in Sec. the loss functions used to compare hidden states and outputs can be easily changed to better match the characteristies of the base mod- els under consideration. We demonstrate the benefit of this additional flexibility by replacing the L distance in Chidden (cquation()) 6) with the L distance for comparing the intermediate representations of ResNets. This choice is motivated by the observation that use of the ReLU nonlinearity results in sparse representations, which suggests the use of the I metric. Figure 10 shows comparison between these two distance functions in the case of ResNet-34 models trained on CIFAR- 100 with "weak" data augmentation (random shifts and horizontal flips) and with "strong" data augmenta- tion (RandAuggeent, Here, we parameterized the meta- metal with a ResNet-50 architecture. The of the L' distance results in clearer separation between the two sets of models. This is flected gualitatively in the distribution of the base model embeddings in model embedding space, and quantitatively in the relative scale of the variance captured by the first principal component.
13

---

Under review as a conference paper at ICLR 2022
Training set size (A 0)
Training trajectories (A = 0)
Model archibecture (A = 0)
Training task (á = 0]
principal companent
200%
02
100%
25%
25%
68
H0.6
principal component
principal component
principal component
principal component
variance
11
comparent index
componant Index
component index
component index
Training set size (A 1)
Training trajectories (A 1)
Model architecture (A
1
Training task (A - 1)
principal componentt
50%
0.5
0.0
0.0
-0.4
-02
-0.2
22
-0.4
0.2
0.0
principal companent
principal component
principal component
principel component
varfance
21
0.5
0.0
component index
component index
component index
component index
Figure 9: PCA plots DYNAMO model embeddings on collections of RNNs with output loss
weight À O. For case of comparison, have also included the plots from Figure with À 1.
distance
distance
principal companent 2
weak strong
0.4
weak strong
-0.3
-0.3
-0.50
0.00
0.25
principal component
principal component
variance
0.01
0.00
16
16
component index
component index
Figure 10: CIFAR-100 ResNet-34 model embeddings using I distance to compare intermediate
representations (left) vs. 2 distance (right). The use of the I distance results in a clearer separa- tion between the two sets of models.
14

---

Under review as a conference paper at ICLR 2022
Model embeddings
Quality word valences
300% 50% Optinal
00
0
-0.4
0.24
-3
00
8:
9:
Figure I1: A map of the word scores (described in equation 8 function of the parameter in model embedding space. Higher scores indicate models that should have better interpretations of words. There is noisy but discernible trend that the score increases 02 decreases (and is highest near the value of the optimal model embedding).
B.4 FURTHER INVESTIGATION OF THE DYNAMICS OF SENTIMENT ANALYSIS
Recall that our trained RNN's implemented sentiment analysis via line attractor dynamics, in which inputted words kicked the hidden state in the 'positive' or 'negative" direction along the line attractor according to the valence of the word (i.e. how positive negative the word is). Figure I how valences assigned to words change
across model embedding space. Given an RNN corresponding to a value we first find a fixed point ht with neutral readout G(h*) se 0. For this fixed point, we compute G(F(x, h²)) for variety of word inputs To produce "score" for the model, we compute
Score(h")=
Σ G(F(2,h"))
Σ G(F(a,h))) - Σ IG(F(x, (8)
NEW goazine
W. Wazzaive
DEWasual
where the set of positive words W positive negative words W negative and neutral words W Wnentral
listed in Table 3 Note that here we are not analyzing context effects, for instance how the string nnot terrible' would be rendered into net-positive valence.
Positive words
exciting, fantastic, amazing, superior, outstanding, superi, magnificent, exceptional, temendous lemible, poor, unaccoptable, shoddy, atrocious, crap. gurbage
Negutive bad,
words
the, 14, a, are, were, can, Fad, was, when, who, to,
Table 3: Lists of good, negative, and neutral words selected to assess the "word valence" quality of model.
C META-MODELS AND TOPOLOGICAL CONJUGACY
In this section describe the notion of topological conjugacy, its relationship the loss function Laddeen and provide speculation as to the interpretation of the results from Section
topological conjugacy Katok Hasselblatt 1997) between dynamical systems defined by maps F X + X and E Y Y is homcomorphism 4 X - satisfying
F(x) ( E (x))
Note that the dynamical systems F and E can have distinct domains. The significance the rela- tionship in equation 9 that dynamics obtained from iterated applications of the map F and E are
15

---

Under review as a conference paper at ICLR 2022
related to each other by the formula
(x) = (0-1 E" (x))..
Thus, and E are topologically conjugate, then their iterates are also topologically conjugate and this means that the dynamics are related by a change of variables. Notice that topological conjugacy an equivalence relation; as such the transitive property tells us that if F ~ E and E D then D.
The notion of topological conjugacy is an important motivation for defining the loss function Cnidden which we recall is given by
Liddden FF,,
Σ IV (ho.t) h4 12.12.
where the hidden states ht are dynamics obtained from base model the hidden states he, are dynamics obtained from the meta-mode F with parameter f, and V is the map from the meta-model hidden states to the base model hidden states. One way of obtaining zero loss is to find topological conjugacy with map V between the meta-model F (at fixed and the base model F, meaning relationship of the form
F(z. hi) (V ()
is convenient to define the notation F2 (2)h) == F(x, h) and (o, == F0,,I, h).. Then we have
h2 =(Fz, Pa.-,
F2,) (V Fo.z, Fo. .... Fo. (V~ Vho,
with he.0 = V- so that Lhiaden [F, 0, V] 0 would hold. As depicted in Figure 2 to obtain zero loss it would actually suffice to find weaker relationship of the form
F(z, Vh) (V (0,,,x,
The difference here is that V need not be invertible.
Using the language of topological conjugacy, we can describe a speculative but plausible interpreta- tion of the results of Section 5 In that section observed that models from the same cluster had very similar dynamical features and performed similarly to the model average of the cluster. This suggests that for each model Fú in the same cluster, we have
(x V. (V, F),,x,h)
where 8 is the centroid of the cluster to which F, belongs. Note that here we replaced 6, with ø, thus assuming both that Lnidden is small and that 6, is sufficiently close to Second, making the hypothesis that there exists inverse V,-1 to the map Vn, the map V,, may provide topological conjugacy between the base model F, and the meta-model Fg evaluated at B. Assuming further that our assumptions hold for all models in the cluster, using the transitivity of topological conjugacy we would conclude that base models belonging to the same cluster are topologically conjugate to one another. This would justify the intuition suggested by Figure & that DYNAMO clusters models according to commonalities topological structures dynamics.
16


Reviews:
[{"review_id": "YxCEnnVWgTk", "reviewer": "Reviewer_k7ES", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of the paper are that the proposed approach is super simple, it depends on a mild set of constraints, and it is quite general. Indeed, the authors show how their idea can be implemented for RNN and CNN models.\n\nThe main weaknesses of the paper are the lack of comparison and the lukewarm results. In particular, only Fig. 5 shows a comparison of the proposed approach. However, MDS is computed instead of PCA, so it is not possible to truly evaluate the comparison with respect to Fig. 2.  Moreover, the discussion on related work at the first paragraph of Sec. 3 is somewhat unsatisfying. The main limitation that was mentioned is the O(N^2) computational cost. However, no analysis was provided of the computational resources requires for the proposed method.\n\nAlso, the results in Figs. 2 & 3 could potentially have been produced using other approaches, among those is even simple PCA on the network weights/hidden-states/other information (possibly padded to match dimensions). For instance, the third panel in Fig. 3 is somewhat expected, given that the underlying structure of RNN/GRU is significantly from one another. \n\nAnother issue is that most results are presented on the sentiment analysis (SA) problem. While SA is a great task for analysis and illustration, it is somewhat a toy example. Thus, it is not clear whether the results presented in this work extend to more challenging tasks solved using RNN models.\n\nFig. 4 is not entirely clear to me, and I am not sure what can be inferred from it. My impression is that the hidden states of the model are not representative enough, but I might be wrong. Please clarify.\n\nFig. 6 is interesting and I would like to understand it better. In particular, do you observe a similar phenomenon when using 100% of the data (instead of 25%/50%). Can you improve the state-of-the-art for this model+task using this approach? What is the difference between the middle and right panels? If you sample in 2D, how are the other dominant dimensions are being set? What are the baseline accuracies?\n\nI am not sure what the results in Fig. 8 show. Please elaborate.\n\nminor comments:\n\n\t- I could not find a formal definition of the term \"manifold\". However, this term is used quite heavily in the manuscript, and thus I would expect at least an intuitive explanation for what it means. For instance, the first paragraph in the introduction claims neural networks form a manifold due to their continuously tunable weights. If this is the case, what about the initial network? Is it also part of this manifold of networks? Asked differently, what is not part of this manifold?\n\n\t- It is claimed that two points in the meta-model embedding space are nearby if they correspond to neural networks with similar dynamics. This is only intuitively implied. Is there a formal guarantee? If not, I suggest to tone-down this claim, as most of the results are focused on a narrow set of architectures.\n\n\t- A similar concern is regarding the convex combination of theta vectors. In particular, even in binary classification problems, the decision boundary is somewhat complex, let alone in a manifold of tasks. It would be interesting to better understand the boundary of the learned manifold.\n\n\t- The tasks explored in this work are classification problems. What about regression tasks?\n\n\t- The dynamical system studied in App. C is not a proper conjugate system per your definition as it has a different domain.", "summary_of_the_paper": "This paper introduces a new approach to studying the population of trained neural networks. Specifically, the author's design and train a meta-model which when given a vector, the meta-model emulates the model associated with the given vector. The authors present a few applications of their framework, including clustering and semi-supervised learning.\n", "summary_of_the_review": "Overall, this paper presents an intriguing idea of exploring the space of learnt networks. The simplicity and generality of the approach are encouraging. The paper could be significantly improved with an extensive empirical evaluation and comparison, and potentially with formal guarantees.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W2F8_iEP8st", "reviewer": "Reviewer_p41a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach here for ensembling together multiple networks is interesting. The argument for embedding together multiple networks via their theta vectors is weakened a bit by decoders V_i being independently considered in Algorithm 1, rather than have a single decoder V that simply takes theta_i as an input in order to figure out how to decode latent meta-states to individual network hidden states (perhaps with padding, truncation, or some simple projection layer to account for different number of neurons). However, the combined meta-model seems to indeed have value in the ensemble setting, especially given the results in Figures 6-7 and section 4.3, therefore justifying empirically the design used here - but it would be good to see some better reasoning and ablation study to see if and why separate decoders are really needed. To me, this aspect is the most appealing in this work, but it is somewhat understudied, especially since no other ensemble approach is considered here to compare whether training the theta has advantages over more traditional boosting and bagging approaches (applied to the same base models). \n\nFor the embedding of networks - the results are rather anecdotal, and while some clear differences are demonstrated, it is hard to see what new insights are provided by them. More fundamentally, the consideration of \"manifold of models\" is unclear here, even when considered in an intuitive way. This is already seen in the introduction where the authors argue that network weights naturally give rise to some manifold structure. I suppose technically this is true in the same sense that any Euclidean space can be considered as a (rather trivial) manifold, but the same can be said for any parametrized space of functions. In fact, a more accurate description of this point is that neural networks provide a subset of functions that can be parameterized in finite (but very high) dimensional space, rather than be considered in their possibly-infinite dimensional ambient function space, thus enabling their optimization. Similarly, the encoding here maps multiple parametrizations (with potentially varying dimensionalities) into a single parameter space. However, it is unclear in which (non-trivial) sense would this space be considered a manifold. Are similar thetas really providing similar models, functions, or architectures? The results show some very coarse organizations where big differences are more or less exhibited (to somewhat larger degree than SVCCA), but it's unclear if finer differences are really mapped to small changes in theta, or if directions in the theta (PCA) space correspond to some clear or interpretable difference in the base models. \n\nFor example, in Figures 3 and 4 (as well as some supplemental figures), several panels are showing the first principal component (capturing much more variance than the rest) aligns with clear differences between configurations (training size, data augmentation, or architecture), but there is also some clear spread in the second component - is this related to some meaningful difference between networks within the same architecture or training regime? Is the varying dimensionality in PCA (based on the 95% explained variance threshold) meaningful? Are individual thetas (prior to PCA projection), or the relations between them via PCA loading, meaningful? The analysis currently provided for these aspects is rather superficial, and it is unclear whether the difference from SVCCA, as demonstrated in Figure 5, is really beneficial or just showing a mere difference between equivalent embeddings. Indeed, while the cluster separation in Figure 5 is perhaps not as clear, or linearly separable, as in figures 3-4, it is quite clear that MDS over SVCCA also organizes the networks according to training size (in concentric rings) or network architecture, and these could probably be nonlinearly separated with an appropriate kernel (and perhaps more than two dimensions). ", "summary_of_the_paper": "This paper proposes an approach to combine together multiple (pretrained) neural networks into a single model that can simulate each of the considered neural networks, both in terms of output and hidden states. The setup here relies on all the considered networks to operate on the same task, and in particular the same input data type (but perhaps with different sampled training sets). The combined meta-model then considers as inputs not only the input data, but also an encoding (which is learned during its training) of each model. This allows model-dependent latent features to be computed by the meta-model, which can then be decoded to hidden states of each network via specialized decoders that can be thought of as auxiliary computational pathways that operate in parallel to the one producing the main output of the meta-model. The model encoding are then used to visualize and explore the relations between different networks (or instantiations), and furthermore, to further tune the combined model towards an interpolation or extrapolation that goes beyond (and seems to outperform) each of the individual networks used to train the meta-model. ", "summary_of_the_review": "The idea in this paper is interesting, and seems promising for an ensemble-learning method interpolating between (or extrapolating from) a collection of neural network base model to a stronger combined meta-model. However, it is not presented or evaluated as such. Instead, this work tries to pose it as a way to learn a \"model manifold,\" and in this context the results are rather superficial and lack clear insights or benefits. Therefore, I would consider it as a borderline paper. It has some potential to be above the acceptance threshold, if it is refocused appropriately and compared to other ensemble methods, but this type of revision might not be sufficiently simple to complete within the tight review schedule for this conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GcXlwpDWCQ-", "reviewer": "Reviewer_9dWJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed algorithm is novel and interesting. The visualization it provided is meaningful and can be of value to the community. However, I have several concerns about the work at its current state. My main concerns are the lack of proper comparison to the literature, which is explained in detail below.\n\nPros:\n- The proposed model is novel, interesting, and easy to understand.\n- The resulting embedding can help better visualize the underlying models.\nCons:\n- Requires a large number of trained models to achieve meaningful embedding. Requires additional data.\n- I would like a clearer comparison to other similarity measures from the literature. \n\nThe paper has several claims on the usefulness of the proposed model. I will go over each of them separately:\n\nVisualization:\nPros: The visualization seems informative and clear, and can be of value to the community.\nCons: Lack clearer comparison to other similarity methods between models. Other methods are only mentioned, but it is unclear which advantages this visualization method has over other ones.\n\nInterpolation:\nThe ability to interpolate models is new and interesting, and in my opinion is a contribution of this paper, although a bit minor by itself.\n\nExtrapolation:\nThe authors claim that the meta-model can reach a better performance than any of its underlying models by extrapolation. While this indeed seems like the case, the value of such a finding is unclear to me. To the best of my understanding, the base models trained in Fig 6 saw only a partial amount of the data, while the meta-model saw additional data points during its training. Therefore, it is unsurprising that it could reach enhanced performance.\n\nSemi-supervised learning:\nIt is hard for me to judge the value of such results, as in my opinion, the comparison here is problematic. As the meta-model sees more data, its accuracy should be compared to other semi-supervised models with similar data, and not the base models.\n\nminor point:\nThe paper lacks organization: some figures/tables are never referenced in the text (Figure 1 and algorithm 1). The figures sometimes are far from their mention in the text. This made the paper much harder to follow than it should have.\n", "summary_of_the_paper": "The authors propose an algorithm that takes several neural networks and outputs an embedding for such networks and a meta-model. This meta-model can take any embedding of a network, and emulate its hidden states and outputs. The authors suggest that these embeddings can measure models similarity, and show how to interpolate and extrapolate between and from the models in the training set.", "summary_of_the_review": "I find this work novel and interesting. However, I have concerns about the 2 main contributions of the paper, which result in a minor contribution of the paper to the community overall. Therefore, I tend to reject the paper in its current state. However, if I missed something which made these contributions more relevant, please let me know and I'll be more than happy increase my score.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1Y8jPvKTvz", "reviewer": "Reviewer_dswE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Quantifying the similarity between two different neural networks trained for the same task is in my opinion an interesting topic which, as correctly mentioned by the authors, has already attracted significant attention. Many methods have been developed to describe the similarity of hidden representations, some of which are cited in the manuscript. In my point of view an open problem in the field is the comparison of the hidden states in  different architectures trained on the same task and dataset. \n\n My concern is that it is not clear if the approach offers a solution to this problem. The examples presented in fig 3 compare networks trained using the same architecture and different training sets, or different training schedules. In these examples, the metamodel has the same architecture of the networks that it describes. Panel 3 presents an analysis of a  GRU and a vanilla RNN, but in that case the metamodel is just another GRU. This gives the impression that the metamodel should just be an extension of all the models which it should describe, and as expressive as the most expressive network which is considered. The theta parameter practically controls which part of the \"ruling network\" should be switched off in order to reproduce the behavior of each target network. Is this impression correct?   \n\nThe key question is if it  is possible to use the approach to make a step forward in the understanding the relationship between hidden representations coded in qualitatively different manner comparing, say  a CNN and a transformer, both trained for image recognition. Or, at least, a network with a VGG and a resnet architecture. ", "summary_of_the_paper": "The manuscript proposes to analyze the similarity of different neural networks trained to solve the same task by developing a meta-model capable of reproducing/emulating both the output and the hidden representation of each individual network.  This meta-model is parametrized by a set of parameters which is common to all the networks, and by other parameters which are network-specific (theta). In this way, each network will correspond to a different theta, and one can analyze the global features of the set of networks by unsupervised manifold learning in the  theta-space.\n", "summary_of_the_review": "I think that the approach is smart and potentially interesting, but the empirical evidence used to demonstrate its usefulness is not fully convincing. In all cases defining which  hidden states should be compared  is  easy (or even trivial, if the architecture is really the same), but in these conditions other approaches for comparing the representations can be used as well. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an algorithm for making a meta-model from an ensemble of models by learning model embedding.

All reviewers appreciated the originality and potential usefulness of the paper. However, they also all think that the work is not completely ready for publication. Both the presentation and quality of the results can be improved. 

There are a lot of good feedback in the discussion that can be used to make an important updated version for the next conference.
Decision: Reject