Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach to combine together multiple (pretrained) neural networks into a single model that can simulate each of the considered neural networks, both in terms of output and hidden states. The setup here relies on all the considered networks to operate on the same task, and in particular the same input data type (but perhaps with different sampled training sets). The combined meta-model then considers as inputs not only the input data, but also an encoding (which is learned during its training) of each model. This allows model-dependent latent features to be computed by the meta-model, which can then be decoded to hidden states of each network via specialized decoders that can be thought of as auxiliary computational pathways that operate in parallel to the one producing the main output of the meta-model. The model encoding are then used to visualize and explore the relations between different networks (or instantiations), and furthermore, to further tune the combined model towards an interpolation or extrapolation that goes beyond (and seems to outperform) each of the individual networks used to train the meta-model. 
main_review: The approach here for ensembling together multiple networks is interesting. The argument for embedding together multiple networks via their theta vectors is weakened a bit by decoders V_i being independently considered in Algorithm 1, rather than have a single decoder V that simply takes theta_i as an input in order to figure out how to decode latent meta-states to individual network hidden states (perhaps with padding, truncation, or some simple projection layer to account for different number of neurons). However, the combined meta-model seems to indeed have value in the ensemble setting, especially given the results in Figures 6-7 and section 4.3, therefore justifying empirically the design used here - but it would be good to see some better reasoning and ablation study to see if and why separate decoders are really needed. To me, this aspect is the most appealing in this work, but it is somewhat understudied, especially since no other ensemble approach is considered here to compare whether training the theta has advantages over more traditional boosting and bagging approaches (applied to the same base models). 

For the embedding of networks - the results are rather anecdotal, and while some clear differences are demonstrated, it is hard to see what new insights are provided by them. More fundamentally, the consideration of "manifold of models" is unclear here, even when considered in an intuitive way. This is already seen in the introduction where the authors argue that network weights naturally give rise to some manifold structure. I suppose technically this is true in the same sense that any Euclidean space can be considered as a (rather trivial) manifold, but the same can be said for any parametrized space of functions. In fact, a more accurate description of this point is that neural networks provide a subset of functions that can be parameterized in finite (but very high) dimensional space, rather than be considered in their possibly-infinite dimensional ambient function space, thus enabling their optimization. Similarly, the encoding here maps multiple parametrizations (with potentially varying dimensionalities) into a single parameter space. However, it is unclear in which (non-trivial) sense would this space be considered a manifold. Are similar thetas really providing similar models, functions, or architectures? The results show some very coarse organizations where big differences are more or less exhibited (to somewhat larger degree than SVCCA), but it's unclear if finer differences are really mapped to small changes in theta, or if directions in the theta (PCA) space correspond to some clear or interpretable difference in the base models. 

For example, in Figures 3 and 4 (as well as some supplemental figures), several panels are showing the first principal component (capturing much more variance than the rest) aligns with clear differences between configurations (training size, data augmentation, or architecture), but there is also some clear spread in the second component - is this related to some meaningful difference between networks within the same architecture or training regime? Is the varying dimensionality in PCA (based on the 95% explained variance threshold) meaningful? Are individual thetas (prior to PCA projection), or the relations between them via PCA loading, meaningful? The analysis currently provided for these aspects is rather superficial, and it is unclear whether the difference from SVCCA, as demonstrated in Figure 5, is really beneficial or just showing a mere difference between equivalent embeddings. Indeed, while the cluster separation in Figure 5 is perhaps not as clear, or linearly separable, as in figures 3-4, it is quite clear that MDS over SVCCA also organizes the networks according to training size (in concentric rings) or network architecture, and these could probably be nonlinearly separated with an appropriate kernel (and perhaps more than two dimensions). 
summary_of_the_review: The idea in this paper is interesting, and seems promising for an ensemble-learning method interpolating between (or extrapolating from) a collection of neural network base model to a stronger combined meta-model. However, it is not presented or evaluated as such. Instead, this work tries to pose it as a way to learn a "model manifold," and in this context the results are rather superficial and lack clear insights or benefits. Therefore, I would consider it as a borderline paper. It has some potential to be above the acceptance threshold, if it is refocused appropriately and compared to other ensemble methods, but this type of revision might not be sufficiently simple to complete within the tight review schedule for this conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an algorithm that takes several neural networks and outputs an embedding for such networks and a meta-model. This meta-model can take any embedding of a network, and emulate its hidden states and outputs. The authors suggest that these embeddings can measure models similarity, and show how to interpolate and extrapolate between and from the models in the training set.
main_review: The proposed algorithm is novel and interesting. The visualization it provided is meaningful and can be of value to the community. However, I have several concerns about the work at its current state. My main concerns are the lack of proper comparison to the literature, which is explained in detail below.

Pros:
- The proposed model is novel, interesting, and easy to understand.
- The resulting embedding can help better visualize the underlying models.
Cons:
- Requires a large number of trained models to achieve meaningful embedding. Requires additional data.
- I would like a clearer comparison to other similarity measures from the literature. 

The paper has several claims on the usefulness of the proposed model. I will go over each of them separately:

Visualization:
Pros: The visualization seems informative and clear, and can be of value to the community.
Cons: Lack clearer comparison to other similarity methods between models. Other methods are only mentioned, but it is unclear which advantages this visualization method has over other ones.

Interpolation:
The ability to interpolate models is new and interesting, and in my opinion is a contribution of this paper, although a bit minor by itself.

Extrapolation:
The authors claim that the meta-model can reach a better performance than any of its underlying models by extrapolation. While this indeed seems like the case, the value of such a finding is unclear to me. To the best of my understanding, the base models trained in Fig 6 saw only a partial amount of the data, while the meta-model saw additional data points during its training. Therefore, it is unsurprising that it could reach enhanced performance.

Semi-supervised learning:
It is hard for me to judge the value of such results, as in my opinion, the comparison here is problematic. As the meta-model sees more data, its accuracy should be compared to other semi-supervised models with similar data, and not the base models.

minor point:
The paper lacks organization: some figures/tables are never referenced in the text (Figure 1 and algorithm 1). The figures sometimes are far from their mention in the text. This made the paper much harder to follow than it should have.

summary_of_the_review: I find this work novel and interesting. However, I have concerns about the 2 main contributions of the paper, which result in a minor contribution of the paper to the community overall. Therefore, I tend to reject the paper in its current state. However, if I missed something which made these contributions more relevant, please let me know and I'll be more than happy increase my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new approach to studying the population of trained neural networks. Specifically, the author's design and train a meta-model which when given a vector, the meta-model emulates the model associated with the given vector. The authors present a few applications of their framework, including clustering and semi-supervised learning.

main_review: The main strengths of the paper are that the proposed approach is super simple, it depends on a mild set of constraints, and it is quite general. Indeed, the authors show how their idea can be implemented for RNN and CNN models.

The main weaknesses of the paper are the lack of comparison and the lukewarm results. In particular, only Fig. 5 shows a comparison of the proposed approach. However, MDS is computed instead of PCA, so it is not possible to truly evaluate the comparison with respect to Fig. 2.  Moreover, the discussion on related work at the first paragraph of Sec. 3 is somewhat unsatisfying. The main limitation that was mentioned is the O(N^2) computational cost. However, no analysis was provided of the computational resources requires for the proposed method.

Also, the results in Figs. 2 & 3 could potentially have been produced using other approaches, among those is even simple PCA on the network weights/hidden-states/other information (possibly padded to match dimensions). For instance, the third panel in Fig. 3 is somewhat expected, given that the underlying structure of RNN/GRU is significantly from one another. 

Another issue is that most results are presented on the sentiment analysis (SA) problem. While SA is a great task for analysis and illustration, it is somewhat a toy example. Thus, it is not clear whether the results presented in this work extend to more challenging tasks solved using RNN models.

Fig. 4 is not entirely clear to me, and I am not sure what can be inferred from it. My impression is that the hidden states of the model are not representative enough, but I might be wrong. Please clarify.

Fig. 6 is interesting and I would like to understand it better. In particular, do you observe a similar phenomenon when using 100% of the data (instead of 25%/50%). Can you improve the state-of-the-art for this model+task using this approach? What is the difference between the middle and right panels? If you sample in 2D, how are the other dominant dimensions are being set? What are the baseline accuracies?

I am not sure what the results in Fig. 8 show. Please elaborate.

minor comments:

	- I could not find a formal definition of the term "manifold". However, this term is used quite heavily in the manuscript, and thus I would expect at least an intuitive explanation for what it means. For instance, the first paragraph in the introduction claims neural networks form a manifold due to their continuously tunable weights. If this is the case, what about the initial network? Is it also part of this manifold of networks? Asked differently, what is not part of this manifold?

	- It is claimed that two points in the meta-model embedding space are nearby if they correspond to neural networks with similar dynamics. This is only intuitively implied. Is there a formal guarantee? If not, I suggest to tone-down this claim, as most of the results are focused on a narrow set of architectures.

	- A similar concern is regarding the convex combination of theta vectors. In particular, even in binary classification problems, the decision boundary is somewhat complex, let alone in a manifold of tasks. It would be interesting to better understand the boundary of the learned manifold.

	- The tasks explored in this work are classification problems. What about regression tasks?

	- The dynamical system studied in App. C is not a proper conjugate system per your definition as it has a different domain.
summary_of_the_review: Overall, this paper presents an intriguing idea of exploring the space of learnt networks. The simplicity and generality of the approach are encouraging. The paper could be significantly improved with an extensive empirical evaluation and comparison, and potentially with formal guarantees.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes to analyze the similarity of different neural networks trained to solve the same task by developing a meta-model capable of reproducing/emulating both the output and the hidden representation of each individual network.  This meta-model is parametrized by a set of parameters which is common to all the networks, and by other parameters which are network-specific (theta). In this way, each network will correspond to a different theta, and one can analyze the global features of the set of networks by unsupervised manifold learning in the  theta-space.

main_review: Quantifying the similarity between two different neural networks trained for the same task is in my opinion an interesting topic which, as correctly mentioned by the authors, has already attracted significant attention. Many methods have been developed to describe the similarity of hidden representations, some of which are cited in the manuscript. In my point of view an open problem in the field is the comparison of the hidden states in  different architectures trained on the same task and dataset. 

 My concern is that it is not clear if the approach offers a solution to this problem. The examples presented in fig 3 compare networks trained using the same architecture and different training sets, or different training schedules. In these examples, the metamodel has the same architecture of the networks that it describes. Panel 3 presents an analysis of a  GRU and a vanilla RNN, but in that case the metamodel is just another GRU. This gives the impression that the metamodel should just be an extension of all the models which it should describe, and as expressive as the most expressive network which is considered. The theta parameter practically controls which part of the "ruling network" should be switched off in order to reproduce the behavior of each target network. Is this impression correct?   

The key question is if it  is possible to use the approach to make a step forward in the understanding the relationship between hidden representations coded in qualitatively different manner comparing, say  a CNN and a transformer, both trained for image recognition. Or, at least, a network with a VGG and a resnet architecture. 
summary_of_the_review: I think that the approach is smart and potentially interesting, but the empirical evidence used to demonstrate its usefulness is not fully convincing. In all cases defining which  hidden states should be compared  is  easy (or even trivial, if the architecture is really the same), but in these conditions other approaches for comparing the representations can be used as well.
review_title: null

Meta-review: This paper introduces an algorithm for making a meta-model from an ensemble of models by learning model embedding.

All reviewers appreciated the originality and potential usefulness of the paper. However, they also all think that the work is not completely ready for publication. Both the presentation and quality of the results can be improved. 

There are a lot of good feedback in the discussion that can be used to make an important updated version for the next conference.
Decision: Reject