Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I first wanted to thank the authors for their proposed approach in this paper. The paper discusses an interesting idea for quantizing the similarity of neural networks, based on weight similarity. However, overall this assumption is based on the fact that similar layers within a particular architecture will learn similar semantics across different runs (and surprisingly authors add tasks to this as well, which I don't understand why this is the case). 

Unfortunately, the paper is hard to read. It is not easy to understand the research questions, and experimental setup. Partly due to overloaded terms, such as “solution” being used for describing multiple different concepts across the paper (solution class, local solution classification, local solution retrieval, none of them particularly well defined in the paper). I highly recommend the authors to describe their findings in a more concrete manner. I did not see any attempts at giving *insights*, mostly numerical comparison. 

My concerns with the methodology of the paper are as follows:

1. What are the findings of the paper? Permutation of the neural networks is certainly an area worth studying. However, in this paper, authors make the assumption that weights across the same layer (say layer #2) are somehow always going to learn similar values (except in permutation) across different runs of the model? Major clarification in this area is required. 

2. I am not quite sure what the term solution class means, or what the authors want the reviewer to believe it means. Please elaborate. Also solution label? This terminology seems a bit obsolete and cumbersome, unless properly defined at the beginning of the paper. 

3. Can you elaborate more on the goals of the experiments? Right now the outcome of the experiments are a bit vague based on lack of hypotheses. 

4. Can you elaborate what the goal of local solution classification is? It is not clear if this is simply the classification accuracy of a trained model, or what is vaguely described in section 3.3

I will make a re-evaluation of the paper after the above questions are answered. Overall, I suggest a rewrite of the paper to make claims, experimental hypotheses and design more clear. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper presents a method called ‘function feature learning’ which do not learn the data distribution but the parameters distribution of several neural networks types. The main idea is to generate many weights from different NNs trained with different random initializations for different subtasks and use them as training data for ‘function feature learning’. The experiments were done on three different datasets.

Overall, the idea is quite interesting and new. However, I’m not 100% sure about the usefulness of the method. The authors claimed to provide more insights of neural networks with their method which I did not see when reading the paper. Furthermore, the authors used a neural networks - a black box model - to provide insights for other neural networks, also black box models. It sounds odd, doesn’t it? Moreover, one assumption from this paper that networks trained with different initializations for the same subtasks produce the same local solution is wrong. Therefore, I’m not 100% sure whether the results produced from all the experiments are trustable. 

In sum, I rate this paper as a borderline paper and lean towards rejection due to several aforementioned uncertain points. 
review_title: Official Blind Review #1

Meta-review: This paper tackles an important problem: understanding if different NN solutions are similar or different. In the current form, however, the main motivation for the approach, and what the empirical results tell us, remains unclear. I read the paper after the updates and after reading reviews and author responses, and still had difficulty understanding the goals and outcomes of the experiments (such as what exactly is being reported as test accuracy and what is meant by: "High test accuracy means that assumptions are reasonable."). We highly recommend that the authors revisit the description of the motivation and approach based on comments from reviewers; further explain what is reported as test accuracy in the experiments; and more clearly highlight the insights obtain from the experiments. 
Decision: Reject