Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Accurately predicting the performance at early stage is important for efficient model selection without incurring too much computation. The paper proposes a neural capacitance metric as a predictive measure to capture the performance of a model on the downstream task using only a handful of early training results. The metric is derived from a line graph mapped from a neural network by modeling the dynamical system of the network.
main_review: The proposed method is novel. Experiments show that the proposed neural capacitance more accurately predicts the accuracy of the model. Ablation study have been performed to study the effect of parameters such as starting epochs, size of training set, etc.

Overall, I think the paper is potentially a good paper, though I'm not an expert in this topic and does not fully understand the mathematical derivation in Section 4.2 and 4.3. My major concern is

- I'm not sure if I fully understand NCP units. Why the weights are randomly initialized and freezed during finetuning? And in Figure 1(b) does the output layer of right stack do the same thing as output layer of left stack for classfication tasks? I do not find the relationship between the new layers and what is described in section 4.2 and 4.3. Why two dense layers are chosen to be the architecture of NCP units? I think this part is missing from the paper and should be elaborated to build the connection between the previous section.

summary_of_the_review: Overall, I think the paper proposes an interesting idea to model the training NN as a dynamical system and is potentially a good paper, but the some part of the method needs more elaboration.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper formulates neural network training as a dynamical system and uses existing theory to formulate a metric (beta_eff) to predict how a pre-trained model will perform on a downstream task without requiring to train the model to convergence.
main_review: **Strengths:**
1. Application of dynamic systems theory to NNs.

While the theory described in the paper is not novel, the application of it to the task of predicting model performance is novel as far as I can tell.

**Weaknesses:**

1. Clarity

The paper uses multiple prior ideas from dynamical systems and machine learning literature but many of these choices are not well motivated or explained. This makes it difficult for a broad ML audience to understand the paper and its significance.
- The paper draws heavily from theory described in Gao et al 2016 which proposes a method to convert the reslience function on a multi-dimensional system to a single dimensional function x_eff where the critical points B_eff are a one-dimensional value. Gao et al 2016 describes this approach for a non-neural network system and it is unclear why the assumptions introduced in this paper apply to neural networks.

The simplifying assumption used in the paper is that the dynamics of a node in a complex multi-dimensional system can be characterized by the average nearest-neighbor activity. However, in a neural network, perturbing a single weight doesn't only affect the gradients of the immediate neighbors of the node. Thus, it is unclear why this approximation is appropriate for neural networks. This is briefly touched upon in section 4.2 where it is stated that "the others’ contribution as a whole is implicitly encoded in the activation gradient" however the connection to the original theory described ini Gao et al 2016 and why this is an appropriate simplification needs more explanation.  

- Next, the motivation for the NCP is not discussed in the paper. By initializing it randomly and freezing it, a random amount of fixed noise is introduced into the process and it is unclear what the purpose of this is. Additionally, is this randomly initialized each time for all the experiments? And if so, are the same experiments repeated with different initializations to control for randomness?

- What is the motivation of using the second order gradient of C to quantify the interaction strength?

2. Correctness
- In Appendix B, section D "PROOF OF THEOREM 1", it is claimed that the gradient vanishes when one of the terms of the product is 0. Then, it is stated that when either goes to 0, the numerator in beta_eff goes to 0. However, it appears that if the gradient (left) term goes to 0, the denominator also goes to 0 since the term is also present in the denominator product. This will result in beta_eff being undefined. Please clarify whether this is accurate.

3. Details
- In the last sentence of section 4.2, the self-dynamics part, f(w_i) is defined as F(w_i*), however, I couldn't find where this was defined. What does this correspond to in the context of neural networks? Additionally, why is g(w_i, w_j) = w_j - w_j*?

4. Figures
- The figures are small and difficult to read. A better presentation of the results such as a table could be added to the main paper while the figures are either added to the appendix or enlarged.
summary_of_the_review: The paper introduced a novel formulation of neural networks based on prior dynamical systems theory for an important machine learning problem. However, there are some parts of the paper that require further explanation and in its current state, I think it is fairly difficult for a broad ML audience to understand the paper and it's significance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework to select the neural networks for downstream tasks. To identify the better generalization model, the authors propose a new metric (Neural Capacitance, NCP) to predict precise learning curves. And the authors provide the theoretical explanations for NCP.

Then the authors have verified the advantages of the proposed method when being applied to different datasets (CIFAR10, CIFAR100, SVHN, Fashion MNIST, Birds) with 17 CNN models.
main_review: I am not an expert on this research topic. maybe I cannot give a precise review.

Strengths:

The whole paper is written in high-quality and a clear manner.

To validate the outstanding performance of the proposed method, the authors conduct experiments on many popular CNN models.

Weakness:

I am not 100% sure about the performance, because the author reports the accuracy with figures rather than tables.

The authors conduct the experiments on well-known models, what about applying this framework to select the subnetwork on many NAS benchmarks e.g., [1].


[1]. Hw-nas-bench: Hardware-aware neural architecture search benchmark 

summary_of_the_review: NA
review_title: null

Meta-review: The paper proposes a method for inferring which of a set of pretrained neural networks, once fine-tuned on a transfer task, will generalize the best. This is accomplished by deriving a quantity based on a mean-field approximation of a dynamical system defined on the adjacency matrix of the weights of a neural network, known as the "neural capacitance". The model selection procedure involves attaching a fixed, randomly initialized network onto the outputs of the pretrained network and fine-tuning for a small number of iterations, and computing the metric; the fixed network is called the "neural capacitance probe" (NCP).

Reviews, though low confidence, awarded borderline scores, and a central concern was clarity and motivation, in particular the role of the NCP. acZh, the highest confidence and most verbose reviewer, echoed these concerns along with specific criticisms, for example about the heavy reliance on Gao et al (2016) without elaboration. The authors have responded in considerable depth but unfortunately the reviewer has not acknowledged these responses. On the NCP, the authors note that this is an approximation to the ideal metric that they have empirically validated.

Reading the updated draft, I find myself still concurring with reviewer acZh in large degree. The draft has improved with the noted additions, such as Appendix G devoted to an explanation of Gao et al (2016), but the presentation is still quite challenging to follow. I am left with fundamental questions about the soundness of the approximation being made, its wider applicability, and the many arbitrary decisions regarding the architecture of the NCP that appear out of nowhere. How sensitive is the procedure to these choices? Did the authors tune these architectural hyperparameters? Using what data? The table of results does not include units, and for a paper proposing a general purpose metric I'd ideally want to see a a robust rationale for hyperparameter selection of method-specific hyperparameters as well as a rigorous statistical treatment of the method's performance. Since it involves an approximation, a comparison to the "ideal" or "exact" procedure on a toy problem where the latter is feasible would strengthen the paper considerably. I do appreciate the breadth of architectures and datasets examined, but I believe the central focus of the paper should be explaining the mathematical motivation (perhaps at a higher level and deferring more detail to the appendix), why precisely it makes sense in the context of neural networks (also raised by acZh, with an answer provided that I believe partially addresses this) and justifying the concrete, approximate instantiation of the method involving the NCP and the hyperparameter selection and evaluation protocol that led you to the particular NCP employed.

At a higher level, this is a very mathematically dense paper that relies considerably on concepts outside of what might be considered typical expertise in the ICLR community, reflected in the confidence scores of the reviewers. While I feel that the issues described above already preclude acceptance at this time, I believe it may be difficult to do the proposed method justice in the short conference paper format, and would suggest to the authors to consider a journal submission instead, where a didactic presentation can be given the full attention it deserves without the difficulty created by length constraints.

Finally, I'd like to apologize to the authors for the non-responsiveness of the Area Chair. The original Area Chair was not able to complete their duty and I have been belatedly assigned this paper to evaluate it, and it is clear that not as much discussion took place as would have been ideal.
Decision: Reject