Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper decreased the trainable neurons in each epoch by exploiting a concept of neuronal equilibrium. If the estimated velocity of the similarity variation between the output of a neuron at time $t$ and $t-1$ is smaller than a threshold, the neuron is regarded as in the equilibrium state and will stop updating its own parameters. After testing the proposed method on different state-of-art tasks, results show that the proposed method can significantly decrease the FLOPs during back propagation with marginal or no performance drop, which is much better than randomly stopping the update of a subset of neurons. 
strengths_and_weaknesses: Strengths:
1. The proposed method can decrease the training FLOPs significantly with very marginal or no performance drop.
2. Compared to traditional pruning method, this method doesn't require additional fine-tuning process and thus simplifies the training process.
3. The proposed method is very straightforward and easy to apply to different methods.

Weaknesses:
1. Evaluating the equilibrium state for neurons also requires additional computational cost. The time complexity of the equilibrium evaluation + the back propagation using NEq is unclear yet.
2. The theoretical basis of the momentum coefficient is unclear.
3. The effect of the number of validation images is unclear.

questions: 1. Evaluating the equilibrium state for neurons also requires additional computational cost. Could you estimate the time complexity of the equilibrium evaluation + the back propagation using NEq and compare it with the time complexity of the baseline back propagation method? Then readers may understand how much it will decrease overall computational cost.
2.  If the momentum coefficient $\mu_{eq}$ is not zero, the similarity will still vary even if $v^t = 0 \rightarrow \Delta \phi^t = \mu_{eq} v^{t-1}$. Why can this state be regarded as equilibrium?
3. The number of validation images is different in different experiments. Does the increase of the validation image affect the time cost severely? How do we choose a reasonable validation set size during training?

limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors investigate a question that fits within into a recent fascinating line of investigation - namely that it is possible to find subsets of neurons with a neural network that, when trained, match the performance of the full network. Being able to rapidly identify this subset of neurons early on in training would enable networks to be trained at a much lower cost. Unfortunately, so far it has been challenging to identify this subset until late in training, at which point it no longer helps. 

The authors pursue a related line of enquiry - is it possible to instead determine that, at a given point in training, a given neuron is "at equilibrium"? This is to say that its activity when driven by a specific input is consistent between training updates. The hypothesis is that such neurons can be excluded from training updates until they exit equilibrium. 

The authors investigate this question empirically, and find some promising results suggesting that this is the case. They test their methods for several different architectures and tasks, finding consistent performance improvements in terms of FLOPS over the baseline. They compare this to a simpler approach in which subsets of neurons are randomly dropped out of training at each epoch. The improvement over this stronger baseline is less significant, although still measurable.
strengths_and_weaknesses: I am not an expert in the field, but I found the core results interesting, seemingly novel, and empirically robust. The methods they suggest seem fairly straightforward to implement. 

In terms of weaknesses, there are several (crucial) aspects of the paper that are unclear -- these are outlined below. In addition, the proposed methods only reduce the FLOPs overhead for training, but do not reduce the network inference cost. This is in contrast to methods that find sparse subnetworks. Finally, I would have liked to see some more discussion on the practical implications of such techniques. As a non-expert, I might naively assume that these techniques would not necessarily lead to significant performance improvements on specialist accelerators. I suspect this is the case for many related techniques, but it would be nice to have these potential limitations more explicitly laid out.
questions: My most important question is the following - what precisely is t in e.g. Eqn 1? Is this before and after training for a given epoch? If so, I don't understand the limit of t->inf in eqn. 2. Relatedly, there are not enough details on the precise algorithm (an algorithm box would be appreciated). In order to be able to decide to "reenable" neurons that have come out of equilibrium, presumably you need to do steps of gradient descent with all neurons being updated? All of these details were unclear to me.

Just below eqn 2, three different conditions on k are laid out. What is the importance of distinguishing between [0,1] and [-1, 0]? This wasn't made particularly clear (I also assume a typo -> (k \in (0;1) should be k \in [0;1]).

Its not clear that all the additional machinery of eqns 3 to 5 is necessary. E.g. Fig 4 suggests that \mu=0 works more or less just as well.

I might naively have assumed that equilibrium would mean k=1. This seems to be the case for essentially no neurons until epoch 150. Do you have a sense of why so many neurons are apparently oscillatory?

"We can extend this dimensioning for the momentum coefficient to any layer assuming that the neurons in the trained model will reach at most equilibrium with de-correlated outputs." - this sentence wasn't super clear to me.

Table 1 is missing a caption, and in particular it would be useful to clarify that FLOPs is averaged across training.

The title of section 4.2 (experimental setup) is a bit confusing, given that you have just talked about an experiment in the preceding sections.

In the limitations section, its a bit unclear how you would extend your line of reasoning to consider groups of neurons.
limitations: As mentioned above, the proposed method only reduces training FLOPs. I further don't have a good sense of how significant the demonstrated savings are as compared to other SOTA techniques.

I don't know of any potential negative societal impacts from this work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors propose a method to selectively choose which neuron's weights to update during training, based on the concept of Neuron Equilibrium.
The main idea is to stop backpropagation through neurons that reach equilibrium, so that no more (possibly useless, but compute-wasteful) backpropagation operations are performed.
While the idea is intriguining, I would imagine the effect to have minimal impact on actual training times (i.e., no more than 1/2-1/3x the original training time, which may not be worth the implementation overhead; see below in Weaknesses); it would be very useful to see a comparison of wallclock training times.

strengths_and_weaknesses: Strengths:
- The idea of working at a neuron instead of connection level, while not per se novel, is useful and can indeed be beneficial.

Weaknesses:
- While training smaller networks or pruning them during training can improve training time of neural networks, it often is not crucial to do. As far as I am aware, network pruning is mostly useful to obtain smaller models for cheaper inference, or for inference on less powerful devices. In those cases, pruning can be just performed after training the larger network.
- Following on my previous point: the main benefit of the authors' approach is that fewer operations are required during training, as the disabled neurons are only used in feedforward activation and not in backpropagation (which usually take a few times more operations than the corresponding feedforward ones). It would be useful to see a comparison of wallclock training times rather than just Backpropagation FLOPS. Looking at wallclock times should be especially interesting since the induced sparsity in backpropagation may actually lead to computational overheads on modern GPUs, where large matrix multiplications are very fast.

Minor:
- Watch out some language issues, e.g., "like the same model already at convergency" -> "at convergence".

questions: See 'weaknesses' above: what is the difference in wallclock training time between a baseline network trained in a typical way, vs using your method?
limitations: See 'weaknesses' above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an early stopping scheme for individual neurons in a neural network. The scheme detects neurons at equilibrium by ever-so-often comparing (between successive samples) the dot product of activations on a holdout/validation set, and freezing neurons when this value is zero. (If the value ceases to be zero, the neuron is correspondingly unfrozen.)

The authors choose hyperparameters using an ablation study, and evaluate the main method on four models across three datasets. They show that their method saves a significant amount of computation in exchange for minor reductions in performance.

strengths_and_weaknesses: The work seems original, but the idea is close enough to the existing pruning literature that it may have been discovered previously. The paper is written clearly and understandably, with adequate reference to the existing body of knowledge, but the evaluation and discussion should be expanded upon. (See next answers for details.) I’m not certain that the work is significant enough for publication; perhaps it would be better suited to a workshop.

questions: It is not clear to me how this algorithm is possible without imposing a great memory overhead – storing the variation of similarities, storing the activation patterns, etc. I would appreciate an accounting of the memory overhead

What is the overall savings in wall-clock time for such computation? In real-world problems, computing dense matrix multiplication is vastly quicker than sparse, and its possible that this reduces the effective savings of the algorithm

What is the standard deviation of the performance and/or savings over a few runs? It is not clear that the differences in performance between approaches (in Table 2) is significant. This is an absolutely critical limitation that must be addressed! (The variation in results is included in Table 1, which is great!)

Neurons may unfreeze during training – what fraction of neurons exhibit this behavior, and how often does this occur?

limitations: - The significance of the results is not established. This is a critical flaw that should be fixed by presenting a standard deviation over a few runs for each row in Table 2.
- The evaluation is relatively simple, particularly for such a general idea. This should be greatly expanded to better understand the performance of this method.
- It is not clear that the FLOPs savings correspond to real-world savings. This should be tested.

Small points:

 - “Convergency” should be “convergence” (lines 49, 144, 262, others) 
 - You can use only the year for the citation in line 64 with “\citeyear”, this avoids repeating the author’s names.
review_title: null

Meta-review: metareview: Despite there not being complete agreement on the novelty of the method presented in the paper, most Reviewers praised the idea of proposing an early stopping scheme that is based on reusing concepts from the pruning literature but with the innovation of shifting the focus from connections to units.
A major criticism moved in the initial reviews was due to doubts on the contribution of the work and in particular its practical benefit in terms of training speedup. However, the rebuttals did a good job at dispelling these doubts.
Following the rebuttals, the paper garnered unanimously positive scores among Reviewers, owing to its significance for bridging the pruning literature with practical training speedup supported by convincing empirical evidence.
confidence: Less certain
Decision: Accept