To update or not to update? Neurons at equilibrium in deep models
Authors: Andrea Bragagnolo
Enzo Tartaglione
Marco Grangetto
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Learning
Equilibrium

Abstract: Recent advances in deep learning optimization showed that, with some a-posteriori information on fully-trained models, it is possible to match the same performance by simply training a subset of their parameters. Such a discovery has a broad impact from theory to applications, driving the research towards methods to identify the minimum subset of parameters to train without look-ahead information exploitation. However, the methods proposed do not match the state-of-the-art performance, and rely on unstructured sparsely connected models.
In this work we shift our focus from the single parameters to the behavior of the whole neuron, exploiting the concept of neuronal equilibrium (NEq). When a neuron is in a configuration at equilibrium (meaning that it has learned a specific input-output relationship), we can halt its update; on the contrary, when a neuron is at non-equilibrium, we let its state evolve towards an equilibrium state, updating its parameters. The proposed approach has been tested on different state-of-the-art learning strategies and tasks, validating NEq and observing that the neuronal equilibrium depends on the specific learning setup.

To update or not to update? Neurons at equilibrium in deep models
Andrea Bragagnolo University of Turin, Italy Synesthesia s.r.l,, Turin, Italy andrea. bragagnolodunito. it
Enzo Tartaglione
LTCI, Télécom Paris,
Institut Polytechnique Paris, France
enzo. tartaarrrlllooo wawaaagiiiieetellleeppepprr fr
Marco Grangetto
University of Turin, Italy
marco, marco. www.gggttoannnttoooo...m it
Abstract
Recent advances in deep learning optimization showed that, with some a- a-posteriori information on fully-trained models, it is possible to match the same performance by simply training subset of their parameters. Such discovery has a broad impact from theory to applications, driving the research towards methods to identify the minimum subset of parameters train without look-ahead information exploitation. However, the methods proposed do not match the state-of-the art performance and rely on unstructured sparsely connected models.
In this work we shift our focus from the single parameters to the behavior of the whole neuron, exploiting the concept of neuronal equilibrium (NEq). When a neuron is in a configuration at equilibrium (meaning that it has learned specific input-output relationship), we can halt update; on the contrary, when neuron is at non-equiibbrium we let its state evolve towards equilibrium state, updating its parameters. The proposed approach has been tested on different state-of-the- art learning strategies and tasks, validating NEq and observing that the neuronal equilibrium depends on the specific learning setup.
Introduction
In recent years, deep learning has become a staple solution to different tasks, such as computer vision, bio-informatics, speech recognition, and many more. Unfortunately, modern neural network architectures are becoming more and more "expensive": performing simple inferences requires lot of computational resources, and training the models even more. Such cost poses several challenges for the research community: the training of a network model is associated with large carbon footprints and the commercialization of Al research (especially for edge devices) is hindered by the resource requirements of the models Strubell et al. (2019).
For several years now, many works in literature have shown that is possible shrink both the size and resource requirements, mainly via quantization Yang et al. (2020); Jin et al. (2022) and pruning Wang al. (2020); (Tang al. (2020); Tanaka et al. (2020). particular, pruning techniques can drastically reduce the number of operations needed to perform inference, without affecting the overall performance of the model. Pruning targets the reduction of parameters in the model, which has advantages after training: unfortunately. it is unable to reduce the cost of the training cycle, and on the contrary, requires, general, iterations of few-shot pruning, followed by fine-tuning Lee et al. (2019): Tartaglione et al. (2018).
This paper has been accepted for publication the 36th Conference Neural Information Processing Systems (NeurIPS 2022).

---

A recent work shows the existence of sub-grue uub-graphs in the whole deep learning model which, when trained in isolation, match the performance of the whole model Frankle and Carbin (2019). This opens the road towards whole field of research, where many approaches are proposed to find these parameters a-priori: this shows a potential environmental impact, considering that a smaller model is trained, leading to a lower energy consumption [Stubblll et al. (201)). Indeed, when training a neural network, the proppapppaattion procedure and the weights update lead to the larger part of FLOPs (compared to forward propagation) Plaut et al. (1986); Baydin et al. (2018) 8). Researchers started to experiment pruning early in the training or even before the training starts in the hope of training a restricted number of parameters. Unfortunately, training sparse network with standard optimizers leads to subpar results Evei et al. (2019) or the final result does not differ much from magnitude pruning at the end of the training Frankle et al. (202T). The causes for such behaviors are still matter of debate among the community 2019: 2021
In this work, NEq, an approach to evaluate whether a given neuron is at equilibrium for the learning dynamics. If the neuron is in such state, its parameters have already reached a target configuration and do not require a further update. Unlike many other recent approaches, NEq disables entire neurons (hence, in a structured way), does not require prior knowledge of the specific task (for example by first training a model to convergence), and automatically self-adapts to the specific learning policy deployed. Unlike pruning techniques, NEq does not remove the neurons' contribution to the output; instead, it only prevents unnecessary updates to their weights: as a result, we reduce the number of operations performed by the back-propagation algorithm and the optimizer.
shift the focus from the single parameter to the whole neuron, and we propose
The rest of the paper is organized as follows. In Sec. _ we discuss the related literature; Sec. presents the concept of neuronal equilibrium and how to evaluate it; Sec. presents the experimental validation inclusive of an ablation study and Sec. draws the conclusions.
2 Related works
It is broadly acknowledged that the typically-deployer deep learning models on the ofate-of-the-art scenarios are over- wver-parametrized Mhaskar and Poggio (2016): [Brutzkus et al. (2018). This ignites two lines of research: reducing the size of these models (with pruning algorithms) or saving computational resources at training time. While the first has been broadly explored, the latter suffered stalemate until a recent work suggested its feasibility Frankle and Carbin (2019).
Pruning strategies. Attempts to reduce the number of parameters from learned models date back to 1989 when Mozer and Smolensky proposed skeletonization, a technique to identify less relevant neurons in a trained model and to remove them 1989 This was accomplished thanks to the evaluation of an error function penalty from a pre-trained model. the same years, LeCun et al. also proposed a work where the information from the second-order derivative of the error function is leveraged to rank the parameters of the trained model on a saliency-like basis LeCun et al.) (190)). In the last decade, thanks to the broad availability of computational resources, pruming approaches gained- back popularity, with approaches including the exploitation of dropout [Mochaaov et al. (2017);
Gomez et al. (219). sensitivity-based approaches Lee al. (2018); Tartaglione et al. (202) (022),
relaxation of l regularization Louizos et al. (2017): Sanh et al. (2020) and optimization of auxiliary parameters Xiao al. (2019). Despite leading to very compact deep models, the demanded training complexity is frequently very high, as many of the most well-kkownn approaches rely on iterative fine- tuning (or few-shot pruning) approaches. Hence, the training complexity for these techniques is larger than training a vanilla model (which in many cases is used as initialization). Much computational complexity could be saved if structured pruning is applied before training the model itself (namely, entire neurons/channels are pruned). Some works on pruning have suggested the possibility of re-allocating previously pruned parameters Tresp et al. (1996) or filters He al. (2018) to learn new functions and minimize the loss, towards higher generalization capability Blalock et al. (2020).
The lottery ticket hypothesis. In a recent paper, Frankle and Carbin provided empirical evidence that, at initialization, there exists sub-grapi of the original deep learning model such that, when trained in isolation, it can match the performance of the complete model Frankle and Carbin (0119). Indeed, the authors claim that the parameters in the very practical perspective, this means that all the parameters not in the selected sub-graph are, de facto, pruned from the model, not requiring any gradient computation. This empirical evidence potentially opens the road to the development of algorithms for pruning at initialization. Unfortunately, at such point, two obstacles are yet to be tackled. From one side, Frankle and Carbin (2019) uses an iterative
have "won at the lottery of ininalization" From a
2

---

poo pan PaINANN CAINNNNION
S Somiiaa und homm
Boo
E2
Viss
E1
Mig Ni
Vig
9/.
Mu
Model at time t-
Model at. time 4
Figure I: For a given time / the model (either in blue or orange) receives samples from the validation set (in red green). The output of the i-th neuron (whose cardinality is N2) depends on both the model's parameters and the specific sample on the validation set. These outputs are squeezed, concatenated and the obtained vector (of size N, ||Evaillo being || the cardinality of the validation set) then normalized, obtaining D
procedure to identify the sub-graph: it simply proves its existence but does not provide method to find directly at initialization. On another side, it focuses on un-structured sub-graphs, meaning that parameters are treated in isolation: is it possible to find structured sub-graphs (or in other words, removing entire nodes and not just ares)?
Beyond the lottery ticket hypothesis. Many works of the last two years have received inspiration from the lottery ticket hypothesis,
the quest determining the early lottery tickets. This is, however, tough task to solve: Frankle et al. 2020) shows that there is region, the very early stages of learning, where the lottery tickets identified with iterative pruning are "not stable" (meaning that tickets extracted different moments of this early stages essentially different). This suggests that, in the very first epochs, the neural network evolves in very different states, making the problem of a-priori identifying winning tickets hard Tartaglione (2022). This is also endorsed other works, including Morcos et al. (2019); | Malach et al. (2020). On the other hand, other approaches reduce the overall complexity of the iterative training by drawing early-bird tickets You et al. (2019) (meaning that they learn the lottery tickets when the model has not yet reached full convergence), even reducing the training data Zhang et al. (202I) or moving the first steps towards srrcctrally--paase winning tickets, yet still at iterative fashion, applying similar concept as Frankle and Carbin to entire neurons and channels Chen al. (2022).
With NEq we learn the important lessons related to the lottery ticket hypothesis, targeting the reduction of computational complexity training time, without exploiting knowledge on pre-training models, rewinding. Hence, we do not target the achievement of sparse architectures, but we aim at determining when a whole neuron requires to be updated when the computation of the gradient for its parameters is not necessary. As such, the comparison with the other presented approaches will be in general unfair, as they require much greater computational complexity for training because of un-structured sparsity (which introduces an overhead in the representation of the tensors) and the iterative strategies. Furthermore, we will observe the possibility, along the training process, that some neurons, already kept in "frozen" state, might unfreeze, requiring additional update steps. Although resource re-allocation has been exploited before Tresp et al. (999);: He et al. (2018), our unfreezing is different it involves learning of specific target function by the neuron, and not learning new ones through their re-allocation. In the next section, the notion of neuronal equilibrium will be presented, well the strategy to determine which neurons will require gradient computation.
3 Neurons at equilibrium
In this section, we will treat the problem of determining when a given neuron, along with the learning dynamics, finds itself at equilibrium. Towards this end. we define v'. as the output the i-th neuron when the input is fed to the whole model trained after epochs. Given set of inputs (where Emat is the validation set), it is possible to compare each n-th element vi.n.g with v),n.g for the same model's is input: what changes are the parameters of the model. Fig. provides an overview of the nomenclature used: the rest of the section we will see how to determine when a neuron is at equilibrium.
Epal
f-1
3

---

3.1 Neuronal equilibrium
In this section, we are interested in evaluating when the relationship between the input the model and the output of the i-th neuron is modified. When this happens, the neuron is at non-equilibrim, meaning that its learned function, in the whole picture (or in other words, taking into account the evolution of the neurons in the previous layers well), is still "evolving". We are interested in identifying the scenarios where the neuron at equilibrium at the net of the interactions with the other neurons. To assess let define the cosine similarity between all the outputs of the i-th neuron at time and at time - 1 for the whole validation set Eval as
N Bing Vi
0
Σ
Σ
EEEval
Here we can determine that, when Φ; = 1, the i-th neuron produces the same (eventually scaled) output between the evaluation at time and at time - 1 for the same input of the model. We say the i-th neuron reaches equilibrium when we have
lim of = k, 8100
(2)
where & E [-1:+1] is some constant value. We can have the following scenarios:
A = L. In this case, the two outputs are perfectly correlated, meaning that the relationship bounding the input of the whole model & and the output of the specific i-th neuron is maintained.
k E (0; 1). The outputs correlate, but we are in presence of an oscillatory behavior (in the sense that the cosine similarity varies by a constant value between consecutive evaluations). This effect can be caused by stochastic effects like high learning ratc/regularization, small batch size, or a combination them.
k E [] Also in this case we are in the presence of oscillatory behavior, but the outputs are anti-correlated or de-correlated.
Here follows the evaluation framework to determine the arrival to an equilibrium state for the i-th neuron.
3.2 Neuron dynamies evaluation
To assess the convergence equilibrium for (2). it is important evaluate the variation of the similarities of over time. Towards this end, let us introduce the variation of similarities
Ao = ot-!
(3)
According to the analysis in Sec. 3.1 in this case, we say we reach equilibrium when Adf - 0. Hence, is useful to keep track of the recent evolution over the similarity scores in the model: towards this end, we can introduce the velocity of the similarity variations:
USs = Aof - Menq 1-1 Adv"
(4)
where jleg is the momentum coefficient. We can rewrite obtaining
making the similarity scores explicit,
Φ + Σ (-1)m
m= [( + (Meg)) )m] 6i-m Deg + 0
(5)
Heg = 0,
where (-)T indicates power of m. we assume o' E [0; 1j (which is the case of ReLUUaativvtedd neurons), in order to prevent (5) from exploding, we need to set Meg E [0; 0.5). We can extend this setup to any layer if we assume that the neurons the trained model will reach equilibrium with anti-correlated outputs.

---

3.3 Selection of trainable neurons at non-equilibrium
To evaluate when a given neuron has reached equilibrium, exploiting 2 we can say that the i-th neuron is at equilibrium when it can satisfy
r'ol <,,
0.
(6)
It is important to notice that, once (6 is satisfied for certain t, in case something changes in the learning dynamics (for example, the learning rate is re-scaled), there may exist some 1 > such that the constraint is not satisfied anymore. When this happens, it means that the neuron is driven towards new states, and is no anymore at equilibrium. Hence, requires to be updated again.
3.4 Overall training scheme
BEGIN
Training epoch)
Trainable neurons
E
at non-equilibrium
Training set
Bag
YES
End of training?
NO
Neuron dynamics
END
evaluation
Validation
Learning steps
Identification of neurons equilibrium (NEq)
Figure 2: Overall training scheme. In orange is the standard training part and in blue the neuron equilibrium evaluation and selection stages (we name this whole part NEq).
The overall training scheme summarized in Fig. 2 The model is trained for one epoch, after which neurons at equilibrium are identified. We split this into phases: in the first (neuron dynamics evaluation), the velocity of the similarities is evaluated according to (4. while in the second (trainable neurons at non-equilibrium) the hidden neurons non-equilibrium, which will be trained for the next epoch, are identified according to (6). For the first epoch, all the neurons are considered at non-equilibrum by default. The evaluation of neurons at equilibrium is agnostic the general training strategy, which can include arbitrary re-scaling for the learning attc/typer-parameeer or the most common optimizers. the next section, we will this procedure on very different architectures, tasks, and learning strategies.
Experiments
In this section, we report the experiments supporting the approach as presented in Sec. 3.4 First, will perform an ablation study, analyzing single contributions for the introduced hyper-parameters and providing an overview of neuronal equilibrium along the training process; then, we will the proposed technique on tfate-of-the-att network architectures, datasets, and learning policies. All experiments were performed using 8 NVIDIA A40 GPUs and the source code uses PyTorch 1.10
4.1 Ablation study
We performed our ablation study training a ResNet-32 He et al. (2015) model on CIFAR- 10 Krizhevsky al.] Unless differently specified, following the syper-parametent setup of Zagoruyko
the source code available at htttp:://gttaub.coo/EDDD0//

---

10-1
Oprop, FLOPs per itter
220
Updated Neurons
Learning Rote
1.0
70
1.0
0.5
10-3
0.0
10.0
Epochs
Epochs
Epochs
(a) ResNet-31 32 trained with SGD,
Epoup, FLOPs per iter.
Learuing Rate
Up-inted Neurons
1.5
Acuurarg
10-7-
0.5
0.5
10-2
50
100
200
100
200
100
200
Epochs
Epochs
Epochs
(h) ResNet-32 trained with Adam.
Figure 3: Back proppagaatntn FLOPs (left, orange), updated neurons (center, green), and classification accuracy (right, red) for ResNet-32 trained on CIFAR-10.
10--
Accuray
80-
Mog 0.0
Byop. FLON
0.5
60-
0.5
Mog
30
200
250
100
150
200
200
Epachs
Epochs
Figure 4: Back-propagation FLOPs (left) and accuracy (right) for different values of Peg for ResNet- 32 trained on CIFAR-10.
and Komodakis 2016), the model is trained using SGD optimizer, with starting learning rate 7 = 0.1 and momentum Hopt = 0.9 and a weight decay of 5 x 10-4 for 250 epochs. The learning rate decayed by factor of 10 after 100 and 150 epochs, using the formulation as in (4). with | llo 50, Peq 0.5, and ε 0.001.
4.1.1 SGD vs Adam
To show that our technique automatically self-adapts to the training policy, we compare the evolution of the FLOPs required for back-propagation step and the number of updated neurons of two different training of the ResNet-32: one using the SGD optimizer with Hopt = 0.9, and the other using the Adam optimizer. For Adam leave the hyper-paramett 3 = 0.9, and B2 0.999) and use the same weight decay as for SGD (5 x 10- Fig. 3 Shows the trends for the training procedures. We the amount of the trained neurons (and the FLOPss required for the backward pass) is higher. This is related the general lack of equilibrium in the neurons of the network: high learning rates, the configuration of the neurons' parameters is subjected to high stochastic noise. As the train progresses, and the network progresses toward its final configuration, fewer and fewer neurons need be updated, Noticeably, Adam drives the neurons towards equilibrium in faster way, as expected; however, in simple tasks like the considered one, converges to lower accuracy scores (92.96% for SGD and 92.01% % for Adam). Furthermore, at the first learning rate decay (epoch 100), for SGD the number of updated neurons first decreases and then increases, such a phenomenon is not present the Adam case. This is explained by the different working principles of the two optimizers: SGD explores the solution space looking for large minima, searching for configurations that prevent equilibrium in high learning rate regimes.
to their default values (n 0.001,
that in the first phase of the train, where 1) high,

---

0.5
0.0
-0.5
-0.5
1.0
1.0
100
150
200
260
50
150
1000
200
Epochs
Epochs
(a) Distribution of &
(b) Distribution of Ag.
1.0
0.5
1.0
50
100
150
200
250
50
100
150
200
200
Epochs
Epochs
(c) Distribation of tho with Mag 0.5.
(d) Distribution of tae with Mag = 0.9.
Figure 5: Neuronal equilibrium--ret quantities for ResNet-32 trained on CIFAR-10. The red line indicates the average.
Table I: Ablation for ResNet-32 trained on CIFAR-10.
(a) Ablation on | loo-
(b) Ablation on
||
Bprop. FLOPs per iteration
Accuracy
Bprop. FLOPss per iteration
Accuracy
500
84.73M 629. 15K
92.70 1. 0.12
0.0
136.7IM 1: 15.34K
92.62 1: 0.23
250
82.62M = 613. 14K
92.80 + 0.43
0.0001
124.45M + 161.766
92.65 + 0.40
100
84.82M = 628.05K
92.81 0.15
0.0005
89.29M + 589.71K
92.69 0.19
50
84.81M = 629.12K
92.96 0.21
0.001
83.62M + 629.22K
92.96 + 0.21
25
=
0.28
:: 59L07K
10
84,91M = 627.1IK
92.70 0.27
0.01
52.53M :: 590.30K
91.23 4: 0.32
5
84,34M = 619.37K
92.57 = 0.48
0.05
16.10M + 254.94K
86.80 - 0.29
2
85.76M = 617.11K
92,80 +: 0.24
0.1
4,97M 1: 186.54K
83.90 -1 0.66
1
85.56M = 626.09K
92.77 + 0.23
0.5
1.78M + 137.9KK
76.78 + 2.57
4.1.2
Distribution of a & choice of Pog
Looking different values for Peg in Fig. accuracy. Despite without warranty from the theory, we tested a very large value for the momentum coefficient (0.9): the convergence of v,, shows that the neurons are in general in very correlated case of equilibrium, with very high values for is in (2). which is also empirically observed in Fig. 5a However, including very large value for Heg maintains the memory very old variations, producing sub-optimal reduction in terms of FLOPs reduction. We find that good compromise, supported by the findings as in Sec. 3.2 is to set Peg to 0.5. Fig. reports the distribution for the velocities for ø Ad, and VAo. observing that the average converges specific l for each of the three learning rates used.
observe for all the values convergence similar
4.1.3 Impact of the validation set size and g
Tab. lb provides empirical evaluation of the impact on the performance and on the FLOPs varying the validation set size. We indeed observe not a significant impact on the performance of the model varying it. Interestingly. the approach produces extremely good results even for extremely low cardinality for the validation set (down to even a single image): this can be explained by the presence of convolutional layers (the only fully-connected layer is the output layer, excluded by default) which even with little images produce high-dimensionality output in every neuron (Fig. I] and by the homogeneity the considered dataset. Investigating the impact of e, instead, we find for very high values of drop in performance, identifying good compromise for classification tasks to 0.001.
7

---

Table 2: Results of the application of NEq to each experimental setup, compared the stochastic approach. We report the average FLOPs per iteration at backpropagation and the final performance of the model evaluated on the test set (values annotated with report the classification accuracy, values annotated with f report the mean loU).
Dataset
Model
Approach
Bprop. FLOPs per iteration
Performance
Baseline
138.94M 1 0.0M
92.85% 1 0.23%1
Stochastic (p 0.2)
1 12,99M 0.00M (-18.68%)
92.78% + 0.19% (-0.07%))
CIFAR-10
ResNet-32
Stochastic (p 0.5)
69.75M 0.00M (-49.8%)
91.88% 1 0.27% (-0.97%)
Stochastic
86.34M + 0.00M (-37.85%)
92.23% + 0.25% (-0.62%)
Neq
84.8IM 0.63M (-38.96%)
92.96%
0.21% (+0.1
Baseline
3.64G 0.0G
69.90% 1 0.04%
Stochastic (p 0.2)
2.94G = 0.00G (-19.26%)
69.42% + 0.16% (-0.48%)
ResNet-18
Stochastic (p 0.5)
1.85G = 0.00G (-49.1 19%)
69.18% 1 0.03% (.0.72%))
Stochastic
2.82G 0.00G (-22.58%)
69.45% + 0.06% (-0.45%)
Neq
2.80G = 0.03G (-23.08%)
69.62% + 0.06% (-0.28%))
ImageNet-IK
Baseline
30.28G + 0.00G
84.71% % = 0.04%
Stochastic (p 0.2)
24.65G 0.00G (-18.6%)
84.54% + 0.04% (-0.83%))
Swin-B
Stochastic (p 0.5)
16.15G + 0.00G (-46.67%)
84.40% = 0.02% (-0.31%)
Stochastic
11.02G 0.00G (-63.67%)
84.27% + 0.04% (-0.44%)
Neq
10.78G 0.02G (-64.39%)
84.35% (-0.36%)
Baseline
305.06G 0.0G
67.71% 1 0.02%
Stochastic (p = 0.2)
248.69G 0.00G (-18.48%)
67.11% + 0.02% (-0.60%))
COCO
DeepLabv3
Stochastic (p 0.5)
163.42G 1 0.00G (-46.43%)
66.91% 1 0.04% (-0.80%)
Stochastic
229.00G
0.00G (-24.93%)
67.02% + 0.03% (-0.69%)
Neg
217.29G + 0.04G (-28.77%)
67.22% + 0.04% (-
4.2
Main experiments
In this section, show the results of the proposed method. For experiments, focused on different state-of-the-an architectures trained on standard classification and semantic segmentation datasets. All the learning policies used are borrowed from other works and un-optimized to test the adaptability of NEq.
ResNet-32 trained on CIFAR-10. The training spans 250 epochs, using SGD as optimizer with momentum Mope = 0.9, weight decay 5 x 10-4 and initial learning rate 7) 0.1, reduced by factor of 10 after 100 and 150 epochs. To evaluate the neuronal equilibrium we used a ||_..// ll of 50 images, Meg = 0.5, and = = 0.001. We used batch size of 100 images during training. ResNet-18 trained on ImageNet- ( as optimizer for 90 epochs, with 77 = 0.1, reduced by factor of 10 every 30 epochs, Hopt = 0.9 and
Krizhevsky al. (012). This model was trained with SGD
weight decay 10-4 using a batch size of 128. We used a E ll of 1.2k images, Heg = 0.5, and 0.001.
Swin Transformer Liu et al. 2021) (Swin-B) trained on ImageNet-IK. To test our technique on more modern models and training policies, we used the Swin-B architecture. Here we trained the model starting from a pre-trained checkpoint trained on ImageNet-211K following the official GitHub repository released under the MIT License. We used
Evat llo of 1.2k images, Heg 0.5, and
E = 0.001.
DeepLabv3 Chen et al. (211)) trained on COCO Lin et al. 2014). Other than classification tasks of varying complexity, we tested our procedure on a semantic segmentation problem. For this experiment, we used DeepLabv3 with a ResNet-50 backbone and the COCO dataset. To train the

---

network, we followed the state-of- of-the-art procedure defined in PyTorch We evaluated the neuronal equilibrium using a | llo of 320 images, Hog 0.5. and 0.02.
4.3
Discussion
The results (average over five different runs) are reported in Tab. For each experiment, we compare our technique with a "stochastic" approach. Namely, we randomly halt, at every epoch and with probability the update of given neuron. We test on three different probabilities: 0.2, 0.5, and probability that is as close as possible to the average over the one achieved by NEq indicated with "*" To evaluate the effectiveness of the proposed procedure we focus on the average computational complexity of the back-propp propagatton for single update iteration (expressed in FLOPs) and the network generalization capabilities at the end of the training. In all the considered scenarios, it is possible to observe a reduction of FLOPs with very marginal or no performance drop for NEq. When compared to the stochastic approach, with fixed probabilities, the amount of saved computation is similar all the considered scenarios, but the loss in performance varies, depending on the specific architectureddatase. On the contrary, NEg remains consistent in performance, self-adapting to the specific setup and saving the largest FLOPs for the given performance. Furthermore, testing the stochastic approach with the same FLOPs saving (hence, even letting that information leak in favor of the stochastic approach), the performance loss is lower.
Limitations. The current approach analyzes the behavior of an entire neuron. However, empirical experiments show that there could be further improvements considering ensembles of neurons. For example, Fig. 5 shows the average value for the similarities close constant but many neurons are still away from the convergence value, meaning that these neurons, at isolation, are still not equilibrium: is the scenario changing when considering the dynamics of groups of neurons? Furthermore, to validate the adaptability of NEq to the most popular training schemes, no optimization of the hyper-parameters for the training procedure has been performed (as it is out of scope for our evaluation). However, higher savings in computational complexity are possible by tuning the training strategy as well. In such a direction, prospectively, will be of interest to design more efficient learning strategies which keep into account the concept of neuronal equilibrium.
5
Conclusions
The work by Frankle and Carbin (019) showed the existence of sub-graphs in deep models which, when trained in isolation, can match the original performance of the whole model. Finding these sub-graphs is, however, a complex task, as the first stages of the learning the model itself is at non-equilibrium. Identifying these with a dynamic strategy, without requiring posterior over the whole training process, is a crucial task to be solved, towards computational resources saving. Differently from the vast majority of the literature which focuses on the identification of sub-graphs without any concrete computational saving (as they rely on iterative or roll-back algorithms), we have introduced the knowledge of neuronal equilibrium, looking for entire structures of the deep model at equilibrium, not requiring further optimization and gradient computation, which self-adapts to very specific experimental setups very different learning scenarios. This work opens the doors toward deeper understanding of the deep neural network's 's learning dynamics and to the development of new training strategies exploiting this knowledge.
Acknowledgements
This research was produced within the framework of Energy4Climate Interdisciplinary Center (E4C) of IP Paris and Ecole des Ponts ParisTech. This research was supported by 3rd Programme d' "nvestissemmnts d' Avenir [ANR-18 101-188EER-0000-000

---

References
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18:1-43, 2018.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learming and systems, 2:129-146, 2020.
A. Brutzkus, A. Globerson, E. Malach, and S. Shalev-Shwartz. Sgd learns over-parameterized networks that provably generalize on linearly separable data. 2018.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous 2017.
convolution for semantic image segmentation. arXiv preprint
Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. arXiv preprini MxKiv:2202.04777 2022.
Utku Evei, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv:/906. 10732, 2019.
J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural networks at initialization: Why we missing the mark? In International Conference Learning Representations, 2021. URL / het/forummiddIgeee Ig---Vc--VVVVM
Aidan N Gomez, Ivan Zhang. Siddhartha Rao Kamalakara, Divyam Madaan, Kevin Swersky, Yarin Gal, and Geoffrey E. Hinton. Learning sparse networks using targeted dropout. arXiv preprint arXiv:1905.13678 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03388, 2015.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 1808. 06866, 2018
Qing Jin, Jian Ren, Richard Zhuang, Sumant Hanumante. Zhengang Li, Zhiyu Chen, Yanzhi Wang, Kaiyuan Yang, and Sergey Tulyakov. F8net: Fixed-point 8-bit only multiplication for network quantization. In International Conference on Learning Representations, 2022. URL https
//openreview net/forummid= Seatmmdddddd_d_ddii___"" CfpJazzXT2
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research).
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
N. Lee, Thalaiyasingam Ajanthan, and Tom. Snip: Single-shot network pruning based connection sensitivity. ArXiv, abs//810.... 10.02334, 2019.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based connection sensitivity. arXiv preprint arXiv:1810.023400 2018
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
10

---

Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2021.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint arXiv:1712.01312, 2017.
Eran Malach, Gilad Yehudai, Shai Shalev-Schward and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine URL http:::/ppcceeedddiiiiiee
Learning Research, pages 6682-6691, PMLR, 13-18 Jul 2020.
mlr.
html
Hrushikesh N Mhaskar and Tomaso Poggio. Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications, 14(06):829 0408889998888-00- 2016.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pages 2498-2507. PMLR, 2017.
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. Advances in neural information processing systems, 32, 2019.
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from network via relevance assessment. In Advances in neural information processing systems, pages 107-115, 1989.
David C Plaut et al. Experiments on learning by back propagation. 1986.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning, Advances in Neural Information Processing Systems, 33:20378 203892 2020.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Anmual Meeting of the Association for Computa- tional Linguistics, pages 3645-3650, Florence, Italy, July 2019, Association for Computational Linguisties. doi: 10.18653/v1/1 1/P19-1355 URL http:////ccaantbloog.ooorgr/////
Hidenori Tanaka, Daniel Kunin, Daniel I. Yamins, and Surya Ganguli. Pruning neural networks with- out any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-6389, 2020.
Ychui Tang. Yunhe Wang. Yixing Xu, Dacheng Tao. Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. arXiv preprint 3rXIv:2010.10732. 2020.
Enzo Tartaglione. The rise of the lottery heroes: why zero-shot pruning is hard. arXiv preprint 2222: 2022.
Enzo Tartaglione, Skjalg Lepsoy, Attllio Fiandrotti, and Gianluca Francini. Learning sparse neural networks via consitivity-drivee regularization. Advances in neural information processing systems, 31, 2018.
Enzo Tartaglione, Andrea Bragagnolo, Francesco Odierna, Attilio Fiandrotti, and Marco Grangetto. Serene: Sensitivity- regularization of neurons for structured sparsity in neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2021.
regularization: towards deep sparse neural networks. Neural Networks, 146:230-22 227, 2022. Enzo Tartaglione, Andrea Bragagnolo, Attlio Fiandrotti, and Marco Grangetto. Loss- based sensitivity
Volker Tresp, Ralph Neuneier, and Hans- Hans-Georg Zimmermann. Early brain damage. Advances in
neural information processing systems, 9, 1996.
Chaoqi Wang, Guodong Zhang, and Roger Grosse Picking winning tickets before training by
preserving gradient flow. International Conference on Learning Representations, 2020. URI. itttps://openrevieee wttp://ooom..dd--/soooooo..

---

Xia Xiao, Zigeng Wang, and Sanguthevar Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. Advances in neural information processing systems, 32, 2019.
Li Yang, Zhezhi He, and Deliang Fan. Harmonious coexistence of structured weight pruning and ternarization for deep neural networks. In AAAI, pages 6623-6630, 2020.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang. Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint 1991 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C. 30.87. URL https: doi. org/10 500.052244...0 C.04//.33..0 80
Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data is more. In International Conference on Machine Learning, pages 12380-12390. PMLR,
2021.
12


Reviews:
[{"review_id": "u9iqIaGNod4", "reviewer": "Reviewer_qWfT", "summary": "This paper proposes an early stopping scheme for individual neurons in a neural network. The scheme detects neurons at equilibrium by ever-so-often comparing (between successive samples) the dot product of activations on a holdout/validation set, and freezing neurons when this value is zero. (If the value ceases to be zero, the neuron is correspondingly unfrozen.)\n\nThe authors choose hyperparameters using an ablation study, and evaluate the main method on four models across three datasets. They show that their method saves a significant amount of computation in exchange for minor reductions in performance.\n", "questions": "It is not clear to me how this algorithm is possible without imposing a great memory overhead – storing the variation of similarities, storing the activation patterns, etc. I would appreciate an accounting of the memory overhead\n\nWhat is the overall savings in wall-clock time for such computation? In real-world problems, computing dense matrix multiplication is vastly quicker than sparse, and its possible that this reduces the effective savings of the algorithm\n\nWhat is the standard deviation of the performance and/or savings over a few runs? It is not clear that the differences in performance between approaches (in Table 2) is significant. This is an absolutely critical limitation that must be addressed! (The variation in results is included in Table 1, which is great!)\n\nNeurons may unfreeze during training – what fraction of neurons exhibit this behavior, and how often does this occur?\n", "limitations": "- The significance of the results is not established. This is a critical flaw that should be fixed by presenting a standard deviation over a few runs for each row in Table 2.\n- The evaluation is relatively simple, particularly for such a general idea. This should be greatly expanded to better understand the performance of this method.\n- It is not clear that the FLOPs savings correspond to real-world savings. This should be tested.\n\nSmall points:\n\n - “Convergency” should be “convergence” (lines 49, 144, 262, others) \n - You can use only the year for the citation in line 64 with “\\citeyear”, this avoids repeating the author’s names.\n\n \n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work seems original, but the idea is close enough to the existing pruning literature that it may have been discovered previously. The paper is written clearly and understandably, with adequate reference to the existing body of knowledge, but the evaluation and discussion should be expanded upon. (See next answers for details.) I’m not certain that the work is significant enough for publication; perhaps it would be better suited to a workshop.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OKegYh7RtmN", "reviewer": "Reviewer_2pYr", "summary": "This paper decreased the trainable neurons in each epoch by exploiting a concept of neuronal equilibrium. If the estimated velocity of the similarity variation between the output of a neuron at time $t$ and $t-1$ is smaller than a threshold, the neuron is regarded as in the equilibrium state and will stop updating its own parameters. After testing the proposed method on different state-of-art tasks, results show that the proposed method can significantly decrease the FLOPs during back propagation with marginal or no performance drop, which is much better than randomly stopping the update of a subset of neurons. ", "questions": "1. Evaluating the equilibrium state for neurons also requires additional computational cost. Could you estimate the time complexity of the equilibrium evaluation + the back propagation using NEq and compare it with the time complexity of the baseline back propagation method? Then readers may understand how much it will decrease overall computational cost.\n2.  If the momentum coefficient $\\mu_{eq}$ is not zero, the similarity will still vary even if $v^t = 0 \\rightarrow \\Delta \\phi^t = \\mu_{eq} v^{t-1}$. Why can this state be regarded as equilibrium?\n3. The number of validation images is different in different experiments. Does the increase of the validation image affect the time cost severely? How do we choose a reasonable validation set size during training?\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method can decrease the training FLOPs significantly with very marginal or no performance drop.\n2. Compared to traditional pruning method, this method doesn't require additional fine-tuning process and thus simplifies the training process.\n3. The proposed method is very straightforward and easy to apply to different methods.\n\nWeaknesses:\n1. Evaluating the equilibrium state for neurons also requires additional computational cost. The time complexity of the equilibrium evaluation + the back propagation using NEq is unclear yet.\n2. The theoretical basis of the momentum coefficient is unclear.\n3. The effect of the number of validation images is unclear.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JwTEKbTjXs", "reviewer": "Reviewer_wKon", "summary": "The authors investigate a question that fits within into a recent fascinating line of investigation - namely that it is possible to find subsets of neurons with a neural network that, when trained, match the performance of the full network. Being able to rapidly identify this subset of neurons early on in training would enable networks to be trained at a much lower cost. Unfortunately, so far it has been challenging to identify this subset until late in training, at which point it no longer helps. \n\nThe authors pursue a related line of enquiry - is it possible to instead determine that, at a given point in training, a given neuron is \"at equilibrium\"? This is to say that its activity when driven by a specific input is consistent between training updates. The hypothesis is that such neurons can be excluded from training updates until they exit equilibrium. \n\nThe authors investigate this question empirically, and find some promising results suggesting that this is the case. They test their methods for several different architectures and tasks, finding consistent performance improvements in terms of FLOPS over the baseline. They compare this to a simpler approach in which subsets of neurons are randomly dropped out of training at each epoch. The improvement over this stronger baseline is less significant, although still measurable.", "questions": "My most important question is the following - what precisely is t in e.g. Eqn 1? Is this before and after training for a given epoch? If so, I don't understand the limit of t->inf in eqn. 2. Relatedly, there are not enough details on the precise algorithm (an algorithm box would be appreciated). In order to be able to decide to \"reenable\" neurons that have come out of equilibrium, presumably you need to do steps of gradient descent with all neurons being updated? All of these details were unclear to me.\n\nJust below eqn 2, three different conditions on k are laid out. What is the importance of distinguishing between [0,1] and [-1, 0]? This wasn't made particularly clear (I also assume a typo -> (k \\in (0;1) should be k \\in [0;1]).\n\nIts not clear that all the additional machinery of eqns 3 to 5 is necessary. E.g. Fig 4 suggests that \\mu=0 works more or less just as well.\n\nI might naively have assumed that equilibrium would mean k=1. This seems to be the case for essentially no neurons until epoch 150. Do you have a sense of why so many neurons are apparently oscillatory?\n\n\"We can extend this dimensioning for the momentum coefficient to any layer assuming that the neurons in the trained model will reach at most equilibrium with de-correlated outputs.\" - this sentence wasn't super clear to me.\n\nTable 1 is missing a caption, and in particular it would be useful to clarify that FLOPs is averaged across training.\n\nThe title of section 4.2 (experimental setup) is a bit confusing, given that you have just talked about an experiment in the preceding sections.\n\nIn the limitations section, its a bit unclear how you would extend your line of reasoning to consider groups of neurons.", "limitations": "As mentioned above, the proposed method only reduces training FLOPs. I further don't have a good sense of how significant the demonstrated savings are as compared to other SOTA techniques.\n\nI don't know of any potential negative societal impacts from this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am not an expert in the field, but I found the core results interesting, seemingly novel, and empirically robust. The methods they suggest seem fairly straightforward to implement. \n\nIn terms of weaknesses, there are several (crucial) aspects of the paper that are unclear -- these are outlined below. In addition, the proposed methods only reduce the FLOPs overhead for training, but do not reduce the network inference cost. This is in contrast to methods that find sparse subnetworks. Finally, I would have liked to see some more discussion on the practical implications of such techniques. As a non-expert, I might naively assume that these techniques would not necessarily lead to significant performance improvements on specialist accelerators. I suspect this is the case for many related techniques, but it would be nice to have these potential limitations more explicitly laid out.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ig2pYMCSamb", "reviewer": "Reviewer_MVNQ", "summary": "The authors propose a method to selectively choose which neuron's weights to update during training, based on the concept of Neuron Equilibrium.\nThe main idea is to stop backpropagation through neurons that reach equilibrium, so that no more (possibly useless, but compute-wasteful) backpropagation operations are performed.\nWhile the idea is intriguining, I would imagine the effect to have minimal impact on actual training times (i.e., no more than 1/2-1/3x the original training time, which may not be worth the implementation overhead; see below in Weaknesses); it would be very useful to see a comparison of wallclock training times.\n", "questions": "See 'weaknesses' above: what is the difference in wallclock training time between a baseline network trained in a typical way, vs using your method?", "limitations": "See 'weaknesses' above.", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of working at a neuron instead of connection level, while not per se novel, is useful and can indeed be beneficial.\n\nWeaknesses:\n- While training smaller networks or pruning them during training can improve training time of neural networks, it often is not crucial to do. As far as I am aware, network pruning is mostly useful to obtain smaller models for cheaper inference, or for inference on less powerful devices. In those cases, pruning can be just performed after training the larger network.\n- Following on my previous point: the main benefit of the authors' approach is that fewer operations are required during training, as the disabled neurons are only used in feedforward activation and not in backpropagation (which usually take a few times more operations than the corresponding feedforward ones). It would be useful to see a comparison of wallclock training times rather than just Backpropagation FLOPS. Looking at wallclock times should be especially interesting since the induced sparsity in backpropagation may actually lead to computational overheads on modern GPUs, where large matrix multiplications are very fast.\n\nMinor:\n- Watch out some language issues, e.g., \"like the same model already at convergency\" -> \"at convergence\".\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept