Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss
Authors: Michael Louis Iuzzolino
Michael Curtis Mozer
Samy Bengio
Venue: NeurIPS 2021 Poster (2021)
Keywords: cascaded networks
anytime prediction
parallel networks
biologically inspired computing
speed-accuracy trade off
noise robustness
cognitive neuroscience
temporal difference methods

Abstract: Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics.  Deep nets typically operate in serial stages wherein each layer completes its computation before processing begins in subsequent layers.   In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading to speed-accuracy trade offs even in feedforward architectures. We explore the consequences of biologically inspired parallel hardware by constructing cascaded ResNets in which each residual block has propagation delays but all blocks update in parallel in a stateful manner. Because information transmitted through skip connections avoids delays, the functional depth of the architecture increases over time, yielding  anytime predictions that improve with internal-processing time. We introduce a temporal-difference training loss that achieves a strictly superior speed-accuracy profile over standard losses and enables the cascaded architecture to outperform state-of-the-art anytime-prediction methods. The cascaded architecture has intriguing properties, including: it classifies typical instances more rapidly than atypical instances; it is more robust to both persistent and transient noise than is a conventional ResNet; and its time-varying output trace provides a signal that can be exploited to improve information processing and inference.

Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Diffrrence Loss
Michael L. Iuzzolino
Google Research, Brain Team Department of Computer Science, University of Colorado
Michael C. Mozer
Google Research, Brain Team
Samy Bengio Google Rescarch, Brain Team*
Abstract
Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics. Deep nets typically operate in serial stages wherein each layer completes its computa- tion before processing begins in subsequent layers. In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading speed-accuracy trade offs even in feedforward architectures. We explore the consequences of biologically inspired parallel hardware by constructing cascaded ResNets in which each residual block has propagation delays but all blocks update in parallel in a stateful manner. Because information trans- mitted through skip connections avoids delays, the functional depth of the architecture increases over time, yielding anytime predictions that improve with internal-processing time. We introduce compora--difference training loss that achieves a strictly superior speed-accuracy profile over standard losses and enables the cascaded architecture outperform state-of-the-art anytime-prediction methods. The cascaded architecture ure has intriguing prop- erties, including: it classifies typical instances more rapidly than atypical instances; it is more robust to both persistent and transient noise than is a conventional ResNet; and its time-varying output trace provides signal that can be exploited to improve information processing and inference.
Since the earliest investigations of artificial neural nets, their design has been informed by biological neural nets 37.. Perhaps the most compelling example the convolutional net for machine vision, which has adopted properties of primate cortical neuroanatomy including a hierarchical layered organization, local receptive fields, and spatial equivariance 12] In this article, we investigate computational consequences of two fundamental propertics of biological information processing systems that have not been considered in the design of deep neural nets. First, the brain consists of massively parallel, dedicated hardware with neurons throughou the corlex updaling continuously and simullaneously, Second, information transmission between newrons introduces time delays DI As result, unrefined and possibly
*Cureetty at Apple.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

incomplete neural state in one region is transmitted to the next region even as the state evolves; and feedforward connectivity yields a speed-accuracy trade off in which the initial response to static input occurs rapidly but can be inaccurate, with the output gradually improving over internal processing time. Following McClelland Si architecture as cascaded.
we refer to such an
Cascaded dynamics contrast sharply with the dynamics of standard feedforward nets, which operate in serial stages, each layer completing its computation before subsequent layers begin processing. Cascaded dynamics are also quite different than the dynamics of vision models with recurrent connections [e.g.. 23 iteratively update, but layer updates are still computed serially with each layer completing its computation and then feeding it immediately to the next layer (or back to itself). Fundamentally, our investigation asks: Supposing we take a step toward biological realism with massively parallel hardware and relatively slow inter-neuron communication, what are the computational benefits and consequences?
25 38 Eri,, which, given static input, may
We construct cascaded networks by introducing propagation delays in deep feedforward nets provided with static imput. We treat the net as massively parallel such that all units across all layers are updated simultaneously and iteratively. We focus on the ResNet architecture L and we introduce a propagation delay into each residual block (Figure ( Because the skip connection permits faster transmission of more primitive perceptual representations, the functional depth of the resulting architecture increases over internal-processee yielding trade off between processing speed and complexity of processing. Consequently, the architecture offers a natural, integral mechanism for making predictions at any point in processing, known as anytime prediction by Speed-accuracy trade offs are a fundamental 22 and human perception has been modeled with deep learning anytime prediction methods
time,
characteristic of human information processing [22
Although we focus on the ResNet, our approach can be incorporated into any model with skip connections (e.g., Highway Nets AS DenseNet [9. U-Net μل] Transformers (22). ). The contrast between serial, one-layer-at-a-tine illustrated in Figures D and Db, respectively. To step through the operation of the cascaded model, at time 1, only the first residual block has received meaningful input, and the model prediction is therefore based only on this block's computation. At time 2, all higher residual blocks have received input from block 1, and the output is therefore based on all blocks' computations, though blocks 2 and above have deficient input. At each subsequent time, all blocks are receiving meaningful input, but it is not until time that block & has reached its asymptotic output because its input does not stabilize until t t- 1. In essence, the cascaded model behaves like WideResNet 500 on the first steps and then becomes a deep Res Net.
model and a cascaded, parallel-update model is
Our work makes the following key contributions.
We demonstrate the superiority of the cascaded architecture to the serial (Figures p..), indicating that parallelism can be exploited in way that has not previously been studied.
We propose and evaluate a novel training objective aimed at improving the predictions of anytime models. This temporal-differene response as quickly as possible. TD training improves the performance of both cascaded and serial architectures. Although a rich literature exists aimed at reducing the number of computational steps required obtain an accurate answer 2 3 [II 15 [[ I7 BN 40,
(TD) loss [0 encourages the most accurate
18 20 24 311 B1
45. 51 521 5Y , all of this work uses a degenerate form of TD for training and our results suggest that these models can be improved using TD.
The cascaded model trained with TD (CAsCADEDTD) tends to respond most rapidly to pro- totypccal exemplars, whereas training with the standard cross-entropy loss (CAscAdedCF)) does not (Figure 2 We assess with three quantitative prototypicality measures, and we further show that CASCADEDTD rapidly converges on the correct semantic family, whereas CASCADEDCE does not. These facts indicate that CASCADEDTD organizes knowledge differently across layers than does CASCADEDCE.
LLike much other research in deep learning 25 H. biology informs our work by providing novel forms of inductive bias. Our goal is to investigate computational consequences of these biases, not to model biological phenomena per se.
2

---

(a)
(b)
readout
(c)
readout
ReLU
Zigh
t=1
1=2
1=3
(=2
Figure 1: (a) ResNet building block, with additional delay component (A, in grey) that convolves temporal kernel with the block output. Details in text. (b) A standard serial ResNet is unrolled in time, with columns depicting time slices. Each rectangle is ResNet block, which may consist of two or more convolutional layers. the serial model, blocks are updated soquentially. Blocks which have not yet been activated are colored white and blocks which have been activated are shown in a hue unique to that block. The input is cyan. The narrow bars within each block signify the activation state of all blocks below that contributing the yellow trapezoid at the top, which enables anytime prediction.
the block's state (via skip connections).
Read from the model The narrow bars inside the trapezoid indicate the block information available at each time for classification (via skip connections). (c) A cascaded ResNet is unrolled in time. In the cascaded model, all blocks update in parallel; however, at each step, they may rely partial updates of lower blocks. As result, multiple processing steps required for layer's activation to reach its asymptotic state. The color intensity (saturation) of a block indicates how close block's activation state is to its asymptotic state.
We show that CASCADED obtains a strictly superior previously proposed anytime prediction models, which all based on a serial architecture.
profile compared to
We demonstrate other virtues of CASCADED is more robust input noise, and its time-varying output trace provides useful signals for meta-cognitive processes separately trained nets that make judgments about the cascaded architecture's accuracy.
Related Work
Prior research on cascaded models. From a psychological perspective, McClelland 36 characterizes human mental computation in terms of hierarchy of leaky integrators that continually transmit partial information as becomes available. We are aware of no work in deep learning static image processing with cascaded models, but there exist two investigations focused on video sequence processing, where the model state from the previous frame is used to efficiently process the next. Fischer et al. μII present streaming rollout framework for recurrent nets and they very bricfly explore the temporal dynamics of cascaded models, showing benefits carly predictions. They present general taxonomy that includes our proposed feodforward cascaded model, but their focus almost entirely on formal definitions and a framework that lays out the space of all well-formed roll out patterns (update orders). In contrast, our focus is almost entirely on training procedures that leverage the dynamics of cascaded models, on early read-out mechanisms, and on the computational consequences of these training and read-out mechanisms.. Kugele et al. 28 focus on spiking neural net dynamics, on time-varying inputs, and on reductions in latency that are obtained as a sequence unfolds due to autocorrelations in the input sequence. Notably, Kugele et al. explore variety of heuristic training losses and they settle on TD(1) as their preferred loss, but they do not explore the rest of the TD(A) family. Our work is complementary.
Carreira et al. 1 present causal video understanding model that performs depth-paralle computation with the objective of improving video processing efficiency via maximizing throughput, minimizing latency, and reducing clock cycles.
Recurvend nets for vision. Recurrent nets have been used in vision (e.g. 20 Ei 40 which adds dimension of internal processing time for every external input (see also
3

---

goloifih
Aclorion
CASCADEDCE
CASCADEDT
stnggrays
stingrays
bicycles
bicycles
(a)
(b)
liona
liona
ropid esemplors
slow exemplors
rapid exemplors
slow exemplors
Figure 2: (a) CIFAR-100 instances categorized rapidly (left) and slowly (right) by a cascaded model trained using a TD loss. (b) same as (a) for a standard cross-entropy loss. The cascaded model with TD loss stratifies instances by typicality, with rapid processing of prototypical views on homogeneous background and slow processing of unusual and cluttered views.
13 However, these models perform serial layerwise updates and therefore fundamentally differ in their operation from cascaded models.
Anytime prediction. Anytime prediction models
9
21
31
B0,
571 assumee serial operation of layers, but allow for predictions to be made from intermediate layers of the architecture. In the simplest case, after t steps, layers have been activated, and at each step, a prediction
55
51
made from the last activated layer [e.g., 16 24 Figure [ illustrates serial model that performs anytime prediction. Some of these models have intrinsic stopping criteria (e.g.. G] others rely on selection of stopping confidence threshold [e.g.. 24 51 54
Temporal-difference learning. TD learning has rich history, mostly in the RL community for value function ostimation. TD can be used for supervised learning as well. In fact, the two previous works in deep learning using cascaded models 5 |I perform boundary case of supervised TD, TD(1), which we show to have inferior performance. A variety of non-cascaded models, both recurrent 200 31
3S
5Yi and feedforward
15 IG 17
51 54, aim to reduce the number of computational steps required to obtain an accurate output. All use TD(1) for training. No previous research has explored the general formulation of TD for improving anytime prediction.
B 24
31
200
15
Deep Cascaded Networks
Many modern deep architecturcs-inchee
ResNet [4] Highway Nets A DenseNet 10
-incorporate skip connections that bypass strictly layered feedforward connectivity, analogous to the architecture of visual cortex [10. Under the biological assumption that signals transmitted through a neural layer are delayed relative to signals that bypass the layer, we construct a cascaded model using ResNets by introducing novel computational component. that delays the transmission of signals from the output of each computational layer, denoted A in Figure D.. Because these delays extend processing in time, the hidden states require a time index. The input to ResNet block i at time is denoted Zt,5. The block transforms this input via. the residual transform, yielding = F(z,,). We conceive of A as tapped delay-line memory of the transform history, Z,,- = [z/, 2|11; ... 21,g,, which is convolved with a temporal kernel k to produce the block output
U-Net 1
2t,+1 ReLU (zz,i+ Zi,
(1)
The kernel 0.... recovers the standard ResNet. in which communication be- tween layers is instant We consider two kernels to introduce time delays. With = (1- a) ... a discrete one-step delay is introduced (OSD for short). With a² a²...], we obtain exponentially weighted smoothing (EWS for short), where larger E [0, 1) yield slower transmission times. Note that both of these special kernels have efficient implementations: the OSD kernel with a one-clement queue and the EWS kernel with a finite (one-step) state vector and the incremental update,
Z aZ, + (1 a)²

---

62.3
Aourad (N)
Acorway
30.5
60.9
CIFAR-10 01
CIFAR-100
TinylmageNet
0.33
03
083
05
Figure Effect of TTD hyperparaneter À on CASCADED'TD test accuracy for three data sets. 1 corresponds to the training methodology all past research on anytime prediction, which is inferior to any À < 1 for all data sets. Shaded eтror bands-hard to see on most curves-ines curves-indicatee =1 SEM, corrected to remove performance variance due to weight initialization and unrelated to 35
We use the OSD kernel for training all models. Modifications of batch norm were required to do time-step-condiiionn which has 8 residual blocks and hence time delays. Note, we conducted experiments with larger ResNets and the additional compute did not affect qualitative properties. We also add time delay the output of the model's first convolutional layer. Consequently, with the OSD kernel, the cascaded model requires 9 updates for the output to reach asymptote. The cascaded and serial models with the same weights will necessarily produce identical asymptotic outputs.
normalization (see Appendix Aj. All experiments use a Nes-11,18,
To obtain a finer temporal granularity evaluation, some simulations switch to the EWS kernel with a = 0.9. Temporal dynamics are qualitatively similar for OSD and EWS, but EWS allows us to better distinguish individual examples in terms of their fine-grain timing, EWS with a 0.9 requires about 70 steps for the output to asymptote. We note that our findings are robust the choice of a, as long as slows transmission.
Training Cascaded Networks with TD(A)
To allow for anytime prediction, we include an output head following each the T residual blocks in both the serial and cascaded models (see Figure D..,, respectively). The output heads may share weights or have separate weights. To encourage correct outputs sooner, use temporal difference (TD) learning 40 over the output sequence. Readers may associate TD methods with reinforcement learning because TD methods have traditionally been used to predict future rewards. However, TD methods are fundamentally designed for supervised learning. We use TD to predict a future outcome the correct classification of an image from sequence of successively more informative states the information flowing through the ResNet at each internal time step.
TD(A) specifies target output Ye at each time t E {1....T} based on the model's actual output det: at future times + for > 0, and the eventual outcome or true target, Vtruel
V/ = (1
Σ X
Dcts
+ X + ytrues
(2)
where E [0, 1] is a free parameter that essentially specifies the time horizon for prediction TD(1) predicts the eventual outcome at each step; TD(0) predicts the model's output at the next step (and the eventual outcome at the final step). Given target Yt and actual output de, we specify a cross-entropy loss, L C Er H(ve, ft), where H(p, q) is the cross-entropy. Note that ye must be treated as a constant, not as a differentiable variable, via a stop gradient (for TensorFlow or Jax) or requires knowledge of all subsequent network states, the beauty of TD methods is that this loss can be computed incrementally (see Appendix I.2.Ip. The edge cases, TD(0) and TD(1), have trivial implementations. Past research has always used TD(1) for specifying intermediate targets, but will show that TD(1) leads to local optima because the model is penalized for being unable to classify correctly at the carliest steps.
grad=False (for PyTorch).. Although Equation g
We define 0 as is generally agreed upon in the algebra community.
5

---

Results
(A) Training
We conducted a sweep over hyperparameter À to determine its effect on asymptotic accuracy of CASCADEDTD. Figure 3 shows results from five replications of CASCADED on CIFAR-100, CIFAR-10, and TinyImageNee with classic studies with linear models 510 Chapter 12]. The same effect is observed where we train a subset of ImageNet. Importantly, À = which is the implicit choice of every previous anytime-predictte Bе
The hyperparameter has a systematic effect, consistent
with high resolution images; see Appendix A.6
model 54 57 obtains the poorest performance for all data sets, significantly worse than A= .5. The essential explanation is that larger À penalize the network for behavior it does not have the capability to achieve: obtaining the asymptotic prediction at the earliest time steps. To paraphrase the classic illustration of TD from Sutton pd if the task is predicting the weather on December 31, no model can predict as accurately on December 1 as on December 30. Selecting À < shortens the prediction horizon; x = O corresponds with requiring prediction only of the weather on the next day.
2 9 15 16 17 18
20
22
30 31
32!
4O 15 51
53
In the rest of the article, we report results for CASCADEDTD with À = 0, or TD(O). Although TD(0) is not optimal for all data sets, it is strictly superior to TD(1) and has trivial implementation because it does not require eligibility traces. Further, it avoids the need for a separate validation set to pick A.
Anytime Prediction and Speed-Accurage Trade Offss
Given a static input, an anytime prediction model attempts to obtain the best classification possible as quickly as possible. Anytime prediction can be performed by both serial and cascaded models. Both yield predictions at each time slice, as depicted by the yellow trapezoids in Figures D,,,, which denote model readout. Critical to anytime prediction is deciding when to terminate processing and initiate response 24 G 5T Following 24 and 5l, we assume that processing terminates when the confidence (probability) for the most likely class riscs above threshold 0. For any 0 one can mcasure the mcan stopping time and the mcan accuracy for all instances in a test sct. By swoeping 0 E [o, 1] and plotting mean accuracy as function of mcan stopping time, one obtains a speed-accuract trade off curve. Figure shows curves for models we'll describe next. The horizontal axis indicates number of simulation time steps, which is linearly related to the matched number of operalions (multiplies. additions, elc.) performed on simulated parallel hardware for all models. (See Appendix C for further details.)
To evaluate the cascaded model, we compare to a recent state-of-the-art method, the Shallow- Deep Network (SDN) [Zl The SDN has the serial architecture depicted in Figure D One critical design decision was whether to have separate read-out heads at cach stop (MULTHHEAA) or shared road-out head (SINCLEHEAD), i.c., whether weights are separate or shared. From the perspective of the cascadod model, which considers the vertical columns of Figure M: to be copies of network unrolled in lime, the SINGLEHEAD approach is natural. The SDN, as a serial model, chose the MULTHEAd approach. We tested all four logical combinations of (SERIAL.TD, CASCADED x (SINCLEHRAD, MULTHEAD). We use SeriaL and CASCADED'TD as shorthand for the SINGLEHEAD variants, and append MULTHHEaD to the consider SERIALCE and CASCADEDCEs which
model name for that version. Additionally, are SINGLEHEAD variants trained with the standard cross entropy loss that penalizes only asymptotic accuracy and does not explicitly attempt to obtain speeded response.
The key observations from Figure which shows speed-accuracy trade offs for the six models on three data sets, are as follows. First, our canonical cascaded model, CASCADEDTD, obtains better anytime prediction than BERIAI.DD-MULTiHE
(i.e., the architecture of SDN). CAsCADEDTD also achieves higher asymptotic accuracy; its accuracy malches that of CASCADEDCF,, a ResNet trained in the standard manner. Thus, cascaded models can exploit parallelism to obtain computational benefits in speeded perception without costs in

---

CIFAR-10
CIFAR-100
TinylmageNet
Goonanen
Me 2011
Geocaaca TD MuchHboe
Srmx.TD
Sesa.TD MucHHizo
SCRALCE
Figure 4: Speed accuracy trade off for three data sets and six models, obtained by varying stopping threshold and measuring mean latency and mean accuracy. CASCADEDTD our parallel anytime prediction model; SERIAL "D-MuLTHHEAD is the state-of-the-art method, SDN
accuracy Second, while MULTIHEAD superior to SINGLEHEAD for serial models, the reverse is true for cascaded models. This finding is consistent with the cascaded architecture's perspective on anytime prediction unrolled iterative estimation, rather than, as cast in SDN, as distinct read out heads from different layers of the network. Third, models trained with TD outperform models trained with standard cross-entropy loss. Training for speeded responses reorganizes knowledge in the network so that earlier layers are more effective in classifying instances. We now turn to better understand what this reorganization entails.
Organization of Knowledge in TD-Trained Cascaded Model
Having examined the response profile of our models over an evaluation set, we now turn to analyzing the response individual instances. Specifically, we ask about the time course of reaching a classification decision. We define the selection latency for an instance to be the minimum number of steps required to reach a confidence threshold on one class and maintain that level going forward, i.e., min {t [ ßr.j 0 v t' 2 ()), where I is the model output, j is an index over classes, and 0 the threshold. The selection latency does not specify whether or not the chosen class is correct. We picked threshold 0 = 0.83 for CAsCADEDTD ~ 10% the test examples failed to reach threshold; results that follow are
such that only robust to this selection.
What determines an instance's selection latency? Figure presents instances that have the lowest and highest latency-labed 'rapid' and 'slow', respectively. For CAsCADed (Figure 2a), notice the homogeneity of the rapid images: the objects are viewed from canonical perspective and lie against a solid background with no clutter in the image. In contrast, the slow images are more varied, both in the object's 's instantiation in the image and the background complexity. Turning to CASCADEDCE (Figure 2 instances do not appear to stratify by prototypicality. In the rest of this section, we formalize the notion of prototypicality with three measures and compute the correlation of cach measure with selection latency for the cascaded model trained with a TD loss (CAsCADEDTD) and with the standard cross-entropy loss (CAsCADEDCE). Our three measures are follows.
Centrality. We compute the cosine distance of an instance's embedding (the penultimate layer activation) and the target-class weight vector. The larger this quantity, the better aligned the two vectors are. Because the weight vector will tend to point near the center of class instances, the cosine distance is a measure of instance centrality.
C-score. Jiang et al. 21 describe an instance-based measure of statistical regularity called the C-score. The C-score is an empirical estimate of the probability that network will generalize correctly to an instance is held out from the training set. It reflects statistical regularity in that an instance similar to many other instances in the training set should have a high C-score.
Human labeling consistency. Peterson et al. 41 collected human labels on images. Most images are consistently labeled, but response distribution indicates inter-human human labeling agreement. Presumably, consistently labeled instances are more prototypical.
ambiguous. The negative entropy of the
*We used = 1 to train SERIAL'TD and DERIACTD-MULTTHHEE MULTHHEAD, as was done for all previously proposed serial models. Could SDN and other serial models be improved with X < 1? In Appendix A.5.2 show that the serial model with À 0 still does not perform well as CASCADED'TD.

---

of instance protolypicality and selection latency for CIFAR-10. Prototypicality measures are formulated such that lower values correspond to higher prototypicality. Since the Spearman coefficient measures the degree to which prototypicality varies with selection latency, large coefficients indicate a correlation between fast responses and the prototypicality of instances, whereas coefficients close to indicate correlation. Here, we show that the prototypicality and selection latency an instance cascaded models are correlated, with CASCADED yielding significantly higher correlations for two out of three measures as compared to CASCADEDCE.
Table 1: Spearman rank correlations between three
Spearman's P
Measure
CASCADEDCE
CASCADED TT
centrality
0.140
0.352
C-score
0.326
0.489
human consistency
0.153
0.142
All three measures are available only for the CIFAR-10 training set. Consequently, we ran 10-fold cross validation the training set, assessing the correlation based on the held out images in each fold. To obtain a gramular selection latency, we use the EWS kernel.
Table ! presents the correlation- Saannnpa'essss p between the three prototypicality measures and negative selection latency for CASCADEDCE and CASCADEDTD. positive cocfficient indicates shorter latency for prototypical instances. The cocfficient is reliably positive (p< < .001) for each of the three typicality mcasures and both models. However, CASCADEDTD obtains reliably higher correlations than CASCADEDCE on two of the three measures (p < .001); they are not significantly different on the human consistency measure (p= = .29). Thus, by these quantilative scores, the TD training procedure leads to better stratification of instances by typicality, line with the qualitative results presented Figure 2 Why does TD training distinguish instances based on prototypicality? Intuitively, prototypical instance shares features with many other class instances. Because these features are frequent in the data set, the TD loss focuses rapidly classifying instances with those features.
Beyond investigating the time course of fine-grain classification, we also examined coarso-grain classification. Forming twenty superclasses from the 100 fine-grain classes of CIFAR- 100, as specified in 20 we examined the probability of correct coarso-grain classification conditional on incorrect fine-grain classification. Zamir et al. [5 refer to this probability compliance, which reflects information being transmitted about coarse calegory even when the specific class cannot be determined. As shown in Figure 5 taxonomic compliance rises faster for CASCADEDTD than for CASCADEDCE. Whereas chance compliance is ,05, CASCADED'TD achieves a compliance probability of .35 after 2 steps. CASCADEDCE requires 8 steps to achieve the same performance. TD training pushes instances to the correct semantic neighborhood sooner,
laronomic
when not to the correct class label. This result further supports the reorganization knowledge for more robust decision making.
Robustness to Input Noise
In previous simulations, we have assumed the input was static while internal processing took
place. Now we consider static inputs with time-varying noise. Figure shows four types of lossy noise we consider CIFAR-10 images. The noise lypes are: (1) Focus: a 16 x 16 loveated patch randomly placed within the image, where regions outside of the patch are Gaussian blurred; (2) Perlin: gradient noise randomly applied to 40% of image pixels; (3)
Figure 5: CASCADED'TD performs coarse-grain classification on CIFAR-100 before fine-train clas- sification, as assessed by measure conditional taxonomic compliance 5inl Graphs are based on runs of each model with different random ini- tializations. Shaded error bands indicate +1 SEM, corrected to remove performance variance due to initial seed 35
0.4
Condijonal Tasonamic Complance
CAB
CABCADEDCE
0.0
2
4
Timeatop.
8

---

Figure 6: Lossy noise types. From left to right: Focus, Perlin, Occlusion, Resolution.
Occlusion: a 16 x 16 occluding patch randomly placod within the image; (4) Resolution: random downsampling by factors of 2x 4x via average pooling followed by k-ncarest upsampling to recover the original dimensionality of 32 x 32.
For each noise lype, CASCADEDCE and CASCADEDTD models are trained with the correspond- ing image translormation type as a data augmentation. Because the external environment changes more rapidly than any snapshot of the environment can be processed, cascaded models will necessarily integrate signals from multiple snapshots. To determine whether signal integration is beneficial for noise suppression, we compared to a serial model that is allowed to fully process each snapshot, which for the architecture requires 9x as many sequential updates as the cascaded model. We therefore refer to the model as SERIALCEX9; it has the same weights as CASCADEDCE.
Two noise variants are applied to test images: persistent, in which a noise sample is drawn independently at each updale slep, and Irunsient, which consists of three stages: first, the model reaches its asymplotic output by noise samples for a variable number steps; and third, the noise-free input is presented until the model returns to its previous asymptotic output. For persistent noise, we assess with asymptotic accuracy; for transient noise, we assess with a measure of drop in integrated performance over the course of the episode, which indicates how quickly the model recovers from noise perturbation (smaller is better). Simulation details provided in Appendix A.4
noise-free input; second, the input is corrupted
Table 2 indicates that CAsCADEDTD obtains a degree of robustness to persistent and transient noise not matched by the alternative models. Although SERIAL performs a full inference pass each noise perturbation, the stateful nature of CASCADEDTD allows it to smooth out noise via slow integration. Although CASCADEDCE, shares the same architecture as CASCADEDTD,, TD training is required to orchestrate the integration of content-specift perceptual information. This experiment indicates that laggy information processing in the cascaded model can be advantageous in noisy environment. In Appendix A.4 we note that the benefit for CASCADED applics only to lossy noisc, not translations and rotations, as one would expect from the perspective of noise averaging.
Meta-cognitive Inference
In this scction, we consider the hypothesis that temporally intermediate outputs from cascaded networks provide additional signals to improve performance. We term this metacognition, by reference to human abilities to reason about our reasoning processes.
The temporal trace of output from CASCADED TD is provided to separate classifier, METACOG- OOD, which is discriminatinelly trained for out-of-distribution (OOD) detection. METACoG- OOD is a fully connected feedforward net with 256-unit hidden layer and sigmoidal output unit for binary prediction: or for in- or out-of-distributio instances, respectively. CIFAR- 10's validation set serves the in-distribution training examples, whereas the validation sets of TinylmageNet,, LSUN, and SVHN serve as OOD training examples; see details in Appendix B.1 The CAsCADEDTD output is represented in one of four ways input to METACOG-O OOD: the confidence of the most probable class, known as the max softmar prediction (MSP),
Table 2: Experiments on persistent and transient input noise applied CIFAR-10. Highlight
indicates the best performance.
Persistent Noise
Transient Noise
Asymptotic Accurucy (%)
Drop in Integrated Performarice
Noise
SERAALCEx9
CASCADEDCE
TASCADRU
SERIAUCE99
CASCADEDCE
CAsCADEDTD
Focus
84.27 1 0.06
83.75 1 0.10
87.31 1 0.04
0.62 1 0.04
0.66 1 0.05
0.00 1 0.01
Occlusion
86.26 + 0.08
82.73 + 0.09
89.76 + 0.05
7.70 + 0.55
8.25 0.72
0.93 + 0.15
Perlin
85.18 + 0.08
84.56 + 0.05
87.67 + 0.08
0.86 + 0.06
0.87
0.07
0.00 + 0.01
Resolution
84.53 +: 0.07
85.40 + 0.07
88.19 + 0.10
0.81 + 0.06
0.53 + 0.05
0.18 + 0.02
9

---

AUMOC
Tewnteers
On CIFAR-10, 10, the output of CASCAD- EDIT over time provides reliable signal for improv- ing OOD detection over using just its asymptotic output. Larger AUROC and smaller FPR are better. All output representations benefit, from the temporal trajectory. Error bars reflect +1 SEM corrected to remove comparison-unrelee unrelated variance K55 is the final max-softmax prediction.
Figure 7:
Boscline
MSF
Entrooy
Sofmaxx
Logis
F9R 9806999866 199
All Teestge
Baseline
Bosa3m2
MFF
Entray
Sohmax
(2) entropy of the class posterior distribution, (3) the class posterior distribution, and (4) the logit representation of the posterior. We investigate whether feeding the output of all time steps to METACOG-OOD leads to improved prediction relative to feeding only the final asymptotic output. Only the latter information is available in standard feedforward net.
Following 3% wC assess OOD performance with AUROC, the area under the ROC curve, and FPR @ 95% TPR, the false positive rate at 95% true positive rate. A bascline metric is computed directly from the final max softmax predictions of the CASCADEDTD model, whereas the other metrics are based on the METACOG-OOF model oulput. Figure 7 indicates that the temporal output trajectory of CASCADED provides a valuable signal for OOD detection. Our goal here was not propose a method for OOD detection, but merely to demonstrate that in principle, there is information about the input that is conveyed by the cascaded model's dynamics but that is not available in traditional classifier's output.
Discussion
We investigated a neglected biologically-motivai architecture in which the bottleneck in neural information processing is transmission delays, not the number of neurons that can update in parallel. We proposed a lemporal-difference (TD) loss that yields improved speed-accuracy trade ofīs. We showed that this model beats the state-of-theart anytime prediction method, partly because of the TD loss and partly because of the model dynamics. The cascaded model has many distinctive properties, including: it classifies prototypical instances more rapidly than outliers; it performs coarse-to- fonfine semantic processing in which general semantic categories are rapidly inferred even when specific class labels are not; it is able to moderate time-varying imput noise; and the temporal trace of the model's output provides an additional signal that can be exploited to improve information processing beyond that provided by the asymptotic model output. Of course, these interesting properties come a computational cost when parallel hardware simulated on existing compute infrast We see three directions in which cascaded nets have particular polential.
For neuroscientists using deep nets model of human vision, cascaded nets approximation to the dynamics of the neural hardware. The properties we investigate neurons operate in parallel, neurons are stateful, and neurons are slow to transmit information- scem likely to have a critical impact on the nature of cortical computing. As one simple illustration, cortical feedback processes are often posited to be critical for explaining differences in processing efficiency of visual stimuli [e.g., 28 Eir. We have shown that these difference might be partly explained by feedforward cascaded dynamics.
better
For hardware rescarchers, cascaded networks are a possible direction for the future design of AI hardware. It direction quite unlike modern GPUs and TPUs, one that exploits massively parallel albeit slow and possibly noisy information processing. Our success in showing strong performance from cascaded models, well as a training procedure to obtain quick and accurate responses, should encourage research in this direction.
For AI research in anytime prediction, we've shown that existing models can be improved with a TD(A) loss; all past research has adopted À = 1, which we show to be inferior to < 1. For researchers who care little about cascaded models per se, cascaded models offer an intriguing method to train serial feedforward models. One can take a serial feedforward model, turn it into a cascaded model for training with TD methods, and then run it in scrial mode. We've 'v shown that TD training can improve asymptotic model accuracy while still providing anytime predictions due to inductive biases imposes on the organization of representations.
10

---

Acknowledgments
The authors thank Tyler Scott, Anirudh Goyal, Pradeep Shenoy, and particularly our three anonymous reviewers for thoughtful comments and feedback on earlier drafts of this work.
References
[1] W. Bialek and F. Rieke. Reliability and information transmission in spiking neurons. Trends in Neurosciences, 15(11) 15(11):428 4344, 1992.
[2] T. Bolukbasi, J. Wang, O. Dekel, and V. Saligrama. Adaptive neural networks for efficient. inference. In International Conference on Machine Learning, pages 527 -36.. PMLR, 2017.
3 A. Bulat, J. Kossaifi, G. Tzimiropoulos, and M. Pantic. Toward fast and accurate human pose estimation via soft-gated skip connections. arXiv preprint arXiv:2002. 11098, 2020.
[4] A. Bulat and G. Trimiropoulos. How far are we from solving the 2D & 3D face alignment problem? Tn Proceedings of the IEFE International Conference on Computer Vision, pages 1021-1030, 2017.
5] J. Carreira, V. Patrancean, L. Mazare, A. Zisserman, and S. Osindero. Massively parallel video networks. In Proceedings of the European Conference on Compuler Vision (ECCV), pages 649 666, 2018.
[6] X. Chen, H. Dai, Y. Li, X. Gao, and L. Song. Learning to stop while learning to predict. H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceediiggs of Machine Learning Research, pages 1520 1530. PMLR, 13 13-18 Jul 2020.
[7] J. Deng, W. Dong, R. Socher, L.-J. K. and L. Fei-Fei. Imagenet: largo-scale hierarchical image database. In 2009 IEEE conference computer vision and pottern recognition, pages 248 255. leee, 2009.
[8 J. J. DiCarlo, D. Zoccolan, and N. C. Rust. How does the brain solve visual object recognition? Neuron, 73(3):415-44) 2012.
9 M. Elbayad, J. Gu, E. Grave, and M. Auli Depth-adaptive transformer. In International Conference on Learning Representations, 2020.
[10] D. J. Felleman and D. C. Van Essen. Distributed hicrarchical processing in the primate cerebral cortex. In Cereb corter. Citeseer, 1991.
[11] V. Fischer, J. Köhler, and T. Pfeil. The streaming rollout of deep networks-towark fully model-parallel execution. In Advances in Newral Information Processing Systems, pages 4039-4050, 2018.
[12] K. Fukushima. Neocognitron: A self-organizing neural network model for mechanism pattern recognition unaffected by shift in position. Biological Cybernetics, 36:193 202, 1980.
[13] A. Graves. Adaptive computation time for recurrent neural networks. CoRR, abs/ htss/163.0000 2016.
[14] K. He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770 778, 2016.
[15] H. HU, D. Dey, M. Hebert, and J. A. Bagnell. Anytime neural network: versatile trade-off between computation and accuracy, 2018.
[16] H. Hu, D. Dey, M. Hebert, and J. A. Bagnell. Learning anytime predictions in neural networks via adaptive loss balancing. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 33, pages 3812 3821, 2019.
11

---

[17] G. Huang, D. Chen, Li, F. Wu, L. van der Maaten, and K. Weinberger. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representalions, 2018.
[18] G. Huang, D. Chen, T. Li, F. Wu, L. van der Maaten, and K. Q. Weinberger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1103 703.09844, 2017.
[19] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 4700 4708, 2017.
[20] M. Tuzzolino, Y. Singer, and M. C. Mozer. Convolutional bipartite attractor networks. or Xiv preprint or Xiv: 906. 03500, 2019.
[21] Z. Jiang, C. Zhang, K. Talwar, and M. C. Mozer. Characterizing structural regularities of labeled data in overparameterized models. In Proceedings of the International Conference on Machine Learning, 2021.
[22] M. Jones, S. Kinoshita, and M. C. Mozer. Optimal response initiation: Why recent experience matters. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2009.
[23] K. Kar, J. Kubilius, K. Schmidt, E. B. Issa, and J. J. DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior. Nature Neuroscience, 04/2019 2019.
[24] Y. Kaya, S. Hong, and T. Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking, In International Conference on Machine Learning, pages 3301 3310. PMLR, 2019.
[25] N. Kriegeskorte. Deep neural networks: A new framework for modeling biological vision and brain information processing. Annual Review of Vision Science, 1(1):417-446, 446, 2015.
[26] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Computer Science, University of Toronto, 2009.
[27] J. Kubilius, M. Schrimpf, A. Nayebi, D. Bear, D. L. K. Yamins, and J. J. DiCarlo. Cornet: Modeling the neural mechanisms of core object recognition. bioRwiv, 2018.
[28] A. Kugele, T. Pfeil, M. Pfeiffer, and E. Chicca. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14:439, 2020,
[29] O. Kumbhar, E. Sizikova, N. J. Majaj, and D. G. Pelli. Anylime prediction as model of human reaction time. CoRR, 244/2011.128555 2020.
[30] G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In International Conference Learning Representations, 2017.
[31] H. Lee and J. Shin. Anytime neural prediction via slicing networks vertically. arxiv preprint Xrxi 807,02609, 2018
[32] S. Leroux, P. Molchanov, P. Simoens, B. Dhoedt, T. Breuel, and Kautz. Jamnn: Iterative and adaptive mobile neural network for efficient image classification, 2018.
[33] S. Liang, Y. Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. ar preprint arxi 1ox.::1:6..00000 706.02690, 2017.
[34] E. S. Marquez, J. S. Hare, and M. Niranjan. Deep cascade learning. IEFE transactions neural networks and learning systems, 29(11):5475 5485, 2018.
[35] M. E. Masson and G. R. Loftus. Using confidence intervals for graphically based data interpretation. Canadian Journal of Eaperimental Psychology/ Revue canadienne de psychologie expérimentale, 57(3))203, 2003.
12

---

[36] J. L. McClelland. On the time relations of mental processes: an examination of systems of processes in cascade. Psychological review, 86(4):287) 1979.
[37] W. S. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathemalical Biophysics, 5(4::115 133, 1943.
[38] L. McIntosh, N. Maheswaranathan, D. Sussillo, and J. Shlens. Recurrent segmentation for variable computational budgets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1648-1657, 2018.
[39] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Fealure Learning 2011, 2011.
[40] A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision, pages 483-499. Springer, 2016.
[41] J. C. Peterson, R. M. Battleday, T. L. Griffiths, and O. Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE International Conference on Compuler Vision, pages 9617 9626, 2019.
[42] R. Rateliff and G. McKoon. The diffussin decision model: theory and data for two-choice decision tasks. Neural Computation, 20(4):873 922, 2008.
[43] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomodical image segmentation. In International Conference on Medical image computing and computer-assiste intervention, pages 234-241. Springer, 2015.
[44] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 252, 2015.
[45] S. Scardapane, M. Scarpiniti, E. Baccarelli, and A. Uncini. Why should we add carly cxits to neural networks? Cognitive Computation, 12(5):9544900 2020.
[46] C. Spocrer, T. C. Kietzmann, J. Mehrer, Charest, and N. Kriegeskorte. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. bioRaiv, 2020.
[47] C. J. Spoerer, P. McClure, and N. Kriegeskorte. Recurrent convolutional neural networks: A better model of biological object recognition. Frontiers in Psychology, 8:1551, 2017.
[48] R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks. arXiv preprint ar 175::500.000 00387, 2015.
[49] R. S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9 44, 1988.
[50] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[51] S. TeerapiiLayanon, B. McDanel, and H.-T. Kung. Branchynel: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pallern Recognition (ICPR), pages 2464 2469. IEEE, 2016.
[52] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[53 X. Wang, Y. Tuo, D. Crankshaw, A. Tumanov, F. Yu, and J. F. Gonzalez. Tdk cascades: Fast deep learning by learning not to overthink. or preprint Xrxiv 00885, 2017.
13

---

[54] L. Yang, Y. Han, X. Chen, S. Song, J. Dai, and G. Huang. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pages 2369 2378, 2020.
[55] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506 03365, 2015.
[56] S. Zagoruyko and N. Komodakis. Wide residual networks. orXiv preprint ar
2016.
[57] A. R. Zamir, T.L. Wu, L. Sun, W. B. Shen, B. E. Shi, J. Malik, and S. Savarese. Feedback networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1308-1317, 2017.
588 S. Zilberstein. Using anytime algorithms in intelligent systems. AT magazine, 17(3):73-7 73,
1996.
14


Reviews:
[{"review_id": "EBcbqAFsKTz", "reviewer": "Reviewer_i5tq", "summary": "Feedforward networks can be converted into networks with temporal cascaded dynamics by introducing a propagation delay and re-training the model with a new loss. Via the use of skip connections, such cascade models can trade-off inference time and make predictions on prototypical images more quickly.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n* The proposed cascade model improves over a previous model in terms of its time/accuracy tradeoff, might be more robust to dealing with certain kinds of noise, and intuitively takes longer to recognize difficult images\n* The authors analyze the proposed model and posit that it classifies prototpical images more quickly than images in unusual views\n* The paper takes a step to implementing meta-cognition with an additional model inspecting the time-dynamics of the cascade model (although this part comes quite short)\n\n**Criticisms**:\n* The proposed cascade model in its current form is limited in its generality: Analyses have only been run on datasets with small image inputs (CIFAR, TinyImagenet) and might not generalize to more natural images such as ImageNet\n* Potential benefits of quick inference for prototypical images are never shown directly with respect to runtimes: one of the primary implied benefits of anytime prediction is the availability of initial class predictions after only little time has passed. This should be shown quantitatively, e.g. as a function of CPU cycles or on (simulated) neuromorphic hardware (e.g. Intel Loihi). Otherwise the paper is left with demonstrating that the proposed CascadedTD model improves over time, but it is unclear how its fast/slow predictions compare with a standard feedforward architecture.\n\nMinor:\n* the paper uses the brain as motivation, but never compares against any brain data. I would personally like this to be made more clear upfront.\n* scholarship: the way the text is written still implies this to be the first model with time delays between neural transmissions. E.g. https://arxiv.org/abs/1804.10123 and https://www.biorxiv.org/content/10.1101/408385v1 address this very idea and are not mentioned in the paper (I pointed this out in an earlier review, so it's quite frustrating to still not see these papers mentioned/discussed. However I did notice improvements in the connection to neuroscience results which I am happy to see.)\n* declaring using lambda=0 in line 187 and then choosing lambda=1 for SerialTD and SerialTD-Multihead is a bit confusing, I think one of these needs to be changed. I realize line 187 refers to only CascadedTD, but that choice was still surprising to me, especially given the improved performance of SerialTD(0) in Fig. A3\n\nEDIT after rebuttal: The authors have responded with good quantitative arguments and laid out conceptual ideas more clearly. These revisions will make for a good paper, I have updated my score 5 -> 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "BVlRabhD04w", "reviewer": "Reviewer_Qwk8", "summary": "The authors propose to tighten the link with biological neural networks by designing a cascaded network that is more compatible with the dynamics of information processing observed in the brain. In cascaded networks, all layers process the information in parallel and are fed with an updated input at each time step. The authors also introduce a loss, the Temporal Difference, that improve the performance of cascaded networks. The authors demonstrate that the cascaded network offers better trade-off accuracy and better noise robustness than serial one. In addition, the authors demonstrate that class instance that are rapidly recognized tend to be more prototypical in cascaded network", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## General comments :\n\nThe paper offers an interesting and novel perspective on the dynamics of neural network. The various experiments are all very relevant, and illustrate well the message of the paper. The two concerns I have are relatively minors : i) I found the paper hard to read because many sentences have just no meaning. Now that the rush of the submission is behind, the authors should take time to carefully re-read the article and correct the typos and also be sure that all the sentences has an English meaning. ii) It would have been interesting to confirm the results of the paper on a higher resolution database (e.g. imageNet).\n\nDespite these 2 comments, I think the presented article is novel enough and should be useful for both the Neuroscientist and Machine Learning Community. This is in general a good article ( My score : 8).\n\n## More specifics comments :\n* Introduction : The concept of anytime prediction should be defined before the section ‘related work’ as it is crucial to understand the origin of the idea behind the paper. I would also suggest to define it before the introduction of figure 1b, which is a good example of an algorithm doing anytime prediction\n\n* Figure 1. The figure 1 is very important to well understand the concept of the paper but is not clear enough on my mind. The color are not well chosen and a color legend is necessary to well grasp the message of the paper.\n\n* Section : ' Deep Cascaded Network'. You said the batch norm has been changed to include ’time-step-conditional normalization’, but there is no further explanation. Could you please explain what is a time-step-conditional normalization ?\n\n## Typos\nHere are some of the typos I found (I quickly give up the idea to note all the typos because of their numbers)\n\nLine 148 : ’The choice of alpha over a had no impact on our findings …’ —> ’ The choice of alpha had no impact on our findings\n\nLine 185 : ‘lambda = 0 corresponds with requiring a prediction only of the wether on the next day’. This sentence has to be corrected\n\nLine 219 : ’SINGLEHEADfor’ —> ’SINGLEHEAD for'\n\nLine 304 : ‘an then the noise-free is presented input until … ‘. This sentence has to be corrected\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not provide any insight on 'Limitations and Negative Impact'", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5 h"}, "extra_text": {}}, {"review_id": "1r8QtnW1AW5", "reviewer": "Reviewer_gw8x", "summary": "This paper proposes to use skip connections in feedforward artificial neural networks to integrate information over multiple time steps by defining a delay for each connection in the network. Extensive experiments with a ResNet18 architecture show that this execution plan leads to an increase in early prediction accuracy when using anytime predictors, to faster predictions when samples are closer to a class prototype, more robustness to noise and a better out-of-distribution (OOD) detection compared to a sequential architecture. Temporal difference learning (TD) is utilized to train the neural network and an analysis is done on CIFAR-10, CIFAR-100 and TinyImageNet to show that the often used value of lambda=1 is suboptimal for these kinds of tasks. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the theory is well explained and the experimental work meticulously shows how the combination of streaming rollout, temporal difference learning and anytime prediction leads to an improved network compared to sequential execution. While being very similar idea-wise to Fischer et al ([10] in the paper), additional experiments extend the previous work and undermine all claims made in the introduction.\n\nStill, a few things are unclear after reading the paper. \n1. I think the work of Fischer et al would have to be discussed in a bit more detail, as the method is very similar in the way that OSD in this paper is exactly the streaming rollout in Fischer et al. There is also follow-up work on streaming rollouts with DenseNets, presented in Kugele et al [1], that is not discussed in the paper. \n\n2.  When scrutinizing TD(lambda) in your experiments, it is mentioned that TD(1) risks to overfit your test set. It is not clear to me, what this means (I guess you mean overfitting on the training set, leading to a worse performance on the test set?), and more importantly, why this should be the case.\n\n3. When introducing Deep Cascaded Networks, it is not clear how exactly the readout for the anytime predictions is done. Are there 9 blocks and 9 time steps, and at each time step t, block output t is read out to make a prediction?\n\n4. In Fig. 4: Why should multi-head be worse? In the worst case, can still learn that all heads are the same. Maybe another explanation could be overfitting?\n\n5. In Tab. 1: As this is in my opinion a non-trivial measure, it would have been good to explain the coefficient in even more detail and how it relates to the results.\n\n6. Generally, it would have been interesting to see experiments with other architectures, e.g. the mentioned U-Net, DenseNet or Transformers.\n\n7. A comparison between EWS and OSD in terms of computational complexity (especially in the backward pass) would have been interesting as well. If I am not mistaken, gradient descent would lead to the problem that it has to backpropagate through all time steps in the case of EWS.\n\nIn summary, this paper is a relevant contribution to improve the understanding of rollouts. I think this paper is on the borderline of being ready for publication and I hope some of my concerns are addressed in the rebuttal. \n\nSmall comments that do not influence my score:\n- ln 149: seems like there are words missing\n- Informal language: ln 213: Let's step; ln 78: home in -> converge\n- ln 304: the word \"input\" has to be before \"is presented\"\n- Overall it looks like the official style file is not used (e.g. sections are not numbered)\n\n\n[1] Efficient Processing of Spatio-Temporal Data Streams With Spiking Neural Networks, A. Kugele, T. Pfeil, M. Pfeiffer, E. Chicca, Front. in Neuroscience, Vol 14 (2020)\n\n\n------\nAfter reading the other reviews and rebuttals, I increased my score from 5 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact were not discussed in detail.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel ANN architecture (which the authors term "cascaded" networks) inspired by the real brain. In brief, this architecture introduces delays in the propagation between computational blocks, such that skip connections serve to provide a rapid inference, whereas over time, the effective depth of the architecture increases. They use this architecture for anytime prediction, and develop a temporal difference based training algorithm that encourages the networks to find rapid solutions to the inference problem when possible. They show that this approach provides some interesting and desirable properties, such as different "reaction times" to prototypical versus atypical samples and greater robustness.

The initial review scores were a mix. Nonetheless, the reviewers were generally positive, although there was some concerns about the generality and potential benefits of the approach for neuromorphic systems. However, after author responses and discussion, though there was still some divergence in scores, the reviewers largely agreed that this paper makes a sufficiently interesting and worthwhile contribution to the field for acceptance.
Decision: Accept (Poster)