Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Feedforward networks can be converted into networks with temporal cascaded dynamics by introducing a propagation delay and re-training the model with a new loss. Via the use of skip connections, such cascade models can trade-off inference time and make predictions on prototypical images more quickly.

main_review: **Strengths**:
* The proposed cascade model improves over a previous model in terms of its time/accuracy tradeoff, might be more robust to dealing with certain kinds of noise, and intuitively takes longer to recognize difficult images
* The authors analyze the proposed model and posit that it classifies prototpical images more quickly than images in unusual views
* The paper takes a step to implementing meta-cognition with an additional model inspecting the time-dynamics of the cascade model (although this part comes quite short)

**Criticisms**:
* The proposed cascade model in its current form is limited in its generality: Analyses have only been run on datasets with small image inputs (CIFAR, TinyImagenet) and might not generalize to more natural images such as ImageNet
* Potential benefits of quick inference for prototypical images are never shown directly with respect to runtimes: one of the primary implied benefits of anytime prediction is the availability of initial class predictions after only little time has passed. This should be shown quantitatively, e.g. as a function of CPU cycles or on (simulated) neuromorphic hardware (e.g. Intel Loihi). Otherwise the paper is left with demonstrating that the proposed CascadedTD model improves over time, but it is unclear how its fast/slow predictions compare with a standard feedforward architecture.

Minor:
* the paper uses the brain as motivation, but never compares against any brain data. I would personally like this to be made more clear upfront.
* scholarship: the way the text is written still implies this to be the first model with time delays between neural transmissions. E.g. https://arxiv.org/abs/1804.10123 and https://www.biorxiv.org/content/10.1101/408385v1 address this very idea and are not mentioned in the paper (I pointed this out in an earlier review, so it's quite frustrating to still not see these papers mentioned/discussed. However I did notice improvements in the connection to neuroscience results which I am happy to see.)
* declaring using lambda=0 in line 187 and then choosing lambda=1 for SerialTD and SerialTD-Multihead is a bit confusing, I think one of these needs to be changed. I realize line 187 refers to only CascadedTD, but that choice was still surprising to me, especially given the improved performance of SerialTD(0) in Fig. A3

EDIT after rebuttal: The authors have responded with good quantitative arguments and laid out conceptual ideas more clearly. These revisions will make for a good paper, I have updated my score 5 -> 7.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to use skip connections in feedforward artificial neural networks to integrate information over multiple time steps by defining a delay for each connection in the network. Extensive experiments with a ResNet18 architecture show that this execution plan leads to an increase in early prediction accuracy when using anytime predictors, to faster predictions when samples are closer to a class prototype, more robustness to noise and a better out-of-distribution (OOD) detection compared to a sequential architecture. Temporal difference learning (TD) is utilized to train the neural network and an analysis is done on CIFAR-10, CIFAR-100 and TinyImageNet to show that the often used value of lambda=1 is suboptimal for these kinds of tasks. 

main_review: Overall, the theory is well explained and the experimental work meticulously shows how the combination of streaming rollout, temporal difference learning and anytime prediction leads to an improved network compared to sequential execution. While being very similar idea-wise to Fischer et al ([10] in the paper), additional experiments extend the previous work and undermine all claims made in the introduction.

Still, a few things are unclear after reading the paper. 
1. I think the work of Fischer et al would have to be discussed in a bit more detail, as the method is very similar in the way that OSD in this paper is exactly the streaming rollout in Fischer et al. There is also follow-up work on streaming rollouts with DenseNets, presented in Kugele et al [1], that is not discussed in the paper. 

2.  When scrutinizing TD(lambda) in your experiments, it is mentioned that TD(1) risks to overfit your test set. It is not clear to me, what this means (I guess you mean overfitting on the training set, leading to a worse performance on the test set?), and more importantly, why this should be the case.

3. When introducing Deep Cascaded Networks, it is not clear how exactly the readout for the anytime predictions is done. Are there 9 blocks and 9 time steps, and at each time step t, block output t is read out to make a prediction?

4. In Fig. 4: Why should multi-head be worse? In the worst case, can still learn that all heads are the same. Maybe another explanation could be overfitting?

5. In Tab. 1: As this is in my opinion a non-trivial measure, it would have been good to explain the coefficient in even more detail and how it relates to the results.

6. Generally, it would have been interesting to see experiments with other architectures, e.g. the mentioned U-Net, DenseNet or Transformers.

7. A comparison between EWS and OSD in terms of computational complexity (especially in the backward pass) would have been interesting as well. If I am not mistaken, gradient descent would lead to the problem that it has to backpropagate through all time steps in the case of EWS.

In summary, this paper is a relevant contribution to improve the understanding of rollouts. I think this paper is on the borderline of being ready for publication and I hope some of my concerns are addressed in the rebuttal. 

Small comments that do not influence my score:
- ln 149: seems like there are words missing
- Informal language: ln 213: Let's step; ln 78: home in -> converge
- ln 304: the word "input" has to be before "is presented"
- Overall it looks like the official style file is not used (e.g. sections are not numbered)


[1] Efficient Processing of Spatio-Temporal Data Streams With Spiking Neural Networks, A. Kugele, T. Pfeil, M. Pfeiffer, E. Chicca, Front. in Neuroscience, Vol 14 (2020)


------
After reading the other reviews and rebuttals, I increased my score from 5 to 6.
limitations_and_societal_impact: Limitations and societal impact were not discussed in detail.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose to tighten the link with biological neural networks by designing a cascaded network that is more compatible with the dynamics of information processing observed in the brain. In cascaded networks, all layers process the information in parallel and are fed with an updated input at each time step. The authors also introduce a loss, the Temporal Difference, that improve the performance of cascaded networks. The authors demonstrate that the cascaded network offers better trade-off accuracy and better noise robustness than serial one. In addition, the authors demonstrate that class instance that are rapidly recognized tend to be more prototypical in cascaded network
main_review: ## General comments :

The paper offers an interesting and novel perspective on the dynamics of neural network. The various experiments are all very relevant, and illustrate well the message of the paper. The two concerns I have are relatively minors : i) I found the paper hard to read because many sentences have just no meaning. Now that the rush of the submission is behind, the authors should take time to carefully re-read the article and correct the typos and also be sure that all the sentences has an English meaning. ii) It would have been interesting to confirm the results of the paper on a higher resolution database (e.g. imageNet).

Despite these 2 comments, I think the presented article is novel enough and should be useful for both the Neuroscientist and Machine Learning Community. This is in general a good article ( My score : 8).

## More specifics comments :
* Introduction : The concept of anytime prediction should be defined before the section ‘related work’ as it is crucial to understand the origin of the idea behind the paper. I would also suggest to define it before the introduction of figure 1b, which is a good example of an algorithm doing anytime prediction

* Figure 1. The figure 1 is very important to well understand the concept of the paper but is not clear enough on my mind. The color are not well chosen and a color legend is necessary to well grasp the message of the paper.

* Section : ' Deep Cascaded Network'. You said the batch norm has been changed to include ’time-step-conditional normalization’, but there is no further explanation. Could you please explain what is a time-step-conditional normalization ?

## Typos
Here are some of the typos I found (I quickly give up the idea to note all the typos because of their numbers)

Line 148 : ’The choice of alpha over a had no impact on our findings …’ —> ’ The choice of alpha had no impact on our findings

Line 185 : ‘lambda = 0 corresponds with requiring a prediction only of the wether on the next day’. This sentence has to be corrected

Line 219 : ’SINGLEHEADfor’ —> ’SINGLEHEAD for'

Line 304 : ‘an then the noise-free is presented input until … ‘. This sentence has to be corrected

limitations_and_societal_impact: The authors did not provide any insight on 'Limitations and Negative Impact'
review_title: null

Meta-review: This paper presents a novel ANN architecture (which the authors term "cascaded" networks) inspired by the real brain. In brief, this architecture introduces delays in the propagation between computational blocks, such that skip connections serve to provide a rapid inference, whereas over time, the effective depth of the architecture increases. They use this architecture for anytime prediction, and develop a temporal difference based training algorithm that encourages the networks to find rapid solutions to the inference problem when possible. They show that this approach provides some interesting and desirable properties, such as different "reaction times" to prototypical versus atypical samples and greater robustness.

The initial review scores were a mix. Nonetheless, the reviewers were generally positive, although there was some concerns about the generality and potential benefits of the approach for neuromorphic systems. However, after author responses and discussion, though there was still some divergence in scores, the reviewers largely agreed that this paper makes a sufficiently interesting and worthwhile contribution to the field for acceptance.
Decision: Accept (Poster)