Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss

Since the earliest investigations of artificial neural nets , their design has been informed by biological neural nets [ 37 ] . Perhaps the most compelling example is the convolutional net for machine vision , which has adopted properties of primate cortical neuroanatomy including a hierarchical layered organization , local receptive fields , and spatial equivariance [ 12 ] . In this article , we investigate computational consequences of two fundamental properties of biological information processing systems that have not been considered in the design of deep neural nets . First , the brain consists of massively parallel , dedicated hardware with neurons throughout the cortex updating continuously and simultaneously . Second , information transmission between neurons introduces time delays [ 1 ] . As a result , unrefined and possibly úCurrently at Apple . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . incomplete neural state in one region is transmitted to the next region even as the state evolves ; and feedforward connectivity yields a speed-accuracy trade o in which the initial response to a static input occurs rapidly but can be inaccurate , with the output gradually improving over internal processing time . Following McClelland [ 36 ] , we refer to such an architecture as cascaded . Cascaded dynamics contrast sharply with the dynamics of standard feedforward nets , which operate in serial stages , each layer completing its computation before subsequent layers begin processing . Cascaded dynamics are also quite di erent than the dynamics of vision models with recurrent connections [ e.g. , 23 , 25 , 38 , 47 ] , which , given a static input , may iteratively update , but layer updates are still computed serially with each layer completing its computation and then feeding it immediately to the next layer ( or back to itself ) . Fundamentally , our investigation asks : Supposing we take a step toward biological realism with massively parallel hardware and relatively slow inter-neuron communication , what are the computational benefits and consequences ? 2 We construct cascaded networks by introducing propagation delays in deep feedforward nets provided with a static input . We treat the net as massively parallel such that all units across all layers are updated simultaneously and iteratively . We focus on the ResNet architecture [ 14 ] and we introduce a propagation delay into each residual block ( Figure 1a ) . Because the skip connection permits faster transmission of more primitive perceptual representations , the functional depth of the resulting architecture increases over internal-processing time , yielding a trade o between processing speed and complexity of processing . Consequently , the architecture o ers a natural , integral mechanism for making predictions at any point in processing , known as anytime prediction [ 58 ] . Speed-accuracy trade o s are a fundamental characteristic of human information processing [ 22 , 42 ] and human perception has been modeled with deep learning anytime prediction methods [ 29 ] . Although we focus on the ResNet , our approach can be incorporated into any model with skip connections ( e.g. , Highway Nets [ 48 ] , DenseNet [ 19 ] , U-Net [ 43 ] , Transformers [ 52 ] ) . The contrast between a serial , one-layer-at-a-time model and a cascaded , parallel-update model is illustrated in Figures 1b and 1c , respectively . To step through the operation of the cascaded model , at time 1 , only the first residual block has received meaningful input , and the model prediction is therefore based only on this block ’ s computation . At time 2 , all higher residual blocks have received input from block 1 , and the output is therefore based on all blocks ’ computations , though blocks 2 and above have deficient input . At each subsequent time , all blocks are receiving meaningful input , but it is not until time t that block t has reached its asymptotic output because its input does not stabilize until t ≠ 1 . In essence , the cascaded model behaves like a WideResNet [ 56 ] on the first steps and then becomes a deep ResNet . Our work makes the following key contributions . • We demonstrate the superiority of the cascaded architecture to the serial ( Figures 1b , c ) , indicating that parallelism can be exploited in a way that has not previously been studied . • We propose and evaluate a novel training objective aimed at improving the predictions of anytime models . This temporal-di erence ( TD ) loss [ 49 ] encourages the most accurate response as quickly as possible . TD training improves the performance of both cascaded and serial architectures . Although a rich literature exists aimed at reducing the number of computational steps required to obtain an accurate answer [ 2 , 3 , 4 , 5 , 11 , 15 , 16 , 17 , 18 , 20 , 24 , 31 , 31 , 38 , 40 , 45 , 51 , 54 , 57 ] , all of this work uses a degenerate form of TD for training and our results suggest that these models can be improved using TD . • The cascaded model trained with TD ( CascadedTD ) tends to respond most rapidly to prototypical exemplars , whereas training with the standard cross-entropy loss ( CascadedCE ) does not ( Figure 2 ) . We assess with three quantitative prototypicality measures , and we further show that CascadedTD rapidly converges on the correct semantic family , whereas CascadedCE does not . These facts indicate that CascadedTD organizes knowledge di erently across layers than does CascadedCE . 2Like much other research in deep learning [ 25 , 8 ] , biology informs our work by providing novel forms of inductive bias . Our goal is to investigate computational consequences of these biases , not to model biological phenomena per se . • We show that CascadedTD obtains a strictly superior speed-accuracy profile compared to previously proposed anytime prediction models , which are all based on a serial architecture . • We demonstrate other virtues of CascadedTD : it is more robust to input noise , and its time-varying output trace provides useful signals for meta-cognitive processes—separately trained nets that make judgments about the cascaded architecture ’ s accuracy . Related Work . Prior research on cascaded models . From a psychological perspective , McClelland [ 36 ] characterizes human mental computation in terms of a hierarchy of leaky integrators that continually transmit partial information as it becomes available . We are aware of no work in deep learning on static image processing with cascaded models , but there exist two investigations focused on video sequence processing , where the model state from the previous frame is used to e ciently process the next . Fischer et al . [ 11 ] present a streaming rollout framework for recurrent nets and they very briefly explore the temporal dynamics of cascaded models , showing benefits to early predictions . They present a general taxonomy that includes our proposed feedforward cascaded model , but their focus is almost entirely on formal definitions and a framework that lays out the space of all well-formed roll out patterns ( update orders ) . In contrast , our focus is almost entirely on training procedures that leverage the dynamics of cascaded models , on early read-out mechanisms , and on the computational consequences of these training and read-out mechanisms . Kugele et al . [ 28 ] focus on spiking neural net dynamics , on time-varying inputs , and on reductions in latency that are obtained as a sequence unfolds due to autocorrelations in the input sequence . Notably , Kugele et al . explore a variety of heuristic training losses and they settle on TD ( 1 ) as their preferred loss , but they do not explore the rest of the TD ( ⁄ ) family . Our work is complementary . Carreira et al . [ 5 ] present a causal video understanding model that performs depth-parallel computation with the objective of improving video processing e ciency via maximizing throughput , minimizing latency , and reducing clock cycles . Recurrent nets for vision . Recurrent nets have been used in vision [ e.g. , 20 , 23 , 27 , 25 , 32 , 47 , 46 ] , which adds a dimension of internal processing time for every external input ( see also [ 13 ] ) . However , these models perform serial layerwise updates and therefore fundamentally di er in their operation from cascaded models . Anytime prediction . Anytime prediction models [ 2 , 9 , 15 , 16 , 17 , 18 , 20 , 24 , 30 , 31 , 34 , 38 , 40 , 45 , 51 , 53 , 54 , 57 ] assume serial operation of layers , but allow for predictions to be made from intermediate layers of the architecture . In the simplest case , after t steps , t layers have been activated , and at each step , a prediction is made from the last activated layer [ e.g. , 16 , 24 ] . Figure 1b illustrates a serial model that performs anytime prediction . Some of these models have intrinsic stopping criteria [ e.g. , 6 ] ; others rely on selection of a stopping confidence threshold [ e.g. , 24 , 51 , 54 ] . Temporal-di erence learning . TD learning has a rich history , mostly in the RL community for value function estimation . TD can be used for supervised learning as well . In fact , the two previous works in deep learning using cascaded models [ 5 , 11 ] perform a boundary case of supervised TD , TD ( 1 ) , which we show to have inferior performance . A variety of non-cascaded models , both recurrent [ 20 , 31 , 38 , 57 ] and feedforward [ 2 , 3 , 4 , 15 , 16 , 17 , 18 , 24 , 31 , 40 , 45 , 51 , 54 ] , aim to reduce the number of computational steps required to obtain an accurate output . All use TD ( 1 ) for training . No previous research has explored the general formulation of TD for improving anytime prediction . Deep Cascaded Networks . Many modern deep architectures—including ResNet [ 14 ] , Highway Nets [ 48 ] , DenseNet [ 19 ] , U-Net [ 43 ] —incorporate skip connections that bypass strictly layered feedforward connectivity , analogous to the architecture of visual cortex [ 10 ] . Under the biological assumption that signals transmitted through a neural layer are delayed relative to signals that bypass the layer , we construct a cascaded model using ResNets by introducing a novel computational component that delays the transmission of signals from the output of each computational layer , denoted in Figure 1a . Because these delays extend processing in time , the hidden states require a time index . The input to ResNet block i at time t is denoted zt , i . The block transforms this input via the residual transform , yielding zÕt , i = F ( zt , i ) . We conceive of as a tapped delay-line memory of the transform history , Z Õt , i = [ zÕt , i zÕt≠1 , i . . . zÕ1 , i ] , which is convolved with a temporal kernel to produce the block output zt , i+1 = ReLU ! zt , i + Z Õt , i `` . ( 1 ) The kernel = [ 1 0 0 . . . 0 ] recovers the standard ResNet in which communication between layers is instantaneous . We consider two kernels to introduce time delays . With = [ 0 1 0 0 . . . 0 ] , a discrete one-step delay is introduced ( OSD for short ) . With = ( 1 ≠ – ) [ 1 – –2 –3 . . . ] , we obtain exponentially weighted smoothing ( EWS for short ) , where larger – œ [ 0 , 1 ) yield slower transmission times . Note that both of these special kernels have e cient implementations : the OSD kernel with a one-element queue and the EWS kernel with a finite ( one-step ) state vector and the incremental update , Z Õt , i = –Z Õt≠1 , i + ( 1 ≠ – ) zÕt , i . We use the OSD kernel for training all models . Modifications of batch norm were required to do time-step-conditional normalization ( see Appendix A ) . All experiments use a ResNet-18 , which has 8 residual blocks and hence 8 time delays . Note , we conducted experiments with larger ResNets and the additional compute did not a ect qualitative properties . We also add a time delay to the output of the model ’ s first convolutional layer . Consequently , with the OSD kernel , the cascaded model requires 9 updates for the output to reach asymptote . The cascaded and serial models with the same weights will necessarily produce identical asymptotic outputs . To obtain a finer temporal granularity at evaluation , some simulations switch to the EWS kernel with – = 0.9 . Temporal dynamics are qualitatively similar for OSD and EWS , but EWS allows us to better distinguish individual examples in terms of their fine-grain timing . EWS with – = 0.9 requires about 70 steps for the output to asymptote . We note that our findings are robust to the choice of – , as long as – slows transmission .

Summary: The authors propose to tighten the link with biological neural networks by designing a cascaded network that is more compatible with the dynamics of information processing observed in the brain. In cascaded networks, all layers process the information in parallel and are fed with an updated input at each time step. The authors also introduce a loss, the Temporal Difference, that improve the performance of cascaded networks. The authors demonstrate that the cascaded network offers better trade-off accuracy and better noise robustness than serial one. In addition, the authors demonstrate that class instance that are rapidly recognized tend to be more prototypical in cascaded network