Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss

Since the earliest investigations of artificial neural nets , their design has been informed by biological neural nets [ 37 ] . Perhaps the most compelling example is the convolutional net for machine vision , which has adopted properties of primate cortical neuroanatomy including a hierarchical layered organization , local receptive fields , and spatial equivariance [ 12 ] . In this article , we investigate computational consequences of two fundamental properties of biological information processing systems that have not been considered in the design of deep neural nets . First , the brain consists of massively parallel , dedicated hardware with neurons throughout the cortex updating continuously and simultaneously . Second , information transmission between neurons introduces time delays [ 1 ] . As a result , unrefined and possibly úCurrently at Apple . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . incomplete neural state in one region is transmitted to the next region even as the state evolves ; and feedforward connectivity yields a speed-accuracy trade o in which the initial response to a static input occurs rapidly but can be inaccurate , with the output gradually improving over internal processing time . Following McClelland [ 36 ] , we refer to such an architecture as cascaded . Cascaded dynamics contrast sharply with the dynamics of standard feedforward nets , which operate in serial stages , each layer completing its computation before subsequent layers begin processing . Cascaded dynamics are also quite di erent than the dynamics of vision models with recurrent connections [ e.g. , 23 , 25 , 38 , 47 ] , which , given a static input , may iteratively update , but layer updates are still computed serially with each layer completing its computation and then feeding it immediately to the next layer ( or back to itself ) . Fundamentally , our investigation asks : Supposing we take a step toward biological realism with massively parallel hardware and relatively slow inter-neuron communication , what are the computational benefits and consequences ? 2 We construct cascaded networks by introducing propagation delays in deep feedforward nets provided with a static input . We treat the net as massively parallel such that all units across all layers are updated simultaneously and iteratively . We focus on the ResNet architecture [ 14 ] and we introduce a propagation delay into each residual block ( Figure 1a ) . Because the skip connection permits faster transmission of more primitive perceptual representations , the functional depth of the resulting architecture increases over internal-processing time , yielding a trade o between processing speed and complexity of processing . Consequently , the architecture o ers a natural , integral mechanism for making predictions at any point in processing , known as anytime prediction [ 58 ] . Speed-accuracy trade o s are a fundamental characteristic of human information processing [ 22 , 42 ] and human perception has been modeled with deep learning anytime prediction methods [ 29 ] . Although we focus on the ResNet , our approach can be incorporated into any model with skip connections ( e.g. , Highway Nets [ 48 ] , DenseNet [ 19 ] , U-Net [ 43 ] , Transformers [ 52 ] ) . The contrast between a serial , one-layer-at-a-time model and a cascaded , parallel-update model is illustrated in Figures 1b and 1c , respectively . To step through the operation of the cascaded model , at time 1 , only the first residual block has received meaningful input , and the model prediction is therefore based only on this block ’ s computation . At time 2 , all higher residual blocks have received input from block 1 , and the output is therefore based on all blocks ’ computations , though blocks 2 and above have deficient input . At each subsequent time , all blocks are receiving meaningful input , but it is not until time t that block t has reached its asymptotic output because its input does not stabilize until t ≠ 1 . In essence , the cascaded model behaves like a WideResNet [ 56 ] on the first steps and then becomes a deep ResNet . Our work makes the following key contributions . • We demonstrate the superiority of the cascaded architecture to the serial ( Figures 1b , c ) , indicating that parallelism can be exploited in a way that has not previously been studied . • We propose and evaluate a novel training objective aimed at improving the predictions of anytime models . This temporal-di erence ( TD ) loss [ 49 ] encourages the most accurate response as quickly as possible . TD training improves the performance of both cascaded and serial architectures . Although a rich literature exists aimed at reducing the number of computational steps required to obtain an accurate answer [ 2 , 3 , 4 , 5 , 11 , 15 , 16 , 17 , 18 , 20 , 24 , 31 , 31 , 38 , 40 , 45 , 51 , 54 , 57 ] , all of this work uses a degenerate form of TD for training and our results suggest that these models can be improved using TD . • The cascaded model trained with TD ( CascadedTD ) tends to respond most rapidly to prototypical exemplars , whereas training with the standard cross-entropy loss ( CascadedCE ) does not ( Figure 2 ) . We assess with three quantitative prototypicality measures , and we further show that CascadedTD rapidly converges on the correct semantic family , whereas CascadedCE does not . These facts indicate that CascadedTD organizes knowledge di erently across layers than does CascadedCE . 2Like much other research in deep learning [ 25 , 8 ] , biology informs our work by providing novel forms of inductive bias . Our goal is to investigate computational consequences of these biases , not to model biological phenomena per se . • We show that CascadedTD obtains a strictly superior speed-accuracy profile compared to previously proposed anytime prediction models , which are all based on a serial architecture . • We demonstrate other virtues of CascadedTD : it is more robust to input noise , and its time-varying output trace provides useful signals for meta-cognitive processes—separately trained nets that make judgments about the cascaded architecture ’ s accuracy . Related Work . Prior research on cascaded models . From a psychological perspective , McClelland [ 36 ] characterizes human mental computation in terms of a hierarchy of leaky integrators that continually transmit partial information as it becomes available . We are aware of no work in deep learning on static image processing with cascaded models , but there exist two investigations focused on video sequence processing , where the model state from the previous frame is used to e ciently process the next . Fischer et al . [ 11 ] present a streaming rollout framework for recurrent nets and they very briefly explore the temporal dynamics of cascaded models , showing benefits to early predictions . They present a general taxonomy that includes our proposed feedforward cascaded model , but their focus is almost entirely on formal definitions and a framework that lays out the space of all well-formed roll out patterns ( update orders ) . In contrast , our focus is almost entirely on training procedures that leverage the dynamics of cascaded models , on early read-out mechanisms , and on the computational consequences of these training and read-out mechanisms . Kugele et al . [ 28 ] focus on spiking neural net dynamics , on time-varying inputs , and on reductions in latency that are obtained as a sequence unfolds due to autocorrelations in the input sequence . Notably , Kugele et al . explore a variety of heuristic training losses and they settle on TD ( 1 ) as their preferred loss , but they do not explore the rest of the TD ( ⁄ ) family . Our work is complementary . Carreira et al . [ 5 ] present a causal video understanding model that performs depth-parallel computation with the objective of improving video processing e ciency via maximizing throughput , minimizing latency , and reducing clock cycles . Recurrent nets for vision . Recurrent nets have been used in vision [ e.g. , 20 , 23 , 27 , 25 , 32 , 47 , 46 ] , which adds a dimension of internal processing time for every external input ( see also [ 13 ] ) . However , these models perform serial layerwise updates and therefore fundamentally di er in their operation from cascaded models . Anytime prediction . Anytime prediction models [ 2 , 9 , 15 , 16 , 17 , 18 , 20 , 24 , 30 , 31 , 34 , 38 , 40 , 45 , 51 , 53 , 54 , 57 ] assume serial operation of layers , but allow for predictions to be made from intermediate layers of the architecture . In the simplest case , after t steps , t layers have been activated , and at each step , a prediction is made from the last activated layer [ e.g. , 16 , 24 ] . Figure 1b illustrates a serial model that performs anytime prediction . Some of these models have intrinsic stopping criteria [ e.g. , 6 ] ; others rely on selection of a stopping confidence threshold [ e.g. , 24 , 51 , 54 ] . Temporal-di erence learning . TD learning has a rich history , mostly in the RL community for value function estimation . TD can be used for supervised learning as well . In fact , the two previous works in deep learning using cascaded models [ 5 , 11 ] perform a boundary case of supervised TD , TD ( 1 ) , which we show to have inferior performance . A variety of non-cascaded models , both recurrent [ 20 , 31 , 38 , 57 ] and feedforward [ 2 , 3 , 4 , 15 , 16 , 17 , 18 , 24 , 31 , 40 , 45 , 51 , 54 ] , aim to reduce the number of computational steps required to obtain an accurate output . All use TD ( 1 ) for training . No previous research has explored the general formulation of TD for improving anytime prediction . Deep Cascaded Networks . Many modern deep architectures—including ResNet [ 14 ] , Highway Nets [ 48 ] , DenseNet [ 19 ] , U-Net [ 43 ] —incorporate skip connections that bypass strictly layered feedforward connectivity , analogous to the architecture of visual cortex [ 10 ] . Under the biological assumption that signals transmitted through a neural layer are delayed relative to signals that bypass the layer , we construct a cascaded model using ResNets by introducing a novel computational component that delays the transmission of signals from the output of each computational layer , denoted in Figure 1a . Because these delays extend processing in time , the hidden states require a time index . The input to ResNet block i at time t is denoted zt , i . The block transforms this input via the residual transform , yielding zÕt , i = F ( zt , i ) . We conceive of as a tapped delay-line memory of the transform history , Z Õt , i = [ zÕt , i zÕt≠1 , i . . . zÕ1 , i ] , which is convolved with a temporal kernel to produce the block output zt , i+1 = ReLU ! zt , i + Z Õt , i `` . ( 1 ) The kernel = [ 1 0 0 . . . 0 ] recovers the standard ResNet in which communication between layers is instantaneous . We consider two kernels to introduce time delays . With = [ 0 1 0 0 . . . 0 ] , a discrete one-step delay is introduced ( OSD for short ) . With = ( 1 ≠ – ) [ 1 – –2 –3 . . . ] , we obtain exponentially weighted smoothing ( EWS for short ) , where larger – œ [ 0 , 1 ) yield slower transmission times . Note that both of these special kernels have e cient implementations : the OSD kernel with a one-element queue and the EWS kernel with a finite ( one-step ) state vector and the incremental update , Z Õt , i = –Z Õt≠1 , i + ( 1 ≠ – ) zÕt , i . We use the OSD kernel for training all models . Modifications of batch norm were required to do time-step-conditional normalization ( see Appendix A ) . All experiments use a ResNet-18 , which has 8 residual blocks and hence 8 time delays . Note , we conducted experiments with larger ResNets and the additional compute did not a ect qualitative properties . We also add a time delay to the output of the model ’ s first convolutional layer . Consequently , with the OSD kernel , the cascaded model requires 9 updates for the output to reach asymptote . The cascaded and serial models with the same weights will necessarily produce identical asymptotic outputs . To obtain a finer temporal granularity at evaluation , some simulations switch to the EWS kernel with – = 0.9 . Temporal dynamics are qualitatively similar for OSD and EWS , but EWS allows us to better distinguish individual examples in terms of their fine-grain timing . EWS with – = 0.9 requires about 70 steps for the output to asymptote . We note that our findings are robust to the choice of – , as long as – slows transmission .

Summary: This paper proposes to use skip connections in feedforward artificial neural networks to integrate information over multiple time steps by defining a delay for each connection in the network. Extensive experiments with a ResNet18 architecture show that this execution plan leads to an increase in early prediction accuracy when using anytime predictors, to faster predictions when samples are closer to a class prototype, more robustness to noise and a better out-of-distribution (OOD) detection compared to a sequential architecture. Temporal difference learning (TD) is utilized to train the neural network and an analysis is done on CIFAR-10, CIFAR-100 and TinyImageNet to show that the often used value of lambda=1 is suboptimal for these kinds of tasks.