Package: continual-inference
A Python library for Continual Inference Networks in PyTorchQuick-start•Docs•Principles•Paper•Examples•Modules•Model Zoo•Contribute•LicenseContinual Inference Networks ensure efficient stream processingMany of our favorite Deep Neural Network architectures (e.g.,CNNsandTransformers) were built with offline-processing for offline processing. Rather than processing inputs one sequence element at a time, they require the whole (spatio-)temporal sequence to be passed as a single input.
Yet,many important real-life applications need online predictions on a continual input stream.
While CNNs and Transformers can be applied by re-assembling and passing sequences within a sliding window, this isinefficientdue to the redundant intermediary computations from overlapping clips.Continual Inference Networks(CINs) are built to ensure efficient stream processing by employing an alternative computational ordering, which allows sequential computations without the use of sliding window processing.
In general, CINs requires approx.L×  fewer FLOPs per prediction compared to sliding window-based inference with non-CINs, whereLis the corresponding sequence length of a non-CIN network. For more details, check out the videos below describing Continual 3D CNNs [1] and Transformers [2].News2022-12-02: ONNX compatibility for all modules is available from v1.0.0. Seetest_onnx.pyfor examples.Quick-startInstallpipinstallcontinual-inferenceExamplecomodules are weight-compatible drop-in replacement fortorch.nn, enhanced with the capability of efficientcontinual inference:importtorchimportcontinualasco#                      B, C, T, H, Wexample=torch.randn((1,1,5,3,3))conv=co.Conv3d(in_channels=1,out_channels=1,kernel_size=(3,3,3))# Same exact computation as torch.nn.Conv3d ✅output=conv(example)# But can also perform online inference efficiently 🚀firsts=conv.forward_steps(example[:,:,:4])last=conv.forward_step(example[:,:,4])asserttorch.allclose(output[:,:,:conv.delay],firsts)asserttorch.allclose(output[:,:,conv.delay],last)# Temporal propertiesassertconv.receptive_field==3assertconv.delay==2See thenetwork compositionandmodel zoosections for additional examples.Library principlesForward modesThe library components feature three distinct forward modes, which are handy for different situations, namelyforward,forward_step, andforward_steps:forward(input)Performs a forward computation over multiple time-steps. This function is identical to the corresponding module intorch.nn, ensuring cross-compatibility. Moreover, it's handy for efficient training on clip-based data.O            (O: output)
         ↑ 
         N            (N: network module)
         ↑ 
 -----------------    (-: aggregation)
 P   I   I   I   P    (I: input frame, P: padding)forward_step(input, update_state=True)Performs a forward computation for a single frame and (optionally) updates internal states accordingly. This function performs efficient continual inference.O+S O+S O+S O+S   (O: output, S: updated internal state)
 ↑   ↑   ↑   ↑ 
 N   N   N   N    (N: network module)
 ↑   ↑   ↑   ↑ 
 I   I   I   I    (I: input frame)forward_steps(input, pad_end=False, update_state=True)Performs a forward computation across multiple time-steps while updating internal states for continual inference (if update_state=True).
Start-padding is always accounted for, but end-padding is omitted per default in expectance of the next input step. It can be added by specifying pad_end=True. If so, the output-input mapping the exact same as that of forward.O            (O: output)
         ↑ 
 -----------------    (-: aggregation)
 O  O+S O+S O+S  O    (O: output, S: updated internal state)
 ↑   ↑   ↑   ↑   ↑
 N   N   N   N   N    (N: network module)
 ↑   ↑   ↑   ↑   ↑
 P   I   I   I   P    (I: input frame, P: padding)__call__Per default, the__call__function operates identically totorch.nnand executes forward. We supply two options for changing this behavior, namely thecall_modeproperty and thecall_modecontext manager. An example of their use follows:timeseries=torch.randn(batch,channel,time)timestep=timeseries[:,:,0]net(timeseries)# Invokes net.forward(timeseries)# Assign permanent call_mode propertynet.call_mode="forward_step"net(timestep)# Invokes net.forward_step(timestep)# Assign temporary call_mode with context managerwithco.call_mode("forward_steps"):net(timeseries)# Invokes net.forward_steps(timeseries)net(timestep)# Invokes net.forward_step(timestep) againCompositionContinual Inference Networks require strict handling of internal data delays to guarantee correspondence betweenforward modes. While it is possible to compose neural networks by definingforward,forward_step, andforward_stepsmanually, correct handling of delays is cumbersome and time-consuming. Instead, we provide a rich interface of container modules, which handles delays automatically. On top ofco.Sequential(which is a drop-in replacement oftorch.nn.Sequential), we provide modules for handling parallel and conditional dataflow.co.Sequential: Invoke modules sequentially, passing the output of one module onto the next.co.Broadcast: Broadcast one stream to multiple.co.Parallel: Invoke modules in parallel given each their input.co.ParallelDispatch: Dispatch multiple input streams to multiple output streams flexibly.co.Reduce: Reduce multiple input streams to one.co.BroadcastReduce: Shorthand for Sequential(Broadcast, Parallel, Reduce).co.Residual: Residual connection.co.Conditional: Conditionally checks whether to invoke a module (or another) at runtime.Composition examples:Residual moduleShort-hand:residual=co.Residual(co.Conv3d(32,32,kernel_size=3,padding=1))Explicit:residual=co.Sequential(co.Broadcast(2),co.Parallel(co.Conv3d(32,32,kernel_size=3,padding=1),co.Delay(2),),co.Reduce("sum"),)3D MobileNetV2 Inverted residual blockContinual 3D version of theMobileNetV2 Inverted residual block.MobileNetV2 Inverted residual block. Source: https://arxiv.org/pdf/1801.04381.pdfmb_conv=co.Residual(co.Sequential(co.Conv3d(32,64,kernel_size=(1,1,1)),nn.BatchNorm3d(64),nn.ReLU6(),co.Conv3d(64,64,kernel_size=(3,3,3),padding=(1,1,1),groups=64),nn.ReLU6(),co.Conv3d(64,32,kernel_size=(1,1,1)),nn.BatchNorm3d(32),))3D Squeeze-and-Excitation moduleContinual 3D version of theSqueeze-and-Excitation moduleSqueeze-and-Excitation block. 
  Scale refers to a broadcasted element-wise multiplication.
  Adapted from: https://arxiv.org/pdf/1709.01507.pdfse=co.Residual(co.Sequential(OrderedDict([("pool",co.AdaptiveAvgPool3d((1,1,1),kernel_size=7)),("down",co.Conv3d(256,16,kernel_size=1)),("act1",nn.ReLU()),("up",co.Conv3d(16,256,kernel_size=1)),("act2",nn.Sigmoid()),])),reduce="mul",)3D Inception moduleContinual 3D version of theInception module:Inception module. Source: https://arxiv.org/pdf/1409.4842v1.pdfdefnorm_relu(module,channels):returnco.Sequential(module,nn.BatchNorm3d(channels),nn.ReLU(),)inception_module=co.BroadcastReduce(co.Conv3d(192,64,kernel_size=1),co.Sequential(norm_relu(co.Conv3d(192,96,kernel_size=1),96),norm_relu(co.Conv3d(96,128,kernel_size=3,padding=1),128),),co.Sequential(norm_relu(co.Conv3d(192,16,kernel_size=1),16),norm_relu(co.Conv3d(16,32,kernel_size=5,padding=2),32),),co.Sequential(co.MaxPool3d(kernel_size=(1,3,3),padding=(0,1,1),stride=1),norm_relu(co.Conv3d(192,32,kernel_size=1),32),),reduce="concat",)Input shapesWe enforce a unified ordering of input dimensions for all library modules, namely:(batch, channel, time, optional_dim2, optional_dim3)OutputsThe outputs produces byforward_stepandforward_stepsare identical to those offorward, provided the same data was input beforehand and state update was enabled. We know that input and output shapes aren't necessarily the same when usingforwardin the PyTorch library, and  generally depends on padding, stride and receptive field of a module.For theforward_stepfunction, this comes to show by someNone-valued outputs. Specifically, modules with adelay(i.e. with receptive fields larger than the padding + 1) will produceNoneuntil the input count exceeds the delay. Moreover,stride> 1 will produceTensoroutputs everystridesteps andNonethe remaining steps. A visual example is shown below:A mixed example of delay and outputs under padding and stride. Here, we illustrate the step-wise operation of two co module layers, l1 with with receptive_field = 3, padding = 2, and stride = 2 and l2 with receptive_field = 3, no padding and stride = 1. ⧇ denotes a padded zero, ■ is a non-zero step-feature, and ☒ is an empty output.For more information, please see thelibrary paper.Handling stateDuring stream processing, network modules which operate over multiple time-steps, e.g., a convolution withkernel_size > 1in the temporal dimension, will aggregate and cache state internally. Each module has its own local state, which can be inspected usingmodule.get_state(). Duringforward_stepandforward_steps, the state is updated unless the forward_step(s) is invoked with anupdate_state = Falseargument.Astate cleanupcan be accomplished viamodule.clean_state().Module libraryContinual Inferencefeatures a rich collection of modules for defining Continual Inference Networks. Specific care was taken to create CIN versions of the PyTorch modules found intorch.nn:Convolutionsco.Conv1dco.Conv2dco.Conv3dPoolingco.AvgPool1dco.AvgPool2dco.AvgPool3dco.MaxPool1dco.MaxPool2dco.MaxPool3dco.AdaptiveAvgPool2dco.AdaptiveAvgPool3dco.AdaptiveMaxPool2dco.AdaptiveMaxPool3dLinearco.Linearco.Identity: Maps input to output without modification.co.Add: Adds a constant value.co.Multiply: Multiplies with a constant factor.Recurrentco.RNNco.LSTMco.GRUTransformersco.TransformerEncoderco.TransformerEncoderLayerFactory: Factory function corresponding tonn.TransformerEncoderLayer.co.SingleOutputTransformerEncoderLayer: SingleOutputMHA version ofnn.TransformerEncoderLayer.co.RetroactiveTransformerEncoderLayer: RetroactiveMHA version ofnn.TransformerEncoderLayer.co.RetroactiveMultiheadAttention: Retroactive version ofnn.MultiheadAttention.co.SingleOutputMultiheadAttention: Single-output version ofnn.MultiheadAttention.co.RecyclingPositionalEncoding: Positional Encoding used for Continual Transformers.Modules for composing and converting networks. Bothcompositionandutilitymodules can be used for regular definition of PyTorch modules as well.Composition modulesco.Sequential: Invoke modules sequentially, passing the output of one module onto the next.co.Broadcast: Broadcast one stream to multiple.co.Parallel: Invoke modules in parallel given each their input.co.ParallelDispatch: Dispatch multiple input streams to multiple output streams flexibly.co.Reduce: Reduce multiple input streams to one.co.BroadcastReduce: Shorthand for Sequential(Broadcast, Parallel, Reduce).co.Residual: Residual connection.co.Conditional: Conditionally checks whether to invoke a module (or another) at runtime.Utility modulesco.Delay: Pure delay module (e.g. needed in residuals).co.Skip: Skip a predefined number of input steps.co.Reshape: Reshape non-temporal dimensions.co.Lambda: Lambda module which wraps any function.co.Constant: Maps input to and output with constant value.co.Zero: Maps input to output of zeros.co.One: Maps input to output of ones.Convertersco.continual: conversion function fromtorch.nnmodules tocomodules.co.forward_stepping: functional wrapper, which enhances temporally localtorch.nnmodules with the forward_stepping functions.We support drop-in interoperability with with the followingtorch.nnmodules:Activationnn.Thresholdnn.ReLUnn.RReLUnn.Hardtanhnn.ReLU6nn.Sigmoidnn.Hardsigmoidnn.Tanhnn.SiLUnn.Hardswishnn.ELUnn.CELUnn.SELUnn.GLUnn.GELUnn.Hardshrinknn.LeakyReLUnn.LogSigmoidnn.Softplusnn.Softshrinknn.PReLUnn.Softsignnn.Tanhshrinknn.Softminnn.Softmaxnn.Softmax2dnn.LogSoftmaxNormalizationnn.BatchNorm1dnn.BatchNorm2dnn.BatchNorm3dnn.GroupNorm,nn.InstanceNorm1d(affine=True, track_running_stats=True required)nn.InstanceNorm2d(affine=True, track_running_stats=True required)nn.InstanceNorm3d(affine=True, track_running_stats=True required)nn.LayerNorm(only non-temporal dimensions must be specified)Dropoutnn.Dropoutnn.Dropout1dnn.Dropout2dnn.Dropout3dnn.AlphaDropoutnn.FeatureAlphaDropoutModel Zoo and BenchmarksContinual 3D CNNsBenchmark results for 1-view testing onKinetics400. For reference,X3D-Lscores 69.3% top-1 acc with 19.2 GFLOPs per prediction.ArchAvg. pool sizeTop 1 (%)FLOPs (G) per stepFLOPs reductionParams (M)CodeWeightsCoX3D-L6471.61.2515.3x6.2linklinkCoX3D-M6471.00.3315.1x3.8linklinkCoX3D-S6464.70.1712.1x3.8linklinkCoSlow6473.16.908.0x32.5linklinkCoI3D6464.05.685.0x28.0linklinkFLOPs reduction is noted relative to non-continual inference.
Note thaton-hardware inferencedoesn't reach the same speedups as "FLOPs reductions" might suggest due to overhead of state reads and writes. This overhead is less important for large batch sizes. This applies to all models in the model zoo.Continual ST-GCNsBenchmark results for onNTU RGB+D 60for the joint modality. For reference,ST-GCNachieves 86% X-Sub and 93.4 X-View accuracy with 16.73 GFLOPs per prediction.ArchReceptive fieldX-Sub Acc (%)X-View Acc (%)FLOPs (G) per stepFLOPs reductionParams (M)CodeCoST-GCN30086.393.80.16107.7x3.1linkCoA-GCN30084.192.60.17108.7x3.5linkCoST-GCN30086.392.40.15107.6x3.1linkHere, you can download pre-trained,model weights for the above architectures on NTU RGB+D 60, NTU RGB+D 120, and Kinetics-400 on joint and bone modalities.Continual TransformersBenchmark results for onTHUMOS14on top of features extracted using a TSN-ResNet50 backbone pre-trained on Kinetics400. For reference,OadTRachieves 64.4 % mAP with 2.5 GFLOPs per prediction.ArchReceptive fieldmAP (%)FLOPs (G) per stepParams (M)CodeCoOadTR-b16464.20.4115.9linkCoOadTR-b26464.40.019.6linkThe library features complete implementations of theone- andtwo-block continual transformer encoders as well.CompatibilityThe library modules are built to integrate seamlessly with other PyTorch projects.
Specifically, extra care was taken to ensure out-of-the-box compatibility with:pytorch-lightningptflopsrideonnxCitation@inproceedings{hedegaard2022colib,title={Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch},author={Lukas Hedegaard and Alexandros Iosifidis},booktitle={European Conference on Computer Vision Workshops (ECCVW)},year={2022}}AcknowledgementThis work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871449 (OpenDR).