DynCNN: An Effective Dynamic Architecture on Convolutional Neural Network for Surveillance Videos

1 INTRODUCTION . Nowadays , with the development of deep learning technologies , large-scale surveillance video analysis for intelligent city draws more and more attention in the real world applications , for instance , in person re-identification ( Ahmed et al . ( 2015 ) ) , vehicle re-identification ( Liu et al . ( 2016 ) ) ( Shen et al . ( 2017 ) ) , pedestrian detection ( Tian et al . ( 2014 ) ) and crowd segmentation ( Kang & Wang ( 2014 ) ) , etc . However , these deep learning methods are extremely computationally expensive—state-of-theart methods for object detection performed on a state-of-the-art NVIDIA P100 GPU run at 10-80 frames per second . Although processing on one video can be done by utilizing one to several topperforming hardware equipments , it still a challenge for large-scale actual deployment—it would cost over 5 billion USD in hardware to analyze over 4 million CCTVs in real time in the UK alone when considering these computational overheads in context ( News ( 2015 ) ) . Those real world applications for intelligent city employed the state-of-the-art network architectures from the ILSVRC competition , e.g . VGG ( Simonyan & Zisserman ( 2014 ) ) , GoogLeNet ( Szegedy et al . ( 2014 ) ) , and ResNet ( He et al . ( 2015 ) ) as their feature extraction architecture . However , these powerful networks tend to be resource-hungry models with high computational demands at inference time to win the competition—processing an one-minute surveillance video through VGG16 network costs 23 TFLOPs . To make the feature extraction architectures more effective in terms of surveillance video , this paper aims to explore ways to reduce the calculation on feature extraction . Figure 1 shows two frame clips of a surveillance video with its corresponding feature map through 3×3 kernel . It is observed that most of the scene between two adjacent frames in the surveillance video are almost the same with merely about 2 % to 15 % difference . Due to the linear characteristic of convolution , the derived feature maps preserve the similarity of input video frames—it can reduce significant amount of calculation if the previous feature map can be reused for the current feature map . Suppose that the frame size is 512×512 while the amount of the changing pixels in feature map is 52k , the calculation can be reduced from 7 MFLOPs to 1.4 MFLOPs , that is 5 times saving ratio . In this paper , we exploit the key characteristic of surveillance video—high scene similarity—and propose a new architecture dynamic convolution which can be directly applied to off-the-shelf model without retraining the weights . Dynamic convolution can inspect the inter-frame variation and thus predict the changing pixels in each corresponding feature map . Accordingly , convolution performs only on those predicted parts to achieve less computation while not affecting the accuracy . Finally , we apply dynamic convolution to one of the state-of-the-art object detector Single Shot MultiBox Detection ( Liu et al . ( 2015 ) ) and test on several surveillance video datasets including overall 45 videos . In addition , we conduct a further analysis in the FLOP computation and execution time of each convolutional layer to improve the understanding of dynamic convolution in practice . 2 PRIOR WORKS . For the last several years , tuning deep learning neural architectures to strike an optimal balance between precision and processing time has been an area of active research . Many methods have been proposed to reduce storage space and computing resources and also made great progress from theoretical research to platform implementation . Basically , they can be divided into three major techniques , effective model design , pruning , and quantization . Effective model design MobileNets ( Howard et al . ( 2017 ) ) is a lightweight network architecture proposed by Google for mobile deployment . The main idea is to disassemble the original convolution calculation into two parts : depthwise convolutions and pointwise convolutions . Although the calculation amount on convolution can greatly be reduced , the precision would be decreased . In addition , the feature map accordingly increases . The increasing data transfer followed by feature map makes MobileNets hard to perform on GPU . To improve the precision , ShuffleNet ( Zhang et al . ( 2017 ) ) proposed two new strategies channel shuffle and pointwise group convolutions based on the concept that helps the information flowing across feature channels . On the other hand , Inception ( Szegedy et al . ( 2016 ) ) introduces bottleneck structure to approximate sparse structures into several dense sub-matrices to achieve more efficient use of computing resources . In the ultra-lightweight neural network SqueezeNet ( Iandola et al . ( 2016 ) ) , the 1x1 and 3x3 convolution kernels are extensively used compared with Inception to achieve more compression on calculation amount . Pruning Besides of the model design , pruning is another efficient way to reduce storage space and computing resources by removing redundant parts of model . Li et al . ( 2016 ) proposed a method for cropping filters of convolutional layer . This method determines the importance of filter by the magnitude of the absolute sum of weights on the filter . After that , the model needs to be trained again . Without lossing greately precision , such method can effectively reduce the complexity of the model . However , heavy analysis on each convolution layer costs lots of time and may occur the concern of one-case model . Liu et al . ( 2017 ) used another factor , the scaling coefficient γ in batch normalization , to determine the importance of the filter . Quantization Network weight sharing quantization is also an important type of network compression method which focuses on the weight representation . The nature of this technique is to find the center value which can represent the original weight distribution by clustering method . Deep Compression proposed by Han et al . ( 2015 ) clusters the weights and replaces them by the center value of each cluster . The transformation between the weights and center values is stored in a codebook . Such transformation greatly compresses the weight representation from a 32-bit floating point number to a short bit number . 3 DYNAMIC CONVOLUTION . Dynamic convolution is an architecture which applies to each layer of convolutional neural network . Figure 2 illustrates a convolution neural network with our dynamic convolution , called dynamic convolutional neural network ( DynCNN ) . The work flow of dynamic convolution consists of three main parts : 1 ) frame differencing ; 2 ) prediction ; 3 ) dyn-convolution . Let IDMt denote the input difference map of t-th frame ( Framet ) derived by frame differencing and iDMt denote the inner difference map derived from the previous inner difference map or input difference map by prediction . Dyn-convolution preserves the feature map ( FM ) from the previous layer and selectively do convolution according to the position which the inner difference map indicates in order to update the feature map . 3.1 FRAME DIFFERENCING . The frame differencing method ( Liu & Hou ( 2012 ) ) is often used in the moving object detection and segmentation methods . In this work , we employ it to inspect the inter-frame variation between two adjacent frames . The basic concept of frame differencing is to subtract two frames to calculate the difference of each pixel . However , in most surveillance video sequences there exist speckle noise which severely affects the difference value . Accordingly , the difference map , as referred to input difference map ( IDM ) , is derived by thresholding the result of frame differencing with the following statement : D ( i , j ) = { 1 , |∆I ( i , j ) | < ΘIDM 0 , otherwise ( 1 ) where ΘIDM denotes the threshold of the statement . The pixel at ( i , j ) is denoted as a changed pixel when D is determined as 1 and we called it changing point . ∆I ( i , j ) in ( 1 ) represents the result of frame differencing as the following expression : ∆I ( i , j ) = ICurr ( i , j ) − IPrev ( i , j ) ( 2 ) where ICurr ( i , j ) and IPrev ( i , j ) in ( 2 ) denote the pixel intensity at position ( i , j ) of the current and previous frame respectively . The value of the threshold ΘIDM is an important factor which determines the amount of convolution calculation . Therefore , the effect of the threshold on the speed and accuracy during the inference phase will be well analyzed in Section 4 . 3.2 PREDICTION . In the convolution process , there exists a diffusion effect in the result if the input is changed . Suppose that the kernel size of convolution is 3×3 , each pixel of the result is determined by the corresponding 9 pixels from the input—the pixel at the same position and its 8 peripheral pixels . On the contrary , when a certain pixel in the input is changed , 9 pixels in the output which the pixel involves will be affected simultaneously and are denoted as impacted pixels . Based on the characteristic of diffusion effect in convolution process , the position of impacted pixels in each feature map can be predicted by using dilation operation and recorded on the inner difference map . 3.3 DYN-CONVOLUTION . The iDM records the position of the impacted pixel of the FM and also implicates which pixel value of the FM needs to be updated . To update the value , the pixels of the previous FM which contributes to it will be re-convoluted and are denoted as needed pixels while this process is called dyn-convolution . In this work , we employ the lastest cuDNN library developed by Nvidia to optimize the performance on calculation speed . However , the convolution calculation provided by the library is only allowed for contiguous blocks instead of discrete blocks with given position . Therefore , we create another memory space , denoted as new array , in GPU to store those needed pixels for the requirement of library . 3.4 PRACTICAL CONCERN AND IMPLEMENTATION . In real surveillance video application , although the calculation reduction on convolution is the main concern of speeding up the overall processing time , the data transfer is another important factor which contributes to the time . Dyn-convolution is intuitive but rude and also not efficient in GPU . Accordingly , we apply two strategies to improve the overall processing time , cell-based partition and continuous convolution . Cell-based Partition Compared with pixel-based map iDM , each FM is divided into several block cells . If any impacted pixel exists in the cell , the cell will be noted as “ impacted ” and recorded on cell difference map ( CDM ) . The dyn-convolution thus performs according to the cell difference map rather than the pixel difference map—the new array stores the needed pixels of impacted cells . Figure 3 illustrates the difference between the naive method and the improved method . In this strategy , the data in new array may contains “ non-needed ” pixels when the cell involves nonimpacted pixels . In addition , new array performs unnecessary convolution on the boundary of two needed pixel blocks , where generates don ’ t-care feature value x . Nevertheless , the needed pixels are reused to reduce the amount of data transfer and the characteristic of cuDNN library—convolution costs less processing time when stride is 1—is contemplated to speed up the convolution process , which indicates that less sacrification on computation while greatly improving the processing time . Continuous Convolution Another part which costs processing time is the layer number . It multiplies the data transfer on a new array . We partition whole convolution layer by pooling process and group the convolution layers up , called convolution group . For each convolution group , the convolution structure is modified from “ once padding before once convolution ” to “ continuous padding first then continuous convolution ” . Continuous convolution aims at directly updating the last layer of the convolution group rather than updating each layer to further reduce the amount of data transfer on new array . Although the updated value may be blemished by the don ’ t-care value x in continuous convolution , it causes insignificant effect on the overall precision while significantly saving the processing time .

Summary: This paper proposes a technique to reduce the compute cost when applying recognition models in surveillance models. The core idea is to analytically compute the pixels that changed across frames and only apply the convolution operation to those pixels. The authors term this as dynamic convolution and evaluate this method on the SSD architecture across datasets like PETS, AVSS, VIRAT.