DYNCNN: AN EFFECTIVE DYNAMIC ARCHITEC-
Year: 2018

Abstract:
The large-scale surveillance video analysis becomes important as the development of intelligent city. The heavy computation resources neccessary for state-of-theart deep learning model makes the real-time processing hard to be implemented. This paper exploits the characteristic of high scene similarity generally existing in surveillance videos and proposes dynamic convolution reusing the previous feature map to reduce the computation amount. We tested the proposed method on 45 surveillance videos with various scenes. The experimental results show that dynamic convolution can reduce up to 75.7% of FLOPs while preserving the precision within 0.7% mAP. Furthermore, the dynamic convolution can enhance the processing time up to 2.2 times.

heading: 1 INTRODUCTION
text: Nowadays, with the development of deep learning technologies, large-scale surveillance video analysis for intelligent city draws more and more attention in the real world applications, for instance, in person re-identification (Ahmed et al. (2015)), vehicle re-identification (Liu et al. (2016))(Shen et al. (2017)), pedestrian detection (Tian et al. (2014)) and crowd segmentation (Kang & Wang (2014)), etc. However, these deep learning methods are extremely computationally expensive—state-of-theart methods for object detection performed on a state-of-the-art NVIDIA P100 GPU run at 10-80 frames per second. Although processing on one video can be done by utilizing one to several topperforming hardware equipments, it still a challenge for large-scale actual deployment—it would cost over 5 billion USD in hardware to analyze over 4 million CCTVs in real time in the UK alone when considering these computational overheads in context (News (2015)).
Those real world applications for intelligent city employed the state-of-the-art network architectures from the ILSVRC competition, e.g. VGG (Simonyan & Zisserman (2014)), GoogLeNet (Szegedy et al. (2014)), and ResNet (He et al. (2015)) as their feature extraction architecture. However, these powerful networks tend to be resource-hungry models with high computational demands at inference time to win the competition—processing an one-minute surveillance video through VGG16 network costs 23 TFLOPs. To make the feature extraction architectures more effective in terms of surveillance video, this paper aims to explore ways to reduce the calculation on feature extraction.
Figure 1 shows two frame clips of a surveillance video with its corresponding feature map through 3×3 kernel. It is observed that most of the scene between two adjacent frames in the surveillance video are almost the same with merely about 2% to 15% difference. Due to the linear characteristic of convolution, the derived feature maps preserve the similarity of input video frames—it can reduce significant amount of calculation if the previous feature map can be reused for the current feature map. Suppose that the frame size is 512×512 while the amount of the changing pixels in feature map is 52k, the calculation can be reduced from 7 MFLOPs to 1.4 MFLOPs, that is 5 times saving ratio.
In this paper, we exploit the key characteristic of surveillance video—high scene similarity—and propose a new architecture dynamic convolution which can be directly applied to off-the-shelf model without retraining the weights. Dynamic convolution can inspect the inter-frame variation and thus predict the changing pixels in each corresponding feature map. Accordingly, convolution performs only on those predicted parts to achieve less computation while not affecting the accuracy. Finally,
we apply dynamic convolution to one of the state-of-the-art object detector Single Shot MultiBox Detection (Liu et al. (2015)) and test on several surveillance video datasets including overall 45 videos. In addition, we conduct a further analysis in the FLOP computation and execution time of each convolutional layer to improve the understanding of dynamic convolution in practice.

heading: 2 PRIOR WORKS
text: For the last several years, tuning deep learning neural architectures to strike an optimal balance between precision and processing time has been an area of active research. Many methods have been proposed to reduce storage space and computing resources and also made great progress from theoretical research to platform implementation. Basically, they can be divided into three major techniques, effective model design, pruning, and quantization.
Effective model design MobileNets (Howard et al. (2017)) is a lightweight network architecture proposed by Google for mobile deployment. The main idea is to disassemble the original convolution calculation into two parts: depthwise convolutions and pointwise convolutions. Although the calculation amount on convolution can greatly be reduced, the precision would be decreased. In addition, the feature map accordingly increases. The increasing data transfer followed by feature map makes MobileNets hard to perform on GPU. To improve the precision, ShuffleNet (Zhang et al. (2017)) proposed two new strategies channel shuffle and pointwise group convolutions based on the concept that helps the information flowing across feature channels. On the other hand, Inception (Szegedy et al. (2016)) introduces bottleneck structure to approximate sparse structures into several dense sub-matrices to achieve more efficient use of computing resources. In the ultra-lightweight neural network SqueezeNet (Iandola et al. (2016)), the 1x1 and 3x3 convolution kernels are extensively used compared with Inception to achieve more compression on calculation amount.
Pruning Besides of the model design, pruning is another efficient way to reduce storage space and computing resources by removing redundant parts of model. Li et al. (2016) proposed a method for cropping filters of convolutional layer. This method determines the importance of filter by the magnitude of the absolute sum of weights on the filter. After that, the model needs to be trained again. Without lossing greately precision, such method can effectively reduce the complexity of the model. However, heavy analysis on each convolution layer costs lots of time and may occur the concern of one-case model. Liu et al. (2017) used another factor, the scaling coefficient γ in batch normalization, to determine the importance of the filter.
Quantization Network weight sharing quantization is also an important type of network compression method which focuses on the weight representation. The nature of this technique is to find the center value which can represent the original weight distribution by clustering method. Deep Compression proposed by Han et al. (2015) clusters the weights and replaces them by the center value of each cluster. The transformation between the weights and center values is stored in a codebook. Such transformation greatly compresses the weight representation from a 32-bit floating point number to a short bit number.

heading: 3 DYNAMIC CONVOLUTION
text: Dynamic convolution is an architecture which applies to each layer of convolutional neural network. Figure 2 illustrates a convolution neural network with our dynamic convolution, called dynamic convolutional neural network (DynCNN). The work flow of dynamic convolution consists of three main parts: 1) frame differencing; 2) prediction; 3) dyn-convolution. Let IDMt denote the input difference map of t-th frame (Framet) derived by frame differencing and iDMt denote the inner difference map derived from the previous inner difference map or input difference map by prediction. Dyn-convolution preserves the feature map (FM) from the previous layer and selectively do convolution according to the position which the inner difference map indicates in order to update the feature map.

heading: 3.1 FRAME DIFFERENCING
text: The frame differencing method (Liu & Hou (2012)) is often used in the moving object detection and segmentation methods. In this work, we employ it to inspect the inter-frame variation between two adjacent frames. The basic concept of frame differencing is to subtract two frames to calculate the difference of each pixel. However, in most surveillance video sequences there exist speckle noise which severely affects the difference value. Accordingly, the difference map, as referred to input difference map (IDM), is derived by thresholding the result of frame differencing with the following statement:
D(i, j) = { 1, |∆I(i, j)| < ΘIDM 0, otherwise
(1)
where ΘIDM denotes the threshold of the statement. The pixel at (i,j) is denoted as a changed pixel when D is determined as 1 and we called it changing point. ∆I(i, j) in (1) represents the result of frame differencing as the following expression:
∆I(i, j) = ICurr(i, j)− IPrev(i, j) (2)
where ICurr(i, j) and IPrev(i, j) in (2) denote the pixel intensity at position (i, j) of the current and previous frame respectively.
The value of the threshold ΘIDM is an important factor which determines the amount of convolution calculation. Therefore, the effect of the threshold on the speed and accuracy during the inference phase will be well analyzed in Section 4.

heading: 3.2 PREDICTION
text: In the convolution process, there exists a diffusion effect in the result if the input is changed. Suppose that the kernel size of convolution is 3×3, each pixel of the result is determined by the corresponding
9 pixels from the input—the pixel at the same position and its 8 peripheral pixels. On the contrary, when a certain pixel in the input is changed, 9 pixels in the output which the pixel involves will be affected simultaneously and are denoted as impacted pixels. Based on the characteristic of diffusion effect in convolution process, the position of impacted pixels in each feature map can be predicted by using dilation operation and recorded on the inner difference map.

heading: 3.3 DYN-CONVOLUTION
text: The iDM records the position of the impacted pixel of the FM and also implicates which pixel value of the FM needs to be updated. To update the value, the pixels of the previous FM which contributes to it will be re-convoluted and are denoted as needed pixels while this process is called dyn-convolution. In this work, we employ the lastest cuDNN library developed by Nvidia to optimize the performance on calculation speed. However, the convolution calculation provided by the library is only allowed for contiguous blocks instead of discrete blocks with given position. Therefore, we create another memory space, denoted as new array, in GPU to store those needed pixels for the requirement of library.

heading: 3.4 PRACTICAL CONCERN AND IMPLEMENTATION
text: In real surveillance video application, although the calculation reduction on convolution is the main concern of speeding up the overall processing time, the data transfer is another important factor which contributes to the time. Dyn-convolution is intuitive but rude and also not efficient in GPU. Accordingly, we apply two strategies to improve the overall processing time, cell-based partition and continuous convolution.
Cell-based Partition Compared with pixel-based map iDM, each FM is divided into several block cells. If any impacted pixel exists in the cell, the cell will be noted as “impacted” and recorded on cell difference map (CDM). The dyn-convolution thus performs according to the cell difference map rather than the pixel difference map—the new array stores the needed pixels of impacted cells. Figure 3 illustrates the difference between the naive method and the improved method.
In this strategy, the data in new array may contains “non-needed” pixels when the cell involves nonimpacted pixels. In addition, new array performs unnecessary convolution on the boundary of two needed pixel blocks, where generates don’t-care feature value x. Nevertheless, the needed pixels are reused to reduce the amount of data transfer and the characteristic of cuDNN library—convolution costs less processing time when stride is 1—is contemplated to speed up the convolution process, which indicates that less sacrification on computation while greatly improving the processing time.
Continuous Convolution Another part which costs processing time is the layer number. It multiplies the data transfer on a new array. We partition whole convolution layer by pooling process and
group the convolution layers up, called convolution group. For each convolution group, the convolution structure is modified from “once padding before once convolution” to “continuous padding first then continuous convolution”. Continuous convolution aims at directly updating the last layer of the convolution group rather than updating each layer to further reduce the amount of data transfer on new array. Although the updated value may be blemished by the don’t-care value x in continuous convolution, it causes insignificant effect on the overall precision while significantly saving the processing time.

heading: 4 EXPERIMENTS
text: This section will evaluate the performance of the dynamic convolution by employing the Single Shot MultiBox Detector 512× 512 model 1 (SSD512) on GTX1080 GPU with cuDNN v5.1. Ground Truth. The existing benchmarks with ground truth for multi-object detection are designed to evaluate the performance of CNN model on still image evaluation, such as Microsoft COCO (Lin et al. (2014)) and PASCAL VOC datasets (Everingham et al. (2010)). However, such benchmarks tested on the surveillance video are not given the ground truth. To well evaluate our proposed dynamic convolution, the ground truth is defined as the detection result of SSD512 model on surveillance video. We choose VOC712Plus SSD 512x512 iter 240000—trained on VOC2007, VOC2012 and COCO, 240000 step iterations, 83.2% mAP on VOC2007 test—as the weight parameters.
Baseline. Dynamic convolution is a mechanism aimed at reducing the computation of original CNN model on surveillance video without affecting the performance. In the experiments, we choose the weight parameters VOC0712 SSD 512x512 iter 120000—trained on VOC2007 and VOC2012, 120000 step iterations, 79.8% mAP on VOC2007 test—as our baseline to evaluate our proposed method. The performance critiria is how close the precision is compared to the baseline.
Dataset. The dynamic convolution is tested on four surveillance video datasets: PETS2009 (Ferryman & Ellis (2014)), AVSS2007 (AVS (2007)), VIRAT (Oh et al. (2011)) and some videos from Youtube. In the surveillance video, person and car are the main objects which appear in the scene and are the most significant targets for surveillance application. Therefore, the performance in the experiments focuses on the objects of person and car. Table 1 lists the charascteristics of four surveillance video datasets, where the scene similarity ratio is defined as the ratio of the number of non-changing points to frame size.

heading: 4.1 ON PETS 2009
text: The PETS 2009 dataset is a benchmark which focuses on several challenges for crowd analysis in public area including the estimation of crowd person count and density, tracking of individual(s) within a crowd, and detection of flow and crowd events. In this dataset, we choose 20 difference scene videos totally involving 4,884 photos as test data. Table 2 presents the result of the performance on objects person and car with the threshold of frame differencing setting at 20. It is clear to see that the proposed dynamic convolution can preserve the precision with respect to baseline, where the difference is only 0.06% and 0.21% on person and car respectively.
1available at https://github.com/weiliu89/caffe/tree/ssd

heading: 4.2 ON AVSS 2007
text: AVSS2007 is a data set for evaluating the algorithm on event detection and tracking. In this dataset, we choose six videos, which contains public surveillance scene for train station and road located in the UK, as our test data including 35,000 images. Table 3 presents the result of the performance on objects person and car with the threshold of frame differencing setting at 20. It is clear to see that the proposed dynamic convolution can preserve the precision with respect to baseline where difference is only 0.16% and 0.42% on person and car respectively.

heading: 4.3 ON VIRAT
text: VIRAT dataset collects broad surveillance videos in terms of various realism scenes including ground camera videos and aerial videos. In this dataset, 10 different scene videos on parking lot, lane and plaza totally involving 63,947 photos are selected as test data. Table 4 presents the result of the performance on objects person and car with the threshold of frame differencing setting at 20. It is clear to see that the proposed dynamic convolution can preserve the precision with respect to baseline, where the difference is only 0.54% and 0.1% on person and car respectively.

heading: 4.4 ON YOUTUBE
text: The collected surveillance videos on Youtube contain 9 scene videos with different viewing angles on the zoo, MRT station, campus and street involving 115,652 images. Table 5 presents the result of the performance on objects person and car with the threshold of frame differencing setting at 20. It is clear to see that the proposed dynamic convolution can preserve the precision with respect to baseline, where the difference is only 0.7% and 0.7% on person and car respectively.

heading: 4.5 CONTINUOUS CONVOLUTION
text: Table 6 shows the average calculation process in each layer over all test videos of PETS 2009 dataset including the DynCNN with and without continuous convolution. The fifth and eighth column represents the new array size of dyn-convolution and the horizontal lines represent the pooling process which is used to delimit the convolution groups. As the layer more deeper, the pruned ratio generally decreases due to the increasing size of needed pixels. Note that the increasing behavior of pruned ratio in each convolution group under DynCNN (w/). In continuous convolution, the inner difference map of the first layer of convolution group directly predicts the impacted pixels of the last layer in advance, which results in that the needed pixels on the new array is more than DynCNN (w/o) and decreases gradually as layer deeper. This phenomenon can also be found in other datasets as shown in Appendix 7.4. Table 7 summarizes the average calculation amount and processing time per frame on each dataset. From this table, although DynCNN (w/o) can prune more calculation amount than DynCNN (w/), DynCNN (w/o) needs to cost more processing time on data transfer. The improvement of the overall processing time is not significant and even invalid in dataset AVSS2007 with respect to baseline. Therefore, the strategy of continuous convolution is demonstrated that can make DynCNN more valuable at the practical application as mentioned in Subsection 3.4.

heading: 5 DISCUSSION

heading: 5.1 THRESHOLD OF FRAME DIFFERENCING
text: The threshold of IDM is used to filter the speckle noise in video. In addition, it can greatly speed up the processing time due to the high scene similarity of video—most difference values derived from frame differencing are introduced by noise. For various surveillance videos, the threshold of IDM should be different, resulting in a single threshold is difficult to be applicable to all surveillance images. To find the best threshold, we conduct a research on the relationship between the precision and calculation amount over different thresholds. Figure 4 shows the test result on 4 datasets.
It is observed from Fig. 4a that the calculation amount exists a significant drop in the threshold interval of 5 to 15 over 4 datasets, which indicates the threshold greater or equal to 15 can filter the speckle noise efficiently in four datasets. However, the precision will be affected as the threshold is greater or equal to 25 as shown in Fig. 4b. Therefore, by trading the calculation amount off against precision, 20 is the best choice of threshold applicable to various surveillance videos and is selected as the default in experiments.
5.2 THE EFFECT ON SCENE SIMILARITY
In Fig. 5, we reveal the distribution of each test surveillance videos on calculation amount and FPS with respect to the average scene similarity. The triangle mark denotes the average of videos in the same dataset while the purple line represents the value of baseline. It is obvious that the higher the
average scene similarity the more the calulation amount decreases. In addition, the FPS also tends to increase as the average scene similarity is higher, especially in the dataset VIRAT. In dataset VIRAT, the most scene of surveillance videos is parking lot where most of time are the still scene, resulting in 99% average scene similarity. The high scene similarity prunes the calculation amount of baseline model up to 75.7% (90.5 GFLOPs to 22.0 GFLOPs) and improves 2.2 times (32.5 to 71.7) of FPS. From the distribution map shown in Fig. 5, it is implied that the surveillance videos generally exist high scene similarity. Furthermore, the proposed dynamic convolution is applicable to all the videos. Therefore, it is believed that dynamic convolution can be applied to the practical applications on surveillance videos.

heading: 6 CONCLUSION
text: This paper exploits the high scene similarity generally existing in surveillance videos and proposes a dynamic convolution to reduce the calculation amount efficiently for heavy-computational surveillance analyses. The dynamic convolution only updates the feature value for the position which is predicted as changed to achieve the reuse of previous feature map. Besides the calculation amount, dynamic convolution also considers the importance of the processing time on pratical applications and employs two strategies to optimize the processing time by sacrificing a few calculation amounts. Compared with the existing technology, the it can directly be applied to the existing convolution neural network architecture without retraining and analyzing weights. Experimental results on four famous and authoritative datasets of surveillance videos provides a powerful evident that the proposed method is applicable to most of surveillance videos. The reduction on the calculation amount can be up to 75.7% while the precision preserves within 0.7% mAP. Furthermore, the processing time can be enhanced up to 2.2 times with respect to baseline. Finally, the aspect of design in the proposed method is completely different from the existing relevant methods. In the future work, we will complement our work with the existing relevent methods to verify the possibility of further acceleration.

heading: 7 APPENDIX

heading: 7.1 DYNAMIC CONVOLUTION MODEL
text: The proposed dynamic convolution is implemented on SSD512 model by caffe framework. Figure 6 and 7 illustrate the first convolution group (gp1) of SSD512 model and that with dynamic convolution (DynSDD512) respectively. In the implementation of dynamic convolution, the original model is combined with three additional layers: 1) Comparison Layer; 2) Position Layer; 3) Recovery Layer. Comparison layer is used to store the current frame and output input difference map (IDM) by comparing with the previous frame. After recieving the IDM, position layer will output the cell index table (CIT), for the position indication, the inner difference table (iDM) and the new array (NA) to next stage. Position layer generates the iDM by directly predicting the impacted pixels of the last layer in gp1 for the continuous convoltion as mentioned in subsection 3.4. Accordingly, the NA is generated according to the iDM through cell-based partition. Finally, recovery layer recieves the feature value from convolution and transfer those value where they locates on feature map through CIT.

heading: 7.2 GPU EXECUTION TIME
text: For a better understanding of the proposed method, we list the average execution time of each layer in detail. Take the PETS2009 dataset for example.

heading: 7.3 EXAMPLE DETECTIONS

heading: 7.4 CALCULATION RESULTS ON OTHER DATASETS

Review summaries:
This paper proposes a technique to reduce the compute cost when applying recognition models in surveillance models. The core idea is to analytically compute the pixels that changed across frames and only apply the convolution operation to those pixels. The authors term this as dynamic convolution and evaluate this method on the SSD architecture across datasets like PETS, AVSS, VIRAT.

The paper addresses the problem of computational inefficiency in video surveillance understanding approaches. It suggests an approach called Dynamic Convolution consists of Frame differencing, Prediction, and Dyn-Convolution steps. The idea is to reuse some of the convolutional feature maps, and frame features particularly when there is a significant similarity among the frames. The paper evaluates the results on 4 public datasets. However, it just compares the approach to a baseline, which is indeed applying convnet on all frames.