A GPU‐free real‐time object detection method for apron surveillance video based on quantized MobileNet‐SSD

IET Image ProcessingVolume 16, Issue 8 p. 2196-2209 ORIGINAL RESEARCHOpen Access A GPU-free real-time object detection method for apron surveillance video based on quantized MobileNet-SSD Zonglei Lyu, Corresponding Author Zonglei Lyu zllv@cauc.edu.cn College of Computer Science and Technology, Civil Aviation University, Tianjin, People's Republic of China Aviation administration of China, Civil Aviation University of China, Tianjin, People's Republic of China Correspondence Zonglei Lyu, College of Computer Science and Technology, Civil Aviation University, Tianjin300300, People's Republic of China. Email: zllv@cauc.edu.cnSearch for more papers by this authorDan Zhang, Dan Zhang College of Computer Science and Technology, Civil Aviation University, Tianjin, People's Republic of China Aviation administration of China, Civil Aviation University of China, Tianjin, People's Republic of ChinaSearch for more papers by this authorJia Luo, Jia Luo College of Computer Science and Technology, Civil Aviation University, Tianjin, People's Republic of ChinaSearch for more papers by this author Zonglei Lyu, Corresponding Author Zonglei Lyu zllv@cauc.edu.cn College of Computer Science and Technology, Civil Aviation University, Tianjin, People's Republic of China Aviation administration of China, Civil Aviation University of China, Tianjin, People's Republic of China Correspondence Zonglei Lyu, College of Computer Science and Technology, Civil Aviation University, Tianjin300300, People's Republic of China. Email: zllv@cauc.edu.cnSearch for more papers by this authorDan Zhang, Dan Zhang College of Computer Science and Technology, Civil Aviation University, Tianjin, People's Republic of China Aviation administration of China, Civil Aviation University of China, Tianjin, People's Republic of ChinaSearch for more papers by this authorJia Luo, Jia Luo College of Computer Science and Technology, Civil Aviation University, Tianjin, People's Republic of ChinaSearch for more papers by this author First published: 20 March 2022 https://doi.org/10.1049/ipr2.12483AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Object detection from apron surveillance video is facing enormous storage pressure and computing overhead. Large cloud server cluster is generally used and high-speed network bandwidth is required, also equipped with powerful GPUs for computing support. The design of hardware-friendly and efficient object detection model is challenging. This paper presents a compression method for outdoor apron surveillance videos, which is further combined with a lightweight detection model to make the inference process independent of GPU. First, the gray level variance of dynamic objects is leveraged to binarize the monitoring images, then an improved MobileNet-SSD algorithm is proposed. Moreover, int8 quantization is performed and bit operations are designed to eliminate the floating-point operation and it can simultaneously accelerate and compress CNN models with only minor performance degradation. Experiment results on a large-scale dataset containing 22k monitoring images demonstrate that the compression ratio of quantized image can achieve up to 21 times, combined with quantized model, the detection on apron surveillance images can reach nearly 25FPS in a pure CPU environment, the mAP is 86.83%, and the model size is compressed to 600 kb. Significantly reduced computational complexity can be applied to embedded devices. 1 INTRODUCTION Surveillance video has the following salient feature. First, the angle and position of the cameras set for the monitoring scene are generally fixed, which makes the background of the surveillance video relatively constant during the time interval. Second, considering that surveillance is the base of public security issues, the existing cameras in public situations are high-resolution, omni-directional and multi-perspective. Massive video data places extremely high requirements on video analysis technology and real-time performance. Video surveillance is the physical basis of the entire airport security system, and processing monitoring data is the core link. At present, the research on apron surveillance video mainly includes video compression and detection [27], as well as surveillance video indexing and retrieval [28]. However, the intelligent processing capacity of the existing monitoring system is insufficient. The application of object detection technology in apron surveillance has the following problems. First, apron monitoring requires clarity and recording duration, which will generate huge data volume. Traditional video surveillance processing depends on cloud server cluster. Massive video data has high computational overhead and high demand for GPU, which causes great storage and calculation pressure on the cloud server and has extremely high requirements for network bandwidth. Second, in addition to the inherent characteristics of the surveillance video, the apron also has its specific physical properties. Apron is generally located in the suburbs, so the view of detection image is relatively broad, which also makes the detection more vulnerable to adverse weather such as rain, fog, and illumination variance as shown in Figure 1. Third, surveillance videos are mostly used in security assurance, and the real-time requirements are growing. It is the current trend to build lightweight models and directly deploy detection models on monitoring device to elevate the intelligent processing capabilities of front-end cameras. Nevertheless, like other monitoring scenarios, only embedded devices with limited computing resources are available in the apron. Low-power devices can hardly load large models and conduct enormous floating-point operations. This poses a great challenge to the design of high-performance models on mobile devices. FIGURE 1Open in figure viewerPowerPoint Examples to illustrate the original (upper) and the corresponding binarized images (below) of apron surveillance videos under complex scenes. (a) Ideal condition. (b) Night illumination. (c) Foggy weather Object detection, as a major task of computer vision, inevitably needs to adapt to edge equipment when moving from laboratory to application, which will face the limited resources. The measurement standards of the previous classical detection algorithms are focus on the detection accuracy, verified on ultra-large datasets such as Pascal VOC (Pascal Visual Object Classes) and COCO (Common Objects in Context). In 2012, AlexNet [8] won the championship in ILSVRC, and then many excellent CNN (Convolutional Neural Network) models emerged, including ResNet [10], GoogLeNet [9], VGGNet [11]. Loading and inferring such large-scale models can perform well on expensive, GPU-based machines. But the embedded environment is subjected to many factors, especially, memory, latency, computing power, and power consumption. Therefore, the parameters and computations of neural networks have gradually become an important indicator to measure the performance of the detection algorithm, which determines the deployment on the small terminals. Recently, researchers have shown an increased interest in lightweight network such as MobileNet [2, 3], ShuffleNet [12], EfficientDet [25]. The evolution of backbone networks reflects that detection algorithms are designed to make trade-off between model performance and model efficiency. The benchmark no longer only focuses on the improvement of accuracy, but sacrifices a certain range of accuracy to obtain faster inference speed. In addition to directly designing novel network architectures, quite a few scholars have studied the compression of the CNN model, among which quantification is a very promising method. This methodology quantizes the weights and activations of models from 32 bit floating point into lower bit representations, and the biggest advantage of model quantization over other compression methods is that it is hardware-friendly. Specifically, when quantified to a particular number of bits, it can bring great hardware acceleration. For binarization [4-7], from Nvidia GPU to x86 platforms, the theoretical performance of 1bit computing is improved by 5–128 times, respectively. And only one additional quantification operation will be introduced, which can benefit from the acceleration of SIMD (Single Instruction Multiple Data Streams). However, low-bit quantization also accompanied by a severe decline in accuracy. In this paper, a detection architecture is presented, which quantify the input images and the MobileNet-SSD model. First, the approach leverages the gray level changes of dynamic instance to separate moving object from the background, processes the three-channel full-precision image into a single-channel binary image, and makes the monitoring image binary. According to the external characteristics of the apron surveillance video, the multi-layer binary images are extracted to quantize the input layer, then the optimal binary image is select by training the threshold model. The advantage of this compression is that the 1-bit encoding method amplifies the contour difference between the object and the background, and is naturally robust under foggy weather condition and illumination variance such as night environment as shown in Figure 1. Further, the processed binary images are sent to the 8-bit quantized MobileNet-SSD model for detection, and a special binary logic operation is designed to eliminate floating-point arithmetic operation. As data exists and calculates in binary form in the computer, so the bit operation can significantly improve the efficiency of the code on the machine. The main contributions of this paper can be summarized as follows: Binarization compression method for apron surveillance videos is proposed. It amplifies the contour difference between the object and the background, and is naturally robust under foggy weather condition and illumination variance. Bit operations are designed to eliminate floating-point operations in the inference of quantized model. This allows complex tasks, such as object detection, to be inferenced without GPU. The detection speed can reach to real-time in pure CPU environments, and the size of the model is compressed to the extent that can be deployed on most edge devices. The size and the inference time of each model is shown in Table 6 The remainder of this paper is organized as follows. Section 2 reviews some related works. Section 3 describes the details of the proposed method. The experimental results are presented and compared with some other advanced algorithms in Section 4. Finally, in Section 5, the algorithm is discussed and the prospect of future application is prospected. 2 RELATED WORKS 2.1 CNN-based object detectors Object detection are mainly categorized into two-stage detectors and one-stage detectors. Two-stage detection is mainly R-CNN series algorithms based on region proposal. For the first time, RCNN uses CNN to break the accuracy bottleneck of traditional detection methods, uses Selective Search to generate candidate boxes, extracts each feature, and then classifies and modifies the position [16]. Fast RCNN extracts feature from the whole input image, and then obtains fixed size features through ROI pooling layer [17]. Faster RCNN proposed Region Proposal Network (RPN) instead of Selective Search, and realized a complete end to end detection model [18]. The core of one-stage detection algorithm is regression. The category confidence probability and position coordinate value of the object are generated directly after a single detection. Among one-stage detectors, YOLO [14, 19] and SSD [15] series and their derived algorithms are now widely used. YOLO uses the Grid Cell mechanism to divide the input image into SxS grids, and each grid is responsible for detecting objects whose centre is in the grid area. Subsequent improvements are based on this idea to optimize the detection head, loss function and backbone [19]. The main highlight of SSD is the feature fusion, which integrates multi-scale feature maps for detection. Besides, SSD follows the Anchor mechanism in Faster-RCNN to generate bounding boxes with different scales and aspect ratios [15]. Generally-speaking, the object detection algorithms first extract features through the backbone network, to transform the input image into a feature map, and then perform two subtasks of classification and coordinate regression on the feature map. Therefore, the evolution of target detection algorithms is usually accompanied by the evolution of the backbone feature extraction network. 2.2 Model compression Numbers are all discrete in computer storage, but due to the precision of representation is very small, therefore, they can be treated as continuous numbers. The essence of quantification is to convert floating-point operation into integer fixed-point operation. The difficulty of quantization is that the numbers become discrete after quantization. As the Table 1 shows, the accuracy of representation varies for different numeric types. Converting high precision to lower precision greatly reduces the likelihood of selecting a value. Using lower numerical types means smaller data representation ranges and sparser values, which will cause loss of precision when quantizing. TABLE 1. Representation ranges in different numerical types Num bits Min value Max value Precision int8 −16 15.875 0.125 FP16 −65,504 65,504 2−10 FP32 −3.4 × 10−38 3.4 × 1038 2−23 According to the training mode, model quantization can be divided into post-training quantization and training-aware quantization. The former is generally used in practical engineering applications, while the academic community pays more attention to the latter, and aims to solve the problem of the disappearance of gradients caused by quantization and difficulty in training. 2.2.1 Post-training quantization This method selects the appropriate quantization operation and calibration operation for the trained network to minimize the quantization loss. At present, INT8 quantization is mostly used in the industry. TensorRT, TensorFlow, PyTorch and many other deep learning frameworks have already enabled or are enabling quantization. The representative method is the IAO proposed by Google, which allows the network to use only integer arithmetic when inferring [1]. 2.2.2 Training-aware quantization Academic research on quantization focuses on how to reduce the precision loss caused by model quantization, which can be further divided into binary quantization [4-7], ternary quantization [20, 21], clustering quantization [22, 23] and mixed precision quantization [24]. Among them, at very low bit (<4bit), binary network (1bit), the accuracy challenge is greater. Since binarization has remarkable hardware acceleration benefits, its representative works are mainly introduced. Binary Connect binarizes the weights and gives the training method of binarized parameters in forward and backward propagation [4]. BNN binarizes weights and activations in training phase, and replaces most arithmetic operations with bit operations [5]. XNOR-Net transforms the parameters of filter and convolution layer into binary, and can approximate convolution by bit operation, which achieves 58 times theoretical calculation acceleration [6]. Based on XNOR-Net, Bi-Real Net introduces a shortcut connection and uses quadratic function to fit the real number activation sign operation, which maintains high reasoning accuracy [7]. Consider a floating variable with range (xmax,xmin) that needs to be quantized to the range (0, Nlevels) where Nlevels = 256 for 8-bits of precision. Derive two parameters: Scale (∆) and Zero-point(z) which map the floating values integers [26]. The strategy can be roughly divided into uniform quantization and non-uniform quantization. Among them, non-uniform quantization requires calculation of scale and zero point. Zero point, also called offset. Because 0 has a special meaning, for example, when padding, the value of 0 is also involved in the calculation. Zero point is an integer number to ensure that there is no quantization error in the mapping process of 0. After adding zero point, the floating-point type 0 will be mapped to a number in the interval (0,255), so that the quantization is more accurate. It is equivalent to offsetting the entire interval after mapping, so the minimum value of floating point corresponds to 0 point. Uniform quantization constrains zero point to the position of 0, and all floating-point numbers are mapped to the interval (−127, 128). 3 METHOD Aiming at the surveillance video features mentioned in the introduction, video-oriented image compression and real-time detection methods for surveillance scenarios are proposed. The algorithm first processes the video sequence to extract the background images during the time interval. Then for the frames to be detected, moving objects in the monitoring video are extracted from the fixed background. Since the binarization process is equivalent to filtering the image noise, binarization can be well applied to outdoor weather changes such as foggy conditions, as the visibility degradation caused by fog can be regarded as white noise. Finally combine the quantized object detection model to obtain the real-time detection results as Figure 2 shows. FIGURE 2Open in figure viewerPowerPoint The flow chart of proposed method 3.1 Background modelling and image binarization Converting a uint8 image with a value range of 0–255 into a tensor with a value range of 0.0–1.0 of float32 is called de-quantization. Accordingly, the process of adjusting the tensor of network output range from 0.0 to 1.0 to image data in uint8 format is quantization. Obviously, de-quantization has no information loss while quantization usually has precision loss. This is because a considerable number of values that cannot be represented by uint8 and can only be rounded into uint8 values. Conversion errors between full-precision and quantitative also arise from rounded clip operations. Digital images are represented in computers as colour images, gray-scale images, binary images, etc. Binary images are a highly memory-saving storage format, where 0 represents the background and 1 represents the foreground. Only 1 Bit per pixel is required to store the information in its entirety. If each pixel is considered as a random variable and has a total of N pixels, then the binary graph has 2N changes, the 8-bit grayscale graph has 28N changes, and the 3-channel RGB image has 224N changes. That is, for images of the same size, the binary graph holds less information, and there are only two possible values or gray level states for each pixel in the image, which also means simpler calculation and faster inference speed. Figure 3 shows the presentation of a same 900-pixel picture in different storage formats. FIGURE 3Open in figure viewerPowerPoint An example of pixel 900 × 900 image on different storage type The most essential factor in identifying objects is the gradient, which means the edge. Just like drawing, the most basic shape is point and line, and the distinction between shape and background is either with or without, represented as 1 and 0 in computer storage, which happens to be the idea of binary quantification. Colour itself is very susceptible to light and other factors and there are variety in the colour of same category. In the application scene of the apron, the category of the detected object, that is, the contour feature, is more often concerned than its colour attribute. Besides, the background of the surveillance video is relatively fixed and the scene is relatively simple. For object detection of such specific scenes, image information can be compressed and encoded first. Images are stored in the computer as stacked matrices, also known as a multi-dimensional tensor. After quantification, the dimension of the tensor decreases, resulting in less storage space and faster inference speed, while gradient information is still preserved. To realize the binarization of the monitoring frames, and reduce the amount of image data and simplify the calculation process of forward reasoning. The first step is to perform background modelling processing on the apron surveillance video. Extract the background of a piece of video at time t (in seconds), assuming that a frame is intercepted every k-seconds, then t/k frames will be obtained. And the two adjacent video frames will be subtracted to obtain the mask. The formula is as follows: m a s k i = 1 , I ( t i ) − I ( t i − 1 ) ≥ T 0 , e l s e , 1 ≤ i ≤ t k \begin{equation}mas{k_i} = \left\{ \def\eqcellsep{&}\begin{array}{l} 1 \def\eqcellsep{&}\begin{array}{*{20}{c}} ,&{I({t_i}) - I({t_{i - 1}}) \ge T} \end{array} \\[6pt] 0 \def\eqcellsep{&}\begin{array}{*{20}{c}} ,&{else} \end{array} \end{array} \right.,1 \le i \le \frac{t}{k}\end{equation} (1)where T represent the threshold of binarization. Further, the background image B ( x , y ) $B(x,y)$ is recalculated by the mask matrix with the value of each pixel in the original image I ( x , y ) ${\rm{I}}(x,y)$ . And the calculation formula is: B ( x , y ) = I ( x , y ) ∗ M , M = m a s k 1 ∩ m a s k 2 ∩ … ∩ m a s k i \begin{equation}B(x,y) = I(x,y)*M,M = mas{k_1} \cap mas{k_2} \cap \ldots \cap mas{k_i}\end{equation} (2) Separating the moving object from the background by using the gray change of dynamic objects. Subtracting the background image B ( x , y ) $B(x,y)$ from the image to be detected f ( x , y ) $f(x,y)$ can obtain the differential image D ( x , y ) $D(x,y)$ . The processing formula is as follows: D ( x , y ) = f ( x , y ) − B ( x , y ) \begin{equation}{\rm{D}}(x,y) = \left| {f(x,y) - B(x,y)} \right|\end{equation} (3) Further perform the binarization operation on the minus image with the binarization threshold T, and binarize the pixels one by one to obtain binarized image Q ( x , y ) $Q(x,y)$ : Q ( x , y ) = 255 , D ( x , y ) ≥ T 0 , e l s e \begin{equation}Q(x,y) = \left\{ \def\eqcellsep{&}\begin{array}{l} 255,D(x,y) \ge T\\[6pt] 0,else \end{array} \right.\end{equation} (4) And an example of binarization process of monitoring image is shown in Figure 4. The core idea of proposed method is to approach the optimal threshold step by step through multiple rounds of iterative operations. Set an initial threshold T0, which divide all the pixels of the image into two categories. One category has pixel values less than or equal to T0 (background area), and the other category has pixel values greater than T0 (foreground area). And then calculate the pixel average value m0 and m1 of the background area and the foreground area separately, and take 2/(m0+m1) as the threshold for the next iteration. When the difference between the threshold obtained in the current round of iterations and the threshold value of the previous round of iterations is small, the iteration ends. The binarized image during the threshold adaptation process is shown in Figure 5. FIGURE 4Open in figure viewerPowerPoint The process of the proposed binarization method. Using the (a) original RGB image, (b) background image to do the differential processing. Further, set threshold to (c) minus image and get (d) binary image FIGURE 5Open in figure viewerPowerPoint The binarized image during the threshold adaptation process 3.2 An improved MobileNet-SSD algorithm The quantization strategy in this paper is shown in Figure 6. After the input image is quantized by the binarization method in Section 3.1, the binary picture is obtained after subtracting the background. The detection model is further quantized in 8-bits, which can make the image is only involved in the calculation of integer numbers in the forward inference process. As the time-consuming floating-point calculations can be removed, and the entire calculation process can be greatly accelerated and simplified. FIGURE 6Open in figure viewerPowerPoint Schematic diagram of image quantization combined with model quantization Traditional SSD [15] algorithm uses multi-scale feature map for object detection, which can extract multi-scale features, reduce the size of each subsequent layer, greatly improve the detection accuracy and speed, but the real-time detection is not so satisfactory. The MobileNet algorithm uses deep convolution layer as the basic layer network, which optimizes the delay while considering the size of the model, but for object detection, the accuracy is still low. Therefore, the combination of the two networks is proposed to achieve real-time and high-precision detection at the same time. Table 2 lists the parameters and calculation complexity comparison of VGG16 and MobileNetV1. Compared with the MobileNet backbone network, VGG16 requires higher GPU memory and larger model size. The lightweight quantized MobileNet-SSD model for real-time detection was proposed to extract features by adjusting the backbone network and using MobileNetV1 [2]. The basic unit of MobileNetV1 is Deeply Separated Convolution (DSC) [13]. TABLE 2. Parameters and calculation complexity comparison Model Params(M) FLOPs MAdd VGG16 138.62 27.17G 54.26G MobileNet 4.25 1.08G 2.15G MobileNet_0.75 2.6 623.87 M 1.23G MobileNet_0.25 0.48 82.45 M 160.28 M Its improvement lies in the use of fractional convolution instead of classical 3D convolution, which reduces the redundant expression of convolution kernel and greatly reduces the amount of computation. The size of the input image is normalized to 320 × 320. From Conv1 to Conv13 are deep separable convolutional layers. The specific architectures are listed by Figure 7. And the basic network part of MobileNet-SSD from Conv1 to Conv13 is completely consistent with the original MobileNet model, and the convolutional layer and the BN layer are merged on the added convolutional layer. The convolved layers of Conv11, Conv13, Conv14, Conv15 are extracted as the detection layer. As shown in the right part of Figure 5, the size of the objects in the apron monitoring screen has certain disciplines. FIGURE 7Open in figure viewerPowerPoint The overall architectures of the improved MobileNet-SSD With the background of airport surveillance video, the Conv16 and 17 layers from the detection network are removed. Correspondingly, the size has changed from the original 6–4 scales. The aspect ratio of the default box has a great influence on the IOU of the default box and the real box, and is also a very important parameter for training the detection model. The original aspect ratio is very suitable for the PASCAL VOC2017 dataset, but it is not applicable for the detection of apron. Therefore, the K-means clustering algorithm is used to cluster the aspect ratio of the instance areas in the apron database to obtain the suitable aspect ratio for detecting. According to the clustering results, the aspect ratio used by the original SSD algorithm has been further adjusted. There are almost no detecting instances in the apron surveillance database with an aspect ratio of more than 3, most of which are at 1, 2, and 2.55. Combined with the aspect ratio of the original SSD algorithm, the number of default boxes and aspect ratios for each detection layer of the MobileNet-SSD algorithm is obtained, see Table 3. The matching strategy of Conv11, Conv13, Conv14, and Conv15 network layers are improved. And the default frame number of Conv11 network layer is improved from 3 to 4. TABLE 3. Feature map size, aspect ratio and number of default box for each detection layer Detection Layer Feature size Original aspect ratio Aspect ratio Original Number of default boxes Number of default boxes Conv11 20 × 20 {1, 2, 1/2} {1, 2, 1/2} 3 4 Conv13 10 × 10 {1, 2, 3, 1/2, 1/3} {1, 2, 2.55, 1/2, 20/51} 6 6 Conv14 5 × 5 {1, 2, 3, 1/2, 1/3} {1, 2, 2.55, 1/2, 20/51} 6 6 Conv15 3 × 3 {1, 2, 3, 1/2, 1/3} {1, 2, 2.55, 1/2, 20/51} 6 6 3.3 Model quantification and inference acceleration In order to convert the floating-point number in the model into an 8-bit integer number, the non-uniform quantization strategy proposed in the Google whitepaper [26] is adopted. Specifically, convert a floating variable with range ( x m i n ${x_{min}}$ , x m a x ${x_{max}}$ ) to the range (0, N l e v e l s ${N_{levels}}$ ) where N l e v e l s ${N_{levels}}$ = 256 for 8-bits of precision. Before quantization, the quantization factor must be calculated, including two parameters: Scale (∆) and Zero-point(z). The quantization factor is used to ensure that the variables in the floating-point interval can be mapped to the value interval of the quantized bit number without any omission. Among them, the calculation formulas of scale and zero point are as follows: Δ = ( x m a x − x m i n ) 255 , z = − x m i n Δ \begin{equation}\Delta = \frac{{({x_{max}} - {x_{min}})}}{{255}},z = - \frac{{{x_{min}}}}{\Delta }\end{equation} (5) Once the scale and zero-point are defined, quantization proceeds as follows: x i n t