# Toward Scalable Video Analytics Using Compressed-Domain Features at the Edge
## 2 Background
### Compressed-Domain-Based Moving Object Detection  
The traditional video coding standards, such as MPEG-1, MPEG-2, H.264, and H.265, are based on the motion estimation process, which involves identifying a correspondence among consecutive images. In video coding standards, the MVs of a current block are correlated with the MVs of neighboring blocks in the current image or in the earlier coded pictures [12; 13]. While intra-picture prediction exploits correlations between spatially neighboring samples, inter-picture prediction makes use of temporal correlation between images to derive a motion-compensated prediction (MCP) for a block of image samples [14]. For intra-prediction, we assume that neighboring blocks possibly correspond to the same moving object with similar motion and the motion of the object is not likely to abruptly change over time. Consequently, using MV in neighboring blocks as the predictor reduces the size of the signaled motion vector difference. For this block-based MCP, a video image is divided into rectangular blocks. Figure 2 shows the general concept of MCP based on a translational motion model. Using a translational motion model, the position of the block in a previously decoded picture is indicated by a motion vector \((\Delta x,\Delta y)\), where \(\Delta x\) specifies the horizontal and \(\Delta y\) the vertical displacement relative to the position of the current block. Note that the motion vectors \(\Delta x\), \(\Delta y\) could be of fractional sample accuracy to more accurately capture the movement of the underlying object. These MVs are coded by entropy coding and placed into a compressed bitstream for delivery to the video decoder, which uses MVs, along with one or more previously decoded pictures, to reconstruct the current image. Therefore, MV information indeed follows the real motion of objects and can be used for tracking. On the other hand, H.264/AVC (Advanced Video Coding) in view of higher resolution and low bandwidth utilization and storage requirement becomes the popular video compression standard for surveillance video. In H.264/AVC, each video sequence is divided into groups of pictures (GOP), comprising at least one intra-code I frame, uni-directionally predicted P frames, and bi-directionally predicted B frames. Typically, the first frame in a GOP is the intra-coded I frame followed by the P and B frames. The I frame uses raw data from the camera, while other frames in a GOP utilize the predictive coding involving motion vectors' displacement. Each frame is partitioned into rectangular areas called macroblocks that are further partitioned into variable size blocks. The motion compensation of displacement is performed at the macroblock and block levels for all intra-coded frames. In the process of motion compression, MVs are obtained for each motion block between the current and reference frames. By minimizing the prediction residual, the MVs represent the temporal displacement between the two block. In recent studies, compressed-domain-based video analytics methods [15; 16] that rely on video coding artifacts of compressed bitstreams, such as MVs, macroblock partition, and quantization coefficients, have been proposed. In [15], the authors applied a probabilistic technique of computer vision for image separation, known as graph cut [17], modeling with MVs rather than pixels and adapted to the additional temporal dimension of video signals. MVs and a spatio-temporal Markov random field (ST-MRF) model that naturally integrates the spatial and temporal aspects of the object's motion for tracking were used. In general, these approaches do not rely on pixels and studies by only using the codec's MVs and block coding modes' extracted bitstream through inexpensive partial decoding. In this manner, computing and storage requirements have been significantly reduced compared to the \"pixel-domain\".