Video coding algorithms encode and decode an entire video frame while feature coding techniques only preserve and communicate the most critical information needed for a given application. This is because video coding targets human per- ception, while feature coding aims for machine vision tasks. Recently, attempts are being made to bridge the gap between these two domains. In this work, we propose to leverage on to the commonality that exists between human vision and machine vision applications by employing cuboids. This is because cuboids, estimated rectangular regions over a video frame, are computationally efﬁcient, has a compact represen- tation and object centric. Such properties are already shown to add value to traditional video coding systems. Herein cuboidal feature descriptors are extracted from the current frame and then employed for accomplishing a machine vi- sion task in the form of object detection. Experimental results show that a trained classiﬁer yields superior average precision when equipped with cuboidal features oriented representation of the current test frame. Additionally, this representation costs 7%less in bit rate if the captured frames are need be communicated to a receiver. Index Terms —Cuboid, HEVC, VCM, Object detection 1. INTRODUCTION Many video analysis systems employ a client-server architec- ture where video signals are captured at the front-end devices and the analyses are carried out at cloud-end servers. In such systems, video data need be communicated from the front end to the server. Prior to this transmission, captured video signal is encoded using a video coding paradigm and at the receiv- ing end the obtained signal is decoded before performing any analysis task. An alternative approach is to extract features from the video data at the client end and then communicate the encoded features rather than the video signal itself. Since capturing of the signal and feature extraction are conducted at the front end, the server becomes less clumsy pertaining to large scale applications and can dedicate most of its resources for the analysis work. However, as the estimated features aredevised for speciﬁc tasks only, they can be difﬁcult to gener- alize to a broad spectrum of machine vision tasks in the cloud end. Traditional video coding standards like H.264/A VC [1] and HEVC [2] are block-based. They are also pixel and frame centric. In order to model the commonality that ex- ists within a video sequence, the frame that need be coded, known as the current frame, is artiﬁcially partitioned into square or rectangular shape blocks. These blocks are formed by grouping together neighboring pixels. After this partition- ing, for each current frame block, commonality modeling is employed to form a prediction of it either using the already coded neighboring blocks belonging to the current frame (intra-prediction) or using motion estimation and compen- sation within a co-located neighborhood region in the set of already coded reference frame(s) (inter-prediction), by minimizing a rate-distortion criterion. Due to the requirements for communicating feature de- scriptors, coding standards like compact descriptors for visual search (CDVS) [3] and compact descriptors for video analysis (CDV A) [4] were developed. In CDVS, to represent the visual characteristics of images, local and global descriptors are de- signed. Deep learning features are employed in CDV A to fur- ther augment the video analysis performance. Although these features showed excellent performance for machine vision tasks, it is not possible to reconstruct full resolution videos for human vision from such features. This results into two successive stages of analysis and compression for machine and human vision [5]. As modern video coding standards like HEVC focus on human vision by reconstructing full resolution frames from the coded bitstream and standards like CDVS and CDV A fo- cus on machine vision tasks and incapable of reconstructing full resolution pictures from the coded feature descriptors; there is a need for human-machine collaborative coding that can leverage on the advances made on the frontiers of video coding and feature coding as well as bridge the gap between these two technologies. For example, in autonomous driving use case, some decisions can be made by the machine itself and for some other decisions humans would interact with ma- chine. Such collaborative coding problem is known as video coding for machines (VCM) [5]. In this direction of work, it was proposed in [6] to extract features in the form of edge maps from a given image which is used to perform machine vision tasks. After that a generative model was employed to perform image reconstruction based on the edge maps and additional reference pixels. Although the objectives of human vision and machine vi- sion are different, human-machine collaborative coding can beneﬁt from exploring the signiﬁcant overlap that exists be- tween these two domains. One important similarity is that human vision is mostly hierarchical. For example, an image of a study room can be described by humans using annota- tions like - over the ﬂoor carpet (global context) there is a desk (local detail) and a laptop (ﬁner detail) is on the desk; on the wall (global context) a clock (local detail) is mounted, etc. Machine vision tasks like object detection can also be mod- eled in this kind of hierarchical way. For instance, in [7] a method is proposed for performing hierarchical object detec- tion in images through deep reinforcement learning [8]. Us- ing scale-space theory [9] and deep reinforcement learning, another work [10] explored optimal search strategies for lo- cating anatomical structures, based on image information at multiple scales. Murshed et al. proposed the hierarchical cuboidal par- titioning of image data (CuPID) algorithm in [11]. In Cu- PID framework, an entire image is initially partitioned into two rectangular shaped regions, known as cuboids, by ﬁnd- ing a hyperplane orthogonal to one of the axes. A greedy optimization heuristic, equipped with sum of the information entropy in the split-pair cuboids as the objective function, is employed to ﬁnd the minimizing hyperplane. This ensures that the obtained split-pair cuboids are maximum dissimilar in terms of image pixel intensity distribution. Next, each cuboid is recursively partitioned by solving the same greedy opti- mization problem. It was shown that cuboids have excellent global commonality modeling capabilities, are object centric and computationally efﬁcient [12–14]. The aforementioned properties of cuboids were leveraged on to traditional video coding in the work of [15]; wherein a reference frame is generated using cuboidal approximation of the target frame and incorporating this additional reference frame into a modiﬁed HEVC encoder outperformed the rate- distortion performance of a standalone HEVC encoder. The obtained gain in delta PSNR and savings in bit rate can be at- tributed to the fact that HEVC employs a partitioning scheme that begins at a rigid ﬁxed size of 6464pixels level and during this process does not take into account the structural properties of the scene need be coded. This sub-optimal par- titioning was improved by incorporating cuboids, which are estimated considering the entire frame’s pixel intensity distri- bution and therefore more object-centric. Building on the work [15], in this paper we investigate the applicability of cuboidal features in video coding target- Fig. 1 . The estimated cuboid map over the current frame, C. This frame is part of the vehicle detection dataset in [16]. The cuboid map, consisting of 70cuboids, is determined by the CuPID algorithm [15]. Each cuboid’s coverage is depicted with white boundary pixels. ing machines. Based on the CuPID algorithm, (i)features are extracted from a given video frame for the machine vi- sion task of object detection. These features costs less bits to encode. Addition to this, (ii)it is possible to reconstruct a full-resolution frame from the obtained feature descriptors alone; unlike the approach in [6] that require additional refer- ence pixels along with edge based feature descriptors in this regard. The reconstructed frame is a coarse representation of the original frame and is capable of preserving important structural properties of the original frame. Finally, (iii)the object detection performance of the cuboidal descriptors set are put to test across varying number of cuboids. The rest of the paper is organized as follows: in section II, we brieﬂy describe the cuboidal feature extraction process from a given video frame and the reconstruction process of a full-resolution frame from those estimated feature descrip- tors. Section III, describes the performance of the cuboidal features over an object detection task. Finally, in section IV , we present our conclusions. 2. DETERMINATION OF CUBOIDAL FEATURES FROM THE CURRENT FRAME USING CUPID Given the original uncompressed current frame C, of resolu- tionXYpixels, the CuPID algorithm hierarchically parti- tions it into ncuboids C(1); C(2); : : : ; C(n), where nis a user- deﬁned number of tiles. The ﬁrst partitioning can produce two half-cuboids C(1) iandC(2) iof size iYand(Xi)Ypix- els respectively. This is achieved by selecting a vertical line x=i+0:5overCfrom the set of lines i2f1;2; : : : ; X1g. Alternatively, Ccould be split into two half-cuboids C(1) x1+j andC(2) x1+jof size XjandX(Yj)pixels, respec- tively by selecting a horizontal line y=j+ 0:5from the set (a)Rcoframe ( n= 100 ):R(n=100) co (b)Rcoframe ( n= 200 ):R(n=200) co (c)Rcoframe ( n= 300 ):R(n=300) co Fig. 2 . Different coarse representations of the current frame C. Each one is a Rcoframe, obtained by varying the number of cuboids, n. of lines j2f1;2; : : : ; Y1g. The ﬁrst optimal split sofCfrom the possible num- ber of splits (X1) + ( Y1) = X+Y2is obtained by solving a greedy optimization [17] problem whose objec- tive function is taken to be a measure of dissimilarity between the candidate cuboid split-pair [15]. After that each obtained half-cuboid is further split recursively until the total number of cuboids obtained from Cbecomes equal to the user input n. Fig. 1 shows an example cuboid map for the frame C. It can be observed that the estimated cuboids are fairly homoge- neous in terms of pixel intensity distribution and they occupy arbitrary shaped rectangular regions. Classes like road, sky and trees are represented using separate larger cuboids and important cues for vehicle detection are also preserved. The estimated cuboid map can be reconstructed at a de- coder side from the optimal partitioning indices fsgn1 i=1 alone. That means, for example, if the ﬁrst optimal partition- ing, which split the entire image into 2cuboids, took place at height (or width) h, the number hneed be encoded and so on. These indices are encoded and augmented into a bit- stream in the way described in [15]. Next, for each obtained cuboid, C(i), from the cuboid map, a feature descriptor, m(i), is computed. This value is taken to be the mean pixel in- tensity (for each image channel of C), considering all the pixels pand their intensities p(x; y)within the coverage of the corresponding cuboid. m(i) j=P p2C(i)pj(x; y) jC(i)j(1) where jrepresents different color component channels of the frame C. The feature descriptors m(i)2Zn are encoded in the way described in [15] and are also part of the bitstream. Having decoded the optimal partitioning indices and the feature descriptors from the communicated bitstream, it is possible to generate a full-resolution frame, Rco. This frame Rcoprovides a coarse representation of the current frame, Csince it is created by replacing every encompassing pixel p2C(i)intensity by the associated cuboidal feature descrip-Table 1 . Savings in bits and complexity requirements from the cuboidal feature descriptors at different scale (varying number of cuboids, n) over HEVC reference. It also reports the corresponding reconstructed Rcoframes PSNR with re- spect to the current frame C. n 100 200 300 Savings in bits:(bcbco) bc71% 38% 4:05% Savings in comp. time:(tctco) tc72:41% 41:38% 3:45% Y-PSNR (in dB) 32:20 32:69 33:16 torm(i). Fig. 3 shows examples of this coarse representa- tion frame Rco. It can be observed that this low-frequency representation Rcoattempt to capture important structure in- formation about the scene and the degree of detailed infor- mation it can communicate tends to increase with growing number of cuboids, n. Table 1 reports the bits requirements, computational complexity and quality (in terms of PSNR) of these Rcoframes. Here, bcandtcstand for bits and computa- tional time (in seconds) required for an HEVC encoder [18] to intra-code (with QP 47) the frame C. The employed system conﬁguration is: Intel Core i7-8650U CPU@1.90 GHz, 32.0 GB RAM. The maximum PSNR is achieved from the frame R(n=300) co and this frame can be encoded at less than 4%bits requirements that of HEVC. Moreover, the encoding process is3:45% times faster compared to the HEVC reference soft- ware [18]. 3. OBJECT DETECTION USING THE CUBOIDAL FEATURE DESCRIPTORS The cuboidal feature descriptors attempt to capture important structural properties of the image at comparatively lower bit and computational complexity requirements. In this work, we propose to utilize them for a machine vision task, in particular theRcoframes are employed in a vehicle detection problem. For the addressed vehicle detection problem, the object detec- (a) HEVC coded test frame with detection conﬁdence scores: f0:79;0:80;0:73;0:81g respec- tively. (b)R(n=300) co test frame with detection conﬁdence scores: f0:78;0:68;0:78;0:86g respec- tively. Fig. 3 . The trained vehicle detector’s performance over differ- ent test frame. (From left to right) HEVC intra-coded version and coarse representation RcoofCrespectively. The detector managed to detect same number of vehicles in the Rcoframe with various conﬁdence scores. Table 2 . Average precision (AP) values of the trained vehicle detector over different representations of the same test set [16]. It also reports the bit rate required to code the test set frames. Testset AP Bit rate (Kbps) HEVC coded 0:9366 110:63 R(n=100) co 0:8158 31:91 R(n=200) co 0:9083 67:53 R(n=300) co 0.9462 102:74 tor You only look once (YOLO) v2 [19] is used. YOLO v2 is a deep learning object detection framework that uses a convo- lutional neural network (CNN) [20] for detection task. Using unoccluded RGB images of the front, rear, left, and right sides of cars on a highway scene, the detector is trained. The CNN used with the vehicle detector, uses a modiﬁed version of the MobileNet-v2 network architecture [21]. 60% of the data is used for training, 10% for validation, and the rest 30% is used for testing the trained detector. Once the detector has been trained, it is used to detect ve- hicles over (i)HEVC coded versions of the original test video frames (refers to Fig. 3(a)) and (ii)cuboidal descriptor ori- ented coarse representation frames ( Rco) of the original test frames (refers to Fig. 3(b)). As can be seen, the detector man- aged to detect same number of vehicles in the R(n=300) co frame like its HEVC coded counterpart. Moreover, for 2vehicles the detector’s detection performance (in terms of higher con- ﬁdence score) improved when compared to the HEVC coded frame. However, for the second vehicle, the conﬁdence score is lower in the case of R(n=300) co frame. This is because that vehicle in the scene is partly occluded by another vehicle and 00.2 0.4 0.6 0.8 10.880.90.920.940.960.981 HEVC RCo(n=100) RCo(n=200) RCo(n=300)Fig. 4 . Precision recall performance of the trained vehicle detector over different representations of the same test set. therefore its cuboidal representation lacks in some structural information. Table 2 reports the performance of the detector over these different test data in the form of average precision. It can be noted that the detector produced the maximum average pre- cision of 0:9462 overR(n=300) co test frames. This result is achieved with a bit rate requirements that of HEVC; in par- ticular a bit rate savings of around 7%is obtained. Bit rate savings improve further (a savings of 39%) if the R(n=200) co test frames are used instead. However, in this case the av- erage precision decreases ( 0:0283 ) compared to the HEVC coded test set. Fig. 4 shows the precision-recall curves for the employed test sets. At almost 80% recall, the precision is more over the test sets R(n=200) co andR(n=300) co , compared to the HEVC coded test set. And these cuboidal descriptor oriented test sets take less bits to encode than HEVC. 4. CONCLUSION Despite the difference in objectives, there exists similarities between human vision and machine vision. A collaborative coding paradigm is required for many use cases. The cuboidal descriptors oriented representation of the current frames were shown to be beneﬁcial in traditional video coding. Leverag- ing on their properties like object oriented, scene structural awareness, compact representation and computational sim- plicity, in this paper, cuboidal descriptors are employed in a machine vision task. Experimental results show that the addressed vehicle detection problem can be solved more ac- curately (average precision improved by 0:01) if cuboidal de- scriptors are used. Along with this a savings in bit rate of around 7%is reported over a HEVC reference. 5.