SYSTEMS AND METHODS FOR REDUCING DISTORTION IN END-TO-END FEATURE COMPRESSION IN CODING OF MULTI-DIMENSIONAL DATA

A device may be configured to reduce distortion in compressed feature data according to one or more of the techniques described herein. In one example, a bitstream including compressed feature data may be decoded according to the video coding standard. A quantization parameter or target bit rate and a picture type may be determined for a decoded picture corresponding to a channel. A distortion reduction engine may be selected based on the quantization parameter and the picture type. The distortion reduction engine may be applied to reduce distortion.

TECHNICAL FIELD

This disclosure relates to coding multi-dimensional data and more particularly to techniques for compression of feature data in an end-to-end network.

BACKGROUND

Digital video and audio capabilities can be incorporated into a wide range of devices, including digital televisions, computers, digital recording devices, digital media players, video gaming devices, smartphones, medical imaging devices, surveillance systems, tracking and monitoring systems, and the like. Digital video and audio can be represented as a set of arrays. Data represented as a set of arrays may be referred to as multi-dimensional data. For example, a picture in digital video can be represented as a set of two-dimensional arrays of sample values. That is, for example, a video resolution provides a width and height dimension of an array of sample values and each component of a color space provides a number of two-dimensional arrays in the set. Further, the number of pictures in a sequence of digital video provides another dimension of data. For example, one second of 60 Hz video at 1080p resolution having three color components could correspond to four dimensions of data values, i.e., the number of samples may be represented as follows: 1920×1080×3×60. Thus, digital video and images are examples of multi-dimensional data. It should be noted that digital video may be represented using additional and/or alternative dimensions (e.g., number of layers, number of views/channels, etc.).

Digital video may be coded according to a video coding standard. Video coding standards define the format of a compliant bitstream encapsulating coded video data. A compliant bitstream is a data structure that may be received and decoded by a video decoding device to generate reconstructed video data. Typically, the reconstructed video data is intended for human-consumption (i.e., viewing on a display). Examples of video coding standards include ISO/IEC MPEG-4 Visual and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), High-Efficiency Video Coding (HEVC), and Versatile video coding (VVC). HEVC is described in High Efficiency Video Coding, Rec. ITU-T H.265, November 2019, which is referred to herein as ITU-T H.265. VVC is described in Versatile Video Coding, Rec. ITU-T H.266, April 2022, which is referred to herein as ITU-T H.266.

Video coding standards may utilize video compression techniques. Video compression techniques reduce data requirements for storing and/or transmitting video data by exploiting the inherent redundancies in a video sequence. Video compression techniques typically sub-divide a video sequence into successively smaller portions (i.e., groups of pictures within a video sequence, a picture within a group of pictures, regions within a picture, sub-regions within a region, etc.) and utilize intra prediction coding techniques (e.g., spatial prediction techniques within a picture) and inter prediction techniques (i.e., inter-picture techniques (temporal)) to generate difference values between a unit of video data to be coded and a reference unit of video data. The difference values may be referred to as residual data. Syntax elements may relate residual data and a reference coding unit (e.g., intra-prediction mode indices and motion information). Residual data and syntax elements may be entropy coded. Entropy encoded residual data and syntax elements may be included in data structures forming a compliant bitstream.

SUMMARY

In general, this disclosure describes various techniques for coding multi-dimensional data, which may be referred to as a multi-dimensional data set (MDDS) and may include, for example, video data, audio data, and the like. It should be noted that in addition to reducing the data requirements for providing multi-dimensional data for human consumption, the techniques for coding of multi-dimensional data described herein may be useful for other applications. For example, the techniques described herein may be useful for so-called machine consumption. That is, for example, in the case of surveillance, it may be useful for a monitoring application running on a central server to be able to quickly identify and track an object from any of a number video feeds. In this case, it is not necessary that the coded video data is capable of being reconstructed to a human consumable form, but only capable of enabling an object to be identified. Object detection is an example of a so-called machine task. As described in further detail below, object detection, segmentation and/or tracking (i.e., object recognition tasks) typically involve receiving an image (e.g., a single image or an image included in a video sequence), generating feature data corresponding to the image, analyzing the feature data, and generating inference data, where inference data may indicate types of objects and spatial locations of objects within the image. Spatial locations of objects within an image may be specified by a bounding box having a spatial coordinate (e.g., x,y) and a size (e.g., a height and a width). This disclosure describes techniques for compressing feature data. In particular, this disclosure describes techniques for mitigating distortion in an end-to-end feature compression network. The techniques described in this disclosure may be particularly useful for allowing machine tasks to be distributed across a communication network. For example, in some applications, an acquisition device (e.g., a video camera and accompanying hardware) may have power and/or computational constraints. In this case, generation of feature data could be optimized for the capabilities at the acquisition device, but, the analysis and inference may be better suited to be performed at one or more devices with additional capabilities distributed across a network. In this case, compression of the feature set may facilitate efficient distribution (e.g., reduced bandwidth and/or latency) of object recognition tasks. It should be noted, as described in further detail below, inference data (e.g., spatial locations of objects within an image) may be used to optimize encoding of video data. (e.g., adjust coding parameters to improve relative image quality in regions where objects of interest are present and the like). Further, a video encoding device that utilizes inference data may be located at a distinct location from acquisition device. For example, a distribution network may include multiple distribution servers (at various physical locations) that perform compression and distribution of acquired video.

It should be noted that as used herein the term typical video coding standard or typical video coding may refer to a video coding standard utilizing one or more of the following video compression techniques: video partitioning techniques, intra prediction techniques, inter prediction techniques, residual transformation techniques, reconstructed video filtering techniques, and/or entropy coding techniques for residual data and syntax elements. For example, the term typical video coding standard may refer to any of ITU-T H.264, ITU-T H.265, ITU-TH.266, and the like, individually or collectively. Further, it should be noted that incorporation by reference of documents herein is for descriptive purposes and should not be construed to limit or create ambiguity with respect to terms used herein. For example, in the case where an incorporated reference provides a different definition of a term than another incorporated reference and/or as the term is used herein, the term should be interpreted in a manner that broadly includes each respective definition and/or in a manner that includes each of the particular definitions in the alternative.

In one example, a method of mitigating distortion in compressed feature data, comprises receiving a bitstream including compressed feature data further coded according a video coding standard, wherein the compressed feature data is a tensor with channel, height, and weight dimensions, decoding the bitstream according to the video coding standard, such that a decoded picture corresponds to a channel, determining a quantization parameter and a picture type for a decoded picture, selecting a distortion reduction engine based on the quantization parameter and the picture type, and applying the distortion reduction engine to the decoded picture.

In one example, a device comprises one or more processors configured to receive a bitstream including compressed feature data further coded according a video coding standard, wherein the compressed feature data is a tensor with channel, height, and weight dimensions, decode the bitstream according to the video coding standard, such that a decoded picture corresponds to a channel, determine a quantization parameter and a picture type for a decoded picture, select a distortion reduction engine based on the quantization parameter and the picture type, and apply the distortion reduction engine to the decoded picture.

In one example, a non-transitory computer-readable storage medium comprises instructions stored thereon that, when executed, cause one or more processors of a device to receive a bitstream including compressed feature data further coded according a video coding standard, wherein the compressed feature data is a tensor with channel, height, and weight dimensions, decode the bitstream according to the video coding standard, such that a decoded picture corresponds to a channel, determine a quantization parameter and a picture type for a decoded picture, select a distortion reduction engine based on the quantization parameter and the picture type, and apply the distortion reduction engine to the decoded picture.

In one example, an apparatus comprises means for receiving a bitstream including compressed feature data further coded according a video coding standard, wherein the compressed feature data is a tensor with channel, height, and weight dimensions, means for decoding the bitstream according to the video coding standard, such that a decoded picture corresponds to a channel, means for determining a quantization parameter and a picture type for a decoded picture, means for selecting a distortion reduction engine based on the quantization parameter and the picture type, and means for applying the distortion reduction engine to the decoded picture.

DETAILED DESCRIPTION

Video content includes video sequences comprised of a series of frames (or pictures). A series of frames may also be referred to as a group of pictures (GOP). For coding purposes, each video frame or picture may divided into one or more regions, which may be referred to as video blocks. As used herein, the term video block may generally refer to an area of a picture that may be coded (e.g., according to a prediction technique), sub-divisions thereof, and/or corresponding structures. Further, the term current video block may refer to an area of a picture presently being encoded or decoded. A video block may be defined as an array of sample values. It should be noted that in some cases pixel values may be described as including sample values for respective components of video data, which may also be referred to as color components, (e.g., luma (Y) and chroma (Cb and Cr) components or red, green, and blue components (RGB)). It should be noted that in some cases, the terms pixel value and sample value are used interchangeably. Further, in some cases, a pixel or sample may be referred to as a pel. A video sampling format, which may also be referred to as a chroma format, may define the number of chroma samples included in a video block with respect to the number of luma samples included in a video block. For example, for the 4:2:0 sampling format, the sampling rate for the luma component is twice that of the chroma components for both the horizontal and vertical directions.

Digital video data including one or more video sequences is an example of multi-dimensional data.FIG.1is a conceptual diagram illustrating video data represented as multi-dimensional data. Referring toFIG.1, the video data includes a respective group of pictures for two layers. For example, each layer may be a view (e.g., a left and a right view) or a temporal layer of video. As illustrated inFIG.1, each layer includes three components of video data (e.g., RGB, BGR, YCbCr, etc.) and each component includes four pictures having width (W)×height (H) sample values (e.g., 1920×1080, 1280×720, etc.). Thus, in the example illustrated inFIG.1, there are 24 W×H arrays of sample values and each array of sample values may be described as a two-dimensional data set. Further, the arrays may be grouped into sets according to one or more other dimensions (e.g., channels, components, and/or a temporal sequence of frames). For example, component 1 of the GOP of layer 1 may be described as a three-dimensional data set (i.e., W×H×Number of pictures), all of the components of the GOP of layer 1 may be described as a four-dimensional data set (i.e., W×H×Number of pictures×Number of components), and all of the components of the GOP of layer 1 and the GOP of layer 2 may described as a five-dimensional data set (i.e., W×H×Number of pictures×Number of components×Number of layers).

Multi-layer video coding enables a video presentation to be decoded/displayed as a presentation corresponding to a base layer of video data and decoded/displayed as one or more additional presentations corresponding to enhancement layers of video data. For example, a base layer may enable a video presentation having a basic level of quality (e.g., a High Definition rendering and/or a 30 Hz frame rate) to be presented and an enhancement layer may enable a video presentation having an enhanced level of quality (e.g., an Ultra High Definition rendering and/or a 60 Hz frame rate) to be presented. An enhancement layer may be coded by referencing a base layer. That is, for example, a picture in an enhancement layer may be coded (e.g., using inter-layer prediction techniques) by referencing one or more pictures (including scaled versions thereof) in a base layer. It should be noted that layers may also be coded independent of each other. In this case, there may not be inter-layer prediction between two layers. A sub-bitstream extraction process may be used to only decode and display a particular layer of video. Sub-bitstream extraction may refer to a process where a device receiving a compliant or conforming bitstream forms a new compliant or conforming bitstream by discarding and/or modifying data in the received bitstream.

A video encoder operating according to a typical video coding standard may perform predictive encoding on video blocks and sub-divisions thereof. For example, pictures may be segmented into video blocks which are the largest array of video data that may be predictively encoded and the largest arrays of video data may be further partitioned into nodes. For example, in ITU-T H.265, coding tree units (CTUs) are partitioned into coding units (CUs) according to a quadtree (QT) partitioning structure. A node may be associated with a prediction unit data structure and a residual unit data structure having their roots at the node. A prediction unit data structure may include intra prediction data (e.g., intra prediction mode syntax elements) or inter prediction data (e.g., motion data syntax elements) that may be used to produce reference and/or predicted sample values for the node. For intra prediction coding, a defined intra prediction mode may specify the location of reference samples within a picture. For inter prediction coding, a reference picture may be determined and a motion vector (MV) may identify samples in the reference picture that are used to generate a prediction for a current video block. For example, a current video block may be predicted using reference sample values located in one or more previously coded picture(s) and a motion vector may be used to indicate the location of the reference block relative to the current video block. A motion vector may describe, for example, a horizontal displacement component of the motion vector (i.e., MVx), a vertical displacement component of the motion vector (i.e., MVy), and a resolution for the motion vector (i.e., e.g., pixel precision). Previously decoded pictures may be organized into one or more to reference pictures lists and identified using a reference picture index value. Further, in inter prediction coding, uni-prediction refers to generating a prediction using sample values from a single reference picture and bi-prediction refers to generating a prediction using respective sample values from two reference pictures. That is, in uni-prediction, a single reference picture is used to generate a prediction for a current video block and in bi-prediction, a first reference picture and a second reference picture may be used to generate a prediction for a current video block. In bi-prediction, respective sample values may be combined (e.g., added, rounded, and clipped, or averaged according to weights) to generate a prediction. Further, a typical video coding standard may support various modes of motion vector prediction. Motion vector prediction enables the value of a motion vector for a current video block to be derived based on another motion vector. For example, a set of candidate blocks having associated motion information may be derived from spatial neighboring blocks to the current video block and a motion vector for the current video block may be derived from a motion vector associated with one of the candidate blocks.

As described above, intra prediction data or inter prediction data may be used to produce reference sample values for a current block of sample values. The difference between sample values included in a current block and associated reference samples may be referred to as residual data. Residual data may include respective arrays of difference values corresponding to each component of video data. Residual data may initially be calculated in the pixel domain. That is, from subtracting sample amplitude values for a component of video data. A transform, such as, a discrete cosine transform (DCT), a discrete sine transform (DST), an integer transform, a wavelet transform, or a conceptually similar transform, may be applied to an array of sample difference values to generate transform coefficients. It should be noted that in some cases, a core transform and a subsequent secondary transforms may be applied to generate transform coefficients. A quantization process may be performed on transform coefficients or residual sample values directly (e.g., in the case, of palette coding quantization). Quantization approximates transform coefficients (or residual sample values) by amplitudes restricted to a set of specified values. Quantization essentially scales transform coefficients in order to vary the amount of data required to represent a group of transform coefficients. Quantization may include division of transform coefficients (or values resulting from the addition of an offset value to transform coefficients) by a quantization scaling factor and any associated rounding functions (e.g., rounding to the nearest integer). Quantized transform coefficients may be referred to as coefficient level values. Inverse quantization (or “dequantization”) may include multiplication of coefficient level values by the quantization scaling factor, and any reciprocal rounding and/or offset addition operations. It should be noted that as used herein the term quantization process in some instances may refer to generating level values (or the like) in some instances and recovering transform coefficients (or the like) in some instances. That is, a quantization process may refer to quantization in some cases and inverse quantization (which also may be referred to as dequantization) in some cases. Further, it should be noted that although in some of the examples quantization processes are described with respect to arithmetic operations associated with decimal notation, such descriptions are for illustrative purposes and should not be construed as limiting. For example, the techniques described herein may be implemented in a device using binary operations and the like. For example, multiplication and division operations described herein may be implemented using bit shifting operations and the like.

Quantized transform coefficients and syntax elements (e.g., syntax elements indicating a prediction for a video block) may be entropy coded according to an entropy coding technique. An entropy coding process includes coding values of syntax elements using lossless data compression algorithms. Examples of entropy coding techniques include content adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), probability interval partitioning entropy coding (PIPE), and the like. Entropy encoded quantized transform coefficients and corresponding entropy encoded syntax elements may form a compliant bitstream that can be used to reproduce video data at a video decoder. An entropy coding process, for example, CABAC, as implemented in ITU-T H.265 or ITU-T H.266 may include performing a binarization on syntax elements. Binarization refers to the process of converting a value of a syntax element into a series of one or more bits. These bits may be referred to as “bins.” Binarization may include one or a combination of the following coding techniques: fixed length coding, unary coding, truncated unary coding, truncated Rice coding, Golomb coding, k-th order exponential Golomb coding, and Golomb-Rice coding. For example, binarization may include representing the integer value of 5 for a syntax element as 00000101 using an 8-bit fixed length binarization technique or representing the integer value of 5 as 11110 using a unary coding binarization technique. As used herein, each of the terms fixed length coding, unary coding, truncated unary coding, truncated Rice coding, Golomb coding, k-th order exponential Golomb coding, and Golomb-Rice coding may refer to general implementations of these techniques and/or more specific implementations of these coding techniques. For example, a Golomb-Rice coding implementation may be specifically defined according to a video coding standard. In the example of CABAC, for a particular bin, a context may provide a most probable state (MPS) value for the bin (i.e., an MPS for a bin is one of 0 or 1) and a probability value of the bin being the MPS or the least probably state (LPS). For example, a context may indicate, that the MPS of a bin is 0 and the probability of the bin being 1 is 0.3. It should be noted that a context may be determined based on values of previously coded bins including bins in a current syntax element and previously coded syntax elements.

FIGS.2A-2Bare conceptual diagrams illustrating examples of coding a block of video data. As illustrated inFIG.2A, a current block of video data (e.g., an area of a picture corresponding to a video component) is encoded by generating a residual by subtracting a set of prediction values from the current block of video data, performing a transformation on the residual, and quantizing the transform coefficients to generate level values. As illustrated inFIG.2B, the current block of video data is decoded by performing inverse quantization on level values, performing an inverse transform, and adding a set of prediction values to the resulting residual. It should be noted that in the examples inFIGS.2A-2B, the sample values of the reconstructed block differs from the sample values of the current video block that is encoded. In particular,FIG.2Billustrates a reconstruction error which is the difference between the current block and the reconstructed block. In this manner, coding may be said to be lossy. However, the difference in sample values may be considered minimally perceptible to a viewer of the reconstructed video. That is, the reconstructed video may be said to be fit for human-consumption. However, it should be noted that in some cases, coding video data on a block-by-block basis may result in artifacts (e.g., so-called blocking artifacts, banding artifacts, etc.) For example, blocking artifacts may cause coding block boundaries of reconstructed video data to be visually perceptible to a user. In this manner, reconstructed sample values may be modified to minimize a reconstruction error and/or minimize perceivable artifacts introduced by a video coding process. Such modifications may generally be referred to as filtering. It should be noted that filtering may occur as part of an in-loop filtering process or a post-loop filtering process. For an in-loop filtering process, the resulting sample values of a filtering process may be used for further reference and for a post-loop filtering process the resulting sample values of a filtering process are merely output as part of the decoding process (e.g., not used for subsequent coding).

Typical video coding standards may utilize so-called deblocking (or de-blocking), which refers to a process of smoothing the boundaries of neighboring reconstructed video blocks (i.e., making boundaries less perceptible to a viewer) as part of an in-loop filtering process. In addition to applying a deblocking filter as part of an in-loop filtering process, a typical video coding standard may utilized Sample Adaptive Offset (SAO), where SAO is a process that modifies the deblocked sample values in a region by conditionally adding an offset value. Further, a typical video coding standard may utilized one or more additional filtering techniques. For example, in ITU-T H.266, a so-called adaptive loop filter (ALF) may be applied.

As described above, for coding purposes, each video frame or picture may divided into one or more regions, which may be referred to as video blocks. It should be noted that in some cases, other overlapping and/or independent regions may be defined. For example, according to typical video coding standards, each video picture may be partitioned to include one or more slices and further partitioned to include one or more tiles. With respect to ITU-T H.266, slices are required to consist of an integer number of complete tiles or an integer number of consecutive complete CTU rows within a tile, instead of only being required to consist of an integer number of CTUs. Thus, in ITU-T H.266, a picture may include a single tile, where the single tile is contained within a single slice or a picture may include multiple tiles where the multiple tiles (or CTU rows thereof) may be contained within one or more slices. Further, it should be noted that ITU-T H.266 provides where a picture may be partitioned into subpictures, where a subpicture is a rectangular region of a CTUs within a picture. The top-left CTU of a subpicture may be located at any CTU position within a picture with subpictures being constrained to include one or more slices Thus, unlike a tile, a subpicture is not necessarily limited to a particular row and column position. It should be noted that subpictures may be useful for encapsulating regions of interest within a picture and a sub-bitstream extraction process may be used to only decode and display a particular region of interest. That is, a bitstream of coded video data may include a sequence of network abstraction layer (NAL) units, where a NAL unit encapsulates coded video data. (i.e., video data corresponding to a slice of picture) or a NAL unit encapsulates metadata used for decoding video data (e.g., a parameter set) and a sub-bitstream extraction process forms a new bitstream by removing one or more NAL units from a bitstream.

FIG.3is a conceptual diagram illustrating an example of a picture within a group of pictures partitioned according to tiles, slices, and subpictures and the corresponding coded video data encapsulated into NAL units. It should be noted that the techniques described herein may be applicable to tiles, slices, subpictures, sub-divisions thereof and/or equivalent structures thereto. That is, the techniques described herein may be generally applicable regardless of how a picture is partitioned into regions. In the example illustrated inFIG.3, Pic3is illustrated as including 16 tiles (i.e., Tile0to Tile15) and three slices (i.e., Slice0to Slice2). In the example illustrated inFIG.3, Slice0includes four tiles (i.e., Tile0to Tile3), Slice1includes eight tiles (i.e., Tile4to Tile11), and Slice2includes four tiles (i.e., Tile12to Tile15). Further, as illustrated in the example ofFIG.3, Pic3includes two subpictures (i.e., Subpicture0and Subpicture1), where Subpicture0includes Slice0and Slice1and where Subpicture1includes Slice2. As described above, subpictures may be useful for encapsulating regions of interest within a picture and a sub-bitstream extraction process may be used in order to selectively decode (and display) a region interest. For example, referring toFIG.3, Subpicture0may corresponding to an action portion of a sporting event presentation (e.g., a view of the field) and Subpicture1may corresponding to a scrolling banner displayed during the sporting event presentation. By organizing a picture into subpictures in this manner, a viewer may be able to disable the display of the scrolling banner. That is, through a sub-bitstream extraction process Slice2NAL unit may be removed from a bitstream (and thus not decoded and/or displayed) and Slice0NAL unit and Slice1NAL unit may be decoded and displayed.

As described above, for inter prediction coding, reference samples in a previously coded picture are used for coding video blocks in a current picture. Previously coded pictures which are available for use as reference when coding a current picture are referred as reference pictures. It should be noted that the decoding order does not necessary correspond with the picture output order, i.e., the temporal order of pictures in a video sequence. According to a typical video coding standard, when a picture is decoded it may be stored to a decoded picture buffer (DPB) (which may be referred to as frame buffer, a reference buffer, a reference picture buffer, or the like). For example, referring toFIG.3, Pic2is illustrated as referencing Pic1. Similarly, Pic3is illustrated as referencing Pic0. With respect toFIG.3, assuming the picture number corresponds to the decoding order, the DPB would be populated as follows: after decoding Pic0, the DPB would include {Pic0}; at the onset of decoding Pic1, the DPB would include {Pic0}; after decoding Pic1, the DPB would include {Pic0. Pic1}; at the onset of decoding Pic2, the DPB would include {Pic0. Pic1}. Pic2would then be decoded with reference to Pic1and after decoding Pic2, the DPB would include {Pic0, Pic1, Pic2}. At the onset of decoding Pic3, pictures Pic0and Pic1would be marked for removal from the DPB, as they are not needed for decoding Pic3(or any subsequent pictures, not shown) and assuming Pic1and Pic2have been output, the DPB would be updated to include (Pic0). Pic3would then be decoded by referencing Pic0. The process of marking pictures for removal from a DPB may be referred to as reference picture set (RPS) management.

FIG.4is a block diagram illustrating an example of a system that may be configured to code (i.e., encode and/or decode) a multi-dimensional data set (MDDS) according to one or more techniques of this disclosure. It should be noted that in some cases an MDDS may be referred to as a tensor. System100represents an example of a system that may encapsulate coded data according to one or more techniques of this disclosure. As illustrated inFIG.4, system100includes source device102, communications medium110, and destination device120. In the example illustrated inFIG.4, source device102may include any device configured to encode multi-dimensional data and transmit encoded data to communications medium110. Destination device120may include any device configured to receive encoded data via communications medium110and to decode encoded data. Source device102and/or destination device120may include computing devices equipped for wired and/or wireless communications and may include, for example, set top boxes, digital video recorders, televisions, computers, gaming consoles, medical imaging devices, and mobile devices, including, for example, smartphones.

Communications medium110may include any combination of wireless and wired communication media, and/or storage devices. Communications medium110may include coaxial cables, fiber optic cables, twisted pair cables, wireless transmitters and receivers, routers, switches, repeaters, base stations, or any other equipment that may be useful to facilitate communications between various devices and sites. Communications medium110may include one or more networks. For example, communications medium110may include a network configured to enable access to the World Wide Web, for example, the Internet. A network may operate according to a combination of one or more telecommunication protocols. Telecommunications protocols may include proprietary aspects and/or may include standardized telecommunication protocols. Examples of standardized telecommunications protocols include Digital Video Broadcasting (DVB) standards, Advanced Television Systems Committee (ATSC) standards, Integrated Services Digital Broadcasting (ISDB) standards, Data Over Cable Service Interface Specification (DOCSIS) standards, Global System Mobile Communications (GSM) standards, code division multiple access (CDMA) standards, 3rd Generation Partnership Project (3GPP) standards, European Telecommunications Standards Institute (ETSI) standards, Internet Protocol (IP) standards, Wireless Application Protocol (WAP) standards, and Institute of Electrical and Electronics Engineers (IEEE) standards.

Storage devices may include any type of device or storage medium capable of storing data. A storage medium may include a tangible or non-transitory computer-readable media. A computer readable medium may include optical discs, flash memory, magnetic memory, or any other suitable digital storage media. In some examples, a memory device or portions thereof may be described as non-volatile memory and in other examples portions of memory devices may be described as volatile memory. Examples of volatile memories may include random access memories (RAM), dynamic random access memories (DRAM), and static random access memories (SRAM). Examples of non-volatile memories may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage device(s) may include memory cards (e.g., a Secure Digital (SD) memory card), internal/external hard disk drives, and/or internal/external solid state drives. Data may be stored on a storage device according to a defined file format.

Referring again toFIG.4, source device102includes data source104, data encoder106, coded data encapsulator107, and interface108. Data source104may include any device configured to capture and/or store multi-dimensional data. For example, data source104may include a video camera and a storage device operably coupled thereto. Data encoder106may include any device configured to receive multi-dimensional data and generate a bitstream representing the data. A bitstream may refer to a general bitstream (i.e., binary values representing coded data) or a compliant bitstream where aspects of a compliant bitstream may be defined according to a standard, e.g., a video coding standard. Coded data encapsulator107may receive a bitstream and encapsulate the bitstream for purposes of storage and/or transmission. For example, coded data encapsulator107may encapsulate bitstream according to a file format. It should be noted that coded data encapsulator107need not necessarily be located in the same physical device as data encoder106. For example, functions described as being performed by data source104, data encoder106and/or coded data encapsulator107may be distributed among devices in a computing system (e.g., at distinct server locations, etc.). Interface108may include any device configured to receive data generated by coded data encapsulator107and transmit and/or store the data to a communications medium. Interface108may include a network interface card, such as an Ethernet card, and may include an optical transceiver, a radio frequency transceiver, or any other type of device that can send and/or receive information. Further, interface108may include a computer system interface that may enable a file to be stored on a storage device. For example, interface108may include a chipset supporting Peripheral Component Interconnect (PCI) and Peripheral Component Interconnect Express (PCIe) bus protocols, proprietary bus protocols, Universal Serial Bus (USB) protocols, IIC, or any other logical and physical structure that may be used to interconnect peer devices.

Referring again toFIG.4, destination device120includes interface122, coded data decapsulator123, data decoder124, and output126. Interface122may include any device configured to receive data from a communications medium. Interface122may include a network interface card, such as an Ethernet card, and may include an optical transceiver, a radio frequency transceiver, or any other type of device that can receive and/or send information. Further, interface122may include a computer system interface enabling a compliant video bitstream to be retrieved from a storage device. For example, interface122may include a chipset supporting PCI and PCIe bus protocols, proprietary bus protocols, USB protocols, I2C, or any other logical and physical structure that may be used to interconnect peer devices. Coded data decapsulator123may be configured to receive and extract a bitstream from an encapsulated format. For example, in the case of video coded according to a typical video coding standard stored on physical medium according to a defined file format, coded data decapsulator123may be configured to extract a compliant bitstream from the file. Data decoder124may include any device configured to receive a bitstream and/or acceptable variations thereof and reproduce multi-dimensional data therefrom. Reproduced multi-dimensional data may then be received by output126. For example, in the case of video, output126may include a display device configured to display video data. Further, it should be noted that data decoder124may be configured to output multi-dimensional data to various types of devices and/or sub-components thereof. For example, data decoder124may be configured to output data to any communication medium. Further, as described above, the techniques described in this disclosure may be particularly useful for allowing machine tasks to be distributed across a communications network. Thus, in some examples, source device102may represent an acquisition device where data source104acquires video data and generates corresponding feature data, data encoder106compresses feature data e.g., according to one or more techniques described herein, and destination device120is a device that performs analysis and inference on the reconstructed feature data. It should be noted, for example, with respect to the example described above, data encoder106and data decoder124may be configured to code multiple types of data. For example, in the case of video data, data encoder106may receive source video and corresponding feature data and generate a compliant bitstream according to a video coding standard and generate a bitstream including compressed feature data, e.g., according to the techniques described herein. In this case, in one example, destination device120may be a headend type of device that reconstructs video (e.g., a high quality representation) and the feature data from a received bitstreams and encodes the reconstructed video based on the feature data, e.g., at output126, for further distribution (e.g., to nodes in a media distribution system).

As described above, data encoder106may include any device configured to receive multi-dimensional data and an example of multi-dimensional data includes video data which may be coded according to a typical video coding standard. As described in further detail below, in some example, techniques for coding multi-dimensional data described herein may be utilized in conjunction with techniques utilized in typical video standards.FIG.5is a block diagram illustrating an example of a video encoder that may be configured to encode video data in accordance with typical video encoding techniques. It should be noted that although example video encoder200is illustrated as having distinct functional blocks, such an illustration is for descriptive purposes and does not limit video encoder200and/or sub-components thereof to a particular hardware or software architecture. Functions of video encoder200may be realized using any combination of hardware, firmware, and/or software implementations. Video encoder200may perform intra prediction coding and inter prediction coding of picture areas, and, as such, may be referred to as a hybrid video encoder. In the example illustrated inFIG.5, video encoder200receives source video blocks. In some examples, source video blocks may include areas of picture that has been divided according to a coding structure. For example, source video data may include CTUs, sub-divisions thereof, and/or another equivalent coding unit. In some examples, video encoder200may be configured to perform additional sub-divisions of source video blocks. It should be noted that the techniques described herein are generally applicable to video coding, regardless of how source video data is partitioned prior to and/or during encoding. In the example illustrated inFIG.5, video encoder200includes summer202, transform coefficient generator204, coefficient quantization unit206, inverse quantization and transform coefficient processing unit208, summer210, intra prediction processing unit212, inter prediction processing unit214, reference block buffer216, filter unit218, reference picture buffer220, and entropy encoding unit222. As illustrated inFIG.5, video encoder200receives source video blocks and outputs a bitstream.

In the example illustrated inFIG.5, video encoder200may generate residual data by subtracting a predictive video block from a source video block. Summer202represents a component configured to perform this subtraction operation. In one example, the subtraction of video blocks occurs in the pixel domain. Transform coefficient generator204applies a transform, such as a DCT or a conceptually similar transform, to the residual block or sub-divisions thereof (e.g., four 8×8 transforms may be applied to a 16×16 array of residual values) to produce a set of transform coefficients. Transform coefficient generator204may be configured to perform any and all combinations of the transforms included in the family of discrete trigonometric transforms, including approximations thereof. Transform coefficient generator204may output transform coefficients to coefficient quantization unit206. Coefficient quantization unit206may be configured to perform quantization on the transform coefficients. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may alter the rate-distortion (i.e., bit-rate vs. quality of video) of encoded video data. In a typical video coding standard, the degree of quantization may be modified by adjusting a quantization parameter (QP) and a quantization parameter may be determined based on signaled and/or predicted values. Quantization data may include any data used to determine a QP for quantizing a particular set of transform coefficients. As illustrated inFIG.5, quantized transform coefficients (which may be referred to as level values) are output to inverse quantization and transform coefficient processing unit208. Inverse quantization and transform coefficient processing unit208may be configured to apply an inverse quantization and an inverse transformation to generate reconstructed residual data. As illustrated inFIG.5, at summer210, reconstructed residual data may be added to a predictive video block. Reconstructed video blocks may be stored to reference block buffer216and used as reference for predicting subsequent blocks (e.g., using intra prediction).

Referring again toFIG.5, intra prediction processing unit212may be configured to select an intra prediction mode for a video block to be coded. Intra prediction processing unit212may be configured to evaluate reconstructed blocks stored to reference block buffer216and determine an intra prediction mode to use to encode a current block. In a typical video coding standard, possible intra prediction modes may include planar prediction modes, DC prediction modes, and angular prediction modes. As illustrated inFIG.5, intra prediction processing unit212outputs intra prediction data (e.g., syntax elements) to entropy encoding unit222.

Referring again toFIG.5, inter prediction processing unit214may be configured to perform inter prediction coding for a current video block. Inter prediction processing unit214may be configured to receive source video blocks, select a reference picture from pictures stored to the reference buffer220, and calculate a motion vector for a video block. A motion vector may indicate the displacement of a prediction unit of a video block within a current video picture relative to a predictive block within a reference picture. Inter prediction coding may use one or more reference pictures. Inter prediction processing unit214may be configured to select predictive block(s) by calculating a pixel difference determined by, for example, sum of absolute difference (SAD), sum of square difference (SSD), or other difference metrics. As described above, a motion vector may be determined and specified according to motion vector prediction. Inter prediction processing unit214may be configured to perform motion vector prediction, as described above. Inter prediction processing unit214may be configured to generate a predictive block using the motion prediction data. For example, inter prediction processing unit214may locate a predictive video block within reference picture buffer220. It should be noted that inter prediction processing unit214may further be configured to apply one or more interpolation filters to a reconstructed residual block to calculate sub-integer pixel values for use in motion estimation. Inter prediction processing unit214may output motion prediction data for a calculated motion vector to entropy encoding unit222.

Referring again toFIG.5, filter unit218receives reconstructed video blocks from reference block buffer216and outputs a filtered picture to reference picture buffer220. That is, in the example ofFIG.5, filter unit218is part of an in-loop filtering process. Filter unit218may be configured to perform one or more of deblocking, SAO filtering, and/or ALF filtering, for example, according to a typical video coding standard. Entropy encoding unit222receives data representing level values (i.e., quantized transform coefficients) and predictive syntax data (i.e., intra prediction data and motion prediction data). It should be noted that data representing level values may include for example, flags, absolute values, sign values, delta values, and the like. For example, significant coefficient flags and the like as provided in a typical video coding standard. Entropy encoding unit222may be configured to perform entropy encoding according to one or more of the techniques described herein and output a bitstream, for example, a compliant bitstream according to a typical video coding standard.

Referring again toFIG.4, as described above, data decoder124may include any device configured to receive coded multi-dimensional data and an example of coded multi-dimensional data includes video data which may be coded according to a typical video coding standard.FIG.6is a block diagram illustrating an example of a video decoder that may be configured to decode video data in accordance with typical video decoding techniques which may be utilized with one or more techniques of this disclosure. In the example illustrated inFIG.6, video decoder300includes an entropy decoding unit302, inverse quantization unit304, inverse transform coefficient processing unit306, intra prediction processing unit308, inter prediction processing unit310, summer312, post filter unit314, and reference buffer316. It should be noted that although example video decoder300is illustrated as having distinct functional blocks, such an illustration is for descriptive purposes and does not limit video decoder300and/or sub-components thereof to a particular hardware or software architecture. Functions of video decoder300may be realized using any combination of hardware, firmware, and/or software implementations.

As illustrated inFIG.6, entropy decoding unit302receives an entropy encoded bitstream. Entropy decoding unit302may be configured to decode syntax elements and level values from the bitstream according to a process reciprocal to an entropy encoding process. Entropy decoding unit302may be configured to perform entropy decoding according any of the entropy coding techniques described above and/or determine values for syntax elements in an encoded bitstream in a manner consistent with a video coding standard. As illustrated inFIG.6, entropy decoding unit302may determine level values, quantization data, and prediction data from a bitstream. In the example, illustrated inFIG.6, inverse quantization unit304receives quantization data and level values and outputs transform coefficients to inverse transform coefficient processing unit306. Inverse transform coefficient processing unit306outputs reconstructed residual data. Thus, inverse quantization unit304and inverse transform coefficient processing unit306operate in a similar manner to inverse quantization and transform coefficient processing unit208described above.

Referring again toFIG.6, reconstructed residual data is provided to summer312. Summer312may add reconstructed residual data to a predictive video block and generate reconstructed video data. A predictive video block may be determined according to a predictive video technique (i.e., intra prediction and inter frame prediction). Intra prediction processing unit308may be configured to receive intra prediction syntax elements and retrieve a predictive video block from reference buffer316. Reference buffer316may include a memory device configured to store one or more pictures (and corresponding regions) of video data. Intra prediction syntax elements may identify an intra prediction mode, such as the intra prediction modes described above. Inter prediction processing unit310may receive inter prediction syntax elements and generate motion vectors to identify a prediction block in one or more reference frames stored in reference buffer316. Inter prediction processing unit310may produce motion compensated blocks, possibly performing interpolation based on interpolation filters. Identifiers for interpolation filters to be used for motion estimation with sub-pixel precision may be included in the syntax elements. Inter prediction processing unit310may use interpolation filters to calculate interpolated values for sub-integer pixels of a reference block. Post filter unit314may be configured to perform filtering on reconstructed video data. For example, post filter unit314may be configured to perform deblocking based on parameters specified in a bitstream. Further, it should be noted that in some examples, post filter unit314may be configured to perform proprietary discretionary filtering (e.g., visual enhancements, such as, mosquito noise reduction). As illustrated inFIG.6, a reconstructed video may be output by video decoder300, for example, to a display.

As described above with respect toFIGS.2A-2B, a block of video data, i.e., an array of data included within a MDDS, may be encoded by generating a residual, performing a transformation on the residual, and quantizing the transform coefficients to generate level values and decoded by performing inverse quantization on level values, performing an inverse transform, and adding the resulting residual to a prediction. An array of data included within a MDDS may also be coded using so-called autoencoding techniques. Generally, autoencoding may refer to a learning technique that imposes a bottleneck into a network to force a compressed representation of an input. That is, an autoencoder may be referred to as a non-linear Primary Component Analysis (PCA) that tries to represent input data in a lower dimensional space. An example of an autoencoder includes a convolution autoencoder that compresses an input using a single convolution operation. Convolution autoencoders may be utilized in so-called deep convolutional neural networks (CNNs).

FIG.7Aillustrates an example of autoencoding using a two-dimensional discrete convolution. In the example illustrated inFIG.7A, a discrete convolution is performed on a current block of video data (i.e., the block of video data illustrated inFIG.2A) to generate an output feature map (OFM), where the discrete convolution is defined according to a padding operation, a kernel, and a stride function. It should be noted that althoughFIG.7Aillustrates a discrete convolution on a two-dimensional input using a two-dimensional kernel, discrete convolution may be performed on higher dimensional data sets. For example, discrete convolution may be performed a three-dimensional input using a three-dimensional kernel (e.g., a cubic kernel). In the case of video data, such a convolution may down-sample video in both the spatial and temporal dimensions. Further, it should be noted that although the example illustrated inFIG.7Aillustrates where a square kernel is convolved over a square input, in other examples, the kernel and/or the input may be non-square rectangles. In the example illustrated inFIG.7A, the 4×4 array of video data is upscaled to a 6×6 array by duplicating the nearest value at the boundary. This is an example of a padding operation. In general, a padding operation increases the size of an input data set by inserting values. In a typical case, zero values may be inserted into an array in order to achieve a particular sized array prior to convolution. It should be noted that padding functions may include one or more of inserting zero's (or another default value) at particular locations, symmetric extension, replicate extension, circular extension at various positions of a data set. For example, for symmetric extension, input array values outside the bounds of the array may be computed by mirror-reflecting the array across the array border along the dimension being padded. For replicate extension, input array values outside the bounds of the array may be assumed to equal the nearest array border value along the dimension being padded. For circular extension, input array values outside the bounds of the array may be computed by implicitly assuming the input array is periodic along the dimension being padded.

Referring again toFIG.7A, an output feature map is generated by convolving a 3×3 kernel over the 6×6 array according to a stride function. That is, the stride illustrated inFIG.7Aillustrates the top-left position of the kernel at a corresponding position in the 6×6 array. That is, for example, at stride position 1, the top-left of the kernel is aligned with the top-left of the 6×6 array. At each discrete position of the stride, the kernel is used to generate a weighted sum. Generated weighted sum values are then used to populate a corresponding position in an output feature map. For example, at position 1 of the stride function, the output of 107 (107= 1/16*107+⅛*107+ 1/16*103+⅛*107+¼*107+⅛*103+ 1/16*111+⅛*111+ 1/16*108) corresponds to the top-left position of the output feature map. It should be noted that in the example illustrated inFIG.7A, the stride function corresponds to a so-called unit stride, i.e., the kernel slides across every position of the input. In other examples, non-unit or arbitrary strides may be used. For example, a stride function may include only the positions 1, 4, 13, and 16 in the stride illustrated inFIG.7Ato generate a 2×2 output feature map. In this manner, in the case of two-dimensional discrete convolution, for an input data having a width, wi, and height, hi, an arbitrary padding function, an arbitrary stride function, and a kernel having a width, wk, and height, hk, may be used to create an output feature map having a desired width, wo, and height, ho. It should be noted, that similar to a kernel, a stride function may be defined for multiple dimensions (e.g., a three-dimensional stride function may be defined). It should be noted that in some cases, for particular kernel size and stride function, the kernel may lie outside of the support region. In some cases, the output at such a position is not valid. In some cases, a corresponding value is derived for the out-of-bound support position, e.g., according to a padding operation.

It should be noted that in the example illustrated inFIG.7A, the 4×4 array of video data is illustrated as being down-sampled to a 2×2 output feature map by selecting the underlined values of the 4×4 output feature map. The 4×4 output feature map is shown for illustration purposes. That is, to illustrate a typical unit stride function. In a typical case, computations would not be made for discarded values. In a typical case, as described above, the 2×2 output feature map could/would be derived by performing the weighted sum operation with the kernel at positions 1, 4, 13, and 16. However, it should be noted that in other examples, so-called pooling operations, such as finding a maximum pooling, may be performed on an input (prior to performing the convolution) or an output feature map to down-sample a data set. For example, in the example illustrated inFIG.7A, the 2×2 output feature map may be generated by taking a local maximum of each 2×2 region in the 4×4 output feature map (i.e., 108, 104, 117, and 108). That is, there may be numerous ways to perform autoencoding that includes performing convolutions on input data in order to represent the data as a down-sampled output feature map.

Finally, as indicated inFIG.7A, an output feature map may be quantized in a manner similar to that described above with respect to transform coefficients (e.g., amplitudes restricted to a set of specified values). In the example illustrated inFIG.7A, the amplitudes of the 2×2 output feature map are quantized by division by 2. In this case, quantization may be described as a uniform quantization defined by:

QOFM(x,y)=round(OFM(x,y)/Stepsize)Where,QOFM(x,y) is a quantized value corresponding position (x, y);OFM(x,y) is a value corresponding position (x, y);Stepsize is a scalar; andround(x) rounds x to the nearest integer.

Thus, for the example illustrated inFIG.7A, Stepsize=2 and x=0 . . . 1, y=0 . . . 1. In this example, at an autodecoder, the inverse quantization for deriving the recovered output feature map, ROFM(x,y) may be defined as follows:

It should be noted that in one example, a respective Stepsize may be provided for each position, i.e., Stepsize(x,y). It should be noted that this may be referred to a uniform quantization, as across the range of possible amplitudes at a position in OFM(x,y) the quantization (i.e., scaling) is same.

In one example, quantization may be non-uniform. That is, the quantization may differ across the range of possible amplitudes. For example, respective Stepsizes may vary across a range of values. That is, for example, in one example, a non-uniform quantization function may be defined as follows:

Where

Further, it should be noted that as described above, quantization may include mapping an amplitude in a range to a particular value. That is, for example, in one example, non-uniform quantization function may be defined as:

Where, valuei+1>valueiand valuei+1−valueidoes not have to equal valuej+1−valuejfor i≠j
The inverse of the non-uniform quantization process, may be defined as:

The inverse process corresponds to a lookup table and may be signaled in the bitstream.

Finally, it should be noted that combinations of the quantization techniques described above may be utilized and in some cases, specific quantization functions may be specified and signaled. For example, quantization tables may be signaled in a manner similar to signaling of quantization tables in ITU-T H.266.

Referring again toFIG.7A, although not shown, but as described in further detail below, entropy encoding may be performed on quantized output feature map data. Thus, as illustrated in the example ofFIG.7A, the quantized output feature map is a compressed representation of the current video block.

As illustrated inFIG.7B, the current block of video data is decoded by performing inverse quantization on the quantized output feature map, performing a padding operation on the recovered output feature map, and convolving the padded output feature map with a kernel. Similar toFIG.2B,FIG.7Billustrates a reconstruction error which is the difference between current block and recovered block. It should be noted that the padding operation performed inFIG.7Bis different than the padding operation performed inFIG.7Aand the kernel utilized inFIG.7Bis different than the kernel utilized inFIG.7A. That is, in the example illustrated inFIG.7B, zero values are interleaved with the recovered output feature map, and the 3×3 kernel in convolved over the 6×6 input using a unit stride resulting in the recovered block of MDDS. It should be noted that such a convolution operation performed during autodecoding may be referred to a convolution-transpose (convT). It should be noted that a convolution-transpose, in some cases may define a specific relationship between kernels at each of an autoencoder and autodecoder and in other cases, the term convolution-transpose may be more general. It should be noted that there may be several ways in which autodecoding may be implemented. That is,FIG.7Bprovides an illustrative case of a convolution-transpose and there numerous ways in which a convolution-transpose (and autodecoding) may be performed and/or implemented. The techniques described herein are generally applicable to autodecoding. For example, with respect to the example illustrated inFIG.7B, in a simple case, each of the four values illustrated in the recovered output feature map may be duplicated to create a 4×4 array (i.e., an array having its top-left four values as108, its top-right four values as102, its bottom-left four values as116, and its bottom-right four values as108). Further, other padding operations, kernels, and/or stride functions may be utilized. Essentially, at an autodecoder, an autodecoding process may be selected in a manner that achieves a desired objective, for example, reducing a reconstruction error. It should be noted the other desired objectives may include reducing visual artifacts, increasing the probability an object is detected, etc.

As described above, with respect toFIGS.7A-7B, autoencoding may be performed on video data to generate a quantized output feature map data. A quantized output feature map is a compressed representation of the current video block. In some cases, that is, based on how autoencoding is performed an output feature map may effectively be a down-sampled version of video data. For example, referring toFIG.7A, the 4×4 array of video data may be compressed to a 2×2 array. In a case where the 4×4 array of video data is one of several 4×4 arrays of video data included in a 1920×1080 resolution picture, autoencoding each 4×4 array as illustrated inFIG.7Aeffectively down-samples the 1920×1080 resolution picture to a 960×540 resolution picture. As described in further detail below, such a down-sampled representation of video data may be coded according to a typical video coding standard.

It should be noted that in addition to performing discrete convolution on two-dimensional (2D) data sets, convolution may be performed on one-dimensional data sets (1D) or on higher dimensional data sets (e.g., 3D data sets). Thus, there are several ways in which video data may be mapped to a multi-dimensional data set. In general, video data may be described as having a number of input channels of spatial data. That is, video data may be described as an Ni×W×H, data set where Niis the number of input channels, W is a spatial width, and H is a spatial height. It should be noted that Ni, in some examples, may be a temporal dimension (e.g., number of pictures). For example, Niin Ni×W×H may indicate a number of 1920×1080 monochrome pictures. Further, in some examples, Ni, may be a component dimension (e.g., number of color components). For example, Ni×W×H may include a single 1024×742 image having RGB components, i.e., in this case, Niequals 3. Further, it should be noted that in some cases, there may be N input channels for both a number of components (e.g., NCi) and a number of pictures (e.g., NPi). In this case, video data may be specified as NCi×NPi×W×H, i.e., as a four-dimensional data set. According to the NCi×NPi×W×H format, an example of 60 1920×1080 monochrome pictures may be expressed as 1×60×1920×1080 and a single 1024×742 RGB image may be expressed as 3×1×1024×742. It should be noted that in these cases, each of the four-dimensional data sets have a dimension having a size of 1, and may be referred to as three-dimensional data sets and respectively simplified to 60×1920×1080 and 3×1024×742. That is, 60 and 3 are both input channels in three-dimensional data sets, but refer to different dimensions (i.e., temporal and component).

As described above, in some cases, a 2D OFM may correspond to a down-sampled component of video (e.g., luma) in both the spatial and temporal dimensions. Further, in some cases, a 2D OFM may correspond to a down-sampled video in both the spatial and component dimensions. That is, for example, a single 1024×742 RGB image, (i.e., 3×1024×742) may be down-sampled to a 1×342×248 OFM. That is, down-sampled by 3 in both spatial dimensions and down-sampled by 3 in the component dimension. It should be noted that in this case, 1024 may be padded by 1 to 1025 and 743 may be padded by 2 to 744, such that each are multiples of 3. Further, in one example, 60 1920×1080 monochrome pictures (i.e., 60×1920×1080) may be down-sampled to a 1×640×360 OFM. That is, down-sampled by 3 in both spatial dimensions and down-sampled by 60 in the temporal dimension.

It should be noted that in the cases above, the down-sampling may be achieved by having a Ni×3×3 kernel with a stride of 3 in the spatial dimension. That is, for the 3×1025×744 data set, the convolution generates a single value for each 3×3×3 data point and for the 60×1920×1080 data set, the convolution generates a single value for each 60×3×3 data point. It should be noted that in some cases, it may be useful to perform discrete convolution on a data set multiple times, e.g., using multiple kernels and/or strides. That is, for example, with respect to the example described above, a number of instances of Ni×3×3 kernels (e.g., each with different values) may be defined and used to generate a corresponding number of instances of OFMs. In this case, the number of instances may be referred to as a number of output channels, i.e., NO. Thus, in the case where an Ni×Wi×Hiinput data set is down-sampled according to a Noinstances of Ni×Wk×Hkkernels, the resulting output data may be represented as NO×WO×HO. Where WOis a function of Wi, Wk, and the stride in the horizontal dimension and HOis a function of Hi, Hk, and the stride in the vertical dimension. That is, each of WOand HOare determined according to spatial down-sampling. It should be noted that in some examples, according to the techniques herein, an NO×WO×HOdata set may be used for object/feature detection. That is, for example, each of the NOdata sets may be compared to one another and relationships in common regions may be used to identify the presence of an object (or another feature) in the original Ni×Wi×Hiinput data set. For example, a comparison/task may be carried out over a multiple of NN layers. Further, an algorithm, such as, for example, a non-max suppression to select amongst available choices, may be used. In this manner, for example, the encoding parameters of a typical video encoder may be optimized based on the NO×WO×HOdata set, e.g., quantization varied based on the indication of an object/feature in video.

In one example, in a case where a number of instances of K×K kernels each having a corresponding dimension equal to a Niis used in processing of an Ni×Wi×Hidataset, the following notation may be used to indicate one of a convolution or convolution transpose, the kernel size, the stride function, and padding function for a convolution, and the number of output dimensions of a discrete convolution:conv2d: 2D convolution, conv2dT: 2D convolution transpose,kK: kernel of size K for all dimensions (e.g., K×K);sS: stride of S for all dimensions (e.g. (S, S));pP: pad by P to both sides of all dimensions with value 0, (e.g., (P, P) for 2D); andnN number of output of channels.

It should be noted that in the example notation provided above, the operations are symmetric, i.e., square. It should be noted that in some examples, the notation may be as follows for general rectangular cases:conv2d: 2D convolution, conv2dT: 2D convolution transpose,kKwKh: kernel of size Kwfor width dimension and Khfor height dimension (e.g., Kw×Kh);sSwSh: stride of Swfor width dimension and Shfor height dimension (e.g., Sw×Sh);pPwPh: pad by Pwto both sides of width dimension and Phto both sides of height dimension (e.g., Pw×Ph); andnN number of output of channels.

It should be noted that in some examples, a combination of the above notation may be used. For example, in some examples, K, S, and PwPhnotation may be used. Further, it should be noted that in other examples, padding may be asymmetric about a spatial dimension (e.g., Pad 1 row above, 2 rows below).

Further, as described above, convolution may be performed on one-dimensional data sets (1D) or on higher dimensional data sets (e.g., 3D data sets). It should be noted that in some cases, the notation above may be generalized for convolutions of multiple dimensions as follows:conv1d: 1D convolution, conv2d: 2D convolution, conv3d: 3D convolutionconv1dT: 1D convolution transpose, conv2dT: 2D convolution transpose, conv3dT: 3D convolution transposekK: kernel of size K for all dimensions (e.g., K for 1D, K×K for 2D, K×K×K for 3D)sS: stride of S for all dimensions (e.g., (S) for 1D, (S, S) for 2D, (S, S, S) for 3D)pP: pad by P to both sides of all dimensions with value 0 (e.g., (P) for 1D, (P, P) for 2D, (P, P, P) for 3D)nN number of output of channels

The notation provided above may be used for efficiently signaling of autoencoding and autodecoding operations. For example, in the case of down-sampling a single 1024×742 RGB image to a 342×248 OFM, as described above, according to 256 instances of kernels may be described as follows:Input data: 3×1024×742Operation: conv2d, k3, s3, p1, n256Resulting Output data: 256×342×248

Similarly, in the case of down-sampling a 60 1920×1080 monochrome pictures to a 640×360 OFM, as described above, according to 32 instances of kernels may be described as follows:Input data: 60×1920×1080Operation: conv2d, k3, s3, p0,2 n32Resulting Output data: 32×640×360

It should be noted that there may be numerous ways to perform convolution on input data in order to represent the data as an output feature map (e.g., 1stpadding, 1stconvolution, 2ndpadding, 2ndconvolution, etc.). For example, the resulting data set 256×342×248 may be further down-sampled by 3 in the spatially dimension and by 8 in the channel dimension and as follows:Input data: 256×342×248Operation: conv2d, k3, s3, p0,1, n32Resulting Output data: 32×114×84

In one example, according to the techniques herein, the operation of an autodecoder may be well-defined and known to an autoencoder. That is, the autoencoder knows the size of the input (e.g., the OFM) received at the decoder (e.g., 256×342×248, 32×640×360, or 32×114×84 in the examples above). This information along with the known k and s of convolution/convolution-transpose stages can be used to determine what the data set size will be at a particular location.

As described above, an example of a machine task includes object recognitions tasks. Object recognition tasks typically involve receiving an image, generating feature data corresponding to the image, analyzing the feature data, and generating inference data. Examples of typical object detection systems include, for example, systems implementing versions of YOLO, RetinaNet, and Faster R-CNN. Detailed descriptions of object detection systems, performance evaluation techniques, and performance comparisons are provided in various technical journals and the like. For example, Redmon et al., “YOLOv3: An Incremental Improvement,” arXiv:1804.02767, 8 Apr. 2018, generally describes YOLOv3 and provides a comparison to other object detection systems. Wu et al., “Detectron2,” at github, facebookresearch, detectron2, 2019 provides libraries and associated documentation for Detectron2 which is a Facebook Artificial intelligence (AI) Research platform for object detection, segmentation and other visual recognition tasks. It should be noted that for explanation purposes, in some cases, the techniques described herein are described with specific example object detection systems (e.g., Detectron2). However, it should be noted that the techniques herein may be generally applicable to other object detection systems.

During an MPEG Meeting in 2020, the Video Coding for Machines (VCM) Group made a decision to adopt Detectron2 as the platform for object detection and instance segmentation.FIG.8illustrates an example where feature data and inference data (i.e., bounding boxes) are generated for image data according to Detectron2. It should be noted that for the sake of brevity a complete detailed description of how Detectron2 generates feature data and inference data is not provided herein. The techniques described herein relate to compressing/decompressing feature data generated according to an object detection system (e.g., Detectron2) which may be useful for distributing operations of the object detection system over a communications network. As illustrated inFIG.8, Detectron2 can be described as including a backbone network unit900and an inference network unit1000. In general, object detection systems include a backbone network that generates feature data and an inference network that generates inference data from the feature data. In Detectron2, a Feature Pyramid Network (FPN), Base-RCNN-FPN, extracts feature maps from an BGR input image at different scales. Detectron2 generates features maps at ¼ scale, ⅛ scale, 1/16 scale, 1/32 scale, and 1/64 scale and at each scale, 256 channels of data are generated. That is, as described above with respect to autoencoding, data is generated for each of 256 instances of kernels at each scale. It should be noted that in Detectron2, at each scale, one or more convolutions and operations are performed to generate feature data (e.g., 7×7 convolution with stride=2 and max pooling with stride=2).FIG.9is a conceptual diagram illustrating an example of generating feature data according to Detectron2. As illustrated inFIG.9, for input data having a width, W, and a height. H, at each scale, i.e., ¼ scale, ⅛ scale, 1/16 scale, 1/32 scale and 1/64 scale, there are 256 channels of feature data. With respect to Dectectron2, the scales of feature data are respectively referred to as P2, P3, P4, P5, and P6. Thus, in a case where, an input image has a size of 1280×800, according to Detectron2, P2, P3, P4, P5, and P6 may have the following respective sizes: 256×320×200; 256×160×100; 256×80×50; 256×40×25; and 256×20×13. As described in further detail below, each of P2, P3, P4, P5, and P6 may be compressed for distribution over a communications network. It should be noted, that as described above, the techniques herein may be generally applicable to other object detection systems. That is, types of backbone networks, other than that used in Detectron2 may generate a different number of channels (e.g., 128, 1024, etc.) at different scales (e.g., ½ and ⅛ scales) and the compression techniques herein may be utilized with such backbone networks.

As described above, an inference network (e.g. inference network unit1000) receives feature data and generates inference data. With respect to Detectron2, and in general, in some examples, an inference network, may be described as including a region proposal network and sub-classes of ROI (regions of interest) heads, which may generally be referred to as a box head. In Detectron2, a region proposal network receives the features maps at ¼ scale, ⅛ scale, 1/16 scale, 1/32 scale, and 1/64 scale, each having 256 channels, as described above, and outputs1000box proposals (which is set as a default) with confidence scores. That is, each of the 1000 box proposals, includes an anchor coordinate, a height, a width, and a score. In general, a region proposal network in Detectron2 can be described as including a RPN head and an RPN output.FIG.10illustrates an example of region proposal network1020including RPN head1022and RPN output1024. In Detectron2, for each feature scale, an RPN head generates objectness logits and anchor deltas. Objectness logits are a probability map of object existence and anchor deltas are a relative box shape and position to anchors. As illustrated inFIG.10, an initial conv2d k3 n256 operation is performed on a feature map. To generate objectness logits a conv2d k1 n3 is performed after the initial conv2d k3 n256 operation. To generate anchor deltas a conv2d k1 n3×4 is performed data after the initial conv2d k3 n256 operation. As illustrated inFIG.10, RPN output1024receives objectness logits and defined parameters including e.g., anchors and ground truth boxes, and generates box proposals. In Detectron2, the generation of box proposals includes anchor generation, ground truth preparation, loss calculation, and proposal selection. Essentially, in Detectron2, the output feature maps of the objectness logits and anchor deltas are associated with ground truth boxes to generate predicted boxes which are scored and the top 1,000 scored boxes are selected as output.

As described above, an inference network may include a box head unit. In general, a box head in Detectron2 can be described as including a ROI pooler, a box head, and a box predictor.FIG.11illustrates an example of box head unit1050including ROI Pooler1052, box head unit1054, and box predictor unit1056. In Detectron2, an ROI pooler pools the rectangular regions of the feature maps that are specified by the box proposals. Essentially, an ROI pooler, generates a tensor which is the collection of cropped instance features which include balanced foreground and background ROIs. In Detectron2, this tensor may have a size of [Nxbatch size, 256, 7, 7], where the ROI size is 7×7. In Detectron2, a box head may be a FastRCNNConvFCHead and a box predictor may be a FastRCNNOutputLayers. It should be noted that an ROI may generate tensors of other sizes. It should be noted that although not shown inFIG.11, prior to input into box head unit1054, the tensor generated from ROI pooler is flattened to a 256×7×7=12,544 tensor.

As illustrated inFIG.11, box head unit1054performs two Linear( ) operations. A Linear( ) operation is specified as follows:Linear(in_features_count, out_features_count, bias)Applies a linear transformation to the incoming data:

Parameters

in_features—size of each input sampleout_features—size of each output samplebias—If set to False, the layer will not learn an additive bias. Default: True

Shape

Input: (N, *, Hin) where * means any number of additional dimensions and Hin=in_featuresOutput: (N, *, Hout) where all but the last dimension are the same shape as the input and Hout=out_features.

Variables

˜Linear.weight—the learnable weights of the module of shape (out_features,in_features). The values are initialized from U(−sqrt{k}, sqrt{k}), where k=1/in_features, and sqrt{ } is a square root operation˜Linear.blas—the learnable bias of the module of shape (out_features). If bias is True, the values are initialized from U(−sqrt{k}, sqrt{k}), where k=1/in_features

Box head unit1054classifies an object within an ROI and fine-tunes the box position and shape. Box predictor unit1056generates classification scores and bounding box predictors. The classification scores and bounding box predictors may be used to output bounding boxes. Typically, in Detectron2, a maximum of 100 bounding boxes are filtered out using non-maximum suppression (NMS). It should be noted the maximum number of bounding boxes is configurable and it may be useful to change the number depending on a particular application.

As described above, in Detectron2, inference data includes bounding boxes. In some applications, it may be useful to have so-called instance segmentation information, which may, for example, provide a per-pixel classification for a bounding box. That is, instance segmentation information may indicate whether a pixel within a bounding box constitutes part of the object. Further, instance segmentation information may, for example, include a binary mask for a ROI. As described above, with respect to the example inFIG.11, an ROI pooler essentially generates tensors which are the collection of cropped instance features and these tensors may be input a FastRCNNConvFCHead box head.

As described above, it is useful for allowing machine tasks to be distributed across a communication network. That is, referring toFIG.12, each of feature extraction network900and inference data generation network1000may distributed across communications medium110, and thus, in some examples located at distinct physical locations. As described above, the VCM Group made a decision to adopt Detectron2 as the platform for object detection and instance segmentation. Dong-Ha Kim, et al. “[VCM Track 1] Compression of FPN Multi-Scale Features for Object Detection Using VVC”, m59562, ISO/IEC JTC 1/SC 29/WG 2, April 2022 (hereinafter Kim1) and Dong-Ha Kim, et al. “[VCM-Track1] Performance of the Enhanced MSFC with Bottom-Up MSFF”, m60197, July 2022 (hereinafter Kim2, Kim1 and Kim2 are collectively referred to herein as Kim) describe processing pipelines where Detectron2 P5, P4, P3, and P2 multi-scale feature data is converted to a single C×H/32×W/32 tensor, where C equals, 256, 192, 144, or 64 channels with a so-called Multi-Scale Feature Fusion (MSFF) module and the single tensor is encoded into a bitstream using ITU-T H.266. The bitstream is decoded according to ITU-T H.266 and each of P5, P4, P3, and P2 are recovered using a so-called Multi-Scale Feature Reconstruction (MSFR) module.FIG.12illustrates an example of the system for compressing and recovering multi-scale feature data. That is, the MSFF modules described in Kim1 and Kim2 may be examples of a feature conversion engine1100and the MSFR modules described in Kim1 and Kim2 may be examples of a feature inverse conversion engine1200.

FIG.13is a block diagram illustrating an example of a feature conversion engine1100. The example feature conversion engine1100may generally correspond to the MSFF module described in Kim2. As illustrated inFIG.13, feature conversion engine1100receives P5, P4, P3, and P2 tensors (e.g., generated according to Detectron2) and generates a single C×H/32×W/32 tensor. As illustrated inFIG.13, feature conversion engine1100includes feature align and concatenation unit1102and squeeze and excitation unit1104. As illustrated inFIG.13, feature align and concatenation unit1102down samples each of the respective 256 channels of P4, P3, and P2 using respective 2D convolution operations, such that each has the same spatial dimensions as P5. For example, as described above, in a case where, an input image has a size of 1280×800, according to Detectron2, P2, P3, P4, and P5 may have the following respective sizes: 256×320×200; 256×160×100; 256×80×50; and 256×40×25. In this case, feature align and concatenation unit1102down samples each of P2, P3, and P4 to 256×40×25. As further illustrated inFIG.13, feature align and concatenation unit1102concatenations P5 and each of down sampled (or resized) P2, P3, and P4 about the channel dimension to generate1024channels (i.e., 256×4) at the size of P5.

As further illustrated inFIG.13, the concatenated feature tensor is input into squeeze and excitation unit1104. It should be noted that for the sake of brevity a complete detailed description of squeeze and excitation networks is not provided herein. However, reference is made to Hu et al., “Squeeze-and-Excitation Networks,” arXiv:1709.01507, 16 May 2019, which describes squeeze and excitation networks in detail. In general, as illustrated inFIG.13, squeeze and excitation unit1104reweighs the concatenated feature tensor and performs channel reduction on the reweighed concatenated feature tensor, such that C channels at the spatial resolution of P5 are output. As illustrated inFIG.13, at the upper branch of squeeze and excitation unit1104respective weights are generated. The concatenated feature tensor is multiplied by a respective weight to reweigh concatenated feature tensor. The global average pooling operation essentially averages each channel and the FC 1, 1, n1024 operations are fully connected layers. Further, inFIG.13, ReLU refers to an operation where ReLU(x)=max (0, x). That is, if an output at the second FC stage is negative, it is set to 0. Further, sigmoid corresponds to the following operation:

Thus, the upper branch of squeeze and excitation unit1104outputs a weight value ranging from 0 to 1 for each channel. After the respective weights are applied, a final convolution operation is used to generate the number of channels to be output, C channels. Kim1 describes where C is equal to 256 and where the output of MSFF module is a 256-channel, W/32×H/32 floating data type tensor. Kim1 further describes where C may be equal to 192 or 144 and Kim2 further described where C may be equal to 64.

It should be noted that Kim2 describes where a so-called bottom-up module may be used to preprocess Detectron2 multi-scale feature data prior to input to feature align and concatenation unit1102.FIG.14illustrates an example of a feature conversion engine1100including bottom-up unit1106. As illustrated inFIG.14, P2 and P5 are directly input into feature align and concatenation unit1102. Additionally, P2 is down scaled using a convolution layer and added to P3. The result is passes through a convolutional layer which fine-tunes the summed feature before it is provided to feature align and concatenation unit1102. Similarly, the fine-tuned feature data corresponding to P3 is down scaled and added to P4, the result of which passes through a convolutional layer which fine-tunes the features data before it is provided to feature align and concatenation unit1102.

Referring again toFIG.12and as described above, the MSFR modules described in Kim1 and Kim2 may be examples of a feature inverse conversion engine1200.FIG.15is a block diagram illustrating an example of a feature inverse conversion engine1200. The example feature inverse conversion engine1200illustrated inFIG.15may generally correspond to the MSFR module described in Kim1 and Kim2. The input to feature inverse conversion engine1200corresponds to the output of feature conversion engine1100. As illustrated inFIG.15, the input to feature inverse conversion engine1200has C channels and as described above, Kim describes where C may be equal to one or 256, 192, 144, or 64. Recovered P5 feature data, P5′, is generated by restoring C to 256. As illustrated inFIG.15, for each of P4, P3, and P2, the input is spatially upscaled to the appropriate size using a 2D convolution and C is restored to 256 using a 2D convolution. Further, as illustrated inFIG.15, recovered P4 feature data, P4′, is generated by adding upscaled P5′ to the spatial and channel restored data corresponding to P4. As illustrated inFIG.15, P5′ is upscaled using a 2D convolution-transpose operation. Similarly, recovered P3 feature data, P3′, is generated by adding upscaled P4′ to the spatial and channel restored data corresponding to P3 and recovered P2 feature data, P2′, is generated by adding upscaled P3′ to the spatial and channel restored data corresponding to P2. It should be noted that although not illustrated inFIG.12, Kim1 describes where P6′ is generated from PS' using a max pooling layer. It should be noted that the process illustrated inFIG.15for recovering feature data may be described as a top-down approach or architecture.

As described above, the MSFF modules described in Kim1 and Kim2 may be examples of a feature conversion engine1100and the MSFR modules described in Kim1 and Kim2 may be examples of a feature inverse conversion engine1200. Z. Zhang, et al., “MSFC: Deep Feature Compression in Multi-Task Network,” in 2021 IEEE International Conference on Multimedia and Expo (ICME), Shenzhen, China: IEEE, July 2021, pp. 1-6. doi: 10.1109/ICME51207.2021.9428258 (hereinafter Zhang) describes another example of a system for compressing and recovering multi-scale feature data. In the example described in Zhang, for input data having a width, W, and a height, H, at each scale, i.e., ¼ scale, ⅛ scale, 1/16 scale, 1/32 scale and 1/64 scale, there are 256 channels of feature data. That is, in Zhang each of P2, P3, P4, P5, and P6 may have the following respective sizes: 256×H/4×W/4; 256×H/8×W/8; 256×H/16×W/16; 256×H/32×W/32; and 256×H/64×W/64. In the example described in Zhang, the MSFF module is similar to the MSFF module in Kim, described above with respect toFIG.13, where a feature align and concatenation unit1102down samples each of the respective 256 channels of P4, P3, and P2 using respective 2D convolution operations, such that each has the same spatial dimensions as P5 and a squeeze and excitation unit1104reweighs the concatenated feature tensor and performs channel reduction on the reweighed concatenated feature tensor, such that C channels at the spatial resolution of P5 are output. As described above, the MSFR module in Kim utilizes a top-down approach. The MSFR module in Zhang utilizes a bottom-up approach.FIG.16is a block diagram illustrating an example of a feature inverse conversion engine1200. The example feature inverse conversion engine1200illustrated inFIG.16may generally correspond to the MSFR module described in Zhang. The input to feature inverse conversion engine1200corresponds to the output of feature conversion engine1100. As illustrated inFIG.16, the input to feature inverse conversion engine1200has C channels. As illustrated inFIG.16, recovered P2 feature data. P2′, is generated by restoring C to 256 and upscaling the input to the dimensions of P2. As further illustrated inFIG.16, for each of P5, P4, and P3, the input is spatially upscaled to the appropriate size using a 2D convolution and C is restored to 256 using a 2D convolution. Further, as illustrated inFIG.16, recovered P3 feature data, P3′, is generated by adding downscaled P2′ to the spatial and channel restored data corresponding to P3. Similarly, recovered P4 feature data. P4′, is generated by adding downscaled P3′ to the spatial and channel restored data corresponding to P4 and recovered P5 feature data. P5′, is generated by adding upscaled P4′ to the spatial and channel restored data corresponding to PS. It should be noted that although not illustrated inFIG.16, Zhang describes where P6′ is generated from PS' using a max pooling layer.

As described above, Kim utilizes ITU-T H.266 for generating a bitstream. In other examples, feature data may be compressed using techniques other than ITU-T H.266. For example, Zhang describes where feature data is compress utilizing a so-called single-stream feature codec (SSFC).FIG.17is an example of a coding system that may encode a multi-dimensional data set in accordance with one or more techniques of this disclosure. As illustrated inFIG.17, the system includes backbone network unit900, feature conversion engine1100, feature inverse conversion engine1200, compression engine2200, decompression engine2300, and communications medium110. Each of backbone network900, feature conversion engine1100, feature inverse conversion engine1200, and communications medium110may include corresponding examples described herein. Compression engine2200may be configured to compress feature data according to one or more of the techniques described herein and decompression engine2300may be configured perform reciprocal operations to reconstruct the feature data. For example, in one example, compression engine2200may correspond to the SSFC encoder and decompression engine2300may correspond to the SSFC decoder described in Zhang. Zhang describes where a SSFC encoder utilizes a convolutional layer, a batch normal layer and a Tanh activation function and performs a channel-wise reduction of feature data. Essentially, the SSFC encoder reduces the number of channels and centers and scales the feature date to the range of [−1,1]. The SSFC decoder in Zhang performs utilizes a convolutional layer, a batch normal layer, and a parametric ReLU to perform a reciprocal operation. A parametric ReLU is a type of ReLU.

In other examples, feature data may be compressed using techniques other than ITU-T H.266 and the SSFC described in Zhang.FIG.18illustrates an example of a compression engine2200andFIG.19illustrates an example of a corresponding decompression engine2300. The compression engine2200inFIG.18and the decompression engine2300inFIG.19may be referred to as an intra feature codec. In one example, the compression engine2200inFIG.18and the decompression engine2300inFIG.19may correspond to the intra feature codec described in K. Misra, et al., “Video Feature Compression for Machine Tasks,” in 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan: IEEE, July 2022, pp. 1-6. doi: 10.1109/ICME52920.2022.9859894 (hereinafter “Misra”). As illustrated inFIG.18, compression engine2200includes residual encoder unit2202, heatmap unit2204, quantizer unit2206, dequantizer unit2208, probability estimator unit2210, and arithmetic encoder unit2212. As illustrated in the example ofFIG.18, compression engine2200receives feature data, for example, C×H/32×W/32 feature data, and essentially generates a bitstream that compresses the input feature data by removing redundancies using residual encoding, spatially down sampling the feature data (e.g., by a factor of 3, 40×25 to 14×9), and reducing the channel count (e.g., by a factor of 4, 256 to 64).FIG.20is an example of a residual encoder unit2202. As illustrated inFIG.20, a sequence of res2d k3 n256 operations and summations are performed resulting in a refinement value being added to the input.FIG.21illustrates a res2d k3 nN operation. As illustrated inFIG.21, for an N channel input a refinement O′ is generated using subsequent convolutions and a ReLU operation and the refinement O′ is added to the input. Referring again toFIG.18, a conv2d k3 s3 p0 nCooperation is performed on the enhanced feature data generated by residual encoder unit2202. The conv2d k3 s3 p0 nCooperation down samples the enhanced feature data about the spatial and channel dimension (i.e., to Co). As illustrated inFIG.18, the result of the conv2d operation is then multiplied by a heatmap prior to quantization by quantizer unit2206. As illustrated inFIG.18, heatmap unit2204receives the result of the conv2d operation and generates a heat map. There may be numerous ways in which heatmap unit2204may be configured to generate a heatmap. In one example, heatmap unit2204provides a weight value from 0 to 1 (non-inclusive), which when multiplied by the input to heatmap unit2204effectively suppresses data without impacting subsequent machine task(s) and enhances data for machine task(s). The scaled enhanced feature data is input into quantizer unit2206which quantizes an input tensor. The output of quantizer unit2206is input into arithmetic encoder unit2212. As illustrated inFIG.18, arithmetic encoder unit2212also receives input from probability estimator unit2210. Probability estimator unit2110determines the Probability Mass Function (PMF) for quantization indices at each location within a tensor. During the determination of a PMF, subset of symbols (quantization indices and therefore dequantized values) that have been decoded in the past may be used to determine the PMF for current location. That is, as illustrated inFIG.18, output of dequantizer unit2108(i.e., decoded past symbols) may be input into probability estimator unit2210. A dequantizer unit performs reciprocal operations to a quantizer unit. Arithmetic encoder unit2212may use an arithmetic coder that makes use of the corresponding PMF when coding a symbol.

As illustrated inFIG.19, decompression engine2300includes arithmetic decoding unit2302, dequantizer unit2208, probability estimator unit2210, and residual decoder unit2304. Each of dequantizer unit2208and probability estimator unit2210may operate as described above. Further, arithmetic decoder unit2302may operate in a reciprocal manner to entropy encoding unit2212. The conv2dT k3 s3 p0 nC operation in decompression engine2300is configured to perform a reciprocal operation to conv2d k3 s3 p0 nCosuch that the size and the number of channels are restored. As illustrated inFIG.19, the output of the conv2dT k3 s3 p0 nC operation is input into residual decoder unit2304.FIG.22illustrates an example of a residual decoder unit2304. As illustrated inFIG.22, a sequence of res2d k3 n256 operations and summations are performed resulting in a refinement value being added to the input. That is, the sequence of res2d k3 n256 operations and summations essentially operate to enhance edges and features.

As described above, reconstructed feature data may be input into an inference network unit. As such, compression performance in terms of machine task accuracy (e.g., classification accuracy) is an important consideration in feature compression. According to the techniques herein, a feature inverse conversion engine may be configured to determine distortion due to compressed feature data being further compressed by a compression engine and reduce distortion in reconstructed feature data. For example, as described above, compressed feature data may be compressed by utilizing ITU-T H.266 (or another video standard), the SSFC described in Zhang, or an intra feature codec. Each of these techniques are not lossless and may introduce distortion. It should be noted that the techniques described herein are not limited to a particular feature conversion engine and compression engine. Further, the techniques described herein are not limited to a particular feature inverse conversion engine.

As described above, video coding may utilize intra prediction, uni-prediction inter prediction, and bi-prediction inter prediction. Typically, a picture coded using only intra prediction is referred to as an I picture, a picture which may utilize intra prediction and uni-prediction is referred to as a P picture, and a picture which may utilize intra prediction, uni-prediction, and bi-prediction is referred to as a B picture (i.e., a picture type may be one of an I picture, a P picture, or a B picture). As further described above, in video coding the degree of quantization may alter the rate-distortion (i.e., bit-rate vs. quality of video) and the degree of quantization may be modified by adjusting a quantization parameter (QP). That is, a QP value may be set in order to achieve a target bit-rate. According to the techniques herein, at the output of a video decoder, a distortion recovery engine may be provided, where a distortion recovery engine is trained to determine and mitigate distortion based on a picture type and a particular quantization parameter (or target bit-rate). That is, when a feature map is compressed, compression distortion present in the feature map may be based on the picture type and the quantization parameter (and/or target bit rate) used to code the compressed feature map. Thus, according to the techniques herein, an MSFR may have different models depending on the encoding picture type and target bitrate (or QP value), where each model is trained with different training data. During recovery of the feature map data, depending on the encoding picture type and target bitrate (or QP value), a different MSFR model may be selected and compression distortion may be reduced using the distortion recovery engine.

FIG.23illustrates an example of a coding system including distortion recovery engines according to the techniques herein. As illustrated inFIG.23, at the output of video decoder300, one of distortion recovery engines2400A-2400C may be selected based on a picture type and the output of a distortion recovery engines2400A-2400C is input into feature inverse conversion engine1200. Feature inverse conversion engine1200may correspond to feature inverse conversion engines described above. As further illustrated inFIG.23, quantization data (e.g., a QP value, a target bit rate, etc.) is input into each distortion recovery engines2400A-2400C. It should be noted that each of distortion recovery engines2400A-2400C may be described as having a distortion recovery engine for each group of QP values. That is, in one example, there may be M×N distortion recovery engines, where each distortion recovery engine corresponds to a different encoding picture type and a target bitrate (or QP) group, where M, N respectively represent the number of encoding picture types and target bitrate (or QP) groups. It should be noted that, in one example, one target bitrate (or QP) group represents a specific bitrate range or QP value range and there is no overlap of bitrates (or QP values) between groups, i.e., target bitrates (or QP values provided for a video coding standard) are partitioned into N groups. Each of the M×N distortion recovery engines may be trained for a particular encoding picture type and target bitrate (or QP value) combination and store corresponding parameters.

In some examples, a distortion recovery engine may include an L layered residual network or a dense network. In other examples, another kind of network may be used.FIG.24is a block diagram illustrating an example of a distortion recover engine according to one or more techniques of this disclosure. In the example illustrated inFIG.24, distortion recover engine2400is a residual network. In the example illustrated inFIG.24, a block of two of res2d k3 n256 operations with a skip connection is repeated to generate an L layered residual network. The number of blocks and layers may correspond to a particular architecture (e.g., L=2*(1+N)). For example, K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition,” arXiv:1512.03385v1, 10 Dec. 2015, describes a residual network with a 34 layer architecture formed using 2 layer blocks. It should be noted that the techniques herein may utilize various network architectures. For example, a distortion recovery engine may include dense net which includes L(L+1)/2 directly connected layers. For example, a distortion recovery engine may include a VGG (Visual Geometry Group architecture. For example, VGG-16 or VGG-19, which respectively include 16 and 19 layers. Further, a distortion recovery engine may include an inception architecture or a GoogLeNet architecture. It should be noted that inFIG.24, the example distortion recovery network the number of channels is aligned with the feature map. In the example illustrated inFIG.24, picture type and a target bitrate (or QP value) are input into distortion recover engine2400, such that for each of the res2d k3 n256 operations, the kernel values for the associated convolution operations are set for a picture type and target bitrate range or QP value group based on training. That is, according to the techniques herein, for a coding system, there exist more than one trained MSFR models, each of which may be related with the specific target bitrate (or QP) range and encoding picture type.

In one example, N target bitrate (or QP) range groups are defined before training and, each group may have representative bitrates or QP values. For example, a QP range of 40 to 45 may have one of 40, 41, 42, 43, 44, and 45 as a representative QP value. In one example, the representative QP value for a QP group may be the mean QP value. In one example, the representative QP value for a QP group may be the minimum QP value. In one example, the representative QP value for a QP group may be the maximum QP value. In one example, the representative QP value for a QP group may be the median QP value. In one example, training may be based on the following process:1. End-to-End training of coding system. That is, determine values and parameters for a feature conversion engine and a feature inverse conversion engine.a. In one example, training may use a relatively high target bitrate (or relatively small QP value, e.g., a minimum QP value allowed by a video coding standard, or a value in the lower third of values).2. Generate SSFC output (i.e., distorted training data) for each encoding picture type and target bitrate (or QP) group using a representative bitrate or QP values for each target bitrate (QP) group.3. Train distortion recovery engine with distorted training data generated at step 2.a. Set feature conversion engine and a feature inverse conversion engine values as the training result of step 1.b. For each encoding type and target bitrate (or QP) group, distortion recovery engine is trained separately.i. Train M×N distortion recovery engines separately with the training data specific to the encoding picture type and the target bitrate (or QP) group.

It should be noted that although in the example described above, target bitrates (or QP values), are used, in other examples, other video characteristics information (e.g., chroma subsampling, etc.) may be used. Further, in one example, luma QP values may be used and in other examples, luma and/or chroma QP values may also be used.

In one example, a distortion recovery engine may be configured for one QP value for each encoding picture type. That is, a QP fusion engine may be utilized with a distortion recovery engine.FIG.25illustrates an example of a QP fusion engine2500utilized with a distortion recovery engine2400.FIG.26illustrates an example of a QP fusion engine. As illustrated inFIG.26, a (l,w,h) dimensional input tensor filled with input QP value is constructed where w, h represent width, height of input feature map, and l represents the number of channels of input tensor is one. This tensor is concatenated with the feature map having a (c,w,h) dimensional tensor, where w, h, and c represent width, height, and the number of channels of the feature map, thereby a (c+l,w,h) dimensional tensor is generated. In this case, switching to a particulate distortion recovery engine (e.g.,2400A-2400C) occurs per picture and is determined based on encoding picture type and the target bitrate (or QP value) for the picture. The following pseudo code illustrates an example of generating a QP fusion output tensor the dimension of which is (c+l,w,h). In the pseudo code, the input of QP fusion network is feature_map (c,w,h) and scalar value representing QP (or bitrate). Using QP value (bitrate), QP_map tensor with (l,w,h) dimension is constructed. The output tensor is generated by concatenating feature_map and QP_map.Pseudo code for QP fusion network#feature_map.shape: (c, w,h)#qp_map.shape: (l,w,h)#output_tensor.shape: (c+1, w,h)QP_map=torch.tensor(np.full((l,w,h), QP)Output=torch.cat ([feature_map, QP_map], 0)

In one example, when a QP fusion engine is utilized, training may be based on the following process:1. End-to-End training of coding system. That is, determine values and parameters for a feature conversion engine and a feature inverse conversion engine.a. In one example, training may use a relatively high target bitrate (or relatively small QP value, e.g., a minimum QP value allowed by a video coding standard, or in the lower third of value).2. Generate SSFC output (i.e., distorted training data) for each encoding picture type and target bitrate (or QP) group using a representative bitrate or QP values for each target bitrate (QP) group. Bitrate (or QP value) of each training data is appended as a component of training data.3. Train distortion recovery engine with distorted training data generated at step 2.a. Set initial network node values of feature conversion engine and a feature inverse conversion engine as the training result of step 1.b. For each encoding type, distortion recovery engine is trained using the training data generated at step 2. If training loss function has bitrate related loss, remove bitrate loss from training loss function. For example, if training loss function is Lagrange cost function of bit rate and detection (or classification) performance (like Equation (1), Equation (2) below) that does not have bitrate cost term is used for training.

It should be noted that, as with any learning-based approach, the selection of a loss function is critical for overall performance. In one example, a Lagrangian cost function (with parameter A) may be used when performing rate-distortion tradeoffs. For distortion, a corresponding task loss function may be used. Loss may be computed for each input feature (Featuren). Further, separate training may be carried out for each bitrate budget (Rbudget,n) resulting in a separate model for each rate point.

In one example, object detection training loss for a Featurenis:

As described above, with respect toFIG.18andFIG.19, a compression engine and a decompression engine may be an intra feature codec. In one example, according to the techniques herein, an intra feature codec may be extended to learn the compression distortion. In one example, according to the techniques herein, a decompression engine may include a distortion recovery engine in order to reduce the distortion due to quantization.FIG.27is a block diagram illustrating an example of a decompression engine including a distortion recovery engine in accordance with one or more techniques of this disclosure. In one example, distortion recovery engine2702may be trained based on the following process:1. End-to-End training of coding system. That is, determine values and parameters for a feature conversion engine and a feature inverse conversion engine.a. In one example, training may use a relatively high target bitrate.2. Generate SSFC output (i.e., distorted training data) for each encoding picture type and target bitrate.3. Train distortion recovery engine with distorted training data generated at step 2.a. Set initial network node values of feature conversion engine and a feature inverse conversion engine as the training result of step 1.b. For each encoding type, distortion recovery engine is trained using the training data generated at step 2. If training loss function has bitrate related loss, remove bitrate loss from training loss function. For example, if training loss function is Lagrange cost function of bit rate and detection (or classification) performance like (like Equation (1), Equation (2) below) that does not have bitrate cost term is used for training.

It should be noted that Misra further describes an example where the output of an inter-predictor network, i.e., a predictor is subtracted from the feature data, which then may be input into an intra feature codec, (e.g., the compression engine and decompression engine illustrated inFIGS.18and19). That is, a residue is may be coded using the techniques described above with respect toFIGS.18and19. The techniques described above with respect toFIG.27are equally applicable regardless of the input to an intra feature codec.

In this manner, video decoder and distortion recovery engine represents an example of a device configured to receive a bitstream including compressed feature data further coded according a video coding standard, wherein the compressed feature data is a tensor with channel, height, and weight dimensions, decode the bitstream according to the video coding standard, such that a decoded picture corresponds to a channel, determine a quantization parameter and a picture type for a decoded picture, select a distortion reduction engine based on the quantization parameter and the picture type, and apply the distortion reduction engine to the decoded picture.