Patent ID: 12244792

DETAILED DESCRIPTION

Embodiments of the present disclosure are now described.

FIG.1is a schematic diagram showing encoding and decoding processes, according to embodiments. In particular,FIG.1shows schematically an example of a closed-loop lossy signal encoding system.

At the encoder, the current frame of an input signal s is received and intra- or inter-predicted (e.g. predicted using signal values from within the current frame s or from one or more previous frames, ŝ, respectively), using a selected one of various predetermined prediction modes. In the example shown inFIG.1, inter-prediction is used. The prediction error (or ‘residual’), e, is transformed, quantized and reconstructed to e′, which is then entropy coded without any further fidelity loss. The produced bitstream from the encoder can then be stored or transmitted over a network to the corresponding decoder.

At the decoder, the reconstructed error signal e′ is added to the reconstructed previous frame ŝ to reconstruct signal s′. Signal s′ is a lossy approximation of s. The reconstruction of signal s′ is also carried out at the encoder in order to use s′ as a reference signal for subsequent inter-prediction for the next frame of the input signal.

In known encoder systems, predictor adaptation functionality (e.g. the functionality deciding which prediction mode is to be used), and the decision of quantization step size, are implemented using a hand-crafted algorithm. This forms the basis for typical video coding standards, such as AVC/H.264, HEVC/H.265, VVC/H.266, AOMedia VP9, AV1, AV2, etc.

In contrast with such systems, the methods disclosed herein provide a pixel-to-decision-mode (PDM) artificial neural network which, once trained, replaces the hand-crafted algorithms for controlling the decisions on prediction modes and quantization settings. This is described in more detail below. With reference toFIG.1, the trained PDM network controls the predictor adaptation block and/or the quantization parameters of the quantizer. When the trained PDM network is deployed and used to determine prediction modes and/or quantization parameters, the remainder of the operation shown inFIG.1is unaltered. This ensures that existing decoders can operate as normal and no change is needed in the bitstream packaging, transport or decoding stages.

The embodiments depicted are applicable to batch processing, i.e. processing a group of images or video frames together without delay constraints (e.g. an entire video sequence), as well as to stream processing, i.e. processing only a limited subset of a stream of images or video frames, or even a select subset of a single image, e.g. due to delay or buffering constraints.

FIG.2shows examples of encoding modes associated with the AVC/H.264 standard. Encoding modes may also be referred to as “decision modes”, in that they comprise modes which are to be decided upon (e.g. selected) for use in encoding. The ‘P’ blocks correspond to inter-predicted blocks of different sizes. The ‘I’ blocks correspond to intra-predicted blocks from the causal neighborhood of blocks (whereby blocks are scanned via raster scan within each frame from the top left to the bottom right). The predefined intra-prediction directions are shown in the bottom-right part ofFIG.2.

The examples shown inFIG.2also depict transform and quantization parameters specified by the AVC/H.264 standard. The transform is a block matrix H, given by:

H=[a…b⋮⋱⋮c…d],
with coefficients a, b, c, d, . . . as specified by the standard. The 1D transform of data block D is given by: X=HD.

Quantization of the (i, j)th sample of X is carried out by:
Xq(i,j)=sign{X(i,j)}[(|X(i,j)|A(Q)+f2L)>>L]
where Q is the quantization level varying from 0 to Qmax, A(Q) is the quantization function, f is a parameter controlling the quantizer dead zone size, and L is the quantization step size.

The reconstruction of samples of data block D is given by:
Xr(i,j)=Xq(i,j)B(Q)
Dr=(HTXr+2N-1E)>>N
where B(Q) is the reconstruction function, selected such that A(Q)B(Q)G2reaches a maximum value and G is the squared norm of the rows of H, E is a matrix of unity values, and N controls the dynamic range of the reconstruction.

The objective of an optimized AVC/H.264 encoder is to convert the input video frame pixels into P or I decision modes, and quantize and encode the residual information such that the required rate, r, and the reconstructed signal distortion, d, are both minimized. This may be formulated as minimizing the regularized rate-distortion cost: r+λd, with λ being the selected regularization coefficient controlling the relative emphasis on rate or distortion. The value of λ as a regularization coefficient may be input to the workflow of the methods described herein, as described in more detail below. If distortion is quantified by multiple functions of measurements (e.g. d1, d2, . . . , dK), then K regularization coefficients (λ1, λ2, . . . , λK) may be used. These multiple distortion functions or measurements could correspond to multiple quality metrics, which may include, but are not limited to, a structural similarity index metric (SSIM), video multimethod assessment fusion (VMAF), mean opinion scores (MOS) from human viewers, and others.

FIGS.3and4show an example workflow of the training process of the neural (or ‘PDM’) network. An example training schematic (which is referred to inFIGS.3and4) is shown inFIG.5.

The first two steps of the overall training workflow (shown inFIG.3) involve the receipt (or selection) of the plurality of encoding modes of a standard coding framework, as well as the image/video sequences to use for training. Any image or video sequences may be used for training. Batches of training data can be selected in a deterministic or random/pseudo-random manner. The third step involves the receipt (or selection) of one or more regularization coefficients that are to be used together with the rate and quality loss functions of the training process. With these sets of inputs, the training workflow shown inFIG.4is executed, and the trained PDM network corresponding to these inputs is derived and can be subsequently be deployed for inference at a given rate and signal quality.

If more rate-quality points are desired (as shown inFIG.3), different regularization coefficients are derived and the process is repeated in order to derive a new instantiation of the PDM network corresponding to a different rate-quality point. If more standard encoding frameworks are desired to be used, then new sets of encoding modes are input and the process shown inFIG.3is repeated.

Referring to the training process shown inFIG.4, given the encoding modes (e.g. prediction, transform and/or quantization parameters) of a given encoder, the first two steps are to: (i) enable all operations to be carried out by differentiable functions; and (ii) approximate the required rate of each encoding mode with a differentiable function.

Concerning step (i), forward and inverse transforms and translational or directional prediction modes are differentiable functions as they involve linear operations. On the other hand, quantization to a set of discrete values is by its nature non-differentiable due to the use of rounding (e.g. a shift operation). Therefore, the noise of rounding is approximated with a differentiable function. For example, an additive uniform noise can be used:
Xq(i,j)=X(i,j)+Δx,
where Δx is additive independent and identically distributed (IID) uniform noise with support width chosen to correspond to the rounding by the integer division carried out when shifting by L bits in the original quantization operation.

Another source of initially non-differentiable operations relates to the hard assignment of prediction modes for inter- or intra-prediction that are derived adaptively based on the input signal blocks and the reference blocks. In such cases, the problem may be expressed as finding the decision mode vector m that minimizes the error of its corresponding inter-/intra-prediction:
m*=argminm{e(m)},
where e(m) expresses the prediction error under decision mode vector m.

Given that the argmin operation has zero gradients almost everywhere with respect to the input and is therefore not differentiable, such decision mode selection functions are converted into differentiable functions by using a straight-through estimator 1argminm(e), where the vector is 1 at index m* and zero everywhere else. −e is then transformed into a continuous categorical distribution that approximates the one-hot distribution by taking the softmax function: es=softmax(−e). The straight-through estimator can then be defined as 1m*=es+stop_gradient(1argminm(es). Therefore, in the forward pass (inference) the stop_gradient( ) function is treated as an identity function and the argmin is computed as normal. However, when training with back-propagation and stochastic gradient descent, in the backward pass only the gradient of the softmax( ) function, es, is used. This ensures that gradients can be back-propagated through such hard assignments of decision (e.g. encoding) modes.

Concerning step (ii), rate estimations can be approximated by continuous and differentiable functions. For example, the rate to encode a given symbol stream can be modelled as a variation of its entropy, or by using divisive normalization techniques that are shown to convert the produced symbols into normal distributions and then assuming independent univariate density models for each sub-band parameterized by a small set of Gaussian kernels.

The next two steps of the training workflow shown inFIG.4involve the establishment of the PDM network structure and the regularized rate-quality loss (RRQL) functions. Concerning the latter, usage of quality metrics such as (MAX_PSNR-PSNR) and (1-SSIM) can take place (where PSNR and SSIM are differentiable distortion functions and MAX_PSNR is the maximum PSNR value of interest), since these quality metrics comprise differentiable functions. The regularization coefficients are provided via external input, and act to balance the impact of each of these quality metrics. Minimizing such quality metrics enables the signal fidelity to be maximized. Other quality metrics can also be used (e.g. 100-VMAF), and the components within such metrics can be approximated by differentiable functions.

Concerning the PDM network structure, the intra-/inter-prediction and/or quantization modes of the utilized standard are converted into operations with parameters that are learnable. For example, the block displacement search of all P modes and the directional prediction modes are unrolled as sequential block difference operations, with different difference weightings. For H.264/AVC, these weights implement the intra/inter-prediction directions of the modes shown inFIG.2. The sequential block difference operations are implemented with trainable parameters for the difference position and relative impact (e.g. the impact of different block sizes vs prediction error is controlled via a regularization coefficient). The sum of absolute differences or the sum of squared differences may be used as a prediction error metric.

Prior to the PDM network structure, image pixels are preprocessed (e.g. filtered) using a neural network structure comprising a multilayer convolutional neural network. This neural pixel processing assists in preprocessing the input image content such that prediction and/or quantization mode selection is optimized according to end-to-end back-propagation learning that incorporates the decoder-side reconstruction stage. Both of these structures are shown inFIG.5, which depicts an example training schematic used within the workflows ofFIGS.3and4. The preprocessing neural network may be separate from the PDM network structure (as depicted in the example shown inFIG.5), or a single network structure may be configured to perform both functions.

When training image/video sequence data is inserted into the training schematic, training of the PDM network takes place based on back-propagation and stochastic gradient descent, and the use of the RRQL functions as losses. Approximated reconstructed frames can also enter the reference buffer (shown inFIG.5) to be used as references for future frames. This leads to the derivation of the trained PDM network. As shown inFIG.5, the output of the PDM network during training is a vector comprising soft decisions, e.g. the probability that a certain encoding mode will be chosen at a given moment.

A virtual encoder module is also used in the framework depicted inFIG.5. The virtual encoder module comprises one or more differentiable functions that are configured to emulate and/or approximate an encoding process (which may be the same as, or different from, the encoding process of an actual encoder). For example, the virtual encoder module may include a prediction component, a frequency transform component, a quantization and entropy encoding component, and a dequantization and inverse transform component. The virtual encoder module is configured to process the training image data using the soft decision modes specified by the PDM network. As such, the virtual encoder module may include differentiable functions that are configured to emulate the operations of the various encoding modes.

The purpose of the virtual encoder module is to emulate a typical image or video encoder using differentiable and learnable components, such as the layers of an artificial neural network. The frequency transform component is any variation of discrete sine or cosine transform or wavelet transform, or an atom-based decomposition. The dequantization and inverse transform component can convert the transform coefficients back into approximated pixel values. The main source of loss for the virtual codec module comes from the quantization component, which emulates any multi-stage dead zone or non-dead zone quantizer. Finally, the entropy coding component can be a continuous differentiable approximation of theoretical (ideal) entropy over transform values, or continuous differentiable representation of a Huffman encoder, an arithmetic encoder, a runlength encoder, or any combination of those that is also made to be context adaptive, i.e., looking at quantization symbol types and surrounding values (context conditioning) in order to utilize the appropriate probability model and compression method.

A rate loss may be calculated by minimizing the rate predicted from the virtual encoder model processing (e.g. virtually encoding and decoding) the quantized coefficients stemming from the soft decision modes. This rate loss is optimized as a function of the weights of the PDM network, by back-propagation using variations of gradient descent methods, in order to train the PDM network. Beyond its utility as a rate estimator, the virtual encoder module also produces distorted (or corrupted) image outputs, which can be used to obtain a quality loss function that is in turn useable to train the PDM network. As described above, the rate and/or quality loss functions can themselves be converted into differentiable operators.

In some embodiments, prior to or during the training of the PDM network itself, any parameters associated with the virtual encoder module can also be empirically tuned or trained with back-propagation and gradient descent methods. This can include, for example, training any transform and quantization parameters that are differentiable, and also the artificial neural network parameters used to represent the non-differentiable mathematical operations of the transform and quantization parts with differentiable approximations, e.g. by using the actual rate to encode the same pixels with a lossy JPEG, MPEG or AOMedia open encoder as a reference.

The PDM network as described herein can comprise any combination of weights connected in a network and having a non-linear function (akin to an activation function of an artificial neural network). An example of such connections and weights is shown inFIG.6(a). An example of the global connectivity between weights and inputs is shown inFIG.6(b). That is,FIG.6(a)shows a combination of inputs x0, . . . , x3with weight coefficients Θ and non-linear activation function g( ), andFIG.6(b)is a schematic diagram showing layers of interconnected activations and weights, forming an artificial neural network. Such examples are trained with back-propagation of errors computed at the output layer, using gradient descent methods. This is shown inFIG.6(c), which depicts schematically the back-propagation of errors δ from coefficient a0(2)of an intermediate layer to the previous intermediate layer using gradient descent.

An example multi-layer neural network processing pipeline is shown inFIG.7. In particular,FIG.7shows a cascade of convolutional (Cony (k×k)) and parametric ReLu (pReLu) layers of weights and activation functions mapping input pixel groups to transformed output pixel groups. Each layer receives codec settings as input, along with the representation from the previous layer. Convolutional layers extend the example ofFIG.6(b)to multiple dimensions, by performing convolution operations between multi-dimensional filters of fixed kernel size (k×k) with learnable weights and the inputs to the layer. In embodiments, some layers have dilated convolutions or pooling components to increase or decrease the resolution of the receptive field. The connectivity of the cascade of convolutional layers and activation functions can also include skip connections, as shown by the connection from the output of the leftmost “Cony (3×3)” layer ofFIG.7to the summation point ofFIG.7. In addition, the entirety of the cascade of multiple layers (also known as a deep neural network) is trainable end-to-end based on back-propagation of errors from the output layer backwards (e.g. as shown inFIG.6(c)), using gradient descent methods.

FIG.8shows an example workflow of the inference process of the PDM network (e.g. during deployment and after training of the PDM network).FIG.9shows an example inference schematic that is used within the workflow ofFIG.8.

As shown inFIGS.8and9, an image or video sequence to be compressed is input to a buffer, before a frame (or frame slice) is passed from the buffer to the trained PDM network. In this example, the trained PDM network includes the convolutional prefiltering neural network configured to prefilter the image data, as described above. The trained PDM network outputs encoding mode decisions to a standard-compliant external encoder, to enable the external encoder to encode the image data using the selected encoding mode(s). As such, in contrast to the training workflow, during inference the processing of the image data according to the selected encoding mode(s) is carried out by an actual encoding block that remains compliant to the standard. That is, the actual (standard-compliant) encoder replaces the virtual encoder that is used during training. Additionally, frames produced as reconstructed frames can be passed back to the buffer to be used as references for future frames.

FIG.10shows a method1000for processing image data using an artificial neural network. The method1000may be performed by a computing device, according to embodiments. The method1000may be performed at least in part by hardware and/or software. The processing is performed prior to encoding the processed image data with an external encoder. The external encoder is operable in a plurality of encoding modes. The plurality of encoding modes may be comprise a plurality of predefined encoding modes (e.g. associated with a particular image or video coding standard), according to embodiments. The artificial neural network may comprise a set of interconnected adjustable weights.

At item1010, image data representing one or more images is received at the artificial neural network. The image data may be retrieved from storage (e.g. in a memory), or may be received from another entity.

At item1020, the image data is processed using the artificial neural network (e.g. by applying the weights of the artificial neural network to the image data) to generate output data indicative of an encoding mode selected from the plurality of encoding modes of the external encoder. The artificial neural network is trained to select using image data an encoding mode of the plurality of encoding modes of the external encoder using one or more differentiable functions configured to emulate an encoding process.

At item1030, the generated output data is outputted from the artificial neural network to the external encoder to enable the external encoder to encode the image data using the selected encoding mode.

In embodiments, the plurality of encoding modes of the external encoder comprises a plurality of prediction modes for encoding image data using predictive coding. The plurality of prediction modes relate to intra-predication and/or inter-prediction.

In embodiments, one or more of the plurality of encoding modes of the external encoder comprises a plurality of quantization parameters useable by the external encoder to encode image data.

In embodiments, the plurality of encoding modes are associated with an image and/or video coding standard.

In embodiments, each of the plurality of encoding modes of the external encoder generates (e.g. enables the generation of) an encoded bitstream having a format that is compliant with an image and/or video coding standard.

In embodiments, the neural network is configured to select the encoding mode from the plurality of encoding modes based on image content of the received image data. For example, the neural network may be configured to select the encoding mode based on pixel data.

In embodiments, the neural network is trained using one or more differentiable functions configured to emulate operations associated with the plurality of encoding modes.

In embodiments, the neural network is trained to optimize a rate score indicative of the bits required by the external encoder to encode output pixel representations generated using the encoding modes. In embodiments, the rate score is calculated using one or more differentiable functions configured to emulate an encoding process. In embodiments, the rate score is calculated using a differentiable rate loss function. In embodiments, the output pixel representations are generated at the neural network.

In embodiments, the neural network is trained to optimize a quality score indicative of the quality of reconstructed pixel representations generated using the encoding modes. In embodiments, the quality score is calculated using one or more differentiable functions configured to emulate an encoding process. In embodiments, the quality score is calculated using a differentiable quality loss function. In embodiments, the quality score is indicative of at least one of: signal distortion in the reconstructed pixel representations; and loss of perceptual and/or aesthetic quality in the reconstructed pixel representations. In embodiments, the reconstructed pixel representations are generated using the artificial neural network.

In embodiments, the neural network is trained using one or more regularization coefficients corresponding to a desired rate-quality operational point.

In embodiments, the method1000further comprises determining one or more loss functions (e.g. errors) based on the generated output data, and adjusting the neural network using back-propagation of values of the one or more loss functions.

In embodiments, the method1000comprises preprocessing (e.g. pre-filtering) the image data prior to processing the image data using the neural network. The preprocessing operation may be performed using an artificial neural network.

In embodiments, the method1000further comprises, at the external encoder, receiving the output data from the neural network, and encoding the image data using the selected encoding mode to generate an encoded bitstream. In embodiments, the encoded bitstream is compliant with an image and/or video coding standard that is associated with the plurality of encoding modes from which the encoding mode is selected.

FIG.11shows a method1100of configuring an artificial neural network for processing image data prior to encoding using an external encoder. The method1100may be performed by a computing device, according to embodiments. The method1100may be performed at least in part by hardware and/or software. The external encoder is operable in a plurality of encoding modes. The plurality of encoding modes may be comprise a plurality of predefined encoding modes (e.g. associated with a particular image or video coding standard), according to embodiments. The artificial neural network comprises a set of interconnected adjustable weights. The neural network is arranged to select using image data an encoding mode of the plurality of encoding modes of the external encoder using one or more differentiable functions configured to emulate an encoding process.

At item1110, image data representing one or more images is received at the neural network.

At item1120, the image data is processed using the neural network to generate output data indicative of an encoding mode of the plurality of encoding modes. In embodiments, the generated output data comprises a soft output, e.g. a vector comprising soft decisions on encoding modes. For example, the generated output data may indicate the probability that a given encoding mode will be selected at a given moment.

At item1130, one or more loss functions are determined based on the generated output data. The one or more loss functions may comprise a quality loss function associated with reconstructed pixel representations generated using the encoding modes (e.g. emulating reconstructed pixel representations derivable by an external decoder). The one or more loss functions may additionally or alternatively comprise a rate loss function associated with output pixel representations generated using the encoding modes (e.g. emulating output pixel representations that may be generated by the external encoder).

At item1140, the weights of the artificial neural network are adjusted based on back-propagation of values of the one or more loss functions.

Embodiments of the disclosure include the methods described above performed on a computing device, such as the computing device1200shown inFIG.12. The computing device1200comprises a data interface1201, through which data can be sent or received, for example over a network. The computing device1200further comprises a processor1202in communication with the data interface1201, and memory1203in communication with the processor1202. In this way, the computing device1200can receive data, such as image data or video data, via the data interface1201, and the processor1202can store the received data in the memory1203, and process it so as to perform the methods of described herein, including processing image data prior to encoding using an external encoder, and optionally encoding the processed image data.

Each device, module, component, machine or function as described in relation to any of the examples described herein may comprise a processor and/or processing system or may be comprised in apparatus comprising a processor and/or processing system. One or more aspects of the embodiments described herein comprise processes performed by apparatus. In some examples, the apparatus comprises one or more processing systems or processors configured to carry out these processes. In this regard, embodiments may be implemented at least in part by computer software stored in (non-transitory) memory and executable by the processor, or by hardware, or by a combination of tangibly stored software and hardware (and tangibly stored firmware). Embodiments also extend to computer programs, particularly computer programs on or in a carrier, adapted for putting the above described embodiments into practice. The program may be in the form of non-transitory source code, object code, or in any other non-transitory form suitable for use in the implementation of processes according to embodiments. The carrier may be any entity or device capable of carrying the program, such as a RAM, a ROM, or an optical memory device, etc.

The present disclosure provides a neural network design that processes input image content and emulates the mathematical operations of the encoding modes (e.g. prediction modes) of a given image/video standard while allowing for the weights and/or connections of the neural network to be trained with back-propagation and stochastic gradient descent. The process that maps encoding decision modes to bitrate is converted into a set of differentiable mathematical functions that can provide for an estimate of the required encoding bitrate. Further, the reconstruction process of the decoder-side image or video frame (or slice) is, in some examples, converted into a set of differentiable mathematical functions that can provide an approximation of the reconstructed pixels. Loss functions are used to estimate quality loss between the input pixels and the approximate reconstruction, and a combination of such loss functions with the rate estimate form a final regularized loss function. End-to-end training of the pixel-to-decision-mode neural network is performed using back-propagation and stochastic gradient descent to minimize the regularized loss function for a set of training data.

As such, the methods disclosed herein convert the encoding process into a fully-neural system, while allowing for backward compatibility to existing standards. This replaces the hand-crafted design previously used for encoder optimization and mode selection with an end-to-end learnable system based on data that is more flexible than the previous hand-crafted methods. Further, the disclosed methods treat the normative part of the standard (e.g. the operations of the decision modes themselves and lossless encoding to produce a standard-compliant bitstream) as an implementation ‘substrate’ that remains unaltered, thereby ensuring standard-compliance.

While the present disclosure has been described and illustrated with reference to particular embodiments, it will be appreciated by those of ordinary skill in the art that the disclosure lends itself to many different variations not specifically illustrated herein.

Where in the foregoing description, integers or elements are mentioned which have known, obvious or foreseeable equivalents, then such equivalents are herein incorporated as if individually set forth. Reference should be made to the claims for determining the true scope of the present invention, which should be construed so as to encompass any such equivalents. It will also be appreciated by the reader that integers or features of the disclosure that are described as preferable, advantageous, convenient or the like are optional and do not limit the scope of the independent claims. Moreover, it is to be understood that such optional integers or features, whilst of possible benefit in some embodiments of the disclosure, may not be desirable, and may therefore be absent, in other embodiments.