Patent Description:
Intra-prediction modes are widely used in picture and video coding. In video coding, intra-prediction modes compete with other prediction modes such as inter-prediction modes such as motion-compensated prediction modes. In intra-prediction modes, a current block is predicted on the basis of neighboring samples, i.e. samples already encoded as far as the encoder side is concerned, and already decoded as far as the decoder side is concerned. Frankly speaking, neighboring sample values are extrapolated into the current block so as to form a prediction signal for the current block with the prediction residual being transmitted in the datastream for the current block. The better the prediction signal is, the lower the prediction residual is and, accordingly, a lower number of bits is necessary to code the prediction residual.

In order to be effective, several aspects should be taken into account in order to form an effective frame work for intra-prediction in a block-wise picture coding environment. For instance, the larger the number of intra-prediction modes supported by the codec, the larger the side information rate consumption is in order to signal the selection to the decoder. On the other hand, the set of supported intra-prediction modes should be able to provide a good prediction signal, i.e. a prediction signal resulting in a low prediction residual.

The present application seeks to provide an intra-prediction mode concept allowing for a more efficient compression of a block-wise picture codec if using the improved intra-prediction mode concept.

Document <CIT> may be construed to disclose an image encoding method that includes: encoding an image including a block by performing, in at least one of intra prediction, inter prediction, and an in-loop filter, a non-linear process by which the input-output relationship becomes non-linear; and encoding an operation parameter of a non-linear filter to be used in the non-linear process.

Document <CIT> may be construed to disclose a technique in which, in an image encoding/decoding device, the prediction direction in a target block, i.e., a block which becomes the target of the intraframe prediction processing, is estimated by taking advantage of pre-encoded blocks which are adjacent to the target block. First, as edge information on decoded images on the adjacent blocks, intensities and angles of the edges are calculated. Next, of the degrees of likelihood calculated with respect to each prediction direction by taking advantage of this edge information and, e.g., a neural network, the prediction direction whose degree of likelihood is the highest is employed as the prediction direction in the target block. Also, a variable-length code table is dynamically created based on the estimated result, which allows a significant reduction in the prediction-direction representing code amount. Document "<NPL> may be construed to disclose a technique for applying deep neural network to improve the state-of-the-art intra prediction. Considering the characteristics of block-based video coding framework, the document proposes a fully connected network for intra prediction where all layers except non-linear ones are fully connected. In the proposed network, the inputs are multiple reference lines of the current block and the output is the prediction for the block. When compared with the traditional intra prediction method, the richer context of current block is exploited.

Document "<NPL>, may be construed to disclose an intra prediction convolutional neural network (IPCNN) for intra prediction, which exploits the rich context of the current block and therefore is capable of improving the accuracy of predicting the current block. Meanwhile, the predictions of the three nearest blocks can also be refined. CNNs are applied to intra prediction for HEVC.

This object is achieved by the subject-matter of the independent claims of the present application. Further developments are set forth in the dependent claims.

Basis for the claimed invention may be found in <FIG> and the associated description. The other Figures relate to unclaimed examples, which provide information useful for understanding the claimed invention. These are described in the following with respect to the Figures, among which:.

In the following, various embodiments are described which assist in achieving a more effective compression when using intra-prediction. Some embodiments achieve the compression efficiency increase by spending a set of intra-prediction modes which are neural network-based. The latter ones may be added to other intra-prediction modes heuristically designed, for instance, or may be provided exclusively. Other embodiments use a neural network in order to perform a selection among a plurality of intra-prediction modes. And even other embodiments make use of both of the just-discussed specialties.

In order to ease the understanding of the following embodiments of the present application, the description starts with a presentation of possible encoders and decoders fitting thereto into which the subsequently outlined embodiments of the present application could be built. <FIG> shows an apparatus for block-wise encoding a picture <NUM> into a datastream <NUM>. The apparatus is indicated using reference sign <NUM> and may be a still picture encoder or a video encoder. In other words, picture <NUM> may be a current picture out of a video <NUM> when the encoder <NUM> is configured to encode video <NUM> including picture <NUM> into datastream <NUM>, or encoder <NUM> may encode picture <NUM> into datastream <NUM> exclusively.

As mentioned, encoder <NUM> performs the encoding in a block-wise manner or block-base. To this, encoder <NUM> subdivides picture <NUM> into blocks a team in units of which encoder <NUM> encodes picture <NUM> into datastream <NUM>. Examples of possible subdivisions of picture <NUM> into blocks <NUM> are set out in more detail below. Generally, the subdivision may end-up into blocks <NUM> of constant size suggest an array of blocks arranged in rows and columns or into blocks <NUM> of different block sizes such as by use of a hierarchical multi-tree subdivisioning with starting the multi-tree subdivisioning from the whole picture area of picture <NUM> or from a pre-partitioning of picture <NUM> into an array of tree blocks wherein these examples shall not be treated as excluding other possible ways of subdivisioning picture <NUM> into blocks <NUM>.

Further, encoder <NUM> is a predictive encoder configured to predictively encode picture <NUM> into datastream <NUM>. For a certain block <NUM> this means that encoder <NUM> determines a prediction signal for block <NUM> and encodes the prediction residual, i.e. the prediction error at which the prediction signal deviates from the actual picture content within block <NUM>, into datastream <NUM>.

Encoder <NUM> may support different prediction modes so as to derive the prediction signal for a certain block <NUM>. The prediction modes, which are of importance in the following embodiments, are intra-prediction modes according to which the inner of block <NUM> is predicted spatially from neighboring, already encoded samples of picture <NUM>. The encoding of picture <NUM> into datastream <NUM> and, accordingly, the corresponding decoding procedure, may be based on a certain coding order <NUM> defined among blocks <NUM>. For instance, the coding order <NUM> may traverse blocks <NUM> in a raster scan order such as row-wise from top to bottom with traversing each row from left to right, for instance. In case of hierarchical multi-tree based subdivisioning, raster scan ordering may be applied within each hierarchy level, wherein a depth-first traversal order may be applied, i.e. leaf notes within a block of a certain hierarchy level may precede blocks of the same hierarchy level having the same parent block according to coding order <NUM>. Depending on the coding order <NUM>, neighboring, already encoded samples of a block <NUM> may be located usually at one or more sides of block <NUM>. In case of the examples presented herein, for instance, neighboring, already encoded samples of a block <NUM> are located to the top of, and to the left of block <NUM>.

Intra-prediction modes may not be the only ones supported by encoder <NUM>. In case of encoder <NUM> being a video encoder, for instance, encoder <NUM> may also support intra-prediction modes according to which a block <NUM> is temporarily predicted from a previously encoded picture of video <NUM>. Such an intra-prediction mode may be a motion-compensated prediction mode according to which a motion vector is signaled for such a block <NUM> indicating a relative spatial offset of the portion from which the prediction signal of block <NUM> is to be derived as a copy. Additionally or alternatively, other non-intra-prediction modes may be available as well such as inter-view prediction modes in case of encoder <NUM> being a multi-view encoder, or non-predictive modes according to which the inner of block <NUM> is coded as is, i.e. without any prediction.

Before starting with focusing the description of the present application onto intra-prediction modes, a more specific embodiment for a possible block-based encoder, i.e. for a possible implementation of encoder <NUM>, as described with respect to <FIG> with then presenting two corresponding embodiments for a decoder fitting to <FIG>, respectively.

<FIG> shows a possible implementation of encoder <NUM> of <FIG>, namely one where the encoder is configured to use transform coding for encoding the prediction residual although this is nearly an example and the present application is not restricted to that sort of prediction residual coding. According to <FIG>, encoder <NUM> comprises a subtractor <NUM> configured to subtract from the inbound signal, i.e. picture <NUM> or, on a block basis, current block <NUM>, the corresponding prediction signal <NUM> so as to obtain the prediction residual signal <NUM> which is then encoded by a prediction residual encoder <NUM> into a datastream <NUM>. The prediction residual encoder <NUM> is composed of a lossy encoding stage 28a and a lossless encoding stage 28b. The lossy stage 28a receives the prediction residual signal <NUM> and comprises a quantizer <NUM> which quantizes the samples of the prediction residual signal <NUM>. As already mentioned above, the present example uses transform coding of the prediction residual signal <NUM> and accordingly, the lossy encoding stage 28a comprises a transform stage <NUM> connected between subtractor <NUM> and quantizer <NUM> so as to transform such a spectrally decomposed prediction residual <NUM> with a quantization of quantizer <NUM> taking place on the transformed coefficients where presenting the residual signal <NUM>. The transform may be a DCT, DST, FFT, Hadamard transform or the like. The transformed and quantized prediction residual signal <NUM> is then subject to lossless coding by the lossless encoding stage 28b which is an entropy coder entropy coding quantized prediction residual signal <NUM> into datastream <NUM>. Encoder <NUM> further comprises the prediction residual signal reconstruction stage <NUM> connected to the output of quantizer <NUM> so as to reconstruct from the transformed and quantized prediction residual signal <NUM> the prediction residual signal in a manner also available at the decoder, i.e. taking the coding loss is quantizer <NUM> into account. To this end, the prediction residual reconstruction stage <NUM> comprises a dequantizer <NUM> which perform the inverse of the quantization of quantizer <NUM>, followed by an inverse transformer <NUM> which performs the inverse transformation relative to the transformation performed by transformer <NUM> such as the inverse of the spectral decomposition such as the inverse to any of the above-mentioned specific transformation examples. encoder <NUM> comprises an adder <NUM> which adds the reconstructed prediction residual signal as output by inverse transformer <NUM> and the prediction signal <NUM> so as to output a reconstructed signal, i.e. reconstruct examples. This output is fed into a predictor <NUM> of encoder <NUM> which then determines the prediction signal <NUM> based thereon. It is predictor <NUM> which supports all the prediction modes already discussed above with respect to <FIG> also illustrates that in case of encoder <NUM> being a video encoder, encoder <NUM> may also comprise an in-loop filter <NUM> with filters completely reconstructed pictures which, after having been filtered, form reference pictures for predictor <NUM> with respect to inter-predicted block.

As already mentioned above, encoder <NUM> operates block-based. For the subsequent description, the block bases of interest is the one subdividing picture <NUM> into blocks for which the intra-prediction mode is selected out of a set or plurality of intra-prediction modes supported by predictor <NUM> or encoder <NUM>, respectively, and the selected intra-prediction mode performed individually. Other sorts of blocks into which picture <NUM> is subdivided may, however, exist as well. For instance, the above-mentioned decision whether picture <NUM> is inter-coded or intra-coded may be done at a granularity or in units of blocks deviating from blocks <NUM>. For instance, the inter/intra mode decision may be performed at a level of coding blocks into which picture <NUM> is subdivided, and each coding block is subdivided into prediction blocks. Prediction blocks with encoding blocks for which it has been decided that intra-prediction is used, are each subdivided to an intra-prediction mode decision. To this, for each of these prediction blocks, it is decided as to which supported intra-prediction mode should be used for the respective prediction block. These prediction blocks will form blocks <NUM> which are of interest here. Prediction blocks within coding blocks associated with inter-prediction would be treated differently by predictor <NUM>. They would be inter-predicted from reference pictures by determining a motion vector and copying the prediction signal for this block from a location in the reference picture pointed to by the motion vector. Another block subdivisioning pertains the subdivisioning into transform blocks at units of which the transformations by transformer <NUM> and inverse transformer <NUM> are performed. Transformed blocks may, for instance, be the result of further subdivisioning coding blocks. Naturally, the examples set out herein should not be treated as being limiting and other examples exist as well. For the sake of completeness only, it is noted that the subdivisioning into coding blocks may, for instance, use multi-tree subdivisioning, and prediction blocks and/or transform blocks may be obtained by further subdividing coding blocks using multi-tree subdivisioning, as well.

A decoder or apparatus for block-wise decoding fitting to the encoder <NUM> of <FIG> is depicted in <FIG>. This decoder <NUM> does the opposite of encoder <NUM>, i.e. it decodes from datastream <NUM> picture <NUM> in a block-wise manner and supports, to this end, a plurality of intra-prediction modes. All the other possibilities discussed above with respect to <FIG> are valid for the decoder <NUM>, too. To this, decoder <NUM> may be a still picture decoder or a video decoder and all the prediction modes and prediction possibilities are supported by decoder <NUM> as well. The difference between encoder <NUM> and decoder <NUM> lies, primarily, in the fact that encoder <NUM> chooses or selects coding decisions according to some optimization suggest, for instance, in order to minimize some cost function which may depend on coding rate and/or coding distortion. One of these coding options or coding parameters may involve a selection of the intra-prediction mode to be used for a current block <NUM> among available or supported intra-prediction modes. The selected intra-prediction mode may then be signaled by encoder <NUM> for current block <NUM> within datastream <NUM> with decoder <NUM> redoing the selection using this signalization in datastream <NUM> for block <NUM>. Likewise, the subdivisioning of picture <NUM> into blocks <NUM> may be subject to optimization within encoder <NUM> and corresponding subdivision information may be conveyed within datastream <NUM> with decoder <NUM> recovering the subdivision of picture <NUM> into blocks <NUM> on the basis of the subdivision information. Summarizing the above, decoder <NUM> may be a predictive decoder operating on a block-bases and besides intra-prediction modes, decoder <NUM> may support other prediction modes such as inter-prediction modes in case of, for instance, decoder <NUM> being a video decoder. In decoding, decoder <NUM> may also use the coding order <NUM> discussed with respect to <FIG> and as this coding order <NUM> is obeyed both at encoder <NUM> and decoder <NUM>, the same neighboring samples are available for a current block <NUM> both at encoder <NUM> and decoder <NUM>. Accordingly, in order to avoid unnecessary repetition, the description of the mode of operation of encoder <NUM> shall also apply to decoder <NUM> as far the subdivision of picture <NUM> into blocks is concerned, for instance, as far as prediction is concerned and as far as the coding of the prediction residual is concerned. Differences lie in the fact that encoder <NUM> chooses, by optimization, some coding options or coding parameters and signals within, or inserts into, datastream <NUM> the coding parameters which are then derived from the datastream <NUM> by decoder <NUM> so as to redo the prediction, subdivision and so forth.

<FIG> shows a possible implementation of the decoder <NUM> of <FIG>, namely one fitting to the implementation of encoder <NUM> of <FIG> as shown in <FIG>. As many elements of the encoder <NUM> of <FIG> are the same as those occurring in the corresponding encoder of <FIG>, the same reference signs, provided with an apostrophe, are used in <FIG> in order to indicate these elements. In particular, adder <NUM>', optional in-loop filter <NUM>' and predictor <NUM>' are connected into a prediction loop in the same manner that they are in encoder of <FIG>. The reconstructed, i.e. dequantized and retransformed prediction residual signal applied to added <NUM>' is derived by a sequence of entropy decoder <NUM> which inverses the entropy encoding of entropy encoder 28b, followed by the residual signal reconstruction stage <NUM>' which is composed of dequantizer <NUM>' and inverse transformer <NUM>' just as it is the case on encoding side. The decoder's output is the reconstruction of picture <NUM>. The reconstruction of picture <NUM> may be available directly at the output of adder <NUM>' or, alternatively, at the output of in-loop filter <NUM>'. Some post-filter may be arranged at the decoder's output in order to subject the reconstruction of picture <NUM> to some post-filtering in order to improve the picture quality, but this option is not depicted in <FIG>.

Again, with respect to <FIG> the description brought forward above with respect to <FIG> shall be valid for <FIG> as well with the exception that merely the encoder performs the optimization tasks and the associated decisions with respect to coding options. However, all the description with respect to block-subdivisioning, prediction, dequantization and re-transforming is also valid for the decoder <NUM> of <FIG>.

Before proceeding with the description of possible embodiments of the present application, some notes shall be made with respect to the above examples. Although not explicitly mentioned above, it is clear that block <NUM> may have any shape. It may be, for instance, of rectangular or quadratic shape. Moreover, although the above description of the mode of operation of encoder <NUM> and decoder <NUM> often mentioned a "current block" <NUM> it is clear that encoder <NUM> and decoder <NUM> act accordingly for each block for which an intra-prediction mode is to be selected. As described above, there may be other blocks as well, but the following description focuses on those blocks <NUM> into which picture <NUM> is subdivided, for which an intra-prediction mode is to be selected.

In order to summarize the situation for a certain block <NUM> for which an intra-prediction mode is to be selected, reference is made to <FIG> shows a current block <NUM>, i.e. a block currently to be encoded or decoded. <FIG> shows a set <NUM> of neighboring samples <NUM>, i.e. samples <NUM> with spatially neighbor block <NUM>. The samples <NUM> within block <NUM> are to be predicted. The prediction signal to be derived is, thus, a prediction for each sample <NUM> within block <NUM>. As already discussed above, a plurality <NUM> of prediction modes are available for each block <NUM> and if block <NUM> is to be intra-predicted, this plurality <NUM> of modes merely comprises inter-prediction modes. A selection <NUM> is performed at encoder and decoder side in order to determine one of the intra-prediction modes out of the plurality <NUM> to be used to predict <NUM> the prediction signal for block <NUM> on the basis of the neighboring sample set <NUM>. The embodiments described further below differ with respect to the available intra-prediction modes <NUM> and the mode of operation with respect to selection <NUM> suggest, for instance, whether side information is set in the datastream <NUM> with respect to selection <NUM> with respect to block <NUM> or not. The description of these embodiments, however, starts with a concrete description providing mathematical details. According to this initial embodiment, the selection for a certain block <NUM> to be intra-predicted is associated with corresponding side information signalization <NUM> and the datastream and the plurality <NUM> of intra-prediction modes comprises a set <NUM> of neural network-based intra-prediction modes as well as a set <NUM> of further intra-prediction mode of heuristic design. One of the intra-prediction modes of mode <NUM> may, for instance, be a DC prediction mode according to which some mean value is determined on the basis of the neighboring sample set <NUM> and this mean value is assigned to all samples <NUM> within block <NUM>. Additionally or alternatively, set <NUM> may comprise inter-prediction modes which may be called angular inter-prediction modes according to which sample values of the neighboring sample set <NUM> are copied into block <NUM> along a certain intra-prediction direction with this intra-prediction direction differing among such angular intra-prediction modes. <FIG> shows that the datastream <NUM> comprises, in addition to the optionally present side information <NUM> concerning the selection <NUM> out of the plurality <NUM> of intra-prediction modes, a portion <NUM> into which the prediction residual encoded which coding may, as discussed above, optionally involve transform coding with quantization in transform domain.

In particular, in order to ease the understanding of the following description of a specific embodiment of the present application, <FIG> shows the general mode of operation for an intra-prediction block at encoder and decoder. <FIG> shows block <NUM> along with the neighboring samples set <NUM> on the basis of which the intra-prediction is performed. It should be noted that this set <NUM> may vary among the intra-prediction modes of the plurality <NUM> of intra-prediction modes in terms of cardinality, i.e. the number of samples of set <NUM> actually used according to the respective intra-prediction mode for determining the prediction signal for block <NUM>. This is, however, for ease of understanding, not depicted in <FIG> shows that encoder and decoder have one neural network <NUM><NUM> to <NUM>KB-<NUM> for each of the neural network-based intra-prediction modes of set <NUM>. Set <NUM> is applied to the respective neural network so as to derive the corresponding intra-prediction mode among set <NUM>. Besides this, <FIG> rather representatively shows one block <NUM> as providing on the basis of the input, namely the set <NUM> of neighboring samples, the one or more prediction signals of the one or more intra-prediction modes of set <NUM>, e.g. the DC mode prediction signal and/or angular intra-prediction mode prediction signal. The following description will show as to how the parameters for the neural networks <NUM>i with i=<NUM>. KB-<NUM> may advantageously be determined. The specific embodiment set out hereinafter, also provides encoder and decoder with another neural network <NUM> which is dedicated to provide a probability value for each neural network-based intra-prediction mode within set <NUM> on the basis of a set <NUM> of neighboring samples which may or may not coincide with set <NUM>. The probability values thus provided when the neural network <NUM> assists in rendering the side information <NUM> for the mode selection more effective. For instance, in the embodiment described below, it is assumed that a variable length code is used to point to one of the intra-prediction modes and at least as far as set <NUM> is concerned, the probability values provided by the neural network <NUM> enable to use the variable length code within the side information <NUM> as an index into an ordered list of intra-prediction modes ordered according to the probability values output by neural network <NUM> for the neural network-based intra-prediction modes within set <NUM>, thereby optimizing or reducing the code rate for the side information <NUM>. To this, as depicted in <FIG>, the mode selection <NUM> is effectively performed depending on both, the probability values provided by the further neural network <NUM> as well as the side information <NUM> within datastream <NUM>.

Let <MAT> be a block of a video frame, i.e. block <NUM>. Assume that B has M pixels. For a fixed color component, let im be the content of a video signal on B. We regard im as an element of <MAT>. Assume that there exists a neighbourhood <MAT> of B that has L pixels and on which an already reconstructed image <MAT> is available, i.e. sample sets <NUM> and <NUM> although they may alternatively differ. By an intra-prediction-function, we mean a function <MAT>. We regard F(rec) as a predictor for im.

What is described next is an algorithm to design, via a data-driven optimization approach, intra-prediction-functions for several blocks B that may occur in a typical hybrid video coding standard, namely set <NUM>. In order to achieve that goal, we took the following main design features into account:.

In the next four sections, a possibility is to describe how one may deal with each of these requirements. More precisely, in section <NUM>, we shall describe how to deal with the first item. In section1. <NUM>, it is described how to handle items <NUM> to <NUM>. In section1. <NUM>, it is described how to take item <NUM> into account. Finally, in section1. <NUM>, it is described how to deal with the last item.

A data driven approach to determine unknown parameters that are used in a video codec is usually set up as an optimization algorithm that tries to minimize a predefined loss function on a given set of training examples. Typically, for a numerical optimization algorithm to work in practice, the latter loss function should satisfy some smoothness requirements.

On the other hand, a video encoder like HEVC performs best when it makes its decisions my minimizing the Rate-Distortion costs D + λ · R. Here, D is the reconstruction error of the decoded video signal and R is the rate, i.e. the number of bits needed to code the video signal. Moreover, <MAT> is a Lagrangian Parameter that depends on the chosen Quantization Parameter.

The true function D + λ · R is typically very complex and is not given by a closed expression on can feed a data driven optimization algorithm with. Thus, we approximate either the whole function D + λ·R or at least the rate function R by a piecewise smooth function.

More precisely, as before let B be a given block <NUM>/ of a video frame <NUM> and let im be the corresponding video signal on B in a fixed color component. Assume that B has M pixels. Then for a prediction candidate <MAT>, we consider the prediction residue res: = <MAT>. For a given Quantization Parameter and a given transform, let R(res) be the rate that a true video encoder needs to signal the quantized transform of res. Moreover, let D(res) be the reconstruction error that arises by dequantization and inverse transform of res. Then we want to determine functions H, <MAT> which are piecewise smooth such that H(res) serves as a good approximation of D(res) + λ · R(res) and such that R̃(res) serves as a good approximation of R(res).

We fix some <MAT> and fix predefined "architectures", i.e. piecewise smooth functions <MAT> and then seek Φ<NUM>, <MAT>, such that we model our functions H and R̃ as <MAT>.

In order to determine the weights Φ<NUM> and Φ<NUM>, on a typical encoder that uses the given hybrid video coding standard we collected a huge set of training examples of prediction residues resj , <MAT>, and the corresponding Rate-Distortion values (D + λR)(resj) respectively only the rate values R(resj) for some finite large index set <IMG>. Then we try to find Φ<NUM> and Φ<NUM> such that they minimize or at least make small the expressions <MAT>.

For that task, we usually use a (stochastic) gradient descent approach.

In this section we describe the algorithm that we set up to design KB intra-predictions for a given block B <NUM>, the ones of st <NUM>, and area Brec <NUM> for already reconstructed samples.

We assume that we are given a predefined "architecture" of our predictions. By this we mean that for some fixed <MAT> we are given a function <MAT> and that we want to determine "weights" <MAT> such that our intra predictions are given as <MAT> where for <MAT> we put <MAT>.

The following section provides details in this regard. The functions in (<NUM>) define the neural network <NUM><NUM>- <NUM>KB -<NUM> in <FIG>.

Next, we model the signalization cost for the intra modes that we try to design by using a second parameter-dependent function <MAT>.

Again, for <MAT>, we define <MAT> by <MAT>.

Again, an example is given in section <NUM> with the function of (<NUM>) representing neural network <NUM> of <FIG>.

We assume that we are given a function <MAT>.

This function, for instance, defines a VLC code length distribution used for side information <NUM>. i.e. the code lengths assocaited by side information <NUM> with cad ponite more of set <NUM>.

For the time being, the k-th component <MAT> of <MAT> shall model the number of bits needed to signal the k-th intra mode that we train.

If R̃ is the function defined in section <NUM>, for given <MAT>, reconstructed image <MAT> and original image <MAT>, we let <MAT> denote the smallest k ∈ {<NUM>,. , KB} with the property that <MAT> for all l ∈ {<NUM>,.

Since <IMG> models the true number of bits for the singalization of an intra mode, its gradient is either zero or undefined. Thus, M allone does not suffice to optimize the weights ΨB via a gradient-descent based algorithm. Thus, we also invoke the cross entropy of an intra mode by transforming the function <MAT> into a probability distribution using the softmax-function. We recall the definition of the latter function. For <MAT> let xi denote the i-th component of x. Then the softmax function <MAT> is defined as <MAT>.

For gradient updates, we will try to minimize the sum of the rate of the residue and the cross entropy of the mode kopt with respect to the latter probability distribution. Thus we define our loss function LossB for the block B as <MAT> <MAT> where <MAT>.

Given the loss function in (<NUM>), we determine the weights <MAT> by a data driven optimization. Thus, if for a finite, large index set <IMG> we are given a set of training examples <MAT> of images imi on B and corresponding reconstructed images reci on Brec, we apply an optimization algorithm, for example based on the (stochastic) gradient descent method, to find weights <MAT> that minimize the expression <MAT>.

<NUM> Specification of the Functions <MAT> and <MAT>.

In this section, we define the form of the functions <MAT> and <MAT> more precisely. Again, recall that some define neural networks <NUM> and <NUM>. Each of these functions consists of a sequence of compositions of functions which are either: <NUM>) An affine transofrmation Aff or <NUM>) A non-linear activation function Act.

By an affine transformation <MAT>, we mean a map that is of the form <MAT> where <MAT> is a linear transformation, i.e. satisfies <MAT> for all <MAT>, and where <MAT>. Each linear map <MAT> is completely determined by a matrix in <MAT>, i.e. corresponds uniquely to a vector <MAT>. Each affine function <MAT> is thus completely determined by m · n + n weights, i.e. by a vector <MAT>. For each <MAT> we shall write AffΘ for the unique affine transformation that corresponds to Θ in the aforementioned way.

By a non-linear activation function <MAT>, we mean a function of the form <MAT>.

Here, (Act(x))i denotes the i-th component of Act(x) and xi denotes the i-th component of x. Finally, <MAT> my be of the form <MAT> or of the form <MAT> although these examples shall not be interpreted as limiting embodiments of the present application to these explicit examples. Other formulae may be used as well such as ρ(z) = log(<NUM> + ez) or any other non-linear function. <MAT> may alternatively be a piecewise smooth function, for example.

Our function <MAT> now looks as follows. We assume that for a fixed <MAT> we are given <MAT> and <MAT> with m<NUM> = L, nk = M, such that <MAT>.

Here, <MAT> and <MAT> are as in (<NUM>). Then, for <MAT> with <MAT>, we define <MAT> <MAT> would, thus, describe a neural network <NUM>i parametrized using paramters Θ. It would be a sequence of linear functions AffΘj and non-linear functions ρ, which, in the present example, are applied alternatingly in the sequence, wherein the parameters Θ comprise the linear function weights in AffΘj In the sequence of linear functions AffΘj and non-linear functions ρ, the pairs of a linear function AffΘj followed by non-linear function ρ would represent a neuron layer, for example, such as the j-th layer, with the number of predecessor nodes preceding this neuron layer j in feed-forward direction of the neural network being determined by dimension m of AffΘj, the number of columns of AffΘj, and the number of neurons of the neuron layer j itself being determined by dimension n of AffΘj, the number of its rows. Each row of AffΘj incorpartes the weights controlling as to how strong a signal strength respectively activation of each of the m predecessor neurons is forwarded to the respective neuron of the neuron layer j which corresponds to the respective row. ρ controlls for each neuron of neuron layer j the non-linear mapping of its linear combination of forwarded predecessor neuron activations onto its own activation. In the above example, there are k such neuron layers. The number of neurons per layer may vary. The number of nuron layers k may vary among the various neural networks <NUM>j, i.e. for different j. Note, that the non-linear function might vary per neurion layer or even per neuron or at some other units.

Similarly, our function <MAT> looks as follows. We assume that for a fixed <MAT> we are given <MAT> and <MAT> with m<NUM>, = L, nk, = KB, such that <MAT>.

Here, <MAT> and <MAT> are as in (<NUM>). Then, for <MAT> <MAT>, with <MAT>, we define <MAT> <MAT> would, thus, describe a neural network <NUM> parametrized using paramters Ψ. It would be a sequence of linear functions AffΨj and non-linear functions ρ, just as it has been described above with respect to the neuron layers concerning the prediction signal computation. The number of neuron layers k' of neural network <NUM> may differ from one or more of the number of neuron layers k of neural networks <NUM>i,.

We extended the algorithm of the previous section so that we can train predictions that complement already existing intra predictions.

Namely, let <MAT> be a set of fixed intra prediction functions that are already available. For example, <MAT> can consist of the DC- or Planar-prediction of HEVC and angular predictions defined according to HEVC; all those predictions may also include a preliminary smoothing of the reconstructed samples. Moreover, we assume that we are given a function <MAT> such that <IMG>(im, rec, k) models the loss of the k-th intra prediciont function <MAT> applied to rec given the original image im.

Then we extend the loss function from (<NUM>) to the loss function <MAT> <MAT>.

Keeping the notations from the end of the previous section, we determine weights <MAT> by minimizing <MAT> on a large set of training examples.

For that purpose, we typically firstly find the weights by optimizing (<NUM>) and then initialize with those weights to find the weights that optimize (<NUM>).

In this section we described how, in the training of our predictions, one may take into account that in a typical video coding standard it is usually possible to split a block into smaller subblocks in various ways and to perform an intra prediction on the smaller subblocks.

Namely, assume that for some <MAT> we are given a set <MAT> of admissible blocks <MAT> together with a set of areas <MAT> such that each <MAT> is a neighborhood of Bi. Typically, <MAT> is a union of two rectangles left and above Bi.

We assume that there exists a block <MAT> such that Bi ⊆ Bmax for each i ∈ {<NUM>,. Let P(BL) be the power set of <IMG>. Then for B ∈ <IMG> we assume that a set <MAT> is given such that for each <MAT> the block B can be written as a disjoint union <MAT>.

For a given color component, let im be an image on Bmax, which, by restriction, we regard as an image im|Bl on Bi for each <MAT>. Moreover, assume that there exists a reconstructed image rec on <MAT>, which, by restriction, we regard as an image <MAT> on <MAT> for each <MAT>.

Keeping the notations of section <NUM>, for each B ∈ <IMG> we seek <MAT> as the set of weights for KB intra prediction-functions <MAT> and we seek <MAT> as weights for the mode prediction function GB. We determine these weights for all B ∈ <IMG> jointly as follows. For B ∈ <IMG> and given sets of weights <MAT>, B' ∈ <IMG>, B' ⊆ B, <NUM> ≤ k ≤ KB,, we put <MAT>.

Moreover, for B' ⊂ B we define ΘB |B' ⊂ ΘB as <MAT>.

As in section <NUM>, we assume that for each B ∈ <IMG> a possibly empty set <MAT> of intra prediction functions is available. We let <MAT>.

Then we define a loss-function <MAT>as follows. We have an ordering ≤ on the set <IMG> via the inclusion of sets. Let <MAT> be the set of all minimal elements in <IMG>. For B ∈ <IMG> we put <MAT> where the latter function is as in (<NUM>).

Next, let B ∈ <IMG> and assume that LossB,total is already defined for all B' ∈ <IMG> with B' ⊆ B.

Finally, given a fixed set of training examples <MAT> of images imi on Bmax, we determine ΘBmax, ΨBmax by minimizing or at least making small the expression <MAT>.

We typically initialize the weights <MAT>, ΨB by firstly minimizing (<NUM>) for each B ∈<IMG> individually.

We consider a hybrid video coding standard in which for a given color component the content of a video signal on a given block <MAT> is to be generated by a decoder. Let M be the number of pixels of B. Moreover, let <MAT> be a fixed neighbourhood of B such that the decoder has at its disposal a reconstructed image rec on Brec. Let L be the number of pixels of Brec. Then we regard rec as an element of <MAT>. We assume that the codec operates by predictive coding on the current block B <NUM>. Then we claim copyright for the following steps that a decoder can perform in order to generate a prediction signal pred on B, which we regard as an element of <MAT>:.

Here, the functions <MAT> are as in (<NUM>).

Then the decoder reconstructs from the bitstream <NUM> a unique index i ∈ {<NUM>,. , KB} that Is also part of datastream <NUM> and puts m: = σ(i).

In the code design to parse the latter index i, it is required that the number of bits needed to signal an index i<NUM> ∈ {<NUM>,. , KB} is less or equal than the number of bits to signal an index i<NUM> ∈ {<NUM>,. , KB} if σ(i<NUM>) ≤ σ(i<NUM>) and if all involved underlying probabilities used by the entropy coding engine are set to equal probability.

If Option One in step <NUM> is true and if the decoder has determined the index m according to the previous step <NUM>, the decoder generates <NUM> the prediction signal pred ∈ <MAT> as <MAT>, i.e. using the selected neural network <NUM>m. Then the decoder proceeds as in the underlying hybrid video coding standard using pred as prediction signal.

The integration of intra prediction functions whose design is based on a data driven learning approach into an existing hybrid video codec. The description had two main parts. In the first part, we described a concrete algorithm for an offline training of intra prediction functions. In the second part, we described how a video decoder may use the latter prediction functions in order to generate a prediction signal for a given block.

Thus, what has been described above in sections <NUM> to <NUM>, is, inter alia, an apparatus for block-wise decoding a picture <NUM> from a datastream <NUM>. The apparatus <NUM> supports a plurality of intra-prediction modes comprising, at least, a set <NUM> of intra-prediction modes according to which the intra-prediction signal for a current block <NUM> of the picture <NUM> is determined by applying a first set <NUM> of neighboring samples of the current block <NUM> onto a neural network <NUM>i. The apparatus <NUM> is configured to select <NUM> for the current block <NUM> one intra-prediction mode out of the plurality <NUM> of intra-prediction modes and predicts <NUM> the current block <NUM> using the one intra-prediction mode, namely using the corresponding neural network <NUM>m having been selected. Although the decoder presented in section <NUM>, had intra-prediction modes <NUM> within the plurality <NUM> of intra-prediction modes supported in addition to the neural network-based ones in set <NUM>, this has been merely an example and needs not to be the case. Further, the above description in sections <NUM> and <NUM> may be varied in that decoder <NUM> does not use, and does not comprise, the further neural network <NUM>. With respect to the optimization described above, this means that the second adder in the inner quality presented in section <NUM> for finding-out <MAT> would not have to be a concatenation of a function MB applied onto any probability value neural network function GB. The optimization algorithm of what, rather, determines suitable parameters for the neural networks <NUM>i in a manner so that the frequency of selection would appropriately follow a code rate indication of MB. For instance, the decoder <NUM> could decode from datastream <NUM> an index for block <NUM> using a variable length code, the code length of which are indicated in MB, and the decoder <NUM> would perform the selection <NUM> based on this index. The index would be part of the side information <NUM>.

A further alternative to the description brought forward above in section <NUM> is that the decoder <NUM> may alternatively derive a ranking among the set <NUM> of neural network-based intra-prediction modes depending on a first portion of the datastream which relates to a neighborhood of the current block <NUM> in order to obtain an ordered list of intra-prediction modes with selecting the intra-prediction mode finally to be used out of the ordered list of intra-prediction modes depending on a second portion of the datastream other than the first portion. The "first portion" may, for instance, relate to a coding parameter or prediction parameter related to one or more block neighboring current block <NUM>. The "second portion" may then be an index, for instance, pointing into, or being an index of, the neural network-based intra-prediction mode set <NUM>. When construed in alignment with above-outlined section <NUM>, the decoder <NUM> comprises the further neural network <NUM> which determines, for each intra-prediction mode of the set <NUM> of intra-prediction modes, a probability value by applying set <NUM> of neighboring samples thereonto and ordering these probability values in order to determine a rank for each intra-prediction mode of set <NUM>, thereby obtaining an ordered list of intra-prediction modes. An index in the datastream <NUM> as part of side information <NUM> is then used as an index into the ordered list. Here, this index may be coded using variable length code for which MB indicates the code length. And as explained above in section <NUM>, in item 4i, according to a further alternative embodiment, decoder <NUM> may use the just-mentioned probability values determined by the further neural network <NUM> for each neural network-based intra-prediction mode of set <NUM> so as to efficiently perform entropy coding of the index into set <NUM>. In particular, the symbol alphabet of this index which is part of the side information <NUM> and used as an index into set <NUM>, would comprise a symbol or value for each of the modes within set <NUM>, and the probability values provided by neural network <NUM> would, in case of neural network <NUM> design according to the above description, provide probability values which would lead to efficient entropy coding in that these probability values closely represent the actual symbol statistics. For this entropy coding, arithmetic coding could be used, for instance, or probability interval partitioning entropy (PIPE) coding.

Favorably, no additional information is necessary for any of the intra-prediction modes of set <NUM>. Each neural network <NUM>i, once advantageously parametrized for encoder and decoder in accordance with, for example, the above description in sections <NUM> and <NUM>, derives the prediction signal for the current block <NUM> without any additional guidance in the datastream. As already denoted above, the existence of other intra-prediction modes besides the neural network-based ones in set <NUM> is optional. They have been indicated above by set <NUM>. In this regard, it should be noted that one possible way of selecting set <NUM>, i.e. the set of neighboring samples forming the input for the intra-prediction <NUM>, may be selected such that this set <NUM> is the same for the intra-prediction modes of set <NUM>, i.e. the heuristic ones, with set <NUM> for the neural network-based intra-prediction modes being larger in terms of the number of neighboring samples included in set <NUM> and influencing the intra-prediction <NUM>. In other words, the cardinality of set <NUM> may be larger for neural network-based intra-prediction modes <NUM> compared to the other modes of set <NUM>. For instance, set <NUM> of any intra-prediction mode of set <NUM> may merely comprise neighboring samples along a one-dimensional line extending alongside to sides of block <NUM> such as the left hand one and the upper one. Set <NUM> of the neural network-based intra-prediction modes may cover an L-shaped portion extending alongside the just-mentioned sides of block <NUM> but being wider than just one-sample wide as set <NUM> for the intra-prediction modes of set <NUM>. In this manner, neural network-based intra-prediction modes may result into a better intra-prediction with a correspondingly lower prediction residual.

As described above in section <NUM>, the side information <NUM> conveyed in the datastream <NUM> to an intra-predicted block <NUM> may comprise a fleck which generally indicates whether the selected intra-prediction mode for block <NUM> is member of set <NUM> or member of set <NUM>. This fleck is, however, merely optional with side information <NUM> indicating, for instance, an index into a whole plurality <NUM> of intra-prediction modes including both sets <NUM> and <NUM>.

The just-discussed alternatives are, in the following, briefly discussed with respect to the <FIG>. define both, decoder and encoder concurrently, namely in terms of their functionality with respect to an intra-predicted block <NUM>. The differences between the encoder mode of operation and the decoder mode of operation with respect to an intra-coded block <NUM> is, on the one hand, the fact that the encoder performs all or at least some of the intra-prediction modes <NUM> available so as to determine at <NUM> a best one in terms of, for instance, some cost function minimizing sense, and that the encoder forms data stream <NUM>, i.e., codes date there into, while the decoder derives the data therefrom by decoding and reading, respectively. <FIG> shows the mode of operation for the above-outlined alternative according to which a flag 70a within the side information <NUM> for block <NUM> indicates whether the intra-prediction mode determined to be the best mode for block <NUM> by the encoder in step <NUM>, is within set <NUM>, i.e., is neural network based intra-prediction mode, or within set <NUM>, i.e., one of the non-neural network based intra-prediction modes. The encoder inserts flag 70a into data stream <NUM> accordingly, while the decoder retrieves it therefrom. <FIG> assumes that the determined intra-prediction mode <NUM> is within set <NUM>. The separate neural network <NUM> then determines a probability value for each neural network based intra-prediction mode of set <NUM> and using these probability values set <NUM> or, to be more precise, the neural network based intra-prediction modes therein are ordered according to their probability values such as in descending order of their probability values, thereby resulting into an ordered list <NUM> of intra-prediction modes. An index 70b being part of the side information <NUM> is then coded by the encoder into data stream <NUM> and decoded therefrom by the decoder. The decoder, accordingly, is able to determine which set of sets <NUM> and <NUM>. The intra-prediction mode to be used for block <NUM> is located in, and to perform the ordering <NUM> of set <NUM> in case of the intra-prediction mode to be used being located in set <NUM>. It might be that an index is also transmitted in data stream <NUM> in case of the determined intra-prediction mode being located in set <NUM>. Thus, the decoder is able to generate the prediction signal for block <NUM> using the determined intra-prediction mode by controlling the selection <NUM> accordingly.

<FIG> shows an alternative according to which the flag 70a is not present in data stream <NUM>. Instead, the ordered list <NUM> would not only comprise the intra-prediction modes of set <NUM>, but also intra-prediction modes of set <NUM>. The index within side information <NUM> would be an index into this greater ordered list and indicate the determined intra-prediction mode, i.e., the one determined be optimization <NUM>. In case of neural network <NUM> providing a probability value for the neural network based intra-prediction mode within <NUM> only, the ranking between intra-prediction modes of set <NUM> relative to the intra-prediction modes of set <NUM> may be determined by other means such as inevitably arranging the neural network based intra-prediction modes of set <NUM> to precede the modes of set <NUM> in the order list <NUM> or to arrange them alternatingly relative to each other. That is, the decoder is able to derive the index from data stream <NUM>, use the index <NUM> as in index into the order list <NUM> with deriving the order list <NUM> from the plurality of intra-prediction modes <NUM> using the probability values output by neural network <NUM>. <FIG> shows a further variant. <FIG> show a case of not using flag 70a, but the flag could be used alternatively. The issue which <FIG> is directed pertains to the possibility that neither encoder nor decoder uses neural network <NUM>. Rather, the ordering <NUM> is derived by other means such as coding parameters conveyed within data stream <NUM> with respect to one or more neighboring blocks <NUM>, i.e., portions <NUM> of a data stream <NUM> which pertains to such one or more neighboring blocks.

<FIG> shows a further variant of <FIG>, namely the one according to which the index 70b is coded using entropy coding and decoded from data stream <NUM> using entropy decoding, commonly denoted using reference sign <NUM>. The sample statistics or the probability distribution used for the entropy coding <NUM> is controlled by the probability values output by neural network <NUM> as explained above, this renders the entropy coding of index 70b very efficient.

For all embodiments 7a to 7d it is true that set <NUM> modes may not be present. Accordingly, the respective module <NUM> may be missing and flag 70a would be unnecessary anyway.

Further, although not shown in any Fig., it is clear that the mode selection <NUM> at the encoder and decoder could be synchronized to each other even without any explicit signaling <NUM>, i.e., without spending any side information. Rather, the selection could be derived from other means such as by taking inevitably the first one of the ordered list <NUM>, or by deriving the index into the order list <NUM> on the basis of coding parameters relating to one or more neighboring blocks. <FIG> shows an apparatus for designing the set of intra-prediction modes of set <NUM> to be used for the block-based picture coding. The apparatus <NUM> comprises a parameterizable network <NUM> which inherits or comprises parameterizable versions of neural networks <NUM><NUM> to <NUM>KB-<NUM> as well as neural network <NUM>. Here, in <FIG>, depicted as individual units, i.e., neural network <NUM><NUM> for providing the probability value for neural network based intra-prediction mode <NUM> to neural network <NUM>KB-<NUM> for providing the probability value associated with the neural network based intra-prediction mode KB-<NUM>. The parameters <NUM> for parametrizing neural networks <NUM> and the parameters <NUM> for parametrizing neural networks <NUM><NUM> to <NUM>KB-<NUM> are input or applied to respective parameter inputs of these neural networks by an updater <NUM>. Apparatus <NUM> has access to a reservoir or a plurality of picture test blocks <NUM> along with corresponding neighboring samples sets <NUM>. Pairs of these blocks <NUM> and their associated neighboring sample sets <NUM> are sequentially used by apparatus <NUM>. In particular, a current picture test block <NUM> is applied to parameterizable neural network <NUM> so that neural networks <NUM> provide a prediction signal <NUM> for each neural network based intra-prediction mode of set <NUM>, and each neural network <NUM> provides a probability value for each of these modes. To this end, these neural networks use their current parameters <NUM> and <NUM>.

In the above description rec has been used to denote the picture test block <NUM>, and <MAT> is the prediction residual <NUM> for mode B and the probability value is <MAT> is the probability value <NUM>. For each mode <NUM>. Kb-<NUM>, there is a cost estimator <NUM> comprised by apparatus <NUM> which computes a cost estimate for the respective mode on the basis of the prediction signal <NUM> obtained for the respective mode. In the above example, cost estimators <NUM> computed the cost estimates as indicated on the left and right hand sides of the inequality in section <NUM>. That is, here, the cost estimators <NUM> also used, for each mode, the corresponding probability value <NUM>. This needs not, however, to be case as already discussed above. The cost estimate, however, is in any case a sum of two add-ins, one of which is an estimate of the coding cost for the prediction residual indicated as the term with R̃ in the above inequality, and another add-in estimating the coding costs for indicating the mode. In order to compute the estimate for the coding cost related to the prediction residual, the cost estimators <NUM> also obtain the original content of the current picture test block <NUM>. The neural networks <NUM> and <NUM> had at their inputs applied thereto the corresponding neighboring sample sets <NUM>. The cost estimate <NUM> as output by cost estimators <NUM> is received by a minimum cost selector <NUM> which determines the mode minimizing or having minimum cost estimate associated therewith. In the above mathematical notation, this has been <MAT>. The updater receives this optimum mode and uses a coding cost function having a first add in forming residual rate estimate depending on the prediction signal <NUM> obtained for the intra-prediction mode of lowest coding estimate, and a second add-in forming a mode signaling side information rate estimate depending on the prediction signal and the probability value obtained for the intra-prediction mode of lowest coding cost estimate as indicated by selector <NUM>. As indicated above, this may be done using a gradient distant. The coding cost function is, thus, differentiable and in the above mathematical representation an example of this function was given in equation <NUM>. Here, the second add-in relating to the mode signaling side information rate estimate computed the cross entropy for the intra-prediction mode of lowest coding cost estimate.

Thus, the updater <NUM> seeks to update parameters <NUM> and <NUM> so as to reduce the coding cost function and then these updated parameters <NUM> and <NUM> are used by the parameterizable neural network <NUM> so as to process the next picture test block of the plurality <NUM>. As discussed above with respect to section <NUM>, there may be a mechanism controlling that primarily those pairs of picture test blocks <NUM> and their associated neighboring sample sets <NUM> are applied for the recursive update process for which the intra-prediction is, in rate distortion sense, preferably done without any block sub-division, thereby avoiding that the parameters <NUM> and <NUM> are optimized too much on the basis of picture test blocks for which, anyway, a coding in units of sub-blocks thereof is more cost effective.

So far, the above-discussed embodiments primarily concern cases where encoder and decoder had within their supported intra-prediction modes <NUM> a set of neural network-based intra-prediction modes. In accordance with the embodiments discussed with respect to <FIG> and <FIG> this needs not to be the case necessarily. <FIG> seeks to outline the mode of operation of an encoder and a decoder in accordance with an embodiment wherein the description thereof is provided in a manner focusing on the differences to the description brought forward above with respect to <FIG>. The plurality <NUM> of supported intra-prediction modes may or may not comprise neural network-based intra-prediction modes and may or may not comprise non-neural network-based intra-prediction modes. The modules <NUM> in <FIG> comprised by encoder and decoder, respectively, in order to provide for each of the supported modes <NUM> the corresponding prediction signal are, accordingly, not necessarily neural networks. As already denoted above, such intra-prediction modes may be neural network-based or they may be heuristically motivated and compute the prediction signal based on a DC intra-prediction mode or an angular intra-prediction mode or any other. Accordingly, these modules <NUM> could be denoted as prediction signal computer. Encoder and decoder in accordance with the embodiment of <FIG>, however, comprise a neural network <NUM>. The neural network <NUM> computes, on the basis of the neighboring sample set <NUM>, probability values for the supported intra-prediction modes <NUM> so that the plurality <NUM> of intra-prediction modes may be turned into the ordered list <NUM>. The index <NUM> within datastream <NUM> for block <NUM> points into this ordered list <NUM>. The neural network <NUM>, thus, assists in lowering the side information rate to be spent for the intra-prediction mode signalization.

<FIG> shows an alternative to <FIG> in that instead of the ordering, entropy de/encoding <NUM> of the index <NUM> is used with controlling the probability or simple statistics thereof, i.e. controlling the entropy probability distribution for entropy de/encoding in en/decoder, according to the probability values determined for the neural network <NUM> for each mode of plurality <NUM>.

<FIG> shows an apparatus for designing or parametrizing neural network <NUM>. It is, thus, an apparatus <NUM> for designing a neural network for assisting in selecting among a set <NUM> of intra-prediction modes. Here, for each mode of set <NUM> there is a corresponding neural network block together forming neural network <NUM> and the parametrizable neural network <NUM> of apparatus <NUM> is merely parametrizable with respect these blocks. For each mode, there is also the prediction signal computer <NUM> which needs, however, not to be parametrizable according to <FIG>. Thus, apparatus <NUM> of <FIG> computes costs estimates for each mode on the basis of the prediction signal <NUM> as computed by the corresponding prediction signal computer <NUM> and, optionally, on the basis of the corresponding probability value as determined by the corresponding neural network block for this mode. On the basis of the resulting costs estimates <NUM>, the minimum cost selector <NUM> selects the mode of the minimum cost estimate and the updater <NUM> updates the parameters <NUM> for the neural <NUM>.

The following is noted with respect to the description of <FIG> and <FIG> and <FIG>. A common feature of the embodiments of <FIG> and <FIG> which is also used by some of the embodiments of <FIG> was the fact that the probability values of the neural network values in order to improve or reduce the overhead associated with the side information <NUM> for signaling the mode determined on the encoder side at the optimization process <NUM> to the decoder. As indicated above with respect to the embodiments of <FIG>, however, it should be clear that the embodiments of <FIG> and <FIG> may be varied to the extent that no side information <NUM> is spent in datastream <NUM> with respect to the mode selection at all. Rather, the probability values output by neural network <NUM> for each mode may be used to synchronize the mode selection between encoder and decoder inevitably. In that case, there would be no optimization decision <NUM> at the encoder side with respect to the mode selection. Rather, the mode to be used among set <NUM> would be determined on encoder and decoder side in the same manner. A similar statement is true with respect to corresponding embodiments of <FIG> when varied so as to not use any side information <NUM> in datastream <NUM>. Back to the embodiments of <FIG> and <FIG>, it is interesting, however, that while the selection process <NUM> at the decoder side is dependent on the probability values output by the neural network in that the ordering or the probability distribution estimate dependency on the probability values varies the interpretation of the side information, as far as the encoder is concerned, the dependency on the probability values may not only affect the coding of the side information <NUM> into datastream <NUM> using, for instance, a respective variable length coding of an index into the ordered list or using entropy en/decoding with a probability distribution estimation depending on the neural network's probability values, but also the optimization step <NUM>: here, the code rate for transmitting side information <NUM> may be taken into account and may, thus, influence the determination <NUM>.

The inventive encoded data stream can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

Claim 1:
Method for training a first neural network (<NUM>) for each of a set (<NUM>) of intra-prediction modes for block-based picture coding for determining an intra-prediction signal (<NUM>) for a current block (<NUM>) by applying a first set (<NUM>) of neighboring samples (<NUM>) of the current block (<NUM>) onto the first neural network (<NUM>, <NUM><NUM>, <NUM>KB-<NUM>) for the respective intra-prediction mode, comprising
applying, for each of the set (<NUM>) of intra-prediction modes, a first set (<NUM>) of neighboring samples (<NUM>) neighboring a first picture test block (<NUM>) onto the first neural network (<NUM>) for the respective intra-prediction mode so as to obtain, for each of the set (<NUM>) of intra-prediction modes, a prediction signal (<NUM>) for the first test block, and a second set (<NUM>) of neighboring samples neighboring the first picture test block onto a second neural network (<NUM>, <NUM><NUM>, <NUM>KB-<NUM>) so as to obtain, for each of the set (<NUM>) of intra-prediction modes, a probability value (<NUM>) indicative of a probability of the respective intra-prediction mode;
determining (<NUM>), for each of the set (<NUM>) of intra-prediction modes, a coding cost estimate (<NUM>) for coding costs related to prediction error coding and mode signalization using the prediction signal (<NUM>) obtained for the respective intra-prediction mode;
updating (<NUM>) first parameters (<NUM>) of the first neural network (<NUM>) for each of the set of intra-prediction modes and second parameters (<NUM>) of the second neural network (<NUM>) so as to reduce a coding cost function having (i) a first addend forming a residual rate estimate depending on the prediction signal (<NUM>) obtained for an intra-prediction mode of the lowest coding cost estimate and (ii) a second addend forming a mode signaling side information rate estimate depending on the prediction signal (<NUM>) and the probability value (<NUM>) obtained for the intra-prediction mode of the lowest coding cost estimate; and
sequentially repeating the applying, determining and updating for a plurality of picture tests blocks with using the first and second parameters (<NUM>, <NUM>) as updated for one picture test block for the application of the next picture test block of the plurality of picture test blocks.