Patent Description:
Current video coding standards such as Versatile Video Coding (VVC) allow to switch interpolation filters for motion compensation depending on a motion vector (MV) resolution, which can be signalled at a block level. In case the MV (or the MV difference MVD) is coded in a particular resolution, e.g. half-sample accuracy, a different interpolation filter could be used to interpolate certain fractional sample positions.

Other new features are reference picture resampling, as disclosed in <NPL>), or adaptive resolution change, as disclosed in <NPL>), which allow referencing previously coded pictures in motion compensated inter-picture prediction having a different resolution/size than the current picture. In order to do that, the referenced picture area is resampled to a block having the same size as the current block. This may lead to the case where several fractional positions are obtained by using different phases of an interpolation filter.

For example, when a 16x16 block references a picture having one fourth of the size in every dimension, the corresponding 4x4 block in the referenced picture needs to be upsampled to 16x16, which can involve different interpolation filters for specific fractional positions / phases. when the MV is signalled in an accuracy that is associated with a smoothing interpolation filter, this filter is applied to the phase this smoothing filter is associated with in reference picture upsampling, while a sharpening interpolation filter may be applied to the other phases.

This mixup can produce visible artefacts and, thus, results into a poorer motion compensated inter predictor which, in turn, increases the prediction error and the bitrate needed to code the prediction residual to achieve equal quality.

The present application seeks to result in a more efficient video coding concept supporting reference picture resampling. This object is achieved by the subject matter of the independent claims.

Preferred embodiments of the present application are described below with respect to the figures, among which:.

The following description of the figures starts with a presentation of a description of an encoder and a decoder of a block-based predictive codec for coding pictures of a video in order to form an example for a coding framework into which embodiments of the present invention may be built in. The respective encoder and decoder are described with respect to <FIG>. Thereinafter the description of embodiments of the concept of the present invention is presented along with a description as to how such concepts could be built into the encoder and decoder of <FIG> and <FIG>, respectively, although the embodiments described with the subsequent <FIG> and following, may also be used to form encoders and decoders not operating according to the coding framework underlying the encoder and decoder of <FIG> and <FIG> such as without intra coded blocks competing with inter coded blocks within one picture, and/or such as without transform based residual coding or the like.

<FIG> shows an apparatus for predictively coding a picture <NUM> into a data stream <NUM> exemplarily using transform-based residual coding. The apparatus, or encoder, is indicated using reference sign <NUM>. <FIG> shows a corresponding decoder <NUM>, i.e. an apparatus <NUM> configured to predictively decode the picture <NUM>' from the data stream <NUM> also using transform-based residual decoding, wherein the apostrophe has been used to indicate that the picture <NUM>' as reconstructed by the decoder <NUM> deviates from picture <NUM> originally encoded by apparatus <NUM> in terms of coding loss introduced by a quantization of the prediction residual signal. <FIG> and <FIG> exemplarily use transform-based prediction residual coding, although embodiments of the present application are not restricted to this kind of prediction residual coding. This is true for other details described with respect to <FIG> and <FIG>, too, as will be outlined hereinafter.

The encoder <NUM> is configured to subject the prediction residual signal to spatial-to-spectral transformation and to encode the prediction residual signal, thus obtained, into the data stream <NUM>. Likewise, the decoder <NUM> is configured to decode the prediction residual signal from the data stream <NUM> and subject the prediction residual signal thus obtained to spectral-to-spatial transformation.

Internally, the encoder <NUM> may comprise a prediction residual signal former <NUM> which generates a prediction residual <NUM> so as to measure a deviation of a prediction signal <NUM> from the original signal, i.e. from the picture <NUM>. The prediction residual signal former <NUM> may, for instance, be a subtractor which subtracts the prediction signal from the original signal, i.e. from the picture <NUM>. The encoder <NUM> then further comprises a transformer <NUM> which subjects the prediction residual signal <NUM> to a spatial-to-spectral transformation to obtain a spectral-domain prediction residual signal <NUM>' which is then subject to quantization by a quantizer <NUM>, also comprised by the encoder <NUM>. The thus quantized prediction residual signal <NUM>" is coded into bitstream <NUM>. To this end, encoder <NUM> may optionally comprise an entropy coder <NUM> which entropy codes the prediction residual signal as transformed and quantized into data stream <NUM>. The prediction signal <NUM> is generated by a prediction stage <NUM> of encoder <NUM> on the basis of the prediction residual signal <NUM>" encoded into, and decodable from, data stream <NUM>. To this end, the prediction stage <NUM> may internally, as is shown in <FIG>, comprise a dequantizer <NUM> which dequantizes prediction residual signal <NUM>" so as to gain spectral-domain prediction residual signal <NUM>‴, which corresponds to signal <NUM>' except for quantization loss, followed by an inverse transformer <NUM> which subjects the latter prediction residual signal <NUM>‴ to an inverse transformation, i.e. a spectral-to-spatial transformation, to obtain prediction residual signal <NUM>"", which corresponds to the original prediction residual signal <NUM> except for quantization loss. A combiner <NUM> of the prediction stage <NUM> then recombines, such as by addition, the prediction signal <NUM> and the prediction residual signal <NUM>"" so as to obtain a reconstructed signal <NUM>, i.e. a reconstruction of the original signal <NUM>. Reconstructed signal <NUM> may correspond to signal <NUM>'. A prediction module <NUM> of prediction stage <NUM> then generates the prediction signal <NUM> on the basis of signal <NUM> by using, for instance, spatial prediction, i.e. intra-picture prediction, and/or temporal prediction, i.e. inter-picture prediction.

Likewise, decoder <NUM>, as shown in <FIG>, may be internally composed of components corresponding to, and interconnected in a manner corresponding to, prediction stage <NUM>. In particular, entropy decoder <NUM> of decoder <NUM> may entropy decode the quantized spectral-domain prediction residual signal <NUM>" from the data stream, whereupon dequantizer <NUM>, inverse transformer <NUM>, combiner <NUM> and prediction module <NUM>, interconnected and cooperating in the manner described above with respect to the modules of prediction stage <NUM>, recover the reconstructed signal on the basis of prediction residual signal <NUM>" so that, as shown in <FIG>, the output of combiner <NUM> results in the reconstructed signal, namely picture <NUM>'.

Although not specifically described above, it is readily clear that the encoder <NUM> may set some coding parameters including, for instance, prediction modes, motion parameters and the like, according to some optimization scheme such as, for instance, in a manner optimizing some rate and distortion related criterion, i.e. coding cost. For example, encoder <NUM> and decoder <NUM> and the corresponding modules <NUM>, <NUM>, respectively, may support different prediction modes such as intra-coding modes and inter-coding modes. The granularity at which encoder and decoder switch between these prediction mode types may correspond to a subdivision of picture <NUM> and <NUM>', respectively, into coding segments or coding blocks. In units of these coding segments, for instance, the picture may be subdivided into blocks being intra-coded and blocks being inter-coded. Intra-coded blocks are predicted on the basis of a spatial, already coded/decoded neighbourhood of the respective block as is outlined in more detail below. Several intra-coding modes may exist and be selected for a respective intra-coded segment including directional or angular intra-coding modes according to which the respective segment is filled by extrapolating the sample values of the neighbourhood along a certain direction which is specific for the respective directional intra-coding mode, into the respective intra-coded segment. The intra-coding modes may, for instance, also comprise one or more further modes such as a DC coding mode, according to which the prediction for the respective intra-coded block assigns a DC value to all samples within the respective intra-coded segment, and/or a planar intra-coding mode according to which the prediction of the respective block is approximated or determined to be a spatial distribution of sample values described by a two-dimensional linear function over the sample positions of the respective intra-coded block with driving tilt and offset of the plane defined by the two-dimensional linear function on the basis of the neighbouring samples. Compared thereto, inter-coded blocks may be predicted, for instance, temporally. For inter-coded blocks, motion vectors may be signalled within the data stream, the motion vectors indicating the spatial displacement of the portion of a previously coded picture of the video to which picture <NUM> belongs, at which the previously coded/decoded picture is sampled in order to obtain the prediction signal for the respective inter-coded block. This means, in addition to the residual signal coding comprised by data stream <NUM>, such as the entropy-coded transform coefficient levels representing the quantized spectral-domain prediction residual signal <NUM>", data stream <NUM> may have encoded thereinto coding mode parameters for assigning the coding modes to the various blocks, prediction parameters for some of the blocks, such as motion parameters for inter-coded segments, and optional further parameters such as parameters for controlling and signalling the subdivision of picture <NUM> and <NUM>', respectively, into the segments. The decoder <NUM> uses these parameters to subdivide the picture in the same manner as the encoder did, to assign the same prediction modes to the segments, and to perform the same prediction to result in the same prediction signal.

<FIG> illustrates the relationship between the reconstructed signal, i.e. the reconstructed picture <NUM>', on the one hand, and the combination of the prediction residual signal <NUM>"" as signalled in the data stream <NUM>, and the prediction signal <NUM>, on the other hand. As already denoted above, the combination may be an addition. The prediction signal <NUM> is illustrated in <FIG> as a subdivision of the picture area into intra-coded blocks which are illustratively indicated using hatching, and inter-coded blocks which are illustratively indicated not-hatched. The subdivision may be any subdivision, such as a regular subdivision of the picture area into rows and columns of square blocks or non-square blocks, or a multi-tree subdivision of picture <NUM> from a tree root block into a plurality of leaf blocks of varying size, such as a quadtree subdivision or the like, wherein a mixture thereof is illustrated in <FIG> in which the picture area is first subdivided into rows and columns of tree root blocks which are then further subdivided in accordance with a recursive multi-tree subdivisioning into one or more leaf blocks.

Again, data stream <NUM> may have an intra-coding mode coded thereinto for intra-coded blocks <NUM>, which assigns one of several supported intra-coding modes to the respective intra-coded block <NUM>. For inter-coded blocks <NUM>, the data stream <NUM> may have one or more motion parameters coded thereinto. Generally speaking, inter-coded blocks <NUM> are not restricted to being temporally coded. Alternatively, inter-coded blocks <NUM> may be any block predicted from previously coded portions beyond the current picture <NUM> itself, such as previously coded pictures of a video to which picture <NUM> belongs, or picture of another view or an hierarchically lower layer in the case of encoder and decoder being scalable encoders and decoders, respectively.

The prediction residual signal <NUM>"" in <FIG> is also illustrated as a subdivision of the picture area into blocks <NUM>. These blocks might be called transform blocks in order to distinguish same from the coding blocks <NUM> and <NUM>. In effect, <FIG> illustrates that encoder <NUM> and decoder <NUM> may use two different subdivisions of picture <NUM> and picture <NUM>', respectively, into blocks, namely one subdivisioning into coding blocks <NUM> and <NUM>, respectively, and another subdivision into transform blocks <NUM>. Both subdivisions might be the same, i.e. each coding block <NUM> and <NUM>, may concurrently form a transform block <NUM>, but <FIG> illustrates the case where, for instance, a subdivision into transform blocks <NUM> forms an extension of the subdivision into coding blocks <NUM>, <NUM> so that any border between two blocks of blocks <NUM> and <NUM> overlays a border between two blocks <NUM>, or alternatively speaking each block <NUM>, <NUM> either coincides with one of the transform blocks <NUM> or coincides with a cluster of transform blocks <NUM>. However, the subdivisions may also be determined or selected independent from each other so that transform blocks <NUM> could alternatively cross block borders between blocks <NUM>, <NUM>. As far as the subdivision into transform blocks <NUM> is concerned, similar statements are thus true as those brought forward with respect to the subdivision into blocks <NUM>, <NUM>, i.e. the blocks <NUM> may be the result of a regular subdivision of picture area into blocks (with or without arrangement into rows and columns), the result of a recursive multi-tree subdivisioning of the picture area, or a combination thereof or any other sort of blockation. Just as an aside, it is noted that blocks <NUM>, <NUM> and <NUM> are not restricted to being of quadratic, rectangular or any other shape.

<FIG> further illustrates that the combination of the prediction signal <NUM> and the prediction residual signal <NUM>"" directly results in the reconstructed signal <NUM>'. However, it should be noted that more than one prediction signal <NUM> may be combined with the prediction residual signal <NUM>"" to result into picture <NUM>' in accordance with alternative embodiments.

In <FIG>, the transform blocks <NUM> shall have the following significance. Transformer <NUM> and inverse transformer <NUM> perform their transformations in units of these transform blocks <NUM>. For instance, many codecs use some sort of DST or DCT for all transform blocks <NUM>. Some codecs allow for skipping the transformation so that, for some of the transform blocks <NUM>, the prediction residual signal is coded in the spatial domain directly. However, in accordance with embodiments described below, encoder <NUM> and decoder <NUM> are configured in such a manner that they support several transforms. For example, the transforms supported by encoder <NUM> and decoder <NUM> could comprise:.

Naturally, while transformer <NUM> would support all of the forward transform versions of these transforms, the decoder <NUM> or inverse transformer <NUM> would support the corresponding backward or inverse versions thereof:.

It should be noted that the set of supported transforms may comprise merely one transform such as one spectral-to-spatial or spatial-to-spectral transform.

As already outlined above, <FIG> have been presented as an example where the inventive concept described further below may be implemented in order to form specific examples for encoders and decoders according to the present application. Insofar, the encoder and decoder of <FIG> and <FIG>, respectively, may represent possible implementations of the encoders and decoders described herein below. <FIG> and <FIG> are, however, only examples. An encoder according to embodiments of the present application may, however, perform block-based encoding of a picture <NUM> using the concept outlined in more detail below and being different from the encoder of <FIG> such as, for instance, in that same does not support intra-prediction, or in that the sub-division into blocks <NUM> is performed in a manner different than exemplified in <FIG>. Likewise, decoders according to embodiments of the present application may perform block-based decoding of picture <NUM>' from data stream <NUM> using the coding concept further outlined below, but may differ, for instance, from the decoder <NUM> of <FIG> in that same does not support intra-prediction, or in that same sub-divides picture <NUM>' into blocks in a manner different than described with respect to <FIG> and/or in that same does not derive the prediction residual from the data stream <NUM> in transform domain, but in spatial domain, for instance.

There are several applications that make use of resolution adaptation for several purposes, e.g. bitrate adaptation for throughput variation or for Region of Interest (Rol) use-cases.

The current VVC draft specifies a process usually referred to as reference picture resampling, which allows having a varying picture size within a video sequence of Rol encoding processes as in the examples shown from <FIG>. For this purpose, the VVC specification draft includes a maximum picture size in the Sequence Parameter Set (SPS), an actual picture size in the Picture Parameter Set (PPS) and scaling window offsets (e.g. red margins in <FIG>) in the PPS that allow deriving the scaling ratios that need to be used between the current picture and the referenced pictures.

After having described a possible implementation of an encoder and decoder framework into which embodiments of the present application could be built into, the description preliminarily refers again to the current VVC development and motivates the specifics of the embodiments outlined later on.

In VVC, the scaling ratios are derived as follows using the width of the pictures considering the scaling windows defined in the PPSs for the current picture (PicOutputWidthL) and the reference picture (fRefWidth): <MAT> <MAT>.

PicOutputWidth and PicOutputHeight are also sometimes referred to as CurrPicScalWinWidth and CurrPicScalWinHeight hereinbelow.

Scale ratios < <NUM> - i.e., RefPicScale values < (<NUM><<<NUM>) - mean that the current picture is bigger than the reference picture and ratios ><NUM> - i.e., RefPicScale values > (<NUM><<<NUM>) mean that the current picture is smaller than the reference picture.

The current VVC draft specifies <NUM> interpolation filters for motion compensation up to <NUM>/<NUM> of a sample using fractional sample interpolation. The first one was designed for the regular motion compensation case, when there is no reference picture resampling, RPR, and it is not an affine mode. A second filter was designed for the case that affine mode is used. The two remaining filters are used for downsampling with factors of <NUM> and <NUM>.

Scaling ratios are allowed to be from <NUM>/<NUM> (8x upsampling) to <NUM> (2x downsampling). Depending on whether an affine mode is used and the scaling ratio, one of the four filters is used. The conditions are as follows:.

For a resolution change where the current picture is bigger than the reference picture or for very small values of ratios when the current picture is smaller than the reference picture (Scaling ratio <= <NUM>. 25x downsampling factor), the regular interpolation filter is used.

The regular interpolation filter used for the cases where no affine mode is used, or there is no RPR (scaling ratio = <NUM>), or the scaling ratio is less than or equal to <NUM>, can apply a specific smoothing filter.

The <NUM>/<NUM> sample regular interpolation filter is defined in VVC as an <NUM>-tap filter. However, the VVC specification defines a special <NUM>-tap smoothing filter that is used in the following case:.

This <NUM>-tap smoothing filter is used when adaptive motion vector resolution is used and the motion vector difference is in half-sample resolution. The filter is replicated in <FIG>.

Given the <NUM>/<NUM> fractional sample accuracy in VVC, the fractional sample position p=<NUM> corresponds to the half-sample position (<NUM>/<NUM>=<NUM>/<NUM>). The variable hpellfldx equal to <NUM> indicates whether the <NUM>-tap smoothing filter (highlighted) is used for the half-sample positions. hpellfldx is set to <NUM> when AmvrShift is equal to <NUM>, which in case no IBC mode is used, indicates half-sample MVD resolution. This is signalled in the bitstream using the syntax amvr_flag equal to <NUM> and amvr_precision_idx equal to <NUM>. See also <FIG>.

When RPR is not used, the smoothing filter is used to generate each sample in the reference block in the cases described above as each sample in the block refers to the same fractional (half-sample) interpolation position. However, when RPR is used, each sample may refer to a different fractional interpolation position.

Note in the following text that a n-sample difference in current bock (x"L - x'L or y"L - y'L) is affected by the scaling ratio.

Therein scaling_win_left_offset could be calculated as SubWidthC x pps_scaling_win_left_offset and scaling_win_top_offset could be calculated as SubHeightC x pps_scaling_win_top_offset.

For instance, let's assume that the current picture is 2x bigger than the reference picture. This, is depicted in <FIG> where samples of the referenced picture and the current picture are shown spatially overlaid with crosses showing samples of the reference(d) picture and block circles showing samples of the current picture. A current 4x4 block of the current picture is shown. The motion vector for that block is shown by an arrow. It is defined at half-pel resolution relative to the pel grid of the samples of the current picture. The resulting block samples' positions within the reference picture, at which the reference picture is to be interpolated to yield the predictor for the block in the current picture, are shown using white circles. The motion vector is exemplarily selected such that same makes that the sample x'L = <NUM> (the upper left sample of the 4x4 block) points to a half-sample position, and the sample x'L = <NUM> (the sample to the right thereof) would point to an integer- or full-sample position, both positions being indicated as being half-pel or full-pel with respect to the reference picture. This situation of leading to pixels associated with different-pel positions - or, in other words, different phases - does not occur with reference and current pictures having the same resolution and the motion vector being defined at half-pel resolution as shown in <FIG>. Similarly, if the scaling would be 4x instead of 2x, and the sample x'L = <NUM> points to a half-sample position, the sample x'L = <NUM> would point to a quarter-sample position and the sample x'L= <NUM> would point to an integer-sample position.

This means that within a single block, some samples would use a smoothing filter and some not, which would lead to an unpleasant visual effect and visible artefacts.

In one embodiment, the derivation of the variable hpellfldx in the motion compenstation step is modified to include the scaling ratio between current and reference picture as follows:
hpellfldx = <NUM> if AmvrShift is equal to <NUM> and scaling ratio == <NUM>, i.e. RefPicScale is equal to <NUM>.

<FIG> illustrates a video decoder according to the present invention. The video decoder <NUM> decodes a video <NUM> from a data stream <NUM> using motion compensation prediction. The motion prediction could be performed in motion prediction portion <NUM>, and is based on first motion vectors <NUM> and second motion vectors <NUM>, which are transmitted in the data stream <NUM>.

The first motion vectors <NUM> are transmitted in the data stream <NUM> at a first resolution being half-sample resolution, and the second motion vectors <NUM> are transmitted in the data stream <NUM> at a second resolution being different from the first resolution.

The motion compensation is performed between first pictures <NUM> of equal picture resolution and second pictures <NUM> of different picture resolution, i.e. RPR is supported or, in other words, motion vectors may point from a current picture to a reference picture of the same resolution as the current picture, with then both forming first pictures, and motion vectors may point from a current picture to a reference picture of a different resolution than the current picture, with then both forming second pictures. The pictures sizes and/or resolution, thus, may vary, and is signalled in the data stream. The motion compensation is performed using interpolation filters <NUM> and <NUM> to obtain sub-sample values within the reference picture, i.e. within the reference sample array.

The video decoder <NUM> selects, for example, in a selection portion <NUM>, the interpolation filter for a predetermined first motion vector, from a first interpolation filter version <NUM> and a second interpolation filter version <NUM>. The second interpolation filter version <NUM> has a higher edge preserving property than the first interpolation filter version <NUM>. As will be shown in more details below, this selection may be specific for samples of a certain phase or, differently speaking, at certain sub-pel positions such as half-pel positions.

The selection of the interpolation filter is depending on whether a current picture, to which the predetermined first motion vector belongs, is equal in picture resolution to the reference sample array, to which the predetermined first motion vector relates. The selection and the check on equality may be done separately for the dimensions, i.e. in horizontal and/or vertical dimension. Additionally or alternatively, the selection could also be depending on a constraint information <NUM> being transmitted in the data stream as will be outlined in more details below.

The dependence of the constraint information <NUM> is not shown in <FIG>.

Further, encoder and decoder can obtain full-sample values within the reference sample array for the predetermined first motion vector without using the interpolation filter. In <FIG>, for instance, the motion vectors shown therein are "first motion vectors" if they are transmitted in the data stream at half-pel resolution such as using AmvrShift=<NUM>. The positions to which the samples of the predicted block are shifted according to the respective motion vector, are shown in <FIG> and <FIG> by circles. Those circles falling onto crosses are "full-sample values". They may directly by determined from the collocated samples of the reference picture (the crosses) without any interpolation. That is, the samples values of the samples of the reference picture, onto which shifted positions of samples of the inter-predicted block fall directly, are directly used as predictor for the samples of the inter-predicted block whose shifted positions fall onto them. Naturally, the same may apply with respect to shifted sample positions of inter-predicted block with second motion vectors, i.e. motion vectors transmitted in the data stream at a resolution other than half-pel.

Also, the decoder can obtain non half-sample sub-sample values by using further interpolation filters, for example using a filter having a higher edge preserving property than the first interpolation filter version. Show again to <FIG>. "Non half-sample sub-sample values" are those circles neither falling onto any reference picture sample, nor laying halfway between two horizontal, vertical or diagonal immediate neighbouring samples of the reference pictures, i.e. neither falling onto any cross, nor laying halfway between two horizontal, vertical or diagonal immediate neighbouring crosses. For same, an interpolation filter with higher edge preserving property is used. See, <FIG>, where this is illustrated for a quarter-pel position: The shifted position of the uppermost and second-from left sample of the block is a quarter pel position. It is sample position <NUM>/<NUM>th of a full sample pitch away to the right from a reference picture sample (the upper left cross in <FIG>). The interpolation filter for to be used for determined the value of that shifted sample, i.e. the uppermost and second-from left white circle, is thus defined in the corresponding entry of the table of <FIG>, i.e. the entry for the <NUM>/<NUM>th positions. It's a FIR filter. The table comprises the filter coefficients to be applied to the reference picture's samples between which the sample position to be interpolated is positioned <FIG> highlights the entry where the interpolation filter is defined and illustrates the weighted some of reference picture samples according to this filter to yield the interpolated quarter-pel sample. Note that horizontal interpolation may be applied first so as to obtain sample values at sub-pel positions between the samples of the reference picture, with then performing vertical interpolation using these interpolated intermediate samples in order to obtain the actually wanted sub-pel samples, if the wanted sub-pel samples are offset, at sub-sample accuracy, from the reference picture samples vertically and horizontally, or vice versa, i.e. firstly vertically and then horizontally. Again, the selection between the two half-pel sample position interpolation filter version might be done separately for horizontal and vertical direction or globally for both direction depending on the equality of picture resolution in both dimensions.

As stated above, the selection can be performed separately for horizontal and vertical interpolation. The selection is illustrated in <FIG> at the two entries concerning half-pel position <NUM>/<NUM>: Which filter to take depends on hpellfldx. The latter variable is set depending on whether, for instance, reference and current pictures have the same resolution. The latter check on equality may be performed separately for a and y as illustrated below by the usage of the terms hpelHorlfldx and hpelVerlfldx. In particular, the second interpolation filter, that is the filter with the higher edge preserving property, might be selected for horizontal interpolation if the current picture and the reference sample array are not equal in horizontal picture resolution. That's the filter defined in the row of the table where hpellfldx=<NUM>. Likewise, for example, the second interpolation filter, that is the filter with the higher edge preserving property, can be selected for vertical interpolation if the current picture and the reference sample array are not equal in vertical picture resolution. And, if the current picture and the reference sample array are not equal in horizontal and vertical picture resolution, the second interpolation filter, that is the filter with the higher edge preserving property, can be selected for horizontal and vertical interpolation. For any direction, for which the second interpolation fitler is not used, the frist interpolation filter is used, i.e. the one in the row of the table where hpellfldx=<NUM>.

The selection of which filter to take among the two half-pel position interpolation filters is, naturally, may interpreted as being performed for all motion vectors, not only the ones being half-pel motion vectors. See in that broad manner, the selection among the two also depends on whether the motion vector is of half-sample resolution or not. If it is, then the selection is done as described so far depending on resolution equality between reference and current pictures, if it is not, then the selection takes the second interpolation filter with the higher edge-preserving property inevitably. as became clear from the above description, the decoder can use an alphabet of one or more syntax elements in the data stream so as to determine the resolution at which a certain motion vector is transmitted in the data stream. For example, an adaptive motion vector resolution might be indicated by an amvr_flag, and if same is set, thereby conforming a deviation from some default motion vector resolution, and an adaptive motion vector resolution precision can be indicated by an index amvr_precision_idx. This syntax is used an decoded by decoder to derive the resolution at which a motion vector of a specific inter-predicted block is transmitted in the data stream, and this syntax is coded correspondingly in order indicate a motion vector's resolution by the encoder.

The decoder and encoder may exclude the half-pel resolution from the set of signalable settings for the motion vector resolution. They can map the alphabet of the one or more syntax elements onto a first set of vector resolutions, which do not comprise half-sample resolution, if one of the following conditions is met (while the mapping is done onto a second set set of vector resolutions including the half-sample resolution, otherwise):.

The decoder would map the alphabet onto a second set of vector resolutions, which comprises half-sample resolution, if none of the above conditions is met.

It is also noted that the data stream can comprise information whether temporal consecutive pictures have same or different horizontal and/or vertical picture resolution dimensions.

Further, as stated above, the current picture can in particular be equal in picture resolution to the reference sample array in horizontal and vertical dimension.

And the reference sample array can be a region, subpicture or picture.

The decoder can also derive the constraint information from the data stream in one of per sequences of pictures, picture-wise, or slice-wise.

<FIG> illustrates a video encoder according to the present invention. Therein the same principles apply as for the decoder. In summary, the video encoder <NUM> encodes a video <NUM> into a data stream <NUM> using motion compensation prediction. The motion prediction could be performed in motion prediction portion <NUM>. The encoder <NUM> indicates in by transmitting in the data stream <NUM> first motion vectors <NUM> and second motion vectors <NUM>.

The motion compensation is performed between first pictures <NUM> of equal picture resolution and second pictures <NUM> of different picture resolution using interpolation filters <NUM> and <NUM> to obtain sub-sample values within the reference picture, i.e. within the reference array.

The video encoder <NUM> selects, for example in a selection portion <NUM>, the interpolation filter for a predetermined first motion vector, from a first interpolation filter version <NUM> and a second interpolation filter version <NUM>. The second interpolation filter version <NUM> has a higher edge preserving property than the first interpolation filter version <NUM>.

The selection of the interpolation filter is depending on whether a current picture, to which the predetermined first motion vector belongs, is equal in picture resolution to the reference sample array, to which the predetermined first motion vector relates, in horizontal and/or vertical dimension. Additionally or alternatively, the selection could also be depending on a constraint information <NUM> to be transmitted in the data stream.

As stated before, the same principles that can be embodied by the decoder can also be embodied by the encoder.

Thus, also the encoder can obtain full-sample values within the reference sample array for the predetermined first motion vector without using the interpolation filter.

Also, the encoder can obtain non half-sample sub-sample values by using further interpolation filters, for example using a filter having a higher edge preserving property than the first interpolation filter version.

As stated above, the selection can be performed separately for horizontal and vertical interpolation.

In particular, the second interpolation filter, that is the filter with the higher edge preserving property, can be selected for horizontal interpolation if the current picture and the reference sample array are not equal in horizontal picture resolution.

Likewise, for example, the second interpolation filter, that is the filter with the higher edge preserving property, can be selected for vertical interpolation if the current picture and the reference sample array are not equal in vertical picture resolution.

And, if the current picture and the reference sample array are not equal in horizontal and vertical picture resolution, the second interpolation filter, that is the filter with the higher edge preserving property, can be selected for horizontal and vertical interpolation.

The selection can further be performed depending on whether the predetermined first motion vector is of half-sample resolution.

Further, in order to select the resolution of the motion vectors, the encoder can refrain from using half-sample resolution for one or more vectors if the current picture is equal in picture resolution to the reference sample array in horizontal and/or vertical dimension.

For the selection, the encoder can map an alphabet of one or more syntax elements in the data stream which indicate a resolution of the predetermined first motion vector. For example, an adaptive motion vector resolution can be indicated by an amvr_flag and an adaptive motion vector resolution precision can be indicated by amvr_precision_idx,.

The encoder can map the alphabet onto a first set of vector resolutions, which do not comprise half-sample resolution, if one of the following conditions is met:.

The encoder can map the alphabet onto a second set of vector resolutions, which comprises half-sample resolution, if none of the above conditions is met.

The encoder can also derive the constraint information from the data stream in one of per sequences of pictures, picture-wise, or slice-wise.

Finally, the above-described principles can also be embodied with a computer program product including a program with software code portions for employing the above principles, when the program is run on a processing device. Further this computer program product can also be embodied as a computer-readable medium on which the software code portions are stored.

The principles laid out above and below can also be embodied as data streams produced by encoding or by the encoder as described in this document.

Let's return to the description of embodiments which amend the current VVC draft. For example, the dependence of the constraint information is not shown in <FIG>.

In one embodiment, the derivation of the variable hpellfldx in the motion compenstation step is modified to incorporate the enable flag for reference picture signalling on the sequence level in the SPS and the smoothing filter coeficients are only used when reference picture resampling is forbidden as follows:
hpellfldx = <NUM> if AmvrShift is equal to <NUM> and sps_ref_picture_resample_enable_flag == <NUM>, i.e. reference picture resampling is disabled.

In another embodiment, a controlling syntax flag is added to the picture or slice header to indicate whether the smoothing filter is disabled for the current picture. Then the hpellfldx is derived as follows:
hpellfldx = <NUM> if AmvrShift is equal to <NUM> and controlling flag is equal to <NUM> (e.g., ph_disable_hpel_smoothing_filter or sh_disable_hpel_smoothing_filter).

In another embodiment, the derivation of the variable AmvrShift is modified to include information about reference picture resampling such as and avoid value of equal to <NUM> when.

In another embodiment, it is a bitstream constraint that AmvrShift is not equal to <NUM> when RPR is used for a reference picture, i.e. if the current picture and reference picture have a non-equal size or the scaling ratio derived from the scaling window is not equal to <NUM>, i.e. RefPicScale is not equal to <NUM>.

It follows, that the horizontal and vertical half sample interpolation filter indices hpelHorlfldx and hpelVerlfldx are derived as follows:.

The inventive data stream can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

The program code may for example be stored on a machine-readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine-readable carrier.

Claim 1:
A video decoder (<NUM>) configured to:
decode a video (<NUM>) from a data stream (<NUM>) using motion compensation prediction to predict current pictures from reference pictures
based on motion vector differences (<NUM>) transmitted in the data stream (<NUM>) at half-sample resolution,
with first reference pictures (<NUM>) of equal picture resolution as respective current pictures for which they are used in prediction and second reference pictures (<NUM>) of different picture resolution than respective current pictures for which they are used in prediction, and
using interpolation filters (<NUM>, <NUM>) to obtain sub-sample values within reference sample arrays formed from respective reference pictures,
select an interpolation filter to obtain half-sample values within a reference sample array, the reference sample array formed from a reference picture by using a motion vector, the interpolation filter selected from a first interpolation filter version (<NUM>) and a second interpolation filter version (<NUM>), wherein the first interpolation filter version (<NUM>) is a <NUM> tap filter with coefficients <NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM> and the second interpolation filter version (<NUM>) is an <NUM> tap filter with coefficients -<NUM>,<NUM>,-<NUM>,<NUM>,<NUM>,-<NUM>,<NUM>,-<NUM>, by
selecting the first interpolation filter version for horizontal interpolation when the reference picture is equal in picture resolution to the current picture in horizontal dimension and selecting the second interpolation filter version for horizontal interpolation when the reference picture is unequal in picture resolution to the current picture in horizontal dimension, and/or
selecting the first interpolation filter version for vertical interpolation when the reference picture is equal in picture resolution to the current picture in vertical dimension and selecting the second interpolation filter version for vertical interpolation when the reference picture is unequal in picture resolution to the current picture in vertical dimension.