Patent Description:
Compression encoding means a series of signal processing techniques for transmitting digitized information through a communication line or techniques for storing information in a form suitable for a storage medium. The medium including a picture, an image, audio, etc. may be a target for compression encoding, and particularly, a technique for performing compression encoding on a picture is referred to as video image compression.

Next-generation video contents are supposed to have the characteristics of high spatial resolution, a high frame rate and high dimensionality of scene representation. In order to process such contents, a drastic increase in the memory storage, memory access rate and processing power will result.

Accordingly, it is required to design a coding tool for processing next-generation video contents efficiently.

disclose bi-directional optical flow (BIO), as a tool to refine motion vectors at the decoding end, thereby improving the efficiency of video coding. Both documents, however, limit their teaching to scenarios in which one of the reference frames precedes the current frame and the other follows it.

In the existing compression technology of a still image or moving image, motion prediction is performed in a prediction block unit when inter-prediction is performed. There is a problem in that prediction precision is deteriorated although prediction blocks of various sizes or shapes are supported in order to search for an optimal prediction block for a current block.

In order to solve this problem, an object of the present invention proposes a method of processing an image by performing motion compensation in a pixel unit upon performing the inter-prediction.

Furthermore, an object of the present invention proposes a method of improving a reference block in a pixel unit by applying an optical flow derivation method upon performing the motion compensation of inter-prediction.

Technical objects to be achieved by the present invention are not limited to the aforementioned technical objects, and other technical objects not described above may be evidently understood by a person having ordinary skill in the art to which the present invention pertains from the following description.

The invention is addressed to a method for processing an image as defined by independent claim <NUM> and a corresponding apparatus as defined by independent claim <NUM>.

Further optional aspects are defined in the dependent claims.

The invention is disclosed in the section relative to "Embodiment <NUM>" with <FIG> and <FIG> and with the other passages of the description referred to in the section disclosing "Embodiment <NUM>". Although they are referred to as "embodiments" the other "embodiments" are to be understood as mere examples, as they do not fall under the scope of the independent claims. More in general, any use of the expressions "embodiment(s)" and "invention" are to be understood to refer to examples, except when referring to elements of "Embodiment <NUM>".

The accompanying drawings, which are included herein as a part of the description for help understanding the present invention, provide embodiments of the present invention, and describe the technical features of the present invention with the description below.

Hereinafter, a preferred embodiment of the present invention will be described by reference to the accompanying drawings. The description that will be described below with the accompanying drawings is to describe exemplary embodiments of the present invention, and is not intended to describe the only embodiment in which the present invention may be implemented. The description below includes particular details in order to provide perfect understanding of the present invention. However, it is understood that the present invention may be embodied without the particular details to those skilled in the art.

In some cases, in order to prevent the technical concept of the present invention from being unclear, structures or devices which are publicly known may be omitted, or may be depicted as a block diagram centering on the core functions of the structures or the devices.

Further, although general terms widely used currently are selected as the terms in the present invention as much as possible, a term that is arbitrarily selected by the applicant is used in a specific case. Since the meaning of the term will be clearly described in the corresponding part of the description in such a case, it is understood that the present invention will not be simply interpreted by the terms only used in the description of the present invention, but the meaning of the terms should be figured out.

Specific terminologies used in the description below may be provided to help the understanding of the present invention. For example, a signal, data, a sample, a picture, a frame, a block, etc may be properly replaced and interpreted in each coding process.

Hereinafter, in this specification, a "processing unit" means a unit in which encoding/decoding processing process, such as prediction, transform and/or quantization, is performed. Hereinafter, for convenience of description, a processing unit may also be called a "processing block" or "block.

A processing unit may be construed as having a meaning including a unit for a luma component and a unit for a chroma component. For example, a processing unit may correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU) or a transform unit (TU).

Furthermore, a processing unit may be construed as being a unit for a luma component or a unit for a chroma component. For example, the processing unit may correspond to a coding tree block (CTB), coding block (CB), prediction block (PB) or transform block (TB) for a luma component. Alternatively, a processing unit may correspond to a coding tree block (CTB), coding block (CB), prediction block (PB) or transform block (TB) for a chroma component. Furthermore, the present invention is not limited thereto, and a processing unit may be construed as a meaning including a unit for a luma component and a unit for a chroma component.

Furthermore, a processing unit is not essentially limited to a square block and may be constructed in a polygon form having three or more vertices.

<FIG> is illustrates a schematic block diagram of an encoder in which the encoding of a still image or video signal is performed, as an embodiment to which the present invention is applied.

Referring to <FIG>, the encoder <NUM> may include a video split unit <NUM>, a subtractor <NUM>, a transform unit <NUM>, a quantization unit <NUM>, a dequantization unit <NUM>, an inverse transform unit <NUM>, a filtering unit <NUM>, a decoded picture buffer (DPB) <NUM>, a prediction unit <NUM> and an entropy encoding unit <NUM>. Furthermore, the prediction unit <NUM> may include an inter-prediction unit <NUM> and an intra-prediction unit <NUM>.

The video split unit <NUM> splits an input video signal (or picture or frame), input to the encoder <NUM>, into one or more processing units.

The subtractor <NUM> generates a residual signal (or residual block) by subtracting a prediction signal (or prediction block), output by the prediction unit <NUM> (i.e., by the inter-prediction unit <NUM> or the intra-prediction unit <NUM>), from the input video signal. The generated residual signal (or residual block) is transmitted to the transform unit <NUM>.

The transform unit <NUM> generates transform coefficients by applying a transform scheme (e.g., discrete cosine transform (DCT), discrete sine transform (DST), graph-based transform (GBT) or Karhunen-Loeve transform (KLT)) to the residual signal (or residual block). In this case, the transform unit <NUM> may generate transform coefficients by performing transform using a prediction mode applied to the residual block and a transform scheme determined based on the size of the residual block.

The quantization unit <NUM> quantizes the transform coefficient and transmits it to the entropy encoding unit <NUM>, and the entropy encoding unit <NUM> performs an entropy coding operation of the quantized signal and outputs it as a bit stream.

Meanwhile, the quantized signal outputted by the quantization unit <NUM> may be used to generate a prediction signal. For example, a residual signal may be reconstructed by applying dequatization and inverse transformation to the quantized signal through the dequantization unit <NUM> and the inverse transform unit <NUM>. A reconstructed signal may be generated by adding the reconstructed residual signal to the prediction signal output by the inter-prediction unit <NUM> or the intra-prediction unit <NUM>.

Meanwhile, during such a compression process, neighbor blocks are quantized by different quantization parameters. Accordingly, an artifact in which a block boundary is shown may occur. Such a phenomenon is referred to a blocking artifact, which is one of important factors for evaluating image quality. In order to decrease such an artifact, a filtering process may be performed. Through such a filtering process, the blocking artifact is removed and the error of a current picture is decreased at the same time, thereby improving image quality.

The filtering unit <NUM> applies filtering to the reconstructed signal, and outputs it through a playback device or transmits it to the decoded picture buffer <NUM>. The filtered signal transmitted to the decoded picture buffer <NUM> may be used as a reference picture in the inter-prediction unit <NUM>. As described above, an encoding rate as well as image quality can be improved using the filtered picture as a reference picture in an inter-picture prediction mode.

The decoded picture buffer <NUM> may store the filtered picture in order to use it as a reference picture in the inter-prediction unit <NUM>.

The inter-prediction unit <NUM> performs temporal prediction and/or spatial prediction with reference to the reconstructed picture in order to remove temporal redundancy and/or spatial redundancy. In this case, a blocking artifact or ringing artifact may occur because a reference picture used to perform prediction is a transformed signal that experiences quantization or dequantization in a block unit when it is encoded/decoded previously.

Accordingly, in order to solve performance degradation attributable to the discontinuity of such a signal or quantization, signals between pixels may be interpolated in a sub-pixel unit by applying a low pass filter to the inter-prediction unit <NUM>. In this case, the sub-pixel means a virtual pixel generated by applying an interpolation filter, and an integer pixel means an actual pixel that is present in a reconstructed picture. A linear interpolation, a bi-linear interpolation, a wiener filter, and the like may be applied as an interpolation method.

The interpolation filter may be applied to the reconstructed picture, and may improve the accuracy of prediction. For example, the inter-prediction unit <NUM> may perform prediction by generating an interpolation pixel by applying the interpolation filter to the integer pixel and by using the interpolated block including interpolated pixels as a prediction block.

The intra-prediction unit <NUM> predicts a current block with reference to samples neighboring the block that is now to be encoded. The intra-prediction unit <NUM> may perform the following procedure in order to perform intra-prediction. First, the intra-prediction unit <NUM> may prepare a reference sample necessary to generate a prediction signal. Furthermore, the intra-prediction unit <NUM> may generate a prediction signal using the prepared reference sample. Next, the intra-prediction unit <NUM> may encode a prediction mode. In this case, the reference sample may be prepared through reference sample padding and/or reference sample filtering. A quantization error may be present because the reference sample experiences the prediction and the reconstruction process. Accordingly, in order to reduce such an error, a reference sample filtering process may be performed on each prediction mode used for the intra-prediction.

The prediction signal (or prediction block) generated through the inter-prediction unit <NUM> or the intra-prediction unit <NUM> may be used to generate a reconstructed signal (or reconstructed block) or may be used to generate a residual signal (or residual block).

<FIG> illustrates a schematic block diagram of a decoder in which decoding of a still image or video signal is performed, as an embodiment to which the present invention is applied.

Referring to <FIG>, the decoder <NUM> may include an entropy decoding unit <NUM>, a dequantization unit <NUM>, an inverse transform unit <NUM>, an adder <NUM>, a filtering unit <NUM>, a decoded picture buffer (DPB) <NUM> and a prediction unit <NUM>. Furthermore, the prediction unit <NUM> may include an inter-prediction unit <NUM> and an intra-prediction unit <NUM>.

Furthermore, a reconstructed video signal output through the decoder <NUM> may be played back through a playback device.

The decoder <NUM> receives a signal (i.e., bit stream) output by the encoder <NUM> shown in <FIG>. The entropy decoding unit <NUM> performs an entropy decoding operation on the received signal.

The dequantization unit <NUM> obtains transform coefficients from the entropy-decoded signal using quantization step size information.

The inverse transform unit <NUM> obtains a residual signal (or residual block) by inverse transforming the transform coefficients by applying an inverse transform scheme.

The adder <NUM> adds the obtained residual signal (or residual block) to the prediction signal (or prediction block) output by the prediction unit <NUM> (i.e., the inter-prediction unit <NUM> or the intra-prediction unit <NUM>), thereby generating a reconstructed signal (or reconstructed block).

The filtering unit <NUM> applies filtering to the reconstructed signal (or reconstructed block) and outputs the filtered signal to a playback device or transmits the filtered signal to the decoded picture buffer <NUM>. The filtered signal transmitted to the decoded picture buffer <NUM> may be used as a reference picture in the inter-prediction unit <NUM>.

In this specification, the embodiments described in the filtering unit <NUM>, inter-prediction unit <NUM> and intra-prediction unit <NUM> of the encoder <NUM> may be identically applied to the filtering unit <NUM>, inter-prediction unit <NUM> and intra-prediction unit <NUM> of the decoder, respectively.

In particular, the inter-prediction unit <NUM> according to the present invention may further include a configuration for performing inter-prediction of a pixel unit. This is described in detail later.

In general, a block-based image compression method is used in the compression technique (e.g., HEVC) of a still image or a video. The block-based image compression method is a method of processing an image by splitting it into specific block units, and may decrease memory use and a computational load.

<FIG> is a diagram for describing a split structure of a coding unit which may be applied to the present invention.

An encoder splits a single image (or picture) into coding tree units (CTUs) of a quadrangle form, and sequentially encodes the CTUs one by one according to raster scan order.

In HEVC, a size of CTU may be determined as one of <NUM>×<NUM>, <NUM>×<NUM>, and <NUM>×<NUM>. The encoder may select and use the size of a CTU based on resolution of an input video signal or the characteristics of input video signal. The CTU includes a coding tree block (CTB) for a luma component and the CTB for two chroma components that correspond to it.

One CTU may be split in a quad-tree structure. That is, one CTU may be split into four units each having a square form and having a half horizontal size and a half vertical size, thereby being capable of generating coding units (CUs). Such splitting of the quad-tree structure may be recursively performed. That is, the CUs are hierarchically split from one CTU in the quad-tree structure.

A CU means a basic unit for the processing process of an input video signal, for example, coding in which intra/inter prediction is performed. A CU includes a coding block (CB) for a luma component and a CB for two chroma components corresponding to the luma component. In HEVC, a CU size may be determined as one of <NUM>×<NUM>, <NUM>×<NUM>, <NUM>×<NUM>, and <NUM>×<NUM>.

Referring to <FIG>, the root node of a quad-tree is related to a CTU. The quad-tree is split until a leaf node is reached. The leaf node corresponds to a CU.

This is described in more detail. The CTU corresponds to the root node and has the smallest depth (i.e., depth=<NUM>) value. A CTU may not be split depending on the characteristics of an input video signal. In this case, the CTU corresponds to a CU.

A CTU may be split in a quad-tree form. As a result, lower nodes, that is, a depth <NUM> (depth=<NUM>), are generated. Furthermore, a node (i.e., leaf node) that belongs to the lower nodes having the depth of <NUM> and that is no longer split corresponds to a CU. For example, in <FIG>, a CU(a), a CU(b) and a CU(j) corresponding to nodes a, b and j have been once split from the CTU, and have a depth of <NUM>.

At least one of the nodes having the depth of <NUM> may be split in a quad-tree form. As a result, lower nodes having a depth <NUM> (i.e., depth=<NUM>) are generated. Furthermore, a node (i.e., leaf node) that belongs to the lower nodes having the depth of <NUM> and that is no longer split corresponds to a CU. For example, in <FIG>, a CU(c), a CU(h) and a CU(i) corresponding to nodes c, h and i have been twice split from the CTU, and have a depth of <NUM>.

Furthermore, at least one of the nodes having the depth of <NUM> may be split in a quad-tree form again. As a result, lower nodes having a depth <NUM> (i.e., depth=<NUM>) are generated. Furthermore, a node (i.e., leaf node) that belongs to the lower nodes having the depth of <NUM> and that is no longer split corresponds to a CU. For example, in <FIG>, a CU(d), a CU(e), a CU(f) and a CU(g) corresponding to nodes d, e, f and g have been three times split from the CTU, and have a depth of <NUM>.

In the encoder, a maximum size or minimum size of a CU may be determined based on the characteristics of a video image (e.g., resolution) or by considering the encoding rate. Furthermore, information about the maximum or minimum size or information capable of deriving the information may be included in a bit stream. A CU having a maximum size is referred to as the largest coding unit (LCU), and a CU having a minimum size is referred to as the smallest coding unit (SCU).

In addition, a CU having a tree structure may be hierarchically split with predetermined maximum depth information (or maximum level information). Furthermore, each split CU may have depth information. Since the depth information represents a split count and/or degree of a CU, it may include information about the size of a CU.

Since the LCU is split in a Quad-tree shape, the size of SCU may be obtained by using a size of LCU and the maximum depth information. Or, inversely, the size of LCU may be obtained by using a size of SCU and the maximum depth information of the tree.

For a single CU, the information (e.g., a split CU flag (split_cu_flag)) that represents whether the corresponding CU is split may be forwarded to the decoder. This split information is included in all CUs except the SCU. For example, when the value of the flag that represents whether to split is '<NUM>', the corresponding CU is further split into four CUs, and when the value of the flag that represents whether to split is '<NUM>', the corresponding CU is not split any more, and the processing process for the corresponding CU may be performed.

As described above, a CU is a basic unit of the coding in which the intra-prediction or the inter-prediction is performed. The HEVC splits the CU in a prediction unit (PU) for coding an input video signal more effectively.

A PU is a basic unit for generating a prediction block, and even in a single CU, the prediction block may be generated in different way by a unit of PU. However, the intra-prediction and the inter-prediction are not used together for the PUs that belong to a single CU, and the PUs that belong to a single CU are coded by the same prediction method (i.e., the intra-prediction or the inter-prediction).

A PU is not split in the Quad-tree structure, but is split once in a single CU in a predetermined shape. This will be described by reference to the drawing below.

<FIG> is a diagram for describing a prediction unit that may be applied to the present invention.

A PU is differently split depending on whether the intra-prediction mode is used or the inter-prediction mode is used as the coding mode of the CU to which the PU belongs.

<NUM>(a) illustrates a PU if the intra-prediction mode is used, and FIG. <NUM>(b) illustrates a PU if the inter-prediction mode is used.

Referring to FIG. <NUM>(a), assuming that the size of a single CU is 2N×2N (N = <NUM>, <NUM>, <NUM> and <NUM>), the single CU may be split into two types (i.e., 2N×2N or N×N).

In this case, if a single CU is split into the PU of 2N×2N shape, it means that only one PU is present in a single CU.

Meanwhile, if a single CU is split into the PU of N×N shape, a single CU is split into four PUs, and different prediction blocks are generated for each PU unit. However, such PU splitting may be performed only if the size of CB for the luma component of CU is the minimum size (i.e., the case that a CU is an SCU).

Referring to FIG. <NUM>(b), assuming that the size of a single CU is 2N×2N (N = <NUM>, <NUM>, <NUM> and <NUM>), a single CU may be split into eight PU types (i.e., 2N×2N, N×N, 2N×N, N×2N, nL×2N, nRx2N, 2N×nU and 2N×nD).

As in the intra-prediction, the PU split of N×N shape may be performed only if the size of CB for the luma component of CU is the minimum size (i.e., the case that a CU is an SCU).

The inter-prediction supports the PU split in the shape of 2N×N that is split in a horizontal direction and in the shape of N×2N that is split in a vertical direction.

In addition, the inter-prediction supports the PU split in the shape of nL×2N, nR×2N, 2N×nU and 2N×nD, which is an asymmetric motion split (AMP). In this case, 'n' means <NUM>/<NUM> value of 2N. However, the AMP may not be used if the CU to which the PU is belonged is the CU of minimum size.

In order to encode the input video signal in a single CTU efficiently, the optimal split structure of the coding unit (CU), the prediction unit (PU) and the transform unit (TU) may be determined based on a minimum rate-distortion value through the processing process as follows. For example, as for the optimal CU split process in a <NUM>×<NUM> CTU, the rate-distortion cost may be calculated through the split process from a CU of <NUM>×<NUM> size to a CU of <NUM>×<NUM> size. The detailed process is as follows.

In the intra-prediction mode, a prediction mode is selected as a PU unit, and prediction and reconstruction are performed on the selected prediction mode in an actual TU unit.

A TU means a basic unit in which actual prediction and reconstruction are performed. A TU includes a transform block (TB) for a luma component and a TB for two chroma components corresponding to the luma component.

In the example of <FIG>, as in an example in which one CTU is split in the quad-tree structure to generate a CU, a TU is hierarchically split from one CU to be coded in the quad-tree structure.

TUs split from a CU may be split into smaller and lower TUs because a TU is split in the quad-tree structure. In HEVC, the size of a TU may be determined to be as one of <NUM>×<NUM>, <NUM>×<NUM>, <NUM>×<NUM> and <NUM>×<NUM>.

Referring back to <FIG>, the root node of a quad-tree is assumed to be related to a CU. The quad-tree is split until a leaf node is reached, and the leaf node corresponds to a TU.

This is described in more detail. A CU corresponds to a root node and has the smallest depth (i.e., depth=<NUM>) value. A CU may not be split depending on the characteristics of an input image. In this case, the CU corresponds to a TU.

A CU may be split in a quad-tree form. As a result, lower nodes having a depth <NUM> (depth=<NUM>) are generated. Furthermore, a node (i.e., leaf node) that belongs to the lower nodes having the depth of <NUM> and that is no longer split corresponds to a TU. For example, in FIG. <NUM>(b), a TU(a), a TU(b) and a TU(j) corresponding to the nodes a, b and j are once split from a CU and have a depth of <NUM>.

At least one of the nodes having the depth of <NUM> may be split in a quad-tree form again. As a result, lower nodes having a depth <NUM> (i.e., depth=<NUM>) are generated. Furthermore, a node (i.e., leaf node) that belongs to the lower nodes having the depth of <NUM> and that is no longer split corresponds to a TU. For example, in FIG. <NUM>(b), a TU(c), a TU(h) and a TU(i) corresponding to the node c, h and I have been split twice from the CU and have the depth of <NUM>.

Furthermore, at least one of the nodes having the depth of <NUM> may be split in a quad-tree form again. As a result, lower nodes having a depth <NUM> (i.e., depth=<NUM>) are generated. Furthermore, a node (i.e., leaf node) that belongs to the lower nodes having the depth of <NUM> and that is no longer split corresponds to a CU. For example, in FIG. <NUM>(b), a TU(d), a TU(e), a TU(f) and a TU(g) corresponding to the nodes d, e, f and g have been three times split from the CU and have the depth of <NUM>.

A TU having a tree structure may be hierarchically split with predetermined maximum depth information (or maximum level information). Furthermore, each spit TU may have depth information. The depth information may include information about the size of the TU because it indicates the split number and/or degree of the TU.

Information (e.g., a split TU flag "split_transform_flag") indicating whether a corresponding TU has been split with respect to one TU may be transferred to the decoder. The split information is included in all of TUs other than a TU of a minimum size. For example, if the value of the flag indicating whether a TU has been split is "<NUM>", the corresponding TU is split into four TUs. If the value of the flag indicating whether a TU has been split is "<NUM>", the corresponding TU is no longer split.

In order to reconstruct a current unit (or current block) on which decoding is performed, the decoded part of a current picture or other pictures including the current unit may be used. A picture (slice) using only a current picture for reconstruction, that is, on which intra-frame prediction only is performed may be called an intra-picture or an I picture (slice), a picture (slice) using a maximum of one motion vector and reference index in order to predict each unit may be called a predictive picture or a P picture (slice), and a picture (slice) using a maximum of two motion vectors and reference indices may be called a bi-predictive picture or a B picture (slice).

The intra-prediction unit performs intra prediction in which the pixel value of a target unit is predicted from reconstructed regions within a current picture. For example, the pixel value of a current unit may be predicted from pixels of units located at the top, left, top left and/or top right of a current unit.

An intra mode may be basically divided into vertical, horizontal, DC, and angular modes depending on the direction of a reference region in which reference pixels used for pixel value prediction are located and a prediction method. In the vertical mode, the pixel value of a region vertically neighboring an object unit is used as the predictor of a current unit. In the horizontal mode, the pixel value of a region horizontally neighboring an object unit is used as a predictor. In the DC mode, an average value of the pixels of reference regions is used as a predictor. Meanwhile, the angular mode corresponds to a case where a reference region is a specific direction, and may indicate a corresponding direction as an angle between a current pixel and a reference pixel. For convenience sake, a predetermined angle and a prediction mode number may be used, and the number of angles uses may be different depending on the size of a target unit.

Some specific mode for various prediction methods may be defined and used. A prediction mode may be transmitted as a value itself indicative of a corresponding mode, but in order to improve transmission efficiency, a method using the prediction mode value of a current unit may be used. In this case, the decoder may obtain the prediction mode of a current unit based on information indicating whether a predictor for a prediction mode is used without any change and a difference between the predictor and an actual value.

Meanwhile, the inter-prediction unit performs inter-prediction for predicting the pixel value of a target unit using information of other reconstructed pictures not a current picture. In this case, a reconstructed picture that belongs to reconstructed pictures stored in the DPB and that is used for inter-prediction is called a reference picture. In the inter-prediction process, an index, motion vector information, etc. indicative of a reference picture including a corresponding reference region may be used to indicate that which reference region is used to predict a current unit.

The inter-prediction may include forward direction prediction, backward direction prediction and bi-directional prediction. The forward direction prediction is prediction using a single reference picture displayed (or output) prior to a current picture temporally. The backward direction prediction means prediction using a single reference picture displayed (or output) after a current picture temporally. To this end, a single piece of motion information (e.g., motion vector and a reference picture index) may be necessary <IMG>. In the bi-directional prediction method, a maximum of two reference regions may be used. The two reference regions may be present in the same reference picture or may be present in different pictures. That is, in the bi-directional prediction method, a maximum of two pieces of motion information (e.g., a motion vector and a reference picture index) may be used. The two motion vectors may have the same reference picture index or may have different reference picture indices. In this case, all the reference pictures may be displayed (or output) prior to a current picture temporally or may be displayed (or output) after a current picture temporally.

Motion information of a current unit may include motion vector information and a reference picture index. The motion vector information may include a motion vector, motion vector prediction (MVP) or a motion vector difference (MVD) and may mean index information specifying a motion vector predictor. A motion vector difference means a difference between a motion vector and a motion vector predictor.

The encoder searches reference pictures for a reference unit most similar to a current unit in an inter-prediction process (i.e., motion estimation), and provides the decoder with a motion vector and reference picture index for the reference unit. The encoder/decoder may obtain the reference unit of the current unit using the motion vector and the reference picture index. The reference unit is present within a reference picture having a reference picture index. Furthermore, the pixel value of a specific unit or an interpolated value may be used as he predictor of the current unit based on the motion vector. That is, motion compensation in which an image of the current unit is predicted from a previously decoded picture is performed using motion information.

Meanwhile, a reference picture list may be configured using pictures used for inter-prediction with respect to a current picture. In the case of a B picture, two reference picture lists are necessary. Hereinafter, for convenience of description, the two reference picture lists are denoted as a reference picture list <NUM> (or LO) and a reference picture list <NUM> (or L1), respectively. Furthermore, a reference picture belonging to the reference picture list <NUM> is called a reference picture <NUM> (or L0 reference picture), and a reference picture belonging to the reference picture list <NUM> is called a reference picture <NUM> (or L1 reference picture). Such a reference picture may use a method of obtaining a motion vector predictor (mvp) using motion information of previously coded units and transmitting only the difference (mvd) of the motion vector predictors (mvp) in order to reduce the amount of transmission related to a motion vector. The decoder calculates a motion vector predictor of a current unit using pieces of motion information of other decoded units and obtains a motion vector value for the current unit using the transmitted difference. In obtaining the motion vector predictor, various motion vector candidate values may be obtained using motion information of already coded units, and one of the various motion vector candidate values may be obtained as a motion vector predictor.

As described above, in general, in the still image or moving image compression technology (e.g., HEVC), a block-based image compression method is used. In particular, when inter-prediction is performed, motion prediction is performed in a prediction block unit. There is still a limit although prediction blocks of various sizes or shapes are supported in order to search for an optimal prediction block for a current block. The reason for this is that a prediction error may be minimized when a pixel unit has a motion vector or has motion vectors of various shapes.

However, since transmitting a motion vector in a pixel unit or supporting the size or shape of an additional prediction block means an increase of additional information to be coded, performance improvement is difficult to expect.

Accordingly, the present invention proposes a method of performing motion compensation in a pixel unit without the partitioning of additional information or a prediction block. By applying a motion compensation method of a pixel unit (or a picture element unit) according to the present invention, a prediction error is reduced owning to motion compensation of a pixel unit and a case where partitioning is performed in a large size is increased. Accordingly, an increase of coding efficiency attributable to a reduction of a split flag can be expected.

Hereinafter, in the description of the present invention, it is assumed that the encoder has determined an inter-prediction index, a reference picture list or motion vector information (e.g., a motion vector, a motion vector predictor or a motion vector difference) indicating whether the reference picture list <NUM>, the reference picture list <NUM> or the bi-directional prediction (i.e., the reference picture lists <NUM> and <NUM>) are used for a current block by performing motion estimation in the aforementioned block unit (e.g., PU unit) and has provided such information to the decoder.

An optical flow refers to a motion pattern, such as an object or which surface or an edge in a view. That is, a pattern of a motion for an object is obtained by sequentially extracting differences between images at a specific time and a previous time. In this case, information about more motions can be obtained compared to a case where a difference between a current frame and a previous fame only is obtained. The optical flow has a very important contribution, such as that it enables a target point of a moving object to be obtained in the visual recognition function of an animal having a sense of view and helps to understand the structure of a surrounding environment. Technically, the optical flow may be used to analyze a three-dimensional image in the computer vision system or may be used for image compression. Several methods of realizing the optical flow have been proposed.

In accordance with the existing motion compensation method adopting the optical flow, the following equation is derived through two assumptions in which when an object moves for a short time, it moves at a specific rate in the state in which a corresponding pixel value is not changed.

A detailed derivation process is as follows.

First, it is assumed that when an object movers for a short time, a corresponding pixel value is not changed. It is assumed that a pixel value at (x, y) coordinates in time t is l(x, y, t) and a pixel value when an object moves δx(=Vx), δy(=Vy) for δt time is l(x + δx, y + δy, t + δt). According to the above assumption, Equation <NUM> below is established.

If a right term in Equation <NUM> is developed in Taylor series, it may be arranged as in Equation <NUM>.

Second, it is assumed that an object moves at a specific rate for a short time. This is described with reference to the following drawing.

<FIG> is an embodiment to which the present invention may be applied and illustrates a bi-directional prediction method of a picture having a steady motion.

<FIG> illustrates that bi-directional reference pictures (Ref) <NUM> and <NUM> are present based on a current picture (Cur Pic) <NUM>.

In this case, as described above, on the assumption that an object has a steady motion, an offset (i.e., a first motion vector) <NUM> from the coordinates of a current processing block <NUM> within the current picture (Cur Pic) <NUM> to the coordinates of a reference block A location within the reference picture <NUM>(Ref0) <NUM> and an offset (i.e., a second motion vector) <NUM> from the coordinates of the current processing block <NUM> within the current picture (Cur Pic) <NUM> to the coordinates of a reference block B location within the reference picture <NUM>(Ref1) <NUM> may be expressed as symmetrical values. That is, a first motion vector <NUM> related to the reference block A location and a second motion vector <NUM> related to the reference block B location may be expressed as having the same size and having opposite directions.

A difference between pixel values in the reference block A location and the reference block B location is arranged as in Equation <NUM> according to the aforementioned two assumptions.

In Equation <NUM>, (i, j) indicates the location of a specific pixel within the current processing block <NUM>.

Furthermore, <MAT>, and <MAT> indicate partial differentiations in an x axis (horizontal axis), a y axis (vertical axis), and a t axis (temporal axis), respectively. Gradients in the x axis and the y axis at the (i, j) location may be expressed as lx(k)[i, j] and ly(k)[i, j] (k = <NUM>, <NUM>), respectively. Furthermore, a gradient in the t axis, that is, a prediction pixel value, may be expressed P (k)[i, j] (k = <NUM>, <NUM>).

It has been assumed that when the object moves for a short time, a corresponding pixel value is not changed. Accordingly, motion vectors Vx(i, j) and Vy(i, j) of a pixel unit that minimize Δ<NUM>(i, j) can be obtained according to Equation <NUM>.

As a result, it is an object of searching for a motion vector in which the pixel value of the A reference block and the pixel value of the B reference block have the same value (or a value having a minimum difference), but an error between the pixels may be great. Accordingly, a motion vector in which a difference between the pixel values is a minimum within a specific window size may be searched for. Accordingly, assuming that a locally steady motion is present based on (i, j) within a window Ω, if the window includes (<NUM>+<NUM>) × (<NUM>+<NUM>), a location within the window may be indicated as (i', j'). In this case, (i', j') satisfies i - M ≤ i' ≤ i + M, j - M ≤ j' ≤ j + M.

Accordingly, a motion vector that minimizes ΣΩΔ<NUM>(i', j') is searched for.

Gx indicates the sum of the gradients in the x axis, Gy indicates the sum of the gradient in the y axis, and δP indicates the sum of gradients in the t axis, that is, the sum of prediction pixel values.

If each term of Equation <NUM> is arranged using Equation <NUM>, it may be expressed as in Equation <NUM>.

If Equation <NUM> is arranged through partial differentiation into Vx and Vy, it is expressed as in Equation <NUM>.

If s1 =ΣΩG<NUM>x, s<NUM> = s4 =ΣΩGxGy, s<NUM> = -ΣΩGxδP, s5 = ΣΩG<NUM>y, s6 = -ΣΩGyδP, Vx and Vy in Equation <NUM> are arranged in to an equation, such as Equation <NUM>.

Accordingly, a predictor may be calculated using Vx and Vy as in Equation <NUM>.

In Equation <NUM>, P[i, j] indicates a predictor for each pixel [i, j] within the current block. P^(<NUM>)[i, j] and P^(<NUM>)[i, j] indicate respective pixel values belonging to the L0 reference block and the L1 reference block, respectively.

The motion vector and reference value of each pixel unit may be obtained using the optical flow according to the above method.

However, the motion vector may be applied only when true bi-directional prediction, that is, the picture order count (POC) of a current picture, is located between the POCs of a reference picture, and is assumed to have the same Vx, Vy bi-directionally without taking into consideration the distance between the current picture and the two reference pictures. Accordingly, improvement is necessary.

Accordingly, the present invention proposes a motion compensation method of a pixel unit, which may be applied regardless of whether the POC of a current picture is located between the POCs of two reference pictures while taking into consideration the distance between the current picture and the two reference pictures. That is, there is proposed a method of generating a prediction block for a current block by deriving a motion vector of a pixel unit and deriving a predictor in a pixel unit based on the derived motion vector of a pixel unit.

In accordance with an embodiment of the present invention, a scaled motion vector is derived by taking into consideration the distance between a current picture and two reference pictures. This is described with reference to the following drawing.

<FIG> is a diagram illustrating a method of performing motion compensation in a pixel unit according to an embodiment of the present invention.

<FIG> illustrates a case where the distance between a current picture (Cur Pic) <NUM> and a L0 reference picture (RefO) <NUM> and the distance between the current picture (Cur Pic) <NUM> and an L1 reference picture (Ref1) <NUM> are different.

Assuming that the distance between the current picture <NUM> and the L0 reference picture <NUM>, that is, a short distance, is Tb and the distance between the current picture <NUM> and the L1 reference picture <NUM>, that is, a long distance, is Td, motion vectors <NUM> and <NUM> of a pixel unit to be obtained are scaled at the ratio of the distance between pictures (i.e., Tb and Td). In this case, the distance between pictures may be determined to be a difference between the POC values of the pictures.

This is expressed into Equation <NUM>.

(Vx(<NUM>), Vy(<NUM>)) indicates the motion vector <NUM> of the L0 reference picture <NUM> for the current block <NUM>, and (Vx(<NUM>), Vy(<NUM>)) indicates the motion vector <NUM> of the L1 reference picture <NUM> for the current block <NUM>. Furthermore, in Equation <NUM>, a scale factor α = Td/Tb.

If Equation <NUM> is applied to them, the difference between pixel values at the reference block A location and the reference block B location is arranged as in Equation <NUM>.

Motion vectors Vx(i, j) and Vy(i, j) of a pixel unit that minimize Δ<NUM>(i, j) may be derived using the same method as that of Embodiment <NUM>.

Furthermore, a predictor may be derived as in Equation <NUM> in each pixel unit of the current block using the motion vectors Vx(i, j) and Vy(i, j) of a pixel unit.

It is assumed that a current picture is a generalized B picture (i.e., a picture to which bi-directional prediction using two reference pictures is applied) and L0 and L1 reference pictures are present temporally in the same direction based on the current picture. In this case, if a reference picture L1' in the opposite direction is present in the DPB, one of reference blocks within the L0 and L1 reference pictures are substituted with a reference block within the reference picture L1' temporally in the opposite direction. That is, a reference block within the reference picture L1' temporally in the opposite direction is generated by scaling any one of the reference blocks within the L0 and L1 reference pictures. For example, a reference picture for the L0 and L1 reference pictures may be used as the reference picture in the opposite direction. This is described with reference to the following drawing.

<FIG>is a diagram illustrating a method of performing motion compensation in a pixel unit according to an embodiment of the present invention.

<FIG> illustrates a case where a current picture (Cur Pic) <NUM> is a B picture and an L0 reference picture (RefO) <NUM> and an L1 reference picture (Ref1) <NUM> are present temporally in the same direction.

In this case, the L1 reference picture (Ref1) <NUM> may be substituted with an L1' reference picture (Ref1') <NUM> so that the L0 reference picture <NUM> and the L1 reference picture <NUM> are located temporally in different directions. In other words, a reference block B within the L1 reference picture <NUM> may be substituted with a reference block B' within the L1' reference picture (Ref1') <NUM>.

In order to derive the reference block B', a motion vector (Vx(<NUM>), Vy(<NUM>))' is generated by scaling the motion vector (Vx(<NUM>), Vy(<NUM>)) of the L1 reference picture (Ref1) <NUM> for the current processing block <NUM> as the L1' reference picture (Ref1') <NUM>. That is, as in Equation <NUM>, the reference block B' may be derived by scaling the motion vector (Vx(<NUM>), Vy(<NUM>)) of the L1 reference picture (Ref1) <NUM> for the current processing block <NUM> temporally in the opposite direction.

In Equation <NUM>, a scale factor α = Tb/Td.

This method is more effective if the reference picture L1', the reference picture (Ref1') <NUM> is present in the distance closer to the current picture in the direction opposite the direction of the L0 reference picture (RefO) <NUM>.

If the L0 reference picture (RefO) <NUM> and the L1 reference picture (Ref1) <NUM> are the same picture, the reference block B' may be derived by scaling the motion vector of the L1 reference picture in the opposite direction.

In contrast, if the L0 reference picture (RefO) <NUM> and the L1 reference picture (Ref1) <NUM> are different pictures, it is effective to scale a reference picture far from the current picture <NUM> (i.e., a reference picture having a great POC difference).

If |POC(Cur) - POC(Ref0)| and |POC(Cur) - POC(Ref1')| are the same, a difference between pixel values at the reference block A location and the reference block B' location is modified as in Equation <NUM>.

Motion vectors Vx(i, j) and Vy(i, j) of a pixel unit that minimize Δ<NUM>(i, j) may be found by the aforementioned method.

Furthermore, a predictor may be calculated as in Equation <NUM> using the motion vectors.

In contrast, if |POC(Cur) - POC(Ref0)| and |POC(Cur) - POC(Ref1')| are different, the aforementioned scale factor α of Embodiment <NUM> may be applied. A difference between pixel values at the reference block A and the reference block B' location is modified as in Equation <NUM>.

In accordance with an embodiment of the present invention, if a current picture is a generalized B picture and only an L0 reference picture and an L1 reference picture are present in the same direction, a motion vector of a pixel unit and a predictor may be calculated using the L0 reference picture and the L1 reference picture. This is described with reference to the following drawing.

<FIG> illustrates a case where a current picture (Cur Pic) <NUM> is a B picture and an L0 reference picture (RefO) <NUM> and an L1 reference picture (Ref1) <NUM> are present in the same direction. Furthermore, it is assumed that the method of Embodiment <NUM> cannot be used because there is no reference picture in a different direction.

Assuming that the distance between the current picture <NUM> and the L0 reference picture <NUM>, that is, a short distance, is Tb and the distance between the current picture <NUM> and the L1 reference picture <NUM>, that is, a long distance, is Td, motion vectors <NUM> and <NUM> of a pixel unit to be calculated are scaled at the ratio of the distance between pictures (i.e., Tb and Td). This is expressed into Equation <NUM>.

(Vx(<NUM>), Vy(<NUM>)) indicates the motion vector <NUM> of the L0 reference picture <NUM> for the current processing block <NUM>, and (Vx(<NUM>), Vy(<NUM>)) indicates the motion vector <NUM> of the L1 reference picture <NUM> for the current processing block <NUM>. Furthermore, in Equation <NUM>, a scale factor γ = Td/Tb.

If this is applied to Equation <NUM>, a difference between pixel values at the reference block A location and the reference block B location is arranged as in Equation <NUM>.

If a current picture is a P picture (i.e., a picture to which unidirectional prediction using a single reference picture is applied), a motion vector that minimizes a difference between pixel values may be found using the reference block A of the current block and the reference block B' of the reference block A.

<FIG> illustrates a case where a current picture (Cur Pic) <NUM> is a P picture and an L0 reference picture (RefO) <NUM> is present. In this case, a motion vector that minimizes a difference between a pixel value at the location of the reference block A of the L0 reference picture (RefO) <NUM> and a pixel value at the location of the reference block B' of the L0' reference picture (RefO) <NUM> may be found.

In this case, the scale factor γ, such as that of Embodiment <NUM>, may be applied depending on the distance of the L0' reference picture (RefO) <NUM>.

A difference between pixel values at the reference block A and reference block B' locations, such as in the example of <FIG>, may be arranged as in Equation <NUM>.

Motion vectors Vx(i, j) and Vy(i, j) of a pixel unit that minimize Δ<NUM>(i, j) may be found by the aforementioned method. Furthermore, a predictor may be calculated as in Equation <NUM> using the motion vectors.

If bi-directional prediction is performed on the reference block of a current block, pixel unit motion compensation for the current block may be performed using any one of two reference blocks. This is described with reference to the following drawing.

In <FIG>, if a current picture (Cur Pic) <NUM> is a P picture and bi-directional prediction is applied to the reference block A of the L0 reference picture (RefO) <NUM> of the current block, a reference picture (an L1' reference picture (Ref1') <NUM> in the case of <FIG>) that belongs to the L0' reference picture (Ref0') <NUM> and L1' reference picture (Ref1') <NUM> of the reference block A of the current block and that is closer to the current picture (Cur Pic) <NUM> is selected. The reference block of the corresponding reference picture is set as B'.

If the distance between the current picture (Cur Pic) <NUM> and the L0' reference picture (Ref0') <NUM> and the distance between the current picture (Cur Pic) <NUM> and the L1' reference picture (Ref1') <NUM> are the same, a reference picture in a predetermined direction may be selected.

A motion vector that minimizes the pixel values of A and B' is calculated. In this case, a scale factor α may be applied as in Embodiment <NUM> depending on the distance of the L1' reference picture (Ref1') <NUM>.

A difference between pixel values at the reference block A and reference block B' locations may be arranged as in Equation <NUM>.

<FIG> is a diagram more specifically illustrating an inter-prediction unit according to an embodiment of the present invention.

Referring to <FIG>, the inter-prediction unit <NUM> (refer to <FIG>) and <NUM> (refer to <FIG>) implements the functions, processes and/or methods proposed in Embodiment <NUM> to Embodiment <NUM>. Specifically, the inter-prediction unit <NUM>, <NUM> may be configured to include a block unit motion vector refinement unit <NUM>, a pixel unit inter-prediction processing unit <NUM>, a predictor derivation unit <NUM> and a block unit inter-prediction processing unit <NUM>. Furthermore, the inter-prediction unit may further include a pixel unit inter-prediction determination unit <NUM>.

The block unit inter-prediction processing unit <NUM> is an element for processing an inter-prediction method defined in the existing still image or moving image compression technology (e.g., HEVC) and is a known technology, and thus a detailed description thereof is omitted.

The block unit motion vector refinement unit <NUM> refines a motion vector for a current block (derived from the block unit inter-prediction processing unit <NUM>) at the distance ratio of the current picture and a first reference picture/second reference picture.

The block unit motion vector refinement unit <NUM> may derive a gradient Ix in the x axis (horizontal) direction and a gradient ly in the y axis (vertical) direction based on a motion vector for a current block, and may apply an interpolation filter to the current block.

In this case, the distance between the current picture and the first reference picture/second reference picture may correspond to a difference between the POC of the current picture and the POC of the first reference picture/second reference picture.

The pixel unit inter-prediction processing unit <NUM> derives a motion vector of a pixel unit for each pixel within a current block based on the motion vector of the current block refined by the block unit motion vector refinement unit <NUM>, and derives a predictor for each pixel within the current block based on the motion vector of a pixel unit.

The predictor derivation unit <NUM> derives a predictor for a current block.

In this case, the predictor derivation unit <NUM> may use a predictor for each pixel within the current block derived by the pixel unit inter-prediction processing unit <NUM> or a predictor for the current block derived by the block unit inter-prediction processing unit <NUM> as the predictor for the current block.

Alternatively, the predictor derivation unit <NUM> may generate a predictor for a current block by the weighted sum of a first predictor for the current block derived by the block unit inter-prediction processing unit <NUM> and a second predictor derived based on a motion vector of a pixel unit derived by a pixel unit motion vector derivation unit <NUM>.

The pixel unit inter-prediction determination unit <NUM> determines whether to apply inter-prediction of a pixel unit (e.g., an optical flow).

For example, if a difference between two predictors derived through the inter-prediction of the existing block unit for a current block is greater than a predetermined threshold, the pixel unit inter-prediction determination unit <NUM> may not apply the inter-prediction of a pixel unit to the current block. If the inter-prediction of a pixel unit is not applied as described above, a predictor for the current block may be derived by the block unit inter-prediction processing unit <NUM>.

Alternatively, from a viewpoint of the encoder, the pixel unit inter-prediction determination unit <NUM> may calculate a rate-distortion cost (RD cost) between a case where inter-prediction of a block unit for a current block is applied and a case where inter-prediction of a pixel unit is applied, and may determine whether to apply inter-prediction of a pixel unit. Furthermore, the pixel unit inter-prediction determination unit <NUM> may signal information about whether to apply inter-prediction of a pixel unit to the decoder through a bit stream. In contrast, from a viewpoint of the decoder, the pixel unit inter-prediction determination unit <NUM> may receive signaling for information about whether to apply inter-prediction of a pixel unit from the encoder for a current block, and may determine whether to apply inter-prediction of a pixel unit.

<FIG> is a diagram illustrating a method of processing an image based on inter-prediction according to an embodiment of the present invention.

Referring to <FIG>, the encoder/decoder refines a motion vector for a current block at the distance ratio of a current picture and a first reference picture/second reference picture (S1201).

The encoder/decoder derives a motion vector of a pixel unit for each pixel within the current block based on the motion vector of the current block refined at step S1201 (S1202).

Furthermore, the encoder/decoder derives a predictor for the current block based on the motion vector of a pixel unit derived at step S1202 (S1203).

Referring to <FIG>, the encoder/decoder refines a motion vector for a current block at the distance ratio of the current picture and a first reference picture/second reference picture (S1101).

In this case, at step S1101, the first reference picture and the second reference picture may be determined using the aforementioned methods of Embodiment <NUM> to Embodiment <NUM>.

For example, the first reference picture and the second reference picture may be located temporally in the same direction or different directions based on the current picture.

Furthermore, if a current picture is a picture to which bi-directional inter-prediction is applied and two reference pictures for the current picture are present temporally in the same direction based on the current picture, one reference block of the two reference pictures may be substituted with a reference block of the second reference picture. In this case, a reference block of the second reference picture may be derived by temporally scaling any one motion vector of the two reference pictures for the current picture based on the current picture in different directions. <IMG>, if the two reference pictures for the current picture are different pictures, a reference block of a reference picture that belongs to the two reference pictures for the current picture and that has a greater POC difference than the current picture may be substituted with a reference block of the second reference picture.

In contrast, if a current picture is a picture to which unidirectional inter-prediction is applied, a reference picture of the current picture may be used as a first reference picture and a reference picture of the first reference picture may be used as a second reference picture. In this case, if the first reference picture is a picture to which bi-directional inter-prediction is applied, a reference picture that belongs to two reference pictures for the first reference picture and that has a smaller POC difference than the current picture may be used as the second reference picture.

The encoder/decoder derives a predictor for each pixel within the current block through (or by applying) inter-prediction of a pixel unit for each pixel within the current block based on the motion vector of the current block refined at step S1101 (S1202).

That is, the encoder/decoder derives a motion vector of a pixel unit for each pixel within the current block based on the refined motion vector of the current block, and derives a predictor for each pixel within the current block based on the derived motion vector of a pixel unit.

Steps S1102 and S1103 may be performed using the aforementioned methods of Embodiment <NUM> to Embodiment <NUM>.

Furthermore, the predictor calculated in Embodiments <NUM> to <NUM> may be applied as follows. One of the following methods may be selected and used or one or more of the methods may be selected, combined and used.

Hereinafter, for convenience of description, the same description as that in the examples of <FIG> and <FIG> is omitted.

The encoder/decoder determines whether a current slice (or picture) is a B slice (or picture) (S1301).

If, as a result of the determination at step S1301, the current slice (or picture) is a B slice (or picture), the encoder/decoder calculates (derives) gradients Ix and ly using a motion vector for a current block derived from the block-based inter-prediction (S1302).

In this case, in the case of an x tab filter (in the case of <FIG>, x=<NUM> and a width W and height H are increased by <NUM> pixels), the encoder/decoder may calculate (derive) the gradients Ix and ly in an interpolated reference picture.

The encoder/decoder interpolates the current block (S1303). As described above, the encoder/decoder may apply the x tab interpolation filter (in the case of <FIG>, x=<NUM> and a width W and height H are increased by <NUM> pixels) to the current block.

Steps S1302 and S1303 are repeatedly performed on an LO reference picture and an L1 reference picture.

The encoder/decoder calculates the refinement of the motion vector for the current block (S1305).

In this case, the encoder/decoder may refine the motion vector by scaling the motion vector for the current block derived from the block-based inter-prediction using the methods described in Embodiments <NUM> to <NUM>.

The encoder/decoder calculates a predictor for each pixel within the current block through inter-prediction of a pixel unit (e.g., an optical flow) for each pixel within the current block based on the refined motion vector (S1306).

In this case, the encoder/decoder may calculate the predictor for each pixel within the current block using the methods described in Embodiments <NUM> to <NUM>.

Furthermore, the encoder/decoder may identically use the predictor for each pixel derived by inter-prediction of a pixel unit as a predictor for the current block.

Meanwhile, if, as a result of the determination at step S1301, the current slice (or picture) is not a B slice (or picture), the encoder/decoder interpolates the current block (S1304).

Step S1304 is repeatedly performed on the L0 reference picture and an L1 reference picture.

The encoder/decoder calculates a predictor for the current block through the existing block-based inter-prediction method (S1307).

Furthermore, the encoder/decoder may use the predictor, derived by the block-based inter-prediction, as a predictor for the current block.

Referring to <FIG>, the encoder/decoder interpolates a current block (S1401).

As described above, the encoder/decoder may apply an x tab interpolation filter) (in the case of <FIG>, x=<NUM> and a width W and height H are increased by <NUM> pixels) to the current block.

Step S1401 is repeatedly performed on an L0 reference picture and an L1 reference picture.

The encoder/decoder calculates a predictor for the current block through the existing block-based inter-prediction method (S1402).

The encoder/decoder determines whether a current slice (or picture) is a B slice (or picture) (S1403).

If, as a result of the determination at step S1403, the current slice (or picture) is a B slice (or picture), the encoder/decoder calculates (derives) gradients Ix and ly using a motion vector for the current block derived from the block-based inter-prediction (S1404).

Step S1404 is repeatedly performed on the L0 reference picture and the L1 reference picture.

The encoder/decoder calculates the refinement of the motion vector for the current block (S1405).

In this case, the encoder/decoder may refine the motion vector for the current block derived from the block-based inter-prediction using the methods described in Embodiments <NUM> to <NUM>.

The encoder/decoder calculates a predictor for each pixel within the current block through inter-prediction of a pixel unit (e.g., an optical flow) for each pixel within the current block based on the refined motion vector (S1406).

The encoder/decoder generates the predictor for the current block by the weighted sum of a first predictor for the current block derived by the block unit inter-prediction at step S1402 and a second predictor derived based on the motion vector of a pixel unit at step S1406 (S1407).

In this case, the encoder/decoder may perform the weighted sum of the first predictor generated as an average value of P^(<NUM>) and P^(<NUM>) for the current block derived by the block unit inter-prediction and the second predictor derived based on the motion vector of a pixel unit.

In this case, the weighting factor of the weighted sum may be differently determined as a slice (or picture) or a block unit.

Furthermore, the weighting factor of the weighted sum may be determined by taking into consideration one or more of the distance between the current picture and a first reference picture/second reference picture (i.e., a POC difference), a difference between the two predictors (i.e., P^(<NUM>) and P^(<NUM>)) generated by the block-based inter-prediction, and the similarity between the motion vector of the first reference picture and the motion vector of the second reference picture.

Furthermore, the encoder/decoder may independently determine the weighting factor of the weighted sum according to the same rule, but the encoder may determine the weighting factor of the weighted sum and provide it to the decoder.

Meanwhile, if, as a result of the determination at step S1403, the current slice (or picture) is not a B slice (or picture), the encoder/decoder uses the predictor derived by the block-based inter-prediction as a predictor for the current block.

Referring to <FIG>, the encoder/decoder interpolates a current block (S1501).

As described above, the encoder/decoder may apply an x tab interpolation filter (in the case of <FIG>, x=<NUM>, and a width W and height H are increased by <NUM> pixels) to the current block.

Step S1501 is repeatedly performed on an L0 reference picture and an L1 reference picture.

The encoder/decoder calculates a predictor for the current block through the existing block-based inter-prediction method (S1502).

The encoder/decoder determines whether a sum of absolute difference (SAD) between reference blocks P^(<NUM>) and P^(<NUM>) for the current block derived by the block unit inter-prediction (i.e., SAD(P^(<NUM>)-P^(<NUM>))) is greater than a predetermined threshold (S1503).

If, as a result of the determination at step S1503, the SAD between the reference blocks P^(<NUM>) and P^(<NUM>) for the current block is smaller than the predetermined threshold, the encoder/decoder determines whether the current slice (or picture) is a B slice (or picture) (S1504).

If, as a result of the determination at step S1504, the current slice (or picture) is a B slice (or picture), the encoder/decoder calculates (derives) gradients |x and ly using a motion vector for the current block derived from the block-based inter-prediction (S1505).

Step S1505 is repeatedly performed on an L0 reference picture and an L1 reference picture.

The encoder/decoder calculates the refinement of the motion vector for the current block (S1506).

The encoder/decoder calculates a predictor for each pixel within the current block through inter-prediction of a pixel unit (e.g., an optical flow) for each pixel within the current block based on the refined motion vector (S1507).

Meanwhile, if, as a result of the determination at step S1503, the SAD between the reference blocks P^(<NUM>) and P^(<NUM>) for the current block is greater than the predetermined threshold or if, as a result of the determination at step S1504, the current slice (or picture) is a B slice (or picture), the encoder/decoder uses the predictor derived by the block-based inter-prediction as a predictor for the current block.

That is, if the SAD between the reference blocks P^(<NUM>) and P^(<NUM>) is greater than the threshold, a predictor to which an optical flow has been is not used because the assumption that a corresponding value is not changed when an object moves within a short time is violated.

Referring to <FIG>, the encoder interpolates a current block (S1601).

As described above, the encoder may apply an x tab interpolation filter (in the case of <FIG>, x=<NUM>, and a width W and height H are increased by <NUM> pixels) to the current block.

Step S1601 is repeatedly performed on an L0 reference picture and an L1 reference picture.

The encoder calculates a predictor for the current block through the existing block-based inter-prediction method (S1602).

The encoder determines whether a current slice (or picture) is a B slice (or picture) (S1603).

If, as a result of the determination at step S1603, the current slice (or picture) is a B slice (or picture), the encoder calculates (derives) gradients Ix and ly using a motion vector for the current block derived from the block-based inter-prediction (S1604).

Step S1604 is repeatedly performed on the L0 reference picture and the L1 reference picture.

The encoder calculates the refinement of the motion vector for the current block (S1605).

In this case, the encoder may refine the motion vector by scaling the motion vector for the current block derived from the block-based inter-prediction using the methods described in Embodiments <NUM> to <NUM>.

The encoder calculates a predictor for each pixel within the current block through inter-prediction of a pixel unit (e.g., an optical flow) for each pixel within the current block based on the refined motion vector (S1606).

In this case, the encoder may calculate the predictor for each pixel within the current block using the methods described in Embodiments <NUM> to <NUM>.

The encoder calculates a rate-distortion cost (RD cost) between a case where inter-prediction of a block unit (e.g., CU or PU unit) for the current block has been applied and a case where inter-prediction of a pixel unit (e.g., an optical flow) has been applied, and determines whether to apply the inter-prediction of a pixel unit (S1607).

Furthermore, the encoder may signal information about whether to apply the inter-prediction of a pixel unit to the decoder through a bit stream.

In the aforementioned embodiments, the elements and characteristics of the present invention have been combined in specific forms. Each of the elements or characteristics may be considered to be optional unless otherwise described explicitly. Each of the elements or characteristics may be implemented in a form in which it is not combined with other elements or characteristics. Furthermore, some of the elements and/or the characteristics may be combined to form an embodiment of the present invention. Order of the operations described in the embodiments of the present invention may be changed. Some of the elements or characteristics of an embodiment may be included in another embodiment or may be replaced with corresponding elements or characteristics of another embodiment. It is evident that an embodiment may be configured by combining claims not having an explicit citation relation in the claims or may be included as a new claim by amendments after filing an application.

The embodiment according to the present invention may be implemented by various means, for example, hardware, firmware, software or a combination of them. In the case of an implementation by hardware, the embodiment of the present invention may be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc..

In the case of an implementation by firmware or software, the embodiment of the present invention may be implemented in the form of a module, procedure or function for performing the aforementioned functions or operations. Software code may be stored in memory and driven by the processor. The memory may be located inside or outside the processor and may exchange data with the processor through a variety of known means.

The detailed description should not be construed as being limitative from all aspects, but should be construed as being illustrative. The scope of the present invention should be determined by reasonable analysis of the attached claims.

Claim 1:
A method of processing an image based on inter-prediction, comprising steps of:
obtaining a first motion vector (<NUM>) of a current block (<NUM>) to which uni-directional inter-prediction using a single reference picture is applied, the first motion vector (<NUM>) indicating a first reference block (A) in a first reference picture (<NUM>),
obtaining a second motion vector of the first reference block, the second motion vector indicating a second reference block in a second reference picture (<NUM>);
determining a scale factor based on a ratio of a difference between a Picture Order Count, POC, of a current picture including the current block and a POC of the first reference picture and a difference between a POC of the current picture and the POC of the second reference picture;
determining a scaled motion vector for each pixel within the current block to minimize a square of a pixel value difference of two corresponding pixels respectively included in the first and second reference blocks by using an optical flow method in consideration of the scale factor;
deriving a predictor for the each pixel within the current block based on the scaled motion vector.