Encoding and decoding method, apparatus and communication system

A method for predicting an image is provided. The method include (i) determining prediction parameters of a current block in a bitstream; (ii) determining a matrix-based intra prediction (MIP) input sample of the current block based on neighboring samples of the current block and the prediction parameters; (iii) setting a shifting number parameter (sW) as a first fixed value; (iv) setting a shifting offset parameter (fO) as a second fixed value; and (v) determining an MIP prediction sample of the current block based on an MIP weighting matrix, the MIP input sample, the shifting offset parameter (fO), and the shifting number parameter (sW).

TECHNICAL FIELD

The present disclosure relates to the field of telecommunication technologies, and in particular, to a method for encoding and decoding images such as pictures or videos.

BACKGROUND

Versatile Video Coding (VVC) is a next generation video compression standard used to replace a current standard such as High Efficiency Video Coding standard (H.265/HEVC). The VVC coding standard provides higher coding quality than the current standard. To achieve this goal, various intra and inter prediction modes are considered. When using these prediction modes, a video can be compressed such that data to be transmitted in a bitstream (in binary form) can be reduced. Matrix-based intra prediction (MIP) is one of these modes. The MIP is an intra prediction mode. When implementing under the MW mode, an encoder or a decoder can derive an intra prediction block based on a current coding block (e.g., a group of bits or digits that is transmitted as a unit and that may be encoded and/or decoded together). However, deriving such prediction blocks may require significant amount of computational resources and additional storage spaces. Therefore, an improved method for addressing this issue is advantageous and desirable.

SUMMARY

When implementing an MIP process, various prediction parameters are determined and then utilized. Traditionally, a few of these prediction parameters can be determined by look-up tables. These look-up tables need to be stored in a component (e.g., a memory, a cache, etc.) of an encoder and/or a decoder and thus require storage spaces. In addition, accessing these look-up tables consumes computing time and resources. Therefore, it is advantageous to have an improved method, apparatus, and system to address the foregoing issue.

The present disclosure provides a method for predicting, encoding, and/or decoding an image based on an MIP process. The MIP process can generate a prediction block of a current block, and the size of the prediction block is smaller than the size of the current block. For example, an “8×8” current block can have a “4×4” prediction block. An MIP prediction block with its size smaller than the current block is derived by performing a matrix calculation, which consumes less computational resources than performing the matrix calculation with a larger block. After the matrix calculation, an upsampling process is applied to the MIP prediction block to derive an intra prediction block that is of the same size of the current block. For example, an “8×8” intra prediction block can be derived from a “4×4” MIP prediction block by invoking the upsampling process of interpolation and/or extrapolation.

More particularly, the present method includes, for example, (i) determining prediction parameters of a current block in a bitstream; (ii) determining an MIP input sample (e.g., “p[x]” in equations (P-1), (P-2), and (P-3) discussed in detail below) of the current block based on neighboring samples of the current block and the prediction parameters; (iii) setting a shifting number parameter (e.g., “sW” in equation (B) discussed in detail below) as a first fixed value; (iv) setting a shifting offset parameter (e.g., “fO” in equation (B) discussed in detail below) as a second fixed value; (v) determining an MIP weighting matrix of the current block based on the prediction parameters; (vi) determining an MIP prediction sample (e.g., “predMip[x][y]” in equation (C) discussed in detail below) of the current block based on the MIP weighting matrix, the MIP input sample, the shifting offset parameter (fO), and the shifting number parameter (sW); and (vii) performing an upsampling process to the MIP prediction sample so as to generate intra predicted samples (e.g., “predSamples[x][y]” in equation (G) discussed in detail below) of the current block.

Without wishing to be bounded by theory, setting either or both of the shifting number parameter and the shifting offset parameter as fixed values effectively improves the overall encoding/decoding efficiency without significantly affecting the accuracy of the encoding/decoding processes. By this arrangement, the present methods provide a solution to significantly shorten computing time and reduce required storage space when implementing MIP processes.

Another aspect of the present disclosure includes a system for encoding/decoding pictures and videos. The system can include an encoding sub-system (or an encoder) and a decoding sub-system (or a decoder). The encoding sub-system includes a partition unit, a first prediction unit, and an entropy coding unit. The partition unit is configured to receive an input video and divide the input video into one or more coding units (CUs). The first intra prediction unit is configured to generate a prediction block corresponding to each CU based on prediction parameters derived from encoding the input video. The entropy coding unit is configured to transform the parameters for deriving the prediction block into a bitstream. The decoding sub-system includes a parsing unit and a second intra prediction unit. The parsing unit is configured to parse the bitstream to get numerical values (e.g., values associated with the one or more CUs). The second intra prediction unit is configured to convert the numerical values into an output video based on the prediction parameters.

DETAILED DESCRIPTION

In order to facilitate the understanding of the present disclosure, the present disclosure will be described more fully hereinafter with reference to the accompanying drawings.

FIG. 1Ais a schematic diagram of a system100according to an embodiment of the present disclosure. The system100can encode, transmit, and decode a picture. The system100can also be applied to encode, transmit and decode a video consisting of a sequence of pictures. More particularly, the system100can receive input pictures, process the input pictures, and generate output pictures. The system100includes an encoding apparatus100aand a decoding apparatus100b. The encoding apparatus100aincludes a partition unit101, a first intra prediction unit103, and an entropy coding unit105. The decoding apparatus100bincludes a parsing unit107and a second intra prediction unit109.

The partition unit101is configured to receive an input video10and then divide the input video10into one or more coding tree units (CTUs) or coding units (CUs)12. The CUs12are transmitted to the first intra prediction unit103. The first intra prediction unit103is configured to derive a prediction block for each of the CUs12by performing an MW process. In some embodiments, based on the sizes of the CUs12, the MIP process has different approaches to handle the CUs12with different sizes. For example, for each type of CUs12, it has a designated MIP size identifier (e.g., 0, 1, 2, etc.).

The first intra prediction unit103first determines prediction parameters (e.g., a width, a height, a size, etc.) of the CU12. Accordingly, the first intra prediction unit103can determine an MIP size identifier of the CU12. The first intra prediction unit103further derives a group of reference samples for the CU12(e.g., using neighboring samples of the CU12, such as above- or left-neighboring samples, discussed in detail with reference toFIG. 3). The first intra prediction unit103then derives an MIP prediction of the CU12based on the group of reference samples and a corresponding MIP weighting matrix. The first intra prediction unit103can use the MIP prediction as an intra prediction14of the CU12. The intra prediction14and the prediction parameters for deriving the intra prediction14are then transmitted to the entropy coding unit105for further process.

The entropy coding unit105is configured to transform the prediction parameters for deriving the intra prediction14into binary form. Accordingly, the entropy coding unit105generates a bitstream16based on the intra prediction14. In some embodiments, the bitstream16can be transmitted via a communication network or stored in a disc or a server.

The decoding apparatus100breceives the bitstream16as input bitstream17. The parsing unit107parses the input bitstream17(in binary form) and converts it into numerical values18. The numerical values18is indicative of the characteristics (e.g., color, brightness, depth, etc.) of the input video10. The numerical values18is transmitted to the second intra prediction unit109. The second intra prediction unit109can then convert these numerical values18into an output video19(e.g., based on processes similar to those performed by the first intra prediction unit103; relevant embodiments are discussed in detail with reference toFIG. 4). The output video19can then be stored, transmitted, and/or rendered by an external device (e.g., a storage, a transmitter, etc.). The stored video can further be displayed by a display.

FIG. 1Bis a schematic diagram illustrating an MIP process S100in accordance with embodiments of the present disclosure. The MW process includes four major steps S101, S102, S103, and S104. In step S101, prediction parameters for the MIP process S100are to be determined. The prediction parameters include a type or size (e.g., indicated by an MIP size identifier, such as “mipSizeId”) of a current block, the number of reference samples in each boundary (e.g., “boundySize”), the number of MIP input samples (e.g., “inSize”), and the dimension of an MIP prediction block (e.g., “predSizexpredSize”) generated by matrix multiplication. Based on the size of the current block, the MIP size identifier can be “0,” “1,” or “2,” which is recorded in parameter “mipSizeId.” Different blocks can have different prediction parameters.

In step S102, the MIP process obtains reference pixels. The references pixels can be from above-neighboring blocks and/or left-neighboring blocks. The pixels from the above-neighboring blocks are stored as parameter “refT” and the pixels from the left-neighboring blocks are stored as parameter “refL.”

In step S103, input samples for the MIP process are determined. The input samples can be determined by three sub-steps, S1031, S1032, and S1033. In sub-step S1031, based on the reference pixels (e.g., from step S102), a downsampling process is performed to generate reference samples. In sub-step S1031, reference sampling areas (or buffer sampling areas) are determined. In some embodiments, a matrix transpose process can be involved in sub-step S1032. In sub-step S1033, a matrix multiplication calculation is performed based on the input samples.

In step S104, intra MW prediction samples are generated. Step S104includes four sub-steps, S1041, S1042, S1043, and S1044. In sub-step S1041, an MIP prediction block is generated based on an MIP weighting matrix, a shifting offset parameter (e.g., “fO”), and a shifting number parameter (e.g., “sW”). In the present disclosure, the shifting offset parameter and the shifting number parameter are set as fixed values. In sub-step S1042, an interpolation process is performed to the MIP prediction block. In sub-step S1043, the MIP prediction block is transposed. In sub-step S1044, the intra MIP prediction samples are generated. In some embodiments, the intra MW prediction samples can be generated by considering the prediction samples from the above-neighboring pixels. In other embodiments, the intra MIP prediction samples can be generated without considering the prediction samples from the above-neighboring pixels (e.g., only considering the prediction samples from the left-neighboring pixels). Details of the MIP process are discussed below with reference toFIG. 3and corresponding equations.

FIG. 2is a schematic diagram of an encoding system200according to an embodiment of the present disclosure. The encoding system200is configured to encode, compress, and/or process an input picture20and generate an output bitstream21in binary form. The encoding system200includes a partition unit201configured to divide the input picture20into one or more coding tree units (CTUs)22. In some embodiments, the partition unit201can divide the picture into slices, tiles, and/or bricks. Each of the bricks can contain one or more integral and/or partial CTUs22. In some embodiments, the partition unit201can also form one or more subpictures, each of which can contain one or more slices, tiles or bricks. The partition unit201transmits the CTUs22to a prediction unit202for further process.

The prediction unit202is configured to generate a prediction block23for each of the CTUs22. The prediction block23can be generated based on one or more inter or intra prediction methods by using various interpolation and/or extrapolation schemes. As shown inFIG. 2, the prediction unit202can further include a block partition unit203, an ME (motion estimation) unit204, an MC (motion compensation) unit205, and an intra prediction unit206. The block partition unit203is configured to divide the CTUs22into smaller coding units (CUs) or coding blocks (CBs). In some embodiments, the CUs can be generated from the CTUs22by various methods such as quadtree split, binary split, and ternary split. The ME unit204is configured to estimate a change resulting from a movement of an object shown in the input picture20or a movement of a picture capturing device that generates the input picture20. The MC unit205is configured to adjust and compensate a change resulting from the foregoing movement. Both the ME unit204and the MC unit205are configured to derive an inter (e.g., at different time points) prediction block of a CU. In some embodiments, the ME unit204and the MC unit205can use a rate-distortion optimized motion estimation method to derive the inter prediction block.

The intra prediction unit206is configured to derive an intra (e.g., at the same time point) prediction block of a CU (or a portion of the CU) using various intra prediction modes including MIP modes. Details of deriving of an intra prediction block using an MIP mode (referred to as “MIP process” hereinafter) is discussed with reference toFIG. 1BandFIG. 3. During the MIP process, the intra prediction unit206first derives one or more reference samples from neighboring samples of the CU, by, for example, directly using the neighboring samples as the reference samples, downsampling the neighboring samples, or directly extracting from the neighboring samples (e.g., Step301ofFIG. 3).

Second, the intra prediction unit206derives predicted samples at multiple sample positions in the CU using the reference samples, an MIP matrix and a shifting parameter. The sample positions can be preset sample positions in the CU. For example, the sample positions can be positions with odd horizontal and vertical coordinate values within the CU (e.g., x=1, 3, 5, etc.; y=1, 3, 5, etc.). The shifting parameter includes a shifting offset parameter and a shifting number parameter, which can be used in shifting operations in generating the predicted samples. By this arrangement, the intra prediction unit206can generate predicted samples in the CU (i.e., “MIP prediction” or “MIP prediction block” refers to a collection of such predicted samples) (e.g., Step302ofFIG. 3). In some embodiments, the sample positions can be positions with even horizontal and vertical coordinate values within the CU.

Third, the intra prediction unit206can derive predicted samples at remaining positions (e.g., those are not sample positions) of the CU (e.g., Step303ofFIG. 3). In some embodiments, the intra prediction unit206can use an interpolation filter to derive the predicted samples at the remaining positions. By the foregoing processes, the intra prediction unit206can generate the prediction block23for the CU in the CTU22.

Referring toFIG. 2, the prediction unit202outputs the prediction block23to an adder207. The adder207calculates a difference (e.g., a residual R) between the output (e.g., a CU in the CTUs22) of the partition unit201and the output (i.e., the prediction block23of the CU) of the prediction block202. A transform unit208reads the residual R, and performs one or more transform operations on the prediction block23to get coefficients24for further uses. A quantization unit209can quantize the coefficients24and outputs quantized coefficients25(e.g., levels) to an inverse quantization unit210. The inverse quantization unit210performs scaling operations on the quantized coefficients25to output reconstructed coefficients26to an inverse transform unit211. The inverse transform unit211performs one or more inverse transforms corresponding to the transforms in the transform unit208and outputs reconstructed residual27.

An adder212then calculates reconstructed CU by adding the reconstructed residual27and the prediction block23of the CU from the prediction unit202. The adder212also forwards its output28to the prediction unit202to be used as an intra prediction reference. After all the CUs in the CTUs22have been reconstructed, a filtering unit213can perform an in-loop filtering on a reconstructed picture29. The filtering unit213contains one or more filters, for example, a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a luma mapping with chroma scaling (LMCS) filter, a neural-network-based filter and other suitable filters for suppressing coding distortions or enhancing coding quality of a picture.

The filtering unit213can then send a decoded picture30(or subpicture) to a decoded picture buffer (DPB)214. The DPB214outputs decoded picture31based on controlling information. The picture31stored in the DPB214may also be employed as a reference picture for performing inter or intra prediction by the prediction unit202.

An entropy coding unit215is configured to convert the pictures31, parameters from the units in the encoding system200, and supplemental information (e.g., information for controlling or communicating with the system200) into binary form. The entropy coding unit215can generate the output bitstream21accordingly.

In some embodiments, the encoding system200can be a computing device with a processor and a storage medium with one or more encoding programs. When the processor reads and executes the encoding programs, the encoding system200can receive the input picture20and accordingly generates the output bitstream21. In some embodiments, the encoding system200can be a computing device with one or more chips. The units or elements of the encoding system200can be implemented as integrated circuits on the chips.

FIG. 3is a schematic diagram illustrating an MIP process in accordance with embodiments of the present disclosure. The MIP process can be implemented by an intra prediction unit (e.g., the intra prediction unit206). As shown inFIG. 3, the intra prediction unit can include a prediction module301and a filtering module302. As also shown inFIG. 3, the MIP process includes three Steps301,302, and303. The MIP process can generate a predicted block based on a current block or a coding block300(such as a CU or partitions of a CU).

In Step301, the intra prediction unit can use neighboring samples31,33of the coding block300to generate reference samples32,34. In the illustrated embodiment, the neighboring samples31are above-neighboring samples, and the neighboring samples33are left-neighboring samples. The intra prediction unit206can calculate an average of the values of every two neighboring samples31,33and set the average of the values as the values of the reference samples32,34, respectively. In some embodiments, the intra prediction unit206can select the value of one from every two neighboring samples31or33as the value of the reference sample32or32. In the illustrated embodiments, the intra prediction unit206derives 4 reference samples32from 8 above-neighboring samples31of the coding block300, and another 4 reference samples34from 8 left-neighboring samples33of the coding block300.

In Step301, the intra prediction unit determines a width and a height of the coding block300and denotes them as variables “cbWidth” and “cbHeight,” respectively. In some embodiments, the intra prediction unit206can adopt a rate-distortion optimized mode decision process to determine an intra prediction mode (e.g., whether an MIP mode is used). In such embodiments, the coding block300can be partitioned into one or more transform blocks, whose width and height are noted as variables “nTbW” and “nTbH,” respectively. When the MIP mode is used as the intra prediction mode, the intra prediction unit determines an MIP size identifier (denoted as variable “mipSizeId”) based on the following conditions A-C.

As an example, if the size of the coding block300is “8×8” (i.e. both “cbWidth” and “cbHeight” are 8), then “mipSizeId” is set as 2. As another example, if the size of the transformed block of the coding block300is “4×4” (i.e. both “nTbW” and “nTbH” are 4), then “mipSizeId” is set as 0. As yet another example, if the size of the coding block300is “4×8,” then “mipSizeId” is set as 1.

In the illustrated embodiments, there are three types of “mipSizeId,” which are “0,” “1,” and “2.” Each type of MIP size identifiers (i.e., variable “mipSizeId”) corresponds to a specific way of performing the MIP process (e.g., use different MIP matrices). In other embodiments, there can be more than three types of MIP size identifiers.

Based on the MIP size identifier, the intra prediction unit can determine variables “(Size” and “predSize” based on the following conditions D-F.

In the illustrated embodiments, “boundarySize” represents a number of reference samples32,34derived from each of the above-neighboring samples31and the left-neighboring samples33of the coding block300. Variable “predSize” is to be used in a later calculation (i.e., equation (C) below).

In some embodiments, the intra prediction unit can also derive variable “isTransposed” to indicate the order of reference samples32,34stored in a temporal array. For example, “isTransposed:” being “0” indicates that the intra prediction unit presents the reference samples32derived from the above-neighboring samples31of the coding block300ahead of the reference samples34derived from the left-neighboring samples33. Alternatively, “isTransposed” being 1 indicates that the intra prediction unit presents the reference samples34derived from the left-neighboring samples33of the coding block300ahead of the reference samples32derived from the above-neighboring samples31. In an implementation of the encoding system200, the value of “isTransposed” is sent to an entropy coding unit (e.g., the entropy coding unit215) as one of the parameters of the MIP process that is coded and written into a bitstream (e.g., the output bitstream21). Correspondingly, in an implementation of a decoding system400inFIG. 4described in this disclosure, the value of “isTransposed” can be received from a parsing unit (e.g., parsing unit401) by parsing an input bitstream (which can be the output bitstream21).

The intra prediction unit can further determine a variable “inSize” to indicate the number of reference samples32,34used in deriving an MIP prediction. A value of “inSize” is determined by the following equation (A). In this disclosure, meanings and operations of all operators in equations are the same as the counterpart operators that are defined in the ITU-T H.265 standard.
inSize=(2*boundarySize)−(mipSizeId==2)?1:0;  (A)

For example, “==” is a relational operator “Equal to”. For example, if “mipSizeId” is 2, then “inSize” is 7 (calculated by (2*4)−1). If “mipSizeId” is 1, then “inSize” is 8 (calculated by (2*4)−0). In some embodiments, the parameter “inSize” can be found in Size-Id Table below.

The intra prediction unit can invoke the following process to derive a group of reference samples32,34, which are stored in array p[x] (“x” is from “0” to “inSize−1”). The intra prediction unit can derive “nTbW” samples from the above-neighboring samples31of the coding block300(and store them in array “refT”) and “nTbH” samples from the left-neighboring samples33(and store them in array “refL”) of the coding block300.

The intra prediction unit can initial a downsampling process on “refT” to get “boundarySize” samples and store the “boundarySize samples” in “refT.” Similarly, the intra prediction unit206can initiate the downsampling process on “refL” to get “boundarySize” samples and store the “boundarySize” samples in “refL.”

In some embodiments, the intra prediction unit can incorporate arrays “refT” and “refL” into a single array “pTemp” based on the order indicated by a variable “isTransposed.” The intra prediction unit can derive “isTransposed” to indicate the order of reference samples stored in a temporal array “pTemp.” For example, “isTransposed” being “0” (or FALSE) indicates that the intra prediction unit presents the reference samples32derived from the above-neighboring samples31of the coding block300ahead of the reference samples34derived from the left-neighboring samples33. In other cases, “isTransposed” being “1” (or TRUE) indicates that the intra prediction unit presents the reference samples34derived from the left-neighboring samples33of the coding block300ahead of the reference samples32derived from the above-neighboring samples31. In some embodiments, in an implementation of the encoding system200, the intra prediction unit can determine a value of “isTransposed” by using a rate-distortion optimization method. In some embodiments, in an implementation of the encoding system200, the intra prediction unit can determine the value of “isTransposed” based on comparisons and/or correlations between neighboring samples32,34and the coding block300. In an implementation of the encoding system200, the value of “isTransposed” can be forwarded to the entropy coding unit (e.g., the entropy coding unit215) as one of the parameters of the MW process to be written in the bitstream (e.g., the output bitstream21). Correspondingly, in an implementation of a decoding system400inFIG. 4described in this disclosure, the value of “isTransposed” can be received from a parsing unit (e.g. parsing unit401) by parsing an input bitstream (which can be the output bitstream21).

The intra prediction unit can determine array “p[x]” (x from “0” to “inSize−1”) based on the following conditions G and H.

In the above condition H, “BitDepth” is a bit depth of a color component of a sample (e.g., Y component) in the coding block300. The symbol “<<” is a bit shifting symbol used in the ITU-T H.265 standard.

Alternatively, the intra prediction unit can derive array p[x] (for x from 0 to “inSize−1” based on the following conditions I and J.

In some embodiments, the intra prediction unit can determine the values of array p[x] by using a unified calculation method without judging the value of “mipSizeId.” For example, the intra prediction unit can append “(1<<(BitDepth−1))” as an additional element in “pTemp,” and calculate p[x] as “pTemp[x]−pTemp[0].”

Equations P-1 and P-2 apply to cases where “mipSizeId” is equal to “0” or “1,” which means that the selected size parameter is within the predetermined range. After determining that the selected size parameter of a current block is in the predetermined range, “1<<(BitDepth−1)” can be determined. Then a difference between “1<<(BitDepth−1)” and “pTemp[0]” can be determined. The difference is then set as p[0]. Then p[x] can be calculated based on the equations P-1 and P-2 above.

Equation P-3 applies to cases where the “mipSizeId” is equal to “2,” which means the selected size parameter is not within the predetermined range. Assuming that the current block is a 4×4 block, which means that there four values in buffer area “pTemp.” These four values are pTemp[0], pTemp[1], pTemp[2], and pTemp[3]. Based on equation P-3 above and these four values, array “p[x]” can be determined. Values in array “p[x]” can be called MIP input samples.

In Step302, the intra prediction unit (or the prediction module301) derives the MIP prediction of the coding block300by using the group of reference samples32,34and an MIP matrix. The MIP matrix is selected from a group of predefined MIP matrices based on its corresponding MIP mode identifier (i.e., variable “mipModeId”) and the MIP size identifier (i.e. variable “mipSizeId”).

The MIP prediction derived by the intra prediction unit includes partial predicted samples35of all or partial sample positions in the coding block300. The MIP prediction is denoted as “predMip[x][y].”

In the illustrated embodiment inFIG. 3, partial predicted samples35are samples marked as grey squares in the current block300. The reference samples32,34in array p[x] derived in Step301are used as an input to the prediction module301. The prediction module301calculates the partial predicted samples35by using the MIP matrix and a shifting parameter. The shifting parameter includes a shifting offset parameter (“fO”) and a shifting number parameter (“sW”). In some embodiment, the prediction module301derives the partial predicted sample35with its coordinate (x, y) based on the following equations (B) and (C):
oW=(1<<(sw−1)−fO*(Σi=0inSize−1p[i])  (B)
predMip[x][y]=(((Σi=0inSize−1mWeight[i][y*predSize+x]*p[i])+oW)>>sW)+pTemp[0] (forxfrom 0 to “predSize−1”,foryfrom 0 to “predSize−1”)  (C)

In equation Q above, “mWeight[i][j]” is an MIP weighting matrix in which matrix elements are fixed constants for both encoding and decoding. Alternatively, in some embodiments, an implementation of the encoding system200uses adaptive MIP matrix. For example, the MIP weighting matrix can be updated by various training methods using one or more coded pictures as input, or using pictures provided to the encoding system200by external means. The intra prediction unit can forward “mWeight[i][j]” to an entropy coding unit (e.g., the entropy coding unit215) when an MIP mode is determined. The entropy coding unit can then write “mWeight[i][j]” in the bitstream, e.g. in one or more special data units in the bitstream containing MIP data. Correspondingly, in some embodiments, an implementation of a decoding system400with adaptive MIP matrix can update MIP matrix using, for example, training method with input of one or more coded pictures or blocks or pictures from other bitstream provided by external meanings, or obtained from parsing unit401by parsing special data units in the input bitstream containing MIP matrix data.

The prediction unit301can determine the values of “sW” and “fO” based on the size of the current block300and the MIP mode used for the current block300. In some embodiments, the prediction unit301can obtain the values of “sW” and “fO” by using a look-up table. For example, Table 1 below can be used to determine “sW.”

In some embodiments, the shifting number parameter “sW” can be set as a first fixed value, such as 5 or 6. In such embodiments, there is no need to use Table 1 above to look up the value of the shifting number parameter “sW.” For example, when “mipSizeId” is equal to “0” or “2,” the shifting number parameter “sW” can be set as “5.” As another example, when “mipSizeId” is equal to “2,” the shifting number parameter “sW” can be set as “5.” Table 2 below shows different settings of the shifting number parameter “sW.” In some embodiments, the shifting number parameter “sW.” can be set by the prediction module301.

In some embodiments, the prediction module can set “sW” as a constant. For example, the prediction module can “sW” as “5” for blocks of various sizes with different MIP modes. As another example, the prediction module301can set “sW” as “6” for blocks of various sizes with different MIP modes. As yet another example, the prediction module can set “sW” as “7” for blocks of various sizes with different MIP modes.

In some embodiments, the prediction unit301can use Table 3 below to determine the shifting offset parameter “fO.”

In some embodiments, the shifting offset parameter “fO” can be set as a second fixed value, such as 23, 32, 46, 56, or 66. The second fixed value has a preferred range of 1-100. In such embodiments, there is no need to use Table 3 above to look up the value of the shifting offset parameter “fO.” In some embodiments, the shifting offset parameter “fO” can be set based on parameter “mipSizeId.” For example, when “mipSizeId” is equal to “0,” the shifting offset parameter “fO” can be set as “34.” As another example, when “mipSizeId” is equal to “1,” the shifting offset parameter “fO” can be set as “23.” As yet another example, when “mipSizeId” is equal to “1,” the shifting offset parameter “fO” can be set as “46.” Table 4 below shows different settings of the shifting offset parameter “fO.” In some embodiments, the shifting offset parameter “fO” can be set by the prediction module301.

In some embodiments, the intra prediction unit can perform a “clipping” operation on the value of the MIP prediction samples stored in array “predMip.” When “isTransposed” is 1 (or TRUE), the “predSize×preSize” array “predMip[x][y] (for x from 0 to “predSize−1; for y from 0 to “predSize−1”) is transposed as “predTemp[y][x]=predMip[x][y]” and then “predMip=predTemp.”

More particularly, when the size of the coding block303is “8×8” (i.e. both “cbWidth” and “cbHeight” are 8), the intra prediction unit can derive an “8×8” “predMip” array.

In Step303inFIG. 3, the intra prediction unit derives predicted samples37of the remaining samples other than the partial samples35in the coding block300. As shown inFIG. 3, the intra prediction unit can use the filtering module302to derive the predicted samples37of the remaining samples other than the partial samples35in the coding block300. An input to the filtering module302can be the partial samples35in step302. The filtering module302can use one or more interpolation filters to derive the predicted samples37of the remaining samples other than the partial samples35in the coding block300. The intra prediction unit (or the filtering module302) can generate a prediction (which includes multiple predicted samples37) of the coding block300and store prediction37in an array “predSamples[x][y]” (for x from 0 to “nTbW−1,” for y from 0 to “nTbH−1”) according to the following conditions K and L.

[CONDITION K] If the intra prediction unit determines that “nTbW” is greater than “predSize” or that “nTbH” is greater than “predSize,” the intra prediction unit initiates an upsampling process to derive “predSamples” based on “predMip.”

[CONDITION L] Otherwise, the intra prediction unit sets the prediction of the coding block300as the MIP prediction of the coding block.

In other words, the intra prediction unit can set “predSamples[x][y] (for x from 0 to “nTbW−1”, for y from 0 to “nTbH−1”) being equal to “predMip[x][y].” For example, the intra prediction unit can set “predSamples” for a coding block with its size equal to “8×8” (i.e. both “cbWidth” and “cbHeight” are 8) as its “predMip[x][y].”

Through the Steps301-303, the intra prediction unit can generate the prediction of the current block300. The generated prediction can be used for further processed (e.g., the prediction block23discussed above with reference toFIG. 2).

FIG. 4is a schematic diagram of a decoding system400according to an embodiment of the present disclosure. The decoding system400is configured to receive, process, and transform an input bitstream40to an output video41. The input bitstream40can be a bitstream representing a compressed/coded picture/video. In some embodiments, the input bitstream40can be from an output bitstream (e.g., the output bitstream21) generated by an encoding system (such as the encoding system200).

The decoding system400includes a parsing unit401configured to parse the input bitstream40to obtain values of syntax elements therefrom. The parsing unit401also converts binary representations of the syntax elements to numerical values (i.e. a decoding block42) and forwards the numerical values to a prediction unit402(e.g., for decoding). In some embodiments, the parsing unit401can also forward one or more variables and/or parameters for decoding the numerical values to the prediction unit402.

The prediction unit402is configured to determine a prediction block43of the decoding block42(e.g., a CU or a partition of a CU, such as a transform block). When it is indicated that an inter coding mode was used to decode the decoding block42, an MC (motion compensation) unit403of the prediction unit402can receive relevant parameters from the parsing unit401and accordingly decode under the inter coding mode. When it is indicated that an intra prediction mode (e.g., an MW mode) is used to decode the decoding block42, an intra prediction unit404of the prediction unit402receives relevant parameters from the parsing unit401and accordingly decodes under the indicated intra coding mode. In some embodiments, the intra prediction mode (e.g., the MIP mode) can be identified by a specific flag (e.g., an MIP flag) embedded in the input bitstream40.

For example, when the MIP mode is identified, the intra prediction unit404can determine the prediction block43(which includes multiple predicted samples) based on the following methods (similar to the Steps301-303described inFIG. 3).

First, the intra prediction unit404derives one or more reference samples from neighboring samples of the decoding block42(similar to Step301inFIG. 3). For example, the intra prediction unit404can generate the reference samples by downsampling the neighboring samples, or directly extracting a portion from the neighboring samples.

The intra prediction unit404can then derive partial predicted samples in the decoding block42using the reference samples, an MIP matrix and a shifting parameter (similar to Step302inFIG. 3). In some embodiments, the positions of the partial predicted samples can be preset in the decoding clock42. For example, the positions of the partial predicted samples can be positions with odd horizontal and vertical coordinate values within the coding block. The shifting parameter can include a shifting offset parameter and a shifting number parameter, which can be used in shifting operations in generating the partial predicted samples.

Finally, if the partial predicted samples of the decoding block42are derived, the intra prediction unit404derives predicted samples of the remaining samples other than the partial predicted samples in the decoding block42(similar to Step303inFIG. 3). For example, the intra prediction unit404can use an interpolation filter to derive the predicted samples, by using the partial predicted samples and the neighboring samples as inputs of the interpolation filter.

The decoding system400includes a scaling unit405with functions similar to those of the inverse quantization unit210of the encoding system200. The scaling unit405performs scaling operations on quantized coefficients44(e.g., levels) from the parsing unit401so as to generate reconstructed coefficients45.

A transform unit406has functions similar to those of the inverse transform unit211in the encoding system200. The transform unit406performs one or more transform operations (e.g., inverse operations of one or more transform operations by the inverse transform unit211) to get reconstructed residual46.

An adder407adds the prediction block43from the prediction unit402and the reconstructed residual46from the transform unit406to get a reconstructed block47of the decoding block42. The reconstructed block47is also sent to the prediction unit402to be used as a reference (e.g., for other blocks coded in an intra prediction mode).

After all the decoding block42in a picture or a subpicture have been reconstructed (i.e., a reconstructed block48is formed), a filtering unit408can perform an in-loop filtering on the reconstructed block49. The filtering unit408contains one or more filters such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a luma mapping with chroma scaling (LMCS) filter, a neural-network-based filter, etc. In some embodiments, the filtering unit408can perform the in-loop filtering on only one or more target pixels in the reconstructed block48.

The filtering unit408then send a decoded picture49(or picture) or subpicture to a DPB (decoded picture buffer)409. The DPB409outputs decoded pictures as the output video41based on timing and controlling information. Decoded pictures49stored in the DPB409can also be employed as a reference picture by the prediction unit402when performing an inter or intra prediction.

In some embodiment, the decoding system400can be a computing device with a processor and a storage medium recording one or more decoding programs. When the processor reads and executes the decoding programs, the decoding system400can receive an input video bitstream and generate corresponding decoded video.

In some embodiments, the decoding system400can be a computing device with one or more chips. The units or elements of the decoding system400can be implemented as integrated circuits on the chips.

FIG. 5is a flowchart illustrating a method500in accordance with an embodiment of the present disclosure. At block501, the method500starts by determining prediction parameters of a current block. In some embodiments, the prediction parameters include parameters for prediction (e.g., “predModeIntra” as defined in the H.265 standard) and size identifiers (e.g., “mipSizeId”). At block502, when the prediction parameters indicate that an MIP mode is applicable, the method500continues to determine an MIP input sample (e.g., values in array “p[x]”) based on neighboring samples. In some embodiments, the neighboring samples can include above-neighboring samples and/or left-neighboring samples.

At block503, the method500continues to determine the product of a shifting offset parameter (fO) and the sum of the MIP input samplings. As shown in equation (B) above, the sum of the MIP input samplings can be “Σi=0inSize1p[i],” and therefore the product can be “fO*(Σi=0inSize−1p[i].”

At block503, the method500continues to determine a first constant based on a shifting number parameter (sW). As shown in equation (B) above, the first constant can be “1<<(sW−1).”

At block504, the method500continues to determine an offset value (oW) by multiplying the first constant and the product. As shown in equation (B) above, the first offset value (oW) can be “(1<<(sW−1))−fO*(Σi=0inSize−1p[i]).”

At block505, the method500determines an MIP weighting matrix based on the prediction parameters. In equation (B) above, the MW weighting matrix is matrix “mWeight.” In some embodiments, the MIP weighting matrix can be generated based on various methods such as training processes involving machine learning (ML) or artificial intelligence (AI).

At block507, the method500continues to determine MIP prediction based on the MIP weighting matrix, the shifting number parameter (sW) and the offset value (oW). The MIP prediction can be matrix “predMip[x][y]” as indicated in equation (C) above. In some embodiments, the MIP prediction can be determined as described in equation (C) above. In some embodiments, the MIP prediction matrix can be determined based on equation (D) below.

In equation (D) above, “[x][y]” are location coordinates of pixels. “x” is for the horizontal direction and “y” is for the vertical direction of the MIP prediction matrix. Parameters “incH,” “predC,” and “incW” are parameters for obtaining matrix values from the MIP weighting matrix. Other parameters in equation (D) have been discussed above with reference to equation (C).

In some embodiments, the shifting offset parameter (fO) can be set as “32” and the shifting number parameter (sW) can be set as 6. In such embodiments, the MIP prediction matrix can be determined based on equations (E) and (F) below.

At block508, the method500generates intra prediction of the current block by a filtering process based on the MIP prediction matrix. If the size of the MIP prediction matrix is the same as the size of the current block, then the method500can set the values in the MIP prediction matrix to the current block as its MIP intra prediction samples (e.g., “preSamples[x][y]”), as shown in Equation (G) below. If not, the method500can perform the filtering process to adjust the size of the MIP prediction matrix. In some embodiments, the filtering process can be a upsampling process or a low-pass filtering process. Embodiments of the upsampling process are discussed in detail above with reference toFIG. 3(e.g., Step303).
predSamples[x][y]=predMip[x][y]  (G)

FIG. 6is a flowchart illustrating a method600in accordance with an embodiment of the present disclosure. At block601, the method600starts by determining prediction parameters of a current block in a bitstream. In some embodiments, the prediction parameters can include “predModeIntra” and “mipSizeId” as defined in the H.265 standard.

At block602, the method600continues by determining a matrix-based intra prediction (MIP) input sample of the current block based on neighboring samples of the current block and the prediction parameters. In some embodiments, the MIP input sample can be the values in array “p[x].” In some embodiments, the neighboring samples include a left-neighboring sample and/or an above-neighboring sample.

Embodiments of determining the MIP input sample are discussed above with reference toFIG. 3(e.g., Step301). For example, the method600can include performing a downsampling process to the neighboring samples to generate a temporary reference array (pTemp[x]) based on the size identifier.

At block603, the method600continues by setting a shifting number parameter (sW) as a first fixed value. In some embodiments, first fixed value can be 5 or 6. At block604, the method600continues by setting a shifting offset parameter (fO) as a second fixed value. In some embodiments, the second fixed value can be 23, 34, or 46.

When the first and second fixed values are set, a first constant “1<<(sW−1)” can be determined. An offset value (oW) can also be calculated (e.g., equation (B)).

At block605, the method600continues by determining an MIP weighting matrix of the current block based on the prediction parameters. Embodiments of the MIP weighting matrix are discussed above with reference toFIG. 3(e.g., Step302).

At block606, the method600continues by determining an MIP prediction sample (e.g., values in array “predMip[x][y]”) of the current block based on the MIP weighting matrix, the MIP input sample, the shifting offset parameter (fO), and the shifting number parameter (sW). Embodiments of the MIP prediction sample are discussed above in detail with reference to equations (C), (D), (E), and (F).

At block607, the method600continues by performing an upsampling process to the MIP prediction sample of the current block so as to generate intra predicted samples (e.g., “predSamples[x][y]”) of the current block. In some embodiments, the MIP prediction sample of the current block can include prediction samples for at least a portion of sampling points of the current block. Embodiments of the MIP prediction samples and MIP weighting matrix are discussed above in detail with reference to equation (G) andFIG. 3(e.g., Step303).

FIG. 7is a schematic diagram of an encoder700according to an embodiment of the present disclosure. As shown, the encoder700includes a first determination unit701, a first computing unit702, and a first prediction unit703. The first determination unit701is to configure prediction parameters of a current block and to determine an MIP input sample (e.g., values in array “p[x]”). In some embodiments, the MIP input sample can be determined based on neighboring samples. The prediction parameters include parameter indicative of which prediction model has been used (e.g., an MIP model) and corresponding parameters (e.g., a size identifier). The first determination unit701can also determine an MIP weighing matrix. The first computing unit702is configured to compute an offset value (e.g., “oW” discussed above) based on a shifting number parameter (e.g., “sW”) and a shifting offset parameter (e.g., “fO”). The first prediction unit703is to generate MIP prediction of the current block based on the MW weighing matrix, the offset value, the shifting number parameter, and the shifting offset parameter.

In some embodiments, the encoder700includes a first inquiry unit704is configured to determine an MIP model of a current block. In such embodiments, the first determination unit701determines an MIP model index of the current block. The first inquiry unit704can then obtain corresponding parameters (e.g., sW, fO, MIP size identifier, etc.) based on the MIP model index.

In the present disclosure, the term “unit” can be a processor, circuitry, software, module, or a combination thereof. In some embodiments, the “unit” can be an integrated component such as a SoC (system on chip). In some embodiments, the “unit” can include a set of instructions stored in a storage media such as a disk, a hard drive, a memory, and so on.

FIG. 8is a schematic diagram of an encoder800according to an embodiment of the present disclosure. The encoder800can include a first communication interface801, a first storage device802, and a first processor803coupled by a first system bus804. The first system bus804can include power lines, control lines, and/or signal lines. The first communication interface801is configured to communicate with other external devices by transmitting and receiving signals. The first storage device802is configured to store data, information, and/or instructions (such of the steps discussed inFIGS. 5 and 6) that can be performed by the first processor803.

The first processor803can be a chip, an integrated circuit, or other devices that can process signals. The first processor803can be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), programmable logic device (PLD), or other suitable logic components.

FIG. 9is a schematic diagram of a decoder900according to an embodiment of the present disclosure. As shown, the decoder900includes an analytic unit901, a second computing unit902, a second prediction unit903, a second prediction unit904, and a second inquiry unit905. The analytic unit901is configured to parse a bitstream so as to obtain prediction parameters of a current block. The second determination unit902is to configure suitable prediction parameters of the current block and to determine an MIP input sample (e.g., values in array “p[x]”). In some embodiments, the MIP input sample can be determined based on neighboring samples. The prediction parameters include parameter indicative of which prediction model has been used (e.g., an MIP model) and corresponding parameters (e.g., a size identifier). The second determination unit903can also determine an MIP weighing matrix. The second computing unit904is configured to compute an offset value (e.g., “oW”) based on a shifting number parameter (e.g., “sW”) and a shifting offset parameter (e.g., “fO”). The second prediction unit904is to generate MIP prediction of the current block based on the MIP weighing matrix, the offset value, the shifting number parameter, and the shifting offset parameter.

In some embodiments, the second inquiry unit905is configured to determine an MIP model of a current block. In such embodiments, the second determination unit902determines an MIP model index of the current block. The second inquiry unit904can then obtain corresponding parameters (e.g., sW, fO, MIP size identifier, etc.) based on the MIP model index.

FIG. 10is a schematic diagram of a decoder1000according to an embodiment of the present disclosure. The decoder1000can include a second communication interface1001, a second storage device1002, and a second processor1003coupled by a second system bus1004. The second system bus1004can include power lines, control lines, and/or signal lines. The second communication interface1001is configured to communicate with other external devices by transmitting and receiving signals. The second storage device1002is configured to store data, information, and/or instructions (such of the steps discussed inFIGS. 5 and 6) that can be performed by the second processor1003.

The second processor1003can be a chip, an integrated circuit, or other devices that can process signals. The second processor1003can be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), programmable logic device (PLD), or other suitable logic components.

FIG. 11is a schematic diagram of an apparatus1100according to an embodiment of the present disclosure. The apparatus1100can be a “sending” apparatus. More particularly, the apparatus1100is configured to acquire, encode, and store/send one or more pictures. The apparatus1100includes an acquisition unit1001, an encoder1102, and a storage/sending unit1103.

The acquisition unit1101is configured to acquire or receive a picture and forward the picture to the encoder1102. The acquisition unit1101can also be configured to acquire or receive a video consisting of a sequence of pictures and forward the video to the encoder1102. In some embodiments, the acquisition unit1101can be a device containing one or more cameras (e.g., picture cameras, depth cameras, etc.). In some embodiments, the acquisition unit1101can be a device that can partially or completely decode a video bitstream to generate a picture or a video. The acquisition unit1101can also contain one or more elements to capture audio signal.

The encoder1102is configured to encode the picture from the acquisition unit1101and generates a video bitstream. The encoder1102can also be configured to encode the video from the acquisition unit1101and generates the bitstream. In some embodiment, the encoder1102can be implemented as the encoding system200described inFIG. 2. In some embodiments, the encoder1102can contain one or more audio encoders to encode audio signals to generate an audio bitstream.

The storage/sending unit1103is configured to receive one or both of the video and audio bitstreams from the encoder1102. The storage/sending unit1103can encapsulate the video bitstream together with the audio bitstream to form a media file (e.g., an ISO-based media file) or a transport stream. In some embodiments, the storage/sending unit1103can write or store the media file or the transport stream in a storage unit, such as a hard drive, a disk, a DVD, a cloud storage, a portable memory device, etc. In some embodiments, the storage/sending unit1103can send the video/audio bitstreams to an external device via a transport network, such as the Internet, a wired network, a cellular network, a wireless local area network, etc.

FIG. 12is a schematic diagram of an apparatus1200according to an embodiment of the present disclosure. The apparatus1200can be a “destination” apparatus. More particularly, the apparatus1200is configured to receive, decode, and render picture or video. The apparatus1200includes a receiving unit1201, a decoder1202, and a rendering unit1203.

The receiving unit1201is configured to receive a media file or a transport stream, e.g., from a network or a storage device. The media file or the transport stream includes a video bitstream and/or an audio bitstream. The receiving unit1201can separate the video bitstream and the audio bitstream. In some embodiments, the receiving unit1201can generate a new video/audio bitstream by extracting the video/audio bitstream.

The decoder1202includes one or more video decoders such as the decoding system400discussed above. The decoder1202can also contain one or more audio decoders. The decoder1202decodes the video bitstream and/or the audio bitstream from the receiving unit1201to get a decoded video file and/or one or more decoded audio files (corresponding to one or multiple channels).

The rendering unit1203receives the decoded video/audio files and processes the video/audio files to get suitable video/audio signal for displaying/playing. These adjusting/reconstructing operations can include one or more of the following: denoising, synthesis, conversion of color space, upsampling, downsampling, etc. The rendering unit1203can improve qualities of the decoded video/audio files.

FIG. 13is a schematic diagram of a communication system1300according to an embodiment of the present disclosure. The communication system1300includes a source device1301, a storage medium or transport network1302, and a destination device1303. In some embodiments, the source device1301can be the apparatus1100described above with reference toFIG. 11. The source device1301sends media files to the storage medium or transport network1302for storing or transporting the same. The destination device1303can be the apparatus1200described above with reference toFIG. 12. The communication system1300is configured to encode a media file, transport or store the encoded media file, and then decode the encoded media file. In some embodiments, the source device1301can be a first smartphone, the storage medium1302can be a cloud storage, and the destination device can be a second smartphone.

The above-described embodiments are merely illustrative of several embodiments of the present disclosure, and the description thereof is specific and detailed. The above embodiments cannot be construed to limit the present disclosure. It should be noted that, a number of variations and modifications may be made by those skilled in the art without departing from the spirit and scope of the disclosure. Therefore, the scope of the present disclosure should be subject to the appended claims.