MULTIPLE MODES AND MULTIPLE TEMPLATES FOR TEMPLATE MATCHING RELATED VIDEO CODING TOOLS

A method of coding video data comprises: determining a template pattern from among a set of two or more template patterns; identifying a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data; obtaining, based on the reference template, a prediction block for a current block of a current picture of the video data; and encoding or decoding the current block based on the prediction block.

TECHNICAL FIELD

BACKGROUND

SUMMARY

In general, this disclosure describes techniques for template matching in video coding. As described herein, a video coder (e.g., a video encoder or a video decoder) may determine a template pattern from among a set of two or more template patterns. The video coder may identify a reference template based on a similarity of the reference template and a current template. The reference template and the current template have a shape defined by the template pattern. The reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data. The video coder may obtain, based on the reference template, a prediction block for a current block of a current picture of the video data. The video coder may encode or decode the current block based on the prediction block.

As mentioned above, the techniques of this disclosure provide for multiple template patterns. The determination of a template pattern from among multiple template patterns may allow a video coder to identify more candidate prediction blocks, such as candidate prediction blocks resulting from applying fusion to multiple candidate prediction blocks. Identifying more candidate prediction blocks may result in better coding efficiency because the video coder may be more likely to identify a candidate prediction block resembling the current block if there are more candidate prediction blocks. When the candidate prediction block closely resembles the current block, the number of bits required to store residual data representing differences between the candidate prediction block and the current block may be less. Hence, encoded video data representing the video data may include fewer bits overall.

In one example, a method of coding video data comprises: determining a template pattern from among a set of two or more template patterns; identifying a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data; obtaining, based on the reference template, a prediction block for a current block of a current picture of the video data; and encoding or decoding the current block based on the prediction block.

In another example, this disclosure describes a device for coding video data, the device comprising: a memory configured to store the video data; and one or more processors implemented in circuitry, the one or more processors configured to: determine a template pattern from among a set of two or more template patterns; identify a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data; obtain, based on the reference template, a prediction block for the current block; and encode or decode the current block based on the prediction block.

In another example, this disclosure describes a computer-readable storage medium is encoded with instructions that, when executed, cause one or more processors to: determine a template pattern from among a set of two or more template patterns; identify a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of video data; obtain, based on the reference template, a prediction block for the current block; and encode or decode the current block based on the prediction block.

DETAILED DESCRIPTION

In general, the use of template-matching coding tools involves identifying a “template” that includes a set of samples above a current block and left of the current block. A video coder (e.g., a video encoder or a video decoder) may then perform a search for a set of samples in a search region that best matches the samples in the template. In the case of an intra template matching coding tool, the search region is within the same picture as the current block. In the case of an inter template matching coding tool, the search region is within a reference picture different from the picture containing the current block. A prediction block may be a block within the search region that is below and right of the identified set of samples. The video coder may determine a motion vector (e.g., a refined motion vector) of the current block as a spatial displacement between the current block and the prediction block. Residual data may represent differences between the current block and the prediction block. The use of such template-matching coding tools may obviate the need for a video encoder to signal the motion vector to a video decoder because the video decoder may independently determine the motion vector in the same way as the video encoder. Thus, the use of template-matching coding tools may reduce the size of encoded video data.

Conventional template-matching coding tools only use one template pattern. In other words, the template only has one shape. Specifically, conventional template-matching coding tools only use a template pattern that includes samples above and left of the current block. However, in accordance with one or more techniques of this disclosure, multiple different template patterns may be used. The use of multiple template patterns may allow video encoders and video decoders to generate additional candidates from which the video encoder or video decoder may select when generating a prediction block. The presence of the additional candidates may further improve video compression performance.

System100as shown inFIG.1is merely one example. In general, any digital video encoding and/or decoding device may perform the template-matching techniques of this disclosure. Source device102and destination device116are merely examples of such coding devices in which source device102generates encoded video data for transmission to destination device116. This disclosure refers to a “coding” device as a device that performs coding (encoding and/or decoding) of data. Thus, video encoder200and video decoder300represent examples of coding devices, in particular, a video encoder and a video decoder, respectively. In some examples, source device102and destination device116may operate in a substantially symmetrical manner such that each of source device102and destination device116includes video encoding and decoding components. Hence, system100may support one-way or two-way video transmission between source device102and destination device116, e.g., for video streaming, video playback, video broadcasting, or video telephony.

In some examples, source device102may output encoded video data to file server114or another intermediate storage device that may store the encoded video data generated by source device102. Destination device116may access stored encoded video data from file server114via streaming or download.

File server114may be any type of server device capable of storing encoded video data and transmitting that encoded video data to the destination device116. File server114may represent a web server (e.g., for a website), a server configured to provide a file transfer protocol service (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a hypertext transfer protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (cMBMS) server, and/or a network attached storage (NAS) device. File server114may, additionally or alternatively, implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, or the like.

Destination device116may access encoded video data from file server114through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on file server114. Input interface122may be configured to operate according to any one or more of the various protocols discussed above for retrieving or receiving media data from file server114, or other such protocols for retrieving media data.

Although not shown inFIG.1, in some examples, video encoder200and video decoder300may each be integrated with an audio encoder and/or audio decoder, and may include appropriate MUX-DEMUX units, or other hardware and/or software, to handle multiplexed streams including both audio and video in a common data stream.

Video encoder200and video decoder300may operate according to a video coding standard, such as ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC) or extensions thereto, such as the multi-view and/or scalable video coding extensions. Alternatively, video encoder200and video decoder300may operate according to other proprietary or industry standards, such as ITU-T H.266, also referred to as Versatile Video Coding (VVC). In other examples, video encoder200and video decoder300may operate according to a proprietary video codec/format, such as AOMedia Video 1 (AV1), extensions of AV1, and/or successor versions of AV1 (e.g., AV2). In other examples, video encoder200and video decoder300may operate according to other proprietary formats or industry standards. The techniques of this disclosure, however, are not limited to any particular coding standard or format. In general, video encoder200and video decoder300may be configured to perform the techniques of this disclosure in conjunction with any video coding techniques that use template matching coding tools.

When operating according to the AV1 codec, video encoder200and video decoder300may be configured to code video data in blocks. In AV1, the largest coding block that can be processed is called a superblock. In AV1, a superblock can be either 128×128 luma samples or 64×64 luma samples. However, in successor video coding formats (e.g., AV2), a superblock may be defined by different (e.g., larger) luma sample sizes. In some examples, a superblock is the top level of a block quadtree. Video encoder200may further partition a superblock into smaller coding blocks. Video encoder200may partition a superblock and other coding blocks into smaller blocks using square or non-square partitioning. Non-square blocks may include N/2×N, N×N/2, N/4×N, and N×N/4 blocks. Video encoder200and video decoder300may perform separate prediction and transform processes on each of the coding blocks.

AV1 also defines a tile of video data. A tile is a rectangular array of superblocks that may be coded independently of other tiles. That is, video encoder200and video decoder300may encode and decode, respectively, coding blocks within a tile without using video data from other tiles. However, video encoder200and video decoder300may perform filtering across tile boundaries. Tiles may be uniform or non-uniform in size. Tile-based coding may enable parallel processing and/or multi-threading for encoder and decoder implementations.

In some examples, a CTU includes a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples of a picture that has three sample arrays, or a CTB of samples of a monochrome picture or a picture that is coded using three separate color planes and syntax structures used to code the samples. A CTB may be an N×N block of samples for some value of N such that the division of a component into CTBs is a partitioning. A component is an array or single sample from one of the three arrays (luma and two chroma) that compose a picture in 4:2:0, 4:2:2, or 4:4:4 color format or the array or a single sample of the array that compose a picture in monochrome format. In some examples, a coding block is an M×N block of samples for some values of M and N such that a division of a CTB into coding blocks is a partitioning.

In some examples, a tile may be partitioned into multiple bricks, each of which may include one or more CTU rows within the tile. A tile that is not partitioned into multiple bricks may also be referred to as a brick. However, a brick that is a true subset of a tile may not be referred to as a tile. The bricks in a picture may also be arranged in a slice. A slice may be an integer number of bricks of a picture that may be exclusively contained in a single network abstraction layer (NAL) unit. In some examples, a slice includes either a number of complete tiles or only a consecutive sequence of complete bricks of one tile.

AV1 includes two general techniques for encoding and decoding a coding block of video data. The two general techniques are intra prediction (e.g., intra frame prediction or spatial prediction) and inter prediction (e.g., inter frame prediction or temporal prediction). In the context of AV1, when predicting blocks of a current frame of video data using an intra prediction mode, video encoder200and video decoder300do not use video data from other frames of video data. For most intra prediction modes, video encoder200encodes blocks of a current frame based on the difference between sample values in the current block and predicted values generated from reference samples in the same frame. Video encoder200determines predicted values generated from the reference samples based on the intra prediction mode.

Intra template matching is now described.FIG.2is a block diagram illustrating an example intra template matching search area. Intra template matching prediction (Intra TMP) is a special intra prediction mode that copies a best prediction block250(e.g., a matching block from the reconstructed part of a current frame, whose L-shaped template (e.g., reference template252) matches a current template254. Current template254is a L-shaped set of samples above and left of a current block256. Current block256may be a PU, CU, or other type of block. For a predefined search range, video encoder200searches for the most similar template to the current template254in a reconstructed part of the current frame and uses the corresponding block as prediction block250. Video encoder200may then signal the usage of the intra template matching mode, and the same prediction operation is performed at video decoder300.

A prediction signal is generated by matching an L-shaped causal neighbor of the current block (e.g., current template254) with another block (e.g., prediction block250) in a predefined search area. In the example ofFIG.2, the predefined search areas may consist of:R1: current CTUR2: top-left CTUR3: above CTUR4: left CTU

Video encoder200and video decoder300may use a sum of absolute differences (SAD) as a cost function. In other words, as video encoder200and video decoder300search the search areas, video encoder200and video decoder300may position reference template252at different locations within the search areas. For each location, video encoder200and video decoder300may apply a cost function (e.g., SAD) to compare the samples in the reference template to the samples within current template254. Video encoder200and video decoder300may select the location based on costs produced by the cost function for the different locations. Thus, within each region, video encoder200and video decoder300may search for the template that has least SAD with respect to the current one and uses its corresponding block as a prediction block.

The dimensions of all regions (SearchRange_w, SearchRange_h) may be set proportional to the block dimension (BlkW, BlkH) to have a fixed number of SAD comparisons per pixel. That is:

Where ‘a’ is a constant that controls the gain/complexity trade-off. In practice, ‘a’ is equal to 5.

In some examples, the intra template matching tool is enabled for CUs with size less than or equal to 64 in width and height. This maximum CU size for intra template matching may be configurable. The intra template matching prediction mode may be signaled at a CU level through a dedicated flag when decoder-side intra mode derivation (DIMD) is not used for the current CU.

Inter template matching is now discussed.FIG.3is a block diagram illustrating an example of template matching performed on a search area around an initial motion vector350. Inter template matching (InterTM) is a decoder-side MV derivation method to refine the motion information of a current CU352by finding the closest match between a template354(i.e., top and/or left neighbouring blocks of current CU352) in a current picture356and a block (i.e., a reference template360having same size as current template354) in a reference picture358. As illustrated inFIG.3, a better MV is searched around the initial motion of current CU352within a [−8, +8]-pel search range. The template matching method in Chen et al., “Description of SDR, HDR and 360° video coding technology proposal by Qualcomm and Technicolor—low and high complexity versions”, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 10thMeeting: San Diego, US, 10-20 Apr. 2018, document JVET-J0021 (hereinafter, “JVET-J0021”) is used with the following modifications: search step size is determined based on AMVR mode and InterTM can be cascaded with bilateral matching process in merge modes.

In AMVP mode, an MVP candidate is determined based on template matching error to select the one which reaches the minimum difference between current template354and a reference block template360, and then InterTM is performed only for this particular MVP candidate for MV refinement. InterTM refines this MVP candidate, starting from full-pel MVD precision (or 4-pel for 4-pel AMVR mode) within a [−8, +8]-pel search range by using an iterative diamond search. The AMVP candidate may be further refined by using cross search with full-pel MVD precision (or 4-pel for 4-pel AMVR mode), followed sequentially by half-pel and quarter-pel ones depending on AMVR mode as specified in Table 1, below. This search process may ensure that the MVP candidate keeps the same MV precision as indicated by the AMVR mode after TM process. In the search process, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than a threshold that is equal to the area of the block, the search process terminates.

In merge mode, video encoder200and video decoder300may apply a similar search method to the merge candidate indicated by the merge index. As Table 1 shows, InterTM may perform all the way down to ⅛-pel MVD precision or skipping those beyond half-pel MVD precision, depending on whether the alternative interpolation filter (that is used when AMVR is of half-pel mode) is used according to merged motion information. When TM mode is enabled, template matching may work as an independent process or an extra MV refinement process between block-based and subblock-based bilateral matching (BM) methods, depending on whether BM can be enabled or not according to its enabling condition check.

Adaptive reordering of merge candidates with template matching (ARMC-TM) is now discussed. Video encoder200and video decoder300may adaptively reorder the merge candidates with template matching TM. Video encoder200and video encoder300may apply the reordering method to regular merge mode, TM merge mode, and affine merge mode (excluding the subblock temporal motion vector prediction (SbTMVP) candidate). For the TM merge mode, video encoder200and video decoder300may reorder merge candidates before the refinement process.

Video encoder200and video decoder300may firstly construct an initial merge candidate list according to given checking order, such as spatial, temporal motion vector predictors (TMVPs), non-adjacent, history-based motion vector predictors (HMVPs), pairwise, virtual merge candidates. Video encoder200and video decoder300may then divide the candidates in the initial list into several subgroups. For the template matching TM merge mode, adaptive decoder-side motion vector refinement (DMVR) mode, video encoder200and video decoder300may each firstly refine merge candidates in the initial list using TM/multi-pass DMVR. Video encoder200and video decoder300may reorder merge candidates in each subgroup to generate a reordered merge candidate list and the reordering is according to costs based on template matching. The index of selected merge candidate in the reordered merge candidate list is signaled to video decoder300. For simplification, merge candidates in the last but not the first subgroup are not reordered. All the zero candidates from the ARMC reordering process are excluded during the construction of Merge motion vector candidates list. The subgroup size is set to 5 for regular merge mode and TM merge mode. The subgroup size is set to 3 for affine merge mode.

Video encoder200and video decoder300may measure the template matching cost of a merge candidate during the reordering process by the SAD between samples of a template of the current block and their corresponding reference samples. The template comprises a set of reconstructed samples neighboring to the current block. Reference samples of the template are located by the motion information of the merge candidate. When a merge candidate utilizes bi-directional prediction, the reference samples of the template of the merge candidate are also generated by bi-prediction as shown inFIG.4.

FIG.4is a conceptual diagram illustrating example template and reference samples of the template in reference pictures. As shown in the example ofFIG.4, a video coder (e.g., video encoder200or video decoder300) is coding (e.g., encoding or decoding) a current block400of a current picture402. Current block400is associated with a merge candidate list that includes one or more bi-prediction merge candidates. Each of the bi-prediction merge candidates includes a List 0 motion vector and a List 1 motion vector. The List 0 motion vector indicates a location within a reference picture in a first reference picture list (i.e., list 0). The List 1 motion vector indicates a location within a reference picture in a second reference picture list (i.e., list 1). In the example ofFIG.4, the List 0 motion vector of a selected merge candidate indicates a location in a reference picture404. The List 1 motion vector of a selected merge candidate indicates a location in a reference picture406. The video coder may then use a template408of current block400to search areas near indicated locations in reference picture404and reference picture406for templates410(i.e., reference samples of a template in reference list 0 and reference list 1). In this way, the video coder may be able to identify a reference block412and a reference block414that are potentially more similar to current block400than reference blocks at locations indicated by the List 0 motion vector and the List 1 motion vector.

Refinement of an initial merge candidate list is now discussed. When multi-pass DMVR is used to derive the refined motion to the initial merge candidate list only the first pass (i.e., PU level) of multi-pass DMVR is applied in reordering. When template matching is used to derive the refined motion, the template size is set equal to 1. Only the above template or left template is used during the motion refinement of TM when the block is flat with block width greater than 2 times of height or narrow with height greater than 2 times of width. TM is extended to perform 1/16-pel MVD precision. The first four merge candidates are reordered with the refined motion in TM merge mode.

FIG.5is a conceptual diagram illustrating example template and reference samples of the template for block with sub-block motion using the motion information of the subblocks of the current block. For subblock-based merge candidates with subblock size equal to Wsub×Hsub, the above template comprises several sub-templates502with the size of Wsub×1, and the left template comprises several sub-templates504with the size of 1×Hsub. As shown inFIG.5, the motion information of the subblocks in the first row and the first column (sub-blocks A-G) of current block506is used to derive the reference samples of each sub-template.

In the reordering process, video encoder200and video decoder300consider a candidate as redundant if the cost difference between a candidate and its predecessor is inferior to a lambda (λ) value e.g., |D1−D2|<λ, where D1and D2are the costs obtained during the first ARMC ordering and λ is the Lagrangian parameter used in the RD criterion at encoder side.

An algorithm for the reordering process may be defined as follows:Determine the minimum cost difference between a candidate and its predecessor among all candidates in the list.If the minimum cost difference is superior or equal to λ, the list is considered diverse enough and the reordering stops.If this minimum cost difference is inferior to λ, the candidate is considered as redundant and the candidate is moved at a further position in the list. This further position is the first position where the candidate is diverse enough compared to the candidate's predecessor.The algorithm stops after a finite number of iterations (if the minimum cost difference is not inferior to λ)

This algorithm may be applied to the Regular, TM, BM and Affine merge modes. A similar algorithm may be applied to the Merge MMVD and sign MVD prediction methods which also use ARMC for the reordering.

The value of λ may be set equal to the λ of the rate distortion criterion used to select the best merge candidate at the encoder side for low delay configuration and to the value λ corresponding to a another QP for Random Access configuration. A set of λ values corresponding to each signaled QP offset is provided in the SPS or in the Slice Header for the QP offsets which are not present in the SPS.

The ARMC design is also applicable to the AMVP mode wherein video encoder200and video decoder300reorder the AMVP candidates according to the TM cost. For the template matching for advanced motion vector prediction (TM-AMVP) mode, an initial AMVP candidate list is constructed, followed by a refinement from TM to construct a refined AMVP candidate list. In addition, an MVP candidate with a TM cost larger than a threshold, which is equal to five times of the cost of the first MVP candidate, is skipped. When wrap-around motion compensation is enabled, the MV candidate shall be clipped with wrap around offset taken into consideration.

Geometric partition mode (GPM) with template matching TM is now described. Template matching is applied to GPM. When GPM mode is enabled for a CU, a CU-level flag is signaled to indicate whether TM is applied to both geometric partitions. Motion information for each geometric partition is refined using TM. When TM is chosen, a template is constructed using left, above or left and above neighboring samples according to partition angle, as shown in Table 2. Video encoder200and video decoder300may then refine the motion by minimizing the difference between the current template and the template in the reference picture using the same search pattern of merge mode with half-pel interpolation filter disabled.

A GPM candidate list may be constructed as follows:1. Interleaved List-0 MV candidates and List-1 MV candidates are derived directly from the regular merge candidate list, where List-0 MV candidates are higher priority than List-1 MV candidates. A pruning method with an adaptive threshold based on the current CU size is applied to remove redundant MV candidates.2. Interleaved List-1 MV candidates and List-0 MV candidates are further derived directly from the regular merge candidate list, where List-1 MV candidates are higher priority than List-0 MV candidates. The same pruning method with the adaptive threshold is also applied to remove redundant MV candidates.3. Zero MV candidates are padded until the GPM candidate list is full.

The GPM-MMVD and GPM-TM are exclusively enabled to one GPM CU. This is done by firstly signaling the GPM-MMVD syntax. When both two GPM-MMVD control flags are equal to false (i.e., the GPM-MMVD are disabled for two GPM partitions), the GPM-TM flag is signaled to indicate whether the template matching is applied to the two GPM partitions. Otherwise (at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is inferred to be false.

Intra block copy (IBC) with template matching is now described. Template Matching is used in IBC for both IBC merge mode and IBC AMVP mode. The IBC-TM merge list is modified compared to the one used by regular IBC merge mode such that the candidates are selected according to a pruning method with a motion distance between the candidates as in the regular TM merge mode. The ending zero motion fulfillment is replaced by motion vectors to the left (−W, 0), top (0, −H) and top-left (−W, −H), where W is the width and H the height of the current CU.

In the IBC-TM merge mode, the selected candidates are refined with the Template Matching method prior to the rate-distortion optimization (RDO) or decoding process. The IBC-TM merge mode has been put in competition with the regular IBC merge mode and a TM-merge flag is signaled.

In the IBC-TM AMVP mode, up to 3 candidates are selected from the IBC-TM merge list. Each of those 3 selected candidates are refined using the Template Matching method and sorted according to their resulting Template Matching cost. Only the 2 first ones are then considered in the motion estimation process as usual.

FIGS.6A-6Dare conceptual diagram illustrating example intra block copy (IBC) reference regions depending on a current coding unit (CU) position. InFIGS.6A-6D, shaded blocks are “causal” with respect to a current block600A,600B,600C, or600D (collectively, “current blocks600”). In other words, the shaded blocks are encoded or decoded prior to current blocks600. Blocks marked with an “X” mark are reference regions that can be used with respect to current blocks600for IBC-TM. The Template Matching refinement for both IBC-TM merge and AMVP modes is quite simple since IBC motion vectors are constrained (i) to be integer and (ii) within a reference region as shown inFIGS.6A-6D. So, in IBC-TM merge mode, all refinements are performed at integer precision, and in IBC-TM AMVP mode, they are performed either at integer or 4-pel precision depending on the AMVR value. Such a refinement accesses only samples without interpolation. In both cases, the refined motion vectors and the used template in each refinement step must respect the constraint of the reference region.

To improve the coding efficiency of template matching, instead of using only one pattern and process of template in TM, template matching may use different template types, store more candidates, and apply fusion to combine these different candidates which are found by the different methods.

For description-wise simplicity, if not otherwise stated, the mentioned TM in this section can refer to the intra template matching, inter template matching, adaptive reordering of merge candidates with template matching (ARMC-TM) or IBC template matching. The disclosed methods can be used solely or in any combination.

Techniques for multiple TM modes by different template patterns are now described. In a first example technique, TM has a template pattern set comprised of multiple template matching patterns, and one or more syntax elements are signaled to indicate one template pattern used in the template matching process. When template matching is used in the current block and a template matching mode is signaled, video encoder200or video decoder300uses the corresponding template pattern of this mode in the template matching process. The template pattern is not constrained to use the adjacent neighboring samples of the current block.

FIGS.7A-7Gare conceptual diagrams illustrating examples of different template patterns accordance to techniques of this disclosure. InFIG.7A, a template700A includes samples left of and above a current CU702A. InFIG.7B, a template700B includes samples left of a current CU702B. InFIG.7C, a template700C includes samples above a current CU702C. InFIG.7D, a template700A includes samples left of an upper half of a current CU702D. InFIG.7E, a template700E includes samples above a left half of a current CU702E. InFIG.7F, a template700F includes samples left of a current CU702F that are not adjacent to current CU702F. InFIG.7G, a template700G includes samples above a current CU702G that are not adjacent to current CU702G.

In another example, as a simplified method of the aforementioned first example technique, the template pattern set is composed of three template matching patterns:(Pattern 1) use both above and left neighboring samples,(Pattern 2) only use above neighboring samples, and(Pattern 3) only use left neighboring samples.

The matching cost of template matching pattern 1 may be derived from the cost of template matching pattern 2 and the cost of template matching pattern 3. In other words, a video coder may calculate a first cost (i.e., the matching cost of a first reference template having template matching pattern 2) based on the differences between samples in a first current template above the current block and samples in a first reference template that has the same shape as the first current template; calculate a second cost (i.e., the matching cost of a second reference template having template matching pattern 3) based on the difference between samples in a second current template left of the current block and samples in a second reference template that has the same shape as the second current template and the same spatial relationship to the first reference template as the first current template has to the second current template; and calculate a third cost (i.e., the cost of a third reference template having template matching pattern 1) as a sum of the first cost and the second cost. In this example, the video coder may repeat this process to determine costs of the first. second, and third reference templates at different locations in a search area. The video coder may then identify a specific reference template based on the determined costs. The video coder may obtain a prediction block based on the identified reference template. Thus, in this example, steps of determining a template pattern from among a set of two or more template patterns and identifying a reference template may be performed together.

In another example, as a simplified method of the aforementioned first example technique, the TM has a base template pattern S, which may be denoted as mode 0. Additional template pattern modes 1 . . . n are all inside the region of S. In other words, the set of template patterns may include a base template pattern S and one or more additional template patterns that fit within a region of the base template pattern S. For example, if template700A inFIG.7Ais the base template pattern, templates700B,700C,700D, and700E could be other template patterns because they are all inside template700A.

In some examples, as a simplified method of the aforementioned first example technique, the template matching cost calculation process always computes the cost of the base template pattern, and sums up two or more of template patterns to determine the cost of the used template pattern. In other words, when identifying a template pattern from among a set of two or more template patterns, a video coder (e.g., video encoder200or video decoder300) may determine a cost for the base template pattern (e.g., using SAD) and may determine costs of the additional template patterns (e.g., using SAD) based on costs of subregions within the region of the base template pattern. The video coder may identify a reference template based on the cost for the base template pattern and the costs for the additional template patterns. For instance, the video coder may identify the reference template that has the lowest cost among reference templates that have the base template pattern and reference templates that have the additional template patterns.

In some examples, as a simplified method of the first example technique, the base template pattern could be template700A ofFIG.7A. In other words, the base template pattern may include reference samples above and left of the current block.

In some examples, as a simplified method of the first example technique, if the current block uses TM, there is another flag to indicate the template pattern of TM, where mode 0 denotes the base template pattern and modes 1 . . . n denote the additional template patterns. Thus, video encoder200may signaling one or more syntax elements that identify the template pattern. Video decoder300may obtain, from a bitstream that includes an encoded representation of the video data, one or more syntax elements that indicate the template pattern. Video decoder300may determine the template pattern based on the one or more syntax elements. In some examples, the syntax of the matching pattern is signaled in one or more of a CU level, a PU level, a CTU level, a slice level, or a picture level. In other words, the syntax elements that indicate the template pattern may be signaled at a coding unit level, a prediction unit level, a coding tree unit level, a slice level, or a picture level.

A technique involving multiple MV candidates derived from the same template pattern and fusion is now described. In a second example technique, the template matching process of a block has a list (i.e., a candidate list) that stores the best N MV candidates with the minimum template matching cost, the Nis larger than 1. For example, if N=2, the template matching process stores two MV candidates. The two stored MV candidates have the minimum and second minimum costs.

Thus, a video coder (e.g., video encoder200or video decoder300) may generate a list of motion vector candidates. As part of generating the list of motion vector candidates, the video coder may determine template matching costs for reference templates in a plurality of reference templates. Each reference template in the plurality of reference templates has a shape defined by the template pattern. Additionally, as part of generating the list of motion vector candidates, the video coder may select two or more of the reference templates based on the template matching costs for the reference templates. The video coder may include candidates indicating motion vectors of the selected reference templates in the list of motion vector candidates. The video coder may determine a selected motion vector in the list of motion vector candidates. The video coder may use the selected motion vector to determine a location of the prediction block for the current block.

In some examples, video encoder200and video decoder300may use the N-th minimum cost MV candidate. For example, video encoder200and video decoder300may use the smallest cost and the third smallest cost MV candidates, the smallest and the fourth smallest cost MV candidates, and so on.

In some examples, in the template matching process, if the distance of the current template matching MV (xcur, ycur) and the temporary best candidate MV (xbest, ybest) is smaller than a threshold, video encoder200and video decoder300do not insert the current template matching MV into the candidate list. In other words, a video coder may exclude a motion vector of a reference template from the list of motion vector candidates based on a comparison of a distance of the motion vector of the reference template and a temporary best candidate motion vector in the list of motion vector candidates.

In some examples, in the template matching process, if the difference of the current template matching block Pcurand the temporary best candidate block Pbestis smaller than a threshold, video encoder200and video decoder300do not insert the current template matching MV into the candidate list. In other words, a video coder may exclude a motion vector of a reference template from the list of motion vector candidates based on a comparison of a difference between a prediction block of a reference template and a prediction block of a temporary best candidate motion vector in the list of motion vector candidates.

If the difference between template matching costs of two MVs is less than a threshold value, video encoder200and video decoder300may remove or prune one MV from the candidate list. For example, video encoder200and video decoder300may remove or prune out the MV with the larger cost out of the two MVs from the candidate list. Thus, a video coder may remove, from the list of motion vector candidates, a candidate that indicates a motion vector of a first one of the selected reference templates based on a comparison of a predefined threshold and a difference of the cost of the first selected reference template and the cost of a second selected reference template.

In another example, video encoder200and video decoder300may generate a predictor (i.e., a predicted motion vector) from a combination of k MV candidates. For instance, a video coder (e.g., video encoder200or video decoder300) may determine a first prediction block based on the first reference template and determine a second prediction block based on a second reference template. The shape of the first reference template and a shape of the second reference template may be defined by a determined template pattern. The video coder may generate the prediction block for the current block based on the first prediction block and the second prediction block.

In one example, video encoder200and video decoder300may generate the predictor from a linear combination of k MV candidates. The 0 to k MV candidates could be selected from 0 to N MV candidates from template matching. The combination can be formulated as shown in the following equation:

where P0. . . Pkare the selected k MV candidates derived from the TM process. In some examples, video encoder200and video decoder300may derive the combining weight wkbased on the template matching cost. Thus, in some examples, a video coder may determine a weight for a first prediction block based on a template matching cost of a first reference template and determine a weight for the second prediction block based on a template matching cost of the second reference template. The video coder may generate samples of the prediction block for the current block as a combination of samples of the first prediction block and the second prediction block weighted according to the weight for the first prediction block and the weight for the second prediction block.

In one example, video encoder200and video decoder300may derive the combined weight wkas the multiplicative inverse of the template matching cost of this candidate Pk. Thus, a video coder may determine the weight for a first prediction block based on a multiplicative inverse of the template matching cost of a first reference template. The video coder may determine the weight for a second prediction block based on a multiplicative inverse of the template matching cost of a second reference template.

In some examples, video encoder200and video decoder300may derive the weights w0. . . wkfor MV candidates P0to Pkin equation (1) based on the SAD, MSE, or block vector (BV) of candidates P0to Pk, respectively. In other words, a video coder may determine the weight for a first prediction block and determine the weight for a second prediction block based on a SAD, an MSE, or a BV.

In another example, a syntax element indicates that whether some of the N MV candidates in the candidate list are selected and apply the linear combination of these MV candidates. In other words, a syntax element may indicate whether two or more reference templates are selected.

In some examples, if a distance of the current template matching MV (xcur, ycur) and other MV in the list, for example the first candidate MV (x0, y0), is smaller or larger than a threshold, the weight of the current template matching block could be set to 0. In other words, a video coder (e.g., video encoder200or video decoder300) may set the weight for the first prediction block to 0 based on a comparison of a predefined threshold and a distance of the motion vector of the first reference template and another motion vector in the list of motion vector candidates.

In another example, if the difference of the current template matching block Pour and the temporary best candidate block Pbestis smaller or larger than a threshold. The weight of the current template matching block could be set to 0. In other words, a video coder may set the weight for the first prediction block to 0 based on a comparison of a predetermined threshold and a difference between the first prediction block and the second prediction block.

In another example, as a simplified method of the aforementioned first example technique, video encoder200and video decoder300generate the MV predictor from the linear combination of the first two candidates. In other words, a video coder may determine a prediction block based on a linear combination of prediction blocks for two motion vector candidates in the list of motion vector candidates.

In another example, as a simplified method of the aforementioned first example technique, the predictor is generated from the linear combination of the first two candidates if the SAD or MSE of that first two candidates is smaller than a threshold; otherwise, the predictor is generated from the first candidate only. In other words, based on costs of two motion vector candidates in the list of motion vector candidates being less than a threshold, a video coder may determine the prediction block based on a linear combination of prediction blocks for the two motion vector candidates.

Techniques involving combinations of multiple template patterns, multiple candidates, and fusion are now described. In this context, fusion may refer to combining two or more candidates to form an additional candidate. In some examples, the MV candidates from multiple template patterns Pmp,0, Pmp,1, . . . Pmp,Mand the MV candidates from multiple MV candidates Pmc,0, Pmc,1, . . . Pmc,Ncould be arbitrarily selected out K MV candidates and a combination may be formed among these K candidates, where K may be larger than 1. In one example, the combination can be a linear combination. Thus, in some examples, a video coder (e.g., video encoder200or video decoder300) may generate a first prediction block using a determined reference template, generate a second prediction block using a motion vector candidate, and generate the prediction block for the current block based on the first prediction block and the second prediction block.

In another example, for some template matching modes, 2 of the candidates from multiple template patterns Pmp,0, Pmp,1, . . . Pmp,Mcould be linear combined. The combination can be formulated as follows:

In equation (2), a and b are the selected MV candidates, and waand wbare weights. Video encoder200and video decoder300may derive the weights waand wbfrom a template matching cost. In some examples, weights waand wbmay be equal (i.e., wa=wb=1).

Thus, a part of obtaining a prediction block for a current block, a video coder may generate a first prediction block using a first reference template having a first shape defined by a first template pattern. The video coder may identify a second reference template based on a similarity of the second reference template and a second current template. The second reference template and the second current template have a shape defined by a second template pattern different from the first template pattern. The second reference template includes second previously reconstructed samples and the second current template includes second reference samples of the current block. The video coder may generate a second prediction block using the second reference template. The video coder may generate the prediction block for the current block based on the first prediction block and the second prediction block. In this example, the video coder may determine a weight for the first prediction block and a weight for the second prediction block based on template matching costs of the first reference template and the second reference template. The video coder may generate samples of the prediction block for the current block as a combination of samples of the first prediction block and the second prediction block weighted according to the weight for the first prediction block and the weight for the second prediction block.

In some examples, a certain template pattern (or certain template patterns) may have a dedicated fusion method, and some other template patterns may not have fusion applied. For example, when left and above template patterns are used, video encoder200and video decoder300may then optionally apply fusion to such patterns. For example, video encoder200and video decoder300may apply fusion with the two smallest-cost candidates, while video encoder200and video decoder300do not apply fusion with other template patterns. In this way, a better trade-off may be achieved because the fusion provides enough diversity while overhead signaling may be reduced by not having fusion for other patterns.

Examples of signaling and syntax are now described. The described additional modes (various patterns, fusion, etc.) may be indicated by a syntax element transmitted in a bitstream to video decoder300. In another example, video encoder200and video decoder300may implicitly derive the used mode for prediction without signaling by following the same process. In one example, the modes may be ordered based on the template matching cost and the smallest cost mode is selected to be used. In some examples, the modes may be ordered, for example by template matching cost, and an index is signaled to this ordered list of modes to identify the one to be used.

In some examples, if template matching is applied in the current block, there is a flag to indicate that whether the multiple candidates are selected and a combination process for the candidate predictors is applied. In some examples, if template matching is applied in the current block, there is a flag to indicate that the type of template pattern which is used in the template matching process.

FIG.8is a block diagram illustrating an example video encoder200that may perform the techniques of this disclosure.FIG.8is provided for purposes of explanation and should not be considered limiting of the techniques as broadly exemplified and described in this disclosure. For purposes of explanation, this disclosure describes video encoder200according to the techniques of VVC and HEVC. However, the techniques of this disclosure may be performed by video encoding devices that are configured to other video coding standards and video coding formats, such as AV1 and successors to the AV1 video coding format.

In the example ofFIG.8, video encoder200includes video data memory830, mode selection unit802, residual generation unit804, transform processing unit806, quantization unit808, inverse quantization unit810, inverse transform processing unit812, reconstruction unit814, filter unit816, decoded picture buffer (DPB)818, and entropy encoding unit820. Any or all of video data memory830, mode selection unit802, residual generation unit804, transform processing unit806, quantization unit808, inverse quantization unit810, inverse transform processing unit812, reconstruction unit814, filter unit816, DPB818, and entropy encoding unit820may be implemented in one or more processors or in processing circuitry. For instance, the units of video encoder200may be implemented as one or more circuits or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Moreover, video encoder200may include additional or alternative processors or processing circuitry to perform these and other functions.

Video data memory830may store video data to be encoded by the components of video encoder200. Video encoder200may receive the video data stored in video data memory830from, for example, video source104(FIG.1). DPB818may act as a reference picture memory that stores reference video data for use in prediction of subsequent video data by video encoder200. Video data memory830and DPB818may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory830and DPB818may be provided by the same memory device or separate memory devices. In various examples, video data memory830may be on-chip with other components of video encoder200, as illustrated, or off-chip relative to those components.

Video data memory830is configured to store received video data. Video encoder200may retrieve a picture of the video data from video data memory830and provide the video data to residual generation unit804and mode selection unit802. Video data in video data memory830may be raw video data that is to be encoded.

Mode selection unit802includes a motion estimation unit822, a motion compensation unit824, and an intra-prediction unit826. Mode selection unit802may include additional functional units to perform video prediction in accordance with other prediction modes. As examples, mode selection unit802may include a palette unit, an intra-block copy unit (which may be part of motion estimation unit822and/or motion compensation unit824), an affine unit, a linear model (LM) unit, or the like. In the example ofFIG.8, motion estimation unit822includes a TM unit832. In accordance with a technique of this disclosure, TM unit832may determine a template pattern from among a set of two or more template patterns. TM unit832may identify a reference template based on a similarity of the reference template and a current template. The reference template and the current template may have shapes defined by the template pattern. The reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data. Motion compensation unit824may obtain, based on the reference template, a prediction block for a current block of a current picture of the video data.

Mode selection unit802generally coordinates multiple encoding passes to test combinations of encoding parameters and resulting rate-distortion values for such combinations. The encoding parameters may include partitioning of CTUs into CUs, prediction modes for the CUS, transform types for residual data of the CUs, quantization parameters for residual data of the CUs, and so on. Mode selection unit802may ultimately select the combination of encoding parameters having rate-distortion values that are better than the other tested combinations.

In general, mode selection unit802also controls the components thereof (e.g., motion estimation unit822, motion compensation unit824, and intra-prediction unit826) to generate a prediction block for a current block (e.g., a current CU, or in HEVC, the overlapping portion of a PU and a TU). For inter-prediction of a current block, motion estimation unit822may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously coded pictures stored in DPB818). In particular, motion estimation unit822may calculate a value representative of how similar a potential reference block is to the current block, e.g., according to sum of absolute difference (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared differences (MSD), or the like. Motion estimation unit822may generally perform these calculations using sample-by-sample differences between the current block and the reference block being considered. Motion estimation unit822may identify a reference block having a lowest value resulting from these calculations, indicating a reference block that most closely matches the current block.

Motion estimation unit822may form one or more motion vectors (MVs) that defines the positions of the reference blocks in the reference pictures relative to the position of the current block in a current picture. Motion estimation unit822may then provide the motion vectors to motion compensation unit824. For example, for uni-directional inter-prediction, motion estimation unit822may provide a single motion vector, whereas for bi-directional inter-prediction, motion estimation unit822may provide two motion vectors. Motion compensation unit824may then generate a prediction block using the motion vectors. For example, motion compensation unit824may retrieve data of the reference block using the motion vector. As another example, if the motion vector has fractional sample precision, motion compensation unit824may interpolate values for the prediction block according to one or more interpolation filters. Moreover, for bi-directional inter-prediction, motion compensation unit824may retrieve data for two reference blocks identified by respective motion vectors and combine the retrieved data, e.g., through sample-by-sample averaging or weighted averaging.

When operating according to the AV1 video coding format, motion estimation unit822and motion compensation unit824may be configured to encode coding blocks of video data (e.g., both luma and chroma coding blocks) using translational motion compensation, affine motion compensation, overlapped block motion compensation (OBMC), and/or compound inter-intra prediction.

When operating according to the AV1 video coding format, intra-prediction unit826may be configured to encode coding blocks of video data (e.g., both luma and chroma coding blocks) using directional intra prediction, non-directional intra prediction, recursive filter intra prediction, chroma-from-luma (CFL) prediction, intra block copy (IBC), and/or color palette mode. Mode selection unit802may include additional functional units to perform video prediction in accordance with other prediction modes.

Mode selection unit802provides the prediction block to residual generation unit804. Residual generation unit804receives a raw, unencoded version of the current block from video data memory830and the prediction block from mode selection unit802. Residual generation unit804calculates sample-by-sample differences between the current block and the prediction block. The resulting sample-by-sample differences define a residual block for the current block. In some examples, residual generation unit804may also determine differences between sample values in the residual block to generate a residual block using residual differential pulse code modulation (RDPCM). In some examples, residual generation unit804may be formed using one or more subtractor circuits that perform binary subtraction.

For other video coding techniques such as an intra-block copy mode coding, an affine-mode coding, and linear model (LM) mode coding, as some examples, mode selection unit802, via respective units associated with the coding techniques, generates a prediction block for the current block being encoded. In some examples, such as palette mode coding, mode selection unit802may not generate a prediction block, and instead generate syntax elements that indicate the manner in which to reconstruct the block based on a selected palette. In such modes, mode selection unit802may provide these syntax elements to entropy encoding unit820to be encoded.

As described above, residual generation unit804receives the video data for the current block and the corresponding prediction block. Residual generation unit804then generates a residual block for the current block. To generate the residual block, residual generation unit804calculates sample-by-sample differences between the prediction block and the current block.

Transform processing unit806applies one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a “transform coefficient block”). Transform processing unit806may apply various transforms to a residual block to form the transform coefficient block. For example, transform processing unit806may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to a residual block. In some examples, transform processing unit806may perform multiple transforms to a residual block, e.g., a primary transform and a secondary transform, such as a rotational transform. In some examples, transform processing unit806does not apply transforms to a residual block.

When operating according to AV1, transform processing unit806may apply one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a “transform coefficient block”). Transform processing unit806may apply various transforms to a residual block to form the transform coefficient block. For example, transform processing unit806may apply a horizontal/vertical transform combination that may include a discrete cosine transform (DCT), an asymmetric discrete sine transform (ADST), a flipped ADST (e.g., an ADST in reverse order), and an identity transform (IDTX). When using an identity transform, the transform is skipped in one of the vertical or horizontal directions. In some examples, transform processing may be skipped.

Quantization unit808may quantize the transform coefficients in a transform coefficient block, to produce a quantized transform coefficient block. Quantization unit808may quantize transform coefficients of a transform coefficient block according to a quantization parameter (QP) value associated with the current block. Video encoder200(e.g., via mode selection unit802) may adjust the degree of quantization applied to the transform coefficient blocks associated with the current block by adjusting the QP value associated with the CU. Quantization may introduce loss of information, and thus, quantized transform coefficients may have lower precision than the original transform coefficients produced by transform processing unit806.

Inverse quantization unit810and inverse transform processing unit812may apply inverse quantization and inverse transforms to a quantized transform coefficient block, respectively, to reconstruct a residual block from the transform coefficient block. Reconstruction unit814may produce a reconstructed block corresponding to the current block (albeit potentially with some degree of distortion) based on the reconstructed residual block and a prediction block generated by mode selection unit802. For example, reconstruction unit814may add samples of the reconstructed residual block to corresponding samples from the prediction block generated by mode selection unit802to produce the reconstructed block.

Filter unit816may perform one or more filter operations on reconstructed blocks. For example, filter unit816may perform deblocking operations to reduce blockiness artifacts along edges of CUs. Operations of filter unit816may be skipped, in some examples.

When operating according to AV1, filter unit816may perform one or more filter operations on reconstructed blocks. For example, filter unit816may perform deblocking operations to reduce blockiness artifacts along edges of CUs. In other examples, filter unit816may apply a constrained directional enhancement filter (CDEF), which may be applied after deblocking, and may include the application of non-separable, non-linear, low-pass directional filters based on estimated edge directions. Filter unit816may also include a loop restoration filter, which is applied after CDEF, and may include a separable symmetric normalized Wiener filter or a dual self-guided filter.

Video encoder200stores reconstructed blocks in DPB818. For instance, in examples where operations of filter unit816are not performed, reconstruction unit814may store reconstructed blocks to DPB818. In examples where operations of filter unit816are performed, filter unit816may store the filtered reconstructed blocks to DPB818. Motion estimation unit822and motion compensation unit824may retrieve a reference picture from DPB818, formed from the reconstructed (and potentially filtered) blocks, to inter-predict blocks of subsequently encoded pictures. In addition, intra-prediction unit826may use reconstructed blocks in DPB818of a current picture to intra-predict other blocks in the current picture.

Video encoder200may output a bitstream that includes the entropy encoded syntax elements needed to reconstruct blocks of a slice or picture. In particular, entropy encoding unit820may output the bitstream.

In accordance with AV1, entropy encoding unit820may be configured as a symbol-to-symbol adaptive multi-symbol arithmetic coder. A syntax element in AV1 includes an alphabet of N elements, and a context (e.g., probability model) includes a set of N probabilities. Entropy encoding unit820may store the probabilities as n-bit (e.g., 15-bit) cumulative distribution functions (CDFs). Entropy encoding unit820may perform recursive scaling, with an update factor based on the alphabet size, to update the contexts.

Video encoder200represents an example of a device configured to encode video data including a memory configured to store video data, and one or more processing units implemented in circuitry and configured to determine a template pattern from among a set of two or more template patterns; identify a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data; obtain, based on the reference template, a prediction block for a current block of a current picture of the video data; and encode the current block based on the prediction block.

In the example ofFIG.9, video decoder300includes coded picture buffer (CPB) memory920, entropy decoding unit902, prediction processing unit904, inverse quantization unit906, inverse transform processing unit908, reconstruction unit910, filter unit912, and DPB914. Any or all of CPB memory920, entropy decoding unit902, prediction processing unit904, inverse quantization unit906, inverse transform processing unit908, reconstruction unit910, filter unit912, and DPB914may be implemented in one or more processors or in processing circuitry. For instance, the units of video decoder300may be implemented as one or more circuits or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Moreover, video decoder300may include additional or alternative processors or processing circuitry to perform these and other functions.

Prediction processing unit904includes motion compensation unit916and intra-prediction unit918. Prediction processing unit904may include additional units to perform prediction in accordance with other prediction modes. As examples, prediction processing unit904may include a palette unit, an intra-block copy unit (which may form part of motion compensation unit916), an affine unit, a linear model (LM) unit, or the like. In other examples, video decoder300may include more, fewer, or different functional components.

When operating according to AV1, motion compensation unit916may be configured to decode coding blocks of video data (e.g., both luma and chroma coding blocks) using translational motion compensation, affine motion compensation, OBMC, and/or compound inter-intra prediction, as described above. Intra-prediction unit918may be configured to decode coding blocks of video data (e.g., both luma and chroma coding blocks) using directional intra prediction, non-directional intra prediction, recursive filter intra prediction, CFL, IBC, and/or color palette mode, as described above.

Entropy decoding unit902may receive encoded video data from the CPB and entropy decode the video data to reproduce syntax elements. Prediction processing unit904, inverse quantization unit906, inverse transform processing unit908, reconstruction unit910, and filter unit912may generate decoded video data based on the syntax elements extracted from the bitstream.

After inverse quantization unit906forms the transform coefficient block, inverse transform processing unit908may apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, inverse transform processing unit908may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotational transform, an inverse directional transform, or another inverse transform to the transform coefficient block.

Furthermore, prediction processing unit904generates a prediction block according to prediction information syntax elements that were entropy decoded by entropy decoding unit902. For example, if the prediction information syntax elements indicate that the current block is inter-predicted, motion compensation unit916may generate the prediction block. In this case, the prediction information syntax elements may indicate a reference picture in DPB914from which to retrieve a reference block, as well as a motion vector identifying a location of the reference block in the reference picture relative to the location of the current block in the current picture. Motion compensation unit916may generally perform the inter-prediction process in a manner that is substantially similar to that described with respect to motion compensation unit824(FIG.8).

TM unit922may perform one or more template-matching operations to generate a prediction block. In accordance with a technique of this disclosure, TM unit922may determine a template pattern from among a set of two or more template patterns. Additionally, TM unit922may identify a reference template based on a similarity of the reference template and a current template. The reference template and the current template have a shape defined by the template pattern. The reference template may include previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data. TM unit922may obtain, based on the reference template, a prediction block for a current block of a current picture of the video data.

If the prediction information syntax elements indicate that the current block is intra-predicted, intra-prediction unit918may generate the prediction block according to an intra-prediction mode indicated by the prediction information syntax elements. Again, intra-prediction unit918may generally perform the intra-prediction process in a manner that is substantially similar to that described with respect to intra-prediction unit826(FIG.8). Intra-prediction unit918may retrieve data of neighboring samples to the current block from DPB914.

Reconstruction unit910may reconstruct the current block using the prediction block and the residual block. For example, reconstruction unit910may add samples of the residual block to corresponding samples of the prediction block to reconstruct the current block.

Filter unit912may perform one or more filter operations on reconstructed blocks. For example, filter unit912may perform deblocking operations to reduce blockiness artifacts along edges of the reconstructed blocks. Operations of filter unit912are not necessarily performed in all examples.

Video decoder300may store the reconstructed blocks in DPB914. For instance, in examples where operations of filter unit912are not performed, reconstruction unit910may store reconstructed blocks to DPB914. In examples where operations of filter unit912are performed, filter unit912may store the filtered reconstructed blocks to DPB914. As discussed above, DPB914may provide reference information, such as samples of a current picture for intra-prediction and previously decoded pictures for subsequent motion compensation, to prediction processing unit904. Moreover, video decoder300may output decoded pictures (e.g., decoded video) from DPB914for subsequent presentation on a display device, such as display device118ofFIG.1.

In this manner, video decoder300represents an example of a video decoding device including a memory configured to store video data, and one or more processing units implemented in circuitry and configured to determine a template pattern from among a set of two or more template patterns; identify a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data; obtain, based on the reference template, a prediction block for a current block of a current picture of the video data; and decode the current block based on the prediction block.

FIG.10is a flowchart illustrating an example method for encoding a current block in accordance with the techniques of this disclosure. The current block may be or include a current CU. Although described with respect to video encoder200(FIGS.1and8), it should be understood that other devices may be configured to perform a method similar to that ofFIG.10.

In this example, video encoder200initially predicts the current block (1000). For example, video encoder200may form a prediction block for the current block. Video encoder200may generate the prediction block using the template-matching techniques of this disclosure.

Video encoder200may then calculate a residual block for the current block (1002). To calculate the residual block, video encoder200may calculate a difference between the original, unencoded block and the prediction block for the current block. Video encoder200may then transform the residual block and quantize transform coefficients of the residual block (1004). Next, video encoder200may scan the quantized transform coefficients of the residual block (1006). During the scan, or following the scan, video encoder200may entropy encode the transform coefficients (1008). For example, video encoder200may encode the transform coefficients using CAVLC or CABAC. Video encoder200may then output the entropy encoded data of the block (1010).

FIG.11is a flowchart illustrating an example method for decoding a current block of video data in accordance with the techniques of this disclosure. The current block may be or include a current CU. Although described with respect to video decoder300(FIGS.1and9), it should be understood that other devices may be configured to perform a method similar to that ofFIG.11.

Video decoder300may receive entropy encoded data for the current block, such as entropy encoded prediction information and entropy encoded data for transform coefficients of a residual block corresponding to the current block (1100). Video decoder300may entropy decode the entropy encoded data to determine prediction information for the current block and to reproduce transform coefficients of the residual block (1102). Video decoder300may predict the current block (1104), e.g., using an intra- or inter-prediction mode as indicated by the prediction information for the current block, to calculate a prediction block for the current block. Video decoder300may generate the prediction block using the template-matching techniques of this disclosure.

Video decoder300may then inverse scan the reproduced transform coefficients (1106), to create a block of quantized transform coefficients. Video decoder300may then inverse quantize the transform coefficients and apply an inverse transform to the transform coefficients to produce a residual block (1108). Video decoder300may ultimately decode the current block by combining the prediction block and the residual block (1110).

FIG.12is a flowchart illustrating an example method for encoding or decoding video data involving template matching according to techniques of this disclosure. In the example ofFIG.12, a video coder (e.g., video encoder200or video decoder300) may determine a template pattern from among a set of two or more template patterns (1200). In some examples, the set of two or more template patterns includes a base template pattern and one or more additional template patterns that fit within a region of the base template pattern. A template having a shape defined by the base template pattern may include reference samples above and left of the current block. In some such examples, the video coder may determine a cost for the base template pattern and may determine costs of the additional template patterns based on costs of subregions within the region of the base template pattern. The video coder may determine the template pattern from among the two or more template patterns based on the cost for the base template pattern and the costs for the additional template patterns. For instance, the video coder may determine which template pattern in the set of two or more template patterns has the lowest cost.

In some examples where the video coder is video encoder200, video encoder200may signal one or more syntax elements that identify the template pattern. In some examples where the video coder is video decoder300, video decoder300may obtain, from a bitstream that includes an encoded representation of the video data, one or more syntax elements that indicate the template pattern and may determine the template pattern based on the one or more syntax elements. In such examples, it may be unnecessary for video decoder300to determine costs for template patterns.

The video coder may identify a reference template based on a similarity of the reference template and a current template (1202). The reference template and the current template have a shape defined by the template pattern. The reference template includes previously reconstructed samples. The current template includes reference samples of a current block of a current picture of the video data. For instance, the video coder may perform a search within a search area of the current picture or a reference picture to identify a reference template having samples similar to samples in the current template.

The video coder may obtain, based on the reference template, a prediction block for a current block of a current picture of the video data (1204). In some examples, bi-directional prediction is used, the reference template is a first reference template, and, as part of obtaining the prediction block for the current block, the video coder may determine a first prediction block based on the first reference template. The video coder may also determine a second prediction block based on a second reference template. In this example, the shape of the first reference template and a shape of the second reference template are defined by the template pattern. The video coder may generate the prediction block for the current block based on the first prediction block and the second prediction block. In some examples, bi-directional prediction is used, the video coder may generate a first prediction block using the reference template, generate a second prediction block using a motion vector candidate, and may generate the prediction block for the current block based on the first prediction block and the second prediction block.

The video coder may encode or decode the current block based on the prediction block (1206). For example, as part of encoding the current block, the video coder may generate residual data indicating differences between the current block and the prediction block. The video coder may apply a transformation to the residual data to generate transform coefficients. The video coder may then quantize the transform coefficients and apply entropy encoding to syntax elements representing the quantized transform coefficients. In an example where the video coder decodes the current block, the video coder may obtain residual data and combine the residual data with the prediction block to reconstruct the current block.

The following numbered clauses illustrate one or more aspects of the devices and techniques described in this disclosure.

Clause 1A. A method of coding video data, the method comprising: determining a template pattern from among a set of two or more template patterns; identifying a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data; obtaining, based on the reference template, a prediction block for a current block of a current picture of the video data; and encoding or decoding the current block based on the prediction block.

Clause 2A. The method of clause 1A, wherein the set of template patterns includes a base template pattern and one or more additional template patterns that fit within a region of the base template pattern.

Clause 3A. The method of clause 2A, wherein identifying the reference template comprises: determining a cost for the base template pattern; and determining costs of the additional template patterns based on costs of subregions within the region of the base template pattern.

Clause 4A. The method of any of clauses 2A and 3A, wherein a template having a shape defined by the base template pattern includes reference samples above and left of the current block.

Clause 5A. The method of any of clauses 1A-4A, further comprising signaling one or more syntax elements that identify the template pattern.

Clause 6A. The method of any of clauses 1A-4A, wherein: the method further comprises obtaining, from a bitstream that includes an encoded representation of the video data, one or more syntax elements that indicate the template pattern; and determining the template pattern comprises determining the template pattern based on the one or more syntax elements.

Clause 7A. The method of any of clauses 5A or 6A, wherein the syntax elements are signaled at a coding unit, prediction unit, coding tree unit, slice, or picture level.

Clause 8A. The method of any of clauses 1A-7A, wherein: the reference template is a first reference template, and the method further comprises: generating a list of motion vector candidates, wherein generating the list of motion vector candidates comprises: determining template matching costs for reference templates in a plurality of reference templates, wherein each reference template in the plurality of reference templates has a shape defined by the template shape; selecting two or more of the reference templates based on the template matching costs for the reference templates, wherein the selected reference templates include the first reference template; and including candidates indicating motion vectors of the selected reference templates in the list of motion vector candidates, determining a selected motion vector in the list of motion vector candidates, wherein the selected motion vector is the motion vector of the first reference template; and using the selected motion vector to determine a location of the prediction block for the current block.

Clause 9A. The method of clause 8A, wherein generating the list of motion vector candidates comprises excluding a motion vector of a reference template from the list of motion vector candidates based on a comparison of a distance of the motion vector of the reference template and a temporary best candidate motion vector in the list of motion vector candidates.

Clause 10A. The method of any of clauses 8A-9A, wherein generating the list of motion vector candidates comprises excluding a motion vector of a reference template from the list of motion vector candidates based on a comparison of a difference between a prediction block of a reference template and a prediction block of a temporary best candidate motion vector in the list of motion vector candidates.

Clause 11A. The method of any of clauses 8A-10A, wherein generating the list of motion vector candidates comprises removing, from the list of motion vector candidates, a candidate that indicates a motion vector of a first one of the selected reference templates based on a comparison of a predefined threshold and a difference of the cost value of the first selected reference template and the cost value of a second selected reference template.

Clause 12A. The method of any of clauses 1A-11A, wherein: the reference template is a first reference template, and obtaining the prediction block comprises: determining a first prediction block based on the first reference template; determining a second prediction block based on a second reference template, wherein the shape of the first reference template and a shape of the second reference template are defined by the template pattern; and generating the prediction block for the current block based on the first prediction block and the second prediction block.

Clause 13A. The method of clause 12A, wherein generating the prediction block for the current block comprises: determining a weight for the first prediction block based on a template matching cost of the first reference template; determining a weight for the second prediction block based on a template matching cost of the second reference template; and generating samples of the prediction block for the current block as a combination of samples of the first prediction block and the second prediction block weighted according to the weight for the first prediction block and the weight for the second prediction block.

Clause 14A. The method of clause 13A, wherein determining the weight for the first prediction block and determining the weight for the second prediction block comprises determining the weight for the first prediction block and determining the weight for the second prediction block based on a sum of absolute differences (SAD), a mean square error (MSE), or a block vector (BV).

Clause 15A. The method of clause 13A, wherein: determining the weight for the first prediction block comprises determining the weight for the first prediction block based on a multiplicative inverse of the template matching cost of the first reference template, and determining the weight for the second prediction block comprises determining the weight for the second prediction block based on a multiplicative inverse of the template matching cost of the second reference template.

Clause 16A. The method of any of clauses 13A-15A, wherein determining the weight for the first prediction block comprises setting the weight for the first prediction block to 0 based on a comparison of a predefined threshold and a distance of the motion vector of the first reference template and another motion vector in the list of motion vector candidates.

Clause 17A. The method of any of clauses 13A-16A, wherein determining the weight for the first prediction block comprises setting the weight for the first prediction block to 0 based on a comparison of a predetermined threshold and a difference between the first prediction block and the second prediction block.

Clause 18A. The method of any of clauses 8A-17A, wherein obtaining the prediction block comprises determining the prediction block based on a linear combination of prediction blocks for two motion vector candidates in the list of motion vector candidates.

Clause 19A. The method of any of clauses 8A-17A, wherein obtaining the prediction block comprises, based on cost values of two motion vector candidates in the list of motion vector candidates being less than a threshold, determining the prediction block based on a linear combination of prediction blocks for the two motion vector candidates.

Clause 20A. The method of any of clauses 8A-19A, wherein a syntax element indicates the whether the two or more reference templates are selected.

Clause 21A. The method of any of clauses 1A-20A, wherein obtaining the prediction block for the current block comprises: generating a first prediction block using the reference template; generating a second prediction block using a motion vector candidate; and generating the prediction block for the current block based on the first prediction block and the second prediction block.

Clause 22A. The method of any of clauses 1A-21A, wherein: the template pattern is a first template pattern, the reference template is a first reference template, and the current template is a first current template, obtaining the prediction block for the current block comprises: generating a first prediction block using the first reference template; identifying a second reference template based on a similarity of the second reference template and a second current template, wherein the second reference template and the second current template have a shape defined by a second template pattern different from the first template pattern, the second reference template includes second previously reconstructed samples and the second current template includes second reference samples of the current block; generating a second prediction block using the second reference template; and generating the prediction block for the current block based on the first prediction block and the second prediction block.

Clause 23A. The method of clause 22A, wherein generating the prediction block for the current block comprises: determining a weight for the first prediction block and a weight for the second prediction block based on template matching costs of the first reference template and the second reference template; and generating samples of the prediction block for the current block as a combination of samples of the first prediction block and the second prediction block weighted according to the weight for the first prediction block and the weight for the second prediction block.

Clause 24A. A device for coding video data, the device comprising one or more means for performing the method of any of clauses 1A-23A.

Clause 25A. The device of clause 24A, wherein the one or more means comprise one or more processors implemented in circuitry.

Clause 26A. The device of any of clauses 24A and 25A, further comprising a memory to store the video data.

Clause 27A. The device of any of clauses 24A-26A, further comprising a display configured to display decoded video data.

Clause 28A. The device of any of clauses 24A-27A, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

Clause 29A. The device of any of clauses 24A-28A, wherein the device comprises a video decoder.

Clause 30A. The device of any of clauses 24A-29A, wherein the device comprises a video encoder.

Clause 31A. A computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to perform the method of any of clauses 1A-23A.

Clause 1B. A method of coding video data, the method comprising: determining a template pattern from among a set of two or more template patterns; identifying a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data; obtaining, based on the reference template, a prediction block for the current block; and encoding or decoding the current block based on the prediction block.

Clause 2B. The method of clause 1B, wherein the set of two or more template patterns includes a base template pattern and one or more additional template patterns that fit within a region of the base template pattern.

Clause 3B. The method of clause 2B, wherein identifying the reference template comprises: determining a cost for the base template pattern; and determining costs of the additional template patterns based on costs of subregions within the region of the base template pattern.

Clause 4B. The method of any of clauses 2B-3B, wherein a template having a shape defined by the base template pattern includes reference samples above and left of the current block.

Clause 5B. The method of any of clauses 1B-4B, further comprising signaling one or more syntax elements that identify the template pattern.

Clause 6B. The method of any of clauses 1B-4B, wherein: the method further comprises obtaining, from a bitstream that includes an encoded representation of the video data, one or more syntax elements that indicate the template pattern; and determining the template pattern comprises determining the template pattern based on the one or more syntax elements.

Clause 7B. The method of any of clauses 1B-6B, wherein: the reference template is a first reference template, and the method further comprises: generating a list of motion vector candidates, wherein generating the list of motion vector candidates comprises: determining template matching costs for reference templates in a plurality of reference templates, wherein each reference template in the plurality of reference templates has the shape defined by the template pattern; selecting two or more of the reference templates based on the template matching costs for the reference templates, wherein the selected reference templates include the first reference template; and including candidates indicating motion vectors of the selected reference templates in the list of motion vector candidates, determining a selected motion vector in the list of motion vector candidates, wherein the selected motion vector is the motion vector of the first reference template; and using the selected motion vector to determine a location of the prediction block for the current block.

Clause 8B. The method of clause 7B, wherein generating the list of motion vector candidates comprises at least one of: excluding a motion vector of a reference template from the list of motion vector candidates based on a comparison of a distance of the motion vector of the reference template and a temporary best candidate motion vector in the list of motion vector candidates, excluding a motion vector of a reference template from the list of motion vector candidates based on a comparison of a difference between a prediction block of a reference template and a prediction block of a temporary best candidate motion vector in the list of motion vector candidates, or removing, from the list of motion vector candidates, a candidate that indicates a motion vector of a first one of the selected reference templates based on a comparison of a predefined threshold and a difference of the cost of the first selected reference template and the cost of a second selected reference template.

Clause 9B. The method of any of clauses 7B-8B, wherein: the reference template is a first reference template, and obtaining the prediction block comprises: determining a first prediction block based on the first reference template; determining a second prediction block based on a second reference template, wherein the shape of the first reference template and a shape of the second reference template are defined by the template pattern; and generating the prediction block for the current block based on the first prediction block and the second prediction block.

Clause 10B. The method of clause 9B, wherein generating the prediction block for the current block comprises: determining a weight for the first prediction block based on a template matching cost of the first reference template; determining a weight for the second prediction block based on a template matching cost of the second reference template; and generating samples of the prediction block for the current block as a combination of samples of the first prediction block and the second prediction block weighted according to the weight for the first prediction block and the weight for the second prediction block.

Clause 11B. The method of clause 10B, wherein: determining the weight for the first prediction block comprises determining the weight for the first prediction block based on a multiplicative inverse of the template matching cost of the first reference template, and determining the weight for the second prediction block comprises determining the weight for the second prediction block based on a multiplicative inverse of the template matching cost of the second reference template.

Clause 12B. The method of any of clauses 10B-11B, wherein determining the weight for the first prediction block comprises at least one of: setting the weight for the first prediction block to 0 based on a comparison of a predefined threshold and a distance of the motion vector of the first reference template and another motion vector in the list of motion vector candidates, or setting the weight for the first prediction block to 0 based on a comparison of a predetermined threshold and a difference between the first prediction block and the second prediction block.

Clause 13B. The method of any of clauses 7B-12B, wherein obtaining the prediction block comprises determining the prediction block based on a linear combination of prediction blocks for two motion vector candidates in the list of motion vector candidates.

Clause 14B. The method of any of clauses 7B-12B, wherein obtaining the prediction block comprises, based on costs of two motion vector candidates in the list of motion vector candidates being less than a threshold, determining the prediction block based on a linear combination of prediction blocks for the two motion vector candidates.

Clause 15B. The method of any of clauses 7B-14B, wherein a syntax element indicates whether the two or more reference templates are selected.

Clause 16B. The method of any of clauses 1B-15B, wherein obtaining the prediction block for the current block comprises: generating a first prediction block using the reference template; generating a second prediction block using a motion vector candidate; and generating the prediction block for the current block based on the first prediction block and the second prediction block.

Clause 17B. The method of any of clauses 1B-16B, wherein: the template pattern is a first template pattern, the reference template is a first reference template, and the current template is a first current template, obtaining the prediction block for the current block comprises: generating a first prediction block using the first reference template; identifying a second reference template based on a similarity of the second reference template and a second current template, wherein the second reference template and the second current template have a shape defined by a second template pattern different from the first template pattern, the second reference template includes second previously reconstructed samples and the second current template includes second reference samples of the current block; generating a second prediction block using the second reference template; and generating the prediction block for the current block based on the first prediction block and the second prediction block.

Clause 18B. The method of clause 17B, wherein generating the prediction block for the current block comprises: determining a weight for the first prediction block and a weight for the second prediction block based on template matching costs of the first reference template and the second reference template; and generating samples of the prediction block for the current block as a combination of samples of the first prediction block and the second prediction block weighted according to the weight for the first prediction block and the weight for the second prediction block.

Clause 19B. A device for coding video data, the device comprising: a memory configured to store the video data; and one or more processors implemented in circuitry, the one or more processors configured to: determine a template pattern from among a set of two or more template patterns; identify a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data; obtain, based on the reference template, a prediction block for the current block; and encode or decode the current block based on the prediction block.

Clause 20B. The device of clause 19B, wherein the set of two or more template patterns includes a base template pattern and one or more additional template patterns that fit within a region of the base template pattern.

Clause 21B. The device of clause 20B, wherein the one or more processors are configured to, as part of identifying the reference template: determine a cost for the base template pattern; and determine costs of the additional template patterns based on costs of subregions within the region of the base template pattern.

Clause 22B. The device of any of clauses 20B-21B, wherein a template having a shape defined by the base template pattern includes reference samples above and left of the current block.

Clause 23B. The device of any of clauses 19B-22B, wherein the one or more processors are further configured to signal one or more syntax elements that identify the template pattern.

Clause 24B. The device of any of clauses 19B-23B, wherein: the one or more processors are further configured to obtain, from a bitstream that includes an encoded representation of the video data, one or more syntax elements that indicate the template pattern; and the one or more processors are configured to, as part determining the template pattern, determine the template pattern based on the one or more syntax elements.

Clause 25B. The device of any of clauses 19B-24B, wherein: the reference template is a first reference template, and the one or more processors are further configured to: generate a list of motion vector candidates, wherein the one or more processors are configured to, as part of generating the list of motion vector candidates: determine template matching costs for reference templates in a plurality of reference templates, wherein each reference template in the plurality of reference templates has the shape defined by the template pattern; select two or more of the reference templates based on the template matching costs for the reference templates, wherein the selected reference templates include the first reference template; and include candidates indicating motion vectors of the selected reference templates in the list of motion vector candidates, determine a selected motion vector in the list of motion vector candidates, wherein the selected motion vector is the motion vector of the first reference template; and use the selected motion vector to determine a location of the prediction block for the current block.

Clause 26B. The device of clause 25B, wherein the one or more processors are configured to, as part of generating the list of motion vector candidates, perform at least one of: excluding a motion vector of a reference template from the list of motion vector candidates based on a comparison of a distance of the motion vector of the reference template and a temporary best candidate motion vector in the list of motion vector candidates, excluding a motion vector of a reference template from the list of motion vector candidates based on a comparison of a difference between a prediction block of a reference template and a prediction block of a temporary best candidate motion vector in the list of motion vector candidates, or removing, from the list of motion vector candidates, a candidate that indicates a motion vector of a first one of the selected reference templates based on a comparison of a predefined threshold and a difference of the cost of the first selected reference template and the cost of a second selected reference template.

Clause 27B. The device of any of clauses 25B-26B, wherein: the reference template is a first reference template, and the one or more processors are configured to, as part of obtaining the prediction block: determine a first prediction block based on the first reference template; determine a second prediction block based on a second reference template, wherein the shape of the first reference template and a shape of the second reference template are defined by the template pattern; and generate the prediction block for the current block based on the first prediction block and the second prediction block.

Clause 28B. The device of clause 27B, wherein the one or more processors are configured to, as part of generating the prediction block for the current block: determine a weight for the first prediction block based on a template matching cost of the first reference template; determine a weight for the second prediction block based on a template matching cost of the second reference template; and generate samples of the prediction block for the current block as a combination of samples of the first prediction block and the second prediction block weighted according to the weight for the first prediction block and the weight for the second prediction block.

Clause 29B. The device of clause 28B, wherein: the one or more processors are configured to, as part of determining the weight for the first prediction block, determine the weight for the first prediction block based on a multiplicative inverse of the template matching cost of the first reference template, and the one or more processors are configured to, as part of determining the weight for the second prediction block, determine the weight for the second prediction block based on a multiplicative inverse of the template matching cost of the second reference template.

Clause 30B. The device of any of clauses 28B-29B, wherein the one or more processors are configured to, as part of determining the weight for the first prediction block, perform at least one of: setting the weight for the first prediction block to 0 based on a comparison of a predefined threshold and a distance of the motion vector of the first reference template and another motion vector in the list of motion vector candidates, or setting the weight for the first prediction block to 0 based on a comparison of a predetermined threshold and a difference between the first prediction block and the second prediction block.

Clause 31B. The device of any of clauses 25B-30B, wherein the one or more processors are configured to, as part of obtaining the prediction block, determine the prediction block based on a linear combination of prediction blocks for two motion vector candidates in the list of motion vector candidates.

Clause 32B. The device of any of clauses 25B-30B, wherein the one or more processors are configured to, as part of obtaining the prediction block, based on costs of two motion vector candidates in the list of motion vector candidates being less than a threshold, determine the prediction block based on a linear combination of prediction blocks for the two motion vector candidates.

Clause 33B. The device of any of clauses 25B-32B, wherein a syntax element indicates whether the two or more reference templates are selected.

Clause 34B. The device of any of clauses 19B-33B, wherein the one or more processors are configured to, as part of obtaining the prediction block for the current block: generate a first prediction block using the reference template; generate a second prediction block using a motion vector candidate; and generate the prediction block for the current block based on the first prediction block and the second prediction block.

Clause 35B. The device of any of clauses 19B-34B, wherein: the template pattern is a first template pattern, the reference template is a first reference template, and the current template is a first current template, the one or more processors are configured to, as part of obtaining the prediction block for the current block: generate a first prediction block using the first reference template; identify a second reference template based on a similarity of the second reference template and a second current template, wherein the second reference template and the second current template have a shape defined by a second template pattern different from the first template pattern, the second reference template includes second previously reconstructed samples and the second current template includes second reference samples of the current block; generate a second prediction block using the second reference template; and generate the prediction block for the current block based on the first prediction block and the second prediction block.

Clause 36B. The device of clause 35B, wherein the one or more processors are configured to, as part of generating the prediction block for the current block: determine a weight for the first prediction block and a weight for the second prediction block based on template matching costs of the first reference template and the second reference template; and generate samples of the prediction block for the current block as a combination of samples of the first prediction block and the second prediction block weighted according to the weight for the first prediction block and the weight for the second prediction block.

Clause 37. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to: determine a template pattern from among a set of two or more template patterns; identify a reference template based on a similarity of the reference template and a current template, wherein the reference template and the current template have a shape defined by the template pattern, the reference template includes previously reconstructed samples and the current template includes reference samples of a current block of a current picture of the video data; obtain, based on the reference template, a prediction block for the current block; and encode or decode the current block based on the prediction block.