Video coding techniques for multi-view video

Techniques are disclosed for coding and decoding video captured as cube map images. According to these techniques, padded reference images are generated for use during predicting input data. A reference image is stored in a cube map format. A padded reference image is generated from the reference image in which image data of a first view contained in reference image is replicated and placed adjacent to a second view contained in the cube map image. When coding a pixel block of an input image, a prediction search may be performed between the input pixel block and content of the padded reference image. When the prediction search identifies a match, the pixel block may be coded with respect to matching data from the padded reference image. Presence of replicated data in the padded reference image is expected to increase the likelihood that adequate prediction matches will be identified for input pixel block data, which will increase overall efficiency of the video coding.

BACKGROUND

The present disclosure relates to coding/decoding systems for multi-view imaging system and, in particular, to use of coding techniques that originally were developed for flat images, for multi-view image data.

Video coding system typically reduced bandwidth of video signals by exploiting spatial and/or temporal redundancy in video content. A given portion of input data (called a “pixel block” for convenience) is compared to a previously-coded image to identify similar content. If the search identifies an appropriate match, the input pixel block is coded differentially with respect to the matching data (a “reference block”) from the prior image. Many modern coding protocols, such as ITU-T H.265, H.264, H.263 and their predecessors, have been designed around these basic principles.

Such video coding protocols operate on an assumption that image data is “flat,” meaning that the image content represents a continuous two-dimensional field of view. Modern video systems are being developed, however, that do not operate under these assumptions.

Multi-view imaging is one application where image data is not flat. Images generated by a multi-view imaging system may represent image data in a two dimensional array of image data but spatial discontinuities may exist in image data contained within the image. Object motion that is relatively small in free space may be represented by large spatial movements within the image data that represents the object. Accordingly, modern coding systems may fail to recognize these instances of motion as an opportunity for differential coding. By failing to recognize such phenomena, such coding systems do not code image data as efficiently as they might.

Accordingly, the inventors recognized a need to improve coding system to accommodate motion effects that may arise in multi-view image data.

DETAILED DESCRIPTION

Embodiments of the present disclosure provide video coding/decoding techniques for cube map images. According to these techniques, padded reference images are generated for use during predicting input data. A reference image is stored in a cube map format. A padded reference image is generated from the reference image in which image data of a first view contained in reference image is replicated and placed adjacent to a second view contained in the cube map image. When coding a pixel block of an input image, a prediction search may be performed between the input pixel block and content of the padded reference image. When the prediction search identities a match, the pixel block may be coded with respect to matching data from the padded reference image. Presence of replicated data in the padded reference image is expected to increase the likelihood that adequate prediction matches will be identified for input pixel block data, which will increase overall efficiency of the video coding.

FIG. 1illustrates a system100in which embodiments of the present disclosure may be employed. The system100may include at least two terminals110-120interconnected via a network130. The first terminal110may have an image source that generates multi-view video. The terminal110also may include coding systems and transmission systems (not shown) to transmit coded representations of the multi-view video to the second terminal120, where it may be consumed. For example, the second terminal120may display the multi-view video on a local display, it may execute a video editing program to modify the multi-view video, or may integrate the multi-view into an application (for example, a virtual reality program), may present in head mounted display (for example, virtual reality applications) or it may store the multi-view video for later use.

FIG. 1illustrates components that are appropriate for unidirectional transmission of multi-view video, from the first terminal110to the second terminal120. In some applications, it may be appropriate to provide for bidirectional exchange of video data, in which case the second terminal120may include its own image source, video coder and transmitters (not shown), and the first terminal110may include its own receiver and display (also not shown). If it is desired to exchange multi-view video bi-directionally, then the techniques discussed hereinbelow may be replicated to generate a pair of independent unidirectional exchanges of multi-view video. In other applications, it would be permissible to transmit multi-view video in one direction (e.g., from the first terminal110to the second terminal120) and transmit “flat” video (e.g., video from a limited field of view) in a reverse direction.

InFIG. 1, the second terminal120is illustrated as a computer display but the principles of the present disclosure are not so limited. Embodiments of the present disclosure find application with laptop computers, tablet computers, smart phones, servers, media players, virtual reality head mounted displays, augmented reality display, hologram displays, and/or dedicated video conferencing equipment. The network130represents any number of networks that convey coded video data among the terminals110-120, including, for example, wireline and/or wireless communication networks. The communication network130may exchange data in circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks and/or the Internet. For the purposes of the present discussion, the architecture and topology of the network130is immaterial to the operation of the present disclosure unless explained hereinbelow.

FIG. 2is a functional block diagram of a coding system200according to an embodiment of the present disclosure. The system200may include an image source210, an image processing system220, a video coder230, a video decoder240, a reference picture store250, a predictor260, a padding unit270and, optionally, a pair of spherical transform units280.1,280.2. The image source210may generate image data as a multi-view image, containing image data of a field of view that extends around a reference point in multiple directions. The image processing system220may convert the image data from the image source210as needed to fit requirements of the video coder230. The video coder230may generate a coded representation of its input image data, typically by exploiting spatial and/or temporal redundancies in the image data. The video coder230may output a coded representation of the input data that consumes less bandwidth than the input data when transmitted and/or stored.

The video decoder240may invert coding operations performed by the video encoder230to obtain a reconstructed picture from the coded video data. Typically, the coding processes applied by the video coder230are lossy processes, which cause the reconstructed picture to possess various errors when compared to the original picture. The video decoder240may reconstruct picture of select coded pictures, which are designated as “reference pictures,” and store the decoded reference pictures in the reference picture store250. In the absence of transmission errors, the decoded reference pictures will replicate decoded reference pictures obtained by a decoder (not shown inFIG. 2).

The predictor260may select prediction references for new input pictures as they are coded. For each portion of the input picture being coded (called a “pixel block” for convenience), the predictor260may select a coding mode and identify a portion of a reference picture that may serve as a prediction reference search for the pixel block being coded. The coding mode may be an inter-coding mode, in which case the prediction reference may be drawn from a previously-coded (and decoded) portion of the picture being coded. Alternatively, the coding mode may be an inter-coding mode, in which case the prediction reference may be drawn from another previously-coded and decoded picture. The predictor260may operate in padded reference image data generated by the padding unit270as described herein.

In an embodiment, the predictor260may search for prediction references of pictures being coded operating on input picture and reference picture that has been transformed to a spherical projection representation. The spherical transform units280.1,280.2may transform the input picture and the reference picture to the spherical projection representations.

When an appropriate prediction reference is identified, the predictor260may furnish the prediction data to the video coder230. The video coder230may code input video data differentially with respect to prediction data furnished by the predictor260. Typically, prediction operations and the differential coding operate on a pixel block-by-pixel block basis. Prediction residuals, which represent pixel-wise differences between the input pixel blocks and the prediction pixel blocks, may be subject to further coding operations to reduce bandwidth further.

As indicated, the coded video data output by the video coder230should consume less bandwidth than the input data when transmitted and/or stored. The coding system200may output the coded video data to an output device290, such as a transmitter (not shown) that may transmit the coded video data across a communication network130(FIG. 1) or a storage device (also not shown) such as an electronic-, magnetic- and/or optical storage medium.

FIG. 3illustrates a cube map image300suitable for use with embodiments of the present invention. As indicated, an omnidirectional camera may capture image data in several fields of view, representing a “front” view,310a “left” view320, a “back” view330, a “right” view340, a “top” view350and a “bottom” view,360respectively. Image data of these views310-360may be assembled into an M×N pixel image according to the spatial relationships that exist among the different fields of view.

FIG. 3(a)illustrates orientation of the views310-360in the larger cube map image300.FIG. 3(b)illustrates orientation of the views310-360about a camera that captures images corresponding to these views310-360. For convenience, the image data captured for each of these fields of view will be described as “views”310-360when describing content of the cube map image300.

FIG. 3(c)is an exploded view of the views' spatial orientation, illustrating edges312-318,322-326,332-336,342-344that occur between the views310-360. Thus, as illustrated inFIG. 3(b), image content from the front view310that is immediately adjacent to edge312is spatially adjacent to pixel content from the left view320that also is immediately adjacent to edge312. Similarly, pixel content from the front view310that is immediately adjacent to edge314is spatially adjacent to pixel content from the right view340that also is immediately adjacent to edge314. Pixel content from the front view310that is immediately adjacent to edges316and318are spatially adjacent to pixel content from the top view350and the bottom view360, respectively, that are immediately adjacent to those edges.

The views310-360may be arranged in the cube map image300to retain continuity across some of the boundaries between the views310-360. As illustrated inFIG. 3(a), image continuity may be maintained between the front view310and the neighboring left, top and bottom views320,350and360along their respective edges312,316and318. Image continuity may be maintained between the left view320and the front and back views310,330, respectively, along edges312and322. Image continuity may be maintained between the back view330and the left and right views320,340respectively along edges322and332.

Image continuity is not maintained, however, across edges314,324,326,334,336,342,344. Thus, image content from the views310-360that are adjacent to these edges will not be in proximity to each other even though they represent adjacent image content. For example, although content from the front view310and the right view340that are adjacent to edge314are adjacent to each other spatially as illustrated inFIG. 3(c), they appear along opposite edges of the cube map image300illustrated inFIG. 3(a). Similarly, image content along the edges324,336and344of the top view350are distant from their counterparts along the edges324,336and344of the left view320, the back view330and the right view340, respectively. Moreover, image content along the edges326,334and342of the bottom view360are distant from their counterparts along the edges326,334and342of the left view320, the back view330and the right view340, respectively.

FIG. 4illustrates a method according to an embodiment of the present disclosure. The method400may process reference pictures arranged in a cube map image format such as shown inFIG. 3(a). For each candidate reference picture, the method400may create padded images in null regions of the source cube map image (box410). The method400also may perform a motion prediction search for an input pixel block across the padded image generated at box410(box420). The method400may determine whether the prediction search generates a match (box430) and, if so, the method400may code the input pixel block predictive using a matching reference block that is identified from the motion prediction search (box440). Otherwise, the method400may code the input pixel block by an alternate technique, such as by intra coding.

FIG. 5illustrates a padded cube map image500according to an embodiment of the present disclosure. The padded cube map image500may include image data from the front, left, back, right, top and bottom views310-360that are generated from creation of a source cube map image, as inFIG. 3(a). Regions of the cube map image300that were null regions370.1,370.2, shown inFIG. 3(a), may contain image data from the views that border the edges324,326,334,336,342and344as necessary to develop continuous image content across those edges. Thus, in the case of null region370.1(FIG. 3), image content of the top view350may be placed as padded images510,520and530and each instance of the top view350may be rotated to align its edges with the edges324,336and344of the left view320, the back view330and the right view340. Similarly, in the case of null region370.2(FIG. 3), image content of the bottom view360may be placed as padded images540,550and560, an each instance of the bottom image360may be rotated to align its edges with the edges326,334and342of the left view320, the back view330and the right view340. InFIG. 5, text of the padded images510-560illustrate rotations of image data that may occur to align data to these edges324,326,334,336,342and344.

Provision of padded images increases likelihood that predictive video coders may detect movement of image content across images. Consider an object illustrated inFIG. 3in the left view320at location Loc1. Image content of the object may have moved from a location Loc2in a top view350in a previously-coded reference frame. Therefore, the image content of the object at location Loc2in the top view350may serve as a prediction reference for the object at location Loc1. In practice, however, a video coder that searches for a prediction match for an object at location Loc1in a frame being coded may not detect the image content at location Loc2of a reference frame, due either to the object's distance from location Loc1in the equirectangular image300, to its orientation, or both.

With use of padded images as illustrated inFIG. 5, a redundant copy of the object may be provided at a location Loc3in a reference frame. The image content of the top view350appears in the padded view510in an orientation that adjoins image content of the top view350at edge324with image content of the left view320at edge324. The location and orientation of the image content at location Loc3is must closer to the object content being coded (at location Loc1) and, therefore, it has a higher likelihood of serving as a basis of prediction by a predictive video coder.

The principles of the present invention find application with cube map image of a variety of formats. Another format is illustrated inFIG. 6, which illustrates front, left, back, right, top and bottom views610-660in an alternative representation with four null regions670.1-670.4(FIG. 6(a)). Here, padded images680.1-680.6may be provided in the null regions670.1-670.4which are drawn from respective ones of the views610-660(FIG. 6(b)). In this example, the padded images680.1and680.4may be derived from the right view640and the padded images680.2-680.3and680.5-680.6may be derived from the top view650.

Returning toFIG. 5, it can be seen that use of padded images does not create image continuity across all edges. For example, with respect to the top view350, continuity is not maintained across edges324,344or336. The padded image510contains data of the top view which does not create continuity across the edge324(even though it does create continuity across the counterpart edge324at the left view320). Further, there is no image data at edges336and344, which represent boundary edges of the image500. Similarly, with respect to the bottom view360, continuity is not maintained across edges326,334or342. The padded image540contains data of the bottom image which does not create continuity across the edge326(even though it does create continuity across the counterpart edge326at the left view320). Further, there is no image data at edges334and342, which represent boundary edges of the image500. Accordingly, prediction searches likely would not identify matches across such boundaries and, optionally, may be constrained to avoid searching across edges324,326,334,336,342,344having discontinuities in image content after padding is applied.

In a further embodiment, a reference image may be expanded by padding about a periphery of the image. Thus, a reference image that is processed by video encoders and decoders as an M×N pixel image may be expanded by amounts ΔM and ΔN, respectively, along a periphery of the image, yielding a (M+2ΔM)×(N+2ΔN) image. Padded image data may be provided along peripheral edges of the M×N pixel image to provide padded image data along edges of the views310,340,350,360at the periphery. Such padded image data may be drawn from the views that abut the peripheral edges in the cube map view. For example, right view data may be provided along a peripheral edge of the front view310and front view data may be provided along a peripheral edge of the right view340. Thus, prediction searches may extend from peripheral edges of the M×N image into the padded regions provided by the ΔM and/or ΔN expansion.

FIG. 7illustrates a method700according to another embodiment of the present disclosure. The method700may be performed for each pixel block of a cube map image being coded. The method700may identify a view associated with a pixel block being coded (box710). Then, for each candidate reference picture that may serve as a prediction reference for the input pixel block, the method700may create a padded reference image using image data from views that are adjacent to the view identified in box710(box720). The method700may perform a motion prediction search730within the padded reference image created at box720(box730). After consideration of the candidate reference pictures, the method700may determine if a prediction search yielded a match (box740). If so, the method700may code the input pixel block predictive using a matching reference block that is identified from the motion prediction search (box750). Otherwise, the method700may code the input pixel block by an alternate technique, such as by intra coding.

FIG. 8illustrates an exemplary cube map image800that may be coded by the method ofFIG. 7.FIG. 8(a)illustrates the cube map image800having front, left, back, right, top and bottom views810-880that are partitioned respectively into pixel blocks.FIG. 8(b)illustrates a padded reference image870that may be generated when a pixel block PB1is coded from a top view850andFIG. 8(c)illustrates a padded reference image880that may be generated when a pixel block PB1is coded from a back view830.

Referring toFIG. 8(b), when a pixel block PB1from a top view850of an input image800is coded, the method700may generate a padded reference image870that includes image data from the top view872of the reference picture and padded images874.1-874.4provided along edges of the top view872. In this instance, the padded images874.1-874.4respectively contain image data of the front view874.1of the reference image, the left view874.2of the reference image, the back view874.3of the reference image and the right view874.4of the reference image. The image data of these views874.1-874.4each may be rotated to provide continuity of image data across edges of the top view872.

The padded reference image870may provide continuous reference picture data along all edges of the view850in which a pixel block PB1is coded. Thus, when coding a pixel block PB1, a video coding system may search for prediction references across edges of the view850in which the pixel block PB1is located.

Similarly, referring toFIG. 8(c), when a pixel block PB2from a back view830of an input image800is coded, the method700may generate a padded reference image880that includes image data from the back view882of the reference picture and padded images884.1-884.4provided along edges of the back view882. In this instance, the padded images884.1-884.4respectively contain image data of the bottom view884.1of the reference image, the right view884.2of the reference image, the top view884.3of the reference image and the left view884.4of the reference image. The image data of these views884.1-884.4each may be rotated to provide continuity of image data across edges of the top view882.

The padded reference image880may provide continuous reference picture data along all edges of the view830in which a pixel block PB2is coded. Thus, when coding a pixel block PB2, a video coding system may search for prediction references across edges of the view80in which the pixel block PB2is located.

The operation of method700may be repeated for pixel blocks of each of the views810-860of an image800being coded.

FIGS. 8(b) and 8(c)each illustrate respective null regions876.1-876.4and886.1-886.4provided in areas between instances of padded image data874.1-874.4and884.1-884.4. In an embodiment, it is unnecessary to provide image data in these null regions. Alternatively, however, it is permissible to replicate padded image data from an adjacent image. For example, null region876.3is adjacent to padded images847.1and847.4; one of the padded images may be replicated in the null region876.3, if desired.

AlthoughFIG. 7illustrates the creation of padded images (box720) may be performed anew for each pixel block being coded, in practice, the creation of a padded image may be performed once and reused for coding all pixel blocks within a given view. Thus, when coding pixel blocks in a top view850of an input image800, a single instance of the padded reference image870may be created for use in coding all pixel blocks from the top view850. Similarly, when coding pixel blocks in a back view830of an input image800, a single instance of the padded reference image880may be created for use in coding all pixel blocks from the back view830.

Moreover, it is not required to use all image data of a given view when building a padded reference image. Instead, is it sufficient to provide a portion of padded image data sufficient to develop image data in a region that corresponds to a search window of the motion prediction search being performed. For example,FIG. 8(a)illustrates an exemplary search window SW provided around pixel block PB1in the top view850of the image800being coded. It is sufficient to develop a padded reference image having data sufficient to cover a region corresponding to a union of the search windows for all pixel blocks of a given view (such as view850). Thus, a padded reference image may be obtained from image data from a reference image corresponding to a co-located view as the pixel block being coded and portions of images adjacent to the co-located view. InFIG. 8(b), a top view872of the reference image is co-located with the view850in which PB1resides and portions of the front, left, back and right views from the reference image may be used to build a padded reference image870that is co-extensive with a union of the search windows for all pixel blocks of the view850. It would not be necessary to use the entirety of the front, left, back and right views from the reference image if the search windows around pixel blocks in the top view850(FIG. 8(a)) cannot reach them.

The method700ofFIG. 7may find application with cube map image data in alternate formats. For example,FIG. 9(a)illustrates a cube map image900having a layout that avoids use of null regions. In this example, the cube map image900contains a front view,910a left view920, a back view930, a right view940, a top view950and a bottom view960respectively, which are developed from fields of view illustrated inFIG. 9(b). The views910-960may be laid out in the image in a regular array, such as the 3×2 array illustrated inFIG. 9(a). In doing so, however, the cube map image900introduces additional discontinuities along view edges that might have been avoided in a different layout (such as the layouts illustrated inFIGS. 3 and 6).

In the example ofFIG. 9, the front, left and back views910,920,930are arranged to preserve image continuity across edges912|928and922|936. Similarly the right, top and bottom views are arranged to preserve image continuity across edges946|954and942|962.

Discontinuities are developed at seams between the front and bottom views910,960, between the left and right views920,940, and between the top and back views930,950. For example, where the front and bottom views910,960meet in the cube map image900, edges916and968are placed adjacent to each other even though they are not adjacent in free space (represented byFIG. 9(b)). Similarly, where the left and right views920,940meet in the cube map image, edges924and944are placed adjacent to each other even though they are not adjacent to each other in free space. And, further, where the back and top views930,950meet in the cube map image900, the edges938and952are placed adjacent to each other but oriented differently (the top view is flipped) from their orientation in free space. These discontinuities are illustrated with dashed lines inFIG. 9(a)where seams between image views that are continuous are represented with solid lines.

Using the technique ofFIG. 7, padded reference images may be developed for the views of cube map image such as illustrated inFIG. 9. When coding pixel block data from a top view950of a cube map image900, padded reference images may be derived from a top view of a reference picture and from padded images derived from front, left, back and right images as illustrated inFIG. 8(b). Similarly, when coding pixel block data from a back view930of a cube map image900, padded reference images may be derived from a back view of a reference picture and from padded images derived from bottom, right, top and left images of the reference picture as illustrated inFIG. 8(c).

In an embodiment, image transformation may be performed on padded image data prior to a motion prediction search. Such transformations may be performed to project image data from the padded image to a domain of the view to which the padded image data is appended.

FIG. 10illustrates one such projection according to an embodiment of the present disclosure. As illustrated inFIG. 10(a), it is possible that image data of an object will appear in multiple views of a cube map image1000. For example, image data of an object Obj (FIG. 10(b)) is illustrated as appearing in both a right view1010and a top view1020of a cube map image1000. Owing to different perspectives of the image sensor(s) that capture image data of these views1010,1020, the object may appear with distortion if the right and top views1010,1020were treated as a single, “flat” image. In an embodiment, padded image data may be subject to transform to counter-act the distortion that arises due to differences among the fields of view.

FIG. 10(c)schematically illustrates operation of a transform according to an embodiment of the present disclosure. In this embodiment, it may be assumed that padded image data from a top view1020is generated for placement adjacent to image data from a right view1010. In this embodiment, a projection of image data from the top view1020is estimated as it appears in a plane of the right view1030. For example, the object Obj (FIG. 10(a)) may be estimated to have a length 11 in the top view. This length occupies an angle α measured from a hypothetical center of the views of the cube map image. From the angle α, a length 12 of the object as it appears in a plane of the right view1010may be derived. Thus, padded image data1030may be developed (FIG. 10(d)) that counter-acts image distortion that may arise from different perspectives of the fields of view and provides improved continuity in image data for prediction purposes.

The principles of the present invention also find application with equirectangular images in spherical projection format.FIG. 11illustrates an application of padding data used with spherically projected image data.FIG. 11(a)illustrates image data of a first view1110in a flat projection andFIG. 11(b)illustrates image data1120of theFIG. 11(a)view transformed according to a spherical projection. Such transforms are common, for example, when mapping data from a top view of an omnidirectional camera to an equirectangular image. Essentially, the view1110may represent data of a “north pole” of an image space.

FIGS. 11(c) and (d)represent an exemplary reference image according to a flat image format (reference number1130) and a spherical projection (reference number1140). During video coding, image data of the spherically projected reference image1140may serve as a prediction reference for a new image, represented by spherically projected image1120. It may occur that, due to the spherical projection of image data, fairly modest changes of motion of data in the flat domain (for example, between pixel blocks1150and1152) may induce large displacements in an equirectangular image, illustrated by motion vector mv inFIG. 11(d).

Image padding, shown inFIG. 11(e), can replicate prediction data along a periphery of the equirectangular image. In the example ofFIG. 11(e), a padded reference image is created by duplicating the content of the reference image1140along its edge1142(FIG. 11(d)), flipping the duplicated image and placing it adjacent to the edge1142. In this manner, the padded reference image creates continuity in image content along the edge1142, which can create shorter motion vectors during prediction searches and thereby lead to improved efficiency in coding.

FIG. 12illustrates a method1200according to an embodiment of the present disclosure. The method1200predicts a search window for a pixel block of an equirectangular image according to motion vectors of previously-coded pixel blocks from the same image. The method1200may project motion vectors of the previously-coded pixel blocks from a domain of the equirectangular image to a spherical domain (box1210). The method1200may estimate a search window of a new pixel block to be coded from the spherically-projected motion vectors of the previously-coded pixel blocks (box1220). The method1200may transform the search window from the spherical projection back to the equirectangular projection of the input image (box1230). Thereafter, the method1200perform a prediction search for a reference within the transformed search window (box1240).

FIG. 13illustrates an exemplary equirectangular image1300that might be processed by the method1200ofFIG. 12. At the time a pixel block1310is coded, other pixel blocks1320,1330from the image1300may already be coded and, thus, motion vectors mv1, mv2may be defined for the coded pixel blocks1320,1330(FIG. 13(a)). These motion vectors mv1, mv2may be projected to a spherical domain1350(FIG. 13(b)). In many instances, the motion vectors mv1, mv2may refer to a co-located region of image content in a spherical projection (FIG. 13(b)) even though the motion vectors mv1, mv2do not refer to co-located regions in an equirectangular format. A search window may be derived from the motion vectors in the spherical projection, for example, by averaging the motion vectors and defining a search region of predetermined size about the resultant vector obtained therefrom. Thereafter, the search window may be transformed back to the domain of the equirectangular image1300.

Transforms between the equirectangular format to the spherical projection may be performed according to the techniques described in co-pending application Ser. No. 15/390,202, filed Dec. 23, 2016, the disclosure of which is incorporated herein.

FIG. 14is a functional block diagram of a coding system1400according to an embodiment of the present disclosure. The system1400may include a pixel block coder1410, a pixel block decoder1420, an in-loop filter system1430, a reference picture store1440, a padding unit1450, a predictor1460, a controller1470, and a syntax unit1480. The padding unit1450may generate padded image data according to one or more of the embodiments of the foregoing discussion. The pixel block coder and decoder1410,1420and the predictor1460may operate iteratively on individual pixel blocks of a picture. The predictor equirectangular1460may predict data for use during coding of a newly-presented input pixel block. The pixel block coder1410may code the new pixel block by predictive coding techniques and present coded pixel block data to the syntax unit1480. The pixel block decoder1420may decode the coded pixel block data, generating decoded pixel block data therefrom. The in-loop filter1430may perform various filtering operations on a decoded picture that is assembled from the decoded pixel blocks obtained by the pixel block decoder1420. The filtered picture may be stored in the reference picture store1440where it may be used as a source of prediction of a later-received pixel block. The syntax unit1480may assemble a data stream from the coded pixel block data which conforms to a governing coding protocol.

The pixel block coder1410may include a subtractor1412, a transform unit1414, a quantizer1416, and an entropy coder1418. The pixel block coder1410may accept pixel blocks of input data at the subtractor1412. The subtractor1412may receive predicted pixel blocks from the predictor1460and generate an array of pixel residuals therefrom representing a difference between the input pixel block and the predicted pixel block. The transform unit1414may apply a transform to the sample data output from the subtractor1412, to convert data from the pixel domain to a domain of transform coefficients. The quantizer1416may perform quantization of transform coefficients output by the transform unit1414. The quantizer1416may be a uniform or a non-uniform quantizer. The entropy coder1418may reduce bandwidth of the output of the coefficient quantizer by coding the output, for example, by variable length code words.

The transform unit1414may operate in a variety of transform modes as determined by the controller1470. For example, the transform unit1414may apply a discrete cosine transform (DCT), a discrete sine transform (DST), a Walsh-Hadamard transform, a Haar transform, a Daubechies wavelet transform, or the like. In an embodiment, the controller1470may select a coding mode M to be applied by the transform unit1415, may configure the transform unit1415accordingly and may signal the coding mode M in the coded video data, either expressly or impliedly.

The quantizer1416may operate according to a quantization parameter QPthat is supplied by the controller1470. In an embodiment, the quantization parameter QPmay be applied to the transform coefficients as a multi-value quantization parameter, which may vary, for example, across different coefficient locations within a transform-domain pixel block. Thus, the quantization parameter QPmay be provided as a quantization parameters array.

The pixel block decoder1420may invert coding operations of the pixel block coder1410. For example, the pixel block decoder1420may include a dequantizer1422, an inverse transform unit1424, and an adder1426. The pixel block decoder1420may take its input data from an output of the quantizer1416. Although permissible, the pixel block decoder1420need not perform entropy decoding of entropy-coded data since entropy coding is a lossless event. The dequantizer1422may invert operations of the quantizer1416of the pixel block coder1410. The dequantizer1422may perform uniform or non-uniform de-quantization as specified by the decoded signal QP. Similarly, the inverse transform unit1424may invert operations of the transform unit1414. The dequantizer1422and the inverse transform unit1424may use the same quantization parameters QPand transform mode M as their counterparts in the pixel block coder1410. Quantization operations likely will truncate data in various respects and, therefore, data recovered by the dequantizer1422likely will possess coding errors when compared to the data presented to the quantizer1416in the pixel block coder1410.

The adder1426may invert operations performed by the subtractor1412. It may receive the same prediction pixel block from the predictor1460that the subtractor1412used in generating residual signals. The adder1426may add the prediction pixel block to reconstructed residual values output by the inverse transform unit1424and may output reconstructed pixel block data.

The in-loop filter1430may perform various filtering operations on recovered pixel block data. For example, the in-loop filter1430may include a deblocking filter1432and a sample adaptive offset (“SAO”) filter1433. The deblocking filter1432may filter data at seams between reconstructed pixel blocks to reduce discontinuities between the pixel blocks that arise due to coding. SAO filters may add offsets to pixel values according to an SAO “type,” for example, based on edge direction/shape and/or pixel/color component level. The in-loop filter1430may operate according to parameters that are selected by the controller1470.

The reference picture store1440may store filtered pixel data for use in later prediction of other pixel blocks. Different types of prediction data are made available to the predictor1460for different prediction modes. For example, for an input pixel block, intra prediction takes a prediction reference from decoded data of the same picture in which the input pixel block is located. Thus, the reference picture store1440may store decoded pixel block data of each picture as it is coded. For the same input pixel block, inter prediction may take a prediction reference from previously coded and decoded picture(s) that are designated as reference pictures. Thus, the reference picture store1440may store these decoded reference pictures.

The padding unit1450may generate padded image data as discussed in the foregoing embodiments. Thus, the padding unit may perform the operations illustrated inFIGS. 4-12to generate padded image data from which the predictor1460may select prediction references.

As discussed, the predictor1460may supply prediction data to the pixel block coder1410for use in generating residuals. The predictor1460may include an inter predictor1462, an intra predictor1463and a mode decision unit1462. The inter predictor1462may receive spherically-projected pixel block data representing a new pixel block to be coded and may search spherical projections of reference picture data from store1440for pixel block data from reference picture(s) for use in coding the input pixel block. The inter predictor1462may support a plurality of prediction modes, such as P mode coding and B mode coding. The inter predictor1462may select an inter prediction mode and an identification of candidate prediction reference data that provides a closest match to the input pixel block being coded. The inter predictor1462may generate prediction reference metadata, such as motion vectors, to identify which portion(s) of which reference pictures were selected as source(s) of prediction for the input pixel block.

The intra predictor1463may support Intra (I) mode coding. The intra predictor1463may search from among spherically-projected pixel block data from the same picture as the pixel block being coded that provides a closest match to the spherically-projected input pixel block. The intra predictor1463also may generate prediction reference indicators to identify which portion of the picture was selected as a source of prediction for the input pixel block.

The mode decision unit1462may select a final coding mode to be applied to the input pixel block. Typically, as described above, the mode decision unit1462selects the prediction mode that will achieve the lowest distortion when video is decoded given a target bitrate. Exceptions may arise when coding modes are selected to satisfy other policies to which the coding system1400adheres, such as satisfying a particular channel behavior, or supporting random access or data refresh policies. When the mode decision selects the final coding mode, the mode decision unit1462may output a non-spherically-projected reference block from the store1440to the pixel block coder and decoder1410,1420and may supply to the controller1470an identification of the selected prediction mode along with the prediction reference indicators corresponding to the selected mode.

The controller1470may control overall operation of the coding system1400. The controller1470may select operational parameters for the pixel block coder1410and the predictor1460based on analyses of input pixel blocks and also external constraints, such as coding bitrate targets and other operational parameters. As is relevant to the present discussion, when it selects quantization parameters QP, the use of uniform or non-uniform quantizers, and/or the transform mode M, it may provide those parameters to the syntax unit1480, which may include data representing those parameters in the data stream of coded video data output by the system1400. The controller1470also may select between different modes of operation by which the system may generate padded reference images and may include metadata identifying the modes selected for each portion of coded data.

During operation, the controller1470may revise operational parameters of the quantizer1416and the transform unit1415at different granularities of image data, either on a per pixel block basis or on a larger granularity (for example, per picture, pet slice, per largest coding unit (“LCU”) or another region). In an embodiment, the quantization parameters may be revised on a per-pixel basis within a coded picture.

Additionally, as discussed, the controller1470may control operation of the in-loop filter1430and the prediction unit1460. Such control may include, for the prediction unit1460, mode selection (lambda, modes to be tested, search windows, distortion strategies, etc.), and, for the in-loop filter1430, selection of filter parameters, reordering parameters, weighted prediction, etc.

In an embodiment, the predictor1460may perform prediction searches using input pixel block data and reference pixel block data in a spherical projection. Operation of such prediction techniques are described in U.S. patent application Ser. No. 15/390,202, filed Dec. 23, 2016 and assigned to the assignee of the present application. In such an embodiment, the coder1400may include a spherical transform unit1490that transforms input pixel block data to a spherical domain prior to being input to the predictor1460. The padding unit1450may transform reference picture data to the spherical domain (in addition to performing the transforms described hereinabove) prior to being input to the predictor1460.

A video parameter set syntax may be modified by adding a new field, shown below as “vps_projection_format_id,” to the as video_parameter_set_rbsp as follows:

The projection format may be signaled in a sequence parameter set (seq_parameter_set_rbsp( )) as follows:

By way of example, the projection-format-id might take the following values:

Additionally, the cube_map_packing_id may be signaled as follows:

Further, the reference_padding_mode field may be coded to identify different transforms applied by an encoder. For example, if reference_padding_mode were set to “0,” it may indicate that no transform were used. If reference_padding_mode were set to “1,” it may indicate that transforms were performed according toFIG. 14. Here again, the number of codes may be expanded as necessary to accommodate other transformations.

FIG. 15is a functional block diagram of a decoding system1500according to an embodiment of the present disclosure. The decoding system1500may include a syntax unit1510, a pixel block decoder1520, an in-loop filter1530, a reference picture store1140, a padding unit1550, a predictor1560, and a controller1570. The syntax unit1510may receive a coded video data stream and may parse the coded data into its constituent parts. Data representing coding parameters may be furnished to the controller1570while data representing coded residuals (the data output by the pixel block coder1110ofFIG. 11) may be furnished to the pixel block decoder1520. The pixel block decoder1520may invert coding operations provided by the pixel block coder1110(FIG. 11). The in-loop filter1530may filter reconstructed pixel block data. The reconstructed pixel block data may be assembled into pictures for display and output from the decoding system1500as output video. The pictures also may be stored in the prediction buffer1540for use in prediction operations. The padding unit1550may generate padded reference images based on metadata contained in the coded data as described in the foregoing discussion. The predictor1560may supply prediction data to the pixel block decoder1520as determined by coding data received in the coded video data stream.

The pixel block decoder1520may include an entropy decoder1522, a dequantizer1524, an inverse transform unit1526, and an adder1528. The entropy decoder1522may perform entropy decoding to invert processes performed by the entropy coder1118(FIG. 11). The dequantizer1524may invert operations of the quantizer1116of the pixel block coder1110(FIG. 11). Similarly, the inverse transform unit1526may invert operations of the transform unit1114(FIG. 11). They may use the quantization parameters QPand transform modes M that are provided in the coded video data stream. Because quantization is likely to truncate data, the data recovered by the dequantizer1524, likely will possess coding errors when compared to the input data presented to its counterpart quantizer1116in the pixel block coder1110(FIG. 11).

The adder1528may invert operations performed by the subtractor1111(FIG. 11). It may receive a prediction pixel block from the predictor1560as determined by prediction references in the coded video data stream. The adder1528may add the prediction pixel block to reconstructed residual values output by the inverse transform unit1526and may output reconstructed pixel block data.

The in-loop filter1530may perform various filtering operations on reconstructed pixel block data. As illustrated, the in-loop filter1530may include a deblocking filter1532and an SAO filter1534. The deblocking filter1532may filter data at seams between reconstructed pixel blocks to reduce discontinuities between the pixel blocks that arise due to coding. SAO filters1534may add offset to pixel values according to an SAO type, for example, based on edge direction/shape and/or pixel level. Other types of in-loop filters may also be used in a similar manner. Operation of the deblocking filter1532and the SAO filter1534ideally would mimic operation of their counterparts in the coding system1100(FIG. 11). Thus, in the absence of transmission errors or other abnormalities, the decoded picture obtained from the in-loop filter1530of the decoding system1500would be the same as the decoded picture obtained from the in-loop filter1150of the coding system1100(FIG. 11); in this manner, the coding system1100and the decoding system1500should store a common set of reference pictures in their respective reference picture stores1140,1540.

The reference picture stores1540may store filtered pixel data for use in later prediction of other pixel blocks. The reference picture stores1540may store decoded pixel block data of each picture as it is coded for use in intra prediction. The reference picture stores1540also may store decoded reference pictures.

The padding unit1550may generate padded reference images based on metadata contained in the coded data as described in the foregoing discussion. Thus, the padding unit1550also may perform operations as described in the foregoingFIGS. 4-11to generate padded reference images on which the predictor1560may operate. In a decoder1500, the type of padded image data will be determined by metadata provided in coded image data identifying padding operations that were performed by an encoder. The padding unit1550may replicate the padding operations to generate padded reference image data that matches the padded reference image data generated by an encoder.

Of course, the padding unit1550need not perform padding operations unless prediction information associated with a coded pixel block references data in a padded region of a padded reference image. Referring toFIG. 8, if an encoder codes pixel block PB1using prediction data from a top view872of a padded reference image870, then the pixel block PB1does not rely on data from any of the padded images874.1-874.4. At a decoder, the padding unit1550need not perform operations to derive padded image data to decode the coded pixel block PB1. On the other hand, a different pixel block (say, PB2) may be coded using data from a padded image884.3(FIG. 8(c)). In this instance, the padding unit1550(FIG. 15) may develop padded image data corresponding to the reference data selected by an encoder. Thus, the decoder1500determines whether padded image data is referenced by prediction before generating padded image data for a given coded pixel block.

As discussed, the predictor1560may supply the transformed reference block data to the pixel block decoder1520. The predictor1560may supply predicted pixel block data as determined by the prediction reference indicators supplied in the coded video data stream. The predictor1560also may replicate the transform techniques described inFIGS. 12-13.

The controller1570may control overall operation of the coding system1500. The controller1570may set operational parameters for the pixel block decoder1520and the predictor1560based on parameters received in the coded video data stream. As is relevant to the present discussion, these operational parameters may include quantization parameters QPfor the dequantizer1524and transform modes M for the inverse transform unit1515. As discussed, the received parameters may be set at various granularities of image data, for example, on a per pixel block basis, a per picture basis, a per slice basis, a per LCU basis, or based on other types of regions defined for the input image.

The foregoing discussion has described operation of the embodiments of the present disclosure in the context of video coders and decoders. Commonly, these components are provided as electronic devices. Video decoders and/or controllers can be embodied in integrated circuits, such as application specific integrated circuits, field programmable gate arrays and/or digital signal processors. Alternatively, they can be embodied in computer programs that execute on camera devices, personal computers, notebook computers, tablet computers, smartphones or computer servers. Such computer programs typically are stored in physical storage media such as electronic-, magnetic- and/or optically-based storage devices, where they are read to a processor and executed. Decoders commonly are packaged in consumer electronics devices, such as smartphones, tablet computers, gaming systems, DVD players, portable media players and the like; and they also can be packaged in consumer software applications such as video games, media players, media editors, and the like. And, of course, these components may be provided as hybrid systems that distribute functionality across dedicated hardware components and programmed general-purpose processors, as desired.

For example, the techniques described herein may be performed by a central processor of a computer system.FIG. 16illustrates an exemplary computer system1600that may perform such techniques. The computer system1600may include a central processor1610, one or more cameras1620, a memory1630, and a transceiver1640provided in communication with one another. The camera1620may perform image capture and may store captured image data in the memory1630. Optionally, the device also may include sink components, such as a coder1650and a display1660, as desired.

The central processor1610may read and execute various program instructions stored in the memory1630that define an operating system1612of the system1600and various applications1616.1-1616.N. The program instructions may perform coding mode control according to the techniques described herein. As it executes those program instructions, the central processor1610may read, from the memory1630, image data created either by the camera1620or the applications1616.1-1616.N, which may be coded for transmission. The central processor1610may execute a program that operates according to the principles of FIG.6. Alternatively, the system1600may have a dedicated coder1650provided as a standalone processing system and/or integrated circuit.

As indicated, the memory1630may store program instructions that, when executed, cause the processor to perform the techniques described hereinabove. The memory1630may store the program instructions on electrical-, magnetic- and/or optically-based storage media.

The transceiver1640may represent a communication system to transmit transmission units and receive acknowledgement messages from a network (not shown). In an embodiment where the central processor1610operates a software-based video coder, the transceiver1640may place data representing state of acknowledgment message in memory1630to retrieval by the processor1610. In an embodiment where the system1600has a dedicated coder, the transceiver1640may exchange state information with the coder1650.

The foregoing description has been presented for purposes of illustration and description. It is not exhaustive and does not limit embodiments of the disclosure to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from the practicing embodiments consistent with the disclosure. Unless described otherwise herein, any of the methods may be practiced in any combination.