Source: http://patents.com/us-9973739.html
Timestamp: 2018-09-25 06:59:39
Document Index: 74327270

Matched Legal Cases: ['Application No. 09820314', 'art 3', 'Application No. 09', 'Application No. 09', 'Application No. 09', 'Application No. 200980151318', 'Application No. 3251', 'art 3', 'art 3']

US Patent # 9,973,739. Sharing of motion vector in 3D video coding - Patents.com
United States Patent 9,973,739
Chen; Ying (San Diego, CA), Hannuksela; Miska (Ruutana, FI)
Family ID: 1000003296354
13/124,641
PCT/FI2009/050834
WO2010/043773
US 20110216833 A1 Sep 8, 2011
61106503 Oct 17, 2008
Current CPC Class: H04N 19/103 (20141101); H04N 19/196 (20141101); H04N 19/30 (20141101); H04N 13/128 (20180501); H04N 19/597 (20141101); H04N 19/61 (20141101); H04N 19/513 (20141101); H04N 2213/003 (20130101)
Current International Class: H04B 1/66 (20060101); H04N 19/103 (20140101); H04N 19/30 (20140101); H04N 19/196 (20140101); H04N 19/61 (20140101); H04N 19/513 (20140101); H04N 19/597 (20140101); H04N 13/00 (20180101); H04N 11/04 (20060101); H04N 11/02 (20060101); H04N 7/12 (20060101)
Field of Search: ;375/240.16,240.25 ;345/421 ;348/42,207.99
7009606 March 2006 Hiraga et al.
2008/0089412 April 2008 Ugur et al.
2010/0020871 January 2010 Hannuksela et al.
2007/047736 Apr 2007 WO
2009/091383 Jul 2009 WO
International Search Report received for corresponding Patent Cooperation Treaty Application No. PCT/FI2009/050834, dated Mar. 23, 2010, 17 pages. cited by applicant .
International Preliminary Report on Patentability received for corresponding Patent Cooperation Treaty Application No. PCT/FI2009/050834, dated Apr. 28, 2011, 10 pages. cited by applicant .
Wu et al., "Improved methods for object-based coding of plenoptic videos", Proceedings of 2005 IEEE International Symposium on Intelligent Signal Processing and Communication Systems, Dec. 13-16, 2005, pp. 481-484. cited by applicant .
Merkle et al., "Efficient compression of multi-view depth data based on MVC", IEEE Proceedings of 3DTV Conference 2007, May 7-9, 2007, 4 pages. cited by applicant .
Tsung et al., "Multiview video hybrid coding system with texture-depth synthesis", Proceedings of 2008 IEEE International Conference on Multimedia and Expo (ICME 2008), Jun. 23-26, 2008, pp. 1581-1584. cited by applicant .
Pourazad et al., "An H.264-based video encoding scheme for 3DTV", Proceedings of 14th European Signal Processing Conference (EUSIPCO 2006), Sep. 4-8, 2006, 5 pages. cited by applicant .
Oh et al., "H.264-Based Depth Map Sequence Coding UsingMotion Information of Corresponding Texture Video", Advances in Image and Video Technology, Lecture Notes in Computer Science, 2006, pp. 898-907. cited by applicant .
Grewatsch et al., "Sharing of motion vectors in 3D video coding", Proceedings of 2004 IEEE International Conference on Image Processing (ICIP), Oct. 24-27, 2004, pp. 3271-3274. cited by applicant .
Redert, et al., "ATTEST: Advanced Three-Dimensional Television System Technologies", Proceedings of the First International Symposium on 3D Data Processing Visualization and Transmission (3DPVT'02), Jun. 19-21, 2002, pp. 313-319. cited by applicant .
Hewage et al., "Comparison of Stereo Video Coding Support in MPEG-4 MAC, H.264/AVC and H.264/SVC", Proceedings of the 4th IET International Conference on Visual Information Engineering (VIE'2007), Jul. 25-27, 2007, pp. 1-6. cited by applicant .
Extended European Search Report received for corresponding European Patent Application No. 09820314.4, dated Jul. 12, 2012, 7 pages. cited by applicant .
Weigand et al., "Joint Draft ITU-T Rec. H.264 |0 ISO/IEC 14496-10 / Amd.3 Scalable video coding", Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG (ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6), JVT-X201, 24th Meeting, Jun. 29-Jul. 5, 2007, 563 pages. cited by applicant .
Vetro et al., "Joint Draft 7.0 on Multiview Video Coding", Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG (ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6), JVT-AA209, 27th Meeting, Apr. 23-29, 2008, 55 pages. cited by applicant .
Series H: Audiovisual and Multimedia Systems Infrastructure of Audiovisual Services--Coding of Moving Video, Advanced Video Coding for Generic Audiovisual Services, H.264, ITU-T, Telecommunication Standardization Sector of ITU, Nov. 2007, pp. 1-564. cited by applicant .
Bourge et al., "ISO/IEC CD 23002-3 Auxiliary Video Data Representation", Video Group, ISO/IEC JTC 1/SC 29/WG 11/N8038, Information technology--MPEG video technologies--Part 3: Auxiliary video data representation, Apr. 11, 2006, pp. 1-.23. cited by applicant .
Vetro et al., "Joint Multiview Video Model (JMVM) 8.0", Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG (ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6), JVT-AA207, 27th Meeting, Apr. 23-29, 2008, pp. 1-20. cited by applicant .
Vetro et al., "Joint Draft 8.0 on Multiview Video Coding", Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG (ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6), JVT-AB204, 28th Meeting, Jul. 20-25, 2008, pp. 1-63. cited by applicant .
Ophir et al., "Wi-Fi (IEEE 802.11) and Bluetooth Coexistence: Issues and Solutions", 2004, pp. 847-852. cited by applicant .
Office Action for European Application No. 09 820 314.4 dated Nov. 14, 2013. cited by applicant .
Office Action for European Application No. 09 820 314.4 dated Mar. 21, 2013. cited by applicant .
Office Action for European Application No. 09 820 314.4 dated Jun. 6, 2014. cited by applicant .
Office Action for Chinese Application No. 200980151318.7 dated Feb. 19, 2014. cited by applicant .
Office Action for India Application No. 3251/CHENP/2011 dated Dec. 26, 2017, 6 pages. cited by applicant.
Assistant Examiner: Mahmud; Farhan
1. A method for encoding a bitstream including a first texture picture, a first depth map picture associated with the first texture picture, a second texture picture, and a second depth map picture associated with the second texture picture, wherein the first depth map picture belongs to a first view and the second depth map picture belongs to a second view and the first depth map picture and the second depth map picture are encoded as auxiliary pictures coded independently of the corresponding texture pictures, the method comprising: predicting the second depth map picture from the first depth map picture using a first motion vector; predicting the second texture picture from the first texture picture using a second motion vector; and encoding the first motion vector and the second motion vector into a bitstream.
3. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and computer program code configured to, with the processor, cause the apparatus to: encode a bitstream including a first texture picture, a first depth map picture associated with the first texture picture, a second texture picture, and a second depth map picture associated with a second texture picture, wherein the first depth map picture belongs to a first view and the second depth map picture belongs to a second view and the first depth map picture and the second depth map picture are encoded as auxiliary pictures coded independently of the corresponding texture pictures; predict the second depth map picture from the first depth map picture using a first motion vector; predict the second texture picture from the first texture picture using a second motion vector; and encode the first motion vector and the second motion vector into a bitstream.
4. A method of decoding a bitstream including a first texture picture, a first depth map picture associated with the first texture picture, a second texture picture, and a second depth map picture associated with the second texture picture, wherein the first depth map picture belongs to a first view and the second depth map picture belongs to a second view and the first depth map picture and the second depth map picture are auxiliary pictures coded independently of the corresponding texture pictures, the method comprising: decoding a first motion vector from a bitstream; decoding a second motion vector from the bitstream; decoding the second depth map picture, wherein the first motion vector is used to predict the second depth map picture from the first depth map picture; and decoding the second texture picture, wherein the second motion vector is used to predict the second texture picture from the first texture picture.
6. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and computer program code configured to, with the processor, cause the apparatus to: decode a bitstream including a first texture picture, a first depth map picture associated with the first texture picture, a second texture picture, and a second depth map picture associated with a second texture picture, wherein the first depth map picture belongs to a first view and the second depth map picture belongs to a second view and the first depth map picture and the second depth map picture are auxiliary pictures coded independently of the corresponding texture pictures; decode a first motion vector from the bitstream; decode a second motion vector from the bitstream; decode the second depth map picture, wherein the first motion vector is used to predict the second depth map picture from the first depth map picture; and decode the second texture picture, wherein the second motion vector is used to predict the second texture picture from the first texture picture.
Multiview video coding has a wide variety of applications, including free-viewpoint video/television, 3D TV, and surveillance applications. Currently, the Joint Video Team (JVT) of ISO/IEC Motion Picture Expert Group (MPEG) and ITU-T Video Coding Expert Group is working to develop a MVC standard, which is becoming an extension of H.264/AVC. These standards are referred to herein as MVC and AVC, respectively. The latest working draft of MVC is described in JVT-AB204, "Joint Draft Multi-view Video Coding," 28th JVT meeting, Hannover, Germany, July 2008, available at ftp3.itu.ch/av-arch/jvt-site/2008_07_Hannover/JVT-AB204.zip.
Besides the features defined in the working draft of MVC, other potential features, particularly those focusing on coding tools, are described in the Joint Multiview Video Model (JMVM). The latest version of JMVM is described in JVT-AA207, "Joint Multiview Video Model (JMVM) 8.0," 24th JVT meeting, Geneva, Switzerland, April 2008, available at ftp3.itu.ch/av-arch/jvt-site/2008_04_Geneva/JVT-AA207.zip.
Furthermore, MPEG has also specified a format for attaching a depth map for a regular video stream in MPEG-C part 3. This specification is described within "Text of ISO/IEC FDIS 23002-3 Representation of Auxiliary Video and Supplemental Information," N8768 of ISO/IEC JTC 1/SC 29/WG 11, Marrakech, Morocco, January 2007.
In MPEG-C part 3, a so-called auxiliary video can be either a depth map or a parallax map. A texture video typically consists of three components, namely one luma component Y, and two chroma components U and V, whereas a depth map only has one component representing the distance between an object pixel and the camera. Generally, a texture video is represented in YUV 4:2:0, 4:2:2 or 4:4:4 format, where one chroma sample (U or V) is coded for each 4, 2, or 1 luma sample, respectively. A depth map is regarded as luma-only video in YUV 4:0:0 format. Depth maps can be inter-coded similarly to inter-coded luma-only texture pictures, and hence coded depth maps can have motion vectors. When representing a depth map, it provides flexibilities in terms of the number of bits used to represent each depth value. For example, the resolution of the depth map can be, for example, 1/4 the width and 1/2 the height of an associated image).
For multiview video content, MVC is the "state-of-art" coding standard. Based on the MVC standard, it is not possible to code depth map videos and texture videos in one MVC bitstream, and in the meantime, enabling motion prediction between depth map images and texture images.
The SVC specification is described in ITU-T Recommendation H.264, "Advanced video coding for generic audiovisual services," November 2007, available at http://www.itu.int/rec/T-REC-H.264/en.
In SVC, there is an output flag "output_flag" in the NAL unit header SVC extension to specify whether a decoded picture is to be output or not. For a video coding layer (VCL) NAL unit belonging to the AVC compatible base layer, output_flag is included in the associated prefix NAL unit.
In SVC, inter-layer coding dependency hierarchy for spatial scalability and CGS are identified by the syntax element dependency_id, while MGS dependency hierarchy is identified by the syntax element quality_id. As is done with temporal_id, these two syntax elements are also signaled in the NAL unit header SVC extension. At any temporal location, a picture of a larger dependency_id value may be inter-layer predicted from a picture with a smaller dependency_id value. However, in CGS, at any temporal location and with an identical dependency_id value, a picture with a quality_id value equal to QL can only use the base quality picture with a quality_id value equal to QL-1 for inter-layer prediction. Those quality enhancement layers with a quality_id greater than 0 are MGS layers.
The coding mode using inter-layer texture prediction is called "IntraBL" mode in SVC. To enable single-loop decoding, only the MBs for which the co-located MBs in the base layer for inter-layer prediction are constrainedly intra-coded can use this mode. A constrainedly intra-coded MB is intra-coded without referring to any samples from the neighboring inter MBs. For spatial scalability, the texture is upsampled based on the resolution ratio between the two layers. In the enhancement layer, the difference between the original signal and the possibly upsampled base layer texture is coded as if it was the motion compensation residue in an inter MB in single-layer coding.
Thus, for multiple-view bitstreams, a MVC-compliant depth coding scenario is contemplated. When multiple views exist, each of which is with a depth map video and a texture video, MVC-compliant depth coding can be applied to enable inter-view prediction between depth map videos and inter-view prediction between texture videos. One manner of coding the multiview content is in an MVC-compliant manner. In accordance with a first embodiment, all of the depth map images are coded as auxiliary pictures while enabling inter-view prediction (indicated by the arrows) between auxiliary pictures in different views, as shown in FIG. 5a. For example, FIG. 5a illustrates that a depth map video from view 1 can utilize depth map videos from views 0 and 2 as prediction references.
In accordance with a second embodiment, all of the depth map images are coded as normal 4:0:0 views while each of the depth map videos is assigned a new view identifier. For example and as shown in FIG. 5b, considering a scenario with three views, the texture videos are coded as views 0 to 2, and the depth map videos are coded as views N, N+1 and N+2. Inter-view motion prediction between depth map video and texture video in each view is applied in this embodiment. It should be noted that motion skip applied in JMVM (indicated by the diagonal arrows between each of the depth map and texture videos) do not take effect in this embodiment. The remaining arrows illustrated in FIG. 5b are again indicative of inter-view prediction. In this case, an SEI message is introduced to enable a renderer to map the view identifier of the depth map to its associated texture video. In FIG. 5b, the inter-view motion prediction is from depth map video to texture video, alternatively, inter-view motion prediction can be done from the texture video to depth map.
Alternatively still, yet another embodiment implementing depth coding in SMVC can be applied where certain views are coded with depth map video while some views are coded without depth map video, as shown in FIG. 5d. In this case, some views, e.g., view 1, have only one dependency layer (the texture video) while other views can have two dependency layers (the depth map as well as the texture video). Moreover, this embodiment can also utilize inter-layer prediction within one view from texture video to depth map video.
With respect to FIG. 4 and as described above, in a first embodiment, the media stream can be an SVC bitstream with a base layer comprising the first and second depth pictures, where the second motion vector is predicatively coded (e.g., using inter-layer motion prediction) on the basis of the first motion vector. Alternatively, in a second embodiment, the media stream can be an SVC bitstream with a base layer comprising the first and second sample pictures, where the first motion vector is predicatively coded (e.g., using inter-layer motion prediction) on the basis of the second motion vector. Additionally, the base layer (in the first embodiment) or an enhancement layer (in the second embodiment) is coded as monochromatic video, where an enhancement layer is coded as a MGS, CGS, or spatial enhancement layer. It should be noted that the base layer (in the first embodiment) or the enhancement layer (in the second embodiment) is indicated "not targeted for output," where an SEI message is encoded to indicate that the media stream comprises a base layer of depth map images (in the fist embodiment). The SEI message can also be encoded to indicate that the media stream comprises an enhancement layer of depth map images (in the second embodiment).
This SEI message, if present, indicates that the coded SVC bitstream has one or more dependency layers (depth map video) of a 4:0:0 format, and from the two dependency layers which have different chroma sampling formats, only inter-layer motion prediction is allowed. The semantics of an SVC joint depth coding SEI message includes a "view info_pre_flag" that when equal to 1, indicates that the view identifier to which this SVC bitstream corresponds is specified. A "view info_pre_flag" equal to 0 indicates that a view identifier is not specified.
Additionally, a "view_id" indicates the view identifier of a view which the decoded video and depth map correspond to.
The MVC depth view identifier mapping SEI message semantics include a "num_depth_views_minus1" parameter that indicates the number of views that are coded with depth map video. Additionally, a "sample_view_id[i]" parameter indicates the view_id of the texture video of the i-th view that is coded with a depth map video. Furthermore, "depth_view_id[i]" indicates the view_id of the depth map video of the i-th view that is coded with a depth map video.
When considering JMVM-compliant depth coding scenarios, an exemplary SEI message syntax can be the same as that described above for MVC-compliant depth coding. With regard to mapping SEI message semantics, like the semantics of a MVC depth_view_identifier mapping SEI message, the following is included: a "num_depth_views_minus1" parameter that indicates the number of views that are coded with depth map video; a "sample_view_id[i]" parameter that indicates the view_id of the texture video of the i-th view that is coded with a depth map video, and "depth_view_id[i]" which indicates the view_id of the depth map video of the i-th view that is coded with a depth map video. Additionally, when present, the bitstream enables motion skip from the depth map video and texture video with a view identifier pair of "depth_view_id[i]" and "sample_view_id[i]". The signaled disparity motion from a view with a "view_id" value equal to a "depth_view_id[i]" value to the view with a "view_id" value equal to the "sample_view_id[i]" value is set to zero.
The coded media bitstream is transferred to a storage 620. The storage 620 may comprise any type of mass memory to store the coded media bitstream. The format of the coded media bitstream in the storage 620 may be an elementary self-contained bitstream format, or one or more coded media bitstreams may be encapsulated into a container file. Some systems operate "live", i.e. omit storage and transfer coded media bitstream from the encoder 610 directly to the sender 630. The coded media bitstream is then transferred to the sender 630, also referred to as the server, on a need basis. The format used in the transmission may be an elementary self-contained bitstream format, a packet stream format, or one or more coded media bitstreams may be encapsulated into a container file. The encoder 610, the storage 620, and the server 630 may reside in the same physical device or they may be included in separate devices. The encoder 610 and server 630 may operate with live real-time content, in which case the coded media bitstream is typically not stored permanently, but rather buffered for small periods of time in the content encoder 610 and/or in the server 630 to smooth out variations in processing delay, transfer delay, and coded media bitrate.
The system includes one or more receivers 650, typically capable of receiving, de-modulating, and de-capsulating the transmitted signal into a coded media bitstream. The coded media bitstream is transferred to a recording storage 655. The recording storage 655 may comprise any type of mass memory to store the coded media bitstream. The recording storage 655 may alternatively or additively comprise computation memory, such as random access memory. The format of the coded media bitstream in the recording storage 655 may be an elementary self-contained bitstream format, or one or more coded media bitstreams may be encapsulated into a container file. If there are multiple coded media bitstreams, such as an audio stream and a video stream, associated with each other, a container file is typically used and the receiver 650 comprises or is attached to a container file generator producing a container file from input streams. Some systems operate "live," i.e. omit the recording storage 655 and transfer coded media bitstream from the receiver 650 directly to the decoder 660. In some systems, only the most recent part of the recorded stream, e.g., the most recent 10-minute excerption of the recorded stream, is maintained in the recording storage 655, while any earlier recorded data is discarded from the recording storage 655.
Various embodiments may be implemented in software, hardware, application logic or a combination of software, hardware and application logic. The software, application logic and/or hardware may reside, for example, on a chipset, a mobile device, a desktop, a laptop or a server. Software and web implementations of various embodiments can be accomplished with standard programming techniques with rule-based logic and other logic to accomplish various database searching steps or processes, correlation steps or processes, comparison steps or processes and decision steps or processes. Various embodiments may also be fully or partially implemented within network elements or modules. It should be noted that the words "component" and "module," as used herein and in the following claims, is intended to encompass implementations using one or more lines of software code, and/or hardware implementations, and/or equipment for receiving manual inputs.
Previous Patent US 9,973,738 | Next Patent US 9,973,740