Abstract:
A method for reconstructing a stereoscopic image stream from a plurality of compressed frames is provided. Each compressed frame consists of a merged image formed by juxtaposing a sampled image frame of a left image and a sampled image frame of a right image. Each sampled image frame has half a number of original pixels disposed at intersections of a plurality of horizontal lines and a plurality of vertical lines in a staggered quincunx pattern in which original pixels surround missing pixels. Each missing pixel is reconstructed according to at least 5 horizontal pixel pairs and 3 vertical pixel pairs in a compressed frame.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Field of the Invention 
         [0002]    The present invention is related to a method and a system for processing a stereoscopic image stream, and more particularly, to a method and a system for reconstructing a stereoscopic image stream from quincunx sampled frames. 
         [0003]    2. Description of the Prior Art 
         [0004]    Three-dimensional (3D) display technology provides more vivid visual experiences than traditional two-dimensional (2D) display technology. In general, the stereoscopic image processing involves two camera systems in which two different images or videos are taken from slightly different camera angles and locations. Techniques to artificially create a perception of depth on a 2D surface include the use of presenting different images to the left and right eyes of the viewer. In such frame sequential 3D display system, a sequence of alternating frames wherein each successive frame carries the image meant for one or the other eye is presented to each eye using shutter glasses having a left-eye lens and a right-eye lens, each of which may be made from electronically controllable liquid crystal assemblies. The lenses are configured to be alternatively switched on and off in sync with the alternating frames such that the right eye only views the right-eye images and the left eye only views the left-eye images. The two series of images are combined by the brain in such a way to perceive depth. 
         [0005]    Most recently released 3D high-definition televisions (HDTVs) operate according to the frame sequential 3D display method described above. However, this doesn&#39;t mean that the input signal to the 3D HDTV has to be in a frame-sequential format. Instead, many 3D HDTVs can process signals in a variety of different formats and perform on-the-fly conversion of the incoming video signal into a frame sequential format. While frame-sequential 3D is part of the blu-ray 3D specification, the video data in a side-by-side format is often preferred when it comes to airing 3D content over cable/air. 
         [0006]      FIGS. 1A-1D  are diagrams illustrating a prior art method for encoding 3D images.  FIG. 1A  depicts an original left-eye frame L and an original right-eye frame R of the same full resolution, such as 1920×1080. In  FIG. 1B , the left-eye frame L and the right-eye frame R are processed by quincunx sampling, which, as well-known to those skilled in the art, is a sampling method by which sampling of odd pixels alternates with sampling of even pixels for consecutive rows, such that the sampled pixels form a checkerboard pattern, thereby resulting in a sampled left-eye frame L′ and a sampled right-eye frame R′. In  FIG. 1C , the remaining pixels are slid in the horizontal direction for providing a down-scaled left-eye frame L″ (consisting of pixels marked by diagonal stripes which go top right to bottom left) with a resolution of 960×1080 and a down-scaled right-eye frame R″ (consisting of pixels marked by diagonal stripes which go top left to bottom right) with a resolution of 960×1080. In  FIG. 1D , the down-scaled left-eye frame L″ and the down-scaled right-eye frame R″ are placed side-by-side, thereby forming the side-by-side frame SBS. 
         [0007]    Since the side-by-side frame SBS contains half the pixels compared to those in the original left-eye frame L and the original right-eye frame R, it can be transmitted more efficiently. However, the missing pixels that have been lost by quincunx sampling need to be reconstructed for playback, normally according to several known interpolation algorithms. Examples of such algorithms include nearest neighbor interpolation which directly applies data from the neighboring pixels to recreate the missing pixel. Unfortunately, such an algorithm produces diagonal line artifacts which result in a deteriorated reconstructed image. There is thus a need for a method capable of generating a stereoscopic image stream with better quality by reconstructing missing pixels of quincunx sampled frames. 
       SUMMARY OF THE INVENTION 
       [0008]    The present invention provides a method for reconstructing a stereoscopic image stream from a plurality of compressed frames each consisting of a merged image formed by juxtaposing a sampled image frame of a left image and a sampled image frame of a right image, each sampled image frame having half a number of original pixels disposed at intersections of a plurality of horizontal lines and a plurality of vertical lines in a staggered quincunx pattern in which original pixels surround missing pixels. For a missing pixel located at an intersection of an n th  horizontal line among an (n−1) th  to an (n+1) th  adjacent horizontal lines and an n th  vertical line among an (n−2) th  to an (n+2) th  adjacent vertical lines, the method includes providing a first intermediate value according to a first horizontal pixel pair having a first pixel located at an intersection of the n th  horizontal line and the (n−1) th  vertical line and a second pixel located at an intersection of the n th  horizontal line and the (n+1) th  vertical line, and then storing the first intermediate in a storage unit; providing a second intermediate value according to a first vertical pixel pair having a third pixel located at an intersection of the (n−1) th  horizontal line and the n th  vertical line and a fourth pixel located at an intersection of the (n+1) th  horizontal line and the n th  vertical line, and then storing the second intermediate in the storage unit; providing a horizontal sensitivity parameter according to the first horizontal pixel pair, at least three neighboring pixels located along the (n−1) th  horizontal line and at least three neighboring pixels located along the (n+1) th  horizontal line, and then storing the horizontal sensitivity parameter in the storage unit; providing a vertical sensitivity parameter according to the first vertical pixel pair, at least two neighboring pixels located along the (n−2) th  vertical line and at least two neighboring pixels located along the (n+2) th  vertical line, and then storing the vertical sensitivity parameter in the storage unit; and reconstructing the missing pixel by weighting the first intermediate value and the second intermediate value according to the horizontal sensitivity parameter and the vertical sensitivity parameter. 
         [0009]    The present invention provides a system for reconstructing a stereoscopic image stream from a plurality of compressed frames. The system includes an image source, a stereoscopic image processor, and a display device. The image source provides the plurality of compressed frames each consisting of a merged image formed by juxtaposing a sampled image frame of a left image and a sampled image frame of a right image, each sampled image frame having half a number of original pixels disposed at intersections of a plurality of horizontal lines and a plurality of vertical lines in a staggered quincunx pattern in which original pixels surround missing pixels. The stereoscopic image processor includes an input buffer for receiving the plurality of compressed frames; a spatial interpolator configured to generate the stereoscopic image stream by reconstructing missing pixels of each of the left and right images by means of interpolation; a storage unit for storing the first intermediate value, the second intermediate value, the horizontal sensitivity parameter and the vertical sensitivity parameter; and an output buffer for outputting the stereoscopic image stream. For a missing pixel located at an intersection of an n th  horizontal line among an (n−1) th  to an (n+1) th  adjacent horizontal lines and an n th  vertical line among an (n−2) th  to an (n+2) th  adjacent vertical lines, the interpolation includes providing a first intermediate value according to a first horizontal pixel pair having a first pixel located at an intersection of the n th  horizontal line and the (n−1) th  vertical line and a second pixel located at an intersection of the n th  horizontal line and the (n+1) th  vertical line; providing a second intermediate value according to a first vertical pixel pair having a third pixel located at an intersection of the (n−1) th  horizontal line and the n th  vertical line and a fourth pixel located at an intersection of the (n+1) th  horizontal line and the n th  vertical line; providing a horizontal sensitivity parameter according to the first horizontal pixel pair, at least three neighboring pixels located along the (n−1) th  horizontal line and at least three neighboring pixels located along the (n+1) th  horizontal line; providing a vertical sensitivity parameter according to the first vertical pixel pair, at least two neighboring pixels located along the (n−2) th  vertical line and at least two neighboring pixels located along the (n+2) th  vertical line; and reconstructing the missing pixel by weighting the first intermediate value and the second intermediate value according to the horizontal sensitivity parameter and the vertical sensitivity parameter; and generating the stereoscopic image stream according to the reconstructed pixel associated with the missing pixel. 
         [0010]    These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0011]      FIGS. 1A-1D  are diagrams illustrating a prior art method for encoding 3D images. 
           [0012]      FIG. 2  is a diagram illustrating a method for reconstructing stereoscopic images according to the present invention. 
           [0013]      FIG. 3  is a block diagram illustrating a 3D display system  100  for implementing the present method. 
       
    
    
     DETAILED DESCRIPTION 
       [0014]      FIG. 2  is a diagram illustrating a method for reconstructing stereoscopic images according to the present invention.  FIG. 2  shows a sampled image frame in the quincunx pattern, wherein the remaining pixels are represented by solid circles and the missing pixels are represented by dotted circles. In an embodiment of the present invention, the remaining pixels located at intersections of 3 adjacent horizontal lines HL n−1 -HL n+1  and 5 adjacent vertical lines VL n−2 -VL n+2  are used for reconstructing a corresponding missing pixel PX which is located at the intersection of the horizontal line HL n  and the vertical line VL n . As depicted in  FIG. 2 , the missing pixel PX may be reconstructed according to the remaining pixels P 1 -P 8 . 
         [0015]    First, the 4 adjacent pixels P 1 -P 4  of the missing pixel PX are used for calculating a first intermediate value M 1  and a second intermediate value M 2 : 
         [0000]        M   1 =( P 1 +P 2)/2  (1)
 
         [0000]        M   2 =( P 3 +P 4)/2  (2)
 
         [0016]    Next, 5 horizontal pixel pairs formed by the pixels P 1 -P 8  are used for calculating a horizontal sensitivity parameter W H , and 3 vertical pixel pairs formed by the pixels P 3 -P 8  are used for calculating a vertical sensitivity parameter W V : 
         [0000]        W   H   =|P 1− P 2|+(| P 5− P 3|+| P 3− P 6|+| P 7 −P 4|+| P 4 −P 8|)/2  (3)
 
         [0000]        W   V   =|P 5 −P 7|+ |P 3− P 4 |+|P 6 −P 8|  (4)
 
         [0017]    If the sum of the horizontal sensitivity parameter W H  and the sum of the vertical sensitivity parameter W V  is equal to zero (W H +W V =0), the missing pixel PX can be reconstructed by averaging the first intermediate value M 1  and the second intermediate value M 2 . If the sum of the horizontal sensitivity parameter W H  and the sum of the vertical sensitivity parameter W V  is not equal to zero (W H +W V ≠0), the missing pixel PX can be reconstructed by weighting the first intermediate value M 1  and the second intermediate value M 2  according to the horizontal sensitivity parameter W H  and the vertical sensitivity parameter W V . The value of the reconstructed missing pixel PX is illustrated as follows: 
         [0000]        PX =( M   1   +M   2 )/2, if  W   H   +W   V =0  (5)
 
         [0000]        PX =( M   2   *W   H   +M   1   *W   V )/( W   H   +W   V ), if  W   H   +W   V ≠0  (6)
 
         [0018]    Normally, most area of an image have smooth changes, which means the original value of a missing pixel before down-sampling is very likely to be similar to its surrounding pixels. However, instead of reconstructing the missing pixel PX by directly averaging the four surrounding pixels P 1 -P 4 , the contribution of the pixels P 5 -P 8  is also taken into account in the present invention. 
         [0019]    As shown in equations (1) and (2), the first intermediate value M 1  is associated with a horizontal pixel pair P 1 -P 2 , and the second intermediate value M 2  is associated with a vertical pixel pair P 3 -P 4 . 
         [0020]    As shown in equation (3), the horizontal sensitivity parameter W H  of the missing pixel PX is associated with five horizontal pixel pairs: two horizontal pixel pairs P 5 -P 3  and P 3 -P 6  located on the horizontal line HL n−1 , two horizontal pixel pairs P 7 -P 4  and P 4 -P 8  located on the horizontal line HL n+1 , and the horizontal pixel pair P 1 -P 2  located on the horizontal line HL n . As shown in equation (4), the vertical sensitivity parameter W V  of the missing pixel PX is associated with three vertical pixel pairs: a vertical pixel pair P 5 -P 7  located on the vertical line VL n−1 , a vertical pixel pair P 6 -P 8  located on the vertical line VL n+1 , and the vertical pixel pair P 3 -P 4  located on the vertical line VL n . 
         [0021]    Since the horizontal sensitivity parameter W H  is calculated according to 5 horizontal pixel pairs and the vertical sensitivity parameter W V  is calculated according to 3 vertical pixel pairs, the present invention may also include normalization. For example, the horizontal sensitivity parameter W H  may include the full contribution of the most dominant horizontal pixel pair P 1 -P 2 , while the contributions of the 4 less dominant horizontal pixel pairs are weighted according to the number of horizontal pixel pairs involved. The vertical sensitivity parameter W V  may include the full contribution of the most dominant vertical pixel pair P 3 -P 4 , while the contributions of the 2 less dominant vertical pixel pairs are weighted according to the number of horizontal pixel pairs involved. In other words, the sum of the 4 horizontal pixel pairs P 5 -P 3 , P 3 -P 6 , P 7 -P 4  and P 4 -P 8  is divided by 2 (weighted by 2/4) in the equation (3), while the 2 vertical pixel pairs P 5 -P 7  and P 6 -P 8  are weighted by 1 in the equation (4). 
         [0022]    The situation indicated by equation (5) only occurs when all the pixel pairs are equal to zero, and the missing pixel PX is reconstructed with 0 in this case. When W H +W V ≠0 as indicated by equation (6), the missing pixel PX is reconstructed with an interpolated value (M 2 *W H +M 1 *W V )/(W H +W V ) which includes the contribution of multiple surrounding pixel pairs. 
         [0023]    In the embodiment illustrated in  FIG. 2 , the 8 remaining pixels P 1 -P 8  located at intersections of 3 adjacent horizontal lines HL n−1 -HL n+1  and 5 adjacent vertical lines VL n−2 -VL n+2  are used for reconstructing the missing pixel PX which is located at the intersection of the horizontal line HL n  and the vertical line VL n . In other embodiments, more remaining pixels may be used for reconstructing the missing pixel PX. For example, assume that the horizontal sensitivity parameter W H  is calculated according to the most dominant horizontal pixel pair P 1 -P 2  and N H  less dominant horizontal pixel pairs, and the vertical sensitivity parameter W V  is calculated according to the most dominant horizontal pixel pair P 3 -P 4  and N V  less dominant vertical pixel pairs. The horizontal sensitivity parameter W H  and the vertical sensitivity parameter W V  may be represented by: 
         [0000]        W   H   =|P 1 −P 2|+ K   H *( DH (1)+ DH (2)+ . . .  DH ( N   H )  (7)
 
         [0000]        W   V   =|P 3 −P 4 |+K   V *( DV (1)+ DV (2)+ . . .  DV ( N   V )  (8)
 
         [0000]      wherein  K   H   *N   H   =K   V   *N   V   (9)
 
         [0024]    DH(1)˜DH(N H ) represent the absolute value of the difference between each corresponding horizontal pixel pairs, and DV(1)˜DV(N V ) represent the absolute value of the difference between each corresponding vertical pixel pairs. The horizontal sensitivity parameter W H  includes the full contribution of the most dominant horizontal pixel pair P 1 -P 2 , while the contributions of the less dominant horizontal pixel pairs are weighted by a normalization factor K H . The vertical sensitivity parameter W V  includes the full contribution of the most dominant vertical pixel pair P 3 -P 4 , while the contributions of the less dominant vertical pixel pairs are weighted by a normalization factor K V . The normalization factors K H  and K V  are associated as indicated in equation (9) so that the values of the horizontal sensitivity parameter W H  and the vertical sensitivity parameter W V  may be contributed by the same number of pixel pairs. 
         [0025]    In addition to the horizontal pixel pair P 1 -P 2 , P 5 -P 3 , P 3 -P 6 , P 7 -P 4  and P 4 -P 8  used in the embodiment of  FIG. 2 , the N H  horizontal pixel pairs may further include other horizontal pixel pairs each located along the adjacent horizontal line H n−1  or H n+1 , or may further include one or more horizontal pixel pairs each located along other horizontal lines, such as H n−2  or H n+2 . In addition to the vertical pixel pair P 3 -P 4 , P 5 -P 7 , and P 6 -P 8  used in the embodiment of  FIG. 2 , the N V  vertical pixel pairs may include other vertical pixel pairs each located along the vertical lines V n−2 , V n  or V n+2 , or may include other vertical pixel pairs each located along other vertical lines, such as V n−4  or V n+4 . 
         [0026]      FIG. 3  is a block diagram illustrating a 3D display system  100  for implementing the present method. The 3D display system  100  includes an image source  10 , a stereoscopic image processor  20 , and a display device  30 . The image source  10  may provide a source 2D video signal S 1  consisting of multiple side-by-side frames representing an image by means of quincunx sampling as previously illustrated. The stereoscopic image processor  20  includes an input buffer  22 , a spatial interpolator  24 , a storage unit  26 , and an output buffer  28 . The input buffer  22  is configured to receive the source 2D video signal S 1  for subsequent processing. The spatial interpolator  24  is configured to interpolate missing pixels in the 2D video signal S 1  using the interpolation method illustrated in equations (1)-(9), thereby generating a 3D video signal S 2  having reconstructed frames each containing half original pixels and half interpolated pixels. The storage unit  26  is used for storing intermediate results during the interpolation. The 3D video signal S 2  may then be transmitted to the display device  30  via the output buffer  28  for playback. In more detail, the stereoscopic image processor  20  may further convert the 3D video signal S 2  into a corresponding format, such as anaglyph or interline . . . etc, according to the type of the display device  30 , and the converting operation may be executed in the post-end of the spatial interpolator  24 . 
         [0027]    Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention.