Method and apparatus for determining view of stereoscopic image for stereo synchronization

Provided is a method and apparatus for determining stereo views of a stereoscopic image. The method includes: selecting a pair of first and second images from a pair of frames of the stereoscopic image, the pair of frames including a base view image and an additional view image. The method further includes determining which of the first image and the second image is the base view image and which of the first image and the second image is the additional view image based on a parallax between the pair of first and second images.

CROSS-REFERENCE TO RELATED PATENT APPLICATION

This application claims priority from Korean Patent Application No. 10-2008-0015446, filed on Feb. 20, 2008, in the Korean Intellectual Property Office, and the disclosures of which are incorporated herein in their entirety by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Methods and apparatuses consistent with the present invention relate to analysis of a view of a stereoscopic image and stereo synchronization, and more particularly, to stereo synchronization used to reproduce a stereoscopic image in a display device based on precisely displaying the stereoscopic image comprising images each having a different view.

2. Description of the Related Art

Stereo sync information indicating a degree of synchronization of a stereoscopic image is needed to reproduce the stereoscopic image. Stereo sync indicates whether a current image is a left view image or a right view image.

Image data only is input into a display device in order to reproduce a three-dimensional (3D) image. Most 3D image content that is currently manufactured and distributed contains left and right view image data but does not include a flag indicating whether a current frame is a left view image or a right view image. In more detail, since only the image data is transmitted when the stereoscopic image is transmitted to the display device, the display device cannot know whether the current frame is the left view image or the right view image.

Although a terminal has been developed to output stereo sync from graphic hardware such as a graphic card, the terminal is not employed in most currently available products, and if employed inflates costs due to a requirement for extra hardware.

FIG. 1is a diagram for explaining a relationship between a video format conversion device110and a display device120. Referring toFIG. 1, a stereoscopic image reproduction system100comprising the video format conversion device110and the display device120converts input left and right view images into a stereoscopic image in a display input format depending on the properties of the display device120. The display device120receives the converted stereoscopic image and reproduces a 3D image having a stereo image effect.

Although stereo sync information of the stereoscopic image may be stored in the properties of the display device120, a cable other than an output cable such as a digital visual interface (DVI) port is necessary for transmitting the stereo sync information, and as specific hardware is required, there is a strong possibility that it may not be supported.

SUMMARY OF THE INVENTION

The present invention provides a method of analyzing a stereoscopic image and determining which of a stereo view each image included in the stereoscopic image is using a display device without an additional input cable of the display device.

The present invention also provides a method of internally generating stereo synchronization information based on a determined stereo view.

The present invention also provides a method of using disparity estimation, and a camera parameter in order to determine a more efficient stereo view.

The present invention also provides a method of reducing viewing fatigue and performing stereo synchronization of a stereoscopic image when stereo synchronization information is fixed.

According to an aspect of the present invention, there is provided a method of determining stereo views of a stereoscopic image, the method comprising: selecting a pair of first and second images from frames of the stereoscopic image, wherein the selected pair correspond to a base view image and an additional view image; and determining which of the first image and the second image is the base view image and which of the first image and the second image is the additional view image based on a parallax between the pair of first and second images.

The method may further comprise: synchronizing the frames of the stereoscopic image based on the determining of which of the first image and the second image is the base image and which of the first image and the second image is the additional view image.

An image may comprise a pair of frames of the stereoscopic image which are a base view image and an additional view image, and the selecting the pair of first and second images may comprise reconstructing the pair of the frames from the image; and selecting the pair of frames as the pair of first and second images.

The determining which of the first image and the second image is the base view image and which of the first image and the second image is the additional view image may comprise: performing a disparity estimation with regard to the pair of first and second images by using the parallax between the pair of first and second images and determining a disparity vector; and determining views of the first and second images based on the determined disparity vector.

The determining of the disparity vector may comprise: obtaining a differential image in which a difference between a value of each pixel of the first image and a value of each corresponding pixel of the second image is recorded; dividing the differential image into a plurality of blocks; and determining which of the plurality of blocks have a sum of absolute values of pixel values greater than a predetermined threshold value; and performing the disparity estimation with regard to the determined blocks.

The determining which of the plurality of blocks have a sum of absolute values of pixel values greater than a predetermined threshold value may comprise: determining which blocks have the sum of absolute values of pixel values greater than the predetermined threshold value among blocks other than a block including a left edge area and a right edge area of the differential image.

The performing of the disparity estimation may comprise: determining, in the first image and the second image, a current block location, a left block location that is shifted in parallel by a predetermined distance from the current block location to the left, and a right block location that is shifted in parallel by the predetermined distance from the current block location to the right, determining a first difference between pixels of the current block location of the first image and pixels of the left block location of the second image; determining a second difference between the pixels of the current block location of the first image and pixels of the current block location of the second image; determining a third difference between the pixels of the current block location of the first image and pixels of the right block location of the second image; comparing the first, second, and third differences; and determining directions of the disparity vector according to the comparison of the first, second, and third differences.

The determining of the directions of the disparity vector may comprise: if the third, second, and first differences are sequentially smaller, determining a direction of the disparity vector to be in a negative direction; and if the first, second, and third differences are sequentially smaller, determining the direction of the disparity vector to be in a positive direction.

The performing of the disparity estimation may comprise: determining, in the first image and the second image, a current line location with regard to pixels in a predetermined horizontal direction of the current block location and a search range including pixels that are shifted in parallel by a predetermined distance from the current line location in a left and right direction, determining differences between pixels of the current line location of the first image and pixels of a line location that is shifted in parallel from the current line location in a left or right direction within the search range of the second image; comparing the determined differences; and determining the disparity vector to be in a direction of a line having the smallest difference according to the comparison of the differences.

The determining of the disparity vector may comprise: performing the disparity estimation with regard to each of the a plurality of corresponding blocks of the first and second images based on the parallax between the plurality of blocks of the first and second images and determining a disparity vector for each of the plurality of corresponding blocks; counting the number of positive directions of the disparity vectors and the number of negative directions of the disparity vectors; and determining a disparity vector of the first and second images to be in a positive direction if there is a greater number of disparity vectors of the plurality of corresponding blocks which are in the positive direction, and determining a disparity vector of the first and second images to be in a negative direction if there is a greater number of disparity vectors of the plurality of corresponding blocks which are in the negative direction.

The determining of the views of the first and second images may comprise: when the stereoscopic image is obtained from a parallel camera, determining the first image as the base view image and the second image as the additional view image if the direction of the disparity vector is positive; and determining the first image as the additional view image and the second image as the base view image if the direction of the disparity vector is negative.

The determining of the views of the first and second images may comprise: when the stereoscopic image is obtained from a cross camera, with regard to a cross point of the cross camera, determining the first image as the base view image and the second image as the additional view image if the direction of the disparity vector of an area before the cross point is positive and the direction of the disparity vector of an area after the cross point is negative; and determining the first image as the additional view image and the second image as the base view image if the direction of the disparity vector of the area before the cross point is negative and the direction of the disparity vector of the area after the cross point is positive.

The determining of the views of the first and second images may comprise: when a perpendicular disparity occurs between the first and second images, determining that the stereoscopic image is obtained from a cross camera.

The determining of the views of the first and second images may comprise: detecting an occlusion region that is seen from one of the first and second images but is not seen from the other one of the first and second images because of being covered by a predetermined object; determining that one of the first and second images in which the occlusion region can be seen is obtained from a point camera of based and additional view sides in which the occlusion region can be seen; and determining the other one of the first and second images in which the occlusion region is not seen as the view image of the other one of the based and additional view sides in which the occlusion region is not seen.

The determining of which of the first image and the second image is the base image and which of the first image and the second image is the additional view image may comprise: extracting, from stereoscopic image data, a first camera parameter with regard to the first image from stereoscopic image and a second camera parameter with regard to the second image; extracting a first parallel translation matrix of the first image from the first camera parameter and a second parallel translation matrix of the second image from the second camera parameter; comparing a component x1 of the first parallel translation matrix and a component x2 of the second parallel translation matrix; and determining views of the first and second images according to the comparison result, wherein the first and second parallel translation matrices indicate how far spatially corresponding coordinates of the first and second images are away from each other with regard to a predetermined world coordinate.

The determining of the views of the first and second images may comprise: if the component x1 is greater than the component x2, determining the first image as the additional view image and the second image as the base view image; and if the component x2 is greater than the component x1, determining the first image as the base view image and the second image as the additional view image.

The method may further comprise: determining a current view arrangement order of the first and second images, and if the current view arrangement order is different from previous view arrangement orders with regard to a predetermined plurality of pairs of first and second images prior to the current pair of first and second images, synchronizing the stereoscopic image based on the previous view arrangement orders.

The synchronizing of the stereoscopic image may comprise: if the current view arrangement order is different from a previous view arrangement order with regard to a pair of previous first and second image, and if the view arrangement orders of a predetermined plurality of pairs of first and second images after the pair of current first and second images are identical to the current view arrangement order, synchronizing the stereoscopic image based on the previous view arrangement orders with regard to the current view arrangement order.

The synchronizing of the stereoscopic image may comprise: if a view arrangement order of a pair of first and second images after the predetermined plurality of pairs of first and second images is identical to the current view arrangement order, synchronizing the stereoscopic image based on the current view arrangement order.

The synchronizing of the stereoscopic image may comprise: if the stereoscopic image is reproduced subsequently after a reproduction of a 2D image, inserting a predetermined number of blank frames between a last frame of the 2D image and a first frame of the stereoscopic image; and synchronizing the stereoscopic image based on the view arrangement order of the pair of first and second images after the blank frames.

According to another aspect of the present invention, there is provided an apparatus for determining stereo views of a stereoscopic image, the apparatus comprising: an image selecting unit which selects a pair of first and second images from frames of the stereoscopic image, wherein the selected pair comprises a base view image and an additional view image; and a view determining unit which determines which of the first image and the second image is the base view image and which of the first image and the second image is the additional view image using a parallax between the pair of first and second images.

The apparatus may further comprise: a stereo synchronizing unit which synchronizes the frames of the stereoscopic image based on the determination of which of the first image and the second image is the base view image and which of the first image and the second image is the additional view image.

According to another aspect of the present invention, there is provided a computer readable recording medium having recorded thereon a program for executing the method of determining stereo views of a stereoscopic image.

Hereinafter, exemplary embodiments of the present invention will be described more fully with reference to the accompanying drawings.

FIG. 2is a block diagram of an apparatus200for determining stereo views of a stereoscopic image according to an exemplary embodiment of the present invention. Referring toFIG. 2, the apparatus200for determining stereo views of the stereoscopic image comprises an image selecting unit210and a view determining unit220to determine stereo views of the stereoscopic image. The apparatus200for determining stereo views of the stereoscopic image comprises a stereo synchronization unit230and a display device240to perform stereo synchronization using the determined stereo views of the stereoscopic image.

The image selecting unit210selects a pair of a first image and a second image that are to be determined and which correspond to a pair of a base view image and an additional view image from the stereoscopic image, and outputs the selected first and second images to the view determining unit220. In the present embodiment, a left view image may be used as the base view image, and a right view image may be used as the additional view image.

If all frames of the stereoscopic image that is input into the image selecting unit210are the base view image and/or the additional view image, then the input frames are alternatively defined to be a first image and a second image and then these are processed, and reproduced

If the stereoscopic image that is input into the image selecting unit210is to include a frame of an image which is a combination of a pair of a base view image and an additional view image that correspond to each other, then a pair of a base view image frame and an additional view image frame are reconstructed from the frame of the image which is a combination of the pair of the base view image and the additional view image. The image selecting unit210selects the pair of first and second images that correspond to the reconstructed pair of base view image frame and additional view image frame.

The view determining unit220uses parallax between the pair of first and second images that are input from the image selecting unit210to determine the first image as one of the base view image and the additional view image, and the second image as the other one of the base view image and the additional view image. The determined view arrangement order is output to the stereo synchronization unit230.

The determined view arrangement order means that the first and second image are determined in order of the left view image and the right view image or in the order of the right view image and the left view image.

The view determining unit220performs disparity estimation with regard to the first and second images, and determines whether each of the first and second images is the base view image or the additional view image based on an obtained disparity vector.

Although not shown, the view determining unit220comprises a differential image obtaining unit that obtains a differential image in which a difference between a value of each pixel of the first image and a value of each pixel of the second image is recorded, a block determining unit that determines a block in which a sum of absolute values of pixel values is greater than a predetermined threshold value among blocks of the differential image, and a disparity estimation performing unit that performs a disparity estimation with regard to the determined block. The detailed operation of the view determining unit220will be described with reference toFIGS. 3 through 10.

The view determining unit220determines each view of the first and second images using a camera parameter extracted from the stereoscopic image. A parallel translation matrix indicates how far spatially coordinates of the first and second images are away from each other with respect to a predetermined world coordinate.

Although not shown, the view determining unit220comprises a camera parameter extracting unit that extracts a first camera parameter with regard to the first image from the stereoscopic image and a second camera parameter with regard to the second image from the stereoscopic image, a parallel translation matrix extracting unit that extracts a first parallel translation matrix that is a parallel translation matrix of the first image from the first camera parameter and extracts a second parallel translation matrix that is a parallel translation matrix of the second image from the second camera parameter, and an x component comparing unit that compares a component x1that is an x component of the first parallel translation matrix and a component x2that is an x component of the second parallel translation matrix.

The view determining unit220determines views of the first and second images according to the result of the x component comparison. For example, if the component x1is greater than the component x2, the first image is determined as the additional view image and the second image is determined as the base view image. Also, if the component x2is greater than the component x1, the first image is determined as the base view image and the second image is determined as the additional view image.

The stereo synchronization unit230synchronizes the stereoscopic image based on the view arrangement order of the pair of first and second images that is input from the view determining unit220, and outputs the synchronized stereoscopic image to the display device240.

The stereo synchronization unit230, in response to an abrupt change in the view arrangement order, compares a current view arrangement order with regard to a pair of current first and second images, a previous view arrangement order with regard to a pair of previous first and second images, and a next view arrangement order with regard to a pair of next first and second images, i.e., three view arrangement orders of three pairs of images, and then performs stereo synchronization with regard to the stereoscopic image. The operation of the stereo synchronization will be described in more detail with reference toFIGS. 11 and 12.

If the stereoscopic image is input after a two-dimensional (2D) image, the stereo synchronization unit230inserts a blank frame in order to reduce a viewer's viewing fatigue due to an abrupt change in a reproduction mode. The detailed description thereof will be described with reference toFIG. 13.

The display device240reproduces the synchronized stereoscopic image that is input from the stereo synchronization unit230. The display device240may reproduce the stereoscopic image that is synchronized according to the apparatus200for determining stereo views of the stereoscopic image without additional stereo synchronization information of stereoscopic image data.

FIG. 3is a detailed block diagram of an apparatus300for determining stereo views of a stereoscopic image according to another exemplary embodiment of the present invention. In the present exemplary embodiment, each frame of the stereoscopic image is one of a base view image or an additional view image.

The image selecting unit210receives a current frame303and delays a frame (305) with regard to the current frame303to obtain a previous frame307. The current frame303is selected as the second image. The previous frame307is selected as the first image.

The apparatus300for determining stereo views of the stereoscopic image comprises a differential image obtaining unit310, a block determining unit320, a disparity estimation performing unit330, and a view determining unit340. The differential image obtaining unit310obtains a differential image showing a difference between each pixel value of the first image and each pixel value of the second image. The block determining unit320determines a block for determining a disparity vector from among blocks of the differential image. The disparity estimation performing unit330performs a disparity estimation with regard to the block determined by the block determining unit320and obtains the disparity vector. The view determining unit340determines whether the first and second images are base view images or additional view images using direction of the disparity vector obtained by the disparity estimation performing unit330.

The apparatus300for determining stereo views of the stereoscopic image compares (350) a current view arrangement order343of a pair of current first and second images and a previous view arrangement order347with regard to a pair of previous images that can be obtained by delaying (345) a frame with regard to the pair of the current images. The apparatus300for determining stereo views of the stereoscopic image determines whether to change (360) or maintain (365) the view arrangement order according to the determined view arrangement order.

FIG. 4is a diagram for explaining a method of obtaining a differential image and determining a block according to an exemplary embodiment of the present invention.

With regard to disparity estimation, it is highly possible for errors to occur in a smooth area of an image and for exact values to be obtained in an edge portion having many feature points. The view determining unit220performs the disparity estimation with regard to a differential image430between a first image410and a second image420.

Since it is difficult to realize the apparatuses200and300for determining stereo views of the stereoscopic image in real-time when disparity estimation is performed with regard to all blocks of the differential image430, the view determining unit220performs the disparity estimation with regard to a part of the differential image430. In more detail, when the differential image430is divided into block units435, a block in which pixel values of two images is highly different is selected in order to determine a disparity vector with regard to a block having a large disparity and many edge components.

In Equation 1, a base view image and an additional view image are respectively a left view image L(i, j) and a right view image R(i, j). The first and second images410and420may be selected as one of the left view image L(i, j) and the right view image R(i, j). The view determining unit220calculates a sum of absolute values of differences between pixels of a block corresponding to a block of the first and second images410and420, i.e., a sum SumResiof absolute values of pixel values of a block of the differential image430.

The view determining unit220performs the disparity estimation with regard to a block having the sum SumResiof absolute values of pixel values of the block of the differential image430which is greater than a predetermined threshold value THResi. If the threshold value THResiis low, the determining is more sensitive to a disparity between two views, and thus the number of operations is high. If the threshold value THResiis high, the determining is less sensitive to a disparity between two views, and thus the number of operations is lower.

The view determining unit220may not perform the disparity estimation with regard to left and right edge blocks440and445of an image in order to increase precision of the disparity estimation. Since the left and right edge blocks440and445of the image have a boundary occlusion block, they are not suitable for the disparity estimation. Therefore, the view determining unit220performs the disparity estimation with regard to the block having the sum SumResiof absolute values of pixel values of the block of the differential image430which is greater than the predetermined threshold value THResiin a block450excluding the left and right edge blocks440and445of the image.

FIG. 5Ais a diagram for explaining a method of estimating each disparity of a base view image and an additional view image in a block unit according to an exemplary embodiment of the present invention. Referring toFIG. 5A, the view determining unit220selects a predetermined block location in order to determine direction of a disparity vector between a block of a first image510and a block of a second image520. Since a current block location determined using the method of determining a block shown inFIG. 4is a base, and base and additional views are left and right views, a relative disparity may be a disparity in left and right directions. The view determining unit220selects a first block location that moves in parallel by a predetermined distance from a current block location to the left, a second block location that is the current block location, and a third block location that moves in parallel by the predetermined distance from the current block location to the right. The predetermined distance of the left and right parallel movement is determined based on the quantity and precision of operation required.

The view determining unit220uses a difference between a block515of the current block location of the first image510and a block523of the first block location of the second image520as a sum of absolute difference (SAD) in order to find the direction of the disparity vector. In more detail, a first SAD value, which is an SAD value between the block515of the current block location of the first image510and the block523of the first block location of the second image520, is determined.

A second SAD value, which is an SAD value between the block515of the current block location of the first image510and a block525of the second block location of the second image520, is determined.

A third SAD value, which is an SAD value between the block515of the current block location of the first image510and a block527of the third block location of the second image520, is determined.

Since the SAD is calculated in the first, second, and third block locations in the present exemplary embodiment, the efficiency of operation is higher than that of a conventional method of performing the disparity estimation with regard to the entire image. Since the SAD values are calculated in units of blocks instead of units of pixels, a more precise direction of the disparity vector is determined and is more effective in a parallel camera structure.

FIG. 5Bis a diagram for explaining a method of estimating each disparity of a base view image and an additional view image in a line unit according to an embodiment of the present invention. Referring toFIG. 5B, the view determining unit220performs a disparity estimation in a line unit in order to increase the efficiency of operation compared to the disparity estimation in a block unit shown inFIG. 5A. A method of processing data in units of lines may be more advantageous to that in units of blocks in terms of a memory access of hardware.

The view determining unit220performs the disparity estimation in a line unit within a search range580between a pixel that moves in parallel by a predetermined distance from a current line location to left and a pixel that moves in parallel by the predetermined distance from the current line location to right. In more detail, SAD values of pixels in a line location that moves in parallel in a left or right direction in a pixel unit are determined within the search range580between pixels565of the current line location of a first image560and pixels575of the current line location of a second image570.

FIG. 6Aillustrates vectors for explaining a unimodal error surface assumption (UESA) according to an embodiment of the present invention. Referring toFIG. 6A, the UESA is used to determine a direction of a disparity vector through a comparison of a plurality of SAD values. In order to obtain an approximation of two optional vectors Vc1620and Vc2630with regard to a current vector v1610, a method of using the UESA will be described with reference toFIG. 6B.

FIG. 6Bis a diagram for explaining a method of determining the direction of a disparity vector through the UESA according to an embodiment of the present invention.

According to the UESA, an error value monotonically increases the farther the error value is away from a point where a whole area minimum error value occurs. Referring toFIG. 6B, SADv1660that is a SAD value between the vector v1610and the vector v1610, SADvc1670that is a SAD value between the vector v1610and the vectors Vc1620, and SADvc2680that is a SAD value between the vector v1610and the vector Vc2630are distributed on a SAD curve650. According to the UESA, the error value monotonically increases in the order of SADv1660, SADvc1670, and SADvc2680; therefore the error value increases in order of vector v1610, Vc1620, and Vc2630.

The view determining unit220compares SAD values of blocks (lines) when performing the disparity estimation in block and line units shown inFIGS. 5A and 5Baccording to the UESA, and determines the direction of the disparity vector.

In more detail, if the first SAD value with regard to the first block location, the second SAD value with regard to the second block location, and the third SAD value with regard to the third block location monotonically increase in the disparity estimation in the block unit shown inFIG. 5A, a SAD value between the block523of the first block location of the second image520and the block515of the current block location of the first image is the smallest, and thus the disparity vector is determined in a negative (−) direction.

Likewise, if the first SAD value with regard to the first block location, the second SAD value with regard to the second block location, and the third SAD value with regard to the third block location monotonically decrease in the disparity estimation in the block unit shown inFIG. 5A, the disparity vector is determined in a positive (+) direction.

In the disparity estimation in the line unit shown inFIG. 5B, a direction in parallel to a predetermined line of the second image570having a smallest SAD with regard to the location565of the current line location of the first image560is determined as the direction of the disparity vector.

A method of determining the first and second images as base view images or additional view images according to the direction of the disparity vector will now be described with reference toFIGS. 7A through 9. A base view and an additional view are left and right views, respectively.

FIG. 7Aillustrates a parallel type camera structure. Referring toFIG. 7A, a left camera710and a right camera720photograph a subject700in a stereo camera structure. The left camera710obtains a left view image. The right camera720obtains a right view image. Since the left camera710and the right camera720are disposed in parallel in the parallel type camera structure, a disparity vector between the left view image and the right view image is not inverted.

FIG. 7Billustrates a cross type camera structure. Referring toFIG. 7B, a left camera760and a right camera770may face a subject750in a crossed arrangement in the cross type camera structure. Therefore, a disparity vector between a left view image and a right view image may be inverted.

FIG. 8is a diagram for explaining a method of determining a view according to the direction of a disparity vector in a parallel type camera structure according to an embodiment of the present invention. Referring toFIG. 8, since the disparity vector is not inverted in the parallel type camera structure, it is possible to easily determine a view of a stereoscopic image if the direction of the disparity vector is known. In order to easily determine the view of the stereoscopic image in the parallel type camera structure, it is needed to be assumed that “when a point V1=(x1,y1) of a left view image and a point V2=(x2,y2) of a right view image correspond to each other, y1and y2are identical to each other with regard to a perpendicular component, and it is satisfied that x1>x2with regard to a horizontal component”.

According to this assumption, when a disparity vector DV is obtained on the condition that x(V1)−x(V2)=x1−x2, if the disparity vector DV is greater than 0, i.e., the disparity vector is disposed in a positive direction, a first image810is determined as a left view image, and a second image820is determined as a right view image.

Likewise, if the disparity vector DV is smaller than 0, i.e., the disparity vector is disposed in a negative direction, the first image is determined as the right view image, and the second image is determined as the left view image.

FIG. 9Aillustrates view directions and a cross point in a cross type camera structure. Referring toFIG. 9A, when a left camera910and a right camera920photograph a subject900, photographing view directions of the left camera910and the right camera920cross each other with regard to a line passing through the cross point905. The photographing view directions of both left and right cameras910and920form the cross point905.

FIG. 9Billustrates images according to views in a cross type camera structure. Referring toFIG. 9B, the images are a left view image930photographed by the left camera910and a right view image940photographed by the right camera920in the cross type camera structure shown inFIG. 9A.

A bottom area of the cross point905of the left view image930is located right, whereas a top area of the cross point905is located left. Likewise, a bottom area of the cross point905of the right view image940is located left, whereas a top area of the cross point905is located right. In more detail, a perpendicular disparity occurs where top and bottom areas of an image are disposed in opposite directions with regard to a cross point.

When the perpendicular disparity occurs between a first image and a second image in the present embodiment, it may be determined that a stereoscopic image is obtained by a cross type camera.

FIG. 9Cillustrates directions of a disparity vector with regard to a cross point in a cross type camera structure. Referring toFIG. 9C, the left view image930and the right view image940shown inFIG. 9Bare disposed above and below in view of the cross point. If such phenomenon is applied to the disparity vector955, the disparity vector955is disposed in a negative direction in an area950below the cross point905, whereas the disparity vector965is disposed in a positive direction in an area960above the cross point905. In more detail, the disparity vector between the left view image930and the right view image940is inverted with regard to the cross point905.

Therefore, views of first and second images are determined based on the direction of the disparity vector according to the location of the cross point in the cross type camera unlike the parallel type camera. For example, if the disparity vector of an area before the cross point is disposed in a positive direction, and the disparity of an area after the cross point is disposed in a negative direction, the first image is determined as the left view image930and the second image is determined as the right view image940. Likewise, if the disparity vector of the area before the cross point is disposed in the negative direction, and the disparity of the area after the cross point is disposed in the positive direction, the first image is determined as the right view image940and the second image is determined as the left view image930.

Since a stereoscopic image obtained by the cross type camera has the same disparity vector direction in the parallel type camera structure between the cross point and the camera, views of two cameras can be determined in the same manner as in the parallel type camera structure.

FIG. 10is a diagram for explaining a method of determining views by using an occlusion region. Referring toFIG. 10, if a post-processing control process such as a process of parallel shifting of a left view image and a right view image is additionally performed, the assumption described with reference toFIG. 8cannot be satisfied. The assumption is also not satisfied when complicated objects are entangled or the image is out of focus.

The view determining unit220uses the occlusion region for supplementing the method of determining views through the disparity estimation including the case where the assumption is not satisfied. The occlusion region refers to a region that is seen from a view but is not seen from another view. The occlusion region mainly occurs at boundary surfaces of an object.

A dark area1003of a hexahedron and a dark area1005of a cone are seen in a front image1000of the hexahedron, a sphere, and the cone. However, if a stereo camera photographs objects of the front image1000and obtains a stereoscopic image, a dark region1013is seen from the hexahedron disposed in left of the left view image, whereas a dark area of the hexahedron is not seen from the hexahedron of the right view image. In more detail, the dark area1003or1013of the hexahedron of the right view image is the occlusion region. Likewise, a dark region1025is seen from the cone disposed in right of the right view image, whereas a dark area of the cone is not seen from the left view image. In more detail, the dark area1005or1025of the cone of the left view image is the occlusion region.

Therefore, it is possible to supplement the method of determining views of the stereoscopic image using the occlusion region. If an area that is not seen from the first image is seen in a right area of an object from the second image with regard to boundary surfaces of the object, the second image is determined as the right view image, and the first image is determined as the left view image. Likewise, if an area that is not seen from the first image is seen in a left area of an object from the second image with regard to boundary surfaces of the object, the second image is determined as the left view image, and the first image is determined as the right view image.

The method of determining directions of the disparity vector is described above with reference toFIGS. 7A through 10. The view determining unit220determines the number of directions of the disparity vector and estimates a trend of the overall direction of all of the disparity vectors. The view determining unit220finally determines the view of the stereoscopic image in a direction having the greatest number of positive disparity vectors and negative disparity vectors among all the directions of the disparity vectors.

FIG. 11is a block diagram of an apparatus for controlling stereo synchronization according to an embodiment of the present invention. Referring toFIG. 11, an apparatus1100for outputting a stereoscopic image comprises an A/D board1110, a timing controller1120, and a stereo control device1160. The A/D board1110receives an analog stereoscopic image signal and converts the analog stereoscopic image signal into a digital stereoscopic image signal. The timing controller1120generates a perpendicular sync signal of the digital stereo image signal that is input from the A/D board1110. The stereo sync signal can be obtained using various methods. The stereo control device1160controls stereo synchronization of the stereoscopic image using the perpendicular sync signal and the stereo sync signal.

A conventional stereo synchronization device generates the stereo sync signal in addition to the stereoscopic image signal through a graphic card and decoder1130, and therefore requires an additional channel for transmitting an additional stereo sync signal. In a conventional toggle switch method, a toggle switch1140is required to be manually operated when a user desires and in order to convert to the stereo sync.

The apparatuses200and300for determining stereo views of the present invention analyze image data of the stereoscopic image signal that is input into a system in the view determining unit1150, thereby synchronizing views of the stereoscopic image according to the determined views. Therefore, the apparatuses200and300for determining stereo views do not need an additional stereo sync signal transmission channel or the toggle switch1140.

FIG. 12Aillustrates stereoscopic images in which a view arrangement order abruptly changes. Referring toFIG. 12A, when stereoscopic images1201through1216are sequentially processed, a view arrangement order of a pair of corresponding first and second images is analyzed. When the view arrangement order of the first and second images is left and right view images, the view arrangement order is indicated as LR. On the other hand, when the view arrangement order of the first and second images is right and left view images, the view arrangement order is indicated as RL.

A view arrangement order1111of the first image1201and a second image1202, a view arrangement order1224of a third image1203and a fourth image1204, a view arrangement order1226of a fifth image1205and a sixth image1206, a view arrangement order1228of a seventh image1207and an eighth image1208, a view arrangement order1230of a ninth image1209and a tenth image1210, a view arrangement order1232of an eleventh image1211and a twelfth image1212, a view arrangement order1234of a thirteenth image1213and a fourteenth image1214, and a view arrangement order1236of a fifteenth image1215and a sixteenth image1216are LR, LR, LR, RL, LR, LR, LR, and LR, respectively.

In more detail, the view arrangement orders are maintained as LR for1224,1224, and1226, are abruptly changed to RL for1228, and are changed back to LR for1230,1232,1234, and1236. Such an abrupt change in the view arrangement order may result in viewing fatigue.

FIG. 12Bis a diagram of a stereo synchronization unit according to an embodiment of the present invention.

In order to prevent viewing fatigue due to an abrupt change in the view arrangement order, the stereo synchronizing unit230performs stereo synchronization of stereoscopic images based on a current view arrangement order, a previous view arrangement order, and a next view arrangement order.

Referring toFIG. 12B, a view arrangement order1272of a first image1251and a second image1252, a view arrangement order1274of a third image1253and a fourth image1254, a view arrangement order1276of a fifth image1255and a sixth image1256, a view arrangement order1278of a seventh image1257and an eighth image1258, a view arrangement order1280of a ninth image1259and a tenth image1260, a view arrangement order1282of an eleventh image1261and a twelfth image1262, a view arrangement order1284of a thirteenth image1263and a fourteenth image1264, and a view arrangement order1286of a fifteenth image1265and a sixteenth image1266are LR, LR, LR, RL, RL, RL, RL, and RL, respectively.

Although the view arrangement orders are changed from LR for1272,1274, and1276to RL for1278, the stereo synchronization unit230does not directly apply the changed view arrangement order to a stereo synchronization operation but observes a next view arrangement order during a predetermined period. For example, if the view arrangement order is changed (i.e. from LR for1276to RL for1278), the view arrangement orders1278,1280,1282, and1284of the four pairs of images1257/12358,1259/1260,1261/1262, and1263/1264are observed, and the changed view arrangement order RL is maintained, the stereo synchronization unit230does not reflect the view arrangement order RL to the four pairs of images1257/12358,1259/1260,1261/1262, and1263/1264but performs the stereo synchronization based on the view arrangement order RL1286of a pair of images1265/1266.

The predetermined period that is observed for determining whether the changed view arrangement order is maintained may be short as perceived by a user although the stereo synchronization unit230does not perform the stereo synchronization according to an actual view arrangement order.

FIG. 13is a diagram of a stereo synchronization unit according to another exemplary embodiment of the present invention. Referring toFIG. 13, the apparatuses200and300for determining stereo views of the present invention analyze parallax from a stereoscopic image and determine views of the stereoscopic image. However, when the view arrangement order is already given, stereo synchronization will be performed according to the given view arrangement order.

Since a viewer feels fatigue if a reproduction of 2D images1301,1302, and1303is abruptly changed to a reproduction of 3D images1307through1315, the stereo synchronization unit230inserts blank images1304,1305, and1306for a predetermined frame period between a 2D image period and a 3D image period in order to supplement an abrupt change in a view reproduction mode. The frame period into which the blank images1304,1305, and1306are inserted must be a period of time short enough so that the viewer does not notice the existence of the blank images1304,1305, and1306.

The stereo synchronization unit230performs stereo synchronization according to the given view arrangement order when the blank images1304,1305, and1306finishes and when a reproduction mode of the 3D images1307through1315starts.

FIG. 14is a flowchart illustrating a method of determining stereo views of a stereoscopic image according to an exemplary embodiment of the present invention. Referring toFIG. 14, a pair of first and second images for a pair of a base view image and an additional view image that correspond to each other is selected from the stereoscopic image (operation1410).

When a frame of the stereoscopic image is a base view image or an additional view image, the frame is selected as the first image or the second image. If a frame of the stereoscopic image includes base view image information and additional view image information, a frame of the base view image and a frame of the additional view image are reconstructed from the frame of the stereoscopic image, and the reconstructed frames of the base view image and the additional view image are selected as the first image or the second image.

A parallax of the pair of the first and second images is used to determine the first image as one of the base view image and the additional view image, and the second image as another of the base view image and the additional view image (operation1420).

In the present embodiment, directions and angles of a disparity vector that are obtained by performing a disparity estimation with regard to the corresponding pair of images are used to determine which view each image is.

An x component of a parallel translation matrix of the pair of images is used as a direction of a difference component to determine each image view.

A method of determining views according to directions of the disparity vector may be changed according to whether a stereoscopic camera structure is a parallel structure or a cross structure. Each image view may be determined according to a direction of an occlusion region in order to supplement the method of determining views according to directions of the disparity vector.

The stereoscopic image is synchronized based on the result that the pair of first and second images is determined as one of the base view image and the additional view image (operation1430).

In order to prevent an abrupt change in the time arrangement order, the stereoscopic image is stereo synchronized according to a view arrangement order of a pair of current images, a view arrangement order of a pair of previous images, and a view arrangement order of a pair of next images.

Also, if a changed view arrangement order is not reflected to the stereo synchronization of the stereoscopic image but instead some view arrangement orders of next images are observed and the changed view arrangement order is maintained, then the stereoscopic image is stereo synchronized based on the changed view arrangement order.

If a view arrangement order is given, a 2D image is reproduced, and a 3D image is reproduced, and a blank image is reproduced during the frames after reproduction of the 2D image is completed and while the 3D image is continued to be reproduced based on the given view arrangement order.

The present invention can be embodied as computer readable code on a computer readable recording medium. The computer readable recording medium may be any data storage device that can store data which can be thereafter read by a computer system. Examples of the computer readable recording medium include read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and the like, as would be understood by one of skill in the art.

Exemplary methods and apparatuses for determining stereo views of a stereoscopic image of the present invention can determine stereo views using a parallax between stereoscopic images of different views without requiring an additional hardware channel for stereo sync information additional to the image data.

Exemplary methods and apparatuses may perform a disparity estimation in units of blocks or lines in order to more exactly and efficiently determine a stereo view of the stereoscopic image and determines stereo views according to directions of a disparity vector. Directions of the disparity vector that can be changed according to a stereo camera structure may be specifically analyzed, thereby analyzing stereo views of various stereoscopic images.

Also, exemplary methods and apparatuses may prevent an abrupt view change based on the determined view image and performs stereo synchronization so as to reproduce the stereoscopic image, thereby reducing fatigue resulting from viewing a 3D stereo image.

Therefore, exemplary methods and apparatuses for determining stereo views of a stereoscopic image can reproduce a conventional stereoscopic image format without an inversion of left and right view images, and deliver a correct stereoscopic effect of the stereoscopic image without an additional channel or a toggle switch for a display device, and without requiring the manual operation of the toggle switch device by the user.

Exemplary methods and apparatuses may efficiently utilize 3D multimedia content and maximize a user's convenience when the user uses a 3D display device. Exemplary methods and apparatuses may utilize a 3D multimedia content display device in a conventional image display device, thereby promoting the 3D display device.