Patent Publication Number: US-8525879-B2

Title: Depth detection method and system using thereof

Description:
This application claims the benefit of Taiwan application Serial No. 098143011, filed Dec. 15, 2009, the subject matter of which is incorporated herein by reference. 
     TECHNICAL FIELD 
     The disclosure relates in general to a depth detection system, and more particularly to a depth detection system for obtaining more reliable depth data using the motion adjustment reference mask technology. 
     BACKGROUND 
     In the modern age, in which the technology is growing with each passing day, the digital content industry including computer motion pictures, digital games, digital learning and mobile applications and services is developed in a flourishing manner. In the existing technology, the stereoscopic image/video has existed, and is expected to enhance the service quality of the digital content industry. 
     Generally speaking, the conventional depth data detecting system adopts the dual camera technology to shoot the target at left and right viewing angles to obtain the left video data and the right video data, and calculates the depth data of each corresponding object according to horizontal offsets between the corresponding left and right video data of the corresponding objects. Generally speaking, the accuracy of the depth data significantly affects the quality of the stereoscopic image data. Thus, it is an important subject of this field to design a depth detection system capable of generating the accurate depth data. 
     SUMMARY 
     The disclosure is directed to a depth detection system adopting a depth estimation apparatus to estimate similarity data of pixel data between left viewing angle video data and right viewing angle video data; to generate a converging parameter through a reference mask according to the similarity data in a selected reference region; and to perform a cyclic iteration operation on the similarity data according to the converging parameter so as to obtain the disparity of each pixel data in the left/right viewing angle video data. The depth detection system of the disclosure further adopts the depth estimation apparatus to verify the disparity, and selectively adjusts the size of the reference mask according to the verified result so as to obtain the disparity of each pixel data with the higher reliability and to correspondingly generate depth information. Thus, compared with the conventional depth detection system, the depth detection system of the disclosure generates the depth information with the higher reliability. 
     According to a first aspect of the present disclosure, a depth detection system including a dual camera apparatus, a horizontal calibration apparatus and a depth estimation apparatus is provided. The dual camera apparatus shoots first video data and second video data, which respectively correspond to a first viewing angle and a second viewing angle. Each of the first and second video data include r×c sets of pixel data, wherein r and c are natural numbers greater than 1. The horizontal calibration apparatus performs horizontal calibration on the first and second video data, and outputs the first and second video data, which are horizontally calibrated. The depth estimation apparatus includes a similarity estimation module, an iteration update module and a control module. The similarity estimation module compares pixel data of the first and second video data, provided by the horizontal calibration apparatus, with each other to obtain initial similarity data, which include r×c initial similarity elements each including d initial similarity elements, wherein d is a natural number greater than 1. The iteration update module selects multiple initial similarity elements to perform an accumulation operation to obtain an iteration parameter according to a reference mask with each of the initial similarity elements serving as a center. The iteration update module performs n times of iteration update operations on the initial similarity data according to the iteration parameter to generate updated similarity data, which include r×c update similarity elements each including d similarity elements. The control module judges whether each of the r×c update similarity elements satisfies a character verification condition. When the r×c update similarity elements satisfy the character verification condition, the control module converts the r×c update similarity elements into depth distribution data. 
     According to a second aspect of the present disclosure, a depth detection method is provided. The method includes the following steps. First, first video data and second video data, which respectively correspond to a first viewing angle and a second viewing angle, are shot. Each of the first and second video data include r×c sets of pixel data, wherein r and c are natural numbers greater than 1. Next, horizontal calibration is performed on the first and second video data. Then, pixel data of the horizontally calibrated first and second video data are compared with each other to obtain initial similarity data. The initial similarity data include r×c initial similarity elements. Each of the r×c initial similarity data include d initial similarity elements, wherein d is a natural number greater than 1. Next, multiple similarity elements are selected according to a reference mask with each of the similarity elements serving as a center, and an accumulation operation is performed on the selected similarity elements to obtain an iteration parameter. Then, n times of iteration update operations are performed on the initial similarity data according to the iteration parameter to generate r×c update similarity elements each including d similarity elements. Next, it is judged whether each of the r×c update similarity elements satisfies a character verification condition. Then, the r×c update similarity elements are converted into depth distribution data when the r×c update similarity elements satisfy the character verification condition. 
     The disclosure will become apparent from the following detailed description of the preferred but non-limiting embodiments. The following description is made with reference to the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram showing a depth detection system according to an embodiment of the disclosure. 
         FIG. 2  is a detailed block diagram showing a depth estimation apparatus  14  of  FIG. 1 . 
         FIG. 3  is a schematic illustration showing r×c×d initial similarity elements in initial similarity data Dis. 
         FIG. 4  is a detailed block diagram showing a range estimation apparatus  18  of  FIG. 1 . 
         FIG. 5  is a flow chart showing a depth detection method according to the embodiment of the disclosure. 
         FIG. 6  is a partial flow chart showing the depth detection method according to the embodiment of the disclosure. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
       FIG. 1  is a block diagram showing a depth detection system  1  according to an embodiment of the disclosure. Referring to  FIG. 1 , the depth detection system  1  of this embodiment includes a dual camera apparatus  10 , a horizontal calibration apparatus  12  and a depth estimation apparatus  14 . The dual camera apparatus  10  shoots video data V 1  and V 2 , which respectively correspond to a first viewing angle and a second viewing angle. For example, the video data V 1  and V 2  are respectively the video data of the left viewing angle and the right viewing angle shot on the same target. The video data V 1  and V 2  include, for example, r×c sets of pixel data, wherein r and c are natural numbers greater than 1. 
     The horizontal calibration apparatus  12  performs horizontal calibration on the video data V 1  and V 2 , and provides horizontally calibrated video data Vhc 1  and Vhc 2  to the depth estimation apparatus  14 . 
     The depth estimation apparatus  14  generates depth distribution data Dd according to the video data Vhc 1  and Vhc 2 .  FIG. 2  is a detailed block diagram showing the depth estimation apparatus  14  of  FIG. 1 . For example, the depth estimation apparatus  14  includes a similarity estimation module  14   a , an iteration update module  14   b  and a control module  14   c.    
     The similarity estimation module  14   a  compares pixel data on the video data Vhc 1  and Vhc 2  to obtain initial similarity data Dis. For example, the initial similarity data Dis include r×c initial similarity elements each including d initial similarity elements, wherein d is a natural number greater than 1. For example, based on each of the r×c sets of pixel data corresponding to the video data Vhc 1  of the left viewing angle, the similarity estimation module  14   a  selects a search window including d sets of pixel data on the video data Vhc 2  corresponding to the right viewing angle and compares each set of pixel data in the video data Vhc 1  with the d sets of pixel data in the search window to obtain the corresponding d initial similarity elements. 
     In one example embodiment, in respect of the pixel data Vhc 1 (R,C) of the video data Vhc 1  corresponding to the position (R,C), the similarity estimation module  14   a  defines a corresponding search window with the pixel data Vhc 2 (R,C), Vhc 2 (R,C+1), Vhc 2 (R,C+2), . . . , Vhc 2 (R,C+d), which are respectively corresponding to the positions (R,C), (R,C+1), (R,C+2), . . . , (R,C+d), of the video data Vhc 2 , wherein R and C are natural numbers respectively smaller than or equal to r and c. The similarity estimation module  14   a  further compares the pixel data Vhc 1 (R,C) with each of the sets of pixel data Vhc 2 (R,C) to Vhc 2 (R,C+d) in the search window to correspondingly obtain the d initial similarity elements. 
     For example, each of the d initial similarity elements in each of the r×c initial similarity elements may be represented by the following equation:
 
 L   0 ( x,y,z )=δ( Vhc 1, Vhc 2, x,y,z )| x= 1,2 , . . . ,r;y= 1,2 , . . . ,c;z= 1,2 , . . . ,d,  
 
wherein the δ function is the similarity function of the image. Because the initial similarity data Dis include the r×c initial similarity elements and each initial similarity element includes the d initial similarity elements, the r×c×d initial similarity elements L 0 (x,y,z)|x=1, 2, . . . , r; y=1, 2, . . . , c; z=1, 2, . . . , d in the initial similarity data Dis may be represented by a three-dimensional similarity space, as shown in  FIG. 3 .
 
     The iteration update module  14   b  selects multiple similarity elements according to one reference mask M with each similarity element in the three-dimensional similarity space serving as a center, and performs an accumulation operation on the selected similarity elements to obtain an iteration parameter P n (n=0, 1, . . . , N). The iteration update module  14   b  further performs N times of iteration update operations on the initial similarity data Dis according to the iteration parameter to generate updated similarity data Dus according to the initial similarity data Dis, wherein N is a natural number greater than 1. Similar to the initial similarity data Dis, the updated similarity data Dus include r×c initial similarity elements, wherein each of the r×c update similarity elements includes d update similarity elements. 
     For example, the iteration update module  14   b  performs the iteration update operation on the initial similarity data Dis according to the iteration parameter and according to the following function:
 
 L   n+1 ( x,y,z )= L   0 ( x,y,z )× P   n   |n= 0,1 , . . . , N  
 
     The iteration parameter P n , relates to the accumulation function S n  (x,y,z). For example, the iteration parameter P n , and the accumulation function S n (x,y,z) respectively satisfy, for example, the following equations: 
                 S   n     ⁡     (     x   ,   y   ,   z     )       =       ∑     (       r   ′     ,     c   ′     ,     d   ′       )       ⁢       L   n     ⁡     (       x   +     r   ′       ,     y   +     c   ′       ,     z   +     d   ′         )                       P   n     =       (         S   n     ⁡     (     x   ,   y   ,   z     )           ∑       (       r   ″     ,     c   ″     ,     d   ″       )     ∈     φ   ⁡     (     x   ,   y   ,   z     )           ⁢       S   n     ⁡     (       r   ″     ,     c   ″     ,     d   ″       )           )     α           
wherein x+r′, y+c′ and z+d′ represent the reference range selected from the three-dimensional similarity space of  FIG. 3  using the reference mask M with the size of r′×c′×d′ and with the coordinates (x,y,z) serving as the center on the condition that the similarity element L 0 (x,y,z) corresponding to the pixel data Vhc 1 ( x,y ) serves as a center point; α is a constant parameter; the accumulation function S n  (x,y,z) represents the accumulation operation of the similarity elements performed on the selected reference range; and the function
 
               ∑       (       r   ″     ,     c   ″     ,     d   ″       )     ∈     φ   ⁡     (     x   ,   y   ,   z     )           ⁢       S   n     ⁡     (       r   ″     ,     c   ″     ,     d   ″       )             
represents the reference summation parameter of the accumulation function S n  (x,y,z) in another selected reference range φ(x,y,z).
 
     The control module  14   c  receives the updated similarity data Dus and judges whether each of the r×c update similarity elements in the updated similarity data Dus satisfies a character verification condition. In one example embodiment, the character verification condition is whether each of the r×c update similarity elements obviously has one unique update similarity element. For example, the control module  14   c  substrates an summation average thereof from the d update similarity elements of the r×c update similarity elements, and judges whether the obtained value is greater than a threshold value to judge whether each of the r×c update similarity elements obviously has the unique update similarity element. 
     When each of the r×c update similarity elements includes one unique update similarity element, it represents that each of the r×c sets of pixel data of the video data Vhc 1  may be mapped to the r×c sets of pixel data of the video data Vhc 2  through the r×c unique update similarity elements in the updated similarity data Dus. Thus, the control module  14   c  can obtain the horizontal displacement quantity of each of the r×c sets of pixel data of the video data Vhc 1 , and the horizontal displacement quantities indicate the horizontal displacements of each of the r×c sets of pixel data of the video data Vhc 1  relative to the pixel data of the video data Vhc 2  having the same image content. Based on the condition that the horizontal distance between the pixel data of the video data Vhc 1  and Vhc 2  relates to the depth of its corresponding image content, the control module  14   c  generates the depth distribution data Dd according to the horizontal displacement quantity. 
     When each of the r×c update similarity elements does not include the unique update similarity element, it represents that each of the r×c update similarity elements cannot definitely indicate the corresponding relationship between the r×c sets of pixel data of the video data Vhc 1  and Vhc 2 . Thus, the control module  14   c  cannot obtain the horizontal displacement quantity, relative to the pixel data of the video data Vhc 2 , of each of the r×c sets of pixel data of the video data Vhc 1  and the corresponding depth distribution data Dd. In this case, the control module  14   c  adjusts the size of the reference mask M to try to refer to more update similarity elements (i.e. select more sets of pixel data by a larger mask in the video data V 1 ) by enlarging the size of the reference mask M when calculating the accumulation function S n (x,y,z). Thus, the opportunity of referring to the mask of the video data V 1  with the texture characteristic can be increased. 
     Thereafter, the control module  14   c  transfers the size (M_size) of the reference mask M back to the iteration update module  14   b  to drive the iteration update module  14   b  to recalculate the iteration parameter according to the adjusted reference mask M, and to regenerate the updated similarity data according to the recalculated iteration parameter. The control module  14   c  further judges whether each of the r×c update similarity elements in the updated similarity data Dus satisfies the character verification condition according to the regenerated updated similarity data Dus. If so, the control module  14   c  may generate the depth distribution data Dd according to the updated similarity data Dus. If not, the control module  14   c  again adjusts the size of the reference mask M and repeats the above-mentioned operation. Thus, the depth detection system  1  according to the embodiment of the disclosure can obtain the similarity data relating to the video data Vhc 1  and Vhc 2 , which has the higher reliability, by the method of dynamically adjusting the size of the reference mask M, so that the depth distribution data Dd with the higher reliability can be obtained. 
     In one example, the depth detection system  1  according to the embodiment of the disclosure further includes a characteristic analyzing apparatus  16  for analyzing the characteristic region in the horizontally calibrated video data Vhc 1  and Vhc 2 , and a range estimation apparatus  18  for estimating the possible depth range of the video data Vhc 1  and Vhc 2  according to the analyzed result of the characteristic region to generate depth range data Ddr. 
     More specifically speaking, the characteristic analyzing apparatus  16  receives and analyzes the video data Vhc 1  and Vhc 2  provided by the horizontal calibration apparatus  12  to obtain characteristic region data Dca 1  from the video data Vhc 1 , and to obtain characteristic region data Dca 2  from the video data Vhc 2 , wherein each of the characteristic region data Dca 1  and Dca 2  includes multiple sets of corresponding minutia point data. For example, the characteristic analyzing apparatus  16  obtains multiple sets of minutia point data by the object dividing technology to indicate several image content objects in the video data Vhc 1  and thus to obtain the characteristic region data Dca 1 . For example, the characteristic region data Dca 1  include the video data of the video data Vhc 1  for displaying the user&#39;s hand (usually having the minimum depth), and the video data of the background region (usually having the maximum depth) in the video data. The same object dividing technology is also applied to the video data Vhc 2  to obtain the characteristic region data Dca 2  from the video data Vhc 2 , wherein the characteristic region data Dca 2  include multiple sets of minutia point data corresponding to the minutia point data in the characteristic region data Dca 1 . 
       FIG. 4  is a detailed block diagram showing the range estimation apparatus  18  of  FIG. 1 . Referring to  FIG. 4 , the depth range estimation apparatus  18  includes, for example, an estimation module  18   a , a statistics module  18   b  and an operation module  18   c . The estimation module  18   a  calculates multiple horizontal displacement quantities between each minutia point in the characteristic region data Dca 1  and each corresponding minutia point in the characteristic region data Dca 2 , and thus converts the horizontal displacement quantities into multiple sets of depth data. Similar to the operation of the control module  14   c , the estimation module  18   a  obtains the depth data Ddc for indicating its corresponding estimated depths according to the horizontal displacement quantities between the characteristic region data Dca 1  in the video data Vhc 1  and the characteristic region data Dca 2  in the video data Vhc 2 . 
     The statistics module  18   b  converts the depth data Ddc into depth statistics distribution data Ddch, and counts the number of minutia points, which may be accumulated on a range of multiple possible depths. For example, the depth statistics distribution data Ddch may be represented by a statistics histogram to indicate the relationship between the depth value and the number of the minutia points thereon. 
     The operation module  18   c  obtains a minimum depth value and a maximum depth value from the depth statistics distribution data Ddch according to a critical condition, determines the depth range data Ddr corresponding to the video data Vhc 1  and Vhc 2  according to the minimum and maximum depth values, and outputs the depth range data Ddr to the depth estimation apparatus  14 . For example, the critical condition is the critical number of the minutia points corresponding to the same depth value. When the minimum depth value is searched, the operation module  18   c  starts the search from the corresponding minimum depth in the depth statistics distribution data Ddch. Once the number of the corresponding minutia points is found to be greater than or equal to the critical number of the minutia points, the operation module  18   c  serves as the minimum depth value. Similarly, when the maximum depth value is searched, the operation module  18   c  starts the search from the corresponding maximum depth of the depth statistics distribution data Ddch to find the maximum depth where the number of the corresponding minutia points is greater than or equal to of the critical number of the minutia points. 
     Thus, the depth estimation apparatus  14  can generate the depth distribution data Dd based on the depth range data Ddr. For example, the similarity estimation module  14   a  in the depth estimation apparatus  14  can determine the value d (i.e., the size of the search window for similarity calculation) according to the depth range data Ddr. 
       FIG. 5  is a flow chart showing a depth detection method according to the embodiment of the disclosure. First, as shown in step (a), the dual camera apparatus  10  shoots the video data V 1  and V 2 , which respectively correspond to the first viewing angle (e.g., the left viewing angle) and the second viewing angle (e.g., the right viewing angle), wherein each of the video data V 1  and V 2  includes r×c sets of pixel data. Next, as shown in step (b), the horizontal calibration apparatus  12  performs horizontal calibration on the video data V 1  and V 2  to provide horizontally calibrated video data Vhc 1  and Vhc 2 . 
     Then, as shown in step (c), the similarity estimation module  14   a  of the depth estimation apparatus  14  compares the pixel data of the video data Vhc 1  and Vhc 2  to obtain the initial similarity data Dis, which include r×c×d initial similarity elements. Next, as shown in step (d), the iteration update module  14   b  of the depth estimation apparatus  14  selects multiple similarity elements according to the reference mask M with each similarity element serving as the center, and performs the accumulation operation on the selected similarity element to obtain the iteration parameter P n . 
     Next, as shown in step (e), the iteration update module  14   b  further performs n times of iteration update operations on the initial similarity data Dis according to the iteration parameter P n  to generate the r×c update similarity elements (i.e., the r×c×d initial similarity elements) according to the r×c initial similarity elements (i.e., the r×c×d initial similarity elements). Then, as shown in step (f), the control module  14   c  judges whether each of the r×c update similarity elements satisfies a character verification condition. If so, step (g) is performed so that the control module  14   c  converts the r×c update similarity elements into the depth distribution data Dd. If not, step (h) is performed so that the control module  14   c  adjusts the size of the reference mask M and then the steps (d), (e) and (f) are repeated to repeatedly obtain the iteration parameter P n , generate the r×c update similarity elements and judge whether each of the r×c update similarity elements satisfies the character verification condition. After the step (f), the step (g) is performed if each of the r×c update similarity elements satisfies the character verification condition, and steps (h) and (d) to (f) are repeated if each of the r×c update similarity elements does not satisfy the character verification condition. 
       FIG. 6  is a partial flow chart showing the depth detection method according to the embodiment of the disclosure. In one example, the method further includes, between the steps (b) and (c), the steps (i) to (l) for calculating the depth range data Ddr to speed up the operation of the step (c). As shown in the step (i), the characteristic analyzing apparatus  16  analyzes the horizontally calibrated video data Vhc 1  and Vhc 2  to obtain the characteristic region data Dca 1  from the video data Vhc 1  and obtain the characteristic region data Dca 2  from the video data Vhc 2 . The characteristic region data Dca 1  and Dca 2  include several corresponding pairs of minutia points. Next, as shown in the step (j), the estimation module  18   a  calculates the horizontal displacement quantities between the minutia points of the characteristic region data Dca 1  and Dca 2 , and converts the horizontal displacement quantities into the depth data Ddc corresponding to each of the minutia points. 
     Then, as shown in the step (k), the statistics module  18   b  converts the depth data Ddc into the depth statistics distribution data Ddch. Thereafter, as shown in the step (l), the operation module  18   c  obtains the minimum depth value and the maximum depth value from the depth statistics distribution data Ddch according to a critical condition, and determines the depth range data Ddr corresponding to the video data Vch 1  and Vch 2  according to the minimum and maximum depth values. 
     The depth detection system according to the embodiment of the disclosure adopts the depth estimation apparatus to estimate the similarity data of the pixel data between the left viewing angle video data and the right viewing angle video data; generates the converging parameter through the reference mask according to the similarity data in a selected reference region; and performs the cyclic iteration operation on the similarity data according to the converging parameter to obtain the disparity of each pixel data in the left viewing angle/right viewing angle video data. The depth detection system according to the embodiment of the disclosure further adopts the depth estimation apparatus to verify the disparity, and selectively adjusts the size of the reference mask according to the verified result to obtain the disparity of each pixel data with the higher reliability and the correspondingly generated depth information. Thus, compared with the conventional depth detection system, the depth detection system according to the embodiment of the disclosure generates the depth information with the higher reliability. 
     In addition, the depth detection system according to the embodiment of the disclosure further adopts the characteristic analyzing apparatus and the range estimation apparatus to obtain the possible depth range for the left viewing angle video data and the right viewing angle video data. Thus, the depth detection system according to the embodiment of the disclosure further increases the operation speed of obtaining the initial similarity data and the depth information. 
     While the disclosure has been described by way of example and in terms of a preferred embodiment, it is to be understood that the disclosure is not limited thereto. On the contrary, it is intended to cover various modifications and similar arrangements and procedures, and the scope of the appended claims therefore should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements and procedures.