Patent ID: 11948333
Assignee: PENG CHENG LABORATORY
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 4:
5. The disparity image fusion method for multiband stereo cameras according to claim 3, wherein the disparity fusion method is: the disparity diagram disparityA obtained by a binocular system in band A and a disparity diagram disparityB obtained by a binocular system in band B are fused according to the homography matrix H, the translation and rotation positional relationships between the binocular system in band A and the binocular system in band B and two confidence marker bits, which is specifically operated as follows:
(c-1) based on disparityA, calculating the coordinate of disparityB corresponding to each position according to the homography matrix, and a corresponding relationship between the coordinates of a pixel pointpA on disparityAand a pixel pointpB matched with pAon disparityB is as follows:

pA=H·pB  (10), wherein H is the homography matrix formed by mapping the disparity diagram obtained in band B to the disparity diagram obtained in band A;
(c-2) judging the disparity value and the confidence value of the matching point for the corresponding position; if the disparity values of the corresponding positions of pA and pB are both 0, no processing is conducted and disparity fusion is ended; if one of the disparity values is 0, the value of the corresponding position of disparityA is filled with non-zero party, and step (c-3) is conducted; if neither value is 0, the confidences of the corresponding positions of pA and pB are judged, and the corresponding position of disparityA is filled with the larger confidence; step (c-1) is conducted;
(c-3) firstly, transforming the disparity valuedB corresponding to position pB into a corresponding disparity value dA under a binocular camera coordinate system in band A, and then filling position pA; representing the coordinate at position pB under the corresponding camera coordinate system of the system in band B as (XB, YB, ZB),wherein the coordinate under the corresponding pixel coordinate system at position pB is represented as (uB, vB), and the coordinate under the corresponding pixel coordinate system at position pA is represented as (uA, vA);
converting pB to the B camera coordinate system:, {
     
      
       
        
         
          X
          B
         
         =
         
          
           
            Z
            B
           
           
            f
            B
           
          
          ⁢
          
           (
           
            
             u
             B
            
            -
            
             c
             ⁢
             
              u
              B
             
            
           
           )
          
         
        
       
      
      
       
        
         
          Y
          B
         
         =
         
          
           
            Z
            B
           
           
            f
            B
           
          
          ⁢
          
           (
           
            
             v
             B
            
            -
            
             c
             ⁢
             
              v
              B
             
            
           
           )
          
         
        
       
      
      
       
        
         
          Z
          B
         
         =
         
          
           
            b
            B
           
           ·
           
            f
            B
           
          
          
           d
           B
          
         
        
       
      
     
    
   
   
    
     (
     11
     )
    
   
  
 

wherein (cuB, cvB) is the principal point position of a reference image of device B; bB and fB are respectively the baseline length and the focal length of the device B; ZB is the coordinate in Z direction at pB; dB is a disparity value corresponding to position pB;
according to the positional relationship between the system in band A and the system in band B, converting pB to the camera coordinate system of the system in band B:

[X′A,Y′A,Z′A]T=R·[XB,YB,ZB]T+t, (X′A, Y′A, Z′A) is the coordinate obtained by converting the point pB under the camera coordinate system of the device in band B to the camera coordinate system of the device in band A; R is a rotation relationship quantity between the camera coordinate systems of the device B and device A, and t is a rotation relationship quantity between the camera coordinate systems of the device B and the device A;, {
     
      
       
        
         
          X
          A
          ′
         
         =
         
          
           
            Z
            
             u
             ⁢
             _
             ⁢
             A
            
            ′
           
           
            f
            A
           
          
          ⁢
          
           (
           
            
             u
             A
            
            -
            
             c
             ⁢
             
              u
              A
             
            
           
           )
          
         
        
       
      
      
       
        
         
          Y
          A
          ′
         
         =
         
          
           
            Z
            
             v
             ⁢
             _
             ⁢
             A
            
            ′
           
           
            f
            A
           
          
          ⁢
          
           (
           
            
             v
             A
            
            -
            
             c
             ⁢
             
              v
              A
             
            
           
           )
          
         
        
       
      
      
       
        
         
          Z
          A
          ′
         
         =
         
          
           
            b
            A
           
           ·
           
            f
            A
           
          
          
           d
           
            z
            ⁢
            _
            ⁢
            A
           
          
         
        
       
      
     
    
   
   
    
     (
     12
     )
    
   
  
 

wherein (cuA, cvA) is the principal point position of a reference image of the device A; bA and fA are respectively the baseline length and the focal length of the device A; Z′u_Ais a depth value obtained from the coordinate X′A; Z′v_A is a depth value obtained from the coordinate Y′A; dz_A is a disparity value obtained according to the coordinate Z′A;
from formula (12), obtaining:, {
     
      
       
        
         
          Z
          
           u
           ⁢
           _
           ⁢
           A
          
          ′
         
         =
         
          
           
            X
            A
            ′
           
           ·
           
            f
            A
           
          
          
           
            u
            A
           
           -
           
            c
            ⁢
            
             u
             A
            
           
          
         
        
       
      
      
       
        
         
          Z
          
           v
           ⁢
           _
           ⁢
           A
          
          ′
         
         =
         
          
           
            Y
            A
            ′
           
           ·
           
            f
            A
           
          
          
           
            v
            A
           
           -
           
            cv
            A
           
          
         
        
       
      
      
       
        
         
          
           Z
           A
          
          ⁢
          ′
         
         =
         
          
           
            b
            A
           
           ·
           
            f
            A
           
          
          
           d
           
            z
            ⁢
            _
            ⁢
            A
           
          
         
        
       
      
     
    
   
   
    
     (
     13
     )
    
   
  
 

thus, from formula (13), obtaining the disparity:, {
     
      
       
        
         
          d
          
           u
           ⁢
           _
           ⁢
           A
          
         
         =
         
          
           
            b
            A
           
           ·
           
            f
            A
           
           ·
           
            (
            
             
              u
              A
             
             -
             
              c
              ⁢
              
               u
               A
              
             
            
            )
           
          
          
           
            X
            A
            ′
           
           ·
           
            f
            A
           
          
         
        
       
      
      
       
        
         
          d
          
           v
           ⁢
           _
           ⁢
           A
          
         
         =
         
          
           
            b
            A
           
           ·
           
            f
            A
           
           ·
           
            (
            
             
              v
              A
             
             -
             
              c
              ⁢
              
               v
               A
              
             
            
            )
           
          
          
           
            Y
            A
            ′
           
           ·
           
            f
            A
           
          
         
        
       
      
      
       
        
         
          d
          
           z
           ⁢
           _
           ⁢
           A
          
         
         =
         
          
           
            b
            A
           
           ·
           
            f
            A
           
          
          
           Z
           A
           ′
          
         
        
       
      
     
    
   
   
    
     (
     14
     )
    
   
  
 

wherein du_A is a disparity value obtained according to the X direction relationship, dv_A is a disparity value obtained according to the Y direction relationship, and dz_A is a disparity value obtained according to the Z direction relationship;
calculating the disparity value that shall be filled at position pA;

dA=α·du_Aαβ·dv_A+γ·dz_A  (15)

wherein α+β+γ=1 and α, β and γ are obtained through experiments; filling dA in position pA.