Patent ID: 11971726
Assignee: WUHAN UNIVERSITY OF SCIENCE AND TECHNOLOGY
Field: Control (Instruments)
Classification: CPC G | IPC G

Claim 3:
4. The method according to claim 1, wherein
in 3), the semantic label of the non-wall corner in the view-angle color image is as follows:

*wk={*data(u,v)k,*types,k}

where, refers to information of the pixel in column u and row v in the view-angle color image at the k-th moment, and*types,k refers to an object category of the pixel in column u and row v in the view-angle color image at the k-th moment;
in 3), the predicted rectangular bounding box of the wall corner and the object type in the predicted rectangular bounding box of the wall corner in the view-angle color image are as follows:

*ek={data(x,y)k,(xnl,ynl,xnh,ynh,Tys)k}

where, *(xn1, yn1, xnh, Tys)k refers to a position of the n-th predicted rectangular bounding box in column x and row y in the view-angle color image at the k-th moment and a wall corner category, and data(x,y)k refers to information of the a pixel in column x and row y in the view-angle color image at the k-th moment;
in 3), coordinate transformation is performed for the semantic label of the non-wall corner in the view-angle color image, and the predicted rectangular bounding box of the wall corner and the object type in the predicted rectangular bounding box in the view-angle color image of the wall corner as follows:
obtaining a pixel coordinate set of non-wall corner semantics in a pixel coordinate system of the view-angle color image in combination with *wk and *ek as follows:

Wk={((iw,jw),type(iw,b,jw))b},b∈[1,B]

where, B refers to a total number of non-wall corner semantic pixel points in the current picture in the set; (iw,jw) refers to a pixel point in row iw and column jw in the picture, and ((iw,jw),type(iw,jw))b refers to that a pixel coordinate of the b-th semantic pixel point in the pixel coordinate set is (iw,jw) and the pixel label is type(iw,jw); and
obtaining a pixel coordinate set of a wall corner semantics in the pixel coordinate system of the view-angle color image as follows:

Ek={((ie,je),Ty(ie,je))t},t∈[1,T]

where, T refers to a total number of wall corner semantic pixel points in the current picture in the set; (ie,je) refers to the pixel point in row ie and column je in the picture, and ((ie,je),Ty(ie,je))t refers to that a pixel coordinate of the t-th semantic pixel point in the pixel coordinate set is (ieje) and the pixel label is Ty(ie,je);
for the coordinate (iw,jw)b of the non-wall corner semantics in the current color image pixel coordinate system obtained above, the depth information is obtained by using the acquired depth map, and the camera calibration parameters are transformed into a camera coordinate system, so as to obtain the three-dimensional point cloud coordinate of the non-wall corner as follows:, {
  
   
    
     
      
       
        
         Z
         w
        
        =
        
         
          d
          w
         
         /
         1000
        
       
      
     
    
    
     
      
       
        X
        w
       
       =
       
        
         (
         
          
           i
           w
          
          -
          
           c
           x
          
         
         )
        
        ·
        
         
          Z
          w
         
         /
         
          f
          x
         
        
       
      
     
    
    
     
      
       
        Y
        w
       
       =
       
        
         (
         
          
           j
           w
          
          -
          
           c
           y
          
         
         )
        
        ·
        
         
          Z
          w
         
         /
         
          f
          y
         
        
       
      
     
    
   
   ;
  
 

where, (Xw,Yw,Zw) refer to the three-dimensional point cloud coordinates of the non-wall corner, the coordinate (iw-jw) of each corresponding pixel point in the non-wall corner semantic pixel coordinate set refer to that the pixel point is located in row iw and column jw in the current color picture, and dw refers to a depth value of the pixel coordinate (iw,jw) in the depth image; cx, cy, fx and fy are camera calibration internal parameters, where cx and cy refer to transverse and vertical pixel numbers of a difference between a center pixel coordinate of the image and an origin pixel coordinate of the image respectively, (cx,cy) is an optical center, and fx and fy refer to transverse and vertical distances from a focal point of a camera to the optical center of the camera respectively;
for the coordinate (ie,je)t of the wall corner semantics in the current color image pixel coordinate system obtained above, the depth information is obtained by using the acquired depth map, and the camera calibration parameters are transformed into the camera coordinate system, so as to obtain the three-dimensional point cloud coordinate of the wall corner;
the point cloud coordinate (Xe,Ye,Ze) of the wall corner semantics in the camera coordinate system is obtained as follows:, {
  
   
    
     
      
       
        
         Z
         e
        
        =
        
         
          d
          e
         
         /
         1000
        
       
      
     
    
    
     
      
       
        X
        e
       
       =
       
        
         (
         
          
           i
           e
          
          -
          
           c
           x
          
         
         )
        
        ·
        
         
          Z
          e
         
         /
         
          f
          x
         
        
       
      
     
    
    
     
      
       
        Y
        e
       
       =
       
        
         (
         
          
           j
           e
          
          -
          
           c
           y
          
         
         )
        
        ·
        
         
          Z
          e
         
         /
         
          f
          y
         
        
       
      
     
    
   
   ;
  
 

where, (Xe,Ye,Ze) refers to a three-dimensional point cloud coordinate of the wall corner obtained through transformation, the coordinate (ieje) of each corresponding pixel point in the wall corner semantic pixel coordinate set refers to that the pixel point is located in the row ie and column je in the current color picture, and de refers to a depth value of the pixel coordinate (ie,je) in the depth image; cx, cy, fx and fy are camera calibration internal parameters, cx and cy refer to transverse and vertical pixel numbers of a difference between a center pixel coordinate of the image and an origin pixel coordinate of the image respectively, (cx, cy) is the optical center, and fx and fy refer to transverse and vertical distances from the focal point of the camera to the optical center of the camera respectively;
in 3), point cloud filtering is performed for the three-dimensional point cloud coordinate (Xw,Yw,Zw) of the non-wall corner and the three-dimensional point cloud coordinate (Xe,Ye,Ze) of the wall corner respectively by using the filter based on the statistical method as follows:
after discrete point clouds are removed from point cloud data by using the filter based on the statistical method, extracting a cluster of point clouds closest to the robot platform from the object point clouds which is equivalent to extracting outer contour point clouds of the object at a view angle, so as to obtain the filtered non-wall corner semantic point cloud coordinate (X′w,Y′w,Z′w) and the filtered wall corner semantic point cloud coordinate (X′e,Y′e,Z′e).