Abstract:
A method and system detects and localizes an object by first acquiring a frame of a three-dimensional (3D) scene with a sensor, and extracting features from the frame. The frame are segmented into segments, wherein each segment includes one or more features, and for each segment, searching an object map for a similar segment, and only if there is a similar segment in the object map, registering the segment in the frame with the similar segment to obtain a predicted pose of the object. The predicted poses are combined to obtain the pose of the object, which can be outputted.

Description:
[0001]    This U.S. Non-Provisional Application is related to U.S. Non-Provisional application Ser. No. ______ (MERL-2882) co-filed herein with and incorporated herein by reference. That Application discloses a system and method for hybrid simultaneous localization and mapping of 2D and 3D data in images acquired by a red, green, blue, and depth sensor of a 3D scene. 
     
    
     FIELD OF THE INVENTION 
       [0002]    This invention relates generally to computer vision and image processing, and more particularly to detecting and tracking objects using images acquired by a red, green, blue, and depth (RGB-D) sensor and processed by simultaneous localization and mapping (SLAM). 
       BACKGROUND OF THE INVENTION 
       [0003]    Object detecting, tracking, and pose estimation can be used in augmented reality, proximity sensing, robotics, and computer vision applications using 3D or RGB-D data acquired by, for example, an RGB-D sensor such as Kinect®. Similar to 2D feature descriptors used for 2D-image-based object detection, 3D feature descriptors that represent the local geometry can be defined for keypoints in 3D point clouds. Simpler 3D features, such as point pair features, can also be used in voting-based frameworks. Those 3D-feature-based approaches work well for objects with rich structure variations, but are not suitable for detecting objects with simple 3D shapes such as boxes. 
         [0004]    To handle simple as well as complex 3D shapes, RGB-D data have been exploited. Hinterstoisser et al. define multimodal templates for the detection of objects, while Drost et al. define multimodal pair features for the detection and pose estimation, see Hinterstoisser et al., “Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes,” Proc. IEEE Int&#39;l Conf. Computer Vision (ICCV), pp. 858-865, November 2011, and Drost et al., “3D object detection and localization using multimodal point pair features,” in Proc. Int&#39;l Conf. 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), pp. 9-16, October 2012. 
         [0005]    Several systems incorporate object detection and pose estimation into a SLAM framework, see Salas-Moreno et al., “SLAM++: Simultaneous localization and mapping at the level of objects,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), June 2013, and Fioraio et al., “Joint detection, tracking and mapping by semantic bundle adjustment,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2013, pp. 1538-1545. Salas-Moreno et al. detect objects from depth maps and incorporate the objects as landmarks in a SLAM map for bundle adjustment. Their method only uses 3D data, and thus requires rich surface variations for objects. Fioraio et al. use a semantic bundle adjustment approach for performing SLAM and object detection simultaneously. Based on a 3D model of the object, they generate a validation graph that contains the object-to-frame and frame-to-frame correspondences among 2D and 3D point features. Their method lacks a suitable framework for object representation, resulting in many outliers after correspondence search. Hence, the detection performance depends on bundle adjustment, which might become slower as the map grows. 
       SUMMARY OF THE INVENTION 
       [0006]    The embodiments of our invention provide a method and system for detecting and localizing objects using a red, green, blue, and depth (RGB-D) image data acquired by a 3D sensor using hierarchical feature grouping. 
         [0007]    The embodiments use a novel compact representation of objects by grouping features hierarchically. Similar to a keyframe being a collection of features, an object is represented as a set of segments, where a segment is a subset of features in a frame. Similar to keyframes, segments are registered with each other in an object map. 
         [0008]    The embodiments use the same process for both offline object scanning and online object detection modes. In the offline scanning mode, a known object is scanned using a hand-held RGB-D sensor to construct an object map. In the online detection mode, a set of object maps for different objects are given, and the objects are detected via an appearance-based similarity search between the segments in the current image and in the object maps. 
         [0009]    If a similar segment is found, the object is detected and localized. In subsequent frames, the tracking is done by predicting the poses of the objects. We also incorporate constraints obtained from the object detection and localization into the bundle adjustment to improve the object pose estimation accuracy as well as the SLAM reconstruction accuracy. The method can be used in a robotic application. For example, the pose is used to pick up an object. Results show that the system is able to detect and pick up objects successfully from different viewpoints and distances. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0010]      FIG. 1  is a schematic of hierarchical feature grouping using object and SLAM maps according to embodiments of the invention; 
           [0011]      FIG. 2  is a schematic of a method and system for object detection and localization according to embodiments of the invention; and 
           [0012]      FIG. 3  is a schematic of a SLAM system and method according to embodiments of the invention. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0013]    Object Detection and Localization 
         [0014]    As shown in  FIG. 2 , the embodiments of our invention provide a method and system  200  for detecting and localizing objects in frames (images)  203  acquired of a scene  202  by, for example, a red, green, blue, and depth (RGB-D) sensor  201 . The method can be used in a simultaneous localization and mapping (SLAM) system and method  300  as shown in  FIG. 3 . In the figures generally, solid lines indicate processes and process flow, and dashed lines indicate data and data flow. The embodiments use segment sets  241  and represent an object in an object map  140  including a set of registered segment sets. 
         [0015]    Both an offline scanning and online detection modes are described in a single framework by exploiting the same SLAM method, which enables instant incorporation of a given object into the system. The invention can be applied to a robotic object picking application. 
         [0016]      FIG. 1  shows our hierarchical feature grouping. A SLAM map  110  stores a set of registered keyframes  115 , each associated with a set of features  221 . We use another hierarchy based on segments  241  to represent an object. A segment contains a subset of features  221  in a keyframe, and an object map  140  includes a set of registered segments. The object map is used for the object detection and pose estimation as described below. In our system, the segments can be generated by depth-based segmentation. 
         [0017]    One contribution of the invention is representing objects based on the hierarchical feature grouping as shown in  FIG. 1 . Just as a keyframe is a collection of features, a subset of features in a frame or image defines a segment. A keyframe-based SLAM system constructs the SLAM map  110  containing keyframes registered with each other. Similarly, we group a set of segments registered with each other to generate the object map  140  corresponding to the object. Because an instance of an object in a frame can contain multiple segments, the object map can contain multiple segments from a single frame. The object map provides a compact representation of the object observed under different viewpoint and illumination conditions. 
         [0018]    Our system exploits the same SLAM method to handle offline object scanning and online object detection modes. Both modes are essential to achieve an object detection and localization that can incorporate a given object instantly into the system. The goal of the offline object scanning is to generate the object map  140  by considering appearance and geometry information of known objects. We perform this process with user interaction. The system displays candidate segments that might correspond to the object to the user. Then, the user selects the segments corresponding to the object in each keyframe that is registered with the SLAM system. 
         [0019]    During online object detection, the system takes a set of object maps corresponding to different objects as the input, and then localizes these object maps with respect to the SLAM map that is generated during the online SLAM session. 
         [0020]    Our system first generates 240 sets of one or more segments  241  from each frame  203  using the depth-based segmentation procedure based on the features. For example, if the object is a box, for a particular view, the features described as be planes, edges and corners, which essentially are associated descriptors of the features. 
         [0021]    An appearance similarity search  260 , using vector of locally aggregated descriptors (VLAD) and the segment sets, is performed to determine similar sets of segments  266 . The searching  260  can use an appearance based similarity search of the object map  140 . If  262  the search is unsuccessful, the segment set is discarded  264 . 
         [0022]    Otherwise, if the search is successful, random sample consensus (RANSAC) registration  270  is performed to localize the segment set in the current frame with the object map. Set of segments with successful  275  RANSAC registration initiate objects in the SLAM map  110  as object landmark candidates. The pose of such objects can then be predicted  280 . 
         [0023]    The pose of each object landmark candidate is refined  285  by a prediction-based registration, and when it is successful, the candidate becomes an object landmark. The list of object landmarks are merged  286  by identifying the refined poses, i.e., if two object landmarks correspond to the same object map and have similar poses, then the landmarks are merged. The refining and merging steps are optional to achieve more accurate results. 
         [0024]    The output includes a detected object and pose  290 . The method can be performed in a processor connected to memory, input/output interfaces and the sensor by buses as known in the art. 
         [0025]    The method can be repeated for a next frame with the sensor at a different viewpoint and pose. 
         [0026]    In subsequent frames, we can use the same prediction-based registration and merging processes to track the object landmarks. Consequently, an object landmark in the SLAM map serves as the representation of the object in the real world. Note that this procedure applies to both the offline object scanning and online object detection modes. In the offline mode, the object map is incrementally constructed using the segment sets specified in the previous keyframes, while in the online mode the object map is fixed. 
         [0027]    Object Detection and Localization Via Hierarchical Feature Grouping 
         [0028]    Our object detection and tracking framework is based in part on a point-plane SLAM system, see Taguchi et al., “Point-plane SLAM for hand-held 3D sensors,” Proc. IEEE Int&#39;l Conf. Robotics and Automation (ICRA), pp. 5182-5189, May 2013. 
         [0029]    That point-plane SLAM system localizes each frame with respect to a SLAM map using both 3D points and 3D planes as primitives. An extended version uses 2D points as primitives and determines 2D-to-3D correspondences as well as 3D-to-3D correspondences to exploit information in regions where the depth is not available, e.g., the scene point is too close or too far from the sensor. 
         [0030]    Our segments include 3D points and 3D planes (but not 2D points) as features, while the SLAM procedure exploits all the 2D points, 3D points, and 3D planes as features to handle the case where the camera is too close or too far from the object and depth information is not available. 
         [0031]    Only segments that have similarity scores greater than a predetermined threshold are returned to eliminate segments that do not belong to any objects of interest. Then the set of segments in the frame are registered with the similar sets of segments in the object map. During the registration, we perform all-to-all descriptor similarity matching between the point features of the two segment sets followed by the RANSAC-based registration  270  that also considers all possible plane correspondences. The segment set that generates the largest number of inliers is used as the corresponding object. If  275  RANSAC fails for all of the k similar segment sets in the object maps, then the segment set extracted from the frame is discarded  264 . 
         [0032]    This step produces object landmark candidates. We consider these object landmarks as candidates, because the segments are only registered with a single segment set in the object map, not with the object map as a whole. An object can also correspond to multiple segments in the frame, resulting in repetitions in this list of object landmark candidates. Thus, we proceed with a pose refinement  285  and merging  286 . 
         [0033]    Prediction-Based Object Registration 
         [0034]    We project all point and plane landmarks of the object map to the current frame based on the predicted pose of the object landmark candidate. Matches between point measurements of the current frame and point landmarks of the object map are determined. We ignore unnecessary matches based on two rules:
   (i) a point measurement is matched with a point landmark when the projected landmark is within a r pixel neighborhood, for example, r is 10; and   (ii) a point measurement is matched with a point landmark when the landmark is at a similar viewing angle when the object map was constructed.   
 
         [0037]    The first rule avoids unnecessary point pairs that are too far on the object, and the second rule avoids performing matches for point landmarks that are behind the object from the current viewing angle of the frame. 
         [0038]    Similarly, a plane measurement is considered a candidate match when it is visible from the viewing angle used for the frame. Note that the object map is matched with the features included in the segments, and with all the features in the frame. Thus, this step does not assume any depth-based segmentation and can work with object landmark candidates initiated using other methods, e.g., 2D-image-based detection methods. 
         [0039]    Merging 
         [0040]    Because an object in the frame can include multiple segments, the list of object landmarks can include redundancies. Therefore, we merge  286  the object landmarks that have similar poses, belonging to the same object. 
         [0041]    SLAM System 
         [0042]      FIG. 3  is a schematic of a SLAM system and method  300  according to the embodiments of the invention that uses the object detection and localization as shown in  FIG. 2 . 
         [0043]    As before, frames are acquired  210 . In step  310 , we determine whether the SLAM map  110  includes any objects. If no, we apply the object detection and localization method  200  to the next frame to produce detected objects and poses  290 . If yes, we apply the prediction-based object localization  320 , followed by the object detection and localization  200 . Step  350  merges object poses. 
         [0044]    Step  360  determines if any of the detected objects are not in the SLAM map, i.e., the objects are new. If not, process the next frame  380 . Otherwise, add  370  a keyframe and the new object to the SLAM map  110 . 
         [0045]    SLAM Map Update 
         [0046]    In a SLAM system, the frame is added to the SLAM map as a keyframe when the pose is different from the poses of any existing keyframes in the SLAM map. We can also add a frame as a keyframe when the frame includes new object landmarks to initialize the object landmarks and maintain the measurement-landmark associations. 
         [0047]    Bundle Adjustment 
         [0048]    Bundel adjustment  340  can be applied to the SLAM map. Bundle adjustment refines the 3D coordinates describing the scene and relative motion obtained from images depicting the 3D points from different viewpoints. The refinement incorporates constraints obtained from the object detection and localization. 
         [0049]    A triplet (k, l, m) denotes an association between feature landmark p l  and feature measurement p m   k  of keyframe k with pose T k . Let I contain the triplets representing all such associations generated by the SLAM system in the current SLAM map. A tuple (k, l, m, o) denotes an object association, such that the object landmark o with pose {tilde over (T)} o  contains an association between the feature landmark p l   o  of the object map and feature measurement p m   k  in keyframe k. I o  contains the tuples representing such associations between the SLAM map and the object map. 
         [0050]    An error E kf  that comes from the registration of the keyframes in the SLAM map is 
         [0000]        E   kf ( p   1   , . . . , p   L   ; T   1   , . . . , T   K )=Σ (k,l,m)∈I    d ( p   l   , T   k   −1 ( p   m   k )),   (1)
 
         [0000]    where d(•,•) denotes the distance between a feature landmark and a feature measurement and T(f) denotes application of transformation T to the feature f. 
         [0051]    An error E obj  due to object localization is 
         [0000]        E   obj ( T   1   , . . . , T   K   ; {tilde over (T)}   1   , . . . , {tilde over (T)}   O )=Σ (k,l,m,o)∈I     o      d ( p   l   o   , {tilde over (T)}   o   T   k   −1 ( p   m   k )).   (2)
 
         [0052]    The bundle adjustment minimizes a total error with respect to the landmark parameters, keyframe poses, and object poses: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         arg 
                          
                         
                             
                         
                          
                         min 
                       
                       
                         
                           ∀ 
                           
                             T 
                             k 
                           
                         
                         , 
                         
                           
                             T 
                             ~ 
                           
                           o 
                         
                         , 
                         
                           p 
                           l 
                         
                       
                     
                      
                     
                       
                         E 
                         kf 
                       
                        
                       
                         ( 
                         
                           
                             p 
                             1 
                           
                           , 
                           … 
                            
                           
                               
                           
                           , 
                           
                             
                               p 
                               L 
                             
                             ; 
                             
                               T 
                               1 
                             
                           
                           , 
                           … 
                            
                           
                               
                           
                           , 
                           
                             T 
                             K 
                           
                         
                         ) 
                       
                     
                   
                   + 
                   
                     
                       
                         E 
                         obj 
                       
                        
                       
                         ( 
                         
                           
                             T 
                             1 
                           
                           , 
                           … 
                            
                           
                               
                           
                           , 
                           
                             
                               T 
                               K 
                             
                             ; 
                             
                               
                                 T 
                                 ~ 
                               
                               1 
                             
                           
                           , 
                           … 
                            
                           
                               
                           
                           , 
                           
                             
                               T 
                               ~ 
                             
                             O 
                           
                         
                         ) 
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
       Effect of the Invention 
       [0053]    The embodiments of the invention provide a method and system for detecting and tracking objects that can be used in a SLAM system. The invention provides a novel hierarchical feature grouping that uses segments, and represents an object as an object map including a set of registered segments. Both the offline scanning and online detection modes are described by a single framework exploiting the same SLAM procedure, which enables instant incorporation of a given object into the system. The method can be used in an object picking application. For example, the pose is used to pick up an object. 
         [0054]    The representations described herein are compact. Namely, there is an analogy between keyframe-SLAM map and segment-object map pairs, respectively. Both use the same features, i.e., planes, 3D points, and 2D points that are extracted from input RGB-D frames. 
         [0055]    Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.