Patent Publication Number: US-11657506-B2

Title: Systems and methods for autonomous robot navigation

Description:
BACKGROUND 
     Autonomous robot navigation requires a good localization and mapping component. A fully autonomous robot utilizes a dense mapping of the environment, which presents segmentation of objects in the environment. The robot can use such information to perform decision making, and complex mobile manipulation. Conventional sparse modeling algorithms execute quickly—for sparse environments the results can provide adequate robot localization. The accuracy of the conventional sparse modeling algorithm localization is impacted by the limited mapping and visualization input data. Conventional semi-dense environment algorithms can execute at a relatively slow pace, but are useful for some situations—for example, autonomous driving. However, the semi-dense information is not adequate to analyze the environment semantically. Conventional dense modeling algorithms are slow and require repeated processing of one single scene. Conventional dense modeling can have application in situations needing complex situational understanding and corresponding task planning. 
     Conventional environmental modeling systems describe features appearing on a two-dimensional (2D) image. Even with robust featuring matching, conventional 2D matching and optimization techniques result in errors that can affect the precision of the mapping transformation matrix. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    illustrates a process for implementing autonomous robot navigation in accordance with embodiments; and 
         FIG.  2    illustrates a system for implementing autonomous robot navigation in accordance with embodiments. 
     
    
    
     DESCRIPTION 
     Embodying systems and methods provide a dense modeling approach for environmental modeling for robot autonomous navigation based on a semantic map created by applying prebuilt image models to captured images of the robot&#39;s environment. 
     Embodying 
     systems and methods create a dense and semantic map of the robot&#39;s environment using existing image files as models for recognition. These image files can be computer-aided design (CAD) models (2D, three-dimensional (3D) wireframe, 3D solid modeling), digital photographs, drawings, and the like. It should be readily understood that embodying systems and methods are not limited to any one image file type and/or nature. For convenience, the following discussion discloses systems and methods that use models from CAD image files. However, other implementations of image files are within the scope of this disclosure. 
     This system can utilize CAD models, match them to objects in the environment captured in images, and map them in the environment. Because the CAD models are dense, the resultant modeling from embodiments is dense as well. Using CAD models results in more robust pose estimation results, which yield increased triangulation accuracy over conventional feature-based matching and optimization techniques. 
     Embodying systems and methods execute significantly faster than conventional surface estimation and modeling approaches; and perform dense modeling faster than conventional dense modeling approaches. Embodying dense modeling approaches improve robot autonomous navigation by providing high resolution results coupled with fast execution speed. 
     In accordance with embodiments, techniques for object segmentation, object matching, object pose estimation, are applied together in an active relearning approach. Object segmentation/matching begins with capturing images of objects in the robot&#39;s environment—including, but not limited to assembly parts, industrial plants, power generation turbines, structures, obstacles, persons, etc.). 
     Various imaging technologies can be used to capture these environmental images. The imaging device can be implemented in the visual spectrum with black and white or color capabilities, or in other spectrums (e.g., infrared, microwave, millimeter, laser, x-ray, etc.). In some implementations, the imaging device can include depth sensing capability (e.g., a red/green/blue plus depth camera (RGB-D)). 
     Object Segmentation 
     In accordance with embodiments, object segmentation is achieved by applying a deep learning-based approach implemented in deep neural networks (DNN) to segment objects from the environment. During a model training process, an object is placed in the environment with annotated pixels. In accordance with embodiments, a foreground DNN performs foreground segmentation, a background DNN performs background segmentation, and a refinement DNN refines the segmentation results. These three neural networks combine to cooperatively create a neural network. 
     Adversarial training can train both the foreground and the background DNNs to recognize boundaries between the foreground and background portions of the images. These two DNNs actively identify the pixels of an input image as foreground elements and background elements. 
     An adversarial strategy is applied to maximize the score and/or probability of correct identification by the two DNNs. In accordance with implementations, Fisher&#39;s linear discriminant approach can be used as an adversarial strategy. The identification information is utilized to maximize the objective represented by equation 1: 
     
       
         
           
             
               
                 
                   
                     J 
                     ⁡ 
                     
                       ( 
                       w 
                       ) 
                     
                   
                   = 
                   
                     
                       
                          
                         
                           
                             m 
                             1 
                           
                           - 
                           
                             m 
                             2 
                           
                         
                          
                       
                       2 
                     
                     
                       
                         S 
                         1 
                         2 
                       
                       + 
                       
                         S 
                         2 
                         2 
                       
                     
                   
                 
               
               
                 
                   ( 
                   
                     EQ 
                     . 
                     
                         
                     
                     ⁢ 
                     1 
                   
                   ) 
                 
               
             
           
         
       
     
     Where, J(w) is the objective; 
     m 1  is the foreground identification from the foreground DNN; 
     m 2  is the background identification from the background DNN; and 
     S is the standard deviation. 
     By applying this approach, candidate objects can be segmented from the background. A refinement DNN is applied to these segmented objects to refine the segmented objects. The value of objective J(w) is used as one input of to the refinement DNN for further refinement. 
     Object Matching and Pose Estimation 
     After further refinement of the image segmentation by the refinement DNN is completed, 2D images of discrete regions of the captured image are available for identification. Identification can be achieved by matching predefined CAD models to the segmented 2D images. Utilizing CAD models reduces the time to identify matches between successive frames; removes false segmented objects; and provides accurate region of interests for pose estimation. 
     Pose estimation can be computed from the matched images identified using the CAD models. In accordance with embodiments, an iterative process is deployed to refine the pose estimation results. This iterative process includes projecting the CAD model image onto the 2D image; computing an error between the projected CAD model and the 2D image; refining the matching identification; and re-computing the object pose. This pose estimation loop can be limited by a predetermined threshold tolerance for the computed error. When the computed error is within the threshold, the loop can terminate. 
     In some implementations, the matching results can be provided to the CAD model library to refine the definition of and object&#39;s key features or points for future segmentation and matching. The refinement of the CAD model library can include applying the robot&#39;s viewing angle into the CAD model of the object. This implementation can have application where the robot is performing a repetitive task—for example, part picking from a bin, inspection of similar facilities or structures, etc. 
     Robot Pose Estimation 
     In accordance with embodiments, robot pose estimation can be done by computing an inverse transformation to find the robot pose  165  of the robot in the world frame (from the robot pose estimate viewpoint/perspective). The robot&#39;s world frame pose  165  can be used as the major localization results for autonomous navigation by the robot. 
     In accordance with embodiments, a localization iterative refinement loop is implemented to improve the localization results. The CAD model for identified objects among the segmented images are again projected onto the 2D images. The position of the model&#39;s projection is adjusted to account for both the object&#39;s and the robot&#39;s pose. Adjusting the model&#39;s pose can reduce any displacement and/or parallax error due to the angular differences in the object&#39;s pose, the robot&#39;s pose, and the CAD model&#39;s drawing perspective. 
     In accordance with embodiments, another matching sequence loop is implemented to quantify the error between a 2D segmented image of an object and the CAD model applied to the object for identification. The error will be used to compute an image Jacobian matrix. Then a Kalman Filter-based optimization approach is used to update the pose of the robot. The matched CAD models can be projected onto the 3D environment to describe the current environment. The robot&#39;s pose can be further refined by combining this projection with the robot pose estimation. 
     Semantic Labeling and Understanding 
     In accordance with embodiments, CAD models that match objects in the captured image are associated with semantic labels and their properties. A semantic map can be compiled from these semantic labels. The label for each object image can include a unique identifier (i.e., unique semantical name), a category, and attributes of the object (e.g., shape size, material, distance, etc.). The robot can further analyze the situational environment by applying the semantic map to achieve its navigational tasks. 
       FIG.  1    illustrates process  100  for implementing autonomous robot navigation in accordance with embodiments. An image of the robot&#39;s environment is captured, step  105 . A deep learning-based approach is applied to the captured image to segment objects from the environment, step  110 . A foreground segmentation DNN can perform foreground segmentation, and a background DNN can perform background segmentation on the captured image. 
     Segmented 2D images can be matched, step  115 , to predefined image files (e.g., CAD models). Matched images are used to compute, step  120 , object pose estimates  160 . The object pose estimate  160  can be refined, step  125 , in an iterative estimation loop. An error can be calculated and used to compare the accuracy of the segmented 2D image to the image file. When the error is within a predetermined tolerance, iterative estimation loop  125  can be exited. In some implementations, the matching result can be used to update the predefined image file. 
     An inverse transformation can be computed, step  130 , to obtain a robot pose estimate. The robot pose estimate can be in a world frame centric to the robot&#39;s viewpoint and/or perspective. This world frame can be used to obtain localization results used by the robot for autonomous navigation. 
     Environmental modeling is performed, step  135 , to improve the localization results. This environmental modeling includes projecting the image file identified as matching the segmented image into the world frame. The projection is adjusted to account for the object&#39;s pose and the robot&#39;s pose in the world frame to create an updated world frame. The updated world frame is used to re-compute the robot pose estimate in step  130 . Iterative refinement loop  140  can be repeated to improve the robot pose estimate. 
     In accordance with embodiments, the robot pose estimate can be provided to iterative estimation loop  125 , where object matching (step  115 ) and object pose estimation (step  120 ) are repeated. This subsequent pass through iterative estimation loop  125  can use the updated image file generated during a previous pass through the iterative estimation loop. Loop  125  can be exited the results are within the predetermined threshold. The updated object pose estimation and updated object matching are provided to iterative refinement loop  140 . When the results of the iterative refinement loop are within a second predetermined accuracy threshold, process  100  continues to step  145 . 
     At step  145 , objects segmented from the captured image that match the image file are associated with semantic labels. The semantic label of each object can include a unique identifier, a category, and object attributes. A semantic map can be compiled, step  150 , from these semantic labels. The robot can achieve its autonomous navigation by applying the semantic map to it environment. 
       FIG.  2    illustrates system  200  for implementing autonomous robot navigation in accordance with embodiments. System  200  can be implemented in robot  205 . Robot  205  can include image capture device(s)  210 . The robot can include other systems, subsystems, and components (not shown) needed to achieve the robot&#39;s task. 
     Control processor  212  can include processor unit  214  and memory unit  216 . The control processor can be in communication with data store  220 , and other elements of system  220 , across data/control bus  218 . Processor unit  214  can execute executable instructions  222 , which cause the processor to perform segmentation, matching, and pose estimation in accordance with embodiments. Memory unit  216  can provide the control processor with local cache memory. 
     In addition to executable instructions  222 , data store  220  can include predefined image files  224 , captured image records  226 , updated image file records  228 , and semantic map  230 . Neural network  240  can include foreground DNN  242 , background DNN  244 , and refinement DNN  246 . 
     Embodying systems and methods provide a localization pose result for autonomous robotic navigation. Embodying systems and methods perform dense modeling to achieve fast object matching with a reduction in false segmentation to provide a pose estimation that incorporates semantic understanding with greater accuracy than conventional approaches. 
     In accordance with some embodiments, a computer program application stored in non-volatile memory or computer-readable medium (e.g., register memory, processor cache, RAM, ROM, hard drive, flash memory, CD ROM, magnetic media, etc.) may include code or executable program instructions that when executed may instruct and/or cause a controller or processor to perform methods discussed herein such as a method of segmenting captured images using prebuilt CAD models to achieve autonomous robot navigation based on semantic mapping, as disclosed above. 
     The computer-readable medium may be a non-transitory computer-readable media including all forms and types of memory and all computer-readable media except for a transitory, propagating signal. In one implementation, the non-volatile memory or computer-readable medium may be external memory. 
     Although specific hardware and methods have been described herein, note that any number of other configurations may be provided in accordance with embodiments of the invention. Thus, while there have been shown, described, and pointed out fundamental novel features of the invention, it will be understood that various omissions, substitutions, and changes in the form and details of the illustrated embodiments, and in their operation, may be made by those skilled in the art without departing from the spirit and scope of the invention. Substitutions of elements from one embodiment to another are also fully intended and contemplated. The invention is defined solely with regard to the claims appended hereto, and equivalents of the recitations therein.