Patent Description:
A typical approach to estimating a three-dimensional pose of a test object involves constructing a database with depth images of known poses of the object. The database is then searched, in a feature space, for the pose of the test object. This approach employs global image descriptors learned using machine learning. Depth images captured by structured light sensors are typically cluttered with noise and a spurious background. Noise sources complicate the task of estimating the pose of the test image from the search of the database. For example, using a global descriptor derived from features extracted from a depth image with the noise and background clutter, the resulting global image descriptors often do not accurately represent the pose of the object.

Prior art is disclosed by <NPL>, <NPL> and <NPL>.

The object of the invention is an improved method relate to automatically estimating a three-dimensional pose of an object from an image captured using a camera with a structured light sensor or other depth camera.

The present invention is defined by the following claims. Further aspects and advantages of the invention are discussed below in conjunction with the preferred embodiments.

The components and the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the embodiments.

A technique is disclosed for automatically estimating the three-dimensional pose of an object from an image captured using a camera with a structured light sensor or other depth camera. Pose estimation may be used in various visual analytics applications involving image search and retrieval. To more accurately estimate a three-dimensional pose of an object, a search is performed in a feature space that is less influenced by noise or background clutter.

To mitigate the influence of noise and background clutter, local image information is exploited. For training, local image patches for each training image are used. Instead of utilizing a single feature database indexing known poses from each of the training images, a feature database for each local image patch is generated indexing known poses from the local image patches generated from training images. For example, local feature databases index local image information for the same poses from the training images. When the trained system is provided a test image, image patches are sampled from the test image and patch-based features are extracted. A search is then performed using the patch-based features in the corresponding local feature database for the respective image patch. A patch-based pose is assigned to each image patch, and a best pose is estimated across all searches performed for the object in the test image. As such, an end-to-end solution is provided for training a system to and for estimating a three-dimensional pose of an object from depth images captured using structured light sensors or other depth cameras.

For both training and pose estimation, image patches are generated. For example, <FIG> illustrates an example of generating image patches from an input depth image. Depth image <NUM> is sampled to generate image patches 103A-103C, 105A-105C and 107A-107C. Depth image <NUM> may be one of a plurality of training images or a test image. As depicted in <FIG>, depth image <NUM> is uniformly sampled into local image patches of the same size and resolution. Alternatively, depth image <NUM> may be non-uniformly sampled. For example, depth image <NUM> may be sampling generating uniform image patches representing the perimeter of the depth image (e.g., patches 103A-103C, 105A, 105B and 107A-107C) and generating a larger, smaller, the same sized, non-uniform image patch representing the center of the depth image (e.g., patch 105B), with greater, lessor or the same resolution. Additional, different or fewer uniform or non-uniform image patches may be generated.

Features are extracted from each image patch using a machine learning artificial agent (e.g. through deep learning using a convolutional neural network). For example, a convolutional neural network is trained using labeled data from the depth images. The training learns what features for a given patch distinguish poses and a classifier for estimating a probability of the pose. The input is the image patch, including the color or gray scale values for each pixel and the depth from the camera for each pixel. Using the extracted features, the feature space is searched to identify nearest neighbor poses of each patch in a database. Because the feature databases are generated using training data representing discrete poses and may not include the exact pose of the test image, the closest pose to the test images (i.e., the nearest neighbor pose) is identified for each image patch. Therefore, for each test image, a plurality of nearest neighbor poses is identified (e.g., one pose for each image patch) and a confidence score may be assigned to each pose. The pose that best fits the object in the test image is selected. In this way, each image patch "votes" for a particular pose by identifying a pose and the likelihood that the pose best fits the object in the test image.

<FIG> illustrates a flowchart diagram for estimating a three-dimensional pose of an object. The method is implemented by the system of <FIG> (discussed below), <FIG> (discussed below) and/or a different system. In examples not being part of the claimed invention, additional different or fewer acts may be provided. For example, one or more acts may be omitted, such as act <NUM>, or one or more of acts <NUM>, <NUM>, <NUM> and <NUM> discussed below with respect to <FIG> may be included. The method is provided in the order shown. Other orders may, in examples not being part of the claimed invention, be provided and/or acts may be repeated. Further, the acts may be performed concurrently as parallel acts. For example, multiple acts <NUM> may be performed concurrently to extract features from the depth image and/or multiple image patches at the same time.

At act <NUM>, a depth image of the object is received. The depth image is captured using a three-dimensional scanner. For example, the three-dimensional scanner is a camera with a structured-light sensor, or a structured-light scanner. A structured-light sensor is a scanner that includes a camera and a projector. The projector projects structured light patterns that are captured by the camera. A multi-shot structured light sensor captures multiple images of a projected pattern on the object. The captured images of the pattern are used to generate the three-dimensional depth image of the object. For example, the depth image may be received as point cloud data. Each pixel is associated with a depth from the sensor. For example, red, green, blue + depth (RGB+D) data is provided.

<FIG> illustrates an example of estimating a three-dimensional pose of an object from a test image. Referring to <FIG>, depth image <NUM> is received. In this example, a depth image <NUM> of an engine of a high speed train is received. The depth image <NUM> may be received from a mobile device, captured by a sensor of the mobile device (e.g., smart phone or tablet). Alternatively, the depth image <NUM> may be captured by a sensor of a computing system.

At act <NUM>, the depth image is sampled to generate a plurality of image patches of the depth image. Referring to <FIG>, depth image <NUM> is sampled to generate a plurality of image patches <NUM>. The depth image may be uniformly or non-uniformly sampled. The image patches are generated in the same fashion as the training images were sampled from a plurality of training images during training (discussed below). The image patches provide local image and depth information for identifying the same or similar poses from image patches generated from the plurality of training images.

At act <NUM>, features are extracted from each of the plurality of image patches. The features are extracted from each patch using a trained or learned convolutional neural network. Other machine learning algorithms or artificial agents may be used in examples not being part of the claimed invention. The convolutional neural network is trained based on the annotated training images. For the convolutional neural network, the training identifies the distinguishing features. These deep-learnt features are then extracted from the image patches. Additionally, features may be extracted from the un-sampled depth image (e.g., the full-image, entire depth image). The features are extracted in the same fashion as or as learned using the training images and training image patches.

At act <NUM>, feature databases are searched for nearest neighbor poses. For example, a plurality of image patch feature databases are searched to find a plurality of nearest neighbor poses for the plurality of image patches. A different nearest neighbor pose is found for each of the image patches using a respective feature database. To find the nearest neighbor poses, a different feature database is searched for each image patch.

According to the invention, a nearest neighbor pose is found for the un-sampled depth image (e.g., the full-image, entire depth image). A full-image feature database is searched for the un-sampled depth image.

Indexing in the databases provides a compact view representation enabling quick search using efficient metrics. For example, the feature databases may be searched by applying L1-norm or L2-norm metrics. Other metrics may be used.

Referring to <FIG>, the nearest neighbor poses for each local image patch of the plurality of image patches <NUM> and a nearest neighbor pose for the full depth image <NUM> form a pool of possible poses. For each patch as well as the full image, nearest neighbors in the corresponding database are found. For example, referring back to <FIG>, a database for patch 103A will be searched for features of patch 103A, a database for patch 103B will be searched for features of patch 103B, and so on through database 107C searched for features of patch 107C. A database for depth image <NUM> may also be searched for depth image <NUM>. This results in a set of nearest neighbors in the feature space, with each nearest neighbor of this set indexing a certain pose.

A confidence score may be assigned to each of the nearest neighbor poses. For example, referring back to <FIG>, image patch 107A has substantially more background than image patch 105B. As such, it is likely that that image patch 107A has substantially more noise or background clutter than image patch 105B. Using this information, a higher confidence score may be assigned to image patch 105B than 107A. Additional and different metrics may be used to assign a confidence score to the nearest neighbor patches. For example, a confidence score may be assigned based on the features extracted from the image patches.

At act <NUM>, a three-dimensional pose of the object is estimated. The three-dimensional pose of the object is estimated based on the pool of nearest neighbor poses. For example, the three-dimensional pose of the object may be chosen based on the confidence score assigned to the each nearest neighbor pose. For example, a confidence score is assigned based on the number of features extracted from each patch, or a confidence score may be assigned based on the level of noise in the patch (e.g., due to a noisy background, etc.).

Other methodologies for estimating the three-dimensional pose of the object based on the pool of nearest neighbor poses may be used in examples not falling under the claimed invention. For example, depth images corresponding to the clustered nearest neighbor poses may be rendered using a three-dimensional computer-aided design model. A refinement algorithm may be applied to each of the rendered images using the input test image as the source, resulting in refined poses corresponding to clustered nearest neighbor poses.

For example, referring to <FIG> at <NUM>, the three-dimensional pose is estimated by rendering a depth image for each of the identified nearest neighbor poses and aligning the rendered depth images with the received depth image of the object.

At <NUM>, a depth image for each of the identified nearest neighbor poses is rendered. The rendered depth image for the nearest neighbor poses will closely resemble the received depth image. Because the feature database includes a discrete number of poses, the nearest neighbor poses, and consequently the rendered images, may be slightly different than the pose from the received depth image. The rendered depth images are rendered using a three-dimensional model of the object (e.g., a three-dimensional computer-aided design (CAD) model). The three-dimensional model of the object may be the three-dimensional CAD model used to design the object, such as the engine of a high speed train depicted in depth image <NUM>.

Further, at <NUM>, the rendered depth images are aligned with the received depth image <NUM>. The rendered depth images representing the nearest neighbor poses is refined prior to choosing the best pose. According to the invention, aligning the rendered depth images is done by applying a mean shift algorithm and a refinement algorithm to each of the rendered depth images. A mean shift algorithm is applied to identify clusters in the feature space of the poses indexed by the pool of nearest neighbors. As such, local maxima in the feature space are determined for the nearest neighbor poses. Cluster centers may then be determined from the identified clusters in the feature space.

The cluster centers may then be refined using a refinement algorithm (e.g., an iterative closest point (ICP) algorithm). For example, using point clouds derived from the rendered depth images, a difference between the derived point clouds and the point cloud of the received depth image <NUM> may be minimized. As such, pose parameters from the rendered images that best describe the received depth image <NUM> are determined with respect to clustered nearest neighbor poses. As such, the poses in the rendered depth images are refined to more closely match the pose of the received depth image <NUM>.

At <NUM>, the best aligned pose is chosen. The best rendered and aligned depth image is the image that is closest to the received depth image. For example, difference images may be used. To utilize difference images, non-zero pixel-regions in the received depth image <NUM> may be identified. Difference images are generated, in the non-zero pixel-regions, between the received depth image <NUM> and the rendered depth images. For each difference image, the number of pixels that having a depth value less than a threshold are counted. The rendered depth image corresponding to the difference image having the fewest pixel counts below the threshold is chosen as the closest image to the received depth image <NUM>. Other selection criterion may be used, such as Euclidian distance between point clouds.

<FIG> illustrates a flowchart diagram of another example of a method for estimating a three-dimensional pose of an object, not falling under the claimed invention. The method of <FIG> represents deep learning features to resolve pose and values for the features of different poses. The databases of sets of deep-learnt features and corresponding poses for the patches and/or entire depth image are created.

The method is implemented by the system of <FIG> (discussed below), <FIG> (discussed below) and/or a different system. Additional, different or fewer acts may be provided. For example, one or more acts may be omitted, such as act <NUM>, or one or more of acts <NUM>, <NUM>, <NUM>, <NUM> and <NUM> discussed above with respect to <FIG> may be included. The method is provided in the order shown. Other orders may be provided and/or acts may be repeated. Further, the acts may be performed concurrently as parallel acts. For example, multiple acts <NUM> and <NUM> may be performed concurrently to extract features from the depth image and multiple image patches at the same time.

At act <NUM>, a plurality of depth images are received. The plurality of depth images are annotated or labeled as training images. <FIG> illustrates an example of constructing feature databases using training images.

Referring to <FIG>, training images <NUM> are received. Each training image of the training images <NUM> is for a different pose of the object. Any number of poses may be used. The training images are from CAD renderings of a model and/or are from actual depth camera captures of the object.

As depicted in <FIG>, a system may be trained to estimate the pose of any object. In this example, the received training images <NUM> are depth images of different poses of an engine of a high speed train. The depth images are labeled for training (for clarity, the labels not depicted in <FIG>). The training images <NUM> are generated or captured to provide image data in the same manner as a captured test image used for estimating a three-dimensional pose of the object.

At act <NUM>, the plurality of depth images are sampled (i.e., segmented, separated, or divided) to generate a plurality of image patches for each the of depth images. The image patches provide local image information for training a system to identify the same or similar poses from image patches generated from a test image. The depth images may be uniformly or non-uniformly sampled to generate uniform or non-uniform image patches.

Referring to <FIG>, training images <NUM> are sampled to form local image patches <NUM> for each training image. In this example, uniformly sampled local patches from each image are generated. The uniform image patches <NUM> provide local image information for each of the training images <NUM>.

At act <NUM>, a machine learning artificial agent, such as a convolutional neural network, is trained to extract features from each of the plurality of image patches of the depth images. Additionally, a convolutional neural network is also trained to extract features from each of the full depth images. In this training stage, the convolutional neural network is trained using labeled image data from the plurality of depth images and/or the plurality of image patches (for clarity, the labels not depicted in <FIG>). Referring to <FIG>, convolutional neural network is trained to extract features from each local image patch <NUM> and each full training image <NUM>.

At act <NUM>, a plurality of image patch feature databases are generated. Each image patch feature database corresponds to a different image patch. Additionally, full-image feature database may be generated corresponding to the plurality of full depth images. The databases are indexed in a compact view representation to enable quick search by using efficient metrics. For example, the feature databases indexed for L1-norm or L2-norm metrics. Other metrics may be used.

For example, referring back to <FIG>, a database for patch 103A will be generated for features of each local image patch 103A from the training images. Likewise, a database for patch 103B will be generated for features of patch 103B, and so on through database 107C generated for features of patch 107C. A database for depth image <NUM> may also be generated for depth image <NUM>. This results in a set of feature databases in the feature space for a plurality of different poses of the object.

Referring to <FIG>, an individual index database is constructed for each of the local image patches <NUM> and the full training images <NUM>. The learned model is used to construct multiple feature databases, where each database corresponds to a certain local image patch from the training images or the full training images. The same machine learnt representation and feature extractor is used for all patches and the full image. For example, the machine learnt representation and feature extractor resizes the input image to a smaller size (e.g., 64x64 compared to full image size 640x480 and patch size 320x240), allowing the same learnt features to be extracted from different sized of input images. Alternatively, different machine learnt representations and feature extractors may be used for different patch locations. The feature databases are built such that the databases correspond to each patch and the full image, the pose from each patch indexed with the same pose as the full image. The databases are built in a such that each patch location has its own database (e.g., if the image is divided into 3x3 patch areas, <NUM> databases are built with <NUM> databases for the different patches and <NUM> database for the full image). All databases may share the same pool of poses. Alternatively, a single database may be built with each entry in the database containing the location information where the patch was extracted from the original image, with which can be related with the associated pose information.

The indexed feature databases may be used to perform the nearest neighbor search, as discussed above in the testing stage. For example, as depicted in <FIG>, the indexed feature database may employ a fast library for approximate nearest neighbors (FLANNs). Other indexing methodologies may be used.

Referring back to <FIG>, the trained model and feature databases are used for estimating a three-dimensional pose of an object. For example, a test image of the object is received. The test image is sampled to generate a plurality of image patches of the test image. A nearest neighbor pose for each of the generated image patches is identified from the plurality image patch databases. Additionally, a nearest neighbor pose may be identified for the full test image from a full-image feature database. A plurality of depth images are rendered based on the nearest neighbor poses. Each of the rendered depth images are aligned with the test image, and the most closely aligned depth image is selected. The pose for the most closely aligned depth image provides or is the pose for the test image. Interpolation from the closest poses may be used for more refined pose estimate.

<FIG> illustrates an embodiment of a system for estimating a three-dimensional pose of an object.

For example, the system is implemented on a computer <NUM>. A high-level block diagram of such a computer <NUM> is illustrated in <FIG>. Computer <NUM> includes a processor <NUM>, which controls the overall operation of the computer <NUM> by executing computer program instructions which define such operation. The computer program instructions may be stored in a storage device <NUM> (e.g., magnetic disk) and loaded into memory <NUM> when execution of the computer program instructions is desired. The memory <NUM> may be local memory as a component of the computer <NUM>, or remote memory accessible over a network, such as a component of a server or cloud system. Thus, the acts of the methods illustrated in <FIG> and <FIG> may be defined by the computer program instructions stored in the memory <NUM> and/or storage <NUM>, and controlled by the processor <NUM> executing the computer program instructions. An image acquisition device <NUM>, such as a three-dimensional scanner, may be connected to the computer <NUM> to input image data to the computer <NUM>. It is also possible to implement the image acquisition device <NUM> and the computer <NUM> as one device. It is further possible that the image acquisition device <NUM> and the computer <NUM> communicate wirelessly through a network.

The Image acquisition device <NUM> is any three-dimensional scanner or other three-dimensional camera. For example, the three-dimensional scanner is a camera with a structured-light sensor, or a structured-light scanner. A structured-light sensor is a scanner that includes a camera and a projector. The projector projects structured light patterns that are captured by the camera. A multi-shot structured light sensor captures multiple images of a projected pattern on the object. The captured images of the pattern are used to generate the three-dimensional depth image of the object.

The computer <NUM> also includes one or more network interfaces <NUM> for communicating with other devices via a network, such as the image acquisition device <NUM>. The computer <NUM> includes other input/output devices <NUM> that enable user interaction with the computer <NUM> (e.g., display, keyboard, mouse, speakers, buttons, etc.). Such input/output devices <NUM> may be used in conjunction with a set of computer programs as an annotation tool to annotate volumes received from the image acquisition device <NUM>. One skilled in the art will recognize that an implementation of an actual computer could contain other components as well, and that <FIG> is a high level representation of some of the components of such a computer for illustrative purposes.

For example, the computer <NUM> may be used to implement a system for estimating a three-dimensional pose of an object. Storage <NUM> and/or memory <NUM> is configured to store the feature databases for the sampled image patches generated using training images or image data. Storage <NUM> and/or memory <NUM> is further configured to store a full-image feature database generated using the full training images or training data. The individual databases may be stored collected together in a single database.

Processor <NUM> is configured to receive a depth image of the object captured by image acquisition device <NUM>, such as a sensor of a mobile device. Processor <NUM> is further configured to generate image patches from the depth image, to identify a nearest neighbor pose for each of the generated image patches from the stored feature databases and to identify a nearest neighbor pose for the received depth image from the stored full-image feature database.

Processor <NUM> is further configured to estimate a three-dimensional pose of the object from the identified nearest neighbor poses. For example, to estimate the three-dimensional pose of the object, the processor <NUM> is configured to render a depth image from each of the identified nearest neighbor poses, to align each of the rendered depth images with the received depth image of the object and to select a rendered depth image that is closest to the received depth image. As such, estimating the three-dimensional pose is based on the selected rendered depth image. Processor <NUM> is also configured to select the rendered depth image closest to the received depth image by generating a difference image for each of the aligned rendered depth images, and to select the aligned rendered depth image corresponding to a difference image with a fewest number of pixels below a depth threshold.

<FIG> illustrates another embodiment of a system for estimating a three-dimensional pose of an object. The system allows for capture of data by a mobile device <NUM> with pose determination by one or both of a remote workstation <NUM> or server <NUM>.

The system <NUM>, such as an imaging processing system, may include one or more of a server <NUM>, a network <NUM>, a workstation <NUM> and a mobile device <NUM>. Additional, different, or fewer components may be provided. For example, additional servers <NUM>, networks <NUM>, workstations <NUM> and/or mobile devices <NUM> are used. In another example, the servers <NUM> and the workstation <NUM> are directly connected, or implemented on a single computing device. In yet another example, the server <NUM>, the workstation <NUM> and the mobile device <NUM> are implemented on a single scanning device. As another example, the workstation <NUM> is part of the mobile device <NUM>. In yet another embodiment, the mobile <NUM> performs the image capture and processing without use of the network <NUM>, server <NUM>, or workstation <NUM>.

The mobile device <NUM> includes sensor <NUM> and is configured to capture a depth image of an object. The sensor <NUM> is a three-dimensional scanner configured as a camera with a structured-light sensor, or a structured-light scanner. For example, the depth image may be captured and stored as point cloud data.

The network <NUM> is a wired or wireless network, or a combination thereof. Network <NUM> is configured as a local area network (LAN), wide area network (WAN), intranet, Internet or other now known or later developed network configurations. Any network or combination of networks for communicating between the client computer <NUM>, the mobile device <NUM>, the server <NUM> and other components may be used.

Claim 1:
A method for estimating a three-dimensional pose of an object, the method comprising:
receiving (<NUM>) a depth image of the object;
sampling (<NUM>) the depth image to generate a plurality of image patches of the depth image;
extracting (<NUM>), using a trained convolutional neural network, features from each of the plurality of image patches and from the depth image;
searching (<NUM>), using the extracted features, a plurality of image patch feature databases and a full-image feature database for a plurality of nearest neighbor poses, the feature databases including a discrete number of poses, wherein a nearest neighbor pose is found for each local image patch of the plurality of image patches using a respective image patch feature database and a nearest neighbor pose is found for the depth image using full-image feature database, the nearest neighbor poses for each local image patch of the plurality of image patches and the nearest neighbor pose for the full depth image form a pool of possible nearest neighbor poses; and
estimating (<NUM>), based on the pool of possible nearest neighbor poses, the three-dimensional pose of the object, the estimating (<NUM>) comprising:
rendering (<NUM>) a depth image from each of the identified nearest neighbor poses using a three-dimensional model of the object;
aligning (<NUM>) each of the rendered depth images with the received depth image of the object by applying a mean shift algorithm and a refinement algorithm to each of the rendered depth images; and
selecting (<NUM>) a rendered and aligned depth image closest to the received depth image, wherein estimating the three-dimensional pose is based on the selected rendered depth image.