Patent Publication Number: US-9836673-B2

Title: System, method and computer program product for training a three dimensional object indentification system and identifying three dimensional objects using semantic segments

Description:
BACKGROUND 
     The present invention relates to the art of machine vision, and more specifically, to a system for training a three dimensional object identification system and identifying three dimensional objects using three dimensional semantic segments. 
     Machine Vision (MV) describes technology and methods used to provide imaging-based automatic inspection and analysis for a variety of applications. Machine vision employs captured images to identify, inspect, and/or manipulate objects undergoing one or more processes. Imaging devices may be combined with a processing unit or maintained separately. When separated, a connection may be made to specialized intermediate hardware, such as a frame grabber. Many MV applications utilize digital cameras capable of direct connection to an image processor without the need for intermediate hardware. 
     Many MV systems rely upon two dimensional (2D) imaging processes that employ various light sources. For example, 2D imaging system may employ visible light, infra-red light, line scan imaging, and X-ray imaging. Typically, 2D imaging systems fall into one of two categories monochromatic images, and color images. 2D imaging systems may also detect portions of an object, or to process an entire image. Systems that process an entire image are often times employed in moving or production line processes. Three-dimensional (3D) imaging may also be employed in MV systems. Generally, 3D imaging includes scanning based triangulation, time of flight, grid based, and stereoscopic processes. 
     Part-based object recognition techniques employ 3D models with semantic points to determine a part center. Such an approach may be time consuming, requiring that the sematic points are manually annotated into the 3D models. Further, this approach does not identify specific regions of a part. Part based object recognition techniques also employ color recognition aide in identification. Colors are sampled at multiple points on an objects surface through pose-annotate training images, a two dimensional projection of the object is generated for each candidate pose, and a pixel-wise template is created. The pixel-wise template is used for matching color and/or texture in a query image. While this approach may be better at identifying specific regions of an object, several drawbacks still exist. Distortions in the pixel-wise template may exist. That is, color at a particular point may vary in each candidate pose due to illumination shortcomings, reflections, and occlusions on the part itself. 
     SUMMARY 
     According to an aspect of an exemplary embodiment, a method of training a three dimensional object identification system and identifying three dimensional objects using semantic segments includes receiving, into a non-volatile memory, an input file containing a geometric description of a three dimensional object having one or more semantic segments and one or more annotations for each of the one or more semantic segments, receiving, into the non-volatile memory one or more training images of the three dimensional object, identifying, through a processor, the one or more segments in the one or more training images, assigning, through a training module, one or more descriptors to the one or more segments, and generating an output file representing a machine vision of the three dimensional object. 
     According to another aspect of an exemplary embodiment, a system includes a central processor unit (CPU), a non-volatile memory operatively connected to the CPU, and a training module configured to analyze an input file containing a geometric description of a three dimensional object having one or more semantic segments, and one or more annotations for each of the one or more semantic segments. The training module includes computer readable program code embodied therewith, the computer readable program code, when executed by the CPU, causes the CPU to: receive, into a non-volatile memory, an input file containing a geometric description of a three dimensional object having one or more semantic segments and one or more annotations for each of the one or more semantic segments, receive, into the non-volatile memory one or more training images of the three dimensional object, identify, through a processor, the one or more segments in the one or more training images, assign, through the training module, one or more descriptors to the one or more segments, and generate an output file representing a machine vision of the three dimensional object. 
     According to another aspect of an exemplary embodiment, a computer program product for analyzing an input file containing a geometric description of a three dimensional object having one or more semantic segments, and one or more annotations for the one or more semantic segments includes a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code, when executed by a processor, causing the processor to: receive, into a non-volatile memory, the input file containing the geometric description of the three dimensional object having one or more semantic segments and the one or more annotations for each of the one or more semantic segments, receive, into the non-volatile memory one or more training images of the three dimensional object, identify, through a processor, the one or more segments in the one or more training images, assign, through a training module, one or more descriptors to the one or more segments, and generate an output file representing a machine vision of the three dimensional object. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The forgoing and other features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which: 
         FIG. 1  is a block diagram illustrating a system for identifying three dimensional objects using semantic segments, in accordance with an exemplary embodiment; 
         FIG. 2  is a flow chart illustrating a method of training a three dimensional object identification system for identifying three dimensional objects having semantic segments, in accordance with an exemplary embodiment; and 
         FIG. 3  depicts a three dimensional object having semantic segments being identified, in accordance with an exemplary embodiment; 
         FIG. 4  depicts a system for identifying three dimensional objects having semantic segments, in accordance with an aspect of an exemplary embodiment; and 
         FIG. 5  depicts a flow chart illustrating a method of identifying three dimensional objects with the system of  FIG. 4 , in accordance with an aspect of an exemplary embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     With reference now to  FIG. 1 , a training system for developing an output filed used in identifying objects having three dimensional (3D) segments is generally indicated at  10 . Training system  10  includes a central processing unit (CPU)  12  and a non-volatile memory  14 . Training system  10  further includes a training module  16 . As will be detailed more fully below, training system  10  receives an input file  18  containing a geometric description of a three dimensional (3D) object having a number of semantic or meaningful segments. Input file  18  is stored in non-volatile memory  14  and accessed by training module  16  through CPU  12 . The geometric description may be embodied in a computer aided design (CAD) file. Input file  18  also includes annotations for each of the semantic segments. The annotations are manually entered into input file  18  and represent an outline of each of the semantic segments. A training image  20  of the object is captured and input into system  10 . Each segment of the training file is analyzed or evaluated, to learn various nuances of the object that may occur in different views. In this manner, training module  16  may generate an output file can be constructed that may be used by a machine to identify the 3D object. 
     In training, descriptors for each semantic segment are identified by training module  16 . The descriptors may take the form of one or more physical characteristics of the segment. The physical characteristics may include color, texture, material characteristics, and/or areas of interest on a segment. For example, an area of interest could register as an area of instability in a captured image. The area of instability may take the form of a decal, a change in the material in a particular segment, the addition of a clear component such as glass, plastic or other such area on a particular segment. During object recognition, known areas of instability may be ignored. Distribution of the segment&#39;s descriptor in the different training images may be used to decide how to identify a particular portion of the segment. If the descriptor assumes one of a few well defined shapes or values, the segment can be identified if one of those shapes or values is measured in a query image. 
     A method of developing output file  24  of an object having semantic segments is indicated generally at  100  in  FIG. 2 . In block  102 , a file containing 3D object geometry is input into non-volatile memory  14 . As noted above, the file includes annotations for one or more semantic segments of the 3D object. For example, a 3D object, indicated generally at  110  in  FIG. 3  may take the form of a tea cup  112 . Tea cup  112  may include a first meaningful or semantic segment  113 , a second semantic segment  114  and a third semantic segment  115 . For example, first semantic segment  113  may define an outer surface of tea cup  112 ; second semantic segment  114  may define a handle for tea cup  112 ; and third semantic segment  115  may define an inner surface of tea cup  112 . The annotations may describe various physical and/or non-physical attributed of each segment  113 - 115 . For example, each descriptor may define one or more of a color, a texture, a material and the like for each segment  113 - 115 . 
     In block  120 , one or more training images are captured of object  110 . The one or more training images may represent various poses, angles or the like of object  110 . The number of training images taken may vary and may generally depend upon object symmetry, and/or object complexity. The training images may be supplied with object pose annotations. In block  130 , training module  16  may identify each semantic segment  113 - 115  using pose annotations and the segment annotations from training file  18 . For example, training module  16  may identify areas in the training image(s) that are associated with each semantic segment  113 - 115  described in training file  18 . At this point, it should be understood, that a pose annotation describes a specification of a 3D object position and orientation in a world coordinate system having six (6) degrees-of-freedom (DOF). Annotation of an object pose in a training image may include (1) intrinsic parameters of a camera such as focal length, sensor specs, lens distortion, and (2) object pose in 3D world coordinates having an origin and orientation that aligns with a sensor plane of the camera. 
     In block  140 , descriptors for each semantic segment  113 - 115  are computed. A descriptor associated with each semantic segment may describe a specified image region as a vector of numeric values incorporating structural information contained in the image region. Information encoded by the descriptor may represent a specification of the texture, color and/or special features, present in the image region. Examples of area descriptors (which do not depend on specific size and shape of the region) may include a hue-Saturation histogram, and/or a co-occurrence histogram. 
     In accordance with an aspect of an exemplary embodiment, training module  16  may evaluate an area of each semantic segment to verify each descriptor. In block  150 , training module  16  may also identify segment stability. For example, training module  16  may analyze a distribution of descriptors for each semantic segment. Training module  16  may determine whether each semantic segment possesses a single stable, e.g., identifiable, appearance, or whether each semantic segment may include multiple stable appearances. Training module  16  may also determine that a segment does not include a stable appearance. 
     In accordance with an aspect of an exemplary embodiment, training module  16  may determine that a semantic segment includes a single stable appearance by identifying a well concentrated unimodal distribution of descriptors. In accordance with another aspect of an exemplary embodiment, training module  16  may determine that a semantic segment includes a single stable appearance by identifying a well concentrated multimodal distribution of descriptors. At this point, it should be understood that a unimodal distribution of n-dimensional unit norm descriptor vectors is “well-concentrated” if an associated standard deviation is below σ max =0.03. It should be further understood that a multi-modal distribution of n-dimensional unit norm descriptor vectors is “well concentrated” if it can be represented as a sum of a number of well concentrated unimodal distributions where pairwise distances between the expectation values of the different modes are all above the threshold of dE=0.2. Determining that a segment does not have a stable appearance, in accordance with an aspect of an exemplary embodiment, may include training module  16  determining that a segment lacks a well concentrated multimodal distribution of descriptors. In block  160 , training module  16  forms output file  24 . 
     Reference will now follow to  FIG. 4  in describing an object recognition system  200  in accordance with an aspect of an exemplary embodiment. Object recognition system  200  includes a central processing unit (CPU)  212  and a non-volatile memory  214 . Object recognition system  200  further includes an object recognition module  216 . As will be detailed more fully below, object recognition system  200  receives output file  24  as well as captured images  218  of object  110 . Output file  24  and captured images  218  are stored in non-volatile memory  214  Object recognition module  216  in conjunction with CPU  212  evaluates the captured images using output file  24  to generate an object identified output  220 . 
     Reference will now follow to  FIG. 5  in describing a method  260  of identifying object  110  using output file  24  generated by method  100  described above. In block  270 , output file  24  is input into non-volatile member  214 . In block  280 , one or more query images  218  are captured of object  110 . In block  290 , object recognition module  216 , through CPU  212 , identifies one or more semantic segments in images  218 . More specifically, semantic segment descriptors provided in input file  24  are used by object recognition module  216  to evaluate a number of candidates for the segment pose in the image. In block  300 , objects are identified as spatial configurations of the semantic segments. The term “spatial configurations” should be understood to mean that objects are identified via recognized associated semantic segments using a Generalized Hough Transform”. In accordance with an aspect of an exemplary embodiment, object recognition module  216 , in reliance on segment descriptors may be able to identify a segment when only a portion of that segment is observable in a query image. Additionally, the segment descriptor enable object recognition modules  216  to identify a segment in a query image that may not have been captured in a training image. For example, the segment descriptor may identify a surface contour, a color, a texture or that like. Thus, even if a portion of a semantic segment is not captured in a training image, the semantic segment may still be identifiable the query image. 
     At this point, it should be understood that a training a system employs semantic segments or a 3D object to develop an output file based on one or more capture training images. In the output file, each semantic segment includes annotated descriptors that are later employed in a machine vision system that may employ object recognition to identify the 3D object. By computing descriptors over homogenous or substantially homogenous segments, fewer training images may be required to develop the output file. Further, the exemplary embodiments enable a part recognition system to recognize parts that may be observed only partially when constructing the output file. The output file may then be employed in a process associated with detecting one or more aspects of the 3D object in a query image. For example, the output file may be employed to detect objects passing along an assembly line to identify areas of interest for a process, object quality and the like. 
     The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, element components, and/or groups thereof. 
     The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated. 
     The flow diagrams depicted herein are just one example. There may be many variations to this diagram or the steps (or operations) described therein without departing from the spirit of the invention. For instance, the steps may be performed in a differing order or steps may be added, deleted or modified. All of these variations are considered a part of the claimed invention. 
     While the preferred embodiment to the invention had been described, it will be understood that those skilled in the art, both now and in the future, may make various improvements and enhancements which fall within the scope of the claims which follow. These claims should be construed to maintain the proper protection for the invention first described. 
     The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.