Patent Publication Number: US-11044400-B1

Title: Frame stitching in human oral cavity environment using intraoral camera

Description:
CROSS-REFERENCE TO PRIOR APPLICATIONS 
     The present application claims priority to U.S. 62/828,650 filed 3 Apr. 2019, which is incorporated herein in its entirety by reference. 
    
    
     FIELD OF THE INVENTION 
     The present invention is a unique method to stitch images of human teeth that are captured by an intraoral camera. The invention outperforms favorably to existing methods and shows a better performance in such cases of low-texture regions. 
     BACKGROUND OF THE INVENTION 
     Along with panoramic radiograph and periodontal checkup, intraoral photos are important visual data for initial diagnosis, treatment planning, active periodontal therapy, reevaluation and maintenance. For example, orthodontic treatment involves the application, over time, of force systems to teeth to correct malocclusion. In order to evaluate tooth movement progress, the orthodontist monitors this movement by means of visual inspection, intra-oral measurements, fabrication of casts, photographs, and radiographs. Therefore, collection of intra-oral images for future treatment is very important. However, dental photography using extra-oral cameras is time consuming and tedious for the patient. 
     Intra-oral cameras enable dentists to capture images of difficult-to-reach areas in the mouth. These intraoral cameras capture a sequence of calibrated images, i.e., each image captures only a part of the jaw. These images must then be “stitched” together to create one panoramic view of the whole jaw. 
     There are a variety of approaches to image stitching. Scale Invariant Feature Transform (“SIFT”) or Speeded Up Robust Features (SURF) are for extracting key points in these methods. A content-aware warping approach generates rectangular images from stitched panoramic images. Autostitch is a commercially available smartphone app. 
     The major drawback in most of the existing stitching approaches is that an alignment function is modeled as 2D homographies. But homographies are justified only if the images correspond to views that differ purely by rotation, or if the imaged scene is effectively planar (e.g., when the scene is sufficiently far away). To overcome this limitation, Zaragoza et al. 2014 proposed the Moving Direct Linear Transformation approach for accurate homography estimation that fine-tunes the projective warp to accommodate the deviations of the input data from the idealized conditions. 
     However, the oral cavity does not allow for images to be captured under ideal conditions. There are often problems with lighting (effect of saliva, tooth discolorization, gum texture, and other sources of specularity) and motion (even inevitable slight motions of the upper/lower jaw may lead to errors far beyond the desired accuracy). Thus, a method is needed to allow for stitching of images obtained under real conditions. 
     SUMMARY OF THE INVENTION 
     This present invention addresses two problems: 1) extracting keypoints in a featureless environment, and 2) stitching images captured in non-ideal conditions. The invention is a novel image stitching algorithm to handle the lack of features necessary for establishing region correspondence in a given scene, which is necessary for 3D reconstruction. Normal maps are estimated to reveal the impacted features of teeth surfaces. The as warps are estimated to enhance the stitching process. Results confirm the high performance of the invention. 
     The present invention can be used under conditions wherein the scene is not rich with features and without the need for devices other than the intra-oral camera and without the need for extra transformation. The invention identifies the required pixel-wise distortion for stitching in contrast to methods of the prior art. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The application file contains at least one drawing executed in color. Copies of this patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. 
         FIG. 1  is a schematic indicating the work flow of the proposed images stitching approach: (a) 2D input image capturing; (b) surface normal extraction; (c) surface normal correction, (d) normal map representation, and (e) image stitching; 
         FIG. 2  is a pair of images showing the planar surface segments of a tooth using normal (a) RGB image and (b) the correspondence normal map; 
         FIG. 3  is a graph showing the structural similarity index metric between the results generated using the present invention and the Ground Truth for ten subjects; 
         FIG. 4  is a set of images showing examples of different stitching approaches for six different subjects, wherein (a) shows the Ground Truth image for each subject, (b) shows the image using the present invention, (c) shows the alignment of the results generated from the present invention to the Ground Truth image, and (d) shows the results using Autostitching; and, 
         FIG. 5  is a pair of images showing (a) a 3D jaw, and (b) its reconstruction from the generated panoramic image. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     The present invention is a unique method to create a 3D reconstruction of a jaw from a plurality of images of human teeth that are captured by an intraoral camera. The method comprises the steps of (1) capturing a plurality of images wherein each image represents a section of the jaw, (2) making a surface normal estimation image for each captured image by extracting the surface normal and using shape from shading; (3) making surface normal corrections using an algorithm to rectify imprecise values impacted by the normal tooth surface, wherein the algorithm corrects a lack of feature region correspondence in a given scene, (4) converting the normal map of images to a gray scale image, and (5) stitching the images together to generate a 3D representative of the jaw. This process, and the images generated at each step in the process, is shown in  FIG. 1 . 
     In a preferred embodiment, the intraoral camera is flown in a curve path to capture the images, about 1 cm from the teeth. Up to 100 frames with a resolution 512×512 may be used for a complete 3D reconstruction. The overlap between the images may be above 70% to cover the whole jaw. An example of one frame is shown in  FIG. 1( a ) . 
     As is known in the art, the surface orientation at a 3D point P in the scene on a surface S is determined by a unit vector perpendicular to the plane tangent to S at P. Shape from Shading (SFS) is an algorithm among the tools used in shape-oriented extraction from a single view. The SFS problem is to find a solution of brightness equation E(x; y)=R(r; q; L) where E(x; y) is the brightness, R(r; q; L) is the surface reflectance map, L is illuminate direction and (r; q) are the surface gradients. Making an assumption of perspective projection and a calibrated camera, a metric representation of teeth and gum surfaces using SFS algorithm is obtained. This approach is used to estimate the surface normal for each captured image. To do this, a unique 3D point P on an object, such as a tooth, is defined by:
 
 P=B   −1 ( sp−b )= g ( s ( x,y ))  Eq. (1)
 
where B and b are decomposed parameters from the projection matrix, p is the corresponding image coordinate and s is the scalar. Because P equals g(s(x, y)) and the normal at a surface point is defined as N=(r×q)/|r×q| wherein
 
             r   =       dg   ⁡     (     s   ⁡     (     x   ,   y     )       )       dx           
and
 
               q   =       dg   ⁡     (     s   ⁡     (     x   ,   y     )       )       dq       ,         
the quantities r and q can be estimated using a Taylor&#39;s series expansion and applying the Jacob iterative method, as is known in the art.  FIG. 1( b )  shows the image of the tooth from the frame of  FIG. 1( a )  after applying the surface normal estimation.
 
     The oral cavity is an unfriendly environment for image acquisition because it contains several factors that make the quality of the images unrealistic. Hence, the SFS cannot provide accurate information of the surface normals relying upon the raw image information. It is known that the surface normals estimation depends on light direction and the surface patches are homogenous and uniformly lit by distant light source. Assuming that the light and the camera are at the same direction, a light direction vector can be determined. A cross product can then be calculated between the normal surface and the light direction vector. After calculating the cross product, a geometric mean filter with a window size 9 is applied. The technique of geometric mean filtering is known in the art. By applying the cross product as well as the mean filtering, the surface patches, which have the same orientation, have the same surface normal un-like other patches, which are on the border between two planar surfaces.  FIG. 1( c )  shows the image of the tooth from the frame of  FIG. 1( a )  after applying the surface normal estimation and the surface normal corrections. 
     As is known in the art, a normal map is a method to represent the surface normals on the image pixels domain. Each pixel of the map stores the direction of the surface normal at that particular point. Normal maps are commonly stored as regular RGB images where the RGB components correspond to the X, Y, and Z coordinates, respectively, of the surface normal. In the present invention, before mapping each component to its corresponding channel in the image domain, the positive and negative values are compressed into a positive interval [0; 1]. The mapping is done to the red and green channels, 
             R   =       (         N   x     ~     +   1     )     2           
and
 
             G   =         (         N   y     ~     +   1     )     2     .           
The Z component is always positive, because this is a tangent space normal map, and it is directly mapped to the blue channel. The values Ñ x , and Ñ y  are values of the SFS map in the x- and y-axis, respectively. The value “1” is added to these values to make sure that the mapping is between [0; 1] and the normal map is generated. In the present invention, the RGB normal map is then converted to a gray-scale image, as shown in  FIG. 1( d ) . As is known in the art, the benefit of generating a normal map of an image is to reveal the impacted geometric properties of the images inside an area, boundary, and shape. An example is shown in  FIG. 2 , wherein the normal map, shown in  FIG. 2( b ) , is rich with of features relative to the captured RGB image, shown in  FIG. 2( a ) .
 
     Features are extracted from the normal map using the KAZE algorithm, a multiscale 2D features detection and description algorithm in nonlinear scale spaces, which uses nonlinear diffusion filtering and makes blurring locally adaptive to the image data with respect to the object boundary and obtaining localization accuracy and distinctiveness. 
     The prescribed image capturing conditions do not lead to views that differ purely by rotation, nor planar scene since the object is near to the camera. Therefore, using a basic homographic warp yields misalignment. To overcome this problem, we warp each part using a location dependent homography. These estimated as-projective-as-possible warps account for data that deviates from the required constraints. Unlike the prior art, the present invention does not use a uniform grid to partition the image, but rather uses estimated normal maps to segment the 2D domain of the image into connected planar segments. Each pixel is assigned a weight that is inversely proportional to its distance from the centroid of the segment. Then pixels of each planar segment are warped using the same homography. A stitching algorithm is used to merge the separate image reconstructions to generate a 3D reconstruction of the entire oral cavity. 
     The present invention allows for stitching images captured in non-ideal conditions, i.e. when the intraoral camera movement is not a pure rotation or the camera is far from the teeth. The present invention maintains perspectives in the stitched images for partial or complete 3D reconstruction of the oral cavity, regardless of the occlusions and uncertainties associated with the jaw. The present invention allows for stitching that is more robust than prior art methods and able to accommodate advances in intraoral camera sensors as well as computational shape from shading. 
       FIGS. 3, 4 and 5  demonstrate the present invention on test subjects.  FIG. 3  is graph showing the structural similarity index metric between the results generated using the present invention and the Ground Truth for ten subjects.  FIG. 4  is a set of images showing examples of different stitching approaches for six different subjects, wherein (a) shows the Ground Truth image for each subject, (b) shows the image using the present invention, (c) shows the alignment of the results generated from the present invention to the Ground Truth image, and (d) shows the results using Autostitching.  FIG. 5  is a pair of images showing (a) a 3D jaw, and (b) its reconstruction from the generated panoramic image. 
     Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the presently disclosed subject matter pertains. Representative methods, devices, and materials are described herein, but are not intended to be limiting unless so noted. The terms “a”, “an”, and “the” refer to “one or more” when used in the subject specification, including the claims. 
     Unless otherwise indicated, all numbers expressing quantities of components, conditions, and otherwise used in the specification and claims are to be understood as being modified in all instances by the term “about”. Accordingly, unless indicated to the contrary, the numerical parameters set forth in the instant specification and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by the presently disclosed subject matter. 
     As used herein, the term “about”, when referring to a value or to an amount of mass, weight, time, volume, concentration, or percentage can encompass variations of, in some embodiments ±20%, in some embodiments ±10%, in some embodiments ±5%, in some embodiments ±1%, in some embodiments ±0.5%, and in some embodiments to ±0.1%, from the specified amount, as such variations are appropriate in the disclosed application.