Method of operating intraoral scanner for fast and accurate full mouth reconstruction

An intraoral scanner includes an image capturing device and a processor. A method of operating the intraoral scanner includes the image capturing device sequentially capturing M images of a buccal bite, the processor generating M sets of buccal bite point clouds according to the M images, the processor matching the M sets of buccal bite point clouds to generate a bite model, when the number of data points of the bite model exceeds a first threshold, the processor computing P sets of bite feature descriptors of the bite model, when a predetermined quantity of bite feature descriptors in a set of bite feature descriptors of the P sets of bite feature descriptors exceeds a second threshold, the processor performing a registration on an upper arch model and a lower arch model to the buccal bite mode to generate a full mouth model.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to intraoral scanning, and in particular, to an operating method of an intraoral scanner.

2. Description of the Prior Art

The intraoral scanner uses laser light to scan the teeth quickly, and then uses software to build a teeth model for medical personnel to perform teeth reconstruction, orthodontic treatments or other clinical applications. The teeth reconstruction may involve the use of dental braces, dental bridges, dental implants and other dentures to reconstruct missing or bad teeth. The orthodontic treatments utilize orthodontic devices to improve abnormal occlusion of teeth. Accurate teeth models are used to prepare suitable dentures or orthodontic devices to lower the risk of dental surgery.

In the related art, the intraoral scanner is used to perform a bite scan to obtain an accurate model. However, the existing bite scan is complicated in procedure and the scan length may be inadequate. A short scan length cannot produce an accurate model. A long scan length will increase the amount of computation of the intraoral scanner, slowing down the model reconstruction speed and causing discomfort to the patient.

SUMMARY OF THE INVENTION

According to an embodiment of the invention, an intraoral scanner includes an image capturing device and a processor, and a method of operating the intraoral scanner includes the image capturing device sequentially capturing M images of a buccal bite, the processor generating M bite point clouds according to the M images, and the processor matching the M bite point clouds to generate a bite model. The method further includes when a quantity of data points of the bite model exceeds a first threshold, the processor computing P sets of bite feature descriptors of the bite model, and when a predetermined quantity of bite feature descriptors in a set of bite feature descriptors of the P sets of bite feature descriptors exceeds a second threshold, the processor performing a registration on an upper arch model and a lower arch model to the buccal model to generate a full mouth model. M and P are positive integers.

According to an embodiment of the invention, an intraoral scanner includes the image capturing device sequentially capturing M images of a buccal bite, the processor generating M sets of bite point clouds according to the M images, the processor down-sampling the M bite point clouds to generate M down-sampled bite point clouds, and the processor matching the M down-sampled bite point clouds to generate a bite model. The method further includes when a quantity of data points of the bite model exceeds a first threshold, the processor computing P sets of bite feature descriptors of the bite model, and when a predetermined quantity of bite feature descriptors in a set of bite feature descriptors of the P sets of bite feature descriptors exceeds a second threshold, the processor performing a registration on an upper arch model and a lower arch model to the buccal model to generate a full mouth model. M and P are positive integers.

DETAILED DESCRIPTION

FIG.1is a block diagram of an intraoral scanner system1according to an embodiment of the invention. The intraoral scanner system1may include an intraoral scanner10and a display12coupled thereto. The intraoral scanner10may be a handheld intraoral scanner, and may be coupled to the display12in a wired or wireless manner. An operator may operate the intraoral scanner10to scan a patient's mouth to reconstruct a full mouth model, and the display12may display the reconstructed full mouth model. Referring toFIG.2,FIG.2shows a schematic diagram of a dental scan area according to an embodiment of the invention. The dental scan area is scanned for reconstructing a full mouth model.

When the intraoral scanner10is used to reconstruct the full mouth model, the intraoral scanner10may scan an upper arch area20to obtain a plurality of upper arch images and create an upper arch model according to the plurality of upper arch images, and scan a lower arch area22to obtain a plurality of lower arch images and create a lower arch model according to the plurality of lower arch images. Next, in order to obtain a relative positional relationship between the upper arch model and the lower arch model, the intraoral scanner10may scan a buccal bite area24to obtain a plurality of bite images, and create a bite model according to the plurality of bite images. The bite model includes a part of the upper arch model and a part of the lower arch model. Finally, the intraoral scanner10may perform a registration on the upper arch model and the lower arch model with respect to the bite model, and adjust the upper arch model and the lower arch model to correct relative positions thereof to generate a full mouth model. Since the bite model is primarily used to align the upper arch model and the lower arch model to increase the accuracy of the full mouth model, the bite model may be a model of a partial buccal bite, and the intraoral scanner10may scan the partial buccal bite to create the bite model. The intraoral scanner10may determine whether a scan area of the buccal bite is sufficient to build the bite model. If so, the operator may be notified to terminate scanning, relieving discomfort to the patient owing to pretense of the intraoral scanner10, reducing an amount of computations of the intraoral scanner10, while reducing the time required to establish the full mouth model. The intraoral scanner system1may perform computations and display the result in real time, enabling the operator to view the scanning result in real time, allowing the operator to check whether the quality of the current full mouth model is satisfactory, reducing the scanning time while enhancing the quality of the full-mouth scan.

The intraoral scanner10may include a processor100, a projection device102, an image capturing device104and a memory106. The processor100may be coupled to the projection device102, the image capturing device104, the memory106, and the display12to control operations thereof. The projection device102may project a pre-programmed pattern onto surfaces of the tooth sample along a predetermined scan path. The imaging device104may scan the tooth sample along the predetermined scan path to obtain a plurality of two-dimensional images of the surface of the object. The tooth sample may be a full upper arch, a partial upper arch, a full lower arch, a partial lower arch, and a buccal bite. The predetermined pattern may be a structured light pattern such as a checkerboard pattern, stripes, circles, a cross pattern, a gray coded pattern, a color coded pattern, other coded patterns or a random pattern. When the predetermined pattern is projected onto the surfaces of the tooth samples of different shapes, textures and/or depths, the pattern will be deformed. The two-dimensional image may show a deformation of the predetermined pattern. The processor100may compute three-dimensional (3D) data points of the surface feature points of the tooth sample according to the original predetermined pattern and the deformed predetermined pattern. A set of 3D data points may be used to generate a 3D model of the tooth sample, referred to as a point cloud. The processor100may generate a plurality of point clouds based on the plurality of 2D images, and match the plurality of point clouds using a matching algorithm and/or a data post-processing program to generate the 3D model of the tooth sample. The memory106may be a non-volatile memory such as a random access memory or a hard drive. The memory106may store images and data points of the plurality of point clouds. Specifically, the memory106may store the upper arch images, the lower arch images, and the bite images, and may store the upper arch model, the lower arch model, and the bite model.

FIG.3is a schematic diagram of a scan method adopted by the intraoral scanner10. The scan method is used to establish the bite model. The intraoral scanner10may scan along a bite scan path Pb to obtain bite images31to34in sequence. The processor100may respectively generate a first bite point cloud to a fourth bite point cloud according to the bite images31to34, and match according to a matching algorithm the adjacent bite point clouds of the first bite point cloud to the fourth bite point cloud to generate the bite model. The matching algorithm may be an iterative closest point algorithm (ICP). For example, when matching the first bite point cloud and the second bite point cloud, the processor100may transform a plurality of data points in the second bite point cloud to reduce or minimize differences between a plurality of data points in the first bite point cloud and the plurality of data points in the second bite point cloud. The intraoral scanner10may scan the upper arch and the lower arch using a similar scan method to create the upper arch model and the lower arch model, respectively.

FIG.4is a schematic diagram of a bite mode Mb according to an embodiment of the invention. The bite model Mb includes a plurality of data points Pn. The data points Pn may include feature points of a tooth, such as an edge or a corner of the tooth, so as to determine the position thereof. The bite model Mb shows that the tooth40of the upper arch and the tooth42of the lower arch are aligned with each other. The processor100may align the upper arch model and the lower arch model using the tooth40and the tooth42of the bite model Mb, respectively, so as to adjust the upper arch model and the lower arch model, and then to generate an accurate full mouth model.

FIG.5is a flowchart of a method500of operating the intraoral scanner10. The method500includes Steps S502to S508for generating an accurate full mouth model. Any reasonable Step change or adjustment is within the scope of the disclosure. Steps S502to S508are detailed as follows:Step S502: Establish the upper arch model and compute a plurality of sets of upper arch feature descriptors;Step S504: Establish the lower arch model and compute a plurality of sets of lower arch feature descriptors;Step S506: Perform a coarse alignment on the upper arch model and the lower arch model according to the plurality of sets of upper arch feature descriptors and the plurality of sets of lower arch feature descriptors to generate a full mouth model;Step S508: Perform a fine alignment on the full mouth model to generate an accurate full mouth model.

In Step S502, the processor100computes a set of upper arch feature descriptors for each data point in the upper arch model, and sequentially computes a plurality of sets of upper arch feature descriptors for all data points in the upper arch model, each set of upper arch feature descriptors describing a geometric relationship between a point of interest and surrounding data points thereof in the upper arch model. In some embodiments, the processor100may first down-sample all data points of the upper arch model to generate a plurality of down-sampled data points of the upper arch model, and sequentially compute the plurality of sets of upper arch feature descriptors for the plurality of down-sampled data points of the upper arch model. Down-sampling may reduce the amount of computations for the processor100to reduce the time for generating the plurality of sets of upper arch feature descriptors. The plurality of sets of upper arch feature descriptors may be stored in the memory106. In Step S504, the processor100computes a set of lower arch feature descriptors for each data point in the lower arch model, and sequentially computes a plurality of sets of lower arch feature descriptors for all data points in the lower arch model, each set of lower arch feature descriptors describing a geometric relationship between a point of interest and surrounding data points thereof in the lower arch model. In some embodiments, the processor100may first down-sample all data points of the lower arch model to generate a plurality of down-sampled data points of the lower arch model, and sequentially compute a plurality of sets of lower arch feature descriptors for the plurality of down-sampled data points of the lower arch model. Down-sampling may reduce the amount of computations of the processor100and reduce the time for generating the plurality of sets of lower arch feature descriptors. The plurality of sets of lower arch feature descriptors may be stored in the memory106. In Step S506, the processor100loads the plurality of sets of upper arch feature descriptors and the plurality of sets of lower arch feature descriptors from the memory106, and performs a coarse alignment on the upper arch model and the lower arch model according to the plurality of sets of upper arch feature descriptors and the plurality of sets of lower arch feature descriptors to generate the full mouth model. The coarse alignment may align a part of the upper teeth in the upper arch model with a part of the lower teeth in the lower arch model.FIGS.7and8show two methods to achieve the coarse alignment, and will be explained in the following paragraphs. In Step S508, the processor100performs a fine alignment on the full mouth model according to the matching algorithm to generate an accurate full mouth model. The matching algorithm may be, for example, an iterative closest point algorithm.

Each set of upper arch feature descriptors or each set of lower arch feature descriptors may include a set of point feature histogram (PFH) data or a set of fast point feature histogram (FPFH) data. A point feature histogram shows the distribution of the geometric relationship between a point of interest and the surrounding data points. The point feature histogram is invariant to the rotational transformation and translational transformation of the point cloud in the 3D space, and may be resistible to the effects of various levels of sampling and various levels of noise. The fast point feature histogram is a simplified version of the point feature histogram. The amount of computation is reduced considerably by simplifying and optimizing the calculation, while retaining most features of the point feature histogram.

FIG.6Ais a schematic diagram of a method of computing a set of point feature histogram data.FIG.6Ashows data points P0to P11. The data points P0to P11may be data points in a point cloud, and the data point P0may be a point of interest. When computing a set of point feature histogram data for the data point P0, the processor100may determine a range R centered at the data point P0and the surrounding data points P1to P5in the range R, and compute a set of feature descriptors for any pair of data points in the data points P0to P5to obtain a 15 (=6*5/2) sets of feature descriptors, and sort the 15 sets of feature descriptors into bins to generate feature histogram data of the set of data points P0. The range R may be, but is not limited to, a spherical range with a radius r. The range R may be set according to a point cloud density. When the point cloud density is small, the range R may be set to a larger value; when the point cloud density is large, the range R may be set to a smaller value. In some embodiments, each set of feature descriptors may include 3 direction angles with respect to a pair of corresponding normals of a pair of data points, and the sorting method may include dividing each directional angle into equal parts at a predetermined interval, e.g., into five parts to obtain 125 (=5*5*5) histogram bins, then sorting the 3 direction angles of each set of feature descriptors to one of 125 histogram bins, and sort the 15 sets of feature descriptors by the same way to generate a set of point feature histogram data. In other embodiments, each set of feature descriptors may include a distance and the 3 direction angles with respect to the pair of corresponding normals, and the sorting method may include dividing a distance range and each directional angle into equal parts at a predetermined interval, e.g., into five parts to obtain 625 (=5*5*5*5) histogram bins, then sorting the distance and the 3 direction angles of each set of feature descriptors to one of 625 histogram bins, and sorting the 15 sets of feature descriptors by the same way to generate a set of point feature histogram data. In other embodiments, each set of feature descriptors may include a plurality of parameters, each parameter may include a maximum parameter range that can be divided at a fixed interval or a variable interval to provide a plurality of histogram bins, and then each set of feature descriptors may be sorted into one of the plurality of histogram bins, and the 15 sets of feature descriptors may be sorted by the same way to generate a set of point feature histogram data. Each set of point feature histogram data may be plotted into a histogram.FIG.6Bis a histogram plotted according to the set of point feature histogram data generated fromFIG.6A, where the horizontal axis represents the histogram bin, and the vertical axis represents the data point ratio (%) of each set of feature descriptors.FIG.6Bshows that the set of histogram data is classified into 3 histogram bins. If the histogram shows that all pieces of histogram data are less than the second threshold Th2, the data point P0is located closely to a substantially flat surface; if the histogram shows that one or more pieces of histogram data are greater than the second threshold Th2, the data point P0is located closely to a feature point of the tooth, e.g., located closely to an edge or a corner of the tooth.

FIG.7is a flowchart of another method700of operating the intraoral scanner10. The method700includes Steps S702to S718for use to realize Step S506inFIG.5. Steps S702to S706are used to generate the bite model Mb. Steps S708to S714are used to determine when to perform a registration on the upper arch model and the lower arch model. Steps S716and S718are used to determine the completion of the full mouth model. Any reasonable step change or adjustment is within the scope of the disclosure. Steps S702to S718are detailed as follows:Step S702: The image capturing device104sequentially captures M images of a buccal bite;Step S704: The processor100generates M bite point clouds according to the M images;Step S706: The processor100matches the M bite point clouds to generate a bite model Mb;Step S708: The processor100determines whether the number of data points Np of the bite model Mb exceeds a first threshold Th1? If so, go to Step S710; if not, go to Step S702;Step S710: The processor100computes P sets of bite feature descriptors of the bite model Mb;Step S712: The processor100determines whether a predetermined number Nf of bite feature descriptors in a set of bite feature descriptors of the P sets of bite feature descriptors exceeds a second threshold Th2? If so, go to Step S714; and if not, go to Step S702;Step S714: The processor100performs a registration on the bite model Mb, the upper arch model, and the lower arch model to generate a full mouth model;Step S716: The processor100determines whether a registration error E is less than the third threshold Th3? If so, go to Step S718; and if not, go to Step S702;Step S718: The display12displays a full mouth model.

When the intraoral scanner10is used to establish the bite model Mb, the image capturing device104sequentially captures M images of the buccal bite of the upper and lower arches (Step S702). The processor100generates M bite point clouds according to the M images (Step S704), and matches the M bite point clouds using the matching algorithm to generate a bite model Mb (Step S706). Next, in Step S708, the processor100determines whether the number of data points Np of the bite model Mb exceeds the first threshold Th1. In some embodiments, the first threshold Th1may be 12,000. When the number of data points Np is less than or equal to the first threshold Th1, the number of data points Np may be insufficient for a registration, and the intraoral scanner1continues to scan the buccal bite and generate an bite model Mb (Steps S702to S706); when the number of data points Np exceeds the first threshold Th1, the number of data points Np may be sufficient for a registration, and the processor100computes P sets of bite feature descriptors of P point cloud data points of the bite model Mb for the point cloud data points of the bite model Mb (Step S710). In some embodiments, in Step S710, the intraoral scanner1may simultaneously compute the P sets of bite feature descriptors and continue to scan the buccal bite to generate the bite model Mb. Each set of bite feature descriptors may include a set of point feature histogram data or a set of fast point feature histogram data.

In Step S712, the processor100determines whether a predetermined number Nf of bite feature descriptors in a set of bite feature descriptors of the P sets of bite feature descriptors exceeds a second threshold Th2. For example, the second threshold Th2may be 30%, 40%, or other proportions, and the predetermined number Nf may be 2000 or other numbers. When the number of bite feature descriptors in the set of bite feature descriptors exceeding the second threshold Th2(such as 30%) is less than the predetermined number Nf (such as less than 2000), the bite model Mb may include a tooth surface but not a tooth feature point, and does not include the upper arch and the lower arch at the same time, the intraoral scanner1continues to scan the buccal bite to generate the bite model Mb and compute more sets of bite feature descriptors (Steps S702to S710). When the number of bite feature descriptors in the set of bite feature descriptors exceeding the second threshold Th2(such as 30%) exceeds the predetermined number Nf (such as more than 2000), the bite model Mb may include tooth feature points and may include a partial upper arch and a partial lower arch at the same time, the processor100performs a registration on the bite model Mb, the upper arch model and the lower arch model to generate a full mouth model (Steps S714). In some embodiments, in Step S714, the intraoral scanner1may simultaneously perform the registration on the bite model Mb, the upper arch model, and the lower arch model, and continue to scan the buccal bite and generate the bite model Mb. The registration algorithm may be a random sample consensus (RANSAC) algorithm. The processor100performs the registration according to the P sets of bite feature descriptors of the bite model Mb and the upper arch feature descriptors of the upper arch model to generate a corrected upper arch model, performs the registration according to the P sets of bite feature descriptors of the bite model Mb and the lower arch feature descriptors of the lower arch model to generate a corrected lower arch model, and combines the corrected upper arch model and the corrected lower arch model to generate a full mouth model. Specifically, if the processor100may use the registration algorithm to find a set of upper arch feature descriptors from the sets of upper arch feature descriptors of the upper arch model to match with a set of bite arch feature descriptors from the P sets of bite arch feature descriptors of the bite model, then the corrected upper arch model will be generated. If the processor100has found a set of lower arch feature descriptors from the lower arch model to match with a set of bite arch feature descriptors from the bite model using the registration algorithm, then the corrected lower arch model will be generated. Later, the processor100combines the corrected upper arch model and the corrected lower arch model to generate the full mouth model and the registration error E of the full mouth model. The registration error E may be the maximum gap between the corrected upper arch model and the corrected lower arch model and/or the maximum overlapping length between the corrected upper arch model and the corrected lower arch model. In some embodiments, the processor100generates a registration confidence parameter after performing the registration on the bite model Mb, the upper arch model, and the lower arch model. When the processor100determines that the registration confidence parameter of the full mouth model reaches the fourth threshold, the full mouth model has a high degree of accuracy, and the processor100computes the registration error E of the full mouth model. When the processor100determines that the registration confidence parameter of the full mouth model has not reached the fourth threshold, the full mouth model has a low accuracy, and the intraoral scanner1repeats Steps S702to S714. In some embodiments, the threshold may be set to 0.98.

In Step716, if the registration error E is less than the third threshold Th3, the full mouth model is determined as accurate, and the display12displays the full mouth model (Step718). If the registration error E is greater than or equal to the third threshold Th3, the full mouth model is determined as inaccurate, and the intraoral scanner1repeats Steps S702to S716. In some embodiments, the registration error E may be the maximum gap between the corrected upper arch model and the corrected lower arch model, and the third threshold Th3may be set to 3%, 5%, or other ratios. In other embodiments, the registration error E may be the maximum overlapping length between the corrected upper arch model and the corrected lower arch model, and the third threshold Th3may be set to zero. In some embodiments, when the registration error E is less than the third threshold Th3, the intraoral scanner10may notify the operator that the reconstruction of the full mouth model is complete, and the scan procedure may be terminated.

Since the registration procedure is computation-intensive and complicated, the method700waits until sufficient number of data points Np of the bite model Mb are accumulated before generating representative bite feature descriptors. Only after the predetermined number Nf of bite feature descriptors exceeds the second threshold Th2, the method700starts detecting the feature points of the tooth and performing the registration process. Therefore, the registration process is performed only when sufficient bite model is obtained, thereby reducing the amount of computations of the intraoral scanner10, reducing the time required for reconstructing a full mouth model, enhancing the quality of a full mouth scan and reducing discomfort of a patient.

FIG.8is a flowchart of another method800of operating the intraoral scanner10. The method800includes Steps S802to S818for use to realize Step S506inFIG.5. Steps S802to S806are used to generate the bite model Mb. Steps S808to S814are used to determine when to perform a registration on the upper arch model and the lower arch model. Steps S816and S818are used to determine the completion of the full mouth model. Any reasonable step change or adjustment is within the scope of the disclosure. Steps S802to S818are detailed as follows:Step S802: The image capturing device104sequentially captures M images of a buccal bite;Step S804: The processor100generates M bite point clouds according to the M images;Step S805: The processor100down-samples the M bite point clouds to generate M down-sampled bite point clouds;Step S806: The processor100matches the M down-sampled bite point clouds to generate a bite model Mb;Step S808: The processor100determines whether the number of data points Np of the point cloud data points of the bite model Mb exceeds a first threshold Th1? If so, go to Step S810; if not, go to Step S802;Step S810: The processor100computes the P sets of bite feature descriptors of the bite model Mb;Step S812: The processor100determines whether a predetermined quantity Nf of bite feature descriptors in a set of bite feature descriptors of the P sets of bite feature descriptors exceeds a second threshold Th2? If so, go to Step S814; and if not, go to Step S802;Step S814: The processor100performs a registration on the bite model Mb, the upper arch model, and the lower arch model to generate a full mouth model;Step S816: The processor100determines whether a registration error E is less than the third threshold Th3? If so, go to Step S818; and if not, go to Step S802;Step S818: The display12displays a full mouth model.

The difference between the operation method800and the operation method700lies in that the operation method800down-samples the bite point cloud in Step S805and uses the down-sampled bite point cloud to generate the bite model Mb in Step S806, thereby further reducing the amount of computations of the intraoral scanner1and reducing the time for generating a full mouth model. The Steps S802, S804, S808to S818of the method800are similar to the Steps S702, S704, and S708to718of the method700, explanations therefor can be found in the preceding paragraphs and will not be repeated here.

The method800uses the down-sampled bite point cloud to generate the bite model Mb, further reducing the amount of the computations of the intraoral scanner1, speeding up reconstruction of the full mouth model, enhancing the quality of a full mouth scan and relieving a patient's discomfort.