Patent Description:
For dental professionals and patients, it is of interest to get a visual impression of the appearance of the patient's face with a modified dental situation, i.e. to visualize the modified dental situation in an image of the face of the patient. Also, the appearance during or after a dental treatment may be of importance for the patient before deciding to undergo such treatment. For this purpose, a virtual preview (virtual mockup) of the dentition modified by dental treatment is helpful for the dentist and may also be used in the course of interactively modifying the treatment plan to get the most favourable aesthetic results.

<CIT> discloses a method for displaying a three dimensional model on an AR display. A plurality of intraoral images of a patient from an intraoral scanner during an intraoral scanning procedure is received and a virtual three dimensional model of at least a portion of a dental arch is generated from the plurality of intraoral images. The three dimensional model, which will be displayed on the AR display, is based on intraoral scanned images of the dental situation of the patient. <CIT> does not disclose model generation by an AR application, as well as following provision of a scan model and alignment of both models for export.

<CIT> discloses a method of designing a dental restoration for a patient, wherein the method includes providing one or more 2D images, where at least one 2D image includes at least one facial feature; providing a 3D virtual model of at least part of the patient's oral cavity; arranging at least one of the one or more 2D images relative to the 3D virtual model in a virtual 3D space such that the 2D image and the 3D virtual model are aligned when viewed from a viewpoint, whereby the 3D virtual model and the 2D image are both visualized in the 3D space. <CIT> does not disclose model generation by an AR application, as well as following provision of a scan model and alignment of both models for export.

<CIT> discloses a method for planning, visualizing, and/or optimizing dental restoration on at least a part of pre-prepared teeth of a patient, based on at least one 3D digital model of at least a part of the pre-prepared teeth. This 3D digital model is then used for providing a 3D digital model of at least part of the prepared teeth, where the prepared teeth are provided by preparing the pre-prepared teeth by dental restorative work. Thus, in <CIT> a 3D scan of the patient's dental situation is the basis for the created dental design model. <CIT> does not disclose model generation by an AR application, as well as following provision of a scan model and alignment of both models for export.

The Youtube video of Kapanu AG with title "Kapanu-Augmented Reality for the Future of Dentistry" is a promotional video demonstrating the use of an augmented reality application in the field of dentistry and in particular the visualisation of a 3D dental model by an AR application based on a scan model. Specifically, the video shows images and videos of faces and mouth regions of people, wherein these images and videos are "augmented" in the sense that a visualization of a virtual object (here: virtual mock-up/dental design model) is superimposed in the mouth region. As a result, people can get a visual impression of the appearance of the persons face with a modified dental situation, i.e. with a denture corresponding to the dental design model. The video shows in different scenes various possible ways of designing a dental design model. Neither the export of the designed dental design model to another, e.g. external, application such as a computer-aided design application nor the technical method how to achieve such an export are shown in said video.

In the context of the present invention, the term "denture" is not necessarily restricted to full dentures but also comprises partial dentures, orthodontic situations or adaptations, or dental restorations such as dental prostheses, including crowns, crown lays, veneers, inlays and onlays, bridges, dental implants, and implant restorations. Accordingly, the term "dental model" includes all models of dental prostheses - such as models of complete and partial dentures - that are used for prosthodontic purposes.

It is desirable to create a denture design in an augmented reality application showing the design in the patient's face and then be able to easily export the design to another application such as a computer-aided design application, e. to create a denture, restoration or other prosthesis from the design.

It is therefore an object of the present invention to provide an improved method that allows transferring data of a three-dimensional dental design model from an augmented reality application to a computer-aided design application.

It is another object to provide such as method wherein a three-dimensional dental design model that has been designed in an augmented reality (AR) application is transferrable to a computer-aided design (CAD) or similar application.

It is a particular object to provide such a method wherein the three-dimensional dental design model transferred to the CAD or similar application can easily be used as a basis to plan a dental treatment.

It is a further object to provide such a method wherein a pose and/or scale of the three-dimensional dental design model is easily adaptable to fit the needs of the receiving CAD application.

It is another object to provide such a method that is performed fully or semi-automatically.

At least one of these objects is achieved by the method of claim <NUM> and/or one of the dependent claims of the present application.

The present invention pertains to a computer-implemented method for designing and exporting design data of a virtual three-dimensional dental design model from an augmented reality (AR) application, the AR application being adapted to visualize the dental design model with a preliminary pose and scale in an image of a face comprising a mouth region with a dental situation. The method comprises.

In particular, the dental design model is brought to a coordinate system of the scan model.

An application to which the design data is exported can be a computer aided design (CAD) application or a similar software, for instance a 3D printer software. This other application can be an external application or an application that is connected to the AR application; for instance, the AR application and a CAD application can be incorporated as sub-applications into the same applications and/or be operable using the same graphical user interface.

According to one embodiment, the method comprises generating aligned design data based on the transformation, wherein the aligned design data comprises at least a surface geometry of the design model.

According to another embodiment, the method comprises generating aligned design data based on the transformation and obtaining an aligned design model by aligning the design model and the scan model in the common coordinate system, wherein the aligned design data comprises at least a surface geometry of the aligned design model.

According to another embodiment, the method comprises exporting the aligned design data to another application. In particular:.

According to another embodiment of the method, calculating the transformation comprises applying inverse transformation to the dental design model to maintain visually the same position of the dental design model with respect to the image.

According to another embodiment, the method comprises aligning the scan model to a known coordinate system before determining the transformation.

In one embodiment, aligning the scan model to the known coordinate system is performed automatically, using heuristics and/or statistical information.

In another embodiment, aligning the scan model to the known coordinate system comprises correcting a scale of the scan model, in particular by orders of magnitudes.

According to a further embodiment of the method, a model based alignment (e. edge based alignment) is used for aligning the dental situation of the scan model and the dental situation in the image, the model based alignment comprising at least:.

A segmentation of teeth and gum in the scan model may be performed using colour information of the scan model and/or using heuristics with position and surface curvature information from the scan model, wherein determining the pose of the scan model is based on the segmentation. The invention also pertains to a computer programme product comprising programme code which is stored on a machine-readable medium, or being embodied by an electromagnetic wave comprising a programme code segment, and having computer-executable instructions for performing one of the above methods.

The invention in the following will be described in detail by referring to exemplary embodiments that are accompanied by figures, in which:.

<FIG> illustrates an exemplary embodiment of a method for exporting a 3D dental design model <NUM>, bringing it into the coordinate system of a 3D scan of an actual dental situation. For instance, to plan a potential dental treatment, a preliminary dental design model is designed using an augmented reality application (AR app) and then transferred to a CAD software.

As illustrated in <FIG>, the input comprises:.

The desired output is an aligned 3D dental design model <NUM>, i.e. a three-dimensional model of designed teeth in the same coordinate system as the scan model <NUM> and matching the scale of the scan model <NUM>.

The 3D scan model <NUM> may have been created by scanning the oral cavity of the patient or by scanning the impression of the dentition taken by impression trays filled with impression material. Creating the 3D scan model <NUM> may be part of the method. Alternatively, an existing model may be retrieved. Such three-dimensional model of the dentition of the patient may already be present, e. as a basis for developing a digital dental treatment plan, for example by adding artificial teeth or other dental restorations or by modifying the dental situation in another manner, for example by correction of teeth positions. The 3D scan model <NUM> may comprise the complete oral cavity or only parts thereof.

The 3D design model <NUM> can be a preliminary design. In particular, the 3D design <NUM> may have a wrong scale or be completely unscaled. The estimated pose <NUM> can be computed automatically with optional user adjustments in the AR app.

As illustrated by the flow chart of <FIG>, an exemplary embodiment of the method <NUM> comprises the following steps:
If the 3D scan model <NUM> is not already aligned (at least roughly) to the known coordinate system, this alignment is done in step <NUM>. <FIG> illustrates, by way of example, the original unaligned position of the 3D scan model <NUM> and the position of the aligned 3D scan model <NUM>' after having performed said alignment <NUM>. The alignment particularly comprises rotation and translation of the scan model <NUM>. For instance, an automated system can use heuristics or statistical information to find the correct alignment, potentially also correcting the scale e. by orders of magnitudes. An auto-alignment, for instance, may comprise the following sub-steps:.

In step <NUM>, the estimated denture pose <NUM> is further improved using edge based alignment. The edge based alignment is illustrated in <FIG> and disclosed in detail in the applicant's <CIT>. This improvement of the pose may be performed automatically with optional fine-tuning by a user.

It may comprise applying inverse transformation to design to keep the design visually at the same position with respect to the image.

The 3D scan model <NUM> of the denture is rendered by the virtual camera as a two-dimensional image of the denture, wherein the virtual camera is operated assuming an estimated positioning which is estimated to coincide with the positioning of the real camera when recording the image <NUM> of the patient's face. Preferably, an aperture angle of the camera optical system should to be known. If it is unknown, it can be estimated instead.

The image <NUM> of <FIG> and the rendered image <NUM>" of the 3D scan model of <FIG> are then separately processed by carrying out feature detection in a dentition area inside the mouth opening in the respective images <NUM>, <NUM>" by performing edge detection and/or color-based tooth likelihood determination in the respective images <NUM>, <NUM>". The image <NUM> does not have to include the entire face, as the region of the mouth opening is sufficient.

For the detected feature (edges or tooth likelihood) or for each of the detected features (edges and tooth likelihood), this results in two detected feature images (one resulting from the camera image <NUM> and one from the rendered image <NUM>") which are then used to calculate a measure of deviation between the detected feature images. Ideally, if the estimated positioning should already coincide with the real positioning of the camera when recording the face image, the measure of deviation would be zero or very small since the detected features (edges or tooth likelihood pattern) would be in identical positions in the two images <NUM>, <NUM>", and therefore there would be no deviation of the detected features in the two images <NUM>, <NUM>". However, in practice there will be a certain deviation at the beginning when an estimated position of the virtual camera is used. For this reason, the position of the virtual camera is varied to a new estimated position. This is iteratively repeated in an optimization process to minimize the deviation measure to determine the best fitting positioning of the virtual camera to arrive at the situation depicted in <FIG>.

To facilitate the deriving of the pose, optionally, a rough segmentation <NUM> of teeth and gum can be performed before step <NUM>. This can be done using heuristics with position and surface curvature information or (if available) using heuristics from colour information. Alternatively, if such a segmentation is already comprised in the mesh, that information can be used. This step is also illustrated in <FIG> showing a mesh without segmentation (teeth and gum having the same colour) and <FIG> showing the same mesh after the segmentation <NUM> of teeth and gum (gum having a darker colour than teeth here).

As illustrated in <FIG>, the scaling and distance can be corrected <NUM> with the assumption that scan model <NUM> and dental design model <NUM> should be at a similar distance to a virtual camera VC. This step solves a potential problem that might occur when an unexperienced user designs the dental design model <NUM> in an augmented reality application and relates to an apparent size of the model in the AR application. Although the dental design model <NUM> may look correct, it could be too far in the front (and thus be smaller than intended) or too far behind the scan model <NUM> (and thus be larger than intended).

In <FIG>, the dental design model <NUM> is much smaller than the scan model <NUM>. As it is also positioned closer to the virtual camera VC, in the augmented reality image of <FIG>, it appears to have the correct size. After having corrected the distance d to the dental design model <NUM> (see <FIG>), the scan model <NUM> and the dental design model <NUM> can be aligned so that the aligned dental design model <NUM> has the correct scale, i.e. so that the apparent size in the image of <FIG> is the same as in the image of <FIG>.

Referring to <FIG> again, finally, the aligned 3D design model <NUM> can be exported <NUM> (e. using STL file format) in the same coordinate system as the scan model <NUM>. In particular, the model is exported to be used in a CAD application. However, the model can also be exported for use in other kinds of applications, such as, for instance, in a 3D printer.

Advantageously, the 3D design model may be exported in the same file together with the scan model. <FIG> illustrate the result by way of example, where the scan model <NUM> and the aligned 3D design model <NUM> are depicted aligned to each other.

In the following, an exemplary embodiment of the method <NUM> is described by way of equations. A 3D point of the design can be projected onto the image with <MAT> and the scan can be visualized in the image with: <MAT> where:.

However, the scan vacan2D_wrong usually does not fit precisely, as the estimated denture pose Dapp and the alignment A both are not exact. An improved denture pose <MAT> needs to be found, such that the fit is as good as possible: <MAT>.

Dcorr can be computed with edge based alignment or defined manually.

According to step <NUM>, with the adjusted denture pose Drefined, the projection of the design should stay at the same position: <MAT> <MAT> with <MAT>.

There is still an ambiguity in scale and distance from camera. As the true scale of the scan is known, the distance where it matches the image is known, too. Assuming that the distance d1 of the design from the camera should be the same as the distance d2 of the scan from the camera, the distance can be corrected with Tdist. For the visual result to stay the same, in step <NUM>, the model needs to be scaled by d2/d1 with Tscale: <MAT>.

Finally, in step <NUM>, the design can be exported in the same coordinate system and same scale as the scan: <MAT>.

<FIG> illustrates an exemplary embodiment of a method (not covered by the present claims) for visualizing an imported 3D dental design model <NUM> in an AR application, bringing a design resulting from a CAD or simulation (e. orthodontics) into a specific pose matching to a facial image <NUM> for an Augmented Reality overlay. The objective is to display the dental design in the image <NUM> in the mouth of a person. For instance, to preview a dental situation in the face of a patient, the situation is designed as a dental model using a CAD application and then transferred to an AR application software.

As illustrated in <FIG>, the input comprises at least one facial image <NUM> with at least part of the mouth visible, a 3D scan model <NUM> of the oral cavity, wherein the 3D scan model <NUM> has a coordinate system that is not necessarily aligned to the coordinate system of the face in the image <NUM>, and the imported 3D dental design model <NUM>, e. having been designed previously in a CAD application.

The output comprises a pose <NUM> of the 3D design model <NUM> corresponding to the image <NUM> and the corresponding rendered image <NUM>.

The 3D scan <NUM> as well as the 3D dental design model <NUM> optionally may comprise colour information and/or a segmentation of teeth and gum.

As illustrated by the flow chart of <FIG>, an exemplary embodiment of the method <NUM> (not covered by the present claims) comprises the following steps:
In a first step <NUM>, the 3D scan <NUM> and the 3D design <NUM> are - at least roughly - aligned to a specified coordinate system. The same transformation is used for both 3D models <NUM>, <NUM>, i.e. both 3D models stay registered relative to each other. The 3D scan <NUM> and the 3D dental design model <NUM> optionally can be provided in the same coordinate system. Otherwise, they can be aligned to be in a common specified coordinate system.

A rough segmentation <NUM> of teeth and gum can then be performed in the 3D scan <NUM>. This can be done using heuristics with position and surface curvature information or (if available) using heuristics from colour information. Alternatively, if such a segmentation is already comprised in the mesh of the 3D scan <NUM>, that information can be used.

Especially if the 3D dental design model <NUM> does not already have colour information, in step <NUM> such colour information can be added to the model <NUM>. This can be done either manually, semi- or fully automatically, e. using statistical information and heuristics. Colour information is important to let the design model <NUM> look more realistic in the AR application.

In step <NUM>, the final pose <NUM> of the 3D design <NUM> is obtained. After step <NUM>, the pose of the 3D design <NUM> is already in the specified coordinate system. The final pose <NUM> may be calculated using pose estimation from the dental AR application. The calculation for instance can be based on information about the segmentation of teeth and gum in the 3D scan <NUM> and on colour information from the 3D dental design model <NUM> (i.e. based on the information obtained in previous steps <NUM> and <NUM>). The final pose <NUM> optionally can be further improved, e. by using a 3D scan for applying edge based alignment as described above with respect to <FIG> and disclosed in the <CIT>.

In the last step <NUM>, the 3D dental design model <NUM> is rendered and a 2D representation of the model having the final pose <NUM> is displayed in the dental AR application, e. in the facial image <NUM> to provide the rendered image <NUM> as a result.

In an exemplary embodiment, a method for designing and exporting design data of a virtual three-dimensional dental design model is provided. The method includes providing an augmented reality visualizer configured to allow designing the dental design model by a user and displaying the dental design model. The design model has a preliminary pose and scale. The dental model is shown in an image of a face and the face image includes at least a portion of a mouth region and a dental situation. A 3D scan model of the dental situation is obtained. The scan model and the dental situation of the image are aligned to obtain a pose of the scan model. The method includes calculating, using a processor, at least one transform to bring the design model and the scan model into a common coordinate system based on the preliminary pose and scale and the pose of the scan model. The method further comprises obtaining an aligned design model by (i) aligning the design model and the scan model in the common coordinate system, and by (ii) calculating a corrected scale of the dental design model and applying the corrected scale so that the scale of the aligned design model matches the scale of the scan model. Calculating the corrected scale comprises calculating a distance of the dental design model to a virtual camera to match a distance of the scan model to the virtual camera.

An export dataset of aligned design data is generated based on the at least one transform. The aligned design data comprises at least one surface geometry of the design model. At least one model may be exported such that both models are aligned in a common coordinate system.

In the context of the present invention, the term "denture" is not necessarily restricted to full dentures but also comprises partial dentures or orthodontic situation/adaptations or dental restorations such as dental prostheses, including crowns, crown lays, veneers, inlays and onlays, bridges, dental implants, implant restorations. Accordingly, the term "dental model" includes all models of dental prostheses as well as the patient situation that could be partial or fully edentulous - such as models of complete and partial dentures - that are used for prosthodontic purposes.

In some embodiments, the present disclosure is implemented using a system having a camera, a processor, an electronic data storage unit, and a display. The camera can be a standard camera, an infrared dot-projection detector, flood illuminator camera, structured-light three-dimensional scanner, standard infrared detector, ultrasonic imaging device, Doppler detector, or any other suitable visualization system capable of capturing information related to a patient's dentition. The processor can be a single processor having one or more cores, or a plurality of processors connected by a bus, network, or other data link. The electronic data storage unit can be any form of non-transitory computer-readable storage medium suitable for storing the data produced by the system. The display can be any display suitable for displaying a digital color or grayscale image.

In some embodiments, the camera, processor, electronic data storage unit, and digital display are components of a single device. The single device may be a smartphone, tablet, laptop computer, personal digital assistant, or other computing device.

In some embodiments, the processor is in communication over a network, which could be wired or wireless, with an external processor used for performing one or more calculation steps and/or a network-attached electronic data storage unit. In some embodiments, the present disclosure makes use of cloud computing to perform one or more calculations steps remotely and/or remote storage to enable the storage of data remotely for collaborative or remote analysis. In some embodiments, the system comprises a plurality of graphical user interfaces to permit multiple users to view or analyze the same data.

In some embodiments, the system operates to provide one or more users with a visualization of a virtual dental model of a patient's teeth, which may be altered to visualize the effect of one or more dental or orthodontic alterations. In some embodiments, this allows the one or more users to visualize a "before" dentition image, i.e., the appearance of a patient's dentition prior to a dental or orthodontic procedure, and an "after" dentition image, i.e., a representation of the expected appearance of a patient's dentition after a proposed dental or orthodontic procedure.

In some embodiments, the system operates by capturing information related to a patient's dentition using a camera, creating a model of the patient's dentition on a processor, fitting a model of a proposed post-alteration dentition to the patient's dentition on the processor, coloring the model of the proposed post-alteration dentition to match an expected real post-alteration coloration, and displaying the fitted model of the proposed post-alteration dentition in place of the patient's actual dentition on a display which otherwise shows the patient's actual facial features. The information related to a patient's dentition, the model of the patient's dentition, and the model of the proposed post-alteration dentition may be stored on an electronic data storage unit. In some embodiments, the operations are performed in real-time.

In some embodiments, a user interface is configured such that a user may view the "before" dentition image and the "after" dentition image simultaneously either side-by-side or with a full or partial overlay.

Claim 1:
Computer-implemented method (<NUM>) for designing a virtual three-dimensional dental design model (<NUM>) using an augmented reality application, the augmented reality application being adapted to visualize the dental design model (<NUM>) with a preliminary pose and scale (<NUM>) in an image (<NUM>) of a face comprising a mouth region with a dental situation, and for exporting the dental design model (<NUM>), the method comprising:
- creating the dental design model (<NUM>) using the augmented reality application, whereafter for exporting the dental design model the following steps are carried out:
- providing a three-dimensional scan model (<NUM>) of the dental situation,
- aligning the scan model (<NUM>) and the dental situation in the image (<NUM>) to obtain a pose of the scan model (<NUM>),
- calculating, based on the preliminary pose and scale (<NUM>) and the pose of the scan model (<NUM>), at least one transformation to bring the design model (<NUM>) and the scan model (<NUM>) into a common coordinate system, in particular wherein the dental design model (<NUM>) is brought to a coordinate system of the scan model (<NUM>), and
- obtaining an aligned design model (<NUM>) by
aligning the design model (<NUM>) and the scan model (<NUM>) in the common coordinate system, and
calculating (<NUM>) a corrected scale of the dental design model (<NUM>) and applying the corrected scale so that the scale of the aligned design model (<NUM>) matches the scale of the scan model (<NUM>), wherein calculating (<NUM>) the corrected scale comprises calculating a distance of the dental design model (<NUM>) to a virtual camera (VC) to match a distance (d) of the scan model (<NUM>) to the virtual camera.