Patent Description:
Conventional cameras transform a three-dimensional view of an object into a two dimensional image. Typically, the depth dimension, corresponding to the distance between the focal plane of the captured image and the camera, is lost. To include a depth characteristic, some optical systems use two cameras to capture a pair of stereo images of the object, much the way our eyes work. Each image of the pair is acquired from a slightly different viewing angle, and the discrepancy between the two images is used to measure depth. An example to this approach is known from <CIT>.

The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools and methods which are meant to be exemplary and illustrative, not limiting in scope.

One aspect provides a system according to claim <NUM>.

In some embodiments, the processor is further configured to indicate on the rendition of the incremental construction a comparison between a viewing distance of the currently acquired image pair and an optimal viewing distance.

In some embodiments, the processor is configured to indicate the comparison by displaying a distance graphic indicator corresponding to the viewing distance of the currently acquired image pair superimposed with the tracking graphic indicator.

The processor is configured to register the currently acquired image pair to a location on the 3D model by: calculating a distance between the camera and the surface as a function of a disparity of the currently acquired image pair, determining a point cloud for the currently acquired image pair using the calculated distance, and mapping multiple key-points in the point cloud to a region on the 3D model in proximity to a most recently added image pair, determining from the mapping when the registration succeeds and when the registration fails, calculating the delta as a function of a discrepancy between the mapped point cloud and the 3D model when the registration succeeds, and wherein adding the currently acquired image pair to the 3D model comprises adding the mapped point cloud to the 3D model.

The processor is configured to determine the point cloud by: binning the image pair, calculating a low resolution distortion from the binned image pair, calculating an average distance from the low resolution distortion, using the average distance to select a distortion correction table corresponding to the average distance, applying the selected distortion table to correct a distortion of the image pair, and wherein the disparity of the image pair comprises a disparity of the corrected image pair, and wherein determining a point cloud comprises determining the point cloud from the corrected image pair.

In some embodiments, the processor is configured to track the incremental construction by indicating on the rendition of the incremental construction when the registration succeeds and when the registration fails.

In some embodiments, the processor is configured to indicate when the registration succeeds by displaying the tracking graphic indicator using a first color, and wherein the processor is configured to indicate when the registration fails by displaying the tracking graphic indicator using a second color.

In some embodiments, the processor is further configured to, responsive to determining that the registration fails, execute a relocking procedure.

In some embodiments, the relocking procedure comprises attempting to register the most recently acquired image pair in proximity to the registered location corresponding to a previously added image pair until the registration succeeds, wherein the previously added image pair is selected from a group consisting of: the ten most recently added image pairs, the twenty most recently added image pairs, the thirty most recently added image pairs, and the forty most recently added image pairs.

In some embodiments, the system further comprises a user interface, wherein the processor is configured to receive a user-indicated location via the user interface, and wherein the relocking procedure comprises registering the most recently acquired image pair in proximity to the user-indicated location on the 3D model.

In some embodiments, the system further comprises a memory, wherein the processor is further configured to retrieve a previously constructed 3D model from the memory, wherein registering the currently acquired image pair comprises registering to a location on the retrieved previously constructed 3D model, and wherein adding the currently acquired image pair at the registered location comprises adding the currently acquired image pair to the previously constructed 3D model.

In some embodiments, rendering the incremental construction further comprises differentiating between the added acquired image pairs and the previously constructed 3D model.

In some embodiments, the rendering the incremental construction of the 3D model on the display comprises continually adjusting the orientation of the 3D model responsive to detected changes in at least one of a horizontal and a vertical viewing angle of said stereoscopic camera.

In some embodiments, the processor is configured to calculate at least one of the horizontal and the vertical viewing angle of said stereoscopic camera from the distortion correction table.

A system and method are disclosed herein for implementing real-time tracking for 3D image reconstruction. A stereoscopic camera scans a surface and provides a stream of image pairs that are incrementally added to construct a high-resolution three-dimensional (3D) model of the surface in real-time. The camera may be a hand-held camera, and thus the user may require feedback to ensure that the camera is positioned at the correct distance and orientation from the surface in order to capture useful, quality images. To provide such feedback, the system tracks the scanning by the camera to the construction of the model in real-time. The construction of the 3D model is displayed to the user while scanning, and the addition of newly acquired images are indicated in real-time on the displayed 3D model, allowing the user to see the relative location of the currently scanned surface on the 3D model.

The description below describes a stereoscopic optical system (<FIG>, and <FIG>) for capturing high resolution image pairs, as well as methods for constructing a 3D model from those image pairs. However, it may be appreciated that this combination is not meant to be limiting, and the method may use other optical systems for acquiring the image pairs necessary to construct the model.

Reference is now made to <FIG> which, taken together, show a stereoscopic optical imaging system for providing multiple high resolution image pairs for constructing a 3D model, in accordance with an embodiment.

An imaging system <NUM>, such as a camera, is provided to capture multiple stereo images of a 3D object <NUM>, such as skin. Camera <NUM> is provided with a front (objective) lens <NUM> for collecting light reflected off object <NUM>. The collected light is transmitted through one or more apertures of a mask <NUM> to a pair of back lenses 106a and 106b, which focus the collected light onto one or more sensors <NUM>, such as may comprise any suitable imaging sensor, for example a charged coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS). Optionally, the collected light may be collected at multiple regions of a single sensor <NUM>, shown in <FIG> as two sensor regions 108a, 108b. Sensor regions 108a, 108b may capture multiple pairs of images of object <NUM> in stereo. The captured image pairs may be received by a processor <NUM> which uses the image pairs to construct a 3D model of object <NUM> and store the model in a memory <NUM>. The constructed 3D model may be rendered on a display <NUM>.

Referring to <FIG>, an exemplary view of sensor <NUM> having two distinct regions, 108a and 108b is shown. Alternatively, multiple individual sensors 108a and 108b may be provided for each of the imaging systems.

Camera <NUM> may constitute two imaging systems, each system comprising one of lenses 106a and 106b and one of sensor regions 108a and 108b, and sharing common objective <NUM>. The two imaging systems may allow imaging along separate optical paths, each corresponding to a different one of viewing angles θ<NUM>, θ<NUM> off of object <NUM>, thereby allowing simultaneous stereo image acquisition. Each of lenses 106a and 106b and sensor regions 108a and 108b may be dedicated one of the optical paths. This design may allow the overall length of the imaging systems to be constrained within a predefined size limitation, such as for implementing within a hand-held device. Alternatively, a separate objective (not shown) may be provided for each optical path.

The imaging systems may be telecentric in the sample space of object <NUM> such as by positioning mask <NUM> in the rear focal plane of objective <NUM>, allowing to decouple the defocusing and magnification of object <NUM>. Optionally, back lenses 106a and 106b may operate under telecentric conditions such as by positioning mask <NUM> in the front focal plane of back lenses 106a and 106b. Optionally, the distance between mask <NUM> to back lenses 106a and 106b may be less than the focal length of back lenses 106a and 106b to allow the images of object <NUM> to expand on reaching the corners of sensor regions 108a and 108b. The telecentric imaging described thus may allow for uniform scaled imaging by providing a constant field of view (FOV), and thus, regions of object <NUM> positioned either above or below the best-focus plane may be imaged at the same size-scale as regions positioned at the optimum focus. This property may be useful when combining the multiple different captured images by processor <NUM> for performing the 3D reconstruction of object <NUM>.

Mask <NUM> may have two or more apertures for transmitting the collected light, each aperture corresponding to a different one of the optical paths. In one implementation, mask <NUM> includes a pair of round holes to produce the desired F-number (F#) at object <NUM>, such as illustrated in <FIG>, where F# is understood to be a measure of the amount of light collected by imaging system <NUM>.

System <NUM> may be designed to image object <NUM> positioned at or near the front focal plane of the front lens such that sensor regions 108a and 108b are infinite conjugates. Thus, light reflected off object <NUM> at angle θ<NUM> may be focused via objective <NUM> through one aperture of mask <NUM> and focused via lens 106a onto sensor region 108a, and light reflected off sample <NUM> at angle θ<NUM> may be focused via objective <NUM> through a second aperture of mask <NUM> and focused via lens 106b onto sensor region 108b. In this manner, different points on object <NUM> imaged at different angles θ<NUM>, θ<NUM> may be mapped onto different regions of the mask plane and different regions of the sensor plane, comprising a different imaging system for each viewing angle. Similarly, light rays reflecting off a single point of object <NUM> at different angles θ<NUM>, θ<NUM> may be parallel when they arrive at mask <NUM>, and transmitted, respectively through the different apertures via back lenses 106a and 106b to sensor regions 108a and 108b. In this manner, the two imaging systems together allow the simultaneous stereo imaging from multiple different angular views of object <NUM>. Optionally, each viewing angle may be imaged sequentially at sensor <NUM>.

The apertures on mask <NUM> may be positioned symmetrically opposite about the viewing axis of camera <NUM>, allowing two slightly different views of the 3D surface to be obtained. The disparity Δ between the two different captured views may be computed and used to determine a depth attribute of the imaged 3D surface. The disparity may be computed as the differences between the lateral (X, Y) positions of one or more identified features in the two images. A 3D map of the imaged object may be formed by computing the disparity between each identified feature in the two captured views. The disparity may be computed using any suitable algorithm such as are known in the art of stereoscopy. The depth can be calculated using the following equation: <MAT>.

Where F<NUM> is the front lens (objective) focal length, F<NUM> is the back lens focal length, b is the aperture spacing, Δ is the disparity, and Z is the depth. Values for F<NUM> may range between <NUM>-<NUM> millimeters (mm) ± <NUM>%, values for F<NUM> may range between <NUM>-<NUM> ± <NUM>%, and values for b may range from <NUM> to <NUM> ± <NUM>%.

Typically, there is a tradeoff between depth of focus, resolution and light level. For round apertures this tradeoff may be described by the following equations: <MAT> <MAT> <MAT> where DA is the diameter of the aperture and λ is the wavelength.

The field of view (FOV) of an imaging system is governed by the diameter DL of objective lens <NUM>, focal length F<NUM>, and F# of the objective, as follows: <MAT> where α is the angle between the right and left images as measured in radians.

Since a large FOV typically requires a large objective, resulting in a heavy and bulky optical system, the FOV may be constrained to allow camera <NUM> to have a size and weight that are suitable for a handheld device. To compensate for a smaller FOV, object <NUM> may be scanned to capture many consecutive image pairs. Equation (<NUM>) may be applied to each of the image pairs acquired using the stereo imaging system above, to calculate the depth attribute, or a 3D point cloud, for each image pair. A registration algorithm may be used to add all the calculated 3D point clouds together and form a large 3D point cloud representing the scanned region.

Any noise of a single 3D point cloud may be accumulated in the registration process, resulting in a significant noise level for the large 3D point cloud. To limit the sensitivity to noise, camera <NUM> may be designed such that the angle α between the left and right images (the image pairs acquired in stereo), may be substantially small, such as ~<NUM>°. Alternatively, the angle α between the left and right images may range from <NUM>° and <NUM>°, or <NUM>° and <NUM>°, or <NUM>° and <NUM>°. Thus, the features in the left image and right image may be very similar, allowing a high degree of accuracy in discerning features along the lateral, x, and vertical, y axes. However, there may remain a non-negligible distortion along the depth, z axis.

Reference is now made to <FIG> which illustrate an optical imaging system <NUM> having a corrective lens to reduce optical distortion, in accordance with another embodiment.

Distortion is an optical aberration which incorrectly maps a point on the real object to the image space. This incorrect mapping of points may have a substantial effect on 3D point clouds. Following Eq. (<NUM>), the relationship dz between z (depth) and disparity (Δ) in a system with a relatively small angle between left and right images, such as ~<NUM>° (<NUM>) is dz~<NUM>Δ. In such a system, even very low distortion, such as tenths of a percent, may have a non-negligible effect on the large 3D point cloud. To avoid such errors, telecentric optical imaging system <NUM> may capture images with very low distortion, such as < <NUM> %. Additionally, telecentric optical imaging system <NUM> provides a constant FOV with a uniform number of pixels.

The distortion in an optical system is also a function of depth; when in focus, an optical system may acquire images with very low distortion, however images acquired far from focus may suffer from high distortion. To address this problem, a lens <NUM> may be provided to reduce the distortion of the optical system and change very gradually along the entire depth of focus, resulting in a relatively low distortion along the depth of focus. By reducing the distortion for each captured image, the cumulative error for the 3D point cloud resulting from the registration may be reduced significantly.

<FIG> shows a single aperture system for imaging object <NUM>. Light reflected off object <NUM> is collected by objective <NUM>. The collected light is focused by corrective lens <NUM> and transmitted through the aperture of mask <NUM>, via one or more back lenses <NUM> onto sensor <NUM>.

The system of <FIG> is substantially similar to that of <FIG> having two imaging systems for two optical paths each corresponding to a different viewing angle of object <NUM>, with the notable difference that corrective lens <NUM> is positioned between objective <NUM> and the aperture plane of mask <NUM>. Lens <NUM> coupled with lens <NUM> may reduce distortion of images acquired using system <NUM>. Light reflected off object <NUM> in two separate optical paths is collected by objective <NUM> coupled with lens <NUM>, focused onto mask <NUM>, and transmitted through multiple aperture pairs of mask <NUM> via back lenses 106a and 106b onto sensor regions 108a and 108b, respectively.

The overall length of the system <NUM> of <FIG> may range, for example, from <NUM> to <NUM>. In one embodiment, the overall length is approximately <NUM> ±<NUM>%. The maximum diameter of system <NUM> may range, for example, from <NUM> to <NUM>, and in one embodiment may be approximately <NUM> ±<NUM>%. The average distortion may range from <NUM>% to <NUM>% and in one embodiment may be approximately <NUM>%. The FOV may range, for example, from 30x30mm to 60x35mm, and in one embodiment, may be approximately 60x30mm ±<NUM>%. The depth of focus may range, for example, from <NUM> to <NUM>, and in one embodiment, may be approximately <NUM> ±<NUM>%. The 3D resolution may range, for example, from <NUM> to <NUM> microns, and in one embodiment, may be <NUM> microns ±<NUM>%.

Sensor regions 108a and 108b configured with either of systems <NUM> or <NUM> may acquire multiple image pairs and transmit them for processing by processor <NUM>. The image acquisition rate and transmittal rate may range from <NUM>-<NUM> Hertz (Hz) ± <NUM>%, or alternatively from <NUM>-<NUM> ± <NUM>%, or <NUM>-<NUM> ± <NUM>%.

Reference is now made to <FIG> which shows a flowchart of a method for the real-time tracking of the scanning of a 3D surface using the system of either of systems <NUM> or <NUM>, in accordance with an embodiment. The image construction and tracking be implemented by processor <NUM> operative with memory <NUM> and display <NUM>.

Multiple pairs of stereoscopic images of a surface are acquired sequentially from a camera (Step <NUM>). A 3D model of the surface is constructed incrementally from the image pairs concurrently with the sequential image acquisition (Step <NUM>). The incremental construction includes performing Steps <NUM>-<NUM>, for each currently acquired image pair: the currently acquired image pair are registered to a location on the 3D model (Step <NUM>), details of which are provided below with respect to <FIG>; if the registration succeeds and if a delta between the registered image pair exceeds a predefined threshold, the currently acquired image pair is added to the 3D model at the registered location (Step <NUM>), such as by performing Step <NUM> of <FIG>. The incremental construction of the 3D model may be rendered on a display (Step <NUM>). The incremental construction of the 3D model may be tracked concurrently with the sequential image acquisition (Step <NUM>), such as by displaying a graphic indicator on the rendition of the incremental construction of the 3D model. The graphic indicator may simultaneously indicate on the rendered 3D model both the registered location, and when the current camera-to surface viewing distance is within a focal range of the camera, and when the current camera-to surface viewing distance is not within a focal range of the camera. This allows the user to observe the scanned surface region using the 3D model as a reference, allowing him to maneuver the camera according to the regions of the model that require additional scanning. It also provides the user with feedback that the images that are currently being scanned are in focus and are thus registered and added to the model.

The tracking indicator may have multiple different attributes to indicate various system parameters, allowing the user to adjust the position of the camera and/or speed of acquisition in response to the indications. For example, the tracking indicator may be displayed as a spot indicating to the user the viewing distance between the camera and the surface for the most recently registered image pair. Optionally, as the user moves the camera closer to the surface to acquire images, the tracking indicator will be smaller. Conversely, as the camera is moved further from the surface to acquire images, the tracking indicator will be larger. Thus the user may follow the position of the spot on the rendered model to view where on the model the currently acquired images are being added and registered, and the tracking indicator on the spot may indicate the distance between the camera and the surface. This tracking feedback may be provided in real-time while scanning the surface, allowing the user to adjust any of the scanning region and the distance to the surface, and the scanning speed, accordingly, and acquire well-focused images from relevant regions of the surface.

Alternatively, the size of the tracking indicator 'spot' may be constant, indicating where on the model the currently acquired image pair is registered, and added. A separate distance indicator may be displayed to indicate to the user a comparison between the current viewing distance and optimal viewing distance for the imaging system, as follows:.

Thus, the combination of the tracking and distance indicators may allow the user to adjust the current viewing distance in real-time while scanning to remain within a range of the optimal focal distance and acquire quality, focused images.

The graphic tracking indicator may indicate when the registration succeeds and thus, the images acquired from the scanning are being added to the model, and conversely when the registration fails. For example, successful registration may be indicated by displaying the indicator in one color, such as green, and failed registration may be indicated by the displaying the indicator in a different color, such as red. A red 'failed registration' indication may allow the user to maneuver the camera while executing a relocking of the image acquisition to the construction of the 3D model, details of which are provided below with respect to <FIG>. Optionally, the user may indicate a new registration location on the 3D model, and may adjust the distance of the camera to focus on the new registration location. The tracking may follow this adjustment providing the user with feedback to successfully relock.

Reference is now made to <FIG>, which show an exemplary implementation of the tracking method described above with respect to <FIG>.

Referring to <FIG>, a tracking indicator <NUM> is shown overlaid on a 3D model <NUM>. Indicator <NUM> is shown as a spot (green) corresponding to an optimal focal distance between the camera and the surface, and The position of indicator <NUM> indicates the location on the model of the most recently registered image pair. The color of indicator <NUM> (green) indicates successful registration of the acquired image stream, and thus the distance between the camera and surface is within the focal range of the imaging system. The absence of the additional distance graphic indicator indicates that the distance is within a predefined range of the optimal focal distance.

Referring to <FIG>, two graphic indicators: distance indicator <NUM> and tracking indicator <NUM> are shown superimposed with each other. The size of distance indictor <NUM> (outer dashed ring) is scaled to correspond to the camera-to-surface distance for the most recently acquired image pair, and the size of tracking indicator <NUM> (inner spot) is constant, indicating to the user that the current scanning distance exceeds the optimal scanning distance. However, the color of indicators <NUM> and <NUM> (green) show that the camera is still within a threshold range of the optimal distance that allows the images acquired from the scanning to be registered. This feedback warns the user not to increase the distance else the registration will fail, and allows the user to adjust the distance to conform with the optimal range.

Referring to <FIG>, two graphic indicators: tracking indicator <NUM> and distance indicator <NUM> are shown superimposed with each other. The size of distance indictor <NUM> (inner dashed ring) is scaled to correspond to the camera-to-surface distance for the most recently acquired image pair, and the size of indicator <NUM> (outer spot) is constant, indicating to the user that the distance is smaller than the optical focal distance. As above, the color attribute of indicators <NUM> and <NUM> (green) indicates that the distance is still within a threshold range of the optimal focal distance, and thus the images acquired from the scanning are being registered successfully. This feedback warns the user not to decrease the distance else the registration will fail, and allows adjusting the distance accordingly.

Referring to <FIG>, the color of indicator <NUM> (red) indicates that the camera is outside the threshold focus range and the acquired images are not being registered. The color attribute of indicator <NUM> may be substantially different than the color attributes of indicators <NUM>- <NUM> of <FIG>, to give a clear indication that the image registration at the current orientation and/or distance failed.

It may be appreciated that the specific graphical shapes and/or colors for indicators <NUM>- <NUM> are exemplary implementations only, and other suitable graphical depictions may be used. Some possible threshold values are <NUM>, or <NUM>, or <NUM>. The color attribute of indicators <NUM>- <NUM> in Figs. 6A-6C may be the same, to indicate successful tracking for all of these scenarios.

Reference is now made to <FIG> which shows a flowchart of method to register of the currently acquired image pair to the 3D model, in accordance with an embodiment. A distance between the camera and the surface is calculated as a function of a disparity of the currently acquired image pair (Step <NUM>), for example, by applying Eq. <NUM> above. A point cloud for the currently acquired image pair is determined using the calculated distance (Step <NUM>). For example, the point cloud may be determined using the method described in greater detail below with respect to <FIG>.

Multiple key points in the point cloud of the currently acquired image pair may determined, and then mapped to corresponding multiple key points of a point cloud of the 3D model (Step <NUM>). The mapping may be used to determine if the registration succeeds or if the registration fails (Step <NUM>). A more detailed description of Step <NUM> is given by steps <NUM>-<NUM> of <FIG>. If the registration succeeds, indicate the success (Step <NUM>), and calculate the delta between the registered image pair and the 3D model, for example as a function of a discrepancy between the mapped point cloud and the 3D model (Step <NUM>). If the delta is significant and exceeds the threshold, the mapped point cloud may be added the to the 3D model (Step <NUM>), corresponding to the adding step <NUM> of <FIG>.

Reference is now made to <FIG> which shows a flowchart of a method to register a point cloud of a currently-acquired image pair to a previously-calculated point cloud, in accordance with an embodiment.

A received high-resolution pair of images is binned, for example by combining the pixel values in a 2x2 or 3x3 or 4x4 or 5x5 or 6x6 binning operation and a low resolution distortion may be calculated from the binned images (Step <NUM>).

An average distance between the camera and the 3D surface is calculated using the low resolution calculation of the distortion (Step <NUM>).

The average distance is used to select a distortion correction table corresponding to the average distance (Step <NUM>). Optionally, the distortion table may be selected from multiple distortion correction tables stored in memory <NUM>.

The selected distortion table is applied to correct the distortion of the image pair (Step <NUM>). The distortion corrected high-resolution image pair is used to calculate a disparity of the corrected image pair, and may be used for the disparity value of Step <NUM> of <FIG>.

Additionally, the distortion corrected images may be used to calculate the angle and the distance between the camera and the surface, as well as determining the 3D point cloud of Step <NUM> of <FIG>, as defined by Eq. <NUM> (Step <NUM>). The angle may be used to reorient the model, as described below with respect to <FIG>.

Multiple key points in the 3D point cloud may be determined (Step <NUM>).

The key points may be mapped to corresponding reference key-points identified in the point cloud of the 3D model, such as may be located in proximity to a region corresponding to a previously successfully added image pair (Step <NUM>). This yields multiple pairs of key points, each pair having a key point from the current point cloud and the previous point cloud (of the 3D model).

Then, a registration transformation matrix is calculated using the matched pairs of key-points, to correctly register (by rotation, distortion, etc.) the two point clouds (Step <NUM>).

If the registration is successful, the method may continue to Step <NUM> of <FIG>. If the registration is not successful, the indicator may indicate the registration failure (Step <NUM>), providing the user with feedback such that he can reposition the camera and acquire images starting from the location indicated by the indicator. Repositioning thus may allow the point clouds determined from subsequently acquired image pairs to be registered and added to the 3D model. For example, if the distance between the camera and surface is greater than a predefined distance, and/or if the acquired images are out of focus, and/or the user moved the camera too quickly across the surface, registration of the acquired images may fail, and the indication of such allows the user to maneuver the camera to allow relocking the image acquisition onto the 3D model construction. Thus, responsive to the registration failure, subsequent image acquisition may be relocked onto the 3D model by attempting to register the most recently acquired image pair at a location on the 3D model that is in proximity to a location corresponding to a previously added image pair, using the method of <FIG>, until the registration succeeds (Step <NUM>). For example, registration may be attempted with respect to previously registered and/or added images, such as going back to the tenth, twentieth, thirtieth, or fortieth most recently added image pairs, allowing to account for the camera's motion away from the most recently added image pair. Thus, the previously added image pair may be selected from a group consisting of: the ten most recently added image pairs, or the twenty most recently added image pairs, or the thirty most recently added image pairs, or the forty most recently added image pairs.

Since the registration algorithm may require more processing for each frame than the tracking algorithm, registering every acquired image pair may result in latency that causes a discrepancy between the restored position on the 3D model and the location of the most recently acquired image pair. To address this, the registration algorithm may skip over some of the acquired images, and use the most recently acquired image pair to relock the image acquisition onto the 3D model. For example, images that are very similar to the registered region, and provide little new information and thus have a small delta with the registered region, may be discarded.

Referring to <FIG>, a time line <NUM>, shown twice for the purpose of clarity, is shown comparing the rates for the image acquisition, registration, and relocking. The image acquisition rate is indicated by the combination of all the arrows, and may remain constant. During the 'attempt relocking' period <NUM>, unless the user has indicated otherwise, the processor attempts to register incoming images to a constructed portion of the 3D model. Registration may be attempted with respect to previously added images, such as the tenth, twentieth, thirtieth most recently added image, and the like. Alternatively, the processor attempts to register the incoming images in proximity to a location on the 3D model indicated by the user. Relocking may require substantial computational resources, and thus the rate for the attempted relocking, indicated by the thick arrows (<NUM>), may be slower than the image acquisition rate. After a failed attempt at relocking, the processor uses the most recently acquired image pair for the subsequent attempt, and discards any images acquired in the interim. In the example shown, registration is attempted for the first, fourth, and ninth acquired image pair, thus the relocking rate may not necessarily be uniform. After attempting to register the first image, corresponding to the first (bold) arrow <NUM> from the left, the acquired images, corresponding to the <NUM>nd and <NUM>rd arrows from the left (thin solid arrows <NUM>) are discarded, and the processor attempts to register the <NUM>th acquired image pair, corresponding to the second bold arrow <NUM> from the left. Similarly, due to latency in attempting to match the <NUM>th pair of images, the <NUM>th, <NUM>th, and <NUM>th images (thin solid arrows <NUM>) are discarded as well. Finally, the processor succeeds in registering the <NUM>th incoming image pair, as indicated by the 'X' under the <NUM>rd bold arrow from the left.

Once the relocking succeeds, the registered period begins, <NUM>. Registration typically requires less computational resources than relocking, since the processor matches the incoming images onto a smaller region of the 3D model, Thus the image acquisition rate may not exceed the registration rate, allowing each incoming image to be registered. However, to conserve computational resources, optionally only images with significant information are added to the 3D model. The newly registered images are compared to the 3D model, and the discrepancy between them is determined. If the discrepancy is less than a threshold value, the images are discarded, as indicated by the thin dashed arrows <NUM>. However, if the discrepancy is greater than the threshold value, and therefore the images contain a significant amount of new information, the images may be added to the 3D model, as indicated by the medium weight arrows <NUM>. Values for the discrepancy threshold may range from <NUM>% to <NUM>%, or from <NUM>% to <NUM>%, or approximately <NUM>%. The image discard rate may be non-uniform, for example, if the user lingers over a region for an extended period, many acquired images may be discarded, and if the user moves the camera over a region that was not previously scanned, few if any images may be discarded.

Reference is now made to <FIG>, which illustrate multiple scanning orientations and corresponding display orientations, in accordance with an embodiment. As the user manually maneuvers the camera about the 3D surface and changes the vertical and/or horizontal viewing angles, the perspective of the displayed 3D model may be adjusted accordingly ("reoriented") such that the currently registered region of the 3D model is prominently displayed, for example displayed at approximately the center of the screen and/or at approximately the center of a display area of the model. Optionally, to prevent the display from jumping or twitching in response to every change in the viewing angles, the display perspective may be adjusted conditional on the change in either of the vertical or horizontal viewing angles exceeding a threshold. Additionally or alternatively, smoothing may be applied to prevent such jumping or twitching, by relaxing sudden movements.

Referring to <FIG>, an initial head-on, front perspective display <NUM> is shown, corresponding to the scanning of a patient's face <NUM> using camera <NUM> from a head-on front viewing angle. The currently scanned region (nose), indicated by white circle 802a is indicated on the displayed 3D model <NUM> at the location 800a corresponding to the registration of the acquired images. In this case, the currently registered location 800a corresponding to the currently scanned region 802a is displayed at the center of the displayed 3D model <NUM>.

Referring to <FIG>, as the user maneuvers camera <NUM> around to the side of the patient's face <NUM>, the camera viewing angle transitions from a front view to a side, profile view <NUM>, to acquire images of the patient's cheek 802b. However, the display of the 3D model <NUM> has not been adjusted, and the perspective remains frontal. As a result, the location on the 3D model where the currently acquired images are registered is not displayed prominently at the center of the display, but rather over to the side, as indicated by the circle 800b corresponding to scanning region 802b.

<FIG> shows the change in the horizontal viewing angle <NUM> and the vertical viewing angle <NUM>, respectively, when moving from the front facing orientation of <FIG> to the side facing orientation of <FIG>. These angles may be measured by either calculating them using the registration locations of the 3D model, or by including a compass or gyroscope with camera <NUM>, or any other suitable means. When the change in the viewing angles exceeds a threshold value, the perspective of the display may be adjusted such that the indication on the 3D model of the currently scanned region is prominently displayed for the user's convenience.

Referring to <FIG>, the orientation of the displayed 3D model <NUM> is shown adjusted to the profile view instead of the frontal view <NUM> shown in <FIG>, such that indication 800b, corresponding to the currently registered location, is located at the center of the displayed model <NUM>.

Reference is now made to <FIG> which shows a flowchart of a method for adjusting the orientation of the display, in accordance with <FIG>. During scanning, the orientation of the displayed 3D model may be continually adjusted to ensure that the graphic indication of the currently registered location on the 3D model is included in the displayed rendition. The horizontal and vertical angles between the viewer viewport and the point cloud viewport may be computed (Step <NUM>). The computed horizontal and vertical angles may be compared to a threshold (Step <NUM>). If the horizontal angle exceeds the threshold, the horizontal angle corresponding to the perspective of the display is adjusted (Step <NUM>). If the vertical angle exceeds the threshold, the vertical angle corresponding to the perspective of the display is adjusted (Step <NUM>).

The re-locking and real-time tracking algorithms described above may be used to track a current scan position onto a previously constructed 3D model, and thus add newly acquired images to the at least partially-constructed 3D model. The graphic indicator may be used to allow the user to navigate the focus of the camera to regions of the surface that had already been included in the previously scanned 3D model, such that newly acquired images can be registered to a location on the retrieved previously constructed 3D model. Once registered, the currently acquired image pair may be added to the previously constructed 3D model at the registered location, as described above. The rendition of the incremental construction may differentiate between the newly added acquired image pairs and the previously constructed 3D model. For example, the newly scanned regions may be rendered in a different color.

Reference is now made to <FIG> which show an implementation of adding newly acquired images to a previously constructed 3D model. <FIG> shows a partially constructed 3D model <NUM> of an individual's face that was stored in a memory <NUM> of processor <NUM> and opened for viewing on display <NUM>. The user may indicate a region on the 3D model for relocking the image acquisition via interface <NUM> as described above. Once relocked, image registration may commence as indicated by graphic indicator <NUM> using the methods described above, to add newly acquired image pairs to the retrieved 3D model. The newly constructed portion of the 3D model <NUM>, corresponding to the newly acquired images, may be rendered in manner to differentiate it from the 3D model that was retrieved from memory <NUM>, such as by rendering the new portion <NUM> in a different color, shown for illustrative purposes only in grey. This allows the user to track the scanning and renewed construction of the 3D model relative to the previously constructed the 3D model.

Reference is now made to <FIG>, which shows a conceptual illustration of a system for tracking the scanning of a person's face in real-time, in accordance with an embodiment. A hand held scanning device <NUM>, such as any of cameras <NUM> or <NUM> above, scans a person's face <NUM>. The location of the scanning is indicated by the circle <NUM> surrounding the corner of the person's mouth. The images acquired from the scanning are used by a processor (not shown) configured with device <NUM> to construct a 3D model <NUM> of the person's face. The construction of the model is rendered in real-time on display <NUM>. While displaying the construction, the location of the currently scanned region is indicated on the rendition of the construction of the 3D model <NUM> as circular indicator <NUM>. Thus, the user of device <NUM> receives real-time feedback with respect to the progress of the scan and can observe and track the construction of the model while scanning, allowing the user to adjust the speed and location of the scanning, accordingly. As can be seen from <FIG>, the user has finished scanning the person's cheek, the 3D model <NUM> of which is constructed and rendered on display <NUM>. The user can now move the camera to the mouth and nose regions of the person's face to continue the 3D model construction, which will be rendered in real-time on display <NUM>, with indicator <NUM> tracking the scanned region on the rendered model.

Reference is now made to <FIG> which is a flowchart of a method for implementing the real-time tracking of the scanning of a 3D surface of Step <NUM> of <FIG> using graphical indicators, in accordance with an embodiment.

A size attribute of a graphical distance indicator may be set to indicate the distance between the stereoscopic camera and the surface, such as by setting the size attribute to correspond to the current camera-to-surface viewing distance for the most recently registered image pair; a shape, pattern, or other visual attribute of the indicator may be set to indicate a comparison between the current viewing distance and the optimal viewing distance; and a color attribute of the indicator may be set to indicate the success of the registration.

The distance between the camera and the surface, such as calculated in Step <NUM> of <FIG>, may be compared to an optimal camera focus distance (Step <NUM>). If the distance is not within the threshold of the optimal focal distance, the color, shape, and size attributes of the indicator may be set to indicate a registration failure (Step <NUM>). If the distance is within the threshold of the optimal focal distance, the color attribute of the indicator may be set to indicate a successful registration (Step <NUM>). If the distance is within the threshold but is greater than the optimal distance, the shape attribute may be set to indicate a 'further than optimal distance' indication (Step <NUM>). If the distance is within the threshold but is smaller than the optimal distance, the shape attribute may be set to indicate a 'nearer than optimal distance' indication (Step <NUM>). The set indicator may be overlaid on the rendered 3D model at a location corresponding to the most recently successfully registered image pair (Step <NUM>). Additionally, or alternatively, any of the indicators may comprise sounding an alert, and/or flashing a light.

The present invention is a system according to claim <NUM>. Other examples not being part of the claimed invention are a method and/or a computer program product.

The computer readable storage medium can be a non-transitory, tangible device that can retain and store instructions for use by an instruction execution device. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, or any suitable combination of the foregoing.

Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

Aspects of the present invention may be described herein with reference to flowchart illustrations and/or diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various examples of the present invention.

Claim 1:
A system, comprising:
a stereoscopic camera (<NUM>) configured to acquire multiple pairs of images of a surface (<NUM>); and
a processor (<NUM>) configured to:
sequentially acquire (<NUM>) multiple pairs of stereoscopic images of a surface from the stereoscopic camera;
incrementally construct (<NUM>) a 3D model (<NUM>, <NUM>, <NUM>) of the surface from the image pairs concurrently with the sequential image acquisition, wherein incrementally constructing comprises, for each currently acquired image pair:
registering the currently acquired image pair to a location on the 3D model (<NUM>);
binning the currently acquired image pair by combining pixel values in a binning operation;
calculating (<NUM>) a low resolution distortion of the binned images;
calculating (<NUM>) an average distance between the camera and the 3D surface, using the calculated low resolution distortion;
selecting (<NUM>) a distortion correction table corresponding to the average distance;
applying (<NUM>) the selected distortion correction table to correct a distortion of the currently acquired image pair;
determining (<NUM>) a point cloud of the corrected, currently acquired image pair; and
adding the currently acquired image pair to the 3D model, by adding (<NUM>) the point cloud to the 3D model at the registered location.