Patent Description:
Three-dimensional scene reconstruction with multiple cameras can benefit from the use of one or more depth sensors as part of a color camera array. Among the different depth sensing principles, indirect Time of Flight (iToF) has become popular.

A hybrid depth sensing device typically consists of a color camera, an infrared projection mechanism and a depth sensor camera built into a single device. Microsoft's Azure Kinect (TradeMark) is an example of such a device. Intrinsic parameters such as focal length, principal point, and lens distortion are typically provided by the manufacturer for both the color camera and the depth sensor. The extrinsic parameters that describe the relative position and rotation between the color and depth cameras of a single device are also provided. The devices are typically designed such that they minimize the baseline between the color sensor and the depth sensor inside the device in order to minimize parallax errors. This is important since, conventionally, the depth map is represented in the coordinates of the image sensor or vice versa.

<FIG> shows an exemplary hybrid camera array. The hybrid camera array comprises three hybrid devices <NUM>. Each hybrid device <NUM> comprises a color sensor <NUM> and a depth sensor <NUM>. The distance between the color sensor <NUM> and the depth sensor <NUM> of a hybrid device <NUM> (i.e. the baseline between these sensors) is shown by arrow <NUM>. The distance between the color sensors <NUM> of two different devices (i.e. the baseline between color sensors) is shown by arrow <NUM>.

When using a single hybrid device <NUM>, the factory calibration will suffice for most applications. However, for applications where a scene needs to be captured from different directions, it is necessary to know the extrinsic parameters (rotation and translation) between multiple different hybrid devices <NUM>. One approach could be to use a calibration pattern. However, this requires a calibration pattern which is not practical in most cases.

Another approach could be to use the depth maps for calibration. However, due to the lack of spatial detail and the presence of noise in the depth maps, such a geometry-based approach is not likely to be very precise.

Existing structure from motion algorithms that use visual feature point detection can provide self-calibration without the need for a calibration pattern using only the visual images from the color sensors. Many years of research has led to these structure from motion tools becoming very accurate and robust. However, structure from motion algorithms alone cannot solve the calibration problem of a hybrid camera array.

A first problem is that structure from motion relies on feature matching between views and hence the views need to be of the same modality (e.g., grey scale or color images). An input of images and depth maps into the structure from motion algorithm will not work since the features cannot be accurately matched across different modalities.

A second problem is that structure from motion provides calibration results (i.e. camera rotation and translation in a common coordinate system) only up to an ambiguous or unknown scale factor, with the same arbitrary dimensionless unit for the camera positions and the point cloud point positions. For many applications this is not a problem since the scale parameter is arbitrary and constructing a 3D geometric model such as a mesh and texturing that model does not require knowledge of scale. However, in a hybrid camera array that includes depth sensors providing depth maps in metric units (e.g., mm) this is a problem since we cannot use, for example, the depth at a given pixel in each depth camera and warp it to one of the cameras (color or depth) of another device. For this we would need to know the camera/device translations in metric units.

Thus, there is a need for a self calibration method which is accurate when used with hybrid camera arrays.

<NPL>, discloses combining bundle adjustment and iterative closest point algorithms to estimate pose parameters of cameras in a multi-view system.

<NPL>, discloses calibrating extrinsic parameters among multiple cameras and a LIDAR sensor using object pose estimation.

According to examples in accordance with an aspect of the invention, there is provided a method for calibrating a camera pose in a hybrid camera array, the hybrid camera array comprising two or more color sensors and one or more depth sensors, the method comprising:.

The scaling may be used to calibrate the up to scale camera poses. In other words, the scaling may be used to obtain camera poses which can provide the real distances between the color sensors and/or depth sensors. The camera poses may be used subsequently for multi-view image/video coding and/or for novel view synthesis, for example.

The up to scale camera poses provide a rotation between the color sensors; and further provide a translation between the color sensors, but only up to an unknown scale factor. The up to scale point cloud provides a measure of depth from the set of images, but only up to an unknown scale factor. The depth maps provide a metric measure of depth - that is, depth values calibrated to real world units (e.g. mm). As such, it has been realized that the up to scale point cloud can be compared to the depth map to thereby estimate the scaling of the up to scale point cloud (and therefore of the up to scale camera poses).

Generating the up to scale camera poses and the up to scale point cloud may comprise inputting the set of images into a structure from motion algorithm, wherein the structure from motion algorithm is configured to output the up to scale camera poses for the color sensors and, optionally, the up to scale point cloud.

Structure from motion algorithms, or SfM algorithms, are commonly used to extract three dimensional structures from two dimensional images. For example, the movement of a color sensor can be deduced up to scale based on the images taken during the movement. Similarly, images of a scene from various color sensors at different positions can also be used instead of using the movement of a single color sensor.

The structure from motion algorithm may be configured to output both the up to scale camera poses and the up to scale point cloud. However, if the structure from motion algorithm does not output the up to scale point cloud or the output up to scale point cloud is not desirable, a new (or additional) up to scale point cloud can be obtained from the up to scale camera poses using, for example, feature detection and triangulation.

Estimating the scaling of the up to scale camera poses and the up to scale point cloud comprises obtaining a plurality of proposed scales and, for each proposed scale: calculating proposed camera poses and a proposed point cloud by applying the proposed scale to the up to scale camera poses and up to scale point cloud respectively, comparing the proposed point cloud to the one or more depth maps and determining one or more error metrics based on the comparison. The scaling is estimated based on the proposed scale with the lowest one or more error metrics.

This could be seen as a "for loop" implemented for various proposed scales where at least one error metric is determined for each proposed scale.

It will be appreciated that various methods exist for obtaining the plurality of proposed scales. In a first example, the proposed scales may be randomly assigned. In a second, preferred, example, an informed guess may be made for one or more of the proposed scales and the other values may be derived from nearby the informed guesses (e.g. ± <NUM>%). Similarly, a "next" proposed scale in the loop used could be based on the error metrics of the previous proposed scales.

In a first comparison method, comparing the proposed point cloud to the one or more depth maps may comprise projecting the proposed point cloud to the views of the one or more depth sensors using the proposed camera poses, evaluating a plurality of depth-based errors between depth values in the one or more depth maps and the corresponding depth values resulting from projecting the proposed point cloud, wherein determining one of the error metrics is based on the plurality of depth-based errors.

The scaling can thus be found by minimizing the errors between the predicted depth associated with the projected points and the measured depth map. This has been found to greatly increase the accuracy and robustness of the estimated scaling.

In a second comparison method, comparing the proposed point cloud may comprise projecting the proposed point cloud to the views of the two or more color sensors using the proposed camera poses thereby to generate projected points, identifying depth map values of the one or more depth maps corresponding to feature points of the proposed point cloud, projecting the depth map values to the views of the two or more color sensors using the proposed camera poses and evaluating a plurality of pixel-based errors, in an imaging plane of each of the two or more color sensors, between the pixel coordinates resulting from projecting the depth map values and the pixel coordinates resulting from projecting the proposed point cloud, wherein determining one of the error metrics is based on the plurality of pixel-based errors.

Using pixel-based errors instead of depth-based errors is typically more intuitive to users. Additionally, the pixel-based based errors somewhat mimic the process of synthesizing novel views in multi-view imaging. This may provide improved results in the synthesis of novel views as the errors have already been minimized between existing views of the color sensors.

Estimating the scaling of the up to scale camera poses and up to scale point cloud may further comprise, for each proposed scale: determining whether one or one or more projected points in the projected point cloud are occluded in the view of any one of the depths sensors and/or the color sensors and based on a particular projected point being occluded, weighting the error corresponding to the particular projected point lower in the one or more error metrics.

Point clouds may include feature points of the back, or the side, of objects which are not visible from the view of the depth sensors. As such, the weighting of the occluded points is preferably lowered in the sum of errors.

For the first comparison method, determining whether one or more projected points in the projected point cloud are occluded may comprise determining whether the depth-based error between the depth of one or more of the projected points and the depth of the depth map exceeds a depth-based occlusion threshold.

For the second comparison method, determining whether one or more projected points in the projected point cloud are occluded may comprise determining whether the pixel-based error between the pixel coordinates of the projected points and the pixel coordinates of the depth map points exceeds a pixel-based occlusion threshold.

Determining whether one or more projected points in the projected point cloud are occluded may comprise identifying projected points within a local neighborhood of one or more central projected points and for each local neighborhood, determining whether the depth-based error and/or the pixel-based error exceed a neighborhood threshold, wherein the neighborhood threshold is optionally dependent on the depth of the identified projected points in the local neighborhood.

Determining whether one or more projected points in the projected point cloud are occluded may comprise, for each projected point: identifying a pixel color for each view of the one or more depth sensors and/or the two or more color sensors based on the set of images and comparing the pixel colors from each view.

Determining whether the projected points are occluded may comprise determining whether a difference between the pixel colors exceeds a pixel color threshold.

The method may further comprise generating a second point cloud using the one or more depth maps and, for each proposed scale: projecting the second point cloud to the views of the one or more depth sensors using the proposed camera poses and evaluating a second plurality of depth-based errors between depth values in the one or more depth maps and the corresponding depth values resulting from the projection of the second point cloud, wherein determining one of the error metrics is based on the second plurality of depth-based errors.

The different error metrics may be weighted based on one or more of: the number of points in the proposed point cloud, the number of points in the second proposed point cloud and a relative importance between the proposed point cloud and the second proposed point cloud.

The invention also provides a computer program product comprising computer program code which, when executed on a computing device having a processing system, cause the processing system to perform all of the steps of the afore-mentioned method.

The invention also provides a system for calibrating a camera pose in a hybrid camera array, the system comprising:.

It will be appreciated that the processor of the system may be further configured to perform any of the afore-mentioned method steps.

The invention provides a method for calibrating a camera pose in a hybrid camera array comprising two or more color sensors and one or more depth sensors. The method comprises obtaining a depth map for each of the depth sensors, obtaining a set of images from the color sensors and generating up to scale camera poses for the color sensors and an up to scale point cloud using the set of images. A scaling of the up to scale camera poses and up to scale point cloud is then estimated using the one or more depth maps.

Existing structure from motion software can output a point cloud as well as the expected camera poses (i.e., orientation and translation). Known packages with structure from motion software include Alice Vision and Colmap. This point cloud is typically used internally to solve the structure from motion problem in a process called bundle adjustment. The points in the point cloud correspond well with detectable features in the images.

Although this point cloud is also only known up to scale, it provides an unexpected alternative to the depth warping and image difference calculation approach. The points in this point cloud consists both of well matchable and distinctive features with sufficient local image texture variation. Moreover, since the points in the point cloud coincide with matching features over the multiple views, few occlusion features will present since those exist due to a combination of textures at different depth levels.

Based on this realization, it is proposed to solve the hybrid camera self-calibration problem by first running a structure from motion algorithm on the color images and then, as a second step, use the point cloud output by the structure from motion algorithm to determine the unknown scale parameter by comparing it with information from the depth map(s).

For a given scale, the point cloud can be first converted to a metric point cloud. The metric point cloud is then warped to all views that contain a depth map. The observed depth value, from the depth maps, is then compared to the depth value of the projected point in the view coordinates. The error metric is hence a geometric metric (instead of an image-based metric, for example).

<FIG> shows a self-calibration process for a hybrid camera array. A sequence of color images <NUM> is obtained from color sensors and depth maps <NUM> are obtained from depth sensors. A structure from motion algorithm <NUM> is used to estimate the scaled camera poses <NUM> of the color sensor, including rotation matrices [R<NUM>,. , RN] and up to scale translation vectors [st<NUM>,. The camera pose of a sensor can be used to warp images to other views. However, in order to warp images to a view corresponding to a different sensor, a scale-resolved, metric, translation vector [t<NUM>,. , tN] is needed.

The structure from motion algorithm <NUM> also outputs an up to scale point cloud [sx<NUM>,. , sxM] <NUM>. This data can be passed to a scale estimation algorithm <NUM> that uses the metric depth map <NUM> to determine the scaling <NUM> of the up to scale camera poses and up to scale point cloud via a scale parameter s.

An exemplary scale estimation algorithm <NUM> has the following steps:.

The minimum of E(s) may be replaced with a threshold minimum Emin. In this case, if E(s) < Emin, the threshold minimum can be updated to the lower value.

<FIG> illustrate the scale estimation process using a single depth sensor <NUM> and first and second scale parameters respectively. Projected points <NUM> correspond to the cloud points projected to the depth sensor <NUM> using the corresponding scale parameter. Metric depth map <NUM> is the depth map observed by the depth sensor <NUM>.

In <FIG>, the first scale parameter selected is too small, thereby resulting in a large error <NUM> between the predicted depth of the projected points <NUM> and the observed depth at corresponding positions on the depth map <NUM>. The corresponding positions on the depth map are the points on the depth map intersected by a vector formed by a projected point <NUM> and the origin of the depth sensor <NUM>.

In <FIG>, the second scale parameters selected, corresponding to a larger scaling, shows a smaller error <NUM> and thus provides a more accurate estimation of the metric translation vector.

<FIG> illustrate the scale estimation process using two depth sensors 506a and 506b. In this case, a plurality of first errors 508a between the projected points <NUM> and the depth map <NUM> from the first depth sensor 506a are calculated and a plurality of second errors 508b between the projected points <NUM> and the depth map <NUM> from the second depth sensor 506a are calculated. The plurality of first errors 508a and the plurality of second errors 508b are summed to determine the total error corresponding to each proposed scale s. Thus, the second depth sensor 506b now also contributes to the total error via its own observed depth map.

In <FIG>, the scale parameter selected is too small, thereby resulting in a large errors 508a and 508b. In <FIG>, the scale parameters selected shows smaller errors (only error <NUM> is visible) and thus provides a more accurate estimation of the metric translation vector.

<FIG> illustrates the scale estimation process when dealing with occluded points <NUM>. Due to a point cloud being sparse, by definition, projected points <NUM> from the back of an object will falsely contribute to the error calculation. This can be seen in <FIG> where the projected point <NUM> corresponds to the back of an object represented by depth map <NUM>. This may lead to a false error <NUM> between the point of the depth map visible to the depth sensor <NUM> and the projected point <NUM> contributing to the total error metric.

A solution may be to assume that objects have a minimum dimension and exclude errors that exceed a given occlusion threshold t (e.g. t = <NUM>). The corresponding error metric could thus be: <MAT> where: <MAT>.

Other approached for handling occlusion could also be used to suppress errors such as using local occlusion reasoning between points in the projected view.

<FIG> illustrates occlusion handling in the scale estimation process using local neighborhoods <NUM>. A local neighborhood <NUM> is provided around a given projected point <NUM> chosen from the plurality of projected points <NUM>. The local neighborhood <NUM> of a given projected point <NUM> can be defined as all other points that lie closer (measured in image space) than a given distance expressed in pixels (e.g., <NUM> pixels).

If the projected point <NUM> has a depth that is larger than a neighborhood threshold, compared to all neighboring points, then it can be concluded that the projected point <NUM> coincides with a point on the object which is not visible from the depth sensor <NUM> (e.g. the back of the object). Consequently, the error between the projected point <NUM> and the observed depth map <NUM> will not accumulate in the error sum.

Alternatively or additionally, pixel colors can be compared between views to determine whether, for a given view, a projected point is occluded or not.

<FIG> illustrates occlusion handling in the scale estimation process using pixel color. If, for a given projected point in a given view, the observed color differs significantly from all color values observed in the other views (for the same projected point), the projected point is likely occluded and hence its contribution to the error sum can be reduced or fully removed. For, example, a weighting may be added to the error sum based on a pixel color comparison.

In <FIG>, two projected points 902a and 902b are shown. Projected point 902a can be seen by all three depth sensors 906a, 906b and 906c and thus the corresponding pixel color (e.g. obtained by warping the color images from the color sensor to the depth sensors) should be similar from all three views. However, projected point 902b can be seen by depth sensors 906a and 906b but cannot be seen by depth sensor 906c. Thus, depth sensor 906c would "see" the pixel color of projected point 902b as being the color at point <NUM>. As such, if the colors of points 902b and <NUM> do not match, the error of point 902b from depth sensor 906c could be ignored in the total error calculation.

Visual feature points (i.e. the points in the point cloud) output by structure from motion algorithm typically correspond with image regions that show spatial texture variation (e.g., corners). Image regions without much texture or texture variation will therefore not generate feature points and thus not be used in the scale estimation process.

To solve this problem, the observed depth maps could be used as a source for additional point cloud data. The depth maps for all views can be converted to a common world space coordinate system using the proposed scale and the camera poses. This gives a depth map-based point cloud with the size of the number of views times the number of depth map pixels per view. For a given proposed scale, two types of point clouds are now available: a scaled version of the point cloud provided by the structure from motion algorithm and a depth map derived point cloud.

Since both the origin and the number of points in each type of point cloud differ significantly, it may be advantageous to normalize each type by the number of points and weigh their contribution separately: <MAT> where α (between <NUM> and <NUM>) weighs the relative importance of the point cloud provided by structure from motion. The particular value of α may be chosen by the skilled person (e.g. via experimentation and/or trial and error).

Returning to <FIG>, the baseline <NUM> between the color sensor <NUM> and the depth sensor <NUM> in a hybrid device <NUM> may be known (e.g. from factory).

The baseline length <NUM> between the color sensors can be estimated. In practice, an average baseline length <NUM> between all color sensors can be estimated. The baseline length <NUM> can then be used to calculate a rough estimate of the scale parameter. A range of proposed scales, around the rough estimate of the scale parameter, can then be used in the scale estimation process as described above.

In an example, one can calculate a rough estimate of the scale parameter by dividing the estimated baseline length <NUM> by an average of the shortest distance between the camera poses. If the color sensors are somewhat uniformly spaced out in 3D space (i.e. the distance between the color sensors does not vary significantly), the rough estimate of the scale parameter may be within <NUM>% of the real scale parameter. Thus, under the assumption of uniformly spaced out cameras, the range of proposed scales may be within <NUM>% of the rough estimate. However, it will be appreciated that, in practice, such assumptions may not always be correct. Thus, it has been found that a sensible range of proposed scales may be within <NUM>% of the rough estimate. Of course, the exact range may depend on the positioning of the color sensors, the processing resources available and the preferences of the skilled person.

The range of proposed scales comprises a plurality of proposed scales.

In an example, the rough estimate may be a guess (e.g. random or based on prior scales). However, this may require more iterations/a larger range of proposed scales and, in some cases, may diverge towards an erroneous scale.

As will be appreciated, the aforementioned error metric is based on depth-based error contributions. In other words, the error metric is based on differences in depth values. However, many users may be more familiar with pixel-based errors (e.g. distances in pixels). As such, a second solution is proposed where pixel-based errors are used to determine the error metric. A sum of residual pixel-based errors (after the optimal scale has been estimated) may be useful as a quality metric, regardless of whether a depth-based or pixel-based error metric was used to guide the selection of the optimal scale.

<FIG> illustrates the scale estimation process using pixel-based errors <NUM>. Feature point <NUM> is a feature point of a proposed scale point cloud. Point <NUM> is a point inferred from the depth map, by identifying the depth value <NUM> corresponding to the line-of-sight to feature point <NUM>. This will be referred to as a depth map point. It has been "un-projected" into 3D space from the imaging plane 1010a of a first color sensor 1006a using the known depth value <NUM> from the depth map. (Here, "un-projection" refers to a process of inferring a <NUM>-D point from a depth value in a <NUM>-D depth map. ) As can be seen, the depth map point <NUM> corresponds to the feature point <NUM> from the view of the first color sensor 1006a.

Using the proposed camera poses of the color sensors 1006a, 1006b and 1006c, the depth map point <NUM> and the feature point <NUM> are re-projected to the imaging planes 1010b and 1010c corresponding to the second color sensor 1006b and third color sensor 1006c respectively.

Pixel-based errors <NUM> can then be found between the re-projections of the feature point <NUM> and the depth map point <NUM> for both imaging planes 1010b and 1010c. Similarly to the depth-based approach, the pixel-based errors can be used to determine an error metric.

Thus, instead of comparing the predicted depth value after projection with the observed depth map value, this alternative pixel-based error approach projects the depth map point <NUM> to various views where it then compares its pixel position with the direct projection of the corresponding point cloud point. This approach results in a total number of error contributions to the error metric equal to the number of feature points in the point cloud times the number of cameras minus one.

Since this error metric involves re-projecting points between views, it resembles the view synthesis process that is typically used during rendering of novel views in multi-view imaging. As such, minimizing a pixel-based error metric may provide further accurate renders of novel views as this should translate to the minimization in visible errors in the novel views.

Of course, one could also use both approaches to obtain different error metrics. These error metrics would comprise different dimensions (i.e. pixel distances and depth differences) and thus combining them may not be desirable unless they are appropriately weighted.

Similarly to the depth-based scale estimation, a second point cloud could also be calculated from the one or more depth maps and used to obtain more depth information of the scene. The use of the second point cloud could be used to determine a second depth-based error metric. It should be noted that the pixel-based error metric cannot be used as-is with a second point cloud that is calculated solely from depth map data. For the points in such a point cloud, there is no information about the correspondence of points between views. Therefore, it is not possible to perform a point-wise comparison. In contrast, the correspondence information is inherent in the point cloud generated by the structure-from-motion algorithm, since this a single "true" <NUM>-D point cloud in <NUM>-D coordinates, which is subsequently projected into the <NUM>-D imaging plane of each sensor.

It will be appreciated that more than one error metric can be determined for a hybrid sensor array set up. The number of error metrics and type of error metric used may depend on the available processing resources and/or user preference.

It will be appreciated that any of the aforementioned occlusion handling approaches can also be used for the pixel-based scale estimation process with minimal changes. For example, the occlusion threshold may be based on distances in pixels instead of distances in metric measurements.

Another solution to the problem of self-calibration which has been considered is to first run a structure from motion algorithm on the color images and then use the depth map(s) to find the correct scale parameter. Theoretically, this could be achieved by warping a color image from one device to another based on the depth map and then taking an image difference measure (e.g., mean squared error). Minimizing this difference over all possible warps of one camera to the other cameras could then result in the best fit scale parameter.

However, this alternative approach may be more sensitive to noise in the depth map - especially for larger baselines. Furthermore, in some instances, the use of an image-based difference measure may cause a bias in the scale estimate, caused by varying surface reflectance resulting in color changes in the image. Additionally, this solution may be relatively more computationally intensive, since all pixels in all images need to be warped to all other images at each iteration in the search for the optimal scale. The alternative solution is likely to be most useful in cases where the baseline is relatively small and the depth maps are expected to have a low level of noise.

As previously discussed, a structure from motion algorithm can be used to obtain a point cloud. In some cases, the structure from motion algorithm will output the point cloud. However, in some cases, it may be preferable to determine a new up to scale point cloud for the scale estimation (optionally using the up to scale camera poses output by the structure from motion algorithm). For example, feature detection and matching can be applied to the set of images and triangulation can used, based on the on the up to scale camera poses, to obtain a new up to scale point cloud. The new up to scale point cloud could be used alone to estimate the scaling, or else it could be used to augment the point cloud from the structure-from-motion algorithm.

The skilled person would be readily capable of developing a processor for carrying out any herein described method. Thus, each step of a flow chart may represent a different action performed by a processor, and may be performed by a respective module of the processor.

As discussed above, the system makes use of processor to perform the data processing. The processor can be implemented in numerous ways, with software and/or hardware, to perform the various functions required. The processor typically employs one or more microprocessors that may be programmed using software (e.g., microcode) to perform the required functions. The processor may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

Functions implemented by a processor may be implemented by a single processor or by multiple separate processing units which may together be considered to constitute a "processor". Such processing units may in some cases be remote from each other and communicate with each other in a wired or wireless manner.

Claim 1:
A method for calibrating a camera pose in a hybrid camera array, the hybrid camera array comprising two or more color sensors and one or more depth sensors, the method comprising:
obtaining a depth map (<NUM>) from each of the one or more depth sensors and a set of images (<NUM>) from the color sensors;
generating up to scale camera poses (<NUM>) for the color sensors and an up to scale point cloud (<NUM>) using the set of images; and
estimating a scaling (<NUM>) of the up to scale camera poses and up to scale point cloud using the one or more depth maps, characterised in that estimating the scaling of the up to scale camera poses and the up to scale point cloud comprises:
obtaining a plurality of proposed scales;
for each proposed scale:
calculating proposed camera poses and a proposed point cloud by applying the proposed scale to the up to scale camera poses and up to scale point cloud respectively;
comparing the proposed point cloud to the one or more depth maps;
determining one or more error metrics based on the comparison; and
estimating the scaling based on the proposed scale with the lowest one or more error metrics.