Patent Description:
Recovering 3D position and/or 3D pose of objects from images has been a longstanding problem in computer vision. Techniques for 3D positioning and pose determination have a variety of applications such as 3D vision inspection, product assembly, goods inspection, human-computer interaction, video surveillance, sports broadcasting, industry robot control, navigation, etc..

One common solution is to use multiple cameras with overlapping fields of view and process individual video streams from the cameras for detection of keypoints, identify correspondence between keypoints in different views, and calculate 3D positions based on the correspondence between keypoints, and optionally temporal information. For such calculation, calibration data representing the position and orientation of the cameras needs to be known with reasonable accuracy. The determination of the calibration data is known as "extrinsic calibration" in the art. The extrinsic calibration may be performed during installation by manual measurements. This approach is time consuming, error prone, and sensitive to subsequent displacement of the cameras. In an alternative, a dedicated equipment of precise and known appearance is presented to the cameras, and the system is operated to calculate the calibration data based on detected images. This approach is also sensitive to subsequent displacement of the cameras, and the dedicated equipment must be used for re-calibration.

It has also been proposed to perform the extrinsic calibration by observing human individuals by the multiple cameras, and specifically to calculate the calibration data based on keypoints detected in the images for the human individuals. This approach allows the extrinsic calibration to be automated and re-calibration of the system to be performed by simply observing people moving around in front of the cameras. One example of this approach is found in the article "<NPL>. Another example is found in the article "<NPL>. Both of these articles propose to perform the extrinsic calibration by minimizing an error function that comprises a first function representing the reprojection error and a second function configured to enforce constant limb lengths over time.

One general challenge of extrinsic calibration is that the detected keypoints in images may suffer from low precision, which is propagated into the calculated calibration data.

It is an objective to at least partly overcome one or more limitations of the prior art.

Another objective is to improve extrinsic calibration of a multi-camera system by use of one or more detected poses as calibration pattern.

Yet another objective is to provide for extrinsic calibration of a multi-camera system with high accuracy and/or robustness.

One or more of these objectives, as well as further objectives that may appear from the description below, are at least partly achieved by a computer-implemented method of calibrating cameras according to the independent claim <NUM>, embodiments thereof being defined by the dependent claims.

Still other objectives, as well as features, aspects and technical effects will appear from the following detailed description, from the attached claims as well as from the drawings.

Embodiments will now be described in more detail with reference to the accompanying schematic drawings.

Embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments are shown. Indeed, the subject of the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure may satisfy applicable legal requirements.

Also, it will be understood that, where possible, any of the advantages, features, functions, devices, and/or operational aspects of any of the embodiments described and/or contemplated herein may be included in any of the other embodiments described and/or contemplated herein, and/or vice versa. In addition, where possible, any terms expressed in the singular form herein are meant to also include the plural form and/or vice versa, unless explicitly stated otherwise. As used herein, "at least one" shall mean "one or more" and these phrases are intended to be interchangeable. Accordingly, the terms "a" and/or "an" shall mean "at least one" or "one or more", even though the phrase "one or more" or "at least one" is also used herein. As used herein, except where the context requires otherwise owing to express language or necessary implication, the word "comprise" or variations such as "comprises" or "comprising" is used in an inclusive sense, that is, to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments. The term "compute", and derivatives thereof, is used in its conventional meaning and may be seen to involve performing a calculation involving one or more mathematical operations to produce a result, for example by use of a computer.

As used herein, the terms "multiple", "plural" and "plurality" are intended to imply provision of two or more items, whereas the term a "set" of items is intended to imply a provision of one or more items.

It will furthermore be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing the scope of the present disclosure.

Like reference signs refer to like elements throughout.

Before describing embodiments in more detail, a few definitions will be given.

As used herein, "object" denotes any animate object that may be represented by a set of keypoints in an image. Further, "live body" denotes the body of an animate or living object such as a human individual or an animal.

As used herein, "scene" denotes a three-dimensional (3D) space that is collectively monitored by two or more cameras, which have at least partly overlapping fields of views.

As used herein, "camera" is an imaging device configured to generate images of a scene by detection of electromagnetic waves and/or mechanical waves that are generated and/or reflected by the scene and objects therein. The camera may be responsive to electromagnetic waves in any wavelength range, including but not limited to ultraviolet, visible or infrared radiation, or any part or combination thereof. The camera may be configured to produce still images or a digital video stream, i.e. a coherent time sequence of images. The respective image is a two-dimensional (2D) representation of the scene, or part thereof, as seen by the camera.

As used herein, "view" or "camera view" refers to the part of the scene that is included within the field of view of a camera. The view may be embodied as an image, for example a digital image.

As used herein, "field of view" has its conventional meaning and denotes the extent of the scene that is observed by the respective camera at any given moment and may be defined as a solid angle through which the camera is sensitive to the electromagnetic/mechanical waves.

As used herein, "keypoint" has its conventional meaning in the field of computer vision and is also known as an interest point. A keypoint is a spatial location or point in an image that defines what is interesting or what stand out in the image and may be defined to be invariant to image rotation, shrinkage, translation, distortion, etc. In the present disclosure, the keypoint is also denoted "feature point" and has a predefined placement on a detected object. In the example of a live body, keypoints may identify one or more joints and/or extremities. Keypoints may be detected by use of any existing feature detection algorithm(s), for example image processing techniques that are operable to detect one or more of edges, corners, blobs, ridges, etc. in digital images. Non-limiting examples of feature detection algorithms comprise SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Feature), FAST (Features from Accelerated Segment Test), SUSAN (Smallest Univalue Segment Assimilating Nucleus), Harris affine region detector, and ORB (Oriented FAST and Rotated BRIEF). Further information about conventional keypoint detectors is found in the article "<NPL>). Further examples of feature detection algorithms are found in the articles "<NPL>, and "<NPL>. Correspondingly, objects may be detected in images by use of any existing object detection algorithm(s). Non-limiting examples include various machine learning-based approaches or deep learning-based approaches, such as Viola-Jones object detection framework, SIFT, HOG (Histogram of Oriented Gradients), Region Proposals (RCNN, Fast-RCNN, Faster-RCNN), SSD (Single Shot MultiBox Detector), You Only Look Once (YOLO, YOLO9000, YOLOv3), and RefineDet (Single-Shot Refinement Neural Network for Object Detection).

As used herein, "scene point" is any point in a scene that is included and detectable as a keypoint in two or more camera views. The scene point has a three-dimensional (3D) location in a common coordinate system.

As used herein, "common coordinate system" refers to a reference system that uniquely defines the 3D location of a scene point in relation to an origin.

As used herein, "calibration data" refers to one or more parameters that define the positions and orientations of the cameras monitoring the scene. The calibration data may be relative or absolute. In some embodiments, the calibration data comprises a so-called camera projection matrix ("camera matrix") for the respective camera. Such a camera matrix may or may not comprise inner ("intrinsic") parameters of the respective camera. In some embodiments, the camera matrix for a camera i is defined as Pi = Ki[Ri Ti], with Ki being the intrinsic matrix of the camera, Ri being a rotation matrix, and Ti being a translation vector. In some embodiments, the intrinsic matrix is known for the respective camera.

As used herein, "pose" refers to a collection of scene points that define the posture of an object in the scene.

Embodiments are related to extrinsic calibration of a multi-camera system based on a calibration pattern formed by one or more poses of one or more objects detected by the cameras in the system. The following description will be given for objects in the form of human individuals, but the disclosure may be applied to any type of animate object.

<FIG> shows an example arrangement of an example multi-camera system <NUM>, denoted "monitoring system" in the following. The monitoring system <NUM> is arranged to monitor a scene <NUM>. In the illustrated example, three individuals <NUM> are in the scene <NUM>. The system <NUM> comprises a plurality of cameras <NUM> which are oriented with their respective field of view <NUM> towards the scene <NUM>. The scene <NUM> is associated with a fixed 3D coordinate system <NUM>, which is denoted "scene coordinate system" or "common coordinate system" herein. The cameras <NUM> may be fixed or moveable. When the system <NUM> has been calibrated, the relative positions and orientations of the cameras <NUM> are known for each image taken. The cameras <NUM> may be synchronized to capture a respective image at approximately the same time, or at least with a maximum time difference which depends on the expected maximum speed of movement of the objects <NUM>. In one example, a maximum time difference of <NUM>-<NUM> seconds may provide sufficient accuracy for normal human motion. The cameras <NUM> are oriented with overlapping fields of view <NUM> and thus produce at least partly overlapping images of the scene <NUM>.

The images captured by the respective camera <NUM> are received by a detection device <NUM>, which is configured to determine one or more keypoints of one or more objects in the respective image. The detection device <NUM> may implement any conventional object detection technique for identifying objects of a generic or specific type in the respective image and may implement any conventional feature detection technique for identifying one or more keypoints of the respective object, for example as described hereinabove. The detection device <NUM> may also pre-process the incoming images, for example for noise reduction, contrast enhancement, etc. In an alternative configuration, the monitoring system <NUM> comprises a plurality of detection devices <NUM>, for example one for each camera <NUM>, where the detection devices <NUM> may be co-located or integrated with the cameras <NUM>.

The detection device <NUM> produces object detection data (ODD), which identifies one or more keypoints of one or more objects detected in the respective image. An example of keypoints that may be detected for a human individual are indicated by open and filled dots in <FIG>. Thus, the respective keypoint may correspond to a joint or an extremity of the human individual. Any number of keypoints may be detected depending on implementation. For example, the ODD may represent each image (view) by a respective view identifier (V1-V3 in <FIG>) and a keypoint position for each keypoint detected for the respective object (K1-K3 in <FIG>). The detected position of the respective keypoint may be given by coordinates in a local and fixed coordinate system <NUM> of the respective view, as exemplified in <FIG>. The detection device <NUM> may also include a confidence score for each object and/or keypoint in the ODD, the confidence score representing a level of certainty for the respective detection of object/keypoint.

The system <NUM> further comprises a positioning device <NUM>, which is configured to operate on the ODD to compute, and possibly track over time, one or more 3D positions of the individual(s) <NUM> in the scene <NUM>. The 3D positions may be given in the scene coordinate system <NUM> and are denoted "scene points" in the following. The scene points, when representing an object (for example, a human individual), defines a "pose" of the object. The positioning device <NUM> may compute the scene points by conventional triangulation. In the example of <FIG>, scene points 10A-10C are computed by identifying intersections of rays that extend from camera centers through the respective keypoint K1-K3 in the views V1-V3. The triangulation may be implemented to also to account for observation errors in the keypoint locations. There are numerous available triangulation methods that may be implemented by the positioning device <NUM>, including but not limited to so-called linear methods, optimal methods or branch-and-bound search methods. Generally, the triangulation operates on calibration data that represents the positions and orientations of the cameras <NUM>.

<FIG> illustrates a common technique of quantifying errors in triangulation, denoted "reprojection". <FIG> presumes that scene points have been computed by triangulation for the keypoints K1-K3 in view V1, for example as illustrated in <FIG>. In the presence of observation errors, the respective scene point will be an approximation, and the calibration data for the respective camera may be used to re-project the scene point onto the respective view, resulting in a reprojection point. <FIG> shows examples of such reprojection points R1-R3. <FIG> illustrates a reprojection of scene points, calculated for object keypoints O1, onto a view V1. The reprojection results in a reprojected pose RO1. The difference between the reprojected pose RO1 and the object keypoints O1 may be quantified as a "total reprojection error", which is computed as an aggregation of distances, for example Euclidean distances, between corresponding keypoints K1-K3 and reprojection points R1-R3 (<FIG>). The aggregation may comprise a sum of the distances, an average of the distances, a sum of squares for the distances, a norm of the distances, etc..

<FIG> is a flow chart of an example method <NUM> for calibrating the cameras in a monitoring system, for example the system <NUM> as shown in <FIG>. In the example of <FIG>, the method <NUM> may be performed jointly by the detection device <NUM> and the positioning device <NUM>. Alternatively, at least part of the calibration method may be performed by a dedicated device, which may be connected to receive the ODD from the detection device <NUM> and/or the images from the cameras <NUM> and may provide the resulting calibration data to the positioning device <NUM>. The method <NUM> will be described with further reference to <FIG>, which exemplifies the data that may be produced by the respective step in <FIG>.

In step <NUM>, image data is received from the cameras. The image data may be in form of raw image data from the respective camera, or a processed version of the raw data. The image data may comprise a single image or a time sequence of images that are generated by the respective camera during a predefined observation time window. In <FIG>, step <NUM> obtains a time sequence of images from the respective camera <NUM>, where the images are represented by vertical lines along a time axis. The cameras <NUM> are arranged to define N different and at least partially overlapping views V1-VN. By jointly processing a time sequence of images, the number of available keypoints is increased significantly, which may be a technical advantage in some embodiments described herein. For example, if the cameras <NUM> operate at a frame rate of <NUM> and the time sequence is captured for <NUM> seconds, each camera generates <NUM> images for processing by the method <NUM>. Thus, even if the number of keypoints in each image may be limited (see below), the total number of available keypoints for processing will be considerable.

In step <NUM>, the image data is processed for feature detection, resulting in the above-mentioned keypoints and keypoint locations. Step <NUM> may apply any conventional object detection technique and/or feature detection technique, for example as described hereinabove. In one example, step <NUM> may operate a human pose estimation algorithm on the image data. Such algorithms are configured to detect people in an image or a time sequence of images, and output the location of joints and extremities of the detected people. It may be noted that it is not necessary to use all available keypoints for calibration. For example, keypoints that are expected to have larger observation errors may be excluded. For a human individual, keypoints representing ears, feet, hands, etc. may be excluded. In <FIG>, step <NUM> results in a set of keypoints for the respective time sequence of images, designated by [F1], [F2],. In these sets, the keypoints may be assigned an object and a time point.

In step <NUM>, the keypoints are associated or matched between the images (views), and optionally across time if time sequences of images are being processed. There are numerous available techniques for association/matching that may be applied by step <NUM>, including but not limited to histograms, SIFT descriptors, appearance models, spatial-temporal continuity, etc. Step <NUM> results in associations of keypoints between images and optionally over time. In <FIG>, these associations are collectively designated by [F1:F2:. If time sequences of images are processed, step <NUM> may be seen to produce a trajectory for each keypoint, given by pairs of keypoint location and time point. To improve the quality of such trajectories, step <NUM> may comprise a sub-step of operating a smoothing function on the respective trajectory. The smoothing function may comprise interpolation and/or low-pass filtering. Such a smoothing results in an updated set of associated keypoints, which may be used as input by one or more of the following steps, including one or more of steps <NUM>-<NUM>.

In step <NUM>, the associated keypoints are processed for calculation of preliminary calibration data for the cameras <NUM>. There are existing techniques for calculating the preliminary calibration data, for example so-called Structure from Motion (SfM) algorithms. Embodiments of an improved technique will be described further below with reference to <FIG>. In <FIG>, the preliminary calibration data for the different cameras <NUM> is designated by θ11, θ2i,.

In step <NUM>, the associated keypoints are processed for calculation of a preliminary pose for the respective object in the scene <NUM> at one or more time points (cf. individuals <NUM> in <FIG>), where each preliminary pose is given by a respective set of scene points (cf. 10A-10C in <FIG>). In some embodiments, the preliminary pose is calculated by triangulation of a subset of the associated keypoints, and by use of the preliminary calibration data. If step <NUM> calculates the preliminary calibration data based on a first subset of associated keypoints, it may be advantageous for step <NUM> to calculate the preliminary pose(s) based on a second subset of associated keypoints, with at least part of the associated keypoints differing between the first and second subsets. The use of such first and second subsets increases the need for keypoints. The number of available keypoints may be increased by jointly processing plural images within an observation time window, as described hereinabove. Step <NUM> results in a set of preliminary poses, one for each object and each time point. In the example of <FIG>, the set of preliminary poses is designated by [Pi]. In some embodiments, the respective pose may be represented by a pose vector of scene point locations.

In step <NUM>, a so-called bundle adjustment is performed for the associated keypoints using the preliminary calibration data and the set of preliminary poses as input. In this context, the bundle adjustment produces refined calibration data for the cameras <NUM>. In the example of <FIG>, the refined (or final) calibration data for the different cameras <NUM> is designated by θ1, θ2,.

The bundle adjustment is a well-known technique of simultaneously refining scene points and calibration data in view of an optimization criterion. Conventionally, bundle adjustment involves minimizing an objective function that defines the above-mentioned total reprojection error. In such an objective function, the scene points and the calibration data are the unknown variables of the objective function. In bundle adjustment, the objective function may be optimized in an iterative fashion, with initial values ("starting values") for the calibration data and the pose being given by the preliminary calibration data and the preliminary pose. However, conventional bundle adjustment is inherently sensitive to the above-mentioned observation errors. Embodiments of an improved technique will be described further below with reference to <FIG>.

In step <NUM>, the method <NUM> provides final camera calibration data, which is the refined calibration data produced by step <NUM>, or a post-processed version thereof. In some embodiments, depending on the common coordinate system, the post-processing may be applied to convert the refined calibration data into a coordinate system with a coordinate axis perpendicular to the ground, for example by rotating the common coordinate system. In some embodiments, post-processing may be applied to estimate the scaling of the monitoring system, for example by detecting images of an object of known length, measuring the time it takes for an object travelling at known speed or acceleration between locations (for example, estimate initial height by measuring time it takes for an object to free fall to the ground), or manipulating images to position keypoints at known locations. The camera calibration data provided by step <NUM> may be used by the processing device <NUM> to compute the scene points when the system <NUM> is operated for monitoring the scene <NUM>.

It is understood that the method <NUM> may be repeated to update the calibration data ("re-calibration"), for example to account for intentional or unintentional changes in the position and/or orientation of one or more of the cameras <NUM>. Such re-calibration may be triggered based on image analysis. For example, in the example with human individuals, the positioning device <NUM> may analyze the distribution of the length of the limbs of individuals over time, to detect a need for re-calibration. If the monitoring system <NUM> is well calibrated, the distribution should have a small standard deviation. Thus, in one example, re-calibration may be trigged when the standard deviation for limb length exceeds a predefined threshold.

<FIG> is a flow chart of an example procedure for calculating the final calibration data in accordance with some embodiments. As indicated, the example procedure may be implemented as part of step <NUM> in <FIG>. Step <NUM> obtains preliminary calibration data for the cameras <NUM> in the system <NUM>. The preliminary calibration data may have been calculated by step <NUM>, or otherwise estimated, for example based on the most recent final calibration data. Step <NUM> obtains a preliminary pose for at least one individual located in the scene and at least one time point. The preliminary pose may have been calculated by step <NUM>. The subsequent processing in steps <NUM>-<NUM> will only operate on the keypoints that correspond to the preliminary pose. To increase the number of keypoints, step <NUM> may thus obtain preliminary poses for more than one individual (if present) and more than one time point. In some embodiments, step <NUM> may obtain a single preliminary pose and the subsequent processing may operate to determine the calibration data based on the single preliminary pose, for example by use of the total reprojection error and the pose error function (below). However, it is to be understood that step <NUM> may obtain a plurality of preliminary poses, for example for different objects in the scene (cf. individuals <NUM> in <FIG>) and/or for different time points during the above-mentioned observation time window. The subsequent processing may operate to determine the calibration data based on the plurality of preliminary poses, not only by use of any of the total reprojection error, the pose error function, and the motion error function (below).

Step <NUM> defines the objective function E to be optimized by step <NUM>. Defining the objective function comprises at least defining 303A a first function and defining 303B a second function. The first function, designated ER herein, represents the total reprojection error and may be defined as described with reference to <FIG>. In some embodiments, ER is defined using, as input data ("known values"), the keypoint locations of the keypoints that correspond to the preliminary pose(s) obtained by step <NUM>, or a subset thereof, and is further defined using, as unknown variables ("latent variables"), the calibration data and the pose(s).

In some embodiments, the first function ER is configured to operate a Huber loss function on the distances between corresponding keypoints and reprojection points to compute difference values that are then aggregated to define the total reprojection error. By definition, the Huber loss function is quadratic for small distances and linear for large distances and thereby reduces the influence of outliers among the distances. Such outliers may be the result of the above-mentioned observation errors.

In some embodiments, the aggregation of distances in the first function ER comprises a weighted sum of difference values, where each difference value represents the reprojection error for a keypoint and its corresponding scene point. The weighted sum comprises a weight for the respective difference value: ∑j(wj · ΔDj), with j designating the respective pair of keypoint and scene point. As understood from the foregoing, the difference value, ΔDj, may be a distance (for example Euclidean), the absolute value of the distance, the square of the distance, an output value of the Huber loss function for the distance, etc. It has been surprisingly found that the distribution of reprojection errors varies in dependence of the distance from the camera to the scene points ("scene point distance"). Specifically, there is an overall trend that scene points further away from the camera have a smaller reprojection error on the camera. In some embodiments, the weight wj for the respective difference value ΔDj is computed as a function of its corresponding scene point distance. For example, the weight may be increased with increasing scene point distance to improve robustness to outliers and thereby improve the accuracy of the total reprojection error in the first function ER. In some embodiments, the weight is computed as a linear function of the scene point distance. In one non-limiting example, the weight is given by: wj = clip(ad + bd · dj, <NUM>, ud), in which dj is the scene point distance, and ad, bd, ud are parameters that may be predefined. Here, a clip function is used to limit the weight to the interval [<NUM>, ud]. It has also surprisingly been found that the distribution of reprojection errors varies in dependence of the above-mentioned confidence score. Specifically, there is an overall trend that keypoints with higher confidence score have a smaller reprojection error on the camera. In some embodiments, the weight wj for a difference value ΔDj is computed as a function of its corresponding confidence score. For example, the weight may be increased with increasing confidence score to thereby improve robustness to outliers. In some embodiments, the weight is computed as a linear function of the confidence score. In one non-limiting example, the weight is given by: wj = clip(as + bs · sj, <NUM>,<NUM>), in which sj is the confidence score, and as, bs are parameters that may be predefined. Here, a clip function is used to limit the weight to the interval [<NUM>,<NUM>]. In some embodiments, the weight may be computed as a function of both the scene point distance and the confidence score, for example by: wj = clip(ad + bd · dj, <NUM>, ud) + clip(as + bs · sj, <NUM>,<NUM>).

The second function, defined by step 303B and designated EQ herein, is a "pose quality function" that determines the quality of the pose(s) with respect to the relative orientation and relative distance between scene points in the respective pose. In some embodiments, EQ is defined to use the pose(s), i.e. groups of scene points, as unknown variable. One general advantage of including EQ in the objective function is that it relaxes the impact of the reprojection errors, and thereby the impact of observation errors, on the optimization. The second function, EQ, is defined to attain high values for poses deemed to be of low quality and low values for poses deemed to be of high quality. The "quality" of a pose refers to the likelihood that the pose is an actual pose that has been attained by the corresponding object in the scene. In the example that the object is a human individual, the second function is thus defined to indicate the likelihood that the pose is a human pose. By accounting for both relative orientation and relative distance between scene points, the second function will have a high ability to assign a proper quality value to each pose. In some embodiments, the second function is configured to determine the quality of the pose as a function of a difference between the pose and one or more valid object poses. Such a second function is also denoted "pose error function" herein and is designated EQP. Optimization of an objective function comprising EQP will encourage the pose to correspond to a plausible pose of the object in the scene. In some embodiments, the second function is configured to determine the quality of the pose based on a comparison between a time sequence of poses, which includes the pose, and a valid object motion. Such a second function is also denoted "motion error function" herein and is designated EQM. Optimization of an objective function comprising EQM will encourage the pose, as included in the time sequence of poses, to correspond to a plausible object motion in the scene.

In some embodiments, the objective function E is a weighted combination of the first and second functions: E = wR · ER + wQ · EQ, with wR and wQ being weights. In some embodiments, EQ is one of the pose error function EQP and the motion error function EQM. In some embodiments, the second function is a weighted combination of the pose error function EQP and the motion error function EQM: EQ = wQP · EQP + wQM · EQM.

It may be noted that the objective function E may include further error functions. In one example, the objective function E comprises a smooth motion error function ESM, which is configured to penalize non-uniform acceleration of keypoints between consecutive time points and to encourage the scene points to represent smooth motion. One example of ESM is described in the above-mentioned article by Takahashi et al, in which the error function comprises third-order differential values for scene point trajectories. By analogy with the above-mentioned keypoint trajectories (step <NUM>), a "scene point trajectory" is a time-sequence of scene points that have been associated across time to represent movement of a specific part of the object in the scene. For objects with complex motion patterns, it may be beneficial to penalize higher-order movement to encourage smooth motion while still allowing for abrupt changes. For example, in human motion, joints may exhibit frequent changes in velocity and acceleration. In some embodiments, ESM may comprise fourth-order differential values for scene point trajectories, for example the l2-norm thereof. In some embodiments, ESM may be normalized by a size parameter for the respective pose, to make ESM independent of scale. In a further example, the objective function E may comprise a length error function EL, which is configured to encourage the reconstructed lengths between pairs of scene points to stay constant over time. For example, for human objects, pairs of scene points at each time point may represent limbs, and the length of such limbs should be substantially constant over time. In some embodiments, EL is configured to operate on the distance between such pairs of scene points as a function of time to generate a measure of the temporal consistency of the distances between the scene points. In one embodiment, EL may compute a coefficient of variation for distances of the respective scene point pair over time and generate the temporal consistency measure by aggregating the coefficients of variation for the different pairs. Here, the coefficient of variation denotes the ratio between the standard deviation of distances and the mean of distances.

It may also be noted that the first function and/or the second function may include one or more further unknown variables, such as object type, activity type, etc..

In step <NUM>, the objective function E defined in step <NUM> is subjected to an optimization procedure, for example iterative, to find values of the unknown variables that optimize the objective function E. In the example that the objective function comprises error functions, the optimization may aim at minimizing the objective function. Any suitable optimization algorithm may be used in step <NUM>, for example a gradient-based algorithm, a direct search method, a Newton-based algorithm, etc..

<FIG> is a flow chart of an example procedure for iteratively optimizing the objective function in accordance with some embodiments. As indicated, the example procedure may be implemented as part of step <NUM> in <FIG>.

In step <NUM>, the starting values for pose and calibration data are determined based on the preliminary pose and the preliminary calibration data. In some embodiments, to reduce ambiguity and complexity caused by pose rotation and scaling, for example in steps <NUM> and <NUM> (<FIG> and below), the starting values for the pose may be determined by aligning the preliminary pose to a reference pose, for example by Procrustes alignment. Thereby, the scene points of the preliminary pose are adjusted in the scene coordinate system <NUM>. In one example, the preliminary pose may be aligned to be upright and face in a predefined direction. In another example, reference locations are defined for a subset of the scene points, and the preliminary pose is transformed so that the subset of scene points are in the reference positions. In some embodiments, to reduce ambiguity caused by object size variation, the (aligned) preliminary pose is normalized, for example by division with its standard deviation.

In some embodiments, for example for human objects, step <NUM> may comprise projecting the respective preliminary pose, given in the scene coordinate system <NUM> (<FIG>), onto a kinematic chain space (KCS) to generate the starting values for the pose. The projection of the preliminary pose results in a KCS matrix, which has limb lengths on its diagonal and an angular representation on the other entries. The limb length corresponds to a distance between pairs of scene points in the pose. In some embodiments, the KCS matrix is computed as: MKCS = M · MT or as its normalized version: MKCS = diag(MKCS)-<NUM> · MKCS · diag(MKCS)-<NUM>, where the function diag operates on a matrix to generate a vector that contains the diagonal elements of the matrix. Here, M is a matrix containing one vector for a respective limb in the pose, the vector being defined by the pair of scene points that corresponds to the limb in the pose. Thus, <MAT> if the number of limbs is L, and MKCS may be regarded as a vector of size L<NUM>. By generating the KCS matrix, the above-described pose alignment and normalization may be omitted since the KCS matrix contains a representation of the angles between the limbs and is unaffected by rotation or scaling. Details of KCS projection are for example described in the articles "<NPL>, and "<NPL>.

Following step <NUM>, the optimization procedure iterates through steps <NUM>-<NUM> until an end criterion is fulfilled, as detected by step <NUM>, whereupon the current calibration data is provided in the monitoring system (step <NUM> in <FIG>). The end criterion may, for example, be given by a predefined number of iterations or a convergence criterion. During each iteration, step <NUM> evaluates the objective function, based on the current values for the pose and the calibration data, and step <NUM> determines updated values for the pose and the calibration data in relation to an optimization criterion. The skilled person understands that the updating step <NUM> is specific to each type of optimization algorithm. In some embodiments, one or more of the iterations comprises steps <NUM>-<NUM>. Step <NUM> identifies a set of keypoints that has a major influence on the objective function, and step <NUM> modifies the set of keypoints to reduce its influence. In some embodiments, step <NUM> first determines the one or more scene points with the largest contribution to the objective function, and then includes the corresponding keypoints in the set of keypoints. Step <NUM> may be performed algorithmically or by brute force testing. Step <NUM> may remove the set of keypoints and the corresponding scene point(s) from the objective function, or apply a weight factor to reduce their impact on the objective function. The rationale for steps <NUM>-<NUM> is that the set of keypoints is likely to be erroneous, and removing it may improve calibration accuracy. In some embodiments, steps <NUM>-<NUM> are performed, for example periodically, towards the end of the optimization procedure.

<FIG> is a flow chart of an example procedure for evaluating the objective function in accordance with some embodiments. As indicated, the example procedure may be implemented as part of step <NUM> in <FIG>.

In step <NUM>, the current pose and calibration data are processed to determine a first value for the total reprojection error, ER.

In step <NUM>, the current pose is processed to determine a second value (pose quality) for the pose error function, EQP. In some embodiments, step <NUM> determines the second value by use of a trained model which is configured to determine the quality of the respective current pose in relation to an ensemble of valid object poses. For example, the trained model may comprise one or more of a neural network, an autoencoder, a PCA analysis function, a Gaussian Mixture Model function, a Normalizing flows function, or any equivalent function.

In some embodiments, the trained model comprises a discriminator of a Generative Adverserial Network (GAN). <FIG> is a block diagram of an example GAN <NUM>, which comprises two neural networks with competing objectives: a discriminator network ("discriminator") <NUM> which is arranged to determine whether a pose is valid or invalid, and a generator network ("generator") <NUM> which is arranged to generate poses that the discriminator <NUM> will classify as valid. An updating module is arranged to update the generator <NUM> and/or the discriminator <NUM> based on the output of the discriminator <NUM>. Training of the GAN <NUM> provides the discriminator <NUM> the ability to detect invalid poses and the generator <NUM> the ability to produce poses that match the distribution of the training data. During training, the generator <NUM> is fed random noise <NUM> and outputs poses <NUM>, i.e. groups of scene points in the scene coordinate system. The discriminator <NUM> is alternatingly provided with the poses <NUM> from the generator <NUM> and valid object poses <NUM>. The valid object poses <NUM> may, for example, be actual poses of objects recorded with an accurate high-speed motion capture system. It may also be beneficial to perturb the valid object poses <NUM> by a small amount of random noise. In one non-limiting example, the GAN <NUM> may be trained in accordance with the article "<NPL>. The valid object poses <NUM> may be realigned and/or normalized, or projected into KCS, as described with reference to step <NUM>. When trained, the discriminator <NUM> is operable to output a score that represents the likelihood that a pose is an invalid object pose, i.e. a low score for valid poses and a high score for invalid poses. The above-mentioned second value may thus be determined by feeding the respective current pose to the trained discriminator <NUM>, where the second value is given as a function of the resulting score.

In a non-limiting example, the generator <NUM> may be built from multiple residual blocks of fully connected (FC) layers. The discriminator <NUM> may comprise one path for feeding poses into a kinematic chain space (KCS) layer, for example as described in the above-mentioned article by Wandt and Rosenhahn. The discriminator <NUM> may further comprise another path built from multiple FC layers. The feature vectors of both paths may be concatenated and fed into another FC layer which outputs the score.

In some embodiments, the trained model used by step <NUM> is a PCA model which has been trained by fitting it to an ensemble of valid object poses, for example actual poses of objects recorded with an accurate high-speed motion capture system. To improve robustness of the fitted model, data augmentation may be performed during training by also fitting the PCA model to one or more copies of the ensemble in which noise, for example Gaussian noise, has been applied to the joint locations (scene points) of the valid object poses. Once the PCA model is fitted, a likelihood or probability value of each pose may be calculated by use of the PCA model. The second value (pose quality) for the pose error function EQP may then be given by the average likelihood of all poses. The calculation of a probability value by use of a fitted PCA model is known in the art and may, for example, be performed based on the teachings in the article "<NPL>.

Reverting to <FIG>, step <NUM> processes a time sequence of poses that includes the current pose to determine a third value (motion quality) for the motion error function, EQM, and step <NUM> combines the first, second and third values into a current value of the objective function. The current value may then be processed by step <NUM> in <FIG>.

It should be noted, as explained further above, that steps <NUM> and <NUM> may process a plurality of current poses in each iteration of the optimization procedure. In such embodiments, the first value may be given by an aggregation of the total projection error for each of the current poses, and the second value may be given by an aggregation of the pose quality for each of the current poses.

<FIG> is a flow chart of an example procedure for determining the third value for the motion error function in accordance with some embodiments. As indicated, the example procedure may be implemented as part of step <NUM> in <FIG>.

In step <NUM>, a time sequence of poses is selected. The selected time sequence of poses may correspond to a sequence of time points within the above-mentioned observation time window. <FIG> shows an example of such a selected sequence <NUM>, where each vertical panel represents a pose and a time point.

In step <NUM>, a prediction function is operated on poses in the time sequence to estimate a valid motion pose at one or more selected time points.

In step <NUM>, a difference is calculated between one or more pairs of valid motion pose and comparison pose. The comparison pose is included in the selected sequence and associated with the same selected time point as the valid motion pose in the respective pair. The difference may, for example, be calculated by aggregation of Euclidean distances between corresponding scene points in the respective pair of valid motion pose and the comparison pose.

In step <NUM>, the third value is calculated as a function of the difference.

An example of the procedure in <FIG> is illustrated in <FIG>. Here, the prediction function <NUM> is operated on all poses in the selected sequence <NUM> except one comparison pose <NUM> to estimate a valid motion pose <NUM> at the time point of the comparison pose <NUM>, and a comparison module <NUM> is operated to compute the difference between the valid motion pose <NUM> and the comparison pose <NUM>. It may be noted that the comparison pose(s) <NUM> need not be subsequent to the selected sequence <NUM>, as shown in <FIG>, but may be located at any time point within the selected sequence.

As indicated by a dashed arrow in <FIG>, steps <NUM>-<NUM> may be repeated any number of times to calculate further differences. The repetitions may differ by selected time sequence and/or by time point(s) of the comparison pose(s). If steps <NUM>-<NUM> are repeated, step <NUM> may aggregate the differences produced by the repetitions to improve the accuracy of the third value.

The prediction function <NUM> may comprise a trained model like the trained model used in step <NUM> and may, for example, comprise one or more of a neural network, an autoencoder, a PCA analysis function, a Gaussian Mixture Model function, a Normalizing flows function, or any equivalent function. The prediction function <NUM> may be trained based on sequences of poses representing valid motion of poses. For example, the GAN <NUM> in <FIG> may be trained by the discriminator <NUM> receiving time sequences of poses <NUM> from the generator <NUM> as well as valid sequences of poses <NUM>. The generator <NUM> and the discriminator <NUM> may be configured as described above. In another non-limiting example, the multiple residual blocks of the discriminator <NUM> may be first applied to individual poses in a time sequence to extract features for individual poses, whereupon residual blocks of convolutions may be applied to the extracted feature for temporal aggregation.

<FIG> is a flow chart of an example procedure for calculating the preliminary calibration data in accordance with some embodiments. As indicated, the example procedure may be implemented as part of step <NUM> in <FIG>. In step <NUM>, a plurality of candidate calibration data is calculated for a plurality of different subsets of the associated keypoints generated by step <NUM>. In step <NUM>, the preliminary calibration data is determined as a function of the plurality of candidate calibration data. By determining a plurality of calibration data for different subsets of associated keypoints, the accuracy and robustness of the preliminary calibration data may be improved since the impact of any single subset of associated keypoints is reduced. By processing different subsets of associated keypoints, the computational load may also be reduced compared to calculating the preliminary calibration data for all available associated keypoints. It is realized that steps <NUM>-<NUM> will greatly benefit from the increase in available keypoints that is achieved by jointly processing a time sequence of images within the above-mentioned observation time window (cf. step <NUM>). In some embodiments, step <NUM> may select the preliminary calibration data among the plurality of calibration data based on any suitable evaluation parameter.

<FIG> is a flow chart of a detailed implementation example of the procedure in <FIG>. The implementation uses a sequence of RANSAC (random sample consensus) procedures to generate and gradually refine the selection among the candidate calibration data. In step <NUM>, a view counter (v) is set to a starting value, for example <NUM>, and a scene point vector S is initialized. In step <NUM>, a pair of views is selected among the available views. The pair of views may be selected based on the number of keypoints that are associated between the views. In some embodiments, the pair of views may be selected to have a number of associated keypoints between a minimum and maximum limit. The minimum limit may be set to ensure a sufficient accuracy. The maximum limit may be set to ensure that the views reproduce the scene from sufficiently different angles, which has also been found to improve accuracy. In step <NUM>, a collection of associated keypoints (OBS_TOT) for the pair of views is determined. The collection may include all associated keypoints for the pair of views. After step <NUM>, the procedure enters a primary iteration over views and a secondary iteration over keypoint subsets. In step <NUM>, a repetition counter (r) is set to a starting value, for example <NUM>. Step <NUM> selects a first group of associated keypoints (OBS1) among OBS_TOT, and step <NUM> calculates candidate calibration data (θ_v_r) as a function of OBS1. Step <NUM> may use any available algorithm for this purpose, for example an eight-point algorithm for the first pair of points (v=<NUM>) and the efficient P-n-P algorithm algorithm for v><NUM>. Step <NUM> selects a second group of associated keypoints (OBS2) among OBS_TOT, and step <NUM> calculates a corresponding group of scene points (GSP2) as a function of OBS2 and θ_v_r calculated by step <NUM>. The first and second keypoint groups OBS1, OBS2 differ by at least one keypoint and may be mutually exclusive. It may be noted that GSP2 need not be (and generally is not) a pose of an object in the scene, since its scene points correspond to OBS2, which is an arbitrary selection of associated keypoints from OBS_TOT. Step <NUM> determines a match score that represents the correspondence between OBS2 and GSP2. In some embodiments, the match score is based on the reprojection errors (RE) for the pair of views, with RE being calculated as a function of GSP2, OBS2 and θ_v_r. Step <NUM> generates a count of the number of inliers (C_r) in GSP2 as a function of RE. In this context, an inlier may be defined as a scene point in GSP2 that yields an RE below a predefined distance threshold. Steps <NUM>-<NUM> are then repeated while incrementing the repetition counter (step <NUM>) until the repetition counter reaches a predefined value (rMAX), as detected by step <NUM>. The first and second groups are selected to differ across the repetitions. It is realized that the iteration of steps <NUM>-<NUM> serves to generate a plurality of candidate calibration data: [θ_v_1,. , θ_v_rMAX] with v=<NUM>, and corresponding evaluation parameter values, [C_1,. The procedure then proceeds to step <NUM>, which selects the preliminary calibration data (θi_v with v=<NUM>) for the pair of views, and thus for the corresponding pair of cameras, among [θ_2_1,. , θ_2_rMAX], for example the candidate calibration data with the largest count of inliers. Step <NUM> calculates a total set of scene points (TOT_v with v=<NUM>) as a function of θi_2 and OBS_TOT. Step <NUM> updates the scene point vector S, which is empty at this stage, by adding GSP_2 to S.

The procedure then proceeds to step <NUM>, which evaluates if there are any remaining views. If so, the procedure continues to step <NUM>, which selects one remaining view based on a selection criterion. In some embodiments, the selection criterion identifies the remaining view that has the largest number of, and/or most evenly distributed, scene points in common with the scene points in vector S. Step <NUM> then increments the view counter and updates OBS_TOT to contain associated keypoints that are present in all views in the current group of views, i.e. the pair of views selected by step <NUM> and the remaining view selected by step <NUM>. The procedure returns to step <NUM> and iterates through steps <NUM>-<NUM>. Step <NUM> then selects the preliminary calibration data (θi_v with v=<NUM>) for the camera corresponding to the selected remaining view. Step <NUM> calculates the total set of scene points (TOT_v with v=<NUM>) for the associated keypoints in OBS_TOT that appear in at least two views in the current group of views. Step <NUM> updates S with any new scene points in TOT_3 compared to TOT_2 (i.e. the "set difference" of TOT_3 and TOT_2). It is realized that the procedure continues until all available views have been processed, resulting in preliminary calibration data for all views ([θi_v]), whereupon step <NUM> outputs ([θi_v], for example for use by step <NUM>, and optionally step <NUM> (<FIG>). In some embodiments, step <NUM> may be omitted and the preliminary pose(s) may be given by the scene points in vector S, or a subset thereof.

The structures and methods disclosed herein may be implemented by hardware or a combination of software and hardware. In some embodiments, such hardware comprises one or more software-controlled computer resources. <FIG> schematically depicts such a computer resource <NUM>, which comprises a processing system <NUM>, computer memory <NUM>, and a communication interface <NUM> for input and/or output of data. The communication interface <NUM> may be configured for wired and/or wireless communication, including communication with the detection device <NUM> and/or the cameras <NUM>. The processing system <NUM> may e.g. include one or more of a CPU ("Central Processing Unit"), a DSP ("Digital Signal Processor"), a microprocessor, a microcontroller, an ASIC ("Application-Specific Integrated Circuit"), a combination of discrete analog and/or digital components, or some other programmable logical device, such as an FPGA ("Field Programmable Gate Array"). A control program 51A comprising computer instructions is stored in the memory <NUM> and executed by the processing system <NUM> to perform any of the methods, operations, procedures, functions, models or steps described in the foregoing. As indicated in <FIG>, the memory <NUM> may also store control data 52B for use by the processing system <NUM>, for example various predefined values such as thresholds, limits, etc. The control program 52A may be supplied to the computing resource <NUM> on a computer-readable medium <NUM>, which may be a tangible (non-transitory) product (e.g. magnetic medium, optical disk, read-only memory, flash memory, etc.) or a propagating signal.

Claim 1:
A computer-implemented method of calibrating cameras (<NUM>) that are arranged to generate at least partly overlapping views (V1-V3) of a scene (<NUM>), said method comprising:
obtaining (<NUM>) preliminary calibration data of the cameras (<NUM>), said preliminary calibration data representing an arrangement of the cameras (<NUM>) in relation to a common coordinate system (<NUM>);
obtaining (<NUM>) a preliminary pose for an object (<NUM>) in the scene (<NUM>), the preliminary pose comprising three-dimensional point locations in the common coordinate system (<NUM>) and being calculated based on feature points detected in the views (V1-V3), wherein the object (<NUM>) is a live body and the point locations that are included in the pose correspond to joints and/or extremities of the live body;
defining (<NUM>) an objective function comprising calibration data and pose as unknown variables, the objective function comprising a first function (303A) that determines an aggregation of differences between the feature points and the pose as projected onto the at least partly overlapping views (V1-V3) by use of the calibration data, and a second function (303B) that determines a quality of the pose;
optimizing (<NUM>) the objective function as a function of the calibration data and the pose, with starting values for the calibration data and the pose given by the preliminary calibration data and the preliminary pose; and
providing (<NUM>) the calibration data that results in an optimization of the objective function, characterized in that the second function (303B) determines the quality of the pose as a function of relative orientation, in the common coordinate system (<NUM>), between point locations included in the pose, and as a function of relative distance, in the common coordinate system (<NUM>), between the point locations included in the pose.