Methods and apparatuses for determining positions of multi-directional image capture apparatuses

A method comprising performing image re-projection on each of a plurality of first images of a scene, thereby to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera; processing the plurality of second images based on a previously generated virtual three dimensional model of the scene, thereby to generate respective positions of the virtual cameras associated with the second images; and determining a position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras.

TECHNICAL FIELD

The present specification relates to methods and apparatuses for determining positions of multi-directional image capture apparatuses.

BACKGROUND

Camera pose registration is an important technique used to determine positions and orientations of image capture apparatuses such as cameras. The recent advent of commercial multi-directional image capture apparatuses, such as 360° camera systems, brings new challenges with regard to the performance of camera pose registration in a reliable, accurate and efficient manner.

Summary

According to a first aspect, this specification describes a method comprising performing image re-projection on each of a plurality of first images of a scene, thereby to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera; processing the plurality of second images based on a previously generated virtual three dimensional model of the scene, thereby to generate respective positions of the virtual cameras associated with the second images; and determining a position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras.

The method of the first aspect may further comprise updating the previously generated virtual three dimensional model with the determined position of the multi-directional image capture apparatus.

The virtual three dimensional model may comprise information describing features of the scene.

The previously generated three dimensional model may have been generated based at least in part on images previously captured by the first multi-directional image capture apparatus.

The previously generated three dimensional model may have been generated based at least in part on images previously captured by a set of one or more multi-directional image capture apparatuses which does not include the first multi-directional image capture apparatus.

Processing the plurality of second images based on the virtual three dimensional model may comprise determining point correspondences between the second images and the virtual three dimensional model, and performing a Direct Linear Transform based on the determined point correspondences to determine positions of each of the virtual cameras.

Determining the position of the multi-directional image capture apparatus based on the generated positions of the virtual cameras may comprise determining a position of a camera of the multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras, and determining the position of the multi-directional image capture apparatus based on the determined position of the camera.

Determining the position of the multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras may comprise determining outliers and inliers in the generated positions of the virtual cameras, and determining the position of the multi-directional image capture apparatus based only on the inliers.

Determining the position of the multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras may further comprise determining the virtual camera having the smallest re-projection error associated therewith, and determining the position of the multi-directional image capture apparatus based only on the position of the virtual camera having the smallest re-projection error associated therewith.

Processing of the plurality of second images may further generate respective orientations of the virtual cameras, and the method of the first aspect may further comprise determining an orientation of the multi-directional image capture apparatus based on one or more of the generated orientations of the virtual cameras.

The position of the multi-directional image capture apparatus may be determined based on both the one or more generated positions of the virtual cameras and the one or more generated orientations of the virtual cameras.

According to a second aspect this specification describes apparatus configured to perform any method described with reference to the first aspect.

According to a third aspect, this specification describes computer-readable instructions which, when executed by computing apparatus, cause the computing apparatus to perform any method described with reference to the first aspect.

According to a fourth aspect, this specification describes apparatus comprising at least one processor, and at least one memory including computer program code, which when executed by the at least one processor, causes the apparatus to: perform image re-projection on each of a plurality of first images of a scene, thereby to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera; process the plurality of second images based on a previously generated virtual three dimensional model of the scene, thereby to generate respective positions of the virtual cameras associated with the second images; and determine a position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras.

The computer program code, when executed by the at least one processor, may further cause the apparatus to: update the previously generated virtual three dimensional model with the determined position of the multi-directional image capture apparatus.

The virtual three dimensional model may comprise information describing features of the scene.

The previously generated three dimensional model may have been generated based at least in part on images previously captured by the first multi-directional image capture apparatus.

The previously generated three dimensional model may have been generated based at least in part on images previously captured by a set of one or more multi-directional image capture apparatuses which does not include the first multi-directional image capture apparatus.

Processing the plurality of second images based on the virtual three dimensional model may comprise determining point correspondences between the second images and the virtual three dimensional model, and performing a Direct Linear Transform based on the determined point correspondences to determine positions of each of the virtual cameras.

Determining the position of the multi-directional image capture apparatus based on the generated positions of the virtual cameras may comprise determining a position of a camera of the multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras, and determining the position of the multi-directional image capture apparatus based on the determined position of the camera.

Determining the position of the multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras may comprise determining outliers and inliers in the generated positions of the virtual cameras, and determining the position of the multi-directional image capture apparatus based only on the inliers.

Determining the position of the multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras may further comprise determining the virtual camera having the smallest re-projection error associated therewith, and determining the position of the multi-directional image capture apparatus based only on the position of the virtual camera having the smallest re-projection error associated therewith.

Processing of the plurality of second images may further generate respective orientations of the virtual cameras, and the computer program code, when executed by the at least one processor, may further cause the apparatus to determining an orientation of the multi-directional image capture apparatus based on one or more of the generated orientations of the virtual cameras.

The position of the multi-directional image capture apparatus may be determined based on both the one or more generated positions of the virtual cameras and the one or more generated orientations of the virtual cameras.

According to a fifth aspect, this specification describes a computer-readable medium having computer-readable code stored thereon, the computer readable code, when executed by at least one processor, causes performance of: performing image re-projection on each of a plurality of first images of a scene, thereby to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera; processing the plurality of second images based on a previously generated virtual three dimensional model of the scene, thereby to generate respective positions of the virtual cameras associated with the second images; and determining a position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras.

The computer-readable code stored on the medium of the fifth aspect may further cause performance of any of the operations described with reference to the method of the first aspect.

According to a sixth aspect, this specification describes apparatus comprising means for performing image re-projection on each of a plurality of first images of a scene, thereby to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera; means for processing the plurality of second images based on a previously generated virtual three dimensional model of the scene, thereby to generate respective positions of the virtual cameras associated with the second images; and means for determining a position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras.

The apparatus of the sixth aspect may further comprise means for causing performance of any of the operations described with reference to the method of the first aspect.

DETAILED DESCRIPTION

In the description and drawings, like reference numerals may refer to like elements throughout.

FIG. 1illustrates a plurality of multi-directional image capture apparatuses10located within an environment.

The multi-directional image capture apparatuses10may, in general, be any apparatus capable of capturing images of a scene12from multiple different perspectives simultaneously. For example, a multi-directional image capture apparatus10may be a 360° camera system (also known as an omnidirectional camera system or a spherical camera system). However, it will be appreciated that a multi-directional image capture apparatus10does not necessarily have to have full angular coverage of its surroundings and may only cover a smaller field of view.

The term “image” used herein may refer generally to visual content. This may be visual content which is captured by a multi-directional image capture apparatus10, or visual content which is derived from visual content captured by a multi-directional image capture apparatus10. For example, an image may be a single frame of a video.

As illustrated inFIG. 1, each multi-directional image capture apparatus10may comprise a plurality of cameras11. The term “camera” used herein may refer to a sub-part of a multi-directional image capture apparatus10which performs the capturing of images.

As illustrated, each of the plurality of cameras11of a multi-directional image capture apparatus10may be facing a different direction to each of the other cameras11of the multi-directional image capture apparatus10. As such, each camera11of a multi-directional image capture apparatus10may have a different field of view, thus allowing the multi-directional image capture apparatus10to capture images of the scene12from different perspectives simultaneously. The fields of view of the different cameras11may be partially overlapping.

Similarly, as illustrated inFIG. 1, each multi-directional image capture apparatus10may be at a different location to each of the other multi-directional image capture apparatuses10. Thus, each of the plurality of multi-directional image capture apparatuses10may capture images of the scene12(via their cameras11) from different perspectives simultaneously. In this way, the multi-directional image capture apparatuses10may be used for a number of different functions. For example, they may be used for functions such as live event broadcasting, video conferencing and virtual reality tourism.

In the example scenario illustrated byFIG. 1, it is possible to perform camera pose registration in order to determine the position and/or orientation of each of the multi-directional image capture apparatuses10within the environment. In particular, it is possible to determine these positions and/or orientations relative to a particular reference coordinate system, which allows the overall arrangement of the multi-directional image capture apparatuses10relative to each other to be determined.

One way of performing camera pose registration is to use a computer vision method. For example, structure from motion (SfM) analysis may be used. SfM is a known technique which is used to estimate three dimensional (3D) structures from two dimensional (2D) images. Broadly speaking, SfM works by determining point correspondences between images (also known as feature matching) and calculating positions and orientations of various features based on the determined point correspondences.

SfM analysis is also capable of outputting the positions and/or orientations of each of the cameras that are associated with the images, which can then be used to determine the position and/or orientation of each of the multi-directional image capture apparatuses10. Such methods for performing camera pose registration are described in UK patent applications GB1620312.7 and GB1702680.8, which are hereby incorporated by reference. As such, by using SfM analysis on images captured by the multi-directional image capture apparatuses10, it is possible to generate a virtual 3D model of the scene12.

A virtual 3D model of the scene12may be a set of information describing various features of the scene12. The virtual 3D model may, for example, include information describing one or more of the positions, orientations and pixel values of various features in the scene12. The position and orientation information may be relative to a particular reference co-ordinate system. The virtual 3D model may also include information indicating correspondences between 3D voxels of the virtual 3D model and 2D pixels in the images from which the virtual 3D model is generated. It will be appreciated that at least some of the information contained in the virtual 3D model may be visually represented by a point cloud.

In some examples, information describing the positions and/or orientations of the multi-directional image capture apparatuses10(obtained, for example, from the SfM analysis described above) may be stored in a dedicated position and orientation database. Alternatively, this information may be included in the virtual 3D model.

After the camera pose registration and the generation of a virtual 3D model as described above, the arrangement of the multi-directional image capture apparatuses10within the environment illustrated byFIG. 1may change with time. For example, as illustrated byFIG. 2A, one of the multi-directional image capture apparatuses10in the arrangement may be moved to a new position and/or orientation. In another example, as illustrated byFIG. 2B, a new multi-directional image capture apparatus10may be added to the arrangement. For instance, if the multi-directional image capture apparatuses10are being used for live broadcasting of a live stage event, then one of the multi-directional image capture apparatuses10may be moved to follow a dancer or singer on the stage, while the other multi-directional image capture apparatuses10may be kept stationary to monitor the stage as a whole. Similarly, a new multi-directional image capture apparatus10may be added to monitor the live stage event.

In the scenario ofFIG. 2A, it may be desirable to determine the new position and/or orientation of the multi-directional image capture apparatus10which has been moved. Similarly, in the scenario ofFIG. 2B, it may be desirable to determine the position and/or orientation of the newly added multi-directional image capture apparatus10.

One way of determining the position and/or orientation of the moved multi-directional image capture apparatus10ofFIG. 2Aor the new multi-directional image capture apparatus10ofFIG. 2Bis to perform the above-mentioned SfM analysis again using new images captured by the multi-directional image capture apparatuses10. However, this approach may be time consuming, computationally intensive and error-prone.

A method for performing camera pose registration which may address some or all of the challenges mentioned above will now be described.

FIG. 3illustrates a multi-directional image capture apparatus10for which camera pose registration is desired. For example, this multi-directional image capture apparatus10may be the multi-directional image capture apparatus10which has been moved inFIG. 2Aor the newly added multi-directional image capture apparatus10inFIG. 2B.

A camera11of the multi-directional image capture apparatus10may capture a first image21. The first image21may be an image of the scene within the field of view20of the camera11. In some examples, the lens of the camera11may be a fish-eye lens and so the first image21may be a fish-eye image (in which the camera field of view is enlarged). However, the method described herein may be applicable for use with lenses and resulting images of other types. For example, the camera pose registration method described herein may also be applicable to images captured by a camera with a hyperbolic mirror in which the camera optical centre coincides with the focus of the hyperbola, and images captured by a camera with a parabolic mirror and an orthographic lens in which all reflected rays are parallel to the mirror axis and the orthographic lens is used to provide a focused image.

The first image21may be processed to generate one or more second images22. More specifically, image re-projection may be performed on the first image21to generate one or more re-projected second images22. For example, if the first image21is not a rectilinear image (e.g. it is a fish-eye image), it may be re-projected to generate one or more second images22which are rectilinear images (as illustrated byFIG. 3). Each generated second image22may at least partially represent a different portion of the first image21. It will be appreciated that, in general, the re-projection may generate any type of second image22, as long as the image type is compatible with the algorithm used to analyse the re-projected images.

Each re-projected second image22may be associated with a respective virtual camera. A virtual camera is an imaginary camera which does not physically exist, but which corresponds to a camera which would have captured the re-projected second image22with which it is associated. A virtual camera is defined by virtual camera parameters which represent the configuration of the virtual camera required in order to have captured to the respective second image22. As such, for the purposes of the methods and operations described herein, a virtual camera can be treated as a real physical camera. For example, each virtual camera has, among other virtual camera parameters, a position and orientation which can be determined.

When a plurality of re-projected second images22are generated (e.g.FIG. 3illustrates nine re-projected second images22being generated), each re-projected second image22may have a different viewing direction compared to each of the other second images22. In other words, the virtual camera of each second image22may have a different orientation compared to each of the other virtual cameras. Similarly, the orientation of each of the virtual cameras may also be different to the orientation of the real camera11which captured the first image21. Furthermore, each virtual camera may have a smaller field of view than the real camera11as a result of the re-projection. The virtual cameras may have overlapping fields of view with each other.

The orientations of the virtual cameras may be pre-set. In other words, the re-projection of the first image21may generate second images22with associated virtual cameras which each have a certain pre-set orientation relative to the orientation of the real camera11. For example, the orientation of each virtual camera may be pre-set such that it has certain yaw, pitch and roll angles relative to the real camera11.

It will be appreciated that, in general, any number of second images22may be generated.

It will also be appreciated that the first images21may correspond to images of the scene at a particular moment in time. For example, if the multi-directional image capture apparatus10is capturing video images, a first image21may correspond to a single video frame of a single camera11, and all of the first images21may be video frames that are captured at the same moment in time.

Having generated the second images22, these images can then be processed based on a previously generated virtual 3D model of the scene, which may be retrieved from a 3D reconstruction database (seeFIG. 5B). In particular, as illustrated byFIG. 4A, a second image22may be processed based on a previously generated virtual 3D model by determining point correspondences between the second image22and the virtual 3D model. For instance, as illustrated byFIG. 4A, correspondences between points in a second image22and points on an object31in the virtual 3D model may be determined.

Following the determination of point correspondences, a Direct Linear Transform may be performed based on the determined point correspondences to determine the position and/or orientation of the virtual camera associated with each second image22. More specifically, a Perspective-n-Point (PnP) problem may be solved to determine the position and/or orientation of each virtual camera. It will be appreciated that for PnP problems, the intrinsic camera parameters (such as focal length and principle point) may be known.

The position and/or orientation of each virtual camera may be determined by solving the following equation:

In the above equation, fxand fyare the focal lengths of the virtual camera along x and y axes, cxand cyare x and y coordinates of the principal point of the virtual camera, r11to r33are terms representing the orientation of the virtual camera, t1to t3are terms representing the position of the virtual camera, X, Y, Z are the coordinates of a 3D point in the virtual 3D model, u and v are the homogeneous coordinates of a 2D point in the second image22corresponding to the 3D point, and s is a uniform scaling factor.

In more detail, the above equation may be solved by multiplying out the matrices and vectors for a plurality of point correspondences to obtain a set of simultaneous equations, which can then be solved to obtain values for r11to r33and t1to t3. It will be appreciated that since position and orientation together represent six degrees of freedom, a minimum of six point correspondences may be needed to determine the position and orientation of each virtual camera.

Following the determination of the position and/or orientation of each of the virtual cameras (for example, for the multi-directional image capture apparatus10illustrated byFIG. 3, the positions and/or orientations of nine virtual cameras may be determined for each of the real cameras11of the multi-directional image capture apparatus10), the position and/or orientation of each of the real cameras11may be determined based on the positions and/or orientations of the virtual cameras.

One way of determining the position and/or orientation of a real camera11is to simply take the position and/or orientation of one of the virtual cameras associated with the real camera11as being the position and/or orientation of the real camera11. The virtual camera chosen for this purpose may be a virtual camera whose orientation is presumed to be the same as the orientation of the real camera11. For example, for the nine virtual cameras associated with the nine second images22ofFIG. 3, the position and/or orientation of the virtual camera associated with the central second image22may be taken to be the position and/or orientation of the real camera11(since the orientation of the central camera in this example may be closest to the orientation of the real camera11). It will be appreciated that any one of the virtual cameras corresponding to the real camera11can be chosen for this purpose (e.g. in the example ofFIG. 3, any of the nine virtual cameras may be chosen). This approach can be used because there may be relatively low error associated with performing point correspondence with a virtual 3D model due to the relatively high amount of information contained within the 3D model. Therefore, the determined position and/or orientation of one of the virtual cameras may be a relatively accurate representation of the position and/or orientation of its associated real camera11.

Alternatively, it may still be desirable to further reduce error, since the determined positions and/or orientations may still have a certain degree of error associated with them due to the spatial distribution of the 3D points used and the limited field of view of the virtual cameras. One way of doing this is to determine outliers and inliers in the generated positions of the virtual cameras, and then determining the position of the real camera11based only on the inliers. This may be performed in the following way for a multi-directional image capture apparatus which has been moved (for example, as illustrated byFIG. 2A).

Referring toFIG. 4B, first, the known previous position of each of the virtual cameras (represented by diamond shapes ofFIG. 4B) of the multi-directional image capture apparatus is determined. The previous position may be the position at a previous time (t−1). This may be achieved by determining an offset vector between the position of the multi-directional image capture apparatus and the position of each of its virtual cameras, the offset vector being relative to a reference co-ordinate system. The offset vector relative to the reference co-ordinate system may be determined according to:
vldevw(t−1)=Rdev(t−1)·vldev
where vwldev(t−1) is the offset vector relative to the reference co-ordinate system at time t−1, Rdev(t−1) is a rotation matrix representing the orientation of the multi-directional image capture apparatus at time t−1, and Vldevis the offset vector between the position of the multi-directional image capture apparatus and the position of the virtual camera relative to a local co-ordinate system of the multi-directional image capture apparatus.

The positions of each of the virtual cameras of the multi-directional image capture apparatus at time (t−1) may then be determined according to:
cl(t−1)=cdev+vldevw(t−1)
where cl(t−1) is the position of one of the virtual cameras of the multi-directional image capture apparatus at time t−1, and cdevis the position of the multi-directional image capture apparatus at time t−1.

Then, the relative translation between each virtual camera position at time t−1 (the previous time) and the corresponding position at time t (the current time) may be determined to obtain a set of relative translations. Median absolute deviation analysis may then be performed on the set of relative translations to determine outlier and inliers. This may be performed according to:

di=Si-Median⁡(𝒮),∀Si∈𝒮dσ=Median⁡({d0,…⁢,dN})inliers=didσ<m,∀i∈N
where S is the set of relative translations of virtual cameras corresponding to one real camera, diis a measure of the difference between a relative translation and the median of all relative translations, dσis the median absolute deviation (MAD), m is a threshold value below which a relative translation is considered an inlier (for example, m may be set to be 2).

In this way, as illustrated byFIG. 4B, a virtual camera11awhich does not deviate from the collective pattern between time t−1 and t is considered an inlier, and a virtual camera11bwhich does deviate from the collective pattern between t−1 and t is considered an outlier. By using only the inliers in further analysis, temporal consistency may be improved.

Following the above inlier and outlier determination, one of the remaining inlier virtual cameras may be chosen based on the re-projection errors associated with the virtual cameras. The re-projection error is an error which may be output by the algorithm which is used to determine point correspondences between the virtual 3D model and the second images. The re-projection error may be a measure of how well a 2D projection of a 3D point is geometrically aligned with a 2D feature. In order to improve geometric consistency, the virtual camera which corresponds to a real camera having the smallest re-projection error associated therewith may be determined from the remaining inlier virtual cameras.

After filtering by enforcing both temporal and geometric consistencies as described above, the position of the virtual camera having the smallest re-projection error may be taken to be the position of its respective real camera (this may be referred to as a “winner takes all approach”).

The orientation of the real camera can be determined from the selected virtual camera according to:
Rl=RvRvl−1
where the rotation matrix of a real camera (Rl) may be determined by multiplying the rotation matrix of a virtual camera (Rv) onto the inverse of the rotation matrix representing the orientation of the virtual camera relative to the orientation of the real camera (Rvl−1).

Once the position and/or orientation of the real camera has been determined, the position and/or orientation of the multi-directional image capture apparatus may be determined based on the position and/or orientation of the real camera. This may be done in the following way.

First, the orientation of the multi-directional image capture apparatus is determined according to:
Rdev=RlRldev−1
where Rdevis a rotation matrix representing the orientation of the multi-directional image capture apparatus, Rlis a rotation matrix representing the orientation of the real camera and Rldev−1is the inverse of a rotation matrix representing the orientation of the real camera relative to the multi-directional image capture apparatus.

Put another way, the rotation matrix of a multi-directional image capture apparatus (Rdev) can be determined by multiplying the rotation matrix of the real camera (Rl) onto the inverse of the matrix representing the orientation of the real camera relative to the orientation of the multi-directional image capture apparatus (Rdev−1).

It will be appreciated that the above described methods are not limited to a matrix representation of orientation. The orientation of the multi-directional image capture apparatus may also be calculated using Euler angle or quaternion representations of orientation. These representations are known in the art and will not be described in detail here.

It will be further appreciated that the above described process may be performed for each real camera11of the multi-directional image capture apparatus10to determine the position and/or orientation of each of the real cameras11of the multi-directional image capture apparatus10.

The position of the multi-directional image capture apparatus10may then be determined in the following way. The position of the real camera relative to the multi-directional image capture apparatus10may be known, and may be represented by vector vldev. However, vldevis relative to a local coordinate system of the multi-directional image capture apparatus. To obtain the position of the real camera11relative to the multi-directional image capture apparatus10(relative the reference coordinate system), vldevmay be rotated according to:
vldevw=Rdevvldev

Where Rdevis the final rotation matrix of the multi-directional image capture apparatus10as determined above, and vwldevis a vector representing the position of the real camera11relative to the multi-directional image capture apparatus10(and relative the reference coordinate system). As such, the position of the real camera11relative to its corresponding multi-directional image capture apparatus (relative the reference coordinate system) may be determined by multiplying the rotation matrix of the multi-directional image capture apparatus10onto the position of the real camera relative to the multi-directional image capture apparatus in the local coordinate system of the multi-directional image capture apparatus.

Therefore, the position of the multi-directional image capture apparatus10may be determined according to:
cdev=cl−vldevw

Where Clrepresents the position vector of the real camera as determined above, vwldevrepresents the position of the real camera relative to the multi-directional image capture apparatus as determined above, and cdevis the position vector of the multi-directional image capture apparatus10in the reference coordinate system. Put another way, a position of the multi-directional image capture apparatus10may be determined by taking the difference between the position vector of the real camera11and the position vector of the real camera relative to the multi-directional image capture apparatus.

Finally, the determined position and/or orientation of the multi-directional image capture apparatus10may be saved in a camera position and orientation database (seeFIG. 5B). The camera position and orientation database may include information describing the position and orientation each of the real cameras of the multi-directional image capture apparatuses and/or the position and orientation of the multi-directional image capture apparatuses themselves. In some examples, the information in the camera position and orientation database may instead be included in the 3D reconstruction database as part of the 3D model. In such examples, the 3D model may be said to be updated based on the determined position of the multi-directional image capture apparatus. In some examples, the 3D reconstruction database and/or the camera position and orientation database may be located at an external server or a computing device coupled with one or more of the multi-directional image capture apparatuses by a wireless or wired connection.

It will be appreciated that the above described methods may be repeated to determine future positions of the multi-directional image capture apparatus. It will also be appreciated that the above described methods may be performed for a plurality of multi-directional image capture apparatuses. For example, if more than one of the plurality of multi-directional image capture apparatuses10ofFIG. 1are moved then the above methods may be performed for all of the multi-directional image capture apparatuses which are moved. Similarly, if more than one multi-directional image capture apparatuses is added to the arrangement, then the above methods may be performed for all of the multi-directional image capture apparatuses added to the arrangement. As such, the positions and orientations of multi-directional image capture apparatuses in the arrangement may be kept up to date.

FIG. 5Ais a flowchart showing examples of operations as described herein.

At operation5.1, a plurality of first images21which are captured by a multi-directional image capture apparatus10may be received. For example, image data corresponding to the first images21may be received at computing apparatus60(seeFIG. 6).

At operation5.2, image re-projection may be performed on each of the first images21to obtain one or more re-projected second images22corresponding to respective virtual cameras.

At operation5.3, the second images22may be processed based on a virtual 3D model to obtain positions and orientations of the virtual cameras. For example, the second images22may be processed using a Direct Linear Transform method.

At operation5.4, positions and orientations of a real camera may be determined based on the positions and orientations of the virtual cameras determined at operation5.3.

At operation5.5, the position and orientation of the multi-directional image capture apparatus may be determined based on the position and orientation of the real camera determined at operation5.4.

At operation5.6, the camera position and orientation database may be updated with the position and orientation of the multi-directional image capture apparatus determined at operation5.5.

It will be appreciated that the position of a real camera11as described herein may be the position of the centre of a lens of the real camera11. Similarly, the position of a virtual camera may be the position of the centre of a virtual lens of the virtual camera. The position of the multi-directional image capture apparatus10may be the centre of the multi-directional image capture apparatus (e.g. if a multi-directional image capture apparatus is spherically shaped, its position may be defined as the geometric centre of the sphere).

FIG. 5Bis a flowchart showing examples of operations and interactions between various functions and databases described herein.

Operations S5.11to S5.14relate to the initial determination of the positions and orientations of a plurality of multi-directional image capture apparatus, for example as described in UK patent applications GB1620312.7 and GB1702680.8.

At operation S5.11, images captured by a plurality of multi-directional image capture apparatus are received.

At operation S5.12, the capture images are processed to generated re-projected images associated with virtual cameras.

At operation5.13, 3D reconstruction is performed by processing the re-projected images. This may be performed using a SfM technique, as described above. As such, a 3D model of the scene captured by the images may be obtained. As illustrated, information describing the scene from the 3D model may be saved to a 3D reconstruction database. Such information may include a 3D point cloud and/or information describing image features.

At operation5.14, the position and orientation of the cameras of the multi-directional image capture apparatuses may be determined. As illustrated, this may be saved to a camera position and orientation database.

Operations S5.21to S5.24may be performed when the arrangement of the multi-directional image capture apparatuses changes as described above with reference toFIGS. 2A and 2B. In this situation, the position and orientation of the cameras of the multi-directional image capture apparatus which caused the change in arrangement (e.g. the one which has moved or the newly added one) may need to be determined.

At operation S5.21, the existing 3D model may be loaded. This may be the 3D model previously generated and saved as a result of operations5.11to5.13. As illustrated, this may be performed by reading the necessary information from the 3D reconstruction database.

At operation S5.22, the multi-directional image capture apparatus may generate new re-projected images associated with virtual cameras based on new images captured by the multi-directional image capture apparatus which caused the change in arrangement. This operation may be performed in the same or a similar way to operation S5.2ofFIG. 5A.

At operation S5.23, the new position and orientation of the virtual cameras may be determined. This operation may be performed in the same or a similar way to operation S5.3ofFIG. 5A. As illustrated, this information may be saved to the 3D reconstruction database to update the 3D model.

At operation S5.24the new positions and orientations of the real cameras may be determined. This operation may be performed in the same or a similar way to operation S5.4ofFIG. 5A. As illustrated, this may be performed using information read from the camera position and orientation database. Also as illustrated, the newly determined positions and orientations may be saved to the camera position and orientation database to update the camera position and orientation database.

It will be appreciated that operations S5.21and5.24may be repeated if the arrangement of the multi-directional image capture apparatuses changes again (for instance, if the apparatus is moving within the environment), in order to continue to update the 3D reconstruction database and the camera position and orientation database. In some examples, operations S5.21to S5.24may be performed periodically or in response to the detection of a change. For example, operations S5.21to S5.24may be performed in response to movement of a multi-directional image capture apparatus or the appearance of a new multi-directional image capture apparatus in the system.

FIG. 6is a schematic block diagram of an example configuration of computing apparatus60, which may be configured to perform any of or any combination of the operations described herein. The computing apparatus60may comprise memory61, processing circuitry62, an input63, and an output64.

The processing circuitry62may be of any suitable composition and may include one or more processors62A of any suitable type or suitable combination of types. For example, the processing circuitry62may be a programmable processor that interprets computer program instructions and processes data. The processing circuitry62may include plural programmable processors. Alternatively, the processing circuitry62may be, for example, programmable hardware with embedded firmware. The processing circuitry62may be termed processing means. The processing circuitry62may alternatively or additionally include one or more Application Specific Integrated Circuits (ASICs). In some instances, processing circuitry62may be referred to as computing apparatus.

The processing circuitry62described with reference toFIG. 6is coupled to the memory61(or one or more storage devices) and is operable to read/write data to/from the memory. The memory61may store thereon computer readable instructions612A which, when executed by the processing circuitry62, may cause any one of or any combination of the operations described herein to be performed. In some examples, various other data, such as the 3D reconstruction database and/or camera position and orientation database ofFIG. 5B, may also be stored in the memory61.

The memory61may comprise a single memory unit or a plurality of memory units upon which the computer-readable instructions (or code)612A is stored. For example, the memory61may comprise both volatile memory611and non-volatile memory612. For example, the computer readable instructions612A may be stored in the non-volatile memory612and may be executed by the processing circuitry62using the volatile memory611for temporary storage of data or data and instructions. Examples of volatile memory include RAM, DRAM, and SDRAM etc. Examples of non-volatile memory include ROM, PROM, EEPROM, flash memory, optical storage, magnetic storage, etc. The memories61in general may be referred to as non-transitory computer readable memory media.

The input63may be configured to receive image data representing the first images21described herein. The image data may be received, for instance, from the multi-directional image capture apparatus10itself or may be received from a storage device. The output may be configured to output any of or any combination of the camera pose registration information described herein. The structural elements ofFIG. 6represent examples of means for performing one or more or any combination of the operations described herein. For example, computing apparatus60may comprise means for performing one or more steps of the methods as described in the claims and throughout the specification.

FIG. 7illustrates an example of a computer-readable medium70with computer-readable instructions (code) stored thereon. The computer-readable instructions (code), when executed by a processor, may cause any one of or any combination of the operations described above to be performed.

Reference to, where relevant, “computer-readable storage medium”, “computer program product”, “tangibly embodied computer program” etc., or a “processor” or “processing circuitry” etc. should be understood to encompass not only computers having differing architectures such as single/multi-processor architectures and sequencers/parallel architectures, but also specialised circuits such as field programmable gate arrays FPGA, application specify circuits ASIC, signal processing devices and other devices. References to computer program, instructions, code etc. should be understood to express software for a programmable processor firmware such as the programmable content of a hardware device as instructions for a processor or configured or configuration settings for a fixed function device, gate array, programmable logic device, etc.

If desired, the different functions discussed herein may be performed in a different order and/or concurrently with each other. Furthermore, if desired, one or more of the above-described functions may be optional or may be combined. Similarly, it will also be appreciated that the flow diagram ofFIG. 5is an example only and that various operations depicted therein may be omitted, reordered and/or combined. For example, it will be appreciated that operation S5.6as illustrated inFIG. 5may be omitted.

It is also noted herein that while the above describes various examples, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications which may be made without departing from the scope of the present invention as defined in the appended claims.