Patent Description:
<NPL>, describes background art concerning geometric camera calibration of central cameras. The techniques for geometric camera calibration of central cameras are incorporated into computer vision libraries such as OpenCV for example.

The premise behind geometric camera calibration for a central camera is that a light ray travels from an object point in three-dimensional real space through the camera optics and a pin hole aperture to an image point in a two-dimensional surface defined by an image sensor. The camera geometry determines how the light rays are mapped from the object points to the image points and geometric camera calibration discovers the mapping.

<CIT> discloses a method for obtaining parameters defining a pixel beam associated with a pixel of an image sensor comprised in an optical device. Calibration is performed using the OpenCV library.

<CIT> discloses a method and apparatus for performing a single view depth and texture calibration. In one embodiment, the apparatus comprises a calibration unit operable to perform a single view calibration process using a captured single view a target having a plurality of plane geometries having detectable features and being at a single orientation and to generate calibration parameters to calibrate one or more of the projector and multiple cameras using the single view of the target.

According to various, but not necessarily all, embodiments of the invention there is provided examples as claimed in the appended claims.

The premise behind geometric camera calibration is that a light ray travels from an object point in three-dimensional real space through the camera optics and aperture to an image point in a two-dimensional surface defined by an image sensor. The camera geometry determines how the light rays are mapped from the object points to the image points and geometric camera calibration discovers the mapping.

The central camera model assumes that the camera aperture is a pin hole. This assumption is not always valid.

The central camera/pin hole aperture assumption may introduce errors. The central camera/pin hole aperture assumption may introduce errors for wide-field of view cameras such as fisheye cameras.

We will use a non-central camera model that assumes a larger aperture (a larger entrance pupil) and the calibration process will account for this larger, non point-like aperture.

Let us consider a mapping M that maps object points at R to an image point at r: <MAT>.

R is a three-dimensional vector (X, Y, Z) positioning an object point in real space. It may be expressed, for example, as [X, Y, Z, <NUM>]T.

r is a two-dimensional vector (x, y). It may, for example, be expressed as [x, y, <NUM>]T. It positions an image point in an image plane of the camera.

M is a mapping that maps the vector R to the vector r. The image point r is the projection of the object point R through the mapping M.

As will be explained below, the mapping will allow light rays travelling from object points to image points to cross the optical axis at different locations.

The mapping M may be expressed as a series of transforms.

The transform E is an extrinsic transform. The transform D is a distortion transform. The transform A is an intrinsic transform.

The transform E is an extrinsic transform. It is typically a joint rotation-translation transform of the form [R,t] where R is a three dimensional rotation matrix and t is a translation vector.

Thus E may have the form <MAT> let <MAT>.

The extrinsic transform E is the same linear mapping between object points and image points in a captured image. It accounts for relative movement of the imaged scene and camera between image captures or, if two images are captured simultaneously from different image sensors the different perspective of the image sensors.

The distortion transform D maps <MAT> to <MAT> <MAT> where.

The mapping X(θ) is a distortion model. It may model radial distortion and/or tangential distortion. However, radial distortion is typically much greater than tangential distortion.

Without loss of generality, the example below will use radial distortion only. One example form of X(θ) is: <MAT> where u is a function of θ. u= θ for an equidistance, also known as equiangular, projection model.

Other projection models may be used. For example:.

Without loss of generality, the example below will use radial distortion and an equiangular projection model.

The intrinsic transform A represents the camera matrix. It comprises in combination of at least a two-dimensional translation matrix <MAT>,where (cx, cy) is the principal point which is normally the image center, and a two-dimensional scaling matrix <MAT>, where fx and fy are scale factors.

In some examples, the intrinsic transform A may also comprise a two-dimensional skew matrix <MAT> where γ is a measure of skewness.

Without loss of generality, the example below will use an intrinsic transform A based on translation and scaling only, <MAT>.

The intrinsic transform A maps <MAT> to <MAT> <MAT>.

Thus considering a single image <MAT> <MAT>.

For the equidistance (also known as equiangular) projection model X(θ) is <MAT>.

It should be noted that other forms of X(θ) are possible. It may, for example, be combined with the scaling matrix S of the intrinsic transform to create a polynomial in terms of r (r<NUM>= x<NUM> + y<NUM> ) and distortion parameters {ki'} instead of a polynomial in terms of θ and distortion parameters {ki}.

In other examples, X(θ) may be expressed as a quotient of polynomials. e.g. <MAT>.

Finally <MAT> is adapted to take account of the NON-CENTRAL camera aperture.

The non-central characteristic of the camera is modelled in the geometric calibration and transform M as an offset ΔZ(R) for each object point R.

An object point R is positioned in three-dimensions by a displacement (Z) along the optical axis and a displacement (X, Y) from the optical axis. The new mapping M transforms displacement (Z) along the optical axis by adding an offset ΔZ defined by a parametric function. The parametric function specifies a viewpoint location, on the optical axis via an entrance pupil, of the object point. A viewpoint is where a light ray travelling from the object point to image point crosses the optical axis.

The camera is symmetric about the optical axis and there the offset ΔZ(R) may be expressed in terms of the off-axis angle θ of incident light relative to the optical axis. The parametric function may therefore specify variation of an entrance pupil position along an optical axis with off-axis angle θ of incident light.

For example, ΔZ=V(θ) <MAT> i.e. <MAT> where
<MAT>, the maximum angle of a light ray from the object point (X, Y, Z) through the aperture measured relative to the optical axis (X=<NUM>, Y=<NUM>).

The parametric function V(θ) may be a non-linear function of the angle θ that the light ray travelling from an object point towards an image point makes with the optical axis as it crosses the optical axis.

The geometric calibration of the camera now involves not only the determination of the intrinsic parameters e.g. scaling parameters fx, fy and translation parameters cx, cy , the distortion transform X(θ) but also the viewpoint offset V(θ).

The geometric calibration starts with using the camera to capture an image of a scene. In one example, the geometric calibration starts with using the camera to capture an image of a two-dimensional calibration pattern. In some but not necessarily all examples, the calibration pattern comprises N easily detectable features fi at known relative positions Ri in the 2D plane. This creates an object set of vectors { Ri }.

Calibration comprises determining the intrinsic parameters e.g. scaling parameters fx, fy and translation parameters cx, cy , the distortion transform X(θ) and the viewpoint offset V(θ).

Optionally, the calibration for the non-central camera may be considered as a 'perturbation' from the central camera model. In this approach, the calibration is first carried out assuming a central camera model to give an initial estimate of some of the intrinsic parameters of the central camera and then calibration is continued using the estimated intrinsic parameters of the central camera model as the initial conditions for calibration using the non-central camera model.

For example, the calibration is first carried out assuming a central camera model to give an initial estimate of some of the intrinsic parameters of the central camera by: assuming a pin hole V(θ) = <NUM>.

Then <MAT> reduces to r = A R where A is a constant for all N of the features fi. <MAT> <MAT> where
[R<NUM>; R<NUM>;. RN]-<NUM> is the right pseudo inverse of [R<NUM>; R<NUM>;.

The problem may now be solved using linear algebra. For example the standard camera calibration algorithms of OpenCV may be used or the pinv() function in Octave, for example.

This provides an initial estimate of some of the intrinsic parameters of the central camera. The calibration is continued using the estimated intrinsic parameters of the central camera model as the initial conditions for calibration using the non-central camera model.

Alternatively the calibration using the non-central camera model may be performed using different initial conditions, for example standard initial conditions or randomly selected initial conditions.

By setting a generalized parametric form for X(θ) and the viewpoint offset V(θ) and assumed initial conditions, then a putative transformation <MAT> can be determined for each mapping from object point Oi to image point Ii.

The distortion transform X(θ) may be expressed as a polynomial in θ , for example an odd polynomial in θ (see above) <MAT> where {ki} are the distortion parameters.

The viewpoint offset V(θ) may be expressed as a non-linear function in θ , for example, a polynomial quotient in θ <MAT> where {qi} are the viewpoint offset parameters.

The polynomials may be expressed to higher orders. V(θ) may be expressed as different parametric functions using viewpoint offset parameters {qi}.

r(p)i can be determined from Ri for each I using non-linear optimization.

For example, a suitable cumulative cost function J can be specified using, for example the mean of the squared distance between a putative image point position r(p)i of feature fi and the actual measured image point position r(m)i of feature fi.

The parameters of M may be found by finding the parameters that minimize J. Thus the distortion parameters {ki} and the viewpoint offset parameters {qi} may be determined. This may also update the other intrinsic parameters e.g. scaling parameters fx, fy and translation parameters cx, cy.

This non-linear optimization may for example be performed using linear/polynomial regression or neural networks or some other non-linear optimization technique.

It will be understood by those skilled in the art that various modifications and adjustments may be desirable to achieve convergence or achieve faster convergence such as, for example, feature scaling and learning rate adjustment.

The process can be repeated for different aperture (stop) values of the camera.

The mapping M has therefore been calibrated.

<FIG> illustrates an example of a method of calibrating a camera by calibrating a parametric mapping that maps between object points and image points.

An object point is positioned in three-dimensions by a displacement (Z) along the optical axis and a displacement (X, Y) from the optical axis the mapping transforms displacement (Z) along the optical axis by adding an offset defined by a parametric function. Thus the mapping uses a parametric function to specify a viewpoint location, on the optical axis via an entrance pupil, of an object point. The viewpoints are where light rays travelling from object points to image points cross the optical axis. The parametric function specifies variation of an entrance pupil position along an optical axis with off-axis angle of incident light. The parametric function comprises a non-linear function of the angle the light ray travelling from an object point towards an image point makes with the optical axis as it crosses the optical axis.

The calibrated mapping M for a non-central camera can be used in the same or similar applications as a calibrated mapping M for a camera. It is however better for a larger range of cameras. It is better for cameras with large fields of view e.g. having a field of view greater than <NUM>° or greater than <NUM>° or <NUM>°.

Examples of suitable applications include:.

<FIG> illustrates a method <NUM> comprising:.

<FIG> illustrates an example of a camera <NUM>.

The camera <NUM> is a non-central camera. As explained above, a locus of viewpoints is specified by V(θ).

The camera comprises an image sensor <NUM>, optics <NUM>, memory <NUM>, controller <NUM> and user interface <NUM>.

The optics <NUM> define an aperture <NUM> (entrance pupil) of finite dimension (not a point, not central). In some but not necessarily all examples, the optics are dioptric (refractive only). In this example, In the optics are comprise one or more refractive lenses. In this example the optics <NUM> provide a large field of view (wide-angle). In this example, the optics provides a field of view greater than <NUM>°, or a field of view greater than <NUM>° or <NUM>°. the optics <NUM> may be a fisheye lens.

In some but not necessarily all examples, the aperture <NUM> is variable is size. The entrance pupil is therefore variable in size.

Implementation of a controller <NUM> may be as controller circuitry. The controller <NUM> may be implemented in hardware alone, have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).

In some but not necessarily all examples, an apparatus may comprise multiple cameras. In this case, the above described intrinsic calibration process is performed for each camera separately and then an extrinsic calibration process if performed to calibrate the multiple camera. Each camera comprises an image sensor <NUM> and optics <NUM>. The apparatus comprises in addition to the multiple cameras a memory <NUM>, controller <NUM> and user interface <NUM> shared by the camera. There is a different extrinsic transform E for the different cameras. The relative movement of part of an imaged scene captured from the multiple different image sensors, while the cameras are stationary, enables determination of the different extrinsic transform E for the multiple different cameras as part of an extrinsic calibration process after intrinsic calibration of each camera.

The memory <NUM> stores a computer program <NUM> comprising computer program instructions (computer program code) that controls the operation of the apparatus <NUM> when loaded into the processor <NUM>. The computer program instructions, of the computer program <NUM>, provide the logic and routines that enables the apparatus to perform the methods illustrated in <FIG>. The processor <NUM> by reading the memory <NUM> is able to load and execute the computer program <NUM>.

As illustrated in <FIG>, the computer program <NUM> may arrive at the apparatus <NUM> via any suitable delivery mechanism <NUM>. The delivery mechanism <NUM> may be, for example, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), an article of manufacture that tangibly embodies the computer program <NUM>. The delivery mechanism may be a signal configured to reliably transfer the computer program <NUM>. The apparatus <NUM> may propagate or transmit the computer program <NUM> as a computer data signal.

As used in this application, the term 'circuitry' refers to all of the following:.

This definition of 'circuitry' applies to all uses of this term in this application, including in any claims. The term "circuitry" would also cover, for example and if applicable to the particular claim element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or other network device.

The blocks illustrated in the <FIG> may represent steps in a method and/or sections of code in the computer program <NUM>. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some blocks to be omitted.

In some but not necessarily all examples, the controller <NUM> is configured to vary the aperture <NUM>.

In some but not necessarily all examples, the memory <NUM> stores one or more data structures <NUM> storing the parameters of the parametric model determined by the geometric calibration- the intrinsic parameters e.g. scaling parameters fx, fy and translation parameters cx, cy , the distortion transform X(θ) but also the viewpoint offset V(θ).

In some but not necessarily all examples, the apparatus <NUM> is configured to communicate data from the apparatus <NUM> with or without local storage of the data in a memory <NUM>, <NUM> at the apparatus and with or without local processing of the data by circuitry or processors at the apparatus <NUM>.

The data may be stored in processed or unprocessed format remotely at one or more devices. The data may be stored in The Cloud.

The data may be processed remotely at one or more devices. The data may be partially processed locally and partially processed remotely at one or more devices.

The data may be communicated to the remote devices wirelessly via short range radio communications such as Wi-Fi or Bluetooth, for example, or over long range cellular radio links. The apparatus may comprise a communications interface such as, for example, a radio transceiver for communication of data.

The apparatus <NUM> may be part of the Internet of Things forming part of a larger, distributed network.

The recording of data may comprise only temporary recording, or it may comprise permanent recording or it may comprise both temporary recording and permanent recording. Temporary recording implies the recording of data temporarily. This may, for example, occur during sensing or image capture, occur at a dynamic memory, occur at a buffer such as a circular buffer, a register, a cache or similar. Permanent recording implies that the data is in the form of an addressable data structure that is retrievable from an addressable memory space of the memory <NUM> and can therefore be stored and retrieved until deleted or over-written, although long-term storage may or may not occur. The use of the term 'capture' in relation to an image relates to temporary recording of the data of the image. The use of the term 'store' in relation to an image relates to permanent recording of the data of the image.

In this brief description, reference has been made to various examples. The use of the term 'example' or 'for example' or 'may' in the text denotes, whether explicitly stated or not, that such features or functions are present in at least the described example, whether described as an example or not, and that they can be, but are not necessarily, present in some of or all other examples. Thus 'example', 'for example' or 'may' refers to a particular instance in a class of examples. It is therefore implicitly disclosed that a feature described with reference to one example but not with reference to another example, can where possible be used in that other example but does not necessarily have to be used in that other example.

Claim 1:
An apparatus (<NUM>) comprising means for calibrating a parametric mapping that maps between object points and image points, said means comprising means for:
capturing an image of a calibration pattern comprising features defining object points; determining, from the image, measured image points that correspond to the object points;
determining, from the mapping, putative image points that correspond to the object points;
characterized by:
minimizing a cumulative cost function dependent upon differences between the measured image points and putative image points to determine parameters of the parametric mapping,
wherein the mapping uses a parametric function to specify points where light rays travelling from object points to image points cross the optical axis.