Patent Description:
In the automotive sector, vision based driver assistance systems (ADAS) are emerging for various applications such as parking assistance, sideline warnings and collision detection. For parking assistance surround view, for example, four cameras can be used to provide a <NUM> degree view of an environment of a vehicle. For collision and sideline detection stereo cameras can be used which are configured to estimate distances and speed of objects in the environment.

For driving autonomy, the vehicle can be enabled to estimate its own path of driving and recognize its three-dimensional (3D) environment while driving. For this, applications may need to provide an appropriate spatial accuracy. Therefore, for example, cameras need to be calibrated and rigidly mounted such that the 3D environment can be estimated from a relative position and orientation of the cameras. In case of stereo view, a pre-calibration can be done in a factory or workshop.

A distance between the cameras is limited to a maximum distance as the rigidly mounted cameras may experience calibration issues due to torsion, vibrations or relative motion of the cameras. Thus, mounting the cameras rigidly to a chassis of the vehicle can be inappropriate for autonomous driving applications as motor vibrations and chassis torsion may affect the distance and orientation of cameras.

<NPL>" propose an end-to-end 3D reconstruction system which combines color, depth and inertial measurements to achieve robust reconstruction with fast sensor motions. IGNACIO <NPL>" describe a full solution to the estimation of the global position of a vehicle in a digital road map by means of visual information alone. The solution is based on a stereo platform used to estimate the motion trajectory of the ego vehicle and a map-matching algorithm, which will correct the cumulative errors of the vision-based motion information and estimate the global position of the vehicle in a digital road map. <NPL>" propose exploiting gyroscope measurements, angular velocity, along with image measurement to compute the relative pose between rolling shutter cameras. <NPL>" present a minimal case solution to the calibrated relative pose problem using <NUM> point correspondences for the case of two known orientation angles. This case is relevant when a camera is coupled with an inertial measurement unit (IMU). <CIT> describes an in-vehicle apparatus including an image-capturing part that is installed on a vehicle and captures an image of a periphery of the vehicle. An image generator generates a bird's-eye image including the vehicle and the periphery of the vehicle viewed from a virtual viewpoint based on the image captured by the image-capturing part. A transmitter transmits the bird's-eye image to a terminal. The image generator changes the virtual viewpoint based on a change request from the terminal and generates the bird's-eye image viewed from the virtual viewpoint after changing the virtual viewpoint.

Hence, there may be a demand for an improved concept to determine a pose of a camera.

This demand can be satisfied by camera systems and methods in accordance with the appended independent and dependent claims.

According to a first aspect not part of the invention defined in the claims, the present disclosure relates to a camera system for a mobile device. The camera system comprises at least one camera which is freely mounted to the mobile device. Further, the camera system comprises at least one motion measurement unit configured to provide motion data of the camera and a data processing circuitry configured to determine a pose of the camera from the motion data.

The mobile device is, for example, a vehicle (e.g. a car, a truck, a bus, a boat, a plane, a motorcycle, a bicycle or an unmanned aerial vehicle) or a handheld device.

The camera can be freely mounted to the mobile device using a damping plate or a (camera) stabilizer to at least partly reduce, compensate or absorb torsion and/or vibrations which may emanate from the mobile device.

The stabilizer, for example, comprises a pivoted support (e.g. a gimbal) which allows a rotation of the camera about at least one axis to mount the camera to the mobile device in a floating position.

The damping plate can be at least partially made of an elastic material to mount the camera freely (elastically) to the mobile device.

Thus, the camera may be considered as loosely coupled or freely or floating mounted to the mobile device.

The motion data, for example, is indicative of an acceleration and/or an angular velocity of the camera. The angular velocity and/or the acceleration for example can be measured by a gyroscope comprised of the motion measurement unit.

Since the camera is freely mounted to the mobile device, the camera may change its pose. The pose can be understood as a relative position and/or orientation within an environment to the mobile device.

The camera, for example, can be subjected to the angular velocity while the mobile device is moving. The data processing unit can determine the pose of the camera, for example, from a relationship between an angular velocity and the pose of the camera.

This may enable a so-called auto-calibration of the camera system with regard to the pose of the camera, making a pre-calibration or a re-calibration (e.g. at a factory or a workshop) of the camera system obsolete.

According to a second aspect not part of the invention defined in the claims, the present disclosure relates to a method for localizing a camera of a camera system for a mobile device. The method comprises providing motion data of the camera, which is freely mounted to the mobile device, using at least one motion measurement unit. The method further comprises determining a pose of the camera from the motion data.

The method, for example, can be executed using the aforementioned camera system. According to a third aspect, the present disclosure relates to a camera system comprising a first camera configured to provide a first sequence of images of an environment; a first motion measurement unit configured to provide first motion data of the first camera;at least one second camera configured to provide a second sequence of images of the environment; a second motion measurement unit configured to provide second motion data of the second camera; and a data processing circuitry configured to determine a relative pose of the first camera and the second camera towards each other from a correlation between the first motion data and the second motion data, determine a position of a target within the environment from a correlation between the first sequence of images and the first motion data; register the position of the target in a digital map of the environment; and detect the target within the second sequence of images using the digital map, the motion data, and the relative pose of the first and the second camera towards each other.

The first and the second camera can be a video/movie camera.

The first and the second motion data, for example, are indicative of an acceleration and/or an angular velocity of the first and the second camera, respectively. The first and the second motion measurement unit, for example, each comprise a gyroscope for measuring the angular velocity or the acceleration acting on the first and the second camera, respectively.

Due to a different pose of the first and the second camera, the first and the second motion data may be different from each other. Further, the first and the second motion data can correlate in such a way that, for example, a distinct difference between the first and the second motion data is indicative of a distinct relative pose to each other.

This may enable an auto-calibration of the camera system to adapt the camera system, for example, to a varying relative pose of the first and the second camera towards each other.

The relative pose, for example, rapidly changes due to vibrations acting on the first and/or the second camera. Alternatively and/or additionally the relative pose may change due to linear thermal expansion with a variation of an ambient temperature.

The auto-calibration may ensure an appropriate spatial accuracy of the camera system which may be required for applications in connection with autonomous driving vehicles.

According to a fourth aspect, the present disclosure relates to a method, comprising providing a first sequence of images of an environment with a first camera; providing first motion data of the first camera; providing a second sequence of images of the environment with a second camera; providing second motion data of the second camera; and determining a relative pose of the first camera and the second camera towards each other from a correlation between the first motion data, the second motion data the first sequence of images, and the second sequence of images, determining a position of a target within the environment from a correlation between the first sequence of images and the first motion data; registering the position of the target in a digital map of the environment; and detecting the target within the second sequence of images using the digital map, the motion data and the relative pose of the first and the second camera.

This method, for example, can be executed using the aforementioned camera system comprising the first and the second camera.

According to a fifth aspect, the present disclosure relates to a computer program comprising instructions which, when executed by at least one processor, causes the processor to perform the method of the fourth aspect.

According to a sixth aspect, the present disclosure relates to a mobile device which comprises the system of the third aspect.

The mobile device can be a vehicle or a handheld device. In particular, the mobile device can be an autonomously driving vehicle.

According to a seventh aspect not part of the invention defined in the claims, the present disclosure relates to a distributed camera system comprising a first camera configured to provide a first sequence of images of an environment a first motion measurement unit configured to provide first motion data of the first camera and a first data processing circuitry. The first data processing circuitry is configured to determine a position of target within the environment from a correlation between the first motion data and the first sequence of images register the position of the target in a digital map of the environment. Further, the first data processing circuitry is configured to provide the digital map of the environment.

The distributed camera system further comprises at least one second camera configured to provide a second sequence of images of an environment, a second motion measurement unit configured to provide second motion data of the second camera and a second data processing circuitry. The second data processing circuitry is configured to determine a relative pose of the first camera and the second camera towards each other from a correlation between the first motion data and the second motion data. The second data processing circuitry is further configured to receive the digital map of the environment and detect the target within at least one of the images of the second sequence of images based on and the digital map and the relative pose of the first camera and the second camera towards each other.

The distributed camera system can be understood as an apparatus with multiple camera system each comprising a camera, a motion measurement unit and a data processing circuitry.

In some embodiments, the first camera, the first motion measurement unit and the first data processing circuitry may be mounted to a first mobile device and the second camera, the second motion measurement unit and the second data processing circuitry may be mounted to a second mobile device.

In some applications it may be desired to track the target with multiple cameras. Thus, it can be useful to identify the target within the second sequence of images.

For this, the digital map can be shared within the first and the second data processing circuitry. The second data processing circuitry thus can estimate a position of the target within one or more images of the second sequence of images by reference to the digital map and the relative pose of the first and the second camera.

The skilled person having benefit from the present disclosure will appreciate that this may be desired for tracking purposes in some applications in which the first and the second camera have different and/or opposite fields of view.

Accordingly, while further examples are capable of various modifications and alternative forms, some particular examples thereof are shown in the figures and will subsequently be described in detail. However, this detailed description does not limit further examples to the particular forms described. Same or like numbers refer to like or similar elements throughout the description of the figures, which may be implemented identically or in modified form when compared to one another while providing for the same or a similar functionality.

It will be understood that when an element is referred to as being "connected" or "coupled" to another element, the elements may be directly connected or coupled via one or more intervening elements. If two elements A and B are combined using an "or", this is to be understood to disclose all possible combinations, i.e. only A, only B as well as A and B, if not explicitly or implicitly defined otherwise. An alternative wording for the same combinations is "at least one of A and B" or "A and/or B". The same applies, mutatis mutandis, for combinations of more than two Elements.

For purposes of autonomous driving, one or more cameras can be utilized to characterize an environment of a vehicle. For this, camera systems can be pre-calibrated to a pose of the cameras. Since those can be mounted to the vehicle, the cameras may be subjected to torsion and/or vibrations which may lead to calibration issues and in some cases to such a characterization of the environment which is inappropriate for autonomous driving purposes.

Hence, there may be a demand for an improved concept for determining a pose of a camera.

<FIG> shows a schematic example of a camera system <NUM> which is not part of the invention defined in the claims. The camera system <NUM> comprises a camera <NUM> and a motion measurement unit <NUM> configured to provide motion data of the camera <NUM>. Further, the camera system <NUM> comprises a data processing circuitry <NUM> configured to determine a pose of the camera <NUM> from the motion data.

The camera <NUM> is freely mounted to a mobile device <NUM>. As can be seen in <FIG>, the camera <NUM> can be freely mounted to the mobile device <NUM> by a stabilizing mounting <NUM> for a floating mounting.

The stabilizing mounting <NUM> can at least partly absorb vibrations coming from the mobile device <NUM>. For example, the stabilizing mounting <NUM> comprises a camera stabilizer (e.g. a gimbal) and/or an elastic mounting, such as a damping plate.

The motion measurement unit <NUM> can comprise an inertial measurement unit (IMU) which is rigidly mounted to the camera <NUM>. The skilled person having benefit from the present disclosure will appreciate that the IMU can provide a portion of the motion data using a combination of one or more accelerometers, gyroscopes, and/or magnetometers. The motion data from the IMU, thus, may be indicative of an acceleration and/or an angular velocity of the camera <NUM>.

Alternatively or additionally, the motion measurement unit <NUM> can comprise a global positioning system (GPS) sensor which is installed at the mobile device <NUM> and configured to provide at least a portion of the motion data. The GPS sensor can provide a geographical position, a velocity of the mobile device <NUM> or a course of the geographical position which may contribute to the motion data.

The motion measurement unit <NUM> may comprise both the IMU and the GPS sensor for a higher operational reliability than each of the IMU and the GPS sensor.

The motion measurement unit <NUM> is coupled to the data processing circuitry <NUM> to communicate the motion data.

Due to a relationship between the pose of the camera <NUM> and the motion data, the data processing circuitry <NUM> can determine the pose of the camera <NUM>.

For example, the camera <NUM> can be displaced from a predefined position by an angular velocity. A displacement of the camera <NUM> from the predefined position may depend on an orientation and an absolute value of the angular velocity. The data processing circuitry <NUM>, for example, is calibrated to determine the displacement and the pose of the camera <NUM> from the angular velocity included in the motion data.

This may enable an auto-calibration of the camera system <NUM> based on the pose of the camera <NUM> on the mobile device <NUM> to recover a spatial accuracy of a characterization of the environment using the camera system <NUM>.

In case of automotive applications, the mobile device <NUM> can be a vehicle. In particular, the mobile device <NUM> can be an autonomously driving vehicle functioning based on a characterization of the environment using the camera system <NUM>.

For example, the autonomously driving vehicle may move on public roads, take evasive maneuvers or output a warning based on the characterization of the environment.

To improve an accuracy of the auto-calibration, the data processing circuitry <NUM> can further involve visual data for determining the pose of the camera <NUM>. For this, the camera <NUM> can be a video/movie camera which records a sequence of images of the environment.

To characterize its pose within the environment, the camera <NUM> can provide a sequence of images of the environment to the processing circuitry <NUM>. This, for example enables the data processing circuitry <NUM> to determine the pose from a correlation between the sequence of images and the motion data, as stated in more detail below.

For a communication of the sequence of images, the camera <NUM> is coupled with the data processing circuitry <NUM>.

As can be seen in <FIG>, the camera <NUM>, for example, provides a sequence of subsequent images of the environment. To simplify matters, a following description of a localization of the camera <NUM> refers to a first image <NUM>, a second image <NUM> and a third image <NUM> of the sequence of images. In general, the sequence of images can comprise more images and the concept described below can also be applied to more images.

Each of the images <NUM>, <NUM> and <NUM> includes an object <NUM> of the environment. Without limitation of generality and to simplify matters, the localization of the camera <NUM> is described by reference to the object <NUM>, whereas in further use cases the localization of the camera <NUM> may involve multiple objects.

A basic idea of the localization of the camera <NUM> is that the data processing circuitry <NUM> determines a change of perspective of the camera <NUM> by comparing the first image <NUM> of the sequence of images with the second image <NUM>.

The (absolute) pose of the camera <NUM> towards the object <NUM> can be derived from a correlation between the motion data and the change of perspective on the object <NUM>, as stated in more detail below.

The data processing circuitry <NUM>, for example, determines a first patch <NUM>-<NUM> within the first image <NUM>. As can be seen in <FIG>, the first patch <NUM>-<NUM> may correspond to a box including a distinct target <NUM>, such as a portion of a contour (e.g. a corner or an edge) of the object <NUM>.

The first patch <NUM>-<NUM> may cover "unique" so-called "photo-differences" of the first image <NUM>. If the images <NUM>, <NUM> and <NUM> are monochrome, the photo-differences, for example, are indicative of a characteristic distribution of greyscale values (greyscale distribution) of pixels included in the first patch <NUM>-<NUM>. Alternatively, the photo-differences can be understood of a distribution of color or contrast.

Subsequently, the data processing circuitry <NUM> can determine a second patch <NUM>-<NUM> indicative of the target <NUM> within the second image <NUM> by comparing the first patch <NUM>-<NUM> with the second image <NUM>.

The first and the second patch <NUM>-<NUM> and <NUM>-<NUM>, for example, are specified by their contour and by a position of their center within the first and the second image <NUM> and <NUM>.

The data processing circuitry <NUM> may arrange the second patch <NUM>-<NUM> within the second image <NUM> such that the second patch <NUM>-<NUM> includes the photo-differences of the first patch <NUM>-<NUM> (within predetermined tolerances). This may ensure that the second patch <NUM>-<NUM> relates to the same target <NUM>.

Due to a change of perspective, the second patch <NUM>-<NUM> may be shifted regarding to the first patch <NUM>-<NUM>. A shift between the first and the second patch <NUM>-<NUM> and <NUM>-<NUM> may imply so-called "affine correspondences" defining a transition between the subsequent patches <NUM>-<NUM> and <NUM>-<NUM>. Those may consider a rotation, a displacement, a warp and/or a shear of the patches <NUM>-<NUM> and <NUM>-<NUM>.

As can be seen in <FIG>, the second patch <NUM>-<NUM>, for example, corresponds to a trapezoid/parallelogram.

The data processing circuitry <NUM> can deduce the change of perspective of the camera <NUM> from the affine correspondences between the first and the second patch <NUM>-<NUM> and <NUM>-<NUM>. For example, the data processing circuitry <NUM> deduces the change of perspective, which, for example, is indicative of an angle, based on principles of homography.

The skilled person having benefit from the present disclosure will appreciate that the change of perspective may enable estimating a visually measurable pose of the camera <NUM> towards the target <NUM> in accordance with principles of visual odometry.

The visually measureable pose, for example, refers to a relative position or relative orientation of the camera <NUM> towards the target <NUM> with respect to other targets.

To obtain the (absolute) pose (e.g. indicative of coordinates or a measure of length) of the camera <NUM> towards the target <NUM>, the data processing circuitry <NUM> can provide a scale to deduce an absolute value for the pose of the camera <NUM> from the visually measureable pose of the camera <NUM>.

For this, the data processing circuitry <NUM> may associate the visual data with the motion data.

In some embodiments of the camera system <NUM>, the data processing circuitry <NUM> determines the pose of the camera <NUM> using a Kalman filter, wherein a control vector of the Kalman filter is specified by the motion data and a measured state is defined by the relative pose derived from to the visual data.

A relation between the motion data and the visual data may provide an observation model of the Kalman filter. With the observation model, the relative pose of the camera <NUM> can be scaled to obtain the pose of the camera <NUM> as an absolute value, for example, indicative of coordinates and/or an angle.

Thus, the visual data and the motion data both may have an influence in determining the pose of the camera <NUM>. Their influence may depend on a weighting factor indicative of an accuracy of the motion data and of the visual data, respectively to adapt the camera system <NUM> to different circumstances (e.g. a dark environment).

The above described concept for determining the pose of the camera <NUM> may also be referred to as "affine flow tracking".

Affine flow tracking may need the motion data and the sequence of images to be time-synchronized.

In some embodiments, the data processing circuitry can be configured to determine a third patch <NUM>-<NUM> indicative of the target <NUM> within a third image <NUM> following the second image <NUM> by comparing the first patch <NUM>-<NUM> with the third image <NUM>. The data processing circuitry <NUM> can determine the third patch <NUM>-<NUM> analogously to the second patch <NUM>-<NUM> by reference to the first patch <NUM>-<NUM>.

This may prevent the third patch <NUM>-<NUM> and subsequent patches from "drifting" away from the target <NUM> due to an error propagation and may provide subpixel accurate matching of subsequent patches with the target <NUM> for both temporal tracking and spatial mapping of subsequent patches by reference to subsequent images of the sequence of images.

Subsequently, the data processing circuitry <NUM> can determine the change of perspective from a shift/affine correspondences between the first and the third patch <NUM>-<NUM> and <NUM>-<NUM> to determine the pose of the camera <NUM>.

Since the third patch <NUM>-<NUM> can be considered as being mapped onto the first patch <NUM>-<NUM> for determining the pose of the camera <NUM>, this concept can be referred to as "affine flow mapping".

Additionally, the data processing circuitry <NUM> can determine a position of the target from the correlation between the change of perspective and the motion data.

For this, the pose of the camera <NUM> may be considered as an origin of a coordinate system. Consequently, the data processing circuitry <NUM> may determine the position of the target <NUM> within the coordinate system from the pose of the camera <NUM> towards the target <NUM>.

The position of the target <NUM> within the coordinate system, for example, is indicative of <NUM>-dimensional coordinates.

In some embodiments of the camera system <NUM>, the data processing circuitry <NUM> is further configured to determine a velocity of the target <NUM> by tracking the position of the target <NUM>.

For this, the data processing circuitry <NUM>, for example, continuously determines the position of the target <NUM>, as stated above, to derive its motion from temporal changes of the position. Implicitly, the data processing circuitry <NUM> can determine the velocity from the motion.

Further, the camera system <NUM> can determine whether the target <NUM> is static or moving within the environment by tracking the position of the target <NUM>.

For example, the camera system <NUM> can analogously track positions of other targets surrounding the target <NUM> to ascertain whether the target <NUM> moves relative to the other targets.

If the position of the other targets does not change over time, those targets can be identified as static targets which do not move within the environment.

If the target <NUM> moves relative to one or more of the static targets, it can be identified as a moving target.

Alternatively, the data processing circuitry <NUM> can compare velocities of several targets to distinguish between static and moving targets.

As illustrated by <FIG>, the data processing circuitry <NUM> may register the position of the target <NUM> in a (<NUM>-dimensional) digital map <NUM> of the environment, for example, by mapping one or multiple of the patches <NUM>-<NUM>, <NUM>-<NUM> and <NUM>-<NUM> onto the position of the target <NUM> within the digital map <NUM>.

Thus, multiple targets can be continuously registered in the digital map <NUM> during a motion of the camera <NUM> along a motion path <NUM>.

The patches <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> are further characterized by a surface normal <NUM>. Consequently, the data processing circuitry <NUM> can characterize surfaces of the environment in terms of their orientation.

As can be seen in <FIG>, the digital map <NUM> enables the data processing circuitry <NUM> to determine a fourth patch <NUM>-<NUM> indicative of the target <NUM> within a fourth image <NUM> based on the pose of the camera <NUM> and the digital map <NUM>.

In some embodiments of the camera system <NUM>, the data processing circuitry <NUM> is configured to compare the position of the target <NUM> with the pose of the camera <NUM> to predict a position of the target <NUM> within the fourth image <NUM>. Consequently, the data processing circuitry <NUM> may align the fourth patch <NUM>-<NUM> with the predicted position.

Additionally, the data processing circuitry <NUM> can be configured to determine whether the fourth patch <NUM>-<NUM> matches with one or more of the preceding/registered patches <NUM>-<NUM>, <NUM>-<NUM> and <NUM>-<NUM> and optionally determine a deviation from the preceding patches <NUM>-<NUM>, <NUM>-<NUM> and <NUM>-<NUM>.

The deviation, for example, enables a (auto-) re-calibration of the pose of the camera <NUM>.

In some cases a patch <NUM>-<NUM> cannot be mapped to target and/or proper patches registered in the digital map <NUM>. Consequently, such targets or patches can be ignored in connection with the characterization of the environment and/or discarded from the digital map <NUM>.

During a motion of the camera <NUM>, the data processing circuitry <NUM> can also continuously detect "new" targets to associate "new patches" <NUM>-<NUM> with those.

The digital map <NUM> can further be shared with a supplementary camera supporting visual odometry for a characterization of the environment. In general, a field of view from the supplementary camera can differ from a field of view of the camera <NUM>.

Sharing the digital map <NUM> can enable the supplementary camera to associate another patch with the target <NUM> by reference to the digital map <NUM> and a pose of the supplementary camera to provide an auto-calibration between the camera <NUM> and the supplementary camera.

This, for example, can be desired in connection with a driver assistance system using multiple differently oriented cameras.

As mentioned above, in some applications can be a demand for using multiple cameras.

In camera systems with multiple cameras, the cameras can experience relative motion to each other or vibrations, for example, in connection with automotive applications. Consequently, a relative position of the multiple cameras can be subjected to variations which may cause calibration issues. Those calibration issues may lead to an insufficient characterization of the environment.

Hence, there is a demand for an improved concept for a characterization of an environment using multiple cameras to overcome such calibration issues emerging from variations of a relative position of the cameras towards each other.

<FIG> illustrates a camera system <NUM>, according to the invention, which comprises a first camera <NUM>-<NUM> and a first motion measurement unit <NUM>-<NUM> which is configured to provide first motion data of the first camera <NUM>-<NUM>. Further, the camera system <NUM> comprises at least one second camera <NUM>-<NUM> and a second motion measurement unit <NUM>-<NUM> configured to provide second motion data of the second camera <NUM>-<NUM>. Additionally, the camera system <NUM> comprises a data processing circuitry <NUM> configured to determine a relative pose of the first camera <NUM>-<NUM> and the second camera <NUM>-<NUM> towards each other from a correlation between the first motion data and the second motion data.

For this, the first and the second motion measurement unit <NUM>-<NUM> and <NUM>-<NUM> are coupled to the data processing circuitry <NUM>.

In some embodiments, the data processing circuitry <NUM> may comprise a first and a second processing circuitry (not shown) each mounted to the first and the second camera <NUM>-<NUM> and <NUM>-<NUM>. This may be desired if the first and the second camera <NUM>-<NUM> and <NUM>-<NUM> are mounted to separate mobile devices.

In some embodiments, the data processing circuitry <NUM> may correspond to the data processing circuitry <NUM> described in connection with the camera system <NUM>.

The relative position of the first and the second camera <NUM>-<NUM> and <NUM>-<NUM> may relate to a relative motion of the cameras <NUM>-<NUM> and <NUM>-<NUM>. Ipso facto, the relative position can be derived from the first and the second motion data indicative of the relative motion.

The first and/or the second motion data are, for example, indicative of an angular velocity and/or accelerations acting on the cameras <NUM>-<NUM> and <NUM>-<NUM>.

The data processing circuitry <NUM>, for example, determines the relative position of the cameras <NUM>-<NUM> and <NUM>-<NUM> from differences between the first and the second motion data.

In this way, the data processing circuitry <NUM> enables an auto-calibration of the camera system <NUM> with reference to the relative position of the cameras <NUM>-<NUM> and <NUM>-<NUM> to recover a spatial accuracy of the camera system <NUM>.

In the camera system <NUM>, the first camera <NUM>-<NUM> is configured to provide a first sequence of images of an environment and the second camera <NUM>-<NUM> is configured to provide a second sequence of images of the environment. In some those embodiments the data processing circuitry <NUM> can be further configured to determine the relative pose of the first camera <NUM>-<NUM> and the second camera <NUM>-<NUM> towards each other from a correlation between the first motion data, the second motion data, the first sequence of images and the second sequence of images.

For this, the first and the second motion data and the first and the second sequence of images may be time-synchronized.

To communicate the first and the second sequence of images to the data processing circuitry <NUM>, each of the cameras <NUM>-<NUM> and <NUM>-<NUM> may be coupled with the data processing circuitry <NUM>.

The data processing circuitry <NUM> may apply concepts of visual odometry to the first and the second sequence of images as described in connection with the aforementioned embodiments of the camera system <NUM>.

For example, the data processing circuitry <NUM> utilizes a Kalman filter to determine the relative pose of the cameras <NUM>-<NUM> and <NUM>-<NUM>.

An observation model of the Kalman filter can be derived from the correlation of affine correspondences between images of the first and the second sequence of images and a difference between the first and the second motion data.

A control vector of the Kalman filter may be specified by the first and the second motion data, whereas a measured state of the relative pose of the cameras <NUM>-<NUM> and <NUM>-<NUM> may be indicative of the affine correspondences.

This may enable the Kalman filter to continuously determine the relative pose of the first and the second camera <NUM>-<NUM> and <NUM>-<NUM> by reference to the first and the second motion data and the first and the second sequence of images.

In this way, the camera system <NUM> may increase an accuracy of the relative position compared to aforementioned embodiments of the camera system <NUM> using (only) the motion data for determining the relative position of the cameras <NUM>-<NUM> and <NUM>-<NUM>.

As can be seen in <FIG>, the first and the second camera are freely mounted (mounted in floating position) to the mobile device <NUM>. For this, each of the cameras <NUM>-<NUM> and <NUM>-<NUM> can be coupled to the mobile device <NUM> using the stabilizing mounting <NUM>.

In this manner, perturbations (e.g. vibrations and/or torsion) coming from the mobile device may be attenuated to improve a spatial accuracy of the first camera <NUM>-<NUM> and/or the second camera <NUM>-<NUM>.

In some embodiments, the first camera <NUM>-<NUM> is mounted to a first mobile device (not shown) and the second camera <NUM>-<NUM> is mounted to a second mobile device (not shown). Each of the first and the second mobile device, for example, is an unmanned aerial vehicle (UAV). Since the UAVs may move relative to each other, the relative pose of the cameras <NUM>-<NUM> and <NUM>-<NUM> may change.

With an aforementioned concept for determining the relative pose of the first and the second camera <NUM>-<NUM> and <NUM>-<NUM>, the camera system <NUM> may be enabled to (continuously) auto-calibrate the camera system <NUM> with regard to the relative pose.

The data processing circuitry <NUM> is further configured to determine a position of a target within the environment from a correlation between the first sequence of images and the first motion data. Further, the data processing circuitry <NUM> is configured to register the position of the target in a digital map of the environment and detect the target within the second sequence of images using the digital map, the motion data and the relative pose of the first and the second camera <NUM>-<NUM> and <NUM>-<NUM>.

This can be desired, for example, if the first and the second camera <NUM>-<NUM> and <NUM>-<NUM> are mounted to different mobile devices or have opposite fields of view like in case of some driving assistance systems.

The data processing circuitry <NUM>, for example, determines the position of the target using the first sequence of images and the first motion data and may register the position of the target in a <NUM>-dimensional map <NUM> as described in connection with the camera system <NUM>. By reference to the relative pose of the first and the second camera <NUM>-<NUM> and <NUM>-<NUM>, the motion data and the digital map, the data processing circuitry <NUM> can determine if the target is within the field of view of the second camera <NUM>-<NUM>. Further, this may enable the data processing circuitry <NUM> to predict a position of the target within images of the second sequence of images with a deviation from a position of the target derived from the second sequence of images.

The skilled person having benefit from the present disclosure may appreciate that this deviation may be used for an auto-calibration of the first and the second camera <NUM>-<NUM> and <NUM>-<NUM> such that the auto-calibration causes the deviation to decrease.

<FIG> schematically illustrates a method <NUM> for localizing a camera of a camera system for a mobile device, not forming part of the invention defined in the claims. The method comprises providing <NUM> motion data of the camera using at least one motion measurement unit, wherein the camera is freely mounted to the mobile device. Further, the method comprises determining <NUM> a pose of the camera from the motion data.

The method <NUM> can be executed, for example, by the camera system <NUM>. For this, at least a portion of the method <NUM> can be performed by the data processing circuitry <NUM> by executing an appropriate computer program.

<FIG> schematically illustrates method <NUM> for localizing multiple cameras of a camera system for a mobile device, not forming part of the invention defined in the claims. The method <NUM> comprises providing <NUM> first motion data of a first camera of the camera system using a first motion measurement unit. The method <NUM> further comprises providing <NUM> second motion data of at least one second camera of the camera system using a second motion measurement unit. Moreover, the method <NUM> provides for determining <NUM> a relative pose of the first camera and the second camera towards each other through a comparison of the first motion data and the second motion data.

Functions of various elements shown in the figures, including any functional blocks labeled as "means", "means for providing a signal", "means for generating a signal. ", etc., may be implemented in the form of dedicated hardware, such as "a signal provider", "a signal processing unit", "a processor", "a controller", etc. as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which or all of which may be shared. However, the term "processor" or "controller" is by far not limited to hardware exclusively capable of executing software, but may include digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage.

Claim 1:
A camera system (<NUM>; <NUM>), comprising:
a first camera (<NUM>-<NUM>) configured to provide a first sequence of images of an environment;
a first motion measurement unit (<NUM>-<NUM>) configured to provide first motion data of the first camera (<NUM>-<NUM>);
at least one second camera (<NUM>-<NUM>) configured to provide a second sequence of images of the environment;
a second motion measurement unit (<NUM>-<NUM>) configured to provide second motion data of the second camera (<NUM>-<NUM>); and
a data processing circuitry (<NUM>; <NUM>) configured to
determine a relative pose of the first camera (<NUM>-<NUM>) and the second camera (<NUM>-<NUM>) towards each other from a correlation between the first motion data and the second motion data,
determine a position of a target within the environment from a correlation between the first sequence of images and the first motion data;
register the position of the target in a digital map (<NUM>) of the environment; and
detect the target within the second sequence of images using the digital map, the motion data, and the relative pose of the first and the second camera (<NUM>-<NUM>; <NUM>-<NUM>) towards each other.