Patent Description:
It is often desired to be able to estimate for example the distance to, or the width of, a vehicle in front of the ego car.

There are some common ways to estimate the distance to an object detected in an image that is captured using a camera mounted in a vehicle. One way is to assume that the observed vehicle has a typical width that is common for that type of vehicle and then use that assumed width together with the bounding box in the image to compute the distance to the vehicle. Another way is to assume that the road is approximately flat and that the mounting angles of the camera relative to the ground are known to compute the distance to the vehicle and its width. Both mentioned ways to compute the distance and width obviously result in erroneous estimates in scenarios where the assumptions are not valid, i.e., when the width of the observed vehicle differs from the assumed width or when the road is not approximately flat. A third option is to use machine learning to train a network to estimate distance and width to observed objects in the image.

A method of estimating a distance to an object using a sequence of images recorded by a monocular camera is disclosed in <CIT>.

<NPL>, discloses an improved visual semantic odometry method.

The problem underlying the present invention is to provide a simple but effective system and method of estimating values like for example the distance to, or the width of, a vehicle in front of the ego car, suited to be run online for example in an Advanced Driving Assistance System (ADAS).

The invention solves this problem with the features of the independent claims.

The invention describes a system and a method that takes as input a video sequence collected by the camera in a motor vehicle. The invention enables an a-posteriori calculation of the road profile the vehicle has moved on from an already collected video sequence, without use of external reference sensors like lidar, radar or a global satellite navigation device. The resulting output may be a 3D profile which describes the road that the vehicle has been driving on through the entire video sequence relative to the camera. The calculated road profile can advantageously be used for several applications in a drivers assistance system, as will be explained later.

The invention is based on the realization that in each time moment, the ground below the vehicle can be assumed to be flat. This means that the method can estimate where the ground is below the 3D camera trajectory through the entire sequence. For this it is advantageous to know the mounting position and/or the mounting angles of the camera in the vehicle. Preferably, therefore, the processing device is adapted to calculate the road profile from the camera motion profile using the mounting position and/or the mounting angles of the camera in the motor vehicle. Also preferably, the processing device is adapted to use the current suspension offset of the vehicle wheels, respectively, in the calculation of said road profile from said camera motion profile, in order to improve accuracy. Odometer data from the wheels are preferably used in the calculation of the camera motion.

In some embodiments, the camera motion profile is calculated by stacking subsequent camera motions from time frame m to m+<NUM>, m+<NUM> to m+<NUM>,. , n-<NUM> to n. This is a simple method for estimating the motion of the camera through the entire video sequence.

In other embodiments, the camera motion profile can be calculated with higher precision from all images between time frames m and n by using anti-causal visual odometry. Anti-causal visual odometry algorithms are known in general to the skilled person.

Preferably, the processing device is adapted to calculate the distance to the object under inspection at time m from a projection of a pixel corresponding to the bottom of the object under inspection in the image at time m onto the calculated road profile. The distance at a past time m is a valuable information since it allows to calculate other information useful in a driver assistance system. In particular, the processing device is preferably adapted to calculate the width of the object under inspection from the calculated distance at time m and the width of the object candidate in the image at time m. Alternatively or in addition, the processing device is preferably adapted to calculate the height of the object under inspection from said calculated distance at time m and the height of the object candidate in the image at time m. In order to have the width and/or the height of the object candidate in the image at time m available for later calculations after time n, the processing device is preferably adapted to store information relating to the bounding box of the particular object candidate at time frame m in a digital memory.

The width and/or height of a detected object at a past time m is a valuable information since it allows to calculate other information useful in a driver assistance system. For example, the calculated width and/or height of the object under inspection can advantageously be used to calculate the current distance to the object under inspection after time frame n in real time. The current distance to an object under inspection is a valuable information in a monocular vision system, since it cannot be obtained from a disparity image like in a stereo vision system. An advantageous application of this aspect of the invention is an Adaptive Cruise Control (ACC) system, where the distance to a vehicle driving in front of the ego vehicle shall be kept constant.

Summarizing the above, given the complete rigid body transformation, i.e. rotation and translation, of the camera motion between consecutive frames, a transformation between arbitrary frames up to the latest one can be computed. The transformation from the camera to the ground under the front wheels is typically also available, which allows for a non-causal estimation of the road topology. Given an estimation of the road topology together with an image measurement of the bounding box of the vehicle in front of the ego vehicle, the distance can be computed accurately. Given this distance to the vehicle and the bounding box, for example the vehicle width can be computed. The vehicle width is typically constant over time, which is why it is interesting to estimate it non-causally.

To estimate the width of a vehicle in front of the ego vehicle the following steps are preferably taken:.

In step b) above, alternatively to calculating all rigid body transformations from i-<NUM> to i, the road topology or road profile could be calculated in a unified manner from all images i where m < i <= n using an anti-casual visual odometry algorithm.

Note that this process can be repeated an arbitrarily number of times to obtain several samples of estimated vehicle widths.

The suggested method does not use assumptions about vehicle width, nor that the road is flat, and thus is not affected by errors resulting from when these assumptions are not satisfied.

In the following the invention shall be illustrated on the basis of preferred embodiments with reference to the accompanying drawings, wherein:.

The vision system <NUM> is mounted, or to be mounted, in or to a motor vehicle and comprises an imaging apparatus <NUM> for capturing images of a region surrounding the motor vehicle, for example a region in front of the motor vehicle. The imaging apparatus <NUM> may be mounted for example behind the vehicle windscreen or windshield, in a vehicle headlight, or in the radiator grille. The imaging apparatus <NUM> comprises an optical imaging device <NUM>, in particular a camera, preferably operating in the visible wavelength range, in the infrared wavelength range, or in both visible and infrared wavelength range, where infrared covers near IR with wavelengths below <NUM> microns and/or far IR with wavelengths beyond <NUM> microns. The imaging apparatus <NUM> comprises one imaging device <NUM> forming a mono imaging apparatus <NUM>. The imaging device <NUM> is preferably a fix focus camera, where the focal length f of the lens objective is constant and cannot be varied.

The imaging apparatus <NUM> is coupled to an on-board data processing device <NUM> adapted to process the image data received from the imaging apparatus <NUM>. The data processing device <NUM> is preferably a digital device which is programmed or programmable and preferably comprises a microprocessor, a microcontroller, a digital signal processor (DSP), and/or a microprocessor part in a System-On-Chip (SoC) device, and preferably has access to, or comprises, a digital data memory <NUM>. The data processing device <NUM> may comprise a dedicated hardware device, like a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU) or an FPGA and/or ASIC and/or GPU part in a System-On-Chip (SoC) device, for performing certain functions, for example controlling the capture of images by the imaging apparatus <NUM> and/or receiving the electrical signal containing the image information from the imaging apparatus <NUM>. The data processing device <NUM>, or part of its functions, can be realized by a System-On-Chip (SoC) device comprising, for example, FPGA, DSP, ARM, GPU and/or microprocessor functionality. The data processing device <NUM> and the memory device <NUM> are preferably realised in an on-board electronic control unit (ECU) and may be connected to the imaging apparatus <NUM> via a separate cable or a vehicle data bus. In another embodiment the ECU and the imaging device <NUM> can be integrated into a single unit, where a one box solution including the ECU and the imaging device <NUM> can be preferred. All steps from imaging, image processing to possible activation or control of safety device <NUM> are performed automatically and continuously during driving in real time.

Image and data processing carried out in the processing device <NUM> advantageously comprises identifying and preferably also classifying possible objects (object candidates) in front of the motor vehicle, such as pedestrians, other vehicles, bicyclists and/or large animals, tracking over time the position of objects or object candidates identified in the captured images, and activating or controlling at least one safety device <NUM> depending on an estimation performed with respect to a tracked object, for example on an estimated collision probability.

The safety device <NUM> may comprise at least one active safety device and/or at least one passive safety device. In particular, the safety device <NUM> may comprise one or more of: at least one safety belt tensioner, at least one passenger airbag, one or more restraint systems such as occupant airbags, a hood lifter, an electronic stability system, at least one dynamic vehicle control system, such as a brake control system and/or a steering control system, a speed control system; a display device to display information relating to a detected object; a warning device adapted to provide a warning to a driver by suitable optical, acoustical and/or haptic warning signals.

The invention is applicable to autonomous driving, where the ego vehicle is an autonomous vehicle adapted to drive partly or fully autonomously or automatically, and driving actions of the driver are partially and/or completely replaced or executed by the ego vehicle.

In the following, preferred embodiments of a vision method under the present invention are described with reference to <FIG>.

In the time frame m, the camera <NUM> of the ego vehicle has detected another vehicle <NUM> driving ahead of the ego vehicle <NUM> in the same direction. The processing device <NUM> assigns a bounding box <NUM> to the detected object or object candidate <NUM>. This is shown in <FIG>. The bounding box <NUM> is a rectangle closely enclosing the corresponding object candidate <NUM> and has a width wb and a height hb. The processing device <NUM> stores the position and the width wb and height hb of the bounding box <NUM>, or more generally the bounding box <NUM> coordinates, at the time frame m in the memory device <NUM>.

In one embodiment, for all subsequent time frames i with m<i<=n, where i is an integer value and the number n will be explained later, the movement of the camera <NUM> from the previous time frame i-<NUM> to the current time frame i is calculated and stored in the memory device <NUM>. The movement or rigid body transformation of the camera <NUM> is the rotation and the translation of the camera <NUM> from time frame i-<NUM> to time frame i. This can be calculated by the processing device <NUM> from the images corresponding to the time frames i, i+<NUM>, or from a video sequence comprising these images, using a so-called visual odometry algorithm. Visual odometry algorithms which can estimate the camera <NUM> motion between two subsequent camera image frames i-<NUM>, i accurately are known to the skilled person. The resulting set of camera positions Cm,. , Cn form a camera <NUM> motion profile C(i) or C(x), see <FIG>.

For all time frames i, where i is an integer value with m<i<=n also the geometric transformation of the camera <NUM> position to a vehicle related ground position is known from the mounting position and mounting angle of the camera <NUM>, and may be prestored in the memory device <NUM>. By means of this geometric transformation, each camera <NUM> position at time i can be transformed into a vehicle related ground position, like the ground position between the front wheels, or the ground position exactly below the camera <NUM>. The resulting sequence of ground positions Rm, Rm+<NUM>, Rm+<NUM>,. , Rn is shown in <FIG>. The processing device <NUM> may interpolate all ground positions from Pm to Pn to achieve a continuous road profile R(x), where m<=x<=n.

In another embodiment, the processing device <NUM> stores all images Im,. In captured at time frames m<=i<=n in the memory device <NUM>, and calculates a set of ground positions Rm, Rm+<NUM>, Rm+<NUM>,. , Rn from all images Im,. In in a unified manner by an anti-causal visual odometry algorithm after time frame n. Again, the processing device <NUM> may interpolate all ground positions from Pm to Pn to achieve a continuous or quasi-continuous road profile R(x), where m<=x<=n.

In a still further embodiment, the processing device <NUM> stores all images Im,. In captured at time frames m<=i<=n in the memory device <NUM>, and directly calculates a continuous road profile R(x), where m<=x<=n, from all images Im,. In in a unified manner by an anti-causal visual odometry algorithm after time frame n.

From the calculated road profile R(i) or R(x), the processing device <NUM> computes the distance dm of the vehicle <NUM> to the camera <NUM> at past time m by projecting a pixel corresponding to the bottom of the object <NUM> under inspection in the image at time m on the calculated road profile R(x), using the mounting position and mounting angle of the camera <NUM>. This is equivalent to finding the intersection <NUM> of the straight line <NUM> from the camera <NUM> to the bottom of the other vehicle <NUM> at time m with the road profile R(x), see top of <FIG>.

The distance dm is calculated after time frame n, which can be defined as the first time frame after the camera <NUM> has passed the position which the rear bottom of the other vehicle <NUM> had at time frame m (see top of <FIG>). Only after time frame n, the complete road profile R(x) used for calculating dm is available. The processing device <NUM> can estimate when time frame n has been reached or passed, in order to start calculation of the road profile R(x). The processing device <NUM> periodically checks whether time frame n has been reached or passed, by calculating a preliminary road profile, and checking whether the straight line <NUM> from the camera <NUM> to the bottom of the other vehicle <NUM> at time m intersects with the preliminary road profile.

From the calculated distance dm, the width w of the other vehicle <NUM> can be easily calculated using the formulas w = ch·wb·dm/f, where wb is the width of the bounding box <NUM> at time m in pixels, f is the focal length of the camera, and ch is a constant giving the relation between pixels and mm of the image sensor of the camera <NUM> in the horizontal direction. Similarly, the height h of the other vehicle <NUM> can be easily calculated using the formulas w = cv·hb·dm/f, where hb is the height of the bounding box <NUM> at time m in pixels, f is the focal length of the camera, and cv is a constant giving the relation between pixels and mm of the image sensor of the camera <NUM> in the vertical direction.

Claim 1:
A mono vision system (<NUM>) for a motor vehicle (<NUM>), comprising a mono camera (<NUM>) adapted to capture images from a surrounding of the motor vehicle (<NUM>), and a processing device (<NUM>) adapted to detect other vehicles (<NUM>) in the surrounding of the motor vehicle (<NUM>) by processing images captured by said mono camera (<NUM>), wherein said
processing device (<NUM>) is adapted to perform the following processing with respect to a particular detected other vehicle (<NUM>):
from a plurality of images spanning a period of time between a time frame m where the other vehicle (<NUM>) is visible by the camera (<NUM>) and a time frame n where the position where the other vehicle (<NUM>) was at time m has been reached or passed by the motor vehicle (<NUM>),
periodically check whether time frame n has been reached by calculating a preliminary road profile and checking whether the straight line (<NUM>) from the camera (<NUM>) to the bottom of the other vehicle (<NUM>) at time m intersects with the preliminary road profile,
calculate a camera (<NUM>) motion profile C(x) between time frame m and time frame n using a visual odometry algorithm; and
after time frame n, calculate a road profile R(x) between
time frames m and n from said calculated camera (<NUM>) motion profile C(x).