Patent Description:
Automotive applications typically require a clear perception of an area surrounding a vehicle. Perception includes, for example, identification of moving objects such as othervehicles and pedestrians, as well as static objects such as road signs and debris. In order to generate a suitable perception, vehicles are equipped with sensors, like cameras, radio detection and ranging (RADAR) and/or light detection and ranging (LIDAR) sensors. Advanced warning systems use information generated by the sensors to warn of potential dangers, and autonomous vehicle control systems use it to safely maneuver the vehicle. The advanced warning systems need to operate in real time, and minimize mistakes in identification and classification of moving objects, while integrating data received by the different sensors.

<CIT> discloses a multi-source target fusion method using a unified time coordinate system and a geographical coordinate system to, respectively, perform a time registration and a spatial registration. A method for generating the multi-sensor target matching result includes: initializing the weight matrix by using the Hungarian algorithm by initializing the target state information obtained by time registration and spatial registration of each sensor target information. The Euclidean distance between the targets of different sensors is used as the weight in the matrix, and then the target matching between multiple sensors is performed by finding a "perfect" match. Dempster-Schafer evidence theory is also applied for target existence fusion and category fusion.

<NPL>, discloses using a common (observation) plane for which, for each of the CCD camera and the radar, an homography is learnt. Multi-object tracking is performed by employing Kalman filtering and data fusion technique underlying a Gaussian mixture form.

The present invention is as defined in the appended set of claims. It is an object of some embodiments of the present invention to provide a system and a method for multi-modal sensor fusion of moving object data.

Other systems, methods, features, and advantages of the present disclosure will become apparent to one with skill in the art upon examination of the following drawings and detailed description.

The present invention, in some embodiments thereof, relates to relates to multi-modal sensor fusion of moving object data, and, more specifically, but not exclusively, to a multi-modal sensor fusion system for moving objects data generated by vehicle sensors.

According to some embodiments of the present invention, there are provided multi-modal sensor fusion systems and methods in which datasets generated from different sensors are integrated for a purpose of presenting data to users in a combined manner. For example, the systems and methods may fuse spatiotemporal streaming data from sensors such as cameras, RADAR(s), or LIDAR(s).

Multi-modal sensor fusion systems present many computational challenges such as: overcoming different coordinate systems attributed to different sensors, incompatible numbers of detections (such as false positives and false negatives), alignment across time between modalities (e.g., different sensing frequencies for different sensors), and noise. Overcoming such computational challenges is important since multi-modal sensor fusion systems are often used in time critical and sensitive situations such as early warning systems aimed at preventing vehicle accidents.

Existing solutions for multi-modal sensor fusion exhibit several drawbacks, for example, in <NPL>, a system is described which fuses radar and image modalities by using trajectories, however, the fusion is performed on a frame-by-frame level, and trajectories are used only as an extra layer of verification.

In <NPL>, a system is described which tracks multiple modalities across time to obtain better estimates of obstacles, relative to use of a single modality, however a solution for matching observations between modalities is not provided.

By contrast, the multi-modal sensor fusion system described herein matches between different modalities by matching between spatiotemporal trajectories, as spatiotemporal trajectories are separately computed for each modality and then inputted into a matching computation that matches between complete trajectories. Matching between trajectories of camera and radar detections may improve robustness, because each trajectory contains more information regarding a moving object than a single camera frame.

Another possible advantage is in assisting offline training and evaluation, by using the system described herein offline in order to generate high-quality estimates of matching between modalities.

Following is a brief description of dataflow of processing of spatiotemporal data, according to some embodiments of the present invention. For brevity, 'spatiotemporal trajectories' may be referred to herein as 'trajectories', 'RADAR or LIDAR' may be referred to herein as 'RADAR/LIDAR', and "variable" and 'attribute' is used herein interchangeably.

Each of the described systems includes and each of the described methods is implemented using a processing circuitry configured to execute a code. The processing circuitry may comprise hardware and firmware and/or software. For example, the processing circuitry may comprise one or more processors and a non-transitory medium connected to the one or more processors and carrying the code. The code, when executed by the one or more processors, causes the system to carry out the operations described herein.

The spatiotemporal data originating from a camera and/or RADAR/LIDAR is buffered, for example, by counting a predetermined number of camera frames, and buffering data from a RADAR/LIDAR corresponding to a time interval according to the predetermined number of camera frames. Next, code for producing spatiotemporal trajectories for each moving object detected by each sensor is executed.

Next, for each pair of spatiotemporal trajectories, a "distance" metric is computed which quantifies their similarity. The trajectories are processed by the code in order to be mapped into a predetermined domain. The predetermined domain comprises a coordinate system according to data originating from the camera sensor, or alternatively, a coordinate system according to data originating from the RADAR/LIDAR sensor, or alternatively, a latent temporal domain comprising a set of locational, directional, and velocity attributes for each moving object.

Processing the spatiotemporal trajectories may include estimation of velocities, and/or estimation of dimensions and/or heights of respective moving objects. For example, a camera typically produces two dimensional data without depth, in a line of view corresponding to camera lens location and orientation, while a RADAR/LIDAR typically produces two dimensional data with depth but without height according to the RADAR/LIDAR orientation and mounted location on a vehicle. In addition, calibration matrices, which provide the relative locations of the camera and radar relative to the ego-vehicle center (center of the vehicle on which they are mounted), and the camera projection attributes (intrinsic and extrinsic), are used in order to align data from originating from the different sensors.

Next, following estimation of velocities and heights for respective trajectories of respective moving objects, and alignment of coordinates, a trajectory matching computation is executed on mapped trajectories, in order to identify each moving object with two respective trajectories. Optionally, the matching computation uses Euclidean distance metric, and matches between trajectories using this metric, by minimizing a sum of distances between pairs across all possible trajectory matching combinations, which may be performed by applying an optimization algorithm.

According some embodiments of the present invention in which a latent temporal domain is used as the predetermined domain for mapping trajectories, a latent state is estimated for each pair of camera and RADAR/LIDAR trajectories which are used as two types of observations of the latent model, and postulate a statistical model that relates them to latent states variables. The statistical model is employed by the matching computation by matching trajectories whose joint latent state has a high probability of originating from a single moving object.

Following the matching computation, an outcome of the matching is outputted to respective components, such as vehicle controller(s), and/or output display(s).

Reference is now made to <FIG>, which is a depiction of system components in a multi-modal sensor fusion system <NUM>, and related vehicular components, according to some embodiments of the present invention. The system <NUM> is used for multi-modal sensor fusion, wherein spatiotemporal sensors <NUM> consist of a camera and a RADAR/LIDAR, according to some embodiments of the present invention. For example, the system <NUM> may be integrated in a vehicle as part of an early warning mechanism, accessed by a driver via a head up display (HUD), in order to present the driver with a stream of fused sensor information, such as velocity, direction, and highlighted indicators of moving objects, which may assist the driver to achieve a reduced reaction time in case of danger of collision with debris, pedestrians, and other vehicles. Alternatively, the system <NUM> may present a vehicle controller responsible for autonomous driving of the vehicle, with fused data regarding moving objects in front of the vehicle, so that the vehicle controller may decide upon an action regarding vehicle navigation based on the moving object fused data.

An I/O interface <NUM> receives raw spatiotemporal data from the spatiotemporal sensors <NUM>, which is then processed by a code stored in the memory storage <NUM>, by execution of the code by the one or more processor(s) <NUM>. The code contains instructions for a process for multi-modal sensor fusion of moving object data, by generating spatiotemporal trajectories from the raw spatiotemporal data, according to a predetermined data sampling criteria, and by matching the generated spatiotemporal trajectories across different modalities, for example, by matching camera trajectories to RADAR/LIDAR trajectories.

Outcomes of the matching of spatiotemporal trajectories are outputted via the I/O interface <NUM> by the one or more processor(s) executing the code instructions, whereas the outputs may be directed to the output display <NUM> and/or the controller <NUM>. For example, if a vehicle using the system is controlled by a driver then the outputs may be directed to the output display <NUM>, whereas if the vehicle is controlled autonomously by the controller <NUM>, then it may use the outputs for obstacle avoidance, and route planning.

Reference is now made to <FIG>, which is an exemplary dataflow of a process for multi-modal sensor fusion of moving object data, according to some embodiments of the present invention. First, as shown at <NUM> and <NUM>, raw camera and RADAR/LIDAR data is received, for example, camera frame data is received at a certain rate as a data stream, and/or RADAR/LIDAR detections are received at a certain frequency.

Next, as shown in <NUM> and <NUM>, the received raw data is checked against a predefined sample rate, which may differ for camera and/or RADAR/LIDAR sensors. The predefined sample rate may be, for example, a predefined number of seconds, a predefined number of camera frames, and/or a predefined number of RADAR/LIDAR detections. If the sample rate is reached, then, as shown in <NUM>, spatiotemporal trajectories are generated for each raw data stream. The spatiotemporal trajectories are generated by an extended object tracking process across time. According to some embodiments of the present invention, a Kalman tracking process is used in order to track and generate the spatiotemporal trajectories.

Next, as shown in <NUM>, the generated spatiotemporal trajectories are mapped onto the predetermined domain. Denote by X, Y, Z horizontal, vertical, and depth axis respectively, then the RADAR/LIDAR domain describes moving objects in a XZ plane and includes a velocity for each moving object, whereas the camera domain describes moving objects in a XY plane. In addition, calibration matrices may be used in order to align the trajectories originating from the different sensors according to the sensors relative position to an ego-vehicle center.

The predetermined domain comprises a domain corresponding to the camera domain, or the RADAR/LIDAR domain, or alternatively, comprises a latent temporal domain, whereas each choice of predetermined domain may convey certain advantages/disadvantages over alternative choices for different embodiments of the present invention. For example, mapping spatiotemporal trajectories onto a latent temporal domain may increase trajectory matching accuracy but may also increase computational complexity.

Next, as shown in <NUM>, following the mapping of the spatiotemporal trajectories onto the predetermined domain, a matching computation is executed between the mapped spatiotemporal trajectories, in order to match between trajectories corresponding to identical moving objects. The matching computation is executed by using a predefined distance metric, for example, using Euclidean distance, and by applying an optimization algorithm to minimize a sum of distances, for example, the Kuhn-Munkres ("Hungarian") algorithm may be used to select a matching (i.e., a subset of all pairs, in which each trajectory appears at most once) that minimizes the sum of distances.

Next, as shown in <NUM>, an outcome of the matching computation is outputted. Matched trajectories may be outputted via the I/O interface <NUM> to the output display <NUM>, and/or to the controller <NUM>.

Reference is also made to <FIG>, which is an exemplary dataflow of a process of mapping spatiotemporal trajectories, as depicted in <NUM>, onto a domain corresponding to a camera domain, according to some embodiments of the present invention. Following generating spatiotemporal trajectories, as shown in <NUM>, camera and RADAR/LIDAR trajectories are received separately, as shown in <NUM> and <NUM>. Next, as shown in <NUM>, for each camera trajectory, given in XY coordinates, a velocity is estimated. Optionally, the velocity is estimated by an average optical flow inside the object boundary. Optical flow is a pattern of apparent motion of image objects between two consecutive frames, caused by a movement of object or camera. Calculating optical flow may be achieved in a number of ways ranging from computer vision algorithms (e.g. the Lucas-Kanade method) to deep neural networks (for example, using FlowNet <NUM>). An optical flow algorithm output typically comprises motion vectors in a camera domain of image pixels. Averaging optical flow inside a boundary of an object provides an estimate of object motion projected to the camera plane. A series of velocity estimations computed over a camera trajectory can be combined in a weighted sum together with a bounding box-based velocity calculation for a more robust estimation. The precise value of the weight in the weighted sum may be learnt from a sample training data set using machine learning techniques.

As shown in <NUM>, a height is estimated for each moving object corresponding to a RADAR/LIDAR trajectory given in XZ coordinates. Optionally, each RADAR/LIDAR trajectory is assigned a predefined identical height, for example, a height of one meter. Next, as shown in <NUM>, the RADAR/LIDAR trajectories, together with their assigned heights are mapped onto the camera domain, optionally using the calibration matrices for coordinate's alignment. Finally, as shown in <NUM>, the mapped spatiotemporal trajectories are outputted for further processing by the mapping computation <NUM>.

Reference is also made to <FIG>, which is an exemplary dataflow of a process of mapping spatiotemporal trajectories onto a domain corresponding to a RADAR/LIDAR domain, as depicted in <NUM>, according to some embodiments of the present invention. First, as shown in <NUM>, camera trajectories are received by the process. Next, as shown in <NUM>, for each camera trajectory a time series of two dimensional spatial bounding boxes is retrieved, wherein each bounding box corresponds to a respective moving object in a respective camera frame. According to some embodiments of the present invention, the bounding boxes are retrieved by applying a predefined camera object detection algorithm. Next, as shown in <NUM>, for each camera trajectory a respective moving object velocity is estimated. The velocities may be estimated in a similar manner as explained for <NUM>, or optionally, comprise of a bounding box based calculation. Next, as shown in <NUM>, the camera trajectories are mapped onto the RADAR/LIDAR domain using coordinate calibration matrices. Finally, as shown in <NUM>, the mapped camera and RADAR/LIDAR trajectories are outputted for further processing by the mapping computation <NUM>.

Next, reference is also made to <FIG>, which is an exemplary dataflow of a process of matching spatiotemporal trajectories using a latent state model, as related to <NUM> and <NUM>, according to some embodiments of the present invention. First, as shown in <NUM>, spatiotemporal trajectories are received by the process following generating trajectories in <NUM>. Next, as shown in <NUM>, a latent state is estimated for each possible combination of pairs of camera and RADAR/LIDAR trajectories. Each spatiotemporal trajectory may be assigned a numerical index for referral, and pairing of trajectories may be performed according to an ascending index order.

Optionally, the latent state is estimated employing a nonlinear double model Kalman filter with smoothing, and using as observations, for each respective trajectory in a respective pairing of trajectories used in a latent state, estimations of respective bounding boxes and respective RADAR/LIDAR detections.

Each latent state contains attributes (x, y, z, V(x), V(y), A(x), A(y), h, w) comprising respectively: Distance from ego-vehicle in the axis of ego-vehicle motion, location on axis orthogonal to motion, a velocity X component, a velocity Y component, an acceleration X component, an acceleration Y component, an object height in a plane facing the ego-vehicle, and an object width in a plane facing the ego-vehicle. Exemplary details of attribute values and update functions as employed by the Kalman filter, are depicted in <FIG>.

Next, as shown in <NUM>, for each estimated latent state, a mapping is performed into the camera and RADAR/LIDAR domains. The purpose of the mappings is to estimate a conditioned probability, as shown in <NUM>, for each mapping, which represents an event that a respective latent state corresponds to a moving object detected by the spatiotemporal sensors, given the respective mapping. For example, a camera observation is preprocessed as a bounding box characterized by top left and bottom right values in the camera domain. Given a latent state, obtaining a camera observation with top left and bottom right values is computed by projecting values derived from the latent state onto the camera domain. Next, a difference from the observed camera attributes is evaluated, and a product of a Gaussian density function with predetermined variances (as detailed in <FIG>) is computed.

Next, the two conditional probabilities corresponding to a respective latent state are summed, and the resulting sum is designated to be a distance between the respective pair of camera and RADAR/LIDAR trajectories.

Next, as shown in <NUM>, a check is performed whether there are any unmapped pairs of camera and RADAR/LIDAR trajectories, and if so selecting pairs of trajectories continues at <NUM>.

Upon completion of pairing trajectories and computing for each pair a conditional probability, as shown in <NUM> a probability matrix Pi,j is generated, wherein P(i, j) is equal to the conditional probability estimated in <NUM> that a respective camera trajectory with an assigned index i and a respective RADAR/LIDAR trajectory with an assigned index j originate from observations of an identical object.

Next, as shown in <NUM>, camera and RADAR/LIDAR trajectories are matched by maximizing a sum of probabilities in P(i, j) such that at most one entry is selected from each row and each column of P(i, j). According to some embodiments of the present invention, the matching is performed by using the Kuhn-Munkres ("Hungarian") algorithm. Next, as shown in <NUM>, the matched trajectories are outputted to <NUM>.

Reference is now made to <FIG>, which is a table of values for attributes of latent states and update functions for using a Kalman filter in evaluating probabilities of latent states, according to some embodiments of the present invention.

As shown in <NUM>, each latent state attributes are each assigned an initial value, which may be determined according to experimentations. As shown in <NUM>, each attribute is assigned an initial probability, wherein the initial probabilities are used by the Kalman filter in a standard way, wherein VarForDeltaAndP(delta,p) represents the variance that satisfies Pr[|X - mean| < delta] = p for X distributed as a normal distribution N(mean,Var).

Reference is now made to <FIG>, which depicts camera and RADAR/LIDAR trajectories, each in a respective domain of an identical scene, according to some embodiments of the present invention. As shown in <NUM>, a camera frame is displayed, which demonstrates automobiles in front of a vehicle. As seen in <NUM>, each automobile is surrounded by a bounding box, and camera trajectories of moving automobiles are mapped onto the frame as sequences of dots, representing automobile detections across multiple time frames. As shown in <NUM>, trajectories originating from RADAR data are displayed, wherein a cone represents boundaries of the RADAR sensor in front of the vehicle. The RADAR trajectories correspond to the same time interval as the camera trajectories in <NUM>. An application of the latent state model is demonstrated in <NUM> and <NUM>, wherein two trajectories marked by ellipsoids, each in a respective domain, are matched as originating from an identical automobile.

Reference is now made to <FIG>, which depicts an exemplary application of an embodiment of the present invention. As shown in <NUM>, a camera frame displays for each automobile a respective bounding box and estimated distance, which results from fusing camera and RADAR data, by matching respective trajectories. For example, the camera display may be of a HUD designated for a driver, in order to alert the driver to any potential dangers such as sudden changes in directions of the surrounding automobiles.

The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the appended claims.

The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

It is expected that during the life of a patent maturing from this application many relevant systems, methods and computer programs will be developed and the scope of the term sensor fusion intended to include all such new technologies a priori.

Claim 1:
A system for computing correspondence between multiple sensor observations of a scene, comprising:
an interface (<NUM>) for receiving first spatiotemporal datasets and second spatiotemporal datasets of a plurality of moving objects in a scene, the first spatiotemporal datasets and second spatiotemporal datasets are based on signals reflected from the plurality of moving objects and originate from a first sensor and a second sensor respectively, where the first sensor and the second sensor capture different types of signal; wherein the first spatiotemporal datasets originate from a camera sensor; wherein the second spatiotemporal datasets originate from a radio detection and ranging, RADAR, or light detection and ranging, LIDAR, sensor; and
processing circuitry (<NUM>), configured to:
generating a plurality of first spatiotemporal object trajectories and a plurality of second spatiotemporal object trajectories based on the first spatiotemporal datasets and the second spatiotemporal datasets respectively;
computing a distance metric between the first and second spatiotemporal object trajectories, by a respective mapping from a respective domain into a predetermined domain; wherein the predetermined domain is a domain comprising a coordinate system according to the first spatiotemporal trajectories originating from the camera sensor, or a domain comprising a coordinate system according to the second spatiotemporal trajectories originating from the RADAR or LIDAR sensor;
executing a matching computation between the mappings of the plurality of first and second spatiotemporal object trajectories by employing the distance metric in order to calculate a similarity between each pair of mappings of first and second spatiotemporal object trajectories, wherein the matching computation pairs trajectories for which a sum of distances between pairs is minimized across all possible trajectory matching combinations; and
outputting an outcome of the matching computation.