Patent Description:
This object is satisfied by a method according to claim <NUM>, a system according to claim <NUM> and a computer-readable storage media according to claim <NUM>.

In this way, the described method can provide high-precision object tracking with quantifiable uncertainty, satisfying stringent requirements of even some of the most advanced driver-assistance and autonomous driving systems.

To execute the example method, an example system includes one or multiple processors configured to perform this method or a computer-readable storage media having instructions that, when executed, configure one or multiple processors to perform this method. The example system can additionally or separately include means for performing this method.

This document also describes other operations of the above-summarized method and other systems set forth herein, as well as means for performing these methods.

The details of one or more aspects of multi-scan sensor fusion for object tracking are described in this document concerning the following figures. The same numbers are used throughout the drawings to reference similar features and components.

Sensor-fusion systems are an important technology for assisted-driving and autonomous-driving systems. Some driving systems use data from multiple types of sensors (e.g., radar systems, lidar systems, vision systems) to monitor objects in an environment of a host vehicle. Advanced driver-assistance systems (e.g., adaptive cruise control (ACC), automatic emergency braking (AEB)), and autonomous-driving systems match tracks from different types of sensors (e.g., radar tracks and vision tracks) to track objects with a high level of precision. Such T2TF techniques are also critical for driving-scene understanding, trajectory planning, and control operations for these advanced driver-assistance and autonomous-driving systems.

Some sensor-fusion systems use T2TF techniques that use a single data scan to match one set of tracks (e.g., radar tracks) to another (e.g., vision tracks). Often the tracks from one sensor type (e.g., radar tracks) are forced to match or associate with the other set of tracks (e.g., vision tracks). Such systems and methodologies cannot generally associate multiple tracks from one sensor type (e.g., multiple radar tracks) to a single track from another sensor type (e.g., a single vision track). In addition, these systems are generally complex, with many ad-hoc parameters involved in the fusion methodology.

In contrast, this document describes less complex and more accurate sensor-fusion techniques and systems to perform T2TF. For example, these techniques and systems can use information from multiple scans to provide a more-nuanced environmental view that supports numerous hypothesis associations between radar tracks and vision tracks (or other tracks from other sensor types). In addition, these techniques and systems can use Dempster-Shafer theory to determine a belief parameter and plausibility parameter for the hypotheses. In this way, the described methods and systems can quantify uncertainty in associations between radar and vision tracks and better fulfill requirements for advanced driver-assistance systems and autonomous-driving systems.

For example, this document describes techniques to perform multi-scan sensor fusion for object tracking. A sensor-fusion system of a vehicle can obtain radar tracks and vision tracks generated for an environment. The sensor-fusion system maintains sets of hypotheses for associations between the vision tracks and the radar tracks based on multiple scans of radar data and vision data. The sets of hypotheses include mass values for the associations. The sensor-fusion system determines a probability value for each hypothesis. Based on the probability value, matches between radar tracks and vision tracks are determined. The sensor-fusion system then outputs the matches to a semi-autonomous or autonomous driving system to control the vehicle's operation. In this way, the described techniques, systems, and methods can provide high-precision obstacle tracking with quantifiable uncertainty.

This section describes just one example of how the described techniques, systems, and methods can perform multi-scan sensor fusion for object tracking. This document describes other examples and implementations.

<FIG> illustrates an example environment <NUM> in which a vehicle <NUM> is equipped with a sensor-fusion system <NUM> that performs multi-scan sensor fusion for object tracking in accordance with this disclosure. Vehicle <NUM> can represent any apparatus or machinery, including operated and unmanned systems used for various purposes. Some non-exhaustive and non-limiting examples of the vehicle <NUM> include other motorized vehicles (e.g., a motorcycle, a bus, a tractor, a semi-trailer truck, watercraft, aircraft, or construction equipment).

In the depicted environment <NUM>, the sensor-fusion system <NUM> is mounted to, or integrated within, the vehicle <NUM>. Vehicle <NUM> can travel on a roadway <NUM> and use the sensor-fusion system <NUM> to navigate the environment <NUM>. Objects may be in proximity to the vehicle <NUM>. For example, <FIG> depicts another vehicle <NUM> traveling in front of and in the same direction as the vehicle <NUM> along the roadway <NUM>.

With the sensor-fusion system <NUM>, vehicle <NUM> has an instrumental or sensor field of view <NUM> that encompasses the other vehicle <NUM>. The sensors (e.g., radar systems, vision systems) can have the same, similar, or different instrumental fields of view <NUM> of the roadway <NUM>. For example, a vision system can project an instrumental field of view <NUM>-<NUM> (not illustrated in <FIG>). A radar system can project an instrumental field of view <NUM>-<NUM> (not shown in <FIG>) from any exterior surface of the vehicle <NUM>. Vehicle manufacturers can integrate at least a part of a vision system or radar system into a side mirror, bumper, roof, or any other interior or exterior location where the field of view <NUM> includes the roadway <NUM>. In general, vehicle manufacturers can design the sensors' location to provide a field of view <NUM> that sufficiently encompasses the roadway <NUM> on which vehicle <NUM> may be traveling. For example, the sensor-fusion system <NUM> can monitor the other vehicle <NUM> (as detected by the sensor systems) on the roadway <NUM> in which the vehicle <NUM> is traveling.

The sensor-fusion system <NUM> includes a fusion module <NUM> and one or more sensor interfaces <NUM>, including a vision interface <NUM>-<NUM> and a radar interface <NUM>-<NUM>. The sensor interfaces <NUM> can include additional sensor interfaces, including another sensor interface <NUM>-n, where n represents the number of sensor interfaces. For example, the sensor-fusion system <NUM> can include interfaces to other sensors (e.g., lidar systems) or other vision systems or radar systems. Although not illustrated in <FIG>, the fusion module <NUM> executes on a processor or other hardware. During execution, the fusion module <NUM> can track objects based on sensor data obtained at the vision interface <NUM>-<NUM> and the radar interface <NUM>-<NUM>. The vision interface <NUM>-<NUM> receives vision or camera data from one or more vision systems of the vehicle <NUM>. The radar interface <NUM>-<NUM> receives radar data from one or more radar systems of the vehicle <NUM>. In particular, the fusion module <NUM> can access the vision interface <NUM>-<NUM> and the radar interface <NUM>-<NUM> to obtain the vision data and the radar data, respectively.

The fusion module <NUM> configures the sensor-fusion system <NUM> to combine the different types of sensor data obtained from the sensor interfaces <NUM> into vision tracks and radar tracks (not illustrated in <FIG>) for tracking objects in the field of view <NUM>. The fusion module <NUM> can determine a plurality of object tracks according to first sensor data (e.g., obtained from the vision interface <NUM>-<NUM>) and identifies another plurality of object tracks according to second sensor data (e.g., obtained from the radar interface <NUM>-<NUM>). Alternatively, the fusion module <NUM> receives the vision tracks and radar tracks via the sensor interfaces <NUM> from the respective sensor systems. The fusion module <NUM> can determine associations between the vision and radar tracks and maintain or store the associations in a track associations data store <NUM>.

The fusion module <NUM> can, for example, use a sliding window with multiple scans of sensor data to create associations between tracks generated by different types of sensor systems. The track associations can be stored in the track associations data store <NUM>. For example, the sensor-fusion system <NUM> can create associations for one or more vision tracks collected or obtained by the vision interface <NUM>-<NUM> with one or more radar tracks collected or obtained by the radar interface <NUM>-<NUM>. A single scan of the environment <NUM> often lacks sufficient contextual information necessary to provide a nuanced environmental view. Because insufficient evidence to support multiple hypothesis association may be obtained from one individual scan, the fusion module <NUM> uses a sliding window with multiple scans of sensor data to create the associations between tracks.

The fusion module <NUM> can determine multiple hypotheses for one or more associations. The fusion module <NUM> can also combine mass values using the Dempster-Shafer theory applied to the multiple hypotheses and generate a fused mass value. The Dempster-Shafer theory provides a framework for determining and reasoning with uncertainty among multiple hypotheses. The Dempster-Shafer theory enables evidence from different sources (e.g., radar tracks and vision tracks) to be combined to arrive at a degree of belief for a hypothesis when considering all available evidence. Each given hypothesis is included in a frame of discernment. Applying Dempster-Shafer theory to sensor fusion is sometimes referred to as Dempster-Shafer fusion. Using Dempster-Shafer fusion, the sensor-fusion system <NUM> can also output a belief parameter (e.g., likelihood) and a plausibility parameter (e.g., confidence) associated with each of the one or more hypotheses based on the fused mass value. This document describes the operations and components of the sensor-fusion system <NUM> and the fusion module <NUM> in greater detail with respect to <FIG>.

Vehicle <NUM> can also include one or more vehicle-based systems (not illustrated in <FIG>) that can use the track assignments, the belief parameters, and the plausibility parameters from the sensor-fusion system <NUM> to operate the vehicle <NUM> on the roadway <NUM>. The vehicle-based systems can include an assisted-driving system and an autonomous-driving system (e.g., an Automatic Cruise Control (ACC) system, Traffic-Jam Assist (TJA) system, Lane-Centering Assist (LCA) system, and L0-L4 Autonomous Driving system). Generally, the vehicle-based systems can use the track assignments provided by the sensor-fusion system <NUM> to operate the vehicle and perform particular driving functions. For example, the assisted-driving system can provide automatic cruise control and monitor for the presence of the other vehicle <NUM> in the lane or an adjacent lane in which the vehicle <NUM> is traveling. As another example, the assisted-driving system can provide alerts when the other vehicle <NUM> crosses a lane marker into the same lane as the vehicle <NUM>.

The autonomous-driving system may move the vehicle <NUM> to a particular location on the roadway <NUM> while avoiding collisions with objects and the other vehicle <NUM> detected by the different systems (e.g., a radar system and a vision system) on the vehicle <NUM>. The track assignments provided by the sensor-fusion system <NUM> can provide information about the location and trajectory of the other vehicle <NUM> to enable the autonomous-driving system to perform a lane change or steer the vehicle <NUM>.

<FIG> illustrates an example automotive system <NUM> configured to perform multi-scan sensor fusion for object tracking in accordance with techniques of this disclosure. The automotive system <NUM> can be integrated within the vehicle <NUM> shown in <FIG> and is described in that context. For example, the automotive system <NUM> includes a controller <NUM> and a sensor-fusion system <NUM>-<NUM>, which is an example of the sensor-fusion system <NUM> of <FIG>. The sensor-fusion system <NUM>-<NUM> and the controller <NUM> communicate over a link <NUM>. Link <NUM> may be wired or wireless and includes a communication bus in some cases. The controller <NUM> performs operations based on information received over the link <NUM>, such as belief parameters and plausibility parameters for multiple hypotheses output from the sensor-fusion system <NUM>-<NUM> as objects in the field of view <NUM> are identified from processing and associating radar tracks and vision tracks.

The controller <NUM> includes a processor <NUM>-<NUM> (e.g., a hardware processor, a processing unit) and a computer-readable storage medium (CRM) <NUM>-<NUM> (e.g., a memory, long-term storage, short-term storage) that stores instructions for an automotive module <NUM>.

The sensor-fusion system <NUM>-<NUM> includes the vision interface <NUM>-<NUM> and the radar interface <NUM>-<NUM>. As discussed above, any number of other sensor interfaces <NUM> may be used, including a lidar interface or other sensor interface <NUM>-n. The sensor-fusion system <NUM>-<NUM> may include processing hardware that includes a processor <NUM>-<NUM> (e.g., a hardware processor, a processing unit) and a CRM <NUM>-<NUM> that stores instructions associated with a fusion module <NUM>-<NUM>, which is an example of the fusion module <NUM> of <FIG>, and a belief and plausibility module <NUM>. The fusion module <NUM>-<NUM> includes a track associations data store <NUM>-<NUM>.

The processors <NUM>-<NUM> and <NUM>-<NUM> can be two separate processing units or a single processing unit (e.g., a microprocessor). The processors <NUM>-<NUM> and <NUM>-<NUM> can also be a pair of or a single system-on-chip of a computing device, a controller, or a control unit. The processors <NUM>-<NUM> and <NUM>-<NUM> execute computer-executable instructions stored within the CRMs <NUM>-<NUM> and <NUM>-<NUM>. As an example, the processor <NUM>-<NUM> can execute the automotive module <NUM> to perform a driving function (e.g., an autonomous lane change maneuver, a semi-autonomous lane-keep feature, an ACC function, a TJA function, an LCA function, or other autonomous or semi-autonomous driving functions) or other operations of the automotive system <NUM>. Similarly, the processor <NUM>-<NUM> can execute the fusion module <NUM>-<NUM> to infer and track objects in the field of view <NUM> based on sensor data obtained from multiple different sensor interfaces <NUM> of the automotive system <NUM>. The automotive module <NUM>, when executing at the processor <NUM>-<NUM>, can receive an indication of one or more objects (e.g., the other vehicle <NUM> illustrated in <FIG>) detected by the fusion module <NUM>-<NUM> in response to the fusion module <NUM>-<NUM> combining and analyzing sensor data obtained from each of the sensor interfaces <NUM>.

Generally, the automotive system <NUM> executes the automotive module <NUM> to perform an automotive function using outputs from the sensor-fusion system <NUM>-<NUM>. For example, the automotive module <NUM> can provide automatic cruise control and monitor for objects in or near the field of view <NUM> to slow vehicle <NUM> and prevent a read-end collision with the other vehicle <NUM>. In such an example, the fusion module <NUM>-<NUM> can provide sensor data or derivative thereof (e.g., belief parameters and plausibility parameters) as output to the automotive module <NUM>. The automotive module <NUM> can also provide alerts or cause a specific maneuver when the data obtained from the fusion module <NUM>-<NUM> indicates one or more objects are crossing in front of the vehicle <NUM>.

For ease of simplicity, the track associations data store <NUM>-<NUM> and the belief and plausibility module <NUM> are described with reference primarily to the vision interface <NUM>-<NUM> and the radar interface <NUM>-<NUM>, without reference to another sensor interface <NUM>-n. However, it should be understood that the fusion module <NUM>-<NUM> can combine sensor data or tracks from more than just two different categories of sensors and can rely on sensor data output from other types of sensors besides just vision and radar systems.

The vision interface <NUM>-<NUM> provides a list of vision-based object tracks (also referred to as "vision tracks" in this disclosure). The vision interface <NUM>-<NUM> outputs sensor data, which can be provided in various forms, such as a list of candidate objects being tracked, along with estimates for each of the objects' position, velocity, object class, and reference angles (e.g., an azimuth angle to a centroid reference point on the object, such as a center of a rear face of the other vehicle <NUM>, and other extent angles to near corners of the rear of the other vehicle <NUM>).

Like the vision interface <NUM>-<NUM>, the radar interface <NUM>-<NUM> can operate independently from the vision interface <NUM>-<NUM> and the other sensor interfaces <NUM>-n. The radar interface <NUM>-<NUM> can maintain a list of detections and corresponding detection times, which are assumed to mostly be tracking on scattered centers of vehicles and other objects it detects. Each detection typically consists of a range, range rate, and azimuth angle. There is generally more than one detection on each vehicle and object unobstructed in the field of view <NUM> and at a reasonably close range to the vehicle <NUM>.

The vision interface <NUM>-<NUM> can estimate azimuth angles and object classifications more accurately than other sensor types. However, the vision interface <NUM>-<NUM> may be deficient in estimating some parameters, such as longitudinal position or range, velocity, and the like. The radar interface <NUM>-<NUM> can accurately measure object range and range rate but may be less accurate in measuring the azimuth angle, generally where vision systems are more accurate. The complementing characteristics of vision and radar systems lead to accuracy benefits in matching the vision tracks and the radar tracks.

The fusion module <NUM>-<NUM> maintains a set of associations between the vision tracks and the radar tracks in the track associations data store <NUM>-<NUM>. The fusion module <NUM>-<NUM> may determine multiple hypotheses for the set(s) of associations between the radar tracks and the vision tracks (e.g., a first hypothesis that one or more radar tracks match to a particular vision track, a second hypothesis that a specific radar track does not correspond to any vision tracks or a third hypothesis that a specific track of vision does not match to any radar tracks). In other aspects, the set(s) of associations may be based on local track confidence and temporal uncertainties.

The fusion module <NUM>-<NUM> may use Dempster-Shafer fusion to combine the mass value of the multiple hypotheses to generate fused mass values. The belief and plausibility module <NUM> can use the fused mass values to generate belief and plausibility parameters for the track associations within each set. The belief and plausibility module <NUM> can also output the belief and plausibility parameters to the automotive module <NUM>.

<FIG> illustrates an example conceptual diagram <NUM> illustrating techniques to perform multi-scan sensor fusion for object tracking in accordance with this disclosure. The conceptual diagram <NUM> includes the fusion module <NUM>-<NUM>, which is an example of the fusion module <NUM> of <FIG> and the fusion module <NUM>-<NUM> of <FIG>.

Vision tracks <NUM> and radar tracks <NUM> are provided as input to the fusion module <NUM>-<NUM>. The vision tracks <NUM> are generated based on vision data. Similarly, the radar tracks <NUM> are generated based on radar data. The fusion module <NUM>-<NUM> includes an input processing module <NUM>, an evidence fusion module <NUM>, and an output processing module <NUM>. The fusion module <NUM>-<NUM> outputs matches <NUM>, which can be provided to the automotive module <NUM>. In general, the input processing module <NUM> can preprocess the vision tracks <NUM> and the radar tracks <NUM> by performing track compensation <NUM>, assess feasibility pairs of vision tracks <NUM> and radar tracks <NUM> by performing feasibility matrix formation <NUM>, and generate mass values for each pair by performing mass extraction <NUM>.

The fusion module <NUM>-<NUM> uses a frame of discernment (FOD) that is defined as a set that describes all possible hypotheses of matching the vision tracks <NUM> to the radar tracks <NUM>. The elements of a FOD are mutually exclusive and exhaustive. For convenience, the vision tracks <NUM> and the radar tracks <NUM> are represented by Equations (<NUM>) and (<NUM>), respectively: <MAT> <MAT> where M represents the number of vision tracks <NUM> and N represents the number of radar tracks <NUM>. The number M of vision tracks <NUM> can be less than, equal to, or greater than the number N of radar tracks <NUM>.

The input processing module <NUM> can define a first FOD by considering matching each radar track <NUM> to the vision tracks <NUM>. The first FOD is defined by Equation (<NUM>): <MAT> where gi denotes the index of feasible vision tracks <NUM> after the gating step, ri is the i-th radar track <NUM>, and Y indicates that ri does not associate with any vision track <NUM>.

The input processing module <NUM> can also define a second FOD by considering matching each vision track <NUM> to the radar tracks <NUM>. The second FOD is defined by Equation (<NUM>): <MAT> where gi denotes the index of feasible radar tracks <NUM> after the gating step, vi is the i-th vision track <NUM>, and Ξ denotes that ri does not associate with any radar track <NUM>.

The input processing module <NUM> assigns a mass value for each element in the FODs based on a different source of information. An individual vision track <NUM>, Tv, and an individual radar track <NUM>, Tr, can be represented, respectively, by Tv = {x<NUM>, x<NUM>, x<NUM>,. , xn<NUM>} with covariance matrix {P<NUM>, P<NUM>, P<NUM>,. , Pn<NUM>} and by Tr = {x̂<NUM>, x̂<NUM>, x̂<NUM>,. , x̂n<NUM>} with covariance matrix {P̂<NUM>, P̂<NUM>, P̂<NUM>,. , P̂n<NUM>}. For convenience, it is assumed that a length of each track is the same (e.g., n<NUM> = n<NUM> = n).

The mass extraction <NUM> can be performed based on distance, bounding box size, object class information, or object heading information. The input processing module <NUM> can calculate the mass value using a distance between two tracks. The distance may be calculated without consideration of covariance using Equation (<NUM>): <MAT>.

Alternatively, the distance between two tracks may be calculated with consideration of covariance using Equation (<NUM>): <MAT>.

The distance between two tracks can also be calculated using the KL divergence as illustrated by Equation (<NUM>): <MAT>.

Given a set of vision tracks <NUM> and radar tracks <NUM>, the input processing module <NUM> can determine the mass values for the first FOD as a function of the distance using Equations (8a) and (8b): <MAT> <MAT>.

The gating threshold can be set for d ≤ α and dΥ = α. For each evidence source, the input processing module <NUM> can assume the threshold is fixed. For the second FOD, the input processing module <NUM> can similarly calculate the mass values as a function of the distance.

The input processing module <NUM> can also calculate the mass value for associations of vision tracks <NUM> and radar tracks <NUM> as a function of the likelihood of their association. From the probability perspective, the probability density function of Tv can be represented by Equation (<NUM>): <MAT>.

The similarity is then described by the negative log-likelihood (NLL) in Equation (<NUM>): <MAT>.

Given a set of vision tracks <NUM> and radar tracks <NUM>, the input processing module <NUM> can determine the mass values for the first FOD as a function of the similarity using Equations (11a) and (11b): <MAT> <MAT>.

For the second FOD, the input processing module <NUM> can similarly calculate the mass values as a function of the similarity with χ ∈ Ωvi→r.

The input processing module <NUM> can also calculate the mass value for associations of vision tracks <NUM> and radar tracks <NUM> as a function of bounding box size information. The bounding box information can be used based on the assumption that an object size is the same in each of its tracks and the matching of those tracks is perfect. Based on these assumptions, the input processing module <NUM> can use the area of the bounding box to calculate mass values. For convenience, the area of the vision track <NUM> is given by Av = {a<NUM>, a<NUM>, a<NUM>,. Similarly, the area of the radar track <NUM> is given by Ar = {â<NUM>, â<NUM>, â<NUM>,. The distance of the area information between two tracks is given by Equation (<NUM>): <MAT>.

Rather than using the area, the input processing module <NUM> can similarly use the length and width directly, as shown in Equation (<NUM>): <MAT> where lhk denotes the length and height information.

Given a set of vision tracks <NUM> and radar tracks <NUM>, the input processing module <NUM> can determine the mass values for the first FOD as a function of the distance of the area from Equations (<NUM>) or (<NUM>) using Equations (14a) and (14b): <MAT> <MAT>.

For the second FOD, the input processing module <NUM> can similarly calculate the mass values as a function of the bounding box size information with χ ∈ Ωvi→r.

The input processing module <NUM> can also calculate the mass value for associations of vision tracks <NUM> and radar tracks <NUM> as a function of object class information. The object class can be defined as C = {c<NUM>, c<NUM>,. Accordingly, the class of the vision track <NUM> is given by Cv = {C<NUM>, C<NUM>, C<NUM>,. Similarly, the class of the radar track <NUM> is given by Cr = {Ĉ<NUM>, Ĉ<NUM>, Ĉ<NUM>,. The distance of the class information between two tracks is given by Equation (<NUM>): <MAT> where H(Ck, Ĉk) is the cross-entropy that is described by Equation (<NUM>): <MAT>.

Given a set of vision tracks <NUM> and radar tracks <NUM>, the input processing module <NUM> can determine the mass values for the first FOD as a function of the distance of the object class information using Equations (17a) and (17b): <MAT> <MAT>.

For the second FOD, the input processing module <NUM> can similarly calculate the mass values as a function of the object class information with χ ∈ Ωvi→r.

The input processing module <NUM> can also calculate the mass value for associations of vision tracks <NUM> and radar tracks <NUM> as a function of object heading information.

The input processing module <NUM> can also calculate the mass value for associations of vision tracks <NUM> and radar tracks <NUM> as a function of object class information. The input processing module <NUM> considers the latest object heading information because this information is partially captured by the position information. The heading of the vision track <NUM> is given by hv. Similarly, the heading of the radar track <NUM> is given by hr. The distance of the heading information between two tracks is given by Equation (<NUM>): <MAT>.

Given a set of vision tracks <NUM> and radar tracks <NUM>, the input processing module <NUM> can determine the mass values for the first FOD as a function of the distance of the heading information using Equations (19a) and (19b): <MAT> <MAT>.

For the second FOD, the input processing module <NUM> can similarly calculate the mass values as a function of the object class information with χ ∈ Ωvi→r. The input processing module <NUM> can similarly use velocity information to extract mass values.

The evidence fusion module <NUM> can determine a belief parameter and a plausibility parameter for each pair by performing stability score calculations <NUM>, generate fused mass values via Dempster-Shafer fusion by performing mass fusion <NUM>, and determine a probability parameter by performing probability calculation <NUM>.

The evidence fusion module <NUM> can use Dempster-Shafer fusion to fuse the mass values of multiple hypotheses. First, based on the mass values for different sources, the evidence fusion module <NUM> uses the Dempster-Shafer theory to obtain the fused mass value for a specific hypothesis. Two information sources can be fused to determine the joint mass value m<NUM>,<NUM>(·) using Equations (<NUM>) through (<NUM>): <MAT> <MAT> where <MAT>.

The expression (<NUM> - K) is the normalization coefficient, in which K denotes the conflict between the evidence, m<NUM>(·) and m<NUM>(·) represent the mass value corresponding to a specific hypothesis for the first evidence and the second evidence, respectively, and B and C are the hypotheses determined by the FOD. The evidence fusion module <NUM> may include a calculation of mass values for each element in a given frame of discernment based on a different source of information (e.g., distance, similarity, track heading information, bounding box information, object class information, or object heading information).

Second, based on the fused mass value, evidence fusion module <NUM> calculates the belief parameter and plausibility parameter. The belief bel(H) and plausibility pl(H) of a specific hypothesis H are given by Equations (<NUM>) and (<NUM>): <MAT> <MAT>.

At initialization, the evidence fusion module <NUM> does not have a mass value available and only uses information from measurements.

The output processing module <NUM> can identify hypotheses without conflict by performing matched hypotheses extraction <NUM> and determine hypotheses with conflicts by performing conflict hypotheses maintenance <NUM>. The output processing module <NUM> outputs the matches <NUM> with improved accuracy as a result of the described processing techniques.

The output processing module <NUM> can extract associations using a probability matrix for the first FOD (described in greater detail with respect to <FIG>) or the second FOD. In particular, the output processing module <NUM> can extract associations that exceed a pre-defined decision threshold (e.g., <NUM>) for probability for the first FOD. If no associations exceed the decision threshold, the output processing module <NUM> can extract associations with ambiguity that exceed a pre-defined significant threshold (e.g., <NUM>). The same extractions are then performed for the second FOD. The extracted associations or hypotheses without a conflict between the first and second FOD and exceed the decision threshold are maintained in a data store. If a conflict between the association exists between the probability matrices or associations that exceed the significant threshold, these hypotheses are maintained in a conflict data store. Hypotheses in the conflict data store are not output as the track associations, but the output processing module <NUM> maintains these hypotheses for the next scan. In the next scan, the output processing module <NUM> prunes a conflict hypothesis if the probability is lower than the significant threshold or accepts the conflict hypothesis if the probability is higher than the decision threshold. The output processing module <NUM> maintains the conflict hypothesis for tracking if it passes the significant threshold but does not pass the decision threshold.

<FIG> illustrates an example diagram <NUM> to calculate distances in generating mass values for multi-scan sensor fusion in accordance with this disclosure. The diagram <NUM> includes a vision track <NUM> defined by points x<NUM>, x<NUM>, x<NUM>, x<NUM>, x<NUM>, x<NUM> (points <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, respectively) and a radar track <NUM> defined by x̂<NUM>, x̂<NUM>, x̂<NUM>, x̂<NUM>, x̂<NUM>, x̂<NUM> (points <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, respectively). The input processing module <NUM> can calculate the distance between a point on the vision track <NUM> and a point on the radar track <NUM> (e.g., the distance between point <NUM> and point <NUM>). For example, a distance <NUM> may be calculated between x<NUM> and x̂<NUM>. In the example diagram <NUM>, the input processing module <NUM> assumes the radar track length is the same as the vision track length. Without consideration of covariance, the distance calculation can be defined by Equation (<NUM>): <MAT>.

Alternatively, with consideration of a covariance matrix defined by P<NUM>, P<NUM>, P<NUM>,. Pn<NUM>, the distance calculation may be defined by Equation (<NUM>): <MAT>.

<FIG> illustrates an example probability matrix <NUM> for hypotheses of associating vision tracks <NUM> and radar tracks <NUM>. The probability matrix <NUM> is formed for the first FOD (e.g., Ωri→v). As described above, a FOD is defined as a set that describes all possible hypotheses of associating the radar tracks <NUM> to the vision tracks <NUM>, or vice versa. The probability matrix <NUM> indicates the probability of each possible pair of track associations in the first FOD. In particular, the probability matrix <NUM> considers a set of radar tracks <NUM> and a set of vision tracks <NUM>. In the probability matrix <NUM>, each element pij is the probability to associate Tri with Tvj based on the mass function corresponding to the first FOD. For example, the probability to associate the first radar track <NUM> (Tr<NUM>) with the first vision track <NUM> (Tv<NUM>) is the probability <NUM> (p<NUM>).

A similar probability matrix can be formed for the second FOD (e.g., Ωvi→r). In that probability matrix, each element p̃ij is the probability to associate Tvj with Tri based on the mass function corresponding to the second FOD. In the second FOD, the probability matrix can also consider the possibility that a single vision track matches to multiple radar tracks. The probabilities can be determined using the pignistic transformation. After gating or clustering, many elements in the probability matrices are zero and fewer possible associations need to be considered.

<FIG> illustrates an example flowchart <NUM> of multi-scan sensor fusion for object tracking in accordance with this disclosure. The operations of the flowchart <NUM> can be performed by the fusion module <NUM>, <NUM>-<NUM>, or <NUM>-<NUM> of <FIG> or a similar fusion module. The flowchart <NUM> can also include fewer or additional operations than those described with reference to <FIG>.

At <NUM>, the fusion module <NUM> receives or obtains vision tracks and radar tracks. For example, the fusion module <NUM> can receive the vision tracks and the radar tracks via the vision interface <NUM>-<NUM> and the radar interface <NUM>-<NUM>, respectively. In other implementations, the fusion module <NUM> can receive vision data and radar data via the vision interface <NUM>-<NUM> and the radar interface <NUM>-<NUM>, respectively.

At <NUM>, the fusion module <NUM> uses multiple scans to initialize the track matching and then uses a sliding window to continue processing matches. The use of multiple scans is described in greater detail with respect to <FIG>.

At <NUM>, the fusion module <NUM> can perform clustering to reduce the number of tracks to process and consider. The clustering can be based on the geometric distance or probability density. It is assumed that the tracks are aligned well in terms of time and the Euclidean distance is used as the metric to cluster tracks. If the time is not well aligned for different tracks, the dynamic time warping distance metric (DTW) can be used.

At <NUM>, the fusion module <NUM> determines whether each radar track and each vision track has previously been matched or associated with another track. If the track has been previously matched, then the fusion module <NUM> calculates a stability score at <NUM> for the match or association. At <NUM>, the fusion module <NUM> determines whether the stability score is greater than the decision threshold. If the stability score is greater than the decision threshold, then the association of the radar track and vision track is output as a match at <NUM>. The fusion module <NUM> can skip the matching process if the matched radar track and vision track are maintained with good confidence it was verified in the previous iteration of the flowchart <NUM>. The fusion module <NUM> uses the stability score to consider the temporal uncertainty of the current window and the uncertainty of how well the track was maintained by each tracker. The stability score is defined using Equation (<NUM>): <MAT> where Δt is the distance of time between current scan and the last scan in which the match was verified. Sv,Δt is defined using Equations (<NUM>) and (<NUM>): <MAT> where <MAT> pd is the detection probability and <MAT> is the likelihood for the tracker. Similarly, Sr,Δt is defined using Equations (<NUM>) and (<NUM>): <MAT> where <MAT> The fusion module <NUM> uses a threshold R to determine whether the matching operation can be skipped in a particular iteration of the flowchart <NUM>. The fusion module <NUM> can also define a break threshold Nbreak as the number of unavailable scans from a specific track. When the number of unavailable scans is number is greater or equal to the break threshold, the matching breaks up. A sliding window is used again for that pair of track associations.

At <NUM>, if the track has not been previously matched or if the stability score is less than the decision threshold, then the fusion module <NUM> calculates a mass value for each potential hypothesis to associate the unmatched track with other tracks using all available measurements. The hypotheses can include one or more radar tracks match to a particular vision track, a radar track does not match to a vision track, or a vision track does not match to a radar track. At <NUM>, after calculating the mass value for each hypothesis, the fusion module <NUM> identifies consistent pairs and conflicting pairs as described with respect to <FIG>. At <NUM>, a state update of the matched pairs is provided to the automotive module <NUM>.

<FIG> illustrate the use of multiple-scan windows to perform multi-scan sensor fusion for object tracking in accordance with this disclosure. <FIG> illustrates an example diagram <NUM> to initialize multiple-scan windows. A window (e.g., windows <NUM>, <NUM>, and <NUM>) may be used for maintaining multiple scan information rather than a single scan by "sliding" the window based on certain criteria. The window may also be used for input selection to the matching algorithm detailed in <FIG>. For example, consider individual vision track information x<NUM>, x<NUM>, x. , xwz, xwz+<NUM>, xwz+<NUM> (vision track data <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, respectively) and radar track information x̂<NUM>, x̂<NUM>, x̂. , x̂wz, x̂wz+<NUM>, x̂wz+<NUM> (radar track data <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, respectively). Individual tracks may be assigned to a window and processed accordingly. For example, a first window <NUM>, a second window <NUM>, and a third window <NUM> are processed sequentially. Because the length of each vision track and radar track is less than the size of an individual window or scan, the track information is accumulated in the sliding windows. The currently available information for each track (e.g., within a particular window) is used to calculate the similarity between different vision tracks and radar tracks.

<FIG> illustrates an example diagram <NUM> that demonstrates the use of a sliding window to perform multi-scan sensor fusion for object tracking. A first window <NUM> at time k includes track information from multiple scans (e.g., vision track data <NUM>, <NUM>, <NUM>, and <NUM> and radar track data <NUM>, <NUM>, <NUM>, and <NUM>). A second window <NUM> at time k+<NUM> includes track information from multiple scans (e.g., vision track data <NUM>, <NUM>, <NUM>, and <NUM> and radar track data <NUM>, <NUM>, <NUM>, and <NUM>). The first window <NUM> and the second window <NUM> represent distinct windows. The track information used at time k+<NUM> may be obtained by "sliding" the window forward a single step or scan from time k. Sliding windows allow the fusion module <NUM> to maintain a history of track data and hypotheses that can then be used to minimize the number of potential matches or reduce the matching complexity.

<FIG> illustrates an example flowchart <NUM> of a method to perform multi-scan sensor fusion for object tracking in accordance with this disclosure. The example flowchart <NUM> is shown as a set of operations <NUM> through <NUM>, which are performed in, but not limited to, the order or combinations in which the operations are shown or described. Further, any of the operations <NUM> through <NUM> may be repeated, combined, or reorganized to provide other methods. In portions of the following discussion, reference may be made to the environment <NUM> and entities detailed above, reference to which is made for example only. The techniques are not limited to performance by one entity or multiple entities.

At <NUM>, a sensor-fusion system of a vehicle obtains radar tracks and vision tracks generated for an environment of a vehicle. For example, the sensor-fusion system <NUM> of the vehicle <NUM> can obtain the vision tracks <NUM> and the radar tracks <NUM> that are generated for the environment <NUM>. The vision tracks <NUM> are generated based on vision data from one or more vision sensors on the vehicle <NUM>. The radar tracks <NUM> are generated based on radar data from one or more radar sensors on the vehicle <NUM>. The radar sensors have a field of view that at least partially overlaps with a field of view of the vision sensors.

At <NUM>, the sensor-fusion system maintains at least one set of hypotheses for associations between the vision tracks and the radar tracks based on multiple scans of the radar data and the vision data. For example, the sensor-fusion system <NUM> can maintain multiple sets of hypotheses for associations between the vision tracks <NUM> and the radar tracks <NUM> based on multiple scans of the radar data and the vision data. The hypotheses include at least one of one or more radar tracks matching a vision track, a vision track matching one or more radar tracks, a radar track not matching a vision track, or a vision track not matching a radar track. In particular, the hypotheses can include a first set of hypotheses (FOD) and a second set of hypotheses (FOD). The first set of hypotheses can include at least one of one or more radar tracks matching a vision track and a radar track not matching a vision track. The second set of hypotheses can include at least one of a vision track matching one or more radar tracks and a vision track not matching a radar track.

The sensor-fusion system <NUM> uses a window of multiple scans of the radar tracks and the vision tracks that includes the radar data and the vision data from a current sensor scan and at least one previous sensor scan. The window steps forward by a single step in each iteration of operation <NUM>. The window initializes with a single scan of the radar tracks and the vision tracks and accumulates additional scans of the radar tracks and the vision tracks in each iteration of the operation <NUM> until the window reaches a predetermined window size.

The sensor-fusion system <NUM> can also determine mass values for the associations between the vision tracks <NUM> and the radar tracks <NUM>. The mass values can be assigned based on distances between tracks, similarities between tracks, heading information, bounding box information, object class information, velocity information, or object heading information. In some implementations, the mass values can be assigned based on at least three of distances between tracks, similarities between tracks, heading information, bounding box information, object class information, velocity information, or object heading information. The sensor-fusion system <NUM> can determine based on the mass values a belief parameter and a plausibility parameter for each hypothesis. The mass values can be combined using a Dempster-Shafer fusion process to generate a fused mass value. The belief parameter and the plausibility parameter can be determined based on the fused mass values.

At <NUM>, the sensor-fusion system determines a probability value for each hypothesis of the at least one set of hypotheses based on mass values. For example, the sensor-fusion system <NUM> can determine a probability matrix for each hypothesis of the sets of hypotheses based on the mass values. The probability value indicates a likelihood that a particular hypothesis is accurate.

At <NUM>, the sensor-fusion system determines one or more matches between one or more radar tracks and a vision track based on the probability value for each hypothesis. For example, the sensor-fusion system <NUM> can determine matches between one or more radar tracks and a vision track based on the probability value for the hypotheses. The matches include one or more hypotheses with probability values that exceed a decision threshold. The sensor-fusion system <NUM> can also extract one or more first hypotheses from the first set of hypotheses and one or more second hypotheses from the second set of hypotheses that have probability values that exceed the decision threshold. The sensor-fusion system <NUM> can then identify the matches as those hypotheses among the first hypotheses and the second hypotheses that exceed the decision threshold.

The sensor-fusion system <NUM> can extract one or more third hypotheses from the first set of hypotheses and one or more fourth hypotheses from the second set of hypotheses with probability values that exceed a significant threshold responsive to no hypothesis among the first set of hypotheses or the second set of hypotheses exceeding the decision threshold. The significant threshold has a lower value (e.g., <NUM>) than the decision threshold (e.g., <NUM>). The sensor-fusion system <NUM> can maintain conflict hypotheses among the first hypotheses and the second hypotheses that do not both exceed the decision threshold and hypotheses that exceed the significant threshold. The sensor-fusion system <NUM> can skip verifying the matches in a subsequent iteration of the operation <NUM> if a stability score of the matches is above a threshold value. The stability score indicates a local track confidence and a temporal uncertainty associated with each match. The sensor-fusion system <NUM> can also maintain the one or more third hypotheses and the one or more fourth hypotheses.

At <NUM>, the sensor-fusion system outputs the one or more matches to a semi-autonomous or autonomous driving system of the vehicle to a tracker. The output of the tracker is then used to control operation of the vehicle. For example, the sensor-fusion system <NUM> can output the matches to a tracker. The output of the tracker can be provided to the automotive module <NUM> to control operation of the vehicle <NUM>. The sensor-fusion system <NUM> can also outputting the belief parameter and the plausibility parameter for the matches to further update the matches and control operation of the vehicle <NUM> based on the belief parameter and the plausibility parameter.

Claim 1:
A method comprising:
iteratively maintaining (<NUM>, <NUM>) at least one set of hypotheses for associations between radar tracks (<NUM>) and vision tracks (<NUM>) within a sliding window (<NUM>, <NUM>, <NUM>, <NUM>) that includes at least two consecutive scans of both radar data for an environment (<NUM>) of a vehicle (<NUM>) obtained using one or more radar sensors on the vehicle (<NUM>) and vision data for the environment (<NUM>) of the vehicle (<NUM>) obtained using one or more vision sensors on the vehicle (<NUM>) for each of the at least two consecutive scans included in the sliding window (<NUM>, <NUM>, <NUM>, <NUM>),
obtaining (<NUM>) the radar tracks (<NUM>) and the vision tracks (<NUM>) based on the radar data from the one or more radar sensors on the vehicle (<NUM>) and the vision data from the one or more vision sensors on the vehicle (<NUM>), respectively, the one or more radar sensors having a first field of view that at least partially overlaps with a second field of view of the one or more vision sensors,
maintaining (<NUM>), based on the at least two consecutive scans of each of the radar data and the vision data, at least one set of hypotheses for associations between the vision tracks (<NUM>) and the radar tracks (<NUM>), the at least one set of hypotheses including mass values for the associations between the vision tracks (<NUM>) and the radar tracks (<NUM>) within the sliding window (<NUM>, <NUM>, <NUM>, <NUM>), and
sliding the sliding window (<NUM>, <NUM>, <NUM>, <NUM>) such that a first one of the at least two consecutive scans is removed from the sliding window (<NUM>) and a next scan consecutively after the two consecutive scans is added to the sliding window (<NUM>);
determining (<NUM>), based on the mass values, a probability value for each hypothesis of the at least one set of hypotheses within the sliding window (<NUM>, <NUM>, <NUM>, <NUM>), the probability value indicating a likelihood that each hypothesis is an accurate association between the vision tracks (<NUM>) and the radar tracks (<NUM>);
determining (<NUM>), based on the probability value for each hypothesis, one or more matches between one or more radar tracks (<NUM>) and a vision track (<NUM>), the one or more matches comprising one or more hypotheses with probability values that exceed a decision threshold; and
outputting (<NUM>), to a semi-autonomous or autonomous driving system of the vehicle (<NUM>), the one or more matches to control operation of the vehicle (<NUM>).