Patent Description:
Air traffic control systems and unmanned traffic management systems monitor aerial vehicles that utilize airspace. A variety of different instruments, sensors, or other monitoring systems may be used simultaneously to track objects (e.g., aerial vehicles). Each sensor may track objects independently. As such, a particular object may be detected by multiple sensors. However, due to measurement uncertainty, instrument error, or other factors, different sensors may report the position of the same object differently. Thus, the actual position of the object may be unclear. Therefore, there is a need for systems and methods to correlate airspace surveillance data from multiple sensors. <NPL> discloses the use of multiple cameras to detect, track and localize multiple moving objects, with the ability to use multiple cameras from different viewing angles and simultaneously track moving objects in the cameras' frames. <NPL>, discloses a low computational overhead density-based fuzzy clustering method for solving the track-to-track association problem in distributed multi-sensor tracking systems. The method gets tracks of targets from different sensors, combines them at a fusion center and then relates them. After that, it uses prior estimated state of each target as basis for determining fuzzy membership of reported tracks of sensors that are related to a single target. The membership degree of each track to a cluster is determined based on the principle of maximum entropy. A weighted combinations of reported clustered tracks of a single target is applied to produce a fused target state.

The invention concerns a method and apparatus as set out in independent claims <NUM> and <NUM>, respectively. Embodiments of the invention are defined as set out in dependent method claims <NUM>-<NUM> and apparatus claims <NUM> and <NUM>.

These and other features, and characteristics of the present technology, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. As used in the specification and in the claims, the singular form of 'a', 'an', and 'the' include plural referents unless the context clearly dictates otherwise.

The present disclosure generally relates to a system for correlating air surveillance data from multiple sensors. The present disclosure is directed to monitoring unmanned aircraft systems (UAS). However, it should be understood that, in other embodiments, the present disclosure may apply to monitoring manned aircraft system or any other airborne objects.

The ability to detect and avoid other aircraft is key to operation of a UAS. For scalable, beyond visual line of sight operations, this requires relying on sensors to perform airspace surveillance and report on proximate air traffic. Any detected targets or other air traffic may be displayed to a human operator of a UAS such that the human operator may gain situational awareness of the airspace and adjust the flight of the UAS accordingly. This airspace surveillance may be performed by a UAS service supplier (USS) or an unmanned traffic management (UTM) system.

A UTM system comprises one or more USS. A USS may manage UAS traffic within a certain geographic area and/or for a certain set of clients. A USS may monitor UAS with a plurality of sensors that gather air surveillance data, including ground based radar tracking using primary and/or secondary radar, and/or by receiving telemetry directly from UAS, that identifies their position. In addition to tracking the position of UAS, a USS may communicate with UAS operators to provide instructions to guide UAS along certain routes to avoid collisions with other UAS and to otherwise manage airspace.

While a single USS may cover a certain geographic area, a plurality of USS may be part of a UTM network to manage air traffic over a larger geographic area. Different USS that are part of a UTM network may communicate with each other to jointly manage UAS air traffic (e.g., one USS may monitor a particular UAS and may hand off control to another USS as the USS is leaving its airspace). In addition, multiple USS may cover overlapping geographic areas, in which case they may communicate with each other to jointly ensure aircraft separation.

When multiple sensors of a UTM monitor overlapping airspace, each sensor may detect a position of an object within the airspace. Furthermore, due to measurement error or uncertainty, instrument error, random noise, or other factors, each sensor may detect a different position for the same object. Accordingly, each sensor may report a different position of the object to the UTM monitoring the airspace. Thus, merely displaying data from all sensors can lead to a cluttered display causing confusion regarding the number and state of actual targets in the airspace. Accordingly, it is desirable to appropriately fuse surveillance data from multiple sensors to provide a combined air picture where each detected target is uniquely represented. Such fusion of sensor data is also helpful in providing algorithmic detect and avoid alerting and guidance.

Disclosed herein are systems and methods of fusing sensor data from multiple sensors to provide a combined air picture to a human operator of a UAS. As disclosed herein, a USS may receive surveillance data from multiple sensors that monitor air traffic. As disclosed herein, the USS may receive formed track data comprising a recent history of positions of targets detected by each sensor.

After receiving surveillance data comprising tracks of detected targets, the USS may aggregate the data from each track. This aggregated data may comprise positions of all targets detected within a threshold duration in the past. The USS may then use a clustering algorithm to perform density-based clustering to determine different clusters of the target positions of the aggregated data.

The USS may then determine a plurality of candidate track-to-cluster associations and rank the candidate associations based on the number of elements from each track in each cluster. The USS may then associate each track with a particular target based on the determined track-to-cluster associations. If multiple sensors have detected the same target, then some targets may be associated with multiple tracks. As such, the USS may estimate a single track for each target based on the tracks associated with each target. The single determined track for each target may then be presented to a user.

<FIG> depicts an example system <NUM> for correlating air surveillance data from multiple sensors. In the example of <FIG>, a plurality of UAS <NUM> (e.g., drones) flying in certain airspace are monitored by a surveillance service provider <NUM> such as a USS and/or a surveillance supplementary data service provider (SDSP). Specifically, the positions of each of the UAS <NUM> are to be tracked by the surveillance service provider <NUM>. However, it should be understood that in other examples, the system <NUM> may track manned aircraft or other types of aerial vehicles and may include tracking systems other than a USS (e.g., a traditional air traffic control system).

The system <NUM> may include a plurality of surveillance instruments or sensors. The system <NUM> may include one or more cooperative sensors and/or one or more non-cooperative sensors. Cooperative sensors rely on cooperative targets that identify themselves, such as or automatic dependent surveillance-broadcast (ADS-B). As such, cooperative sensors are able to establish an identity of a target being tracked. Alternatively, non-cooperative sensors do not communicate with the target and are unable to directly establish an identify of a target.

In the example of <FIG>, the system <NUM> includes a first sensor <NUM> and a second sensor <NUM>. Each of the first and second sensors <NUM>, <NUM> may track the positions of the UAS <NUM> and/or other targets and transmit air surveillance data to the surveillance service provider <NUM>. Each of the first and second sensors <NUM>, <NUM> may comprise a variety of sensors or surveillance instruments including ground-based primary radar, ADS-B receivers, cellular data receivers, receivers that receive GPS data from the UAS <NUM> (including but not limited to Remote ID, or UTM provided telemetry data), SSR/MLAT systems, or any other electro-optic or other types of sensors that can track the positions of the UAS <NUM>. Each of the first and second sensors <NUM>, <NUM> may comprise cooperative or non-cooperative sensors. While the example of <FIG> shows a first sensor <NUM> and a second sensor <NUM>, it should be understood that, in other examples, the system <NUM> may include any number of sensors that track the positions of the UAS <NUM> and transmit data to the surveillance service provider <NUM>.

In the example of <FIG>, the first and second sensors <NUM>, <NUM> transmit data to the surveillance service provider <NUM> over a network <NUM>. As such, the network <NUM> may introduce latency and drop-outs. This may introduce error into the data received by the surveillance service provider <NUM> from the first and second sensors <NUM>, <NUM>. The disclosed techniques for correlating data from multiple sensors may allow the surveillance service provider <NUM> to accurately track the UAS <NUM> despite the error introduced by the network <NUM> as well as other sources of error.

The surveillance service provider <NUM> may receive surveillance data from the first sensor <NUM> and the second sensor <NUM>. The data received by the surveillance service provider <NUM> from the first and second sensors <NUM>, <NUM> may include positions of the UAS <NUM> measured by each of the sensors <NUM>, <NUM>. In embodiments, the data received by the surveillance service provider <NUM> from the first and second sensors <NUM>, <NUM> comprises a formed target track rather than raw detection data. That is, each of the first and second sensors <NUM>, <NUM> transmits, to the surveillance service provider <NUM>, a track of a detected object comprising a plurality of recently detected positions of the object. If a sensor is tracking multiple objects, the sensor may transmit multiple tracks corresponding to the multiple objects to the surveillance service provider <NUM>.

For example, <FIG> shows two tracks <NUM>, <NUM> that may be transmitted by the first sensor <NUM> and two tracks <NUM>, <NUM> that may be transmitted by the second sensor <NUM>. In the example of <FIG>, the track transmitted by the first sensor <NUM> comprises the five most recent positions of a target and the track transmitted by the second sensor <NUM> comprises the seven most recent positions of a target. However, it should be understood that in other examples, sensors may transmit tracks comprising any number of recent positions of a target.

The surveillance service provider <NUM> may combine the data received from the first and second sensors <NUM>, <NUM> to determine accurate positions of the UAS <NUM>, using the techniques described herein. In some examples, the surveillance service provider <NUM> may receive data directly from the UAS <NUM>.

Now referring to <FIG>, components of the surveillance service provider <NUM> are schematically depicted. In the illustrated example, the surveillance service provider <NUM> comprises a server computing device. However, in other examples, the surveillance service provider <NUM> may be any type of computing device (e.g., mobile computing device, personal computer, etc.). Additionally, while the surveillance service provider <NUM> is depicted in <FIG> as a single piece of hardware, this is also merely an example. More specifically, the surveillance service provider <NUM> may represent a plurality of computers, servers, databases, etc. In some examples, the surveillance service provider <NUM> may be configured as a general-purpose computer with the requisite hardware, software, and/or firmware. In other examples, the surveillance service provider <NUM> may be configured as a collection of cooperating computing devices or even as a special purpose computer designed specifically for performing the functionality described herein.

As illustrated in <FIG>, the surveillance service provider <NUM> may include a processor <NUM>, input/output hardware <NUM>, network interface hardware <NUM>, a data storage component <NUM>, and a non-transitory memory component <NUM>. The memory component <NUM> may be configured as volatile and/or nonvolatile computer readable medium and, as such, may include random access memory (including SRAM, DRAM, and/or other types of random access memory), flash memory, registers, compact discs (CD), digital versatile discs (DVD), and/or other types of storage components. Additionally, the memory component <NUM> may be configured to store operating logic <NUM>, a surveillance data reception module <NUM>, a data aggregation module <NUM>, a clustering module <NUM>, a track-to-cluster association module <NUM>, a track-to-target association module <NUM>, and a target state estimation module <NUM> (each of which may be embodied as a computer program, firmware, or hardware, as an example). A network interface <NUM> is also included in <FIG> and may be implemented as a bus or other interface to facilitate communication among the components of the surveillance service provider <NUM>.

The processor <NUM> may include any processing component configured to receive and execute instructions (such as from the data storage component <NUM> and/or the memory component <NUM>). The input/output hardware <NUM> may include a monitor, keyboard, mouse, printer, camera, microphone, speaker, touch-screen, and/or other device for receiving from, and sending data to the surveillance service provider <NUM>. The network interface hardware <NUM> may include any wired or wireless networking hardware, such as a modem, LAN port, wireless fidelity (Wi-Fi) card, WiMax™ card, mobile communications hardware, and/or other hardware for communicating with the first and second sensors <NUM>, <NUM>, the UAS <NUM>, and other networks and/or devices.

The data storage component <NUM> may store surveillance data received from the sensors <NUM>, <NUM> (e.g., previous tracked positions of the UAS <NUM>). By storing sensor data received from the sensors <NUM>, <NUM>, the surveillance service provider <NUM> may analyze tracks of the various UAS <NUM>. As used herein, a track comprising a sequence of measured positions of a target. The use of the track data stored in the data storage component <NUM> is discussed in further detail below.

Included in the memory component <NUM> are operating logic <NUM>, the surveillance data reception module <NUM>, the data aggregation module <NUM>, the clustering module <NUM>, the track-to-cluster association module <NUM>, the track-to-target association module <NUM>, and the target state estimation module <NUM>. The operating logic <NUM> may include an operating system and/or other software for managing components of the surveillance service provider <NUM>.

The surveillance data reception module <NUM> may receive data from the first and second sensors <NUM>, <NUM>. In examples comprising additional sensors, the surveillance data reception module <NUM> may receive data from additional sensors as well. The data received by the surveillance data reception module <NUM> may comprise positions of one or more targets (e.g., the UAS <NUM>) tracked by the first and second sensors <NUM>, <NUM>. As explained above, the data transmitted by the first and second sensors <NUM>, <NUM> may comprise tracks of targets. As such, the surveillance data reception module <NUM> may receive tracks of targets. The received data may also comprise a time stamp associated with each data point (e.g., each detected position of a target). After receiving surveillance data, the surveillance data reception module <NUM> may store the received data in the data storage component <NUM>.

The data aggregation module <NUM> aggregates data from all of the tracks received by the surveillance data reception module <NUM> up to a certain predetermined threshold duration in the past (e.g., five seconds). That is, all received positions for all tracks within the threshold duration are combined. The aggregated data may include time stamps, target locations, target altitudes, velocity, bearings or headings, and the like. <FIG> shows an example aggregation of tracks <NUM>, <NUM>, <NUM>, and <NUM> that may be performed by the data aggregation module <NUM>. In the example of <FIG>, the tracks <NUM> and <NUM> represent tracks of a first target detected by the first and second sensors <NUM>, <NUM>, respectively and the tracks <NUM>, <NUM> represent tracks of a second target detected by the first and second sensors <NUM>, <NUM>, respectively. Because of noise, measurement error, or other factors, the tracks <NUM> and <NUM> are slightly different even though they represent the same target. Similarly, the tracks <NUM> and <NUM> are slightly different from each other despite representing the same target. As such, the techniques described herein allow the surveillance service provider <NUM> to determine a single track for each of the two targets being tracked.

Referring back to <FIG>, the clustering module <NUM> performs density-based clustering of the data aggregated by the data aggregation module <NUM>. The clustering is performed for all positions of all tracks aggregated by the data aggregation module <NUM>. In one example, the clustering module <NUM> uses the known density-based spatial clustering of applications with noise (DBSCAN) data clustering algorithm. In other examples, the clustering module <NUM> may use other clustering algorithms. Specifically, the clustering module <NUM> analyzes the positions of the UAS <NUM> within a certain threshold duration aggregated by the data aggregation module <NUM> and groups them into a plurality of clusters based on their density using unsupervised learning techniques. Density and/or distance metrics for the clustering algorithm may be determined from sensor characteristics (e.g., error standard deviation) and/or user input.

<FIG> shows an example clustering performed by the clustering module <NUM> on the tracks <NUM>, <NUM>, <NUM>, and <NUM>. As can be seen in <FIG>, the clustering module <NUM> has found two clusters <NUM> and <NUM> based on the tracks <NUM>, <NUM>, <NUM>, <NUM>. For example, cluster <NUM> includes all the positions from tracks <NUM> and <NUM> as well as the two most recent positions from track <NUM> and the most recent position from track <NUM>. For example, cluster <NUM> includes the earliest three positions from track <NUM> and the earliest six positions from track <NUM>. Thus, the clusters <NUM> and <NUM> do not exactly match the positions of the first and second targets tracked by the first and second sensors <NUM>, <NUM>. The remaining techniques described below attempt to resolve this difference.

Referring back to <FIG>, the track-to-cluster association module <NUM> creates candidate track-to-cluster associations for each track received by the surveillance data reception module <NUM>. That is, the track-to-cluster association module <NUM> determines possible associations between the tracks received by the surveillance data reception module <NUM> and the clusters determined by the clustering module <NUM>. The track-to-cluster association module <NUM> then ranks the candidate track-to-cluster associations based on the clustering performed by the clustering module <NUM>. Specifically, the track-to-cluster association module <NUM> may rank a track-to-cluster association higher when there are more elements of the cluster in the track. For example, as shown in <FIG>, all five positions of track <NUM> and all seven positions of track <NUM> are included in cluster <NUM>, whereas only two of the five points of track <NUM> and one of seven points of track <NUM> are included in cluster <NUM>. Thus, the track-to-cluster association module <NUM> may rank an association between track <NUM> and cluster <NUM> higher than other associations for track <NUM>. Similarly, the track-to-cluster association module <NUM> may rank associations between track <NUM> and cluster <NUM>, track <NUM> and cluster <NUM>, and track <NUM> and cluster <NUM> higher than other associations for tracks <NUM>, <NUM>, and <NUM>.

Referring back to <FIG>, the track-to-target association module <NUM> associates certain tracks received from different sensors as corresponding to the same target based on the ranking of the candidate track-to-cluster associations determined by the track-to-cluster association module <NUM>. The track-to-target association module <NUM> may also associate tracks with targets based on other rules. For example, for cooperative sensors, each track may be associated with an ID of a particular target. Thus, the track-to-target association module <NUM> may exclude tracks received from cooperative sensors from targets that do not match that ID of the target received from the cooperative sensor.

In the example of <FIG>, the track-to-target association module <NUM> associates tracks <NUM> and <NUM> with a first target, as shown by grouping <NUM>. The track-to-target association module <NUM> further associates tracks <NUM> and <NUM> with a second target, as shown by grouping <NUM>. These associations may be made based on the associations made by the track-to-cluster association module <NUM> and the rankings thereof discussed above.

Referring back to <FIG>, the target state estimation module <NUM> estimates a target state (as a single track) for each target detected by the first and second sensors <NUM>, <NUM> by combining the tracks associated with the target by the track-to-target association module <NUM>. For example, in the example of <FIG>, the target state estimation module <NUM> may combine the tracks <NUM> and <NUM> of grouping <NUM> into a single track for a first target and may combine the tracks <NUM> and <NUM> of grouping <NUM> into a single track for a second target.

The target state estimation module <NUM> may use a variety of techniques to combine multiple tracks correlated together into a single track. In one example, the target state estimation module <NUM> may select the track from a preferred sensor as the state estimate for the target. In another example, the target state estimation module <NUM> may use a Kalman filter to determine a state estimate track for the target based on the tracks associated with the target. In another example, the target state estimation module <NUM> may use the known Covariance intersection algorithm to combine the tracks associated with a target into a single state estimate track. In another example, the target state estimation module <NUM> may use particle filtering to combine the tracks associated with a target into a single state estimate track. In other examples, the target state estimation module <NUM> may use other techniques to combine multiple tracks associated with a target into a single state estimate track for the target.

It should be understood that the components illustrated in <FIG> are merely illustrative and are not intended to limit the scope of this disclosure. More specifically, while the components in <FIG> are illustrated as residing within the surveillance service provider <NUM>, this is a nonlimiting example. In some embodiments, one or more of the components may reside external to the surveillance service provider <NUM>.

As mentioned above, the various components described with respect to <FIG> may be used to correlate airspace surveillance data from multiple sensors. An illustrative example of the various processes is described with respect to <FIG>. Although the steps associated with the blocks of <FIG> will be described as being separate tasks, in other embodiments, the blocks may be combined or omitted. Further, while the steps associated with the blocks of <FIG> will be described as being performed in a particular order, in other embodiments, the steps may be performed in a different order.

Referring now to <FIG>, a flow chart is shown for correlating airspace surveillance data from multiple sensors, according to one or more embodiments shown and described herein. At step <NUM>, the surveillance data reception module <NUM> receives surveillance data from a plurality of sensors. The received sensor data may comprise a track comprising a plurality of recently detected positions of a target. After receiving the airspace surveillance data, the surveillance data reception module <NUM> may store the data in the data storage component <NUM>.

At step <NUM>, the data aggregation module <NUM> aggregates the data received by the surveillance data reception module <NUM>. The data aggregation module <NUM> may aggregate received tracks comprising positions of targets within a threshold duration into the past.

At step <NUM>, the clustering module <NUM> performs density-based clustering of the data aggregated by the data aggregation module <NUM>. In some examples, the clustering module <NUM> may use the DBSCAN clustering algorithm. In other examples, the clustering module <NUM> may use other clustering algorithms to perform clustering.

At step <NUM>, the track-to-cluster association module <NUM> creates a plurality of candidate associations between tracks received by the surveillance data reception module <NUM> and clusters determined by the clustering module <NUM>. The track-to-cluster association module <NUM> further ranks the candidate associations based on the number elements from a track in a cluster.

At step <NUM>, the track-to-target association module <NUM> associates each track received by the surveillance data reception module <NUM> to a target based on the ranking of track to cluster associations performed by the track-to-cluster association module <NUM>. The track-to-target association module <NUM> may also determine the association between tracks and targets based on other rules such as ID-based exclusion for tracks received from cooperative sensors.

At step <NUM>, the target state estimation module <NUM> determines a state estimate comprising a single track for each target associated with one or more tracks by the track-to-target association module <NUM>. The target state estimation module <NUM> may use a variety of techniques to fuse multiple tracks associated with a target into a single track including selecting a track from a preferred sensor or using techniques such as Kalman filtering or covariance intersection. The method of <FIG> may be repeated either on an event basis (e.g., when there are diverging tracks for a single target) and/or periodically to account for new sensor data.

It should now be understood that the devices, systems, and methods described herein allow multi-sensor correlation of airspace surveillance data. A USS may receive airspace surveillance data from multiple sensors. The airspace surveillance data may comprise one or more tracks associated with one or more targets comprising recently detected positions of the one or more targets. The USS may aggregate the received data including all detected positions of targets within a threshold distance into the past.

The USS may then perform density-based clustering of the aggregated data. The USS may determine candidate track-to-cluster associations for the aggregated data and may rank each of the candidate track-to-cluster associations based on the number of elements from a track in a cluster. The greater the number of elements of a track that are in a cluster for a candidate association, the higher the rank of that association.

The USS may then perform track-to-target associations by associating each track with a target based on the ranking of the track-to-cluster associations. The USS may then determine a state estimate comprising a single estimated track for each target associated with one or more tracks.

Claim 1:
A computer-implemented method comprising:
receiving (<NUM>) airspace surveillance data from a plurality of sensors, the airspace surveillance data comprising tracks associated with one or more targets, wherein the tracks comprise recent positions of the one or more targets measured by the plurality of sensors;
aggregating (<NUM>) the airspace surveillance data received up to a certain predetermined threshold duration in the past to obtain aggregated data, wherein the aggregated data combines all positions of all tracks within the predetermined threshold duration;
performing (<NUM>) density-based clustering of the aggregated data to obtain a plurality of clusters;
determining (<NUM>) one or more candidate track-to-cluster associations between the tracks and the clusters, wherein the candidate track-to-cluster associations are ranked according to the number of target positions from a track that are included in a cluster;
associating (<NUM>) each of the tracks with one of the one or more targets based on the ranking of the candidate track-to-cluster associations; and
estimating (<NUM>) a state for each of the one or more targets by fusing the tracks associated with the same one or more targets into a single track.