Patent Description:
Applications in the millimeter-wave frequency regime have gained significant interest in the past few years due to the rapid advancement in low cost semiconductor technologies, such as silicon germanium (SiGe) and fine geometry complementary metal-oxide semiconductor (CMOS) processes. Availability of high-speed bipolar and metal-oxide semiconductor (MOS) transistors has led to a growing demand for integrated circuits for millimeter-wave applications at e.g., <NUM>, <NUM>, <NUM>, and <NUM> and also beyond <NUM>. Such applications include, for example, automotive radar systems and multi-gigabit communication systems. In some radar systems, the distance between the radar and a target is determined by transmitting a frequency modulated signal, receiving a reflection of the frequency modulated signal (also referred to as the echo), and determining a distance based on a time delay and/or frequency difference between the transmission and reception of the frequency modulated signal. Accordingly, some radar systems include a transmit antenna to transmit the radio-frequency (RF) signal, and a receive antenna to receive the reflected RF signal, as well as the associated RF circuits used to generate the transmitted signal and to receive the RF signal. In some cases, multiple antennas may be used to implement directional beams using phased array techniques. A multiple-input and multiple-output (MIMO) configuration with multiple chipsets can be used to perform coherent and non-coherent signal processing as well. <CIT> and <CIT> represent prior art from patent literature. <NPL> and <NPL>] represent further prior art articles.

A method as defined in claim <NUM> and a radar system as defined in claim <NUM> are provided. The dependent claims define further embodiments. The radar system may be configure to perform on of the methods of claims <NUM> to <NUM>.

The making and using of the embodiments disclosed are discussed in detail below. It should be appreciated, however, that the present invention provides many applicable inventive concepts that can be embodied in a wide variety of specific contexts. The specific embodiments discussed are merely illustrative of specific ways to make and use the invention, and do not limit the scope of the invention.

The description below illustrates the various specific details to provide an in-depth understanding of several example embodiments according to the description. The embodiments may be obtained without one or more of the specific details, or with other methods, components, materials and the like. In other cases, known structures, materials or operations are not shown or described in detail so as not to obscure the different aspects of the embodiments. References to "an embodiment" in this description indicate that a particular configuration, structure or feature described in relation to the embodiment is included in at least one embodiment. Consequently, phrases such as "in one embodiment" that may appear at different points of the present description do not necessarily refer exactly to the same embodiment. Furthermore, specific formations, structures or features may be combined in any appropriate manner in one or more embodiments.

Embodiments of the present invention will be described in a specific context, a millimeter-wave radar-based tracker for people sensing. Examples which are not claimed may be used for tracking other targets (e.g., animals, vehicles, robots, etc.) and/or may operate in regimes different than millimeter-wave.

In an embodiment of the present invention, a millimeter-wave radar is used to track human targets based on features extracted from the detected targets. In some embodiments, some or all of the extracted features used to associate a target to a track are not based on a motion model. Thus, some embodiments are advantageously capable of detecting human targets without knowledge (or with little knowledge) of the motion and/or localization (actual or predicted) of the human targets. Therefore, some embodiments are advantageously capable of tracking human targets using low frame-rates. In some embodiments, using low frame-rates advantageously allows for power savings, which may extend battery life in battery powered applications, and/or may advantageously allow for compliance with regulatory requirements (such as FCC requirements) associated with maximum duty cycle (maximum frame rate) for the radar operation, without sacrificing tracking performance.

In some embodiments, features, such as range, Doppler, and/or angle, are additionally tracked and are also used for associating detected targets to tracks based on a motion model, which may advantageously increase the tracking performance. For example, the range and Doppler velocity at a previous time step may be used to predict the location of the target at a future time step, and such information may be used to increase the confidence that a target assignment is correct (e.g., by using a gating region of expected locations for the target at the future time step).

A radar, such as a millimeter-wave radar, may be used to detect and track humans. For example, <FIG> shows a schematic diagram of millimeter-wave radar system <NUM>, according to an embodiment of the present invention. Millimeter-wave radar system <NUM> includes millimeter-wave radar sensor <NUM> and processing system <NUM>.

During normal operation, millimeter-wave radar sensor <NUM> operates as a frequency-modulated continuous-wave (FMCW) radar sensor and transmits a plurality of TX radar signals <NUM>, such as chirps, towards scene <NUM> using transmitter (TX) antenna <NUM>. The radar signals <NUM> are generated using RF and analog circuits <NUM>. The radar signals <NUM> may be in the <NUM> to <NUM> range. The objects in scene <NUM> include one or more humans, which may be moving or idle, for example. Other objects may also be present in scene <NUM>, other moving or static objects, such as furniture, machinery, mechanical structures, walls, etc..

The radar signals <NUM> are reflected by objects in scene <NUM>. The reflected radar signals <NUM>, which are also referred to as the echo signal, are received by receiver (RX) antennas 116a and 116b. RF and analog circuits <NUM> processes the received reflected radar signals <NUM> using, e.g., band-pass filters (BPFs), low-pass filters (LPFs), mixers, low-noise amplifier (LNA), and/or intermediate frequency (IF) amplifiers in ways known in the art to generate an analog signal xouta(t) and xoutb(t).

The analog signal xouta(t) and xoutb(t) are converted to raw digital data xout_dig(n) using ADC <NUM>.

The raw digital data xout_dig(n) is processed by processing system <NUM> to detect humans and their positions, and to track the detected humans.

Although <FIG> illustrates a radar system with a two receiver antennas <NUM>, it is understood that more than two receiver antennas <NUM>, such as three or more, may also be used.

Although <FIG> illustrates a radar system with a single transmitter antenna <NUM>, it is understood that more than one transmitter antenna <NUM>, such as two or more, may also be used.

Controller <NUM> controls one or more circuits of millimeter-wave radar sensor <NUM>, such as RF and analog circuit <NUM> and/or ADC <NUM>. Controller <NUM> may be implemented, e.g., as a custom digital or mixed signal circuit, for example. Controller <NUM> may also be implemented in other ways, such as using a general purpose processor or controller, for example. In some embodiments, processing system <NUM> implements a portion or all of controller <NUM>.

Processing system <NUM> may be implemented with a general purpose processor, controller or digital signal processor (DSP) that includes, for example, combinatorial circuits coupled to a memory. In some embodiments, processing system <NUM> may be implemented as an application specific integrated circuit (ASIC). In some embodiments, processing system <NUM> may be implemented with an ARM, RISC, or x86 architecture, for example. In some embodiments, processing system <NUM> may include an artificial intelligence (AI) accelerator. Some embodiments may use a combination of hardware accelerator and software running on a DSP or general purpose microcontroller. Other implementations are also possible.

In some embodiments, millimeter-wave radar sensor <NUM> and a portion or all of processing system <NUM> may be implemented inside the same integrated circuit (IC). For example, in some embodiments, millimeter-wave radar sensor <NUM> and a portion or all of processing system <NUM> may be implemented in respective semiconductor substrates that are integrated in the same package. In other embodiments, millimeter-wave radar sensor <NUM> and a portion or all of processing system <NUM> may be implemented in the same monolithic semiconductor substrate. Other implementations are also possible.

As a non-limiting example, RF and analog circuits <NUM> may be implemented, e.g., as shown in <FIG>. During normal operation, VCO <NUM> generates a radar signal, such as a linear frequency chirp (e.g., from <NUM> to <NUM>, or from <NUM> to <NUM>), which is transmitted by transmitting antenna <NUM>. The VCO <NUM> is controlled by PLL <NUM>, which receives a reference clock signal (e.g., <NUM>) from reference oscillator <NUM>. PLL <NUM> is controlled by a loop that includes frequency divider <NUM> and amplifier <NUM>.

The TX radar signal <NUM> transmitted by transmitting antenna <NUM> is reflected by objects in scene <NUM> and received by receiving antennas 116a and 116b. The echo received by receiving antennas 116a and 116b are mixed with a replica of the signal transmitted by transmitting antenna <NUM> using mixer 146a and 146b, respectively, to produce respective intermediate frequency (IF) signals xIFa(t) xIFb(t) (also known as beat signals). In some embodiments, the beat signals xIFa(t) xIFb(t) have a bandwidth between <NUM> and <NUM>. Beat signals with a bandwidth lower than <NUM> or higher than <NUM> is also possible.

Beat signals xIFa(t) xIFb(t) are filtered with respective low-pass filters (LPFs) 148a and 148b and then sampled by ADC <NUM>. ADC <NUM> is advantageously capable of sampling the filtered beat signals xouta(t) xoutb(f) with a sampling frequency that is much smaller than the frequency of the signal received by receiving antennas 116a and 116b. Using FMCW radars, therefore, advantageously allows for a compact and low cost implementation of ADC <NUM>, in some embodiments.

The raw digital data xout_dig(n), which in some embodiments include the digitized version of the filtered beat signals xouta(t) and xoutb(t), is (e.g., temporarily) stored, e.g., in matrices of Nc x Ns per receiver antenna <NUM>, where Nc is the number of chirps considered in a frame and Ns is the number of transmit samples per chirp, for further processing by processing system <NUM>.

In some embodiments, ADC <NUM> is a <NUM>-bit ADC with multiple inputs. ADCs with higher resolution, such as <NUM>-bits or higher, or with lower resolution, such as <NUM>-bits, or lower, may also be used. In some embodiments, an ADC per receiver antenna may be used. Other implementations are also possible.

<FIG> shows a sequence of chirps <NUM> transmitted by TX antenna <NUM>, according to an embodiment of the present invention. As shown by <FIG>, chirps <NUM> are organized in a plurality of frames and may be implemented as up-chirps. Some embodiments may use down-chirps or a combination of up-chirps and down-chirps, such as up-down chirps and down-up chirps. Other waveform shapes may also be used.

As shown in <FIG>, each frame may include a plurality of chirps <NUM> (also referred to, generally, as pulses). For example, in some embodiments, the number of pulses in a frame is <NUM>. Some embodiments may include more than <NUM> pulses per frame, such as <NUM> pulses, <NUM> pulses, or more, or less than <NUM> pulses per frame, such as <NUM> pulses, <NUM> pulses, <NUM> or less. In some embodiments, each frame includes only a single pulse.

Frames are repeated every FT time. In some embodiments, FT time is <NUM>. A different FT time may also be used, such as more than <NUM>, such as <NUM>, <NUM>, <NUM>, or more, or less than <NUM>, such as <NUM>, <NUM>, or less.

In some embodiments, the FT time is selected such that the time between the beginning of the last chirp of frame n and the beginning of the first chirp of frame n+<NUM> is equal to PRT. Other embodiments may use or result in a different timing.

The time between chirps of a frame is generally referred to as pulse repetition time (PRT). In some embodiments, the PRT is <NUM>. A different PRT may also be used, such as less than <NUM>, such as <NUM>, <NUM>, or less, or more than <NUM>, such as <NUM>, or more.

The duration of the chirp (from start to finish) is generally referred to as chirp time (CT). In some embodiments, the chirp time may be, e.g., <NUM>. Higher chirp times, such as <NUM>, or higher, may also be used. Lower chirp times, may also be used.

In some embodiments, the chirp bandwidth may be, e.g., <NUM>. Higher bandwidth, such as <NUM> or higher, or lower bandwidth, such as <NUM>, <NUM>, or lower, may also be possible.

In some embodiments, the sampling frequency of millimeter-wave radar sensor <NUM> may be, e.g., <NUM>. Higher sampling frequencies, such as <NUM> or higher, or lower sampling frequencies, such as <NUM> or lower, may also be possible.

In some embodiments, the number of samples used to generate a chirp may be, e.g., <NUM> samples. A higher number of samples, such as <NUM> samples, or higher, or a lower number of samples, such as <NUM> samples or lower, may also be used.

<FIG> shows a flow chart of embodiment method <NUM> for people tracking, according to an embodiment of the present invention. Method <NUM> may be implemented by processing system <NUM>.

During steps 302a and 302b, raw ADC data xout_dig(n) is received, e.g., from millimeter-wave radar sensor <NUM>. As shown, the raw ADC data xout_dig(n) includes separate baseband radar data from multiple antennas (e.g., <NUM> in the example shown in <FIG>).

During steps 304a and 304b, signal conditioning, low pass filtering and background removal are performed on the raw ADC data of the respective antenna <NUM>. The raw ADC data xout_dig(n) radar data are filtered, DC components are removed to, e.g., remove the Tx-Rx self-interference and optionally pre-filtering the interference colored noise. Filtering may include removing data outliers that have significantly different values from other neighboring range-gate measurements. Thus, this filtering also serves to remove background noise from the radar data.

During steps 306a and 306b, 2D moving target indication (MTI) filters are respectively applied to data produced during steps 304a and 304b to remove the response from static targets. The MTI filter may be performed by subtracting the mean along the fast-time (intra-chirp time) to remove the transmitter-receiver leakage that perturbs the first few range bins, followed by subtracting the mean along the slow-time (inter-chirp time) to remove the reflections from static objects (or zero-Doppler targets).

During steps 308a and 308b, a series of FFTs are performed on the filtered radar data produced during steps 306a and 306b, respectively. A first windowed FFT having a length of the chirp is calculated along each waveform for each of a predetermined number of chirps in a frame of data. The FFTs of each waveform of chirps may be referred to as a "range FFT. " A second FFT is calculated across each range bin over a number of consecutive periods to extract Doppler information. After performing each 2D FFT during steps 308a and 308b, range-Doppler images are produced, respectively.

During step <NUM>, a minimum variance distortionless response (MVDR) technique, also known as Capon, is used to determine angle of arrival based on the range and Doppler data from the different antennas. A range-angle image (RAI) is generated during step <NUM>. In some embodiments, a range-Doppler-angle data cube is generated during step <NUM>.

During step <NUM>, an ordered statistics (OS) Constant False Alarm Rate (OS-CFAR) detector is used to detect targets. The CFAR detector generates a detection image in which, e.g., "ones" represent targets and "zeros" represent non-targets based, e.g., on the power levels of the RAI, by comparing the power levels of the RAI with a threshold, points above the threshold being labeled as targets ("ones") while points below the threshold are labeled as non-targets ("zeros).

In some embodiments, targets present in the detection image generated during step <NUM> are clustered during step <NUM>, e.g., based on similar feature characteristics, such as empirical mode decomposition (EMD), and/or scale invariant feature transform (SIFT), associated with the detected targets. In some embodiments, other types of features of the detected targets, such as motion model-based features based on, e.g., range, Doppler, and/or angle, may also be used to cluster cells together. In some embodiments, metrics such as correlation and/or Wasserstein distance may be used to determine the similarities between clusters. In some embodiments, the feature-based clustering is performed by using k-means clustering, in which targets are grouped (clustered) based on having similar features to the one of k clusters having the nearest mean of such (e.g., combined) features.

For example, in some embodiments, a vector of features includes a plurality of features (e.g., intrinsic mode functions (IMFs) and/or number of IMFs, which are associated with EMD, and/or magnitude M(m,n) and/or phase φ(m,n), which are associated with SIFT), where each channel describes a type of feature (e.g., IMFs, number of IMFs, magnitude M(m,n) and/or phase φ(m,n)). Each channel may be described as a Gaussian distribution (taking mean and variance over available vectors of the same feature). A weighted sum over all the different Gaussian distributions over the channels is obtained to provide a descriptor for each cell, where the descriptor is associated with all the feature types and which may be a value or vector that is indicative of the characteristics (features) of the associated cluster and which may be used to determine how similar are clusters. Such descriptor is used for clustering, e.g., using the k-means clustering algorithm.

In some embodiment, a density-based spatial clustering of applications with noise (DBSCAN) algorithm may also be used to associate targets to clusters during step <NUM>. The output of DBSCAN is a grouping of the detected points into particular targets. DBSCAN is a popular unsupervised algorithm, which uses minimum points and minimum distance criteria to cluster targets, and may be implemented in any way known in the art. Other clustering algorithms may also be used.

In some embodiments, thus, clustering results in the radar image (e.g., RAI or RDI) or data cube being divided into groups of cells with similar descriptors. In some embodiments, each cluster corresponds to a (e.g., potential) detected target. Since the spread of features is not necessarily uniform, in some embodiments, each cluster is not necessarily equal. Thus, in some embodiments, the radar image or data cube is divided into clusters of cells, but each cluster of cells is not necessarily of the same size (e.g., does not have the same number of cells/sub-cells). During step <NUM>, detected (clustered) targets are associated with respective tracks. As will be described in more detail later, in some embodiments, detected targets are associated to respective tracks using feature-based template matching (during step <NUM>). For example, in some embodiments, geometric features are used during step <NUM> for template matching. A geometric feature may be understood as a feature that is recognizable despite changes in rotation of the target, as well as changes in the range, Doppler velocity, and angle of the centroid of the target. In some embodiments a geometric feature may include a physical geometric feature, such as physical edges of the target (e.g., from the radar image). In some embodiments, additionally or alternatively, a geometric feature may include a metric (e.g., a vector, function, or group of functions) based on the relationship between cells of the raw data (e.g., of the data cube), such as the relationship between range cells, Doppler velocity cells, and/or angle cells. Examples of such metric include functions extracted using functional decomposition of the data cube, gradients of the data cube, and/or statistical properties of the data cube (such as histograms/PDF of the data cube). Examples of geometric features include according to the claims EMD features and not claimed examples include SIFT features.

In some embodiments, geometric features allow for identification of a target without relying on a motion model. In some embodiments, geometric features allow for distinguishing between tracked targets.

In some embodiments, geometric features such as EMD, and/or SIFT (not claimed), are tracked for each target. For each clustered cell (for each detected target) a feature vector is generated for each time step i with values of each feature associated with the clustered cell. Detected targets at time step i + <NUM> are assigned to respective tracks based on the similarities between feature vectors (e.g., based on the error between the feature vectors), e.g., using Hungarian assignment. For example, in some embodiments, a similarity measure is identified between feature clusters at consecutive time steps (e.g., i, and i + <NUM>), and the assignments that minimize the error (e.g., increase correlation) between feature clusters is selected for track assignment.

In some embodiments, the data association step (<NUM>) may include, additionally, data association methods that do not rely on featured-based template matching.

In some embodiments, the data assignment of detected targets (clusters) to tracks relies on the geometric features of the cluster and does not rely (or does not rely solely) on the actual physical locations and/or velocities of the detected targets.

During step <NUM>, track filtering is performed, e.g., for tracking a target over time. For example, in some embodiments, the unscented Kalman filter is used to perform track filtering during step <NUM>. For example, in some embodiments, the features (e.g., SIFT, EMD, range, Doppler, angle, deep learning-based parameters, and/or other parameters associated with the track) are, e.g., additional features used to perform data association (which may also be tracked by the Kalman filter). The unscented Kalman filter may also track localization of each track and may rely on the track history of such localization to enhance data association. The Kalman filter may be implemented in any way known in the art.

It is understood that although targets may be identified using template matching (during step <NUM>) that may not include spatial and/or movement information (e.g., range, Doppler, angle), such localization information may still be tracked during step <NUM>. Thus, in some embodiments, featured-based template matching (step <NUM>) is an enabler for data association in environments, such as low frame rate, and/or multi-target scenarios, and/or distributed radar implementations in which relying in localization information alone may be difficult.

During step <NUM>, track management tasks, such as generating tracks and killing tracks are performed. For example, during step <NUM>, track initializations, re-initialization, and/or tracks killing may be performed, e.g., based on whether detected targets are no longer in the field-of-view (in scene <NUM>), or re-entered the field of view, for example.

In some embodiments, steps <NUM>, <NUM>, and <NUM> may be implemented in different order. For example, in some embodiments, track initialization (during step <NUM>) may be performed before performing step <NUM>.

<FIG> illustrates target association using template matching, according to an embodiment of the present invention. Template matching, e.g., as illustrated in <FIG>, may be performed, e.g., during step <NUM>.

As shown in <FIG>, humans A and B move within field-of-view <NUM> as time progresses (from time step i to time step i + <NUM>). In some embodiments the time between time steps i and i + <NUM> may be, e.g., <NUM> (for a <NUM> frame per second radar system). In some embodiments, faster frame rates may be used. As will be described in more details later, in some embodiments, slower frame rates, such as <NUM> frames per second radar systems (where the time between time steps i and i + <NUM> is, e.g., <NUM>) or slower, may advantageously be used while preserving the ability to effectively track targets.

As shown in <FIG>, humans A and B are located at time step i in locations <NUM> and <NUM>, respectively. At time step i + <NUM>, humans A and B are located in locations <NUM> and <NUM>, respectively. Since at time step i + <NUM> the detected target at location <NUM> is closer to location <NUM>, and the detected target at location <NUM> is closer to location <NUM>, a conventional association method using probabilistic data association filter (PDAF) would likely associate detected human B (at location <NUM>) to the track of human A and detected human A (at location <NUM>) to the track of human B. As shown in <FIG>, using featured-based assignment advantageously allows for correct track assignment without increasing the duty cycle (without reducing the size of time step i so that targets are detected closer in time) by relying on features that are not based (or not based solely) on a motion model. Instead, in some embodiments, detected targets are assigned to track based on the level of correlation between features at, e.g., consecutive time steps. Thus, some embodiments advantageously allow for tracking humans at low duty cycles, such as <NUM> frames per second or slower.

Since some embodiments rely on geometric features that are not based on a motion model (such as such as EMD, and not claimed SIFT) to track targets, some embodiments are advantageously suitable for tracking targets using distributed radar, where a human may move from fields-of-view of different radars and where the radars may lack information about movement of a target outside their own filed-of-view. For example, <FIG> illustrates target association using template matching using distributed radar system <NUM> using <NUM> radars systems, according to an embodiment of the present invention. In some embodiments, each radar system used in distributed radar system <NUM> is implemented as millimeter-wave radar system <NUM>. Template matching, e.g., as illustrated in <FIG> (and as explained in more detailed later), may be performed, e.g., during step <NUM> of each radar of distributed radar system <NUM>. In some embodiments, distributed radar system <NUM> may include more than <NUM> radars, such as <NUM>, <NUM>, <NUM> or more.

As shown in <FIG>, humans A and B move between fields-of-view <NUM> and <NUM> as time progresses (from time step i to time step i + <NUM>). For example, humans A and B are located at time step i in locations <NUM> and <NUM>, respectively. At time step i + <NUM>, humans A and B are located in locations <NUM> and <NUM>, respectively. Since the radar with field-of-view <NUM> detects a single target (Human A at location <NUM>) at time step i and a single target (Human B at location <NUM>) at time i + <NUM>, and since the radar with field-of-view <NUM> detects a single target (Human B at location <NUM>) at time step i and a single target (Human A at location <NUM>) at time i + <NUM>, a conventional association method using PDAF would likely associate detected human B (at location <NUM>) to the track of human A and detected human A (at location <NUM>) to the track of human B. As shown in <FIG>, using featured-based assignment advantageously allows for correct track assignment when a target moves between fields-of view of different radars in a distributed radar system. In some embodiments, a controller <NUM> shared between radars of distributed radar system <NUM> is used to identify features of tracked targets, and the common controller <NUM> may be used to perform the tracking functions. For example, in some embodiments, common controller <NUM> may perform steps <NUM>, <NUM> and <NUM>, while processing system <NUM> of each radar <NUM> of the distributed radar system <NUM> performs steps <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>.

In some embodiments, controller <NUM> may be implemented as part of processing system <NUM> of one of the radars <NUM> of distributed radar system <NUM>. In some embodiments, controller <NUM> may be implemented externally to processing systems <NUM> of radar systems <NUM> of distributed radar system <NUM>, e.g., with a general purpose processor, controller or digital signal processor (DSP) that includes, for example, combinatorial circuits coupled to a memory. In some embodiments, processing system <NUM> may be implemented as an application specific integrated circuit (ASIC). In some embodiments, processing system <NUM> may be implemented with an ARM, RISC, or x86 architecture, for example. In some embodiments, processing system <NUM> may include an artificial intelligence (AI) accelerator. Some embodiments may use a combination of hardware accelerator and software running on a DSP or general purpose microcontroller. Other implementations are also possible.

For example, in some embodiments, the features of the detected target along with spatial and movement parameters are passed to a central processor <NUM> (external to each radar of distributed radar system <NUM>) for data association and tracking. In some embodiments, one of the processing systems <NUM> of the distributed radar system <NUM> may operate as the central processor <NUM>, e.g., for data association and tracking.

<FIG> shows a flow chart of embodiment method <NUM> for performing data association using template matching, according to an embodiment of the present invention. Step <NUM> may be performed as method <NUM>.

As shown in <FIG>, for each of the L clusters (e.g., identified during step <NUM>), features associated with time step i are extracted during steps <NUM> and <NUM>, and are compared with features of the, e.g., L clusters at time step i + <NUM> during steps <NUM> and <NUM> to generate L error vectors. In some embodiments, step <NUM> may be considered part of step <NUM> (e.g., such as a way to implement step <NUM>).

During step <NUM>, assignments between L clusters at time step i and L clusters at time step i + <NUM> are made, e.g., to minimize the error between error vectors (e.g., so that the summation of the errors of the error vectors between assigned clusters is minimized). In some embodiments, Hungarian assignment is used during step <NUM> to associate clusters at time step i to clusters at time step i + <NUM>. In some embodiments, each cluster at time step i + <NUM> is associated to the tracks that corresponds to the associated cluster at time i.

In some embodiments, applying Hungarian assignment comprises:.

In some embodiments, the number of clusters at time steps i and i + <NUM> are different (e.g., since a previously detected target disappears, or a new target arrives to the field-of view). In some such embodiments, assignment is made to minimize the error between vectors at each time step, and the error of the additional vectors that are not associated with a corresponding vector at the other time step are assigned a default error. In some embodiments, for each unassigned cluster or target, and error counter is used to count how many times there are unsigned targets, and corresponding tracks are killed when the counter reaches a predetermined threshold (e.g., <NUM>, <NUM>, etc.).

In some embodiments, a motion model that relies on features such as range, Doppler, and/or angle, is also used for associating detected targets to tracks, which may advantageously increase the tracking performance. For example, the range and Doppler velocity at a previous time step may be used to predict the location of the target at a future time step, and such information may be used to increase the confidence that a target assignment is correct (e.g., by using a gating region of expected locations for the target at the future time step), and where the confidence level is used to associate the target to a track. Thus, in some embodiments, associating a target to a track is further based on a motion-based model, e.g., based on range, Doppler, and/or angle.

<FIG> shows a flow chart of embodiment method <NUM> for extracting EMD features (step <NUM>) and computing the error associated with the extracted EMD features (step <NUM>), according to an embodiment of the present invention. In some embodiments, step <NUM> may be performed as step <NUM>, and the portion of steps <NUM> associated with comparing EMD features may be performed as step <NUM>.

During step <NUM>, EMD feature extraction is performed. EMD may be understood as a way to decompose a signal data into intrinsic mode functions (IMF) of instantaneous frequencies contained in the original signal data. The disintegrated signal components form a complete or nearly orthogonal basis of the original data signal. In some embodiments, EMD is performed on raw data (e.g., the Doppler signal) associated with the cluster cell. For example, in some embodiments, EMD is performed on raw data (e.g., from the data cube) associated with a particular cluster cell at time step i.

During step <NUM>, the IMFs having an energy higher than a predetermined value are identified. During step <NUM>, the identified IMFs are sorted (e.g., in ascending order). During step <NUM>, the error between the sorted identified IMFs at time i (for each cluster cell) and the sorted IMFs of each cluster at time i + <NUM> is computed to generate, e.g., L, error values for each cluster cell at time i (which may be used as part of the error vectors in step <NUM>).

In some embodiments, the error between IMFs at time steps i and i + <NUM> is determined by using, e.g., means square error. In some embodiments, the number of IMF above the threshold may also be used to determine the error between clusters at time steps i and i + <NUM>.

In some embodiments, the EMD features (e.g., the sorted IMFs) between all clusters at time step i and all clusters at time step i + <NUM>, are compared, and the cluster pairs at time steps i and i + <NUM> resulting in the lowest means square error are associated.

<FIG> shows waveforms of a disintegration of a data signal into multiple intrinsic frequency components through EMD, according to an embodiment of the present invention.

In the embodiment of <FIG>, three dominant modes (IMF1, IMF2, and IMF3) out of ten modes have an energy higher than the predetermined threshold.

<FIG> shows a flow chart of a not claimed method <NUM> for extracting SIFT features (step <NUM>) and computing the error associated with the extracted SIFT features (step <NUM>). In some embodiments, step <NUM> may be performed as step <NUM>, and the portion of step <NUM> associated with comparing SIFT features may be performed as step <NUM>.

During step <NUM>, SIFT feature extraction is performed. SIFT may be understood as a pattern recognition method for detecting features of a radar image (e.g., RDI, RAI) that are, e.g., invariant to scaling, rotation, translation, and geometrical distortion, or any affine distortion of the image. For example, in some embodiments, SIFT feature extraction is obtained by computing gradients between the different cells of an image. For example, for each clustered cell, magnitude M(m,n) and phase φ(m,n) may be determined by applying Equations <NUM> and <NUM>: <MAT> <MAT> where X(m,n) are the sub-cells of the clustered cell of the radar image, m,n being the location of the sub-cell.

In some embodiments, SIFT feature extraction is performed on, e.g., RDI, RAI, or data cube (e.g., from steps 308a, 308b, and/or <NUM>) on a region associated with a particular cluster cell at time step i, by, e.g., using Equations <NUM> and <NUM>.

During step <NUM>, the error between the magnitude and/or phase of the cluster cell at time i and of each of the other clusters at time i + <NUM> is computed to generate L error values (which may be used as part of the error vectors in step <NUM>). For example, in some embodiments, a correlation value r may be used, which may be given by <MAT> where x is the magnitude M or phase φ vector at time i, x is the mean of the magnitude M or phase φ vectors at time i, y is the magnitude M or phase φ at time i + <NUM>, and y is the mean of the magnitude M or phase φ vectors at time i + <NUM>. In embodiments in which both magnitude M or phase φ are used, Equation <NUM> is applied respectively to each metric (magnitude M and phase φ) and an (e.g., weighted) average between the two correlations r obtained with Equation <NUM> is used as a single metric associated with the SIFT feature of the clustered cell.

In some embodiments, the correlation r between SIFT features is computed between all clusters at time step i and all clusters at time step i + <NUM>, e.g., by using Equation <NUM>. In some embodiments, the clusters at time steps i and i + <NUM> having the highest correlation and greater than a predetermined correlation threshold are associated. In some embodiments, the predetermined correlation threshold is between <NUM> and <NUM>.

<FIG> show an input radar image and corresponding histogram of gradients with SIFT features, respectively.

<FIG> shows a flow chart of not claimed method <NUM> for computing the Wasserstein distance between clusters at time steps i and i + <NUM>. In some embodiments, step <NUM> may be performed as step <NUM>.

During step <NUM>, the Wasserstein distance is computed between clusters at time steps i and i + <NUM>. The Wasserstein distance (also referred to as earth mover's distance, Wasserstein metric or Kantorovich-Rubinstein metric) is a mathematical function that computes, in addition to the similarities between two probability distributions, the distance between the two probability distributions. It is understood that the term "distance," as used with respect to the Wasserstein metric, is the distance between distributions and not necessarily a physical distance. For example, the Wasserstein metric may be used to determine the distance between SIFT features of two clusters, and/or between EMD features of two clusters. The Wasserstein metric may be a physical distance in other scenarios, e.g., when used with respect to RAI or RDI data.

In some embodiments, the raw data (e.g., a vector of different feature values, such as SIFT and/or EMD, as well as, e.g., range, Doppler and/or angle) associated with each cluster is modeled (approximated) as a Gaussian distribution having a respective mean µ and standard deviation σ. For example, in an embodiment having a cluster (P1) at time step i having mean µ<NUM> and standard deviation σ<NUM>, and a cluster (P<NUM>) at time step i + <NUM> having mean µ<NUM> and standard deviation σ<NUM>, the Wasserstein metric may be computed as <MAT> where <MAT> is the L2 norm.

In some embodiments, the Wasserstein metric is computed between all clusters at time step i and all clusters at time step i + <NUM>, e.g., by using Equation <NUM>. In some embodiments, the clusters at time steps i and i + <NUM> having the lowest Wasserstein distance are associated.

<FIG> shows image <NUM> having two clusters at time i and i + <NUM>.

Each cluster cell <NUM>, <NUM>, <NUM>, and <NUM> is approximated as a Gaussian distribution. In some embodiments, all possible Wasserstein distances between time steps i and i + <NUM> are determined, which in this embodiment is four Wasserstein distances, namely, W<NUM>,<NUM> (between clusters <NUM> and <NUM>), W<NUM>,<NUM> (between clusters <NUM> and <NUM>), W<NUM>,<NUM> (between clusters <NUM> and <NUM>), and W<NUM>,<NUM> (between clusters <NUM> and <NUM>). Each Wasserstein distance (W<NUM>,<NUM>, W<NUM>,<NUM>, W<NUM>,<NUM>, W<NUM>,<NUM>) may be computed using Equation <NUM>. In some embodiments, clusters having the lowest Wasserstein distance are associated.

In some embodiments, image <NUM> is a representation of features (e.g., SIFT, EMD) and the Wasserstein metric is used to determine the similarities between features. In some embodiments, image <NUM> is a radar image (e.g., RAI, RDI), and the Wasserstein metric is used for motion-based tracking (e.g., using Euclidean distance).

In some embodiments, a single type of feature is used during template matching. For example, in some embodiments, step <NUM> is performed by performing steps <NUM> and <NUM>, and assigning clusters at time steps i and i + <NUM>, e.g., by minimizing the total means square error. In some embodiments, step <NUM> is performed by performing steps <NUM> and <NUM>, and assigning clusters at time steps i and i + <NUM>, e.g., by maximizing the correlation between clusters. In some embodiments, step <NUM> is performed by performing step <NUM> based solely on EMD features or based solely on SIFT features (not claimed), and assigning clusters at time steps i and i + <NUM>, e.g., by minimizing the Wasserstein distance (e.g., minimizing the summation of the Wasserstein distances between matched clusters).

<FIG> illustrates a method for performing data association using template matching, as applied to the example of <FIG>.

As shown, <FIG> illustrates a non-limiting example in which, EMD feature extraction (step <NUM>) and SIFT feature extraction (step <NUM>) are used to extract geometric features for clusters <NUM> and <NUM> (at time step i) and clusters <NUM> and <NUM> (at time step i + <NUM>), and the Wasserstein metric (step <NUM>) is computed between each of the clusters detected at time step i (<NUM> and <NUM>) and each of the clusters detected at time step i + <NUM> (<NUM> and <NUM>) for each of the geometric features extracted.

As shown in <FIG>, IMFs higher than a predetermined threshold are identified for each cluster identified at time step i (<NUM> and <NUM>) and at time step i + <NUM> (<NUM> and <NUM>). Magnitude M and phase φ associated with SIFT features are also extracted for each cluster identified at time step i (<NUM> and <NUM>) and at time step i + <NUM> (<NUM> and <NUM>). Resulting vectors of features V<NUM>, V<NUM>, V<NUM>, and V<NUM> associated with clusters <NUM>, <NUM>, <NUM> and <NUM>, respectively, are generated. The vectors of features V<NUM>, V<NUM>, V<NUM>, and V<NUM> are modeled as Gaussian distributions, and the Wasserstein distance is computed between each cluster identified at time step i (<NUM> and <NUM>) and at time step i + <NUM> (<NUM> and <NUM>) for each feature (in this example, IMF, M, and φ). Thus, a resulting error vector of error metrics (in this example Wasserstein distances, although other metrics, such as correlation, can also be used) is generated for each possible assignment between clusters identified at time step i and i + <NUM> (in this example, <NUM>/<NUM>, <NUM>/<NUM>, <NUM>/<NUM>, and <NUM>/<NUM>).

A similarity metric is then generated, e.g., by using a (e.g., weighted) average of each of the errors (in this example Wasserstein distances) inside each of the error vectors. Thus, the similarity metric may be used as a metric indicating how similar two clusters are. Assignment is then made (e.g., using Hungarian assignment) to minimize the total error. For example, in this example, the sum of errors D402_406 and D404_408 is lower than the sum of the sum of errors D404_406 and D402_408 and thus clusters <NUM> and <NUM> are matched and clusters <NUM> and <NUM> are matched (as also shown in <FIG>).

In some embodiments, such as shown in <FIG>, more than one type of feature may be used to perform template matching. In such embodiments, it is possible that a first type of feature (e.g., EMD) is indicative of a first matching (e.g., matching cluster A at time step i with cluster A' at time step i + <NUM>, and matching cluster B at time step i with cluster B' at time step i + <NUM>), and a second type of feature (e.g., SIFT) is indicative of a second different matching (e.g., matching cluster A at time step i with cluster B' at time step i + <NUM>, and matching cluster B at time step i with cluster A' at time step i + <NUM>). In some such embodiments, the average error may be used to determine a final cluster assignment to a particular track. In some embodiments, a weighted average (with, e.g., predetermined coefficients) may be used instead of averaging. In some embodiments, the function F in Equation <NUM> may be modified to account for, e.g., averaging or applying weighted coefficients to respective error metrics associated with respective features.

As shown in <FIG>, template matching may be performed with one or more of EMD and SIFT features. Some embodiments may include different features for template matching. For example, some embodiments may rely on deep-learning based feature extraction and association through correlation by using deep convolutional neural networks (DCNN) for processing the data cube (range-Doppler-angle data cube, e.g., from the output of step <NUM>) for extracting geometric features to be used during the template matching step to determine similarities between clusters (e.g., during steps <NUM> and/or <NUM>) by determining the correlation and/or Wasserstein distance between such deep-learning based geometric features.

For example, in some embodiments, the output of the Lth layer of a DCNN may be given by (Wl,Hl,Dl), where Wl,Hl are the width and height of each feature map and Dl is the dimension/number of feature maps at Lth layer. In some embodiments, instead of one layer, multiple layer outputs can be treated as extracted features, for e.g., layer L and layer L+<NUM>. In some embodiments, the DCNN is trained to learn distinct geometric features by using supervised learning with a data set including multiple (e.g., human) targets performing a plurality of activities (e.g., walking, running, standing idle, etc.).

In some embodiments, template matching is performed, in addition to one or more of EMD, SIFT and/or deep learning-based geometric features, on a motion model relying on one or more of range, Doppler, and angle.

Claim 1:
A method for tracking human targets, the method comprising:
receiving data from a radar sensor (<NUM>) of a radar (<NUM>);
processing the received data to detect human targets;
identifying a first geometric feature of a first detected human target at a first time step, the first detected human target being associated to a first track, wherein identifying the first geometric feature comprises:
performing a first empirical mode decomposition, EMD, on received data of the first time step associated with the first detected human target,
identifying first intrinsic mode functions, IMFs, from the first EMD that are higher than a predetermined threshold, and
sorting the first IMFs;
identifying a second geometric feature of a second detected human target at a second time step, wherein identifying the second geometric feature comprises:
performing second EMD on received data of the second time step associated with the second detected human target,
identifying second IMFs from the second EMD that are higher than the predetermined threshold, and
sorting the second IMFs;
determining an error value based on the first and second geometric features, wherein determining the error value comprises determining a mean square error based on the sorted first and second IMFs; and
associating the second detected human target to the first track based on the error value.