Patent Description:
Applications in the millimeter-wave frequency regime have gained significant interest in the past few years due to the rapid advancement in low cost semiconductor technologies, such as silicon germanium (SiGe) and fine geometry complementary metal-oxide semiconductor (CMOS) processes. Availability of high-speed bipolar and metal-oxide semiconductor (MOS) transistors has led to a growing demand for integrated circuits for millimeter-wave applications at <NUM>, <NUM>, <NUM>, and <NUM> and also beyond <NUM>. Such applications include, for example, static and moving object detection and tracking.

In some radar systems, the distance between the radar and a target is determined by transmitting a frequency modulated signal, receiving a reflection of the frequency modulated signal (also referred to as the echo), and determining a distance based on a time delay and/or frequency difference between the transmission and reception of the frequency modulated signal. Accordingly, some radar systems include a transmit antenna to transmit the radio-frequency (RF) signal, a receive antenna to receive the RF, as well as the associated RF circuitry used to generate the transmitted signal and to receive the RF signal. In some cases, multiple antennas may be used to implement directional beams using phased array techniques. A multiple-input and multiple-output (MIMO) configuration with multiple chipsets can be used to perform coherent and non-coherent signal processing as well.

In some settings, static objects coexist with moving objects. For example, in an indoor setting, static objects, such as furniture and walls, coexist with moving objects such as humans. An indoor setting may also include object exhibiting periodic movements, such as fans. Doppler analysis has been used to distinguish between moving and static objects.

Publication <NPL> describes a real-time behavior detection system using millimeter wave radar. Radar is used to sense the micro-Doppler information of targets. A convolution neural network (CNN) is further implemented in the detection and classification of the human motion behaviors using this information.

Publication <CIT> describes a system for characterization of the motion of a human based on a radar that uses Doppler spectrograms and a particle filter.

Publication <CIT> describes method for target classification that also classifies an activity type and level based on either radar or sonar data.

Publication <NPL> decribes determination of human activities (such as walking, falling or creeping) based on radar Doppler data using conventional, data driven and deep learning classification approaches.

There may be a demand for providing an improved concept for a method for tracking a target using a millimeter-wave radar and a millimeter-wave radar system.

Such a demand may be satisfied by the subject matter of any of the claims.

In accordance with an embodiment, a method for tracking a target using a millimeter-wave radar includes: receiving radar signals using the millimeter-wave radar; generating a range-Doppler map based on the received radar signals; detecting a target based on the range-Doppler map; tracking the target using a track; generating a predicted activity label based on the track, where the predicted activity label is indicative of an actual activity of the target; generating a Doppler spectrogram based on the track; generating a temporary activity label based on the Doppler spectrogram; assigning an uncertainty value to the temporary activity label, where the uncertainty value is indicative of a confidence level that the temporary activity label is an actual activity of the target; and generating a final activity label based on the uncertainty value; wherein generating the final activity label comprises, when the uncertainty value is above a predetermined threshold, using the predicted activity label as the final activity label, and when the uncertainty value is below the predetermined threshold, using the temporary activity label as the final activity label.

In accordance with an embodiment, a millimeter-wave radar system includes: a millimeter-wave radar configured to transmit and receive radar signals; and a processor including: a radar processing block configured to generate a range-Doppler map based on the radar signals received by the millimeter-wave radar; a target detector block configured to detect a target based on the range-Doppler map; a tracker configured to: track the target using a track, and generate a predicted activity label based on the track, where the predicted activity label is indicative of an actual activity of the target; a feature extraction block configured to generate a Doppler spectrogram based on the track; a classifier configured to generate a temporary activity label based on the Doppler spectrogram; and a classification gating block configured to receive an uncertainty value associated to the temporary activity label and produce gating data based on the uncertainty value, the predicted activity label, and the temporary activity classification, where the uncertainty value is indicative of a confidence level that the temporary activity label is an actual activity of the target, and where the tracker is configured to generate a final activity label based on the gating data; wherein the tracker is configured to use the predicted activity label as the final activity label when the uncertainty value is above a predetermined threshold, and to use the temporary activity label as the final activity label when the uncertainty value is below the predetermined threshold.

In accordance with an embodiment, a method for tracking a target using a millimeter-wave radar includes: receiving radar signals using the millimeter-wave radar; generating a range-Doppler map based on the received radar signals; detecting a target based on the range-Doppler map; using a tracker to track the target using a track; using a classifier to generate a temporary activity label based on an output of the tracker; and using the tracker to generate a final activity label based on an output of the classifier, where the tracker tracks activity labels of the target using state variables; wherein the tracker uses a predicted activity label as the final activity label when an uncertainty value associated with the temporary activity label is above a predetermined threshold, and uses the temporary activity label as the final activity label when the uncertainty value is below the predetermined threshold, wherein the tracker generates the predicted activity label based on the state variables.

The making and using of the embodiments disclosed are discussed in detail below. It should be appreciated, however, that the present invention provides many applicable inventive concepts that can be embodied in a wide variety of specific contexts.

The description below illustrates the various specific details to provide an in-depth understanding of several example embodiments according to the description. Known structures, materials or operations are not shown or described in detail so as not to obscure the different aspects of the embodiments. References to "an embodiment" in this description indicate that a particular configuration, structure or feature described in relation to the embodiment is included in at least one embodiment. Consequently, phrases such as "in one embodiment" that may appear at different points of the present description do not necessarily refer exactly to the same embodiment. Furthermore, specific formations, structures or features may be combined in any appropriate manner in one or more embodiments.

Embodiments of the present invention will be described in a specific context, a millimeter-wave radar that integrates a tracker that tracks human targets with a classifier that classifies human activities. Some embodiments may be used to track targets other than humans, such as an animal (e.g., a dog or cat) or a robot. Some embodiments may be used to classify targets using a criteria different from human activities, such as classifying the type of target (e.g., whether it is human or non-human).

In an embodiment of the present invention, a millimeter-wave radar includes a tracker that has an integrated classifier for classifying human target activities. The tracker implements the integrated classifier by using state variables corresponding to possible human activities. The tracker updates the state variables associated with human activities based on an output of a classifier that generates a temporary classification based on an output of the tracker. In some embodiments, thus, the tracker is advantageously capable of tracking, in addition to the localization of the human target, the corresponding human activities.

A millimeter-wave is used to detect and track targets, such as humans.

For example, <FIG> shows radar system <NUM>, according to an embodiment of the present invention. Radar system <NUM> includes millimeter-wave radar <NUM> and processor <NUM>. In some embodiments, millimeter-wave radar <NUM> includes processor <NUM>.

During normal operation, millimeter-wave radar <NUM> transmits a plurality of radiation pulses <NUM>, such as chirps, towards scene <NUM>. In some embodiments the chirps are linear chirps (i.e., the instantaneous frequency of the chirp varies linearly with time).

The transmitted radiation pulses <NUM> are reflected by objects in scene <NUM>. The reflected radiation pulses (not shown in <FIG>), which are also referred to as the echo signal, are detected by millimeter-wave radar <NUM> and processed by processor <NUM> to, for example, detect and track targets, such as humans.

The objects in scene <NUM> may include static humans, such as lying human <NUM>, humans exhibiting low and infrequent motions, such as standing human <NUM>, and moving humans, such as running or walking human <NUM> and <NUM>. The objects in scene <NUM> may also include static objects, such as furniture and periodic movement equipment (not shown). Other objects may also be present in scene <NUM>.

Processor <NUM> analyses the echo data to determine the location of humans using signal processing techniques. For example, in some embodiments, a range FFT is used for estimating the range component of the location of a detected human (e.g., with respect to the location of the millimeter-wave radar). The azimuth component of the location of the detected human may be determined using angle estimation techniques.

In some embodiments, a range-Doppler map (image) is generated from the echo data, and a two-dimensional (2D) moving target identification (MTI) is performed on the range-Doppler map to detect moving targets.

Processor <NUM> may be implemented as a general purpose processor, controller or digital signal processor (DSP) that includes, for example, combinatorial circuits coupled to a memory. In some embodiments, the processor <NUM> may be implemented with an ARM architecture, for example. In some embodiments, processor <NUM> may be implemented as a custom application specific integrated circuit (ASIC). In some embodiments, processor <NUM> includes a plurality of processors, each having one or more processing cores. In other embodiments, processor <NUM> includes a single processor having one or more processing cores. Other implementations are also possible. Some embodiments may be implemented as a combination of hardware accelerator and software running on a DSP or general purpose micro-controller.

In some embodiments, millimeter-wave radar <NUM> operates as a FMCW radar that includes, a millimeter-wave radar sensor circuit, a transmitting antenna, and at least two receiving antennas. Millimeter-wave radar <NUM> transmits and receives signals in the <NUM> to <NUM> range. Alternatively, frequencies outside of this range, such as frequencies between <NUM> and <NUM>, or frequencies between <NUM>, and <NUM>, may also be used.

In some embodiments, the echo signals received by the receiving antennas of millimeter-wave radar <NUM> are filtered and amplified using band-pass filter (BPFs), low-pass filter (LPFs), mixers, low-noise amplifier (LNAs), and intermediate frequency (IF) amplifiers in ways known in the art.

The echo signals are then digitized using one or more analog-to-digital converters (ADCs) for further processing. Other implementations are also possible.

Detecting and tracking human target(s), e.g., in an indoor environment, may be desirable for a variety of reasons. Conventional methods for tracking a target assume that that the target is a single point in the range-Doppler map. In a conventional range-Doppler processing chain, the cluster of detections obtained is used to obtain a single bin in a range-Doppler image to determine range and Doppler components of the target detected. Such single bin is then fed into the tracker for tracking the target. For example, in conventional radar signal processing, the range, Doppler and angle of arrival may be detected for the single point target. Such components are then fed into the tracker for tracking purposes.

The motion model for a conventional tracker may be expressed as <MAT> where k represents a discrete time step, Δt is the time between each time step, px is the position of the target in the x direction, py is the position of the target in the y direction, vx is the velocity of the target in the x direction, and vy is the velocity of the target in the y direction.

In some radar systems, such as in a millimeter-wave radar system, a human target may exhibit a double spread across range and Doppler bins as reflections are received from different parts of the human body during movement of the human target. For example, <FIG> shows a range-Doppler map of a moving human captured with a millimeter-wave radar system, according to an embodiment of the present invention. As shown in <FIG>, a human target may exhibit peaks at different locations of the range-Doppler map corresponding to different portions of the human target body, such as the right foot <NUM>, left foot <NUM>, and torso and hands <NUM>.

In some embodiments, a tracker may use a coordinated turn motion model, such as described in co-pending <CIT>, and entitled "Human Target Tracking System and Method". For example, in some embodiments, the coordinated turn motion model used by the tracker may be given by <MAT> where k represents a discrete time step, Δt is the time between each time step, px is the position of the centroid of the human target in the x direction, py is the position of the centroid of the human target in the y direction, lx is the bounding box dimension in the x direction, ly is the bounding box dimension in the y direction, vlx is the rate of change in the x dimension of the bounding box, vly is the rate of change in the y dimension of the bounding box, vc is the radial velocity of the centroid of the human target, θ is the angle of arrival of the human target, and ω is the rate of change of the angle of arrival (AoA) of the human target, where the bounding box may have a rectangular shape and surround the spread in Doppler and range of a target.

In an embodiment, parameters px, py, lx, ly, vlx, vly, vc, θ, and/or ω represent states of a tracker (e.g., implemented with a Kalman filter, such as the unscented Kalman filter) that tracks a human target with a track. These states are obtained from measurements of millimeter-wave radar system <NUM>. For example, in an embodiments, millimeter-wave radar system <NUM> measures r (range of the target from the millimeter-wave radar sensor), θ (angle of arrival - angle of the target), vc (radial velocity of the target), Lr (bounding box dimension across range), and Ld (bounding box dimension across Doppler). These measurements may collectively be referred to as <MAT>.

The Zmeas measurements may be converted into states of the tracker by <MAT> where ω, vlx, and vly are initialized as zero.

At each time step, a new set of measurements Zmeas is collected, and the track is updated based on such new set of measurements Zmeas. For example, the, e.g., unscented Kalman filter computes (for the track) predicted states px, py, lx, ly, vlx, vly, vc, θ, and ω, for every time step (e.g., using Equation <NUM>). Due to Bayes recursive approach, these states may embody information from all measurements available from time <NUM> :k - all measurements of the track). Such predicted states may be converted into the form of a predicted measurement by <MAT>.

The Mahalanobis distance may be computed between the predicted measurements Zpred and a new set of measurements Zmeas by <MAT> where S is the covariance matrix between Zmeas and Zpred.

In some embodiments, the new set of measurement Zmeas is considered a valid measurement of the target being tracked by the track if the distance Md is lower than a predetermined threshold. The multi-dimensional region in which the distance Md is lower than the predetermined threshold is referred to as the gating region or expected region associated to the track.

In many applications, it may be desirable to classify a target. For example, in some embodiments, it may be advantageous to classify a target as a human or non-human. For example, the lights of a room may be automatically be turned on when target is determined to be a human target (e.g., as opposed to a non-human target, such as a dog, for example).

Other classifications may also be advantageous. According to the invention, human targets are classified based on the activities being performed (e.g., walking, running, standing, sitting, lying down, etc.). For example, if a target is determined to be a human target, some embodiments may further assign an activity classification to such human target. Such activity classification may be used to enhance the localization tracking of the human target. For example, a human target may have a higher probability of remaining in the same place when the human target is performing the activity of sitting, than when the human target is performing the activity of running. Such activity classification may also be used for other purposes. For example, a system may take an action based on what type of activity the detected human is performing. For example, a system may turn on a warning light when a human target is detected performing the activity of lying in the floor, or from transitioning from walking to lying in the floor. In other words, the obtained (e.g., activity) label may be used to further track the target, e.g., by updating state variables of the tracker based on the (e.g., activity) label.

To classify a target, in examples not according to the invention a classifier may be used in open loop. For example, <FIG> shows block diagram <NUM> illustrating classifier <NUM> generating a target classification based on an output of tracker <NUM>. As shown in <FIG>, after a radar (e.g., millimeter-wave radar <NUM> together with processor <NUM>) performs target detection <NUM>, a tracker <NUM> tracks such target. Tracker localization (e.g., identifying the location of the target) is generated by the tracker <NUM>. A feature extraction <NUM> is performed based on the output of tracker <NUM>. The extracted feature (e.g., a Dopler spectrogram) is fed to a classifier <NUM> for classification purposes. The classifier then classifies the target based on the feature received. In some embodiments, detection parameters (parameters associated with <NUM>, <NUM>, and <NUM>) are updated each frame while the target classification is updated every N frames, where N may be, e.g., <NUM>.

Possible classifications include, for example, whether a target is human or non-human. Other classifications include, for example, activity classifications. For example, when a target is identified as a human target, possible activity classifications may include: running, walking, sleeping, sitting, waving, falling, cooking, working, etc..

During normal operation, misclassifications of a target may occur for a variety of reasons. For example, a radar, such as millimeter-wave radar <NUM>, may produce outputs that, when processed, may result in misclassifications. <FIG> illustrate scenarios that, when fed into a classifier, may result in misclassification of the target.

<FIG> shows graph <NUM> of a human target that is not detected during a portion of time, as shown by region <NUM>. During a misdetection, a track of the tracker <NUM> may not get associated with a detection, and, therefore, there may not be a feature extraction during that frame, which is a source of error that may result in misclassification.

<FIG> shows graph <NUM> of two human targets (<NUM> and <NUM>) located in the same range bins in a radar system having two receiving antennas. Generally, a radar system having only two receiving antennas is capable of localizing, at most, one target. When two targets are present in such system, an under-determined system of equations may result in the inability to extract features independently for each target, which is a source of error that may result in misclassification.

<FIG> shows graph <NUM> of a human target that is transitioning between activities (e.g., from walking to standing to sitting). When a human target performs an activity that was not previously used for training the classifier <NUM> (e.g., the activity was not part of the library used to train the classifier), the classifier <NUM> may assign a wrong classification to the target. Transients that may occur between transitions from a first activity (e.g., walking) to a second activity (e.g., standing) may also be a source of error that may result in misclassification.

In an embodiment, the state variables of a tracker include state variables associated with human activities. The tracker, thus, predicts, during a prediction step, the next human activity of the human target based on the history of the track. A classifier generates a temporary target classification based on features extracted from an output of the tracker. During the prediction step, the tracker assigns an uncertainty value to the temporary target classification (e.g., using Bayes recursion). The tracker then updates the human activity state variables with the predicted value when the classification uncertainty value is above a threshold, or with the temporary target classification when the uncertainty value is below the threshold. Then tracker then generates a target classification based on the updated human activity state variables.

By generating the target classification using the tracker, embodiments advantageously improve the classification accuracy when compared to applications that use a classifier in open loop configuration (without using the tracker to track human activities and/or without feeding the output of the classifier back to the tracker). For example, when the classifier produces a temporary human activity classification with a high uncertainty value (e.g., because the human activity is a new activity that was not used during the training of the classifier) the tracker uses the predicted activity instead.

Other advantages include that, in some embodiments, the localization tracking is enhanced by using the activity classification state. For example, when a first human target is sitting and a second human target walks near the location of the first human target, the data association when the human targets path cross may be enhance by the knowledge that the first human target is sitting (and, e.g., the probability of the first human target to remain sitting is high), and the second human target is walking (and, e.g., the probability of the second human target to remain walking is high). Thus, in this example, e.g., target detections near the first and second human targets corresponding to a target that is not moving are assigned to the track of the first human target while target detections near the first and second human targets corresponding to a moving target are assigned to the track of the second human target.

In an embodiment, the tracker includes, in addition to other state variables (e.g., as shown in Equations <NUM> and <NUM>) state variables a<NUM> to an, such as shown by <MAT> where a<NUM> corresponds to a first human activity, a<NUM> corresponds to a second human activity, and an corresponds to an nth human activity, where n is an integer number higher than <NUM>.

In some embodiments, the tracker performs, for state variables a<NUM> to an, a human activity prediction step given by <MAT> where k represents a discrete time step, and f is a transition matrix based on transition probabilities between human activities. In some embodiments, the transition probabilities are not linear. In some embodiments, the prediction step is performed every N frames, where N may be, e.g., <NUM>.

<FIG> shows human activity state transition diagram <NUM>, according to an embodiment of the present invention. As shown in <FIG>, only four activities are used in this embodiment. It is understood that fewer than four activities, such as, e.g., three or two, or more than four activities, such as five, eight, ten, or more, may also be used.

As a non-limiting example, in an embodiment, state a<NUM> corresponds to waiving, state a<NUM> corresponds to walking, state as corresponds to standing, and state a<NUM> corresponds to sitting. The probabilities for each state may or may not be the same. For example, in an embodiment, probability p<NUM> (e.g., for remaining sitting once the human target is already sitting) may be, e.g., <NUM>, probability p<NUM> (e.g., for remaining walking once the human target is already walking) may be, e.g., <NUM>, and probability p<NUM> (e.g., for remaining standing once the human target is already standing) may be, e.g., <NUM>.

The probabilities for transitioning between states may or may not be reciprocal. For example, probability p<NUM> (e.g., for transitioning from standing to walking) may be, e.g., <NUM> while probability p<NUM> (e.g., for transitioning from walking to standing) may be, e.g., <NUM>. And the probability p<NUM> (e.g., for transitioning from walking to sitting) may be, e.g., <NUM> while probability p<NUM> (e.g., for transitioning from sitting to walking) may be, e.g., <NUM>.

In some embodiments, the probability pi for the associated activity ai may be calculated by <MAT> where Q is the number of states, and Ai may be given by <MAT> where Wij is the transition weight for transitioning from activity i to activity j. In other words, Equation <NUM> may be understood as a softmax operation that normalizes the values of Ai to obtain a probability value pi for the activity ai.

<FIG> shows block diagram <NUM> for detecting, tracking, and classifying activities of human targets, according to an embodiment of the present invention. Blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> may be implemented, e.g., by processor <NUM>.

Radar processing block <NUM> generates a radar image based on radar signals received by millimeter-wave radar <NUM>. In some embodiments the radar image is a range-Doppler image. In other embodiments, the radar image is a range cross-range image. For example, in some embodiments, a millimeter-wave radar having two receiving antennas receive radar signals from the field of view (e.g., from reflections from human targets) and generate respective range-Doppler maps. Two dimensional (2D) MTI filters are applied to the respective range-Doppler maps, which results are coherently integrated. After coherent integration, a range-Doppler image is generated.

Target detector block <NUM> detects potential targets based on the received radar image. For example, in some embodiments, an order statistics (OS) constant false alarm rate (CFAR) (OS-CFAR) detection is performed by target detector <NUM>. Such CFAR detector generates a detection image in which, e.g., "ones" represent targets and "zeros" represent non-targets based, e.g., on the power levels of the range-Doppler image. For example, in some embodiments, the CFAR detector compares the power levels of the range-Doppler image with a threshold, and points above the threshold are labeled as targets while points below the threshold are labeled as non-targets. Although targets may be indicated by ones and non-targets may be indicated by zeros, it is understood that other values may be used to indicate targets and non-targets.

In an embodiment, the OS-CFAR detector uses k-th quartile/median instead of mean CA-CFAR (cell average). Using the k-th quartile /median may advantageously be more robust to outliers.

Detection gating block <NUM> generates a list of targets and associated parameters based on the detection image, data from tracker <NUM>, and, e.g., Equations <NUM>-<NUM>. For example, in some embodiments, targets present in the detection image are clustered using, e.g., a density-based spatial clustering of applications with noise (DBSCAN) algorithm. Parameter estimations are then performed for each clustered target, e.g., using one or more of Equations <NUM>-<NUM>. Detection gating block <NUM> then uses the estimated parameters, predicted parameters received from tracker <NUM>, and, e.g., Equations <NUM>, to screen for useful data, e.g., using ellipsoidal gating. Detection gating block <NUM> then generates the list of targets and associated estimated screened parameters. The parameters estimated and screened by detection gating block <NUM> include, for example, angle of arrival (AoA), bounding box dimensions (e.g., lx and ly), rate of change in bounding box dimensions (e.g., vlx and vly), etc., for the respective targets.

Tracker block <NUM> tracks each target using a respective track by performing a prediction step and an update step for each track. For example, during the prediction step, tracker <NUM> generates a prediction of the state variables of the tracker (for parameters associated with each track) based on the respective history of each respective track (e.g., using one or more of Equations <NUM>-<NUM>, and <NUM>-<NUM>). During the update step, tracker 808associates detected targets with respective tracks based on the output of detection gating block <NUM>.

Tracker block <NUM> may also generate and kill targets based on the history of the track and based on the new received measurements (e.g., the detected target and associated parameters). Tracker block <NUM> may also perform track filtering. For example, in some embodiments, tracker block <NUM> is implemented using an unscented Kalman filter or particle filter to predict one or more parameters associated with each target, such as the range, angle, bounding box dimensions, based on the history of the track and based on new measurements received.

Featured extraction block <NUM> extract features for each target from an output of tracker block <NUM>. For example, in some embodiments, a Doppler spectrogram is extracted from the history of the corresponding track tracked by tracker block <NUM>. The Doppler spectrogram is used by a classifier block <NUM>, such as a long short-term memory (LSTM) classifier, to generate a temporary target activity classification.

Classification gating block <NUM> receives the temporary target activity classification from classifier <NUM>, receives from tracker <NUM> an uncertainty value associated with such temporary target activity classification, and generates a gating data output to indicate whether to use or not to use the temporary target classification based on the uncertainty associated with the temporary target activity classification. For example, in some embodiments, an ellipsoidal gating is used to determine whether a particular temporary target classification should be used (e.g., whether an uncertainty value associated with the temporary target classification is above or below a predetermined threshold). For example, in some embodiments, classification gating block <NUM> determines a first (Mahalanobis) distance between the temporary target activity classification and the predicted activity classification (generated by tracker <NUM>). For example, in some embodiments, the Mahalanobis distance Md_act may be computed between the predicted measurements Zpred and a new set of measurements Zmeas by <MAT> where <MAT> is the covariance matrix between the activity predicted by the tracker Zpred_act and the activity measured by the classifier Zmeas_act.

The determination of the classification distance is also based on the uncertainty value associated with the temporary target activity classification. When the classification distance is lower than a threshold, classification gating block <NUM> generates a gating data output to signal tracker <NUM> to use the temporary target activity classification. When the classification distance is higher than the threshold, classification gating block <NUM> generates a gating data output to signal tracker <NUM> to use the predicted activity classification.

As shown in <FIG>, tracker <NUM> receives the temporary target activity classification (e.g., from classifier <NUM> or classification gating block <NUM>) and generates the uncertainty value associated with the temporary target activity classification (e.g., during the prediction step). For example, the tracker uses the classification probabilities from previous time steps to generate an uncertainty value associated with the temporary target activity classification.

Tracker block <NUM> also receives the gating data from classification gating block <NUM> and updates the state variables associated with human activities based on such gating data. For example, in some embodiments, tracker block <NUM> updates the human activity state variables with the value predicted by tracker block <NUM> when the gating data output indicates that the classification distance is higher than a threshold, and with the temporary target activity classification value when the classification distance is lower than the threshold. Other state variables may also be updated based on the updated human activity state variables. Then tracker block <NUM> generates a target activity classification and target localization based on the updated state variables. In some embodiments, the target activity classification generated by tracker block <NUM> is generated using an arg max function. For example, the human activity having the maximum probability weight is the activity selected as the target activity by tracker <NUM>.

As shown in <FIG>, tracker block <NUM> and classifier block <NUM> operate in a closed-loop manner. In some embodiments, classifier block <NUM> benefits from tracker block <NUM> because the tracker block <NUM> uses the assumption of Gaussian states and the uncertainty associated with it. Therefore, using tracker block <NUM> allows classifier block <NUM> to provide, with respect to a target classification, an associated probability and an associated uncertainty (since, in some embodiments, the Bayesian approach of tracker <NUM> adds an uncertainty measure to the output of classifier <NUM> that aids in the selection of the next human activity).

In some embodiments, tracker block <NUM> benefits from classifier block <NUM> because the state variables track the associated activities of the target, thereby enhancing the prediction step of the tracker (e.g., since more information may be used for the prediction). For example, when two targets cross, the activities associated with each target can help tracker block <NUM> to associate the right detections to the right tracks. In other words, in some embodiments, tracking human activity states with tracker block <NUM> improves the probability of correct assignment of the detected targets with respective tracks.

Advantages of some embodiments include an improved response to target misclassifications by using a target classification predicted by the tracker instead of a target classification determined by a classifier and having high uncertainty. In some embodiments, the tracker advantageously achieves a result similar to Bayesian deep learning without implementing the complex neural networks that are conventionally associated with Bayesian deep learning.

<FIG> shows a flow chart of embodiment method <NUM> for tracking human targets and corresponding activities, according to an embodiment of the present invention. Method <NUM> is implemented by millimeter-wave radar <NUM> and processor <NUM>.

During step <NUM>, millimeter-wave radar <NUM> transmits radar signals (e.g., <NUM>) towards a scene (e.g., <NUM>). For example, millimeter-wave radar <NUM> may transmit radar chirps organized in frames. During step <NUM>, millimeter-wave radar <NUM> receives the reflected radar signals.

During step <NUM>, a radar image is generated, e.g., by radar processing block <NUM>, based on the received reflected radar signals. During step <NUM>, a detection image (e.g., a matrix including "<NUM>" for detected targets and "<NUM>" otherwise) is generated, e.g., by target detector block <NUM>.

During step <NUM>, the detected targets of the detection image are clustered, e.g., by detection gating block <NUM>. For example, since a target human may exhibit a double spread across range and Doppler bins, it is possible that multiple detected targets in the detection image belong to a single human target. During step <NUM>, targets with high probability of belonging to a single human target are clustered together.

During step <NUM>, parameters of each clustered target are determined based on the radar measurements (e.g., based on the received reflected radar signals and subsequent processing). During step <NUM>, parameters such as r (range of the target from the millimeter-wave radar sensor), θ (angle of arrival - angle of the target), vc (radial velocity of the target), Lr (bounding box dimension across range), and Ld (bounding box dimension across Doppler) are determined for each clustered target.

During step <NUM>, a tracker tracking human targets, such as tracker <NUM>, generates a prediction for detection state variables for each human target being tracked based on the history of the corresponding track. Detection state variables may include one or more of the variables of Equations <NUM> and <NUM>, for example.

During step <NUM>, detection gating is performed, e.g., by detection gating block <NUM>. For example, for each tracked target, a detection distance may be determined between the measured parameters and the predicted parameters, e.g., using Equation <NUM>. When the measurement of a target is inside the gating region (i.e., the detection distance is below a predetermined threshold), the tracker updates the detection state variables of such target with the measured values during step <NUM>. When the measurement of the target is outside the gating region (i.e., the detection distance is above the predetermined threshold), the tracker updates the detection state variables of such target with the predicted values during step <NUM>.

During step <NUM>, a Doppler spectrogram is generated based on the updated detection state variables and on the radar image, e.g., by feature extraction block <NUM>. During step <NUM>, a temporary target activity classification is generated for each tracked human target, e.g., by classifier <NUM>. For example, classifier <NUM> may generate, for each tracked human target, a vector including probabilities for each possible activity. The temporary target activity classification (the activity label) of each tracked target is the activity having the highest probability of the corresponding vector.

During step <NUM>, predicted values for activity state variables are generated (e.g., by tracker <NUM>) for each track, e.g., using Equation <NUM>. During step <NUM>, an uncertainty value is associated with each temporary target activity classification based on the predicted value of the corresponding activity state variable of the corresponding target e.g., using Bayes recursion.

During step <NUM>, classification gating is performed, e.g., by classification gating block <NUM>. For example, in some embodiments, for each tracked target, a classification distance may be determined by computing a Mahalanobis distance between the temporary target activity classification and the predicted activity classification. When the temporary target activity classification of a target is inside the gating region (i.e., the classification distance is below a predetermined threshold), the tracker updates the classification state variables of such target with the temporary target activity classification during step <NUM>. When the temporary target activity classification of the target is outside the gating region (i.e., the classification distance is above the predetermined threshold), the tracker updates the classification state variables of such target with the predicted values during step <NUM>.

During step <NUM>, the final target classification is generated for each target based on the updated classification state variables of the tracker. During step <NUM>, the localization of the target is generates for each target based on the updated detection state variables of the tracker.

In an embodiment, a classifier uses data associated with human activity state transitions to improve human target activity classification. A first stage of the classifier is trained with human activity snippets. A second stage of the classifier is trained with human activity transition snippets.

<FIG> shows classifier <NUM> for classifying human activities, according to an embodiment of the present invention. Classifier <NUM> and <NUM> may be implemented as classifier <NUM>. Classifier <NUM> is implemented with an activity model that includes a first stage including bidirectional LSTM network <NUM>, fully connected (FC) layers <NUM>, and softmax layer <NUM>, and a second stage including FC layer <NUM>.

During normal operation, classifier <NUM> receives as input N spectrograms x<NUM> to xN, corresponding to N (consecutive) frames (e.g., from features extracted from tracker <NUM> or <NUM>). In some embodiments, N is an integer number higher than <NUM>, such as <NUM>, <NUM>, <NUM> or higher.

Bidirectional LSTM network <NUM> generates human target activity vectors based on respective input spectrograms x<NUM> to xN, which, after being processed by fully connected layer <NUM> and softmax layer <NUM>, result in output activity probabilities y<NUM> to yN. Fully connected layer <NUM>, which implements the transition matrix f from Equation <NUM>, receives the output activity probabilities y<NUM> to yN and generates corresponding final activity probabilities ŷ<NUM> to ŷN. Vector ŷN, which in some embodiments is produced every N frames, is a vector that includes the probabilities of each of the human activities considered. The human activity of vector ŷN having the higher probability corresponds to the "target classification" of <FIG>, and to the "temporary target activity classification" of <FIG>.

Bidirectional LSTM network <NUM>, fully connected layers <NUM> and <NUM>, and softmax layer <NUM> may be implemented in any way known in the art.

In some embodiments, classifier <NUM> is trained in two steps. A first step trains the first stage of classifier <NUM> while a second step trains the second stage of classifier <NUM>.

During the first step of training classifier <NUM>, first spectrogram snippets associated with human activities, where the first spectrogram snippets are truncated Doppler spectrograms that do not include transitions between activities, are supplied as inputs x<NUM> to xN. Bidirectional LSTM network <NUM> and layers <NUM> and <NUM> are trained while monitoring outputs y<NUM> to yN (e.g., as described with respect to <FIG>).

After the first stage of classifier <NUM> is trained during the first step of training classifier <NUM>, the second stage of classifier <NUM> is trained. During the second step of training classifier <NUM>, second spectrogram snippets associated with human activity transitions are supplied as inputs x<NUM> to xN. Fully connected layer <NUM> is trained while monitoring outputs ŷ<NUM> to ŷN (e.g., as described with respect to <FIG>). Bidirectional LSTM network <NUM> and layers <NUM> and <NUM> are not modified during the second step of training classifier <NUM>.

<FIG> shows Doppler spectrogram <NUM> of human activities and activity transitions of a human target, according to an embodiment of the present invention. As shown in <FIG>, a human target may transition between activities during a period of time. During the first step of training classifier <NUM>, the first spectrogram snippets may include snippets similar to activity snippets shown in <FIG> (e.g., walk, stand, sit, and wave). During the second step of training classifier <NUM>, the second spectrogram snippets may include snippets similar to transition snippets shown in <FIG> (e.g., transition from walk to stand, transition from stand to sit, and transition from sit to wave).

<FIG> illustrate plots <NUM>, <NUM>, and <NUM> showing actual activity classification results, the output of classifier <NUM> (e.g., <NUM>), and the output of tracker <NUM>, respectively, according to an embodiment of the present invention. Plots <NUM>, <NUM>, and <NUM> of <FIG> corresponds to Doppler spectrogram <NUM> of <FIG>. As shown by <FIG>, the output of tracker <NUM> (curve <NUM>) is closer to the actual activity classification result (curve <NUM>) than the output of classifier <NUM> (curve <NUM>). In other words, <FIG> shows an improved performance when using tracker <NUM> for classification of human activity when compared to using classifier <NUM> in open loop.

<FIG> show confusion matrices for classifier <NUM> and tracker <NUM>, respectively, according to an embodiment of the present invention. The confusion matrices of <FIG> correspond to plots <NUM>, <NUM>, and <NUM> of <FIG> and to Doppler spectrogram <NUM> of <FIG>. As illustrated in <FIG>, in an embodiment, the tracker <NUM> shows improved accuracy, with <NUM>% total actual correct predictions when compared to the classifier <NUM> accuracy alone, which shows <NUM>% total actual correct predictions.

By including activity transitions in the training of the classifier, some embodiments advantageously minimize or eliminate misclassifications arising from transitions between activities.

<FIG> illustrates block diagram <NUM> showing a machine learning pipeline for machine language based feature extraction and identification that can be used to train classifier <NUM> to classify human targets based on human activities. The top portion <NUM> of <FIG> is devoted to the processing of training features (e.g., Doppler spectrograms snippets) for configuring the first and second portions of classifier <NUM>. The bottom portion <NUM> is devoted to the processing new measurements using the trained classifier <NUM> (e.g., trained as shown in top portion <NUM>).

As shown in the top portion <NUM> of <FIG>, training data <NUM> is transformed into feature vectors <NUM> and corresponding labels <NUM>. Training data <NUM> represents the raw data (e.g., echo). Feature vectors <NUM> represent sets of generated vectors that are representative of the training data <NUM>. Labels <NUM> represent user metadata associated with the corresponding training data <NUM> and feature vectors <NUM>. For example, during the first step of training classifier <NUM>, training data <NUM> includes first spectrogram snippets associated with human activities and corresponding labels (e.g., walk, stand, sit, wave, etc.). During the second step of training classifier <NUM>, training data <NUM> includes second spectrogram snippets associated with human activity transitions with human activities and corresponding labels (e.g., transition from walk to stand, transition from stand to sit, transition from sit to wave, etc.).

As shown, training data <NUM> is transformed into feature vectors <NUM> using image formation algorithms. Data preparation block <NUM> represents the initial formatting of raw sensor data, and data annotation block <NUM> represents the status identification from training data <NUM>.

During training, one or more radar images are taken of a controlled environment that includes one or more static and moving targets (e.g., humans, moving machinery, furniture, and other moving equipment) using, e.g., a millimeter-wave radar. In some cases, multiple radar images are recorded to increase the accuracy of identification. Machine learning algorithm <NUM> evaluates the ability of a prediction model <NUM> to identify feature vectors and iteratively updates training data <NUM> to increase the classification accuracy of the algorithm. The training performance of the machine learning algorithm may be determined by calculating the cross-entropy performance. In some embodiments, the machine learning algorithm <NUM> iteratively adjusts image formation parameters for a classification accuracy of at least <NUM>%. <NUM> Alternatively, other classification accuracies could be used.

Machine learning algorithm <NUM> may be implemented using a variety of machine learning algorithms known in the art. For example, a random forest algorithm or neural network algorithm, such as a ResNet-<NUM> or other neural network algorithm known in the art, may be used for classification and analysis of feature vectors <NUM>. During the iterative optimization of feature vectors <NUM>, a number of parameters of image formation <NUM> may be updated.

Claim 1:
A method for tracking a target (<NUM>) using a millimeter-wave radar, the method comprising:
receiving radar signals (<NUM>) using the millimeter-wave radar;
generating a range-Doppler map based on the received radar signals (<NUM>);
detecting a target (<NUM>) based on the range-Doppler map;
tracking the target (<NUM>) using a track;
generating a predicted activity label (<NUM>) based on the track, wherein the predicted activity label is indicative of an actual activity of the target;
generating a Doppler spectrogram (<NUM>) based on the track;
generating a temporary activity label (<NUM>) based on the Doppler spectrogram;
assigning an uncertainty value (<NUM>) to the temporary activity label, wherein the uncertainty value is indicative of a confidence level that the temporary activity label is an actual activity of the target; and
generating a final activity label (<NUM>) based on the uncertainty value; wherein generating the final activity label comprises, when the uncertainty value is above a predetermined threshold, using the predicted activity label as the final activity label, and when the uncertainty value is below the predetermined threshold, using the temporary activity label as the final activity label.