Patent Description:
Applications in the millimeter-wave frequency regime have gained significant interest in the past few years due to the rapid advancement in low cost semiconductor technologies, such as silicon germanium (SiGe) and fine geometry complementary metal-oxide semiconductor (CMOS) processes. Availability of high-speed bipolar and metal-oxide semiconductor (MOS) transistors has led to a growing demand for integrated circuits for millimeter-wave applications at e.g., <NUM>, <NUM>, <NUM>, and <NUM> and also beyond <NUM>. Such applications include, for example, automotive radar systems and multi-gigabit communication systems.

In some radar systems, the distance between the radar and a target is determined by transmitting a frequency modulated signal, receiving a reflection of the frequency modulated signal (also referred to as the echo), and determining the distance based on a time delay and/or frequency difference between the transmission and reception of the frequency modulated signal. Accordingly, some radar systems include a transmit antenna to transmit the radio-frequency (RF) signal, and a receive antenna to receive the reflected RF signal, as well as the associated RF circuits used to generate the transmitted signal and to receive the RF signal. In some cases, multiple antennas may be used to implement directional beams using phased array techniques. A multiple-input and multiple-output (MIMO) configuration with multiple chipsets can be used to perform coherent and non-coherent signal processing as well.

Besides determining only the distance to a target, tracking of detected targets is desirable in many applications, for example to determine how a target moves from radar frame to radar frame. While this is comparatively straight forward in case a single target is present, for example a single person, it may become a greater challenge in case of multiple targets, for example multiple persons in a scene.

One approach used for tracking uses a digital filtering of the acquired radar data, for example using a extended or unscented Kalman filter. A Kalman filter or other digital filter may perform the tracking based on various parameters.

<CIT> discloses a radar device according to the preamble of claim <NUM>.

<CIT> discloses a machine learning model which comprises a policy network.

A radar device as defined in claim <NUM> and a method as defined in claim <NUM> are provided. The dependent claims define further embodiments.

According to an embodiment, a radar device is provided, comprising:.

According to another embodiment, a method is provided, comprising:.

The above summary is merely intended to give a brief overview over some embodiments and is not to be construed as limiting, as other embodiments may include different features from the ones discussed above.

In the following, various embodiments will be described in detail referring to the attached drawings. These embodiments are given by way of example only and are not to be construed as limiting any way. For example, while embodiments are described including a plurality of features (for example components, elements, acts, events, steps or the like), in other embodiments some of these features may be omitted, or may be replaced by alternative features. Furthermore, in addition to the features explicitly described, other features may be provided, for example features used in conventional radar devices and associated methods.

Details or variations described with respect to one of the embodiments may also be applied to other embodiments and therefore will not be described repeatedly. Features from different embodiments may be combined to form further embodiments.

Turning now to the Figures, <FIG> is a block diagram of a radar device according to an embodiment. The radar device of <FIG> includes a radar frontend <NUM> which transmits radar signals (TX) towards a scene <NUM> and receives reflected radar signals (RX) from scene <NUM>, for example reflected from objects like persons. Radar frontend <NUM> processes the received radar signals. Such a processing may for example include mixing and the like. Radar frontend <NUM> may be implemented in any conventional manner known to the skilled person. A particular implementation will be discussed further below referring to <FIG>.

Radar frontend outputs analog radar signals ra to processing circuitry <NUM>. Processing circuitry <NUM> further processes and digitizes the radar signals and outputs digital radar data rd, for example so called radar images. Digital radar data rd is provided to a digital filter <NUM> in order to obtain information about objects in scene <NUM>, said information labelled td in <FIG>. For example, information td may be tracking information about objects like persons. Digital filter <NUM> may be a extended or unscented Kalman filter. Apart from the digital filtering, further conventional processing may be applied to obtain information td.

Digital filter <NUM> may perform the filtering based on various parameters, also referred to as hyperparameters herein. For example, such parameters may be parameters of the state transition model or a control input model used by the Kalman filter. Other hyperparameters include process angle variance or measurement noise. In a particular example, digital filter <NUM> is a tracking filter and may use about <NUM> adjustable tracking-parameters. These may include for example six parameters for noise measurement/measuring noise, five parameters for processing noise, two parameters for opening and deleting tracks and, finally, a Mahalanobis distancebased gating parameter that decides if a measurement is associated with any track. In summary, the hyperparameters describe and model the scene, where the tracker is performing the location of the target.

The optimum parameters for processing radar signals obtained from scene <NUM> depend on the scene and the environment. For example, the optimum parameters may depend on a number of objects, for example a number of people to track, noise, false tracking targets, movement of the target, and the general environment of the objects to be tracked. Optimized parameters for a particular scene tend to give better results than general parameters.

Therefore, in the radar device of <FIG>, a machine learning logic <NUM> sets the parameters p of digital filer <NUM> based on digital radar data rd2 which may be the same as digital radar data rd, and/or may include data in another stage of the processing pipeline processing the received radar signals.

A machine learning logic, as used herein, refers to an entity using approaches that learn based on training data or other data to improve its performance. Common types of machine learning logic include artificial neural networks, decision trees, support vector machines and the like. In embodiments, a machine learning logic is used which includes at least a policy network, which provides the parameters p based on data rd2, and a critic network which provides at least one reward value, also referred to as Q-Value, for the choices the policy network makes. An example embodiment of such a machine learning logic is illustrated in <FIG>.

As shown in <FIG>, in an embodiment the machine learning logic includes a policy network <NUM> and a critic network <NUM>. A policy network is sometimes also referred to as actor network herein. Policy network <NUM> and critic network <NUM> may both be implemented as artificial neural networks, for example convolutional neural networks (CNN), including a plurality of layers. Policy network <NUM>, in a trained state, receives radar data rd2 and outputs the parameters p for digital filter <NUM> of <FIG>. Critic network <NUM> also receives radar data rd2 or any other data indicative of the reflected radar signals and also the parameters set by policy network <NUM> and outputs a plurality of reward values, also referred to as Q-Values, herein. To this end, critic network <NUM> includes a plurality of heads h1 to hn, where n is equal to or greater than <NUM>. For example, n may be at or around <NUM>.

In this contribution, head is the terminal part of a network (in this case, the critic network), which is purposed to output a value.

A head, as used herein, outputs a value which is independent from other given heads, coming from the same network. As will be explained later, different heads may be trained using different training data. For example, the different heads may include completely separate neural networks, together forming critic network <NUM>, or may for example have common input layers and/or intermediate layers, but different output layers and/or other intermediate layers, i.e. may have some common layers, but additionally comprise different layers. Training of the critic network <NUM> will also be explained later in greater detail.

As the heads h1 to hn were trained differently, e.g. using different training data, they yield different reward values. In case the scene to be processed, as represented by radar data rd2, is within the trained range, however, the reward values output by heads h1 to hn will be comparatively close to each other, i.e. have a narrow distribution. If, however, the scene <NUM> to be processed is rather outside of a trained range of the network, the distribution will be broader. The trained range is e.g. a range of objects (number, type or the like)covered by the training data For instance, training data in a training phase of the machine learning logic of <FIG> may include scenes with up to three objects to be tracked by digital filer <NUM>. If a scene to be processed then includes four or five objects, this is outside the trained range, and typically leads to a broader distribution of the reward values provided by heads h1 to hn. The reward value is generated by heads h1 to hn are provided to an out of distribution (OOD) detection <NUM>. Based on the distribution of the reward values, OOD detection <NUM> detects if the scene <NUM> to be processed is outside the training range. For example, OOD detection <NUM> may compare the standard deviation of the distribution of reward values generated by heads h1 to hn with a predefined threshold value, and if the standard deviation exceeds the threshold value, decides on an out of distribution event. In other embodiments, OOD detection <NUM> may check if one or more values generated by heads h1 to hn are outside an expected distribution width, obtained for example based on training data. This, in turn, indicates that the result td of digital filter <NUM>, which was obtained based on parameters p set by machine learning logic <NUM>, may not be reliable, for example that the tracking result may be erroneous. In other words, the out of distribution event indicates a comparatively high likelihood that information, e.g. tracking information, td output by digital filter <NUM> is erroneous, for example that the tracking result does not correctly indicate the actual moving objects like persons in scene <NUM>.

<FIG> is a flow chart illustrating a method according to an embodiment. The method of <FIG> may be implemented in the radar device illustrated with reference to <FIG> or other radar devices, for example radar devices as explained further below. To avoid repetitions, the method of <FIG> will be discussed referring to the previous explanations for <FIG>.

At <NUM> in <FIG>, the method comprises receiving reflected radar signals from a scene, as explained with reference to <FIG>.

At <NUM>, the method comprises processing the reflected radar signal to obtain a digital signal, for example by radar frontend <NUM> and processing circuitry <NUM> of <FIG>.

At <NUM>, the method comprises setting a filter parameter of a digital filter based on the digital signal, as explained for digital filter <NUM> in <FIG>, using a machine learning logic like machine learning logic <NUM>, for example implemented as explained with respect to <FIG>.

At <NUM>, the method comprises filtering the digital signal based on the filter parameters to obtain information about objects, for example tracking data, as explained for digital filter <NUM> of <FIG>.

At <NUM>, the method comprises providing a plurality of reward values for the setting of the filter parameters, for example by heads h1 to hn of <FIG>.

At <NUM>, the method comprises an out of distribution detection based on the plurality of reward values, for based on a distribution of the reward values as explained above.

Example implementation details of the device of <FIG>, in particular radar frontend <NUM>, processing circuitry <NUM> and digital filter <NUM>, will now be explained referring to <FIG>. However, it is to be noted that this is merely an example implementation, and other conventional implementations may also be used.

As mentioned above, a radar, such as a millimeter-wave radar, may be used to detect and track humans. For example, <FIG> shows a schematic diagram of millimeter-wave radar device <NUM>, according to an embodiment of the present invention. Millimeter-wave radar device <NUM> includes millimeter-wave radar sensor <NUM> and processing system <NUM>.

During normal operation, millimeter-wave radar sensor <NUM> operates as a frequency-modulated continuous-wave (FMCW) radar sensor and transmits a plurality of TX radar signals <NUM>, such as chirps, towards scene <NUM> using transmitter (TX) antenna <NUM>. The radar signals <NUM> are generated using RF and analog circuits <NUM>. The radar signals <NUM> may be in the <NUM> to <NUM> range. The objects in scene <NUM> may include one or more humans, which may be moving or idle, for example. Other objects may also be present in scene <NUM>, other moving or static objects, such as furniture, machinery, mechanical structures, walls, etc..

The radar signals <NUM> are reflected by objects in scene <NUM>. The reflected radar signals <NUM>, which are also referred to as the echo signal, are received by receiver (RX) antennas 116a and 116b. RF and analog circuits <NUM> processes the received reflected radar signals <NUM> using, e.g., band-pass filters (BPFs), low-pass filters (LPFs), mixers, low-noise amplifier (LNA), and/or intermediate frequency (IF) amplifiers in ways known in the art to generate an analog signal xouta(t) and xoutb(t), which are examples for signal ra of <FIG>.

The analog signals xouta(t) and xoutb(t) are converted to raw digital data xout_dig(n) using ADC <NUM>. The raw digital data xout_dig(n) is processed by processing system <NUM> to detect humans and their positions, and to track the detected humans. Although <FIG> illustrates a radar system with a two receiver antennas <NUM>, it is understood that more than two receiver antennas <NUM>, such as three or more, may also be used.

Although <FIG> illustrates a radar system with a single transmitter antenna <NUM>, it is understood that more than one transmitter antenna <NUM>, such as two or more, may also be used.

Controller <NUM> controls one or more circuits of millimeter-wave radar sensor <NUM>, such as RF and analog circuit <NUM> and/or ADC <NUM>. Controller <NUM> may be implemented, e.g., as a custom digital or mixed signal circuit, for example. Controller <NUM> may also be implemented in other ways, such as using a general purpose processor or controller, for example. In some embodiments, processing system <NUM> implements a portion or all of controller <NUM>.

Processing system <NUM> may be implemented with a general purpose processor, controller or digital signal processor (DSP) that includes, for example, combinatorial circuits coupled to a memory. In some embodiments, processing system <NUM> may be implemented as an application specific integrated circuit (ASIC). In some embodiments, processing system <NUM> may be implemented with an ARM, RISC, or x86 architecture, for example. In some embodiments, processing system <NUM> may include an artificial intelligence (AI) accelerator. Some embodiments may use a combination of hardware accelerator and software running on a DSP or general purpose microcontroller. Other implementations are also possible.

In some embodiments, millimeter-wave radar sensor <NUM> and a portion or all of processing system <NUM> may be implemented inside the same integrated circuit (IC). For example, in some embodiments, millimeter-wave radar sensor <NUM> and a portion or all of processing system <NUM> may be implemented in respective semiconductor substrates that are integrated in the same package. In other embodiments, millimeter-wave radar sensor <NUM> and a portion or all of processing system <NUM> may be implemented in the same monolithic semiconductor substrate. Other implementations are also possible.

As a non-limiting example, RF and analog circuits <NUM> may be implemented, e.g., as shown in <FIG>. During normal operation, VCO <NUM> generates a radar signal, such as a linear frequency chirp (e.g., from <NUM> to <NUM>, or from <NUM> to <NUM>), which is transmitted by transmitting antenna <NUM>. The VCO <NUM> is controlled by PLL <NUM>, which receives a reference clock signal (e.g., <NUM>) from reference oscillator <NUM>. PLL <NUM> is controlled by a loop that includes frequency divider <NUM> and amplifier <NUM>.

The TX radar signal <NUM> transmitted by transmitting antenna <NUM> is reflected by objects in scene <NUM> and received by receiving antennas 116a and 116b. The echo received by receiving antennas 116a and 116b are mixed with a replica of the signal transmitted by transmitting antenna <NUM> using mixer 146a and 146b, respectively, to produce respective intermediate frequency (IF) signals xIFa(t) xIFb(t) (also known as beat signals). In some embodiments, the beat signals xIFa(t) xIFb(t) have a bandwidth between <NUM> and <NUM>. Beat signals with a bandwidth lower than <NUM> or higher than <NUM> is also possible.

Beat signals xIFa(t) xIFb(t) are filtered with respective low-pass filters (LPFs) 148a and 148b and then sampled by ADC <NUM>. ADC <NUM> is advantageously capable of sampling the filtered beat signals xouta(t) xoutb(t) with a sampling frequency that is much smaller than the frequency of the signal received by receiving antennas 116a and 116b. Using FMCW radars, therefore, advantageously allows for a compact and low cost implementation of ADC <NUM>, in some embodiments. The raw digital data xout_dig(n), which in some embodiments include the digitized version of the filtered beat signals xouta(t) and xoutb(t), is (e.g., temporarily) stored, e.g., in matrices of Nc x Ns per receiver antenna <NUM>, where Nc is the number of chirps considered in a frame and Ns is the number of transmit samples per chirp, for further processing by processing system <NUM>.

In some embodiments, ADC <NUM> is a <NUM>-bit ADC with multiple inputs. ADCs with higher resolution, such as <NUM>-bits or higher, or with lower resolution, such as <NUM>-bits, or lower, may also be used. In some embodiments, an ADC per receiver antenna may be used. Other implementations are also possible.

<FIG> shows a sequence of chirps <NUM> transmitted by TX antenna <NUM>, according to an embodiment of the present invention. As shown by <FIG>, chirps <NUM> are organized in a plurality of frames and may be implemented as up-chirps. Some embodiments may use down-chirps or a combination of up-chirps and down-chirps, such as up-down chirps and down-up chirps. Other waveform shapes may also be used.

As shown in <FIG>, each frame may include a plurality of chirps <NUM> (also referred to, generally, as pulses). For example, in some embodiments, the number of pulses in a frame is <NUM>. Some embodiments may include more than <NUM> pulses per frame, such as <NUM> pulses, <NUM> pulses, or more, or less than <NUM> pulses per frame, such as <NUM> pulses, <NUM> pulses, <NUM> or less. In some embodiments, each frame includes only a single pulse. Frames are repeated every FT time. In some embodiments, FT time is <NUM>. A different FT time may also be used, such as more than <NUM>, such as <NUM>, <NUM>, <NUM>, or more, or less than <NUM>, such as <NUM>, <NUM>, or less.

In some embodiments, the FT time is selected such that the time between the beginning of the last chirp of frame n and the beginning of the first chirp of frame n+<NUM> is equal to PRT. Other embodiments may use or result in a different timing.

The time between chirps of a frame is generally referred to as pulse repetition time (PRT). In some embodiments, the PRT is <NUM>. A different PRT may also be used, such as less than <NUM>, such as <NUM>, <NUM>, or less, or more than <NUM>, such as <NUM>, or more.

The duration of the chirp (from start to finish) is generally referred to as chirp time (CT). In some embodiments, the chirp time may be, e.g., <NUM>. Higher chirp times, such as <NUM>, or higher, may also be used. Lower chirp times, may also be used.

In some embodiments, the chirp bandwidth may be, e.g., <NUM>. Higher bandwidth, such as <NUM> or higher, or lower bandwidth, such as <NUM>, <NUM>, or lower, may also be possible.

In some embodiments, the sampling frequency of millimeter-wave radar sensor <NUM> may be, e.g., <NUM>. Higher sampling frequencies, such as <NUM> or higher, or lower sampling frequencies, such as <NUM> or lower, may also be possible. In some embodiments, the number of samples used to generate a chirp may be, e.g., <NUM> samples. A higher number of samples, such as <NUM> samples, or higher, or a lower number of samples, such as <NUM> samples or lower, may also be used.

<FIG> shows a flow chart illustrating operation of an embodiment <NUM> of processing system <NUM> for people tracking, according to an embodiment of the present invention.

During steps 302a and 302b, raw ADC data xout_dig(n) is received, e.g., from millimeter-wave radar sensor <NUM>. As shown, the raw ADC data xout_dig(n) includes separate baseband radar data from multiple antennas (e.g., <NUM> in the example shown in <FIG>).

During steps 304a and 304b, signal conditioning, low pass filtering and background removal are performed on the raw ADC data of the respective antenna <NUM>. The raw ADC data xout_dig(n) radar data are filtered, DC components are removed to, e.g., remove the Tx-Rx self-interference and optionally pre-filtering the interference colored noise. Filtering may include removing data outliers that have significantly different values from other neighboring rangegate measurements. Thus, this filtering also serves to remove background noise from the radar data.

During steps 306a and 306b, 2D moving target indication (MTI) filters are respectively applied to data produced during steps 304a and 304b to remove the response from static targets. The MTI filter may be performed by subtracting the mean along the fast-time (intra-chirp time) to remove the transmitter-receiver leakage that perturbs the first few range bins, followed by subtracting the mean along the slowtime (inter-chirp time) to remove the reflections from static objects (or zero-Doppler targets).

During steps 308a and 308b, a series of FFTs are performed on the filtered radar data produced during steps 306a and 306b, respectively. A first windowed FFT having a length of the chirp is calculated along each waveform for each of a predetermined number of chirps in a frame of data. The FFTs of each waveform of chirps may be referred to as a "range FFT. " A second FFT is calculated across each range bin over a number of consecutive periods to extract Doppler information. After performing each 2D FFT during steps 308a and 308b, range-Doppler images are produced, respectively.

During step <NUM>, a minimum variance distortionless response (MVDR) technique, also known as Capon, is used to determine angle of arrival based on the range and Doppler data from the different antennas. A range-angle image (RAI) is generated during step <NUM>. In some embodiments, a range-Doppler-angle data cube is generated during step <NUM>.

During step <NUM>, an ordered statistics (OS) Constant False Alarm Rate (OS-CFAR) detector is used to detect targets. The CFAR detector generates a detection image in which, e.g., "ones" represent targets and "zeros" represent non-targets based, e.g., on the power levels of the RAI, by comparing the power levels of the RAI with a threshold, points above the threshold being labeled as targets ("ones") while points below the threshold are labeled as non-targets ("zeros). In some embodiments, targets present in the detection image generated during step <NUM> are clustered during step <NUM>, e.g., based on similar feature characteristics, such as empirical mode decomposition (EMD), and/or scale invariant feature transform (SIFT), associated with the detected targets. In some embodiments, other types of features of the detected targets, such as motion model-based features based on, e.g., range, Doppler, and/or angle, may also be used to cluster cells together. In some embodiments, metrics such as correlation and/or Wasserstein distance may be used to determine the similarities between clusters. In some embodiments, the feature-based clustering is performed by using k-means clustering, in which targets are grouped (clustered) based on having similar features to the one of k clusters having the nearest mean of such (e.g., combined) features.

For example, in some embodiments, a vector of features includes a plurality of features (e.g., intrinsic mode functions (IMFs) and/or number of IMFs, which are associated with EMD, and/or magnitude M(m,n) and/or phase ϕ(m,n), which are associated with SIFT), where each channel describes a type of feature (e.g., IMFs, number of IMFs, magnitude M(m,n) and/or phase ϕ(m,n)). Each channel may be described as a Gaussian distribution (taking mean and variance over available vectors of the same feature). A weighted sum over all the different Gaussian distributions over the channels is obtained to provide a descriptor for each cell, where the descriptor is associated with all the feature types and which may be a value or vector that is indicative of the characteristics (features) of the associated cluster and which may be used to determine how similar are clusters. Such descriptor is used for clustering, e.g., using the k-means clustering algorithm.

In some embodiment, a density-based spatial clustering of applications with noise (DBSCAN) algorithm may also be used to associate targets to clusters during step <NUM>. The output of DBSCAN is a grouping of the detected points into particular targets. DBSCAN is a popular unsupervised algorithm, which uses minimum points and minimum distance criteria to cluster targets, and may be implemented in any way known in the art. Other clustering algorithms may also be used.

In some embodiments, thus, clustering results in the radar image (e.g., RAI or rdI) or data cube being divided into groups of cells with similar descriptors. In some embodiments, each cluster corresponds to a (e.g., potential) detected target. Since the spread of features is not necessarily uniform, in some embodiments, each cluster is not necessarily equal. Thus, in some embodiments, the radar image or data cube is divided into clusters of cells, but each cluster of cells is not necessarily of the same size (e.g., does not have the same number of cells/sub-cells).

During step <NUM>, detected (clustered) targets are associated with respective tracks. As will be described in more detail later, in some embodiments, detected targets are associated to respective tracks using feature-based template matching (during step <NUM>). For example, in some embodiments, geometric features are used during step <NUM> for template matching. A geometric feature may be understood as a feature that is recognizable despite changes in rotation of the target, as well as changes in the range, Doppler velocity, and angle of the centroid of the target. In some embodiments a geometric feature may include a physical geometric feature, such as physical edges of the target (e.g., from the radar image). In some embodiments, additionally or alternatively, a geometric feature may include a metric (e.g., a vector, function, or group of functions) based on the relationship between cells of the raw data (e.g., of the data cube), such as the relationship between range cells, Doppler velocity cells, and/or angle cells. Examples of such metric include functions extracted using functional decomposition of the data cube, gradients of the data cube, and/or statistical properties of the data cube (such as histograms/PDF of the data cube). Examples of geometric features include EMD features and SIFT features.

In some embodiments, geometric features allow for identification of a target without relying on a motion model. In some embodiments, geometric features allow for distinguishing between tracked targets.

In some embodiments, geometric features such as EMD, and/or SIFT, are tracked for each target. For each clustered cell (for each detected target) a feature vector is generated for each time step i with values of each feature associated with the clustered cell. Detected targets at time step i + <NUM> are assigned to respective tracks based on the similarities between feature vectors (e.g., based on the error between the feature vectors), e.g., using Hungarian assignment. For example, in some embodiments, a similarity measure is identified between feature clusters at consecutive time steps (e.g., i, and i + <NUM>), and the assignments that minimize the error (e.g., increase correlation) between feature clusters is selected for track assignment.

In some embodiments, the data association step (<NUM>) may include, additionally, data association methods that do not rely on featured-based template matching.

In some embodiments, the data assignment of detected targets (clusters) to tracks relies on the geometric features of the cluster and does not rely (or does not rely solely) on the actual physical locations and/or velocities of the detected targets.

In other embodiments, the data assignment based on geometry may be omitted, and the data assignment may rely on velocity. In yet other embodiments, steps <NUM> to <NUM> may be omitted, and Kalman filtering may by directly applied to the output of step <NUM>.

During step <NUM>, track filtering is performed, e.g., for tracking a target over time. For example, in some embodiments, the unscented Kalman filter is used to perform track filtering during step <NUM>. For example, in some embodiments, the features (e.g., SIFT, EMD, range, Doppler, angle, deep learning-based parameters, and/or other parameters associated with the track) are, e.g., additional features used to perform data association (which may also be tracked by the Kalman filter). The unscented Kalman filter may also track localization of each track and may rely on the track history of such localization to enhance data association. The Kalman filter may be implemented in any way known in the art. In other embodiments, a extended Kalman filter may be used.

It is understood that although targets may be identified using template matching (during step <NUM>) that may not include spatial and/or movement information (e.g., range, Doppler, angle), such localization information may still be tracked during step <NUM>. Thus, in some embodiments, featured-based template matching (step <NUM>) is an enabler for data association in environments, such as low frame rate, and/or multi-target scenarios, and/or distributed radar implementations in which relying in localization information alone may be difficult.

During step <NUM>, track management tasks, such as generating tracks and killing tracks are performed. For example, during step <NUM>, track initializations, re-initialization, and/or tracks killing may be performed, e.g., based on whether detected targets are no longer in the field-of-view (in scene <NUM>), or re-entered the field of view, for example.

In some embodiments, steps <NUM>, <NUM>, and <NUM> may be implemented in different order. For example, in some embodiments, track initialization (during step <NUM>) may be performed before performing step <NUM>.

Data association <NUM> and template matching <NUM> may also be omitted in other embodiments.

The parameters of unscented Kalman filter <NUM> are set by machine learning logic <NUM> already described referring to <FIG>. This may be based on an output of data association step <NUM> or any other output in the processing pipeline in particular the one shown in <FIG>, for example also based on the digital data xout_dig(n). This in turn means that any of this data may be used as training data (the training process will be described further below).

Next, machine learning logic <NUM> and training thereof will be described in greater detail.

In some embodiments, during training, the policy network, the critic network, and in some embodiments also one or more value networks are trained using the data coming from the processed scene recorded by the radar, e.g. the radar data rd or rd2 mentioned above. Once there is a convergence in the training process (i.e. no improvement over time), the training is concluded. In inference, i.e. in actual use of the device including the machine learning logic after training, both policy and critic network are used, where the policy network chooses hyperparameters for the digital filter, e.g. tracking filter, and the critic network determines out-of-distribution characteristics of the captured scene as will be explained below in more detail.

In some embodiments the machine learning logic may be based on the so called actor-critic approach, where an actor, for example the policy network of <FIG>, and a critic network are used. The policy network is trained based on reward values output by the critic network. Simultaneously, the critic network may be trained. In conventional reinforcement learning, in contrast to supervised learning, no specific annotated training data is necessary, but reward values may be generated based a measurable result, for example. In embodiments discussed herein, in contrast to such "pure" reinforcement learning, training data may be annotated, meaning that for example the number of objects to be tracked in a scene and/or their movement is a priori known for the training data, such that the desired result of the tracking is known.

<FIG> shows a simple scheme of reinforcement learning. A machine learning logic <NUM> in response to a state of the environment performes an action At on the environment <NUM>, which leads to an updated state of the environment St+<NUM> and reward values Rt+<NUM>. The machine learning logic, besides the state St acts on the reward value Rt. In the present example, this means that generally for a given radar data, the machine learning logic sets certain tracking parameters, which lead to tracked data as to tracked data as a result.

In some implementations, at first the policy network may be trained, then the critic network may be trained, and then a joint training may be performed. Conventional reinforcement learning is able to optimize a scene dependent set of tracking parameters in an environment it has been trained on.

The environment, in this case, indicates circumstances under which a scene is taken, for example presence of other objects, electromagnetic disturbers etc. The scene, however, is a term generally designating the number of objects to be tracked and their movement, in a precise time step. Generally, the tracking performance of conventional approaches like püure actor-critic based approaches is very sensitive on the environment.

Therefore, in some embodiments techniques based on so called meta-reinforcement learning techniques are used, which is a technique essentially to generalize learning. The concept is for example described in<NPL>. Meta-reinforcement is visualized in <FIG>. An inner loop <NUM> corresponds to the reinforcement learning scheme of <FIG>, where machine learning logic <NUM>, also referred to as agent, receives training signals. An outer loop <NUM> represents the meta-reinforcement learning. Training is performed based on a distribution of environments, and for example some environments may be used for training the "inner loop", i.e. the reinforcement learning, and other environments may be used for the "outer loop" <NUM>, i.e. the meta-reinforcement learning, which allows a generalization. The meta-reinforcement learning may be performed as off policy meta-reinforcement learning via probabilistic context variables, as described for example in <NPL>.

This approach in turn is based on the so called soft actor critic approach. Soft actor critic can be thought of as being an actor critic method (essentially a reinforcement learning method) that maximizes the long term reward and the long term entropy.

<FIG> and <FIG> are diagrams illustrating a detailed machine learning logic according to an embodiment, in the embodiment of <FIG> and <FIG> including a main network group <NUM>, a context network group <NUM> and a target network group <NUM>. Main network group <NUM> includes a policy network <NUM>, a critic network <NUM> and a value network <NUM>. Context network group <NUM> includes a context network <NUM>. Target network group <NUM> includes a value network <NUM>, which, to distinguish from value network <NUM>, will be referred to as target value network in the following. Networks <NUM> to <NUM> each may be implemented as neural networks including a plurality of layers. Embodiments are not limited to any particular type of layers, and for example fully connected layers, convolutional layers, recurrent layers, etc may be used.

In <FIG>, "a" represents the action by policy network <NUM>, in the present embodiments the setting of parameters of the digital filter. This is in response to a state "s", i.e. the captured radar signals and/or digital data derived therefrom. "s'" is a next state. "r" denotes the reward value for the action "a" generated by the critic network <NUM>, and additionally by value net <NUM> or target value net <NUM>. Together, s, a, r, s' from the context c. Training is performed on a plurality of so called tasks, wherein each task consists of a set of states, actions, a transition function and a reward function. The transition function is different for every task, but unknown.

For updating, a policy loss function <NUM>, a context loss function <NUM>, a value loss function <NUM> and a critic loss function <NUM> may be used. The updating is then performed corresponding to a gradient of these functions. These loss functions in some embodiments may be defined as follows: <MAT> <MAT> <MAT> <MAT> <MAT>.

In the above equations, E is the Expectation operator. Expectation is always over the subscripted variables of E. s~B means state s from a sampled Batch B, a~π is the action from the current policy and z~q is the context from the context network. Π generally represents the policy. KL is the KL-Divergence as distance measure, and "exp" is the exponential function.

The policy _loss Lpolicy is the expected KL Divergence between the policy distribution and the exponential Q-Value which is guaranteed to improve the policy. ZΘ normalizes the distribution and is intractable but does not contribute to gradient computation. z means, there is no gradient computation through this variable. Similar for V. LKL is a constraint loss ensuring that the context variable is not exploding. The use of context variables may be as described in Rakelly et. al, "Efficient Off-Policy Meta-Reinforcement Learning voa Probabilistic Context Variables", arXiv:<NUM>. 08254v1, which proposes a standard normal distribution that would result in r(z) = N(<NUM>,<NUM>), which also has a closed form solution.

The context network <NUM>, based on the context, provides context variables z, z', which are taken into account by value network <NUM> and target value network <NUM>. For this, the context variable is sampled from the context network. Afterwards, the context variable is an additional input to the policy network and the critic network. Value network <NUM> and target value network <NUM> may have the same structure, but target value network <NUM> may be updated less often, to provide stability.

The training, given the plurality of training tasks, including scenes where the tracked objects are known are as follows:
First, the value network <NUM> and the target value network <NUM> with the same network weights. Furthermore, the context variable is initialized for every task as a standard normal distribution.

For each respective task, trajectories executed with the actions "a" of policy network <NUM> are stored in the replay memory D(i) for that specific task. For every training task for updating/evaluating, the following steps are then performed:.

<FIG> then shows the situation for the last replay memory, and function V being output by target value net in <FIG>, corresponding to the estimated soft value function. <FIG> can be seen as representing outer loop <NUM> of <FIG>, while <FIG> can be seen as representing inner loop <NUM>. While <FIG> and <FIG> illustrate the training phase, the critic network and in some embodiments also the context network are also used after training, for the out of distribution detection. This will be explained next.

<FIG> illustrates the out of distribution detection In example of <FIG>, each of critic network <NUM> and a target critic network <NUM>, which may be used for stability purposes during training, similar to target value network <NUM> of <FIG>, have a plurality of heads (as explained with reference to <FIG> for the critic network), leading to a plurality of respective Q-Values, in the example of <FIG> ten Q-Values 1001_1 - 1001_10 for target value network <NUM> and ten Q-Values 1002_1 - 1002_10 for value network <NUM>. Every sampled head learns the same target, but due to the random sampling of replay memories during training explained above, each head sees different data during training, leading to a diversity between the heads. During update and learning, each head at <NUM> is sampled with a probability p; Qi~B(<NUM>, p). Each head is then updated with the TD loss according to block <NUM>. The equation shown in block <NUM> describes the gradient update for the network, in particular critic Network with respect to multiple heads (see e.g. <FIG>). Expectation is computed over the training samples in the batch from the replay memory. First term is the predicted Q-Value given the state and action. Second term is the reward minus the predicted future reward from the target critic network for the next state and the current action.

This diversity between the heads can then be used for out of distribution detection later. For example, a mean value µ and variance σ may be calculated along all heads from the training data, i.e. from the Q-Values resulted in the training phase. A variance scale α may then be defined, for example α=<NUM>, and an out of distribution threshold range cr may be defined as cr=µ ± ασ.

Whenever one of the heads during normal operation (for example heads h1 to hn of <FIG>) outputs a Q-Value outside the range of cr, the measurement is classified as out of distribution. Alternatively, as mentioned above the variance of the Q values output may be calculated and compared to a threshold.

<FIG> shows this concept applied to radar data. Here, range angle images and range Doppler images <NUM> to <NUM> from single and from multiple frames are provided as input, and Q-Value estimation network <NUM> (for example critic network <NUM> of <FIG>, or value network <NUM> of <FIG>) outputs respective Q-Values, for example <NUM> Q-Values, based on which then an out of distribution detection is performed.

In some embodiments, for generating the Q-Values additionally input noise may be taken into account. Such an embodiment is illustrated in <FIG>. A radar frame <NUM> can be characterized by a mean value µ and a variance σ regarding the signal, and a variable ε~N(µ,σ) may be provided which reflects the intensity of the radar signal and the noise. Mean µ and variance σ may be calculated over range angle intensity values of the radar data. This variable is used as input noise <NUM> modifying the Q-Values 1203_1 to 1230_10 output by critic network <NUM>. For example, ε may be added as noise in a layer of the critic network, for example in a second last layer before the output. In this case, for example for radar frames including a high number of people, the diversity of the Q-Values generated by the heads is enhanced, which may lead to an improved out of distribution detection.

To illustrate this concept, <FIG> illustrates a standard deviation of critic heads over a number of people in the scene. In the example shown, for example the networks have been trained for scenes with up to three people. Due to this, the standard deviation for up to three people is low, but increases significantly when more people, i.e. four or five people, to be tracked are in the scene. As four or five people is outside the range the networks were trained with, the detection and tracking is not as reliable, and this may be detected by the out of distribution detection.

Claim 1:
A radar device (<NUM>), comprising:
a radar frontend (<NUM>, <NUM>) configured to send radar signals and to receive reflected radar signals,
processing circuitry (<NUM>, <NUM>) configured to provide digital radar data (rd, rd2) based on the received reflected radar signals,
a digital filter (<NUM>, <NUM>, <NUM>) configured to process the digital radar data (rd, rd2) in order to obtain information (td) about objects which reflected the radar signals, and
a machine learning logic (<NUM>, <NUM>), characterized by comprising:
a policy network (<NUM>, <NUM>) configured to set the parameters (p) of the digital filter based on the digital radar data, and
a reward value generating network (<NUM>, <NUM>, <NUM>) including a plurality of heads (h1 ... hn), each head (h1...hn) configured to provide a respective expected reward value for a setting of parameters (p) by the policy network (<NUM>),
wherein the radar device (<NUM>) is further configured to detect that a scene (<NUM>, <NUM>) captured by the radar device (<NUM>) is not reliably processable based on a distribution of the expected reward values generated by the plurality of heads (h1...hn).