Patent Description:
Various use cases are known that rely on tracking a <NUM>-D position of a target. Example use cases include human-machine interfaces (HMIs): here, the <NUM>-D position of a user-controlled object implementing the target can be tracked. It can be determined whether the target performs a gesture. It could also be determined whether the target actuates an input element of a user interface (UI).

<CIT> discloses a radar system than can recognize gestures. Two-Dimensional gestures usable with touch-sensitive displays can be recognized. Touch-free control of a smart device is provided.

Further related radar systems and technologies are disclosed in <CIT>, <CIT> and <CIT>.

Accordingly, there may be a need for determining a robust and accurate estimate of the position of a movable target.

Various examples of the disclosure generally describe techniques of using a neural network algorithm for processing a radar measurement dataset such as a <NUM>-D point cloud or a <NUM>-D map generated by a radar sensor. The neural network algorithm may provide an output dataset indicative of a position estimate of a target such as, e.g., a hand, a part of a hand - e.g., a finger or a palm - or a handheld object. In some examples, the neural network algorithm may provide an output dataset that classifies a gesture performed by the target.

Next, some examples for possible implementations of the disclosure are provided.

A computer-implemented method according to claim <NUM> includes, in particular, obtaining a radar measurement dataset. The radar measurement dataset is indicative of depth positions of data points of a scene. The scene is observed by a radar unit. The scene includes a target. The target is selected from the group consisting of a hand, a part of a hand, and a handheld object. The method also includes processing the radar measurement dataset using at least one neural network algorithm, to obtain an output data set. The output data set includes one or more position estimates of the target defined with respect to a predefined reference coordinate system. The predefined reference coordinate system is associated with the scene.

A computer program or a computer-program product or a computerreadable storage medium includes program code. The program code can be loaded and executed by at least one processor. Upon loading and executing the program code, the at least one processor performs a method. The method includes obtaining a radar measurement dataset. The radar measurement dataset is indicative of depth positions of data points of a scene. The scene is observed by a radar unit. The scene includes a target. The target is selected from the group consisting of a hand, a part of a hand, and a handheld object. The method also includes processing the radar measurement dataset using at least one neural network algorithm, to obtain an output data set. The output data set includes one or more position estimates of the target defined with respect to a predefined reference coordinate system. The predefined reference coordinate system is associated with the scene.

A computer-implemented method of performing a training of at least one neural network algorithm according to claim <NUM> is provided. The neural network algorithm is for processing a radar measurement dataset to obtain an output dataset. The output dataset includes one or more position estimates of a target. The method includes, in particular, processing multiple training radar measurement datasets that are indicative of depth positions of data points of the scene that is observed by a radar unit. The scene includes the target. The target is selected from the group consisting of a hand, a part of a hand, and a handheld object. The method also includes obtaining ground-truth labels for the multiple training radar measurement datasets. The ground-truth labels each include one or more positions of the target defined with respect to a predefined reference coordinate system. The predefined reference coordinate system is associated with the scene. Also, the method includes performing the training based on the multiple training radar measurement datasets and the ground-truth labels.

A computer program or a computer-program product or a computerreadable storage medium includes program code. The program code can be loaded and executed by at least one processor. Upon loading and executing the program code, the at least one processor performs a method of performing a training of at least one neural network algorithm is provided. The neural network algorithm is for processing a radar measurement dataset to obtain an output dataset. The output dataset includes one or more position estimates of a target. The method includes processing multiple training radar measurement datasets that are indicative of depth positions of data points of the scene that is observed by a radar unit. The scene includes the target. The target is selected from the group consisting of a hand, a part of a hand, and a handheld object. The method also includes obtaining ground-truth labels for the multiple training radar measurement datasets. The ground-truth labels each include one or more positions of the target defined with respect to a predefined reference coordinate system. The predefined reference coordinate system is associated with the scene. Also, the method includes performing the training based on the multiple training radar measurement datasets and the ground-truth labels.

A device includes a processor and a memory. The processor can load program code from the memory. Upon loading the program code, the processor obtains a radar measurement dataset that is indicative of depth positions of data points of a scene observed by a radar unit. The scene includes a target. The target is selected from the group consisting of a hand, a part of a hand, and a handheld object. The processor also processes the radar measurement dataset using at least one neural network algorithm to obtain an output data set. The output data set includes one or more position estimates of the target defined with respect to a predefined reference coordinate system that is associated with the scene.

A device includes means for obtaining a radar measurement dataset. The radar measurement data set is indicative of depth position of data points of a scene observed by a radar unit. The scene includes a target. The target is selected from the group consisting of a hand, a part of a hand, and a handheld object. The device also includes means for processing the radar measurement dataset using at least one neural network algorithm to obtain an output dataset. The output dataset includes one or more position estimates of the target defined with respect to a predefined coordinate system that is associated with the scene.

A device includes a module for obtaining a radar measurement dataset that is indicative of depth positions of data points of a scene. The scene is observed by a radar unit. The scene includes a target. The target is selected from the group consisting of a hand, a part of a hand, and a handheld object. The device also includes a module for processing the radar measurement dataset using at least one neural network algorithm to obtain an output data set. The output dataset includes one or more position estimates of the target defined with respect to a predefined reference coordinate system that is associated with the scene.

A device includes a processor and a memory. The device is for performing a training of at least one neural network algorithm. The at least one neural network algorithm is for processing a radar measurement dataset to obtain an output dataset including one or more position estimates of a target. The processor can load program code from the memory and execute the program code. Upon executing the program code, the processor obtains multiple training radar measurement datasets that are indicative of depth position of data points of a scene observed by a radar unit. The scene includes the target. The target is selected from the group consisting of a hand, a part of a hand, and handheld object. The processor also obtains ground-truth labels for the multiple training radar measurement datasets. The ground-truth labels each include one or more positions of the target defined with respect to a predefined reference coordinate system that is associated with the scene. The processor also performs the training based on the multiple training radar measurement data sets and the ground-truth labels.

A device includes a processor and a memory. The device is for performing a training of at least one neural network algorithm. The at least one neural network algorithm is for processing a radar measurement dataset to obtain an output dataset including one or more position estimates of a target. The device includes means for obtaining multiple training radar measurement datasets that are indicative of depth position of data points of the scene observed by a radar unit. The target is selected from the group consisting of a hand, a part of a hand, and handheld object. The device also includes means for obtaining ground-truth labels for the multiple training radar measurement datasets. The ground-truth labels each include one or more positions of the target defined with respect to a predefined reference coordinate system associated with scene. The device also includes means for performing the training based on the multiple training radar measurement data sets and the ground-truth labels.

A device includes a processor and a memory. The device is for performing a training of at least one neural network algorithm. The at least one neural network algorithm is for processing a radar measurement dataset to obtain an output dataset including one or more position estimates of a target. The device includes a module for obtaining multiple training radar measurement datasets that are indicative of depth position of data points of the scene observed by a radar unit. The target is selected from the group consisting of a hand, a part of a hand, and handheld object. The device also includes a module for obtaining ground-truth labels for the multiple training radar measurement datasets. The ground-truth labels each include one or more positions of the target defined with respect to a predefined reference coordinate system associated with scene. The device also includes a module for performing the training based on the multiple training radar measurement data sets and the ground-truth labels.

In the following, examples of the disclosure will be described in detail with reference to the accompanying drawings. It is to be understood that the following description of examples is not to be taken in a limiting sense. The scope of the disclosure is not intended to be limited by the examples described hereinafter or by the drawings, which are taken to be illustrative only.

Hereinafter, techniques will be described that facilitate determining an estimate of a position (position estimate) of a movable target. According to the various examples described herein, it is possible to determine one or more position estimates of the target. For example, it would be possible to determine multiple position estimates associated with different points in time, i.e., a time series of position estimates. It would also be possible to determine a single position estimate at a certain point in time.

Generally speaking, the position of the target can be tracked over the course of time.

In the various examples described herein, the one or more position estimates can be defined with respect to a predefined reference coordinate system. By determining one or more position estimates of the target that are defined with respect to a predefined reference coordinate system, an increased flexibility with respect to subsequent application-specific postprocessing can be provided. In particular, various techniques are based on the finding that depending on the particular application, different post-processing may be required, e.g., different gestures may need to be classified or different interactions between a user and an HMI may need to be monitored. Accordingly, by determining the one or more position estimates of the target, various such postprocessing applications are facilitated. This is, in particular, true if compared to tracking algorithms which provide, as output data, a classification with respect to application-specific gesture classes.

As a general rule, the one or more position estimates could be defined with respect to a cartesian coordinate system or a polar coordinate system. The reference coordinate system may be defined with respect to one or more reference points, e.g., a sensor - e.g., a radar unit - used to observe a scene.

As a further general rule, a <NUM>-D or <NUM>-D or a <NUM>-D position estimate may be determined. , one, two or three coordinates may be used to specify the position estimate.

As a general rule, it would be possible to use a regression to determine the position estimate in a continuously-defined result space of coordinates. For instance, it would be possible to determine the position of the fingertip of a hand using continuous horizontal and vertical components. In other examples, it would be possible to use a classification to determine the position estimate in a discrete result space of coordinates. In another example, it would be possible to rely on a discrete grid representation - i.e., discretized grid cells -, e.g., associated with input elements a UI, and then determine the position to lie within one of the grid cells.

Various kinds of targets can be tracked using the techniques described herein. According to various examples, it would be possible to determine the one or more position estimates of a target such as a hand or a part of a hand or a handheld object. For example, it would be possible to determine the position estimate of a fingertip or the palm of a hand. By such techniques, it is possible to facilitate user interaction with input elements of a UI of an HMI.

According to various examples, one or more position estimates of the target can be determined based on a radar measurement dataset. A radar unit can be used to acquire raw data and the radar measurement dataset can then be determined based on the raw data.

According to the various examples disclosed herein, a millimeterwave radar unit may be used that operates as a frequency-modulated continuous-wave (FMCW) radar that includes a millimeterwave radar sensor circuit, a transmitter, and a receiver. A millimeter-wave radar unit may transmit and receive signals in the <NUM> to <NUM> range. Alternatively, frequencies outside of this range, such as frequencies between <NUM> and <NUM>, or frequencies between <NUM> and <NUM>, may also be used.

A radar unit can transmit a plurality of radiation pulses, such as chirps, towards a scene. This refers to a pulsed operation. In some embodiments the chirps are linear chirps, i.e., the instantaneous frequency of the chirp varies linearly with time.

A Doppler frequency shift can be used to determine a velocity of the target. Raw data provided by the radar unit can thus indicate depth positions of multiple objects of a scene. It would also be possible that velocities are indicated.

As a general rule, there are various options for processing the raw data to obtain the radar measurement dataset. Examples of such processing of raw data acquired by a radar unit are described in A. Hazra, Deep Learning Applications of Short-Range Radars, ArTech House, <NUM>. An example processing will be described later on in connection with <FIG>.

Depending on the processing of the raw data, different forms of the radar measurement dataset can be obtained. Two possible options are disclosed in TAB. <NUM> below.

The radar measurement dataset may not only encode depth information - as explained in connection with TAB. <NUM> -, but may optionally include additional information, e.g., may be indicative of a velocity of respective object points of the scene included in the radar measurement dataset. Another example for additional information could be reflectivity.

According to various examples, a machine-learning (ML) algorithm is used to obtain an output dataset that includes one or more position estimates of the target defined with respect to a predefined reference coordinate system associated with the scene that is observed by the radar unit. The ML algorithm operates based on the radar measurement dataset. The ML algorithm can thus be referred to as a ML tracking algorithm, because the position of the target is tracked.

As a general rule, output data can be used in various use cases. According to some examples, it is possible that the output data is used to control an HMI. The HMI may react to certain gestures and/or an actuation of an input element of a UI. As a general rule, a gesture can be defined by a certain movement (e.g., having a certain shape or form) and optionally velocities or accelerations performed by the target. The HMI may employ a UI. The UI may include one or more input elements that are defined with respect to the field-of-view (FOV) of the radar unit. For example, it is possible to determine, based on the output data, whether the target addresses a certain input element, e.g., by hovering without movement in an area associated with that input element. It could then be judged whether the certain input element is actuated, e.g., if the target addresses the certain input element for a sufficiently long time duration. A specific type of use case employing such an HMI would be the tracking of a palm or finger or a handheld pointing device (such as a stylus) on and above a touchscreen of an infotainment system or a screen for ticket machines for touchless sensing.

An example implementation of the ML algorithm is a neural network algorithm (hereinafter, simply neural network, NN). An NN generally includes a plurality of nodes that can be arranged in multiple layers. Nodes of a given layer are connected with one or more nodes of a preceding layer at their input, and one or more nodes of a subsequent layer. Skip connections between non-adjacent layers are also possible. Such connections are also referred to as edges. The output of each node can be computed based on the values of each one of the one or more nodes connected to the input. Nonlinear calculations are possible. Different layers can perform different transformations such as, e.g., pooling, max-pooling, weighted or unweighted summing, non-linear activation, convolution, etc..

The calculation performed by the nodes are set by respective weights associated with the nodes. The weights can be determined in a training of the NN. For this, an iterative numerical optimization can be used to set the weights. A loss function can be defined between an output of the NN in its current training state and a ground truth label; the training can then minimize the loss function. Details with respect to the training will be described later on in connection with <FIG>.

More specifically, the training can set the weights so that the NN can extract the position of the target from a noisy representation of the scene; for instance, the target may be a fingertip or a handheld object such as a stylus and the scene may include the entire hand as well as scene clutter and/or measurement noise. This can occur due to inaccuracies of the measurement process, inaccuracies in the calibration, and/or inaccuracies in the processing of raw data. During the training phase, the NN obtains an ability to compensate some inaccuracies of the radar chip calibration and processing of the raw data.

In particular, it has been observed that using an NN for tracking a target can have benefits if compared to conventionally parametrized algorithms. In particular, it would be possible to flexibly adjust the NN to different types of radar units. Typically, different types of radar units can exhibit different measurement noise and/or clutter. The signal characteristics can vary from radar unit to radar unit. Accordingly, by being able to dynamically (re-)train a NN for different types of radio units, and overall increased accuracy can be obtained.

Various options are available for implementing a NN. Some of these options are summarized in TAB.

According to some examples, it would be possible that multiple NNs are combined in a super-network. The super-network can have a recurrent structure. Recurrent NN (RNNs) allow outputs of a previous iteration - e.g., associated with data points of a previous point in time - to be used as inputs in a subsequent iteration while having hidden states. The super-network can include multiple cells. Each cell can receive an input of data points associated with a respective point in time, as well as an output of a further cell, e.g., a cell that receives, as a respective input, data points associated with an earlier and/or later point in time. This may help to enforce inter-frame consistency, e.g., avoid sudden jumps or discontinuous behavior of the position estimates. For example, the radar measurement dataset can include a time series of frames, each frame being indicative of depth positions of the data points of the scene at a respective point in time. For example, each frame can include a <NUM>-D map or a <NUM>-D point cloud, cf. The output dataset can then include a time series of multiple position estimates of the target, the time series of the multiple position estimates being associated with the time series of the frames.

According to various examples described herein, it would be possible that the RNN can include a self-attention module as a part of an encoder-decoder transformer architecture. Such architecture is, in general, known from: <NPL>). The self-attention module can be used to model a temporal interaction between the frames of the time series of frames. Generally, the self-attention module can determine relationships between the data points of the scene at multiple points in time. Also, it would be possible to infer changes between the position estimates of the target at the multiple points in time. Thereby, it would be possible to infer a velocity estimate of the target. The output dataset can be augmented with such velocity estimates that are inferred by the RNN.

There are various implementations of RNNs known in the prior art and it is possible to use such RNNs in the various examples described herein. For instance, the RNN could be selected from the group consisting of: a Long Short Term Memory (LSTM) RNN, a Gated Recurrent Unit (GRU) RNN, and a bidirectional RNN, an autoregressive RNN with a Transformer encoder-decoder.

The LSTM RNN has feedback connections between its cells. For instance, the cell of an LSTM RNN can remember values over certain time intervals. A forget gate can be defined that deletes data. An example of the LSTM RRN is described in: <NPL>.

An example of the GRU RNN is described in <NPL>). The GRU RNN does not require memory cells as the LSTM RNN.

Bidirectional RNNs are described in <NPL>.

<FIG> schematically illustrates aspects with respect to a measurement setup.

A radar unit <NUM> includes two transmitters <NUM>, <NUM> that can transmit electromagnetic waves towards a scene. A receiver <NUM> of the radar unit <NUM> can detect reflected electromagnetic waves backscattered from objects in the scene. A FOV <NUM> of the radar unit <NUM> is illustrated. The FOV <NUM> is dimensioned so that the radar unit <NUM> can observe a scene <NUM>. Next, details with respect to the scene <NUM> will be explained.

Illustrated in <FIG> is a target <NUM>, here a fingertip of a hand of a user. Using a NN, it is possible to obtain an output dataset that includes one or more position estimates of the fingertip target <NUM> defined with respect to a reference coordinate system <NUM> that is associated with the scene <NUM>. In particular, it is possible that the NN is trained to denoise the scene <NUM> by filtering scene clutter <NUM> of the scene <NUM> and/or noise associated with the measurement process of the radar unit <NUM>. , the output dataset can be indicative of the one or more position estimates, yet with suppressed noise and/or scene clutter <NUM>.

In the example of <FIG>, a system <NUM> includes the radar unit <NUM>, as well as a display <NUM>. A UI <NUM> includes multiple input elements <NUM>-<NUM> that are at predefined positions with respect to the display <NUM> or, more generally, with respect to the reference coordinate system <NUM>. For instance, a graphical illustration can be depicted in the sections of the display <NUM> associated with a respective one of the input elements <NUM>-<NUM>. A user may point towards the respective one of the input elements <NUM>-<NUM> to actuate the respective input element <NUM>-<NUM>. Such actuation can then trigger a function that may be associated with a semantic context of the graphical illustration. This is only one example for an implementation of the UI <NUM>.

<FIG> schematically illustrates aspects with respect to a device <NUM> that can be used according to the various examples disclosed herein for tracking a position of a target such as the target <NUM> of <FIG>. The device <NUM> includes a processor <NUM> and a memory <NUM> and an interface <NUM> to communicate with other devices, e.g., a radar unit such as the radar unit <NUM> or a control device of an HMI. The processor <NUM> can load program code from the memory <NUM> and execute the program code. Upon executing the program code, the processor <NUM> can perform techniques as described herein, such as: obtaining a radar measurement dataset that is indicative of depth positions of data points of a scene; processing the radar measurement dataset, e.g., using a NN, to thereby obtain an output dataset that includes one or more positions of the target; training the NN; postprocessing the output dataset, e.g., to classify a movement of the target; controlling an HMI; implementing an HMI; preprocessing the radar measurement dataset, e.g., to discard/remove velocities before providing the radar measurement dataset as an input to the NN; processing raw data of a radar unit to determine a radar measurement dataset; etc..

<FIG> schematically illustrates signal processing according to various examples. <FIG> illustrates a processing flow of multiple logical operations.

At box <NUM>, a radar unit - e.g., the radar unit <NUM> of <FIG>; also cf. <NUM> - is used to acquire raw data <NUM>. The raw data <NUM> is suitable for determining the <NUM>-D position of the target <NUM>. Thus, a Z-position of the target <NUM> can be measured (cf. reference coordinate system <NUM> in <FIG>).

The raw data <NUM> is then pre-processed at box <NUM>, to obtain a radar measurement dataset <NUM>; the radar measurement dataset includes depth data, i.e., is indicative of depth positions of data points of a scene observed by the radar unit <NUM>. Example implementations of the radar measurement dataset <NUM> have been discussed in connection with TAB. <NUM> and include a <NUM>-D point cloud <NUM>-<NUM> and a <NUM>-D map <NUM>-<NUM>.

The radar measurement dataset <NUM> can be subject to measurement noise stemming from imperfections of the radar unit <NUM>, e.g., noise associated with the measurement process, an imperfect calibration, etc. The radar measurement dataset <NUM> can alternatively or additionally include scene clutter, e.g., originating from multi-path reflections, background objects, etc..

Next, tracking of the <NUM>-D position of the target is performed at box <NUM>. An NN - cf. <NUM> - can be employed. Thereby, an output dataset <NUM> is obtained. The output dataset comprises one or more position estimates of the target <NUM>. The measurement noise can be reduced.

A use-case specific application may be executed at box <NUM> based on the output dataset <NUM>. For example, gesture classification could be executed based on the one or more position estimates. For example, a UI including multiple input elements can be used to control an HMI (cf. It is possible that the application provides an output to the user such that a continuous usermachine interface - illustrated in <FIG> by the dashed arrows - is implemented.

<FIG> illustrates details with respect to an example implementation of box <NUM>. The raw data <NUM> corresponds to multiple range Doppler maps obtained from multiple receivers of the radar unit <NUM>. The range-Doppler map/image is generated by performing FFT along fast-time and slow-time of the received intermediate frequency signal. Following which <NUM>-D Moving Target Indicator (2D MTI) is implemented as first order final impulse response (FIR) filter removing the background clutter and reflections from static targets. The OS-CFAR detection algorithm detects the range-Doppler bins that are indicative of a target(s). Once the range-Doppler bins are detected, 2D digital beamforming (e.g., <NUM>-D Capon) is applied to transform the data across virtual channels from rectangular array configuration into <NUM>-D azimuth-elevation image. Then, OS-CFAR is applied to detect corresponding azimuth and elevation pixels which are representative of the same target(s). Using the detected azimuth, elevation values and the corresponding range values, and applying the polar coordinate conversion leads to (x,y,z) point clouds.

Hence, the radar measurement dataset <NUM> is obtained. In the illustrated example a vector is obtained that specifies distance/range, angle and speed of a center of the target, cf. <NUM>, example II. It would also be possible to obtain a <NUM>-D point cloud.

<FIG> is a flowchart of a method according to various examples. For example, the method of <FIG> could be executed by the processor <NUM> upon loading program code from the memory <NUM>. Optional boxes are labeled with dashed lines in <FIG>.

The method of <FIG> illustrates aspects with respect to inference using one or more NNs. In particular, one or more position estimates of a tracked target are inferred. The one or more NNs are pre-trained. For instance, multiple NNs may be included in cells of an RNN super-network.

At <NUM>, a radar measurement dataset is obtained. For example, a radar measurement dataset <NUM> as discussed in connection with <FIG> could be obtained.

Obtaining a radar measurement dataset may include loading the radar measurement dataset from a memory.

Obtaining the radar measurement dataset may include receiving the radar measurement dataset via an interface from a radar unit (cf. <FIG>: interface <NUM>).

Obtaining the radar measurement dataset at box <NUM> may, e.g., include controlling radar acquisition at box <NUM>. For this, control data may be transmitted to a radar unit to trigger a radar measurement (cf. <FIG>: box <NUM>), e.g., via an interface such as the interface <NUM> in <FIG>.

Obtaining the radar measurement dataset may optionally include processing raw data obtained from a radar unit at box <NUM>. An example implementation of such processing has been discussed in connection with <FIG>, and <FIG>: box <NUM>.

Obtaining the radar measurement dataset may include pre-processing the radar measurement dataset at box <NUM>. For example, it would be possible to remove velocities from the radar measurement dataset. Sometimes, it would be possible that the radar measurement dataset is indicative of velocities of data points of the scene. Such velocities could be obtained from Doppler frequency shifts. It would be possible to remove such velocities.

Such techniques focus on utilizing implicit non-radial velocity components instead of explicit radial components and therefore don't rely on Doppler frequency shifts.

For instance, often the radar measurement dataset may only be indicative of a radial velocity, i.e., a component of the velocity towards or away from the transmitter of the radar unit. Non-radial components may not be indicated, since they cannot be observed by the radar measurement process. On the other hand, for some use cases, such non-radial components may be of particular interest. Accordingly, it may be beneficial to altogether remove velocities from the radar measurements that and then rather infer velocities when processing the radar measurement dataset using at least one NN. Then, a higher accuracy can be obtained in the tracking of the position. The tracking can be particularly robust. Training of the at least one NN can be simplified, because the dimensionality of the radar measurement dataset can be reduced.

At box <NUM>, the radar measurement dataset - that may or may not be indicative of velocities of the data points of the scene, as explained above in connection with box <NUM> - is processed using at least one NN to thereby obtain an output dataset (cf. <FIG>: box <NUM>).

In some examples, the output dataset could be indicative of a gesture classification. , one or more identifiers of one or more gestures selected from a predefined plurality of gesture classes could be included in the output dataset.

In other examples, the output dataset may include one or more position estimates of the target. The one or more position estimates are defined with respect to a predefined reference coordinate system associated with the scene. This means that the one or more position estimates may not yet be classified with respect to a predefined gesture classes. This provides for increased flexibility in the post-processing.

It is, in particular, possible that the output dataset is indicative of multiple position estimates, thereby specifying a time dependency of the location of the target. The target may be at rest in which case the multiple position estimates do not deviate significantly; or may be performing a movement.

It would be optionally possible to post-process the output dataset at box <NUM>. Application-specific post-processing is possible (cf. <FIG>: box <NUM>). For instance, it would be possible to post-process the output dataset using a classification to identify a gesture from multiple predefined gesture classes. , it would be possible to post-process the output dataset to classify the one or more position estimates and optionally velocity estimates with respect to predefined gesture classes.

As a general rule, there can be movement gestures or rest gestures. An example of a rest gesture would be a static rest of the target at or above an input element of a UI. Here, the target does not move or only move slightly with respect to the dimensions of the input element. A respective example will be described later on in connection with <FIG>. Examples of movement gestures would be: swipe gesture, e.g., a directional swipe; pinch gesture; double-click gesture; etc. Details with respect to classifying a movement of the target will be described later on in connection with <FIG>.

Examples of NNs have been discussed in connection with <FIG>. An example NN <NUM> is also illustrated in <FIG>. Here, the radar measurement dataset <NUM> is provided to the NN <NUM> which includes multiple layers <NUM>-<NUM>. These layers can perform operations, e.g., to reduce a dimensionality (encoding), to thereby obtain a feature vector of reduced dimensionality that implements the output dataset <NUM>. The feature vector specifies the position estimate <NUM>. An output layer may be a regression layer or a classification layer, e.g., to provide the one or more position estimates using discretized or continuous coordinates defined with respect to the reference coordinate system. The discretized coordinates may be defined with respect to input elements of a UI, such as the input elements <NUM>-<NUM>.

The feature vector indicative of the position estimate <NUM> in the illustrated example has three entries, i.e., encodes a <NUM>-D position estimate, e.g., in a Cartesian coordinate system or another coordinate system and using continuous coordinates. This is only one option. In another option, it would be possible that the feature vector has <NUM>-D <NUM>-D dimensionality, e.g., where discretized coordinates are used that may encode a particular input element is encoded addressed by the target <NUM>.

<FIG> schematically illustrates aspects with respect to a CNN, cf. <NUM>: example I. Here, the radar measurement dataset <NUM> is provided as a <NUM>-D map. A first layer of the CNN is then obtained by performing convolutions between the radar measurement dataset <NUM> and a kernel <NUM>.

<FIG> schematically illustrates the radar measurement dataset <NUM> including a time series of frames <NUM>. Each frame <NUM> is indicative of the depth position of respective data points of the scene at the respective point in time. For example, each frame <NUM> could include a <NUM>-D map <NUM>-<NUM> or a <NUM>-D point cloud <NUM>-<NUM> encoding depth positions of data points of the scene <NUM> at a respective point in time, cf.

Specifically, <FIG> illustrates an acquisition interval <NUM> in time domain over the course of which the respective radar measurements are acquired.

In the illustrated example, a super-network <NUM> has recurrent structure and includes multiple cells <NUM>-<NUM>. Each one of the multiple cells <NUM>-<NUM> can include a respective NN <NUM>-<NUM> - <NUM>-<NUM>, e.g., one of the NNs as discussed in connection with TAB. Each cell receives, as a further input, an output of a preceding cell: this is the recurrent structure. There could also be feedback connection (not shown in <FIG>).

Then, the output dataset <NUM> then includes a time series of multiple position estimates <NUM> of the target <NUM> defined with respect to the respective predefined reference coordinate system <NUM>. The time series of the multiple position estimates <NUM> is associated with a time series of frames <NUM>, i.e., covers the same acquisition interval <NUM>.

According to various examples, the super-network <NUM> can augment the output dataset <NUM> with inferred velocity estimates <NUM> of the target <NUM>. This means that the velocity estimates <NUM> are not the observed Doppler velocities, but rather hidden variables inferred from the radar measurement dataset <NUM>. As illustrated in <FIG>, it is possible to obtain a respective velocity estimate <NUM> for each point in time within the acquisition interval <NUM>. The super-network <NUM> thus augments the output dataset <NUM> accordingly.

As a general rule, it would be possible to obtain a <NUM>-D velocity estimate, e.g., indicative of the magnitude of the velocity (cf. In other examples, a <NUM>-D or <NUM>-D velocity estimate could be obtained that is indicative of the magnitude and direction of the velocity.

<FIG> schematically illustrates aspects with respect to a movement gesture <NUM> performed by the fingertip target <NUM>. In the illustrated example, the fingertip target <NUM> performs a left-to-right swipe gesture so that over the course of time multiple positions <NUM>-<NUM> are adhered. For example, it would be possible to obtain the output dataset <NUM> including a time series of multiple position estimates <NUM> corresponding to the positions <NUM>-<NUM>. The gesture <NUM> is also characterized by certain velocities <NUM>-<NUM>. It would be possible that the output dataset <NUM> is augmented with velocity estimates <NUM> being indicative of the velocities <NUM>-<NUM> of the fingertip target <NUM>.

In the illustrated example (but also in other examples), it would be possible that the NN used to determine multiple position estimates <NUM> includes a regression layer to provide the multiple position estimates <NUM> of the fingertip target <NUM> using continuous coordinates of the reference coordinate system <NUM>. Such use of continuous coordinates of the reference coordinate system <NUM> can have the advantage that the postprocessing to classify the gesture can work accurately and comparably robust, specifically for movement gestures as illustrated in <FIG>.

<FIG> schematically illustrates a static rest gesture <NUM> of the fingertip target <NUM>. The fingertip target <NUM> rests in an area associated with the input element <NUM> of the user interface <NUM>. A respective position <NUM>, as well as motion blur <NUM> - e.g., due to measurement noise - is illustrated in <FIG>.

In the illustrated example (but also in other examples), it would be possible that one or more position estimates <NUM> of the target <NUM> are specified by the output dataset <NUM> using discretized coordinates of the reference coordinate system <NUM>. In particular, it would be possible that the discretized coordinates are associated with the input elements <NUM>-<NUM> of the UI <NUM>. These input elements have predefined locations in the reference coordinate system <NUM>. Accordingly, it would be possible to include an indicator that is indicative of a particular input element <NUM>-<NUM>. To this end, the NN used to determine the one or more position estimates may include a classification layer to provide the discretized coordinates. This can help to quickly and reliably infer the position estimate at an accuracy tailored the application defined by the UI <NUM>. , noise can be suppressed. Sudden jumps between adjacent input elements <NUM>-<NUM> could be suppressed by appropriately setting properties of an RNN.

<FIG> is a flowchart of a method according to various examples. <FIG> illustrates that inference using at least one NN to determine one or more position estimates of a target at box <NUM> is preceded by a training at box <NUM>. In the training of box <NUM>, one or more parameters of the at least one NN can be set. While above various examples have been described with respect to the inference at box <NUM> (cf. <FIG>), next, various aspects will be described with respect to the training at box <NUM>.

<FIG> is a flowchart of a method according to various examples. For example, the method of <FIG> could be executed by the processor <NUM> upon loading program code from the memory <NUM>.

The method of <FIG> illustrates aspects with respect to training of a NN. The NN can be trained to determine one or more position estimates of a target. As such, the method of <FIG> can implement box <NUM> of the method of <FIG>.

For example, a training in accordance with <FIG> could be executed for each type of radar unit. This is because different types of radar units can exhibit different measurement noise or measurement-process related inaccuracies. By re-training the NN for different types of radar units, a tailored tracking algorithm can be provided having increased accuracy.

At box <NUM>, multiple training radar measurement datasets that are indicative of depth positions of data points of a scene observed by a radar unit are obtained. The training radar measurement dataset may be obtained from actual measurements or could also be synthesized.

The scene includes a target that is, according to various examples, selected from the group consisting of a hand, a part of a hand, and a handheld object.

At box <NUM>, ground-truth labels are obtained for the multiple training radar measurement data sets.

These ground-truth labels can each include one or more positions defined with respect to a predefined reference coordinate system.

The ground-truth labels could be obtained from manual annotation. It would also be possible to use another tracking algorithm, e.g., an interacting multi-model tracking algorithm considering, e.g., an unscented Kalman filter and a coordinated turn model; then, for low uncertainties of such algorithm, it would be possible to derive a respective ground-truth label. It would also be possible to obtain ground-truth labels from another sensor, e.g. RGB and/or depth camera, multi-view camera setup with or without markers.

Then, it is possible to perform - at box <NUM> - the training based on the multiple training radar measurement data sets, as well as the ground-truth labels.

For instance, to perform the training, it would be possible to determine a difference between position estimates determined by the NN in its current training state and the positions indicated by the ground-truth labels and, based on this difference, determine a respective loss value of a predefined loss function. Then, in an iterative optimization process it is possible to reduce the loss value. Respective techniques of training are, in general, known to the skilled person.

According to various examples, it would be possible that the scene includes classes associated with background objects. In other words, the training radar measurement data sets may include such clutter. Then, it would be possible to train the NN to perform denoising by suppressing the clutter, by using the ground-truth labels that are indicative of the positions of the target, but not of the clutter.

According to various examples, it would be possible that at least some of the multiple training radar measurement datasets comprise a time series of frames that are associated with a time duration during which the target performs a static rest gesture (cf. <FIG> where the static rest gesture <NUM> has been explained; cf. <FIG> where a respective time series of frames <NUM> has been discussed). Each frame of the time series of frames can be indicative of the depth position data points of the scene at the respective point in time during the time period. Then, the depth positions of the respective data points of the scene associated with the target at the points in time can exhibit a motion blur (cf. <FIG>, motion blur <NUM>). The motion blur can be associated with movement of the target and and/or the scene clutter <NUM>. Thus, by considering the ground-truth labels that remain unaffected of the motion blur, it is possible to train the NN to suppress motion blur. This can be, in particular, helpful for scenarios that aim at detecting actuation of a given input element <NUM>-<NUM>. Here, sudden jumps due to motion blur <NUM> should be avoided.

More generally, it would be possible that the one or more position estimates of the target are specified using discretized coordinates of the reference coordinate system. Then, the ground-truth labels for the at least some of the multiple training radar measurement data sets can statically specify a given one of the discretized coordinates for the respective points in time.

<FIG> schematically illustrates a device <NUM> according to various examples. The device <NUM> can be configured to perform techniques as described herein, e.g., with respect to processing a radar measurement dataset. The device <NUM> includes a module <NUM> for obtaining a radar measurement dataset. The module <NUM> could, e.g., implement box <NUM> and/or box <NUM> of <FIG>. The module <NUM> could implement box <NUM> of the method of <FIG>.

The device <NUM> also includes a module <NUM> for processing the radar measurement dataset that is obtained at box <NUM>. For instance, the module <NUM> could implement box <NUM> and/or box <NUM> of <FIG>. Box <NUM> could implement box <NUM> of the method of <FIG> and optionally box <NUM>.

Summarizing, radar measurement datasets are processed using a NN. The output of NN is a predicted position of a fingertip (or hand, palm) in a predefined reference coordinate system, e.g., relative to a screen or generally input elements of a UI. It would be possible to postprocess such position estimates to provide a classification of a gesture performed by the target. In other examples, it would also be possible to directly perform a gesture classification.

During the training, the NN obtains an ability to compensate, at least to some degree, for inaccuracies of the radar chip calibration and pre-processing. The NN can be quickly retrained to recognize new gesture types without method change.

Although the invention has been shown and described with respect to certain preferred embodiments, equivalents and modifications will occur to others skilled in the art upon the reading and understanding of the specification. The present invention includes all such equivalents and modifications and is limited only by the scope of the appended claims.

Claim 1:
A computer-implemented method, comprising:
- obtaining (<NUM>) a radar measurement dataset (<NUM>) indicative of depth positions of data points of a scene (<NUM>) observed by a radar unit (<NUM>), the scene (<NUM>) comprising a target (<NUM>), the target (<NUM>) being selected from the group consisting of a hand, a part of a hand, and a handheld object, and
- processing (<NUM>) the radar measurement dataset (<NUM>) using at least one neural network algorithm (<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>) to obtain an output dataset (<NUM>), the output dataset (<NUM>) comprising one or more position estimates (<NUM>) of the target (<NUM>) defined with respect to a predefined reference coordinate system (<NUM>) associated with the scene (<NUM>),
wherein the radar measurement dataset (<NUM>) comprises a time series of frames (<NUM>), each frame of the time series of frames (<NUM>) being indicative of the depth positions of the respective data points of the scene (<NUM>) at a respective point in time,
wherein the at least one neural network algorithm (<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>) comprises multiple neural network algorithms included in multiple cells (<NUM>, <NUM>, <NUM>) of a super-network (<NUM>) with recurrent structure,
the method being characterised in that the output dataset (<NUM>) comprises a time series of multiple position estimates (<NUM>) of the target (<NUM>) defined with respect to the predefined reference coordinate system (<NUM>), the time series of the multiple position estimates (<NUM>) being associated with the time series of frames (<NUM>),
wherein the super-network (<NUM>) augments the output dataset (<NUM>) with inferred velocity estimates (<NUM>) of the target (<NUM>).