Patent ID: 12216229

Corresponding numerals and symbols in different figures generally refer to corresponding parts unless otherwise indicated. The figures are drawn to clearly illustrate the relevant aspects of the preferred embodiments and are not necessarily drawn to scale.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The making and using of the embodiments disclosed are discussed in detail below. It should be appreciated, however, that the present invention provides many applicable inventive concepts that can be embodied in a wide variety of specific contexts. The specific embodiments discussed are merely illustrative of specific ways to make and use the invention, and do not limit the scope of the invention.

The description below illustrates the various specific details to provide an in-depth understanding of several example embodiments according to the description. The embodiments may be obtained without one or more of the specific details, or with other methods, components, materials and the like. In other cases, known structures, materials or operations are not shown or described in detail so as not to obscure the different aspects of the embodiments. References to “an embodiment” in this description indicate that a particular configuration, structure or feature described in relation to the embodiment is included in at least one embodiment. Consequently, phrases such as “in one embodiment” that may appear at different points of the present description do not necessarily refer exactly to the same embodiment. Furthermore, specific formations, structures or features may be combined in any appropriate manner in one or more embodiments.

Embodiments of the present invention will be described in a specific context, a deep CNN (DCNN) for millimeter-wave (mmWave) radar-based (human) target classification or radar-based (human) target localization that uses a parametric two-dimensional (2D) or 3D CNN layer for receiving raw digital data from an analog-to-digital converter (ADC) of the millimeter-wave radar. In some embodiments, the DCNN uses an L-dimensional CNN layer for receiving the raw digital data, where L is a positive integer greater than or equal to 2. Some embodiments may generate other information about the target in addition, or instead of target classification and/or target localization. Some embodiments may be implemented in radars operating in regimes different than millimeter-wave and/or for targets other than human targets, such as animals or autonomous machines, for example.

In an embodiment of the present invention, a DCNN receives raw digital data from an ADC of a millimeter-wave radar, and processes the raw digital data to extract features for classification (e.g., of human activities) directly from the raw digital data without using conventional preprocessing methods (such as background mean subtraction, range discrete Fourier transform, and/or Doppler fast Fourier transform). In some embodiments, the initial (first) layer of the DCNN (which receives the raw digital data from the ADC) is implemented as a constrained 2D convolutional layer. In some embodiments, the constrained 2D convolutional layer uses 2D sinc filter kernels. In other embodiments, the constrained 2D convolutional layer uses 2D Morlet wavelet filter kernels. Other filter kernels, such as filter kernels based on Fractional Fourier Transform, and Discrete Cosine Transform, may also be used.

People sensing and activity classification have increasing application potential in various areas, such as physical security, defense, and surveillance. In industrial and consumer space, for example, human activity recognition finds applications in smart homes, human-machine interfaces and elderly fall-motion monitoring systems. For example, knowledge of the performed activity in a room can enable smart control of the energy consumption, such as HVAC and lighting. Furthermore, knowledge of the performed human activity facilitates ubiquitous smart home solution by, e.g., discerning the user's intent.

Human activity recognition also enables sensing and recognition of elderly fall-motion. Elderly falls are a leading cause of death in elderly population. In some cases, an elderly fall may lead to major restrictions to the overall mobility of the individual if medical assistance is not provided immediately.

Some conventional human activity recognition systems are based on cameras and computer vision approaches. These systems are generally accurate and relatively easy to implement. However, camera systems may suffer from lack of privacy and may be sensitive to illumination conditions, which may render some camera systems unsuitable for some applications.

Radars may also be used to effectively sense human activities. Radars may offer privacy preserving and illumination-invariance properties, and are capable of being aesthetically concealed in the operating environment.

FIG.1shows a schematic diagram of exemplary radar system100. Radar system100includes millimeter-wave radar sensor102, processor120, and artificial intelligence (AI) accelerator122.

During normal operation, millimeter-wave radar sensor102operates as a frequency-modulated continuous-wave (FMCW) radar sensor and transmits a plurality of radar signals106, such as chirps, towards scene130using transmitter (TX) antenna114. The radar signals106are generated using RF and analog circuits104. The radar signals106may be in the 20 GHz to 122 GHz range.

The objects in scene130may include idle humans, such as lying human134and standing human136, and moving humans, such as walking human132. The objects in scene108may also include static objects (not shown), such as furniture, walls, and periodic movement equipment, such as fans. Other objects may also be present in scene120.

The transmitted radar signals106are reflected by objects in scene120. The reflected radar signals108, which are also referred to as the echo signal, are received by receiver (RX) antenna116. RF and analog circuits104processes the received reflected radar signals108using, e.g., band-pass filters (BPFs), low-pass filters (LPFs), mixers, low-noise amplifier (LNA), and/or intermediate frequency (IF) amplifiers in ways known in the art to generate an analog signal xout(t).

The analog signal xout(t) is converted to raw digital data xout_dig(n) using ADC112. The raw digital data xout_dig(n) is pre-processed by processor120and then processed by AI accelerator122to classify the activity of a human in scene130.

Controller110controls one or more circuits of millimeter-wave radar sensor102, such as RF and analog circuit104and/or ADC112.

Processor104may be implemented in any way known in the art, such as a general purpose processor, controller or digital signal processor (DSP) that includes, for example, combinatorial circuits coupled to a memory.

AI accelerator122is designed to accelerate artificial intelligence applications, such as artificial neural networks and machine learning. AI accelerator122may be implemented in any way known in the art.

FIG.2Ashows a flow chart of method200for pre-processing and processing the raw digital data xout_dig(n). Method200includes a preprocessing step205, which includes steps202and204, a feature extraction step206, and a classification step208. Steps202,204, and206are performed by processor120. Step208is performed by AI accelerator122.

During step202, a 1D moving target indication (MTI) filter is applied to the raw digital data xout_dig(n) to remove the response from static targets (such as, e.g., chairs, tables and walls) and also of the transmitter-receiver leakage, which affects the first few range bins. The MTI filter may be performed by subtracting the mean along the fast-time (intra-chirp time) to remove the transmitter-receiver leakage that perturbs the first few range bins, followed by subtracting the mean along the slow-time (inter-chirp time) to remove the reflections from static objects (or zero-Doppler targets).

During step204, 2D windowing is applied to the filtered digital data xfiltered_dig(n) along the fast-time as well slow-time dimensions, followed by a 2D fast Fourier transform (FFT) to generate a 2D matrix representing the received energy spectrum over range and velocity, also known as range-Doppler image (RDI).

During method206, feature image extraction is performed on the range-Doppler image to generate an RDI video. The RDI video can be expressed as

vRDI(p,l,k)=❘"\[LeftBracketingBar]"∑m=1Ust⁢∑n=1Uft⁢w⁡(m,n)⁢s⁡(m,n,k)⁢e-j⁢2⁢π⁡(m.pUst+n.lUft)❘"\[RightBracketingBar]"(1)
where Ustis the FFT size along slow-time, Uftis the FFT size along fast-time, w(m,n) is the 2D weighting function along the fast-time and slow-time, s(m,n,k) is the ADC data (xout_dig) on the kth frame, where the indexes n and m sweep along the fast-time and slow-time axes, respectively. The l and p indexes sweep along the range and Doppler axes, respectively.

During step208, the RDI video is fed to a DCNN or LSTM, which classifies the human activity of the detected human target based on the RDI video as well as on the training of the DCNN or LSTM.

FIG.2Bshows a flow chart of method220for pre-processing and processing the raw digital data xout_dig(n). Method220includes a preprocessing step205, which includes steps202and204, a feature extraction step222, and a classification step224. Steps202,204, and222are performed by processor120. Step224is performed by AI accelerator122. Steps202and204of method220are performed in a similar manner as in method200.

During step222, feature image extraction is performed on the range-Doppler image by marginalizing over range to generate a Doppler spectrum. Each generated Doppler spectrum includes information about the macro-Doppler component as well as the micro-Doppler component due to hand and leg movements of the detected human target while performing an activity.

The Doppler spectrum from consecutive frames is stacked one after another to generate a 2D image. The stacked Doppler spectrum across consecutive frames is referred to as Doppler spectrogram, and includes information about the instantaneous Doppler spectral content and the variation of the Doppler spectral content over time.

The Doppler spectra of the slow-time data from the kth radar frame on the selected L range bins can be expressed
S(p,k)=Σl=1LvRDI(p,l,k)  (2)
where VRDImay be given by Equation 1.

During step224, the Doppler spectrogram is fed to a DCNN or long short-term memory (LSTM) neural network, which classifies the human activity of the detected human target based on the spectrogram as well as on the training of the DCNN or LSTM.

In an embodiment of the present invention, the preprocessing step (e.g.,205) and the feature extraction step (e.g., steps206or222) are omitted. Instead, a neural network is used to generate, e.g., human activity classification, directly from the raw digital data from the ADC. By directly operating on the raw digital data from the ADC using a DCNN, some embodiments advantageously reduce computational complexity as well as eliminating the need for a DSP for preprocessing.

FIG.3shows a schematic diagram of radar system300, according to an embodiment of the present invention. Radar system300includes millimeter-wave radar sensor102, and artificial intelligence (AI) accelerator322.

As shown,FIG.3illustrates a possible implementation of millimeter-wave radar sensor102, according to an embodiment of the present invention. Other implementations are also possible.

As shown inFIG.3, in some embodiments, millimeter-wave radar sensor102includes reference oscillator302, phased-locked-loop (PLL)304, voltage controlled oscillator (VCO)306, frequency divider308, amplifier310, mixer316, low-pass filter (LPF)318, and ADC112.

During normal operation, VCO306generates a linear frequency chirp (e.g., from 57 GHz to 64 GHz), which is transmitted by transmitting antenna114. The VCO is controlled by PLL304, which receives a reference clock signal (e.g., 80 MHz) from reference oscillator302. PLL304is controlled by a loop that includes frequency divider308and amplifier310.

The linear chirp transmitted by transmitting antenna114is reflected by objects in scene130and received by receiving antenna116. The echo received by transmitting antenna116is mixed with a replica of the signal transmitted by transmitting antenna114using mixer316to reduce an intermediate frequency (IF) signal xIF(t) (also known as the beat signal). In some embodiments, the beat signal xIF(t) has a bandwidth between 10 kHz and 1 MHz. A beat signal xIF(t) with a bandwidth lower than 10 kHz or higher than 1 MHz is also possible.

The beat signal xIF(t) is filtered with low-pass filter (LPF)318and then sampled by ADC112. ADC112is advantageously capable of sampling the filtered beat signal xout(t) with a sampling frequency that is much smaller than the frequency of the signal received by receiving antenna116. Using FMCW radars, therefore, advantageously allows for a compact and low cost implementation of ADC112, in some embodiments.

The raw digital data xout_dig(n), which in some embodiments is the digitized version of the filtered beat signal xout(t), is (e.g., temporarily) stored (e.g., in matrices of Nc×Ns, where Ncis the number of chirps considered in a frame and Nsis the number of transmit samples per chirp) for further processing.

In some embodiments, ADC112is a 12-bit ADC. ADCs with higher resolution, such as 14-bits or higher, or with lower resolution, such as 10-bits, or lower, may also be used.

As shown inFIG.3, in some embodiments, AI accelerator322is used to process the raw digital data xout_dig(n) from ADC112to classify the activity of a target in scene130. AI accelerator322may be implemented in any way known in the art.

Although, as shown inFIG.3, some embodiments use AI accelerator322to implement the neural network to process the raw digital data xout_dig(n) to classify the activity of a target in scene130, other hardware implementations, different from an AI accelerator, or in addition to an AI accelerator may also be used. For example, some embodiments may implement the neural network used for classifying the target (e.g., such as DCNN500,800or900, as described in more detailed below) using a general purpose processor, controller or digital signal processor (DSP) that includes, for example, combinatorial circuits coupled to a memory. In some embodiments, the neural network may be implemented with an ARM or x86 architecture, for example. In some embodiments, the neural network may be implemented using a custom application specific integrated circuit (ASIC) and/or using a combination of hardware accelerator and software running on a DSP or general purpose micro-controller. Other implementations are also possible.

FIG.4shows a sequence of chirps106transmitted by TX antenna114, according to an embodiment of the present invention. As shown byFIG.4, chirps106are organized in a plurality of frames and may be implemented as up-chirps. Some embodiments may use down-chirps or a combination of up-chirps and down-chirps.

The time between chirps of a frame is generally referred to as pulse repetition time (PRT). In some embodiments, the PRT is 5 ms. A different PRT may also be used, such as less than 5 ms, such as 4 ms, 2 ms, or less, or more than 5 ms, such as 6 ms, or more.

Frames of chirps106may include a plurality of chirps. For example, in some embodiments, each frame of chirps includes 16 chirps. Some embodiments may include more than 16 chirps per frame, such as 20 chirps, 32 chirps, or more, or less than 16 chirps per frame, such as 10 chirps, 8 chirps, or less. In some embodiments, each frame of chirps includes only a single chirp.

Frames are repeated every FT time. In some embodiments, FT time is 50 ms. A different FT time may also be used, such as more than 50 ms, such as 60 ms, 100 ms, 200 ms, or more, or less than 50 ms, such as 45 ms, 40 ms, or less.

In some embodiments, the FT time is selected such that the time between the beginning of the last chirp of frame n and the beginning of the first chirp of frame n+1 is equal to PRT. Other embodiments may use or result in a different timing.

Some activities can be distinguished by analyzing their unique range-velocity profiles (e.g., as shown inFIGS.2A and2B). Some activities may have very different range-velocity profiles, such as walking and standing idle. However, some activities may exhibit only slight differences in their range-velocity profiles. For example, working on a laptop and sitting idle on a chair may only differ by the slight movement of the hands exhibited by the human working on the laptop. Thus, a higher resolution on specific frequency bands may be required in order to accurately distinguish these actions. However, when using preprocessing steps, such as steps205,206and222, the whole observable range-velocity space is discretized in equal bins.

In an embodiment of the present invention, a first 2D convolutional layer of a neural network uses a plurality of time-domain band-pass filters (e.g., using 2D sinc filter kernels) that are trained so that their respective cutoff frequencies are adjusted according to the needs of a particular application. Thus, in some embodiments, replacing the preprocessing steps (e.g., such as steps205,206and222) by a DCNN that uses a first 2D convolutional layer that is constrained to optimize cutoff frequencies of a plurality of time-domain band-pass filters advantageously results in improved accuracy during activity recognition when compared with implementations using explicit preprocessing steps, such as steps205,206and222. By constraining the first 2D convolutional layer to a predefined shape (such as using 2D sinc filter kernels or other types of predefined kernels), some embodiments advantageously achieve faster training convergence times and help mitigate or avoid the problem of getting stuck in local minima.

FIG.5shows a block diagram of DCNN500for activity classification, according to an embodiment of the present invention. DCNN500includes constrained 2D convolutional layer502and a plurality of additional layers504. Additional layers504may include, for example, one or more convolutional layers (including complex and non-complex convolutional layers), fully-connected layers, recurrent layers, pooling layers, and/or dense layers. In some embodiments, additional layers504may be implemented using known neural network architectures, such as U-Net or atrous convolutional layers. DCNN500may be implemented, e.g., in AI accelerator322.

As shown inFIG.5, 2D convolutional layer502is implemented with a plurality of 2D sinc filters that are constrained such that only cutoff frequencies and/or bandwidth of each 2D sinc filter are trainable (as opposed to an unconstrained 2D convolutional layer, in which all parameters of each filter are trainable, where each filter is not constrained to be of a specific type, such as of the 2D sinc filter type). For example, in some embodiments, each 2D sinc filter of 2D convolutional layer502may be given by

sinc2⁢D(n,m;flst,bst,flft,bft)=w⁡(n,m)⁢hN,fsst(n;flst,bst)⁢hM,fsft(m;flft,bft)(3)
where

hN,fsst(n;flst,bst)
is a slow-time 1D sinc filter, N is a length of the slow-time 1D sinc filter, f1stis a lower cutoff frequency of the slow-time 1D sinc filter, bstis a bandwidth of the slow-time 1D sinc filter, n is a slow-time filter parameter index (n is an integer between 0 and N, inclusive),

hM,fsft(m;flft,bft)
is a fast-time 1D sinc filter, M is a length of the fast-time 1D sinc filter, f1ftis a lower cutoff frequency of the fast-time 1D sinc filter, bftis a bandwidth of the fast-time 1D sinc filter, m is a fast-time filter parameter index (m is an integer between 0 and M, inclusive), and w(n,m) is a 2D cosine weighting function. The 2D cosine weighting function may be given by

w⁡(n,m)=14⁢(1+cos(2⁢π⁢n-⌊N2⌋N))*(1+cos(2⁢π⁢m-⌊M2⌋M))(4)
and the slow-time 1D sinc filter and the fast-time 1D sinc filter may be given by

hK,fs(k,fl,b)=2⁢(fl+b)⁢sinc(2⁢(fl+b)·k-⌊K2⌋fs)-2⁢fl·sinc(2⁢fl·k-⌊K2⌋fs)(5)
where K is the length of a 1D sinc filter, k is an integer between 0 and K, inclusive, fsis a sampling frequency of the data to be filtered (e.g., xout_dig(n)), flis a lower cutoff frequency, b is a bandwidth of the 1D sinc filter, and k is a filter parameter index, where

sinc⁡(x)=sin⁡(π⁢x)π⁢x.

In the 2D sinc filters sinc2Dof 2D convolutional layer502, as defined by Equation 3, the trainable parameters (also referred to as the hyperparameters), are the lower cutoff frequencies (f1stand f1ft) and the bandwidths (bstand bft) of the slow-time and fast-time 1D sinc filters, respectively. It is understood that, e.g., the bandwidths (bstand bft) or lower cutoff frequencies hyperparameters may be replaced by, e.g., the higher cutoff frequencies of the slow-time and fast-time 1D sinc filters, respectively, or by a center frequency, without affecting performance.

During training of DCNN500, the constrained 2D convolutional layer502is initialized according to the definition 2D sinc filters and only the hyperparameters are allowed to be learned. As a result, the trained filters of the constrained 2D convolutional layer502are 2D bandpass filters (e.g., with rectangular shape in the frequency domain) that have their respective cutoff frequencies optimized based on the training data.

FIG.6shows an exemplary 2D sinc filter sinc2D, as used in constrained 2D convolutional layer502, in time and frequency domains, according to an embodiment of the present invention. As shown inFIG.6, the 2D sinc filter sinc2Dis a 2D band-pass filter that exhibits clear cutoff frequencies in the frequency domain, as illustrated by the rectangular shape in plot620. As shown inFIG.6, the 2D sinc filter sinc2Dis capable of extracting joint range and velocity features of data to be filtered (e.g., xout_dig(n)).

In some embodiments, the 2D sinc filters used in constrained 2D convolutional layer502include a rotational parameter for rotating the 2D sinc filters with respect to the pseudo range domain and pseudo Doppler domain. In such embodiments, each 2D sinc filter of 2D convolutional layer502may be given by

ϕsinc(n,m;flst,bst,flft,bft)=4⁢w⁡(n,m)(6)((flst+bst)⁢(flft+bft)⁢sinc⁡(2⁢(flst+bst)⁢tst′,2⁢(flft+bft)⁢tft′)-(flst+bst)⁢flft⁢sinc⁡(2⁢(flst+bst)⁢tst′,2⁢flft⁢tft′)-flst(flft+bft)⁢sinc⁡(2⁢flst⁢tst′,2⁢(flft+bft)⁢tft′)+flst⁢flft⁢sinc⁡(2⁢flst⁢tst′,2⁢tft′))⁢wheresinc⁡(x,y)=sin⁡(π⁢x)⁢sin⁡(π⁢y)π2⁢xy(7)tst′=n-⌊N2⌋f0st⁢cos⁡(α)-m-⌊M2⌋f0st⁢sin⁡(α)(8)tft′=n-⌊N2⌋f0st⁢sin⁡(α)+m-⌊M2⌋f0ft⁢cos⁡(α)(9)
where α is a rotation angle, N is a length of the 2D sinc filter in slow-time, f1stis a lower cutoff frequency of the 2D sinc filter in slow-time, bstis a bandwidth of the 2D sinc filter in slow-time, n is a slow-time filter parameter index (n is an integer between 0 and N, inclusive), M is a length of the 2D sinc filter in fast-time, f1ftis a lower cutoff frequency of the 2D sinc filter in fast-time, bftis a bandwidth of the 2D sinc filter in fast-time, m is a fast-time filter parameter index (m is an integer between 0 and M, inclusive), and w(n,m) is a 2D cosine weighting function (e.g., as given in Equation 4).

When the rotation angle α is equal to 0, Equation 6 can be expressed as Equation 3.FIG.7illustrate the impact of the angle of rotation α with respect to the pseudo range domain and pseudo Doppler domain, according to an embodiment of the present invention. Pseudo range and Doppler domain are used here for representation of the feature maps that are obtained after applying, e.g., 2D sinc, convolutions.

As shown inFIG.7, an exemplary 2D sinc filter704is rotated with respect to the pseudo range and pseudo Doppler domains by rotation angle α. An exemplary 2D sinc filter702having the rotation angle α equal to 0, can be expressed using Equation 4.

In some embodiments, DCNN500is implemented with a categorical cross-entropy as a loss function. Other loss functions, such as mean square error may also be used.

In some embodiments, convolutional layer502and any of the additional layers504may use a rectified linear unit (ReLU) as activation function. Other activation functions, such as Sigmoid, and leaky ReLU, may also be used.

FIG.8shows a block diagram of DCNN800, according to an embodiment of the present invention. DCNN800is implemented during training as DCNN820, and is implemented during normal operation (after training) as DCNN802.

As shown, DCNN800includes constrained 2D convolutional layer502and additional layers804, where additional layers804represent a possible implementation of additional layers504.

As shown inFIG.8, additional layers804includes, maxpool layers806and810, unconstrained 2D convolutional layer808, dense layer812, and softmax classifier layer814. During training, DCNN800additionally includes dropout layers822,824, and826after each convolutional and dense layer.

During training, DCNN800receives batches (e.g., of 128 samples each) of training data and generates, with softmax classifier layer814, an M-element vector that corresponds to the classification of the respective data, where M is equal to or greater than 2. For example, in some embodiments, M is equal to 6, corresponding to activities classes: “empty room,” “walking,” “idle,” “arm movement,” “waving,” and “working,” for example. The output vector may be of the form [“empty room” “walking” “idle” “arm movement” “waving” “working”], for example. The training data includes a dataset that includes recordings of the five different human activities (“walking,” “idle,” “arm movement,” “waving,” and “working,”) plus recordings of the empty room, where each recording of the dataset is pre-labeled with the corresponding class (e.g., “empty room,” “walking,” “idle,” “arm movement,” “waving,” and “working”).

In some embodiments, the output vector includes confidence values (i.e., the probability that a particular label is correct). In such embodiments, an output vector [0.01 0.75 0.1 0.5 0.4 0.5] may be interpreted as the respective data having 1% probability of corresponding to an “empty room” classification, a 75% probability of corresponding to a “walking” classification, a 10% probability of corresponding to an “idle” classification, a 5% probability of corresponding to an “arm movement” classification, a 4% probability of corresponding to an “waving” classification, and a 5% probability of corresponding to an “working” classification. In such scenario, the respective data may be assigned the classification with highest confidence (in this non-limiting example, classified as “walking”).

In some embodiments, during training, constrained 2D convolutional layer502is initialized to cover the entire range-Doppler space. For example, in some embodiments, constrained 2D convolutional layer502may be initialized by directly defining 2D sinc filters. In other embodiments, constrained 2D convolutional layer502may be initialized by generating the 2D sinc filters using 1D sinc filters. Other initialization schemes are also possible.

For example, in some embodiments, during training, constrained 2D convolutional layer502is initialized with 2D sinc filters. In some embodiments, Nst1D sinc filters are initialized to equally divide the slow-time frequency into bands of size

Bst=fs,st2⁢Nst,
and Nft1D sinc filters are initialized to equally divide the fast-time frequency into bands of size

Bft=fs,ft2⁢Nft
where fs,stand fs,ftare the sampling frequency of the slow-time data and fast-time data, respectively, and Nstand Nftare the number of filters in the slow-time and fast-time directions, respectively. The initial set of Nsttimes Nftfilters is obtained by applying the outer product of each 1D slot-time sinc filter with each 1D fast-time sinc filter, thereby covering the complete observable frequency domain. Therefore, there are no separated filters for slow-time and fast-time. By initializing the 2D sinc filters of constrained 2D convolutional layer502in this manner, some embodiments avoid preferential frequency areas by initialization.

Since the sampling frequency in fast-time direction fs,ftmay be orders of magnitude higher than the sampling frequency in the slow-time direction fs,st, in some embodiments, the cutoff frequencies and bandwidths of the short-time and fast-time filters of the constrained 2D convolutional layer502are normalized to a value between 0 and 1, inclusive, to, e.g., allow for equal training in both filter dimensions (slow-time and fast-time).

The unconstrained convolutional layer808and dense layer812are initialized using the “Glorot” initialization scheme, for example.

During training, the hyperparameters of constrained 2D convolutional layer502, as well as the trainable parameters of the additional layers804are trained based on the training data. For example, during training, the output vector generated by DCNN800is compared with the pre-labels of the respective data batch, and the trainable weights of the neural network are adjusted so that the classification of a respective batch of data corresponds to the respective pre-labels. The model (the neural network800) is refined by running a plurality of training data batches, e.g., hundreds or thousands of training data batches.

In some embodiments, an optimizer, such as an RMSprop optimizer, is used to optimize DCNN800. Other optimizers, such as a gradient decent optimizer, and gradient decent with momentum optimizer, may also be used. In some embodiments, the learning rate is lris 0.0001, with ρ of 0.9 and ε of 10−8. Other learning parameter values may also be used.

Dropout layers822,824, and826are used during training to help create redundancy in the neural network and prevent overfitting by randomly removing nodes (e.g., randomly zeroing weights on the previous layer) and corresponding edges to/from the removed nodes of the neural network. For example, during training, the sequence of layers is502,822,806,808,824,810,812,826, and814. In some embodiments, 20% of the nodes are removed by each of the dropout layers822,824, and826.

Constrained 2D convolutional layer502may be implemented with 65 filters in the slow-time dimension and 33 filters in the fast-time dimension. A different number of filters may also be used.

The maxpool layers806and810may be implemented with pooling sizes of 8×2, and 4×2, respectively. Other pooling sizes may also be used.

Unconstrained 2D convolutional layer808may be implemented with 50 filters of size 3×3. In some embodiments, more than 50 filters, such as 60 or more, or less than 50 filters, such as 45 filters, or less, may be used. In some embodiments, filters of size different than 3×3, such as 4×4, or 2×2, may also be used.

After maxpool layer810, the tensor is flattened and fed into dense layer812, which may have a size of, e.g., 32. In some embodiments, dense layer may have a size different than 32, such as higher than 32 (e.g., 35, 40, or higher), or lower than 32 (e.g., 28, 24, or lower).

After dense layer812, softmax classifier layer814generates the classes. In the embodiment in which 6 classes are considered, softmax classifier814has a size of 6 (corresponding to each of the 6 classes). Some embodiments may be implemented softmax classifier814with a size smaller than 6, such as 5, 4, 3 or 2, or with a size higher than 6, such as 7, 8, 10, or higher.

Once trained, DCNN800may be used to classify objects, such as to classify activities of humans in scene130. During normal operation (after training), the sequence of layers of DCNN800is502,806,808,810,812, and814. Constrained 2D convolutional layer502receives raw digital data xout_dig(n) from ADC112and filters it by convolving the raw digital data xout_dig(n) with the plurality of trained 2D sinc filters. The filtered data is pooled using maxpool layer806, to, e.g., smoothen the data, e.g., by applying averaging. The pooled data is then filter with trained unconstrained 2D convolutional layer808. The output of convolutional layer808is pooled using maxpool layer810to, e.g., smoothen the data, e.g., by applying averaging and, e.g., to decrease dimensionality. The tensor generated by maxpool layer810is then flattened and fed into dense layer812followed by softmax classifier layer814. Softmax classifier layer814generates an output vector with probabilities associated with each classification.

Advantages of some embodiments include that, by constraining the first 2D convolutional layer to a particular filter shape (e.g., such as with 2D sinc filters), some embodiments advantageously allow for faster convergence during the training process of the neural network when compared to an unconstrained convolutional layer. In some embodiments, constraining the first 2D convolutional layer to a particular filter shape (e.g., such as with 2D sinc filters) has the additional advantage of helping to overcome the problems of getting stuck in local minima.

Some embodiments may implement the first constrained convolutional layer of the DCNN with filters other than 2D sinc filters. For example,FIG.9shows a block diagram of DCNN900for activity classification, according to an embodiment of the present invention. DCNN900operates in a similar manner as DCNN500. DCNN900, however, implements the first constrained 2D convolutional layer902with Morlet wavelet filters instead of 2D sinc filters.

A Morlet wavelet may be understood as the multiplication of an underlying frequency (carrier) by a Gaussian window (envelope). In some embodiments, each 2D Morlet wavelet filter of 2D convolutional layer902may be given by

ϕwave(n,m;fcst,σst,fcft,σft)=gN,M(n,m;σst,σft)⁢cos(2⁢π⁢fcst·n-⌊N2⌋N)⁢cos(2⁢π⁢fcft·m-⌊M2⌋M)(10)
where N is a slow-time filter length, n is a slow-time filter parameter index (n is an integer between 0 and N, inclusive), M is a fast-time filter length, m is a fast-time filter parameter index (m is an integer between 0 and M, inclusive), σstis a slow-time standard deviation, σftis a fast-time standard deviation, fcstis a slow-time center frequency, fcftis a fast-time center frequency, fsstis a slow-time sampling frequency, fsftis a fast-time sampling frequency, and gN,M(n, m; σst, σft) may be given by

gN,M(n,m;σst,σft)=12⁢π⁢σst⁢σft⁢e-((nN-⌊N2⌋)22⁢σst2+(mM-⌊M2⌋)22⁢σft2)(11)

In the 2D Morlet wavelet filters ϕwave(n, m; fst, σst, fft, σft) of 2D convolutional layer902, as defined by Equation 10, the trainable parameters (the hyperparameters), are the center frequencies (fcstand fcft) and the standard deviations (σstand σft) of the wavelets. Similar to the 2D sinc filters used in constrained 2D convolutional layer502, wavelet filters can adjust the frequency area of interest by, e.g., adjusting the center frequencies (fcstand fcft) of the filters. Additionally, however, wavelets can adjust the time-frequency resolution by adjusting the standard deviation (σstand σft) of the Gaussian part of the wavelet.

As shown by Equation 10, since the Morlet wavelet is the result of the product of a cosine function and a Gaussian window function, the frequency response of the resulting Morlet wavelet also has a Gaussian shape. For example,FIG.10shows an exemplary 2D Morlet wavelet filter ϕwave(n, m; fst, σst, fft, σft), as used in constrained 2D convolutional layer902, in time and frequency domains, according to an embodiment of the present invention. As shown inFIG.10, the Morlet wavelet filter does not exhibit a clear cutoff frequency (as shown by plot1020). The standard deviations of the Gaussian function in time domain and in frequency domain are indirect proportional, where decreasing the width of the Gaussian function in time domains results to an increased width in the frequency domain (which shows the time-frequency resolution tradeoff).

Similar to the 2D sinc filters used in constrained 2D convolutional layer502, the 2D Morlet wavelet filters used in constrained 2D convolutional layer902may include a rotational parameter for rotating the 2D Morlet wavelet filters with respect to the pseudo range domain and pseudo Doppler domain. In such embodiments, each 2D Morlet wavelet filter of 2D convolutional layer902may be given by
ϕwave(n,m;fcst,σst,fcft,σft)=gN,M(n,m;σst,σft)cos(2πfcstt′st)cos(2πfcftt′ft)   (12)
where t′stand t′ftmay be given by Equations 8 and 9, respectively, where a is a rotation angle, N is a slow-time filter length, M is a fast-time filter length, σstis a slow-time standard deviation, σftis a fast-time standard deviation, fcstis a slow-time center frequency, fcftis a fast-time center frequency, fsstis a slow-time sampling frequency, fsftis a fast-time sampling frequency, gN,M(n, m; σst, σft) may be given by Equation 11.

When the rotation angle α is equal to 0, Equation 12 can be expressed as Equation 10.

In some embodiments, during training, constrained 2D convolutional layer902is initialized to cover the entire range-Doppler space. For example, in some embodiments, constrained 2D convolutional layer902may be initialized by directly defining 2D Morlet wavelet filters. In other embodiments, constrained 2D convolutional layer902may be initialized by generating the 2D Morlet wavelet filters using 1D Morlet wavelet filters. Other initialization schemes are also possible.

For example, in some embodiments, Nst1D Morlet wavelet filters and Nft1D Morlet wavelet filters are initialized to equally divide the slow-time and fast-time dimensions into equal bands. Both time axes are normalized to a value between 0 and 1, inclusive. The initial set of Nsttimes Nftfilters is obtained by applying the outer product of each 1D slot-time Morlet wavelet filter with each 1D fast-time Morlet wavelet filter, thereby covering the complete observable frequency domain. In some embodiments, the standard deviation is initialized to 0.6 (other values may also be used). In some embodiments, the trainable weights (the hyperparameters) of the Morlet wavelet filters are also normalized by mapping to a value range between 0 and 1, inclusive.

By using wavelet filters in the first constrained 2D convolutional layer of the DCNN, some embodiments advantageously allows for adjusting the time-frequency resolution of the filters of the first 2D convolutional layer by, e.g., adjusting the standard deviation of the Gaussian distributions associated with the wavelet filters.

FIGS.11-17show experimental setup or results of DCNN800and DCNN900, implemented using radar system300, according to embodiments of the present invention.

For generating the experimental results, radar system300was implemented using up-chirps, as shown inFIG.4, with a single chirp per frame and a PRT of 1 ms, where each chirp is generated with 128 samples and has a duration of 64 ρs. Each chirp had a ramp start frequency fminof 59.5 GHz, a ramp stop frequency fmaxof 60.5 GHz, and a bandwidth B of 1 GHz. ADC112was implemented as a 12-bit ADC operating with a 2 MHz sampling frequency. The range resolution was 15 cm, with a maximum range of 9.6 m, and a maximum Doppler velocity of 1.25 m/s. The elevation and azimuth of millimeter-wave radar sensor102(the direction of the center of the beam) were 70° and 120°, respectively.

For generating the experimental results, additional layers504of DCNN900were implemented in a similar manner as additional layers804, and, during training, DCNN900included dropout layers822,824, and826, as implemented in DCNN800.

For generating the experimental results, DCNN800and900were trained for activity classification with a dataset that included five different human activities plus a recording of an empty room. DCNN800and900, therefore, were trained to identify six classes: “empty room,” “walking,” “idle,” “arm movement,” “waving,” and “working.”

FIG.11shows an experimental setup for data recording with a test person performing the activity “working,” according to an embodiment of the present invention. As shown inFIG.11, millimeter-wave radar sensor102is located in a corner of a room, where the room includes a desk in front of millimeter-wave radar sensor102, and a chair facing millimeter-wave radar sensor102.

To record the class “walking,” a single human randomly walked around the room ofFIG.11. The class “idle” was split into two recordings: in the first recording, a human was standing in front of the millimeter-wave radar sensor102; and in the second recording, the human was sitting at the table facing towards the millimeter-wave radar sensor102. To record the class “arm movement,” a human was recorded randomly moving his arms while standing in the room ofFIG.11. To record the class “waving,” a human was waving with his hand at different positions in the room ofFIG.11, facing towards millimeter-wave radar sensor102. To record the class “working,” a human is recorded working in his laptop while sitting in a chair, as shown inFIG.11. To record the class “empty room,” the room ofFIG.11was recorded with the presence of a human. During training for generating the experimental results, each human activity was performed by the same human, and each class was recorded for about 18 minutes in total.

FIG.12shows the number of samples per class of the training dataset1200used to train DCNN800and900for generating the experimental results, according to an embodiment of the present invention. Each sample of training dataset1200has 2048 chirps. The samples were generated by cutting out 2048 chirps with an overlap of 512 chirps from the recordings. Since the PRT was 1 ms, each sample captures 2.048 seconds. As shown inFIG.12, for each activity, about 700 samples are available for training per class.

For comparison purposes, results for three other neural networks trained using the same training dataset1200were also generated. A first neural network was implemented in a similar manner as DCNN800, but having an unconstrained 2D convolutional layer instead of constrained convolutional layer502. A second neural network was implemented receiving Doppler spectrogram as input (from step222) instead of raw digital data from ADC112. A third neural network was implemented receiving RDI video (from step206) instead of raw digital data from ADC112.

The five neural networks (DCNN800and900, and the three other neural networks) were trained using training dataset1200for 20 epochs (except that the spectrogram-based neural network was trained for 100 epochs to allow for convergence), where an epoch indicates the number of passes of the entire training dataset the machine learning algorithm has completed

FIG.13shows the cumulative gains of initial 2D filters of convolutional layers502and902, according to embodiments of the present invention. Plots1302,1304, and1306show the initial cumulative gains (after initialization but before training) in range, velocity and the joint range-velocity gain, respectively, of the 2D sinc filters of constrained convolutional layer502. Plots1312,1314, and1316show the initial cumulative gains (after initialization but before training) in range, velocity and the joint range-velocity gain, respectively, of the 2D Morlet wavelet filters of constrained convolutional layer902. Plots1322,1324, and1326show the initial cumulative gains (after initialization but before training) in range, velocity and the joint range-velocity gain, respectively, of the unconstrained convolutional layer when initialized using the Glorot initialization scheme. As shown inFIG.13, the initial gains are approximately constant over the whole space.

During training, the filter parameters are iteratively optimized.FIG.14shows the cumulative gains of 2D filters of convolutional layers502and902after 20 epochs of training, according to embodiments of the present invention. Plots1402,1404, and1406show the trained cumulative gains in range, velocity and the joint range-velocity gain, respectively, of the 2D sinc filters of constrained convolutional layer502. Plots1412,1414, and1416show the trained cumulative gains in range, velocity and the joint range-velocity gain, respectively, of the 2D Morlet wavelet filters of constrained convolutional layer902. Plots1422,1424, and1426show the trained cumulative gains in range, velocity and the joint range-velocity gain, respectively, of the unconstrained convolutional layer.

As shown inFIG.14, the cumulative gain of the 2D sinc filters and the 2D Morlet wavelet filters exhibit a bandpass characteristic. Therefore, the resulting cumulative gain of the 2D sinc filters and the 2D Morlet wavelet filters are similar, except that the 2D Morlet wavelet gain is smoother due to the nature of its filter shape in frequency domain.

However, the resulting shape of the unconstrained convolutional layer is different from the shape of the 2D sinc filters or the 2D Morlet wavelet filters. A reason for the difference in shape of the unconstrained 2D convolutional layer is because the search space of the unconstrained convolutional layer is not constrained to a particular filter type. Therefore, although the 2D sinc filter and the 2D wavelet filter are within the search space of the unconstrained 2D convolutional layer, such search space is orders of magnitude larger than the search space of a 2D convolutional layer constrained to a particular filter type. Thus, arriving to a comparable solution using an unconstrained 2D convolutional layer may require longer training time compared to using a constrained 2D convolutional layer, such as502and902. Additionally, an unconstrained 2D convolutional layer may get stuck during training at a local minima and may fail to arrive at a solution with a performance comparable to using a constrained 2D convolutional layer, such as502and902.

FIG.15shows the accuracy, standard deviation, and F1-scores of DCNN800, DCNN900, and the three additional networks, according to an embodiment of the present invention. As shown inFIG.15, after 20 epochs of training, DCNN800and DCNN900achieve an accuracy of 98.9% and 99.5%, respectively, which is significantly better than the accuracy achieved by the DCNN implemented using an unconstrained first 2D convolutional layer, using the spectrogram-based neural network (even though the spectrogram-based neural network was trained for 100 epochs to allow for convergence) or the RDI video-based neural network. As shown, DCNN800and DCNN900converge, achieving an accuracy of near 100% in less than 21 epochs of training.

Limits on the spectrogram-based and RDI video-based approaches are due, in part, to their respective preprocessing steps. For example, the lack of range information in spectrogram-based approaches may have a detrimental impact on class prediction, which may be exacerbated when analyzing activities of multiple humans simultaneously. As another example, the STFT used for generating the RDIs equally discretizes the range as well as the velocity domains. However, some activities, such as “idle” and “working” exhibit very slight movements. As a result, their features share similar range-Doppler bins and thus, the STFT processed data is very similar for both actions, which makes the classification task difficult.

FIG.16show confusion matrices for RDI video-based classification and classification using DCNN900, according to an embodiment of the present invention. As shown inFIG.16, the “idle” and “working” classes exhibit a significantly lower level of accuracy using RDI video-based neural network classification (as shown by confusion matrix1602) when compared with using DCNN900(as shown by confusion matrix1622).

Advantages of some embodiments include that, by allowing the neural network to operate directly from raw digital data (e.g., from ADC112) instead of using preprocessing steps (such as spectrogram-based or RDI video-based preprocessing), better accuracy is achieved.

Although using an unconstrained 2D convolutional layer that directly operates on raw digital data (e.g., from ADC112) may not exhibit the limitations of spectrogram-based or RDI-based implementations, constraining the search space using specific types of filters, such as 2D sinc filters and 2D wavelet filters leads to a reduction in the search space and advantageously allows some embodiments to arrive to a solution that is at least close to the global minima (the global optimum) while significantly reducing the training effort when compared to using unconstrained convolutional layer.

FIG.17shows model sizes of DCNN800, DCNN900, and a DCNN implemented using an unconstrained 2D convolutional layer as the first layer, according to an embodiment of the present invention. As shown, layer502was implemented with 64 filters of 65 by 33, layer902was implemented with 64 filters of 129 by 33, and unconstrained convolutional layer was implemented with 64 filters of 65 by 33.

As shown inFIG.17, DCNN800and DCNN900have a size that is less than half of the size of the unconstrained DCNN. The reason for such smaller size is because the first layer of DCNN800and900is significantly smaller than the size of the first layer of the unconstrained 2D convolutional layer. The reason for the smaller size of layers502and902is that only four hyperparameters are trainable (64 filters times 4 hyperparameters equals 256), with the rest of the weights being fixed (compared with the unconstrained convolutional layer, in which all weights are trainable).

FIG.18shows a block diagram of DCNN1800for target and/or activity classification, according to an embodiment of the present invention. DCNN1800includes constrained 2D convolutional layer1802and a plurality of additional layers504. DCNN1800may be implemented, e.g., in AI accelerator322. In some embodiments, constrained convolutional layer1802is a constrained L dimensional convolutional layer, where L is a positive integer greater than or equal to 2.

Constrained 2D convolutional layer1802may be implemented as constrained 2D convolutional layer502or902. In some embodiments, constrained 2D convolutional layer1802may be implemented using a Fractional Fourier Transform or using a Discrete Cosine Transform filters. Other filter types that include the global minima within their search space may also be used. In some embodiments, filters with a search space that includes only local minima may also be used.

In some embodiments, the filter kernels of constrained 2D convolutional layer1802have a size of 10×10 or higher, such as 65×33, for example.

DCNN1800may be used for human activity classification, e.g., in a similar manner as DCNN800and DCNN900. DCNN1800may also be used for other types of classification. For example, in some embodiments, DCNN1800may be used for gesture sensing applications, in which each gesture (e.g., a human gesture, such as gesturing with the hands) corresponds to a class. In such embodiments, DCNN1800is trained using a dataset based on the gestures to be recognized.

In some embodiments, DCNN1800may be used for people detection applications, in which objects are classified as humans or not humans. In such embodiments, DCNN1800is trained using a dataset based on humans and non-humans.

Some embodiments may implement other types of classifications. For example, in some embodiments, the set of classes include a class indicative of the number of humans present (e.g., to count the number of humans in a room). In some embodiments, the set of classes include a class indicative of the presence of a human, and a class indicative of the absence of human. Other classifications are also possible.

Some embodiments implement a multi-layer approach to the first layers of the DCNN1800. For example, in some embodiments, additional constrained convolutional layers follow the first constrained convolutional layer. For example,FIG.19shows a block diagram of DCNN1900for target and/or activity classification, according to an embodiment of the present invention. DCNN1900is a possible implementation of DCNN1800and includes constrained 2D convolutional layer1802and a plurality of additional layers1904. DCNN1900may be implemented, e.g., in AI accelerator322.

Additional layers1904include a second constrained 2D convolutional layer1902, and a plurality of additional layers1906. In some embodiments, additional layers504may be implemented as additional layers1904. In some embodiments, additional layers1906may be implemented as additional layers804. Other implementations are also possible.

In some embodiments, filters implemented with constrained convolutional layer1802and/or constrained convolutional layer1902, and/or additional layers1906can be complex. For example, in some embodiments, learned cosine kernels are implemented as complex kernels, in which the same real kernel is phase-shifted by 90° and convolved with the input.

In some embodiments, constrained convolutional layer1902may be implemented as an L-dimensional convolutional layer, where L is a positive integer greater than or equal to 2. The number of dimensions of constrained convolutional layer1902may be the same or different than the number of dimensions of constrained convolutional layer1802.

DCNN1900operates in a similar manner as DCNN1800. DCNN1900, however, includes second constrained 2D convolutional layer1902that operates in combination with the first constrained 2D convolutional layer1802. For example, in some embodiments, the first constrained 2D convolutional layer1802implements a plurality of coarse filters that generate respective channels of data. The data within each channel of data is then downsampled (e.g., by using a stride greater than 1 in the convolution or pooling layer), e.g., to reduce computational complexity. The second constrained 2D convolutional layer1902then operates only within the previous filtered frequency areas of the respective channels instead of covering the entire observable search space.

FIG.20shows exemplary plots2002and2022of the frequency response of first and second constrained 2D convolutional layers1802and1902, respectively, according to an embodiment of the present invention. In the embodiment ofFIG.20, 2D sinc filters were used to implement the first and second constrained 2D convolutional layers1802and1902. Other embodiments may use other filter types.

In some embodiments, the first and second constrained 2D convolutional layers1802and1902may be implemented with filters of different type. For example, in some embodiments, the first constrained 2D convolutional layer1802may be implemented with 2D sinc filters while the second constrained 2D convolutional layer1902may be implemented with 2D Morlet wavelet filters. Other implementations are also possible.

As shown in plot2002, the first constrained 2D convolutional layer1802uses coarse filters that are trained to adjust their cutoff frequencies based on the training dataset. As shown in plot2022, the filters of second constrained 2D convolutional layer1902are also trained to adjust their cutoff frequencies based on the training dataset, but their search space is restricted to be within the cutoff frequencies determined by the first constrained 2D convolutional layer1802.

In some embodiments, the first and second constrained 2D convolutional layers1802and1902are trained simultaneously. For example, in some embodiments, L2 norm conditions the search space of the filters of second constrained 2D convolutional layer1902(as a soft constraint). In some embodiments, explicit boundaries condition the search space of the filters of second constrained 2D convolutional layer1902(hard constraint).

As shown, e.g., inFIG.18, DCNN1800may generate an output that corresponds to a classification of a target (e.g., human activity classification, gesture recognition, people detection, people counting, etc.) based, e.g., on a predefined set of classes. In some embodiments, DCNN1800may generate other outputs, in addition to, or instead of, an output indicative of the classification of a target, e.g., based on a predefined set of classes. For example,FIG.21shows a block diagram of DCNN1800for generating 2D radar images, according to an embodiment of the present invention.

As shown inFIG.21, DCNN1800can be trained to produce radar images based on raw digital data xout_dig(n) from ADC112. In some embodiments, DCNN1800may produce 2D radar images, such as range-Doppler images (RDIs), range-angle images (RAIs), and Doppler-angle images (DAIs), for example. Some embodiments, thus, may implement pre-processing and feature extractions steps (e.g., steps205and206ofFIGS.2A, and steps205and222ofFIG.2B) implicitly by the DCNN1800.

DCNN1800may be trained to produce radar images by using a training dataset generated from (e.g., radar and/or camera) recordings, e.g., of one or more humans performing one or more activities in scene130, such as walking, working, standing, waving, arm movement, etc. Each sample of the training dataset is pre-labeled with a corresponding 2D radar image, e.g., with an artificial filter manually removing the ghost targets and adding real targets whenever missed in the generated 2D radar image. The difference between the obtained 2D image (at the output of DCNN1800) and the corresponding pre-labeled 2D image is used as the error for training DCNN1800.

FIG.22shows exemplary 2D radar images2202and2222during training of DCNN1800, according to an embodiment of the present invention. Radar image2222is a pre-labeled 2D RDI that corresponds to a recording of 4 humans as they walk inside a room.

As shown, radar image2222shows 4 identified targets2204,2206,2208, and2210. For example, in some embodiments, radar image2222is a matrix in which a 1 represents that a target is located at that identified range-velocity point, and a 0 represents that no target is located at that range-velocity point.

During training, for example, the recording of the 4 humans walking inside the room is fed to DCNN1800, in which, for example, 2D radar image2202is generated. As shown, there are differences between the generated radar image2202and the pre-labeled radar image2222. For example, human2210is not detected in radar image2202. The difference between radar images2202and2222represents the error, which is used to optimize DCNN1800, e.g., by adjusting the hyperparameters of constrained convolutional layer1802and/or other weights of additional layers504, e.g., based on a loss function, such as means square error.

In some embodiments, the pre-labeled radar images are generated by using, e.g., a camera that records images simultaneously with the radar recordings for generating the training dataset. In some embodiments, the pre-labeled radar images are generated, e.g., by performing steps205and206on the radar recordings, instead of or in addition to using the camera images.

In some embodiments, the pre-labeled images are manually labeled by a user based on knowledge of the targets and corresponding activities. In some embodiments, at least some of the pre-labeled radar images are generated synthetically. For example, in some embodiments, radar images with multiple humans are synthetically computed by performing a data augmentation step during training based on multiple single human radar images, e.g., by superimposing multiple, single human, radar images. For example, two single human images, in which the humans are not close together in the corresponding radar images, are superimposed to generate a two human radar image.

Although RDIs have been used to illustrate a possible method of training DCNN1800for generating radar images, it is understood that other radar images may also be used, such as RAIs and DAIs, for example.

In some embodiments, the output of DCNN1800is further processed, e.g., to track targets, count people, or other applications. For example,FIG.23shows a schematic diagram of radar system2300, according to an embodiment of the present invention. Radar system2300includes millimeter-wave radar sensor102, AI accelerator322, and processor1910.

As shown inFIG.23, processor2310may be to post-process the output of DCNN1800, which includes information about targets in scene130, such as target classification information and/or radar images. For example, in some embodiments, during normal operation, radar images with target location information (such as radar image2222) are fed to processor2310, in which targets are clustered, e.g., to group detected targets as a single target (e.g., to group detected hands, torso and feet as a single human target).

Processor2310may cluster targets based on the output of DCNN1800using, for example, density-based spatial clustering of applications with noise (DBSCAN), other clustering methods may also be used.

In some embodiments, processor2310is used to track targets and the activities of the targets. For example, in some embodiments, processor2310may track which activities a detected human target is performing over time, such as sitting, then standing, then walking, then working, etc.

In some embodiments, a Kalman filter may be used to track one or more targets based on radar images received from DCNN1800. In some embodiments, the Kalman filter also tracks associated target classifications (e.g., associated human activities, gestures, etc.) of the detected targets based on a classification output of DCNN1800. Tracking methods other than using a Kalman filter, or in addition to using a Kalman filter, may also be used. For example, some embodiments may use a particle filter instead of a Kalman filter, for tracking targets.

Processor2310may be used for other post-processing activities, in addition to, or instead of, clustering and/or tracking targets. Processor2310may be implemented in a similar manner as processor120. In some embodiments, processor2310and AI accelerator322are integrated in a single integrated circuit (IC).

Advantages of some embodiments include minimizing the data flow of the radar system. For example, in radar system2300, data flows from millimeter-wave radar102, to AI accelerator322(e.g., for classification), then to processor2310(for post-processing). An approach based on radar system100would instead exhibit a data flow from millimeter-wave radar102, to processor120(for preprocessing), then to AI accelerator122(for classification), then back to processor120(for post-processing).

In some embodiments, a first constrained convolutional layer may be used to feed its output to a plurality of additional layers paths. For example,FIG.24shows a block diagram of DCNN2400, according to an embodiment of the present invention. DCNN2400may be used, for example, for generating target localization data.

DCNN2400includes constrained 3D convolutional layer2402, and additional layers2404,2406, and2408. It is understood that constrained 3D convolutional layer2402is a possible implementation of constrained convolutional layer1802. Each of additional layers2404,2406, and2408may be implemented, e.g., as additional layers504. In some embodiments, additional layers2404,2406, and2408, have the same architecture (e.g., the same sequence of identical layers), and may be initialized in a similar manner (although the weights of the respective trainable parameters after training may differ), e.g., but performing pointwise convolutions along different axis. For example, in some embodiments, additional layers2404,2406, and2408keep all layer parameters the same, but additional layers2404perform pointwise convolution along channels dimension to generate reconstructed RDIs, additional layers2406perform pointwise convolution along fast-time dimension to generate reconstructed DAIs; additional layers2408perform pointwise convolution along slow-time dimension to generate reconstructed RAIs.

In other embodiments, some, or all of additional layers2404,2406, and2408may have different architectures and/or may be initialized in a different manner from each other.

Integration step2410may be performed on the outputs of additional layers2404,2406, and2408to generate localization data. For example, in some embodiments, integration step2410is implemented with soft information transfers between layers, such as by using softmax layers (e.g., with connections between additional layers2402,2406and/or2408). Other embodiments may implement integration step2410by performing signal processing operations to integration the reconstructed RDIs, DAIs and/or RAIs to generate localization data.

In some embodiments, constrained convolutional layer2402is implemented with a plurality of 3D sinc filters. In some embodiments, each of the dimensions of the 3D sinc filters corresponds to slow-time, fast-time, and channels, respectively, where each channel corresponds to a data stream from a (real or virtual) RX antenna associated with millimeter-wave radar sensor102. For example, in an embodiment in which 2 real antennas are used for receiving reflected radar signals108, the number of channels is 2. More than 2 antennas may also be used.

In some embodiments, the 3D sinc filters of c constrained convolutional layer2402are phase-shifted. For example, if one channel has kernel w1, the next channel is applied w1·cos(θ), the next channel is applied w1·cos(2θ), and so on, where θ is the angle of the target.

The RX antennas used for each channel may be implemented in any way known in the art. For example, in some embodiments, 1 TX antenna and 3 TX antennas are implemented in an L-shape configuration for a 3-channel implementation. Other implementations are also possible.

During normal operation, the output of constrained 3D convolutional layer is fed to one or more paths of additional layers, such as additional layers2404,2406and/or2408. The output of the one or more additional layers may be integrated by integration step2410to generate localization data along range, Doppler and angle. As shown inFIG.24, the localization data may have 3 dimensions, such as range, Doppler, and angle. In some embodiments, the generated localization data may be in the form of 2D range-Doppler images (RDIs), 2D range-angle images (RAIs) and/or 2D Doppler-angle images (DAIs).

Additional layers2404receives the output from constrained 3D convolutional layer2402and generates reconstructed RDIs, (e.g., similar to RDI2222), in which the location of detected targets are identified in the range-Doppler domain. As shown inFIG.24, additional layers2404operate on the received input by pointwise convolving along the channel domain, thus generating slices along fast-time-slow-time domain, where pointwise convolution along a first domain may be understood as learned weighted summation along such first domain.

Additional layers2406receives the output from constrained 3D convolutional layer2402and generates reconstructed DAI, in which the locations of detected targets are identified in the Doppler-angle domain. As shown inFIG.24, additional layers2406operate on the received input by pointwise convolving along fast-time, thus generating slices along Doppler-channel domain.

Additional layers2408receives the output from constrained 3D convolutional layer2402and generates reconstructed RAI, in which the locations of detected targets are identified in the range-angle domain. As shown inFIG.24, additional layers2408operate on the received input by pointwise convolving along slow-time, thus generating slices along fast time-channel domain.

In some embodiments, only two of the additional layers (e.g., only additional layers2404and2408) are implemented.

In some embodiments, the integration layer2410may be omitted. For example, in some embodiments, only one of the additional layers (such as one of additional layers2404,2406, or2408) is implemented.

In an embodiment having 2 channels, constrained 3D convolutional layer2402may be implemented with, e.g., F1×F22D sinc filters associated with the first channel, and F1×F22D sinc filters associated with the second channel filters, where, e.g., the second F1×F22D sinc filters associated with the second channel are implemented as cos(θ) times the F1×F22D sinc filters associated with the first channel. In some embodiments, the K kernels additionally provide the sum of the results of the F1×F22D sinc filters associated with the first channel, and the results of F1×F22D sinc filters associated with the second channel filters into an F1×F2matrix. Therefore, the K kernels would transform the input along channels from 2 to K. In some embodiments, F1is equal to 36, F2, is equal to 65. Other values for F1and F2are also possible.

In some embodiments, the kernel values are normalized so that the 2D sinc filters associated with the first channel have a value of w1=1 for all angles θ. Each of the F1×F2filters associated with the second channel may be given by

sincch⁢2=w1⁢cos⁡(2⁢π⁢d⁢sin⁢(θ)λ)(13)
where d is the distance between the two RX antennas, λ is the wavelength of the transmitted signal (e.g., the center frequency of the chirp), and θ is the angle of the target.

In some embodiments,

d/λ=0.5and⁢cos⁡(2⁢π⁢d⁢sin⁢(θ)λ)=cos⁡(π⁢sin⁢(θ)),
where θ∈[−90, 90] and cos(π sin(θ))∈[cos(−π), cos(π)]. Thus, in some embodiments, each kthtrainable kernels are cos(−π+k*(2π/K)), and the trained spatial frequency are

cos⁡(-π+k*(2⁢πK)+bwa),
K is the number of angle kernels, and bwais the learnable parameter. Thus, the learned kernels for the second channel is

w2=w1⁢cos⁡(-π+k*(2⁢πK)+bwa).

In some embodiments, unlike fast-time and slow-time domain where the learned kernels are applied through convolution operation, the learned kernels along antennas are applied through 1×1 convolution/fully-connected connections. In an embodiment with K=29 (angle kernels), and M overall kernels, the 4D kernel map after constrained 3D convolutional layer2402is 128×32×29×M.

In some embodiments, these kernels can be implemented through complex weights, where the kthkernel can be expressed as

w2k=w1k(cos⁡(-π+k*(2⁢πK)+bwa)+j⁢sin⁡(-π+k*(2⁢πK)+bwa))(14)

Example embodiments of the present invention are summarized here. Other embodiments can also be understood from the entirety of the specification and the claims filed herein.

Example 1. A method including: transmitting a plurality of radar signals using a millimeter-wave radar sensor towards a target; receiving a plurality of reflected radar signals that correspond to the plurality of transmitted radar signals using the millimeter-wave radar; mixing a replica of the plurality of transmitted radar signals with the plurality of received reflected radar signals to generate an intermediate frequency signal; generating raw digital data based on the intermediate frequency signal using an analog-to-digital converter; processing the raw digital data using a constrained L dimensional convolutional layer of a neural network to generate intermediate digital data, where L is a positive integer greater than or equal to 2, and where the neural network includes a plurality of additional layers; and processing the intermediate digital data using the plurality of additional layers to generate information about the target.

Example 2. The method of claim1, where generating information about the target includes classifying the target based on a set of classes.

Example 3. The method of one of examples 1 or 2, where generating information about the target includes providing a location of the target using a radar image.

Example 4. The method of one of examples 1 to 3, where the radar image is a range-Doppler image (RAI), a range-angle image (RAI) or a Doppler-angle image (DAI).

Example 5. The method of one of examples 1 to 4, where L is equal to 2.

Example 6. The method of one of examples 1 to 5, where a kernel size of a filter of the constrained L dimensional convolutional layer is higher than 10 by 10.

Example 7. The method of one of examples 1 to 6, where processing the raw digital data using the constrained L dimensional convolutional layer includes processing the raw digital data using a 2D sinc filter of the constrained L dimensional convolutional layer.

Example 8. The method of one of examples 1 to 7, where the 2D sinc filter is defined by

sinc2⁢D(n,m;flst,bst,flft,bft)=w⁡(n,m)⁢hN,fsst(n;flst,bst)⁢hM,fsft(m;flft,bft)
where

hN,fsst(n;flst,bst)
is a slow-time 1D sinc filter, N is a length of the slow-time 1D sinc filter, flstis a lower cutoff frequency of the slow-time 1D sinc filter, bstis a bandwidth of the slow-time 1D sinc filter, n is an integer between 0 and N, inclusive,

hM,fsft(m;flft,bft)
is a fast-time 1D sinc filter, M is a length of the fast-time 1D sinc filter, flftis a lower cutoff frequency of the fast-time 1D sinc filter, bftis a bandwidth of the fast-time 1D sinc filter, m is an integer between 0 and M, inclusive, w(n,m) is a 2D cosine weighting function, wherein the slow-time 1D sinc filter and the fast-time 1D sinc filter are defined by

hK,fs(k,fl,b)=2⁢(fl+b)⁢sinc(2⁢(fl+b)·k-⌊K2⌋fs)-2⁢fl·sinc(2⁢fl·k-⌊K2⌋fs)
where K is a length of a 1D sinc filter, k is an integer between 0 and K, inclusive, fsis a sampling frequency of a signal to be filtered, flis a lower cutoff frequency, b is a bandwidth of the 1D sinc filter, and k is a filter parameter index.

Example 9. The method of one of examples 1 to 7, where the 2D sinc filter is defined by

ϕsinc(n,m;flst,bst,flft,bft)=4⁢w⁡(n,m)((flst+bst)⁢(flft+bft)⁢sinc⁡(2⁢(flst+bst)⁢tst′,2⁢(flft+bft)⁢tft′)-(flst+bst)⁢flft⁢sinc⁡(2⁢(flst+bst)⁢tst′,2⁢flft⁢tft′)-flst(flft+bft)⁢sinc⁡(2⁢flst⁢tst′,2⁢(flft+bft)⁢tft′)+flst⁢flft⁢sinc⁡(2⁢flst⁢tst′,2⁢tft′))where⁢sinc⁡(x,y)=sin⁡(π⁢x)⁢sin⁡(π⁢y)π2⁢xytst′=n-⌊N2⌋fsst⁢cos⁡(α)-m-⌊M2⌋fsft⁢sin⁡(α)tft′=n-⌊N2⌋fsst⁢sin⁡(α)+m-⌊M2⌋fsft⁢cos⁡(α)
where α is a rotation angle and wherein α is different from 0, wherein N is a length of the 2D sinc filter in slow-time, flstis a lower cutoff frequency of the 2D sinc filter in slow-time, bstis a bandwidth of the 2D sinc filter in slow-time, n is an integer between 0 and N, inclusive, M is a length of the 2D sinc filter in fast-time, flftis a lower cutoff frequency of the 2D sinc filter in fast-time, bftis a bandwidth of the 2D sinc filter in fast-time, m is an integer between 0 and M, inclusive, and w(n,m) is a 2D cosine weighting function.

Example 10. The method of one of examples 1 to 5, where processing the raw digital data using the constrained L dimensional convolutional layer includes processing the raw digital data using a 2D Morlet wavelet filter of the constrained L dimensional convolutional layer.

Example 11. The method of one of examples 1 to 5 or 10, where the 2D Morlet wavelet filter is defined by

ϕwave(n,m;fcst,σst,fcft,σft)=gN,M(n,m;σst,σft)⁢cos(2⁢π⁢fcst·n-⌊N2⌋fsst)⁢cos(2⁢π⁢fcft·m-⌊M2⌋fsft)
where N is a slow-time filter length, n is an integer between 0 and N, inclusive, M is a fast-time filter length, m is an integer between 0 and M, inclusive, σstis a slow-time standard deviation, σftis a fast-time standard deviation, fcstis a slow-time center frequency, fcftis a fast-time center frequency, fsstis a slow-time sampling frequency, fsftis a fast-time sampling frequency, and wherein gN,M(n, m; σst, σft) is defined by

gN,M(n,m;σst,σft)=12⁢π⁢σst⁢σft⁢e-((nN-⌊N2⌋)22⁢σst2+(mM-⌊M2⌋)22⁢σft2).

Example 12. The method of one of examples 1 to 5, or 10, where the 2D Morlet wavelet filter is defined by

ϕwave(n,m;fcst,σst,fcft,σft)=gN,M(n,m;σst,σft)⁢cos⁡(2⁢π⁢fcst⁢tst′)⁢cos⁡(2⁢π⁢fcft⁢tft′)⁢wheretst′=n-⌊N2⌋fsst⁢cos⁡(α)-m-⌊M2⌋fsft⁢sin⁡(α)tft′=n-⌊N2⌋fsst⁢sin⁡(α)-m-⌊M2⌋fsft⁢cos⁡(α)
wherein α is a rotation angle and wherein α is different from 0, wherein N is a slow-time filter length, n is an integer between 0 and N, inclusive, M is a fast-time filter length, m is an integer between 0 and M, inclusive, σstis a slow-time standard deviation, σftis a fast-time standard deviation, fcstis a slow-time center frequency, fcftis a fast-time center frequency, fsstis a slow-time sampling frequency, fsftis a fast-time sampling frequency, and wherein gN,M(n,m,σst,σft) is defined by

gN,M(n,m;σst,σft)=12⁢π⁢σst⁢σft⁢e-((nN-⌊N2⌋)22⁢σst2+(mM-⌊M2⌋)22⁢σft2).

Example 13. The method of one of examples 1 to 12, further including training the neural network by: initializing the neural network; and after initializing the neural network, feeding training data to the constrained L dimensional convolutional layer, where trainable weights of the constrained L dimensional convolutional layer include at least one of a slow-time cutoff frequency, a slow-time bandwidth, a fast-time cutoff frequency, and a fast-time bandwidth.

Example 14. The method of one of examples 1 to 13, further including normalizing the slow-time and fast-time cutoff frequencies, and normalizing the slow-time and fast-time bandwidths.

Example 15. The method of one of examples 1 to 14, further including training the neural network by: initializing the neural network; and after initializing the neural network, feeding a training data to the constrained L dimensional convolutional layer, where trainable weights of the constrained L dimensional convolutional layer include a center frequency and a standard deviation for each of the L dimensions.

Example 16. The method of one of examples 1 to 15, further including training the neural network for less than 21 epochs.

Example 17. The method of one of examples 1 to 16, where the plurality of additional layers includes a first maxpool layer followed by an unconstrained 2D convolutional layer followed by a second maxpool layer, followed by a dense layer, and followed by a softmax layer.

Example 18. The method of one of examples 1 to 17, where the target is a human target.

Example 19. The method of one of examples 1 to 18, where generating information about the target includes classifying the target based on a set of classes, and where the set of classes includes classes indicative of human activities.

Example 20. The method of one of examples 1 to 19, where the set of classes includes a walking class indicative of a human walking, an idle class indicative of an idle human, a random arm movements class indicative of a human exhibiting random arm movements, a waving class indicative of a human performing hand waving movements, and a working class indicative of a sitting human working with a computer.

Example 21. The method of one of examples 1 to 20, where the set of classes includes classes indicative of human gestures.

Example 22. The method of one of examples 1 to 21, where the set of classes includes a first class indicative of the presence of a human, and a second class indicative of the absence of a human.

Example 23. The method of one of examples 1 to 22, where the set of classes includes classes indicative of the number of humans present.

Example 24. The method of one of examples 1 to 23, further including tracking the target based on the generated information about the target.

Example 25. The method of one of examples 1 to 24, where the plurality of radar signals are a plurality of chirps.

Example 26. The method of one of examples 1 to 25, where the plurality of additional layer includes a first additional constrained convolutional layer.

Example 27. The method of one of examples 1 to 26, where generating information about the target includes generating a range-Doppler radar image indicative of a location of the target, the method further including processing the intermediate digital data using a second plurality of additional layers to generate a range-angle radar image indicative of the location of the target.

Example 28. A radar system including: a millimeter-wave radar sensor including: a transmitting antenna configured to transmit a plurality of radar signals towards a target; a receiving antenna configured to receive a plurality of reflected radar signals; a mixer configured to mix a replica of the plurality of transmitted radar signals with the plurality of received reflected radar signals to generate an intermediate frequency signal; an analog-to-digital converter (ADC) configured to generate, at an output of the ADC, raw digital data based on the intermediate frequency signal; and an artificial intelligence (AI) accelerator having an input coupled to the output of the ADC, and configured to: receive the raw digital data from the ADC, and process the raw digital data using a constrained L dimensional convolutional layer of a neural network to generate intermediate digital data, where L is a positive integer greater than or equal to 2, and where the neural network includes a plurality of additional layers; and process the intermediate digital data using the plurality of additional layers to generate, at an output of the AI accelerator, data associated with the target.

Example 29. The radar system of example 28, further including a digital signal processor (DSP) having an input coupled to the output of the AI accelerator, where the AI accelerator is directly connected to an output of the millimeter-wave radar sensor.

Example 30. The radar system of one of examples 28 or 29, where the DSP is configured to track a target based on the output of the AI accelerator.

Example 31. The radar system of one of examples 28 to 30, where the AI accelerator and the DSP are integrated in the same integrated circuit.

Example 32. A radar system including: a millimeter-wave radar configured to transmit a plurality of chirps towards a target, and to receive a plurality of reflected chirps; a mixer configured to mix a replica of the plurality of transmitted chirps with the plurality of received reflected chirps to generate an intermediate frequency signal; an analog-to-digital converter (ADC) configured to generate, at an output of the ADC, raw digital data based on the intermediate frequency signal; and a processor having an input coupled to the output of the ADC, and configured to: receive the raw digital data from the ADC, and process the raw digital data using a neural network having a first constrained two dimensional convolutional layer followed by a plurality of additional layers to generate, at an output of the plurality of additional layers, data associated with the target.

Example 33. The radar system of example 32, where the processor is an artificial intelligence (AI) accelerator.

While this invention has been described with reference to illustrative embodiments, this description is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of the invention, will be apparent to persons skilled in the art upon reference to the description. It is therefore intended that the appended claims encompass any such modifications or embodiments.