Patent Description:
In recent trends, human computer interaction (HCI) is not just limited to specific hardware such as mouse and keyboard but has broadened to include human sensory modes such as gestures, speech, and facial patterns. Gesture based HCI is one of the most important and attractive technique that has been widely adopted and diverse sensing modalities such as camera, wearable devices, Radio Frequency, and ultrasound are explored. Among these wide gamut of gesture detection techniques, due to the limitations such as dependence on lighting, requirement of specialized hardware etc., the ultrasound based approach looks attractive.

Gesture based HCI has numerous applications on resource constrained edge platforms such as robots, mobile phones etc. In conventional methods, the classification of gestures is achieved via deep neural networks involving convolution (CNN). However, these approaches demand large memory and computation power to run efficiently, thus limiting their use in power and memory constrained edge devices. Document <CIT> discloses systems and methods for touchless sensing and gesture recognition using continuous wave sound signals. Continuous wave sound, such as ultrasound, emitted by a transmitter may reflect from an object, and be received by one or more sound receivers. Sound signals may be temporally encoded. Received sound signals may be processed to determine a channel impulse response or calculate time of flight. Determined channel impulse responses may be processed to extract recognizable features or angles. Extracted features may be compared to a database of features to identify a user input gesture associated with the matched feature. Angles of channel impulse response curves may be associated with an input gesture. Time of flight values from each receiver may be used to determine coordinates of the reflecting object. Embodiments may be implemented as part of a graphical user interface. Embodiments may be used to determine a location of an emitter. Document <NPL>", discloses that hand rehabilitation robot can assist the patients in completing rehabilitation exercises. Usually these rehabilitation exercises are designed according to Fugl-Meyer Assessment(FMA). Surface electromyography(sEMG) signal is the most commonly used physiological signal to identify the patient's movement intention. However, recognizing the hand gesture based on the sEMG signal is still a challenging problem due to the low amplitude and non-stationary characteristics of the sEMG signal. In this paper, eight standard hand movements in FMA are selected for the active exercises by hand rehabilitation robots. A total of <NUM> volunteers' sEMG signals are collected in the course of the experiment. Four time domain features, integral EMG(IEGM), root mean square(RMS), zero crossings(ZC) and energy percentage(EP), are used to identify hand gestures. A feedforward spiking neural network receives the above time domain feature data, and combines the population coding with the Spikeprop learning algorithm to realize the accurate recognition of hand gestures. The experimental results show that: (<NUM>) the spiking neural network can achieve a satisfactory classification accuracy by using only <NUM> neurons; (<NUM>) the classification accuracy using all four features are highest with an accuracy of <NUM>%; (<NUM>) under the same number of neurons, the classification accuracy of the spiking neural network is higher than that of the multilayer perceptron, radial basis function network and support vector machine. This demonstrates the fact that spiking neural networks can achieve a satisfactory classification accuracy with a smaller network size.

Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems.

In one aspect, there is provided a processor implemented method for gesture detection using Spiking Neural Networks as defined in claim <NUM>. In another aspect, there is provided a processor implemented system for gesture detection using Spiking Neural Networks as defined in claim <NUM>. In yet another aspect, there are provided one or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by system of claim <NUM> cause gesture detection using Spiking Neural Networks is defined in claim <NUM>.

Ultrasound based gesture detection using sound navigation and ranging (SONAR) principle has been extensively explored in literature. The key advantage of this technique is that it uses off-the-shelf available speaker and microphone setup. Gesture detection methods based on ultrasound can broadly be classified under the following categories: i) Fine finger tracking followed by gesture detection, ii) Doppler shift based approach, and iii) Channel impulse response (CIR) image based approach. On the contrary,<NPL>. ") described a simple approach which cannot be classified under the aforementioned approaches by using MIMO acoustic setup and making use of simple raw time domain signal features. But as shown in Yiallourides et al. itself, their approach can only detect few gestures and furthermore the accuracy is quite low. Now, as pointed out in literature, while the finger tracking based approaches are not suitable for complex gesture detection (single reflector model), the doppler shift approaches suffer from poor resolution and cannot distinguish minor gestures accurately. On the other hand, the CIR image approach based on least square estimation has shown to perform better compared to other categories. One of the key problems of using the ultrasound based method as discussed in the literature (e.g., refer "<NPL>. " - also referred as Wang et al. ) is the ill effect of Frequency Selective Fading (FSF) due to multiple reflections emanating from complex gestures. Wang et al. proposed to overcome this FSF problem by using the frequency hopping technique but this leads to reduced available bandwidth at any given instant due to which the CIR estimation may suffer.

Gesture based HCI has numerous applications on resource constrained edge platforms such as robots, mobile phones etc. In most of the aforementioned methods, the classification of gestures is achieved via deep neural networks involving convolution (CNN). However, these approaches demand large memory and computation power to run efficiently, thus limiting their use in power and memory constrained edge devices. Lately, mammalian brain inspired spiking neural networks (SNN) which runs on neuromorphic platforms that are both data and energy efficient are extensively being considered for edge use cases.

In the present disclosure, method described herein implement an ultrasound based robust low power edge compatible gesture detection system which uses MIMO like setup in the acoustic range of <NUM> - <NUM> (e.g., depending on the hardware support and specifications) and leverages the diversity to efficiently alleviate the problem of FSF. It is observed through experiments conducted the CIR image for various gestures are sparse in nature and hence the system of the present disclosure estimates CIR by imposing the l<NUM>-norm penalty as it is well known to promote sparse solutions. The popular iterative shrinkage threshold algorithm (ISTA) is used for estimating this sparse CIR; however, in the implementation the unfolded variant of ISTA, Learned ISTA (LISTA as known in the art) (after suitable training) is employed for efficient deployment. Due to the advantages of SNNs as mentioned above, SNNs are used by the system and method of the present disclosure for gesture classification from these CIR images. Because of the discontinuous nature of voltage spikes in an SNN, supervised training using established techniques are difficult. An easier way is to convert a trained ANN into an SNN via ANN-to-SNN conversion techniques which retains similar classification accuracy while gaining on energy consumption. Here, the present disclosure and its system and method designed and trained a <NUM>-layer CNN for gesture classification and then converted it into an equivalent SNN. The performance benefit of SNN of the system of the present disclosure is compared against conventional approach (e.g., Ultragesture - (this being better performing than most other competing techniques)). The results indicate that the CIR image obtained with sparsity prior looks much better compared to the least squares approach used in literature. In addition, the classification performance of the converted SNN shows an improvement of around <NUM>% compared to the state-of-the-art Ultragesture. Moreover, converted SNN is found to have 3x less number of operations than its CNN counterpart making the former more energy efficient. This makes the system described herein a robust edge deployable system.

<FIG> depicts an exemplary acoustic system <NUM> for gesture detection using Spiking Neural Networks, in accordance with an embodiment of the present disclosure. In an embodiment, the system <NUM> may also be referred as acoustic system or gesture detection system or gesture recognition system and interchangeably used herein. In an embodiment, the system <NUM> includes one or more hardware processors <NUM>, communication interface device(s) or input/output (I/O) interface(s) <NUM> (also referred as interface(s)), and one or more data storage devices or memory <NUM> operatively coupled to the one or more hardware processors <NUM>. The one or more processors <NUM> may be one or more software processing components and/or hardware processors. In an embodiment, the hardware processors can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) is/are configured to fetch and execute computer-readable instructions stored in the memory. In an embodiment, the system <NUM> can be implemented in a variety of computing systems, such as laptop computers, notebooks, hand-held devices (e.g., smartphones, tablet phones, mobile communication devices, and the like), workstations, mainframe computers, servers, a network cloud, and the like.

The I/O interface device(s) <NUM> can include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like and can facilitate multiple communications within a wide variety of networks N/W and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. In an embodiment, the I/O interface device(s) can include one or more ports for connecting a number of devices to one another or to another server.

The memory <NUM> may include any computer-readable medium known in the art including, for example, volatile memory, such as static random-access memory (SRAM) and dynamic-random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an embodiment, a database <NUM> is comprised in the memory <NUM>, wherein the database <NUM> comprises information on transmitted signal, reflecting (or reflected) signal, sparsity prior serving as a constraint, etc. The database <NUM> further comprises a plurality of CIR images, gesture being recognized, and the like. The memory <NUM> further comprises various technique(s) such as Channel Impulse Response (CIR) estimator, logical operations, interpolation technique(s), filtering/up-sampling technique(s), padding technique(s), modulation technique(s), various band pass filter(s), processing technique(s) that include quadrature demodulation technique(s), and the like. Further, the memory <NUM> further comprises gesture recognition technique(s), quantization technique(e) and the like. Furthermore, the memory <NUM> comprises a Convolution Neural Network (CNN), a trained spike neural network (or a Spike Neural Network (SNN) being trained, and the like. The above-mentioned technique(s) are implemented as at least one of a logically self-contained part of a software program, a self-contained hardware component, and/or, a self-contained hardware component with a logically self-contained part of a software program embedded into each of the hardware component (e.g., hardware processor <NUM> or memory <NUM>) that when executed perform the method described herein. The memory <NUM> further comprises (or may further comprise) information pertaining to input(s)/output(s) of each step performed by the systems and methods of the present disclosure. In other words, input(s) fed at each step and output(s) generated at each step are comprised in the memory <NUM> and can be utilized in further processing and analysis.

<FIG>, with reference to <FIG>, depicts an exemplary high level block diagram of the acoustic system <NUM> for gesture detection using Spiking Neural Networks, in accordance with an embodiment of the present disclosure, in accordance with an embodiment of the present disclosure. More specifically, the system <NUM> in <FIG> includes a transceiver block comprising a transmitter block and a receiver block, a CIR image estimation block, a Spiking Neural Network block for gesture detection. Speakers comprised in the transceiver block transmit distinct waveforms in the <NUM> - <NUM> band and the microphones receive the reflected signals from the user. Using these reflected signals and the transmitted waveforms, m*n CIR images (e.g., <NUM> CIR images, wherein m*n=<NUM> and <NUM> indicates the number of channels, for instance, <NUM> speakers (e.g., say 'm') and <NUM> microphones (e.g., say 'n') as used in the present disclosure setup) are generated by imposing the l<NUM>-norm constraint. During the training phase, the <NUM>-layer CNN is trained natively using backpropagation on m*n CIR images (e.g., <NUM> CIR images). The trained network weights are then quantized in order to reduce their memory footprint. Finally, the quantized CNN is converted into equivalent SNN via ANN-to-SNN conversion. During testing phase, m*n CIR images are encoded into spike domain before being fed to the SNN for final classification wherein gesture detection is performed. The description of each block/component depicted in <FIG> is better understood by way of examples and in conjunction with <FIG> and/or steps of <FIG>.

<FIG>, with reference to <FIG>, depicts an exemplary high level block diagram of transmitter and receiver components of the acoustic system for gesture detection using Spiking Neural Networks, in accordance with an embodiment of the present disclosure.

<FIG>, with reference to <FIG>, depicts an exemplary flow chart illustrating an acoustic method for gesture detection using Spiking Neural Networks comprised in the systems of <FIG>, in accordance with an embodiment of the present disclosure. In an embodiment, the system(s) <NUM> comprises one or more data storage devices or the memory <NUM> operatively coupled to the one or more hardware processors <NUM> and is configured to store instructions for execution of steps of the method by the one or more processors <NUM>. The steps of the method of the present disclosure jwill now be explained with reference to components of the system <NUM> of <FIG>, the block diagram of the system <NUM> depicted in <FIG> and its components depicted in <FIG>, and the flow diagram as depicted in <FIG>.

In an embodiment of the present disclosure, at step <NUM>, the one or more hardware processors <NUM> transmit, via a plurality of speakers (e.g., speakers depicted in <FIG> and <FIG>), a plurality of modulated signals to a user. Prior to transmitting, via the plurality of speakers, the plurality of modulated signals to the user, the system <NUM> performs a logical operation on two pseudo random sequences obtained from a generator polynomial, to obtain a plurality of spreading sequence codes. More specifically, an XOR operation is performed on the two pseudo random sequences to obtain a plurality of spreading sequence codes. It is to be understood by a person having ordinary skill in the art or person skilled in the art that such XOR operation shall not be construed as limiting the scope of the present disclosure. In other words, any operation other than XOR operation may also be performed on the two pseudo random sequences to obtain the plurality of spreading sequence codes wherein each of the two pseudo random sequences has a length of predefined symbols. Further, the plurality of spreading sequence codes is interpolated using the interpolating technique comprised in the memory <NUM> to obtain a plurality of interpolated sequences. The plurality of interpolated sequences is then filtered using a filtering technique to obtain a plurality of filtered sequences. The filtering technique applied herein by the system <NUM> is a low pass filtering technique, in an example embodiment of the present disclosure. Further, the plurality of filtered sequences is appended with zeros to obtain a plurality of padded signals wherein the system <NUM> applies a padding technique on the plurality of filtered sequences to append zeros to the plurality of filtered sequences. Further, the plurality of padded signals is modulated using a modulation technique to obtain the plurality of modulated signals. the steps of filtering the plurality of interpolated sequences, appending the plurality of filtered sequences, and modulating the plurality of padded signals are performed such that each of the plurality of modulated signals obtained for transmission ranges between a first pre-defined acoustic transmission band and a second pre-defined acoustic transmission band.

Referring to steps of <FIG>, at step <NUM> of the present disclosure, the one or more hardware processors <NUM> receive, via a plurality of microphones, a plurality of reflected signals from the user, in response to the plurality of transmitted modulated signals. In an embodiment, the expressions "transmitted modulated signals" "transmitted signals" and "modulated signals", may be interchangeably used herein. In an embodiment, the step of receiving, via a plurality of microphones, a plurality of reflected signals from the user, in response to the plurality of transmitted modulated signals comprises: receiving, at the plurality of microphones, a plurality of signals based on the plurality of transmitted modulated signals; applying, at the plurality of microphones, a quadrature demodulation to the plurality of received signals to obtain a plurality of demodulated signals; and filtering, at the plurality of microphones, the plurality of demodulated signals to obtain the plurality of reflected signals.

The above steps of transmitting, via the plurality of speakers, the plurality of modulated signals to the user and receiving, via a plurality of microphones, a plurality of reflected signals from the user, in response to the plurality of transmitted modulated signals are better understood by way of depiction in <FIG>. The above steps may be further better understood by way of following description:.

As shown in <FIG> and <FIG>, the transceiver consists of a plurality of speakers and microphones that mimics a multiple-input multiple-output (MIMO) acoustic transmitter and receiver setup (e.g., m*n MIMO) respectively. The speakers send periodic frames which then gets reflected by the moving hand/finger and these reflected signals are received by the microphones. In the present disclosure, <NUM>*<NUM> MIMO setup is implemented by the system and method described herein. Such MIMO setup shall not be construed as limiting the scope of the present disclosure. In other words, there can be as many number of speakers and as many number of microphones implemented in the system <NUM> of FIGS as shown. It is to be further understood that based on the number of speakers and microphones implemented, m*n CIR images (or CIR coefficients) are estimated. In the present disclosure's transceiver design, to help in better estimation of the CIR, the system <NUM> transmit different sequences of same length having small cross correlation from the speakers. While one can use the well-known spreading sequences that are widely used in multi-user CDMA communication, here in the method of the present disclosure, the system employs Gold codes (e.g., refer "<NPL>") which exhibit good correlation properties. This helps in reducing the interference and thus aids in better estimation of the channel. In the transceiver of the system <NUM>, Gold codes are used that are generated by performing XOR operation on two pseudo random sequences having a length of <NUM> symbols (e.g., also referred as a length of predefined symbols and interchangeably used herein). The generator polynomials for the two sequences are z<NUM> + z<NUM> + z<NUM> + z<NUM> + <NUM> and z<NUM> + z<NUM> + <NUM> respectively. Let ci(n), i = {<NUM>, <NUM>} denote the two different sequences of length <NUM>, which are interpolated by 'p' times (e.g., p=<NUM> times in the present disclosure and such number of interpolations shall not be construed as limiting the scope of the present disclosure), followed by low pass filtering of 'k' kHz (e.g., k=<NUM> and such frequency application for filtering shall not be construed as limiting the scope of the present disclosure). Further to reduce the effect of inter frame interference, the sequence is appended with zeros such that the total length of each frame is of duration 'y' ms (e.g., <NUM> comprising <NUM> samples - such padding application with zeros shall not be construed as limiting the scope of the present disclosure). Let this filtered, up sampled/interpolated, and zero padded signal be denoted by xi(n), which is then modulated by a carrier of fc = q kHz (e.g., q=<NUM> and such modulating application with <NUM> shall not be construed as limiting the scope of the present disclosure) by multiplying it with <MAT>. The above upsampling, filtering and modulating operations are performed to restrict the acoustic transmission band between <NUM> (e.g., the first pre-defined acoustic transmission band) to <NUM> (e.g., the second first pre-defined acoustic transmission band). This acoustic band is chosen because it is almost inaudible to human ears and further, most commercially available speakers and microphones show good frequency response in this band. Each frame comprising of these <NUM> samples is looped back and played through the speakers at a sampling rate of 'r' kHz (e.g., r=<NUM> and such sampling rate of <NUM> shall not be construed as limiting the scope of the present disclosure. These transmitted modulated signals get reflected from the hand and the reflected signals are captured by the microphones. As a first step, at each microphone as shown in <FIG>, the system <NUM> applies the quadrature demodulation by multiplying it with the signal <MAT>, <MAT>. The demodulated signal(s) is/are then filtered with a <NUM> low pass filter to obtain the complex baseband signals, yj(n), j = <NUM>, <NUM>, which are received at the two microphones (e.g., also referred the plurality of reflected signals and interchangeably used herein). The transmitted and the received sequence xi(n) and yj(n), i,j = <NUM>, <NUM> respectively (also referred as the plurality of modulated signals/the plurality of transmitted modulated signals and the plurality of reflected signals are used for CIR estimation.

In this regard, at step <NUM> of the present disclosure, the one or more hardware processors <NUM> process, via a Channel Impulse Response (CIR) estimator, the plurality of transmitted modulated signals and the plurality of reflected signals to obtain a plurality of CIR images. The plurality of transmitted modulated signals and the plurality of reflected signals are processed by the CIR estimator (or also referred as CIR estimation block) wherein a plurality of CIR coefficients are estimated based on the plurality of transmitted modulated signals, and the plurality of reflected signals using a sparsity prior serving as a constraint and the plurality of CIR coefficients are concatenated to obtain the plurality of CIR images. The above step of processing the plurality of transmitted modulated signals and the plurality of reflected signals via the CIR estimator to obtain the plurality of CIR images is better understood by way of following description:.

The reflected signal(s) from the hand comprise of multiple reflections from different points depending upon the gesture and thus can aptly be modeled by a multipath channel. This multipath channel is characterized by an L tap finite impulse response filter. The received signal at the jth microphone, yj(n) can be expressed as: <MAT> where <MAT> denotes the L tap CIR of the reflected signal(s) from ith speaker to the jth microphone and ηj(n) denotes the additive white Gaussian noise. Addition of or introduction to white Gaussian noise is optional. In the present disclosure, the system <NUM> considered the total number of channel taps L to be <NUM> which approximately translates to <NUM>. The above equation can be represented in the following matrix as: <MAT> where for any i,j = <NUM>, <NUM>, h<NUM>j, h<NUM>j herein referred as hj, the received signal(s) yj = [yj(<NUM>),yj(<NUM>),yj(P - <NUM>)]T and Xi is a matrix of dimension P × L which can be expressed as: <MAT>.

The value of P is chosen such that P + L = <NUM> which corresponds to the length of each transmitted frame. hj denotes the CIR at a particular time index. In other words, hj is the CIR coefficient (also referred as coefficient and interchangeably used herein) being estimated at a particular time index. To estimate CIR using equation (<NUM>), a simple least square similar can be employed. But it is now important to observe from <FIG>, which provides a CIR illustration corresponding to few gestures, that the CIR at any given particular time index is sparse in nature. More specifically, <FIG>, with reference to <FIG>, depicts <NUM> CIR images corresponding to two complimentary (a) anticlockwise and (b) clockwise finger rotation respectively, in accordance with an embodiment of the present disclosure. In other words, only few taps in hj tend to be significant, while most of them can be neglected. Hence, in the present disclosure, the system <NUM> estimates the CIR images by solving the following optimization: <MAT>.

The regularizer ∥hj∥<NUM> is introduced since it is well known that l<NUM>-norm promotes sparse solution, wherein the sparse solution is also referred as sparsity prior serving as the constraint, and λ is a hyper-parameter which controls between mean square error (MSR) and the sparsity prior serving as the constraint. The above equation can be solved by using iterative shrinkage threshold algorithm (ISTA) (e.g., refer "<NPL>. ") whose (k + <NUM>)th iterative update is given as: <MAT> where for any y, γ, soft(y, γ) = sign(y)max (<NUM>, |y| - γ) and α is the learning rate. On implementation, the system <NUM> observed that for most instances, the above solution was converging with less than <NUM> iterations. However, for efficient implementation, the system <NUM> has used the unfolded variant of ISTA, LISTA (learned iterative shrinkage threshold algorithm) comprising <NUM> layers with appropriate training. <MAT> is found using corresponding <MAT>, from which the four CIRs <MAT>, i, j = <NUM>, <NUM> can easily be separated. This CIR estimation is repeated for every <NUM> i.e., corresponding to the length of each transmitted frame. By concatenating the CIR (or CIR coefficients) at every time index and considering only the magnitude, the CIR images are obtained as shown in <FIG>. Since the system <NUM> employs a <NUM>*<NUM> MIMO like setup, it is highly unlikely that all channels (e.g., <NUM> channels) would be in deep fade, hence making it robust against the ill effects of FSF. Further, is it noticed from <FIG> that the distinctive <NUM> CIR images correspond to two complimentary clockwise and anti-clockwise finger rotations. These distinctive images shall be used for gesture detection. In other words, at step <NUM> of the present disclosure, the one or more hardware processors <NUM> recognize, via a Spiking Neural Network (SNN), a gesture performed by the user based on the plurality of CIR images. In an embodiment, the step of recognizing, via the Spiking Neural Network (SNN), the gesture performed by the user based on the plurality of CIR images comprises converting the plurality of CIR images into a spike domain; extracting, one or more features of the spike-domain using one or more spiking neurons comprised in the SNN; and recognizing the gesture performed by the user from the extracted one or more features by using the SNN. In an embodiment, the Spiking Neural Network is obtained by training a Convolutional Neural Network (CNN) using training data comprising a plurality of CIR images corresponding to one or more users to obtain a trained CNN; quantizing the trained CNN to obtain a quantized CNN; and converting the quantized CNN to the SNN. The training of CNN and obtaining SNN are better understood by way of following description:.

A <NUM>-layered convolutional architecture comprising of three convolution layers and two fully connected layers, as shown in <FIG>, is designed to train on four-CIR images (also referred as <NUM>-channel CIR images and interchangeably used herein). More specifically, <FIG>, with reference to <FIG>, depicts an exemplary <NUM>-Layer Convolutional Neural Network (CNN) architecture for gesture classification with <NUM> CIR images, in accordance with an embodiment of the present disclosure. Each convolution layer learns a set of filters of sizes 7x7, 5x5, and 3x3 that are capable of extracting features from a 2D-receptive field (e.g., inner rectangular/square window) of their respective inputs. After each convolution layer, a max-pooling layer with kernel 4x4 is introduced that imparts some degree of generalization and translation invariance to the extracted features. Moreover, it helps down sampling the input space. Of the next two dense fully connected layers, the last one is the final classification output layer giving the gesture probabilities. Both dense layers learn the mapping between the spatial features and gesture probabilities. ReLu activation is used for both convolution and dense layers except for the final output layer which uses a SoftMax function. Dropout is used for regularizing the network with a drop-off probability of <NUM>. Categorical cross entropy loss is employed to guide the gradient descent for estimation of optimal network weights as shown in the following equation (<NUM>), where cn denotes the actual probability of nth gesture class occurrence and ĉn is the SoftMax output from the output layer of CNN.

Next, Quantization is performed on the trained CNN in order to reduce its memory footprint and improve its latency. The system <NUM> applies weight and activation quantization from single-precision floating point (<NUM>-bits) to byte sized unsigned integers (<NUM>-bits). This is done using binning the <NUM>-bit floating point range into <NUM> unique values. Quantization Aware Training (QAT) which is the optimal way to estimate these bin values from training data, is used here. It is to be understood by a person having ordinary skill in the art or person skilled in the art that such use of <NUM>-layer CNN shall not be construed as limiting the scope of the present disclosure.

Finally, the quantized CNN is converted into an equivalent SNN. This is done by approximately matching the output of an ANN neuron to the firing rate of a spiking neuron. Here, a corresponding SNN network is constructed using Integrate- and-Fire (IF) neuron model as shown in the following equation (<NUM>). <MAT> Vl(t) represents the membrane potential vector of the spiking neurons at time t in layer <NUM>, Il(t) represents the residual potential vector at time t and sl(t) represents the spiking activity of the neurons where Vth is the threshold potential of the spiking neurons. <IMG> is the Heavyside Step function, bl gives the bias term for the neurons of layer l and u is a vector comprising of all ones. The membrane potential of the IF neuron models is modified as shown in equation (<NUM>) below to reduce the error in the approximation of ReLU activation with firing rate.

Weights for each layer are normalized with 99th-percentile value of ReLU activations of that layer as shown in equation (<NUM>) below, where λl represents the 99th-percentile value of ReLU activations in l-th layer during training.

Softmax function is applied to the membrane potentials of the final output layer in the converted SNN, and the resultant values are treated as the probability of occurrence of corresponding gesture class. Max pooling layers are implemented in the converted SNN by means of a Hard Winner-Take-All mechanism (among neurons in the pooling window) where the neuron which spikes first, inhibits all the other neuron in its window from activating. For Max pooling, instead of directly solving maximum activity problem in spike domain, the system <NUM> approximates maximum spiking neuron with first spiking neuron. In the testing phase, to test the performance of the converted multi-layer SNN, the four CIR images in the real-valued space need to be encoded into spike domain before being fed to the SNN. The system and method of the present disclosure used a rate-based Poisson encoding scheme which treats the real-value as the rate of a Poisson process. Thus, for each CIR channel pixel value, an independent spike train containing the information in the form of firing rate is obtained. These spike trains can be directly fed to the input layer of the converted SNN to obtain predicted gesture probabilities.

The system and method of the present disclosure have collected data for <NUM> gestures from <NUM> different subjects using the inbuilt speakers and microphones of DELL® Precision laptop. The gestures considered for experiments by the system <NUM> are taken from Ling et al. (e.g., refer "<NPL>. ") and is shown in <FIG>. More specifically, <FIG>, with reference to <FIG>, depicts a plurality of gestures (e.g., <NUM> gestures) considered along with a corresponding CIR image, in accordance with an embodiment of the present disclosure. Each subject performs individual gesture <NUM> times. For the purpose of comparison, the system <NUM> chose technique from Ling et al. because it has been shown in <NPL>. ) that Ling et al. performed better than most other competing methods. Hence, from the recorded reflections, CIR images of each gesture are estimated by the system <NUM> for the following two cases: i) Dataset of the present disclosure: Four CIR images, using <NUM>-layer LISTA, ii) Ultragesture dataset: two CIR images using LS (Least Square) approach to mimic the setup of Ling et al. (one speaker and two microphones). Further, <FIG> also shows the distinctive one channel CIR image (h22) corresponding to different gestures.

First, the system and method of the present disclosure provide a comparison between the quality of CIR images estimated with the proposed sparsity prior based approach and the LS based approach that are used in state-of-the-art methods (e.g., refer "<NPL>"). For the sake of illustration, <FIG> shows only one channel CIR image estimated using both the aforementioned approaches. It is noticed/observed from <FIG> that the least square based approach provides a degraded CIR image compared to the LISTA based approach of the present disclosure, which thus aids in better gesture detection. More specifically, <FIG>, with reference to <FIG>, depicts CIR images obtained (a) using the UltraGesture (conventional approach), and (b) with the SNN as implemented by the system <NUM> of <FIG> for a push-pull gesture, in accordance with an embodiment of the present disclosure. Next, results for gesture detection are provided herein by the system and method of the present disclosure. For each gesture, for both the above-mentioned datasets, the ratio of training and test data is <NUM>:<NUM>. The CNN network has been created on Python <NUM> using TensorFlow v2. <NUM> and has been tested on an <NUM> GB Nvidia Turing Architecture GPU. ANN-to-SNN conversion and the simulated run of SNN was also done on the same system using TensorFlow.

The system <NUM> and the method of the present disclosure have tested the classification performance using (i) the trained CNN, (ii) the Quantized CNN and (iii) the converted SNN. As shown in Table <NUM>, the mean testing accuracy for SNN as implemented by the system <NUM> with these three networks are <NUM>%, <NUM>% and <NUM>% respectively. Each of these accuracy values is higher than corresponding accuracy values obtained for Ultragesture dataset, thanks to better CIR estimation and the robustness to ill effects of fading due to MIMO like setup in SNN as implemented by the system <NUM> and the method of the present disclosure.

The active power consumption of a neuromorphic hardware is mainly contributed by the spiking network's total number of synaptic operations (SOP). Following (<NUM>) and the method mentioned in Sorbaro et al. (e.g., refer "<NPL>. "), total number of synaptic operation for the SNN of the system <NUM> is found to be ~<NUM> while that for the CNN is ~<NUM> (considering matrix multiplication only). This converted SNN can be implemented on neuromorphic platforms such as Brainchip Akida (e.g., refer "<NPL>"), Intel® Loihi (e.g., refer "<NPL>. "), etc. to achieve further power benefit (~100x). <FIG> shows the confusion matrix for converted SNN for <NUM> gesture classes. More specifically, <FIG>, with reference to <FIG>, depicts a confusion matrix for Gest-SNN dataset (SNN dataset as used by the present disclosure), in accordance with an embodiment of the present disclosure. Though most of the classes are correctly classified, Click and Double Click being single point gestures, are sometimes confused by the network. Also, the mean value of the class wise Average Precision (AP) is found to be <NUM> and mean of Average Recall (AR) is found to be <NUM> for the SNN of the system <NUM>.

The system and method of the present disclosure implemented an ultrasound based system or acoustic system which uses CIR image and SNN for gesture classification providing an improvement of <NUM>% over existing state-of-the-art. The system leverages the MIMO diversity by using a plurality of speakers and microphones and estimates the CIR with the assumption of sparsity. Use of SNN, created via ANN-to-SNN conversion on a trained <NUM>-layer CNN, brings in energy benefit, thanks to lesser number of operations. From these results, it can be concluded that SNN as implemented by the system and method of the present disclosure is a good alternative, frugal and robust gesture detection system compatible for deployment on resource constrained edge platforms.

It is to be understood that the scope of the protection is extended to such a program and in addition to a computer-readable means having a message therein; such computer-readable storage means contain program-code means for implementation of one or more steps of the method, when the program runs on system according to claim <NUM>.

Claim 1:
A processor implemented method, comprising:
transmitting, via a plurality of speakers, a plurality of modulated signals to a user (<NUM>);
receiving, via a plurality of microphones, a plurality of reflected signals from the user, in response to the plurality of transmitted modulated signals (<NUM>);
processing, via a Channel Impulse Response, CIR, estimator, the plurality of transmitted modulated signals and the plurality of reflected signals using a sparsity prior serving as a constraint to obtain a plurality of CIR images (<NUM>), wherein processing the plurality of transmitted modulated signals and the plurality of reflected signals using the sparsity prior serving as a constraint comprises:estimating the plurality of CIR images by solving the optimization equation using iterative shrinkage threshold algorithm, ISTA, given by: <MAT> wherein hj is a tap CIR of the reflected signal from the jth microphone, yj is received signal at the jth microphone, X is a matrix of dimension P × L , wherein L denotes number of taps and P is chosen such that P+L=<NUM>, and λ is a hyper-parameter which controls between mean square error, MSR, and the sparsity prior serving as the constraint; and
recognizing, via a Spiking Neural Network, SNN, a gesture performed by the user based on the plurality of CIR images (<NUM>), wherein the Spiking Neural Network is obtained by: training a Convolutional Neural Network, CNN, using training data comprising a plurality of CIR images corresponding to one or more users to obtain a trained CNN;
quantizing the trained CNN to obtain a quantized CNN; and
converting the quantized CNN to the SNN, and
wherein the quantized CNN is converted to the SNN by performing an approximate matching of a corresponding output of an CNN neuron comprised in the CNN to a firing rate of a spiking neuron comprised in the SNN.