Encoding Event-Based Gesture Data With a Hybrid SNN Guided Variational Auto-encoder
Authors: Kenneth Michael Stewart
Andreea Danielescu
Timothy Shea
Emre Neftci
Venue: ICLR 2022 Submitted (2022)
Keywords: Neuromorphic Computing
Variational Auto-encoders
Representation Learning
Spiking Neural Networks
Self-supervised Learning

Abstract: Commercial mid-air gesture recognition systems have existed for at least a decade, but they have not become a widespread method of interacting with machines. These systems require rigid, dramatic gestures to be performed for accurate recognition that can be fatiguing and unnatural. To address this limitation, we propose a neuromorphic gesture analysis system which encodes event-based gesture data at high temporal resolution. Our novel approach consists of an event-based guided Variational Autoencoder (VAE) which encodes event-based data sensed by a Dynamic Vision Sensor (DVS) into a latent space representation suitable to compute the similarity of mid-air gesture data. We show that the Hybrid Guided-VAE achieves 87% classification accuracy on the DVSGesture dataset and it can encode the sparse, noisy inputs into an interpretable latent space representation, visualized through T-SNE plots. We also implement the encoder component of the model on neuromorphic hardware and discuss the potential for our algorithm to enable real-time, self-supervised learning of natural mid-air gestures.

Under review as a conference paper at ICLR 2022
ENCODING EVENT-BASED GESTURE DATA WITH A HYBRID SNN GUIDED VARIATIONAL AUTO-ENCODER
Anonymous authors Paper under double-blind review
ABSTRACT
Commercial mid-air gesture recognition systems have existed for at least decade, but they have not become a widespread method of interacting with machines. These systems require rigid, dramatic gestures to be performed for accurate recog- nition that can be fatiguing and unnatural. To address this limitation, pose a neuromorphic gesture analysis system which encodes event-based gesture data at high temporal resolution. Our novel approach consists of an event-based guided Variational Autoencoder (VAB) which encodes event-based data sensed by a Dynamic Vision Sensor (DVS) into a latent space representation suitable to compute the similarity of mid-air gesture data. We show that the Hybrid Guided- the DVSGesture dataset and it
pro-
VAE achieves 87% classification accuracy encode the sparse, noisy inputs into an interpretable latent space representation. visualized through T-SNE plots. We also implement the encoder component of the model neuromorphic hardware and discuss the potential for our algorithm to enable real-time, self-supervised learning of natural mid-air gestures.
INTRODUCTION
Computers that interact with users through text and spoken language have become ubiquitous. However, people also communicate a great deal of information through body language and mid- air gestures, which are often closely coupled with speech (Fröhlich et al. 2019). This mode of interaction is beneficial in a variety of human computer interaction applications because it is natural, touchless, and efficient. For example, mid-air gestures can be used to interact with car infotainment systems, making interaction safer by eliminating the need for drivers to look at a touch interface instead of the road while driving (Young et al., 2020 May et al. 2017). Mid-air gesture interaction can also be beneficial in high-use public areas, reducing the need for touch based systems which can transmit pathogens from person to person. Additionally, interactive systems that leverage mid-air gestures are more accessible for people with hearing speech impairments than other touchless alternatives, such as voice (Ballati et al. 2018).
Despite numerous potential applications and recent progress in sensors capable of detecting mid-air gestures (Lichtsteiner et al. 2008b Zhang 2012 Keselman et al. 2017 Wang et al. (2016), accurate recognition remains a challenge. Changing backgrounds and lighting conditions, wide variation in how users perform the same gesture, and a need for large data sets are just a few of the reasons that accurate recognition in real-world environments is difficult. Requiring that users perform prede- fined, rigid gestures is not a solution, as people have difficulty reproducing and remembering rigid, precise movements (Hakim et al. 2019). A reliably accurate gesture recognition system needs to address these issues by recognizing natural gestures despite their wide ranging variability without relying on large data sets recorded users' homes and sent to the cloud, which would be invasive and a security risk for home owners and their families. One way to address these issues is to adapt in real-time user's individual gestures by measuring the similarity of incoming gestures to exist- ing gesture classes. On the sensing side, Neuromorphic Dynamic Vision Sensors (DVS) inspired by the biological retina are well suited to this task (Brandli et al. 2014) because they capture tem- poral, pixel-wise intensity changes approach has key advantages over traditional RGB cameras, such as faster response times, better temporal resolution, and invariance to static image features like lighting and background. Thus, raw DVS sensor data intrinsically emphasizes the dynamic movements that comprise most natural gestures. However, effectively processing DVS event streams remains an open challenge. Events are asynchronous and spatially sparse, making it challenging to directly apply conventional vision algorithms (Gallego et al. 2020 Gallego et al. 2019)
sparse stream of binary events (Gallego et al.) 2019). This

---

Under review as a conference paper at ICLR 2022
Spling Narra Natwork Ercedkr
in: Loss Exc. Lesa
Pyes
TW
p
CNN Deccdor
VAE
Figure I: The Hybrid Guided- VAE architecture. Streams of gesture events recorded using a Dy- namic Vision Sensor (DVS) are input into Spiking Neural Network (SNN) that encodes the spatio- temporal features of the input data into a latent structure 2. P and Q are pre-synaptic traces and U is the membrane potential of the spiking neuron. For clarity, only a single layer of the SNN is shown here and refractory states I are omitted. To help disentangle the latent space, portion of the 2 equal to the number of target features y" is input into a classifier that trains each latent variable to encode these features (Exc. Loss). The remaining 2, noted (m are input into different classi- fier that adversarially trains the latent variables to not encode the target features so they encode for other features instead (Inh. Loss). The latent state is decoded back into x: using the conventional deconvolutional decoder layers.
Spiking Neural Networks (SNNs) can efficiently process and learn from event-based data while taking advantage of temporal information (Neftci et al.] 2019). SNN models inspired by the biological cortex and can be used for hierarchical feature extraction from the precise timing of events through event-by cent-by-event processing (Gerstner et al. 2014). Recent work demonstrated how SNNs can be trained end-to-end using gradient backpropagation in time and standard autodifferentiation tools, making the integration of SNNs possible as part of modern machine learning and deep learning methods Zenke & Neftci, 2021 Bellec et al. 2019 Shrestha & Orchard 2018
Here, take advantage of this capability by incorporating a convolutional SNN into Variational Autoencoder (VAE) to encode spatio-temporal streams of events recorded by the DVS (figure []. The goal of the VAE is to embed the streams of DVS events into a latent space which facilitates the evaluation of gesture similarity. To best use the underlying hardware, we implement to process the DVS data, with an SNN-based encoder and a conventional (non-spiking) convolutional network decoder. To ensure the latent space represents features which are perceptually salient and useful for gesture recognition, gesture variation. the Appendix we show an ablation study that demonstrates the capabilities of the guided VAE to disentangle features of variation over other methods.
hybrid VAE
guided VAE to disentangle the features that account for
Our Hybrid Guided- VAE encodes the gesture data in a way that allows automatically measure and analyze the similarity of gestures, allowing us to cluster similar gestures and assign pseudo- labels to novel ones. The key contributions of this work are:
1. End-to-end trainable event- based SNNs for processing neuromorphic sensor data event-by- event and embedding them in latent space.
2. A Hybrid Guided-VAB that encodes event-based camera data in latent space representa- tion of salient gesture features for clustering and pseudo-labeling.
3. A groof-of-concepp implementation of the Hybrid Guided- VAE on Intel's Loihi Neuromor- phic Research Processor.
The ability to encode gestures into a disentangled latent representation from DVS data is a key feature to enable mid-air gesture recognition systems that are less rigid and more natural because they can adapt to each user.
2

---

Under review as a conference paper at ICLR 2022
2
RELATED WORK
2.1 MEASURING GESTURE SIMILARITY
Mid-airge gesture data are obtained through gesture elicitation studies, the goal of which is to collect user-defined gestures and then group similar gestures together to determine the gestural language for an application. One way the language is determined is by clustering gestures together based on a set of criteria (Ali et al. 2018) (Villrreal-Narvazz et al., (020). Examples of criteria used include, but are not limited to, which body part is used, body part trajectory, and body pose. Clustering is often done manually with people assigning labels and providing explanations as to why they labeled a gesture a certain way based on predefined clustering criteria. In this way, gestures frequently used for a particular interaction can be identified. There are several issues with this approach. The clus- tering criteria used evaluate which gestures are similar are chosen subjectively by researchers and therefore vary from study to study (Connell et al., (213a) (Connell et al.) 2013b) (Silpasuwanchai & Ren 2014) (Vatavu 2012). Using subjective criteria contrasts with the goal of gesture elicita- tion studies, which is find an objective consensus among the elicited gestures. The class labels given to the mid air gestures are also subjective and can vary across cultures. After clustering, a similarity score is calculated. with researchers continuously refining the appropriate similarity score to use (Wobbrock al. 2009) (Vatavu & Wobbrock 2015) (Tsandilas) 2018) (Vatavu) 2019). Ad- ditionally, because human labelers are needed, the process is tedious and expensive, which is why researchers are turning to crowd sourcing to reduce the time taken to cluster gesture data (Ali et al., 2018 (Ali et al. 2019).
To circumvent these issues, our work seeks to automate the process of computing gesture similar- ity for clustering. With recent advances in deep learning. the process of class labeling for image classification and object recognition tasks can be automated (Bah al.) (018) or semi-automated PPagdettoosapol et al. 2020). Because mid air gesture data can be collected in the form of images or video, deep learning can be used for automatic gesture class labeling by learning similarity mea- sure between the gestures. However, is still challenging problem because salient image features may be missing in a scene or can be mistaken as part of the background. Additionally, mid air ges- tures are dynamic movements, for accurate recognition, which can be computationally expensive. Here, we automate the process of clustering mid air gestures for labeling using spatio-temporal event-based data streamed from a DVS. To automatically and objectively measure the similarity of gestures between and across classes for clustering and automatic labeling, we build novel Hybrid Guided-VAF model that can take advantage of the DVS' temporal resolution and robustness while being end-to-end trainable using gradient descent variational objective. Hybrid VAEs that combine both spiking and ANN layers have been used before on DVS event data for predicting optical flow, with the hybrid architecture efficiently processes the sparse spatio-temporal event inputs while preserving the spatio-temporal nature of the events (Lee et al. 2020
apatio-tempone
information (i.e. video) processing is necessary
2.2 VARIATIONAL AUTOENCODERS
VAEs are type of generative model which deal with models of distributions p(z), defined over data points x E X (Kingma & Welling, 2013). A VAE commonly consists of two networks, 1) an encoder (Enc) that encodes the captured dependencies of a data sample x into a latent representation 2; and 2) decoder (Dec) that decodes the latent representation back to the data space making reconstruction ã. Using Gaussian assumptions for the latent space:
Enc(x) = q(>l>)= = (=)p(()), E(x)).
(1)
3 Dec(z) p(z(z),
(2)
where q is the encoding probability model into latent states 2 that are likely to produce x, and p is the decoding probability model conditioned 2. The functions yuxx) and E(x) are deterministic functions whose parameters can be trained through gradient-based optimization. Using variational approach, the VAE loss consists of the sum of two terms resulting from the variational lower bound:
log p(x) Egnn log log p(alz) DKL( (x(2[)))))))(
(3)
Lu
Eprior
The first term is the expected log likelihood of the reconstructed data computed using samples of the latent space, and the second term acts as a prior, where Dx is the Kullback-Leiie divergence. The VAE loss is thus formulated to maximize the variational lower bound by maximizing - -yrrior and Lu. VAE's latent space captures salient information for representing the data X, and thus similarities in the data (Larsen al. 2016).
3

---

Under review as a conference paper at ICLR 2022
2.3 DISENTANGLING VARIATIONAL AUTOENCODERS
VAEs do not necessarily disentangle all the factors of variation, which can make the latent space dif- ficult to interpret and use. Several approaches have been developed to improve the disentangling of the latent representation, such as Beta VAE (Higgins et al. 2017) and Total Correlation VAE (Chen) et al., 2018). For this reason, we employ Guided-VAE, which has been developed specifically to disentangle the latent space representation of key features supervised fashion (Ding et al,, 2020). We describe here the supervised Guided-VAL algorithm, which is the basis of our hybrid model described in the next section.
To learn a disentangled representation, supervised Guided- VAE trains latent variables to encode existing ground-truth labels while making the rest of the latent variables uncorrelated with that label. The supervised Guided-VAE model targets the generic generative modeling task by using an adversarial excitation and inhibition formulation. This is achieved by minimizing the discriminative loss for the desired latent variable while maximizing the minimal classification error for the rest of the latent variables. For N training data samples X (#1...., and M features with ground- truth labels, let
(Z1, 2m, ... @ P/m where the 2m define the "guided" latent variable capturing feature m, and 2/m represents the rest of the latent variables. Let ym (m(xn) be a one-hot vector representing the ground-truth label for the m-th feature of sample In. For each feature m,
the excitation and inhibition losses are defined as follows:
LEne (2, m. max Ce
E B (z..) Peg Pem (y
Um (In) (mm)) ((m))))m))
(4)
Lint m) Rm
max
E (wn) log (y = Ym (mn()n))emm))
where Gn a classifier making a prediction on the m-th feature in the guided space and km is classifier making a prediction over in the unguided space 2/m- By training these classifiers adversarially with the VAE's encoder, it learns to disentangle the latent representation, with &m
representing the target features and Am representing any features other than the target features.
METHODS
3.1 DYNAMIC VISION SENSORS AND PREPROCESSING
Dynamic Vision Sensors (DVS) are type of event-based sensor that record event streams at a high temporal resolution and are compatible with SNNs (Liu Delbruck 2010). Inspired by the human retina, DVS sensors detect brightness changes on logarithmic scale with a user-tunable threshold, instead of RGB pixels like typical cameras. An event consists of its location x, y. timestamp 1, and the polarity p E [OFF, ON representing the direction of change. Using a dense representation, the DVS event stream is denoted S. N+ indicating the number of events that occurred in the time bin (t,t + At) with space-time coordinates (z.y.p.f). At is the temporal discretization and equal Ims in our experiments. The recorded DVS event stream is provided as input to the Hybrid Guided VAE network implemented on a GPU (network dynamics described in the following section). Each polarity of the event stream (z. y- P) is fed onto one of the two channels of the first convolutional layer. Within the time step. most pixels have value zero, and very few have values larger than two. Note that time is not represented as a separate dimension in the convolutional layer, but through the dynamics of the SNN.
The VAE targets take different form because the decoder is not SNN. Time surfaces (TS) widely used to preprocess event-based spatio-temporal features (Lagorce et al., (017). TS can be constructed by convolving an exponential decay kernel through time in the event stream as follows:
Tst a.y.P = c * DVS.a with c' -
(5)
where 7 is time constant. Here, we convolve over the time length of the input gesture data stream S S'S..... This results in two 2D images, one for each polarity, that are used as VAE targets( i.e., for the reconstruction loss).
4

---

Under review as a conference paper at ICLR 2022
3.2 HYBRID GUIDED VARIATIONAL AUTO-ENCODER
Gestures recorded using DVS camera produce streams of events containing rich spatio-temporal patterns of the gesture. Event-based computer vision algorithms typically extract hand encoded statistics and use these in their models. While efficient, this approach discards important spatio- temporal features from the data (Gallego et al. 2019).
Rather than manually selecting feature set, we process the raw DVS events while preserving key spatio-temporal features using a spiking neural network (SNN) trained end-to-end in the Hybrid Guided-VAR architecture shown in Figure
key advantage of the VAE is that the loss can be optimized using gradient backpropagation. To retain this advantage in our hybrid VAE, we must ensure that the encoder SNN is also trainable through gradient descent. Until recently, several challenges hindered this: the spiking nature of neu- rons' nonlinearity makes it cindiffferentiblee and the continuous-time dynamics raise a challenging temporal credit assignment problem. These challenges are solved by the surrogate gradients ap- proach (Neftci al., 2019), which formulates the SNN as an equivalent binary RNN, and employs a smooth surrogate network for the purposes of computing the gradients. Our Hybrid Guided-Vat VAE uses a convolutional SNN to encode the spatio-temporal streams in the latent space, and non- spiking convolutional decoder reconstruct the TS of the data. We chose because the SNN can bridge computational time scales by extracting slow and relevant factors of variation in the gesture (Wiskott & Sejnowski 2002) from fast event streams recorded by the DVS. We chose a conventional (non-spiking) decoder for three reasons: (1) for gesture similarity esti- mation, we are mainly interested in the latent structure produced by the encoder, rather than the generative features of the network, (2) as we demonstrate in the results section, a dedicated neuro- morphic processor (Indiveri et al. 2011 Davies et al.) 2018) only requires the encoder to produce this latent structure, and (3) SNN training is compute- and memory- intensive. Thus a conventional decoder enables us to dedicate more resources to the SNN encoder.
wvent-based encoder
Our Hybrid Guided- VAE network architecture is shown in Figure the architecture description provided in T Descriptions of the excitatory and inhibitory networks necessary for "guid- ing" of the VAE are provided in the Appendix. The SNN encoder consists of four discrete leaky integrate and fire convolutional layers (see following section) followed by linear layers, and outputs a pair vectors (j. 2) for sampling the latent state z. According to the guided VAE, part of or all of latent state z is inputed into one of three connected networks, the excitation classifier, the adversarial inhibition classifier,
the decoder. The target features for the excitatory network given as one-hot encoded vectors of length M. The excitation classifier is jointly trained with the encoder to train the first M latent variables to only encode information relevant to the corresponding target feature. The inhibition classifier takes as input the remaining latent variables in the latent space, mm. and are adversarially trained on two sets of targets. One set of targets are the same target features that the excitation classifier trains on. The other set of target features is vector of length M but all of the values are set to 0.5 indicating that none of the values correspond to any target. The inhibition classifier is jointly trained with the encoder to train the remaining 2,m latent variables to not encode any information relevant to target features forcing them to instead encode information for other features. The decoder is transposed convolutional network that takes the full latent state as input to construct the TS, denoted ā in Figure
Table I: Hybrid Guided- VAE architecture
Layer Input
Okapui
Layer Type
12x
DYSTER SNN LIF Enceler
Za
16x16x22
32c7ptsl
16x 16x36
la
16x 16x32
64c7ptst
16x166x64
64c7p0k1
8x8x644
7
la
8x8x64
8
ppsc
8x
9
la
8x 128
10
128
PIO JANN)
12
100
E(U") (ANN)
13
128
ANN Decader
14
125c/p0s2
4x4xx 128
15
64c4p1s2
8x8xx6
16
32c4p1s2
16x32
ompрt
204p1c2
32x32xx2
Surface
Ya represents YxY sum pooling, Xcxp2s8 represents x convolution filters (XXY) with pudding
5

---

Under review as a conference paper at ICLR 2022
3.3 ENCODER SNN DYNAMICS
To take full advantage of the event-based nature of the DVS input stream and its rich temporal features, the data is encoded using an SNN. SNNs be formulated as a type of recurrent neural network with binary activation functions (Figure [T] (Neftci et al. 2019). With this formulation, SNN training can be carried out using standard tools of autodiflerentiation. In particular. to best match the dynamics of existing digital neuromorphic hardware implementing SNNs (Davies al. 2018 Furber et al. 2014), our neuron model consists of a discretized Leaky Integrate and Fire (LIF) neuron model with time step At (Kaiser et al. 20200:
U.= -
WW Wop UnA Rt + bi,
PJta aPj + (1 (1- =)
(6)
Ofts1 BQ} + (1 -)S
=
where the constants a exp( Timm
and
exp(
AL reflect the decay dynamics of the Tayn
membrane potential and the synaptic state during a At timestep. where Tmem and Tayn are membrane and synaptic time constants, respectively. The time step in experiments was fixed to At Ims. R2 here implements the reset and refractory period of the neuron (with dynamcis similar P), and states P., Qi are pre-synaptic traces that capture the leaky dynamics of the membrane potential and the synaptic currents. S = OUU)) represents the spiking non-linearity, computed using the unit step function, where (U;) = 0 if U; < Unh otherwise 1. We distinguish here the input spike train S from the output spike train St. Following the surrogate gradient approach (Neftci et al. 2019), for the purposes of computing the gradient, the derivative of θ is replaced with the derivative of the fast sigmoid function ZZnkee & Ganguli, 2017). Note that equation (6) is equivalent to a discrete-time version of the spike response model with linear filters (Gerstner Kistler, 2002). Similar networks were used for classification tasks on the DVSGesture dataset, leading to state-of-the-art accuracy on that task (Kaiser et al.) 2020 [Shrestha& & Orchard (2018). In the Appendix, we show how these equations can be obtained via discretization of the common LIF neuron.
The SNN follows a convolutional architecture, as described in Table encoding the input sequence St. into a membrane potential variable U in the final layer. The network computes uu(²) and X(U') as in a conventional VAE, but uses the final membrane potential state U². Thanks to the chosen neural dynamics, the TS can be naturally computed by our network. In fact, using appro- priate choice of = Tsyn for computing the TS, it becomes exactly equivalent to the pre-synaptic trace Q' of our network (See Appendix). Hence, our choice of input and target corresponds to an autoencoder in the space of pre-synaptic traces Q'
3.4 DATASETS
We trained and evaluated the model using the Neuromorphic MNIST (NMNIST) and IBM DVS- Gesture datasets, both of which were collected using DVS sensors (Lichtsteiner et al., 2008a Posch jet al., 2011). NMNIST consists of 32 32, 300ms event data streams of MNIST images recorded with a DVS (Orchard et al. 2015). The dataset contains 60,000 training event streams and 10,000 test event streams.
The IBM DVSGesture dataset (Amir al. 2017) consists of recordings of 29 different individuals performing 10 different gestures, such as clapping, and an 'other' gesture class containing gestures that do not fit into the first 10 classes (Amir et al., 2017). The gestures are recorded under four different lighting conditions, so each gesture is also labeled with the associated lighting condition under which it was performed. Detailed analyses of the 'other' class and the four lighting conditions are included in the Appendix. Samples from the first 23 subjects were used for training and the last 6 subjects were used for testing. The training set contains 1078 samples and the test set contains 264 samples. Each sample consists of about 6 seconds of the gesture being repeatedly performed. In our work we scale each sample to 32x32 and only use a randomly sampled 200ms sequence to match real time learning conditions. For both datasets, to reduce memory requirements, the gradients were truncated to 100 time steps (i.e. 100ms worth of data).
For both datasets, the model learns a latent space encoding that can be used to reconstruct the digits or gestures and to classify instances accurately.
6

---

Under review as a conference paper at ICLR 2022
3.5 NEUROMORPHIC HARDWARE IMPLEMENTATION
As a first step towards a self-supervised gesture recognition system based on the Hybrid Guided- VAE, we developed a proof-of -o-ccct implementation of our SNN encoder which can be trained and run on Intel Loihi. Since only the encoder is required for estimating the feature embeddings, is not necessary to map the decoder onto the Loihi. We trained the neuromorphic encoder with our Hybrid Guided- VAE method with a couple key differences in order for the encoder to work on Loihi as follows: 1) To train with the same neuron model and quantization as the Loihi used a differentiable functional simulator of the Loihi chip (Shrestha Orchard (2018): Stewart et al. 2020a)), which allows one-to-one mapping of trained networks onto the hardware; and 2) The and Σ parts of the network were made spiking and used the quantized membrane potential of the neuron for the latent representation instead of ANN trained full precision values.
4 RESULTS
4.1 NMNIST ACCURACY AND LATENT SPACE
The use of the NMNIST spiking digit classification data allows to test the ability of the algorithm learn to encode spiking input data in manner that preserves information needed for accurate classification and captures salient features in the latent space. Trained on the NMNIST dataset, the excitation classifier achieved both training and test accuracy of approximately 99% indicating that the SNN encoder learned latent representation that clearly disentangles digit classes.
We used T-SNE to visualize the learned representations of the algorithm. T-SNE embeds both the local and global topology of the latent space into a low- dimensioonl space suitable for visualization (Mukherjee al. 2019). allowing and lighting conditions. As shown in Figure 2 each digit class in the NMNIST dataset clearly separable in the latent space, with only a few data points inaccurately clustered.
to observe clustering and separation between gesture classes
Figure 2: T-SNE plot of the zm portion of the latent space of the encoded NMNIST dataset. Color of the data points corresponds to the digit classes. Clear separation between classes indicates that the algorithm lears to encode the spiking data into latent space that strongly emphasizes class-relevant features over other variation.
4.2 DVSGESTURE ACCURACY, LATENT SPACE, AND RECONSTRUCTIONS
To analyze the learned representation of gestures in the latent space we examine the accuracy of the excitation classifier in correctly identifying a gesture, the T-SNE projections of the different parts of the latent space, and traversals of the latent space. We compared our results against partially ablated models (no-guiding, and no-spiking convolutional encoder) with results in the Appendix. Finally, we observe the quality of the embeddings based on our own DVS recordings of novel gestures.
The excitation classifier results on the DVSGesture dataset achieves training accuracy of approx- imately 97% and test accuracy of approximately 87%. Qualitatively, the SNN encoder learns disentangled latent representation of features unique to each gesture class but has some difficulty distinguishing between gestures that are very similar.
7

---

Under review as a conference paper at ICLR 2022
original
decoded
Capppigg
Right Wave
Left Wave
Right Clokwwise
Right Counter
Left Clockwise
Left Counter
Arm
Roll
Drums
Guitar
Figure 3: Original (top) and reconstructed (bottom) time-surfaces for a sample gesture from each class. The reconstructions reflect the location of each gesture but with some smoothing of the detail.
In Figure 3 sample gesture from each of the gesture classes is visualized as a TS. Colors in the samples correspond to the TS value of the events at the end of the sequence. Note that the TS leaves a significant amount of fine detail intact. In contrast, encoding and then decoding the samples results in reconstruction that preserves the general structure of the gesture but smooths out some of the detail. Note that, for the purposes of estimating gesture similarity, the fidelity of the reconstruction is less important than the capacity of the model to disentangle the gesture classes. Furthermore, dis- entangling autoencoders are known to provide lower quality reconstructions compared to unguided VAEs (Chen al., 2018).
We use T-SNE to examine the capacity of the network to disentangle salient gesture features in the latent space. Figure shows a T-SNE plot of the guided portion of the latent space trained on the DVSGesture dataset. This plot indicates clear clustering of gesture classes, with between similar gestures such as left arm clockwise and counterclockwise. This global structure in the learned representations indicates the Hybrid Guided-VAF is identifying useful, class-relevant features in the encoder and suppressing noise and unhelpful variability from the spiking sensor.
overlap
To test the generalization of the learned encoder, we evaluated how the VAE model performs when provided new gesture data captured in a new enviromment intended to replicate ecologically valid conditions for a real-world gesture recognition system. We recorded gestures belonging two new classes, right and left swipe down, which are not present in the DVSGesture dataset. We used a different DVS sensor (the DAVIS 240C sensor (Brandli et al. 20144) and processed the data with the trained Hybrid Guided-VAB. Each gesture was repeated three times for approximately 3s by the same subject under the same lighting conditions.
Figure (Left) shows the new gesture TS and associated T-SNE embeddings in the portion of the latent space. The right swipe down gestures were represented by the model similar to right hand wave gestures, as indicated by their proximity in the T-SNE plot of the latent space. Similarly, the left swipe down gestures were represented most similar to left hand wave gestures. Interestingly, both new classes cluster near the edges of the existing classes, possibly indicating the presence of
Hand Clapping
Right
Right Hand Wave Left Hand Wave Air Drums
Wave
Right Arm CW
Right Arm CcW
Left arm
Arm CCW
Drums
Guiter
New
Right Swipe Down
Left Swipe Down
Figure 4: (Left) A T-SNE plot of the Zm portion of the latent space of the encoded DVSGesture dataset. Additionally, projections of the 2. portion of the latent space of encoded new gestures we recorded using a different DVS, and not part of the DVSGesture dataset are shown. Bottom color plots are the TS of the new gestures. (Right) T-SNE plot of the 2m portion of the latent space disentanglement of three gesture classes implemented on the Intel Loihi.
8

---

Under review as a conference paper at ICLR 2022
feature gradient. These results demonstrate that the Hybrid Guided- is capable of appropriately representing novel gestures manner that could support pseudolabeling. With additional data points of new gestures, the hybrid Guided-VAE can eventually learn new classes of gestures on
As additional tool to investigate the structure of the latent space representations, we use latent traversals to interpret the features of the space. Traversals consist of a set of generated TS based on positions in the space computed using off polarity events. The positions traverse a line in the latent space, revealing how the network encodes salient features.
Figure 5 contains two traversals illustrating the shift in position and intensity of motion in the TS en- coded the latent variables and 23. Because those features correspond to distinguishing. salient characteristics of the gesture classes "Right Hand Wave" and "Left Hand Wave", this encoding allows the model to disentangle the gestures.
Z,=max
Other
Zye max
Figure 5: Traversals of the latent space learned from the DVSGesture dataset. (Top) Beginning with the right hand wave latent variable maximized and the left hand wave variable minimized, traverse along the latent space by gradually decreasing the right hand wave latent variable and increasing the left hand wave latent variable. Note the initial TS shows a small, focal area of motion in the top-left corresponding to the participant's right hand waving. (Bottom) The latent traversal along all of the non-target vm latent variables illustrates the relative insensitivity of the model to these features.
4.3
NAUROMORPHIC IMPLEMENTATION RESULTS
Figure (Right) shows the latent space representations of three gesture classes mapped using the neuromorphic encoder. With just three classes, the model rumning on Loihi demonstrates separation between gestures and global structure, but these features are not as defined as in the conventional model. With all ten classes of gestures used to train the neuromorphic encoder, there was clear disentanglement or obvious structure to the latent space. possible that the lack of separation for the more challenging 10-way task is due to the low-precision integers used for synaptic weights and membrane potentials of spiking neurons on the neuromorphic chip, thus adding variability to the embedded positions. In future work we will test our algorithm on more precise hardware and adopt powerful on-chip learning algorithms such as SOEL (Stewart et al.) 2020b).
CONCLUSIONS
We presented novel algorithm to process DVS sensor data and show that capable of learning highly separable, salient features of real-world gestures. The Hybrid Guided- VAE contains an en- coder model that learns to represent extremely sparse. digh-dimensional visual data captured at the sensor small number of latent dimensions. The encoder is jointly trained by two classifiers such that the latent space disentangles and accurately represents target features. The algorithm represents significant step towards self- self-supervised gesture recognition by measuring the similarity of gesture metric. addition, we demonstrated first-
data in real-time with a learned, perceptually-relevan of-its-kind implementation of the algorithm on neuromorphic hardware. Due to the sparse nature of event-hased data and processing, the SNN encoding implementation offers significant benefits for applications the edge, including extremely low power usage and the ability learn on-device to avoid intrusive remote data aggregation. This could enable flexible gesture recognition capabil- ities to be embedded into home electronies mobile devices, where computing power is limited and privacy paramount. While the model performance currently decreases running on the neu- romorphic device compared to the conventional GPU-based implementation, improvements to the neuromorphic hardware the adoption of more powerful learning algorithms, e.g. (Stewart al. 2020b), could alleviate these limitations. These techniques may owe some measure of their success to the computational principles they share with human perceptual systems and we expect that this approach will open possibilities for interaction between humans and intelligent machines.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Abdullah X. Ali, Meredith Ringel Morris, and Jacob O. Wobbrock. Crowdsourcing similarity judgments for agreement analysis in end-user elicitation studies. UIST '18, pp. 177-188, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450359481 doi: 10.1445/244333.3242261.. URL ptttps://do..oo otg//100.........o 1145/ 0240553242587.2 3242621
Abdullah X. Ali, Meredith Ringel Morris, and Jacob Wobbrock. Crowdlicit: A system for conducting distributed end-user elicitation and identification studies. In Proceedings of the
2019 CHI Conference Human Factors in Computing Systems, CHI '19, pp. 1-12, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450359702. doi: 011444/2290055330000000 URL http::ssss:o.o.oo/////o// 1140330000010030005000 3300485
Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al. A low power, fully event-based gesture recognition system. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7243-7252, 2017.
M Bah, Adel Hafiane, and Raphael Canals, Deep learning with unsupervised data labeling for weed detection in line crops in uav images. Remote Sensing. 10(11):1690, Oct 2018. ISSN 2072-4292. doi: 10.3390/rs 1690.. URL htt::tt::/ //tttt:://xx.. doi. www./o.//......... 00t.0/330000001000
Fabio Ballati, Fulvio Corno, and Luigi De Russis. Assessing virtual assistant capabilities with ital- ian dysarthric speech. Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '18, pp. 93-101, New York, NY. USA, 2018. Associ- ation for Computing Machinery. ISBN 9781450356503,
doi: 10.1145/32349553333354, URL https: wtrg/000rgg11... 1145/ /1/144//34444 3234495. 3236354
Guillaume Bellec, Franz Scherr, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets. arXiv preprint arXiv: 090019904993 2019.
Christian Brandli, Raphael Berner, Minhao Yang. Shih-Chi Liu, and Tobi Delbruck. A 240x 180 130 db 3 jas latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(10))2333-233 2341, 2014.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources dis- entanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL httptttpppppppepe.p.eiese wew.eeeddddinnnnenii.seee....eee new.ips....... cc/ http:////..1eeeeeeeacaaaaa.2....
Papp:99997222222222ppp..... pdf
Sabrina Connell, Pei-Yi Kuo, Liu Liu, and Anne Marie Piper. A wizard-of-oz elicitation study examining child-defined gestures with a whole-body interface. In Proceedings of the 12th International Conference on Interaction Design and Children, IDC '13, pp. 277-280, New York, NY, USA, 201 2a. Association for Computing Machinery. ISBN 978 4503 19188 doi: 10. 00.1145/245760.2444 248488583. 2485823. htlllttts::/eee.
org/10. 1145/24851552 60 2485823
Sabrina Connell, Pei-Yi Kuo, Liu Liu, and Anne Marie Piper. A wfzard-of-oz elicitation study examining child-defined gestures with whole-body interface. In Proceedings of the 12th International Conference on Interaction Design and Children, IDC '13, pp. 277-280, New York, NY, USA, 2013b. Association for Computing Machinery. ISBN 978 9781450319188. doi: 10.1145/2485760.24533 2485582.. WtRLhttss:// /tt::///oo.ooo www./o............... 1141/ 48874866700 2485823
M. Davies, N. Srinivasa, T. H. Lin, G. Chinya, P. Joshi, A. Lines, A. Wild, and H. Wang. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, PP(99):111, 2018. ISSN 0272-1732. doi: 10. 1..1009/MM.011 18.112130355
Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang, Max Welling, and Zhuowen Tu. Guided variational autoencoder for disentanglement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Marlen Fröhlich, Christine Sievers, Simon W. Townsend, Thibaud Gruber, and Carel P. van Schaik. Multimodal communication and language origins: integrating gestures and vocaliza-
tions. Biological Reviews, 1809-1998, 2019. doi: 10 12535 URL https: 7/on1 in bbra ry. wi www.lyy..........o...........eee www.oo///o//aaaoo 111...1111bbr...... 12555
10

---

Under review as a conference paper at ICLR 2022
Steve B Furber, Francesco Galluppi, Sally Temple, Luis Plana, et al. The spinnaker project. Pro- ceedings of the IEEE, 102(5):652 665, 2014.
G. Gallego, T. Delbruck, G. M. Orchard, C. Bartolozzi, B. Taba, A. Censi, Leutenegger, A. Davi- son, J. Conradt, K. Daniilidis, and D. Scaramuzza. Event-based vision: A survey. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2020. doi: 10.1 10910TAMM..200000 3008413.
Guillermo Gallego, Tobi Delbruck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. arXiv preprint arXiv: 1904.04455, 2019.
W. Gerstner and W. Kistler. Spiking Neuron Models. Single Neurons, Populations. Plasticity. Cam bridge University Press, 2002.
Wulfram Gerstner, Wernr M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From
single neurons to networks and models of cognition. Cambridge University Press, 2014.
Noorkholis Luthfil Hakim, Timothy K. Shih, Sandeli Priyanwada Kasthuri Arachchi, Wisnu Aditya, Yi-Cheng Chen, and Chih- Yang Lin. Dynamic hand gesture recognition using 3dcnn and Istm with fsmm context-aware model. Sensors (Basel, Switzerland), 19(24):5429, Dec 2019. ISSN doi: 1.3390/s112254299 31 31835404[pmid).
1424-8220.
URL https:// ncbi mnmm nih. mtl....i...oo
31835404
I. Higgins, Loic Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
G. Indiveri, B. Linares- Barreecoo T.J. Hamilton, A. van Schaik, R. Etienne-Cummings. T. Delbruck, S.-C. Liu, P. Dudek, P. Häfliger, Renaud, Schemmel, G. Cauwenberghs, J. Arthur, Hynna, F. Folowosele, S. Saighi, T. Baraan-Gotarredona, J. Wijekoon, Y. Wang, and K. Boahen. Neuro- morphic silicon neuron circuits. Frontiers in Neuroscience, 5:1-23, 2011. ISSN 1662-453X. doi: 10.3389/fmins. 201 1.00073.
Jacques Kaiser, Hesham Mostafa, and Emre Neftci. Synaptic plasticity dynamics for deep con- tinuous local learning (decolle). Frontiers in Neuroscience, 14:424, 2020. ISSN 1662-453X. doi: 10.3389/fmins.2220.ooo 3389/fnins 2020. 00424 00424
front ienetiiessin Uttphtts:: http://ww....oorrroooorooooora/a wttp:///eerrarr.........
Leonid Keselman, John Iselin Woodfill, Anders Grunnet-lepsen, and Achintya Bhowmik. Intel realsense stereoscopic depth cameras. In Proceedings of IEEE Conference Computer Viston and Puttern Recognition (CVPR) Workshops, July 2017.
Diederik Kingma and Max Welling. arXiv:1312.6114 2013.
arXiv preprint Auto-encoding variational bayes.
X. Lagorce, G. Orchard, F. Galluppi. B. E. Shi, and R. B. Benosman. Hots: A hierarchy of event- based time-surfaces for pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(7): 39(7):134--135,, 2017. doi: 10. 109/TPAMM.2011.5547077
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. volume 48 Proceedings of Machine Learning Research, pp. 1558 1566, New York, New York, USA, 20-22 Jun 2016. PMLR. URL http::/proceed: ngs m wrwss//sssee...... v48 rsenl 1 html
Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis, and Kaushik Roy. Spike-flownet: Event-based optical flow estimation with energy-efficient hybrid neural networks, 2020.
P. Lichtsteiner, C. Posch, and T. Delbruck. A 128x128 120 dB 15 us latency asynchronous temporal contrast vision sensor. Solid-State Circuits, IEEE Journal of. 43(2):566-576, Feb. 2008a. ISSN 0018-9200. doi: 10. 109//SSC.200 91199114337
P. Lichtsteiner, C. Posch, and T. Delbruck. An 128x128 120dB 15us-latency temporal contrast vision sensor. IEEE J. Solid State Circuits, 43(2):566-577 2008b.

---

Under review as a conference paper at ICLR 2022
S.-C. Liu and T. Delbruck. Neuromorphic sensory systems. Current Opinion in Neurobiology, 20 (3):288-224 295, 2010. doi: wtt.://..con.220000.0
Keenan R. May, Thomas M. Gable, and Bruce N. Walker. Designing in-vehicle air gesture set using elicitation methods. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI '17, ,, pp. 74-83, New York, NY. USA, 2017. Association for Computing Machinery. ISBN 978 1500351508. doi: 10.1145) 3122986. 123015.
URL
3123015
Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. Clustergan: Latent space clustering in generative adversarial networks. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01)::610-4500 Jul. 2019. doi: 16.9/69/
URL https:
7/ojs. awai.aaa.......... www//irgg....oo...o..oo...o....e /ttp://////////rrieeecc//////eea
AAAI /wt.//AAAAaaaaeaaaaaaaaaaaaaaaea
E. O. Neftei, H. Mostafa, and F. Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- hased optimization to spiking neural networks. IEEE Sig- nal Processing Magazine, 36(6):51-63, Nov 2019. doi: 10. 20.1009/
URL
attps://ieeexplee wwt.::eeeeeeeeeeeeeeeeeeeeeee wwtp/eraaam///////o.../.......ae 79/ 888 05875444488889--8888 pdf
ieee
Mchdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. Boosting self-supervised learning via knowledge transfer. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, jun 2018. doi: 10. 2..1199/cp::2211800999 URL htt.:::tpd/oo.. org//0.
110952Fevpr. 2018. 0097
Garrick Orchard, Ajinkya Jayawant, Gregory K. Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saceades. Frontiers Neuroscience, 9, nov 2015. doi: 23389//inns.2115.004.
C. Posch, D. Matolin, and R. Wohlgenannt. A qvga 143 db dynamic range frame-free pwm image sensor with lossless pixel-level video compression and time-domain cds. Solid-State Circuits, IEEE Journal of, (661):255922555 jan. 2011. ISSN 0018-9200. doi: 10.1 00.100///SC..2000200550000
Krittaphat Pugdeethosapol, M. Bishop, Dennis B. Bowen, and Q Qiu. Automatic image labeling with click supervision on acrial images. 2020 International Joint Conference on Neural Networks (IJCNN) pp. 1 8, 2020.
Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems, pp. 1412 1421, 2018.
Chaklam Silpasuwanchai and Xiangshi Ren. Jump and shoot! prioritizing primary and alterna- tive body gestures for intense gameplay. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '14, pp. 951 951-954, New York, NY, USA, 2014. Associa- tion for Computing Machinery. ISBN 9781450324731. doi: 10.1445/2552288.5557100 URL
https::/doll http/oi..oggg...////////// 11111114555555
2557107
K. Stewart, G. Orchard, S. B. Shrestha, and E. Neftei. Online few-shot gesture learning on a neuromorphic processor. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 10(4):512-521, Oct 2020a. doi: 10. 20000//ETTASS.20..33333333 //ieeexplore. leee htt.ee/..r...aaatartcc/ceeeeeeee
URL https:
K. Stewart, G. Orchard, S. B. Shrestha, and E. Neftci. Online few-shot gesture learning neuro- morphic processor. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 10 (4):512-521, 2020b. doi: 10. 10.1109/JTTAS. 00110000ECAAS22200002225555 2000.33355.00
Theophanis Tsandilas. Fallacies of agreement: critical review of consensus assessment methods for gesture elicitation. ACM Trans. Comput.-Humm Interact, 25(3), June 2018. ISSN 1073-0516. doi: 10.1145/3182166 URL https: /wi. orc.l.org/1100 1145 /185511
Radu-Dania Vatavu. User-defined gestures for free-hand control. In Proceedings of the 10th European Conference on Interactive TV and Video, EurolTV '12, pp. 45-48, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 47814503 doi: 10.1145/2325616 2325626. URL httpt::t/////.... www.ooooorrgg//// 1ttp:/02...15/222511111 22255222332611622222222 2325626
12

---

Under review as a conference paper at ICLR 2022
Radu-Danis Vatavu. The dissimilarity-consee approach to agreement analysis in gesture elic- itation studies. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pp. 1-13, New York, NY, USA, 2019. Association for Computing Machin- ery. ISBN 9781450359702. doi: 10.1145/3290005 3300454
httph: org/1..........
MadueDaniel Vatavu and Jacob O. Wobbrock. Formalizing agreement analysis for elicitation studies: New measures, significance test, and toolkit. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI 15, pp. 1325-1334, New York. NY, USA, 2015. Association for Computing Machinery. ISBN 978 45033 1456. doi: 10.1 10.1145/270222222702222.
URL http:http:///oooooo org/ 1et...115/22222222 . 2702223
Santiago Villarreal-Narvaez, Jean Vanderdonckt, Radu-Daniel Vatavu, and Jacob O. Wobbrock. A systematic review of gesture elicitation studies: What can learn from 216 studies? In Pro-
ceedings of the 2020 ACM Designing Interactive Systems Conference, DIS '20. pp. 855-872, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450369749, doi: 10.1145/33772333395551. uttpttps::////// //doi. org/10 1145/ 311453355723633998 3395511
Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Pro- ceedings of the 29th Annual Symposium User Interface Software and Technology, UIST '16, pp. 851-860, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978 078145034189 1899. doi: 04114/2284511..985555
URL /ttp::///ci.org/1000 1145/
2984511 2984565
Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invari- ances. Neural computation, 14(4):715-770, 2002.
Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew Wilson. User-defined gestures for surface computing. In Proceedings of the SIGCHI Conference on Human Factors in Com- puting Systems, CHI '09, pp. 1093-1092. New York, NY, USA, 2009. Association for Com- puting Machinery. ISBN 9781605582467. //doi. 10. 11 0444555110000 15188866 6518866
doi: 10.1145/1518701.1 15177001111116666 URL https:
Gareth Young, Hamish Milne, Daniel Griffiths, Elliot Padfield, Robert Blenkinsopp, and Orestis Georgiou. Designing mid-air haptic gesture controlled user interfaces for cars. Proceedings of the ACM on Human-Computer Computer Interaction, 4(EICS): Jun 2020. ISSN 2573-0142. doi: 10. 10.1145/3397869, httphtt:: /xx doi orrpiooor//loo 1145/3397869
Zenke and E. O. Neftei. Brain-inspired learning on neuromorphic substrates. Proceedings of the IEEE, pp. 1-16, 2021. doi: 10. 10.160//PR0C.2200.045555
Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multi-layer spiking neural networks. arXiv preprint arXiv: /1146, 2017.
Z. Zhang. Microsoft kinect sensor and its effect. IEEE MultiMedia, 19(2):4-0,, 2012.
A APPENDIX
A.1 ABLATION STUDY
We present the results of an ablation study of our Hybrid Guided-VAN method to demonstrate why used this method for latent space disentanglement. We compare the Hybrid Guided- in our work to a hybrid VAE with the guided part ablated, as well as a Guided- VAF that does not use an SNN encoder and instead use CNN encoder, and an ordinary CNN VAE. Comparing the clustering of the hybrid VAFS and the CNN VAES in Figure 6. both the CNN and hybrid VAES able to disentangle and cluster the latent space when using the guided method, with our hybrid method in Figure 6b) showing less overlap between clusters and therefore better disentanglement. For classifi- cation from the latent space representation of the data, T that the CNN and hybrid VAEs both achieve high accuracy on both datasets, with the hybrid VABS achieving higher performance. Therefore, our hybrid guided VAE approach is more suitable for latent space disentanglement with data taken from event-based sensors. The disentangled latent space shows the hybrid VAES learn measure similarity in event-based data, such as gesture similarity, which could be used future work for self-supervised learning from unlabeled data streamed to neuromorphic hardware using an approach similar to (Noroozi et al. 2018).
13

---

Under review as a conference paper at ICLR 2022
Table 2: Classification from latent space
Algorithm
Datasct
Train
Test
Hybrid Guided VAE
DVSGesture
97.6%
86.8%
NMNIST
99.0%
98.2%
CNN Guided VAE
DVSGiestur
86.7%
82.3%
NMNIST
97.2%
96.8%
Hybrid VAE
DVSGesture
99.7%
38.3%
NMNIST
96.8%
92,4%
CNN VAE
DVSGesture
97.5%
28, 28.4%
NMNIST
96.8%
91.5%
now
Highr
Ann andll
Left Sodpe Down
Fight Sie Dowen
night Swipa Down
goi Down
(a) CNN Guided VAE
(b) Hybrid Guided VAE
Chagping
drm
Pagid tiw Mave
Letk com
Right
Alr Sulter
Right Swice Down
Left Swipe Dawnn
(c) CNN VAE
(d) Hybrid VAE
Figure 6: TSNE plots of the DVSGesture latent space. The images below each figure show novel gestures and where they are placed in the latent space by the different models.
A.2 SNN DYNAMICS CORRESPOND THE DISCRETIZATION OF LIF NEURON MODEL.
We use the LIF model with current-based synaptic dynamics and a relative refractory period defined
in Kaiser et al. (2020). The dynamics of the membrane potential U, of neuron is determined by the tollowing differential equations:
U.(t) Vv((t) R.. b,.
d
Tmem dt
V((t) = Vi(t) + 4(t),
(7)
Fres dt
(1(() = R2(1) + 5,(),,
S.(t) =5(t/ t2)
with St(t) representing the spike train of neuron i spiking at times t/. where ó is the Dirac delta. The constant b, represents the intrinsic excitability of the neuron. The two states U and V is interpreted as special case a two-compartment model, with one dendritic (V) compartment and one somatic (U) compartment. The reset mechanism is captured with the dynamics of R, The factors Tmem and Tref time constants of the membrane and reset dynamics, respectively. I, denotes the total
14

---

Under review as a conference paper at ICLR 2022
synaptic current of neuron i, expressed as:
d Tayn di
- 1:())+ Σ (t), jEpre
(8)
where Waj is the synaptic weights between pre-synaptic neuron j and post-synaptic neuron i. Be- cause Vi and I are linear with respect to the weights Wiz. the dynamics of Vú can be rewritten as:
Vi(1) Σ W()),(t) jEpre
Timem Pt(() +
d Q2(() Q5(1) (t)
Taym dt
The states and Q describe the traces the membrane and the current-based synapse, respectively. For each incoming spike, each trace undergoes a jump of height 1 and otherwise decays exponen- tially with time constant Tmem (for P) and Tayn (for Q). Weighting the trace P3 with the synaptic weight W; results the Post-Synaptic Potentials (PSPs) of neuron caused by input neuron j. The equations ( define spike response model Gerstner et al. (2014) with linear filters.
In digital system, the continuous dynamics above are simulated in discrete time. Therefore the equations above are discretized by integration over one time step At Kaiser al. (2020), leading to equations (7) in the main article. Discretized equations for R are:
R 7R, (1 (t),
(10)
with 7 = exp( Free)
A.3
TIME SURFACES WITH EXPONENTIAL KERNELS ARE PRE-SYNAPTIC TRACES Q
In continuous time, the Time Surface (TS) is defined as:
(E* (tn)(t) = S," - (it=s)Smn(sddl with c(s) =
(11)
We demonstrate here that the solution of Q is equal to (e Smn (t) when Taym The differential equation governing Q( linear and can be integrated in straightforward manner:
d ((t) (t) + (tn
Tayn dt
Q(t) l' exp( 1-5 Sin
(12)
Taya
(E (t))
where we have assumed Q(0) 0. The last line is true if Tayn
A.4 GUIDING OTHER FACTORS OF VARIATION: LIGHTING
A key feature of the guided VAE is to incorporate alternative features not directly related to the gesture class disentangle the factors of variation in the data. To demonstrate this, in separate experiment, we trained on the lighting condition provided in the DVSGesture dataset instead of the gesture class. The T-SNE projection of the Zm latent space is shown in Figure 7 The model clusters the lighting conditions with This is likely due to the fact that the lighting conditions under which the gestures are performed are combinations of the labeled lighting conditions, with the label given to the most prominent lighting condition used Amir et al. (2017).
overlap between LED lighting and the other lighting conditions.
15

---

Under review as a conference paper at ICLR 2022
natural Buorescent
Figure 7: A T-SNE plot of the guided 2m latent space using lighting condition labels.
Hand Clapping Hand
Hand Wave
Right Arm CW
Right Arm COW
Left Arm CW
Left Arm CCW
Arm Roll
Drums
Figure 8: A T-SNE plot of the 2m portion of the latent space of the encoded DVSGesture dataset. Six unlabeled gestures (labeled "other" in the dataset) are shown below time-surfaces connected to the respective embedded positions in the latent space. Each of the unclassified gestures embedded near gesture classes which exhibit similar motion.
A.5 LABELING UNLABELED GESTURES
Figure shows a T-SNE plot of the guided portion of the latent space trained on the DVSGesture dataset. The latent space is clustered by the gesture that the excitation classifier targeted for each of the 2m, with some overlap between very similar gestures such left arm clockwise versus left arm counterclockwise. The bottom of the figure shows samples from the "other" class in the Dys Gesture dataset in relation to the T-SNE projection of the guided latent space. The embedded positions of these gestures show which gesture classes they are most similar to. For example, gestures with certain salient features, such as a salient left hand, are placed with the cluster of the gesture class that has the same salient feature, such as the left hand wave class. Thus, the latent states obtained for these gestures can provide the basis for aseudolabeling of new gestures by measuring the similarity between unclassified samples and existing gesture classes in the latent space.
A.6 HYBRID GUIDED VAE CLASSIFIER ARCHITECTURES
Table
Hybrid Guided- VAE excitation classifier architecture
Layer
Kernel
Output
Layer Type
input
I0
2t
2
100
Linear layers
3
100
output
10
Feature (One Hot Target)
16

---

Under review as a conference paper at ICLR 2022
Table 4: Hybrid Guided-VAl VAE inhibition classifier architecture
Layer
Kernel
Output
Layer Type
input
90
Zret
2
100
Linear layers
3
100
output
10
Feature (One Hot Target)
output
10
Feature (No Hot Target)
A.7 SYSTEM SPECIFICATIONS FOR MEASUREMENT
The training of the hybrid Guided-VAL VAE model and the latent space classifier were performed with Arch Linux 5.6. 10, and Py'Torch 1.6.0. The machine consists of AMD Ryzen Threadripper CPUs with 64GB RAM and Nvidia GeForce RTX 1080Ti GPUs. For the neuromorphic hardware imple- mentation, an Intel Nahuku board consisting of Loihi chips running on the Intel Neuromorphic Research Community (INRC) cloud were used with Nx SDK 1.0. The machine consists of an Intel Xcon E5-2650 CPU with 4GB RAM.
17


Reviews:
[{"review_id": "YI5XW0QPyGi", "reviewer": "Reviewer_K3XW", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed SNN encoder method is interesting. T-SNE plots show that it is successful in disentangling the gesture classes.\n2. The paper covers the literature well.\n3. The hardware implementation demonstrates the low-power realization capability.\n\nWeaknesses:\n1. First of all, the motivation behind a neuromorphic system to address the rigid gesture limitation is not clear. In fact, it can also be possible to address the rigid gesture limitation with traditional sensors. Is the very high temporal resolution of DVS really necessary to successfully handle natural gestures? This motivation is not convincing. If so, authors should be able to support their claims by some experimental comparison or by a reference to a prior work.\n2. This paper does not present experimental evaluation to justify the claims, which is a major drawback. It only reports accuracy on the training and test set of the DVSGesture dataset, and presents some T-SNE based analysis. Quantitative comparisons against some competing methods are required to be able to justify the effectiveness of the proposed approach.\n\nMinor issues:\nBoth arxiv and journal versions of the same paper are cited: Gallego et al. 2020 & Gallego et al. 2019. Is there a reason for that?\n\n", "summary_of_the_paper": "This paper proposes a hybrid spiking neural network (SNN) guided variational auto-encoder (VAE) in order to overcome the limitation of rigid and dramatic gestures that are required in commercial mid-air gesture recognition systems. The motivation is that this limitation causes unnatural movements and thus fatigue. Authors claim that the proposed neuromorphic system overcomes this limitation by means of event-based data of high temporal resolution, and that it enables self-supervised learning of natural mid-air gestures for on-device learning advantage. For this purpose, a VAE latent space representation is employed, which is shown to be effective in clustering and pseudo-labeling tasks. Authors obtain 87% classification accuracy on the DVSGesture dataset, and claim that their method can learn highly separable and salient features. Their work involves a neuromorphic hardware implementation as well to demonstrate the real-time potential.", "summary_of_the_review": "Although the method and the proof-of-concept implementation is interesting and the analyses including the latent space are informative, the motivation of this work is not clear to me and the paper lacks sufficient evaluation to support the claims. I explain these weaknesses above. This paper looks like an unfinished study. Therefore my decision is rejection of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "FSwxAX84VQK", "reviewer": "Reviewer_LZrg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe this work brings some interesting concepts and several good ideas that could be of interest for practitioners. However I think its final outcome only goes halfway. I feel the architecture proposed is not really leveraged to prove an actual benefit over existing approaches. Experimental setup is somehow limited to prove the input is properly captured and reconstructed but does not show other benefits beyond that neither is evaluated against current SOA approaches.\n\nAs a practitioner in the HAR domain I’d rather see a less ambitious paper, but with a more complete story. I believe authors try to combine multiple relatively novel bits (DVS, SNN, VAE) but at the end it is not clear what each one brings to the table. I understand these technologies have been around for a while and the idea may be that all of them sum up for the work’s contribution. But for gesture detection/modelling, for instance, a work showing actual benefit of SNN+VAE over more traditional LSTM+CNN , with an adequate experimental setup and evaluation, would make a proper contribution.\n\nAs another suggestion, I think the manuscript’s current title and abstract are a bit misleading toward the gesture recognition/HAR domain. Honestly I do not think authors should change the domain, but the focus of the experimentation. In my opinion, having NMNIST for this paper, given it’s not gesture related and it’s limited complexity, does not contribute much. I’d rather see a more gesture/action related experimental setup.\n\nOverall I believe the work would truly benefit from a more comprehensive evaluation, adding to the experimental setup diverse datasets, including comparatives and -as author already note- performing ablation studies to evaluate the  contribution of each component.  \n\nBy the way I looked for the ablation experiments in the Appendix and was not able to find them. Probably I missed them, just mentioning it so authors can verify they are attaching the correct appendix. In any case, if you could add these experiments to a document/pdf that’d be great, make them easier to spot.\n\nThe overall writing quality of the paper is very good. It’s well written, well structured, easy to read and I consider the figures/tables included illustrate the proposal correctly.\n", "summary_of_the_paper": "Authors propose a gesture modelling approach based on two fairly novel components, the use of a DVS camera and a SNN working in conjunction with a VAE. Using the DVS cameras as input in the space/time domain, this is recurrently processed by the SNN and its latent representation modelled by the VAE. Authors show examples of this representation using two DVS captured datasets. They show the feasibility of this approach for time based events scenarios.\n", "summary_of_the_review": "The paper presents several interesting ideas (DVS+SNN+VAE) in the shape of the proposed architecture. The novelty component I believe is justified since I’m not familiarized with similar approaches in the gesture modeling domain. However, and this is the reason for my recommendation, the evaluation of the proposal somehow only relies on the capacity of the system for modelling the input. It’s not analyzed the contribution of each of the pieces proposed and it’s not measured its performance against other approaches. The experimental setup is what I think diminishes significantly the relevance of the work for the conference audience.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "56jbX0M7MI", "reviewer": "Reviewer_t5op", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The work focuses on novel sensors that smartly comprises visual data, triggering events only when changes occur in the image for every location. Also, event streams have very high temporal resolution and HDR. \n+ The solution explicitly takes advantage of the temporal resolution, employing a SNN that uses a variational autoencoder for the learning. \n+ Authors present the results evaluating on a widely used dataset for DVS data, showing how their solution achieves correct classification for different gestures, and also showing how the clustering using the autoencoder clearly separates different sub-gestures or parts, that could potentially help with generalization as well\n\n- The motivation for the use of event-based sensors is sometimes confusing. Specifically, authors say an event-based recognition system helps with the problem of requiring rigid gestures. However, the paper does not demonstrate how and why. Some experiments are needed to actually prove that the system presented \"recognizes natural gestures despite their wide ranging variability\" \n- Authors do not compare with other works in the state of the art. There are multiple solutions already for gesture recognition, and specially using the DVS gesture dataset. \n- There is no clear section in the experimental section that shows results with a classification metric\n- Authors also mention several times their implementation of the solution in Loihi. From the paper right now, it is not clear if they implemented the whole solution (it seems it was implemented partially) and there is not data about any hardware implementation with no details about the resources, the performance, the energy consumption, etc. In fact, I would remove this part from the paper since it makes the objectives of the work a bit more diffuse", "summary_of_the_paper": "The papers presents a method for gesture recognition using event-based sensors. Also, instead of using conventional machine learning techniques based on images, authors present a solution that employs a guided variational autoencoder that is plugged into a SNN network, making a the whole system a biologically-inspired solution.\nAuthors show the efficacy of their method on a dataset with event streams that is publicly available. \nFinally, authors also mention they partly implemented the solution on the Intel neuromorphic chip Loihi   ", "summary_of_the_review": "My recommendation is that the paper should not be published in its present form. Although the solution is novel and interesting and addresses a problem that is relevant to the community, the lack of comparison to other works in the state of the art, makes the work hard to assess (the comparison to only 1 work in the appendix is not enough either)\nSecondly, the experimental section is not complete, and needs to be improved adding evaluation metrics and a proper discussion about them\nFinally, the section about the implementation in the neuromorphic chip seems superficial and might need to be left out to focus the paper in the solution/method itself. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-mrVz1kjb6D", "reviewer": "Reviewer_6Az2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n1. The proposed approach is technically sound. It is especially good that the authors show the results after implementing their network on neuromorphic hardware.\n\n2. The experimental results show that VAE architecture learns almost perfectly separated clusters for the target gesture classes, and classification accuracy of 97% on neuromorphic MNIST dataset and 87% on the IBM DVSGesture dataset.\n\n### Weaknesses\nThe full extent of the benefits offered by the different components of the proposed approach was unclear from my reading of the paper.\n1. Why is the VAE architecture necessary for gesture detection? Since the authors already employ separate classifiers to disentangle the latent features, what additional benefit is provided by the decoder component for feature learning? This would have been best illustrated with an ablation study by removing the decoder from the network architecture.\n\n2. Further, performing an ablation study consisting of only one classifier, and therefore no disentanglement of the latent features, would have clearly justified the need for disentanglement for the purpose of gesture recognition in the first place.\n\n3. While it is appreciable that the proposed network reads event data at the resolution of 1 ms, how useful is it to read the data at such a high resolution? Have the authors performed any experiments of sampling the data at lower temporal resolutions and observing the corresponding detection performance? My understanding is that processing the input data at higher resolutions is more expensive. Moreover, for such high resolutions as 1 ms, the network may be flooded with redundant data (consisting of almost identical event data in a majority of the adjacent frames) since the rate of change of gestures is presumably much lower, hence most of the network's processing ends up being wasteful.", "summary_of_the_paper": "The authors present a method to detect mid-air gestures by transforming event-based data captured by a Dynamic Vision Sensor (DVS) into meaningful latent representations. The proposed method consists of a variational autoencoder (VAE) built using spiking neural networks (SNNs) to encode the event data at a high temporal resolution. The VAE is coupled with a pair of classifiers to disentangle the latent features into event-aware and event-agnostic features, and detect the gestures based on the event-aware features. Experimental results on two datasets show that the proposed approach outperforms a convolutional neural network (CNN) based VAE baseline and is able to cluster the features for the different gesture classes in the latent space.", "summary_of_the_review": "Overall, the proposed approach provides excellent results for gesture recognition on two benchmark datasets and goes the extra step of implementing the neural network on neuromorphic hardware. However, I have some questions regarding the benefits of the individual components of the neural network and concerns regarding the proportion of wasteful computation done by the network, as detailed under \"weaknesses\". I invite the authors to address these comments and believe that the paper would subsequently make a strong contender for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 3 rejections and 1 marginal accept. Reviewers were unanimous in that empirical evaluation is lacking. No rebuttal was submitted and I have no reason to overturn the reviewers' decisions. I recommendation this paper be rejected.
Decision: Reject