Associative Memory Augmented Asynchronous Spatiotemporal Representation Learning for Event-based Perception
Authors: Uday Kamal
Saurabh Dash
Saibal Mukhopadhyay
Venue: ICLR 2023 notable top 25% (2023)
Keywords: associative memory
memory augmented neural network
spatiotemporal representation
event-based camera
event-based perception
object recognition
attention
set processing

Abstract: We propose $\textit{EventFormer}$, a computationally efficient event-based representation learning framework for asynchronously processing event camera data. EventFormer treats sparse input events as a spatially unordered set and models their spatial interactions using self-attention mechanism. An associative memory-augmented recurrent module is used to correlate with the stored representation computed from past events. A memory addressing mechanism is proposed to store and retrieve the latent states only $\textit{where}$ these events occur and update them only $\textit{when}$ they occur. The representation learning shift from input space to the latent memory space resulting in reduced computation cost for processing each event. We show that EventFormer achieves 0.5$\%$ and 9$\%$ better accuracy with 30000$\times$ and 200$\times$ less computation compared to the state-of-the-art dense and event-based method, respectively, on event-based object recognition datasets.

Published as a conference paper at ICLR 2023
ASSOCIATIVE MEMORY CHRONOUS SPATIOTEMPORAL REPRESENTATION LEARNING FOR AVENT-BASED PERCEPTION
AUGMENTED ASYN-
Uday Kamal", Saurabh Dash" Saibal Mukhopadhyay
School of Electrical and Computer Engineering
Georgia Institute of Technology (ukamal 6, saurabhdash @gatech. edu, sa i (ecee wateeegatechh... edu
ABSTRACT
We propose EventFormer- a computationally efficient event-based representation learning framework for asynchronously processing event camera data. Event- Former treats sparse input events as spatially unordered set and models their spatial interactions using self-attention mechanism. An associative memory- augmented recurrent module is used to correlate with the stored representation computed from past events. A memory addressing mechanism is proposed store and retrieve the latent states only where these events occur and update them only when they occur. The representation learning shift from input space the latent memory space resulting in reduced computation cost for processing each event. We show that EventFormer achieves 0.5% and 9% better accuracy with 30000x and 200x less computation compared to the state-of-the-art dense and event-based method, respectively, on event-based object recognition datasets.
INTRODUCTION
Ultra-low power, high dynamic range (> 120dB), high temporal resolution, and low latency makes event-based cameras (Brandli et al. 2014: Suh et al. 2020: Son et al.. 2017 77: Finateu all (020) attractive for real-time machine vision applications such robotics and autonomous driv- ing (Falanga et al., 2020: Hagenaars et all 2020: Sun et al. 2021: Zhu et al., 2018 Gehrig et all. 102 Convolutional and recurrent neural network-based based methods, originally developed for frame- based cameras, have demonstrated good perception accuracy on event camera (Gehrie et al. 2019 Baldwin a 2022: Canniei all, (220b)). But they rely temporal aggregation of the events to create a frame-lik- dense representation as input thereby discarding the inherent sparsity of event data and resulting in high computational cost (Figure 2). Recent works have explored event-based processing methods for object recognition to exploit data sparsity. Examples of such methods based representation relying on hand-crafted features (Lagorce et al., 2016 Sironi et all 201 & Ramesh et al. 2019). 3D space-time event-cloud (Wang et all 2019), and Graph- based methods et all, 202 c Schaefer et al. 2022). These methods adopt event-based processing to achieve lower computational costs but do not achieve similar performance compared to the dense- representation based methods (Figure 2) This necessitates computationally efficient algorithms that exploit sparsity and achieve high accuracy.
include
We propose an associative memory-augmented asynchronous representation learning framework for event-based perception, hereafter referred to EventFormer, that enables computationally ef- ficient event-basee based processing with high performance (Figure D. As events are triggered asyn- chronously, an event-based processing algorithm must generate and maintain a higher-order repre- sentation from the events, and efficiently update that representation to correlate a new event with the past events across space and time. One way to address this to include a recurrent module at each pixel to track history of past events (Cannici et all (2020a). However, the associated process- ing and memory requirement of such method increases exponentially with the number of pixels. Motivated by recent works in memory augmented neural networks k Kumar et all 2016; Ma et al..
"quall contribution
Author CohereAI

---

Published as a conference paper at ICLR 2023
Event stream
Frame
Graph
Point cloud
Set
Atradaiia
1
Event- baaed P Perception
CNN
GNN
PointNet
EventForrer
New
(+12A)
(a)
(b)
(c)
Figure I: Comparison with existing works. (a) Event-based perception algorithms where percep- tion latency & is proportional to the event-generation rate( Á). (h) Existing works include (left) dense processing that aggregates events into a frame and use CNN, event-based processing includ- ing (mid) GNN on spatiotemporal event-graph, (right) PointNet like architectures treating events as point-cloud. These methods either re-process (frame and point cloud) or store the past events (graph) to spatiotemporally correlate with new events. (c) EventFormer encodes the spatiotemporal interaction of the past events into a compact latent memory space to efficiently retrieve and correlate with the new events without requiring to store or re-process the past events.
2018 Karunaratne al., 2021). we address the preceding challenge by maintaining a spatiotempo- ral representation associated with past events, occurring at various pixels, as the hidden states of an Associative Memory. As learning spatiotemporal correlation is shifted from the high-dimension input (pixel) space to compact latent memory space, EventFormer requires an order of magni- tude lower floating point operations (FLOPs) per event to update the memory. To the best of our knowledge, EventFormer is the first associative -ugmentdd spatiotemporal representation learning method for event-based perception. The key contributions this paper are:
EventFormer maps the spatiotemporal representation of the incoming event stream into the hidden states of an associative memory and uses lightweight perception head directly operating on these states to generate perception decisions.
The spatiotemporal representation update mechanism activates only 'when" and only 'where' there is new event using unstructured set-based processing without requiring to store and re- re-process the past events.
We propose a new query-key association memory access mechanism to enable spa- tial location-aware memory access retrieve the past states of the current event locations.
Given a new event (or set of events), our model generates spatial representation by computing their spatial interaction through self-attention mechanism (Vaswani et al.. 2017)and retrieves the past spatiotemporal states related to that pixel location(s) from the memory. The retrieval process exploits a novel location-based query and content-based key-value association mechanism to extract correlations among the past events neighboring pixels. Once we get the past states, along with our present spatial representation, a recurrent module takes them as input and generates the refined state information. The associated hidden states of the memory are updated with the new information. We evaluate EventFormer on object recognition task from event-camera data. In our experiments on existing benchmark datasets, N-Caltech101 (Orchard al], 2015) and N-Cars (Sironi et al. 2018). EventFormer shows an excellent computational advantage over the existing methods (both dense and event-based) while achieving performance comparable to the dense methods (Figure
Related Work: Dense methods convert events to dense frame-like representation and process them with standard deep learning models such CNNs (Maqueda et al., 2018 Gehrig et al. 2019: Cannici et all 2020a). These methods are synchronous in nature as they generate dense inputs by binning and aggregating events in time and generate output only when the entire bin is pro- cessed. Event-based methods update their representations with a new event and generate new out- put. Methods such as (Lagorce all, 2016: Sironi et all, 2018 Ramesh et all (019) compute time- ordered representation (also known as time-surface) in an event- based manner with fewer compu- tations. However, their reliance fixed, hand-tuned representation results in sub-optimal perfor-
2

---

Published as a conference paper at ICLR 2023
mance compared to the data-driven methods (Gehrig et al 2019). Recently, graph-based methods (Messikommer et al., 2020; Li et al. 202 Schaefer et al., 2022) have been considered that gener- ate spatiotemporal event graph and process them with GNN. For new event, the graph is updated by removing nodes associated with past events and adding new ones. As these methods require subsampling the events to reduce the size of the graph (and hence, computation), performance is reduced as well. Also, they need to store the past events inside the graph to correlate with the new event causing additional computation and memory overhead. Point cloud-based methods can inherently process unstructured event camera data in a permutation invariant manner (Wang et al., 2019: Vemprala et al., 2021). However, they process over sliding window of time and need to re-process the past events for every new event to establish correlations leading to redundant com- putations. Memory Augmented Neural Networks typically combine an external associative memory with a recurrent controller (Kumar et all 2016 Ma et al. 20118 Karunaratne et al. (021) resulting in enhanced memory capacity compared to the vanilla recurrent module (Cho et al. 2014). Can- nici et al. (2020) propose pixel-wise memory-based representation learning for event-camera data that stores states at each pixel location (Cannici et al., 2020a).. However, it requires a com- putationally expensive feature extractor spatiotemporal correlations among these states not considered. Event camera has also been applied for more complex tasks including egomotion, mo- tion segmentation Parameshwara Pagranntaiion et all bb22 and depth prediction (Hidalgo- Carrió et al. (02(). To broadly categorize, these methods either fall into dense frame-based representation where clas- sical image processing (Mitrokhin et all 2018) and end-to-end learning-baad -bandd methods (Zhu al., 2019: Mitrokhin et al. (019)) have been applied or 3D-point cloud representation where GNNs has been adopted(Mirrokhin et al., 2020) to learn to perform these dense prediction tasks. While most of the existing works in this direction adopt a dense processing -aaend encoder-decoder structure, spike-based asynchronous formulation has also been explored in (Parameshwara et al. (021a).
2 METHOD
Preliminaries and Problem Formulation: An event-based camera consists of sensors at every pixel location that respond to the change of brightness asynchronously and in- dependently (Gallego et all (022). Mathe-
85
Ours
Cannict al PURUL
Recognition Accuracy [%]
Gebrig
2019
80
matically, a set of events produced at pixel locations (xr, y);
< y < H) during a time interval 5 can be defined as a time-ordered sequence, & {(rs, Vi, P:) 4 E T = sup, 43 where i denotes the event triggering timestamps, pt E
< W,0 <
Lictl
202
75
Messikommer 2020
Schaefer et
2022
{-1,1} as the polarity (relative change in brightness), and T as the total time of obser- vation. In this work, we consider the spatial locations of the events as the input, that is: E, f(z,, 91)}. Our goal is to learn com- pute efficient parametric mapping F : & + Me, that convert the raw event sequence E, into a suitable representation Mg, e Im xd (where x d defines the dimension the fea- ture space) with event-based processing capa- bility. This implies that F needs have the capability to update its representation as new event(or a new event sequence) arrives without storing or recomputing the past events.
Event- based Dense
65
Sironi et
2018
10-1
10
10*
10a
10a
10
Computational Complexity [MFLOPSevent]]
Recognition performance compu- tation. On N-Caltech101 dataset, EventFormer has about 30000 lower compute cost than the SoTA dense method (Cannici et al_ (020bb with 9% improvement in accuracy compared to the best performing event-based method (Li et all, (021c)).
Figure 2:
soon
Overview of the Framework: Eventformer consists of positional encoder (Li et al. 202 fol-
lowed by a pairwise interaction module to compute the spatial interactions among the events (Figure 3). A recurrent module takes this output, Z, the current input state, and computes the spatiotem- poral representation X, by retrieving the past hidden states H4-1 stored in an associative memory, M. The output of this recurrent module is used to update the memory representation Mr, which
3

---

Published as a conference paper at ICLR 2023
List of Events
Retrieve Past States
Associative Memory
{i)
7
Read
M
M
M
14-11
FF pred
Xt
M's
M
Classification Head
R
Refine
Write
Xt
Positional Encoder
Pairwise Interaction
Recurrent Module
Update New States
Figure EventFormer architecture. EventFormer generates positional embedding Te from list of new events time I, and calculates their pairwise interaction, Z, using self-attention. recurrent unit extracts the past states Hz-1 associated with F, from an associative memory and computes & the current states. This is further encoded into the latent memory space, Mr to process future events. linear layer uses M, to predict the target class.
Algorithm 1: Compute the event-based representation through EventFormer
Input: A list of n 2-dimensional pixel coordinates of the events at time I: Tt { yi) 1.m)
Output: Prediction vector Y; E R' where denotes the number of classes.
Hyperparameter: The dimension of the representation vector d, row size of the memory m, number of stacks for the Refine operator R. Initialize the memory representation with the learned initializer, Ma E Rm
1 मe < (x,,) = Va (cos(z, W,) sin(x,W?))
2 Hi-1 + Read(TT,
Read the past memornyreprresentation
3 foreach r E R do
Tt f- Refiinee (Te, m.);
Compute event interactions
5
Z, 2 TT;
6 X, + R(2,, H4-1);
D Compute the current states Compute new memory representation
M' + (r,te(M,,1,,.)
8 at sig sigmoid(( (
D Compute the update factor Update the memory
9 Mr = a.Mi-1 + (1 - M')
10 Mi
flat + Reshape M, into 1-dimensional vector;
Y, < -FPprea(MM)))
> Prediction at time t
12 return Y,.
used by a classification head for the recognition task. To facilitate these operations, EventFormer has three unique Operators: Read to retrieve the past representation, Write/Erase to update the memory with new representation, and, Ref ine to compute the spatial correlation among the events. Both Read and Write/Erase operators use multihead residual attention (Vaswani et al. (017)) as building block. The Refine operator adopts an efficient version of this building block (Shen all 202t) to address the quadratic memory and compute requirement (that can become computation- ally prohibitive for very fast-moving object generating large number events at a given time) of the traditional dot-product attention mechanism (Vaswani et all 2017). Algorithm I shows implementation of the overall Eventformer architecture.
Operator details: An Operator(A, B3) takes A Rxd and B ga input and maps them into the Q (query), K (key), and V (value) space using a linear transformation. The attention mechanism maps the Q outputs as follows:
(1) QK E Raxm measures the pairwise similarity of the query and key vectors. a(QK E Rnd weighted sum of the value vectors where the weights are computed using scaled softmax tivation a(-) sof tmax(./Vil), The multihead residual attention computes multiple attention by
(,nnQ,KKVV: = a(QKT)
4

---

Published as a conference paper at ICLR 2023
Multihead Residual Attention
Operator Buillding Block
Memory Operators
Memory Update
LayerNorm
Write/
Frite/Erase Erase Operator
PoodForward(F7
Part Memeeyy
Spalicooppee
Queeray
Key
Value
(Mu-i)
Layer
FR
Fi
FF.
Read Operator
Write
Erass
^f
ef
Mullibuud Attention
B
A
Query Key
Value
Part Momucy
(M...
Mirnara
(a)
(b)
(c)
(d)
Figure 4: Details of the memory operations. (a) block diagram of the multihead residual atten- tion, (b) composition of an Operator(A, B), (c) Read operator calculates query from positional embedding मi and key-value pair from past memory representation, Mr-1. The Write and Erase operator computes query from M,-1 and key-value pair from X, and, (d) An update to the as- sociative memory occurs with a linear combination of the new value be added M's and past representation Mr-1 weighted by an elementwise scaling factor generated by the Erase operator.
projecting Q, K. and V onto separate d/h dimensional space, respectively. The output is linear combination of h-different attentions (Attn(...,, .. a)) applied to each of these h projections:
MultiHead (Q, K, V: w. concatenate(O"... ("",,, W"
(2)
where O = Attn(QW,, KW VW"; a;)
(3)
w = (W7, W WTy E Rdd & and Wo E Rdd are learnable parameters. following residual block completes the operation and generates output, O as follows:
O Operator B) == LayerNorm(O,, + FF)(O)))
(4)
where O₁ LayerNorm ( MultiHead K,
K,V; V;
a))
(5)
LayerNorm is layer normalization function (Ba et al. 2016). Figure 4 illustrates these operations.
EventFormer generates positional embedding by mapping the event sequence from a low-dimensional (2D) space to higher-dimensional feature space. This is done by a learnable Fourier feature-based positional encoder. At any given time 6, the positional encoder,
Positional encoder:
II 24 R=x2 ne E Raxd parameterized by W, E R maps a list of 2-dimensional events, to d-dimensional feature space, n, using the following equation:
mt II(x;)) = (concatenate (?") sin(x;
(6)
Spatial correlation: To compute refined representation that considers spatial correlation among the events, we compute self-attention among the positional embedding n, by using the Refine operator. The efficient attention mechanism modifies the Equation aas follows:
EfficientAtta((. K, ai 0x) = a, (Q) (ak
(7)
where ay and aş denote the row-wise and column-wiss scaled softmax activation, respectively. A closer look at Equation reveals revvals that the memory and computation complexity has been reduced to (dd-+d22 and (d2)) respectively, which is linear with respect to the number evenns,, n. Finally, stack multiple Refine operators to compute Z, [xdd as their higher-order interactions.
Associative memory augmented recurrent module: key iiboccaation-bssee memory access architecture with separate association mechanism for state retrieval and update operation (Figure 4. We define M₁ E mm as the stored memory repre- sentation time t. The Read operator retrieves the past representation H4-1 Rx using the current event locations मe It computes the query vectors from n, and key, value vectors using the past stored representation Mr-1- The final retrieved state is weighted sum of the projected M
The proposed associative memory has query-
5

---

Published as a conference paper at ICLR 2023
Time progression
(a)
(b)
(c)
SSCJ
Figure Temporal evolution memory representation. (a) and (b) Change in memory represen- lation over time for samples from two different classes. (c) tSNE plot of the memory representation for 10 randomly selected classes from N-Caltech101 dataset.
where the weights computed through the association between the event locations (Tτ) and the abstract memory addresses (keys projected from M,-1). This implies that a particular memory representation gets more weight if its corresponding address (key) has higher similarity with the query (i.e., positional embedding). The retrieved H4-1 is used as the past hidden state of the recur- rent module, R, compute the current spatiotemporal representation, X, E Ra We use gated recurrent unit (GRU) as the recurrent module. The Write and Erase operator jointly update M, with new information. The Write operator computes the query vectors from M,-1 and the key- value pair from X₂ to query the location of the memory that needs to be updated and generate the new representation, MM' Rxd The purpose of Erase operator is calculate set of element- wise scaling factors, a E [Rexxd 0 < < 1] to control the amount of update at each memory location. Additional details can be found in Appendix 0 Finally, we update the memory using:
a (1- (1-at)M',
Classification head: A single feedforward layer FF (.pred(.) that maps the flattened memory represen- tation, M flat E mR to the prediction vector, Y; E IR where e is the number of classes.
3 EXPERIMENTS AND RESULTS
We evaluate the performance of our method on standard event-based object recognition benchmarks: N-Caltech101 and N-Cars (see Appendix for details). We consider two metrics: recognition accuracy and the number of floating point operations required to update the representation for each new event (MFLOPs/ev).
Datasets and metrics:
EventFormer follows end-to-end feed-forward layer-based implementation. We use 32x32 and 16x 16 as memory dimensions mRd) for N-Caltech10) and N-Cars dataset, re- spectively. Unless otherwise specified, all the multihead residual attention blocks use h (number of heads) = 4, and the number stacks for the Refine operator, R Additional details including training hyperparameters and time intervals be found in Appendix B
Implementation:
Representation Learning in Associative Memory: We first visualize the evolution the learned representation in the associative memory for different classes of 1-Caltech101 (Figure (5) Initially, the memory representation starts from the same initial state for all classes. As events are observed time, the memory states start form unique patterns for different classes. The tSNE plot in

---

Published as a conference paper at ICLR 2023
14
Cless Clas 2
STA
2013
280 1839 aED 1001 169
Pesll
11.2
300 300 5000 9300 1930 1300
A003
1900
5000
01.00
5230
Figure 6: Memory representation dynamics over sequential class update. (a) Change in pre- diction probability of the target class over time. (b) Change in memory representation when class changes occur. (c) Average memory update activity for a large-scale experiment similar to (a) and (b), and (d) their corresponding perception latency distribution.
1.0
0.95
5.35
3401
500
Number cunrletiee owarts
Profuct profle
+0.55
3093 (991 S000 1399 1300
0.80
8000
8500
0000
9500
10000
10500
1100
Number of cunullttvee events
(1)
Figure 7: Memory representation dynamics over sequential input transformation. (a) Change in prediction probability of the target class over time. (b) Change in memory representation when transformed input appears. (c) Average memory update activity for a large-scale experiment similar to (a) and (b), and (d) their corresponding perception latency distribution.
Figure shows that the learned representations for various classes form distinct cluster enabling separation of the class boundaries even with a relatively simple classifier head.
Temporal Update of the Learned Representation with Class Change: We study the temporal up- dates the memory states when samples from two different classes are passed to EventFormer in streaming manner (Figure 6). The memory updates its representation the object class changes, enabling accurate classification results over time. We randomly take 3000 samples from the Caltech10 101 dataset consisting of sequential class change (3000 unique class combinations). We plot the distribution of the number of events required for the target class output to reach a 0.6 confi- dence, hereafter referred to the perception latency (Figure (d)). The median perception latency is ~ 120 and ~ 60 events for the initial and new class, respectively. The perception latency for the initial class (red) has wider distribution since the memory has adapt its representation from the initial state. We also plot the mean and standard deviation of the Frobenius norm of the differ- ence between two successive memory representations || Mella hereafter referred to as the
memory update activity, for the same experiment (Figure 5 (c)). Higher memory update activity,
observed when developing the representation of the initial class, stabilizes with more events of the same class. The change in the class also results in increased memory update activity but less than the initial case. This shows that EventFormer can adapt to new representations in real-time and re-use important features already computed in the past thereby reducing perception latency.
Temporal Update of Learned Representation with Input Transformation: We study the temporal update of the memory states over time while objects from the same class but with different transfor- mations (rotation) are passed to EventFormer in a streaming manner (Figure With the incoming events from the rotated sample, observe degradation in prediction probability due to represen-

---

Published as a conference paper at ICLR 2023
(a)
(b)
0.8
0.002
0.0332
0.7
0.106
0.0339
Relative computation
0.6
OWrite/Erase
Refine
Read
0.413
(normalized)
0.413
Positional Encoder
Read
Recurrent Module
Refine
Write
Erase
100
1000
Figure 8: FLOP analysis. (a) Normalized FLOP contribution from the key components of Event- Former (for n=1). (b) Change in relative compute cost for the Operators with increasing n.
Associative memory
Separate write and erase operator
Refine operator
Experiment
Accuracy f
MFLOPs/ev
X
0.670
0.014
X
0.752
0.041
X
0.703
0.035
(d)
0.848
0.048
Table 1: Ablation study of various key components of EventFormer on the N-Caltech101 dataset.
tation update, which stabilizes over time. We consider 3000 random samples from N-Caltech101 with combinations of two random rotations between 10 to 30 degrees. We observe that the mem- ory update activity is less for input transformation of the same example (Figure 77 (c)), compared to completely new example (Figure (c)). This shows that EventFormer can preserve learned representation for class even under input transformation enabling less memory update activity.
Computational Complexity: We analytically derive the computational complexity model of Event- Former (details in Appendix C We study the FLOP contributions from the key components in Figure g For single event (n=1), most of the FLOP costs are from the memory update block due its (md2 + md + d²) compute complexity. The cost of memory update diminishes with larger while the Read and Ref ine operator starts to dominate (Figure (b)).
Ablation study on the components: We con- sider the experiments where EventFormer is implemented without (a): the associative mem- ory, (b): separate Write and Erase operator, and (c): Ref ine operator and compare the re- sults with (d): the complete EventFormer to
Hyperparameter
Accuracy 1 MFLOPs/ev J
0.759
0,042
R
0.848
0.048
4
0.864
0.061
analyze the impact of different components. For experiment (a), pool the X; to 1- dimensional vector (Vemprala et all, (2021) to deal with the variable-sized sequence. Also, we make the hidden size of the recurrent module m d so that the prpd receives the same-sized input. In experiment (b), we linearly project the output of the Write operator into a m2d space and split it into two equal-sized vectors to get M', and The results shown in Table I Merging the write and erase operation marginally reduces the MFLOPs/ev, but at the expense of less flexibility during memory updates and hence, lower accuracy. Without the Refine operator, the model saves more computation but lacks the higher-order interaction modeling among the events, causing marginal accuracy loss. Finally, absence of associative memory reduces maximum compu-
0.550
0.012
m(= d)
32
0.848
0.048
48
0.882
0.108
Table 2: Performance of EventFormer N- Caltech101 with different hyperparameters.
8

---

Published as a conference paper at ICLR 2023
N N-Caltech101
N-Cars
Methods
Representation
Asyne.
Accuracy
MFLOPSev
Accuracy
MFLOPs/ev +
H-First
Spike
0.054
0.561
Gabor-SNN
Spike
0,284
0.789
HOTS
Time-Surface
V
0,210
54.0
0.624
14.0
HATS
Time-Surface
V
0,642
4.3
0.902
0.03
DART
Time-Surface
0.664
EST
Event-Histogram
0.817
4150
0.925
1050
Matrix-LSTM
Event-Histogram
0.843
1580
0.926
1250
YOLE
Voxel-Grid
0.702
3659
0.927
328.16
AsyNet
Voxel-Grid
0.745
202
0,944
EvS-S
Graph
0.761
11.5
0.931
AEGNN
Graph
0.668
0.369
0.945
0.03
Ours
Unstructured Set
0.848
0.048
0.943
0.013
Table
Performance comparison with state-of-the-an event-based and dense methods.
tation but also causes the model to lose spatial information (due to the pooling operation) resulting in minimum performance.
Effect of Hyperparameters: We study the impact of hyperparameters R and m(= - d) (Table 2) We observe that increasing their values result in better performance but at the cost of higher MFLOPs/ev. Therefore, we follow initial setting (R=2 and m(= d)=32) have better balance between accuracy and compute cost while comparing with the state-of-the-art methods.
Comparison with the state-of-the-art (SoTA): We compare EventFormer with SoTA dense and event-based methods on these two datasets (Table 3. Following the previous work (Schaefer et all 2022), report the average MFLOPs/ev on window of 25000 events. Methods utilizing hand hand-crafted features: H-First (Orchard et all 2015).. Gabor-SNN (Bovik et al. 1990). HOTS (Lagorce et al., 2016). HATS (Sironi et all, 2018). DART (Ramesh et al.. 2019). require fewer MFLOPs/ev while performing worse than the data-driven methods. EST (Gehrig et al. 2019) and Matrix-I (Cannici all (020h) are synchronous methods that learn the optimal representation and achieve higher accuracy. However, they require compute-intensive intensive feature extractors to work on the learned representations. Whereas EventFormer does not require any complex feature extractors since the learned representation in the memory captures both the spatial and temporal information. can achieve similar performance with 30000x less computation. YOLE (Cannici et al. 2019) and AsyNet (Messikommer al., (020) modify the CNNs to enable asynchronous, sparse processing to reduce the compute cost. However, they do not consider the temporal information in the events. EvS- eL all (021c) and AEGNN(SScaaffrr all, 2022) graph-based methods with asynchronous and efficient graph nodes update. However, they use sub-sampling to restrict the graph size from growing prohibitively large, leading to suboptimal performance. Our method, on the other hand, inherently learns to leverage useful events to update the memory without requiring any redundant computation (such as radius search). The memory mechanism enables learning better temporal cor- relation compared to the traph-based methods. As result, our method achieves 9% better accuracy on N-Caltech101 while being x00x more efficient compared to (Li al. 2021c).
Computation Latency: The latency of EventFormer implemented with PyTorch on Nvidia RTX3090 is 4.5ms/event on N-Caltech101 dataset which 10x faster than the existing most effi cient method (Schaefer et all (52ms/eeen))))) Unlike their method, do not require searching over large graph to update the representation which reduces computational latency.
4 CONCLUSION AND FUTURE WORK
We propose a novel memory- auggente representation learning framework-EventFFomee for asyn- chronous and efficient ovent-based perception. EventFormer learns to store, retrieve and update its memory representation in the latent form of higher-order spatiotemporal dynamics of the events that allow it to achieve high performance with minimal compute cost. Future works include EventFormer more complex tasks including object detection, depth estimation, and optical flow prediction.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: arXiv:1607.0663 2016.
Raymond Baldwin, Ruixu Liu, Mohammed Mutlaq Almatrafi, Vijayan K Asari, and Keigo Hi- rakawa. Time-ordered recent event (tore) volumes for event cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Alan C. Bovik, Mariannaa Clark, and Wilson S. Geisler. Multichannel texture analysis using localized spatial filters. IEEE transactions on pattern analysis and machine intelligence, 12(1):55-73, 1990.
Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240 180 130 db 3 us latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(10):2333 2341, 2014. doi: 10.1 (0.1109//SSC22214442271
Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. Asynchronous convolu- tional networks for object detection in neuromorphic cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. A differentiable recur- rent surface for asynchronous event-based data. In European Conference on Computer Vision, pp- 136-152. Springer, 2020a.
Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. Matrix-Istm: a differ- entiable recurrent surface for asynchronous event-based data. In Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK, pp. 23-28, 2020b.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 2014.
Davide Falanga, Kevin Kleber, and Davide Scaramuzza. Dynamic obstacle avoidance for quadrotors with event cameras. Science Robotics. 5(40) 12, 2020.
Thomas Finateu, Atsumi Niwa, Daniel Matolin, Koya Tsuchimoto, Andrea Mascheroni, Etienne Reynaud, Pooria Mostafalu, Frederick Brady, Ludovic Chotard, Florian LeGoff. Hirotsugu Takahashi, Hayato Wakabayashi, Yusuke Oike, and Christoph Posch. 5.10 a 1280x720 back- illuminated stacked temporal contrast weent-based vision sensor with 4.86um pixels, 1 1.066geps readout, programmable event-rate controller and compressive data-formatting pipeline. In 2020 IEEE International Solid- State Circuits Conference (ISSCC), pp. 112-114, 2020. doi: D0.10099SSCCC19992220009999
Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154-180, 2020.
Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end learning representations for asynchronous event-based data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5633-56444 2019.
Mathias Gehrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. Dsec: stereo event camera dataset for driving scenarios. IEEE Robotics and Automation Letters, 6(3):4947-4954, 2021.
Jesse Hagenaars, Federico Paredes- Vallés, Sander M Bohté, and Guido CHE De Croon. Evolved neuromorphic control for high speed divergence-based landings mavs. IEEE Robotics and Automation Letters, 5(4):6239-6244 2020.
Javier Hidalgo-Cariin Carrió, Daniel Gehrig, and Davide Scaramuzza. Learning monocular dense depth from events. In 2020 International Conference 3D Vision (3DV), pp. 534 534-5433 IEEE, 2020.
10

---

Published as a conference paper at ICLR 2023
Geethan Karunaratne, Manuel Schmuck, Manuel Le Gallo, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi. Robust high-dimen sigh-dimenssonal memory-augmenteed neural networks. Nature commumications, 12(1):1-12, 2021.
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit lyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In International conference on machine learning, pp. 1378-1387. PMLR, 2016.
Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E Shi, and Ryad Benosman. Hots: hierarchy of event- based time-surfaces for pattern recognition. IEEE transactions pattern analysis and machine intelligence, 39(7):1346-1333 2016.
Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable fourier features for multi- dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34:15816-15822 2a
Yijin Li, Han Zhou, Bangbang Yang. Ye Zhang, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang- Graph-based asynchronous event processing for rapid object recognition. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), pp. 934-943, October 2021b.
Yijin Li, Han Zhou, Banghang Yang, Ye Zhang, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang. Graph-based asynchronous event processing for rapid object recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 934 943, 2021c
Chao Ma. Chunhua Shen, Anthony Dick, Qi Wu, Peng Wang, Anton van den Hengel, and Ian Reid. Visual question answering with memory-augmente networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6975-6984, 2018.
Ana I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso Gareía, and Davide Scaramuzza. Event-based vision meets deep learning on steering prediction for self-driving cars. Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 5419-5427, 2018.
Nico Messikommer, Daniel Gehrig, Antonio Loquercio, and Davide Scaramuzza. Event-based asyn- chronous sparse convolutional networks. In European Conference on Computer Vision, pp. 415- 431. Springer, 2020.
Anton Mitrokhin, Cornelia Fermüller, Chethan Parameshwara, and Yiannis Aloimonos. Event-based moving object detection and tracking. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1-9. IEEE, 2018.
Anton Mitrokhin, Chengxi Ye, Cornelia Fermüller, Yiannis Aloimonos, and Tobi Delbruck. Ev- imo: Motion segmentation dataset and learning pipeline for event cameras. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6105-6112. IEEE, 2019.
Anton Mitrokhin, Zhiyuan Hua, Cornelia Fermuller, and Yiannis Aloimonos. Learning visual mo- the tion segmentation using event surfaces. In Proceedings the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 14414-14423, 2020.
Garrick Orchard, Cedric Meyer, Ralph Etienne-Cumming Cummings, Christoph Posch, Nitish Thakor, and Ryad Benosman. Hfirst: A temporal approach to object recognition. IEEE transactions on pattern analysis and machine intelligence, 37(10):202--2044 2015.
Chethan M Parameshwara, Simin Li, Cornelia Fermüller, Nitin J Sanket, Matthew Evanusa, and Yiannis Aloimonos. Spikems: Deep spiking neural network for motion segmentation. In 202/ JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3414-3420. IEEE, 2021a
Chethan M Parameshwara, Nitin Sanket, Chahat Deep Singh, Comelia Fermüiler, and Yiannis Aloimonos. O-mms: Zero-shot multi-motion segmentation with monocular event camera. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. IEEE, 2021b.
11

---

Published as a conference paper at ICLR 2023
Bharath Ramesh, Hong Yang, Garrick Orchard, Ngoc Anh Le Thi, Shihao Zhang, and Cheng Xiang- Dart: distribution aware retinal transform for event-based cameras. IEEE transactions on pattern analysis and machine intelligence,
2019.
Simon Schaefer, Daniel Gehrig, and Davide Scaramuzza. Aegnn: Asynchronous event-based graph neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12371-12381 2022.
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference appli- cations of computer vision, pp. 3531-3539, 2021.
Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-base object classification. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1731-1740, 2018.
Bongki Son, Yunjae Suh, Sungho Kim, Heejae Jung, Jun-Seok Kim, Changwoo Shin, Keunju Park, Kyoobin Lee, Jinman Park, Jooyeon Woo, Yohan Roh, Hyunku Lee, Yibing Wang. Ilia Ovsiannikov, and Hyunsurk Ryu. 4.1 640x480 dynamic vision sensor with 9um pixel and 300meps address-event representation. 2017 IEEE International Solid-State Circuits Confer- doi: 10. N0.1100/SSSCC.211..777222
ence (ISSCC), pp. 66-67, 2017.
Yunjae Suh, Seungnam Choi, Masamichi Ito, Jeongseok Kim, Youngho Lee, Jongseok Seo, Heejae Jung. Dong-Hee Yeo, Seol Namgung, Jongwoo Bong, Sehoon Yoo, Seung-Hun Shin, Doowon Kwon, Pilkyu Kang, Seokho Kim, Hoonjoo Na, Kihyun Hwang, Changwoo Shin, Jun-Seok Kim, Paul K. J. Park, Joonseok Kim, Hyunsurk Ryu, and Yongin Park. A 1280x960 dynamic vision sensor with a 4.95-um pixel pitch and motion artifact minimization. In 2020 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-5, 2020. doi: 10. N0.1109/SCAAS4533..222 9180436.
Sihao Sun, Giovanni Cioffi, Coen De Visser, and Davide Scaramuzza. Autonomous quadrotor flight despite rotor failure with onboard vision sensors: Frames vs. events. IEEE Robotics and Automa- tion Letters, 6(2):580-587, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Sai Vemprala, Sami Mian, and Ashish Kapoor. Representation learning for event-based visuomotor policies. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4712-4724. Curran Associates, Inc., 2021.
Qinyi Wang, Yexin Zhang, Junsong Yuan, and Yilong Lu. Space-time event clouds for gesture recog- nition: From rgb cameras to event cameras. In 2019 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 1826-1835. IEBE, 2019.
Alex Zihao Zhu, Dinesh Thakur, Tolga Özaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Dani- ilidis. The multivehicle stereo event camera dataset: An event camera dataset for 3d perception. IEEE Robotics and Automation Letters, 3(3):2032-200 20399, 2018.
Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Danilidis.. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 989-997, 2019.
12

---

Published as a conference paper at ICLR 2023
Supplement to "Associative Memory Augmented Asynchronous Spatiotemporal Representation Learning for Event-based Perception"
A APPENDIX
In this supplementary material, we provide experimental and additional details on EventFormer.
B DETAILS OF THE NOTATION USED IN THIS WORK
Type
Description
Notation
Dimension
number of events
dimension of went-representation row size the memory number of classes
d
Scaler parameters
m
positional encoder recurrent module
11(.)
Functions
R(..))
classifier head
FF FFpred
sequence of events at time positional embedding associative memory representation t retrieved state from the memory (past hidden state for the recurrent module) representation with higher order interaction among the events current spatiotemporal representation new memory representation at scaling factors to control memory update prediction at time
St
n x X d
M
m X d X d
Hi-1
Vector/Matrix
2.
d
n d
M' Oig Y
m X d
m d
1 1xc
Table 4: Different notations used to describe the operations of EventFormer
C DERIVATION OF COMPUTATIONAL COMPLEXITY MODEL OF EVENTFORMER
Here derive the computational complexity model of the proposed EventFormer architecture. Let us define n as the number of events to be processed within time-interval т, d as the dimension of the in the memory, h be the number of heads in the residual
the number of attention blocks (we use the same / in all the blocks), r as the number of stacks for the Refine operator, and be the number of class for the recognition task.
representation,
(i) Positional Encoder: Our positional encoder block consists of single matrix multiplication with the input, that to compute x, W we need 2nd multiplications and nd 1) additions, so total 3nd FLOPs.
(ii) Read Operator: Read operator consists of residual multihead attention block with h heads. To begin with, calculating the query matrix for each head, we need ni n())² multiplication and n $ additions, in total / (2 1) FLOPs. Similarly for the key and value matrix, need total 2m A (2 1) FLOPs. Then, to compute QKT, we need additional mn(2 -1) computation. Similarly the product (QKT) requires n (2m 1) FLOPs. Upto now for each head, we require
total FLOPP:
13

---

Published as a conference paper at ICLR 2023
d d h (2 1) + 2m (2 -1 + mn(2 d
d (2m 1),
- -1)
h
Therefore, for / heads, it becomes:
d nd(2 h - 1) 2md(2 1) + h2m - 1) + nd(2m - 1)
The following 2 residual connection involves pointwise addition of a d dimensional matrix and the out-projection involves an additional nd(2d 1) FLOPs. Finally, total FLOPs count for the Read(T.; becomes:
- hmm d 1111 1) + nd(2m 1) + 2nd + 2nd(2d - 1) nd(2 - 1) + 2md(2 - 1)
d
d
(iii) Recurrent Module: Our recurrent module GRU that involves matrix multiplication (update, reset, hidden) for the current input, सy and hidden state H each. Therefore matrix mul- tiplication in total followed by 5 elementwise addition and elementwise multiplication. Therefore, the total computation becomes:
6nd(2d 1) + 7nd
(iv) Refine Operator:
Unlike the Read(.,.) operator, it projects the same input, X, to query, key
and value space. Therefore, total computation for such project becomes 3n d à (2 1), followed by
additional 2 - 1) and n (2 1) FLOPs for K V, and Q(K computation. Therefore, for h heads and considering the additional cost for residual operations and out-projections, the final cost becomes:
3nd(2 d 1)
d² (2n 1) ndl2 d h 1) + 2nd 2nd(2d - 1)
here accounts for the number stacks.
(iv) Write and Erase Operator: Similar to the Read(...) operator, we can calculate the total FLOPS required for the Write( Zz) and Erase (M Z,) operation combined as follows:
d 1) + h2 d 2 md(22 d - 1) 2nd(2
1) + md(2n 1) + 2md + 2md -1) + 3md
here, the additional 3md accounts for the 2 pointwise addition and multiplication in Equation
The final output is calculated by vector(E m-ar)--m Rmdxc) mul- tiplication, which requires an additional c(2md 1) FLOPs.
(iv) Perception Output:
Total FL.OPs: The total FLOPs accounting all the above computation becomes:
14

---

Published as a conference paper at ICLR 2023
nd(2 d h 11-1 1) + 2md(2 1) hmm (2 d 1) + nd(2m 1) + 2nd 2nd(2d 1)
d
3rd
Positional Encoder
Read Operation
3nd(2 d
d (2n 1) + nd h - 1) + 2nd + 2nd(2d -
Gnd((2 - 1) + 7nd +r
1) +
h
h
Recurrent Module
Operator
1) 22 d h
1) + hamn(2 d h
1) + md(2n 2md + 2md(2d -1) 11) + 3md
+2 md((
h
Memory Update (includes both Write and Erase operation)
c(2md - 1)
Output (9)
D DETAILS OF THE DATASETS
N-Caltech101 Orchard et al. 2015) is converted to event space from standard frame-based dataset (Caltech101). It has 8246 event sequences from 101 class categories following the original Cal- tech101 dataset. N-Caltech10) uses an event camera to record event streams generated by moving images displayed on monitor.
N-Cars (Sironi all 2018) is real-world event-camera dataset with 24039 event sequences from 2-class instances (car or background). N-Cars utilizes an event camera mounted moving recording its surroundings in a real-world setting.
E TRAINING DETAILS
Data Preparation: For both the datasets, we use the official test set portion to report the result and split the remaining into 90% and 10% ratios for training and validation purposes. The event coordinates are normalized by dividing them by the height and width (180 x 240 for N-Caltech and 100 120 for N-Cars) of the frame resolution. We use 50 ms long sequence for each sample to reduce training complexity. Each sample is chunked by Ims window, resulting in 50 50-sequences for each sample.
Optimization: We use the standard categorical cross-entropy as the loss function to train the network with Adam (Kingma & Ba, 2014) with batch size 128 for N-Cars and 64 for N-Caltech101 with an initial learning rate of 1e that decreases by a factor of 5 after every 25 epochs. The memory representation is initialized by set of learnable parameters M, Rg We also use a dropout of 0.2 the memory representation before passing them to the final classification layer. To avoid gradient explosion during training, we use gradient clipping with a maximum gradient norm of 100.
F ADDITIONAL ABLATION EXPERIMENT
Impact of polarity: In our current formulation, we did not consider polarity information of the events during the event-encoding part. We conduct an additional experiment evaluate the impact of polarity by taking them as an additional input alongside the positional coordinates of the events. In this formulation, we consider &,- ((x,, Ms p.)] where P: denotes the polarity of the event the (I;, ys) location. At a given time f, while we use the same positional encoder, II to process Ti, we use separate MLP layer parameterized by Wpot E gdx1 to map the list of n event polarities, Pe to d-dimensional feature space, po,, Finally, add and po, together to generate a refined that now contains both the polarity and positional information. We keep the subsequent operations unchanged. Following the same training and evaluation procedure on N-Caltech101 dataset, this modified architecture achieved a slightly better accuracy of 0.849 (compared to the original 0.848) with negligible additional compute cost.
15

---

Published as a conference paper at ICLR 2023
EventFormer+ Dense Decoder
Dense Network
Eacoda
Duxdbe
Decoder
Atlantion
Monuryy
Part
skip omneetion
(a)
(b)
Figure 9: Network architectures for dense prediction tasks. (a) Convolution based hierarchical encoder-decoder structure with skip connections in between. (b) Possible adaptation of EventFormer to work with existing decoder structure with no skip-connection.
G FEASIBILITY STUDY OF APPLYING EVENTFORMER ON COMPLEX TASK
Motion Segmentation: Our primary focus in this work was to develop an event-based spa- tiotemporal representation learning framework and we used classification as an example task to demonstrate the efficacy of our method. In this section, we further evaluate our method on dense task-motion segmentation as exam- ple of tasks more complex than classification. Unlike the classification task, motion segmen- tation is a dense prediction task. Existing works [Mitrokhin al. 2019: Sanket all, 2020: Parameshwara et al. (021)) on such dense task leverage hierarchical encoder-decoder network architecture where the encoder encodes the high-dimensional space into a compact latent- space while the decoder decodes it back to its original shape. Therefore, applying Event- Former such dense tasks requires its own de- coder to reconstruct the dense output from its latent memory representation. While designing novel decoder architecture is out of the scope of this work, we use the existing convolution-basee decoder structure for this experiment. We reshape a x b) give it 2D-
IoU
MFLOPs/ev
Method
boxes
floor
(Encoder)
EV-IMO
0.70
0.59
4786.7
EVDodgeNet SpikeMS
0.67
0.61
2575.0
0.59
0.46
18.5
EventFormer (m=64,d=32)
0.47
0.44
0.064
EventFormer (m=64.d=64)
0.49
0.43
0.193
EventFormer (m=256,d=32)
0.55
0.53
0.160
Table 5: Comparison of preliminary Event- Former adaptation with other methods on EV- IMO dataset.
the memory representation M Id into M, Raxbxd (where positional bias and pass it to the decoder as shown in Figure. [0 that end-to-end training with the decoder enables it learn the required 2D-latent representation the encoded space.
Dataset and Experimental Setup: We train and evaluate our method on a sub-set of EV-IMO dataset (Mitrokhin all. 2019). EV-IMO consists of challenging scenarios where different objects move at varying speeds and directions. A monocular event- baset camera captures the motions. Since our primary objective for this experiment is only to provide some insights on how to scale our method on complex tasks, we use a subset of the available five different sequences (boxes and floor) to reduce the training time. We center-crop the events by a 256 x 256 spatial window (i.c., we do not consider the events that fall outside this window) and use 20ms long sequence for every sample. Each sequence is chunked by a 1ms temporal window resulting in 20 sequences per sample. We use Adam with learning rate of 5 10-3 and trained for 80 epochs. We consider the Intersection Union (IoU) and MFLOPs/ev as the performance metrics. Since all the methods (including ours) employ similar decoder structures, we consider only the encoder part compare the MFLOPs/ev.
16

---

Published as a conference paper at ICLR 2023
Aggregated Event Frames
(a)
0.7
Ground Truth Mask
0.4
t= 1ms
t= 2ms
t= 3m5
1 5.5m
[= 15ms
0.0
12
16
20
Ertn (ins)
Predicted
Aggregated Event Frames
(b)
Ground Truth Mask
0.6
6.5
t= 1ms
t= 2ms
t= 3ms
t= 5ms
t 15ms
6.0
12
16
20
time (mn4)
Predicted Masks
Figure 10: Qualitative motion-segmeentation performance on EV-IMO dataset.
Preliminary Performance: We compare the per- formance with existing state-of-the-arr dense and event-based motion- mnotion-semmentttion methods (Table.
0.7
0.5
(5). Both the EV-IMO (Mitrokhin et al. 2019) and EVDodgeNet (Sanket et al. 2020) are dense processing- methods that aggregate events into frame-based representation.
0.4
Dense
0.2
SpikeMS Parameshwara et all 2)22)) considers the incoming events as spikes and adopts a spiking-neural newwokk (SNN) based encoder-decoder structure to process them asynchronously. For our method, we consider three different settings:
0.1
Oure
0
12
16
20
time (ms)
Figure 11: Incremental prediction perfor- mance comparison. Our method can con- verge to its peak performance significantly faster than the existing methods.
64, b 8, d 32, m = 64, a = b 8, d = 64, and m = 256, = 16, d 32. We observe that the better performance of the dense methods comes with the cost of much higher computational complexities. Our method can achieve comparable performance with SpikeMS while being 115x more efficient. We also observe that the performance of our method improves with higher memory capacity across the space. An- other interesting observation is that the performance gain from higher m is much larger compared the higher d. This implies that representation with more fine-grained spatial information is nec- essary for dense prediction tasks. This is also helpful in terms of computational benefits since FLOP model has a sub-linear relationship with m. Our unstructured formulation enables us to per- form incremental prediction, even for dense tasks. Figure. | shows the comparison with existing methods on such incremental prediction tasks. For the dense method, we integrate the events with incremental time-window starting from Ims up to 20ms with Ims interval. As shown, method can outperform both methods at the early stage, requiring much fewer events. We also visualize the method
qualitative performance of our method in Figure. [00 Although
achieve high loU very fast, it fails to capture the local details of the object. This is because, unlike existing methods, our decoder can utilize any skip connections. Such skip connections are necessary for the ex- isting mncodr-ddcoddr network architectures to recover the local spatial information that may have been lost during the encoding process (Drozdzal et al., 2016). These results indicate that existing decoder architectures may not be optimum to work on our representation and novel architectural innovations are required this regard. We leave this as a potential future research direction.
17

---

Published as a conference paper at ICLR 2023
175
200
MFLOPs/ev
EventFormer FLOPS
175
150
150
125
125
100
100
75
50
75
2000
25
1750
1500
1250
250 500
500
1000
750
1000
1200
500
d
2500
250
m
750
2000
0
Figure 12: Scaling EventFormer compute cost with increasing m and d from 16 2048.
Scaling of Computational Complexity: Our compute model derived in Appendix C shows ex- ponential relation with while sub-linear relation with m. For more complex and dense tasks with higher spatial resolution, EventFormer may need to scale up its memory size at the cost of additional compute overhead. We conduct an additional experiment tional cost would increase with higher memory size, and the result is shown in Figure [2] We ob- serve that the compute cost increases at significantly lower rate for higher m compared to higher d. For both m d 2048, our EventFormer formulation requires about 210 MFLOPs/ev, which is still 10x lower compared to the existing dense encoder architecture of EveDodgeNet (Snkeet al., 2020).
better understand how our computa-
H LATENCY SCALING FOR LARGER EVENT COUNTS
We perform additional timing experiments un- derstand the change in latency with increasing number of events. Table Ø shows the result. We observe that our latency has highly sub-linear relationship with respect to the number of events. Also, the throughput increases with the number of events thanks to the parallel processing capabilities of the modern hardware accelerators (GPU in our case). We also measure the time required for a dense
Latency (ms)
Throughput (Kev/s)
1
4.53
0.22
10
4.62
2.16
100
4.78
20.92
1000
4.91
203.66
10000
7.54
1333.33
method (Matrix-LSTM (Cannici et all, (2020b) to process a block of 30000 events which is 35.93ms for N-Caltech101 dataset. is noteworthy to men- tion that Matrix-LSTM utilizes its own highly opti- mized CUDA kernels to reduce their latency signif- icantly. While developing such optimized kernel for EventFormer is out of the scope of this work, we also expect similar latency reduction for method through such software optimization.
20000
14.15
1413.42
30000
20.8
1442.31
Table 6: EventFormer latency and through- put with increasing number of events.
18

---

Published as a conference paper at ICLR 2023
N-Caltech101
N-Cars
Methods
Representation
Async. Accuracy
Latency(ms).
Accuracy
Latency(ms)
H1-First
Spike
0.054
0.561
Gabor-SNN
Spike
0.284
0.789
0.071 (Intel 17)
HOTS
Time-Sun Time-Surface
0.210
0.624
0.038 (Intel i7)
HATS
Time-Son Sune-Sufface
0.642
0.902
0.002 (Intel i7)
DART
Sune-Surface
V
0.664
EST
Event-Hie Histogram
X
0.817
0,925
Matrix-LSTM
Ovent-Histogralt
X
0.843
0.926
0.002(GTXX 1080T))
YOLE
Noxel-Grid
0.702
0.927
AsyNet
Voxel-Grid
0.745
0.944
EvS-S
Graph
0.761
0.004[Intel
0.931
AEGNN
Graph
0.668
0.945
Ours
Unstructured Set
0.848
0.0007 (RTX3090)
0.943
0.0005 (RTX3090)
Table 7: Lateney and Accuracy comparison with state-of-the- event-based and dense methods.
I DETAILS OF THE MEMORY OPERATORS:
During Read operation, we want to know: what are the past states at the current event location? To do so, we use the multihead residual attention block perform query-key-based associations in the memory. To be more specific, we query the past memory representation M using the positional embedding of the current event locations, πe. The complete retrieval of the past hidden representation, H4-1 involves the following operations:
Read:
H = Read M,-1) Mr-1)
= LayerNorm(O, FFF((O))
(10)
where O, = Layerrorrmmmt MultiHead(Q... Kr, Vr; w, w.a).
(11)
Here, Qr represents the query vector calculated from Te, and K,, V, represents the key and value vectors computed from the MM-1-
Similar to the Read operator, we adopt residual multi-head attention block for Write operator to calculate the new memory representation, MM,.. However, this time we compute the query vectors from M,- and key-value pair from the refined spatiotemporal representation, X, The idea here is that we want to query the location of the memory that needs to be updated while contents to updated are provided by the new representation.
Write and Erase:
M'. Write((Me-1, LayerNormmO FFF
(12)
where O = LayerNorm (M MultiHeaddQu.. Kw, Vwi a.)
(13)
Here, Que represents the query vector calculated from MM-1, and K V., represents the key and value vectors computed from the X. We also introduce Erase operator (follows the same operations of Write operator) that calculates set of element- wise scaling factors, ar {RRaxd 0 az S 1} to control the strength of update:
ar eignoid(Erase((MMi..))
(14)
J LATENCY COMPARISON:
direct comparison of the latency of our and other methods in Table 7 is challenging as differ- ent methods have provided their latency measurements using different hardware configurations and software setups. However, we still show the latency/event reported in the original papers for the methods and corresponding hardware configuration. A common methodology was used in all these works where the average latency/event was measured considering the average run time per sample divided by the average number of events per sample.
19

---

Published as a conference paper at ICLR 2023
REFERENCES
Michal Drozdzal, Eugene Vorontsov, Gabriel Chartrand, Samuel Kadoury, and Chris Pal. The impor- tance of skip connections in biomedical image segmentation. In Deep learning and data labeling for medical applications, pp. 179-187. Springer, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Anton Mitrokhin, Chengxi Ye, Coreelaa Fermüller, Yiannis Aloimonos, and Tobi Delbruck. Ev- imo: Motion segmentation dataset and learning pipeline for event cameras. In 2019 IEEF/RSI International Conference on Intelligent Robots and Systems (IROS), pp. 6105-6112 12. IEEE, 2019.
Chethan M Parameshwara, Simin Li, Comella Fermüller, Nitin J Sanket, Matthew S Evanusa, and Yiannis Aloimonos. Spikems: Deep spiking neural network for motion segmentation. 2021 JEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3414-3420. IEEE, 2021.
Nitin Sanket, Chethan M Parameshwara, Chahat Deep Singh, Ashwin Kuruttukulam, Cornelia Fermüller, Davide Scaramuzza, and Yiannis Aloimonos. Evdodgenet: Deep dynamic obstacle dodging with event cameras. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 10651-10657 IEEE, 2020.
20


Reviews:
[{"review_id": "wnO1F3xrfjv", "reviewer": "Reviewer_AhYC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a high performance asynchronous encoding scheme for event-based camera data, which uses associative memory and preserves the spatio-temporal relationships between events in a compact and efficient manner. The authors evaluate the pipeline on recognition tasks, and achieve high speedups without sacrificing the accuracy of the predictions.\n", "summary_of_the_review": "The asynchronous frame-less nature of the event stream is a big problem for event-based processing, and most modern methods either sacrifice information contained in the events, or the performance. I believe the paper has a potential to improve on the current state of the art, at least in terms of the encoding of the events. The only major weakness I see is the lack of focus / evaluation on depth and motion, with motion estimation (egomotion or segmentation) being the major applications of the event camera. But, the approach is still valuable and I would be willing to accept this paper provided the authors make their method available.", "strength_and_weaknesses": "The paper presents and efficient way to encode and incrementally update the event stream, which is a pressing issue for any event-based processing pipeline; I believe the method could have a big impact if made publicly available to the community.\n\nThe evaluation is performed on N-Caltech101 and N-Cars for recognition tasks. While valid, the event camera (specifically - spatio-temporal event relations) emphasizes motion over the appearance, and an evaluation on any egomotion dataset or at least a more challenging recognition / segmentation would strengthen the paper.\nWe suggest for the future work (or, a revision of this paper if feasible) authors to add EVIMO (https://better-flow.github.io/evimo/) dataset to evaluation - egomotion, segmentation and depth estimation are all valuable applications for event cameras. MVSEC (https://daniilidis-group.github.io/mvsec/) is another\ngreat source of data, with both egomotion and depth.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cNwjtjQmleY", "reviewer": "Reviewer_rsty", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a transformer memory architecture for processing events from an event camera. A block of events are converted into positional embeddings, passed through self attention, and then fused with a memory tensor which is updated with events over time. The memory tensor can then be decoded at any time to produce the desired output. Qualitative examples are provided for the dynamics of the learned memory, and experiments are performed on image classification on the N-Caltech 101 and N-Cars datasets, where the proposed method outperforms previous state of the art, at significantly lower MFLOPs/ev.", "summary_of_the_review": "I believe that this work can serve as a good model for future memory-based event processing architectures. The overall architecture is simple and well motivated, and the experiments show improvements over prior work. Overall, my main concerns are the claims of efficiency given the latency results and the ability to scale up the method.", "strength_and_weaknesses": "The motivation of having a stored memory for event processing is compelling, and the proposed work uses many of the current techniques that have demonstrated efficacy (attention, RNNS). The provided qualitative examples are quite interesting, showing that the memory differentiates between different classes for classification from t-SNE, and can update its state when the input changes to a different class. The results from the classification experiments are also strong, showing strong classification accuracy mostly above prior state of the art, at significantly less compute. \n\nThe method is presented as asynchronous, but does seem to require some level of batching of events at input. The Refine stage requires pairwise interactions between the input events, so it seems like this stage would have no effect if a single event was passed in. The computational latency of this method also seems quite high (4.5ms per event), as a single 30k event block would take over two seconds. This seems to go against the arguments presented in Table 3, where efficiency is reported in terms of MFLOPs/ev. For 'synchronous' methods which batch events into tensor representations, they should be able to process 30k event blocks at > 10Hz. Obviously there are tradeoffs for each, but currently the synchronous methods are only described as being more expensive.\n\nAnother concern is whether this method would scale up for more difficult tasks. It seems like the memory tensor is at most 32 x 32, which makes sense for classification. However, if a dense task was provided (e.g. optical flow), or something like detection, would it be sufficient to scale up the size of the memory? At what point does this become prohibitively expensive? 32 x 32 seems fairly small for a learned representation.\n\nThe outline of the method mostly is clear, but the Read, Write and Erase sections are a little hard to follow. While the other parts are clearly outlined in equations, these three are condensed entirely in text under 'Associative memory augmented recurrent module', and it's not clear what goes where. Some clarifications here could help the reader.\n\nTable 3: AEGNN slightly outperforms the proposed method for N-Cars and should be bolded.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U2p1DtB01z", "reviewer": "Reviewer_rHjG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new event representation framework EventFormer, which can process sparse events asynchronously and efficiently. The author proposes a memory-based mechanism to retrieve the past representation, store the memory with new representation and update the hidden states of associative memory. The paper claims that the method obtains high performance with minimal compute cost on the N-Caltech101 and N-Cars dataset.", "summary_of_the_review": "Overall, I think the design is interesting and seems to work to some extent. However, authors are suggest to apply the proposed representation on more advanced tasks such as object detection and optical flow estimation instead of only classification.", "strength_and_weaknesses": "Pros: \n1. This paper is well written and clear in general. \n2. The proposed EventFormer seems to perform well on two event-based object recognition datasets. \n3. The hybrid transformer and rnn architecture design seems novel.\n\nCons:\n1. The input of EventFormer is a list of events {(x_i, y_i)}, does it mean that the polarity information of events is lost? Does it have any influence to the performance?\n2. As is mentioned in paper, the MFLOPs/ev on average window of 25000 events are reported. However, 50ms long sequence is used for each sample in supplementary material. What exactly does that mean?\n3. The resolution of N-Caltech101 should be 180×240, but the paper says 180×260, is this a typo or is there pre-processing?\n4. How is the efficiency of the proposed method when dealing with high spatial resolution events such as the ones produced by Prophesee Gen4? This is especially important for tasks such as object detection, semantic segmentation and optical flow estimation.\n5. Authors are suggested to provide inference speed other than flops of models.\n6. What about performance of the proposed method on other tasks such as object detection, depth estimation and optical flow estimation?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NgTltiXqn55", "reviewer": "Reviewer_wCjB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A memory-augmented representation learning model, EventFormer, is proposed for asynchronous event-based perception. To achieve this goal, the EventFormer learns to store, retrieve and update its memory representation in the latent form of higher-order spatiotemporal dynamics of the events. The idea is interesting and also obtains good performance on two classification tasks. As the drawbacks of this paper, many typos can be found in the manuscript. I also wonder how about the performance on the long-term event representation learning and also other challenging tasks, such as event-based object detection, event-based visual tracking, etc. ", "summary_of_the_review": "A memory-augmented representation learning model, EventFormer, is proposed for asynchronous event-based perception. To achieve this goal, the EventFormer learns to store, retrieve and update its memory representation in the latent form of higher-order spatiotemporal dynamics of the events. The idea is interesting and also obtains good performance on two classification tasks. \nThe following issues can be addressed in future versions to further improve the quality of this work. \n1. the writing of this paper still needs further improvements, many typos can be found, even in the figure; \n2. the performance on the long-term event streams, and challenging tasks is not verified. ", "strength_and_weaknesses": "Strength \n1. the idea to adopt the Transformer and memory scheme for event-based representation learning is interesting and new for this problem. \n2. the model works well on short-term and simple classification datasets, including n-caltech101, and n-cars. \n\n\nWeaknesses\n1. the writing of this paper still needs further improvements, many typos can be found, even in the figure; \n2. the performance on the long-term event streams, and challenging tasks is not verified. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%