Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A memory-augmented representation learning model, EventFormer, is proposed for asynchronous event-based perception. To achieve this goal, the EventFormer learns to store, retrieve and update its memory representation in the latent form of higher-order spatiotemporal dynamics of the events. The idea is interesting and also obtains good performance on two classification tasks. As the drawbacks of this paper, many typos can be found in the manuscript. I also wonder how about the performance on the long-term event representation learning and also other challenging tasks, such as event-based object detection, event-based visual tracking, etc. 
strength_and_weaknesses: Strength 
1. the idea to adopt the Transformer and memory scheme for event-based representation learning is interesting and new for this problem. 
2. the model works well on short-term and simple classification datasets, including n-caltech101, and n-cars. 


Weaknesses
1. the writing of this paper still needs further improvements, many typos can be found, even in the figure; 
2. the performance on the long-term event streams, and challenging tasks is not verified. 
clarity,_quality,_novelty_and_reproducibility: this paper is clearly written, but with some typos. The implementation details are enough for re-implementation. 
summary_of_the_review: A memory-augmented representation learning model, EventFormer, is proposed for asynchronous event-based perception. To achieve this goal, the EventFormer learns to store, retrieve and update its memory representation in the latent form of higher-order spatiotemporal dynamics of the events. The idea is interesting and also obtains good performance on two classification tasks. 
The following issues can be addressed in future versions to further improve the quality of this work. 
1. the writing of this paper still needs further improvements, many typos can be found, even in the figure; 
2. the performance on the long-term event streams, and challenging tasks is not verified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new event representation framework EventFormer, which can process sparse events asynchronously and efficiently. The author proposes a memory-based mechanism to retrieve the past representation, store the memory with new representation and update the hidden states of associative memory. The paper claims that the method obtains high performance with minimal compute cost on the N-Caltech101 and N-Cars dataset.
strength_and_weaknesses: Pros: 
1. This paper is well written and clear in general. 
2. The proposed EventFormer seems to perform well on two event-based object recognition datasets. 
3. The hybrid transformer and rnn architecture design seems novel.

Cons:
1. The input of EventFormer is a list of events {(x_i, y_i)}, does it mean that the polarity information of events is lost? Does it have any influence to the performance?
2. As is mentioned in paper, the MFLOPs/ev on average window of 25000 events are reported. However, 50ms long sequence is used for each sample in supplementary material. What exactly does that mean?
3. The resolution of N-Caltech101 should be 180×240, but the paper says 180×260, is this a typo or is there pre-processing?
4. How is the efficiency of the proposed method when dealing with high spatial resolution events such as the ones produced by Prophesee Gen4? This is especially important for tasks such as object detection, semantic segmentation and optical flow estimation.
5. Authors are suggested to provide inference speed other than flops of models.
6. What about performance of the proposed method on other tasks such as object detection, depth estimation and optical flow estimation?

clarity,_quality,_novelty_and_reproducibility: The paper is well written and the hybrid transformer and rnn design has certain novelty.
summary_of_the_review: Overall, I think the design is interesting and seems to work to some extent. However, authors are suggest to apply the proposed representation on more advanced tasks such as object detection and optical flow estimation instead of only classification.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a high performance asynchronous encoding scheme for event-based camera data, which uses associative memory and preserves the spatio-temporal relationships between events in a compact and efficient manner. The authors evaluate the pipeline on recognition tasks, and achieve high speedups without sacrificing the accuracy of the predictions.

strength_and_weaknesses: The paper presents and efficient way to encode and incrementally update the event stream, which is a pressing issue for any event-based processing pipeline; I believe the method could have a big impact if made publicly available to the community.

The evaluation is performed on N-Caltech101 and N-Cars for recognition tasks. While valid, the event camera (specifically - spatio-temporal event relations) emphasizes motion over the appearance, and an evaluation on any egomotion dataset or at least a more challenging recognition / segmentation would strengthen the paper.
We suggest for the future work (or, a revision of this paper if feasible) authors to add EVIMO (https://better-flow.github.io/evimo/) dataset to evaluation - egomotion, segmentation and depth estimation are all valuable applications for event cameras. MVSEC (https://daniilidis-group.github.io/mvsec/) is another
great source of data, with both egomotion and depth.

clarity,_quality,_novelty_and_reproducibility: I really enjoyed the quality of the illustrations, and the literature review is very detailed and gives the reader a good overview of the existing approaches to event encoding. Maybe the review section could benefit from more diversity, especially among papers that tackle egomotion, motion segmentation and depth - for example
 - Learning visual motion segmentation using event surfaces (3D, graph representation)
 - https://arxiv.org/pdf/1903.07520.pdf (dense image-like representation)
 - https://arxiv.org/pdf/1803.04523.pdf (image-like representation, classic pipeline)

In regards to novelty, I think the paper presents a relatively new (at least, to the field) problem and addresses it directly, while most of the existing papers to not consider the problem of event encoding as an independent one.
summary_of_the_review: The asynchronous frame-less nature of the event stream is a big problem for event-based processing, and most modern methods either sacrifice information contained in the events, or the performance. I believe the paper has a potential to improve on the current state of the art, at least in terms of the encoding of the events. The only major weakness I see is the lack of focus / evaluation on depth and motion, with motion estimation (egomotion or segmentation) being the major applications of the event camera. But, the approach is still valuable and I would be willing to accept this paper provided the authors make their method available.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a transformer memory architecture for processing events from an event camera. A block of events are converted into positional embeddings, passed through self attention, and then fused with a memory tensor which is updated with events over time. The memory tensor can then be decoded at any time to produce the desired output. Qualitative examples are provided for the dynamics of the learned memory, and experiments are performed on image classification on the N-Caltech 101 and N-Cars datasets, where the proposed method outperforms previous state of the art, at significantly lower MFLOPs/ev.
strength_and_weaknesses: The motivation of having a stored memory for event processing is compelling, and the proposed work uses many of the current techniques that have demonstrated efficacy (attention, RNNS). The provided qualitative examples are quite interesting, showing that the memory differentiates between different classes for classification from t-SNE, and can update its state when the input changes to a different class. The results from the classification experiments are also strong, showing strong classification accuracy mostly above prior state of the art, at significantly less compute. 

The method is presented as asynchronous, but does seem to require some level of batching of events at input. The Refine stage requires pairwise interactions between the input events, so it seems like this stage would have no effect if a single event was passed in. The computational latency of this method also seems quite high (4.5ms per event), as a single 30k event block would take over two seconds. This seems to go against the arguments presented in Table 3, where efficiency is reported in terms of MFLOPs/ev. For 'synchronous' methods which batch events into tensor representations, they should be able to process 30k event blocks at > 10Hz. Obviously there are tradeoffs for each, but currently the synchronous methods are only described as being more expensive.

Another concern is whether this method would scale up for more difficult tasks. It seems like the memory tensor is at most 32 x 32, which makes sense for classification. However, if a dense task was provided (e.g. optical flow), or something like detection, would it be sufficient to scale up the size of the memory? At what point does this become prohibitively expensive? 32 x 32 seems fairly small for a learned representation.

The outline of the method mostly is clear, but the Read, Write and Erase sections are a little hard to follow. While the other parts are clearly outlined in equations, these three are condensed entirely in text under 'Associative memory augmented recurrent module', and it's not clear what goes where. Some clarifications here could help the reader.

Table 3: AEGNN slightly outperforms the proposed method for N-Cars and should be bolded.
clarity,_quality,_novelty_and_reproducibility: The method is mostly quite clear, although some improvements in the read/write/erase section would be appreciated. Overall, the method is compelling, and shows improvements over prior state of the art. 
summary_of_the_review: I believe that this work can serve as a good model for future memory-based event processing architectures. The overall architecture is simple and well motivated, and the experiments show improvements over prior work. Overall, my main concerns are the claims of efficiency given the latency results and the ability to scale up the method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents EventFormer, a representation learning framework for asynchronous spatiotemporal representation that uses external associative memory for efficient event-based processing.
The authors revised the paper to address all major concerns -- especially those regarding timing --  of the reviewers and included a series of appendices to further strengthen their contribution.
The idea is novel and I like the approach which, despite few limitations, will make a very interesting addition to the conference.
justification_for_why_not_higher_score: Interesting work but I am just not sure it would be good for an oral.
Decision: Accept: notable-top-25%