<figure><img src="image_1.png" /></figure>

HAL
open science

# Event-Based localization with Spiking Neural Networks: How Much is needed ?

Geoffroy Keime, Nicolas Cuperlier, Benoit R Cottereau

<figure><img src="image_2.png" /></figure>

## To cite this version:

Geoffroy Keime, Nicolas Cuperlier, Benoit R Cottereau. Event-Based localization with Spiking Neural Networks: How Much is needed ?. SINFRA, Jun 2024, Singapour, France. hal-04925955

HAL Id: hal-04925955

https://hal.science/hal-04925955v1

Submitted on 3 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Open licence - etalab
# Event-Based localization with Spiking Neural Networks: How Much is needed ?

**Geoffroy Keime**¹,², **Nicolas Cuperlier**¹,³, **Benoit R. Cottereau**¹,²

1: IPAL, CNRS IRL 2955, Singapore 2: CerCo, CNRS UMR 5549, Toulouse III Université 3: ETIS, ENSEA CNRS UMR8051 CY Cergy Paris Université
[geoffroy.keime@cnrs.fr](mailto:geoffroy.keime@cnrs.fr)

The aim of this project is to develop **bio-inspired and energy efficient computational models for navigation** in space. These models should be able to achieve **Visual Place Recognition (VPR), Odometry** and finally **Visual Simultaneous Localization and Mapping (V-SLAM)** from **spiking neural networks (SNNs)** processing data collected by an **event-based camera**.

**Keywords:** Navigation, Event-based camera, Spiking Neural Network, Unsupervised learning, Bio-inspiration

<figure><img src="image_4.png" /></figure>

**Hypothesis:** Our proposed architecture (SNNs with Events) can provide very good performances while only using a limited number of spikes.

## Event-based cameras

<figure><img src="image_5.png" /></figure>

standard
camera
output:

<figure><img src="image_6.png" /></figure>

event
camera
output:

<figure><img src="image_7.png" /></figure>

Send an event (**spike**) every time a luminance increment ('ON' event) or decrement ('OFF' event) is detected in the visual scene.

Very high temporal resolution (~1µs) and robustness to motion blur.

## Spiking Neural Network

<figure><img src="image_8.png" /></figure>

Layer (N - 1)

Layer N

<figure><img src="image_9.png" /></figure>

Leaky integrate and fire (LIF)

<figure><img src="image_10.png" /></figure>

Hierarchical Model with Leaky integrate and fire (LIF) neurons.
Unsupervised learning using an Hebbian plasticity rule: the Spike
Timing-Dependent Plasticity (STDP)

## Bibliography

S. Hussaini, M. Milford, and T. Fischer, "Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments," IEEE Robot. Autom. Lett., vol. 7, no. 2, pp. 4094-4101, Apr. 2022, doi: 10.1109/LRA.2022.3149030

P. U. Diehl and M. Cook, "Unsupervised learning of digit recognition using spike-timing-dependent plasticity," Front. Comput. Neurosci., vol. 9, Aug. 2015, doi: 10.3389/fncom.2015.00099.

E. O. Neftci, H. Mostafa, and F. Zenke, "Surrogate Gradient Learning in Spiking Neural Networks." arXiv, May 03, 2019. Accessed: May 30, 2024. [Online].

W. Fang et al., "SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence," Sci. Adv., vol. 9, no. 40, p. eadi1480, Oct. 2023, doi: 10.1126/sciadv.adi1480

A. Z. Zhu, D. Thakur, T. Ozaslan, B. Pfrommer, V. Kumar, et al. Daniilidis, "The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception," IEEE Robot. Autom. Lett., vol. 3, no. 3, p. 2032-2039, juill. 2018, doi: 10.1109/LRA.2018.2800793.

## Datasets

* MVSEC

* Nordland

* Brisbane

* NYC Event VPR

<figure><img src="image_11.png" /></figure>

Event Based data captured during navigation with visual changes

## Preliminary Results on VPR

* **Unsupervised learning** with STDP using different places captured under **variable conditions** (seasons, weather...)

* Training leads to place-selective neurons but **validation on larger datasets is needed**.

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

Synaptic weights of one place-selective neuron after STDP learning (left) and associated place (right)

<figure><img src="image_14.png" /></figure>

## Take Home Message

We aim to develop an **energy-efficient** computational model for navigation based on SNNs processing event data.

Preliminary results suggest that this model can achieve **very good performances** at VPR using a **limited number of spikes**.

## What's Next?

* Finalizing VPR with SNN and Events (Tests + Validation on different datasets)

* Comparisons with **neurobiological data** (place cells, conjonctive neurons...)

* Add **Optical Flow** and **Odometry** (event-based extraction of optic flow and inner-motion)

* How to autonomously detect and **sample space**