<figure><img src="image_1.png" /></figure>

HAL
open science

# Modélisation et exploration d'architectures neuromorphiques pour les systèmes embarqués haute-performance

Edgar Lemaire

► To cite this version:

Edgar Lemaire. Modélisation et exploration d'architectures neuromorphiques pour les systèmes embarqués haute-performance. Réseau de neurones [cs.NE]. Université Côte d'Azur, 2022. Français.
NNT: 2022COAZ4009 . tel-03665844v2

HAL Id: tel-03665844

https://theses.hal.science/tel-03665844v2

Submitted on 24 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
UNIVERSITÉ
CÔTE D'AZUR

<figure><img src="image_2.png" /></figure>

ÉCOLE DOCTORALE
SCIENCES ET
TECHNOLOGIES DE
L'INFORMATION ET DE
LA COMMUNICATION

THALES

## THÈSE DE DOCTORAT

# Modélisation et Conception d'Architectures Neuromorphiques pour les Systèmes Embarqués Haute-Performance

### Edgar LEMAIRE

Laboratoire d'Antennes, Électronique et Télécommunications (LEAT) & Thales Research & Technology (TRT-Fr)

**Présentée en vue de l'obtention du grade de docteur en Electronique d'Université Côte d'Azur**

**Dirigée par** : Benoît Miramond, LEAT

**Co-Dirigée par** : Sébastien Bilavarn, LEAT

**Co-Encadrée par** : Hadi Saoud, Thales

**Soutenue le** : 08 Mars 2022

**Devant le jury, composé de :**

Simon Thorpe, Dir. de Recherche, CerCO

Gilles Sassatelli, Dir. de Recherche, LIRMM

Olivier Bichler, Ingé. Chercheur, CEA-LIST

Jean Martinet, Professeur, I3S

Philippe Millet, Resp. pôle Innovation, Nexter

Hadi Saoud, Ingé. Chercheur, Thales

Benoît Miramond, Professeur, LEAT

Sébastien Bilavarn, Maitre de Conf., LEAT

Cifre K

anRT
ASSOCIATION NATIONALE
RECHERCHE TECHNOLOGIE
# Modélisation et Conception d'Architectures Neuromorphiques pour les Systèmes Embarqués Haute-Performance

Jury :

Rapporteurs

Simon Thorpe, Directeur de Recherche, Centre de Recherche Cerveau et Cognition (CerCO) - CNRS/Université Toulouse 3 Paul Sabatier

Gilles Sassatelli, Directeur de Recherche, Laboratoire d'Informatique, de Robotique et de Microélectronique de Montpellier (LIRMM) – CNRS/Université Montpellier 2

Examinateurs :

Jean Martinet, Professeur des Universités, Laboratoire d'Informatique, Signaux et Systèmes de Sophia antipolis (I3S) – CNRS/Université Côte d'Azur

Olivier Bichler, Ingénieur Chercheur, Laboratoire d'Intégration des Systèmes et Technologies (LIST) – CEA Tech

Philippe Millet, Responsable du Pôle Innovation, Nexter

Hadi Saoud, Ingénieur Chercheur, Thales Research & Technology (TRT), Thales

Benoît Miramond, Professeur des Universités, Laboratoire d'Eléctronique, Antenne et Télécommunications (LEAT) – CNRS/Université Côte d'Azur

Sébastien Bilavarn, Maître de Conférences, Laboratoire d'Eléctron
# Résumé en Français

Les réseaux de neurones profonds ont permis des progrès sans précédent dans le domaine de l'apprentissage automatique. En imitant le calcul parallèle et distribué du cerveau, de tels algorithmes permettant en effet d'émuler n'importe quelle fonction, jusqu'aux plus complexes. Leurs applications vont de la vision par ordinateur (classification, segmentation, détection...), au traitement du language naturel en passant par la prédiction de séries temporelles. Dans tous ces domaines, les réseaux de neurones profonds n'ont cessé de repousser les limites de l'intelligence artificielle. D'autre part, nous assistons par ailleurs à l'émergence progressive de l'internet des objets (Internet of Things, IoT). Ces systèmes embarqués connectés ont un besoin grandissant en capacité de traitement, et les réseaux de neurones profonds semblent alors tout indiqués pour accomplir cette tâche. Qu'il s'agisse de navigation autonome dans des drones ou dans des voitures sans pilote, de reconnaissance faciale dans les téléphones portables ou plus généralement de traitement périphérique (Edge Computing) dans les réseaux de capteurs, les applications embarquées de l'apprentissage profond sont déjà nombreuses. Cependant, les algorithmes neuronaux sont complexes et particulièrement gourmands en ressources de calcul. En matière de consommation énergétique, ces derniers semblent en effet incompatibles avec la nature contrainte des systèmes d'IoT. Dans cette optique, l'approche de l'Électronique Neuromorphique consiste à s'inspirer du cerveau biologique pour en mimer l'efficacité énergétique. En effet, il s'agit du modèle de processeur neuronal le plus abouti à notre connaissance, affichant une consommation d'à peine 20 watts. À travers l'utilisation de modèles de neurones bio-inspirés proposés par les neurosciences computационnelles, l'approche Neuromorphique a pour objectif de réduire la consommation énergétique de l'intelligence artificielle embarquée. Cette approche repose sur des arguments prometteurs: la simplicité du calcul impulsionnel, l'aspect binaire des synapses et l'encodage "épars" (sparse). Cependant, la littérature scientifique spécialisée semble manquer de comparaison étendue de l'impact du domaine de codage sur l'efficacité énergétique des accélérateurs neuronaux. De ce fait, la contribution principale de cette thèse est une comparaison détaillée des domaines de codage impulsionnels et formels sur FPGA. Cette étude implique plusieurs cas d'usage et différents niveaux de parallélisme. Ce volet de la thèse a été mené a bien au moyen du développement d'un estimateur de ressources et d'énergie pour les architectures neuronales sur FPGA. Cet outil permet d'accélérer l'exploration de l'espace des applications. D'autre part, nous proposons de quantifier les gains potentiels au moyen d'une métrique et d'un modèle de haut niveau: le ratio d'activité synaptique. Cette métrique permet d'évaluer rapidement les gains potentiels d'énergies offerts par le domaine impulsionnel pour une application donnée. Nous l'utilisons notamment pour trouver de nouvelles familles de réseaux impulsionnels prometteuses en termes énergétiques. Enfin, nous mettons à profit les enseignements de cette étude pour proposer l'accélération neuronale hybride, une architecture mélant les deux domaines d'encodage (impulsionnel et formel). L'architecture a été embarquée à bord du satellite OPS-SAT (lancé en Décembre 2019) et testée en vol. Ce faisant, il s'agit de la toute première architecture neuromorphique fonctionnelle dans l'éspace.

*Mots Clefs* - Réseaux de Neurones Artificiels, Réseaux de Neurones Impulsionnels, Eléctronique Numérique, Architecture Matérielle, Modélisation, Consommation énergétique, Systèmes Embarqués, Informatique en Périphérie, Eléctronique Neuromorphique, FPGA
Abstract

Deep Neural Networks have recently pushed unprecedented progress in the field of Machine Learn-
ing. Those algorithms mimic the parallel and distributed computation paradigm of the biological
brain to fit any desired functions, including the most complex ones. The range of applications for
Deep Learning is very wide: from computer vision (classification, segmentation, detection...) to
natural language processing and time-series prediction. Those domains were all greatly impacted
by Deep Neural Networks, since they are continuously pushing back the limits of Artificial Intel-
ligence. On the other hand, the emergence of the Internet of Things in recent years have opened
a brand new range of applications for Deep Learning. Indeed, the processing abilities of those
algorithms is particularly appealing for such autonomous systems like IoT. From navigation and
obstacle detection in drones and self-driving cars, to face recognition in smartphones, or more
generally for Edge Computing in networks of sensors, Deep Learning seems all indicated for the
task. However, Deep Learning models are very complex and energy-intensive. In terms of energy
consumption, they appear not to comply with the highly-constrained nature of embedded systems.
To cope with this issue, the Neuromorphic Computing approach consists in taking inspiration from
the biological brain. Our brain is indeed the most efficient neural processor anyone has ever heard
of, with only 20W of average power consumption. By using bio-inspired models from Computa-
tional Neurosciences, the Neuromorphic approach thus aims in reproducing the energy efficiency of
the brain. This choice is backed by several objective assessments, like the hardware-friendliness of
the spiking synaptic operation, the sparse computation or the lightweight communication between
neurons. However, the literature seems to lack broad and fair comparisons of the impact on neural
coding on the energy consumption of hardware neural networks. Therefore, the first contribution
of this work is a fair and extensive comparison between spiking and formal coding domains on
FPGA. The study also takes low-level implementation into account, like parallelism. It was made
possible by the development of an hardware-footprint estimator for hardware neural networks on
FPGA, which drastically facilitates and accelerates the design and applications space exploration.
Moreover, we propose an high-level metric and energy model to evaluate the energy consumption
of neuromorphic accelerators. This model named Synaptic Activity Ratio (SAR) is based on the
number of spikes per synapse and the ratio of energy consumption between spiking and formal
synaptic operations. The model is indeed able to determine whether a given CNN model (tested
on a given dataset) could bring energy savings on FPGA. This contribution also facilitates the
exploration of applications relatively to neural coding domains. For instance, we also use this
technique to find novel SNN representations which seem better adapted to hardware acceleration
than the basic rate-coded conversion approach. Finally, we use the knowledge acquired from this
study to introduce the novel concept of neural coding hybridization. We propose an hardware
architecture which uses both formal domain for feature-extraction and spiking domain for classi-
fication. The architecture was embedded on-board OPS-SAT satellite (launched Dec. 2019) for
in-flight test, being the world first neuromorphic architecture in space.

*Keywords* - Artificial Neural Networks, Spiking Neural Networks, Digital Electronics, Hardware Architecture, Modeling, Energy Consumption, Embedded Systems, Edge Computing, Neuromorphic Electronics, FPGA
Acknowledgements

I would like to express my deepest gratitude to my supervisors: Benoît Miramond, Sébastien Bilavarn, Hadi Saoud and Philippe Millet for their unwavering help and support during the three years of my thesis. I would like to thank them for believing in me with this project although I had very little experience and knowledge on digital electronics; for teaching me so much professionally, technically and humanely; and for the support they brought me all along those three years.

I would also like to express my gratitude to Simon Thorpe, Gilles Sassatelli, Olivier Bichler and Jean Martinet for accepting to be part of my thesis jury. It is a great honor for me to defend my work in front of such prominent researchers.

I would also like to thank my colleagues of both LEAT and Thales RT for their support and the many friendship we built. In particular, I would like to sincerely thank Nassim Abderrahmane for all the work we have accomplished together, and all the help he offered me in the early stages of my thesis.

In addition, I would like to thank my parents Frédérique and Bruno Lemaire, my sister Félicie Lemaire, my grand-parents, uncles, aunts, cousins and friends for their support, encouragement and prodding. Each one of them had a role to play in the achievement of this work. I would also like to thank Manon Philip for her unwavering support and love, which helped me going forward during hard times.

This work was funded by Thales and ANRT (Agence Nationale de la Recherche Technologique), and hosted at both Thales Research Technology's LCHP (High Performance Computing Laboratory) and CNRS & Université Côte d'Azur LEAT (Electronics, Antennas and Telecommunication Laboratory).
# List of Publications

## Journal Papers

Abderrahmane, N., Lemaire, E., & Miramond, B. (2020). Design space exploration of hardware spiking neurons for embedded artificial intelligence. Neural Networks, 121, 366-386.

Lemaire, E., Miramond, B., Bilavarn, S., Saoud, H. & Abderrahmane, N., (2022). Synaptic Activity and Hardware Footprint of Spiking Neural Networks in Digital Neuromorphic Systems. ACM Transactions on Embedded Computing Systems (Accepted for publication)

## Conference Papers

Lemaire, E., Moretti, M., Daniel, L., Miramond, B., Millet, P., Feresin, F., & Bilavarn, S. (2020, October). An FPGA-based Hybrid Neural Network accelerator for embedded satellite image classification. In 2020 IEEE International Symposium on Circuits and Systems (ISCAS) (pp. 1-5). IEEE.

## Book Chapters

Chapter 12 : Space Use-Case, in the book :
Jahre, M., Göhringer, D., & Millet, P. (Eds.). (2021). Towards Ubiquitous Low-power Image Processing Platforms. Springer International Publishing.

## Workshop Papers

Férésin, F., Kervennic, E., Bobichon, Y., Lemaire, E., Abderrahmane, N., Bahl, G., ... & Ben-guigui, M. (2021). In space image processing using AI embedded on system on module: example of OPS-SAT cloud segmentation. In 2nd European Workshop on On-Board Data Processing.

## Patents

FISO neuron model, submitted in September 2019, in proceeding.
<table>
  <tbody>
    <tr>
      <td>2.19</td>
      <td>Schematic overview of the FINN architecture found in S2N2 [13]. The SWU (Sliding Window Unit) flattens the input data and forwards it to the MVTU (Matrix Vector Threshold Unit). Each Processing Element (PE) inside the MVTU processes one output channel and has a number of SIMD (Single Instruction on Multiple Data) lanes that read from input channels and multiply the input by kernel weights in parallel.</td>
      <td>29</td>
    </tr>
    <tr>
      <td>2.20</td>
      <td>Illustration of the Neural Processing Unit of SPLEAT [14]. This NPU is used to emulate a convolution, pooling or fully-connected spiking layer in order to build a full SNN on FPGA.</td>
      <td>30</td>
    </tr>
    <tr>
      <td>3.1</td>
      <td>High level representation of PADS architecture, for a 2-layers spiking MLP. FCi stands for the i<sup>th</sup> fully-connected layer with Ni neurons.</td>
      <td>36</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Block Diagram of the Spike Generation Cell.</td>
      <td>37</td>
    </tr>
    <tr>
      <td>3.3</td>
      <td>Spike emission process flowchart. Initialization is highlighted in green, emission in red.</td>
      <td>38</td>
    </tr>
    <tr>
      <td>3.4</td>
      <td>Architecture of a generic NPU in PADS, represented with 4 input synapses. Register barriers are represented as rectangles.</td>
      <td>41</td>
    </tr>
    <tr>
      <td>3.5</td>
      <td>Architecture of a PADS Input-NPU, optimized for the input FC layer</td>
      <td>42</td>
    </tr>
    <tr>
      <td>3.6</td>
      <td>Architecture of the PADS' Terminate Delta Module with 4 output neurons.</td>
      <td>43</td>
    </tr>
    <tr>
      <td>3.7</td>
      <td>Workflow for synthesis, simulation and reporting using Vivado Design Suite (2019.1). The source code (VHDL) is depicted on the left, and the resulting report on the right.</td>
      <td>44</td>
    </tr>
    <tr>
      <td>3.8</td>
      <td>Evolution of test accuracy, number of input &amp; output spikes and execution time (in clock cycles) with respect to MaxPeriod. All measurements are averaged on 100 samples. Topology: 784-10 on MNIST.</td>
      <td>48</td>
    </tr>
    <tr>
      <td>3.9</td>
      <td>Evolution of test accuracy, number of input &amp; output spikes and execution time (in clock cycles) with respect to input size. All measurements are averaged on 100 samples. Single FC layer SNN on MNIST dataset with various input sizes.</td>
      <td>49</td>
    </tr>
    <tr>
      <td>4.1</td>
      <td>Illustration of the CIAR cloud segmentation task. On the left, the original image taken by OPS-SAT. On the right, the resulting segmentation map, with cloudy patches in yellow. Source: [15]</td>
      <td>54</td>
    </tr>
    <tr>
      <td>4.2</td>
      <td>a) Sample of MNIST dataset, b) Sample of GTSRB dataset</td>
      <td>55</td>
    </tr>
    <tr>
      <td>4.3</td>
      <td>a) Sample of CIFAR 10 dataset, b) Sonar echoes from Mines VS Rocks dataset [16]</td>
      <td>56</td>
    </tr>
    <tr>
      <td>4.4</td>
      <td>Examples of spectrograms from Google Speech Commands dataset [17]. Spoken Digits dataset use in this work is a subset of Google Speech Commands.</td>
      <td>56</td>
    </tr>
    <tr>
      <td>4.5</td>
      <td>Illustration of the over-the-air RadioML 2018 recording setup, found in [18]. A host computer generates the desired RF signal, which is emitted and received via two universal software radio peripherals (USRP).</td>
      <td>57</td>
    </tr>
    <tr>
      <td>4.6</td>
      <td>Evolution of spike filtering rate on CIFAR-10 task</td>
      <td>61</td>
    </tr>
    <tr>
      <td>4.7</td>
      <td>Spike Count Evolution Δ = 5 and Δ = 20 in Layer #0, Layer #6 and in total, for various Θ<sub>IF</sub> on CIFAR-10 task. The vertical axis is logarithmic</td>
      <td>62</td>
    </tr>
    <tr>
      <td>4.8</td>
      <td>Evolution of λ against number of parallel MACs, for Xilinx Zedboard and ZCU102 targets.</td>
      <td>67</td>
    </tr>
    <tr>
      <td>4.9</td>
      <td>Unified synaptic trace format</td>
      <td>68</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>4.10</td>
      <td>SAR (above) and accuracy (below) on OPSSAT, MNIST, GTSRB and Mines Vs Rocks datasets classification, for Δ = 5, 10 and 20. λ<sub>ZE</sub> and λ<sub>ZC</sub> are also represented. Spiking inference with N2D2.</td>
      <td>70</td>
    </tr>
    <tr>
      <td>4.11</td>
      <td>SAR (above) and accuracy (below) on Spoken Digits, RadioML 2018 and CIFAR-10 datasets classification, for Δ = 5, 10 and 20. λ<sub>ZE</sub> and λ<sub>ZC</sub> are also represented. Spiking inference with N2D2. Additionally, for CIFAR-10 θ<sub>IF</sub> varies between 0.5, 1 and 2.</td>
      <td>71</td>
    </tr>
    <tr>
      <td>4.12</td>
      <td>Activity ratio on MNIST using a larger topology, obtained using N2D2. Spiking Accuracy = 98,73%; Formal Accuracy = 99,19%</td>
      <td>73</td>
    </tr>
    <tr>
      <td>4.13</td>
      <td>Spike train period with respect to element intensity according to the rate-coding policy, with PeriodMax = 1 × 10<sup>11</sup> fs and PeriodMax = 1 × 10<sup>7</sup> fs (default N2D2 parameters used in our experiments)</td>
      <td>74</td>
    </tr>
    <tr>
      <td>4.14</td>
      <td>Distribution of element intensity in our benchmark datasets. Left are image datasets, right are 1D datasets. a) MNIST, b) Mines VS Rocks, c) GTSRB, d) Spoken Digits, e) CIFAR-10, f) RadioML2018</td>
      <td>76</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>Overview of the high-level estimation framework</td>
      <td>80</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>Design space for a) FC layers (30 points), b) Conv &amp; Pool layers (24 points each)</td>
      <td>82</td>
    </tr>
    <tr>
      <td>5.3</td>
      <td>3D interpolations of LUTs (left) and FFs (right) for Fully-Connected layers. First row: SPLEAT (red) vs HLS (blue), second row: PADS (red) vs VGT (blue). Third row: SPLEAT (red) vs PADS (blue).</td>
      <td>84</td>
    </tr>
    <tr>
      <td>5.4</td>
      <td>3D interpolations of active and idle power for Fully-Connected layers. Left: SPLEAT, right: PADS.</td>
      <td>86</td>
    </tr>
    <tr>
      <td>5.5</td>
      <td>3D interpolations of Block RAM (left) and DSPs (right) for Fully-Connected layers. First row: SPLEAT (red) vs HLS (blue), second row: PADS (red) vs VGT (blue). Third row: SPLEAT (red) vs PADS (blue).</td>
      <td>87</td>
    </tr>
    <tr>
      <td>5.6</td>
      <td>3D interpolations of Active Power for Fully-Connected layers. First row: SPLEAT (red) vs HLS (blue), second row: PADS (red) vs VGT (blue). Third row: SPLEAT (red) vs PADS (blue).</td>
      <td>88</td>
    </tr>
    <tr>
      <td>5.7</td>
      <td>a) Simple spike trace example with 4 input synapses and 4 time increments, b) Illustration of PADS parallel and pipelined process.</td>
      <td>90</td>
    </tr>
    <tr>
      <td>5.8</td>
      <td>Illustration of SPLEAT sequential process.</td>
      <td>91</td>
    </tr>
    <tr>
      <td>5.9</td>
      <td>a) FPGA power consumption model, b) Dynamic and Static power VS LUT occupation (%) on Zedboard</td>
      <td>92</td>
    </tr>
    <tr>
      <td>5.10</td>
      <td>Layer-wise LUT estimation in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</td>
      <td>93</td>
    </tr>
    <tr>
      <td>5.11</td>
      <td>Layer-wise RAM and DSP estimation in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</td>
      <td>94</td>
    </tr>
    <tr>
      <td>5.12</td>
      <td>Comparisons of resource usage in VGT, HLS, PADS and SPLEAT. Left: OPSSAT, Right: Spoken Digits. Top: Full network, Bottom: Classification Stage</td>
      <td>95</td>
    </tr>
    <tr>
      <td>5.13</td>
      <td>Layer-wise and total SAR for a) OPS-SAT (Δ = 4) and b) Spoken Digits (Δ = 20).</td>
      <td>97</td>
    </tr>
    <tr>
      <td>5.14</td>
      <td>Layer-wise execution time (per image) estimation in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</td>
      <td>98</td>
    </tr>
    <tr>
      <td>5.15</td>
      <td>Layer-wise Power estimation in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</td>
      <td>99</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>5.16</td>
      <td>Comparisons of inference time (per image) and power usage in VGT, HLS, PADS and SPLEAT. Left: full network, Right: classification stage. Top: time, Bottom: power</td>
      <td>100</td>
    </tr>
    <tr>
      <td>5.17</td>
      <td>Layer-wise energy estimation (per image) in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</td>
      <td>102</td>
    </tr>
    <tr>
      <td>5.18</td>
      <td>Comparisons of energy consumption per image in VGT, HLS, PADS and SPLEAT. Left: full network, Right: classification stage. Top: power, Bottom: timing</td>
      <td>103</td>
    </tr>
    <tr>
      <td>5.19</td>
      <td>Confrontation of SAR model and energy estimations for 7 datasets of the benchmark: MNIST, OPSSAT, GTSRB, CIFAR-10, Mines VS Rocks, Spoken Digits and RadioML 2018. Validation is made separately for parallel and sequential architectures. Each subfigure is divided between SAR &amp; &lambda; values (top) and energy estimations (bottom).</td>
      <td>105</td>
    </tr>
    <tr>
      <td>5.20</td>
      <td>SAR and energy estimations without static power consumption, on MNIST (&Delta; = 20) and OPSSAT(&Delta; = 4)</td>
      <td>107</td>
    </tr>
    <tr>
      <td>5.21</td>
      <td>SAR and energy estimations for MNIST with &Delta; = 5 (left) and MNIST with &Delta; = 20 after applying the DSP saturation correction (right)</td>
      <td>108</td>
    </tr>
    <tr>
      <td>6.1</td>
      <td>Activity distribution in CNN layers for OPSSAT (left) and Spoken Digits (right)</td>
      <td>114</td>
    </tr>
    <tr>
      <td>6.2</td>
      <td>Spike train period with respect to element intensity according to the rate-coding policy, with PeriodMax = 1timestep and PeriodMax = 100timesteps</td>
      <td>115</td>
    </tr>
    <tr>
      <td>6.3</td>
      <td>Layer-wise and total SAR for a) OPS-SAT (&Delta; = 4) and b) Spoken Digits (&Delta; = 20).</td>
      <td>118</td>
    </tr>
    <tr>
      <td>6.4</td>
      <td>Hardware footprint estimation of VGT, SPLEAT, Hybrid VGT-SPLEAT and Hybrid VGT-PADS on OPS-SAT RGB dataset with &Delta; = 4, MinPeriod = 1 and MaxPeriod = 100. a) LUT, b) Registers, c) Block RAM, d) DSP, e) Power, f) Inference time and g) Energy.</td>
      <td>119</td>
    </tr>
    <tr>
      <td>6.5</td>
      <td>Hardware footprint estimation of VGT, SPLEAT, Hybrid VGT-SPLEAT and Hybrid VGT-PADS on Spoken Digits RGB dataset with &Delta; = 4, MinPeriod = 1 and MaxPeriod = 100. a) LUT, b) Registers, c) Block RAM, d) DSP, e) Power, f) Inference time and g) Energy.</td>
      <td>121</td>
    </tr>
    <tr>
      <td>6.6</td>
      <td>a: OPS-SAT CubeSat being tested before launch. b: OPS-SAT FlatSat platform. Photo credits: TU Graz [19].</td>
      <td>123</td>
    </tr>
    <tr>
      <td>6.7</td>
      <td>Impression of OPS-SAT in low earth orbit [20]</td>
      <td>124</td>
    </tr>
    <tr>
      <td>6.8</td>
      <td>Schematic of OPS-SAT architecture. The yellow part is for the "technical" bus, and the blue part is for the payload. Source: [19]</td>
      <td>125</td>
    </tr>
    <tr>
      <td>6.9</td>
      <td>Illustration of the CIAR Hybrid Neural Network Architecture.</td>
      <td>126</td>
    </tr>
    <tr>
      <td>7.1</td>
      <td>Illustration of the Send-on-Delta spike encoding process used in [21], for 2 IF neurons encoding a 1D temporal signal. Left: network setup. Right: input and output data.</td>
      <td>131</td>
    </tr>
    <tr>
      <td>7.2</td>
      <td>Illustration of the Leaky Integrate &amp; Fire neuron process. The membrane potential is shown in blue, input spikes in green and output spikes in red.</td>
      <td>132</td>
    </tr>
    <tr>
      <td>7.3</td>
      <td>SAR and Accuracy measurements on the benchmark of datasets with S2NET framework. Samples are presented for 5, 10 and 20 time-steps. The &lambda; value is depicted in orange for the Zedboard and red for the ZCU102 (see Section 4.3.2). All measurements are averaged on 10 runs.</td>
      <td>136</td>
    </tr>
    <tr>
      <td>7.4</td>
      <td>Hardware architecture of: a) FISO neuron with 4 input pixels, b) LIF neuron with 4 input synapses. Register barriers are shown in gray or colored rectangles. Data stored in memory is depicted in light blue squares.</td>
      <td>140</td>
    </tr>
  </tbody>
</table>
# List of Tables

<table><tr><td>2.1</td><td>Logic resources, execution time, power and energy of FPGA SNN accelerators found in the literature. Topology nomenclature: KcWsX = Convolution layer with K filters of size W<sup>2</sup> and a stride of X, KpWsX = Pooling layer with K filters of size W<sup>2</sup> and a stride of X. Fully-connected layers are referred to by their number of output neurons.</td><td>22</td></tr><tr><td>3.1</td><td>TensorFlow Keras parameters used for training in all our experiments.</td><td>44</td></tr><tr><td>3.2</td><td>Hardware resources measures for PADS and VGT MLPs on OPS-SAT Grayscale Dataset: 28 × 28 - 100 - 2</td><td>45</td></tr><tr><td>4.1</td><td>Large CNN topology for Spoken Digits dataset. This CNN achieves 97.34% accuracy.</td><td>58</td></tr><tr><td>4.2</td><td>List of CNN topologies for the classification benchmark</td><td>59</td></tr><tr><td>4.3</td><td>Accuracy and synaptic activity measurements with N2D2 on all datasets of the benchmark.</td><td>60</td></tr><tr><td>4.4</td><td>Logic resources, power and energy comparison for 16-bit ACC and MAC operations on Xilinx xc7z020 FPGA. The MAC operation has been implemented with and without DSPs.</td><td>66</td></tr><tr><td>4.6</td><td>Large CNN topology for MNIST dataset</td><td>69</td></tr><tr><td>4.7</td><td>Network-wise Synaptic Activity Ratio for the dataset benchmark, with varying Δ and Θ<sub>IF</sub> values. The cells are green when SAR is above λ<sub>ZE</sub> and λ<sub>ZC</sub>, yellow when it is between the two and red when it is below.</td><td>72</td></tr><tr><td>5.1</td><td>Dynamic power consumption in PADS with and without power gating, on the benchmark of datasets. VGT dynamic power usage is shown for comparison. Results are shown for the classification stage (FC layers) only. The results have been obtained through the estimation framework, for Δ = 20 (best accuracy).</td><td>109</td></tr><tr><td>6.1</td><td>CNN topologies for a) OPS-SAT and b) Spoken Digits datasets.</td><td>115</td></tr><tr><td>6.2</td><td>Input sizes and spike generation min and max period values for full CNN and Hybrid classification stages on OPS-SAT and Spoken Digits dataset.</td><td>117</td></tr><tr><td>6.3</td><td>Measurements and estimations for VGT, SPLEAT and Hybrid VGT-PADS on OPS-SAT Dataset. The difference is expressed in % of the measurement.</td><td>126</td></tr><tr><td>7.1</td><td>Parameters used for training with S2NET framework</td><td>134</td></tr><tr><td>7.2</td><td>Comparison of accuracy and network-wise SAR with Conversion with Rate-coding and SGL with SoD. Bold letters show the best-case accuracy for reach method and task.</td><td>135</td></tr><tr><td>7.3</td><td>Inference time of PADS-V1 and PADS-V2 on MNIST and OPS-SAT hybrid.</td><td>143</td></tr><tr><td>7.4</td><td>Width of the spike encoding window in PADS-V1 and PADS-V2 on the benchmark of datasets, for best-case accuracy. The width is expressed in both timesteps and clock-cycles.</td><td>143</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>1</td>
      <td>LUT, FF, Block RAM and DSP occupation measures for Convolution layers at design-space measurement points. Measures obtained after hardware synthesis on Xilinx Vivado Design Suite, targeting Xilinx Zedboard</td>
      <td>166</td>
    </tr>
    <tr>
      <td>2</td>
      <td>LUT, FF, Block RAM and DSP occupation measures for Pooling layers at design-space measurement points. Measures obtained after hardware synthesis on Xilinx Vivado Design Suite, targeting Xilinx Zedboard</td>
      <td>166</td>
    </tr>
    <tr>
      <td>3</td>
      <td>LUT, FF, Block RAM and DSP occupation results for Fully-Connected layers at design-space measurement points. Measures obtained after hardware synthesis on Xilinx Vivado Design Suite, targeting Xilinx Zedboard</td>
      <td>167</td>
    </tr>
    <tr>
      <td>4</td>
      <td>Power measures for Fully-Connected layers at design-space measurement points. Measures obtained after hardware synthesis and simulation on Xilinx Vivado Design Suite targeting Xilinx Zedboard</td>
      <td>167</td>
    </tr>
    <tr>
      <td>5</td>
      <td>Power measures for Convolution layers at design-space measurement points. Measures obtained after hardware synthesis and simulation on Xilinx Vivado Design Suite targeting Xilinx Zedboard</td>
      <td>168</td>
    </tr>
    <tr>
      <td>6</td>
      <td>Power measures for Pooling layers at design-space measurement points. Measures obtained after hardware synthesis and simulation on Xilinx Vivado Design Suite targeting Xilinx Zedboard</td>
      <td>168</td>
    </tr>
    <tr>
      <td>7</td>
      <td>Duration results for Fully-Connected layers at design-space measurement points. Measures obtained by calculation and validated using post-synthesis simulation.</td>
      <td>169</td>
    </tr>
    <tr>
      <td>8</td>
      <td>Duration results for Convolution layers at design-space measurement points. Measures obtained by calculation and validated using post-synthesis simulation.</td>
      <td>169</td>
    </tr>
    <tr>
      <td>9</td>
      <td>Duration results for Pooling layers at design-space measurement points. Measures obtained by calculation and validated using post-synthesis simulation.</td>
      <td>170</td>
    </tr>
  </tbody>
</table>
# Contents

<table><tr><td>1</td><td>Introduction</td><td>1</td></tr><tr><td>1.1</td><td>History of Deep Learning</td><td>1</td></tr><tr><td>1.2</td><td>Energy Consumption and Embedded Systems</td><td>2</td></tr><tr><td>1.3</td><td>Bio-inspired neurons for Machine Learning</td><td>2</td></tr><tr><td>1.4</td><td>Problem statement and outline</td><td>3</td></tr><tr><td>2</td><td>State of the Art &amp; Contributions</td><td>5</td></tr><tr><td>2.1</td><td>Formal neurons</td><td>6</td></tr><tr><td>2.1.1</td><td>Feed-forward NN and CNNs</td><td>7</td></tr><tr><td>2.1.2</td><td>Error gradient backpropagation algorithm</td><td>9</td></tr><tr><td>2.2</td><td>FNN in hardware</td><td>9</td></tr><tr><td>2.2.1</td><td>Hardware neural networks on digital hardware</td><td>10</td></tr><tr><td>2.2.2</td><td>Example of FNN accelerators for FPGA</td><td>12</td></tr><tr><td>2.3</td><td>Spiking Neural Networks</td><td>14</td></tr><tr><td>2.3.1</td><td>Principle of Spiking Neural Networks</td><td>15</td></tr><tr><td>2.3.2</td><td>Spike encoding</td><td>18</td></tr><tr><td>2.3.3</td><td>Training SNNs</td><td>19</td></tr><tr><td>2.3.4</td><td>Terminate Delta</td><td>20</td></tr><tr><td>2.4</td><td>SNNs in hardware</td><td>21</td></tr><tr><td>2.4.1</td><td>Advantages of SNNs in Hardware</td><td>21</td></tr><tr><td>2.4.2</td><td>Literature review</td><td>21</td></tr><tr><td>2.4.3</td><td>Confronting Spiking and Formal Neural Networks</td><td>28</td></tr><tr><td>2.5</td><td>Conclusion</td><td>31</td></tr><tr><td>2.6</td><td>Contributions</td><td>32</td></tr><tr><td>2.6.1</td><td>Synaptic Activity</td><td>32</td></tr><tr><td>2.6.2</td><td>Quantitative comparison of formal and spiking domains</td><td>32</td></tr><tr><td>2.6.3</td><td>Cartography of applications and neural coding domains</td><td>33</td></tr><tr><td>2.6.4</td><td>How to benefit from spiking domain?</td><td>33</td></tr><tr><td>3</td><td>Spiking Neural Networks parallel implementation: PADS</td><td>34</td></tr><tr><td>3.1</td><td>Hardware Architecture</td><td>35</td></tr><tr><td>3.1.1</td><td>Spike Generation Cell</td><td>35</td></tr><tr><td>3.1.2</td><td>Neural Processing Unit</td><td>39</td></tr><tr><td>3.1.3</td><td>Terminate Delta Module</td><td>41</td></tr><tr><td>3.2</td><td>Hardware Synthesis Results</td><td>42</td></tr><tr><td>3.2.1</td><td>Methodology</td><td>43</td></tr><tr><td>3.2.2</td><td>Comparison with VGT</td><td>45</td></tr><tr><td>3.2.3</td><td>Spike Generation Overhead</td><td>46</td></tr><tr><td>3.2.4</td><td>Conclusions on PADS hardware implementation</td><td>49</td></tr><tr><td>3.3</td><td>Conclusion</td><td>50</td></tr></table>
4 Synaptic Activity Ratio & Energy Modeling 52

<table><tr><td>4.1 Representative Datasets</td><td>53</td></tr><tr><td>4.1.1 Used topologies</td><td>54</td></tr><tr><td>4.2 Accuracy and synaptic activity measurements</td><td>55</td></tr><tr><td>4.2.1 Methods</td><td>56</td></tr><tr><td>4.2.2 Synaptic activity results</td><td>58</td></tr><tr><td>4.2.3 Discussions on Synaptic Activity results</td><td>62</td></tr><tr><td>4.3 Synaptic Activity ratio</td><td>63</td></tr><tr><td>4.3.1 Energy Consumption Model</td><td>63</td></tr><tr><td>4.3.2 The value of λ</td><td>65</td></tr><tr><td>4.4 Synaptic Activity Ratio measurements</td><td>67</td></tr><tr><td>4.4.1 Synaptic Activity Ratio evaluation software</td><td>67</td></tr><tr><td>4.4.2 Network-wise SAR & theoretical cartography</td><td>69</td></tr><tr><td>4.4.3 Data type and rate-coding</td><td>73</td></tr><tr><td>4.4.4 Layer-wise SAR & hybridization</td><td>73</td></tr><tr><td>4.5 Conclusion</td><td>74</td></tr></table>

5 Hardware Footprint and High-Level Estimations 77

<table><tr><td>5.1 Motivations</td><td>79</td></tr><tr><td>5.1.1 Speed-up cartography and exploration</td><td>79</td></tr><tr><td>5.1.2 Layer-wise approach</td><td>79</td></tr><tr><td>5.1.3 Level of parallelism</td><td>79</td></tr><tr><td>5.2 Framework</td><td>80</td></tr><tr><td>5.2.1 Hardware measurements database</td><td>80</td></tr><tr><td>5.2.2 Execution time</td><td>85</td></tr><tr><td>5.2.3 Spiking hardware inference simulator</td><td>90</td></tr><tr><td>5.3 Ressource estimations</td><td>92</td></tr><tr><td>5.3.1 Layer-wise estimation</td><td>93</td></tr><tr><td>5.3.2 Network-wise estimation</td><td>95</td></tr><tr><td>5.3.3 Conclusions</td><td>96</td></tr><tr><td>5.4 Inference time and power estimations</td><td>96</td></tr><tr><td>5.4.1 Layer-wise estimation</td><td>96</td></tr><tr><td>5.4.2 Network-wise estimation</td><td>100</td></tr><tr><td>5.4.3 Conclusions</td><td>101</td></tr><tr><td>5.5 Energy estimations</td><td>101</td></tr><tr><td>5.5.1 Layer-wise estimation</td><td>102</td></tr><tr><td>5.5.2 Network-wise estimation</td><td>103</td></tr><tr><td>5.5.3 Conclusions on energy estimations</td><td>104</td></tr><tr><td>5.6 Validation of the SAR model</td><td>106</td></tr><tr><td>5.7 Conclusion</td><td>107</td></tr><tr><td>5.8 Outlooks</td><td>110</td></tr><tr><td>5.8.1 Improvement of the SAR metric</td><td>110</td></tr><tr><td>5.8.2 Improvement of the estimation framework</td><td>110</td></tr><tr><td>5.8.3 Studying the level of parallelism</td><td>111</td></tr><tr><td>5.8.4 Hybridization and other spike encoding methods</td><td>111</td></tr></table>
<table><tbody><tr><td><strong>6 Neural coding domain hybridization</strong></td><td><strong>112</strong></td></tr><tr><td>6.1 Motivations</td><td>113</td></tr><tr><td>6.1.1 SAR and footprint variability</td><td>113</td></tr><tr><td>6.1.2 Distribution of activity</td><td>115</td></tr><tr><td>6.1.3 Formal convolutions and spiking classification</td><td>116</td></tr><tr><td>6.2 Estimations on hybrid architectures</td><td>116</td></tr><tr><td>6.2.1 Methodology</td><td>116</td></tr><tr><td>6.2.2 OPS-SAT</td><td>117</td></tr><tr><td>6.2.3 Spoken Digits</td><td>120</td></tr><tr><td>6.2.4 Discussions on hybrid estimations</td><td>122</td></tr><tr><td>6.3 Hybrid hardware implementation</td><td>123</td></tr><tr><td>6.3.1 Context</td><td>124</td></tr><tr><td>6.3.2 VGT-PADS Hybrid Architecture</td><td>124</td></tr><tr><td>6.4 Conclusion</td><td>127</td></tr><tr><td>6.4.1 Outlook</td><td>128</td></tr><tr><td><strong>7 Enhancing PADS: FISO &amp; LIF as Recurrent neurons</strong></td><td><strong>129</strong></td></tr><tr><td>7.1 Theoretical Background</td><td>130</td></tr><tr><td>7.1.1 Send on Delta spike encoding</td><td>130</td></tr><tr><td>7.1.2 LIF Neuron</td><td>131</td></tr><tr><td>7.1.3 LIF as recurrent neurons</td><td>132</td></tr><tr><td>7.1.4 Surrogate Gradient Learning</td><td>133</td></tr><tr><td>7.1.5 Output decoding: readout layer</td><td>133</td></tr><tr><td>7.1.6 Application in the S2NET framework</td><td>133</td></tr><tr><td>7.1.7 Static input samples</td><td>134</td></tr><tr><td>7.2 Accuracy &amp; SAR results</td><td>134</td></tr><tr><td>7.3 PADS V2</td><td>138</td></tr><tr><td>7.3.1 Architecture</td><td>138</td></tr><tr><td>7.3.2 Inference Time Results</td><td>141</td></tr><tr><td>7.4 Conclusions &amp; Outlooks</td><td>144</td></tr><tr><td><strong>8 Conclusions and outlooks</strong></td><td><strong>145</strong></td></tr><tr><td>8.1 Conclusion</td><td>145</td></tr><tr><td>8.2 Outlooks</td><td>147</td></tr><tr><td>8.2.1 Short term and work-specific perspectives</td><td>147</td></tr><tr><td>8.2.2 Middle term outlooks and insights</td><td>148</td></tr><tr><td><strong>APPENDIX 1: CIAR project</strong></td><td><strong>159</strong></td></tr><tr><td><strong>APPENDIX 2: Custom SoC platform</strong></td><td><strong>161</strong></td></tr><tr><td>.1 Programmable Logic modules</td><td>161</td></tr><tr><td>.2 CPU &amp; Embedded Linux</td><td>162</td></tr><tr><td><strong>APPENDIX 3: Additional figures</strong></td><td><strong>164</strong></td></tr><tr><td><strong>APPENDIX 4: Measurements</strong></td><td><strong>166</strong></td></tr><tr><td><strong>APPENDIX 5: Raw Estimations</strong></td><td><strong>171</strong></td></tr></tbody></table>
# Chapter 1

## Introduction

In recent years, the field of artificial intelligence has been widely dominated by Deep Learning. This field of Machine Learning uses Artificial Neural Networks which model the biological neural networks in mathematical equations. This approach enables to emulate more and more complex cognitive functions in automated algorithms, such as image classification, object detection, natural language processing or decision making. Since it first appeared in the foreground of research in the 90’s, all those domains have been revolutionized by Deep Learning. The complexity of the models growing further and further, the limits of Machine Learning are constantly pushed back, involving more and more application domains and industrial actors.

### 1.1 History of Deep Learning

Long before its modern supremacy, the premises of Deep Learning dates back to the very beginning of computer science. In the same school of thought of Alan Turing, researchers like McCulloch and Pitts worked to demonstrate that cognition could be modeled through mathematical functions. In 1943, they proposed the first artificial neuron model [22]. In this breakthrough publication, the authors demonstrated that a network of such formal neurons was able to emulate propositional logic functions. Later in 1957, Rosenblatt proposed the first application of formal neurons to automated classification: the perceptron [23]. The perceptron is made of a single layer of formal neurons. The major contribution of this pioneering publication is the learning algorithm. By iterating over a set of labeled training samples, the algorithm automatically adjusts the synaptic weights to minimize the output error. In doing so, the authors proposed the first supervised learning algorithm. However, the technique was limited to single layers of neurons, which were only able to solve binary classification problems. This limitation was leveraged in another major breakthrough when Werbos proposed the Multi-Layer Perceptron (MLP) and gradient Backpropagation (BP) algorithm in 1974 [24]. The MLP is considered by many as the first modern neural network model, and the BP algorithm is still a fundamental element of modern Machine Learning. The BP algorithm allows to train several successive perceptron layers at once, hence the name Multi-Layer Perceptron. The deeper meaning of this work is even more revolutionary: an MLP of arbitrary size and depth is theoretically able to solve any non-linear problem. However, the feasibility of this algorithm was limited by the available computing power in the early 70’s.

Following Moore’s law, the exponential growth of computing power made it possible in 1986, when Rumelhart proposed the first implementation of the BP algorithm in a feed-forward MLP [25]. This work opened the gates to what has since become modern Deep Learning by using deeper and deeper networks, the models were able to solve more and more complex tasks in various application fields. The most prominent of which was computer vision: extracting information from images for classification, segmentation or detection. In 1989, LeCun proposed the first Convolution Neural Network (CNN) [26] trained with the BP algorithm. The innovation of this model is to give a structure to the layers of neurons according to specificities of the application. For computer
vision, the convolution layers are designed to extract spatial patterns. LeCun later generalized the CNN model to various tasks in the fields of image classification, speech recognition and time-series prediction [27]. The convolution layers are still widely used in Deep Learning to this date, and are one of the key components of modern neural network architectures like VGG-16 [28]. In recent years, other Formal Neural Networks (FNN) models have emerged for various domains of application, like Transformer neural networks for Natural Language Processing [29]. In doing so, the models are getting deeper and deeper and more and more complex. For example, the recent GPT-3 transformer model proposed by the OpenAI team in 2020 [30] uses 175 billions of parameters, i.e. 175 billions of explicit synaptic connections.

## 1.2 Energy Consumption and Embedded Systems

Neural network models becoming more and more complex, they have higher and higher computational requirements for training and inference. For example, the GPT-3 transformer model with 175 billion parameters requires a bare minimum of 350GB of GPU memory for inference. That is equivalent to 8 Nvidia A6000 GPUs; the latest and most powerful deep-learning oriented High-Performance Computing (HPC) device. This represents a power consumption of 2400W. Therefore, the energetic scalability of Machine Learning starts to raise questions among the scientific and public communities. But this energetic issue is even more problematic in the field of embedded systems. Indeed, the recent emergence of the Internet of Things (IoT) opened a brand new field of applications for neural networks. From autonomous navigation in drones or self-driving cars, to advanced robotics and augmented reality, the need for such embedded artificial intelligence is just starting to rise. By definition, those systems are isolated from the power grid, and must run on battery or energy harvested from the environment (mostly solar panels). Quite straightforwardly, the energy budget of Deep Learning is not compliant with such constrained systems.

An approach chosen by many researchers in the field of neural network acceleration is to take inspiration from the most advanced and efficient neural processor anyone has ever heard of: the biological brain. The human brain is said to feature more than 10 000 billions synaptic connections per cm³. On the other hand, it consumes no more than 20W in average. Since the volume of human brain is roughly 1450cm³, that is an average of $10^{-17}W$ per synapse. For comparison, the most advanced neural network model (GPT-3) running on the most advanced hardware available (Nvidia A6000) uses approximately $10^{-8}W$ per synapse (175 billions synapses and 2400W for 8 A6000 GPUs). That is, if we omit the nature of the network itself (its topology and actual neuron dynamics) and if we consider that the number of synapses is a reliable metric to compare the brain and a deep learning model, the first is approximately one billion times more energy-efficient than the latter. It seems that we indeed have a lot to learn from biology regarding low-power neural processing.

## 1.3 Bio-inspired neurons for Machine Learning

In the field of Machine Learning, the main interest is to develop powerful models applied to solving real-world problems, like image segmentation for self-driving cars or natural language processing for virtual assistants. However, another field of research has long paid interest to artificial neural networks for other motivations. Neurosciences and Cognitive Sciences use them as a tool to
simulate cognitive functions, providing a better understanding of the underlying mechanisms of the human mind. For this purpose, computational Neurosciences are interested in the biological plausibility of the neuron models. In doing so, many biological neuron models have been proposed, mimicking the event-based nature of the brain. In contrast with Machine Learning which uses static real-valued information encoding, the biological brain uses action potentials: brief electrical impulses better known by the name of spikes. In Spiking Neural Networks (SNNs), information is encoded temporally with spikes. The strong hypothesis at the basis of our approach is the following: spike encoding is one of the major sources of power efficiency in the biological brain, compared to formal Machine Learning models. This hypothesis is backed by several factual assessments. In the widely used Integrate & Fire spiking neuron model, multiplication-accumulation operations are replaced with accumulation. The latter uses fewer logic and power than the first [31], hence the expectations of a better hardware-efficiency. Moreover, the sparse encoding of information in SNNs enables a sparse computation. In other words, the design is only active upon processing input spikes and remains idle otherwise. This feature is known as "frame-free", in contrast with the frame-constrained nature of the formal approach. Additionally, spikes enable light-weight communication between neurons, further reducing the hardware footprint.

In the light of these assessments, a recent trend in the field of neural network acceleration is to use bio-inspired neurons in deep-learning network models. It consists in developing application specific hardware architectures to support the inference of deep SNN models. By using bio-inspired neuron and event-based computation, expectations are a major reduction of energy consumption. This is a particularly appealing promise for embedded systems. In all, SNNs are expected to leverage the energy consumption limitations of embedded artificial intelligence. The thesis lies precisely in this field of the research: Neuromorphic Computing. More precisely, this work deals with the hardware acceleration of Spiking Neural Networks on FPGA, and the associated energy savings.

## 1.4 Problem statement and outline

However, there is still a lack of clear, fair and extensive comparison between formal and spiking hardware neural networks in the literature. The hardware-efficiency promises remain unproven outside of very specific cases, and most importantly: to be quantified. In this thesis, we address the most widely used type of SNNs to our knowledge: rate-coded SNNs converted from formal models. The goal of this thesis is to evaluate the potential energy efficiency gains of this model over conventional Formal Neural Networks in the context of FPGA accelerators. More precisely, this thesis aims in providing a cartography of neural coding domains and applications. In other words, the goal is to determine which type of application would be better suited to spiking domain acceleration in terms of energy gains. Moreover, this study follows a hardware-software co-design approach: the model and the underlying hardware has to be designed according to one another. In doing so, our goal is also to find innovative ways to benefit from spiking neurons in hardware and vice versa. The contributions of this work relatively to the literature will be described further at the end of the State-of-the-Art section (Chapter 2).

This document begins with a detailed state-of-the-art (Chapter 2), including a theoretical background of formal and spiking neural network models and hardware acceleration. We also provide an overview of the existing architectures and few publications addressing the comparison of the two neural coding domains. In Chapter 3, we describe our prototype of hardware accelerators for fully-connected spiking layers: PADS. It is based on rate-coding, and is used as an experimental
testbench to study the energy consumption of neuromorphic systems. In Chapter 4, we propose the Synaptic Activity Ratio (SAR) metric and the associated high-level energy estimation model. The model is used to rapidly determine whether a model or application is suitable for spiking acceleration, in terms of energy savings over formal acceleration. In Chapter 5, we propose an estimation framework based on low-level measurements and FPGA modeling. The framework enables rapid estimation of resources, power usage, inference time and energy consumption of formal and spiking accelerator under high and low level of parallelism. The framework enables a fast and reliable cartography of application, neural coding domain and implementation choices. It is applied to a benchmark of datasets, and the energy estimations are confronted with the SAR-based model. In Chapters 6 and 7, we propose two hardware implementations based on PADS architecture. In the light of the previous studies, those architectures intends to leverage the potential energy savings expected from neuromorphic acceleration. In Chapter 6, we propose neural network hybridization in order to tailor the neural coding domain according to synaptic activity at layer level. In Chapter 7, we use our acquired knowledge and models to find a more suitable SNN representation for neuromorphic acceleration. That is, a novel representation in which SNNs are viewed as Recurrent Neural Networks on a finite (and defined) number of timestep [32]. Using SoD spike encoding [33] and Surrogate Gradient Learning, this new model of SNNs achieve lower SAR than rate-based SNNs for even accuracy. We also propose an adaptation of the PADS architecture to this new model.

<figure><img src="image_3.png" /><figcaption>Figure 1.1: Neuromorphic Computing is a trans-disciplinary approach between Deep Learning, Digital Electronics and Computational Neurosciences.</figcaption></figure>
# Chapter 2

# State of the Art & Contributions

## Chapter contents

<table><tr><td>2.1 Formal neurons</td><td>6</td></tr><tr><td>2.1.1 Feed-forward NN and CNNs</td><td>7</td></tr><tr><td>2.1.2 Error gradient backpropagation algorithm</td><td>9</td></tr><tr><td>2.2 FNN in hardware</td><td>9</td></tr><tr><td>2.2.1 Hardware neural networks on digital hardware</td><td>10</td></tr><tr><td>2.2.2 Example of FNN accelerators for FPGA</td><td>12</td></tr><tr><td>2.3 Spiking Neural Networks</td><td>14</td></tr><tr><td>2.3.1 Principle of Spiking Neural Networks</td><td>15</td></tr><tr><td>2.3.2 Spike encoding</td><td>18</td></tr><tr><td>2.3.3 Training SNNs</td><td>19</td></tr><tr><td>2.3.4 Terminate Delta</td><td>20</td></tr><tr><td>2.4 SNNs in hardware</td><td>21</td></tr><tr><td>2.4.1 Advantages of SNNs in Hardware</td><td>21</td></tr><tr><td>2.4.2 Literature review</td><td>21</td></tr><tr><td>2.4.3 Confronting Spiking and Formal Neural Networks</td><td>28</td></tr><tr><td>2.5 Conclusion</td><td>31</td></tr><tr><td>2.6 Contributions</td><td>32</td></tr><tr><td>2.6.1 Synaptic Activity</td><td>32</td></tr><tr><td>2.6.2 Quantitative comparison of formal and spiking domains</td><td>32</td></tr><tr><td>2.6.3 Cartography of applications and neural coding domains</td><td>33</td></tr><tr><td>2.6.4 How to benefit from spiking domain?</td><td>33</td></tr></table>
<figure><img src="image_4.png" /><figcaption>Figure 2.1: Left: commonly used activation functions in formal neurons: hyperbolic tangent (Tanh), sigmoid and Rectified Linear Unit (ReLU). Right: Schema of FC layer, with 5 input and 2 output neurons.</figcaption></figure>

Before going into more details on the works and results obtain to achieve the thesis goals, we briefly introduce the State of the Art of neuromorphic accelerators. First, we provide information on conventional Formal Neural Networks (FNN) and their hardware implementations. Then, we introduce Spiking Neural Networks (SNN) and how they differ from classical FNNs. More precisely, we are going to explain how those differences are expected to bring resource, power and energy savings to neuromorphic computing. In doing so, we describe some existing SNN accelerators and techniques aiming to benefit from the spiking domain promises.

## 2.1 Formal neurons

Formal Neurons are very roughly inspired from biological models in their behaviour. They are designed to integrate incoming weighted activations from several uphill neurons, just like the biological neuron does with incoming action potentials in its soma. Each of those activations are weighted by a synaptic-weight. From this "soma activity", which is also referred to as "membrane potential", an output activation is computed thanks to an activation function. This output activation is then propagated to downhill neurons. This behavior is summarized in Equation 2.1.

$$y_j^l = f(s_j^l), \quad s_j^l = \sum_{i=0}^{N_{l-1}-1} w_{ij} * y_i^{l-1} \qquad (2.1)$$

With $y_j^l$ being the output of the $j^{th}$ neuron of layer $l$, $s_j^l$ the membrane potential of the $j^{th}$ neuron of layer $l$ and $w_{ij}$ the synaptic weight between the $i^{th}$ neuron of layer $l-1$ and the $j^{th}$ neuron of layer $l$, and $f()$ the non-linear activation function. The activation function is usually a non-linear function such as hyperbolic tangent (Tanh), sigmoid or Rectified Linear Unit (ReLU). Those activations functions are shown in Figure 2.1a.

However, this single artificial neuron is not useful in itself. It starts to gain interest when using several interconnected neurons, through which information flows successively. This set of interconnected neurons is what is called an Artificial Neural Network.
2.1.1 Feed-forward NN and CNNs

Artificial Neural Networks come in various types and families. Those types are called topologies, which characterize the organization of neurons in the network. In the present work, we deal with feed-forward, layer-based neural networks. If some types of networks are unstructured, such as reservoir neural networks [34] or liquid state machines ( [35]), feed-forward neural networks are formed of successive layers, through which information flows from the input to the output, always in the same direction. Therefore, we will not cover topologies such as Recurrent Neural Networks, in which activation does not only flows downstream, but can be branched to itself or upstream layers. A layer is a set of neurons, which are connected to neurons of the previous and following layers. In the Machine Learning community, the layer corresponds more to the connection scheme between two set of neurons, than the set of neuron in itself. This connection policy will indeed serve to emulate a transformation of information between two set of neurons. The most basic type of layer is the Fully-Connected (FC) layer. An illustration of FC layers is shown in Figure 2.1b.

It implies an all-to-all connection between upstream and downstream neurons of a layer. This function is used to classify data, and can be used on its own to build a Multi-Layer Perceptron (MLP), which is composed of several successive FC layers. It has been stated by Hornik et. al. [36] that an MLP composed of two fully-connected layers was sufficient to approximate any non-linear separable function with an arbitrary precision, i.e. it is sufficient to solve any classification task. However, when using only FC layers, complex classification tasks (i.e., with a lot of classes, large images, complex objects, rotations...) requires an incredible amount of neurons in those layers. When adding layers, the problem might be solved with fewer neurons, but this still requires a lot of neurons and connections, which implies a huge computation and memory requirements. In order to solve this problem, more specific layers have been proposed by LeCun et. al. [37] in 1980: Convolutional and Pooling Layers. Convolutional layers emulate a convolution operation using one or several filters. This type of layer is well suited for the extraction of spatial or temporal patterns. Moreover, this layer implies a lower number of synaptic weights, as they are shared among filters. A convolution layer with 5 filters of size 3×3 only has 3×3×5 = 45 weights, which helps mitigating memory requirements of Artificial Neural Networks. The Convolutional layer is represented in Figure 2.2. In this figure, two steps of the Convolutional process are represented. The first step is the convolution of the orange patch with the filter, which results in the orange activation in the output feature map. The last step is also shown in blue. All intermediate steps are omitted for clarity. Pooling layers on the other hand emulate a sub-sampling operation. This is used to mitigate the computation and memory requirements of Artificial Neural Networks, as Convolutional layers may result in large output size due to a large number of filters or large input vector size. Similarly to Convolutional layers, Pooling layers associate an output neuron to an input patch. However, the output activation is not obtained by convolving a filter, but rather using Maximum or Average operations: Max-Pooling layers compute the output activation as the maximum value of the associated input patch, whereas Average-Pooling layers compute the output activation as the average of the patch. The combination of Convolution and Pooling layers form a Feature-Extraction (FE), which is useful to pre-process the data before classification through FC layers. Basically, a typical Convolutional Neural Network such as LeNet5 [37] (Figure 2.3) is composed of a FE stage followed by an MLP.
<figure><img src="image_5.png" /><figcaption>Figure 2.2: Schema of a Convolutional layer, with a 5x5x1 input data, a single 3x3 filter and a stride of 1.</figcaption></figure>

<table><caption>Figure 2.3: Summary of LeNet-5 CNN. Conv is for Convolutional Layer, Pool is for Pooling layers and D is for FC layers. The visuals are obtained using N2D2 software [1]</caption><thead><tr><th>Layer</th><th>Input</th><th>Conv 1</th><th>Pool 1</th><th>Conv 2</th><th>Pool 2</th><th>D 1</th><th>D 2</th><th>D 3</th></tr></thead><tbody><tr><th>Hyper Parameters</th><td></td><td>N<sub>filters</sub> = 6<br>W<sub>filters</sub> = 5<br>S = 1</td><td>N<sub>filters</sub> = 6<br>W<sub>filters</sub> = 2<br>S = 2</td><td>N<sub>filters</sub> = 16<br>W<sub>filters</sub> = 5<br>S = 1</td><td>N<sub>filters</sub> = 16<br>W<sub>filters</sub> = 2<br>S = 2</td><td>N<sub>out</sub> = 256</td><td>N<sub>out</sub> = 84</td><td>N<sub>out</sub> = 10</td></tr><tr><th>Out size</th><td>28x28x1</td><td>24x24x6</td><td>12x12x6</td><td>8x8x16</td><td>4x4x16</td><td>256</td><td>84</td><td>10</td></tr><tr><th>Illustration</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>
2.1.1.1 Advanced layers

In recent years, new types of layers emerged in addition to the three aforementioned fundamental layer structures. A short list of the most important advanced layers is given below for information.

* Batch-Normalization layer [38]: normalizes activation to stabilize and speed-up learning

• Recurrent layers [39]: short term memory for temporal signal processing

* Residual layer [40]: recurrent connection across several layers

* Softmax layer [41]: output layer used for categorical classification training

In the present work, only the Softmax layer will be used for training in formal domain.

2.1.2 Error gradient backpropagation algorithm

In both Artificial Neural Networks and their biologic inspiration, the information is not located
inside neurons, but rather inside the connection between neurons: the so called synapses. Indeed,
those connections are endowed with synaptic weights, a coefficient that modulate the sign and
intensity of the information flowing through. The knowledge contained in the network is therefore
represented by the distribution of those weights among synapses. The whole goal of Machine
Learning is to find the weight distribution so that the neural network behavior fits the desired
function. In a small topology such as LeNet5 [37] (see 2.3), there are 369174 synaptic weights,
and this number is exploding when addressing state-of-the-art CNN topologies, such as VGG-
16 [28] which implies 138 millions parameters or even Transformer networks [29] that can reach
billions. Understandably, an automatic learning algorithm is used to tune those weights. The most
common family of training algorithms is based on Error Gradient Backpropagation [42] [26] [43].
The basic idea of this process is to learn statistical biases by iterating on large labelled training
datasets. To do so, the learning algorithm iterates across training samples, starting with a random
weight distribution. Each sample is associated to a label, which represents the target output of the
network for this data. The data is passed at the input of the network, and the output is retrieved:
this is the forward path. An error is computed between the actual output and the expected one
(computed from a label or a specific ground truth value), using various kind of loss functions. At
this stage, one is able to tune the weights of the last layer in order to minimize this error. The
error is then propagated backwards, layer-by-layer, from the output towards the input. At each
layer, the weights are automatically tuned to minimize the layer-wise error. More information on
this process, including the mathematical principles and equations are presented in [42]. However,
as we do not study learning algorithm, such details is out of scope of this work.

2.2 FNN in hardware

Now that the principles of Formal Neural Networks have been explained, this section will focus
on the state-of-the-art regarding the deployment of such models on hardware platforms. It should
be noted that in this whole work, we focus on neural network inference and training is considered
out of scope: it is always performed in software, and the hardware platforms only come into
consideration for the inference issue. At this stage, there are two major possibilities for hardware
artificial neural network implementation: digital or analog systems. Digital implementations have
3 major advantages over analog circuits:
* The development and deployment frameworks are more mature and accessible for digital systems.

* The digital paradigm enable multiplexing, sequencing etc. which brings scalability.

* Analog circuits are more subject to electrical noise and technology variability, even more when scaled-up to larger system sizes.

Moreover, if analog circuits are usually more compact than equivalent digital circuits, this trend reverses when addressing smaller than 22nm CMOS technology [44]. Following Moore's law, the 22nm and smaller CMOS technology is democratizing fast, so digital circuits also take the advantage regarding compactness. For all those reasons, we only address digital implementations in this work. Digital systems come in 4 main families:

* Central Processing Units (CPUs): the most generic and common type of programmable processors, used in every computers, smartphones, and general purpose electronic devices. This is a purely centralized Von Neumann type of architecture [45], were computation is highly sequential in a fast and generic core, which communicates with external memories.

* Graphical Processing Units (GPUs) [46]: a more specific type of accelerator, originally used for graphics rendering in video-games or special effects softwares. GPUs are highly parallel programmable processors, composed of a very large number of small and simple cores, dividing large computations into small portions.

* Application Specific Integrated Circuits (ASICs) [47]: a type of accelerator which is specifically designed for a given task, offering an optimal use of hardware resources (area, power, energy...) on this task. However, this type of accelerator is very expensive to produce, and is only conceivable for large production scales.

* Field-Programmable Gate Arrays (FPGAs) [48]: this type of device is composed of an array of processing elements (Look-Up-Tables, Registers, Multiplexers, Digital Signal Processing units...), which can be configured to build complex circuits using an hardware synthesis software. The architecture is described using a Hardware Description Language [49] (VHDL [50], Verilog [51]...), and automatically mapped to the device. The resulting accelerator can be highly specific and thus makes a near-optimal use of resources for a given task. FPGAs are way cheaper than ASICs to exploit, but make a less optimal use of resources, considering all the logic used to program the device, and the unused elements of the board. Moreover, FPGAs are re-programmable, which makes this technology well suited for early prototyping and research.

In the next section, a literature review will help us compare hardware neural networks on those 4 types of digital hardware targets.

### 2.2.1 Hardware neural networks on digital hardware

In [2], the authors presented the recent advances in the field of Neural Network acceleration regarding compression, algorithm optimization and hardware optimizations. The authors also proposed a separation of hardware acceleration in three levels: structure level, algorithm level and implementation level. The structure level deals with optimizations related to the network topology, such as synapse pruning (removing synapses with insignificant weights), weight redundancy and
<figure><img src="image_6.png" /><figcaption>Figure 2.4: Summary of comparisons between CPU, GPU, ASIC and FPGA for neural network acceleration in terms of power consumption and computing power [2].</figcaption></figure>

layer decomposition. The algorithm level deals with efficient algorithms for training, and are thus considered out of scope. Finally, the implementation level deals with hardware implementation platforms and techniques for CPU, GPU, FPGA, ASIC and other novel technologies such as memristor-based neural networks [52]. Concerning this implementation level, the authors made a comparison of various CNN topologies on various hardware targets, in order to measure and compare the subsequent efficiencies. Their results are summed-up in Figure 2.4, which shows the power consumption and computing power (GOPS: Giga Operation Per Second) of various neural networks on various hardware targets. In this figure, we can see that CPU is the least optimal hardware, offering low computing power for high power consumption. On the other hand GPUs offer the highest possible computing power, but with the highest power consumption. Finally, ASICs and FPGAs seem a good trade-off between computing power and power consumption, as they are specifically designed to optimize both these aspects. ASICs still offer lower power consumption, as it is even more specific than FPGA and does not imply programming logic and unused elements.

In [3], the authors also proposed an in-depth comparison of Neural Network acceleration regarding speed and performance per Watt, using various CNN topologies and hardware targets. The results of their study is given in Figure 2.5. If their study mainly deals with binarized neural networks, they also proposed results for non-binarized models. On those graphics, one might see that, for non-binary models, CPU and GPUs have quite similar performances, with a slight advantage for the CPU in both metrics. On the other hand, FPGA and ASIC architectures always offer better performance and speed-up than CPU and GPUs, in all configurations. This result is an interesting preliminary justification for SNN that belongs to also use binary coding whose activations are scheduled in time.

In [4], Nurvitadhi et al. performed an in depth comparison on Neural Network acceleration targeting FPGA and GPU, regarding performance and performance per Watt. The authors also evaluated the influence of pruning, data compacity and binarization on the performance on both hardware. To do so, the authors compared various Deep Neural Networks (DNN) topologies on Nvidia Titan X, the most powerful GPU on the market, with Xilinx Stratix 10, the latest version of large Xilinx FPGA device. The authors also evaluated Xilinx Arria 10, a smaller recent FPGA device. The authors also developed a FPGA-based DNN accelerator architecture, which they used
<figure><img src="image_7.png" /><figcaption>Figure 2.5: Comparison of CPU, GPU, ASIC and FPGA for binary neural network acceleration in terms of speed and performance per Watt, on various neural network topologies [3].</figcaption></figure>

in their comparison. The results of the comparison are shown in Figure 2.6, for various baseline and
several optimizations: pruning, compact data and binarization. Concerning raw performance, the
only case where Stratix 10 FPGA is below the TITAN X is for baseline DNN. With optimizations,
the FPGA outperforms the GPU in all cases. Concerning performance per Watt (i.e. energy
efficiency), the FPGA performs better than the GPU in every experiment. Finally, the authors
conclude that Neural Network acceleration should focus on FPGAs rather than GPUs, in the light
of their measurements. However, most of these comparisons do not take into account the impact
of compression on network accuracy. This metric has to be considered in addition to the speed
and performance.

## 2.2.2 Example of FNN accelerators for FPGA

In the light of this literature review, it is clear that specific circuits such as FPGAs and ASICs are
the most suited to Neural Network acceleration in the context of energy-and-resource-constrained
embedded systems. Although there is a slight advantage for ASICs in that regard, FPGA's
reconfigurability offers interesting prototyping capabilities, for a much lower cost than ASICs.
Hence, in our work, we focus on FPGA-based neuromorphic accelerators. In this section, we
will specifically describe two FPGA accelerators for Formal Neural Networks: VGT, which is
an architecture for fully parallel FNN acceleration, and C-HLS, which is a fully-sequential FNN
accelerator. Those two accelerators will be used in our application benchmark in Chapter 5. For
information, a wide survey of existing formal accelerators is proposed in [53].

### 2.2.2.1 VGT accelerator

VGT (VHDL Generation Tool) [5] is a framework for Formal Neural Network deployment, which automatically generates VHDL code from an high-level description. In this work, the generated circuits are referred to as VGT architectures. It should be noted that the subsequent VGT accelerators is implemented with a high level of parallelism. Convolution layers operate in a
<figure><img src="image_8.png" /><figcaption>Figure 2.6: Comparison of FPGA (Stratix 10) and GPU (Titan X) for Deep Neural Networks in terms of performance and performance per Watt, for various optimizations: pruning, compact data and binarization. [4].</figcaption></figure>

streaming fashion: the input feature-maps are flattened, and streamed pixel by pixel to the layer, in parallel for each input channels. In the same way, the output feature-maps are streamed in parallel one pixel at a time. The architecture of the convolution itself is made of two stages: a Processing Element for weight-multiplication (i.e. convolution of input channels with filters), and an adder-tree for filter-wise integration (i.e. sum of weighted inputs to compute output feature-maps). The full architecture is available in Figure 2.7. There is a Processing Element for each input channel. Each Processing Element operates weight multiplications in parallel for each convolution filter. Using a sliding window, weight multiplications are performed in a pipelined systolic fashion [10]. An illustration of a Processing Element architecture is available in Figure 2.8. After the systolic arrays, an adder-tree is used to sum the results filter-wise, so that there is one output value per output-filter. As the input feature maps are streamed to the PEs, and thanks to the pipelined architecture, the convolution layer will produce one output value at a time for each filter, in a streaming fashion.

The max-pooling layers of VGT works similarly to the convolution layers, except that there is no weight multiplication, and that the adder-tree is replaced with comparator-tree in order to extract the maximum value of each input patch. An illustration for the max-pooling layer is available in Figure 2.9

Finally, the fully-connected layer of VGT is implemented as a parallel matrix-multiplication architecture. The architecture is similar to that of the convolution layer depicted in 2.7. In contrast with Convolution and Pooling layers, the input of the FC layer is not streamed but available all in one piece. This is possible thanks to the flattening layer, that will not be described here. The FC layer itself is made, as for Convolution layers, of a PE stage and an adder-tree stage. In the
<figure><img src="image_9.png" /><figcaption>Figure 2.7: Illustration of a VGT-generated convolution layer architecture [5].</figcaption></figure>

PE stage, there is one PE per input channel. Those PE stages perform weight multiplications in a fully-parallel way: all input activations are multiplied by all synaptic weights in parallel, thus for a 100-100 layer, there are 100 × 100 parallel multiplications. Then, the results are summed-up to obtain the output activations, using pipelined adder-trees. There is one adder-tree per output neuron so all output activations are processed in parallel.

### 2.2.2.2 C-HLS accelerator

The second FNN accelerator, which we call C-HLS accelerator, has been developed by Sebastien Bilavarn in the LEAT laboratory. This architecture is derived from a C code completely designed to support HLS. This code is a layer based implementation allowing 2D convolutions, ReLU, max pooling and fully connected layers. Layers are configurable (size, input, kernels, ...) such that typical neural network topologies like LeNet can be fully specified and simulated. Therefore this code can also be used to generate RTL IP cores that can be further integrated with a CPU and system bus to be quickly ran on Xilinx devices (Vivado HLS and SDSoC methodology). Loop level parallelism can be explored introducing pipelining pragmas in the original designs, but this is not addressed in this study, where this accelerator is parameterized in a fully-sequential fashion. This choice has been made in order to enhance the comparison between sequential and parallel accelerators in terms of hardware footprint and performance in Chapter 5.

## 2.3 Spiking Neural Networks

In previous section, we mainly focused on "hardware-level" optimizations. If we refer to the terminology proposed by [2] (introduced in section 2.2.1), there are other possibilities for optimization at algorithmic and structural level. In this PhD thesis, we also study the algorithmic-level optimizations by addressing a brain-inspired type of neuron: spiking neurons. Consequently, in this work, we investigate how a neural coding paradigm shift towards spiking domain could help mitigating logic-resources occupation, power and energy consumption of FPGA accelerators. Of course, this matter is always addressed with hardware considerations in mind: the aim is to build an hardware
<figure><img src="image_10.png" /><figcaption>Figure 2.8: Illustration of a Processing Element in VGT convolution layers [5].</figcaption></figure>

platform which takes advantages of the model, and find a model which takes advantages of the hardware possibilities. But first, a short review of Spiking Neural Networks principles will be given to introduce all the concepts involved in our study. Then, hardware implementations of Spiking Neural Networks will be discussed.

### 2.3.1 Principle of Spiking Neural Networks

Spiking Neural Network is a type of brain-inspired Neural Network that emerged from the neurosciences, where it aimed in emulating the biological brain behavior for scientist to study and understand the physical mechanisms underlying information propagation in biological neural networks. The first model to emerge was the very simple Integrate and Fire (IF) neuron model proposed by Lapicque in 1907 [54]. Later, more biologically plausible neuron models were proposed such as the Hodgkin-Huxley (HH) [55] neuron which used a vast set of non-linear differential equations in order to model the mechanism of action potential initiation and propagation in Giand Squid axons. Figure 2.10 shows the biological plausibility of various spiking neurons (i.e. how close they mimick the real physical mechanisms occurring in the biological brain) against their implementation cost in terms of Floating-Point Operation Per Seconds (FLOPS). As it can be seen in this representation, the more a model is biologically plausible, the higher its implementation cost. Thus the HH neurons, and other complex models are not well suited to our search for resource and power efficiency in Machine Learning tasks. For this purpose, we tend to look for models with the lowest implementation cost possible, which we found in the IF neuron model. Moreover, it has been shown by Brette et. al. [56] that the IF-based neuron models are in fact very realistic models when modeling spiking activity in the brain, and are widely used in the literature for that purpose [57] [58]. As this model is both hardware-friendly and effective, we mostly use IF neurons in our works.
<figure><img src="image_11.png" /><figcaption>Figure 2.9: Illustration of a VGT-generated max-pooling layer architecture [5].</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 2.10: Biological plausibility of Spiking Neuron models against implementation as stated in [6]</figcaption></figure>

The IF neuron dynamics are governed by the differential equation 2.2.

$$ \begin{aligned} \frac{dU_i^l(t)}{dt} &= I_i^l(t) - S_i^l(t) \times \theta \\ S_i^l(t) &= \Theta(U_i^l(t) - \theta) \end{aligned} \qquad (2.2) $$

Where $U_i^l$ is the membrane potential of neuron i of layer l, $S_i^l$ the state of the axon (output synapse) of neuron i, $\theta$ is the neuron threshold and $\Theta$ is the Heaviside step function. Moreover, R is the input resistance of the neuron and $I_i^l$ is the input current generated by the input spikes as shown in Equation 2.3:

$$ I_i^l(t) = \sum_j w_{i,j} \times S_j^{l-1}(t) \qquad (2.3) $$

Where $S_j^{l-1}(t)$ is the input spike train and $w_{i,j}$ are the input synaptic weights. A common approach
<figure><img src="image_13.png" /><figcaption>Figure 2.11: Illustration of the Integrate & Fire neuron model. Top: network setup, bottom: membrane potential, input and output activity.</figcaption></figure>

is to approximate the solution of the differential equation 2.2 using the Euler method. The discrete solution is shown in Equation 2.4:

$$U_{i}^{l}[t+1] = U_{i}^{l}[t] + \sum_{j} w_{i,j} \times S_{j}^{l-1}[t] - S_{i}^{l}[t] \times \theta \quad (2.4)$$

In other words, the membrane potential is incremented by the sum of the weighted input activations (i.e. input weights since input activations are binary), and decremented by $\theta$ whenever the neuron threshold is exceeded. Additionally, over-passing the threshold triggers an output spike ($S_{i}^{l}[t] = 1$). This process is summarized in Figure 2.11. The neuron setup depicted in the figure features a red neuron receiving input spikes from the blue and green neurons. The membrane potential graph shows the impact of weighted input spikes on the red neuron potential. The bottom graphs show the input and output spikes from the point of view of the red neuron. As shown in this Figure, the IF neuron integrates temporal and spatial information and produces an output which is propagated towards the following neurons. Thus, its behavior is not far different to that of a Formal neuron, except that i) multiplication-accumulation (MAC) operations are replaced by accumulation operations (ACC), and ii) the model is event-based: spikes are received and processed asynchronously. Hence, SNN acts as a subset of binary networks whose activations are scheduled in time.

Those IF neurons are used to replace standard Formal neurons in mostly any kind of Neural Network Topology. Thus, their exits Spiking Multi-Layer Perceptrons [59], Spiking Convolutional Neural Networks [58], Spiking Auto-Encoders [60]... The only difference is the model of neuron and the encoding of information in the network. In all, SNN process information in a dynamical
fashion using binary activations, whereas FNNs process is static but uses real-valued activations.

### 2.3.2 Spike encoding

If an FNN works on classical data (static images, videos...), SNNs work with spikes. Thus, the network's input data must be in event-based format. There are two ways to obtain such event-based data: either by using a specific event-based sensor, or by transcoding classical data toward spiking domain.

Event-based sensors are mostly present in the field of vision sensors, with the recent breakthrough of Event-Based Camera, also called Dynamic Vision Sensors or Artificial Retinas [61]. This type of sensor is directly inspired from the biological retina behavior: it is made of an array of sensors, each of which detects intensity gradients and generate an event whenever the measured gradient exceeds a certain threshold. Those events usually follow the Adress Event Representation (AER) policy, in which information is represented by a packet containing locations and timestamps of events. The generated event flux then represent the motion in an efficient way, as only the moving objects generate events, and not the static background. Other type of event-based sensor exist for other type of data, such as artificial cochleas for audio data [62], or even artificial event-based skins for tactile sensors [63]. Thanks to the AER representation, event-based sensors produce less information than their classical counterpart, as most of pixels are idle. Moreover, the data itself is lightweight, as it only encodes position and timestamp. Moreover, such data can be directly interpreted by an SNN, and doesn't require a dedicated transcoding pre-processing.

However, event-based sensing is a novel technology which is not yet widespread in industry. In most use-cases, only conventional frame-based sensors are available. Such applications thus require a dedicated pre-processing to encode data into spikes. Their are 3 main spike encoding policies explained below. It should be noted that in our explanations, we deal with images and pixels, but all the notions are translatable towards other type of data.

* Rate coding [64], which consists in generating spike trains for each input pixel whose frequency are proportional to input data;

* Latency coding [65], which consists in generating a single spike per input pixel, where the emission date is proportional to the pixel's intensity;

* Order coding [66] [67], which consists in emitting one spike per pixel in the order of intensity. Pixels are sorted by intensity: the brightest pixel spike first, followed by the second brightest, and so on.

Those three different methods are illustrated in Figure 2.12 for image transcoding case. The pixel intensity is represented by grayscale squares, and the generated spikes by the red arrows on the time axis. The rate coding approach usually offers the best performance, but generates a high number of spikes that drastically impacts both the computation density of the processing and its latency [64]. The latency coding [68] offers inferior performances, but enables to drastically reduce the computation density by reducing the number of spikes flowing into the network. However, the computation duration remains high, due to the width of the time-window [64]. Finally, the order coding seems the most promising: it reduces performance but also reduces both the event density and the computation duration by compressing the spike emission window [66]. However, rate-coding was used during most of this thesis. At the time this work started, that was indeed the most reliable spike encoding policy in terms of network accuracy. Moreover, rate-coding is
<figure><img src="image_14.png" /><figcaption>Figure 2.12: Different spike coding methods: a. is rate coding, b. is latency coding, c. is order coding.</figcaption></figure>

widespread enough so that it is well documented and supported in most SNN frameworks. The rate-coding formula used in this thesis is given in Equation 2.5.

$$ P = \frac{1}{\text{MaxFreq} + (\text{MinFreq} - \text{MaxFreq}) \times v}, \quad (2.5) \\ v = \frac{1 - i}{255} $$

Where P is the spike train period associated to a pixel of value i (8-bit integer), MaxFreq and MinFreq are respectively the maximum and minimum spike train frequencies.

It should be noted that even if most of this work is based on such rate-coded SNNs, other types of encoding where studied during the thesis. In Chapter 7, we address timestep-constrained SNNs trained through Surrogate Gradient Learning [32] [69]. This type of SNN drastically differs from the rate-coded representation.

### 2.3.3 Training SNNs

In this subsection, training techniques for Spiking Neural Networks will be presented. It should be noted that in this work, we focus on classification tasks. If FNNs mostly use variants of the well-known Backpropagation algorithm [25] [26], in SNNs the learning phase can be performed by three very different techniques:

1. Neural network conversion

2. Spiking Time Dependant Plasticity (STDP)

3. Spiking Backpropagation

The first training technique, and the most widely used in the literature, is neural network conversion [70] [57]. This method consists in training a neural network using ReLU neurons in Formal domain using conventional Backpropagation algorithm, and then transferring the learned synaptic weights to a Spiking Neural Network with the exact same topology. Some attention has to be given to the compatibility of the trained network with SNNs: for example, there are no mature ways of implementing Average Pooling layers in SNNs, thus we would rather use Max Pooling layers. Similarly, batch normalization techniques are difficult to apply in SNNs, thus it should not be used in training. Conversion is simple to set, and offers state-of-the-art classification performance in spiking domain. It is compatible with the vast majority of SNN deployment frameworks and neuromorphic accelerators, thus this is the method we have chosen to use in our work.
Secondly, Spiking Time Dependant Plasticity (STDP) [71] [72] is an unsupervised and on-line
training method which exploits the causality effect between spikes to perform data clustering. On-
line means that the training phase is not properly distinct from the test phase, as the network is
continuously learning. Unsupervised means the training data is unlabeled: i.e. it does not require
human effort to label the training set. Indeed, the clustering relies on dataset intrinsic statistics,
and not on human-chosen classes. Additionally, STDP is a local learning rule: each synaptic
weight is tuned by the relative spiking times of uphill and downhill neurons. If an input spike
causes an output spike (i.e. if an output spike is generated shortly after receiving an input spike),
the synapse is potentiated. On the other hand, when an input spike does not cause an output
spike, the connection is depreciated. This local aspect makes STDP more hardware-friendly than
traditional Backpropagation. Moreover, the computation itself is much simpler, as it does not
require complex partial derivative computation. However, this training method is very recent and
still under development. Notably, STDP does not provide state-of-the-art performance on most
classification tasks [73].

The third training method is Spiking Backpropagation [74] [75], also called Surrogate Gradient
Learning [32]. This method is an adaptation of the traditional Error Gradient Backpropagation
algorithm to Integrate & Fire spiking neurons. Indeed, in Error Gradient Backpropagation, the
weight increment (increase or decrease) is proportional to the derivative of the activation function.
However, as we have seen in section 2.3.1, the activation function of the IF neuron is a simple
threshold, i.e. an Heaviside function. The point is that first, this function is not differentiable in 0,
and more importantly, that its derivative equals to 0 elsewhere, thus implying a null weight update.
In order to get around this limitation and still apply Backpropagation to networks of IF neurons,
the creators of Spiking Backpropagation used an approximation of the Heaviside function (usually
a sigmoid or hyperbolic-tangent-based function) to perform the backward path, while the forward
path uses the normal Heaviside activation function. This method has the advantages of offering
state-of-the-art classification performance on most datasets, all the while using fewer spikes and
inference time than converted SNNs [73]. Most of the thesis is based on rate-coded converted SNNs,
but Spiking Backpropagation has recently gained visibility and so have its promises for hardware
acceleration. Hence, Chapter 7 addresses the potential benefits and hardware implementation of
SNNs trained via Surrogated Gradient Learning.

2.3.4 Terminate Delta

It has been shown in section 2.3.2. how to encode conventional data like images and time series
into spikes. In this subsection, the Terminate Delta process is explained. It is used in this work
to interpret the output spikes and retrieve the predicted class accordingly. This classification
procedure is taken from N2D2 [1], a Neural Network training and deployment framework which
enables conversion towards spiking domain.

This process consists in determining the most active neuron, using a margin whose character-
ized by the parameter Delta. With a Δ value of 2, it means that the most spiking neuron would
be enacted after it has spiked two times more than the second most active neuron. Indeed, with
rate-based SNNs, spikes arrive frequently at the output. Hence, tuning Δ enables to take account
of more or less spikes before enacting the result. A high Δ value will give better classification
performances, but higher execution time and higher number of spiking events in the network.
Thus, the Delta parameter is an interesting parameter to tune the trade-off duration and energy
against accuracy.
## 2.4 SNNs in hardware

Spiking domain is expected to bring drastic hardware footprint reduction to neural network accelerators compared to conventional formal coding domain. Indeed, the simplicity of the Integrate & Fire (IF) neuron model, coupled with the lightweight binary activation, could bring both logic resource reduction and power efficiency.

### 2.4.1 Advantages of SNNs in Hardware

Spiking Neural Networks have several advantages over Formal Neural Networks when addressing hardware implementations. The first lies in the activation integration mechanism: the synaptic operation. For Integrate & Fire spiking neurons, as described in Equations 2.4, the integration consists in accumulating a synaptic weight whenever a spike is received. Basically, this consists in a simple Accumulation (ACC) operation. For formal neurons on the other hand, the synaptic operation is a Multiplication-And-Accumulation (MAC), as the weight is multiplied by the real-valued activation. This difference has two main impacts on FPGA implementations: the MAC uses a Digital Signal Processing (DSP) unit, which is a scarce resource on most FPGAs. Moreover, the MAC operation is considered to be at least ten times more energy-intensive than the ACC operation. Indeed, on 45nm CMOS, Panda et. al. [76] reported $3.2pJ$ for a 32-bit MAC against $0.1pJ$ for a 32-bit ACC. The ratio of energy consumption between MAC and ACC operation will be studied further in Chapter 4.

Consequently, hardware SNNs should be more energy-efficient than FNNs [31] [77] [78], based on the energy cost of their respective synaptic operations. Additionally, spikes can be represented by binary (1-bit) signals, which offers lightweight connections between neuron, where formal activations are encoded on signal from 8 to 64 bits. This enables better scalability for parallel implementations, which involves a large number of physical connections between hardware neurons [79]. Finally, the event-based processing also offers interesting properties for highly-constrained embedded systems. Indeed, in SNNs, activations are scattered in time in a sparse fashion, whereas all synapses of a layer are activated simultaneously in FNNs. In other words, the information in SNNs is dynamic: it is spread on a temporal dimension, which offers low instantaneous computing density [79]. In an FNN however, the behavior is fully static, implying high computation density. This difference enables for better multiplexing possibilities, as well as low-power implementations. However, this low instantaneous dynamic power might be counterbalanced by longer processing time, thus energy consumption might not follow the same trend. However, energy is not the only constraint, and power is often a limiting factor in embedded-systems, in solar-powered systems for example.

### 2.4.2 Literature review

In this section, we give a short list of recent works in the field of FPGA SNN accelerators. For the architectures where the information was available, logic resources, execution time, power and energy consumption are summarized in Table 2.1. As some architectures are not named by their authors, we refer to them with names inspired from their respective article titles.
<table><caption>Table 2.1: Logic resources, execution time, power and energy of FPGA SNN accelerators found in the literature. Topology nomenclature: KcWsX = Convolution layer with K filters of size W² and a stride of X, KpWsX = Pooling layer with K filters of size W² and a stride of X. Fully-connected layers are referred to by their number of output neurons.</caption><thead><tr><th>Name</th><th>Year</th><th>Data</th><th>Topology</th><th># Synapses</th><th>Ressources</th><th>Power (mW)</th><th>Time per image (ms)</th><th>Energy per image (mJ)</th></tr></thead><tbody><tr><td>“ConfConvNode” [7]</td><td>2018</td><td>events</td><td>CNN: 32x32-6c5s1-6p2s2-4c5s1-4p2s2-8c5s1-4c2s1</td><td>904</td><td>21K LUTs<br>38K FFs<br>6.5 Mb BRAM</td><td>7.7</td><td>25</td><td>0.2<br>(220 nJ/syn)</td></tr><tr><td>“Large Scale SNN” [80]</td><td>2012</td><td>events</td><td>Toroidal: 2048 neurons</td><td>2,048M</td><td>80K LUTs<br>91K FFs<br>20 DSPs<br>8.6 Mb BRAM</td><td>?</td><td>?</td><td>?</td></tr><tr><td>“Polychronous SNN” [8]</td><td>2013</td><td>events</td><td>Unstructured 4096 neurons</td><td>1,150M</td><td>135K LUTs<br>581K FFs<br>14.4 Mb BRAM</td><td>?</td><td>?</td><td>?</td></tr><tr><td>“Low Power SNN” [9]</td><td>2020</td><td>frames</td><td>CNN: 28x28-64c5s1-64p2s2-64c5s1-64p2s2-128-10</td><td>37,248</td><td>140K LUTs<br>81K FFs<br>16.5 Mb BRAM</td><td>4600</td><td>6.1</td><td>35<br>(940 nJ/syn)</td></tr><tr><td>“IIR SNN” [11]</td><td>2020</td><td>frames</td><td>MLP: 28x28-500-500-10</td><td>647,000</td><td>125K LUTs<br>185K FFs<br>1028 DSPs<br>3.1 Mb BRAM</td><td>4500</td><td>0.52</td><td>2<br>(3.1 nJ/syn)</td></tr><tr><td>HFirst [12]</td><td>2015</td><td>events</td><td>CNN: 128x128x1-12c7s2-12p4s1-36c8s2-36p32s1</td><td>2892</td><td>17 DSP<br>6 Mb BRAM</td><td>250</td><td>0.002</td><td>0.5<br>(170 nJ/syn)</td></tr><tr><td>S2N2 [13]</td><td>2021</td><td>time-series &amp; frames</td><td>CNN: 16x16-64c5s1-64c5s1-128c3s1-128c3s1-1024-24</td><td>554368</td><td>102K LUTs<br>34K FFs<br>40 DSPs<br>12.7 Mb BRAM</td><td>?</td><td>0.031</td><td>?</td></tr><tr><td>SPLEAT [14]</td><td>2020</td><td>frames</td><td>CNN: 28x28-6c5s1-6p2s2-16c5s1-16p2s2-84-10</td><td>22894</td><td>4.8K LUTs<br>3.2K FFs<br>1 DSP<br>0.45 Mb BRAM</td><td>315</td><td>1.4</td><td>0.5<br>(22 nJ/syn)</td></tr></tbody></table>

### 2.4.2.1 “ConfConvNode”

In [7], Camunas-Mesa et. al. introduced a configurable event-driven architecture for convolutional layers targeting FPGAs. This architecture addressed IF neurons, and a conversion technique was used for training. The core of this architecture is a Processing Element (PE), a building block for user-defined spiking CNN architectures. For a better understanding, a schematic of the circuit of a PE is given in Figure 2.13. The PE involved a convolutional processing unit and a routing module. The convolution processing unit was used to convolve a spike with the convolution filters, and perform the IF neuron thresholding operation. The routing module was in charge of reading the input spike addresses and sending the output spikes to their destination neuron. In the paper, the architecture was used to classify poker cards filmed in real time (25ms per sample) using a Dynamic Vision Sensor (DVS) [61] (see section 2.3.2). In real-time, the architecture used 7.7mW and 192.5µJ. The authors also evaluated the architecture on slowed DVS recordings: using a slow-down factor of 10, the architecture used 5.25mW, and 0.85mW with a slow-down factor of 100.

### 2.4.2.2 “Large Scale SNN”

In [80], Cheung et. al. introduced a fully-parallel SNN hardware architecture targeting off-the-shelf FPGA-based systems. As this architecture primarily targeted cortical simulation, the implemented neuron was an Izhikevich model [6], but the authors claimed that it could be easily changed to a more hardware-friendly IF model. The design was highly-pipelined, ensuring low ex-
<figure><img src="image_15.png" /><figcaption>Figure 2.13: Illustration of the Convolutional Processing Element found in [7].</figcaption></figure>

execution time. Moreover, the architecture involved a weight-distribution module, which was used to parallelize accesses, distributions and integrations of synaptic weights concurrently. Additionally, the synaptic weights were arranged in consecutive RAM locations, enabling to retrieve several data in a single RAM access. This weight-distribution module is illustrated in figure 2.14. Moreover, the architecture worked in a fully event-driven way thanks to FIFO queues between neurons. This architecture supports 64000 neurons at a maximum rate of $1.39 \times 10^9$ spikes/s, with a spike-rate delivery up to 1.4 times faster than GPU at the time of the study (2012). However, this metric is not standard and not comparable to other architectures. Moreover, the number of spikes strongly depends on the data and spike encoding. The impact of data on number of spikes will be studied in details in Chapter 4.

### 2.4.2.3 "Polychronous SNN"

Reference [8] introduced a hardware architecture for polychronous Spiking Neural Networks [81]. This type of neurons is based on latency coding rather than rate coding: spikes are associated with precise latencies so that they arrive simultaneously to their destination neuron and cause it to fire, despite being emitted asynchronously. The neuron circuit is illustrated in figure 2.15. In this figure, timers are used to detect when pre-synaptic spikes arrive within 3ms of each other. It should be noted that these delay is programmable. The network was trained in hardware using a kind of STDP learning rule, adapted to programmable delay synapses. Using this approach, this type of SNN is able to process complex spatio-temporal patterns. The architecture was highly multiplexed to ease scalability. In the article, the authors implemented a 4000 neurons network with 1.15 million programmable delay synapses on FPGA, and achieved real-time simulation of cortical neurons. The authors also claim that their implementation is robust to noise from random input spikes. This approach is promising for hardware implementations of Machine Learning applications: encoding information in relative timing of events could increase the quantity of information conveyed by spikes while reducing their number. However, STDP rules usually provide lower accuracy than other methods, thus the interest of this technique remains to be proven on classification tasks.
<figure><img src="image_16.png" /><figcaption>Figure 2.14: Illustration of the weight distribution technique used in [7] to limit the memory access rate. module has 3 input synapse data per cycle. The neuron 1 connects to neurons 2, 3, 7, 8 and 9 in the network, and neuron 1 is fired in the last time step. The figure shows the accumulation of three synaptic weights ($w_{3,1}$, $w_{8,1}$ and $w_{9,1}$).</figcaption></figure>

#### 2.4.2.4 “Low Power SNN”

In [9], Ju et. al. presented an SNN architecture targeting FPGA for integration in low-power systems, supporting all basic CNN layer types. The authors addressed the hardware-friendly IF neuron model, and used network conversion for training, with a weight balancing method [82] to optimize spike emission rate. Several optimizations were made to increase hardware efficiency of the architecture. First, authors used a 8-bit fixed-point dynamic. Moreover, convolutions were implemented in a parallel, pipelined and systolic fashion using shift-registers, ensuring fast processing. For a better understanding, a schematic representation of the pipelined parallel convolution implementation is shown in Figure 2.16. Additionally, the whole network was fully pipelined to further reduce processing latency. The architecture has been implemented on a Xilinx's ZCU102 FPGA, achieving 98.94% recognition rate on MNIST at 164 FPS. This is equivalent to an acceleration of 41× against CPU and 22× against GPU. The architecture used 4.6W and 35mJ per image. Additionally, the authors proposed an hardware-efficient implementation for spiking Pooling layers.
<figure><img src="image_17.png" /><figcaption>Figure 2.15: Polychronous neuron circuit overview found in [8].</figcaption></figure>

### 2.4.2.5 "IIR SNN"

In [11], Fang et. al. proposed a novel way to implement for SNNs in hardware, and introduced their subsequent architecture. The accelerator supported all CNN standard layers. The authors used Leaky IF neurons, which they demonstrated to be equivalent to Time Encoding Machines. In the light of this demonstration, the authors developed a flexible model in which SNNs are emulated by a network of Infinite Impulse Response (IIR) units, which drastically reduce latency by enabling neurons to encode information on small time windows. In the architecture, each layer is emulated by a Processing Element in a multiplexed fashion, and it features a layer-level pipeline for execution time reduction. The whole architecture is generated through High Level Synthesis (HLS), as part of a global framework. This framework covers all steps from training, quantization, architecture generation and deployment. An illustration of the full framework is given in Figure 2.17. Training is performed thanks to surrogate gradient back-propagation (see section 2.3.3) using PyTorch. The framework involves resource optimization by using pragmas in the C-HLS code to increase or decrease parallelism according to the network size. Using their framework and architecture, the authors obtained up to 2.9x speedup compared to Intel's Loihi [83], 38.5x compared to SpiNNaker [84], 1.9x compared to TrueNorth [85] and 3.3x compared to Minitaur [86]. Concerning energy consumption per image, the authors achieved a 33.6x reduction compared to Nvidia RTX 5000, 2.6x compared to Loihi and 2.82x against SpiNNaker.
<figure><img src="image_18.png" /><figcaption>Figure 2.16: Illustration of the pipelined parallel convolution implementation found in [9]. The input data is first flattened in rows, and fed in a shift register. A systolic array of Convolution Units [10] then performs the operation in a pipelined fashion. A: Situation at clock T, B: situation at clock T+1.</figcaption></figure>

### 2.4.2.6 HFirst

In [12], Orchard et. al. proposed an SNN architecture tailored for DVS signal processing, using Address Event Representation (AER), for spatial pattern recognition. The accelerator supported all standard CNN layers. According to the authors, the asynchronous nature of events ("frame free" data) frees computation and communication from the rigidity of conventional clocked systems. Using timing of spikes in object recognition, the authors demonstrated a drastic simplification of computation. Indeed, a simple asynchronous temporal-winner-take-all (temporal WTA) pooling operation was used instead of more complex synchronous operations usually implemented in SNN accelerators. This temporal WTA drastically mitigated the processing latency as there was no need to accumulate spikes to perform pooling. The FPGA accelerator was built in a highly parallel and pipelined fashion and was able to reach real-time recognition of DVS data such as written characters, written digits, and playing cards, with a latency inferior to 2µs. Example of DVS recordings of such samples are shown in Figure 2.18. Additionally, the power consumption of the system depended on the nature of the scene. A scene without any movement used 100mW, but this value climbed up to 250mW for scenes with a lot of action (i.e. a lot of input events).

### 2.4.2.7 S2N2

S2N2 was introduced in [13], and was an FPGA architecture for streaming SNNs, which implemented Leaky IF neurons and time coding policy. The architecture supported all standard CNN layers. In this work, the term "streaming" was used to describe the temporal nature of data, which is streamed at the input of the network. The authors stated that SNN hardware implementation usually involved systolic array-based computation [10]. This kind of systems benefits from a supposed high sparsity in spiking events to reduce computational requirements. This work
<figure><img src="image_19.png" /><figcaption>Figure 2.17: Illustration of the training, compression, conversion and hardware deployment framework proposed in [11].</figcaption></figure>

showed however that this assumption was not always true in the case of data with large temporal dimensions, such as DeepSig RadioML 2018 dataset [18]. The authors proposed an architecture adapted to such streaming SNNs developed using FINN framework [87]: this framework is tailored for binary Neural Network, and has been customized by the authors to support SNN development and deployment. As it is shown in Figure 2.19, the architecture flattened input data into streams and convolution was operated in a parallel SIMD (Single Instruction on Multiple Data) fashion. The resulting architecture involved intra and inter-layer parallelism, and a pipeline to efficiently process data streams. The authors obtained a precision of 68.5% on RadioML 2018 dataset.

### 2.4.2.8 SPLEAT

SPLEAT [14] is an architecture for configurable SNN deployment on FPGA, which uses IF neurons. The architecture supports convolutional, pooling and fully-connected layers, and supports both temporal and rate coding policies thanks to a configurable spike generation cell. The architecture has been designed to be easily configured and deployed on FPGA. It features inter-layer parallelism (i.e. each layer is emulated by a dedicated Neural Processing Unit) and layer-wise pipeline to reduce execution time. For a better understanding, an illustration of the architecture of such Neural Processing Unit is given in Figure 2.20. Each Neural Processing Unit emulates a given layer in a highly sequential fashion, enabling high scalability. Due to this implementation choice, the FPGA logic occupation of this architecture is really low. On MNIST dataset, in average, the architecture used 315mW, with an execution time of roughly 100ms per image. It should be noted that the processing time varies widely between two images, as the number of spikes vary from one sample to the other. The architecture also features a specific module for output classification, which interprets the spikes using a terminate delta process (see section 2.3.4).
<figure><img src="image_20.png" /><figcaption>Figure 2.18: Illustration of stabilized DVS data used for training and testing in [12].</figcaption></figure>

### 2.4.2.9 Conclusion

In this short literature review, we have seen that implementation choices vary greatly from one architecture to the other, depending on the chosen spiking neuron model and spike encoding policy. In all cases, much effort was given by the authors to reduce processing latency, by using pipelines and systolic architectures in some cases. Moreover, authors often developed specific hardware-friendly layers or neuron models in order to reduce hardware footprint of the subsequent implementations, in a software-and-hardware co-design fashion. Overall, this literature review shows that SNNs are promising for hardware footprint reduction of NN implementation on FPGA. However, this study lacks information on fair comparisons between hardware SNNs and FNNs. Therefore, this matter is specifically addressed in the next section.

## 2.4.3 Confronting Spiking and Formal Neural Networks

In this section, we introduce and describe previous literature works which tackle the comparison between formal and spiking neural network implementations on FPGA. In a first part, we address logic resources usage comparisons, and in a second part, energy consumption comparisons.

### 2.4.3.1 Logic resources comparison

In [88], Khacef et. al. compared a Multi-Layer Perceptron (MLP, i.e. a neural network made of several fully-connected layers) implementations on FPGA in both spiking and formal domains, in identical conditions. The SNN used IF neurons and rate-coding to translate static images into spike trains. The authors demonstrated a reduction of the FPGA logic resources occupation by 50% when using spikes compared to formal coding on MNIST dataset, showing that a single coding paradigm switch with no further optimizations could drastically reduce logic resources usage of NN implementations. Moreover, the authors showed a decrease of 59% in dynamic power usage, which is coherent with the difference in logic resources usage. However, this worked only compared the neuron layers, and did not take in account the resource overhead involved by spike encoding and output interpretation. Indeed, the spikes must be encoded into spike trains using rate-coding, which requires an additional encoding module in the FPGA. In a similar way, the output decoding via terminate delta process requires a dedicated module in FPGA. Thus, the
<figure><img src="image_21.png" /><figcaption>Figure 2.19: Schematic overview of the FINN architecture found in S2N2 [13]. The SWU (Sliding Window Unit) flattens the input data and forwards it to the MVTU (Matrix Vector Threshold Unit). Each Processing Element (PE) inside the MVTU processes one output channel and has a number of SIMD (Single Instruction on Multiple Data) lanes that read from input channels and multiply the input by kernel weights in parallel.</figcaption></figure>

total logic occupation might be counterbalanced by those two modules. Despite efforts to find other comparisons between formal and spiking domains in terms of FPGA logic occupation, none was found. Consequently, there is a need for further work to confirm those results, and generalize to other datasets and more complex neural network topologies.

### 2.4.3.2 Energy comparison

On the other hand, the comparisons of SNN and FNN FPGA accelerators regarding computing needs and energy consumption have been studied on several occasions in the literature. More than comparing implementations, those works focus on building a theoretical model to evaluate the potential energy consumption reduction (or increase) when using spiking domain compared to formal domain.

In [89], Han *et. al.* proposed to evaluate energy benefits of SNNs when compared to FNNs. To do so, they proposed that energy benefits were higher at lower spike firing rates, *i.e.* a lower number of spikes means lower number of operations and hence lower energy consumption. The authors proposed to evaluate the influence of the time-window length on spike firing rate. In
<figure><img src="image_22.png" /><figcaption>Figure 2.20: Illustration of the Neural Processing Unit of SPLEAT [14]. This NPU is used to emulate a convolution, pooling or fully-connected spiking layer in order to build a full SNN on FPGA.</figcaption></figure>

timing-aware systems such as this one, time is represented as discrete increment named time-steps. Thus, the “time window” is the number of time-steps available for spike encoding. The study concluded that spike density drastically increased with respect to the number of time-steps. Thus, lower number of time-steps should be preferred in order to minimize energy consumption.

In [31], Sengupta et. al. proposed a novel ANN-to-SNN conversion technique for deep neural networks, such as VGG-16 [28] and deep Residual topologies. This conversion process uses threshold and weight balancing techniques in order to optimize the spiking activity. Using this method, the authors performed the first ever SNN test on the full ImageNet 2012 dataset [90], achieving 30.04% top-1 error rate and 10.99% top-5 error rate. In this paper, the authors also proposed to compare the original FNN and converted SNN regarding number of operations. The authors found that, with VGG-16 architecture, there were 1.9× more ACC operations in the SNN than MAC operations in the FNN, and 2.4× for the Residual architecture. The authors proposed this metric in order to evaluate both the relevance of their conversion technique in terms of computation reduction and energy efficiency.

In [77], the authors proposed to evaluate and compare the energy cost of formal and spiking domain in a theoretical approach. The authors focused on rate-coded SNNs, as they claim it is the most used and straightforward way to deploy SNN using conversion training technique. They proposed two equivalent hardware implementations for spiking and a formal neurons and deduced a theoretical energy cost for each of them, by dividing the processing into operations of known energy cost (multiplication, accumulation, RAM access...). The authors estimated that the spiking neuron energy consumption was lower to that of the formal neuron if and only if the number of spike per synapse was inferior to 1.72. According to the authors, such value is hardly realistic
in the case of rate-coded SNNs. They concluded that efforts must be taken to benefit from the event-based nature of SNNs. For example, they proposed to address the spike density problem by developing novel, high-sparsity spike encoding methods.

In [78], the authors proposed a compression technique associated to FNN-to-SNN conversion, aiming in minimizing spiking activity. In doing so, the authors hope to reduce energy consumption of the subsequent SNN deployment. The iterative compression technique is attention-guided: attention-maps derived from an uncompressed “teacher” model are used to prune the synaptic connections of a “student” model. The model is then converted toward spike domain, obtaining up to 33% compression rate on VGG-16 without dramatic performance loss. The authors used an IF neuron model. They also proposed a spiking back-propagation learning technique based on surrogate gradient (see 2.3.3). This method is used to re-train the model after compression and conversion, in order to reduce the number of time-steps used for inference. Using this full compression, conversion and re-training framework, the authors demonstrated near state-of-the-art classification accuracies on CIFAR-10 (91.28%) and CIFAR-100 (64.98%), using deep VGG-16 or ResNet topologies, with drastically lower spiking activity than other methods. The authors measured that their compressed SNN used 38.7 times fewer energy than an equivalent uncompressed ANN, and 12.2 times fewer energy than an equivalent ANN with identical compression on CIFAR-10 dataset. Compared to un uncompressed SNN, energy consumption was divided by 2.5. The authors also proposed a method to link energy consumption to spiking activity. Indeed, the authors approximate the energy consumption of an network to the energy consumption of its main operation: ACC for an SNN and MAC for an FNN. Counting the number of spikes, the authors were able to estimate the relative energy consumption of an FNN and a derived converted SNN. This approximation was used to give estimations of relative energy consumption in SNNs and FNNs on CIFAR-10, CIFAR-100 and Tiny ImageNet datasets. In all cases, the authors estimated that SNNs brought energy savings, and even-more when using the proposed compression and re-training technique with high sparsity targets.

## 2.5 Conclusion

In this section, we have presented previous works studying comparisons of spiking and formal hardware neural network implementations. In terms of logic resources usage, a single paper [88] was found, which found interesting area and power usage savings when shifting from formal to spiking paradigm. However, the literature clearly lacks sufficient comparisons: the only paper which proposes such a study focuses on a single MLP architecture, which is not sufficient to draw general conclusions. To do so, extensive comparisons covering various layer sizes, number of layers, and type of layers (convolution and pooling) must be undertaken. Moreover, this study does not take spike encoding in account, which could counter-balance the results. Finally, this comparison only covers fully-parallel implementations, and should be extended to other architectural choices and level of parallelism.

On the other hand, many papers tackle the energy-efficiency comparison between formal and spiking hardware neural networks in a theoretical manner. In all the mentioned studies, authors have drawn a strong dependence of energy consumption to spike firing rate. Accordingly, a lower number of spikes during processing means a lower energy consumption. In some of those papers [31] [78], researchers proposed to sum-up the energy consumption of an hardware NN by the number of synaptic operations. Those synaptic operations are simplified in their energy estimation models, so a formal synaptic operation is a MAC, and a spiking one is an ACC. Moreover, au-
thors in [89], [31] and [78] proposed techniques to reduce the spiking activity at software level, by using novel conversion or training techniques. However, there is still a lack for fair and extensive comparison between hardware FNN and SNNs, covering various topologies, datasets, and architectural choices. Moreover, the comparisons are often theoretical and their is a lack of quantified comparison between formal and spiking accelerators. Additionally, the correlation between spiking activity and energy consumption must be studied further and quantified, in order to ensure the validity of such an high-level energy estimation model. Indeed, this model is much simpler than reality: it does not take memory access, level of parallelism, or other architectural choices in account.

## 2.6 Contributions

In this section, we provide details on the positioning and contribution of this thesis regarding the above described state of the art in Neuromorphic Engineering. Each of the following subsection is dedicated to a particular aspect of those contributions.

### 2.6.1 Synaptic Activity

In the literature, spiking activity is considered as a reliable metric to evaluate energy consumption in Neuromorphic systems [31] [77] [78]. It is a common idea that the energy consumption of an hardware neural network can be approximated by the cost of synaptic operations: Multiplication-Accumulation (MAC) for formal neurons and Accumulation (ACC) for spiking ones. Since the MAC operation is more energy-intensive than the other, SNNs are expected to consume less energy. However, the number of operations in an SNN is unpredictable: it depends on synaptic activity. If there is more than one spike per synapses, *i.e.* more ACCs in the SNN than MACs in the FNN the SNN might consume more energy overall. One of the contribution of this thesis is to investigate the correlation between synaptic activity and energy consumption of hardware SNNs (Chapter 4).

### 2.6.2 Quantitative comparison of formal and spiking domains

The bibliography proposed in this section outlines a lack of fair and extensive comparison between formal and spiking implementations of neural networks on FPGA. To our knowledge, a single paper [88] proposes a quantitative measure of the logic resources savings when using spike coding compared to formal coding. Regarding energy consumption comparison, some papers [31] [77] [78] address the issue at a theoretical level, based on the number of operations in each coding domain (ACC and MAC). However, those papers does not perform clear power or energy measure, and only produce theoretical high-level indications. Consequently, one of the objectives of this thesis is to perform an extensive comparison between formal and spiking implementations on FPGA, using various level of parallelism, and on an extensive application benchmark. Rather than directly proposing a benchmark of applications and accelerators, we propose an estimation framework. The framework provides resource, power, inference time and energy estimations for formal and spiking neural networks under two extreme level of parallelism: fully-parallel and fully-multiplexed implementations. In doing so, the framework eases the extensive comparison between formal and spiking coding domains. This framework is described in Chapter 5, and applied to a range of representative datasets.
### 2.6.3 Cartography of applications and neural coding domains

Some of the aforementioned datasets are images, and others are vector datasets (Fourier Transform of voice recordings, sonar echoes...). Some have high number of classes, and some are binary tasks. This approach aims in proposing a comparison in a wide, representative range of applications. In doing so, we also investigate the conditions that seems suitable to neuromorphic acceleration. Indeed, some tasks may be suited to spikes while some others might not. The goal of this thesis is therefore to propose a cartography of neural coding domain with respect to applications. More generally, we are interested in finding the suitable conditions for spiking neural networks. Such conditions ranges from the resolution of input data to the depth of the network and the number of classes. Moreover, this cartography is complemented by the influence of parallelism on neuromorphic accelerators.

### 2.6.4 How to benefit from spiking domain ?

The proposed study led to several assessment regarding the energy consumption of rate-coded SNN accelerators on FPGA. In addition to the cartography of applications, we also propose several contributions and insights regarding hardware implementation of SNNs. Those contributions are:

* Neural coding domain hybridization (Chapter 6): tailoring coding domain to synaptic activity at layer level, in order to draw the best from both worlds.

* A prototype of hardware implementation of timestep-constrained SNNs trained through Surrogate Gradient Learning and using Send-on-Delta spike encoding. This method is expected to reduce synaptic activity and inference time in SNNs while maintaining state-of-the-art classification accuracy [73] [32] [91].

The development of those architectures was led by the insights provided by the study of energy consumption in neuromorphic accelerators. In both cases, the goal is to reduce synaptic activity to minimize both the number of operations (i.e. spikes) and temporal sparsity of spikes.
# Chapter 3

## Spiking Neural Networks parallel implementation: PADS

### Chapter contents

<table><tr><td>3.1 Hardware Architecture</td><td>35</td></tr><tr><td>3.1.1 Spike Generation Cell</td><td>35</td></tr><tr><td>3.1.2 Neural Processing Unit</td><td>39</td></tr><tr><td>3.1.3 Terminate Delta Module</td><td>41</td></tr><tr><td>3.2 Hardware Synthesis Results</td><td>42</td></tr><tr><td>3.2.1 Methodology</td><td>43</td></tr><tr><td>3.2.2 Comparison with VGT</td><td>45</td></tr><tr><td>3.2.3 Spike Generation Overhead</td><td>46</td></tr><tr><td>3.2.4 Conclusions on PADS hardware implementation</td><td>49</td></tr><tr><td>3.3 Conclusion</td><td>50</td></tr></table>
In this chapter, we are going to introduce PADS (Parallel Architecture for Dense Spiking layers), an FPGA accelerator for spiking fully-connected layers. In a first section, we explain the hardware implementation of PADS in details. Then, the architecture is compared to an equivalent highly-parallel accelerator for Formal Neural Networks. This comparison brings our first insights on the cartography of suitable application domains and contexts for PADS accelerator. Moreover, a System-on-Chip (SoC) architecture is developed for PADS testing and deployment in FPGA.

## 3.1 Hardware Architecture

PADS is developed as a prototype for rate-based Spiking Neural Networks acceleration on FPGA. In doing so, the goal is to provide an experimental prototype for the thesis, serving for experiments and further improvements. Two major choices have been made during development: first, PADS only covers Fully-Connected layer. That is, in order to reduce development time before obtaining a prototype. Second, PADS uses a highly parallel implementation. This choice is motivated by two aspects: On the one hand, to reduce execution time and mitigate dynamic encoding drawback. On the other hand to facilitate development for the aforementioned reasons.

PADS uses the Integrate & Fire neuron model described in Section 2.3.1. The architecture comes in the form of a parameterizable VHDL code. The topology is defined in a package file, and is automatically mapped in the FPGA during synthesis. Therefore, each PADS implementation is specific to a defined MLP topology, in contrast with a programmable core such as Xilinx DPU [92]. PADS is designed to run with a clock period of 10ns (100MHz). In PADS, each neuron is physically implemented by a dedicated Neural Processing Unit (NPU). Each synapse is also physically instantiated in the design by a dedicated wire. Apart from spiking fully-connected layers of IF neurons, the architecture features a Spike Generation Cell (GenCell) for spike encoding (Section 2.3.2), and a Terminate Delta Module (TDM) for output decoding (Section 2.3.4). An overview of the architecture is presented in Figure 3.1. On this figure, each NPU is represented with a different color. The output of each NPU is connected to all downstream neurons in a fully-parallel fashion. Only two layers are represented in the figure but there can be any number of layers.

In Figure 3.1, the colored arrows are hardware synapses: 1-bit signals carrying NPU output spikes. The black arrows emerging from the Spike Generation Cell are also hardware synapses but contain an 8-bit address signal in addition to the 1-bit spike signal. The reason behind this address signal will be explained further in 3.1.1. The *inputPixel* signal carries input data in a streaming fashion, one pixel at a time in a given order. The *outClass* signal carries the identifier of the winning class, and the *stopNetwork* signals is triggered when the Terminate Delta condition is reached.

### 3.1.1 Spike Generation Cell

The Spike Generation Cell, also called GenCell, is in charge of the spike encoding of input data into spike trains following the rate-coding policy (Section 2.3.2). In the model, each input pixel is associated to an input neuron which emits spikes at a rate derived from the rate-coding equation (Equation 2.5). The GenCell emulates all input neurons in a sequential fashion. The GenCell receives a stream of input pixels (8-bit integers) at a rate of one pixel per clock cycle. The GenCell's output is made of two signals: a 8-bit address signal, and a 1-bit spike signal. The
<figure><img src="image_23.png" /><figcaption>Figure 3.1: High level representation of PADS architecture, for a 2-layers spiking MLP. FCi stands for the $i^{th}$ fully-connected layer with Ni neurons.</figcaption></figure>

corresponding architecture is given in Figure 3.2, and the spike generation process is illustrated by the flowchart given in Figure 3.3. Two processing phases can be distinguished:

* The initialization phase: In the first time-step ($v_time=0$) the GenCell receives the input input frame in a streaming fashion, one pixel per clock cycle. Each pixel is associated with a period corresponding to the frequency of the spike train to be generated (Equation 2.5).

* The spike emission phase: On the following time-steps ($v_time \approx 0$), the input periods are compared to the current $v_time$. A spike is emitted when $v_time \equiv 0 \pmod{P}$ (P the spike train Period). In other words, a spike is generated when the remainder of the division between $v_time$ and $P$ is zero.

Before going in more details, it should be noted that the rate-coding policy is pre-computed off-line to simplify the implementation. Pixels being 8-bit encoded, there are 256 possible pixel values and thus 256 possible spike train periods. The 256 periods are computed off-line and stored in a hard-coded LUT. The period corresponding to a pixel value $i$ is stored at the $i^{th}$ cell of the aforementioned LUT.

### 3.1.1.1 Double Counter

The GenCell features a nested double counter which paces the spike generation process. The outer counter controls discrete “virtual” time increments. Indeed, the spike train periods are defined in terms of “virtual” time-steps rather than real time. This “virtual” time is referred to as $v_time$ in the following. At each $v_time$ increment, an inner counter $C_{pixel}$ scans the input neurons, at a rate of one per clock cycle and generates a spike if $v_time \% P = 0$. For example, the GenCell time-step length (i.e. $v_time$ increment) on MNIST is 784 clock cycles (28x28p samples). The value of $C_{pixel}$ thus points to the current active input neuron (i.e. input pixel). Moreover, the output address signal ($o_addr$) always carries the value of $C_{pixel}$. In all, the double counter acts as a finite state machine which controls the GenCell behavior.
<figure><img src="image_24.png" /><figcaption>Figure 3.2: Block Diagram of the Spike Generation Cell.</figcaption></figure>
<figure><img src="image_25.png" /><figcaption>Figure 3.3: Spike emission process flowchart. Initialization is highlighted in green, emission in red.</figcaption></figure>
### 3.1.1.2 Initialization Phase

The initialization phase occurs when $v_time=0$. During this phase the GenCell receives the input data stream at a rate of one pixel per clock cycle. The stream must be synchronized with the $C_{pixel}$ counter so that each input neuron can be associated to the right pixel value. For each pixel, the corresponding spike train period is retrieved in the rate-coding LUT. The subsequent period value is stored in the Period LUT (P-LUT). Additionally, a copy of the value is stored in the Next Spiking Time LUT (NST-LUT) as explained in the following. The address in the LUT is given by the neuron counter ($C_{pixel}$). The NST-LUT is used to prevent the use of a hardware-intensive modulo operator. That is, the NST-LUT directly indicates the “next-spiking time”. When $C_{pixel}$ reaches the last pixel, $v_time$ is incremented and the Spike Emission Phase begins.

### 3.1.1.3 Spike Emission Phase

The spike emission phase begins when $v_time>0$. The process is written in pseudo-code in Algorithm 1, and is also shown in the red section of the flowchart in Figure 3.3.

**Algorithm 1**

Pseudo-code of the Spike Emission Phase in the GenCell process

$v_time \leftarrow 1$
while stop_network \neq True do
  $C_{pixel} \leftarrow 0$
  for n in [0, $N_{pixel-1}$] do
    if NST_n = $v_time$ then
      $o$\_spike \leftarrow True
      $o$\_addr \leftarrow $C_{pixel}$
      NST_n \leftarrow NST_n + P_n
    else
      $o$\_spike \leftarrow False
      $C_{pixel} \leftarrow C_{pixel} + 1$
      $v_time \leftarrow v_time + 1$

At each $v_time$ increment, the GenCell scans all the input neurons one by one. For a given neuron $n$ ($C_{pixel} = n$), the Next Spiking Time $NST_n$ is retrieved from the NST-LUT. The period $P_n$ is retrieved from the P-LUT. If $v_time=NST_n$, a spike is emitted. It should be noted that the corresponding neuron address ($C_{pixel} = n$) is carried by $o_addr$ output signal. The Next Spiking Time is incremented by $P_n$ and the new value is stored in the $n^{th}$ cell of the NST-LUT. This whole process has been developed to avoid using a modulo operator, which is not adapted to hardware implementations and caused timing violations at 100MHz. This process is repeated for each input neuron. When the last neuron is reached, $v_time$ is incremented and the process starts again from the first neuron until reaching the termination condition explained in Section 3.1.3.

### 3.1.2 Neural Processing Unit

The Neural Processing Unit (NPU) is in charge of emulating a single IF spiking neuron. There are two types of NPUs: one is specifically designed for the first layer, and one is generic for all the
other layers. In our model the membrane potential accumulator is not reset after a spike emission. The IF threshold $\theta_{IF}$ is rather subtracted from the accumulator.

### 3.1.2.1 Generic NPU

The generic NPU (G-NPU) architecture is made of three stages: the weight-affectation stage, the integration stage, and the accumulator & threshold stage. This architecture is represented in Figure 3.4 for an example neuron with 4 input synapses.

The weight affectation stage is composed of one multiplexer per input synapse: if a synapse is active (i.e. a spike is received though that synapse), the multiplexer forwards the corresponding synaptic weight and 0 otherwise. It should be noted that the synaptic weights are hard-coded in the G-NPU design. Then, the resulting set of weights and zeros are summed together in the integration stage. Summation is performed by a 2-by-2 pipelined adder-tree. In doing so, the architecture is able to process 1 spike per input synapse and per clock cycle. Register barriers after each stage ensure synchronization and limit the critical path of the design. The adder-tree is automatically generated thanks to hand-coded VHDL functions and packages.

The resulting sum is added to the membrane potential accumulator (orange register in Figure 3.4). If the result is above the neuron threshold ($\Theta_{IF}$), a spike is emitted and $\Theta_{IF}$ is subtracted from the membrane potential accumulator. Otherwise, no spike is emitted and the membrane potential remains unchanged. This process is pipelined in the same fashion than the rest of the architecture. Its implementation is described in Algorithm 2.

**Algorithm 2**

Pseudo-code of the PADS NPU accumulation & threshold stage process.

$potential\_spike(t) \leftarrow potential(t-1) + val(t) - \Theta_{IF}$

$potential\_nospike(t) \leftarrow potential(t-1) + val(t)$

if $potential(t-1) > \Theta_{IF}$ then

$o\_spike \leftarrow True$

$potential(t) \leftarrow potential\_spike(t-1)$

else

$o\_spike \leftarrow False$

$potential(t) \leftarrow potential\_nospike(t-1)$

In the pseudo-code, $Val(t)$ is the adder-tree output at clock-cycle t, and $potential(t)$ is the membrane potential at clock-cycle t. $potential\_spike(t)$ is represented by the red register on Figure 3.4, and $potential\_nospike(t)$ by the orange one.

### 3.1.2.2 First Layer NPU

The first layer NPU (FL-NPU) is based on the generic architecture presented above, but optimized for the first layer. As seen in 3.1.1, the GenCell is sequential and emits at most one spike per cycle. The Gencell also features an address signal which carries the address of the emitting neuron. Thus, the input parallelism featured in the generic NPU is useless in this context: only one synapse will be stimulated at a time. A simplified version is therefore designed to save hardware resources. As this layer is often the biggest in an MLP topology (highest number of neurons and synapses), this optimization might bring interesting resource savings (Section 3.2). The FL-NPU architecture is given in Figure 3.5. This architecture is made of two stages: the weight affectation stage and
<figure><img src="image_26.png" /><figcaption>Figure 3.4: Architecture of a generic NPU in PADS, represented with 4 input synapses. Register barriers are represented as rectangles.</figcaption></figure>

the accumulation & threshold stage. In the weight affectation stage, the weight is retrieved in a Look-Up-Table (LUT) using the address signal (*i_addr*). It should be noted that the weight LUTs are stored in Block RAM during synthesis, in contrast with the G-NPU in which the synaptic weights are hard-coded. The accumulation & threshold stage works exactly like in the generic NPU design.

### 3.1.3 Terminate Delta Module

The Terminate Delta Module (TDM) is in charge of the Terminate Delta procedure introduced in Section 2.3.4. The TDM counts output spikes on the last layer of the SNN, and determines when a neuron has spiked $\Delta$ times more than any other neuron. $\Delta$ is a user-defined parameter. In some cases, the Terminate Delta condition is never achieved, and the network runs indefinitely. This problem arises when there are not enough output spikes, or when the concurrence between several neurons is too fierce. To cope with this issue, the TDM also features a timeout mechanism which forces the network to stop after a certain number of clock cycles. The TDM implementation is pipelined like all the other modules, and is able to receive one spike per synapse at each clock cycle. The architecture is illustrated in Figure 3.6, where the timeout implementation and register barriers are not shown for the sake of clarity.

The TDM is made of 4 stages: a counter stage, two trees of max operators, and the delta stage. First, the counter stage counts the number of output spikes on-the-fly. The counts are stored in an array of registers indexed in the same order as the output neurons. In the second stage, a pipelined tree of maximum operators is used to find the index of the maximum value in the array. This value ($Max_1$) and its index ($ID\_Max_1$) are stored in registers. $Max_1$ is replaced by
<figure><img src="image_27.png" /><figcaption>Figure 3.5: Architecture of a PADS Input-NPU, optimized for the input FC layer</figcaption></figure>

a zero in the array, which is passed to the next maximum-tree stage. This second pipelined tree of maximum operators finds the maximum value ($Max_2$) in the new array. At this phase, the TDM has thus retrieved the maximum spike count ($Max_1$), the second maximum spike count ($Max_2$), and the index of the most spiking neuron ($ID\_Max_1$). Those three values are passed to the delta stage. The absolute value of $Max_1-Max_2$ is computed. If it is greater than $\Delta$, the Terminate Delta condition is reached and $ID\_Max_1$ is the winning class. Otherwise, the process continues until the condition is met or the timeout procedure forces the network to stop.

## 3.2 Hardware Synthesis Results

In this section, PADS is synthesized on FPGA. We analyze the hardware synthesis results in terms of logic resources usage, power usage, execution time and energy consumption. PADS will be compared to the VGT architecture, a parallel accelerator for Formal Neural Networks described in Section 2.2.2.1. The VGT architecture uses the same level of parallelism and implementation choices than PADS. Like in PADS, each neuron of the model is hard-coded in the design. VGT neurons have a similar structure made of three stages: A weight affectation stage (MAC operation), a tree of adders for input activity integration, and a ReLu stage for the activation function. The whole neuron is pipelined like PADS. This similarity enables a fair comparison. Some additional experiments on PADS are also presented in a second time. Those experiments aim at understanding the influence of various SNN-specific hyper-parameters on SNN acceleration. In the remaining of this document, we oppose static and dynamic processing. Dynamic refers to time-based computation, like in rate-based SNNs. On the other hand, FNNs are static models.
<figure><img src="image_28.png" /><figcaption>Figure 3.6: Architecture of the PADS’ Terminate Delta Module with 4 output neurons.</figcaption></figure>

### 3.2.1 Methodology

PADS and VGT architectures are applied to the OPS-SAT [93] classification task. A simple MLP topology is used. The Terminate Delta Module is configured with $\Delta = 4$. The GenCell is configured with $MaxPeriod = 10$ and $MinPeriod = 1$.

#### 3.2.1.1 Dataset Specifications

The grayscale OPS-SAT dataset is made of 28x28p grayscale patches extracted from 1920x1080p RGB satellites images taken by the OPS-SAT embedded camera (See Chapter 6.3.1.1 and Appendix 8.2.2). The original satellite images have variable light exposures, noise levels and orientations. The task is to classify each patch as "cloud" or "no-cloud". The dataset is made of 100 860 patches extracted from 16 full-size HD pictures. There are 40 078 cloud patches and 60 782 no-cloud patches. The dataset is split in two random batches, with 90% for training and 10% for testing.

#### 3.2.1.2 Training with TensorFlow Keras

A Multi-Layer Perceptron (MLP) with topology 28 × 28 - 100 - 2 is used for classification. The MLP is trained on this dataset using TensorFlow with Keras front-end. The training hyperparameters are listed in table 3.1. The synaptic weights are clipped between -1 and 1. This weight clipping is used before conversion towards spiking domain in order to balance weights and neuron threshold ($\Theta_{IF}$). This method is inspired from N2D2 [1], and is analogous to a weight-balancing technique [82]. In addition, a Softmax [94] activation function is used in the output layer during training in formal domain. The test score obtained in formal domain using TensorFlow is 71%. For test in spiking domain, the TF-trained weights are exported to N2D2 for spiking inference. The result spiking test score is also 71% with $\Delta = 4$. For comparison, the original formal CNN obtained 81% accuracy on RGB patches (instead than grayscale).
<table><caption>Table 3.1: TensorFlow Keras parameters used for training in all our experiments.</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Loss</td><td>Categorical Crossentropy</td></tr><tr><td>Optimizer</td><td>Adam</td></tr><tr><td># Epochs</td><td>50</td></tr><tr><td>Batch Size</td><td>128</td></tr><tr><td>Validation Splot</td><td>0.1</td></tr></tbody></table>

<figure><img src="image_29.png" /><figcaption>Figure 3.7: Workflow for synthesis, simulation and reporting using Vivado Design Suite (2019.1). The source code (VHDL) is depicted on the left, and the resulting report on the right.</figcaption></figure>

It should be noted that N2D2 has been widely used for SNN experiments during this PhD thesis as it was one of the only framework to offer both training and SNN conversion at the time work began. Moreover, N2D2 is developed by the CEA LIST which is part of the french technological research ecosystem.

### 3.2.1.3 Hardware Synthesis & Power Estimation

After learning, the trained synaptic weights are exported as VHDL packages for PADS and VGT architectures. PADS and VGT are both configured with the same 28 × 28 - 100 - 2 MLP topology. Both architectures are synthesized separately using Vivado Design Suite, the Xilinx framework for hardware synthesis, targeting the XCZU9EG family device of a ZCU102 board. The synthesis, simulation and reporting workflow is illustrated in Figure 3.7. A post-synthesis simulation is performed afterwards using the Vivado XSIM simulation tool. Vivado XSIM tool uses a custom VHDL testbench to provide input signals for the design. A specific testbench is developed for each architecture. For PADS the data is streamed by the testbench as required by the GenCell design. For VGT on the other hand, the pixels are injected all at once as specified in the VGT design. The aim of this realistic simulation is to generate a SAIF (Switching Activity Interchange Format) file. This file contains log of all signal switches occurring in the architecture during simulation. It is later used in the Vivado Power Analysis tool in order to estimate the power usage of the IP
<table><caption>Table 3.2: Hardware resources measures for PADS and VGT MLPs on OPS-SAT Grayscale Dataset: 28 × 28 - 100 - 2</caption><thead><tr><th>Module</th><th>GENCELL</th><th colspan="2">FC1</th><th colspan="2">FC2</th><th>T DELTA</th><th colspan="2">TOTAL</th></tr><tr><th>Architecture</th><th>PADS</th><th>PADS</th><th>VGT</th><th>PADS</th><th>VGT</th><th>PADS</th><th>PADS</th><th>VGT</th></tr></thead><tbody><tr><td>LUT (#)</td><td>9 944</td><td>7 662</td><td>940 659</td><td>1 574</td><td>1 776</td><td>138</td><td>19 318</td><td>942 435</td></tr><tr><td>FF (#)</td><td>283</td><td>2 798</td><td>671 168</td><td>2 065</td><td>1 772</td><td>85</td><td>5 231</td><td>672 940</td></tr><tr><td>BRAM (#)</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td></tr><tr><td>DSP (#)</td><td>0</td><td>0</td><td>755</td><td>0</td><td>40</td><td>0</td><td>0</td><td>795</td></tr><tr><td>DYNAMIC POWER (mW)</td><td>196</td><td>39</td><td>4 465</td><td>12</td><td>12</td><td>1</td><td>248</td><td>4 477</td></tr></tbody></table>

based on its simulated internal activity. The synthesis results for both VGT and PADS are given in Table 3.2.

### 3.2.2 Comparison with VGT

At first glance, Table 3.2 clearly shows that logic resources requirements are lower for PADS.

#### 3.2.2.1 DSP usage

First, VGT uses 795 DSPs, whereas PADS doesn't use any. This substantial gap is directly related to the fundamental difference between formal and spiking coding domain. As mentioned in Section 2.3.1, SNNs integrate information via simple ACC operations, whereas FNNs require MAC operations. During synthesis, Vivado maps each parallel MAC operation on a DSP (Digital Signal Processor). DSPs are specifically optimized for that purpose. On the other hand, ACC operations are directly mapped in the Programmable Logic. DSP being a scarce resource in FPGAs, PADS seems preferable in that regard.

#### 3.2.2.2 LUTs and Registers in FC layers

Second, VGT uses 49 times more Look-Up-Tables (LUTs) and 129 times more registers (also called Flip-Flops, FF). As shown in Table 3.2, this significant difference is mostly due to the first layer (FC1). For FC1 specifically, VGT uses 123 times more LUTs and 240 times more FFs than PADS. On the other hand the resource usage of the second layer (FC2) is similar in PADS and VGT. Indeed, there are two types of Neural Processing Units (NPU) in PADS: a FL-NPU optimized for the first layer, and a G-NPU for all the others. As explained in 3.1.2 the G-NPU has a large input parallelism which implies an adder-tree for input integration. Instead, the first layer NPU only has a single input and an addressing mechanism which only requires one adder. Consequently, the G-NPU is much more resource intensive.

#### 3.2.2.3 Power Usage

Third, VGT uses 18 times more power than PADS overall, and the overhead is mostly due to FC1, where VGT uses 114 times more power than PADS. This observation matches with the resource usage results. This is quite straightforward, as more resources implies more signals and thus, more signal toggles during execution. This in turns implies a higher power usage. Moreover, computation is sparse in PADS due to the rate-coding policy. This implies a low signal toggle rate
in the design. On the other hand, VGT works on static data, thus the signal toggle rate is much higher. This difference explains the power usage difference.

So, PADS provides interesting resource and power savings compared to VGT. The absence of DSPs in PADS design provide a better scalability, since this resource is scarce on FPGA circuits. The lower power usage could also benefit to low-power applications, such as in solar-powered systems.

### 3.2.3 Spike Generation Overhead

Table 3.2 highlights the resource overhead induced in PADS by the GenCell and the Terminate Delta Module (TDM). Indeed VGT does not require such specific modules, dedicated to spike encoding and decoding. Concerning the TDM, the overhead is not very significant: it is only accountable for 0.7% of LUTs and 1.6% of registers in PADS. On the other hand, the overhead caused by the GenCell is much noteworthy. It represents 51.4% of LUTs, and 6.5% of registers of the total design (GenCell+PADS+TDM). Thus, optimizing the GenCell is interesting to further reduce the hardware footprint of PADS.

#### 3.2.3.1 Execution Time & Energy Overhead

The resource overhead is not the only issue with this module. The sequential output allows using a hardware-friendly FL-NPU in the first layer. However, the drawback on execution time is very significant. PADS takes 8000 clock cycles in average for classification of OPS-SAT 28x28 grayscale patches, whereas VGT takes only 23. At 100MHz this is equivalent to 83µs for PADS and 230ns for VGT. PADS energy consumption is thus 2.1 × 10⁻⁵J per image in average, whereas it is only 6.5 × 10⁻⁷J in VGT. In other words, PADS is 32 times more energy intensive than VGT. Thus, PADS architecture does not comply with our initial intuition that spiking domain mitigates energy consumption. This observation is entirely due to the execution time overhead. It should be noted that this is not a strict disadvantage for PADS. The energy consumption is higher, but spread across extended period of times. Thus, PADS is compliant with low-power applications, such as solar-powered systems (satellites...).

#### 3.2.3.2 Causes of the timing Overhead

The difference in execution time is quite straightforward: VGT works in a static and synchronous fashion, where all inputs are presented at once for each layer. Oppositely, PADS works in a dynamic and asynchronous fashion because of the event-based nature of spiking domain. This mechanism intrinsically results in higher execution time, and is even more visible when using a sequential spike-encoding module like the GenCell. As mentioned earlier in Section 3.1.1, the GenCell scans the input vector entirely at each time-step, which results in timing overhead.

In the remaining of this subsection, we propose two experiments to investigate the behavior of the GenCell. More specifically, we study the influence of spatial and temporal resolution of spike encoding on the behavior of the GenCell. Temporal resolution refers to the number of periods available for spike encoding. Spatial resolution refers to the input size. The aim of those experiments is also to evaluate a potential trade-off between execution time and prediction accuracy.
### 3.2.3.3 Influence of Temporal Resolution

As a reminder, the rate-coding formula is given in Equation 3.1.

$$ 
\begin{aligned}
P &= \frac{1}{\text{MaxFreq} + (\text{MinFreq} - \text{MaxFreq}) \times v}, \\
v &= \frac{1 - i}{255}
\end{aligned} 
\qquad (3.1) 
$$

Where P is the spike train period associated to a pixel of value i (8-bit integer), MaxFreq and MinFreq are respectively the maximum and minimum spike train frequencies.

As mentioned in Section 3.1.1, the GenCell operates in a time-step based fashion rather than real-time. Spike train periods are thus expressed in terms of time-steps. According to Equation 3.1, spike trains are encoded with periods ranging from MaxPeriod = $\frac{1}{\text{MinFreq}}$ to MinPeriod = $\frac{1}{\text{MaxFreq}}$. The difference between MinPeriod and MaxPeriod characterizes the temporal resolution of spike encoding. There is thus a finite number of possible periods between MinPeriod and MaxPeriod. Those parameters must have a strong influence on execution time as they govern the spike emission rate. MinPeriod is set to 1, so that spike trains are able to begin at the first time-step. With a fixed MinPeriod = 1, MaxPeriod defines the temporal resolution by itself.

Under those conditions, Figure 3.8 shows the influence of MaxPeriod, i.e. temporal resolution. In this experiment PADS is configured with a small mono-layer 784-10 topology. The classifier is applied to MNIST dataset. TensorFlow Keras is used for training with the parameters listed in 3.1. The weights are exported in a VHDL package, and PADS is configured with the 784-10 topology. For each value of MaxPeriod, we perform behavioral simulation at Register Transfer Level (RTL) using Modelsim, on a hundred MNIST samples. We measure the average prediction accuracy, number of input & output spikes, and execution time with respect to MaxPeriod.

Figure 3.8 shows that high MaxPeriod values (i.e. high temporal resolutions) offers significantly better accuracy, lower execution times and lower number of input spikes. However, the number of output spikes is slightly lower for low temporal resolutions. Indeed, low temporal resolution implies that fewer periods are available for spike encoding. In other words, very different pixels might be encoded with identical spike trains. Thus, the network behavior is deteriorated which explains the heavily degraded accuracy (solid black line) at low MaxPeriod values, and the increased execution time. The execution time is higher because of the altered behavior: the Terminate Delta Module has more difficulty to determine the winning class under such conditions. Moreover, lower MaxPeriod value does not only imply lower resolution but higher input spike rates as shown in the Figure (red bars). As explained previously, power usage is directly related to signal toggles in the design. Higher spike rates results in higher signal toggle rates which in turns increase dynamic power usage.

Consequently, high MaxPeriod values (i.e. high temporal resolutions) can help mitigating execution time, and further reduce power usage. However, execution time remains very high with 9000 clock cycles at best. This execution time is even higher than with the 784-100-2 topology used for OPS-SAT grayscale patch classification (8000 clock cycles). Consequently, smaller topologies does not imply lower execution times. Fine tuning the temporal resolution does not enable to compete with VGT either, in terms of processing speed.
<figure><img src="image_30.png" /><figcaption>Figure 3.8: Evolution of test accuracy, number of input & output spikes and execution time (in clock cycles) with respect to MaxPeriod. All measurements are averaged on 100 samples. Topology: 784-10 on MNIST.</figcaption></figure>

### 3.2.3.4 Influence of Spatial Resolution

In this paragraph we study the influence of spatial resolution on the GenCell behavior and subsequent execution time. Because of the sequential nature of the GenCell, the time-step length depends on the input size. The process requires to scan the whole "Next Spiking Time" vector (NST-LUT) (Section 3.1.1) at each time-step at a rate of one element per clock cycle. For example, on MNIST dataset, the time-step length is 784 clock cycles. Quite straightforwardly the execution time could be mitigated by reducing the input size. In other words, input down-sampling could provide interesting speed-ups at the cost of a loss of information.

For that purpose, we study the influence of input down-sampling on PADS behavior. A single FC layer SNN is used with $I_{\text{size}}$ inputs and 10 outputs. $I_{\text{size}}$ is the input size varying from 28 to 10. A specific network is trained for each input size using Tensorflow Keras with the parameters given in 3.1. A *Resize* layer from Keras front-end is used at the input. This layer down-samples the input data to the required $I_{\text{size}}$ value. Moreover, the resized samples are also exported in VHDL testbenches for later simulation. Each set of trained weights is exported in a dedicated VHDL package and PADS is configured successively with each topology. Each iteration is simulated at RTL level using Modelsim, using the previously exported down-sampled data. Each simulation covers a hundred of such samples. We measure the variations of average accuracy, number of input & output spikes and execution time with respect to input size. The results of this experiment are shown in Figure 3.9. It should be noted that we use MinPeriod = 1 and MaxPeriod=100.

The MLP topology changes with input size, thus the network is retrained each time. Thus, variations of testing accuracy are not only caused by spatial resolution. Intrinsic variability of training also plays a role. Therefore, the difference between formal and spiking accuracy should be studied, rather than spiking test accuracy only. In doing so, the training variability is mitigated. This difference is referred to as "accuracy gap". The accuracy gap increases towards low input
<figure><img src="image_31.png" /><figcaption>Figure 3.9: Evolution of test accuracy, number of input & output spikes and execution time (in clock cycles) with respect to input size. All measurements are averaged on 100 samples. Single FC layer SNN on MNIST dataset with various input sizes.</figcaption></figure>

sizes (i.e. low spatial resolutions). Accordingly, the converted SNN supported by PADS seems less robust to down-sampling than the original FNN tested in TensorFlow.

Moreover, smaller input sized means fewer neurons in the input layer. Fewer neurons in turns generate fewer spikes thus the number of input spike per image (red bars) increases with respect to input size. On the other hand, the number of output spike per image is higher for low input sizes. Thus, at low spatial resolutions, fewer input spikes generates more output spikes than at high spatial resolutions. This phenomenon confirms the degradation of the network’s behavior for higher down-sampling rates. As in the previous experiment (temporal resolution), loss of information also increases execution time. Consequently, reducing spatial resolution through down-sampling of input data does not mitigate latency, and heavily degrades accuracy. Thus, down-sampling is not viable to reduce latency in PADS. This experiment also shows that PADS is more affected by spatial down-sampling than the original TensorFlow model.

### 3.2.4 Conclusions on PADS hardware implementation

In this section, PADS is compared to VGT on OPS-SAT grayscale dataset with a 784 – 100 – 2 MLP. Under such conditions, PADS enables drastic logic resources savings relatively to VGT. The most significant difference lies in DSP usage. If VGT uses a lot of DSPs for MAC operations, PADS uses only LUTs and Flip-Flops for ACC operations. A major difference was also found in the first layer, where the specific NPU enabled drastic resources savings in terms of LUTs and
Registers. At the same time, PADS provides substantial power savings relatively to VGT with 248mW and 4477mW respectively (18 times less).

However, the dynamic nature of information and the sequential behavior of the GenCell are responsible for the timing overhead in PADS. Indeed, PADS is in average 30 times slower than VGT on the OPS-SAT grayscale dataset. Despite power savings, it consumes in average $2.1 \times 10^{-5}$J per image, against $6.5 \times 10^{-7}$J for VGT. PADS uses more energy, but spread across a longer processing time. Fine-tuning the temporal resolution of rate encoding mitigates the GenCell latency. However, VGT still remains several dozens of times faster than PADS, even with optimal temporal resolution. On the other hand, down-sampling the input data does not mitigate latency: the loss of spatial information seems to make the classification task harder, increasing the execution time. Additionally, one simple way to solve the temporal sparsity of spike encoding could be a parallelization of the GenCell process. For this purpose, a fully-parallel version of the GenCell was developed with simplified hardware implementation. After preliminary experiments on MNIST, the parallel GenCell saturates the Zedboard resources on its own. Therefore, this approach is unrealistic. On the other hand, proposing an intermediate level of parallelism would have required too much development for the purpose of our prototype. Moreover, this feature would have been incompatible with the NPU design of PADS.

In the light of these considerations, a first cartography of favorable application domains for PADS arises. PADS is more suited to systems constrained by the resource (e.g. drones, satellites..) or power budget (e.g. solar or wind powered systems), whereas VGT is more suited to systems constrained by energy (e.g. battery-powered systems) or time (e.g. real-time applications).

Because PADS is more resource and power efficient, it is preferable to use VGT in the context of highly constrained solar-powered embedded systems, such as Satellites. Indeed, size, weight and power are the most limiting factors, rather than overall energy consumption. The logic resource and dynamic power savings implies that it might be used in more constrained systems than VGT would which are limited by instantaneous power rather than overall energy consumption. This is also true for other types of solar powered devices. On the other hand, PADS is not adapted to battery-powered systems, as energy becomes the limiting factor in such conditions. PADS seems less adapted to real-time systems either because of its increased execution time. At this point it should be noted that all presented results are constrained to rate-coded MLP deployed on highly parallel hardware. Moreover, they are constrained to OPS-SAT classification task. In Chapter 5, those results are generalized to low-level of parallelism and a benchmark of 7 datasets (images and 1D signals).

## 3.3 Conclusion

In this Chapter, we have presented the PADS architecture. PADS is a highly-parallel Spiking MLP architecture featuring a Spike Generation Cell, two variants of Neural Processing Units, and a Terminate Delta Module. PADS was developed as a first prototype of SNN hardware accelerator in the thesis, serving as a toolbox to investigate such implementations and support later improvements. PADS was compared to an equivalent highly-parallel formal architecture: VGT. During this comparison, PADS demonstrated drastic logic resources and power savings compared to VGT. However, the dynamic nature of rate-coding and the sequential implementation of the Spike Generation Cell were found responsible for a severe increase in execution time. Consequently, PADS energy-intensiveness was a lot greater than that of VGT. However, in PADS, this energy consumption is spread across time.
This observation led us to our firsts insights concerning the cartography between applications and coding domain. PADS seems well suited to highly resource and power constrained systems, such as satellites. Indeed, such systems have drastic constraints in terms of size and weight. Saving resources is thus helpful in that regard. Moreover, solar-powered devices like satellites are limited in terms of instantaneous power usage rather than overall energy consumption. VGT does not comply with such constraints because of the high instantaneous power usage when PADS seems well suited to this type of systems. On the other hand, VGT is much more adapted for battery-powered systems, where overall energy consumption is the limiting factor. Moreover, PADS seems less adapted to real-time systems because of its long response time.

However, those insights are only true for very specific cases. Only two architectures where compared, which does not covers other implementation choices, such as sequential architectures. Moreover, this comparison only covered a single MLP topology, and a single application (OPS-SAT grayscale patch classification). Additionally, this comparison only stands for conversion-based and rate-based SNN models. In Chapter 7, we address a novel type of SNN models. Since rate-based converted SNNs are responsible for a significant inference time overhead, we have indeed looked for alternative choice in the literature. Moreover, our experiment concerning the spatial resolution influence on PADS did not provide the expected results. Indeed, reducing spatial resolution was intended to reduce execution time, but the opposite phenomenon arose. This finding was imputed to loss-of-information. Consequently, the idea of Neural Network Hybridization emerged. It will be introduced in details in Section 6.

In addition to this work, a SoC design was developed to support PADS hardware deployment. The role of this platform is to support hardware deployment of PADS and other hardware acceler- ator on FPGA. In doing so, it provides a useful testbench to test the architecture in real conditions and probe the actual power consumption of the design. The latter was not undertaken during this thesis, due to the lack of material and knowledge at that time. However, the SoC will enable such study in further works. The design is described in details in Appendix 8.2.2. For information, it features a Petalinux OS running on the Cortex ARM A-53 of the board to control and send data to the accelerator in the programmable logic. The SoC features all necessary interfaces to enable direct communication between the Linux user-space and the accelerator (kernel drivers, AXI interface...).
# Chapter 4

## Synaptic Activity Ratio & Energy Modeling

### Chapter contents

<table><tr><td>4.1 Representative Datasets</td><td>53</td></tr><tr><td>4.1.1 Used topologies</td><td>54</td></tr><tr><td>4.2 Accuracy and synaptic activity measurements</td><td>55</td></tr><tr><td>4.2.1 Methods</td><td>56</td></tr><tr><td>4.2.2 Synaptic activity results</td><td>58</td></tr><tr><td>4.2.3 Discussions on Synaptic Activity results</td><td>62</td></tr><tr><td>4.3 Synaptic Activity ratio</td><td>63</td></tr><tr><td>4.3.1 Energy Consumption Model</td><td>63</td></tr><tr><td>4.3.2 The value of λ</td><td>65</td></tr><tr><td>4.4 Synaptic Activity Ratio measurements</td><td>67</td></tr><tr><td>4.4.1 Synaptic Activity Ratio evaluation software</td><td>67</td></tr><tr><td>4.4.2 Network-wise SAR & theoretical cartography</td><td>69</td></tr><tr><td>4.4.3 Data type and rate-coding</td><td>73</td></tr><tr><td>4.4.4 Layer-wise SAR & hybridization</td><td>73</td></tr><tr><td>4.5 Conclusion</td><td>74</td></tr></table>
This chapter is dedicated to the study of Synaptic Activity in Spiking Neural Networks. Synaptic Activity refers to events occurring in synapses, i.e. spikes. As explained earlier in Section 2.4.3.2, Formal Neural Networks (FNN) work in a synchronous fashion with one and only one activation per synapse. On the other hand, the dynamic behavior of SNNs imply an unpredictable number of spikes per synapse and per sample.

The effective number of spikes is crucial when addressing SNN hardware implementations, as it could imply a computation overhead. The computation overhead could counter-balance the advantage of Accumulation (ACC) over Multiplication-Accumulation (MAC) (Section 2.3.1). Hence, the energy savings brought by spiking domain are thus not guaranteed. In this chapter, we study how synaptic activity varies under the influence of dataset, topology and various SNN-specific hyper parameters.

The raw spike count is relevant to compare computational efficiency in different SNN applications or topologies, but it does not help in the comparison with FNNs. Therefore, we propose the Synaptic Activity Ratio (SAR) metric: the ratio between $N_{spikes}$ and $N_{synapse}$. This metric is inspired from the literature (see Section 2.4.3.2) and is used to assess the computational overhead induced by execution in spiking domain. In doing so, we provide a high-level energy consumption model, based on the relative computation requirements of FNNs and SNNs. The model assimilates the processing of a neuromorphic accelerator to that of its synaptic operations: Multiplication-Accumulation (MAC) for FNNs and Accumulation (ACC) for SNNs. Indeed, Kundu et. al. [78] have already shown that computation in neural network accelerators is largely dominated by synaptic operations.

The raw synaptic activity measurements and subsequent SAR results are applied to a benchmark of Machine Learning datasets. Those datasets cover different types of data, with various spatial dimensions and complexities. This study aims in providing a high-level cartography of applications and coding domains.

## 4.1 Representative Datasets

The seven classification datasets of the benchmark address two types of data: Spatial data and 1D vectors. Each type of data can be mono or multi-channel. There are three datasets for each type of data, seven in total. For each type, there are three datasets with increasing complexity. In this work, state-of-the-art classification accuracy is used as an indicator of complexities. Spatial and vector data are used to evaluate the impact of data type on synaptic activity and SAR.

Image classification datasets are:

1. OPS-SAT RGB [93]: Classification of patches (28x28x3) extracted satellite images in two classes: cloud and no-cloud. The state-of-the-art accuracy is 81% [93]. This task is part of a larger cloud segmentation application. Figure 4.1 features an example full-size satellite of OPS-SAT dataset on the left. Additionally, the segmentation mask resulting from patch classification is shown on the left.

2. MNIST [95]: Handwritten digits classification task, the most basic and used benchmark application in the SNN community. It is made of ten classes of 28x28 grayscale images. The state-of-the-art accuracy is 99.84% [96]. Some samples of MNIST are shown in Figure 4.2a.

3. GTSRB [97]: German traffic signs classification task, a more complex image classification set with 43 classes of 32x32 RGB images. Its state-of-the-art accuracy is 99.71% [98]. Figure 4.2b presents some examples of GTSRB images.
<figure><img src="image_32.png" /><figcaption>Figure 4.1: Illustration of the CIAR cloud segmentation task. On the left, the original image taken by OPS-SAT. On the right, the resulting segmentation map, with cloudy patches in yellow. Source: [15]</figcaption></figure>

4. CIFAR-10 [99]: common objects classification task, an even more complex image classification task, commonly used in NN community, with 10 classes of 32x32 RGB images. Its state-of-the-art accuracy is 99.70% [100]. A selection of CIFAR-10 samples are shown in Figure 4.3a.

1D data classification datasets are:

1. Mines versus Rocks [16]: classification of sonar echoes from rocks and military mines in the ground. It is made of two classes of 60-elements vectors, thus it is quite a simple and small classification task. The state-of-the-art accuracy on this task is 90.4% [16]. The task is yet very simple: the input resolution is very low, and there is only two output classes. The low accuracy is mostly due to the low amount of samples in the dataset. Figure 4.3b shows two sonar echoes examples for rocks and mines.

2. Spoken Digits [101]: this dataset is made of the ten groups of 0 to 9 audio digits, extracted from the Google Speech Commands [17] dataset. A Mel Frequency Cepstral Coefficients transform (MFCC) with 39 × 13 dimensional features was applied to those audio recordings. When this chapter was written (September 2021), no publication mentioning classification accuracy on this dataset was found. However, an accuracy of 97.34% is achieved using the topology given in Table 4.1. Some examples of spectrograms from Google Speech Commands dataset are given in Figure 4.4,

3. Deepsig RadioML 2018 [18]: A dataset which includes both synthetic simulated signals and over-the-air recordings of 24 digital and analog modulation types of RF signals. An illustration of the RF recording experimental setup is given in Figure 4.5. The dataset is made of 24 classes of 1024-elements vectors with 2 channels, and it represents a difficult classification task. Its state-of-the-art classification accuracy is 64% [102].

### 4.1.1 Used topologies

Each set of training and testing of formal CNNs takes important amount of time (especially with the selected AI framework, i.e. N2D2), so using state-of-the-art deep neural networks would have
<figure><img src="image_33.png" /><figcaption>Figure 4.2: a) Sample of MNIST dataset, b) Sample of GTSRB dataset</figcaption></figure>

required an unmanageable simulation time. In addition, inference in spike-domain also takes a large amount of time (that is proportional to the complexity and depth of the selected topology). Therefore, we could not use state-of-the-art CNNs to perform this study. However, we made an exploration of topologies and hyper-parameters to provide the best results possible in the limits of our computational power and available time. Instead, a custom CNN was used for each dataset (topology description is available in Tables 4.2). With those topologies, the goal is not to achieve state-of-the-art accuracy, but rather to compare SNNs and FNNs in terms of accuracy and Synaptic Activity Ratio.

## 4.2 Accuracy and synaptic activity measurements

In this section, we study the impact of dataset and CNN topology on synaptic activity and network accuracy. Additionally, we study the impact of two SNN-specific hyper-parameters: $\Delta$ and $\Theta_{IF}$.

4.2.0.0.1 **Studying $\Delta$ influence** Indeed, as explained in Section 2.3.4, this parameter has a strong impact on the behavior of the network: increasing $\Delta$ means that the Terminate Delta Module will wait for a larger difference between the most active neuron and the second most active. In other words, the Terminate Delta Module (TDM, Section 3.1.3) will wait for more output spikes. Thus the total number of spikes flowing in the network will also increase. On the other hand, it is usually admitted that a higher $\Delta$ will have a positive impact on prediction accuracy [64]. Thus, one of our concerns is to study the possible trade-off between accuracy and synaptic activity.

4.2.0.0.2 **Studying $\Theta_{IF}$ influence** A higher firing threshold in Integrate & Fire (IF) ($\Theta_{IF}$) will cause fewer spikes to pass to the next layer, impacting the overall synaptic activity. However,
<figure><img src="image_34.png" /><figcaption>Figure 4.3: a) Sample of CIFAR 10 dataset, b) Sonar echoes from Mines VS Rocks dataset [16]</figcaption></figure>

<figure><img src="image_35.png" /><figcaption>Figure 4.4: Examples of spectrograms from Google Speech Commands dataset [17]. Spoken Digits dataset use in this work is a subset of Google Speech Commands.</figcaption></figure>

it is difficult to predict how the network will react to such a change in ΘIF: the synaptic activity could decrease because fewer spikes are allowed to pass, but the Terminate Delta Procedure could in turn last longer and thus allow more spikes to flow in the network, resulting in an equal number of spikes overall. Hence, studying the influence of this parameter is also one of our concerns. The impact of ΘIF will be assessed on CIFAR-10 only, because its application to all datasets would require unrealistic amount of simulation time.

4.2.1 Methods

A few definitions are provided in the following before going into further details:

**task:** the combination formed by a dataset and its associated CNN,

**experiment:** the ensemble formed by the task and the set of hyper-parameters used for training and testing,

**trial:** is the realization of an experiment (usually, 10 trials are executed for each experiment in order to obtain averaged and statistically representative results).
<figure><img src="image_36.png" /><figcaption>Figure 4.5: Illustration of the over-the-air RadioML 2018 recording setup, found in [18]. A host computer generates the desired RF signal, which is emitted and received via two universal software radio peripherals (USRP).</figcaption></figure>

Each trial is made using N2D2 [1], thus the accuracy and synaptic activity measurements are software simulation results. However, the resulting hardware architecture configured with the trained weights and identical hyper-parameters should reproduce the behavior of N2D2. Consequently, we are confident that the synaptic activity measured in N2D2 will be the same in a subsequent hardware implementation using PADS (Chapter 3) or SPLEAT [14] (Section 2.4.2.8).

### 4.2.1.0.1 Default hyper-parameters
For each trial, N2D2 is configured with the required dataset and its corresponding CNN topology. The hyper-parameters are set to selected values:

* There are no on-the-fly pre-processing on input data;

* *MinPeriod* is set to 1;

* *MaxPeriod* is set to 100;

* if not mentioned otherwise, the default value of Δ is 4;

* the default Integrate & Fire threshold $\Theta_{IF}$ value is 1.

### 4.2.1.0.2 Experimental method
For each trial:

* the dataset is split with 80% of samples for training, 10% for validation and 10% for testing;
<table><caption>Table 4.1: Large CNN topology for Spoken Digits dataset. This CNN achieves 97.34% accuracy.</caption><thead><tr><th>Layer</th><th>Type</th><th># Kernels</th><th>Kernel size</th><th>Stride</th><th>Output Shape</th></tr></thead><tbody><tr><td>0</td><td>Input</td><td></td><td></td><td></td><td>(507,1)</td></tr><tr><td>1</td><td>Conv</td><td>32</td><td>5</td><td>1</td><td>(503,3)</td></tr><tr><td>2</td><td>MaxPool</td><td>3</td><td>2</td><td>2</td><td>(252,3)</td></tr><tr><td>3</td><td>Conv</td><td>64</td><td>5</td><td>1</td><td>(248,5)</td></tr><tr><td>4</td><td>MaxPool</td><td>5</td><td>2</td><td>2</td><td>(124,5)</td></tr><tr><td>6</td><td>Conv</td><td>128</td><td>5</td><td>1</td><td>(120,5)</td></tr><tr><td>7</td><td>MaxPool</td><td>5</td><td>2</td><td>2</td><td>(60,5)</td></tr><tr><td>8</td><td>FC</td><td></td><td></td><td></td><td>250</td></tr><tr><td>9</td><td>FC</td><td></td><td></td><td></td><td>100</td></tr><tr><td>10</td><td>FC</td><td></td><td></td><td></td><td>10</td></tr></tbody></table>

* training is performed until the "learning plateau" is reached, where the network accuracy reaches a stable value and is not increased by further training epochs;

* the trained network is first tested in formal domain, providing a baseline test accuracy; then, the network is transcoded and tested in spiking domain, using N2D2 as well;

* accuracy and synaptic activity are logged during the test (e.g. accuracy of learning and testing, for each of formal and spiking domain);

* Each experiment is averaged on 10 runs.

### 4.2.2 Synaptic activity results

The experimental protocol is applied to each of the 6 tasks of our benchmark. The results obtained after averaging all measurements on 10 trials are presented in Table 4.3. The synaptic activity is expressed as the number of output spikes of that layer, also called spike count.

#### 4.2.2.0.1 Influence of $\Delta$: When analyzing those results, we observe that the spiking accuracy is always lower than formal accuracy. In other word, transcoding towards spiking domain implies a loss of accuracy. However, this degradation seems to be mitigated by increasing $\Delta$.

Let us imagine a situation where a pixel is encoded in a spike-train with Period = 100. If the Terminate Delta condition is reached before the 100th time-step, the prediction would be enacted before any spike is emitted by this pixel. Thus, the network would have stopped with incomplete information. Increasing $\Delta$ implies to wait for more output spikes. In that regard, increasing $\Delta$ should result in better accuracy and higher spike counts.

Our results confirms this statement. For example, in the GTSRB task, the $\Delta$ = 20 experiment provides an accuracy of 93.5% against 83.3% with $\Delta$ = 5. Moreover, the average total spike count is 2.1 times higher with $\Delta$ = 20. This trend is found in all other experiments. Thus, tuning $\Delta$ enables to optimize the trade-off between accuracy and synaptic activity.

#### 4.2.2.0.2 Influence of $\Theta_{IF}$: We also study the influence of $\Theta_{IF}$ on accuracy and synaptic activity with CIFAR-10 task. The results (Table 4.3) show that, for a constant value of ($\Delta$), higher values $\Theta_{IF}$ provide significantly better accuracies. For $\Delta$ = 5, the accuracy is 11.45% with
<table><caption>Table 4.2: List of CNN topologies for the classification benchmark</caption><thead><tr><th>Layer</th><th>Type</th><th># Kernels</th><th>Kernel size</th><th>Stride</th><th>Output Shape</th></tr></thead><tbody><tr><td>0</td><td>Input</td><td></td><td></td><td></td><td>(28,28,3)</td></tr><tr><td>1</td><td>Conv</td><td>3</td><td>5</td><td>1</td><td>(24,24,3)</td></tr><tr><td>2</td><td>MaxPool</td><td>3</td><td>2</td><td>2</td><td>(12,12,3)</td></tr><tr><td>3</td><td>Conv</td><td>5</td><td>5</td><td>1</td><td>(8,8,5)</td></tr><tr><td>4</td><td>MaxPool</td><td>5</td><td>2</td><td>2</td><td>(4,4,5)</td></tr><tr><td>5</td><td>FC</td><td></td><td></td><td></td><td>10</td></tr><tr><td>6</td><td>FC</td><td></td><td></td><td></td><td>2</td></tr></tbody></table><table><thead><tr><th>Layer</th><th>Type</th><th># Kernels</th><th>Kernel size</th><th>Stride</th><th>Output Shape</th></tr></thead><tbody><tr><td>0</td><td>Input</td><td></td><td></td><td></td><td>(32,32,3)</td></tr><tr><td>1</td><td>Conv</td><td>32</td><td>5</td><td>1</td><td>(28,28,32)</td></tr><tr><td>2</td><td>MaxPool</td><td>32</td><td>2</td><td>2</td><td>(14,14,32)</td></tr><tr><td>3</td><td>Conv</td><td>32</td><td>5</td><td>1</td><td>(10,10,32)</td></tr><tr><td>4</td><td>MaxPool</td><td>32</td><td>2</td><td>2</td><td>(5,5,32)</td></tr><tr><td>6</td><td>FC</td><td></td><td></td><td></td><td>120</td></tr><tr><td>7</td><td>FC</td><td></td><td></td><td></td><td>43</td></tr><tr><td>8</td><td>FC</td><td></td><td></td><td></td><td>10</td></tr></tbody></table><table><thead><tr><th>Layer</th><th>Type</th><th># Kernels</th><th>Kernel size</th><th>Stride</th><th>Output Shape</th></tr></thead><tbody><tr><td>0</td><td>Input</td><td></td><td></td><td></td><td>(60,1)</td></tr><tr><td>1</td><td>FC</td><td></td><td></td><td></td><td>200</td></tr><tr><td>2</td><td>FC</td><td></td><td></td><td></td><td>2</td></tr></tbody></table><table><thead><tr><th>Layer</th><th>Type</th><th># Kernels</th><th>Kernel size</th><th>Stride</th><th>Output Shape</th></tr></thead><tbody><tr><td>0</td><td>Input</td><td></td><td></td><td></td><td>(1024,2)</td></tr><tr><td>1</td><td>Conv</td><td>32</td><td>5</td><td>1</td><td>(1020,32)</td></tr><tr><td>2</td><td>MaxPool</td><td>32</td><td>2</td><td>2</td><td>(510,32)</td></tr><tr><td>3</td><td>Conv</td><td>64</td><td>5</td><td>1</td><td>(506,64)</td></tr><tr><td>4</td><td>MaxPool</td><td>64</td><td>2</td><td>2</td><td>(253,64)</td></tr><tr><td>6</td><td>Conv</td><td>128</td><td>5</td><td>1</td><td>(249,128)</td></tr><tr><td>7</td><td>MaxPool</td><td>128</td><td>2</td><td>2</td><td>(124,128)</td></tr><tr><td>8</td><td>FC</td><td></td><td></td><td></td><td>256</td></tr><tr><td>9</td><td>FC</td><td></td><td></td><td></td><td>125</td></tr><tr><td>10</td><td>FC</td><td></td><td></td><td></td><td>24</td></tr></tbody></table>

$\Theta_{IF} = 0.5$, 31.8% with $\Theta_{IF} = 1$ and 57.8% with $\Theta_{IF} = 2$. As there are 10 classes, the accuracy with $\Theta_{IF=0.5}$ and $\Delta = 5$ is close to randomness.

However, high values of $\Theta_{IF}$ implies higher synaptic activity. For $\Delta = 5$, the total spike count is 1375 with $\Theta_{IF} = 0.5$, 32967 with $\Theta_{IF} = 1$ and 313019 with $\Theta_{IF} = 2$. Those trends are also visible with $\Delta = 10$ and $\Delta = 20$. Thus, $\Theta_{IF}$ also enables a trade-off between accuracy and synaptic activity.

However, this behavior is quite surprising. Indeed, lowering $\Theta_{IF}$ in a layer means easing output spike firing. Thus, a synaptic activity overhead could be expected for low values of $\Theta_{IF}$. Our hypothesis is that spikes indeed flow more easily through the network with $\Theta_{IF=0.5}$, including a large number of “undesired” spikes reaching the output. This phenomenon is illustrated in Figure 4.6. This figure shows the spike filtering rate for all 9 experiments on CIFAR-10 task. The filtering rate is the ratio between the numbers of input spikes and output spike as given by Equation 4.1.
<table><caption>Table 4.3: Accuracy and synaptic activity measurements with N2D2 on all datasets of the benchmark</caption><thead><tr><th>Task</th><th>FNN<br>ACC. (%)</th><th>Δ</th><th>SNN<br>ACC. (%)</th><th>Layer 0<br>(# spikes)</th><th>Layer 1<br>(# spikes)</th><th>Layer 2<br>(# spikes)</th><th>Layer 3<br>(# spikes)</th><th>Layer 4<br>(# spikes)</th><th>Layer 5<br>(# spikes)</th><th>Layer 6<br>(# spikes)</th><th>Layer 7<br>(# spikes)</th><th>Layer 8<br>(# spikes)</th><th>Layer 9<br>(# spikes)</th><th>Total<br>(# spikes)</th></tr></thead><tbody><tr><td rowspan="3">OPS-SAT<br>(Std. dev.)</td><td rowspan="3"><b>91.95</b><br>(0.37)</td><td>5</td><td><b>91.78</b><br>(0.52)</td><td>200<br>(84)</td><td>216<br>(60)</td><td>104<br>(25)</td><td>79<br>(20)</td><td>37<br>(6)</td><td>8<br>(1)</td><td>4<br>(0)</td><td></td><td></td><td></td><td>648<br>(196)</td></tr><tr><td>10</td><td><b>91.88</b><br>(0.83)</td><td>683<br>(252)</td><td>561<br>(116)</td><td>243<br>(38)</td><td>228<br>(36)</td><td>85<br>(9)</td><td>18<br>(2)</td><td>10<br>(0)</td><td></td><td></td><td></td><td>1828<br>(453)</td></tr><tr><td>20</td><td><b>91.92</b><br>(1.21)</td><td>9 117<br>(14 351)</td><td>1 057<br>(477)</td><td>394<br>(136)</td><td>379<br>(141)</td><td>114<br>(32)</td><td>26<br>(11)</td><td>13<br>(3)</td><td></td><td></td><td></td><td>11 100<br>(15 151)</td></tr><tr><td rowspan="3">MNIST<br>(Std. dev.)</td><td rowspan="3"><b>99.2</b><br>(0.2)</td><td>5</td><td><b>97.3</b><br>(0.6)</td><td>128<br>(12)</td><td>870<br>(56)</td><td>391<br>(23)</td><td>481<br>(38)</td><td>269<br>(19)</td><td>65<br>(5)</td><td>38<br>(3)</td><td></td><td></td><td></td><td>2242<br>(156)</td></tr><tr><td>10</td><td><b>98.9</b><br>(0.5)</td><td>263<br>(16)</td><td>2 525<br>(106)</td><td>1 028<br>(34)</td><td>895<br>(94)</td><td>501<br>(45)</td><td>80<br>(4)</td><td>55<br>(2)</td><td></td><td></td><td></td><td>5 347<br>(301)</td></tr><tr><td>20</td><td><b>99.1</b><br>(0.6)</td><td>501<br>(22)</td><td>3 921<br>(188)</td><td>1 570<br>(60)</td><td>1 388<br>(144)</td><td>768<br>(69)</td><td>141<br>(5)</td><td>98<br>(3)</td><td></td><td></td><td></td><td>8 387<br>(491)</td></tr><tr><td rowspan="3">GTSRB<br>(Std. dev.)</td><td rowspan="3"><b>95.5</b><br>(0.8)</td><td>5</td><td><b>83.3</b><br>(5.1)</td><td>4 083<br>(549)</td><td>8 195<br>(1 373)</td><td>3 052<br>(503)</td><td>2 054<br>(448)</td><td>1 231<br>(243)</td><td>145<br>(19)</td><td>136<br>(14)</td><td>175<br>(21)</td><td></td><td></td><td>19 071<br>(3 170)</td></tr><tr><td>10</td><td><b>90.6</b><br>(1.8)</td><td>6 285<br>(937)</td><td>10 852<br>(2 583)</td><td>3 979<br>(927)</td><td>3 260<br>(690)</td><td>1 936<br>(391)</td><td>259<br>(53)</td><td>272<br>(42)</td><td>366<br>(58)</td><td></td><td></td><td>27 209<br>(5 681)</td></tr><tr><td>20</td><td><b>93.5</b><br>(1.4)</td><td>8 947<br>(1 239)</td><td>15 888<br>(3 167)</td><td>5 779<br>(1 112)</td><td>4 574<br>(948)</td><td>2 738<br>(547)</td><td>435<br>(76)</td><td>481<br>(64)</td><td>646<br>(91)</td><td></td><td></td><td>39 488<br>(7 244)</td></tr><tr><td rowspan="10">CIFAR-10<br>(Std. dev.)</td><td rowspan="10"><b>69.8</b><br>(1.1)</td><td>5<br>Θ<sub>IF</sub> = 1</td><td><b>31.8</b><br>(6.1)</td><td>4 772<br>(953)</td><td>16 167<br>(2 281)</td><td>6 275<br>(803)</td><td>3 395<br>(519)</td><td>2 129<br>(254)</td><td>87<br>(9)</td><td>82<br>(8)</td><td></td><td></td><td></td><td>32 967<br>(4 837)</td></tr><tr><td>10<br>Θ<sub>IF</sub> = 1</td><td><b>50.2</b><br>(2.7)</td><td>10 655<br>(591)</td><td>30 490<br>(2 482)</td><td>11 446<br>(846)</td><td>7 169<br>(775)</td><td>4 479<br>(414)</td><td>220<br>(9)</td><td>201<br>(8)</td><td></td><td></td><td></td><td>64 660<br>(5 125)</td></tr><tr><td>20<br>Θ<sub>IF</sub> = 1</td><td><b>57.8</b><br>(1.9)</td><td>22 919<br>(771)</td><td>58 790<br>(4 786)</td><td>22 091<br>(1 591)</td><td>15 943<br>(1 668)</td><td>10 074<br>(932)</td><td>531<br>(21)</td><td>482<br>(20)</td><td></td><td></td><td></td><td>130 830<br>(9 789)</td></tr><tr><td>5<br>Θ<sub>IF</sub> = 2</td><td><b>59.2</b><br>(0.9)</td><td>56 165<br>(1 592)</td><td>65 231<br>(6 567)</td><td>24 978<br>(2 134)</td><td>9 298<br>(1 244)</td><td>6 005<br>(667)</td><td>161<br>(12)</td><td>60<br>(3)</td><td></td><td></td><td></td><td>161 898<br>(12 409)</td></tr><tr><td>10<br>Θ<sub>IF</sub> = 2</td><td><b>59.2</b><br>(0.9)</td><td>83 542<br>(1 740)</td><td>97 492<br>(10 141)</td><td>37 323<br>(3 213)</td><td>141 341<br>(1 881)</td><td>9 225<br>(1 006)</td><td>254<br>(19)</td><td>98<br>(4)</td><td></td><td></td><td></td><td>242 275<br>(18 004)</td></tr><tr><td>20<br>Θ<sub>IF</sub> = 2</td><td><b>59.2</b><br>(0.9)</td><td>110 520<br>(1 522)</td><td>120 233<br>(13 565)</td><td>49 826<br>(4 322)</td><td>19 464<br>(2 506)</td><td>12 484<br>(1 341)</td><td>352<br>(37)</td><td>140<br>(7)</td><td></td><td></td><td></td><td>313 019</td></tr><tr><td>5<br>Θ<sub>IF</sub> = 0.5</td><td><b>11.45</b><br>(0.002)</td><td>5<br>(1)</td><td>104<br>(14)</td><td>64<br>(8)</td><td>603<br>(75)</td><td>388<br>(43)</td><td>68<br>(3)</td><td>143<br>(6)</td><td></td><td></td><td></td><td>1376<br>(150)</td></tr><tr><td>10<br>Θ<sub>IF</sub> = 0.5</td><td><b>11.54</b><br>(0.002)</td><td>68<br>(15)</td><td>1 310<br>(193)</td><td>673<br>(82)</td><td>2 780<br>(322)</td><td>1 617<br>(161)</td><td>137<br>(6)</td><td>316<br>(13)</td><td></td><td></td><td></td><td>6 900<br>(792)</td></tr><tr><td>20<br>Θ<sub>IF</sub> = 0.5</td><td><b>12.14</b><br>(0.008)</td><td>574<br>(185)</td><td>7 714<br>(1 314)</td><td>3 401<br>(480)</td><td>6 713<br>(619)</td><td>3 797<br>(268)</td><td>257<br>(12)</td><td>656<br>(39)</td><td></td><td></td><td></td><td>23 112<br>(2 916)</td></tr><tr><td rowspan="3">Mines VS<br>Rocks<br>(Std. dev.)</td><td rowspan="3"><b>79.2</b><br>(9.8)</td><td>5</td><td><b>74.2</b><br>(5.8)</td><td>136<br>(69)</td><td>299<br>(159)</td><td>8<br>(2)</td><td></td><td></td><td></td><td></td><td></td><td></td><td>443<br>(230)</td></tr><tr><td>10</td><td><b>75</b><br>(4.4)</td><td>259<br>(72)</td><td>576<br>(158)</td><td>20<br>(8)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>855<br>(238)</td></tr><tr><td>20</td><td><b>78.8</b><br>(5.6)</td><td>490<br>(232)</td><td>1 090<br>(499)</td><td>34<br>(24)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>1 614<br>(755)</td></tr><tr><td rowspan="3">Spoken<br>Digits<br>(Std. dev.)</td><td rowspan="3"><b>88.55</b><br>(2.3)</td><td>5</td><td><b>38.2</b><br>(10.1)</td><td>13 918<br>(1 684)</td><td>24 908<br>(17 681)</td><td>13 348<br>(8 975)</td><td>4 839<br>(772)</td><td>3 223<br>(561)</td><td>1 013<br>(280)</td><td>884<br>(243)</td><td>93<br>(3)</td><td>43<br>(3)</td><td></td><td>61 473<br>(30 202)</td></tr><tr><td>10</td><td><b>71.6</b><br>(4.7)</td><td>103 463<br>(17 488)</td><td>5 776<br>(3 534)</td><td>3 148<br>(1 914)</td><td>2 899<br>(736)</td><td>1 891<br>(496)</td><td>1 839<br>(560)</td><td>1 367<br>(422)</td><td>221<br>(16)</td><td>308<br>(17)</td><td></td><td>120 912<br>(25 083)</td></tr><tr><td>20</td><td><b>87.5</b><br>(4.2)</td><td>228 752<br>(26 123)</td><td>12 988<br>(8 818)</td><td>6 933<br>(4 481)</td><td>5 727<br>(948)</td><td>5 705<br>(682)</td><td>3 068<br>(932)</td><td>2 351<br>(708)</td><td>387<br>(28)</td><td>578<br>(57)</td><td></td><td>244 128<br>(42 777)</td></tr><tr><td rowspan="3">RadioML<br>(Std. dev.)</td><td rowspan="3"><b>42</b><br>(0.3)</td><td>5</td><td><b>20.5</b><br>(5.6)</td><td>16 791<br>(3 542)</td><td>108 234<br>(23 076)</td><td>64 131<br>(13 455)</td><td>12 5167<br>(24 690)</td><td>86 702<br>(16 732)</td><td>13 201<br>(1 936)</td><td>12 185<br>(1 846)</td><td>191<br>(19)</td><td>119<br>(15)</td><td>265<br>(34)</td><td>42 6986<br>(85 345)</td></tr><tr><td>10</td><td><b>31.6</b><br>(4.9)</td><td>32 167<br>(2 479)</td><td>206 629<br>(26 715)</td><td>121 349<br>(15 074)</td><td>232 232<br>(25 878)</td><td>160 043<br>(16 370)</td><td>23 934<br>(2 201)</td><td>22 191<br>(2 058)</td><td>471<br>(54)</td><td>302<br>(26)</td><td>714<br>(68)</td><td>800 033<br>(90 923)</td></tr><tr><td>20</td><td><b>37.5</b><br>(3.8)</td><td>508 836<br>(1 290)</td><td>325 052<br>(36 528)</td><td>190 171<br>(20 596)</td><td>360 255<br>(30 068)</td><td>248 090<br>(18 679)</td><td>38 796<br>(2 584)</td><td>36 050<br>(2 412)</td><td>895<br>(75)</td><td>595<br>(37)</td><td>1442<br>(83)</td><td>98 4637<br>(112 352)</td></tr></tbody></table>

$$FR(L_i, L_j) = \frac{N^i}{N^j} \qquad (4.1)$$

Where $FR(L_i, L_j)$ is the filtering rate from layer $i$ to layer $j$ and $N^i$ the spike count at layer $i$.

For $\Theta_{IF=1}$ and $\Theta_{IF=2}$, the filtering rate is always larger than one, thus there is always more input spikes than output spikes: the SNN acts as a spike (low-pass) filter. $\Theta_{IF=0.5}$ experiments demonstrate the opposite: there are more output spikes than input spikes. This observation backs our hypothesis that a low threshold generates “undesired” spikes in the intermediate layers. This phenomenon is also responsible for the drastically deteriorated accuracy, as the Terminate Delta procedure is altered by undesired spikes.
<figure><img src="image_37.png" /><figcaption>Figure 4.6: Evolution of spike filtering rate on CIFAR-10 task</figcaption></figure>

4.2.2.0.3 Combined effect of Δ and ΘIF Moreover, Δ and ΘIF have a combined effect on accuracy and synaptic activity. We have seen that Δ had an impact on accuracy in GTSRB and other tasks. This is also the case on CIFAR-10 with ΘIF = 0.5 and ΘIF = 1. But with ΘIF = 2, the accuracy is constant for all values of Δ. However, the synaptic activity is still affected. With ΘIF = 2, the total synaptic activity for Δ = 5 is 161898, 242275 for Δ = 10 and 313019 for Δ = 20. In order to further investigate the combined effect of Δ and ΘIF, we analyze the Spike Count Evolution (SCE) between Δ = 5 and Δ = 20 against ΘIF, as described in Equation 4.2.

$$ SCE^l(\Delta_1, \Delta_2) = \frac{N_{\Delta_1}^l}{N_{\Delta_2}^l} \qquad (4.2) $$

Where SCE^l(Δ₁, Δ₂) is the SCE of layer l between Δ₁ and Δ₂, and N^lΔ₁ the spike count at layer l for Δ₁.

The SCE is measured in the input layer (Layer #0), in the output layer (Layer #6) and in total. The results are given in Figure 4.7. The vertical axis is logarithmic. The general trend is that SCE decreases with ΘIF, i.e. the spike count is more affected by Δ with low ΘIF values. Moreover, for ΘIF = 1 and ΘIF = 2, the SCE is higher for the output layer than for the input. This trend is drastically reversed when addressing ΘIF = 0.5: the SCE is more than 50 times higher in the input layer than in the output layer. If we refer to Figure 4.6, we can thus assume that the evolution of filtering rate is mostly due to the SCE in the input layer. In other words, the SNN behavior is deteriorated by the excessive number of spikes in the first layer.