<figure><img src="image_38.png" /><figcaption>Figure 4.7: Spike Count Evolution $\Delta = 5$ and $\Delta = 20$ in Layer #0, Layer #6 and in total, for various $\Theta_{IF}$ on CIFAR-10 task. The vertical axis is logarithmic</figcaption></figure>

To prevent this behavior, $\Theta_{IF}$ could be decreased locally in the first layer, as proposed in the Spike Select method proposed in [64]. However, preliminary experiments showed that accuracy was drastically affected by the Spike Select technique, thus it was let out of scope of our study. Further work is therefore required in that regard.

#### 4.2.2.0.4 Influence of dataset
At first glance, the synaptic activity measurements are indeed very different from one task to another. However, the number of synapses involved has an impact on this metric. To isolate the intrinsic effect of dataset, synaptic activity can be normalized with respect to number of synapses. This metric is proposed in the next section: Synaptic Activity Ratio (SAR).

### 4.2.3 Discussions on Synaptic Activity results

In this section, we have studied the influence of the Terminate Delta criterion ($\Delta$) and the Integrate & Fire neuron threshold ($\Theta_{IF}$) on synaptic activity and accuracy of spiking CNNs on various datasets. Both SNN-specific hyper-parameters highlight the existing trade-off between synaptic activity and accuracy.

Increasing the value of $\Delta$ improves accuracy at the expense of a greater synaptic activity. Similarly, increasing the value of $\Theta_{IF}$ also improves accuracy and increases synaptic activity. Moreover, the filtering rate of the network is severely affected by $\Theta_{IF}$. With $\Theta_{IF=0.5}$, there are
more output spikes than input spikes. On the other hand, with $\Theta_{IF} = 1$ and $\Theta_{IF} = 2$ we observe the opposite: the network acts as a strong spike filter, with fewer output spikes at the output than at the input.

Additionally, some combined effects of $\Delta$ and $\Theta_{IF}$ have been found. If Delta has a strong influence on accuracy with $\Theta_{IF} = 0.5$ and $\Theta_{IF} = 1$, this is not the case when $\Theta_{IF} = 2$: the accuracy remains the same for all values of $\Delta$. Moreover, as explained above, the influence of $\Delta$ on the filtering rate varies with $\Theta_{IF}$: for $\Theta_{IF} = 0.5$ the filtering rate increases with respect to $\Delta$, but we observe the opposite for $\Theta_{IF} = 1$ and $\Theta_{IF} = 2$. Lastly, for $\Theta_{IF} = 0.5$, the SCE was greater in the input layer. On the other hand, the SCE was greater in the output layer for $\Theta_{IF} = 1$ and $\Theta_{IF} = 2$.

Consequently, this study helps to understand the behavior of a rate-coded Spiking Neural Network, and how it reacts to different hyper-parameters. More specifically, the study highlighted a strong trade-off between accuracy and synaptic activity. As discussed in our experiments on PADS architecture (Section 3.2), higher synaptic activity implies higher signal toggle rate in the design. In turns, an higher toggle rate implies higher dynamic power usage. Thus, there exists an interesting trade-off between power usage and accuracy. The SAR eases the exploration of such trade-off, as it only requires software-level information.

However, those results only enable the comparison between various experiments on a same task. As explained previously, the spike count has to be normalized to the number of synapse in order to evaluate the impact of the dataset independently from the topology. Moreover, one of our main concern is comparing SNNs and FNNs. Hence, we introduce a metric called "Synaptic Activity Ratio" in the next section. Indeed, in addition to enabling comparison between different SNN topologies, this metric could also enable high-level comparison of SNNs and FNNs in terms of number of operations, and thus, in terms of relative energy consumption per sample.

Lastly, the investigation was limited to converted rate-coded SNNs. The results could be different using other spike-coding techniques, advanced weight and threshold balancing methods, or even totally different SNN model representation. However, the Synaptic Activity Ratio metric proposed in the next section is designed to be adapted to any SNN model.

## 4.3 Synaptic Activity ratio

The Synaptic Activity Ratio (SAR) is a novel metric inspired from previous works, which proposes to count operations in SNNs and equivalent FNNs to assess their relative energy efficiency. The number of spikes per synapse was proposed in various forms in the literature [89] [31] [78] to assess potential energy gains of SNNs. This metric remains however qualitative and lacks formalization. In this section, we propose a formalization through the SAR. Moreover, we develop an high-level model for energy-wise cartography of applications and coding domains that is based on the SAR metric. This model will be confronted to actual energy estimations in Chapter 5. It should be noted that the model is only adapted to FPGA implementations.

### 4.3.1 Energy Consumption Model

As a reminder, we call synaptic operation the integration of an input activation in the neuron. In both FNNs and SNNs, the computation is widely dominated by synaptic operations [78] in Convolution and Fully-Connected layers. This observations comes from measurements of Floating Point Operation per second (FLOP/s) and energy in SNNs and ANNs is directly proportional with
the number of synapses remaining after pruning, that is the number of ACC and MAC operations respectively. Hence, all other computations are neglected in our model. Consequently, the energy consumption of a NN model can be expressed as the energy consumption of its synaptic operations, as shown in Equation 4.3

$$ 
\begin{aligned}
E_{\text{FNN}} &= N_{\text{MAC}} * E_{\text{MAC}} \\
E_{\text{SNN}} &= N_{\text{ACC}} * E_{\text{ACC}}
\end{aligned}
\qquad (4.3) $$

Where $E_{\text{FNN}}$ (respectively $E_{\text{SNN}}$) is the energy consumption of a formal (respectively spiking) neural network implementation, and $E_{\text{MAC}}$ (respectively $E_{\text{ACC}}$) is the energy cost of a MAC (respectively ACC) operation.

In this model, the execution time of the architecture is assimilated to the execution time of synaptic operations (MAC & ACC). The rest of the computation is neglected just as for energy consumption, since synaptic operations are largely dominant in neural network processing.

Let $\lambda = \frac{E_{\text{MAC}}}{E_{\text{ACC}}}$ be the ratio between the energy consumptions of a MAC and an ACC operation. The value of $\lambda$ depends on the implementation technology of each neuron (digital or analog, using LUTs/FFs or dedicated DSPs, semiconductor technology etc.). For example, in the literature we found that Panda et. al. [76], working with 32nm CMOS technology and 32-bit dynamics, measured 3.2pJ energy consumption for a MAC, and 0.1pJ for an ACC. Hence, and for that specific case, $\lambda = 32$. Integrating $\lambda$ into Equation 4.3 yields to Equation 4.4, which defines the relative energy model between spiking and formal domains.

$$ \frac{E_{\text{SNN}}}{E_{\text{FNN}}} = \frac{1}{\lambda} \times \frac{N_{\text{ACC}}}{N_{\text{MAC}}} \qquad (4.4) $$

In a Formal Neural Network, there is one single activation per synapse, as information is static and synchronous. Therefore, there is one and only one MAC operation per synapse. Thus $N_{\text{MAC}} = N_{\text{synapses}}$. On the other hand, in SNNs the number of activation is the number of spikes. Synaptic activity is non-deterministic, but depends on several hyper-parameters (Section 4.2) and random initialization. Hence, $N_{\text{ACC}} = N_{\text{spikes}}$. Replacing this equality in Equation 4.4 yields to Equation 4.5.

$$ 
\begin{aligned}
\frac{E_{\text{SNN}}}{E_{\text{FNN}}} &= \frac{1}{\lambda} \times \frac{N_{\text{spikes}}}{N_{\text{synapses}}} \\
\frac{E_{\text{SNN}}}{E_{\text{FNN}}} &= \frac{1}{\lambda} \times \text{SAR}
\end{aligned}
\qquad (4.5) $$

Indeed, SAR is defined as the ratio between the spike count and the number of synapses. Additionally, Equation 4.6 represents an alternative SAR metric defined at layer level. This layer-level SAR is a fully novel proposition, whose aim is to study the SAR metric at a finer granularity. In details, layer-level SAR can be expressed using the output spike count from the previous layer, enabling easy computation based on the N2D2 logs. It should be noted that this layer-wise SAR only makes sense for Convolution or Fully-Connected layers. Pooling layers are not covered as computation in the network is dominated by Convolution and Full-Connected layers [78].
$$
\begin{align}
\mathrm{SAR}^l &= \frac{N_{\mathrm{ACC}}^l}{N_{\mathrm{MAC}}^l} \nonumber \\
\mathrm{SAR}^l &= \frac{N_{\mathrm{input spikes}}^l}{N_{\mathrm{input synapses}}^l} \nonumber \\
\mathrm{SAR}^l &= \left\{ \begin{array}{ll} \frac{N_{\mathrm{output spikes}}^{l-1} * N_{\mathrm{neurons}}^l}{N_{\mathrm{neurons}}^{l-1} * N_{\mathrm{neurons}}^l} & \text{if } l \text{ is FC} \\ \\ \frac{N_{\mathrm{output spikes}}^{l-1} * N_{\mathrm{filters}}^l * (F_{\mathrm{width}}^l - S^l)^2}{N_{\mathrm{neurons}}^{l-1} * N_{\mathrm{filters}}^l * (F_{\mathrm{width}}^l - S^l)^2} & \text{if } l \text{ is Conv} \end{array} \right. \tag{4.6} \\
\mathrm{SAR}^l &= \frac{N_{\mathrm{output spikes}}^{l-1}}{N_{\mathrm{neurons}}^{l-1}} \nonumber
\end{align}
$$

With:

• SAR^l the Synaptic Activity Ratio for layer l,

• $N_{ACC}^l$ the number of ACC for spiking layer $l$ and $N_{MAC}^l$ the number of MAC for its formal counterpart,

• $N_{input spikes}^l$ the number of input spikes for a layer $l$ for an inference on a single sample,

• $N_{input synapses}^l$ the number of input synapses of a layer $l$

• $N_{neurons}^l$ the number of output neurons of a layer $l$,

• $N_{filters}^l$ the number of filters, $F_{width}^l$ the width of the filter and $S_l$ the stride of a convolution layer $l$.

### 4.3.2 The value of $\lambda$

In this subsection, we focus on the value of $\lambda$, the energy consumption ratio between MAC and ACC operations. More specifically, we aim at finding the $\lambda$ value corresponding to our specific hardware target: Xilinx xc7z020 FPGA (Zedboard). The $\lambda$ value will be useful when interpreting the SAR results. Indeed, according to Equation 4.5, $\lambda$ provides a frontier between coding domains: if $SAR < \lambda$, $E_{SNN} < E_{FNN}$ and vice versa.

In FPGAs, operations are usually mapped to the programmable logic, in LUTs and Flip-Flops. However, Xilinx boards feature specific circuits for Digital Signal Processing: DSPs. Those processing units are optimized for multiplication operation. During hardware synthesis, Vivado maps MAC operations on DSPs. However, DSP is a scarce resource on FPGAs: there are only 220 DSPs on a Zedboard, and 2520 on a ZCU102. To cope with this scarcity, Vivado is able to map several parallel MAC operations on a single DSP by increasing its working frequency. Indeed, DSPs have a maximum working frequency of 741MHz [103]. Usually, we work with a clock frequency of 100MHz, thus a single DSP can be shared for up to 7 parallel MAC operations. In all, Vivado is able to synthesize up to 1540 different MAC operations on the 220 available DSPs of a Zedboard. We thus have the following statement for Zedboard:

• For 0 to 1540 MAC operations, Vivado uses DSPs
<table><caption>Table 4.4: Logic resources, power and energy comparison for 16-bit ACC and MAC operations on Xilinx xc7z020 FPGA. The MAC operation has been implemented with and without DSPs.</caption><thead><tr><th></th><th>ACC</th><th>MAC<br>w/o DSP</th><th>MAC<br>w DSP</th></tr></thead><tbody><tr><td>LUT<br>(#)</td><td>16</td><td>253</td><td>16</td></tr><tr><td>FF<br>(#)</td><td>80</td><td>112</td><td>48</td></tr><tr><td>DSP<br>(#)</td><td>0</td><td>0</td><td>1</td></tr><tr><td>Dynamic Power<br>(mW)</td><td>1.6</td><td>7.1</td><td>1.8</td></tr><tr><td>Energy<br>(pJ)</td><td>16</td><td>71</td><td>18</td></tr></tbody></table>

• Above 1540 MAC operations, Vivado uses LUTs and FFs.

In order to characterize the $\lambda$ value, we have measured the hardware footprint of ACC operation as well as MAC operation with and without using DSPs on Xilinx Zedboard (Table 4.4). In this table, we have $\lambda_1 = \frac{Energy_{MAC DSP}}{Energy_{ACC}} = 1.1$ and $\lambda_2 = \frac{Energy_{MAC no DSP}}{Energy_{ACC}} = 4.4$. When DSPs are used the ratio is named $\lambda_1$, and $\lambda_2$ otherwise. Therefore, the value of $\lambda$ for a Xilinx Zedboard is defined as shown in Equation 4.7.

$$ \lambda = \begin{cases} 1.1 & \text{if } N_{\text{MAC}} \le 1400 \\ \frac{1.1 \times 1400 + 4.4 \times (N_{\text{MAC}} - 1400)}{N_{\text{MAC}}} & \text{if } N_{\text{MAC}} \ge 1400 \end{cases} \quad (4.7) $$

Where $N_{\text{MAC}}$ is the number of parallel MAC operations.

Furthermore, Equation 4.7 can be generalized to other Xilinx boards, as shown in Equation 4.8.

$$ \lambda = \begin{cases} \lambda_1 & \text{if } N_{\text{MAC}} \le 7 \times N_{\text{MAC}}^{\text{max}} \\ \frac{\lambda_1 \times 7 \times N_{\text{MAC}}^{\text{max}} + \lambda_2 \times (N_{\text{MAC}} - 7 \times N_{\text{MAC}}^{\text{max}})}{N_{\text{MAC}}} & \text{if } N_{\text{MAC}} \ge 7 \times N_{\text{MAC}}^{\text{max}} \end{cases} \quad (4.8) $$

Where $N_{\text{MAC}}$ is the number of parallel MAC operations, $N_{\text{MAC}}^{\text{max}}$ is the number of available DSPs on the board, $\lambda_1$ is the energy consumption ($\frac{E_{\text{MAC}}}{E_{\text{ACC}}}$) using DSPs and $\lambda_2$ without using DSPs.

For ZCU102, we measure the energy of ACC and MAC operations with and without DSPs: $E_{\text{ACC}}^{ZCU102} = 18fJ$, $E_{\text{MAC w DSP}}^{ZCU102} = 27fJ$ and $E_{\text{MAC w no DSP}}^{ZCU102} = 103fJ$. Thus, we found $\lambda_1^{ZCU102} = 1.5$, and $\lambda_2^{ZCU102} = 5.7$. Moreover, in the case of ZCU102, there are 2520 available DSPs, thus $N_{\text{MAC}}^{\text{max}} = 2520$. Using Equations 4.7 and 4.8, we compute $\lambda$ for Zedboard and ZCU102, with respect to the number of synapses in the design (i.e. the number of MAC operations). The results are provided in Figure 4.8.

In order to estimate $\lambda$ for Zedboard ($\lambda_{\text{ZE}}$) and ZCU102 ($\lambda_{\text{ZC}}$), we count the number of MACs in VGT [5] architecture with respect to the network topology. Consequently, the SAR metric is specifically adapted to a comparison between PADS and VGT, two equivalent parallel neuro-morphic accelerators. Equations 4.9 are derived from the VGT code. The code is available on
<figure><img src="image_39.png" /><figcaption>Figure 4.8: Evolution of λ against number of parallel MACs, for Xilinx Zedboard and ZCU102 targets.</figcaption></figure>

GitHub [104]. Those equations enable to compute the number of MACs required for Conv and FC layers in VGT architecture.

$$ 
\begin{align} 
N_{\text{Conv}}^{\text{MAC}} &= N_{\text{In}}^{\text{Chan}} \times N_{\text{Out}}^{\text{Chan}} \times H_{\text{Filter}} \times W_{\text{Filter}} \tag{4.9} \\ 
N_{\text{FC}}^{\text{MAC}} &= N_{\text{In}}^{\text{Neur}} \times N_{\text{Out}}^{\text{Neur}} 
\end{align} 
$$

Where $N_{\text{Conv}}^{\text{MAC}}$ is the number of MACs in a VGT convolution layer with $N_{\text{In}}^{\text{Chan}}$ input and $N_{\text{Out}}^{\text{Chan}}$ output channels, using a convolution filter of height $H_{\text{Filter}}$ and width $W_{\text{Filter}}$. $N_{\text{FC}}^{\text{MAC}}$ is for Fully-Connected layers having $N_{\text{In}}^{\text{Neur}}$ input and $N_{\text{Out}}^{\text{Neur}}$ output neurons.

We estimate the total number of MACs for each topology using Equation 4.9. The subsequent counts are shown in Table 4.5a. In the end, the values of $\lambda_{\text{ZE}}$ and $\lambda_{\text{ZC}}$ given in Table 4.5a are derived from Equation 4.5 based on the number of MACs shown in Table 4.5a. The resulting $\lambda$ values are used in the next section. It helps determining if an application is suitable for spiking domain regarding SAR: if $\text{SAR} < \lambda$, $E_{\text{SNN}} < E_{\text{FNN}}$ and vice versa.

## 4.4 Synaptic Activity Ratio measurements

Based on the synaptic activity results of Section 4.2 (Table 4.3), we compute the layer-wise and network-wise SAR for each experiment. To do so, a python software is developed. The program takes synaptic activity trace from N2D2 and computes layer-wise and network-wise SAR.

### 4.4.1 Synaptic Activity Ratio evaluation software

The SAR estimation software works on N2D2 output files: synaptic activity logs. Those logs contains the synaptic trace of all layers of the network for one image. A dedicated python plug-in catches those files on the fly as they are automatically overwritten by N2D2 at each log. For each layer, 50 activity logs are stored in a auto-generated folder tree. The software then translates
(a) Number of MAC operations in the VGT architecture adapted to each CNN of our benchmark (topologies in Tables 4.2).

<table><thead><tr><th>Task</th><th>Number of MACs</th></tr></thead><tbody><tr><td>MNIST</td><td>24 894</td></tr><tr><td>GTSRB</td><td>137 682</td></tr><tr><td>CIFAR-10</td><td>260 680</td></tr><tr><td>Mines VS Rocks</td><td>12 400</td></tr><tr><td>Spoken Digits</td><td>137 692</td></tr><tr><td>RadioML 2018</td><td>40 722 383</td></tr></tbody></table>

(b) Estimated values of $\lambda$ on the dataset benchmark using VGT architecture on Zedboard and ZCU102

<table><thead><tr><th>Task</th><th>λ<sub>ZE</sub></th><th>λ<sub>ZC</sub></th></tr></thead><tbody><tr><td>OPSSAT</td><td>1.5</td><td>1.35</td></tr><tr><td>MNIST</td><td>4.21</td><td>2.72</td></tr><tr><td>GTSRB</td><td>4.37</td><td>5.15</td></tr><tr><td>CIFAR-10</td><td>4.38</td><td>5.42</td></tr><tr><td>Mines VS Rocks</td><td>4.03</td><td>1.50</td></tr><tr><td>Spoken Digits</td><td>4.25</td><td>3.32</td></tr><tr><td>RadioML 2018</td><td>4.40</td><td>5.70</td></tr></tbody></table>

<figure><img src="image_40.png" /><figcaption>Figure 4.9: Unified synaptic trace format</figcaption></figure>

the activity logs into a unified format: a 2D matrix with flattened input synapses as rows and timesteps as columns. This format is shown in Figure 4.9. Each cell contains the state of an input synapse i at a given timestep: 1 if it carries a spike and 0 otherwise.

To build this unified synaptic trace format, the real-time synaptic trace of N2D2 is discretized. The temporal sampling rate (*timestep length*) is set according to the real PADS GenCell behavior (see Section 3.1.1). A function automatically scans all input synaptic traces and determines the minimum delay between two spikes $\delta t$. As the GenCell emits at most one spike per timestep, the temporal resolution of the discretization process is set to $\delta t$. As a reminder, the timestep duration of PADS GenCell is equal to the input size in clock cycles (784 clock cycles for MNIST, 3072 for GTSRB...). The program then counts the total number of spikes per synapse and computes the SAR according to formulas 4.5 and 4.6 for network-wise and layer-wise SAR respectively. Layer-wise and network-wise SAR are computed for each experiment of the benchmark using the dynamic SAR estimator software.
<table><caption>Table 4.6: Large CNN topology for MNIST dataset</caption><thead><tr><th>Layer</th><th>Type</th><th># Kernels</th><th>Kernel size</th><th>Stride</th><th>Output Shape</th></tr></thead><tbody><tr><td>0</td><td>Input</td><td></td><td></td><td></td><td>(28,28,1)</td></tr><tr><td>1</td><td>Conv</td><td>32</td><td>5</td><td>1</td><td>(24,24,32)</td></tr><tr><td>2</td><td>MaxPool</td><td>32</td><td>2</td><td>2</td><td>(12,12,32)</td></tr><tr><td>3</td><td>Conv</td><td>32</td><td>5</td><td>1</td><td>(8,8,32)</td></tr><tr><td>4</td><td>MaxPool</td><td>32</td><td>2</td><td>2</td><td>(4,4,32)</td></tr><tr><td>6</td><td>FC</td><td></td><td></td><td></td><td>120</td></tr><tr><td>7</td><td>FC</td><td></td><td></td><td></td><td>84</td></tr><tr><td>8</td><td>FC</td><td></td><td></td><td></td><td>10</td></tr></tbody></table>

### 4.4.2 Network-wise SAR & theoretical cartography

The SAR results are presented in Figures 4.10 and 4.11. In those figures, $C_i$ stands for the $i^{th}$ convolution layer, and $FC_i$ stands for the $i^{th}$ fully-connected layer. In this section, we analyze the network-wise SAR results (in gray) on the dataset benchmark. These values are compared to $\lambda_{ZE}$ and $\lambda_{ZC}$ to assess the suitability to spiking domain. The network-wise results are compiled in Table 4.7. In this table, the cells are green when the SAR is above $\lambda_{ZE}$ and $\lambda_{ZC}$. This is the best case for spiking domain according to Equation 4.5. The cells are yellow when SAR is between the two $\lambda$ values. In this case, the suitability depends on the hardware target. When SAR is below both $\lambda$ values, the case is favorable to formal domain and cells are in red.

The network-wise SAR increases with $\Delta$ in each topology, as expected from the raw synaptic activity analysis. Moreover, the SAR seems to increase with task complexity. The SAR is indeed higher in CIFAR-10 than in GTSRB, which is higher than in MNIST. The same trend is visible for vector data (Mines VS Rocks, Spoken Digits and RadioML 2018). In other words, spiking acceleration seems better suited for simple tasks, at least for rate-based models. Additionally, the SAR is generally higher for 1D vector datasets at comparable complexity level. For both Spoken Digits and RadioML datasets, the best case accuracy ($\Delta = 20$) fails to meet the $\lambda$. According to the proposed model, the SNN will thus use more energy than an equivalent FNN on those tasks. More generally, the SAR results tend to indicate that spiking domain is more suited to image datasets than 1D vector datasets. It should be noted that this statement is constrained to converted rate-coded SNNs and parallel accelerators. When analyzing those results in the light of the lambda values of Table 4.5b, we observe that spiking domain is preferable in environments where DSPs are saturated. Indeed, $\lambda$ increases when DSPs are saturated (Figure 4.8). This is visible on the figures, as $\lambda_{ZE}$ is usually lower than $\lambda_{ZC}$. As large topology uses more MACs, and thus more DSPs, this statement should imply that spiking domain takes advantage from topology size. For example, OPS-SAT is a very small CNN topology, thus the values of $\lambda$ are small. Thus, OPS-SAT with $\Delta = 20$ is not suitable to spiking domain even if the SAR is low (2.2). However, topologies associated to complex datasets are larger than that of simple datasets. In order to separate the effect of topology size and dataset complexity, we use a larger CNN topology on MNIST and measure SAR with N2D2. The larger topology is provided in Table 4.6. The subsequent SAR measurements and lambda values are shown in Figure 4.12.

The SAR for MNIST with either the baseline (Figure 4.10) or larger topology (Figure 4.12) are very close. The values of $\lambda_{ZE}$ and $\lambda_{ZC}$ are much greater in the larger CNN ($\lambda_{ZE}^{OPSSAT} = 1.5$ and $\lambda_{ZE}^{RadioML} = 4.40$), thus this implementation is even more favorable to spiking domain, according to the SAR energy model. This experiment thus comforts our hypothesis that SAR is related
<figure><img src="image_41.png" /><figcaption>Figure 4.10: SAR (above) and accuracy (below) on OPSSAT, MNIST, GTSRB and Mines Vs Rocks datasets classification, for $\Delta = 5, 10$ and $20$. $\lambda_{ZE}$ and $\lambda_{ZC}$ are also represented. Spiking inference with N2D2.</figcaption></figure>
<figure><img src="image_42.png" /><figcaption>Figure 4.11: SAR (above) and accuracy (below) on Spoken Digits, RadioML 2018 and CIFAR-10 datasets classification, for $\Delta = 5, 10$ and $20$. $\lambda_{ZE}$ and $\lambda_{ZC}$ are also represented. Spiking inference with N2D2. Additionally, for CIFAR-10 $\theta_{IF}$ varies between 0.5, 1 and 2.</figcaption></figure>
<table><caption>Table 4.7: Network-wise Synaptic Activity Ratio for the dataset benchmark, with varying $\Delta$ and $\Theta_{IF}$ values. The cells are green when SAR is above $\lambda_{ZE}$ and $\lambda_{ZC}$, yellow when it is between the two and red when it is below.</caption><thead><tr><th>Task</th><th>FNN<br>ACC. (%)</th><th>Δ</th><th>SNN<br>ACC. (%)</th><th>SAR</th></tr></thead><tbody><tr><td rowspan="3">OPSSAT RGB<br>(Std. dev.)</td><td rowspan="3">91.95<br>(0.37)</td><td>5</td><td>91.78<br>(0.52)</td><td>0.39</td></tr><tr><td>10</td><td>91.88<br>(0.83)</td><td>0.92</td></tr><tr><td>20</td><td>91.92<br>(1.21)</td><td>2.22</td></tr><tr><td rowspan="3">MNIST<br>(Std. dev.)</td><td rowspan="3">99.2<br>(0.2)</td><td>5</td><td>97.3<br>(0.6)</td><td>0.58</td></tr><tr><td>10</td><td>98.9<br>(0.5)</td><td>0.82</td></tr><tr><td>20</td><td>99.1<br>(0.6)</td><td>1.24</td></tr><tr><td rowspan="3">GTSRB<br>(Std. dev.)</td><td rowspan="3">95.5<br>(0.8)</td><td>5</td><td>83.3<br>(5.1)</td><td>2.24</td></tr><tr><td>10</td><td>90.6<br>(1.8)</td><td>3.59</td></tr><tr><td>20</td><td>93.5<br>(1.4)</td><td>7.70</td></tr><tr><td rowspan="10">CIFAR-10<br>(Std. dev.)</td><td rowspan="10">69.8<br>(1.1)</td><td>5<br>Θ<sub>IF</sub> = 0.5</td><td>11.45<br>(6.1)</td><td>0.05</td></tr><tr><td>10<br>Θ<sub>IF</sub> = 0.5</td><td>11.54<br>(1.9)</td><td>0.23</td></tr><tr><td>20<br>Θ<sub>IF</sub> = 0.5</td><td>12.14<br>(0.9)</td><td>0.73</td></tr><tr><td>5<br>Θ<sub>IF</sub> = 1</td><td>31.8<br>(0.9)</td><td>0.94</td></tr><tr><td>10<br>Θ<sub>IF</sub> = 1</td><td>50.2<br>(2.7)</td><td>2.57</td></tr><tr><td>20<br>Θ<sub>IF</sub> = 1</td><td>57.8<br>(1.9)</td><td>7.57</td></tr><tr><td>5<br>Θ<sub>IF</sub> = 2</td><td>59.2<br>(0.9)</td><td>7.88</td></tr><tr><td>10<br>Θ<sub>IF</sub> = 2</td><td>59.2<br>(0.9)</td><td>11.77</td></tr><tr><td>20<br>Θ<sub>IF</sub> = 2</td><td>59.2<br>(0.9)</td><td>15.64</td></tr><tr><td rowspan="3">Mines VS<br>Rocks<br>(Std. dev.)</td><td rowspan="3">79.2<br>(9.8)</td><td>5</td><td>74.2<br>(5.8)</td><td>0.58</td></tr><tr><td>10</td><td>75<br>(4.4)</td><td>1.07</td></tr><tr><td>20</td><td>78.8<br>(5.6)</td><td>3.34</td></tr><tr><td rowspan="3">Spoken<br>Digits<br>(Std. dev.)</td><td rowspan="3">88.55<br>(2.3)</td><td>5</td><td>38.2<br>(10.1)</td><td>2.04</td></tr><tr><td>10</td><td>71.6<br>(4.7)</td><td>6.41</td></tr><tr><td>20</td><td>87.5<br>(4.2)</td><td>8.24</td></tr><tr><td rowspan="3">RadioML<br>(Std. dev.)</td><td rowspan="3">42<br>(0.3)</td><td>5</td><td>20.5<br>(5.6)</td><td>6.20</td></tr><tr><td>10</td><td>31.6<br>(4.9)</td><td>11.45</td></tr><tr><td>20</td><td>37.5<br>(3.8)</td><td>17.41</td></tr></tbody></table>
<figure><img src="image_43.png" /><figcaption>Figure 4.12: Activity ratio on MNIST using a larger topology, obtained using N2D2. Spiking Accuracy = 98,73%; Formal Accuracy = 99,19%</figcaption></figure>

to intrinsic task complexity, and not topology size. In the next section of this chapter (Section 4.4.3), we focus on the relation between data type (spatial or vector) and SAR. The aim is to explain why vector datasets demonstrate higher SAR than images. Our intuition is that vector datasets are not intrinsically unfavorable to spiking domain. Rather, our hypothesis is that the current rate-coding policy (Equation 2.5) is not adapted to the data distribution.

### 4.4.3 Data type and rate-coding

As explained in section 2.3.2, the rate-coding policy associates spike trains with high frequency to pixels with high intensity, and *vice versa*. The spike train period with respect to input intensity is represented in Figure 4.13. For a better readability, the vertical axis is logarithmic. According to this Figure, rate-coding makes a clear distinction between elements of very low intensity and others. On the other hand, Figure 4.14 provides the distribution of intensity in input data for all datasets of the benchmark. In this figure, image datasets (on the left) features wide intensity distribution, ranging from 0 to 255. On the other hands, the distribution is much narrower for vector datasets (on the right): there is no (or very few) elements of low intensity in such data. The current rate-coding function (Figure 4.13) is therefore not suited for data with narrow intensity distribution. In the current situation, rate encoding implies a drastic loss of information of in the case of vector datasets. This phenomenon explains why such data have poor accuracy and high SARs. The SAR and accuracy could be improved by rescaling input data on $[0; 255]$ or by moving to another spike coding strategy.

### 4.4.4 Layer-wise SAR & hybridization

Besides the aforementioned network-wise results, the SAR was also computed at the layer level following Equation 4.6. In SAR result figures (Figures 4.10 and 4.11), the convolution layers (Conv) are shown in variations of blue, and the fully-connected (FC) layers are shown in nuances
<figure><img src="image_44.png" /><figcaption>Figure 4.13: Spike train period with respect to element intensity according to the rate-coding policy, with $PeriodMax = 1 \times 10^{11}$ fs and $PeriodMax = 1 \times 10^{7}$ fs (default N2D2 parameters used in our experiments)</figcaption></figure>

of green. This finer granularity level offers a different view on this metric. It shows how SAR varies from one layer to another in the same experiment. As seen in the results, the SAR indeed varies greatly from one layer to the other. For example in RadioML task with $\Delta = 20$, the SAR in layer *Conv1* (24.9) is nine times higher than in *FC1* (2.8). This is the most extreme case of intra-topology SAR variability. The gap is not so high in other experiments, but remains always significant.

According to those layer-wise results and based on the SAR energy model, some layers seem more adapted to spiking domain than others within the same network. Moreover, some layers even show a local SAR favorable to spiking domain (below or between $\lambda_{ZE}$ and $\lambda_{ZC}$), whereas others are not. This is the case in Spoken Digits with $\Delta = 20$; where FC2 is the only layer whose SAR is below $\lambda_{ZE}$. In other words, the layer-wise SAR of some layer seem suitable for spiking domain acceleration while the overall SAR does not. Under such circumstances, adapting the coding domain at layer-level could be a fruitful compromise between spiking and formal implementation.

## 4.5 Conclusion

In this chapter, we proposed a list of machine learning datasets with various levels of difficulty, covering both spatial (images) and vector data (sonar echoes, Fourier transform of voice recordings and RF signals). This benchmark represents a wide variety of possible applications for embedded neural networks. Synaptic activity was measured on this benchmark using various CNN topologies. The influence of SNN-specific hyper-parameters on synaptic activity and SAR was studied, namely $\Delta$ and $\Theta_{IF}$. This study highlighted the trade-off between synaptic activity and accuracy. Moreover, studying the influence of $\Delta$ and $\Theta_{IF}$ helped us to understand the behavior of spiking neural
networks in various conditions.

We also propose a model to assess the potential energy savings provided by spiking acceleration over conventional formal implementations. This model is based on the Synaptic Activity Ratio metric (i.e. average number of spikes per synapse), and on the relative cost of MAC and ACC operations in FPGA. Based on a widespread approximation in literature [78] [31], the model assimilates the energy consumption of an hardware neural network to that of the synaptic operations: MACs in FNNs, ACCs in SNNs. In order to refine this approach, we propose a model to evaluate $\lambda = \frac{E_{MAC}}{E_{ACC}}$ on FPGA targets. The model takes the saturation of DSP into account, which depends on the target device and on the FNN accelerator baseline design (VGT in this case). The position of the SAR relatively to $\lambda$ therefore defines the suitability of the model to spiking acceleration: $(SAR) < \lambda$ indicates that the spiking model should use less energy than the formal one in hardware. Thanks to the model, we are thus able to propose a fast, high-level cartography of suitable applications for neuromorphic acceleration based on rate-coded SNNs. For instance, the model is applied to a benchmark of datasets using the N2D2 [1] framework.

The results show that simpler tasks provide better SAR independently from topology size. Accordingly, simple tasks seems more suited to rate-based spiking domain. Additionally, spiking domain was more suited to images than 1D data. In response, we propose that tailoring the rate-coding policy to the data dynamic range could be a solution. The same kind of study should be applied to other types of spike coding (time, rank order...). The SAR metric and $\lambda$ model can be used to determine the suitability of a spike encoding rule to hardware acceleration. Finally, this method could be applied to other families of SNNs: in Chapter 7, we use the SAR metric and $\lambda$ model on a novel, timestep-constrained SNN model proposed by Neftci et. al. [32]. Lastly, we have studied layer-wise SAR, and found great variability of SAR within a same network. Some layers show SAR lower than $\lambda$, whereas others don't. More generally, we propose that coding domain hybridization could be an interesting idea, by tailoring the coding domain at a finer granularity, and thus benefiting from spiking domain locally in the architecture. This proposition will be studied further when addressing coding domain hybridization in Chapter 6.
<figure><img src="image_45.png" /><figcaption>Figure 4.14: Distribution of element intensity in our benchmark datasets. Left are image datasets, right are 1D datasets. a) MNIST, b) Mines VS Rocks, c) GTSRB, d) Spoken Digits, e) CIFAR-10, f) RadioML2018</figcaption></figure>
# Chapter 5

## Hardware Footprint and High-Level Estimations

### Chapter contents

<table><tr><td>5.1 Motivations</td><td>79</td></tr><tr><td>5.1.1 Speed-up cartography and exploration</td><td>79</td></tr><tr><td>5.1.2 Layer-wise approach</td><td>79</td></tr><tr><td>5.1.3 Level of parallelism</td><td>79</td></tr><tr><td>5.2 Framework</td><td>80</td></tr><tr><td>5.2.1 Hardware measurements database</td><td>80</td></tr><tr><td>5.2.2 Execution time</td><td>85</td></tr><tr><td>5.2.3 Spiking hardware inference simulator</td><td>90</td></tr><tr><td>5.3 Ressource estimations</td><td>92</td></tr><tr><td>5.3.1 Layer-wise estimation</td><td>93</td></tr><tr><td>5.3.2 Network-wise estimation</td><td>95</td></tr><tr><td>5.3.3 Conclusions</td><td>96</td></tr><tr><td>5.4 Inference time and power estimations</td><td>96</td></tr><tr><td>5.4.1 Layer-wise estimation</td><td>96</td></tr><tr><td>5.4.2 Network-wise estimation</td><td>100</td></tr><tr><td>5.4.3 Conclusions</td><td>101</td></tr><tr><td>5.5 Energy estimations</td><td>101</td></tr><tr><td>5.5.1 Layer-wise estimation</td><td>102</td></tr><tr><td>5.5.2 Network-wise estimation</td><td>103</td></tr><tr><td>5.5.3 Conclusions on energy estimations</td><td>104</td></tr><tr><td>5.6 Validation of the SAR model</td><td>106</td></tr><tr><td>5.7 Conclusion</td><td>107</td></tr><tr><td>5.8 Outlooks</td><td>110</td></tr><tr><td>5.8.1 Improvement of the SAR metric</td><td>110</td></tr><tr><td>5.8.2 Improvement of the estimation framework</td><td>110</td></tr><tr><td>5.8.3 Studying the level of parallelism</td><td>111</td></tr><tr><td>5.8.4 Hybridization and other spike encoding methods</td><td>111</td></tr></table>
In this chapter, we propose a framework for high level prototyping of neuromorphic accelerators on FPGA. Based on high-level information (description of the CNN and software inference logs), the framework is able to accurately estimate logic resources occupation, power, execution time and energy of various FPGA NN accelerators. The goal of its purpose is to enable the exploration of design spaces and application use cases for neuromorphic acceleration. Most importantly, the framework focuses on the comparison of formal and spiking domains under two opposed levels of parallelism. The framework involves estimation for the following designs:

* PADS: the fully-parallel spiking architecture for FC layers introduced in Chapter 3.

* SPLEAT: the fully-sequential spiking architecture described in Section 2.4.2.8.

* VGT: the fully-parallel formal architecture described in Section 2.2.2.1.

* C-HLS: the fully-sequential formal architecture described in Section 2.2.2.2.

The framework is built upon two main bricks: the hardware-footprint database and the spiking inference simulator. The first is a structured set of hardware measurements (LUT,FF, RAM, DSP, Dynamic Power, Execution Time) derived from a vast hardware synthesis and simulation campaign. It should be noted that the database required 264 sets of hardware synthesis, post-synthesis simulation and power analysis using Xilinx Vivado toolchain. That is more than 300 hours of computation, not counting all the software crashes and unused results. Interpolation of the database enables reliable resource estimations. Inference time and dynamic power estimations for formal architectures (VGT and C-HLS) are obtained in the same way, as they only depend on the CNN topology. However in spiking architecture (PADS and SPLEAT), time and power also depends on synaptic activity. That is the role of the second brick of framework: the spiking inference simulator. This software uses information from the hardware footprint database and N2D2 synaptic activity logs. It provides reliable time and power estimations for SPLEAT and PADS. Finally, energy estimations are derived from time and power estimations. The framework is described in details in Section 5.2. The framework is built at layer level. Hence, the database and inference simulator provide layer-wise estimations, which are combined at network-level. A Specific software is developed to simulate network-level pipeline in VGT, PADS and SPLEAT. C-HLS architecture does not feature inter-layer pipeline and is thus not concerned.

The framework is applied to the benchmark of Machine Learning datasets mentioned in Section 4.1. It should be noted that this corresponds to 143 layer-wise measurements. Obtaining hardware measurements would have required 100+ hours of computation using conventional methods. The framework reduces this duration to a few minutes. In doing so, we provide insights and trends on the cartography of applications, coding domain and architectural choices. In this document, explanations focus on OPS-SAT and Spoken Digits datasets for clarity and simplicity purpose, but all the other results are listed in Appendix B. Resource estimations are provided in Section 5.3, power and inference time in Section 5.4 and energy in Section 5.5. When analyzing the results, the goal will be to find the conditions in which SNNs are preferable over FNNs, and quantify the savings. Moreover, the energy estimations results are used to validate the SAR energy model in Section 5.6.

Additionally, it should be noted that all hardware experiments performed in this section have been made on Xilinx Zedboard. The comparisons between two architectures can be extrapolated to other board, but the absolute values provided are only valid for this specific hardware target. Indeed, low-level optimizations and CMOS technology can change from one board to the other and drastically change the subsequent resource, power and energy estimations.
# 5.1 Motivations

One of the main goals of this thesis is to provide a cartography of applications, neural coding domains and parallelism. To do so, the most direct approach would have been to provide direct hardware measurements on a defined benchmark. Instead, we have chosen to propose a framework and software for layer-wise estimation. In this section, we provide details on the elements which motivated this approach.

## 5.1.1 Speed-up cartography and exploration

As mentioned in the introduction, hardware measurements of a single design requires extended periods of time for synthesis, post-synthesis simulation and power analysis. Moreover, the number of parameters involved in our cartography is nearly infinite. For example: the CNN topology, the initial weight distribution, the threshold of Intergate & Fire neurons, the $\Delta$ value in the Terminate Delta, the rate encoding frequency ranges... Each combination of parameters requires hours of work and computation, thus it is unrealistic to provide an exhaustive exploration. A framework which enables low-level estimations based on high-level information therefore drastically mitigates the time requires for exploration. In doing so, this work enables to rapidly draw trends in the cartography. The benchmark presented in this section (and in Appendix .2) would have required approximately 500 hours of computation alone. Thanks to the framework, results can be obtained in less than an hour. Moreover, the framework can be used by other students and researchers to extend the benchmark proposed in this Chapter.

## 5.1.2 Layer-wise approach

The layer-wise approach of the framework is motivated by three distinct aspects. First, layers are the basic bloc of feed-forward neural network topologies. Providing layer-wise results for the most common types (convolution, max-pooling and fully-connected) enables to reconstruct a wide range of CNN topologies. Second, fully-parallel implementations of large designs such as RadioML 2018 CNN (Table 4.2) often overpass the available resources of the target board (Zedboard). In such conditions, synthesis duration is largely extended. Moreover, the results are distorted by low-level optimization, altering generalization of measurements. Performing synthesis at layer-level solves this issue as designs are smaller. Third and last, the layer-wise approach gives the opportunity to study the possibilities of hybridization. This feature will be used further in Chapter 6.

## 5.1.3 Level of parallelism

The four architectures that compose our benchmark are either fully sequential or fully parallel. Those are only prototypes and a realistic architecture should involve an intermediate level of parallelism. However, studying the two extremes of the spectrum yields insights on the underlying trends. More than proposing a real benchmark for the four chosen architecture, the goal is rather to study the influence of parallelism on the comparison between formal and spiking coding domains. Moreover, the layer-wise estimations will also provide insights on the tuning of parallelism at layer level in spiking architectures. Such information lacks in literature and could facilitate the adoption of spiking technology in industry.
<figure><img src="image_46.png" /><figcaption>Figure 5.1: Overview of the high-level estimation framework</figcaption></figure>

## 5.2 Framework

In this section, we describe the high-level estimation framework in details. An overview of the framework is available in Figure 5.1. The structure is divided in three main areas, illustrated in shades of red, green and blue. Each area corresponds with one section of the framework. The green section addresses the hardware measurement database. It also involves the python API developed for interpolation of measurements. The logic resources estimations (LUT, FF, RAM, DSP) for all architectures are obtained directly after interpolation, as well as VGT dynamic power and C-HLS power and execution time. The green area will be described in Subsection 5.2.1. The red section addresses the generation of spike traces from the N2D2 activity logs. It has already been described in the previous chapter (Section 4.4.1). Finally, the blue section covers the inference simulators. The role of this area is to compute dynamic power and inference times for spiking accelerators based on spike traces, active and idle power from the database, and single spike execution times. The single spike execution times are computed separately (Subsection 5.2.2.3). The hardware spiking inference simulator is described in Subsection 5.2.3. Additionally, a dedicated high-level behavioral simulator computes VGT inference times (Subsection 5.2.2.2). It has been developed to simulate the inter-layer pipeline in this architecture.

### 5.2.1 Hardware measurements database

We begin by describing the green area of the framework in Figure 5.1. That is, the hardware measurement database and python interpolation API. In a first time, we describe the structure of the database *i.e. design spaces* and the experimental protocol settled for hardware measurements.
Then, we describe and shortly discuss the 3D interpolation of hardware metrics on the design spaces.

### 5.2.1.1 Design Spaces & Measurement Campaign

The database should provide information at layer level for each of the four architectures (SPLEAT, C-HLS, PADS and VGT). There are 3 types of layers: Convolution (CONV), Pooling (POOL) and Fully-Connected (FC). The database is thus divided in 12 sets, one for each combination of layer and architecture. Those sets are called “design spaces” in the remaining of this work. We focus on 2D design spaces, i.e. design spaces that are governed by two parameters. The number of dimensions of a design space corresponds to the number of considered variables (i.e. hyper-parameters). In our case, the study is limited to 2D design spaces. The corresponding hyper parameters are:

* For Convolution layers: Number of filters ($N_{\text{fil}}$) and width of input ($S_{\text{in}}$);

* For Pooling layers: Number of input channels ($N_{\text{chan}}^{\text{in}}$) and input width ($N_{\text{chan}}^{\text{out}}$);

* For Fully-Connected layers: Number of input neurons ($N_{\text{neur}}^{\text{in}}$) and number of output neurons ($N_{\text{neur}}^{\text{out}}$).

There are other hyper-parameters such as the number of input channels in convolutions or the stride in max-poolings. However, each additional dimension multiplies the number of hardware synthesis required to build the database. Using more than 2D design spaces would imply an unmanageable amount of time. A preliminary study (which is not detailed here) have shown that hardware footprint is mostly governed by the two selected hyper-parameters. That is, apart from the number of input channels in convolution layers ($N_{\text{chan}}^{\text{in}}$). This parameter has as strong influence that must be taken in account. In a dedicated paragraph at the end of this subsection, we provide the approximation made to acknowledge this parameter. However, $N_{\text{chan}}^{\text{in}}$ is not considered in the convolution design space, and the approximation is only applied during interpolation.

Each design space is thus defined by the range of its defining variables. For each layer, the ranges are tuned to cover all layers involved in the CNNs of the benchmark benchmark (Section 4.1.1). It should be noted that the first fully-connected layer (FC1) of RadioML 2018 CNN is not covered, due to its eccentricity. RadioML-FC1 has 15872 input neurons, where the second largest layer only has 800. It is thus omitted in all the following results.

* For Convolution layers: $N_{\text{fil}}$ varies from 1 to 50, and $S_{\text{in}}$ from 1 to 128 (128 × 128p images);

* For Pooling layers: $N_{\text{fil}}$ varies from 1 to 50, and $S_{\text{in}}$ from 1 to 128;

* For Fully-Connected layers: $N_{\text{neur}}^{\text{in}}$ varies from 1 to 800 and $N_{\text{neur}}^{\text{out}}$ varies from 1 to 250.

The other parameters have the following default values:

* For Convolution layers: Filter size is 5 × 5, Stride is 1 and $N_{\text{chan}}^{\text{in}}$ = 1 during the measurement campaign;

* For Pooling layers: Filter size is 2 × 2, Stride is 2 and $N_{\text{chan}}^{\text{in}}$ = $N_{\text{filters}}$;

* For Fully-Connected layers, there are no other hyper-parameters.
<figure><img src="image_47.png" /><figcaption>Figure 5.2: Design space for a) FC layers (30 points), b) Conv & Pool layers (24 points each)</figcaption></figure>

We select a set of points in each design space. 30 for Fully-connected layers, and 24 for Convolution and Pooling. Those points are the same for each architecture, except for PADS which does not cover Convolution and Pooling. At each point, we measure the hardware footprint of each accelerator: PADS, VGT, SPLEAT and C-HLS. The resulting Design Spaces with measurement points are shown in Figure 5.2a for FC layers and Figure 5.2b for convolution and pooling layers. The design spaces of convolution and pooling layers are identical.

The synthesis and measurement protocol for each point is the same as described in Section 3.2.1. For a given measurement point, each architecture is configured according to the coordinates of the point. The four designs are synthesized independently using Vivado Design Suite. A VHDL testbench is configured for each architecture and used for post-synthesis simulation using Vivado XSIM. This VHDL testbench contains realistic input stimuli for the layer. Those stimuli are inspired from previous experiments. During post-synthesis simulation, all signal switches are logged in a SAIF file (Switching Activity Interchange Format). This file is then used in the Vivado Power Analyzer to estimate dynamic power consumption. The realistic stimuli in the testbenches ensure an accurate estimation. Finally, logic resource and power reports are stored in XML format. This protocol is repeated four times (four architectures) for each of the 78 measurement points. All the synthesis, post-synthesis simulation, reporting and storage has been automated via bash and TCL scripts. A bash scripts iterates over all measurement points, and launches a dedicated TCL script for each. The TCL script opens Vivado, loads source files, performs synthesis and post-synthesis simulation. Afterward, a TCL command automatically generates and stores power and resources reports in XML format.

#### 5.2.1.1.1 $N_{chan}^{in}$ in convolution layers
As explained in previous section, $N_{chan}^{in}$ is not considered in the design space of convolution layers. In this section, we explain how it is approximated differently for each architecture.

* PADS is not concerned at it does not feature convolution.
* SPLEAT is event-based, thus the only change is the number of bits required to encode an event (the channel address is encoded in binary). Preliminary experiments have shown that the impact of such modification is negligible even for large number of input channels.

* C-HLS is fully-sequential, thus the number of input channels multiplies the execution time but does not affect resources and power. The number of channels is therefore addressed later

* In VGT, input channels are processed in parallel. Thus, input channels multiplies the logic resources and power footprint, but doesn't affect execution time.

### 5.2.1.1.2 Idle power usage measurements
PADS and SPLEAT are spiking accelerators which use a sparse temporal information coding. That is, the circuit is in idle state between two spikes. Consequently, the overall dynamic power usage is computed as a weighted average of active and idle power consumptions in the spiking hardware inference simulator (Subsection 5.2.3.2). Hence we measure both active and idle power consumption for spiking architecture. That is, using two distinct testbenches: one with regular input stimuli, and the other with empty signals.

### 5.2.1.2 3D interpolation

All the raw results of the aforementioned measurement campaign are available in tables in Appendix 2. As explained previously, those results are used in estimation through interpolation. For example, in order to obtain the LUT usage for a PADS FC layer with a given configuration, all LUT measurements for PADS FC design space are placed on a regular grid and interpolated using the python Scipy Interpolate library. This mechanism is fully automated through a python software which retrieves the data in the XML files, performs interpolation and retrieves the result. The logic resource estimations for all architectures, as well as dynamic power for VGT and C-HLS are directly obtained through interpolation. For SPLEAT and PADS, inference simulation is required to obtain accurate estimations based on interpolated active and dynamic power. In this subsection, we show the 3D surface graphs resulting for interpolation of all measurements. Those results provide interesting preliminary insights on the design space exploration. For that purpose, they are presented in three sets of comparisons:

* A comparison between SPLEAT and C-HLS, to visualize the influence of spiking domain on sequential architectures

* A comparison between PADS and VGT, for the same influence on parallel architectures

* A comparison between SPLEAT and PADS, for the influence of parallelism on spiking domain in hardware

In this section, we limit our scope to Fully-Connected layers. First because PADS only covers FC layers, and second because the results are similar for convolution and pooling layers in other architectures. Moreover, the number of graphs and discussions would be two great covering all three types of layers.
Look Up Tables

<figure><img src="image_48.png" /></figure>

Registers (FFs)

<figure><img src="image_49.png" /><figcaption>Figure 5.3: 3D interpolations of LUTs (left) and FFs (right) for Fully-Connected layers. First row: SPLEAT (red) vs HLS (blue), second row: PADS (red) vs VGT (blue). Third row: SPLEAT (red) vs PADS (blue).</figcaption></figure>
5.2.1.2.1 **Look-Up-Tables & Registers** The results for LUTs and Registers (Flip-Flops, FF) are shown in Figure 5.3. LUTs and FFs are the two basic logic elements of the Programmable Logic. Every design uses some, which is not the case of more specialized elements like RAM or DSP. Moreover, RAM and DSPs can be synthesized in LUTs and FFs when the dedicated elements are saturated. Thus, LUTs and Registers usage is a global indicator of FPGA occupation. The interpolation surfaces for LUTs (left) and FFs (right) are shown as 3D surfaces. The first row depicts SPLEAT against C-HLS, the second depicts PADS against VGT, and the third SPLEAT vs PADS.

5.2.1.2.2 **Memory & DSPs** The 3D surface plots for Block RAM (BRAM) and DSP usage in Fully-Connected layers are provided in Figure 5.5. RAM and DSPs are specialized logic elements, in contrast with LUTs and Flip Flops. DSP are used for multiplication-accumulation operations. Block Memory is used to store data mostly in sequential architectures. As explained in previous subsection, LUTs and FFs can be used instead of RAM and DSP when such scarce resources are saturated. This could affect logic occupation and power usage. Thus, those two resources are also valuable indicators of FPGA occupation.

5.2.1.2.3 **Power** The power interpolation results are divided in two metrics: active and idle power usages. Active power corresponds to a situation where the architecture is receiving and processing input stimuli. On the other hand, idle power corresponds to an architecture in idle state. Active power is measured for all architectures, but idle power is only considered in spiking architectures. Indeed, only spiking architecture are susceptible of being in idle state during some part of processing. The results of interpolation on active power measurement are given in Figure 5.6. The interpolation of idle power on fully-connected spiking layers are given in 5.4. In idle power figures, active power is also represented for comparison.

The results shows that idle power is very similar to active power in all cases. Thus, the design does not benefit from the event-based coding in terms of energy consumption. Power-gating or clock-gating techniques might be used to nullify power usage in idle state. This possibility will be explored further in outlooks and discussions. The results presented in this section give a global view resource and power usages in neuromorphic accelerators depending on parallelism and coding domain. If sequential accelerators might seem more scalable at first, this is not guaranteed for larger topologies because of RAM saturation. However, SPLEAT demonstrates drastically lower resource and power usages than PADS on current design spaces. On the other hand, SPLEAT uses a little more resources and power than C-HLS, although the difference is not significant in front of the available resources. PADS demonstrate both lower power and resource usage than VGT across all design space. Moreover, the metrics increases faster for VGT. Thus, PADS is more scalable than VGT among parallel implementation both in terms of logic resources and power usage. Finally, idle power in spiking architecture is very close to active power. Thus, no major power savings should be expected from the asynchronous behavior of PADS and SPLEAT.

## 5.2.2 Execution time

In this section, we describe how are made estimations of inference time. The method is different for each architecture.
<figure><img src="image_50.png" /><figcaption>Figure 5.4: 3D interpolations of active and idle power for Fully-Connected layers. Left: SPLEAT, right: PADS.</figcaption></figure>
<figure><img src="image_51.png" /><figcaption>Figure 5.5: 3D interpolations of Block RAM (left) and DSPs (right) for Fully-Connected layers. First row: SPLEAT (red) vs HLS (blue), second row: PADS (red) vs VGT (blue). Third row: SPLEAT (red) vs PADS (blue).</figcaption></figure>
ACTIVE POWER

<figure><img src="image_52.png" /><figcaption>Figure 5.6: 3D interpolations of Active Power for Fully-Connected layers. First row: SPLEAT (red) vs HLS (blue), second row: PADS (red) vs VGT (blue). Third row: SPLEAT (red) vs PADS (blue).</figcaption></figure>
### 5.2.2.1 C-HLS inference time interpolation

C-HLS is fully sequential and does not feature inter-layer pipeline. That is, the execution time of a layer only depends on its configuration. Hence, execution time in C-HLS is dealt with as resource estimations: it is measured at each measurement point in the design spaces, and interpolated using the same Scipy Interpolate python library. Moreover, the total execution time is the sum of layer-wise results, as no pipeline is involved.

### 5.2.2.2 VGT behavioral simulation

VGT features inter-layer pipeline, therefore the behavior of previous layers affects the following ones. Consequently, the inference time of a layer does not only depend on the configuration of the layer, but on the configuration of all previous layers. A simple python software is thus developed for high-level behavioral simulation of VGT. The simulator reproduces the behavior of VGT layers and inter-layer pipeline without performing the actual computations. It has been developed by analyzing the VHDL code and studying behavioral simulations of the architecture in Modelsim software. Moreover, the simulator has been validated on several cases by comparison with Modelsim behavioral simulations. It is used to provide accurate layer-wise and network inference time estimations for VGT.

### 5.2.2.3 Spiking architectures: single spike execution time

In spiking architectures, inference time is estimated using a dedicated spiking inference simulator. The simulator is based on two elements: the synaptic traces obtained from N2D2 spiking inference, and the single spike execution time. The simulator will be detailed later in Subsection 5.2.3. In this paragraph, we provide information on those single spike execution times. That is, the time required to process a single input spike in either SPLEAT or PADS. In order to obtain equations that define execution time with respect to layer configuration, the VHDL code of SPLEAT and PADS is analyzed in depth. The analysis yields to Equations 5.1 and 5.2 for execution times (in clock cycles) of SPLEAT and PADS respectively.

$$ 
\begin{aligned}
\delta t_{\text{SPLEAT Conv}} &= 4 + \left(\frac{K}{S}\right)^2 \times N_K \\
\delta t_{\text{SPLEAT Pool}} &= 1 + K^2 \\
\delta t_{\text{SPLEAT FC}} &= 3 + N_{\text{out}}
\end{aligned}
\qquad (5.1) $$

With $\delta t$ the processing times (in clock cycles) for a single spike, $K$ the kernel size (five for convolutions, two for max-pooling), $S$ the stride (one for convolutions, two for max-pooling), $N_K$ the number of kernels, and $N_{\text{out}}$ the number of output neurons. For a better understanding of those equations, more material on SPLEAT is available in [14].

$$ \delta t_{\text{PADS FC}} = 2 + \log_2(N_{\text{input}}) \qquad (5.2) $$

With $\delta t$ the processing time (in clock cycle) for a single spike and $N_{\text{input}}$ the number of input neurons. This equation directly illustrates the process of the Generic Neural Processing Unit (NPU) of PADS shown in Figure 3.4 (Section 3.1.2). Indeed, $\log_2(N_{\text{input}})$ is the depth of the adder tree in the Generic NPU. The 2 additional clock cycles are caused by the threshold & firing process.
<figure><img src="image_53.png" /><figcaption>Figure 5.7: a) Simple spike trace example with 4 input synapses and 4 time increments, b) Illustration of PADS parallel and pipelined process.</figcaption></figure>

The two equations are verified on several measurement points using Modelsim behavioral simulation before the measurement campaign. Both equations are integrated in python functions to be used later in the spiking hardware inference simulator (Section 5.2.3).

### 5.2.3 Spiking hardware inference simulator

In this subsection, we describe the spiking hardware inference simulator used to estimate inference time and dynamic power usage of spiking architectures. It is based on spike traces derived from N2D2 synaptic logs, single spike processing times, as well as active and idle power interpolations.

#### 5.2.3.1 Inference time

Inference time estimation in SPLEAT and PADS is based on the same spike trace used by the SAR estimator in Section 4.4.1. The spike trace is converted from N2D2 format to a unified format. That is, each layer is associated with a spike trace in the form of a 2D matrix of size N×M. N is the number of input synapses, and M the number of time increments. Indeed, time is discretized in increments of one clock cycle (10ns at 100MHz). As a reminder, N2D2 spiking inference is configured to mimic the spike generation process of the GenCell. Thus N2D2 synaptic activity effectively represents the real input stimuli of hardware spiking layers, as it involves the exact same network with identical hyper-parameters. A simple spike trace with 4 output synapses and 4 time increments (i.e. 4 clock cycles) is provided in Figure 5.7a. The inference time estimation is also based on the single spike processing times ($\delta t_{\text{PADS}}$) and $\delta t_{\text{SPLEAT}}$ obtained in 5.2.2. The estimation protocol is different for PADS and SPLEAT, as described below.

##### 5.2.3.1.1 Simulation for PADS
PADS is a fully-parallel pipelined architecture, therefore it is able to receive up to one spike per synapse and per clock cycle. In other words, PADS is able to process one spike trace column per clock cycle. The pipeline process is illustrated in Figure 5.7b. In this Figure, $\delta t_{\text{PADS}}$ = 3 clock cycles, i.e. the pipeline is three stage deep. A python simulator
<figure><img src="image_54.png" /><figcaption>Figure 5.8: Illustration of SPLEAT sequential process.</figcaption></figure>

is developed to compute execution time based on spike traces derived from N2D2. Knowing the values of $\delta_{t\text{CLK}}$ (10ns at 100MHz) and $\delta_{t\text{PADS}}$ (Section 5.2.2), it is able to provide accurate time estimations from high-level spike trace information.

### 5.2.3.1.2 Simulation for SPLEAT
SPLEAT on the other hand is a highly sequential, with no intra-layer pipeline. Hence, spikes must be processed one by one. SPLEAT is thus able to process at most one spike-trace cell per clock cycle. SPLEAT is event-based, thus empty cells are not processed. The ensuing process is illustrated in Figure 5.8. Like for PADS, a python simulator is developed to compute inference time based on spike traces, $\delta_{t\text{CLK}}$ and $\delta_{t\text{SPLEAT}}$ values. In the figure, $\delta_{t\text{SPLEAT}} = 3$, like in PADS figure

Finally, SPLEAT and PADS are both event-based architectures which feature inter-layer pipeline. The propagation of spikes in the network is neglected in front of the actual processing time, thus we consider that all layers are active simultaneously. In consequence the network-level execution time is computed as the maximum layer-wise execution time in both PADS and SPLEAT. In other words, the network-wise inference time is the maximum layer-wise inference time (among all layers of the architecture).

### 5.2.3.2 Power estimator

In FPGA, power consumption is divided between static and dynamic components. Dynamic power consumption ($P^{\text{Dynamic}} = P^{\text{run}}$) is the power consumed by signal switches inherent to the processing state. Static power consumption is the power consumed by the designed when powered on, but without any clock or signal input. Static power consumption is composed of the board static consumption ($P^{\text{empty}}$), and the design static consumption ($P^{\text{idle}}$). This FPGA energy model was proposed in [105] and illustrated in Figure 5.9a. Moreover, let $P^{\text{Static}} = P^{\text{empty}} + P^{\text{idle}}$. Experiments show that $P^{\text{idle}}$ is negligible in front of $P^{\text{empty}}$, thus we consider $P^{\text{Static}} = P^{\text{empty}}$. The results of such experiment is illustrated in Figure 5.9b. It shows evolution of dynamic and static power consumption against FPGA occupation. Static power consumption is not affected by design size, thus $P^{\text{Static}} = P^{\text{empty}}$. Thus, in the model $P^{\text{Static}}$ is considered as an offset which depends on the device. For the Zedboard that is 0.250W.

For formal architectures (VGT and C-HLS), dynamic power consumption depends only on configuration. It is thus directly obtained through interpolation of the power measurements database. For spiking architectures on the other hand (PADS and SPLEAT), power consumption depends on active power consumption, idle power consumption and synaptic activity. As a reminder, active power consumption corresponds to the Neural Processing Unit (NPU) state when it is busy processing spikes. Idle power corresponds to the NPU state when awaiting input stimuli. Both have been measured for SPLEAT and PADS in Section 5.2.1.2.3. Dynamic power estimation is computed as an average between active and idle state. The python code for inference time esti-
<figure><img src="image_55.png" /><figcaption>Figure 5.9: a) FPGA power consumption model, b) Dynamic and Static power VS LUT occupation (%) on Zedboard</figcaption></figure>

mation is reused to count the number of active and idle clock cycles during processing. Like for inference time, PADS and SPLEAT dynamic power consumption is estimated separately. Based on an input spike trace, the simulator outputs the proportion of time passed in active ($n_{active}$) and idle ($n_{idle}$) states, in number of clock cycles. Dynamic power estimation is computed according to Equation 5.3. It should be noted that $n_{active} + n_{idle}$ is equal to $\Delta t_{PADS}$ or $\Delta t_{SPLEAT}$ for PADS or SPLEAT respectively.

$$P^{\text{Dynamic}} = \frac{n_{\text{active}} \times P^{\text{active}} + n_{\text{idle}} \times P^{\text{idle}}}{n_{\text{active}} + n_{\text{idle}}} \qquad (5.3)$$

## 5.3 Resource estimations

The estimation framework is applied to the benchmark of seven datasets: MNIST, OPS-SAT, GTSRB, CIFAR-10, Mines VS Rocks, Spoken Digits and RadioML 2018. All raw estimation results are available in Appendix .2. In this document, we focus on two interesting cases: OPS-SAT and Spoken Digits tasks. This section is dedicated to the logic resource estimations of the four accelerators (VGT, PADS, C-HLS and SPLEAT) on the benchmark of datasets. That is Look-Up-Tables (LUTs), Registers (Flip-Flops, FFs), Block RAMs (RAMs) and Digital Signal Processors (DSPs) estimations. Before going into further details, a few notation should be introduced to simplify explanations. The estimations are noted $Metric_{Arch}^{Obj}$, where:

* $Metric_{VGT}^{Obj}$ is the considered resource (LUT,FF,DSP,RAM),

* $Obj$ is the considered object, it can be a full CNN (MNIST, OPS-SAT...), a sub-part of the CNN (CL: classification, FE: feature extraction) or a specific layer (C1: first convolution, P2: second pooling, FC1: first fully-connected),

* $Arch$ is the considered architecture (PADS,VGT,SPLEAT,C-HLS).

Therefore, $LUT_{VGT}^{MNIST}$ refers to the LUT usage the full MNIST CNN implemented with VGT. $RAM_{PADS}^{GTSRB-FC2}$ refers to the RAM usage of the second FC layer of GTSRB implemented with PADS. Consequently, the ratio between two estimations is noted $\frac{Metric_{Arch}^{Obj}}{Metric_{Arch}^{Obj}}$. It is used to describe savings or overheads between two architectures.

Moreover, the estimation results discussed through three sets of comparisons: PADS vs. VGT, SPLEAT vs. C-HLS and SPLEAT vs. PADS. The two main goals of the study are indeed the comparison of formal and spiking domains and the influence of parallelism on spiking accelerators.
<figure><img src="image_56.png" /><figcaption>Figure 5.10: Layer-wise LUT estimation in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</figcaption></figure>

5.3.1 Layer-wise estimation

First, we provide layer-wise resource estimations. Figure 5.10 features the layer-wise LUT usage for the four architectures on OPS-SAT and Spoken Digits datasets. It should be noted that FF usage is strongly correlated with LUT usage, thus it is not shown here. The FF usage graph is available in Appendix .2. Figure 5.11 shows the layer-wise RAM and DSP usage.

Estimations yields that $LUT_{SPLEAT}^{Conv} \approx LUT_{C-HLS}^{Conv}$. However, it also shows that $LUT_{SPLEAT}^{Pool} \approx 1.5 \times LUT_{C-HLS}^{Pool}$. Therefore, SPLEAT is not suited to max-pooling layers. The estimations are coherent with the literature, as transcoding max-pooling layers towards spiking domain is widely accepted as challenging [106]. In literature, authors propose to incorporate pooling operation in convolution layers. That is, by using a stride of two [14] or specific synaptic weight matrices [107]. Such a mechanism could thus benefit to spiking CNN hardware implementations. Additionally, $LUT_{SPLEAT}^{FC} \approx 2.5 \times LUT_{C-HLS}^{FC}$. Thus, SPLEAT uses more LUTs in both Pool and FC layers. This is explained by low-level differences between SPLEAT and C-HLS. The latter is automatically generated from synthetisable C code. Pragmas ensure the maximum level of multiplexing during synthesis. SPLEAT on the other hand is designed by hand and does not feature such
<figure><img src="image_57.png" /><figcaption>Figure 5.11: Layer-wise RAM and DSP estimation in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</figcaption></figure>

low-level optimizations. The difference in implementation choices explains the difference between expectations and estimations. Indeed, the literature reports up to 50% resource savings [88] when using spiking rather than formal acceleration. Figure 5.11 shows that RAM and DSP usage is similar in C-HLS in Convolution and FC layers. Similarly to LUT estimations, that is not true for max-pooling layers. Indeed, C-HLS max-pooling layers mostly use RAM whereas SPLEAT max-pooling layers mostly use DSPs. In both cases, usage remains very low and is far from reaching the board capacity.

The comparison between PADS and VGT (FC layers) yields that $LUT_{\text{VGT}}^{\text{OPSSAT-FC}} \approx LUT_{\text{VGT}}^{\text{OPSSAT-FC}}$. However, $LUT_{\text{VGT}}^{\text{Spoken-FC1}} \approx 2.5 \times LUT_{\text{PADS}}^{\text{Spoken-FC1}}$ and 1.2 for FC2. Hence, PADS enables resource savings over VGT for large layer sizes. This is coherent with the 3D plots of LUT interpolation surface shown in Figure 5.3 (first column, second raw), which shows that VGT resource usage increases faster than PADS. The savings offered by PADS are even greater regarding DSP usage. Indeed, VGT uses a lot of DSPs whereas PADS uses none. In Spoken Digits CNN, both FC1 and FC2 reach the maximum DSP capacity of the board on their own (Zedboard has 220 DSPs available). The saturation of DSP by VGT is responsible for the LUT overhead observed in large FC layers. In all, PADS offers 100% reduction of DSP usage which is a critical and scarce resource in FPGAs, and 2.5 reduction of LUT usage in large FC layers. This is thus an interesting
<figure><img src="image_58.png" /><figcaption>Figure 5.12: Comparisons of resource usage in VGT, HLS, PADS and SPLEAT. Left: OPSSAT, Right: Spoken Digits. Top: Full network, Bottom: Classification Stage</figcaption></figure>

feature for embedded systems in which size is critical.

Lastly, the comparison between PADS and VGT resource estimations yields that SPLEAT LUT usage is far below that of PADS. This is quite straightforward due to the very different level of parallelism. Moreover, PADS resource usage strongly depends on layer size. This is not the case for SPLEAT. Thus, the resource overhead of PADS is greater in larger layers: $LUT_{PADS}^{Spoken-FC1} \approx 1700 \times LUT_{SPLEAT}^{Spoken-FC1}$. In the much smaller OPSSAT-FC2 layer, the ratio is only 4. That is, SPLEAT is preferable over PADS regarding LUT usage. As a counterpart, PADS reduces Block RAM usage by 100% compared to SPLEAT. However, SPLEAT BRAM usage is negligible in front of the available resource, so that PADS savings are not significant.

### 5.3.2 Network-wise estimation

After discussing layer-wise resource results, we propose to compare accelerators at network level. The results provided in Figure 5.12 are obtained by summing the layer-wise estimations. It should be noted that effect of DSP and BRAM saturation is not taken in account here. Therefore, the total DSP and BRAM usage may overpass the board limit in some cases (Zedboard). This approximation is the same for all architectures, thus the results remain comparable. Improvements regarding the acknowledgment of DSP and BRAM saturation will be detailed in the outlooks of this Chapter (Section 5.8).

The top row of Figure 5.12 features the resource usage estimations for full CNNs. PADS does not cover convolution and pooling layer therefore it does not appear in those results. The bottom row features resource estimation for the fully-connected stages including PADS. The network-
wise estimations confirm the conclusions derived from layer-level results. In general, we observe $LUT_{SPLEAT} \approx 2 \times LUT_{C-HLS}$. The network wise DSP and Block RAM usage are equivalent in C-HLS and SPLEAT in both OPS-SAT and Spoken Digits datasets. The classification-stage level estimations for VGT and PADS also confirms the layer-wise results: PADS offers 50% LUT saving in Spoken Digits, and 100% DSP savings in both datasets. PADS and VGT feature very similar low-level implementation choises, except for the coding domain. Hence, those results confirm that spiking domain provides reliable resource savings in highly parallel architectures, especially for large network sizes. Lastly, the results also confirm the conclusions derived from the comparison of SPLEAT and PADS at layer level. That is, SPLEAT uses much fewer resources than PADS, especially in larger FC layers.

### 5.3.3 Conclusions

As explained earlier, fully-parallel implementations are unrealistic for state-of-the-art CNN im-
plementations. For simple tasks implying small topologies, high-level of parallelism is however
conceivable. In this context, spiking implementations could bring substantial resource savings.
That is mitigating size and weight in the system. In satellite applications such as OPS-SAT cloud
segmentation [93], small CNNs are sufficient for acceptable accuracy, and size is one of the most
limiting factors. Such application could therefore benefit from spiking highly-parallel implemen-
tations instead of formal ones. The same conclusions can be made for other niches applications,
like object tracking in drones [108] also involves small CNNs and drastic size constraints. Such
applications could also benefit from spiking highly-parallel acceleration. In both cases, one might
expect to reduce LUT usage by 50% and DSP usage by 100%. Thus, smaller FPGAs can be used
for the same task, reducing weight and size of the system. Furthermore, the resource estimations
have confirmed that max-pooling operation is challenging for spiking domain. Indeed, SPLEAT
uses ten times more resources than C-HLS for max-pooling layers. Such layers might be integrated
to convolution operations as proposed in literature [14] [106].

## 5.4 Inference time and power estimations

In this section, we describe the inference time and power estimations for the four accelerators on
OPS-SAT and Spoken Digits datasets. Like for other results, the estimations for all datasets of the
benchmark are available in Appendix 2. Like for resources, we begin with layer-wise estimations
and give network level results afterwards. The same notations are used: Metric<sub>Arch</sub><sup>Obj</sup>, where Metric
可以 be either *T* for time and *P* for power consumption. Additionally, *DP* and *SP* stands for
dynamic and static power respectively. Like for resource results, estimations are discussed by
comparing SPLEAT to C-HLS, PADS to VGT and SPLEAT to PADS. Before going into further
details, the layer-wise and total SAR obtained on OPS-SAT and Spoken Digits are reminded in
Figure 5.13. For OPS-SAT, formal test accuracy is 91.2% and spiking test accuracy is 91.1%
(Δ = 4). For Spoken Digits, formal accuracy is 88.55% and spiking is 87.5% (Δ = 20).

### 5.4.1 Layer-wise estimation

The layer-wise inference time estimations are depicted in Figures 5.14. For spiking accelerators,
the results are averaged on 100 images. Indeed execution time of spiking accelerators depends on
<figure><img src="image_59.png" /><figcaption>Figure 5.13: Layer-wise and total SAR for a) OPS-SAT ($\Delta = 4$) and b) Spoken Digits ($\Delta = 20$).</figcaption></figure>

synaptic activity, i.e. on the input sample. The estimations are obtained with MinPeriod = 1 timesteps and MaxPeriod = 1000 timesteps.

As expected, C-HLS is much slower than SPLEAT for convolutions and FC layers. C-HLS convolutions are a hundred to a thousand times slower than SPLEAT. For FC layers, that is three to a hundred times slower. That is the counterpart of the resource savings brought by the intense multiplexing of C-HLS architecture. It should be noted that SPLEAT inference time is affected by SAR. Indeed the ratio of inference time $\frac{T_{\text{Conv\&FC}}^{\text{C-HLS}}}{T_{\text{SPLEAT}}^{\text{Conv\&FC}}}$ is greater for layers with low SAR (Figure 5.13). In other words, SPLEAT convolutions and FC are always faster than C-HLS and even more for low layer-wise SAR. Thus, the estimations confirms that SPLEAT inference time is coherent with layer-wise SAR. For max-pooling layers, SPLEAT is not always faster: On Spoken Digits, SPLEAT is three to ten times slower than C-HLS. This observation confirms the incompatibility of max-pooling operation and spike coding.

The comparison between PADS and VGT yields that PADS is slower than VGT in all studied layers. In both FC1 layers, PADS is two times slower. In FC2 layers, that is five times. Therefore, the resource savings of PADS FC layers are made at the expense of inference time. In Figure 5.13, $SAR^{\text{OPSSAT-FC}} < 1$. That is, there are fewer synaptic operations in the SNN than there are in the FNN. In the light of this metric, one might expect PADS to be faster in OPS-SAT FC layers. However, the temporal sparsity of spikes is not taken in account in SAR. This sparsity is intrinsic to rate encoding but strengthen by the Spike Generation Cell design. In consequence, PADS is slower than VGT even when SAR<1. Moreover, this time the inference time ratios are not consistent with SAR trends. That is, $\frac{T_{\text{Spoken-FC2}}}{T_{\text{Spoken-FC1}}}$ is greater than $\frac{T_{\text{Spoken-FC1}}}{T_{\text{Spoken-FC2}}}$ although $SAR^{\text{Spoken-FC1}} > SAR^{\text{Spoken-FC2}}$. That is also caused by the temporal sparsity of spikes, which is non deterministic and not taken in account by the SAR metric.

The comparison between PADS and SPLEAT yields interesting results to understand the influence of parallelism on rate-coded SNN accelerators. Indeed, SPLEAT and PADS have similar inference time for all FC layers except Spoken-FC1. That is surprising, as the parallelism and pipeline of PADS should enable a drastic speed-up. As shown in Figure 5.13, $SAR^{\text{Spoken-FC1}}$ is a lot higher than in other layers. When SAR is low, SPLEAT is able to process spikes on-the-fly, i.e. before the next spike arrives. In such condition, the parallelism and pipeline of PADS are useless, and the two architectures have the same inference time. On the other hand, things changes for higher SAR since spikes arrive much more frequently or even simultaneously at the input of
<figure><img src="image_60.png" /><figcaption>Figure 5.14: Layer-wise execution time (per image) estimation in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</figcaption></figure>

the layer. Thus, SPLEAT is not able to process spikes as they arrive, hence the observed time overhead. Under such conditions, the parallelism of PADS is better exploited and the architecture is much faster than SPLEAT. This observation deeply strengthen the role of the SAR metric: it can be used to tune parallelism at layer level in spiking architectures. Using lower level of parallelism when SAR is low can bring substantial resource savings (Section 5.3) without time overhead.

The layer-wise dynamic power estimations are provided in Figure 5.15. At this stage, only dynamic power is taken in account. Static power only makes sense at network level: it is related to the whole design and board. Estimations demonstrates that SPLEAT dynamic power usage is lower than C-HLS in all layers of both CNNs, except Spoken Digits-FC1. That is, even in max-pooling layers which caused resource and time overhead. $DP_{SPLEAT}$ is 2 to 4 times lower than $DP_{C-HLS}$, except in Spoken Digits FC1 where $DP_{SPLEAT} \approx 2 \times DP_{C-HLS}$. Overall, SPLEAT provides greater power savings over C-HLS when SAR is low. This is coherent with previous observations on inference time: a low SAR means that SPLEAT latency is masked by the temporal sparsity of spikes. In such conditions, SPLEAT is often idle and waiting for inputs. On the other
<figure><img src="image_61.png" /><figcaption>Figure 5.15: Layer-wise Power estimation in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</figcaption></figure>

hand, C-HLS processes pixels without interruption, thus the dynamic power is higher. Additionally, C-HLS power overhead in pooling-layers can be attributed to the Block RAMs. In all, highly sequential spiking accelerators provides substantial power savings over formal ones. Those savings are correlated to layer-wise SAR.

The second comparison yields that PADS provides significant dynamic power savings over VGT. That is, $DP_{PADS}$ is two three times smaller than $DP_{VGT}$. That is, the time overhead of highly-parallel spiking accelerators is compensated by dynamic power savings. Moreover, the power savings are also correlated with SAR. That is, $\frac{DP_{VGT}}{DP_{PADS}}$ follows the same trend as SAR.

Lastly, the comparison between SPLEAT and PADS confirms the discussions on computation density. SPLEAT dynamic power usage is drastically lower than PADS as expected from the parallelism. Moreover, $\frac{DP_{PADS}}{DP_{SPLEAT}}$ is higher for higher SAR: 60 in Spoken-FC1 (SAR=16) and only 2.3 in OPSSAT-FC2 (SAR=0.8). The computation density explains such results: as explained, PADS processes spikes in parallel when SAR is high. Such high computation density increases dynamic power usage, which is visible in the results. This observation strengthen our statement that the SAR metric is a crucial tool for layer-level tuning of parallelism in spiking accelerators.
<figure><img src="image_62.png" /><figcaption>Figure 5.16: Comparisons of inference time (per image) and power usage in VGT, HLS, PADS and SPLEAT. Left: full network, Right: classification stage. Top: time, Bottom: power</figcaption></figure>

In doing so, one is not only able to optimize inference time for resource usage, but also dynamic power.

### 5.4.2 Network-wise estimation

After studying layer-wise results, inference time and power estimations are provided at network level in Figure 5.16.

Network-wise inference time estimations are depicted in the top row of Figure 5.16. Like for resource results, the figure is divided between full CNN estimations (left) and classification stage estimations (right). Inference time estimations are discussed first and power will be addressed after. The comparison of SPLEAT and C-HLS yields that SPLEAT is even faster at network-level. Indeed, SPLEAT features an inter-layer pipeline whereas C-HLS does not. SPLEAT is 13,000 times faster on OPS-SAT, and 6,500 times on Spoken Digits. The trend is consistent with SAR and previous observations: SAR is higher in Spoken Digits, thus spike density is higher and SPLEAT is not able to process all spikes on-the-fly. The comparison between PADS and VGT are also consistent with layer-wise results. On OPS-SAT PADS classification stage is 1.5 times slower than VGT. On Spoken Digits that is 4.9 times. Like for sequential architectures, this is coherent with the SAR metric. Lastly, the comparison of PADS and SPLEAT yields that SAR is also interesting for tuning parallelism at network-level. Indeed, for OPS-SAT network-level times are equivalent. For Spoken Digits where SAR is higher, PADS is a hundred time faster. In such context parallelism is required to mitigate latency in high SAR applications but useless otherwise.

The network-wise power estimations are depicted in the bottom row of Figure 5.16. In contrast
with layer-wise results, the graphs feature the static power of the Xilinx Zedboard (0.250 W). Like for time estimations, the network-level power is coherent with aforementioned layer-wise results: spiking domain enables substantial power savings over formal domain. SPLEAT reduces dynamic power by a factor two compared to C-HLS, and PADS enables 50% savings over VGT. Is should be noted that PADS power savings are slightly improved in OPS-SAT, i.e. low SAR. However, the savings are negligible in front of the common static power offset. That is, the overall power usage (dynamic+static) will remain unchanged. This is true for FPGA where static power is high. However, static power is much lower in ASICs. Such systems could therefore highlight the power savings provided by spiking domain. Lastly, the comparison between PADS and SPLEAT is consistent with the layer-level discussions: SAR is higher in Spoken Digits, thus computation density in PADS is higher. That explains the increased power consumption difference in Spoken Digits. However, the differences are also negligible in front of static power.

### 5.4.3 Conclusions

The study of time and power estimations yields several strong conclusions. The layer-wise results confirmed that max-pooling layers are challenging for spiking domain. SPLEAT max-pooling layers indeed suffer severe latency overhead compared to C-HLS when SAR is high. That is, despite the higher level of multiplexing found in C-HLS architecture. However, SPLEAT is much faster than C-HLS overall (factor 6500 to 13000), all while reducing dynamic power usage by a factor two. The differences in processing time is mostly due to low-level implementation choices, and the power savings are small in front of the static power offset, which mitigates the proposed results. On the other hand, PADS is slower than VGT (factor 1.5 to 5) due to the temporal sparsity of spikes. This temporal sparsity is intrinsic to rate encoding, but increased by the current sequential spike generation process. Reducing spike temporal sparsity is therefore crucial to compete with highly parallel formal accelerators. Such improvement will be detailed in the outlooks of this chapter. Regarding power, PADS also provides savings which are small in front of the static power offset.

Moreover, the results discussed in this section strengthen the role of the SAR metric. Indeed, it can be used to tune parallelism at both layer and network-level. That is, high level of parallelism should be used to mitigate timing when SAR is high. Otherwise, parallelism is useless and does not provide speed-up. For example, using SPLEAT instead of PADS in OPSSAT Classification stage does not imply latency overhead, but provide 90% savings in resource and dynamic power. Realistic architectures should feature intermediate level of parallelism, which can be tuned using SAR. Further work should be undertaken to quantify the relation between SAR and parallelism in spiking accelerators.

## 5.5 Energy estimations

Finally, timing and power are combined to provide energy estimations ($E = P \times \Delta T$). In this section, we proceed like for other estimations: layer-wise estimations are addressed first, followed by network-level results. Energy is widely accepted as the most important and universal limiting factor for embedded systems. The results of this section are thus crucial for the cartography of applications and neural coding domains. In this section, the same notations are used to simplify explanations. The estimations are noted $E_{Arch}^{Obj}$, where E stands for energy consumption. It should be noted that static power is taken in account in network-wise energy estimations, but not in layer-wise.
<figure><img src="image_63.png" /><figcaption>Figure 5.17: Layer-wise energy estimation (per image) in VGT, HLS, PADS and SPLEAT. Top: OPSSAT RGB, bottom: Spoken Digits</figcaption></figure>

5.5.1 Layer-wise estimation

Layer-wise energy estimations for VGT, C-HLS, PADS and SPLEAT on OPS-SAT (above) and Spoken Digits (below) datasets are provided in Figure 5.17. The comparison of SPLEAT and C-HLS yields that the first provide drastic power savings in Conv and FC layers. That is, $E_{SPLEAT}^{Conv}$ is forty to two thousand times lower than $E_{C-HLS}^{Conv}$. In FC layers, energy is reduced by a factor 2 to 200. Moreover, SPLEAT energy savings are higher for low layer-wise SAR. That is, $\frac{E_{SPLEAT}^{Conv \ \& \ FC}}{E_{C-HLS}^{Conv \ \& \ FC}}$ is loosely proportional to the layer-wise SAR. On the other hand, pooling layers imply an energy consumption overhead in Spoken Digits CNN. Thus, the power savings observed in SPLEAT pooling layers over VGT are not significant enough to compensate the latency overhead. Those estimations confirm that max-pooling layers should be avoided in spiking architectures, particularly when SAR is high.

In highly parallel architectures, the layer-wise energy savings provided by PADS is less straightforward. According to Figure 5.17, PADS and VGT have nearly identical energy consumption in FC1$^{OPS-SAT}$. In FC2$^{OPS-SAT}$, PADS uses 2 times more energy than VGT. Hence PADS does
<figure><img src="image_64.png" /><figcaption>Figure 5.18: Comparisons of energy consumption per image in VGT, HLS, PADS and SPLEAT. Left: full network, Right: classification stage. Top: power, Bottom: timing</figcaption></figure>

not provide reliable-energy savings over VGT despite the low layer-wise SAR measured in OPS-SAT. In Spoken Digits, $E_{\text{PADS}}^{\text{Spoken-FC1}}$ is 20% lower than $E_{\text{VGT}}^{\text{Spoken-FC1}}$, whereas $E_{\text{PADS}}^{\text{Spoken-FC2}}$ is three times higher than $E_{\text{VGT}}^{\text{Spoken-FC2}}$. More than a confirmation that PADS does not provide reliable energy savings at layer-level, this second result is incoherent with SAR measurements. That is, $SAR^{\text{Spoken-FC1}} > SAR^{\text{Spoken-FC2}}$, but $\frac{E_{\text{PADS}}^{\text{Spoken-FC1}}}{E_{\text{VGT}}^{\text{Spoken-FC1}}} < \frac{E_{\text{PADS}}^{\text{Spoken-FC2}}}{E_{\text{VGT}}^{\text{Spoken-FC2}}}$. Therefore, the SAR is not correlated to layer-wise energy ratios in highly-parallel architectures.

Lastly, the layer-wise comparison of PADS and SPLEAT yields that the first is only preferable to the second when SAR is high, in terms of energy consumption. When SAR is low (OPSSAT-FC1&FC2 and Spoken-FC1), SPLEAT consumes 2.3 to 9 times fewer energy than PADS. On Spoken-FC2 on the other hand (high SAR), SPLEAT consumes 2.3 times more energy than PADS. This observation confirms the role of SAR in determining parallelism at layer-level for resource, time, power and energy optimization.

### 5.5.2 Network-wise estimation

The network-level estimations are provided in Figure 5.18. Like for previous estimation metrics, the results are split between full-network (top) and classification stage (bottom). The comparison between C-HLS and SPLEAT confirms that the latter provides drastic energy savings over the first. That is, $E_{\text{HLS}}^{\text{OPSSAT}} \approx 1000 \times E_{\text{SPLEAT}}^{\text{OPSSAT}}$ and $E_{\text{HLS}}^{\text{Spoken}} \approx 25 \times E_{\text{SPLEAT}}^{\text{Spoken}}$. Those ratios are consistent with the network-level SAR measurements (0.4 in OPS-SAT and 8.3 in Spoken-Digits). That is, SPLEAT advantage is greater for low SAR.
In the field of highly parallel architectures, PADS uses more energy than VGT in both datasets. More precisely $E_{\text{PADS}}^{\text{OPSSAT}} \approx 1.3 \times E_{\text{VGT}}^{\text{OPSSAT}}$ and $E_{\text{PADS}}^{\text{Spoken}} \approx 3 \times E_{\text{VGT}}^{\text{Spoken}}$. Those ratios are consistent with the network-wise SAR, and indicates that PADS energy overhead is mitigated for low SAR. That is, the estimations confirm our hypothesis that spiking domain is better adapted to low SAR. The energy overhead is due to the inference time overhead in PADS, which is caused by the temporal sparsity of spike encoding. Hence, reducing temporal sparsity of spike encoding is crucial to compete with formal implementation in terms of energy. Moreover, those results are not consistent with the expected cartography based on SAR study (Section 4.4). The correlation between estimation-based cartography and SAR-based cartography will be discussed in details in Section 5.6, for both high and low level of parallelism.

Finally, the comparison between PADS and SPLEAT confirms the role of SAR in tailoring parallelism for spiking implementations. Indeed, SPLEAT uses fewer energy than PADS on OPSSAT where SAR is low (10%), but more on Spoken Digits where SAR is high (17 times more). That is, it is preferable to use SPLEAT than PADS on OPSSAT. That is not only for the slight energy savings but also for the drastic resource savings provided by SPLEAT over PADS. On the other hand, when SAR is high like in Spoken Digits, higher level is necessary to mitigate latency and energy.

### 5.5.3 Conclusions on energy estimations

Energy is widely accepted as the most limiting factor for embedded systems. The conclusions derived from the energy estimations are thus crucial for the cartography of applications and coding domains. The network-level estimations have shown that SPLEAT provides drastic energy savings over C-HLS, regardless of SAR. However, SPLEAT is even more advantageous for low SAR. In such conditions, energy consumption is reduced by a factor 1000. However, the conclusions are mitigated by the difference of low-level implementation in the two sequential architectures. The comparison is not exactly made with all other things equal, thus the savings might also come from architectural choices. Further experiments should be undertaken with more comparable sequential accelerators to confirm our findings. On the other hand, VGT and PADS are much more similar to each other, increasing our confidence in the following results. That is, PADS uses 1.3 to 3 times more energy than VGT on OPSSAT and Spoken Digits respectively. This overhead is caused by the temporal sparsity of spikes, due to the rate-encoding itself but also to the Spike Generation Cell implementation. This issue should be addressed urgently to leverage spiking domain energy benefits. For both datasets, the SAR-based cartography proposed in Section 4.4 is partially contradicted by energy estimations. This issue will be addressed in the following section (5.6).

Concerning layer-wise estimations, the results confirmed that max-pooling layers are challenging for SNNs, and must be avoided in spiking hardware implementations. That is, by using higher strides in convolution as proposed in the literature [14] [106]. Moreover, the results confirmed the role of SAR in tailoring parallelism, whether at layer or network level. Indeed, the highly parallel design of PADS is underused in low SAR cases, which creates unnecessary resource and power overhead without reducing inference time. On the other hand, parallelism is necessary to mitigate latency when SAR is high.
<figure><img src="image_65.png" /><figcaption>Figure 5.19: Confrontation of SAR model and energy estimations for 7 datasets of the benchmark: MNIST, OPSSAT, GTSRB, CIFAR-10, Mines VS Rocks, Spoken Digits and RadioML 2018. Validation is made separately for parallel and sequential architectures. Each subfigure is divided between SAR & λ values (top) and energy estimations (bottom).</figcaption></figure>
## 5.6 Validation of the SAR model

One of the goals of this chapter is also to confront the SAR energy model with energy estimations. In doing so, the goal is to evaluate the reliability of the metric, and its usability. More specifically, SAR does not take low-level implementation choices into account such as parallelism or pipeline. As a reminder, the SAR energy model intends to determine whether an application is suitable to spiking acceleration. It is interpreted as a ratio of energy consumption (modulo a constant $\lambda$) between equivalent formal and spiking applications. The model is reminded in Equation 5.4.

$$ 
\begin{aligned}
\text{SAR} &= \frac{E_{\text{SNN}}}{E_{\text{FNN}}} \times \lambda, \\
\lambda &= \frac{E_{\text{MAC}}}{E_{\text{ACC}}}
\end{aligned}
\qquad (5.4) $$

Where $E_{\text{SNN}}$ and $E_{\text{FNN}}$ are the energy consumption of equivalent spiking and formal accelerators, and $E_{\text{MAC}}$ and $E_{\text{ACC}}$ are the energy consumption of MAC and ACC accelerators. The value of $\lambda$ depends on the device and on the number of MACs in the formal design, as explained in Section 4.3.2 (Equation 4.8). In this case, $\lambda$ is computed for the Xilinx Zedboard (Zynq-7020). It depends on the design, thus it is computed for each task of the benchmark separately with parallel (PADS/VGT) and sequential (SPLEAT/C-HLS) architectures. According to the model, spiking domain enables energy savings when SAR>$\lambda$ and vice versa. The SAR and $\lambda$ values are shown alongside energy estimations in Figure 5.19. To allow comparison with PADS, all results are limited to the classification stage. The full-network comparisons for SPLEAT and C-HLS are shown in Appendix 2.

In the Figure, cases where the SAR model is correlated with energy estimations are marked with a green tick. Otherwise, graphs are marked with a red cross. For parallel implementations, the SAR model is accurate in five cases out of seven. For sequential implementations on the other hand, the model is wrong in five cases out of seven. A simple explanation can be given for this difference. PADS and VGT have very similar implementations (i.e. same level of parallelism and same intra and inter-layer pipeline). On the other hand, we have already explained the low-level differences between SPLEAT and C-HLS. The SAR model is made to compare equivalent architectures thus it is not fully adapted for C-HLS and SPLEAT. Thus, the SAR-based cartography is sensitive to low-level implementation differences.

On the other hand the model is more accurate for the comparison of VGT and PADS. The model still fails in two cases: MNIST and OPSSAT. There are three points which explain the incorrectness of the model on those two cases. First, energy ratio between PADS and VGT is smaller in MNIST and OPSSAT than in other tasks. Hence, the difference of energy consumption might fall within the margin of error of our model. Second, the SAR model does not acknowledge the temporal sparsity of spikes. The sparsity has a strong influence on inference time and therefore energy consumption, which is not represented in the SAR model. Those two first explanations will serve for future improvements of the SAR model. Third, the SAR model is based on the design activity (i.e. dynamic power) and neglects the static component. In FPGAs, static power is high and often represents in a significant proportion of the overall power usage. In ASICs on the other hand, the static component is much less significant. In all, the SAR model seems better suited to ASIC implementation than FPGAs. Figure 5.20 shows the energy estimations for MNIST and OPSSAT when neglecting the static power consumption. In this context, the model is more accurate: the prediction becomes valid for OPSSAT, but still fails on MNIST.
<figure><img src="image_66.png" /><figcaption>Figure 5.20: SAR and energy estimations without static power consumption, on MNIST ($\Delta$ = 20) and OPSSAT($\Delta$ = 4)</figcaption></figure>

Lastly, it should be noted that all results were obtained with $\Delta$ = 20. That is to ensure the best accuracy, at the expanse of synaptic activity. Indeed the goal of this section is not to provide an energy benchmark for SNN and FNN but rather to validate the SAR model. In Section 4.2, we have shown that SAR increases with $\Delta$. This parameter could be fine-tuned to obtain the best compromise between accuracy and SAR. For example the energy estimation from VGT and PADS on MNIST with $\Delta$ = 5 is provided in Figure 5.21a. In this case, the SAR model is valid, and PADS consumes fewer energy than VGT. The counterpart is a loss of SNN accuracy (98.74% to 97.9%). This observation confirms that tuning $\Delta$ leverages the trade-off between energy consumption and performance in rate-coded hardware SNNs. Moreover, the figure shows that PADS provides substantial energy saving over VGT in Mines Versus Rocks. Our intuition is that the small size of input (i.e. small timestep length in the spike generation process) enables inference time savings and thus energy savings.

## 5.7 Conclusion

In this chapter, a framework for hardware-footprint estimation of neuromorphic accelerators was provided. The framework covers wide range of layer sizes for 4 different architectures: PADS and SPLEAT for spiking domain, VGT and C-HLS for formal domain. SPLEAT and C-HLS are highly sequential while VGT and PADS are highly parallel. The framework is based on an hardware measurement campaign, design space interpolation and estimation of inference time and power. In doing so, the framework drastically facilitates the cartography of applications, neural coding domain and level of parallelism. Indeed, the results presented in this section would have required hundreds of hours of synthesis and simulation. Thanks to the framework, such results are obtained in a few minutes.

The framework was applied to Spoken Digits and OPSSAT datasets. Energy is the most commonly limiting factor in embedded systems, thus it is the main metric used for cartography. In this context, the suitability of spiking domain is not straightforward. SPLEAT provided substantial
<figure><img src="image_67.png" /><figcaption>Figure 5.21: SAR and energy estimations for MNIST with $\Delta = 5$ (left) and MNIST with $\Delta = 20$ after applying the DSP saturation correction (right)</figcaption></figure>

energy savings over C-HLS (up to a factor 1000), but the architectures have different low-level implementations. Therefore, the generalization of conclusions is mitigated. On the other hand, PADS and VGT have very similar implementations and the comparison is more reliable. In such conditions, spiking domain is responsible for increasing energy consumption by a factor 1.3 on OPSSAT and 3 on Spoken Digits. However, further experiments have shown that PADS could bring energy savings for very low SAR, like on MNIST with $\Delta = 5$. However, the current SAR-based cartography under-estimates PADS energy consumption and is inconsistent with energy estimations. Methods to improve the reliability of the SAR-based cartography will be discussed in the outlooks of this chapter. Moreover, PADS reduces energy consumption by a factor 3.3 in Mines Versus Rocks, despite the SAR predictions. That is thanks to the small size of Mines Versus Rocks samples, which reduces the timestep of the spike generation process in turns reducing inference time and energy consumption. Hence, spiking domain is also adapted to datasets with small samples. In all, the energy overhead of PADS is due to the temporal sparsity of spikes in the current spike encoding. Reducing the temporal sparsity of spikes is thus crucial in order to compete with formal architectures regarding inference time and energy consumption.

However, there are some applications where energy is not the only limiting factor. In drones and satellite applications for example, size is crucial. In solar-powered systems on the other hand, power consumption is the most limiting aspect. Spiking applications could therefore be suited to such applications, as it enables reliable resource and power savings over formal domain. That is, PADS reduces the LUT usage by 50% and DSP usage by 100%, while providing 50% savings in dynamic power. In applications such as on-board satellite cloud segmentation use-case [93] (Section 6.2.2) or object-tracking in drones [108], spiking domain should be preferred over formal inference.

Additionally, the comparison of SPLEAT and PADS yields that high level of parallelism does not always provide speed-up in spiking domain. That is, for low SAR, PADS did not reduce inference time over SPLEAT, both at layer and network level. In such conditions, using lower level
<table><caption>Table 5.1: Dynamic power consumption in PADS with and without power gating, on the benchmark of datasets. VGT dynamic power usage is shown for comparison. Results are shown for the classification stage (FC layers) only. The results have been obtained through the estimation framework, for $\Delta = 20$ (best accuracy).</caption><thead><tr><th rowspan="2">Classification Stage</th><th colspan="3">Dynamic Power (mW)</th></tr><tr><th>PADS No PG</th><th>PADS PG</th><th>VGT</th></tr></thead><tbody><tr><td>OPSSAT</td><td>0.07</td><td>0.03</td><td>0.15</td></tr><tr><td>MNIST</td><td>0.97</td><td>0.57</td><td>2.07</td></tr><tr><td>GTSRB</td><td>4.04</td><td>2.07</td><td>12.4</td></tr><tr><td>CIFAR-10</td><td>0.08</td><td>0.02</td><td>0.19</td></tr><tr><td>MVR</td><td>1.07</td><td>1.05</td><td>1.26</td></tr><tr><td>Spoken Digits</td><td>1.53</td><td>0.98</td><td>3.32</td></tr></tbody></table>

of parallelism brings drastic resource and power savings (up to 90%) without increasing inference time. Furthermore, PADS and SPLEAT are both prototypes with extreme level of parallelism. An accomplished accelerator should involve an intermediate level of parallelism, possibly different from one layer to the other. In this context, SAR can be used to tune parallelism at both layer and network level. Such possibility will be investigated further in the outlooks.

### 5.7.0.0.1 Discussions on power usage
In this chapter, we have seen that the power savings provided by spiking domain could be penalized in two ways:

* First, a large part of the dynamic power usage in FPGA systems is due to clock signals. The clocks still use power when the design is in idle state, i.e. waiting input spikes. Hence, idle states and active states are equivalent in terms of dynamic power consumption. The Neural Processing Units of PADS are often in such idle state due to the high temporal sparsity of spike encoding. That is the intrinsic sparsity of rate-encoding coupled with the sequential Spike Generation Cell. Therefore, the dynamic power savings brought by the simpler computation are over-compensated by the consumption of idle state.

* Second, power consumption has two components: dynamic power usage mentioned above, and static power consumption. If the dynamic power usage is related to the intrinsic signal toggles in the design, the static power can be considered as an offset depending on the device. For FPGAs, the static component stands for a large part of the overall power consumptions (approximately 25% in a Zedboard at 100% occupation, in average). In turns, the power savings are often not significant in front of the static power offset for FPGA targets. That is even more true with small designs that occupies small portions of the programmable logic. In such conditions, the dynamic power savings are not visible at system level.

In order to simulate the effect of a strong power gating on spiking architecture, idle power consumption is set to 0 in our estimation framework. This approach neglects the cost of the additional hardware required to manage clock-gating, but it still provides a good estimation of the potential gains. The results on a few datasets are shown in Table 5.1 for information. Hence, this preliminary experiments show that spiking accelerator could indeed mitigate the temporal sparsity of spikes through a power-gated implementations. Such feature should be studied in further work.
On the other hand, a good solution to enhance the power savings of neuromorphic circuits is to address target with lower static-to-dynamic power consumption ratio. For example, ASIC technology demonstrates much lower static power consumptions than FPGAs for equivalent designs. This technology was not available during the PhD thesis, but it should be addressed in further work, possibly at simulation level. Moreover, addressing ASICs instead of FPGA is a much easier task for our estimation approach. If FPGAs have limited resources which creates side-effect during synthesis, this is not so much the case with ASICs. Therefore, such devices seem better suited for our estimation framework approach and this possibility should be studied further.

In all, addressing both of those issues to enhance the power savings of spiking accelerators could in turn leverage the expected energy savings.

## 5.8 Outlooks

In this section, we provide some outlooks on the future improvement of the work presented in this section.

### 5.8.1 Improvement of the SAR metric

As shown in the previous section, the SAR metric is not always able to predict which coding domain is preferable in terms of energy consumption. In highly-parallel accelerators (PADS and VGT) that is because the SAR metric does not take spike sparsity into account. To cope with this issue, the SAR metric should be improved by multiplying the result by a variable representing spike sparsity as described in Equation 5.5.

$$ \frac{E_{SNN}}{E_{FNN}} = \left( \frac{SAR}{\lambda} \right) \times \gamma \qquad (5.5) $$

Where $\gamma$ is the sparsity factor. A possibility would be to set $\gamma$ to the average time between two consecutive input spikes. Such model should be tested and validated experimentally in further work.

### 5.8.2 Improvement of the estimation framework

Other sources of error lies in the energy estimation framework. First, the saturation of DSP is taken in account in the SAR model, but not in the network-wise estimations. Indeed, the sum of layer-wise DSP usage often overpasses the limit of the synthesis target (Zedboard). This could cause inconsistency between the SAR model and energy estimations. To cope with this issue, the saturation of DSPs can be approximated using the Table 4.4 of Section 4.3.2. Indeed, this table shows the hardware cost (resource, power and energy) for a MAC acceleration implemented with and without DSP. Hence, the exceeding DSPs can be converted in LUTs and Registers, and the energy consumption corrected accordingly. Such work has been done on MNIST ($\Delta = 20$) as shown in Figure 5.21b. The correction does reduce the error margin, but the SAR-based estimation remains incorrect.

Another way to cope with the issue could be to use ASIC estimations instead of FPGA. Such designs are not affected by such resource saturation, thus no correction is required. Moreover, the FPGA synthesis tool (Vivado) used to obtain all results performs low-level optimizations depending on the board limitations. Such mechanisms could affect the consistency and generalization of the
estimations. Targeting ASIC rather than FPGA could also avoid such concerns. Additionally, the static power consumption in FPGAs is high and often masks the energy savings provided by spiking domain. In ASICs, the static component is much smaller in front of dynamic power usage. As power savings are made on the dynamic component, ASIC technology could highlight the energy benefits of spiking domain. The estimator will be adapted to ASIC technology in further work.

Additionally, C-HLS should be replaced by an architecture closer to SPLEAT in terms of low-level implementation choices. For example, a more relevant formal sequential architecture should have the same inter-layer pipeline as SPLEAT, and the same intra-layer level of parallelism. In doing so, the comparison of SAR predictions and energy estimations will gain relevance.

## 5.8.3 Studying the level of parallelism

Lastly, this work demonstrated that PADS did not reduce latency compared to SPLEAT when SAR is low. That is, such high level of parallelism is not necessary to sustain the actual density of computation. SPLEAT and PADS are two prototypes which feature two opposed extreme levels of parallelism, but a realistic architecture must involve an intermediate level of parallelism. The goal is to find the lowest possible level of parallelism that does not affect inference time. For example, by using several SPLEAT Neural Processing Unit per layer. In doing so, the quantity of allocated resources and power is strictly adapted to the density of spikes. In further work, an analytic model should be constructed to study this optimal point. SPLEAT will also be adapted to feature intermediate level of parallelism.

## 5.8.4 Hybridization and other spike encoding methods

This study also brought several insights and perspectives for the remaining of this work. First, coding domain should be tailored to SAR at layer level. For example in Spoken Digits, spiking domain is preferable in FC1 but not in FC2 (in terms of energy consumption). In the light of those observations, using spiking domain on one layer and formal domain on the other could bring the best from both worlds to a same topology. In the next chapter of this thesis (Chapter 6), such concept will be investigated and applied to OPS-SAT and Spoken Digits dataset.

Moreover, temporal sparsity of spikes in the current encoding technique is responsible for the energy overhead in the studied neuromorphic implementations. Hence, reducing temporal sparsity is a crucial issue that must be tackled. Two possibilities arises. First, the temporal sparsity depends on the timestep length in the Spike Generation Cell. This timestep is proportional to the input size. In Chapter 3, down-sampling the input resulted in a loss of information that increased inference time and lowered accuracy. Hybridization could cope with this issue, as spike encoding can be made on intermediate activation (i.e. between two layers rather than at the input). Indeed, the size of deep layers are often lower than the size of samples. Second, the temporal sparsity is also intrinsic to rate-based SNNs, as information is encoded through time. In Chapter 7, we investigate a novel type of SNN with lower temporal sparsity and constrained number of timesteps. In the current rate-based SNN model, inference time is not deterministic and drastically vary from one image to the other (up to a factor ten). This could cause timing violation issues for integration in a system. The constrained number of timesteps used for spike encoding in Chapter 7 solves this issue that was let aside in our previous discussions.
# Chapter 6

# Neural coding domain hybridization

## Chapter contents

<table><tr><td>6.1 Motivations</td><td>113</td></tr><tr><td>6.1.1 SAR and footprint variability</td><td>113</td></tr><tr><td>6.1.2 Distribution of activity</td><td>115</td></tr><tr><td>6.1.3 Formal convolutions and spiking classification</td><td>116</td></tr><tr><td>6.2 Estimations on hybrid architectures</td><td>116</td></tr><tr><td>6.2.1 Methodology</td><td>116</td></tr><tr><td>6.2.2 OPS-SAT</td><td>117</td></tr><tr><td>6.2.3 Spoken Digits</td><td>120</td></tr><tr><td>6.2.4 Discussions on hybrid estimations</td><td>122</td></tr><tr><td>6.3 Hybrid hardware implementation</td><td>123</td></tr><tr><td>6.3.1 Context</td><td>124</td></tr><tr><td>6.3.2 VGT-PADS Hybrid Architecture</td><td>124</td></tr><tr><td>6.4 Conclusion</td><td>127</td></tr><tr><td>6.4.1 Outlook</td><td>128</td></tr></table>
The idea of tailoring the coding domain at layer level has been mentioned several times in this document. Rather than using a fully-formal or fully-spiking accelerator, the inter-layer variability of SAR and hardware footprint estimation suggests that hybridization could bring benefits and mitigate drawbacks from both worlds. Therefore we propose a novel approach for neuromorphic systems: hybrid acceleration. i.e. a combination of formal and spiking accelerators. In this chapter, we focus on a specific type of hybridization:

* Formal feature extraction stage (Convolution & Pooling) for example with VGT,

* Transcoding of feature-maps achieved with the Spike Generation Cell of PADS 3.

* Spiking classification stage (Fully-connected) for example using SPLEAT or PADS Neural Processing Units

First, we introduce the motivations that led to the development of neural coding hybridization. This concept is applied to OPS-SAT and Spoken Digit datasets. We use the estimation framework described in Chapter 5 to estimate energy consumptions on VGT-PADS and VGT-SPLEAT hybrid architectures. For comparison, results for VGT and SPLEAT architectures are also provided. In doing so, the aim of hybridization is to benefit from the low power and logic usage of spiking accelerators, while mitigating their intrinsic time overhead. Third and last, a prototype of VGT-PADS hybrid accelerator for OPS-SAT cloud segmentation task is proposed. This architecture is embedded on-board OPS-SAT satellite for in-flight testing. The hardware measurement, inference time and power usages obtained in real conditions are confronted to the estimations.

## 6.1 Motivations

In this section, we provide a list of elements which motivated the hybridization approach. Each of the following subsections is dedicated to one particular motivation.

### 6.1.1 SAR and footprint variability

Results of Chapters 4 and 5 have shown that SAR and energy estimations varied drastically from one layer to another within a same spiking-CNN. As a result, some layers of a model might be suited to spiking acceleration while others are not. This is the case in Spoken Digits ($\Delta = 20$) as reminded in Figure 6.3, where only fully-connected layers have SAR below $\lambda$. As explained previously, $\lambda$ is a value specific to the target (Zedboard) and design which represents the ratio of energy consumption between a MAC and an ACC operation (see 4.3.2). Using spiking coding domain in the classification stage (FC layers) of Spoken Digits might thus bring energy savings locally in the architecture.

#### 6.1.1.1 Input size and loss of information

The influence of the spatial resolution of input data was measured in Section 3.2.3. As explained in Chapter 3 (Section 3.1.1), the temporal sparsity of spike encoding depends on input size. More precisely, the time increment of the spike encoding process (timestep) is equivalent to the sample size in clock cycles. For OPS-SAT, that is $28 \times 28 \times 3 = 2352$ clock cycles (23520 ns). Thus, down-sampling input size was expected to increase the input spike rate through decreasing the GenCell timestep. However, the opposite was observed and down-sampling input size resulted in an
<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /><figcaption>Figure 6.1: Activity distribution in CNN layers for OPS-SAT (left) and Spoken Digits (right)</figcaption></figure>
<table><caption>Table 6.1: CNN topologies for a) OPS-SAT and b) Spoken Digits datasets.</caption><caption>(a)</caption><thead><tr><th>Layer</th><th>Type</th><th># Kernels</th><th>Kernel size</th><th>Stride</th><th>Output Shape</th></tr></thead><tbody><tr><td>0</td><td>Input</td><td></td><td></td><td></td><td>(28,28,3)</td></tr><tr><td>1</td><td>Conv</td><td>3</td><td>5</td><td>1</td><td>(24,24,3)</td></tr><tr><td>2</td><td>MaxPool</td><td>3</td><td>2</td><td>2</td><td>(12,12,3)</td></tr><tr><td>3</td><td>Conv</td><td>5</td><td>5</td><td>1</td><td>(8,8,5)</td></tr><tr><td>4</td><td>MaxPool</td><td>5</td><td>2</td><td>2</td><td>(4,4,5)</td></tr><tr><td>5</td><td>FC</td><td></td><td></td><td></td><td>10</td></tr><tr><td>6</td><td>FC</td><td></td><td></td><td></td><td>2</td></tr></tbody></table><table><caption>(b)</caption><thead><tr><th>Layer</th><th>Type</th><th># Kernels</th><th>Kernel size</th><th>Stride</th><th>Output Shape</th></tr></thead><tbody><tr><td>0</td><td>Input</td><td></td><td></td><td></td><td>(507,1)</td></tr><tr><td>1</td><td>Conv</td><td>3</td><td>5</td><td>1</td><td>(503,3)</td></tr><tr><td>2</td><td>MaxPool</td><td>3</td><td>2</td><td>2</td><td>(252,3)</td></tr><tr><td>3</td><td>Conv</td><td>5</td><td>5</td><td>1</td><td>(248,5)</td></tr><tr><td>4</td><td>MaxPool</td><td>5</td><td>2</td><td>2</td><td>(124,5)</td></tr><tr><td>6</td><td>Conv</td><td>5</td><td>5</td><td>1</td><td>(120,5)</td></tr><tr><td>7</td><td>MaxPool</td><td>5</td><td>2</td><td>2</td><td>(60,5)</td></tr><tr><td>8</td><td>FC</td><td></td><td></td><td></td><td>100</td></tr><tr><td>9</td><td>FC</td><td></td><td></td><td></td><td>10</td></tr></tbody></table>

<figure><img src="image_70.png" /><figcaption>Figure 6.2: Spike train period with respect to element intensity according to the rate-coding policy, with *PeriodMax* = 1timestep and *PeriodMax* = 100timesteps</figcaption></figure>

extended processing time and lower accuracy. That is because of the arbitrary loss of information caused by down-sampling. The idea behind hybridization is thus to use the first layers of a CNN as a pre-processing for down-sampling without loss of information. Indeed, deeper layer activations often have lower resolution than the input sample. That is the case for FC1 in both OPS-SAT and Spoken Digits CNN topologies 6.1. In doing so, we hope to reduce the spike temporal sparsity and therefore reduce inference time.

### 6.1.2 Distribution of activity

In Chapter 4 (Section 4.4.3), we have shown that data distribution in vector datasets was not suited to the current rate-encoding policy. As a reminder, the rate-encoding rule is pictured in Figure 6.2: the graph shows that rate-coding policy discriminates pixels with very low intensity from others. Therefore, narrow data distribution such as that of Spoken Digits samples are poorly represented by the rate encoding policy. On the other hand, the distribution of intermediate activation are depicted in Figure 6.1 for each layer of OPS-SAT and Spoken Digits CNNs (Pooling layers have no influence on activity distribution, thus they are not represented). The CNN topologies are described in Table 6.1. In the following, we refer to the output of the feature extraction stage as "latent representation", a common use in Machine Learning. In Figure 6.1, the latent representation (Conv2 for OPS-SAT and Conv3 for Spoken Digits) have sparse distributions, with
a lot of pixels with very low intensity. This distribution thus appears to be much more adapted to the rate-encoding policy than the input samples. As explained in section 4.4.3, a better spike-encoding should result in a more efficient processing, i.e. less spikes are needed for classification. In turns, this is expected to reduce SAR and in turns inference time and energy consumption of spiking accelerators.

### 6.1.3 Formal convolutions and spiking classification

Moreover, it should be noted that this work focuses on a specific hybridization scheme where the feature extraction stage operates in formal domain and classification stage in spiking domain. There are a lot of different possibilities of hybridization and our specific approach relies in three elements: First, the number of successive transcoding (passing from one coding domain to another) must not be too great, as encoding and decoding spiking information requires specific hardware which counter-balance the resource and power savings of spiking accelerators. Therefore, we limit our approach to two successive coding domains. Second, a spike encoding module was already developed earlier in the thesis (GenCell), whereas the spike decoding module must be developed from scratch. Indeed, the current Terminate Delta Module only works for categorical classification and cannot provide accurate decoding. Therefore, we concentrate on one single type of transcoding: formal towards spikes. Third and last PADS (which has been developed during this thesis: Chapter 3) only covers spiking fully-connected layers in its current development state, thus it is only able to replace the classification stage for hybridization.

## 6.2 Estimations on hybrid architectures

In this section, we apply the estimation framework to hybrid architectures on OPS-SAT and Spoken Digits dataset. Such hybrid estimation requires a specific method described in the following.

### 6.2.1 Methodology

The hybridization approach involves a few differences with the estimations provided in Chapter 5. Indeed, the inference time and dynamic power estimations for the hybrid architectures cannot be made on the original N2D2 activity logs. Indeed, spiking activity is influenced by hybridization, such as a reduction of temporal sparsity as explained in the motivations. This difference must be taken in account as it is one of the major appeal of hybridization: reducing spike encoding temporal sparsity enables to decrease inference time and therefore reduce energy consumption. For each task, the fully-connected stage must be tested in spiking domain separately on a new dataset of latent representation. To do so, we build a dataset of OPS-SAT latent representation and perform spiking inference test using N2D2. The spike encoding is configured with the new *MinPeriod* and *MaxPeriod* values found in Table 6.2. As a reminder, the period values depends on the GenCell timestep length, which itself depends on the sample size: the timestep length is the input size expressed in clock cycles (1clk=10ns at 100MHz). All input sizes and corresponding period hyper-parameters are listed in Table 6.2 for convenience. The timestep used to encode latent representation is indeed much smaller than that required to encode input samples, which promises to reduce encoding temporal sparsity and improve processing speed in the spiking classification stages.
<table><caption>Table 6.2: Input sizes and spike generation min and max period values for full CNN and Hybrid classification stages on OPS-SAT and Spoken Digits dataset.</caption><thead><tr><th rowspan="2"></th><th colspan="3">OPS-SAT</th><th colspan="3">Spoken Digits</th></tr><tr><th>Input shape (size)</th><th>MinPeriod (ns)</th><th>MaxPeriod (ns)</th><th>Input shape (size)</th><th>MinPeriod (ns)</th><th>MaxPeriod (ns)</th></tr></thead><tbody><tr><th>Full network</th><td>28x28x3 (2352)</td><td>23 520</td><td>2 352 000</td><td>507x1x1</td><td>5 070</td><td>507 000</td></tr><tr><th>Hybrid classification stage</th><td>80x1x1 (80)</td><td>800</td><td>80 000</td><td>295x1x1 (295)</td><td>2 950</td><td>295 000</td></tr></tbody></table>

As it is not possible to retrieve the latent representation directly from N2D2, we use Tensor-Flow instead. To do so, the synaptic weights are exported from N2D2 to TensorFlow, and the intermediates activation are extracted by running the feature-extraction stage and logging the output. Each resulting feature-map is labeled according to the original label of the sample. The classification stage is then tested in spiking domain using N2D2 configured with the periods of Table 6.2. The subsequent synaptic logs are used to estimate time, power and energy in the spiking classification stages of VGT-SPLEAT and VGT-PADS using the estimation framework presented in Chapter 5. For VGT and SPLEAT, the estimations are obtained as described in Chapter 5. It should be noted that there are a few difference between N2D2 and TensorFlow formal inference. Thus, there are slight differences between the estimations provided in this section and that of Chapter 5.

Additionally and in contrast with Chapter 5, we take into account the duration of the Gen-Cell initialization phase in the inference time estimations. This initialization phase lasts for one timestep, as described in Chapter 3. The corresponding time overhead is added to the overall inference time estimations (FE stage for full network and CL stage for hybrid network). Moreover, we also take into account the resource and power usages of the GenCell. To do so, the GenCell are synthesized separately, in the right configurations, to provide accurate resource and power estimations. In future work, those metrics will be directly obtained by incorporating GenCell estimations to the overall framework.

## 6.2.2 OPS-SAT

In this section, we apply the hardware footprint estimator to OPS-SAT dataset for VGT, SPLEAT, VGT-PADS and VGT-SPLEAT. OPS-SAT dataset seem well suited to hybridization for two main reasons: first, the OPS-SAT feature maps are 30 times smaller than the input samples, which enables to reduce temporal sparsity of spike encoding by a factor 30. Second, the activity distribution is wider in feature-maps than in input samples. Moreover, the OPS-SAT use-case of CIAR project offers to deploy and test the architecture directly on-board a satellite. That is a good opportunity for such a proof of concept. The OPS-SAT use-case will be presented in details in 6.3.1. As a reminder, the CNN used for OPS-SAT classification is available in Table 6.1a.

### 6.2.2.0.1 SAR on OPS-SAT
SAR and accuracy measurements for OPS-SAT spiking CNN are reminded in Figure 6.3. In this section we focus on $\Delta = 4$, as the accuracy improvements achieved for higher $\Delta$ values are minimal and are not worth the SAR overhead (see complete Figures in Chapter 4). With $\Delta = 4$, both formal and spiking test reached 81 % accuracy using
<figure><img src="image_71.png" /><figcaption>Figure 6.3: Layer-wise and total SAR for a) OPS-SAT ($\Delta = 4$) and b) Spoken Digits ($\Delta = 20$).</figcaption></figure>

N2D2. All layer-wise SAR measurements for OPSSAT $\Delta = 4$ are lower than the $\lambda$ thresholds, which means all layers are suitable for spiking conversion and not only the classification stage. That is according to the SAR model described in Chapter 4 (Section 4.3.2). As a reminder, $\lambda$ is the ratio of energy consumption between a MAC operation and an ACC operation. It depends on the hardware target and the number of MACs involved in the formal architecture (VGT). The values have been plotted for Zedboard ($\lambda_{ZE}$ in orange) and ZCU102 ($\lambda_{ZC}$ in red). In this chapter, we focus on $\lambda_{ZE}$ has estimations are made for Zedboard. In Figure 6.3, all layers of the OPS-SAT-related CNN are already suitable for spiking domain, according to the $\lambda$ model. Therefore, the goal is not to tailor coding domain with respect to SAR, but rather to benefit from the lower input resolution and wide activation distribution (Figure 6.1) of the latent representation. As a reminder, OPS-SAT samples have 2352 values while OPS-SAT latent representation only have 80.

### 6.2.2.0.2 Estimations on OPS-SAT
Before going into further details, we remind the notations used to compare estimations (already used in Chapter 5): Metric<sub>Arch</sub><sup>Task-Layer</sup> stands for the estimation of a metric on a given dataset, for a given layer (or group of layers) and architecture. For example $E_{SPLEAT}^{Spoken-Conv}$ is the Energy consumption of SPLEAT Convolution layers on Spoken Digits dataset. Moreover, the Classification stage is noted CL, and the Feature-Extraction stage is noted FE.

The resource, time, power and energy estimations on OPS-SAT dataset are provided in Figure 6.4 for VGT, SPLEAT, hybrid VGT-SPLEAT and VGT-PADS. As already stated in previous chapter, the most limiting factor in most embedded systems is energy. Therefore, we begin by discussing the impact of hybridization in the light of energy consumption. In this chapter, we refer to the Feature Extraction stage (convolution and pooling layers) as FE, and the classification stage (fully-connected) as CL. As shown in the bottom row of the Figure, hybridization enables to drastically reduce the energy overhead of spiking neural networks. For VGT-SPLEAT, on the one hand using the formal VGT FE stage reduces $E^{FE}$ by a factor 11.5. On the other hand, hybridization enables to reduce the GenCell timestep from 2352 clock cycles (i.e. sample size) to 80 clock cycles (i.e. feature-maps size). This allows to drastically reduce the rate-encoding temporal sparsity, and thus inference time and energy: $E_{SPLEAT}^{CL} \approx 20 \times E_{VGT-SPLEAT}^{CL}$. This is also visible in the inference time results: $T_{SPLEAT}^{CL} \approx 27 \times T_{VGT-SPLEAT}^{CL}$. In all, the results show hybridization with SPLEAT enables to reach the same energy consumption than a pure VGT ($E_{VGT} \approx E_{VGT-SPLEAT}$).
<figure><img src="image_72.png" /><figcaption>Figure 6.4: Hardware footprint estimation of VGT, SPLEAT, Hybrid VGT-SPLEAT and Hybrid VGT-PADS on OPS-SAT RGB dataset with $\Delta = 4$, *MinPeriod* = 1 and *MaxPeriod* = 100. a) LUT, b) Registers, c) Block RAM, d) DSP, e) Power, f) Inference time and g) Energy.</figcaption></figure>
Similarly to VGT-SPLEAT, hybridization with PADS enables to reduce the energy consumption in the classification stage. Comparing with energy estimations of Chapter 5 for PADS (Figure 5.18) yields that $E_{\text{PADS}}^{\text{CL}} \approx 2.5 \times E_{\text{VGT-PADS}}^{\text{CL}}$. Furthermore, energy estimations without hybridization issued that $E_{\text{PADS}}^{\text{CL}} > E_{\text{SPLEAT}}^{\text{CL}}$. As a reminder, that is because PADS parallelism is not exploited due to spike temporal sparsity of OPS-SAT task: $T_{\text{SPLEAT}}^{\text{CL}} \approx T_{\text{PADS}}^{\text{CL}}$ while $P_{\text{SPLEAT}}^{\text{CL}} < P_{\text{PADS}}^{\text{CL}}$. With hybridization, we observe that $T_{\text{VGT-SPLEAT}}^{\text{CL}} \approx 4 \times T_{\text{VGT-PADS}}^{\text{CL}}$. That is, temporal sparsity of spike is higher and SPLEAT is no longer able to process spike at the same rate than PADS. Consequently, we observe that $E_{\text{VGT-SPLEAT}}^{\text{CL}} \approx 4 \times E_{\text{VGT-PADS}}^{\text{CL}}$. In other words, hybridization increases the parallelism requirements and PADS is therefore preferable in this context. Furthermore, we observe that VGT-PADS CL stages uses even fewer energy than pure VGT CL stage: $E_{\text{VGT}}^{\text{CL}} \approx 2 \times E_{\text{VGT-PADS}}^{\text{CL}}$. That is, hybridization enables to draw energy savings from PADS, whereas it was not the case in Chapter 5. However, at network level $E_{\text{VGT}} \approx E_{\text{VGT-PADS}}$ because most of the energy consumption lies in the VGT FE stage used in both. Therefore, hybridization does not provide significant energy savings on OPS-SAT.

However, using the SPLEAT classification stage instead of VGT enables significant resource savings: the overall LUT and FF usage are reduced by 30% and DSP usage by 80%. Dynamic power usage is also reduced by 10% which is not significant. That is, VGT-SPLEAT hybridization enables to reduce the resource usage of OPS-SAT classification implementation without the usual energy overhead attributed to SPLEAT. For VGT-PADS, the resource savings are much less significant as the OPS-SAT CNN is small: as shown in Chapter 5, PADS resource savings over VGT arise for larger layer sizes. Therefore, we expect more significant resources savings on Spoken Digits Dataset.

In all, estimations show that hybridization offers an interesting trade-off between the resource efficiency of spiking domain and speed of formal accelerators. However, OPS-SAT dataset is not the best application case for hybridization. Indeed, most of computation in VGT on OPS-SAT dataset takes place in the feature-extraction stage. Thus, the transcoding of the classifications stage does not drastically change the overall footprint. In the next section (6.2.3), hybridization is applied to the Spoken Digits CNN, where classification stage is much more important.

### 6.2.3 Spoken Digits

In this section, we apply the hybridization principle to Spoken Digits datasets whose CNN topology is reminded in Table 6.1. Spoken Digits is well suited to hybridization: the feature-maps are smaller than the input, and show a wide pixel distribution. The narrow intensity distribution of Spoken Digits samples have already been pointed as a cause for the bad SAR scores of spiking CNNs in Section 4.4.3. The much wider distribution of feature-maps should provide a better rate encoding of data. In doing so, we hope to decrease SAR and inference time compared to the original SNN.

6.2.3.0.1 SAR on Spoken Digits
The SAR distribution in Spoken Digits is shown in Figure 6.3. In this section, we focus on $\Delta = 20$ to reach the same accuracy as the formal model and provide a fair context for comparisons. As shown in the figure, the FC2 layer is the only one whose SAR is below both $\lambda_{ZE}$ values for $\Delta = 20$. According to the model, FC2 is thus the only suitable layer for acceleration in spiking domain, whereas the feature extraction stage is not. Thus, in this case the interest of hybridization is not only to benefit from the latent representation small resolution and wide activation distribution, but also to benefit from the low SAR of the FC2 layer.

The hardware estimations for VGT, SPLEAT, VGT-SPLEAT and VGT-PADS are given in Figure 6.5. Like for OPS-SAT, we begin by discussing the energy estimations as this is the most
<figure><img src="image_73.png" /><figcaption>Figure 6.5: Hardware footprint estimation of VGT, SPLEAT, Hybrid VGT-SPLEAT and Hybrid VGT-PADS on Spoken Digits RGB dataset with $\Delta = 4$, MinPeriod = 1 and MaxPeriod = 100.
a) LUT, b) Registers, c) Block RAM, d) DSP, e) Power, f) Inference time and g) Energy.</figcaption></figure>
limiting metric in embedded systems. Using the VGT FE stage instead of SPLEAT FE stage enables substantial energy savings: $E_{\text{SPLEAT}}^{\text{FE}} \approx 327 \times E_{\text{VGT-SPLEAT}}^{\text{FE}}$. However, in contrast with OPS-SAT hybridization increases the energy consumption of SPLEAT. Indeed, $E_{\text{VGT-SPLEAT}}^{\text{CL}} \approx 3 \times E_{\text{SPLEAT}}^{\text{FE}}$. As explained in Chapter 5, the density of spikes is so higher in Spoken Digits FC1 (SAR$^{\text{Spoken-FC1}} \approx 2.8$) that SPLEAT layers are not able to process spikes on-the-fly, even without hybridization. Therefore, reducing temporal sparsity through hybridization does not speed-up SPLEAT inference time. In all, $E_{\text{VGT-SPLEAT}}^{\text{Spoken}}$ is slightly higher than $E_{\text{SPLEAT}}^{\text{Spoken}}$. Furthermore, VGT-SPLEAT is much more energy-intensive than VGT: $E_{\text{VGT-SPLEAT}}^{\text{Spoken}} \approx 86 \times E_{\text{VGT}}^{\text{Spoken}}$ That is, SPLEAT is not suited to hybridization in tasks with high SAR.

The conclusion is very different for VGT-PADS: thanks to the high level of parallelism, PADS is able to process spikes on-the-fly regardless of density. Therefore, reducing temporal sparsity enables to drastically reduce inference time and energy consumption in PADS classification stage. If we refer to the results of Chapter 5, $E_{\text{PADS}}^{\text{CL}} \approx 3600 \times E_{\text{VGT-PADS}}^{\text{CL}}$. Without hybridization, PADS classification stage consumes more energy than VGT (Chapter 5). We observe the opposite with hybridization: $E_{\text{VGT}}^{\text{CL}} \approx 1.6 \times E_{\text{VGT-PADS}}^{\text{CL}}$. In contrast with OPS-SAT, this saving is significant at network level as most of VGT energy consumption lies in the classification stage (80%). Indeed, VGT-PADS provides 30% energy savings over VGT at network level. Therefore, hybridization with PADS provides energy savings where PADS alone does not.

Moreover, energy savings provided by VGT-PADS also come with substantial resource and power savings. Hybridization with PADS enables to reduce LUT usage by a factor 2.4, register usage by 40% and DSP usage by 80%. Moreover, VGT-PADS dynamic power is 45% lower than pure VGT (43% for total power). Those savings are much more significant than on OPS-SAT dataset for two reasons. First, the layers are larger than in OPS-SAT and PADS resource savings arises for larger topologies. Second, most of the resource usage of VGT is located in the classification stage thus savings are significant at network level. In all, VGT-PADS hybridization seems very promising for embedded acceleration, as it enables significant savings in all metrics except inference time. Indeed, VGT-PADS is still 23% slower than pure VGT, but this overhead remains manageable. On the other hand, VGT-SPLEAT also provide substantial resource savings. However it is not necessary to discuss those results as SPLEAT provides even more resource savings for fewer energy.

## 6.2.4 Discussions on hybrid estimations

In the light of the estimations on OPS-SAT and Spoken Digits dataset, hybridization seems a good opportunity of trade-off between spiking and formal hardware acceleration. Indeed, the classification stage benefits from the resource and power efficiency of spiking domain all the while mitigating the inference time of spiking implementations. This second aspect is achieved by both using a faster formal feature-extraction stage and reducing the GenCell timestep length. Indeed, decreasing the timestep length reduces the temporal sparsity of spike encoding. This enables a drastic speed-up in spiking accelerators provided that they are able to sustain the increased spike density. On OPS-SAT, the SAR is low enough (SAR $\approx 0.4$). Therefore SPLEAT is able to process spikes on-the-fly, even with a lower spike temporal sparsity. VGT-SPLEAT thus competes with pure VGT in terms of energy consumption on OPS-SAT. However, a higher level of parallelism is required for hybridization on Spoken Digits which features a higher SAR (SAR $\approx 4$). Under such conditions, spikes arrive faster than the processing ability of VGT-SPLEAT, causing a severe time and energy overhead. On the other hand, VGT PADS is designed to process spikes on-the-fly regardless of the density. Therefore, hybridization with PADS reduces energy consumption by 30%
<figure><img src="image_74.png" /><figcaption>Figure 6.6: a: OPS-SAT CubeSat being tested before launch. b: OPS-SAT FlatSat platform. Photo credits: TU Graz [19].</figcaption></figure>

at network-level. On OPS-SAT, the savings are less significant and the energy consumptions are similar at network level. Hybridization also provides substantial resource savings. Like for energy, those savings are more important when the classification stage is larger: VGT-PADS reduces LUT usage by 40%, DSP usage by 80% and total power usage by 43% on Spoken Digits dataset. For the same reason, the savings are less significant on OPS-SAT dataset.

In all, hybridization implies a higher spike density, which requires a higher level of parallelism than a full spiking topology. If the architecture is able to benefit from the lower temporal sparsity of spikes, hybridization provides significant energy savings compared to fully-spiking implementations. Furthermore, hybridization offers significant energy and resource savings over formal implementations, provided that the classification stage is significant in front of the convolution stage. In this work, we only address a simple case where hybridization is limited to the classification stage.

However, hybridization could be made earlier in the network. The trade-off between the speed of formal accelerators and resource and power efficiency of spiking accelerators must be explored further in future works. The SAR metric and estimation framework will enable to find the optimal hybridization scheme for each topology.

In the following section, we propose an hardware implementation for the VGT-PADS hybrid accelerator. The estimations will be confronted to real hardware measurements in order to verify the benefits of hybridization. In doing so, we also provide hardware measurements on VGT and SPLEAT implementations for OPS-SAT dataset. In doing so, we propose to evaluate the reliability of our estimation framework, and strengthen the confidence in the conclusions of Chapter 5.

## 6.3 Hybrid hardware implementation

In this section, we describe the hardware implementation of the VGT-PADS hybrid neuromorphic accelerator. This accelerator is developed in the context of CIAR project and applied to the OPS-SAT cloud segmentation application. The architecture was embedded onboard OPS-SAT satellite for in-flight testing on real images. In doing so, the system was both the first spiking neural network accelerator in space, and the first hybrid neural network in space. The hardware measurements are confronted with the estimation results.
<figure><img src="image_75.png" /><figcaption>Figure 6.7: Impression of OPS-SAT in low earth orbit [20]</figcaption></figure>

### 6.3.1 Context

#### 6.3.1.1 OPS-SAT orbital laboratory

The hybrid neural network accelerator (VGT-PADS) was developed in collaboration with the CIAR project hosted by IRT Saint-Exupéry. The CIAR project addresses deployment of FPGA neural network accelerators in spatial application. More specifically, the CIAR project is implied in experimentation on-board OPS-SAT [109] provided by the European Space Agency (ESA). This satellite is a testing platform for European researchers in the satellite field. It is a CubeSat [110], and measures 10 × 10 × 30cm. It weighs 5.5kg. The main advantage of CubeSats is their small size and low weight, which facilitates the access to orbit. Figure 6.6 shows pictures of OPS-SAT and the associated “FlatSat” platform. The “FlatSat” architecture is identical to the satellite and serves as a test platform for applications and experiments. For illustration, an impression of OPS-SAT in low earth orbit is shown in Figure 6.7.

OPS-SAT includes various systems to support a large set of experiments. Each of these modules is specifically hardened for space applications. A description of the full satellite platform is provided in Figure 6.8. However, our experiment only involves the SoC Processor Payload and the HD camera. The camera is used to take photos of the earth to test application in-flight. The OPS-SAT dataset pictures were also taken from this camera, before being manually selected and labeled by CIAR project members. The SoC is a hardened MitySOM, which contains an ARM CPU, DDR4 memory and a Cyclone V FPGA. The VGT-PADS hybrid architecture is developed targeting this board.

### 6.3.2 VGT-PADS Hybrid Architecture

During CIAR project, the VGT-PADS hybrid hardware architecture was developed aiming in reducing resource and power usage of neural network acceleration for space applications, all the while reducing the intrinsic latency of spiking accelerators. In the present section, we describe the VGT-PADS hardware implementation and confront actual hardware measurements with estimations. Such comparison is made on VGT, SPLEAT and VGT-PADS accelerators. It should be
<figure><img src="image_76.png" /><figcaption>Figure 6.8: Schematic of OPS-SAT architecture. The yellow part is for the "technical" bus, and the blue part is for the payload. Source: [19]</figcaption></figure>

noted that VGT-SPLEAT was not implemented in hardware, due to the aforementioned estimated latency and energy overhead. Therefore it is not featured in this section.

### 6.3.2.1 Architecture

The VGT-PADS hybrid architecture combines a formal feature extraction stage (conv+pool) with a spiking classification stage (fully-connected). The formal feature extraction stage is implemented using the VGT architecture (Section 2.2.2.1), and the classification stage using PADS FC layers (Section 3.1). The interface between the two is based on the Spike Generation Cell (GenCell). An additional module retrieves the VGT latent representation, performs flattening and streams pixels to the GenCell. A Terminate Delta Module (TDM) is used to decode the spiking output and retrieve the winning class. The architecture is shown in Figure 6.9. The output of VGT feature maps are stored in FIFO queues, one for each channel. When the last pixel is received, the *FM_VALID* signal is raised, triggering the flattening process. Pixels are streamed one by one to the GenCell, following a user-defined flattening policy.

In the following subsection, we provide hardware measurements performed on the VGT-PADS hybrid accelerator, alongside hardware measurements for SPLEAT and VGT. The three accelerators are tested on OPS-SAT dataset. The measurements are confronted to the estimations to confirm the benefits of hybridization and strengthen the confidence in our estimation framework.

### 6.3.2.2 Results

All three architectures are configured for the OPS-SAT classification tasks, with the synaptic weights resulting from N2D2 training and conversion. Each architecture is synthesized and implemented using Vivado design suite targeting Xilinx Zedboard. The resulting hardware measurement
<figure><img src="image_77.png" /><figcaption>Figure 6.9: Illustration of the CIAR Hybrid Neural Network Architecture.</figcaption></figure>

<table><caption>Table 6.3: Measurements and estimations for VGT, SPLEAT and Hybrid VGT-PADS on OPS-SAT Dataset. The difference is expressed in % of the measurement.</caption><thead><tr><th rowspan="2"></th><th colspan="3">VGT</th><th colspan="3">VGT-PADS</th><th colspan="3">SPLEAT</th></tr><tr><th>MEAS.</th><th>EST.</th><th>DIFF (%)</th><th>MEAS.</th><th>EST.</th><th>DIFF (%)</th><th>MEAS.</th><th>EST.</th><th>DIFF (%)</th></tr></thead><tbody><tr><td>Time (ns)</td><td>8 835</td><td>9 850</td><td>11.5</td><td>10 985</td><td>9 191</td><td>16.3</td><td>248 073</td><td>216 997</td><td>13.2</td></tr><tr><td>Dynamic Power (W)</td><td>0.21</td><td>0.58</td><td>171.5</td><td>0.15</td><td>0.52</td><td>256.2</td><td>0.03</td><td>0.04</td><td>43.9</td></tr><tr><td>Static Power (W)</td><td>0.26</td><td>0.25</td><td>4</td><td>0.25</td><td>0.25</td><td>0</td><td>0.25</td><td>0.25</td><td>0</td></tr><tr><td>LUT (#)</td><td>22 549</td><td>26 500</td><td>17.5</td><td>17 245</td><td>25 900</td><td>50.2</td><td>2 398</td><td>1 730</td><td>27.9</td></tr><tr><td>FF (#)</td><td>30 671</td><td>25 200</td><td>17.8</td><td>23 996</td><td>25 600</td><td>6.7</td><td>724</td><td>951</td><td>31.3</td></tr><tr><td>BRAM (#)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>3.5</td><td>3</td><td>14.3</td></tr><tr><td>DSP (#)</td><td>760</td><td>255</td><td>70</td><td>200</td><td>60</td><td>70</td><td>8.00</td><td>6</td><td>25</td></tr><tr><td>Energy (nJ)</td><td>4 152</td><td>8 176</td><td>96.9</td><td>4 394</td><td>7 077</td><td>61.1</td><td>69 469</td><td>62 923</td><td>9.4</td></tr></tbody></table>

(resources, power, inference time and energy consumption averaged on 10 samples) are given in Table 6.3 alongside corresponding estimations. The difference between both values is expressed in % of the real measurement. The error margin is displayed in green when it is below 30%, orange when it is around 50% and red when it is greater than 70%. It should be noted that the hardware measurements are different from the original results provided in the CIAR publication [93]. That is because our laboratory uses a different toolchain (Xilinx) than that of CIAR project (Quartus), which imply an incompatibility between the two designs.

Most of the estimations fall within a 15% margin of error with respect to the actual hardware measurement, which is considered acceptable seeing the order of magnitudes involved. However, there appears to be two major sources of errors in our estimations: dynamic power estimation and DSP usage estimation. First, the substantial error for DSP usage estimation can be explained by the difficulty of predicting the low-level Vivado optimizations. Indeed, Vivado is able to multiplex several MAC operations in a single DSP or use LUT and Registers instead. Moreover, the effect of DSP saturation is difficult to extrapolate from layer-level to network-level. However, for both
VGT-PADS and VGT, DSP usage is under-estimated while LUT and Register usage is over-estimated: thus, we assume that Vivado used LUTs and Registers in place of some DSPs in the design. However, the two errors compensate each other (DSPs are used instead of LUTs). Overall, the DSP usage is difficult to estimate due to saturation, multiplexing and alternative synthesis optimizations. To cope with this problem, the hardware database must be rebuilt with no DSP constraints during synthesis. In doing so, we aim in obtaining a flat resource usage estimation that is less influenced by low-level optimizations. Moreover, a source of error arises from the approximation made on VGT convolution layers. In Chapter 5.2.1.1, we explained that the estimations are obtained for 1 input channel, and multiplied by the number of channels. This approximation is valid for resource estimations, but not for power. Indeed, synthesis performs low-level optimization when all channels are synthesized at once. Therefore, the database should be extended to take into account the number of input channels in VGT convolution layers (3D design space).

Second, the substantial error in dynamic power estimation is related to the same kind of issue. Indeed, the network-level dynamic power estimation is computed as the sum of layer-wise estimations. However, Vivado performs low-level optimization such as resource and clock sharing during synthesis. As clocks are responsible for a very significant proportion of dynamic power at layer-level (around 30% in average), clock sharing brings significant dynamic power reduction at network-level. Therefore, the dynamic power estimations demonstrate a wide margin of error compared to actual measurements. In turns, this also affects the energy estimations derived from power and time. However, the conclusions derived from estimations in the previous subsection are still valid, as the comparisons are not too much affected by this error: VGT and VGT-PADS still demonstrate a similar energy consumption, while SPLEAT is much more energy intensive. Therefore, those results strengthen the confidence in the resource and time estimation results, and confirm the benefits of hybridization. However, the dynamic power estimation must be improved to reduce the error margin. This will be addressed in the outlooks section of this Chapter (Section 6.4.1).

## 6.4 Conclusion

Estimations and hardware measurement results have shown that hybridization represents an interesting trade-off between formal and spiking implementations. It is the most useful in topologies such as Spoken Digits, where the classification stage represents most of VGT energy consumption. On this dataset, VGT-PADS hybrid architecture provides 30% energy savings over pure VGT architecture. It uses 40% fewer LUTs, 80% fewer DSPs and consumes 43% less power. Those substantial savings are promising for the field of embedded artificial intelligence. On OPS-SAT dataset, the classification stage is less significant, thus resource and energy savings are less significant. As explained before, this study focuses on a precise type of architecture, where the feature extraction stage operates in formal domain while the classifier is works with spikes. PADS should be extended to convolution and pooling layers to enable hybridization earlier in the network. The optimal hybridization scheme must be determined for each application in further work, by exploring the trade-off between resource savings and energy savings. More details are given in the outlooks section (Section 6.4.1). Moreover, this study shows that hybridization increases the parallelism and pipeline requirements in the spiking classifier, as spikes arrive more frequently due to the lower temporal sparsity of encoding. This is particularly visible in Spoken Digits results where SAR is high. In such context, SPLEAT demonstrates a severe latency overhead despite the
reduction of temporal sparsity. Therefore, the level of parallelism must be fine-tuned according
to the actual temporal sparsity of spikes and SAR in further works. This will be detailed in the
outlooks section.

Additionally, the confrontation of estimations on hardware measurements for VGT, VGT-PADS and SPLEAT applied to OPS-SAT dataset strengthen the confidence in the estimation framework, and confirms the benefits of hybridization. Indeed, most metrics where estimated with a margin of error of 15%. Dynamic power estimations suffered severe errors due to unaccounted low-level optimizations performed by Vivado, but all qualitative comparisons remain valid. The improvement of power estimations will be addressed further in the outlooks. Moreover, those results strengthen our confidence in the conclusions drawn from comparisons of formal and spiking accelerators in Chapter 5.

6.4.1 Outlook

In order to improve the estimation framework, we intend to rebuild the hardware footprint
database in future work. That is, by using a specific synthesis policy which does not limit the
number of available resources. In doing so, interpolation will provide more accurate results, es-
pecially for DSP usage. Indeed, such flat resource estimations are less influenced by low-level
synthesis optimizations of Vivado. Moreover, we propose to analyze the clock sharing optimiza-
tions performed by Vivado when combining layers at network-level. In doing so, we intend in
refining the dynamic power estimation and provide more accurate absolute values. Moreover, as
explained in Chapter 5, the estimator can be used to determine the optimal level of parallelism in
spiking accelerators. This perspective is even more appealing for hybridization where parallelism
requirements are higher.

Finally, another way to improve the energy efficiency of hardware SNNs is to addressed new
type of spike coding which demonstrate lower temporal sparsity and SAR. In Chapter 7, we will also
address such new representation with SNNs trained through Surrogate Gradient Learning (SGL)
[32] [73]. In addition to lower SAR, this type of SNN is timestep-constrained, thus inference time
does not depend on synaptic activity but rather on a user defined hyper-parameter. Such feature
could increase the appeal for hardware SNN accelerators as it facilitates the timing management
at system level. The SAR for SG learning will be measured on the dataset benchmark, and PADS
will be adapted to this new SNN model.
# Chapter 7

## Enhancing PADS: FISO & LIF as Recurrent neurons

### Chapter contents

<table><tr><td>7.1 Theoretical Background</td><td>130</td></tr><tr><td>7.1.1 Send on Delta spike encoding</td><td>130</td></tr><tr><td>7.1.2 LIF Neuron</td><td>131</td></tr><tr><td>7.1.3 LIF as recurrent neurons</td><td>132</td></tr><tr><td>7.1.4 Surrogate Gradient Learning</td><td>133</td></tr><tr><td>7.1.5 Output decoding: readout layer</td><td>133</td></tr><tr><td>7.1.6 Application in the S2NET framework</td><td>133</td></tr><tr><td>7.1.7 Static input samples</td><td>134</td></tr><tr><td>7.2 Accuracy & SAR results</td><td>134</td></tr><tr><td>7.3 PADS V2</td><td>138</td></tr><tr><td>7.3.1 Architecture</td><td>138</td></tr><tr><td>7.3.2 Inference Time Results</td><td>141</td></tr><tr><td>7.4 Conclusions & Outlooks</td><td>144</td></tr></table>
In this chapter we address a novel representation of SNNs proposed by Neftci et. al. in [32]. This new type of SNNs represents Leaky Integrate and Fire (LIF) neurons as Recurrent Neurons with binary outputs and inputs. Indeed, LIF and Recurrent Neurons share major similarities such as their similar architecture, temporal dynamics and training through weight adjustment. Considering the SNN as an RNN enables to use existing and mature training methods, such as the well-established Backpropagation Through Time (BPTT) algorithm. This algorithm is adapted to the LIF neuron activation function using the Surrogate Gradient Learning (SGL) technique [32] [91]. This method has two main advantages: first SNNs trained through Surrogate Gradient Learning demonstrated lower spiking activity and inference time for comparable accuracy than converted rate-coded SNNs [73]. Moreover, training in spiking domain enables to penalize spiking activity during training. This novel model of SNNs is promising for hardware acceleration. Indeed, the previous chapters of this thesis have shown that minimizing spiking activity (i.e. SAR) and inference time is a crucial matter to leverage energy savings in neuromorphic accelerators.

The SNN model proposed in [32] is described in details in Section 7.1. We also describe the S2NET framework proposed by Zimmer et. al. in [21]. This pytorch-based training framework implements the Surrogate Gradient Learning technique alongside Send-on-Delta spike encoding. It is used to measure accuracy and SAR on the benchmark of datasets (MNIST, OPSSAT, GTSRB, CIFAR-10, Mines VS Rocks, Spoken Digits and RadioML 2018) in Section 7.2. The results are confronted with that of rate-coded SNNs obtained in Chapter 4. Lastly, we propose a first prototype of hardware implementation of such SGL-trained SNN model. The architecture is based on PADS and adapted to the LIF dynamics of S2NET.

# 7.1 Theoretical Background

In this section, we describe the SNN model proposed in [21]. Rather than rate-encoding like in previous chapters of this thesis, the authors used the Send-on-Delta [33] technique which tailors temporal sampling to the signal gradient. This encoding scheme is described in Section 7.1.1. Moreover, the model is based on the equivalence between LIF neurons and recurrent neural cells. The LIF neuron dynamics are described in Section 7.1.2, and the equivalence with recurrent neurons is discussed in Section 7.1.2. Thanks to this equivalence, SNNs can be trained using algorithms developed for Recurrent Neural Networks (RNN) such as the Backpropagation Through Time (BPTT) algorithm. In section 7.1.4, we describe the Surrogate Gradient Learning technique proposed in [32]. This algorithm is an adaptation of the BPTT to the LIF activation function. The output spike decoding process is described in Section 7.3.1.4. In Section 7.1.6, we describe the simplification to the model brought by the S2NET framework. Since this framework and the SNN model were originally intended for temporal signal processing, Section 7.1.7 describes how the model is adapted to static samples.

## 7.1.1 Send on Delta spike encoding

The Send-on-Delta (SoD) spike encoding method is inspired from [33], in which Miskowicz et. al. proposed a strategy for information encoding in event-based sensors. This method is adapted in the S2NET framework which is used in this chapter. It is based on a signal-dependent sampling scheme which is able to represent information in fewer spikes (sparse coding) than conventional rate-based methods. According to the authors and to the conclusions of this thesis so far, such a feature could provide substantial energy savings in clock-gated or power-gated hardware SNN
<figure><img src="image_78.png" /><figcaption>Figure 7.1: Illustration of the Send-on-Delta spike encoding process used in [21], for 2 IF neurons encoding a 1D temporal signal. Left: network setup. Right: input and output data.</figcaption></figure>

implementations (see outlooks of Chapter 5). The SoD method is based on lateral connections between spiking neurons. In Figure 7.1, the SoD method is illustrated for a simplistic case where 2 Integrate & Fire (IF) neurons encode a 1D temporal signal. The goal of SoD is to detect increase or decrease in the input data. The IF neuron is able to detect an increase in input signal through a threshold mechanism and membrane potential reset. In Figure 7.1, the threshold is noted Θ. The green neuron detects increases thanks to a positive synaptic weight whereas the red neuron detects decreases thanks to a negative weight. As shown in the figure, the sampling rate is not arbitrary fixed but depends on the temporal variations. Moreover, the lateral connections enables to reset the membrane potential of both neurons whenever one of them emits a spike.

This mechanism can be generalized to populations of N neurons to process temporal input signals with N dimensions. For example, this mechanism can be applied to layers of Gabor filters [111] which are used to detect edges in 2D images. In this chapter, the SoD encoding is used in all layers of the networks. That is, the first layer of neurons performs spike encoding of analog inputs through SoD, and the following layers of LIF neurons do the same of spiking inputs. Additionally, the parameters of SoD (synaptic weights and neuron threshold) are learned during training. The resulting spike encoding scheme is therefore optimized to minimize spiking activity and maximize accuracy.

### 7.1.2 LIF Neuron

The LIF neuron is similar to the IF neuron used in the previous chapter of this thesis. The difference lies in the leakage of the membrane potential responsible for the Leaky Integrate & Fire appellation. The dynamics of this neuron illustrated in Figure 7.2 and described in Equation 7.1:

$$ \tau_{mem} \frac{dU_i^l(t)}{dt} = -(U_i^l(t) - U_{rest}) + R I_i^l(t) \quad (7.1) $$

Where $U_i^l$ is the membrane potential of the i'th neuron of layer l, $τ_{mem}$ is the membrane time constant, $U_{rest}$ is the resting potential, R the input resistance and I the input current. Whenever the membrane potential overpasses the neuron threshold $θ$, a spike is emitted in output and the
<figure><img src="image_79.png" /><figcaption>Figure 7.2: Illustration of the Leaky Integrate & Fire neuron process. The membrane potential is shown in blue, input spikes in green and output spikes in red.</figcaption></figure>

membrane potential is instantaneously decreased by $\theta - U_{\text{rest}}$. The output spike train is noted $S_i^l(t) = \Theta(U_i^l(t) - \theta)$ where $\Theta$ is the Heaviside step function. Incorporating this threshold term in Equation 7.1 yields:

$$ \frac{dU_i^l(t)}{dt} = -\frac{1}{\tau_{\text{mem}}}((U_i^l(t) - U_{\text{rest}}) + R I_i^l(t)) - S_i^l(t)(\theta - U_{\text{rest}}) \quad (7.2) $$

The input current is generated by input spikes coming from neurons of the layer $l-1$ as described in Equation 7.3.

$$ I_i^l(t) = \sum_j w_{i,j} \times S_j^{l-1}(t) + \sum_j v_{i,j} \times S_j^l(t) \quad (7.3) $$

Where $\sum_j w_{i,j} \times S_j^{l-1}(t)$ is the feed-forward term (influence of input spikes) and $\sum_j v_{i,j} \times S_j^l(t)$ is the recurrent term for lateral connections (see Section 7.1.1). $w_{i,j}$ and $v_{i,j}$ are the synaptic weights of the input and recurrent synapses respectively. In all, the LIF neuron is able to process temporal information thanks to the membrane potential leakage. Indeed, the relative timing of input spikes has no influence on the IF neuron potential, whereas it does in the LIF neuron. This features also decreases the number of spikes in the networks, as all membranes are constantly decreasing.

### 7.1.3 LIF as recurrent neurons

The solution of the differential equation 7.1 can be approximated on discrete time through the Euler Method as shown in Equation 7.4. Moreover, we set $U_{\text{rest}} = 0$ and $R = 1$.

$$ U_i^l[t+1] = e^{-\frac{\Delta t}{\tau_{\text{mem}}}} \times (U_i^l[t] - S_i^l[t] \times \theta) + (1 - e^{-\frac{\Delta t}{\tau_{\text{mem}}}}) \times (\sum_j w_{i,j} \times S_j^{l-1}[t] + \sum_j v_{i,j} \times S_j^l[t]) \quad (7.4) $$

Where $\Delta t$ is the time increment between two timesteps. According to Neftci et. al. [32], those equations precisely describe a Recurrent Neural Network with binary output and whose activation
function is an Heaviside (threshold) function. Thanks to this equivalence, the network of LIF neurons can be trained using the well-established RNN training methods, such as the BPTT algorithm. The BPTT algorithm is an adaptation of the static BP algorithm for dynamic pattern recognition used in formal RNNs. However, the Heaviside activation function is not adapted to the BPTT algorithm. In the next subsection, we describe how BPTT is adapted to networks of LIF neurons through Surrogate Gradient Learning.

### 7.1.4 Surrogate Gradient Learning

In the BPTT algorithm, the adjustment of synaptic weights is proportional to the partial derivatives of the activation function. Since the activation function of the LIF neuron is an Heaviside step-function, two issues arise:

* First, the Heaviside function is not derivable in 0,

* Second and most importantly, the derivative of the Heaviside function is 0 on $R*$.

* In all, the weight adjustment factor cannot be computed in zero, and it is equal to zero everywhere else.

To cope with these issues, Neftci et. al. [32] proposed to use an approximation of the Heaviside function in the backpropagation algorithm: a sigmoid function. In other words, the activation function is a sigmoid during the backward path, and an Heaviside during the forward path. Using this technique, several papers in the literature have demonstrated state-of-the-art classification accuracy with reduced synaptic activity [73] [32] [91] [21]. As explained before, training in spiking domain enables to optimize the spiking activity: the number of spikes can be penalized at network-level by taking it into account in the loss function. In the S2NET framework [21], all layers of the networks are trained using the Surrogate Gradient Learning technique. That is, including the first layer which performs SoD encoding of analog input data.

### 7.1.5 Output decoding: readout layer

Finally, the output spiking activity is decoded through a readout layer. This layer is made of non-firing neurons with no lateral connections. In classification tasks, each class is associated with an output neuron like for Terminate Delta (Chapter 2). The probability of each class is defined by the normalized average membrane potential of the corresponding neuron. This method has been proposed by Zimmer et. al. in [21] as a simplification of the model developed by Neftci in [32], which implied a softmax function. According to the authors, the training is more stable with an average function. Moreover, the latter is more hardware-friendly than the complex softmax function.

### 7.1.6 Application in the S2NET framework

In the next section, we use the S2NET framework proposed by Zimmer et. al. in [21], which implements the aforementioned SNN model and Surrogate Gradient Learning in PyTorch. Using this framework, we measure accuracy and SAR on the benchmark of datasets. The results will be confronted to that of rate-coded converted SNNs. In this framework, the neuron threshold ($\theta$) is a trainable parameter which is common to the layer. The neuron threshold is analogous to $\Delta$
<table><caption>Table 7.1: Parameters used for training with S2NET framework</caption><thead><tr><th>Parameter</th><th>Value</th></tr></thead><tbody><tr><td>Sigma value</td><td>5</td></tr><tr><td>Learning Rate</td><td>0.0005</td></tr><tr><td>Weight Decay</td><td>0.00004</td></tr><tr><td>Reg. Loss Coef.</td><td>0</td></tr><tr><td>Optimizer</td><td>RAdam</td></tr><tr><td>Gamma</td><td>0.8</td></tr><tr><td>Step Size</td><td>3</td></tr><tr><td># Epochs</td><td>100</td></tr></tbody></table>

in the Send-on-Delta encoding method, but we use $\theta$ for the sake of consistency with previous chapters. Moreover, the exponential decay of the membrane potential is approximated by a linear decay ($\beta$). This parameter is also trained through Surrogate Gradient Learning, and shared among all neurons of a layer. Finally, in the S2NET framework uses a threshold balancing technique is applied: the neuron threshold $\theta$ is normalized by the norm of the synaptic weight matrix. The specific LIF dynamics implemented in the S2NET framework are summarized in Equation 7.5:

$$ 
\begin{aligned}
U_i^l[t+1] &= (U_i^l[t] - S_i^l[t] \times \theta^l \times B_i^l) \times \beta^l + (\sum_j w_{i,j} \times S_j^{l-1}[t] + \sum_j v_{i,j} \times S_j^l[t]) \times (1 - \beta^l) \\
S_i^l[t] &= \Theta(U_i^l[t] - \theta^l \times B_i^l) \\
B_i^l &= ||w_{i,j}||^2
\end{aligned} 
\qquad (7.5) $$

Where $U_i^l[t]$ is the membrane potential of neuron $i$ of layer $l$ at timestep $t$, $S_j^{l-1}[t]$ is the state of the input synapse coming from neuron $j$ of layer $l-1$ at time $t$, $w_{i,j}$ is the synaptic weight between neuron $i$ and $j$, $\beta^l$ is the membrane decay factor of layer $l$, $\theta^l$ the threshold of neurons in layer $l$ and $\Theta$ is the Heaviside function. The threshold is normalized by $B_i^l$ to balance weights and threshold. It should be noted that $w_{i,j}$, $\beta^l$ and $\theta^l$ are trainable parameters, the two latter being common to all neurons of a same layer.

### 7.1.7 Static input samples

The SNN model described in this section was originally developed to process temporal data, such as time series, video recordings or data from dynamic vision sensors. In practice, the continuous time is discretized in timesteps like shown in Equations 7.4. Moreover, the model is also applicable to static data like the one used in our benchmark of dataset (Chapter 4). To do so, the same input sample is presented for several successive time-steps. The number of timesteps will define the spike encoding window for all layers of the neural network, i.e. If the input sample is presented for five timesteps, spikes will be encoded on five timesteps in all successive layers.

## 7.2 Accuracy & SAR results

The model described in previous section (Section 7.1) is applied to the benchmark of datasets: MNIST, GTSRB, CIFAR-10, Mines Vs. Rocks, Spoken Digits and RadioML 2018. For each dataset, we measure accuracy and SAR for various number of timesteps. As explained before, the
<table><caption>Table 7.2: Comparison of accuracy and network-wise SAR with Conversion with Rate-coding and SGL with SoD. Bold letters show the best-case accuracy for each method and task.</caption><thead><tr><th rowspan="2">Task</th><th colspan="3">Conversion \& Rate</th><th colspan="3">SGL \& SoD</th></tr><tr><th>Δ</th><th>SNN ACC. (%)</th><th>SAR</th><th>TS</th><th>SNN ACC. (%)</th><th>SAR</th></tr></thead><tbody><tr><td rowspan="3">MNIST<br>λ<sub>ZE</sub>=4.2 / λ<sub>ZC</sub>=2.7</td><td>5</td><td>97.3</td><td>0.58</td><td>5</td><td>98.5</td><td>0.60</td></tr><tr><td>10</td><td>98.9</td><td>0.82</td><td>10</td><td>98.6</td><td>1.41</td></tr><tr><td>20</td><td>99.1</td><td>1.24</td><td>20</td><td>98.6</td><td>3.12</td></tr><tr><td rowspan="3">GTSRB<br>λ<sub>ZE</sub>=4.4 / λ<sub>ZC</sub>=5.2</td><td>5</td><td>83.3</td><td>2.24</td><td>5</td><td>96.7</td><td>1.18</td></tr><tr><td>10</td><td>90.6</td><td>3.59</td><td>10</td><td>96.8</td><td>2.33</td></tr><tr><td>20</td><td>93.5</td><td>7.70</td><td>20</td><td>96.6</td><td>4.92</td></tr><tr><td rowspan="3">CIFAR-10<br>λ<sub>ZE</sub>=4.4 / λ<sub>ZC</sub>=5.4</td><td>5</td><td>31.8</td><td>0.94</td><td>5</td><td>53.4</td><td>0.82</td></tr><tr><td>10</td><td>50.2</td><td>2.57</td><td>10</td><td>53.7</td><td>1.69</td></tr><tr><td>20</td><td>57.8</td><td>7.57</td><td>20</td><td>53.7</td><td>3.58</td></tr><tr><td rowspan="3">Mines V Rocks<br>λ<sub>ZE</sub>=4.0 / λ<sub>ZC</sub>=1.5</td><td>5</td><td>74.2</td><td>0.58</td><td>5</td><td>83.8</td><td>1.48</td></tr><tr><td>10</td><td>75</td><td>1.07</td><td>10</td><td>83.3</td><td>3.11</td></tr><tr><td>20</td><td>78.8</td><td>3.34</td><td>20</td><td>75.7</td><td>6.42</td></tr><tr><td rowspan="3">Spoken Digits<br>λ<sub>ZE</sub>=4.4 / λ<sub>ZC</sub>=5.4</td><td>5</td><td>38.2</td><td>2.04</td><td>5</td><td>79.7</td><td>1.24</td></tr><tr><td>10</td><td>71.6</td><td>6.41</td><td>10</td><td>80.7</td><td>2.94</td></tr><tr><td>20</td><td>87.5</td><td>8.24</td><td>20</td><td>81.5</td><td>6.49</td></tr><tr><td rowspan="3">RadioML 2018<br>λ<sub>ZE</sub>=4.4 / λ<sub>ZC</sub>=5.7</td><td>5</td><td>20.5</td><td>6.20</td><td>5</td><td>58.4</td><td>0.98</td></tr><tr><td>10</td><td>31.6</td><td>11.45</td><td>10</td><td>60.0</td><td>2.21</td></tr><tr><td>20</td><td>37.5</td><td>17.41</td><td>20</td><td>60.7</td><td>4.85</td></tr></tbody></table>
<figure><img src="image_80.png" /><figcaption>Figure 7.3: SAR and Accuracy measurements on the benchmark of datasets with S2NET framework. Samples are presented for 5, 10 and 20 time-steps. The λ value is depicted in orange for the Zedboard and red for the ZCU102 (see Section 4.3.2). All measurements are averaged on 10 runs.</figcaption></figure>
static samples are presented for a given number of time-steps. This number defines the width of the spike-encoding window. In other words, if the sample is presented for three timesteps, spikes are encoded on three timesteps in all the network. Since a larger time-window enables to encode richer information using more spikes, we suspect a trade-off between SAR and accuracy which can be leveraged by tuning the number of timesteps. The results of this set of experiments is shown in Figure 7.3. All values are averaged on 10 runs. The same parameters listed in 7.1 are used for all tasks to enhance the intrinsic influence of the data. It should be noted that OPS-SAT is not covered in this study due to technical issues.

We begin by comparing the network-wise SAR and accuracy obtained with S2NET with the previous results obtained with Conversion and rate-encoding using N2D2 (Chapter 4). In order to facilitate the discussion, the network-level results are summarized in Table 7.2. As a reminder, the SAR is considered suitable for spiking acceleration when it is below the λ value, according to the model proposed in Section 4.3.2. λ represents the ratio of energy consumption between a MAC operation and an ACC operation on FPGA. It depends on the target board and on the baseline formal design (VGT formal parallel accelerator). In Table 7.2, we provide λ<sub>ZC</sub> for the ZCU102 board, and λ<sub>ZE</sub> for the Zedboard. The SAR is displayed in green if it is below both λ values, orange between and red over. Moreover, the best-case accuracy is displayed in bold letters to enhance comparison between the two training methods. According to the results, Surrogate Gradient Learning does not always provide better accuracy than conversion. The latter indeed performs better on MNIST, CIFAR-10 and Spoken Digits datasets. However, the same default training parameters (Table 7.1) were used for all S2NET training runs. That is to enhance the comparison between datasets, but a fine tuning could provide better classification accuracy in further works. Indeed, Surrogate Gradient Learning is able to reach state-of-the-art accuracy according to the literature. On the other hand, SGL performs significantly better than conversion on GTSRB and RadioML 2018 datasets. On the latter, the best-accuracy is nearly two times higher with Surrogate Gradient Learning.

However, the main matter of this study is not accuracy but Synaptic Activity Ratio (SAR). According to the results, SGL achieves substantially lower SAR than conversion for most datasets. That is apart from MNIST, in which SAR are very close for both methods. As depicted in Table 7.2, the best-case accuracy for conversion is reached for SAR deemed unsuitable for spiking acceleration (red). Using SGL, not only a better accuracy is reached, but the SAR is also suitable for spiking acceleration (green). In Spoken Digits, the best case accuracy with SGL is associated to a red SAR. However, the SAR becomes favorable to spikes if the number of timesteps is reduced from 20 to 10. Under such conditions, the model only loses 1% accuracy compared to the best-case. For Conversion, the only configuration suitable for hardware acceleration is Δ = 5, but the accuracy is only 38.2%. In all, Surrogate Gradient Learning and Send-on-Delta encoding are more adapted to hardware acceleration than conversion and rate-coding. Furthermore, increasing the number of timesteps does not always provide a better accuracy. For MNIST, CIFAR-10 and GTSRB, increasing the number of timesteps over 10 degrades accuracy. Therefore, increasing the number of timesteps arbitrarily is counter-productive and each model should be tuned experimentally.

Additionally, the reduction of synaptic activity is not the only advantage of the SNN model proposed in the S2NET framework. As explained before, the spikes are encoded on a time-window with fixed length. This is a major difference with rate encoding in which the number of timesteps is non deterministic and depends on spiking activity and Terminate Delta termination condition. For hardware acceleration in highly parallel architectures, this means that the inference time is fully deterministic. The integration of such hardware accelerator in wider systems is therefore
facilitated by the easier management of timings. Moreover, the timestep constraint of this SNN model enables to tune the width of the spike encoding window. On the other hand, the window width is data dependent in rate-coding: it cannot be tuned and can grow to very large numbers in some cases. This matter will be studied at the end of the next section.

## 7.3 PADS V2

In this section, we propose an hardware architecture dedicated to the acceleration of SNN inference, using the model described in the previous section (7.1). This architecture is based on PADS: it is fully parallel and pipelined. Moreover, only fully-connected layers are supported. In the following, the previous version of PADS is referred to as PADS-V1, and the new iteration as PADS-V2. The architecture is described in details in Section 7.3.1. Then, we confront PADS-V1 and PADS-V2 in terms of inference time on a few examples. The aim of this study is to determine if this SNN model used in recent frameworks such as S2NET [21] could provide lower inference time than rate-coded converted SNNs in FPGA accelerators.

### 7.3.1 Architecture

The architecture is made of three distinct modules, each of which is based on the Generic Neural Processing Unit of PADS-V1. Those modules are:

* The Formal Input Spiking Output (FISO) layer to encode pixels into spike trains,

* The LIF layer made of LIF neurons for hidden layers

* The Readout layer made of non-firing neurons and a dedicated module to compute the maximum membrane potential value.

The architecture is fully-parallel, meaning that each neuron is physically implemented in hardware like in PADS-V1. Moreover, the architecture features an end-to-end pipe-line from spike-encoding (FISO layer) to spike-decoding (Readout layer) to reduce inference time. After describing each module in a dedicated subsection below, the last paragraph describes the mechanism of pipe-line synchronization implemented in PADS-V2. It should be noted that this feature was not implemented in PADS-V1. Before going into further details, we provide some simplifications of the model in the first subsection.

#### 7.3.1.1 Simplification

First, the lateral connections are not implemented in the current stage of development. However, the S2NET network enables to deactivate lateral connection for training and testing. Without lateral connections, the accuracy is slightly degraded and the number of spikes is higher, but the model is still functional. This feature is used to validate the architecture in its current state of development. Moreover, the membrane potential equations of S2NET are expressed differently to facilitate hardware implementation. The goal of those simplifications is to enable as much off-line computation as possible. In doing so, we avoid costly hardware operations such as multiplication and divisions. As a reminder, the original model proposed in [21] (without lateral connections) is
shown in Equations 7.6:

$$
\begin{align}
U_i^l[t+1] &= (U_i^l[t] - S_i^l[t] \times \theta^l \times B_i^l) \times \beta^l + \left(\sum_j w_{i,j} \times S_j^{l-1}[t]\right) \times (1 - \beta^l) \tag{7.6} \\
S_i^l[t] &= \Theta(U_i^l[t] - \theta^l \times B_i^l) \nonumber \\
B_i^l &= ||w_{i,j}||^2 \nonumber
\end{align}
$$

Where $U_i^l[t]$ is the membrane potential of neuron $i$ of layer $l$ at timestep $t$, $S_j^{l-1}[t]$ is the state of the input synapse coming from neuron $j$ of layer $l-1$ at time $t$, $w_{i,j}$ is the synaptic weight between neuron $i$ and $j$, $\beta^l$ is the membrane decay factor of layer $l$, $\theta^l$ the threshold of neurons in layer $l$ and $\Theta$ is the Heaviside function. The threshold is normalized by $B_i^l$ to balance weights and threshold. It should be noted that $w_{i,j}$, $\beta^l$ and $\theta^l$ are trainable parameters, the two latter being common to all neurons of a same layer. The equation of the membrane potential can be expressed in a different manner:

$$
\begin{align}
U_i^l[t+1] &= U_i^l[t] \times \beta^l - S_i^l[t] \times \theta^l \times B_i^l \times \beta^l + \left(\sum_j w_{i,j} \times (1-\beta^l) \times S_j^{l-1}[t]\right) \nonumber \\
S_i^l[t] &= \Theta(U_i^l[t] - \theta^l \times B_i^l) \tag{7.7} \\
B_i^l &= ||w_{i,j}||^2 \nonumber
\end{align}
$$

In this equation, a lot of multiplications can be made off-line. That is the case for the following products: $\theta^l \times B_i^l \times \beta^l$, $w_{i,j} \times (1 - \beta^l)$, $\theta^l \times B_i^l$ and $||w_{i,j}||^2$. Those values are therefore pre-processed and either stored in RAMS (synaptic weights $w_{i,j} \times (1 - \beta^l)$) or hard-coded in the neuron. Deporting those operations off-line enables to relieve the number of on-line operations supported by the programmable logic.

Moreover, the Readout layer process is also simplified. In the S2NET model, the Readout layer is made of non-firing leaky neurons. At the end of processing, the winning neuron is selected as the one whose average membrane potential is the highest. It should be noted that this is a temporal average across the discrete time window. However, the goal of the Readout layer is mostly to determine the most active output neuron. The same thing can be achieved by using non-leaky in addition to non-firing neuron, and finding the maximum total membrane potential rather than the maximum average. In doing so, the architecture is relieved from costly multiplication (leakage) and division (average) operations.

### 7.3.1.2 Formal Input Spiking Output layer

The FISO layer is designed to encode input data into spike trains according to the Send-on-Delta method (Section 7.1.1), except it does not feature lateral connections to inhibit all neurons when a spike is emitted. In contrast with the GenCell used in the previous chapter, the FISO layer combines both the spike encoding process and the first layer of neurons. This implementation relieves the architecture from the overhead imputed to the dedicated spike generation module. It should be noted that the concept of FISO neuron was first proposed during the early stages of the thesis. At that time, we were not aware of the S2NET-like SNN models, and the FISO neuron was intended to facilitate hybridization between Formal Neural Networks and rate-coded SNNs. This led to a patent filing in September 2019 concerning the concept of hybrid neural processing unit.

The FISO layer is implemented in parallel: there is one FISO neuron for each input neuron in the model. The architecture of the FISO neuron is depicted in Figure 7.4a. It derives closely from
<figure><img src="image_81.png" /><figcaption>Figure 7.4: Hardware architecture of: a) FISO neuron with 4 input pixels, b) LIF neuron with 4 input synapses. Register barriers are shown in gray or colored rectangles. Data stored in memory is depicted in light blue squares.</figcaption></figure>

the Generic Neural Processing Unit (NPU) of PADS-V1. We only address the differences between the FISO neuron and the PADS-V1 NPU in this section. For more information on the hardware implementation, refer to the description of PADS-V1 NPU in Section 3.1.2. The main differences are the following: first, the weight affectation stage, which is replaced by a weight multiplication stage. Indeed, the integration of analog input requires to multiply weight with input activations. Second, the membrane potential decay: the membrane potential (orange register in Figure 7.4a) is multiplied with the decay factor ($\beta$) before weight accumulation (blue and red registers). The other differences have no influence on the implementation: the multiplication of synaptic weights with $1 - \beta$ is performed off-line, and so are the products $\theta \times B \times \beta$ and $\theta \times B$. Moreover, the architecture features an *enable* port to ensure synchronization in the pipeline. This mechanism is described in a dedicated paragraph at the end of this section.

### 7.3.1.3 LIF layer

The LIF neuron is similar to the FISO neuron, except that it features a weight affectation stage like in the original PADS-V1, since the inputs are binary signals. The architecture of the LIF neuron is given in Figure 7.4b. More detail on this implementation can be found in the description of PADS-V1 NPU in Section 3.1.2.

### 7.3.1.4 Readout layer

The Readout layer includes both the last layer of neurons and the selection of the winning class. It is illustrated in Figure 7.5a with 4 input synapses and 3 output neurons. The neurons are non-leaky and non-firing neurons which forwards their membrane potential to a comparator. The comparator is a pipelined tree of maximum operators designed to retrieve the maximum membrane potential (i.e. most active neuron). The *CLK* (clock), *RST* (reset) and *EN* (enable) signals are only depicted in the first readout neuron for the sake of simplicity.
<figure><img src="image_82.png" /><figcaption>Figure 7.5: a) Architecture of the Readout layer with 4 input synapses and 3 neurons. b) Overview of the full architecture of PADS V2.</figcaption></figure>

### 7.3.1.5 Pipeline & synchronization

A global overview of an MLP with three layers is depicted in Figure 7.5b. In order to ensure the synchronization of the pipeline, each layer is associated with a control unit. To better illustrate this matter, a chronogram of the network process is provided in Figure 7.6. The layers are synchronized using a *valid* signal. On the chronogram, the pipeline depth of the FISO layer is two clock cycles. Therefore, the FISO neurons starts processing input upon receiving the input *valid* signal (val⁰). When data reaches the end of the pipeline (after two clock cycles), the output *valid* signal (val¹) is raised. Moreover, the valid signal remains up for the duration of the time-window, e.g. three clock cycles for three time-steps in the chronogram. This mechanism is repeated in each successive layer to ensure an end-to-end pipeline from spike encoding (FISO layer) to decoding (Readout layer). It should be noted that the pipeline depth in Figure 7.6 have been arbitrarily fixed to two, three and four clock cycles. The real pipe-line depth can be calculated depending on the layer configuration, as described in the next section.

### 7.3.2 Inference Time Results

The PADS-V2 architecture proposed in the previous section enables drastic acceleration compared to PADS-V1. This is achieved by both the timestep constraint and the fully-parallel spike encoding. In PADS-V1, the Spike Generation Cell is a dedicated module which encodes information sequentially. The sequential choice was made to limit the resource overhead caused by spike encoding: preliminary experiments have shown that a fully-parallel Spike Generation Cell for OPS-SAT dataset already saturates the Zedboard resources. In PADS-V2 the spike encoding is parallel, but it is directly supported by the first layer of neurons (FISO layer). In doing so, the cost of parallelization is mitigated. Evaluation of the resource cost of PADS-V2 should be
<figure><img src="image_83.png" /><figcaption>Figure 7.6: Chronogram of the PADS-V2 pipelined process at network-level</figcaption></figure>

undertaken in further work. However, preliminary experiments shows that this implementation is much faster than PADS-V1. Both architectures are applied to two different tasks: a three-layer MLP (784-100-10) for MNIST classification, and a smaller MLP (80-10-2) for hybrid OPS-SAT classification (Chapter 6). The inference times are listed in Table 7.3.

Moreover, we provide the width of spike encoding window for both PADS-V1 and PADS-V2, for all datasets of the benchmark. Those values correspond to the best case accuracy of each method, and it is averaged on 10 samples. The results are shown in Table 7.4. In those results, the width of the spike encoding window is often smaller for rate-encoding in terms of timesteps. However, the sequential GenCell implies a much higher width in terms of clock cycle, since a timestep lasts for several clock-cycles (Chapter 3.1.1). As a reminder, parallelization of the Spike Generation Cell was found to be very resource-intensive in Chapter 3.1.1. Therefore, reducing the number of clock cycles for spike encoding through parallelization of the GenCell is not a viable solution. In PADS-V2 on the other hand, the width in timesteps is equal to the width in clock cycle, thanks to the fully-parallel spike encoding. In this case, parallelization is accessible since i) the spike generation is incorporated among a layer of LIF neurons ii) the SoD generation process is much simpler than rate coding in terms of hardware. Therefore, PADS-V2 spike encoding is temporally compact, reducing the overall inference-time by several order of magnitudes compared to PADS-V1. Moreover, the spike-encoding window width is a user-defined parameter in PADS-V2, whereas it is unpredictable and data-dependent in PADS-V1. Thus, the number of timesteps can grow out of control, like for RadioML-2018 in Table 7.4. In PADS-V2, the width of the window is fixed in advance, thus the problem does not arise. The variability of inference time in PADS-V1 is also a significant drawback because it complicates its integration in wider systems: in PADS-V2, the predictable inference time is much easier for timing management.

The preliminary experiments show that PADS-V2 is indeed much faster than PADS V1. That is thanks to the timestep-constrained representation of the SNN model, and to the end-to-end pipeline and fully-parallel implementation. Moreover, PADS-V2 architecture is much less influenced by input size than PADS-V1, as the spike encoding process is no longer dependent on input size. Thus, the new model of SNN seems much more adapted to hardware acceleration than rate-coded converted models studied in the previous chapters. Indeed, inference time was found to be the major source of energy overhead in spiking accelerators in Chapters 5 and 6. The timestep-constrained SNN model presented in this chapter, and the PADS-V2 architecture, could
<table><caption>Table 7.3: Inference time of PADS-V1 and PADS-V2 on MNIST and OPS-SAT hybrid</caption><thead><tr><th></th><th>PADS<br>V1</th><th>PADS<br>V2</th></tr></thead><tbody><tr><td>MNIST<br>784-100-10</td><td>~15 000 clk</td><td>39 clk</td></tr><tr><td>OPS-SAT (Hybrid)<br>80-10-2</td><td>~120 clk</td><td>27 clk</td></tr></tbody></table>

<table><caption>Table 7.4: Width of the spike encoding window in PADS-V1 and PADS-V2 on the benchmark of datasets, for best-case accuracy. The width is expressed in both timesteps and clock-cycles.</caption><thead><tr><th rowspan="3"></th><th colspan="4">Spike encoding window width</th></tr><tr><th colspan="2">PADS-V1<br>best acc.</th><th colspan="2">PADS-V2<br>best acc.</th></tr><tr><th>TS</th><th>CLK</th><th>TS</th><th>CLK</th></tr></thead><tbody><tr><td>MNIST</td><td>5</td><td>3920</td><td>10</td><td>10</td></tr><tr><td>GTSRB</td><td>5</td><td>15360</td><td>10</td><td>10</td></tr><tr><td>CIFAR-10</td><td>14</td><td>43008</td><td>10</td><td>10</td></tr><tr><td>MVR</td><td>9</td><td>540</td><td>5</td><td>5</td></tr><tr><td>Spoken Digits</td><td>10</td><td>5070</td><td>20</td><td>20</td></tr><tr><td>RadioML 2018</td><td>80</td><td>245760</td><td>20</td><td>20</td></tr></tbody></table>

therefore leverage significant energy savings. However, further work is required to deploy and test the architecture in hardware, in order to evaluate the resource, power and energy consumption of this architecture. Moreover, PADS-V2 is a prototype architecture and an operational design should involve a lower level of parallelism.

Additionally, the inference time in PADS-V2 is fully deterministic and only depends on the layer configuration. That is a major difference with PADS-V1, in which inference time is unpredictable and variable from one sample to another. This features facilitates the implementation of PADS-V2 in wider systems, easing timing management between the components. The inference time of PADS-V2 depends on the pipeline depth of each layer, as shown in the chronogram of Figure 7.6. The total execution time is expressed in Equation 7.8:

$$ \Delta T^{\text{Total}} = \Delta T_{\text{pipe}}^{\text{FISO}} + \sum_{j}^{N^{\text{hidden}}} \Delta T_{\text{pipe}}^{\text{j-LIF}} + \Delta T_{\text{pipe}}^{\text{RD}} + N^{\text{ts}} \quad (7.8) $$

Where $\Delta T^{\text{Total}}$ is the total execution time (in clock cycles), $\Delta T_{\text{pipe}}^{\text{layer}}$ is the pipe-line depth (in clock cycles) of a given layer, $N^{\text{hidden}}$ is the number of hidden layers of LIF neurons and $N^{\text{ts}}$ is the number of time-steps for sample encoding. For FISO and LIF layers, the pipeline depth is computed as shown in Equation 7.9. It is the depth of the adder-tree, plus three clock cycles for threshold, firing and leakage.

$$ \Delta T_{\text{pipe}}^{\text{FISO-LIF}} = \log2(N^{\text{input}}) + 3 \quad (7.9) $$

Where $N^{\text{input}}$ is the number of input synapses. The pipeline depth of the readout layer is computed like shown in Equation 7.10. This time, the processing is divided in two separate pipe-line. For