# FPGA-based CNN Acceleration using Pattern-Aware Pruning

Léo Pradels, Silviu-Ioan Filip, Olivier Sentieys, Daniel Chillet, Thibaut Le Calloch

<figure><img src="image_1.png" /></figure>

To cite this version:

Léo Pradels, Silviu-Ioan Filip, Olivier Sentieys, Daniel Chillet, Thibaut Le Calloch. FPGA-based CNN Acceleration using Pattern-Aware Pruning. AICAS 2024 - IEEE 6th International Conference on AI Circuits and Systems, Apr 2024, Abu Dhabi, United Arab Emirates. pp.228-232, 10.1109/AICAS59952.2024.10595865. hal-04689673

HAL Id: hal-04689673

https://inria.hal.science/hal-04689673v1

Submitted on 26 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# FPGA-based CNN Acceleration using Pattern-Aware Pruning

Léo Pradels

Univ. Rennes, Inria, CNRS, IRISA
F-35000 Rennes, France

Email: leo.pradels@inria.fr

Silviu-Ioan Filip

Univ. Rennes, Inria, CNRS, IRISA
F-35000 Rennes, France

Email: silviu.filip@inria.fr

Olivier Sentieys

Univ. Rennes, Inria, CNRS, IRISA
F-35000 Rennes, France

Email: olivier sentieys@inria.fr

Daniel Chillet

Univ. Rennes, Inria, CNRS, IRISA
F-35000 Rennes, France

Email: daniel.chillet@irisa.fr

Thibaut Le Calloch

Safran Electronics & Defense,
F-95610 Eragny, France

Email: thibaut.le-calloch@safrangroup.com

**Abstract**—While convolutional neural networks (CNNs) have demonstrated exceptional performance in computer vision, optimizing FPGA-based CNN accelerators remains a challenge due to resource constraints. This is especially true for sequential designs, which are limited by external memory access. Despite the benefits of sparsity, most existing sparse accelerators are sequential and memory-bound. We introduce an innovative dataflow CNN architecture enriched with structured sparsity through pattern pruning at tile level. In our approach, pattern pruning serves as a fine-tuning step, effectively reducing FPGA resource consumption, including memory and logic. Experimental results indicate better latency than other dataflow approaches, while maintaining competitive accuracy compared to state-of-the-art unstructured pruning methods. We demonstrate the versatility of our approach in image classification and super-resolution applications, where we achieve a consistent 30 frames per second across a wide range of image sizes on the Set5 dataset.

# I. INTRODUCTION & RELATED WORK

Convolutional Neural Networks (CNNs) achieve state-of-the-art results in many computer vision tasks (e.g., image restoration), but their embedded deployment is complex due to their computational, energy, and memory requirements.

While there are several compression techniques available (such as quantization and neural architecture search), pruning is an often-used approach to lessen the impact of the networks on the device. The goal of pruning is to zero-out parameters and/or activations with minimal impact on model accuracy. Techniques vary in granularity, with fine-grained methods [1] offering high accuracy, but at increased hardware complexity, and coarse-grained approaches [1] prioritizing hardware acceleration. Recent research emphasizes medium-grained sparsity that balances hardware acceleration and model accuracy. For instance, density bound block (DBB) methods [2]–[4] enforce a constant number of nonzero (NNZ) terms inside a block of weights, whereas pattern-based methods use recurring sparse shapes at the kernel level [5], [6].

There are two main accelerator families. Sequential Layer (SL) architectures [7]–[10] use a general and flexible hardware block (e.g. optimized for general matrix multiplication operations) that executes layers one after another. This flexibility

usually comes at a cost, since mapping computations to the hardware can lead to under-utilization and limitations due to external memory bandwidth. They are also characterized by energy-intensive DDR accesses [11]. Dataflow architectures [12]–[14] (a.k.a. inter-layer pipelining) on the other hand, compute in parallel (in a pipelined fashion) all the layers on a target FPGA. This kind of method reduces off-chip traffic, but the resulting designs are constrained by the number of layers that can be implemented in parallel due to the limited availability of on-chip memory, digital signal processing (DSP) blocks, and logic (LUTs and FFs) [15]. It is also challenging to schedule each layer to minimize on-chip footprint.

The majority of sparse accelerators are based on SL architectures [16], [17]. They handle a variety of pruning granularities, with a strong focus on unstructured sparsity [18], [19], which is naturally the most common (in weights or feature maps). Few options use structured pruning, such as pattern acceleration [20] or DBB [2], [21]. Some sparse pipeline accelerators have nonetheless been proposed recently. For instance, in [22], the focus is on unstructured pruning, but the approach requires complex additional hardware to decompress data, as well as extra metadata for sparse encodings [23]. Pruning at the filter-level is tackled in [24], but needs a lot of metadata that is highly correlated to the pruning percentage [23].

We perform FPGA-based CNN acceleration with a dataflow design that consumes little energy (due to reduced external memory access) and resources. It is a pattern-based pruning approach at the tile level, used in a pipelined setting. Our main contribution is a novel two-step process, which accelerates a CNN according to latency and accuracy constraints:

1. reach a target latency $\Delta_t$ through a new tiled-based pruning method and a mapping procedure that takes into account FPGA resource availability and the CNN architecture characteristics;

2. based on the tile shapes and pruning percentages found in Step 1, improve accuracy by varying the $N_\rho$ best patterns per layer. Increasing $N_\rho$ comes at a modest cost in terms of logic resources.
<figure><img src="image_3.png" /><figcaption>Fig. 1: Pruning compression ratio comparison (inspired by [23, Fig. 8]) with metadata included using various methods on a convolutional layer of size 128 × 128 × 3 × 3 and block shape 64 × 3 × 3. Elements are stored as 8-bit integers. NNZ is the ideal baseline, in the sense that only nonzero values are stored, without any metadata.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2: Memory storage of weights (50% density, 6 NNZ) with $N_\rho = 3$ different patterns $\{\rho_0, \rho_1, \rho_2\}$ and 4 tiles $\{T_i\}_{i=0}^3$, computed sequentially in 4 cycles with 6 multiplication units (DSPs).</figcaption></figure>

The paper is structured as follows. In Section II we overview our pattern-based FPGA accelerator, whereas in Section III we present our proposed two-step latency-guided pruning flow. Hardware and accuracy results are discussed in Section IV, with concluding remarks in Section V.

## II. ACCELERATOR ARCHITECTURE

Our FPGA dataflow accelerator is characterized by how it treats sparse data, in a low-overhead pattern-based manner.

### A. Sparse Data Encoding

There are various ways to represent sparse data in memory [23], [25, Sec. V.A]. Sparse accelerators can use for example bitmask, run-length encoding (RLE), NNZ coordinates (called COO) or compressed sparse column/row representations. The encoding has an impact on the metadata needed, making certain methods more desirable depending on the level of sparsity (see Fig. 1). We introduce a DBB-based approach that favors patterns at the tile level, i.e., in a layer each tile is limited to a number of patterns $N_\rho$, and each pattern has the same percentage of NNZ terms. Metadata in this case is not affected by the level of sparsity, which differs from most state-of-the-art methods. It is instead linked to both $N_\rho$ (i.e., $O(\lceil \log_2(N_\rho) \rceil)$ space for $N_\rho$ patterns) and the group/tile shape (i.e., the number of tiles per layer). Due to space limitations, we only briefly describe it in Section III-B and focus more on the pattern-based accelerator design.

We impose a constant number of patterns per layer. This saves weight memory storage by saving only the NNZ values and the pattern number corresponding to the tile (see Fig. 2).

<table><caption>TABLE I: Pruning methods and granularity comparison of top-1 accuracy on ImageNet-1K [26] with $N_\rho$ patterns per layer. Results correspond to finetuning a Pytorch [27] ResNet18 reference model (torchvision weights) for 14 epochs.</caption><thead><tr><th rowspan="2">Pruning Method</th><th colspan="4">Pruning (%)</th></tr><tr><th>Ref (0)</th><th>20</th><th>50</th><th>80</th></tr></thead><tbody><tr><td>Fine-grained</td><td></td><td>68.812</td><td>68.288</td><td>65.442</td></tr><tr><td>DBB (unbounded $N_\rho$)</td><td></td><td>68.224</td><td>67.934</td><td>64.682</td></tr><tr><td><strong>Pattern DBB (Ours)</strong></td><td><strong>256</strong></td><td><strong>67.956</strong></td><td><strong>67.49</strong></td><td><strong>63.098</strong></td></tr><tr><td>($N_\rho$)</td><td><strong>100</strong></td><td><strong>67.956</strong></td><td><strong>67.00</strong></td><td><strong>62.016</strong></td></tr><tr><td></td><td><strong>32</strong></td><td><strong>67.888</strong></td><td><strong>66.2</strong></td><td><strong>60.680</strong></td></tr><tr><td></td><td><strong>5</strong></td><td><strong>67.836</strong></td><td><strong>66</strong></td><td><strong>59.018</strong></td></tr><tr><td>Filter-based</td><td></td><td>65.508</td><td>57.172</td><td>30.412</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Fig. 3: Global architecture of the accelerator, with $N$ NNZ elements per tile of size $k_x \cdot k_y \cdot tc_i$, with $tc_i$ tile size on input channels.</figcaption></figure>

The pattern indexes can be hardcoded (see Fig. 3(b)) on the FPGA or stored in SRAM on the device. The number of patterns also impacts accuracy, with a larger $N_\rho$ bringing the results close to a vanilla DBB design (unbounded $N_\rho$) and fine-grained pruning, as can be seen in Table I.

### B. Pattern Pruning Accelerator Architecture

The overall architecture is shown in Fig. 3(a), where the only access to DDR is to read/write the network inputs and outputs. Within each layer, we buffer $k_y$ (kernel size) rows of each feature map (FM) as in [14] in order to minimize the on-chip memory footprint of FMs. As described in Section II-A, weights are stored in on-chip memory in compressed format.

We have $tc_o$ pattern processing elements (PPEs), where $tc_o$ is the tiling size for output channels, which is proportional to the number of output channels. Within each PPE (see Fig. 3(b)), $N$ NNZ values are computed in parallel. This is possible due to DBB pruning, which allows us to constrain the number of DSPs per layer in proportion to the number of NNZ values per tile. The pattern index $\rho$ stored with a weight tile will facilitate using the FMs with the corresponding weights (see the use of multiplexers in Fig. 3(b)).

The tile dimensions $tc = (tc_i, tc_o)$ will impose the parallelism for each layer. This directly influences the layer latencies $\{\Delta^{(\ell)}\}_{\ell=1}^L$ of a target $L$ layer network and impacts
<figure><img src="image_6.png" /><figcaption>Fig. 4: Condensed view of our latency-guided pattern pruning process. User inputs are marked with red.</figcaption></figure>

the size of intermediate FIFOs between layers. Large latency differences between layers induce larger FIFOs, increasing on-chip memory usage. In Section III-A2, we introduce an algorithm to reduce intermediate FIFO footprint.

### III. PATTERN PRUNING IN A DATAFLOW SETTING

We propose a two-step flow (see Fig. 4). Step ① gradually increases (in increments of $\delta_p$) the global pruning percentage $p_{global}$ and determines appropriate tile shapes $\{tc^{(\ell)}\}_{\ell=1}^L$ and per layer pruning percentages $\{p^{(\ell)}\}_{\ell=1}^L$ until the target latency $\Delta_t$ can be assured. Step ② calibrates the number of patterns $N_\rho$ by layer in an attempt to reach a target accuracy.

#### A. Step ①: Latency-guided pruning

1) Per-layer pruning level estimation: For the current $p_{global}$ pruning level, we estimate how much we should prune each layer of the network to meet this threshold, while implicitly trying to limit model accuracy degradation as much as possible. We use a fast weight magnitude-based [1] approach that tends to give good results in post-training scenarios; the $p_{global}$ weights with the lowest scores are pruned out. The resulting pruned network will give us the layer percentages $\{p^{(\ell)}\}_{\ell=1}^L$.

2) Exploring the mapping design space: Our goal is to maximize throughput under a fixed DSP budget. In a fully pipelined architecture, throughput will be limited by the layer with the largest latency, which depends on $tc$ and the number of multiply-accumulate (MAC) operations in the layer. CNNs are in general characterized by variability in the number of MACs per layer. To balance and optimize latency, it is therefore important to tune the block/tile size per layer (i.e., the mapping parameters). Overall, our approach depends on:

* Number of DSPs $M_{DSP}$ allocated for the network;

* Number of MACs $N_{MAC}^{(\ell)}$, for each layer $\ell$:

$$N_{MAC}^{(\ell)} = \begin{cases} H_o^{(\ell)} W_o^{(\ell)} C_o^{(\ell)} C_i^{(\ell)} k_x^{(\ell)} k_y^{(\ell)} & \ell \text{ is a convolution} \\ C_o^{(\ell)} C_i^{(\ell)} & \ell \text{ is fully connected} \end{cases} \quad (1)$$

and pruning percentage $p^{(\ell)}$ for each layer $\ell$.

The quantities in Eq. (1) correspond to the number of I/O channels in layer $\ell$ ($C_{i/o}$), the I/O feature map dimensions ($H_{i/o}$, $W_{i/o}$) and convolutional kernel size ($k_x^{(\ell)}$, $k_y^{(\ell)}$).

Summarized in Algorithm 1, its main steps are:

1) Construct profiles of all possible tiling configurations $B_{tc}^{(\ell)}$, together with their associated layer latency estimations $\Delta_{tc}^{(\ell)}$ and required number of DSPs $N_{DSP}^{\Delta_{tc}^{(\ell)}}$.

2) We use an exhaustive search on the Cartesian product of the sets of tile configurations for all layers. For optimized global latency, we search for configurations with minimal pairwise layer latency gaps (parameterized with $\tau > 0$).

### Algorithm 1 Global mapping

**Require:** weight dimensions ($C_o^{(\ell)}$, $C_i^{(\ell)}$, $k_x^{(\ell)}$, $k_y^{(\ell)}$) for each layer $\ell$, maximum latency deviation $\tau$ between two layers, layer pruning percentages $\{p^{(1)}, p^{(2)}, \dots, p^{(L)}\}$

**Ensure:** mapping parameters $C_{net}$ and latency $\Delta_{C_{net}}$

/* 1. Create network profiles: */

1: **for** $\ell = 1$ **to** $L$ **do**

2: $D_o^{(\ell)} \leftarrow \text{get\_divisors}(C_o^{(\ell)})$

3: $D_i^{(\ell)} \leftarrow \text{get\_divisors}(C_i^{(\ell)})$

4: $C^{(\ell)} \leftarrow D_o^{(\ell)} \times D_i^{(\ell)}$

5: **for** $tc = (tc_o, tc_i)$ **in** $C^{(\ell)}$ **do**

6: $B_{tc}^{(\ell)} \leftarrow \begin{cases} \{tc_o, tc_i, k_x^{(\ell)}, k_y^{(\ell)}\} & \ell \text{ is a convolution layer} \\ \{tc_o, tc_i\} & \ell \text{ is fully connected layer} \end{cases}$

7: $\Delta_{tc}^{(\ell)} \leftarrow N_{MAC}^{(\ell)} / (\prod_{k \in B_{tc}^{(\ell)}} k)$

8: $N_{DSP}^{\Delta_{tc}^{(\ell)}} \leftarrow [(1 - p^{(\ell)}) \prod_{k \in B_{tc}^{(\ell)}} k]$

9: **end for**

10: **end for**

/* 2. Find optimized mappings for pipeline scheduling: */

11: $\mathcal{C} \leftarrow C^{(1)} \times C^{(2)} \times \dots \times C^{(L)}$

/* $\mathcal{C} = \{tc^{(1)}, \dots, tc^{(L)}\}$ */

12: $\mathcal{C}_{min} \leftarrow \{C \in \mathcal{C} | \forall (i, j) \in [1; L]^2 \ i \neq j, |\Delta_{tc_i}^{(i)} - \Delta_{tc_j}^{(j)}| \le \tau\}$

/* 3. Check DSP consumption: */

13: **for** $C$ **in** $\mathcal{C}_{min}$ **do**

14: $N_{DSP}^{C} \leftarrow \sum_{\ell=1}^{L} N_{DSP}^{\Delta_{tc}^{(\ell)}}$

15: **end for**

16: $\mathcal{C}_{DSP} \leftarrow \arg \min_{C \in \mathcal{C}_{min}} \left\{ \max_{1 \le \ell \le L} (\Delta_{tc(\ell)}^{(\ell)}) \mid N_{DSP}^{C} \le M_{DSP} \right\}$

17: $\Delta_{C_{net}} \leftarrow \min_{C \in \mathcal{C}_{min}} \left\{ \max_{1 \le \ell \le L} (\Delta_{tc(\ell)}^{(\ell)}) \mid N_{DSP}^{C} \le M_{DSP} \right\}$

/* 4. SRAM optimization: */

18: $C_{net} \leftarrow \arg \min_{C \in \mathcal{C}_{DSP}} \left\{ \sum_{\ell=1}^{L} N_{SRAM_{in}}^{(\ell),C} + N_{SRAM_{post-acc}}^{(\ell),C} \right\}$

3) DSP scarcity is the limiting factor in increasing block size, which is why it is crucial to optimize their allocation according to the pruning percentages $\{p^{(\ell)}\}_{\ell=1}^L$ and ensure that the total number of allocated DSPs $N_{DSP}^{C}$ does not exceed the limit $M_{DSP}$.

4) Different mappings $C$ can have the same latency $\Delta_{tc}^{(\ell)}$ and DSP consumption $N_{DSP}^{\Delta_{tc}^{(\ell)}}$, but a different SRAM cost. It corresponds to the number of SRAM blocks needed for input feature maps $N_{SRAM_{in}}^{(\ell)}$ and the number of blocks needed for post-accumulation $N_{SRAM_{post-acc}}^{(\ell)}$:

$$N_{SRAM_{in}}^{(\ell)} = k_y^{(\ell)} \cdot \left[ \frac{tc_i^{(\ell)} \cdot N_{In}^{\text{Bits}}}{N_{SRAM}^{\text{Bits}}} \right] \cdot \left[ \frac{W_i^{(\ell)} \cdot C_i^{(\ell)}}{tc_i^{(\ell)} \cdot N_{SRAM}^{\text{line}}} \right],$$

$$N_{SRAM_{post-acc}}^{(\ell)} = \left[ \frac{tc_o^{(\ell)} \cdot N_{Post-acc}^{\text{Bits}}}{N_{SRAM}^{\text{Bits}}} \right] \cdot \left[ \frac{W_o^{(\ell)}}{N_{SRAM}^{\text{line}}} \right],$$

where $N_{In}^{\text{Bits}}$ is the word length of the input activations (e.g. 8 bits), $N_{Post-acc}^{\text{Bits}}$ is the word length of the post-accumulation results, and $N_{SRAM}^{\text{Bits}}$, $N_{SRAM}^{\text{line}}$ are the word length of an SRAM and its number of lines (e.g. 4096), respectively. We choose the mapping that consumes the least SRAM globally.
<table><caption>TABLE II: Resource (LUT/FF) percentage usage results for DBB with COO encoding, and our pattern-based DBB with varying $N_ρ$ on a 2D convolution with kernel size $k = 3$. Various pruning percentages $p^{(t)}$ are used, with $tc_o = 3$ and $tc_i = 6$. Clock frequency is 200MHz.</caption><thead><tr><th rowspan="2">N<sub>ρ</sub></th><th colspan="5">Pruning (%)</th></tr><tr><th>10</th><th>30</th><th>50</th><th>70</th><th>90</th></tr></thead><tbody><tr><td>1</td><td>1.1/1.4</td><td>1/1.4</td><td>0.9/1.3</td><td>0.8/1.1</td><td>0.6/0.7</td></tr><tr><td>4</td><td>2.5/1.9</td><td>2.4/1.8</td><td>2.1/1.9</td><td>1.6/1.7</td><td>1.1/1.4</td></tr><tr><td>8</td><td>3.3/1.9</td><td>3.2/1.9</td><td>2.8/1.7</td><td>2.2/1.7</td><td>1.5/1.6</td></tr><tr><td>16</td><td>3.5/2.1</td><td>4.0/2.0</td><td>3.8/2.0</td><td>3.1/2.0</td><td>1.8/1.9</td></tr><tr><td>32</td><td>3.7/2.1</td><td>4.9/2.0</td><td>4.6/2.0</td><td>3.6/2.0</td><td>2.4/1.9</td></tr><tr><td>100</td><td>5.7/2.1</td><td>7.5/2.0</td><td>6.4/2.0</td><td>5.5/2.0</td><td>3.5/1.9</td></tr><tr><td>256</td><td>7.5/2.1</td><td>9.5/2.0</td><td>8.7/2.0</td><td>7.8/2.0</td><td>4.5/2.0</td></tr><tr><td>COO</td><td>9.0/2.4</td><td>11.8/2.5</td><td>11.4/2.6</td><td>8.8/3.0</td><td>5.0/2.2</td></tr></tbody></table>

As long as the model latency is far from the user's latency requirements ($\Delta_{net} - \Delta_t > \epsilon_\Delta$, with $\epsilon_\Delta < 0$ a small magnitude threshold) we increment the global pruning percentage $p_{global} = p_{global} + \delta_p$ and we reiterate Step ①.

## B. Step ②: Accuracy-aware pattern pruning refinement

This step reduces the accuracy loss induced in the pruned model by increasing the number of patterns $N_ρ$ per layer. We use a *k-means* clustering method [28] to find the best set of patterns for acceleration. Once the patterns have been determined, we fine-tune the network for a few epochs. Finally, we perform quantization-aware training [29] (QAT) to quantify weights and activations to 8 bits over several more epochs.

# IV. EXPERIMENTS AND DISCUSSION

We perform a suite of experiments on a Xilinx Zynq UltraScale+ MPSoC ZCU104 using *High-Level Synthesis* (HLS) with Vitis HLS 2022.2. At the layer level, Table II considers a 128 × 128 × 3 × 3 2D convolution with a ($tc_i = 6$, $tc_o = 3$) tile size and various pruning percentages and number of patterns $N_ρ$. Logic consumption (LUTs and FFs) can be finely controlled by varying $N_ρ$. The difference between our method and vanilla DBB (shown in Table II using a COO encoding defined by {$k_x, k_y, tc_i$} coordinates) is due to the impact of the multiplexers used for routing inputs in the layer.

Image enhancement tasks are often deployed on embedded systems. We therefore test our method with VDSR-based [30] CNNs (54 features and varying number of layers $L$) on a super-resolution task with a ×2 scale on the Set5 [31] dataset. Peak Signal to Noise Ratio (PSNR) (in dB) is used as a comparison criterion between the reference image and the network output. We compare against a dense accelerator generated using the FINN [12] framework that deploys pipelined models on FPGAs. Its input is an ONNX-formatted model fine-tuned with QAT for 80 epochs and 8-bit fixed-point values (weights and activations) using Brevitas [32]. We also adapted our accelerator to use DBB pruning with COO encoding as another comparison point. COO encoding tends to have very good behavior (latency and energy consumption) at high pruning percentages [25]. Pruning in this case is also done at the *tc* tile-level, consistent with our methodology, without imposing any pattern structure. While our design can handle arbitrarily

<table><caption>TABLE III: Comparison between FINN, DBB+COO pruning, and our method using VDSR CNNs. Image shapes are 3 × S × S. Values in bold are below the FPS/PSNR targets or above the FPGA resources limit. DSP, LUT, BRAM, URAM usage is given in (%), $p = p_{global}$ in (%), PSNR in (dB), and V quantify the validity of the result.</caption><thead><tr><th>Method</th><th>L</th><th>S</th><th>FPS/PSNR</th><th>LUT/DSP</th><th>B/URAM</th><th>p</th><th>V</th></tr></thead><tbody><tr><td>FINN</td><td></td><td></td><td>62.00/36.32</td><td>27.3/58.3</td><td>35.9/0</td><td>0</td><td>✓</td></tr><tr><td>COO</td><td>200</td><td></td><td>30.10/36.83</td><td>9.14/29</td><td>17.3/21.9</td><td>1</td><td>✓</td></tr><tr><td>N<sub>ρ</sub> = 2</td><td></td><td></td><td>31.49/36.83</td><td>5.5/29</td><td>9.5/21.9</td><td>1</td><td>✓</td></tr><tr><td>FINN</td><td></td><td></td><td><strong>15.43/36.32</strong></td><td>36.9/58.9</td><td>36.2/0</td><td>0</td><td>✗</td></tr><tr><td>COO</td><td>5</td><td>400</td><td>43.25/37.09</td><td><strong>121/79.6</strong></td><td>35.6/18.8</td><td>30</td><td>✗</td></tr><tr><td>N<sub>ρ</sub> = 9</td><td></td><td></td><td>46.30/37.06</td><td>36.3/94.2</td><td>11.2/18.8</td><td>30</td><td>✓</td></tr><tr><td>FINN</td><td></td><td></td><td><strong>3.85/36.32</strong></td><td>56.2/58.9</td><td>36.2/0</td><td>0</td><td>✗</td></tr><tr><td>COO</td><td>800</td><td></td><td>30.63/36.97</td><td><strong>191/51.2</strong></td><td>38.9/25.0</td><td>90</td><td>✗</td></tr><tr><td>N<sub>ρ</sub> = 9</td><td></td><td></td><td>34.72/36.97</td><td>36.5/52.7</td><td>20/28.1</td><td>90</td><td>✓</td></tr><tr><td>FINN</td><td></td><td></td><td>31.44/36.79</td><td>45.3/76</td><td>46.8/0</td><td>0</td><td>✓</td></tr><tr><td>COO</td><td>200</td><td></td><td>30.10/36.83</td><td>23.9/76</td><td>42.2/37.5</td><td>1</td><td>✓</td></tr><tr><td>N<sub>ρ</sub> = 2</td><td></td><td></td><td>31.49/36.83</td><td>12.7/76</td><td>21.5/37.5</td><td>1</td><td>✓</td></tr><tr><td>FINN</td><td></td><td></td><td><strong>7.70/36.79</strong></td><td>66.8/76</td><td>46.8/0</td><td>0</td><td>✗</td></tr><tr><td>COO</td><td>10</td><td>400</td><td>32.40/36.87</td><td><strong>498/89.2</strong></td><td>71.0/59.4</td><td>80</td><td>✗</td></tr><tr><td>N<sub>ρ</sub> = 9</td><td></td><td></td><td>38.20/36.88</td><td>48/94.8</td><td>35.1/59.4</td><td>80</td><td>✓</td></tr><tr><td>FINN</td><td></td><td></td><td><strong>1.93/36.79</strong></td><td><strong>109/76</strong></td><td>46.8/0</td><td>0</td><td>✗</td></tr><tr><td>COO</td><td>800</td><td></td><td>30.40/36.85</td><td><strong>316/90.1</strong></td><td>83.2/56.3</td><td>93</td><td>✗</td></tr><tr><td>N<sub>ρ</sub> = 9</td><td></td><td></td><td>34.48/36.85</td><td>26.4/99.1</td><td>61.5/62.5</td><td>93</td><td>✓</td></tr><tr><td>FINN</td><td></td><td></td><td>31.43/36.73</td><td><strong>71.6/123</strong></td><td>75.6/0</td><td>0</td><td>✗</td></tr><tr><td>COO</td><td>200</td><td></td><td>29.90/36.83</td><td><strong>101/97.9</strong></td><td>60.9/53.1</td><td>20</td><td>✗</td></tr><tr><td>N<sub>ρ</sub> = 9</td><td></td><td></td><td>30.30/36.82</td><td>26/98.8</td><td>33.3/53.1</td><td>20</td><td>✓</td></tr><tr><td>FINN</td><td></td><td></td><td><strong>7.70/36.73</strong></td><td><strong>105/123</strong></td><td>75.6/0</td><td>0</td><td>✗</td></tr><tr><td>COO</td><td>15</td><td>400</td><td>35.44/36.75</td><td><strong>540/70.1</strong></td><td>88.3/90.6</td><td>90</td><td>✗</td></tr><tr><td>N<sub>ρ</sub> = 9</td><td></td><td></td><td>44.67/36.75</td><td>62.9/79.3</td><td>58.0/90.6</td><td>90</td><td>✓</td></tr><tr><td>FINN</td><td></td><td></td><td><strong>3.85/36.73</strong></td><td><strong>204.6/246</strong></td><td><strong>151.3/0</strong></td><td>0</td><td>✗</td></tr><tr><td>COO</td><td>800</td><td></td><td>30.29/36.74</td><td><strong>434/87.5</strong></td><td>113/87.5</td><td>96</td><td>✗</td></tr><tr><td>N<sub>ρ</sub> = 9</td><td></td><td></td><td>33.55/36.74</td><td>63/91</td><td>85.4/93.8</td><td>96</td><td>✓</td></tr></tbody></table>

shaped inputs, square images are used due to a limitation in FINN. We vary $L$, showcasing throughput in frames per second (FPS), hardware resource consumption, and PSNR.

The results are compiled in Table III, with target thresholds of 30 FPS and 200MHz and a PSNR accuracy greater or equal than 36dB (as a baseline, bicubic interpolation upscaling has a PSNR of 33.66dB [30]). Whereas our method is always within limits, the FINN-generated dense accelerator cannot guarantee FPS or resource use for most designs except when $S = 200$ and $L \in \{5, 10\}$. The sparse COO-based accelerator can achieve good accuracy and FPS, but has an increased logic consumption (LUTs) and uses more internal memory compared to our method. This is due to the complicated routing needed for the COO encoding. The BRAM overhead is due to metadata storage requirements for the COO encoding (seen for instance in Fig. 1).

# V. CONCLUSION AND FUTURE WORK

In this paper, we present a fast two-step method for implementing CNN models on FPGA using an unrolled pipelined design that limits read/write access to external memory. To this end, we propose a new dataflow sparse accelerator that uses DBB-based pattern pruning at the tile level, making efficient use of available DSPs within a fixed logic resource budget. We also show that our approach does not degrade network accuracy considerably compared to other pruning methods. In the future, we plan to explore how changing the number of patterns per layer can lead to even less logic consumption.
REFERENCES

[1] T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste, "Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Deep Neural Networks," *The Journal of Machine Learning Research*, vol. 22, no. 1, pp. 10882-11005, 2021.

[2] J. Li and A. Louri, "AdaPrune: An Accelerator-Aware Pruning Technique for Sustainable CNN Accelerators," *IEEE Transactions on Sustainable Computing*, vol. 7, no. 1, pp. 47-60, 2021.

[3] M. Zhu, T. Zhang, Z. Gu, and Y. Xie, "Sparse Tensor Core: Algorithm and Hardware Co-Design for Vector-wise Sparse Neural Networks on Modern GPUs," in *52nd Annual IEEE/ACM Int. Symp. on Microarchitecture (MICRO)*, 2019, pp. 359-371.

[4] A. Zhou, Y. Ma, J. Zhu, J. Liu, Z. Zhang, K. Yuan, W. Sun, and H. Li, "Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch," *arXiv preprint arXiv:2102.04010*, 2021.

[5] S. Liu, B. Ren, X. Shen, and Y. Wang, "CoCoPIE: Making Mobile AI Sweet As PIE-Compression-Compilation Co-Design Goes a Long Way," *ArXiv*, vol. abs/2003.06700, 2020.

[6] Y. Cai, H. Li, G. Yuan, W. Niu, Y. Li, X. Tang, B. Ren, and Y. Wang, "YOLObile: Real-Time Object Detection on Mobile Devices via Compression-Compilation Co-Design," in *AAAI Conference on Artificial Intelligence*, 2020.

[7] Y.-H. Chen, T. Krishna, J. Emer, and V. Sze, "Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks," in *IEEE International Solid-State Circuits Conference (ISSCC)*, 2016, pp. 262-263.

[8] N. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers et al., "In-Datacenter Performance Analysis of a Tensor Processing Unit," in *44th Annual Int. Symp. on Computer Architecture (ISCA)*, 2017, pp. 1-12.

[9] A. Xilinx. (2020) Vitis AI. [Online]. Available: https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html

[10] J. Choquette, W. Gandhi, O. Giroux, N. Stam, and R. Krashinsky, "NVIDIA A100 Tensor Core GPU: Performance and Innovation," *IEEE Micro*, vol. 41, no. 2, pp. 29-35, 2021.

[11] V. Sze, Y. hsin Chen, T.-J. Yang, and J. S. Emer, "Efficient Processing of Deep Neural Networks," *Synthesis Lect. on Computer Arch.*, 2020.

[12] Y. Umuroglu, N. J. Fraser, G. Gambardella, M. Blott, P. Leong, M. Jahre, and K. Vissers, "FINN: A Framework for Fast, Scalable Binarized Neural Network Inference," in *ACM/SIGDA Int. Symp. on FPGAs*, 2017, pp. 65-74.

[13] J. Cai, Y. Wei, Z. Wu, S. Peng, and K. Ma, "Inter-layer Scheduling Space Definition and Exploration for Tiled Accelerators," in *50th Annual Int. Symp. on Computer Architecture (ISCA)*, 2023, pp. 1-17.

[14] K. Goetschalckx and M. Verhelst, "Breaking High-Resolution CNN Bandwidth Barriers With Enhanced Depth-First Execution," *IEEE J. on Emer. and Select. Topics in Circ. and Syst.*, vol. 9, no. 2, pp. 323-331, 2019.

[15] L. Mei, K. Goetschalckx, A. Symons, and M. Verhelst, "DeFiNES: Enabling Fast Exploration of the Depth-first Scheduling Space for DNN Accelerators through Analytical Modeling," in *IEEE Int. Symp. on High-Performance Computer Architecture (HPCA)*, 2023, pp. 570-583.

[16] J. Wang, S. Yu, Z. Yuan, J. Yue, Z. Yuan, R. Liu, Y. Wang, H. Yang, X. Li, and Y. Liu, "PACA: A Pattern Pruning Algorithm and Channel-Fused High PE Utilization Accelerator for CNNs," *IEEE Trans. on Comp.-Aided Design of Int. Circ. and Syst.*, vol. 41, no. 11, pp. 5043-5056, 2022.

[17] Y. Wang, C. Zhang, Z. Xie, C. Guo, Y. Liu, and J. Leng, "Dual-side Sparse Tensor Core," in *ACM/IEEE 48th Annual Int. Symp. on Computer Architecture (ISCA)*, 2021, pp. 1083-1095.

[18] Y.-H. Chen, T.-J. Yang, J. Emer, and V. Sze, "Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices," *IEEE J. on Emer. and Select. Topics in Circ. and Syst.*, vol. 9, no. 2, pp. 292-308, 2019.

[19] R. Hojabr, A. Sedaghati, A. Sharifian, A. Khonsari, and A. Shriraman, "SPAGHETTI: Streaming Accelerators for Highly Sparse GEMM on FPGAs," in *IEEE Int. Symp. on High-Performance Computer Architecture (HPCA)*, 2021, pp. 84-96.

[20] Z. Tan, J. Song, X. Ma et al., "PCNN: Pattern-based Fine-Grained Regular Pruning towards Optimizing CNN Accelerators," in *57th ACM/IEEE Design Automation Conference (DAC)*, 2020, pp. 1-6.

[21] Z.-G. Liu, P. N. Whatmough, Y. Zhu, and M. Mattina, "S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Acceleration," in *IEEE Int. Symp. on High-Perf. Comp. Arch. (HPCA)*, 2022, pp. 573-586.

[22] Y. Yang, J. S. Emer, and D. Sanchez, "ISOSeles: Accelerating Sparse CNNs through Inter-Layer Pipelining," in *IEEE Int. Symp. on High-Performance Computer Architecture (HPCA)*, 2023, pp. 598-610.

[23] S. Dave, R. Baghdadi, T. Nowatzki, S. Avancha, A. Shrivastava, and B. Li, "Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights," *Proceedings of the IEEE*, vol. 109, no. 10, pp. 1706-1752, 2021.

[24] M. Shimoda, Y. Sada, and H. Nakahara, "FPGA-Based Inter-layer Pipelined Accelerators for Filter-Wise Weight-Balanced Sparse Fully Convolutional Networks with Overlapped Tiling," *Journal of Signal Processing Systems*, vol. 93, pp. 499-512, 2021.

[25] Y. N. Wu, P.-A. Tsai, A. Parashar, V. Sze, and J. S. Emer, "Sparseloop: An Analytical Approach To Sparse Tensor Accelerator Modeling," in *ACM/IEEE Int. Symp. on Microarchitecture (MICRO)*, 2022.

[26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet: Large Scale Visual Recognition Challenge," *Int. Journal of Computer Vision (IJCV)*, vol. 115, no. 3, pp. 211-252, 2015.

[27] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, "Automatic Differentiation in PyTorch," 2017.

[28] S. Lloyd, "Least Squares Quantization in PCM," *IEEE Transactions on Information Theory*, vol. 28, no. 2, pp. 129-137, 1982.

[29] M. Courbariaux, Y. Bengio, and J.-P. David, "BinaryConnect: Training DNNs with binary weights during propagations," in *NIPS*, 2015.

[30] J. Kim, J. K. Lee, and K. M. Lee, "Accurate Image Super-Resolution Using Very Deep Convolutional Networks," *IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)*, pp. 1646-1654, 2015.

[31] M. Bevilacqua, A. Roumy, C. Guillemot, and M.-L. Alberi Morel, "Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding," in *British Machine Vision Conf. (BMVC)*, 2012.

[32] A. Pappalardo, "Xilinx/brevitas," 2023. [Online]. Available: https://doi.org/10.5281/zenodo.3333552