# Efficient CNN Architecture Design Techniques for Real-time Embedded Systems Deployment

Senthil Yogamani

<figure><img src="image_1.png" /></figure>

## To cite this version:

Senthil Yogamani. Efficient CNN Architecture Design Techniques for Real-time Embedded Systems Deployment. 2024. hal-04584152

HAL Id: hal-04584152

https://hal.science/hal-04584152

Preprint submitted on 23 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Efficient CNN Architecture Design Techniques for Real-time Embedded Systems Deployment

Senthil Yogamani

<figure><img src="image_1.png" /></figure>

To cite this version:

Senthil Yogamani. Efficient CNN Architecture Design Techniques for Real-time Embedded Systems Deployment. 2024. hal-04584152

HAL Id: hal-04584152

https://hal.science/hal-04584152v1

Preprint submitted on 23 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# 1

## Efficient CNN Architecture Design
Techniques for Real-time Embedded
Systems Deployment

Senthil Yogamani

## CONTENTS

<table><tr><td>1.1</td><td>Introduction</td><td>2</td></tr><tr><td>1.2</td><td>Hardware Accelerators for CNN</td><td>4</td></tr><tr><td>1.2.1</td><td>Custom CNN HW IPs</td><td>4</td></tr><tr><td>1.2.2</td><td>FPGAs</td><td>4</td></tr><tr><td>1.2.3</td><td>GPUs</td><td>5</td></tr><tr><td>1.2.4</td><td>DSPs</td><td>5</td></tr><tr><td>1.2.5</td><td>CPUs</td><td>6</td></tr><tr><td>1.3</td><td>Building Blocks of CNN</td><td>6</td></tr><tr><td>1.3.1</td><td>Convolutional building block</td><td>6</td></tr><tr><td>1.3.2</td><td>Baseline FCN architecture</td><td>7</td></tr><tr><td>1.3.3</td><td>Formalized extensions</td><td>7</td></tr><tr><td>1.3.4</td><td>Performance consideration/metrics</td><td>9</td></tr><tr><td>1.4</td><td>Taxonomy of Optimization Methods</td><td>10</td></tr><tr><td>1.4.1</td><td>Quantization</td><td>11</td></tr><tr><td>1.4.2</td><td>Efficient filter design</td><td>14</td></tr><tr><td>1.4.3</td><td>Model Compression</td><td>19</td></tr><tr><td>1.4.4</td><td>Data/domain dependent optimizations</td><td>21</td></tr><tr><td>1.5</td><td>Towards a recipe for efficient design</td><td>23</td></tr><tr><td>1.5.1</td><td>Meta Learning</td><td>24</td></tr><tr><td>1.5.2</td><td>Design Exploration & Guidelines</td><td>25</td></tr><tr><td>1.6</td><td>Conclusion</td><td>26</td></tr></table>

Majority of CNN architecture design is aimed at topping leader-boards of public benchmarks by increasing the complexity to achieve diminishing returns. They are typically over-specified by a large margin and can be optimized by a factor of 10x-100x with small reduction in accuracy. These networks are typically not suitable for embedded deployment in spite of the rapid increase in
# 1

## Efficient CNN Architecture Design
Techniques for Real-time Embedded
Systems Deployment

Senthil Yogamani

## CONTENTS

<table><tr><td>1.1</td><td>Introduction</td><td>2</td></tr><tr><td>1.2</td><td>Hardware Accelerators for CNN</td><td>4</td></tr><tr><td>1.2.1</td><td>Custom CNN HW IPs</td><td>4</td></tr><tr><td>1.2.2</td><td>FPGAs</td><td>4</td></tr><tr><td>1.2.3</td><td>GPUs</td><td>5</td></tr><tr><td>1.2.4</td><td>DSPs</td><td>5</td></tr><tr><td>1.2.5</td><td>CPUs</td><td>6</td></tr><tr><td>1.3</td><td>Building Blocks of CNN</td><td>6</td></tr><tr><td>1.3.1</td><td>Convolutional building block</td><td>6</td></tr><tr><td>1.3.2</td><td>Baseline FCN architecture</td><td>7</td></tr><tr><td>1.3.3</td><td>Formalized extensions</td><td>7</td></tr><tr><td>1.3.4</td><td>Performance consideration/metrics</td><td>9</td></tr><tr><td>1.4</td><td>Taxonomy of Optimization Methods</td><td>10</td></tr><tr><td>1.4.1</td><td>Quantization</td><td>11</td></tr><tr><td>1.4.2</td><td>Efficient filter design</td><td>14</td></tr><tr><td>1.4.3</td><td>Model Compression</td><td>19</td></tr><tr><td>1.4.4</td><td>Data/domain dependent optimizations</td><td>21</td></tr><tr><td>1.5</td><td>Towards a recipe for efficient design</td><td>23</td></tr><tr><td>1.5.1</td><td>Meta Learning</td><td>24</td></tr><tr><td>1.5.2</td><td>Design Exploration & Guidelines</td><td>25</td></tr><tr><td>1.6</td><td>Conclusion</td><td>26</td></tr></table>

Majority of CNN architecture design is aimed at topping leader-boards of public benchmarks by increasing the complexity to achieve diminishing returns. They are typically over-specified by a large margin and can be optimized by a factor of 10x-100x with small reduction in accuracy. These networks are typically not suitable for embedded deployment in spite of the rapid increase in
computational power available. There is still a large need to optimize or reduce
the size of the network by orders of magnitude for computer vision applica-
tions. There is an increasing larger community which is focused on designing
efficient networks. However, there is lag between efficient design community
and architecture design due to rapid progress. In several cases, the proposed
optimized networks in the community do not consider utilizing the parallelism
in the hardware effectively because of lack of embedded experience. The mo-
tivation of this paper to systematically summarize efficient design techniques
and provide guidelines for a application designer. We perform a case study by
benchmarking semantic segmentation for autonomous driving.

1.1 Introduction

There has been rapid progress in various perception tasks due to deep learn-
ing which can be deployed in real world autonomous driving systems such as
object detection [52, 72, 47], depth estimation [32, 33], localization [63], mo-
tion estimation [46], semantic segmentation [6, 53, 51], blockage and adverse
weather detection [65, 66, 64], trajectory prediction [24], multi-task learn-
ing [59, 37, 7] and near-field perception systems [12, 50, 18, 34]. The trend in
Deep Learning is to use large models to beat competition scores. In industrial
applications, there is a computational bound because of limited resources on
embedded. There is some work done attempting to quantify the accuracy gain
normalized to computations used [5]. According to this paper, GoogleNet is a
winner by a large margin and VGG is highly inefficient. There are many op-
timization methods which report large runtime improvements with minimal
loss of accuracy going to lower resolution or sparse convolutions. Most of these
studies are done on Alexnet (or similar) which is a highly redundant network.
There is no evaluation of a combination of all these features.

There is also a gradation of computational resources across various projects
and various sizes of CNN would be needed! The above graph is a demonstration
of the accuracy/complexity trade-off of various CNN types. The graph below
is from the paper [22] which does a similar comparison including the meta-
architectures like Faster R-CNN, R-FCN and SSD.

Motivation:

1/ exploration of network topologies has provided community with a huge quantity of research papers most of them build advances one on top of another. It is sometimes hard for newcomers to see the whole picture and the key components of a given architecture. Here we intend to explain the full history which had led to current state of the art.

2/ Early architectures were not applicable to real time embedded design and recent trends to reduce parameters for memory footprint and simplify architecture to minimize computation cost are highlighted in this survey
computational power available. There is still a large need to optimize or reduce
the size of the network by orders of magnitude for computer vision applica-
tions. There is an increasing larger community which is focused on designing
efficient networks. However, there is lag between efficient design community
and architecture design due to rapid progress. In several cases, the proposed
optimized networks in the community do not consider utilizing the parallelism
in the hardware effectively because of lack of embedded experience. The mo-
tivation of this paper to systematically summarize efficient design techniques
and provide guidelines for a application designer. We perform a case study by
benchmarking semantic segmentation for autonomous driving.

1.1 Introduction

There has been rapid progress in various perception tasks due to deep learn-
ing which can be deployed in real world autonomous driving systems such as
object detection [52, 72, 47], depth estimation [32, 33], localization [63], mo-
tion estimation [46], semantic segmentation [6, 53, 51], blockage and adverse
weather detection [65, 66, 64], trajectory prediction [24], multi-task learn-
ing [59, 37, 7] and near-field perception systems [12, 50, 18, 34]. The trend in
Deep Learning is to use large models to beat competition scores. In industrial
applications, there is a computational bound because of limited resources on
embedded. There is some work done attempting to quantify the accuracy gain
normalized to computations used [5]. According to this paper, GoogleNet is a
winner by a large margin and VGG is highly inefficient. There are many op-
timization methods which report large runtime improvements with minimal
loss of accuracy going to lower resolution or sparse convolutions. Most of these
studies are done on Alexnet (or similar) which is a highly redundant network.
There is no evaluation of a combination of all these features.

There is also a gradation of computational resources across various projects
and various sizes of CNN would be needed! The above graph is a demonstration
of the accuracy/complexity trade-off of various CNN types. The graph below
is from the paper [22] which does a similar comparison including the meta-
architectures like Faster R-CNN, R-FCN and SSD.

Motivation:

1/ exploration of network topologies has provided community with a huge quantity of research papers most of them build advances one on top of another. It is sometimes hard for newcomers to see the whole picture and the key components of a given architecture. Here we intend to explain the full history which had led to current state of the art.

2/ Early architectures were not applicable to real time embedded design and recent trends to reduce parameters for memory footprint and simplify architecture to minimize computation cost are highlighted in this survey
<figure><img src="image_3.png" /><figcaption>**FIGURE 1.1**
Accuracy of various networks normalized to OPS (reproduced figure).</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 4: Accuracy stratified by object size, meta-architecture and feature extractor. We fix the image resolution to 300.</figcaption></figure>

<caption>**FIGURE 1.2**
Accuracy of various networks normalized to OPS (reproduced figure).</caption>

3/ help people to pick the right direction for their DNN work given their constraints and help to understand how their architecture are efficient

4/ not a benchmark comparing all existing architectures but presenting the
<figure><img src="image_3.png" /><figcaption>**FIGURE 1.1**
Accuracy of various networks normalized to OPS (reproduced figure).</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 4: Accuracy stratified by object size, meta-architecture and feature extractor. We fix the image resolution to 300.</figcaption></figure>

<caption>**FIGURE 1.2**
Accuracy of various networks normalized to OPS (reproduced figure).</caption>

3/ help people to pick the right direction for their DNN work given their constraints and help to understand how their architecture are efficient

4/ not a benchmark comparing all existing architectures but presenting the
4 *Efficient CNN Architecture Design Techniques for Real-time Embedded Systems Deployment*

key ideas and the specificities of each architecture component to show in which
way they are efficient

**Related work:** The motivation of this paper to summarize efficient design techniques and provide guidelines for a application designer. [60] is an excel-lent reference which is the closest to our attempt, the perspective is different though, their perspective is from a hardware design view.

**Paper structure:** The rest of the paper is structured as follows. Section 1.2 provides a distilled summary of necessary hardware details. Section 1.3 provides a short overview of convolutional neural networks and run-time related aspects. Section 1.4 is the main section of the paper where various optimization ideas are discussed in an organized manner. Section 1.5 provides guidelines of putting the ideas together for designing an efficient network, there are trade-offs specific to hardware. Finally, section 1.6 summarizes the paper and provides potential future directions.

## 1.2 Hardware Accelerators for CNN

Majority of the CNN inference run-time and optimizations are reported on Nvidia platforms. The optimization techniques can be highly depend on the type of processor. Thus we provide an overview of various architectures from the perspective of CNN deployment.

1.2.1 Custom CNN HW IPs

There are two primary CNN accelerator types namely Matrix Multiply and Explicit Convolution.

**ASIC** (Application-specific Integrated Circuit) implements the entire algorithm in hardware with minimal flexibility like modification of parameters coded via register settings. If the algorithm is standardized, ASIC provides the best performance for the given silicon. But it requires a lot of initial investment which impacts time to market and there is a risk of being redundant particularly for the area of computer Vision where the algorithms are not standardized and could change drastically. Mobileye is one company who primarily uses ASIC. But moving from EyeQ3 to EyeQ4, they have shifted to flexible processors. Toshibas TMP device has HW acceleration for object detection feature (HOG) and 3D reconstruction (SFM) and Renesas provides an IMR HW module to perform distortion correction.
4 *Efficient CNN Architecture Design Techniques for Real-time Embedded Systems Deployment*

key ideas and the specificities of each architecture component to show in which
way they are efficient

**Related work:** The motivation of this paper to summarize efficient design techniques and provide guidelines for a application designer. [60] is an excel-lent reference which is the closest to our attempt, the perspective is different though, their perspective is from a hardware design view.

**Paper structure:** The rest of the paper is structured as follows. Section 1.2 provides a distilled summary of necessary hardware details. Section 1.3 provides a short overview of convolutional neural networks and run-time related aspects. Section 1.4 is the main section of the paper where various optimization ideas are discussed in an organized manner. Section 1.5 provides guidelines of putting the ideas together for designing an efficient network, there are trade-offs specific to hardware. Finally, section 1.6 summarizes the paper and provides potential future directions.

## 1.2 Hardware Accelerators for CNN

Majority of the CNN inference run-time and optimizations are reported on Nvidia platforms. The optimization techniques can be highly depend on the type of processor. Thus we provide an overview of various architectures from the perspective of CNN deployment.

1.2.1 Custom CNN HW IPs

There are two primary CNN accelerator types namely Matrix Multiply and Explicit Convolution.

**ASIC** (Application-specific Integrated Circuit) implements the entire algorithm in hardware with minimal flexibility like modification of parameters coded via register settings. If the algorithm is standardized, ASIC provides the best performance for the given silicon. But it requires a lot of initial investment which impacts time to market and there is a risk of being redundant particularly for the area of computer Vision where the algorithms are not standardized and could change drastically. Mobileye is one company who primarily uses ASIC. But moving from EyeQ3 to EyeQ4, they have shifted to flexible processors. Toshibas TMP device has HW acceleration for object detection feature (HOG) and 3D reconstruction (SFM) and Renesas provides an IMR HW module to perform distortion correction.
1.2.2 FPGAs

FPGA (Field Programmable Gate Array) is closely related to ASICs but has a key advantage of being re-configurable at run-time. From the application perspective, this means the HW can adaptively transform to an accelerator needed for that particular scenario. For instance, if the algorithm has to be drastically different for low and high speeds, the FPGA can transform into one of these as required. But because ASIC is hard-coded to be optimized for one algorithm, FPGA will be lagging in performance. Due to recent progress in FPGA technology, the gap is closing. Xilinx and Altera are the major FPGA suppliers who offer very similar performance FPGAs. Suitable for rapid changing CNN architectures instead of fixed function ASICs.

1.2.3 GPUs

GPU (Graphics Processing Unit) was traditionally designed for graphics ac-
celeration. It can be used for view rendering/remapping and for drawing out-
put overlays using OpenGL. Some of the GPUs can perform only 2D graphics
via OpenVG. They are slowly being re-targeted to be used as additional re-
source for vision algorithms through OpenCL. Nvidia is leading this way pro-
viding a General Purpose GPU processor for vision. With respect to Vision
algorithms, CUDA (Compute Unified Device Architecture) of Nvidia is signif-
icantly more powerful than other GPUs provided by ARM (Mali), PowerVR
(SGX) and Vivante. The performance power of Nvidia GPUs are growing at
a significantly faster rate compared to other processors to suit the growth in
the field of automotive vision applications. CUDA uses SIMT (single instruc-
tion multiple threads) with threading done in hardware. It has a limitation
of dealing with load/store intensive operations which is improved by high
data-bandwidth provided to it.

1.2.4 DSPs

SIMD engines are quite popular in the application areas of image processing. This is because the input data is 8-bit fixed point and the initial stages of image processing are typically embarrassingly parallel. These processors are typically designed to be power-efficient by avoiding floating-point and having a simplified processing pipeline. For instance, performance/watt of TI's Embedded Vision Engine (EVE) [43] is ~8X than that of A15. On the other hand, it lacks flexibility and it is typically used for the pixel-level initial stages of the pipeline. TI and Renesas have small width (8/16) SIMD processors where Mobileye and Freescale make use of a large SIMD width (76/128) processor.

DSPs (Digital Signal Processors) are traditional processors designed for
low-power acceleration of signal processing algorithms. Most of them including
TI's C66x and Toshiba's MPE are VLIW (Very Large Instruction Word) based
1.2.2 FPGAs

FPGA (Field Programmable Gate Array) is closely related to ASICs but has a key advantage of being re-configurable at run-time. From the application perspective, this means the HW can adaptively transform to an accelerator needed for that particular scenario. For instance, if the algorithm has to be drastically different for low and high speeds, the FPGA can transform into one of these as required. But because ASIC is hard-coded to be optimized for one algorithm, FPGA will be lagging in performance. Due to recent progress in FPGA technology, the gap is closing. Xilinx and Altera are the major FPGA suppliers who offer very similar performance FPGAs. Suitable for rapid changing CNN architectures instead of fixed function ASICs.

1.2.3 GPUs

GPU (Graphics Processing Unit) was traditionally designed for graphics ac-
celeration. It can be used for view rendering/remapping and for drawing out-
put overlays using OpenGL. Some of the GPUs can perform only 2D graphics
via OpenVG. They are slowly being re-targeted to be used as additional re-
source for vision algorithms through OpenCL. Nvidia is leading this way pro-
viding a General Purpose GPU processor for vision. With respect to Vision
algorithms, CUDA (Compute Unified Device Architecture) of Nvidia is signif-
icantly more powerful than other GPUs provided by ARM (Mali), PowerVR
(SGX) and Vivante. The performance power of Nvidia GPUs are growing at
a significantly faster rate compared to other processors to suit the growth in
the field of automotive vision applications. CUDA uses SIMT (single instruc-
tion multiple threads) with threading done in hardware. It has a limitation
of dealing with load/store intensive operations which is improved by high
data-bandwidth provided to it.

1.2.4 DSPs

SIMD engines are quite popular in the application areas of image processing. This is because the input data is 8-bit fixed point and the initial stages of image processing are typically embarrassingly parallel. These processors are typically designed to be power-efficient by avoiding floating-point and having a simplified processing pipeline. For instance, performance/watt of TI's Embedded Vision Engine (EVE) [43] is ~8X than that of A15. On the other hand, it lacks flexibility and it is typically used for the pixel-level initial stages of the pipeline. TI and Renesas have small width (8/16) SIMD processors where Mobileye and Freescale make use of a large SIMD width (76/128) processor.

DSPs (Digital Signal Processors) are traditional processors designed for
low-power acceleration of signal processing algorithms. Most of them including
TI's C66x and Toshiba's MPE are VLIW (Very Large Instruction Word) based
superscalar processors. VLIW is an efficient architecture scheme to provide
flexibility compared to SIMD. They also exhibit a form of SIMD using packed
arithmetic i.e. aliasing 4 byte instructions in a word instruction. The gap of
performance normalized to cost of DSP is becoming closer because of lowered
cost and performance improvements of NEON and its ubiquity.

1.2.5 CPUs

GPP (General Purpose Processor) is typically the master control processor of an SOC. Some flavour of ARM Cortex Application processors is commonly used as the GPP. The flavours vary from A9, A15 to A53 and A57. The former two are the popular GPP in the current generation devices and the latter two are 64-bit roadmap processors. NEON is a SIMD engine which can accelerate image processing. Because of the ubiquitous nature of multimedia, importance of NEON has grown and it is becoming more tightly integrated. The only major exception of an SOC not using ARM is Mobileye EyeQ which uses MIPS cores instead. Toshibas TMPV7600 uses their proprietary MeP low power RISC cores but they use A9 in some of the TMPV75 processors. [35] discusses efficient CNN for ARM v7.

1.3 Building Blocks of CNN

In this section, we provide an overview of building blocks of CNN and ab-
stracted design techniques in popular architectures.

1.3.1 Convolutional building block

[45] provides a systematic evaluation of all the hyper-parameters and design criteria. Modular design of replicating building blocks has been successfully demonstrated in ResNet and GoogleNet. We use a simple building block which is a 5x5 or 3x3 convolution with an optional pooling and non-linearity. 3x3 is a popular choice because of smaller number of computations. It has an optional stride as well. From the analysis in [45], we choose ReLU or cross-channel maxout which generalizes ReLU and leaky ReLU. Cross-channel can also be utilized as output channel reduction mechanism. Resolution of the weights play a key role in computational complexity and memory bandwidth. Typically 32-bit floating point is used but there is a lot of empirical evidence to show that 8-bit is sufficient without losing much accuracy. Some layers can optionally be with 16-bit resolution in case a larger dynamic range is needed. De-convolution is needed for the decoder part and there are three ways to achieve it namely upconv, unpooling and deconv. We choose deconv because
superscalar processors. VLIW is an efficient architecture scheme to provide
flexibility compared to SIMD. They also exhibit a form of SIMD using packed
arithmetic i.e. aliasing 4 byte instructions in a word instruction. The gap of
performance normalized to cost of DSP is becoming closer because of lowered
cost and performance improvements of NEON and its ubiquity.

1.2.5 CPUs

GPP (General Purpose Processor) is typically the master control processor of an SOC. Some flavour of ARM Cortex Application processors is commonly used as the GPP. The flavours vary from A9, A15 to A53 and A57. The former two are the popular GPP in the current generation devices and the latter two are 64-bit roadmap processors. NEON is a SIMD engine which can accelerate image processing. Because of the ubiquitous nature of multimedia, importance of NEON has grown and it is becoming more tightly integrated. The only major exception of an SOC not using ARM is Mobileye EyeQ which uses MIPS cores instead. Toshibas TMPV7600 uses their proprietary MeP low power RISC cores but they use A9 in some of the TMPV75 processors. [35] discusses efficient CNN for ARM v7.

1.3 Building Blocks of CNN

In this section, we provide an overview of building blocks of CNN and ab-
stracted design techniques in popular architectures.

1.3.1 Convolutional building block

[45] provides a systematic evaluation of all the hyper-parameters and design criteria. Modular design of replicating building blocks has been successfully demonstrated in ResNet and GoogleNet. We use a simple building block which is a 5x5 or 3x3 convolution with an optional pooling and non-linearity. 3x3 is a popular choice because of smaller number of computations. It has an optional stride as well. From the analysis in [45], we choose ReLU or cross-channel maxout which generalizes ReLU and leaky ReLU. Cross-channel can also be utilized as output channel reduction mechanism. Resolution of the weights play a key role in computational complexity and memory bandwidth. Typically 32-bit floating point is used but there is a lot of empirical evidence to show that 8-bit is sufficient without losing much accuracy. Some layers can optionally be with 16-bit resolution in case a larger dynamic range is needed. De-convolution is needed for the decoder part and there are three ways to achieve it namely upconv, unpooling and deconv. We choose deconv because
it mimics a convolution block via transposed convolution.

Building block parameterization:

* Convolution Kernel size = 3x3 or 5x5

* Convolution Type = conv or deconv

* Data Resolution = 8-bit or 16-bit

* Stride = 1 or 2

* Non-linearity = ReLU or cross-channel maxout

* Pooling = max (2x2)

Conv operations typically use up to 70-90% of operations. Conv operations are very efficient for parallelization and it is commonly used technique to replace non-conv layers like FC layers by GAP layers.

### 1.3.2 Baseline FCN architecture

The simplest CNN architecture is a linear cascade of convolution blocks typically known as Fully convolutional networks (FCN). Standard design is to progressively reduce feature resolution and increase channels. The image width and height are powers of 2 (eg: 640x480) and thus a progressive down-sampling by a factor of 2. Design constraints are listed below. Figure 1.3 illustrates this.

Efficient design constraints:

* Image width/height are usually powers of 2

* Efficient because it is just dropping of alternative pixels

* Channel up-sampling by a factor of 2 creates a symmetric design

* Some layers have no up-sampling to increase non-linearity

* Five scales of down sampling when height degrades to H/32

* 2D structure is important for decoder, cannot be flattened

* Progressive down-sampling by a factor of 2

* May add more layers at same scale (to be determined empirically)

* Minimum of 10 layers in encoder

* Minimum of 2 layers in decoder (deconv layers)

* Wider vs Deeper layers
it mimics a convolution block via transposed convolution.

Building block parameterization:

* Convolution Kernel size = 3x3 or 5x5

* Convolution Type = conv or deconv

* Data Resolution = 8-bit or 16-bit

* Stride = 1 or 2

* Non-linearity = ReLU or cross-channel maxout

* Pooling = max (2x2)

Conv operations typically use up to 70-90% of operations. Conv operations are very efficient for parallelization and it is commonly used technique to replace non-conv layers like FC layers by GAP layers.

### 1.3.2 Baseline FCN architecture

The simplest CNN architecture is a linear cascade of convolution blocks typically known as Fully convolutional networks (FCN). Standard design is to progressively reduce feature resolution and increase channels. The image width and height are powers of 2 (eg: 640x480) and thus a progressive down-sampling by a factor of 2. Design constraints are listed below. Figure 1.3 illustrates this.

Efficient design constraints:

* Image width/height are usually powers of 2

* Efficient because it is just dropping of alternative pixels

* Channel up-sampling by a factor of 2 creates a symmetric design

* Some layers have no up-sampling to increase non-linearity

* Five scales of down sampling when height degrades to H/32

* 2D structure is important for decoder, cannot be flattened

* Progressive down-sampling by a factor of 2

* May add more layers at same scale (to be determined empirically)

* Minimum of 10 layers in encoder

* Minimum of 2 layers in decoder (deconv layers)

* Wider vs Deeper layers
<figure><img src="image_5.png" /><figcaption>FIGURE 1.3</figcaption></figure>

Baseline network with convolution blocks and its parameters (/2 indicates downsampling, number at the end indicate number of channels)

1.3.3 Formalized extensions

We abstracted out the key design ideas based on survey of various networks,
some of which is illustrated in Fig 1.3. We provide canonical references for
each concept.

1) Wider bank of filters: Some networks have utilized wider bank of filters and shown better capacity. This paper shows that 16 wider layers can attain similar capacity of 1000 layers. This can have training advantages of parallelism for training/inference. It can also enable feature dependent gating. Fig 1.4 (top right) illustrates this.

2) Cross channel filters: Cross channel filters can model functional depen-
dencies across channels by marginalizing a 3D convolution block. Inception
used it successfully and Xception optimized it. Fig 1.4 (bottom) illustrates
this.

3) Cross layer filters: The cross layer connections provides exponential ensemble over the function. Resnet skip connections were the simplest way of doing it. Then DenseNet took it to the extreme where all the layers have connections from their previous ones (illustrated in Fig 1.4 top left). Highway net utilized learnable skip connections from data.

Figure 1.4 Illustration of extension filters like cross layer filters (top left), wider filters (top right) and cross channel filters (bottom)

4) Split branching and summation joining: There is a popular trend of summation junctions heavily used. It was introduced by Resnet for skip connections. It is much more efficient than concatenation joining which increases feature dimensionality. It comes for free leveraging DMA_3D in CNN HW block. Other usage is in ResNeXt which extends for more parallel paths (link) and PolyNet generalizes it and achieves efficiency (link).

5) Handling scale of objects: Theoretical insights show CNNs are not scale invariant. There is plenty of empirical evidence to illustrate explicit handling of scale. This is very important in AD for handling small objects. Dilated convolutions are popular for handling multi-scale and can be mapped as multi-scale images with various strides. Preferred design choice is to do multi-scale feature maps as used in SSD incorporating multi-scale feature maps (link) and
<figure><img src="image_5.png" /><figcaption>FIGURE 1.3</figcaption></figure>

Baseline network with convolution blocks and its parameters (/2 indicates downsampling, number at the end indicate number of channels)

1.3.3 Formalized extensions

We abstracted out the key design ideas based on survey of various networks,
some of which is illustrated in Fig 1.3. We provide canonical references for
each concept.

1) Wider bank of filters: Some networks have utilized wider bank of filters and shown better capacity. This paper shows that 16 wider layers can attain similar capacity of 1000 layers. This can have training advantages of parallelism for training/inference. It can also enable feature dependent gating. Fig 1.4 (top right) illustrates this.

2) Cross channel filters: Cross channel filters can model functional dependencies across channels by marginalizing a 3D convolution block. Inception used it successfully and Xception optimized it. Fig 1.4 (bottom) illustrates this.

3) Cross layer filters: The cross layer connections provides exponential ensemble over the function. Resnet skip connections were the simplest way of doing it. Then DenseNet took it to the extreme where all the layers have connections from their previous ones (illustrated in Fig 1.4 top left). Highway net utilized learnable skip connections from data.

Figure 1.4 Illustration of extension filters like cross layer filters (top left), wider filters (top right) and cross channel filters (bottom)

4) Split branching and summation joining: There is a popular trend of summation junctions heavily used. It was introduced by Resnet for skip connections. It is much more efficient than concatenation joining which increases feature dimensionality. It comes for free leveraging DMA_3D in CNN HW block. Other usage is in ResNeXt which extends for more parallel paths (link) and PolyNet generalizes it and achieves efficiency (link).

5) Handling scale of objects: Theoretical insights show CNNs are not scale invariant. There is plenty of empirical evidence to illustrate explicit handling of scale. This is very important in AD for handling small objects. Dilated convolutions are popular for handling multi-scale and can be mapped as multi-scale images with various strides. Preferred design choice is to do multi-scale feature maps as used in SSD incorporating multi-scale feature maps (link) and
<figure><img src="image_6.png" /><figcaption>FIGURE 1.4</figcaption></figure>

Illustration of extension filters like cross layer filters (top left), wider filters (top right) and cross channel filters (bottom) [reproduced figures].

MS-CNN - unified multi-scale (link). Fig 1.6 illustrates different mechanisms to handle scale and preferred design choice.

1.3.4 Performance consideration/metrics

* Runtime and indirectly fps are the most important aspect to achieve real-time interactive system.

* Number of Operations is a way to capture the computational capacity, this is captured via TOPS/GOPS. There is a theoretical value for each network and then practical run-time on the hardware.

* CNNs are very memory intensive and memory bandwidth could become a bottleneck. In a typical computer vision algorithm, the data sized as we
<figure><img src="image_6.png" /><figcaption>FIGURE 1.4</figcaption></figure>

Illustration of extension filters like cross layer filters (top left), wider filters (top right) and cross channel filters (bottom) [reproduced figures].

MS-CNN - unified multi-scale (link). Fig 1.6 illustrates different mechanisms to handle scale and preferred design choice.

1.3.4 Performance consideration/metrics

* Runtime and indirectly fps are the most important aspect to achieve real-time interactive system.

* Number of Operations is a way to capture the computational capacity, this is captured via TOPS/GOPS. There is a theoretical value for each network and then practical run-time on the hardware.

* CNNs are very memory intensive and memory bandwidth could become a bottleneck. In a typical computer vision algorithm, the data sized as we
<figure><img src="image_7.png" /><figcaption>**FIGURE 1.5**</figcaption></figure>

Illustration of extension filters like cross layer filters (top left), wider filters (top right) and cross channel filters (bottom) [reproduced figures].

<figure><img src="image_8.png" /><figcaption>**FIGURE 1.6**</figcaption></figure>

Illustration of various ways to handle scale. Preferred way is to do multiscale feature map (bottom right) [reproduced figures].

progress through the pipeline. In CNN, the data increases by a large factor and then almost remains the same.

* The network also has a lot of parameters which is part of code size. This causes loading of program cache and associated overheads.
<figure><img src="image_7.png" /><figcaption>**FIGURE 1.5**</figcaption></figure>

Illustration of extension filters like cross layer filters (top left), wider filters (top right) and cross channel filters (bottom) [reproduced figures].

<figure><img src="image_8.png" /><figcaption>**FIGURE 1.6**</figcaption></figure>

Illustration of various ways to handle scale. Preferred way is to do multiscale feature map (bottom right) [reproduced figures].

progress through the pipeline. In CNN, the data increases by a large factor and then almost remains the same.

* The network also has a lot of parameters which is part of code size. This causes loading of program cache and associated overheads.
# 1.4 Taxonomy of Optimization Methods

## 1.4.1 Quantization

In Deep Learning applications, standard data format used to be 32-bit floating point for CNN models but full precision (fp32) operations are computationally very intensive both for training and inference. The need to make inference predictions scalable with real time low resource embedded system has boosted research to reduce the precision of data representation in order to achieve more efficient inference computation. There is plenty of empirical evidence that it is possible to reduce to a lower resolution like INT8 with minimal loss in accuracy. Reducing precision can be done not only for the weights but also for the activations. This will reduce computational cost and at the same time will decrease memory needs.

First we will focus on low precision optimizations for inference on pre-trained neural networks like INT8 inference most HW vendors (Nvidia, TI, Renesas) have bet on. Then we will present some works that investigated the low precision training with some successful results to train from scratch networks with binary weights (and activations) using full precision during training.

**Low Resolution Offline Conversion:** This section is then dedicated to introduce techniques to quantize a pretrained model without precision restriction (using fp-32) and somehow convert it automatically to a int-8 low precision model for efficient inference. Approximating full precision floating point values with 8-bit integers introduces quantization noise. Different techniques explore how to convert the full precision weights to low precision while maintaining the level of accuracy as high as possible.

The usual most common low inference pipeline used to deploy hardware accelerated neural networks is currently relying on int8 precision for both weights and activations. This has been enabled by the advent of 8-bit hardware such as Google TPU (link) and software support of 8-bit inference by popular DL frameworks such as Google Tensorflow (link) or Nvidias TensorRT library (link).

Even simple fp-32 to int-8 quantization can be diverse. Indeed quantization process can be either linear on non linear, symmetric or asymmetric, with saturation or without saturation. In this paragraph we will detail the key components of a given solution which is the Tensor RT 8-bit Inference process:

* 8-bit GPU Instructions Recent Nvidia GPUs introduce new 8-bit integer 4-element vector dot product (DP4A) instructions which enables efficient computation.

* FP32 Scaling Factor The quantization process uses a scaling factor for a given tensor
# 1.4 Taxonomy of Optimization Methods

## 1.4.1 Quantization

In Deep Learning applications, standard data format used to be 32-bit floating point for CNN models but full precision (fp32) operations are computationally very intensive both for training and inference. The need to make inference predictions scalable with real time low resource embedded system has boosted research to reduce the precision of data representation in order to achieve more efficient inference computation. There is plenty of empirical evidence that it is possible to reduce to a lower resolution like INT8 with minimal loss in accuracy. Reducing precision can be done not only for the weights but also for the activations. This will reduce computational cost and at the same time will decrease memory needs.

First we will focus on low precision optimizations for inference on pre-trained neural networks like INT8 inference most HW vendors (Nvidia, TI, Renesas) have bet on. Then we will present some works that investigated the low precision training with some successful results to train from scratch networks with binary weights (and activations) using full precision during training.

**Low Resolution Offline Conversion:** This section is then dedicated to introduce techniques to quantize a pretrained model without precision restriction (using fp-32) and somehow convert it automatically to a int-8 low precision model for efficient inference. Approximating full precision floating point values with 8-bit integers introduces quantization noise. Different techniques explore how to convert the full precision weights to low precision while maintaining the level of accuracy as high as possible.

The usual most common low inference pipeline used to deploy hardware accelerated neural networks is currently relying on int8 precision for both weights and activations. This has been enabled by the advent of 8-bit hardware such as Google TPU (link) and software support of 8-bit inference by popular DL frameworks such as Google Tensorflow (link) or Nvidias TensorRT library (link).

Even simple fp-32 to int-8 quantization can be diverse. Indeed quantization process can be either linear on non linear, symmetric or asymmetric, with saturation or without saturation. In this paragraph we will detail the key components of a given solution which is the Tensor RT 8-bit Inference process:

* 8-bit GPU Instructions Recent Nvidia GPUs introduce new 8-bit integer 4-element vector dot product (DP4A) instructions which enables efficient computation.

* FP32 Scaling Factor The quantization process uses a scaling factor for a given tensor
<figure><img src="image_9.png" /><figcaption>**FIGURE 1.7**</figcaption></figure>

INT-8 TensorRT Quantization: Linear Symmetric Quantization with saturation (left) and 8-bit integer 4-element vector dot product Instruction [reproduced figures].

* Linear Symmetric Quantization with saturation Using a threshold to saturate quantized values are very useful for activations to improve accuracy. Threshold is chosen differently per layer in a way that minimizes loss of information between original distribution and quantized distribution (using KL divergence).

* Use of Calibration Dataset To define threshold value you need to run fp32 inference on a calibration dataset.

This technique enable to run INT8 models inference with resulting accuracy comparable with the FP32 baseline.

**Training Neural Networks with Low Precision:** An alternative to quantize trained network is to directly train network with low resolution. Low resolution can be either applied to weights or activations or even gradients. Compared to offline low resolution conversion, training neural networks with low precision results in less or more degraded accuracy. Accuracy is more sensitive to low resolution activations than low resolution weights and even more sensitive to gradients quantization. Different level of quantization can be considered and possibly as low as binarization.

**Binary networks:** Binary networks could act as the gateway to Deep Neural Networks for low-memory, low-computational devices such as mobile phones. Depending on the techniques, binarization can be of the two kinds : either the weights are binarized (memory size is reduced) or both the weights and activations (memory size and computational cost are reduced). Here are some of the most popular works.

**BNN [10]** (Binarized Neural Networks) introduces a method to train neural networks with both binary weights and activations at run time replacing floating point multiplications with bitwise operations such as XNOR-count
<figure><img src="image_9.png" /><figcaption>**FIGURE 1.7**</figcaption></figure>

INT-8 TensorRT Quantization: Linear Symmetric Quantization with saturation (left) and 8-bit integer 4-element vector dot product Instruction [reproduced figures].

* Linear Symmetric Quantization with saturation Using a threshold to saturate quantized values are very useful for activations to improve accuracy. Threshold is chosen differently per layer in a way that minimizes loss of information between original distribution and quantized distribution (using KL divergence).

* Use of Calibration Dataset To define threshold value you need to run fp32 inference on a calibration dataset.

This technique enable to run INT8 models inference with resulting accuracy comparable with the FP32 baseline.

**Training Neural Networks with Low Precision:** An alternative to quantize trained network is to directly train network with low resolution. Low resolution can be either applied to weights or activations or even gradients. Compared to offline low resolution conversion, training neural networks with low precision results in less or more degraded accuracy. Accuracy is more sensitive to low resolution activations than low resolution weights and even more sensitive to gradients quantization. Different level of quantization can be considered and possibly as low as binarization.

**Binary networks:** Binary networks could act as the gateway to Deep Neural Networks for low-memory, low-computational devices such as mobile phones. Depending on the techniques, binarization can be of the two kinds : either the weights are binarized (memory size is reduced) or both the weights and activations (memory size and computational cost are reduced). Here are some of the most popular works.

**BNN [10]** (Binarized Neural Networks) introduces a method to train neural networks with both binary weights and activations at run time replacing floating point multiplications with bitwise operations such as XNOR-count
operations and left or right bit shifts. This enables to achieve a huge gap in
reducing computation complexity, memory footprint and power efficiency.

Even if during inference all operations uses binary arithmetic, during training real-valued variables are required to enable SGD optimization. Actually only weights and activations are binarized to efficiently run the forward pass. Real-valued variables are used for both the gradients and the weights updates involved in the backpropagation process during training.

As a result, two different (binarized and real-valued) versions of the weights
are jointly used. The weights are stored in real-valued accumulators and are
binarized in each iteration for forward propagation and gradient computations.

In addition to be computationally efficient using bit-wise operations, BNNs
are very power efficient not only because fp32 operations consume more energy
but also because they use less memory access (which cost augments with
memory size). This is then expected to reduce energy consumption drastically
(more than 32 times)

eBNN [44] (Embedded Binarized Neural Networks) extends this idea to even more efficient version for inference dedicated to very low memory devices. Basically, BNN addresses the issue of minimizing the memory required to store weights. eBNN on the other hand focuses on minimizing the overall memory used during inference including the memory used for temporaries (intermediate results between layers).

The trick consists only in reordering the computation of inference which
enables to use only a single floating point temporary for the entire network
achieving a 32x space reduction for temporaries which are especially signifi-
cant for convolutional networks. There is however a computational counterpart
since some recomputation may occurs for overlapping pooling. This optimiza-
tion is completely opposite to GPU programming idea which requires large
memory to allow parallel programming.

**XNOR-Net** [54] has more recently proposed a similar weight and activation binarized net with two major additions. First they prove an optimal way to binarize with introducing a scaling factor. In addition they change the bloc structure to reorder layers sequence from (Conv BN Activation Pool) to (BN BinActivation BinConv Pool) enabling to decrease the quantization loss. XNOR-Nets offer the possibility of running state-of-the-art networks on CPUs (rather than GPUs) in real-time.

**Quantized networks:** Hubara and Courbariaux [23] extended their work on BNN to a more general approach with Quantized Neural Networks a.k.a. QNN. This enable to fix a trade-off between model compression and accuracy loss. They pushed further the scope of quantization in DNN training since they achieved a full quantized training including low precision 6-bit gradients, which are very sensitive to reduced precision. Fig 1.8 shows that for a given GoogleNet architecture 4-bit weights and activations QNN achieves 66.5 Top1 Accuracy on Imagenet only 5 below standard full precision accuracy. Combining it with 6-bit gradients during training does not hurt the accuracy. 1-2-6 bit
operations and left or right bit shifts. This enables to achieve a huge gap in
reducing computation complexity, memory footprint and power efficiency.

Even if during inference all operations uses binary arithmetic, during training real-valued variables are required to enable SGD optimization. Actually only weights and activations are binarized to efficiently run the forward pass. Real-valued variables are used for both the gradients and the weights updates involved in the backpropagation process during training.

As a result, two different (binarized and real-valued) versions of the weights
are jointly used. The weights are stored in real-valued accumulators and are
binarized in each iteration for forward propagation and gradient computations.

In addition to be computationally efficient using bit-wise operations, BNNs
are very power efficient not only because fp32 operations consume more energy
but also because they use less memory access (which cost augments with
memory size). This is then expected to reduce energy consumption drastically
(more than 32 times)

eBNN [44] (Embedded Binarized Neural Networks) extends this idea to even more efficient version for inference dedicated to very low memory devices. Basically, BNN addresses the issue of minimizing the memory required to store weights. eBNN on the other hand focuses on minimizing the overall memory used during inference including the memory used for temporaries (intermediate results between layers).

The trick consists only in reordering the computation of inference which
enables to use only a single floating point temporary for the entire network
achieving a 32x space reduction for temporaries which are especially signifi-
cant for convolutional networks. There is however a computational counterpart
since some recomputation may occurs for overlapping pooling. This optimiza-
tion is completely opposite to GPU programming idea which requires large
memory to allow parallel programming.

**XNOR-Net** [54] has more recently proposed a similar weight and activation binarized net with two major additions. First they prove an optimal way to binarize with introducing a scaling factor. In addition they change the bloc structure to reorder layers sequence from (Conv BN Activation Pool) to (BN BinActivation BinConv Pool) enabling to decrease the quantization loss. XNOR-Nets offer the possibility of running state-of-the-art networks on CPUs (rather than GPUs) in real-time.

**Quantized networks:** Hubara and Courbariaux [23] extended their work on BNN to a more general approach with Quantized Neural Networks a.k.a. QNN. This enable to fix a trade-off between model compression and accuracy loss. They pushed further the scope of quantization in DNN training since they achieved a full quantized training including low precision 6-bit gradients, which are very sensitive to reduced precision. Fig 1.8 shows that for a given GoogleNet architecture 4-bit weights and activations QNN achieves 66.5 Top1 Accuracy on Imagenet only 5 below standard full precision accuracy. Combining it with 6-bit gradients during training does not hurt the accuracy. 1-2-6 bit
<table><thead><tr><th>Model</th><th>Top-1</th><th>Top-5</th></tr></thead><tbody><tr><td colspan="3">Binarized activations+weights, during training and test</td></tr><tr><td>BNN</td><td>47.1%</td><td>69.1%</td></tr><tr><td colspan="3">Quantize weights and activations during training and test</td></tr><tr><td>QNN 4-bit</td><td>66.5%</td><td>83.4%</td></tr><tr><td colspan="3">Quantize activation,weights and gradients during training and test</td></tr><tr><td>QNN 6-bit</td><td>66.4%</td><td>83.1%</td></tr><tr><td colspan="3">No Quantization (standard results)</td></tr><tr><td>GoogleNet - our implementation</td><td>71.6%</td><td>91.2%</td></tr></tbody></table>

FIGURE 1.8

Classification test error rates of the GoogleNet model trained on the ImageNet 1000 classification task.

quantization scheme (weight-activation-gradients) seems to be a sweet spot to take in account the relative sensitivity to quantization.

**Ternary Weight networks:** Another special designed quantized networks are Ternary weights networks a.k.a.TWN [38] which constrain the weights to be ternary valued : -1 / 0 / +1. As other quantized weight networks, they address the storage and computational issues. 2-bit storage requirement is needed to store the weights achieving a x16 model compression rate compared to full precision networks. TWN have a extra 0 value weight in addition to Binary Weights Networks which enhance they expressive ability but does not hurt their hardware friendly ability. Indeed O terms don't need to be accumulated for any MAC operation. This work considers only weight quantization.

1.4.2 Efficient filter design

Compared to fully connected network architecture, convolutional neural networks drastically save a huge number of parameters and computational cost by introducing local connections (receptive field defines by filter kernel size) and parameters sharing within a channel (same filter is used in each point of a feature map based on translation symmetry assumption). However CNN still have huge redundancy and most parameters within model architecture are useless. Decreasing model size by limiting depth and features map numbers is a trivial but not optimal strategy since it directly impacts the model accuracy. Different approaches have been investigating to address this issue trying to reduce model size with as little accuracy degradation as possible.

Efficiency linked to architecture design is a challenge to find the optimal trade off between representation capability and computation complexity. This
<table><thead><tr><th>Model</th><th>Top-1</th><th>Top-5</th></tr></thead><tbody><tr><td colspan="3">Binarized activations+weights, during training and test</td></tr><tr><td>BNN</td><td>47.1%</td><td>69.1%</td></tr><tr><td colspan="3">Quantize weights and activations during training and test</td></tr><tr><td>QNN 4-bit</td><td>66.5%</td><td>83.4%</td></tr><tr><td colspan="3">Quantize activation,weights and gradients during training and test</td></tr><tr><td>QNN 6-bit</td><td>66.4%</td><td>83.1%</td></tr><tr><td colspan="3">No Quantization (standard results)</td></tr><tr><td>GoogleNet - our implementation</td><td>71.6%</td><td>91.2%</td></tr></tbody></table>

FIGURE 1.8

Classification test error rates of the GoogleNet model trained on the ImageNet 1000 classification task.

quantization scheme (weight-activation-gradients) seems to be a sweet spot to take in account the relative sensitivity to quantization.

**Ternary Weight networks:** Another special designed quantized networks are Ternary weights networks a.k.a. TWN [38] which constrain the weights to be ternary valued : -1 / 0 / +1. As other quantized weight networks, they address the storage and computational issues. 2-bit storage requirement is needed to store the weights achieving a x16 model compression rate compared to full precision networks. TWN have a extra 0 value weight in addition to Binary Weights Networks which enhance they expressive ability but does not hurt their hardware friendly ability. Indeed O terms don't need to be accumulated for any MAC operation. This work considers only weight quantization.

1.4.2 Efficient filter design

Compared to fully connected network architecture, convolutional neural networks drastically save a huge number of parameters and computational cost by introducing local connections (receptive field defines by filter kernel size) and parameters sharing within a channel (same filter is used in each point of a feature map based on translation symmetry assumption). However CNN still have huge redundancy and most parameters within model architecture are useless. Decreasing model size by limiting depth and features map numbers is a trivial but not optimal strategy since it directly impacts the model accuracy. Different approaches have been investigating to address this issue trying to reduce model size with as little accuracy degradation as possible.

Efficiency linked to architecture design is a challenge to find the optimal trade off between representation capability and computation complexity. This
can be done at different levels which we will review in detail: the kernel filter in the spatial extent; the kernel filter in the channel extent and the connections between distant layers. First group of model improvement is related to filter kernel spatial simplification and regularization. Second group focuses on advanced architecture redefining connections between two consecutive convolutional layers to reduce number of implicated parameters. The third group is dedicated to rethinking connections between non adjacent convolutional layers introducing the concept of skip-connections.

**Scaling the architecture size by limiting depth, channels or complex architecture:** GoogleNet increased depth of the networks with much lower complexity than vanilla stacking conv layers architecture. darknet approach vanilla architecture with no additional trick tradeoff between speed, memory / complexity and param size squeezed vs darknet efficiency can have multiple aspects.

**Regularizing filter kernels:**

**Filter kernels spatial size:** Simonyan & Zisserman [57] democratize with VGG-16 the use of small 3x3 kernels as opposed to the very first convnets as AlexNet[31] and ZF-Net[74] whose design were relying on large receptive fields kernels at least in the early layers. Simonyan & Zisserman discussed that stacking three 3x3 convolutional layers has an effective 7x7 receptive layers but can perform it with less parameters and computation. This is also a way to introduce regularization by forcing kxk layers to have a decomposition through 3x3 layers (with some additional non linearity in between).

**Filter kernel spatial factorization:** Other even more efficient ways to decompose kxk layers is through the combination of asymmetric convolution (1xk kernel associated to a kx1 kernel) and have been used by Paszke [49]. Spatial Factorization into asymmetric convolutions has also been considered when upgrading the architecture of inception module to its second version in [62]. For 3x3 convolution, Szegedy and al have evaluated that this factorized two-layer solution (using a 3x1 convolution followed by a 1x3 convolution) is 33% cheaper for the same number of output filters, if the number of input and output filters is equal. In practice, they showed that this trick gave good results only in the early layers. Ioannou and al in [27] referred to this factorization as sequential separable filters and gave an insight of a more generic way of defining low-rank filters in CNN.

**Other filter kernel regularizers:** In some interpretation dilated convolutions presented by Yu and Koltun in [73] can also be interpreted as a convolution kernel regularizer.

**Regularizing filter kernel through activation functions:** More recently, some strategies to compute multiple features maps with the same filter kernel have been presented as a way to reduce filters redundancy. Based on this idea, Shang and al. built new advanced activation function called Concatenated Rectified Linear Unit in [56]. They observed that early conv
can be done at different levels which we will review in detail: the kernel filter in the spatial extent; the kernel filter in the channel extent and the connections between distant layers. First group of model improvement is related to filter kernel spatial simplification and regularization. Second group focuses on advanced architecture redefining connections between two consecutive convolutional layers to reduce number of implicated parameters. The third group is dedicated to rethinking connections between non adjacent convolutional layers introducing the concept of skip-connections.

**Scaling the architecture size by limiting depth, channels or complex architecture:** GoogleNet increased depth of the networks with much lower complexity than vanilla stacking conv layers architecture. darknet approach vanilla architecture with no additional trick tradeoff between speed, memory / complexity and param size squeezed vs darknet efficiency can have multiple aspects.

**Regularizing filter kernels:**

**Filter kernels spatial size:** Simonyan & Zisserman [57] democratize with VGG-16 the use of small 3x3 kernels as opposed to the very first convnets as AlexNet[31] and ZF-Net[74] whose design were relying on large receptive fields kernels at least in the early layers. Simonyan & Zisserman discussed that stacking three 3x3 convolutional layers has an effective 7x7 receptive layers but can perform it with less parameters and computation. This is also a way to introduce regularization by forcing kxk layers to have a decomposition through 3x3 layers (with some additional non linearity in between).

**Filter kernel spatial factorization:** Other even more efficient ways to decompose kxk layers is through the combination of asymmetric convolution (1xk kernel associated to a kx1 kernel) and have been used by Paszke [49]. Spatial Factorization into asymmetric convolutions has also been considered when upgrading the architecture of inception module to its second version in [62]. For 3x3 convolution, Szegedy and al have evaluated that this factorized two-layer solution (using a 3x1 convolution followed by a 1x3 convolution) is 33% cheaper for the same number of output filters, if the number of input and output filters is equal. In practice, they showed that this trick gave good results only in the early layers. Ioannou and al in [27] referred to this factorization as sequential separable filters and gave an insight of a more generic way of defining low-rank filters in CNN.

**Other filter kernel regularizers:** In some interpretation dilated convolutions presented by Yu and Koltun in [73] can also be interpreted as a convolution kernel regularizer.

**Regularizing filter kernel through activation functions:** More recently, some strategies to compute multiple features maps with the same filter kernel have been presented as a way to reduce filters redundancy. Based on this idea, Shang and al. built new advanced activation function called Concatenated Rectified Linear Unit in [56]. They observed that early conv
layers of well-known convnets such AlexNet capture both negative and posi-
tive phase information through learning pairs of negatively correlated lters.
CReLU exploits this redundancy by duplicating linear response of convolution
and negating the copy before concatenating it with the original response. This
process which makes more efficient use of trainable parameters has been
adopted by Kim and al in the initial layers of the PVANet architecture [30]
to built lightweight network for real-time object detection.

**Regularizing filter kernel through symmetry:**

Similar filter weights reuse to reduce redundancy has been taken advantage of
by Cohen and Welling with Group equivariant Convolutional Neural Networks
(G-CNNs) in [9] by exploiting symmetries. CNN equivariance to translation
is then extended to G-CNN equivariance to rotation and even reflections.

To address the problem of efficiency, we saw that it is possible to reduce
redundancy by giving some more constraints to the different filter kernels.

**Input feature maps channels low dimensional embedding / convolutional layer design:** Enlarging our view on the global convnet architecture, we will know discuss how we can shape the structure of the volume of the input layer more efficient when defining the connections between two consecutive layers. Indeed, even if CNN parameters are shared by neurons within the same feature maps, the number of parameter involved in the computation of a single feature map is directly linked to the number of channels of the previous layers. Different strategies have been elaborated to a alleviates this constraint such as feature maps dimensionality reduction by 1x1 convolution, group convolution or more recently exploited separable convolutions.

**1x1 convolutions are non linear dimensionality reduction** : Almost every recent architecture rely on 1x1 conv. Among first papers to have introduced such 1x1 convolutions are [41] [57] [61] [62] [25]. Lin and al in [41] have presented this 1x1 convolution under a specific angle linked to Goodfellow and al work on Maxout Networks [13]. They defined it as a cross channel (or cross feature map) parametric pooling layer. Each pooling layer performs weighted linear recombination on the input feature maps, which then go through a rectifier linear Unit which is also equivalent to a convolution layer with 1x1 convolution kernel.

Standard convolution operation filters existing features in order to produce
new representation. 1x1 convolution are in some way different from larger ker-
nel convolution since they do not intend to extract hierarchical features on
top of earlier features maps but instead try to combine existing features maps
together. Simonyan & Zisserman [57] used 1x1 convolutions as linear projec-
tion but onto a space of the same dimensionality. Rapidly, many contributions
turned the 1x1 conv usage into projection in order to reduce channel dimen-
sionality of input volume, sometimes coupled with symmetrical 1x1 expansion
convolutions [49] or feature maps concatenations [61]. This usage is most of
the time coupled within architectural high-level modules [61] [25] or bottle-
neck [49]. In addition [49] further simplifies 1x1 convolution by removing biases
without any impact on accuracy.
layers of well-known convnets such AlexNet capture both negative and posi-
tive phase information through learning pairs of negatively correlated lters.
CReLU exploits this redundancy by duplicating linear response of convolution
and negating the copy before concatenating it with the original response. This
process which makes more efficient use of trainable parameters has been
adopted by Kim and al in the initial layers of the PVANet architecture [30]
to built lightweight network for real-time object detection.

**Regularizing filter kernel through symmetry:**

Similar filter weights reuse to reduce redundancy has been taken advantage of
by Cohen and Welling with Group equivariant Convolutional Neural Networks
(G-CNNs) in [9] by exploiting symmetries. CNN equivariance to translation
is then extended to G-CNN equivariance to rotation and even reflections.

To address the problem of efficiency, we saw that it is possible to reduce
redundancy by giving some more constraints to the different filter kernels.

**Input feature maps channels low dimensional embedding / convolutional layer design:** Enlarging our view on the global convnet architecture, we will know discuss how we can shape the structure of the volume of the input layer more efficient when defining the connections between two consecutive layers. Indeed, even if CNN parameters are shared by neurons within the same feature maps, the number of parameter involved in the computation of a single feature map is directly linked to the number of channels of the previous layers. Different strategies have been elaborated to a alleviates this constraint such as feature maps dimensionality reduction by 1x1 convolution, group convolution or more recently exploited separable convolutions.

**1x1 convolutions are non linear dimensionality reduction** : Almost every recent architecture rely on 1x1 conv. Among first papers to have introduced such 1x1 convolutions are [41] [57] [61] [62] [25]. Lin and al in [41] have presented this 1x1 convolution under a specific angle linked to Goodfellow and al work on Maxout Networks [13]. They defined it as a cross channel (or cross feature map) parametric pooling layer. Each pooling layer performs weighted linear recombination on the input feature maps, which then go through a rectifier linear Unit which is also equivalent to a convolution layer with 1x1 convolution kernel.

Standard convolution operation filters existing features in order to produce
new representation. 1x1 convolution are in some way different from larger ker-
nel convolution since they do not intend to extract hierarchical features on
top of earlier features maps but instead try to combine existing features maps
together. Simonyan & Zisserman [57] used 1x1 convolutions as linear projec-
tion but onto a space of the same dimensionality. Rapidly, many contributions
turned the 1x1 conv usage into projection in order to reduce channel dimen-
sionality of input volume, sometimes coupled with symmetrical 1x1 expansion
convolutions [49] or feature maps concatenations [61]. This usage is most of
the time coupled within architectural high-level modules [61] [25] or bottle-
neck [49]. In addition [49] further simplifies 1x1 convolution by removing biases
without any impact on accuracy.
Iandola and al in [25] explicitly formalized the non linear dimensionality reduction within their microarchitectural fire modules composed of two sub-layers called squeeze layers and expand layers. With this fire module architecture they managed to reach AlexNet-level accuracy on Imagenet challenge with 50x fewer parameters. To achieve this performance they followed different strategies one of which is to decrease the number of input channels to kxk filters (k=3 in their work). Their reasoning is very elementary but yet ultra efficient : The total quantity of parameters in this layer is (number of input channels) * (number of filters) * (k\*k). So, to maintain a small total number of parameters in a CNN, it is important not only to decrease the number of kxk filters, but also to decrease the number of input channels to the kxk filters. Using squeeze layers (1x1 conv) they can decrease the number of input channels to kxk filters the way they want. As a matter of fact, these 1x1 convolutions are very light regarding parameters and computation and have a huge impact on next layer efficiency reducing its input volume. Recent architectures widely rely on this strategy.

**Filter groups:** Another similar way to minimize involved parameters in a given layer is filter groups. This very old concept dates back to 2012 and Krizhevsky and al in AlexNet [31] implementation whose architecture was specifically designed with two separate convolutional filter groups across most of the layers. As explained by the authors, the main motivation of this design was at that time to enable training across two GPUs to overcome memory constraints, filter groups allowing more efficient model-parallelization across the GPUs. What has been seen during many years as an engineering trick has more recently been presented as an efficient technique to remove parameters in the model and at the same to learn better representations.

In a given layer standard convolutional filters have the same channel dimension c as the input volume on which they operate. (Every single filter sees all feature map channels within the input volume). Introducing filter grouping, we can design a model where filters operate on a fraction of this input volume by creating g independent groups of filters with associated groups of feature map input channels. (Every single filter now sees only a fraction of the feature map channels within the input volume reducing filter dimensions in the channel extent from h.w.Ci to h.w.Ci/g). This change does not affect the dimensions of the input and output feature maps but significantly reduces computational complexity and the number of model parameters.This design will also benefit from the side effect to structure learned filters into distinct groups, as illustrated by black-and-white and colour filters in the two group filters architecture of the first AlexNet layer. ResNext architecture [71] proposed a model which integrates group convolution within ResNet building blocks. Ioannou and al combined this filters groups with 1x1 low dimensional embedding to build root modules in [26]. Huang and al in [20] combined it with meta learning to introduced learned group convolution. They both achieved
Iandola and al in [25] explicitly formalized the non linear dimensionality reduction within their microarchitectural fire modules composed of two sub-layers called squeeze layers and expand layers. With this fire module architecture they managed to reach AlexNet-level accuracy on Imagenet challenge with 50x fewer parameters. To achieve this performance they followed different strategies one of which is to decrease the number of input channels to kxk filters (k=3 in their work). Their reasoning is very elementary but yet ultra efficient : The total quantity of parameters in this layer is (number of input channels) * (number of filters) * (k\*k). So, to maintain a small total number of parameters in a CNN, it is important not only to decrease the number of kxk filters, but also to decrease the number of input channels to the kxk filters. Using squeeze layers (1x1 conv) they can decrease the number of input channels to kxk filters the way they want. As a matter of fact, these 1x1 convolutions are very light regarding parameters and computation and have a huge impact on next layer efficiency reducing its input volume. Recent architectures widely rely on this strategy.

**Filter groups:** Another similar way to minimize involved parameters in a given layer is filter groups. This very old concept dates back to 2012 and Krizhevsky and al in AlexNet [31] implementation whose architecture was specifically designed with two separate convolutional filter groups across most of the layers. As explained by the authors, the main motivation of this design was at that time to enable training across two GPUs to overcome memory constraints, filter groups allowing more efficient model-parallelization across the GPUs. What has been seen during many years as an engineering trick has more recently been presented as an efficient technique to remove parameters in the model and at the same to learn better representations.

In a given layer standard convolutional filters have the same channel dimension c as the input volume on which they operate. (Every single filter sees all feature map channels within the input volume). Introducing filter grouping, we can design a model where filters operate on a fraction of this input volume by creating g independent groups of filters with associated groups of feature map input channels. (Every single filter now sees only a fraction of the feature map channels within the input volume reducing filter dimensions in the channel extent from h.w.Ci to h.w.Ci/g). This change does not affect the dimensions of the input and output feature maps but significantly reduces computational complexity and the number of model parameters.This design will also benefit from the side effect to structure learned filters into distinct groups, as illustrated by black-and-white and colour filters in the two group filters architecture of the first AlexNet layer. ResNext architecture [71] proposed a model which integrates group convolution within ResNet building blocks. Ioannou and al combined this filters groups with 1x1 low dimensional embedding to build root modules in [26]. Huang and al in [20] combined it with meta learning to introduced learned group convolution. They both achieved
18*Efficient CNN Architecture Design Techniques for Real-time Embedded Systems Deployment*

state-of-the-art architecture regarding model efficiency.

**Depth-wise Separable Convolutions:** Some similar approach to effi-
ciently reduce the channel extent of filter kernel size is the use of depth-wise
separable convolution which pushes it to the extreme case where every sin-
gle filter sees the input volume through a learned projection on a 1-D space,
i.e a single input channel. These convolution were first used in the Xception
model [8]. They can be interpreted as an extreme version of the GoogleNet
inception module which consists of channel-wise spatial convolution followed
by a point-wise 1x1 convolution projecting the channels output onto a new
channel space. However they differ from Inception module in two ways : the
channel-wise spatial convolution is performed first and there is no ReLU non
linearities. In other words, a depth-wise separable convolution split a stan-
dard 3D convolution into two successive layers : first one filters and second on
combines outputs from first layers.

Wang and al in [69] and [68] build similar architecture using the same
concept they named factorized CNN with intra channel convolution. Com-
plexifying the model they added ReLus and residual connections.Flattened
Convolutional Neural Networks [29] coupled separable convolution and asym-
metric convolution factorization in the spatial extent. Mobilenets [19] propose
hyper-parametric model to propose light model for resource restricted appli-
cation that can trade-off latency and accuracy

**Efficiency : Depth-wise Separable Convolutions vs group Convolution:**

If we look at the parameter reduction capability of these two approaches, we
can discuss about their comparative efficiency. Considering convolutional layer
between two features map volume with M channels for input volume and N
channels for output volume in a basic scenario of k\*k spatial filter kernel.
Standard convolution will have k\*k\*N\*M parameters Separable Convolution
will have k\*k\*M + 1\*1\*N\*M parameters corresponding to a 1/N + 1/k2 re-
duction Group Convolution will have k\*k\*N\*M/g parameters corresponding
to a g reduction Same reduction ratio is valid for computation In separable
convolution all the complexity is then related to 1x1 convolution. Combining it
with group convolution within 1x1 conv layer provides even more efficient ar-
chitecture. This combined strategy inside a ResNet bottleneck with additional
shuffle channel to maintain information flow has been illustrated by Zhang and
al with their ShuffleNet architecture in [75]. This decreased complexity could
be used to increase channels numbers to enable better information represen-
tation. Model is derived in different complexity linked to hyper parameter
enabling trade off between complexity and accuracy.

**Advanced Skip Connections between distant layer connections:**

In the search of efficient network architectures, some trend has been to look
in a different direction than the one which consists in stacking up custom
convolutional layers and has investigated on some more advanced types of
connection between layers.
18*Efficient CNN Architecture Design Techniques for Real-time Embedded Systems Deployment*

state-of-the-art architecture regarding model efficiency.

**Depth-wise Separable Convolutions:** Some similar approach to effi-
ciently reduce the channel extent of filter kernel size is the use of depth-wise
separable convolution which pushes it to the extreme case where every sin-
gle filter sees the input volume through a learned projection on a 1-D space,
i.e a single input channel. These convolution were first used in the Xception
model [8]. They can be interpreted as an extreme version of the GoogleNet
inception module which consists of channel-wise spatial convolution followed
by a point-wise 1x1 convolution projecting the channels output onto a new
channel space. However they differ from Inception module in two ways : the
channel-wise spatial convolution is performed first and there is no ReLU non
linearities. In other words, a depth-wise separable convolution split a stan-
dard 3D convolution into two successive layers : first one filters and second on
combines outputs from first layers.

Wang and al in [69] and [68] build similar architecture using the same
concept they named factorized CNN with intra channel convolution. Com-
plexifying the model they added ReLus and residual connections.Flattened
Convolutional Neural Networks [29] coupled separable convolution and asym-
metric convolution factorization in the spatial extent. Mobilenets [19] propose
hyper-parametric model to propose light model for resource restricted appli-
cation that can trade-off latency and accuracy

**Efficiency : Depth-wise Separable Convolutions vs group Convolution:**

If we look at the parameter reduction capability of these two approaches, we
can discuss about their comparative efficiency. Considering convolutional layer
between two features map volume with M channels for input volume and N
channels for output volume in a basic scenario of k\*k spatial filter kernel.
Standard convolution will have k\*k\*N\*M parameters Separable Convolution
will have k\*k\*M + 1\*1\*N\*M parameters corresponding to a 1/N + 1/k2 re-
duction Group Convolution will have k\*k\*N\*M/g parameters corresponding
to a g reduction Same reduction ratio is valid for computation In separable
convolution all the complexity is then related to 1x1 convolution. Combining it
with group convolution within 1x1 conv layer provides even more efficient ar-
chitecture. This combined strategy inside a ResNet bottleneck with additional
shuffle channel to maintain information flow has been illustrated by Zhang and
al with their ShuffleNet architecture in [75]. This decreased complexity could
be used to increase channels numbers to enable better information represen-
tation. Model is derived in different complexity linked to hyper parameter
enabling trade off between complexity and accuracy.

**Advanced Skip Connections between distant layer connections:**

In the search of efficient network architectures, some trend has been to look
in a different direction than the one which consists in stacking up custom
convolutional layers and has investigated on some more advanced types of
connection between layers.
**Residual connections:** First early attempt have been done by Microsoft Research teams which introduced residual connections in [17]. These residual connections can be seen as identity mappings between distant layers. First goal was to face the challenge of training deeper nets with more effective gradient flow during training process. As a result it helps in the same time to build more efficient feature representation thanks to more hierarchical discriminative power with deeper nets and indirect feature reuse.

**Densely connected nets:** Residual connections were the initial approach to extend the basic simplistic vanilla convnet architecture where some layer incoming connections come from the only previous layer. Densely Connected Convolutional Networks, presented by Huang and al in [21], extended further the ideas of skips connections designing a model where some given layer is connecting each layer to every other layer in a feed-forward fashion. As a result, where traditional L layers neural networks have L connections paths, this new design enables $L(L+1)/2$ direct connections paths, meaning that for a given layer, all preceding layers features maps are considered as inputs, enabling feature reuse and combination of low level features with high level features. Introducing direct connections between any two layers (within the same dense block) any learned feature map can be accessed by all subsequent layers, leading to more compact models encouraged by feature reuse. Additionally to feature reuse and model compactness, Densely Connected Convolutional Networks has others advantages among which gradient flow and features propagation nice properties and deep supervision.

**Densely connected nets with increasing growth rate:** More recent architecture presented by [20] and called Condensenet extends Densely connected nets in two ways. First it introduced exponentially increasing growth rate. Original DenseNet design adds k new feature maps at each layer, where this k value is called the growth rate. Based on the fact deeper layers in Densenet used more high-level features than low-level features, the authors modified the growth rate in a way the short range connections were strengthened with k value getting bigger in the deeper layers of the network. The second improvement rely on learned grouped convolutions, which are based on filter groups we introduced in previous subsections coupled with an advanced learning strategy which enabled to automatically select important connections and subsequently prune away a fraction of unimportant filters for which weights have the lowest magnitude. This modular connectivity within the net is directly related to meta learning concepts we will detailed in the next part of this survey.

### 1.4.3 Model Compression

Deep compression could be defined as a field dedicated to compressing learned models after training. As opposed to previously presented meta learning techniques this family of algorithms could be viewed as post-processing enabling to drastically reduce the number of parameters describing the whole network.
**Residual connections:** First early attempt have been done by Microsoft Research teams which introduced residual connections in [17]. These residual connections can be seen as identity mappings between distant layers. First goal was to face the challenge of training deeper nets with more effective gradient flow during training process. As a result it helps in the same time to build more efficient feature representation thanks to more hierarchical discriminative power with deeper nets and indirect feature reuse.

**Densely connected nets:** Residual connections were the initial approach to extend the basic simplistic vanilla convnet architecture where some layer incoming connections come from the only previous layer. Densely Connected Convolutional Networks, presented by Huang and al in [21], extended further the ideas of skips connections designing a model where some given layer is connecting each layer to every other layer in a feed-forward fashion. As a result, where traditional L layers neural networks have L connections paths, this new design enables $L(L+1)/2$ direct connections paths, meaning that for a given layer, all preceding layers features maps are considered as inputs, enabling feature reuse and combination of low level features with high level features. Introducing direct connections between any two layers (within the same dense block) any learned feature map can be accessed by all subsequent layers, leading to more compact models encouraged by feature reuse. Additionally to feature reuse and model compactness, Densely Connected Convolutional Networks has others advantages among which gradient flow and features propagation nice properties and deep supervision.

**Densely connected nets with increasing growth rate:** More recent architecture presented by [20] and called Condensenet extends Densely connected nets in two ways. First it introduced exponentially increasing growth rate. Original DenseNet design adds k new feature maps at each layer, where this k value is called the growth rate. Based on the fact deeper layers in Densenet used more high-level features than low-level features, the authors modified the growth rate in a way the short range connections were strengthened with k value getting bigger in the deeper layers of the network. The second improvement rely on learned grouped convolutions, which are based on filter groups we introduced in previous subsections coupled with an advanced learning strategy which enabled to automatically select important connections and subsequently prune away a fraction of unimportant filters for which weights have the lowest magnitude. This modular connectivity within the net is directly related to meta learning concepts we will detailed in the next part of this survey.

### 1.4.3 Model Compression

Deep compression could be defined as a field dedicated to compressing learned models after training. As opposed to previously presented meta learning techniques this family of algorithms could be viewed as post-processing enabling to drastically reduce the number of parameters describing the whole network.
**Atomic Pruning : Weight Pruning** Most of the key concepts of deep compression have been introduced by Song Han and al building compression framework around the very basic idea of weight pruning [16]. This was presented as a way to alleviate the predefined architecture constraints by trying to learn both weights and connections in a three steps process. Conventional optimization process during training will minimize some cost function combining data loss and regularization loss. Regularization loss will push the parameters towards zero value to avoid overfitting and build simple rules. This leads the author to the conclusion that parameters with value close to zero describes connections which are not important for the learnt model. Their 3-steps method first learns a model in a classical manner; then prunes unimportant connections based on their weight value and finally performs a last step which finetunes the remaining connections. This technique leads to impressive compression results by reducing the number of parameters of Alexnet by a factor x9 with no loss in accuracy. In complementary works [15] the author further extended its pipeline with parameter quantization, weight sharing and huffman coding demonstrating a size reduction of 50x of the parameter set for Alexnet. The main motivation of deep compression is to reduce model size for offline storage in nonvolatile memory or for network transfers. It is also required for small internal memory architectures like FGPA where there is very little internal memory. The major drawback of this technique is that the resulting compressed model is sparse and that the gain in computation speed is far below the parameter reduction factor. Indeed most common computation platform do not take benefit of sparsity with their highly parallel dense matrix computation scheme.

**Sparsity should be powered by Hardware acceleration** On the other hand dedicated hardware enable to have significant gain in computation speed and energy efficiency. Following this idea, Song Han presented in [14] an efficient inference engine that performs inference on the compressed DNN. This design takes advantage of compressed DNN sparsity with sparse matrix-vector multiplication with weight sharing. The resulting computation speed are impressive achieving 189 and 13 speed gain when compared to CPU and GPU implementations of the same DNN without compression. In top of that, model compression enables networks to fit in on-chip SRAM which reduces energy consumption per memory read by a factor x120 compared from fetching weights from DRAM. Combined with energy savings related to sparsity and weight sharing, total energy consumption gain reaches a x2800 factor.

**Structured Pruning : Filter Pruning** A very similar approach to weight pruning is filter pruning. Whereas weight pruning results in sparse connectivity pattern, removing whole filters and their associated feature maps preserve dense connectivity. Consequently computation cost reduction does not rely on sparse convolution libraries or dedicated hardware and existng efficient BLAS libraries for dense matrix multiplication can be further used. Li and al in [39] detailed a method with the same principles as weight pruning selecting filters to prune by computing sum of its absolute kernel weights and
**Atomic Pruning : Weight Pruning** Most of the key concepts of deep compression have been introduced by Song Han and al building compression framework around the very basic idea of weight pruning [16]. This was presented as a way to alleviate the predefined architecture constraints by trying to learn both weights and connections in a three steps process. Conventional optimization process during training will minimize some cost function combining data loss and regularization loss. Regularization loss will push the parameters towards zero value to avoid overfitting and build simple rules. This leads the author to the conclusion that parameters with value close to zero describes connections which are not important for the learnt model. Their 3-steps method first learns a model in a classical manner; then prunes unimportant connections based on their weight value and finally performs a last step which finetunes the remaining connections. This technique leads to impressive compression results by reducing the number of parameters of Alexnet by a factor x9 with no loss in accuracy. In complementary works [15] the author further extended its pipeline with parameter quantization, weight sharing and huffman coding demonstrating a size reduction of 50x of the parameter set for Alexnet. The main motivation of deep compression is to reduce model size for offline storage in nonvolatile memory or for network transfers. It is also required for small internal memory architectures like FGPA where there is very little internal memory. The major drawback of this technique is that the resulting compressed model is sparse and that the gain in computation speed is far below the parameter reduction factor. Indeed most common computation platform do not take benefit of sparsity with their highly parallel dense matrix computation scheme.

**Sparsity should be powered by Hardware acceleration** On the other hand dedicated hardware enable to have significant gain in computation speed and energy efficiency. Following this idea, Song Han presented in [14] an efficient inference engine that performs inference on the compressed DNN. This design takes advantage of compressed DNN sparsity with sparse matrix-vector multiplication with weight sharing. The resulting computation speed are impressive achieving 189 and 13 speed gain when compared to CPU and GPU implementations of the same DNN without compression. In top of that, model compression enables networks to fit in on-chip SRAM which reduces energy consumption per memory read by a factor x120 compared from fetching weights from DRAM. Combined with energy savings related to sparsity and weight sharing, total energy consumption gain reaches a x2800 factor.

**Structured Pruning : Filter Pruning** A very similar approach to weight pruning is filter pruning. Whereas weight pruning results in sparse connectivity pattern, removing whole filters and their associated feature maps preserve dense connectivity. Consequently computation cost reduction does not rely on sparse convolution libraries or dedicated hardware and existng efficient BLAS libraries for dense matrix multiplication can be further used. Li and al in [39] detailed a method with the same principles as weight pruning selecting filters to prune by computing sum of its absolute kernel weights and
sorting filters according to it, then pruning the ones with smallest sum values and finally retraining modified DNN to regain accuracy. In a fairly different way Wen and al [70] proposed filter pruning using model structure learning using group lasso which is an efficient regularization to learn sparse structures. In fact their method is even more general than filter regularization since the Structured Sparsity Learning (SSL) method can regularize any structure (i.e., lters, channels, lter shapes, and layer depth) of DNNs. This learning technique acts like a compression method to learn compressed model from bigger DNN reducing compuattion cost without hurting the computation pipeline by generating hardware friendly structred saprsity. Experiments proved an average 5.1and 3.1speedups against CPU and GPU on reference AlexNet.

**Miscellaneous topics:**

**Topological redundancy:** Topology in this context refers to the spatial structure of the data flow graph of CNN. Firstly, there is inherent symmetry in the weight space [2] which can be captured by a Riemannian manifold as tried out in Segnet [1]. Secondly, the functional graph might be over parameterized and some links/nodes could be removed without affecting the accuracy significantly. The NIPS 2015 paper by Nvidias Dally et al [16] was the first one to show a large improvement by reducing the topological redundancy in the graph and achieving a improvement of 13x less parameters for VGG16. They use a greedy method to sequentially remove less important links/edges and re-train. As in classical sparse matrix operations, sparse algorithmsarent cache flow and SIMD friendly and causes a huge penalty. Typical rule of thumb is to have at least 10x sparsity to go with a sparse algorithm. There is a recent paper which tries to find structured sparsity which is parallelization and cache friendly. It is called Structured sparsity learning (SSL) which is open source and more elegant than Dallys paper [70].

**Filter Redundancy:** For large convolution filters, a low rank factorization is a common technique even outside CNNs. In particular, a rank-1 factorization using SVD creates separability across horizontal and vertical dimensions and very fast data traversal [58]. As CNN uses 3D tensors, the factorization can be in 3 dimensions using tensor SVD. I had this idea of jointly decomposing all the filters in the filter bank by a bayesian optimization search scheme. This exploits joint redundancies across the filters. [36] discusses fast versions of convolution filtering.

**Compression of Parameters:** Deep Compression [15] demonstrated a size reduction of 50x of the parameter set. The motivation is that parameter set is very large and good to compress for offline storage in nonvolatile memory or network transfers. It is also required for small internal memory architectures like FGPA where there is very little internal memory. Squeezenet [25] is another open source method which optimizes by 50x.
sorting filters according to it, then pruning the ones with smallest sum values and finally retraining modified DNN to regain accuracy. In a fairly different way Wen and al [70] proposed filter pruning using model structure learning using group lasso which is an efficient regularization to learn sparse structures. In fact their method is even more general than filter regularization since the Structured Sparsity Learning (SSL) method can regularize any structure (i.e., lters, channels, lter shapes, and layer depth) of DNNs. This learning technique acts like a compression method to learn compressed model from bigger DNN reducing compuattion cost without hurting the computation pipeline by generating hardware friendly structred saprsity. Experiments proved an average 5.1and 3.1speedups against CPU and GPU on reference AlexNet.

**Miscellaneous topics:**

**Topological redundancy:** Topology in this context refers to the spatial structure of the data flow graph of CNN. Firstly, there is inherent symmetry in the weight space [2] which can be captured by a Riemannian manifold as tried out in Segnet [1]. Secondly, the functional graph might be over parameterized and some links/nodes could be removed without affecting the accuracy significantly. The NIPS 2015 paper by Nvidias Dally et al [16] was the first one to show a large improvement by reducing the topological redundancy in the graph and achieving a improvement of 13x less parameters for VGG16. They use a greedy method to sequentially remove less important links/edges and re-train. As in classical sparse matrix operations, sparse algorithmsarent cache flow and SIMD friendly and causes a huge penalty. Typical rule of thumb is to have at least 10x sparsity to go with a sparse algorithm. There is a recent paper which tries to find structured sparsity which is parallelization and cache friendly. It is called Structured sparsity learning (SSL) which is open source and more elegant than Dallys paper [70].

**Filter Redundancy:** For large convolution filters, a low rank factorization is a common technique even outside CNNs. In particular, a rank-1 factorization using SVD creates separability across horizontal and vertical dimensions and very fast data traversal [58]. As CNN uses 3D tensors, the factorization can be in 3 dimensions using tensor SVD. I had this idea of jointly decomposing all the filters in the filter bank by a bayesian optimization search scheme. This exploits joint redundancies across the filters. [36] discusses fast versions of convolution filtering.

**Compression of Parameters:** Deep Compression [15] demonstrated a size reduction of 50x of the parameter set. The motivation is that parameter set is very large and good to compress for offline storage in nonvolatile memory or network transfers. It is also required for small internal memory architectures like FGPA where there is very little internal memory. Squeezenet [25] is another open source method which optimizes by 50x.
### 1.4.4 Data/domain dependent optimizations

**Cascade CNN:** One of the first successfully machine learning application in computer vision is Viola-jones algorithm [67] which is based on adaboost cascade for efficient computation.

To improve the efficiency of our prototype, we are looking into one strategy of cascade rejection. For example, in the picture above, we can eliminate the objects we are not interested in (sky, buildings, vegetation) within the CNN encoder pipeline within first stages of CNN layers. This strategy was first implemented by the AlexNet guy and others at Google for fast PD (link) and there is also some extension to semantic segmentation (link). At high level, the idea is similar to adaboost cascade but done within the CNN layers. To explain it naively at high level, imagine there is a 9 layer network and it is split into 3 stages. Apart from the end2end loss for object detection accuracy, first stage has an additional loss function to reject pixels and only accepted pixels go to second stage (similar to rejection cascade in our PD). The standard proposal generation mechanisms like Faster-RCNN can be viewed as a special case with one rejection stage which generates proposals.

**Cascaded Classifiers:** Concatenation of several classifiers where output from one classifier is fed into the next one. Coarse-to-fine strategy: Reject some false positives at each stage. Efficient: reject east false candidates rapidly, avoid computational costs of the next stages. Accurate: Later classifiers: get more and more discriminative.

One can take advantage of intermediate layers to construct Cascaded Rejection Classifiers. Filter negative object proposals in a cascaded manner Classifiers applied and learnt on different convolutional layers in order to exploit features at different scales. Speeds up detection, maintain high accuracy. Consider CNN as a cascade of layers [40].

**Shared encoder multi-task CNN:** Multi-task learning (MTL) is a sub-field of machine learning in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately. Neural networks have capability to learn manifold representations from large amounts of high dimensional data. It is a well known fact that these representations are generic and transferable knowledge for learning related tasks in a much efficient way. Recent times Deep Learning based methods have shown a tremendous success in Multi Task Learning. In this report we outline significant and novel deep learning architectures related to Multi Task Learning.

We propose a network architecture with shared encoder which can be jointly learned. The main advantages are increased efficiency, scalability to add more tasks leveraging previous features and better generalization through inductive transfer (learning transferable features for tasks) [48].
### 1.4.4 Data/domain dependent optimizations

**Cascade CNN:** One of the first successfully machine learning application in computer vision is Viola-jones algorithm [67] which is based on adaboost cascade for efficient computation.

To improve the efficiency of our prototype, we are looking into one strategy of cascade rejection. For example, in the picture above, we can eliminate the objects we are not interested in (sky, buildings, vegetation) within the CNN encoder pipeline within first stages of CNN layers. This strategy was first implemented by the AlexNet guy and others at Google for fast PD (link) and there is also some extension to semantic segmentation (link). At high level, the idea is similar to adaboost cascade but done within the CNN layers. To explain it naively at high level, imagine there is a 9 layer network and it is split into 3 stages. Apart from the end2end loss for object detection accuracy, first stage has an additional loss function to reject pixels and only accepted pixels go to second stage (similar to rejection cascade in our PD). The standard proposal generation mechanisms like Faster-RCNN can be viewed as a special case with one rejection stage which generates proposals.

**Cascaded Classifiers:** Concatenation of several classifiers where output from one classifier is fed into the next one. Coarse-to-fine strategy: Reject some false positives at each stage. Efficient: reject east false candidates rapidly, avoid computational costs of the next stages. Accurate: Later classifiers: get more and more discriminative.

One can take advantage of intermediate layers to construct Cascaded Rejection Classifiers. Filter negative object proposals in a cascaded manner Classifiers applied and learnt on different convolutional layers in order to exploit features at different scales. Speeds up detection, maintain high accuracy. Consider CNN as a cascade of layers [40].

**Shared encoder multi-task CNN:** Multi-task learning (MTL) is a sub-field of machine learning in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately. Neural networks have capability to learn manifold representations from large amounts of high dimensional data. It is a well known fact that these representations are generic and transferable knowledge for learning related tasks in a much efficient way. Recent times Deep Learning based methods have shown a tremendous success in Multi Task Learning. In this report we outline significant and novel deep learning architectures related to Multi Task Learning.

We propose a network architecture with shared encoder which can be jointly learned. The main advantages are increased efficiency, scalability to add more tasks leveraging previous features and better generalization through inductive transfer (learning transferable features for tasks) [48].
<figure><img src="image_10.png" /><figcaption>FIGURE 1.9</figcaption></figure>

Cascaded CNN [reproduced figure].

**Computational efficiency:** The simple and easy to understand intuition behind shared features is improving computational efficiency. Say there are 2 classes and 2 independent networks taking 50% of processing power individually. If there is a possibility of say 30% sharing across the 2 networks, each network can re-use the additional 15% to do a slightly larger network individually. There is plenty of empirical evidence to show that the initial layers of the network are task-independent (oriented gabor filters) and we should be able to do some level of sharing, more the better.

**Generalization and accuracy:** Ignoring computational efficiency, the networks which are jointly learnt tend to generalize better and be more accurate. This is why transfer learning on imagenet is very popular where there network learns very complex classes like differentiating between specific species of dogs (link). Because the subtle differences between say two species say Labradaor or Pomeranian are learnt, they are better at detecting a simpler task of dog detection. Another argument is that there is less possibility of over-fitting for a particular task when they are jointly learnt.
<figure><img src="image_10.png" /><figcaption>FIGURE 1.9</figcaption></figure>

Cascaded CNN [reproduced figure].

**Computational efficiency:** The simple and easy to understand intuition behind shared features is improving computational efficiency. Say there are 2 classes and 2 independent networks taking 50% of processing power individually. If there is a possibility of say 30% sharing across the 2 networks, each network can re-use the additional 15% to do a slightly larger network individually. There is plenty of empirical evidence to show that the initial layers of the network are task-independent (oriented gabor filters) and we should be able to do some level of sharing, more the better.

**Generalization and accuracy:** Ignoring computational efficiency, the networks which are jointly learnt tend to generalize better and be more accurate. This is why transfer learning on imagenet is very popular where there network learns very complex classes like differentiating between specific species of dogs (link). Because the subtle differences between say two species say Labradaor or Pomeranian are learnt, they are better at detecting a simpler task of dog detection. Another argument is that there is less possibility of over-fitting for a particular task when they are jointly learnt.
<figure><img src="image_11.png" /><figcaption>FIGURE 1.10</figcaption></figure>

Shared encoder multi-task encoder

1.5 Towards a recipe for efficient design

1.5.1 Meta Learning

In the previous section we presented in detail many possible architecture de-
signs DL scientists and engineers should consider or navigate to find to very
efficient solution to the problem they want to model and solve with DNN.
In this section we will introduce the concept of meta-learning which can be
defined as tha automation of this process by implementing algorithms which
can select and optimize model architecture.

The goal is to have a network with a good generalization, so that it classies
unseen data correctly. In order to achieve this, important aspects are the net-
works architectures (how many layers and connections between them) and
hyperparameters (parameters that control the behavior of the intermediate
functional blocks). The choice of the architecture and hyperparameters is of-
ten done manually by experts using their domain knowledge; but sometimes
new architectures are discovered by chance. This process is obviously time-
consuming and tedious, and yet a lot of potential is wasted due to missing
tuning of neural networks. Automatic optimization algorithms such as evolu-
tionary algorithms or Bayesian optimization exist that can optimize almost
arbitrary objective functions. In the context of machine learning these meth-
ods are also known as AutoML or Metalearning. A special challenge in the
domain of Deep Learning is the fact that the evaluation of a new network
architecture can require days of training, even on modern computers with
graphic accelerators (GPU). This technical report summarizes the state of
the art methods in Metalearning with special respect to the demands in Deep
Learning.

**Hyperparameters Optimization** One very basic example to illustrate
<figure><img src="image_11.png" /><figcaption>FIGURE 1.10
Shared encoder multi-task encoder</figcaption></figure>

## 1.5 Towards a recipe for efficient design

### 1.5.1 Meta Learning

In the previous section we presented in detail many possible architecture designs DL scientists and engineers should consider or navigate to find to very efficient solution to the problem they want to model and solve with DNN. In this section we will introduce the concept of meta-learning which can be defined as the automation of this process by implementing algorithms which can select and optimize model architecture.

The goal is to have a network with a good generalization, so that it classies unseen data correctly. In order to achieve this, important aspects are the networks architectures (how many layers and connections between them) and hyperparameters (parameters that control the behavior of the intermediate functional blocks). The choice of the architecture and hyperparameters is often done manually by experts using their domain knowledge; but sometimes new architectures are discovered by chance. This process is obviously time-consuming and tedious, and yet a lot of potential is wasted due to missing tuning of neural networks. Automatic optimization algorithms such as evolutionary algorithms or Bayesian optimization exist that can optimize almost arbitrary objective functions. In the context of machine learning these methods are also known as AutoML or Metalearning. A special challenge in the domain of Deep Learning is the fact that the evaluation of a new network architecture can require days of training, even on modern computers with graphic accelerators (GPU). This technical report summarizes the state of the art methods in Metalearning with special respect to the demands in Deep Learning.

**Hyperparameters Optimization** One very basic example to illustrate
this approach is the field of hyperparameters optimization [4]. As you should
know hyperparameters choice is a key concept to master when you want to
succeed in learning your model. In practical the set of hyperparameters to set
can be quite large including global hyperparameters (optimizer, learning rate,
momentum, weight initialization...) and architecture design hyperparameters
(layer type, layer size, activation function, pooling type ...). Navigating the
hyperparameters space to find the best setting is sometimes a pain for DL
newcomers. To overcome this challenge many techniques have been presented
to automate this time consuming process. From grid search to random search
[3] or many variations have been studied to efficiently sample the hyperpa-
rameters space to find proper settings.

Faster Convolutional Architecture Search for Semantic Segmentation -
Also [11] (Audi) uses this approach with Q-Learning and a speeded-up version
of learning (bandit like, see below). They apply it for semantic segmentation
and achieve good results. Training time is 3 weeks on 1 GPU.

* Use google's Xception network or microsoft's resnet as the base CNN network as they give the best bang for the buck wrt computation. Avoid fatty networks like VGG16.

* This resnet based semantic segmentation which ranks #1 in cityscapes is a scalable architecture which can be parameterized for various depth and width (link).

* Design a larger network which is easier to learn and then reduce to smaller ones (mimicking networks idea). It is also good to have a gradation of network complexity vs accuracy graph.

* Re-target it to fit in the necessary computational resource based on GMAC budget available in the project. At the moment, I am manually calculating GMAC figures for various networks as a first order approximation. Good to find a tool, didn't find one so far.

* 8-bit quantization using Ristretto (open source). This offers a great advantage of memory bandwidth and should be the first step as memory is gonna be the bottleneck. Then try to perform SSL optimizations to exploit topological sparsity.

* Filter redundancy ideas could conflict with SSL and it is hard to determine upfront and need to be empirically experimented. This has less value as it has become a standard to use smaller filters like 3x3 and 1x1.

1.5.2 Design Exploration & Guidelines

Deep learning is a rapidly progressing area of research. Most of the research
is disparate in which the various ideas developed in different architectures
this approach is the field of hyperparameters optimization [4]. As you should
know hyperparameters choice is a key concept to master when you want to
succeed in learning your model. In practical the set of hyperparameters to set
can be quite large including global hyperparameters (optimizer, learning rate,
momentum, weight initialization...) and architecture design hyperparameters
(layer type, layer size, activation function, pooling type ...). Navigating the
hyperparameters space to find the best setting is sometimes a pain for DL
newcomers. To overcome this challenge many techniques have been presented
to automate this time consuming process. From grid search to random search
[3] or many variations have been studied to efficiently sample the hyperpa-
rameters space to find proper settings.

Faster Convolutional Architecture Search for Semantic Segmentation -
Also [11] (Audi) uses this approach with Q-Learning and a speeded-up version
of learning (bandit like, see below). They apply it for semantic segmentation
and achieve good results. Training time is 3 weeks on 1 GPU.

* Use google's Xception network or microsoft's resnet as the base CNN network as they give the best bang for the buck wrt computation. Avoid fatty networks like VGG16.

* This resnet based semantic segmentation which ranks #1 in cityscapes is a scalable architecture which can be parameterized for various depth and width (link).

* Design a larger network which is easier to learn and then reduce to smaller ones (mimicking networks idea). It is also good to have a gradation of network complexity vs accuracy graph.

* Re-target it to fit in the necessary computational resource based on GMAC budget available in the project. At the moment, I am manually calculating GMAC figures for various networks as a first order approximation. Good to find a tool, didn't find one so far.

* 8-bit quantization using Ristretto (open source). This offers a great advantage of memory bandwidth and should be the first step as memory is gonna be the bottleneck. Then try to perform SSL optimizations to exploit topological sparsity.

* Filter redundancy ideas could conflict with SSL and it is hard to determine upfront and need to be empirically experimented. This has less value as it has become a standard to use smaller filters like 3x3 and 1x1.

1.5.2 Design Exploration & Guidelines

Deep learning is a rapidly progressing area of research. Most of the research
is disparate in which the various ideas developed in different architectures
are not formalized because of the lack of theory. Hence it is hard to combine
ideas from two top networks from an application development perspective.
Additionally the main area of active research in deep learning is on image
recognition problems (as in ImageNet challenge) and the ideas trickle down
to semantic segmentation. Additionally efficiency is typically not a design
criteria in academic research as majority of leading networks are very large
comprising of hundreds of layers and employ ensemble of several networks. The
work in [5] compared various networks' accuracy normalized to the amount of
computation and shows ResNet and GoogleNet are efficient architectures. This
suffers from the same problem of treating the different networks independently
and hence does not formalize and combine ideas.

Some good design choices that are accepted with-in the community are
presented:

(1) The use of 3x3 convolutions similar to VGG architectures [57] turned to be useful experimentally. Especially in scenarios where you care about the resolution of your input such as segmentation. Since larger filter size will cause reduction in the image resolution.

(2) The dilated convolution is considered to be the best practice in segmentation as it increases the receptive field without downgrading resolution. Although in our comparative evaluation Dilation8 was not suitable for real-time applications. However that can be due to their use of a deep network to build upon.

(3) For real-time performance shallow networks can be useful for segmentation with a compromise in the accuracy.

(4) Batch normalization [28] turned to be a very useful trick for better con-
vergence during training in our experiments. This is due to the reduction of
change in distribution of network activations. They termed that as reduction
of internal covariate shift. The covariate shift occurs due to change of networks
parameters during training.

(5) The resolution of the input image largely affects the segmentation, al-
though it seems as a tiny detail. We found that higher input image resolution
can help with segmenting small objects like pedestrian. Also, using random
crops to help reduce the class imbalance can further help the segmentation.
This can be seen in Dilation8 results, they use random crops from the image
that is then upsampled as input.

(6) Skip connections is widely used in segmentation architectures such as FCN8s [42] and U-net [55]. However, the extensive use of these skip connections can lead to overhead in memory bandwidth.
are not formalized because of the lack of theory. Hence it is hard to combine
ideas from two top networks from an application development perspective.
Additionally the main area of active research in deep learning is on image
recognition problems (as in ImageNet challenge) and the ideas trickle down
to semantic segmentation. Additionally efficiency is typically not a design
criteria in academic research as majority of leading networks are very large
comprising of hundreds of layers and employ ensemble of several networks. The
work in [5] compared various networks' accuracy normalized to the amount of
computation and shows ResNet and GoogleNet are efficient architectures. This
suffers from the same problem of treating the different networks independently
and hence does not formalize and combine ideas.

Some good design choices that are accepted with-in the community are
presented:

(1) The use of 3x3 convolutions similar to VGG architectures [57] turned to be useful experimentally. Especially in scenarios where you care about the resolution of your input such as segmentation. Since larger filter size will cause reduction in the image resolution.

(2) The dilated convolution is considered to be the best practice in segmentation as it increases the receptive field without downgrading resolution. Although in our comparative evaluation Dilation8 was not suitable for real-time applications. However that can be due to their use of a deep network to build upon.

(3) For real-time performance shallow networks can be useful for segmentation with a compromise in the accuracy.

(4) Batch normalization [28] turned to be a very useful trick for better con-
vergence during training in our experiments. This is due to the reduction of
change in distribution of network activations. They termed that as reduction
of internal covariate shift. The covariate shift occurs due to change of networks
parameters during training.

(5) The resolution of the input image largely affects the segmentation, al-
though it seems as a tiny detail. We found that higher input image resolution
can help with segmenting small objects like pedestrian. Also, using random
crops to help reduce the class imbalance can further help the segmentation.
This can be seen in Dilation8 results, they use random crops from the image
that is then upsampled as input.

(6) Skip connections is widely used in segmentation architectures such as FCN8s [42] and U-net [55]. However, the extensive use of these skip connections can lead to overhead in memory bandwidth.
1.6 Conclusion

We provided an overview of various processors for implementing deep learning
algorithms. Then we de-constructed a wide array of CNN architecture into its
constituent building blocks and discussed various efficient design techniques
in a taxonomic way. Then we detail how to construct an efficient CNN for
the particular hardware. Finally, we perform a case study by benchmarking
semantic segmentation for autonomous driving.
1.6 Conclusion

We provided an overview of various processors for implementing deep learning
algorithms. Then we de-constructed a wide array of CNN architecture into its
constituent building blocks and discussed various efficient design techniques
in a taxonomic way. Then we detail how to construct an efficient CNN for
the particular hardware. Finally, we perform a case study by benchmarking
semantic segmentation for autonomous driving.
<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>
<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>
Bibliography

[1] Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Symmetry-invariant optimization in deep networks. *arXiv preprint* arXiv:1511.01754, 2015.

[2] Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Understanding symmetries in deep networks. *arXiv preprint* arXiv:1511.01029, 2015.

[3] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. *Journal of Machine Learning Research*, 13(Feb):281–305, 2012.

[4] James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In *Advances in Neural Information Processing Systems*, pages 2546–2554, 2011.

[5] Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. An analysis of deep neural network models for practical applications. *arXiv preprint* arXiv:1605.07678, 2016.

[6] Sumanth Chennupati, Venkatraman Narayanan, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Learning panoptic segmentation from instance contours. In *2021 IEEE International Conference on Robotics and Automation (ICRA)*, pages 9586–9593. IEEE, 2021.

[7] Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, pages 0–0, 2019.

[8] François Chollet. Xception: Deep learning with depthwise separable convolutions. *arXiv preprint* arXiv:1610.02357, 2016.

[9] Taco Cohen and Max Welling. Group equivariant convolutional networks. In *International Conference on Machine Learning*, pages 2990–2999, 2016.

[10] Matthieu Courbariaux, Itay Hubara, COM Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training neural networks with weights and activations constrained to+ 1 or–.
Bibliography

[1] Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Symmetry-invariant optimization in deep networks. *arXiv preprint arXiv:1511.01754*, 2015.

[2] Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Understanding symmetries in deep networks. *arXiv preprint arXiv:1511.01029*, 2015.

[3] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. *Journal of Machine Learning Research*, 13(Feb):281–305, 2012.

[4] James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In *Advances in Neural Information Processing Systems*, pages 2546–2554, 2011.

[5] Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. An analysis of deep neural network models for practical applications. *arXiv preprint arXiv:1605.07678*, 2016.

[6] Sumanth Chennupati, Venkatraman Narayanan, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Learning panoptic segmentation from instance contours. In *2021 IEEE International Conference on Robotics and Automation (ICRA)*, pages 9586–9593. IEEE, 2021.

[7] Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, pages 0–0, 2019.

[8] François Chollet. Xception: Deep learning with depthwise separable convolutions. *arXiv preprint arXiv:1610.02357*, 2016.

[9] Taco Cohen and Max Welling. Group equivariant convolutional networks. In *International Conference on Machine Learning*, pages 2990–2999, 2016.

[10] Matthieu Courbariaux, Itay Hubara, COM Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training neural networks with weights and activations constrained to+ 1 or–.
[11] Rupesh Durgesh12, Gerald Krell, Andrii Iegorov, Peter Schuberth, and Felix Friedmann. Faster convolutional architecture search for semantic segmentation.

[12] Ciarán Eising, Jonathan Horgan, and Senthil Yogamani. Near-field perception for low-speed vehicle automation using surround-view fish-eye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(9):13976-13993, 2021.

[13] Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. arXiv preprint arXiv:1302.4389, 2013.

[14] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: efficient inference engine on compressed deep neural network. In *Proceedings of the 43rd International Symposium on Computer Architecture*, pages 243–254. IEEE Press, 2016.

[15] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.

[16] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In *Advances in Neural Information Processing Systems*, pages 1135–1143, 2015.

[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770–778, 2016.

[18] Jonathan Horgan, Ciarán Hughes, John McDonald, and Senthil Yogamani. Vision-based driver assistance systems: Survey, taxonomy and advances. In *2015 IEEE 18th International Conference on Intelligent Transportation Systems*, pages 2032–2039. IEEE, 2015.

[19] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.

[20] Gao Huang, Shichen Liu, Laurens van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. arXiv preprint arXiv:1711.09224, 2017.

[21] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016.
[11] Rupesh Durgesh12, Gerald Krell, Andrii Iegorov, Peter Schuberth, and Felix Friedmann. Faster convolutional architecture search for semantic segmentation.

[12] Ciarán Eising, Jonathan Horgan, and Senthil Yogamani. Near-field perception for low-speed vehicle automation using surround-view fish-eye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(9):13976-13993, 2021.

[13] Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. arXiv preprint arXiv:1302.4389, 2013.

[14] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: efficient inference engine on compressed deep neural network. In *Proceedings of the 43rd International Symposium on Computer Architecture*, pages 243–254. IEEE Press, 2016.

[15] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.

[16] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In *Advances in Neural Information Processing Systems*, pages 1135–1143, 2015.

[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770–778, 2016.

[18] Jonathan Horgan, Ciarán Hughes, John McDonald, and Senthil Yogamani. Vision-based driver assistance systems: Survey, taxonomy and advances. In *2015 IEEE 18th International Conference on Intelligent Transportation Systems*, pages 2032–2039. IEEE, 2015.

[19] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.

[20] Gao Huang, Shichen Liu, Laurens van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. arXiv preprint arXiv:1711.09224, 2017.

[21] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016.
[22] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. *arXiv preprint arXiv:1611.10012*, 2016.

[23] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. *arXiv preprint arXiv:1609.07061*, 2016.

[24] Ciarán Hughes, Sunil Chandra, Ganesh Sistu, Jonathan Horgan, Brian Deegan, Sumanth Chennupati, and Senthil Yogamani. Drivespace: towards context-aware drivable area detection. *Electronic Imaging*, 31:1–9, 2019.

[25] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 0.5 mb model size. *arXiv preprint arXiv:1602.07360*, 2016.

[26] Yani Ioannou, Duncan Robertson, Roberto Cipolla, and Antonio Criminisi. Deep roots: Improving cnn efficiency with hierarchical filter groups. *arXiv preprint arXiv:1605.06489*, 2016.

[27] Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. *arXiv preprint arXiv:1511.06744*, 2015.

[28] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. *arXiv preprint arXiv:1502.03167*, 2015.

[29] Jonghoon Jin, Aysegul Dundar, and Eugenio Culurciello. Flattened convolutional neural networks for feedforward acceleration. *arXiv preprint arXiv:1412.5474*, 2014.

[30] Kye-Hyeon Kim, Sanghoon Hong, Byungseok Roh, Yeongjae Cheon, and Minje Park. Pvanet: Deep but lightweight neural networks for real-time object detection. *arXiv preprint arXiv:1608.08021*, 2016.

[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In *Advances in neural information processing systems*, pages 1097–1105, 2012.

[32] Varun Ravi Kumar, Marvin Klingner, Senthil Yogamani, Markus Bach, Stefan Milz, Tim Fingscheidt, and Patrick Mäder. Svdistnet: Self-supervised near-field distance estimation on surround view fisheye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(8):10252–10261, 2021.
[22] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. *arXiv preprint arXiv:1611.10012*, 2016.

[23] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. *arXiv preprint arXiv:1609.07061*, 2016.

[24] Ciarán Hughes, Sunil Chandra, Ganesh Sistu, Jonathan Horgan, Brian Deegan, Sumanth Chennupati, and Senthil Yogamani. Drivespace: towards context-aware drivable area detection. *Electronic Imaging*, 31:1–9, 2019.

[25] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 0.5 mb model size. *arXiv preprint arXiv:1602.07360*, 2016.

[26] Yani Ioannou, Duncan Robertson, Roberto Cipolla, and Antonio Criminisi. Deep roots: Improving cnn efficiency with hierarchical filter groups. *arXiv preprint arXiv:1605.06489*, 2016.

[27] Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. *arXiv preprint arXiv:1511.06744*, 2015.

[28] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. *arXiv preprint arXiv:1502.03167*, 2015.

[29] Jonghoon Jin, Aysegul Dundar, and Eugenio Culurciello. Flattened convolutional neural networks for feedforward acceleration. *arXiv preprint arXiv:1412.5474*, 2014.

[30] Kye-Hyeon Kim, Sanghoon Hong, Byungseok Roh, Yeongjae Cheon, and Minje Park. Pvanet: Deep but lightweight neural networks for real-time object detection. *arXiv preprint arXiv:1608.08021*, 2016.

[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In *Advances in neural information processing systems*, pages 1097–1105, 2012.

[32] Varun Ravi Kumar, Marvin Klingner, Senthil Yogamani, Markus Bach, Stefan Milz, Tim Fingscheidt, and Patrick Mäder. Svdistnet: Self-supervised near-field distance estimation on surround view fisheye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(8):10252–10261, 2021.
[33] Varun Ravi Kumar, Stefan Milz, Christian Witt, Martin Simon, et al. Near-field depth estimation using monocular fisheye camera: A semi-supervised learning approach using sparse lidar data. In *CVPR Workshop*, volume 7, page 2, 2018.

[34] Varun Ravi Kumar, Senthil Yogamani, Hazem Rashed, Ganesh Sitsu, et al. Omnidet: Surround view cameras based multi-task visual perception network for autonomous driving. *IEEE Robotics and Automation Letters*, 6(2):2830–2837, 2021.

[35] Liangzhen Lai, Naveen Suda, and Vikas Chandra. Not all ops are created equal! *arXiv preprint arXiv:1801.04326*, 2018.

[36] Andrew Lavin and Scott Gray. Fast algorithms for convolutional neural networks. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 4013–4021, 2016.

[37] Isabelle Leang, Ganesh Sistu, Fabian Bürger, Andrei Bursuc, et al. Dynamic task weighting methods for multi-task networks in autonomous driving systems. In *2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)*, pages 1–8. IEEE, 2020.

[38] Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. *arXiv preprint arXiv:1605.04711*, 2016.

[39] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. *arXiv preprint arXiv:1608.08710*, 2016.

[40] Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, and Xiaoou Tang. Not all pixels are equal: Difficulty-aware semantic segmentation via deep layer cascade. *arXiv preprint arXiv:1704.01344*, 2017.

[41] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. *arXiv preprint arXiv:1312.4400*, 2013.

[42] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 3431–3440, 2015.

[43] Dipan Kumar Mandal, Jagadeesh Sankaran, Akshay Gupta, Kyle Castille, Shraddha Gondkar, Sanmati Kamath, Pooja Sundar, and Alan Phipps. An embedded vision engine (eve) for automotive vision processing. In *Circuits and Systems (ISCAS), 2014 IEEE International Symposium on*, pages 49–52. IEEE, 2014.

[44] Bradley McDanel, Surat Teerapittayanon, and HT Kung. Embedded binarized neural networks. *arXiv preprint arXiv:1709.02260*, 2017.
[33] Varun Ravi Kumar, Stefan Milz, Christian Witt, Martin Simon, et al. Near-field depth estimation using monocular fisheye camera: A semi-supervised learning approach using sparse lidar data. In *CVPR Workshop*, volume 7, page 2, 2018.

[34] Varun Ravi Kumar, Senthil Yogamani, Hazem Rashed, Ganesh Sitsu, et al. Omnidet: Surround view cameras based multi-task visual perception network for autonomous driving. *IEEE Robotics and Automation Letters*, 6(2):2830–2837, 2021.

[35] Liangzhen Lai, Naveen Suda, and Vikas Chandra. Not all ops are created equal! *arXiv preprint arXiv:1801.04326*, 2018.

[36] Andrew Lavin and Scott Gray. Fast algorithms for convolutional neural networks. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 4013–4021, 2016.

[37] Isabelle Leang, Ganesh Sistu, Fabian Bürger, Andrei Bursuc, et al. Dynamic task weighting methods for multi-task networks in autonomous driving systems. In *2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)*, pages 1–8. IEEE, 2020.

[38] Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. *arXiv preprint arXiv:1605.04711*, 2016.

[39] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. *arXiv preprint arXiv:1608.08710*, 2016.

[40] Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, and Xiaoou Tang. Not all pixels are equal: Difficulty-aware semantic segmentation via deep layer cascade. *arXiv preprint arXiv:1704.01344*, 2017.

[41] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. *arXiv preprint arXiv:1312.4400*, 2013.

[42] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 3431–3440, 2015.

[43] Dipan Kumar Mandal, Jagadeesh Sankaran, Akshay Gupta, Kyle Castille, Shraddha Gondkar, Sanmati Kamath, Pooja Sundar, and Alan Phipps. An embedded vision engine (eve) for automotive vision processing. In *Circuits and Systems (ISCAS), 2014 IEEE International Symposium on*, pages 49–52. IEEE, 2014.

[44] Bradley McDanel, Surat Teerapittayanon, and HT Kung. Embedded binarized neural networks. *arXiv preprint arXiv:1709.02260*, 2017.
[45] Dmytro Mishkin, Nikolay Sergievskiy, and Jiri Matas. Systematic evaluation of convolution neural network advances on the imagenet. *Computer Vision and Image Understanding*, 2017.

[46] Eslam Mohamed, Mahmoud Ewaisha, Mennatullah Siam, Hazem Rashed, Senthil Yogamani, Waleed Hamdy, Mohamed El-Dakdouky, and Ahmad El-Sallab. Monocular instance motion segmentation for autonomous driving: Kitti instancemotseg dataset and multi-task baseline. In *2021 IEEE Intelligent Vehicles Symposium (IV)*, pages 114–121. IEEE, 2021.

[47] Sambit Mohapatra, Senthil Yogamani, Heinrich Gotzig, Stefan Milz, and Patrick Mader. Bevdnet: bird's eye view lidar point cloud based real-time 3d object detection for autonomous driving. In *2021 IEEE International Intelligent Transportation Systems Conference (ITSC)*, pages 2809–2815. IEEE, 2021.

[48] Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Fast scene understanding for autonomous driving. *arXiv preprint arXiv:1708.02550*, 2017.

[49] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. *arXiv preprint arXiv:1606.02147*, 2016.

[50] Maximilian Pöpperli, Raghavendra Gulagundi, Senthil Yogamani, and Stefan Milz. Capsule neural network based height classification using low-cost automotive ultrasonic sensors. In *2019 IEEE Intelligent Vehicles Symposium (IV)*, pages 661–666. IEEE, 2019.

[51] Hazem Rashed, Ahmad El Sallab, Senthil Yogamani, and Mohamed ElHelw. Motion and depth augmented semantic segmentation for autonomous navigation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, pages 0–0, 2019.

[52] Hazem Rashed, Eslam Mohamed, Ganesh Sistu, Varun Ravi Kumar, Ciarán Eising, Ahmad El-Sallab, and SK Yogamani. Fisheyeyolo: Object detection on fisheye cameras for autonomous driving. In *Proceedings of the Machine Learning for Autonomous Driving NeurIPS 2020 Virtual Workshop, Virtual*, volume 11, 2020.

[53] Hazem Rashed, Senthil Yogamani, Ahmad El-Sallab, Pavel Krizek, and Mohamed El-Helw. Optical flow augmented semantic segmentation networks for automated driving. In *Proceedings of the International Conference on Computer Vision Theory and Applications*, 2019.

[54] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In *European Conference on Computer Vision*, pages 525–542. Springer, 2016.
[45] Dmytro Mishkin, Nikolay Sergievskiy, and Jiri Matas. Systematic evaluation of convolution neural network advances on the imagenet. *Computer Vision and Image Understanding*, 2017.

[46] Eslam Mohamed, Mahmoud Ewaisha, Mennatullah Siam, Hazem Rashed, Senthil Yogamani, Waleed Hamdy, Mohamed El-Dakdouky, and Ahmad El-Sallab. Monocular instance motion segmentation for autonomous driving: Kitti instancemotseg dataset and multi-task baseline. In *2021 IEEE Intelligent Vehicles Symposium (IV)*, pages 114–121. IEEE, 2021.

[47] Sambit Mohapatra, Senthil Yogamani, Heinrich Gotzig, Stefan Milz, and Patrick Mader. Bevdnet: bird's eye view lidar point cloud based real-time 3d object detection for autonomous driving. In *2021 IEEE International Intelligent Transportation Systems Conference (ITSC)*, pages 2809–2815. IEEE, 2021.

[48] Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Fast scene understanding for autonomous driving. *arXiv preprint arXiv:1708.02550*, 2017.

[49] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. *arXiv preprint arXiv:1606.02147*, 2016.

[50] Maximilian Pöpperli, Raghavendra Gulagundi, Senthil Yogamani, and Stefan Milz. Capsule neural network based height classification using low-cost automotive ultrasonic sensors. In *2019 IEEE Intelligent Vehicles Symposium (IV)*, pages 661–666. IEEE, 2019.

[51] Hazem Rashed, Ahmad El Sallab, Senthil Yogamani, and Mohamed ElHelw. Motion and depth augmented semantic segmentation for autonomous navigation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, pages 0–0, 2019.

[52] Hazem Rashed, Eslam Mohamed, Ganesh Sistu, Varun Ravi Kumar, Ciarán Eising, Ahmad El-Sallab, and SK Yogamani. Fisheyeyolo: Object detection on fisheye cameras for autonomous driving. In *Proceedings of the Machine Learning for Autonomous Driving NeurIPS 2020 Virtual Workshop, Virtual*, volume 11, 2020.

[53] Hazem Rashed, Senthil Yogamani, Ahmad El-Sallab, Pavel Krizek, and Mohamed El-Helw. Optical flow augmented semantic segmentation networks for automated driving. In *Proceedings of the International Conference on Computer Vision Theory and Applications*, 2019.

[54] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In *European Conference on Computer Vision*, pages 525–542. Springer, 2016.
[55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In *International Conference on Medical Image Computing and Computer-Assisted Intervention*, pages 234–241. Springer, 2015.

[56] Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In *International Conference on Machine Learning*, pages 2217–2225, 2016.

[57] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. *arXiv preprint arXiv:1409.1556*, 2014.

[58] Amos Sironi, Bugra Tekin, Roberto Rigamonti, Vincent Lepetit, and Pas-cal Fua. Learning separable filters. *IEEE Transactions on Pattern Anal-ysis and Machine Intelligence*, 37(1):94–106, 2015.

[59] Ganesh Sistu, Isabelle Leang, and Senthil Yogamani. Real-time joint object detection and semantic segmentation network for automated driving. *arXiv preprint arXiv:1901.03912*, 2019.

[60] Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer. Efficient processing of deep neural networks: A tutorial and survey. *arXiv preprint arXiv:1703.09039*, 2017.

[61] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 1–9, 2015.

[62] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 2818–2826, 2016.

[63] Nivedita Tripathi and Senthil Yogamani. Trained trajectory based automated parking system using Visual SLAM. In *Proceedings of the Computer Vision and Pattern Recognition Conference Workshops*, 2021.

[64] Michal Uricár, David Hurych, Pavel Krizek, et al. Challenges in designing datasets and validation for autonomous driving. In *Proceedings of the International Conference on Computer Vision Theory and Applications*, 2019.

[65] Michal Uřičář, Pavel Křížek, Ganesh Sistu, and Senthil Yogamani. Soil-ingnet: Soiling detection on automotive surround-view cameras. In *2019 IEEE Intelligent Transportation Systems Conference (ITSC)*, pages 67–72. IEEE, 2019.
[55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In *International Conference on Medical Image Computing and Computer-Assisted Intervention*, pages 234–241. Springer, 2015.

[56] Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In *International Conference on Machine Learning*, pages 2217–2225, 2016.

[57] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. *arXiv preprint arXiv:1409.1556*, 2014.

[58] Amos Sironi, Bugra Tekin, Roberto Rigamonti, Vincent Lepetit, and Pas-cal Fua. Learning separable filters. *IEEE Transactions on Pattern Anal-ysis and Machine Intelligence*, 37(1):94–106, 2015.

[59] Ganesh Sistu, Isabelle Leang, and Senthil Yogamani. Real-time joint object detection and semantic segmentation network for automated driving. *arXiv preprint arXiv:1901.03912*, 2019.

[60] Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer. Efficient processing of deep neural networks: A tutorial and survey. *arXiv preprint arXiv:1703.09039*, 2017.

[61] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 1–9, 2015.

[62] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 2818–2826, 2016.

[63] Nivedita Tripathi and Senthil Yogamani. Trained trajectory based automated parking system using Visual SLAM. In *Proceedings of the Computer Vision and Pattern Recognition Conference Workshops*, 2021.

[64] Michal Uricár, David Hurych, Pavel Krizek, et al. Challenges in designing datasets and validation for autonomous driving. In *Proceedings of the International Conference on Computer Vision Theory and Applications*, 2019.

[65] Michal Uřičář, Pavel Křížek, Ganesh Sistu, and Senthil Yogamani. Soil-ingnet: Soiling detection on automotive surround-view cameras. In *2019 IEEE Intelligent Transportation Systems Conference (ITSC)*, pages 67–72. IEEE, 2019.
[66] Michal Uricár, Jan Ulicny, Ganesh Sistu, Hazem Rashed, et al. Desoiling dataset: Restoring soiled areas on automotive fisheye cameras. In *Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops*, pages 0–0, 2019.

[67] Paul Viola and Michael Jones. Rapid object detection using a boosted cascade of simple features. In *Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on*, volume 1, pages I–I. IEEE, 2001.

[68] Min Wang, Baoyuan Liu, and Hassan Foroosh. Design of efficient convolutional layers using single intra-channel convolution, topological subdivision and spatial bottleneck structure.

[69] Min Wang, Baoyuan Liu, and Hassan Foroosh. Factorized convolutional neural networks. *arXiv preprint arXiv:1608.04337*, 2016.

[70] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In *Advances in Neural Information Processing Systems*, pages 2074–2082, 2016.

[71] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 5987–5995. IEEE, 2017.

[72] Lucie Yahiaoui, Jonathan Horgan, Brian Deegan, Senthil Yogamani, Ciarán Hughes, and Patrick Denny. Overview and empirical analysis of isp parameter tuning for visual perception in autonomous driving. *Journal of Imaging*, 5(10):78, 2019.

[73] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. *arXiv preprint arXiv:1511.07122*, 2015.

[74] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In *European conference on computer vision*, pages 818–833. Springer, 2014.

[75] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. *arXiv preprint arXiv:1707.01083*, 2017.
[66] Michal Uricár, Jan Ulicny, Ganesh Sistu, Hazem Rashed, et al. Desoiling dataset: Restoring soiled areas on automotive fisheye cameras. In *Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops*, pages 0–0, 2019.

[67] Paul Viola and Michael Jones. Rapid object detection using a boosted cascade of simple features. In *Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on*, volume 1, pages I–I. IEEE, 2001.

[68] Min Wang, Baoyuan Liu, and Hassan Foroosh. Design of efficient convolutional layers using single intra-channel convolution, topological subdivision and spatial bottleneck structure.

[69] Min Wang, Baoyuan Liu, and Hassan Foroosh. Factorized convolutional neural networks. *arXiv preprint arXiv:1608.04337*, 2016.

[70] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In *Advances in Neural Information Processing Systems*, pages 2074–2082, 2016.

[71] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 5987–5995. IEEE, 2017.

[72] Lucie Yahiaoui, Jonathan Horgan, Brian Deegan, Senthil Yogamani, Ciarán Hughes, and Patrick Denny. Overview and empirical analysis of isp parameter tuning for visual perception in autonomous driving. *Journal of Imaging*, 5(10):78, 2019.

[73] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. *arXiv preprint arXiv:1511.07122*, 2015.

[74] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In *European conference on computer vision*, pages 818–833. Springer, 2014.

[75] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. *arXiv preprint arXiv:1707.01083*, 2017.