Please fill in the "Bandwidth-Aware Quantization Control for Priority-Based Task Management in Heterogeneous Edge Computing Systems; In heterogeneous edge computing environments comprising a main core and a Neural Processing Unit (NPU) that share memory resources, effective management of memory bandwidth is crucial for optimal system performance. This thesis proposes a dynamic quantization strategy that adapts quantization levels based on real-time memory bandwidth conditions and application priorities. When the main core executes high-priority tasks, the system prioritizes its memory bandwidth usage by reducing the quantization bits of the NPU tasks, thereby mitigating memory congestion. Conversely, when NPU tasks are prioritized, the main core adjusts accordingly to balance the memory bandwidth allocation. The proposed framework ensures efficient utilization of shared memory resources, enhancing overall system performance while maintaining the necessary accuracy for critical applications. Experimental results demonstrate that the adaptive quantization approach effectively manages memory bandwidth, achieving significant performance improvements in heterogeneous edge computing scenarios." section by referring to the format below.



Chapter 1. INTRODUCTION

1.1 Overheads of Inference of Convolutional Neural Networks
The rapid adoption of the Convolutional Neural Network (CNN) has
drastically increased the demand for high computational capabilities. Since the
training and inference processes of the CNN models require the massive
transfer of weight parameters and arithmetic operations on the parameters,
computing for CNN applications shows characteristics of both massive
computational complexity and memory consumption. For example, CNN
inference on image data with a dimension of 3 x 224 x 224 using a relatively
simple AlexNet model results in a computational complexity of 0.729 GFLOPs
and a memory consumption of 0.82GB. The same task with a relatively complex
VGG-16 model results in the computational complexity of 16 GFLOPs and the
memory consumption of 1.46GB [1] [2] [3]. Mitigating the overheads of the
CNN inference tasks has been a crucial goal, especially in low-cost mobile or
embedded platforms with limited computing resources.

1.2 Weight Quantization for Model Compression
Weight quantization has been one of the most widely adopted model
compression techniques for relieving both overheads of CNN computation.
Weight quantization expresses weight parameters using data formats with lower
precision. For example, the most popular INT8 quantization scheme transforms
the weight parameters originally stored in the 32-bit single-precision floating
point data format (FP32) into the 8-bit integer data format (INT8). Weight
quantization reduces the memory space taken up by the weight proportionally
to the ratio of the bit-width while replacing complex floating-point operations
with simpler integer operations to mitigate the computational complexity of the
CNN inference tasks. One of the hardware-friendly quantization schemes,
called Power-of-Two (POT) quantization, further reduce the computational
complexity and improve the performance of the CNN inference tasks. The POT
quantization scheme quantizes the weights of filters into power-of-two values,
which significantly reduces the computational complexity by replacing the
multiplication operation with the bit-shift operation [4] [5] [6].

1.3 Processing-in-Memory Technique
For the CNN Inference workloads' heavily memory-bound aspects,
Processing-in-Memory (PIM) technique which eliminates data transfer
between the Core Side and the Memory Side has been regarded as a notable
hardware solution for the overheads of CNN Inference. Virtually all types of
memories including DRAM, SRAM, PCM, MRAM, RRAM and even those
implemented as caches in the processors have been utilized as base for PIM
technique.

1.4 Limitations of PIM and POT Quantization Techniques
Yet, both PIM and POT Quantization techniques have shortcomings to be
solved. Most PIM systems usually come with significant amount of area
overhead due to additional hardware such as buffers and ALUs for full support
of compute instructions.
Besides, POT Quantization brings about significant degree of accuracy
drop in CNN inference unless retrained for dozens of epochs, making it unfit for
accuracy sensitive use cases such as medical image classification and security
systems.

1.5 Research Goal
In this paper, we leverage the POT quantization to implement a novel PEC
architecture that utilizes shifters for performing multiplication operations at a
low cost. We also introduce a filter-wise POT quantization scheme that
quantizes the weight parameters to the Power-of-Two data format in filterwise
manner to better fit the random weight distribution of the convolution
layers. The filter-wise quantization scheme achieves better accuracy than the
naive model-wise Power-of-Two quantization scheme by quantizing only the
parts of the filters in the neural network.
We also study the impact of re-training on the inference accuracy of the
POT quantized CNN models to determine the sweet spot on the trade-off
between quantization degree and the accuracy.
Besides, we design the framework named Cooperative Acceleration of
CNN Inference in Multicore Processors (i.e., The Cooperative Acceleration).

1.6 Outline of the Paper
This paper is structures as follows: In Section II, we review the backgrounds,
limitations, and motivations for our new design in CNN Quantization, Powerof-
Two Quantization, Processing-in-Memory Techniques. In the following
Section III, we describe the key techniques of our acceleration scheme: Basic
In-Cache Processing Architecture, The Filter-Wise Quantization Scheme for
PIM, and the Cooperative Acceleration of CNN Inference. In the next Section
IV, we put detail on Data Placement Schemes for CNN parameters for improved
performance and hardware utilization. We describe the data placement schemes
in both inter-LLC-slice and intra-LLC-slice granularities. In Section V, we
design the detailed microarchitecture of the cooperative acceleration throughout
the whole cache hierarchy. In the following Section VI, we simulate the
Inferences of three CNN models under consideration on our cooperative
acceleration system, and compare its latency, energy consumption and area
consumption. Finally, in Section VII, we summarize the study and verify the
benefits brought on by the Cooperative Acceleration.
The contributions of this work are:
• We show that each convolutional layer in the CNN models show different
sensitivity towards the Power-of-Two quantization.
• We show that Power-of-Two quantization of only the parts of the convolution
filters can quantize the significant amount of weight parameters with small
accuracy drop.
• We show that the L3 cache of off-the shelf CPU can be transformed into a
massively parallel processing unit.
• We show that the filter-wise quantization scheme can improve throughput of PIM-based CNN accelerators
• We show that the Cooperative Acceleration of CNN Inference with between
the Host Core and the PIM-Enabled Cache benefits the system in latency,
throughput, thermal and energy overhead aspects.

A. Abbreviations and Acronyms

 FX - Fixed Point
 POT - Power of Two
 CNN - Convolutional Neural Network
 PEC - PIM Enabled Cache
 STD - Standard Deviation
 SPU - Slice Processing Unit
 SA – Sense Amplifier.