Please write Chapter 3. Motivation based on the content of the document below.

Abstract
Quantization for Memory Augmentation on
Edge Devices

In heterogeneous edge computing environments, where a main core and a Neural Processing Unit (NPU) share memory resources, effective memory bandwidth management is critical for achieving optimal system performance. This thesis presents a dynamic quantization strategy that leverages Post-Training Quantization (PTQ) to adaptively adjust the bit-width of NPU weights in real-time based on current memory bandwidth conditions and application priorities. During the model loading phase, the NPU employs a Look-Up Table (LUT) command to quantize weights to 8-bit, 6-bit, or 4-bit (or lower) representations and stores them in memory using a segment table for immediate access. This approach allows pre-stored quantized weights to be selected and executed on-the-fly, enabling the system to dynamically modify the quantization bit-width as needed. When the main core executes high-priority tasks, the system prioritizes its memory bandwidth by reducing the NPU’s quantization bits, thereby alleviating memory congestion. Conversely, when NPU tasks take precedence, the memory bandwidth allocation is adjusted to favor the NPU, ensuring balanced performance across components. The proposed framework facilitates efficient utilization of shared memory resources, enhancing overall system performance while maintaining the necessary accuracy for critical applications.

Keywords: Dynamic Quantization, Post-Training Quantization, Neural Processing Unit, Memory Bandwidth Management, Edge Computing 
Chapter 1. INTRODUCTION

1.1 Challenges in Heterogeneous Edge Computing Systems

A. 연구방법론
   
The proliferation of Internet of Things (IoT) devices and the increasing demand for real-time data processing have propelled the adoption of edge computing. Heterogeneous edge computing systems, which integrate diverse processing units such as general-purpose CPUs and specialized Neural Processing Units (NPUs), offer enhanced performance and energy efficiency by catering to varied computational tasks. However, this heterogeneity introduces significant challenges in resource management, particularly in balancing computational workloads and memory bandwidth utilization. The shared memory architecture between the main core and the NPU often becomes a bottleneck, leading to contention and reduced system performance. Effective management of memory bandwidth is critical to ensure seamless task execution, especially when handling priority-based applications that demand different levels of computational resources and memory access patterns.
 
1.2 Shared Memory Resource Management

In heterogeneous edge computing environments, the main core and the NPU typically share a common memory pool. This shared architecture necessitates sophisticated memory management strategies to prevent contention and ensure that high-priority tasks receive the necessary bandwidth without being impeded by lower-priority operations. Traditional memory management techniques focus on static allocation or simple dynamic allocation based on fixed priorities. However, these approaches often fail to adapt to the dynamic nature of real-time applications, leading to suboptimal performance and inefficient resource utilization. Dynamic memory bandwidth management that can respond to instantaneous system conditions and task priorities is essential for maintaining high throughput and low latency in edge computing systems.
 
1.3 Quantization Techniques in Edge Computing

Quantization is a widely adopted technique in deep learning model optimization, especially for deployment on resource-constrained edge devices. By reducing the precision of neural network weights and activations, quantization decreases memory footprint and computational complexity, enabling faster inference and lower energy consumption. Fixed-point quantization schemes, such as INT8, are particularly favored for their compatibility with existing hardware accelerators. However, the choice of quantization levels significantly impacts both the accuracy of the neural network and the efficiency of memory bandwidth utilization. Adaptive quantization strategies that can dynamically adjust quantization levels based on real-time system conditions and task priorities present a promising avenue for enhancing the performance of heterogeneous edge computing systems.
 
1.4 Limitations of Current Approaches

Despite the advancements in quantization and memory management techniques, several limitations persist in current heterogeneous edge computing systems:
Static Quantization Levels: Most existing quantization methods employ fixed quantization levels that do not adapt to varying memory bandwidth conditions or task priorities, leading to inefficiencies in resource utilization.
Lack of Priority-Based Management: Traditional memory management approaches often do not account for the varying priorities of tasks, resulting in scenarios where high-priority tasks may experience delays due to contention with lower-priority operations.
Memory Congestion: Shared memory architectures can suffer from congestion, especially when multiple processing units simultaneously access memory with high bandwidth demands, degrading overall system performance.
Accuracy Trade-offs: Aggressive quantization can lead to significant accuracy losses in neural network models, particularly in scenarios requiring high precision, such as medical image analysis or security applications.
Addressing these limitations is crucial for developing robust and efficient heterogeneous edge computing systems capable of handling diverse and demanding real-time applications.
 
1.5 Research Goal

This thesis aims to address the challenges of memory bandwidth management and task prioritization in heterogeneous edge computing systems through the development of a dynamic quantization strategy. The primary objectives are as follows:
Dynamic Quantization Control: Develop an adaptive quantization mechanism that adjusts the quantization levels of NPU tasks in real-time based on current memory bandwidth conditions and the priorities of executing tasks.
Priority-Based Task Management: Implement a priority-based scheduler that allocates memory bandwidth preferentially to high-priority tasks executed by the main core, while dynamically reducing the quantization bits for lower-priority NPU tasks to alleviate memory congestion.
Efficient Resource Utilization: Ensure optimal utilization of shared memory resources by balancing memory bandwidth allocation between the main core and the NPU, thereby enhancing overall system performance without compromising the accuracy required for critical applications.
Performance Evaluation: Conduct comprehensive experiments to evaluate the effectiveness of the proposed adaptive quantization approach in managing memory bandwidth and improving system performance in various heterogeneous edge computing scenarios.
By achieving these objectives, the proposed framework seeks to enhance the performance and reliability of heterogeneous edge computing systems, making them more suitable for deployment in environments with stringent real-time processing and accuracy requirements.
 
1.6 Outline of the Thesis

This thesis is structured as follows:
Chapter 2: Background and Related Work
This chapter reviews the existing literature on heterogeneous edge computing systems, memory bandwidth management, and quantization techniques. It discusses the limitations of current approaches and identifies gaps that the proposed research aims to address.
Chapter 3: System Architecture and Design
This chapter presents the overall architecture of the proposed bandwidth-aware quantization control framework. It details the dynamic quantization strategy, priority-based task management mechanisms, and the integration between the main core and the NPU.
Chapter 4: Dynamic Quantization Mechanism
This chapter delves into the specifics of the adaptive quantization strategy, including the algorithms used to adjust quantization levels based on real-time memory bandwidth conditions and task priorities.
Chapter 5: Priority-Based Task Scheduling
This chapter describes the design and implementation of the priority-based scheduler that dynamically allocates memory bandwidth to tasks based on their priority levels.
Chapter 6: Experimental Setup and Evaluation
This chapter outlines the experimental setup, including the hardware and software environments used for testing. It presents the methodology for evaluating the performance of the proposed framework and discusses the results obtained from various benchmark scenarios.
Chapter 7: Discussion and Future Work
This chapter interprets the experimental results, highlighting the strengths and potential areas for improvement in the proposed approach. It also suggests directions for future research to further enhance memory bandwidth management and task prioritization in heterogeneous edge computing systems.
Chapter 8: Conclusion
This chapter provides a summary of the research contributions, reiterates the significance of the findings, and concludes the thesis.
 
Chapter 2. Background

1.1 Model Quantization

Model quantization is a technique used to reduce the computational and memory requirements of neural networks by representing the model parameters and activations with lower-precision numerical values. Instead of using the standard 32-bit floating-point format, quantization maps these values to a limited set of levels, often using 8-bit integers or even fewer bits. This process results in a smaller model size and can lead to faster inference times, which is particularly beneficial for deploying deep neural networks on resource-constrained devices such as mobile phones and embedded systems.

Quantization leverages the redundancy in neural networks, where many parameters can tolerate reduced precision without significantly impacting the model's performance. By constraining the weights and activations to a smaller set of possible values, quantization reduces the amount of data that needs to be stored and processed.
There are two primary types of quantization methods:
Uniform Quantization: Uses evenly spaced quantization levels across the range of values.
Non-Uniform Quantization: Employs quantization levels that are unevenly spaced, often tailored to the distribution of the data.
   

A.1 Uniform Quantization
   
Uniform quantization divides the range of possible values into equally spaced intervals and maps each value to the nearest quantization level within these intervals. This approach simplifies the quantization and dequantization processes, as the quantization levels are regularly spaced, allowing for efficient computation.

Uniform quantization involves three main steps:
Scaling (Normalization): The first step is to adjust the original data values so that they fit within a standard range, typically between zero and one. This is achieved by shifting all the values so that the smallest value becomes zero and then scaling them so that the largest value becomes one. All other values are proportionally adjusted to fall within this new range. This normalization ensures that the data fits within a predictable and manageable scale.
Quantization (Mapping to Levels): Once the data is scaled, each value is assigned to the nearest quantization level. These quantization levels are evenly distributed within the standardized range (from zero to one). The number of quantization levels determines how many distinct values the data can take after quantization. Each scaled data point is rounded to the nearest quantization level, effectively reducing the infinite possibilities of the original data to a finite set of levels.
Inverse Scaling (De-normalization): After quantization, the values are mapped back to the original data range. This involves performing the inverse operations of the initial scaling step. The quantized values are expanded to match the original scale by multiplying them by the range of the original data and then adding back the minimum value. This step restores the data to its original scale but with the simplified set of quantized values.


A.2 Non-Uniform Quantization
   
Non-uniform quantization allocates quantization levels unevenly across the range of values, typically assigning more levels where the data is denser. This method is designed to minimize the quantization error for data that follows a specific distribution, such as Gaussian or Laplacian distributions commonly found in neural network weights and activations.
Non-uniform quantization involves several key steps, including level optimization—determining the optimal set of quantization levels based on the distribution of the data, which means figuring out where to place the quantization levels so that they best represent the data with minimal error—using techniques such as the Lloyd-Max Algorithm, an iterative method that seeks to minimize the total distortion or error introduced during quantization by adjusting both the quantization levels and the thresholds between them to find the best configuration, and K-Means Clustering, which involves grouping the data into clusters based on their values where each cluster represents a quantization level and the center of the cluster (centroid) becomes the quantized value, particularly useful because data points within a cluster are similar, so representing them with a single value introduces minimal error; then, in the quantization phase, mapping each data value to the nearest quantization level determined in the previous step, meaning that for each original value, you find the quantization level closest to it and use that as its new representation, thereby aiming to minimize the overall difference between the original data and the quantized data by assigning values to the nearest level; and finally, codebook storage, which involves creating a codebook containing the quantization levels and assignments to perform quantization and later reconstruct the original data (dequantization), facilitating compression and decompression by allowing both the sender and receiver (or the quantization and dequantization processes) to consistently interpret the quantized data, ensuring accurate data reconstruction within the limits of the quantization. 
 
Chapter 3. Motivation