Analog Computing for AI Sometimes Needs Correction by Digital Computing: Why and When

Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-11 09:37:01

Abstract: Analog computing is a compute-in-memory technology that allows multiple dot-product operations to be performed in parallel, and can extremely accelerate matrix-vector multiplications in AI. However, computation in analog computing devices is imperfect due to its non-idealities. This can severely degrade its accuracy, and prevents the analog computing from being widely used.
In this paper, we propose a correction-based approach to mitigate these non-idealities of analog computing. The proposed method exploits the confidence score calculated from analog computing, and leverages digital computing to correct the result if the confidence is low. We propose two algorithms that efficiently improve accuracy without requiring offline profiling or training. First, the digital rate-based correction algorithm optimizes the accuracy with given the target digital computing usage rate. Second, the confidence threshold based correction algorithm balances the digital computing usage rate and the accuracy by finding the appropriate threshold from online profiling. We use several image recognition models to show that the proposed algorithms improve the accuracy of AI on analog computing by efficiently utilizing the digital computing.

Main paper: 
 introduction: 
 Analog computing is a compute-in-memory technology based on the law of physics. An analog computing device has crossbars of memory cells like other memory devices. If voltages are applied to one side, the results of dot-products between these voltages and the conductance stored in the cells are derived using Ohm's and Kirchhoff's laws. This phenomenon allows Matrix-Vector Multiplication (MVM), a key operation in AI models, to be performed in O(1) time complexity. In addition, since the matrix elements reside in the crossbars as the conductance during computation, the memory bandwidth requirements for MVM operations are significantly reduced. These advantages make analog computing a promising solution to extremely improve the performance and energy efficiency of AI, compared to digital computing devices, like GPUs or NPUs.However, analog computing introduces computational noise due to its non-idealities, resulting in imperfect results and limiting its adoption. There are many sources of computational noises. First, noise can be introduced when converting input value to voltage level. Second, the matrix elements need to be programmed as the conductance level of the memory cells, but it is difficult to fine-tune the conductance level. Moreover, the conductance can leak over time. Third, the dot-product result is derived as the current level, which can be affected by electrical interference or wire capacity limitations. Fourth, converting the current level to digital output is prone to errors, and its resolution may be limited.Submitted to the Second Workshop on Machine Learning with New Compute Paradigms at NeurIPS (MLNCP 2024). Do not distribute.To mitigate these non-idealities, several approaches have been proposed. Some works exploit redundancy when storing or processing data to improve computational accuracy [1, 5] . Several architectural studies have been conducted [23, 4] . For example, a large shared analog-to-digital converter (ADC) enhances bit resolution without overhead [23] . Some works optimize AI models for analog devices, such as analog-aware training to mitigate the conductance leakage effect [20, 25] and tailored stochastic gradient descent (SGD) algorithm [26] . In addition, hybrid analog-digital architectures have been proposed [8, 12, 14, 16, 17] . In these architectures, digital computing handles the accuracy-sensitive parts while analog computing handles the rest to increase energy efficiency.In this paper, we propose a correction-based approach to overcome the non-ideality of analog computing. In our approach, we first use analog computing devices to perform AI models and check the confidence of the result. Then, if the result is not confident enough, we correct it with digital computing. To determine confidence, we use the difference between the maximum and the second maximum softmax value [22] . We propose two algorithms for the correction. The first algorithm improves the accuracy of the AI model while maintaining the target digital computing device usage rate. The second algorithm determines the appropriate confidence threshold from the profiling phase. Then, in the running phase, the threshold is used to determine whether the result should be corrected. The proposed algorithms do not require any offline profiling or training.We show the feasibility of the proposed mechanisms with an analog computing device emulator [21] .With four image classification AI models, we achieve near digital accuracy, with only 22%-62% support of digital computing. The proposed mechanisms are independent of the hardware architecture and can be used with any AI model that ends with a softmax model, including LLMs. In future work, we will apply the proposed mechanisms to other AI models including LLMs, and analyze the latency and power consumption with the detailed hardware simulation.2 Background and Related Work Here's how it works: if we program the conductance of each memory cell as G i,j , and then apply a voltage V i to each row, then according to Ohm's law and Kirchhoff's law, the current flowing through the j-th column is equivalent to i G ji • V i . Based on this principle, matrix-vector multiplications can be implemented in a fully parallel fashion, in a single time step. 
analog computing and its non-idealities: 
 𝐼 ! = # "#! $ 𝐺 !" × 𝑉 " WL BL V n I 1 V 2 V 1 I 2 I m G 11 G 21 G m1As mentioned above, analog computing offers great efficiency through highly parallelized, in-situ computations. It has fast computation speed and high energy efficiency. In addition, it alleviates the memory bandwidth bottleneck by performing computations where the data resides.However, analog computing is inherently vulnerable thus it is difficult to achieve the same level of computational accuracy as digital computing. Analog devices in nature have programming noise, read noise, temporal resistance drift, limited bit-level precision, analog-digital conversion loss, etc [3, 4, 10, 18, 23, 25] . These non-ideal factors might degrade computational accuracy. Therefore, for practical use, it is important to maintain a certain level of accuracy in AI computation under these non-idealities of analog computation.To alleviate the accuracy degradation of AI computation caused by non-ideality, many previous works have been proposed. [4, 23, 25] proposed novel architectural designs that can minimize the impact of conversion loss bit-level precision limitation and analog-digital conversion loss. [1, 5] presented a bit-level divide-and-merge mechanism to recover from the bit-level precision limit. [15] proposed a time-domain conversion technique and buffer for storing intermediate analog values to minimize losses in analog-digital conversion. [20, 25] proposed a new AI training algorithm that is robust to temporal conductance drift. Some work [11, 18] aimed to minimize the stuck-at faults using redundant arrays and re-mapping weights. Tiki-Taka [26] addressed the inefficiency and high error rates of stochastic gradient descent (SGD) on analog devices using an auxiliary array.On the other hand, several works introduced hybrid designs that combines analog computing with highly accurate digital computing to compensate for non-ideality. Rashed et al. [8] proposed an architecture that compute the most significant bits (MSBs) on digital logic and the least significant bits (LSBs) on analog crossbars. Nandakumar et al. [17] presented a design where high-precision digital computing accumulates the partial sum of the results calculated in an analog computing device and updates the precise weights in the analog computing device. ReHy [12] delegated feed-forward propagation to analog computing, while backward propagation, which requires precise computation, is handled by digital computing for CNN training. 3D-ReG [14] is a heterogeneous architecture that vertically integrates ReRAM-based analog device and GPU to improve DNN training efficiency.HARDSEA [16] aimed to implement self-attention in transformer models using a hybrid design that offloads dense relevance predictions to analog computing devices and handles sparse, precise computations with digital computing. 
design: 
 In this paper, we propose analog-digital hybrid AI computation that, only when the AI computation in analog computing devices seems to be wrong, digital device steps and corrects the answer. This approach takes advantage of both the high efficiency of analog computing and the high accuracy of digital computing.The problems we propose to solve in our approach are: (1) How do we know that an analog AI computation is wrong? In the real world, the correct answer is unknown. (2) How does digital computing decide whether to intervene? We need a strategy that balances analog computing's efficiency and digital computing's correctness. In this section, we discuss the answers to both questions.To predict the correctness of analog computation, we use the confidence score of the AI model.There are several ways to define a confidence score [22] , and we choose the difference between the top two softmax values from the last layer. The softmax values from the last layer represent the likelihood of each item or class being selected as the output, so the difference between the top two softmax values represents the confidence of the best answer compared to the second best answer.The principle behind our approach is that when the difference between the top two softmax values is sufficiently large, there is little chance that their order will be reversed due to analog calculation errors. Figure 2 (a) presents the relationship between confidence score and actual correctness of four AI models run on a digital device. It has two plots: the left with data from accurate predictions, and the right from incorrect ones. The x-axis is the confidence score range, and the y-axis indicates the number of cases that their data have the confidence score belonging to the range in the x-axis. These plots demonstrate a clear trend: accurate predictions are likely to have higher confidence score, and wrong predictions usually have lower confidence score. Figure 2 (b) shows results when using analog computing, emulated through IBM AIHWKIT [21] , with further details provided in Section 4. In analog computing, the trend between confidence and accuracy remains consistent.To address computational errors in analog computing devices, our solution involves initially performing the computations on the analog device. If confidence is sufficiently high, we accept the result; otherwise, the result is corrected by the digital accelerators. Obtaining confidence score is straightforward: the AI model first identifies the maximum softmax value, and during this process, the second highest softmax value is naturally obtained with little extra effort. To offload the computation, we can take advantage of the hybrid approach by transferring only the input data, which is likely to reside in the host memory, to the digital accelerators.We propose two mechanisms for determining a confidence threshold for triggering digital device intervention. The threshold is determined based on either (1) the target utilization rate of digital computation, or (2) continuous online profiling. Both mechanisms are based on online profiling, not requiring offline profiling. This enables immediate adaptation without pre-processing AI model on both analog and digital computing devices to collect data in advance.In this method, the system administrator determines the target utilization rate of digital computation, α%, taking into account the available resources and power conditions. The AI model initially processes incoming data on an analog device first, generating a confidence score and a result. In case the confidence score ranks within the lowest α%, the computation is transferred to a digital accelerator, which reprocesses the input and updates the result accordingly. This approach primarily uses analog computing, engaging digital computing only for the least confident α% cases.Algorithm 1 describes the overall procedure. An online histogram tracks the distribution of confidence, from which the rank of a specific confidence is determined. Assuming the confidence falls into bin index k, then its rank isk -1 i=0 count of bin(i) + randint.(0, count of bin(k))From the calculated rank, we can get the percentile of the confidence. When the percentile is lower than α, we re-run the AI model in a digital accelerator with same input and update the result. This mechanism is designed to determine the optimal confidence threshold through online profiling.It is intended to meet scenarios in that administrators do not need to set a digital computing usage rate or prefer the system to manage settings autonomously.Algorithm 2 describes the entire procedure, which consists of two phases: profiling and execution.In the profiling phase, input data is simultaneously processed by both analog and digital computing devices to collect the distribution of confidence. At the end of the profiling phase, a threshold is established. In the execution phase, the AI model is executed on analog computing devices and the results are adjusted by digital computing if the confidence is less than or equal to the threshold value.The key here is to determine the appropriate threshold. If the threshold is set too low, digital computing may be unable to calibrate the results, potentially degrading the accuracy of AI results. Conversely, if the threshold is set too high, excessive digital computing will be employed, which could undermine the efficiency of analog computing.To determine the threshold, we specifically profile cases where the analog and digital computing results differ. As digital computation is typically more accurate, the different result indicates that the analog result might be incorrect. Therefore, we profile these cases to determine the range of 'confidence score that needs to be corrected'. We set the threshold with two different options, the 70-th and 90-th percentile of these profiled confidence scores. Profiling is conducted periodically, and the threshold is updated via a weighted moving average, with a weight of 0.5.It is possible that no cases is found in which the results of analog and digital computing differ. In such a case, we set the confidence threshold for profiling to 0. This is a reasonable assumption because it is highly probable that the results of analog and digital computing will be identical.1: data id ← data id + 1 2: if data id %phase len < prof rate × phase len then ▷ Profiling phase 3:Compute the AI model with analog and digital computing devices in parallel 4:if Result analog ̸ = Result digital then 5:profile.append(confidence score from analog computing) 6:end if 7: else 8:if data id %phase len == prof rate × phase len then ▷ Start of execution phase 9:threshold ′ = δ%th percentile of profiled confidence scores 10:threshold ← threshold × β + threshold ′ × (1 -β) 11: end if 12:Compute the AI model with analog computing device 13:if confidence score ≤ threshold then 14:Correct the result with digital computing device 15:end if 16: end if 4 Initial Results 
experimental methodology: 
 In this paper, we focus on evaluating the accuracy of AI computation and the usage of digital computing. Accuracy is the key target metric, and digital computing usage can indicate the achievable energy efficiency and performance improvements of the proposed mechanisms. We leave a detailed analysis of latency and power consumption as future work.To emulate computational noises on analog computing devices, we utilize the IBM AIHWKIT [21] .We mostly use default configurations: PCM-like memory cells, 512×512 crossbar arrays, 8-bit DAC, and 8-bit ADC. Note that our mechanism is orthogonal to the characteristics of analog computing devices, and evaluation with various device configurations is our on-going work.To evaluate the accuracy of analog computing devices, we choose the image classification tasks as it is easy to verify the correctness with labeled dataset. We use four image classification models, ResNet-18 [9] , Inception-v4 [24] , ViT-Base, and ViT-Large [7] . For the dataset, we use ImageNet data [6] since it has a large number of classes. In addition, it has 50,000 images with labels for validation. We use the validation set as input data for evaluation. based online profiling accurately predicts the confidence rank of the result. We also verified that actual digital computing usage is close to the target rate, but the graph is omitted due to space limits. Four graphs show similar trends. The proposed algorithm achieves accuracy close to that of digitalonly computation while increasing the usage of analog computing to 38-78%, thereby enhancing computing efficiency. The profiling-based threshold setting algorithm performs comparably to the ideal case, suggesting that online profiling is sufficient, even with just a 5% profiling rate. Increasing the profiling rate may have better results, however, it is less efficient since both analog and digital computations are required during profiling. Comparing the green and blue graphs, a trade-off between accuracy and efficiency is observed when setting the threshold at the 70th and 90th percentile, with the 90th percentile providing more achievable results. 
discussion and conclusion: 
 In this paper, we propose a correction-based approach to overcome analog computing's non-ideality.We find that the confidence from the result of the AI model can be a good indicator of its correctness, and improve the overall accuracy while minimizing digital computing usage. Our early experimental results show that the proposed algorithms based on confidence score efficiently optimize the digital computing usage to achieve accuracy close to digital computing.Our future work is as follows. First, detailed hardware simulation is required to prove the efficiency of the proposed algorithms. Especially, the latency and energy efficiency are affected by hardware configurations of analog computing. We will evaluate our mechanisms with various analog computing devices using the detailed analog hardware simulator, e.g., DNN+NeuroSim [13, 19] .Second, we can apply the proposed methods to other AI models, including Large Language Models (LLMs). Since the confidence based mechanisms have been used for LLMs [2, 22] , we believe that our approach is easily adopted for LLMs. If the capacity of the analog computing device is limited, analog computing can be applied to only the initial layers or the distillation models. 
 
 Appendix:  



Reviews: review: title: title: Review for Analog Computing for AI Sometimes Needs Correction by Digital Computing: Why and When
main_review: main_review: The paper presents an algorithm to automatically coordinate analog and digital compute to balance the power efficiency and speed benefits of analog computing with the noise resiliency and accuracy of digital computing. Specifically, the authors propose to execute a digital model when needed based on the confidence of the output provided by the analog model. The confidence of the analog model is estimated by the distribution of softmax outputs, and thus, this methodology could be generally applied to many types of networks including LLMs.

Implicitly this approach relies on there being a connection between instances where the analog network deviates from the digital network and instances where the outputs of the analog network have low confidence. While the intuitions described in section 3.1 make sense, it is difficult to conceptualize the distribution of error at the output of the analog network. It would be great if additional quantitative analysis could be provided to highlight the degree to which the chosen confidence metric predicts deviations between the output of the analog model and outputs of the digital model. Specifically, while figure 2 shows how confidence reflects when digital and analog outputs are incorrect, it does not seem to show how confidence reflects when digital and analog outputs are different, and that is the challenge that is resolved by the proposed method.

Other than this, the paper is well-written and effective in discussing a couple intuitive strategies for choosing when to execute the digital model. Additionally, the authors experimentally validate their algorithm and demonstrate digital-like performance with reduced computation. It would also be good to include more discussion on how future work could potentially provide more precise estimates of when the analog and digital models deviate in order to further limit the amount of digital network executions.
title: title: The authors introduce two algorithms aimed at approving the accuracy of analog hardware by offloading a fraction of the computation to digital hardware
main_review: main_review: This paper presents two methods to remedy the reduced accuracy of analog hardware caused by its non-idealities. Both methods are based on estimating the confidence of an analog neural networks output, and then correcting low-confidence outputs by performing an additional run with a digital-only neural network. The authors test these methods by using a employing a software library that simulates analog electroinic crossbar arrays. Results indicate that the accuracy could be improved by up to 20 percentage points (depending on the model and method used). Overall, I think this is a nicely written paper and I am in favor of accepting it; I just have some minor comments and questions:

- The authors use analog computing pretty much synonymously with analog electronic crossbar arrays. For completeness, I think it should be mentioned that analog computing also encompasses optical or photonic computing (and potentially even mechanical or other forms of computing, though these are rather niche). Since their method is hardware and model agnostic, it should also be applicable to other kinds of hardware.
- It would be interesting to learn what happens if one combines the correction-based approach presented here with other methods aimed at overcoming analog hardware non-idealities such as hardware aware training. Would the combined approach increase the accuracy even further or would the improvements be neglegible? 
- As the authors themselves point out, applying the proposed methods to other AI models would be very illuminating. Even the 4 models shown in this paper exhibit significant variation in how much digital computation is needed to get an accuracy close to that of digital hardware.
- In Algorithms 1 and 2, the variable names could be more clearly explained in the text. For example, the variable $\beta$ is introduced in line 10 in Algorithm 2 without explanation of what it means. From context, one can infer that it is the weight of the moving average, but this is never explicitly said in the text. In fact, in the main text the weight of the moving average is said to be equal to 0.5 always. Adding the definition of these variables would make the algorithms a lot more readable.
- There is a typo on line 168: It should be either _no case is found_ or _no cases are found_.