# HAIPO: Hybrid AI Algorithm-Based Post-Fabrication Optimization for Modern 3D NAND Flash Memory
## 5 Evaluation  
**Methodology. ** We validate our HAIPO using two types of 320 ea modern 3D NAND flash chips: (_i_) 3D TLC NAND flash chips and (_ii_) the 3D QLC NAND flash chips. To investigate the effectiveness of BS (Bayesian simulation), we also evaluate the optimization using only DR and NR (HAIPO\({}_{\text{DN}}\)). For each type of chip, we perform post-fabrication optimization using one of HAIPO, HAIPO\({}_{\text{DN}}\), and HAIPO(-) on 40 blocks in 320 NAND flash chips that are randomly selected from different locations. After post-fabrication optimization, we evaluate the final reliability and performance of the selected NAND flash chips with the device parameters obtained from the three techniques. All test procedures for reliability and performance follow commercial industry standards [22]. Since our proposed technique is the first attempt to automatically explore the optimal eFuse combination to satisfy various memory requirements such as reliability and performance, it is challenging to compare the results of our technique with those of other prior works with the same research goal. Therefore, the effectiveness of our work is verified across different steps in implementing our technique. **Reliability and Performance. ** Figure 12 compares the reliability and performance of the three post-fabrication optimization techniques in the TLC and QLC NAND flash chips. We measure VSum (the higher, the better) and average program latency tPROG (the lower, the better), respectively. Figure 11: Additional optimization modules: (**a**) Process-aware individual chip optimization, (**b**) Multi-objective optimization using an adaptive fitness function. We make two observations from Figure 12. First, HAIPO significantly improves reliability by 11% (8.8%) and performance by 6.2% (12%) on average for the TLC (QLC) NAND flash chips compared to HAIPO(-). The performance and reliability benefits clearly show the effectiveness of HAIPO at optimizing device parameters that are hard to find in manual-based optimization. Second, BS (Bayesian simulation) is the key factor for HAIPO to meet the reliability requirement in modern NAND flash memory. HAIPO\({}_{\text{DN}}\) also provides considerable benefits over HAIPO(-) (4.1% and 6.4% in reliability and performance, respectively, on average) but still fails to satisfy the reliability requirement in QLC NAND flash chips. This clearly shows that the Bayesian simulation plays a critical role in improving the effectiveness of our HAIPO. To further confirm the advantage of HAIPO on reliability, we investigate the \(V_{th}\) distribution in TLC NAND flash chips. As shown in Figure 13, HAIPO improves the reliability over HAIPO(-) significantly, resulting in (_i_) narrower distribution of all \(V_{th}\) states (i.e., improving VSum) and (_ii_) improving the amount of Disturb in the erased cells (i.e., lowering the RBers). In addition, to assess the efficiency of HAIPO, we examine how well our technique optimizes reliability and performance across generations. Figure 14 shows the change in reliability and performance of 3D TLC NAND flash memory according to the generations. The blue dots indicate the results of individual eFuses configurations, and the red solid line represents the center value of reliability and performance. In the initial step, VSum values are very widely distributed because a large number of eFuse configurations are examined. In the mid-step, the variation of the VSum values is shrunk, and the slope of the optimization gradually decreases. From the results, we can identify that if we aim to further optimize our HAIPO, we should focus on improving the optimization speed in the mid and final steps of our technique. Figure 12: Reliability (VSum) and performance (tPROG) comparisons of three post-fabrication optimization techniques for the TLC and QLC NAND flash chips. Figure 13: Comparison of \(V_{th}\) distributions in TLC 3D NAND flash memory across three post-fabrication optimization techniques. **Per-WL Optimization.