# A Survey on Memory Subsystems for Deep Neural Network Accelerators
## 3 3D Memory Integration
The key characteristic of this model is that the continuous functions as multiplication and activation can be considered as stepwise functions with minimal DNN inference. The RAPIDNN architecture proposed in [8] maps all the functionalities into the memory block. It utilizes a single level memristor device, rather than multilevel memristors. The RAPIDNN network consists of two modules: DNN composer (software) and accelerator (hardware) as depicted in Figure 2:0 DNN composer creates a table of each neural network  
Figure 2:1 Outline of RAPIDNN architecture [8]. Figure 2:2 The composition of a vault [7]. operation, which can be stored in accelerator memory blocks. This sequence of operations is shown in Figure 2:3  
In [9], a reliability improvement scheme in a DNN accelerator with 3D memory integration has been proposed. High bandwidth memory (HBM) is a new concept introduced in [9]. In 3D-stacked memories, there is a different soft error rate (SER) in every layer, unlike planar memories. Since from the top of the package material alpha particles rise up, the top-most layer shows a higher SER than the remaining layers. In addition, the bottom-most layer also shows a high rate of SER due to the heat released from the interposer and logic units. As shown in Figure 13, the uppermost and lower-most layers show 10 times higher SER in comparison with the remaining layers. Authors in [9] mention that DNN possesses very accurate results in areas such as image classification. The majority of computations in DNNs are produced in the hidden layer--consisting of weighted filters. For every layer stored in the stacked DRAM dice, a frequently applied input feature map is able to extract the corresponding output feature map. In this process, the same weight filter, stored in the stacked DRAM dice, is read from numerous interactions by the memory. If faults are injected into weights, a duplicate fault effect can be caused by repetitive memory accesses. Authors in [9] find that all the weights in AlexNet are below  
Figure 2:4 Interpretation of computation of a neuron to sequence of operations in RAPIDNN network [8]. Figure 2:5 SER of different layers in a 3D DRAM integration [10]. than 1. The 29 th and 30 th bits of all the weights are 0 and 1 when they are expressed in 32-bit floating point. Two following cases show that flipping just one bit has a high impact on accuracy. * Case-1: flipping the 29 th bit (1\(\rightarrow\)0) for 500 weights in each layer of DNN. * Case-2: flipping the 30 th bit (0\(\rightarrow\)1) for 5 weights in each layer of DNN. Therefore, faults on weights, and everything stored in memory layers, create abnormality in operations. [PERSON] et al. in [9] shows that designing fault-tolerant DNN accelerators for reaching high accuracy levels is a topic that should be taken into consideration. Deeptrain [11] presents a novel architecture using the 3D stacking processing-in-memory (PIM) paradigm. It applies to the hybrid memory cube (HMC) concept for accelerating training and inference [11]. Deeptrain integrates a logic layer in the lower layer and high-bandwidth DRAM layers in upper layers for eliminating off-chip DRAM accesses. Deeptrain is for DNN training using an embedded platform, yet it does not possess the quality of back propagation of pooling layers and weight updates by the DNN. An overview of Deeptrain architecture has been displayed in Figure 2:6  
Every programmable memory address generator (PMAG) is equipped with a vault. This PMAG unit is a key component in Deeptrain. PMAG realizes different data flows in Deeptrain based on a state machine. PMAG maps various data (gradients, parameters, and inputs) to individual vaults. PMAG couples partitioning as well as data mapping along with the bus communication fabric. PMAG can be configured in regards to numerous computational kernels in the training phase. In recent studies, 3D memory stacking-based architectures for DNN have been introduced. These structures utilize DRAM stacking over the processing elements.