# PCM-2R: Accelerating MLC PCM Writes via Data Reshaping and Remapping

Authors: Feng Hong; Jianquan Zhang; Shigui Qi; Zheng Li
Journal: Mobile Information Systems
 (2022)

---

## Abstract

Multilevel cell (MLC) phase change memory (PCM) shows great potential in terms of capacity and cost compared with single-level cell (SLC) PCM by storing multiple bits in one physical PCM cell. However, poor write performance is a huge challenge for MLC PCM. In general, write latency of MLC PCM is 10 to 100X longer compared with DRAM technology. Considerable write latency greatly degrades the overall system performance and restricts the application of MLC PCM. Actually, several chips compose a memory DIMM to match the wide interface of data bus. The data of a write request, i.e., a cache line block, are distributed to multiple PCM chips. As a result, the write service time is determined by the chips with the most data amount. Conventional PCM write schemes do not care for the modified-byte distribution among PCM chips and it just waits for the completion of the chip with the most amount of data. However, it is observed that (1) the conventional PCM write scheme suffers from unbalanced modified-byte distribution that some PCM chips bear too many modified bytes while some chips are kept idle for long times. (2) The modified-byte distribution shows some unique patterns that some bytes are changed more frequently compared with others. (3) MLC PCM shows significant asymmetry considering only MSB or LSB transitions. Based on these observations, in order to solve the poor write problem of PCM, this article presents a novel PCM write scheme called PCM-2R. The key ideas behind our proposed scheme are to reshape the data to evenly distribute the cache line blocks among all chips based on their modified-byte distribution pattern to avoid unbalanced distribution and then remap modified bytes to fast region after decoupling MLC PCM cells considering the state transition asymmetries. The evaluation results show that PCM-2R achieves 51% read latency reduction, 37% write latency reduction, 1.9X IPC improvement, 41% running time reduction, 2.2X throughout improvement, and 52% energy reduction compared with the baseline. Moreover, compared with the state-of-the-art write schemes, PCM-2R achieves 0.2X more IPC improvement and 0.2X throughout improvement.


---

## Body

## 1. Introduction

Modern data-intensive applications have exhibited a growing demand for storage-class memory, e.g., Internet of Things (IOT) and big data analytics. Phase Change Memory (PCM) is one of the most promising memory technologies since conventional DRAM technology faces some great challenges in scalability, energy consumption, and stability under the finer feature size [1–3]. Despite the advantages of nonvolatile and low energy consumption, multilevel cell (MLC) PCM shows great potential in terms of capacity and cost compared with single-level cell (SLC) PCM by storing multiple bits in one physical PCM cell [4, 5]. However, poor write performance and endurance are huge challenges for implementing PCM into the memory hierarchy [6–8]. In general, write latency of MLC PCM is 4 to 8X slower and 10 to 100X longer compared with SLC PCM and DRAM technology, respectively [9, 10]. The considerable write latency greatly degrades the overall system performance and restricts the application of MLC PCM. There are several disadvantages and limitations in PCM write [11]. Firstly, due to the complex program-and-verify scheme, the programming time is quite long. Secondly, due to the high write power consumption constraint, the number of cells that can be written in parallel is restricted [12–14]. As a result, data can only be written in write division mode and it takes several write operations to finish the whole write request [15–17]. These limitations may lead to potential performance degradation if we add PCM to the memory hierarchy. In general, the computer system is usually organized as a hierarchical architecture, including CPUs, caches, main memory, etc. The exchange unit between LLC (last level cache) and main memory is cache line. In order to match the wide interface of data bus, several chips compose a memory DIMM. The data of a write request are distributed to multiple PCM chips. As we mentioned, the PCM-based main memory cannot program all data once due to circuit constraints and the size of cache line block is much larger than the size of the operation unit. As a result, it takes several write operations for each chip to finish the data write and the write service time is determined by the chip with the most data amount [18].Conventional PCM write schemes do not care for the modified-byte distribution among PCM chips and it just waits for the completion of the chip with the most amount of data. However, we observe the following:(i)
The modified-byte distribution is quite unbalanced that some PCM chips bear too many modified bytes and the other chips are not. As a result, the idle chips have to wait for the busy chips and are kept idle for long times, which causes consequent low parallelism and high latency.(ii)
The modified-byte distribution shows some unique characteristics, such as cyclical, etc. In general, some bytes are changed more frequently compared with others and can be summarized into several common patterns, i.e., first half concentrated, second half concentrated, crossed, and others.In addition, we also observe the following:(iii)
MLC PCM shows significant asymmetry considering only MSB or LSB transitions. The worst transition latency of LSB is only 0.84X compared with the latency of MSB.Based on these key observations, to effectively reduce the write latency and improve the write parallelism of MLC PCM-based memory, we present a novel PCM write scheme called PCM-2R. The proposed scheme reshapes the data to evenly distribute the cache line block among all chips based on its modified-byte distribution pattern to avoid unbalanced distribution and then remaps modified bytes in fast region after decoupling MLC PCM cells considering the state transition asymmetries. The main contributions of this work are summarized as follows:(i)
We detect the modified-byte distribution patterns inside cache line write service and summarize it into several command cases. In general, the modified-byte distribution shows significantly cyclical pattern and the cycle length is typically 8 bytes. The modified-byte distribution shows some unique patterns that some bytes are changed more frequently compared with others. In addition, the modified-byte distribution is quite unbalanced under conventional scheme that some PCM chips bear too many modified bytes.(ii)
We propose a data reshaping method to distribute the modified bytes evenly and avoid unbalanced distribution among PCM chips to boost the write parallelism. Based on the three special modified-byte distribution patterns, data reshaping scheme reconstructs the relationship between the chips and data.(iii)
We propose a data remapping method to leverage MLC write asymmetries in state transition latency and remap modified bytes to fast region to reduce the write service time. The cache line is divided into fast write half line and slow write half line by decoupling MLC cells. Since all modified bytes are concentrated on the front position of the cache line after data reshaping, our proposed scheme can further improve the write performance via remapping most data to the fast region.(iv)
We build a multicore computer system on the full-system simulators and the evaluation results show that PCM-2R decreases 51% read latency and 37% write latency and earns 1.9X IPC improvement, 41% running time reduction, 2.2X throughput improvement, and 51% energy reduction compared with the baseline under multiprogrammed SPEC 2006 and multithreaded PARSEC 2.0 workloads. In addition, compared with the state-of-the-art write schemes [19, 20], PCM-2R achieves 0.2X more IPC improvement and 0.2X throughout improvement.The remainder of this paper is structured as follows. Section2 describes the background and presents the motivations and the key observations of this study. Section 3 introduces the architecture and implementation of proposed schemes. Section 4 presents and analyzes the experimental results. Section 5 summaries the related work. Finally, Section 6 offers conclusions.


## 2. Background and Motivations

### 2.1. Background

Phase Change Memory (PCM) is an extremely promising competitor to DRAM for some outstanding features such as better technology scaling, nonvolatility, and low idle power consumption [21]. Different from charge-based DRAM technology, PCM takes advantage of resistance of phase change material (chalcogenide glass, such as Ge2Sb2Te5) to store digital information and it does not need to do refresh operations. In general, amorphous state material exhibits higher resistance compared with the crystalline state material. By applying different intensity of current pulses and programming time to the phase change material, the chalcogenide material can be manipulated between the amorphous state and the crystalline state [22, 23].As shown in Figure1, PCM utilizes the tight resistance ranges to present digital information (logical “0” or “1”). The switching of logic “0” to “1” is defined as SET operation and the switching of logic “1” to “0” is defined as RESET operation. Single-level cell (SLC) stores only one bit inside one physical PCM cell and it takes two resistance levels in total. It is practical to store multiple bits in one cell by dividing more resistance levels, such as the multilevel cell (MLC) technology. Multilevel cell (MLC) PCM shows great potential in terms of capacity and cost compared with single-level cell (SLC) PCM by storing multiple bits in one PCM cell [24].Figure 1
Resistance distributions of (a) SLC and (b) MLC PCM cells.
(a)(b)Program-and-verify (P&V) write strategy (or iterative write algorithm) is widely used in MLC PCM cell programming. According to the previous art Mercury [25], the write approaches can be concluded into two typical kinds, i.e., SET to RESET (referred to as S2R) and RESET to SET (referred to as R2S), as shown in Figure 2. By leveraging the adaptive programming techniques named State-aware Adaptive Programming, the transitions between different MLC cell states can be realized using S2R or R2S methods. R2S programming involves multiple short RESET pulses and each pulse gradually reduces the cell resistance. The RESET pulses continue until the cell reaches the given resistance value. In constant, S2R programming starts with a long SET pulse and the material is set to a lower resistance level state. After that, the resistance level is gradually increased with individual RESET pulses. In general, different MLC cell states take use of adaptive programming schemes and the service time depends on the initial state and the destination state, as shown in Table 1. Programming of PCM cells is more elaborate and complex than DRAM technology. As a result, the programming time is much longer. Even worse, MLC PCM involves more narrow resistance ranges of each data value, which increases the difficulty of programming and greatly expands the operation delay. Poor write performance has become a huge challenge for MLC PCM. In general, write latency of MLC PCM is 10 to 100X longer compared with DRAM technology. This poor write degrades the overall system performance and greatly restricts the application of MLC PCM [26].Figure 2
MLC PCM P&V strategies.Table 1
MLC PCM state transitions.
Destination initial0001101100—Init + R2SInit + R2SInit + R2S01Init + R2S—Init + R2SInit + R2S10Init + S2RInit + S2R—S2R (without init)11Init + S2RInit + S2RInit + S2R—In this paper, we consider 2-bit MLC PCM that one cell stores two logic bits. According to Mercury model [25], the state transition latency is summarized in Figure 3. The latency is normalized to the “00” to “10” latency. We highlight two novel insights that (1) MLC PCM shows significant asymmetry between state transitions. The programming time varies much and depends on the initial state and the destination state. In general, write latency of middle states “01”/“10” is much larger than states “00”/“11.” (2) MLC PCM shows significant asymmetry considering only MSB or LSB state transitions. The worst transition latency of LSB is only 0.84X compared with the latency of MSB, as shown in the red part in Figure 3.Figure 3
MLC PCM state transition latency according to prototype Mercury [25].There are several limitations in PCM write. The number of cells that can be programmed concurrently is restricted due to the high programming power consumption and write circuit constraints. As a result, the whole data can only be written in write division mode and it takes several write operations to finish the whole write request [13, 14]. When PCM is incorporated into the main memory, these limitations will further amplify the shortcomings of PCM. In the memory hierarchy, the exchange unit between LLC (last level cache) and main memory is cache line. In general, several chips compose a memory DIMM in order to match the wide interface of the data bus. The data of a write request, i.e., a cache line block, are distributed to multiple PCM chips to improve the write parallelism. However, the PCM-based main memory cannot program all data in one operation due to circuit constraints. As a result, it takes several write operations to complete the write service for the cache line and the write service time is determined by the chips with the most data amount.


### 2.2. Motivations and Observations

Conventional PCM write schemes use the naive DIMM organization and the cache line data are sequentially assigned to multiple chips. It does not care for the modified-byte distribution among PCM chips and it just waits for the completion of the chip with the most amount of data. Giving a simple example as shown in Figure4, assuming the cache line size is 64 bytes, a memory DIMM is composed of four 16-bit wide PCM chips (the size of the write operation per chip is 16 bits); i.e., one chip can only operate two bytes in parallel. Under conventional byte distribution method, the cache line block is sequentially assigned to multiple chips. Conventional method does not care for the modified-byte distribution, which causes potential parallelism and performance degradation. In this example, chip 1 consumes 8 write units to complete the entire data write while the other chips take only 1 write unit to finish the service. Chip 0, chip 2, and chip 3 are idle and have to wait for the completion of chip 1 since it has the most data amount among all PCM chips. If the data are distributed evenly across all chips, the write parallelism can be significantly improved and the write latency can be greatly reduced.Figure 4
Conventional byte distribution and PCM write division scheme.In order to measure the degree of uneven distribution, we count the modified-byte distribution of memory requests (we assume the cache line is 64-byte size) and the results are shown in Figure5. The detailed system parameters and benchmark information can be found in Section 4. We draw three key observations from the experimental results:(1)
Cyclical. The modified-byte distribution shows significantly cyclical pattern and the cycle length is typically 8 bytes.(2)
Unique. The modified-byte distribution shows some unique patterns that some bytes are changed more frequently compared with others.(3)
Unbalanced. The modified-byte distribution is quite unbalanced under conventional scheme that some PCM chips bear too many modified bytes.Moreover, based on the MLC transition latency shown in Figure3, we observe the following:(4)
Asymmetries. The worst transition latency of LSB is only 0.84X compared with the latency of MSB considering only MSB or LSB.Figure 5
Modified-byte distribution of memory requests using SPEC CPU2006 and PARSEC 2.0 benchmarks. Each cache line is composed of 64 bytes.Y-axis represents the proportion of modified bytes of different benchmarks.According to observation “cyclical,” the cyclical pattern implies that if we can solve the problem in one or two cycles, the whole problem can be solved. Observation “unique” shows that if we can effectively deal with the problem according to the unique data distribution patterns, we can get significant endurance and performance improvement of PCM-based main memory. According to observation unbalanced, the quite unbalanced modified-byte distribution results in the circumstance that some PCM chips bear too many modified bytes. The write service time is determined by the chips with the most data amount, which causes write parallelism and performance degradation. Observation “asymmetries” hints that we can get further performance improvement by decoupling the PCM cells and distributing data to the fast regions (regions with only LSB state transitions).


## 2.1. Background

Phase Change Memory (PCM) is an extremely promising competitor to DRAM for some outstanding features such as better technology scaling, nonvolatility, and low idle power consumption [21]. Different from charge-based DRAM technology, PCM takes advantage of resistance of phase change material (chalcogenide glass, such as Ge2Sb2Te5) to store digital information and it does not need to do refresh operations. In general, amorphous state material exhibits higher resistance compared with the crystalline state material. By applying different intensity of current pulses and programming time to the phase change material, the chalcogenide material can be manipulated between the amorphous state and the crystalline state [22, 23].As shown in Figure1, PCM utilizes the tight resistance ranges to present digital information (logical “0” or “1”). The switching of logic “0” to “1” is defined as SET operation and the switching of logic “1” to “0” is defined as RESET operation. Single-level cell (SLC) stores only one bit inside one physical PCM cell and it takes two resistance levels in total. It is practical to store multiple bits in one cell by dividing more resistance levels, such as the multilevel cell (MLC) technology. Multilevel cell (MLC) PCM shows great potential in terms of capacity and cost compared with single-level cell (SLC) PCM by storing multiple bits in one PCM cell [24].Figure 1
Resistance distributions of (a) SLC and (b) MLC PCM cells.
(a)(b)Program-and-verify (P&V) write strategy (or iterative write algorithm) is widely used in MLC PCM cell programming. According to the previous art Mercury [25], the write approaches can be concluded into two typical kinds, i.e., SET to RESET (referred to as S2R) and RESET to SET (referred to as R2S), as shown in Figure 2. By leveraging the adaptive programming techniques named State-aware Adaptive Programming, the transitions between different MLC cell states can be realized using S2R or R2S methods. R2S programming involves multiple short RESET pulses and each pulse gradually reduces the cell resistance. The RESET pulses continue until the cell reaches the given resistance value. In constant, S2R programming starts with a long SET pulse and the material is set to a lower resistance level state. After that, the resistance level is gradually increased with individual RESET pulses. In general, different MLC cell states take use of adaptive programming schemes and the service time depends on the initial state and the destination state, as shown in Table 1. Programming of PCM cells is more elaborate and complex than DRAM technology. As a result, the programming time is much longer. Even worse, MLC PCM involves more narrow resistance ranges of each data value, which increases the difficulty of programming and greatly expands the operation delay. Poor write performance has become a huge challenge for MLC PCM. In general, write latency of MLC PCM is 10 to 100X longer compared with DRAM technology. This poor write degrades the overall system performance and greatly restricts the application of MLC PCM [26].Figure 2
MLC PCM P&V strategies.Table 1
MLC PCM state transitions.
Destination initial0001101100—Init + R2SInit + R2SInit + R2S01Init + R2S—Init + R2SInit + R2S10Init + S2RInit + S2R—S2R (without init)11Init + S2RInit + S2RInit + S2R—In this paper, we consider 2-bit MLC PCM that one cell stores two logic bits. According to Mercury model [25], the state transition latency is summarized in Figure 3. The latency is normalized to the “00” to “10” latency. We highlight two novel insights that (1) MLC PCM shows significant asymmetry between state transitions. The programming time varies much and depends on the initial state and the destination state. In general, write latency of middle states “01”/“10” is much larger than states “00”/“11.” (2) MLC PCM shows significant asymmetry considering only MSB or LSB state transitions. The worst transition latency of LSB is only 0.84X compared with the latency of MSB, as shown in the red part in Figure 3.Figure 3
MLC PCM state transition latency according to prototype Mercury [25].There are several limitations in PCM write. The number of cells that can be programmed concurrently is restricted due to the high programming power consumption and write circuit constraints. As a result, the whole data can only be written in write division mode and it takes several write operations to finish the whole write request [13, 14]. When PCM is incorporated into the main memory, these limitations will further amplify the shortcomings of PCM. In the memory hierarchy, the exchange unit between LLC (last level cache) and main memory is cache line. In general, several chips compose a memory DIMM in order to match the wide interface of the data bus. The data of a write request, i.e., a cache line block, are distributed to multiple PCM chips to improve the write parallelism. However, the PCM-based main memory cannot program all data in one operation due to circuit constraints. As a result, it takes several write operations to complete the write service for the cache line and the write service time is determined by the chips with the most data amount.


## 2.2. Motivations and Observations

Conventional PCM write schemes use the naive DIMM organization and the cache line data are sequentially assigned to multiple chips. It does not care for the modified-byte distribution among PCM chips and it just waits for the completion of the chip with the most amount of data. Giving a simple example as shown in Figure4, assuming the cache line size is 64 bytes, a memory DIMM is composed of four 16-bit wide PCM chips (the size of the write operation per chip is 16 bits); i.e., one chip can only operate two bytes in parallel. Under conventional byte distribution method, the cache line block is sequentially assigned to multiple chips. Conventional method does not care for the modified-byte distribution, which causes potential parallelism and performance degradation. In this example, chip 1 consumes 8 write units to complete the entire data write while the other chips take only 1 write unit to finish the service. Chip 0, chip 2, and chip 3 are idle and have to wait for the completion of chip 1 since it has the most data amount among all PCM chips. If the data are distributed evenly across all chips, the write parallelism can be significantly improved and the write latency can be greatly reduced.Figure 4
Conventional byte distribution and PCM write division scheme.In order to measure the degree of uneven distribution, we count the modified-byte distribution of memory requests (we assume the cache line is 64-byte size) and the results are shown in Figure5. The detailed system parameters and benchmark information can be found in Section 4. We draw three key observations from the experimental results:(1)
Cyclical. The modified-byte distribution shows significantly cyclical pattern and the cycle length is typically 8 bytes.(2)
Unique. The modified-byte distribution shows some unique patterns that some bytes are changed more frequently compared with others.(3)
Unbalanced. The modified-byte distribution is quite unbalanced under conventional scheme that some PCM chips bear too many modified bytes.Moreover, based on the MLC transition latency shown in Figure3, we observe the following:(4)
Asymmetries. The worst transition latency of LSB is only 0.84X compared with the latency of MSB considering only MSB or LSB.Figure 5
Modified-byte distribution of memory requests using SPEC CPU2006 and PARSEC 2.0 benchmarks. Each cache line is composed of 64 bytes.Y-axis represents the proportion of modified bytes of different benchmarks.According to observation “cyclical,” the cyclical pattern implies that if we can solve the problem in one or two cycles, the whole problem can be solved. Observation “unique” shows that if we can effectively deal with the problem according to the unique data distribution patterns, we can get significant endurance and performance improvement of PCM-based main memory. According to observation unbalanced, the quite unbalanced modified-byte distribution results in the circumstance that some PCM chips bear too many modified bytes. The write service time is determined by the chips with the most data amount, which causes write parallelism and performance degradation. Observation “asymmetries” hints that we can get further performance improvement by decoupling the PCM cells and distributing data to the fast regions (regions with only LSB state transitions).


## 3. PCM-2R Design

Keeping these key insights in mind, we present a novel write scheme named PCM-2R for MLC PCM-based memory. We first characterize the unique distribution patterns and then introduce our data reshaping and remapping methods as well as the architectural design.


### 3.1. Modified-Byte Distribution Patterns

According to our observations, the modified-byte distribution patterns can be classified into several cases, as shown in Table2 (C refers to the byte that is changed and U refers to state unchanged).(i)
First half concentrated: This pattern implies that the modified bytes are mainly concentrated on the first half of the 8 bytes’ cycle. The distribution profile shape is similar to letter “Z.” In other words, the distribution of one cycle is like “CCCCUUUU,” where “C” and “U” present that the byte is changed (modified) and unchanged (unmodified) compared with the original data, respectively.(ii)
Second half concentrated: This pattern is opposite to the first half concentrated that the modified bytes are concentrated on the second half of the 8 bytes’ cycle. The distribution looks like “UUUUCCCC.”(iii)
Crossed: The modified-byte distribution is crossed over the 8 bytes’ cycle and the distribution type is “CCUUCCUU.”(iv)
Others: The other remaining cases that we do not care for in this work are, for example, the uniform distribution and other irregular cases.Table 2
Modified-byte distribution patterns.
PatternsDescriptionExamplesInstancesFirst half concentratedModified bytes are concentrated in the first half of each 8 bytesCCCCUUUUleslie3d, gobmk, libquan-tum, canneal, and vipsSecond half concentratedModified bytes are concentrated in the second half of each 8 bytesUUUUCCCCzeusmp and bwavesCrossedModified bytes are crossed in each 8 bytesCCUUCCUUastar, freqmine, ferret, and facesimOthersThe other remaining cases—dedup, bzip2, and x264The percentage of distribution patterns is shown in Figure6. We observe that these three distribution patterns occupy more than 20% of memory accesses in some workloads such as canneal, astar, bwaves, gobmk, leslie3d, libquantum, and zeusmp. However, in some benchmarks, the circumstance is not obvious, such as dedup, bzip2, and x264. Since the modified-byte distribution shows some cyclical patterns, the unbalanced distribution results in potential performance and endurance problems. The service time is determined by the chip with most write work, which results in the parallelism degradation. In addition, some bytes are changed more frequently and these cells are more likely to be worn out. Keeping these typical patterns in mind, PCM-2R enhances the write parallelism and performance by leveraging data reshaping and remapping strategies.Figure 6
Percentage of distribution patterns using SPEC CPU2006 and PARSEC 2.0 benchmarks.


### 3.2. Data Reshaping and Remapping

#### 3.2.1. Data Reshaping

In order to avoid unbalanced data distribution, we propose a novel data reshaping method by leveraging the special characteristics shown in the modified-byte distribution. The primary job of reshaping scheme is to reconstruct the data and distribute the modified bytes to all chips evenly.The data reshaping scheme is based on modified-byte distribution patterns and is divided into three cases, as shown in Figure7. Under the conventional method, chip 0 corresponds to byte (8 × i) and byte (8 × i + 1), where i = {0, 1, 2, …, 7}. Data reshaping scheme reconstructs the relationship between the chips and data. Since the distribution has periodic characteristics, we take 2 write cycles for example. In detail, (1) if the distribution pattern is first half concentrated, the proposed reshaping scheme uses static data reorganization that exchanges the second half of the first 8 bytes and the first half of the second 8 bytes. Under this circumstance, all chips have the same amount of modified bytes, i.e., 2 modified bytes, and the programming time of each time is almost equal. Relatively, the write latency is halved (two write units are reduced to one write unit) as shown in Figure 7(a). (2) If the distribution pattern is second half concentrated, similarly, our proposed reshaping method exchanges the first half of the first 8 bytes and the second half of the second 8 bytes. Thus, the modified-byte is evenly distributed to all chips, as shown in Figure 7(b). (3) If the distribution pattern is crossed, however, exchange occurs between the first half of first 8 bytes and second 8 bytes, and second half of first 8 bytes and second 8 bytes. Different from previous circumstances, we only exchange part of the data. As shown in Figure 7(c), bytes 2 and 3 are exchanged with bytes 8 and 9. In the same way, bytes 6 and 7 are exchanged with bytes 12 and 13. The data are equally distributed among the individual chips.Figure 7
Data reshaping scheme based on different modified-byte distribution. (a) Data reshaping for first half concentrated; (b) data reshaping for second half concentrated; (c) data reshaping for crossed.


#### 3.2.2. Data Remapping

Inspired by prior art [27–30], we propose a novel bit-mapping scheme named data remapping to improve write performance and reduce energy consumption. The primary goal of data remapping is to remap the data after reshaping to fast region of MLC PCM by decoupling the cells considering the asymmetry between MSB and LSB state transitions. For illustrative purpose, we assume the cache line is 512 bits with 256 physical cells. The conventional bit-mapping scheme is shown in Figure 8. In general, one PCM cell stores two consequent bits and the whole cache line is distributed in continuous physical PCM cells. Under this circumstance, writing a cache block involves all of the bits (both the LSB and MSB bits) of the 256 PCM cells. Recently, SPCM mapping scheme [19, 20] is proposed to utilize the asymmetries of MLC PCM. By consolidating the physical units of the two cache lines, the memory lines are divided into two parts, i.e., one stored in MSBs in two PCM physical lines and the other stored in LSBs in two PCM lines. As shown in Figure 8, cache line block A is stored in the MSBs of two physical lines and cache line block B involves all LSBs of two physical lines. Since the LSBs write latency is much shorter than MSBs latency, the write service time of cache block B is reduced. In other words, only half of the cache lines can be accelerated.Figure 8
Remapping scheme.To explore and exploit the asymmetries between LSB and MSB transitions, we propose the data remapping scheme to speed up the write operation after implementing data reshaping scheme. As shown in Figure8, our proposed remapping scheme involves only one physical PCM line. The cache line has both MSB and LSB bits similar to conventional cache line. Differently, our design exploits interleaved mapping, and the first half of the cache line is stored in the LSBs of one physical line while the second half of the cache line is stored in the MSBs of the same PCM line. In other words, the cache line is divided into fast write half line and slow write half line by decoupling MLC cells. In general, our design differs from the conventional scheme and SPCM in the following aspects. On one hand, unlike conventional mapping scheme, the continuous bits are mapped to different physical cells. Compared with SPCM, data remapping scheme involves only one physical line for one cache line while SPCM has to write two different PCM lines for one cache block. On the other hand, the first half of the cache line is composed of all LSB bits to leverage its fast transition speed (fast write half line). Since all modified bytes are centered on the front position of the cache line after data reshaping, our proposed scheme can further improve the write performance via remapping the data to the fast “region” for all cache lines. However, SPCM can only accelerate half of the cache line write based on the write address (odd address or even address).As shown in Figure9, conventional PCM write scheme involves 8 write units without considering the actual modified-byte patterns. PCM-2R tries to reshape the data according to its data pattern. After the data reshaping method (first half concentrated in this case), the data amount of all chips is balanced and most modified bytes are centered on the front position. The number of write units is reduced since there are no modified bytes in some write units after data reshaping. Then the first half cache line is remapped to fast write region and the second half is remapped to slow write region. Fast region has shorter write latency and overall write performance is accelerated again by leveraging data remapping scheme.Figure 9
Putting data reshaping and remapping all together.


### 3.3. Architectural Design

The architectural design of PCM-2R, including proposed data reshaping and remapping schemes, is illustrated in Figure10. In general, PCM-2R involves the innovations in both memory controller and PCM array. The memory controller-side includes write buffer, modified-byte distribution checking, data reshaping and write control modules for write operations and read buffer, and data reconstruction and read control modules for read operations. The PCM array-side includes write control logic, write circuits, row/column decoders, and physical cells.Figure 10
Architecture of PCM-2R.In general, the DCW (Data-Comparison-Write) scheme [31] is adopted to reduce the energy consumption. During a write operation, DCW first reads out the old data and then writes the new data only when the old and new data are different. We add a middle layer named modified-byte distribution checking to provide necessary meta information to the data reshaping module designed in the memory controller. The memory controller first reads the old data stored in the PCM chips and compares the old data with the new data to be written and then records the modified-byte distribution. For the write request to be processed, the memory controller maintains a 64-bit array and each bit indicates the state of the 64 bytes’ cache line data. If one byte is modified, i.e., the new data are different from the stored old data, the value is labeled with “1.” In detail, PCM-2R uses the violent crack method and tries to find the most suitable data reshaping method or keep the original data. The modified-byte array is transformed according to each data reshaping method and our design estimates the number of write units of all proposed methods in parallel. For each data chunk (8 bytes per chunk and the cache line is 64 bytes), if any bytes need to be modified, i.e., the corresponding bits in the modified-byte array are 1, the write unit counter increases by one. Through the prior estimates about the time consumption based on the modified-byte distribution results, the reshaping selection can be released into very simple operations, as shown in Figure 11. Once the data reshaping mechanism is determined, an additional tag per cache line is introduced to indicate the pattern.Figure 11
Reshaping methods selection.The data reshaping and reconstruction are relative, as shown in Figure10. For write operation, the memory controller reshapes the data based on its pattern and stores the tags with reshaped data into the array. For read operation, the memory controller has to reconstruct the data based on the tags. Since the schemes involved in PCM-2R are quite simple, the data reshaping and reconstruction can be achieved through simple circuits, as, respectively, shown in Figures 12 and 13. The RTL designs of reshaping and reconstruction schemes involve only one multiplexer and several OBUFs; the time and space overhead caused by reshaping and reconstruction is small.Figure 12
Reshaping RTL designs.Figure 13
Reconstruction RTL designs.In addition, the data remapping mechanisms are designed at the hardware level and implemented in the MLC PCM array control logic. The key design of proposed remapping scheme is the write control logic. Write control logic determines which physical cells are adopted to store the data to be written. As shown in Figure14, the write control logic includes clock, write buffer, finite state machine, and write signals generation modules. The finite state machine selectively chooses the data through the multiplexer and generates write control signals to the actual write circuits. By leveraging the write control logic, the first half data are written into fast write half line and the second half data are written into slow write half line.Figure 14
Write control logic.


### 3.4. Overhead

In PCM-2R, a read operation is issued to compare the old data and the original data and triggers an extra time overhead. However, compared with the slow write operation, the overhead raised by an extra read operation is acceptable. It is worth noting that the key component of PCM-2R is the data reshaping mechanism selection, which affects the overall performance and introduces extra storage overhead. For each write request, the memory controller maintains a 64-bit array to indicate the state of the cache line data. Since the calculation involved is not complicated and can be released into simple gate operations, the time overhead is quite low. Moreover, the data reshaping and reconstruction methods deliver 64 LUTS (look-up-tables) overhead according to the experimental results on Xilinx FPGA [32]. There are four selected distribution patterns in PCM-2R, so the space overhead is 2 bits per 64B cache line (0.4%). In addition, the reshaping scheme has 1.8 to 4.3 ns overhead and the reconstruction scheme has extra 1.8 to 5.2 ns overhead. Since the write control logic only controls the write data selection, the overhead to support the data control is not significant. It is worth noting that PCM-2R does not have any PCM DIMM or PCM array changes, so there is no extra time, area, and energy overhead.


## 3.1. Modified-Byte Distribution Patterns

According to our observations, the modified-byte distribution patterns can be classified into several cases, as shown in Table2 (C refers to the byte that is changed and U refers to state unchanged).(i)
First half concentrated: This pattern implies that the modified bytes are mainly concentrated on the first half of the 8 bytes’ cycle. The distribution profile shape is similar to letter “Z.” In other words, the distribution of one cycle is like “CCCCUUUU,” where “C” and “U” present that the byte is changed (modified) and unchanged (unmodified) compared with the original data, respectively.(ii)
Second half concentrated: This pattern is opposite to the first half concentrated that the modified bytes are concentrated on the second half of the 8 bytes’ cycle. The distribution looks like “UUUUCCCC.”(iii)
Crossed: The modified-byte distribution is crossed over the 8 bytes’ cycle and the distribution type is “CCUUCCUU.”(iv)
Others: The other remaining cases that we do not care for in this work are, for example, the uniform distribution and other irregular cases.Table 2
Modified-byte distribution patterns.
PatternsDescriptionExamplesInstancesFirst half concentratedModified bytes are concentrated in the first half of each 8 bytesCCCCUUUUleslie3d, gobmk, libquan-tum, canneal, and vipsSecond half concentratedModified bytes are concentrated in the second half of each 8 bytesUUUUCCCCzeusmp and bwavesCrossedModified bytes are crossed in each 8 bytesCCUUCCUUastar, freqmine, ferret, and facesimOthersThe other remaining cases—dedup, bzip2, and x264The percentage of distribution patterns is shown in Figure6. We observe that these three distribution patterns occupy more than 20% of memory accesses in some workloads such as canneal, astar, bwaves, gobmk, leslie3d, libquantum, and zeusmp. However, in some benchmarks, the circumstance is not obvious, such as dedup, bzip2, and x264. Since the modified-byte distribution shows some cyclical patterns, the unbalanced distribution results in potential performance and endurance problems. The service time is determined by the chip with most write work, which results in the parallelism degradation. In addition, some bytes are changed more frequently and these cells are more likely to be worn out. Keeping these typical patterns in mind, PCM-2R enhances the write parallelism and performance by leveraging data reshaping and remapping strategies.Figure 6
Percentage of distribution patterns using SPEC CPU2006 and PARSEC 2.0 benchmarks.


## 3.2. Data Reshaping and Remapping

### 3.2.1. Data Reshaping

In order to avoid unbalanced data distribution, we propose a novel data reshaping method by leveraging the special characteristics shown in the modified-byte distribution. The primary job of reshaping scheme is to reconstruct the data and distribute the modified bytes to all chips evenly.The data reshaping scheme is based on modified-byte distribution patterns and is divided into three cases, as shown in Figure7. Under the conventional method, chip 0 corresponds to byte (8 × i) and byte (8 × i + 1), where i = {0, 1, 2, …, 7}. Data reshaping scheme reconstructs the relationship between the chips and data. Since the distribution has periodic characteristics, we take 2 write cycles for example. In detail, (1) if the distribution pattern is first half concentrated, the proposed reshaping scheme uses static data reorganization that exchanges the second half of the first 8 bytes and the first half of the second 8 bytes. Under this circumstance, all chips have the same amount of modified bytes, i.e., 2 modified bytes, and the programming time of each time is almost equal. Relatively, the write latency is halved (two write units are reduced to one write unit) as shown in Figure 7(a). (2) If the distribution pattern is second half concentrated, similarly, our proposed reshaping method exchanges the first half of the first 8 bytes and the second half of the second 8 bytes. Thus, the modified-byte is evenly distributed to all chips, as shown in Figure 7(b). (3) If the distribution pattern is crossed, however, exchange occurs between the first half of first 8 bytes and second 8 bytes, and second half of first 8 bytes and second 8 bytes. Different from previous circumstances, we only exchange part of the data. As shown in Figure 7(c), bytes 2 and 3 are exchanged with bytes 8 and 9. In the same way, bytes 6 and 7 are exchanged with bytes 12 and 13. The data are equally distributed among the individual chips.Figure 7
Data reshaping scheme based on different modified-byte distribution. (a) Data reshaping for first half concentrated; (b) data reshaping for second half concentrated; (c) data reshaping for crossed.


### 3.2.2. Data Remapping

Inspired by prior art [27–30], we propose a novel bit-mapping scheme named data remapping to improve write performance and reduce energy consumption. The primary goal of data remapping is to remap the data after reshaping to fast region of MLC PCM by decoupling the cells considering the asymmetry between MSB and LSB state transitions. For illustrative purpose, we assume the cache line is 512 bits with 256 physical cells. The conventional bit-mapping scheme is shown in Figure 8. In general, one PCM cell stores two consequent bits and the whole cache line is distributed in continuous physical PCM cells. Under this circumstance, writing a cache block involves all of the bits (both the LSB and MSB bits) of the 256 PCM cells. Recently, SPCM mapping scheme [19, 20] is proposed to utilize the asymmetries of MLC PCM. By consolidating the physical units of the two cache lines, the memory lines are divided into two parts, i.e., one stored in MSBs in two PCM physical lines and the other stored in LSBs in two PCM lines. As shown in Figure 8, cache line block A is stored in the MSBs of two physical lines and cache line block B involves all LSBs of two physical lines. Since the LSBs write latency is much shorter than MSBs latency, the write service time of cache block B is reduced. In other words, only half of the cache lines can be accelerated.Figure 8
Remapping scheme.To explore and exploit the asymmetries between LSB and MSB transitions, we propose the data remapping scheme to speed up the write operation after implementing data reshaping scheme. As shown in Figure8, our proposed remapping scheme involves only one physical PCM line. The cache line has both MSB and LSB bits similar to conventional cache line. Differently, our design exploits interleaved mapping, and the first half of the cache line is stored in the LSBs of one physical line while the second half of the cache line is stored in the MSBs of the same PCM line. In other words, the cache line is divided into fast write half line and slow write half line by decoupling MLC cells. In general, our design differs from the conventional scheme and SPCM in the following aspects. On one hand, unlike conventional mapping scheme, the continuous bits are mapped to different physical cells. Compared with SPCM, data remapping scheme involves only one physical line for one cache line while SPCM has to write two different PCM lines for one cache block. On the other hand, the first half of the cache line is composed of all LSB bits to leverage its fast transition speed (fast write half line). Since all modified bytes are centered on the front position of the cache line after data reshaping, our proposed scheme can further improve the write performance via remapping the data to the fast “region” for all cache lines. However, SPCM can only accelerate half of the cache line write based on the write address (odd address or even address).As shown in Figure9, conventional PCM write scheme involves 8 write units without considering the actual modified-byte patterns. PCM-2R tries to reshape the data according to its data pattern. After the data reshaping method (first half concentrated in this case), the data amount of all chips is balanced and most modified bytes are centered on the front position. The number of write units is reduced since there are no modified bytes in some write units after data reshaping. Then the first half cache line is remapped to fast write region and the second half is remapped to slow write region. Fast region has shorter write latency and overall write performance is accelerated again by leveraging data remapping scheme.Figure 9
Putting data reshaping and remapping all together.


## 3.2.1. Data Reshaping

In order to avoid unbalanced data distribution, we propose a novel data reshaping method by leveraging the special characteristics shown in the modified-byte distribution. The primary job of reshaping scheme is to reconstruct the data and distribute the modified bytes to all chips evenly.The data reshaping scheme is based on modified-byte distribution patterns and is divided into three cases, as shown in Figure7. Under the conventional method, chip 0 corresponds to byte (8 × i) and byte (8 × i + 1), where i = {0, 1, 2, …, 7}. Data reshaping scheme reconstructs the relationship between the chips and data. Since the distribution has periodic characteristics, we take 2 write cycles for example. In detail, (1) if the distribution pattern is first half concentrated, the proposed reshaping scheme uses static data reorganization that exchanges the second half of the first 8 bytes and the first half of the second 8 bytes. Under this circumstance, all chips have the same amount of modified bytes, i.e., 2 modified bytes, and the programming time of each time is almost equal. Relatively, the write latency is halved (two write units are reduced to one write unit) as shown in Figure 7(a). (2) If the distribution pattern is second half concentrated, similarly, our proposed reshaping method exchanges the first half of the first 8 bytes and the second half of the second 8 bytes. Thus, the modified-byte is evenly distributed to all chips, as shown in Figure 7(b). (3) If the distribution pattern is crossed, however, exchange occurs between the first half of first 8 bytes and second 8 bytes, and second half of first 8 bytes and second 8 bytes. Different from previous circumstances, we only exchange part of the data. As shown in Figure 7(c), bytes 2 and 3 are exchanged with bytes 8 and 9. In the same way, bytes 6 and 7 are exchanged with bytes 12 and 13. The data are equally distributed among the individual chips.Figure 7
Data reshaping scheme based on different modified-byte distribution. (a) Data reshaping for first half concentrated; (b) data reshaping for second half concentrated; (c) data reshaping for crossed.


## 3.2.2. Data Remapping

Inspired by prior art [27–30], we propose a novel bit-mapping scheme named data remapping to improve write performance and reduce energy consumption. The primary goal of data remapping is to remap the data after reshaping to fast region of MLC PCM by decoupling the cells considering the asymmetry between MSB and LSB state transitions. For illustrative purpose, we assume the cache line is 512 bits with 256 physical cells. The conventional bit-mapping scheme is shown in Figure 8. In general, one PCM cell stores two consequent bits and the whole cache line is distributed in continuous physical PCM cells. Under this circumstance, writing a cache block involves all of the bits (both the LSB and MSB bits) of the 256 PCM cells. Recently, SPCM mapping scheme [19, 20] is proposed to utilize the asymmetries of MLC PCM. By consolidating the physical units of the two cache lines, the memory lines are divided into two parts, i.e., one stored in MSBs in two PCM physical lines and the other stored in LSBs in two PCM lines. As shown in Figure 8, cache line block A is stored in the MSBs of two physical lines and cache line block B involves all LSBs of two physical lines. Since the LSBs write latency is much shorter than MSBs latency, the write service time of cache block B is reduced. In other words, only half of the cache lines can be accelerated.Figure 8
Remapping scheme.To explore and exploit the asymmetries between LSB and MSB transitions, we propose the data remapping scheme to speed up the write operation after implementing data reshaping scheme. As shown in Figure8, our proposed remapping scheme involves only one physical PCM line. The cache line has both MSB and LSB bits similar to conventional cache line. Differently, our design exploits interleaved mapping, and the first half of the cache line is stored in the LSBs of one physical line while the second half of the cache line is stored in the MSBs of the same PCM line. In other words, the cache line is divided into fast write half line and slow write half line by decoupling MLC cells. In general, our design differs from the conventional scheme and SPCM in the following aspects. On one hand, unlike conventional mapping scheme, the continuous bits are mapped to different physical cells. Compared with SPCM, data remapping scheme involves only one physical line for one cache line while SPCM has to write two different PCM lines for one cache block. On the other hand, the first half of the cache line is composed of all LSB bits to leverage its fast transition speed (fast write half line). Since all modified bytes are centered on the front position of the cache line after data reshaping, our proposed scheme can further improve the write performance via remapping the data to the fast “region” for all cache lines. However, SPCM can only accelerate half of the cache line write based on the write address (odd address or even address).As shown in Figure9, conventional PCM write scheme involves 8 write units without considering the actual modified-byte patterns. PCM-2R tries to reshape the data according to its data pattern. After the data reshaping method (first half concentrated in this case), the data amount of all chips is balanced and most modified bytes are centered on the front position. The number of write units is reduced since there are no modified bytes in some write units after data reshaping. Then the first half cache line is remapped to fast write region and the second half is remapped to slow write region. Fast region has shorter write latency and overall write performance is accelerated again by leveraging data remapping scheme.Figure 9
Putting data reshaping and remapping all together.


## 3.3. Architectural Design

The architectural design of PCM-2R, including proposed data reshaping and remapping schemes, is illustrated in Figure10. In general, PCM-2R involves the innovations in both memory controller and PCM array. The memory controller-side includes write buffer, modified-byte distribution checking, data reshaping and write control modules for write operations and read buffer, and data reconstruction and read control modules for read operations. The PCM array-side includes write control logic, write circuits, row/column decoders, and physical cells.Figure 10
Architecture of PCM-2R.In general, the DCW (Data-Comparison-Write) scheme [31] is adopted to reduce the energy consumption. During a write operation, DCW first reads out the old data and then writes the new data only when the old and new data are different. We add a middle layer named modified-byte distribution checking to provide necessary meta information to the data reshaping module designed in the memory controller. The memory controller first reads the old data stored in the PCM chips and compares the old data with the new data to be written and then records the modified-byte distribution. For the write request to be processed, the memory controller maintains a 64-bit array and each bit indicates the state of the 64 bytes’ cache line data. If one byte is modified, i.e., the new data are different from the stored old data, the value is labeled with “1.” In detail, PCM-2R uses the violent crack method and tries to find the most suitable data reshaping method or keep the original data. The modified-byte array is transformed according to each data reshaping method and our design estimates the number of write units of all proposed methods in parallel. For each data chunk (8 bytes per chunk and the cache line is 64 bytes), if any bytes need to be modified, i.e., the corresponding bits in the modified-byte array are 1, the write unit counter increases by one. Through the prior estimates about the time consumption based on the modified-byte distribution results, the reshaping selection can be released into very simple operations, as shown in Figure 11. Once the data reshaping mechanism is determined, an additional tag per cache line is introduced to indicate the pattern.Figure 11
Reshaping methods selection.The data reshaping and reconstruction are relative, as shown in Figure10. For write operation, the memory controller reshapes the data based on its pattern and stores the tags with reshaped data into the array. For read operation, the memory controller has to reconstruct the data based on the tags. Since the schemes involved in PCM-2R are quite simple, the data reshaping and reconstruction can be achieved through simple circuits, as, respectively, shown in Figures 12 and 13. The RTL designs of reshaping and reconstruction schemes involve only one multiplexer and several OBUFs; the time and space overhead caused by reshaping and reconstruction is small.Figure 12
Reshaping RTL designs.Figure 13
Reconstruction RTL designs.In addition, the data remapping mechanisms are designed at the hardware level and implemented in the MLC PCM array control logic. The key design of proposed remapping scheme is the write control logic. Write control logic determines which physical cells are adopted to store the data to be written. As shown in Figure14, the write control logic includes clock, write buffer, finite state machine, and write signals generation modules. The finite state machine selectively chooses the data through the multiplexer and generates write control signals to the actual write circuits. By leveraging the write control logic, the first half data are written into fast write half line and the second half data are written into slow write half line.Figure 14
Write control logic.


## 3.4. Overhead

In PCM-2R, a read operation is issued to compare the old data and the original data and triggers an extra time overhead. However, compared with the slow write operation, the overhead raised by an extra read operation is acceptable. It is worth noting that the key component of PCM-2R is the data reshaping mechanism selection, which affects the overall performance and introduces extra storage overhead. For each write request, the memory controller maintains a 64-bit array to indicate the state of the cache line data. Since the calculation involved is not complicated and can be released into simple gate operations, the time overhead is quite low. Moreover, the data reshaping and reconstruction methods deliver 64 LUTS (look-up-tables) overhead according to the experimental results on Xilinx FPGA [32]. There are four selected distribution patterns in PCM-2R, so the space overhead is 2 bits per 64B cache line (0.4%). In addition, the reshaping scheme has 1.8 to 4.3 ns overhead and the reconstruction scheme has extra 1.8 to 5.2 ns overhead. Since the write control logic only controls the write data selection, the overhead to support the data control is not significant. It is worth noting that PCM-2R does not have any PCM DIMM or PCM array changes, so there is no extra time, area, and energy overhead.


## 4. Performance Evaluation

### 4.1. Experimental Environment

In order to evaluate the efficiency and effectiveness of the proposed schemes, we use the full-system GEM5 simulator [33] combined with the cycle-level NVMain module [34], which is a cycle accurate memory simulator for NVMs. We use the same configuration parameters from prior similar studies [35] and the evaluation parameters used in this study are shown in Table 3. In general, we simulate a 4-core system with ALPHA-like processors and model the whole memory hierarchy including L1, L2, last level DRAM caches, and 16 GB MLC PCM main memory. DRAM cache is adopted for improving the performance and mitigating the endurance of PCM-based main memory [36–38]. The main memory has 4 channels in total, with 2 ranks per channel and 8 banks per rank. The memory controller adopts the FRFCFS-WQF scheduling proposed by previous art [39, 40]. The workloads used in this study are summarized in Table 4. We adopt a set of multithreaded and multiprogrammed programs in PARSEC 2.0 [41] and SPEC 2006 suites [42], respectively. All of these benchmarks involve intensive and nonintensive memory accesses and have different memory Read Per Kilo Instructions (RPKI) and memory Write Per Kilo Instructions (WPKI) rates. In addition, five billion instructions are simulated for each workload after fast-forwarding one billion instructions considering program initialization and cache warm-up. In general, we use the DCW [31] as the baseline and combine the DCW with the silentstore scheme [43, 44] as the comparison. All the comparison techniques have the same technology node (22 nm) and simulation configurations. PCM-R combines proposed reshaping method with DCW while PCM-2R puts reshaping and remapping all together. We also try to compare our proposed designs with state-of-the-art bit-mapping [3] and SPCM designs [19, 20]. Specifically, the row buffer hit/miss is considered in the evaluation and the time overhead caused by data reshaping, reconstruction, and remapping is also involved in our evaluation.Table 3
Parameters of simulation.
ParameterValueProcessor4 cores, 2 GHz, ALPHA-likeL1 cache32 kB I-cache, 32 kB D-cache, 2 cycles’ access latencyL2 cache8-way, 2 MB private per core, 20 cycles’ access latencyL3 DRAM cache16-way, 32 MB shared, 50 cycles’ access latencyCache organization64-byte line size, write back, LRU replacementMemory controllerFRFCFS-WQF scheduling, 32-entry R/W queuesMemory organization16 GB MLC PCM, 4 channels with 2 ranks each, 8 banks per rankMemory timingRead 250 ns, write 2μs (“00” to “10,” 1.0X)PCM organization4 chips per DIMM, the size of write unit is 64 bits (32 2 bit MLC cells)Table 4
Characteristics of the evaluated workloads.
WorkloadRPKIWPKISPEC 2006, multiprogrammedAstar: 4 copies of astar0.530.38bwaves: 4 copies of bwaves11.195.67bzip2: 4 copies of bzip28.814.64gobmk: 4 copies of gobmk0.140.13leslie3d: 4 copies of leslie3d7.112.59libquantum: 4 copies of libquantum10.956.93zeusmp: 4 copies of zeusmp13.563.18PARSEC 2.0, multithreadedcanneal1.881.31dedup0.350.33facesim0.860.71ferret0.230.15freqmine0.320.33vips1.271.18x2640.180.20


### 4.2. Memory Read Latency

The write latency reduction significantly benefits the read latency due to the wait time reduction of read requests [45]. Although PCM-2R does not improve the read performance directly, it improves the write parallelism and read latency benefits from the write service time reduction. Figure 15 shows the experimental results of read latency. On average, PCM-R shows 48% read latency reduction compared with the baseline. In comparison, silentstore and bit-mapping + SPCM show, respectively, 40% and 45% read latency improvement and PCM-2R shows almost 51% read latency reduction by leveraging data remapping scheme. One workload shows significant performance improvement (facesim) and outperforms the state-of-the-art scheme by 25% more read latency reduction because the modified-byte pattern is obvious and the amount is balanced after data reshaping and remapping methods. It is noteworthy that our schemes are not effective in some workloads with balanced modified-byte distribution, such as dedup, bzip2, and x264. In other words, PCM-R and PCM-2R are not effective for all benchmarks and efficient for some workloads with special unbalanced distribution patterns.Figure 15
Read latency.


### 4.3. Memory Write Latency

Figure16 illustrates the comparison of memory write latency of different PCM write schemes. The improvement of write latency is not significant compared with the increase of read latency since the memory controller we used in this study adopts the read priority scheduling and the write requests are delayed until reaching the HighWaterMark [39, 40]. On average, we observe that the write latency of PCM-R and PCM-2R is only 67% and 63% compared with the baseline, respectively. The write latency of silentstore is 78% of the baseline. In comparison, the write latency of bit-mapping + SPCM is 68% compared with the baseline and PCM-2R outperforms the state-of-the-art PCM write scheme greatly. Moreover, our designs achieve better performance improvement with the memory intensiveness increases with more obvious modified-byte distribution patterns (among the workloads canneal, facesim, astar, bwaves, leslie3d, and zeusmp). However, some workloads show less performance improvement, such as dedup, ferret, vips, x264, bwaves, and bzips, since the modified-byte distribution is almost balanced in these benchmarks and our proposed schemes are less effective for the balanced distribution.Figure 16
Write latency.


### 4.4. IPC Speedup

The metric IPC speedup is used to evaluate the system performance of different schemes. IPC speedup is equal to the IPC of existing scheme divided by the IPC of baseline. The results of IPC speedup are shown in Figure17. PCM-2R has the best performance in general and shows about 1.9X speedup compared with the baseline. PCM-R presents 1.8X speedup and silentstore scheme indicates 1.6X speedup compared with the DCW on average. In comparison, bit-mapping + SPCM shows 1.7X speedup compared with baseline on average. Our proposed schemes achieve IPC improvement due to the reduction of memory write service time. Since we adopt O3 (out-of-order) CPUs, more instructions are sent and finished and the IPC improves subsequently. Similar to the results of write latency, two memory-intensive workloads show significant speedup (more than 3.5X, as shown in workloads bwaves, leslie3d, and libquantum). It is demonstrated that the reduction of read and write latency can lead to significant increase in system performance and our designs are effective.Figure 17
IPC speedup.


### 4.5. Program Running Time

The program running time is also an important indicator of the performance of the main memory system. The faster the load/store instructions finish, the better problem running time can get. Our designs can get program running time reduction since both reshaping and remapping methods accelerate the write operation and the service time is reduced, as shown in Figure18. Silentstore scheme and PCM-R schemes show 32% and 38% running time reduction compared with the baseline scheme, respectively. In comparison, bit-mapping + SPCM and PCM-2R show 35% and 41% running time reduction compared with the DCW scheme, respectively. PCM-2R outperforms silentstore scheme, bit-mapping + SPCM, and PCM-R by 9%, 6%, and 3%, respectively.Figure 18
Running time.


### 4.6. Memory Throughput

Write throughput is one of the most important metrics of the PCM-based main memory. Figure19 shows the results of throughput normalized to the baseline. Our mechanisms show significant throughput improvement in all workloads. On average, the throughput improvement under PCM-R scheme is 2.0X. Four workloads show more than 2.0X improvement (canneal, bwaves, libquantum, and zeusmp). When employing the PCM-2R scheme, the throughput is increased significantly. The average improvement is 2.2X and 5 out of 14 workloads show more than 2.1X improvement over the baseline (canneal, ferret, bwaves, libquantum, and zeusmp) since their modified-byte distribution is very close to the given three patterns in this work. In comparison, silentstore and bit-mapping + SPCM schemes can get 1.8X and 1.9X throughput enhancement on average, respectively.Figure 19
Throughput enhancement.


### 4.7. Energy Consumption

Although our mechanisms do not reduce the data amount directly, the memory energy consumption is reduced since data reshaping and remapping significantly decrease the number of write operations and reduce the overall write service time. Figure20 illustrates the energy consumption of compared schemes normalized to the baseline. PCM-R reduces energy consumption by up to 49% for all 14 workloads on average. PCM-2R decreases the energy consumption by 52% on average. The compared silentstore and bit-mapping + SPCM schemes can get 43% and 48% energy consumption on average compared with the baseline, respectively.Figure 20
Energy consumption.


### 4.8. Sensitivity Analysis

DRAM buffer is widely used in PCM-based main memory system to hide the long write latency and a suitable DRAM cache size can effectively improve the performance of PCM main memory system. In this section, we try to explore the design space of proposed PCM-2R schemes with various DRAM cache and evaluate the impact of different LLC DRAM size on the effectiveness and efficiency of PCM-2R. Figure21(a) shows the running time reduction (compared with the baseline) sensitivity to varying LLC DRAM size (32 MB, 64 MB, and up to 128 MB). On average, the running time reduction decreases from 37% to 25% with the LLC size increases. For some workloads, the degradation is quite obvious, such as canneal, facesim, freqmine, vips, x264, gobmk, leslie3d, and zeusmp. This is because larger LLC DRAM cache absorbs most of the main memory access requests; the service time of different schemes is almost the same in DRAM cache. For some workloads, the effect of different DRAM size is small, such as bzip2 and libquantum. Moreover, dedup shows performance improvement with DRAM size increases. This is because the memory accesses patterns are strengthened and have more expected patterns shown in Table 2. The experimental results of IPC speedup sensitivity to different DRAM cache size are similar to running time reduction, as shown in Figure 21(b). The IPC speedup results are also compared with the baseline. On average, the IPC speedup is 1.9X, 1.8X, and 1.6X when DRAM is 32 MB, 64 MB, and 128 MB, respectively. In general, IPC speedup decreases in most workloads expect dedup. For freqmine, astar, and gobmk, there are almost no memory accesses when DRAM is set to 128 MB and IPC speedup is 1X compared with the baseline, i.e., no improvement. In summary, our proposed schemes are still effective and efficient with varying LLC DRAM size.Figure 21
Sensitivity analysis: (a) running time reduction sensitivity; (b) IPC speedup sensitivity.


## 4.1. Experimental Environment

In order to evaluate the efficiency and effectiveness of the proposed schemes, we use the full-system GEM5 simulator [33] combined with the cycle-level NVMain module [34], which is a cycle accurate memory simulator for NVMs. We use the same configuration parameters from prior similar studies [35] and the evaluation parameters used in this study are shown in Table 3. In general, we simulate a 4-core system with ALPHA-like processors and model the whole memory hierarchy including L1, L2, last level DRAM caches, and 16 GB MLC PCM main memory. DRAM cache is adopted for improving the performance and mitigating the endurance of PCM-based main memory [36–38]. The main memory has 4 channels in total, with 2 ranks per channel and 8 banks per rank. The memory controller adopts the FRFCFS-WQF scheduling proposed by previous art [39, 40]. The workloads used in this study are summarized in Table 4. We adopt a set of multithreaded and multiprogrammed programs in PARSEC 2.0 [41] and SPEC 2006 suites [42], respectively. All of these benchmarks involve intensive and nonintensive memory accesses and have different memory Read Per Kilo Instructions (RPKI) and memory Write Per Kilo Instructions (WPKI) rates. In addition, five billion instructions are simulated for each workload after fast-forwarding one billion instructions considering program initialization and cache warm-up. In general, we use the DCW [31] as the baseline and combine the DCW with the silentstore scheme [43, 44] as the comparison. All the comparison techniques have the same technology node (22 nm) and simulation configurations. PCM-R combines proposed reshaping method with DCW while PCM-2R puts reshaping and remapping all together. We also try to compare our proposed designs with state-of-the-art bit-mapping [3] and SPCM designs [19, 20]. Specifically, the row buffer hit/miss is considered in the evaluation and the time overhead caused by data reshaping, reconstruction, and remapping is also involved in our evaluation.Table 3
Parameters of simulation.
ParameterValueProcessor4 cores, 2 GHz, ALPHA-likeL1 cache32 kB I-cache, 32 kB D-cache, 2 cycles’ access latencyL2 cache8-way, 2 MB private per core, 20 cycles’ access latencyL3 DRAM cache16-way, 32 MB shared, 50 cycles’ access latencyCache organization64-byte line size, write back, LRU replacementMemory controllerFRFCFS-WQF scheduling, 32-entry R/W queuesMemory organization16 GB MLC PCM, 4 channels with 2 ranks each, 8 banks per rankMemory timingRead 250 ns, write 2μs (“00” to “10,” 1.0X)PCM organization4 chips per DIMM, the size of write unit is 64 bits (32 2 bit MLC cells)Table 4
Characteristics of the evaluated workloads.
WorkloadRPKIWPKISPEC 2006, multiprogrammedAstar: 4 copies of astar0.530.38bwaves: 4 copies of bwaves11.195.67bzip2: 4 copies of bzip28.814.64gobmk: 4 copies of gobmk0.140.13leslie3d: 4 copies of leslie3d7.112.59libquantum: 4 copies of libquantum10.956.93zeusmp: 4 copies of zeusmp13.563.18PARSEC 2.0, multithreadedcanneal1.881.31dedup0.350.33facesim0.860.71ferret0.230.15freqmine0.320.33vips1.271.18x2640.180.20


## 4.2. Memory Read Latency

The write latency reduction significantly benefits the read latency due to the wait time reduction of read requests [45]. Although PCM-2R does not improve the read performance directly, it improves the write parallelism and read latency benefits from the write service time reduction. Figure 15 shows the experimental results of read latency. On average, PCM-R shows 48% read latency reduction compared with the baseline. In comparison, silentstore and bit-mapping + SPCM show, respectively, 40% and 45% read latency improvement and PCM-2R shows almost 51% read latency reduction by leveraging data remapping scheme. One workload shows significant performance improvement (facesim) and outperforms the state-of-the-art scheme by 25% more read latency reduction because the modified-byte pattern is obvious and the amount is balanced after data reshaping and remapping methods. It is noteworthy that our schemes are not effective in some workloads with balanced modified-byte distribution, such as dedup, bzip2, and x264. In other words, PCM-R and PCM-2R are not effective for all benchmarks and efficient for some workloads with special unbalanced distribution patterns.Figure 15
Read latency.


## 4.3. Memory Write Latency

Figure16 illustrates the comparison of memory write latency of different PCM write schemes. The improvement of write latency is not significant compared with the increase of read latency since the memory controller we used in this study adopts the read priority scheduling and the write requests are delayed until reaching the HighWaterMark [39, 40]. On average, we observe that the write latency of PCM-R and PCM-2R is only 67% and 63% compared with the baseline, respectively. The write latency of silentstore is 78% of the baseline. In comparison, the write latency of bit-mapping + SPCM is 68% compared with the baseline and PCM-2R outperforms the state-of-the-art PCM write scheme greatly. Moreover, our designs achieve better performance improvement with the memory intensiveness increases with more obvious modified-byte distribution patterns (among the workloads canneal, facesim, astar, bwaves, leslie3d, and zeusmp). However, some workloads show less performance improvement, such as dedup, ferret, vips, x264, bwaves, and bzips, since the modified-byte distribution is almost balanced in these benchmarks and our proposed schemes are less effective for the balanced distribution.Figure 16
Write latency.


## 4.4. IPC Speedup

The metric IPC speedup is used to evaluate the system performance of different schemes. IPC speedup is equal to the IPC of existing scheme divided by the IPC of baseline. The results of IPC speedup are shown in Figure17. PCM-2R has the best performance in general and shows about 1.9X speedup compared with the baseline. PCM-R presents 1.8X speedup and silentstore scheme indicates 1.6X speedup compared with the DCW on average. In comparison, bit-mapping + SPCM shows 1.7X speedup compared with baseline on average. Our proposed schemes achieve IPC improvement due to the reduction of memory write service time. Since we adopt O3 (out-of-order) CPUs, more instructions are sent and finished and the IPC improves subsequently. Similar to the results of write latency, two memory-intensive workloads show significant speedup (more than 3.5X, as shown in workloads bwaves, leslie3d, and libquantum). It is demonstrated that the reduction of read and write latency can lead to significant increase in system performance and our designs are effective.Figure 17
IPC speedup.


## 4.5. Program Running Time

The program running time is also an important indicator of the performance of the main memory system. The faster the load/store instructions finish, the better problem running time can get. Our designs can get program running time reduction since both reshaping and remapping methods accelerate the write operation and the service time is reduced, as shown in Figure18. Silentstore scheme and PCM-R schemes show 32% and 38% running time reduction compared with the baseline scheme, respectively. In comparison, bit-mapping + SPCM and PCM-2R show 35% and 41% running time reduction compared with the DCW scheme, respectively. PCM-2R outperforms silentstore scheme, bit-mapping + SPCM, and PCM-R by 9%, 6%, and 3%, respectively.Figure 18
Running time.


## 4.6. Memory Throughput

Write throughput is one of the most important metrics of the PCM-based main memory. Figure19 shows the results of throughput normalized to the baseline. Our mechanisms show significant throughput improvement in all workloads. On average, the throughput improvement under PCM-R scheme is 2.0X. Four workloads show more than 2.0X improvement (canneal, bwaves, libquantum, and zeusmp). When employing the PCM-2R scheme, the throughput is increased significantly. The average improvement is 2.2X and 5 out of 14 workloads show more than 2.1X improvement over the baseline (canneal, ferret, bwaves, libquantum, and zeusmp) since their modified-byte distribution is very close to the given three patterns in this work. In comparison, silentstore and bit-mapping + SPCM schemes can get 1.8X and 1.9X throughput enhancement on average, respectively.Figure 19
Throughput enhancement.


## 4.7. Energy Consumption

Although our mechanisms do not reduce the data amount directly, the memory energy consumption is reduced since data reshaping and remapping significantly decrease the number of write operations and reduce the overall write service time. Figure20 illustrates the energy consumption of compared schemes normalized to the baseline. PCM-R reduces energy consumption by up to 49% for all 14 workloads on average. PCM-2R decreases the energy consumption by 52% on average. The compared silentstore and bit-mapping + SPCM schemes can get 43% and 48% energy consumption on average compared with the baseline, respectively.Figure 20
Energy consumption.


## 4.8. Sensitivity Analysis

DRAM buffer is widely used in PCM-based main memory system to hide the long write latency and a suitable DRAM cache size can effectively improve the performance of PCM main memory system. In this section, we try to explore the design space of proposed PCM-2R schemes with various DRAM cache and evaluate the impact of different LLC DRAM size on the effectiveness and efficiency of PCM-2R. Figure21(a) shows the running time reduction (compared with the baseline) sensitivity to varying LLC DRAM size (32 MB, 64 MB, and up to 128 MB). On average, the running time reduction decreases from 37% to 25% with the LLC size increases. For some workloads, the degradation is quite obvious, such as canneal, facesim, freqmine, vips, x264, gobmk, leslie3d, and zeusmp. This is because larger LLC DRAM cache absorbs most of the main memory access requests; the service time of different schemes is almost the same in DRAM cache. For some workloads, the effect of different DRAM size is small, such as bzip2 and libquantum. Moreover, dedup shows performance improvement with DRAM size increases. This is because the memory accesses patterns are strengthened and have more expected patterns shown in Table 2. The experimental results of IPC speedup sensitivity to different DRAM cache size are similar to running time reduction, as shown in Figure 21(b). The IPC speedup results are also compared with the baseline. On average, the IPC speedup is 1.9X, 1.8X, and 1.6X when DRAM is 32 MB, 64 MB, and 128 MB, respectively. In general, IPC speedup decreases in most workloads expect dedup. For freqmine, astar, and gobmk, there are almost no memory accesses when DRAM is set to 128 MB and IPC speedup is 1X compared with the baseline, i.e., no improvement. In summary, our proposed schemes are still effective and efficient with varying LLC DRAM size.Figure 21
Sensitivity analysis: (a) running time reduction sensitivity; (b) IPC speedup sensitivity.


## 5. Related Work

In this section, we focus on the recent research related to this work. Du et al. [14] observed that the bit distribution is quite unbalanced among PCM groups and proposed a low-overhead D-XOR bit-mapping strategy to balance the distribution of modified bits. The write service time is reduced due to the balanced workload distribution. Jiang et al. [46] proposed a mechanism to decouple the 2-bit STT-RAM cells and improve the performance by leveraging the cell asymmetries. The cache lines are divided into RFWS lines (read fast and write slow) and RSWF lines (read low and write fast). Using efficiency prediction method, the data are placed into the right position and the write time is reduced. Hoseinzadeh et al. [19, 20] tried to improve the critical read latency by leveraging the asymmetries between the MSB and LSB bits of MLC cells. By decoupling the cells, the memory is divided into two parts. One part has almost the same read latency compared with SLC cell and the other remains the same as the MLC cell. The authors also proposed a pairing write queue to address the write problem raised by the line stripping method. Similar to this mechanism, Yoon et al. [28] proposed data mapping and buffering techniques to apply the key ideas to MLC PCM including cells decoupling and data prediction. The difference is that this strategy takes advantage of the row buffer space and divides the row buffer in two parts for RFWS regions and RSWF regions. Yue and Zhu [27] present a different decoupling method compared with prior art. This strategy divides one cache line into two half lines rather than dividing the whole memory into two different-type lines. Based on the key insights that the read operation usually involved the data of the first half of a cache line, the proposed bit-mapping scheme reduces the critical read latency. Zhang et al. [35] proposed a PCM write scheme named TriState-SET to improve the poor PCM write performance by choosing the fastest state transition among all possible transitions. Palangappa and Mohanram [47, 48] proposed compression expansion coding by combining the data compression algorithm and the energy and latency asymmetries in MLC write. Qureshi et al. [6] proposed write cancellation method by cancelling the currently executing write operation and then performing the critical read request first to improve the overall system performance. The authors also modeled the MLC PCM write and proposed an extended model for iterative write. Based on the proposed modeling, write pausing tries to pause the ongoing write request and reperform it after all read requests are issued. Jiang et al. [49] proposed a write truncation method to improve the write operations in MLC PCM. MLC PCM write involves multiple write iterations, which results in long write time. Finding that there is a long tail effect in the write iterations, write truncation method stops the iterations after reaching a certain number of times. In order to solve the problem of data errors, ECC methods are introduced. Jiang et al. [39] presented FPB method to improve the write parallelism of MLC PCM. The authors assumed and proved that the required power is reduced with the number of iterations increases. By implementing a fine-grained power budgeting method, two writes can be served in parallel under constraints and the service time is reduced. Huang and Hua [50] proposed a write-friendly and fast-recovery scheme for security metadata in nonvolatile memories by instantly persisting the modifications of security metadata without extra memory writes. Zhang et al. [51, 52] proposed two microarchitectural designs to enhance the write operations of nonvolatile memories. These methods can also be applied in conjunction with our techniques.


## 6. Conclusion

In this paper, we proposed PCM-2R, a novel MLC PCM write scheme based on the key observations that the modified-byte distribution is quite unbalanced among all chips and shows some unique patterns. Keeping these special patterns in mind, we try to boost the write by reshaping the data to distribute the cache line block among all chips to avoid unbalanced distribution and then remapping modified bytes to fast region after decoupling MLC PCM cells considering the asymmetries in state transitions. Evaluation results of read latency, write latency, IPC improvement, and running time reduction of PARSEC 2.0 and SPEC 2006 workloads demonstrate the effectiveness and efficiency of PCM-2R. Our results show that PCM-2R reduces 51% read latency, 37% write latency, 41% running time, and 52% energy and gets 1.9X IPC and 2.2X throughput improvement on average compared with the baseline scheme. The experimental results also show that PCM-2R outperforms the state-of-the-art bit-mapping and SPCM schemes by 4.7% and 3.6% more read latency and write latency, 0.2X more IPC improvement, 2.8% more running time reduction, 0.2X more throughput improvement, and 3.5% more energy reduction.Actually, there is an increasing demand for large capacity memory to run modern data-intensive applications, such as IOT, artificial intelligence algorithms, big data analytics, and so on. The proposed PCM-2R design can offer storage-class memory; therefore it is well suited for these use-case scenarios. As PCM has been partly commercially used, e.g., Intel 3D X-Point, we believe PCM-2R can be used in these real world use-case scenarios in near future.



---

*Source: 9552517-2022-07-16.xml*