Wrapping Operations for Atomicity and Durability: A Position Paper on How to Simplify NVM Programming for Extreme Performance

Problem Statement: The ability to access emerging high capacity storage class memories (SCM) [3] in fine-grained units is a tremendous advantage for the growing set of applications involving graph traversals, pointer chasing and columnar databases. SCM provides persistence at instruction granularity in stream with computing, providing new opportunities and challenges for structuring data-centric computation. Rethinking application design to benefit from a persistent memory tier [6] has the potential to deliver ordersof-magnitude gains as shown by in-memory data processing software such as VoltDB[1], SAP HANA[2], and IBM DB2 [8]. The use of SCM also enables energy-efficient handling of vast amounts of data that are accessed infrequently. For application development requiring quick turnaround, this unification of storage and memory into a single directlyaccessed persistent storage memory tier is a mixed blessing. It pushes upon developers the burden of ensuring that SCM stores are ordered correctly, flushed from caches, and if interrupted by a crash, do not leave objects in inconsistent states or cause corruption of metadata. While memory fences let a developer control the order in which stores from one CPU are made visible to others, the default memory store semantics supported by processors make no guarantees of when the value being written by a store instruction will actually be reflected in memory banks. Cache flush instructions guarantee update of the backing memory soon; but the writes are also done asynchronously from volatile store buffers, and so, maintain the risk of losing an update in the face of an uncontrolled machine restart. To address this gap, processor manufacturers will need to provide instructions to wait until a designated store has drained at the controller; we refer to such instructions generically as PSYNC (persistent sync) in this paper. How to use persistent syncs, memory fences, and cache flushes to ensure properly ordered, all-ornothing updates of a sequence of stores to arbitrary addresses is a challenge that software designers will need to confront to benefit from the instruction-granular persistence of SCM. This position paper advocates a software library approach to liberate programmers from the burden of explicit state management in SCM. Even if a programmer were to flush caches synchronously after each store in a train of correlated writes (by combining flushing with PSYNC), a machine failure that interrupts the sequence can still result in an unrecoverable system unless the state at which the sequence was interrupted is precisely knowable at the time of recovery. A journal or other metadata also in persistent memory needs to track the progress of persistent memory writes and make it available in order to effect recovery – in a manner similar to log-based recovery in databases [5]. A subtle but important aspect of any solution is the need to deal with the processor cache hierarchy. The extreme degree to which high performance applications are vulnerable to memory stalls when running on a modern superscalar CPU means that a good solution must exploit caches as efficiently as possible. However, since the cache subsystem independently evicts cache lines and writes their values to memory, these uncontrolled evictions complicate solutions by causing partial and out-of-order updates to SCM. When combined with the need to also flush metadata writes, it is difficult to maintain high performance when out-of-order execution is continuously impeded by flushes, PSYNCs, and memory fences. SoftWrAP Approach. The SoftWrAP (for Software based Write Aside Persistence) approach we propose is a software derivative of an earlier hardware solution WrAP [4]. In WrAP [4] a controller on the SCM maintains a log of updates and handles spurious cache evictions with an overflow structure called a victim cache. The SoftWrAP approach provides a programmer with wrap library calls to demarcate a region of code (as shown Figure 1) from within which stores (called wrapped stores) are to be written to SCM in an all-or-nothing manner. When an SCM location X is updated for the first time it is associated with an address X' in DRAM via a hash map which redirects all wrapped accesses from X to X'. Cache eviction of X’ does not change X, which is updated in a controlled way by the run time. Programmer Annotated Atomic Region in SoftWrAP