Patent Publication Number: US-11042296-B1

Title: System and method of handling journal space in a storage cluster with multiple delta log instances

Description:
BACKGROUND 
     Clustered storage systems (also referred to herein as “storage clusters”) employ various techniques and/or methodologies to protect and/or distribute electronic data and/or metadata. In response to receipt of a write input/output (IO) request for a storage object (e.g., a volume, a logical unit (LU), a file system) from a host computer, a data storage processor (also referred to herein as a “storage node”) of a storage cluster may write pending changes (also referred to herein as “deltas” or “delta updates”) associated with a metadata page to a journal in local memory before storing the metadata delta updates to a storage array. Once the metadata delta updates have been written to the journal, the storage node sends an acknowledgement message to the host computer that issued the write IO request. The storage node subsequently stores the delta updates to the associated metadata page in the storage array. 
     SUMMARY 
     In a storage cluster, a storage node can write metadata delta updates to both a metadata delta log in volatile memory and a journal in persistent memory (collectively referred to herein as the “delta log infrastructure”). The metadata delta log in the volatile memory can include a first set of data containers and a second set of data containers. Each data container of the first and second sets can be associated with a respective metadata page. A storage node of the storage cluster can write delta updates associated with a metadata page to the metadata delta log in the volatile memory, filling one or more of the first set of data containers with the metadata delta updates. The first set of data containers receiving the metadata delta updates can be designated as the “active” set of data containers, and the second set of data containers can be designated as the “de-staging” set of data containers. The storage node can also write copies of metadata delta updates (also referred to herein as “raw delta updates”) to the journal (also referred to herein as the “raw delta log”) in the persistent memory. 
     Once one or more of the active set of data containers have been filled, the “active” and “de-staging” designations of the respective first and second sets of data containers can be switched. In other words, the “active” designation of the first set of data containers can be switched to the “de-staging” designation, and the “de-staging” designation of the second set of data containers can be switched to the “active” designation. In response to switching the “active” and “de-staging” designations of the respective first and second sets of data containers, additional delta updates associated with a metadata page can be written to fill one or more of the current designated “active” set of data containers. Further, the metadata delta updates written to the current designated “de-staging” set of data containers can be stored to a storage array, and allocated space in the raw delta log storing the raw delta updates can be reclaimed. 
     In such a storage cluster, a storage node can write delta updates for different types of metadata to the delta log infrastructure. For example, the different metadata types can include reference count metadata that specify reference counts to storage objects in the storage cluster, mapping metadata such as logical-to-virtual and/or virtual-to-physical mapping metadata, and/or any other suitable metadata type(s). Because different metadata types can have different amortization rates, different metadata size parameters, and so on, the delta log infrastructure can be optimized for handling the different metadata types. For example, a metadata delta log instance can be provided and optimized for each metadata type. Further, each metadata delta log instance can have its own associated raw delta log. 
     However, such a configuration of the delta log infrastructure has shortcomings due at least in part to increased write amplification and/or increased recovery complexity. For example, in such a configuration, one or more IO transactions can include writing raw delta updates for multiple metadata types to multiple raw delta logs. Because raw delta updates written to multiple raw delta logs in one or more IO transactions can be dependent upon one another, recovery mechanisms for recovering the delta log infrastructure to a consistent state can be highly complex. At least some of theses shortcomings can be addressed by alternatively configuring the delta log infrastructure to include a single raw delta log for use with multiple delta log instances. However, such an alternative configuration of the delta log infrastructure also has shortcomings due to difficulties in safely reclaiming allocated space in the single raw delta log. 
     Techniques are disclosed herein for handling journal space in a storage cluster with multiple delta log instances. The storage cluster can include a plurality of storage nodes, in which each storage node can include multiple instances of metadata delta logs in volatile memory and a single journal (“raw delta log”) in persistent memory. Each metadata delta log can include a first set of data containers initially designated as the “active” set of data containers, and a second set of data containers initially designated as the “de-staging” set of data containers. Each metadata delta log can also be optimized for handling a particular type of metadata. In the disclosed techniques, a storage node can receive a plurality of write IO requests for one or more storage objects from one or more host computers. The plurality of write IO requests can direct the storage node to write pending changes (“deltas,” “delta updates”) for one or more different metadata types to the respective metadata delta logs. 
     In response to receipt of the plurality of write IO requests, the storage node can write delta updates for each respective metadata type to a metadata delta log optimized for handling the respective metadata type, filling one or more of the current designated “active” set of data containers with the metadata delta updates. The storage node can also write copies of metadata delta updates (“raw delta updates”) to the single raw delta log to persist them for use in recovering the respective metadata delta logs to a consistent state in the event of a disaster, data loss, and/or data corruption. Once one or more of the active set of data containers of the metadata delta log have been filled, the storage node can switch the designation of the current designated “active” set of data containers from “active” to “de-staging,” as well as switch the designation of the current designated “de-staging” set of data containers from “de-staging” to “active.” Further, once the “active” and “de-staging” designations have been switched, the storage node can write a bookmark to the raw delta log in the persistent memory, as well as to a bookmark list in the volatile memory. In certain implementations, the bookmark can be configured as a record having several attributes, including a “type” attribute specifying the respective metadata type handled by the metadata delta log, an “identifier” attribute specifying the current designated “active” set of data containers included in the metadata delta log, and an “offset” attribute specifying the current allocation offset in the raw delta log. 
     Once “active” and “de-staging” designations of one or more sets of data containers have been switched for the respective metadata delta logs, and bookmarks have been written to each of the raw delta log and the bookmark list, the storage node can initiate de-staging operations to be performed in a background process. The storage node can then determine whether at least a first de-staging operation for delta updates of a first metadata type and a second de-staging operation for delta updates of a second metadata type have been performed in the background process. If the storage node determines that the first de-staging operation has been performed before the second de-staging operation, then the storage node can remove a first bookmark specifying the first metadata type from the bookmark list, checking to see whether the first bookmark was removed from a tail of the bookmark list. If the first bookmark was indeed removed from the tail of the bookmark list (i.e., the first bookmark was the “oldest” bookmark in the bookmark list), then the storage node can perform a memory space reclaim operation to reclaim the space between a tail of the raw delta log and the first bookmark written to the raw delta log. 
     If the storage node determines that the second de-staging operation has been performed before the first de-staging operation, then the storage node can remove a second bookmark specifying the second metadata type from the bookmark list, checking to see whether the second bookmark was removed from the tail of the bookmark list. If the second bookmark was not removed from the tail of the bookmark list (i.e., the first bookmark remains at the tail of the bookmark list), then the storage node can defer performing a memory space reclaim operation on the raw delta log until the first de-staging operation for delta updates of the first metadata type has been performed in the background process. In this way, memory space reclamation can be more safely performed in a storage node that includes multiple instances of metadata delta logs and a single raw delta log. 
     In certain embodiments, a method of handling journal space in a storage cluster with multiple delta log instances includes writing delta updates of a first metadata type to both a respective set of data containers in a first delta log instance and a raw delta log, and, in response to one or more of the respective set of data containers in the first delta log instance being filled with at least some of the delta updates, writing a first bookmark for the first metadata type to both the raw delta log and a bookmark list. The method further includes, in response to a first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to a storage array: if the first bookmark is determined to be an oldest bookmark in the bookmark list, reclaiming at least first space between a tail of the raw delta log and the first bookmark written to the raw delta log; and if the first bookmark is determined not to be the oldest bookmark in the bookmark list, deferring the reclaiming of the first space in the raw delta log. 
     In certain arrangements, the method further includes, in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array, removing the first bookmark for the first metadata type from the bookmark list. 
     In certain arrangements, the method further includes writing delta updates of a second metadata type to both a respective set of data containers in a second delta log instance and the raw delta log, and, in response to one or more of the respective set of data containers in the second delta log instance being filled with at least some of the delta updates, writing a second bookmark for the second metadata type to both the raw delta log and the bookmark list. 
     In certain arrangements, the method further includes, in response to a second de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the second delta log instance to the storage array: determining that the second bookmark is not the oldest bookmark in the bookmark list; and, deferring reclaiming second space in the raw delta log. 
     In certain arrangements, the method further includes, in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array: determining that the first bookmark is the oldest bookmark in the bookmark list; and reclaiming at least the first space and the second space between the tail of the raw delta log and the second bookmark written to the raw delta log. 
     In certain arrangements, the method further includes setting a threshold specifying a minimum allowed free space in the raw delta log. 
     In certain arrangements, the method further includes determining that free space in the raw delta log is below the set threshold, and forcing at least the first de-staging operation to be performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array. 
     In certain embodiments, a storage node of a storage cluster includes a volatile memory including a first delta log instance and a bookmark list, the first delta log instance having a respective set of data containers; a persistent memory including a raw delta log; and, processing circuitry configured to execute program instructions to write delta updates of a first metadata type to both the respective set of data containers in the first delta log instance and the raw delta log, and, in response to one or more of the respective set of data containers in the first delta log instance being filled with at least some of the delta updates, to write a first bookmark for the first metadata type to both the raw delta log and the bookmark list. The processing circuitry is further configured to execute the program instructions, in response to a first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to a storage array: if the first bookmark is determined to be an oldest bookmark in the bookmark list, to reclaim at least first space between a tail of the raw delta log and the first bookmark written to the raw delta log; and, if the first bookmark is determined not to be the oldest bookmark in the bookmark list, to defer the reclaiming of the first space in the raw delta log. 
     In certain arrangements, the processing circuitry is further configured to execute the program instructions, in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array, to remove the first bookmark for the first metadata type from the bookmark list. 
     In certain arrangements, the processing circuitry is further configured to execute the program instructions to write delta updates of a second metadata type to both a respective set of data containers in a second delta log instance and the raw delta log, and, in response to one or more of the respective set of data containers in the second delta log instance being filled with at least some of the delta updates, to write a second bookmark for the second metadata type to both the raw delta log and the bookmark list. 
     In certain arrangements, the processing circuitry is further configured to execute the program instructions, in response to a second de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the second delta log instance to the storage array: to determine that the second bookmark is not the oldest bookmark in the bookmark list; and, to defer reclaiming second space in the raw delta log. 
     In certain arrangements, the processing circuitry is further configured to execute the program instructions, in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array: to determine that the first bookmark is the oldest bookmark in the bookmark list; and, to reclaim at least the first space and the second space between the tail of the raw delta log and the second bookmark written to the raw delta log. 
     In certain arrangements, the raw delta log is configured as a ring buffer having a head and a tail, and the processing circuitry is further configured to execute the program instructions to write the delta updates of the first metadata type at the head of the raw delta log. 
     In certain arrangements, the processing circuitry is further configured to execute the program instructions to write the first bookmark for the first metadata type at the head of the raw delta log. 
     In certain arrangements, the bookmark list is configured as a linked list having a head and a tail, and the processing circuitry is further configured to execute the program instructions to write the first bookmark for the first metadata type at the head of the linked list. 
     In certain arrangements, the processing circuitry is further configured to execute the program instructions to set a threshold specifying a minimum allowed free space in the raw delta log. 
     In certain arrangements, the processing circuitry is further configured to execute the program instructions to determine that free space in the raw delta log is below the set threshold, and to force at least the first de-staging operation to be performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array. 
     In certain embodiments, a computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method that includes writing delta updates of a first metadata type to both a respective set of data containers in a first delta log instance and a raw delta log, and, in response to one or more of the respective set of data containers in the first delta log instance being filled with at least some of the delta updates, writing a first bookmark for the first metadata type to both the raw delta log and a bookmark list. The method further includes, in response to a first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to a storage array: if the first bookmark is determined to be an oldest bookmark in the bookmark list, reclaiming at least first space between a tail of the raw delta log and the first bookmark written to the raw delta log; and, if the first bookmark is determined not to be the oldest bookmark in the bookmark list, deferring the reclaiming of the first space in the raw delta log. 
     Other features, functions, and aspects of the present disclosure will be evident from the Detailed Description that follows. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The foregoing and other objects, features, and advantages will be apparent from the following description of particular embodiments of the present disclosure, as illustrated in the accompanying drawings, in which like reference characters refer to the same parts throughout the different views. 
         FIG. 1  is a block diagram of an exemplary data storage environment, in which techniques can be practiced for handling journal space in a storage cluster with multiple delta log instances; 
         FIG. 2  is a block diagram of volatile memory, persistent memory, and storage array components of a storage node in the data storage environment of  FIG. 1 ; 
         FIGS. 3 a -3 g    are block diagrams describing an illustrative example of handling journal space in a storage cluster with multiple delta log instances; and 
         FIG. 4  is a flow diagram of an exemplary method of handling journal space in a storage cluster with multiple delta log instances. 
     
    
    
     DETAILED DESCRIPTION 
     Techniques are disclosed herein for handling journal space in a storage cluster with multiple delta log instances. The disclosed techniques can include writing delta updates for a respective type of metadata to a current designated “active” set of data containers in a metadata delta log instance and raw delta updates for respective metadata types to a raw delta log, switching a designation of the current designated “active” set of data containers from “active” to “de-staging” once one or more of the current designated “active” set of data containers has been filled, and writing a bookmark for the respective metadata type to both the raw delta log and a bookmark list. The disclosed techniques can further include determining that a de-staging operation has been performed in a background process for writing the delta updates from the current designated “de-staging” set of data containers to a storage array, removing the bookmark for the respective metadata type from the bookmark list, determining that the removed bookmark was the “oldest” bookmark in the list, and reclaiming the space between a tail of the raw delta log and the bookmark written to the raw delta log. In this way, memory space reclamation can be more safely performed in a storage node that includes multiple delta log instances and a single raw delta log. 
       FIG. 1  depicts an illustrative embodiment of an exemplary data storage environment  100 , in which techniques can be practiced for handling journal space in a storage cluster with multiple delta log instances. As shown in  FIG. 1 , the data storage environment  100  can include a plurality of host computers  102 . 1 ,  102 . 2 , . . . ,  102 . n , at least one data storage processor  104  (also referred to herein as a “storage node”), a storage array  106 , and a communications medium  103  that includes at least one network  108 . For example, each of the plurality of host computers  102 . 1 , . . . ,  102 . n  can be configured as a web server computer, a file server computer, an email server computer, an enterprise server computer, and/or any other suitable client/server computer or computerized device. The plurality of host computers  102 . 1 , . . . ,  102 . n  can be configured to provide, over the network  108 , storage input/output (IO) requests (e.g., small computer system interface (SCSI) commands, network file system (NFS) commands) to the storage node  104 . Such storage IO requests (e.g., write IO requests, read IO requests) can direct the storage node  104  to write and/or read data blocks, data pages, data files, or any other suitable data elements to/from volumes (VOLs), logical units (LUs), file systems, and/or any other suitable storage targets, such as a plurality of storage targets  118 . 1 ,  118 . 2 , . . . ,  118 . m  maintained in the storage array  106 . 
     The communications medium  103  can be configured to interconnect the plurality of host computers  102 . 1 , . . . ,  102 . n  with the storage node  104  to enable them to communicate and exchange data and/or control signaling. As shown in  FIG. 1 , the communications medium  103  can be illustrated as a “cloud” to represent different communications topologies such as a backbone topology, a hub-and-spoke topology, a loop topology, an irregular topology, and so on, or any suitable combination thereof. As such, the communications medium  103  can include copper-based data communications devices and cabling, fiber optic devices and cabling, wireless devices, and so on, or any suitable combination thereof. The communications medium  103  can be further configured to support storage area network (SAN)-based communications, network attached storage (NAS)-based communications, local area network (LAN)-based communications, metropolitan area network (MAN)-based communications, wide area network (WAN)-based communications, wireless communications, distributed infrastructure communications, and/or any other suitable communications. 
     The storage node  104  can be connected directly to the storage array  106  or by an optional network infrastructure  110 , which can include an Ethernet network, an InfiniBand network, a fiber channel network, and/or any other suitable network(s). As shown in  FIG. 1 , the storage node  104  can include a communications interface  112 , processing circuitry  114 , and a memory  116 . The communications interface  112  can include one or more of an Ethernet interface, an InfiniBand interface, a fiber channel interface, and/or any other suitable communications interface. The communications interface  112  can further include SCSI target adapters, network interface adapters, and/or any other suitable adapters for converting electronic, optical, and/or wireless signals received over the network  108  to a form suitable for use by the processing circuitry  114 . 
     The memory  116  can include volatile memory such as a random-access memory (RAM)  122  or any other suitable volatile memory, as well as persistent memory such as a nonvolatile random-access memory (NVRAM)  124  or any other suitable persistent memory. The memory  116  can also store a variety of software constructs realized in the form of specialized code and data  128  (e.g., program instructions) that can be executed by the processing circuitry  114  to carry out the techniques and/or methods disclosed herein. The memory  116  can further include an operating system  126 , such as a Linux operating system (OS), a Unix OS, a Windows OS, or any other suitable operating system. 
     The processing circuitry  114  can include one or more physical storage processors and/or engines configured to execute the specialized code and data  128 , as well as data movers, director boards, blades, IO modules, storage drive controllers, switches, and/or any other suitable computer hardware or combination thereof. For example, the processing circuitry  114  can execute the specialized code and data  128  as program instructions out of the memory  116 , process storage IO requests (e.g., write IO requests, read IO requests) issued by the respective host computers  102 . 1 , . . . ,  102 . n , and/or store data and/or metadata to the storage array  106  in the data storage environment  100 , which can be a clustered RAID environment. 
     As shown in  FIG. 1 , the storage array  106  can include a coded target  120 , which can be configured to store coded data for use in regenerating lost or corrupted data on one or more of the storage targets  118 . 1 ,  118 . 2 , . . . ,  118 . m . In certain implementations, the coded target  120  can be stored to a hard disk drive (HDD) configured to store parity data in a RAID array. Alternatively (or in addition), the storage targets  118 . 1 ,  118 . 2 , . . . ,  118 . m  and the coded target  120  can be stored to one or more HDDs, solid state drives (SSDs), flash devices, and/or any other suitable storage device(s). It is noted that the storage node  104  can include a keyboard, a mouse, and/or any other suitable  10  device(s), an uninterruptable power supply (UPS), and/or any other suitable storage node component(s). 
     In the context of the processing circuitry  114  being implemented using one or more processors executing the specialized code and data  128 , a computer program product can be configured to deliver all or a portion of the specialized code and data  128  to the respective processor(s). Such a computer program product can include one or more non-transient computer-readable storage media, such as a magnetic disk, a magnetic tape, a compact disk (CD), a digital versatile disk (DVD), an optical disk, a flash drive, a solid state drive (SSD), a secure digital (SD) chip or device, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and so on. Further, the non-transient computer-readable storage media can be encoded with sets of program instructions for performing, when executed by the respective processor(s), the various techniques and/or methods disclosed herein. 
       FIG. 2  depicts volatile memory, persistent memory, and storage array components that can be included in the storage node  104  of  FIG. 1 . As shown in  FIG. 2 , the volatile memory components can be embodied in the RAM  122 , and include multiple instances of metadata delta logs  204 . 1 ,  204 . 2 , . . . ,  204 . p . The metadata delta log  204 . 1  can include a first set  210  of data buckets (also referred to herein as data “containers”) H 1 , H 2 , H 3 , . . . , H N  (in which “N” is any suitable positive integer) and an associated bloom filter (BF)  206 , as well as a second set  212  of data buckets (“containers”) H 1 , H 2 , H 3 , . . . , H N  and an associated bloom filter (BF)  208 . Likewise, the metadata delta log  204 . 2  can include a first set  218  of data buckets (“containers”) H 1 , H 2 , H 3 , . . . , H N  and an associated bloom filter (BF)  214 , as well as a second set  220  of data buckets (“containers”) H 1 , H 2 , H 3 , . . . , H N  and an associated bloom filter (BF)  216 . In general, the metadata delta log  204 . p  (in which “p” is any suitable positive integer) can include a first set  226  of data buckets (“containers”) H 1 , H 2 , H 3 , . . . , H N  and an associated bloom filter (BF)  222 , as well as a second set  228  of data buckets (“containers”) H 1 , H 2 , H 3 , . . . , H N  and an associated bloom filter (BF)  224 . Each of the first and second sets of data containers H 1 , H 2 , H 3 , . . . , H N  included in the respective metadata delta logs  204 . 1 , . . . ,  204 . p  can be configured to store delta updates to metadata pages. For example, each of the data containers H 1 , H 2 , H 3 , . . . , H N  can be configured as a binary tree (“b-tree”) structure or any other suitable data structure. 
     As described herein, the processing circuitry  114  of the storage node  104  can execute the specialized code and data  128  as program instructions out of the memory  116 , and process storage IO requests (e.g., write IO requests, read IO requests) issued by the respective host computers  102 . 1 , . . . ,  102 . n  to write/read data and/or metadata to/from the storage array  106 . For example, each portion of metadata (e.g., each metadata delta update) can be stored to a metadata page on the storage array  106 . Further, each metadata delta update can specify a change to a metadata page on the storage array  106 . 
     In certain implementations, the storage node  104  can convert a plurality of metadata delta updates into a plurality of delta update tuples  202 , respectively. As shown in  FIG. 2 , each of the plurality of delta update tuples  202  can include several tuple parameters, such as a logical index “Li” of a corresponding metadata page “i,” an entry index “Ei” that refers to a specific delta update entry or offset within the metadata page “i,” a record or delta type “Tp” that defines the size or type “p” of the delta update entry, and a payload or value “V” of the delta update entry. The storage node  104  can also write delta update tuples for particular delta types T 1 , T 2 , . . . , T p  to the metadata delta logs  204 . 1 , . . . ,  204 . p , respectively, in the volatile memory (i.e., the RAM  122 ). For example, the storage node  104  can write delta update tuples “Li:Ei:T 1 :V” to the metadata delta log  204 . 1  (as illustrated by a path  236 ), as well as write delta update tuples “Li:Ei:T 2 :V” to the metadata delta log  204 . 2  (as illustrated by a path  238 ). In general, the storage node  104  can write delta update tuples “Li:Ei:T p :V” to the metadata delta log  204 . p  (as illustrated by a path  240 ), in which “p” is any suitable positive integer. The storage node  104  can subsequently store the delta update tuples written to the metadata delta logs  204 . 1 ,  204 . 2 , . . . ,  204 . p  to a metadata page store  234  (as illustrated by paths  244 ,  246 , . . . ,  248 , respectively) included in the storage array  106 . 
     In certain implementations, the storage node  104  can determine the target data containers H 1 , H 2 , H 3 , . . . , or H N  for the respective delta update tuples Li:Ei:T 1 :V in the first set  210  of data containers included in the metadata delta log  204 . 1  based on a predetermined hash function of the logical index “Li” of the corresponding metadata page. Likewise, the storage node  104  can determine the target data containers H 1 , H 2 , H 3 , . . . , or H N  for the respective delta update tuples Li:Ei:T 2 :V in the first set  218  of data containers included in the metadata delta log  204 . 2  based on a predetermined hash function of the logical index “Li” of the corresponding metadata page. In general, the storage node  104  can determine the target data containers H 1 , H 2 , H 3 , . . . , or H N  for the respective delta update tuples Li:Ei:T p :V in the first set  226  of data containers included in the metadata delta log  204 . p  (in which “p” is any suitable positive integer) based on a predetermined hash function of the logical index “Li” of the corresponding metadata page. In such implementations, the various data containers H 1 , . . . , H N  included in the respective metadata delta logs  204 . 1 , . . . ,  204 . p  can be referred to as hash-based sorted buckets (HBSBs). 
     As shown in  FIG. 2 , the storage node  104  can write copies of the delta update tuples Li:Ei:T p :V (also referred to herein as “raw delta updates”) for the particular delta types T 1 , T 2 , . . . , T p  to a single journal (also referred to herein as the “raw delta log”)  205  (as illustrated by a path  242 ) included in the persistent memory (i.e., the NVRAM  124 ). In the event of a disaster, data loss, and/or data corruption, the storage node  104  can replay the raw delta log  205  to apply the delta update tuples written thereto to the metadata delta logs  204 . 1 ,  204 . 2 , . . . ,  204 . p  included in the volatile memory (i.e., the RAM  122 ), thereby recovering the metadata delta logs  204 . 1 ,  204 . 2 , . . . ,  204 . p  to a consistent state. In certain implementations, the raw delta log  205  can be configured as a ring buffer having a “head”  230  and a “tail”  232 . Further, the storage node  104  can write each respective tuple Li:Ei:T p :V for a particular delta type T 1 , T 2 , . . . , or T p  to the head  230  of the ring buffer, and subsequently release allocated space for the respective tuple Li:Ei:T p :V from the tail  232  of the ring buffer. As such, the raw delta log  205  can store the respective tuples Li:Ei:T p :V in time order (e.g., from oldest to newest). In certain implementations, the storage node  104  can determine that the raw delta log  205  is full, initiate one or more de-staging operations to store at least some of the delta update tuples written to the respective metadata delta logs  204 . 1 ,  204 . 2 , . . . ,  204 . p  to the metadata page store  234  (as illustrated by the paths  244 ,  246 , . . . ,  248 , respectively) included in the storage array  106 , and subsequently release the space for the delta update tuples from the tail  232  of the ring buffer. 
     During operation, the storage node  104  can receive a plurality of write IO requests for one or more of the storage targets  118 . 1 , . . . ,  118 . m  from one or more of the host computers  102 . 1 , . . . ,  102 . n . The plurality of write IO requests can direct the storage node  104  to write pending changes (“deltas,” “delta updates”) for one or more different metadata types to the respective metadata delta logs  204 . 1 , . . . ,  204 . p  included in the RAM  122 . In response to receipt of the plurality of write IO requests, the storage node  104  can write delta updates for a respective metadata type to a metadata delta log instance optimized for handling the respective metadata type. For example, the metadata delta log  204 . 1  can be optimized for handling the metadata type “T 1 ,” and the metadata delta log  204 . 2  can be optimized for handling the metadata type “T 2 .” In general, the metadata delta log  204 . p  (in which “p” is any suitable positive integer) can be optimized for handling the metadata type “T p .” 
     With regard to the functionality of the metadata delta log  204 . 1  included in the RAM  122 , the storage node  104  can write delta updates for the metadata type “T 1 ” to fill one or more of the first set  210  of data containers H 1 , H 2 , H 3 , . . . , H N  with the metadata delta updates. It is noted that the first set  210  of data containers H 1 , H 2 , H 3 , . . . , H N  included in the metadata delta log  204 . 1  can initially be designated as the “active” set, while the second set  212  of data containers H 1 , H 2 , H 3 , . . . , H N  included in the metadata delta log  204 . 1  can initially be designated as the “de-staging” set. The storage node  104  can also write copies of delta updates (“raw delta updates”) to the raw delta log  205  included in the NVRAM  124 . Once one or more of the first set  210  of data containers H 1 , H 2 , H 3 , . . . , H N  have been filled, the storage node  104  can switch the designation of the first set  210  of data containers from “active” to “de-staging,” as well as switch the designation of the second set  212  of data containers from “de-staging” to “active.” 
     In response to the switching of the “active” and “de-staging” designations for the first and second sets  210 ,  212  of data containers of the metadata delta log  204 . 1 , the storage node  104  can write a first bookmark to the raw delta log  205 . In certain implementations, such a bookmark can be configured as a record with several attributes, including a “type” attribute “T p ” specifying the metadata type handled by the metadata delta log  204 . p , an “identifier” attribute “I” specifying the current designated “active” set of data containers included in the metadata delta log  204 . p , and an “offset” attribute “A p ” specifying the current allocation offset in the raw delta log  205  (in which “p” is any suitable positive integer). The storage node  104  can also write the bookmark record having the attributes T p , I, A p  to a bookmark list in the RAM  122 . In certain implementations, the bookmark list can be configured as a linked list (or any other suitable memory structure) having a head and a tail, and the bookmark record having the attributes T p , I, A p  can be written at the head of the bookmark list. The storage node  104  can then initiate a transaction commit operation (also referred to herein as a “de-staging operation”) to be performed in a background process, de-staging or otherwise writing the delta updates for the respective metadata type (e.g., the metadata type “T 1 ”) from the current designated “de-staging” set (e.g., the first set  210  or the second set  212 ) of data containers to an associated metadata page in one of the plurality of storage targets  118 . 1 , . . . ,  118 . m  maintained in the storage array  106 . 
     It is noted that each of the remaining metadata delta logs  204 . 2 , . . . ,  204 . p  included in the RAM  122  can have the same functionality as the metadata delta log  204 . 1  described hereinabove. It is further noted that when one or more of the “active” set of data containers for a respective metadata delta log  204 . 1 , . . . , or  204 . p  is full, the designation of the set of data containers can be switched from “active” to “de-staging,” and a de-staging operation can be initiated for a respective metadata type T 1 , T 2 , . . . , or T p  from the full set of data containers. The order of the de-staging operations for the metadata delta logs  204 . 1 , . . . ,  204 . p  can therefore be unpredictable. Such de-staging operations can also be relatively long processes. Most of the time, de-staging operations are performed for all of the metadata types T 1 , T 2 , . . . , T p  to assure that the overall de-staging operation is a smooth process. In addition, for each metadata delta log  204 . 1 , . . . ,  204 . p , while the “active” set of data containers are being filled, the “de-staging” set of data containers can be de-staged. Once the “de-staging” set of data containers have completely de-staged, the next switch of the “active” and “de-staging” designations can occur. 
     Having written raw delta updates to the raw delta log  205  for the different metadata types T 1 , T 2 , . . . , T p , as well as written bookmarks specifying the respective metadata types T 1 , T 2 , . . . , T p , to both the raw delta log  205  and the bookmark list, the storage node  104  can determine whether at least a first de-staging operation for delta updates of a first metadata type (e.g., the metadata type “T 1 ”) and a second de-staging operation for delta updates of a second metadata type (e.g., the metadata type “T 2 ”) have been performed in the background process. If the storage node  104  determines that the first de-staging operation has been performed before the second de-staging operation, the storage node  104  can remove the first bookmark having the attributes T 1 , I, A 1  from the bookmark list, checking to see whether the first bookmark was removed from the tail of the bookmark list. In other words, the storage node  104  can check to see whether the first bookmark was the “oldest” bookmark in the bookmark list. If the first bookmark was indeed removed from the tail of the bookmark list, then the storage node  104  can safely perform a memory space reclaim operation to reclaim the space between a tail of the raw delta log and the first bookmark written to the raw delta log. 
     If the storage node  104  determines that the second de-staging operation has been performed before the first de-staging operation, then the storage node  104  can remove a second bookmark having the attributes T 2 , I, A 2  from the bookmark list, checking to see whether the second bookmark was removed from the tail of the bookmark list. In other words, the storage node  104  can check to see whether the second bookmark having the attributes T 2 , I, A 2  was the “oldest” bookmark in the bookmark list. If the second bookmark having the attributes T 2 , I, A 2  was not removed from the tail of the bookmark list (i.e., the first bookmark remains at the tail of the bookmark list), then the storage node  104  can defer performing a memory space reclaim operation on the raw delta log  205  until the first de-staging operation for the metadata type T 1  has been performed in the background process. 
     The disclosed techniques for handling journal space in a storage cluster with multiple delta log instances will be further understood with reference to the following illustrative example, and  FIGS. 3 a -3 g   . In this example, a storage node of a storage cluster writes copies of delta updates (“raw delta updates”) for different metadata types T 1 , T 2 , T 3  to a single raw delta log  302  (see, e.g.,  FIG. 3 a   ) included in a persistent memory (e.g., NVRAM), and determines whether to perform a memory space reclaim operation (or defer performing a memory space reclaim operation) on the raw delta log  302 . The raw delta log  302  is configured as a ring buffer having a “head”  306  and a “tail”  308 . In addition, the storage node  104  writes bookmarks having attributes T p , I, A p  (e.g., p=1, 2, or 3) to a bookmark list  304  (see, e.g.,  FIG. 3 d   ) in a volatile memory (e.g., RAM). The bookmark list  304  is configured as a linked list having a “head”  310 , a “tail”  312 , and a terminator node  310 . 
     In this example, the storage node writes delta updates for the metadata type “T 1 ” to a first metadata delta log, filling one or more of a current designated “active” set of data containers with the metadata delta updates. The storage node also writes copies of delta updates (“raw delta updates”) to space specified by an allocation offset “A 1 ” in the raw delta log  302  (see  FIG. 3 a   ). Once one or more of the current designated “active” set of data containers in the first metadata delta log have been filled, the storage node switches the designation of the current designated “active” set from “active” to “de-staging.” Having switched the designation for the set of data containers from “active” to “de-staging,” the storage node writes a bookmark “B 1 ” at the head  306  of the raw delta log  302  (i.e., at the end of the space specified by “A 1 ”) (see  FIG. 3 a   ). The storage node also writes the bookmark “B 1 ” having attributes T 1 , 1, A 1  at the head  310  of the bookmark list  304  (see  FIG. 3 d   ). It is noted that the identifier attribute “I” that specifies the current designated “active” set of data containers in the bookmark “B 1 ” is indicated as “1” for purposes of illustration. 
     Further in this example, the storage node writes delta updates for the metadata type “T 2 ” to a second metadata delta log, filling one or more of a current designated “active” set of data containers with the metadata delta updates. The storage node also continues to write raw delta updates to space specified by an allocation offset “A 2 ” in the raw delta log  302  (see  FIG. 3 b   ). Once one or more of the current designated “active” set of data containers in the second metadata delta log have been filled, the storage node switches the designation of the current designated “active” set from “active” to “de-staging.” Having switched the designation for the set of data containers from “active” to “de-staging,” the storage node writes a bookmark “B 2 ” at the head  306  of the raw delta log  302  (i.e., at the end of the space specified by “A 2 ”) (see  FIG. 3 b   ). The storage node also writes the bookmark “B 2 ” having attributes T 2 , 1, A 2  at the head  310  of the bookmark list  304  (see  FIG. 3 d   ). It is noted that the identifier attribute “I” that specifies the current designated “active” set of data containers in the bookmark “B 2 ” is indicated as “1” for purposes of illustration. 
     Still further in this example, the storage node writes delta updates for the metadata type “T 3 ” to a third metadata delta log, filling one or more of a current designated “active” set of data containers with the metadata delta updates. The storage node also continues to write raw delta updates to space specified by an allocation offset “A 3 ” in the raw delta log  302  (see  FIG. 3 c   ). Once one or more of the current designated “active” set of data containers in the third metadata delta log have been filled, the storage node switches the designation of the current designated “active” set from “active” to “de-staging.” Having switched the designation for the set of data containers from “active” to “de-staging,” the storage node writes a bookmark “B 3 ” at the head  306  of the raw delta log  302  (i.e., at the end of the space specified by “A 3 ”) (see  FIG. 3 c   ). The storage node also writes the bookmark “B 3 ” having attributes T 3 , 1, A 3  at the head  310  of the bookmark list  304  (see  FIG. 3 d   ). It is noted that the identifier attribute “I” that specifies the current designated “active” set of data containers in the bookmark “B 3 ” is indicated as “1” for purposes of illustration. 
     In addition, the storage node initiates one or more transaction commit operations (also referred to herein as “de-staging operations”) to be performed in one or more background processes, de-staging or otherwise writing the delta updates for a respective metadata type (e.g., the metadata type “T 1 ,” “T 2 ,” or “T 3 ”) from a current designated “de-staging” set of data containers to an associated metadata page in a storage target maintained in a storage array. Further, the storage node determines whether at least a first de-staging operation for delta updates of the metadata type “T 1 ,” a second de-staging operation for delta updates of the metadata type “T 2 ,” and a third de-staging operation for delta updates of the metadata type “T 3 ” have been performed in the background process(es). 
     In this example, the storage node determines that the first de-staging operation for delta updates of the metadata type “T 1 ” has been performed before both the second de-staging operation and the third de-staging operation. In this case, the storage node removes the bookmark “B 1 ” having the attributes T 1 , I, A 1  from the bookmark list  304 , and checks to see whether the bookmark “B 1 ” was removed from the tail  312  of the bookmark list  304 . In other words, the storage node checks to see whether the bookmark “B 1 ” was the “oldest” bookmark in the bookmark list  304 . If the bookmark “B 1 ” was indeed removed from the tail  312  of the bookmark list  304  (i.e., the bookmark “B 1 ” was the “oldest” bookmark in the bookmark list  304 ), then the storage node can safely perform a memory space reclaim operation to reclaim the space specified by “A 1 ” (as well as the space occupied by the bookmark “B 1 ”) in the raw delta log  302 . As shown in  FIG. 3 e   , once the bookmark “B 1 ” is removed from the bookmark list  304 , the bookmarks “B 2 ” and “B 3 ” remain in the bookmark list  304 . As further shown in  FIG. 3 e   , the memory space reclaim operation can be safely performed by advancing the tail  308  of the raw delta log  302  to the start of the space specified by “A 2 .” 
     If, however, the storage node determines that the second de-staging operation for delta updates of the metadata type “T 2 ” has been performed before both the first de-staging operation and the third de-staging operation, then the storage node removes the bookmark “B 2 ” having the attributes T 2 , I, A 2  from the bookmark list  304 , and determines that the bookmark “B 2 ” was not removed from the tail  312  of the bookmark list  304 . In other words, the storage node determines that the bookmark “B 2 ” was not the “oldest” bookmark in the bookmark list  304 . As shown in  FIG. 3 f   , once the bookmark “B 2 ” is removed from the bookmark list  304 , the bookmarks “B 1 ” and “B 3 ” remain in the bookmark list  304 . The storage node therefore defers performing a memory space reclaim operation on the raw delta log  302  until at least the first de-staging operation for delta updates of the metadata type “T 1 ” has been performed in the background process(es). As further shown in  FIG. 3 f   , because the memory space reclaim operation cannot be safely performed by the storage node at this time, the space specified by “A 1 ,” “A 2 ,” and “A 3 ,” as well as their corresponding bookmarks “B 1 ,” “B 2 ,” and “B 3 ,” remain in the raw delta log  302 . 
     The storage node then determines that the first de-staging operation for delta updates of the metadata type “T 1 ” has been performed in the background process(es). Further, the storage node removes the bookmark “B 1 ” having the attributes T 1 , I, A 1  from the bookmark list  304 , and determines that the bookmark “B 1 ” was removed from the tail  312  of the bookmark list  304 . In other words, the storage node determines that the bookmark “B 1 ” was the “oldest” bookmark in the bookmark list  304 . As shown in  FIG. 3 g   , once the bookmark “B 1 ” is removed from the bookmark list  304 , only the bookmark “B 3 ” remains in the bookmark list  304 . Because the bookmark “B 1 ” was indeed removed from the tail  312  of the bookmark list  304  (i.e., the bookmark “B 1 ” was the “oldest” bookmark in the bookmark list  304 ), the storage node can safely perform a memory space reclaim operation to reclaim the adjacent space specified by “A 1 ” and “A 2 ” (as well as the space occupied by the bookmarks “B 1 ” and “B 2 ”) in the raw delta log  302 . As described hereinabove, the storage node previously determined that the second de-staging operation for delta updates of the metadata type “T 2 ” has been performed in the background process(es). As further shown in  FIG. 3 g   , the memory space reclaim operation can be safely performed by advancing the tail  308  of the raw delta log  302  to the start of the space specified by “A 3 .” Once the storage node subsequently determines that the third de-staging operation for delta updates of the metadata type “T 3 ” has been performed, the sole bookmark “B 3 ” having the attributes T 3 , I, A 3  can be removed from the bookmark list  304 , and a memory space reclaim operation can be safely performed to reclaim the space specified by “A 3 ” (as well as the space occupied by the bookmark “B 3 ”) in the raw delta log  302 . 
     A method of handling journal space in a storage cluster with multiple delta log instances is described below with reference to  FIG. 4 . As depicted in block  402 , delta updates of a metadata type are written to both a set of data containers in a delta log instance and a raw delta log. The term “active” corresponds to a designation initially applied to the set of data containers in the delta log instance. As depicted in block  404 , in response to the designation of the set of data containers in the delta log instance being switched from “active” to “de-staging,” a bookmark for the metadata type is written to both the raw delta log and a bookmark list. As depicted in block  406 , in response to a de-staging operation being performed for writing the delta updates of the metadata type from the set of data containers in the delta log instance to a storage array: the bookmark for the metadata type is removed from the bookmark list; if the bookmark was an oldest bookmark in the bookmark list, space is reclaimed in the raw delta log; and, if the bookmark was not the oldest bookmark in the bookmark list, the reclaiming of the space in the raw delta log is deferred. 
     Having described the above illustrative embodiments, other alternative embodiments or variations can be made and/or practiced. For example, it was described herein with reference to the illustrative example that, if the storage node determines that the bookmark “B 2 ” for the metadata type “T 2 ” was not removed from the tail  312  of the bookmark list  304  (i.e., the bookmark “B 2 ” was not the “oldest” bookmark, but, rather, the bookmark “B 1 ” was the “oldest” bookmark in the bookmark list  304 ), then the storage node can defer performing a memory space reclaim operation on the raw delta log  302  until an additional de-staging operation (e.g., a de-staging operation for the metadata type “T 1 ”) has been performed in a background process. As an alternative (or addition) to the disclosed techniques, in order to address the case where such a de-staging operation for the metadata type “T 1 ” is not performed for an extended time (e.g., the metadata type “T 1 ” may have become idle), the storage node can set a minimum allowed free space threshold for the raw delta log, and monitor the free space in the raw delta log  302  to determine whether or not it is below the set threshold. If it is determined that the free space in the raw delta log  302  is below the set threshold, then the storage node can force the additional de-staging operation (e.g., the de-staging operation for the metadata type “T 1 ”) to be performed, thereby assuring that at least the minimum allowed free space in the raw delta log is maintained. 
     It was further described herein that, in the event of a disaster, data loss, and/or data corruption, a storage node can replay the raw delta log to apply the delta update tuples written thereto to the multiple instances of metadata delta logs in the volatile memory, thereby recovering the metadata delta logs to a consistent state. As an alternative (or addition) to the disclosed techniques, the storage node can further replay the raw delta log to apply the several bookmarks written thereto to the bookmark list in the volatile memory, thereby recovering the bookmark list to a consistent state. 
     Several definitions of terms are provided below for the purpose of aiding the understanding of the foregoing description, as well as the claims set forth herein. 
     As employed herein, the term “storage system” is intended to be broadly construed to encompass, for example, private or public cloud computing systems for storing data, as well as systems for storing data comprising virtual infrastructure and those not comprising virtual infrastructure. 
     As employed herein, the terms “client,” “host,” and “user” refer, interchangeably, to any person, system, or other entity that uses a storage system to read/write data. 
     As employed herein, the term “storage device” may refer to a storage array including multiple storage devices. Such a storage device may refer to any non-volatile memory (NVM) device, including hard disk drives (HDDs), solid state drives (SSDs), flash devices (e.g., NAND flash devices, NOR flash devices), and/or similar devices that may be accessed locally and/or remotely (e.g., via a storage attached network (SAN)). A storage array (drive array, disk array) may refer to a data storage system used for block-based, file-based, or object storage. Storage arrays can include, for example, dedicated storage hardware containing HDDs, SSDs, and/or all-flash drives. A data storage entity may be any one or more of a file system, object storage, a virtualized device, a logical unit (LU), a logical unit number (LUN), a logical volume, a logical device, a physical device, and/or a storage medium. An LU may be a logical entity provided by a storage system for accessing data from the storage system and may be used interchangeably with a logical volume. An LU or LUN may be used interchangeably with each other. A LUN may be a logical unit number for identifying an LU and may also refer to one or more virtual disks or virtual LUNs, which may correspond to one or more virtual machines. A physical storage unit may be a physical entity such as a drive or disk or an array of drives or disks for storing data in storage locations that can be accessed by address. A physical storage unit may be used interchangeably with a physical volume. 
     As employed herein, the term “storage medium” may refer to one or more storage media such as a hard drive, a combination of hard drives, flash storage, a combination of flash storage, a combination of hard drives, flash storage, and other storage devices, and/or any other suitable types or combinations of computer readable storage media. A storage medium may also refer to both physical and logical storage media, include multiple levels of virtual-to-physical mappings, and include an image or disk image. A storage medium may be computer-readable and may be referred to as a computer-readable program medium. 
     As employed herein, the term “IO request” or simply “IO” may be used to refer to an input or output request such as a data read request or data write request. 
     As employed herein, the terms, “such as,” “for example,” “e.g.,” “exemplary,” and variants thereof describe non-limiting embodiments and mean “serving as an example, instance, or illustration.” Any embodiments described herein using such phrases and/or variants are not necessarily to be construed as preferred or more advantageous over other embodiments, and/or to exclude the incorporation of features from other embodiments. In addition, the term “optionally” is employed herein to mean that a feature or process, etc., is provided in certain embodiments and not provided in other certain embodiments. Any particular embodiment of the present disclosure may include a plurality of “optional” features unless such features conflict with one another. 
     While various embodiments of the present disclosure have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present disclosure, as defined by the appended claims.