Storage system employing MRAM and array of solid state disks with integrated switch

A storage system includes a central processing unit (CPU) subsystem including a CPU, a physically-addressed solid state disk (SSD) that is addressable using physical addresses associated with user data, provided by the CPU, to be stored in or retrieved from the physically-addressed SSD in blocks. Further, the storage system includes a non-volatile memory module, the non-volatile memory module having flash tables used to manage blocks in the physically addressed SSD, the flash tables include tables used to map logical to physical blocks for identifying the location of stored data in the physically addressed SSD. Additionally, the storage system includes a peripheral component interconnect express (PCIe) switch coupled to the CPU subsystem and a network interface controller coupled through a PCIe bus to the PCIe switch, wherein the flash tables are maintained in the non-volatile memory modules thereby avoiding reconstruction of the flash tables upon power interruption.

BACKGROUND OF THE INVENTION

Field of the Invention

This invention relates generally to storage systems, and particularly to storage systems utilizing physically-addressed solid state disk (SSD).

Background

Solid State Drives (SSDs) using flash memories have become a viable alternative to Hard Disc Drives (HDDs) in many applications. Such applications include storage for notebook, tablets, servers and network-attached storage appliances. In notebook and tablet applications, storage capacity is not too high, and power and or weight and form factor are key metric. In server applications, power and performance (sustained read/write, random read/write) are key metrics. In network-attached storage appliances, capacity, power, and performance are key metrics with large capacity being achieved by employing a number of SSDs in the appliance. SSD may be directly attached to the system via a bus such as SATA, SAS or PCIe.

Flash memory is a block-based non-volatile memory with each block organized into and made of various pages. After a block is programmed into the flash memory, it must be erased prior to being programmed again. Most flash memory require sequential programming of pages within a block. Another limitation of flash memory is that blocks can only be erased for a limited number of times, thus, frequent erase operations reduce the life time of the flash memory. A flash memory does not allow in-place updates. That is, it cannot overwrite existing data with new data. The new data are written to erased areas (out-of-place updates), and the old data are invalidated for reclamation in the future. This out-of-place update causes the coexistence of invalid (i.e. outdated) and valid data in the same block.

Garbage Collection (GC) is the process to reclaim the space occupied by the invalid data, by moving valid data to a new block and erasing the old block. But garbage collection results in significant performance overhead as well as unpredictable operational latency. As mentioned, flash memory blocks can be erased for a limited number of times. Wear leveling is the process to improve flash memory lifetime by evenly distributing erases over the entire flash memory (within a band).

The management of blocks within flash-based memory systems, including SSDs, is referred to as flash block management and includes: Logical to Physical Mapping; Defect management for managing defective blocks (blocks that were identified to be defective at manufacturing and grown defective blocks thereafter); Wear leveling to keep program/erase cycle of blocks within a band; Keeping track of free available blocks; and Garbage collection for collecting valid pages from a number of blocks (with a mix of valid and invalid page) into one block and in the process creating free blocks are examples of block management required to effectuate writing and programming of flash memory. Flash block management requires maintaining various tables referred to as flash block management tables (or “flash tables”). These tables are generally proportional to the capacity of SSD.

Generally, the flash block management tables can be constructed from metadata maintained on flash pages. Metadata is non-user information written on a page. Such reconstruction is time consuming and generally performed very infrequently upon recovery during power-up from a failure (such as power fail). In one prior art technique, the flash block management tables are maintained in a volatile memory, and as mentioned, the flash block management tables are constructed from metadata maintained in flash pages during power-up. In another prior art technique, the flash block management tables are maintained in a battery-backed volatile memory, utilized to maintain the contents of volatile memory for an extended period of time until power is back and tables can be saved in flash memory. In yet another prior art technique, the flash block management tables are maintained in a volatile random access memory (RAM), the flash block management tables are periodically and/or based on some events (such as a Sleep Command) saved (copied) back to flash, and to avoid the time consuming reconstruction upon power-up from a power failure additionally a power back-up means provides enough power to save the flash block management tables in the flash in the event of a power failure. Such power back-up may comprise of a battery, a rechargeable battery, or a dynamically charged super capacitor.

Flash block management is generally performed in the solid state drive (SSD) and the tables reside in the SSD. Alternatively, the flash block management may be performed in the system by a software or hardware, commands additionally include commands for flash management commands and the commands use physical addresses rather than logical addresses. An SSD with commands using physical addresses is referred to as Physically-Addressed SSD. The flash block management tables are maintained in the (volatile) system memory.

A storage system (also referred to as “storage array”, or “storage appliance”) is a special purpose computer system attached to a network, dedicated to data storage and management. The storage system may be connected to Internet Protocol (IP) Network running Network File System (NFS) protocol or Common Internet File System (CIFS) protocol or Internet Small Computer System (iSCSI) protocol or to a Storage Area Network (SAN) such as Fiber Channel (FC) or Serial Attached SCSI (SAS) for block storage.

These storage systems typically provide one or two network ports and one or more external network switches are required to connect multiple hosts to such systems. External network switches are costly and take rack space in the space constraint data centers.

There are also substantial latencies and processing associated with the above mentioned protocols which makes the storage system slow to respond.

In a storage system employing physically-addressed SSD which maintains the flash block management tables on the system memory that has no power back-up means for the system and no power back-up means for the system memory, the flash block management tables that reside in the system memory are lost and if copies are maintained in the flash onboard the SSD, the copies may not be updated and/or may be corrupted if power failure occurs during the time a table is being saved (or updated) in the flash memory.

Hence, during a subsequent power up, during initialization, the tables have to be inspected for corruption due to power fail and, if necessary, recovered. The recovery requires reconstruction of the tables to be completed by reading metadata from flash pages and results in further increase in delay for system to complete initialization. The process of complete reconstruction of all tables is time consuming, as it requires metadata on all pages of SSD to be read and processed to reconstruct the tables. Metadata is non-user information written on a page. This flash block management table recovery, during power-up, further delays the system initialization, the time to initialize the system is a key metric in many applications.

Yet another similar problem of data corruption and power fail recovery arises in SSDs and also Hard Disc Drives (HDDs) when write data for write commands (or queued write commands when command queuing is supported) is cached in a volatile system memory and command completion issued prior to writing to media (flash or HDD). It is well known in the art that caching write data for write commands (or queued write commands when command queuing is supported) and issuing command completion prior to writing to media significantly improves performance.

Additionally, file systems and storage systems employ journaling or logging for error recovery, the journal or log associated with a command or commands is saved in a persistent storage. In the event of a power fail or system crash or failure, the journal or log is played back to restore the system to a known state.

As mentioned before, in some prior art techniques, a battery-backed volatile memory is utilized to maintain the contents of volatile memory for an extended period of time until power returns and tables can be saved in flash memory.

Battery backup solutions for saving system management data or cached user data during unplanned shutdowns are long-established but have certain disadvantage including up-front costs, replacement costs, service calls, disposal costs, system space limitations, reliability and “green” content requirements.

What is needed is a system employing physically-addressed SSD to reliably and efficiently preserve flash block management tables in the event of a power interruption as well as having a low latency and supporting number of hosts.

SUMMARY OF THE INVENTION

Briefly, a storage system includes a central processing unit (CPU) subsystem including a CPU, a physically-addressed solid state disk (SSD) that is addressable using physical addresses associated with user data, provided by the CPU, to be stored in or retrieved from the physically-addressed SSD in blocks. Further, the storage system includes a non-volatile memory module, the non-volatile memory module having flash tables used to manage blocks in the physically addressed SSD, the flash tables include tables used to map logical to physical blocks for identifying the location of stored data in the physically addressed SSD. Additionally, the storage system includes a peripheral component interconnect express (PCIe) switch coupled to the CPU subsystem and a network interface controller coupled through a PCIe bus to the PCIe switch, wherein the flash tables are maintained in the non-volatile memory modules thereby avoiding reconstruction of the flash tables upon power interruption.

These and other objects and advantages of the invention will no doubt become apparent to those skilled in the art after having read the following detailed description of the various embodiments illustrated in the several figures of the drawing.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

Referring now toFIG. 1, a storage system100is shown, in accordance with an embodiment of the invention. The system100is shown to include a Central Processor Unit (CPU) subsystem20(also known herein as “processor” or “host”), a system memory30, a non-volatile memory (NVM) module40, and a bank of physically-addressed solid state disks (SSD)70, in accordance with an embodiment of the invention. A “Bank”, as used herein, refers to one or more.

The CPU subsystem20of system100is shown to include a multi-core CPU22, a memory controllers24shown coupled to a memory bus32to the memory30, a PCIe controller28, an NVM controller26shown coupled to NVM module40. The NVM module40is shown coupled to the NVM controller26of CPU subsystem20through a NVM bus42. The memory30is shown coupled to the memory controllers24through memory bus32.

The system100further includes a network interface controller (NIC)50and a peripheral component interconnect express bus (PCIe) switch60. The NIC50is shown coupled through a PCIe bus52and network interface54for connection to the network. The PCIe switch60is shown coupled to the PCIe controller28of the CPU subsystem20through a PCIe bus62.

The NIC50is shown to receive input through the network interface54.

The storage array70is shown to include a bank of storage array elements72. Each storage array element72is shown to include a bank of flash memories74and connects to the PCIe switch60through the PCIe bus64.

The NIC50includes circuitry required to communicate with a specific physical layer and data link layer for receiving and transmitting information packets including command/status and data, as is known to those in the industry. The NIC50further includes circuitry required for communicating with upper layer protocols (layer above data link layer, such as transport layer, application layer, . . . ), as is also known in the industry.

In some embodiments, the network interface54is a Gigabit or ten Gigabit Ethernet running Internet Small Computer System Interface (iSCSI) and in other embodiments, it is a Serial Attached SCSI (SAS) or Fiber Channel (FC), which are generally used with block storage protocols. In yet other embodiments, the network interface52is Gigabit or ten Gigabit Ethernet running network file storage (NFS) protocol. All of the foregoing interfaces are known in the art. In particular the Ethernet capabilities are either integrated into the CPU subsystem20or implemented via a low-cost dedicated NIC50, connected directly to CPU subsystem20or through the PCIe switch60and PCIe bus62to CPU subsystem20as shown inFIG. 1.

In operation, a network switch is connected to the NIC50through the network interface54. Multiple hosts can utilize the storage system100to perform read and write operations through the network switch. The NIC50receives commands from different hosts and directs them to the CPU subsystem20. The CPU subsystem20processes commands received by the NIC50through the network interface54from the network switch and their associated payload; creates new commands and data structures required by storage array elements72of storage array70in the DRAM module30and the NVM module40; and notifies the storage array elements72of storage array70accordingly. Each storage array element72independently communicates with the PCIe controller28of the CPU subsystem20through the PCIe switch60via the PCIe busses64and62to retrieve their command and data structures. Once the storage array elements72have processed and completed their commands, they send completion status to the CPU subsystem20. The CPU subsystem20collects the completion status from all the storage array elements72and formulates status corresponding to the host commands.

In some embodiment, the PCIe switch60may be integrated into the PCIe controller28or the NIC50may be coupled directly to the CPU subsystem20. In other embodiment, the PCIe switch60may include multiple and cascaded PCIe switches to provide the fan-out required by the storage system100. By expanding the number of PCIe ports and PCIe lanes, the storage system can supports many physically-addressed SSD to provide the large storage capacities required for these class of systems. If the storage system lacks the number of PCIe ports and PCIe lane to support the required capacity, it will make such system disadvantageously smaller in capacity than desired.

Referring now toFIG. 2, a storage system110is shown in accordance with another embodiment of the invention. The storage system110is analogous to the storage system100ofFIG. 1except that the NVM module44is shown coupled to the PCIe switch60through the PCIe bus46in the embodiment ofFIG. 2whereas in storage system100, the NVM module40is shown coupled to the NVM controller26of the CPU subsystem20. In the storage system110, operationally, the system100and110behave analogously with the exception of the manner in which the NVM module44is accessed by the CPU subsystem20. That is, in the storage system100, CPU subsystem20accesses the NVM module40through the NVM bus42whereas in the storage system110, the CPU subsystem10accesses the NVM module44through the PCIe switch60and the PCIe controller28.

Referring now toFIG. 3, a storage system120is shown in accordance with yet another embodiment of the invention. The storage system120is analogous to the storage system110ofFIG. 2except that the storage system120further includes a network switch80coupled to the NIC50through the network interface54and host1through host ‘n’ are coupled to the network switch80through the network interfaces82with ‘n’ being an integer value. The embodiment ofFIG. 3allows host1through host ‘n’ to be directly connected to the storage system120therefore eliminating the need for an external expensive high speed Ethernet network switch. Integrating the network switch into the storage system will improve the storage rack utilization and cost associated with implementing optimized data centers.

Referring now toFIG. 4, a shared PCIe based storage system130is shown in accordance with an embodiment of the invention. Storage system130is analogous to the storage system100ofFIG. 1except that the storage system130lacks the NIC50and network interface54. In this embodiment, the host1through the host ‘n’ communicate directly with the PCIe controller28of CPU subsystem20via PCIe busses68and therefore eliminating the cost and latency associated with the NIC controller50of the embodiment ofFIG. 1.

There are substantial amount of latencies and processing associated with the protocols used in NIC based storage system which makes such storage storage system slow to respond. To avoid these latencies, a shared PCIe based storage system provides PCIe ports as means for direct connections to network of hosts via PCIe cables and therefore bypasses the software stacks and protocol processing associated with use of NIC. The shared PCIe based storage system provides a high speed and high performance, sharing infrastructure using existing protocols such as non-volatile memory express (NVMe).

The shared PCIe based storage system looks like an individual direct attached storage (DAS) to each host without the limitation of DAS. DAS as well known in the industry is dedicated to the server that is installed in and can't be shared with other servers. It also means potential waste of resource when the entire capacity of the DAS is not required by that server. The shared PCIe based storage system solves these problems by providing a shared storage among the network of hosts. The storage system management tools will allocate storage to each host based on their requirements and more importantly, files across the network hosts can be shared.

FIG. 5shows a shared PCIe based storage system140, in accordance with yet another embodiment of the invention. The shared PCIe based storage system130is analogous to the storage system140with the exception of the host1through host n shown coupled to the PCIe controller28of the CPU subsystem20though PCIe switch67via the PCIe bus63. In this embodiment, PCIe switch67is used to extend the PCIe ports and PCIe lanes of PCIe controller28.

The embodiment ofFIG. 5allows host1through host ‘n’ to be directly connected to the storage system140therefore eliminating the need for an external expensive high speed PCIe switch. Integrating the PCIe switch into the storage system will improve the storage rack utilization and cost associated with implementing optimized data centers.

FIG. 6ashows shared PCIe based storage system150, in accordance with another embodiment of the invention. Shared PCIe based storage system150is analogous to the storage system140with the exception of the PCIe switches. Shared PCIe based storage system140is shown to have two PCIe switches67and61whereas the shared PCIe based storage system150integrates the two switches into a single PCIe switch63.

FIG. 7shows a storage system160, in accordance with another embodiment of the invention. Storage system160is analogous to the storage system150ofFIG. 6with the exception of the NVM module. In the storage system150ofFIG. 6, the NVM module40is shown coupled directly to the CPU subsystem20whereas in the storage system160ofFIG. 7, the NVM module42is shown coupled to the CPU subsystem10through a PCIe switch63.

FIG. 8shows a storage system170, in accordance with another embodiment of the invention. The storage system170is analogous to the storage system150ofFIG. 6and the storage system160with the exception of the NVM module in that in the storage system160ofFIG. 7, the NVM module44is shown coupled to the PCIe controller28through a PCIe switch63whereas in the storage system170ofFIG. 8, the NVM module90is shown coupled directly to the PCIe controller28of the CPU10through the PCIe bus92.