Patent Description:
Sometimes, a machine may need additional memory for operation. While increasing the normal memory (often Dynamic Random Access Memory (DRAM)) is an option, DRAM may be expensive. Other types of storage may serve to expand memory, exchanging performance for cost. But the technology to access such other types of storage may limit how the storage may be used, reducing its utility.

A need remains to support expanding memory while reducing the performance costs. From <CIT>it is known a method and system for providing a dual memory access to a non-volatile memory device using expanded memory addresses and direct memory access (DMA).

Embodiments of the invention provide the ability to route commands to a storage device. When a command is received from an application, a mode switch may determine whether the command is a command to be handled by a storage device using a block mode or using a direct access for files (DAX) mode. By that there is no need for the application to determine the best mode for a particular command.

The drawings described below are examples of how embodiments of the invention may be implemented, and are not intended to limit embodiments of the invention. Individual embodiments of the invention may include elements not shown in particular figures and/or may omit elements shown in particular figures. The drawings are intended to provide illustration and may not be to scale.

The examples shown in <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG> are important for understanding the invention as set in the appended claims but do not illustrate an embodiment of the invention as such. <FIG> shows a machine including a storage device that may be used to extend the memory.

Aspects and embodiments of the invention will be described in detail in the following invention, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to enable a thorough understanding of the invention. It should be understood, however, that persons having ordinary skill in the art may practice the invention without some of these specific details.

For example, a first module could be termed a second module, and, similarly, a second module could be termed a first module, without departing from the scope of the invention.

The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the description of the invention and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The components and features of the drawings are not necessarily drawn to scale.

Sometimes, a machine may need additional memory for operation. While increasing the memory (often Dynamic Random Access Memory (DRAM)) is an option, DRAM may be expensive. Other types of storage may serve to expand memory, exchanging performance for cost. But the technology to access such other types of storage may limit how the storage may be used, reducing its utility.

Using a storage device, such as a Solid State Drive (SSD), to expand memory in the machine may increase storage by a considerable amount for less than the cost of a comparable amount of memory. For example, the average cost of a <NUM> gigabyte (GB) DRAM module may be around $<NUM>, or about $<NUM> per GB. On the other hand, a <NUM> terabyte (TB) SSD may cost around $<NUM>, or about $<NUM> per GB. So, while DRAM may be faster than an SSD, DRAM may cost approximately <NUM> times as much per GB as an SSD. The cost savings of using an SSD may thus offset the fact that an SSD may be slower to access than DRAM.

But because SSDs may be designed to store large amounts of data, SSDs may also be designed to support reading and writing large amounts of data at a time. For example, while a command to load data from or store data to memory might read or write only, say, <NUM> bytes per command, an SSD read or write command may process data in chunks of, say, <NUM> kilobytes (KB). But a host processor may not know that the storage is an SSD, and may attempt to access the storage as though it was memory. Therefore, accessing large amounts of data from the storage in an SSD as though it was memory may take longer than accessing comparable amounts of data from the storage in an SSD when using the SSD as a typical storage device.

Embodiments of the invention may overcome these problems by using two interfaces to an SSD being used to extend the memory in the machine. For small amounts of data, conventional load/store commands may be used. But when large amounts of data are to be accessed from storage, and in particular when large amounts of data are to be written to storage, the SSD may be accessed using write commands. Since write commands may support larger amounts of data than store commands, data may be written to storage more efficiently using write commands than using store commands. The operating system may determine which mode to use when storing/writing data to the storage.

<FIG> shows a machine including a storage device that may be used to extend the memory. In <FIG>, machine <NUM>, which may also be termed a host or a system, may include processor <NUM>, memory <NUM>, and storage device <NUM>. Processor <NUM> may be any variety of processor. (Processor <NUM>, along with the other components discussed below, are shown outside the machine for ease of illustration: embodiments of the invention may include these components within the machine. ) While <FIG> shows a single processor <NUM>, machine <NUM> may include any number of processors, each of which may be single core or multi-core processors, each of which may implement a Reduced Instruction Set Computer (RISC) architecture or a Complex Instruction Set Computer (CISC) architecture (among other possibilities), and may be mixed in any desired combination.

Processor <NUM> may be coupled to memory <NUM>. Memory <NUM> may be any variety of memory, such as flash memory, Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Persistent Random Access Memory, Ferroelectric Random Access Memory (FRAM), or Non-Volatile Random Access Memory (NVRAM), such as Magnetoresistive Random Access Memory (MRAM) etc. Memory <NUM> may also be any desired combination of different memory types, and may be managed by memory controller <NUM>. Memory <NUM> may be used to store data that may be termed "short-term": that is, data not expected to be stored for extended periods of time. Examples of short-term data may include temporary files, data being used locally by applications (which may have been copied from other storage locations), and the like.

Processor <NUM> and memory <NUM> may also support an operating system under which various applications may be running. These applications may issue requests (which may also be termed commands) to read data from or write data to either memory <NUM>. When storage device <NUM> is used to support applications reading or writing data via some sort of file system, storage device <NUM> may be accessed using device driver <NUM>. But when storage device <NUM> is used to extend memory <NUM>, processor <NUM> may issue commands to load and/or store data as though storage device <NUM> were additional memory <NUM>. In such embodiments, the storage offered by storage device <NUM> may be used by processor <NUM> as though storage device <NUM> were itself memory <NUM>, even though storage device <NUM> may use a different physical structure and different access mechanism than memory <NUM>. In the remainder of this document, unless context implies otherwise, it should be understood that storage device <NUM> is being used to extend memory <NUM>, and may not be used by an application to store and retrieve data (files, objects, or other data types) as a storage device used for more traditional purposes.

In embodiments of the invention where storage device <NUM> is used to extend memory <NUM>, a cache-coherent interconnect protocol, such as the Compute Express Link® (CXL) protocol, or Peripheral Component Interconnect Express (PCIe) protocol, may be used to extend memory <NUM> using storage device <NUM>. (Compute Express Link is a registered trademark of the Compute Express Link Consortium, Inc. ) In embodiments of the invention, connection <NUM> may be used to enable processor <NUM> to communicate with storage device <NUM>. But embodiments of the invention may also enable access to storage device <NUM> via other mechanisms, such as device driver <NUM> via an interface such as the PCIe interface.

While <FIG> shows one storage device <NUM>, there may be any number (one or more) of storage devices in machine <NUM>.

While <FIG> uses the generic term "storage device", embodiments of the invention may include any storage device formats that may benefit from the use of computational storage units, examples of which may include hard disk drives and Solid State Drives (SSDs). Any reference to "SSD" below should be understood to include such other embodiments of the invention. In addition, while the discussion above (and below) focuses on storage device <NUM> as being associated with a computational storage unit, embodiments of the invention may extend to devices other than storage devices that may include or be associated with a computational storage unit. Any reference to "storage device" above (and below) may be understood as also encompassing other devices that might be associated with a computational storage unit.

<FIG> shows details of machine <NUM> of <FIG>. In <FIG>, typically, machine <NUM> includes one or more processors <NUM>, which may include memory controllers <NUM> and clocks <NUM>, which may be used to coordinate the operations of the components of the machine. Processors <NUM> may also be coupled to memories <NUM>, which may include random access memory (RAM), read-only memory (ROM), or other state preserving media, as examples. Processors <NUM> may also be coupled to storage devices <NUM>, and to network connector <NUM>, which may be, for example, an Ethernet connector or a wireless connector. Processors <NUM> may also be connected to buses <NUM>, to which may be attached user interfaces <NUM> and Input/Output (I/O) interface ports that may be managed using I/O engines <NUM>, among other components.

<FIG> shows processor <NUM> of <FIG> communicating with storage device <NUM> of <FIG> using two modes. In <FIG>, in one mode processor <NUM> may communicate with storage device <NUM> using mode <NUM>. In mode <NUM>, storage device <NUM> may include host-managed device memory (HDM) <NUM>. Processor <NUM> may access HDM <NUM> in the same manner as memory <NUM> of <FIG>: processor <NUM> may issue load and store commands (represented as command <NUM>) to read data from and write data to HDM <NUM>. HDM <NUM>, from the perspective of processor <NUM>, may be thought of as another memory module just like memory <NUM>. The underlying data may actually be stored in storage <NUM>, with HDM <NUM> acting as a "front end" for processor <NUM> to access the data.

In an extended memory system, processor <NUM> may use a range of addresses, some of which may be in memory <NUM> of <FIG> and some of which may be in storage device <NUM>. For example, consider a situation where memory <NUM> of <FIG> includes eight gigabytes (GB) of memory, and storage device <NUM> includes eight GB of storage. Processor <NUM> may use addresses 0x0 <NUM><NUM> through 0x1 FFFF FFFF to access data in memory <NUM> of <FIG>, and addresses 0x2 <NUM><NUM> through 0x3 FFFF FFFF to access data in storage device <NUM>. As far as processor <NUM> is concerned, machine <NUM> of <FIG> offers <NUM> GB of memory, accessible through addresses 0x <NUM><NUM> through 0x3 FFFF FFFF: processor <NUM> may not be concerned with what forms of storage are used to provide this <NUM> GB of memory. But given a particular address, a virtual memory management may know where the requested data actually resides.

When processor <NUM> issues a request to access a particular address, this address is termed a "physical address". For data stored in memory <NUM> of <FIG>, this term is reasonable: data at address, say, 0x0 <NUM><NUM> may be located at that actual address in memory <NUM> of <FIG>. But for data stored on storage device <NUM>, this term "physical address" might not be descriptive. For example, some storage devices, such as Solid State Drives (SSDs), may use translation layers to map a particular address as used by processor <NUM> to the address where the data is actually stored. In this manner, as discussed further with reference to <FIG> below, such storage devices may relocate data as needed without having to inform processor <NUM> of the new address where the data is stored. Thus, context may matter in understanding whether the term "physical address" refers to the actual physical location on storage device <NUM> where the data is stored or merely some other logical identifier for where the data is stored on storage device <NUM> (with storage device <NUM> handling a mapping from the "physical address" used by processor <NUM> to the actual storage location of the data on storage device <NUM>).

In addition to these operations, the virtual memory management may also manage a page table. A page table permits an application to use virtual addresses, which may be mapped to physical addresses. For every data used by an application, there may be an entry in the page table that associates the virtual address used by the application and the physical address (at least, from the perspective of processor <NUM>) in the extended memory system where the data is stored. In this manner, two or more applications may use the same virtual address, which may be mapped to different physical addresses, which may prevent the two applications from accessing the same data. (Of course, if the applications want to share data, then their respective page tables may map virtual addresses to the same physical address.

But there may be some complications in using storage device <NUM> to extend memory <NUM> of <FIG>. One complication may be the size of storage device <NUM> itself, and whether the entire storage device <NUM> is exposed to processor <NUM>. In some embodiments of the invention, HDM <NUM> may be equal in size to storage <NUM> offered by storage device <NUM>. In other embodiments of the invention, storage <NUM> may be larger than HDM <NUM>. For example, storage <NUM> might be approximately <NUM> times as large as HDM <NUM>.

In some embodiments of the invention where storage <NUM> is larger than HDM <NUM>, HDM <NUM> may be mapped to a particular set of addresses in storage <NUM>. Processor <NUM> may be unable to access data in addresses that do not map to addresses in HDM <NUM>. In such embodiments of the invention, the larger space in storage <NUM> may be lost, and storage device <NUM> may appear to only offer as much storage as HDM <NUM> offers.

In other embodiments of the invention where storage <NUM> is larger than HDM <NUM>, processor <NUM> may still be able to access any data from storage <NUM>. For example, consider the situation where HDM <NUM> includes <NUM> GB, and storage <NUM> includes <NUM> GB. If processor <NUM> attempts to access an address that is outside the address set currently supported by HDM <NUM>, storage device <NUM> may flush (i.e., write) any data from HDM <NUM> to storage <NUM>, then load another <NUM> GB of data from storage <NUM> into HDM <NUM>.

Another complication may be the unit of access for data on storage device <NUM>. For example, memory <NUM> of <FIG> is typically byte-addressable: processor <NUM> may attempt to read or write an individual byte within memory <NUM> of <FIG>. But storage device <NUM> might not be byte-addressable. For example, SSDs are typically written or read in units of a page or a block. A page may be, for example, approximately <NUM> KB or <NUM> KB in size (although other page sizes may also be used). A block may contain, for example, <NUM> pages or <NUM> pages (although blocks may include other numbers of pages). Therefore, a block may be many KB or MB in size: for example, a block that includes <NUM> pages of <NUM> KB each may include <NUM> MB of storage. Reading or writing <NUM> MB of data just to access a single byte may be inefficient.

To address this problem, storage device <NUM> may include buffer <NUM>. Buffer <NUM> may be a module such as a DRAM module (similar to the modules that may make up memory <NUM> of <FIG>). HDM <NUM> may support byte-addressable access to buffer <NUM>. Storage device <NUM> may then load and flush data between storage <NUM> and buffer <NUM> as needed. Thus, when processor <NUM> attempts to access an address from HDM that maps to an address in storage <NUM> not currently loaded in buffer <NUM>, storage device <NUM> may write any changed data in buffer <NUM> to storage <NUM>, then load a new portion of storage <NUM> into buffer <NUM> to enable processor <NUM> to access the data via HDM <NUM>.

A third complication in using storage device <NUM> to extend memory <NUM> of <FIG> may be the basic unit of access. For example, when processor <NUM> accesses memory <NUM> of <FIG>, processor <NUM> may load and store data in units of, say, <NUM> bytes (which may be the size of a cache line). But as discussed above, storage device <NUM> may use the page or the block as the basic unit of access, which may be larger (potentially orders of magnitude larger) than the size of the basic unit of access to memory <NUM> of <FIG>.

While using a smaller unit of access to access storage device <NUM> might not be detrimental (storage device <NUM>, via HDM <NUM> and buffer <NUM>, may manage accessing data using the smaller sized unit of access), using a smaller unit of access may be inefficient, particularly where large amounts of data are being transferred. For example, to send <NUM> KB (which may be the size of a page) worth of data using <NUM> bytes (the basic unit of access for memory <NUM>) at a time may involve <NUM> load or store commands, whereas a single native command for storage device <NUM> may enable transferring that amount of data.

To address this complication, processor <NUM> may also communicate with storage device <NUM> using mode <NUM>. In mode <NUM>, storage device <NUM> may issue direct memory access (DMA) commands, such as command <NUM>, to storage device <NUM>. These commands may use the native command set supported by storage device <NUM> to access data, such as an application attempting to access a file via a file system might use. For transferring larger amounts of data between processor <NUM> and storage <NUM>, using native commands may be more efficient, and might lead to a <NUM>% improvement (at least for bulk data access) in access to storage device <NUM>.

In some embodiments of the invention, command <NUM> may include any native command supported by storage device <NUM>. In other embodiments of the invention, command <NUM> may be limited to write commands. In such embodiments of the invention, requests to access data from storage <NUM> may be handled via command <NUM> in mode <NUM>. The reason mode <NUM> might not be used to handle data read operations may be to ensure data consistency: if data is read via mode <NUM>, information about what data might need to be written back to storage <NUM> (for example, from a write-back cache in processor <NUM>) may be lost, which might ultimately lead to data loss. In the remainder of this document, commands <NUM> issued via mode <NUM> may be limited to commands to write data to storage <NUM> (not to read data from storage <NUM>). But embodiments of the invention may include commands <NUM> as including commands to read data from storage <NUM>, provided that consistency and synchronization issues may be otherwise resolved.

<FIG> shows an operating system in machine <NUM> of <FIG> communicating with storage device <NUM> of <FIG> using two modes, according to embodiments of the invention as set in the appended claims. In <FIG>, application <NUM> communicates with operating system <NUM>, which in turn communicates with storage device <NUM>.

Application <NUM> issues commands, such as commands <NUM> and <NUM> of <FIG>, to access data from storage device <NUM>. Application <NUM> may issue write command <NUM> to write data to storage device <NUM>. Note that in this situation, application <NUM> may be aware that storage device <NUM> is acting to extend memory <NUM> of <FIG> but may issue the appropriate commands to perform a page- (or block-) sized write command on storage device <NUM>. Write command <NUM> may be delivered to operating system <NUM>, which may offer block mode <NUM>. In block mode <NUM>, access to data on storage device <NUM> is provided as with any storage device offering block access. Operating system may enable access to a file such as file <NUM>, store data from file <NUM> in page cache <NUM>, manage the blocks of data in file <NUM> (which may be identified using logical block addresses (LBAs) or other logical identifiers) using block management <NUM>, and use device driver <NUM> to access storage device <NUM>, sending write command <NUM> to storage device <NUM> via transmitter <NUM>.

Storage device <NUM> includes interface <NUM>-<NUM>. Interface <NUM>-<NUM> enables using the block mode to access storage device <NUM>: for example, commands received over interface <NUM>-<NUM> may be expected to use the PCIe protocol, and may be interpreted using PCIe module <NUM>.

As discussed further with reference to <FIG>, below, in identifying a particular data being requested from storage device <NUM>, application <NUM> may use a logical identifier, such as an LBA. This LBA may be an identifier within LBA space <NUM>. Storage device may map this LBA to a physical address, such as a physical block address (PBA) on storage device <NUM> where the data is actually stored in storage space <NUM>. This mapping may be performed using, for example, a flash translation layer. In this manner, application <NUM> may refer to the data without having to know exactly where the data is stored on storage device <NUM>, and without having to update the identifier if the data is relocated within storage device <NUM>.

On the other hand, application may issue load/store command <NUM> to read/write data from memory <NUM> of <FIG>, which may be extended by storage device <NUM>. Load/store command <NUM> may be delivered to operating system <NUM>, which also offers direct access for files (DAX) mode <NUM>. Application <NUM> may use mmap() to perform memory mapping, which may enable byte-addressable access to the data in the extended memory system. When operating system <NUM> receives load/store command <NUM>, operating system <NUM> may use mode <NUM> of <FIG> to process load/store command <NUM>. In mode <NUM> of <FIG>, operating system <NUM> may provide the command to virtual memory management <NUM>, which may then deliver the command to storage device <NUM>. Virtual memory management <NUM> may examine the virtual address used in load/store command <NUM> and map that virtual address to a physical address. From this physical address, virtual memory management <NUM> may identify where the data is actually stored: on storage device <NUM> or memory <NUM> of <FIG>, for example. Once virtual memory management <NUM> has determined that load/store command <NUM> should be directed to storage device <NUM>, operating system <NUM> may then send load/store command <NUM> to storage device <NUM> via transmitter <NUM>. (If virtual memory management <NUM> determines that load/store command <NUM> should be directed to memory <NUM>, load/store command <NUM> may travel a different path than if load/store command <NUM> is sent to storage device <NUM>: the latter may travel across a bus such as a PCIe bus, whereas the former may travel across a memory bus, which may involve a different transmitter than transmitter <NUM>.

Storage device <NUM> includes interface <NUM>-<NUM>. Interface <NUM>-<NUM> enables using the DAX mode to access storage device <NUM>: for example, commands received over interface <NUM>-<NUM> may be expected to use the CXL. mem protocol, and may be interpreted using CXL. mem module <NUM>. As discussed above with reference to <FIG>, while the load/store command received at interface <NUM>-<NUM> may use a physical address which may be within physical addresses <NUM>, this physical address may actually be another logical identifier for the data. Storage device <NUM> may then map physical addresses <NUM> to storage space <NUM> (which may or may not use a flash translation layer, depending on whether the physical addresses may be directly mapped). Note that storage space <NUM> is mapped to by both physical addresses <NUM> and LBA space <NUM>.

Not shown in <FIG> are HDM <NUM> and buffer <NUM> of <FIG>. HDM <NUM> and buffer <NUM> of <FIG> may be included to support the use of CXL. mem module <NUM> and the mapping of physical addresses <NUM> to storage space <NUM>.

Operating system also determines whether load/store command <NUM> (particularly when the command is store command <NUM>) should be better handled using block mode access. In such embodiments of the invention, operating system <NUM> uses mode <NUM> of <FIG> to process load/store command <NUM>. In mode <NUM> of <FIG>, mode switch <NUM> directs command <NUM> to be handled using block mode <NUM>, even though the command was issued using DAX mode <NUM>. Once operating system <NUM> has directed command <NUM> to be handled using block mode <NUM>, the command may be processed similarly as though application <NUM> had issued write command <NUM>.

Note that using mode switch 475results in load/store command <NUM> being processed as though it was write command <NUM>. Because the format of the commands may differ, before processing command <NUM> using block mode <NUM>, operating system <NUM> uses modifier <NUM> to modify load/store command <NUM> into a format similar to write command <NUM> before processing the command using block mode <NUM>. In this manner, storage device <NUM> may recognize and process the modified command. But despite mode switch <NUM> processing load/store command <NUM> as though it was write command <NUM>, as far as application <NUM> is concerned, application <NUM> is still accessing data in an extended memory system. Application <NUM> may not be aware that the data is stored in storage <NUM> or that operating system <NUM> may ultimately process load/store command <NUM> using write commands.

There are a number of ways in which operating system <NUM> makes the decision to handle store command <NUM> using block mode <NUM>. One way operating system <NUM> makes the determination to use block mode <NUM> to handle store command <NUM> based on a parameter of operating system <NUM>. For example, in some embodiments of the invention, operating system <NUM> may have a system-wide parameter (at least for use within the user space including application <NUM>) that may indicate whether store command <NUM> should be handled using block mode <NUM>. If this parameter is set, then store command <NUM> may be handled using block mode <NUM>; if this parameter is not set, then store command <NUM> may be handled using DAX mode <NUM>. Application <NUM> may then alter this parameter as appropriate to specify whether store command <NUM> should be handled using block mode <NUM> or DAX mode <NUM>.

In some embodiments of the invention as set in the appended claims, a file in operating system <NUM>, such as file <NUM>, includes a value and/or an attribute (such as a property associated with the file handled as metadata by operating system <NUM>) that specifies whether store command <NUM> should be handled using block mode <NUM> or DAX mode <NUM>: application <NUM> may change this value and/or attribute in file <NUM> as appropriate to specify whether store command <NUM> should be handled using block mode <NUM> or DAX mode <NUM>. Operating system <NUM> may access this value from file <NUM>, or this attribute for file <NUM>, as appropriate when processing store command <NUM> to make the determination. Note that file <NUM> may include multiple values and/or attributes, each of which may specify differently whether block mode <NUM> or DAX mode <NUM> should be used. For example, file <NUM> may include a value and/or an attribute associated with each file being written and/or read, which value and/or attribute may indicate the appropriate mode to use for accessing data from that file.

In some embodiments of the invention as set in the appended claims, operating system <NUM> determines the size of the data being accessed. If the size of the data being accessed satisfies a threshold (for example, fi the number of blocks being written is greater than some threshold number), then operating system <NUM> handles store command <NUM> using block mode <NUM>; otherwise, operating system <NUM> handles store command <NUM> using DAX mode <NUM>.

In some embodiments of the invention, an Application Programming Interface (API) to make system calls to change the mode to be used. For example, application <NUM> may use an API to change an attribute of file <NUM> to change the mode between block mode <NUM> and DAX mode <NUM>. APIs may also be used for other purposes.

In some embodiments of the invention, operating system <NUM> may use combinations of these approaches to determine whether to handle a particular command <NUM> using block mode <NUM> or DAX mode <NUM>. Note too that operating system <NUM> may also use command <NUM> itself as part of the determination process: for example, if only store commands may be handled using block mode <NUM>, then if command <NUM> is a load command, operating system <NUM> may hand command <NUM> using DAX mode <NUM>, regardless of any other parameter, attribute, value, or size that might suggest that command <NUM> ought to be handled using block mode <NUM>.

Because processor <NUM> of <FIG> may not block while write command <NUM> is being performed, processor <NUM> of <FIG> may execute other commands. But if these other commands were to include load/store commands <NUM>, the result could be data inconsistency. For example, consider the situation where write command <NUM> is being performed, and load command <NUM> is received. If load command <NUM> requests data that is being written by the pending write command <NUM>, then the data that is loaded may not be the data that was to be written using write command <NUM>. To avoid this data inconsistency problem, in some embodiments of the invention load command <NUM> may be blocked while write command <NUM> is incomplete. In other embodiments of the invention, operating system <NUM> may determine whether load command <NUM> attempts to load data being written by write command <NUM>, and may block load command <NUM> only if load command <NUM> attempts to load data being written by write command <NUM>. In some embodiments of the invention, operating system <NUM> may similarly block store command <NUM> (to avoid the possibility of write command <NUM> and store command <NUM> both attempting to write data to the same file, which could leave the data in storage device <NUM> in an unknown state (either as written by write command <NUM>, as written by store command <NUM>, or partially written by write command <NUM> and partially written by store command <NUM>).

<FIG> shows a Solid State Drive (SSD) used to extend memory <NUM> of <FIG>. In <FIG>, SSD <NUM> includes interfaces <NUM>-<NUM> and <NUM>-<NUM>. Interfaces <NUM>-<NUM> and <NUM>-<NUM> may be interfaces used to connect SSD <NUM> to machine <NUM> of <FIG>. As shown, SSD <NUM> may include more than one interface <NUM>-<NUM> and <NUM>-<NUM>: for example, one interface might be used for block-based read and write requests, and another interface might be used for DAX mode read and write requests. While <FIG> suggests that interfaces <NUM>-<NUM> and <NUM>-<NUM> are physical connections between SSD <NUM> and machine <NUM> of <FIG>, interfaces <NUM>-<NUM> and <NUM>-<NUM> may also represent protocol differences that may be used across a common physical interface. For example, SSD <NUM> might be connected to machine <NUM> using a U. <NUM> or an M. <NUM> connector, but may support block-based requests and DAX mode requests: handling the different types of requests may be performed by different interface <NUM>-<NUM> and <NUM>-<NUM>.

SSD <NUM> may also include host interface layer <NUM>, which may manage interfaces <NUM>-<NUM> and <NUM>-<NUM>. If SSD <NUM> includes more than one interface <NUM>-<NUM> and <NUM>-<NUM>, a single host interface layer <NUM> may manage all interfaces, SSD <NUM> may include a host interface layer for each interface, or some combination thereof may be used.

SSD <NUM> may also include SSD controller <NUM>, various channels <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>, along which various flash memory chips <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> may be arrayed (flash memory chips <NUM>-<NUM> through <NUM>-<NUM> may be referred to collectively as flash memory chips <NUM>). SSD controller <NUM> may manage sending read requests and write requests to flash memory chips <NUM>-<NUM> through <NUM>-<NUM> along channels <NUM>-<NUM> through <NUM>-<NUM> (which may be referred to collectively as channels <NUM>). Although <FIG> shows four channels and eight flash memory chips, embodiments of the invention may include any number (one or more, without bound) of channels including any number (one or more, without bound) of flash memory chips.

Within each flash memory chip, the space may be organized into blocks, which may be further subdivided into pages, and which may be grouped into superblocks. Page sizes may vary as desired: for example, a page may be <NUM> KB of data. If less than a full page is to be written, the excess space is "unused". Blocks may contain any number of pages: for example, <NUM> or <NUM>. And superblocks may contain any number of blocks. A flash memory chip might not organize data into superblocks, but only blocks and pages.

While pages may be written and read, SSDs typically do not permit data to be overwritten: that is, existing data may be not be replaced "in place" with new data. Instead, when data is to be updated, the new data is written to a new page on the SSD, and the original page is invalidated (marked ready for erasure). Thus, SSD pages typically have one of three states: free (ready to be written), valid (containing valid data), and invalid (no longer containing valid data, but not usable until erased) (the exact names for these states may vary).

But while pages may be written and read individually, the block is the basic unit of data that may be erased. That is, pages are not erased individually: all the pages in a block are typically erased at the same time. For example, if a block contains <NUM> pages, then all <NUM> pages in a block are erased at the same time. This arrangement may lead to some management issues for the SSD: if a block is selected for erasure that still contains some valid data, that valid data may need to be copied to a free page elsewhere on the SSD before the block may be erased. (In some embodiments of the invention, the unit of erasure may differ from the block: for example, it may be a superblock, which as discussed above may be a set of multiple blocks.

Because the units at which data is written and data is erased differ (page vs. block), if the SSD waited until a block contained only invalid data before erasing the block, the SSD might run out of available storage space, even though the amount of valid data might be less than the advertised capacity of the SSD. To avoid such a situation, SSD controller <NUM> may include a garbage collection controller (not shown in <FIG>). The function of the garbage collection may be to identify blocks that contain all or mostly all invalid pages and free up those blocks so that valid data may be written into them again. But if the block selected for garbage collection includes valid data, that valid data will be erased by the garbage collection logic (since the unit of erasure is the block, not the page). To avoid such data being lost, the garbage collection logic may program the valid data from such blocks into other blocks. Once the data has been programmed into a new block (and the table mapping LBAs to physical block addresses (PBAs) updated to reflect the new location of the data), the block may then be erased, returning the state of the pages in the block to a free state.

SSDs also have a finite number of times each cell may be written before cells may not be trusted to retain the data correctly. This number is usually measured as a count of the number of program/erase cycles the cells undergo. Typically, the number of program/erase cycles that a cell may support mean that the SSD will remain reliably functional for a reasonable period of time: for personal users, the user may be more likely to replace the SSD due to insufficient storage capacity than because the number of program/erase cycles has been exceeded. But in enterprise environments, where data may be written and erased more frequently, the risk of cells exceeding their program/erase cycle count may be more significant.

To help offset this risk, SSD controller <NUM> may employ a wear leveling controller (not shown in <FIG>). Wear leveling may involve selecting data blocks to program data based on the blocks' program/erase cycle counts. By selecting blocks with a lower program/erase cycle count to program new data, the SSD may be able to avoid increasing the program/erase cycle count for some blocks beyond their point of reliable operation. By keeping the wear level of each block as close as possible, the SSD may remain reliable for a longer period of time.

SSD controller <NUM> may include flash translation layer (FTL) <NUM> (which may be termed more generally a translation layer, for storage devices that do not use flash storage), HDM <NUM>, buffer <NUM>, and firmware <NUM>. FTL <NUM> may handle translation of LBAs or other logical IDs (as used by processor <NUM> of <FIG>) and physical block addresses (PBAs) or other physical addresses where data is stored in flash chips <NUM>-<NUM> through <NUM>-<NUM>. FTL <NUM>, may also be responsible for relocating data from one PBA to another, as may occur when performing garbage collection and/or wear leveling. HDM <NUM> and buffer <NUM> are discussed with reference to <FIG> above. Firmware <NUM> may be stored on an appropriate hardware module (such as a Programmable Read Only Memory PROM), an Erasable PROM (EPROM), and Electrically Erasable PROM (EEPROM), or some other variant thereof. Firmware <NUM> may be responsible for managing what data is in buffer <NUM>, and may flush data from buffer <NUM> to flash memory chips <NUM> and load data from flash memory chips <NUM> into buffer <NUM>.

<FIG> shows a flowchart of an example procedure for operating system <NUM> of <FIG> to switch between modes in communicating with storage device <NUM> of <FIG>. In <FIG>, at block <NUM>, operating system <NUM> of <FIG> may switch to block mode <NUM> of <FIG>: for example, if application <NUM> of <FIG> issues write command <NUM> of <FIG>, or if operating system <NUM> of <FIG> determines that store command <NUM> of <FIG> should be processed using block mode <NUM> of <FIG>. At block <NUM>, operating system <NUM> of <FIG> may process the command as write command <NUM> of <FIG> in block mode <NUM> of <FIG>.

At block <NUM>, operating system <NUM> of <FIG> may determine whether application <NUM> of <FIG> has issued load command <NUM> of <FIG> (or if operating system <NUM> of <FIG> determines that a further store command <NUM> of <FIG> should be processed using DAX mode <NUM> of <FIG>. If not, then processing may return to block <NUM> to handle further write commands in block mode <NUM> of <FIG>.

But if application <NUM> of <FIG> issues load command <NUM>, or if operating system <NUM> of <FIG> determines that store command <NUM> is to be handled using DAX mode <NUM> of <FIG>, then at block <NUM> (<FIG>) operating system <NUM> of <FIG> may switch to DAX mode <NUM> of <FIG>. At block <NUM>, operating system <NUM> of <FIG> may map the file being loaded by application <NUM> of <FIG>. This process may involve determining one or more virtual addresses that may be used by application <NUM> of <FIG> and the associated physical addresses in the extended memory system where the data is stored, and configuring the page table for application <NUM> of <FIG> accordingly. At block <NUM>, operating system <NUM> of <FIG> may send load/store command <NUM> of <FIG> to storage device <NUM> of <FIG>.

Finally, at block <NUM>, operating system <NUM> of <FIG> may determine whether application <NUM> of <FIG> has issued write command <NUM> of <FIG> (or if operating system <NUM> of <FIG> determines that a further store command <NUM> of <FIG> should be processed using block mode <NUM> of <FIG>. If not, then processing may return to block <NUM> to handle further load/store commands in DAX mode <NUM> of <FIG>. Otherwise, processing may return to block <NUM> of <FIG> to process write commands in block mode <NUM> of <FIG>.

<FIG> shows a flowchart of an example procedure for storage device <NUM> of <FIG> to receive commands using two modes. In <FIG>, at block <NUM>, storage device <NUM> of <FIG> may receive command <NUM> of <FIG> via interface <NUM>-<NUM> of <FIG>. At block <NUM>, controller <NUM> of <FIG> may process command <NUM> of <FIG>. At block <NUM>, storage device <NUM> of <FIG> may receive command <NUM> of <FIG> via interface <NUM>-<NUM> of <FIG>. At block <NUM>, controller <NUM> of <FIG> may process command <NUM> of <FIG>.

<FIG> shows a flowchart of an alternative example procedure for storage device <NUM> of <FIG> to receive commands using two modes, according to embodiments of the invention. In <FIG>, at block <NUM>, storage device <NUM> of <FIG> may grant processor <NUM> of <FIG> access to HDM <NUM> of <FIG>. At block <NUM>, storage device <NUM> of <FIG> may manage HDM <NUM> of <FIG> using buffer <NUM> of <FIG>. This management may include firmware <NUM> of <FIG> managing data consistency between buffer <NUM> of <FIG> and storage <NUM> of <FIG>.

At block <NUM>, storage device <NUM> of <FIG> may receive command <NUM> of <FIG> via interface <NUM>-<NUM> of <FIG>. At block <NUM>, storage device <NUM> of <FIG> may process command <NUM> of <FIG> using controller <NUM> of <FIG>. For example, controller <NUM> of <FIG> may perform load/store operations on buffer <NUM> of <FIG>, and may also write data to storage <NUM> of <FIG> or read data from storage <NUM> of <FIG>.

At block <NUM> (<FIG>), storage device <NUM> of <FIG> may receive command <NUM> of <FIG> via interface <NUM>-<NUM> of <FIG>. At block <NUM>, storage device <NUM> of <FIG> may process command <NUM> of <FIG> using controller <NUM> of <FIG>. For example, controller <NUM> of <FIG> may perform a write command for data to storage <NUM> of <FIG>.

Because write command <NUM> of <FIG> may update data that is stored in buffer <NUM> of <FIG>, at block <NUM>, storage device <NUM> of <FIG> may flush buffer <NUM> of <FIG>, to ensure that any data in buffer <NUM> of <FIG> that has been updated is written to storage <NUM> of <FIG>. At block <NUM>, storage device <NUM> of <FIG> may reload buffer <NUM> of <FIG> with data from storage <NUM> of <FIG>. Finally, at block <NUM>, because buffer <NUM> of <FIG> may have been updated with new data, storage device <NUM> of <FIG> may remap HDM <NUM> of <FIG> to buffer <NUM> of <FIG> to reflect the updated data, particularly if the size of a file is changed.

<FIG> shows a flowchart of an example procedure for operating system <NUM> of <FIG> to send commands to storage device <NUM> of <FIG> using two modes, according to embodiments of the invention as set in the appended claims. In <FIG>, at block <NUM>, operating system <NUM> of <FIG> receives command <NUM> of <FIG> or command <NUM> of <FIG> from processor <NUM> of <FIG>. At block <NUM>, operating system <NUM> of <FIG> may determine whether command <NUM> of <FIG> or command <NUM> of <FIG> should be handled using block mode <NUM> of <FIG> or DAX mode <NUM> of <FIG>. At block <NUM>, operating system <NUM> of <FIG> then selects interface <NUM>-<NUM> or <NUM>-<NUM> of <FIG> of storage device <NUM> of <FIG>. The selection of interface <NUM>-<NUM> or <NUM>-<NUM> of <FIG> may depend on command <NUM> of <FIG> or command <NUM> of <FIG>, among other possible information. For example, as discussed above with reference to <FIG>, if application <NUM> of <FIG> issues write command <NUM>, then block mode <NUM> of <FIG> may be used, which may lead to the selection of interface <NUM>-<NUM> of <FIG>; if application <NUM> of <FIG> issues load/store command <NUM>, then either block mode <NUM> of <FIG> or block mode <NUM> of <FIG>is used, depending on mode switch <NUM> of <FIG>, which leads to the selection of either interface <NUM>-<NUM> of <FIG> or interface <NUM>-<NUM> of <FIG>.

<FIG> shows a flowchart of an alternative example procedure for operating system <NUM> of <FIG> to send commands to storage device <NUM> of <FIG> using two modes, according to embodiments of the invention as set in the appended claims.

In <FIG>, at block <NUM>, operating system <NUM> of <FIG> receives command <NUM> of <FIG> or command <NUM> of <FIG> from processor <NUM> of <FIG>. At block <NUM>, operating system <NUM> of <FIG> may determine that command <NUM> of <FIG> or command <NUM> of <FIG> accesses data from storage device <NUM> of <FIG> (for example, the command might be a command that may be processed within operating system <NUM> of <FIG>, without accessing data from storage device <NUM> of <FIG>). At block <NUM>, operating system <NUM> of <FIG> then selects interface <NUM>-<NUM> or <NUM>-<NUM> of <FIG> of storage device <NUM> of <FIG>. The selection of interface <NUM>-<NUM> or <NUM>-<NUM> of <FIG> may depend on command <NUM> of <FIG> or command <NUM> of <FIG>, among other possible information. At block <NUM>, modifier <NUM> of <FIG> modifies command <NUM> of <FIG> to be in a format that looks like command <NUM> of <FIG>. For example, modifier <NUM> of <FIG> may replace the command name of command <NUM> of <FIG> with the command name of command <NUM> of <FIG>, and may replace the physical address of command <NUM> of <FIG> with the LBA associated with the data to be written as expected by storage device <NUM> of <FIG>. Modifier <NUM> of <FIG> may also modify the reorder parameters in the command, and may modify how the command and data are packaged (e.g., packetized) for transmission to storage device <NUM> of <FIG>. Block <NUM> may be omitted, as shown by dashed line <NUM>. Finally, at block <NUM>, transmitter <NUM> of <FIG> may transmit the command (modified or not) to selected interface <NUM>-<NUM> or <NUM>-<NUM> of <FIG>.

In <FIG>, some embodiments of the invention are shown. But a person skilled in the art will recognize that other embodiments are also possible, by changing the order of the blocks, by omitting blocks, or by including links not shown in the drawings.

The Dual Mode Solid State Drive (SSD) may support host-managed device memory (HDM) memory (that is, memory of the Dual Mode SSD that the host central processing unit (CPU) may access via the CXL. mem protocol) registration when the device is enumerated. For example, if the Dual Mode SSD is configured (in device firmware) to define an HDM of <NUM> GB, then the host CPU can see <NUM> GB of memory in the device.

The Dual Mode SSD may support memory access via CXL. mem and may support DMA engine for CXL.

The Dual Mode SSD may be designed to support two modes, memory access mode and storage access mode. The memory access mode may support CXL. mem, and the storage access model may support Peripheral Component Interconnect Express (PCIe). io may be used to access control registers and CXL configuration space.

A device driver may support an integrated device driver for both Non-Volatile Memory Express (NVMe) and CXL protocols.

In fsdax mode using a Non-Volatile Dual In-Line Memory Module (NVDIMM), the SSD may support two modes: block and dax mode. Block mode may enable read() and write() on direct memory access (DMA) via PCIe. DAX mode may enable load/store by CPU via CXL. This approach may have some data consistency issues due to maintaining two device access paths. In order to avoid the consistency issues, the SSD can include some principles for file system and memory access. The Dual Mode SSD may not permit "read()" via PCIe.

An application may create a file and may set its mode. If the application sets the file as a block mode, write() operations may be done by page cache on the filesystem, and actual data transfer may be done by DMA via an NVMe device driver. If the application sets the file as a dax mode, the application may invoke mmap() and may submit load/store commands via CXL.

The file system may maintain metadata and should track the changes on the file. If the file is changed by CXL. mem, the metadata may become corrupt. To avoid this concern: (i) the application may "write" with block mode when the total number of blocks change in the given file exceeds a threshold; (ii) the application may "load" and "store" with dax mode only; (iii) the "store" on dax mode may be for mapped pages only: the SSD may not allow to map additional pages to "store" via CXL. mem; (iv) remapping on dax mode may be required when "write" in block mode changes the size of file; and (v) "load" by CXL. mem may not be allowed when 'write' by PCIe is not completed.

The Dual Mode SSD may operate as a memory device, not a storage device.

The Dual Mode SSD may provide a huge size memory extension (equivalent with SSD capacity) with lower performance against DRAM. This tradeoff may be efficient for some memory size bounded applications.

The reason to maintain dual mode is because bulk data transfer via DMA (storage access) may be faster than writing via CXL. mem (memory access mode).

Software may support Application Programming Interfaces (APIs) on fsdax using NVDIMM.

Embodiments of the invention include a storage device that supports dual mode access. If a command sent to a storage device may be better served using block mode access, the command may be sent as a block mode command. If a command sent to the storage device may be better served using DAX mode, the command may be sent as a DAX mode command. The operating system determines the best mode for the command based on the command itself, an operating system parameter, a file value, a file attribute, or the size of the data being accessed. Once the operating system has determined the best mode, the operating system selects an appropriate interface on the dual mode storage device, to which the command may be delivered.

The following discussion is intended to provide a brief, general description of a suitable machine or machines in which certain aspects of the invention may be implemented. The machine or machines may be controlled, at least in part, by input from conventional input devices, such as keyboards, mice, etc., as well as by directives received from another machine, interaction with a virtual reality (VR) environment, biometric feedback, or other input signal. As used herein, the term "machine" is intended to broadly encompass a single machine, a virtual machine, or a system of communicatively coupled machines, virtual machines, or devices operating together. Exemplary machines include computing devices such as personal computers, workstations, servers, portable computers, handheld devices, telephones, tablets, etc., as well as transportation devices, such as private or public transportation, e.g., automobiles, trains, cabs, etc..

The machine or machines may include embedded controllers, such as programmable or non-programmable logic devices or arrays, Application Specific Integrated Circuits (ASICs), embedded computers, smart cards, and the like. The machine or machines may utilize one or more connections to one or more remote machines, such as through a network interface, modem, or other communicative coupling. Machines may be interconnected by way of a physical and/or logical network, such as an intranet, the Internet, local area networks, wide area networks, etc. One skilled in the art will appreciate that network communication may utilize various wired and/or wireless short range or long range carriers and protocols, including radio frequency (RF), satellite, microwave, Institute of Electrical and Electronics Engineers (IEEE) <NUM>, Bluetooth®, optical, infrared, cable, laser, etc..

Various embodiments may be described by reference to or in conjunction with associated data including functions, procedures, data structures, application programs, etc. which when accessed by a machine results in the machine performing tasks or defining abstract data types or low-level hardware contexts. Associated data may be stored in, for example, the volatile and/or non-volatile memory, e.g., RAM, ROM, etc., or in other storage devices and their associated storage media, including hard-drives, floppy-disks, optical storage, tapes, flash memory, memory sticks, digital video disks, biological storage, etc. Associated data may be delivered over transmission environments, including the physical and/or logical network, in the form of packets, serial data, parallel data, propagated signals, etc., and may be used in a compressed or encrypted format. Associated data may be used in a distributed environment, and stored locally and/or remotely for machine access.

Various embodiments may include a tangible, non-transitory machine-readable medium comprising instructions executable by one or more processors, the instructions comprising instructions to perform the elements of the inventions as described herein.

The various operations of methods described above may be performed by any suitable means capable of performing the operations, such as various hardware and/or software component(s), circuits, and/or module(s). The software may comprise an ordered listing of executable instructions for implementing logical functions, and may be embodied in any "processor-readable medium" for use by or in connection with an instruction execution system, apparatus, or device, such as a single or multiple-core processor or processor-containing system.

Claim 1:
A system, comprising:
a processor (<NUM>);
a memory (<NUM>) coupled to the processor (<NUM>);
a storage device (<NUM>) coupled to the processor (<NUM>), the storage device (<NUM>) including a storage (<NUM>) for data, a controller (<NUM>) to manage the data in the storage (<NUM>), a first interface (<NUM>-<NUM>) configured to enable access to the storage device (<NUM>) using a block mode, and a second interface (<NUM>-<NUM>) configured to enable access to the storage device (<NUM>) using a direct access for files, DAX, mode (<NUM>), the storage device (<NUM>) configured to extend the memory (<NUM>), wherein the storage device (<NUM>) includes a Solid State Drive, SSD; and
an operating system (<NUM>) to offer block mode (<NUM>) when a command issued by an application (<NUM>) is a block mode command (<NUM>) and to offer DAX mode (<NUM>) when a command issued by the application (<NUM>) is a DAX command (<NUM>),
wherein the operating system (<NUM>) comprises:
a mode switch (<NUM>) to select one of the first interface (<NUM>-<NUM>) and the second interface (<NUM>-<NUM>) for an issued DAX command (<NUM>) based at least in part on one of a parameter of the operating system (<NUM>), a metadata of a file (<NUM>) or a data size associated with the command;
a modifier (<NUM>) to convert the issued DAX command (<NUM>) to a block mode command when the selected interface is the first interface (<NUM>-<NUM>); and
a transmitter (<NUM>) to transmit the converted command to the storage device (<NUM>) using the selected first interface (<NUM>-<NUM>), and
wherein the controller (<NUM>) is configured to process the transmitted command (<NUM>; <NUM>).