Patent Description:
The NVMe (non-volatile memory express) specification is a logical-device interface specification for accessing non-volatile storage media which is attached to a host computing device via a Peripheral Component Interconnect (PCI) Express (PCIe) bus. The NVMe specification may be used with a variety of non-volatile storage media, such as solid state drives (SSDs). One focus of the NVMe specification relates to input/output (I/O) communication between a host device, which may access and/or write to the non-volatile storage media, and a memory device which includes the non-volatile storage media. In this regard, NVMe implements a paired submission queue and completion queue mechanism, with host software on the host device placing commands into the submission queue. Completions are placed onto the associated completion queue by the memory device controller of the storage device.

An NVMe device implementing the NVMe specification may process write commands in a Relaxed Ordering mechanism, in which write commands sent by a requester are reordered and completed by a completer in an order that is different from the original order in which the write commands were sent. Patent application <CIT> teaches completing write commands in the order in which they are received and maintaining a table for each zone to track where data is written.

This may pose problems in an NVMe device implementing the Zoned Namespaces (ZNS) specification, which requires that zones be written sequentially. Under the ZNS specification, each zone of the device address space has a write pointer which keeps track of the position of the next write. Zone write commands advance the write pointer to the end of the newly written data. Preserving the sequential order of the zone write commands is especially important to ensure that the write pointer is advanced to the appropriate positions in the address space, so that zone write commands are executed at the correct locations in the non-volatile memory.

Commands may be scrambled out of order in the following situations:.

In view of the above, a zoned namespaces (ZNS) storage computing device is provided. The ZNS storage computing device comprises a processor, and non-volatile memory comprising a plurality of zones including a given zone. The processor is configured to execute a zone writing program to receive zone write commands, and responsive to receiving the zone write commands, execute the zone write commands on the given zone of the non-volatile memory of the storage computing device in an order specified by zone write sequence numbers included in a zone descriptor for the given zone.

In view of the above issues, the present disclosure describes a system and method for preserving the order in which zone write commands are issued on a NVMe ZNS storage device which is connected to a host computing device via a PCI Express (PCIe) bus.

Referring to <FIG>, a computing system <NUM> comprises a storage device <NUM> including a processor <NUM>, firmware <NUM>, non-volatile memory <NUM>, and a data bus <NUM> coupled to the processor <NUM> and the non-volatile memory <NUM>. The processor is configured to execute a zone writing program <NUM> or logic to receive zone write commands <NUM>, and responsive to receiving the zone write commands <NUM>, execute the zone write commands <NUM> on a given zone of the non-volatile memory <NUM>. The processor <NUM> is a microprocessor, which can include an application specific integrated circuit (ASIC). The firmware <NUM> may be configured as permanent software programmed into a read-only memory of the storage device <NUM>.

The processor <NUM> of the storage device <NUM> communicates with a memory controller <NUM> of a host device <NUM> via a PCIe bus <NUM>. The storage device <NUM> may be a rack mounted storage platform containing individual solid state drives, in one particular example. The non-volatile memory <NUM> of the storage device <NUM> is divided into zoned namespaces, including a plurality of zones 40a, 40b. Each zone 40a, 40b of the non-volatile memory <NUM> is a contiguous range of logical block addresses that are managed as a single unit. The number of zones 40a, 40b in the non-volatile memory <NUM> is not particularly limited, and the non-volatile memory <NUM> in the example of <FIG> is depicted as having x zones.

The host device <NUM> and the storage device <NUM> store one or a plurality of zone descriptor 24a, 24b, which are data structures containing information about the zones 40a, 40b of the non-volatile memory <NUM> of the storage device <NUM>. Each zone 40a, 40b has an associated zone descriptor 24a, 24b which contains a set of attributes including associated zone write sequence numbers (ZWSN) 26a, 26b, which record an order in which zone write commands <NUM> are issued by the memory controller <NUM> of the host device <NUM> for each zone 40a, 40b. The storage device <NUM> may retrieve the zone descriptors 24a, 24b and the zone write sequence numbers 26a, 26b by sending a zone management receive command <NUM> to the host device <NUM> to request a zone descriptor 24a, 24b. Responsive to receiving the zone management receive command <NUM>, the host device <NUM> sends one or a plurality of zone descriptors 24a, 24b to the storage device <NUM>. Accordingly, the storage device <NUM> can execute the zone write commands <NUM> in an order which is specified by zone write sequence numbers 26a, 26b included in the zone descriptor 24a, 24b for a given zone 40a, 40b.

Although only two zone descriptors 24a, 24b are depicted in the example of <FIG>, it will be appreciated that the processor <NUM> may retrieve any number of zone descriptors corresponding to the zones of the non-volatile memory <NUM> on which zone write commands are to be executed. The zone descriptors 24a, 24b may include support bits in the reserved fields of their data structures indicating that zone write sequence numbers are supported. The reserved fields of the zone write commands <NUM> may be configured to include the zone write sequence numbers 26a, 26b, so that the storage device <NUM> may retrieve the zone descriptors 24a, 24b and the zone write sequence numbers 26a, 26b from the reserved fields of the zone write commands.

Turning to <FIG>, an expanded view is depicted of the storage device <NUM> of <FIG>. Each zone write command <NUM> for each zone is assigned a zone write sequence number 26a, 26b starting with zero for the first sequence number. In the example of <FIG>, the zone write sequence numbers 26a in the zone zero descriptor 24a comprise write commands 26a(<NUM>)-(<NUM>) for zone zero 40a. The zone write sequence numbers 26b in the zone one descriptor 24b comprise write commands 26b(<NUM>)-(<NUM>) for zone one 40b. For zone zero 40a, the offset0 write command is assigned the SN0 sequence number 26a(<NUM>), the offset1 write command is assigned the SN1 sequence number 26a(<NUM>), the offset2 write command is assigned the SN2 sequence number 26a(<NUM>), and the offset3 write command is assigned the SN3 sequence number 26a(<NUM>), so that the offset0, offset1, offset2, and offset3 write commands are executed on the logical block addresses of zone zero 40a in this specified order.

Similarly, for zone one 40b, the offset0 write command is assigned the SN0 sequence number 26b(<NUM>), the offset1 write command is assigned the SN1 sequence number 26b(<NUM>), the offset2 write command is assigned the SN2 sequence number 26b(<NUM>), and the offset3 write command is assigned the SN3 sequence number 26b(<NUM>), so that the offset0, offset1, offset2, and offset3 write commands are executed on the logical block addresses of zone one 40b in this specified order.

Responsive to receiving a zone open command <NUM> from the host device <NUM>, the processor <NUM> opens a given zone on the non-volatile memory <NUM> for executing zone write commands. In the example of <FIG>, the zone open command <NUM> causes the processor <NUM> to open zone zero 40a and zone one 40b for executing zone write commands <NUM>. Each zone is assigned a current zone write sequence number 30a, 30b so that zone zero 40a is assigned a first current zone write sequence number 30a, and zone one 40b is assigned a second current zone write sequence number 30b. When a given zone is opened, the current zone write sequence number 30a, 30b for the given zone is reset to zero. The processor <NUM> then receives and holds the zone write commands <NUM> in a submission queue <NUM> preserving an order in which the zone write commands <NUM> were received by the processor <NUM>. The zone open command <NUM> may be configured with a zone write sequence number support bit 34a to indicate to the storage device <NUM> that that the host device <NUM> supports zone write sequence numbers 26a, 26b.

As depicted in the example of <FIG>, the zone commands in the submission queue <NUM> are scrambled due to a bad link quality. The submission queue <NUM> includes a first zone write command <NUM>(<NUM>) specifying offset1 for zone <NUM>, a second zone write command <NUM>(<NUM>) specifying offset3 for zone <NUM>, a third zone write command <NUM>(<NUM>) specifying offset0 for zone <NUM>, a fourth zone write command <NUM>(<NUM>) specifying offset2 for zone <NUM>, a fifth zone write command <NUM>(<NUM>) specifying offset0 for zone <NUM>, a sixth zone write command <NUM>(<NUM>) specifying offset2 for zone <NUM>, a seventh zone write command <NUM>(<NUM>) specifying offset1 for zone <NUM>, and an eighth zone write command <NUM>(<NUM>) specifying offset3 for zone <NUM>, in this order.

The processor <NUM> then compares the entries of the submission queue <NUM> to the zone write sequence numbers 26a, 26b included in the zone descriptor 24a, 24b, comparing the current zone write sequence number 30a, 30b of the zone descriptor 24a, 24b with the write commands in the submission queue <NUM>. The processor <NUM> may iterate through the submission queue <NUM> in a loop to compare each zone write command in the submission queue <NUM> to the current zone write sequence number 30a, 30b. For example, when the current zone write sequence number 30a for zone <NUM> is zero, the processor <NUM> compares the zone write sequence number 26a(<NUM>) for zone <NUM> at sequence number zero (SN0) with the first zone write command <NUM>(<NUM>) in the submission queue <NUM>. When the first zone write command <NUM>(<NUM>) does not match the current write sequence number 30a or does not meet the criteria for aborting the submission queue <NUM>, then the processor <NUM> compares the current write sequence number 30a to the next zone write command in the submission queue <NUM>: the second zone write command <NUM>(<NUM>). When the processor <NUM> is configured as a multi-core processor, then the processor <NUM> may use a compare-and-swap (CAS) instruction to perform the comparisons in shared memory.

The processor <NUM> then determines whether the current zone write sequence number 30a of the zone descriptor 24a matches the write command in the submission queue <NUM>, whether the zone write command in the submission queue <NUM> has a sequence number which is greater than the current zone write sequence number 30a of the zone descriptor 24a, whether the write command in the submission queue <NUM> has a sequence number which is less than the current zone write sequence number 30a of the zone descriptor 24a, and whether a timeout timestamp attached to a zone write command has expired.

When the current zone write sequence number 30a of the zone descriptor 24a matches the write command in the submission queue <NUM>, then the write command is executed by the processor <NUM>, the current zone write sequence number 30a is incremented by one, the write command is removed from the submission queue <NUM>, and a write command confirmation <NUM> or acknowledgment is sent to the host device <NUM>. For example, when the current zone write sequence number 30a is zero, and the fifth zone write command <NUM>(<NUM>) of the submission queue <NUM> matches the current zone write sequence number 26a(<NUM>) at sequence number zero (SN0), then the fifth zone write command <NUM>(<NUM>) is executed by the processor <NUM>, the current zone write sequence number 30a is incremented by one, the fifth zone write command <NUM>(<NUM>) is removed from the submission queue <NUM>, and a write command confirmation <NUM> is sent to the host device <NUM>.

When the zone write command in the submission queue <NUM> has a sequence number which is greater than the current zone write sequence number 30a of the zone descriptor 24a, then the write command is held as pending in the submission queue <NUM>, and a timeout timestamp is attached to the write command, recording the time at which the write command was held as pending in the submission queue <NUM>. For example, when the current zone write sequence number 30a is zero, and the first zone write command <NUM>(<NUM>) of the submission queue <NUM> has a sequence number of one, which is greater than the current zone write sequence number 30a of zero, then the first zone write command <NUM>(<NUM>) is held in the submission queue <NUM>, and a timeout timestamp is attached to the write command. When a zone write command with a timeout timestamp is not executed by the processor <NUM> within a predetermined time period specified by the timeout stamp, then the processor <NUM> determines that the zone write command has expired, and aborts all the write commands in the submission queue <NUM>.

When the write command in the submission queue <NUM> has a sequence number which is less than the current zone write sequence number 30a of the zone descriptor 24a, then the processor <NUM> aborts all the write commands in the submission queue <NUM>. For example, when the current zone write sequence number 30a is three, and the zone write command <NUM>(<NUM>) in the submission queue <NUM> is two, then the processor <NUM> aborts all the write commands in the submission queue <NUM>.

Referring back to <FIG>, operations of a second storage device <NUM> are described in detail. In the second storage device <NUM>, a processor is configured as firmware <NUM> to execute the zone writing program <NUM>. Since the features of the second storage device <NUM> are similar to the first storage device <NUM> with the exception of the firmware <NUM>, the detailed description thereof is abbreviated here. Like parts in this example are numbered similarly and share their functions with the exception of the firmware <NUM>, and will not be described again except as below for the sake of brevity.

The second storage device <NUM> includes firmware <NUM>, non-volatile memory <NUM>, and a data bus <NUM> coupled to the firmware <NUM> and the non-volatile memory <NUM>. The firmware <NUM> is configured to execute zone writing program <NUM> or logic to perform zone write commands <NUM> on the non-volatile memory <NUM> according to the zone writing program <NUM>. The firmware <NUM> may be configured as permanent software programmed into a read-only memory of the storage device <NUM>. The host device <NUM> and the storage device <NUM> store one or a plurality of zone descriptor 124a, 124b, which are data structures containing information about the zones 140a, 140b of the non-volatile memory <NUM> of the storage device <NUM>.

The firmware <NUM> of the storage device <NUM> communicates with a memory controller <NUM> of a host device <NUM> via a PCIe bus <NUM>. Responsive to receiving a zone open command <NUM> from the host device <NUM>, the firmware <NUM> opens a given zone for executing zone write commands. In the example of <FIG>, the zone open command <NUM> causes the firmware <NUM> to open zone zero 140a and zone 140b for executing zone write commands <NUM>. When a given zone is opened, the current sequence number 130a, 130b for the given zone is reset to zero. The firmware <NUM> then receives and holds the zone write commands <NUM> in a submission queue <NUM> preserving an order in which the zone write commands <NUM> were received by the firmware <NUM>.

The firmware <NUM> compares the entries of the submission queue <NUM> to the zone write sequence numbers 126a, 126b of the zone descriptor 124a, 124b, comparing the current zone write sequence number 130a, 130b of the zone descriptor 124a, 124b with the write commands in the submission queue <NUM>. When the storage device <NUM> is configured as a multi-core processor, then the firmware <NUM> may use CAS instruction to perform the comparisons in shared memory.

The firmware <NUM> then determines whether the current zone write sequence number 130a of the zone descriptor 124a matches the write command in the submission queue <NUM>, whether the zone write command in the submission queue <NUM> has a sequence number which is greater than the current zone write sequence number 130a of the zone descriptor 124a, whether the write command in the submission queue <NUM> has a sequence number which is less than the current zone write sequence number 130a of the zone descriptor 124a, and whether a timeout timestamp attached to a zone write command has expired.

When the current zone write sequence number 130a of the zone descriptor 124a matches the write command in the submission queue <NUM>, then the write command is executed by the firmware <NUM>, the current zone write sequence number 130a is incremented by one, the write command is removed from the submission queue <NUM>, and a write command confirmation <NUM> or acknowledgment is sent to the host device <NUM>.

When the write command in the submission queue <NUM> has a sequence number which is greater than the current zone write sequence number 130a of the zone descriptor 124a, then the write command is held as pending in the submission queue <NUM>, and a timeout timestamp is attached to the write command, recording the time at which the write command was held as pending in the submission queue <NUM>. When a zone write command with a timeout timestamp is not executed by the firmware <NUM> within a predetermined time period, then the firmware <NUM> determines that the write command has expired, and aborts all the write commands in the submission queue <NUM>.

When the write command in the submission queue <NUM> has a sequence number which is less than the current zone write sequence number 130a of the zone descriptor 124a, then the firmware <NUM> aborts all the write commands in the submission queue <NUM>.

The reserved fields of the zone write commands <NUM> may be configured to include the zone write sequence numbers 126a, 126b. The zone open command <NUM> may be configured with a zone write sequence number support bit 134a to indicate to the storage device <NUM> that that the host device <NUM> supports zone write sequence numbers 126a, 126b.

<FIG> illustrates a flowchart of a computerized method <NUM> for executing zone write commands on a given zone of the non-volatile memory of a storage device in an order specified by zone write sequence numbers included in a zone descriptor for the given zone in accordance with a zone writing program. The following description of computerized method <NUM> is provided with reference to the software and hardware components described above and shown in <FIG> and <FIG>. It will be appreciated that computerized method <NUM> also may be performed in other contexts using other suitable hardware and software components.

At step <NUM>, zone management receive command is sent to host device. At step <NUM>, a zone descriptor and zone write sequence numbers are retrieved from the host device. Each of the plurality of zones of the non-volatile memory of the storage device has an associated zone descriptor, and each zone descriptor has associated zone write sequence numbers, which record an order in which zone write commands are issued by a memory controller of the host device.

At step <NUM>, a zone open command is received from the host device. At step <NUM>, responsive to receiving the zone open command, a given zone is opened on the non-volatile memory for executing zone write commands. At step <NUM>, zone write commands are received from the host device. At step <NUM>, zone write commands from the host device are held in a submission queue.

At step <NUM>, the zone write commands of the submission queue are compared to the zone write sequence numbers. It is determined whether the current zone write sequence number of the zone descriptor matches the write command in the submission queue, whether the zone write command in the submission queue has a sequence number which is greater than the current zone write sequence number of the zone descriptor, whether the write command in the submission queue has a sequence number which is less than the current zone write sequence number of the zone descriptor, and whether a timeout timestamp attached to a zone write command has expired.

At step <NUM>, responsive to determining that the current zone write sequence number of the zone descriptor matches the write command in the submission queue, the zone write command is executed on the given zone of the non-volatile memory of the storage device in an order specified by the zone write sequence numbers included in the zone descriptor for the given zone. At step <NUM>, it is determined whether all the zone write commands in the submission queue have been executed.

At step <NUM>, responsive to determining that all the zone write commands in the submission queue have been executed, a write command confirmation or acknowledgment is sent to the host device, and at step <NUM>, the zone write operation is terminated.

At step <NUM>, responsive to determining that all the zone write commands in the submission queue have not been executed, the current zone write sequence number is incremented by one. At step <NUM>, the zone write command is removed from the submission queue. At step <NUM>, the next zone write command is identified in the submission queue for comparison to the current zone write sequence number, and the method <NUM> returns to step <NUM>.

At step <NUM>, responsive to determining that the write command in the submission queue has a sequence number which is less than the current zone write sequence number of the zone descriptor, or responsive to determining that a timeout timestamp attached to a zone write command has expired, all write commands in the submission queue are aborted.

At step <NUM>, responsive to determining that the zone write command in the submission queue has a sequence number which is greater than the current zone write sequence number of the zone descriptor, the zone write command is held as pending in the submission queue. At step <NUM>, a timeout timestamp is attached to the held zone write command. At step <NUM>, the next zone write command is identified in the submission queue for comparison to the current zone write sequence number, and the method <NUM> returns to step <NUM>.

The above-described system and method allow zone write commands in an NVMe device implementing the ZNS specification to be executed in the original order in which the write commands were sent by a host device. Accordingly, the original sequential order of the zone write commands can be preserved, and zone write commands can be prevented from being executed out of order in various situations, including command fetches, command data transfers, controller-to-firmware APIs, and multi-core processing.

Fig. <NUM> schematically shows a non-limiting embodiment of a computing system <NUM> that can enact one or more of the methods and processes described above. Computing system <NUM> may embody the storage device <NUM> or host device <NUM> described above and illustrated in <FIG> and <FIG>. Computing system <NUM> may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), and/or other computing devices, and wearable computing devices such as smart wristwatches and head mounted augmented reality devices.

Computing system <NUM> may optionally include a display subsystem <NUM>, input subsystem <NUM>, communication subsystem <NUM>, and/or other components not shown in Fig. <NUM>.

Non-volatile storage device <NUM> may include optical memory (e.g., CD, DVD, HD-DVD, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), or other mass storage device technology.

Display subsystem <NUM> may include one or more display devices.

When included, input subsystem <NUM> may comprise or interface with one or more user-input devices such as a keyboard, mouse, or touch screen.

It will be appreciated that "and/or" as used herein refers to the logical disjunction operation, and thus A and/or B has the following truth table.

Claim 1:
A zoned namespaces, ZNS, storage computing device comprising:
a processor (<NUM>); and
non-volatile memory (<NUM>) comprising a plurality of zones (40a, 40b) including a given zone, wherein
the processor (<NUM>) is configured to execute a zone writing program to:
receive zone write commands (<NUM>); and
responsive to receiving the zone write commands (<NUM>), execute the zone write commands (<NUM>) on the given zone of the non-volatile memory (<NUM>) of the storage computing device (<NUM>) in an order specified by zone write sequence numbers (24a, 24b) included in a zone descriptor for the given zone,
wherein the processor holds zone write commands from a host device in a submission queue; and
the processor compares the zone write commands of the submission queue to the zone write sequence numbers,
wherein one of:
when a current zone write sequence number matches a zone write command in the submission queue, then the zone write command is executed by the processor, the current zone write sequence number is incremented by one, and the zone write command is removed from the submission queue;
when a zone write command in the submission queue has a sequence number which is greater than a current zone write sequence number, then the zone write command is held as pending in the submission queue, and a timeout timestamp is attached to the zone write command; or
when a zone write command in the submission queue has a sequence number which is less than a current zone write sequence number, then the processor aborts all the zone write commands in the submission queue.