Patent Description:
A host computer ("host") communicates with a solid state drive (SSD) to obtain data services provided by the SSD, such as reading data, storing data, and erasing data. The solid-state drive (SSD) is a data storage device that includes non-volatile memory, such as NAND (Not-And) Flash memory, to store persistent digital data. The SSD may be configured to emulate a hard disk drive (HDD), e.g., a device that stores persistent digital data on a magnetic surfaces of rapidly rotating platters. Current SSDs include one of three physical interfaces via which the SSDs can be accessed: serial attached small computer system interface (SAS), serial AT attachment (SATA), and peripheral component interconnect express (PCIe) (e.g., a U. <NUM> connector).

Non-volatile memory express (NVMe) is a logical interface that defines a register level interface for accessing a non-volatile memory (e.g., an SSD) over a PCIe bus. As an example, a host computer may use NVMe to access a prior art SSD having a PCIe physical interface. A driver executed on the host computer converts NVMe commands to PCIe signals, and vice versa.

NVMe over Fabrics (NVMe-oF) is a protocol designed to use message-based commands to transfer data between a host computer and a target SSD or system over a network, such as Ethernet, Fibre Channel (FC) or InfiniBand, thus extending the distance over which SSDs can be accessed. NVMe-oF uses a message-based model for communication between a host and a target storage device (e.g., an SSD), and includes a transport-mapping mechanism for sending and receiving commands and responses related to memory operations. As an example, a host computer may use NVMe-oF to access a prior art SSD via a network, where the SSD is included in a storage subsystem having an NVMe-oF interface device. The NVMe-oF interface device converts NVMe commands in NVMe-oF messages from the host to PCIe signals, and converts PCIe signals to NVMe commands, and includes the NVMe commands in NVMe-oF messages to the host.

<CIT> discloses a storage device, comprising: a chassis; non-volatile storage media disposed on the chassis; a network interface connector integrated with the chassis. The network interface connector integrated with the chassis is structured to be directly inserted into a network switch; and control logic disposed on the chassis and configured to enable access to the non-volatile storage media through the network interface connector.

<CIT> describes a eSSD system which includes at least one primary SSD, at least one secondary SSD, an Ethernet switch, and a storage-offload (SoE) controller. The SoE controller may operate in a replication mode and/or an erasure-coding mode. In either mode, the SoE controller receives a first write command sent from a remote device to at least one primary SSD. In the replication mode, the SoE controller sends a second write command to the at least one secondary SSD to replicate at the at least one secondary SSD data associated with the first write command. In the erasure-coding mode, the SoE determines erasure codes associated with the first write command and manages distribution of the write data and associated erasure codes.

<CIT> describes a data replication system which has a chassis including a plurality of eSSDs, a fabrics switch, and a baseboard management controller (BMC). The BMC configures one of the plurality of eSSDs as an active eSSD and one or more of the plurality of eSSDs as one or more passive eSSDs. The fabrics switch of the chassis is programmed to forward packets destined for the active eSSD to both the active eSSD and the one or more passive eSSDs. In response to a host data write command received from the host, the active eSSD stores the host data and sends an address and an instruction corresponding to the host data to the one or more passive eSSDs. Each of the one or more passive eSSDs stores a copy of the host data using the address and the instruction received from the active eSSD and the host data received in the packets forwarded by the fabrics switch.

It is therefore the object of the present invention to provide a solid state drive (SSD) with a built-in network interface device to enable communication over a network fabric in a more efficient manner.

In an embodiment, a unitary solid state drive (SSD) assembly comprises: a non-volatile memory (NVM); a processor communicatively coupled to the NVM and configured to implement a communication protocol configured for accessing solid state memories over a communication network; a network interface device communicatively coupled to the processor, wherein the network interface device is configured to communicate via a network fabric according to a network communication protocol, and wherein the NVM, the processor, and the network interface device are arranged in a unitary assembly. The unitary SSD assembly further comprises a network connector coupled to the network interface device.

In another embodiment, a method of accessing an NVM of a unitary SSD assembly includes: receiving, at a network interface device of the unitary SSD assembly, a packet that includes information related to accessing the NVM, the packet received from a network fabric communicatively coupled to the network interface device; determining, at the network interface device of the unitary SSD assembly, that a destination network address in a header of the packet matches a network address of the unitary SSD assembly; and in response to determining that the network address in the header of the packet matches the network address of the unitary SSD assembly: decapsulating, at a processor of the unitary SSD assembly, a data unit from the packet, the data unit corresponding to a communication protocol for accessing storage devices, and using information in the data unit to access the NVM.

An SSD device includes a built-in network interface device (such as an Ethernet network interface device) for providing the SSD with connectivity to a network (such as an Ethernet network). SSDs with built-in network interface devices enable easier and/or more affordable scalability of storage capacity. A storage subsystem comprises multiple such SSDs and a network fabric (e.g., including an Ethernet switch) that communicatively couples the SSDs and provides network access to the SSDs, according to an embodiment. The storage subsystem can be easily scaled by adding additional SSDs with built-in network interface devices and coupling such SSDs to the network fabric. Similarly, a storage system can be easily scaled additionally or alternatively by adding additional such storage subsystems.

SSDs with built-in network interface devices enable functions that prior art SSDs do not provide. A storage subsystem comprises multiple SSDs with respective built-in network interface devices, the multiple SSDs communicatively coupled via a network fabric. In such a storage subsystem, data can be copied, moved, etc., from one SSD to another via the fabric and without first transferring any of the data to a host computer.

<FIG> is a simplified diagram of a unitary SSD assembly <NUM> with a built-in network interface device. In an example unitary SSD assembly <NUM>, the built in network interface device comprises an Ethernet network interface device <NUM>, and <FIG> is described in the context of an Ethernet network for ease of explanation. In other embodiments, however, the built in network interface device comprises another suitable network interface device configured to communicatively couple to another suitable network fabric, such as a Fibre Channel (FC) network interface device configured to communicatively couple to an FC network fabric, an InfiniBand network interface device configured to communicatively couple to an InfiniBand network fabric, etc..

The unitary SSD assembly <NUM> also comprises a non-volatile memory (NVM) <NUM>, such as a NAND (Not-And) Flash memory or another suitable solid state memory. In an embodiment, the NVM <NUM> comprises an array of NVM memories.

The unitary SSD assembly <NUM> additionally comprises a processor <NUM>. In some embodiments, the processor <NUM> is configured to execute machine readable instructions stored in a memory (not shown) coupled to and/or integrated with the processor <NUM>. The machine readable instructions, when executed by the processor <NUM>, cause the processor <NUM> to implement one or more communication protocol layers, or sublayers thereof, related to accessing the NVM <NUM> via a network fabric, as will be described in more detail below. In some embodiments, the processor <NUM> additionally or alternatively comprises a hardware controller that includes one or more hardware state machines and/or one or more pipelined hardware processors configured to implement one or more communication protocol layers, or sublayers thereof, related to accessing the NVM <NUM> via a network fabric, as will be described in more detail below. In some embodiments, the processor <NUM> additionally or alternatively comprises one or more hardware tables configured for packet processing, memory management, etc. In some embodiments, the processor <NUM> is implemented on an application specific integrated circuit (ASIC).

In some embodiments, the processor <NUM> is configured to implement a memory controller to access the NVM <NUM>. In other embodiments, the unitary SSD assembly <NUM> further comprises a memory controller (not shown) separate from the processor <NUM>.

In some embodiments, the unitary SSD assembly <NUM> further comprises volatile memory <NUM>, such as a random access memory (RAM) or another suitable volatile memory. The memory <NUM> is used for one or both of i) acting as a memory cache for the NVM <NUM>, and ii) temporarily storing messages, commands, etc., generated by, or to be processed by, the processor <NUM>, the messages, commands, etc., related to memory access operations, according to various embodiments. In some embodiments, the memory <NUM> is omitted from the unitary SSD assembly <NUM>.

At least when at least two components of the SSD assembly <NUM> are included in separate integrated circuit (IC) devices and mounted on one or more printed circuit boards (PCBs) (or another suitable substrate), the at least two components are communicatively coupled by a PCIe network <NUM>, which includes one or more PCIe busses and optionally one or more PCIe switches. For example, in an embodiment, the Ethernet network interface device <NUM>, the NVM <NUM>, and the processor <NUM> are communicatively coupled by the PCIe network. As another example, the Ethernet network interface device <NUM>, the NVM <NUM>, the processor <NUM>, and the memory <NUM> are communicatively coupled by the PCIe network, according to another embodiment.

In some embodiments, one or more components of the SSD assembly <NUM> are implemented on a single integrated circuit (IC) device within a unitary IC chip package. For example, the Ethernet network interface device <NUM> and the processor <NUM> are implemented on a single IC device within a unitary IC chip package, and the Ethernet network interface device <NUM> and the processor <NUM> are not communicatively coupled via the PCIe network <NUM> but rather via an interface circuit on the single IC, according to an embodiment. In another embodiment, the Ethernet network interface device <NUM> and the processor <NUM> are implemented on a multi-chip module (MCM) within a unitary IC chip package, and the Ethernet network interface device <NUM> and the processor <NUM> are not communicatively coupled via the PCIe network <NUM> but rather via an interface circuit on the MCM, according to an embodiment.

In another embodiment, the Ethernet network interface device <NUM>, the NVM <NUM>, the processor <NUM>, and the memory <NUM> (if included), are implemented on a single IC device within a unitary IC chip package, and the PCIe network <NUM> is omitted. In another embodiment, the Ethernet network interface device <NUM>, the NVM <NUM>, the processor <NUM>, and the memory <NUM> (if included), are implemented in an MCM within a unitary IC chip package, and the PCIe network <NUM> is omitted.

In another embodiment, the Ethernet network interface device <NUM>, the NVM <NUM>, and the processor <NUM> are implemented on a single IC device within a unitary IC chip package, and the PCIe network <NUM> is optionally omitted. In another embodiment, the Ethernet network interface device <NUM>, the NVM <NUM>, and the processor <NUM> are implemented in an MCM within a unitary IC chip package, and the PCIe network <NUM> is optionally omitted.

In other embodiments, two or more of the components of the unitary SSD assembly <NUM> are implemented using stacked IC technology and included within a unitary IC chip package. Similarly, in some embodiments, two or more of the components of the unitary SSD assembly <NUM> are implemented using package on package (PoP) technology.

The unitary SSD assembly <NUM> also comprises an Ethernet connector <NUM> coupled to the Ethernet network interface device <NUM>. In various embodiments, the Ethernet connector <NUM> comprises an RJ45 connector, an M8 connector, an M12 connector, or another suitable Ethernet connector.

In some embodiments, one or more IC chip packages and/or PoP modules that include at least the Ethernet network interface device <NUM>, the NVM <NUM>, the processor <NUM>, and the memory <NUM> (if included), are mounted on a unitary substrate <NUM>. In some embodiments that include the unitary substrate <NUM>, the Ethernet connector <NUM> is also mounted on the unitary substrate <NUM>. In other embodiments, the Ethernet connector <NUM> is not mounted on the unitary substrate <NUM>. In an embodiment, the unitary substrate <NUM> comprises a PCB. In other embodiments, the unitary substrate <NUM> comprises another suitable substrate that is configured for mounting one or more IC chip packages and/or PoP modules.

In some embodiments, two or more IC chip packages and/or PoP modules that include at least two of the Ethernet network interface device <NUM>, the NVM <NUM>, the processor <NUM>, and the memory <NUM> (if included), are mounted on two or more suitable substrates.

In some embodiments, the unitary SSD assembly <NUM> includes a housing that contains the Ethernet network interface device <NUM>, the NVM <NUM>, the processor <NUM>, the PCIe network <NUM> (if included), and the memory <NUM> (if included). <FIG> is a diagram of an example housing <NUM> of the unitary SSD assembly <NUM>, according to an embodiment. In an embodiment, the housing <NUM> encloses, at least partially, the Ethernet network interface device <NUM>, the NVM <NUM>, the processor <NUM>, the PCIe network <NUM> (if included), and the memory <NUM> (if included).

In an embodiment, the housing <NUM> defines an aperture <NUM>. The aperture <NUM> is shaped to accommodate the Ethernet connector <NUM>, according to an embodiment. In some embodiments, the housing <NUM> also defines one or more other apertures (not shown) that serve various other purposes such as one or more of: i) accommodating other connectors (not shown), ii) providing ventilation, iii) providing access to components within the housing, etc..

As discussed above, although <FIG> were described in the context of the built-in network interface device of the unitary SSD assembly comprising an Ethernet network interface device, in other embodiments the built-in network interface device of the unitary SSD assembly <NUM> comprises another suitable network interface device such as an FC network interface device configured to communicatively couple to an FC network fabric, an InfiniBand network interface device configured to communicatively couple to an InfiniBand network fabric, etc. In embodiments in which the unitary SSD assembly <NUM> comprises an FC network interface device, the connector <NUM> is a suitable FC connector such as a Lucent connector (LC) connector, a multiple-fiber push on (MPO) connector, or another suitable FC connector, and the aperture <NUM> is shaped to accommodate the FC connector. In embodiments in which the unitary SSD assembly <NUM> comprises an InfiniBand network interface device, the connector <NUM> is a suitable InfiniBand connector, and the aperture <NUM> is shaped to accommodate the InfiniBand connector.

In some embodiments, an SSD with a built-in network interface device is enclosed (at least partially) in a housing having a suitable form factor, such as a standard hard disk drive (HDD)/SSD form factor as defined by the Storage Networking Industry Association (SNIA), such as a <NUM>-inch form factor (e.g., with maximum dimensions substantially equal to (i.e., within <NUM>% of) a width of <NUM> inches, a depth of <NUM> inches, and height of <NUM> to <NUM> inches), a <NUM>-inch long form factor (e.g., with maximum dimensions substantially equal to (i.e., within <NUM>% of) a width of <NUM> inches, a depth of <NUM> inches, and height of <NUM> to <NUM> inches), a <NUM>-inch short form factor (e.g., with maximum dimensions substantially equal to (i.e., within <NUM>% of) a width of <NUM> inches, a depth of <NUM> inches, and height of <NUM> to <NUM> inches), a 1U short form factor (e.g., with maximum dimensions substantially equal to (i.e., within <NUM>% of) a width of <NUM> inches, a depth of <NUM> inches, and height of <NUM> inches), a 1U long form factor (e.g., with maximum dimensions substantially equal to (i.e., within <NUM>% of) a width of <NUM> inches, a depth of <NUM> inches, and height of <NUM> inches), a <NUM>-inch form factor (e.g., with maximum dimensions substantially equal to (i.e., within <NUM>% of) a width of <NUM> inches, a depth of <NUM> inches, and height of <NUM> to <NUM> inches), or a <NUM>-inch form factor (e.g., with maximum dimensions substantially equal to (i.e., within <NUM>% of) a width of <NUM> inches, a depth of <NUM> inches, and height of <NUM> inches).

In some embodiments, the housing has a maximum width of <NUM> inches, a maximum depth of <NUM> inches, and a maximum height of <NUM> inches.

In other embodiments, the housing has suitable dimensions than those discussed above.

In some embodiments, the unitary SSD assembly <NUM> does not include a housing such as the housing <NUM> of <FIG>. For example, in an embodiment, components of the unitary SSD assembly <NUM> such as described above (e.g., the Ethernet network interface device <NUM>, the NVM <NUM>, the processor <NUM>, the PCIe network <NUM> (if included), the memory <NUM> (if included), and the connector <NUM>) are mounted on a single unitary substrate such as a PCB. In another embodiment, a bracket is mounted to the single unitary substrate, the bracket shaped for mounting the unitary SSD assembly <NUM> to another PCB, to a housing of another device such as a server, a storage subsystem, etc. In some embodiments, the bracket defines an aperture that accommodates the connector <NUM>. In some embodiments in which the single unitary substrate is mounted to the bracket, the connector <NUM> is mounted to the bracket and is not mounted to the single unitary substrate. In some embodiments, the PCB has a maximum width of <NUM> inches and a maximum depth of <NUM> inches. In other embodiments, the PCB has other suitable dimensions.

In another embodiment, the single unitary substrate is mounted within a frame (comprising metal, plastic, etc.). In an embodiment, the frame is configured for mounting the unitary SSD assembly <NUM> to another PCB, to a housing of another device such as a server, a storage subsystem, etc. In some embodiments, the frame defines an aperture that accommodates the connector <NUM>. In some embodiments in which the single unitary substrate is mounted within a frame, the connector <NUM> is mounted to the frame and is not mounted to the single unitary substrate. In some embodiments, the frame has a maximum width of <NUM> inches and a maximum depth of <NUM> inches. In other embodiments, the PCB has other suitable dimensions.

As yet another example, in an embodiment, components of the unitary SSD assembly <NUM> such as described above (e.g., the Ethernet network interface device <NUM>, the NVM <NUM>, the processor <NUM>, the PCIe network <NUM> (if included), the memory <NUM> (if included), and the connector <NUM>) are mounted within a frame to form a unitary component. In an embodiment, the unitary component is configured for mounting to another PCB, to a housing of another device such as a server, a storage subsystem, etc. In some embodiments in which the components of the unitary SSD assembly <NUM> are mounted within a frame, two or more of the components of the unitary SSD assembly <NUM> are mounted to two or more PCBs, and the two or more PCBs are communicatively coupled such as by cables, ribbon cables, wires, etc.; the two or more PCBs are mounted within the frame to form the unitary component. In some embodiments, the frame has a maximum width of <NUM> inches and a maximum depth of <NUM> inches. In other embodiments, the PCB has other suitable dimensions.

Referring again to <FIG>, the Ethernet network interface device <NUM> is associated with one or more network addresses that uniquely identify the unitary SSD assembly <NUM> within a communication network, according to an embodiment. The network address is a suitable logical or physical address, such as a media access control (MAC) address, an Internet Protocol (IP) address, etc. In embodiments in which the network interface device <NUM> is an FC network interface device, the one or more network addresses include one or more FC addresses. In embodiments in which the network interface device <NUM> is an InfiniBand network interface device, the one or more network addresses include one or more local identifiers (LIDs).

In some embodiments, the network interface device <NUM> is not a general purpose network interface device <NUM>, but rather is a reduced functionality (e.g., as compared to a general purpose network interface device) network interface that comprises network interface front end interface logic configured with functionality to interface a protocol specialized for accessing storage devices over a network fabric (e.g., NVMe-oF, NVMe over TCP (NVMe/TCP), etc.) with a network fabric protocol such as Ethernet, Fibre Channel, Infiniband, etc. In some embodiments, the network interface device <NUM> comprises Ethernet to NVMe-oF (or NVMe/TCP) front end interface logic. In other embodiments, the network interface device <NUM> comprises FC to NVMe-oF (and/or FC over TCP/IP (FCIP) front end interface logic. In other embodiments, the network interface device <NUM> comprises InfiniBand to NVMe-oF (and/or TCP/IP over InfiniBand (IPoIB) front end interface logic.

The Ethernet network interface device <NUM> is configured to implement an Ethernet communication protocol (e.g., providing services up to an including a data link layer in the Open Systems Interconnection (OSI) model), according to an embodiment. In an embodiment, the Ethernet network interface device <NUM> is configured to receive, from the processor <NUM>, a data unit that is to be forwarded to a network fabric (e.g., an Ethernet fabric), encapsulate the data unit in an Ethernet frame, and transmit the Ethernet frame to the network fabric via the connector <NUM>. Additionally, the Ethernet network interface device <NUM> is configured to receive, from the network fabric (e.g., the Ethernet fabric) and via the connector <NUM>, an Ethernet frame, and determine whether a destination network address in the Ethernet frame matches a network address of the unitary SSD assembly <NUM>, according to an embodiment. The Ethernet network interface device <NUM> is further configured to, when the destination network address in the Ethernet frame matches the network address of the unitary SSD assembly <NUM>, decapsulate a data unit from the Ethernet frame and forward the data unit to the processor <NUM>, e.g., via the PCIe network <NUM> (when present).

In embodiments in which the network interface device <NUM> comprises an FC network interface device, the FC network interface device <NUM> is configured to implement an FC communication protocol, according to an embodiment. In an embodiment, the FC network interface device <NUM> is configured to receive, from the processor <NUM>, a data unit that is to be forwarded to a network fabric (e.g., an FC fabric), encapsulate the data unit in an FC frame, and transmit the FC frame to the network fabric via the connector <NUM>. Additionally, the FC network interface device <NUM> is configured to receive, from the network fabric (e.g., the FC fabric) and via the connector <NUM>, an FC frame, and determine whether a destination network address in the FC frame matches a network address of the unitary SSD assembly <NUM>, according to an embodiment. The FC network interface device <NUM> is further configured to, when the destination network address in the FC frame matches the network address of the unitary SSD assembly <NUM>, decapsulate a data unit from the FC frame and forward the data unit to the processor <NUM>, e.g., via the PCIe network <NUM> (when present).

In embodiments in which the network interface device <NUM> comprises an InfiniBand network interface device, the InfiniBand network interface device <NUM> is configured to implement an InfiniBand communication protocol, according to an embodiment. In an embodiment, the InfiniBand network interface device <NUM> is configured to receive, from the processor <NUM>, a data unit that is to be forwarded to a network fabric (e.g., an InfiniBand fabric), encapsulate the data unit in an InfiniBand packet, and transmit the InfiniBand packet to the network fabric via the connector <NUM>. Additionally, the InfiniBand network interface device <NUM> is configured to receive, from the network fabric (e.g., the InfiniBand fabric) and via the connector <NUM>, an InfiniBand packet, and determine whether a destination network address in the InfiniBand packet matches a network address of the unitary SSD assembly <NUM>, according to an embodiment. The InfiniBand network interface device <NUM> is further configured to, when the destination network address in the FC frame matches the network address of the unitary SSD assembly <NUM>, decapsulate a data unit from the InfiniBand packet and forward the data unit to the processor <NUM>, e.g., via the PCIe network <NUM> (when present).

In an embodiment, the processor <NUM> is configured to implement one or more communication protocol layers above the communication protocol layer(s) implemented by the network interface device <NUM>, the one or more communication protocol layers implemented by the processor <NUM> for facilitating the transfer of the data between the NVM <NUM> and an external device (external to the unitary SSD assembly <NUM>), via a network fabric coupled to the connector <NUM>. For example, the processor <NUM> implements an NVMe driver for converting NVMe commands and/or data to PCIe bus signals, and converting PCIe bus signals to NVMe commands and/or data, according to an embodiment. In some embodiments in which the unitary SSD assembly <NUM> omits a PCIe network, NVMe driver converts NVMe commands/data to signals for accessing the NVM <NUM>, and converts signals from the NVM <NUM> to NVMe commands/data.

As another example, the processor <NUM> implements a Remote Direct Memory Access (RDMA) protocol for transferring data to or from the NVM <NUM>, according to an embodiment. In various embodiments, the processor <NUM> implements RDMA over converged Ethernet (RoCE), RDMA over TCP/IP, etc..

The processor <NUM> implements an NVMe over Fabrics (NVMe-oF) protocol for facilitating transferring data between the NVM <NUM> with an external device (external to the unitary SSD assembly <NUM>), via a network fabric coupled to the connector <NUM>. The processor <NUM> uses the NVMe-oF protocol to establish a logical connection with an external device (external to the unitary SSD assembly <NUM>), via a network fabric coupled to the connector <NUM>. The processor <NUM> uses the NVMe-oF protocol to exchange commands and/or data (associated with accessing memory devices) with an external device (external to the unitary SSD assembly <NUM>), via a network fabric coupled to the connector <NUM>.

In various other embodiments, the processor <NUM> additionally or alternatively implements one of, or any suitable combination of two or more of: i) RDMA over the User Data Protocol (UDP) as a transport layer, ii) iWARP (a networking protocol that implements RDMA over the Transmission Control Protocol (TCP) and the Internet Protocol (IP) (TCP/IP)), iii) NVMe over TCP as a transport layer (NVMe/TCP), etc..

Associated with transferring data from the NVM <NUM> to a memory in an external device (e.g., external to the unitary SSD assembly <NUM>), the processor <NUM> uses the NVMe-oF protocol to open a logical connection to the external device over the network fabric. The logical connection is an NVMe-oF transport layer connection based on Ethernet, Fibre Channel, InfiniBand, etc., according to various illustrative embodiments. Alternatively, the connection <NUM> is based on RDMA, RoCEv2, iWARP, NVMe/TCP, or another suitable protocol, in other embodiments. The connection is opened prior to transfer of data from the NVM <NUM>, in an embodiment.

As an illustrative example, the Ethernet interface device <NUM> receives an Ethernet packet from the network fabric via the connector <NUM>. The Ethernet network interface device <NUM> decapsulates a capsule from the Ethernet packet, and provides the capsule to the processor <NUM>. The capsule contains one or more commands such as an NVMe read command, data to be stored in the NVM <NUM>, scatter gather lists (SGLs) which indicate network addresses of data in NVM <NUM> to be retrieved, etc. In response to contents of the capsule, the processor <NUM> (implementing the NVMe-oF protocol, the NVMe/TCP protocol, etc.) generates PCIe signals for accessing the NVM <NUM> according to the contents of the capsule. In some embodiments in which the PCIe network <NUM> is omitted, the processor <NUM> generates suitable signals for accessing the NVM <NUM> according to the contents of the capsule.

As another example, the processor <NUM> (implementing the NVMe-oF protocol, the NVMe/TCP protocol, etc.) generates a capsule (or a TCP/IP frame) that contains data read from the NVM <NUM>. The processor <NUM> provides the capsule to the Ethernet network interface device <NUM>, which generates an Ethernet packet that includes the capsule. The Ethernet interface device <NUM> then transmits the Ethernet packet to the network fabric via the connector <NUM>.

<FIG> is a diagram of a storage system <NUM> that utilizes unitary SSD assemblies such as the unitary SSD assembly <NUM> of <FIG>. The storage system <NUM> comprises a host <NUM> coupled to an interconnect <NUM> which in turn is coupled to a storage subsystem <NUM> having a network fabric <NUM> and three unitary SSD assemblies <NUM>, <NUM>, and <NUM>. The interconnect <NUM> comprises a network such as a local area network (LAN), a wide area network (WAN), or another suitable interconnect or network that communicatively couples the host <NUM> and the storage subsystem <NUM>. The interconnect <NUM> includes one or more intermediary devices, such as one or more interface switches and/or routers, coupled through wired and/or wireless interconnections, according to some embodiments.

One or both of the storage subsystem <NUM> and the host <NUM> are located in a rack for mounting multiple electronic modules, according to an embodiment. The rack includes multiple mounting slots referred to as bays, each designed to hold a hardware unit such as the host <NUM> or the storage subsystem <NUM>. In some examples, the rack may include a top-of-rack switch which provides connectivity between the hardware units and a remote network and/or hardware units in other racks. In an embodiment, the interconnect <NUM> comprises a backplane of the rack. Further, more than one storage subsystem <NUM> and/or more than one host <NUM> are coupled to the interconnect <NUM>, in various embodiments. Similarly, the storage subsystem <NUM> is communicatively coupled to another storage subsystem and/or another host in another rack (not shown), such as via a top-of-rack switch, according to some embodiments.

The host <NUM> includes any type of host, such as a computer processor or a network of computers and/or processors, according to various embodiments. Further, the host <NUM> is not necessarily limited to a single host device, and may represent a plurality of host devices. In an embodiment, the host <NUM> includes a memory <NUM> in the form of dynamic random access memory (DRAM) and/or other suitable memory, a processor <NUM>, such as a central processing unit (CPU), and a network interface card (NIC) <NUM>. The processor <NUM> is implemented on one or more IC devices and is configured to execute machine readable instructions stored in the memory <NUM> (or another memory (not shown) to perform arithmetical, logical, input/output (I/O) and other operations. To facilitate storage of data in the storage subsystem <NUM>, the host <NUM> utilizes the NIC <NUM> to access memory in the storage subsystem <NUM>. The NIC <NUM> facilitates transferring data over the interconnect <NUM> between the host <NUM> and the storage subsystem <NUM>. In various embodiments, the NIC <NUM> comprises an Ethernet network interface device, an FC network interface device, an InfiniBand network interface device, etc..

The unitary SSD devices <NUM>, <NUM>, <NUM> are configured to establish logical connections with the host <NUM> via the network fabric <NUM>, according to an embodiment. For example, as illustrated in <FIG>, the processor <NUM> establishes a logical connection <NUM> with the unitary SSD device <NUM> via the network fabric <NUM>. For example, the processor <NUM> establishes the logical connection <NUM> with the unitary SSD device <NUM> using NVMe-oF, NVMe/TCP, etc., in various embodiments, or using other suitable protocols in other embodiments.

The storage system <NUM> also includes a storage subsystem <NUM> coupled to the interconnect <NUM>. The storage subsystem <NUM> comprises a unitary SSD assembly <NUM> (e.g., corresponding to the unitary SSD assembly <NUM> or another suitable unitary SSD assembly with a built-in network interface device), communicatively coupled to a network fabric <NUM>. The host <NUM> is configured to establish a logical connection with the unitary SSD assembly <NUM> and to transfer data via the network fabric <NUM>, according to an embodiment.

<FIG> is a diagram of an example Ethernet frame <NUM> that the unitary SSD assembly <NUM> is configured to generate and transmit, and receive and process, according to an embodiment. The frame <NUM> includes an Ethernet Layer <NUM> header <NUM> and an Ethernet type field <NUM> which indicate that the frame <NUM> is associated with an Ethernet protocol. In an embodiment, the header <NUM> includes a transmitter MAC address (e.g., a MAC address of a device is transmitting the Ethernet frame <NUM>) and a receiver MAC address (e.g., a MAC address of a device that is to receive the Ethernet frame <NUM>). The example frame <NUM> may include an IP header <NUM> which indicates a source IP network address and a destination IP network address. The source IP network address may be an IP address of a device that transmitted the frame <NUM> and the destination IP network address may be an IP address of a device to where the frame <NUM> is to be sent. The MAC addresses and/or the IP network addresses may facilitate directing the frame from the unitary SSD assembly <NUM> or to the SSD assembly via a network fabric such as the network fabric <NUM>/<NUM> (<FIG>). In some embodiments, the MAC addresses and/or the IP network addresses in the Ethernet frame <NUM> include a network address of a unitary SSD assembly. A payload <NUM> of the Ethernet frame <NUM> includes an NVMe-oF capsule <NUM> which includes NVMe commands, responses, SGLs, etc., associated with the transfer of the data to/from a unitary SSD assembly <NUM>. The frame <NUM> may include other data as well such as port IDs associated with a logical connection between the unitary SSD assembly <NUM> and another network device to facilitate the routing of the frame to or from unitary SSD assembly over the logical connection.

<FIG> is a flow diagram of a method <NUM> for accessing an NVM of a unitary SSD assembly, such as the unitary SSD assembly discussed above with reference to <FIG>. The method <NUM> is implemented by the unitary SSD assembly <NUM> of <FIG>, and the method <NUM> is described with reference to <FIG>. In other embodiments, the method <NUM> is implemented by another suitable unitary SSD assembly with a built-in network interface device configured to communicate via a network fabric.

At block <NUM>, a network interface device (e.g., an Ethernet network interface device, an FC network interface device, an InfiniBand network interface device, etc.) of the unitary SSD assembly receives a packet (e.g., an Ethernet frame, an FC frame, an Infiniband packet, etc.) that includes information related to accessing the NVM. In an embodiment, the packet is received from a network fabric (e.g., an Ethernet fabric, an FC fabric, an InfiniBand fabric, etc.) communicatively coupled to the network interface device.

In an embodiment, the packet is received at block <NUM> via a network connector (e.g., an Ethernet connector, an FC connector, an InfiniBand connector, etc.) coupled to the network interface device.

At block <NUM>, the Ethernet network interface device of the unitary SSD assembly determines that a destination network address in a header of the packet received at block <NUM> matches a network address of the unitary SSD assembly.

At block <NUM>, in response to determining at block <NUM> that the destination network address in the header of the packet received at block <NUM> matches the network address of the unitary SSD assembly, the network interface device decapsulates a data unit from the packet received at bloc <NUM>. In an embodiment, the data unit decapsulated from the packet corresponds to a communication protocol for accessing storage devices. In an embodiment, data unit decapsulated from the packet corresponds to the NVMe-oF protocol. In another embodiment, data unit decapsulated from the packet corresponds to the RDMA protocol. In another embodiment, data unit decapsulated from the packet corresponds to the RCoE protocol. In another embodiment, data unit decapsulated from the packet corresponds to the NVMe protocol.

At block <NUM>, also in response to determining at block <NUM> that the destination network address in the header of the packet received at block <NUM> matches the network address of the unitary SSD assembly, a processor of the unitary SSD assembly uses information in the data unit to access the NVM.

In some embodiments, the NVM, the processor and the network interface device are integrated onto a unitary substrate, and using information in the data unit to access the NVM at block <NUM> comprises retrieving data from the NVM and transferring the retrieved data from the NVM to the processor via the unitary substrate without first transferring the retrieved data to any component not integrated onto the unitary substrate.

In some embodiments, the unitary substrate is a single PCB, and the NVM, the processor, and the network interface device are mounted on the single PCB, and transferring the retrieved data from the NVM to the processor via the unitary substrate comprises transferring the retrieved data from the NVM to the processor via the single PCB without first transferring the retrieved data to any component not mounted onto the single PCB.

In some embodiments, the NVM, the processor, and the network interface device are implemented on multiple IC chiplets, and the multiple IC chiplets are integrated onto the unitary substrate in an MCM; and transferring the retrieved data from the NVM to the processor via the unitary substrate comprises transferring the retrieved data from the NVM to the processor within the MCM without first transferring the retrieved data to any component outside of the MCM.

In some embodiments, the NVM, the processor, and the network interface device are included within a housing; and transferring the retrieved data from the NVM to the processor comprises transferring the retrieved data from the NVM to the processor without first transferring the retrieved data to any component outside of the housing.

In some embodiments, the NVM, the processor, and the network interface device are implemented on a single IC; and transferring the retrieved data from the NVM to the processor comprises transferring the retrieved data from the NVM to the processor without first transferring the retrieved data to any component not implemented on the single IC.

In some embodiments, the NVM, the processor, and the network interface device are implemented as at least two ICs in a multi-chip module (MCM); and transferring the retrieved data from the NVM to the processor comprises transferring the retrieved data from the NVM to the processor without first transferring the retrieved data to any component not in the MCM.

In some embodiments, the NVM, the processor, and the network interface device are implemented as at least two integrated circuits (ICs) in a unitary package on package (PoP) integrated circuit device; and transferring the retrieved data from the NVM to the processor comprises transferring the retrieved data from the NVM to the processor without first transferring the retrieved data to any component not in the PoP integrated circuit device.

The method <NUM> further comprises: generating, at the processor, a further data unit corresponding to the communication protocol for accessing storage devices, the further data unit generated to include data retrieved from the NVM; providing, by the processor, the further data unit to the Ethernet network interface device; encapsulating, at the network interface device, the further data unit within a further packet (e.g., a further Ethernet frame, a further FC frame, a further Infiniband packet, etc.); and transmitting, by the network interface device, the further packet to the network fabric.

<FIG> is a simplified diagram of the unitary SSD assembly <NUM> with a built-in network interface device. The unitary SSD assembly <NUM> is a more specific embodiment of the unitary SSD assembly <NUM> of <FIG>, and like-numbered elements are not described in detail for purposes of brevity. Although <FIG> is discussed in the context of the Ethernet protocol, the unitary SSD assembly <NUM> is suitable for use with other network fabric protocols such as FC, InfiniBand, etc., according to various embodiments.

In some embodiment, the method <NUM> of <FIG> is implemented by the unitary SSD assembly <NUM> of <FIG>. In other embodiments, the unitary SSD assembly implements other suitable methods other than the method <NUM>.

In the example unitary SSD assembly <NUM>, the built in network interface device comprises an Ethernet network interface device <NUM>, and <FIG> is described in the context of an Ethernet network for ease of explanation. In other embodiments, however, the built in network interface device comprises another suitable network interface device configured to communicatively couple to another suitable network fabric, such as a FC network interface device configured to communicatively couple to an FC network fabric, an InfiniBand network interface device configured to communicatively couple to an InfiniBand network fabric, etc..

The unitary SSD assembly <NUM> comprises a processor <NUM>. In an embodiment, the processor <NUM> corresponds to the processor <NUM> of <FIG>. The processor <NUM> comprises a plurality of hardware logic components including NVMe-oF to NVMe converter logic <NUM>, NVMe front end interface logic <NUM>, and SSD controller logic <NUM>. The NVMe-oF to NVMe converter logic <NUM> is configured to implement NVMeoF protocol functions and to encapsulate/decapsulate NVMe commands/responses/data to/from NVMe-oF capsules, according to an embodiment. In another embodiment, the processor <NUM> comprises NVMe/TCP to NVMe converter logic <NUM> is configured to implement NVMe/TCP protocol functions and to encapsulate/decapsulate NVMe commands/responses/data to/from TCP/IP frames, according to an embodiment.

The NVMe front end interface logic <NUM> is configured to convert NVMe commands/responses/data to/from signals useable by the SSD controller <NUM>, according to an embodiment. The SSD controller <NUM> is a suitable SSD controller configured to write data to and read data from the NVM memory <NUM>.

In some embodiments, the SSD assembly also comprises a control processor <NUM> that is configured to perform functions such as administrative handling, monitoring of the network interface device <NUM> and/or the processor <NUM>, set up of the network interface device <NUM>, and/or the processor <NUM>, and/or the NVM <NUM>, etc. The control processor <NUM> is included as a component of the processor <NUM>, in an embodiment. The control processor <NUM> is a separate component from the processor <NUM>, in another embodiment. The control processor <NUM> is configured to execute machine readable instructions stored in a memory (not shown) coupled to and/or integrated with the control processor <NUM>. The machine readable instructions, when executed by the control processor <NUM>, cause the control processor <NUM> to perform functions such as administrative handling, monitoring of the network interface device <NUM> and/or the processor <NUM>, set up of the network interface device <NUM>, and/or the processor <NUM>, and/or the NVM <NUM>, etc..

At least some of the various blocks, operations, and techniques described above may be implemented utilizing hardware, a processor executing firmware instructions, a processor executing software instructions, or any combination thereof. When implemented utilizing a processor executing software or firmware instructions, the software or firmware instructions may be stored in any suitable computer readable memory such as a random access memory (RAM), a read only memory (ROM), a flash memory, etc. The software or firmware instructions may include machine readable instructions that, when executed by one or more processors, cause the one or more processors to perform various acts.

Claim 1:
A unitary solid state drive, SSD, assembly (<NUM>), the unitary SSD assembly (<NUM>) comprising:
a non-volatile memory, NVM (<NUM>);
a processor (<NUM>) communicatively coupled to the NVM;
a network interface device (<NUM>) communicatively coupled to the processor (<NUM>), wherein the network interface device is configured to communicate via a network fabric (<NUM>) according to a NVM express over Fabrics, NVMe-oF, communication protocol, and wherein the NVM (<NUM>), the processor (<NUM>), and the network interface device (<NUM>) are arranged in a unitary assembly; and
a network connector (<NUM>) coupled to the network interface device (<NUM>),
wherein the processor (<NUM>) is configured to implement the NVMe-oF communication protocol and characterized in that to use same to establish a logical connection to an external solid state memory over the network fabric (<NUM>), thereby enabling transfer of data between the NVM (<NUM>) and the external solid state memory without first transferring any of the data to a host computer.