Patent Description:
Storage architectures for network storage systems often employ "top-of-rack" (TOR) switches, such as TOR Ethernet, Fibre Channel or InfiniBand switches. (InfiniBand is a trademark of System I/O, Inc. ) The TOR switches can be expensive and consume rack space within the rack enclosure of the system. As such, there is a need for improved switching and storage system architectures, particularly, but not exclusively, for use with Ethernet, Fibre Channel or InfiniBand.

<CIT> discloses systems and methods for a storage adapter device for communicating with network storage. In some implementations, the storage adapter device comprises a host interface. In these implementations, the host interface may be configured to communicate with a host device using a local bus protocol. <CIT> discloses a bridge device for connecting a USB <NUM> host device with a plurality of downstream, non-USB <NUM> mass storage devices, such as SATA or PATA devices. <CIT> discloses service nodes that may be ASICs that can provide the functionality of a switch or a router. The service nodes can be configured in a circular replication chain, thereby providing benefits such as high reliability.

In some aspects, the switching component and the protocol interface are configured for use with one or more of Ethernet, Fibre Channel and InfiniBand. The one or more storage devices may include, for example, protocol-specific storage components (such as Ethernet attached storage, Fibre Channel attached storage, or InfiniBand attached storage), servers, or switch attached components within the rack enclosure. In some aspects, the switching component includes: one or more high port count switches configured to provide the TOR switching; and one or more low port count switches configured for product connectivity.

In some aspects, a high port count switch comprises: a protocol-specific switch having a high port count connector for connecting to an input data pipe providing data channels in accordance with a particular protocol; a protocol-specific channel-to-high speed serial converter having high port count serial lane connectors for connecting to the protocol-specific switch; and a high speed serial switch having high port count serial lane connectors for connecting to the protocol-specific channel-to-high speed serial converter and to external downstream components. The protocol-specific channel-to-high speed serial converter comprises, in some examples, an Ethernet-to-nonvolatile memory express (NVMe) bridge, a Fibre Channel-to-NVMe bridge, or an InfiniBand-to-NVMe bridge. The high speed serial switch may be, for example, a Peripheral Component Interconnect Express (PCIe) switch. A baseboard management controller (BMC) is connected to the protocol-specific switch of the high port count switch.

In some aspects, a low port count switch comprises: a protocol-specific switch having a low port count connector for connecting to an input data pipe providing data channels in accordance with a particular protocol; a protocol-specific channel-to-high speed serial converter having low port count serial lane connectors for connecting to the protocol-specific switch; and a high speed serial switch having low port count serial lane connectors for connecting to the protocol-specific channel-to-high speed serial converter and to external downstream components.

In some aspects, the protocol interface comprises a personality module configured as a mezzanine card that includes one or more mezzanine connectors. The mezzanine card may be configured, in some examples, for co-planar mounting to a printed circuit board assembly (PCBA) that includes the switching component of the IOM and, in other examples, for vertical mounting to the PCBA.

In some aspects, one or more servers, storage components, and switch-attached components are coupled to the IOM and coupled to one another. In some aspects, a plurality of IOMs are provided along with a plurality of servers.

In another aspect, a method is provided for use with an IOM of a network storage system. The method includes: routing data through a switching component configured to provide TOR switching of the data to determine a particular device destination for the data within the network storage system; and routing the data through a protocol interface configured to convert a protocol of the data from an input data protocol to a protocol for use with the particular device destination of the data.

In some aspects of the method, the switching component and the protocol interface are configured for use with one or more of Ethernet, Fibre Channel and InfiniBand, and routing the data through the protocol interface is performed to convert a protocol of the data from a particular one of Ethernet, Fibre Channel and InfiniBand to a protocol for use with the particular device destination of the data, such as NVMe or PCIe.

In yet another aspect, an apparatus is provided for use within a network storage system, the apparatus comprising: means for routing data through a switching component of an IOM configured to provide TOR switching of the data to determine a particular device destination for the data within the network storage system; and means for routing the data through a protocol interface configured to convert a protocol of the data from an input data protocol to a protocol for use with the particular device destination of the data.

The word "exemplary" or "embodiment" is used herein to mean "serving as an example, instance, or illustration. " Any implementation or aspect described herein as "exemplary" or as an "embodiment" is not necessarily to be construed as preferred or advantageous over other aspects of the disclosure. Likewise, the term "aspects" does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

Embodiments will now be described in detail with reference to the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the aspects described herein. It will be apparent, however, to one skilled in the art, that these and other aspects may be practiced without some or all of these specific details. In addition, well known steps in a method of a process may be omitted from flow diagrams presented herein in order not to obscure the aspects of the disclosure. Similarly, well known components in a device may be omitted from figures and descriptions thereof presented herein in order not to obscure the aspects of the disclosure.

Storage architectures for network storage systems often employ "top-of-rack" (TOR) switches, such as TOR Ethernet, Fibre Channel or InfiniBand switches.

<FIG> illustrates a network storage system (or "rack") <NUM> that has TOR switches. Briefly, the network storage system <NUM> includes two traditional TOR switches <NUM><NUM> and <NUM><NUM>, a set of standard servers <NUM><NUM>, <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, a set of attached storage components <NUM><NUM>, <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, (such as Ethernet attached storage, Fibre Channel attached storage, or InfiniBand attached storage), and a set of additional switch attached components <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>. The attached storage devices <NUM><NUM>, <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may be Network Attached Storage (NAS) devices such as NAS drives or NAS systems, which may be configured to act, e.g., as centralized network storage for use over a local network. NAS devices are often attached over a Local Area Network (LAN) using Ethernet but can be configured for use with other physical transports such as InfiniBand or Fibre Channel. The additional switch attached components <NUM><NUM>, <NUM><NUM>, and <NUM><NUM> may be any other devices attached to the other components of the rack, such as routers or modems. Data pipes <NUM> are connected to the TOR switches <NUM><NUM> and <NUM><NUM> to provide input/output from the network storage system <NUM>. The TOR switches <NUM><NUM> and <NUM><NUM> are in turn connected to one another and to the standard servers <NUM><NUM>, <NUM><NUM>, <NUM><NUM> and <NUM><NUM> via intermediate lines <NUM>. The standard servers <NUM><NUM>, <NUM><NUM>, <NUM><NUM> and <NUM><NUM> are in turn connected to one another and to attached storage components <NUM><NUM>, <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, and to the additional switch attached components <NUM><NUM>, <NUM><NUM>, and <NUM><NUM> the via lines <NUM>. With this arrangement, three data paths or pipes (<NUM>, <NUM>, and <NUM>) are provided.

In accordance with aspects of the present disclosure, network storage architectures are provided wherein the functionality of the TOR switch is moved into an input/output (I/O) module (IOM) of the system. Aspects of the present disclosure cover Fibre Channel, InfiniBand and Ethernet switching. Exemplary IOMs described herein provide both the protocol interface for the back-end media devices within the enclosure (e.g., IOM functionality) and TOR switch functionality as well. This configuration allows switching to be dis-aggregated from the TOR switch and distributed throughout the data network of the rack. This, in turn, also allows for the reduction in size/quantity and/or removal of the TOR from the overall system. By "dis-aggregated" it is meant that, instead of providing a TOR that provides substantially all of the primary switching in the TOR, the switching is distributed among the data network components of the rack, using architectures and components described in detail below.

Aspects of the present disclosure cover both low port count and high port count systems. A low port count system is suited for expansion (or downstream) enclosures, where limited port counts are required. The high port count switches may be best suited for "Head" (or upstream) enclosures where the TOR functionality is being replaced. Both low and port count switches can be used for optimal implementation of this solution.

Another aspect of the present disclosure is the capability to plug a mezzanine card onto the base IOM to make the carrier/switch board support any available device back end. The IOM supports high-speed switch (e.g., Fibre Channel, Ethernet of InfiniBand) connections from the switch to a connector system (e.g., mezzanine, back plane, card edge of the small outline dual in-line memory module (SODIMM) style) and a similar connector system that allows the card to interface to the system. The personality module functionality can be populated as a mezzanine style card using any of the connector systems noted above. The mezzanine connector system allows a personality module to be connected between the back end device interface and the front-end switch system. This essentially provides a configurable interface system that supports a protocol agnostic enclosure. A protocol agnostic enclosure that supports a configurable protocol based on the incorporation of a unique plug in personality module is thus provided for use within the enterprise storage space.

The personality module functionality can also be populated by hard soldering onto the carrier card assembly without the use of the mezzanine interface. In this approach, the functions of the personality module are still performed but not on a replaceable mezzanine card. In the hard soldered solution, the card is replaced to support other interfaces, but the dis-aggregated switch functionality noted above is maintained.

Aspects of the disclosure cover both single board and multiple board implementations to provide a configurable interface system that supports a protocol agnostic enclosure. An exemplary implementation described below incorporates multiple Ethernet to nonvolatile memory express (NVMe) bridge devices within the personality card function. Other implementations are described below as well. According to the invention the disclosure includes connecting a baseboard management controller (BMC) to the switch directly, thus allowing all switch connected devices to access the BMC without a dedicated management port. Additional aspects of the disclosure relate to the population of a switch chip (e.g., Ethernet, Fibre Channel or InfiniBand) onto a Peripheral Component Interconnect Express (PCIe) plug in card. This applies to both industry standard form factors (e.g., half height, half length (HHHL), full height, half length (FHHL), full height, full length (FHFL), half height, full length (HHFL)) and non-industry standard form factors.

Notably, many of these aspects of the disclosure allow using the PCIe plug in card as a true switch, not a NIC (Network Interface Card). This differs from implementations intended to be host ports to provide data connections to a server or host device. Such implementations are typically connected to a PCIe bus and provide data to the host over that bus. Herein, instead, the switch card (in at least some examples) only obtains power from that PCIe slot. The switch operates as a stand-alone switch, primarily receiving power from the PCIe slot. No connections over the PCIe lanes are needed. Auxiliary data management connections may be made within the PCIe slot, but those auxiliary connections may be, e.g., for environmental management only.

<FIG> illustrates a network storage system (rack) <NUM> with an IOM configured for use with Ethernet to provide for dis-aggregated switching. The network storage system <NUM> includes, in this particular example, two high port count dis-aggregated switches <NUM><NUM>-<NUM><NUM>, three low port count dis-aggregated switches <NUM><NUM>-<NUM><NUM>, a set of standard servers <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>, a pair of Ethernet attached storage components <NUM><NUM> and <NUM>, and a set of additional switch attached components <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>. Data pipes/paths <NUM> are connected to the dis-aggregated switches <NUM><NUM>-<NUM><NUM> and <NUM><NUM>-<NUM><NUM> to provide input/output from the network storage system <NUM>. The dis-aggregated switches <NUM><NUM>-<NUM><NUM> and <NUM><NUM>-<NUM><NUM> are in turn connected via data paths/pipes <NUM> to one another and to the other components of the rack <NUM>: the standard servers <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>, the Ethernet attached storage components <NUM><NUM> and <NUM><NUM>, and the additional switch attached components <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>. With this configuration, two data paths/pipes (<NUM> and <NUM>) are provided (i.e., one fewer that the rack <NUM> of <FIG>, which does not include dis-aggregated switches).

In the rack system <NUM> of <FIG>, the dis-aggregated switches <NUM><NUM>-<NUM><NUM> and <NUM><NUM>-<NUM><NUM> incorporate Ethernet switching only. No separate Ethernet TOR switches are required. The high port count dis-aggregated switches <NUM><NUM> - <NUM><NUM> thus replace the separate TOR Ethernet switches. The other switches of the rack <NUM> may be low port count switches used from product connectivity. Note that data pipes <NUM> provide the data In/Out for the rack <NUM>. The data pipes <NUM> provide for data traffic internal to the rack (i.e., traffic among the various illustrated components). In contrast with the rack <NUM> of <FIG>, which employs three data pipes/paths, the rack <NUM> employs only two data pipes/paths. Notably, whereas the rack <NUM> of <FIG> employs intermediate data pipes/paths <NUM> to couple the standard servers <NUM><NUM>, <NUM><NUM>, <NUM><NUM> and <NUM><NUM> to the other components of rack <NUM>, the rack <NUM> can omit the intermediate data pipes/paths. The features illustrated in this figure and in the other figures described herein are merely illustrative. For example, the number of servers or other components shown in the figures are merely exemplary. Also, the number of TOR components of predecessor systems that can be replaced using the systems and architectures described herein are merely exemplary.

<FIG> illustrates a network storage system (rack) <NUM> with an IOM configured for use with InfiniBand to provide for dis-aggregated switching. The network storage system <NUM> has an architecture similar to the architecture of network storage system <NUM> and hence will only briefly be described. The network storage system <NUM> includes, in this particular example, two high port count dis-aggregated switches <NUM><NUM>-<NUM><NUM>, three lowport count dis-aggregated switches <NUM><NUM>-<NUM><NUM>, a set of standard servers <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>, a pair of InfiniBand attached storage components <NUM><NUM> and <NUM><NUM>, and a set of additional switch attached components <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>. Data pipes/paths <NUM> to provide input/output from the network storage system <NUM>. Data paths/pipes <NUM> provide coupling among the components of the rack <NUM>. No separate InfiniBand TOR switches are required.

<FIG> illustrates a network storage system (rack) <NUM> with an IOM configured for use with Fibre Channel to provide for dis-aggregated switching. The network storage system <NUM> has an architecture similar to the architecture of network storage system <NUM> and hence will only briefly be described. The network storage system <NUM> includes, in this particular example, two high port count dis-aggregated switches <NUM><NUM>-<NUM><NUM>, three low port count dis-aggregated switches <NUM><NUM>-<NUM><NUM>, a set of standard servers <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>, a pair of Fibre channel attached storage components <NUM><NUM> and <NUM><NUM>, and a set of additional switch attached components <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>. Data pipes/paths <NUM> to provide input/output from the network storage system <NUM>. Data paths/pipes <NUM> provide coupling among the components of the rack <NUM>.

<FIG> illustrates a network storage system (rack) <NUM> with an IOM configured for use with Ethernet, InfiniBand, and Fibre Channel to provide for dis-aggregated switching. The network storage system <NUM> includes, in this particular example, two high port count dis-aggregated switches <NUM><NUM>-<NUM><NUM>, three low port count dis-aggregated switches <NUM><NUM>-<NUM><NUM>, a set of standard servers <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>, an Ethernet attached storage component <NUM><NUM>, an InfiniBand attached storage component <NUM><NUM>, a Fibre channel attached storage component <NUM><NUM>, and a pair of additional switch attached components <NUM><NUM> and <NUM><NUM>. Data pipes/paths <NUM> to provide input/output from the network storage system <NUM>. Data paths/pipes <NUM> provide coupling among the components of the rack <NUM>. In the rack <NUM> of <FIG>, the dis-aggregated switches <NUM><NUM>-<NUM><NUM> and <NUM><NUM>-<NUM><NUM> incorporate Ethernet, Fibre Channel and/or InfiniBand Switching with multiple switches per IOM. No separate Ethernet, Fibre Channel and/or InfiniBand TOR switches are required.

<FIG> is a bock diagram of a general purpose switch card or fabric card <NUM> incorporating features of the IOMs of <FIG>, wherein a BMC is connected to the Ethernet/Fibre channel/InfiniBand switch directly, allowing all switch connected devices to access the BMC without a dedicated management port. The fabric card <NUM> includes an Ethernet/Fibre channel and/or InfiniBand switch <NUM>, an Ethernet/InfiniBand/Fibre Channel-to-high speed serial integrated circuit <NUM> (which may be a combination of separate integrated circuits (ICs)), and a high speed serial switch <NUM>. A BMC <NUM> is connected to the Ethernet/Fibre channel and/or InfiniBand switch <NUM>. The connection of the BMC <NUM> to the switch <NUM> allows management to be performed over in-band high speed links instead of dedicated management ports. Note that the dashed boxes in <FIG> represent possible combinations for personality module circuitry incorporating the components. A first dashed box <NUM> corresponds to a configuration for personality module circuitry that collectively incorporates each of the illustrated switch and circuit components. A second dashed box <NUM> corresponds to a configuration for personality module circuitry that incorporates the switch <NUM>, the integrated circuit <NUM>, and the BMC <NUM> but excludes the high speed serial switch <NUM>. A third dashed box <NUM> corresponds to a configuration for personality module circuitry that incorporates integrated circuit <NUM>, the BMC <NUM>, and the high speed serial switch <NUM> but excludes the switch <NUM>. A fourth dashed box <NUM> corresponds to a configuration for personality module circuitry that incorporates just the integrated circuit <NUM>.

<FIG> also illustrates exemplary data transfer data pipes or paths (each have N total lanes). A set of N Ethernet/Fibre channel and/or InfiniBand lanes <NUM> connect the Ethernet/Fibre channel and/or InfiniBand switch <NUM> to external upstream components. A set of N high speed serial lanes <NUM> connect switch <NUM> to IC <NUM>. A set of N high speed serial lanes <NUM> connect IC <NUM> to high speed serial switch <NUM>. A set of N high speed serial lanes <NUM> connect high speed serial switch <NUM> to external downstream components.

<FIG> is a bock diagram of a low port count switch card or fabric card <NUM> incorporating features of the general purpose IOM fabric card of <FIG> for an Ethernet example. The low port count device of <FIG> is well-suited for expansion (or downstream) enclosures, where limited port counts are required, and may be used as the low port count dis-aggregated switches <NUM><NUM>-<NUM><NUM> of <FIG>. The low port fabric card <NUM> includes an Ethernet switch <NUM>, a set of four Ethernet to NVMe bridges <NUM><NUM>-<NUM><NUM>, and a pair of <NUM> lane "generation <NUM>" PCIe switches <NUM><NUM>-<NUM><NUM>. A BMC <NUM> is connected to the Ethernet switch <NUM>. A dashed block <NUM> indicates that the Ethernet switch <NUM> and the set of four Ethernet to NVMe bridges <NUM><NUM>-<NUM><NUM> may be configured as one device or chip set. Among other functions, the Ethernet switch <NUM> determines the particular downstream device (e.g., a particular storage drive) for receiving particular portions of input data (e.g., particular data packets) and routes those particular portions of the data to the downstream device. Since the various storage drives are connected to the Ethernet switch <NUM> via different bridges <NUM><NUM>-<NUM><NUM>, the Ethernet switch <NUM> determines, as part of its processing, which particular bridge to route particular portions (packets) of the data through. Hence, among other features, the Ethernet switch <NUM> provides a means for receiving input data in a first protocol from a plurality of input connectors and for determining particular means for bridging from among a plurality of means for bridging to receive particular portions of the input data. A dashed block <NUM> indicates that the PCIe switches <NUM><NUM>-<NUM><NUM> may be configured as another separate device or chip set. <FIG> also illustrates exemplary data transfer data pipes or paths. A set of eight <NUM> gigabit (Gb) Ethernet lanes <NUM> connect the Ethernet switch <NUM> to external upstream components. A set of four <NUM> Gb lanes <NUM> connect switch <NUM> to the NVMe bridges <NUM><NUM>-<NUM><NUM>. A set of four X16 lanes <NUM> connect NVMe bridges <NUM><NUM>-<NUM><NUM> to the two PCIe switches <NUM><NUM>-<NUM><NUM>. A set of four 28X lanes <NUM> connect the two PCIe switches <NUM><NUM>-<NUM><NUM> to external downstream components. In the particular example of <FIG>, each of the lanes <NUM> can be coupled to fourteen drives. A similar configuration as shown in <FIG> can be provided for use with InfiniBand or Fibre Channel. Note that the switches <NUM><NUM>-<NUM><NUM> need not be generation <NUM> PCIe switches, but other generations, e.g., generation <NUM> and beyond, or other, non-PCIe switches.

<FIG> is a bock diagram of a high port count switch card or fabric card <NUM> incorporating features of the general purpose fabric card of <FIG> for an Ethernet example. The high port count device of <FIG> is well-suited for Head (or upstream) enclosures, where the TOR functionality is being replaced, and may be used as the high port count dis-aggregated switches <NUM><NUM>-<NUM><NUM> of <FIG>. (Both low and high port count switches may be used in overall systems as shown in <FIG>) The high port fabric card <NUM> includes an Ethernet switch <NUM>, a set of four Ethernet to NVMe bridges <NUM><NUM>-<NUM><NUM>, and a pair of <NUM> lane "generation <NUM>" PCIe switches <NUM><NUM>-<NUM><NUM>. A BMC <NUM> is connected to the Ethernet switch <NUM>. A dashed block <NUM> indicates that the Ethernet switch <NUM> and the set of Ethernet to NVMe bridges <NUM><NUM>-<NUM><NUM> may be configured as one device or chip set. A dashed block <NUM> indicates that the PCIe switches <NUM><NUM>-<NUM><NUM> may be configured as another separate device or chip set. <FIG> also illustrates exemplary data transfer data pipes or paths. A set of N <NUM> Gb Ethernet lanes <NUM> connect the Ethernet switch <NUM> to external upstream components. A set of four <NUM> Gb lanes <NUM> connect switch <NUM> to the NVMe bridges <NUM><NUM>-<NUM><NUM>. A set of four X16 lanes <NUM> connect NVMe bridges <NUM><NUM>-<NUM><NUM> to the two PCIe switches <NUM><NUM>-<NUM><NUM>. A set of four 28X lanes <NUM> connect the two PCIe switches <NUM><NUM>-<NUM><NUM> to external downstream components. In the particular example of <FIG>, each of the 28X lanes <NUM> can be coupled to fourteen NVM drives. Thus, whereas the low port count example of <FIG> provides for eight connections into the Ethernet switch <NUM>, the high port count example of <FIG> provides for N connections into the Ethernet switch <NUM> (where N can be any practical number but is typically substantially larger than <NUM> and, in some examples, may be <NUM>). Note also that in an embodiment not covered by the scope of the claims, a BMC need not be provided and so, in some implementations, a device with all of the components of <FIG> may be provided while omitting the BMC or incorporating the BMC functionality into other components.

<FIG> illustrates a low port count system <NUM> that includes six IOMs. Three of the IOMs, denoted IOM A <NUM><NUM>-<NUM><NUM>, are connected to one another and to a set of three servers <NUM><NUM>-<NUM><NUM> via the various connection lines/paths shown in the figure. The other three of the IOMs, denoted IOM B <NUM><NUM>-<NUM><NUM>, are connected to one another and to the set of three servers <NUM><NUM>-<NUM><NUM> via the various connection lines/paths shown in the figure, some of which are <NUM> connectors (where G is another designator for gigabit (Gb)) and others are <NUM> connectors, such as Ethernet connectors. This configuration provides for distributed switching while providing relatively low cost with enhanced resiliency. The configuration also eliminates the need for convention TOR switches, as discussed above.

As explained above, a mezzanine card may be plugged onto the base IOM of a system to enable a carrier/switch board to support any available device back end. The IOM may support high-speed switch (Fibre Channel, Ethernet of InfiniBand) connections from the switch to a connector system (mezzanine, back plane, card edge of SODIMM style) and a similar connector system that allows the card to interface to the system. This personality module functionality can be populated as a mezzanine style card using any of the connector systems described above. The mezzanine connector system thus allows a "personality module" to be connected between the back end device interface and the front-end switch system so as to provide a configurable interface system that supports a protocol agnostic enclosure. Note also that single board and multiple board implementations are provided (to provide the configurable interface system that supports the protocol agnostic enclosure).

<FIG> illustrates a first exemplary protocol agnostic IOM <NUM>, which includes a switch (Ethernet, InfiniBand, or Fibre Channel) <NUM> and a personality module <NUM>. The personality module <NUM> may include, for example, various CPU, GPU, NIC, SAS, SATA, RAID, Ethernet, NVMe, Fibre Channel, InfiniBand, any protocol or compute interface components so as to provide a protocol agnostic system that can accommodate a wide variety of protocols). The personality module <NUM> includes a first mezzanine connection <NUM> for connecting the personality module <NUM> to the switch <NUM>, and a second mezzanine connection <NUM> for connecting the personality module <NUM> to other components (not shown). As shown, the switch may have N ports <NUM> for use as a host interface. The first mezzanine connection <NUM> may accommodate an equal number of high speed serial lanes <NUM> for connecting to the switch <NUM>. The second mezzanine connection <NUM> may accommodate an equal number of high speed serial lanes <NUM> for connecting to the other components (via, for example, a midplane/driveplane connector, discussed below). In this particular example, the switch fabric may be configured on an IOM PCBA (with the IOM PCBA shown in <FIG>).

<FIG> illustrates a second exemplary protocol agnostic IOM <NUM>, which includes a switch (Ethernet, InfiniBand, or Fibre Channel) <NUM> and a personality module <NUM>. The personality module <NUM> may include, for example, various CPU, GPU, NIC, SAS, SATA, RAID, Ethernet, NVMe, Fibre Channel, InfiniBand, any protocol or compute interface components so as to provide a protocol agnostic system that can accommodate a wide variety of protocols). In this example, a first mezzanine connection <NUM> is provided between the switch <NUM> and its N connection lines <NUM>. A second mezzanine connection <NUM> is provided on the personality module <NUM> for connecting the personality module <NUM> to other components (not shown). The personality module <NUM> may accommodate an equal number of high speed serial lanes <NUM> for connecting to the switch <NUM>. The second mezzanine connection <NUM> may accommodate an equal number of high speed serial lanes <NUM> for connecting to the other components. In this particular example, the switch fabric may be configured with switch circuitry on a personality module card (with the PM card shown in <FIG>).

<FIG> illustrates a two examples of the physical structure of a protocol agnostic IOM that includes a personality module as shown in <FIG> and <FIG>. In a first exemplary physical structure <NUM>, a personality module card <NUM> is mounted co-planar with an IOM PCBA <NUM> (i.e., mounted over the IOM PCBA and parallel with the IOM PCBA). A pair of mezzanine connectors <NUM> and <NUM> (e.g., connectors <NUM> and <NUM> of <FIG>) connect the personality module card <NUM> to the IOM PCBA <NUM>. N host interface connector <NUM> is provided on one end of the IOM PCBA <NUM>. A midplane/driveplane connector <NUM> is provided at the other end of the IOM PCBA <NUM>, which may provide N high speed serial lanes (as shown in <FIG>). In a second exemplary physical structure <NUM>, a personality module card <NUM> is mounted perpendicular to an IOM PCBA <NUM>. A pair of mezzanine connectors <NUM> and <NUM> (e.g., connectors <NUM> and <NUM> of <FIG>) connect opposing sides of the personality module card <NUM> to the IOM PCBA <NUM>. N host interface connector <NUM> is again provided on one end of the IOM PCBA <NUM>. A midplane/driveplane connector <NUM> is again provided at the other end of the IOM PCBA <NUM>, which may provide N high speed serial lanes (as shown in <FIG>).

<FIG> illustrates another co-planar physical structure <NUM> having a personality module card <NUM> mounted co-planar with an IOM PCBA <NUM>. In this example, a card edge connector <NUM> connects the personality module card <NUM> to the IOM PCBA <NUM>. N host interface connector <NUM> is provided on one end of the IOM PCBA <NUM>. A midplane/driveplane connector <NUM> is provided at the other end of the IOM PCBA <NUM>, which may provide N high speed serial lanes (as shown in <FIG> and <FIG>). In some examples, the card edge connector <NUM> can be at a right angle with the personality module <NUM> and mounted in space above the IOM PCBA <NUM>. In other examples, the card edge connector <NUM> can be co-planar with the personality module card <NUM> mounted in a cut out in the IOM PCBA <NUM>.

As discussed above, the switch chip (e.g., Ethernet, Fibre Channel or InfiniBand) may be populated onto a PCIe plug in card and may be used in connection with industry standard form factor (e.g., HHHL, FHHL, FHFL, HHFL) as well as non-industry standard form factors. The PCIe plug in card may be used a true switch, not a NIC (Network Interface Card). As noted, other implementations are often intended to be host ports to provide data connections to a server or host device and are typically connected to the PCIe bus to provide data to the host over that bus. In the devices of the next two figures, the switch card obtains its power from the PCIe slot. The switch thus operates as a stand-alone switch, primarily receiving power from the PCIe slot. No connections over the PCIe lanes are needed. Auxiliary data management connections may be made within the PCIe slot but (in at least some examples) the auxiliary data management connections are for environmental management only.

<FIG> illustrates an exemplary physical structure for a PCIe-based switch module PCBA <NUM>. The switch module PCBA <NUM> includes a switch <NUM> (e.g., Ethernet, InfiniBand, or Fibre Channel) mounted in the center of the PCBA <NUM>, with various additional devices also mounted to the PCBA <NUM>. In this example, the additional devices include a power control and conversion device <NUM>, a monitoring device <NUM> for monitoring signals, and supporting circuitry <NUM>, <NUM>, and <NUM>. Various interconnection lines are shown that connect the components to one another and to input/output connectors, including a set of <NUM>. N host connectors <NUM> and a set of <NUM>. N internal switch port connectors <NUM>.

<FIG> illustrates another exemplary physical structure for a PCIe-based switch module PCBA <NUM>. The switch module PCBA <NUM> includes a switch <NUM> (e.g., Ethernet, InfiniBand, or Fibre Channel) mounted in the center of the PCBA <NUM>, with various additional devices also mounted to the PCBA <NUM>. In this example, the additional devices are generally represented as supporting circuitry <NUM>. Interconnection lines are not shown in the illustration of <FIG>, but similar connections as in <FIG> may be used to connect the various components to one another and to input/output connectors, including a set of <NUM>. N host connectors <NUM> and a set of <NUM>. N internal switch port connectors <NUM>.

<FIG> is a perspective view of an exemplary embodiment of a fabric box <NUM> with an integrated switch which may be configured in accordance with the various features of <FIG>, discussed above. Switch ports are integrated into the fabric modules. The device distributed switching through the rack system (not shown) it is installed in, and eliminates the need for a TOR switch (of the type shown in <FIG>).

<FIG> is a block diagram illustrating an IOM <NUM> for use within a network storage system. The IOM <NUM> includes a switching component <NUM> configured to provide TOR switching for data to be routed from input connectors to one or more storage devices within a rack enclosure. The IOM <NUM> also includes a protocol interface <NUM> configured to convert a protocol of the data from an input data protocol to a protocol for use with the one of more storage devices within the rack enclosure. Examples of these components are described above in detail above with reference to <FIG>. As discussed in those examples, the network storage system may be a rack-based system wherein the various components of the system, including the IOM <NUM> and the various storage devices, are mounted within the enclosure of the rack.

<FIG> is a flow chart illustrating a method <NUM> that may be performed by the IOM <NUM> of <FIG> or other suitably-equipped systems, devices or apparatus. Briefly, at <NUM>, an IOM of a network storage system routes data through a switching component of the IOM that is configured to provide TOR switching of the data to determine a particular device destination for the data within the network storage system. At <NUM>, the IOM then routes the data through a protocol interface of the IOM that is configured to convert a protocol of the data from an input data protocol to a protocol for use with the particular device destination of the data. The switching component may be, for example, switching component <NUM> of <FIG> and the protocol interface may be, for example, the protocol interface <NUM> of <FIG>. Examples of the operations of <FIG> are described in greater detail with reference to <FIG>.

<FIG> is a flow chart further illustrating a method <NUM> that may be performed by the IOM <NUM> of <FIG> or other suitably-equipped systems, devices or apparatus. Briefly, at <NUM>, an IOM of a rack-based network storage system routes data through a switching component of the IOM that is configured to provide TOR switching of Ethernet, Fibre Channel and/or InfiniBand protocol data to determine a particular device destination for the data within the network storage system, such as a particular protocol-specific storage component, server, or switch attached components (mounted within a rack enclosure) that uses nonvolatile memory express (NVMe) and/or Peripheral Component Interconnect Express (PCIe) or other protocols. At <NUM>, the IOM routes data through a switching component of the IOM that is configured to provide TOR switching of the data to determine a particular device destination for the data within the network storage system. At <NUM>, the IOM then routes the data through a protocol interface of the IOM that is configured to convert the protocol of the data (Ethernet, Fibre Channel and/or InfiniBand) to a protocol for use with the particular device destination of the data, such as NVMe and/or PCIe or other protocols. The switching component may be, for example, switching component <NUM> of <FIG> and the protocol interface may be, for example, the protocol interface <NUM> of <FIG>. Examples of the operations of <FIG> are described in detail above with reference to <FIG>.

<FIG> is a block diagram illustrating an IOM <NUM> for use within a network storage system. The IOM <NUM> includes a first switching component <NUM> configured to receive input data from a plurality of input connectors for routing to a plurality of storage devices of the network storage system and further configured to determine particular bridge components from among a plurality of bridge components to receive particular portions of the input data. The IOM <NUM> also includes a plurality of bridge components <NUM> configured to receive data from the first switching component <NUM> in a first protocol and convert the data to a second protocol for use with particular storage devices of the plurality of storage devices. The IOM <NUM> also includes at least one second switching component <NUM> configured to receive data in the second protocol from at least one of the plurality of bridge components <NUM> for routing to particular storage devices among the plurality of storage devices. Examples of these components are described above in detail above with reference to <FIG>. As discussed in those examples, the network storage system may be a rack-based system wherein the various components of the system, including the IOM <NUM> and the various storage devices, are mounted within the enclosure of the rack.

Claim 1:
An input/output module, IOM (<NUM>), configured for insertion into a mounting slot of a rack enclosure of a network storage system, the IOM comprising:
a first switching component (<NUM>) configured to receive input data from a plurality of input connectors for routing to a plurality of storage devices (<NUM>) of the network storage system;
a plurality of bridge components (704_1 - 704_4) configured to receive data from the first switching component in a first protocol and to convert the data to a second protocol for particular storage devices of the plurality of storage devices;
wherein the first switching component is configured to determine particular bridge components from among the plurality of bridge components to receive particular portions of the input data;
at least one second switching component (<NUM>) configured to receive data in the second protocol from at least one of the plurality of bridge components for routing to particular storage devices among the plurality of storage devices; and
a baseboard management controller, BMC, (<NUM>, <NUM>, <NUM>) directly connected to the first switching component.