Adjustable data storage drive module carrier assembly

A data storage sled is provided. The data storage sled includes a circuit card assembly comprising connectors that couple to drive modules and a host connector for coupling the data storage sled to an external connector, an enclosure comprising hinged covers each configured to cover apertures in the enclosure thorough which individual ones of the drive modules can be inserted into an associated connector on the circuit card assembly. The data storage sled also includes drive module mounting assembles configured to hold the individual ones of the drive modules into the associated connector by at least including movable mounting features to accommodate varying lengths among the individual ones of the drive modules.

BACKGROUND

Computer systems typically include bulk storage systems, such as magnetic disk drives, optical storage devices, tape drives, or solid state storage drives, among other storage systems. As storage needs have increased in these computer systems, networked storage systems have been introduced which store large amounts of data in a storage environment physically separate from end user computer devices. These networked storage systems typically provide access to bulk data storage over one or more network interfaces to end users or other external systems. In addition to storage of data, remote computing systems include various processing systems that can provide remote computing resources to end users. These networked storage systems and remote computing systems can be included in high-density installations, such as rack-mounted environments.

However, as the densities of networked storage systems and remote computing systems increase, various physical limitations can be reached. These limitations include density limitations based on the underlying storage technology, such as in the example of large arrays of rotating magnetic media storage systems. These limitations can also include computing density limitations based on the various physical space requirements for network interconnect as well as the large space requirements for environmental climate control systems.

Bulk storage systems are traditionally limited in the number of devices that can be included per host and also have large physical space requirements due to the separate physical packaging of individual storage drives, which can be problematic in storage environments where higher capacity, redundancy, and reliability is desired. These shortcomings can be especially pronounced with the increasing data storage and retrieval needs in networked, cloud, and enterprise environments.

OVERVIEW

A data storage sled is provided. The data storage sled includes a circuit card assembly comprising connectors that couple to drive modules and a host connector for coupling the data storage sled to an external connector, an enclosure comprising hinged covers each configured to cover apertures in the enclosure thorough which individual ones of the drive modules can be inserted into an associated connector on the circuit card assembly. The data storage sled also includes drive module mounting assembles configured to hold the individual ones of the drive modules into the associated connector by at least including movable mounting features to accommodate varying lengths among the individual ones of the drive modules.

In another example, a data storage assembly comprising a number of removable data storage sleds is presented. The data storage assembly includes guide fins configured to partition the data storage assembly into a plurality of bays each configured to hold an associated data storage sled, and pins included in each of the plurality of bays to engage ejection springs of the data storage sleds.

DETAILED DESCRIPTION

The examples herein discuss various data storage assemblies, such as data storage sleds. These data storage sleds are configured to carry many drive modules, such as solid state drive (SSD) cards, and couple these SSD cards to a host interface. In many examples herein, the SSD cards and host interface comprise PCIe interfaces. The data storage sleds can be inserted and removed in a data storage system which employs a PCIe fabric for switching PCIe traffic among a plurality of data storage sleds, processing modules, and network interfaces.

As a first example of a data storage sled,FIG. 1is presented.FIG. 1is a diagram illustrating storage sled100. Storage sled100includes various features and elements, andFIG. 1illustrates at least storage sled body110, drive module covers111, cover hinges112, body fasteners113, cover closure features114, side grip features115, electromagnetic interference (EMI) contacts116, EMI shield117, sled host connector119, and eject button131. Although not shown inFIG. 1for clarity and due to the selected view, storage sled100also includes other features and elements as detailed in subsequent figures and discussion below. The mechanical and structural elements of storage sled100and associated mechanical and structural elements inFIGS. 1-11can comprise machined, forged, cast, extruded, or other manufactured elements. These elements can be made of metals, metallic compounds, polymers, plastics, ceramics, or other materials, including combinations and variations thereof.

Storage sled100is configured to carry one or more drive modules, such as assembly drive module120inFIG. 4, among other drive modules. In this example, up to four drive modules can be held by storage sled100, with one carrier assembly covered by an associated cover111after being inserted into storage sled100. Each of covers111can be opened by a user or operator using associated ones of cover closure features114. Each of covers111can be hinged at hinge112which allow a user to rotate the associated cover outward from sled100to gain access to an aperture in which a drive module is inserted and removed. In some examples, hinge112is spring-loaded and upon pressing a latch or friction-fit feature of closure114, hinge112allows cover111to open outward. When spring-loaded, hinge112can provide resistance to closure by a user and a latch or mating features of cover closure114to keep cover111closed and flush with body110.

Storage sled body110is composed of at least two shell portions, namely a top body and a bottom body (such as elements801-802inFIG. 8). The at least two body portions enclose a printed circuit board, sled insertion elements, and drive module mounting elements, and are secured together with fasteners113, such as screws, rivets, clips, welds, and the like. Storage sled body110is configured to be slid into associated mating features of a larger enclosure or storage assembly which can hold many storage sleds, as featured inFIGS. 10-11below. Latch features, such as pictured inFIG. 7, are included on the bottom of sled100and are configured to engage with associated latch mates within the larger storage assembly to provide purchase of storage sled100within the associated mounting rails or features as well as provide resistance to removal of storage sled100from the associated mounting features in the storage unit. Eject button131is coupled to the latch features to allow for ejection and removal of storage sled100from the storage assembly. Side grip features115allow for a human operator to grip storage sled100for insertion and removal from an associated storage assembly. Once slid into associated mounting rails, guides, or features of the larger storage assembly, sled host connector119is configured to electrically and mechanically mate with an associated external connector, such as a connector on a midplane or backplane PCB of the larger storage assembly.

Additionally,FIG. 1shows EMI contacts116and EMI shield117. EMI contacts116and EMI shield117are configured to electrically mate with at storage sled body110to form a Faraday cage with the top body and bottom body. EMI shield117is shown with holes or perforations of a predetermined size. This predetermined size can be driven by emission characteristics of the electrical components of storage sled100or by electrical interference provided by an environment that storage sled100is positioned within. EMI contacts116and EMI shield117can each be composed of metallic or conductive materials. When storage sled body110is composed of non-conductive materials, then a metallic layer, such as a coating or foil, can line inner or outer surfaces of storage sled body110to provide shielding, which electrically contacts EMI contacts116and EMI shield117. In some examples, EMI contacts116and EMI shield117are formed from the same piece of conductive material, such as a stamped metal part that is folded and welded or soldered together to form EMI contacts116and EMI shield117. In other examples, EMI contacts116and EMI shield117are formed from a machined metallic part or formed from separate pieces and electrically bonded together. EMI contacts116comprise one or more metallic fingers or springs which electrically contact both body110and an associated storage assembly into which storage sled100is inserted or into a rack-mounted assembly into which the storage assembly is inserted. Thus, body110and EMI shield117form a Faraday cage around electrical components of storage sled100and EMI contacts116are configured to touch external conductive surfaces and hold contact with the surfaces using a spring force formed by EMI contacts116extending outward from sled100.

FIG. 2is a diagram illustrating various views of storage sled100. Side view200shows storage sled100from one of the sides, and highlights some of the elements discussed inFIG. 1. Top view201shows storage sled100from the top, and also highlights some of the elements discussed inFIG. 1. Bottom view202shows storage sled100from the bottom, and also highlights some of the elements discussed inFIG. 1.

Additionally, the bottom view202illustrates additional features latch130and insertion spring118. Latch130is coupled to internal mechanism and eject button131. Latch130fits into an accompanying slot when storage sled100is inserted into a storage assembly, such as shown by sled latch holes1020ofFIG. 10. Ejection spring118is included an associated channel in storage sled100. When storage sled100is removed from a storage assembly, then spring118is expanded or uncompressed. When storage sled100is inserted into a storage assembly, a mating feature, such as a compression pin or protrusion, in the storage assembly contacts spring118and compresses spring118upon insertion of storage sled100. When eject button131is pressed and latch130is raised to allow storage sled100to be removed from the storage assembly, tension in compressed spring118aids removal of storage sled100from the storage assembly by pushing storage sled100outward from an associated bay in the storage assembly. Insertion of storage sled100in an incorrect orientation is prevented by the mating feature, such as the pin or protrusion, in the storage assembly.

A channel is included in the enclosure of storage sled100to allow for the mating feature or pin to engage spring118, but this channel is not found on the opposite side of sled100, and insertion upside-down (or backwards) in the storage assembly is prevented past a predetermined point. This prevents damage to connector119or to electrical and mechanical components to storage sled100by improper insertion. A further discussion of the latch and spring mechanisms are found inFIGS. 7-9.

FIG. 3is a diagram illustrating drive module120insertion into storage sled100. InFIG. 3, cover111is opened by pivoting around hinge112. In some examples, hinge112is coupled to a spring which holds cover111open when a user opens cover111. Cover111reveals aperture301into which module120is inserted. Module120has at least one edge connector121which is inserted into connector159in storage sled100. Module120is then pressed downward into aperture301and end mount151of module holder150secures to an end of module120opposite of connector121. End mount151is tensioned by a spring and allows for various lengths of drive modules. A finger of an operator can pull end mount151to compress the associated spring and allow for insertion of drive module120. Coarse adjust arm152also allows for various lengths of drive modules, but for larger adjustments than end mount151. Sled100can accommodate up to four drive modules120, although a different number can be employed. Further discussion of mounting of drive module120is discussed inFIGS. 4-5.

Drive module120includes solid state memory chips122, and edge connector121. In some examples, the solid state storage elements of drive module120include memory chips, such as chips122which can comprise flash memory, magnetic random access memory, phase change memory, memristors, or other solid state memory devices. Drive module120can comprise different form factors of drive modules. Drive module120can include other components and elements, such as mounting holes, processors, interface chips, passive components, holdup circuitry, among other components and elements. In some examples, drive module120comprises a M.2 solid state drive (SSD) module, which can have a variable dimensions, such as width and length, based on model, manufacturer, storage capacity, or embedded features, among other factors. Other examples include MiniPCI express cards, mini-serial ATA (mSATA) cards, and the like, with varying dimensions. For example, end mount151and coarse adjust features152and153allow for varying lengths of M.2 SSD modules, such as lengths of 16, 26, 30, 38, 42, 60, 80 and 110 millimeters (mm), among others. Advantageously, commodity or off-the-shelf M.2 SSD modules (or other drive modules) can be deployed in storage sled100using the adjustable mounting features.

Removal of drive module120from storage sled100is as follows. Cover111, flush with body110when closed, is opened using a fingernail or small item that engages cover closure114. Cover111is opened by a user, or by a spring associated with hinge112to expose drive module120in aperture301. A finger or small object can engage a grip on drive holder150to slide drive end mount151away from drive module120. Drive end mount151is spring or resistance loaded and provides force to hold drive module120into drive connector159. Drive module120can then be removed from associated connector159and from sled100. Advantageously, sled100allows for tool-less insertion and removal of many drive modules.

Thus, the drive module mounting assembly comprising at least drive holder150, slide drive end mount151, and arms152allow for coarse adjustment features and fine adjustment features. The coarse adjustment features are configured to select among standardized lengths of the drive modules and the fine adjustment features configured to hold an associated drive module in an associated connector and compensate for variability in the standardized lengths of the drive modules. As discussed herein, the coarse adjustment features comprise arms152on the drive module mounting assembles which locate into side channels153of the enclosure or body110of sled100to select among the standardized lengths of the drive modules. To provide specific examples of these coarse and fine adjustable mounts,FIGS. 4 and 5are presented.

FIG. 4is a diagram illustrating a drive module mounting mechanism400. InFIG. 4, drive module120can be mounted and held securely in storage sled100. Specifically, connector159is mounted onto an associated circuit board of storage sled100and connector121of drive module120is then inserted into connector159and held in place by adjustable mounting features of sled100.

FIG. 4shows drive holder150with drive end mount151, coarse adjustment arms152, and one or more springs154. Drive end mount151is configured to mate with an end or edge of drive module120. In some examples, drive module120includes a screw cutout124for mounting in systems which employ screwed mounting schemes. However, in sled100, a tool-less design is employed which does not require a screw to secure drive module120in sled100. Drive end mount151can engage this screw cutout for additional purchase onto drive module120, although this configuration is not required. Drive holder150is fitted into coarse adjust channels153on either side of sled100, and coarse adjust arms152can slide along the length of sled100within channels153.

Coarse adjust arm152also allows for various lengths of drive modules, typically for larger adjustments than end mount151. Fingers formed into channels153can set discrete coarse length selection points for arms152. For example,FIG. 4shows three coarse adjustment lengths indicated by the wider fingers formed into channels153. These coarse adjust features in channels153allow for varying discrete lengths of M.2 SSD modules, such as at least three discrete lengths comprising 110 mm, 80 mm, and 60 mm lengths, among others. More than three predetermined locations can be employed, such as when each finger sets a discrete length. When drive module120is not inserted, arms152can slide in channel153to adjust drive holder150into a desired discrete coarse length adjustments.

In some examples, arms152can locate into the pictured finger slots in channels153at predetermined locations to allow for the discrete length adjustments. Arms152can locate into the finger slots from above or below. In some examples, a predetermined number of widened finger slots155can be employed which set the coarse adjustment levels by matching widened slots155to a width of rectangular features on the ends of arms152.FIG. 3shows widened finger slots155, which are widened relative to the other finger slots. The non-widened finger slots are not required with a predetermined number of widened slots is employed. However, manufacturability of the body pieces can be aided in having the finger slots in some examples.

In further examples, screw157is configured to mate with one or more predetermined holes in body110to further secure drive holder150into any of the predetermined coarse adjustment lengths. In another example, screw157can engage into a channel in body110to allow for any number of coarse adjustment lengths along the channel. Arms152may be made flexible to enable a snapping feature of arms152to locate arms152into the finger slots, at least by providing by a spring force due to flexible action of arms152with fulcrum points where arms152attach to drive holder150.

End mount151is tensioned by a spring and allows for various fine length adjustments of drive modules. For example, although a drive module might be designed for a particular length, such as 110 mm, 80 mm, and 60 mm lengths, variation in actual drive modules installed into sled100might vary due to manufacturing tolerances, inaccuracies of reported specifications, or other variations. Coarse adjust arms152can be positioned for a particular length, but drive holder can provide purchase and tension onto drive module120to hold drive module into connector159. During insertion, a finger of an operator can pull end mount151to compress the associated spring and allow for insertion of drive module120. End mount151engages an end or edge of drive module120and when released by the operator spring154expands and places pressure onto drive module120to hold securely into connector159and in sled110.

FIG. 5provides a further detailed exploded view of the drive mount elements. Drive end mount151can be seen as a separate element which can slide with respect to a piece which houses springs154in spring channels154C and forms coarse adjust arms152. Further elements can be employed to hold drive holder150in a coarse position within channels153or sled120. Specifically, mounting screw157and mounting plate156can be employed to fasten drive holder150relative to a top or bottom body of sled110. Variations on these elements are possible, such as a different number or configuration of spring elements or fastener elements.

FIGS. 6 and 7provide a further view of storage sled100, namely sectioned internal views.FIG. 6illustrates at least two drive modules120mounted into storage sled100.FIGS. 6 and 7also provide a view of various sled insertion and ejection features. Sled latch130is attached to flexible spring member136which is fastened at one end to body110using one or more fasteners and free to flex at the other end, forming a spring lever action that can lift and lower latch130through an associated latch aperture in body110.

To insert storage sled100into a storage assembly, an operation can slide storage sled100into an associated slot of the storage assembly. Spring force provided by latch spring135keeps latch130in a normally lowered or extended position. However, a tapered or wedge shape on an entry surface of latch130(facing the direction of connector119) allows for insertion sliding action to compress spring135and raise latch130into body110of sled100when latch130contacts a surface of the storage assembly. When a proper depth of insertion is reached for sled100, a hole or aperture in the surface of the storage assembly allows for latch130to drop down and engage the aperture due to the spring force provided by latch spring135. The hole or apertures can be seen inFIG. 10, such as holes1020. Once inserted, removal or inadvertent ejection of sled100is prevented by latch130protruding into the hole or aperture of the surface of the storage assembly. Latch130can have a generally square or abrupt shape on an exit surface of latch130(facing the direction of eject button131) to resist ejection. Furthermore, ejection spring118is compressed upon insertion by a tab, pin, or other feature on the surface of the storage assembly on which sled100slides during insertion.

To eject storage sled100from a storage assembly, eject button131is able to be pushed by an operator, and consequently move eject lever134which can engage latch spring135. Latch spring135is raised by a wedge feature on eject lever134and thus moved by pushing eject button131. Movement of eject lever134and eject button131is opposed by a spring force provided by latch spring135and fasteners136. Flexing latch spring135upward also raises attached latch130, which then disengages with a mating aperture in a storage assembly into which storage sled100is inserted. The ejection process allows storage sled100to slide within a storage assembly into which sled100was previously inserted. Ejection spring118, previously compressed by an insertion process, provides ejection force for sled100to move outward from a mating connector of the storage assembly and the storage assembly itself.

In further examples, a lighted ejection or insertion indicator132can be coupled to eject button131. This indicator may be customized to appear as a logo or other shape or size. Various transparent or translucent features can allow light to be emitted from indicator132, such as glass, acrylic, or other polymer. In some examples, a light, such as a light emitting diode, can be included in indicator132which can indicate various properties of sled100. These properties can include insertion or ejection status, such as lit when properly inserted and unlit when ejected. These properties can include other functional properties, such as an operational status of elements of sled100. In the examples seen inFIG. 7, light pipe133is employed to carry light from indicator circuit137to indicator132. Light pipe133can comprise a fiber optic element or polymer light guide, among other elements. In examples where a light is mounted into indicator132, light pipe133can instead comprise one or more wires or signal links. Indicator circuit137can be communicatively coupled to processing elements of sled100, such as processor811.

FIG. 8illustrates an exploded view of storage sled100. Similar elements as discussed above are shown inFIG. 8, although variations are possible. Additionally, various electrical components are shown inFIG. 8. Specifically,FIG. 8includes a sled printed circuit board, indicated by circuit card assembly (CCA)810, which can mount up to four drive modules through associated apertures301into connectors159, two each on the top and bottom of CCA810. A CCA includes a printed circuit board along with attached electrical and mechanical components. CCA810can be an example of CCA1250inFIG. 12, although variations are possible. CCA810includes drive module connectors159, sled connector119, processor810, Peripheral Component Interconnect Express (PCIe) switch812, and holdup circuitry813. Although one CCA810is shown inFIG. 8which has four drive module connectors soldered thereto, other examples can employ individual CCAs for each drive module connector, or variations thereof. Further description of these elements is include inFIG. 12.

FIG. 9illustrates end views of storage sled100. Connector end view900shows EMI contacts116, sled connector119, ejection spring hole118H, and ejection spring channel118A, among other features. Sled connector119inserts into a mating connector of a backplane, midplane, or other CCA associated with a storage assembly, such as shown inFIG. 12. Ejection spring hole118H allow for a cavity into which ejection spring118can be held, and compressed/expanded. Channel118A allows for a pin or other raised feature on an associated storage assembly to engage spring118and to compress spring118during insertion. Channel118A also allows for only one orientation of insertion of sled100into a storage assembly, as the pin or other raised feature on the storage assembly would impact features of body110and prevent further insertion of sled100due to interference with the pin or other raised feature.

Outer end view901shows EMI contacts116, EMI shield117, indicator132, and eject button131, among other features. EMI shield117forms an air-porous electrical shield for components of sled100to provide for cooling airflow while attenuating or blocking predetermined frequency ranges of radiative electromagnetic energy.

FIG. 10illustrates storage assembly1000into which storage sleds can be inserted and structurally contained.FIG. 10shows an empty assembly, andFIG. 11shows a filled assembly. Assembly1000can hold up to eight storage sleds, such as storage sled100. Sled latch holes1020can engage sled latches for the insertion and ejection processes discussed above. Sled alignment fins1021provide channels for storage sleds to be inserted and inhibit side-to-side motion. Sled alignment fins1021also ensure proper alignment of storage sleds during insertion to provide for proper mating of storage sled electrical connectors to connectors associated with assembly1000. Assembly1000can be inserted into a further assembly or enclosure, such as a rackmount assembly or rackmount enclosure. Assembly handle1010can provide for user or operator insertion and removal of assembly1000in the rackmount enclosure, such as in a 3U enclosure. Assembly fastener1011secures assembly1000into the rackmount enclosure, and can include screws, snap features, latches, or other fasteners.

FIG. 11illustrates a fully assembled “octet” of storage sleds. EMI contacts116can be seen as protruding from each sled. In some examples, EMI contacts116electrically contact with conductive surfaces of assembly1000, while in other examples, EMI contacts116allow for electrical contact with conductive surfaces of the rackmount enclosure into which assembly1000is inserted. Combinations of configurations of EMI contacts116can be employed.

Although not required in all examples, heatsinks can be deployed onto various components of the examples discussed herein. The heatsinks generally each include a heat dissipating member which can allow for heat transfer of elements to the surrounding environment. In some examples each heatsink is coupled to bulk metal portions of body110of sleds100or to case portions of assembly1000. Multiple heat sinks can be employed for various chips and electrical components. Thermal interface elements can be employed, such as heat pads, glues, adhesives, thermal compounds, and the like, to better conduct heat from the associated components. In some examples, heatsinks are thermally coupled to assembly1000to further dissipate heat or spread heat along the body of assembly1000.

FIG. 12illustrates a block diagram of storage module1210, as an example of any of storage sleds110ofFIG. 13.FIG. 12also illustrates an example physical configuration of storage module1210as shown for circuit card assembly1250. Storage module1210includes four storage drives1211, PCIe switch1212, processor1220, power control module1221, and holdup circuit1222. Power control module1221distributes power to each element of storage module1210over associated power links1230-1232. Power control module1221can selectively enable/disable power for each power link. Further communication links can be included for intra-sled communication between the various elements of storage module1210.

PCIe switch1212communicates with an interconnect module (not pictured) over links1240. Links1240are included in a data plane for transferring user data, such as that discussed forFIG. 13. Each of links1240comprises a PCIe link with four lanes, namely a “x4” PCIe link. More than one PCIe link1240is provided for load balancing, redundancy, and failover protection for storage module1210. In some examples, PCIe switch1212has links1240connected to non-transparent (NT) interfaces or ports, where one or more host systems (such as a processor on a processing module) can interact with storage drives1211in a redundant or failover configuration. PCIe switch1212also communicates with four storage drives111over associated x4 PCIe links1241.

Processor1220communicates over at least sideband links1249. Sideband links1249can include Universal Serial Bus (USB), SMBus, JTAG, Inter-Integrated Circuit (I2C), controller area network bus (CAN), or any other communication interface, and in some examples is provided over portions of PCIe links1240. In this example, processor1220includes I2C interface1225and USB interface1226for communication over sideband links1249. I2C interface1225and USB interface1226can be included in separate circuitry or included in similar elements as processor1220. Processor1220and PCIe switch1212can communicate over an associated communication link1233, which can be an I2C or a PCIe link, among other link types.

Each storage drive1211comprises a solid state drive (SSD) in this example, and communicates with external systems over an associated PCIe interface included in each storage drive1211. The solid state storage media of storage drives1211can comprise flash memory, static RAM, NAND flash memory, NOR flash memory, memristors, or other solid state media. Instead of or in addition to solid state media, each storage drive1211can comprise magnetic storage, such as hard disk drives, tape drives, magnetoresistive memory devices, and the like, or can comprise optical storage, such as phase change memory. Each storage drive1211can receive read transactions and write transactions issued by a host system, such as a processor of a processing sled node. Responsive to a read transaction, storage drive1211can retrieve data identified by the read transaction and transfer the data for delivery to the associated host. Responsive to a write transaction, storage drive1211can write data that accompanies the write transaction to storage media associated with storage drive1211.

In some examples, each storage drive1211comprises a circuit card assembly (CCA) which is separate from CCA1250and with a mini-PCI Express connector or other connector that interfaces with a connector on CCA1250. CCA1250comprises one or more printed circuit boards1251that couple to the various elements of storage module1210. In other examples, each storage drive1211comprises one or more flash memory chips with a PCIe interface which is soldered onto CCA1250. In yet other examples, each storage drive1211comprises one or more separate solid state disk drives or magnetic hard disk drives along with associated enclosures and circuitry. PCIe switch1212comprises a PCIe cross connect switch for establishing switched connections between any PCIe interfaces handled by PCIe switch1212. In some examples, PCIe switch1212comprises a PLX Technology PEX8725 10-port, 24 lane PCIe switch chip.

Processor1220comprises one or more microprocessors, processing devices, multi-core processors, processing circuitry, or other processing system. Processor1220can include one or more non-transitory memory devices, such as RAM, solid state storage, or other memory to store instructions that are executable by processor1220to operate as discussed herein. In some examples, processor1220comprises an ARM microcontroller, ARM microprocessor, field-programmable gate array (FPGA), application specific integrated circuit (ASIC), application specific processor, or other microprocessor or processing elements. Processor1220can monitor usage statistics, traffic status, or other usage information through link1233. PCIe switch1212can track this usage information during normal operation and data transfer with storage drives1211, and processor1220can retrieve this usage information as needed over link1233.

Power control module1221includes circuitry to selectively provide power to any of the elements of storage module1210. Power control module1221can receive control instructions from processor1220or over any of PCIe links1240. In some examples, power control module1221comprises processing elements discussed above for processor1220, or is included in the elements of processor1220. Power control module1221can receive power over power link1223as a power source for the various elements of storage module1210. Holdup circuit1222includes energy storage devices for storing power received over power link1223for use during power interruption events, such as loss of source power. Holdup circuit1222can include capacitance storage devices, such as an array of capacitors. Further discussion of examples of power control circuitry is found below.

As seen inFIG. 12, arrows indicate a bidirectional power flow over link1223. Power can be accepted by module1210when input power is available, such as from a mating connector or midplane. Power can be redistributed to other modules by module1210over link1223when input power is not available, such as during power interruption events. When module1210is removed from a mating connector, then power can be bled off into associated power sink circuitry. Although one power link1223is shown, it should be understood that more than one link can be included, such as separate input and output links or separate links for different voltage levels.

FIG. 13is a system diagram illustrating storage system1300. Storage system1300includes three different module types inFIG. 13, namely storage sleds1310, interconnect modules1320, and processing modules1330. Although this example shows many storage sleds, 2 interconnect modules, and 6 processing modules. Any number of sleds or modules can be includes, such as 48 storage sleds or 64 storage sleds, along with a different number of interconnect or processing modules. Some examples can distribute functionality of each interconnect module1320among two or more modules. Additionally, power supply modules and associated power and control distribution links can also be included, but are omitted inFIG. 13for clarity.

A module typically comprises physical support structure and enclosure that includes circuitry, printed circuit boards, semiconductor systems, and structural elements. The modules are insertable and removable from a rackmount style of enclosure. In some examples, the elements ofFIG. 13are included in a 3U chassis for mounting in a larger rackmount environment. It should be understood that the elements ofFIG. 13can be included in any physical mounting environment, and need not include any associated enclosures or rackmount elements.

Holdup circuitry is included on each sled1310to provide power to the associated sled when input power has been lost or removed for the sled. In some examples, the sled is removed from an associated mating connector and input power is lost due to the removal. In other examples, power is lost to system1300, such as during a facility power outage or when an associated power supply fails. Similar holdup circuitry can be included on the other various modules of system1300. Specifically, holdup circuitry1325is included on interconnect modules1320and holdup circuitry1335is included on processing modules1330.

Turning to the example of storage sled1310, the various holdup circuitry is also accompanied by a power controller circuit to selectively provide power to the elements of storage sled1310. The power controller can receive control instructions from a processor of storage sled1310or from other processors or modules, such as over the Inter-Integrated Circuit (I2C), Ethernet, or Universal Serial Bus (USB) sideband interfaces discussed herein. Storage sled1310can receive power over one or more power links as a power source for the various elements of storage sled1310. Holdup circuitry includes energy storage devices for storing power received over the power link for use during power interruption events, such as loss of source power. Holdup circuitry can include capacitance storage devices, such as an array of capacitors. Further discussion of examples of power control circuitry is found below.

As seen inFIG. 13, storage sleds1310can each provide self-power during power interruption events. Also, storage sleds1310can each redistribute power to other storage sleds. This redistributed power can be transferred to other storage sleds1310or to other modules inFIG. 13, such as interconnect module1320or processing module1330. Typically, a storage sled will use any associated holdup power to commit in-flight write data associated with pending write operations before power down of the associated sled. The in-flight write data can be committed to storage drives of the associated storage sled, or can be committed to other non-volatile memory such as a non-volatile write cache which can hold write data until power is restored. In-flight write operations can also be held in non-volatile memory of interconnect module1320or processing module1330if the write operations have not yet reached an associated storage sled. Once any in-flight write data has been committed to non-volatile memory, then excess or remaining holdup power can be redistributed to other modules. In some examples, no pending write operations are present when input power is lost, and a larger amount of excess power is available on a particular storage sled. This excess power can be redistributed to a different storage sled to aid that sled in commit processes for associated write operations. Advantageously, excess holdup power of one sled or module can be used to power operations of another sled or module during power interruptions.

A plurality of storage sleds1310are included in system1300. Each storage sled1310includes one or more storage drives, such as four each. Each storage sled1310also includes Peripheral Component Interconnect Express (PCIe) switches, processors, and control system elements. PCIe switches of each storage sled1310communicate with one or more on-sled storage drives over associated PCIe links. PCIe switches of each storage sled1310also are communicatively coupled to an on-sled processor or control system for traffic statistics retrieval and status monitoring, among other operations. PCIe switches of each storage sled1310communicate over one or more PCIe links1340with an associated PCIe switch1321of an interconnect module1320.

Each PCIe switch1321of interconnect modules1320communicate over associated PCIe links1342with associated PCIe switch1332of one or more processing modules1330. PCIe switch1332communicates with one or more associated processing systems1331as well as over one or more cross-connect PCIe links1343. Interconnect modules1320also each include a plurality of PCIe switches1322for interconnecting processor modules, such as processor modules1330. PCIe switches1322are included for processor module cross-connect, and communicate with ones of PCIe switches1333in associated processing modules1330over processor module cross-connect links1341. PCIe switches1333communicate with ones of processing systems1331over PCIe links1334.

In the example inFIG. 13, PCIe switches1321and1332(and associated PCIe links) are included in a data plane of system1300, and used for carrying storage data between storage sleds1310and processing modules1330. PCIe switches1322and1333(and associated PCIe links) are included in a control plane of system1300, and used for carrying user control data and control signaling between processing modules.

Each processing module1330communicates over one or more PCIe links1335through PCIe switches1333with external expansion cards or external PCIe ports. In some examples, the external expansion cards include network interface cards for communicating over TCP/IP networks or carrying iSCSI traffic, among other network traffic types. These packet links are illustrated by packet network links1344. External access to storage system1300is provided over ones of packet network links1344, such as for end user access to data stored on storage sleds1310.

Each processing module1330can also communicate with other processing modules, such as those in other storage assemblies or 3U enclosures, over one or more inter-module packet network interfaces1345. In some examples, inter-module packet network interfaces1345include network interface cards for communicating over Ethernet or TCP/IP (Transmission Control Protocol (TCP)/Internet Protocol) networks for exchanging storage packets between processing modules. Further operation of inter-module storage packet exchange over Ethernet is discussed in the examples herein.

The PCIe switches discussed herein can comprise PCIe crosspoint switches, which logically interconnect various ones of the associated PCIe links based at least on the traffic carried by each PCIe link. Each PCIe switch port can comprise a non-transparent (NT) or transparent port. An NT port can allow some logical isolation between endpoints, while a transparent port does not allow logical isolation, and has the effect of connecting endpoints in a purely switched configuration. Access over an NT port or ports can include additional handshaking between the PCIe switch and the initiating endpoint to select a particular NT port or to allow visibility through the NT port. In other examples, a domain-based PCIe signaling distribution can be included which allows segregation of PCIe ports of a PCIe switch according to user-defined groups.

PCIe can support multiple bus widths, such as x1, x4, x8, x16, and x32, with each multiple of bus width comprising an additional “lane” for data transfer. PCIe also supports transfer of sideband signaling, such as System Management Bus (SMBus) interfaces and Joint Test Action Group (JTAG) interfaces, as well as associated clocks, power, and bootstrapping, among other signaling. Although PCIe is used inFIG. 13, it should be understood that different communication links or busses can instead be employed, such as Ethernet, Serial Attached SCSI (SAS), FibreChannel, Thunderbolt, Serial Attached ATA Express (SATA Express), among other interconnect, network, and link interfaces. Any of the links inFIG. 13can each use various communication media, such as air, space, metal, optical fiber, or some other signal propagation path, including combinations thereof. Any of the links inFIG. 13can include any number of PCIe links or lane configurations. Any of the links inFIG. 13can each be a direct link or might include various equipment, intermediate components, systems, and networks. Any of the links inFIG. 13can each be a common link, shared link, aggregated link, or may be comprised of discrete, separate links.

InFIG. 13, any processing system1331on any processing module1330has logical visibility to all storage drives in all storage sleds1310. Any processing system1331can transfer data for storage on any storage drive and retrieve data already stored on any storage drive. Thus, ‘m’ number of storage drives can be coupled with ‘n’ number of processors to allow for a large, scalable architecture with a high-level of redundancy and density.

To provide visibility of each processing system1331to any storage sled1310, various techniques can be employed. In a first example, a particular processing system1331manages (instantiates/binds) a subset number of the total quantity of storage sleds, such as 16 storage drives spanning 4 storage sleds, and handles transactions for that subset of storage drives, such as read and write transactions. Each processing system1331, however, has memory-mapped visibility to the storage drives managed by any other processing system1331. When a transaction is desired for a storage drive not managed by a particular processing system, the particular processing system uses the memory mapped access to all storage drives for that transaction. The transaction can be transferred and transitioned to the appropriate processing system that manages that storage drive associated with the data of the transaction. PCIe switches1322and1333are used to transfer data between processing systems so that a particular processing system or processor can store the data in the storage sled or sleds that is managed by that particular processing system, even though the data might be received over a network interface associated with a different processing system.

In operation, such as a write operation, data can be received over any network interface1344by any processing system1331of any processing module1330. For example, the write operation can be a write operation received over network link1344from an end user employing an iSCSI protocol. The processing system that receives the write operation determines if it physically manages the storage drive or drives associated with the write operation, and if it does, then the processing system transfers the data for storage on the associated storage drives over data plane PCIe links1333. If the processing system determines that it does not physically manage the storage drive or drives associated with the write operation, then the processing system transfers the write operation to another processing sled that includes the processing system that does manages the storage drive or drives over cross connect links1334. Data striping can be employed by any processing system to stripe data for a particular write transaction over any number of storage drives, such as over all of the storage sleds that include storage drives managed by the particular processing system.

In this example, the PCIe interfaces associated with each processing system1331have 64-bit address spaces, which allows an addressable space of 264bytes, leading to at least 16 exbibytes of byte-addressable memory. The 64-bit PCIe address space can shared by all processing systems1331for memory mapping to storage drives on storage sleds. Thus, while each particular processing system1331actually manages a subset of the total storage drives on storage sleds, all processors1331have visibility to, and can initiate read/write transactions to, any of storage drives on storage sleds. A managing processing system1331that manages a particular storage drives on storage sleds receives write/read transactions and any associated data from an initiating processing system1331by at least using the memory mapped PCIe address space.