Peripheral storage card with offset slot alignment

A storage card insertable into a host system is provided that includes a plurality of storage devices connectors. The storage card include slot offset features to offset a circuit board of the storage card from a host system slot alignment. This offset provides for storage device connector placement on both sides of the storage card. The storage card also can include a Peripheral Component Interconnect Express (PCIe) switch circuit configured to communicatively couple the PCIe signaling of the plurality of storage device connectors and PCIe signaling of a host connector of the storage card, where the PCIe switch circuit is configured to receive storage operations over the PCIe signaling of the host connector of the storage card and transfer the storage operations for delivery over the PCIe signaling of selected ones of the plurality of storage device connectors.

BACKGROUND

Computer systems typically include bulk storage systems, such as magnetic disk drives, optical storage devices, tape drives, or solid-state storage drives, among other storage systems. As storage needs have increased in these computer systems, networked storage systems have been introduced which store large amounts of data in a storage environment physically separate from end user computer devices. These networked storage systems typically provide access to bulk data storage over one or more network interfaces to end users or other external systems. In addition to storage of data, remote computing systems include various processing systems that can provide remote computing resources to end users. These networked storage systems and remote computing systems can be included in high-density installations, such as rack-mounted environments.

However, as the densities of networked storage systems and remote computing systems increase, various physical limitations can be reached. These limitations include density limitations based on the underlying storage technology, such as in the example of large arrays of rotating magnetic media storage systems. These limitations can also include computing density limitations based on the various physical space requirements for network interconnect as well as the large space requirements for environmental climate control systems.

In addition to physical space limitations, these bulk storage systems have been traditionally limited in the number of devices that can be included per host, which can be problematic in storage environments where higher capacity, redundancy, and reliability is desired. These shortcomings can be especially pronounced with the increasing data storage and retrieval needs in networked, cloud, and enterprise environments.

OVERVIEW

Various computer peripheral cards, devices, systems, methods, and software are provided herein. In one example, a storage card insertable into a host system includes a plurality of storage devices connectors. The storage card include slot offset features to offset a circuit board of the storage card from a host system slot alignment. This offset provides for storage device connector placement on both sides of the storage card. The storage card also includes a Peripheral Component Interconnect Express (PCIe) switch circuit configured to communicatively couple the PCIe signaling of the plurality of storage device connectors and PCIe signaling of a host connector of the storage card, where the PCIe switch circuit is configured to receive storage operations over the PCIe signaling of the host connector of the storage card and transfer the storage operations for delivery over the PCIe signaling of selected ones of the plurality of storage device connectors.

DETAILED DESCRIPTION

In the examples herein, various storage cards are shown that have one or more storage drives to provide data storage capabilities for a host system. A storage card can be insertable into a slot or mating connector of a host system. A storage card might comprise a peripheral add-in card. Typically, host systems can have one or more slots available for plugging in of various add-in cards which couple to the host system via one or more edge connectors. The add-in cards discussed herein can be insertable into a mating socket of the associated host system, such as a motherboard or daughterboard of a computer or server system. A slot cover might be included with the add-in card, such as seen herein. Various electronics, processing elements, storage elements, or other elements might be included in an add-in card to supplement the capabilities of the host system. In some cases, add-in cards might be added into a chassis or rack-mounted system external from the host system or systems.

In many add-in cards, the printed circuit board (PCB) includes an integral edge connector formed from a portion of the PCB and metallic traces. The positioning of the slot connector with regard to the slot opening can lead to a larger stackup height possible for components mounted on a first side of the PCB versus a second side of the PCB. Although this configuration is sufficient for many add-in cards, such as fixed or soldered-on components it can be difficult to include large amounts of modular storage drives that mate to the PCB via associated connectors. The examples herein provide for enhanced PCB configurations which provide an offset from the slot connector within the slot dimensions to allow for modular storage devices on both sides of the PCB. Advantageously, an increase in storage density and component quantity can be achieved.

FIGS. 1A and 1Bare presented to illustrate example physical configurations of storage cards.FIG. 1Aillustrates side ‘A’ of an example physical configuration of storage card100.FIG. 1Billustrates side ‘B’ of an example physical configuration of storage card100. Storage card100is formed from at least two printed circuit boards in this example. Primary circuit board (also referred to herein as a printed circuit board)151includes many of the components of storage card100. Secondary circuitry board (PCB)152includes edge connector153and one or more further connectors for coupling to primary circuit board151. Slot cover163is optionally included to cover a slot opening in a host system.

Storage card100includes one or more storage drive connectors111which can couple to one or more storage drives113, PCIe switch112, auxiliary interface115, and holdup capacitors124. Although now shown inFIGS. 1A and 1Bfor clarity, a power control module is included that distributes power to each element of storage card100over associated power links. Power control module, such as power control521ofFIG. 5, can selectively enable/disable power for each power link and provide holdup power to on-board components using at least holdup capacitors124. Further communication links can be included for intra-card communication between the various elements of storage card100.

Storage card100includes one or more storage drives113, such as two on each PCB side shown inFIGS. 1A and 1B. Storage drives113are arranged on both sides of storage card100. As will be discussed in further examples below, solid state drive (SSD) connectors111can comprise M.2 connectors, and each storage drive113can comprise M.2 compatible SSDs. Other SSD types and connectors might instead be employed, such as Enterprise & Datacenter Storage Form Factor (EDSFF) storage devices that use the SFF-TA-1002 x4 or x8 connectors, among others. The connectors couple to the same primary printed circuit board (PCB)151in this example, albeit with some connectors disposed on a front side (side A) and some connectors disposed on a back side (side B).

Storage card100might comprise a HHHL (half-height half-length) PCIe peripheral card. Other card sizes can be employed, such as a FHHL (full-height half-length) or FHFL (full-height full-length), or HHFL (half-height full length), among others. Storage card100comprises a compact two-sided arrangement which can fit into a single-width PCIe slot, and onto a HHHL sized PCIe card. In this example, the M.2 SSDs comprise either 110 millimeter (mm) or 80 mm length M.2 SSDs. Other sizes of M.2 SSDs can be included, such as lengths of 16, 26, 30, 38, 42, 60, 80 and 110 mm and widths of 12, 16, 22 and 30 mm M.2 end supports can be included to structurally supports an end of each M.2 SSD which is opposite of the connector end.

FIGS. 2A and 2Bare presented to illustrate another example physical configuration of storage cards. Specifically,FIGS. 2A and 2Bboth illustrate side ‘B’ of an example physical configuration of storage card100. In this configuration, a rigid secondary PCB is employed as secondary circuitry board152. Secondary circuitry board152communicatively couples to primary circuit board151via one or more connectors, which can be referred to herein as mezzanine connectors. Likewise, secondary circuitry board152can be referred to herein as a mezzanine board. Connector155is included on secondary circuitry board152, while connector154is included on primary circuit board151. Connectors154-155carry signaling and power to and from primary circuit board151from secondary circuit board152. Secondary circuit board152includes edge connector153which couples into a mating connector in a host system or other peripheral card slot.

Secondary circuitry board152, when coupled to primary circuit board151, provides for an offset for primary circuit board151away from edge connector153. The offset distance can vary, and might be established by a stackup height of connectors154-155. This offset provides for additional clearance on the ‘back’ side B of storage ecard100to mount one or more storage drives113via connectors111. Although slot cover163is optional, additional offset spacers160can be included to couple slot cover163to primary circuit board151. These offset spacers160can ensure that slot cover163fits into mating slot cover mounting features of a host system. Offset spacers160have a thickness corresponding to an offset provided by secondary circuitry board152and connectors154-155.

FIG. 3illustrates a top view of storage card100. This top view highlights the offset between primary circuit card151and secondary circuit card152. InFIG. 3, a slot width301is provided which encompasses a total width of a host slot into which storage card100can fit. Storage card100typically occupies less than the width indicated by slot width301, although multi-slot width cards can be provided in other examples.

However, instead of having primary circuit card151aligned with PCIe socket140, secondary circuit card152is aligned with PCIe socket140. Slot offset302is provided between primary circuit card151and secondary circuit card152which offsets primary circuit card151from socket140. Edge connector153on secondary circuit card152is aligned with socket140for insertion of storage card100into socket140and the corresponding host slot. As can be seen inFIG. 3, storage drives113can be mounted onto either side of primary circuit board151due in part to the offset created by secondary circuit board152from socket140. When a slot cover is employed, offset spacers160can be employed to establish offset303between the slot cover and primary circuit board151.

FIG. 4is presented to illustrate an alternative arrangement of the secondary circuit board.FIG. 4illustrates storage card400which includes storage drives413on both sides of primary circuit board451. Optional slot cover463can be included, as well as offset spacers460for slot cover463. PCIe switch circuitry412can also be included to electrically couple interfaces of storage drives413to host signaling associated with edge connector453.

In this configuration, secondary circuit board452includes edge connector453as well as flexible connector457. Flexible connector457as well as one or more standoffs458provide for an offset between edge connector453and primary circuit board451. Additional connectors may be included on one or more among primary circuit board451and secondary circuit board452to couple with flex connector457. However, direct soldered connections might instead be employed to couple with flex connector457. A detailed view401is included inFIG. 4to further illustrate flexible circuit connection457between primary circuit board451and secondary circuit board452. Flexible circuit connection457carries PCIe signaling, sideband signaling, power connections, among other electrical signaling between primary circuit board451and secondary circuit board452. Flexible circuit connection457might comprise one or more sections of flexible circuitry, among various rigid portions. Rigid-flex circuit elements can be employed with both flexible and rigid portions.

FIG. 5Aillustrates an example schematic representation of a first side ‘A’ of storage card510.FIG. 5Billustrates an example schematic representation of a second side ‘B’ of storage card510.FIG. 6illustrates an example schematic configuration of storage card510. Storage card510includes four storage drives511, PCIe switch512, processor520, power control module521, and holdup circuit522. Power control module521distributes power to each element of storage card510over associated power links530-532. Power control module521can selectively enable/disable power for each power link. Further communication links can be included for intra-card communication between the various elements of storage card510.

Components511,512,515,520,521,522, and570are mounted on circuit board551in this example. Additionally, circuit board551includes one or more connectors for coupling to circuit board553. For example, mezzanine connector552can be employed to physically and communicatively couple circuit board551to circuit board553. As discussed in the preceding Figures, an offset is provided between circuit board551to circuit board553. This offset provides for mounting of storage drives511on both sides of storage card510while still allowing for storage card510to fit into a single slot of a host system. Optional slot cover555can be included along with associated offset elements to cover a slot opening associated with the slot into which storage card510is inserted.

Storage card510includes one or more storage drives511, such as four shown inFIGS. 5A, 5B, and 6. Two storage drives511are arranged on a front side ‘A’ of storage card510, while two storage drives511are arranged on a back side ‘B’ of storage card510. A first set of connectors is employed for the side A rank of storage drives, while a second set of connectors is employed for the side B rank of storage drives. InFIG. 5A, a +z axis comprises an axis oriented out of the drawing, and an offset is provided by mezzanine connector552between circuit board551and circuit board553along the z-axis. One or more storage drives can be included in each rank, with associated connectors570that couple to the storage drives. As will be discussed in further examples below, connectors570can comprise M.2 connectors, and each storage drive511can comprise M.2 solid state drives (SSD). The connectors couple to the same printed circuit board (PCB)551in this example.

Storage card510also includes one or more Peripheral Component Interconnect Express (PCIe) switches, processors, and control system elements. PCIe switch512communicates with one or more on-card storage drives over associated PCIe links. PCIe switch512is also communicatively coupled to an on-card processor or control system for traffic statistics retrieval, power monitoring, status monitoring, among other operations.

PCIe switch512communicates with a host system or host module (not pictured) over PCIe link540. PCIe link540can comprise a PCIe link with multiple lanes, such as a “x4” PCIe link, although a different number of PCIe lanes can be employed. Additionally, more than one PCIe link540can be employed for load balancing, redundancy, and failover protection for storage card510. PCIe switch512also communicates with four storage drives511over associated x4 PCIe links541, although a different number of storage drives can be employed. PCIe can support multiple bus widths, such as x1, x4, x8, x16, and x32, with each multiple of bus width comprising an additional “lane” for data transfer. The PCIe links discussed herein can be any generation of PCIe links, such as generations 3, 4, 5, or Gen-Z generations, among others. Storage card510also supports transfer of sideband signaling549, such as System Management Bus (SMBus) interfaces and Joint Test Action Group (JTAG) interfaces, as well as associated clocks, power, and bootstrapping, among other signaling.

PCIe interface540can carry NVMe (NVM Express) traffic issued by a host processor or host system. NVMe (NVM Express) is an interface standard for mass storage devices, such as hard disk drives and solid-state memory devices. NVMe can supplant serial ATA (SATA) interfaces for interfacing with mass storage devices in personal computers and server environments. However, these NVMe interfaces are limited to one-to-one host-drive relationship, similar to SATA devices. In the examples discussed herein, a PCIe interface is employed to transport NVMe traffic and present a multi-drive system as one or more NVMe virtual logical unit numbers (VLUNs) over a PCIe interface.

In NVMe operations, such as an NVMe write operation, data can be received over any of PCIe links540or515for any storage drive511. For example, a write operation can be an NVMe write operation received over PCIe link540from a device employing an NVMe protocol transported over a PCIe interface. In another example, the write operation can be an NVMe write operation received over PCIe link540or515from an external device employing an NVMe protocol transported over a PCIe interface. An associated storage drive can receive the NVMe traffic over an associated PCIe interface541and respond accordingly, such as with a write confirmation or with read data in the case of an NVMe read operation.

In further examples, processor520can handle PCIe traffic for the storage drives and manage the storage drives in a logical manner. For example, data striping can be employed by processor520to stripe data for a particular write transaction over any number of storage drives511, such as over all of the storage drives or a subset of the storage drives. Likewise, data redundancy can be employed to mirror data over any of storage drives511. In further examples, ones of storage drives511are presented as one or more logical drives or logical volumes to a host system, such as one or more NVMe virtual logical units (VLUNs). Processor520can manage striping, mirroring, or logical volume establishment and presentation. In a first example, processor520receives all PCIe traffic for storage drives511over PCIe interface533and distributes to appropriate storage drives511to achieve striping, mirroring, or logical volumes. In other examples, processor520monitors traffic in PCIe switch512and instructs PCIe switch512to direct PCIe traffic to appropriate storage drives to achieve striping, mirroring, or logical volumes.

As mentioned above, processor520can present the storage resources of storage card510as a VLUN, such as NVMe VLUNs. Processor520can present any number of VLUNs to an external system over a PCIe interface, such as any of PCIe links540or515. These VLUNs can be presented as an NVMe target. An NVMe target can present the storage resources of storage card510as a single storage target, such as emulating a single storage drive, over a PCIe interface. In this manner, a plurality of storage drives that comprise any number of storage drives511can be presented as a single NVMe target to an external system over a PCIe interface. Processor520can receive NVMe storage traffic, such as NVMe frames, and distribute these storage transactions for handling by an assigned storage drive511. In other examples, processor520monitors NVMe storage traffic in PCIe switch512and instructs PCIe switch512to direct PCIe traffic to appropriate storage drives to achieve VLUNs or NVMe targets.

Auxiliary PCIe interface515can optionally be included in storage card510. Auxiliary PCIe interface515can be employed to connect two or more PCIe storage cards to each other for transfer of user data, storage operations, status, control signaling, operational information, or other data between storage cards, such as two of storage card510. Auxiliary PCIe interface515can comprise a different PCIe bus width or lane allocation than host PCIe interface540. Auxiliary PCIe interface515can couple a PCIe interface provided by PCIe switch512to a PCIe interface of another storage card which can be included in the same host system as storage card510or included in another host system. Additionally, a connector can be employed with interface515to connect among the various storage cards using associated cabling. In some examples, mini-SAS connectors and cabling are employed and are configured to carry PCIe signaling of auxiliary PCIe interface515. Auxiliary PCIe interface515can also include non-PCIe signaling, such as sideband interfaces549or other interfaces.

Auxiliary PCIe interface515can be used for cluster interconnect and can terminate at external connectors, such as mini-Serial Attached SCSI (SAS) connectors which are employed to carry PCIe signaling over mini-SAS cabling. In further examples, MiniSAS HD cables are employed that drive 12 Gb/s versus 6 Gb/s of standard SAS cables. 12 Gb/s can support PCIe Gen 3. Connector associated with interface515can comprise mini-SAS connectors that comprise mini-SAS jacks. Associated cabling can comprise SAS cabling which can include associated shielding, wiring, sheathing, and termination connectors.

PCIe switch512comprises one or more PCIe crosspoint switches, which logically interconnect various ones of the associated PCIe links based at least on the traffic carried by each PCIe link. PCIe switch512establishes switched connections between any PCIe interfaces handled by PCIe switch512. Each PCIe switch port might comprise a non-transparent (NT) or transparent port, as well as ports isolated by domain-based logical port segregation. An NT port can allow some logical isolation between endpoints, while a transparent port does not allow logical isolation, and has the effect of connecting endpoints in a purely switched configuration. Access over an NT port or ports can include additional handshaking between the PCIe switch and the initiating endpoint to select a particular NT port or to allow visibility through the NT port. In other examples, a domain-based PCIe signaling distribution can be included which allows segregation of PCIe ports of a PCIe switch according to user-defined groups. In some examples, PCIe switch512comprises one or more among PLX/Broadcom/Avago PEX8796 24-port, 96 lane PCIe switch chips, PEX8725 10-port, 24 lane PCIe switch chips, PEX97xx chips, PEX9797 chips, or other PEX87xx/PEX97xx chips.

Although PCIe link540is used inFIGS. 5A and 5B, it should be understood that additional or different communication links or busses can be employed, such as Ethernet, Serial Attached SCSI (SAS), FibreChannel, Thunderbolt, Serial Attached ATA Express (SATA Express), among other interconnect, network, and link interfaces. Any of the links inFIGS. 5A and 5Bcan each use various communication media, such as air, space, metal, optical fiber, or some other signal propagation path, including combinations thereof. Any of the PCIe links inFIGS. 5A and 5Bcan include any number of PCIe links or lane configurations. Any of the links inFIGS. 5A and 5Bcan each be a direct link or might include various equipment, intermediate components, systems, and networks. Any of the links inFIGS. 5A and 5Bcan each be a common link, shared link, aggregated link, or may be comprised of discrete, separate links.

Processor520can optionally communicate over at least sideband links549. Sideband links549can include Universal Serial Bus (USB), SMBus, JTAG, Inter-Integrated Circuit (I2C), controller area network bus (CAN), or any other communication interface, and in some examples is provided over portions of PCIe link540. In this example, processor520includes I2C interface525and USB interface526for communication over sideband links549. I2C interface525and USB interface526can be included in separate circuitry or included in similar elements as processor520. Processor520and PCIe switch512can communicate over an associated communication link533, which can be an I2C or a PCIe link, among other link types.

Each storage drive511comprises a solid-state drive (SSD) in this example, and communicates with external systems over an associated PCIe interface included in each storage drive511. Each storage drive511comprises an individual M.2 SSD card, which communicates over an associated PCIe interface541, which can comprise PCIe interfaces such as described for PCIe interface540, although variations are possible. The solid-state storage media of storage drives511can comprise flash memory, static RAM, NAND flash memory, NOR flash memory, memristors, magnetic random-access memory (MRAM), or other solid-state media. Instead of or in addition to solid state media, each storage drive511can comprise magnetic storage, such as hard disk drives, tape drives, magnetoresistive memory devices, and the like, or can comprise optical storage, such as phase change memory.

Each storage drive511can receive read transactions and write transactions issued by a host system, such as a host processor. Responsive to a read transaction, each storage drive511can retrieve data identified by the read transaction and transfer the data for delivery to the associated host. Responsive to a write transaction, each storage drive511can write data that accompanies the write transaction to storage media associated with storage drive511. Data striping can be employed by storage card510to stripe data for a particular write transaction over any number of storage drives511.

Each storage drive511comprises an M.2 circuit card which is separate from circuit board551and circuit board553and includes a mini-PCI Express connector or other connector that interfaces with a connector on circuit board551. Circuit board551and circuit board553each comprise one or more printed circuit boards that couple to the various elements of storage card510. In other examples, each storage drive511comprises one or more flash memory chips with a PCIe interface which is soldered onto circuit board551. In yet other examples, each storage drive511comprises one or more separate solid-state disk drives or magnetic hard disk drives along with associated enclosures and circuitry. In the examples shown inFIGS. 5A and 5B, storage card510is shown as an HHHL card (half-height half-length PCIe card). Although card510can instead be a FHHL card (full-height half-length PCIe card), FHFL card (full-height full-length PCIe card), or HHFL (half-height full length) in other examples.

Processor520comprises one or more microprocessors, processing devices, multi-core processors, processing circuitry, or other processing system. Processor520can include one or more non-transitory memory devices, such as RAM, solid state storage, or other memory to store instructions that are executable by processor520to operate as discussed herein. In some examples, processor520comprises an ARM microcontroller, ARM microprocessor, field-programmable gate array (FPGA), application specific integrated circuit (ASIC), application specific processor, or other microprocessor or processing elements. Processor520can comprise any processing elements discussed below for processor721ofFIG. 7or control processor800ofFIG. 8. Processor520can monitor usage statistics, traffic status, or other usage information through link533. PCIe switch512can track this usage information during normal operation and data transfer with storage drives511, and processor520can retrieve this usage information as needed over link533.

Holdup circuitry522is included on storage card510to provide power to the storage card when input power has been lost or removed for the storage card. In some examples, the storage card is removed from an associated mating connector and input power is lost due to the removal. In other examples, power is lost to a host system into which storage card510is connected, such as during a facility power outage or when an associated power supply fails.

The various holdup circuitry is also accompanied by a power controller circuit521to selectively provide power to the elements of storage card510. The power controller can receive control instructions from a processor of storage card510or from other processors or modules, such as over the Inter-Integrated Circuit (I2C), Ethernet, or Universal Serial Bus (USB) sideband interfaces, or over a PCIe interface. Storage card510can receive power over one or more power links as a power source for the various elements of storage card510, and these power links can be included in a PCIe connector of storage card510. Holdup circuitry522includes energy storage devices for storing power received over the power link for use during power interruption events, such as loss of source power. Holdup circuitry522can include capacitance storage devices, such as an array of capacitors. Further discussion of examples of power control circuitry is found below.

Although processor520and power controller521are shown as separate elements inFIGS. 5A and 5B, it should be understood that processor520and power controller521can be included in the same processing circuitry. In some examples, processor520and power controller521comprise an ARM-compatible microprocessor or microcontroller, although other circuitry can be employed.

Storage card510can provide self-power during power interruption events. Typically, storage card510will use any associated holdup power to commit in-flight write data associated with pending write operations before power down of circuitry of storage card510. The in-flight write data can be committed to associated storage drives511, or can be committed to other non-volatile memory such as a non-volatile write cache which can hold write data until power is restored. Once any in-flight write data has been committed to non-volatile memory, then excess or remaining holdup power can be held for future use, bled off into dummy loads, or redistributed to other cards over PCIe power links or other power links.

In some examples, no pending write operations are present when input power is lost, and a larger amount of excess power is available on storage card510. This excess power can be redistributed to a different storage card to aid that storage card in commit processes for associated write operations. Advantageously, excess holdup power of one storage card can be used to power operations of another storage card during power interruptions. This redistributed power can be transferred to other storage cards or other PCIe cards over power links523when included in PCIe links540or515.

Power control module521includes circuitry to selectively provide power to any of the elements of storage card510. Power control module521can receive control instructions from processor520or over link530. In some examples, power control module521comprises processing elements discussed above for processor520, or is included in the elements of processor520. Power control module521can receive power over power link523as a power source for the various elements of storage card510. Holdup circuit522includes energy storage devices for storing power received over power link523for use during power interruption events, such as loss of source power. Holdup circuit522can include capacitance storage devices, such as an array of capacitors. Further discussion of examples of power control circuitry is found below.

In some examples, bidirectional power flow is possible over link523. Power can be accepted by storage card510when input power is available, such as from a mating connector. Power can be redistributed to other storage cards by module510over link523when input power is not available, such as during power interruption events. When storage card510is removed from a mating connector, then power can be bled off into associated power sink circuitry. Although one power link523is shown, it should be understood that more than one link can be included, such as separate input and output links or separate links for different voltage levels.

FIG. 7is a block diagram illustrating power control system700. Power control system700can be included on any of the storage cards discussed herein, such as the power controller or holdup circuitry portions of storage card100ofFIGS. 1A and 1Bor storage card510ofFIGS. 5A, 5B, and 6, among others. Power control system700illustrates power controller720, which can be an example of any of the power control modules or processor discussed herein, such as power control module521or processor520ofFIG. 6. Power controller720includes processor721, communication interface722, and power circuitry723. Each of the elements of power controller720are communicatively coupled.

Communication interface722communicates over communication links724, which can include any of the communication link protocols and types discussed herein. Communication interface722can include transceivers, network interface equipment, bus interface equipment, and the like. In operation, communication interface722receives control instructions from another processing unit over communication links724. Communication links724also communicate with elements of the card that power controller720is employed on. For example, on a storage card, communication links724receive write data commit status of storage drives, power control instructions from other processors or processing systems, and can communicate over a PCIe interface or sideband communications of a PCIe interface.

Processor721includes any processor or processing system discussed herein, and controls the operations of power controller720, such as initiating power up of storage card elements, initiating power down of storage card elements, monitoring usage statistics for a storage card or for other storage cards.

To further describe the circuitry and operation of processor721, a detailed view is provided, although variations are possible. Processor721includes communication interface740and processing system750. Processing system750includes processing circuitry751, random access memory (RAM)752, and storage753, although further elements can be included. Example contents of storage753are further detailed by software modules754-756.

Processing circuitry751can be implemented within a single processing device but can also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing circuitry751include general purpose central processing units, microprocessors, application specific processors, and logic devices, as well as any other type of processing device. In some examples, processing circuitry751includes physically distributed processing devices, such as cloud computing systems.

Communication interface740includes one or more communication and network interfaces for communicating over communication networks or discrete links, such as communication interface722, or further serial links, packet networks, the Internet, and the like. The communication interfaces can include one or more local or wide area network communication interfaces which can communicate over Ethernet or Internet protocol (IP) links. Communication interface740can include network interfaces configured to communicate using one or more network addresses, which can be associated with different network links. Examples of communication interface740include network interface card equipment, transceivers, modems, and other communication circuitry. Although communication interface740and communication interface722are both shown inFIG. 7, it should be understood that these can comprise different interfaces or combined into the same communication interface module, and can communicate over links724.

RAM752and storage753together can comprise a non-transitory data storage system, although variations are possible. RAM752and storage753can each comprise any storage media readable by processing circuitry751and capable of storing software. RAM752can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Storage753can include non-volatile storage media, such as solid state storage media, flash memory, phase change memory, or magnetic memory, including combinations thereof. RAM752and storage753can each be implemented as a single storage device but can also be implemented across multiple storage devices or sub-systems. RAM752and storage753can each comprise additional elements, such as controllers, capable of communicating with processing circuitry751.

Software stored on or in RAM752or storage753can comprise computer program instructions, firmware, or some other form of machine-readable processing instructions having processes that when executed a processing system direct processor721to operate as described herein. For example, software drives processor721to monitor operating statistics and status for a storage card, monitor power status for the cards and modules, and instruct power circuitry723to control flow of holdup power or operational power, among other operations. The software can also include user software applications. The software can be implemented as a single application or as multiple applications. In general, the software can, when loaded into a processing system and executed, transform the processing system from a general-purpose device into a special-purpose device customized as described herein.

Software modules754-756each comprise executable instructions which can be executed by processor721for operating power controller720according to the operations discussed herein. Specifically, statistical monitor754monitors usage status or usage statistics for elements of a storage card. The usage statistics include data transfer rates of links, error rates of links, a cumulate number of errors of links, among other statistics. The usage statistics can be collected and stored by processor721in a data structure, such as a database or table and stored in storage753, RAM752, or other storage elements. Power monitor755monitors power inrush statistics during a power-up process, power status statistics, power active status, voltage levels, phase measurements, current draw, holdup circuit status or levels, card/module insertion status, thermal levels, among other statistics. Power control756instructs power circuitry to power up or power down an associated storage card or module responsive to statistical monitor754or power monitor755, among other signals such as discrete signals monitored by power circuitry723. Power control756can power up or power down a card or module responsive to data commit status of associated storage drives or other circuitry, responsive to insertion status, or other factors.

Software modules754-756can reside in RAM752during execution and operation by processor721, and can reside in storage space753during a powered-off state, among other locations and states. Software modules754-756can be loaded into RAM752during a startup or boot procedure as described for computer operating systems and applications.

Storage753can include one or more storage systems comprising flash memory such as NAND flash or NOR flash memory, MRAM, phase change memory, magnetic memory, among other solid state storage technologies. As shown inFIG. 7, storage753includes software modules754-756stored therein. As described above, storage753can store software modules754-756in one or more non-volatile storage spaces during a powered-down state of processor721, among other operating software, such as operating systems.

Processor721is generally intended to represent a computing system where at least software modules754-756are deployed and executed in order to render or otherwise implement the operations described herein. However, processor721can also represent any computing system on which at least software modules754-756can be staged and from where software modules754-756can be distributed, transported, downloaded, or otherwise provided to yet another computing system for deployment and execution, or yet additional distribution.

Power circuitry723includes various power control, voltage regulation, power holdup, and other circuitry. Power circuitry723receives power from a power source, such as off-card power link735, and distributes power to on-card elements over ones of power links725.

As a specific example of power circuitry723, various elements are shown inFIG. 7. These elements include buck-boost module731, flow control module732, on-card distribution module733, holdup capacitors734, and dummy load735. Buck-boost module731comprises one or more switching power regulators that receive power from a power source, such as off-card power link735, and boosts a voltage associated with the power source to a holdup voltage for holdup capacitors734. In this example, the power source is provided at +12 VDC and the holdup capacitors734are driven at +80 VDC, although different voltages can be employed, such as 125 VDC or higher voltages on holdup capacitors734. Buck-boost module731can also take the energy stored by holdup capacitors734and step-down the voltage to a lower voltage, such as 12 VDC for driving on-card or off-card elements using the energy stored in holdup capacitors734. Processor721can communicate with buck-boost731to instruct buck-boost731to enter a buck mode or a boost mode. Buck-boost731can receive control signals or instructions from processor721, such as over general purpose I/O of processor721.

To control the flow of energy between on-card power and holdup power, flow control module732is employed. Flow control module732includes various power switching elements, such as transistor switches, analog switches, solid state switches, diodes, and the like. When external off-card power is available, such as over link735, then flow control732can provide this power to on-card distribution module733and to buck-boost module731for charging holdup capacitors734. When external off-card power is not available, then flow control732can allow power stored in holdup capacitors734and stepped-down by buck-boost module731to flow to on-card distribution module733instead of off-card power of link735. Also, as discussed below, when excess energy remains in holdup capacitors734after an associated storage card of power controller720has had all elements powered down and data committed, then this excess energy can be directed by flow control module732to off-card consumers over link735. In this manner, excess energy stored in holdup devices of power controller720can be used to provide power to other cards or devices during a shutdown or commit process. The commit process includes writing any in-flight write data to non-volatile memory. The non-volatile memory can include storage drives of a storage card, or can include separate non-volatile memory dedicated to power-down caching of in-flight data. If the associated storage card of power controller720is instead removed from a chassis or connector, then this excess energy of holdup capacitors734can be safely bled off using dummy load735. Flow control module732can receive control signals or instructions from processor721, such as over general purpose I/O of processor721.

On-card distribution module733includes various power flow and switching circuitry to direct electrical power to various elements of a storage card, such as storage drives, PCIe switches, and the like, over links725. Links725can comprise the various power links discussed herein for the various cards. On-card distribution module733includes various power switching elements, such as transistor switches, analog switches, solid state switches, diodes, and the like. On-card distribution module733can receive control signals or instructions from processor721, such as over general purpose I/O of processor721.

Dummy load735can include resistive loads, such as heat dissipating electrical elements to bleed off excess energy of a holdup circuit, such as holdup capacitors734. In some examples, dummy load735comprises a high-output light emitting diode (LED) which can efficiently bleed off excess energy using the light output of the LED. This LED can also indicate that energy still remains in the holdup circuit, warning a user of a particular storage card that potentially dangerous or damaging voltages and energies might still exist on a storage card. When a card is inserted into a connector, the LED is normally off. However, when a storage card is removed from a connector, then the LED would be instructed to illuminate and indicate that energy was being bled off of the storage card using the LED. When the LED finally turned off, due to insufficient energy remaining on a card, then the operator can know that dangerous or damaging voltages and energies no longer exist on the storage card. If the LED cannot bleed all of the energy quickly enough, then additional resistive elements can be employed in parallel to assist the LED indicator. Cover plates for the various higher voltage elements, such as capacitors, of system700can be employed.

FIG. 8is a block diagram illustrating processing system800. Processing system800illustrates an example of any of the power control modules or card processors discussed herein, such as power control module521or processor520ofFIG. 6, or power controller720ofFIG. 7. In addition, processing system800can be illustrative of any processing system a storage card discussed herein.

Control processor800includes communication interface801and processing system810. Processing system810includes processing circuitry811, random access memory (RAM)812, and storage813, although further elements can be included. Example contents of RAM812are further detailed in RAM space820, and example contents of storage813are further detailed in storage system860.

Processing circuitry811can be implemented within a single processing device but can also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing circuitry811include general purpose central processing units, microprocessors, application specific processors, and logic devices, as well as any other type of processing device. In some examples, processing circuitry811includes physically distributed processing devices, such as cloud computing systems.

Communication interface801includes one or more communication and network interfaces for communicating over communication links, networks, such as packet networks, the Internet, and the like. The communication interfaces can include PCIe interfaces, serial links, such as SPI links, I2C links, USB links, UART links, or one or more local or wide area network communication interfaces which can communicate over Ethernet or Internet protocol (IP) links. Communication interface801can include network interfaces configured to communicate using one or more network addresses, which can be associated with different network links. Examples of communication interface801include network interface card equipment, transceivers, modems, and other communication circuitry.

RAM812and storage813together can comprise a non-transitory data storage system, although variations are possible. RAM812and storage813can each comprise any storage media readable by processing circuitry811and capable of storing software. RAM812can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Storage813can include non-volatile storage media, such as solid-state storage media, flash memory, phase change memory, or magnetic memory, including combinations thereof. RAM812and storage813can each be implemented as a single storage device but can also be implemented across multiple storage devices or sub-systems. RAM812and storage813can each comprise additional elements, such as controllers, capable of communicating with processing circuitry811.

Software stored on or in RAM812or storage813can comprise computer program instructions, firmware, or some other form of machine-readable processing instructions having processes that when executed a processing system direct control processor800to operate as described herein. For example, software can drive processor800to monitor operating statistics and status for various storage cards and other modules, monitor power status for the cards and modules, and instruct power circuitry to control flow of holdup power or operational power, control power down or reset of various on-board storage drives, control performance throttling, among other operations. The software can also include user software applications, application programming interfaces (APIs), or user interfaces. The software can be implemented as a single application or as multiple applications. In general, the software can, when loaded into a processing system and executed, transform the processing system from a general-purpose device into a special-purpose device customized as described herein.

RAM space820illustrates a detailed view of an example configuration of RAM812. It should be understood that different configurations are possible. RAM space820includes applications821and operating system (OS)822. Software applications823-825each comprise executable instructions which can be executed by processor800for operating a power controller or other circuitry according to the operations discussed herein. Specifically, statistical monitor823monitors usage status or usage statistics for elements of cards and modules. The usage statistics include data transfer rates of links, error rates of links, a cumulate number of errors of links, among other statistics. The usage statistics can be collected and stored by processor800in a data structure, such as a database or table and stored in storage813, RAM812, or other storage elements. Power monitor824monitors power statistics during a power up, operational, or power-down processes, power status statistics, power active status, voltage levels, phase measurements, current draw, holdup circuit status or levels, card/module insertion status, thermal levels, among other statistics. Power control825instructs power circuitry to power up or power down an associated drive, card, circuitry, or module responsive to statistical monitor823or power monitor824, among other signals such as discrete signals monitored by associated power circuitry. Power control825can power up or power down a card or module responsive to data commit status of associated storage drives or other circuitry, responsive to insertion status, or other factors.

Applications821and OS822can reside in RAM space820during execution and operation of control processor800, and can reside in storage system860during a powered-off state, among other locations and states. Applications821and OS822can be loaded into RAM space820during a startup or boot procedure as described for computer operating systems and applications.

Storage system860illustrates a detailed view of an example configuration of storage813. Storage system860can comprise flash memory such as NAND flash or NOR flash memory, MRAM, phase change memory, magnetic memory, among other solid state storage technologies. As shown inFIG. 8, storage system860includes system software861. As described above, system software861can be in a non-volatile storage space for applications and OS during a powered-down state of control processor800, among other operating software.

Control processor800is generally intended to represent a computing system with which at least software861and821-825are deployed and executed in order to render or otherwise implement the operations described herein. However, control processor800can also represent any computing system on which at least software861and821-825can be staged and from where software861and821-825can be distributed, transported, downloaded, or otherwise provided to yet another computing system for deployment and execution, or yet additional distribution.