Patent Description:
Corporations operating large-scale computing systems invest significant amounts of capital to establish and maintain the equipment comprising the computing systems. These computing systems are typically stored in data centers, which require expensive hardware and equipment, as well as real estate and personnel to maintain the equipment stored in the data centers. To minimize costs, data center racks and the equipment thereon are designed to be compact and capable of functioning over extended periods of time, as well as modular to accommodate changing architecture and configuration of components within the computing system.

Given the high power outputs of each computing system, as well as the other equipment in the data rack and in the data center, high levels of heat are generated. Significant heat within and around the computing systems threaten the lifespan and operation of the computing system.

Liquid cooling is one method in which to remove heat in the system and to maintain components, such as chip assemblies and microprocessors, within the system within operating temperature limits. Liquid cooling allows for removal of the excess heat with heat transfer fluid pumped into a cooling device and the heated return cooling liquid pumped out of the device.

Document <CIT> relates to a cooling system for a data center.

Document <CIT> relates to sectioned manifolds for pumped immersion-cooling of electronic components.

The presently claimed subject-matter is defined in independent claim <NUM>. There is provided a system for thermal management of computing devices comprising a central supply line coupled to a fluid source of cooling liquid; a central return line coupled to the fluid source; a rack supply manifold coupled to the central supply line; a rack return manifold coupled to the central return line; and a loop assembly coupled to the rack supply manifold and the rack return manifold. The cooling loop assembly can be coupled to at least one cooling device that is electrically connected to a circuit board, the cooling loop assembly further comprising multiple inlet connectors connected to the rack supply manifold; and multiple outlet connectors connected to the rack return manifold. The multiple inlet connectors may provide a passageway for the cooling liquid from the central supply line into the cooling loop assembly for distribution to the at least one cooling device. The multiple outlet connectors may provide a passageway to the rack return manifold for the cooling liquid exiting the at least two of the cooling devices.

In one example, each multiple inlet connector may be coupled to a board supply line that feeds fluid from the rack supply manifold into the cooling loop assembly and into the at least one cooling device. The cooling loop assembly can further comprise a board inlet manifold and secondary supply lines. The secondary supply lines may connect the board inlet manifold to the at least one cooling device.

In another example, the cooling loop assembly can additionally or alternatively comprise board return lines coupled to each multiple outlet connector. The cooling liquid may return from the at least one cooling device flows through each of the board return lines and the multiple outlet connectors. Additionally or alternatively, the cooling loop assembly can comprise a board outlet manifold and secondary return lines connecting the multiple outlet connectors to the board outlet manifold.

In another example, the at least one cooling device is two or more cooling devices, and the multiple inlet connectors are at least two inlet connectors. Additionally, in some examples, the multiple outlet connectors may be at least two outlet connectors. In some examples, the rack supply manifold and the rack return manifold may be positioned on opposed sides of the two or more cooling devices. In other examples, the rack supply manifold and the rack return manifold are positioned on a same side of the two or more cooling devices.

According to another example of this aspect, the multiple inlet connectors and the multiple outlet connectors are first multiple inlet connectors and first multiple outlet connectors. The cooling loop assembly may be a first cooling loop assembly and the system may further include a second cooling loop assembly coupled to the rack supply manifold and rack return manifold. The first cooling loop assembly may overlie the second cooling loop assembly. The second cooling loop assembly may be coupled to at least one other cooling device connected to another circuit board.

In another example, the rack supply manifold may be a first rack supply manifold and the rack return manifold may be a first rack return manifold. The system may further comprise a second central supply line; a second central return line; a second rack supply manifold coupled to the second central supply line; and a second rack return manifold coupled to the second central return line. At least one of the multiple inlet connectors may be connected to the second rack supply manifold. At least one of the multiple outlet connectors may be connected to the second rack return manifold. The at least one of the multiple inlet connectors may provide a passageway for the cooling liquid from the second central supply line into the cooling loop assembly for distribution to the at least one cooling device. The at least one of the multiple outlet connectors provide a passageway to the second rack return manifold for the cooling liquid to exit the at least one of the cooling devices.

The multiple inlet connectors and the multiple outlet connectors may comprise quick disconnect couplings.

Furthermore, there is disclosed a system comprising a circuit board; a plurality of microelectronic assemblies electronically connected to the circuit board; a plurality of cooling devices; and a cooling loop assembly. The plurality of cooling devices may overlie a first microelectronic assembly and a second microelectronic assembly of the plurality of microelectronic assemblies and be configured to cool the first and second microelectronic assemblies. The cooling loop assembly may be coupled to the plurality of the cooling devices and overlie the circuit board. The cooling loop assembly can comprise multiple inlet connectors configured to be coupled to an assembly fluid source external to the circuit board; and multiple outlet connectors configured to be coupled to an assembly return line external to the circuit board. The multiple inlet connectors provide a passageway for cooling liquid from the assembly fluid source into the cooling loop assembly for distribution to the at least two of the cooling devices. The multiple outlet connectors may provide a passageway to the assembly return line for the cooling liquid exiting the at least two of the cooling devices.

In one example, each of the multiple inlet connectors is attached to a board supply line overlying the circuit board and that feeds fluid into the cooling loop assembly and into the at least two of the cooling devices. Additionally, an outlet return line may overlie the circuit board and be connected to each of the multiple outlet connectors. The cooling liquid returning from each of the at least two of the cooling devices may flow through each of the outlet return lines and the multiple outlet connectors. Additionally, the inlet connectors and outlet connectors are quick disconnect couplings and the outlet connectors are quick disconnect couplings.

Furthermore, there is provided a system for thermal management of computing devices comprising a rack supply manifold connected to a supply line; a rack return manifold connected to a central return line; and a first cooling loop assembly and a second cooling loop assembly coupled to the rack supply manifold and the rack return manifold. The first cooling loop assembly may be coupled to at least one cooling device that is electrically connected to a circuit board. The first cooling loop assembly comprises at least one inlet connector coupled to the rack supply manifold. The at least one inlet connector provides a passageway for cooling liquid from the rack supply manifold into the first cooling loop assembly for distribution to the at least one cooling device. At least one outlet connector may be coupled to the rack return manifold. The at least one outlet connector may provide a passageway for cooling liquid exiting the at least one cooling device to the rack return manifold. At least one of the at least one inlet connector and the at least one outlet connector is a plurality of inlet connectors or outlet connectors. The second cooling loop assembly may be coupled to at least one other cooling device connected to the circuit board. The second cooling loop assembly may comprise at least one inlet connector of the second cooling loop assembly coupled to the rack supply manifold; and at least one outlet connector of the second cooling loop assembly coupled to the rack return manifold. The at least one inlet connector may provide a passageway for the cooling liquid from the rack supply manifold into the second cooling loop assembly for distribution to the at least one other cooling device. The at least one outlet connector may provide a passageway to the rack return manifold for the cooling liquid exiting the at least one other cooling device. At least one of the at least one inlet connector and the at least one outlet connector of the second cooling loop assembly is a plurality of inlet connectors or outlet connectors.

In one example, the at least one connector of the first cooling loop assembly and the at least one connector of the second cooling loop assembly comprise a plurality of inlet connectors.

Similar reference numbers and designations in the various drawings indicate like elements. In the drawings:.

Liquid cooling systems and cold plates are used to dissipate heat within chip assemblies, computing systems, and data server rack systems. Liquid cooled processors require higher coolant flow rate to maintain the computing performance and system reliability. As chip power increases, liquid flow rate must also increase to keep up with increased heat created by the chips. This can lead to significant rise in pressure drop through connections for cooling loop assemblies. Further, pumping power consumed by pressure drop in connectors does not contribute to cooling.

To address the shortcomings associated with supplying a higher flow rate at increased power and cost, an improved liquid cooling system is disclosed. In accordance with aspects of the disclosure, the improved cooling system can include a single cooling loop with multiple quick disconnects provided at the supply interface and multiple disconnects provided at the return interface. This allows for the lowest pressure drop through the connections and the cooling loop, versatility, and a design that is easily scalable for increased number of supply and returns. Additionally, it can prevent mechanical conflict in the machine due to prior art single supply and return hoses.

<FIG> is an example system <NUM> that includes a cooling loop assembly <NUM> with multiple inlets and multiple outlets for thermal management and liquid cooling of microelectronic assemblies, including chips, such as semiconductor chips, or other devices (not shown) on a circuit board <NUM> of a computing device <NUM>. The circuit board <NUM> may include a motherboard, main board or the like. In this example, the cooling loop assembly <NUM> may be the only loop implemented on the circuit board <NUM> for cooling all components connected to the cooling loop assembly <NUM> on the circuit board <NUM>, but the cooling loop assembly <NUM> can be implemented with other loops on the circuit board <NUM>, as well. In this example, cooling loop assembly <NUM> is fluidly connected to each of the cooling devices 130a,130b, 130c,130d on the circuit board <NUM>. Each cooling device 130a, 130b, 130c, 130d may include a respective inlet 132a, 132b, 132c, 132d and a corresponding outlet 134a, 134b, 134c, 134d. Cooling loop assembly <NUM> and corresponding computing device <NUM> may be one of a plurality of cooling loops and computing devices in a data center server rack (see <FIG>), but in other examples the cooling loop may be utilized for a standalone individual computing device and cooling loop.

Cooling loop assembly <NUM> may include at least two inlet connectors: a first inlet connector 140a and a second inlet connector 140b, but a single inlet connection or more than two inlet connectors may also be utilized. Cooling loop assembly <NUM> further includes at least two outlet connectors: a first outlet connector 142a and a second outlet connector 142b, but a single outlet connector or more than two outlet connectors may also be utilized. In this example, first and second inlet connectors 140a, 140b may be coupled or connected to a single liquid cooling inlet source or manifold (not shown in this view) of a larger cooling system (see <FIG>), and first and second outlet connectors may be coupled or connected to a single liquid cooling outlet or manifold (not shown in this view).

The first and second inlet connectors 140a, 140b, as well as the first and second outlet connectors 142a, 142b may be quick disconnect couplings or fittings that are designed to provide fast and easy connection and disconnection of lines to the respective inlet source and outlet source. Examples of quick disconnect fittings can include snap type (spring loaded ball latching), non-latching, double shut-off, and dry break quick disconnect fittings. Utilizing quick disconnect fittings can provide a higher pressure drop than simpler or other types of mechanical fittings. Further, quick disconnect fittings do not require the use of tools to assemble and disassemble. In other examples, different types of connectors may be utilized, such as quick connectors.

A first board supply line 144a connects the first inlet connector 140a to the board inlet manifold <NUM> and a second board supply line 144b connects the second inlet connector 140b to the board inlet manifold <NUM>. Cooling liquid is distributed across the circuit board <NUM> along a length of the board inlet manifold <NUM>. This enables cooling liquid to reach and be distributed to each of the cooling devices 130a, 130b, 130c,130d. As shown, the board inlet manifold <NUM> is positioned adjacent at least one of the edges of each of the cooling devices 130a,130b,130c,130d. But, in other examples, the board inlet manifold <NUM> can be positioned anywhere on the circuit board <NUM> and may be any shape or size. Secondary supply lines 146a,146b,146c,146d may extend from the board inlet manifold <NUM> to the inlets 132a, 132b, 132c, 132b of each respective cooling device 130a,130b,130c,130d.

A first return line 148a joins the first outlet connector 142a to the board outlet manifold <NUM> and a second return line 148b joins the second outlet connector 142b to the board outlet manifold <NUM>. The board outlet manifold <NUM> can run parallel to the board inlet manifold <NUM> and in this example, extends along a length L of the circuit board and adjacent at least one edge of each of the cooling devices 130a, 130b, 130c, 130d. Secondary return lines 154a, 154b, <NUM>54c, 154d extend between the board outlet manifold <NUM> and each of the outlets 134a, 134b, 134c, 134d of the respective cooling devices 130a, 130b, 130c, 130d.

The supply and return lines, as well as any secondary supply and return lines and manifolds may be hoses, pipes, or other structures that enable fluid to move from a liquid source through the inlet connections, through the cooling loop assembly and cooling devices, and back out through the return lines to an outlet. In some examples, the supply and return lines may be flexible hoses or pipes to allow for easier user handling. In other examples, the supply and return lines may be more rigid to better allow for automation of the connection and disconnection of the return and supply lines. In still other examples, the system may incorporate both flexible and rigid lines.

The cooling devices 130a,130b,130c,130d may be conventional liquid cooling devices, such as cold plates or other devices that can be used in connection with liquid cooling of a chip, or microelectronic element, device, or assembly. The cooling devices 130a, 130b, 130c, 130d are shown having an identical configuration, but in other examples, one or more of the cooling devices 130a, 130b, 130c, 130d may differ. Further, the cooling devices 130a,130b,130c,130d may be arranged in any configuration, and any number of cooling devices may be arranged in the system.

The loop assembly <NUM> may continually pump cooling liquid through the inlet supply lines and heated return cooling liquid through the outlet lines. The cooling liquid may be any known cooling liquid from a continuous source of cooling liquids or a chiller system. Cooling liquid may include, for example, water, deionized water, inhibited glycol and water solutions, dielectric fluids, and any suitable cooling liquid.

During operation, cooling liquid from a primary source, such as a rack supply manifold <NUM> (see <FIG>), can enter the loop assembly <NUM> through the first and second inlets of the loop assembly <NUM> at the first and second quick disconnects 140a, 140b and first and second board supply lines 144a,144b in the direction shown by the arrows in the figure. Incoming cooling liquid, which will be represented by a diagonal cross-hatching pattern in the examples described herein, will then be dispersed along the length of the board inlet manifold <NUM>, and into each of the secondary inlet lines 146a,146b,146c,146d. The flow of cooling liquid will continue through each of the secondary inlet lines 146a,146b,146c,146d and into the inlets 132a,132b,132c,132d of the respective cooling devices 130a,130b,130c,130d. Heated surfaces of the cooling devices 130a,130b,130c,130d will heat and increase the temperature of the cooling liquid. Heated return cooling liquid, which will be represented by square cross hatching in the examples described herein, exits each cooling device 130a, 130b, 130c, 130d through the respective cooling device outlets 134a, 134b, 134c, 134d and flows to the board outlet manifold <NUM> through secondary outlet lines 154a, 154b, 154c, 154d in the direction shown by the arrows. Heated return cooling liquid will be dispersed from the board outlet manifold <NUM> through first and second return lines 148a, 148b and first and second outlet connectors 142a,142b, each of which are connected to a single primary outlet, such as rack return manifold (see <FIG>). It is to be appreciated that the arrows used throughout the descriptions of the various examples herein indicate a direction of flow of liquid in the respective cooling loop assemblies.

<FIG> is a schematic view showing an example system 100A that includes the first and second inlet connectors 140a, 140b and first and second outlet connectors 142a,142b of primary cooling loop <NUM> and computing device <NUM> on a shelf <NUM> of a computing device rack <NUM>. A plurality of other computing devices <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> and cooling loop assemblies <NUM>-<NUM>, <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> are also shown on shelves <NUM> of the computing device rack <NUM>. The computing device rack <NUM> in this example is one in a series of three computing device racks, as shown in <FIG>, but in other examples, computing device rack <NUM> may be a single data rack connected to at least one cooling system, or may be one in a series of any number of racks connected to the at least one cooling system. <FIG> illustrates the multiple inlet and outlet connection points to the cooling liquid source and liquid return outlet. The racks and computing devices are shown in one arrangement, but any arrangement may be utilized.

A rack supply manifold <NUM> can extend the length of the computing device rack <NUM> and may be connected to a central cooling supply line <NUM>. Rack supply manifold <NUM> can distribute cooling liquid from the central cooling liquid supply line <NUM> along the length of the entire rack <NUM> and provide a source of cooling liquid to each cooling loop assembly <NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> of each respective computing device <NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>,<NUM>-<NUM>. A rack return manifold <NUM> can similarly extend the length of the opposed side of the computing device rack <NUM>, provide a return outlet for heated cooling liquid returning from the system, and pump liquid out of the computing device rack <NUM> and into a central return line <NUM> to return heated cooling liquid to a system source for cooling. The rack inlet and rack return manifolds <NUM>,<NUM> extend along the sides of the rack, but in other examples, one or both of the rack inlet and rack return manifolds can be positioned anywhere adjacent the computing device rack <NUM>, including, for example, the front, rear, or on the same side of the server rack.

First inlet connector 140a of cooling loop assembly <NUM> is shown joined to a first rack inlet connection point 170a on the rack supply manifold <NUM> and a second inlet connector 140b of the cooling loop assembly <NUM> is joined to a second rack inlet connection point 170b on the rack supply manifold <NUM>. First inlet connectors 140c, 140e, <NUM>,140i of the remaining cooling loop assemblies <NUM>-<NUM> through <NUM>-<NUM> are also joined to rack supply manifold <NUM> at respective first rack inlet connection points 170c, 170e, <NUM>, 170i. Similarly, second inlet connectors 140d,140f,<NUM>,140j are joined to rack supply manifold <NUM> at respective second rack inlet connection points 170d,170f, <NUM>,170j.

Second outlet connector 142a of cooling loop assembly <NUM> is shown joined to a first rack outlet connection point 172a on the rack return manifold <NUM>, and a second outlet connector 142b of the cooling loop assembly <NUM> is joined to a second rack outlet connection point 172b on the rack outlet inlet manifold <NUM>. First outlet connectors 142c, 142e, <NUM>, 142i of the remaining cooling loop assemblies <NUM>-<NUM> through <NUM>-<NUM> are also joined to rack return manifold <NUM> at respective first rack outlet connection points 172c, 172e, <NUM>, 172i. Similarly, second outlet connectors 142d,142f,<NUM>,142j are joined to rack return manifold <NUM> at respective second rack outlet inlet connection points 172d,172f, <NUM>,172j.

With reference to <FIG>, a schematic coolant distribution system <NUM> comprises computing device rack <NUM>, as one in a series of three data racks, but in other examples, any number of racks may be utilized. The coolant distribution system <NUM> can also comprise a central supply and return sub-system <NUM> includes a source or storage tank <NUM> of cooling liquid, chilling unit <NUM>, central distribution unit <NUM>, and central supply line <NUM> and central return line <NUM>. The coolant distribution system <NUM> contains the pumping systems for coolant circulation, and cooling units to remove the heat from the coolant by transferring the heat to datacenter process water. System <NUM> can also be a standalone system that removes the heat from the coolant by transferring the heat to air through a fan coil or by transferring the heat to a refrigerant through a chiller system.

The coolant distribution system pumps and distributes cooling liquid through the central cooling liquid inlet line <NUM> and into each of the rack supply manifolds <NUM>, <NUM>-<NUM>,<NUM>-<NUM> and each first and second connector (not shown) of each computing device (not shown) in the respective computing device racks <NUM>, <NUM>-<NUM>, <NUM>-<NUM>. Heated return cooling liquid will exit the respective computing device racks <NUM>, <NUM>-<NUM>, <NUM>-<NUM> through the respective rack return manifolds <NUM>, <NUM>-<NUM>,<NUM>-<NUM> and into the central cooling liquid outlet line <NUM> and back to the central distribution unit <NUM>. The central distribution unit <NUM> will then pump the heated return cooling liquid into the cooling unit <NUM> where the heated return cooling liquid will be cooled and then circulated back into the central distribution unit <NUM> and pumped back into the system. It is to be appreciated that the components of the central supply and return sub-system <NUM> provides one example for cooling and distributing cooling liquid into the central inlet and outlet lines and throughout each of the racks <NUM>,<NUM>-<NUM>,<NUM>-<NUM>, but numerous other examples may be used to accomplish the same features. For example, the lines may run underground, additional or fewer cooling components may be used.

Referring to <FIG>, another schematic example <NUM> of a cooling loop assembly <NUM> is shown. This example is otherwise similar to the example of <FIG>, except that the first and second inlet connectors 240a,240b and the first and second outlet connectors 242a,242b, as well as the supply lines 2244a,244b, return lines <NUM>,248b, secondary outlet lines 254a,254b,254c,254d, and the board outlet manifold <NUM> are positioned on the same side of the circuit board <NUM> of the computing device <NUM>. As shown, the flow of cooling liquid through the cooling loop <NUM> will follow a circular pattern, as shown by the arrows, so as to allow for cooling liquid to return to the same side of the circuit board <NUM>, instead of a more linear pattern from one end of the board to the other, as in the example of <FIG>. For example, cooling liquid from a rack supply manifold flows <NUM> (<FIG>) flows through first and second inlet connectors 240a,240b and through first and second supply lines 244a,244b to the board inlet manifold <NUM>. Cooling liquid is distributed from the board inlet manifold <NUM> to inlets 232a,232b,232c,232d of cooling devices 230a,230b,230c,230d through secondary inlet lines 246a,246b,246c,246d. Heated return cooling liquid will exit the cooling devices 230a,230b,230c,230d through their respective outlets 234a,234b,234c,234d and flow to the board outlet manifold <NUM>. First and second return lines 248a,248b will distribute cooling liquid from the board outlet manifold <NUM> to connectors 242a,242b and the rack return manifold <NUM>.

As better seen in the example rack system 200A of <FIG>, in this cooling loop <NUM> configuration, rack supply manifold <NUM> and rack return manifold <NUM> can be positioned directly adjacent one another and on the same side of computing device <NUM> and/or the cooling devices. Cooling loop assembly <NUM> and computing device <NUM> can be positioned on a shelf of computing device rack <NUM>. First and second inlet connection points 270a,270b are positioned on the rack supply manifold <NUM>, and first and second outlet connection points 272a,272b are positioned on the rack return manifold. Cooling loop assembly <NUM> is joined to the first and second inlet connection points 270a,270b via first and second inlet connectors 242a,242b and first and second board supply lines 244a,244b. Additionally, cooling loop assembly <NUM> is joined to the first and second outlet connection points 272a,272b via first and second outlet connectors 242a,242b and first and second return lines 248a,248b.

As in the previous example, one or more similar computing devices <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> with similar cooling loop assemblies <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> can be provided on additional shelves <NUM> of a computing device rack <NUM>, all of which are coupled to the rack supply manifold <NUM> and a rack return manifold <NUM>. A plurality of additional first and second inlet connectors 242c-j, as well as corresponding first and second board supply lines 244c-244j are joined to the rack supply manifold <NUM> at respective rack inlet connection points 270c-j. Additional first and second outlet connectors 272c-j and corresponding first and second return lines 248c-j are also joined to the rack outlet manifold <NUM> at respective rack outlet connection points 272c-j. In other examples, some of the computing devices may be connected to a separate set of rack supply and return manifolds.

<FIG> illustrates a schematic cooling distribution system <NUM> that comprises a computing device rack <NUM>, as well as any number of additional computing device racks, such as computing device racks <NUM>-<NUM>,<NUM>-<NUM>, as well as a cooling and return sub-system <NUM>. The cooling and return sub-system <NUM> can include a cooling liquid supply or storage tank <NUM>, a chilling system <NUM>, a central distribution unit <NUM>, a central supply line <NUM> and central return line <NUM>. The components of the central supply and return sub-system <NUM> can be identical to the previous example and is not discussed at length in this example, but other cooling and return sub-systems may be utilized herein. For ease of illustration, the components within the computing device racks are not shown. Each of the computing device racks <NUM>,<NUM>-<NUM>,<NUM>-<NUM> include rack supply manifolds <NUM>,<NUM>-<NUM>,<NUM>-<NUM> and rack return manifolds <NUM>, <NUM>-<NUM>,<NUM>-<NUM>, positioned on the same side of the respective computing device racks <NUM>,<NUM>-<NUM>,<NUM>-<NUM>. As in the previous example, cooling liquid can be distributed to each of the computing device racks <NUM>,<NUM>-<NUM>,<NUM>-<NUM> through a central supply line <NUM>, which feeds into each of the rack supply manifolds <NUM>,<NUM>-<NUM>,<NUM>-<NUM>. As described in <FIG>, cooling liquid can be distributed from the central supply line and rack supply manifolds into each of the computing devices through the first and second inlet connectors and board supply lines. Heated return cooling liquid can then travel through the return lines and outlet connectors and up through the rack return manifolds <NUM>, <NUM>-<NUM>,<NUM>-<NUM> into the central return line <NUM>. The heated cooling liquid can then be processed and cooled down in the components of the central supply and return sub-system <NUM> and recirculated through the system again.

Although the supply lines 244a,244b and return lines 248a,248b that extend to the rack supply manifold <NUM> and rack return manifold <NUM> are shown having example lengths, in other examples, the length of the supply and return lines can vary widely. This can allow for the cooling loop assembly <NUM> to be alternatively arranged on a computing device rack <NUM> in a similar configuration as the loop assembly <NUM> of <FIG> and <FIG>. For example, supply lines 244a,244b would extend to the rack supply manifold <NUM> of <FIG> on one side of the computing device rack <NUM>, and return lines 246a,246b having a sufficient length to extend across the circuit board <NUM> can reach and be connected to the rack return manifold <NUM> of <FIG> positioned on a different side, including an opposite side of the computing device rack <NUM> of <FIG>. The reverse is also true. Cooling loop assembly shown in <FIG>, can also be configured and positioned on a computing device rack <NUM> of <FIG> relative to the directly adjacent rack supply manifold <NUM>, and the rack return manifold <NUM>.

The configurations shown in <FIG> can be easily scalable and can allow for an increased number of supply and return lines. Such configurations can provide low pressure drop through the lines. Furthermore, such designs can provide for versatility in redundancy of the cooling loop designs, as will be discussed in more detail with regard to the example redundant systems described with regard to <FIG>.

<FIG> is another example system <NUM> that includes a cooling loop assembly <NUM> where the inlet configuration differs from the previous examples. In this example, an inlet tee <NUM> can be used to join a first inlet connector 440a and a second inlet connector 440b together so as to create a branched inlet connector or connection <NUM>. As shown, the first connector inlet line 436a extends between the inlet tee <NUM> and the first inlet connector 440a. A second connector inlet line 436b extends between the inlet tee <NUM> and the second inlet connector 440b. The inlet tee <NUM> allows for distribution of combined cooling liquid flowing from the first inlet connector 440a and the second connector 440b to the board inlet manifold <NUM> through a board supply line <NUM>. Use of a branched inlet connection <NUM> can help to avoid conflicts arising from the use of additional multiple inlet lines, as well as allow for increased space by eliminating extra lines. It can allow for retrofitting from existing single supply and return designs, and also provides a configuration that allows for versatility in creating a redundant system, such as in the examples discussed below regarding <FIG>.

An outlet tee connector <NUM> can be used to join a first outlet connector 442a and a second outlet connector 442b together so as to create a branched outlet connector or connection <NUM>. As shown, the first connector outlet line 438a extends between the outlet tee connector <NUM> and the first outlet connector 442a. A second connector outlet line 438b extends between the outlet tee connector <NUM> and the second outlet connector 442b. The outlet tee connector <NUM> allows for the distribution of heated return cooling liquid flowing from the rack return manifold <NUM> through the return line <NUM> and into the outlet tee connector <NUM>. At the outlet tee connector <NUM> the heated return cooling liquid can be divided between the first connector outlet line 438a and the second connector outlet line 438b.

During operation, cooling liquid will be pumped from a liquid source through the first and second connectors 440a,440b and through the inlet tee <NUM> to the board supply line <NUM>. The board supply line <NUM>, which extends from the inlet tee <NUM> to the board inlet manifold <NUM>, can further distribute the combined cooling liquid entering the cooling loop assembly <NUM> from the first inlet connector 440a and the second inlet connector 440b to the board manifold <NUM>. Cooling liquid in the board manifold <NUM> will be further distributed to the inlets 432a, 432b,432c,432d of the cooling devices 430a,430b,430c,430d through respective secondary inlet lines 446a,446b,446c,446d.

Heated return cooling liquid exits the outlets 434a,434b,434c,434d of each respective cooling device 430a,430b,4340c,430d and flows to the board outlet manifold <NUM> through secondary outlet lines 454a,454b,454c,454d. Heated return cooling liquid will then flow from board outlet manifold <NUM> to the outlet tee connector <NUM>, where the flow will then be divided between the first and second connector lines 438a,43b as the flow continues to the first and second outlet connectors 442a,442b and to an outlet, such as a rack return manifold.

<FIG> illustrates a schematic example of a rack system 400a that includes computing device <NUM> and cooling loop assembly <NUM> as part of a larger computing device rack <NUM>, as well as branched inlets or dual inlet and dual outlet connection points to the cooling liquid source and liquid return outlet for each computing device <NUM>-<NUM>, <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> on the respective shelves <NUM>. The racks and computing devices are shown in one arrangement, but any arrangement may be utilized. As in the previous examples, rack supply manifold <NUM> may be connected with central supply line <NUM> and rack return manifold <NUM> can be connected to the central return line <NUM>. With reference to the first shelf <NUM> and computing device <NUM>, cooling liquid can enter the rack supply manifold <NUM> and be distributed through the first inlet connectors 440a at the first rack inlet connection point 470a and the second inlet connector 440b at the second rack inlet connection point 470b. The inlet tee <NUM> will combine the flow of cooling liquid through the first connector line 436a and the second connector line 436b and direct the flow through the board supply line <NUM>. The flow will be distributed throughout the rest of the cooling loop assembly <NUM> as previously described and heated return cooling liquid will exit the computing device <NUM> through the board return line <NUM>. The outlet tee <NUM> will then divide the volume of the heated return cooling liquid and direct the heated return cooling liquid through the first and second outlet lines 438a,438b and outlet connectors 442a,442b and into the rack return manifold <NUM> through the rack connection points 470a,470b. The flow will then be distributed to the central outlet line <NUM>.

Additional computing devices <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> can include a similar configuration. Cooling liquid will enter the rack supply manifold <NUM> and be further distributed through the first inlet connectors 440c, 440e, <NUM>, 440i at each respective first rack inlet connection point 470c, 470e, <NUM>, 470i, as well as through the second inlet connectors 440c,440e,<NUM>,440i at each respective second rack inlet connection point 470d,470f,<NUM>,470j. Heated return cooling liquid will exit each of the computing devices <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> through the respective first outlet connectors 442c, 442e, <NUM>, 442i at each first rack outlet connection point 472c, 472e, <NUM>, 472i, as well as through the respective second outlet connectors 442d, 442f, <NUM>, 442j at each second rack outlet connection point 472d, 472f, <NUM>, 472j. In this example, each of the cooling loop assemblies <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM> include a similar configuration as cooling loop assembly <NUM> where a tee is utilized to create branched inlets and outlets, but in other examples, any of the cooing loop assembly configurations may differ, including the number of inlets and outlets and the type of connections. The rack <NUM> can also be part of a larger system of racks, as previously described with regard to <FIG> and <FIG>.

In the example described, the first and second inlet connectors 440a,440b are joined together by inlet tee <NUM>, and the first and second outlet connectors 442a,442b are joined together by an outlet tee connector <NUM>. In other examples, one set of connectors may have a direct connection to a respective board manifold and the other set of connectors may include a tee connector and have an indirect connection to the respective board manifold. For example, the first and second inlet connectors 440a,440b may be directly joined to the board inlet manifold <NUM>, as described in any one of the foregoing examples, and the first and second outlet connectors 442a,442b may be indirectly joined to the board outlet manifold through the outlet tee connector <NUM> and return line <NUM>. Conversely, an opposite configuration is also possible in which the first and second inlet connectors 440a,440b are joined together by the inlet tee <NUM> and indirectly connected to the board inlet manifold <NUM> through the board supply line <NUM>, and the first and second outlet connectors 442a,442b are directly connected to the board outlet manifold <NUM>.

In the previous examples, a single cooling loop assembly is disclosed that includes an equal number of multiple inlet connection points to a single fluid inlet source and multiple outlet connection points to the fluid outlet source, when further distributing cooling liquid to and from the cooling devices through a board manifold. In other example configurations, including any of the configurations being described herein, the number of inlet connection points and outlet connection points can be optimized to achieve desired thermal and mechanical properties in the cooling loop assembly, as well as the overall system. For example, the number of inlet and outlet connection points does not need to be the same and fewer inlet and/or outlet connection points can be utilized in the cooling loop assembly.

For example, with reference to <FIG>, a cooling loop assembly <NUM> for thermal management of components and cooling devices 530a,530b,530c,530d on a circuit board <NUM> of a computing device <NUM> is shown. The cooling loop assembly <NUM> is otherwise identical to the cooling loop <NUM> of <FIG>, except that in this example, only one outlet connector <NUM> is present in the cooling loop assembly <NUM>. As shown, cooling loop assembly <NUM> includes first and second inlet connectors 540a,540b connected to a board inlet manifold <NUM>, but only includes one outlet connector <NUM> that is connected to a board outlet manifold <NUM>. As shown by the arrows in <FIG>, the flow of cooling liquid through the board inlet manifold <NUM> and into each of the cooling devices 530a,530b,530c,530d, as well as the flow of the heated return cooling liquid out of the cooling devices 530a,530b,530c,530d and to the board outlet manifold <NUM> otherwise remains the same.

Multiple loop assemblies which include either or both multiple inlet connection points or multiple outlet connection points are also disclosed. <FIG> is one example of a circuit board <NUM> of a computing device <NUM> with a first cooling loop assembly 610A and a second cooling loop assembly <NUM>.

The first cooling loop assembly <NUM> includes two inlets and one outlet. As shown, a branched inlet, as previously disclosed, includes a first inlet connector 640a and a second inlet connector 640b joined together by an inlet tee <NUM>. A single board supply line <NUM> connects the tee connector <NUM> to a board inlet manifold <NUM>, which is connected to the cooling devices 630a, 630b along secondary inlet lines 646a. Secondary outlet lines 654a,654b extend from the cooling devices 630a,630b to the board outlet manifold <NUM>, which in turn is connected to outlet connector <NUM> and outlet <NUM> by a single return line <NUM>. As in the previous examples, cooling liquid can flow in the directions shown by the arrows.

The second cooling loop assembly <NUM> includes one inlet <NUM> having an inlet connector <NUM> and two outlets 726a,726b with corresponding outlet connectors 726a,726b. Cooling liquid flows into supply line <NUM> through inlet <NUM> connector <NUM>. Supply line <NUM> feeds into a board manifold <NUM>, which is configured to distribute cooling liquid to cooling devices 730a,730b through secondary inlet lines 746a,746b. Heated return cooling liquid that exits cooling devices 730a, 730b will flow through secondary outlet lines 754a,754b to the board outlet manifold <NUM>. Return outlet lines <NUM>,748b will distribute heated return cooling liquid from the board outlet manifold to outlet 726a,726b and connectors 742a,742b.

Any number of cooling devices may be implemented within a cooling loop assembly of a system. In any of the examples described herein, it may be desired to include a cooling loop assembly that has only one cooling device or at least one cooling device. In other examples, at least two, at least three, or at least five cooling devices can be used in the system according to aspects of the disclosure.

<FIG> illustrates an example system <NUM> that includes an example cooling loop assembly <NUM> coupled to a single cooling device <NUM> overlying a circuit board <NUM> of a computing device <NUM>. At least three inlet connectors 2140a,2140b,2140c can be used to supply cooling liquid to the cooling device <NUM>, but in other examples greater or fewer inlet connectors may be used. In some examples, three inlet connectors 2140a,2140b,2140c can be coupled to a board inlet manifold <NUM> by board supply lines 2144a, 2144b, 2144c, but in other examples greater or fewer outlet connectors may be used. At least three outlet connectors 2142a,2142b,2142c are coupled to the cooling device <NUM>. In this example, board return lines 2148a,2148b,2148c can directly connect the outlet connectors 2142a,2142b,2142c to the board return manifold <NUM>, and a secondary return line <NUM> couples the board return manifold <NUM> to the outlet <NUM> of cooling device <NUM>. Similarly, board supply lines 2144a,2144b,2142c can directly connect the inlet connectors 2140a,2140b,2140c to the board inlet manifold <NUM>, and a secondary supply line <NUM> couples the board supply manifold <NUM> to the inlet <NUM> of the cooling device <NUM>. When the supply and return lines are oriented in this configuration, the outlet connectors 2142a,2142b,2142c can directly connect to a rack return manifold (not shown) oriented on one side of the cooling device <NUM>, and the inlet connectors 2140a,2140b,2140c can connect directly connect to a rack supply manifold (not shown) oriented on the opposite side of the cooling device <NUM>. In some examples, the rack supply and rack return manifolds may be oriented as shown in <FIG>.

<FIG> shows an alternative system <NUM>-<NUM> in which inlet connectors 2140a,2140b,2140c and outlet connectors 2142a,2142b,2142c are coupled to cooling device <NUM> in a different configuration. The lines can instead be oriented so that the outlet connectors 2142a,2142b,2142c and inlet connectors 2140a,2140b,2140c, the corresponding supply lines 2144a, 2144b, 2144c, <NUM> and return lines 2138a, 2148b,2148c, <NUM>, and the board inlet manifold <NUM> and board return manifold <NUM> can be joined to a rack return manifold (not shown) and a rack supply manifold (not shown) oriented on the same side of the cooling device <NUM>, which is also oriented on the same side of the circuit board <NUM>. For example, the inlet connectors 2140a,2140b,2140c and outlet connectors 2142a,2142b,2142c can join to the rack supply manifold <NUM> and rack return manifold <NUM> shown in <FIG>.

<FIG> shows another example system <NUM>-<NUM> with a branched inlet connector or connection <NUM> and a branched outlet connector or connection <NUM>. The branched inlet and outlet connections can be identical to the example disclosed in <FIG> and are not described in detail. In this configuration, both the inlet connectors 2240a,2240b,2240c and outlet connectors 2242a,2242b,2242c are positioned on one side or the same side of the cooling device <NUM>. Since only one cooling device <NUM> is provided on the system, a board supply line <NUM> can directly connect the branched inlet connector <NUM>, including inlet connectors 2240a,2240b and the inlet tee <NUM>, to the inlet <NUM> of the cooling device <NUM>. Similarly, a board return line and a board return line <NUM> can couple or directly connect the branched outlet connectors <NUM>, including outlet connectors 2242a,2242b and outlet tee <NUM>, to the outlet <NUM> of the cooling device <NUM>. This can allow for connection to a rack supply manifold and a rack return manifold positioned on the same side of the cooling device, as in prior examples. Alternatively, either one of the inlet connectors and outlet connectors may be positioned on an opposite side of the cooling device <NUM>. This can allow for connection to a rack supply manifold and a rack return manifold positioned on opposite sides of the cooling device, as in prior examples.

The use of multiple inlets and outlets, as well as multiple loop assemblies on a single circuit board can be used to create redundancy in the overall cooling system, while reducing the pressure drop at the inlet and outlet couplings/connectors. <FIG> illustrates an example system <NUM> that includes a circuit board <NUM> of a computing device <NUM>, cooling devices 830a,830b,830c,830d, and two cooling loop assemblies, a first cooling loop assembly <NUM> and a second cooling loop <NUM>. In this configuration, each of the first and second loop assemblies <NUM>,<NUM> has an independent assembly source inlet manifold and an assembly outlet manifold. Both the first and second cooling loop assemblies <NUM>, <NUM> can feed cooling liquid into and out of the same cooling devices 830a,830b,830c,830d. As will be described in more detail below, in the event one or a portion of the first and/or second cooling loops <NUM>,<NUM> malfunctions, the other cooling loop, which has a separate liquid source, remains in operation, thereby allowing for redundancy of the system.

In this example, the two cooling loop assembles are disposed on opposed sides of the cooling devices 830a,830b,830c,830d, but the cooling loops can remain in fluid communication in any configuration relative to the cooling devices 830a,830b,830c,830d. First cooling loop assembly <NUM> is positioned on the left side of the circuit board <NUM>, whereas the second cooling loop assembly <NUM> is positioned on the right side of the circuit board <NUM>. The first cooling loop assembly <NUM> is similar to the cooling loop assembly <NUM> previously described with regards to <FIG>. Similar reference numerals will be used to identify similar components and a full discussion is not repeated here. The inlets 826a,826b, including first and second inlet connectors 840a,840b of the first cooling loop assembly <NUM> can be connected to a first cooling liquid supply source that is external to the circuit board, such as a rack supply manifold (not shown) via first and second connectors 840a,840b. The outlets 826a,826b, including first and second outlet connectors 842a,842b are similarly connected to an outlet external to the circuit board <NUM>, such as rack return manifold (not shown).

Second cooling loop assembly <NUM> is a redundant cooling loop assembly that is a mirror image of first cooling loop assembly <NUM> and operates in a similar way and where similar reference numerals will be used to identify similar features. Cooling liquid flowing toward cooling devices 830a,830b,830c and in the inflow direction are illustrated with a dotted pattern. The outlets 824a,<NUM> of the second cooling loop assembly <NUM>, including first and second outlet connectors 940a,940b, are connected to a redundant and second cooling liquid supply source that is external to the circuit board, such as a rack supply manifold <NUM> (<FIG>). The outlets 926a,926b are connected to a second assembly outlet, such as rack return manifold <NUM> (<FIG>). Heated return cooling liquid flowing away from cooling devices 830a,830b,830c and in the outflow direction are illustrated with a diamond pattern.

Each of the cooling devices 830a,830b,830c,830d include dual inlets and outlets, although in other examples the devices include only a single inlet and a single outlet. As shown, cooling devices 830a,830b,830c,830d have first cooling device inlets 832a,832b,832c,832d and respective second cooling device inlets 932a,932b,932c,932d, as well as respective first cooling device outlets 834a,834b,834c,834d and respective second cooling device outlets 934a,934b,934c,934d. In this configuration, the first cooling device inlets 832a,832b,832c,832d are respectively connected to secondary supply lines 846a,846b,846c,846d of the first cooling loop assembly <NUM>. The second cooling device inlets 932a,932b,932c,932d are respectively connected to secondary supply lines 846a,846b,846c,846d of the second cooling loop assembly <NUM>. Similarly, first cooling device outlets 834a,<NUM>,834c,834d are connected to respective secondary return lines 854a,854b,854c,854d of the first cooling loop assembly <NUM>. Second cooling device outlets 934a,934b,934c,934d are respectively connected to secondary return lines of the second cooling loop assembly <NUM>.

During operation, cooling liquid flows into the first inlets 832a,832b,832c,832d and second inlets 932a,932b,932c,932d of the respective cooling devices 830a,830b,830c,830d from cooling liquid supplied by both the first cooling loop <NUM> and the second cooling loop <NUM>. Cooling liquid will flow through the first and second inlet connectors 840a,840b of the first loop assembly <NUM> to the respective cooling devices 830a,830b,830c,830d. In one example, cooling liquid will flow through the first and second board supply lines 844a,844b, to the board supply manifold <NUM>, and through respective secondary supply lines 846a,846b,846c,846d to respective first inlets 832a,832b,832c,832d of the respective cooling devices 830a,830b,830c,830d. Similarly, a second source of cooling liquid will flow through the first and second inlet connectors 940a,940b of the second loop assembly <NUM> to the respective second inlets 932a,932b,932c,932d of the respective cooling devices 830a,830b,830c,830d. In one example, cooling liquid will flow through the first and second board supply lines 944a,944b, to the board supply manifold <NUM>, and through respective secondary supply lines 946a,946b,946c,946d to respective second inlets 932a,932b,932c,932d.

Cooling liquid within each of the cooling devices 830a,830b,830c,830d will then exit from the respective cooling device through first outlets 834a,834b,834c,834d and second outlets 934a,94b,934c,934d. In one example, cooling liquid from the first outlets 834a,834b,834c,834d will flow through secondary supply lines 854a,854b,854c,854d of the first cooling liquid assembly to the board return manifold <NUM>, as well as through primary return lines 848a,848b and to the outlets 826a,826b through outlet connectors 826a,826b. Cooling liquid from the second outlets 934a,934b,934c,934d of the second loop assembly <NUM> will also direct flow to the outlet connectors 842a,842b. In one example, heated cooling liquid will flow from the respective second outlets 934a,934b,934c,934d through secondary supply lines 954a,954b,954c,954d of the second cooling assembly <NUM> to the board return manifold <NUM>, as well as through primary return lines 948a,948b and through outlet connectors 926a,926b of the second loop assembly.

Turning to <FIG>, a rack assembly <NUM>, which includes circuit board <NUM> of computing device <NUM> and first and second cold plate loop assemblies <NUM>,<NUM>, is shown positioned on a top shelf <NUM> of a larger data rack <NUM>, and cooling system. Additional computing devices <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> are positioned on additional shelves <NUM> within the rack <NUM>. First cooling liquid assemblies <NUM>-<NUM>,<NUM>,<NUM>,<NUM>-<NUM>,<NUM>-<NUM> and the redundant second cooling liquid assemblies <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> are shown extending into each of the respective computing devices <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>.

For ease of illustration, the components within the computing device <NUM> are not illustrated in detail and instead portions of the first cold loop assembly <NUM> and the second cold loop assembly <NUM> that are positioned within computing device <NUM> are represented by a broken line rectangle. Since the first and second cold loop assemblies <NUM>,<NUM> overlap along the inlets and outlets of the cooling devices (<FIG>), the broken line rectangles are shown overlapping.

The rack assembly <NUM> includes a first rack supply manifold <NUM> and a second rack supply manifold <NUM> that supplies cooling liquid to all computing devices <NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> on the rack with supply lines attached to the respective first rack supply manifold <NUM> and second rack supply manifold. Similarly, a first rack return manifold <NUM> and a second rack return manifold <NUM> provide an outlet for heated return cooling liquid exiting the respective computing devices <NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> through supply lines attached to the first rack return manifold and second rack return manifold <NUM>. A first central supply line <NUM> feeds cooling liquid into the first rack supply manifold <NUM> and first central return line <NUM> distributes return cooling liquid back to the cooling central supply and return subsystem <NUM> (<FIG>). A second central supply line <NUM> feeds cooling liquid into the second rack supply manifold <NUM> and first central return line <NUM> distributes heated return cooling liquid from the second manifold <NUM> to the central supply and return subsystem <NUM> (<FIG>).

The first rack supply manifold <NUM> includes a plurality of inlets or openings along its length, including first rack inlet connection points 870a, 870c,870e,<NUM>,870i and plurality of second rack inlet connection points 870b,870d,870f,<NUM>,870j. The first rack return manifold <NUM> similarly includes a plurality of openings along its length, including first rack outlet connection points 872a, 872c,872e,<NUM>,872i and a second rack outlet connection points 872b,872d,872f,<NUM>,872j.

The second rack supply manifold <NUM> and return manifold <NUM> are redundant manifolds that provide a secondary or redundant source of cooling liquid for each computing device on the shelves <NUM>, and allow for distribution of heated return cooling liquid from a separate cooling liquid source, as discussed in more detail in <FIG>. The second rack supply manifold <NUM> includes a plurality of inlets or openings along its length to allow for connection of supply lines to the manifold, including first rack inlet connection points 970a, 970c, 970e,<NUM>,970i and plurality of second rack inlet connection points 970b,970d,970f,<NUM>,970j. The first rack return manifold <NUM> similarly includes a plurality of openings along its length to allow for connection of return lines to the manifold, including first rack outlet connection points 972a, 972c, 972e,<NUM>,972i and a second rack outlet connection points 972b,972d,972f,<NUM>,972j.

With reference to the first computing device <NUM> in the assembly <NUM>, at one end of the computing device <NUM> the first cooling loop assembly <NUM> is connected to the first rack supply manifold <NUM> and rack return manifold <NUM>. The first inlet connector 840a of the first cooling loop assembly <NUM> is joined to the first connection points 870a on the first rack supply manifold <NUM> and the second inlet connector 840b of the first cooling loop assembly <NUM> is connected to the second connection point 870b. This provides for fluid communication between the first rack supply manifold <NUM> and the first cooling loop assembly <NUM>. Similarly, the first outlet connector 842a is connected to the first outlet connection point 872a and second outlet connector 842b is connected to the second outlet connection point 872b on the first rack return manifold <NUM>. This allows for fluid communication between the first rack return manifold <NUM> and the first cooling loop assembly <NUM>.

At the other end of the computing device <NUM> the second cooling loop assembly <NUM> is connected to the second rack supply manifold <NUM> and second rack return manifold <NUM>. The second rack supply manifold <NUM> provides a redundant cooling liquid supply to the second cooling loop assembly <NUM>. As shown, the first inlet connector 940a of the second cooling loop assembly <NUM> is joined to the first connection points 970a on the second rack supply manifold <NUM> and the second inlet connector 940b of the second cooling loop assembly <NUM> is connected to the second connection point 970b. This provides for fluid communication between the redundant and second rack supply manifold <NUM> and the second cooling loop assembly <NUM>. Similarly, the first outlet connector 942a is connected to the first outlet connection point 972a and second outlet connector 942b is connected to the second outlet connection point 972b on the second rack return manifold <NUM>. This allows for fluid communication between the second rack return manifold <NUM> and the second cooling loop assembly <NUM>.

Each of the remaining computing device <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> includes respective first cooling loop assemblies <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> and redundant and second cooling loop assemblies <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> that include similar structure and operate in a manner identical to computing device <NUM>. For example, each of the first cooling loop assemblies <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> are connected to the first rack supply manifold <NUM> and the first rack return manifold <NUM> as described with regard to the first cooling loop <NUM> and are not further described in detail. Second cooling loops assemblies <NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> are also similarly attached to the second rack supply manifold <NUM> and second return manifold <NUM> as described with regard to the second cooling loop <NUM> and are not further described in detail. In other examples, the cooling loop assemblies used in the system may vary.

With reference to <FIG> a schematic redundant cooling distribution system <NUM> is shown that comprises computing data rack <NUM>, as well as at least one additional computing device rack <NUM>-<NUM>. In other examples, any number of computing device racks may be utilized in the system. The redundant system <NUM> includes a first central supply and return subsystem <NUM> and a redundant and second central supply and return subsystem <NUM>. The first subsystem <NUM> provides for circulation of cooling liquid into each of the computing device racks <NUM>,<NUM>-<NUM> and out of the computing device racks <NUM>,<NUM>-<NUM>. As shown, the subsystem <NUM> supplies cooling liquid to the first rack supply manifolds <NUM>,<NUM>-<NUM> and the first cooling loop assemblies and returns the heated return cooling liquid from the second cooling loop assemblies through the first rack return manifolds <NUM>,<NUM>-<NUM>. The first supply and first return manifolds are shown positioned on the left side of the computing device racks <NUM>,<NUM>-<NUM>. This second subsystem provides a redundant supply of cooling liquid through the redundant and second supply manifolds <NUM>,<NUM>-<NUM> to the second cooling loop assemblies of the computing device racks <NUM>,<NUM>-<NUM>. Similarly, heated return cooling liquid exiting the second cooling loop assemblies of the computing devices in the respective racks <NUM>,<NUM>-<NUM> flow through the second return manifolds <NUM>, <NUM>-<NUM>. The components of the second subsystem <NUM> may be independent of the first subsystem <NUM> to the extent that it includes its own set of system components and operates on a loop that is independent of the loop in the first subsystem <NUM>. With separate cooling liquid supplies in the system and separate returns, this system ensures that even if one of the components in the first subsystem <NUM> fails, such as the central distribution unit <NUM>, the redundant supply and return provided by the second subsystem <NUM> will ensure that the components within the computing devices are continually cooled. In a scenario where the central distribution unit <NUM> fails, the computing devices within the computing devices racks <NUM>,<NUM>-<NUM> would overheat without the presence of the subsystem <NUM>. In other examples, one or more components of the first and second subsystems <NUM>,<NUM> may be shared.

The operation of the first central supply and return subsystem <NUM> is identical to the examples previously described. A water source <NUM> may pump a cooling liquid, such as water, into cooling unit <NUM>. Cooling unit <NUM> can be any conventional cooling unit <NUM> capable of cooling water within the unit to a pre-set temperature. Water from cooling unit <NUM> will then be pumped into the central distribution unit <NUM>, which will further pump and distribute cooling liquid through the central cooling liquid inlet line <NUM> and into each of the first rack supply manifolds <NUM>, <NUM>-<NUM> and into each of the first and second connectors (not shown) of each computing device (not shown) in the respective computing device racks <NUM>, <NUM>-<NUM>. Heated return cooling liquid will exit the respective computing device racks <NUM>, <NUM>-<NUM> through the respective first rack return manifolds <NUM>, <NUM>-<NUM> and into the central cooling liquid outlet line <NUM> and back to the central distribution unit <NUM>. The CDU will then pump the heated return cooling liquid into the cooling unit <NUM> where the heated return cooling liquid will be cooled and then circulated back into the central distribution unit <NUM> and pumped back into the system. It is to be appreciated that the central supply and return sub-system <NUM> provides one method and components for providing, cooling, and distributing the cooling liquid into the central inlet and outlet lines, and numerous other examples may be used to accomplish the same function. For example, the lines may run underground, additional or fewer cooling components may be used.

The operation of the second central supply and return subsystem <NUM> is identical to the examples previously described. A water source <NUM> may pump a cooling liquid, such as water, into cooling unit <NUM>. Cooling unit <NUM> can be any conventional cooling unit <NUM> capable of cooling water within the unit to a pre-set temperature. Water from cooling unit <NUM> will then be pumped into the central distribution unit <NUM>, which will further pump and distribute cooling liquid through the central cooling liquid inlet line <NUM> and into each of the first rack supply manifolds <NUM>, <NUM>-<NUM> and into each of the first and second connectors (not shown) of each computing device (not shown) in the respective computing device racks <NUM>, <NUM>-<NUM>. Heated return cooling liquid will exit the respective computing device racks <NUM>, <NUM>-<NUM> through the respective first rack return manifolds <NUM>, <NUM>-<NUM> and into the central cooling liquid outlet line <NUM> and back to the central distribution unit <NUM>. The CDU will then pump the heated return cooling liquid into the cooling unit <NUM> where the heated return cooling liquid will be cooled and then circulated back into the central distribution unit <NUM> and pumped back into the system. It is to be appreciated that the central supply and return sub-system <NUM> provides one method and components for providing, cooling, and distributing the cooling liquid into the central inlet and outlet lines, and numerous other examples may be used to accomplish the same function. For example, the lines may run underground, additional or fewer cooling components may be used.

While the first and second subsystems <NUM>,<NUM> are identical in this example, they may differ in other examples or applications.

<FIG> is an assembly <NUM> that is a variation of the assembly <NUM> shown in <FIG>. The only difference between the two assemblies is that in the present assembly <NUM>, the first cooling loop assembly <NUM> only includes one inlet connector <NUM> and one outlet connector <NUM>. Similarly, the second cooling loop assembly <NUM> only includes one inlet connector <NUM> and one outlet connector <NUM>. Otherwise, the assembly <NUM> is identical to assembly <NUM>, such that the first and second cooling loop assemblies <NUM>,<NUM> are identical to the assemblies of <FIG> and operate in the same manner.

<FIG> shows an example system that incorporates the system <NUM> of <FIG> into a computing device rack system <NUM>. The system is otherwise similar to <FIG>, and includes computing devices <NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> on shelves <NUM> of the computing device rack <NUM>. On one side of the computing devices <NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> and shelves <NUM>, the first loop assemblies <NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> are connected to the first rack supply manifold <NUM> and the first rack return manifold <NUM>. On the other side of the computing devices <NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>, second loop assemblies <NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM>,<NUM>-<NUM> are connected to the second rack supply manifold <NUM> and the second rack return manifold <NUM>. The remaining components in the system are otherwise identical to the previous examples disclosed herein, including, for example, the configuration of the computing device rack system and the overall cooling distribution system respectively shown in <FIG>.

The redundant systems shown in <FIG> provide a configuration that allows for multiple supply and return connectors for each computing device rack. Less pressure drop can be achieved across the systems, as well as enable rack level coolant supply redundancy. Furthermore, it is to be appreciated that redundant systems can incorporate any loop assembly configurations disclosed herein, as well as variations thereof.

Numerous variations of the aforementioned examples discussed in <FIG> are within the scope of the disclosure. For example, any number of inlet connectors, outlet connectors, supply lines, return lines, secondary lines and manifolds may be utilized in various configurations relative to one or more cooling devices. The cooling devices may be further arranged in series, as opposed to being parallel to one another.

The rack supply and rack return manifolds may be provided on any side of the computing device rack, and any number of rack supply and rack return manifolds may be utilized. The number of rack supply and rack return manifolds in the system does not need to be equal, such that there may be more rack supply manifolds than rack return manifolds, and vice versa.

The computing devices in computing device racks in at least <FIG>, <FIG>, <FIG>, <FIG> and <FIG> are not limited to the example cooling loops <NUM> to <NUM>-<NUM>, <NUM> to <NUM>-<NUM>, <NUM> to <NUM>-<NUM>, <NUM> to <NUM>-<NUM> used in the respective computing devices. Instead, all of the supply and return lines, secondary lines, and board manifolds within the cooling loop assembly may utilize lines that may vary in length so as to reach and connect with the rack supply manifold and rack return manifold. This allows for one cooling loop assembly to connect with any number of rack supply and rack return manifolds in various configurations. For example, the supply and return lines of the cooling loop assemblies of <FIG>, <FIG>, <FIG>, <FIG>, <FIG> and <FIG> can be flexible, to allow for connection to the rack supply manifolds (<NUM>,<NUM>,<NUM>,<NUM>) and rack return manifolds (<NUM>,<NUM>,<NUM>,<NUM>) of respective <FIG> and <FIG>.

Conventional liquid cooling systems rely on a single inlet to the loop assembly and a single outlet from the loop assembly for thermal management of components on a single board of a computing device. That is, at the shelf level, in some examples, only single supply and return lines and connectors are used to supply cooling liquid to components within computing devices in the system. In some examples, creasing the number of inlets and inlet connectors at the supply lines at the shelf level can improve thermal management of the computing devices, as well as improve overall efficiency and cost of the entire system. For example, using multiple inlet and outlet in parallel, reduces the flow rate in each inlet and outlet connectors, therefore reducing the pressure drop through the couplings. According to aspects of the disclosure, in some examples, this makes it possible to use smaller connectors on the rack manifold, which makes it easier for rack manifold manufacturing, reliability, and serviceability. Moreover, with small connectors in the system, lower force is required to connect and disconnect the quick disconnect couplings.

<FIG> is a system chart comparing the pressure drop to flow rate of two systems. Curve A represents the flow rate to pressure for a system with only a single supply and return connections to a single cooling loop for each computing device on a shelf of a computing device rack. As shown, the higher the flow rate, the greater pressure that is required to move the fluid through the system. Curve B represents a system curve for a system that utilizes multiple inlet/outlet connections to a single loop for an individual circuit board of a computing device on a shelf such as the concepts and examples disclosed in <FIG>. As shown, increasing the number of connections and inlets at the shelf level reduces the pressure drop. For a given pump curve, an increase Δ in flow rate can therefore be achieved in a system with multiple inlet connections and outlet connections to a single loop, as compared to the flow rate of a system utilizing conventional single inlet and outlet connections to a single loop. Thus, using a system that utilizes multiple supply and return connections at the shelf level and to a single circuit board and loop can achieve increased flow rate at lower pressure for the overall system.

With reference to <FIG>, the Head-Pressure curve to flow rate curve is identical to <FIG>, except that a lower speed pump curve has also been added. This reflects that for the same flow rate achieved on Curve A (single inlet/outlet connections to a single loop for each board on a shelf in the system), the same flow rate is achieved on Curve B (multiple inlet/outlet connections to a single loop for each board on a shelf in the system) at a lower pump speed. With reference to the power-flow rate curve, this also means that less power is required to achieve the same flow rate. This amounts to a cost savings on cost to run the system, as well as wear and tear on the components utilizing the system. Further, smaller pumps can be utilized in the system design to save on overall cost.

According to another example, the present disclosure provides a system including a central supply line and a central return line connected to a fluid source of cooling liquid. A rack supply manifold is connected to the central supply line. A rack return manifold is connected to the central return line. A liquid cooling loop assembly may be coupled to at least one cooling device connected to a circuit board. Multiple inlet connectors may be connected to the rack supply manifold. Multiple outlet connectors may be connected to the rack return manifold. The multiple inlet connectors provide a passageway for the cooling liquid from the central supply line into the liquid cooling loop assembly for distribution to the at least one cooling device. The multiple outlet connectors provide a passageway to the rack return manifold for the cooling liquid exiting the at least one cooling device.

Claim 1:
A system (<NUM>) for thermal management of computing devices (<NUM>) comprises:
a central supply line (<NUM>) coupled to a fluid source of cooling liquid;
a central return line (<NUM>) coupled to the fluid source;
a rack supply manifold (<NUM>) coupled to the central supply line (<NUM>);
a rack return manifold (<NUM>) coupled to the central return line (<NUM>); and
a cooling loop assembly (<NUM>) coupled to the rack supply manifold (<NUM>) and the rack return manifold (<NUM>), the cooling loop assembly (<NUM>) coupled to at least one cooling device (130a, 130b, 130c, 130d) electrically connected to a circuit board (<NUM>), the cooling loop assembly (<NUM>) further comprising:
multiple board supply lines (144a, 144b);
multiple inlet connectors (140a, 140b) connected to the rack supply manifold (<NUM>); and
multiple outlet connectors (142a, 142b) connected to the rack return manifold (<NUM>);
the multiple inlet connectors (140a, 140b) providing a passageway for the cooling liquid from the central supply line (<NUM>) into the cooling loop assembly (<NUM>) for distribution to the at least one cooling device (130a, 130b, 130c, 130d), and
the multiple outlet connectors (142a, 142b) providing a passageway to the rack return manifold (<NUM>) for the cooling liquid exiting the at least one cooling device (130a, 130b, 130c, 130d),
wherein each of the multiple inlet connectors (140a, 140b) is coupled to a corresponding one of the multiple board supply lines (144a, 144b) that feeds fluid from the rack supply manifold (<NUM>) into the cooling loop assembly (<NUM>), and
wherein the cooling loop assembly (<NUM>) comprises a board inlet manifold (<NUM>) and secondary supply lines (146a, 146b, 146c, 146d), each board supply line (144a, 144b) connecting a corresponding one of the multiple inlet connectors (140a, 140b) to the board inlet manifold (<NUM>), and the secondary supply lines (146a, 146b, 146c, 146d) connecting the board inlet manifold (<NUM>) to the at least one cooling device (130a, 130b, 130c, 130d).