Liquid cooling apparatus and method for facilitating cooling of an electronics system

Apparatus and method are provided for facilitating liquid cooling one or more components of an electronic subsystem chassis disposed within an electronics rack. The apparatus includes a rack-level coolant manifold assembly and at least one movable chassis-level manifold subassembly. The rack-level coolant manifold assembly includes a rack-level inlet manifold and a rack-level outlet manifold, and each movable chassis-level manifold subassembly includes a chassis-level coolant inlet manifold coupled in fluid communication with the rack-level inlet manifold, and a chassis-level coolant outlet manifold coupled in fluid communication with the rack-level outlet manifold. The chassis-level manifold subassembly is slidably coupled to the electronics rack to facilitate access to one or more removable components of the electronic subsystem chassis. In one embodiment, the electronics subsystem chassis is a multi-blade center system having multiple removable blades, each blade being an electronics subsystem.

TECHNICAL FIELD

The present invention relates to apparatuses and methods for facilitating cooling of an electronics system, such as a multi-blade center system, and more particularly, to apparatuses and methods for facilitating liquid-cooling of selected electronic components of an electronics system without impacting serviceability of the electronics system.

BACKGROUND OF THE INVENTION

The power dissipation of integrated circuit chips, and the modules containing the chips, continues to increase in order to achieve increases in processor performance. This trend poses a cooling challenge at both the module and system level. Increased air flow rates are needed to effectively cool high power modules and to limit the temperature of the air that is exhausted into the computer center.

In many server applications, processors along with their associated electronics (e.g., memory, disc drives, power supplies, etc.) are packaged in removable drawer or blade configurations disposed within a housing. Typically, the components are cooled by air moving in parallel air flow paths, usually front-to-back, impelled by one or more air-moving devices (e.g., fans or blowers). In some cases, it may be possible to handle increased power dissipation within a single drawer or blade chassis by providing greater air flow, through the use of a more powerful air-moving device, or by increasing the rotational speed (i.e., RPMs) of an existing air-moving device. However, this approach is becoming problematic at the system level.

The sensible heat load carried by air exiting the electronics rack is stressing the ability of room air-conditioning to effectively handle the load. This is especially true for large installations with “server farms” or large banks of computer racks close together. In such installations, liquid cooling of selected components is an attractive technology to manage the higher heat fluxes. The liquid coolant absorbs the heat dissipated by selected components/modules in an efficient manner. Typically, the absorbed heat is ultimately transferred from the liquid to an outside environment, whether air or liquid-cooled.

SUMMARY OF THE INVENTION

The shortcomings of the prior art are overcome and additional advantages are provided through the provision of a liquid cooling apparatus for an electronics rack comprising at least one electronic subsystem chassis. The liquid cooling apparatus includes: a rack-level coolant manifold assembly and at least one movable chassis-level manifold subassembly. The rack-level coolant manifold assembly includes a rack-level inlet manifold and a rack-level outlet manifold, and is configured to mount to the electronics rack. The at least one movable chassis-level manifold subassembly is configured to couple to the electronics rack, adjacent to at least one electronic subsystem chassis, and pass liquid coolant to one or more components of the at least one electronic subsystem chassis for facilitating cooling of the one or more components thereof. The at least one chassis-level manifold subassembly includes a chassis-level coolant inlet manifold and a chassis-level coolant outlet manifold. The chassis-level coolant inlet manifold is coupled in fluid communication with the rack-level inlet manifold and the chassis-level coolant outlet manifold is coupled in fluid communication with the rack-level outlet manifold. The at least one movable chassis-level manifold subassembly, when coupled to the electronics rack, is slidable relative to the electronics rack to facilitate access to a removable component of the at least one electronic subsystem chassis, wherein the at least one electronic subsystem chassis includes at least one removable component.

In another aspect, a cooled electronics rack is provided. The cooled electronics rack includes an electronics system and a liquid cooling apparatus for facilitating cooling of one or more components of at least one electronics subsystem of the electronics system. The electronics system includes the at least one electronics subsystem to be cooled, an electronics rack, and at least one air-moving device. The electronics rack at least partially surrounds and supports the at least one electronics subsystem, and includes an air inlet side and an air outlet side. The air inlet and air outlet sides respectively enable ingress and egress of air through the electronics rack, including through the at least one electronics subsystem to be cooled. The at least one air-moving device causes air to flow from the air inlet side of the electronics rack, across the at least one electronics subsystem to the air outlet side of the electronics rack. The liquid cooling apparatus includes a rack-level coolant manifold assembly and at least one movable chassis-level manifold subassembly. The rack-level coolant manifold assembly, which is mounted to the electronics rack, includes a rack-level inlet manifold and a rack-level outlet manifold. The at least one movable chassis-level manifold subassembly is movably coupled to the electronics rack adjacent to the at least one electronics subsystem to be cooled to pass liquid coolant to the one or more components of the at least one electronic subsystem for facilitating cooling of the one or more components thereof. The at least one movable chassis-level manifold subassembly includes a chassis-level coolant inlet manifold and a chassis-level coolant outlet manifold. The chassis-level coolant inlet manifold is coupled in fluid communication with the rack-level inlet manifold, and the chassis-level coolant outlet manifold is coupled in fluid communication with the rack-level outlet manifold. The at least one movable chassis-level manifold subassembly is slidable relative to the electronics rack to facilitate access to a removable component of the at least one electronics subsystem, wherein each electronics subsystem of the at least one electronics subsystem includes at least one removable component.

In a further aspect, a method is provided for facilitating cooling of an electronics rack comprising at least one electronic subsystem chassis. The method includes: providing a rack-level coolant manifold assembly including a rack-level inlet manifold and a rack-level outlet manifold, the rack-level coolant manifold assembly being configured to mount to the electronics rack; providing at least one movable chassis-level manifold assembly configured to couple to the electronics rack adjacent to the at least one electronic subsystem chassis and pass liquid coolant to one or more components of the at least one electronic subsystem chassis for facilitating cooling of at least one component thereof, wherein the at least one movable chassis-level manifold subassembly includes a chassis-level coolant inlet manifold and a chassis-level coolant outlet manifold; mounting the rack-level coolant manifold assembly to the electronics rack, and coupling the at least one movable chassis-level manifold subassembly to the electronics rack adjacent to the at least one electronic subsystem chassis; and coupling in fluid communication the chassis-level coolant inlet manifold and the rack-level inlet manifold, and coupling in fluid communication the chassis-level coolant outlet manifold and the rack-level outlet manifold, wherein the at least one movable chassis-level manifold subassembly is slidably adjustable relative to the electronics rack to facilitate access to a removable component of the at least one electronic subsystem chassis, wherein the at least one electronic subsystem chassis comprises at least one removable component.

DETAILED DESCRIPTION OF THE INVENTION

As used herein, the term “electronics system”, includes any housing, frame, rack, compartment, blade server system, etc., having one or more heat generating components of a computer system, and may be, for example, a stand alone computer processor having high, mid or low end processing capability. In one embodiment, an electronics system may comprise multiple electronics subsystems, each having one or more heat generating components disposed therein requiring cooling. “Electronics subsystem” refers to any blade, book, node, etc., having one or more heat generating electronic components. Electronics subsystems of an electronics system may be movable or fixed relative to the electronics system, with the blades of a blade center system being one example of subsystems of an electronics system to be cooled. In another example, the electronics system may comprise an electronics rack having one or more multi-blade center systems disposed therein, with each multi-blade center system being an example of an electronic subsystem chassis containing a plurality of electronics subsystems (e.g., removable blades) having one or more components to be cooled. As used herein, “electronic subsystem chassis” refers to any sub-housing, drawer, compartment, etc., containing one or more electronics subsystems of an electronics system, such as an electronics rack.

Reference is made below to the drawings, which are not drawn to scale and are simplified for ease of understanding, and wherein the same reference numbers used throughout different figures designate the same or similar components.

FIG. 1Adepicts one embodiment of an electronics rack100comprising a stack of multi-blade center systems110, as well as supporting power supplies, networking equipment, etc.

FIG. 1Billustrates one embodiment of a multi-blade center system110, one example of which is marketed by International Business Machines Corporation, of Armonk, N.Y. By way of specific example, multi-blade center system110may comprise a stand alone server system which incorporates scalable computing functionality up to, for example, fourteen high performance servers (or blades).

As shown inFIG. 1B, multi-blade center system110includes an electronics subsystem chassis120and multiple removable blades130. As one example, each removable blade130is an electronics subsystem, such as a server of a multi-server electronics system. A first flange131and second flange132with openings133are also illustrated. Typically, flanges131,132are used to secure the multi-blade center system within an electronics rack, such as depicted inFIG. 1A. Airflow135is through an air inlet side136of multi-blade center system110to an air outlet side137, and is established, for example, by two or more air-moving devices (not shown) disposed at the back portion of the system housing. Electrical and networking infrastructure is also located near the back of electronics subsystem chassis120.

FIGS. 1C & 1Ddepict one embodiment of a removable blade130of the electronic subsystem chassis. As illustrated inFIG. 1D, removable blade130includes, for example, multiple processors above which conventionally reside respective air-cooled heat sinks140. In this example, each removable blade is a complete computer system, or subsystem, and includes, for example, Direct Access Storage Device (DASD)141and Dual In-Line Memory Modules (DIMMs)142. Electrical connectors143are provided for electrically connecting blade130to the respective electronic subsystem chassis120(FIG. 1B). Corresponding electrical connectors are disposed within the electronic subsystem chassis near the back thereof for making electrical connection to connectors143when the blade is inserted into the chassis in operational position.

By way of specific example, a typical blade center chassis today is 9 U tall, and houses14field-replaceable blades, each containing two central processing units (CPUs). A standard electronics rack that is 42 U tall can thus accommodate four such blade center chassises (each 9 U tall), for a total of 56 blades and 112 CPU modules. International Business Machines Corporation markets three versions of a blade center chassis, namely, the BC, BCH and BC-Telco versions.FIGS. 1A-1Dillustrate one example of a BCH chassis marketed by International Business Machines Corporation, however, the concepts presented herein are readily applicable to any blade center chassis configuration, as well as to other electronic subsystem housing variants. Further, the liquid cooling apparatus described herein is readily adaptable to use with any housing version with a removable component configuration.

Advantageously, liquid cooling of an electronics rack such as depicted inFIG. 1Aprovides increased cooling at the module and rack level, and enables higher performance systems than currently feasible using air-cooling alone. Further, a liquid cooling apparatus, such as described below, improves energy efficiency by eliminating or reducing requirements of one or more data center air-conditioning units; that is, by rejecting heat to the liquid coolant, which in one example, is subsequently rejected to the ambient environment outside of the data center. With a hybrid liquid-air cooling approach such as described herein, the power consumption of the air moving devices within the electronics rack may also be reduced, further reducing acoustic noise within the data center. Additionally, a reduced form factor of the processor's thermal solution is provided, thus allowing more functionality to be packaged within a single subsystem or blade. This added functionality could be memory, hard drives, or other devices, which would allow for a more competitive offering within the market place. Also, in the blade configuration ofFIG. 1D, there is a reduction in pre-heating of air flowing over the memory DIMMs by liquid cooling the processors. This pre-heating of the air is currently a significant problem, with liquid-cooling of the processors substantially alleviating the issue.

FIG. 2is a schematic of one embodiment of a cooled electronics rack, generally denoted200, in accordance with an aspect of the present invention. Cooled electronics rack200includes an electronics rack100having a plurality of electronic subsystem chassises110stacked within the rack. In this example, four electronic subsystem chassises110are illustrated, with each electronic subsystem chassis being (in one example), a multi-blade center system such as depicted inFIGS. 1B-1D. As illustrated inFIG. 2, each electronic subsystem chassis includes a back plane201, into which the respective removable blades are electrically inserted, and one or more air-moving devices205located near the back of each electronic subsystem chassis. These air-moving devices205cause air flow202to flow from an air inlet side111to an air outset side112of electronics rack110through the multiple electronic subsystem chassises110.

One embodiment of the liquid cooling apparatus, generally denoted210, is illustrated inFIG. 2. In this embodiment, a liquid cooling unit220is disposed in a lower portion of electronics rack100. Liquid cooling unit220includes, for example, a liquid-to-liquid heat exchanger (not shown) for extracting heat from coolant flowing through a primary coolant loop230of liquid cooling apparatus210and dissipating the heat within a facility coolant loop219comprising a facility coolant supply line221and facility coolant return line222. In one example, facility coolant supply line221and facility coolant return line222couple liquid cooling unit220to a data center facility coolant supply and return (not shown). Liquid cooling unit220further includes an appropriately sized reservoir, pump, and optional filter, for moving liquid coolant through primary coolant loop230. In one embodiment, primary coolant loop230includes a rack-level inlet manifold231and a rack-level outlet manifold232, which are coupled to liquid cooling unit220via, for example, flexible hoses and respective quick disconnect couplings225,226. The flexible hoses allow the rack-level manifolds to be mounted within, for example, a door of the electronics rack hingedly mounted to the air inlet side of the electronics rack in a manner similar to that described in co-pending, commonly assigned U.S. patent application Ser. No. 11/763,678, filed Jun. 15, 2007, entitled “Liquid-Based Cooling Apparatus for an Electronics Rack”, the entirety of which is hereby incorporated herein by reference. In one example, rack-level inlet manifold231and rack-level outlet manifold232each comprise an elongate, rigid tube vertically mounted to an electronics rack100.

In the embodiment illustrated, the rack-level coolant manifold assembly, comprising rack-level inlet manifold231and rack-level outlet manifold232is in fluid communication with multiple movable chassis-level manifold subassemblies240. As illustrated, each movable chassis-level manifold subassembly is coupled to the electronics rack to reciprocate vertically (as indicated by arrows211) adjacent to an associated electronic subsystem chassis110. In the embodiment illustrated, respective quick disconnect couplings235,236are employed to coupled the rack-level inlet manifold and rack-level outlet manifold to each movable chassis-level manifold subassembly240, using for example appropriately sized, flexible rack-level tubing.

Each movable chassis-level manifold subassembly240includes a chassis-level coolant inlet manifold241and a chassis-level coolant outlet manifold242, disposed, in this example, within a common structure, but isolated to prevent the direct flow of coolant therebetween. The chassis-level coolant inlet manifold241of each movable chassis-level manifold subassembly240is coupled via flexible tubing243to a first cold plate250of two series-coupled cold plates250,251within each removable blade of an associated electronic subsystem chassis110. Further, flexible tubing244couples each liquid-cooled cold plate251to the chassis-level coolant outlet manifold242of the respective movable chassis-level manifold subassembly240. These flexible tubes243,244are sized and provided with sufficient flexibility to allow the associated movable chassis-level manifold subassembly240to reciprocate within a designed extent of travel, as illustrated by arrows211. Each chassis-level flexible tubing243,244couples to the respective chassis-level coolant inlet manifold241, or chassis-level coolant outlet manifold242via an appropriately sized quick disconnect coupling246,247. In one implementation example, the previously air-cooled heat sinks of the blade referenced inFIG. 1Dare removed and replaced with corresponding liquid-cooled cold plates250,251through with liquid coolant passes in-series, as illustrated inFIG. 2.

As noted, in one example, the rack-level inlet manifold231and rack-level outlet manifold232comprise rigid structures, while the remaining tubing coupling the rack-level inlet and outlet manifolds to liquid cooling unit220, and to movable chassis-level manifold subassemblies240are flexible tubing, as is the tubing coupling movable chassis-level manifold subassembly240to the respective sets of series-coupled cold plates. Note that two series-coupled cold plates are illustrated and described herein by way of example only. The liquid cooling apparatus could readily be applied to use with a single liquid-cooled cold plate, or to an electronic subsystem comprising multiple liquid-cooled cold plates, either coupled in parallel or in series fluid communication with the associated movable chassis-level manifold subassembly.

FIG. 3is a partial representation of one embodiment of a movable chassis-level manifold subassembly240shown coupled in fluid communication to multiple sets of two series-connected cold plates250,251, which as noted above, are assumed (in one embodiment) to be coupled to respective heat-generating components (such as microprocessors) disposed within respective blades of a multi-blade center system such as depicted inFIGS. 1B & 2.

In this embodiment, removable chassis-level manifold subassembly240includes two separate manifolds, that is, the chassis-level coolant inlet manifold and the chassis-level coolant outlet manifold. Inlet to the chassis-level coolant inlet manifold is via a respective hose connection300, and outlet of coolant from the chassis-level coolant outlet manifold is via a hose connection301, as discussed above in connection withFIG. 2. Similarly, a plurality of quick disconnect couplings246facilitate coupling the chassis-level coolant inlet manifold to first cold plates250of the respective pairs of series-connected cold plates250,251(disposed within the blades of the multi-blade center system), and a plurality of quick disconnect couplings247facilitate coupling the chassis-level coolant outlet manifold to each second cold plate251of the pairs of series-connected cold plates250,251for facilitating return of liquid coolant from the cold plates to the manifold. Flexible tubes or hoses243,244couple movable chassis-level manifold subassembly240to the respective sets of cold plates. As noted, these flexible chassis-level tubes243,244are of sufficient length to allow for movement of chassis-level manifold subassembly240as illustrated inFIGS. 4A-5B. Appropriate tubing252couples the first cold plate and second cold plate of each pair of series-connected cold plates in fluid communication.

FIGS. 4A & 4Billustrate a partial embodiment of a cooled electronics rack200, in accordance with an aspect of the present invention. In this embodiment, a single electronic subsystem chassis110is illustrated, with the chassis including a plurality of removable, vertically-oriented blades130, as well as multiple horizontally-oriented power supply units135, which are also selectively removable. The liquid cooling apparatus includes a rack-level inlet manifold231and a rack-level outlet manifold232, which in one embodiment are rigid structures mounted to the electronics rack. Further illustrated inFIGS. 4A & 4Bis a movable chassis-level manifold subassembly240, one embodiment of which is described above in connection withFIGS. 2 & 3. Movable chassis-level manifold subassembly240again couples to one or more cold plates disposed within the individual removable blades130of the respective electronics subsystem chassis (for example, multi-blade center system) disposed within the cooled electronics rack. The movable chassis-level manifold subassembly240distributes liquid coolant to the individual blades of the associated chassis to facilitate liquid cooling of one or more components of the individual blades as explained above. In the example described above in connection withFIGS. 2 & 3, two series-connected cold plates are illustrated coupled and providing liquid cooling to two processors disposed within each blade. The movable chassis-level manifold subassembly240couples to the individual removable blades via flexible tubing243,244. The plurality of quick disconnect couplings246are shown in this figure, with the corresponding plurality of quick disconnect couplings247(seeFIG. 3) being disposed behind couplings246.

In one example, movable chassis-level manifold subassembly240couples to the electronics rack via a chassis-level manifold support400mounted directly to the respective electronic subsystem chassis, for example, through openings133in first and second flanges131,132of the electronics subsystem chassis120illustrated inFIG. 1B. Chassis-level manifold support400includes, in one embodiment, cylindrical rods401, upon which movable chassis-level manifold subassembly240slidably adjusts. A spring-loaded latch mechanism, such as a push pin410, is provided for engaging respective aligned holes411in rods401to allow for a level, fixed positioning of the movable chassis-level manifold subassembly in any one of two or more positions, with five aligned sets of holes411being shown.

In a first position, illustrated inFIG. 4A, the movable chassis-level manifold subassembly is positioned in a location below removable blades130of the multi-blade center system110, for example, to allow servicing or removal of one or more blades of the multi-blade center system. Removal of a blade is accomplished by disconnecting the appropriate flexible tubing243,244using the quick disconnects246,247, and sliding the blade out horizontally from the electronics rack.

In a second position, illustrated inFIG. 4B, movable chassis-level manifold subassembly240has been raised with respect to the associated electronic subsystem chassis110to allow, for example, access to the lower power supply units135disposed within the chassis. In this embodiment, in either position illustrated inFIG. 4Aor4B, power supply unit135in the upper portion of the chassis is readily accessible for servicing or removal.

FIGS. 5A & 5Bdepict an alternate embodiment of a cooled electronics rack200′ employing stacked electronic subsystem chassises110′. In this embodiment, each electronic subsystem chassis comprises (as one example) a multi-blade center system, including a plurality of vertically-oriented removable blades each, for example, housing a respective electronic subsystem with one or more processors to be liquid-cooled using the liquid cooling apparatus described herein. In this electronic subsystem chassis embodiment, the removable power supply units illustrated above in connection withFIGS. 1B,2&4are omitted. Therefore, the movable chassis-level manifold subassembly240and its associated chassis-level manifold support400′ are configured to allow the movable chassis-level manifold subassembly to reside in at least two positions, one of which is illustrated inFIG. 5Aover the associated electronic subsystem chassis, and the other of which is illustrated inFIG. 5B, over an adjacent electronic subsystem chassis, which allows for the removal of one or more selected blades of the associated electronic subsystem chassis, as described above in connection with the embodiment ofFIGS. 4A & 4B.

FIGS. 6A-6Cillustrate an alternate embodiment of a chassis-level manifold support, generally denoted600, in accordance with an aspect of the present invention. In this embodiment, chassis-level manifold support600is sized and configured to mount to a respective electronic subsystem chassis120, which may comprise a multi-blade center system, such as described above. As shown inFIG. 6A, the removable blades130of the multi-blade center system are oriented vertically within the electronic subsystem chassis120, with only two blades being shown by way of example. The chassis-level manifold support600includes a left support601and a right support602, each of which is mounted to a respective side of the electronic subsystem chassis. Left support601and right support602are configured to constrain movement of movable chassis-level manifold subassembly240to a vertical direction relative to the respective electronic subsystem chassis120. In this embodiment, spring-loaded handles603,604are mounted to the movable subassembly and provided in association with left support601and right support602to allow for the selective latching of each side of movable chassis-level manifold subassembly240to a selected pair of aligned pin holes605in left support601, and pin holes606in right support602. In operation, a technician applies force to each side handle603,604to compress these springs, and adjusts the movable chassis-level manifold subassembly240to the desired height relative to the left support601and right support602. The technician then relaxes the applied force, and the springs coupled to the handles latch the movable chassis-level manifold subassembly into the selected pin hole level605,606on the left and right supports601,602. In this embodiment, the movable chassis-level manifold subassembly comprises a first plurality of quick disconnect couplings246and a second plurality of quick disconnect couplings247, which couple flexible tubing243,244to one or more components of each removable blade to be liquid-cooled, as described above. In addition, hose connections300,301couple the chassis-level coolant inlet manifold and the chassis-level coolant outlet manifold to the rack-level inlet manifold and the rack-level outlet manifold, respectively (such as illustrated above in connection withFIG. 2).

As illustrated inFIGS. 6B & 6C, chassis-level manifold extensions610are provided at respective ends of the movable chassis-level manifold subassembly240to facilitate coupling the spring-biased handles602,603to the movable chassis-level manifold subassembly. Alternatively, the manifold subassembly240could be configured at its ends to include an appropriately designed flange sized and positioned to allow for bolting of the respective spring-biased handle to the ends of the manifold subassembly.