Method and apparatus for cooling an equipment enclosure through closed-loop, liquid-assisted air cooling in combination with direct liquid cooling

A method and an apparatus for cooling, preferably within an enclosure, a diversity of heat-generating components, with at least some of the components having high-power densities and others having low-power densities. Heat generated by the essentially relatively few high-power-density components, such as microprocessor chips for example, is removed by direct liquid cooling, whereas heat generated by the more numerous low-power or low-watt-density components, such as memory chips for example, is removed by liquid-assisted air cooling in the form of a closed loop comprising a plurality of heating and cooling zones that alternate along the air path.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the cooling of enclosures, such as racks, for diverse types of equipment, such as heat-producing electronics, through a combination of air and liquid cooling for very high total power levels of the equipment.

For instance, as heat is generated during operation of electronic equipment, such as that comprising an integrated-circuit chip (IC), the thermal resistance between chip junction and the medium employed for the removal of heat must be sufficiently small in order to provide a junction temperature that is low enough to insure the continued reliable operation of the equipment. However, the problem of adequate heat removal becomes ever more difficult to solve as chip geometry is scaled down and operating speeds of the electronic equipment are increased, resulting in an increased power density (W/cm2) at the surface of the chip. The problem is further exacerbated when different types of chips in close proximity with each other possess different cooling requirements. For example, in a computer system, a processor chip may have a much higher power density than closely located memory chips. Furthermore, as another example, different types of chips may have different maximum-allowable junction temperatures. Such cooling requirements impose mechanical and thermal packaging challenges to the equipment design that can limit the performance thereof. In the current technology, the power density of processor and other kinds of high-performance chips is rapidly approaching levels that exceed the capability of forced-air cooling, necessitating the use of liquid cooling for some applications and installations in order to be able to attain the requisite degree of cooling for the equipment.

The cooling of computer racks and other types of electronic equipment is typically accomplished by forced-air cooling; however, liquid-assisted air cooling and direct-liquid cooling, frequently with water as the cooling medium, have also been widely employed. This concept is discussed in Richard C. Chu, et al., “Review of Cooling Technologies for Computer Products”,IEEE Transactions on Device and Materials Reliability, Vol. 4, No. 4, pp. 568-585, (December 2004). In liquid-assisted air cooling, a liquid-cooled heat exchanger is placed in a heated air stream in order to extract heat and reduce the air temperature before it is expelled into the room. Chu, et al. (Supra) also describe the problems encountered with data-center thermal management, in which the power dissipated for each equipment rack is approaching 30 kW. In a typical modern data center, water-cooled air-conditioning units or other external cooling devices are used to provide, through perforations in a raised floor or through ducts, a stream of chilled air to the computers, in which the air is heated, and downstream of which the air is returned to air-conditioning units so as to be chilled again. Significant problems encountered with this approach include the need for the large circulatory volume of air required to adequately cool the electronics, the extensive raised-floor space required to handle this air volume, the accompanying high acoustic noise levels encountered in the room, and the difficulty of controlling the airflow in the room to prevent already-hot air from re-circulating into the electronics, thereby potentially leading to overheating and electronic failure of the equipment. Moreover, the computer machine room can be uncomfortable for human occupancy because of large temperature differences present between room areas cooled by the cold inlet air and room areas heated by the hot outlet air. It is noted hereby that traditional data-center cooling is basically quite similar to liquid-assisted air cooling in that, in both instances, heat is initially transferred from the electronics to air. The difference resides in the location of the subsequent heat transfer from air to liquid: in traditional data-center cooling, this air-to-liquid heat transfer occurs outside the computer racks, typically in air-conditioning units where the liquid is water, whereas in liquid-assisted air cooling, it occurs within the computer racks.

2. Discussion of the Prior Art

Various methods and apparatus have been developed in the technology for the purpose of imparting adequate cooling to diverse types of operating equipment, such as electronic devices functioning at high power levels and which generate considerable amounts of heat, which must be dissipated.

Chu, et al., U.S. Pat. No. 6,819,563 B1, which is commonly assigned to the present assignee, and the disclosure of which is incorporated herein by reference, discloses a method and system for augmenting the air cooling of rack-mounted electronics systems by using a cooling fluid to cool air entering the system, and to remove a portion of the heat dissipated by the electronics being cooled. A drawback in the adding of heat exchangers to an electronic rack is due to an increased flow resistance that reduces airflow through the rack. In the patent, the air path is an open loop, whereas the present invention is directed to the provision of a closed-loop air path inside the rack.

Chu, et al., U.S. Pat. No. 6,775,137 B2, which is commonly assigned to the present assignee, and the disclosure of which is incorporated herein by reference, relates to an enclosure apparatus that provides for a combined air and liquid cooling of rack-mounted, stacked electronic components. A beat exchanger is mounted on the side of the stacked electronic components, and air flows from the front to the back within the enclosure, impelled by air-moving devices mounted behind the electronic components. A drawback in adding a heat exchanger to the side of an electronics rack is the requirement for an increase in floor space. Moreover, a front-to-back airflow within the confines of the rack does not allow for the use of a continuous midplane for the electronic components.

Patel, et al, U.S. Pat. No. 6,628,520 describes an apparatus for housing electronic components that includes an enclosure, mounting boards with electronic components mounted thereon, a supply plenum for cooling air, one or more outlets, which are directed toward the mounting boards, one or more heat-exchanging devices, and one or more blowers. A significant limitation in this arrangement resides in that inlet and outlet plenums for the air are needed along opposite sides of the electronics rack, in addition to the space required at the top and bottom of the rack, which is used to reverse the direction of the air flow.

Ishimine, et al., U.S. Pat. No. 6,621,707 pertains to an electronic apparatus comprising a motherboard, multi-chip modules mounted to the motherboard, cooling members for cooling the multi-chip modules, a refrigeration unit for cooling the cooling members to room temperature or lower, and a connection structure to releasably couple each multi-chip module thermally and mechanically to the refrigeration unit. In contrast therewith, the present invention does not use a refrigeration unit, or require a substantially hermetically sealed box structure, or a drying means for supplying dry air for cooling purposes.

Sharp, et al, U.S. Pat. Nos. 6,506,111 B2 and 6,652,373 B2 each describe a rack with a closed-loop airflow and a heat exchanger. The air flows vertically up one side of the rack, horizontally across electronics devices, vertically down the other side of the rack, and then across a heat exchanger located in the base, or optionally on the tops, of the rack. Because the air path is much shorter for memory cards near the heat exchanger location, a perforated plate is included in one of the vertical paths to enable adjustment of the airflow across the various memory cards to match some desired, e.g., constant distribution. The present invention does not require the vertical plenums, which occupy valuable space, or the perforated plate.

Parish IV, et al., U.S. Pat. No. 6,462,949 B1 discloses a cooling apparatus using “low-profile extrusions” to cool electronic components mounted on a board or card, whereas contrastingly, the present invention does not use any “low-profile extrusions” or similar structure.

Miller, et al., U.S. Pat. No. 6,305,180 B1 discloses a system for cooling electronic equipment using a chiller unit between adjacent racks for returning cooled air to ambient Contrastingly, the present invention is distinct from the foregoing because in the system described therein, the air is re-circulated within the rack rather than being expelled to the ambient environment.

Go, et al., U.S. Pat. No. 5,144,531 is directed to a liquid cooling system comprising cold plates attached to their respective circuit modules, quick couplers for connecting flexible hoses to these cold plates, a supply duct, and a return duct to form strings of cold plates, which are connected between the supply duct and the return duct. Valved quick couplers are used for the connection to the supply duct and the return duct, and valveless quick couplers are used for the connection to the cold plates. In contrast, pursuant to the present invention, a quick connect is not used to connect to the cold plate, though quick connects may be employed for the connections to each individual blade.

Koltuniak, et al., U.S. Pat. No. 3,749,981 describes a modular power supply wherein the power modules, each with its own fans, are mounted inside a sealed cabinet. Also mounted inside the cabinet are cooling modules, each with its own fan and heat exchanger. This patent represents an early example in the technology of an air-recirculation system requiring shared airflow plenums that occupy valuable space.

Ward, et al., U.S. Pat. No. 3,387,648 pertains to a cabinet-enclosed cooling system for electronic modules mounted on a modular chassis, wherein the chassis is extensible from the cabinet. This is an example of an air-recirculation system that requires, at the front and back of the assembly, shared vertical air plenums which unnecessarily occupy valuable space.

In implementing the construction of high-performance computer systems, it is desirable to be able to electrically interconnect as many processor chips and memory cards as possible while using conventional and economically priced electronic packaging methods. Thereby, the more densely and closely packed the electronics are, the more difficult they are to cool, because space is required for air circulation and for heat sinks. One method of achieving dense packaging of the electronic components is to build modular units called “blades”, each of which contains one or more processors and memory card(s). Multiple blades are then plugged into a common electrical backplane, or midplane, which, because of its high wiring density, provides for a high-speed and cost-effective inter-blade communication. Moreover, the modularity of blades allows for the sharing of common system resources, and facilitates servicing and configuration changes. Blade-type packaging is not limited to computer systems, but may also be employed for switch systems, or other types of information processing, and for matching and/or mixing of different functions within a single rack or enclosure.

Two features of conventional blade-style packaging essentially limit the performance achievable by the electronic components located within a rack:

Racks with blade-style packaging frequently employ vertical backplanes (or midplanes) in conjunction with front-to-back airflow cooling arrangements, thereby requiring airflow holes to be formed in the backplane. Such holes, to a significant extent, block wiring channels in the backplane, thereby greatly reducing the number of I/O's (input/output electrical signaling interconnections) available for connection to the attached blades. Moreover, in such a rack, the relatively small airflow cross-section provided by the holes in the backplane limits total power dissipation to about 30 kW. This aspect is disclosed in the publication by M. J. Crippen, et al., “BladeCenter packaging, power, and cooling”,IBM J. Res. &Dev., Vol. 49 No. 6, November 2005, pp. 887-904.

2. Total Reliance on Air-Cooling

As a cooling fluid, air is advantageous vis-à-vis water because it effortlessly bathes myriad heat-producing electronic devices in a safe, insulating cooling fluid. However, air is disadvantageous in comparison with water because its small heat capacity per unit volume, 3500 times smaller than water, limits the power density that may be cooled, and requires a considerable amount of airflow space, which restricts packaging density.

The above-mentioned features of conventional blade-style packaging, front-to-back airflow and total reliance on air cooling, must be clearly improved upon in order to solve the following problems, which are currently in evidence:

(a) limited total power that can be dissipated in a blade-style rack,

(b) limited packaging density due to space required for airflow,

(b) high engineering cost of customized airflow solutions for conventional raised-floor data centers,

(c) excessive data-center noise encountered due to air movers and airflow, and

(d) discomfort encountered by personnel in data centers due to non-uniform air temperatures.

SUMMARY OF THE INVENTION

The current invention implements two ideas in a unique and novel manner: first, the use of vertical airflow with vertical backplanes or midplanes; and second, the combined use of both air and water as coolants, in an arrangement that exploits the strengths of both fluids.

As a result, whereas conventional, air-cooled, blade-style packaging limits total power to 30 kW in a noisy rack occupying 2′×3′ of floor space (5 kW/ft2), a prototype embodiment of the present invention, with a realistic mix of heat-producing components, will successfully cool a total power of 81 kW in a quiet rack occupying 2.7′×5′ of floor space (6 kW/ft2). Moreover, the prototype embodiment indicates that future embodiments could readily cool over 100 kW in such a rack (>7.4 kW/ft2).

Accordingly, the present invention provides for a method and an apparatus for cooling, preferably within an enclosure, a diversity of heat-generating components, with at least some of the components having high power densities and others having low power densities. Direct liquid cooling is used to remove heat generated by a relatively small number of high-power-density components exemplified by microprocessor chips, whereas novel, closed-loop, liquid-assisted air cooling is used to remove heat generated by a relatively large number of low-watt-density components exemplified by memory chips.

In effectuating direct liquid cooling, microchannel coolers or other types of cold heads are attached directly to the high-power-density components. In effectuating closed-loop liquid-assisted air cooling, air travels upwardly in the front half of an enclosure through relatively narrow rectangular packages, referred to as “blades”, which contain diverse heat-generating components, and which are positioned in multiple rows located one above the other. Air-to-liquid heat exchangers are interleaved between rows of blades in order to cool the air emerging from each respective blade row before entering the next row. The heat that the air removes from the blade row is transferred in its entirety to the liquid, and is thereby removed from the enclosure, with the air being thereby assisted in its cooling task by means of the liquid. The blades are ordinarily attached to the front side of one or more central, vertical circuit cards, referred to as “midplanes”. At the top of the front stack of blades, the air then travels through a first set of air movers that divert the air towards the rear half of the enclosure, and into a first high-pressure plenum. From this top-and-rear-located high-pressure plenum, the air then travels downwardly within the rear half of the enclosure through additional rows of blades attached to the other side of the midplanes, and finally through a second set of air movers that divert the air towards the front half of the enclosure and into a second high-pressure plenum. From this bottom-and-front-located high-pressure plenum, the air again travels upwardly through the front blades, thereby completing a closed loop. This closed-loop, liquid-assisted air cooling architecture enables multiple blades to be connected to the front and rear of the midplanes, thereby facilitating the provision of low-cost, densely arranged, high-performance electrical interconnections within the rack. Because air flowing through the blades travels substantially in parallel with the respective midplane, the midplane does not need to be provided with air-circulation holes, which would tend to block wiring channels, thereby imparting an important advantage to this structural arrangement. Moreover, no vertically directed air plenums, which occupy valuable floor space, are needed in this structure.

On each side of the midplanes, each horizontal row of blades is mechanically supported by a blade cage having bottom and top surfaces, which are substantially open in order to allow for a large volume of a vertical flow of air at a low pressure loss, another important advantage of this arrangement in comparison with the conventional practice of flowing air through small holes formed in the midplanes or backplanes.

Because the air-to-liquid heat exchanger interposed downstream of each blade cage removes from the air, on an average, all heat absorbed therein, the combination of a blade cage and a heat exchanger is thermally neutral for the air; in essence, the air temperature increases from T1to T2as it passes through a blade cage, but then decreases from T2to T1as it passes through the heat exchanger immediately downstream thereof. The air thereby traverses its closed loop, through M blade cages and M heat exchangers, without any net increase in its temperature. Inasmuch as the air loop is enclosed in the rack, and the walls of the rack are insulated with an acoustic-transmission-loss material, the invention provides for a much quieter, more comfortable room for personnel than that encountered in conventional installations where noisy air movers located in the rack expel to the room large amounts of hot air, which must be collected by air-conditioning units that create additional noise. Eliminating this prior-art construction is yet another important advantage of the present invention.

The liquid in the air-to-liquid heat exchangers is normally carried in piping that is distinct from the piping used to carry the liquid for direct-liquid cooling, so the two liquids may differ, but are typically both water, often with anti-corrosion, algicide, and other additives. In order to save vertical space, a coolant-distribution manifolds for direct-liquid cooling of each blade cage is placed immediately in front of a heat exchanger adjacent to the blade cage, thereby ensuring that a pair of hoses, which connect each blade to the inlet or outlet manifolds, may be disconnected for blade removal without having to disturb other blades. Furthermore, in order to save space in the direction normal to the midplanes, the quick disconnects for the manifolds are mounted at an angle. Such efficient packaging permits a large quantity of electronics to be housed within a small amount of space, thereby presenting another advantage of the invention.

BRIEF DESCRIPTION OF THE INVENTION

Referring now specifically to the disclosure,FIG. 1shows a perspective representation, andFIGS. 2A and 2Bshow, respectively equivalent front and side elevations, of a preferred embodiment of an enclosure1, such as for computer or electronic components, and its heat-producing contents. External walls and doors2, which cover the frame3of enclosure1, shown inFIG. 3, are omitted inFIG. 1for purposes of visual accessibility to the interior.FIGS. 2A and 2Bshow an imaginary, Cartesian xyz coordinate system4whose xz plane divides the enclosure1into a front region5(y<0) and a rear region6(y>0), and whose yz plane divides the enclosure into two halves (x>0 and x<0). All other Figures show similar xyz coordinate systems whose axes extend in all cases, in parallel with the yz coordinate system4ofFIGS. 2A and 2B, but the origins of which may differ.

Many of the drawing figures are true-scale representative diminutions of a full-scale, thermal-prototype enclosure built to embody the concepts contained in this application. Frequent reference will be made hereinbelow to the specifics of this prototype embodiment, but it should be understood that the invention is not limited thereto. The x×y×z dimensions of the thermal-prototype enclosure shown inFIG. 3, including the external walls and doors2, but excluding bulk power supplies that are housed in a separate enclosure (as shown inFIG. 22), are 0.81×1.52×2.13 meters (32×60×84 inches). Experiments indicate that by using the cooling schemes described herein, up to 81 kW of heat (55.8 kW liquid-assisted air cooled, 25.5 kW direct-liquid cooled) may be dissipated in the thermal-prototype enclosure1with only modest component temperatures (71° C. worst case, based on over 11,000 measured locations).

Within the front region5, a central-front region {y<0, z1<z<z2} encloses an integer number MFof front heat exchangers7, MFfront blade rows8, and MFfront quick-connect manifolds9. Along the z direction, the front blade rows8are interleaved with the front heat exchangers7. A front quick-connect manifold9, which supplies the blade row8immediately thereabove with cooling liquid for direct-liquid cooling, is located on the −y side of each front heat exchanger7. Each front blade row8comprises an integer number NFof front blades10, which are arrayed along the x direction. Each front blade10is a package, generally having the shape of a rectangular parallelepiped, that contains front heat-producing components11. The heat generated by the heat-producing components11is generally the result of Joule heating encountered in electrical circuits.

Similarly, within the rear region6, a central-rear region {y>0, z1<z<z2} encloses an integer number MRof rear heat exchangers12, MRrear blade rows13, and MRrear quick-connect manifolds14. Along the z direction, the rear blade rows13are interleaved with the rear heat exchangers12. The interleaved ordering of rear blade rows13and rear heat exchangers12is opposite to that of front blade rows8and front heat exchangers7; that is, where the z-wise order of front heat exchangers7and front blade rows8(from bottom to top) is7-8-7-8-7-8-7-8as shown in the drawings, then the z-wise order of rear heat exchangers12and rear blade rows13(from bottom to top) is13-12-13-12-13-12-13-12. One of the rear quick-connect manifolds14is located on the +y side of each rear heat exchanger12. Each rear blade row13comprises an integer number NRof rear blades15that contain rear heat-producing components16.

Although front blade rows8are illustrated as being identical to rear blade rows13, and front heat exchangers7are illustrated as being identical to rear heat exchangers12, it is possible to accommodate dissimilar blade rows and heat exchangers, provided that the heat exchanger immediately downstream of each blade row is sized appropriately to remove the heat load thereof. Moreover, although MF=NF=4 is shown, other values of MFand NF, which may not be necessarily equal, are within the scope of the disclosure. Likewise, although MR=NR=4 is shown, other values of MRand NR, not necessarily equal, are also applicable. Furthermore, although the z-wise orders7-8-7-8-7-8-7-8and13-12-13-12-13-12-13-12are shown in the front region5and rear region6respectively, the opposite orders,8-7-8-7-8-7-8-7and12-13-12-13-12-13-12-13, are also possible. In addition, z-wise arrangements using fewer heat exchangers, such as8-8-7-8-8-7and12-13-13-12-13-13, are also contemplated.

Each front blade10and each rear blade15is electrically connected to a midplane (17,18), which is an electrical circuit card upstanding in the xz plane, and whose function resides in providing electrical power to, and electrical communication between, the blades that are connected thereto. The enclosure1may contain one or more midplanes, although one large midplane is often preferred so as to provide connectivity between as many blades as possible. However, because the maximum size of circuit cards may be limited by logistics of manufacturability, two or more midplanes may exist in the enclosure1. As an example,FIG. 2Bshows two midplanes, a lower midplane17and an upper midplane18. In this case, with four rows of front blades (MF=4) and four rows of rear blades (MR=4), all front blades10in the lower two front blade rows8connect to the −y surface of the lower midplane17via front midplane connectors19, and all rear blades15in the lower two rear blade rows13connect to the +y surface of the lower midplane17via rear midplane connectors20. Likewise, all front blades10in the upper two front blade rows8connect to the −y surface of the upper midplane18via front midplane connector19, and all rear blades15in the upper two rear blade rows13connect to the +y surface of the upper midplane18via rear midplane connectors20where at least one connector is provided for each blade, but multiple connectors could also be used for each blade.

In general, although not necessarily, the front heat-producing components11may be divided into two types: low-power-density front heat-producing components21and high-power-density front heat-producing components22. Similarly, the rear heat-producing components16may be divided into low-power-density rear heat-producing components27and high-power-density rear heat-producing components28. A low-power-density heat-producing component may be defined, for example, as having a worst-case surface heat flux less than P; a high-power-density heat-producing component may then be defined as having a worst-case surface heat flux that exceeds P, where as typical value of P may be 75 W/cm2. Although, inFIG. 2B, high-power-density components22,28are shown only at the centers of the blades in the y direction, the invention is not restricted thereto; in general, low-power-density components21,27and high-power-density components22,28may be located anywhere within the three-dimensional volume occupied by front blades10or rear blades15. Furthermore, classification of a component as “low-power-density” or “high-power-density” depends largely on the distribution of heat generation therewithin, because not only the peak heat flux P at a “hotspot”, but also the physical size of the hotspot, must be considered. Moreover, P also depends on the highest permissible temperature Tmaxof a component: the lower the required value of Tmax, the lower the definition of P must be.

The represented prototype embodiment uses mockup heat-producing components, such as resistors and thermal test chips, instead of real heat-producing components such as processor and memory chips. In order to monitor the temperature of the mockup heat-producing components, over 11,000 temperature sensors are placed near selected components throughout the prototype enclosure1. Power dissipation of the mockup low-power-density heat-producing components is 1744 watts per blade; power dissipation of the mockup high-power-density components is 800 watts per blade. The prototype enclosure has space for 32 blades, as shown inFIGS. 2A and 2B, and thus can accommodate a total enclosure power of 81 kW, which far exceeds the capabilities of conventional equipment enclosures. The plan-form power density of the prototype embodiment is 81 kW/13.3 ft26 kW/ft2. This far exceeds the power density of typical data centers, which are designed to be cooled by the use of circulating air.

For the two types of heat-producing components, the present invention provides two different cooling solutions; namely, closed-loop liquid-assisted air cooling for low-power-density components21,27, and direct-liquid cooling for high-power-density components22,28,

Referring toFIG. 1, closed-loop liquid-assisted air-cooling employs a closed loop23of circulating air that is confined within the enclosure1. The closed loop23comprises a front airflow segment24flowing towards +z in the central-front region (y<0; z1<z<z2) of the enclosure1, a top airflow segment25flowing towards +y in a top region {z>z2} of the enclosure, a rear airflow segment26flowing towards −z in the central-rear region (y>0; z1<z<z2) of the enclosure, and a bottom airflow segment27flowing towards −y in a bottom region {z<z1} of the enclosure1.

As air flows along the front airflow segment24, it is alternately cooled by one of the front heat exchangers7and then heated by one of the front blade rows8; this cooling-and-heating cycle occurs MFtimes along the front airflow segment24. Similarly, as air flows along the rear airflow segment26, it is alternately heated by one of the rear blade rows13and then cooled by one of the rear heat exchangers12; with this heating-and-cooling cycle occurring MRtimes along the rear airflow segment26. Thus, along the front airflow segment24, each adjacent combination of front heat exchanger7, front blade row8, and front quick-connect manifold9represents a thermally neutral front unit28; whereby on average, each streamline of air in the front airflow segment is cooled by one of the heat exchangers from a temperature T2to a lower temperature T1, but is then reheated by the following front blade row from temperature T1to the original temperature T2. Likewise, each combination of rear heat exchanger12, rear blade row13, and rear quick-connect manifold14represents a thermally neutral rear unit29. Because the aforesaid z-wise order7-8-7-8-7-8-7of front heat exchangers7and front blade rows8is arranged opposite to the z-wise order13-12-13-12-13-12-13-12of rear heat exchangers12and rear blade rows13, the desired, alternating order of heat exchangers and blade rows is maintained, in the air-stream direction, as the air moves (at the top of the enclosure1) from the front region5to the rear region6, and conversely as it moves (at the bottom of the enclosure1) from back to front.

InFIG. 1, this closed loop23is depicted diagrammatically as the rectangle that is delineated by an upper-front airflow corner30, an upper-rear airflow corner31, a lower-rear airflow corner32, and a lower-front airflow corner33. Movement of air along the closed air loop23is driven by an upper air-moving assembly34(FIGS. 2A and 2B) that comprises an integral number KUof upper fans, such as35,36,37,38located in a top-front region {y<0; z>z2} of the enclosure1, as well as by a lower air-moving assembly39that comprises an integral number KLof lower fans such as40,41,42,43, located in a bottom-rear region {y>0; z<z1} of the enclosure1. In the prototype embodiment shown inFIG. 1andFIGS. 2A and 2B, the number of upper fans in the first air-moving assembly34is KU=4; likewise, the number of lower fans in the second air-moving assembly39is KL=4. In such an embodiment, the closed air path23, shown schematically inFIG. 1as the single rectangle (30,31,32,33), is more accurately represented, as shown inFIGS. 2A and 2B, as two concentric rectangles; namely, an inner rectangle44and an outer rectangle45. The inner rectangle44represents air driven by the pair of top-inner fans (37,38) and the pair of bottom-inner fans (42,43). The outer rectangle45represents air driven by the pair of upper-outer fans (35,36) and the pair of lower-outer fans (40,41). It should be emphasized that, notwithstanding this illustration of the airflow as one discrete loop23or as two discrete loops44and45, in actuality an infinite number of parallel streamlines flow along such closed paths, bathing the entire volume occupied by the blade rows (8,13), the heat exchangers (7,12), and the air-moving assemblies (34,39) in the air stream.

Other arrangements of air movers are also within the scope of the invention. For example, arrays of axial-flow fans may be interleaved between blade cages and heat exchangers so as to replace or supplement the air-moving power of the shown centrifugal fans.

In the closed loop23, the only empty spaces needed for air plenums are a top-rear region46{y>0; z<Z2} and a bottom-front region47{y>0; <Z2}. Consequently, no floor space is lost along the sides or front and back of the rack for air distribution, because there are no air plenums that extend vertically in the enclosure. Thus, except for the space occupied by the external walls or doors2, the full “foot-print” of the rack is available for electronics, which are contained in the blades (10,15).

Air in the closed loop23increases in temperature from a cool temperature T1to a warm temperature T2as it flows through each blade row (8.13), because the air convectively absorbs heat dissipated by the low-power-density components21,27(as shown inFIG. 2B) therein. However, the heated air is immediately restored to the cool temperature T1as it flows through the heat exchanger (7,12) that immediately follows the blade row in the air-stream direction. Thus, each adjacent combination of front heat exchanger7and front blade row8is thermally neutral for the air. Consequently, in traversing through the closed loop23, the air is heated and cooled MF+MRtimes, with no net change in temperature being encountered during steady-state operation.

The heat exchangers7and12are air-to-liquid heat exchangers in whose liquid side is circulated an air-assisting liquid48. All heat dissipated by the low-power-density components is removed from the enclosure1by the air-assisting liquid48, which is typically, but not necessarily, water that communicates with an external chilled-water system (not shown). This chilled-water system must provide, by means well known in the art, reasonably-clean, non-corrosive, above-dew-point water for use in the heat exchangers (7,12). Clean water is required to prevent heat-exchanger fouling (which can compromise heat-transfer performance); non-corrosive water is required to prevent corrosion of metal plumbing; and above-dew-point water is desired to avoid water in the air from condensing on the surfaces of the heat-exchangers.

For direct-liquid cooling of the high-power-density heat-producing components22,28, a direct-cooling liquid49is conveyed thereto in a manner described below, whence the components' heat load is primarily transferred directly to the direct-cooling liquid49by solid-to-solid conduction and solid-to-liquid convection. Thus, virtually all heat dissipated by the high-power-density components is removed from the enclosure1by the direct-cooling liquid49, which is typically water communicating with an external chilled-water system. Again, the chilled-water system must provide, by means well known in the art, filtered, non-corrosive, above-dew-point water for use in direct liquid cooling of the high power density components22,28.

Because both types of cooling, i.e., closed-loop liquid-assisted air cooling and direct-liquid cooling, reject heat to liquids within the enclosure1, the entire enclosure appears to an outside observer to be liquid cooled. Yet in reality, air cooling is used to advantage internally, because the low-power-density components21,27, treated with liquid-assisted air cooling, are ordinarily numerous and therefore difficult and expensive to treat with direct-liquid cooling.

In principle, the direct-cooling liquid49and the air-assisting liquid48may be independent of each other, and may resultingly operate at different temperatures, thereby allowing, for example, very low-temperature high-power-density components (e.g. processors) to be combined with higher-temperature, air-cooled low-power-density components (e.g. memory chips). It is not ordinarily contemplated to have any components at temperature low enough so that condensation forms under typical computer machine room conditions. However, if very or extremely low temperatures are required, the relatively well-sealed volume of air inside the enclosure1may be dehumidified by suitable means well known in the art.

Discussed hereinbelow in specific detail are various components and operating aspects of the inventive apparatus employed for implementation of the novel cooling method.

Thermally Neutral Units

The structure of a thermally neutral front unit28, which comprises one of the front heat exchangers7, the front blade row8directly thereabove, and the front quick-connect manifold9directly in front thereof, is described in detail below. The structure of a rear thermally neutral unit, which comprises one of the rear heat exchanger12, the rear blade row13directly therebeneath, and the rear quick-connect manifold14directly therebehind, is similar, possibly even identical, except for the inverted z-wise order of components, as discussed previously in connection withFIGS. 2A and 2B.

FIGS. 4 and 5illustrate two views of one of the MFthermally neutral front units2B, which, for explanatory purposes, is assumed to be the thermally neutral unit at the lower left ofFIG. 2B. The front blade row8comprises NFfront blades10housed together in a front blade cage50, which mechanically supports the front blades within the enclosure1, and which locates the blades10for slidable connection in the −y direction relative to the midplane17, to which blades in a rear blade cage (not shown inFIGS. 4 and 5) may also be connected. As shown, the midplane preferably extends in the z direction above this thermally neutral front unit (assumed above to be the lowest in the enclosure1), so that instead of the midplane being shared only by the lowest horizontal blade rows front and rear, it is shared by more than one horizontal row. For example,FIG. 2shows lower and upper midplanes (17,18) that each provide connectivity between four blade rows: two front blade rows8and two rear blade rows13. The midplane17may also extend beyond the blade cage50in the !x direction, as shown, to allow space for connections that bring electrical power to the midplane, which in turn distributes power to the front and rear blade rows (8,13).

In a prototype embodiment, each prototype front blade10has dimensions of 120 mm×560 mm×305 mm in the x, y, and z directions, respectively, and each prototype blade cage50has xyz dimensions of 573 mm×605 mm×311 mm. Each prototype heat exchanger has dimensions of 540 mm×605 mm×48 mm in the x, y, and z directions, respectively. The prototype thermally neutral units28are stacked on a 375-mm pitch in the z direction.

FIG. 4shows the NF=4 front blades located in the positions they would normally occupy during operation when plugged at right angles into the midplane17.FIG. 5shows the leftmost blade disconnected from the midplane and partially withdrawn from the blade cage, thereby illustrating the manner in which a blade is removed from the blade cage for servicing or replacement.FIG. 5also shows the rightmost two blades omitted, thereby revealing the front air-to-liquid heat exchanger7therebeneath.

Arrows inFIGS. 4 and 5indicate the flow of cooling fluids through the thermally neutral front unit28. The closed loop23of circulating air, which cools the low-power-density components21, flows in the +z direction. In the prototype embodiment, the volumetric flow rate of air along the closed loop23is determined by a detailed scan of measured velocities over a typical blade. The scanned velocities vary between about 3.0 and 5.0 m/s. The average air velocity is 3.4 m/s over the 480 mm×535 mm cross-sectional area of the loop. Integrating the velocity over the cross-sectional area, the total volumetric flow rate along the closed loop23is 0.873 m3/s (1850 standard cubic feet per minute). Because the air is cooled by heat exchange to liquid eight times around the prototype-embodiment's closed loop23, it is to be appreciated that this 1850 CFM is equivalent to 14,800 CFM of conventional air cooling, because the latter does not use multiple heat exchanges from air to liquid. Such a large equivalent flow rate of air is extremely difficult to accomplish (with an enclosure of the size used in the prototype embodiment) by means other than those taught by the present invention.

The air-assisting liquid48, which cools the closed loop23of circulating air before it enters the next blade-row thereabove, enters the heat exchanger7through a heat-exchanger supply fitting51and exits through a heat-exchanger return fitting52. In a prototype embodiment, the air-assisting liquid employed is water, with a volumetric flow rate through each heat exchanger of approximately 11.4 liter/minute (3.0 gallons/minute).

Direct-cooling liquid49, which cools high-power-density components22, enters the quick-connect manifold9through a manifold supply pipe53and exits through a manifold return pipe54. In the prototype embodiment, the direct-cooling liquid is preferably water, with a volumetric flow rate to the quick-connect manifold9of approximately 8.0 liters/minute (2.11 gallons/minute), which imparts a flow to each identical blade of 2.0 liters/minute (0.53 gallons/minute).

Blade Cages

FIG. 6illustrates the front blade cage50, showing all blades having been removed, thereby revealing its structure as possessing solid side surfaces55; an open front cage surface56to allow insertion of front blades; a slotted rear cage surface57with slots58to allow connectors near the +y edge of the front blades to mate to the midplane connectors19on the front surface of one of the midplanes (17,18); a bottom cage surface59having large rectangular bottom-flow-through holes60; and a top cage surface61having similar, large rectangular top-flow-through holes62. The bottom flow-through holes60and top-flow-through holes62together allow for airflow through the front blades in the +z direction. Between the NFbottom-flow-through holes are (NF−1) bottom blade guides63formed from the sheet metal of the bottom cage surface59. Likewise, between the top-flow-through holes, top blade guides64are formed from the sheet metal of the top cage surface61. Each blade guide (63,64) has a U-shaped cross section in the xz plane that provides guidance for the blades as they prepare to engage the midplane connectors19. The U-shaped cross section also imparts considerable stiffness to the guide itself, thereby preventing excessive bending of the bottom guides63under the weight of the front blades10, which bear on the bottom guides as the blades are inserted and removed.

As shown inFIG. 5, each front blade10has a left sheet-metal skin65on its −x face and a right sheet-metal skin66on its +x face. Each of these faces has a hemmed top edge67that slides within one of the top blade guides64while the blade is being inserted or withdrawn, and a hemmed bottom edge68that similarly slides within one of the bottom blade guides63. Hemming the edges prevents galling the blade guides (63,64) as the edges slide on them. Each bottom blade guide63is wide enough to accept two hemmed bottom edges68; one belonging to the right sheet-metal skin66of the blade to its left, and the other belonging to the left sheet-metal skin65of the blade to its right. Likewise, each top blade guide64is wide enough to accept two hemmed top edges67, one from a blade to its left, and another from a blade to its right. As shown inFIG. 6, special guides69at the extreme left and right of the blade cage, both top and bottom, are wide enough to accept one hemmed edge only. To assist in aligning the blade in the x direction so that the hemmed edges properly engage the blade guides, tapered starting blocks70, affixed to the bottom and top cage surfaces (59,61) are provided between blade guides (63,64,69).

Referring toFIG. 4, the direct-cooling liquid49in the manifold supply pipe53is supplied to one of the blades10, for example the second blade from the right inFIG. 4, by flowing first through a supply quick connect71that is attached (for the purpose of minimizing the y dimension of the enclosure1) at an acute angle to the manifold supply pipe53, then through a manifold-supply elbow fitting72, thereafter through a flexible supply hose73, and finally through a blade-supply elbow fitting74. If it is desired to balance flow between various blades, a control valve may be inserted between each flexible supply hose73and the corresponding blade-supply elbow fitting74. In a prototype embodiment, needle valves such as sold under the registered trademark “SWAGELOK”, Model SS-IR56, may be used for this purpose. Flow of the direct-cooling liquid49through the blade itself is described hereinbelow. The direct-cooling liquid is returned from the blade to the manifold return pipe54by flowing first through a blade-return elbow fitting75, then through a flexible return hose76, then through a manifold-return elbow fitting77, and finally through a return quick connect78. The flexible supply hose73and flexible return hose76must be flexible to permit operation of the supply quick connect71and the return quick connect78. For example, 85 durometer polyurethane hose may be suitable for hoses73,76. In the prototype embodiment, 6.35- mm-I.D., 9.53-mm-O.D. hose of this type is used, with “SWAGELOK”, Model SS-IR56(Reg. ™) connections.

A quick-connect, well known in the art, is a two-piece plumbing connection that provides rapid, easy, virtually dripless connection and disconnection of a fluid line. The two pieces are referred to as “body” and “stem”; the body is the larger (female) half of the connection; the stem is the smaller (male) half. Each piece has a shut-off valve, whereby when the two halves are disconnected with fluid flowing in the line, the flow is automatically stopped in both disconnected halves of the line. When the two halves are re-connected, the flow automatically restarts. Such convenient disconnection and reconnection are essential to the equipment in this invention, inasmuch as the equipment is prone to occasional failure, and thus requires occasional servicing or replacement. Although high-quality quick connects are quite reliable and virtually dripless, the supply and return quick connects71,78in the preferred embodiment are located, as shown inFIG. 4, in front of the blades, with no electronic components located directly therebeneath. Such a location is preferred to avoid any possibility of direct-cooling liquid49dripping onto the electronics.

As shown by the empty blade positions inFIG. 5, the manifold supply pipe has attached thereto, at NFlocations (one for each of the NFblades in the blade row), an angle block79and a quick-connect supply body80. The angle block is designed to orient the quick connects71,78at an acute angle to the manifold pipes53,54, rather than at a right angle, in order to minimize the y dimension of the enclosure1. Similarly, the manifold return pipe54has attached thereto, at NFlocations, one of the angle blocks79and a quick-connect return stem81. Each blade10has a quick-connect supply stem82(seen more clearly inFIG. 10) that leads to the supply hose73and a quick-connect return body83that leads to the return hose76. One of the blades10may be quickly connected to the flow of direct-cooling liquid49by connecting the blade's quick-connect supply stem82to the manifold's quick-connect supply body80, and by also connecting the blade's quick-connect return body83to the manifold's quick-connect return stem81. The connections are arranged this way, with supply and return connections having opposite genders, to avoid any possibility of erroneous connection. In a prototype embodiment, the quick-connect bodies and stems may be of the type sold under the registered trademark “SWAGELOK” Models SS-QTM2A-B-4PM and QTM2-D-4PM, respectively.

Heat Exchangers

In a prototype embodiment, the front heat exchangers7and rear heat exchangers12are identical; details of such a heat exchanger, as well as the quick-connect manifold9attached thereto, are shown inFIG. 7. The construction of this device, known as a copper-tube, aluminum-fin air-to-liquid heat exchanger, is well known in the art of heat-exchanger fabrication. It comprises a copper supply fitting84that supplies the air-assisting liquid48from an external chilled-liquid system (not shown) to a copper-pipe supply header85, and a copper return fitting86that returns the air-assisting liquid48from a copper-pipe return header87to the chilled-liquid system. In the heat exchanger, flow of the air-assisting liquid occurs through an integer number NCof copper piping circuits88that in parallel convey the air-assisting liquid from the supply header85to the return header87. One end of each piping circuit88is connected to the supply header85by a supply feeder89; the other end of each piping circuit is connected to the return header87by a return feeder90. Each piping circuit88comprises an integer number NPof straight copper pipes91that extend back and forth along the +x and −x directions through tight holes in finely spaced aluminum fins92. The NPstraight copper pipes are connected at their ends by NP−1 U-turn copper fittings93, thereby to form a continuous meandering path from supply header to return header. All along this meandering path, the air-assisting liquid48absorbs heat; heat is transferred first by convection from hot air in the closed air loop23to the aluminum fins92and straight copper pipes91, then by conduction through the aluminum fins92and the straight copper pipes91, and finally by convection from the interior of the copper pipes to the air-assisting liquid48within the tubes. Surrounding the finned area of the heat exchanger7is a four-sided C-channel frame94,95,96,97whose right side95is shown partially cutaway inFIG. 7in order to reveal the piping circuits88. Holes in the frame sides95,97support the piping circuits.

In the prototype embodiment, the number of piping circuits88is NC=7, the number of passes per circuit is NP=6, the outer diameter of copper pipes in piping circuits88is 9.5 mm, the outer diameter of the header pipes (85,87) is 16 mm, the fins92are 0.1 mm thick on 1.5 mm centers, and the height of each fin in the z direction is 44 mm. The finned area of the heat exchanger covers the full cross-sectional area of the front blade row8that it must cool; for the prototype embodiment, the x and y dimensions of this area are 480 mm and 530 mm, respectively.

A space-saving advantage of the prototype embodiment resides in that the quick-connect manifold9nestles inside the C-channel-frame's front member94, being attached thereto by means of scalloped clamps98that cradle the manifold supply and return pipes53,54. The front half of each clamp, visible inFIG. 7, cradles the front surfaces of the pipes53,54; while the rear half of each clamp, not visible inFIG. 7, cradles the rear surfaces of the pipes. The rear half of each clamp is affixed to the C-channel-frame's front member94. To secure the pipes53,54to the front member94, the front and rear halves of each clamp98are pulled together by a screw that passes through a hole99in the front half of the scalloped clamp, passes between the two pipes53,54, and engages a threaded hole in the rear half of the scalloped clamp.

Plumbing Connections for Heat Exchangers

The heat-exchanger's supply header85is connected to a heat-exchanger supply riser100, shown schematically inFIG. 8and pictorially inFIG. 9, that supplies the air-assisting liquid48from a first chilled-liquid system (not shown) at a supply temperature TS1to an entire column of heat exchangers, which are connected in parallel. If the first chilled-liquid system is a chilled-water system, as is well known in the art, then the temperature of water in the heat-exchanger supply riser100is typically TS1=18 to 20° C. in order to be safely above the dew-point temperature of typical computer-room environments.

The heat exchanger's return header87is connected to a heat-exchanger return riser101, shown schematically inFIG. 8and pictorially inFIG. 9, that returns the air-assisting liquid48from an entire column of heat exchangers, connected in parallel, to the chilled-liquid system. This return water has been warmed to a return temperature TR1by absorption of heat from the closed air loop23. If the chilled-liquid system is a typical chilled-water system, the typical return temperature is TR1=25-27° C., such that the temperature rise TR1-TS1of the water across the heat exchanger is typically within a preferable range of about 5-10° C., predicated on the disclosed system.

In a prototype embodiment, the heat load of the low-power-density components per front blade row8is experimentally about 5.4 kW to 6.9 kW. This heat load is adequately cooled by one of the prototype heat exchangers7when it carries a flow rate of approximately 11.4 liter/min (3.0 gallon/min) of the air-assisting liquid48.

Plumbing Connections for Quick-Connect Manifolds

Referring to the schematicFIG. 8and the pictorial representation inFIG. 9, a front quick-connect supply riser102supplies the direct-cooling liquid49from a second chilled-liquid system (not shown), at a supply temperature TS2, to MFfront quick-connect-manifold supply pipes53, one of which belongs to each of the MFfront quick-connect manifolds9. Likewise, a rear quick-connect supply riser103supplies the direct-cooling liquid49from the second chilled-liquid system, at the supply temperature TS2, to MRrear quick-connect-manifold supply pipes104, one of which belongs to each of the MRrear quick-connect manifolds14. If the second chilled-liquid system is a chilled-water system, as is well known in the art, then the temperature of water in the quick-connect supply risers102,103is typically TS2=18 to 20° C. in order to be safely above the dew-point temperature of typical computer-room environments.

Still referring to the schematicFIG. 8and the pictorialFIG. 9, MFfront quick-connect-manifold return pipes54(one per front quick-connect manifold) return the direct-cooling liquid49to a front quick-connect return riser105, and thence to the return side of the second chilled-liquid system. Similarly, MRrear quick-connect manifold return pipes106(one per rear quick-connect manifold) return the direct-cooling liquid49to a rear quick-connect return riser107, and thence to the return side of the second chilled-liquid system. Because the direct-cooling liquid has absorbed, from the front and rear blades, the heat that was dissipated by the high-power-density components therein, the return water has a temperature TR2that is higher than TF2. If the second chilled-liquid system is a chilled-water system, as is well known in the art, then the temperature of water in the quick-connect return risers105is typically TR2=25 to 27° C.; that is, the flow rate through the manifolds is typically adjusted to produce a water-temperature rise,TηTR2-TS2, of about 5-10° C.

Prototype Blade

One of the front blades10used in the prototype embodiment is illustrated inFIG. 10and inFIG. 11.FIG. 10shows the blade from the −x direction, whereasFIG. 11shows the blade from the +x direction. In order to display the blade's internal structure, its left sheet-metal skin65is hidden inFIG. 10, whereas its right sheet-metal skin66is hidden inFIG. 11. From the description above, it is evident that the particular structure of this blade is merely an example of the type of equipment that may be cooled in accordance with the invention; in general, the invention applies regardless of the locations of the low-power-density and high-power-density heat-producing devices within the volume of the blades. Nevertheless, the blade structure described herein has several advantages, as elucidated hereinbelow.

The blade comprises a blade circuit card108having a front surface facing the −x direction (shown inFIG. 10) and a rear surface facing the +x direction (shown inFIG. 11). At the +y edge of the blade circuit card108, the midplane connectors19are electrically connected thereto. Also electrically connected to the blade circuit card108are four types of heat-producing components: first, four groups of DIMMs (“Dual-In-Line Memory Modules”), a standard format for carrying computer-memory chips, including an upper-front DIMM array109, a lower-front DIMM array110, an upper-rear DIMM array111, and a lower-rear DIMM array112; second, two processor modules, including an upper processor module113and a lower processor module114; third, four DIMM-power converters, including an upper-front DIMM-power converter115, a lower-front DIMM power converter116, an upper-rear DIMM power converter117, and a lower-rear DIMM power converter118; and fourth, two processor-power converters, including an upper processor power converter119and a lower processor power converter120. Note that the power converters115-118include fined heat sinks, which are visible inFIGS. 10 & 11and which obscure the active electronic components used for power conversion that are mounted to circuit card108. Of all these heat-producing components, only the processor modules (113,114) are high-power-density, direct-liquid cooled components; all of the others are low-power-density, air-cooled components.

Each “power-converter” component115-120delivers low-voltage, high-amperage power to the DIMM array109-112or to the processor module113,114that lies directly opposite on the other side of the blade circuit card108. For example, the upper-front DIMM power converter115(FIG. 11) delivers power to the upper-front DIMM array109(FIG. 10); the lower-rear DIMM-power converter118delivers power to the lower-rear DIMM array112. Likewise, the upper-processor power converter119delivers power to the upper processor module113, and the lower-processor power converter120delivers power to the lower processor module114. This arrangement, in which each power converter lies directly opposite the component it powers, produces very short electrical paths from the power converters to their respective loads, thereby providing a low-loss means of delivering the low-voltage, high-amperage power, and representing an advantage of this invention.

An additional advantage of this invention resides in that the DIMM arrays109-112are arranged such that, in traversing a blade, no streamline of air passes through more than one DIMM array, thereby preventing overheated air that would lead to poor cooling of components furthest downstream. For example (FIG. 10), on its path through the blade, an air streamline121passes through the lower-rear DIMM power converter118and then through the upper-front DIMM array109. Because the DIMM power converter118dissipates only about 18% as much heat as the lower-rear DIMM array112that it powers, the arrangement of DIMMs shown is, from a cooling viewpoint, far superior to an alternative arrangement having all DIMM arrays109-112on one side of the blade circuit card108and all DIMM power converters115-118on the other side, because in that case, some air streamlines would pass through two DIMM arrays.

Both of the advantages cited above, i.e., short electrical paths for power delivery and efficient DIMM arrangement to avoid overheated air, derive from components being placed on both sides of the blade circuit card108. This is possible only if the blade circuit card stands in a plane parallel to yz which lies, as shown inFIG. 12, midway between the blade's left sheet-metal skin65and its right sheet-metal skin66. This must be done while maintaining the −z and +z faces of the blade open so as to allow for a vertical airflow. Referring toFIG. 11andFIG. 12, these requirements are met by suspending the blade circuit card108and its components from an upper angle bracket122and a lower angle bracket123that are attached to a tailstock129, and from a left U-channel strut124and a right U-channel strut125that are attached to the left sheet-metal skin65and the right sheet-metal skin66, respectively.

Referring toFIGS. 10 and 11, in the prototype embodiment, each air-cooled DIMM array109-112comprises 16 double-high DIMMs126on 11-mm centers. Prototype DIMM cards are thermal mockups, containing simple resistors to generate heat, rather than real DRAM and hub chips. Each mockup DIMM dissipates either 20 W or 26 W of heat, depending on its configuration, so that each DIMM array109-112dissipates about 320 W (with 20 W DMMs) or 416 W (with 26 W DMMs). Air-cooled DIMM power converters115-118and processor power-converters119-120, which typically dissipate only 18% as much heat as the devices they power, are not simulated thermally in the prototype embodiment. However, each prototype blade has, near the midplane connectors19, additional air-cooled heat-producing components (not shown in the Figures) that dissipate 80 W. Thus the total air-cooled heat dissipation per prototype blade is (4)(320)+80=1360 W (with 20 W DIMMs) or (4)(416)+80=1744 W (with 26 W DIMMs). In order to measure the thermal performance of the prototype embodiment, over 11,000 temperature sensors are located near heat-producing components on the mockup DIMM cards throughout the prototype enclosure1. With 1360 air-cooled watts per blade, and water entering the heat exchangers7at 15° C. (slightly lower than the range 18-20° C. suggested hereinabove, because humidity in the prototype's laboratory environment is controlled such that 15° C. is well above dew point), the highest temperature measured by these sensors is 56° C. With 1744 air-cooled watts per blade, the highest air-cooled-component temperature measured is 71° C. The highest temperatures are typically located near the downstream edges of DIMM cards, where ambient air is warmest due to heating thereof by components upstream.

In the prototype embodiment, each mockup processor module contains an 18.5×18.5 mm silicon heater chip dissipating 350 W of heat, yielding an average power density of 1.02 W/mm2. This heat is removed by the direct-cooling liquid49that flows in a processor cooling head127(FIG. 10). For the prototype embodiment, the direct-cooling liquid is water, which enters the cooling head127at 10° C. (lower than the range 18-20° C. suggested hereinabove, because humidity in the prototype's laboratory environment is controlled such that 10° C. is above dew point). The prototype cooling head, a silicon-microchannel cooler, is attached to the silicon chip in such a way as to remove the chip's heat efficiently, in order to maintain the chip at the lowest possible temperature. Thermal sensors integrated into the silicon heater chip illustrate that, with water flowing at 1 liter/min through the cooling head, the maximum temperature on the silicon chip is about 35° C. This represents a total thermal resistance (chip to inlet water) of 0.07° C./W. Using the chip area given above, this is equivalent to an area-normalized thermal resistance of 24° C./(W/mm2). In an alternative embodiment, the prototype blades are populated with heater chips generating 96 W (0.28 W/mm) that are air cooled by Heatlane™ heatsink technology. This air-cooled solution provides a total thermal resistance of 0.27° C./W, which is equivalent to an area-normalized resistance of 92° C./(W/mm2). Thus, the direct-water-cooled solution has nearly four times the cooling capability of the air-cooled solution.

Referring toFIG. 10, the direct-cooling liquid49which cools the processor modules enters the blade through the supply hose73and blade-supply elbow fitting74, as previously described, thereafter to a feed-through supply fitting128that passes through a hole in the blade tailstock129, then to a blade supply pipe130, then to a flow meter131which verifies that an adequate flow of direct-cooling liquid is present before power is applied to the processor modules113-114and finally to a blade-supply manifold132that feeds two hoses, including an upper-processor supply hose133and a lower-processor supply hose134, which convey the direct-cooling liquid49to the cooling heads127that cool the upper processor module113and lower processor module114. After passing through the cooling heads127, the direct-cooling liquid flows through an upper-processor return hose135and a lower-processor return hose136, whose flows are combined in a blade-return manifold137. Referring now toFIG. 11, the blade-return manifold137discharges this flow to a return elbow fitting138that passes through a hole in the blade circuit card108and delivers the flow to a blade return pipe139, from there to a return feed-through fitting140that passes through a hole in the blade tailstock129, then to the blade return elbow fitting75, and finally to the return hose76.

Air Movers

The air-moving assemblies34,39are now described in more specific detail, along with related issues such as acoustic insulation, sealing, flow control, and fan failure.

Each of the fans35-38and40-43driving the closed-loop airflow23is preferably of the type known as a “centrifugal fan” or “blower”, because such fans naturally cause the air to turn a right-angle corner. Thus, if the upper fans35-38are of the centrifugal type, they naturally cause the air to turn at the upper-front airflow corner30(FIG. 1); similarly, if the lower fans40-43are centrifugal, they naturally cause the air to turn at the lower-rear airflow corner32.

Referring toFIG. 13and assuming the use of centrifugal fans, each of the upper fans35-38has an axis of rotation that is parallel to the z-axis, an intake air-stream141flowing toward +z, and an exhaust air-stream flowing142flowing toward +y. The latter flow direction is achieved by a fan-and-housing assembly143, as shown inFIG. 13, wherein each upper fan35-38is enclosed in a housing144having two open sides; i.e., an intake side145facing −z and an exhaust side146facing +y. Similarly, each of the lower fans40-43has an axis of rotation that is extended in parallel with the z axis, an air-intake direction pointing toward −z, and an air-exhaust direction pointing toward −y. The latter is achieved (assuming that upper and lower fans are identical) by the fan-and-housing assembly144, which for the lower fans is oriented so that the open intake side145faces −z and the open exhaust side146faces −y. In other words, if lower fans and upper fans are identical, then a lower fan40-43in its housing144is merely an upside down version of an upper fan35-38in its housing144.

Because the air loop23is closed, the noise created by the moving air, and in particular the noise created by the fans35-38and40-43, may be acoustically isolated inside the enclosure1, thereby minimizing annoyance to nearby personnel, and protecting their hearing. In contrast, acoustical isolation is much more difficult to achieve for conventional enclosures where the air used for air cooling therewithin flows across the enclosure boundary to the outside. Improved acoustic isolation is thus a key advantage of the present invention. Acoustic isolation of the fan and air noise within the enclosure1is readily accomplished by lining all inside surfaces of its outer shell, especially walls and doors2, with a layer147of an acoustic insulation, as shown inFIG. 3. This insulation should preferably be of the type known as a “transmission-loss material”, which attenuates the transmission of acoustic energy therethrough. A transmission-loss material is primarily characterized by its mass, the greater the mass per unit area of the layer, the greater the attenuation. In a prototype embodiment of this invention, the acoustic insulation used was a 1″-thick (25.4-mm-thick) layer of SOUNDMAT (registered trademark) PB material, which is a self-adhesive transmission-loss material, made by SoundCoat corporation, that includes a “barrier layer” (transmission-loss layer) having an areal density of 1 lb/ft2(4.88 N/m2).

Referring again toFIGS. 1,2A and2B, the top-rear region46{y>0; z>z2} of the enclosure1contains only air, thereby providing a high-pressure plenum in which the closed air loop23turns the upper-rear airflow corner31, the air being driven to execute this turn because of the favorable pressure gradient created in the −z direction by the low-pressure intake of the lower fans40-43. Likewise, a bottom-front region47of the enclosure10contains only air, thereby providing a high-pressure plenum in which the closed air loop23turns the lower-front airflow corner33, the air being driven to execute this turn because of the favorable pressure gradient created in the +z direction by the low-pressure intake of the upper fans35-38.

The fans in each air-moving assembly (34,39) are arranged so that their air streams do not substantially interfere. This is achieved by placing the fans in an over-and-under, fore-and-aft arrangement shown inFIG. 14, which depicts, without fan housings144, the upper fans (35-38) of the upper air-moving assembly34. Because of the fore-and aft arrangement, outer-intake airstreams150and151of the outer fans (35,36) do not interfere with inner-intake airstreams152and153of the inner fans (37,38). Also, because of the over-and-under arrangement, outer-exhaust airstreams154and155of the outer fans (35,36) do not interfere and with inner-exhaust airstreams156and157of the inner fans (37,38).

In the prototype embodiment, all fans (upper fans35-38and lower fans40-43) are preferably backward-curved centrifugal fans having a 250-mm-diameter rotating wheel158comprising eleven backward-curved blades159. The pressure-flow performance of each such fan is enhanced with a flared inlet ring160, which guides air smoothly into the fan. When such an inlet ring is used, space in the z direction may be saved, as shown, by packaging the fans so that the wheel158of each inner fan37,38partially overlaps (in the z direction) the inlet ring160of the corresponding outer fan35,36. The rotating wheel158is attached to the armature of a motor whose stator161is affixed to the housing144.

FIG. 15shows four of the fan-and-housing assemblies143arranged in the aforesaid over-and-under, fore-and-aft configuration. Each housing144comprises a sheet-metal box162whose −z and +y faces are open (as shown previously inFIG. 13), a top-hat-shaped flange163into which the stator161nestles and to which it is affixed, and a connector assembly164on whose +y face is located a male fan connector165that provides, via a local fan cable166, for connection of electrical power to the fan's motor, as well as connection of electrical signals from the fan's tachometer.

FIG. 16shows the upper air-moving assembly34, which in the prototype embodiment comprises, in addition to the four fan-and-housing assemblies40, a sheet-metal four-fan enclosure167whose surfaces facing the −z and +y directions are substantially open, as shown inFIG. 17, so as to minimize aerodynamic resistance that would impede intake airstreams150,152and exhaust airstreams154,156. Referring again toFIG. 16, the upper air-moving assembly34also comprises an array of four female fan connectors168that are attached to the four-fan enclosure167and which mate with the male-fan connectors165on the fan-and-housing assemblies143in order to provide the fans with electrical power and signals that originate from remote equipment (not shown), and are carried to the female fan connectors by remote fan cables169, which are retained and protected by surrounding skids170. A similar design pertains to the lower air-moving assembly39, which is just an upside-down replica of the upper air-moving assembly34.

A handle171is provided on each fan-and-housing assembly143to facilitate its removal from the upper air-moving assembly34should a fan fail. Removal of one of the outer fans35,36, which automatically disconnects its male fan connector165from the female fan connector168, is accomplished by releasing a latch172and pulling the handle171in the −y direction, as illustrated with regard to the outer right fan36inFIG. 18. This figure also reveals a support box173. There are two such boxes, as shown, whose function is to hold the outer fans35,36in position against the force of gravity while still allowing unimpeded flow of air in the z direction. In the prototype embodiment, removing one of the outer fans35,36(and one of the support boxes173) is prerequisite to removing one of the inner fans37,38; however, this is necessary only if the outer and inner fans overlap in the z direction to save space (as previously described in connection withFIG. 14). If there is no z overlap, then the outer and inner fans may be made independently removable, like conventional drawers.

It is important that the closed air loop23be reasonably tightly sealed to ensure that air will not leak from high-pressure areas to low-pressure areas, because such leaks would diminish the amount of cooling air that actually circulates in the closed air loop23. Sealing is particularly important in the vicinity of the top-rear-region148and bottom-front region149, because these areas are high-pressure plenums that readily leak air to the atmospheric pressure surrounding the enclosure1and to other low-pressure regions such as the fan intakes. For example, in the prototype embodiment, the positive pressure in the plenums is approximately 280 to 350 Pa (depending on conditions), and the pressure difference across the stack of front thermally neutral units is about 400 Pa. To prevent depressurization of the high-pressure plenums when the doors2of the enclosure are opened for access, a plenum box174, which is open on its −y and −z faces, encloses the top-rear region148, and another such box (shown inFIG. 19), which is open only on its +y and +z faces, encloses the bottom-front region149. In addition, in order to prevent aerodynamic short-circuiting of the fans, it is important that the high-pressure plenums be sealed off from the low-pressure fan intakes, by placing sealing means between the +z surface of the lower fan unit39and +y surface of the lowest front blade cage50.

With the prototype embodiment, it has been ascertained experimentally that a splitter plate175shown inFIG. 19, which bisects the plenum box174parallel to the yz plane, aids in distributing the air emerging from the lower fans more evenly in the plenum box, thereby sharing the air more evenly between the +x and −x halves of the enclosure1and leading to better thermal performance (lower maximum temperature) of air-cooled heat-producing components arranged in the enclosure. Without the splitter plate175, the air emerging from the fans tends to favor the −x side of the plenum box174, apparently due to the clockwise rotation of the fans inFIG. 19. The splitter plate175helps prevent too much air from flowing to the −x side of the plenum box by guiding the flow of air from the right-side fans to stay in the right half of the plenum box. The maximum air-cooled temperature in the prototype enclosure1was thereby reduced by about 6° C.

FIG. 20shows the topmost thermally neutral front unit28as well as the upper air-moving assembly34. To allow for failure of one of the fans35-38without overheating the air-cooled heat-producing components21, the preferred embodiment of the invention has a separation S between the +z face of the blade cage50and the −z a face of the four-fan enclosure167. It must be noted that when a fan such as38fails, in order to prevent aerodynamic short-circuiting of the other fans through the open aperture created by the failed fan, it is necessary that the failed-fan's exhaust area ABCD be sealed by a pivoting flat plate (not shown) that is hinged along BC and normally blown open by the airstream157, but which falls under the force of gravity or a spring when there is a cessation in the failed-fan's exhaust airstream157. Thus, the air that normally travels upward through the failed fan (airstream153) must be imparted an alternative route; otherwise, heat-producing components directly beneath the failed fan, in the blade cage50, will overheat. This alternative route is provided for by the separation S, which allows alternative air paths such as176through the other, still-functioning fans. LetT be the increase in worst-case temperature of heat-producing components due to a fan failure. In the prototype embodiment, experiments showT=32° C. with S=10 mm (the smallest value of S tested), but thatT=2° C. with S=35 mm. That is, with S=35 mm, the system and functioning thereof is virtually immune to the failure of a fan.

Referring toFIG. 21, alternative embodiments may use other techniques to allow for fan failure without sacrificing the vertical space represented by S. For example, even with S=0, an alternative path for the airstream normally handled by one of the outer fan35,36may be provided by permanent openings177in the side walls of the support boxes173that otherwise separate the intake airstreams of the two outer fans. Then, for example, if fan36fails, an alternative air path such as178is possible, which prevents heat-producing components beneath the failed fan from overheating. If one of the inner fans37,38fails, providing an alternative air path with S=0 is more difficult than for the outer fans, because, for example, the wall179between inner fan38and outer fan36seals the high-pressure exhaust of fan38from the low-pressure intake of fan36, and thus must not pass air under normal circumstances when all fans are running. However, it is possible to construct a louvered opening in wall179, whose louvers would open only when fan38fails.

Bulk Power Supply

FIG. 22illustrates the manner in which bulk power supplies, which convert electrical power from AC to DC, may be integrated into the enclosure1described by this invention. Such supplies, which ordinarily are 90% efficient and thus dissipate internally as heat about one-ninth (11%) of the power they deliver, are needed for typical heat-producing components, such as computer processor and memory chips. Off-the-shelf bulk-power-supply modules180are readily available that provide 4 kW of power in a package having xyz dimensions of about 127 mm×385 mm×127 mm.FIG. 22illustrates one embodiment of the invention in which twenty-four such power-supply modules (enough for an 80 kW rack with redundancy) are housed in their own power-supply enclosure181that abuts the −x face of the enclosure frame3. The power-supply enclosure181is aerodynamically isolated from the main enclosure1so that the closed loop of air23in the main enclosure1does not enter the power-supply enclosure181. The power-supply modules180are arranged in two stacks; i.e., a front stack182whose modules180are removable (for repair and replacement) from the −y face of the power-supply enclosure181; and a rear stack183whose modules180are similarly removable from the +y face of the power-supply enclosure. As shown, the front stack draws inlet cooling air184towards the +y direction; the rear stack draws inlet cooling air185toward the −y direction. Both stacks182,183exhaust cooling air186towards the +z direction through an aperture187in the power-supply enclosure. Thus, for such off-the-shelf power-supply modules180, the relatively small mount of heat dissipated in the power supplies is expelled to room air, and must be treated conventionally by external air-handling units. For example, if the power required in the enclosure1is 80 kW and the power supplies are 90% efficient, 8.9 kW is expelled to room air.

An alternative embodiment may employ custom bulk power-supply modules that admit bottom-to-top airflow, thereby allowing the power-supply modules to employ, lice the blade cages10,15, closed-loop liquid-assisted air cooling, in keeping with the inventive objective of eliminating the air-cooling burden on the machine room.