Abstract:
A cooling apparatus and method including a plurality of heat-producing devices positioned in a plurality of cabinets arranged in a row that allows flow of a first fluid through the heat-producing devices and cabinets where the flow is directed from an upstream end of the row to a downstream end of the row. The cabinets have a space therebetween wherein a heat exchanger is positioned between and adjacent to the cabinets, thereby the cabinets and heat exchangers alternate in the row. Each heat exchanger allows flow of a second fluid therethrough for cooling the first fluid. A fluid-moving device is positioned adjacent the heat-producing devices for encouraging flow of the first fluid through the cabinets&#39; heat-producing devices and through the heat exchangers, thereby encouraging heat transfer in each of the heat exchangers from the first fluid to the second fluid.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention is related to devices and methods for cooling heat-producing equipment, and more specifically, is related to devices for cooling heat-producing electronic equipment arranged in a row of cabinets. 
       BACKGROUND OF THE INVENTION 
       [0002]    Referring to  FIG. 1  and the Cartesian coordinate system which comprises an x axis  102 , a y axis  104 , and a z axis  106  that are mutually orthogonal, a known air-cooling apparatus  100 , described in US Patent  7 , 085 , 133 , which is incorporated by reference herein in its entirety, includes a row of cabinets  108 , including cabinets  110 ,  112 ,  114 ,  116  arrayed along the x axis  102 . The row of cabinets  108  includes a first cabinet  110  located at the +x end of the row and a last cabinet  116  located at the −x end of the row. An arbitrary number of additional interior cabinets, such as cabinets  112  and  114  shown in  FIG. 1 , are positioned between the first cabinet  110  and the last cabinet  116 . 
         [0003]    An intake end-plenum  118 , which includes a sloping wall  120 , abuts the row of cabinets  108  at an upstream face  110   a  of the first cabinet  110  to direct cooled air thereto. An exhaust end-plenum  122 , which includes a sloping wall  124 , is adjacent to a downstream face  116   b  of the last cabinet  116  to direct exhaust air therefrom. Interposed between each pair of adjacent cabinets is a combined-plenum unit  126  that comprises both an intake plenum  128  and an exhaust plenum  130 . Within each combined-plenum unit  126 , the intake plenum  128  and the exhaust plenum  130  are separated from each other by a sloping wall  132 . The combined plenum units  126  are mounted to the cabinets  110 ,  112 , and  114  such that the exhaust plenums  130  thereof abut the cabinets&#39; downstream surfaces  110   b,    112   b,  and  114   b  respectively, and the intake plenums  128  thereof abut the cabinets&#39; upstream surfaces  112   a,    114   a,  and  116   a,  respectively. Each cabinet  110 ,  112 ,  114 ,  116  contains heat-producing electronics  134  arranged to allow airflow parallel to the x direction  102 . Therefore, air-moving devices  136  in each cabinet are arranged to induce and encourage an S-shaped airflow  138 . This type of cooling means is used, for example, in IBM°&#39;s Bluegene®/L and Bluegene®/P supercomputers. The abutted row  108  of cabinets  110 ,  112 ,  114 ,  116  and plenums  118 ,  122 ,  126  stand in a room  140  on a raised floor  142  that is above and substantially parallel to a sub-floor  144 . The raised floor  142  typically comprises a regular two-dimensional array of removable tiles  146  having pitch p in the x  102  and y  104  directions. Cooling air  148  is supplied to an under-floor space  150  between the raised floor  142  and the sub-floor  144  by a plurality of air-conditioning units  152  that are also known in the art. 
         [0004]    Cooling one of the interior cabinets  112 ,  114  is accomplished by the S-shaped air-stream  138  passing through a hole  154  in the raised floor, and thereafter through the intake plenum  128 . Drawn by the air-moving devices  136 , the S-shaped air stream  138  travels over the heat-producing electronics  134 , exiting the cabinet through the exhaust plenum  130 . After the S-shaped air-stream  138  exits the exhaust plenum  130 , it is returned to an open top surface  156  of the air conditioning units  152 . Cooling of the first cabinet  110  or last cabinet  116  is similar to that for interior cabinets  114 , except that the air enters the first cabinet  110  through the intake end plenum  118 , and air exits the last cabinet  116  through the exhaust end plenum  122 . 
         [0005]    The known cooling apparatus  100  is deficient because it imposes at least the following several requirements on the room  140  and on the design of the cabinets  110 ,  112 ,  114 ,  116 . First, each cabinet must be fed by an airflow rate V sufficient to keep all the cabinet&#39;s internal electronics  134  sufficiently cool. For cabinets that dissipate large quantities of heat, this requirement is often burdensome on the infrastructure of the room  140  because it requires significant investment in air-conditioning units  152 , a large under-floor space  150 , and a disruption of airflow patterns to other, already-existing equipment in the room. 
         [0006]    Second, at the interface between any of the intake plenums  118 ,  128  and the abutting cabinets  110 ,  112 ,  114 ,  116  where the air-stream  138  first turns, the flow must be managed carefully, with appropriately designed turning aids, to avoid stagnation regions causing the electronics  134  to reach higher temperatures. This requirement is difficult to achieve in designing the cabinet, and despite best design efforts may be defeated by unusual raised-floor conditions, such as those where the distance between the raised floor  142  and the sub-floor  144  is too small, or where the hole  154  is partially obstructed by either structural members of the raised floor  142  or by equipment such as wires in under-floor space  150 . 
         [0007]    Third, in order to achieve high packing density of cabinets, the combined plenum unit  126  must be narrow. Thus, air must flow vertically through a relatively narrow intake plenum  128  and exhaust plenum  130 . This requirement inevitably incurs pressure loss, leading to reduced flow rate V and increased temperature of the electronics  134 . 
         [0008]    Fourth, holes  154  must be cut in the raised floor  142  underneath each of the intake plenums  118  and  128 . To avoid non-uniform flow leading to hotspots in the cabinet, the holes  154  must not be obstructed by structural members supporting the raised floor. Unobstructed holes are difficult to insure for all installations, because raised-floors are not standard worldwide, for example, the pitch p of the removable tiles  146  may differ from country to country. 
         [0009]    Therefore, a need exists for an improved cooling apparatus and method of cooling a row of cabinets  108  that houses electronic equipment  134 . It would be desirable, without sacrificing airflow through any particular item of the electronics  134 , for the cooling apparatus to operate with the least possible total airflow, thereby minimizing both the cost of air-conditioning equipment  152  and the level of acoustical noise in the room  140 . Further, it would be desirable to minimize constricted air passageways, such as the narrow plenums  128  and  130 , that unduly limit airflow. Moreover, it would be desirable to avoid turns in the airflow path, such as those in the S-shaped airflow path  138 , thereby to eliminate hotspots caused by flow non-uniformities and boundary-layer separation. Finally, it would be desirable to improve cabinet-packing density by minimizing the amount of space devoted exclusively to air handling, such as that occupied by plenums  118 ,  122 , and  126 . 
       SUMMARY OF THE INVENTION 
       [0010]    In an aspect of the invention, a cooling apparatus includes a plurality of heat-producing devices positioned in a plurality of cabinets arranged in a row allowing flow of a first fluid through the heat-producing devices and cabinets. The flow of the first fluid is directed from an upstream end of the row to a downstream end of the row such that an upstream heat-exchanger side abuts a downstream cabinet side the cabinets positioned in spaced relation to each other and defining a space therebetween. A plurality of heat exchangers are positioned at least partially in the spaces between the cabinets and adjacent to the cabinets. Thereby the cabinets and the heat exchangers alternate in the rows, each heat exchanger allowing flow of a second fluid therethrough for cooling the first fluid. At least one fluid-moving device positioned adjacent the heat-producing devices for encouraging the flow of the first fluid through the cabinets&#39; heat-producing devices and through the heat exchangers, thereby encouraging the transfer of heat from the first fluid to the second fluid in the heat exchangers. 
         [0011]    In a related aspect, at least one fluid-moving device is positioned between the heat-producing devices of each cabinet and the heat exchanger immediately downstream of the heat-producing device. 
         [0012]    In a related aspect, the apparatus further includes a first fluid-moving device positioned between the heat-producing device and the heat exchanger, and a second fluid-moving device is positioned between the heat exchanger and the cabinet immediately downstream of the heat exchanger. 
         [0013]    In a related aspect, the apparatus further includes a plurality of first fluid-moving devices positioned between the heat-producing devices and a plurality of heat exchangers, and a plurality of second fluid-moving devices each positioned between the heat exchangers and a front of the plurality of cabinets. In an embodiment of the apparatus, the first fluid may be air. Further, the heat-producing devices may be electronic devices, and further may be heat-producing devices such as computers or computer processors. 
         [0014]    In a related aspect, a plenum is positioned at an upstream side of a first cabinet of the plurality of cabinets for directing incoming ambient air. 
         [0015]    In a related aspect, a first plenum is positioned at an upstream side of a first cabinet of the plurality of cabinets for guiding the direction of incoming ambient air, and a second plenum is positioned at a downstream side of a last cabinet of the plurality of cabinets for guiding the direction of outgoing ambient air. 
         [0016]    In a related aspect, the second fluid is water. In another embodiment of the invention, the heat exchanger includes ingress and egress tubes carrying the second fluid, to remove heat from the first fluid. In another embodiment, the flow of the first fluid is directed in a closed loop. 
         [0017]    In a related aspect, the apparatus further includes a plurality of fluid-moving devices positioned adjacent an upstream side and a downstream side of the heat-producing devices for encouraging flow of the first fluid through the cabinets&#39; heat-producing devices and through the heat exchangers. 
         [0018]    In a related aspect, the apparatus further includes a vertical barrier dividing the cabinets into a front portion and a rear portion, and circulating the first fluid in a closed loop between the front and rear portions. Additionally, the apparatus may include a horizontal barrier dividing the cabinets into an upper portion and a lower portion, and circulating the first fluid in a closed loop between the upper and lower portions. 
         [0019]    In another aspect of the invention, a cooling system in an enclosed room includes a plurality of heat-producing devices positioned in a plurality of cabinets arranged in a row allowing a flow of a first fluid through the heat-producing devices and cabinets. The flow of the first fluid is directed from an upstream end of the row to a downstream end of the row, and the cabinets are positioned in spaced relation to each other and define a space therebetween. A plurality of heat exchangers are positioned at least partially in the spaces between the cabinets and adjacent to the cabinets. Thereby, the cabinets and the heat exchangers alternate in the rows such that an upstream heat-exchanger side abuts a downstream cabinet side, and each heat exchanger allows flow of a second fluid therethrough for cooling the first fluid. At least one fluid-moving device is positioned adjacent the heat-producing devices for encouraging the flow of the first fluid through the cabinets&#39; heat-producing devices and through the heat exchangers, thereby encouraging in each of the heat exchangers a transfer of heat from the first fluid to the second fluid. A first plenum adjacent an upstream side of a first cabinet for directing the flow of the first fluid as it enters the row of cabinets. A last plenum adjacent a downstream side of a last cabinet for directing the flow of the first fluid exiting the row of cabinets. 
         [0020]    In a related aspect, the first fluid is cycled in a closed loop within the enclosed room. In an alternative embodiment, the system further comprises a raised floor in the enclosed room, wherein the raised floor supports the plurality of cabinets, and the first fluid is directed through holes in the raised floor. In a further aspect, each of the heat exchangers provide, at its downstream side, a temperature of the first fluid that is substantially the same as the temperature of the first fluid when entering the upstream side of the first cabinet. 
         [0021]    In another aspect, a method for cooling includes: (a) positioning a plurality of heat-producing devices in a plurality of cabinets arranged in a row; (b) positioning a plurality of heat exchangers in a space between the cabinets and adjacent to the cabinets, thereby alternating the cabinets and the heat exchangers in the row; (c) directing flow of a first fluid through the heat-producing devices, cabinets, and heat exchangers for cooling the first fluid; and (d) positioning a plurality of fluid-moving devices adjacent the heat-producing devices for encouraging flow of the first fluid through the cabinets&#39; heat-producing devices and through the heat exchangers, thereby encouraging heat transfer from the first fluid to a second fluid in each of the heat exchangers. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0022]    These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings, in which: 
           [0023]      FIG. 1  is a front elevational view of a prior art cooling apparatus depicting a row of cabinets with interleaved airflow plenums; 
           [0024]      FIG. 2  is a front elevational view of a cooling apparatus according to an embodiment of the present invention depicting heat exchangers between cabinets in a row; 
           [0025]      FIG. 3  is a front elevational view of an apparatus according to another embodiment of the invention depicting differently arranged plenums; 
           [0026]      FIG. 4  is a front elevational view of an apparatus according to another embodiment of the invention without a plenum on the air-intake end of the row of cabinets; 
           [0027]      FIG. 5  is a front elevational view of an apparatus according to another embodiment of the invention without plenums at either the air-intake end or the air-exhaust end of the row of cabinets; 
           [0028]      FIG. 6  is a front elevational view of an apparatus according to another embodiment of the invention depicting differently arranged plenums; 
           [0029]      FIG. 7  is a front elevational view of an apparatus according to another embodiment of the invention depicting first and second air-moving devices; 
           [0030]      FIG. 8  is a plan view of an apparatus according to another embodiment of the invention depicting a vertical barrier for dividing the cabinets and heat exchangers into front and rear portons; and 
           [0031]      FIG. 9  is a front elevational view of an apparatus according to another embodiment of the invention depicting a horizontal barrier for dividing the cabinets and heat exchangers into upper and lower portions. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0032]    Referring to  FIG. 2 , an illustrative embodiment of a cooling apparatus  200  according to the present invention uses the same reference numerals for like elements as the prior art apparatus  100  shown in  FIG. 1 . However, the apparatus  200  differs from the prior art apparatus  100  in at least two significant ways. First, on the downstream faces of each cabinet  110 ,  112 ,  114 ,  116 , the present invention employs, in contrast to the prior art air plenums  126 ,  122 , a series of air-to-water heat exchangers  210 ,  212 ,  214 ,  216 . Second, the present invention uses, in place of the prior art&#39;s multiple S-shaped air paths  138 , a single, row-wise airflow path  218  that travels substantially in the −x direction, straight through an entire flow-through row  220 . The flow-through row  220  comprises the cabinets  110 ,  112 ,  114 ,  116 ; the heat exchangers  210 ,  212 ,  214 ,  216 , and optionally an intake plenum and an exhaust plenum such as a bottom-intake plenum  222 , and a bottom-exhaust plenum  224 , respectively. 
         [0033]    The heat exchangers  210 ,  212 ,  214 ,  216  make possible the row-wise airflow path  218 . Referring to the graph  244  of air temperature vs. horizontal coordinate x at the top of  FIG. 2 , the heat-producing electronics in cabinet  110  cause the temperature of the air circulating along air path  218  to rise from T 0  to T 1  as it traverses cabinet  110  from the cabinet&#39;s upstream face  110   a  at x=x 0  to the downstream face  110   b  at x=x 1 . The air-to-water heat exchanger  210  is typically a tube-and-fin heat exchanger well known in the art, wherein warm air passes over the heat-exchanger&#39;s fins and a cold liquid flows in the heat exchanger&#39;s tubes, thereby allowing heat to be transferred from the air to the liquid. The liquid is supplied to each heat exchanger from an external liquid-chilling system via a supply pipe  240 , and is returned to the liquid-chilling system via a return pipe  242 . Therefore, in traversing the heat exchanger  210  from x 1  to x 2 , the temperature of the air, being cooled by the externally chilled liquid, drops from T 1  to T 0 . Thus, the combination of cabinet  110  and heat exchanger  210  is thermally neutral for the air. This air-temperature cycle is repeated for subsequent cabinets and heat exchangers: the air is warmed to temperature T 1  a second time while traversing cabinet  112  in the region x 2  to x 3 , is cooled a second time to temperature T 0  by the heat exchanger  212  in the region x 3  to x 4 , is warmed a third time to temperature T 1  while traversing cabinet  114  in the region x 4  to x 5 , is cooled a third time to temperature T 0  by heat exchanger  214  in the region x 5  to x 6 , is warmed a fourth time to temperature T 1  by cabinet  116  in the region x 6  to x 7 , and is finally cooled a fourth time to temperature T 0  by heat exchanger  216  in the region x 7  to x 8 . Thus, the entire flow-through row  220  is thermally neutral for the air; that is, the air returns to the under-floor space  150  at temperature T 0 , ready to repeat the cycle. Because the air path  218  is closed, the temperatures T 0  and T 1  will automatically float to whatever values cause equilibrium to occur. Thus, it is necessary to choose heat exchangers  210 ,  212 ,  214 ,  216  and air-moving devices  136  such that acceptable temperatures are obtained for the worst-case heat dissipation of electronics  134 . Heat exchanges  210 ,  212 ,  214 ,  216  are described in U.S. patent application Ser. No. 11/939,165, filed Nov. 13, 2007, now abandoned, the disclosure of which is hereby incorporated herein by reference in its entirety. Temperature control of a cooling fluid is also discussed in copending U.S. patent application Ser. No. 12/483,542, filed Jun. 12, 2009, the disclosure of which is hereby incorporated herein by reference in its entirety. 
         [0034]    Again referring to  FIG. 2 , the row-wise airflow path  218  is now described in detail. Air enters the first cabinet  110  from the under-floor space  150 , flowing upward through row-intake hole  226  in the raised-floor  142 , and through the perforated metal screen  228 , which may be necessary, depending on the nature of the electronics, to prevent the escape of electromagnetic radiation therefrom into the room  140 . The row-wise airflow path  218  moves upward through the bottom-intake plenum  222  to the first cabinet  110  of the flow-through row  220 . The air-moving devices  136  within the cabinets  110 ,  112 ,  114 ,  116  encourage the row-wise airflow path  218  through each cabinet  110 ,  112 ,  114 ,  116 , and thereby through the entire flow-through row  220 . An intake-end wall  230  of the bottom-intake plenum  222  may, if desired, slant inward toward the top of the first cabinet  110 , inasmuch as upper cross-sections of the intake plenum  222  handle far less airflow than lower cross-sections, and thus require less cross-sectional area. Alternatively, the intake-end wall  230  may be substantially vertical, or removed altogether. In the latter case, the flow-through row  220  draws air from the room  140  rather than from the under-floor space  150 . 
         [0035]    The row-wise airflow path  218  exits the last cabinet  116  of the flow-through row  220 , flowing downward through a perforated-metal exhaust screen  232  whose function is similar to that of the perforated-metal intake screen  228 , downward through a row-exhaust hole  234  in the raised-floor  142 , and thereby into the under-floor space  150 . An exhaust-end wall  236  of the bottom-exhaust end plenum  224  may, if desired, slant outward toward the bottom of the last cabinet  116 , inasmuch as upper cross-sections of the bottom-exhaust plenum  224  handle far less airflow than lower cross-sections, and thus require less cross-sectional area. Alternatively, the exhaust-end wall  230  may be substantially vertical, or removed altogether. In the latter case, the flow-through row  220  exhausts air to the room  140  rather than to the under-floor space  150 . 
         [0036]    Referring to  FIG. 3 , another embodiment of the invention is a cooling apparatus  300  that includes a top-exhaust plenum  324  instead of the bottom-exhaust plenum  224  previously shown in  FIG. 2 . The top-exhaust plenum  324  is identical to bottom-exhaust plenum  224  except that it is rotated 180 degrees about the x axis, such that top-exhaust plenum  324  is wide at the top, by virtue of a sloping end wall  336 , thereby to accommodate greater airflow at upper cross sections than at lower cross sections In the cooling apparatus  300 , a row-wise airflow  318  behaves as in cooling apparatus  200 , except that in apparatus  300 , the airflow  318  exits the row  220  flowing upward through the top-exhaust end plenum  324 , which has an opening  334  at the top. A perforated metal exhaust screen  332  at the top of top-exhaust plenum  324  serves the same purpose as screen  232  in plenum  224 , as discussed previously. As with the apparatus  200  shown in  FIG. 2 , and also pertaining to the embodiments shown in  FIGS. 4 ,  6  and  7 , depending on the nature of the electronics  134 , it may not be necessary to include the perforated metal screen  332  to prevent the escape of electromagnetic radiation from the flow-through row  220 . 
         [0037]    Referring to  FIG. 4 , another alternative embodiment of the invention is a cooling apparatus  400 , where no intake plenum is used. In this embodiment, airflow  418  enters the flow-through row of cabinets  220  directly from the room  140 . The airflow exits the apparatus  400  as in the apparatus  300  shown in  FIG. 3 . Pertaining to this embodiment as well as to that shown on  FIG. 5 , to prevent the escape of electromagnetic radiation from the flow-through row  220 , it may be necessary, depending on the nature of the electronics  134 , to affix to the upstream surface  110   a  of the first cabinet  110  a perforated metal screen  428 , through which air flows immediately prior to entering cabinet  110 . 
         [0038]    Referring to  FIG. 5 , another alternative embodiment of the invention is a cooling apparatus  500  where no intake-end plenum or exhaust-end plenum is used. In this embodiment, airflow  518  exhausts from the last cabinet  116  directly to the room  140 . Airflow  518  is otherwise identical to airflow  418  discussed with reference to  FIG. 4 . To prevent the escape of electromagnetic radiation from the flow-through row  220 , it may be necessary, depending on the nature of the electronics  134 , to affix to the downstream surface  116   b  of the last cabinet  116  a perforated metal screen  532 . 
         [0039]    Referring to  FIG. 6 , another alternative embodiment of the invention is cooling apparatus  600 , where a top-intake end plenum  622  and the top-exhaust end plenum  324  are used. The top-intake plenum  622  is identical to the bottom-intake plenum  222 , shown in  FIG. 2 , except that it is rotated 180 degrees about the x axis, such that the top-intake plenum  622  is wide at the top, by virtue of a sloping end wall  630 , thereby to accommodate greater airflow at upper cross sections than at lower cross sections. In this embodiment, an airflow  618  enters the flow-through row  220  downward through the top-intake end plenum  622  and exits the flow-through row  220  upward through the top-exhaust end plenum  324 . 
         [0040]    Referring to  FIG. 7 , another embodiment of the invention is a cooling apparatus  700 , which is similar to the apparatus  200  shown in  FIG. 2 . However, in the apparatus  700  shown in  FIG. 7 , the heat-exchanger  210  is replaced by a heat-exchanger assembly  710  that comprises, in addition to the heat exchanger  210 , an array of air-moving devices  760 , such as axial-flow fans. Likewise, the heat exchangers  212 ,  214 , and  216  shown in  FIG. 2  are replaced, in apparatus  700 , by heat-exchanger assemblies  712 ,  714 ,  716  respectively, which comprise, in addition to heat exchangers  212 ,  214 , and  216  respectively, air-moving devices  762 ,  764 , and  766  respectively. Thus, the cooling apparatus  700  includes air-moving devices  760 ,  762 ,  764 ,  766  that supplement the air-moving devices  136  within the cabinets  110 ,  112 ,  114 ,  116 . Alternatively, depending, for example, on the cost and pressure-rise requirements of the cooling system and on the space required by the electronics, the air-moving devices  760 ,  762 ,  764 ,  766  may replace the air-moving devices  136  contained within the cabinets  110 ,  112 ,  114 ,  116 . 
         [0041]    The heat-exchanger assemblies  710 ,  712 ,  714 ,  716 , although described above for use with the airflow arrangement of the cooling apparatus  200  shown in  FIG. 2 , may also be used with any of the other airflow arrangements, as shown in cooling apparatuses  300 ,  400 ,  500 , and  600  of  FIGS. 3-6 , respectively. 
         [0042]    Referring to  FIG. 8 , another embodiment of the invention is a cooling apparatus  800 , wherein each of the cabinets  110 ,  112 ,  114 ,  116  is internally divided into a front portion  802  and a rear portion  804 . Note that  FIG. 8  is a plan view, as specified by the orientation of the x, y, and z axes  102 ,  104 ,  106  respectively, whereas  FIGS. 1-7  and  9  are front elevational views. In each cabinet, the portions  802 ,  804  are separated from each other by a vertical cabinet barrier  806  that substantially prevents air flow across it. The barrier  806  lies substantially parallel to an xz plane spanned by the x and z axes. Likewise, each of the heat-exchangers  210 ,  212 ,  214 ,  216  comprises, in this embodiment, a vertical heat-exchanger barrier  808  that substantially prevents airflow across it. The cabinet barriers  806  and the heat-exchanger barriers  808  are substantially co-planar. A first closed-end plenum  810  is abutted to the upstream face  110   a  of the first cabinet  110 , and a second closed-end plenum  812  is abutted to a downstream face  216   b  of the heat exchanger  216 . Front air-moving devices  814  in the front portion  802  of the cabinets  110 ,  112 ,  114 ,  116  are configured to drive a closed-horizontal-loop air-stream  818  in the −x direction, while rear air-moving devices  816  in the rear portion  804  of the cabinets  110 ,  112 ,  114 ,  116  are configured to drive the closed-horizontal-loop air stream  818  in the +x direction, such that the air stream  818  circulates in a closed loop about the vertical z axis  106 . That is, the closed-horizontal-loop air-stream  818  flows toward +x in the rear portion  804  of the cabinets  100 ,  112 ,  114 ,  116  and heat exchangers  210 ,  212 ,  214 ,  216 , then toward −y in the first closed-end plenum  810 , then toward −x in the front portion  802  of the cabinets and heat exchangers, and finally toward +y in the second closed-end plenum  812 , thus completing a closed loop. This closed-loop embodiment is advantageous because it imposes no air-handling burden on the room  140 , and because it provides very quiet operation of the air moving devices  814 ,  816 , particularly when the cabinets  110 ,  112 ,  114 ,  116 , heat-exchanger assemblies  210 ,  212 ,  214 ,  216 , and closed-end plenums  810 ,  812  are acoustically insulated, because people in the room  140  are shielded from the noise of air movers and flowing air. 
         [0043]    Again referring to the apparatus  800  shown in  FIG. 8 , it should be noted that the closed-horizontal-loop air stream  818 , at its +x end, traverses two sets of heat-producing electronics  134 , in the rear portion  804  of the first cabinet  110  and in the front portion  802  of the first cabinet  110 , without any intervening heat exchanger to cool the air. If this causes the air to become unacceptably warm in the front portion  802  of cabinet  110 , so as to compromise cooling of the electronics  134  therein, then an additional heat exchanger identical to  210  may be abutted to the +x surface of the first cabinet  110 . 
         [0044]    Referring to  FIG. 9 , another embodiment of the invention is a cooling apparatus  900 , wherein each of the cabinets  110 ,  112 ,  114 ,  116  is internally divided into a lower portion  902  and an upper portion  904 . In each cabinet, the portions  902 ,  904  are separated from each other by a horizontal cabinet barrier  906  that substantially prevents air flow across it. Barrier  906  lies substantially parallel to an xy plane spanned by the x and y axes Likewise, each of the heat-exchangers  210 ,  212 ,  214 ,  216  comprises, in this embodiment, a horizontal heat-exchanger barrier  908  that substantially prevents air flow across it. The cabinet barriers  906  and the heat-exchanger barriers  908  are substantially co-planar. A first closed-end plenum  910  is abutted to the upstream face  110   a  of the first cabinet  110 , and a second closed-end plenum  912  is abutted to a downstream face  216   b  of the heat exchanger  216 . Lower air-moving devices  914  in the lower portion  902  of the cabinets  110 ,  112 ,  114 ,  116  are configured to drive a closed-vertical-loop air-stream  918  in the −x direction, while upper air-moving devices  916  in the upper portion  904  of the cabinets  110 ,  112 ,  114 ,  116  are configured to drive the closed-vertical-loop air stream  918  in the +x direction, such that the air stream  918  circulates in a closed loop about the horizontal y axis  104 . More specifically, the closed-horizontal-loop air-stream  918  flows toward +x in the upper portion  904  of the cabinets  100 ,  112 ,  114 ,  116  and heat exchangers  210 ,  212 ,  214 ,  216 , then toward −z in the first closed-end plenum  810 , then toward −x in the lower portion  902  of the cabinets and heat exchangers, and finally toward +y in the second closed-end plenum  812 , thus completing a closed loop. This closed-loop embodiment, shown in  FIG. 9 , is advantageous for the same acoustic reason described earlier in connection with apparatus  800  shown in  FIG. 8 . 
         [0045]    Again referring to the apparatus  900  shown in  FIG. 9 , it should be noted that the closed-horizontal-loop air stream  918 , at its +x end, traverses two sets of heat-producing electronics  134 , in the upper portion  904  of the first cabinet  110  and in the lower portion  902  of the first cabinet  110 , without any intervening heat exchanger to cool the air. If this causes the air to become unacceptably warm in the lower portion  902  of cabinet  110  so as to compromise cooling of the electronics  134  therein, then an additional heat exchanger identical to  210  may be abutted to the +x surface of the first cabinet  110 . 
         [0046]    Additionally, other embodiments and variations are possible keeping with the spirit and scope of the invention, for example, although the embodiments presented herein have included “air-to-water heat exchangers”, the heat exchangers may use other fluids. In another example, the water supply and return pipes  240 ,  242  may enter the heat-exchangers  210 ,  212 ,  214 ,  216  from the top rather than from the bottom. 
         [0047]    All the embodiments of the current invention, including those represented as cooling apparatuses  200 ,  300 ,  400 ,  500 ,  600 ,  700 ,  800 , and  900 , shown in  FIGS. 2-9 , respectively, have a number of significant advantages over the prior-art apparatus  100  shown in  FIG. 1 , including those discussed hereinafter. A first advantage is that the total airflow required in the room  140 , and the associated acoustical noise, are greatly reduced by the invention vis-à-vis the prior art, leading to greater acoustical comfort for humans in the room  140 , and to less disruption of airflow if the room houses an existing installation of other equipment. Quantitatively, if volumetric flow rate V of air is required to cool each cabinet, and there are N cabinets in a row, then the prior art requires a total flow rate of NV per row, whereas the present invention which requires only V per row. This is a factor of N improvement that allows installation of such cabinets in buildings unable to support large amounts of airflow, and also reduces the total amount of airflow noise. 
         [0048]    Second, many fewer air-conditioning units  152  are required in the room  140  by the invention than by the prior art, leading to lower capital investment in air-conditioning units  152  and lower energy cost to drive air-moving devices therein. According to the invention, the heat load of electronics  134  is transferred from the air locally to water flowing in pipes  240 ,  242  of heat exchangers  210 ,  212 ,  214 ,  216 . Therefore, the flow-through row  220  puts no thermal load on the room  140 , and thus requires only minimal air-conditioning for general dehumidification, and ancillary heat loads. In contrast, the prior-art row  108  dissipates all its heat load to the room, thus requiring, if the number of cabinets and the power dissipation therein is large, a great number of air-conditioning units  152 . 
         [0049]    Third, the prior-art&#39;s narrow airflow plenums  126 , shown in  FIG. 1 , are eliminated. Such narrow plenums are required by the prior art to achieve compact packaging along the flow-through row  220 , and to insure that the holes  154  in the raised floor  142  match the periodicity p of the raised-floor tiles  146 . However, air velocity is high in the narrow airflow plenums  126 , typically much larger than in the cabinet itself, because the cross-sectional area normal to the airstream is much smaller in the plenum than in the cabinet. Thus pressure drop in the airflow plenums  126  is large, and airflow rate through the prior-art electronics  134  is thereby restricted, increasing the temperature therein and reducing the lifetime and performance thereof. In the invention, this source of pressure drop is eliminated. Some pressure loss occurs in the invention&#39;s heat exchangers  210 ,  212 ,  214 ,  216 , but because the cross-sectional area of the heat exchanger is large, air velocity is low, and therefore pressure drop is relatively small. 
         [0050]    Fourth, flow non-uniformities that occur in the prior art are eliminated. Specifically, the narrowness of the prior art&#39;s airflow plenums  126  cause flow separation at locations near the upstream faces  110   a,    112   a,    114   a,    116   a  of the cabinets wherever the airflow cannot negotiate a tight turn around a sharp edge. In the wake of such separation is a stagnation region of very-low-velocity airflow that causes very high temperatures of the electronics  134  therein. The tendency to separate may be minimized by widening the prior-art combined plenums  126 , but this is highly undesirable in the prior art, because of the desire to achieve a compact footprint of the row  108  of cabinets and plenums, and because of the aforementioned requirement to match the periodicity of the holes  154  with the pitch p of the removable tiles  146 . In contrast, embodiments  400  and  500  of the current invention require no air turn upstream of any electronics  134 , so the problem of flow separation is completely eliminated. All other embodiments require just one air turn per row  220 , upstream of the first cabinet  110 . Because the invention has only one intake plenum per row  220  rather than one intake plenum per cabinet as in the prior art, beneficial widening of the intake plenum, mentioned above, has, for the invention, much less impact on the footprint of a row  220  than a similar widening would have for the prior-art row  108 . That is, widening each of the prior-art&#39;s inlet plenums ( 118  and  128 ) by an amount d widens the prior-art cabinet row  108  by an amount Nd, where N is the number of cabinets per row. In contrast, widening the invention&#39;s intake end plenum ( 222  or  622 , depending on the embodiment) by the same amount d widens the invention&#39;s flow-through row  220  merely by d, a factor-of-N improvement over the prior art. 
         [0051]    Fifth, the prior art&#39;s need to turn the air twice in each cabinet  110 ,  112 ,  114 ,  116  is eliminated by the invention. By replacing the prior-art&#39;s S-shaped air-streams  138 , with the single, row-wise airflow path  218  most or all of the air turns are eliminated. Specifically, instead of two 90-degree turns per cabinet in the prior-art apparatus  100 , there are only four turns per row in apparatuses  200 ,  700 ,  800 , and  900 ; only two turns per row in apparatuses  300  and  600 ; only one turn per row in apparatus  400 ; and zero turns per row in apparatus  500 . Fewer turns is desirable because turning air incurs pressure drop and thereby reduces airflow, raising the temperature, shortening the life and compromising the performance of the electronics  134 . 
         [0052]    Sixth, compared to the prior art, the invention provides additional space for air-moving devices. As shown by apparatus  700  in  FIG. 7 , an air-to-water heat exchanger specified by this invention, such as  210 , need not occupy the entire space between the adjacent cabinets  110  and  112 ; instead, some of this space may be occupied by the array of air-moving devices  760 , which either supplement or replace the air-moving devices  136  internal to cabinet  110 . If air-moving devices  760 ,  762 ,  764 ,  766  supplement air-moving devices  136 , then the pressure rise of the system (and hence the air velocity) is greatly increased, a benefit that may be used either to reduce the temperature of the electronics, or to cool more electronics or more powerful electronics. If, instead, the air-moving devices  760 ,  762 ,  764 ,  766  replace air-moving devices  136 , then the space vacated by  136  may beneficially be used to house more electronics  134  in cabinet  110 . 
         [0053]    Seventh, the periodic, large airflow holes  154  in the raised floor  142  of the prior-art apparatus  100  are eliminated by this invention, thereby reducing the system&#39;s dependence on the pitch p of removable tiles  146  of the raised floor  142 . For example, in apparatus  200  shown in  FIG. 2 , pitch C of cabinets along a row, defined as C η x 8 -x 6  η x 6 -x 4  η x 4 -x 2  η x 2 -x 0 , is substantially unconstrained by the pitch p of the raised-floor tiles  95 , because the only holes therein are small holes for the supply and return pipes  240  and  242 . However, in the prior art, the holes  154  are large, and thus it is more important that the cabinet pitch C and the tile pitch p be more closely synchronized, to avoid interfering with struts that support the raised floor  142 . Toward this end, in the prior art, C and p are preferably related by a simple proportion such as mC=np where m and n are small integer such as (m, n)=(1,2) or (m, n)=(2,3). No such restriction applies to the invention. 
         [0054]    Eighth, redundancy of the air-moving devices  136  is improved by the invention vis-à-vis the prior art. Specifically, along a flow-through row of cabinets  220 , air-moving devices  136  sharing a common streamline back each other up, such that failure of a single air-moving device  136  is much less significant than for the prior-art&#39;s separate, S-shaped airstreams  138 , wherein failure of an air-moving device can cause the temperature of nearby electronics to rise. For apparatus  700 , similar redundancy is achieved for the supplementary, or alternative, series of air movers  762 ,  764 ,  766 ,  768 . 
         [0055]    Ninth, the invention improves cabinet-packing density vis-à-vis the prior art, thereby saving valuable floor space and also improving electrical-signaling performance between cabinets by allowing shorter cables. Specifically, the stream-wise (x) dimension of one of the heat exchangers assemblies  210 ,  212 ,  214 ,  216  is typically far smaller than the x dimension of one of the prior art&#39;s combined plenum units  126 , because the heat-exchanger&#39;s x dimension need only be large enough to accommodate tubes and fins to transfer heat from air to water, whereas the combined plenum unit&#39;s x dimension must be large enough to accommodate, through the intake plenum  128  and the exhaust plenum  130 , the large volumetric flow-rate of air, denoted V, that is needed to cool electronics  134 . For example, in the IBM® BlueGene/P® supercomputer, which comprises electronics  134  in each cabinet dissipating as much as 40 kW, and whose (x, y, z) cabinet dimensions are (70 cm, 89 cm, 180 cm), the x dimension of one of the heat exchangers  210 ,  212 ,  214 ,  216  need only be 10 cm, whereas the x dimension of the combined plenum unit  126  must be 52 cm in order to accommodate V=2.35 m3/s (5000 CFM). Thus, cooling BlueGene/P according to the current invention saves about 42 cm of width per cabinet, which is about 47% of the width of the cabinet itself. 
         [0056]    Thereby, the present invention clearly is advantageous for at least the reasons above in use with a supercomputer requiring rows of cabinets such as IBM®&#39;s BLUEGENE®, by the single stream of air flowing through a row of cabinets, passing alternately through cabinets and heat exchangers, instead of flowing air separately through each cabinet. 
         [0057]    While the present invention has been particularly shown and described with respect to preferred embodiments thereof, it will be understood by those skilled in the art that changes in forms and details may be made without departing from the spirit and scope of the present application. It is therefore intended that the present invention not be limited to the exact forms and details described and illustrated herein, but falls within the scope of the appended claims.