Patent Description:
Computer users often focus on the speed of computer microprocessors (e.g., megahertz and gigahertz). Many forget that this speed often comes with a cost-higher power consumption. This power consumption also generates heat. That is because, by simple laws of physics, all the power has to go somewhere, and that somewhere is, in the end, conversion into heat. A pair of microprocessors mounted on a single motherboard can draw hundreds of watts or more of power. Multiply that figure by several thousand (or tens of thousands) to account for the many computers in a large data center, and one can readily appreciate the amount of heat that can be generated. The effects of power consumed by the critical load in the data center are often compounded when one incorporates all of the ancillary equipment required to support the critical load.

Many techniques may be used to cool electronic devices (e.g., processors, memories, networking devices, and other heat generating devices) that are located on a server or network rack tray. For instance, forced convection may be created by providing a cooling airflow over the devices. Fans located near the devices, fans located in computer server rooms, and/or fans located in ductwork in fluid communication with the air surrounding the electronic devices, may force the cooling airflow over the tray containing the devices. In some instances, one or more components or devices on a server tray may be located in a difficult-to-cool area of the tray; for example, an area where forced convection is not particularly effective or not available.

The consequence of inadequate and/or insufficient cooling may be the failure of one or more electronic devices on the tray due to a temperature of the device exceeding a maximum rated temperature. While certain redundancies may be built into a computer data center, a server rack, and even individual trays, the failure of devices due to overheating can come at a great cost in terms of speed, efficiency, and expense.

<CIT> discloses a device and a system for cooling heat generating electronics. The system has a plurality of cold plates, which are adapted to transfer heat from a plurality of computer components to a cooling fluid. A supply line supplies the cooling fluid to and from these cold plates. The System also has a housing with one or more racks to support the computer component(s). The racks may also support the cold plates so that the cold plates are in thermal communication with at least one computer component.

<CIT> discloses a cooling apparatus and method for cooling one or more electronic components of an electronic subsystem of an electronics rack. The cooling apparatus includes a heat sink, which is configured to couple to an electronic component, and which includes a coolant-carrying channel for coolant to flow therethrough. The coolant provides two-phase cooling to the electronic component, and is discharged from the heat sink as coolant exhaust, which comprises coolant vapor to be condensed. The cooling apparatus further includes a node-level condensation module, associated with the electronic subsystem, and coupled in fluid communication with the heat sink to receive the coolant exhaust from the heat sink. A controller automatically controls the liquid-cooling of the heat sink and/or the liquid-cooling of the node-level condensation module. Documents <CIT> and <CIT> disclose further data center liquid cooling systems including control valves.

This disclosure describes a cooling system, for example, for rack mounted electronic devices (e.g., servers, processors, memory, networking devices or otherwise) in a data center. In various disclosed implementations, the cooling system includes a cold plate cooling system according to the present disclosure providing controllable liquid (e.g., water) cooling for server or network heat generating devices, such as processors, memory modules, switches, and other devices. The cold plate cooling system includes a tray-mounted control valve to meter a cooling liquid flow based on particular tray operating parameters and computer-implemented methods.

In an example implementation, a data center cooling system includes a server rack sub-assembly that includes at least one motherboard mounted on a support member and a plurality of heat generating electronic devices mounted on the at least one motherboard; at least one cold plate positioned in thermal communication with at least a portion of the plurality of heat generating electronic devices, the cold plate configured to receive a flow of a cooling liquid circulated through a supply conduit fluidly coupled to a liquid inlet of the at least one cold plate, through the at least one cold plate, and through a return conduit fluidly coupled to a liquid outlet of the at least one cold plate; and a modulating control valve attached to either of the motherboard or the support member and positioned in either of the supply conduit or the return conduit, the modulating control valve configured to adjust a rate of the flow of the cooling liquid based at least in part on an operating condition of at least one of the plurality of heat generating electronic devices.

In an aspect combinable with the example implementation, the at least one cold plate includes a plurality of cold plates.

In another aspect combinable with any one of the previous aspects, each of the plurality of cold plates is mounted to a respective heat generating electronic device of the plurality of heat generating electronic devices.

In another aspect combinable with any one of the previous aspects, the supply conduit is directly coupled to each liquid inlet of the plurality of cold plates, and the return conduit is directly coupled to each liquid outlet of the plurality of cold plates.

In another aspect combinable with any one of the previous aspects, the supply conduit is directly coupled to a liquid inlet of one of the plurality of cold plates, and the return conduit is directly coupled to a liquid outlet of one of the plurality of cold plates.

In another aspect combinable with any one of the previous aspects, the supply conduit is directly coupled to liquid inlets of a portion of the plurality of cold plates, and the return conduit is directly coupled to liquid outlets of a portion of the plurality of cold plates.

Another aspect combinable with any one of the previous aspects further includes a controller communicably coupled to the modulating control valve.

In another aspect combinable with any one of the previous aspects, the controller is attached to either of the motherboard or the support member.

In another aspect combinable with any one of the previous aspects, the controller is configured to perform operations including determining a value of the operating condition of the at least one heat generating electronic device; and adjusting the modulating control valve to open or close based on the determined value.

In another aspect combinable with any one of the previous aspects, determining the value of the operating condition of the at least one heat generating electronic device includes determining an operating temperature for each of the plurality heat generating electronic devices; determining a maximum operating temperature from the determined operating temperatures; and calculating a thermal margin between the maximum operating temperature and at least one of the operating temperatures.

In another aspect combinable with any one of the previous aspects, adjusting the modulating control valve to open or close based on the determined value includes adjusting the modulating control valve to open based on the calculated thermal margin greater than a threshold thermal margin value.

In another aspect combinable with any one of the previous aspects, determining the value of the operating condition of the at least one heat generating electronic device includes determining a supply cooling liquid temperature; determining a return cooling liquid temperature; and calculating a temperature difference between the supply cooling liquid temperature and the return cooling liquid temperature.

In another aspect combinable with any one of the previous aspects, adjusting the modulating control valve to open or close based on the determined value includes adjusting the modulating control valve to open based on the calculated temperature difference greater than a threshold temperature difference value.

In another aspect combinable with any one of the previous aspects, determining a supply cooling liquid temperature includes receiving a measured temperature value from an inlet temperature sensor in thermal communication with the flow of the cooling liquid through the supply conduit.

In another aspect combinable with any one of the previous aspects, determining a return cooling liquid temperature includes receiving a measured temperature value from an outlet temperature sensor in thermal communication with the flow of the cooling liquid through the return conduit.

In another example implementation, a method for cooling heat generating devices in a data center includes circulating a flow of a cooling liquid to a server rack sub-assembly that includes at least one motherboard mounted on a support member and a plurality of heat generating electronic devices mounted on the at least one motherboard; circulating the flow of the cooling liquid through a supply conduit to a liquid inlet of at least one cold plate positioned in thermal communication with at least a portion of the plurality of heat generating electronic devices; circulating the flow of the cooling liquid through the at least one cold plate to receive heat into the cooling liquid from the portion of the plurality of heat generating electronic devices; circulating the flow of the heated cooling liquid from a liquid outlet of the at least one cold plate through a return conduit; and controlling a modulating control valve attached to either of the motherboard or the support member to adjust a rate of the flow of the cooling liquid based at least in part on an operating condition of at least one of the plurality of heat generating electronic devices.

In an aspect combinable with the example implementation, the at least one cold plate includes a plurality of cold plates, each of the plurality of cold plates mounted to a respective heat generating electronic device of the plurality of heat generating electronic devices.

Another aspect combinable with any one of the previous aspects further includes circulating the flow of cooling liquid directly from the supply conduit to each liquid inlet of the plurality of cold plates; and circulating the flow of the heated cooling liquid directly from each liquid outlet of the plurality of cold plates to the return conduit.

Another aspect combinable with any one of the previous aspects further includes circulating the flow of cooling liquid directly from the supply conduit to a liquid inlet of one of the plurality of cold plates; heating the cooling liquid in the one of the plurality of cold plates; circulating the flow of the heated cooling liquid from a liquid outlet of the one of the plurality of cold plates to a liquid inlet of another of the plurality of cold plates; further heating the heated cooling liquid in the another of the plurality of cold plates; and circulating the flow of the further heated cooling liquid directly from a liquid outlet of the another of the plurality of cold plates to the return conduit.

Another aspect combinable with any one of the previous aspects further includes circulating the flow of cooling liquid directly from the supply conduit to liquid inlets of a first portion of the plurality of cold plates; heating the cooling liquid in the portion of the plurality of cold plates; circulating the flow of the heated cooling liquid from liquid outlets of the first portion of the plurality of cold plates to liquid inlets of a second portion of the plurality of cold plates; further heating the heated cooling liquid in the second portion of the plurality of cold plates; and circulating the flow of the further heated cooling liquid directly from liquid outlets of the second portion of the plurality of cold plates to the return conduit.

Another aspect combinable with any one of the previous aspects further includes determining a value of the operating condition of the at least one heat generating electronic device; and adjusting the modulating control valve to open or close based on the determined value.

Various implementations of a data center cooling system according to the present disclosure may include one, some, or all of the following features. For example, a server rack cold plate cooling system may allow for increased cooling (e.g., through an increase in a cooling fluid flow rate or decrease in cooling fluid temperature) for high power (or high density) servers or other data center electronic devices, while also allowing decreased cooling (e.g., through a decrease in a cooling fluid flow rate or increase in cooling fluid temperature) for low power (or low density) servers or other data center electronic devices. Thus, the server rack cold plate cooling system may provide cooling "on demand" for each specific server tray sub-assembly within a data center (which may be hundreds, thousands, tens of thousands or more). As another example, the server rack cold plate cooling system may provide for lower total time average cooling fluid flow usage and higher efficiency at the data center level. Further, the server rack cold plate cooling system may allow for short periods of time when power can be high by enabling high cooling fluid flow rate usage at such times. For example, peak server performance may be achievable without limiting the demands of the server rack. Thus, a data center may be more easily provisioned on average power load usage rather than peak (or nameplate, or worst case) power load usage.

According to the invention, a cold plate cooling system according to the present disclosure provides controllable liquid (e.g., water) cooling for server or network heat generating devices, such as processors, memory modules, switches, and other devices. The cold plate cooling system includes a tray-mounted control valve to meter a cooling liquid flow based on particular tray operating parameters and computer-implemented methods. For example, in some aspects, the control valve is embedded inside a cold plate loop, which is thermally coupled to the heat generating devices of a server tray (or server tray sub-assembly). There may be tens, hundreds, thousands, tens of thousands or more of such trays in a data center.

The cooling liquid (e.g., water) control may allow for scalability and a high level of modularity and allows each tray to function at a different liquid flow rate depending on the potentially unique configuration (or operating condition) of each tray. Thus, cold plate cooling systems of the present disclosure may accommodate variability in tray-to-tray power levels as well as variability of the cooling solution in terms of manufacturing variation and variability of assembly.

According to the invention, in operation of the cold plate cooling system, the tray-mounted control valve (or valves) can be adjusted between a minimum to a maximum level. In some aspects not being part of the invention, the minimum level is zero percent open, and in some aspects not being part of the invention, the maximum level is <NUM> percent open. A control algorithm executed by a control system determines or measures particular temperatures associated with the tray for which the cold plate cooling system is providing cooling. If heat generating device power increases, thereby increasing such temperatures, the control valve may be actuated to open more to reduce such temperatures to a target value (or values). In contrast, if heat generating device power decreases, thereby decreasing such temperatures, the control valve may be actuated to close more to allow such temperatures to rise to a target value (or values).

<FIG> illustrates an example system <NUM> that includes a server rack <NUM>, e.g., a <NUM> inch or <NUM> inch server rack, and multiple server rack sub-assemblies <NUM> mounted within the rack <NUM>. Although a single server rack <NUM> is illustrated, server rack <NUM> may be one of a number of server racks within the system <NUM>, which may include a server farm or a co-location facility that contains various rack mounted computer systems. Also, although multiple server rack sub-assemblies <NUM> are illustrated as mounted within the rack <NUM>, there might be only a single server rack sub-assembly. Generally, the server rack <NUM> defines multiple slots <NUM> that are arranged in an orderly and repeating fashion within the server rack <NUM>, and each slot <NUM> is a space in the rack into which a corresponding server rack sub-assembly <NUM> can be placed and removed. For example, the server rack sub-assembly can be supported on rails <NUM> that project from opposite sides of the rack <NUM>, and which can define the position of the slots <NUM>.

The slots <NUM>, and the server rack sub-assemblies <NUM>, can be oriented with the illustrated horizontal arrangement (with respect to gravity). Alternatively, the slots <NUM>, and the server rack sub-assemblies <NUM>, can be oriented vertically (with respect to gravity), although this would require some reconfiguration of the evaporator and condenser structures described below. Where the slots are oriented horizontally, they may be stacked vertically in the rack <NUM>, and where the slots are oriented vertically, they may be stacked horizontally in the rack <NUM>.

Server rack <NUM>, as part of a larger data center for instance, may provide data processing and storage capacity. In operation, a data center may be connected to a network, and may receive and respond to various requests from the network to retrieve, process, and/or store data. In operation, for example, the server rack <NUM> typically facilitates the communication of information over a network with user interfaces generated by web browser applications of users who request services provided by applications running on computers in the datacenter. For example, the server rack <NUM> may provide or help provide a user who is using a web browser to access web sites on the Internet or the World Wide Web.

The server rack sub-assembly <NUM> may be one of a variety of structures that can be mounted in a server rack. For example, in some implementations, the server rack sub-assembly <NUM> may be a "tray" or tray assembly that can be slidably inserted into the server rack <NUM>. The term "tray" is not limited to any particular arrangement, but instead applies to motherboard or other relatively flat structures appurtenant to a motherboard for supporting the motherboard in position in a rack structure. In some implementations, the server rack sub-assembly <NUM> may be a server chassis, or server container (e.g., server box). In some implementations, the server rack sub-assembly <NUM> may be a hard drive cage.

Referring to <FIG>, the server rack sub-assembly <NUM> includes a frame or cage <NUM>, a printed circuit board <NUM>, e.g., motherboard, supported on the frame <NUM>, one or more heat-generating electronic devices <NUM>, e.g., a processor or memory, mounted on the printed circuit board <NUM>, and one or more cold plates <NUM>. One or more fans (not shown) can also be mounted on the frame <NUM>.

The frame <NUM> can include or simply be a flat structure onto which the motherboard <NUM> can be placed and mounted, so that the frame <NUM> can be grasped by technicians for moving the motherboard into place and holding it in position within the rack <NUM>. For example, the server rack sub-assembly <NUM> may be mounted horizontally in the server rack <NUM> such as by sliding the frame <NUM> into the slot <NUM> and over a pair of rails in the rack <NUM> on opposed sides of the server rack sub-assembly <NUM> - much like sliding a lunch tray into a cafeteria rack. Although <FIG> illustrates the frame <NUM> extending below the motherboard <NUM>, the frame can have other forms (e.g., by implementing it as a peripheral frame around the motherboard) or may be eliminated so that the motherboard itself is located in, e.g., slidably engages, the rack <NUM>. In addition, although <FIG> illustrates the frame <NUM> as a flat plate, the frame <NUM> can include one or more side walls that project upwardly from the edges of the flat plate, and the flat plate could be the floor of a closed-top or open-top box or cage.

The illustrated server rack sub-assembly <NUM> includes a printed circuit board <NUM>, e.g., a motherboard, on which a variety of components are mounted, including heat-generating electronic devices <NUM>. Although one motherboard <NUM> is illustrated as mounted on the frame <NUM>, multiple motherboards may be mounted on the frame <NUM>, depending on the needs of the particular application. In some implementations, the one or more fans (not shown) can be placed on the frame <NUM> so that air enters at the front edge (at the left hand side in <FIG>) of the server rack sub-assembly <NUM>, closer to the front of the rack <NUM> when the sub-assembly <NUM> is installed in the rack <NUM>, flows (as illustrated) over the motherboard <NUM>, over some of the heat generating components on the motherboard <NUM>, and is exhausted from the server rack assembly <NUM> at the back edge (at the right hand side), closer to the back of the rack <NUM> when the sub-assembly <NUM> is installed in the rack <NUM>. The one or more fans can be secured to the frame <NUM> by brackets. Thus, the fans can pull air from within the frame <NUM> area and push the air after it has been warmed out the rack <NUM>. An underside of the motherboard <NUM> can be separated from the frame <NUM> by a gap.

As shown, each cold plate heat exchanger (or "cold plate") <NUM> contacts the electronic device <NUM> so that heat is drawn by conductive heat transfer from the electronic device <NUM> to the cold plate <NUM>. For example, the cold plate <NUM> is in conductive thermal contact with the electronic device <NUM>. In particular, the bottom of the cold plate <NUM> contacts the top of the electronic device <NUM>. In operation, a working liquid <NUM>, such as water, glycol, or another cooling liquid, is circulated (e.g., pumped) from a cooling liquid source (not shown) through a supply conduit <NUM> to each cold plate <NUM>. Heat from the electronic device <NUM> causes the working liquid <NUM> in the cold plate <NUM> to rise in temperature. The heated liquid <NUM> then passes through return conduit <NUM>, through control valve <NUM> to the cooling liquid source. The cooling liquid source may be a mechanical refrigeration cooling source or an evaporative cooling source, for example. Thus, the cooling liquid <NUM> may remove heat from the electronic device <NUM>.

As shown, the control valve <NUM> (positioned in the return conduit <NUM> in this example) is mounted on the server tray sub-assembly <NUM>, and in this example, on the frame <NUM>. In alternative examples, the control valve <NUM>, which may be a plug or ball valve as examples, may be mounted on another component of the server tray sub-assembly <NUM>, such as the mother board <NUM>.

As further shown in this example, the control valve <NUM> is communicably coupled (e.g., through a wired or wireless connection) to a controller <NUM>. More specifically, a valve actuator (not shown) that is part of or attached to the control valve <NUM> may be communicably coupled to the control <NUM> to receive commands (e.g., to open or close) from the controller <NUM>. In example implementations, the controller <NUM> is a proportional integral derivative (PID) controller. But in alternative implementations, the controller <NUM> may be another form of a microprocessor-based controller or even a mechanical, electro-mechanical, pneumatic, or hydraulic controller.

In some aspects, the controller <NUM> may also be communicably coupled to other components on the server rack sub-assembly <NUM> that are not shown, such as for example, temperature sensors, pressure sensors, or the electronic devices <NUM> themselves. For example, the controller <NUM> may be communicably coupled to temperature sensors in thermal communication with the electronic devices <NUM> so as to measure or determine an operating temperature of the respective electronic devices <NUM>. In some aspects, such temperature sensors may be mounted on the presumed hottest portion of the respective electronic devices <NUM>. In addition or alternatively, the controller <NUM> may be communicably coupled to temperature sensors positioned on the sub-assembly <NUM> to measure operating temperature of the sub-assembly <NUM>, generally.

In some aspects, the controller <NUM> may be in communication with the electronic devices <NUM> to determine or measure, for example, operating power, operating utilization, or both. For example, in some aspects, heat generated by the electronic devices <NUM> may be related (directly or otherwise) to an amount of electric power consumed by the devices <NUM>. Further, heat generated by the electronic devices <NUM> may be related (directly or otherwise) to a utilization percentage of the devices <NUM>. Thus, by determining or measuring consumed electrical power or utilization of the electronic devices <NUM>, an amount of heat (and thus temperature) may be at least indirectly determined by the controller <NUM>.

<FIG> illustrate schematic top views of example implementations of a server rack sub-assembly <NUM> that includes example implementations of a cold plate cooling system. In the example implementations of <FIG>, the cold plate cooling system includes, for instance, one or more cold plate heat exchangers, or cold plates, <NUM> that are fluidly coupled to the control valve <NUM> that is communicably coupled to the controller <NUM>. In these examples, the control valve <NUM> is mounted to the server tray sub-assembly <NUM> and, more specifically, the frame <NUM>. While four cold plates <NUM> are shown in these examples, there may be more or fewer as may be dictated, for instance, by a number of processors (e.g., electronic devices <NUM>) mounted on the server tray sub-assembly <NUM>. Thus, in some aspects, there may be a one-to-one ratio of processors (e.g., electronic devices <NUM>) to cold plates <NUM>. In alternative examples, there may be a different ratio of processors (e.g., electronic devices <NUM>) to cold plates <NUM>. Further, in this example, there is a single control valve <NUM> for the multiple cold plates <NUM> (e.g., a <NUM>:<NUM> ratio); alternatively, there may be other arrangements in which there may be a <NUM>:<NUM> ratio, <NUM>:<NUM> ratio, or other ratio of control valves <NUM> to cold plates <NUM>. In that manner, variability in intra-tray power levels of the electronic devices <NUM> can be accommodated and each electronic device <NUM> can be cooled at a different liquid flow rate.

As shown in these example, a supply conduit <NUM> is fluidly coupled to inlets of the cold plates <NUM> while a return conduit <NUM> is fluidly coupled to outlets of the cold plates <NUM>. Generally, the supply conduit <NUM> is fluidly coupled to a supply side of a cooling liquid source (not shown), such as, for example, a chiller, evaporative cooling source, or otherwise, while the return conduit <NUM> is fluidly coupled to a return side of the cooling liquid source. In this example, the control valve <NUM> is fluidly coupled within the return conduit <NUM>; alternatively, the control valve <NUM> is fluidly coupled within the supply conduit <NUM>.

In some aspects, the controller <NUM> may be communicably coupled to temperature sensors (not shown) that are on or in thermal contact with the electronic devices <NUM> on which the cold plates <NUM> are mounted. Such temperature sensors may measure operating temperature values of each of the electronic devices <NUM>.

Each of <FIG> show an example arrangement of the cold plates <NUM> with respect to how they are fluidly coupled within the cold plate cooling system. For example, turning to <FIG> specifically, this figure shows an example implementation of the cold plate cooling system in which the cold plates <NUM> are fluidly connected in parallel. Thus, as shown, each of the cold plates <NUM> is directly coupled to the supply conduit <NUM> (at inlets of the respective cold plates <NUM>). Further, each of the cold plates <NUM> is directly coupled to the return conduit <NUM> (at outlets of the respective cold plates <NUM>). In this example, since each cold plate <NUM> is directly coupled to the supply conduit <NUM> (e.g., there is not another cold plate <NUM> fluidly connected between the supply conduit <NUM> and each other cold plate <NUM>), a temperature of a cooling liquid supply at inlets of each cold plate <NUM> is identical or substantially the same (e.g., accounting for potential temperature differences at inlets caused by heat gain in the supply conduit <NUM>). While each cold plate <NUM> may, therefore, receive the cooling liquid at an identical or similar temperature, a temperature of the cooling liquid at outlets of the cold plates <NUM> may vary, e.g., due to different heat loads (e.g., caused by different power usages) of each of the electronic devices <NUM> thermally coupled to the respective cold plates <NUM>.

Turning to <FIG>, this figure shows an example implementation of the cold plate cooling system in which a pair of cold plates <NUM> are fluidly connected in series, and each pair (two shown) is fluidly connected in parallel with another pair. Thus, as shown, two of the four cold plates <NUM> (labeled cold plates 132a) are directly coupled to the supply conduit <NUM> (at inlets of the respective cold plates 132a). An inlet of each of the other two cold plates <NUM> (labeled cold plates 132b) is fluidly coupled to an outlet of one of the two directly coupled cold plates 132a; thus, each of the other two cold plates 132b are indirectly coupled to the supply conduit <NUM> through the cold plates 132a. Each of the cold plates 132b is directly coupled to the return conduit <NUM> (at outlets of the respective cold plates 132b). In this example, since each cold plate 132a is directly coupled to the supply conduit <NUM> (e.g., there is not another cold plate <NUM> fluidly connected between the supply conduit <NUM> and each cold plate 132a), a temperature of a cooling liquid supply at inlets of each cold plate 132a is identical or substantially the same (e.g., accounting for potential temperature differences at inlets caused by heat gain in the supply conduit <NUM>). However, a temperature of the cooling liquid supply at inlets of each cold plate 132b may be higher than the temperature of the cooling liquid supply at inlets of the cold plates 132a due to heat transferred in the cold plates 132a to the cooling liquid. Further, in some aspects, a temperature of the cooling liquid supply at inlets of the respective cold plates 132b may be different due to differences in amount of heat transferred in the cold plates 132a to the cooling liquid. Thus, temperature of the cooling liquid at outlets of the cold plates 132a and 132b may vary, e.g., due to different heat loads (e.g., caused by different power usages) of each of the electronic devices <NUM> thermally coupled to the respective cold plates <NUM> (cold plates 132a and 132b).

Turning to <FIG>, this figure shows an example implementation of the cold plate cooling system in which the cold plates <NUM> are fluidly connected in series. Thus, as shown, one of the four cold plates <NUM> (labeled cold plate 132a) is directly coupled to the supply conduit <NUM> (at an inlet of the cold plate 132a). The outlet of cold plate 132a is directly coupled to the inlet of the cold plate 132b; he outlet of cold plate 132b is directly coupled to the inlet of the cold plate 132c; and the outlet of cold plate 132c is directly coupled to the inlet of the cold plate 132d. In this example, since only cold plate 132a is directly coupled to the supply conduit <NUM> (e.g., there is not another cold plate <NUM> fluidly connected between the supply conduit <NUM> and the cold plate 132a), a temperature of a cooling liquid supply at the inlet of the cold plate 132a is likely to be less than a temperature of the cooling liquid supply at the inlet of cold plate 132b. Likewise, a temperature of the cooling liquid supply at the inlet of the cold plate 132b is likely to be less than a temperature of the cooling liquid supply at the inlet of cold plate 132c, and a temperature of the cooling liquid supply at the inlet of the cold plate 132c is likely to be less than a temperature of the cooling liquid supply at the inlet of cold plate 132d (e.g., due to heat gained into the cooling liquid at each of the respective cold plates 132a, 132b, and 132c).

<FIG> are graphs that illustrate relationships between operating variables of the cold plate cooling system of one or more of <FIG>. For example, graph <NUM> shown in <FIG> shows a relationship between device temperature on a y-axis <NUM> and power dissipated into a cold plate on x-axis <NUM>. Thus, as shown, curve <NUM> shows the relationship between how hot an electronic device <NUM> is relative to the heat power from the electronic device <NUM> dissipated into the cold plate <NUM> that is mounted to the device <NUM>. As shown in graph <NUM>, there is a linear relationship (e.g., for constant cooling flow rate through the cold plate <NUM>) between device temperature and heat dissipated into the cold plate <NUM>.

Turning to <FIG>, this figure includes graph <NUM>, which shows a relationship between a hottest device temperature (e.g., the hottest electronic device <NUM> on the server tray sub-assembly <NUM>) on a y-axis <NUM> and cooling liquid flow rate through a cold plate on x-axis <NUM>. Thus, as shown, curve <NUM> shows the relationship between the temperature of the hottest electronic device <NUM> on the sub-assembly <NUM> relative to how much (e.g., volumetric flow per unit time) cooling liquid flows through the cold plate <NUM> that is mounted to the hottest device <NUM>. As shown in graph <NUM>, there is a non-linear relationship between hottest device temperature and cooling liquid flow rate through the cold plate <NUM>. In some aspects, graph <NUM> illustrates how a known relationship of temperature (e.g., of the hottest electronic device <NUM>) and flow rate (e.g., of the cooling liquid flow rate) may be characterized and functionally described (e.g., temperature depends on flow rate and a change of flow rate with the valve <NUM> changes temperature).

Turning to <FIG>, this figure includes graph <NUM>, which shows a relationship between cooling liquid flow rate through the cold plate cooling system on a y-axis <NUM> and control valve open percentage on x-axis <NUM>. Thus, as shown, curve <NUM> shows the relationship between the liquid flow rate relative to the open state (e.g., percentage) of the control valve <NUM>. As shown in graph <NUM>, there is a non-linear relationship between flow rate and control valve open percentage. In some aspects, graph <NUM> illustrates the direct relationship between valve opening percentage and flow rate through the valve, such that, e.g., if the valve maintains a minimum open percentage, then transient heat increases may be handled with little change in the opening percentage above the minimum.

<FIG> is a flowchart that illustrates an example method <NUM> of operating a cold plate cooling system of one or more of <FIG>. For example, in some aspects, method <NUM> may illustrate an example method for operating one or more of the cold plate cooling systems of <FIG> according to, for instance, one or more measured temperature values of one or more of the electronic devices <NUM> of the server tray sub-assembly <NUM>. In some aspects, method <NUM> may be performed by or with the controller <NUM>. Method <NUM> may begin at step <NUM>, which includes measuring one or more temperatures of heat generating devices, such as the electronic devices <NUM>. For example, in some aspects, the controller <NUM> is communicably coupled to the electronic devices <NUM> to receive temperature values, or to temperature sensors in thermal communication with the electronic devices <NUM>.

Method <NUM> may continue at step <NUM>, which includes determining a hottest heat generating device on a server tray, such as the server rack sub-assembly <NUM>. For example, upon polling each of the electronic devices <NUM> for operational temperature values (or temperature sensors thermally coupled to the electronic devices <NUM>), the controller <NUM> may determine the highest operating temperature, e.g., the hottest electronic device <NUM>.

Method <NUM> may continue at step <NUM>, which includes calculating a thermal margin between a temperature of the hottest heat generating device and at least one of the other measured temperatures (e.g., in step <NUM>). For example, the controller <NUM> may calculate the thermal margin by calculating the difference between the temperature of the hottest electronic device <NUM> and, for example, the temperature of the coldest electronic device <NUM> on the server tray sub-assembly <NUM>. Alternatively, the controller <NUM> may calculate the thermal margin by calculating the difference between the temperature of the hottest electronic device <NUM> and, for example, an average temperature of all of the electronic devices <NUM> on the server tray sub-assembly <NUM>. In some aspects, there is an actual thermal margin (ATM) (e.g., based on actual temperatures being measured) and a target or threshold thermal margin (TTM) (e.g., which is a setpoint or specified value).

Method <NUM> may continue at step <NUM>, which includes a determination of whether the calculated thermal margin meets a threshold thermal margin value. For example, after the thermal margin is calculated by the controller <NUM>, the controller <NUM> may compare the calculated thermal margin to a pre-determined or otherwise programmed thermal margin setpoint (or threshold). For example, an example thermal margin setpoint may be <NUM>. If the calculated thermal margin is less than the setpoint (e.g., the electronic devices <NUM> are being adequately cooled), the controller <NUM> determines that the decision in step <NUM> is "yes" and method <NUM> may end or repeat at step <NUM>.

If the calculated thermal margin is greater than the setpoint (e.g., the electronic devices <NUM> are not being adequately cooled), the controller <NUM> determines that the decision in step <NUM> is "no" and method <NUM> may continue at step <NUM>, which includes adjusting a position (e.g., a percentage open) of a control valve. For example, the controller <NUM> may actuate the control valve <NUM> to open more than a current position of the valve <NUM> to, e.g., allow a greater flow rate of the cooling liquid to flow through the cold plate cooling system to increase a heat dissipation rate of the electronic devices <NUM> into the cold plates <NUM>. Step <NUM> may continue, for example, back to step <NUM> after the control valve <NUM> is adjusted.

<FIG> illustrates a schematic top view of another example implementation of a server rack sub-assembly that includes an example implementation of a cold plate cooling system. This figure shows an example implementation of the cold plate cooling system in which the cold plates <NUM> are fluidly connected in parallel (similar to <FIG>). Thus, as shown, each of the cold plates <NUM> is directly coupled to the supply conduit <NUM> (at inlets of the respective cold plates <NUM>). Further, each of the cold plates <NUM> is directly coupled to the return conduit <NUM> (at outlets of the respective cold plates <NUM>). In this example, since each cold plate <NUM> is directly coupled to the supply conduit <NUM> (e.g., there is not another cold plate <NUM> fluidly connected between the supply conduit <NUM> and each other cold plate <NUM>), a temperature of a cooling liquid supply at inlets of each cold plate <NUM> is identical or substantially the same (e.g., accounting for potential temperature differences at inlets caused by heat gain in the supply conduit <NUM>). While each cold plate <NUM> may, therefore, receive the cooling liquid at an identical or similar temperature, a temperature of the cooling liquid at outlets of the cold plates <NUM> may vary, e.g., due to different heat loads (e.g., caused by different power usages) of each of the electronic devices <NUM> thermally coupled to the respective cold plates <NUM>.

In this example, a temperature sensor <NUM> is mounted in or to the supply conduit <NUM> (or in thermal communication with the supply conduit <NUM>). Also, as shown, a temperature sensor <NUM> is mounted in or to the return conduit <NUM> (or in thermal communication with the return conduit <NUM>). Thus, temperature sensor <NUM> can measure a temperature of the cooling liquid in the supply conduit <NUM>, while the temperature sensor <NUM> can measure a temperature of the cooling liquid in the return conduit <NUM>.

These temperature sensors <NUM> and <NUM> may be communicably coupled to the controller <NUM>. Thus, the cold plate cooling system shown in this example of <FIG> may be controlled based at least partly on one or both of the temperature values of the cooling liquid provided to the controller by temperature sensors <NUM> and <NUM> (e.g., cooling liquid temperature control). Turning to <FIG>, this figure shows a flowchart that illustrates an example method <NUM> of operating the cold plate cooling system of <FIG>. For example, in some aspects, method <NUM> may illustrate an example method for operating the cold plate cooling system of <FIG> according to, for instance, the one or more measured temperature values of the cooling liquid used to cool the electronic devices <NUM>. In some aspects, method <NUM> may be performed by or with the controller <NUM>. Method <NUM> may also be used (along with temperature sensors <NUM> and <NUM>) to operate the example cold plate cooling systems of <FIG> as well. Method <NUM> may begin at step <NUM>, which includes measuring an inlet cooling liquid temperature (Tin). For example, the controller <NUM> may poll (or receive from) the temperature sensor <NUM> a temperature of the cooling liquid that is flowing through the supply conduit <NUM>.

Method <NUM> may continue at step <NUM>, which includes measuring an outlet cooling liquid temperature (Tout). For example, the controller <NUM> may poll (or receive from) the temperature sensor <NUM> a temperature of the cooling liquid that is flowing through the return conduit <NUM>. Steps <NUM> and <NUM> may be performed in an alternative order, or may be performed simultaneously as well.

Method <NUM> may continue at step <NUM>, which includes calculating a temperature delta between Tin and Tout (Tdelta = Tout-Tin). For example, the controller <NUM> may calculate Tdelta at a particular time of operation of the cold plate cooling system.

Method <NUM> may continue at step <NUM>, which includes a determination of whether the calculated temperature delta meets a threshold temperature delta value. For example, after the Tdelta is calculated by the controller <NUM>, the controller <NUM> may compare the calculated Tdelta to a pre-determined or otherwise programmed Tdelta setpoint (or threshold). For example, an example Tdelta setpoint may be <NUM>. If the calculated Tdelta is less than the setpoint (e.g., the electronic devices <NUM> are being adequately cooled), the controller <NUM> determines that the decision in step <NUM> is "yes" and method <NUM> may end or repeat at step <NUM>.

If the calculated Tdelta is greater than the setpoint (e.g., the electronic devices <NUM> are not being adequately cooled), the controller <NUM> determines that the decision in step <NUM> is "no" and method <NUM> may continue at step <NUM>, which includes adjusting a position (e.g., a percentage open) of a control valve. For example, the controller <NUM> may actuate the control valve <NUM> to open more than a current position of the valve <NUM> to, e.g., allow a greater flow rate of the cooling liquid to flow through the cold plate cooling system to increase a heat dissipation rate of the electronic devices <NUM> into the cold plates <NUM>. Step <NUM> may continue, for example, back to step <NUM> after the control valve <NUM> is adjusted.

<FIG> is a graph that illustrates a relationship between operating variables of the cold plate cooling system of <FIG>. This figure includes graph <NUM>, which shows a relationship between Tdelta on a y-axis <NUM> and heat power dissipated on the server tray sub-assembly <NUM> on x-axis <NUM>. Thus, as shown, curve <NUM> shows the relationship between the Tdelta relative to how much heat is removed from the electronic devices <NUM> on the sub-assembly <NUM> (e.g., with the cold plate cooling system of <FIG>). As shown in graph <NUM>, there is a linear relationship between Tdelta and heat dissipation from the electronic devices <NUM> such that the greater the Tdelta (e.g., greater the actual thermal margin), the more power is dissipated in the electronic device <NUM>.

<FIG> is a flowchart that illustrates inventive method <NUM> of operating a cold plate cooling system of the present disclosure. Method <NUM> illustrates method for operating the cold plate cooling system of any one of <FIG> (or <FIG>) according to the one or more measured temperature values of the cooling liquid used to cool the electronic devices <NUM> as well as percent open constraints on the control valve <NUM>. Method <NUM> is performed by or with the controller <NUM>. Method <NUM> begins at step <NUM>, which includes measuring one or more temperatures of heat generating devices, such as the electronic devices <NUM>. For example, in some aspects, the controller <NUM> is communicably coupled to the electronic devices <NUM> to receive temperature values, or to temperature sensors in thermal communication with the electronic devices <NUM>.

Method <NUM> continues at step <NUM>, which includes determining a hottest heat generating device on a server tray, such as the server rack sub-assembly <NUM>. Upon polling each of the electronic devices <NUM> for operational temperature values (or temperature sensors thermally coupled to the electronic devices <NUM>), the controller <NUM> determines the highest operating temperature, e.g., the hottest electronic device <NUM>.

Method <NUM> continues at step <NUM>, which includes calculating a thermal margin between a temperature of the hottest heat generating device and at least one of the other measured temperatures (e.g., in step <NUM>). For example, the controller <NUM> may calculate the thermal margin by calculating the difference between the temperature of the hottest electronic device <NUM> and, for example, the temperature of the coldest electronic device <NUM> on the server tray sub-assembly <NUM>. Alternatively, the controller <NUM> may calculate the thermal margin by calculating the difference between the temperature of the hottest electronic device <NUM> and, for example, an average temperature of all of the electronic devices <NUM> on the server tray sub-assembly <NUM>.

Method <NUM> may continue at step <NUM>, which includes a determination of whether the calculated thermal margin meets a threshold thermal margin value. For example, after the thermal margin is calculated by the controller <NUM>, the controller <NUM> may compare the calculated thermal margin to a pre-determined or otherwise programmed thermal margin setpoint (or threshold). For example, an example thermal margin setpoint may be <NUM>. If the calculated thermal margin is less than the setpoint (e.g., the electronic devices <NUM> are being adequately cooled), the controller <NUM> determines that the decision in step <NUM> is "yes" and method <NUM> may repeat at step <NUM>.

If the calculated thermal margin is greater than the setpoint (e.g., the electronic devices <NUM> are not being adequately cooled), the controller <NUM> determines that the decision in step <NUM> is "no" and method <NUM> may continue at step <NUM>, which includes adjusting a position (e.g., a percentage open) of a control valve. For example, the controller <NUM> may actuate the control valve <NUM> to open more than a current position of the valve <NUM> to, e.g., allow a greater flow rate of the cooling liquid to flow through the cold plate cooling system to increase a heat dissipation rate of the electronic devices <NUM> into the cold plates <NUM>.

Method <NUM> may continue at step <NUM>, which includes determining the new valve position to which the control valve <NUM> was adjusted in step <NUM>. Steps <NUM> and <NUM> may, in some aspects, be performed by the controller <NUM> simultaneously.

Method <NUM> may continue at step <NUM>, which includes a determination of whether the new valve position (determined in step <NUM>) is less than a minimum open percentage. If the controller <NUM> determines that the determined valve percent open position is less than the minimum open percentage, then method <NUM> continues to step <NUM>, which includes setting (e.g., actuating) the control valve <NUM> to the minimum percent open position (which may be predetermined or otherwise programmed in the controller <NUM>).

If the controller <NUM> determines that the determined valve percent open position is greater than the minimum open percentage, then method <NUM> continues to step <NUM>, which includes a determination of whether the new valve position (determined in step <NUM>) is greater than a maximum open percentage. If the controller <NUM> determines that the determined valve percent open position is greater than the maximum open percentage, then method <NUM> continues to step <NUM>, which includes setting (e.g., actuating) the control valve <NUM> to the maximum percent open position (which may be predetermined or otherwise programmed in the controller <NUM>). If the controller <NUM> determines, based on steps <NUM> and <NUM>, that the control valve open percentage is between the minimum and maximum set positions, then method <NUM> may end or continue back to step <NUM>.

<FIG> is a graph that illustrates a relationship between operating variables of the cold plate cooling system according to the operating method of <FIG>. This figure includes graph <NUM>, which shows a relationship between cooling liquid flow rate through the cold plate cooling system on a y-axis <NUM> and open percentage of control valve <NUM> on x-axis <NUM>. Thus, as shown, curve <NUM> shows the relationship between the cooling liquid volumetric flow rate relative to the percent open of the control valve <NUM>. As shown in graph <NUM>, there is a non-linear relationship between flow rate and control valve open percentage.

<FIG> is a schematic diagram of a control system (or controller) <NUM>. The system <NUM> can be used for the operations described in association with any of the computer-implemented methods described previously, for example as or as part of the controller <NUM> or other controllers described herein. For example, the system <NUM> may be used in providing local control for particular ones of or small groups of server rack sub-assemblies, or in providing master control over an entire data center or multiple data centers of such units. Moreover, the system <NUM> may describe computing resources that may operate as the loads to be cooled by the systems and methods described above.

The system <NUM> is intended to include various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The system <NUM> can also include mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally the system can include portable storage media, such as, Universal Serial Bus (USB) flash drives. For example, the USB flash drives may store operating systems and other applications. The USB flash drives can include input/output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device.

The system <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, and an input/output device <NUM>. Each of the components <NUM>, <NUM>, <NUM>, and <NUM> are interconnected using a system bus <NUM>. The processor <NUM> is capable of processing instructions for execution within the system <NUM>. The processor may be designed using any of a number of architectures. For example, the processor <NUM> may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor.

In one implementation, the processor <NUM> is a single-threaded processor. In another implementation, the processor <NUM> is a multi-threaded processor. The processor <NUM> is capable of processing instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a user interface on the input/output device <NUM>.

The features described can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier, e.g., in a machine-readable storage device for execution by a programmable processor; and method steps can be performed by a programmable processor executing a program of instructions to perform functions of the described implementations by operating on input data and generating output. The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result.

Additionally, such activities can be implemented via touchscreen flat-panel displays and other appropriate mechanisms.

The features can be implemented in a control system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular implementations of particular inventions.

Claim 1:
A data center cooling system, comprising:
a server rack sub-assembly (<NUM>) that comprises at least one motherboard (<NUM>) mounted on a support member (<NUM>) and a plurality of heat generating electronic devices (<NUM>) mounted on the at least one motherboard (<NUM>);
at least one cold plate (<NUM>) positioned in thermal communication with at least a portion of the plurality of heat generating electronic devices, the cold plate configured to receive a flow of a cooling liquid (<NUM>) circulated through a supply conduit (<NUM>) fluidly coupled to a liquid inlet of the at least one cold plate, through the at least one cold plate, and through a return conduit (<NUM>) fluidly coupled to a liquid outlet of the at least one cold plate; and
a modulating control valve (<NUM>) mounted on either of the motherboard (<NUM>) or the support member and positioned in either of the supply conduit or the return conduit, the modulating control valve configured to adjust a rate of the flow of the cooling liquid in response to receiving a command from a controller (<NUM>), the command configured to adjust the valve based at least in part on an operating condition of at least one of the plurality of heat generating electronic devices;
the controller (<NUM>) communicably coupled to the modulating control valve (<NUM>), and configured to:
determine (<NUM>) an operating temperature for each of the plurality of heat generating electronic devices;
determine (<NUM>) a maximum operating temperature corresponding to a hottest electronic device from the determined operating temperatures; and
calculate (<NUM>) a thermal margin between the maximum operating temperature and a minimum or average of the determined operating temperatures; and
adjust (<NUM>) the modulating control valve to open or close based on the determined thermal margin to a new valve position between a predetermined minimum open percentage higher than <NUM>% and a predetermined maximum open percentage lower than <NUM>%.