Patent Description:
Liquid submersion cooled electronic systems and devices are known. One example of an array of liquid submersion cooled electronic devices is an array of liquid submerged servers (LSS's) arranged in a rack system. An example of an array of LSS's in a rack system is disclosed in <CIT>, <CIT>, and <CIT>. Another example of an array of liquid submersion cooled electronic devices is disclosed in <CIT>. Patent Application Publication No. <CIT> discloses an apparatus for cooling of computing system environments and, more particularly, to immersion cooling of electronic components used in large computing systems environments having one or more servers.

Liquid submersion cooling devices and systems are described that use a cooling liquid, for example a dielectric cooling liquid, to submersion cool individual electronic devices or an array of electronic devices. In embodiments described herein, the electronic device includes a non-pressurized (or "zero" pressure) device housing defining an interior space where pressure in the interior space equals, or is only slightly greater than, pressure outside the non-pressurized device housing.

A liquid submersion cooled electronic device is provided according to claim <NUM>.

One example application of the liquid submersion cooling devices and systems described herein is for use with an array of LSS's arranged in a rack system. However, the concepts described herein can be used in other applications where arrays of electronic devices are liquid submersion cooled, including, but not limited to, blade servers, disk arrays/storage systems, solid state memory devices, storage area networks, network attached storage, storage communication systems, routers, telecommunication infrastructure/switches, wired, optical and wireless communication devices, cell processor devices, printers, power supplies, etc..

The liquid submersion cooling devices and systems described herein can be used in any area that could benefit from the advantages of liquid submersion cooling. In one example, the liquid submersion cooling devices and systems can be used in blockchain computing (cryptocurrency) applications, for example in either ASIC or GPU computer mining configurations. The liquid submersion cooling devices and systems can also be used in deep learning applications, for example in multi-GPU configurations supporting maximum bandwidth and direct memory access (DMA) of high performance GPUs. The liquid submersion cooling devices and systems can also be used in artificial intelligence and high-performance computing (HPC) clusters with multiple co-processor configurations, for example multi-GPU configurations supporting DMA capabilities of GPU co-processors. Many other applications and uses of the liquid submersion cooling devices and systems described herein are possible and contemplated.

The liquid submersion cooling devices and systems described herein do not require fully sealed electronic device housings, which helps to reduce cost and simplifies access to the electronics for service and modifications. Liquid submersion cooling also has superior cooling efficiency compared to air cooling, thereby reducing power requirements and associated operating costs.

The cooling liquid used to cool the electronics in the electronic devices described herein is a dielectric liquid. The cooling liquid is preferably a single phase dielectric cooling liquid. It is preferred that the single phase dielectric cooling liquid have a high enough thermal transfer capability and heat capacity to handle the amount of heat being generated by the submerged heat generating electronic components so that the cooling liquid does not change state from a liquid to a gas during the heat absorption process. Submersion cooling of the heat generating electronic components means that enough of the cooling liquid is present so that one or more of the heat generating electronic components are partially or fully submerged in the dielectric cooling liquid in direct intimate contact with the dielectric cooling liquid.

The heat-generating electronic component(s) to be submerged in the cooling liquid can be any electronic component(s) that generate heat and that one may wish to cool by partially or fully submerging the electronic components in the cooling liquid. For example, the electronic components can include one or more processors, for example a CPU and/or a GPU, one or more power supplies, one or more switches, one or more data storage drives, one or more memory modules, and other electronic components. The electronic systems formed by the electronic components include, but are not limited to, blade servers, disk arrays/storage systems, solid state memory devices, storage area networks, network attached storage, storage communication systems, routers, telecommunication infrastructure/switches, wired, optical and wireless communication devices, cell processor devices, printers, power supplies, and the like.

<FIG> illustrate an example of a liquid submersion cooled electronic device <NUM>. The device <NUM> includes a device housing <NUM> that is formed by a bottom liquid tight tray <NUM> defining an interior space <NUM> and a cover <NUM> that removably fits on the tray <NUM> to prevent contaminants from falling into cooling liquid that is contained with the interior space <NUM> of the tray <NUM>. The tray <NUM> has side walls and a bottom wall that define the interior space <NUM>, and at least a partially open top. In the illustrated example, the entire top of the tray <NUM> is shown as being open. However, in other embodiments, only a portion of the top of the tray <NUM> can be open. The cover <NUM> is removably disposed over the open portion of the top of the tray <NUM>.

The housing <NUM> can be referred to as non-pressurized (or "zero" pressure) or minimally pressurized so that the pressure (or vacuum) in the interior space <NUM> equals, or is only slightly greater/less than, pressure outside the device housing. For example, the pressure in the interior space <NUM> can equal ambient pressure. In another embodiment, the pressure in the interior space may be a small pressure having a value which may be small and difficult to measure, for example up to about <NUM> psi. So a non-pressurized device housing as used herein is intended to encompass the interior space <NUM> having zero pressure (i.e. the pressure in the interior space equals ambient pressure) as well as encompass small pressures/vacuums, for example a pressure of up to about <NUM> psi greater than ambient pressure. This is in contrast to some device housings for liquid submersion cooled electronics which may be referred to as pressurized or sealed housings which might typically operate at positive measureable pressure levels greater than ambient air pressure as a consequence of being connected to other similarly liquid immersion cooled electronic devices involving the same fluid loop and experiencing the pressure produced by a centralized or remote pump which produces fluid circulation by creating a positive pressure on the outlet of the pump and a corresponding negative or lower pressure on the inlet to the pump.

The minimization of pressure between the interior space <NUM> and the ambient can be achieved in any suitable manner. For example, in one embodiment a pressure relief/equalization mechanism <NUM>, such as an air vent or check valve or other pressure relief/equalization mechanism, can be provided in the cover <NUM> as illustrated in <FIG> to provide air communication between the interior space and the ambient. In another embodiment, pressure minimization can be achieved simply as a result of the cover <NUM> not fitting closely or sealing with the tray <NUM>. Because the housing <NUM> is not intended to be pressurized, there is no need to seal and pressurize the device <NUM>. However, the tray <NUM> does need to be sealed or leak proof to prevent leakage of cooling liquid therefrom that will be disposed within the interior space <NUM>. Minimization of pressure in the device <NUM> is permissible because the dielectric cooling liquid is recirculated within the tray <NUM> and the heat exchanger (described below) and there is no need for the device <NUM> to operate at a pressure that is different than the pressure of the ambient environment.

As best seen in <FIG> and <FIG>, various heat generating electronic components <NUM> are disposed within the interior space <NUM>. The electronic components <NUM> can vary based on the type of electronic system the device <NUM> is to form. Examples of electronic components <NUM> that can be used includes, but is not limited to, one or more processors, for example a CPU and/or a GPU, one or more power supplies, one or more switches, one or more data storage drives, one or more memory modules, and other electronic components. The electronic systems formed by the electronic components can include, but are not limited to, blade servers, disk arrays/storage systems, solid state memory devices, storage area networks, network attached storage, storage communication systems, routers, telecommunication infrastructure/switches, wired, optical and wireless communication devices, cell processor devices, printers, power supplies, and the like.

A dielectric cooling liquid is disposed in the interior space <NUM> with the dielectric cooling liquid partially or fully submerging and in direct contact with at least some the heat generating electronic components <NUM>. The level of the dielectric liquid in the tray <NUM> will be sufficient to partially or fully submerse the electronic components that one wishes to submersion cool.

A cooling liquid distribution circuit is provided for distributing the cooling liquid within the device <NUM>. In the embodiment illustrated in <FIG> and <FIG>, the distribution circuit includes one or more pumps <NUM> within the interior space <NUM> and having a pump inlet in fluid communication with the bulk cooling liquid contained in the interior space <NUM> and a pump outlet. The illustrated example shows two of the pumps <NUM>, one pump <NUM> used as a primary pump and the other pump <NUM> used as a back-up pump in case of failure of the primary pump. A control valve <NUM> can be provided that is fluidly connected to the outlet of each pump <NUM> and which can be controlled by a suitable pump controller based on monitored performance of the pumps <NUM> to select which pumps outlet will be used. The pumps <NUM> can be partially or fully submerged in the cooling liquid, or the pumps <NUM> may not be submerged but have inlets in the cooling liquid.

A heat exchanger <NUM> is disposed within the interior space <NUM> and has an inlet in fluid communication with the pump outlet via the control valve <NUM>, and an outlet. The heat exchanger <NUM> can have any configuration that is suitable for reducing the temperature of the returning cooling liquid. In the illustrated example, the heat exchanger <NUM> is configured as a liquid-to-liquid heat exchanger that is connected to an external cooling fluid loop <NUM> which supplies a secondary cooling liquid to the heat exchanger <NUM>. However, the heat exchanger <NUM> can be a liquid-to-air heat exchanger or any other configuration that can reduce the temperature of the returning cooling liquid.

The heat exchanger <NUM> can be mounted at any suitable location within the housing <NUM>. In the illustrated example, the heat exchanger <NUM> is shown as being mounted on the interior facing surface of an end wall <NUM> of the tray <NUM>. The heat exchanger <NUM> may or may not be partially or fully submerged in the cooling liquid disposed within the interior space <NUM>.

Referring to <FIG> along with <FIG>, a liquid distribution manifold <NUM> is disposed within the interior space <NUM> and has an inlet <NUM> that is in fluid communication with the outlet of the heat exchanger <NUM> via a supply line <NUM>, and a plurality of manifold outlets <NUM> leading from the manifold <NUM>. The liquid distribution manifold <NUM> distributes the cooling liquid to targeted ones of the electronic components <NUM> before the cooling liquid enters the bulk cooling liquid within the interior space <NUM>.

The returning cooling liquid can be directed from the manifold <NUM> directly onto some of the electronic components <NUM>, such as CPUs, GPUs, power supplies, switches, or the like. For example, as shown in <FIG> and <FIG>, one or more open top trays <NUM> are disposed within the interior space <NUM>, and some of the electronic components <NUM> are disposed in the trays <NUM>. Supply tubes <NUM> extend from the outlets <NUM> to each tray <NUM> in order to direct the returning cooling liquid into the tray <NUM>. The trays <NUM> retain the cooling liquid around the electronic component(s) <NUM> located within the trays <NUM>. One or more liquid outlets or weirs <NUM> are formed in a side wall of each tray <NUM> from which dielectric cooling liquid exits the space defined by the tray <NUM>. In use, each tray <NUM> is designed to fill with the cooling liquid to a level sufficient to liquid submersion cool the electronic component(s) <NUM> with the trays <NUM>. The cooling liquid then spills out from the weir(s) <NUM> and flows by gravity into the bulk cooling liquid in the remainder of the interior space <NUM>, where it can then be pumped by the pump <NUM> to the heat exchanger <NUM> for cooling. Some of the electronic components within the interior space <NUM> but not within one of the trays <NUM> may also be partially or fully submerged in the bulk cooling liquid contained in the interior space <NUM>.

The fluid distribution manifold <NUM> can be configured to help proportion the flow of the cooling liquid to each of the outlets <NUM> to optimally manage the flow from each of the outlets <NUM> to the trays <NUM>. For example, the sizes of the outlets <NUM> can be varied, the sizes of the supply tubes <NUM> can be varied, adjustable valves can be provided in the outlets <NUM> or in the tubes <NUM>, or the like. Management of the flow is useful in order to direct the proper amount of and/or the correct pressure of the returning cooling liquid.

The weir(s) <NUM> is disposed at the maximum dielectric cooling liquid level of the tray <NUM> wherein the weir(s) <NUM> establishes the level of the dielectric cooling liquid within the tray <NUM> and establishes a volumetric rate of flow of the dielectric cooling liquid within the tray <NUM> that is needed for the cooling of the heat generating electronic component(s) within the tray <NUM>. As used throughout this description and claims, a weir is an outlet for the cooling liquid where the cooling liquid exits via gravity without using pump pressure connected to the weir. A weir is different than, and distinct from, an outlet which during use is intended to be connected to a pump so that pump pressure causes the cooling liquid to exit through the outlet, such as the outlets 52b, <NUM> described in <CIT>.

In this embodiment, the pumps <NUM> and the heat exchanger <NUM> are disposed at a first end of the device housing <NUM>, and the liquid distribution manifold <NUM> is disposed at a second end of the device housing <NUM> opposite to the first end. However, other arrangements are possible.

<FIG> illustrates another example of a liquid submersion cooled electronic device <NUM> that is similar to the device <NUM> and like elements are referenced using the same reference numerals. In this embodiment, the heat exchanger <NUM> is disposed outside the interior space <NUM> of the device housing so that the cooling liquid exits the device housing to be cooled. For example, the heat exchanger <NUM> can be mounted on the exterior surface of the end wall <NUM> of the tray <NUM>. Although not illustrated, the device <NUM> will include a cover like the cover <NUM> in <FIG>.

<FIG> illustrates another example of a liquid submersion cooled electronic device <NUM> that is similar to the device <NUM> and like elements are referenced using the same reference numerals. In this embodiment, the heat exchanger <NUM> is disposed outside the interior space <NUM> of the device housing so that the cooling liquid exits the device housing to be cooled. In this embodiment, the heat exchanger <NUM> is not mounted on the end wall <NUM> of the tray <NUM>, but is instead mounted at an exterior location separate from the device <NUM> so that the heat exchanger <NUM> is not mounted on the device housing. Although not illustrated, the device <NUM> will include a cover like the cover <NUM> in <FIG>.

<FIG> illustrates another example of a liquid submersion cooled electronic device <NUM> that is similar to the device <NUM> and like elements are referenced using the same reference numerals. In this embodiment, both the heat exchanger and the pump(s) are disposed outside the interior space <NUM> of the device housing. In addition, the pump(s) and the heat exchanger are incorporated into a common unit referred to as cooling distribution unit <NUM>. The cooling distribution unit <NUM> is illustrated as not mounted on the end wall <NUM> of the tray <NUM>, but is instead mounted at an exterior location separate from the device <NUM>. However, the cooling distribution unit <NUM> could be mounted on the exterior surface of the end wall <NUM>. Although not illustrated, the device <NUM> will include a cover like the cover <NUM> in <FIG>.

Referring to <FIG>, a plurality of the devices <NUM> of <FIG> are illustrated as being disposed together in a vertical array <NUM>, for example on a rack <NUM>. Alternatively, the devices <NUM> can be used individually and separately from one another. In one example implementation, a plurality of the electronic devices <NUM> can be arranged into a plurality of vertically spaced rows on the rack <NUM>. <FIG> illustrates a vertical manifold <NUM> mounted on the rack <NUM> that is part of the external cooling fluid loop <NUM> and is used to bring the secondary cooling liquid to the heat exchanger <NUM>. A similar vertical array on the rack <NUM> can be implemented for the electronic devices <NUM>, <NUM>, <NUM>.

The pumps described herein can be adaptively controlled by the pump controller depending upon desired performance of the electronic device(s). For example, the pumps can be controlled to operate in series or in parallel. In addition, the pumps can be operated in a redundant mode where a second pump acts as a back-up to the first pump in the event of failure of the first pump.

Claim 1:
A liquid submersion cooled electronic device (<NUM>, <NUM>, <NUM>, <NUM>), comprising:
a device housing (<NUM>) defining an interior space (<NUM>), the device housing including a tray (<NUM>) with an at least partially open top and a cover (<NUM>) removably attached to the tray and disposed over the at least partially open top;
at least one heat generating electronic component (<NUM>) disposed within the interior space of the device housing;
a dielectric cooling liquid in the interior space, the dielectric cooling liquid partially or fully submerging and in direct contact with the at least one heat generating electronic component;
a pump (<NUM>) having a pump inlet in fluid communication with the interior space and a pump outlet; and
a heat exchanger (<NUM>) having a heat exchanger inlet in fluid communication with the pump outlet and having a heat exchanger outlet in fluid communication with the interior space;
characterized in that the device housing (<NUM>) is non-pressurized.