Localized fluid acceleration in immersed environment

A cooling system comprises an information technology (IT) container including a plurality of IT chambers, a fluid supply channel disposed at a bottom of the IT container to receive the fluid from a cooling unit and to supply the fluid to the IT chambers, a fluid return channel disposed on a top of the IT chambers to return the fluid received from the IT chambers to the cooling unit, a fluid acceleration channel disposed separately from the fluid return channel to return at least some of the fluid to the cooling unit in an accelerated manner, and one or more pumps disposed between at least some of the IT chambers immerged in the fluid environment and the fluid acceleration channel to increase a flowrate of the fluid from the corresponding IT chambers to the cooling unit via the fluid acceleration channel. For example, each IT chamber to store fluid and to populate IT equipment submerged in the fluid for immersion cooling. In an embodiment, the IT chamber includes acceleration sections.

FIELD OF THE INVENTION

Embodiments of the present invention relate generally to data centers. More particularly, embodiments of the invention relate to immersion cooling for data centers.

BACKGROUND

Heat removal is a prominent factor in a computer system and data center design. The number of high performance electronics components such as high performance processors packaged inside servers have steadily increased, thereby increasing the amount of heat generated and dissipated during the ordinary operations of the servers. The reliability of servers used within a data center decreases if the environment in which they operate is permitted to increase in temperature over time. Maintaining a proper thermal environment is critical for normal operations of these servers in data centers, as well as the server performance and lifetime. It requires more effective and efficient heat removal solutions especially in the cases of cooling these high performance servers.

Immersion cooling technology has brought many attentions recently. Many efforts are focusing on the fluid selection, information technology (IT) side design, material compatibilities, test and verification, and so on. Most of the solutions utilize existing cooling infrastructure (cooling water/chilled water) or system. In some of the solutions, a coolant distribution unit (CDU) is used to form an external cooling loop and an internal immersion cooling fluid loop. The external cooling loop can be adapted to any type of existing data center cooling infrastructures. These solutions may not fully utilize the advantages of immersion cooling.

DETAILED DESCRIPTION

The IT Hardware industry is a critical market for many reasons: it plays a crucial role in business competitiveness, service quality and availability, and also plays a significant role in the infrastructure TCO. IT Hardware is closely linked with the profit of an organization. It is one of the core competencies of the internet giant, cloud computing service providers, as well as high performance computing and AI computing related business service users and providers who build, operate, compute, store and manage other IT hardware platforms and infrastructures. The majority of the hyper-scale owners are customizing full-stacks of these hardware systems. For instance, in the rapidly growing cloud computing business, the performance and cost (both capital cost and operation cost) of computing and storage hardware systems, clusters and infrastructure, all require the service providers to create customized systems that fit their individual needs the best. These markets require continuous innovation. An efficient system design and operation benefits the service providers in multiple aspects in a long term. The key to this is to develop continuously with more resilience, efficiency, and cost effective solutions and architecture. By increasing the power density of the high end processors, the power difference among the high power components and low power components increases, thereby requiring the thermal management industry to continuously provide both feasible and efficient solutions.

The first problem the current solution aims to solve is the high power density electronics' thermal management. With the increasing power of the IT equipment and power density in the system, thermal management becomes more and more challenging. The traditional air cooling systems become a bottle neck for the development and deployment of higher power computing and higher performance computing based electronics, including chips.

The second focus on the current work is to increase the operating efficiency and system capability of immersion cooling. While immersion cooling may be a proper solution for the high power density thermal management, however, the efficiency and capability of a system with an immersion cooling system still falls short of fulfilling the requirement. The current disclosure purposes to change the internal fluid management and fluid dynamics to improve the cooling system's performance and capability. Since the fluid is more efficiently managed, the system capability will increase, especially in the cases of high power density and low power density, which are co-packages on the same system.

For immersion cooling solutions, the single phase solution still requires an implementation of cooling hardware such as heat sink, to extend and maximize the heat transfer area for the high power density components. This is due to the nature of fluid circulation in a purely immersed environment, where the circulation may not sufficient for such high power density components' cooling requirement. The solution proposed in the current invention enables localized fluid acceleration and cooling enhancement for these components.

Since immersion cooling is designed for high power systems, it is common for some of the immersion tanks or containers to reach a very high power density. In such high power systems (either tank or container), the power distribution is greatly uniformly distributed, which mean that the IT power dissipation as well as thermal requirement are completely different. Advanced designs are required for such scenarios.

Solution versatility and resilience are also key requirements for modern IT hardware. The current work aims to improve the solution versatility and resilience which may be compatible for different cases as well as IT generation upgrades.

The previous design for immersion cooling does not provide efficient fluid management nor localized fluid management. The previous solution predominantly characterizes a high performance heat sink to solve back for the high power density challenge in an immersion cooling environment. This solution is limited by the power envelop of load and space, for implementing large heat sinks and so on.

The current design introduces a localized precision cooling enhancement for the IT, which proves to be especially useful for high power density equipment. The design enables the following features: Localized fluid management and acceleration for high heat load regions, advanced control design for separate regions, multi-layer fluid management and fluid dynamics in immersed environments, hardware design for the IT and IT container, flexible reconfigurable for different IT equipment and use cases for both could data center and edge computation systems.

In an embodiment, a cooling system comprises an information technology (IT) container including a plurality of IT chambers, a fluid supply channel disposed at a bottom of the IT container to receive the fluid from a cooling unit and to supply the fluid to the IT chambers, a fluid return channel disposed on a top of the IT chambers to return the fluid received from the IT chambers to the cooling unit, a fluid acceleration channel disposed separately from the fluid return channel to return at least some of the fluid to the cooling unit in an accelerated manner, and one or more pumps disposed between at least some of the IT chambers and the fluid acceleration channel to increase a flowrate of the fluid from the corresponding IT chambers to the cooling unit via the fluid acceleration channel. For example, each IT chamber to store fluid and to receive IT equipment submerged in the fluid for immersion cooling.

In an embodiment, each of the one or more pumps are configured to pump the fluid from the fluid supply channel through the corresponding IT chamber to the fluid acceleration channel.

In an embodiment, at least one IT equipment includes a regular section containing an IT component that generates heat and an acceleration section containing a local heat exchange element attached to the IT component, both the regular section and the acceleration section integrated as one complete system.

In an embodiment, a pump associated with the IT equipment is disposed in the acceleration section to increase the flowrate of the fluid in the acceleration section.

In an embodiment, the local heat exchange element includes a heat sink elevated to the acceleration section. For example, a cooling system comprises cooling hardware such as a heat sink to extend and maximize the heat transfer area for the high power density components.

In an embodiment, the fluid acceleration channel is disposed on top of the fluid return channel. In an embodiment, both acceleration channel and fluid return channel are connected to one return channel.

In an embodiment, the flowrate of the fluid in an IT chamber is controlled by a corresponding pump based on a local temperature of the fluid within the corresponding IT chamber.

In an embodiment, each pump is attached to the fluid acceleration channel and coupled to and removable from the fluid return channel of the IT container.

In an embodiment, the fluid acceleration channel and the pumps are attached to the lid, such that when the lid is lifted, the fluid acceleration channel and the pumps are lifted and removed from the IT chambers. In an embodiment, when the pumps are lifted/removed from the IT chambers, the main fluid recirculation may still function. However, since the acceleration pumps are lifted/removed from the fluid, there are no acceleration functions.

FIG. 1is a block diagram illustrating a data center system according to one embodiment. Referring toFIG. 1, data center immersion cooling system100is referred to as a data center system with immersion cooling. In one embodiment, data center immersion cooling system100includes data center or data center unit101coupled to external cooling unit102. External cooling unit102may be an indirect evaporative cooling (IDEC) unit. Cooling unit102includes a heat exchanger105, which may be a liquid-to-liquid heat exchanger or an air-to-liquid heat exchanger. Typically, heat exchanger105includes a primary loop106and a secondary loop107. Primary loop106is utilized to circulate external cooling source such as external air or external liquid. Secondary loop107is utilized to circulate internal cooling liquid to exchange heat with the external cooling material of primary loop106.

In one embodiment, data center101includes an immersion tank103filled with the internal cooling liquid, i.e., immersion cooling liquid. Although there is one immersion tank shown herein, more immersion tanks can also be included within data center101. Immersion tank103contains one or more server systems104and each server blade includes one or more IT components (e.g., processors, memory, storage devices). Server systems104are submerged in the internal cooling liquid. The internal cooling liquid is thermally conductive dielectric liquid designed to extract the heat from the server systems. Such cooling technique is referred to as immersion cooling.

Server immersion cooling is a computer cooling practice by which computer components or servers are submerged in a thermally conductive dielectric liquid. For example, common dielectrics which are suitable for immersion cooling are typically oil-based. Server immersion cooling has the potential of becoming a popular server cooling solution for green data centers, as it allows them to drastically reduce their energy load, regardless of their PUE. Servers and other IT hardware cooled by immersion cooling do not require fans, thus these are removed.

Referring back toFIG. 1, according to one embodiment, data center101includes a liquid supply line111and a liquid return line112coupled to the secondary side of the heat exchanger105of cooling system102to form the secondary loop. In addition, liquid supply line111is coupled to an intake port of immersion tank103and liquid return line112is coupled to an outlet port of immersion tank103. Liquid supply line111is configured to receive the cooling liquid from heat exchanger105and to distribute the cooling liquid to immersion tank103. Liquid return line112is configured to receive the cooling liquid carrying the heat exchanged from server blades104from immersion tank103and to return the cooling liquid back to heat exchanger105for heat exchange.

In addition, a liquid pump115may be disposed on liquid return line112to pump and circulate the cooling liquid to flow within the secondary loop. In addition, multiple pumps may be designed in the system (on main supply line111or on main return line112for redundant purpose. Note that if there are multiple immersion tanks within data center101, there will be multiple pairs of liquid supply line and liquid return line to couple the immersion tanks with heat exchanger105of cooling system102. Unlike conventional cooling systems, the secondary loop107via liquid supply line111, immersion tank103, and liquid return line112is a single heat transfer loop without using a CDU in between. Typically, a CDU also includes a heat exchanger having a primary loop and a secondary loop therein, which will form multiple loops between cooling system102and immersion tank103. Also note that liquid pump115may be disposed on liquid supply line111or alternatively, there may be multiple liquid pumps, one disposed on liquid supply line111and another one disposed on liquid return line112.

FIG. 2is a block diagram illustrating an example of a data center system200with immersion cooling according to another embodiment. For example,FIG. 2shows the system design view from the front side. The immersion cooling system200may be utilized as a part of cooling system100of data center101as shown inFIG. 1. In an embodiment, the fluid supply channel213is designed on the bottom of the container209. In an embodiment, the fluid supply channel213is the only source for supply fluid to the system in this architecture. In an embodiment, the main section includes multiple IT chambers (e.g.,209a,209b). For example, the IT chambers (e.g.,209a209b) are filled with immersion cooling fluid and it is used for populating IT equipment (e.g.,211a,211b). InFIG. 1, for example, IT equipment (e.g.,211a,211b) are shown populated and aligned in the IT chambers (e.g.,209a,209b). In an embodiment, the fluid return channel207is used for the main system fluid return. In an embodiment, on the top of the IT equipment (e.g.,211a,211b) there is a section (e.g.,205a,205b) where a pump (e.g.,215a,215b) is packaged. In an embodiment, the pump (e.g.,215a,215b) is the localized pump which is used for accelerating the corresponding IT equipment (e.g.,211a,211b) it is designed and assembled for. In an embodiment, the localized pumps (e.g.,215a,215b) are connected to the fluid acceleration channel203. For example, the fluid acceleration channel203can be understood as a channel or a loop. In an embodiment, the cooling system200includes a lid on the top of the fluid acceleration channel203. For example, the lid is the very top cover of the system200.

FIG. 3is a block diagram illustrating an example of a data center system300with immersion cooling according to an embodiment.FIG. 3shows another system configuration according to an embodiment. In an embodiment, this design is a highly combined design with IT equipment301. For example, it can be seen that the basic sections are the same as shown inFIG. 2. However, the IT equipment301are designed in two layers/sections303,305, even though these two sections303,305are both integrated together as one complete system according to an embodiment. In an embodiment, the acceleration section305can be understood as an extension of the cooling area of portion302. In an embodiment, in the acceleration section305, the major cooling device309such as heat sinks are elevated to this section305. In an embodiment, the other section303is the regular section which packaged the electronics. In an embodiment, since the system are designed in two sections303,305, it is more convenient to design and implement acceleration section305for the IT equipment301. In an embodiment, only the acceleration section305is designed with acceleration pump307accordingly. For example, the acceleration sections305can be custom designed based on the IT populated in the immersion system, and it also provides possibilities for system design and operating efficiency.

In an embodiment, the acceleration pump307is dedicated for the major cooling device309in the acceleration section305. In some embodiment, the location of the acceleration pump307is adjustable connected with the acceleration channel311. In some embodiment, the acceleration pump307is coupled to and removable from the regular return channel313.

In an embodiment, the acceleration pump307can be integrated to acceleration channel311. In an embodiment, the acceleration channel311is not in the main immersion environment, and once the acceleration pump307is installed, it connects the immersion fluid with the acceleration channel311.

In an embodiment, the acceleration channel311includes a fluid blind mating port such as male portion. In an embodiment, the acceleration pump side may include female portion to connect with male portion of the acceleration pump side for connections. In an embodiment, connections are blind mating quick disconnections.

In an embodiment, it can be understood as that the acceleration pump307is configured as a connecting component between the main immersion environment and the acceleration channel311.

In an embodiment, the acceleration pump307is packaged within an additional channel315, then the additional channel315is connected to the acceleration channel311. In an embodiment, the acceleration pump307within the additional channel315pumping the fluid to accelerate the corresponding region309. In an embodiment, the additional channel315has the acceleration pump307packaged inside. In an embodiment, one or more additional channels is connected to the acceleration channel to form a complete acceleration loop between the immersion tank to the converging channel.

In an embodiment, the acceleration channel can be individually dedicated connected to the converging channel. In an embodiment, the acceleration channel311can be merged together and parallel (in terms of fluid flowing) with the regular channel and then converges to converging channel. For example, the fluid is supplied from the main supply to the immersion system, and the fluid will flow through the electronics and then return to the converging channel before finally reaching the cooling unit. In an embodiment, if there are one or more acceleration channels, a portion of the fluid will be accelerated with a higher flow rate and pass through its corresponding acceleration channel and then to converge in the converging channel. This is an illustration of the parallel concept.

FIG. 4is a block diagram illustrating an example of a data center system400with immersion cooling according to another embodiment.FIG. 4presents a side of design according to an embodiment. It can be seen the main supply411and main return417of the system are designed at the bottom and left side on the top. For example, it needs to be mentioned that these designs can be in different locations in the system, and the main functions are supply and return. In an embodiment, the localized pump419and the corresponding channel413are located on the top of the server and connected to the fluid acceleration channel413. For example, it needs to be noted that this may require some server chassis optimization to have a better structural level adaption with the localized pump section.

In an embodiment, the arrow411,413,417shows the fluid flowing direction. For example, the current one shows that fluid goes into the immersion container411through the supply channel409and pass the PCB405including all the chips407before leaving the container. In an embodiment, the fluid417normally leaves the container through the fluid return channel415. Also, the fluid413can be localized accelerated by the pump419and leaves the container faster through the fluid acceleration channel403according to an embodiment. In an embodiment, the lo

FIG. 5is a block diagram illustrating an example of a data center system500with immersion cooling according to another embodiment. For example,FIG. 5shows that one of the design that integrate the acceleration channel503and acceleration pump/pumps505are integrated on the top lid501of the immersion container/tank. In an embodiment, different lid501can be switched with different acceleration configurations. For example, this can be understood as the top section501,503,505of the system. In an embodiment, the location of the acceleration pump505can be changed to be coupled to channel503under lid501. In some embodiment, the acceleration pump505, the acceleration channel503, the top lid501and/or the entire top section can be dynamically reconfigured.

FIG. 6is a block diagram illustrating an example of a top section600of a data center system with immersion cooling according to another embodiment. For example,FIG. 6shows that the top section600can be flexible configured based on the IT characteristics populated in the system. For example, the container maybe populated with some GPU based high performance systems and storage system. In an embodiment, only the dedicated spaces where the GPU systems are populated are assembled with localized pumps601,603,605. For example, the location of each acceleration pump (e.g.,601,603,605) can be adjusted based on the corresponding acceleration section. In an embodiment, acceleration pumps601,603,605are coupled to acceleration channel under lid. In some embodiment, the acceleration pumps601,603,605, the acceleration channel, the top lid and/or the entire top section can be dynamically reconfigured. In an embodiment, one or more acceleration pumps (e.g.,601,603,605) may be in idle based on the need of cooling enhancement. In an embodiment, all acceleration pumps may be in idle when no cooling enhancement is needed.

FIG. 7is a block diagram illustrating an example of a data center system700with immersion cooling according to another embodiment. For example,FIG. 7presents a schematic representation of the system and the control design. In an embodiment, the immersion tank703is connected to the cooling unit701. In an embodiment, the cooling unit701can be understood as any fluid sources or cooling unit, either in a primary loop725,707or secondary loop725,709. In an embodiment, the system pump705is used on the main supply loop705and the pump speed is controlled by the immersion system power (e.g.,721), either the rated power or actual operating power. In an embodiment, the pump speed can be set as a constant speed based on the system characterization. In an embodiment, one or more acceleration pumps713,715are forming separate loops717,719form the immersion tank703and the coolant unit701. In an embodiment, it can be seen that the pumps713,715pump fluid form the immersion tank703precisely from certain area or locations, and the fluid717,719pumped by these acceleration pumps713,715forming together in the fluid acceleration channel711,719. In an embodiment, the localized acceleration pump713,715are controlled through the measurements of the localized temperature (e.g., T723) such as GPU temperatures or other high power density electronics' temperatures. It needs to be motioned that each individual acceleration pump (e.g.,713,715) may be connected to different sensors723a,723b, or input control signals721.

FIG. 8is a block diagram illustrating an example of a data center system800with immersion cooling under a normal operation mode according to an embodiment. For example,FIG. 8shows the fluid801,803flowing manner during the normal operating mode. In an embodiment, under the normal operating mode, the supply fluid801flows from the cooling unit701to the immersion tank703and the return fluid803flows from the immersion tank703to the cooling unit701.

FIG. 9is a block diagram illustrating an example of a data center system900with immersion cooling under a single acceleration mode according to another embodiment. For example,FIG. 9shows the fluid flowing diagram when one of the acceleration907functions. In an embodiment, under the single acceleration mode, the supply fluid901flows from the cooling unit701to the immersion tank703, the return fluid903flows from the immersion tank703to the cooling unit701, and the first acceleration fluid907,905also flows from the immersion tank703to the cooling unit701.

FIG. 10is a block diagram illustrating an example of a data center system1000with immersion cooling under a multiple acceleration mode according to another embodiment. For example,FIG. 10shows system fluid flowing diagram when multiple localized fluid acceleration1007,1009are in active mode. In an embodiment, under the multiple acceleration mode, the supply fluid1001flows from the cooling unit701to the immersion tank703. In some embodiment, the return fluid1003flows from the immersion tank703to the cooling unit701. In an embodiment, he first acceleration fluid1007and the second acceleration fluid1009flows from the immersion tank703to the cooling unit701through acceleration channel1011,1005.