Patent Publication Number: US-8982552-B2

Title: System for providing physically separated compute and I/O resources in the datacenter to enable space and power savings

Description:
BACKGROUND 
     Conventional computer servers typically incorporate the compute resources, e.g., central processing unit (CPU) and memory, and input/output (I/O) adaptors within the same enclosure in a datacenter. The few systems that make use of disaggregated I/O typically contain some I/O functionality that still export specific I/O fabrics that are still locally tied to the server. As a result, these hardware types are physically close to each other, and must be powered and cooled in the datacenter assuming this close proximity. 
     Server enclosures containing CPUs &amp; memory continue to demand air cooling because the enclosures incorporate specialized I/O devices and other components that cannot be cooled by alternate cooling methods other than air cooling, e.g., exclusive heat conduction to the rack. 
     Servers that do have disaggregated I/O typically remain located near I/O equipment because I/O link cabling between these resources tends to be local to the server and there is often no need to separate them further. 
    
    
     
       DESCRIPTION OF THE DRAWINGS 
       The detailed description will refer to the following drawings in which like numbers refer to like objects, and in which: 
         FIGS. 1A ,  1 B illustrate exemplary configured racks for providing physically separated compute and I/O resources in the datacenter to enable space and power savings; 
         FIG. 2  is a top view of the exemplary data center floor plan for providing physically separated compute and I/O resources in the datacenter to enable space and power savings; 
         FIG. 3  is a flow chart illustrating an embodiment of a method for providing physically separated compute and I/O resources in the datacenter to enable space and power savings; and 
         FIG. 4  illustrates exemplary hardware components of a computer that may be used in connection with the method for providing physically separated compute and I/O resources in the datacenter to enable space and power savings. 
     
    
    
     DETAILED DESCRIPTION 
     Traditional server computing systems incorporate input/output (I/O) resources, i.e., I/O hardware, along with the compute resources, i.e., compute hardware, typically because of the need to communicate between the compute and I/O resources at fast speeds. Examples of compute resources include central processing unit (CPU) and memory. 
     An embodiment of a system and method disaggregate I/O resources (i.e., hardware and devices) from a server&#39;s compute resources, such as CPU and memory, by moving the server&#39;s local I/O devices to a remote location apart from the server&#39;s compute resources. An embodiment uses optical technology to separate direct-attach I/O root ports from CPUs and memory in a server architecture and to accomplish the fast communication speeds needed between the compute resources and long distances associated with remotely located I/O resources. Specifically, an embodiment uses fiber-optic cables (i.e., optical cables) and electrical-to-optical conversion to facilitate communication between the compute resources and the I/O resources. The compute resources and the remotely located I/O resources can be designed differently to allow for liquid cooling exclusively for the compute resources and air cooling for the I/O resources. 
     Further, the datacenter may be segregated into equipment locales that differ in their cooling requirements. With the segregation of the compute and I/O resources, the floor-space rack density of the compute resources can be increased, thus increasing power and cooling efficiency, and providing a safe way to integrate liquid cooling at the rack-level. As a result, datacenter power and cooling can be performed more efficiently, thus saving cost at the datacenter level. 
     Further, the optical cables can connect many servers to many I/O devices and use fewer links than traditional I/O fabrics. The I/O devices may be housed in a separate I/O enclosure, which may use traditional air cooling in the datacenter. Without the overhead of having high-powered CPUs and memories present in the I/O enclosure, these I/O devices will consume less energy using the traditional air cooling infrastructure of the datacenter. 
       FIGS. 1A ,  1 B and  2  illustrate an exemplary conductively-cooled compute rack  110  (shown in  FIGS. 1A and 2 ) that is physically separated from an exemplary air-cooled I/O rack  150  (shown in  FIGS. 1B and 2 ). These figures show a conductively cooled compute-rack for illustration purposes only. One skilled in the art will appreciate that other types of liquid-cooling can equally be applied. 
     Referring to  FIG. 1A , the exemplary compute rack  110  includes compute books  120 , which houses compute hardware, such as memory  106  and CPU  108 . The compute hardware typically uses more power than the I/O hardware, thus liquid cooling is preferred for the compute hardware. After the separation of the compute and I/O hardware, liquid cooling can be used to cool the compute hardware by providing a central cooling zone. Specifically, the compute rack  110  may include heat pipes  104  that transfer heat from major heat producers, such as the memory  106  and the CPU  108 , to a central conductive liquid-to-liquid heat exchanger  140  permitting the attachment of rack-based cold plates  102  (i.e., heat flow plates) located in the rear of the compute rack. The central heat exchanger is supplied with cool water  144 , and warm water  146  leaves it as heat is exchanged. The central heat exchanger  140 , can also connect to other components cooled with conventional fans such as the power supplies  148 . To cool these components, a sealed compartment in the product can be designed to provide a closed-loop path of air that is directly into a smaller air-to-liquid heat exchanger  142  that connects to the same central liquid-to-liquid heat exchanger at the rear of the compute rack. Another valid instance of an exclusively liquid-cooled rack, other than that detailed here, is the use of self-contained closed-loop air conditioning (AC) units that attach to a single rack and provide cool air to the front of the compute rack while collecting heated air at the back of the rack. 
     Referring to  FIG. 1A , all compute hardware may be located at the compute rack front  130 , with the cold plates  102  located at the rear of the compute rack  110 . The only cables needed to connect to the remote I/O racks are located at the compute rack front  130 , where optical communications to the externally located I/O resources are found. Specifically, with continued reference to  FIG. 1A , compute rack electrical to optical (EO) conversion devices  112  are located at the compute rack front near compute rack optical ports  114 . Compute rack optical cables  214  (shown in  FIG. 2 ) connect the compute rack optical ports  114  to an optical cable mezzanine  240  (shown in  FIG. 2 ), which is connected to the externally located I/O resources. As a result, the system and method provide liquid cooling at the server rack without bringing liquid into the server equipment itself. Datacenter operators prefer liquid cooling in the datacenter because the operators often have water lines on the floor attached to the CRAC units, but not to the actual electronics. Quick disconnects are not needed as all liquid cooling interfaces are conduction plates. 
     Referring to  FIG. 1B , the I/O rack  150  includes I/O cards  152 . At the rear  180  of the I/O rack, I/O cables  182  facilitate communications between the I/O cards  152  and other parts of the data center infrastructure such as network devices and storage devices. I/O rack EO conversion devices  162  are located at the I/O rack front  170 , or alternately at the I/O rack rear  180 , near I/O rack optical ports  164 . I/O rack optical cables  264  (shown in  FIG. 2 ) connect the I/O rack optical ports  164  to the optical cable mezzanine  240  (shown in  FIG. 2 ). Air cooling is used for the I/O rack  150  to cool the I/O hardware, with cool air in  174  provided to the cold aisle at front  170  of the I/O rack  150  and hot air  176  exhausted at the rear  180  of the I/O rack  150 . 
       FIG. 2  is a top view of the exemplary liquid-cooled data center room  100  and exemplary air-cooled data center room  200 . The liquid-cooled room  100  is connected to the air-cooled room  200  using compute rack optical cables  214 , the optical cable mezzanine  240 , and I/O rack optical cables  264 . The compute rack optical cables  214  connect the compute rack optical ports  114  (shown in  FIG. 1A ) at the compute rack front  130  to the optical cable mezzanine  240 . Since liquid cooling is used exclusively in the room containing compute racks  110 , access aisles  210  can be maintained at unregulated room temperature, which permits this section of the data center to be economized for cooling infrastructure. On the other hand, traditional air cooling is used for the I/O rack  150 , with cold aisles  270  at the I/O rack front  170  and hot aisles  280  at the I/O rack rear  180 . Cold room air conditioning (CRAG) units  290  are used for air cooling of the I/O hardware, such as the I/O cards  152  (shown in  FIG. 1B ). 
     Separating the compute resources from the I/O resources achieves cost savings associated with power and cooling of server equipment in the datacenter. The datacenter infrastructure can be optimized around the type of equipment being deployed in these different sections of the datacenter. For example, the CPU and memory may be placed in a datacenter room that requires little air movement since the liquid cooling plumbing to the room can remove all of the heat involved in these types of products. In an adjacent room, conventional heating, ventilation, and air conditioning device (HVAC) or CRAC air conditioning units may be utilized for the I/O hardware. The cost savings involved within the datacenter may be used to offset the extra cost involved in optically cabling between the compute and I/O resources. 
     The advantages of the system for separating compute and I/O resources in the datacenter to enable space and power savings are as follows. The I/O hardware are separated from the server&#39;s compute hardware, such as CPU and memory, opening the opportunity to design products separately from each other. If products can be designed separately from each other, different means of cooling can be used for each product. Liquid cooling can be used for the compute hardware, while air cooling can be used for the I/O hardware, without the need to co-join cooling methods into a single product. The system further facilitates more efficient setup of datacenter infrastructure in order to save cost of power and cooling to servers. Without the I/O hardware, the server uses less floor space in the datacenter, thus saving electricity, equipment, and facilities cost to datacenter operators. 
     When the system is conductively cooled using a central heat exchanger  140 , the system provides liquid cooling at the rack-level without bringing liquid into the same enclosure as the compute hardware itself. Quick disconnects are not needed since all liquid cooling interfaces are conduction plates, i.e., cold plates  102 . Adoption of liquid cooling into the compute rack  110  may be more favorable and may lead to quicker deployment and faster savings for datacenter customers. 
     Further, the remote I/O devices are connected to the server using a switched communications fabric, which is more generic by connecting many servers to many I/O devices. As a result, the datacenter operator has more freedom to separate the server from the I/O devices at longer distances, and to separate different equipment into different locales of the datacenter. 
       FIG. 3  is a flow chart illustrating an embodiment of a method  300  for providing physically separated compute and I/O resources in the datacenter to enable space and power savings. The method  300  starts  302  by applying liquid cooling exclusively to one or more compute devices located at a compute rack of a server infrastructure (block  310 ). The method  300  applies air cooling to one or more I/O devices located at an I/O rack, which is remotely located from the compute rack (block  320 ). The method  300  uses one or more compute rack EO conversion devices at the front of the compute rack to connect the one or more compute devices to optical cables (block  330 ). The method  300  further uses one or more I/O rack EO conversion devices at front of the I/O rack to connect the I/O devices to the optical cables (block  340 ). The method  300  ends at block  350 . 
       FIG. 4  illustrates exemplary hardware components of a computer that may be used in connection with the method for providing physically separated compute and input/output resources in the datacenter to enable space and power savings. The computer has exclusively liquid-cooled racks  440  and air-cooled racks  432 ,  434 . The exclusively liquid-cooled racks  440  contain a server with external input/output  444 , which typically includes a memory  402 , a processors  414 , I/O fabric devices  446 , and network fabric devices  448 . The air-cooled racks  432  contain external input/output products  420 , which typically include input/output fabric devices  436  and input/output cards  408 . The air-cooled racks  434  include a secondary storage device  412 , conventional servers  410 , and input &amp; display devices  416 . The secondary storage  412 , the conventional servers  410 , the input &amp; display devices  416 , the input/output cards  408 , and the network fabric devices may be connected using a network  418  such as the Internet or other type of computer or telephone network. The input/output fabric devices  446  on the exclusively liquid-cooled racks  440  and the input/output fabric devices  436  on the air-cooled racks  432  may be connected using an optical input/output fabric  450 . 
     The memory  402  may include random access memory (RAM) or similar types of memory. The secondary storage device  412  may include a hard disk drive, floppy disk drive, CD-ROM drive, flash memory, or other types of non-volatile data storage, and may correspond with various databases or other resources. The processor  414  may execute instructions to perform the method steps described herein. These instructions may be stored in the memory  402 , the secondary storage  412 , or received from the Internet or other network. The input &amp; display devices  416  may include, respectively, any device for entering data into the computer  400 , such as a keyboard, keypad, cursor-control device, touch-screen (possibly with a stylus), or microphone, and any type of device for presenting a visual image, such as, for example, a computer monitor, flat-screen display, or display panel. An output device connected to the input/output cards  408  may include any type of device for presenting data in hard copy format, such as a printer, and other types of output devices including speakers or any device for providing data in audio form. The computer can possibly include multiple input devices, output devices, and display devices. 
     Although the computer is depicted with various components, one skilled in the art will appreciate that the computer can contain additional or different components. In addition, although aspects of an implementation consistent with the method for providing physically separated compute and I/O resources in the datacenter to enable space and power savings are described as being stored in memory, one skilled in the art will appreciate that these aspects can also be stored on or read from other types of computer program products or computer-readable media, such as secondary storage devices, including hard, disks, floppy disks, or CD-ROM; or other forms of RAM or ROM. The computer-readable media may include instructions for controlling the computer to perform a particular method. 
     The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the invention as defined in the following claims, and their equivalents, in which all terms are to be understood in their broadest possible sense unless otherwise indicated.