TUNABLE STACK-UP DIMM FORM FACTOR COLD PLATE WITH EMBEDDED PELTIER DEVICES FOR ENHANCED COOLING CAPABILITY

An apparatus is described. The apparatus includes a metallic chamber having a first outer surface with first Peltier devices and a second outer surface with second Peltier devices. The first and second outer surfaces face in opposite directions such that the first Peltier devices are to cool first semiconductor chips that face the first outer surface and the second Peltier devices are to cool second semiconductor chips that face the second outer surface.

BACKGROUND

System design engineers face challenges, especially with respect to high performance data center computing, as both computers and networks continue to pack higher and higher levels of performance into smaller and smaller packages. Creative packaging and cooling solutions are therefore being designed to keep pace with such aggressively designed systems.

DETAILED DESCRIPTION

FIGS. 1athrough 1eshow different views of a cold plate101having embedded Peltier devices102_1through102_4. As described in more detail further below, the Peltier devices102_1through102_4make thermal contact with the package lids of a Dual In-Line Memory Module's (DIMM's) semiconductor chips thereby providing enhanced/active cooling capacity. Here, thermal contact means actual contact or a thermal resistance that is equal to or less than that achieved with actual contact (e.g., by placing a thermal interface material between the Peltier devices and the semiconductor chips).

FIG. 1ashows a first side view whileFIG. 1bshows a second side view.FIG. 1c, shows a system integration view in which the cold plate101is placed between neighboring DIMMs121_1,121_2. Here, the cold plate101has a DIMM-like form factor so that it can be readily inserted between the neighboring DIMMs121_1,121_2.

As can be seen inFIG. 1c, the Peltier devices102are thermally coupled to the package lids of the DIMMs' semiconductor chips120. A Peltier device is typically composed of a multi-layer structure that includes an n type semiconductor and a p type semiconductor. When a DC current is driven through the multi-layer structure, one side of the Peltier device becomes hot while the other side of the Peltier devices become cold. Essentially, in response to the DC current, the Peltier device transfers heat from the cold side to the hot side.

The Peltier devices102on the cold plate101are therefore arranged such that their cold side is facing outward and in thermal contact with the DIMM semiconductor chips120while their hot side is facing inward and in thermal contact with a cold plate101.

As observed in the particular embodiment ofFIGS. 1a, 1band 1cthere are multiple Peltier devices102_1through102_4on each side of the cold plate101and each Peltier device is positioned to be in thermal contact with a different DIMM semiconductor chip package lid. For instance, in the specific example ofFIGS. 1a, 1band 1c, each side of a DIMM has four semiconductor memory chip packages and there are four Peltier devices102_1through102_4on the corresponding side of the cold plate for each such memory chip package.

Four memory chips per DIMM side is exemplary for ease of drawing. In practice there can be multiple rows of more than four memory chip on a single side of a DIMM. For instance, a DIMM can have four rows of sixteen X4 memory chips per DIMM side for a total of 64 memory chips per DIMM side. In this case there can be 64 Peltier devices per DIMM side. Alternatively, a single Peltier device can be designed to be in thermal contact with more than one DIMM semiconductor chip. For example, a larger surface area Peltier device could be placed in thermal contact with, e.g., two or four memory chips which reduces the number of Peltier devices on the side of the cold plate (e.g., from64to32or16).

In operation, heat generated by the DIMM's semiconductor chips120are transferred to the chips' respective package lids. The heat is then transferred to the cold side of the Peltier devices102which transfer the heat to the hot side of the Peltier devices102. The heat from the hot side of the Peltier devices102is then transferred to a cold plate101.

The cold plate101, in various embodiments, is a hollow chamber through which fluid continually flows. Here, referring toFIG. 3a, the cold plate101includes a cooled fluid input103_1and a warmed fluid output103_2. Cooled fluid enters the cooled fluid input103_1and flows through the cold plate101. While the fluid is flowing through the cold plate101it absorbs heat provided to the cold plate101by the hot sides of the Peltier devices102_1through102_4. As discussed at length above, the heat provided by the Peltier devices102_1through102_4is largely the heat generated by the corresponding DIMM semiconductor chips120.

With the fluid absorbing heat generated by the DIMM semiconductor chips while it flows through the cold plate101, the fluid will exit the cold plate101as warmed fluid from warmed fluid output port103_2. The warmed fluid, after it exits the cold plate101, flows to a cooling apparatus of some kind which removes the heat from the fluid and converts the fluid back to cooled fluid. The cooled fluid is then routed back to the cooled fluid input103_1of the cold plate101and the process repeats.

Notably, as described above, each Peltier device102_1through102_4has a DC current driven through it so that the Peltier devices exhibit the desired heat transfer effect. As such,FIG. 1ashows a pair of electrical wires104running to each Peltier device102_1through102_4from electrical edge connector tabs105_1,105_2(for ease of drawing, only one wire is labeled with reference number104). Per Peltier device, a first wire receives DC current from the host through edge connector tab105and injects the Peltier device with the DC current. A second wire receives the DC current from the Peltier device and transports it back to the edge connector tab.

In the specific embodiment ofFIG. 1a, there are four Peltier devices on both sides of the cold plate. The left side edge connector tab105_1includes the wire connections for the “left hand side” Peltier devices on both the observed front and non-observed back of the cold plate. The right side edge connector tab105_2includes the wire connections for the “right hand side” Peltier devices on both the observed front and non-observed back of the cold plate.

For illustrative ease none ofFIG. 1a, 1bor1cshow the mechanical, electrical or fluidic connections between the host (e.g., the PC motherboard of a computer) and the cold plate101.

FIGS. 1dand 1e, by contrast, depict an embodiment for these connections. Specifically, to address the fluidic connections, the cooled fluid input port103_1plugs into a cooled fluid manifold106and the warmed fluid output port103_2plugs into a warmed fluid manifold107. To address the electrical connections, the edge connector tabs105_1,105_2plug into electrical sockets/connectors108_1,108_2.

With respect to the fluidic connections, importantly, both manifolds106,107run an extended distance before and after the location of the cold plate101so that other cold plates (not shown inFIG. 1e) can also plug into them. Said another way, the manifolds106,107are arranged to run off both edges of a bank of DIMMs with interleaved cold plates between them. DIMMs can plug into connectors on either side of the cold plate resulting in the structure ofFIG. 1c. Here, DIMM connectors109_2,109_3off the “right side” of cold plate101are visible inFIG. 1e, whereas, DIMM connector109_1off the “left side” of the cold plate101and DIMM connector109_2off the right side of cold plate101are visible inFIG. 1c.

Importantly, another cold plate can be placed in thermal contact with one of the exposed DIMM sides ofFIG. 1cand the resulting DIMM/cold plate/DIMM/cold plate structure can be repeated to realize an entire bank of DIMMs with interleaved cold plates. Here, the manifolds106,107are extended from their depiction inFIG. 1eso that they run off the edges of the DIMMs and cold plates in the DIMM bank so that each DIMM's fluidic input and output ports103_1,103_2can plug into the corresponding manifolds106,107.

In operation, cooled fluid runs through the cooled fluid manifold106and feeds cooled fluid to each DIMM's cooled fluid input port103_1. The DIMMs warm the respective fluid that flows through them. The DIMMs then collectively emit warmed fluid from their warm fluid output ports103_2into the warmed fluid manifold107. The warmed fluid in warmed fluid manifold107is routed to a liquid cooling apparatus of some kind which converts the warmed fluid to cooled fluid. The cooled fluid is returned to the cooled fluid manifold106and the process repeats.

With respect to the electrical connections,FIGS. 1dand 1edepict the host side electrical connectors108_1,108_2into which the cold plate's electrical tabs105_1,105_2. Here, tabs105_1,105_2are essentially edge connectors that slide into connectors108_1,108_2. The host board110includes the circuitry (e.g., current sources) to drive the DC currents through the Peltier devices. As such, these currents first flow into the host connectors108_1,108_2and then the tabs105_1,105_2, and then through the Peltier devices102_1through102_4. The currents are then returned from the Peltier devices back through the edge connectors105_1,105_2and into host connectors108_1,108_2.

Again, only four Peltier devices per side are depicted. In other embodiments, many more Peltier devices can be present per side. As such, more than two edge connector tabs105_1,105_2may extend from the cold plate (or, only one longer edge connector tab extends from the cold plate).

Regardless, in various embodiments, the fluidic and electrical connectors are sufficient mechanical connections between the host PC board110and cold plate101to rigidly support the location and placement of the cold plate101in the system.

With an overview of the cold plate101, its operation and system integration having been described, various additional features of the cold plate's structure and fabrication are next described.

FIGS. 2athrough 2gdepict a design and manufacturing process for the cold plate surface. For ease of discussion, the processing associated with only one Peltier device and its associated wiring is presented.

FIG. 2ashows a cold plate surface201prior to the placement and attachment of a Peltier device and its associated wiring. Here, the left hand side ofFIG. 2adepicts a top down view whereas the right hand side depicts a side view along ray221. As observed in the top down view there are grooves or depressions202formed in the surface of the cold plate where the Peltier devices and its wires are to be placed.

Here, in various embodiments the cold plate is metallic (composed of a metal (e.g., copper) or metal alloy (e.g., bronze)) so that it transfers heat from the Peltier devices to the internal liquid with little thermal resistance. The depressions202can be formed, e.g., by a mold that shapes the cold plate surface during manufacturing of the cold plate, or, the depressions202can be etched into the cold plate's surface after the cold plate manufacturing process has yielded a smooth, flat surface cold plate surface.

FIG. 2bpresents the same cold plate surface with the same depressions as inFIG. 2abut where the right hand side view is looking along a different ray222. Here, the right hand side ofFIG. 2alooks along ray221which follows the depression trenches end to end, whereas, the right hand side ofFIG. 2bdoes not look along the depression trenches. As such, the right hand side ofFIG. 2ashows the surface of the depressions whereas the right side ofFIG. 2bshows the surface of the cold plate where no depression exists.

The followingFIGS. 2cthrough 2fare viewed along the ray221that follows the depression(s) made for the wires and Peltier device.

As observed inFIG. 2c, an epoxy223is coated inside the depressions where the wires are to be placed. In various embodiments, the epoxy223not only has the epoxy-like characteristics (e.g., is applied as a gel and then can be hardened/cured) but also has high electrical resistance (is a dielectric) and has some appreciable elasticity. Here, as will be more clear in the following discussion, the dielectric property of the epoxy223electrically isolates the wires from the conductive cold plate201while the elastic property helps preserve mechanical integration of the wiring by absorbing any thermal mismatch differences between the cold plate and the wires.

As observed inFIG. 2d, wires224are placed into the epoxy223before the epoxy has been cured (e.g., while the epoxy is a gel). Here, the wires224can be rigid metallic strips or thinner strips of metallic foil (e.g., composed of metal or a metal alloy). Notably, looking at the top down view on the left hand side ofFIG. 2d, the wires224are thinner than the width of the depressions so that the epoxy223resides not only beneath the wires224(as observed in the right hand side ofFIG. 2d) but also along the sides of the wires224.

The epoxy223is then allowed to cure (harden) which secures the wires224to the cold plate. As mentioned above, even after hardening the epoxy223has some elasticity to absorb differences in thermal coefficient expansion between the wires224and the cold plate201. The epoxy223can be implemented with various epoxies that are soft/flexible after curing.

As observed inFIG. 2e, the Peltier device225is next placed in the depression that is reserved for the Peltier device. The Peltier device225has electrical leads226that are approximately aligned with (e.g., slightly higher than) the wires224. The electrical226leads are then soldered to the wires224. In an embodiment, the Peltier device225is firmly pressed down upon into the cold plate before the leads226are bent and/or soldered to the wires224which coarsely keeps the Peltier device225against into the cold plate after the solder has reflowed and hardened.

After the leads226have been soldered to the wires224, referring toFIG. 2f, the Peltier device225is again firmly pressed into the cold plate surface and the depressed regions surrounding the Peltier device225and the wires224are covered with another layer of epoxy227.

The second layer227, when applied, has some viscosity so that it can surround the leads226and the periphery of the Peltier225device along the depression floor and in the volume of space in-between. The firm pressure applied to the Peltier device225is maintained during curing of the second epoxy227so that the Peltier device225continues to be firmly pressed into the cold plate after the second epoxy227has hardened. So doing keeps low thermal resistance between the Peltier device225and the cold plate. The second epoxy227is also electrically insulating in various embodiments to electrically isolate the leads226and wires224. The second epoxy227can be implemented with various epoxies including epoxies that are harder (less soft) after curing than the first epoxy223, or, epoxies that are thicker than the first epoxy223.

FIG. 2gshows another view of the structure ofFIG. 2fbut along a ray222that does not point along where the depressions were made. Here, looking at the right hand side view, some of the second epoxy227is visible. The top of the Peltier device225is nevertheless manifestly protruding from the surface of the cold plate thereby forming the leading surface that will make contact with the corresponding DIMM semiconductor chip package lids.

In an extended embodiment, before the second epoxy227is applied, the floor of the depression around the periphery of the Peltier device225is lined with the first epoxy223. The second epoxy227is then applied as described above but on top of the first epoxy223around the periphery of the Peltier device. Here, the first epoxy223helps absorb thermal expansion differences between the second epoxy227and the cold plate without affecting the firm downward pressure applied to the Peltier device225.

In various embodiments, cold plates are manufactured by forming two separate plate halves (one for each side), forming and processing the halves as described above and then sealing the halves together to form the complete cold plate (space exists between the two halves within the cold plate thereby forming the internal chamber). In various embodiments an O-ring is placed between the two halves to not only seal the chamber but also assist in the placement of the Peltier devices on the semiconductor package chip lids of the neighboring DIMMs. More details concerning such an O-ring are provided in more detail below with respect toFIGS. 4 and 5.

FIG. 3shows an internal structure of a cold plate301. As observed inFIG. 3, the cold plate301includes internal fins302that form a structured fluidic channel within the cold plate301that improves the efficiency of the cooling across the surface area of the Peltier devices (which are on the other side of the cold plate wall).

More particularly, the fins302form an alternating series of upward and downward vertical flows within the cold plate301. Here, by designing the locations of the fins302such that a Peltier device is centered within a particular vertical or downward flow, each Peltier device will receive a full “wash” of the cold plate's fluid on the underside of the region of the cold plate where it is located. This helps improve the heat transfer from the Peltier device to the fluid.

FIG. 4shows an exploded view of additional possible features of the cold plate. As observed inFIG. 4, the cold plate is implemented as left401_1and right401_1portions that are sealed together to form a complete cold plate401which internal fluidic channels. An O-ring411is seated between the portions401_1,401_2. The cold plate halves401_1have threaded holes in the corners. Screws412are threaded into the through holes and the cold plate portions401_1,401_2are compressed together with the O-ring in between411. The O-ring411preserves the mechanical integrity of the fluidic channels within the cold plate (prevents leaks) by elastically filling various pits/grooves or other imperfections in the cold plates that would by themselves result in a leak.

The O-ring411can also add a dimension of mechanical tolerance in applications having, e.g., aggressively small spacings between DIMMs. Specifically, the screws can be tightened further to compress the cold plate portions401_1,401_2closer together (the O-ring411becomes more compressed) thereby reducing the width of the cold plate. By contrast, if the mechanical spacings are more forgiving, the screws need not be tightened as much which relaxes the compression of the O-ring411and causes the portions401_1,401_2to expand away from each other slightly thereby expanding the width of the cold plate. In either the extremely compressed or less compressed scenarios the O-ring411prevents leaks.

This later technique of expanding the cold plate portions401_1,401_2away from one another can also be used to ensure good thermal contact between the Peltier devices and the chips on the cold plate's neighboring DIMMs. For example, in order to mechanically integrate the cold plate in between neighboring DIMMs, the screws are tightly torqued to compress the cold plate portions tightly together thereby minimizing the width of the DIMM (from outer surface to outer surface).

The minimal DIMM width allows the DIMM to be easily inserted in between two neighboring DIMMs. After the cold plate has been fixed in place (e.g., secured to the manifolds and electrical host connectors) the screws are loosened which cause the cold plate portions401_1,401_2to expand away from each other until their Peltier devices make thermal contact with their corresponding DIMMs (the O-ring expands in response to the loosening of the screws which causes the cold plate portions401_1,401_2to move away from each other).

FIG. 5shows a related mechanical integration approach in which the cold plates for, e.g., an entire DIMM bank, are “stacked up” with their interleaved DIMMs in between and mounted together as a single cohesive unit (for ease of drawingFIG. 5shows the stack of cold plates without the interleaving DIMMs in between the cold plates). The screws are tightened to secure the stack and DIMMs in position such that the DIMMs are aligned with their host board connectors (and the cold plates are aligned with their host board connectors).

The entire unit (multiple DIMMs interleaved with multiple cold plates) is then plugged into the host board (the multiple DIMMS and cold plates are plugged into the host system in parallel). The screws can then be adjusted as described above to ensure the Peltier devices are in contact with the chip lids of their neighboring DIMMs.

Note that the particular embodiments ofFIGS. 4 and 5suggest the fluidic input and output ports do not plug into manifolds that run along the bottoms of the cold plates but rather attach to conduits (not shown) that are located above the cold plates (looking atFIGS. 4 and 5, the fluidic input and output ports are on the upper edges of the cold plate(s)). Similar (e.g., upper edge) connections can also exist with respect to the electrical connections. Thus, the stacked-up and O-ring approaches described just above are not limited to a cold plate having bottom side fluidic and/or electrical connections as discussed above with respect toFIGS. 1athrough1e.

In various embodiments, the above described cold plate embodiments, whether individually connected to the host PC board as described above with respect toFIGS. 1athrough 1e, or, in a stack-up solution as described above with respect toFIG. 5, are capable of being inserted between tightly pitched DIMM solutions such as a Joint Electron Device Engineering Council (JEDEC) dual data rate “5” (DDR5) solution having a 300 mm pitch between neighboring DIMMs (the cold plate can be properly position within the 300 mm spacing).

Although the above teachings have emphasized a cold plate in which liquid coolant continually flows through the cold plate, in other embodiments, conceivably, a vapor chamber could be used instead of a cold plate (a so called “two phase” cooling approach). In a vapor chamber approach, the vapor chamber is structured similar to the above described cold plate except that the liquid within the chamber boils in response to the heat that is received from the Peltier devices. The boiling creates vapor. The vapor is then cooled (condensed) back to a liquid thereby removing heat from the liquid. The vapor can be condensed entirely within the DIMM form factor chamber in which case fluidic connections need not be made to the chamber. Alternatively, or in combination, the vapor can be vented from the DIMM form factor chamber (e.g., into manifolds), condensed externally, and the resulting cooled liquid returned to the chamber (through a cooled fluid input conduit/connection).

The teachings above can be applied to the cooling apparatus600ofFIG. 6.FIG. 6depicts a general cooling apparatus600whose features can be found in many different kinds of semiconductor chip cooling systems. As observed inFIG. 6, one or more semiconductor chips within a package602(such as memory chips) are mounted to an electronic circuit board601(such as a DIMM). A cold plate603, such as the cold plates described above, is thermally coupled with the package602(e.g., by way of a Peltier device608that is attached to the cold plate) so that the cold plate603receives heat generated by the one or more semiconductor chips (the cold plate603can also be referred to as a vapor chamber in the case of two-phase cooling systems).

Liquid coolant is within the cold plate603. If the system also employs air cooling (optional), a heat sink604can be thermally coupled to the cold plate603. Warmed liquid coolant and/or vapor605leaves the cold plate603to be cooled by one or more items of cooling equipment (e.g., heat exchanger(s), radiator(s), condenser(s), refrigeration unit(s), etc.) and pumped by one or more items of pumping equipment (e.g., dynamic (e.g., centrifugal), positive displacement (e.g., rotary, reciprocating, etc.))606. Cooled liquid607then enters the cold plate603and the process repeats.

With respect to the cooling equipment and pumping equipment606, cooling activity can precede pumping activity, pumping activity can precede cooling activity, or multiple stages of one or both of pumping and cooling can be intermixed (e.g., in order of flow: a first cooling stage, a first pumping stage, a second cooling stage, a second pumping stage, etc.) and/or other combinations of cooling activity and pumping activity can take place.

Moreover, the intake of any equipment of the cooling equipment and pumping equipment606can be supplied by the cold plate of one semiconductor chip package or the respective cold plate(s) of multiple semiconductor chip packages.

In the case of the later (intake received from cold plate(s) of multiple semiconductor chip packages), the semiconductor chip packages can be components on a same electronic circuit board or multiple electronic circuit boards. In the case of the later (multiple electronic circuit boards), the multiple electronic circuit boards can be components of a same electronic system (e.g., different boards in a same server computer) or different electronic systems (e.g., electronic circuit boards from different server computers). In essence, the general depiction ofFIG. 6describes compact cooling systems (e.g., a cooling system contained within a single electronic system), expansive cooling systems (e.g., cooling systems that cool the components of any of a rack, multiple racks, a data center, etc.) and cooling systems in between.

AlthoughFIG. 6shows the Peltier device608in direct contact with a semiconductor chip package, in other embodiments one or more intervening structure(s) can exist along the thermal path between the Peltier device and the semiconductor chip package (e.g., thermal interface material (TIM)).

The following discussion concerningFIGS. 7, 8 and 9are directed to systems, data centers and rack implementations, generally. As such,FIG. 7generally describes possible features of an electronic system that can include one or more DIMMs that are cooled according to the teachings above.FIG. 8describes possible features of a data center that include such electronic systems.FIG. 9describes possible features of a rack having one or more such electronic systems installed into it.

FIG. 7depicts an example system. System700includes processor710, which provides processing, operation management, and execution of instructions for system700. Processor710can include any type of microprocessor, central processing unit (CPU), graphics processing unit (GPU), processing core, or other processing hardware to provide processing for system700, or a combination of processors. Processor710controls the overall operation of system700, and can be or include, one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), or the like, or a combination of such devices.

Certain systems also perform networking functions (e.g., packet header processing functions such as, to name a few, next nodal hop lookup, priority/flow lookup with corresponding queue entry, etc.), as a side function, or, as a point of emphasis (e.g., a networking switch or router). Such systems can include one or more network processors to perform such networking functions (e.g., in a pipelined fashion or otherwise).

In one example, system700includes interface712coupled to processor710, which can represent a higher speed interface or a high throughput interface for system components that needs higher bandwidth connections, such as memory subsystem720or graphics interface components740, or accelerators742. Interface712represents an interface circuit, which can be a standalone component or integrated onto a processor die. Where present, graphics interface740interfaces to graphics components for providing a visual display to a user of system700. In one example, graphics interface740can drive a high definition (HD) display that provides an output to a user. High definition can refer to a display having a pixel density of approximately 100 PPI (pixels per inch) or greater and can include formats such as full HD (e.g., 1080p), retina displays, 4K (ultra-high definition or UHD), or others. In one example, the display can include a touchscreen display. In one example, graphics interface740generates a display based on data stored in memory730or based on operations executed by processor710or both. In one example, graphics interface740generates a display based on data stored in memory730or based on operations executed by processor710or both.

Memory subsystem720represents the main memory of system700and provides storage for code to be executed by processor710, or data values to be used in executing a routine. Memory subsystem720can include one or more memory devices730such as read-only memory (ROM), flash memory, volatile memory, or a combination of such devices. Memory730stores and hosts, among other things, operating system (OS)732to provide a software platform for execution of instructions in system700. Additionally, applications734can execute on the software platform of OS732from memory730. Applications734represent programs that have their own operational logic to perform execution of one or more functions. Processes736represent agents or routines that provide auxiliary functions to OS732or one or more applications734or a combination. OS732, applications734, and processes736provide software functionality to provide functions for system700. In one example, memory subsystem720includes memory controller722, which is a memory controller to generate and issue commands to memory730. It will be understood that memory controller722could be a physical part of processor710or a physical part of interface712. For example, memory controller722can be an integrated memory controller, integrated onto a circuit with processor710. In some examples, a system on chip (SOC or SoC) combines into one SoC package one or more of: processors, graphics, memory, memory controller, and Input/Output (I/O) control logic circuitry.

In various implementations, memory resources can be “pooled”. For example, the memory resources of memory modules installed on multiple cards, blades, systems, etc. (e.g., that are inserted into one or more racks) are made available as additional main memory capacity to CPUs and/or servers that need and/or request it. In such implementations, the primary purpose of the cards/blades/systems is to provide such additional main memory capacity. The cards/blades/systems are reachable to the CPUs/servers that use the memory resources through some kind of network infrastructure such as CXL, CAPI, etc.

Any of the memory solutions described above can be implemented with DIMMs that are cooled according to the teachings above.

While not specifically illustrated, it will be understood that system700can include one or more buses or bus systems between devices, such as a memory bus, a graphics bus, interface buses, or others. Buses or other signal lines can communicatively or electrically couple components together, or both communicatively and electrically couple the components. Buses can include physical communication lines, point-to-point connections, bridges, adapters, controllers, or other circuitry or a combination. Buses can include, for example, one or more of a system bus, a Peripheral Component Interconnect express (PCIe) bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, Remote Direct Memory Access (RDMA), Internet Small Computer Systems Interface (iSCSI), NVM express (NVMe), Coherent Accelerator Interface (CXL), Coherent Accelerator Processor Interface (CAPI), Cache Coherent Interconnect for Accelerators (CCIX), Open Coherent Accelerator Processor (Open CAPI) or other specification developed by the Gen-z consortium, a universal serial bus (USB), or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus.

In one example, system700includes interface714, which can be coupled to interface712. In one example, interface714represents an interface circuit, which can include standalone components and integrated circuitry. In one example, multiple user interface components or peripheral components, or both, couple to interface714. Network interface750provides system700the ability to communicate with remote devices (e.g., servers or other computing devices) over one or more networks. Network interface750can include an Ethernet adapter, wireless interconnection components, cellular network interconnection components, USB (universal serial bus), or other wired or wireless standards-based or proprietary interfaces. Network interface750can transmit data to a remote device, which can include sending data stored in memory. Network interface750can receive data from a remote device, which can include storing received data into memory. Various embodiments can be used in connection with network interface750, processor710, and memory subsystem720.

In one example, system700includes one or more input/output (I/O) interface(s)760. I/O interface760can include one or more interface components through which a user interacts with system700(e.g., audio, alphanumeric, tactile/touch, or other interfacing). Peripheral interface770can include any hardware interface not specifically mentioned above. Peripherals refer generally to devices that connect dependently to system700. A dependent connection is one where system700provides the software platform or hardware platform or both on which operation executes, and with which a user interacts.

In one example, system700includes storage subsystem780to store data in a nonvolatile manner. In one example, in certain system implementations, at least certain components of storage780can overlap with components of memory subsystem720. Storage subsystem780includes storage device(s)784, which can be or include any conventional medium for storing large amounts of data in a nonvolatile manner, such as one or more magnetic, solid state, or optical based disks, or a combination. Storage784holds code or instructions and data in a persistent state (e.g., the value is retained despite interruption of power to system700). Storage784can be generically considered to be a “memory,” although memory730is typically the executing or operating memory to provide instructions to processor710. Whereas storage784is nonvolatile, memory730can include volatile memory (e.g., the value or state of the data is indeterminate if power is interrupted to system700). In one example, storage subsystem780includes controller782to interface with storage784. In one example controller782is a physical part of interface714or processor710or can include circuits in both processor710and interface714.

Such non-volatile memory devices can be placed on a DIMM and cooled according to the teachings above.

In an example, system700can be implemented as a disaggregated computing system. For example, the system700can be implemented with interconnected compute sleds of processors, memories, storages, network interfaces, and other components. High speed interconnects can be used such as PCIe, Ethernet, or optical interconnects (or a combination thereof). For example, the sleds can be designed according to any specifications promulgated by the Open Compute Project (OCP) or other disaggregated computing effort, which strives to modularize main architectural computer components into rack-pluggable components (e.g., a rack pluggable processing component, a rack pluggable memory component, a rack pluggable storage component, a rack pluggable accelerator component, etc.).

Although a computer is largely described by the above discussion ofFIG. 7, other types of systems to which the above described invention can be applied and are also partially or wholly described byFIG. 7are communication systems such as routers, switches and base stations.

FIG. 8depicts an example of a data center. Various embodiments can be used in or with the data center ofFIG. 8. As shown inFIG. 8, data center800may include an optical fabric812. Optical fabric812may generally include a combination of optical signaling media (such as optical cabling) and optical switching infrastructure via which any particular sled in data center800can send signals to (and receive signals from) the other sleds in data center800. However, optical, wireless, and/or electrical signals can be transmitted using fabric812. The signaling connectivity that optical fabric812provides to any given sled may include connectivity both to other sleds in a same rack and sleds in other racks.

Data center800includes four racks802A to802D and racks802A to802D house respective pairs of sleds804A-1and804A-2,804B-1and804B-2,804C-1and804C-2, and804D-1and804D-2. Thus, in this example, data center800includes a total of eight sleds. Optical fabric812can provide sled signaling connectivity with one or more of the seven other sleds. For example, via optical fabric812, sled804A-1in rack802A may possess signaling connectivity with sled804A-2in rack802A, as well as the six other sleds804B-1,804B-2,804C-1,804C-2,804D-1, and804D-2that are distributed among the other racks802B,802C, and802D of data center800. The embodiments are not limited to this example. For example, fabric812can provide optical and/or electrical signaling.

FIG. 9depicts an environment900that includes multiple computing racks902, each including a Top of Rack (ToR) switch904, a pod manager906, and a plurality of pooled system drawers. Generally, the pooled system drawers may include pooled compute drawers and pooled storage drawers to, e.g., effect a disaggregated computing system. Optionally, the pooled system drawers may also include pooled memory drawers and pooled Input/Output (I/O) drawers. In the illustrated embodiment the pooled system drawers include an INTEL® XEON® pooled computer drawer908, and INTEL® ATOM™ pooled compute drawer910, a pooled storage drawer912, a pooled memory drawer914, and a pooled I/O drawer916. Each of the pooled system drawers is connected to ToR switch904via a high-speed link918, such as a 40 Gigabit/second (Gb/s) or 100 Gb/s Ethernet link or an 100+ Gb/s Silicon Photonics (SiPh) optical link. In one embodiment high-speed link918comprises an 600 Gb/s SiPh optical link.

Again, the drawers can be designed according to any specifications promulgated by the Open Compute Project (OCP) or other disaggregated computing effort, which strives to modularize main architectural computer components into rack-pluggable components (e.g., a rack pluggable processing component, a rack pluggable memory component, a rack pluggable storage component, a rack pluggable accelerator component, etc.).

Multiple of the computing racks900may be interconnected via their ToR switches904(e.g., to a pod-level switch or data center switch), as illustrated by connections to a network920. In some embodiments, groups of computing racks902are managed as separate pods via pod manager(s)906. In one embodiment, a single pod manager is used to manage all of the racks in the pod. Alternatively, distributed pod managers may be used for pod management operations. RSD environment900further includes a management interface922that is used to manage various aspects of the RSD environment. This includes managing rack configuration, with corresponding parameters stored as rack configuration data924.

Any of the systems, data centers or racks discussed above, apart from being integrated in a typical data center, can also be implemented in other environments such as within a bay station, or other micro-data center, e.g., at the edge of a network.

Some examples may be implemented using or as an article of manufacture or at least one computer-readable medium. A computer-readable medium may include a non-transitory storage medium to store program code. In some examples, the non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. In some examples, the program code implements various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, API, instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof.

To the extent any of the teachings above can be embodied in a semiconductor chip, a description of a circuit design of the semiconductor chip for eventual targeting toward a semiconductor manufacturing process can take the form of various formats such as a (e.g., VHDL or Verilog) register transfer level (RTL) circuit description, a gate level circuit description, a transistor level circuit description or mask description or various combinations thereof. Such circuit descriptions, sometimes referred to as “IP Cores”, are commonly embodied on one or more computer readable storage media (such as one or more CD-ROMs or other type of storage technology) and provided to and/or otherwise processed by and/or for a circuit design synthesis tool and/or mask generation tool. Such circuit descriptions may also be embedded with program code to be processed by a computer that implements the circuit design synthesis tool and/or mask generation tool.