# Evolution of System Embedded Optical Interconnect in Sub-Top-of-Rack Data Center Systems
###### Abstract  
In this paper we review key technological milestones in system embedded optical interconnects in data centers that have been achieved between 2014 and 2020 on major European Union research and development projects. This includes the development of proprietary optically enabled data storage and switch systems and optically enabled data storage and compute subsystems. We report on four optically enabled data center system demonstrators: LightningValley, ThunderValley2, Pegasuswhere customers can outsource their data storage and increased computing requirements to very large and secure data centers typically comprising at least 100,000 servers and associated data storage and network switching. These \"hyperscale\" data centers are run by internet content providers (ICPs) such as Amazon, Google and Microsoft and provide the dedicated compute, storage and server equipment required to meet the remote and diverse data processing and storage requirements of cloud environments. However, in order to cope with rapidly changing customer demand, the architectures underlying the data centers themselves need to evolve, and a critical part of that evolution is the deployment of optical connections at all levels of the data center environment. Printed circuit boards (PCBs) are at the heart of all modern information and communication technology (ICT) systems. The increase in data communication speeds incurs a toll on ICT systems, such as servers and switches, when higher frequency electronic signals are conveyed along the metal channels used in conventional PCBs. As frequencies of these electronic signals increase, dielectric absorption, skin effect and other resistive loss mechanisms attenuate them more strongly, while signal reflections, signal skew and interference from other electronic channels distort their integrity. Furthermore, the environmental effects of system operation, such as temperature and humidity, cause changes in the circuit board substrate, thus altering the carefully balanced characteristics of the electronic channels. Many of these constraints can be mitigated to some degree, however, at an ever-mounting cost to the overall system design and with an increasing power penalty. Embedded optical interconnect technologies, whether deployed at the cable level, circuit board level or chip level, offer significant performance and power advantages over conventional electronic interconnect. Performance gains include higher data rates, reduced electromagnetic interference, reduced power consumption, higher channel density and a corresponding reduction in the amount of cable or PCB materials used. Therefore, in order to cope with the exponential increase in capacity, processing power and bandwidth density inside information communication systems, there has been a trend over the past decade to migrate optical channels down from the higher communication tier optical fiber networks into the data communication system enclosure itself. One area in which this is particularly apparent is in modern data centers, where the migration of optical interconnect into top-of-rack (TOR) or other network switch enclosures can substantially mitigate the communications bottlenecks resulting from the increase in both data rate and internal interconnect link lengths. As is discussed toward the end of the paper, a great deal of research is currently underway into \"copackaged optics\" (CPO) for data center switch ASICs, which have aggregated bandwidths expected to exceed 100 Tb/s by 2026. Intel has been leading the effort to develop CPO based on silicon photonics microtracnseivers or \"chiplets\" coassembled with switch ASICs onto common carriers [1; 2; 3] to accommodate these astronomical bandwidths in data center switch ASICs. However, the vast majority of systems in data centers, what we refer to as sub-TOR systems, are servers and data storage arrays, which in hyperscale data centers number in the hundreds of thousands.