# Evolution of System Embedded Optical Interconnect in Sub-Top-of-Rack Data Center Systems
## 8 Copackaged Optical Assembly--The Ultimate Migration of System-Level Optical Interconnect  
Hyperscale data centers are very large data centers typically comprising at least hundreds of thousands of servers and associated storage and networking capacity, and they are run by major internet content providers, such as Google, Microsoft, Amazon and Alibaba, to provide \"cloud\" services. By 2021 hyperscale data centers have become the dominant form of data center in the world, overtaking private and enterprise data centers, with most organizations outsourcing their data storage requirements to these highly secure facilities with guaranteed quality of service. Fiber-optic infrastructure is already well established in hyperscale data centers to connect together thousands of racks filled with servers, storage arrays and switch enclosures. External fiber-optic connections are made to optical transceivers, which are plugged onto the \"face-plate\" of such enclosures, hence the term face-plate pluggable (FPP) modules; however, the exponential increase in data consumption is pushing the bandwidth of data center switch enclosures to the point where there is now not enough space on the face-plate for the transceivers to meet this demand. As shown in Figure 16g, data center switch ASICs of 51 Tbps are expected to be introduced by 2024, and those of 100 Tbps are expected before 2027, which represent substantially higher bandwidths than the bisection bandwidths reported in the demonstration platforms in this paper. There are now massive efforts underway to move optical transceiver cores or \"chiplets\" into the switch ASIC package itself, in so-called copackaged optical (CPO) assemblies as shown in Figure 16a [29]. As reported in this paper, migration of optical interconnect to the system, board and chip level will be required to substantially mitigate the communications bottlenecks resulting from the increase in both per-lane bandwidth and internal interconnect link lengths. At the board level, this migration will see the development of electro-optical circuit boards comprising detachable fiber shuffles, but in the future, embedded polymer or glass  
Figure 15: Converged rack-scale data center system [23]. waveguides [30] may play an increasing role if the propagation losses can be reduced to be comparable to those of fiber. While front-pluggable and midboard transceivers are evolving, at the chip level, photonic integrated circuit technology (e.g., silicon photonics) has opened the door to the deployment of microtransceiver chiplets (Figure 16d-f) in CPO modules. Front-panel, midplane, backplane and midboard passive optical connectors will be an essential part of this ecosystem connecting system to board to chip. In this paper we have reported on the first generation of midboard transceivers introduced in 2012; however, by 2020 these fixed modules had become largely obsolete due to lack of field replaceability. The second generation of midboard transceivers specified by the Consortium for On-Board Optics (COBO) are pluggable onto the middle of the board [34]; however, as they are still in separate modules, they would take up a lot of valuable board area. Therefore, it is expected that in such switch enclosures midboard transceivers will be bypassed altogether with a combination of FPPs and CPO modules used to accommodate the huge data center switch bandwidths. Currently, a 12.8 Tbps switch can be accommodated by 32 400 G face-plate pluggable (FPP) transceiver modules, which fill the face-plate area of a standard 1 RU high enclosure.