# Inter-Data Center RDMA: Challenges, Status, and Future Directions
## 3 Inter-DC RDMA Requirements and Challenges  
The growing trend of DC interconnection (DCI) is driven by two key factors: the rise of inter-DC applications and resource constraints. **Emerging inter-DC applications:** Many applications, such as cloud services, data analytics, machine learning, and graph processing, involve large datasets distributed across multiple DCs. These applications commonly adhere to the \"moving compute to data\" model, necessitating efficient data transfer and processing across DCs. For instance, scientific research such as the InfiniCloud project interconnects supercomputing centers around the nation to facilitate collaborative research efforts. In the past few years, federated cloud computing has been mooted as a means to tackle the demands of data privacy management and international digital sovereignty, which also requires data processing in geographically distributed DCs. In addition to the data requirements, the substantial computing power required for LLM training in recent years has led to widespread discussions on effectively utilizing the computing resources of multiple DCs to support large-scale LLM pre-training tasks. **Resource constraints:** At the same time, the expansion of large DCs is limited by land, power, and connectivity constraints. To address this, large CSPs have adopted a strategy of using multiple interconnected DCs through dedicated fibers to serve a region rather than relying on a single mega-DC. This approach, known as DCI, involves fiber lengths ranging from tens of kilometers for city-wide interconnections to thousands of kilometers for nation-wide interconnections. For instance, China has been developing a national computing system that interconnects computing nodes and DC clusters across the country to balance resource distribution, called the China Computing Network (C\({}^{2}\)NET) [42, 43]. This system leverages computing infrastructures located in the west, where land and electricity costs are lower, to meet the computing demands of the east. Owing to the aforementioned reasons, the interconnection of multiple DCs to support various emerging applications will emerge as an important trend in the future. In recent years, with the conditions for wide-area interconnection among DCs gradually being perfected, some rely on the existing wide area network (WAN) infrastructure, while others deploy dedicated optical fibers to provide high-speed interconnection [44, 45, 46]. Thus, for inter-DC applications, directly establishing RDMA connections over DCI is highly advisable. Firstly, these services typically employ RDMA for communication within the DC due to stringent performance requirements. When these services communicate across DCs, it is better to continue using existing RDMA technology to achieve single-connection high throughput over long-distance links. Second, using RDMA can maintain API consistency and reduce the complexity of application deployment across DCs. However, deploying current DCN-specific RDMA solutions across DCs presents several significant challenges, primarily due to the differences in network conditions compared to intra-DC environments. **Congestion control:** Existing intra-DC RDMA solutions (e.g., DCQCN [25], TIMELY [29], and IRN [28]) leverage congestion control algorithms to ensure a lossless network environment. However, these algorithms struggle with the unique conditions of inter-DC networks, leading to impaired performance. Inter-DC links have much longer propagation delays compared to intra-DC links. The inter-DC latency varies from 1 millisecond to several tens of milliseconds. For instance, city-wide interconnections within 100 km result in a latency of approximately 1 millisecond, whereas some nation-wide or continent-wide interconnections can exceed 1000 km, bringing a 10-millisecond level latency [47]. This increased latency heavily affects the performance of existing RDMA congestion control algorithms such as DCQCN, TIMELY, and HPCC.