# System Innovations in Inter Data Center Transport Networks

Loukas Paraschis, Harald Bock, Parthiban Kandappan, Bernd

Sommerkorn-Krombholz, Joao Pedro, Abhinava Sadasivarao, Sharfuddin

Syed, Jeff Rahn, Paul Doolan, Biao Lu

► To cite this version:

Loukas Paraschis, Harald Bock, Parthiban Kandappan, Bernd Sommerkorn-Krombholz, Joao Pedro, et al.. System Innovations in Inter Data Center Transport Networks. 23th International IFIP Conference on Optical Network Design and Modeling (ONDM), May 2019, Athens, Greece. pp.444-451, 10.1007/978-3-030-38085-4_38. hal-03200666

HAL Id: hal-03200666

https://inria.hal.science/hal-03200666

Submitted on 16 Apr 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# System Innovations in Inter Data Center Transport Networks

Loukas Paraschis¹, Harald Bock², Parthiban Kandappan¹, Bernd Sommerkorn-Krombholz², Joao Pedro³, Abhinava Sadasivarao¹, Sharfuddin Syed¹, Jeff Rahn¹, Paul Doolan¹ and Biao Lu¹

¹ Infinera Corporation, Sunnyvale, USA

² Infinera Corporation, Munich, Germany

³ Infinera Corporation, Lisbon, Portugal
lparaschis@infinera.com

**Abstract.** We review the most important WDM DCI system innovations. State-of-the-art coherent WDM transmission has already exceeded 6 b/s/Hz, using subcarrier modulation. Moreover, the adoption of software innovations in automation and programmability, that DCI pioneered in transport networks, has also simplified operations and enables the emergence of “open” transport architectures. Combining these advancements with emerging network analytics frameworks allows exciting innovations in network design and management optimization.

**Keywords:** DCI, WDM, SDN, coherent, optical transport, network design and analytics.

## 1 Introduction

Traffic on the networks interconnecting data centers (DCs), referred to as data center interconnect (DCI) networks, has grown more than any other transport network traffic type [1, 2, 3], and has been projected to grow by at least two more orders of magnitude [3, 4]. The economics associated with this growth have motivated the building of dedicated DCI networks, and of a new class of purpose-built systems that are optimized for the DCI requirements [5, 6]. In many respects, the growth of DCI has been the most significant development in optical transport networking this decade, and its most significant evolution since the major transitions from TDM to IP/MPLS and WDM [6, 7].

While DCI has a few things in common with traditional telecommunication transport, for example most of the current WDM technology employed in DCI is the same as that used in telecom networks, the DCI transport networks have a substantial number of unique characteristics (both architectural and operational) that have motivated the development of a new class of DCI-optimized packet and optical transport systems [5, 6]. More specifically, DCI-optimized transport systems have been developed to address the DC operational environment, with requirements for lower power
and cost per Gb/s, for simpler DCI routing that focus on maximizing throughput ra-
ther than routing scale, and for high capacity, typically point-to-point, WDM systems
that maximize spectral efficiency by employing state-of-the-art coherent transmission
(summarized in Section II) [4, 5, 6].

Moreover, DCI transport networking has also pioneered the extensive adoption of
significant software innovations [6, 8-13] in programmability, automation, manage-
ment abstraction, and control-plane disaggregation, typically referred collectively as
SDN (summarized in Section III). This new DCI-optimized infrastructure is increas-
ingly being deployed globally and has leveraged SDN to enable important innovations
in open transport network architectures. In addition these new networks account for
some of the most spectrally-efficient fiber deployments, that already exceed 6 b/s/Hz
even in very long subsea routes, leveraging digital subcarriers and advancements in
photonic-integration [14, 15] and will, by 2020, improve by an additional 20% lever-
aging constellation shaping [16] and further advancements in photonic-integration
[17]. This paper summarizes the main WDM system innovations that have facilitated
this explosive DCI evolution, and then presents (in section 4) some important recent
innovations in optical transport design achieved by combining the advancements in
coherent open WDM with extensible software/SDN infrastructure. More specifical-
ly, this novel networking paradigm usually couples recent advancements in stream-
ing-telemetry methodologies [10, 13] with emerging network analytics frameworks
[13, 18]. These are often combined with machine-learning [18], to improve capacity
(e.g. optical margin) optimization [18, 19], as well as enhancing (e.g. predictive)
management and control [13, 20].

## 2 Coherent WDM Transmission

Widespread adoption of coherent communication in WDM networks has enabled significant increases in per-fiber capacities. A decade ago, on-off keying modulation could support 1-2 Tb/s on a single fiber. This increased to 5 Tb/s using the first generation of coherent signaling, and further expanded to 30-40 Tb/s with the best available technology today.

First-generation coherent systems supported QPSK modulation and hard-decision
forward error correction. Fiber capacity was increased using two orthogonal polariza-
tions of the optical signal. In addition, the ability to use coherent mixing to select a
single channel enabled inexpensive colorless de-multiplexing. Dispersion compensa-
tion could also be accomplished using digital signal processing (DSP). As a result,
flexible add/drop line systems have changed dramatically to leverage the capabilities
of coherent systems. Since the initial introduction, significant improvements have
made their way into coherent systems. The use of DSP has expanded to the transmit-
ter, enabling higher order modulation(s) than QPSK, and introducing near-Nyquist
shaping to improve spectral efficiency. ADC and DAC technology have both im-
proved dramatically. Lasers have been tailored for low linewidth to support higher
order modulation. Advanced signal processing such as Nyquist subcarriers have fur-
ther improved nonlinear performance and tolerance to dispersion. All these improve-
ments have helped increase bandwidth and drive down cost per bit.

The most spectrally-efficient deployed fiber networks today already exceed 6
b/s/Hz, leveraging digital subcarriers and advancements in photonic integration [14,
15]. By 2020, constellation shaping [16] and further advancements in photonic-
integration [17] will provide an additional 20% improvement of spectral density for a
given reach performance.

3 Software Innovations and SDN

The unique operational characteristics of the cloud DCs have also given rise to novel software requirements, and innovation opportunities for DCI [5, 6, 8-11]. More specifically, DC operators pioneered the pervasive use of DevOps and software automation techniques, initially to serve their hyperscale compute infrastructure needs [8, 11, 21, 22]. Incorporating equivalent software innovations to advance the functionality of DCI networks has led to the introduction of network programmability, automation, management abstraction, and control-plane disaggregation, often referred collectively as SDN transport [6, 8-13]. While many of these SDN innovations were initially introduced in DCI packet transport, their more recent adoption in DCI optical systems has been an even more radical innovation because traditional optical network management has previously been based on proprietary (vendor-specific) NMS [5, 9]. The most notable example has been the use of extensive API frameworks based on YANG data models, and the related NETCONF, RESTCONF, or gRPC interfaces, which we identify as model-driven networking (MDN) [9-13]. These APIs enable DCI network operators to develop new transport automation and abstraction frameworks [10, 11]. OpenConfig is one such widely adopted API [10] and is currently supported by all major DCI transport system vendors [3, 12, 13]. Other, more recent, important MDN efforts are also in progress, aiming for enhancements in the MDN robustness and functionality, beyond OpenConfig [11, 13].

MDN innovations have also catalyzed newer forms of performance monitoring, particularly streaming telemetry which was pioneered in DCI transport and aimed to resolve the limitations of the traditional SNMP data pull approaches [8, 10, 11]. These new telemetry frameworks, have enabled two important innovations in WDM systems: First, they facilitate more manageable reporting of a greater number of network and system parameters, such as transmit and receive optical power, Pre-FEC and Post-FEC statistics, amplifier parameters, dispersion, severity of alarms, client and line side laser temperatures, device/port up/down status, etc. Second, and even more important, MDN based non-proprietary frameworks allow the end-user extensive flexibility in defining the desired content (more or less info), the method (e.g. data encoding mechanism), and the granularity (from milliseconds to hours) of the network monitoring mechanisms; e.g. [12]. Such advanced monitoring flexibility also allows transport to be more effectively integrated in the network management and control planes, and more easily combined with machine-learning techniques [21], towards advanced network analytics [13, 19]. Network analytics aim to identify,
based on operator defined trigger points, potential drifts in parameters and notify net-
work operations for actionable remedial steps that would allow for sufficient time to
anticipate and plan repair maintenance and recovery, minimizing potential down
times [11, 20-22]. For DCI operators such new network analytics frameworks, based
on innovations in streaming telemetry methodologies, have been increasingly consid-
ered an important evolution of network management and mediation and have recently
been combined with cognitive systems [11, 20]. A typical cognitive system, compris-
ing of utilizing PM telemetry streaming, along with policy-based operations and
maintenance, was demonstrated in a proof-of-concept by a leading North American
service provider [20]. In that example, the system continually monitors the bandwidth
utilization and based on real-time analytics, takes policy defined action to increase the
available bandwidth by creating additional services automatically.

Along with the increased openness and programmability comes the need to ad-
vance the system control-plane and network management abstraction beyond vendor
specific, usually proprietary, implementations. This effort is part of a much wider
networking effort to adopt intent-based configuration frameworks [8], which is not
specific to transport. It requires that network layers are controlled by an SDN Control-
ler or NMS that maintains global network state and monitors the entire network (in-
cluding optical transport) for changes. Based on this global state (and the operator
intent), the Controller decides when the network needs to transition from one state to
another. In the simplest optical example of capacity expansion from A to B, where A
is a steady-state optical network operating at capacity X, and B is the optical network
with increased capacity X' > X, the Controller sends the entire configuration (includ-
ing ones that don't need change) to each NE. In such declarative configuration man-
agement (DCM), the NEs identify and apply only the required change e.g., turn-up
extra wavelengths, with the (combined) end-result being the increase in network ca-
pacity to X'. Much like the other such SDN innovations, DCM has been initially in-
troduced in network switching and routing systems. However, we consider the recent
extension of DCM, and more generally intent-based networking to WDM transport,
which was again pioneered in next generation DCI WDM systems [13], to be very
exciting because it enables network operations and capacity to be optimized (poten-
tially dynamically) based on network parameters and operator policy rules [13].

Finally, DCI network element MDN programmability is now being enhanced to
accommodate third-party software agent extensibility, which would allow applica-
tions developed by a network operator to interact with the DCI transport system NEs
[11, 13]. For example, [13] described the first to our knowledge, implementation in a
WDM system of such SW-agent based operational extensibility, which specifically
focused on network analytics applications. Note that the advancements in MDN, in-
cluding the DCM, and SW agent extensibility, benefit all transport use-cases, not just
DCI, becoming particularly valuable in improving the operational efficiency (OpEx)
of large-scale deployments [20].

Moreover, while the first explicit goal of these software innovations has been to
improve operational efficiency, and thus reduce OpEx, these innovations can also
improve CapEx by enabling open and vendor-agnostic network management. In this
sense, they are becoming the first important step towards open line-system (OLS)
architectures, which can then be combined with emerging network analytics and ma-
chine-learning, to improve capacity (e.g. optical margin) optimization [18, 19, 23],
which we discuss in the next section.

4 Innovations in Network Design and Optimization

Recently, for network planning purposes, system providers and even more DCI opera-
tors have moved away from vendor agnostic, offline planning tools towards novel
strategies to optimize the capacity-reach trade-off of their network. The original ap-
proach of stacking margins within traditional planning tools that reflect component
performance variations, system aging, network evolution and other network level
effects is truly not efficient. In order to cope with the outlined need for OpEx and
CapEx reductions a new paradigm in planning is required. Leveraging new technolo-
gies such as performance, baud- and bit-rate flexible transponders, and leveraging the
ability of streaming telemetry data enable live monitoring and Current State of Life
determination of the optical performance, providing the current present margin in the
network.

In a very recent field trial in a Tier 1 European service provider, the benefit of this
paradigm has been demonstrated [25], and the Infinera Aware technology has been
proven to address it in combination of a new class of transponder [26]. We refer to
this transponder class as autonomous intelligent transponder (AIT) as it takes ad-
vantage of the host of information today's digital signal processors (DSP) provide
such as signal-to-noise ratio (SNR), accumulated chromatic dispersion and differen-
tial group delay, to name only a few, in order to autonomously adapt transmission
parameters to the current quality of the transmission link. The main benefit of this
approach is that channel capacity (or any other suitable metric) is maximized at every
point in time without manual setup or configuration. We conducted the field trial in
Telia Carrier's live European backbone network. The bi-directional link used con-
nects Munich, Zürich, Strasbourg, and Frankfurt over Infinera's hiT 7300 multi-haul
line system. The link consists of 20 spans of standard single mode fiber (SSMF) with
a total link length of 1500 km, 14 inline amplifiers (half pure EDFA and half hybrid
EDFA Raman), and 7 ROADMs. The AITs were installed in Munich (MCH), simpli-
fying configuration and operation during the trial. Apart from other live traffic chan-
nels our channels under test occupied the spectrum from 191.6 THz to 192.0 THz in
both directions with 50 GHz channel spacing. The AIT channel was placed in the
middle with four 100 Gb/s DP-4QAM neighbor channels on either side.

The BER of the neighbor channels, which were provided by Infinera's commercial
Groove G30 platform, were monitored during the whole trial to capture any impact
from their adaptively switching AIT neighbor. We discovered that in its current im-
plementation the standalone AIT capacity solution does not achieve optimum effi-
ciency in the presence of nonlinearities coming mainly from direct neighbor channels,
i.e. it provides an OSNR margin larger than necessary. We conclude that with differ-
ent slopes of the BER vs. OSNR curves in the presence of non-linearities, the actual
margin is underestimated. We chose to estimate the amount of nonlinear distortion
and real OSNR as a result of running the Infinera Aware Technology solution [18]. We gained 1.0 dB in OSNR and 20% higher bit rate by combination of AIT and Aware in the field trial scenario.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Optimum AIT bit rate vs. launch power per channel after 1500 km plus additional 50 km NZDSF span. Orange and blue curves show AIT stand-alone results with 0.5 dB OSNR margin (blue curve with and orange curve without neighbor channels). Red markers show AIT stand-alone result with 1.5 dB OSNR margin (circle) and AIT + Aware result with 1.5 dB ASE margin (star).</figcaption></figure>

This example clearly demonstrates how this new approach to optimizing optical performance in a network can help increase available bandwidth in a given installation and so maximize network utilization. Combining this performance analytics approach with machine learning techniques will open an additional range of functions and possibilities that will support planning and operations within an open networking environment.

One specific use-case that is relevant in this context is the prediction of the optical reach of new lightpaths in disaggregated and open DWDM line systems. Usually, there is no agreed and truly valid performance model for such a multi-vendor and sometimes even multi-technology environment. Upgrading these networks with additional capacity is not straightforward as optical performance of new channels can only be predicted with very limited accuracy.

This challenge can be overcome by the combination of optical performance monitoring on the installed system with machine learning techniques. The accuracy of such predictions will obviously improve over time as the number of installed lightpaths increases, generating more training data to retrain a machine learning-based performance estimator. and can significantly simplify operational and planning procedures in open DWDM networking.

In particular, the information generated by the Infinera Aware Technology during a network optimization cycle, can be used successfully by machine learning tools to
accurately predict performance of new lightpaths in an open line system with dis-
aggregated transponders [19].

5 Conclusion

This paper summarized the evolution and innovations of purpose-built DCI transport
systems that enable some of the most spectrally-efficient fiber networks deployed
today. Most notably, state-of-the-art coherent WDM transmission has leveraged sub-
carrier modulation to exceed 6 b/s/Hz even in subsea routes. Also, it will soon incor-
porate practical implementations of constellation shaping to enhance the system per-
formance by an additional 1 dB and offer finely granular optimization of channel
capacity per optical link. At the same time, DCI networks are leading the way in the
extensive adoption of software innovations to simplify operations and enable open
transport architectures. Often collectively referred to as the introduction of SDN,
these innovations really include major steps towards increased network programma-
bility, automation, management abstraction, and control-plane disaggregation. More-
over, important new innovations in optical transport design are achieved combining
the advancements in coherent WDM with an open and extensible software/SDN infra-
structure. This novel networking paradigm usually combines recent advancements in
streaming-telemetry methodologies with emerging network analytics frameworks, and
more recently being often combined with machine-learning, to improve capacity (e.g.
optical margin) optimization, as well as network operations based on enhanced (e.g.
predictive) management and control. More specifically, the example use-cases dis-
cussed here demonstrate the promise of analytics tools as well as machine learning
technology in the operation of disaggregated DWDM environments.

6 Acknowledgement

We would like to acknowledge insightful interactions related to this work with many
colleagues in the industry and academia, and especially Anders Lindgren and Stefan
Melin at Telia Company.

References

1. U. Holzle, “A Ubiquitous Cloud Requires a Transparent Network”, IEEE/OSA Conference on Optical Fiber Communications (OFC), Plenary (2017).

2. V. Vusirikala, “SDN Enabled Programmable, Dynamic Optical Layer”, European Conference on Communications (ECOC), Plenary (2017).

3. Global Cloud Index, www.cisco.com/c/en/us/solutions/collateral/service-provider/global-cloud-index-gci/white-paper-c11-738085.html, last accessed 2019/03/21.

4. James Hamilton, “*How Many Data Centers Needed World-Wide*”, https://perspectives.mvdirona.com/2017/04/how-many-data-centers-needed-world-wide, last accessed 2019/03/24.
5. B. Koley et al., "Future Needs of WDM Transport for Inter-datacenter Interconnections", IEEE/OSA Conference on Optical Fiber Communications (OFC), M2E1 (2014).

6. L. Paraschis, R. Kannan, "Innovations in Inter Data Center Transport Networks", Chapter 17 in Optical Fiber Telecommunications VII, Elsevier, ISBN 978-0128165027 (2019).

7. L. Paraschis, "Advancements in Metro Regional and Core Transport Network Architectures for the Next-Generation Internet", Chapter 18 (pp. 793–817) in Optical Fiber Telecommunications VI Volume B, Systems and Networks, Elsevier, ISBN 978-0123969606 (2016).

8. B. Koley, "The Zero Touch Network", IEEE 12th International Conference on Network and Service Management, Keynote (2016).

9. L. Paraschis, "SDN innovations in WAN", Optical Internetworking Forum (OIF), Plenary presentation (oif2015.083) (2015).

10. A. Shaikh et al., "Vendor-neutral Network Representations for Transport SDN", IEEE/OSA Conference on Optical Fiber Communications (OFC), Th4G.3, (2016).

11. Symposium on "Transport Network Management and Analytics innovations", IEEE/OSA Conference on Optical Fiber Communications (OFC) Tu3H (2018).

12. A. Sadasivarao et al., "High Performance Streaming Telemetry in Optical Transport Networks", IEEE/OSA Conference on Optical Fiber Communications (OFC), Tu3D.3 (2018).

13. A. Sadasivarao et al., "Demonstration of Advanced Open WDM Operations and Analytics, based on an Application-Extensible, Declarative, Data Model Abstracted Instrumentation Platform", IEEE/OSA Conference on Optical Fiber Communications (OFC), M3Z.1 (2019).

14. J. Rahn et al., "DSP-Enabled Frequency Locking for Near-Nyquist Spectral Efficiency Superchannels utilizing Integrated Photonics", IEEE/OSA Conference on Optical Fiber Communications (OFC), W1B.3 (2018).

15. S. Grubb et al., "Real-time 16QAM Transatlantic Record Spectral Efficiency of 6.21 b/s/Hz Enabling 26.2 Tbps Capacity", IEEE/OSA Conference on Optical Fiber Communications (OFC), M2E.6 (2019).

16. R. Maher et al., "Constellation Shaped 66 GBd DP-1024QAM Transceiver with 400 km Transmission over Standard SMF", Proc. European Conference on Communications (ECOC), PDPB2 (2017).

17. R. Going et al., "Multi-channel InP-based Coherent PICs with Hybrid Integrated SiGe Electronics Operating up to 100 GBd, 32QAM" Proc. European Conference on Optical Communication (ECOC) Th.PDP.C.3 (2017).

18. J. Slovak et al., "Aware Optical Networks: Leaving the Lab" Journal of Optical Communications and Networking (JOCN), vol. 11, no. 2, p. A134 (2018).

19. R.M. Morais and J. Pedro "Machine Learning Models for Estimating Quality of Transmission in DWDM Networks" Journal of Optical Communications and Networking (JOCN), vol. 10, no. 10, p.D84 (2018).

20. Infinera Press Release, https://www.infinera.com/centurylink-and-infinera-on-the-path-toward-the-cognitive-network, last accessed 2019/03/24.

21. L. A. Barroso et al., "The Datacenter as a Computer - An Introduction to the Design of Warehouse-Scale Machines", Morgan & Claypool (2009).

22. V. Gill et al., "Worse is Better", NANOG 49, San Francisco June 14 (2010).

23. X. Wu et al., "Applications of Artificial Neural Networks in Optical Performance Monitoring", Journal of Lightwave Technology, (JLT) (Volume: 27, Issue: 16, Aug.15, 2009).

24. R. Singh et al, "RADWAN: Rate Adaptive Wide Area Network". Proc. SIGCOMM, (2018).

25. Telia Carrier Press Release, https://www.teliacarrier.com/Press-room/Press-releases/Mar-13-2018.html, last accessed 2019/03/24.
26. Infinera Press Release, https://www.infinera.com/telia-carrier-infinera-demonstrate-industry-first-autonomous-intelligent-transponder-live-network-field-trial, last accessed 2019/03/25.