# Programmable Flex-E and X-Ethernet Networks for Traffic Isolation in Multi-tenant Environments

Kostas Katsalis, Rixin Li

► To cite this version:

Kostas Katsalis, Rixin Li. Programmable Flex-E and X-Ethernet Networks for Traffic Isolation in Multi-tenant Environments. 23th International IFIP Conference on Optical Network Design and Modeling (ONDM), May 2019, Athens, Greece. pp.610-622, 10.1007/978-3-030-38085-4_52. hal-03200659

HAL Id: hal-03200659

https://inria.hal.science/hal-03200659

Submitted on 16 Apr 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Programmable Flex-E and X-Ethernet Networks
for Traffic Isolation in Multi-Tenant
Environments

Kostas Katsalis¹ and Rixin Li²

¹ Huawei Technologies Munich, Germany
kostas.katsalis@huawei.com

² Huawei Technologies, China
lirixin@huawei.com

**Abstract.** In this work we investigate the coupling of Flex-E and X-Ethernet technologies as a means to realize interface slicing and fast switching respectively. Flex-E is investigated as a key technology that is able to split a physical interface into isolated sub-channels, decouple MAC rate from PHY rate and achieve *hard* bandwidth isolation. X-Ethernet technology introduces Ethernet PCS switching, eliminates table lookup and buffer queuing and will be investigated as a fast switching mechanism that is exploiting the PCS layer relay. We present evaluation results from implementation activities that showcase the ability of an integrated Flex-E/X-Ethernet solution to enable network slicing features in transport networks. Furthermore we present a technical approach on the way the control plane of Flex-E/X-Ethernet network can be integrated with an orchestration and management solution.

**Keywords:** Flex-Ethernet · X-Ethernet · Network Slicing · Service differentiation · Throughput guarantees

# 1 Introduction

In order to couple with the strain raised and satisfy the strict requirements of 5G systems, significant changes have been undergoing in transport networking. Under the umbrella of 3GPP-based mobile cellular networks, transport networks support connectivity services between the various Disaggregated Radio Access Networks (DA-RAN) network components (3GPP TR38.816, TR38.801).

With respect to the terminology used by 3GPP (3GPP TR23.799, TR38.803) as transport network we consider the Fronthaul (FH), Midhaul (MH) and Backhaul (BH) type of communication networks that are used to interconnect Network Functions (NFs). Note that NFs can be either Physical (PNFs) or Virtual Network Functions (VNFs). In the case where the DA-RAN paradigm is adopted, NFs reside in the Radio Unit (RU), Centralized Unit (CU), Distributed Unit (DU) and in the Core Network (CN) [1]. Furthermore, in 3GPP TS23.501
where the 5G Architecture is described, future network designs also consider network operations in the light of Network Slicing [8].

In this paper we investigate Flex-E and X-Ethernet technologies as a means to realize transport network slicing with the support of fast switching. Flex-E will be investigated as a key technology that is able to split a physical interface into isolated sub-channels, decouple MAC rate from PHY rate and achieve "hard" bandwidth isolation. X-Ethernet technology introduces Ethernet PCS switching, eliminates table lookup and buffer queuing and will be investigated as a fast switching mechanism that is exploiting PCS layer relay.

Traditional deployments where relying on fiber technologies and PDH T1/E1 connections, ATM, SONET/SDH, like also MPLS to support transport network connectivity services. However, the last decade we witness a spiraling adoption of Ethernet technology as a means to implement transport networking for the mobile network. A flat Ethernet solution simplifies provisioning and planning, while also helps to radically reduce OPEX. Solutions like Ethernet over MPLS, Ethernet over SONET/SDH, Packet over SONET (PoS), Ethernet over DWDM Ethernet over OTN and so on, are inexorably driving the Ethernet technology as the de facto standard to support mobile network connectivity. Open Internet working Forum (OIF), a global industry forum for advanced inter-operable optical networking solutions, devised Flex-E in OIF-FLEXE-01.0 - Flex Ethernet Implementation Agreement (IA). The key features of Flex-E are the ability to create larger links out of multiple slower links, multiplex lower speed traffic from different clients in higher speed links, while also utilize a specific portion of some PHY link. As we will analyze in the following Flex-E can serve as an ideal technology to realize the concept of Network Slicing in transport networks and offer guaranteed performance and services per tenant even in scenarios of extreme flow multiplexing.

The main contributions of this paper are the following:

- We present the primitives of operation of Flex-E and X-Ethernet technologies as key enabling technologies towards slicing the transport network.

- We present evaluation results from real testbed experimentation. As we will analyze Flex-E is able to provide strict throughput guarantees, while X-Ethernet is able to bring switching time cost up to level of few nanoseconds.

- Our demonstrator considers the case of channelization where several lower-speed links are created for different ethernet clients.

- We provide insight information on the way a SDN solution could be incorporated in an integrated platform and how an overall orchestration and management solution could be designed.

This paper is organized as follows. In section 2 we present the related work. In section 3, we present the primitives of operation of Flex-E and X-Ethernet technologies. In section 4, we present evaluation results. In section 5 we elaborate on the way programmability can be enabled in Flex-E, X-Ethernet networks and the way they can be integrated with orchestration and management systems. We conclude our study and present future research directions in section 6.
## 2 Related Work

Flex-E enables Ethernet-based services to be mapped over a next-generation optical transport network with the most efficient utilization of capacity possible. It was originally conceived to meet the challenges of Internet Content Providers (ICPs) for higher capacities and dynamicity, and as a new mechanism for the data center interconnect (DCI). It was originally proposed by OIF in Interoperability Agreement (IA) OIF-FLEXE-01.0 on March 2016. A detailed description of the technology can be found by OIF in IA OIF-FLEXE-02.0 (2018), IA OIF-FLEXE-01.1 (2017), IA OIF-FLEXE-01.0 (2016) and [7]. The forthcoming Flex-E 2.1 project is about prototyping Flex-E over 50Gbps PHY applications and extending the recently released Flex-E 2.0. Flex-E 2.1 will further specify a new Flex-E frame and multiplexing format and address applications with lower bandwidth needs. Flex-E neighbor discovery is discussed in IA OIF-FLEXE-ND-01.0 (2018). Potential use cases were considered within other organizations such as the Ethernet Alliance and described by certain industrial players like Google in [3]. Proprietary deployments considering an integrated approach of the control and management plane of Flex-E over OTN were also contributed by Huawei [12]. Regarding Flex-E implementation landscape Huawei incorporates Flex-E in PTN990 router series, IXIA presented a demo in OFC 2016 with Altera/Intel, with Flex-E 2x100GbE and Ciena provides the Flex-E Liquid Spectrum solution. A 400 Gbps of Flex-E traffic was sent over four bonded 100 GbE interfaces on network demonstration in OFC 2018 by OIF and Ethernet Alliance.

In [11] a survey on backhaul network technologies is presented with an analysis on the relevant synchronization issues. A detailed analysis on the topic of transport network slicing is presented in [6]. Virtualization techniques for different types of optical networks and technologies where granularities may vary, according to the node and link characteristics. For example, in a wavelength switched network with optical cross connects (OXCs) and Reconfigurable Optical Add-Drop Multiplexing (ROADMs) virtualization approaches are different from networks with sub-wavelength granularity of switching and control [10]. In [5] hardware and spectrum resource virtualization techniques for the optical network are investigated. An adaptive transponder design is analyzed for WXC/WDM, while also multiflow transponder mechanisms for right-sized subchannels with the necessary number of subtransponders. Flex-O is described in ITU-T G.709.1/Y.1331.1 recommendation. Flex-O provides OTN interfaces with comparable functionality as to what was introduced in Flex-E for Ethernet interfaces. It provides an interoperable system interface for OTUCn transport signals; while it enables higher capacity ODUflex and OTUCn, by means of bonding *m* standard-rate interfaces (see ITU-T G.709.1/Y.1331).

# 3 Transport Network Slicing with Flex-E and X-Ethernet

## 3.1 Flex-E Basic Operation & Concepts

In principle, in the OSI stack in the Data Link Layer Logical Link Control (LLC) is performing multiplexing network protocols over the same Media Access
Control Sublayer (MAC), which is used for addressing and channel access control mechanisms. Reconciliation Sublayer (RS) processes PHY local/remote fault messages, while in the PHY Layer Physical Coding Sublayer (PCS) performs auto-negotiation and coding.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Flex-E layer between Ethernet MAC and PCS (Physical Coding Sublayer). Additional FlexE Shim distribute/aggregate sub-layer in PCS/PMD</figcaption></figure>

Flex-E technology is introduced as a thin layer, known as Flex-Shim, and being able to support data rates out of the conventional range offered by current Ethernet standards. In more detail, the Flex-E Shim layer (Fig.1) is responsible for the mapping of Flex-E clients (Ethernet flows) to groups of PHYs. From a layering perspective Flex-E Shim is introduced between the Ethernet MAC and the PCS sublayers. The main idea behind Flex-E is to decouple the actual PHY layer speed from the MAC layer speed of a client. Flex-E is based on a time-division multiplexing mechanism that is able to drive the asynchronous Ethernet flows over a synchronous schedule over multiple PHY layers. The main operational components of Flex-E are depicted in Table 3.1.

Currently there are three operational scenarios supported by Flex-E, which describe different ways the MAC layer speed is related with the corresponding PHY speed, allowing a distinct manner for multiplexing clients in time:

- *Bonding*: allows a MAC layer speed higher than a single PHY by grouping multiple PHYs to serve a flow (e.g. support a 200G MAC over two bonded 100GBASE-R PHYs).

- *Sub-rating*: MAC layer speed is less than the actual PHY. Allows the MAC layer to use a portion of a PHY to serve a flow (e.g. support 50Gbps MAC over a 100GBASE-R PHY).

- *Channelization*: enables multiple Flex-E clients over a shared single PHY or bounded PHY via the means of time division multiplexing in the Flex-Shim (e.g. support 150Gbps and a 50Gbps MAC over two bonded 100GBASE-R).
<caption>**Table I: Flex-E main operational components**</caption>

**Flex-E Client** is an Ethernet flow based on a MAC data rate that may or may not correspond to any Ethernet PHY rate. The MAC rates currently supported are 10, 40, and m x 25 Gb/s. FlexE 2.0 augments FlexE 1.0 by providing support for FlexE Groups composed of m x 200 Gb/s Ethernet PHYs and m x 400 Gb/s Ethernet PHYs, and several other features.

**Flex-E Group** is a group of Ethernet PHYs that are bonded together. OIF supports Flex-E groups composed of one or more bonded 100GBASE-R PHYs. Higher rates like 400GbE are under development in IEEE P802.3bs and will be supported in future Flex-E releases.

**Flex-E Shim** is the layer that maps or de-maps the Flex-E clients over a Flex-E group. This procedure relies on a calendar-based slot scheduling. Essentially a set of slots are assigned to each client, according to the MAC layer speed.

Hybrids of these scenarios are also possible, for instance a sub-rate of a bonded PHY supporting 250Gbps MAC over three bonded 100GBASE-R PHYs. These options allow increased resource flexibility for 5G and fine-tuning of the offered rate depending on the usage.

## 3.2 X-Ethernet technology

X-Ethernet is a Huawei proprietary technology, where X stands for extended distance, expanded granularity and extremely low latency. X-Ethernet introduces Ethernet PCS switching based on the interface offered by Flexible-Ethernet. The basic switching unit is FlexE Client. The switch device will redirect FlexE Clients (64B/66B block streams) from its inbound port to its outbound port without waiting for the arrival of the whole Ethernet frame for FCS checksum and forwarding decision with table lookup. Therefore, all the time consuming procedure, such as encapsulation/decapsulation, queuing and table lookup, can be removed. We give another name to the PCS switching that is PCS Non-Stop Switch (NSS). What's more, the remaining procedure processing time is predictable, which results in deterministic device latency. Idle insertion or deletion according to IEEE 802.3 may be performed to rate-adapt FlexE Client to the Flex Group. Details of the technology can be found in [9].

# 4 Performance Evaluation

## 4.1 Flex-E testbed, experiment description and evaluation results

We demonstrate and evaluate Flex-E technology, while verifying the theoretical framework proposed by OIF. The system demonstrator under test, is depicted in Fig. 2 where for the implementation of the testbed two Huawei Optix PTN 990 are used. A software patch on VRP V100R008C10, supports the necessary Flex-E functionality. PTN devices are primarily used on bearer networks that carry various services, such as mobile communication and enterprise users' services. It
transports packet services on the network and converges them to an IP/MPLS
backbone network.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Flex-E operational scenarios</figcaption></figure>

In principle channelisation enables multiple Flex-E clients over a shared single PHY or bounded PHY via the means of time division multiplexing in the Flex-Shim (e.g. support 150G and a 50 MAC over two bonded 100GBASE-R PHYs). In this demonstrator one 100G link between the two routers is “splitted” using the Flex-E technology. The splitting is made by creating Flex-E channels. The Flex-E clients are identified using VLAN technology and each VLAN is mapped to a specific Flex-E channel.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Flex-E allocated slots per channel (4 channels supported)</figcaption></figure>

As presented in Fig. 3 for the demonstrator we allocate different time slots for
two Flex-E channels, namely *Channel 0* and *Channel 2*. The mapping of channels
in Flex-E flows is made on a VLAN basis where for the current implementation
the lower two bits of the channel ID and VLAN ID correspond to each other
(e.g., VLAN 4 maps to channel 0 and VLAN 5 to channel 1 and VLAN 6 in
channel 2 and so on). In Fig. 3 a sample channel configuration is presented for
router 1 (R1 prompt) while a similar configuration also exists for router 2. In
the example depicted, slots 5 to 9 are allocated in channel 2, while all the other
slots (1-4 and 6-20) are allocated in channel 0. With this allocation channel 0
receives 15/20=3/4 capacity (translated to scheduling opportunities) from the
100G interface, channel 2 receives 1/4 from the link capacity, while channels 1
<figure><img src="image_5.png" /><figcaption>Fig. 4. Flex-E throughput guarantees per flow</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 5. (b) and (c) Delay and jitter results for two different Flex-E channels</figcaption></figure>

and 3 are blocked. Because of the specific flow mapping implementation, channel 0 will support not only VLAN 4 but also all VLAN-ids where the ending bits are 00 (like 0b000, which is VLAN 8) and so on. Different Flex-E client mapping mechanisms are under consideration and development.

To create multiple Ethernet flows with the necessary VLAN identification, traffic has been generated by using a Huawei 100GE traffic generator, which was controlled using Tesegine 2.0 V300R006C10B410 software. Using the Tesegine 2.0 traffic generator software, we were able to generate multiple concurrent flows with different configuration options of all the frames/packets fields (like src/dst MAC address, VLAN-id, src/dst IP address etc.). As a baseline experiment, we created two flows: Flow-1 (VLAN-id 4) that could be critical traffic and Flow-2 (VLAN-id 6) being the background traffic. The goal of the demonstrator was to showcase the ability of Flex-E channelisation to provide precise capacity shares between the two competing channels/flows. Channel statistics were obtained using interface counters that were updated to report per channel reports.

From all the experiments performed, perfect isolation was achieved; not only the capacity ratios were respected but also channels 1 and 3 that could block
the traffic are not mapped to "open" Flex-E channels and thus had no effect on performance. Even though we only present here a representative subset of results, our conclusions apply to a wide range of experiments with the adjustment of different network parameters. In Fig. 4 and Fig. 5 we present the evaluation results of the experiment described above with two Flex-e channels serving to Ethernet flows that are identified using their VLAN-id. As we can see, the 100G link is precisely sliced exploiting time scheduling on the Flex-Calendar. Flex-E channel 0 throughput is 75 Gb/s on average while channel 2 throughput is 25 Gb/s on average (5 slots out of 25) Fig. 4. An important observation however is that although Flex-e technology is able to provide precise throughput guarantees per flow using very low level slicing, is not able to differentiate delay and jitter per channel for each Flex-E client. This phenomenon was however expected and is depicted in Fig. 5(a) and Fig. 5(b) where, as we can observe, both flows experience similar delay and jitter performance.

## 4.2 X-Ethernet testbed, experiment description and evaluation results

A X-Ethernet testbed was used to carry demonstration activities, where for each X-Ethernet switch the solution is based on a FPGA board, six 100G CFP2 optical module slots, two 10G SFP+ optical module slots, one Ethernet interface slot. The FPGA board is mainly composed by one Virtex UltraScale chip, one ZYNQ chip, two DDR3 SDRAM chips, one DDR4 SODIMM, two Quad-SPI flashes and one Micro SD.

In the following, we provide experimentation results from the execution of three test scenarios that were carried out and showcase the ability of X-Ethernet technology to satisfy challenging switching requirements for a Flex-E network. Three X-Ethernet prototypes are connected to each other and formed a network. A controller (PC) configures each of the devices via the RS232 port on each device. A CPRI tester is used to generate CPRI option 7 traffic. A network performance tester is used to generate Ethernet traffic with 100 Gb/s maximum bit rate. CPRI is injected into the X-Ethernet node device 1 (XE1) and transport to XE2, then loop back to the CPRI tester. The Ethernet traffic pass through three XE devices one by one and loop back to the network performance tester. It should be noted that the connection between XE devices is through 100G links. Particularly, CPRI and Ethernet traffic share the same 100G link between XE No.1 and XE2. Moreover, the connection between Network performance tester and XE devices are also 100G link. Network performance tester generate an Ethernet traffic that has an effective data rate ranges from 0 to 100 Gb/s.

**Experiment T1:** Setup the test environment as shown in Fig. 6(a). We configured end-to-end Ethernet traffic flow between XE nodes in two scenarios: Flex-E tunnel allocated 2 and 10 slots respectively. The TestCenter generated and send packets of length 128 bytes.

**Experiment T2:** Setup the test environment as shown in Fig. 6(b). We configured end-to-end Ethernet traffic flows between three XE nodes. The TestCenter generate and send packets of length 128 bytes. In both experiments we
<figure><img src="image_7.png" /><figcaption>Fig. 6. X-Ethernet experiment topology.</figcaption></figure>

modified traffic packet length to 1518 bytes in sequence, and repeated the above operation, while the metric of interest is traffic latency. The experiment results are depicted in the following table:

<table><caption>Fig. 7. X-Ethernet experimentation results.</caption><thead><tr><td>Slot number</td><td>Packet length</td><td>T1 (µs)</td><td>T2 (µs)</td><td>Delay of P nodes (µs)</td></tr></thead><tbody><tr><td rowspan="2">2</td><td>128 byte</td><td>2.988</td><td>3.539</td><td>0.551</td></tr><tr><td>1518 byte</td><td>4.309</td><td>4.903</td><td>0.594</td></tr><tr><td rowspan="2">10</td><td>128 byte</td><td>2.116</td><td>2.683</td><td>0.567</td></tr><tr><td>1518 byte</td><td>2.59</td><td>3.156</td><td>0.566</td></tr></tbody></table>

For the packet streams of different packet length at different rates, X-Ethernet exhibits its ultra-low latency forwarding capability at around 0.5 use. Compared to the classic router/switch performance (30 us to 200 ms), X-Ethernet has a huge advantage in carrying latency sensitive services. Thanks to its L1.5 Non Stop Switching mechanism, all the time consuming procedures like table lookup, queuing, buffering, etc., are removed. What's more, due to its TDM like mechanism, it provides end-to-end hard isolation pipe, which guarantees a nanosecond level jitter in the transmission.

## 5 Enabling programmability on Flex-E, X-Ethernet networks an integration with 5G-OS

Software Defined Networking (SDN) is a technology paradigm that drives innovation on the network segment but mainly inside the data center, it has just recently applied also in the wireless access but also in the transport network.

Although there is a proposal to adopt GMPLS as the Flex-E control plane, is not yet standardized [4]. To enable programmability on the Flex-E/X-Ethernet solution a NETCONF protocol-based solution with out-of-band signaling could be also adopted. In the case of NETCONF protocol, a NETCONF server should operate on the switching device to interpret messages send from a remote SDN
controller implementing the NETCONF client. NETCONF is connected oriented using TCP while messages are encoded in XML and encrypted by SSH. An SDN controller like ODL or RYU could be used to implement this functionality and be directly connected to an orchestration and management solution like the 5G Operating System (5G-OS) proposed and designed in the context of the 5G-PICTURE project [2]. A 5G-OS instance can be used to manage services and slices on top of the infrastructure provided by multiple Infrastructure Provider(s). A 5G-OS is composed of different orchestration, management and control components that are tightly integrated to control the operational and business aspects of multiple technological and administrative domains. A simplified version of the 5G-OS integrated with a NETCONF based solution for the programmability of the X-Ethernet solution is depicted in Fig.8. In 5G-OS a controller within a single domain is referred to as a Domain Controller(DC), similarly an orchestrator within a single domain is called a Domain Orchestrator(DO). An orchestrator that spans multiple DOsand is responsible for full service instantiation is called a Multi-Domain Orchestrator(MDO). Depending on the NETCONF approach with the appropriate YANG models RESTCONF API can also be exploited. RESTCONF is a REST like protocol running over HTTP for accessing data defined in YANG using datastores defined in NETCONF.

<figure><img src="image_8.png" /><figcaption>Fig. 8. NETCONF-based programmability of Flex-E/X-Ethernet switches</figcaption></figure>

In both GMPLS and NETCONF cases, new data models need to be devised that expose the Flex-E information and functionalities to the control plane. Although the design of YANG models is possible over RSVP, new YANG models are expected to emerge for Flex-E. As in all control plane models, the design
primitives for the Flex-E control plane are security, scalability and fast conver-
gence. To on-board on 5G-OS a possible service descriptor should include:

- Flex-E Group provisioning, configuration and instantiation operations: Routers must advertise the type of Flex-E support that they offer, the current calendar allocation and information like link delay and node delay. Regarding capabilities exposure auto-negotiation procedures also need to be defined.

- Flex-E calendar scheduling: The control plane must be able to provide an efficient mechanism for the optimal assignment of PHYs to a specific group, while also consider for the optimal slot allocation in the group calendar for each Flex-E client.

- Establishment of Flex-E multi-hop paths: Existing solutions consider a pre-configured Command-Line Interface (CLI) based Flex-E group configuration and client assignment. Note that the most important functionality in order to have a functional Flex-E setup is that for each PHY the mux and demux share the same sub-calendar. Otherwise, it would be impossible to decode the slot information to a specific Flex-E client. In a multi-hop setup this information sharing can be challenging.

- Dynamic calendar switching configurations: Control plane must support dynamic switching between calendar configurations (A or B) and allow modifying the configuration of Flex-E clients into calendar slots, based on SLAs and performance criteria.

Regarding X-Ethernet, note that it is a Huawei proprietary experimental solu-
tion and the focus in the current development phase is on the dataplane opera-
tions. No YANG models are currently available or under development. However,
these are planned to be implemented the following period, exploiting NETCONF
protocol and enabling integration with the 5G-OS. For an orchestration and
management system like 5G-OS a possible X-Ethernet service descriptor should
include: VLAN ID, Flex-E client bandwidth, Flex-E group ID, PHY ID, Flex-E
calendar slot number, performance monitoring information, system-type, data
mode, client signal type.

6 Conclusions and future work

In this work we investigated the coupling of Flex-E and X-Ethernet technologies as a means to realize interface slicing and fast switching respectively. Flex-E is a key technology that is able to split a physical interface into isolated sub-channels, decouple MAC rate from PHY rate and achieve "hard" bandwidth isolation. X-Ethernet technology introduces Ethernet PCS switching, eliminates table lookup and buffer queuing and will be investigated as a fast switching mechanism that is exploiting PCS layer relay. We presented evaluation results from implementation activities that showcase the ability of an integrated Flex-E/X-Ethernet solution to enable programmability and network slicing features for future transport networks. As Flex-E is a living standard, we are following all the recent amendments that we are willing to incorporate into our testing
infrastructure. Furthermore, we are also investigating solutions where on the top of the end-to-end pipe that Flex-E is able to construct, virtual networks (like VLAN or enhanced VPN solutions based) can be constructed and operate depending on network slice requirements.

## Acknowledgements

This work has received funding from the EU H2020 5G-PICTURE project under grant agreement no 762057.

## References

1. C. Chang, N. Nikaein, O. Arouk, K. Katsalis, A. Ksentini, T. Turletti, and K. Samdanis. Slice orchestration for multi-service disaggregated ultra-dense rans. *IEEE Communications Magazine*, 56(8):70–77, August 2018.

2. S. Dräxler, H. Karl, H. R. Kouchaksaraei, A. Machwe, C. Dent-Young, K. Katsalis, and K. Samdanis. 5g os: Control and orchestration of services on multi-domain heterogeneous 5g infrastructures. In *2018 European Conference on Networks and Communications (EuCNC)*, pages 1–9, June 2018.

3. Tad Hofmeister, Vijay Vusirikala, and Bikash Koley. How can flexibility on the line side best be exploited on the client side? In *Optical Fiber Communication Conference*, pages W4G–4. Optical Society of America, 2016.

4. I Hussain, R Valiveti, Q Wang, L Andersson, M Chen, and H Zheng. Gmpls routing and signaling framework for flexible ethernet (flexe). *draft-izh-camp-fiexe-fwk-05*. *IETF-Draft*, 2018.

5. Masahiko Jinno, Hidehiko Takara, Kazushige Yonenaga, and Akira Hirano. Virtualization in optical networks from network level to hardware level. *Journal of Optical Communications and Networking*, 5(10):A46–A56, 2013.

6. Alexandros Kaloxylos, Christian Mannweiler, et al. Network slicing. *5G System Design: Architectural and Functional Considerations and Long Term Research*, pages 181–205, 2018.

7. K. Katsalis, L. Gatzikis, and K. Samdanis. Towards Slicing for Transport Networks: The Case of Flex-Ethernet in 5G. In *2018 IEEE Conference on Standards for Communications and Networking (CSCN)*, pages 1–7, Oct 2018.

8. K. Katsalis, N. Nikaein, E. Schiller, A. Ksentini, and T. Braun. Network Slices toward 5G Communications: Slicing the LTE Network. *IEEE Communications Magazine*, 55(8):146–154, Aug 2017.

9. Rixin Li, Renlei Wang, et al. X-ethemet: Enabling integrated fronthaul/backhaul architecture in 5g networks. In *2017 IEEE Conference on Standards for Communications and Networking (CSCN)*, pages 121–125. IEEE, 2017.

10. Reza Nejabati, Eduard Escalona, Shuping Peng, and Dimitra Simeonidou. Optical network virtualization. In *15th International Conference on Optical Network Design and Modeling-ONDM 2011*, pages 1–5. IEEE, 2011.

11. Orawan Tipmongkolsilp, Said Zaghloul, and Admela Jukan. The evolution of cellular backhaul technologies: Current issues and future trends. *IEEE communications surveys & tutorials*, 13(1):97–113, 2011.

12. R. Vilalta, R. Martínez, R. Casellas, R. Muñoz, Y. Lee, L. Fei, P. Tang, and V. López. Network slicing using dynamic flex ethernet over transport networks. In *ECOC*, pages 1–3, Sept 2017.