# Resource Allocation in Multicore Elastic Optical Networks: A Deep Reinforcement Learning Approach

Authors: Juan Pinto-Ríos; Felipe Calderón; Ariel Leiva; Gabriel Hermosilla; Alejandra Beghelli; Danilo Bórquez-Paredes; Astrid Lozada; Nicolás Jara; Ricardo Olivares; Gabriel Saavedra
Journal: Complexity
 (2023)

---

## Abstract

A deep reinforcement learning (DRL) approach is applied, for the first time, to solve the routing, modulation, spectrum, and core allocation (RMSCA) problem in dynamic multicore fiber elastic optical networks (MCF-EONs). To do so, a new environment was designed and implemented to emulate the operation of MCF-EONs - taking into account the modulation format-dependent reach and intercore crosstalk (XT) - and four DRL agents were trained to solve the RMSCA problem. The blocking performance of the trained agents was compared through simulation to 3 baselines RMSCA heuristics. Results obtained for the NSFNet and COST239 network topologies under different traffic loads show that the best-performing agent achieves, on average, up to a four-times decrease in blocking probability with respect to the best-performing baseline heuristic method.


---

## Body

## 1. Introduction

Due to the ever-growing number of users, devices, and networking applications, Internet traffic keeps on increasing, more than doubling every two years, to levels that will lead to an eventual capacity crunch of the current core optical networks [1, 2]. Big technological companies, such as Google, Meta, Amazon, Netflix, Apple, and Microsoft now account for more than half of Internet traffic, and the introduction of 5G is expected to accelerate the growth of emerging heavy app users consuming 1 terabyte per month [3].Various solutions to deal with this constant traffic growth have been proposed, ranging from greater efficiency in using currently deployed optical resources to expanding the capacity of the optical transport network. Examples of the former and latter are Elastic Optical Networks (EONs) [4] and multicore optical fiber (MCF) [5], respectively. EONs [6] divide the spectrum into narrow slots called frequency slot units (FSU), usually of 12.5 GHz width [7]. In EON communication, each connection uses as many adjacent slots as needed, thereby improving the spectral usage efficiency [8]. Under dynamic operation, EONs [9] can establish and release connections on-demand. MCF extends the fiber capacity by adding multiple cores within the same cladding. Thus, the capacity of a single fiber is significantly increased given that each core can be considered as an extra optical medium [10].One of the first cases for the support of elastic optical networks was given by data-intensive applications running on multidata center systems [11]. Later, the need for elastic optical networks was highlighted for applications such as cloud-based IoT services [12], cloud-fog computing [13] as well as critical support for the 5G communication infrastructure [14] and the applications associated with it, such as Ultra High Definition videos, Telemedicine, and Smart City/Industry/Factory/Home [15]. Similarly, MCF has been identified as a complement of elastic optical networks to deliver the high capacity required by current and future applications, as well as the driving force to provide cost-efficient solutions for high-capacity submarine cables [16, 17], a key infrastructure underpinning Internet. Currently, applications requiring the combination of MCF [18] with the efficient use of the spectrum offered by dynamic elastic optical networks [19] have also been identified in scenarios such as intradata center networks [20, 21]. Going beyond the current technological state, expected Multimedia 3D Services for 6G networks such as Tactile/Haptic Internet, Video Games/Streaming as a 3D Service, and Deep-Sea Sightseeing [15] will certainly require a network capacity that only will be provided by the combination of MCF and dynamic EONs, termed as dynamic MCF-EON from now on.One of the main challenges of dynamic MCF-EONs is the design of efficient routing, modulation, spectrum, and core assignment (RMSCA) strategies for establishing optical connections with as low blocking probability as possible. Most RMSCA proposals use heuristic approaches that consider the impact of intercore crosstalk (intercore XT) on optical signal quality, as described in [19, 22–25]. Although rule-based heuristics are computationally simple, their performance depends on the ability of the designer to detect the best set of rules defining the heuristic behavior [26]. In recent years, it has been shown that in most cases, deep reinforcement learning (DRL) techniques applied to solve resource allocation problems in dynamic elastic optical networks outperform rule-based systems [27, 28]. DRL has the ability to explore solutions other than those detected by the expert knowledge of the human designer. As a result, it has the potential of generating new nonobvious policies from the experience gained after training in a relevant environment [29].


### 1.1. Related Work

In dynamic scenarios, DRL was applied to solve the routing, modulation, and spectrum assignment (RMSA) problem in single-domain EONs [27, 28, 30, 31], multidomain EONs [32], multiband EONs [33, 34] ,and survivable EONs operating under shared protection [35]; the problem of energy-efficient traffic grooming in fog-cloud EONs [36], the problem of establishing and reconfiguring multicast sessions in EONs [37], the fragmentation mitigation problem [38], and the resource allocation problem with advanced reservation (AR) in EONs for cloud-edge computing [39]. Only one previous work has studied the application of DRL on MCF networks [40], but this work focused on fixed-grid networks. In this paper, we extend the work reported in [27, 30, 31] by applying DRL to dynamic MCF-EONs for the first time.In the context of dynamic MCF or MCF-EON networks, with the exception of [40], only supervised machine-learning techniques have been applied so far. These consist of techniques for making inferences based on expert-labeled data. Thus, instead of taking actions, supervised learning algorithms perform estimations or classifications [41]. For example, the authors of [42, 43] used supervised learning to predict future connection requests in dynamic MCF-EONs to perform a crosstalk-aware resource allocation in advance. Instead, the authors in [44] used machine learning to estimate the intercore XT to then execute a crosstalk-aware allocation algorithm. All these studies have used machine learning as an auxiliary process to improve the heuristic allocation, either by predicting future traffic or transmission quality. In none of them, machine learning had direct participation in the decision-making related to resource allocation.


### 1.2. Paper Contribution

To the best of our knowledge, there are no previous studies on applying DRL to solve the RMSCA problem in dynamic MCF-EONs. In this paper, we present, for the first time, the implementation and testing of a new dynamic MCF-EON environment where four different DRL agents are trained to solve the RMSCA problem. The results obtained by the best-performing agent are then compared to 3 baseline heuristics.The rest of this article is organized as follows: Section2 presents the DRL system developed, Section 3 describes the performance evaluation experiments, and Section 4 concludes the paper.


## 1.1. Related Work

In dynamic scenarios, DRL was applied to solve the routing, modulation, and spectrum assignment (RMSA) problem in single-domain EONs [27, 28, 30, 31], multidomain EONs [32], multiband EONs [33, 34] ,and survivable EONs operating under shared protection [35]; the problem of energy-efficient traffic grooming in fog-cloud EONs [36], the problem of establishing and reconfiguring multicast sessions in EONs [37], the fragmentation mitigation problem [38], and the resource allocation problem with advanced reservation (AR) in EONs for cloud-edge computing [39]. Only one previous work has studied the application of DRL on MCF networks [40], but this work focused on fixed-grid networks. In this paper, we extend the work reported in [27, 30, 31] by applying DRL to dynamic MCF-EONs for the first time.In the context of dynamic MCF or MCF-EON networks, with the exception of [40], only supervised machine-learning techniques have been applied so far. These consist of techniques for making inferences based on expert-labeled data. Thus, instead of taking actions, supervised learning algorithms perform estimations or classifications [41]. For example, the authors of [42, 43] used supervised learning to predict future connection requests in dynamic MCF-EONs to perform a crosstalk-aware resource allocation in advance. Instead, the authors in [44] used machine learning to estimate the intercore XT to then execute a crosstalk-aware allocation algorithm. All these studies have used machine learning as an auxiliary process to improve the heuristic allocation, either by predicting future traffic or transmission quality. In none of them, machine learning had direct participation in the decision-making related to resource allocation.


## 1.2. Paper Contribution

To the best of our knowledge, there are no previous studies on applying DRL to solve the RMSCA problem in dynamic MCF-EONs. In this paper, we present, for the first time, the implementation and testing of a new dynamic MCF-EON environment where four different DRL agents are trained to solve the RMSCA problem. The results obtained by the best-performing agent are then compared to 3 baseline heuristics.The rest of this article is organized as follows: Section2 presents the DRL system developed, Section 3 describes the performance evaluation experiments, and Section 4 concludes the paper.


## 2. DRL for Dynamic MCF-EONs

A DRL system can be summarized as an agent (an entity equipped with a learning algorithm) that—during its training phase—learns to make good decisions by interacting with an environment [45, 46].In the context of RMSCA, the agent must learn to allocate optical resources to connection requests such that they are not blocked. Blocking can happen due to physical impairments or lack of spectral continuity or contiguity in the chosen route. A good allocation decision makes the environment give the agent a high-value reward.Formally, a DRL system can be modeled as a Markov Decision Process (MDP) described by the 6-tuple {S, A, T, R, s0, γ} [29], where the following takes place:(i)
S (States): Set of possible states describing the status of the system. In this work, the state st is described by the link spectrum utilization, at time step t, of each candidate route per core between the source and destination node of connection request crt. The latter is defined by the tuple o,d,h,b, where o is the source node, d is the destination node, h is the holding time of the request, and b is the bitrate of the demand.(ii)
A (Actions): Set of actions the agent can take. In this work, an action at at time step t is a triplet k,c,j, where k is the selected route (out of K pre-computed routes) c the identifier of the core (out of C cores), and j the identifier of a block of contiguous slots that can accommodate the demand of crt (out of J blocks).(iii)
Tst+1st,at (Transition probability): Probability distribution that the system transits to state st+1, given the system is in state st and the agent takes action at when receiving connection request crt.(iv)
Rst,at,st+1 (Reward): The reward function that defines the immediate reward rt received when transiting to state st+1 due to action at while in state st. In this work, the reward is designed to be equal to 1 if the request is accepted and −1 if it is rejected.(v)
s0 (Initial state): The state of the network at the start of the decision process. In this work, this state corresponds to all routes having all spectrum slots available in all links and cores.(vi)
γ (Discount factor): A parameter ∈⌈0,1 that sets the importance of current and future rewards. This factor adjusts the process of exploration and exploitation of agents in the environment [29].The evolution of the DRL system defined above is as follows: During a training episode—made of a finite amount of time steps—an agent learns to make good decisions by interacting with the environment at each time stept [45, 46]. To do so, upon receiving a connection request crt with the system in state st, the agent generates an action at. Such action makes the environment transit to the state st+1 with probability Tst+1st,at and the agent receives a reward Rtst,at,st+1. The objective of the agent is maximizing the expected future discounted reward. Thus, by repeating this process during the training episode, the agent will learn a policy π∗as that leads to maximizing the return function, Γt, defined as(1)Γt=∑t∈t,∞γt′−t⋅Rt′.The details of the state modeling are as follows: We extend the state defined in [27] by considering the different cores. Thus, the state is represented as an array of 1×2V+1+2J+3⋅K⋅C elements, where V is the number of nodes of the optical network. The extended state is then given by(2)st=o,d,h,b,zk,c1,j,zk,c2,jj∈1,…,J,zk,c3,zk,c4,zk,c5k∈1,…,K|c∈1,…,C,where a one-hot encoding is used to identify the origin and destination nodes. Each subcomponent z in the vector mentioned above is as follows: For each core c∈C and route k∈K between o and d nodes, zk,c1,j is the size of j−th block that can accommodate the connection request. zk,c2,j is the index of the first slot of each block j. The third component, zk,c3, is the number of FSUs required to establish the connection (given the modulation format used). Finally, zk,c4, the average number of FSUs available in all J blocks in route k and core c is included, and zk,c5 is the total number of FSUs available in the route k in core c.In our resource allocation problem, the environment is programmed to represent the operation and constraints of a dynamic MCF-EON. When a connection request arrives during the training phase, the agent decides what resources to allocate. At the beginning of its training, the agent makes random decisions (exploration process). Then, the environment determines whether the set of resources identified by the agent is feasible and gives the agent feedback about the quality of its decision. This information, stored in the experience buffer of the agent, allows the agent to learn. As a result, it starts to select better actions (exploitation process) for future requests. Better actions result in the agent earning a high cumulative reward. After an agent has finished its training stage, it can be evaluated (testing stage) by having it to process a new set of connection requests.The implementation of any DRL system is done in two stages as follows:(i)
Stage 1: Environment Design and Implementation. The environment is a program that receives the agent’s action, processes it, and sends back feedback. The specific feedback depends on the results of the agent’s action on the environment. The environment must consider the characteristics and constraints of the existing system to process the action. In the case of an optical network, the environment must manage information about the network topology and status and model the network operation (including physical phenomena related to the signal transmission and spectrum allocation constraints).(ii)
Stage 2: Agent Training. The agent must first acquire knowledge about the environment. This training is done by exploration and exploitation. When exploring, the agent selects random actions to learn how the environment reacts and stores such knowledge. When exploiting stored knowledge, the agent makes informed decisions to select the following action: During exploration and exploitation, the agent receives feedback from the environment, which the agent uses to update its knowledge (policy). In this way, the agent’s training progresses.In the following section, these two stages are described in detail in the context of dynamic MCF-EONs.


### 2.1. Stage 1: Environment Design and Implementation

In this work, the toolkitOptical RL-Gym, developed by Natalino and Monti [31] to facilitate the implementation and replicability of deep reinforcement learning environments for optical networks was extended by creating a new environment: DeepRMSCAEnv. Such an environment encapsulates all the necessary functions to simulate an MCF-EON.The right part of Figure1 shows a schematic of the implemented environment, including its main components and interactions. Dashed and thick lines modules are modules from the Optical RL-Gym toolkit that had to be modified and developed from scratch, respectively, to model an MCF-EON environment correctly.Figure 1
Interaction between a DRL agent and the MCF-EON environment developed: DeepRMSCAEnv.The environment can be considered made of an event-driven dynamic MCF-EON simulator and a feature engineering module. The former is responsible for processing the connection requests according to the agent’s action and sending the relevant information to the feature engineering module. The latter is responsible for preparing and sending feedback to the agent (reward and observation).The dynamic MCF-EON simulator consists of five components. Two of these store data about the network as follows:(i)
Network Information. This component stores the graph representation of the network and the link capacity, considering the multicore nature of links. It also stores the K alternatives routes for each source-destination pair, the modulation format used as a function of the route length distance, and the network information, coded as in [27].(ii)
Network Utilization. This component stores the utilization of each slot (available or used) for each network link and core.The remaining 3 components perform specific tasks(i)
Traffic Generator. This component is responsible for the random generation of connection establishment and release requests. At time step t, connection request crt is sent to the agent and the request processor component. Connection release requests are sent only to the request processor.(ii)
Request Processor. This component receives several inputs. The first two are the connection establishment or release request and the action of the agent (in the case of a connection request). When connection requestscrt are received at time step t, the Request Processor module receives st from the Route Utilization module and sends it to the agent, then the Request Processor module waits for the action of the agent. Once the action, at, is received, the Request Processor first determines the number of slots the connection requires. To do so, the most efficient modulation format that ensures a QoT [47] is first selected (QoT has been transformed into a maximum reach, as shown in Table 1). The calculation of the number of slots is the same described in Section 2 of [27]. Next, it checks the network topology (input from the network information module) and the network utilization (input from the network utilization module) to evaluate the availability of the resources selected by the agent. It also obtains information from the XT calculator component regarding the feasibility of the allocation in terms of crosstalk. If resources are available and a positive answer is received from the XT calculator, then resources are allocated, and the corresponding information is updated on the network utilization module. Information about a successful establishment is also sent to the Reward Generator module. If resources cannot be allocated, information about the failed establishment is sent to the Reward Generator component only. When a connection release is received, the Request Processor component updates the network utilization module to make the released resources available.(iii)
XT Calculator. This component calculates the intercore crosstalk (XT), defined as the interference between optical connections in neighboring cores using the same frequency slots. It receives information about the resources selected by the agent’s action at (length of the links composing the route and core) and the route-level utilization information from the network utilization module and evaluates the XT. For generic MCF systems, with any number of cores in any geometric arrangement, the steps to calculate the mean XT affecting a connection established in core x are as follows:(a)
Calculate the mean XT per unit of length between corex and adjacent core y, wx,y as(3)wx,y=2g2qβΛx,y,whereg, q, β, and Λ are the coupling coefficient, radius of curvature (or bending), constant propagation, and the distance between cores x and y, respectively.(b)
Calculate the total mean XT affecting corex, XTx, by adding the crosstalk contribution of all its adjacent cores. That is,(4)XTx=∑y=1nwx,y⋅L,wheren is the number of cores adjacent to core x and L the length of the link.Table 1
Maximum reach for each modulation format [50].
Modulation formatMax. reach (km)64QAM25032QAM50016QAM10008QAM2000QPSK4000BPSK8000For the specific case where cores follow a triangular or hexagonal geometric arrangement and different pairs of cores are equidistant, equation (5) has been found to be a better approximation to calculate XTx [19], as given as follows:(5)XTx=n−n⋅exp−n+1⋅wL1+n⋅exp−n+1⋅wL,where, as in equation (4), n represents the number of cores neighbouring x, and L is the length of the link. The term w is given by equation (3) (subindices have been dropped since the distance between all core pairs is assumed to be the same).An XT threshold value for different modulation formats is defined in [48, 49] such that the signal quality is acceptable. If XT exceeds this predefined threshold (summarized in Tables 1 and 2), a negative answer is sent to the request processor (−1). Otherwise, a positive answer is sent (1).Table 2
XT threshold for each modulation format [49].
Modulation formatXT threshold (dB)64QAM−3432QAM−2716QAM−258QAM−21QPSK−18BPSK−14The feature engineering module in Figure1 is responsible for preparing the information to be sent back to the agent. It is made of three components as follows:(i)
Reward Generator. This component calculates the numerical reward to be sent to the agent depending on the information received from the Request Processor component. In this work, a successful resource allocation returns a reward equal to 1 and a failed allocation equal to −1. Connections can be rejected due to a lack of spectrum resources along the route selected by the agent, because of crosstalk among cores exceeding the predefined threshold, or because the length of the route selected by the agent is longer than the maximum optical reach of any modulation format (such limit depends on the modulation format and a bit-error-rate threshold, as in Table 1 of [50].(ii)
Routes Utilization. This component receives the routing information from the network information component and the utilization state of the slots in the K shortest routes between the origin and destination nodes of connection request crt from the network utilization module. This information is then consolidated in a 1D vector made of K⋅C⋅J elements, where K is the number of alternative routes, C is the number of cores, and J is the number of blocks with enough available slots to establish the connection request being processed.(iii)
Experience Data Generator. This component builds the information to be stored in the Experience Buffer which is a collection of tuples at,st,rt,st+1 generated during the training process.


### 2.2. Stage 2: Agent Training

The left side of Figure1 shows the interaction between the agent and the DeepRMSCAEnv environment during the training stage.The agent aims to maximize its long-term reward. That is, selecting actions leads to the highest number of connection requests established. To achieve this goal, the agent is built considering two main components.


#### 2.2.1. Policy

This component is where the knowledge of the behavior of the agent is embedded. At a given time,t receives a connection establishment request, crt as input along with state st, and action at is outputted. The action is defined by 3 integer numbers, namely, a route identifier k (selected out of K possible precomputed routes), a core identifier c (selected out of C possible cores), and the identifier j of the block selected. These values define which route, core, and spectrum resources should be assigned to each request. As the agent successfully allocates more connection requests, the policy becomes better. At the end of the training, the policy is expected to allow the agent to define which action has the highest probability of not being blocked. Figure 2 shows a simplified example of two possible actions that might be taken by the agent, given a specific st.Figure 2
Example of a connection request from node 5 to 3, requesting 3 slots (2 for data, 1 as guard band).K = 2, meaning 2 routes. The routes spectral use is represented by white blocks (available FSUs) and red and grey ones (occupied for data and as guard bands, respectively).On the left part of the figure, a 5-node network topology and a connection establishment request of 2 slots between nodes 5 and 3 are shown. The demand is represented by the red boxes (2 slots in this case) plus the grey box (1 slot used as a guard band). The number of slots required to serve the connection (red squares) is determined by the modulation format, using the same method presented in [27]. One guard band of 1 slot is considered for each connection request to achieve a good trade-off between the quality of transmission and the blocking probability [51].Let us assume that the network is equipped with three cores per link, and the agent can select either route 1k=1, represented by the red link in the topology, or route 2 k=2 by the green links. In addition to a route, the agent must also select a core and a slot. On the right side of the figure, the spectrum utilization of both routes is shown. Red and grey squares represent used FSUs. A row of squares represents the slot utilization in a specific core for a specific route. Thus, the three rows on the upper and lower part of the figure represent the slot utilization on the three cores of the first and second routes, respectively.If the agent selects Action 1, depicted in the upper part of the figure, then actionat=1,2,1 is sent back to the environment, signaling that the agent selects slot 11 as the initial slot on route 1 in core 2 to establish the connection. The thunderbolt symbol in route 1 represents the presence of crosstalk exceeding the acceptable threshold. In this case, the request will be rejected, and a reward of −1 will be sent to the agent. Instead, if the agent selects Action 2, depicted on the lower part of the figure, then action at=2,2,1 is sent back to the environment. This action leads to a successful connection establishment, and the agent receives a reward equal to 1. During the training stage, the policy component should be updated to select Action 2 over Action 1 (for this state st), leading to a higher reward.


#### 2.2.2. Learning Algorithm

This component receives the Experience Data from the environment and, based on that information, updates the policy to produce actions that maximize the expected cumulative long-term reward. In this study, we consider learning algorithms compatible with the action space. The action space used has a multidiscrete nature because the action is defined by multiple discrete values (route, core, and slot identifier). Thus, the learning algorithms available in the Stable-Baselines [52] library that was compatible with a multidiscrete space state were selected (as also done in [27, 31]). These are as follows:(i)
Advantage Actor-Critic (A2C) [53] and Actor-Critic using Kronecker-Factored Trust Region (ACKTR) [54]. These are approaches based on the actor-critic algorithm [53], which has two interacting neural networks. The actor uses a dense neural network to process and update the policy obtained. The critic uses a separated neural network to evaluate the quality of the policy by calculating the “value function” [45]. Both algorithms differ in how they update their neural networks’ weights. A2C does that by using the feedback the critic’s network gives to the actor’s network, whilst ACKTR uses a Kronecker-factored approximation [56], which is a method that optimizes the stochastic gradient descent.(ii)
Proximal Policy Optimization (PPO2) [55] and Trust Region Policy Optimization (TRPO) [56]. These learning algorithms use only one neural network, whose weights are updated based on the policy gradient descent. They differ in the way the policy gradient descent is approached. TRPO avoids sudden changes in the neural network weights, updating only those that do not differ by a greater distance than what the Kullback–Leibler restriction (relative entropy) [56] allows. Instead, PPO2 does not impose limits on the neural network weights’ changes to optimize the policy’s descent curve.


## 2.1. Stage 1: Environment Design and Implementation

In this work, the toolkitOptical RL-Gym, developed by Natalino and Monti [31] to facilitate the implementation and replicability of deep reinforcement learning environments for optical networks was extended by creating a new environment: DeepRMSCAEnv. Such an environment encapsulates all the necessary functions to simulate an MCF-EON.The right part of Figure1 shows a schematic of the implemented environment, including its main components and interactions. Dashed and thick lines modules are modules from the Optical RL-Gym toolkit that had to be modified and developed from scratch, respectively, to model an MCF-EON environment correctly.Figure 1
Interaction between a DRL agent and the MCF-EON environment developed: DeepRMSCAEnv.The environment can be considered made of an event-driven dynamic MCF-EON simulator and a feature engineering module. The former is responsible for processing the connection requests according to the agent’s action and sending the relevant information to the feature engineering module. The latter is responsible for preparing and sending feedback to the agent (reward and observation).The dynamic MCF-EON simulator consists of five components. Two of these store data about the network as follows:(i)
Network Information. This component stores the graph representation of the network and the link capacity, considering the multicore nature of links. It also stores the K alternatives routes for each source-destination pair, the modulation format used as a function of the route length distance, and the network information, coded as in [27].(ii)
Network Utilization. This component stores the utilization of each slot (available or used) for each network link and core.The remaining 3 components perform specific tasks(i)
Traffic Generator. This component is responsible for the random generation of connection establishment and release requests. At time step t, connection request crt is sent to the agent and the request processor component. Connection release requests are sent only to the request processor.(ii)
Request Processor. This component receives several inputs. The first two are the connection establishment or release request and the action of the agent (in the case of a connection request). When connection requestscrt are received at time step t, the Request Processor module receives st from the Route Utilization module and sends it to the agent, then the Request Processor module waits for the action of the agent. Once the action, at, is received, the Request Processor first determines the number of slots the connection requires. To do so, the most efficient modulation format that ensures a QoT [47] is first selected (QoT has been transformed into a maximum reach, as shown in Table 1). The calculation of the number of slots is the same described in Section 2 of [27]. Next, it checks the network topology (input from the network information module) and the network utilization (input from the network utilization module) to evaluate the availability of the resources selected by the agent. It also obtains information from the XT calculator component regarding the feasibility of the allocation in terms of crosstalk. If resources are available and a positive answer is received from the XT calculator, then resources are allocated, and the corresponding information is updated on the network utilization module. Information about a successful establishment is also sent to the Reward Generator module. If resources cannot be allocated, information about the failed establishment is sent to the Reward Generator component only. When a connection release is received, the Request Processor component updates the network utilization module to make the released resources available.(iii)
XT Calculator. This component calculates the intercore crosstalk (XT), defined as the interference between optical connections in neighboring cores using the same frequency slots. It receives information about the resources selected by the agent’s action at (length of the links composing the route and core) and the route-level utilization information from the network utilization module and evaluates the XT. For generic MCF systems, with any number of cores in any geometric arrangement, the steps to calculate the mean XT affecting a connection established in core x are as follows:(a)
Calculate the mean XT per unit of length between corex and adjacent core y, wx,y as(3)wx,y=2g2qβΛx,y,whereg, q, β, and Λ are the coupling coefficient, radius of curvature (or bending), constant propagation, and the distance between cores x and y, respectively.(b)
Calculate the total mean XT affecting corex, XTx, by adding the crosstalk contribution of all its adjacent cores. That is,(4)XTx=∑y=1nwx,y⋅L,wheren is the number of cores adjacent to core x and L the length of the link.Table 1
Maximum reach for each modulation format [50].
Modulation formatMax. reach (km)64QAM25032QAM50016QAM10008QAM2000QPSK4000BPSK8000For the specific case where cores follow a triangular or hexagonal geometric arrangement and different pairs of cores are equidistant, equation (5) has been found to be a better approximation to calculate XTx [19], as given as follows:(5)XTx=n−n⋅exp−n+1⋅wL1+n⋅exp−n+1⋅wL,where, as in equation (4), n represents the number of cores neighbouring x, and L is the length of the link. The term w is given by equation (3) (subindices have been dropped since the distance between all core pairs is assumed to be the same).An XT threshold value for different modulation formats is defined in [48, 49] such that the signal quality is acceptable. If XT exceeds this predefined threshold (summarized in Tables 1 and 2), a negative answer is sent to the request processor (−1). Otherwise, a positive answer is sent (1).Table 2
XT threshold for each modulation format [49].
Modulation formatXT threshold (dB)64QAM−3432QAM−2716QAM−258QAM−21QPSK−18BPSK−14The feature engineering module in Figure1 is responsible for preparing the information to be sent back to the agent. It is made of three components as follows:(i)
Reward Generator. This component calculates the numerical reward to be sent to the agent depending on the information received from the Request Processor component. In this work, a successful resource allocation returns a reward equal to 1 and a failed allocation equal to −1. Connections can be rejected due to a lack of spectrum resources along the route selected by the agent, because of crosstalk among cores exceeding the predefined threshold, or because the length of the route selected by the agent is longer than the maximum optical reach of any modulation format (such limit depends on the modulation format and a bit-error-rate threshold, as in Table 1 of [50].(ii)
Routes Utilization. This component receives the routing information from the network information component and the utilization state of the slots in the K shortest routes between the origin and destination nodes of connection request crt from the network utilization module. This information is then consolidated in a 1D vector made of K⋅C⋅J elements, where K is the number of alternative routes, C is the number of cores, and J is the number of blocks with enough available slots to establish the connection request being processed.(iii)
Experience Data Generator. This component builds the information to be stored in the Experience Buffer which is a collection of tuples at,st,rt,st+1 generated during the training process.


## 2.2. Stage 2: Agent Training

The left side of Figure1 shows the interaction between the agent and the DeepRMSCAEnv environment during the training stage.The agent aims to maximize its long-term reward. That is, selecting actions leads to the highest number of connection requests established. To achieve this goal, the agent is built considering two main components.


### 2.2.1. Policy

This component is where the knowledge of the behavior of the agent is embedded. At a given time,t receives a connection establishment request, crt as input along with state st, and action at is outputted. The action is defined by 3 integer numbers, namely, a route identifier k (selected out of K possible precomputed routes), a core identifier c (selected out of C possible cores), and the identifier j of the block selected. These values define which route, core, and spectrum resources should be assigned to each request. As the agent successfully allocates more connection requests, the policy becomes better. At the end of the training, the policy is expected to allow the agent to define which action has the highest probability of not being blocked. Figure 2 shows a simplified example of two possible actions that might be taken by the agent, given a specific st.Figure 2
Example of a connection request from node 5 to 3, requesting 3 slots (2 for data, 1 as guard band).K = 2, meaning 2 routes. The routes spectral use is represented by white blocks (available FSUs) and red and grey ones (occupied for data and as guard bands, respectively).On the left part of the figure, a 5-node network topology and a connection establishment request of 2 slots between nodes 5 and 3 are shown. The demand is represented by the red boxes (2 slots in this case) plus the grey box (1 slot used as a guard band). The number of slots required to serve the connection (red squares) is determined by the modulation format, using the same method presented in [27]. One guard band of 1 slot is considered for each connection request to achieve a good trade-off between the quality of transmission and the blocking probability [51].Let us assume that the network is equipped with three cores per link, and the agent can select either route 1k=1, represented by the red link in the topology, or route 2 k=2 by the green links. In addition to a route, the agent must also select a core and a slot. On the right side of the figure, the spectrum utilization of both routes is shown. Red and grey squares represent used FSUs. A row of squares represents the slot utilization in a specific core for a specific route. Thus, the three rows on the upper and lower part of the figure represent the slot utilization on the three cores of the first and second routes, respectively.If the agent selects Action 1, depicted in the upper part of the figure, then actionat=1,2,1 is sent back to the environment, signaling that the agent selects slot 11 as the initial slot on route 1 in core 2 to establish the connection. The thunderbolt symbol in route 1 represents the presence of crosstalk exceeding the acceptable threshold. In this case, the request will be rejected, and a reward of −1 will be sent to the agent. Instead, if the agent selects Action 2, depicted on the lower part of the figure, then action at=2,2,1 is sent back to the environment. This action leads to a successful connection establishment, and the agent receives a reward equal to 1. During the training stage, the policy component should be updated to select Action 2 over Action 1 (for this state st), leading to a higher reward.


### 2.2.2. Learning Algorithm

This component receives the Experience Data from the environment and, based on that information, updates the policy to produce actions that maximize the expected cumulative long-term reward. In this study, we consider learning algorithms compatible with the action space. The action space used has a multidiscrete nature because the action is defined by multiple discrete values (route, core, and slot identifier). Thus, the learning algorithms available in the Stable-Baselines [52] library that was compatible with a multidiscrete space state were selected (as also done in [27, 31]). These are as follows:(i)
Advantage Actor-Critic (A2C) [53] and Actor-Critic using Kronecker-Factored Trust Region (ACKTR) [54]. These are approaches based on the actor-critic algorithm [53], which has two interacting neural networks. The actor uses a dense neural network to process and update the policy obtained. The critic uses a separated neural network to evaluate the quality of the policy by calculating the “value function” [45]. Both algorithms differ in how they update their neural networks’ weights. A2C does that by using the feedback the critic’s network gives to the actor’s network, whilst ACKTR uses a Kronecker-factored approximation [56], which is a method that optimizes the stochastic gradient descent.(ii)
Proximal Policy Optimization (PPO2) [55] and Trust Region Policy Optimization (TRPO) [56]. These learning algorithms use only one neural network, whose weights are updated based on the policy gradient descent. They differ in the way the policy gradient descent is approached. TRPO avoids sudden changes in the neural network weights, updating only those that do not differ by a greater distance than what the Kullback–Leibler restriction (relative entropy) [56] allows. Instead, PPO2 does not impose limits on the neural network weights’ changes to optimize the policy’s descent curve.


## 2.2.1. Policy

This component is where the knowledge of the behavior of the agent is embedded. At a given time,t receives a connection establishment request, crt as input along with state st, and action at is outputted. The action is defined by 3 integer numbers, namely, a route identifier k (selected out of K possible precomputed routes), a core identifier c (selected out of C possible cores), and the identifier j of the block selected. These values define which route, core, and spectrum resources should be assigned to each request. As the agent successfully allocates more connection requests, the policy becomes better. At the end of the training, the policy is expected to allow the agent to define which action has the highest probability of not being blocked. Figure 2 shows a simplified example of two possible actions that might be taken by the agent, given a specific st.Figure 2
Example of a connection request from node 5 to 3, requesting 3 slots (2 for data, 1 as guard band).K = 2, meaning 2 routes. The routes spectral use is represented by white blocks (available FSUs) and red and grey ones (occupied for data and as guard bands, respectively).On the left part of the figure, a 5-node network topology and a connection establishment request of 2 slots between nodes 5 and 3 are shown. The demand is represented by the red boxes (2 slots in this case) plus the grey box (1 slot used as a guard band). The number of slots required to serve the connection (red squares) is determined by the modulation format, using the same method presented in [27]. One guard band of 1 slot is considered for each connection request to achieve a good trade-off between the quality of transmission and the blocking probability [51].Let us assume that the network is equipped with three cores per link, and the agent can select either route 1k=1, represented by the red link in the topology, or route 2 k=2 by the green links. In addition to a route, the agent must also select a core and a slot. On the right side of the figure, the spectrum utilization of both routes is shown. Red and grey squares represent used FSUs. A row of squares represents the slot utilization in a specific core for a specific route. Thus, the three rows on the upper and lower part of the figure represent the slot utilization on the three cores of the first and second routes, respectively.If the agent selects Action 1, depicted in the upper part of the figure, then actionat=1,2,1 is sent back to the environment, signaling that the agent selects slot 11 as the initial slot on route 1 in core 2 to establish the connection. The thunderbolt symbol in route 1 represents the presence of crosstalk exceeding the acceptable threshold. In this case, the request will be rejected, and a reward of −1 will be sent to the agent. Instead, if the agent selects Action 2, depicted on the lower part of the figure, then action at=2,2,1 is sent back to the environment. This action leads to a successful connection establishment, and the agent receives a reward equal to 1. During the training stage, the policy component should be updated to select Action 2 over Action 1 (for this state st), leading to a higher reward.


## 2.2.2. Learning Algorithm

This component receives the Experience Data from the environment and, based on that information, updates the policy to produce actions that maximize the expected cumulative long-term reward. In this study, we consider learning algorithms compatible with the action space. The action space used has a multidiscrete nature because the action is defined by multiple discrete values (route, core, and slot identifier). Thus, the learning algorithms available in the Stable-Baselines [52] library that was compatible with a multidiscrete space state were selected (as also done in [27, 31]). These are as follows:(i)
Advantage Actor-Critic (A2C) [53] and Actor-Critic using Kronecker-Factored Trust Region (ACKTR) [54]. These are approaches based on the actor-critic algorithm [53], which has two interacting neural networks. The actor uses a dense neural network to process and update the policy obtained. The critic uses a separated neural network to evaluate the quality of the policy by calculating the “value function” [45]. Both algorithms differ in how they update their neural networks’ weights. A2C does that by using the feedback the critic’s network gives to the actor’s network, whilst ACKTR uses a Kronecker-factored approximation [56], which is a method that optimizes the stochastic gradient descent.(ii)
Proximal Policy Optimization (PPO2) [55] and Trust Region Policy Optimization (TRPO) [56]. These learning algorithms use only one neural network, whose weights are updated based on the policy gradient descent. They differ in the way the policy gradient descent is approached. TRPO avoids sudden changes in the neural network weights, updating only those that do not differ by a greater distance than what the Kullback–Leibler restriction (relative entropy) [56] allows. Instead, PPO2 does not impose limits on the neural network weights’ changes to optimize the policy’s descent curve.


## 3. Performance Evaluation

Table3 lists the values of the main parameters used to train the agents. In terms of network parameters, we consider two topologies, namely, the NSFNet (mills [57]) and the COST239 (Batchelor [58]). For each one, we assume 100 FSUs and 3 cores arranged in a triangular geometry per link, and the available modulation formats are BPSK, QPSK, 8-QAM, and 16-QAM. These simplifications have been considered due to memory constraints. The same number of slots was considered in [33]. As in [59], we use (5) to calculate the XT.Table 3
Network, traffic and training parameters.
ParametersValueNetwork parametersTopologiesNSFNet [50] and COST239 [51]Number of cores3Number of FSU by link100Modulation formatsBPSK, QPSK, 8-QAM, 16-QAMTraffic parametersBit rates (Gb/s)Uniformly distributed in [25–100] GbpsAgent training parametersPrecomputed candidate routes5Number of connection requests per episode50 [27]Simulated requests per training160,000Agent’s learning algorithm parametersBy-default [52]Agent’s hyperparametersBy-default [31]Regarding the traffic characteristics, we assume a fully dynamic behavior, where connection establishment requests arrive as a Poisson process and connection holding times follow a negative exponential distribution. The bitrate associated to each connection is uniformly selected from the range [25–100] Gbps, as in [27]. Finally, regarding the agents (one per learning algorithm), they will select one out of 5 precomputed routes, one out of 3 cores, and the identifier j of the FSU block for the connection considering a total of 100 FSUs. Agents will be trained in episodes made of 50 connection requests each (to simplify backpropagation in the dense neural network used by the agent by delivering small batches of data continuously), and the whole training session will consider a total of 160,000 connection requests. The parameters of the four agents will be the ones set by default in the agent’s library Stable Baselines [52]. The DRL system developed is available in a Git repository (The new environment, under the name DeepRMSCAEnv, is available at https://gitlab.com/IRO-Team/deeprmsca-a-mcf-eon-enviroment-for-optical-rl-gym/).


### 3.1. Preliminary Training Results

Results for the training process of 4 agents are presented. The following discussion about the results obtained is valid only for the hyperparameters used for each agent defined in Table3.The agents TRPO, PPO2, A2C, and ACKTR were trained with a traffic load of 250 Erlang, as in [27].Figures3 and 4 show the reward accumulated by the different agents during their training in the NSFNet and COST239 topologies, respectively. Given that each episode is made of 50 connection requests, the maximum reward achievable by an agent is 50. It can be seen that the A2C and TRPO agents are the only ones reaching values close to the maximum expected reward in both topologies with an average reward of 49 and 47, respectively, with TRPO exhibiting slightly better performance. On the other hand, the PPO2 and ACKTR agents did not perform well using the default parameters. PPO2 performed well on the COST239 topology (reward oscillated around 42) but not on NSFNet (reward oscillated around 30). In both topologies, the agent got stuck to the same value of reward from the very beginning, showing no signs of learning. In the case of ACKTR, the default parameters were not suitable for this task either. Not only the agent got low values of reward in both topologies, but in the COST239 topology, the reward obtained decreased during several periods of the training process, never again exceeding the value obtained in the first 10,000 timesteps.Figure 3
Accumulated reward for the A2C, PPO2, TRPO, and ACKTR agents in the NSFNet topology.Figure 4
Accumulated reward for the A2C, PPO2, TRPO, and ACKTR agents in the COST239 topology.The A2C (Actor-Critic) learning algorithm filters those agents’ actions leading to a low reward. Such filtering is possible thanks not only to the feedback received from the environment but also to the feedback given to the Actor (neural network in charge of applying the policy) by the Critic (neural network in charge of evaluating the quality of the policy used through the Value function). As a result, the agent starts with low values of reward (exploration phase) to then quickly increasing its reward per episode (exploitation phase) as the training progresses. Such behavior can be observed in Figures3 and 4, where the A2C agent requires a few episodes to achieve a reward close to 50 and exhibits one of the best results in both topologies.ACKTR (Actor-Critic using Kronecker-Factored Trust Region) is a trust-region optimization algorithm for actor-critic methods with gradient update sped up by means of the Kronecker-factored approximation. The effectiveness of the trust-region method is highly dependent on the learning algorithm’s parameters. In practice, using the by-default parameters of the Stable Baselines led to the following: (a) the weights of the actor’s neural network not being updated, trapping the agent in a local optimum, as seen in Figure3 (NFSNet topology) and (b) not finding the trust region, resulting in random actions, as seen in Figure 4 (COST 239 topology).PPO2 uses a different approach by updating the gradient more frequently than other methods. As a result, it can find a good policy more quickly than other methods, as shown in Figures3 and 4. However, Figure 3 shows that it also gets stuck in a local optimum. Most probably this is due to the use of the by-default learning algorithm’s parameters of Stable Baselines.Finally, TRPO combines the policy gradient method of PPO2, but it also uses a trust region to avoid radical changes in the update of the neural network weights. The size of the trust region is aimed to avoid increasing the relative entropy of information based on the factor Kullback−Lieber As a result, it improves slowly and monotonically, as seen in Figures3 and 4. For the problem studied here, this agent achieved the highest cumulative reward.Please notice that parameter tuning is out of the scope of this work, as our aim was to show the potential of DRL as a solution for the dynamic resource allocation in MCF-EONs.Figures5 and 6 show the evolution of the blocking probability during the training process of the same agents for the NFSNet and COST239 topologies, respectively. For comparison, the dashed red line shows the blocking probability obtained by one of the baseline heuristics, kSP-FF-FCA. This heuristic has a list of 5 precomputed routes, sorted from shortest k=1 to longest k=5. When a connection request arrives, the heuristic attempts to establish the connection in the shortest path of the list k=1, applying the first-fit policy for spectrum allocation and first-fit crosstalk-aware for core allocation, as described in [60]. The same procedure is repeated for the following route in the list if unsuccessful: After attempting all paths, the connection is rejected if there are no available resources.Figure 5
Blocking probability for A2C, PPO2, TRPO, and ACKTR in NSFNet Topology.Figure 6
Blocking probability for A2C, PPO2, TRPO, and ACKTR in COST239 Topology.From the figure, we can see that once the agents are in steady-state, TRPO and A2C agents outperform the heuristic, improving blocking of 24.3% and 73.9% for the NSFNet topology and 14.51% and 38.71% for the COST239 topology, respectively.Given the excellent performance of the TRPO agent in both topologies, in the following section, this agent will be trained for different traffic loads, and then its performance will be contrasted with that of the heuristics selected in [61].


### 3.2. TRPO Training Results

The TRPO agent was trained for traffic loads between 500 and 3000 Erlang, in steps of 500. Figures7 and 8 show the evolution of the blocking probability achieved by the TRPO agent as the training process progresses for different traffic loads for the NSFNet and COST239 topologies, respectively. It can be seen that the agent exhibits consistent behavior, with the blocking probability increasing with the traffic load, as expected. It can also be seen that at the beginning of the training process, the agent obtains a high blocking probability due to the exploration process. When the exploitation process starts, the blocking probability is reduced until it converges to a steady value. This happens after 150 thousand timesteps for the NFSNet and 130 thousand timesteps for COST239, irrespective of the traffic load. Given this steady value, we assume training has finished and the trained agent can now be evaluated in a testing setting.Figure 7
Blocking probability progress for TRPO agent training in NSFNet Topology.Figure 8
Blocking probability progress for TRPO agent training in COST239 Topology.


### 3.3. TRPO Agent VS. Heuristic: Blocking Performance

Figures9 and 10 show the blocking probability achieved by the trained TRPO agent and the same heuristics, selected for blocking evaluation in the survey [50], namely, KSP-FF-FCA [61], KSP-RF-RCA [61], and KSP-SCMA XT/demand-aware [22]. Results assume operation in the C-band (320 FSU) for the NSFNet and COST239 topologies, respectively. The three heuristics apply alternated routing. KSP-FF-FCA uses the First Fit policy to select core and spectrum, KSP-RF-RCA applies a random policy to select core and spectrum, and KSP-SCMA XT/demand aware allocates different parts of the spectrum and core depending on the bitrate of the connection request. If the connection request’s demand is below a bitrate’s threshold, a First-Fit allocation policy is applied for spectrum and core assignment as long as the cross-talk levels are not exceeded; otherwise, a Last-Fit policy is applied if the connection request’s demand is above the threshold.Figure 9
Blocking probability steady average of TRPO agent trained in NSFNet topology.Figure 10
Blocking probability steady average of TRPO agent trained in COST239 topology.Compared to the best-performing heuristic, KSP-SCMA XT/demand-aware, a significant improvement in the blocking performance of the DRL approaches is observed. For example, in the NFSNet topology, at the highest load studied, the TRPO agent exhibits a blocking probability of about1.9⋅10−2, about four times slower than the blocking of 8.5⋅10−2 achieved by the heuristic. On average, considering both topologies and loads over 2000 Erlang, TRPO achieves a 4-times decrease in blocking concerning the best heuristic, being ideal for the future scenario of demand for connection requests [64] and highlighting the benefits of applying DRL techniques to the RMSCA problem.Finally, our results show that the trained agent can generalise policies for different traffic loads and spectrum resources and outperform the rule-based heuristics. The improved performance comes from the ability of the DRL to explore solutions other than those detected by the expert knowledge of the human designer of the heuristics. We have also observed that training in adverse conditions achieves good results. That is, training the agent at high traffic loads makes the agent to perform well at lower traffic loads whereas training the agent using links with reduced capacity leads to the agent to perform better in links with increased capacity. In line with previous research [33], such generalisation was not observed in terms of topology: The agent trained in the NFSNet topology did not perform well in the COST 239 topology and vice versa. Studying the benefit of using Graph Neural Networks to overcome the lack of topology generalisation is part of current research [63].


## 3.1. Preliminary Training Results

Results for the training process of 4 agents are presented. The following discussion about the results obtained is valid only for the hyperparameters used for each agent defined in Table3.The agents TRPO, PPO2, A2C, and ACKTR were trained with a traffic load of 250 Erlang, as in [27].Figures3 and 4 show the reward accumulated by the different agents during their training in the NSFNet and COST239 topologies, respectively. Given that each episode is made of 50 connection requests, the maximum reward achievable by an agent is 50. It can be seen that the A2C and TRPO agents are the only ones reaching values close to the maximum expected reward in both topologies with an average reward of 49 and 47, respectively, with TRPO exhibiting slightly better performance. On the other hand, the PPO2 and ACKTR agents did not perform well using the default parameters. PPO2 performed well on the COST239 topology (reward oscillated around 42) but not on NSFNet (reward oscillated around 30). In both topologies, the agent got stuck to the same value of reward from the very beginning, showing no signs of learning. In the case of ACKTR, the default parameters were not suitable for this task either. Not only the agent got low values of reward in both topologies, but in the COST239 topology, the reward obtained decreased during several periods of the training process, never again exceeding the value obtained in the first 10,000 timesteps.Figure 3
Accumulated reward for the A2C, PPO2, TRPO, and ACKTR agents in the NSFNet topology.Figure 4
Accumulated reward for the A2C, PPO2, TRPO, and ACKTR agents in the COST239 topology.The A2C (Actor-Critic) learning algorithm filters those agents’ actions leading to a low reward. Such filtering is possible thanks not only to the feedback received from the environment but also to the feedback given to the Actor (neural network in charge of applying the policy) by the Critic (neural network in charge of evaluating the quality of the policy used through the Value function). As a result, the agent starts with low values of reward (exploration phase) to then quickly increasing its reward per episode (exploitation phase) as the training progresses. Such behavior can be observed in Figures3 and 4, where the A2C agent requires a few episodes to achieve a reward close to 50 and exhibits one of the best results in both topologies.ACKTR (Actor-Critic using Kronecker-Factored Trust Region) is a trust-region optimization algorithm for actor-critic methods with gradient update sped up by means of the Kronecker-factored approximation. The effectiveness of the trust-region method is highly dependent on the learning algorithm’s parameters. In practice, using the by-default parameters of the Stable Baselines led to the following: (a) the weights of the actor’s neural network not being updated, trapping the agent in a local optimum, as seen in Figure3 (NFSNet topology) and (b) not finding the trust region, resulting in random actions, as seen in Figure 4 (COST 239 topology).PPO2 uses a different approach by updating the gradient more frequently than other methods. As a result, it can find a good policy more quickly than other methods, as shown in Figures3 and 4. However, Figure 3 shows that it also gets stuck in a local optimum. Most probably this is due to the use of the by-default learning algorithm’s parameters of Stable Baselines.Finally, TRPO combines the policy gradient method of PPO2, but it also uses a trust region to avoid radical changes in the update of the neural network weights. The size of the trust region is aimed to avoid increasing the relative entropy of information based on the factor Kullback−Lieber As a result, it improves slowly and monotonically, as seen in Figures3 and 4. For the problem studied here, this agent achieved the highest cumulative reward.Please notice that parameter tuning is out of the scope of this work, as our aim was to show the potential of DRL as a solution for the dynamic resource allocation in MCF-EONs.Figures5 and 6 show the evolution of the blocking probability during the training process of the same agents for the NFSNet and COST239 topologies, respectively. For comparison, the dashed red line shows the blocking probability obtained by one of the baseline heuristics, kSP-FF-FCA. This heuristic has a list of 5 precomputed routes, sorted from shortest k=1 to longest k=5. When a connection request arrives, the heuristic attempts to establish the connection in the shortest path of the list k=1, applying the first-fit policy for spectrum allocation and first-fit crosstalk-aware for core allocation, as described in [60]. The same procedure is repeated for the following route in the list if unsuccessful: After attempting all paths, the connection is rejected if there are no available resources.Figure 5
Blocking probability for A2C, PPO2, TRPO, and ACKTR in NSFNet Topology.Figure 6
Blocking probability for A2C, PPO2, TRPO, and ACKTR in COST239 Topology.From the figure, we can see that once the agents are in steady-state, TRPO and A2C agents outperform the heuristic, improving blocking of 24.3% and 73.9% for the NSFNet topology and 14.51% and 38.71% for the COST239 topology, respectively.Given the excellent performance of the TRPO agent in both topologies, in the following section, this agent will be trained for different traffic loads, and then its performance will be contrasted with that of the heuristics selected in [61].


## 3.2. TRPO Training Results

The TRPO agent was trained for traffic loads between 500 and 3000 Erlang, in steps of 500. Figures7 and 8 show the evolution of the blocking probability achieved by the TRPO agent as the training process progresses for different traffic loads for the NSFNet and COST239 topologies, respectively. It can be seen that the agent exhibits consistent behavior, with the blocking probability increasing with the traffic load, as expected. It can also be seen that at the beginning of the training process, the agent obtains a high blocking probability due to the exploration process. When the exploitation process starts, the blocking probability is reduced until it converges to a steady value. This happens after 150 thousand timesteps for the NFSNet and 130 thousand timesteps for COST239, irrespective of the traffic load. Given this steady value, we assume training has finished and the trained agent can now be evaluated in a testing setting.Figure 7
Blocking probability progress for TRPO agent training in NSFNet Topology.Figure 8
Blocking probability progress for TRPO agent training in COST239 Topology.


## 3.3. TRPO Agent VS. Heuristic: Blocking Performance

Figures9 and 10 show the blocking probability achieved by the trained TRPO agent and the same heuristics, selected for blocking evaluation in the survey [50], namely, KSP-FF-FCA [61], KSP-RF-RCA [61], and KSP-SCMA XT/demand-aware [22]. Results assume operation in the C-band (320 FSU) for the NSFNet and COST239 topologies, respectively. The three heuristics apply alternated routing. KSP-FF-FCA uses the First Fit policy to select core and spectrum, KSP-RF-RCA applies a random policy to select core and spectrum, and KSP-SCMA XT/demand aware allocates different parts of the spectrum and core depending on the bitrate of the connection request. If the connection request’s demand is below a bitrate’s threshold, a First-Fit allocation policy is applied for spectrum and core assignment as long as the cross-talk levels are not exceeded; otherwise, a Last-Fit policy is applied if the connection request’s demand is above the threshold.Figure 9
Blocking probability steady average of TRPO agent trained in NSFNet topology.Figure 10
Blocking probability steady average of TRPO agent trained in COST239 topology.Compared to the best-performing heuristic, KSP-SCMA XT/demand-aware, a significant improvement in the blocking performance of the DRL approaches is observed. For example, in the NFSNet topology, at the highest load studied, the TRPO agent exhibits a blocking probability of about1.9⋅10−2, about four times slower than the blocking of 8.5⋅10−2 achieved by the heuristic. On average, considering both topologies and loads over 2000 Erlang, TRPO achieves a 4-times decrease in blocking concerning the best heuristic, being ideal for the future scenario of demand for connection requests [64] and highlighting the benefits of applying DRL techniques to the RMSCA problem.Finally, our results show that the trained agent can generalise policies for different traffic loads and spectrum resources and outperform the rule-based heuristics. The improved performance comes from the ability of the DRL to explore solutions other than those detected by the expert knowledge of the human designer of the heuristics. We have also observed that training in adverse conditions achieves good results. That is, training the agent at high traffic loads makes the agent to perform well at lower traffic loads whereas training the agent using links with reduced capacity leads to the agent to perform better in links with increased capacity. In line with previous research [33], such generalisation was not observed in terms of topology: The agent trained in the NFSNet topology did not perform well in the COST 239 topology and vice versa. Studying the benefit of using Graph Neural Networks to overcome the lack of topology generalisation is part of current research [63].


## 4. Conclusion

This paper presents a deep reinforcement learning approach applied for the first time in the literature to solve the routing, modulation format, spectrum, and core allocation problem in dynamic multicore elastic optical networks. Simulation results show that the deep reinforcement learning approach offers a significant performance advantage over the best heuristic strategy studied.Further research on improving the DRL approach performance should focus on hyperparameter tuning, applying transfer learning techniques or graph neural networks to cover a broader range of topologies with decreased computational effort, increasing the size of the data to be processed to study fibers with more cores and investigating different reward schemes that differentiate the reward according to the cause of blocking (e.g. crosstalk, capacity unavailability, fragmentation, or optical reach).Additionally, we would like to explore explainability techniques that might help understand how the agent makes its decisions to improve current heuristics.We expect these results and the code made available in the Git repository to help the research community study the benefits of deep reinforcement learning in the area of optical networks.



---

*Source: 4140594-2023-03-02.xml*