Patent Description:
Network managers typically use their discretion to identify and overwrite IGP metrics of links between nodes to address network inefficiencies or failures and to meet SLOs. Typically, network managers need to analyze the routing results at the steady-state, when everything in the network domain works as expected but also at various failure states, when some components in the network domain fail. Due to the large number of links and failure states, it is extremely challenging, and in some instances not possible, to evaluate and decide optimal IGP metrics to overwrite while considering all possible failure states. Thus, in practice, IGP metric overwrites usually result from local analysis with only a few failure states. Additionally, network managers typically have no ability to evaluate the quality of a given set of IGP metrics absent implementing them. As a result, the proposed IGP metrics are usually not optimal. Poor IGP metrics can lead to various issues, such as elevated network failure risk, latency, poor user experience, and higher network build costs.

Moreover, the baseline IGP metrics are generally correlated with latency. In this regard, links assigned with baseline IGP metrics indicative of low latency are often selected for routing paths as they likely correspond to the shortest, and thus quickest paths on the network. However, baseline IGP metrics based on latency do not work well in all network settings. In this regard, many properties other than latency can contribute to the decision on the best routing paths for a network. For instance, a path between a source node and destination node may have the lowest latency but may also have the highest failure probability and/or limited bandwidth capabilities that are detrimental to data transmission on the network domain. As IGP metrics are not typically correlated with properties such as failure probability and capacity, the baseline IGP metrics cannot account for these properties.

<NPL>), discloses machine learning (ML)-guided routing for intradomain traffic engineering, wherein different ML paradigms are applied for flow optimization.

The claimed subject matter is defined by the independent claims. Dependent claims describe embodiments thereof. The present disclosure relates to optimizing interior gateway protocol (IGP) metrics using reinforcement learning (RL). An IGP metric optimizer may optimize IGP metrics with respect to any deterministic network-based optimization objective function. The inputs to the IGP metric optimizer may be the network cross-layer topology, a list of demands, a set of probabilistic failures and a subset of links to tune. The IGP metric optimizer may use RL to optimize the objective function to determine an updated set of IGP metrics based on the inputs. The updated set of IGP metrics may then be used to assign routing paths between nodes for a network domain.

One aspect of the disclosure provides a method for tuning IGP metrics for a network domain. The method includes receiving, by one or more processors, a topology (G) of a network and a set of flows (F); receiving, by the one or more processors, an objective function; and optimizing, by the one or more processors using reinforcement learning, the objective function based on the received topology and the one or more flows F.

Another aspect of the disclosure provides a system comprising one or more processors. The one or more processors are configured to receive a topology (G) of a network domain and a set of flows (F); receive an objective function; and optimize, using reinforcement learning, the objective function based on the received topology and the one or more flows F.

Yet another aspect of the disclosure provides a non-transitory computer readable medium storing instruction, that when executed by one or more processors, cause the one or more processors to: receive a topology (G) of a network domain and a set of flows (F); receive an objective function; and optimize, using reinforcement learning, the objective function based on the received topology and the one or more flows F.

In some instances, the topology G equals (V, E), where V is a set of nodes on the domain network and E is the set of edges between each node in the set of nodes on the domain network.

In some instances, each of one or more flows F equals {f_j}, j = <NUM>. |F|, where j is the index of the flow and each flow f_j is a tuple comprising (src_j, dst_j, demand_j, SLO_j), where src_j and dst_j are the source and destination node, respectively, demand_j is the size of the flow, and SLO_j is the service level objective (SLO) requirement for the flow.

In some instances, a network utility is determined by a routing simulator for each failure state. In some examples, optimizing the objective function is further based on the network utility of each failure state determined by the routing simulator.

In some instances, updated IGP metrics are determined based on the optimization of the objective function. In some examples, the IGP metrics for the network domain are updated with the updated IGP metrics.

The technology described herein address the deficiencies of known techniques for identifying and overwriting IGP metrics, by identifying and overwriting IGP metric values with values that improve network performance determined by optimizing an objective function formulated as a reinforcement learning problem solvable by an IGP metric optimizer. The IGP metric optimizer is a framework that is configured to optimize the given objective function by tuning the IGP metrics of given links. The updated set of IGP metrics may then be used to assign routing paths between nodes for a network domain.

The technology described herein is advantageous because it provides an end-to-end solution that can automatically tune IGP metrics to optimize an arbitrary objective function for an arbitrary pair of network topology, network states, including failure states, demands, and policies. Moreover, the technology provides the ability to handle many objective functions. For example, traditional optimization-based formulation requires explicit mathematical expressions of the objective function, dynamics function, constraint functions, etc., leading to problems that are so complex, that current systems are unable to consider many failure states at the same time. The IGP metric optimizer is capable of handling large numbers of failure states, learn from previous training experiences, and accelerate training in future tasks.

<FIG> illustrates an example software architecture and dataflow of an IGP metric optimizer <NUM>. As shown, the software architecture includes a routing simulator <NUM> and reinforcement learning (RL) agent <NUM>. The IGP metric optimizer <NUM> receives a set of inputs <NUM>. The inputs include network topology <NUM> of a network domain, demands <NUM>, probable failure states <NUM> of the links connecting nodes in the network domain, and an objective function <NUM>. Although not shown, the inputs <NUM> may include a subset of links within the network topology to tune. Based on some or all of the inputs <NUM>, the routing simulator <NUM> determines a network utility <NUM> for a set of flows and network topology at some or all failure states. The RL agent <NUM> uses the determined network utilities along with the inputs <NUM> to optimize the objective function <NUM> to determine updated IGP metric values <NUM>.

The updated IGP metric values may be provided to the routing simulator <NUM> which may determine an updated network utility based on the updated IGP metrics. Based on the updated network utility the RL agent <NUM> may optimize the objective function <NUM> to determine a new set of updated IGP metric values. This process may continue indefinitely, with the routing simulator <NUM> determining updated network utility values and the RL agent <NUM> determining new, updated IGP metric values. The process may be stopped by a network manager or other user of the IGP metric optimizer <NUM>, such as when the network manager determines the updated IGP metric values are good enough. Alternatively, the process may be stopped after a predetermined number of runs, a predetermined time period, after the change to updated IGP metric values between runs is below a threshold value, etc. The final set of updated IGP metric values may be output by the IGM metric optimizer <NUM>, illustrated as the outputted IGP metric values <NUM> in <FIG>. The outputted IGP metric values may be used to assign routing paths between nodes in the network domain.

<FIG> illustrates an example system <NUM> including network devices for performing aspects of the present disclosure. The system <NUM> includes network devices 230a-230e (collectively "network devices <NUM>"), including computing devices 230a and 230b. All network devices may be communicatively coupled to a network <NUM>.

The network devices <NUM>, such as network devices 230c-230e may include switches, routers, modems, gateways, software-defined-network applications, or other such components that may be connected to or otherwise form part of the network <NUM>. The network devices <NUM> may include physical devices, virtualized devices, replicas, etc. In some examples, network devices may include computing devices such as servers, general purpose computers, PDAs, tablets, mobile phones, smartwatches, terminals, set top boxes, and other such devices. For instance, and as further illustrated in <FIG>, network devices 230a and 230b are computing devices. In another example, network device 230c may be a top-of-rack switch that has a set of servers attached to it. In yet another example, network device 230d may be a router that is attached to a plurality of switches, which in turn connect to a plurality of computing devices. Additionally, network devices <NUM> may also include services that are implemented on the network devices. Although <FIG> illustrates only network devices 230a and 230b as computing devices, the system <NUM> can include any number of computing devices. Moreover, although only network devices 230a-230e are illustrated in system <NUM>, the system may include any number of network devices.

Although <FIG> illustrates the network devices 230a-230e as being connected directly to the network <NUM>, the network devices may be connected to the network <NUM> via other network devices. For instance, network devices 230a and 230b may be connected to the network <NUM> through another network device, such as one of network devices 230c-230e. Similarly, network devices 230c-230e may be connected to the network <NUM> via other network devices.

Network devices <NUM> may be coupled to other network devices to form a link on a communication pathway on the network <NUM>. For example, a first network device may connect to a second network device, which may connect to other network devices or directly to the network. <FIG> illustrates an example network domain <NUM> including network devices <NUM>-<NUM>, also referred to herein as nodes. The nodes may be connected by edges <NUM>-<NUM>, also referred to herein as links. For example, node <NUM> is connected to node <NUM> via links <NUM>, <NUM>, and <NUM>, with link <NUM> being between nodes <NUM> and <NUM>, link <NUM> being between nodes <NUM> and <NUM>, and link <NUM> being between nodes <NUM> and <NUM>. In some instances, nodes may be connected via multiple paths. For instance, node <NUM> is illustrated as being connect to node <NUM> via another including links <NUM>, <NUM>, and <NUM>, with link <NUM> being between nodes <NUM> and <NUM>, link <NUM> being between nodes <NUM> and <NUM>, and link <NUM> being between nodes <NUM> and <NUM>. The routing path selected between nodes may be based on updated IGP metrics, as described further in.

A network domain may include a collection of network devices, such as network devices <NUM>, referred to herein as "nodes," that share a common domain. In some instances, a network domain may include a domain and one or more sub-domains. Each domain and sub-domain may be under common control by one or more administrators.

Network devices may include components typically present in general purposes computers, servers, and routers. For instance, and as further illustrated in <FIG>, network device 230a is a computing device containing a processor <NUM> and memory <NUM>. The memory <NUM> can store information accessible by the processor <NUM>, including instructions <NUM> that can be executed by the processor <NUM>. Memory <NUM> can also include data <NUM> that can be retrieved, manipulated or stored by the processor <NUM>. The memory <NUM> may be a type of non-transitory computer readable medium capable of storing information accessible by the processor <NUM>, such as a hard-drive, solid state drive, flash drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, write-capable, and read-only memories. The subject matter disclosed herein may include different combinations of the foregoing, whereby different portions of the instructions <NUM> and data <NUM> are stored on different types of media. The processor <NUM> can be a well-known processor or other lesser-known types of processors. Alternatively, the processor <NUM> can be a dedicated controller such as an ASIC.

The instructions <NUM> can be a set of instructions executed directly, such as machine code, or indirectly, such as scripts, by the processor <NUM>. In this regard, the terms "instructions," "steps" and "programs" can be used interchangeably herein. The instructions <NUM> can be stored in object code format for direct processing by the processor <NUM>, or other types of computer language including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. The instructions <NUM> may provide for implementing an IGP metric optimizer, such as IGP metric optimizer <NUM>, including RL agent <NUM> and routing simulator <NUM>, as described herein.

The data <NUM> can be retrieved, stored or modified by the processor <NUM> in accordance with the instructions <NUM>. For instance, although the system and method are not limited by a particular data structure, the data <NUM> can be stored in computer registers, in a distributed storage system as a structure having a plurality of different fields and records, or documents, or buffers. The data <NUM> can also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII or Unicode. Moreover, the data <NUM> can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories, including other network locations, or information that is used by a function to calculate relevant data.

Although <FIG> functionally illustrates the processor <NUM> and memory <NUM> as being within the same block, it will be understood by those of ordinary skill in the art that the processor and memory may actually comprise multiple processors and memories that may or may not be stored within the same physical housing. For example, some of the instructions and data may be stored on removable CD-ROM and others within a read-only computer chip. Some or all of the instructions and data may be stored in a location physically remote from, yet still accessible by, the processor. Similarly, the processor may actually comprise a collection of processors that may or may not operate in parallel.

Computing device 230b, and any other network devices, may be configured similarly to computing device 230a. In this regard, computing device 230b may have some or all of the same components of computing device 230a. For example, computing device 230b includes a processor <NUM> and memory <NUM> storing instructions <NUM> and data <NUM>. Moreover, computing device 230b may include other components normally found in a personal computer such as a CD-ROM/DVD/Blu-ray drive, hard drive, and a display device <NUM>, for example, a monitor having a screen, a projector, a touch-screen, a small LCD screen, a television, or another device such as an electrical device that can be operable to display information processed by a processor, speakers, a modem and/or network interface device, user input <NUM>, such as a mouse, keyboard, touch screen or microphone, and all of the components used for connecting these elements to one another. Other computing devices and network devices in accordance with the systems and methods described herein may be configured similar to computing devices 230a and 230b.

The components in system <NUM>, including network devices 230a-230e and storage system <NUM>, may be capable of direct and indirect communication such as over network <NUM>. For example, using a network socket, the network device 230a can communicate with another network device attached to network <NUM>, through an Internet protocol. The network devices 230a-230e can set up listening sockets that may accept an initiating connection for sending and receiving information. The network <NUM> itself can include various configurations and protocols including the Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks, collections of private networks, such as a network domain, etc., using general communication protocols and/or communication protocols proprietary to one or more companies. The network <NUM> can support a variety of short- and long-range connections. The short- and long-range connections may be made over different bandwidths, such as <NUM> to <NUM> (commonly associated with the Bluetooth® standard), <NUM> and <NUM> (commonly associated with the Wi-Fi® communication protocol); or with a variety of communication standards, such as the LTE® standard for wireless broadband communication. The network <NUM>, in addition or alternatively, can also support wired connections between the devices 260a-260e, as well as with the storage system <NUM>, including over various types of Ethernet connection.

As described herein, aspects of the disclosure can be implemented according to a variety of different configurations and quantities of computing devices, including in paradigms for sequential or parallel processing, or over a distributed network of multiple devices. Moreover, aspects of this disclosure can be implemented in digital circuits, computer-readable storage media, as one or more computer programs, or a combination of one or more of the foregoing. The computer-readable storage media can be non-transitory, e.g., as one or more instructions executable by a cloud computing platform and stored on a tangible storage device.

For example, the IGP metric optimizer <NUM> may be implemented by multiple computing devices, such as computing devices 230a and 230b. For example, computing device 230a may implement the RL agent <NUM> and computing device 230b may implement the routing simulator. In other examples, inputs <NUM> may be provided by computing devices 230a, 230b, and/or some other network device or storage system. In some instances, the IGP metric optimizer may be implemented by one or more computing devices not connected to the network.

The IGP metric optimizer <NUM> receives a set of inputs <NUM>, as shown in <FIG>. The routing simulator <NUM>, within the IGP metric optimizer <NUM>, determines a network utility <NUM> for a set of flows and network topology at some or all failure states. The RL agent <NUM> uses the determined network utilities along with the inputs <NUM> to optimize the objective function <NUM> to determine updated IGP metric values <NUM>. This process may repeat, with the updated IGP metric values may be provided to the routing simulator <NUM>. The routing simulator may determine an updated network utility for each link based on the updated IGP metrics. Using these updated IGP metrics, the RL agent <NUM> may optimize the objective function <NUM> again to determine a new set of updated IGP metric values. Upon completion of the process, the IGP metric optimizer <NUM> may output IGP metric values that can be used to assign routing paths between nodes in the network domain.

The IGP metric optimizer <NUM> may operate by solving the following Problem <NUM>, as outlined below.

the IGP metric optimizer optimizes the objective function u over all IGP metrics given the topology G and the set of flows F to determine an optimal positive-integer-valued IGP metric (IGPO) that maximizes network utility. (Problem <NUM>).

Network utility may include any type of measure of the network and its configuration, such as transient risks, network steady-state risks, and user experience. In one example, network utility may include how much flow is unrouted at each failure state. In another example, network utility may include how much flow will be dropped or violate its routing policy shortly after a failure occurs.

The IGP metric optimizer <NUM> outputs an IGPo for each link. The outputted IGPO for each link may be provided to the routing simulator <NUM> to simulate the implementation of the IGPO metrics. In this regard, the routing simulator <NUM> may use the IGPO to get routing results, such as paths and demand routed in each path, for any given flow or any set of flows, at any given failure state. The routing simulator <NUM> may output routing paths for each flow at some or all states, including steady-state and failure states.

Although each flow f_j in the set of flows F is described as comprising a tuple including elements src_j, dst_j, demand_j, and SLO_j, the tuple may include more or fewer elements. Additionally, although demand_j is described as being the flow size measured in Gbps, the flow may be measured in any metric, such as MBps, mbps, etc. An example flow may include a network node in San Francisco (the source node), a node in New York City (the destination node). The demand of the flow may be 100Gbps and the SLO may be <NUM>% availability. There may not be any direct link or adjacency between the nodes in San Francisco and New York, so the flow may traverse multiple links on the network to route from the node in San Francisco to the node in New York City.

The topology G and set of flows F are typically known and provided as inputs, such as inputs <NUM>, to the IGP metric optimizer <NUM>. However, when IGP metrics are also considered as optimization variables, determining an objective function u for the IGP metric optimizer <NUM> may be difficult. In this regard, it may be difficult to find a satisfactory objective function because of the complexity of the problem. For example, the size of the network, the number of flows, and the number of failure states all add complexity to the problem. Moreover, there may be many different types of network utilities that a network manager wants to encode into the objective function, but it may only be possible to solve for a single scalar objective function. Further yet, each objective function has to be modeled and designed, which may itself be difficult.

Once an objective function is developed, the objective function may be used to evaluate routing results. Thus its value depends directly on routing results and indirectly on IGP metrics. A routing result specifies the amount of traffic that goes through each feasible routing path for all the flows F in topology G. Typically, all the feasible routing paths should be used to minimize the amount of unrouted demand. However, the number of feasible routing paths grows exponentially with the number of edges E. Further, the number of possible failure states grows exponentially as topology G grows. Thus, evaluating all feasible routing paths and failure states through the objective function can require large amounts of memory and processing power. The IGP metric optimizer <NUM>, which includes an RL agent <NUM> described herein, may be used to optimize a wide range of objective functions, so long as their values can be determined by the IGP metrics of all the links. Such objective functions may include deterministic objective functions. Accordingly, the same IGP metric optimizer can be used to solve a wide range of optimization problems.

The IGP metric optimizer <NUM> includes a routing simulator <NUM> and an RL agent <NUM>, as shown in <FIG>. The routing simulator <NUM> may be considered a black box that takes IGP metric configurations as inputs and gives network utilities as outputs. The RL agent <NUM> keeps a value function that predicts the quality of any given IGP metric, and a policy that decides the next IGP metric to sample. As training goes on, the RL agent <NUM> collects more IGP metrics and observes their respective network utilities. Based on the collected IGP metrics and their respective network utilities, the RL agent <NUM> can update its value function and policy, and increase the likelihood to sample IGP metrics with more favorable network utilities.

To train the RL agent, a routing simulator that can deterministically return a routing result for any given topology G of a network domain, a set of Flows F, and IGP metric is used, as further shown in <FIG>. Accordingly, Problem <NUM> can be modified into Problem <NUM>, where when given:.

the IGP metric optimizer may optimize the objective function for an optimal positive-integer-valued IGP metric (IGPO) that maximizes u(f(G, F, IGP)) over all IGP metrics.

For the IGP metric optimizer <NUM>, the IGP metric generation process may be modeled as a Markov decision process (MDP) and optimized by an RL agent, such as RL agent <NUM>. There are two components for a standard RL problem including an environment <NUM> and an agent <NUM>. The environment <NUM> and agent 402interact with each other, as shown in <FIG>. In this regard, the environment <NUM> tracks its current state 's', takes state transitions in response to the agent's actions, and gives reward feedback 'r' to the agent <NUM>. <NUM> The agent observes the environment's state and takes actions 'a' in the environment <NUM>. The agent <NUM> and environment <NUM> typically interact for multiple steps, which can be either finite or infinite. The objective is to find a policy for the agent <NUM>, which is a mapping from environment states to distributions over agent actions, that maximizes the expected total (or discounted) reward (referred to herein as utility,) throughout the interaction.

The interaction between an environment <NUM> and the agent <NUM> may be modeled as a MDP M = (S, A, T, R, γ), where: S is a set of states, A is a set of actions, T: S × A → D(S) is a mapping from state-action pairs to distributions over the next state, R: S × A <MAT>, is a mapping from state-action pairs to reward values; and γ is a discount factor, typically between <NUM> and <NUM>. When solving the RL problem, the goal is to find a policy that maps from states to action π: S → (A) distributions, that maximize the expected discounted reward:
Note that when γ = <NUM> objective J is the expected total reward.

As explained above, the IGP metric generation process may be modeled as an MDP and the goal of the RL agent <NUM> is to find a policy to maximize the expected reward, or utility, when interacting with the MDP. In this regard, the objective function may be part of the reward function of the MDP. The interaction between the environment <NUM> and the agent <NUM>, which may be compared to RL agent <NUM>, is modeled as an MDP. A complete set of IGP metrics may be generated in multiple steps. At each step, the agent <NUM> may observe its current state, which includes the already-decided IGP metrics, and determine an action to take. Each action taken by the agent may decide the IGP metric of one link. After the agent <NUM> takes its action, the MDP may respond by transiting the current state to another one that includes the newly decided IGP metric. The environment <NUM> may send a reward signal to the RL agent. After multiple steps, when all the links get IGP metrics, the reward will be the network utility for the newly generated IGP metric. Before that, the reward at each step is <NUM>.

In an implementation, each link on a network may be assigned an index. Then for each link, an IGP metric value may be assigned in the same order as the link indices. The procedure for generating IGP metrics is illustrated in <FIG> illustrates an IGP metric generation procedure for an <NUM>-link network topology across nine steps. In the first <NUM> steps, the RL agent, such as RL agent <NUM> of the IGP metric optimizer <NUM>, generates the IGP metric of one link at each step. In the last step, the RL agent evaluates the whole IGP metric using the objective function.

In step <NUM>, corresponding to time t = <NUM>, the network utility for all links is <NUM>. At step <NUM>, corresponding to time t = <NUM>, the IGP metric for a first link is determined to be <NUM>. At step <NUM>, corresponding to time t=<NUM>, an IGP metric for a second link is determined to be <NUM>. This process continues, with an IGP metric being found for each link through time t = <NUM> at step <NUM>. At step <NUM>, corresponding to time t = <NUM>, the IGP metrics of all links is determined, as further shown in <FIG>. After determining a complete set of IGP metrics, the network utility may be evaluated. For clarity, not all steps are illustrated.

The interpretation of each component in the MDP for the IGP metric optimizer is as follows:.

Any deterministic function may be used as an objective function. However, a proper network utility function may increase the success of RL-based optimization. In this regard, the objective function is typically a proxy of desired properties for a network domain. Although many variables may be of interest while evaluating a routing result, such as network risk, maximum latency, minimum link availability, there is only one utility function that can be optimized for each experiment. Therefore some tradeoffs have to be made among these variables, such as assigning different weights to different variables, or adding extra penalty terms only if some variables go beyond some given thresholds. Still, the selection of weights and thresholds is problem-dependent.

The RL agent <NUM> may solve the RL problem using a deep RL approach called proximal policy optimization (PPO). Although other RL approaches can also be used, such as the Reinforce algorithm and search-based methods such as coordinate ascent, simulated annealing, and regularized evolution. With PPO, the RL agent, such as RL agent <NUM>, may approximate two functions including a value function and a policy function. Each function may be represented as a neural network. The value function may map each state to a predicted value, which is the predicted future discounted reward when the RL agent <NUM> starts from this state and takes its current policy. The policy maps each state into a distribution over actions.

An illustrative block diagram of the algorithm is shown in <FIG>. Each iteration of the training process contains two steps including a sampling step followed by an updating step. In the sampling step, the RL agent, labeled PPO Agent <NUM>, which may be compared with RL agent <NUM> and agent <NUM>, may interact with the environment by taking its policy to generate new IGP metric samples. The samples are saved in an episodic replay buffer (ERB) <NUM>, which may be considered a queue of some fixed size. Since IGP metrics are generated in multiple steps, each IGP metric may be interpreted as a trajectory in the ERB. Each trajectory is composed of a sequence of transitions, where each transition, at a time 't', includes the state before the transition (st), the action that is taken (at), the next state (st+<NUM>), and the reward received for this transition (rt). The trajectory is the basic unit that can be sampled from the ERB. In other words, transitions corresponding to the same IGP metric can only be sampled at the same time from the ERB.

In the updating step, the RL agent <NUM> may randomly sample a certain number of trajectories from the ERB and use the trajectories to update both the value network and the policy network. Training may be done by gradient descent. The algorithm may terminate either after a given number of iterations, or when the RL agent fails to make enough progress in the last given number of iterations.

Claim 1:
A method for tuning IGP metrics for a network domain, comprising:
receiving, by one or more processors (<NUM>), a topology (<NUM>), G, of a network (<NUM>) and a set of flows, F, wherein the topology G equals V, E, where V is a set of nodes (<NUM>) on the domain network and E is the set of edges between each node in the set of nodes (<NUM>) on the domain network;
receiving, by the one or more processors (<NUM>), an objective function (<NUM>);
optimizing, by the one or more processors (<NUM>) using reinforcement learning, the objective function (<NUM>) based on the received topology and the one or more flows F; and being characterized by:
determining, by a routing simulator (<NUM>), a network utility (<NUM>) for each failure state (<NUM>) in the set of edges E;
wherein optimizing the objective function (<NUM>) is further based on the network utility of each failure state (<NUM>) determined by the routing simulator.