# Results and Achievements of the ALLIANCE Project: New Network Solutions for 5G and Beyond
## 3 The KDN Framework
### Evaluation of a DRL + GNN Agent for Routing Optimisation in Optical Networks  
A main limitation of existing DRL-based solutions for networks is that they are not able to generalise properly across different network scenarios (e.g., different topologies, configurations) than those seen during the training phase. This limits their applicability to commercial products, so that DRL agents can be trained in controlled testbeds and then be deployed in any real-world network, regardless of its topology. The reason behind this limitation is that existing DRL solutions for networking use standard neural networks (e.g., fully connected, convolutional) that are not suited to learn data from computer networks, whose information is intrinsically structured as graphs [26]. In virtue of the generalisation capabilities of GNNs to model network-related information (structured as graphs), we argue that the integration of such models into DRL agents may enable the building of more efficient solutions for network optimisation. In particular, the use of GNN could help the DRL agent generalise better to different scenarios from those observed during the training phase, which is an essential learning aspect to achieve functional solutions for networking. In this vein, the GNN model can help the agent acquire deeper knowledge about networks, by learning how the state of different network elements (e.g., forwarding devices, flows) relate to a particular target optimisation goal (e.g., maximise congestion). As a result, in the ALLIANCE project we have proposed a novel architecture that integrates a GNN model into a DRL algorithm [25]. We show that the resulting DRL + GNN agent can learn the complex relationships between the different elements of the network, reason about these relationships, and optimise the routing configuration over an Optical Transport Network scenario. In particular, we consider a KDN/SDN-based scenario, where a centralised controller receives information of new traffic demands to be routed _{source, destination, estimated bandwidth}_, and our DRL + GNN agent--running in the controller-- selects the best src-dst path for each demand, with the ultimate goal of maximising the amount of traffic volume served by the network, which is a classic Traffic Engineering goal. The proposed DRL + GNN agent implements Deep Q-Network (DQN) [19] as a reinforcement learning algorithm, and a custom GNN model (coded in TensorFlow) that incorporates state information from links (use) and their relations according to the network topology and the traffic demands routed through the network. In our experiments, we generate different network optimisation episodes, which are defined by: a network topology, and a sequence of src-dst traffic demands randomly generated that the DRL agent must incrementally allocate on particular sequences of lightpaths (i.e., end-to-end paths). We train the DRL agent for 1000 episodes, where each episode ends when the agent allocates a demand to a path (i.e., sequence of lightpaths) that does not have enough available capacity. The immediate reward is the traffic volume of the current traffic demand if it was successfully allocated by the agent, and zero otherwise. Thus, the agent is intended to maximise in the long-term the amount of traffic volume successfully routed. Please note that since the generation of traffic demands is random, the agent cannot exploit any meaningful information to predict the traffic demands that will come in the future, which makes this problem more challenging. Figure 2a,b show a comparison of the proposed DRL + GNN agent [25] against a state-of-the-art DRL solution using fully connected neural networks (SoA DRL) [30], also presented as an earlier work within the ALLIANCE project. In Figure 2a, we can observe that when both DRL-based solutions are trained and evaluated in the same network topology (NSFNet; 14 nodes), they achieve a similar score (i.e., amount of src-dst traffic volume successfully routed through the network).