## 6 Evaluation
### DRL Agent Training  
As previously described, the routing and scheduling agents are independently trained using the I2I topology shown in Figure 3, but some training-related concepts are common to both agents. First, all of the service requests between TFCs and other TFCs or TMSs have been artificially created and saved in a dataset. This allows for fair comparisons of model performance when changes are made to the implementation. Once the agent is trained, additional datasets are used to evaluate the created models. To simulate network load, a configurable number of background data flows is routed and scheduled at the beginning of each episode. This ensures the agents can find links or positions within the hyperperiod that lack sufficient resources, forcing them to explore other valid routes. Background traffic data flows share the same properties as regular flows: they are routed along the shortest available path and scheduled at the first available position starting from the current one. Background traffic data flows are described by the same types of service requests as regular data flows. Specifically, 300 background traffic data flows are generated at the beginning of each episode between all 13 nodes in the network. These background flows remain constant throughout the episode--no new service requests are generated, and no existing connections terminate. This behavior is based on the assumption that control channels between TFCs and TMSs are established as long-term connections with minimal variability over time. Each training round for both agents lasts 500,000 time steps, which takes approximately one hour to complete.  
Several training rounds are launched for the routing agent to enable it to learn how to establish optimal paths between any pair of TSN switches. Figure 5 illustrates the evolution of rewards and losses experienced by the agent during the training process. The blue lines indicate the actual results gathered during training and the orange lines indicate averaged values. The figure shows a steady increase in the average reward as the agent continues to explore and learn. Eventually, the reward reaches a steady state, where it stabilizes at a more constant level. The final training reward of the routing model is 103.87 points, indicating that the agent learned optimally most of the routes. Regarding loss, it remains consistently low throughout the entire training process, highlighting the stability and efficiency of the training methodology.  
For the scheduling agent, several training rounds are also launched. The objective is to enable the agent to be able to schedule data flows such that the residence time in intermediate nodes remains as low as possible. Figure 6 illustrates the evolution of the rewards and losses during the training process. The average rewards quickly increased and stabilized at a constant level above nine points. However, the actual evolution of rewards provides deeper insights. The minimum reward value reached since the early stages of training was around \(-1\), indicating that when the agent fails to schedule a data flow optimally, it often selects the subsequent position, resulting in only minor penalties. This behavior demonstrates the agent's ability to adapt and minimize penalties even in suboptimal scenarios. The final average reward is 9.23 points. Regarding loss, although it is significantly higher than in the routing training rounds, it becomes stable early in the training process, reflecting the model's convergence and reliability.