## 6 Evaluation
### Simulation Results  
As previously described, the models that are trained are also evaluated to test their performance, with all evaluations using the pre-trained routing and scheduling models. The evaluations involved assigning resources to 1000 service requests and analyzing the results at the end of each episode. The joint evaluation of the routing and scheduling models follows a sequential decision-making process between the two agents. Given that scheduling states depend on routing actions (as shown in Figure 7), the evaluation begins by sending the first environmental state to the routing agent. The routing agent decides on the first link to form the path. Subsequently, the routing agent's action is used to determine the position vector of the link where the routing agent is about to route data  
Figure 5: The reward function over episodes (**left** plot) and the loss function over steps (**right** plot) of a routing model during training. Figure 6: The reward function over episodes (**left** plot) and the loss function over steps (**right** plot) of a routing model during training. frames. The scheduling agent then decides which position should be used for scheduling data frames. Both actions are then sent back to the environment, allowing it to transition to the next state. This process is repeated until the service request is fully processed. In terms of routing, Table 2 shows the obtained results. The rows represent all the routes originating from each TSN switch, while the columns represent all routes ending at each TSN switch. Each field is marked with the following symbols:  
* \(\surd\) indicates that the shortest path is always established. * \(\sim\) indicates that the data flows can reach their destination without following the shortest path but within the maximum delay bound. * \(\times\) indicates that no path can be established within the maximum delay bound. In summary, there are 156 combinations of source and destination TSN switches, representing 156 optimal routes to learn. Among these, 111 are optimally learned (representing 71% of cases), 3 routes are partially learned optimally while still reaching their destinations within delay limits, 38 routes are established without following optimal paths but stay below the delay limits (24% of cases), 2 routes can only be partially established, and 2 routes cannot be established at all. Regarding scheduling, 97% of operations are optimal, with the maximum error being one position, which adds an extra residence time equivalent to a single slot time (100 microseconds). The remaining 3% of operations are suboptimal because of the model's  
performance limitations. However, the agent has learned a policy that is able to add the minimum possible sojourn time when it is not making optimal decisions. Figure 8 shows the cumulative distribution function (CDF) of relative delays after jointly evaluating the routing and the scheduling models. The deviation was calculated as the difference between the actual end-to-end delay (from the agents' decisions) and the theoretical delay of the optimal route. Approximately 80% of data flows reach their destinations with the minimum possible delay, and 95% of flows experienced extra delays of 6 milliseconds or less. Most of the extra delays are multiples of 1 millisecond due to routing issues, while smaller increments, in multiples of 100 microseconds, were caused by scheduling issues. Overall, routing issues have a more significant impact on extra delay than scheduling issues. The models that are created are tested with several background traffic patterns. Specifically, several evaluations with 600 and 900 background traffic data flows are performed. Regarding routing, adding more background traffic data flows does not affect the number of routes that are learned, as the routes that are not established keep very low values for all cases. The most significant changes are due to the congestion of optimal paths that is experienced in the network when the background traffic level is increased.