## 6 Evaluation
### Agent Description and Configuration  
Both the routing and scheduling agents are implemented using the Rainbow algorithm [25], an enhanced version of the deep Q-network (DQN), which is one of the most popular value-based DRL algorithms. Rainbow incorporates several improvements to DQN, focusing on enhancements to the neural networks and the replay buffer. The deployed neural networks include noisy layers and a physical differentiation of value and advantage layers, following the Rainbow architecture. Additionally, Rainbow employs two copies of the same neural network for training and target evaluation. The replay buffer is also enhanced to prioritize experiences that provide greater learning value. This modified version is called Prioritized Experience Replay (PER). Moreover, the learning process is further enhanced by calculating distributions of expected Q-values instead of single discrete values. Additionally, a mechanism called n-step learning is introduced to assess how well it is taking any action in the long term. The noisy layers of the neural network force the agent to explore until it learns how to ignore the noise, so it is used as an alternative exploration strategy to \(\epsilon\)-greedy. Moreover, an ADAM optimizer is used for updating the neural network. The complete set of hyperparameters configured for both agents is provided in Table 1.  
A deeper description of all the hyperparameters is provided to facilitate the understanding of the selected values:  
* The replay buffer size represents the number of experiences that can be accumulated in the replay buffer. Experiences are used to train the agent as long as they are stored at the replay buffer. Once the replay buffer is full, the earliest gathered experiences are discarded. In total, 1,000,000 samples is a typical value, as larger ones do not usually provide better results, and smaller ones provoke slower training convergence. * The batch size is the amount of experiences that are used for training the agent at each time step. In general, as this value increases, training becomes faster but less accurate, so a trade-off must be established. Based on previous experience, we use a size of 32 samples. * The target network update periodicity configures how often the weights and biases of the online neural network are copied to the target neural network. As this value decreases, training becomes faster, but the risk of losing convergence increases. As in the case of the batch size, a trade-off must be achieved. We use a periodicity of 1000 time steps due to previous experience. * The discount factor modifies the relevance of long-term future rewards against short-term ones. Values for this hyperparameter are usually high, which is recommended when the last reward of the episode is very relevant to the agent. For this reason, we use a discount factor equal to 0.99. * The learning rate is the factor used to update the neural network weights during the backpropagation process. As an ADAM optimizer is used, the provided value is just the initial one, as the optimizer modifies it during training. It is recommended to use initial values between 0.0001 and 0.01, so we use the minimum value of this range. * The soft update coefficient is related to the percentage of weights not passed to the target network when updating it. This mechanism provides smoother variations in the target network, which, as a consequence, enhances learning stability. We use a soft update coefficient of 0.005 to benefit from this feature. * Both hyperparameters related to the PER mechanism (\(\alpha\) and \(\beta\)) are used for giving priorities to those experiences from which the agent can learn better policies. While alpha is the hyperparameter that provides the desired priorities, it might over-prioritize some experiences, so beta is used as a scaling factor. Typical values for \(\alpha\) are between