## Materials and Methods
### Hyperparameter Tuning  
Grid Search  
Before presenting any results, hyperparameter optimization is applied as a final step in machine learning. Some parameters in Equation (2) are used for controlling the learning process and must be set appropriately to maximize the learning method [19]. This is the case of the learning rate \(\alpha\) and discount factor or discount rate \(\gamma\), two parameters between 0 and 1 that evaluate how much information overwrites past information and how important are the obtained rewards compared to those obtained in further steps [20], respectively. The learning rate or step-size parameter (\(\alpha\)) affects how much the difference between a previous and a new Q-value is considered. Generally, constant small learning rates are effective in nonstationary problems in which the reward probabilities change over time. In those cases, more weight is given to recent rewards than long-past rewards. In contrast, when true values of the actions are assumed to be stationary, then the sample-average method (Equation (3)) may be used in order to average the rewards incrementally and guaranteed convergence to the true action values. \[\alpha_{t}(a)=\frac{1}{k_{a}} \tag{3}\]  
where \(k_{a}\) indicates the amount of preceding selections of action a. On the whole, the step-size parameter is essential to maximize the speed of learning. Large step-sizes cause oscillations in the value function in comparison with small ones that lead to longer learning rates [21; 22]. Regarding the discount rate (\(\gamma\)), it must be selected considering the time horizon of the relevant rewards for a particular action. The agent's discounted reward \(G_{t}\) as a function of the discount rate can be expressed as:  
\[G_{t}\doteq R_{t+1}+\gamma R_{t+2}+\gamma^{2}R_{t+3}+...=\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1} \tag{4}\]  
where \(0\leq\gamma\leq 1\) determines the present value of future rewards. \(\gamma=0\) only maximizes immediate rewards and \(\gamma=1\) takes future rewards into account strongly. In many application it is nearly always arbitrarily chosen by researchers to be near the 0.9 point [23]. Figure 3: Example of synthetic images of four different RGB colors (Black, Yellow, Red and Blue) that simulate the background, medical suit, blood traces and camera position of the HORUS problem, respectively. Emulates: (**a**) least zoom, (**b,c**) intermediate zoom, (**d**) maximum zoom. The trade-off between exploration and exploitation, one of the key challenges that arise in RL [24], is also considered at this stage. The dilemma comprises choosing between what has been learned (exploitation) or some other action to possibly increase knowledge (exploration). In general, the greedy method (that selects the action with highest learned Q-value) gets stuck performing suboptimal actions compared to the \(\epsilon\)-greedy method (Equation (5)) that uses \(\epsilon\) as a probability to randomly select an action, independently of action-value estimates [16]. The \(\epsilon\)-greedy method follows a policy \(\pi\):  
\[\pi(s)=\left\{ \begin{aligned} & random\ action\ from\ \mathbf{A}(s)& if\ \epsilon<\epsilon\\ & argmax_{a\in\mathbf{A}(s)}\ Q(s,a)& otherwise \end{aligned} \right. \tag{5}\]  
where \(0\leq\epsilon\leq 1\) is a uniform random number computed at each time step. Slightly more sophisticated strategies are Boltzmann exploration (softmax) or some derivatives of \(\epsilon\)-greedy method method that reduce \(\epsilon\) over time to try to get best of both high and low values [16; 21]. The final step in our work will be to apply grid search to exhaustively find an optimal subset of these hyperparameters. Grid-search is used to find these optimal values of the policy which lead to the most accurate predictions [19]. For this reason, the behaviour of different agents has been evaluated according to the following metrics:  
1. Average number of timesteps per episode: minimum number of steps or the shortest path, is sought per episode. 2.