# Petri Net Toolbox for Multi-Robot Planning under Uncertainty
## 3 Materials and Methods
### Policy Synthesis
\]  
where \(P_{\geq 1}\) represents the set of places with at least one token in a given marking, and \(\Lambda\) is the set of all enabled uncontrollable immediate transitions as defined in Section 3.2. The existence of an equivalent MDP for the GSPNR enables the use of dynamic programming solvers to obtain the optimal policy. In this case we implement value iteration [26]. The yielded optimal policy \(\pi^{*}_{M_{G^{\prime}}}\) dictates which actions should be taken in each state, and, in the GSPNR, this corresponds to selecting an immediate transition for each marking and is defined as \(\pi^{*}_{G^{\prime}}\). As the GSPNR to MDP conversion algorithm explores the GSPNR reachability graph it also builds a function that allows mapping the equivalent MDP states back to GSPNR markings. This function, defined as \(Map_{s}^{m}:R(G^{\prime})\to S\), sets which state in \(M_{G^{\prime}}\) corresponds each marking in \(G^{\prime}\). It enables the conversion from the optimal policy obtained in the MDP, \(\pi^{*}_{M_{G^{\prime}}}\), to the optimal policy in the GSPNR, \(\pi^{*}_{G^{\prime}}\). Thus, the optimal policy in the GSPNR is defined as \(\pi^{*}_{G^{\prime}}(m)=\pi^{*}_{M_{G^{\prime}}}(Map_{s}^{m}(m))\forall m\in R (G)\). Figure 6 shows an example of the GSPNR to MDP conversion, where \(Map_{s}^{m}([2,0,0])=S_{1}\), \(Map_{s}^{m}([1,1,0])=S_{2}\) and \(Map_{s}^{m}([1,0,1])=S_{3}\). Consider that the reward function gives a reward of 1 in place \(P_{2}\) and zero everywhere else. Thus, value iteration will output the following optimal policy: \(\pi^{*}_{M_{G^{\prime}}}(S_{1})=\bot\), \(\pi^{*}_{M_{G^{\prime}}}(S_{2})=t_{3}\) and \(\pi^{*}_{M_{G^{\prime}}}(S_{3})=t_{4}\). Finally, the \(Map_{s}^{m}\) function allows mapping \(\pi^{*}_{M_{G^{\prime}}}\) into \(\pi^{*}_{G^{\prime}}\). The policy in state \(S_{2}\) and state \(S_{3}\) has a direct correspondence. In \(S_{1}\) the action \(L\) represents waiting for the outcome of executing action \(P_{1}\), that is, waiting for the uncontrollable event \(t_{1}\) or \(t_{2}\) to trigger. The obtained policy is optimal in the designed GSPNR model. Thus, it is limited by how well the model captures reality. So inaccurate models may lead to erroneous policies that produce unwanted behaviors. To detect these issues, before moving to the real scenario deployment, this toolbox provides a model simulation method implemented under the _evaluate_policy()_ API function. The model simulation is a Monte Carlo based method that samples from the model by firing transitions and progressing through the state space collecting timestamps. It provides the states, actions selected, and the duration of each action during the simulated horizon. The simulation horizon is given by the  
Figure 6: (**Left**) GSPNR example; (**Right**) Equivalent MDP obtained from the GSPNR on the left. user as the amount of time in seconds that the simulation should run. The immediate transitions are fired according to the given policy and the timed transitions according to the exponential distribution generated from their rates. The user can then compute, from the simulation data, multiple properties that help evaluating the policy performance and the model correctness. For instance, it can assess: the mean time to reach a certain state; the mean time spent in each state; the probability of reaching a certain state within a given horizon; the average throughput of a certain action; the average number of robots in each action or decision state; or how the robot probability distribution per action or decision state evolves over time. Even when the model is correct this can be used to make it more efficient and realistic through the analysis of how changes in the GSPNR model affect these properties. On top of that, it is possible to simulate a high number of states and transitions in a short amount of time since the method does not need to actually wait for the actions to execute.