Related Work
Uncertain MDPs have (often implicitly) been used by reinforcement learning (RL) algorithms, for instance, to optimize the exploration/exploitation trade-off by guiding the RL agent towards unexplored parts of the environment, following the optimism in the face of uncertainty principle [Jaksch et al., 2010 , Fruit et al., 2017 ] .
We use the same principle in the exploration phase of our procedure, but compute robust policies as output to account for the uncertainty in an adversarial (or pessimistic) way.
Similarly, uncertain MDPs are used to compute robust policies when the data available is limited [Nilim and Ghaoui, 2005 , Wiesemann et al., 2013 , Russel and Petrik, 2019 ] .
Such robustness is connected to a pessimistic principle that has been effective in offline RL settings [Lange et al., 2012 ] , where the agent only has access to a fixed dataset of past trajectories, meaning it needs to base decisions on limited data [Rashidinejad et al., 2021 , Buckman et al., 2021 , Jin et al., 2021 ] .
Likewise, our method may be stopped early and return a policy before the problem is fully explored.

Robust RL concerns the standard RL problem, but explicitly accounts for input disturbances and model errors [Morimoto and Doya, 2005 ] .
Often there is a focus on ensuring that a reasonable performance is achieved during data collection.
To that end, Lim et al. [ 2013 ] and Derman et al. [ 2019 ] sample trajectories using a robust policy, which can slow down the process to find an optimal policy.
We assume sampling access to the underlying MDP, which allows us to use more efficient exploration.
It should be noted that we only use uMDPs as an intermediate model towards learning a standard MDP, whereas in robust RL the model itself may also be an (adversarial) uncertain MDP.
As a result, our method converges to an MDP, while robust RL attempts to learn and possibly converge to an uncertain MDP.

Learning MDPs from data is also related to model learning , which typically assumes no knowledge about the states and thus iteratively increases the set of states [Vaandrager, 2017 ] .
In [Tappler et al., 2019 , 2021 ] the L^{*} algorithm for learning finite automata is adapted for MDPs.
These methods only yield point estimates of probabilities, and make strong assumptions on the structure of the MDP that is being learned. Ashok et al. [ 2019 ] use PAC-learning to estimate the transition functions of MDPs and stochastic games in order to perform statistical model checking with a PAC guarantee on the resulting value.
In [Bacci et al., 2021 ] the Baum-Welch algorithm for learning hidden Markov models is adapted to learn MDPs.

Finally, the literature distinguishes two types of uncertainty: aleatoric and epistemic uncertainty [Hüllermeier and Waegeman, 2021 ] .
Aleatoric uncertainty refers to the uncertainty generated by a probability distribution, like the transition function of an MDP, and is also known as irreducible uncertainty .
In contrast, epistemic uncertainty is reducible by collecting and accounting for (new) data.
Our work can be seen as adding an additional layer of epistemic uncertainty on the probability distributions of the transition function that is then, by gathering and including more data, reduced.