Problem Statement and Outline of the Procedure
We have an unknown but fixed MDP M=(S,s_{I},A,P,R) , which we will refer to as the true MDP , an initial prior uMDP \mathcal{M}=(S,s_{I},A,\mathbb{I},\mathcal{P},R) , and a specification \varphi which we want to satisfy.
A discussion of prior (and other parameter) choices follows in Section 6 .

We assume that the underlying graph of the true MDP is known.
In particular: transitions that do not exist in the true MDP (transitions of probability 0 ) do also not exist in the uMDP, transitions of probability 1 in the true MDP are assigned the point interval [1,1] in the uMDP, and any other transition of non-zero probability p has an interval I\in\mathbb{I} in the uMDP.

Under Assumption 1 and Definition 2 , we construct the initial prior uMDP \mathcal{M} to have transitions of probability 0 and 1 exactly where the true MDP M has these too, and interval transitions [\varepsilon,1-\varepsilon] for all other transitions, with \varepsilon>0 free to choose. In particular, our approach does not require \varepsilon to be smaller than the smallest probability p>0 occurring in M , which we also do not assume to be known.
Alternatively, in case further knowledge is available, one may use any other prior uMDP as long as it satisfies Assumption 1 .
Our learning problem is expressed as follows:

In the following we outline our anytime-learning procedure as illustrated in Figure 1 .

Input. We start with an initial prior uMDP \mathcal{M} and a specification \varphi we wish to verify.
We assume (black box) access to the unknown true MDP M to sample trajectories from, or alternatively, assume a (constant) stream of observations from the true MDP.

Robust policy computation. We compute a robust policy \pi for the pessimistic extension of \varphi , i.e. \varphi_{P} , in the uncertain MDP \mathcal{M} , together with the worst-case performance of the specification in the current uMDP.
If the specification contains an explicit threshold, the value can be compared against the threshold for automatic termination.
In case of specifications that optimize a probability or reward, termination needs to be done manually, as it is impossible to tell if the maximum (minimum) was achieved.

Anytime learning. If the result from step 2 is unsatisfactory, we start learning:

Exploration. We sample one or more trajectories from the true MDP M , using the optimism in the face of uncertainty principle.

Update. We update the intervals of the uMDP \mathcal{M} in accordance with the newly collected data.
This update yields a new uMDP that more faithfully captures all collected data up to this point than the previous uMDP.

Repeat. We start again at step 2 with this new uMDP until (manual) termination.

Output. The process may be stopped at any moment and yields the latest uMDP \mathcal{M} together with robust policy \pi and the performance of \pi on \mathcal{M} .

The effects of this procedure are illustrated in Figure 2 .
In 2(a) , we see an example MDP M to learn, and 2(b) shows the assumed knowledge about M . 2(c) shows the initial uMDP \mathcal{M} constructed from 2(b) , using a (symbolic or explicit) lower bound \varepsilon>0 to ensure that all lower bounds of \mathcal{M} are strictly greater than zero.
In 2(d) , we see an intermediate learned uMDP.
Some intervals may already have successfully converged towards the probability of that transition in the true MDP, while others may be very inaccurate due to a low sample size and thus a bad estimate.
Finally, 2(e) depicts the learned uMDP converging towards the true MDP.