RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
Sequential decision-making in realistic scenarios is inherently subject to uncertainty, commonly captured via probabilities. Markov decision processes (MDPs) are the standard model to reason about such decision-making problems [Puterman, 1994 , Bertsekas, 2005 ] .
Safety-critical scenarios require assessments of correctness which can, for instance, be described by temporal logic [Pnueli, 1977 ] or expected reward specifications.
A fundamental requirement for providing such correctness guarantees on MDPs is that probabilities are precisely given.
Methods such as variants of model-based reinforcement learning [Moerland et al., 2020 ] or PAC-learning [Strehl et al., 2009 ] can learn MDPs by deriving point estimates of probabilities from data to satisfy this requirement.
This derivation naturally carries the risk of statistical errors.
Optimal policies are highly sensitive to small perturbations in transition probabilities, leading to sub-optimal outcomes such as a deterioration in performance [Mannor et al., 2007 , Goyal and Grand-Clement, 2020 ] .

Uncertain MDPs (uMDPs; also known as interval or robust MDPs) extend MDPs to incorporate such statistical errors by introducing an additional layer of uncertainty via uncertainty sets on the transition function [Nilim and Ghaoui, 2005 , Wiesemann et al., 2013 , Goyal and Grand-Clement, 2020 , Rigter et al., 2021 ] .
The solution of a uMDP is a robust policy that allows an adversarial selection (i.e., the worst-case scenario) of probabilities within the uncertainty set, and induces a worst-case performance (a conservative bound on, e.g., the reachability probability or expected reward).
The problem of computing such robust policies, also called robust verification , is solved using value iteration or convex optimization, where the uncertainty sets are convex [Wolff et al., 2012 , Puggelli et al., 2013 ] .
Our approach.
We study the problem of learning an MDP from data.
We propose an iterative learning method which uses uMDPs as intermediate models and is able to adapt to new data which may be inconsistent with prior assumptions.
Furthermore, the method is task-aware in the sense that the learning procedure respects temporal logic specifications.
In particular, our method learns intervals of probabilities for individual transitions.
This Bayesian anytime-learning approach employs intervals with linearly updating conjugate priors [Walter and Augustin, 2009 ] , and can iteratively improve upon a uMDP that approximates the true MDP we wish to learn.
This method not only decreases the size of each interval, but may also increase it again in case of a so-called prior-data conflict where new data suggests the actual probability lies outside the current interval.
Consequently, a newly learned interval does not need to be a subset of its prior interval.
This property makes our method especially suitable to learn MDPs where the transition probabilities change.
Alternatively, we also include probably approximately correct (PAC) intervals via Hoeffding’s inequality [Hoeffding, 1963 ] , which introduces a correctness guarantee for each transition.

We summarize the key features of our learning method, and what sets it apart from other methods.

An anytime approach. The ability to iteratively update intervals that are not necessarily subsets of each other allows us to design an anytime-learning approach .
At any time, we may stop the learning and compute a robust policy for the uMDP that the process has yielded thus far, together with the worst-case performance of this policy against a given specification.
This performance may not be satisfactory, e.g., the worst-case probability to reach a set of critical states may be below a certain threshold.
We continue learning towards a new uMDP that more faithfully captures the true MDP due to the inclusion of further data.
Thereby, we ensure that the robust policy gradually gets closer to the optimal policy for the true MDP.

Specification-driven. Our method features the possibility to learn in a task-aware fashion, that is, to learn transitions that matter for a given specification.
In particular, for reachability or expected reward (temporal logic) specifications which require a certain set of target states to be reached, we only learn and update transitions along paths towards these states.
Transitions outside those paths do not affect the satisfaction of the specification.

Adaptive to changing environment dynamics. When using linearly updating intervals, our approach is adaptive to changing environment dynamics.
That is, if during the learning process the probability distributions of the underlying MDP change, our method can easily adapt and learns these new distributions.
Preliminaries
A discrete probability distribution over a finite set X is a function \mu\colon X\to[0,1]\subset\mathbb{R} with \sum_{x\in X}\mu(x)=1 .
We write \mathcal{D}(X) for the set of all discrete probability distributions over X , and by |X| we denote the number of elements in X .
For any closed interval I\subseteq\mathbb{R} we write \underline{I} and \overline{I} for the lower and upper bounds of the interval, that is, I=[\underline{I},\overline{I}] .

A Markov decision process (MDP) is a tuple (S,s_{I},A,P,R) with S a finite set of states, s_{I}\in S the initial state, A a finite set of actions, P\colon S\times A\times S\to[0,1] with \forall s,a\in S\times A,\sum_{s^{\prime}}P(s,a,s^{\prime})=1 (such that P(s,a)\in\mathcal{D}(S) ) the probabilistic transition function, and R\colon S\times A\to\mathbb{R}_{\geq 0} the reward (or cost) function.

A trajectory in an MDP is a finite sequence (s_{0},a_{0},s_{1},a_{1},\dots,s_{n})\in(S\times A)^{*}\times S where s_{0}=s_{I} and P(s_{i},a_{i},s_{i+1})>0 for 0\leq i<n .
A memoryless policy is a function \pi\colon S\to\mathcal{D}(A) .
If \pi maps to Dirac distributions, then it is a memoryless deterministic (or pure) policy.
Applying a policy to an MDP M resolves all the non-deterministic choices and yields an (induced) discrete-time Markov chain (DTMC); see, e.g., Puterman [ 1994 ] for details.

An uncertain MDP (uMDP) is a tuple (S,s_{I},A,\mathbb{I},\mathcal{P},R) where S,s_{I},A,R are as for MDPs, \mathbb{I} is a set of probability intervals \mathbb{I}=\{[a,b]\mid 0<a\leq b\leq 1\} , and \mathcal{P}\colon S\times A\times S\to(\mathbb{I}\cup\{0\}) is the uncertain transition function , assigning either a probability interval, or the exact probability 0 to any transition.

Uncertain MDPs can be seen as an uncountable set of MDPs that only differ in their transition functions.
For a transition function P , we write P\in\mathcal{P} if for every transition the probability of P lies within the interval of \mathcal{P} , i.e., P(s,a,s^{\prime})\in\mathcal{P}(s,a,s^{\prime}) for all (s,a,s^{\prime})\in S\times A\times S .
We only allow intervals with a lower bound greater than zero, to ensure a transition cannot vanish under certain distributions generated by the uncertainty.
This assumption is standard in uMDPs [Wiesemann et al., 2013 , Puggelli et al., 2013 ] .
Applying a policy to a uMDP yields an induced interval Markov chain [Jonsson and Larsen, 1991 ] .
Specifications.
We consider reachability or expected reward (cost) specifications.
The value \mathbb{P}^{M}_{\pi}(\lozenge\,T) is the probability to reach a set of target states T\subseteq S in the MDP M under the policy \pi , also referred to as the performance of \pi . 1 1 1 We will omit the policy \pi and the MDP M whenever they are clear from the context. Likewise, \textrm{R}^{M}_{\pi}(\lozenge\,T) describes the expected accumulated reward to reach T under \pi .
Note that for probabilistic specifications, we can replace the formula \lozenge\,T by more general reach-avoid specifications using the until operator from temporal logic.

Formally, the specification \mathbb{P}_{\text{Max}}(\lozenge\,T)=\max_{\pi}\mathbb{P}_{\pi}(\lozenge\,T) expresses that the probability of eventually reaching the target set T\subseteq S should be maximized.
Likewise, the specification \text{R}_{\text{Max}}(\lozenge\,T) requires the expected reward for reaching T to be maximized.
For minimization, we write \mathbb{P}_{\text{Min}}(\lozenge\,T) and \text{R}_{\text{Min}}(\lozenge\,T) , respectively.
Besides optimizing a probability or reward, a specification may also express an explicit user-provided threshold to compare the performance of a policy to.

For uMDPs, we define optimistic and pessimistic specifications.
In optimistic specifications, we assume the best-case scenario of the uncertainty to satisfy the specification by also minimizing (or maximizing) over the uncertainty set, written as \mathbb{P}_{\text{MinMin}}(\lozenge\,T)=\min_{\pi}\min_{P\in\mathcal{P}}\mathbb{P}^{\mathcal{M}[P]}_{\pi}(\lozenge\,T) (or \mathbb{P}_{\text{MaxMax}}(\lozenge\,T) ), where the first Min ( Max ) signals what the decision-maker is trying to achieve, and the second what the uncertainty does.
In pessimistic specifications, the uncertainty does the opposite of the goal: \mathbb{P}_{\text{MaxMin}}(\lozenge\,T) or \mathbb{P}_{\text{MinMax}}(\lozenge\,T) .
The notation is similar for reward specifications.
A (standard) specification \varphi can be extended to be optimistic or pessimistic by adding a second Min or Max .
We write \varphi_{O} for its optimistic extension, and \varphi_{P} for its pessimistic extension.

For an MDP M , the aim is to compute a policy \pi that either optimizes a given specification \varphi , or whose performance respects a given threshold that, e.g., provides an upper bound on the probability of reaching a set of critical states.
Common methods are value iteration or linear programming [Puterman, 1994 , Baier and Katoen, 2008 ] .
For uncertain MDPs \mathcal{M} , the goal is to compute a policy \pi that satisfies an optimistic or pessimistic specification \varphi_{O} or \varphi_{P} .
In the latter case, we call \pi a robust policy .
Optimal policies in uMDPs can be computed via robust dynamic programming or convex optimization [Wolff et al., 2012 , Puggelli et al., 2013 ] .
Bayesian Learning
We assume access to trajectories through the true MDP.
Due to the Markov property of the MDP, i.e., the fact that the transition probabilities only depend on the current state and not on any further history, we may split each trajectory \tau into separate sets of independent experiments where a state-action pair (s,a) is sampled and a successor state s_{i} is observed; see also Appendix A of [Strehl and Littman, 2008 ] .
We count the number of occurrences of the transition (s,a,s_{i}) in each trajectory \tau , and the number of occurrences of the state-action pair (s,a) in each trajectory \tau , denoted by \#(s,a,s_{i}) and \#(s,a) , respectively. In the following, we introduce the two approaches of learning intervals: the standard approach of PAC learning via Hoeffding’s inequality which provides a PAC guarantee on the value of the learned model, and a new approach using linearly updating intervals (LUI), which is more flexible due to the inclusion of prior-data conflicts and its closure properties.
Finally, we introduce a minor modification to the LUI approach, making it also applicable to situations where the underlying MDP changes over time.
Such a modification is not possible in PAC learning as Hoeffding’s inequality assumes independent samples from a fixed distribution.
Learning PAC Intervals
We use the standard method of maximum a-posteriori probability (MAP) estimation to infer point estimates of probabilities;
see Appendix A for details.
These point estimates can then easily be turned into probably approximately correct (PAC) intervals via Hoeffding’s inequality.
Given N=\#(s,a) samples and a fixed error rate \gamma , we use Hoeffding’s inequality [Hoeffding, 1963 ] to compute the interval size \delta_{P}=\sqrt{\nicefrac{{\log(\nicefrac{{2}}{{\gamma_{P}}})}}{{2N}}} .
This \delta_{P} may then be used to construct intervals that are PAC with respect to the true transition probability.
To lift the PAC guarantee to the entire learned MDP, and consequently also the optimal value for some specification, we need to distribute \gamma over all transitions with probabilities strictly between zero and one, i.e., \gamma_{P}=\nicefrac{{\gamma}}{{\sum_{(s,a)\in S\times A}|\mathbf{Succ}_{>1}(s,a)|}} , where \mathbf{Succ}_{>1}(s,a)=\{s^{\prime}\in S\mid 0<P(s,a,s^{\prime})<1\} is the set of successor states of each (s,a) with more than one successor. 2 2 2 The published version of this paper contains an error in the formula for \delta_{P} , which has been corrected here. Using this \delta_{P} , we then construct the intervals

\mathcal{P}(s,a,s_{i})=[\max(\varepsilon,\tilde{P}(s,a,s_{i})-\delta_{P}),\min(\tilde{P}(s,a,s_{i})+\delta_{P},1)], (1)

where \tilde{P} is the (MAP) point estimate, and \varepsilon is again a small value to ensure that the interval lower bounds are non-zero.
As a result, we have the following Proposition, whose proof is a direct application of Hoeffding’s inequality.

The true MDP M^{*} lies within the learned uMDP \mathcal{M} with probability greater or equal to 1-\gamma .

This is a standard result and also used in, e.g., PAC statistical model checking [Ashok et al., 2019 ] .
Learning Linearly Updating Probability Intervals
We use the Bayesian approach of intervals with linearly updating conjugate priors [Walter and Augustin, 2009 ] to learn intervals of probabilities.
Each uncertain transition \mathcal{P}(s,a,s_{i}) is assigned a prior interval \mathcal{P}_{i}=[\underline{\mathcal{P}}_{i},\overline{\mathcal{P}}_{i}] , and a prior strength interval [\underline{n}_{i},\overline{n}_{i}] that represents a minimum and maximum number of samples on which the prior interval is based.
The greater the values of the strength interval, the more emphasis is placed on the prior, and the more data is needed to significantly change the prior when computing the posterior.
The greater the difference between the \underline{n}_{i} and \overline{n}_{i} , the greater the difference between a prior-data conflict and a prior-data agreement .

The interval [\underline{\mathcal{P}}_{i},\overline{\mathcal{P}}_{i}] can be updated to [\underline{\mathcal{P}}_{i}^{\prime},\overline{\mathcal{P}}_{i}^{\prime}] , using N=\#(s,a) and k_{i}=\#(s,a,s_{i}) , as follows:

\underline{\mathcal{P}}_{i}^{\prime}=\begin{cases}\frac{\overline{n}_{i}\underline{\mathcal{P}}_{i}+k_{i}}{\overline{n}_{i}+N}& if \forall j.\frac{k_{j}}{N}\geq\underline{\mathcal{P}}_{j} (prior-data agreement),\[4.30554pt] \frac{\underline{n}_{i}\underline{\mathcal{P}}_{i}+k_{i}}{\underline{n}_{i}+N}& if \exists j.\frac{k_{j}}{N}<\underline{\mathcal{P}}_{j} (prior-data conflict).\end{cases} (2) \overline{\mathcal{P}}_{i}^{\prime}=\begin{cases}\frac{\overline{n}_{i}\overline{\mathcal{P}}_{i}+k_{i}}{\overline{n}_{i}+N}& if \forall j.\frac{k_{j}}{N}\leq\overline{\mathcal{P}}_{j} (prior-data agreement),\[4.30554pt] \frac{\underline{n}_{i}\overline{\mathcal{P}}_{i}+k_{i}}{\underline{n}_{i}+N}& if \exists j.\frac{k_{j}}{N}>\overline{\mathcal{P}}_{j} (prior-data conflict).\end{cases} (3)

The strength interval is updated by adding the number of samples N to it: [\underline{n}_{i}^{\prime},\overline{n}_{i}^{\prime}]=[\underline{n}_{i}+N,\overline{n}_{i}+N] .

The initial values for the priors of each state-action pair can be chosen freely subject to the constraints 0<\underline{\mathcal{P}}_{i}\leq\overline{\mathcal{P}}_{i}\leq 1 , and \overline{n}_{i}\geq\underline{n}_{i}\geq 1 .
Key properties of linearly updating probability intervals.
Convergence in the infinite run. Under the assumption that the true MDP does not change, each interval will converge to the exact transition probability when the total number of samples processed tends to infinity, regardless of how many samples are processed per iteration [Walter and Augustin, 2009 ] .
This assumption is, however, not required for our work.
If the true MDP changes over time, or is adversarial (i.e., a uMDP), our method is still applicable, but will not converge to a fixed MDP.

Prior-data conflict. When the estimated probability \nicefrac{{k_{i}}}{{N}} lies outside the current interval, a so-called prior data conflict occurs.
Consequently, if at some point we derive an interval that does not contain the true transition probability, the method will correct itself later on.

Closure properties under updating. Finally, updating is closed in two specific ways.
First, any interval of probabilities is updated again to a valid interval of probabilities, and second, any set of intervals at a state-action pair that contains a valid distribution over the successor states will again contain a distribution over successor states after updating.

A key requirement for computing robust policies on uMDPs is that the lower bound of every interval is strictly greater than zero (see Definition 2 ).
This closure property is formalized as follows.

For any valid prior interval [\underline{\mathcal{P}},\overline{\mathcal{P}}] with 0<\underline{\mathcal{P}}\leq\overline{\mathcal{P}}\leq 1 , we have that the posterior [\underline{\mathcal{P}}^{\prime},\overline{\mathcal{P}}^{\prime}] computed via Definition 3 also satisfies 0<\underline{\mathcal{P}}^{\prime}\leq\overline{\mathcal{P}}^{\prime}\leq 1 .

Furthermore, we also have closure properties at each state-action pair.
In particular, if we choose our prior intervals such that there is at least one valid probability distribution at the state-action pair, then the posterior intervals will again contain a valid probability distribution.
We formalize this notion by examining the sum of the lower and upper bounds of the intervals in the following Theorem:

We have the following bounds on the set of posterior distributions.
In case of a prior data agreement, we have that the sum of the posterior bounds is bounded by the sum of the prior bounds, and the value 1 . That is,

\sum_{i}\underline{\mathcal{P}}_{i}\leq\sum_{i}\underline{\mathcal{P}}_{i}^{\prime}\leq 1,\qquad 1\leq\sum_{i}\overline{\mathcal{P}}_{i}^{\prime}\leq\sum_{i}\overline{\mathcal{P}}_{i}

In case of a prior-data conflict, the sum of posterior bounds is no longer necessarily bounded by the prior, and we only have

\sum_{i}\underline{\mathcal{P}}_{i}^{\prime}\leq 1\leq\sum_{i}\overline{\mathcal{P}}_{i}^{\prime}

Note, however, that this last constraint ( 5 ) is already sufficient to ensure that there is a valid probability distribution at the state-action pair.

The proofs for both Theorem 1 and 2 can be found in Appendix B .
Efficient exploration
Above, we assumed that a set of trajectories was given.
To actually obtain the trajectories, we use the well-established optimism in the face of uncertainty principle [Munos, 2014 ] .
We compute the optimal policy for the optimistic extension (see Section 3 ) of the specification \varphi , i.e. \varphi_{O} , in the current uMDP and use this policy for exploration.
To make exploration specification-driven, we only sample transitions along trajectories towards the target state(s) of the specification.
When the last seen state has probability zero or one to reach the target for reachability, or reward zero or infinity, we restart.
States satisfying these conditions can be found by analyzing the graph of the true MDP [Baier and Katoen, 2008 ] .
Trajectories and iterations.
As explained in Section 4 , our method is iterative in terms of updating the uMDP \mathcal{M} and computing a robust policy.
After updating the uMDP, we also compute a new exploration policy, based on the new uMDP.
Each iteration consists of processing at least one, but possibly more trajectories.
To determine how many trajectories to collect, we use a doubling-counting scheme, where we keep track of how often every state-action pair and transition is visited during exploration [Jin et al., 2020 ] .
An iteration is completed when any of the counters is doubled with respect to the previous iteration.
A detailed description of this schedule is given in Appendix C .
Learning under changing environments
Previously, we assumed a fixed unknown true MDP M^{*} to learn.
But what if the transition probabilities of M^{*} suddenly change?
More precisely, we assume two unknown true MDPs, M_{1}^{*} and M_{2}^{*} with the same underlying graph but (possibly) different transition probabilities.
After an unknown number of interactions with the true environment M_{1}^{*} , we suddenly continue interacting with M_{2}^{*} .
We modify our LUI approach by introducing a bound on the strength of the prior, n_{\text{Max}}=[\underline{n}_{\text{Max}},\overline{n}_{\text{Max}}] , and update the strength intervals by the following rule

[\underline{n}_{i}^{\prime},\overline{n}_{i}^{\prime}]=[\min(\underline{n}_{i}+N,\underline{n}_{Max}),\min(\overline{n}_{i}+N,\overline{n}_{Max})].

The probability intervals themselves are updated in the same way as before in Definition 3 .
By limiting the prior strength in this way, we trade some rate of convergence for adaptability.
The weaker the prior, the greater the effect of a prior-data conflict, and hence adaptability to new data.
When suddenly changing environments, new data will likely lead to prior-data conflicts, and thus a higher adaptability of the overall learning method.

We add randomization to the pure optimistic exploration policy.
We introduce a hyperparameter \xi\in[0,1] , and follow with probability \xi the action of the optimistic policy, and distribute the remaining 1-\xi uniformly over the other actions, yielding a memoryless randomized policy.
Experimental Evaluation
We implement our approach, with both linearly updating intervals (LUI) and PAC intervals (PAC), in Java on top of the verification tool PRISM [Kwiatkowska et al., 2011 ] , together with a variant of value iteration to compute robust policies for uMDPs with convex uncertainties [Wolff et al., 2012 ] . 3 3 3 The implementation is available at https://github.com/LAVA-LAB/luiaard . We compare our method to point estimates derived via MAP-estimation (MAP) and with uMDPs learned by the UCRL2 reinforcement learning algorithm [Jaksch et al., 2010 ] (UCRL2).
We modify UCRL2 to make it more comparable to our setting.
In particular, we use optimistic policies for exploration, but robust policies to compute the performance, in contrast to the standard UCRL2 setting which only uses optimistic policies, see Appendix E for further details.

Without knowledge about the true MDP apart from Assumption 1 , we have to define an appropriate prior interval for every transition.
We set \varepsilon=\texttt{1e-4} as constant and define the prior uMDP with intervals \mathcal{P}_{i}=[\varepsilon,1-\varepsilon] and strength intervals [\underline{n}_{i},\overline{n}_{i}]=[5,10] at every transition \mathcal{P}(s,a,s_{i}) , as in Figure 2(c) .
For MAP, we use a prior of \alpha_{i}=10 for all i .
The same prior is used for the point estimates of both PAC and UCRL2, together with an error rate of \gamma=0.01 .