RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Evaluation metrics.
We consider two metrics to evaluate the four learning methods.

Performance. How does the robust policy computed on the learned model perform on the true MDP?
We evaluate the probability of satisfying the given specification \mathbb{P}^{M}_{\pi}(\lozenge\,T) or expected reward \textrm{R}^{M}_{\pi}(\lozenge\,T) of the robust policy \pi computed after each update of the model.

Performance estimation error. How well do we expect a robust policy to perform on the true MDP based on the performance on the intermediate learned uMDP?
We compute the difference between the performance of the robust policy on the learned uMDP (the worst-case performance) and the performance on the true MDP.
While values closer to zero are preferable, we do not accept methods with positive estimation errors, since this indicates their estimated performance is not a lower (conservative) bound on the actual performance of the policy.
In particular, an estimation error above zero shows the policy is misleading in terms of predicting its performance.

We benchmark our method using several well-known environments: the Chain Problem [Araya-López et al., 2011 ] , Aircraft Collision Avoidance [Kochenderfer, 2015 ] , a slippery Grid World [Derman et al., 2019 ] , a 99-armed Bandit [Lattimore and Szepesvári, 2020 ] , and two version of a Betting Game [Bäuerle and Ott, 2011 ] .
For details on all these environments we refer to Appendix D .
We highlight the Betting Game and Chain Problem environments here, as they will be used to explain some of the key observations we make from our experimental results.

Betting Game. The agent starts with 10 coins and attempts to maximize the number of coins after 6 bets.
When a bet is won, the number of coins placed is doubled; when lost, the number of coins placed is removed.
The agent may bet 0 , 1 , 2 , 5 , or 10 coins.
We consider two versions of the game, one which is favourable to the player, with a win probability of 0.8 , and one that is unfavourable with win probability 0.2 .
After 6 bets the player receives a reward equal to the number of coins left.
The specification is to maximize the reward.

Chain Problem. We consider a chain of 30 states.
There are three actions, one progresses with probability 0.95 to the next state, and resets the model to the initial state with probability 0.05 .
The second action does the same, but with reversed probabilities.
The third action has probability 0.5 for both cases.
Every action gets a reward of 1 .
As specification, we minimize the reward to reach the last state of the chain.
Results
We present an excerpt of our experimental results here, and refer the reader to Appendix F for the full set of results, which in particular also includes the estimation error for all environments, an additional model error metric, and further experiments regarding different priors and changing environments.
All experiments were performed on a machine with a 4GHz Intel Core i9 CPU, using a single core.
Each experiment is repeated 100 times, and reported with a 95\% confidence interval.

Figure 3 shows the performance of the robust policies computed via each learning method against the number of trajectories processed on the different environments.
We first note that in all environments, our LUI method is the first to find an optimal policy.
Depending on the environment, the performance of LUI and PAC may be roughly equivalent (Chain, Grid).
When also taking the estimation error into account (Figure 4 ), we see LUI outperforming other methods on that metric.
UCRL2 is the slowest to converge to an optimal policy.
This due to UCRL2 being a reinforcement learning algorithm, and thus it is slower in reducing the intervals in favour of a broader exploration.
Recovering from bad estimates.
On the Chain environment, we see LUI and PAC (around trajectory 5 ), and UCRL2 (around trajectory 10^{3} ) choose the wrong action(s), with an decrease in performance as a result.
This is most likely due to getting a bad estimate on some of the transitions later on in the chain, leading to many resets to the initial state, and thus decreasing the performance significantly.
While all three methods manage to recover and then find the optimal policy, UCRL2 takes significantly longer: only after trajectory 10^{5} ,
where LUI and PAC only need about 100 trajectories.
MAP-estimation typically sits between LUI and PAC in terms of performance.
It is less sensitive to mistakes like the one discussed above, but is less reliable in providing a conservative bound on its performance, as will be discussed below.
Furthermore, we see that in the unfavourable Betting Game, only MAP-estimation gives sub-optimal performance, due to bad estimates.
It is able to recover from this, but needs almost 10^{4} trajectories to do so.
Due to the low win probability in this Betting Game, a robust policy on the uMDPs is by default an optimal policy for the true MDP, and we see that LUI, PAC, and UCRL2 do not change to a sub-optimal policy.
Robust policies are conservative.
Consider Figure 4 .
We note the undesirable behaviour of having an Estimation Error above zero, which means the performance of the policy on the learned model was higher than the performance of that policy on the true MDP.
MAP-estimation is particularly susceptible to this, while all three uMDP methods yield policies that are conservative in general, though some exceptions exist as shown in the full results in Appendix F .
LUI is adaptive to changing environments.
Finally, we investigate the behaviour of the learning methods when after a fixed number of trajectories the probabilities of the true MDP change, as introduced in Section 5.4 .
Figure 5 shows the performance of the robust policy for each learning method on the Chain environment, results for the same experiment on the Betting Game can be found in Appendix F .
After {\dagger}\in\{10^{2},\dots,10^{5}\} trajectories, we change the environment by swapping the transition probabilities, for three different bounds on the strength intervals.
We similarly bound the MAP-estimation priors for MAP and UCRL.
PAC is omitted from this experiment as PAC guarantees lose all meaning when changing the underlying distribution.
After the change in environment, the new optimal policy has to use the opposite action from the previously optimal policy.
We see that LUI is the only method capable of converging to optimal policies both before and after the change in environment.
Furthermore, the lower the bounds on the prior strength, the faster it adapts to the change.
We conclude that LUI is an effective approach to deal with learning an MDP under potential adversarial sampling conditions.
Conclusion and Future Work
We presented a new Bayesian method that learns uMDPs to approximate an MDP, either via linearly updating intervals, or PAC-intervals.
Robust policies computed on learned uMDPs are shown to be conservative and reliable in predicting their performance when applied on the MDP that is being learned.
The approach of linearly updating intervals is also effective at continuously learning a potentially changing environment.
For future work, we aim to extend our method to uncertain POMDPs [Suilen et al., 2020 , Cubuktepe et al., 2021 ] .
While we do not see any immediate negative societal impacts of our work, we acknowledge that potential misuse of our work cannot be ruled out due to the generality of MDPs.
Chain problem.
We consider a larger version of the chain problem Araya-López et al. [ 2011 ] with 30 -states.
There are three actions, one progresses with probability 0.95 to the next state, and resets the model to the initial state with probability 0.05 .
The second action does the same, but with reversed probabilities.
The third action has probability 0.5 for both progressing and resetting.
Every action gets a reward of 1 .
As specification, we minimize the reward to reach the last state of the chain: \text{R}_{\text{Min}}(\lozenge\,T) .
This can be seen as an instance of the stochastic shortest path problem [Bertsekas, 2005 ] .
Aircraft collision avoidance.
We model a small, simplified instance of the aircraft collision avoidance problem Kochenderfer [ 2015 ] .
Two aircraft, one controlled, one adversarial, approach each other.
The controlled aircraft can increase, decrease, or stay at the current altitude with a success probability of 0.8 .
The adversarial aircraft can do the same, but does so with probabilities 0.3 , 0.3 , 0.4 respectively.
To goal is to maximize the probability of the two aircraft passing each other without a collision, expressed by the temporal logic specification \mathbb{P}_{\text{Max}}(\neg\text{collision}\mbox{$\,{\sf U}\,$}\text{passed}) .
Grid world.
We consider a slippery 10\times 10 grid world as in Derman et al. [ 2019 ] , where a robot starts in the north-west corner and has to navigate towards a target position.
The robot can move in each of the four cardinal directions with a success probability of 0.55 , and a probability of 0.15 to move in each of the other directions.
Throughout the grid, the robot has to avoid traps.
The model has 100 states and 1450 transitions.
We consider safety specifications that maximize the probability for reaching the north-east (NE) corner without getting trapped, i.e. \mathbb{P}_{\text{Max}}(\neg\text{trapped}\mbox{$\,{\sf U}\,$}\text{NE}) .
Bandit.
We consider a 99 -armed bandit Lattimore and Szepesvári [ 2020 ] , where each arm (action) has an increased probability of success, 0.01,0.02,\dots,0.99 respectively. The goal is to find the action that has the highest probability of success: \mathbb{P}_{\text{Max}}(\lozenge\,\text{success}) .
Betting game.
We consider a betting game from [Bäuerle and Ott, 2011 ] , in which an agent starts with 10 coins and attempts to maximize the number of coins after 6 bets.
When a bet is won, the amount of coins placed is doubled, when lost the amount of coins placed is removed.
The agent may place bets of 0 , 1 , 2 , 5 , and 10 coins.
We consider two versions of the game, one which is FAVOURABLE to the player, with a win probability of 0.8 , and one that is unFAVOURABLE to the player, with a win probability of 0.2 .
After six bets the player receives a state-based reward that equals the number of coins left.
This yields an MDP of 300 states and 1502 transitions.
The specification is to maximize the reward after for reaching a terminal state after these six bets.
Figure 1 : Procedure outline.; Procedure outline.
(a) The true MDP.; (b) Assumption 1 .; (c) Initial prior.; (d) Intermediate uMDP.; (e) Converging towards the true MDP.; Process flow on an example MDP.
Figure 3 : Comparison of the performance of robust policies on different environments against the number of trajectories processed (on log-scale). The dashed line indicates the optimal performance.; Comparison of the performance of robust policies on different environments against the number of trajectories processed (on log-scale). The dashed line indicates the optimal performance.
Figure 4 : Estimation error on the two Betting Game environments and the Bandit, against the number of trajectories processed (on log-scale).; Estimation error on the two Betting Game environments and the Bandit, against the number of trajectories processed (on log-scale).
Figure 5 : Environment change on the Chain Problem at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=0.8 in the exploration.; Environment change on the Chain Problem at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=0.8 in the exploration.
Figure 6 : Prior strength evaluation on the Grid environment.; Prior strength evaluation on the Grid environment.
Figure 7 : Performance, model error, and estimation error for all four learning methods on all six environments.; Performance, model error, and estimation error for all four learning methods on all six environments.
Figure 8 : Environment change on the Chain environment at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=0.8 in the exploration.; Environment change on the Chain environment at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=0.8 in the exploration.
Figure 9 : Environment change on the Chain environment at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=1.0 in the exploration.; Environment change on the Chain environment at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=1.0 in the exploration.
Figure 10 : Environment change on the Betting Game environment at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=0.8 in the exploration.; Environment change on the Betting Game environment at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=0.8 in the exploration.
Figure 11 : Environment change on the Betting Game environment at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=1.0 in the exploration.; Environment change on the Betting Game environment at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=1.0 in the exploration.