SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which, after taking exponents, justifies the equivalence to (11). Given this equivalence and Lemma 3.1, it is clear that (11) has a unique solution. To be more explicit, suppose for a contradiction that (11) does not have exactly one solution (up to  $\|\cdot\|$ -equivalence of solutions). Then by the equivalence between the two Bellman equations, (9) would either have no solutions or more than one solution which contradicts the existence and uniqueness guaranteed by Lemma 3.1. Finally, (12) follows from the form of the maximiser at each Bellman step.
For brevity, we will fix a  $b \in \mathcal{B}$  and drop it from our notation. Also write  $\pi = \pi(\cdot|b)$  and  $\pi_a = \pi(a|b)$ . The Lagrangian for the constrained problem is
$$
\mathcal {L} (\pi , \lambda) = \sum_ {a, o} P (o \mid a) \pi_ {a} \log \left[ \frac {P (o \mid a) \pi_ {a}}{U ^ {*} (a , o)} \right] + \lambda \left(\sum_ {a} \pi_ {a} - 1\right). \tag {37}
$$
We require, in addition, that the minimiser  $\pi^{*}$  (which exists due to the Weierstrass extreme value theorem) is such that  $\pi_{a}^{*} \geq 0$  for each  $a \in \mathcal{A}$ . The first order necessary conditions give
$$
\pi_ {a} = e ^ {- (1 + \lambda)} \exp [ - \Pi (a) ] \quad \forall a \in \mathcal {A} \tag {38}
$$
and the constraint equation gives
$$
1 = \sum_ {a} \pi_ {a} = e ^ {- (1 + \lambda)} \sum_ {a} \exp [ - \Pi (a) ]. \tag {39}
$$
Hence the only candidate for the minimiser is  $\pi^{*}$  such that
$$
\pi_ {a} ^ {*} = \frac {\exp [ - \Pi (a) ]}{\sum_ {\hat {a} \in \mathcal {A}} \exp [ - \Pi (\hat {a}) ]} \quad \forall a \in \mathcal {A}. \tag {40}
$$
The Hessian of  $\mathcal{L}$  is positive definite for any  $\lambda$  and  $\pi \in \Delta (\mathcal{A})$ , so we conclude that  $\pi^{*}$  is a minimiser. Finally, that  $\pi_{a}^{*}\geq 0$  for every  $a\in \mathcal{A}$  is clear from (40).
Proof. Fix an  $\hat{a} \in \mathcal{A}$  and  $b \in \mathcal{B}$ . If we set
$$
\pi^ {\hat {a}} (a \mid b) := \left\{ \begin{array}{l l} 1, & a = \hat {a} \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {41}
$$
$$
U ^ {\hat {a}} (a, o \mid b) := P (o \mid \hat {a}, b) \pi^ {\hat {a}} (a \mid b) \in \mathcal {U} \tag {42}
$$
then the constraint (20) is satisfied trivially. Straightforward computations show that the constraint (21) is satisfied by  $U^{\hat{a}}(a,o|b)$  with  $(\rho, \bar{U})$  as defined in (22) and (23).
A.7 Details of Experimental Scenarios
A.7.1 Details of Navigation1 Scenario
The robot can move in the four cardinal directions with 0.1 probability of actuator failure. If the realised movement leads to a collision with an obstacle or the edge of the map, no movement occurs and the robot remains in its current position. If the robot's true state is a landmark, the robot receives a position reading uniformly in the  $9 \times 9$  grid around the robot's true state. Outside the landmarks, the robot receives no observation. The robot receives a penalty of -100 for entering a danger zone and a reward of +300 for entering a goal state. In both cases, the problem terminates. Every other state incurs a reward of -1. The discount parameter was 0.99. The robot's initial belief was equally distributed between two initial positions that were uniformly sampled from the southern-most row of the map.
A.7.2 Details of Navigation2 Scenario
Similar to Navigation1, the robot's action space consists of moves anywhere in the four cardinal directions NORTH, SOUTH, EAST, WEST. To simulate noise in the robot's actuator's, actions fail with 0.1 probability, and if this occurs, the robot moves randomly in a direction orthogonal to the one specified. If the realised movement leads to a collision with an obstacle or the edge of the
A.6 Algorithm REFSOLVER
map, no movement occurs and the robot remains in its current position. If the robot's true state is a landmark, the robot receives a position reading uniformly in the  $9 \times 9$  grid around the robot's true state. Otherwise, the robot receives no observation. The robot receives a reward of +600 for being in a goal state, and -3 for being in any other state. The discount parameter was  $\gamma = 0.99$ . The robot's initial belief was equally distributed between two initial positions that were uniformly sampled from the southern-most row of the map.
A.7.3 Details of Navigation3 Scenario
The state and observation spaces have six dimensions which represent the position  $(x,y,z)$  and orientation (roll, pitch, yaw) of the drone. There are 12 actions that increase or decrease one dimension by 0.2 units. Action and observation noise is sampled from a multivariate Gaussian distribution with 0 mean and 0.032 standard deviation. If an action results in a collision with the terrain, the robot receives a -500 penalty, and the world terminates. Reaching the goal area receives a 1000 reward, and each step incurs a -1 penalty. The drone's initial configuration is  $(-4.0, -1.0, 0.0, 0.0, 0.0, 0.0)$  and the goal are the ball with centre  $(10.4, 3.0, 0.45)$  and radius 0.25. The start and goal are not in a direct line and the robot needs to go over the hill near the goal area to reach its destination. The discount parameter was  $\gamma = 0.99$ . The robot's initial belief is populated with its exact start position.
A.8 Source Code
We include the source code for REFSOLVER, which is developed on top of pomdp.py as a Supplementary Material. https://github.com/RDLLab/ref_pomdp_neurips23