SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Roberto Cipollone
Sapienza University of Rome  
cipollone@diag.uniroma1.it
Anders Jonsson
Universitat Pompeu Fabra anders.jonsson@upf.edu
Alessandro Ronca
University of Oxford  
alessandro.ronca@cs.ox.ac.uk
Mohammad Sadegh Talebi
University of Copenhagen m.shahi@di.ku.dk
This paper deals with offline (or batch) Reinforcement Learning (RL) in episodic Regular Decision Processes (RDPs). RDPs are the subclass of Non-Markov Decision Processes where the dependency on the history of past events can be captured by a finite-state automaton. We consider a setting where the automaton that underlies the RDP is unknown, and a learner strives to learn a near-optimal policy using pre-collected data, in the form of non-Markov sequences of observations, without further exploration. We present RegORL, an algorithm that suitably combines automata learning techniques and state-of-the-art algorithms for offline RL in MDPs. RegORL has a modular design allowing one to use any off-the-shelf offline RL algorithm in MDPs. We report a non-asymptotic high-probability sample complexity bound for RegORL to yield an  $\varepsilon$ -optimal policy, which makes appear a notion of concentrability relevant for RDPs. Furthermore, we present a sample complexity lower bound for offline RL in RDPs. To our best knowledge, this is the first work presenting a provably efficient algorithm for offline learning in RDPs.
Most reinforcement learning (RL) algorithms hinge on the Markovian assumption, i.e. that the underlying system transitions and rewards are Markovian in some natural notion of (observable) state, and hence, the distribution of future observations depends only on the current state-action of the system. This fundamental assumption allows one to model decision making using the powerful framework of Markov Decision Processes (MDPs) [1]. However, there are many application scenarios where rewards are issued according to temporal conditions over histories (or trajectories), and others where the environment itself evolves in a history-dependent manner. As a result, Markovian approaches may prove unsuitable for modeling such situations. These scenarios can be appropriately modeled as Non-Markovian Decision Processes (NMDPs) [2, 3].
NMDPs describe environments where the distribution on the next observation and reward is a function of the history. In these environments, behaving optimally may also require to take histories into account. For example, a robot may receive a reward for delivering an item only if the item was previously requested, and a self-driving car is more likely to skid and lose control if it previously rained. Also, consider a mobile robot that has to track an object which may disappear from its field of view. The object is likely to be found again in the same place where it was seen last time. This requires the agent to remember, hence to act according to information in its interaction history. In general, an NMDP can show an arbitrary dependency on the history or trace, preventing efficient learning. Consequently, recent research has focused on tractable sub-classes of NMDPs. In Regular
Decision Processes (RDPs) [3], the next observation and reward distributions depend on regular properties of the history, which can be captured by a deterministic finite-state automaton. This determines the existence of a finite state space where states are determined by histories, and where the Markov property is regained.
In this paper, we investigate offline RL in episodic RDPs, where the goal is to find a near-optimal policy using a pre-collected dataset, with minimal possible size, generated by a fixed behavior policy (and without further exploration). Offline RL in MDPs has received extensive attention recently, and provably sample efficient algorithms have been proposed for various settings. Despite the extensive and rich literature on MDPs, comparatively little work exists on offline RL in NMDPs. The scarcity of results may likely be attributed to the difficult nature of the problem rather than the lack of interest.
Partially-Observable Markov Decision Processes (POMDPs) [4] are also NMDPs, and RDPs can be seen as the subclass of POMDPs that enjoy the property of having hidden states determined by the history of observations. This is a key property that allows one to take advantage of a set of planning and learning techniques that do not apply to arbitrary POMDPs. Planning in POMDPs is computationally intractable [5], and two common approaches to solve (and learn) them rely on maintaining either a belief state or a finite history of observations. Maintaining and updating a belief state is worst-case exponential in the size of the original observation space, while the latter approach yields a space whose size is exponential in the history length. State-of-the-art work on offline RL in POMDPs considers restricted classes of POMDPs such as undercomplete POMDPs (e.g., [6, 7]), which cannot be used to model all RDP instances. General POMDPs are only considered under assumptions such as the possibility of reaching every belief state in a few steps [8] or ergodicity [9]. While existing offline RL algorithms for solving POMDPs cannot guarantee provable learning in a generic RDP, the structural properties of RDPs indicate that they can be solved more efficiently using techniques that are carefully tailored to their structure. Exploiting the structure in RDPs is thus key in designing provably sample-efficient learning algorithms.
1.1 Summary of Contributions
We formalize offline RL in RDPs (Section 2), and establish a first, to the best of our knowledge, sample complexity lower bound thereof (Section 5). We introduce an algorithm, called Reg0RL, that learns  $\varepsilon$ -optimal policies for any RDP, in the episodic setting. At the core of Reg0RL, there is a component called ADACT-H, which is a variant of ADACT [10], carefully tailored to episodic RDPs. ADACT-H learns a minimal automaton that underlies the unknown RDP without prior knowledge. The output automaton is further used to derive a Markov abstraction of data to be used by any off-the-shelf algorithm for offline RL in episodic MDPs. We present a sample-complexity bound for ADACT-H to return a minimal underlying automaton with high probability. This bound substantially improves the existing bound for the original ADACT, and can be of independent interest. In view of the modular design of Reg0RL, the total sample complexity is controlled by twice that of ADACT-H (Theorem 6) and that for the incorporated off-the-shelf algorithm. We also present another variant of ADACT-H, called ADACT-H-A. In contrast to ADACT-H that learns a complete RDP, ADACT-H-A only reconstructs a subset of states that are likely under the behavior policy, in relation to an input accuracy parameter. As such, ADACT-H-A renders more aligned with the practice of RL than ADACT-H. Furthermore, we provide a first lower-bound for offline RL in RDPs that involves relevant parameters for the problem, such as the RDP single-policy concentrability, which extends an analogous notion for MDPs from the literature. Finally, if contrasted to both online learning in RDPs and automata learning, our results suggest possible improvements in sample complexity results for both areas.
Offline RL in MDPs. There is a rich and growing literature on offline RL, and provably sample efficient algorithms have been proposed for various settings of MDPs; see, e.g., [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. For example, in the case of episodic MDPs, it is established that the optimal sample size in offline RL depends on the size of state-space, episode length, as well as some notion of concentrability, reflecting the distribution mismatch between the behavior and optimal policies. A closely related problem is off-policy learning; see, e.g., [21, 22, 23] and the recent survey [24].
Online RL in RDPs. Several algorithms for online RL in RDPs exist [25, 26, 27] but complexity bounds are only given in [26] for the infinite-horizon discounted setting. The sample complexity bounds in [26] are not immediately comparable to ours, due to the different setting. Importantly, the algorithm in [26] uses the uniform policy for learning, and it therefore might be adapted to our setting only under the assumption that the behaviour policy is uniform. Even in this case, our bounds show an improved dependency on several key quantities. Furthermore, we provide a sample complexity lower bound, whereas their results are limited to showing that a dependency on the quantities occurring in their upper bounds is necessary.
The online RL algorithms in [28, 29, 30, 31] have been developed for formalisms that are closely related to RDPs, and such algorithms can be applied to RDPs. However, these algorithms are not proven to be sample efficient.
POMDPs. Every RDP can be seen as a POMDP whose hidden dynamics evolves according to its finite-state automaton. However, RL in POMDPs is a largely open problem. Even for a known POMDP, computing a near-optimal policy is PSPACE-complete [5]. For unknown dynamics, which is the setting considered here, favourable bounds have been obtained for the class of undercomplete POMDPs [6, 7], which does not include all RDPs, or alternatively, under other assumptions such as few-step reachability [8] or ergodicity [9]. This relationship between RDPs and POMDPs can be also seen from the notion of state. In fact, the automaton state of an RDP is an instance of information state, as defined in [32], and of belief, as in classic POMDP literature [33].
**PSRs.** Predictive State Representations (PSRs) [34, 35, 36, 37] are general descriptions of dynamical systems that capture POMDPs and hence RDPs. There exist polynomial PAC bounds for online RL in PSRs [38]. Nonetheless, these bounds are looser than the one we show here, since they must necessarily consider a wider class of models. Moreover, although a minimum core set for PSRs is similar to a minimal RDP, the bounds feature a number of quantities that are specific to PSRs (e.g., regularity parameter) and do not immediately apply to RDPs.
Other Non-Markovian Settings. Feature MDPs and state representations both share the idea of having a map from histories to a state space. This is analogous to the map determined by the transition function of the automaton underlying an RDP. Algorithmic solutions for feature MDPs are based on suffix trees, and they cannot yield optimal performance in our setting [29, 30]. The automaton of an RDP can be seen as providing one kind of state representation [39, 40, 41, 42]. The existing bounds for state representations show a linear dependency on the number of candidate representations, which is exponential in the number of states in our case. A similar dependency is also observed in [43]. RL with non-Markovian rewards is considered in [44, 45, 46, 47, 48, 49]. The idea of a map from histories to states is also found in [32]. Non-Markovianity is also introduced by logical specifications that the agent is required to satisfy [50, 51, 52, 53, 54]; however, it is resolved a priori from the known specification. The convergence properties of Q-learning over a (known) underlying state space such as the one of an RDP are studied in [55].
Learning PDFA. Our algorithms for learning an RDP borrow and improve over techniques for learning Probabilistic-Deterministic Finite Automata (PDFA) [56, 57, 58, 10, 59, 60]. Our algorithm builds upon the state-of-the-art algorithm ADACT [10], and we derive bounds that are a substantial improvement over the ones that would be obtained from a straightforward application of any existing PDFA-learning algorithm to the offline RL setting.
We provide additional literature review in Appendix A.
Notations. Given a set  $\mathcal{V}$ ,  $\Delta(\mathcal{V})$  denotes the set of probability distributions over  $\mathcal{V}$ . For a function  $f: \mathcal{X} \to \Delta(\mathcal{V})$ ,  $f(x, y)$  is the probability of  $y$  given  $x$ . Further, we write  $y \sim f(x)$  to abbreviate  $y \sim f(x, \cdot)$ . For  $y \in \mathcal{V}$ , we use  $\mathbb{1}_y \in \Delta(\mathcal{V})$  to denote the Kronecker delta defined as  $\mathbb{1}_y(y) = 1$  and  $\mathbb{1}_y(y') = 0$  for each  $y' \in \mathcal{V}$  such that  $y' \neq y$ . Given an event  $E$ ,  $\mathbb{I}(E)$  denotes the indicator function of  $E$ , which equals 1 if  $E$  is true, and 0 otherwise, e.g.,  $\mathbb{1}_y(y') = \mathbb{I}(y = y')$ . For any integer  $Z \geq 0$ , we let  $[Z] := \{0, \dots, Z\}$ . Given a set  $\mathcal{X}$ , for  $k \in \mathbb{N}$ ,  $\mathcal{X}^k$  represents the set of sequences of length  $k$  whose elements are from  $\mathcal{X}$ . Also,  $\mathcal{X}^* = \cup_{k=0}^{\infty} \mathcal{X}^k$ . The notation  $\widetilde{\mathcal{O}}(\cdot)$  hides poly-logarithmic terms.
2.1 Episodic Regular Decision Processes
We first introduce generic episodic decision processes. An episodic decision process is a tuple  $\mathcal{P} = \langle \mathcal{O},\mathcal{A},\mathcal{R},\bar{T},\bar{R},H\rangle$ , where  $\mathcal{O}$  is a finite set of observations,  $\mathcal{A}$  is a finite set of actions,  $\mathcal{R}\subset [0,1]$  is a finite set of rewards and  $H\geq 1$  is a finite horizon. As is common in automata theory, we use sequences  $a_{m}r_{m}o_{m}\dots a_{n}r_{n}o_{n}$  to denote traces of actions, rewards and observations, and concatenation  $\mathcal{ARO} = \{aro:a\in \mathcal{A},r\in \mathcal{R},o\in \mathcal{O}\}$  to denote sets of sequences. Let  $\mathcal{E}_t = (\mathcal{ARO})^{t + 1}$  be the set of traces of length  $t + 1$ , and let  $e_{m:n}\in \mathcal{E}_{n - m}$  denote a trace from time  $m$  to time  $n$ , included. A trajectory  $e_{0:T}$  is the full trace generated until time  $T$ . We assume that a trajectory  $e_{0:T}$  can be partitioned into episodes  $e_{\ell :\ell +H}\in \mathcal{E}_H$  of length  $H + 1$ , and that the dynamics at time  $T = k(H + 1) + t$ ,  $t\in [H]$ , are conditionally independent of the previous episodes and all rewards, i.e. the dynamics only depend on  $a_{k(H + 1)}o_{k(H + 1)}\dots a_{T}o_{T}$ . For  $t\in [H]$ , let  $\mathcal{H}_t = (\mathcal{AO})^{t + 1}$  denote the relevant part of the trajectory for decision making, and let  $\mathcal{H} = \cup_{t = 0}^{H}\mathcal{H}_t$ . We refer to elements in  $\mathcal{H}$  as histories, even though they are not complete trajectories. In each episode  $e_{0:H}$ ,  $a_0 = a_\perp$  is a dummy action used to initialize the distribution on  $\mathcal{H}_0$ . The transition function  $\bar{T}:\mathcal{H}\times \mathcal{A}\to \Delta (\mathcal{O})$  and the reward function  $\bar{R}:\mathcal{H}\times \mathcal{A}\to \Delta (\mathcal{R})$  only depend on the history of the current episode. Given  $\mathcal{P}$ , a generic policy is a function  $\pi :(\mathcal{AO})^{*}\to \Delta (\mathcal{A})$  that maps trajectories to distributions over actions. The value function  $V^{\pi}:[H]\times \mathcal{H}\to \mathbb{R}$  of a policy  $\pi$  is a mapping that assigns real values to histories. For  $h\in \mathcal{H}$ , it is defined as  $V^{\pi}(H,h)\coloneqq 0$  and
$$
V ^ {\pi} (t, h) := \mathbb {E} \left[ \sum_ {i = t + 1} ^ {H} r _ {i} \mid h, \pi \right], \forall t <   H, \forall h \in \mathcal {H} _ {t}. \tag {1}
$$
For brevity, we write  $V_{t}^{\pi}(h) \coloneqq V^{\pi}(t,h)$ . The optimal value function  $V^{*}$  is defined as  $V_{t}^{*}(h) \coloneqq \sup_{\pi} V_{t}^{\pi}(h), \forall t \in [H], \forall h \in \mathcal{H}_{t}$ , where  $\sup$  is taken over all policies  $\pi : (\mathcal{A}\mathcal{O})^{*} \to \Delta(\mathcal{A})$ . Any policy achieving  $V^{*}$  is called optimal, which we denote by  $\pi^{*}$ ; namely  $V^{\pi^{*}} = V^{*}$ . Solving  $\mathcal{P}$  amounts to finding  $\pi^{*}$ . In what follows we consider simpler policies of the form  $\pi : \mathcal{H} \to \Delta(\mathcal{A})$  mapping histories to distributions over actions. Let  $\Pi_{\mathcal{H}}$  denote the set of such policies. It can be shown that  $\Pi_{\mathcal{H}}$  always contains an optimal policy, i.e.  $V_{t}^{*}(h) \coloneqq \max_{\pi \in \Pi_{\mathcal{H}}} V_{t}^{\pi}(h), \forall t \in [H], \forall h \in \mathcal{H}_{t}$ . An episodic MDP is an episodic decision process whose dynamics at each timestep  $t$  only depends on the last observation and action [1].
Episodic RDPs. An episodic Regular Decision Process (RDP) [3, 25] is an episodic decision process  $\mathbf{R} = \langle \mathcal{O},\mathcal{A},\mathcal{R},\bar{T},\bar{R},H\rangle$  described by a finite transducer (Moore machine)  $\langle \mathcal{Q},\Sigma ,\Omega ,\tau ,\theta ,q_0\rangle$  , where  $\mathcal{Q}$  is a finite set of states,  $\Sigma = \mathcal{A}\mathcal{O}$  is a finite input alphabet composed of actions and observations,  $\Omega$  is a finite output alphabet,  $\tau :\mathcal{Q}\times \Sigma \to \mathcal{Q}$  is a transition function,  $\theta :\mathcal{Q}\rightarrow \Omega$  is an output function, and  $q_{0}\in \mathcal{Q}$  is a fixed initial state [61, 62, 63]. The output space  $\Omega = \Omega_{\mathrm{o}}\times \Omega_{\mathrm{r}}$  consists of a finite set of functions that compute the conditional probabilities of observations and rewards, meaning  $\Omega_{\mathrm{o}}\subset \mathcal{A}\to \Delta (\mathcal{O})$  and  $\Omega_{\mathrm{r}}\subset \mathcal{A}\to \Delta (\mathcal{R})$  . For simplicity, we use two output functions,  $\theta_{\circ}:\mathcal{Q}\times \mathcal{A}\to \Delta (\mathcal{O})$  and  $\theta_{\mathrm{r}}:\mathcal{Q}\times \mathcal{A}\to \Delta (\mathcal{R})$  , to denote the individual conditional probabilities. Let  $\tau^{-1}$  denote the inverse of  $\tau$  , i.e.  $\tau^{-1}(q)\subseteq \mathcal{Q}\times \mathcal{A}\mathcal{O}$  is the subset of state-symbol pairs that map to  $q\in \mathcal{Q}$  . In this context, an input symbol is an element of AOr. We use  $A,R,O,Q$  to denote the cardinality of  $A,\mathcal{R},\mathcal{O},\mathcal{Q}$  , respectively, and assume  $A\geq 2$
An RDP  $\mathbf{R}$  implicitly represents a function  $\bar{\tau}:\mathcal{H}\to \mathcal{Q}$  from histories in  $\mathcal{H}$  to states in  $\mathcal{Q}$ , recursively defined as  $\bar{\tau}(h_0)\coloneqq \tau(q_0,a_0o_0)$  and  $\bar{\tau}(h_t)\coloneqq \tau(\bar{\tau}(h_{t-1}),a_ot_t)$ . The dynamics and of  $\mathbf{R}$  are defined as  $\bar{T}(h,a,o) = \theta_{\mathrm{o}}(\bar{\tau}(h),a,o)$  and  $\bar{R}(h,a,r) = \theta_{\mathrm{r}}(\bar{\tau}(h),a,r)$ ,  $\forall h\in \mathcal{H},\forall aro\in \mathcal{ARO}$ . Episodic RDPs are acyclic, i.e. the states can be partitioned as  $\mathcal{Q} = \mathcal{Q}_0\cup \dots \cup \mathcal{Q}_{H+1}$ , where each  $\mathcal{Q}_{t+1}$  is the set of states generated by histories in  $\mathcal{H}_t$  for each  $t\in [H]$ . An RDP is minimal if its Moore machine is minimal. Since there is nothing to predict at time  $H+1$ , a minimal RDP contains a single state  $q_{H+1}$  in  $\mathcal{Q}_{H+1}$ . To ensure that an acyclic RDP  $\mathbf{R}$  is minimal, we introduce a designated termination observation  $o_\perp$  in  $\mathcal{O}$  and define  $\tau(q_{H+1},ao_\perp) = q_{H+1}$  and  $\theta_{\mathrm{o}}(q_{H+1},a) = \mathbb{1}_{o_\perp}$  for each  $a\in \mathcal{A}$ . Hence,  $q_{H+1}$  is absorbing and the states in  $\mathcal{Q}$  implicitly count how many steps are left until we observe  $o_\perp$ . Without  $o_\perp$ , a Moore machine could potentially represent all episodes using fewer than  $H+2$  states.
Since the conditional probabilities of observations and rewards are fully determined by the current state-action pair  $(q, a)$ , an RDP  $\mathbf{R}$  adheres to the Markov property over its states, but not over the observations. Given a state  $q_{t} \in \mathcal{Q}$  and an action  $a_{t} \in \mathcal{A}$ , the probability of the next transition is
$$
\mathbb {P} \left(r _ {t}, o _ {t}, q _ {t + 1} \mid q _ {t}, a _ {t}, \mathbf {R}\right) = \theta_ {\mathrm {r}} \left(q _ {t}, a _ {t}, r _ {t}\right) \theta_ {\mathrm {o}} \left(q _ {t}, a _ {t}, o _ {t}\right) \mathbb {I} \left(q _ {t + 1} = \tau \left(q _ {t}, a _ {t} o _ {t}\right)\right).
$$
(a)Cookie domain.
(b) Agent's view.
Figure 1: The cookie domain: The agent can only see what is in the current room [28].
Evidently, in the special case where an RDP is Markovian in both observations and rewards, it reduces to an episodic MDP. More precisely, any episodic MDP with actions  $\mathcal{A}$ , states  $\mathcal{O}$  and horizon  $H$  can be represented by some episodic RDP with states  $\mathcal{Q} \subseteq \mathcal{O} \times [H + 1]$  and inputs  $\mathcal{AO}$ .
An important class of policies for RDPs are the regular policies. Given an RDP  $\mathbf{R}$ , a policy  $\pi : \mathcal{H} \to \Delta(\mathcal{A})$  is called regular if  $\pi(h_1) = \pi(h_2)$  whenever  $\bar{\tau}(h_1) = \bar{\tau}(h_2)$ , for all  $h_1, h_2 \in \mathcal{H}$ . Let  $\Pi_{\mathbf{R}}$  denote the set of regular policies for  $\mathbf{R}$ . Regular policies exhibit powerful properties. First, under a regular policy, suffixes have the same probability of being generated for histories that map to the same RDP state. Second, there exists at least one optimal policy that is regular.
Proposition 1. Consider an RDP  $\mathbf{R}$ , a regular policy  $\pi \in \Pi_{\mathbf{R}}$  and two histories  $h_1$  and  $h_2$  in  $\mathcal{H}_t$ ,  $t \in [H]$ , such that  $\bar{\tau}(h_1) = \bar{\tau}(h_2)$ . For each suffix  $e_{t+1:H} \in \mathcal{E}_{H-t-1}$ , the probability of generating  $e_{t+1:H}$  is the same for  $h_1$  and  $h_2$ , i.e.  $\mathbb{P}(e_{t+1:H} \mid h_1, \pi, \mathbf{R}) = \mathbb{P}(e_{t+1:H} \mid h_2, \pi, \mathbf{R})$ .
Proposition 2. Each RDP  $\mathbf{R}$  has at least one optimal policy  $\pi^{*} \in \Pi_{\mathbf{R}}$ .
Due to Proposition 2, when solving an RDP  $\mathbf{R}$ , we can restrict our search to the set of regular policies  $\Pi_{\mathbf{R}}$ . A regular policy can be compactly defined as  $\pi : \mathcal{Q} \to \Delta(\mathcal{A})$ , where  $\pi(q_0) = \mathbb{1}_{a_\perp}$  always selects the dummy action  $a_\perp$ , and its value function as  $V^\pi : [H] \times \mathcal{Q} \to \mathbb{R}$ .
Next, we define occupancy measures for RDPs. Given a regular policy  $\pi : \mathcal{Q} \to \Delta(\mathcal{A})$  and  $t \in [H]$ , let  $d_t^\pi \in \Delta(\mathcal{Q}_t \times \mathcal{AO})$  be the induced probability distribution over states in  $\mathcal{Q}_t$  and input symbols in  $\mathcal{AO}$ , recursively defined as  $d_0^\pi(q_0, a_0 o_0) := \theta_\circ(q_0, a_0, o_0)$  and
$$
d _ {t} ^ {\pi} \left(q _ {t}, a _ {t} o _ {t}\right) := \sum_ {\left(q, a o\right) \in \tau^ {- 1} \left(q _ {t}\right)} d _ {t - 1} ^ {\pi} \left(q, a o\right) \pi \left(q _ {t}, a _ {t}\right) \theta_ {\mathrm {o}} \left(q _ {t}, a _ {t}, o _ {t}\right) \quad t > 0.
$$
We also overload the notation by writing  $d_t^\pi(q_t, a_t) = \sum_{o \in \mathcal{O}} d^\pi(q_t, a_t o)$ . Of particular interest is the occupancy distribution  $d_t^* := d_t^{\pi^*}$ , associated with an optimal policy  $\pi^*$ .
Example 1 (The cookie domain [28]). The cookie domain (Figure 1a) has three rooms connected by a hallway. The agent (purple triangle) can move in the four cardinal directions. When pressing a button in the orange room, a cookie randomly appears in either the green or the blue room. The agent receives a reward of  $+1$  for eating the cookie and it may then press the button again. This domain is partially observable since the agent can only see what is in the room that it currently occupies (Figure 1b). The cookie domain can be modelled as an episodic RDP with states  $\mathcal{Q} = [H + 1] \times \mathcal{O} \times \mathcal{U}$ , with  $\mathcal{U} = \{u_1, u_2, u_3, u_4\}$ . The value of  $\mathcal{U}$  is  $u_1$  when the button has not been pressed yet (or not pressed since the last cookie was eaten). The value is  $u_2$  when the button has been pressed, but the agent has not visited the blue or green room yet. In this state, the environment has a  $50\%$  probability of generating the observation of a cookie when the agent enters either room for the first time. If the agent visits the green room and finds no cookie, the value becomes  $u_3$ , meaning that the cookie is in the blue room. The meaning of  $u_4$  is dual to that of  $u_3$ .
2.2 Offline RL in RDPs