SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tianwei Ni
Mila, Université de Montréal  
tianwei.ni@mila.quebec
Michel Ma
Mila, Université de Montréal  
michel.ma@mila.quebec
Benjamin Eysenbach
Princeton University  
eysenbach@princeton.edu
Mila, Université de Montréal  
pierre-luc.bacon@mila.quebec
Reinforcement learning (RL) algorithms face two distinct challenges: learning effective representations of past and present observations, and determining how actions influence future returns. Both challenges involve modeling long-term dependencies. The Transformer architecture has been very successful to solve problems that involve long-term dependencies, including in the RL domain. However, the underlying reason for the strong performance of Transformer-based RL methods remains unclear: is it because they learn effective memory, or because they perform effective credit assignment? After introducing formal definitions of memory length and credit assignment length, we design simple configurable tasks to measure these distinct quantities. Our empirical results reveal that Transformers can enhance the memory capability of RL algorithms, scaling up to tasks that require memorizing observations 1500 steps ago. However, Transformers do not improve long-term credit assignment. In summary, our results provide an explanation for the success of Transformers in RL, while also highlighting an important area for future research and benchmark design. Our code is open-sourced<sup>1</sup>.
In recent years, Transformers (Vaswani et al., 2017; Radford et al., 2019) have achieved remarkable success in domains ranging from language modeling to computer vision. Within the RL community, there has been excitement around the idea that large models with attention architectures, such as Transformers, might enable rapid progress on challenging RL tasks. Indeed, prior works have shown that Transformer-based methods can achieve excellent results in offline RL (Chen et al., 2021; Janner et al., 2021; Lee et al., 2022), online RL (Parisotto et al., 2020; Lampinen et al., 2021; Zheng et al., 2022; Melo, 2022; Micheli et al., 2022; Robine et al., 2023), and real-world tasks (Ouyang et al., 2022; Ichter et al., 2022) (see Li et al. (2023); Agarwal et al. (2023) for the recent surveys).
However, the underlying reasons why Transformers achieve excellent results in the RL setting remain a mystery. Is it because they learn better representations of sequences, akin to their success in computer vision and NLP tasks? Alternatively, might they be internally implementing a learned algorithm, one that performs better credit assignment than previously known RL algorithms? At the core of these questions is the fact that RL, especially in partially observable tasks, requires two distinct forms of temporal reasoning: (working) memory and (temporal) credit assignment. Memory refers to the ability to recall a distant past event at the current time (Blankenship, 1938; Dempster, 1981),
1https://github.com/twni2016/Memory-RL
while credit assignment is the ability to determine when the actions that deserve current credit occurred (Sutton, 1984). These two concepts are also loosely intertwined – learning what bits of history to remember depends on future rewards, and learning to assign current rewards to previous actions necessitates some form of (episodic) memory (Zilli and Hasselmo, 2008; Gershman and Daw, 2017).
Inspired by the distinct nature of memory and credit assignment in temporal dependencies, we aim to understand which of these two concepts Transformers address in the context of RL. Despite empirical success in prior works, directly answering the question is challenging due to two issues. First, many of the benchmarks used in prior works, such as Atari (Bellemare et al., 2013) and MuJoCo (Todorov et al., 2012), require minimal memory and short-term credit assignment, because they closely resemble MDPs and their rewards are mostly immediate. Other benchmarks, such as Key-to-Door (Hung et al., 2018; Mesnard et al., 2021), entangle medium-term memory and credit assignment. Second, there is a lack of rigorous definition of memory and credit assignment in RL (Osband et al., 2020). The absence of quantifiable measures creates ambiguity in the concept of "long-term" often found in prior works.
In this paper, we address both issues concerning memory and credit assignment to better understand Transformers in RL. First, we provide a mathematical definition of memory lengths and credit assignment lengths in RL, grounded in common understanding. We distinguish the memory lengths in reward, transition, policy, and value, which relate to the minimal lengths of recent histories needed to preserve the corresponding quantities. Crucially, our main theoretical result reveals that the memory length of an optimal policy can be upper bounded by the reward and transition memory lengths. For credit assignment, we adopt the forward view and define its length as the minimal number of future steps required for a greedy action to start outperforming any non-greedy counterpart.
The definitions provided are not only quantitative in theory but also actionable for practitioners, allowing them to analyze the memory and credit assignment requirements of many existing tasks. Equipped with these tools, we find that many tasks designed for evaluating memory also evaluate credit assignment, and vice versa, as demonstrated in Table 1. As a solution, we introduce toy examples called Passive and Active T-Mazes that decouple memory and credit assignment. These toy examples are configurable, enabling us to perform a scalable unit-test of a sole capability, in line with best practice (Osband et al., 2020).
Lastly, we evaluate memory-based RL algorithms with LSTMs or Transformers on our configurable toy examples and other challenging tasks to address our research question. We find that Transformers can indeed boost long-term memory in RL, scaling up to memory lengths of 1500. However, Transformers do not improve long-term credit assignment, and struggle with data complexity in even short-term dependency tasks. While these results suggest that practitioners might benefit from using Transformer-based architectures in RL, they also highlight the importance of continued research on core RL algorithms. Transformers have yet to replace RL algorithm designers.
2 Measuring Temporal Dependencies in RL
MDPs and POMDPs. In a partially observable Markov decision process (POMDP)  $\mathcal{M}_O = (\mathcal{O},\mathcal{A},P,R,\gamma,T)^2$ , an agent receives an observation  $o_t \in \mathcal{O}$  at step  $t \in \{1,\dots,T\}$ , takes an action  $a_t \in \mathcal{A}$  based on the observed history  $h_{1:t} := (o_{1:t},a_{1:t-1}) \in \mathcal{H}_t^3$ , and receives a reward  $r_t \sim R_t(h_{1:t},a_t)$  and the next observation  $o_{t+1} \sim P(\cdot \mid h_{1:t},a_t)$ . The initial observation  $h_{1:1} := o_1$  follows  $P(o_1)$ . The total horizon is  $T \in \mathbb{N}^+ \cup \{+\infty\}$  and the discount factor is  $\gamma \in [0,1]$  (less than 1 for infinite horizon). In a Markov decision process (MDP)  $\mathcal{M}_S = (\mathcal{S},\mathcal{A},P,R,\gamma,T)$ , the observation  $o_t$  and history  $h_{1:t}$  are replaced by the state  $s_t \in S$ . In the most generic case, the agent is composed of a policy  $\pi(a_t \mid h_{1:t})$  and a value function  $Q^\pi(h_{1:t},a_t)$ . The optimal value function  $Q^*(h_{1:t},a_t)$  satisfies  $Q^*(h_{1:t},a_t) = \mathbb{E}[r_t \mid h_{1:t},a_t] + \gamma \mathbb{E}_{o_{t+1} \sim P(|h_{1:t},a_t|)}[\max_{a_{t+1}} Q^*(h_{1:t+1},a_{t+1})]$  and induces a deterministic optimal policy  $\pi^*(h_{1:t}) = \operatorname{argmax}_{a_t} Q^*(h_{1:t},a_t)$ . Let  $\Pi_\mathcal{M}^*$  be the space of all optimal policies for a POMDP  $\mathcal{M}$ .
2The classic notion of POMDPs (Kaelbling et al., 1998) includes a state space, but we omit it because we assume that it is unknown.
For any  $k \in \{1, \ldots, t\}$ , let  $o_{t-k:t}$  denote the sequence  $(o_{t-k}, \ldots, o_t)$ , and similarly for  $a_{t-k:t}$ . In particular, let  $h_{t:t} = o_t$  to align with the MDP setting, and let  $h_{t':t} = \emptyset$  for  $t' > t$  and  $h_{t':t} = h_{1:t}$  for  $t' < 1$ .
Below, we provide definitions for context-based policy and value function, as well as the sum of  $n$ -step rewards given a policy. These concepts will be used in defining memory and credit assignment lengths.
Context-based policy and value function. A context-based policy  $\pi$  takes the recent  $l_{\mathrm{ctx}}(\pi)$  observations and actions as inputs, expressed as  $\pi (a_t\mid h_{t - l_{\mathrm{ctx}}(\pi) + 1:t})$ . Here,  $l_{\mathrm{ctx}}(\pi)\in \mathbb{N}$  represents the policy context length. For simplicity, we refer to context-based policies as "policies". Let  $\Pi_k$  be the space of all policies with a context length of  $k$ . Markovian policies form  $\Pi_1$ , while  $\Pi_T$ , the largest policy space, contains an optimal policy. Recurrent policies, recursively taking one input at a time, belong to  $\Pi_{\infty}$ ; while Transformer-based policies have a limited context length. Context-based value function  $Q_{n}^{\pi}$  of a POMDP  $\mathcal{M}$  is the expected return conditioned on the past  $n$  observations and actions, denoted as  $Q_{n}^{\pi}(h_{t - n + 1:t},a_{t}) = \mathbb{E}_{\pi ,\mathcal{M}}\big[\sum_{i = t}\gamma^{i - t}r_{i}\mid h_{t - n + 1:t},a_{t}\big]$ . By definition,  $Q^{\pi}(h_{1:t},a_{t}) = Q_{T}^{\pi}(h_{1:t},a_{t}),\forall h_{1:t},a_{t}$ .
The sum of  $n$ -step rewards. Given a POMDP  $\mathcal{M}$  and a policy  $\pi$ , the expectation of the discounted sum of  $n$ -step rewards is denoted as  $G_{n}^{\pi}(h_{1:t},a_{t}) = \mathbb{E}_{\pi,\mathcal{M}}\left[\sum_{i=t}^{t+n-1}\gamma^{i-t}r_{i}\mid h_{1:t},a_{t}\right]$ . By definition,  $G_{1}^{\pi}(h_{1:t},a_{t}) = \mathbb{E}[r_{t}\mid h_{1:t},a_{t}]$  (immediate reward), and  $G_{T-t+1}^{\pi}(h_{1:t},a_{t}) = Q^{\pi}(h_{1:t},a_{t})$  (value).
2.1 Memory Lengths
In this subsection, we introduce the memory lengths in the common components of RL, including the reward, transition, policy, and value functions. Then, we will show the theoretical result of the relation between these lengths.
The reward (or transition) memory length of a POMDP quantifies the temporal dependency distance between current expected reward (or next observation distribution) and the most distant observation.
Definition 1.A (Reward memory length  $m_{\text{reward}}^{\mathcal{M}}$ ). For a POMDP  $\mathcal{M}$ ,  $m_{\text{reward}}^{\mathcal{M}}$  is the smallest  $n \in \mathbb{N}$  such that the expected reward conditioned on recent  $n$  observations is the same as the one conditioned on full history, i.e.,  $\mathbb{E}[r_t \mid h_{1:t}, a_t] = \mathbb{E}[r_t \mid h_{t-n+1:t}, a_t], \forall t, h_{1:t}, a_t$ .
Definition 1.B (Transition memory length  $m_{\mathrm{transit}}^{\mathcal{M}}$ ). For a POMDP  $\mathcal{M}$ ,  $m_{\mathrm{transit}}^{\mathcal{M}}$  is the smallest  $n \in \mathbb{N}$  such that next observations are conditionally independent of the rest history given the past  $n$  observations and actions, i.e.,  $o_{t+1} \perp h_{1:t}, a_t \mid h_{t-n+1:t}, a_t$ .
Next, policy memory length is defined based on the intuition that while a policy may take a long history of observations as its context, its action distribution only depends on recent observations. Therefore, policy memory length represents the temporal dependency distance between current action and the most distant observation.
Definition 1.C (Policy memory length  $l_{\mathrm{mem}}(\pi)$ ). The policy memory length  $l_{\mathrm{mem}}(\pi)$  of a policy  $\pi$  is the minimum horizon  $n \in \{0, \dots, l_{\mathrm{ctx}}(\pi)\}$  such that its actions are conditionally independent of the rest history given the past  $n$  observations and actions, i.e.,  $a_t \perp h_{t - l_{\mathrm{ctx}}(\pi) + 1:t} \mid h_{t - n + 1:t}$ .
Lastly, we define the value memory length of a policy, which measures the temporal dependency distance between the current value and the most distant observation.
Definition 1.D (Value memory length of a policy  $l_{\mathrm{value}}^{\mathcal{M}}(\pi)$ ). A policy  $\pi$  has its value memory length  $l_{\mathrm{value}}^{\mathcal{M}}(\pi)$  as the smallest  $n$  such that the context-based value function is equal to the value function, i.e.,  $Q_{n}^{\pi}(h_{t - n + 1:t},a_{t}) = Q^{\pi}(h_{1:t},a_{t}), \forall t,h_{1:t},a_{t}$ .
Now we prove that the policy and value memory lengths of optimal policies can be upper bounded by the reward and transition memory lengths (Proof in Appendix A).
Theorem 1 (Upper bounds of memory lengths for optimal policies). For optimal policies with the shortest policy memory length, their policy memory lengths are upper bounded by their value memory lengths, which are further bounded by the maximum of reward and transition memory lengths.
$$
l _ {\text {m e m}} \left(\pi^ {*}\right) \leq l _ {\text {v a l u e}} ^ {\mathcal {M}} \left(\pi^ {*}\right) \leq \max  \left(m _ {\text {r e w a r d}} ^ {\mathcal {M}}, m _ {\text {t r a n s i t}} ^ {\mathcal {M}}\right) := m ^ {\mathcal {M}}, \quad \forall \pi^ {*} \in \underset {\pi \in \Pi_ {\mathcal {M}} ^ {*}} {\operatorname {a r g m i n}} \left\{l _ {\text {m e m}} (\pi) \right\} \tag {1}
$$
Thm. 1 suggests that the constant of  $m^{\mathcal{M}}$  can serve as a proxy to analyze the policy and value memory lengths required in a task, which are often challenging to directly compute, as we will demonstrate in Sec. 3.4
2.2 Credit Assignment Lengths
Classically, temporal credit assignment is defined with a backward view (Minsky, 1961; Sutton, 1984) – how previous actions contributed to current reward. The backward view is specifically studied through gradient back-propagation in supervised learning (Lee et al., 2015; Ke et al., 2018). In this paper, we consider the forward view in reinforcement learning – how current action will influence future rewards. Specifically, we hypothesize that the credit assignment length  $n$  of a policy is how long the greedy action at the current time-step starts to make a difference to the sum of future  $n$ -step rewards.
Before introducing the formal definition, let us recall the common wisdom on dense-reward versus sparse-reward tasks. Dense reward is generally easier to solve than sparse reward if all other aspects of the tasks are the same. One major reason is that an agent can get immediate reward feedback in dense-reward tasks, while the feedback is delayed or even absent in sparse-reward tasks. In this view, solving sparse reward requires long-term credit assignment. Our definition will reflect this intuition.
Definition 2.A (Credit assignment length of a history given a policy  $c(h_{1:t}; \pi)$ ). Given a policy  $\pi$  in a task, the credit assignment length of a history  $h_{1:t}$  quantifies the minimal temporal distance  $n^5$ , such that a greedy action  $a_t^*$  is strictly better than any non-greedy action  $a_t'$  in terms of their  $n$ -step rewards  $G_n^\pi$ . Formally, let  $A_t^* \coloneqq \operatorname{argmax}_{a_t} Q^\pi(h_{1:t}, a_t)^6$ , define
$$
c \left(h _ {1: t}; \pi\right) := \min  _ {1 \leq n \leq T - t + 1} \left\{n \mid \exists a _ {t} ^ {*} \in A _ {t} ^ {*}, \text {s . t .} G _ {n} ^ {\pi} \left(h _ {1: t}, a _ {t} ^ {*}\right) > G _ {n} ^ {\pi} \left(h _ {1: t}, a _ {t} ^ {\prime}\right), \forall a _ {t} ^ {\prime} \notin A _ {t} ^ {*} \right\} \tag {2}
$$
Definition 2.B (Credit assignment length of a policy  $c(\pi)$  and of a task  $c^{\mathcal{M}}$ ). Credit assignment length of a policy  $\pi$  represents the worst case of credit assignment lengths of all visited histories. Formally, let  $d_{\pi}$  be the history occupancy distribution of  $\pi$ , define  $c(\pi) := \max_{h_{1:t} \in \mathrm{supp}(d_{\pi})} c(h_{1:t}; \pi)$ .
Further, we define the credit assignment length of a task  $\mathcal{M}$ , denoted as  $c^{\mathcal{M}}$ , as the minimal credit assignment lengths over all optimal policies<sup>7</sup>, i.e.,  $c^{\mathcal{M}} \coloneqq \min_{\pi^{*} \in \Pi_{\mathcal{M}}^{*}} c(\pi^{*})$ .
As an illustrative example, consider the extreme case of a sparse-reward MDP where the reward is only provided at the terminal time step  $T$ . In this environment, the credit assignment length of any policy  $\pi$  is  $c(\pi) = T$ . This is because the credit assignment length of the initial state  $s_1$  is always  $c(s_1; \pi) = T$ , as the performance gain of any greedy action at  $s_1$  is only noticeable after  $T$  steps.
Our definition of credit assignment lengths has ties with the temporal difference (TD) learning algorithm (Sutton, 1988). With the tabular TD(0) algorithm, the value update is given by  $V(s_{t}) \gets V(s_{t}) + \alpha (r_{t} + \gamma V(s_{t+1}) - V(s_{t}))$ . This formula back-propagates the information of current reward  $r_{t}$  and next value  $V(s_{t+1})$  to current value  $V(s_{t})$ . When we apply the TD(0) update backwards in time for  $n$  steps along a trajectory, we essentially back-propagate the information of  $n$ -step rewards to the current value. Thus, the credit assignment length of a state measures the number of TD(0) updates we need to take a greedy action for that state.
3 Environment Analysis of Memory and Credit Assignment Lengths
By employing the upper bound  $m^{\mathcal{M}}$  on memory lengths (Thm. 1) and credit assignment length  $c^{\mathcal{M}}$  (Def. 2.B), we can perform a quantitative analysis of various environments. These environments include both abstract problems and concrete benchmarks. In this section, we aim to address two questions: (1) Do prior environments actually require long-term memory or credit assignment? (2) Can we disentangle memory from credit assignment?
${}^{4}$  Thm. 1 also applies to MDPs,where it provides a quick proof of the well-known fact that optimal policies and values of MDPs can be Markovian:  ${m}_{\text{reward }}^{\mathcal{M}},{m}_{\text{transit }}^{\mathcal{M}}$  are both at most 1,so the policy and value memory lengths are also at most 1 .
The  $n$  is guaranteed to exist because a greedy action must be the best in  $T - t + 1$ -step rewards (i.e.  $Q$ -value).
<sup>6</sup>If  $A_{t}^{*} = \mathcal{A}$  , i.e. all actions are greedy, then the set of  $n$  contains all integers between 1 and  $T - t + 1$
If there are multiple optimal policies,  $c(\pi^{*})$  may be not unique. See Appendix A for an example.
Table 1: Estimated memory and credit assignment lengths required in prior tasks, as defined in Sec. 2. The top block shows tasks designed for memory, while the bottom one shows tasks designed for credit assignment. All tasks have a horizon of  $T$  and a discount factor of  $\gamma \in (0,1)$ . The  $T$  column shows the largest value used in prior work. The terms "long" and "short" are relative to horizon  $T$ . We manually estimate these lengths from task definition and mark tasks purely evaluating long-term memory or credit assignment in black.
Task M	T	l_mem(π*)	mM	cM
Memory	Reacher-pomdp (Yang and Nguyen, 2021)	50	long	long	short
Memory Cards (Esslinger et al., 2022)	50	long	long	1
TMaze Long (Noise) (Beck et al., 2019)	100	T	T	1
Memory Length (Osband et al., 2020)	100	T	T	1
Mortar Mayhem (Pleines et al., 2023)	135	long	long	≤ 25
Autoencode (Morad et al., 2023)	311	T	T	1
Numpad (Parisotto et al., 2020)	500	long	long	short
PsychLab (Fortunato et al., 2019)	600	T	T	short
Passive Visual Match (Hung et al., 2018)	600	T	T	≤ 18
Repeat First (Morad et al., 2023)	831	2	T	1
Ballet (Lampinen et al., 2021)	1024	≥ 464	T	short
Passive Visual Match (Sec. 5.1; Our experiment)	1030	T	T	≤ 18
172 MiniGrid-Memory (Chevalier-Boisvert et al., 2018)	1445	≤ 51	T	≤ 51
Passive T-Maze (Eg. 1; Ours)	1500	T	T	1
152 Memory Maze (Pasukonis et al., 2022)	4000	long	long	≤ 225
HeavenHell (Esslinger et al., 2022)	20	T	T	T
T-Maze (Bakker, 2001)	70	T	T	T
Goal Navigation (Fortunato et al., 2019)	120	T	T	T
T-Maze (Lambrechts et al., 2021)	200	T	T	T
Spot the Difference (Fortunato et al., 2019)	240	T	T	T
PyBullet-P benchmark (Ni et al., 2022)	1000	2	2	short
PyBullet-V benchmark (Ni et al., 2022)	1000	short	short	short
Credit Assignment	Umbrella Length (Osband et al., 2020)	100	1	1	T
Push-r-bump (Yang and Nguyen, 2021)	50	long	long	long
Key-to-Door (Raposo et al., 2021)	90	short	T	T
Delayed Catch (Raposo et al., 2021)	280	1	T	T
Active T-Maze (Eg. 2; Ours)	500	T	T	T
Key-to-Door (Sec. 5.2; Our experiment)	530	short	T	T
Active Visual Match (Hung et al., 2018)	600	T	T	T
Episodic MuJoCo (Ren et al., 2022)	1000	1	T	T
3.1 How Long Do Prior Tasks Require Memory and Credit Assignment?
To study memory or credit assignment in RL, prior works propose abstract problems and evaluate their algorithms in benchmarks. A prerequisite for evaluating agent capabilities in memory or credit assignment is understanding the requirements for them in the tested tasks. Therefore, we collect representative problems and benchmarks with a similar analysis in Sec. 3.2, to figure out the quantities  $m^{\mathcal{M}}$  related to memory lengths and  $c^{\mathcal{M}}$  related to credit assignment length.
Regarding abstract problems, we summarize the analysis in Appendix B. For example, decomposable episodic reward assumes the reward is only given at the terminal step and can be decomposed into the sum of Markovian rewards. This applies to episodic MuJoCo (Ren et al., 2022) and Catch with delayed rewards (Raposo et al., 2021). We show that it has policy memory length  $l_{\mathrm{mem}}(\pi^{*})$  at most 1 while requiring  $m_{\mathrm{reward}}^{\mathcal{M}}$  and  $c^{\mathcal{M}}$  of  $T$ , indicating that it is really challenging in both memory and credit assignment. Delayed environments are known as difficult to assign credit (Sutton, 1984). We show that delayed reward (Arjona-Medina et al., 2019) or execution (Derman et al., 2021) or observation (Katsikopoulos and Engelbrecht, 2003) with  $n$  steps all require  $m_{\mathrm{reward}}^{\mathcal{M}}$  of  $n$ , while only delayed rewards and execution may be challenging in credit assignment.
As for concrete benchmarks, Table 1 summarizes our analysis, with detailed descriptions of these environments in Appendix B. First, some tasks designed for long-term memory (Beck et al., 2019; Osband et al., 2020; Hung et al., 2018; Yang and Nguyen, 2021; Esslinger et al., 2022) or credit
assignment (Osband et al., 2020) indeed solely evaluate the respective capability. Although some memory tasks such as Numpad (Parisotto et al., 2020) and Memory Maze (Pasukonis et al., 2022) do require long-term memory with short-term credit assignment, the exact memory and credit assignment lengths are hard to quantify due to the complexity. In contrast, our Passive T-Maze introduced in Sec. 3.2, provides a clear understanding of the memory and credit assignment lengths, allowing us to successfully scale the memory length up to 1500.
Conversely, other works (Bakker, 2001; Lambrechts et al., 2021; Esslinger et al., 2022; Hung et al., 2018; Raposo et al., 2021; Ren et al., 2022) actually entangle memory and credit assignment together, despite their intention to evaluate one of the two. Lastly, some POMDPs (Ni et al., 2022) only require short-term memory and credit assignment, which cannot be used for any capability evaluation.
3.2 Disentangling Memory from Credit Assignment
We can demonstrate that memory and credit assignment are independent concepts by providing two examples, all with a finite horizon of  $T$  and no discount. The examples have distinct pairs of  $(\max (r_{\mathcal{M}},m_{\mathrm{transit}}^{\mathcal{M}}),c^{\mathcal{M}}):(T,1),(T,T)$  respectively.
We adapt T-Maze (Fig. 1), a POMDP environment rooted in neuroscience (O'Keefe and Dostrovsky, 1971) and AI (Bakker, 2001; Osband et al., 2020), to allow different credit assignment lengths. Passive and active concepts are borrowed from Hung et al. (2018).
Figure 1: T-Maze task. In the passive version, the agent starts at the oracle state ("S" and "O" are the same state), while the active version requires that the agent navigate left from the initial state to collect information from the oracle state.