SlowGuess/NeurIPS_2023_OCR huggingface dataset:
David Abel
dmabel@google.com
Andre Barreto
andrebarreto@google.com
Benjamin Van Roy
benvanroy@google.com
Doina Precup
doinap@google.com
Hado van Hasselt
hado@google.com
Satinder Singh
baveja@google.com
In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward. However, this perspective is based on a restricted view of learning as finding a solution, rather than treating learning as endless adaptation. In contrast, continual reinforcement learning refers to the setting in which the best agents never stop learning. Despite the importance of continual reinforcement learning, the community lacks a simple definition of the problem that highlights its commitments and makes its primary concepts precise and clear. To this end, this paper is dedicated to carefully defining the continual reinforcement learning problem. We formalize the notion of agents that "never stop learning" through a new mathematical language for analyzing and cataloging agents. Using this new language, we define a continual learning agent as one that can be understood as carrying out an implicit search process indefinitely, and continual reinforcement learning as the setting in which the best agents are all continual learning agents. We provide two motivating examples, illustrating that traditional views of multi-task reinforcement learning and continual supervised learning are special cases of our definition. Collectively, these definitions and perspectives formalize many intuitive concepts at the heart of learning, and open new research pathways surrounding continual learning agents.
In *The Challenge of Reinforcement Learning*, Sutton states: "Part of the appeal of reinforcement learning is that it is in a sense the whole AI problem in a microcosm" [56]. Indeed, the problem facing an agent that learns to make better decisions from experience is at the heart of the study of Artificial Intelligence (AI). Yet, when we study the reinforcement learning (RL) problem, it is typical to restrict our focus in a number of ways. For instance, we often suppose that a complete description of the state of the environment is available to the agent, or that the interaction stream is subdivided into episodes. Beyond these standard restrictions, however, there is another significant assumption that constrains the usual framing of RL: We tend to concentrate on agents that learn to solve problems, rather than agents that learn forever. For example, consider an agent learning to play Go: Once the agent has discovered how to master the game, the task is complete, and the agent's learning can stop. This view of learning is often embedded in the standard formulation of RL, in which an agent interacts with a Markovian environment with the goal of efficiently identifying an optimal policy, at which point learning can cease.
But what if this is not the best way to model the RL problem? That is, instead of viewing learning as finding a solution, we can instead think of it as endless adaptation. This suggests study of the continual reinforcement learning (CRL) problem [47, 48, 25, 27], as first explored in the thesis by
Ring [46], with close ties to supervised never-ending [10, 39, 43] and continual learning [47, 48, 26, 54, 41, 42, 49, 22, 30, 45, 4].
Despite the prominence of CRL, the community lacks a clean, general definition of this problem. It is critical to develop such a definition to promote research on CRL from a clear conceptual foundation, and to guide us in understanding and designing continual learning agents. To these ends, this paper is dedicated to carefully defining the CRL problem. Our definition is summarized as follows:
The CRL Problem (Informal)
An RL problem is an instance of CRL if the best agents never stop learning.
The core of our definition is framed around two new insights that formalize the notion of "agents that never stop learning": (i) we can understand every agent as implicitly searching over a set of history-based policies (Theorem 3.1), and (ii) every agent will either continue this search forever, or eventually stop (Remark 3.2). We make these two insights rigorous through a pair of logical operators on agents that we call generates and reaches that provide a new mathematical language for characterizing agents. Using these tools, we then define CRL as any RL problem in which all of the best agents never stop their implicit search. We provide two motivating examples of CRL, illustrating that traditional multi-task RL and continual supervised learning are special cases of our definition. We further identify necessary properties of CRL (Theorem 4.1) and the new operators (Theorem 4.2, Theorem 4.3). Collectively, these definitions and insights formalize many intuitive concepts at the heart of continual learning, and open new research pathways surrounding continual learning agents.
We first introduce key concepts and notation. Our conventions are inspired by Ring [46], the recent work by Dong et al. [16] and Lu et al. [32], as well as the literature on general RL by Hutter [23, 24], Lattimore [28], Leike [29], Cohen et al. [12], and Majeed [36].
Notation. We let capital calligraphic letters denote sets  $(\mathcal{X})$ , lower case letters denote constants and functions  $(x)$ , italic capital letters denote random variables  $(X)$ , and blackboard capitals denote the natural and real numbers  $(\mathbb{N},\mathbb{R},\mathbb{N}_0 = \mathbb{N}\cup \{0\})$ . Additionally, we let  $\Delta (\mathcal{X})$  denote the probability simplex over the set  $\mathcal{X}$ . That is, the function  $p:\mathcal{X}\times \mathcal{Y}\to \Delta (\mathcal{Z})$  expresses a probability mass function  $p(\cdot \mid x,y)$ , over  $\mathcal{Z}$ , for each  $x\in \mathcal{X}$  and  $y\in \mathcal{Y}$ . Lastly, we use  $\neg$  to denote logical negation, and we use  $\forall_{x\in \mathcal{X}}$  and  $\exists_{x\in \mathcal{X}}$  to express the universal and existential quantifiers over a set  $\mathcal{X}$ .
2.1 Agents and Environments
We begin by defining environments, agents, and related artifacts.
Definition 2.1. An agent-environment interface is a pair  $(\mathcal{A},\mathcal{O})$  of countable sets  $\mathcal{A}$  and  $\mathcal{O}$  where  $|\mathcal{A}|\geq 2$  and  $|O|\geq 1$ .
We refer to elements of  $\mathcal{A}$  as actions, denoted  $a$ , and elements of  $O$  as observations, denoted  $o$ . Histories define the possible interactions between an agent and an environment that share an interface.
Definition 2.2. The histories with respect to interface  $(\mathcal{A},\mathcal{O})$  are the set of sequences of action-observation pairs,
$$
\mathcal {H} = \bigcup_ {t = 0} ^ {\infty} (\mathcal {A} \times \mathcal {O}) ^ {t}. \tag {2.1}
$$
We refer to an individual element of  $\mathcal{H}$  as a history, denoted  $h$ , and we let  $hh'$  express the history resulting from the concatenation of any two histories  $h, h' \in \mathcal{H}$ . Furthermore, the set of histories of length  $t \in \mathbb{N}_0$  is defined as  $\mathcal{H}_t = (\mathcal{A} \times \mathcal{O})^t$ , and we use  $h_t \in \mathcal{H}_t$  to refer to a history containing  $t$  action-observation pairs,  $h_t = a_0 o_1 \ldots a_{t-1} o_t$ , with  $h_0 = \emptyset$  the empty history. An environment is then a function from the set of all environments,  $\mathcal{E}$ , that produces observations given a history.
Definition 2.3. An environment with respect to interface  $(\mathcal{A},\mathcal{O})$  is a function  $e:\mathcal{H}\times \mathcal{A}\to \Delta (\mathcal{O})$
This model of environments is general in that it can capture Markovian environments such as Markov decision processes (MDPs, Puterman, 2014) and partially observable MDPs (Cassandra et al., 1994), as well as both episodic and non-episodic settings. We next define an agent as follows.
Definition 2.4. An agent with respect to interface  $(\mathcal{A},\mathcal{O})$  is a function  $\lambda :\mathcal{H}\to \Delta (\mathcal{A})$
We let  $\Lambda$  denote the set of all agents, and let  $\Lambda$  denote any non-empty subset of  $\Lambda$ . This treatment of an agent captures the mathematical way experience gives rise to behavior, as in "agent functions" from work by Russell and Subramanian [50]. This is in contrast to a mechanistic account of agency as proposed by Dong et al. [16] and Sutton [58]. Further, note that Definition 2.4 is precisely a history-based policy; we embrace the view that there is no real distinction between an agent and a policy, and will refer to all such functions as "agents" unless otherwise indicated.
2.2 Realizable Histories
We will be especially interested in the histories that occur with non-zero probability as a result of the interaction between a particular agent and environment.
Definition 2.5. The realizable histories of a given agent-environment pair,  $(\lambda ,e)$ , define the set of histories of any length that can occur with non-zero probability from the interaction of  $\lambda$  and  $e$ ,
$$
\mathcal {H} ^ {\lambda , e} = \bar {\mathcal {H}} = \bigcup_ {t = 0} ^ {\infty} \left\{h _ {t} \in \mathcal {H} _ {t}: \prod_ {k = 0} ^ {t - 1} e \left(o _ {k + 1} \mid h _ {k}, a _ {k}\right) \lambda \left(a _ {k} \mid h _ {k}\right) > 0 \right\}. \tag {2.2}
$$
Given a realizable history  $h$ , we will refer to the realizable history suffixes,  $h'$ , which, when concatenated with  $h$ , produce a realizable history  $hh' \in \tilde{\mathcal{H}}$ .
Definition 2.6. The realizable history suffixes of a given  $(\lambda, e)$  pair, relative to a history prefix  $h \in \mathcal{H}^{\lambda, e}$ , define the set of histories that, when concatenated with prefix  $h$ , remain realizable,
$$
\mathcal {H} _ {h} ^ {\lambda , e} = \bar {\mathcal {H}} _ {h} = \left\{h ^ {\prime} \in \mathcal {H}: h h ^ {\prime} \in \mathcal {H} ^ {\lambda , e} \right\}. \tag {2.3}
$$
We abbreviate  $\mathcal{H}^{\lambda ,e}$  to  $\bar{\mathcal{H}}$ , and  $\mathcal{H}_h^{\lambda ,e}$  to  $\bar{\mathcal{H}}_h$ , where  $\lambda$  and  $e$  are obscured for brevity.
2.3 Reward, Performance, and the RL Problem
Supported by the arguments of Bowling et al. [7], we assume that all of the relevant goals or purposes of an agent are captured by a deterministic reward function (in line with the reward hypothesis [57]).
Definition 2.7. We call  $r: \mathcal{A} \times \mathcal{O} \to \mathbb{R}$  a reward function.
We remain agnostic to how the reward function is implemented; it could be a function inside of the agent, or the reward function's output could be a special scalar in each observation. Such commitments do not impact our framing. When we refer to an environment we will implicitly mean that a reward function has been selected as well. We remain agnostic to how reward is aggregated to determine performance, and instead adopt the function  $v$  defined as follows.
Definition 2.8. The performance,  $v: \mathcal{H} \times \mathbb{A} \times \mathcal{E} \to [\mathrm{v}_{\min}, \mathrm{v}_{\max}]$  is a bounded function for fixed constants  $\mathrm{v}_{\min}, \mathrm{v}_{\max} \in \mathbb{R}$ .
The function  $v(\lambda, e \mid h)$  expresses some statistic of the received future random rewards produced by the interaction between  $\lambda$  and  $e$  following history  $h$ , where we use  $v(\lambda, e)$  as shorthand for  $v(\lambda, e \mid h_0)$ . While we accommodate any  $v$  that satisfies the above definition, it may be useful to think of specific choices of  $v(\lambda, e \mid h_t)$ , such as the average reward,
$$
\lim  _ {k \rightarrow \infty} \inf  _ {k \rightarrow \infty} \frac {1}{k} \mathbb {E} _ {\lambda , e} [ R _ {t} + \dots + R _ {t + k} | H _ {t} = h _ {t} ], \tag {2.4}
$$
where  $\mathbb{E}_{\lambda ,e}\big[\dots \mid H_t = h_t\big]$  denotes expectation over the stochastic process induced by  $\lambda$  and  $e$  following history  $h_t$ . Or, we might consider performance based on the expected discounted reward,  $v(\lambda ,e\mid h_t) = \mathbb{E}_{\lambda ,e}[R_t + \gamma R_{t + 1} + \ldots \mid H_t = h_t]$ , where  $\gamma \in [0,1)$  is a discount factor.
The above components give rise to a simple definition of the RL problem.
Definition 2.9. An instance of the RL problem is defined by a tuple  $(e, v, \Lambda)$  as follows
$$
\Lambda^ {*} = \arg \max  _ {\lambda \in \Lambda} v (\lambda , e). \tag {2.5}
$$
This captures the RL problem facing an agent designer that would like to identify an optimal agent  $(\lambda^{*}\in \Lambda^{*})$  with respect to the performance  $(v)$ , among the available agents  $(\Lambda)$ , in a particular environment  $(e)$ . We note that a simple extension of this definition of the RL problem might instead consider a set of environments (or similar alternatives).
3 Agent Operators: Generates and Reaches
We next introduce two new insights about agents, and the logical operators that formalize them:
1. Theorem 3.1: Every agent can be understood as searching over another set of agents.
2. Remark 3.2: Every agent will either continue their search forever, or eventually stop.
We make these insights precise by introducing a pair of logical operators on agents: (1) a set of agents generates (Definition 3.4) another set of agents, and (2) a given agent reaches (Definition 3.5) an agent set. Together, these operators enable us to define learning as the implicit search process captured by the first insight, and continual learning as the process of continuing this search indefinitely.
3.1 Operator 1: An Agent Basis Generates an Agent Set.
The first operator is based on two complementary intuitions.
From the first perspective, an agent can be understood as searching over a space of representable action-selection strategies. For instance, in an MDP, agents can be interpreted as searching over the space of policies (that is, the space of stochastic mappings from the MDP's state to action). It turns out this insight can be extended to any agent and any environment.
The second complementary intuition notes that, as agent designers, we often first identify the space of representable action-selection strategies of interest. Then, it is natural to design agents that search through this space. For instance, in designing an agent to interact with an MDP, we might be interested in policies representable by a neural network of a certain size and architecture. When we design agents, we then consider all agents (choices of loss function, optimizer, memory, and so on) that search through the space of assignments of weights to this particular neural network using standard methods like gradient descent. We codify these intuitions in the following definitions.
Definition 3.1. An agent basis (or simply, a basis),  $\Lambda_{\mathrm{B}}\subset \mathbb{A}$  , is any non-empty subset of  $\mathbb{A}$
Notice that an agent basis is a choice of agent set,  $\Lambda$ . We explicitly call out a basis with distinct notation  $(\Lambda_{\mathrm{B}})$  as it serves an important role in the discussion that follows. For example, we next introduce learning rules as functions that switch between elements of an agent basis for each history.
Definition 3.2. A learning rule over an agent basis  $\Lambda_{\mathrm{B}}$  is a function,  $\sigma : \mathcal{H} \to \Lambda_{\mathrm{B}}$ , that selects a base agent for each history.
We let  $\Sigma$  denote the set of all learning rules over  $\Lambda_{\mathrm{B}}$ , and let  $\Sigma$  denote any non-empty subset of  $\Sigma$ . A learning rule is a mechanism for switching between the available base agents following each new experience. Notice that learning rules are deterministic; while a simple extension captures the stochastic case, we will see by Theorem 3.1 that the above is sufficiently general in a certain sense. We use  $\sigma(h)(h)$  to refer to the action distribution selected by the agent  $\lambda = \sigma(h)$  at any history  $h$ .
Definition 3.3. Let  $\Sigma$  be a set of learning rules over some basis  $\Lambda_{\mathrm{B}}$ , and let  $e$  be an environment. We say that a set  $\Lambda$  is  $\Sigma$ -generated by  $\Lambda_{\mathrm{B}}$  in  $e$ , denoted  $\Lambda_{\mathrm{B}} \vDash_{\Sigma} \Lambda$ , if and only if
$$
\forall_ {\lambda \in \Lambda} \exists_ {\sigma \in \Sigma} \forall_ {h \in \bar {\mathcal {H}}} \lambda (h) = \sigma (h) (h). \tag {3.1}
$$
Thus, any choice of  $\Sigma$  together with a basis  $\Lambda_{\mathrm{B}}$  induces a family of agent sets whose elements can be understood as switching between the basis according to the rules prescribed by  $\Sigma$ . We then say that a basis generates an agent set in an environment if there exists a set of learning rules that switches between the basis elements to produce the agent set.
Definition 3.4. We say a basis  $\Lambda_{\mathrm{B}}$  generates  $\Lambda$  in  $e$ , denoted  $\Lambda_{\mathrm{B}} \notin \Lambda$ , if and only if
$$
\Lambda_ {\mathrm {B}} \varepsilon_ {\Sigma} \Lambda . \tag {3.2}
$$
Intuitively, an agent basis  $\Lambda_{\mathrm{B}}$  generates another agent set  $\Lambda$  just when the agents in  $\Lambda$  can be understood as switching between the base agents. It is in this sense that we can understand agents as searching through a basis—an agent is just a particular sequence of history-conditioned switches over a basis. For instance, let us return to the example of a neural network: The agent basis might represent a specific multilayer perceptron, where each element of this basis is an assignment to the network's weights. The learning rules are different mechanisms that choose the next set of weights in
(a) Generates  $(\Lambda_{\mathrm{B}} \notin \Lambda)$
(b) Sometimes Reaches  $(\lambda_1 \rightsquigarrow \Lambda_{\mathrm{B}})$
Figure 1: A visual of the generates (left) and sometimes reaches (right) operators. (a) Generates: An agent basis,  $\Lambda_{\mathrm{B}}$ , comprised of three base agents depicted by the triangle, circle, and square, generates a set  $\Lambda$  containing agents that can each be understood as switching between the base agents in the realizable histories of environment  $e$ . (b) Sometimes Reaches: On the right, we visualize  $\lambda_{1} \in \Lambda$  generated by  $\Lambda_{\mathrm{B}}$  (from the figure on the left) to illustrate the concept of sometimes reaches. That is, the agent's choice of action distribution at each history can be understood as switching between the three basis elements, and there is at least one history for which the agent stops switching—here, we show the agent settling on the choice of the blue triangle and never switching again.
response to experience (such as gradient descent). Together, the agent basis and the learning rules generate the set of agents that search over choices of weights in reaction to experience. We present a cartoon visual of the generates operator in Figure 1(a).
Now, using the generates operator, we revisit and formalize the central insight of this section: Every agent can be understood as implicitly searching over an agent basis. We take this implicit search process to be the behavioral signature of learning.
Theorem 3.1. For any agent-environment pair  $(\lambda, e)$ , there exists infinitely many choices of a basis,  $\Lambda_{\mathrm{B}}$ , such that both (1)  $\lambda \notin \Lambda_{\mathrm{B}}$ , and (2)  $\Lambda_{\mathrm{B}} \notin \{\lambda\}$ .
Due to space constraints, all proofs are deferred to Appendix B.
We require that  $\lambda \notin \Lambda_{\mathrm{B}}$  to ensure that the relevant bases are non-trivial generators of  $\{\lambda\}$ . This theorem tells us that no matter the choice of agent or environment, we can view the agent as a series of history-conditioned switches between basis elements. In this sense, we can understand the agent as if it were carrying out a search over the elements of some  $\Lambda_{\mathrm{B}}$ . We emphasize that there are infinitely many choices of such a basis to illustrate that there are many plausible interpretations of an agent's behavior—we return to this point throughout the paper.
3.2 Operator 2: An Agent Reaches a Basis.
Our second operator reflects properties of an agent's limiting behavior in relation to a basis. Given an agent and a basis that the agent searches through, what happens to the agent's search process in the limit: does the agent keep switching between elements of the basis, or does it eventually stop? For example, in an MDP, many agents of interest eventually stop their search on a choice of a fixed policy. We formally define this notion in terms of an agent reaching a basis according to two modalities: an agent (i) sometimes or (ii) never reaches a basis.
Definition 3.5. We say agent  $\lambda \in \mathbb{A}$  sometimes reaches  $\Lambda_{\mathrm{B}}$  in  $e$ , denoted  $\lambda \stackrel{\sim}{\rightarrow} \Lambda_{\mathrm{B}}$ , if and only if
$$
\exists_ {h \in \bar {\mathcal {H}}} \exists_ {\lambda_ {\mathrm {B}} \in \Lambda_ {\mathrm {B}}} \forall_ {h ^ {\prime} \in \bar {\mathcal {H}} _ {h}} \lambda \left(h h ^ {\prime}\right) = \lambda_ {\mathrm {B}} \left(h h ^ {\prime}\right). \tag {3.3}
$$
That is, for at least one realizable history, there is some base agent  $(\lambda_{\mathrm{B}})$  that produces the same action distribution as  $\lambda$  forever after. This indicates that the agent can be understood as if it has stopped its search over the basis. We present a visual of sometimes reaches in Figure 1(b). By contrast, we say an agent never reaches a basis just when it never becomes equivalent to a base agent.
Definition 3.6. We say agent  $\lambda \in \mathbb{A}$  never reaches  $\Lambda_{\mathrm{B}}$  in  $e$ , denoted  $\lambda \not\to \Lambda_{\mathrm{B}}$ , iff  $\neg (\lambda \not\to \Lambda_{\mathrm{B}})$ .
<sup>1</sup>We use as if in the sense of the positive economists, such as Friedman [19].
The reaches operators formalize the intuition that, since every agent can be interpreted as if it were searching over a basis, every agent will either (1) sometimes, or (2) never stop this search. Since (1) and (2) are simply negations of each other, we can now plainly state this fact as follows.
Remark 3.2. For any agent-environment pair  $(\lambda, e)$  and any choice of basis  $\Lambda_{\mathrm{B}}$  such that  $\Lambda_{\mathrm{B}} \notin \{\lambda\}$ , exactly one of the following two properties must be satisfied:
$$
(1) \lambda \not \sim \Rightarrow \Lambda_ {\mathrm {B}}, \quad (2) \lambda \not \sim \Rightarrow \Lambda_ {\mathrm {B}}. \tag {3.4}
$$
Thus, by Theorem 3.1, every agent can be thought of as implicitly searching over an agent basis, and by Remark 3.2, every agent will either (1) sometimes, or (2) never stop this search. We take this implicit search process to be the behavioral signature of learning, and will later exploit this perspective to define a continual learning agent as one that continues its search forever (Definition 4.1). Our analysis in Section 4.4 further elucidates basic properties of both the generates and reaches operators, and Figure 1 presents a cartoon visualizing the intuition behind each of the operators. We summarize all definitions and notation in a table in Appendix A.
Considerations on the Operators. Naturally, we can design many variations of both  $\sqrt{}$  and  $\mathbb{E}$ . For instance, we might be interested in a variant of reaches in which an agent becomes  $\epsilon$ -close to any of the basis elements, rather than require exact behavioral equivalence. Concretely, we highlight four axes of variation that can modify the definitions of the operators. We state these varieties for reaches, but similar modifications can be made to the generates operator, too:
1. Realizability. An agent reaches a basis (i) in all histories (and thus, all environments), or (ii) in the histories realizable by a given  $(\lambda, e)$  pair.
2. History Length. An agent reaches a basis over (i) infinite or, (ii) finite length histories.
3. Probability. An agent reaches a basis (i) with probability one, or (ii) with high probability.
4. Equality or Approximation. An agent reaches a basis by becoming (i) equivalent to a base agent, or (ii) sufficiently similar to a base agent.
Rather than define all of these variations precisely for both operators (though we do explore some in Appendix C), we acknowledge their existence, and simply note that the formal definitions of these variants follow naturally.