SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4 Continual Reinforcement Learning
We now provide a precise definition of CRL. The definition formalizes the intuition that CRL captures settings in which the best agents do not converge—they continue their implicit search over an agent basis indefinitely.
4.1 Definition:Continual RL
We first define continual learning agents using the generates and never reaches operators as follows.
Definition 4.1. An agent  $\lambda$  is a continual learning agent in  $e$  relative to  $\Lambda_{\mathrm{B}}$  if and only if the basis generates the agent  $(\Lambda_{\mathrm{B}} \notin \{\lambda\})$  and the agent never reaches the basis ( $\lambda \not\to \Lambda_{\mathrm{B}}$ ).
This means that an agent is a continual learner in an environment relative to  $\Lambda_{\mathrm{B}}$  if the agent's search over  $\Lambda_{\mathrm{B}}$  continues forever. Notice that an agent might be considered a continual learner with respect to one basis but not another; we explore this fact more in Section 4.4.
Then, using these tools, we formally define CRL as follows.
Definition 4.2. Consider an RL problem  $(e, v, \Lambda)$ . Let  $\Lambda_{\mathrm{B}} \subset \Lambda$  be a basis such that  $\Lambda_{\mathrm{B}} \not\subset \Lambda$ , and let  $\Lambda^{*} = \arg \max_{\lambda \in \Lambda} v(\lambda, e)$ . We say  $(e, v, \Lambda, \Lambda_{\mathrm{B}})$  defines a CRL problem if  $\forall \lambda^{*} \in \Lambda^{*} \lambda^{*} \nrightarrow \Lambda_{\mathrm{B}}$ .
Said differently, an RL problem is an instance of CRL just when all of the best agents are continual learning agents relative to basis  $\Lambda_{\mathrm{B}}$ . This problem encourages a significant departure from how we tend to think about designing agents: Given a basis, rather than try to build agents that can solve problems by identifying a fixed high-quality element of the basis, we would like to design agents that continue to update their behavior indefinitely in light of their experience.
4.2 CRL Examples
We next detail two examples of CRL to provide further intuition.
Q-Learning in Switching MDPs. First we consider a simple instance of CRL based on the standard multi-task view of MDPs. In this setting, the agent repeatedly samples an MDP to interact with from a fixed but unknown distribution [64, 9, 2, 25, 20]. In particular, we make use of the switching MDP environment from Luketina et al. [33]. The environment  $e$  consists of a collection of  $n$  underlying MDPs,  $m_1, \ldots, m_n$ , with a shared action space and environment-state space. We refer to this environment-state space using observations,  $o \in O$ . The environment has a fixed constant positive probability of 0.001 to switch the underlying MDP, which yields different transition and reward functions until the next switch. The agent only observes each environment state  $o \in O$ , which does not reveal the identity of the active MDP. The rewards of each underlying MDP are structured so that each MDP has a unique optimal policy. We assume  $v$  is defined as the average reward, and the basis is the set of  $\epsilon$ -greedy policies over all  $Q(o, a)$  functions, for fixed  $\epsilon = 0.15$ . Consequently, the set of agents we generate,  $\Lambda_{\mathrm{B}} \notin \Lambda$ , consists of all agents that switch between these  $\epsilon$ -greedy policies.
Now, the components  $(e,v,\Lambda ,\Lambda_{\mathrm{B}})$  have been defined, we can see that this is indeed an instance of CRL: None of the base agents can be optimal, as the moment that the environment switches its underlying MDP, we know that any previously optimal policy will no longer be optimal in the next MDP following the switch. Therefore, any agent that converges (in that it reaches the basis  $\Lambda_{\mathrm{B}}$ ) cannot be optimal either for the same reason. We conclude that all optimal agents in  $\Lambda$  are continual learning agents relative to the basis  $\Lambda_{\mathrm{B}}$ .
We present a visual of this domain in Figure 2(a), and conduct a simple experiment contrasting the performance of  $\epsilon$ -greedy continual Q-learning (blue) that uses a constant step-size parameter of  $\alpha = 0.1$ , with a convergent Q-learning (green) that anneals its step size parameter over time to zero. Both use  $\epsilon = 0.15$ , and we set the number of underlying MDPs to  $n = 10$ . We present the average reward with  $95\%$  confidence intervals, averaged over 250 runs, in Figure 2(b). Since both variants of Q-learning can be viewed as searching over  $\Lambda_{\mathrm{B}}$ , the annealing variant that stops its search will under-perform compared to the continual approach. These results support the unsurprising conclusion that it is better to continue searching over the basis rather than converge in this setting.
Continual Supervised Learning. Second, we illustrate the power of our CRL definition to capture continual supervised learning. We adopt the problem setting studied by Mai et al. [35]. Let  $\mathcal{X}$  denote a set of objects to be labeled, each belonging to one of  $k\in \mathbb{N}$  classes. The observation space  $\mathcal{O}$  consists of pairs,  $o_{t} = (x_{t},y_{t})$ , where  $x_{t}\in \mathcal{X}$  and  $y_{t}\in \mathcal{Y}$ . Here, each  $x_{t}$  is an input object to be classified and  $y_{t}$  is the label for the previous input  $x_{t - 1}$ . Thus,  $\mathcal{O} = \mathcal{X}\times \mathcal{Y}$ . We assume by convention that the initial
(a) Switching MDP Visual
(b) Switching MDP Results
Figure 2: A visual of a grid world instance of the switching MDPs problem (left) [33], and results from an experiment contrasting continual learning and convergent Q-learning (right). The environment pictured contains  $n$  distinct MDPs. Each underlying MDP shares the same state space and action space, but varies in transition and reward functions, as indicated by the changing walls and rewarding locations (stars, circles, and fire). The results pictured on the right contrast continual Q-learning (with  $\alpha = 0.1$ ) with traditional Q-learning that anneals its step-size parameter to zero over time.
label  $y_0$  is irrelevant and can be ignored. The agent will observe a sequence of object-label pairs,  $(x_0, y_0), (x_1, y_1), \ldots$ , and the action space is a choice of label,  $\mathcal{A} = \{a_1, \dots, a_k\}$  where  $|\mathcal{Y}| = k$ . The reward for each history  $h_t$  is  $+1$  if the agent's most recently predicted label is correct for the previous input, and  $-1$  otherwise:
$$
r \left(a _ {t - 1} o _ {t}\right) = r \left(a _ {t - 1} y _ {t}\right) = \left\{ \begin{array}{l l} + 1 & a _ {t - 1} = y _ {t}, \\ - 1 & a _ {t - 1} = \text {o t h e r w i s e}. \end{array} \right. \tag {4.1}
$$
Concretely, the continual learning setting studied by Mai et al. [35] supposes the learner will receive samples from a sequence of probability distributions,  $d_0, d_1, \ldots$ , each supported over  $X \times Y$ . The  $(x, y) \in X \times Y$  pairs experienced by the learner are determined by the sequence of distributions. We capture this distributional shift in an environment  $e$  that shifts its probability distribution over  $O$  depending on the history to match the sequence,  $d_0, d_1, \ldots$ .
Now, is this an instance of CRL? To answer this question precisely, we need to select a  $(\Lambda, \Lambda_{\mathrm{B}})$  pair. We adopt the basis  $\Lambda_{\mathrm{B}} = \{\lambda_{\mathrm{B}} : x \mapsto y_i, \forall_{y_i \in \mathcal{Y}}\}$  that contains each classifier that maps each object to each possible label. By the universal set of learning rules  $\Sigma$ , this basis generates the set of all agents that search over classifiers. Now, our definition says the above is an instance of CRL just when every optimal agent never stops switching between classifiers, rather than stop their search on a fixed classifier. Consequently, if there is an optimal classifier in  $\Lambda_{\mathrm{B}}$ , then this will not be an instance of CRL. If, however, the environment imposes enough distributional shift (changing labels, adding mass to new elements, and so on), then the only optimal agents will be those that always switch among the base classifiers, in which case the setting is an instance of CRL.
4.3 Relationship to Other Views on Continual Learning
The spirit of continual learning has been an important part of machine learning research for decades, often appearing under the name of "lifelong learning" [63, 62, 53, 55, 51, 3, 4], "never-ending learning" [39, 43] with close ties to transfer-learning [61, 60], meta-learning [52, 17], as well as online learning and non-stationarity [5, 40, 13, 6, 31]. In a similar vein, the phrase "continuing tasks" is used in the classic RL textbook [59] to refer explicitly to cases when the interaction between agent and environment is not subdivided into episodes. Continual reinforcement learning was first posed in the thesis by Ring [46]. In later work [47, 48], Ring proposes a formal definition of the continual reinforcement learning problem—The emphasis of Ring's proposal is on the generality of the environment: rather than assume that agents of interest will interact with an MDP, Ring suggests studying the unconstrained case in which an agent must maximize performance while only receiving a stream of observations as input. The environment or reward function, in this sense, may change over time or may be arbitrarily complex. This proposal is similar in spirit to general RL, studied by Hutter [24], Lattimore [28], Leike [29], and others [12, 37, 36] in which an agent interacts with an unconstrained environment. General RL inspires many aspects of our conception of CRL; for instance, our emphasis on history-dependence rather than environment-state comes directly from general RL. More recently, Khetarpal et al. [25] provide a comprehensive survey of the continual reinforcement learning literature. We encourage readers to explore this survey for a detailed history of the subject. In the survey, Khetarpal et al. propose a definition of the CRL problem that emphasizes the non-stationarity of the underlying process. In particular, in Khetarpal et al.'s definition, an agent interacts with a POMDP in which each of the individual components of the POMDP—such as the state space or reward function—are allowed to vary with time. We note that, as the environment model we study (Definition 2.3) is a function of history, it can capture time-indexed non-stationarity. In this sense, the same generality proposed by Khetarpal et al. and Ring is embraced and retained by our definition, but we add further precision to what is meant by continual learning by centering around a mathematical definition of continual learning agents (Definition 4.1).
4.4 Properties of CRL
Our formalism is intended to be a jumping off point for new lines of thinking around agents and continual learning. We defer much of our analysis and proofs to the appendix, and here focus on highlighting necessary properties of CRL.
2For other surveys, see the recent survey on continual robot learning by Lesort et al. [30], a survey on continual learning with neural networks by Parisi et al. [42], a survey on transfer learning in RL by Taylor and Stone [60], and a survey on continual image classification by Mai et al. [35].
Theorem 4.1. Every instance of  $CRL(e, v, \Lambda, \Lambda_{\mathrm{B}})$  necessarily satisfies the following properties:
1. If  $\Lambda \neq \Lambda_{\mathrm{B}} \cup \Lambda^{*}$ , then there exists a  $\Lambda_{\mathrm{B}}'$  such that (1)  $\Lambda_{\mathrm{B}}' \notin \Lambda$ , and (2)  $(e, v, \Lambda, \Lambda_{\mathrm{B}}')$  is not an instance of CRL.
2. No element of  $\Lambda_{\mathrm{B}}$  is optimal:  $\Lambda_{\mathrm{B}}\cap \Lambda^{*} = \emptyset$
3. If  $|\Lambda|$  is finite, there exists an agent set,  $\Lambda^{\circ}$ , such that  $|\Lambda^{\circ}| < |\Lambda|$  and  $\Lambda^{\circ} \notin \Lambda$ .
4. If  $|\Lambda|$  is infinite, there exists an agent set,  $\Lambda^{\circ}$ , such that  $\Lambda^{\circ} \subset \Lambda$  and  $\Lambda^{\circ} \nVdash \Lambda$ .
This theorem tells us several things. The first point of the theorem has peculiar implications. We see that as we change a single element (the basis  $\Lambda_{\mathrm{B}}$ ) of the tuple  $(e, v, \Lambda_{\mathrm{B}}, \Lambda)$ , the resulting problem can change from CRL to not CRL. By similar reasoning, an agent that is said to be a continual learning agent according to Definition 4.1 may not be a continual learner with respect to some other basis. We discuss this point further in the next paragraph. Point (2.) notes that no optimal strategy exists within the basis—instead, to be optimal, an agent must switch between basis elements indefinitely. As discussed previously, this fact encourages a departure in how we think about the RL problem: rather than focus on agents that can identify a single, fixed solution to a problem, CRL instead emphasizes designing agents that are effective at updating their behavior indefinitely. Points (3.) and (4.) show that  $\Lambda$  cannot be minimal. That is, there are necessarily some redundancies in the design space of the agents in CRL—this is expected, since we are always focusing on agents that search over the same agent basis. Lastly, it is worth calling attention to the fact that in the definition of CRL, we assume  $\Lambda_{\mathrm{B}} \subset \Lambda$ —this suggests that in CRL, the agent basis is necessarily limited in some way. Consequently, the design space of agents  $\Lambda$  are also limited in terms of what agents they can represent at any particular point in time. This limitation may come about due to a computational or memory budget, or by making use of a constrained set of learning rules. This suggests a deep connection between bounded agents and the nature of continual learning, as explored further by Kumar et al. [27]. While these four points give an initial character of the CRL problem, we note that further exploration of the properties of CRL is an important direction for future work.
Canonical Agent Bases. It is worth pausing and reflecting on the concept of an agent basis. As presented, the basis is an arbitrary choice of a set of agents—consequently, point (1.) of Theorem 4.1 may stand out as peculiar. From this perspective, it is reasonable to ask if the fact that our definition of CRL is basis-dependant renders it vacuous. We argue that this is not the case for two reasons. First, we conjecture that any definition of continual learning that involves concepts like "learning" and "convergence" will have to sit on top of some reference object whose choice is arbitrary. Second, and more important, even though the mathematical construction allows for an easy change of basis, in practice the choice of basis is constrained by considerations like the availability of computational resources. It is often the case that the domain or problem of interest provides obvious choices of bases, or imposes constraints that force us as designers to restrict attention to a space of plausible bases or learning rules. For example, as discussed earlier, a choice of neural network architecture might comprise a basis—any assignment of weights is an element of the basis, and the learning rule  $\sigma$  is a mechanism for updating the active element of the basis (the parameters) in light of experience. In this case, the number of parameters of the network is constrained by what we can actually build, and the learning rule needs to be suitably efficient and well-behaved. We might again think of the learning rule  $\sigma$  as gradient descent, rather than a rule that can search through the basis in an unconstrained way. In this sense, the basis is not arbitrary. We as designers choose a class of functions to act as the relevant representations of behavior, often limited by resource constraints on memory or compute. Then, we use specific learning rules that have been carefully designed to react to experience in a desirable way—for instance, stochastic gradient descent updates the current choice of basis in the direction that would most improve performance. For these reasons, the choice of basis is not arbitrary, but instead reflects the ingredients involved in the design of agents as well as the constraints necessarily imposed by the environment.
4.5 Properties of Generates and Reaches
Lastly, we summarize some of the basic properties of generates and reaches. Further analysis of generates, reaches, and their variations is provided in Appendix C.
Theorem 4.2. The following properties hold of the generates operator:
1. Generates is transitive: For any triple  $(\Lambda^1, \Lambda^2, \Lambda^3)$  and  $e \in \mathcal{E}$ , if  $\Lambda^1 \not\subsetneq \Lambda^2$  and  $\Lambda^2 \not\subsetneq \Lambda^3$ , then  $\Lambda^1 \not\subsetneq \Lambda^3$ .
2. Generates is not commutative: there exists a pair  $(\Lambda^1, \Lambda^2)$  and  $e \in \mathcal{E}$  such that  $\Lambda^1 \not\in \Lambda^2$ , but  $\neg (\Lambda^2 \nmid e \Lambda^1)$ .
3. For all  $\Lambda$  and pair of agent bases  $(\Lambda_{\mathrm{B}}^{1},\Lambda_{\mathrm{B}}^{2})$  such that  $\Lambda_{\mathrm{B}}^{1}\subseteq \Lambda_{\mathrm{B}}^{2}$  if  $\Lambda_{\mathrm{B}}^{1}\vDash \Lambda$  , then  $\Lambda_{\mathrm{B}}^{2}\vDash \Lambda$
4. For all  $\Lambda$  and  $e\in \mathcal{E},\wedge \nmid e\Lambda$
5. The decision problem, Given  $(e, \Lambda_{\mathrm{B}}, \Lambda)$ , output True iff  $\Lambda_{\mathrm{B}} \notin \Lambda$ , is undecidable.
The fact that generates is transitive suggests that the basic tools of an agent set—paired with a set of learning rules—might be likened to an algebraic structure. We can draw a symmetry between an agent basis and the basis of a vector space: A vector space is comprised of all linear combinations of the basis, whereas  $\Lambda$  is comprised of all valid switches (according to the learning rules) between the base agents. However, the fact that generates is not commutative (by point 2.) raises a natural question: are there choices of learning rules under which generates is commutative? We suggest that a useful direction for future work can further explore an algebraic perspective on agents.
We find many similar properties hold of reaches.
Theorem 4.3. The following properties hold of the reaches operator:
1.  $\nrightarrow$  and  $\nrightarrow$  are not transitive.
2. "Sometimes reaches" is not commutative: there exists a pair  $(\Lambda^1, \Lambda^2)$  and  $e \in \mathcal{E}$  such that  $\forall_{\lambda^1 \in \Lambda^1} \lambda^1 \rightsquigarrow \Lambda^2$ , but  $\exists_{\lambda^2 \in \Lambda^2} \lambda^2 \not\Rightarrow \Lambda^1$ .
3. For all pairs  $(\Lambda, e)$ , if  $\lambda \in \Lambda$ , then  $\lambda \rightsquigarrow \Lambda$ .
4. Every agent satisfies  $\lambda \prec \infty \land$  in every environment.
5. The decision problem, Given  $(e,\lambda ,\Lambda)$  output True iff  $\lambda \rightsquigarrow \Lambda$  is undecidable.
Many of these properties resemble those in Theorem 4.2. For instance, point (5.) shows that deciding whether a given agent sometimes reaches a basis in an environment is undecidable. We anticipate that the majority of decision problems related to determining properties of arbitrary agent sets interacting with unconstrained environments will be undecidable, though it is still worth making these arguments carefully. Moreover, there may be interesting special cases in which these decision problems are decidable (and perhaps, efficiently so). We suggest that identifying these special cases and fleshing out their corresponding efficient algorithms is an interesting direction for future work.
In this paper, we carefully develop a simple mathematical definition of the continual RL problem. We take this problem to be of central importance to AI as a field, and hope that these tools and perspectives can serve as an opportunity to think about CRL and its related artifacts more carefully. Our proposal is framed around two new insights about agents: (i) every agent can be understood as though it were searching over an agent basis (Theorem 3.1), and (ii) every agent, in the limit, will either sometimes or never stop this search (Remark 3.2). These two insights are formalized through the generates and reaches operators, which provide a rich toolkit for understanding agents in a new way—for example, we find straightforward definitions of a continual learning agent (Definition 4.1) and learning rules (Definition 3.2). We anticipate that further study of these operators and different families of learning rules can directly inform the design of new learning algorithms; for instance, we might characterize the family of continual learning rules that are guaranteed to yield continual learning agents, and use this to guide the design of principled continual learning agents (in the spirit of continual backprop by Dohare et al. [14]). In future work, we intend to further explore connections between our formalism of continual learning and some of the phenomena at the heart of recent empirical continual learning studies, such as plasticity loss [34, 1, 15], in-context learning [8], and catastrophic forgetting [38, 18, 21, 26]. More generally, we hope that our definitions, analysis, and perspectives can help the community to think about continual reinforcement learning in a new light.
The authors are grateful to Michael Bowling, Clare Lyle, Razvan Pascanu, and Georgios Piliouras for comments on a draft of the paper, as well as the anonymous NeurIPS reviewers that provided valuable feedback on the paper. The authors would further like to thank all of the 2023 Barbados RL Workshop participants and Elliot Catt, Will Dabney, Sebastian Flennerhag, András György, Steven Hansen, Anna Harutyunyan, Mark Ho, Joe Marino, Joseph Modayil, Rémi Munos, Evgenii Nikishin, Brendan O'Donoghue, Matt Overlan, Mark Rowland, Tom Schaul, Yannick Shroecker, Rich Sutton, Yunhao Tang, Shantanu Thakoor, and Zheng Wen for inspirational conversations.