Title: A Definition of Continual Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Continual Reinforcement Learning

Reinforcement Learning

Lifelong Reinforcement Learning

Continual Learning
Abstract: In a standard view of the reinforcement learning problem, an agent’s goal is to efficiently identify a policy that maximizes long-term reward. However, this perspective is based on a restricted view of learning as finding a solution, rather than treating learning as endless adaptation. In contrast, continual reinforcement learning refers to the setting in which the best agents never stop learning. Despite the importance of continual reinforcement learning, the community lacks a simple definition of the problem that highlights its commitments and makes its primary concepts precise and clear. To this end, this paper is dedicated to carefully defining the continual reinforcement learning problem. We formalize the notion of agents that “never stop learning” through a new mathematical language for analyzing and cataloging agents. Using this new language, we define a continual learning agent as one that can be understood as carrying out an implicit search process indefinitely, and continual reinforcement learning as the setting in which the best agents are all continual learning agents. We provide two motivating examples, illustrating that traditional views of multi-task reinforcement learning and continual supervised learning are special cases of our definition. Collectively, these definitions and perspectives formalize many intuitive concepts at the heart of learning, and open new research pathways surrounding continual learning agents.

A Definition of Continual Reinforcement Learning

Abstract

In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward. However, this perspective is based on a restricted view of learning as finding a solution , rather than treating learning as endless adaptation . In contrast, continual reinforcement learning refers to the setting in which the best agents never stop learning. Despite the importance of continual reinforcement learning, the community lacks a simple definition of the problem that highlights its commitments and makes its primary concepts precise and clear. To this end, this paper is dedicated to carefully defining the continual reinforcement learning problem. We formalize the notion of agents that 'never stop learning' through a new mathematical language for analyzing and cataloging agents. Using this new language, we define a continual learning agent as one that can be understood as carrying out an implicit search process indefinitely, and continual reinforcement learning as the setting in which the best agents are all continual learning agents. We provide two motivating examples, illustrating that traditional views of multi-task reinforcement learning and continual supervised learning are special cases of our definition. Collectively, these definitions and perspectives formalize many intuitive concepts at the heart of learning, and open new research pathways surrounding continual learning agents.

1 Introduction

In The Challenge of Reinforcement Learning , Sutton states: 'Part of the appeal of reinforcement learning is that it is in a sense the whole AI problem in a microcosm' [56]. Indeed, the problem facing an agent that learns to make better decisions from experience is at the heart of the study of Artificial Intelligence (AI). Yet, when we study the reinforcement learning (RL) problem, it is typical to restrict our focus in a number of ways. For instance, we often suppose that a complete description of the state of the environment is available to the agent, or that the interaction stream is subdivided into episodes. Beyond these standard restrictions, however, there is another significant assumption that constrains the usual framing of RL: We tend to concentrate on agents that learn to solve problems, rather than agents that learn forever. For example, consider an agent learning to play Go: Once the agent has discovered how to master the game, the task is complete, and the agent's learning can stop. This view of learning is often embedded in the standard formulation of RL, in which an agent interacts with a Markovian environment with the goal of efficiently identifying an optimal policy, at which point learning can cease.

But what if this is not the best way to model the RL problem? That is, instead of viewing learning as finding a solution , we can instead think of it as endless adaptation . This suggests study of the continual reinforcement learning (CRL) problem [47, 48, 25, 27], as first explored in the thesis by

Ring [46], with close ties to supervised never-ending [10, 39, 43] and continual learning [47, 48, 26, 54, 41, 42, 49, 22, 30, 45, 4].

Despite the prominence of CRL, the community lacks a clean, general definition of this problem. It is critical to develop such a definition to promote research on CRL from a clear conceptual foundation, and to guide us in understanding and designing continual learning agents. To these ends, this paper is dedicated to carefully defining the CRL problem. Our definition is summarized as follows:

The CRL Problem (Informal)

An RL problem is an instance of CRL if the best agents never stop learning.

The core of our definition is framed around two new insights that formalize the notion of 'agents that never stop learning': (i) we can understand every agent as implicitly searching over a set of history-based policies (Theorem 3.1), and (ii) every agent will either continue this search forever, or eventually stop (Remark 3.2). We make these two insights rigorous through a pair of logical operators on agents that we call generates and reaches that provide a new mathematical language for characterizing agents. Using these tools, we then define CRL as any RL problem in which all of the best agents never stop their implicit search. We provide two motivating examples of CRL, illustrating that traditional multi-task RL and continual supervised learning are special cases of our definition. We further identify necessary properties of CRL (Theorem 4.1) and the new operators (Theorem 4.2, Theorem 4.3). Collectively, these definitions and insights formalize many intuitive concepts at the heart of continual learning, and open new research pathways surrounding continual learning agents.

2 Preliminaries

We first introduce key concepts and notation. Our conventions are inspired by Ring [46], the recent work by Dong et al. [16] and Lu et al. [32], as well as the literature on general RL by Hutter [23, 24], Lattimore [28], Leike [29], Cohen et al. [12], and Majeed [36].

Notation. We let capital calligraphic letters denote sets ( X ), lower case letters denote constants and functions ( G ), italic capital letters denote random variables ( -), and blackboard capitals denote the natural and real numbers ( N GLYPH<150> R , N 0 = N [ f 0 g ). Additionally, we let GLYPH<1> 'X' denote the probability simplex over the set X . That is, the function ? : X GLYPH<2> Y ! GLYPH<1> 'Z' expresses a probability mass function ? 'GLYPH<1> j GGLYPH<150> H ' , over Z , for each G 2 X and H 2 Y . Lastly, we use : to denote logical negation, and we use 8 G 2X and 9 G 2X to express the universal and existential quantifiers over a set X .

2.1 Agents and Environments

We begin by defining environments, agents, and related artifacts.

Definition 2.1. An agent-environment interface is a pair 'A GLYPH<150> O' of countable sets A and O where jAj GLYPH<21> 2 and jOj GLYPH<21> 1 .

We refer to elements of A as actions , denoted 0 , and elements of O as observations , denoted > . Histories define the possible interactions between an agent and an environment that share an interface.

Definition 2.2. The histories with respect to interface 'A GLYPH<150> O' are the set of sequences of actionobservation pairs,

We refer to an individual element of H as a history , denoted GLYPH<17> , and we let GLYPH<17> GLYPH<17> 0 express the history resulting from the concatenation of any two histories GLYPH<17> GLYPH<150> GLYPH<17> 0 2 H . Furthermore, the set of histories of length C 2 N 0 is defined as H C = 'A GLYPH<2> O' C , and we use GLYPH<17> C 2 H C to refer to a history containing C action-observation pairs, GLYPH<17> C = 0 0 > 1 GLYPH<149> GLYPH<149> GLYPH<149> 0C GLYPH<0> 1 >C , with GLYPH<17> 0 = ; the empty history. An environment is then a function from the set of all environments, E , that produces observations given a history.

Definition 2.3. An environment with respect to interface ( A GLYPH<150> O' is a function 4 : H GLYPH<2>A ! GLYPH<1> 'O' .

This model of environments is general in that it can capture Markovian environments such as Markov decision processes (MDPs, Puterman, 2014) and partially observable MDPs (Cassandra et al., 1994), as well as both episodic and non-episodic settings. We next define an agent as follows.

Definition 2.4. An agent with respect to interface 'A GLYPH<150> O' is a function GLYPH<23> : H ! GLYPH<1> 'A' .

We let GLYPH<3> denote the set of all agents, and let GLYPH<3> denote any non-empty subset of GLYPH<3> . This treatment of an agent captures the mathematical way experience gives rise to behavior, as in 'agent functions' from work by Russell and Subramanian [50]. This is in contrast to a mechanistic account of agency as proposed by Dong et al. [16] and Sutton [58]. Further, note that Definition 2.4 is precisely a history-based policy; we embrace the view that there is no real distinction between an agent and a policy, and will refer to all such functions as 'agents' unless otherwise indicated.

2.2 Realizable Histories

We will be especially interested in the histories that occur with non-zero probability as a result of the interaction between a particular agent and environment.

Definition 2.5. The realizable histories of a given agent-environment pair, ' GLYPH<23> GLYPH<150> 4 ' , define the set of histories of any length that can occur with non-zero probability from the interaction of GLYPH<23> and 4 ,

Given a realizable history GLYPH<17> , we will refer to the realizable history suffixes , GLYPH<17> 0 , which, when concatenated with GLYPH<17> , produce a realizable history GLYPH<17> GLYPH<17> 0 2 ¢ H .

Definition 2.6. The realizable history suffixes of a given ' GLYPH<23> GLYPH<150> 4 ' pair, relative to a history prefix GLYPH<17> 2 H GLYPH<23> GLYPH<150>4 , define the set of histories that, when concatenated with prefix GLYPH<17> , remain realizable,

We abbreviate H GLYPH<23> GLYPH<150>4 to ¢ H , and H GLYPH<23> GLYPH<150>4 GLYPH<17> to ¢ H GLYPH<17> , where GLYPH<23> and 4 are obscured for brevity.

2.3 Reward, Performance, and the RL Problem

Supported by the arguments of Bowling et al. [7], we assume that all of the relevant goals or purposes of an agent are captured by a deterministic reward function (in line with the reward hypothesis [57]). Definition 2.7. We call A : AGLYPH<2> O ! R a reward function .

We remain agnostic to how the reward function is implemented; it could be a function inside of the agent, or the reward function's output could be a special scalar in each observation. Such commitments do not impact our framing. When we refer to an environment we will implicitly mean that a reward function has been selected as well. We remain agnostic to how reward is aggregated to determine performance, and instead adopt the function E defined as follows.

Definition 2.8. The performance , E : H GLYPH<2> GLYPH<3> GLYPH<2> E ! » vmin GLYPH<150> vmax … is a bounded function for fixed constants vmin GLYPH<150> vmax 2 R .

The function E ' GLYPH<23> GLYPH<150> 4 j GLYPH<17> ' expresses some statistic of the received future random rewards produced by the interaction between GLYPH<23> and 4 following history GLYPH<17> , where we use E ' GLYPH<23> GLYPH<150> 4 ' as shorthand for E ' GLYPH<23> GLYPH<150> 4 j GLYPH<17> 0 ' . While we accommodate any E that satisfies the above definition, it may be useful to think of specific choices of E ' GLYPH<23> GLYPH<150> 4 j GLYPH<17> C ' , such as the average reward, where E GLYPH<23> GLYPH<150>4 » GLYPH<1> GLYPH<1> GLYPH<1> j GLYPH<29> C = GLYPH<17> C … denotes expectation over the stochastic process induced by GLYPH<23> and 4 following history GLYPH<17> C . Or, we might consider performance based on the expected discounted reward, E ' GLYPH<23> GLYPH<150> 4 j GLYPH<17> C ' = E GLYPH<23> GLYPH<150>4 » 'C , GLYPH<15> 'C , 1 , GLYPH<149> GLYPH<149> GLYPH<149> j GLYPH<29> C = GLYPH<17> C … , where GLYPH<15> 2 » 0 GLYPH<150> 1 ' is a discount factor.

The above components give rise to a simple definition of the RL problem.

Definition 2.9. An instance of the RL problem is defined by a tuple ' 4GLYPH<150> EGLYPH<150> GLYPH<3> ' as follows

This captures the RL problem facing an agent designer that would like to identify an optimal agent ( GLYPH<23> GLYPH<3> 2 GLYPH<3> GLYPH<3> ) with respect to the performance ( E ), among the available agents ( GLYPH<3> ), in a particular environment ( 4 ). We note that a simple extension of this definition of the RL problem might instead consider a set of environments (or similar alternatives).

3 Agent Operators: Generates and Reaches

We next introduce two new insights about agents, and the logical operators that formalize them:

Theorem 3.1: Every agent can be understood as searching over another set of agents.

Remark 3.2: Every agent will either continue their search forever, or eventually stop.

We make these insights precise by introducing a pair of logical operators on agents: (1) a set of agents generates (Definition 3.4) another set of agents, and (2) a given agent reaches (Definition 3.5) an agent set. Together, these operators enable us to define learning as the implicit search process captured by the first insight, and continual learning as the process of continuing this search indefinitely.

3.1 Operator 1: An Agent Basis Generates an Agent Set.

The first operator is based on two complementary intuitions.

From the first perspective, an agent can be understood as searching over a space of representable action-selection strategies. For instance, in an MDP, agents can be interpreted as searching over the space of policies (that is, the space of stochastic mappings from the MDP's state to action). It turns out this insight can be extended to any agent and any environment.

The second complementary intuition notes that, as agent designers, we often first identify the space of representable action-selection strategies of interest. Then, it is natural to design agents that search through this space. For instance, in designing an agent to interact with an MDP, we might be interested in policies representable by a neural network of a certain size and architecture. When we design agents, we then consider all agents (choices of loss function, optimizer, memory, and so on) that search through the space of assignments of weights to this particular neural network using standard methods like gradient descent. We codify these intuitions in the following definitions.

Definition 3.1. An agent basis (or simply, a basis), GLYPH<3> B GLYPH<26> GLYPH<3> , is any non-empty subset of GLYPH<3> .

Notice that an agent basis is a choice of agent set, GLYPH<3> . We explicitly call out a basis with distinct notation ( GLYPH<3> B) as it serves an important role in the discussion that follows. For example, we next introduce learning rules as functions that switch between elements of an agent basis for each history.

Definition 3.2. A learning rule over an agent basis GLYPH<3> B is a function, GLYPH<30> : H ! GLYPH<3> B , that selects a base agent for each history.

We let GLYPH<6> denote the set of all learning rules over GLYPH<3> B, and let GLYPH<6> denote any non-empty subset of GLYPH<6> . A learning rule is a mechanism for switching between the available base agents following each new experience. Notice that learning rules are deterministic; while a simple extension captures the stochastic case, we will see by Theorem 3.1 that the above is sufficiently general in a certain sense. We use GLYPH<30> ' GLYPH<17> '' GLYPH<17> ' to refer to the action distribution selected by the agent GLYPH<23> = GLYPH<30> ' GLYPH<17> ' at any history GLYPH<17> .

Definition 3.3. Let GLYPH<6> be a set of learning rules over some basis GLYPH<3> B , and let 4 be an environment. We say that a set GLYPH<3> is GLYPH<6> -generated by GLYPH<3> B in 4 , denoted GLYPH<3> B ' 4 GLYPH<6> GLYPH<3> , if and only if

Thus, any choice of GLYPH<6> together with a basis GLYPH<3> B induces a family of agent sets whose elements can be understood as switching between the basis according to the rules prescribed by GLYPH<6> . We then say that a basis generates an agent set in an environment if there exists a set of learning rules that switches between the basis elements to produce the agent set.

Definition 3.4. We say a basis GLYPH<3> B generates GLYPH<3> in 4 , denoted GLYPH<3> B ' 4 GLYPH<3> , if and only if

Intuitively, an agent basis GLYPH<3> B generates another agent set GLYPH<3> just when the agents in GLYPH<3> can be understood as switching between the base agents. It is in this sense that we can understand agents as searching through a basis-an agent is just a particular sequence of history-conditioned switches over a basis. For instance, let us return to the example of a neural network: The agent basis might represent a specific multilayer perceptron, where each element of this basis is an assignment to the network's weights. The learning rules are different mechanisms that choose the next set of weights in

<latexi sh

1_b

64="NOC

U

ym

3V

H+

Q/X

7o

>A

k

n

c

rd

E

fS

T9

v

B

2

G

p

Y

uF

5

8

IM

W

K

P

j

L

0

J

z

w

Z

D

g

q

R

<latexi sh

1_b

64="S2ywr

8qVX

un p+07

dC

N

Y

>A

3

c

f

5

O

L

Hv

k

U

JK

Z

mz

B

P

MD

/

gF

TW

Q

G

9

o

R

I

j

E

<latexi sh

1_b

64="S2ywr

8qVX

un p+07

dC

N

Y

>A

3

c

f

5

O

L

Hv

k

U

JK

Z

mz

B

P

MD

/

gF

TW

Q

G

9

o

R

I

j

E

<latexi sh

1_b

64="S2ywr

8qVX

un p+07

dC

N

Y

>A

3

c

f

5

O

L

Hv

k

U

JK

Z

mz

B

P

MD

/

gF

TW

Q

G

9

o

R

I

j

E

<latexi sh

1_b

64="S2ywr

8qVX

un p+07

dC

N

Y

>A

3

c

f

5

O

L

Hv

k

U

JK

Z

mz

B

P

MD

/

gF

TW

Q

G

9

o

R

I

j

E

'

/u1D456

Λ

B

Λ

/u1D706

1

/u1D706

2

.

.

.

/u1D706

/u1D456

.

.

.

/u1D706

1

/u1D706

2

.

.

.

/u1D706

/u1D456

.

.

.

/u1D706

1

/u1D706

2

.

.

.

/u1D706

/u1D456

.

.

.

/u1D706

1

/u1D706

2

.

.

.

/u1D706

/u1D456

.

.

.

(a) Generates (

GLYPH<3>

B

4

GLYPH<3>

)

/u1D706

1

/u1D706

2

.

.

.

/u1D706

1

/u1D706

2

.

.

.

/u1D706

/u1D456

.

.

.

/u1D706

1

/u1D706

2

.

.

.

/u1D706

/u1D456

.

.

.

(b) Sometimes Reaches (

GLYPH<23>

1

4

GLYPH<3>

B)

/u1D706

response to experience (such as gradient descent). Together, the agent basis and the learning rules generate the set of agents that search over choices of weights in reaction to experience. We present a cartoon visual of the generates operator in Figure 1(a).

Now, using the generates operator, we revisit and formalize the central insight of this section: Every agent can be understood as implicitly searching over an agent basis. We take this implicit search process to be the behavioral signature of learning.

Theorem 3.1. For any agent-environment pair ( GLYPH<23> GLYPH<150> 4 ), there exists infinitely many choices of a basis, GLYPH<3> B , such that both (1) GLYPH<23> 8 GLYPH<3> B , and (2) GLYPH<3> B ' 4 f GLYPH<23> g .

Due to space constraints, all proofs are deferred to Appendix B.

We require that GLYPH<23> 8 GLYPH<3> B to ensure that the relevant bases are non-trivial generators of f GLYPH<23> g . This theorem tells us that no matter the choice of agent or environment, we can view the agent as a series of history-conditioned switches between basis elements. In this sense, we can understand the agent as if 1 it were carrying out a search over the elements of some GLYPH<3> B. We emphasize that there are infinitely many choices of such a basis to illustrate that there are many plausible interpretations of an agent's behavior-we return to this point throughout the paper.

3.2 Operator 2: An Agent Reaches a Basis.

Our second operator reflects properties of an agent's limiting behavior in relation to a basis. Given an agent and a basis that the agent searches through, what happens to the agent's search process in the limit: does the agent keep switching between elements of the basis, or does it eventually stop? For example, in an MDP, many agents of interest eventually stop their search on a choice of a fixed policy. We formally define this notion in terms of an agent reaching a basis according to two modalities: an agent (i) sometimes or (ii) never reaches a basis.

Definition 3.5. We say agent GLYPH<23> 2 GLYPH<3> sometimes reaches GLYPH<3> B in 4 , denoted GLYPH<23> 4 GLYPH<3> B , if and only if

That is, for at least one realizable history, there is some base agent ' GLYPH<23> B ' that produces the same action distribution as GLYPH<23> forever after. This indicates that the agent can be understood as if it has stopped its search over the basis. We present a visual of sometimes reaches in Figure 1(b). By contrast, we say an agent never reaches a basis just when it never becomes equivalent to a base agent.

Definition 3.6. We say agent GLYPH<23> 2 GLYPH<3> never reaches GLYPH<3> B in 4 , denoted GLYPH<23> GLYPH<157> 4 GLYPH<3> B , iff :' GLYPH<23> 4 GLYPH<3> B ' .

<latexi sh

1_b

64="

y

Dvr

B

NGFf

Tz nuU

S

>A

3

c

V

d

k

E

7

0

g

K

q

I

C

M

p

w

O

L

Yo

/Q

+2

9W

H

J

5

8

m

P

X

<latexi sh

1_b

64="S2ywr

8qVX

un p+07

dC

N

Y

>A

3

c

f

5

O

L

Hv

k

U

JK

Z

mz

B

P

MD

/

gF

TW

Q

G

9

o

R

I

j

E

<latexi sh

1_b

64="S2ywr

8qVX

un p+07

dC

N

Y

>A

3

c

f

5

O

L

Hv

k

U

JK

Z

mz

B

P

MD

/

gF

TW

Q

G

9

o

R

I

j

E

<latexi sh

1_b

64="S2ywr

8qVX

un p+07

dC

N

Y

>A

3

c

f

5

O

L

Hv

k

U

JK

Z

mz

B

P

MD

/

gF

TW

Q

G

9

o

R

I

j

E

The reaches operators formalize the intuition that, since every agent can be interpreted as if it were searching over a basis, every agent will either (1) sometimes, or (2) never stop this search. Since (1) and (2) are simply negations of each other, we can now plainly state this fact as follows.

Remark 3.2. For any agent-environment pair ( GLYPH<23> GLYPH<150> 4 ) and any choice of basis GLYPH<3> B such that GLYPH<3> B ' 4 f GLYPH<23> g , exactly one of the following two properties must be satisfied:

Thus, by Theorem 3.1, every agent can be thought of as implicitly searching over an agent basis, and by Remark 3.2, every agent will either (1) sometimes, or (2) never stop this search. We take this implicit search process to be the behavioral signature of learning, and will later exploit this perspective to define a continual learning agent as one that continues its search forever (Definition 4.1). Our analysis in Section 4.4 further elucidates basic properties of both the generates and reaches operators, and Figure 1 presents a cartoon visualizing the intuition behind each of the operators. We summarize all definitions and notation in a table in Appendix A.

Considerations on the Operators. Naturally, we can design many variations of both 4 and ' 4 . For instance, we might be interested in a variant of reaches in which an agent becomes & -close to any of the basis elements, rather than require exact behavioral equivalence. Concretely, we highlight four axes of variation that can modify the definitions of the operators. We state these varieties for reaches, but similar modifications can be made to the generates operator, too:

Realizability. An agent reaches a basis (i) in all histories (and thus, all environments), or (ii) in the histories realizable by a given ' GLYPH<23> GLYPH<150> 4 ' pair.

History Length. An agent reaches a basis over (i) infinite or, (ii) finite length histories.

Probability. An agent reaches a basis (i) with probability one, or (ii) with high probability.

Equality or Approximation. An agent reaches a basis by becoming (i) equivalent to a base agent, or (ii) sufficiently similar to a base agent.

Rather than define all of these variations precisely for both operators (though we do explore some in Appendix C), we acknowledge their existence, and simply note that the formal definitions of these variants follow naturally.

4 Continual Reinforcement Learning

We now provide a precise definition of CRL. The definition formalizes the intuition that CRL captures settings in which the best agents do not converge -they continue their implicit search over an agent basis indefinitely.

4.1 Definition: Continual RL

We first define continual learning agents using the generates and never reaches operators as follows. Definition 4.1. An agent GLYPH<23> is a continual learning agent in 4 relative to GLYPH<3> B if and only if the basis generates the agent ( GLYPH<3> B ' 4 f GLYPH<23> g ) and the agent never reaches the basis ( GLYPH<23> GLYPH<157> 4 GLYPH<3> B ).

This means that an agent is a continual learner in an environment relative to GLYPH<3> B if the agent's search over GLYPH<3> B continues forever. Notice that an agent might be considered a continual learner with respect to one basis but not another; we explore this fact more in Section 4.4.

Then, using these tools, we formally define CRL as follows.

Definition 4.2. Consider an RL problem ' 4GLYPH<150> EGLYPH<150> GLYPH<3> ' . Let GLYPH<3> B GLYPH<26> GLYPH<3> be a basis such that GLYPH<3> B ' 4 GLYPH<3> , and let GLYPH<3> GLYPH<3> = arg max GLYPH<23> 2 GLYPH<3> E ' GLYPH<23> GLYPH<150> 4 ' . We say ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> B ' defines a CRL problem if 8 GLYPH<23> GLYPH<3> 2 GLYPH<3> GLYPH<3> GLYPH<23> GLYPH<3> GLYPH<157> 4 GLYPH<3> B .

Said differently, an RL problem is an instance of CRL just when all of the best agents are continual learning agents relative to basis GLYPH<3> B. This problem encourages a significant departure from how we tend to think about designing agents: Given a basis, rather than try to build agents that can solve problems by identifying a fixed high-quality element of the basis, we would like to design agents that continue to update their behavior indefinitely in light of their experience.

4.2 CRL Examples

We next detail two examples of CRL to provide further intuition.

Q-Learning in Switching MDPs. First we consider a simple instance of CRL based on the standard multi-task view of MDPs. In this setting, the agent repeatedly samples an MDP to interact with from a fixed but unknown distribution [64, 9, 2, 25, 20]. In particular, we make use of the switching MDP environment from Luketina et al. [33]. The environment 4 consists of a collection of = underlying MDPs, < 1 GLYPH<150> GLYPH<149> GLYPH<149> GLYPH<149> GLYPH<150> <= , with a shared action space and environment-state space. We refer to this environment-state space using observations, > 2 O . The environment has a fixed constant positive probability of 0 GLYPH<149> 001 to switch the underlying MDP, which yields different transition and reward functions until the next switch. The agent only observes each environment state > 2 O , which does not reveal the identity of the active MDP. The rewards of each underlying MDP are structured so that each MDP has a unique optimal policy. We assume E is defined as the average reward, and the basis is the set of & -greedy policies over all & ' >GLYPH<150> 0 ' functions, for fixed & = 0 GLYPH<149> 15. Consequently, the set of agents we generate, GLYPH<3> B ' 4 GLYPH<3> , consists of all agents that switch between these & -greedy policies.

Now, the components ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> B ' have been defined, we can see that this is indeed an instance of CRL: None of the base agents can be optimal, as the moment that the environment switches its underlying MDP, we know that any previously optimal policy will no longer be optimal in the next MDP following the switch. Therefore, any agent that converges (in that it reaches the basis GLYPH<3> B) cannot be optimal either for the same reason. We conclude that all optimal agents in GLYPH<3> are continual learning agents relative to the basis GLYPH<3> B.

We present a visual of this domain in Figure 2(a), and conduct a simple experiment contrasting the performance of & -greedy continual Q-learning (blue) that uses a constant step-size parameter of GLYPH<13> = 0 GLYPH<149> 1, with a convergent Q-learning (green) that anneals its step size parameter over time to zero. Both use & = 0 GLYPH<149> 15, and we set the number of underlying MDPs to = = 10. We present the average reward with 95% confidence intervals, averaged over 250 runs, in Figure 2(b). Since both variants of Q-learning can be viewed as searching over GLYPH<3> B, the annealing variant that stops its search will under-perform compared to the continual approach. These results support the unsurprising conclusion that it is better to continue searching over the basis rather than converge in this setting.

Continual Supervised Learning. Second, we illustrate the power of our CRL definition to capture continual supervised learning. We adopt the problem setting studied by Mai et al. [35]. Let X denote a set of objects to be labeled, each belonging to one of : 2 N classes. The observation space O consists of pairs, >C = ' GC GLYPH<150> HC ' , where GC 2 X and HC 2 Y . Here, each GC is an input object to be classified and HC is the label for the previous input GC GLYPH<0> 1. Thus, O = XGLYPH<2> Y . We assume by convention that the initial

/u1D45A

1

/u1D45A

2

/u1D45A

3

/u1D45A

/u1D45B

(a) Switching MDP Visual

Q-learning  (continual)

Q-learning  (anneal)

(b) Switching MDP Results

4.5

4.0

3.5

Average Reward

3.0

2.5

2.0

mmmn

1.5

1.0

0.5

0

5000

10000

15000

20000

25000

Steps

<latexi sh

1_b

64="J

wj

QnB

W

7K

8Y

zFvVd0rfE

>A

c

9

2

+

m

P

OgC

Z

U

L

X

y

I

k

R

T

G

3

/

M

D

H

N

q

u

p

<latexi sh

1_b

64="

Oj

Pu

YJ2W8

A/T

c

>

Q

n

V

9

F

7

+d

m

E

gC

Z

U

B

S5

y

I

k

R

w

G

3

0

r

M

K

fD

H

N

z

q

p

<latexi sh

1_b

64="yqD

g3Zw5

r

Bp

m

v

+

M

>A

Q

n

cjV

9

2F

7W

d

P

E

O

CJ

U

L

X

S

o

I

k

R

T

G

Y0/

K

8f

H

N

z

u

<latexi sh

1_b

64="Z0

rzvX

q

2

D

UQBmHOfA

>

n

cjV

9

F

7W+d

P

E

gCN

k

L

S5

TK

o/

J

w

R

8

y

Y

I

M

G

3

p

u

label H 0 is irrelevant and can be ignored. The agent will observe a sequence of object-label pairs, ' G 0 GLYPH<150> H 0 ' GLYPH<150> ' G 1 GLYPH<150> H 1 ' GLYPH<150> GLYPH<149> GLYPH<149> GLYPH<149> , and the action space is a choice of label, A = f 0 1 GLYPH<150> GLYPH<149> GLYPH<149> GLYPH<149> GLYPH<150> 0: g where jYj = : . The reward for each history GLYPH<17> C is , 1 if the agent's most recently predicted label is correct for the previous input, and GLYPH<0> 1 otherwise:

Concretely, the continual learning setting studied by Mai et al. [35] supposes the learner will receive samples from a sequence of probability distributions, 3 0 GLYPH<150> 3 1 GLYPH<150> GLYPH<149> GLYPH<149> GLYPH<149> , each supported over X GLYPH<2>Y . The ' GGLYPH<150> H ' 2 X GLYPH<2>Y pairs experienced by the learner are determined by the sequence of distributions. We capture this distributional shift in an environment 4 that shifts its probability distribution over O depending on the history to match the sequence, 3 0 GLYPH<150> 3 1 GLYPH<150> GLYPH<149> GLYPH<149> GLYPH<149> .

Now, is this an instance of CRL? To answer this question precisely, we need to select a ' GLYPH<3> GLYPH<150> GLYPH<3> B ' pair. We adopt the basis GLYPH<3> B = f GLYPH<23> B : G 7! H8 GLYPH<150> 8 H8 2Yg that contains each classifier that maps each object to each possible label. By the universal set of learning rules GLYPH<6> , this basis generates the set of all agents that search over classifiers. Now, our definition says the above is an instance of CRL just when every optimal agent never stops switching between classifiers, rather than stop their search on a fixed classifier. Consequently, if there is an optimal classifier in GLYPH<3> B, then this will not be an instance of CRL. If, however, the environment imposes enough distributional shift (changing labels, adding mass to new elements, and so on), then the only optimal agents will be those that always switch among the base classifiers, in which case the setting is an instance of CRL.

4.3 Relationship to Other Views on Continual Learning

The spirit of continual learning has been an important part of machine learning research for decades, often appearing under the name of 'lifelong learning' [63, 62, 53, 55, 51, 3, 4], 'never-ending learning' [39, 43] with close ties to transfer-learning [61, 60], meta-learning [52, 17], as well as online learning and non-stationarity [5, 40, 13, 6, 31]. In a similar vein, the phrase 'continuing tasks' is used in the classic RL textbook [59] to refer explicitly to cases when the interaction between agent and environment is not subdivided into episodes. Continual reinforcement learning was first posed in the thesis by Ring [46]. In later work [47, 48], Ring proposes a formal definition of the continual reinforcement learning problem-The emphasis of Ring's proposal is on the generality of the environment: rather than assume that agents of interest will interact with an MDP, Ring suggests studying the unconstrained case in which an agent must maximize performance while only receiving a stream of observations as input. The environment or reward function, in this sense, may change over time or may be arbitrarily complex. This proposal is similar in spirit to general RL , studied by Hutter [24], Lattimore [28], Leike [29], and others [12, 37, 36] in which an agent interacts with an unconstrained environment. General RL inspires many aspects of our conception of CRL; for instance, our emphasis on history-dependence rather than environment-state comes directly from general RL. More recently, Khetarpal et al. [25] provide a comprehensive survey of the continual reinforcement learning literature. We encourage readers to explore this survey for a detailed history of the subject. 2 In the survey, Khetarpal et al. propose a definition of the CRL problem that emphasizes the non-stationarity of the underlying process. In particular, in Khetarpal et al.'s definition, an agent interacts with a POMDP in which each of the individual components of the POMDP-such as the state space or reward function-are allowed to vary with time. We note that, as the environment model we study (Definition 2.3) is a function of history, it can capture time-indexed non-stationarity. In this sense, the same generality proposed by Khetarpal et al. and Ring is embraced and retained by our definition, but we add further precision to what is meant by continual learning by centering around a mathematical definition of continual learning agents (Definition 4.1).

4.4 Properties of CRL

Our formalism is intended to be a jumping off point for new lines of thinking around agents and continual learning. We defer much of our analysis and proofs to the appendix, and here focus on highlighting necessary properties of CRL.

Theorem 4.1. Every instance of CRL ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> B ' necessarily satisfies the following properties:

If GLYPH<3> < GLYPH<3> B [ GLYPH<3> GLYPH<3> , then there exists a GLYPH<3> 0 B such that (1) GLYPH<3> 0 B ' 4 GLYPH<3> , and (2) ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> 0 B ' is not an instance of CRL.

No element of GLYPH<3> B is optimal: GLYPH<3> B \ GLYPH<3> GLYPH<3> = ; .

If j GLYPH<3> j is infinite, there exists an agent set, GLYPH<3> GLYPH<14> , such that GLYPH<3> GLYPH<14> GLYPH<26> GLYPH<3> and GLYPH<3> GLYPH<14> ' 4 GLYPH<3> .

If j GLYPH<3> j is finite, there exists an agent set, GLYPH<3> GLYPH<14> , such that j GLYPH<3> GLYPH<14> j 5 j GLYPH<3> j and GLYPH<3> GLYPH<14> ' 4 GLYPH<3> .

This theorem tells us several things. The first point of the theorem has peculiar implications. We see that as we change a single element (the basis GLYPH<3> B) of the tuple ' 4GLYPH<150> EGLYPH<150> GLYPH<3> B GLYPH<150> GLYPH<3> ' , the resulting problem can change from CRL to not CRL. By similar reasoning, an agent that is said to be a continual learning agent according to Definition 4.1 may not be a continual learner with respect to some other basis. We discuss this point further in the next paragraph. Point (2.) notes that no optimal strategy exists within the basis-instead, to be optimal, an agent must switch between basis elements indefinitely. As discussed previously, this fact encourages a departure in how we think about the RL problem: rather than focus on agents that can identify a single, fixed solution to a problem, CRL instead emphasizes designing agents that are effective at updating their behavior indefinitely. Points (3.) and (4.) show that GLYPH<3> cannot be minimal . That is, there are necessarily some redundancies in the design space of the agents in CRL-this is expected, since we are always focusing on agents that search over the same agent basis. Lastly, it is worth calling attention to the fact that in the definition of CRL, we assume GLYPH<3> B GLYPH<26> GLYPH<3> -this suggests that in CRL, the agent basis is necessarily limited in some way. Consequently, the design space of agents GLYPH<3> are also limited in terms of what agents they can represent at any particular point in time. This limitation may come about due to a computational or memory budget, or by making use of a constrained set of learning rules. This suggests a deep connection between bounded agents and the nature of continual learning, as explored further by Kumar et al. [27]. While these four points give an initial character of the CRL problem, we note that further exploration of the properties of CRL is an important direction for future work.

Canonical Agent Bases. It is worth pausing and reflecting on the concept of an agent basis. As presented, the basis is an arbitrary choice of a set of agents-consequently, point (1.) of Theorem 4.1 may stand out as peculiar. From this perspective, it is reasonable to ask if the fact that our definition of CRL is basis-dependant renders it vacuous. We argue that this is not the case for two reasons. First, we conjecture that any definition of continual learning that involves concepts like 'learning' and 'convergence' will have to sit on top of some reference object whose choice is arbitrary. Second, and more important, even though the mathematical construction allows for an easy change of basis, in practice the choice of basis is constrained by considerations like the availability of computational resources. It is often the case that the domain or problem of interest provides obvious choices of bases, or imposes constraints that force us as designers to restrict attention to a space of plausible bases or learning rules. For example, as discussed earlier, a choice of neural network architecture might comprise a basis-any assignment of weights is an element of the basis, and the learning rule GLYPH<30> is a mechanism for updating the active element of the basis (the parameters) in light of experience. In this case, the number of parameters of the network is constrained by what we can actually build, and the learning rule needs to be suitably efficient and well-behaved. We might again think of the learning rule GLYPH<30> as gradient descent, rather than a rule that can search through the basis in an unconstrained way. In this sense, the basis is not arbitrary . We as designers choose a class of functions to act as the relevant representations of behavior, often limited by resource constraints on memory or compute. Then, we use specific learning rules that have been carefully designed to react to experience in a desirable way-for instance, stochastic gradient descent updates the current choice of basis in the direction that would most improve performance. For these reasons, the choice of basis is not arbitrary, but instead reflects the ingredients involved in the design of agents as well as the constraints necessarily imposed by the environment.

4.5 Properties of Generates and Reaches

Lastly, we summarize some of the basic properties of generates and reaches. Further analysis of generates, reaches, and their variations is provided in Appendix C.

Theorem 4.2. The following properties hold of the generates operator:

Generates is transitive: For any triple ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 GLYPH<150> GLYPH<3> 3 ' and 4 2 E , if GLYPH<3> 1 ' 4 GLYPH<3> 2 and GLYPH<3> 2 ' 4 GLYPH<3> 3 , then GLYPH<3> 1 ' 4 GLYPH<3> 3 .

For all GLYPH<3> and pair of agent bases ' GLYPH<3> 1 B GLYPH<150> GLYPH<3> 2 B ' such that GLYPH<3> 1 B GLYPH<18> GLYPH<3> 2 B , if GLYPH<3> 1 B ' 4 GLYPH<3> , then GLYPH<3> 2 B ' 4 GLYPH<3> .

Generates is not commutative: there exists a pair ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 ' and 4 2 E such that GLYPH<3> 1 ' 4 GLYPH<3> 2 , but :' GLYPH<3> 2 ' 4 GLYPH<3> 1 ' .

For all GLYPH<3> and 4 2 E , GLYPH<3> ' 4 GLYPH<3> .

The decision problem, Given ' 4GLYPH<150> GLYPH<3> B GLYPH<150> GLYPH<3> ' , output True iff GLYPH<3> B ' 4 GLYPH<3> , is undecidable.

The fact that generates is transitive suggests that the basic tools of an agent set-paired with a set of learning rules-might be likened to an algebraic structure. We can draw a symmetry between an agent basis and the basis of a vector space: A vector space is comprised of all linear combinations of the basis, whereas GLYPH<3> is comprised of all valid switches (according to the learning rules) between the base agents. However, the fact that generates is not commutative (by point 2.) raises a natural question: are there choices of learning rules under which generates is commutative? We suggest that a useful direction for future work can further explore an algebraic perspective on agents.

We find many similar properties hold of reaches.

Theorem 4.3. The following properties hold of the reaches operator:

4 and GLYPH<157> 4 are not transitive.

For all pairs ' GLYPH<3> GLYPH<150> 4 ' , if GLYPH<23> 2 GLYPH<3> , then GLYPH<23> 4 GLYPH<3> .

'Sometimes reaches' is not commutative: there exists a pair ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 ' and 4 2 E such that 8 GLYPH<23> 1 2 GLYPH<3> 1 GLYPH<23> 1 4 GLYPH<3> 2 , but 9 GLYPH<23> 2 2 GLYPH<3> 2 GLYPH<23> 2 GLYPH<157> 4 GLYPH<3> 1 .

Every agent satisfies GLYPH<23> 4 GLYPH<3> in every environment.

The decision problem, Given ' 4GLYPH<150> GLYPH<23> GLYPH<150> GLYPH<3> ' , output True iff GLYPH<23> 4 GLYPH<3> , is undecidable.

Many of these properties resemble those in Theorem 4.2. For instance, point (5.) shows that deciding whether a given agent sometimes reaches a basis in an environment is undecidable. We anticipate that the majority of decision problems related to determining properties of arbitrary agent sets interacting with unconstrained environments will be undecidable, though it is still worth making these arguments carefully. Moreover, there may be interesting special cases in which these decision problems are decidable (and perhaps, efficiently so). We suggest that identifying these special cases and fleshing out their corresponding efficient algorithms is an interesting direction for future work.

5 Discussion

In this paper, we carefully develop a simple mathematical definition of the continual RL problem. We take this problem to be of central importance to AI as a field, and hope that these tools and perspectives can serve as an opportunity to think about CRL and its related artifacts more carefully. Our proposal is framed around two new insights about agents: (i) every agent can be understood as though it were searching over an agent basis (Theorem 3.1), and (ii) every agent, in the limit, will either sometimes or never stop this search (Remark 3.2). These two insights are formalized through the generates and reaches operators, which provide a rich toolkit for understanding agents in a new way-for example, we find straightforward definitions of a continual learning agent (Definition 4.1) and learning rules (Definition 3.2). We anticipate that further study of these operators and different families of learning rules can directly inform the design of new learning algorithms; for instance, we might characterize the family of continual learning rules that are guaranteed to yield continual learning agents, and use this to guide the design of principled continual learning agents (in the spirit of continual backprop by Dohare et al. [14]). In future work, we intend to further explore connections between our formalism of continual learning and some of the phenomena at the heart of recent empirical continual learning studies, such as plasticity loss [34, 1, 15], in-context learning [8], and catastrophic forgetting [38, 18, 21, 26]. More generally, we hope that our definitions, analysis, and perspectives can help the community to think about continual reinforcement learning in a new light.

References

Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507 , 2023.

David Abel, Yuu Jinnai, Yue Guo, George Konidaris, and Michael L. Littman. Policy and value transfer in lifelong reinforcement learning. In Proceedings of the International Conference on Machine Learning , 2018.

Haitham Bou Ammar, Rasul Tutunov, and Eric Eaton. Safe policy search for lifelong reinforcement learning with sublinear regret. In Proceedings of the International Conference on Machine Learning , 2015.

Megan M Baker, Alexander New, Mario Aguilar-Simon, Ziad Al-Halah, S´ ebastien MR Arnold, Ese Ben-Iwhiwhu, Andrew P Brna, Ethan Brooks, Ryan C Brown, Zachary Daniels, et al. A domain-agnostic approach for characterization of lifelong learning systems. Neural Networks , 160:274-296, 2023.

Peter L Bartlett. Learning with a slowly changing distribution. In Proceedings of the Annual Workshop on Computational Learning Theory , 1992.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. Advances in Neural Information Processing Systems , 2014.

Michael Bowling, John D. Martin, David Abel, and Will Dabney. Settling the reward hypothesis. In Proceedings of the International Conference on Machine Learning , 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 2020.

Emma Brunskill and Lihong Li. PAC-inspired option discovery in lifelong reinforcement learning. In Proceedings of the International Conference on Machine Learning , 2014.

Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka, and Tom Mitchell. Toward an architecture for never-ending language learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2010.

Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the AAAI Conference on Artificiall Intelligence , 1994.

Michael K Cohen, Elliot Catt, and Marcus Hutter. A strongly asymptotically optimal agent in general environments. arXiv preprint arXiv:1903.01021 , 2019.

Travis Dick, Andr´ as Gy¨ orgy, and Csaba Szepesvari. Online learning in Markov decision processes with changing cost sequences. In Procedings of the International Conference on Machine Learning , 2014.

Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325 , 2021.

Shibhansh Dohare, Juan Hernandez-Garcia, Parash Rahman, Richard Sutton, and A Rupam Mahmood. Loss of plasticity in deep continual learning. arXiv preprint arXiv:2306.13812 , 2023.

Shi Dong, Benjamin Van Roy, and Zhengyuan Zhou. Simple agent, complex environment: Efficient reinforcement learning with agent states. Journal of Machine Learning Research , 23 (255):1-54, 2022.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning , 2017.

Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences , 3(4):128-135, 1999.

Milton Friedman. Essays in positive economics . University of Chicago press, 1953.

Haotian Fu, Shangqun Yu, Michael Littman, and George Konidaris. Model-based lifelong reinforcement learning with Bayesian exploration. Advances in Neural Information Processing Systems , 2022.

Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 , 2013.

Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences , 24(12):1028-1040, 2020.

Marcus Hutter. A theory of universal artificial intelligence based on algorithmic complexity. arXiv preprint cs/0004001 , 2000.

Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probability . Springer Science & Business Media, 2004.

Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research , 75: 1401-1476, 2022.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114(13):3521-3526, 2017.

Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, and Benjamin Van Roy. Continual learning as computationally constrained reinforcement learning. arXiv preprint arXiv:2307.04345 , 2023.

Tor Lattimore. Theory of general reinforcement learning . PhD thesis, The Australian National University, 2014.

Jan Leike. Nonparametric general reinforcement learning . PhD thesis, The Australian National University, 2016.

Timoth´ ee Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia D´ ıaz-Rodr´ ıguez. Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information fusion , 58:52-68, 2020.

Yueyang Liu, Benjamin Van Roy, and Kuang Xu. A definition of non-stationary bandits. arXiv preprint arXiv:2302.12202 , 2023.

Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, and Zheng Wen. Reinforcement learning, bit by bit. Foundations and Trends in Machine Learning , 16(6):733-865, 2023. ISSN 1935-8237.

Jelena Luketina, Sebastian Flennerhag, Yannick Schroecker, David Abel, Tom Zahavy, and Satinder Singh. Meta-gradients in non-stationary environments. In Proceedings of the Conference on Lifelong Learning Agents , 2022.

Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Proceedings of the International Conference on Machine Learning , 2023.

Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing , 469:28-51, 2022.

Sultan J Majeed. Abstractions of general reinforcement Learning . PhD thesis, The Australian National University, 2021.

Sultan Javed Majeed and Marcus Hutter. Performance guarantees for homomorphisms beyond Markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence , 2019.

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. Elsevier, 1989.

Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, et al. Never-ending learning. Communications of the ACM , 61(5):103-115, 2018.

Claire Monteleoni and Tommi Jaakkola. Online learning of non-stationary sequences. Advances in Neural Information Processing Systems , 16, 2003.

Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. 2018.

German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks , 113:54-71, 2019.

Emmanouil Antonios Platanios, Abulhair Saparov, and Tom Mitchell. Jelly bean world: A testbed for never-ending learning. arXiv preprint arXiv:2002.06306 , 2020.

Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, 2014.

Matthew Riemer, Sharath Chandra Raparthy, Ignacio Cases, Gopeshh Subbaraj, Maximilian Puelma Touzel, and Irina Rish. Continual learning in environments with polynomial mixing times. Advances in Neural Information Processing Systems , 2022.

Mark B Ring. Continual learning in reinforcement environments . PhD thesis, The University of Texas at Austin, 1994.

Mark B Ring. Child: A first step towards continual learning. Machine Learning , 28(1):77-104, 1997.

Mark B Ring. Toward a formal framework for continual learning. In NeurIPS Workshop on Inductive Transfer , 2005.

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems , 2019.

Stuart J Russell and Devika Subramanian. Provably bounded-optimal agents. Journal of Artificial Intelligence Research , 2:575-609, 1994.

Paul Ruvolo and Eric Eaton. ELLA: An efficient lifelong learning algorithm. In Proceedings of the International Conference on Machine Learning , 2013.

Tom Schaul and J¨ urgen Schmidhuber. Metalearning. Scholarpedia , 5(6):4650, 2010.

J¨ urgen Schmidhuber, Jieyu Zhao, and Nicol N Schraudolph. Reinforcement learning with self-modifying policies. In Learning to Learn , pages 293-309. Springer, 1998.

Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Proceedings of the International Conference on Machine Learning , 2018.

Daniel L Silver. Machine lifelong learning: Challenges and benefits for artificial general intelligence. In Proceedings of the Conference on Artificial General Intelligence , 2011.

Richard S Sutton. Introduction: The challenge of reinforcement learning. In Reinforcement Learning , pages 1-3. Springer, 1992.

Richard S Sutton. The reward hypothesis, 2004. URL http://incompleteideas.net/rlai. cs.ualberta.ca/RLAI/rewardhypothesis.html .

Richard S Sutton. The quest for a common model of the intelligent decision maker. arXiv preprint arXiv:2202.13252 , 2022.

Richard S Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . MIT Press, 2018.

Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(Jul):1633-1685, 2009.

Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Advances in Neural Information Processing Systems , 1995.

Sebastian Thrun. Lifelong learning algorithms. Learning to Learn , 8:181-209, 1998.

Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems , 15(1-2):25-46, 1995.

Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical Bayesian approach. In Proceedings of the International Conference on Machine learning , 2007.

A Notation

We first provide a table summarizing all relevant notation.

B Proofs of Presented Results

We next provide proofs of each result from the paper. Our proofs make use of some extra notation: we use ) as logical implication, and we use GLYPH<15> 'X' to denote the power set of any set X . Lastly, we use 8AGLYPH<18>X and 9AGLYPH<18>X as shorthand for 8A2 GLYPH<15> 'X' and 9A2 GLYPH<15> 'X' respectively.

B.1 Section 3 Proofs

Our first result is from Section 3 of the paper.

Theorem 3.1. For any pair ( GLYPH<23> GLYPH<150> 4 ), there exists infinitely many choices of a basis, GLYPH<3> B , such that both (1) GLYPH<23> 8 GLYPH<3> B , and (2) GLYPH<3> B ' 4 f GLYPH<23> g .

Proof of Theorem 3.1.

Choose a fixed but arbitrary pair ' GLYPH<23> GLYPH<150> 4 ' . Then, enumerate the realizable histories, H GLYPH<23> GLYPH<150>4 , and let GLYPH<17> 1 denote the first element of this enumeration, GLYPH<17> 2 the second, and so on.

Then, we design a constructive procedure for a basis that, when repeatedly applied, induces an infinite enumeration of bases that satisfy the desired two properties. This constructive procedure for the : -th basis will contain : , 1 agents, where each agent is distinct from GLYPH<23> , but will produces the same action as the agent every : , 1 elements of the history sequence, GLYPH<17> 1 GLYPH<150> GLYPH<17> 2 GLYPH<150> GLYPH<149> GLYPH<149> GLYPH<149> .

For the first ( : = 1) basis, we construct two agents. The first, GLYPH<23> 1 B , chooses the same action distribution as GLYPH<23> on each even numbered history: GLYPH<23> 1 B ' GLYPH<17> 8 ' = GLYPH<23> ' GLYPH<17> 8 ' . Then, this agent will choose a different action distribution on the odd length histories: GLYPH<23> 1 B ' GLYPH<17> 8 , 1 ' < GLYPH<23> ' GLYPH<17> 8 , 1 ' , for 8 any even natural number. The second agent, GLYPH<23> 2 B will do the opposite to GLYPH<23> 1 B : on each odd numbered history GLYPH<17> 8 , 1 , GLYPH<23> 2 B ' GLYPH<17> 8 , 1 ' < GLYPH<23> ' GLYPH<17> 8 , 1 ' , but on every even numbered history, GLYPH<23> 2 B ' GLYPH<17> 8 ' = GLYPH<23> ' GLYPH<17> 8 ' .

Observe first that by construction, GLYPH<23> < GLYPH<23> 1 B , and GLYPH<23> < GLYPH<23> 2 B , since there exist histories where they choose different action distributions. Next, observe that the basis, GLYPH<3> B = f GLYPH<23> 1 B GLYPH<150> GLYPH<23> 2 B g , generates f GLYPH<23> g in 4 through the following set of learning rules, GLYPH<6> : given any realizable history, GLYPH<17> 2 H GLYPH<23> GLYPH<150>4 , check whether the history has an even or odd numbered index in the enumeration. If odd, choose GLYPH<23> 1 B , and if even, choose GLYPH<23> 2 B .

More generally, this procedure can be applied for any : :

where we use the notation » GLYPH<17> … == 8 to express the logical predicate asserting that the modulos of the index of GLYPH<17> in the enumeration GLYPH<17> 1 GLYPH<150> GLYPH<17> 2 GLYPH<150> GLYPH<149> GLYPH<149> GLYPH<149> is equal to 8 .

Further, < GLYPH<23> ' GLYPH<17> ' simply refers to any choice of action distribution that is unequal to GLYPH<23> ' GLYPH<17> ' . Thus, for all natural numbers : GLYPH<21> 2, we can construct a new basis consisting of : base agents that generates GLYPH<23> in 4 , but does not contain the agent itself. This completes the argument. GLYPH<3>

B.2 Section 4 Proofs

We next present the proofs of results from Section 4.

B.2.1 Theorem 4.1: Properties of CRL

We begin with Theorem 4.1 that establishes basic properties of CRL.

Theorem 4.1. Every instance of CRL ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> B ' satisfies the following properties:

If GLYPH<3> < GLYPH<3> B [ GLYPH<3> GLYPH<3> , there exists a GLYPH<3> 0 B such that (1) GLYPH<3> 0 B ' 4 GLYPH<3> , and (2) ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> 0 B ' is not an instance of CRL.

No element of GLYPH<3> B is optimal: GLYPH<3> B \ GLYPH<3> GLYPH<3> = ; .

If j GLYPH<3> j is infinite, there exists an agent set, GLYPH<3> GLYPH<14> such that GLYPH<3> GLYPH<14> GLYPH<26> GLYPH<3> and GLYPH<3> GLYPH<14> ' 4 GLYPH<3> .

If j GLYPH<3> j is finite, there exists an agent set, GLYPH<3> GLYPH<14> such that j GLYPH<3> GLYPH<14> j 5 j GLYPH<3> j and GLYPH<3> GLYPH<14> ' 4 GLYPH<3> .

We prove this result in the form of three lemmas, corresponding to each of the four points of the theorem (with the third lemma, Lemma B.3, covering both points 3. and 4.). Some of the lemmas make use of properties of generates and reaches that we establish later in Appendix C.

Lemma B.1. For all instances of CRL ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> B ' , if GLYPH<3> < GLYPH<3> B [ GLYPH<3> GLYPH<3> , then there exists a choice GLYPH<3> 0 B such that (1) GLYPH<3> 0 B ' 4 GLYPH<3> , and (2) ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> 0 B ' is not an instance of CRL.

Proof of Lemma B.1.

Recall that a tuple ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> B ' is CRL just when all of the optimal agents GLYPH<3> GLYPH<3> do not reach the basis. Then, the result holds as a straightforward consequence of two facts. First, we can always construct a new basis containing all of the optimal agents, GLYPH<3> GLYPH<14> B = GLYPH<3> B [ GLYPH<3> GLYPH<3> . Notice that GLYPH<3> GLYPH<14> B still generates GLYPH<3> by property three of Theorem 4.2. Further, since both GLYPH<3> B and GLYPH<3> GLYPH<3> are each subsets of GLYPH<3> , and by assumption GLYPH<3> < GLYPH<3> B [ GLYPH<3> GLYPH<3> (so there is at least one sub-optimal agent that is not in the basis), it follows that GLYPH<3> GLYPH<14> B GLYPH<26> GLYPH<3> . Second, by Proposition C.15, we know that every element GLYPH<23> GLYPH<14> B 2 GLYPH<3> GLYPH<14> B will always reach the basis, GLYPH<23> GLYPH<14> B GLYPH<3> 4 GLYPH<3> GLYPH<14> B . Therefore, in the tuple ' 4GLYPH<150> EGLYPH<150> GLYPH<3> GLYPH<150> GLYPH<3> GLYPH<14> B ' , each of the optimal agents will reach the basis, and therefore this is not an instance of CRL. GLYPH<3>

Lemma B.2. No element of GLYPH<3> B is optimal: GLYPH<3> B \ GLYPH<3> GLYPH<3> = ; .

Proof of Lemma B.2.

The lemma follows as a combination of two facts.

First, recall that, by definition of CRL, each optimal agent GLYPH<23> 2 GLYPH<3> GLYPH<3> satisfies GLYPH<23> GLYPH<3> GLYPH<157> 4 GLYPH<3> B.

Therefore, since sometimes reaches ( 4 ) and never reaches ( GLYPH<157> 4 ) are negations of one another, we conclude that no basis element can be optimal. GLYPH<3>

Second, note that by Lemma B.11, we know that each GLYPH<23> B 2 GLYPH<3> B satisfies GLYPH<23> B 4 GLYPH<3> B.

Before stating the next lemma, we note that points (3.) and (4.) of Theorem 4.1 are simply expansions of the definition of a minimal agent set, which we define precisely in Definition C.4 and Definition C.5.

Lemma B.3. For any instance of CRL, GLYPH<3> is not minimal.

Proof of Lemma B.3.

We first show that GLYPH<3> cannot be minimal. To do so, we consider the cases where the rank (Definition C.3) of GLYPH<3> is finite and infinite separately.

(Finite Rank GLYPH<3> .)

If rank ' GLYPH<3> ' is finite and minimal, then it follows immediately that there is no agent set of smaller rank that generates GLYPH<3> . By consequence, since GLYPH<3> B GLYPH<26> GLYPH<3> and GLYPH<3> B ' 4 GLYPH<3> , we conclude that GLYPH<3> cannot be minimal. X

(Infinite Rank GLYPH<3> .)

If rank ' GLYPH<3> ' is infinite and minimal, then there is no proper subset of GLYPH<3> that uniformly generates GLYPH<3> by definition. By consequence, since GLYPH<3> B GLYPH<26> GLYPH<3> and GLYPH<3> B ' 4 GLYPH<3> , we conclude that GLYPH<3> cannot be minimal. X

This completes the argument of both cases, and we conclude that for any instance of CRL, GLYPH<3> is not minimal. GLYPH<3>

B.2.2 Theorem 4.2: Properties of Generates

Next, we prove basic properties of generates.

Theorem 4.2. The following properties hold of the generates operator:

Generates is transitive: For any triple ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 GLYPH<150> GLYPH<3> 3 ' and 4 2 E , if GLYPH<3> 1 ' 4 GLYPH<3> 2 and GLYPH<3> 2 ' 4 GLYPH<3> 3 , then GLYPH<3> 1 ' 4 GLYPH<3> 3 .

For all GLYPH<3> and pair of agent bases ' GLYPH<3> 1 B GLYPH<150> GLYPH<3> 2 B ' such that GLYPH<3> 1 B GLYPH<18> GLYPH<3> 2 B , if GLYPH<3> 1 B ' 4 GLYPH<3> , then GLYPH<3> 2 B ' 4 GLYPH<3> .

Generates is not commutative: there exists a pair ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 ' and 4 2 E such that GLYPH<3> 1 ' 4 GLYPH<3> 2 , but :' GLYPH<3> 2 ' 4 GLYPH<3> 1 ' .

For all GLYPH<3> and 4 2 E , GLYPH<3> ' 4 GLYPH<3> .

The decision problem, Given ' 4GLYPH<150> GLYPH<3> B GLYPH<150> GLYPH<3> ' , output True iff GLYPH<3> B ' 4 GLYPH<3> , is undecidable.

The proof of this theorem is spread across the next five lemmas below.

The fact that generates is transitive suggests that the basic tools of an agent set-paired with a set of learning rules-might be likened to an algebraic structure. We can draw a symmetry between an agent basis and the basis of a vector space: A vector space is comprised of all linear combinations of the basis, whereas GLYPH<3> is comprised of all valid switches (according to the learning rules) between the base agents. However, the fact that generates is not commutative (by point 2.) raises a natural question: are there choices of learning rules under which generates is commutative? An interesting direction for future work is to explore this style of algebraic analysis on agents.

Lemma B.4. Generates is transitive: For any triple ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 GLYPH<150> GLYPH<3> 3 ' and 4 2 E , if GLYPH<3> 1 ' 4 GLYPH<3> 2 and GLYPH<3> 2 ' 4 GLYPH<3> 3 , then GLYPH<3> 1 ' 4 GLYPH<3> 3 .

Proof of Lemma B.4.

Assume GLYPH<3> 1 ' 4 GLYPH<3> 2 and GLYPH<3> 2 ' 4 GLYPH<3> 3 . Then, by Proposition C.4 and the definition of the generates operator, we know that

where GLYPH<6> 1 and GLYPH<6> 2 express the set of all learning rules over GLYPH<3> 1 and GLYPH<3> 2 respectively. By definition of a learning rule, GLYPH<30> , we rewrite the above as follows,

Then, consider a fixed but arbitrary GLYPH<23> 3 2 GLYPH<3> 3 . We construct a learning rule defined over GLYPH<3> 1 as GLYPH<30> 1 : H ! GLYPH<3> 1 that induces an equivalent agent as follows. For each realizable history, GLYPH<17> 2 ¢ H , by Equation B.5 we know that there is an GLYPH<23> 2 such that GLYPH<23> 3 ' GLYPH<17> ' = GLYPH<23> 2 ' GLYPH<17> ' , and by Equation B.4, there is an GLYPH<23> 1 such that GLYPH<23> 2 ' GLYPH<17> ' = GLYPH<23> 1 ' GLYPH<17> ' . Then, set GLYPH<30> 1 : GLYPH<17> 7! GLYPH<23> 1 such that GLYPH<23> 1 ' GLYPH<17> ' = GLYPH<23> 2 ' GLYPH<17> ' = GLYPH<23> 3 ' GLYPH<17> '

Since GLYPH<17> and GLYPH<23> 3 were chosen arbitrarily, we conclude that

But, by the definition of GLYPH<6> , this means there exists a learning rule such that

This is exactly the definition of GLYPH<6> -generation, and by Proposition C.4, we conclude GLYPH<3> 1 ' 4 GLYPH<3> 3 . GLYPH<3>

Lemma B.5. Generates is not commutative: there exists a pair ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 ' and 4 2 E such that GLYPH<3> 1 ' 4 GLYPH<3> 2 , but :' GLYPH<3> 2 ' 4 GLYPH<3> 1 ' .

Proof of Lemma B.5.

The result follows from a simple counterexample: consider the pair

Note that since GLYPH<23> 8 is in both sets, and GLYPH<3> 1 is a singleton, we know that GLYPH<3> 2 ' 4 GLYPH<3> 1 in any environment. But, by Proposition C.6, we know that GLYPH<3> 1 cannot generate GLYPH<3> 2 . GLYPH<3>

Lemma B.6. For all GLYPH<3> and pair of agent bases ' GLYPH<3> 1 B GLYPH<150> GLYPH<3> 2 B ' such that GLYPH<3> 1 B GLYPH<18> GLYPH<3> 2 B , if GLYPH<3> 1 B ' 4 GLYPH<3> , then GLYPH<3> 2 B ' 4 GLYPH<3> .

Proof of Lemma B.6.

The result follows as a natural consequence of the definition of generates. Recall that GLYPH<3> 1 B ' 4 GLYPH<3> just when,

where again GLYPH<23> GLYPH<30> 1 ' GLYPH<17> ' B 2 GLYPH<3> 1 B is the base agent chosen by GLYPH<30> 1 ' GLYPH<17> ' . We use superscripts GLYPH<6> 1 and GLYPH<30> 1 to signify that GLYPH<30> 1 is defined relative to GLYPH<3> 1 B , that is, GLYPH<30> 1 : H ! GLYPH<3> 1 B 2 GLYPH<6> 1 .

But, since GLYPH<3> 1 B GLYPH<18> GLYPH<3> 2 B , we can define GLYPH<6> 2 = GLYPH<6> 1 and ensure that GLYPH<3> 2 B ' 4 GLYPH<6> 2 GLYPH<3> , since the agent basis GLYPH<3> 1 B was already sufficient to generate GLYPH<3> . Therefore, we conclude that GLYPH<3> 2 B ' 4 GLYPH<3> . GLYPH<3>

Lemma B.7. For all GLYPH<3> and 4 2 E , GLYPH<3> ' 4 GLYPH<3> .

Proof of Lemma B.7.

This is a direct consequence of Proposition C.18.

GLYPH<3>

Lemma B.8. The decision problem, AGENTSGENERATE , Given ' 4GLYPH<150> GLYPH<3> B GLYPH<150> GLYPH<3> ' , output True iff GLYPH<3> B ' 4 GLYPH<3> , is undecidable.

Proof of Lemma B.8.

We proceed as is typical of such results by reducing AGENTSGENERATE from the Halting Problem.

In particular, let < be a fixed but arbitrary Turing Machine, and F be a fixed but arbitrary input to be given to machine < . Then, HALT defines the decision problem that outputs True iff < halts on input F .

We construct an oracle for AGENTSGENERATE that can decide HALT as follows. Let 'A GLYPH<150> O' be an interface where the observation space is comprised of all configurations of machine < . Then, we consider a deterministic environment 4 that simply produces the next configuration of < when run on input F , based on the current tape contents, the state of < , and the location of the tape head. Note that all three of these elements are contained in a Turing Machine's configuration, and that a single configuration indicates whether the Turing Machine is in a halting state or not. Now, let the action space A consist of two actions, f 0 no-op GLYPH<150> 0 halt g . On execution of 0 no-op no-op, the environment moves to the next configuration. On execution of 0 halt, the machine halts. That is, we restrict ourselves to the singleton agent set, GLYPH<3> , containing

the agent GLYPH<23> GLYPH<14> that outputs 0 halt directly following the machine entering a halting configuration, and 0 no-op otherwise:

Using these ingredients, we take any instance of HALT, ' <GLYPH<150>F ' , and consider the singleton agent basis: GLYPH<3> 1 B = f 0 no-op g .

We make one query to our AGENTSGENERATE oracle, and ask: GLYPH<3> 1 B ' 4 GLYPH<3> . If it is True, then the histories realizable by ' GLYPH<23> GLYPH<14> GLYPH<150> 4 ' pair ensure that the single agent in GLYPH<3> never emits the 0 halt action, and thus, < does not halt on F . If it is False, then there are realizable histories in 4 in which < halts on F . We thus use the oracle's response directly to decide the given instance of HALT. GLYPH<3>

B.2.3 Theorem 4.3: Properties of Reaches

We find many similar properties hold for reaches.

Theorem 4.3. The following properties hold of the reaches operator:

4 and GLYPH<157> 4 are not transitive.

For all pairs ' GLYPH<3> GLYPH<150> 4 ' , if GLYPH<23> 2 GLYPH<3> , then GLYPH<23> 4 GLYPH<3> .

Sometimes reaches is not commutative: there exists a pair ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 ' and 4 2 E such that 8 GLYPH<23> 1 2 GLYPH<3> 1 GLYPH<23> 1 4 GLYPH<3> 2 , but 9 GLYPH<23> 2 2 GLYPH<3> 2 GLYPH<23> 2 GLYPH<157> 4 GLYPH<3> 1 .

Every agent satisfies GLYPH<23> 4 GLYPH<3> in every environment.

The decision problem, Given ' 4GLYPH<150> GLYPH<23> GLYPH<150> GLYPH<3> ' , output True iff GLYPH<23> 4 GLYPH<3> , is undecidable.

Again, we prove this result through five lemmas that correspond to each of the above properties.

Many of these properties resemble those in Theorem 4.2. For instance, point (5.) shows that deciding whether a given agent sometimes reaches a basis in an environment is undecidable. We anticipate that the majority of decision problems related to determining properties of arbitrary agent sets will be undecidable, though it is still worth making these arguments carefully. Moreover, there may be interesting special cases in which these decision problems are decidable (and perhaps, efficiently so). Identifying these special cases and their corresponding efficient algorithms is another interesting direction for future work.

Lemma B.9. 4 and GLYPH<157> 4 are not transitive.

Proof of Lemma B.9.

We construct two counterexamples, one for each of 'sometimes reaches' ( 4 ' and 'never reaches' ( GLYPH<157> 4 ).

Counterexample: Sometimes Reaches. To do so, we begin with a tuple ' 4GLYPH<150> GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 GLYPH<150> GLYPH<3> 3 ' such that both

We will show that there is an agent, GLYPH<23> 1 2 GLYPH<3> 1 , such that GLYPH<23> 1 GLYPH<157> 4 GLYPH<3> 3 , thus illustrating that sometimes reaches is not guaranteed to be transitive. The basic idea is that sometimes reaches only requires an agent stop its search on one realizable history. So, GLYPH<23> 1 4 GLYPH<3> 2 might happen on some history GLYPH<17> , but each GLYPH<23> 2 2 GLYPH<3> 2 might only reach GLYPH<3> 3 on an entirely different history. As a result, reaching GLYPH<3> 2 is not enough to ensure the agent also reaches GLYPH<3> 3 .

In more detail, the agent sets of the counterexample are as follows. Let A = f 0 1 GLYPH<150> 0 2 g and O = f > 1 GLYPH<150> > 2 g . Let GLYPH<3> 2 be all agents that, after ten timesteps, always take 0 2. GLYPH<23> 1 is simple: it

always takes 0 1, except on one realizable history, GLYPH<17> GLYPH<14> , (and all of the realizable successors of GLYPH<17> GLYPH<14> , GLYPH<23> GLYPH<150>4 ), where it switches to taking 0 2 after ten timesteps. Clearly GLYPH<23> 1 4 GLYPH<3> 2 , since

H GLYPH<17> GLYPH<14> after ten timesteps, we know there will be some GLYPH<23> 2 such that GLYPH<23> 1 ' GLYPH<17> GLYPH<14> GLYPH<17> 0 ' = GLYPH<23> 2 ' GLYPH<17> GLYPH<14> GLYPH<17> 0 ' for all realizable history suffixes GLYPH<17> 0 . Now, by assumption, we know that GLYPH<23> 2 4 GLYPH<3> 3 . This ensures there is a single realizable history GLYPH<17> such that there is an GLYPH<23> 3 where GLYPH<23> 2 ' GLYPH<17> GLYPH<17> 0 ' = GLYPH<23> 3 ' GLYPH<17> GLYPH<17> 0 ' for any realizable suffix GLYPH<17> 0 . To finish the counterexample, we simply note that this realizable GLYPH<17> can be different from GLYPH<17> GLYPH<14> and all of its successors. For example, GLYPH<17> GLYPH<14> might be the history containing only > 1 for the first ten timesteps, while GLYPH<17> could be the history containing only > 2 for the first ten timesteps. Thus, this GLYPH<23> 1 never reaches GLYPH<3> 3 , and we conclude the counterexample. X

Counterexample: Never Reaches. The instance for never reaches is simple: Let A = f 0 1 GLYPH<150> 0 2 GLYPH<150> 0 3 g , and GLYPH<3> 1 = GLYPH<3> 3 . Suppose all agents in GLYPH<3> 1 (and thus GLYPH<3> 3 ) only choose actions 0 1 and 0 3. Let GLYPH<3> 2 be a singleton, GLYPH<3> 2 = f GLYPH<23> 2 g such that GLYPH<23> 2 : GLYPH<17> 7! 0 2. Clearly, every GLYPH<23> 1 2 GLYPH<3> 1 will never reach GLYPH<3> 2 , since none of them ever choose 0 2. Similarly, GLYPH<23> 2 will never reach GLYPH<3> 3 , since no agents in GLYPH<3> 3 choose 0 2. However, by Proposition C.15 and the assumption that GLYPH<3> 1 = GLYPH<3> 3 , we know 8 GLYPH<23> 1 2 GLYPH<3> 1 GLYPH<23> 1 GLYPH<3> 4 GLYPH<3> 3 . This directly violates transitivity. X

This completes the argument for all three cases, and we conclude. GLYPH<3>

Lemma B.10. Sometimes reaches is not commutative: there exists a pair ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 ' and 4 2 E such that 8 GLYPH<23> 1 2 GLYPH<3> 1 GLYPH<23> 1 4 GLYPH<3> 2 , but 9 GLYPH<23> 2 2 GLYPH<3> 2 GLYPH<23> 2 GLYPH<157> 4 GLYPH<3> 1 .

Proof of Lemma B.10.

The result holds as a straightforward consequence of the following counterexample. Consider the pair of agent sets

Note that since GLYPH<23> 8 is in both sets, and GLYPH<3> 1 is a singleton, we know that GLYPH<23> 4 GLYPH<3> 1 in any environment by Lemma B.11. But, clearly GLYPH<23> 9 never reaches GLYPH<3> 1 , since no agent in GLYPH<3> 1 ever chooses 0 1. GLYPH<3>

Lemma B.11. For all pairs ' GLYPH<3> GLYPH<150> 4 ' , if GLYPH<23> 2 GLYPH<3> , then GLYPH<23> 4 GLYPH<3> .

Proof of Lemma B.11.

The proposition is straightforward, as any GLYPH<23> 2 GLYPH<3> will be equivalent to itself in behavior for all histories. GLYPH<3>

Lemma B.12. Every agent satisfies GLYPH<23> 4 GLYPH<3> in every environment.

Proof of Lemma B.12.

This is again a direct consequence of Proposition C.18.

GLYPH<3>

Lemma B.13. The decision problem, AGENTREACHES , Given ' 4GLYPH<150> GLYPH<23> GLYPH<150> GLYPH<3> ' , output True iff GLYPH<23> 4 GLYPH<3> , is undecidable.

Proof of Lemma B.13.

We again proceed by reducing AGENTREACHES from the Halting Problem.

In particular, let < be a fixed but arbitrary Turing Machine, and F be a fixed but arbitrary input to be given to machine < . Then, HALT defines the decision problem that outputs True iff < halts on input F .

We construct an oracle for AGENTREACHES that can decide HALT as follows. Consider the same observation space used in the proof of Lemma B.8: Let O be comprised of all configurations of machine < . Then, sequences of observations are simply evolution of different Turing Machines processing possible inputs. We consider an action space, A = f 0 halted GLYPH<150> 0 not GLYPH<0> yet g , where agents simply report whether the history so far contains a halting configuration.

Then, we consider a deterministic environment 4 that simply produces the next configuration of < when run on input F , based on the current tape contents, the state of < , and the location of the tape head. Note again that all three of these elements are contained in a Turing Machine's configuration.

Using these ingredients, we take any instance of HALT, ' <GLYPH<150>F ' , and build the singleton agent set GLYPH<3> B containing only the agent GLYPH<23> halted : GLYPH<17> 7! 0 halted that always reports the machine as having halted. We then consider whether the agent that outputs 0 not GLYPH<0> yet indefinitely until < reports halting, at which point the agent switches to 0 halted .

We make one query to our AGENTREACHES oracle, and ask: GLYPH<23> 4 GLYPH<3> B. If it is True, then the branching agent eventually becomes equivalent to GLYPH<23> halted in that they both indefinitely output 0 halted on at least one realizable history. Since 4 is deterministic, we know this equivalence holds across all histories. If the query reports False, then there is no future in 4 in which < halts on F , otherwise the agent would become equivalent to GLYPH<23> halted. We thus use the oracle's response directly to decide the given instance of HALT. GLYPH<3>

C Additional Analysis

Finally, we present a variety of additional results about agents and the generates and reaches operators.

C.1 Additional Analysis: Generates

We first highlight simple properties of the generates operator. Many of our results build around the notion of uniform generation , a variant of the generates operator in which a basis generates an agent set in every environment. We define this operator precisely as follows.

Definition C.1. Let GLYPH<6> be a set of learning rules over some basis GLYPH<3> B . We say that a set GLYPH<3> is uniformly GLYPH<6> -generated by GLYPH<3> B , denoted GLYPH<3> B j =GLYPH<6> GLYPH<3> , if and only if

Definition C.2. We say a basis GLYPH<3> B uniformly generates GLYPH<3> , denoted GLYPH<3> B j = GLYPH<3> , if and only if

We will first show that uniform generation entails generation in a particular environment. As a consequence, when we prove that certain properties hold of uniform generation, we can typically also conclude that the properties hold for generation as well, though there is some subtlety as to when exactly this implication will allow results about j = to apply directly to ' 4 .

Proposition C.1. For any ' GLYPH<3> B GLYPH<150> GLYPH<3> ' pair, if GLYPH<3> B j = GLYPH<3> , then for all 4 2 E , GLYPH<3> B ' 4 GLYPH<3> .

Proof of Proposition C.1.

Recall that in the definition of uniform generation, GLYPH<3> B j = GLYPH<3> , we require,

Now, contrast this with generates with respect to a specific environment 4 ,

The only difference in the definitions is that the set of histories quantified over is H in the former, and ¢ H = H GLYPH<23> GLYPH<150>4 in the latter.

Since ¢ H GLYPH<18> H for any choice of environment 4 , we can conclude that when Equation C.3, it is also the case that Equation C.4 holds, too. Therefore, GLYPH<3> B j = GLYPH<3> ) GLYPH<3> B ' 4 GLYPH<3> for any 4 . GLYPH<3>

We next show that the subset relation implies generation.

Proposition C.2. Any pair of agent sets ' GLYPH<3> small GLYPH<150> GLYPH<3> big ' such that GLYPH<3> small GLYPH<18> GLYPH<3> big satisfies

Proof of Proposition C.2.

The result follows from the combination of two facts. First, that all agent sets generate themselves. That is, for arbitrary GLYPH<3> , we know that GLYPH<3> ' 4 GLYPH<3> , since the trivial set of learning rules,

that never switches between agents is sufficient to generate the agent set.

Second, observe that removing an agent from the generated set has no effect on the generates operator. That is, let GLYPH<3> 0 = GLYPH<3> n GLYPH<23> , for fixed but arbitrary GLYPH<23> 2 GLYPH<3> . We see that GLYPH<3> ' 4 GLYPH<3> 0 , since GLYPH<6> tr is sufficient to generate GLYPH<3> 0 , too. By inducting over all removals of agents from GLYPH<3> , we reach our conclusion. GLYPH<3>

Next, we establish properties about the sets of learning rules that correspond to the generates operator.

Proposition C.3. For any ' GLYPH<3> B GLYPH<150> GLYPH<6> GLYPH<150> GLYPH<3> ' such that GLYPH<3> B j =GLYPH<6> GLYPH<3> , it holds that

Proof of Proposition C.3.

We proceed toward contradiction, and assume j GLYPH<3> j 7 j GLYPH<6> j . Then, there is at least one learning rule GLYPH<30> 2 GLYPH<6> that corresponds to two or more distinct agents in GLYPH<3> . Call this element GLYPH<30> GLYPH<14> , and without loss of generality let GLYPH<23> 1 and GLYPH<23> 2 be two distinct agents that are each generated by GLYPH<30> GLYPH<14> in the sense that, for every GLYPH<17> 2 H . But, by the distinctness of GLYPH<23> 1 and GLYPH<23> 2 , there must exist a history GLYPH<17> in which GLYPH<23> 1 ' GLYPH<17> ' < GLYPH<23> 2 ' GLYPH<17> ' . We now arrive at a contradiction as such a history cannot exist: By Equation C.8, we know that GLYPH<23> 1 ' GLYPH<17> ' = GLYPH<30> GLYPH<14> ' GLYPH<17> '' GLYPH<17> ' = GLYPH<23> 2 ' GLYPH<17> ' for all GLYPH<17> . GLYPH<3>

We see that the universal learning rules, GLYPH<6> , is the strongest in the following sense.

Proposition C.4. For any basis GLYPH<3> B and agent set GLYPH<3> , exactly one of the two following properties hold:

The agent basis GLYPH<3> B uniformly generates GLYPH<3> under the set of all learning rules: GLYPH<3> B j = GLYPH<6> GLYPH<3> .

There is no set of learning rules for which the basis GLYPH<6> -uniformly generates the agent set: :9 GLYPH<6> GLYPH<18> GLYPH<6> GLYPH<3> B j =GLYPH<6> GLYPH<3> .

Proof of Proposition C.4.

The proof follows from the law of excluded middle. That is, for any set of learning rules GLYPH<6> , either it generates GLYPH<3> or it does not. If it does generate GLYPH<3> , by Lemma B.6 so does GLYPH<6> . By consequence, if GLYPH<6> does not generate GLYPH<3> , neither do any of its subsets. GLYPH<3>

Furthermore, uniform generation is also transitive.

Theorem C.5. Uniform generates is transitive: For any triple ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 GLYPH<150> GLYPH<3> 3 ' , if GLYPH<3> 1 j = GLYPH<3> 2 and GLYPH<3> 2 j = GLYPH<3> 3 , then GLYPH<3> 1 j = GLYPH<3> 3 .

Proof of Theorem C.5.

Assume GLYPH<3> 1 j = GLYPH<3> 2 and GLYPH<3> 2 j = GLYPH<3> 3 . Then, by Proposition C.4 and the definition of the uniform generates operator, we know that

where GLYPH<6> 1 and GLYPH<6> 2 express the set of all learning rules over GLYPH<3> 1 and GLYPH<3> 2 respectively. By definition of a learning rule, GLYPH<30> , we rewrite the above as follows,

Then, consider a fixed but arbitrary GLYPH<23> 3 2 GLYPH<3> 3 . We construct a learning rule defined over GLYPH<3> 1 as GLYPH<30> 1 : H ! GLYPH<3> 1 that induces an equivalent agent as follows. For each history, GLYPH<17> 2 H , by Equation C.12 we know that there is an GLYPH<23> 2 such that GLYPH<23> 3 ' GLYPH<17> ' = GLYPH<23> 2 ' GLYPH<17> ' , and by Equation C.11, there is an GLYPH<23> 1 such that GLYPH<23> 2 ' GLYPH<17> ' = GLYPH<23> 1 ' GLYPH<17> ' . Then, set GLYPH<30> 1 : GLYPH<17> 7! GLYPH<23> 1 such that GLYPH<23> 1 ' GLYPH<17> ' = GLYPH<23> 2 ' GLYPH<17> ' = GLYPH<23> 3 ' GLYPH<17> ' . Since GLYPH<17> and GLYPH<23> 3 were chosen arbitrarily, we conclude that

But, by the definition of GLYPH<6> , this means there exists a learning rule such that

This is exactly the definition of GLYPH<6> -uniform generation, and by Proposition C.4, we conclude GLYPH<3> 1 j = GLYPH<3> 3 . GLYPH<3>

Next, we show that a singleton basis only generates itself.

Proposition C.6. Any singleton basis, GLYPH<3> B = f GLYPH<23> g , only uniformly generates itself.

Proof of Proposition C.6.

Note that generates requires switching between base agents. With only a single agent, there cannot be any switching, and thus, the only agent that can be described as switching amongst the elements of the singleton set GLYPH<3> B = f GLYPH<23> g is the set itself. GLYPH<3>

C.1.1 Rank and Minimal Bases

As discussed in the paper, one natural reaction to the concept of an agent basis is to ask how we can justify different choices of a basis. And, if we cannot, then perhaps the concept of an agent basis is disruptive, rather than illuminating. In the main text, we suggest that in many situations, the choice of basis is made by the constraints imposed by the problem, such as the available memory. However, there are some objective properties of different bases that can help us to evaluate possible choices of a suitable basis. For instance, some bases are minimal in the sense that they cannot be made smaller while still retaining the same expressive power (that is, while generating the same agent sets). Identifying such minimal sets may be useful, as it is likely that there is good reason to consider only the most compressed agent bases.

To make these intuitions concrete, we introduce the rank of an agent set.

Definition C.3. The rank of an agent set, rank ' GLYPH<3> ' , is the size of the smallest agent basis that uniformly generates it:

For example, the agent set,

has rank ' GLYPH<3> ' = 2, since the basis,

uniformly generates GLYPH<3> , and there is no size-one basis that uniformly generates GLYPH<3> by Proposition C.6.

Using the notion of an agent set's rank, we now introduce the concept of a minimal basis . We suggest that minimal bases are particular important, as they contain no redundancy with respect to their expressive power. Concretely, we define a minimal basis in two slightly different ways depending on whether the basis has finite or infinite rank. In the finite case, we say a basis is minimal if there is no basis of lower rank that generates it.

Definition C.4. An agent basis GLYPH<3> B with finite rank is said to be minimal just when there is no smaller basis that generates it,

In the infinite case, as all infinite rank bases will have the same effective size, we instead consider a notion of minimiality based on whether any elements can be removed from the basis without changing its expressive power.

Definition C.5. An agent basis GLYPH<3> B with infinite rank is said to be minimal just when no proper subset of GLYPH<3> B uniformly generates GLYPH<3> B .

Notably, this way of looking at minimal bases will also apply to finite rank agent bases as a direct consequence of the definition of a minimal finite rank basis. However, we still provide both definitions, as a finite rank basis may not contain a subset that generates it, but there may exist a lower rank basis that generates it.

Corollary C.7. As a Corollary of Proposition C.2 and Definition C.4, for any minimal agent basis GLYPH<3> B , there is no proper subset of GLYPH<3> B that generates GLYPH<3> B .

Regardless of whether an agent basis has finite or infinite rank, we say the basis is a minimal basis of an agent set GLYPH<3> just when the basis uniformly generates GLYPH<3> and the basis is minimal.

Definition C.6. For any GLYPH<3> , a minimal basis of GLYPH<3> is any basis GLYPH<3> B that is both (1) minimal, and (2) GLYPH<3> B j = GLYPH<3> .

A natural question arises as to whether the minimal basis of any agent set GLYPH<3> is unique. We answer this question in the negative.

Proposition C.8. The minimal basis of a set of agents is not necessarily unique.

Proof of Proposition C.8.

To prove the claim, we construct an instance of an agent set with two distinct minimal bases.

Let A = f 0 0 GLYPH<150> 0 1 g , and O = f > 0 g . We consider the agent set containing four agents. The first two map every history to 0 0 and 0 1, respectively, while the second two alternate between 0 0

and 0 1 depending on whether the history is of odd or even length:

Note that there are two distinct subsets that each universally generate GLYPH<3> :

Next notice that there cannot be a singleton basis by Proposition C.6, and thus, both GLYPH<3> 0 GLYPH<150> 1 B and GLYPH<3> 2 GLYPH<150> 3 B satisfy (1) j GLYPH<3> 0 GLYPH<150> 1 B j = j GLYPH<3> 2 GLYPH<150> 3 B j = rank ' GLYPH<3> ' , and (2) both GLYPH<3> 0 GLYPH<150> 1 B j = GLYPH<3> , GLYPH<3> 2 GLYPH<150> 3 B j = GLYPH<3> . GLYPH<3>

Beyond the lack of redundancy of a basis, we may also be interested in their expressive power. For instance, if we compare two minimal bases, GLYPH<3> 1 B and GLYPH<3> 2 B , how might we justify which to use? To address this question, we consider another desirable property of a basis: universality .

Definition C.7. An agent basis GLYPH<3> B is universal if GLYPH<3> B j = GLYPH<3> .

Clearly, it might be desirable to work with a universal basis, as doing so ensures that the set of agents we consider in our design space is as rich as possible. We next show that there is at least one natural basis that is both minimal and universal.

Proposition C.9. The basis,

is a minimal universal basis :

GLYPH<3> GLYPH<14> B j = GLYPH<3> : The basis uniformly generates the set of all agents.

GLYPH<3> GLYPH<14> B is minimal.

Proof of Proposition C.9.

We prove each property separately.

First, we show that the basis is universal: GLYPH<3> GLYPH<14> B j = GLYPH<3> . Recall that this amounts to showing that,

Let GLYPH<23> 2 GLYPH<3> and GLYPH<17> 2 H be fixed but arbitrary. Now, let us label the action distribution produced by GLYPH<23> ' GLYPH<17> ' as ? GLYPH<23> ' GLYPH<17> ' . Let > refer to the last observation contained in GLYPH<17> (or ; if GLYPH<17> = GLYPH<17> 0 = ; ). Now, construct the agent GLYPH<23> GLYPH<14> B : > 7! ? GLYPH<23> ' GLYPH<17> ' . By construction of GLYPH<3> GLYPH<14> B , this agent is guaranteed to be a member of GLYPH<3> GLYPH<14> B , and furthermore, we know that GLYPH<23> GLYPH<14> B produces the same output as GLYPH<23> on GLYPH<17> . Since both GLYPH<23> and GLYPH<17> were chosen arbitrarily, the construction will work for any choice of GLYPH<23> and GLYPH<17> , and we conclude that at every history, there exists a basis agent GLYPH<3> GLYPH<14> B 2 GLYPH<3> GLYPH<14> B that produces the same probability distribution over actions as any given agent. Thus, the first property holds. X

2. GLYPH<3> GLYPH<14> B is minimal.

Second, we show that GLYPH<3> GLYPH<14> B is a minimal basis of GLYPH<3> . Recall that since rank ' GLYPH<3> GLYPH<14> B ' = 1 , the definition of a minimal basis means that:

To do so, fix an arbitrary proper subset of GLYPH<3> B 2 GLYPH<15> ' GLYPH<3> GLYPH<14> B ' . Notice that since GLYPH<3> B is a proper subset, there exists a non-empty set GLYPH<3> B such that,

Now, we show that GLYPH<3> B cannot uniformly generate GLYPH<3> by constructing an agent from GLYPH<3> B. In particular, consider the first element of GLYPH<3> B, which, by construction of GLYPH<3> GLYPH<14> B , is some mapping from O to a choice of probability distribution over A . Let us refer to this agent's output probability distribution over actions as ? . Notice that there cannot exist an agent in GLYPH<3> GLYPH<14> B that chooses ? , otherwise GLYPH<3> B would not be a proper subset of GLYPH<3> GLYPH<14> B . Notice further that in the set of all agents, there are infinitely many agents that output ? in at least one history. We conclude that GLYPH<3> B cannot uniformly generate GLYPH<3> , as it does not contain any base element that produces ? . The set GLYPH<3> B was chosen arbitrarily, and thus the claim holds for any proper subset of GLYPH<3> GLYPH<14> B , and we conclude. X

This completes the proof of both statements.

GLYPH<3>

Corollary C.10. As a direct consequence of Proposition C.9, every universal basis has infinite rank.

C.1.2 Orthogonal and Parallel Agent Sets

Drawing inspiration from vector spaces, we introduce notions of orthogonal and parallel agent bases according to the agent sets they generate.

Definition C.8. A pair of agent bases ( GLYPH<3> 1 B GLYPH<150> GLYPH<3> 2 B ) are orthogonal if any pair ' GLYPH<3> 1 GLYPH<150> GLYPH<3> 2 ' they each uniformly generate satisfy

Naturally this definition can be modified to account for environment-relative generation ( ' 4 ), or to be defined with respect to a particular set of learning rules in which case two bases are orthogonal with respect to the learning rule set just when they generate different agent sets under the given learning rules. As with the variants of the two operators, we believe the details of such formalisms are easy to produce.

A few properties hold of any pair of orthogonal bases.

Proposition C.11. If two bases GLYPH<3> 1 B GLYPH<150> GLYPH<3> 2 B are orthogonal, then the following properties hold:

GLYPH<3> 1 B \ GLYPH<3> 2 B = ; .

Neither GLYPH<3> 1 B nor GLYPH<3> 2 B are universal.

Proof of Proposition C.11.

We prove each property independently.

We proceed toward contradiction. That is, suppose that both GLYPH<3> 1 B is orthogonal to GLYPH<3> 2 B , and that GLYPH<3> 1 B \ GLYPH<3> 2 B < ; . Then, by the latter property, there is at least one agent that is an element of

both bases. Call this agent GLYPH<23> GLYPH<14> B . It follows that the set GLYPH<3> GLYPH<14> B = f GLYPH<23> GLYPH<14> B g is a subset of both GLYPH<3> 1 B and GLYPH<3> 2 B . By Proposition C.2, it follows that GLYPH<3> 1 B j = GLYPH<3> GLYPH<14> B and GLYPH<3> 2 B j = GLYPH<3> GLYPH<14> B . But this contradicts the fact that GLYPH<3> 1 B is orthogonal to GLYPH<3> 2 B , and so we conclude. X

We again proceed toward contradiction. Suppose without loss of generality that GLYPH<3> 1 B is universal. Then, we know GLYPH<3> 1 B j = GLYPH<3> . Now, we consider two cases: either GLYPH<3> 2 B generates some non-empty set, GLYPH<3> 2 , or it does not generate any sets. If it generates a set GLYPH<3> 2 , then we arrive at a contradiction as GLYPH<3> 2 \ GLYPH<3> < ; , which violates the definition of orthogonal bases. If if does not generate a set, this violates the definition of a basis, as any basis is by construction non-empty, and we know that containing even a single element is sufficient to generate at least one agent set by Proposition C.6. Therefore, in either of the two cases, we arrive at a contradiction, and thus conclude the argument. X

This concludes the proof of each statement.

GLYPH<3>

Corollary C.12. For any non-universal agent basis GLYPH<3> B , there exists an orthogonal agent basis, GLYPH<3> y B .

Conversely, two agent bases GLYPH<3> 1 B GLYPH<150> GLYPH<3> 2 B are parallel just when they generate the same agent sets.

Definition C.9. A pair of agent bases ' GLYPH<3> 1 B GLYPH<150> GLYPH<3> 2 B ' are parallel if for every agent set GLYPH<3> , GLYPH<3> 1 B j = GLYPH<3> if and only if GLYPH<3> 2 B j = GLYPH<3> .

Proposition C.13. If two bases GLYPH<3> 1 B GLYPH<150> GLYPH<3> 2 B are parallel, then the following properties hold:

Both GLYPH<3> 1 B j = GLYPH<3> 2 B and GLYPH<3> 2 B j = GLYPH<3> 1 B .

rank ' GLYPH<3> 1 B ' = rank ' GLYPH<3> 2 B ' .

GLYPH<3> 1 B is universal if and only if GLYPH<3> 2 B is universal.

Proof of Proposition C.13.

We prove each property separately.

The claim follows directly from the definition of parallel bases. An agent set GLYPH<3> is uniformly generated by GLYPH<3> 1 B if and only if it is uniformly generated by GLYPH<3> 2 B . Since by Proposition C.2 we know both GLYPH<3> 1 B j = GLYPH<3> 1 B and GLYPH<3> 2 B j = GLYPH<3> 2 B , we conclude that both GLYPH<3> 1 B j = GLYPH<3> 2 B and GLYPH<3> 2 B j = GLYPH<3> 1 B . X

Recall that the definition of rank refers to the size of the smallest basis that uniformly generates it,

Now, note that by property (1.) of the proposition, both sets uniformly generate each other. Therefore, we know that

since the smallest set that generates each basis is no larger than the basis itself, or the other basis.

3. GLYPH<3> 1 B is universal if and only if GLYPH<3> 2 B is universal.

The claim again follows by combining the definitions of universality and parallel: If GLYPH<3> 1 B is universal, then by definition of parallel bases, GLYPH<3> 2 B must uniformly generate all the same agent sets including GLYPH<3> , and therefore GLYPH<3> 2 B is universal, too. Now, if GLYPH<3> 1 B is not universal, then it does not uniformly generate GLYPH<3> . By the definition of parallel bases, we conclude GLYPH<3> 2 B does not generate GLYPH<3> as well. Both directions hold for each labeling of the two bases without loss of generality, and we conclude. X

This completes the argument for each property, and we conclude.

C.2 Analysis: Reaches

We now establish other properties of the reaches operator. Several of these results are based on a third modality of the reaches operator: always reaches , in which an agent eventually reaches an agent basis in all histories realizable in a given environment. We define this precisely as follows.

Definition C.10. We say agent GLYPH<23> 2 GLYPH<3> always reaches GLYPH<3> B , denoted GLYPH<23> GLYPH<3> 4 GLYPH<3> B , if and only if

The nested quantifiers allows the agent to become equivalent to different base behaviors depending on the evolution of the interaction stream. For example, in an environment that flips a coin to determine whether 0 heads or 0 tails is optimal, the GLYPH<23> might output 0 heads indefinitely if the coin is heads, but 0 tails otherwise. In this case, such an agent will still always reach the basis GLYPH<3> B = f GLYPH<23> 1 B : GLYPH<17> 7! 0 heads GLYPH<150> GLYPH<23> 2 B : GLYPH<17> 7! 0 tails g . Notice that we here make use of the notation, ¢ H C : 1 GLYPH<17> , which refers to all history suffixes of length C or greater, defined precisely as

We first show that the always reaches operator implies sometimes reaches.

Proposition C.14. If GLYPH<23> GLYPH<3> 4 GLYPH<3> , then GLYPH<23> 4 GLYPH<3> .

Proof of Proposition C.14.

Assume GLYPH<23> GLYPH<3> 4 GLYPH<3> . That is, expanding the definition of always reaches, we assume

Further recall the definition of can reach GLYPH<23> GLYPH<3> B is as follows

Then, the claim follows quite naturally: pick any realizable history GLYPH<17> 2 ¢ H . By our initial assumption that GLYPH<23> GLYPH<3> 4 GLYPH<3> , it follows (by Equation C.26) that there is a time C and a realizable history suffix GLYPH<17> GLYPH<14> for which

By construction of GLYPH<17> GLYPH<17> GLYPH<14> 2 ¢ H GLYPH<17> , we know GLYPH<17> GLYPH<17> GLYPH<14> is a realizable history. Therefore, there exists a realizable history, GLYPH<17> GLYPH<3> = GLYPH<17> GLYPH<17> GLYPH<14> , for which 9 GLYPH<23> B 2 GLYPH<3> B 8 GLYPH<17> 0 2 ¢ H GLYPH<17> GLYPH<23> ' GLYPH<17> GLYPH<3> GLYPH<17> 0 ' = GLYPH<23> B ' GLYPH<17> GLYPH<3> GLYPH<17> 0 ' holds. But this is exactly the definition of can reach, and therefore, we conclude the argument. GLYPH<3>

Next, we highlight the fact that every agent in a basis also reaches that basis.

Proposition C.15. For any agent set GLYPH<3> , it holds that GLYPH<23> GLYPH<3> 4 GLYPH<3> for every GLYPH<23> 2 GLYPH<3> .

Proof of Proposition C.15.

GLYPH<3>

The proposition is straightforward, as any GLYPH<23> 2 GLYPH<3> will be equivalent to itself in behavior for all histories. GLYPH<3>

We further show that, unlike sometimes and never reaches, always reaches is transitive.

Proposition C.17. Always reaches is transitive.

Proof of Proposition C.17.

We proceed by assuming that both 8 GLYPH<23> 1 2 GLYPH<3> 1 GLYPH<23> 1 GLYPH<3> 4 GLYPH<3> 2 and 8 GLYPH<23> 2 2 GLYPH<3> 2 GLYPH<23> 2 GLYPH<3> 4 GLYPH<3> 3 and show that it must follow that 8 GLYPH<23> 1 2 GLYPH<3> 1 GLYPH<23> 1 GLYPH<3> 4 GLYPH<3> 3 . To do so, pick a fixed but arbitrary GLYPH<23> 1 2 GLYPH<3> 1 , and expand GLYPH<23> 1 GLYPH<3> 4 GLYPH<3> 2 as

Now, consider for any realizable history GLYPH<17> GLYPH<17> GLYPH<14> GLYPH<17> 0 , we know that the corresponding GLYPH<23> 2 that produces the same action distribution as GLYPH<23> 1 also satisfies GLYPH<23> 2 GLYPH<3> 4 GLYPH<3> 3 . Thus, there must exist some time ¢ C at which, any realizable history ¢ GLYPH<17> ¢ GLYPH<17> GLYPH<14> , will satisfy 9 GLYPH<23> 3 2 GLYPH<3> 3 8¢ GLYPH<17> 0 2 ¢ ¢ H GLYPH<23> 2 ' ¢ GLYPH<17> ¢ GLYPH<17> GLYPH<14> ¢ GLYPH<17> 0 ' = GLYPH<23> 3 ' ¢ GLYPH<17> ¢ GLYPH<17> GLYPH<14> ¢ GLYPH<17> 0 . ¢

But then there exists a time ( C ), that ensures every GLYPH<23> 2 2 GLYPH<3> 2 will have a corresponding GLYPH<23> 3 2 GLYPH<3> 3 with the same action distribution at all subsequent realizable histories.

Therefore,

Thus, rewriting,

But this is precisely the definition of always reaches, and thus we conclude.

GLYPH<3>

Next, we show two basic properties of the set of all agents: it uniformly generates all agent sets, and it is always reached by all agents.

Proposition C.18. For any 4 , the set of all agents GLYPH<3> (i) uniformly generates all other agent sets, and (ii) is always reached by all agents:

Proof of Proposition C.18.

The property holds as a straightforward consequence of Proposition C.2: Since any set GLYPH<3> is a subset of GLYPH<3> , it follows that GLYPH<3> j = GLYPH<3> . X

The property holds as a straightforward consequence of Proposition C.15: Since every agent satisfies GLYPH<23> 2 GLYPH<3> , it follows that GLYPH<23> 4 GLYPH<3> . X

This concludes the argument of both statements.

GLYPH<3>

C.3 Figure: Set Relations in CRL

Finally, in Figure 3 we present a visual depicting the set relations in CRL between an agent basis GLYPH<3> B, an agent set it generates GLYPH<3> , and the three agent sets corresponding to those agents in GLYPH<3> that (i) sometimes, (ii) never, or (iii) always reach the basis. First, we highlight that we visualize GLYPH<3> B as a subset of GLYPH<3> since we define GLYPH<3> B GLYPH<26> GLYPH<3> in CRL (Definition 4.2). However, there can exist triples ' GLYPH<3> B GLYPH<150> GLYPH<3> GLYPH<150> 4 ' such that GLYPH<3> B ' 4 GLYPH<3> , but that GLYPH<3> B is not a subset of GLYPH<3> . Such cases are slightly peculiar, since it means that the basis contains agents that cannot be expressed by the agent set GLYPH<3> . Such cases are not in line with our definition of CRL, so we instead opt to visualize GLYPH<3> B as a subset of GLYPH<3> . Next, notice that the basis is a subset of both the agents that always reach the basis and the agents that sometimes reach the basis-this follows directly from the combination of Proposition C.14 and point (3.) of Theorem 4.3. By similar reasoning from Proposition C.14, we know that the set of agents that always reaches GLYPH<3> B is a subset of the agents that sometimes reach the basis. Further, since sometimes and never reaches are negations of one another (Remark 3.2), observe that the two sets are disjoint, and together comprise the entirety of GLYPH<3> . Lastly, we know that the set of optimal agents, GLYPH<3> GLYPH<3> , contains only agents that never reach the basis, and thus the set GLYPH<3> GLYPH<3> is disjoint from GLYPH<3> B and the set of agents that sometimes reach GLYPH<3> B.

Λ

{

/u1D706

∈

Λ

:

/u1D706

/squiggleright

/u1D452

Λ

B

}

{

/u1D706

∈

Λ

:

/u1D706

/squiggleright

/

/u1D452

Λ

B

}

{

/u1D706

∈

Λ

:

/u1D706

/square

/squiggleright

/u1D452

Λ

B

}

Λ

B

Λ

∗

<latexi sh

1_b

64="q3v8

BWSDJ

U

29

RCMP

X

7

>A

oFn

c

VpZ

y

E

5L

G

Ym

r

j

K

k

T

d

z

+

Q

/

O

w

g

0

f

I

N

H

u

<latexi sh

1_b

64="JW5CH7Z

3L8

O

rkNf

Bq

SQ

>A

nX

c

Vp

9y2E

m

v

jP

R

z

0

u

Y

G

M

U

K+

D/

w

g

o

F

d

I

T

<latexi sh

1_b

64="+VAk8y

JLgC

FmE

G5PI

o

>

H

c

p

3

d70

vT

Y

K

R

SZ

U

uw

z

9

ND

q

X

W

/

B

M

Of

Q

n

r

j

2

<latexi sh

1_b

64="wT

V

coj

SP

QkF5Ou fDBXG8

>A

E3

p

y

d

R7

9rHv

CW

q

K

Z

n

J

L

g

/

Y

N

+

m

z

0

2

I

M

U

<latexi sh

1_b

64="

y

Dvr

B

NGFf

Tz nuU

S

>A

3

c

V

d

k

E

7

0

g

K

q

I

C

M

p

w

O

L

Yo

/Q

+2

9W

H

J

5

8

m

P

X

<latexi sh

1_b

64="NOC

U

ym

3V

H+

Q/X

7o

>A

k

n

c

rd

E

fS

T9

v

B

2

G

p

Y

uF

5

8

IM

W

K

P

j

L

0

J

z

w

Z

D

g

q

R

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper looks at developing a foundation for continual reinforcement learning (CRL). The authors develop definitions and insights, aiming to formalize the intuitive concepts in continual learning fields. They also provide two examples of CRL to illustrate the difference between traditional RL and CRL, which is viewing learning as endless adaptation instead of finding a solution.\n\nSTRENGTHS:\n1, The concept of CRL as agent designer aiming to identify an optimal agent among available agents is innovative.\n\n2, The use of rigorous math tools and definition in this paper could provide valuable insight into continual learning and potentially inspire future CRL algorithms development.\n\nWEAKNESSES:\n1, Though it's not the focus and goal of this paper, the idea of agent basis contains redundancies and may hinder the transformation of it into a memory and computation efficient CRL algorithm.\n\n2, The training process of agent basis is unclear in Figure 2 (b). I guess two examples are presented to show the effectiveness of CRL, but it's not surprising that CRL can beat RL without adaptation in these changing envs. More online learning benchmarks would help to achieve a fair comparison.\n\n3, The paper is not well structured, and only few related works are mentioned. Continual Learning is discussed in many fields, with different names (e.g., online learning, few-shot adaptation, adaptive control), it would be valuable to provide a broader literature review.\n\nQUESTIONS:\nThe high-level idea of CRL training is understandable, but I am unsure about the training details in Section 4.2 CRL examples. Including a pseudocode description could clarify the training process and help understand the implementation details.\n\nLIMITATIONS:\nOverall, the paper requires a lot of additional work to become publishable. Adding figures and pseudocode would enhance readability and understanding. Furthermore, conducting more experiments to demonstrate the effectiveness of the proposed CRL idea would strengthen the paper's contributions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

comments: SUMMARY:
In this paper, the authors develop a simple mathematical definition of the continual RL problem. These definitions, insights, and results formalize many intuitive concepts at the heart of continual learning, and may open new research pathways surrounding continual learning agents.

STRENGTHS:
The mathematical definition of continual reinforcement learning give us some inspirations when thinking of the continual learning, multi-task RL, continual supervised learning, and so on.

WEAKNESSES:
1.The abstract is too simple that readers cannot get enough information and key ideas from it.
2.This paper is not easy to follow, for the overall mathematical definitions and analysis. I hope the authors can give us more cases or experiments using the definitions proposed in this paper to highlights the benefits of following such definitions.

QUESTIONS:
1.What information can we get from Figure 2(a) Switching MDP Visual?  Do authors just want to emphasis “Each underlying MDP shares the same state space and action space, but varies in transition and reward functions”? 
2.How can we prove or evaluate the definitions, the proposed operators, and so on?
3.Can authors add some ablation studies in this paper?

LIMITATIONS:
Please modify the paper as the weaknesses and questions above, and add limitation analysis in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a mathematical formulation for the problem of continual reinforcement learning in an infinite horizon setting.\n\nSTRENGTHS:\n1. The authors propose a new mathematical formalism for continual RL.\n2. The paper can be of interest to mathematically inclined readers and could potentially lead to theoretical results in continual reinforcement learning.\n\nWEAKNESSES:\n1. Abstract: there already are foundations for CRL. You cited a survey paper [16] and multiple other existing works. As unconventional as it is, I wouldn’t be against a short abstract, but I do not believe the current one will do.\n2. Missed important opportunities to connect your formalism with existing (continual) RL. E.g. eq 2.1 states an infinite horizon, which is required for example in Theorem 3.1, but the very standard “infinite horizon” vocabulary never appears. I think the paper is not connected enough to existing (continual) RL literature.\n3. Generally, I’m not convinced by motivations. The paper says it is important, but I did not really see a convincing motivation for abstracting the problem of continual RL. The theorems provide results that can be conveyed in significantly easier ways with existing formulations: e.g. non commutativity in thm 5.2 signifies that in continual RL, some policy transitions cannot be “unlearnt” in some settings. The formalism incurs a very significant overhead over such high level language – I believe that the authors can improve the motivations to justify why their formalism is worth it.\n\n\n**Typos and suggestions:**\n1. L10: we instead of We\n2. L41: I would use $\\mathcal{Z}$ instead of $\\mathcal{X}$ so it flows better in the following sentence.\n3. L84: phrasing “instead opt for simply the bounded…”\n4. L93: did you mean the sum of rewards to start at index 0 ? or t+1, t+2, etc.\n5. L95: I know it got absorbed into v, but I would still prepend this line with “Given <initial conditions> $h_0$, any tuple…” since your work aims to formalise rigorously the problem.\n6. “We use as if in the sense of the positive economists” this is terminology that should be defined in the paper.\n7. L334: “generates is commutative” -> are\n8. L338: I recommend not defining objects used in a main body theorem in the appendix. \n9. I believe that the insights, as stated in the conclusion, are trivial; perhaps “insight” isn’t the word that should be used (using the Cambridge dictionary definition of “insight” “ a clear, deep, and sometimes sudden understanding of a complicated problem or situation” – I do not believe the idea for example that agents in CRL either can or cannot find a set of policies/behaviours is insightful).\n\nQUESTIONS:\nNo.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper lays out a foundation for continual reinforcement learning (CLR) from the ground up—establishing definitions for the purpose of building towards a technical definition of CRL itself, proving various properties of CRL, and through employing simple CRL examples, demonstrating some of these properties.\n\nSTRENGTHS:\n- The writing is exceptionally clear and precise.\n- The framework is well thought-out; the definitions are clear and useful, and the framework as a whole is both technical and intuitive. Importantly the provided definition of CRL seems to get at the root of what researchers have always vaguely meant by CRL.\n- There is a rich foundation set for future work (e.g. exploring “connections between our formalism of continual learning and some of the phenomena at the heart of recent empirical continual learning studies, such as plasticity loss, in-context learning, and catastrophic forgetting”; line 364).\n\nWEAKNESSES:\n- I understand the appeal of a short abstract (i.e. it is to the point, and (albeit somewhat pompously) implies some special finality or importance to the contributions in the work), but the resolution of the abstract, for good reason, usually sits between the resolution of the title and the paper itself. The abstract in this case does not provide much more information than the title, so any reader that wants a quick overview of the paper must read the paper itself, defeating the purpose of the abstract. I suggest the authors to reconsider employing such a short abstract.\n- I feel that the paper could have developed more connections to previous literature, performed more experiments building off of this framework, etc., i.e. more validation for why we should care about this framework. Better (nonstationary) environments to evaluate agents on? Agent structures? Assessment of how well current methods fit into/exploit elements of this framework? The framework is nice, but it would be useful to see how it can situated into current literature, either by retroactively analyzing current agents/environments w.r.t. this framework, or designing new agents/environments from the principles this framework provides.\n- Maybe more real-world examples of changing environments? Or where we want the behavior to change in the same environment (but then we just have an oscillating agent…). Something about the environment changing (or being better understood over time) is fundamental to continual RL)\n- It took me a number of readings to understand agent bases; i.e. the relationship between $\\Lambda$ and $\\Lambda_B$.\n\nQUESTIONS:\n- Do we really want agents that will adapt their policy forever? Or are the environments we care about simply non-stationary, i.e. because the world changes, and the agent is required to learn more about the environment in order to continue to act optimally? Maybe the policy (in some high-level sense) should remain the same, and it is the world model that should be adapted? E.g. suppose the expressed behavior of the agent policy can be boiled down to “helping humans”, and this policy is fixed, given the agents current understanding of the world as context. Perhaps only this context needs to be updated, i.e. as the world changes, and/or the agent learns more about the existing world. This seems to fit within the proposed framework, if we consider the inferred world model as part of the agent. And while it does seem likely that the discovery of new contexts would require updating the policy, it still appears to me to be beneficial to disentangle these aspects explicitly. What are the authors' thoughts on this?\n\nLIMITATIONS:\n- The authors have discussed the limitations thoroughly, and I have listed them above as a strength as there are many directions ripe for future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-19"}

comments: SUMMARY:
The authors propose a new definition of continual reinforcement learning where an optimal continual learning agent will not converge to a fixed policy. This is formalized through the generate operator, which defines the "searching" between different policies; and the reach operator, which defines whether the agent will stop searching.

STRENGTHS:
1. This paper provides a unique definition of the continual learning problem, which is formalized by their mathematical framework.
2. Based on their framework, the authors were able to prove a number of interesting theorems.

WEAKNESSES:
1. It is not immediately obvious how this definition can inspire new learning algorithms.
2. The underlying concept that an optimal learning agent should not converge is not surprising. The author also mentioned that it's an "unsurprising conclusion that it is better to track than converge" for their toy non-stationary environment.

QUESTIONS:
1. Can you elaborate on how this formal definition of continual learning can address practical problems such as plasticity loss, in-context learning, and catastrophic learning as mentioned in the Discussion section?

LIMITATIONS:
The authors mention the limitations implicitly as future work in the discussion section. Essentially, since this paper is almost purely theoretical, they treat the lack of empirical study as future work or limitation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis is an ambitious paper.  The paper notes that the problem of \"Continual Reinforcement Learning\" lacks a rigorous definition and seeks to provide one.  It mathematically defines the reinforcement learning problem and then explores the conditions in which an instance of the RL problem is a CRL problem.\n\nThe legwork is done through the new \"generates\" and \"reaches\" operators.\n\nThe \"generates\" operator accepts two arguments: 1. a set of \"basis\" agents, and 2. a particular agent (which may not be in that set, but is exhibits some combined behavior of the basis agents).  It returns boolean truth value.  The \"generates\" operator expresses if there exists a learning rule that can search over \"basis\" agents (with respect to an environment / historical observations) and find a combination of them that is equivalent to the particular agent in the argument. This function is also undecidable in general.\n\nLoosely, the \"reaches\" operator (wrt an environment) accepts two arguments: 1. a particular agent 2. a set of \"basis\" agents It returns a boolean truth value.  It returns True when an agent's behavior settles on an element in the basis set.  The authors actually define a more precise \"sometimes reaches\" and \"never reaches\" operator. This is also undecidable.\n\nA learned agent will always choose a behavior equivalent to some basis agent in the context of the environment and history.\n\nUsing these formalisms, a CRL agent is one that never reaches an agent basis, and a CRL problem is one where the best agent is a CRL agent. \n\nNotably, a problem is a CRL problem depending on the choice of basis.\n\nThe paper performs mathematical analysis on these operators, relates them to two example instances of CRL, and provides a thoughtful discussion for new questions that can be asked about agent basis sets.\n\nSTRENGTHS:\nThe informal definition \"An RL problem is an instance of CRL is the best agents never stop learning\" is intuitive and clear.\n\nI always appreciate explicit definitions for notation. (although defining logical negation, and the quantifiers might not be necessary in a conference paper). I type checked several of the equations and everything seems to work out nicely. The theory gives a good feel for how someone might implement a framework for exploring agents in the language of this theory using a strongly typed language.\n\nThe paper gives explicit examples and experiments demonstrating intuitive properties of this theory in the setting of Q-learning and finite Markov decision processes. As well as a more complex example in continual supervised learning.\n\nThe paper proves several basic properties of the new reaches and generates operators. I have not checked the proofs for correctness, but I also don't see any obvious problems.\n\nSection 5 made me delete a lot of my questions because it answered them. The idea of exploring choices of the basis is mathematically interesting.\n\nThe paper raises more questions than it answers.\n\nWEAKNESSES:\nThe abstract is terse - problematically so. Even if this succeeds at being a theoretically sound definition of continual reinforcement learning, I think it is important to elaborate at least a little bit more. Note this is the main reason I'm rating this a 5 / 10. Given a better abstract I think this is a 7/10, but I think it is critical to give a better overview to the researchers' attention you are competing for.\n\nA 9 page paper with a 20 page appendix full of proofs may be better suited for a journal.\n\nThe paper raises more questions than it answers.\n\nQUESTIONS:\nOn line 47 the authors explicitly call out that the set A and O are countable.  Does this imply that some other (non-numeric - i.e. non-ℝ) sets may be uncountable in this definition?\n\nI think the order of appendix A and appendix B should be switched. Provide the notation first. (The table was very helpful in reviewing).\n\n\nI became confused when I first read Theorem 3.1. Particularly when the selected agent was not an element of the agent basis. In hindsight it makes sense that the new agent is a (linear?) combination behaviors in the agent basis set, but I think the paper could make that more clear. It wasn't intuitive what \"switches\" meant to me. Or that lambda(h) in Figure 1 was a combination of the basis agents conditioned on history. (At first I thought the learning rule was just choosing an element of the basis set).\n\nIn Theorem 3.1 are the infinite choices of basis countable?\n\nIf a problem's status as CRL or not depends on the choice of basis, and there are an infinite choices for the basis, then is this framework all that useful for thinking about problems?\n\nTheorem 5.1.2 suggests that it is possible to reduce any CRL problem to an RL problem via a change of basis.\n\nIs it possible to reduce any CRL problem to a RL problem by finding a function that changes the basis? Are there problems where such a mapping does not exist?\n\nHave you formalized any of the proofs into a proof assistant like Lean? I took a peek at the appendix, but I simply don't have the time to verify these proofs. Being able to point to an automatically checked proof in each theorem would greatly increase the quality of the paper.\n\nLIMITATIONS:\nAn optimal CRL agent could be difficult to rectify with the alignment problem. In contrast, it might be the case that deciding if a CRL agent will remain aligned with its creators is undecidable. That question is worth exploring.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper the authors propose a formal framework for reasoning about continual reinforcement learning problems. To this end they introduce mathematical definitions for environments and agents which serve as a basis for defining the general reinforcement learning setting as well as the continual setting. In particular, the authors tie the problem of continual learning to the learning progress an agent makes and two operators describing its limitting learning behaviour. Finally, they provide two examples and derive sevaral characteristics of the proposed operators.\n\nSTRENGTHS:\nThe problem of continual RL is most certainly relevant to the community and will arguably become more prominent as real-world agents adapting throughout their deployment become more common-place. To my knowledge the proposed formalism is original in its attempt to characterise this setting formally.\n\nOverall I found the framework sound and its description easy to follow. I agree with the authors that such a formalism has the potential to open up new perspectives on and analysis of (long-term) learning behaviour of agents.\n\nWEAKNESSES:\nThroughout the paper the terms \"agent\" and \"behaviour\" are used almost interchangeably. However, a formal definition of behaviours is missing, making it somewhat unclear whether or not the two terms are actually meant to be equivalent. If so, the statement \"we can understand every agent as implicitly searching over a set of behaviors\" needs clarification. If not, I would encourage the authors to formally define what a \"behaviour\" constitutes.\n\nA related issue is the separation of agents and their learning rules. Arguably, an agent parameterised by a recurrent network is a continual learner as it changes its \"behaviour\" (in the sense of the one-step policy) based on the history giving rise to its internal state. However, under the provided definition such an agent would not be considered a continual learner unless it updates its weights.\n\nFinally, the paper would benefit from a more in-depth discussion of how previous approaches to continual RL compare to the proposed framework or can be embedded into it. Similarly, the authors mention several avenues for future research such as the examination of catastrophic forgetting - it might be interesting to include at least a characterisation of these problems in the new framework to inspire future work.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}