SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507, 2023.
[2] David Abel, Yuu Jinnai, Yue Guo, George Konidaris, and Michael L. Littman. Policy and value transfer in lifelong reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2018.
[3] Haitham Bou Ammar, Rasul Tutunov, and Eric Eaton. Safe policy search for lifelong reinforcement learning with sublinear regret. In Proceedings of the International Conference on Machine Learning, 2015.
[4] Megan M Baker, Alexander New, Mario Aguilar-Simon, Ziad Al-Halah, Sébastien MR Arnold, Ese Ben-Iwhiwhu, Andrew P Brna, Ethan Brooks, Ryan C Brown, Zachary Daniels, et al. A domain-agnostic approach for characterization of lifelong learning systems. *Neural Networks*, 160:274–296, 2023.
[5] Peter L Bartlett. Learning with a slowly changing distribution. In Proceedings of the Annual Workshop on Computational Learning Theory, 1992.
[6] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. Advances in Neural Information Processing Systems, 2014.
[7] Michael Bowling, John D. Martin, David Abel, and Will Dabney. Settling the reward hypothesis. In Proceedings of the International Conference on Machine Learning, 2023.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
[9] Emma Brunskill and Lihong Li. PAC-inspired option discovery in lifelong reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2014.
[10] Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka, and Tom Mitchell. Toward an architecture for never-ending language learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2010.
[11] Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the AAAI Conference on Artificial Intelligence, 1994.
[12] Michael K Cohen, Elliot Catt, and Marcus Hutter. A strongly asymptotically optimal agent in general environments. arXiv preprint arXiv:1903.01021, 2019.
[13] Travis Dick, András György, and Csaba Szepesvari. Online learning in Markov decision processes with changing cost sequences. In Proceedings of the International Conference on Machine Learning, 2014.
[14] Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325, 2021.
[33] Jelena Luketina, Sebastian Flennerhag, Yannick Schroecker, David Abel, Tom Zahavy, and Satinder Singh. Meta-gradients in non-stationary environments. In Proceedings of the Conference on Lifelong Learning Agents, 2022.
[34] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Proceedings of the International Conference on Machine Learning, 2023.
[35] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28-51, 2022.
[36] Sultan J Majeed. Abstractions of general reinforcement Learning. PhD thesis, The Australian National University, 2021.
[37] Sultan Javed Majeed and Marcus Hutter. Performance guarantees for homomorphisms beyond Markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[38] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In *Psychology of learning and motivation*, volume 24, pages 109–165. Elsevier, 1989.
[39] Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, et al. Never-ending learning. Communications of the ACM, 61(5):103-115, 2018.
[40] Claire Monteleoni and Tommi Jaakkola. Online learning of non-stationary sequences. Advances in Neural Information Processing Systems, 16, 2003.
[41] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. 2018.
[42] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71, 2019.
[43] Emmanouil Antonios Platanios, Abulhair Saparov, and Tom Mitchell. Jelly bean world: A testbed for never-ending learning. arXiv preprint arXiv:2002.06306, 2020.
[44] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
[45] Matthew Riemer, Sharath Chandra Raparthy, Ignacio Cases, Gopeshh Subbaraj, Maximilian Puelma Touzel, and Irina Rish. Continual learning in environments with polynomial mixing times. Advances in Neural Information Processing Systems, 2022.
[46] Mark B Ring. Continual learning in reinforcement environments. PhD thesis, The University of Texas at Austin, 1994.
[47] Mark B Ring. Child: A first step towards continual learning. Machine Learning, 28(1):77-104, 1997.
[48] Mark B Ring. Toward a formal framework for continual learning. In NeurIPS Workshop on Inductive Transfer, 2005.
[49] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 2019.
[50] Stuart J Russell and Devika Subramanian. Provably bounded-optimal agents. Journal of Artificial Intelligence Research, 2:575-609, 1994.
[51] Paul Ruvolo and Eric Eaton. ELLA: An efficient lifelong learning algorithm. In Proceedings of the International Conference on Machine Learning, 2013.
[52] Tom Schaul and Jürgen Schmidhuber. Metalearning. Scholarpedia, 5(6):4650, 2010.
[53] Jürgen Schmidhuber, Jieyu Zhao, and Nicol N Schraudolph. Reinforcement learning with self-modifying policies. In Learning to Learn, pages 293-309. Springer, 1998.
[54] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Proceedings of the International Conference on Machine Learning, 2018.
[55] Daniel L Silver. Machine lifelong learning: Challenges and benefits for artificial general intelligence. In Proceedings of the Conference on Artificial General Intelligence, 2011.
[56] Richard S Sutton. Introduction: The challenge of reinforcement learning. In Reinforcement Learning, pages 1-3. Springer, 1992.
[57] Richard S Sutton. The reward hypothesis, 2004. URL http://incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewardhypothesis.html.
[58] Richard S Sutton. The quest for a common model of the intelligent decision maker. arXiv preprint arXiv:2202.13252, 2022.
[59] Richard S Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[60] Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(Jul):1633-1685, 2009.
[61] Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Advances in Neural Information Processing Systems, 1995.
[62] Sebastian Thrun. Lifelong learning algorithms. Learning to Learn, 8:181-209, 1998.
[63] Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(1-2):25-46, 1995.
[64] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical Bayesian approach. In Proceedings of the International Conference on Machine learning, 2007.
We first provide a table summarizing all relevant notation.
Notation	Meaning	Definition
A	Actions
O	Observations
Ht	Length t histories	Ht=(A×O)t
H	All histories	H=∪t=0∞Ht
h	A history	h∈H
hh'	History concatenation
ht	Length t history	ht∈Ht
H̅=Hλ,e	Realizable histories	H̅=∪t=0∞{ht∈Ht:Πt-1k=0e(ok|hk,ak)λ(ak|hk)>0}
H̅h=Hλ,e	Realizable history suffixes	H̅h={h'∈H:h'h'∈Hλ,e}
e	Environment	e:H×A→Δ(A)
E	Set of all environments
λ	Agent	λ:H→Δ(A)
∧	Set of all agents
Λ	Set of agents	Λ⊆ ∧
ΛB	Agent basis	ΛB⊂ ∧
r	Reward function	r: A×O→R
v	Performance	v:H×A×E→[vmin, vmax]
σ	Learning rule	σ:H→ΛB
Σ	Set of all learning rules
Σ	Set of learning rules	Σ⊆ Σ
ΛB ⇔ Σ Λ	Σ-generates	∀Λ∃σ∈Σ∀h∈Hλ(h)=σ(h)(h)
ΛB ⇔ Λ	Generates	∃Σ⊆Σ ΛB ⇔ Σ Λ
ΛB |= Σ Λ	Universally Σ-generates	∀Λ∃σ∈Σ∀h∈Hλ(h)=σ(h)(h)
ΛB |= Λ	Universally generates	∃Σ⊆Σ ΛB |= Σ Λ
λ ⇔ ΛB	Sometimes reaches	∃h∈H∃λB∈ΛB∀h'∈Hhλ(hh')=λB(hh')
λ ⇔ ΛB	Never reaches	¬(λ ⇔ ΛB)
λ □ ⇔ ΛB	Always reaches	∀h∈H∃t∈N0∀h'∈Ht:∞∃λB∈ΛB∀h'∈Hhλ(hh'h')=λB(hh'h')
Table 1: A summary of notation.
B Proofs of Presented Results
We next provide proofs of each result from the paper. Our proofs make use of some extra notation: we use  $\Rightarrow$  as logical implication, and we use  $\mathcal{P}(\mathcal{X})$  to denote the power set of any set  $\mathcal{X}$ . Lastly, we use  $\forall_{\mathcal{A} \subseteq \mathcal{X}}$  and  $\exists_{\mathcal{A} \subseteq \mathcal{X}}$  as shorthand for  $\forall_{\mathcal{A} \in \mathcal{P}(\mathcal{X})}$  and  $\exists_{\mathcal{A} \in \mathcal{P}(\mathcal{X})}$  respectively.
B.1 Section 3 Proofs
Our first result is from Section 3 of the paper.
Theorem 3.1. For any pair  $(\lambda, e)$ , there exists infinitely many choices of a basis,  $\Lambda_{\mathrm{B}}$ , such that both (1)  $\lambda \notin \Lambda_{\mathrm{B}}$ , and (2)  $\Lambda_{\mathrm{B}} \notin \{\lambda\}$ .
Proof of Theorem 3.1.
Choose a fixed but arbitrary pair  $(\lambda, e)$ . Then, enumerate the realizable histories,  $\mathcal{H}^{\lambda, e}$ , and let  $h^1$  denote the first element of this enumeration,  $h^2$  the second, and so on.
Then, we design a constructive procedure for a basis that, when repeatedly applied, induces an infinite enumeration of bases that satisfy the desired two properties. This constructive procedure for the  $k$ -th basis will contain  $k + 1$  agents, where each agent is distinct from  $\lambda$ , but will produce the same action as the agent every  $k + 1$  elements of the history sequence,  $h^1, h^2, \ldots$ .
For the first  $(k = 1)$  basis, we construct two agents. The first,  $\lambda_{\mathrm{B}}^{1}$ , chooses the same action distribution as  $\lambda$  on each even numbered history:  $\lambda_{\mathrm{B}}^{1}(h^{i}) = \lambda (h^{i})$ . Then, this agent will choose a different action distribution on the odd length histories:  $\lambda_{\mathrm{B}}^{1}(h^{i + 1})\neq \lambda (h^{i + 1})$ , for  $i$  any even natural number. The second agent,  $\lambda_{\mathrm{B}}^{2}$  will do the opposite to  $\lambda_{\mathrm{B}}^{1}$ : on each odd numbered history  $h^{i + 1}$ ,  $\lambda_{\mathrm{B}}^{2}(h^{i + 1})\neq \lambda (h^{i + 1})$ , but on every even numbered history,  $\lambda_{\mathrm{B}}^{2}(h^{i}) = \lambda (h^{i})$ .
Observe first that by construction,  $\lambda \neq \lambda_{\mathrm{B}}^{1}$ , and  $\lambda \neq \lambda_{\mathrm{B}}^{2}$ , since there exist histories where they choose different action distributions. Next, observe that the basis,  $\Lambda_{\mathrm{B}} = \{\lambda_{\mathrm{B}}^{1}, \lambda_{\mathrm{B}}^{2}\}$ , generates  $\{\lambda\}$  in  $e$  through the following set of learning rules,  $\Sigma$ : given any realizable history,  $h \in \mathcal{H}^{\lambda, e}$ , check whether the history has an even or odd numbered index in the enumeration. If odd, choose  $\lambda_{\mathrm{B}}^{1}$ , and if even, choose  $\lambda_{\mathrm{B}}^{2}$ .
More generally, this procedure can be applied for any  $k$ :
$$
\Lambda_ {\mathrm {B}} ^ {k} = \left\{\lambda_ {\mathrm {B}} ^ {1}, \dots , \lambda_ {\mathrm {B}} ^ {k + 1} \right\}, \quad \lambda_ {\mathrm {B}} ^ {i} (h) = \left\{ \begin{array}{l l} \lambda (h) & [ h ] = = i, \\ \neq \lambda (h) & \text {o t h e r w i s e}, \end{array} \right. \tag {B.1}
$$
where we use the notation  $[h] == i$  to express the logical predicate asserting that the modulus of the index of  $h$  in the enumeration  $h^1, h^2, \ldots$  is equal to  $i$ .
Further,  $\neq \lambda(h)$  simply refers to any choice of action distribution that is unequal to  $\lambda(h)$ . Thus, for all natural numbers  $k \geq 2$ , we can construct a new basis consisting of  $k$  base agents that generates  $\lambda$  in  $e$ , but does not contain the agent itself. This completes the argument.
B.2 Section 4 Proofs
We next present the proofs of results from Section 4.
B.2.1 Theorem 4.1: Properties of CRL
We begin with Theorem 4.1 that establishes basic properties of CRL.
Theorem 4.1. Every instance of  $CRL(e, v, \Lambda, \Lambda_{\mathrm{B}})$  satisfies the following properties:
1. If  $\Lambda \neq \Lambda_{\mathrm{B}} \cup \Lambda^{*}$ , there exists a  $\Lambda_{\mathrm{B}}'$  such that (1)  $\Lambda_{\mathrm{B}}' \vDash \Lambda$ , and (2)  $(e, v, \Lambda, \Lambda_{\mathrm{B}}')$  is not an instance of CRL.
2. No element of  $\Lambda_{\mathrm{B}}$  is optimal:  $\Lambda_{\mathrm{B}}\cap \Lambda^{*} = \emptyset$
3. If  $|\Lambda|$  is finite, there exists an agent set,  $\Lambda^{\circ}$  such that  $|\Lambda^{\circ}| < |\Lambda|$  and  $\Lambda^{\circ} \nVdash \Lambda$ .
4. If  $|\Lambda|$  is infinite, there exists an agent set,  $\Lambda^{\circ}$  such that  $\Lambda^{\circ} \subset \Lambda$  and  $\Lambda^{\circ} \nVdash \Lambda$ .
We prove this result in the form of three lemmas, corresponding to each of the four points of the theorem (with the third lemma, Lemma B.3, covering both points 3. and 4.). Some of the lemmas make use of properties of generates and reaches that we establish later in Appendix C.
Lemma B.1. For all instances of  $CRL(e, v, \Lambda, \Lambda_{\mathrm{B}})$ , if  $\Lambda \neq \Lambda_{\mathrm{B}} \cup \Lambda^{*}$ , then there exists a choice  $\Lambda_{\mathrm{B}}'$  such that (1)  $\Lambda_{\mathrm{B}}' \notin \Lambda$ , and (2)  $(e, v, \Lambda, \Lambda_{\mathrm{B}}')$  is not an instance of  $CRL$ .
Proof of Lemma B.1.
Recall that a tuple  $(e, v, \Lambda, \Lambda_{\mathrm{B}})$  is CRL just when all of the optimal agents  $\Lambda^{*}$  do not reach the basis. Then, the result holds as a straightforward consequence of two facts. First, we can always construct a new basis containing all of the optimal agents,  $\Lambda_{\mathrm{B}}^{\circ} = \Lambda_{\mathrm{B}} \cup \Lambda^{*}$ . Notice that  $\Lambda_{\mathrm{B}}^{\circ}$  still generates  $\Lambda$  by property three of Theorem 4.2. Further, since both  $\Lambda_{\mathrm{B}}$  and  $\Lambda^{*}$  are each subsets of  $\Lambda$ , and by assumption  $\Lambda \neq \Lambda_{\mathrm{B}} \cup \Lambda^{*}$  (so there is at least one sub-optimal agent that is not in the basis), it follows that  $\Lambda_{\mathrm{B}}^{\circ} \subset \Lambda$ . Second, by Proposition C.15, we know that every element  $\lambda_{\mathrm{B}}^{\circ} \in \Lambda_{\mathrm{B}}^{\circ}$  will always reach the basis,  $\lambda_{\mathrm{B}}^{\circ} \rightsquigarrow \Lambda_{\mathrm{B}}^{\circ}$ . Therefore, in the tuple  $(e, v, \Lambda, \Lambda_{\mathrm{B}}^{\circ})$ , each of the optimal agents will reach the basis, and therefore this is not an instance of CRL.
Lemma B.2. No element of  $\Lambda_{\mathrm{B}}$  is optimal:  $\Lambda_{\mathrm{B}}\cap \Lambda^{*} = \emptyset$
Proof of Lemma B.2.
The lemma follows as a combination of two facts.
First, recall that, by definition of CRL, each optimal agent  $\lambda \in \Lambda^{*}$  satisfies  $\lambda^{*} \nrightarrow \Lambda_{\mathrm{B}}$ .
Second, note that by Lemma B.11, we know that each  $\lambda_{\mathrm{B}} \in \Lambda_{\mathrm{B}}$  satisfies  $\lambda_{\mathrm{B}} \stackrel{\varkappa}{\rightsquigarrow} \Lambda_{\mathrm{B}}$ .
Therefore, since sometimes reaches  $(\wedge \rightarrow)$  and never reaches  $(\neg \rightarrow)$  are negations of one another, we conclude that no basis element can be optimal.
Before stating the next lemma, we note that points (3.) and (4.) of Theorem 4.1 are simply expansions of the definition of a minimal agent set, which we define precisely in Definition C.4 and Definition C.5.
Lemma B.3. For any instance of CRL,  $\Lambda$  is not minimal.
Proof of Lemma B.3.
We first show that  $\Lambda$  cannot be minimal. To do so, we consider the cases where the rank (Definition C.3) of  $\Lambda$  is finite and infinite separately.
(Finite Rank  $\Lambda$ .)
If  $\mathrm{rank}(\Lambda)$  is finite and minimal, then it follows immediately that there is no agent set of smaller rank that generates  $\Lambda$ . By consequence, since  $\Lambda_{\mathrm{B}} \subset \Lambda$  and  $\Lambda_{\mathrm{B}} \notin \Lambda$ , we conclude that  $\Lambda$  cannot be minimal.
(Infinite Rank  $\Lambda$ )
If  $\mathrm{rank}(\Lambda)$  is infinite and minimal, then there is no proper subset of  $\Lambda$  that uniformly generates  $\Lambda$  by definition. By consequence, since  $\Lambda_{\mathrm{B}} \subset \Lambda$  and  $\Lambda_{\mathrm{B}} \not\in \Lambda$ , we conclude that  $\Lambda$  cannot be minimal.  $\checkmark$
This completes the argument of both cases, and we conclude that for any instance of CRL,  $\Lambda$  is not minimal.
B.2.2 Theorem 4.2: Properties of Generates
Next, we prove basic properties of generates.
1. Generates is transitive: For any triple  $(\Lambda^1, \Lambda^2, \Lambda^3)$  and  $e \in \mathcal{E}$ , if  $\Lambda^1 \not\in \Lambda^2$  and  $\Lambda^2 \not\in \Lambda^3$ , then  $\Lambda^1 \not\in \Lambda^3$ .
2. Generates is not commutative: there exists a pair  $(\Lambda^1, \Lambda^2)$  and  $e \in \mathcal{E}$  such that  $\Lambda^1 \not\subset \Lambda^2$ , but  $\neg (\Lambda^2 \nmid e \Lambda^1)$ .
3. For all  $\Lambda$  and pair of agent bases  $(\Lambda_{\mathrm{B}}^{1},\Lambda_{\mathrm{B}}^{2})$  such that  $\Lambda_{\mathrm{B}}^{1}\subseteq \Lambda_{\mathrm{B}}^{2}$  if  $\Lambda_{\mathrm{B}}^{1}\vDash \Lambda$  , then  $\Lambda_{\mathrm{B}}^{2}\vDash \Lambda$
4. For all  $\Lambda$  and  $e\in \mathcal{E},\wedge \nmid e\Lambda$
5. The decision problem, Given  $(e,\Lambda_{\mathrm{B}},\Lambda)$ , output True iff  $\Lambda_{\mathrm{B}} \nleq \Lambda$ , is undecidable.
The proof of this theorem is spread across the next five lemmas below.
The fact that generates is transitive suggests that the basic tools of an agent set—paired with a set of learning rules—might be likened to an algebraic structure. We can draw a symmetry between an agent basis and the basis of a vector space: A vector space is comprised of all linear combinations of the basis, whereas  $\Lambda$  is comprised of all valid switches (according to the learning rules) between the base agents. However, the fact that generates is not commutative (by point 2.) raises a natural question: are there choices of learning rules under which generates is commutative? An interesting direction for future work is to explore this style of algebraic analysis on agents.
Lemma B.4. Generates is transitive: For any triple  $(\Lambda^1, \Lambda^2, \Lambda^3)$  and  $e \in \mathcal{E}$ , if  $\Lambda^1 \not\subsetneq \Lambda^2$  and  $\Lambda^2 \not\subsetneq \Lambda^3$ , then  $\Lambda^1 \not\subsetneq \Lambda^3$ .
Proof of Lemma B.4.
Assume  $\Lambda^1 \not\in \Lambda^2$  and  $\Lambda^2 \not\in \Lambda^3$ . Then, by Proposition C.4 and the definition of the generates operator, we know that
$$
\forall_ {\lambda^ {2} \in \Lambda^ {2}} \exists_ {\sigma^ {1} \in \Sigma^ {1}} \forall_ {h \in \bar {\mathcal {H}}} \lambda^ {2} (h) = \sigma^ {1} (h) (h), \tag {B.2}
$$
$$
\forall_ {\lambda^ {3} \in \Lambda^ {3}} \exists_ {\sigma^ {2} \in \mathbb {E} ^ {2}} \forall_ {h \in \bar {\mathcal {H}}} \lambda^ {3} (h) = \sigma^ {2} (h) (h), \tag {B.3}
$$
where  $\Sigma^1$  and  $\Sigma^2$  express the set of all learning rules over  $\Lambda^1$  and  $\Lambda^2$  respectively. By definition of a learning rule,  $\sigma$ , we rewrite the above as follows,
$$
\forall_ {\lambda^ {2} \in \Lambda^ {2}} \forall_ {h \in \bar {\mathcal {H}}} \exists_ {\lambda^ {1} \in \Lambda^ {1}} \lambda^ {2} (h) = \lambda^ {1} (h), \tag {B.4}
$$
$$
\forall_ {\lambda^ {3} \in \Lambda^ {3}} \forall_ {h \in \bar {\mathcal {H}}} \exists_ {\lambda^ {2} \in \Lambda^ {2}} \lambda^ {3} (h) = \lambda^ {2} (h). \tag {B.5}
$$
Then, consider a fixed but arbitrary  $\lambda^3\in \Lambda^3$ . We construct a learning rule defined over  $\Lambda^1$  as  $\sigma^1:\mathcal{H}\to \Lambda^1$  that induces an equivalent agent as follows. For each realizable history,  $h\in \bar{\mathcal{H}}$ , by Equation B.5 we know that there is an  $\lambda^2$  such that  $\lambda^3 (h) = \lambda^2 (h)$ , and by Equation B.4, there is an  $\lambda^1$  such that  $\lambda^2 (h) = \lambda^1 (h)$ . Then, set  $\sigma^1:h\mapsto \lambda^1$  such that  $\lambda^1 (h) = \lambda^2 (h) = \lambda^3 (h)$
Since  $h$  and  $\lambda^3$  were chosen arbitrarily, we conclude that
$$
\forall_ {\lambda^ {3} \in \Lambda^ {3}} \forall_ {h \in \bar {\mathcal {H}}} \exists_ {\lambda^ {1} \in \Lambda^ {1}} \lambda^ {3} (h) = \lambda^ {1} (h).
$$
But, by the definition of  $\Sigma$ , this means there exists a learning rule such that
$$
\forall_ {\lambda^ {3} \in \Lambda^ {3}} \exists_ {\sigma^ {1} \in \mathbb {Z} ^ {1}} \forall_ {h \in \bar {\mathcal {H}}} \lambda^ {3} (h) = \sigma^ {1} (h) (h).
$$
This is exactly the definition of  $\Sigma$ -generation, and by Proposition C.4, we conclude  $\Lambda^1 \vDash \Lambda^3$ .
Lemma B.5. Generates is not commutative: there exists a pair  $(\Lambda^1, \Lambda^2)$  and  $e \in \mathcal{E}$  such that  $\Lambda^1 \nVdash \Lambda^2$ , but  $\neg (\Lambda^2 \nVdash \Lambda^1)$ .
Proof of Lemma B.5.
The result follows from a simple counterexample: consider the pair
$$
\Lambda^ {1} = \left\{\lambda_ {i}: h \mapsto a _ {1} \right\}, \quad \Lambda^ {2} = \left\{\lambda_ {i}: h \mapsto a _ {1}, \lambda_ {j}: h \mapsto a _ {2} \right\}.
$$
Note that since  $\lambda_{i}$  is in both sets, and  $\Lambda^1$  is a singleton, we know that  $\Lambda^2 \notin \Lambda^1$  in any environment. But, by Proposition C.6, we know that  $\Lambda^1$  cannot generate  $\Lambda^2$ .
Lemma B.6. For all  $\Lambda$  and pair of agent bases  $(\Lambda_{\mathrm{B}}^{1},\Lambda_{\mathrm{B}}^{2})$  such that  $\Lambda_{\mathrm{B}}^{1}\subseteq \Lambda_{\mathrm{B}}^{2}$ , if  $\Lambda_{\mathrm{B}}^{1}\not\in\Lambda$ , then  $\Lambda_{\mathrm{B}}^{2}\not\in\Lambda$ .
Proof of Lemma B.6.
The result follows as a natural consequence of the definition of generates. Recall that  $\Lambda_{\mathrm{B}}^{1} \models^{\varepsilon} \Lambda$  just when,
$$
\begin{array}{l} \exists_ {\Sigma^ {1} \subseteq \Xi} \Lambda_ {\mathbf {B}} ^ {1} \varepsilon_ {\Sigma^ {1}} \Lambda (B.6) \\ \equiv \exists_ {\Sigma^ {1} \subseteq \mathbb {Z}} \exists_ {\sigma^ {1} \in \Sigma^ {1}} \forall_ {h \in \bar {\mathcal {H}}} \lambda (h) = \lambda_ {\mathrm {B}} ^ {\sigma^ {1} (h)} (h), (B.7) \\ \end{array}
$$
where again  $\lambda_{\mathrm{B}}^{\sigma^{1}(h)} \in \Lambda_{\mathrm{B}}^{1}$  is the base agent chosen by  $\sigma^{1}(h)$ . We use superscripts  $\Sigma^{1}$  and  $\sigma^{1}$  to signify that  $\sigma^{1}$  is defined relative to  $\Lambda_{\mathrm{B}}^{1}$ , that is,  $\sigma^{1}: \mathcal{H} \to \Lambda_{\mathrm{B}}^{1} \in \Sigma^{1}$ .
But, since  $\Lambda_{\mathrm{B}}^{1} \subseteq \Lambda_{\mathrm{B}}^{2}$ , we can define  $\Sigma^{2} = \Sigma^{1}$  and ensure that  $\Lambda_{\mathrm{B}}^{2} \not\subsetneq \Sigma^{2} \Lambda$ , since the agent basis  $\Lambda_{\mathrm{B}}^{1}$  was already sufficient to generate  $\Lambda$ . Therefore, we conclude that  $\Lambda_{\mathrm{B}}^{2} \not\subsetneq \Lambda$ .
Lemma B.7. For all  $\Lambda$  and  $e\in \mathcal{E},\mathbb{A}\notin \Lambda$
Proof of Lemma B.7.
This is a direct consequence of Proposition C.18.
Lemma B.8. The decision problem, AGENTSGENERATE, Given  $(e, \Lambda_{\mathrm{B}}, \Lambda)$ , output True iff  $\Lambda_{\mathrm{B}} \not\in \Lambda$ , is undecidable.
Proof of Lemma B.8.
We proceed as is typical of such results by reducing AGENTSGENERATE from the Halting Problem.