Title: The World Is Bigger! A Computationally-Embedded Perspective on the Big World Hypothesis
Venue: NeurIPS 2025
Accepted: yes
Keywords: big world hypothesis

computational constraints

continual learning

reinforcement learning

algorithmic complexity
Abstract: Continual learning is often motivated by the idea, known as the big world hypothesis, that ``the world is bigger'' than the agent. Recent problem formulations capture this idea by explicitly constraining an agent relative to the environment. These constraints lead to solutions in which the agent continually adapts to best use its limited capacity, rather than converging to a fixed solution. However, explicit constraints can be ad hoc, difficult to incorporate, and may limit the effectiveness of scaling up the agent's capacity. In this paper, we characterize a problem setting in which an agent, regardless of its capacity, is constrained by being embedded in the environment. In particular, we introduce a computationally-embedded perspective that represents an embedded agent as an automaton simulated within a universal (formal) computer. Such an automaton is always constrained; we prove that it is equivalent to an agent that interacts with a partially observable Markov decision process over a countably infinite state-space. We propose an objective for this setting, which we call interactivity, that measures an agent's ability to continually adapt its behaviour by learning new predictions. We then develop a model-based reinforcement learning algorithm for interactivity-seeking, and use it to construct a synthetic problem to evaluate continual learning capability. Our results show that deep nonlinear networks struggle to sustain interactivity, whereas deep linear networks sustain higher interactivity as capacity increases.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a theoretical framework for continual learning based on the big world hypothesis, where agents are implicitly constrained by being embedded within a computationally universal and local environment. The authors formalize such agents as embedded automata and define a new objective called interactivity, which uses Kolmogorov complexity to quantify how much an agent’s future behavior can be both complex and predictable from its past. They prove that any finite-capacity agent maximizing interactivity must continually adapt, as stopping learning is provably suboptimal. To operationalize this, they develop an RL algorithm that approximates interactivity via prediction error and evaluate it on a synthetic behavioral self-prediction benchmark, where an agent learns from its own nonstationary experience stream. The work is mostly theoretical but provides a novel, general perspective on continual adaptation without requiring explicit task boundaries or external rewards.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Originality:\nThe paper presents a novel perspective on continual learning grounded in implicit constraints through computational embedding, rather than explicit resource limitations. The formalization of a universal-local environment and the use of Kolmogorov complexity to define the agent’s objective (interactivity) is highly original and conceptually deep.\n- Significance:\nBy framing continual learning in terms of interactivity, the paper offers a general and theoretically motivated setting in which any non-adaptive agent is provably suboptimal. This challenges standard assumptions in RL and provides a new foundation for studying continual adaptation without reliance on task segmentation or explicit forgetting.\n- Theoretical Rigor (Quality):\nThe paper is mathematically precise and includes formal definitions and proofs for all key claims, deferred to a comprehensive appendix. Notably, the results show that interactivity is upper bounded by agent capacity (Theorem 1) and that agents that stop learning are suboptimal (Theorem 2).\n- Clarity of Core Ideas:\nDespite the abstract nature of the concepts, the paper clearly communicates its main constructions—universal-local environments, embedded automata, and interactivity—with helpful analogies (e.g., Conway’s Game of Life) and visual aids. The contrast with traditional RL and AIXI is particularly helpful for positioning the contribution.\n\nWeaknesses:\n- Experiments (Quality / Significance):\nThe only experiment is a synthetic benchmark (behavioral self-prediction) without comparison to existing continual learning algorithms. While the experiment supports the theoretical claims, its simplicity (max 2 layers) limits practical insight into the applicability or scalability of the framework.\n- Clarity of Implementation Details:\nAlthough the theoretical setup is well presented, the reinforcement learning algorithm for maximizing interactivity is described somewhat abstractly. Important aspects like how the inner/outer learning loops interact, and the concrete form of predictions and TD errors, are difficult to follow without significant effort.\n- Limited Practical Connection:\nThe paper is mostly theoretical, and the benchmark does not involve realistic tasks or environments. This makes it unclear how the interactivity framework might inform or improve existing continual learning methods in practice, especially in high-dimensional domains.\n- Heavy Theoretical Overhead:\nThe reliance on Kolmogorov complexity, while elegant, leads to uncomputability in general and requires approximations that introduce complexity and potential instability (as discussed in the limitations). The practical utility of interactivity as a training signal may thus be limited without further simplification.\n\nQUESTIONS:\n1. Your use of universal-local environments (e.g., Conway’s Game of Life) is well-motivated as a constructive proof that embedded agents can exist. However, it’s not entirely clear why locality is essential beyond enabling the embedding. Could you clarify whether locality plays any further role in shaping the agent’s adaptation dynamics or in bounding interactivity?\n2. The proposed interactivity measure uses Kolmogorov complexity and is approximated via agent-relative prediction error. How general is this framework in practice? For instance, would interactivity remain a meaningful or tractable objective in high-dimensional or real-world sensory domains (e.g., vision, language)? What kinds of practical approximations might be required?\n3. Your behavioral self-prediction benchmark is clean and well-motivated, but it lacks comparisons to continual learning baselines. Can you elaborate on why existing baselines (e.g., EWC, replay buffers) were not included, and what challenges prevent comparison?\n4. Your learning algorithm for interactivity involves a form of bi-level optimization (meta-gradients), which is known to be expensive and unstable. Could you clarify whether this is essential to your formulation, or whether simpler approximations (e.g., first-order variants, bootstrapped objectives) might suffice?\n5. Your current experiments only include 2-layer networks. What's the rationale behind this choice, and how does the method extend to deeper networks?\n\nLIMITATIONS:\n- Practical Scalability:\nThe paper proposes an elegant but computationally heavy optimization procedure. While the authors mention second-order complexity and meta-gradients, they do not discuss the practical infeasibility of scaling to larger models. A clearer discussion of how the method might break down or require approximation in such settings would be valuable.\n- Limited Empirical Validation:\nThe synthetic benchmark is theoretically motivated but very narrow in scope. The authors should discuss whether their current results demonstrate efficacy beyond toy settings, and that future work is needed to bridge this gap.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework to think about agent/environment design by considering how the agent is embedded into the environment, and how the agent’s capabilities of computation are inherently limited based on how the agent is embedded in the environment. It proposes an intrinsic measure of interactivity which determines what the agent should be optimizing to learn more about its environment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Very unique paper and brings valuable insights.\n- The paper is systematic and I think the ideas presented in this paper are deep. It would be a nice addition to the conference.\n- I like that the experiments use self-prediction as a loss since it is aligned with intrinsic motivation losses.\n- The relaxation to measure complexity conditioned on some horizon H is clever to make this metric more practically useful (Section 5.3)\n- I appreciate the measure of interactivity is based on the undiscounted average reward setting, which is also an underappreciated/less studied setting.\n\nWeaknesses\n- The empirical results seem a bit rushed. I understand the focus does not have to be on a full empirical analysis, but I think some of the discussion from Appendix B2 should be moved up and the discussion should relate the results to the main thesis of the paper better.\n- The surprising finding of instability of linear methods is also not discussed as much, and deserves some more attention.\n\nQUESTIONS:\nIs the main idea the following: we treat the environment as a computing machine, which contains the agent (automaton), which is also a computing machine. Then since the agent is embedded in the environment, its computation abilities are upper bounded by what the environment allows and what it presents to the agent (partial observability). And then finally, the goal is for the agent to maximize interactivity, which is basically about learning about a future that is maximally complex (since more to learn about) but also you want your past to reduce complexity of future (since it tells us that the past behaviors were “useful”.\n\nLIMITATIONS:\nYes the paper acknowledges the computational limits of the algorithm to maximize interactivity.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn \"The World Is Bigger: A Computationally-Embedded Perspective on the Big World Hypothesis\" the authors propose a new general problem setting for studying (a version of) the continual learning problem. This is achieved first by defining what the author's call a \"universal-local environment\" into which an agent may be embedded. The embedded agent is explicitly constrained by construction and is shown to be representable as a POMDP. Furthermore, an intrinsic new measure is proposed for assessing the agents ability to adapt to its future environment, and that maximizing this measure is structurally a continual RL problem.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper presents what I would consider to be a bold contribution to the continual learning area\n* The paper presents a valuable extension (in spirit at least) to AIXI, which was an important conceptual contribution to the field\n* The formalism for much of the work in the continual learning set up is ad hoc, and this is a contribution towards placing those efforts on more stable grounds\n* The paper's motivation and writing are clear (if perhaps its technical presentation might be improved slightly) \n* The authors are to be commended for generating *results* from their construction (i.e. presenting an end-to-end utilization of the setup) with all that that entails\n\n\nWeaknesses:\n* Beyond an abstract theoretical construction it is not immediately clear to me if this will be of practical use to the community. Beyond the recovery of the \"common desideratum of the continual learning problem\"(which indeed is likely by explicit construction) it's not clear that studying agents/algorithms in this setting will provide insight to real-world systems\n* There are a lot of contributions here: the idea of \"universal-local\" environments, the definition of boundaries Markov processes, how to embed agents therein, demonstrating that this embedded construction has the desired properties and representations, the introduction of a suitable objective measure, and how that may be solved with TD methods etc., but with all that said, the empirical results feel almost incidental, and indeed weakly supported. \n* Minor typos here and there:\n> typo: line 76: \"Interactivity (as) viewed...\" -or- \"Interactivity (when) ~can be~ viewed from...\"\n> typo: line 325 \"... algorithms that stops learning...\"\n> type: line 327 \"We trained two a linear network and a non-linear network...\" (trained two what?)\n> typo: line 352 \"... is not be appropriate...\"\n\nQUESTIONS:\n1. Any idea why \"(linear methods) ... always lead to performance collapse\"?\n2. Interactivity as defined appears to be one sensible measure, were others considered and rejected?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors define a class of environments, \"universal Markov environments\" equivalent to universal Turing machines, and a \"locality\" condition. Authors then formalize agents/automata within these environments, and define interactivity as a complexity-theoretic measurement of the ability to adapt future behavior to past experience. Authors further define agent-relativized complexity and show they can approximate this with a value function trained with RL TD learning. Authors prove that agents with the goal of maximize interactivity necessarily employ continual learning; stopping learning is necessarily suboptimal. Authors implement this method on an environment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The idea of universal-local environments is interesting, and I like the example of Conway's Game of Life. I like the idea of having a general computational framework that can describe agency in terms of complexity.\n\nWeaknesses\n- I'm unsure if Kolmogorov complexity is the right complexity measure to use to describe agency- it may be, but it is also very general and uncomputable. It's unclear if the approximation the authors define is useful for applications.\n- I don't think \"implicitly constrained\" is defined sufficiently.\n- I'm not sure the authors explain why interactivity is a measure that should be optimized, beyond that it is a measure that requires continual learning by Theorem 2.\n- Weak experiments: There is just one set of experiments in Figure 4, and they are not described in Section 6 in sufficient detail. For example, the nature of the environment is not described.\n\nQUESTIONS:\n- Which definitions have already been defined in the literature, and which parts are novel contributions?\n- Section 1 states that \"Interactivity is similar to previously considered intrinsic motivation objectives 70 (Chentanez et al., 2004; Schmidhuber, 2010), and specifically predictive information (Bialek et al., 71 2001; Still and Precup, 2012). However, interactivity differs because of its formulation in terms 72 of behaviours using Kolmogorov complexity. This makes interactivity better suited to sequential 73 decision making in the constrained and partially observable setting that we consider.\" Is there a reason predictive information is less-well suited to sequential decision making? I would have thought that interactivity being defined based on Kolmogorov complexity would make it _worse_ for decision making settings.\n- I would explicitly define \"capacity\".\n- Is RL the most natural technique to learn in these environments?\n\nLIMITATIONS:\n- Interactivity can only be measured through an approximation, the utility of which is unclear.\n- Experiments showing the usefulness of these measures are not robust.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}