Cognitive Predictive Processing: A Human-inspired Framework for Adaptive Exploration in Open-World Reinforcement Learning
Authors: boheng liu
Ziyu Li
Chenghua Duan
YuTian Liu
Zhuo Wang
Xiuxing Li
Qing Li
Xia Wu
Venue: NeurIPS 2025 poster ()
Keywords: Open-World Reinforcement Learning
Human-inspired Artificial Intelligence
Cognitive Architectures
Primary area: reinforcement_learning
TL;DR: Human-inspired cognitive mechanisms improve reinforcement learning performance in open-world environments by 4.6% while reducing completion steps by 7.1%.

Abstract: Open-world reinforcement learning challenges agents to develop intelligent behavior in vast exploration spaces. Recent approaches like LS-Imagine have advanced the field by extending imagination horizons through jumpy state transitions, yet remain limited by fixed exploration mechanisms and static jump thresholds that cannot adapt across changing task phases, resulting in inefficient exploration and lower completion rates.
  Humans demonstrate remarkable capabilities in open-world decision-making through a chain-like process of task decomposition, selective memory utilization, and adaptive uncertainty regulation.
  Inspired by human decision-making processes, we present Cognitive Predictive Processing (CPP), a novel framework that integrates three neurologically-inspired systems: a phase-adaptive cognitive controller that dynamically decomposes tasks into exploration, approach, and completion phases with adaptive parameters;
  a dual-memory integration system implementing dual-modal memory that balances immediate context with selective long-term storage; 
  and an uncertainty-modulated prediction regulator that continuously updates environmental predictions to modulate exploration behavior.
  Comprehensive experiments in MineDojo demonstrate that these human-inspired decision-making strategies enhance performance over recent techniques, with success rates improving by an average of 4.6\% across resource collection tasks while reducing task completion steps by an average of 7.1\%. 
  Our approach bridges cognitive neuroscience and reinforcement learning, excelling in complex scenarios that require sustained exploration and strategic adaptation while demonstrating how neural-inspired models can solve key challenges in open-world AI systems.

Cognitive Predictive Processing: A Human-inspired Framework for Adaptive Exploration in Open- Reinforcement Learning
Boheng Liu', Ziyu Li
Chenghua Duan',
Yutian Liu',
Zhuo Wang', Xiuxing Li', Qing Li', Xia Wu
Beijing Institute of Technology
Tangshan Research Institute, Beijing Institute of Technology Scheng@bit.edu.. cn, Syyul@@bt.edu.com
Abstract
Open-world reinforcement learning challenges agents develop intelligent behav- ior in vast exploration spaces. Recent approaches like LS- L.S-mmgiine have advanced the field by extending imagination horizons through jumpy state transitions, yet remain limited by fixed exploration mechanisms and static jump thresholds that cannot adapt across changing task phases, resulting inefficient exploration and lower completion rates. Humans demonstrate remarkable capabilities in open- world decision- maaiing through a chain-like process of task decomposition, selec- tive memory utilization, and adaptive uncertainty regulation. Inspired by human decision- makkin processes, we present Cognitive Predictive Processing (CPP), a novel framework that integrates three caurologically-inspire systems: a phase- adaptive cognitive controller that dynamically decomposes tasks into exploration, approach, and completion phases with adaptive parameters; dual-memory inte- gration system implementing dual-modal memory that balances immediate context with selective long-term storage; and an tor that continuously updates environmental predictions to modulate exploration behavior. Comprehensive experiments in MineDojo demonstrate that these human- inspired Dacision-making strategies enhance performance over recent techniques, with success rates improving by an average of 4.6% across resource collection tasks while reducing task completion steps by an average of 7.1%. Our approach bridges cognitive neuroscience and reinforcement learning, excelling in complex scenarios that require sustained exploration and strategic adaptation while demonstrating how neural-insee inspiree models can solve key challenges open-world AI systems.
prediction regula-
Introduction
Open-world reinforcement learning (RL) challenges agents to develop intelligent behavior in environ- ments characterized by unbounded exploration spaces, diverse interaction possibilities, and limited perceptual information [1, 31, 35]. These environments demand flexible decision-making capabil- ities that can generalize across varied scenarios and objectives while operating under substantial uncertainty [12]. Contemporary embodied AI systems struggle with efficient exploration, contextual adaptation, and progressive learning in such complex domains [8], highlighting the need for more sophisticated cognitive architectures that can emulate human-
human-inspired inspired reasoning and adaptation [26].
Recent advancements in model-based reinforcement learning have made significant progress toward addressing these challenges [1, 11, 16, 28, 29, 31, 45). DreamerV3 [16] employs world models to
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Cognitive Predictive Processing (CPP)
Phase Adaptive
Goal
Cognitive Controltar
dage
Fonsitio
Enviromentt
Uncertaintty- Modulated Prediction
enhanced
Dual-Mamory Integration
Octain water
Agant
Phose Ansie- Cognittve Controller
Dual Memory Intogration System
Uncertaiity-Moddue Modulated Prediction Regulator
Exploratory
Approach
Completion
Working Mamorr Ii med ste processing)
Code
Eppoodic Memoey Lonn torm storage?
Forecaust
f
Internal Foreeast
Update
Parcogtual
Compatiion
Reward onstyote
Search
Extenatee scarninn
Expler Expleration
Bélanced
Concentrane Targei
intoge alion
Action
Active Inference
Out tree
Obtain
sand
Ostain
Mine
Iron ore
Exploratory Phase
Approach Phase
Completion Phase
Figure 1: Cognitive Predictive Processing architecture for open- world reinforcement learning. The figure shows our integrated architecture that enhances agent exploration through three key components. Minecraft imagery and content shown in this figure are the property of Mojang Studios and Microsoft Corporation, used for academic illustration purposes.
improve sample efficiency through imagination-basee planning, achieving impressive results across diverse domains. Building on this foundation, LS- LS-Imagine [28] introduces jumpy state transitions guided by affordance maps, extending imagination horizons beyond the short-term constraints of previous approaches. While these methods have advanced the state of the art, they rely on fixed exploration mechanisms and static jump thresholds that cannot adapt to changing task phases. This limitation leads to inefficient exploration patterns and behavioral stagnation, particularly in complex scenarios requiring sustained exploration and strategic reasoning.
Humans excel in open-world environments through sophisticated cognitive processes, and unlike the fixed exploration mechanisms of existing algorithms, human behavior can be characterized by three complementary cognitive strategies that enable adaptive decision-makinn in a cascading process[15, 21]. First, when confronting complex tasks, humans naturally decompose problems into distinct phases [4. 13), transforming global uncertainty into manageable local challenges. This phase- spasess decomposition allows for targeted optimization of exploration parameters tailored to each environmental context, creating foundation for efficient problem-solving. Second, operating within this phase-structured framework, human behavior has been effectively modeled using selective dual-memory systems [39] that dynamically balance innovation and conservation. In these models, working memory focuses on immediate cnvironmental features while episodic memory stores struc- tured experiential. Third, as phases progress, humans have been observed to regulate exploration through predictive processing [41] that establishes adaptive feedback loops. By continuously com- paring expected outcomes with actual environmental feedback, this mechanism dynamically adjusts exploration strategies, increasing precision in familiar contexts while promoting flexibility in novel situations. These integrated cognitive mechanisms, operating in sequential yet complementary man- ner, enable humans to overcome precisely the static exploration strategies and contextual adaptation challenges that limit current Al approaches in open-world environments.
Inspired by human decision-making processes, we present Cognitive Predictive Processing (CPP), a framework that enhances open-world reinforcement learning by integrating principles from cognitive

---

neuroscience [23]. The approach, and completion phases with distinct computational characteristics, monitoring reward patterns and environmental features to determine phase transitions [42]. The dual-memory integration system parallels human working and episodic memory systems [10], providing contextual awareness and selective retention of significant experiences. The ensertainty-modulaae prediction regulator applics hierarchical predictive coding [6]. generating environmental predictions and responding to uncertainty through precision-weighee integration of sensory inputs and internal models. Figure I shows the integrated CPP architecture and how its key components interact to enable adaptive exploration in open-world environments.
cognitive controller segments problems into exploration,
Our contributions include: (1) Cognitive Predictive Processing, a novel framework that integrates numan-inspired decision makinn mechanisms into open- open-world reinforcement learning: (2) a phase- adaptive cognitive controller that draws inspiration from humans' natural problem decomposition into distinct phases, enabling adaptive exploration parameters across different task phases; (3) dual-memon mual-memory integration system inspired by human selective memory systems that balance immediate context with critical historical experiences, providing both short-term working memory and selective long-term storage; (4) an
prediction regulator that leverages principles from human predictive processing that adaptively responds to environmental uncertainty, adjusting explo- ration behavior through environmental prediction. Across multiple resource collection tasks, our framework not only improves success rates by average of 4.6% but also reduces task completion steps by an average of 7.1% demonstrating consistent performance advantages in structured tasks that require contextual adaptation and strategic planning.
2 Related Work
2.1 Open-World Reinforcement Learning
Open-world reinforcement learning confronts challenges of vast state spaces, sparse rewards, and long-horizon tasks. Recent advances in model-based approaches have significantly improved capa- bilities in such environments. DreamerV3 [16] extends earlier world model techniques [17, 18] by learning predictive representations from limited experiences, though these models typically struggle with limited temporal horizons [14]. Several approaches address morizon planning challenges: LS- Imagine [28] introduces jumpy imagination for efficient extended planning, while STEVE-1 1 [31], MineDreamer [46], and VPT [1] leverage language models or human demonstrations to bootstrap policies for complex environments like Minecraft. Despite these advances, exploration in open-world environments remains challenging, with most methods employing fixed exploration mechanisms that fail to adapt to changing task demands [19, 34). Traditional exploration strategies including intrinsic motivation [27, 32] and novelty -sekkigg [36] show promise but employ static parameters and heuristics that cannot adapt to different phases of the learning process [33]. This contrasts with human exploration, which dynamically adjusts strategies based on task progress and environmental feedback. HyperX [49] introduces uncertainty-guidee exploration through novelty bonuses and prediction discrepancies in reiffrreinort learning contexts, though these exploration signals diminish during training and focus on cross-task task adaptation rather than continuous within-tase modulation. Recent approaches like Spatial [37] and skill-oriented frameworks [45] implement more struc- tured exploration but still rely on fixed exploration mechanisms rather than truly adaptive reasoning. Unlike these approaches, our work introduces human-insppn -immared framework that dynamically adjusts exploration strategies based on task progress, directly addressing the gap between static computational methods and flexible exploration in open-world environments.
2.2 Cognitive Mechanisms in Intelligent Systems
Cognitive-inspired approaches have shown considerable success in various AI domains through cognitive architectures like ACT-R [38] and SOAR [25] for structured problem-solving, while memory-augmentedd systems [7, 30] have improved sample efficiency in reinforeement learning tasks. Meanwhile, Botvinick et al. [3] highlight that structured cognitive processes remain crucial for developing capable RL agents. However, these traditional approaches have not been effectively applied for open- spen-world exploration, where environments present unique challenges including vast exploration spaces, sparse rewards, and complex interactions. Human cognition excels in such real-world scenarios through flexible phase decomposition [42], memory integration [5, 24, 39].
3

---

and uncertainty prediction [6], offering valuable inspiration for open-world reinforcement learning. Our work addresses this research gap by integrating cognitive mechanisms into unified framework specifically designed for open-world reinforcement learning. significantly enhancing performance in complex environments that have previously challenged conventional methods.
Cognitive Predictive Processing (CPP) Framework
Environment
Agent Architecture
water,
Input
Affordance Mapping
Phase-Adapiive Cognitive Controller
Vieual Embedding
Ebplersion
Appresion
Palicy Optimization
Reward Memory
Dusl-memory
integration eyslem
Wirking Mericly
Reward
Stagnation
Phase-Adaptive Cognitivee Controller
Exploration Stage (Aggressive jumps)
World Model
Adaptive
Uncertainty Madulatte Prediction Regulater
Approech (Belanced exploration)
Jumping
Completion Stage (Conservative jumps]
Long Short-Tene Term Imagination (Fixed Thresholds)
Long Short-Term Imagination (CPP)
Fixed Thresholds
Aggressive
Conservative
Figure Cognitive Predictive Processing framework. The figure shows our integrated architecture with three key components: Phase-Adaptive Cognitive Controller for mhase-based task decomposition, Dual-Memory Integration System for selective experience utilization, Eevertainty-Modulaaee Prediction Regulator for uncertainty regulation.
Method
3.1 Problem formulation
We formulate cognitive predictive processing in open-world reinforcement learning as enhanced partially observable Markov decision process (POMDP). Our approach operates on visual observa- tions or while generating control signals az, receiving sparse environmental rewards r,, and lacking access to internal environment states.
Our cognitive framework augments the traditional state-action space with a cognitive state vector 4/y (M, o.), where m, represents the dual-memory state and Φ, indicates the cognitive phase. The world model learns tuples {D; (ot, 4t, 4t, MM, the Jı, A., Gt)), where M₁ represents the affordance map. je indicates the jumping flag, A, denotes time steps between transitions, and G represents cumulative discounted rewards.
Our memory system maintains both short-term working memory W, {(or-1, Ke, ) with capacity K, and long-term episodic memory & = {(o;, a;, ri, (r),&)))ee where s; (detailed in Appendix A.2) represents the surprise value and I denotes indices of significant experiences. The surprise value s; quantifies experience significance through reward prediction error, observation prediction error, and cognitive state transitions.
k=
The phase-adaptive cognitive controller decomposes tasks into phases di (dmmplore gapproach, (complete), each with corresponding exploration parameters n())) (weeplore (dh)), hujmpp(pp that control exploration weight and jump bias. Phase transitions follow P( Se, Oc, Tt, Wt, x So(ot, Pt, Ex), where So integrates observations, rewards and memory information.

---

Precision weighting = fw fu( modulates the balance between sensory input and predictions over window of N previous steps. Policy learning occurs on trajectories [(h,, dt, ft. the, jt: At, Gz)]. where hN represents the deterministic recurrent state and 2, rep- resents the stochastic state. The mapping (h+, 2t) (ot, Mc) is performed by the world model's encoder, transforming observations and affordance maps into internal representations.
3.2 Overview of cognitive predictive processing
In this section, we present the details of Cognitive Predictive Processing, which enhances open- open-world reinforcement learning through the following algorithm steps:
1. Phase-adaptive cognitive controller (Sec. 3.3): We continuously monitor task progress through reward patterns and behavioral sequences, dynamically classifying the current state into exploration, approach, or completion phases with corresponding parameter configura- tions.
2. Dual-memory integration system (Sec. 3.4):
We implement a two-stream memory system that combines short-term working memory for immediate context with selective long-term episodic memory for significant experiences.
3. Uncortainty-modulated prediction regulator (Sec. 3.5): The agent maintains a hierarchi- cal predictive model generating expectations about environmental states, with prediction errors driving both learning and adaptive parameter tuning based on uncertainty estimates.
4. Phase-adaptive world modeling (Sec. 3.6): The world model and policy are tested with phase-specific parameters that adapt the current cognitive phase, integrating memory retrievals and predictive signals for context-sensitive decision-making.
5. Data collection: During interaction, we collect enhanced data including cognitive states, memory activations, and prediction errors alongside traditional observations and rewards, feeding this information back into the learning process.
6. Iterate Steps 1-5.
The integrated architecture is illustrated in Figure 2, showing how our cognitive mechanisms interact enhance agent exploration.
3.3 Phase-adaptive cognitive controller
Inspired by human problem-solving strategies, implement cognitive controller that decomposes complex tasks into meaningful phases. This controller continuously monitors task progress and adaptively classifies the current state into one of three cognitive phases:
6 E Exexplore gapproach ccomplete]
(I)
Each phase represents a distinct cognitive mode with specialized parameter configurations that optimize the exploration-exxlootaioo trade-off. For each phase, we define specific exploration parameters:
n((r) exptore ( (immp
where veeppioreedd) controls the exploration weight and hoump (d)) determines the jump bias, with high exploration during the exploration phase, balanced parameters in the approach phase, and low exploration with conservative jump bias in the completion phase.
The phase classification process integrates multiple information sources:
P( 1 lo., or, rt. &,, x fo(or, ras W,,.,)
(3)
where Sa phase classification function, phase transitions are governed by a progress tracking mechanism detailed in Appendix A.I.
5

---

3.4
Dual-memory integration system
Our memory architecture simulates human cognitive processes by implementing parallel memory systems. Working memory provides immediate context while episodic memory stores significant experiences. These systems interact guide decision-makingg
m, (Wi, Ex)
The working memory W; maintains limited capacity buffer of recent experiences:
W/ = (or-k. kk at -, (t-k;
(5)
where = 20 represents capacity constraint simulating human working memory limitations. Work- ing memory provides immediate context for decision-making through similarity- attention mechanism:
(k) = cxp(cos(f(o)),f(e.oo)
(6)
j== cxp(cos(f(o)), f(e.)) f(e.o)))
where cos( ((eoo)) measures the cosine similarity between current observation and episodic memories, f extracts feature representations from observations, and (x, represents the attention weight for experience at time k.
The episodic memory selectively stores experiences based on their surprise value s;
& = E U {(o,, Cr Fr, Cr)) s; T.
where quantifies experience significance through reward prediction error and cognitive state transitions, and 7t (defined in Appendix A.2) dynamic threshold that adapts based on recent experiences.
This dual-modal memory system enables the agent simultaneously leverage immediate context through working memory and historical successes through episodic memory, creating cognitive foun- dation that supports higher-level reasoning processes. Detailed memory retrieval and consolidation mechanisms are provided in Appendix A.2.
3.5 Uncertainty-modulated prediction regulator
Our uncertainty-modulated prediction regulator implements hierarchical model that generates expectations about future states and establishes adaptive feedback loops, where prediction errors only drive learning but also dynamically adjust exploration strategies. The system maintains predictions at multiple levels:
6++1 So(he, ze, Ot, Φt)
(8)
4+1 = Sr(he, 2t, 0t, (t)
where Jo and fr neural network prediction functions that generate next-observation and next- reward forecasts respectively. These functions form the core of the world model's predictive capabil- ities, enabling anticipatory reasoning across different cognitive phases. Prediction enrors between expectations and observations drive both learning and adaptive behavior:
c = ||o 0z112
(9)
E Ir, F.l
These errors are combined into uncertainty estimate that modulates exploration parameters:
Us Wo E. Wr Ef + Wtrend Eyrend
(10)

---

where frrend represents the reward trend enror (formally defined in Appendix A.3). Higher uncertainty encourages broader exploration, while lower uncertainty enables more focused behavior. This mecha- nism adapts jump thresholds based on environmental predictability, fostering effective exploration in new areas while exploiting familiar regions. The detailed implementation uncertainty estimation and parameter adaptation is provided in Appendix A.3.
3.6 Phase-adaptive world modeling
Our integrated cognitive architecture enhances both world model learning and policy optimization by incorporating phase-specific parameters, memory retrievals, and uncertainty estimates into a unified framework. The world model architecture extends typical model-based reinforcement learning with cognitive conditioning:
2. po no (1)
f, ~ po (fr hr 2z, Øt)
(11)
o pa (ō: hi 24, 22)
Each component is conditioned on the current cognitive phase ot, enabling specialized processing different reasoning phases. The jumping mechanism incorporates phase-specific thresholds and uncertainty estimates to adaptively balance exploration:
j. = [max((a, > 0iump(( (6.,
(12)
where j. is the jumping indicator and @ is the adaptive threshold described in Appendix A.3. Memory retrievals provide relevant historical context for current decisions:
R₂ = fe where j argsort (cos( f()))))
(13)
For behavior learning, we adapt the actor-critic approach with memory- and phase-aware components:
a2 To 21, Re, Φi) V uy (he, 2t, R2, 9+)
(14)
This integrated approach transforms how the agent processes information and makes decisions across different task phases, enabling adaptive exploration that becomes increasingly focused as tasks progress from exploration to completion.
4 Experiments
This section details our comprehensive evaluation of the Cognitive Predictive Processing framework in open-world reinforcement learning scenarios. Our evaluation focuses on two key objectives: (1) comparing CPP against state-of of-thoort approaches in overall task performance, and (2) analyzing the contribution of individual cognitive components agent capabilities.
4.1
Experimental setup
4.1.1
Benchmark and tasks
We evaluate our approach on the challenging MineDojo benchmark [9], which provides diverse set of open-world tasks in Minecraft. Following previous work, we focus on five representative tasks requiring different exploration and interaction strategies: (1) harvest log in plains, (2) harvest water with bucket, (3) harvest sand, (4) shear sheep, and (5) mine iron ore. For fair comparison, we maintain the same environmental configurations across all methods, including identical starting conditions, maximum episode lengths, and reward structures as detailed in Table 1, All experimental results represent averages from three independent runs to ensure statistical reliability.

---

Table 1: Details of evaluation tasks in MineDojo benchmark.
Task
Description
Initial Tools
Initial Environment
Max Steps
Harvest log in plains
"Cut tree"
None
Plains biome
1000
Harvest water
"Obtain water"
Bucket
Near water source
1000
Harvest sand
"Obtain sand"
None
Near beach
1000
Shear sheep
"Obtain wool"
Shears
Sheep within 15 blocks
1000
Mine iron ore
"Mine iron ore"
Stone pickaxe
Underground cave system
2000
Table 2: Success rates (%) across MineDojo tasks (mean + std over runs). Results for VPT, STEVE-1, and DreamerV3 are from LS -S-mmaiine [28], while LS -S--magine and CPP results represent our evaluations.
Model
Cut tree
Obtain water
Obtain sand
Obtain wool
Mine iron ore
VPT [1]
6.97
0.61
12.99
1.94
0.00
STEVE-1 [31]
57.00
6.00
37.00
3,00
0.00
DreamerV3 [16]
53.33
55.72
59.88
25.13
16.79
IS--nnee [28]
88.3343.21
75.67=3.93
57.33=4.66
56.67:4.43
10.66+1.55
CPP (Ours)
90.00+2.65
80.33+3.79
60.00
69.33+6.04
12.00+1.8.
4.1.2 Baselines and implementation
We compare CPP against state-of-the-art methods in open-world reinforcement learning: DreamerV3 [16], LS-Imagine [28], STEVE-1 [31]. and VPT [1]. Our implementation builds upon the LS- Imagine codebase, integrating our cognitive components while maintaining the core world model architecture. All models are trained for x 10 environment steps. Detailed implementation specifics and hyperparameter settings are provided in Appendix B.
4.2
Performance evaluation
4.2.1 Overall task success
Table demonstrates that CPP outperforms baseline methods in the majority evaluated tasks. the harvest log task in plains, CPP achieves 90.00% success rate compared 88.33% for LS LS-Imagine, representing 1.9% relative improvement. The performance differential increases in the harvest water task, where CPP reaches 80.33% success rate compared to LSS LS-Imagine's 75.67%, constituting 6,2% relative improvement. For the obtain sand task, CPP delivers a consistent improvement of 4.7% beyond Imagine. The most significant performance gain appears in the obtain wool task with 22.3% relative improvement, demonstrating CPP's efficacy in scenarios involving mobile targets. Unlike static resources in other tasks, sheep movement introduces additional uncertainty, requiring continuous adaptation of exploration strategies and contextual awareness to track and approach dynamic objectives.
In the mine iron ore task, DreamerV3 achieves 16.79% success rate compared to CPP's 12.00%. This specific difference stems from DreamerV3's architecture that particularly excels in enclosed spaces with limited visibility. Despite this, CPP still outperforms LS-Imagine in this complex scenario, demonstrating 12.6% relative improvement. Across all other tasks, CPP consistently outperforms both DreamerV3 and I-agine,,,, confirming our framework's effectiveness in most open-world environments while revealing specific optimization opportunities for future work.
4.2.2 Sample efficiency
Beyond success rates, Table reveals CPP's efficiency characteristics across different task types. CPP excels in structured interaction tasks, achieving first success in the harvest log task in just 340.48 steps compared to 363.27 steps for LS-Imae and 71..22 steps for DreamerV3, a 52. % reduction versus the latter. This efficiency extends to water collection (3.6% reduction) and wool gathering (18.2% 2% reduction), where CPP's phase-adaptive controller effectively transitions between exploration modes. DreamerV3's fixed exploration parameters prove advantageous in homogencous resource

---

Table 3: Average steps to first success across MineDojo tasks (mean + std over 3 runs). Results for VPT, STEVE-I, and DreamerV3 are from LS Imagine [28], while LS-Imagine and CPP results represent our evaluations.
Model
Cut tree
Get water
Get sand
Get wool
Mine iron
VPT [1]
963.32
987.65
880.54
987.49
STEVE-1 [31]
752.47
989.07
770.40
992.36
DreamerV3 [16]
711.22
628.79
548.76
841.14
1789.06
LS-Imaginee [28]
363.27=28.45
473.05=35.67
612.27+42.18 18
558.04+31.27
1874.07=127.35
CPP (Ours)
340,48422.13
455.81+29.84
567.84238.76
456.45+39.52
1867.53:115.66
Table Ablation results showing success rates (%) across tasks (mean + std over 3 runs). "PACC": Phase-Adaptive Cognitive Controller. "DMIS": Dual-Memory Integration System. "UMPR":
Uncertainty-Moduulteee Prediction Regulator.
Model Variant
Cut a tree
Obtain water
Obtain sand
Obtain wool
Mine iron ore
Full CPP
90.00+2.65
80.33+3.79
60.00+4.58
69.33+6.04
12.00
w/o PACC
84.33+4.16
72.67=5.28
54.33+6.12
52.00+7.35
7.33#3.21
w/o DMIS
86.67+3.51
75.00=4.73
57.67=5.46
61.67+5.89
8.67+3.06
w/o UMPR
87.33+3.28
76.67+4.52
55.00
63.00#5.24
9.33+2.94
identification tasks like sand collection (548.76 76 versus 567.84 steps) and iron mining (1789.06 versus 1867.53 steps). Notably, water collection shows different patterns: water bodies possess distinctive visual features (blue coloration, reflective properties, flowing patterns) that create strong perceptual signals our dual-memory system effectively encodes and retrieves, whereas sand blocks have more subtle visual characteristics that can be confused with similar terrain elements, making Dreamer V3's consistent exploration more effective for such visually ambiguous but static resources. This complementary performance highlights CPP's design intention: while DreamerV3 excels in direct resource identification through consistent exploration, CPP prioritizes adaptive behavior across changing task phases. This tradeoff delivers superior performance in most realistic scenarios requiring contextual adaptation, especially in complex tasks with dynamic elements. Different architectural approaches optimize for different environmental characteristics, explaining the observed performance patterns. Additional trajectory analysis in Appendix C further illustrates these task-specific efficiency patterns.
4.3
Ablation studies
To understand the contribution of each cognitive component, we perform ablation studies by sys- tematically removing individual mechanisms from the full CPP framework. Table presents these results across all tasks. Removing the cognitive controller produces the largest performance drops, particularly complex tasks with 24.99% relative decrease in shear sheep task. The dual--meory system shows substantial impact on water collection (5.33 percentage points) and shear sheep (7.66 percentage points) tasks, reflecting its importance for contextual adaptation. While our uncertainty quantification uses weighted linear combinations with fixed parameters rather than full Bayesian approaches, this design choice balances computational efficiency with biological plausibility, as neuroscience evidence suggests that uncertainty representations in biological systems often manifest as relatively simple combinations of prediction errors [48]. The consistent performance contributions across diverse tasks validate this simplified yet effective approach.
Each ablated variant maintains reasonable performance, demonstrating that individual cognitive mechanisms provide complementary capabilities. However, the full integration yields optimal results across all task categories, confirming that human-inspired cognitive mechanisms work synergistically in open- world reinforcement learning.

---

Conclusions and limitations
Our experimental results demonstrate that Cognitive Predictive Processing enhances reinforcement learning in open-world environments through three synergistic components: phase-adaptive cogni- tive controller for task decomposition, dual-memory integration system for experience utilization, and an uncertaintymmodulteed mooduated prediction regulator for uncertainty regulation. Compared to state-of- the-art methods, CPP significantly improves success rates (by 6.2% in water collection and 22.3% in wool collection tasks) while reducing completion steps by up 6.3%, confirming the effectiveness of integrating human-inspired cognitive principles into reinforcement learning systems.
Despite these advances, several limitations remain. A notable limitation appears in the mine iron ore task, where our method did not match DreamerV3's performance, suggesting that complex navigation in confined spaces with limited visibility requires specialized mechanisms beyond our current implementation. Additionally, our three-phase decomposition framework (exploration, approach, completion) is well-suited for goal-directed tasks with spatial components, but may face challenges in certain task categories: highly unstructured environments where phase boundaries blur significantly, extremely long-howe long-horizon tasks requiring numerous intermediate subgoals where simple three-phase decomposition may be insufficient, and highly adversarial settings requiring rapid replanning due to dynamic obstacles.
Future work could address these limitations through enhanced spatial reasoning components, broader enviromment evaluations, and closer integration with neuroscientific models. A particularly promising direction involves enabling adaptive learning rules where the system could "learn how to learn" through meta- eeta-gradient approaches, allowing learning parameters to evolve based on accumulated task experience rather than remaining static [47]. Furthermore, extending CPP complex multi-stage goals (such as crafting tasks requiring sequential resource collection) would leverage our dual-memory system's ability to organize experiences across subtasks, while adaptation goal-conditioned settings could be achieved by conditioning phase classification and memory retrieval on goal embeddings. Extending CPP real-world applications, particularly in domains such as assistive robotics where adaptive exploration strategies could benefit navigation and manipulation tasks in human-centered environments, represents an exciting and impactful future direction for this research.
Acknowledgement
This work supported by the State Key Program of National Natural Science of China (Grant No. 62236001), Hebei Natural Science Foundation (Grant No. F2024105033), and Beijing Municipal Natural Science Foundation (Grant No. L247011). 1)..
10

---

References
[1] Baker B. Akkaya I, Zhokov P. et al. Video pretraining (vpt): Learning act watching unlabeled online videos[J]. Advances Neural Information Processing Systems, 2022, 35: 24639-24654.
[2] Bavard Lebreton M, Khamassi M, Reference point centering range adaptation enhance human reinforcement learning at the cost irrational preferences[]]. Nature communications, 2018, 9(1): 4503.
[3] Botvinick M. Ritter Wang X, et al. Reinforcement learning. and slow[J]. Trends cognitive sciences, 2019, 23(5): 408-422.
[4] Botvinick M M. Niv Y. Barto A G. Hierarchically organized behavior and its neural foundations: reinforcement learning perspective[]]. cognition, 2009, 113(3): 262-280.
[5] Chandra S, Sharma S, Chaudhuri R, et al. Episodic and associative memory from spatial scaffolds in the hippocampus(J]. Nature, 2025: 1-13.
[6] Chang H. Tang W. Wulf A M, et al. Sleep microstructure organizes memory replay[J]. Nature, 2025: 1-9.
[7] Chateau-Laurent Н. Alexandre F. Contextual Control of Hopfield Networks in a Hippocampal Model[[]//Proceedingg of the Annual Meeting of the Cognitive Science Society. 2024, 46.
[8] Fan L, Liang M. Li Y. et al. Evidential active recognition: Intelligent and prudent open-world embodied perceptiontC//PPoccceee CC//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 16351-16361.
[9] Fan L, Wang G, Jiang Y. al. Minedojo: Building open-ended embodied agents with internet-scale knowledge[J]. Advances Neural Information Processing Systems, 2022, 35: 18343-18362.
[10] Fan Y. Wang M. Fang et al. vio-dimensional neural geometry underpins hierarchical organization of sequence in human working memory(J]. Nature Human Behaviour, 2025, 9(2): 360-375.
[11] Feng Wang X, Zhou Z, et al. EvoAgent: Agent Autonomous Evolution with Continual World Model for Long- Long-Horizon TJsks[J]. arXiv preprint arXiv:2502.05907 2025.
[12] Fujimoto T, Suetterlein J. Chatterjee S, et al. Assessing the impact of distribution shift on reinforcement learning performance(]]. arXiv preprint arXiv:2402.03590, 2024.
[13] Gershman $ J. Deconstructing the human algorithms for exploration[J]. Cognition, 2018, 173: 34-42.
[14] Guo Z, Thakoor S, Pislar M. et al. Byol-explore: Exploration by bootstrapped prediction[J]. Advances in neural information processing systems, 2022, 35: 31855-31870.
[15] Hassabis D, Kumaran D, Summerfield C, et al. Neurosciencc-inspiree artificial intelligence(J). Neuron, 2017, 95(2): 245-258.
[16] Hafner D, Pasukonis J. J, et al. Mastering diverse control tasks through world models[J]. Nature. 2025: 1-7.
[17] Hafner D. Lillicrap T, Ba J. et al. Dream control: Leamnnng behaviors latent imagination(J], arXiv preprint arXiv: 1920v33912.00603, 2019.
[18] Hafner D. Lillicrap T. Norouzi M. et al. Mastering atari with discrete world models[J]. arXiv preprint arXiv:2010.02193, 2020.
[19] Hafner Pasukonis J, al. Mastering diverse domains through world models[J]. arXiv preprint arXiv:2301.044 1044230 2023.
[20] J, Lepori A. Franke Linking forward-pass dynamics Transformers and real-time human processing[J]. arXiv preprint 1rXiv.2504.14107, 2025.
[21] Lee D. Seo H. Jung M W. Neural basis of reinforcement learning and decision making[J]. Annual review of neuroscience, 2012, 35(1): 287-308.
[22] Kaefer K. Stella F, McNaughton L. et al. Replay, the default mode network and the cascaded memory systems model[J]. Nature Reviews Neuroscience, 2022, 23(10): 628-640.
[23] Kriegeskorte N, Douglas P K. Cognitive computational neuroscience(J] Nature neuroscience, 2018, 21(9): 1 1148-1160 1160.

---

[24] Kveim V A. Salm L. Ulmer T. et al. Divergent recruitment of developmentally defined neuronal ensembles supports memory dynamics(J]. Science, 2024, (87(66) eadk0997.
[25] Laird E. The Soar cognitive architecture(M]. MIT press, 2019.
[26] LeCun Y. A path towards autonomous machine intelligence version 0.9. 2022-06-27[J]. Open Review, 2022, 62(1): 1-62.
[27] Li Yang Wang Z, et al. Auto me-reward: Automated dense reward design with large language models for minecraft( (C/prooccdcs// of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 16426- 16435.
[28] Li J. Wang Q, Wang Y, et al. Open- Open-World Reinforcement Learning over Long Short-Term Imagination[C]. [C] ICLR, 2025.
[29] Li Xu X. Xu Z. et al. LARM: Large Auto-Regressive Model for Long- Morizon Embodied Intelligence(J]. arXiv preprint arXiv:2405.
2024.
[30] Li Zhu D, Hu Y, et al. Neural episodic control with state abstraction[]]] arXiv preprint arXiv:2301.1 14990, 2023.
[31] Lifshitz S, Paster K, Chan H, al. Steve-1: generative model for text-to-behavior minecraft[J]. Advances in Neural Information Processing Systems, 2023, 36: 69900-69929.
[32] Liu B, Li X, Zhang J, et al. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems[J]. arXiv preprint Si5i:25504001990 2025.
[33] Luo S, Jiang M. Zhang S. et al. Experiment-free exoskeleton assistance via learning in simulation[]]. Nature, 2024, 630(8016): 353-359.
[34] Meng Y. Bing Z. Yao X. et al. Preserving and combining knowledge in robotic lifelong reinforcement learning[J]. Nature Machine Intelligence, 2025: 1-14.
[35] Parmar J, Chouhan Raychoudhury V, et al. Open-world machine learning: applications, challenges, and opportunities(]] ACM Computing Surveys, 2023, 55(10): 1-37.
[36] Peng X B, Guo Y. Halper L, et al. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters[]]. ACM Transactions On Graphics (TOG), 2022, 41(4): 1-17.
[37] Qu D. Song H, Chen Q, al. SpatialVLA: Exploring Spatial Representations for Visual-Language- Model[J]. arXiv preprint arXiv:2501. 155300 2025.
[38] Ritter F F, Tehranchi F, Oury J D. ACT ACT-R: R: A cognitive architecture for modeling cognition[]]. Wiley Interdisciplinary Reviews: Cognitive Science, 2019, 10(3): e1488.
[39] Sun W, Advani M. Spruston N, et al. Organizing memories for generalization in complementary learning systems[J]. Nature neuroscience, 2023, 26(8): 1438-1448.
[40] Tamura M, Spellman
Rosen A M, et al. Hippocampal-prefroon theta-gamma coupling during performance of spatial working memory task(J]. Nature communications, 2017, 8(1): 2182.
[41] Trudel N. Scholl J, Klein-Fliigee M C, et al. Polarity of uncertainty representation during exploration and exploitation in ventromedial prefrontal cortex[J]. Nature Human Behaviour, (021,5(1) 83-98.
[42] Wang Y, Chiew V. On the cognitive process of human problem solving[J]. Cognitive systems research, 2010, 11(1): 81-92.
[43] Wittmann M K, Lin Y, Pan D, et al. Basis functions for complex social decisions in dorsomedial frontal cortex[J]. Nature, 2025: 1-11.
[44] Yang S. Nachum O. Y. et Foundation models for decision making: Problems, methods, and opportunities(J] arXiv preprint MEXiv:2303.04129 2023.
[45] Yuan H, Zhang C, Wang H. et al. Skill reinforcement learning and planning for open-world long-horizon [J. arXiv preprint arXiv:2303. 111.2333...63,, 2023.
[46] Zhou E. Qin Y. Yin Z, et al. Minedreamer: Learning to follow instructions via chain- of-in-o-iing ofaaiinattioon for simulated world control[J]. arXiv preprint 1000:2403..2007,, 2024.
12

---

[47] Sousa M. Bujalski P. Cruz B F, et al. A multiddimensional distributional map of future reward in dopamine neurons[J]. Nature, 2025: 1-9.
[48] Trudel N, Scholl J, Klein-Flüige M C, et al. Polarity of uncertainty representation during exploration and exploitation in ventromedial prefrontal cortex[J]. Nature Human Behaviour, 2021, 5(1): 83-98.
[49] Zintgraf L M. Feng L, Lu C, et al. Exploration in approximate hyper-state space for meta reinforcement Learning[[[/itermaaaeee Conference Machine Learning. PMLR, 2021: 12991-13001.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and introduction accurately present our contributions: CPP framework with three components, performance improvements (4.6% success rate increase, 7.1 % step reduction), and applications. These claims are supported by experiments in Sec. 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Sec. acknowledging simplified cognitive approximations and performance limitations in complex navigation tasks. Additional failure mode analysis is provided Appendix C with visualizations Figures 4-5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Mathematical formulation is presented in Sec. (equations 1-14) with imple- mentation details in appendices (equations 15-28). All components are formally specified with proper notation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification; Sec. 4.1 and Appendix B details the MineDojo benchmark, tasks, and baselines. Appendix B contains hyperparameters (Table 5) and environment setup. Algorithm 1 provides complete pseudocode.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We provide detailed algorithmic descriptions (Algorithm 1), mathematical formulations (Sec. 3, equations 1-14), hyperparameters (Table 5), and implementation
details (Appendix B) sufficient for reproduction. The code will not be publicly released due to commercial considerations.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
mublic/guides Sewiidde/Codeuumissseee wtt.:/uwwmsssooooiiioooooooooo.e oooiillyyy)))) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. /t/pc//ppllc..... hwtp//g//////eee///e Sttpss//ooSSbiiiiseeee ioammissionPp sissssoliccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Sec. 4.1 describes evaluation tasks and baselines. Training details are in Appendix B. Hyperparameters are in Table 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
16

---

Justification: We report standard deviations for all main experimental results based on three independent runs with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix B.2 specifies "NVIDIA L40s GPUs" and "1 10 environment steps" for training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https :tops:////urrrr /tt.//neurr/////e httpp..cppppppe cc/public/ Cttollic//tiiittoeeee wtw.hhhhchieeeeeeeeeeee
Answer: [Yes]
Justification: We followed the NeurlPS Code of Ethics. Research focuses on reinforcement learning gaming environments without sensitive data or harmful applications. Minecraft content is appropriately credited to Mojang/Microsoft.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
17

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work is not related to any private or personal data, and there's no explicit negativesocial impacts. This research advances fundamental Al capabilities without direct harmful applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Paper does not release models or data with misuse risks. Research focuses on reinforcement learning Minecraft environments only.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
18

---

Justification:
Minecraft content attribution in Figure 1. Moje benchmark [9] properly cited, and all baselines (DreamerV3, LS- -m-mmagine, STEVE-1, VPT) referenced [16]-[1].
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer: [NA]
Justification:
No human subjects involved; all experiments conducted in simulation environ- ments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Research methodology is based on reinforcement learning with cognitive mechanisms; no LLMs used.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: httpps://neurrii htterrppp.ccccoofeeenneeeeee///e /LLM) for what should or should not be described.
20

---

A Detailed Implementation of Cognitive Mechanisms
A.1 Phase-Adaptive Cognitive Controller
Phase transitions in the cognitive controller are governed by a progress tracking mechanism that integrates reward trends and memory activations:
-i--- max (os(((o))) ((e.o))
Progress,
(15)
cEE.
where (oe)) ((.oo) measures the similarity between current observation and episodic memories, k is the reward averaging window size. When progress exceeds phase-specific thresholds, the system transitions to the next cognitive phase:
papproack if 0i-1 desplore
djeploree and Progress, > reeplore
Φ
gromplete
if Φt-1 gapproach and Progress, papproach otherwise
(16)
where peppoore 0.4 and rapproack 0.7 are threshold values for phase transitions, The system implements stagnation detection based on consecutive low-reward episodes:
Stagnation, =
Σ
Tr, $)
> A
(17)
where m = 50 is the detection window, 0.05 represents the reward significance threshold below which steps are considered non-prodee coooruduetive, and X 0.8 the required proportion of low-reward steps. Upon detecting stagnation, the system reverts to exploration:
d x ceplorre if Stagnation, =
(18)
This dynamic adaptation ensures recovery from unproductive behavioral patterns by resetting to exploratory strategies when necessary.
A.2 Dual-memory integration system
The surprise value for episodic memory encoding quantifies the significance an experience through three components:
Ir: Pel + |lo, 6:ll2 + I + Φ:-1]
(19)
where fr is the predicted reward based current policy, Or is the predicted observation, and [[ A Φ: is an indicator function that equals when a cognitive phase transition occurs.
The surprise threshold 7; adapts based on a moving average of recent surprise values:
7 = 3 T(-1 + (1 3) median((s.-x1
(20)
where B = 0.9 is a smoothing factor and K, = 50 is the window size for computing the median surprise. This mechanism ensures that the agent improves, only increasingly significant experiences are stored.
Episodic memory retrieval operates through similarity-bart -baedd access:
sim e.o, dr. c.d) (o)) + + (1-7) I[0 c.6)
(21)
where ʃ extracts feature representations from observations, cos computes cosine similarity, and 7 = 0.7 balances the importance observation similarity versus cognitive phase matching. The top-K retrieval operation selects the most relevant episodic memories:
21

---

R. {ez1 ) where j e argsort eec, (sim(o,, c.o, 92, e.o))
(22)
with K, 5 representing the number of retrieved memories. These retrieved memories provide relevant historical context that enhances decision makinn, particularly during similar situations or cognitive phases encountered previously.
A.3
Uncerrainnty-modulaeed prediction regulator
Our uncertainty-modulaeed prediction regulator calculates prediction errors across multiple channels estimate environmental uncertainty:
E |1o, 6
E
|r frl
(23)
Strend
Σ
Tthreshold
N
i=t-N.
where N, 20 is the reward history window size and "threshold 0.05 is the reward significance threshold. These errors contribute to the overall uncertainty estimate:
24 Wo EZ + Wr Ef + Wtrend Eirend
(24)
with fixed weights w. 0.5, w, = 0.3, and Wtrend = 0.2.
Uncertainty directly modulates the jump threshold used for extended planning:
giump giump ( T, S
(25)
where 4)ump (ox) represents Stase-specifie base thresholds as detailed in Table 5.
The trend adjustment T, responds to recent reward patterns:
0.1
if < 0.05
T
0.1
if > 0.2
(26)
0.0
otherwise
where F. is the moving average recent rewards, computed as the arithmetic mean over a sliding window of the 20 most recent reward signals.
The stagnation adjustment increases linearly with consecutive no-reward steps:
S, min 0.01 c.)
(27)
The stagnation counter ot tracks periods without significant rewards:
if > 0.05
C
(28)
a--+1 +1 otherwise
This adaptive threshold mechanism encourages broader exploration during periods of high uncer- tainty or reward stagnation, while enabling more focused behavior when the environment becomes predictable.
22

---

Table 5: Key hyperparameters of the Cognitive Predictive Processing framework.
Component
Parameter
Value
Exploration Phase Threshold (repooree)
0.4
Approach Phase Threshold (poproaach)
0.7
Exploration Weight (w e
expplore
0.8
Cognitive Controller
Exploration Weight (w expplree
apprroach
0.5
Exploration Weight (w (oanerre
0.2
Stagnation Detection Window (m)
50
Base Jump Threshold
0.6
Jump Threshold Explore Phase (frage (graplore))
0.5
Jump Threshold Approach Phase (Omame (gspproach))
0.6
Jump Threshold Complete phase (0) (o (dcomplete)
0.7
Working Memory Capacity (K)
20
Dual-Memory System
Surprise Threshold Base (Tras)
0.3
Observation Error Weight (w,)
0.5
Prediction Regulator
Reward Error Weight (w,)
0.3
Transition Error Weight (Wtrend)
0.2
B Detailed Experimental Setup
B.1 Environmental Configuration Details
We configured the MineDojo environment with standardized parameters across all evaluation tasks to ensure fair comparison between methods. All environments shared common settings including appropriate break speed multipliers (100), consistent reward structures point upon target acquisi- tion), and task-specific prompts for affordance map generation. Tasks were initialized with relevant starting conditions as described in Table 1, with identical environmental parameters maintained across all compared methods. For each task, utilized dedicated fine-tuned UNet models to generate approprrtee affordance maps, with fast reset disabled for consistency. All experimental results averaged across three independent runs with same random seeds ensure statistical reliability.
B.2 Baseline Implementation Details
Our evaluation methodology utilized the LS-IImginee codebase the implementation foundation for our CPP framework. For rigorous comparative evaluation, we directly incorporated the reported performance metries for DreamerV3, VPT, and STEVE-1 from the LS-IImgine paper [28], rather than re-implementing these methods. To ensure fair comparison, we implemented CPP using identical training procedures, hyperparameters, and model architecture components described in the LS-Imagine repository, maintaining consistency in imagination horizon (15 steps), network dimensions (encoder: cnn-depth 96, mip-units 1024; decoder: same configuration), optimizer settings, and training duration (1 x 10 steps). This approach allowed us to isolate the performance improvements attributable our cognitive components while maintaining methodological consistency with established benchmarks, and all experiments were conducted on NVIDIA L40s GPUs.
B.3 Hyperparameter Configuration
Our CPP implementation integrates the cognitive components with carefully selected parameters detailed in Table 5. These parameter selections were informed by cognitive science principles that model human problem-solving strategies. For other world model parameters, we used the same configuration as the Imagine baseline to ensure fair comparison across all evaluated methods, including network dimensions, optimizer settings, and training duration.
23

---

Obtain
Octain
sand
Obtain
wool
Vine ron
Exploratory Phase
Approach Phase
Completion Phase
Figure 3: CPP agent behavior across five MineDojo tasks. Each task progression shows three cognitive phases: exploration (left), approach (middle), and completion (right). Top rows show agent observations while middle rows display affordance maps.
C Extended Performance Analysis
C.1 Trajectory Analysis with Affordance Maps
Figure showcases CPP's adaptive exploration through affordance maps that evolve from diffuse patterns during exploration concentrated activation during completion. This progression directly correlates with the decreasing exploration weight parameters across cognitive phases. The obtain wool task demonstrates particularly clear transitional patterns. Importantly, when target objects initially occluded (such as sheep obscured by forests), the stagnation detection mechanism (Equations 17-18) identifies lack of progress and maintains the exploration phase with diffuse affordance patterns until visual detection occurs, at which point the maps transition to concentrated activation. These visualizations provide evidence that CPP's cognitive mechanisms enhance exploration coherence, supporting the quantitative performance improvements observed.
C.2 Completion Trajectory Analysis
Figure 4 illustrates CPP failure modes while Figure 5 presents LS-IIm failure cases. In the harvest log task (top row), the agent becomes trapped in terrain cavities (highlighted in red), rendering affordance mapping ineffective. The obtain sand task (middle row) failures occur when environmental generation creates water bodies without adjacent sand blocks. In the obtain wool task (bottom row), the agent fails identify a black sheep (highlighted in red) despite direct visibility, with affordance map activation indicating perceptual models trained on white sheep failed to generalize to variant appearances. LSS Imagine exhibits different failure patterns, predominantly exploration timeouts across all tasks. Most notably in the water collection task (red highlight), LS-Imagine identifies water sources but exhibits behavioral stagnation, repeatedly executing similar actions without progress. This highlights critical advantage of CPP's stagnation detection mechanism, which dynamically resets exploration parameters when detecting repetitive behaviors. The comparative analysis reveals that while CPP enhances exploration efficiency, the failure modes demonstrate remarkable parallels

---

Octain
sand
Figure 4: CPP failure modes across tasks. Red highlights indicate critical failure points: terrain cavity in the harvest log task preventing further exploration; water without adjacent sand in the obtain sand task challenging environmental expectations; and unrecognized black sheep in the obtain wool task revealing perceptual limitations after training primarily with white sheep.
to human cognitive processing errors. Similar to how human and Transformer processing are affected by equivalent stimulus properties [20], CPP's inability to recognize black sheep becoming trapped in terrain cavities mirrors human perceptual biases and environmental fixation.
C.3 Reward Pattern Analysis
experimental tasks, revealing three key ad- vantages of CPP. First, CPP achieves initial high-value rewards more rapidly (within first 50 steps), establishing effective exploration trajectories earlier. Second, during middle phases (steps 400-700) when both methods encounter "reward deserts," CPP maintains more frequent non-zero signals, demonstrating the effectiveness of dual-memory system in guiding exploration during limited feedback. Finally, later phases (steps 700-1000), CPP exhibits more continuous, goal-oriented reward patterns with concentrated high-values between steps 800-900, while LS-Imagine displays more intermittent peaks. This confirms how our cognitive phase transition effectively shifts from ex- ploration to completion phases. The MineCLIP score comparison (Figure 6 quantitatively validates these observations, showing CPP's superior performance with statistically significant improvements (p < 0.05) in the water collection and sheep shearing tasks, where overllpppin error bars indicate reliable performance advantages.
Figure 6 presents intrinsic reward patterns
D Hyperparameter Sensitivity Analysis for Cognitive Predictive Processing
We conducted sensitivity analysis on two critical parameters of our CPP framework to validate design choices and assess robustness. For exploration weight parameters (Figure 7 a), we tested five configurations controlling phase-progression patterns. Our default implementation (B: 0.55-.5))) achieves optimal performance with 90.0% success rate and 340 completion steps. Configurations with parameters (D) or reverse progression (E) show significantly degraded performance, confirming that phase-based adaptation is essential for effective task completion.
25

---

Cul a
Obtan
Mine irnn ore
Figure 5: LS- LS-Imagine failure modes across tasks. Red highlight in sand collection task shows behavioral stagnation where the agent fails to progress despite locating sand.
LS-Imagine
(Ours)
(a) Harvest log in plains
(b) Harvest water
(c) Harvest sand
(d) Shear sheep
(e) Mine iron ore
(f) Comparison in MineCLIP reward
Figure 6: Reward pattern comparison between CPP and IS-Inagine. CPP (red) shows earlier high-value rewards, consistent signals during exploration phases, and more concentrated reward patterns later phases compared to LS-Imagine (blue).
26

---

Eepad Jousert DARI
Fspal Seryne Packranna
Lucattion Hee'' Colfer Sia Lusher--hee -C780 -Garon
A1 383
(a)
(b)
Figure 7: Hyperparameter sensitivity analysis. (a) Exploration weight parameters across five configurations, showing success rates and completion steps. Configuration represents our default implementation. (b) Surprise threshold base impact success rate, with vertical line indicating default value (0.3). All analyses conducted on the harvest log plains task.
The surprise threshold base parameter governs episodic memory selectivity and exhibits an inverted U-shaped relationship with performance (Figure b). The default value of 0.3 achieves peak success rate, while values that are too low (0.1-0.2) or too high (0.5-0.7) both reduce performance through different mechanisms. These results validate our parameter choices and demonstrate that CPP's cognitive mechanisms operate optimally within specific ranges that align with neurologically-insprree principles.
E
Algorithm Pseudocode
This appendix provides the complete pseudocode for the Cognitive Predictive Processing framework for open-world reinforcement learning.
27

---

Algorithm Cognitive Predictive Processing Framework
1: Initialize world model parameters 0, policy parameters vh, working memory W/ 0, episodic memory & = 6, cognitive phase Φ demplore precision weights w (Wor Wpy Werend))
2: Initialize replay buffer B with random experience.
3: while not converged do
4: Sample transitions from B.
5: Update world model parameters
6: for episode do
7: Receive initial observation Og.
8:
Initialize states ho, 20.
9:
for time step = 0... T until termination do
Compute Progress, using Eq. (15).
11:
Update cognitive phase Ø₁ according to Eq. (16).
Check for stagnation via Eq. (17) and reset phase if needed via Eq. (18).
13:
Set exploration parameters n7(x)) = (w (Øt), bump (Φt)) from Eq. (2).
Update working memory W, {(o at-s -t=x) per Eq. (5).
15:
Compute surprise value s, using Eq. (19).
Adapt surprise threshold 7, using Eq. (20). if st > then
17:
18:
Update episodic memory & E U {(or, ae, Tr, 8t)) via Eq. (7). end if
19:
Retrieve relevant memories R2 using similarity function in Eq. (21), Eq. (22).
21:
Generate predictions ō fi41 using Eq. (8).
22:
Compute prediction errors E8, E Etren via Eq. (23).
23:
Calculate uncertainty U4 = Wo Eg w. ET Wtrend Strend from Eq. (24).
Adjust jump threshold @iump using Eq. (25, 26, 28).
25:
Determine jump flag j: = [[maa((a)) > 0fump (Øc, Ut)] via Eq, (12).
26:
Select action de (ae (h, 2,, Rr, 0x) using Eq. (14).
27:
Execute action a,, observe 01+1, Tr+1-
28:
Update cognitive state the (me, Φ1) where my = (W,, E,) per Eq. (4).
29:
Store transition (Ot, at, Ft, Mr, Ve. A., G.) in
30:
if j: 1 (Jumpy transition detected) then Construct long-term data with jumpy transition information.
31:
32:
end if
33:
end for Sample transitions batch from B.
34:
35:
Update world model by conditioning on cognitive phase d, via Eq. (11). Compute phase-specific value function V, Vp 2t, Rr, dr) via Eq. (14).
36:
37:
Update policy with phase-specific entropy regularization.
end for
39:
end while
28


Reviews:
[{"review_id": "2ILtynNikS", "reviewer": "Reviewer_fxzZ", "summary": "* Introduces cognitive predictive processing (CPP) framework inspired by human decision-making, for the purpose of improving exploration in open-world environments\n* Involves three components: (1) module that decomposes tasks into phases, (2) a dual-memory integration system, and (3) an uncertainty-aware prediction module\n* Evaluated on MineDojo, demonstrating improved performance on resource collection tasks", "questions": "* How does the uncertainty-modulated exploration compare to HyperX [1]? This seems like a relevant work either way. \n* Which other evaluation domains were considered, and why was MineDojo selected among them? It is true we are lacking a diverse range of open-world environments in the literature; would simply like to understand other considerations and potential issues with them\n\n[1] https://arxiv.org/abs/2010.01062", "limitations": "Yes; although more details about limitations introduced by the three distinct phases could be discussed more at length (are there real-world tasks that fall outside of this framework? if not, and this is really very general, a stronger, more explicit explanation of this would be useful too)", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n* Paper is easy to follow, and the presentation is polished\n* The architectural decisions make sense, and appear novel\n* The evaluation environment is appropriate given the stated aims\n* The paper contains sufficient details for reproducibility, and links to source code\n* The results indeed appear promising\n\nWeaknesses\n* Performance is not consistently better than baselines (addressed as limitation)\n* Related, the architecture (especially the manually-defined \"phases\") seems a bit brittle, and may not be applicable to all tasks of interest\n* The approach is evaluated only on one domain", "was_revised": true, "final_justification": "I stand by my initial review and therefore maintain my current score.", "rebuttal": "We sincerely thank you for your thorough and positive assessment of our work. We are encouraged by your recognition that our paper is \"easy to follow\" with \"polished presentation,\" that our \"architectural decisions make sense and appear novel,\" and that our evaluation is \"appropriate\" with \"sufficient details for reproducibility.\" We particularly appreciate your acknowledgment that our results \"indeed appear promising\" and your overall favorable rating.\n ___\n-\tQ1: How does the uncertainty-modulated exploration compare to HyperX [1]? This seems like a relevant work either way.\n\n- A1: Thank you for highlighting this relevant work. Regarding the comparison between our uncertainty-modulated exploration and HyperX, we observe both conceptual similarities and important distinctions that highlight the contributions of our approach. HyperX addresses uncertainty-guided exploration by introducing two exploration bonuses during meta-training: (1) a novelty bonus on approximate hyper-states using random network distillation, and (2) a bonus based on the discrepancy between predicted and observed rewards/transitions. Their approach focuses primarily on learning good task-adaptation behavior across different tasks in a meta-learning context, with exploration bonuses that tend toward zero during training.  \n\n- A1: Our uncertainty-modulated prediction regulator differs in several significant ways: First, while HyperX operates in a meta-reinforcement learning context focusing on cross-task exploration, our CPP framework addresses the challenge of continuous adaptation within a complex open-world environment. This fundamental difference leads to distinct uncertainty quantification approaches. Second, CPP implements a more comprehensive uncertainty estimation through multiple integrated prediction error channels (Eq. 10, 23-24): (1) observation prediction errors that capture perceptual uncertainty, (2) reward prediction errors that quantify value uncertainty, and (3) reward trend errors that detect changes in progress patterns. This multi-channel integration enables our system to distinguish between different sources of uncertainty rather than treating them uniformly. Third, unlike HyperX's exploration bonuses that are designed to eventually disappear, our uncertainty estimates continuously modulate exploration parameters throughout task execution. As shown in Eq. 25-28, our approach dynamically adjusts jump thresholds based on current uncertainty levels, making exploration more aggressive when uncertainty is high and more conservative when it is low. This continuous modulation is crucial for open-world tasks where uncertainty varies greatly across different regions and phases.   \n\n- A1: The effectiveness of our approach is particularly evident in tasks requiring adaptive uncertainty-guided exploration, such as the 'obtain wool' task, where CPP achieves a 22.3% improvement over LS-Imagine. This task involves tracking dynamic targets (sheep) where uncertainty changes rapidly as the agent moves, requiring precise modulation of exploration behavior based on prediction confidence. Our ablation studies (Table 3) further validate the contribution of the uncertainty-modulated prediction regulator, showing performance drops of 2.67-6.33 percentage points across tasks when this component is removed, demonstrating its significant impact on system performance.We will incorporate this relevant reference and clearly articulate these distinctions in our revised manuscript.\n___\n-\tQ2 : Which other evaluation domains were considered, and why was MineDojo selected among them? It is true we are lacking a diverse range of open-world environments in the literature; would simply like to understand other considerations and potential issues with them\n\n- A2: Regarding your question about evaluation domains, we considered multiple environments including Atari, ProcGen, DMLab, and Atari100k before selecting MineDojo. Our choice was guided by MineDojo's unique advantages for evaluating adaptive exploration: (1) It offers a true open-world setting with minimal constraints on agent behavior; (2) It provides diverse interaction possibilities requiring both broad exploration and precise manipulation; (3) It features tasks with varying complexity levels, from simple resource collection to multi-step interactions; (4) It has established benchmarks with prior work enabling fair comparison. While additional environments would strengthen evaluation (a limitation we acknowledge in Sec. 5), MineDojo's complexity makes it a representative testbed for open-world exploration strategies. We plan to extend our approach to other environments in future work.\n ___\n- L1: although more details about limitations introduced by the three distinct phases could be discussed more at length (are there real-world tasks that fall outside of this framework? if not, and this is really very general, a stronger, more explicit explanation of this would be useful too)\n- A3: We sincerely appreciate your insightful observation about the need for more detailed discussion of our framework's limitations. You raise an important point about generalizability. Our three-phase approach (exploration, approach, completion) is well-suited for goal-directed tasks with spatial components that involve resource location, navigation, and manipulation, common in robotics, embodied AI, and open-world gaming environments. This architecture naturally maps to how agents typically progress from broad exploration to targeted completion in these domains, as demonstrated by our successful application across diverse MineDojo tasks.\n- A3: However, we acknowledge limitations in certain task categories: (1) highly unstructured environments where phase boundaries blur significantly, such as continuous-control problems without clear goal states; (2) extremely long-horizon tasks requiring numerous intermediate subgoals where a simple three-phase decomposition may be insufficient; and (3) highly adversarial settings where rapid replanning is necessary due to dynamic obstacles or opponents. As our ablation studies show (Table 3), even when the phase-adaptive component is removed, our memory and uncertainty mechanisms still provide performance benefits, suggesting a degree of robustness. Following your valuable feedback, we will include a more comprehensive analysis of our framework's limitations and generality in the revised paper, including a more explicit discussion of task categories where adaptations to the basic three-phase structure might be necessary.\n ___\nWe are grateful for your insightful questions, which have helped us better articulate our contributions in relation to existing work. Your suggestion to compare with HyperX has been particularly valuable in positioning our research within the literature. We will incorporate all your feedback in our revised paper to strengthen the presentation and contextualization of our work."}, {"review_id": "u0RBzJQVOV", "reviewer": "Reviewer_TSWQ", "summary": "This paper introduces CPP, a framework for RL designed to improve how agents explore and make decisions in open-world environments like Minecraft. \n\nInspired by real cognition, the CPP framework integrates three components to create more flexible and adaptive agents: A phase-adaptive cognitive controller that breaks down tasks, a dual-memory integration system that balances short & long-term memory and an uncertainty-modulated prediction regulator that adjusts exploration based on environmental predictability.", "questions": "The current model assumes the learning rule f_θ is static throughout the entire course of an animal's learning. Is it possible that animals \"learn how to learn\"—that is, the learning rule itself is non-stationary and adapts as the animal gains more experience with the task structure?\n\n\nThe quantitative definitions of surprise and uncertainty are based on simple, weighted linear sums of different error signals with fixed, hand-tuned weights (e.g. ω_o, ω_r, ω_trend ). This heuristic approach seems to be less principled than other methods in the RL for quantifying uncertainty.\n\nCould you provide a stronger justification for these specific heuristic formulations?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n\nThe agent succeeded more often and did so faster. CPP reduced the # steps to finish tasks by ~ 7.1%. In the harvest log task, it was > 50% more efficient than DreamerV3. \n\nThe authors’ ablation studies were useful and well done. Each cognitive component was confirmed to contribute meaningfully to the agent performance. \n\nWeakness:\nThe improvement is relatively modest - 4.6 % averaged over five tasks, and CPP actually gets worse on mine iron-ore, where DreamerV3 still leads. So the demonstrated benefits are a starting point rather than an eventual gamechanger (yet).\n\nThe paper's central premise is its \"human-like framework\" built on neurologically-inspired systems. While the concepts are well motivated, the actual implementations often rely on hand tuned heuristics, which is not very adaptive. Many critical parameters that govern the cognitive behavior are hard-coded, e.g. fixed progress thresholds for explore & approach. Similarly, the uncertainty calculation uses a fixed weighting of different error types.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful review. We greatly appreciate your recognition of our work's strengths, particularly your acknowledgment that \"the agent succeeded more often and did so faster,\" reducing task completion steps by ~7.1%, and that our \"ablation studies were useful and well done\" with each component confirmed to \"contribute meaningfully to agent performance.\" Your balanced assessment of both our contributions and limitations has been extremely valuable.\n ___\n-\tQ1: Learning rule adaptation in animals and AI systems \n- A1: Your question about whether animals \"learn how to learn\" raises a fascinating and important direction. You've identified a key limitation in our current implementation, where the learning parameters θ remain static throughout training. Your insight aligns perfectly with recent neuroscience findings [1] demonstrating that biological systems indeed adapt their learning mechanisms based on accumulated task experience. We find this suggestion particularly compelling and are already exploring its implementation through meta-gradient approaches that would allow our framework to adapt learning rules based on task performance and environmental complexity. Rather than simply using fixed learning rates across all phases, this would enable truly adaptive learning that evolves with experience. We will expand our discussion of future work in Section 5 to include this promising direction, with proper attribution to your valuable suggestion. Such an approach could potentially address the performance inconsistencies you noted in tasks like iron ore mining.\n ___\n-\tQ2: Justification for heuristic uncertainty formulations  \n- A2: We appreciate your critical assessment of our uncertainty quantification approach. You're right to question our use of weighted linear combinations with fixed parameters (ω_o, ω_r, ω_trend). While more complex Bayesian approaches might seem more principled, our design choices were motivated by two key considerations: First, computational efficiency in already computationally intensive environments bayesian uncertainty estimation would significantly increase the computational burden during online learning in complex environments. Second, our approach draws inspiration from neuroscience literature [2] suggesting that even in biological systems, uncertainty representations often manifest as relatively simple combinations of prediction errors rather than full Bayesian computations. Our ablation studies (Table 3) demonstrate that even with these simplified heuristics, the uncertainty regulator contributes significantly to performance (2.67-6.33 percentage points across tasks). Nevertheless, we acknowledge this limitation and will strengthen our justification in Section 4.3, including a more detailed analysis of how different uncertainty formulations affect performance across tasks.\n ___\nWe sincerely thank you for your thoughtful questions that have pushed us to better articulate and improve our approach. Your suggestions regarding adaptive learning rules and uncertainty estimation have provided clear directions for enhancing both our current paper and future research. We will incorporate these improvements in our revised manuscript to address the limitations you've identified while maintaining the strengths you recognized in our work.\n ___\n[1] Sousa M, Bujalski P, Cruz B F, et al. A multidimensional distributional map of future reward in dopamine neurons[J]. Nature, 2025: 1-9.  \n [2] Trudel N, Scholl J, Klein-Flügge M C, et al. Polarity of uncertainty representation during exploration and exploitation in ventromedial prefrontal cortex[J]. Nature Human Behaviour, 2021, 5(1): 83-98."}, {"review_id": "fkor8tNYMh", "reviewer": "Reviewer_hvog", "summary": "This paper introduces cognitive predictive processing (CPP), a neurologically inspired architecture for open-world reinforcement learning. Based on insights on human planning and decision-making, CPP integrates three cognitive components: (1) a phase-adaptive cognitive controller that decomposes tasks into exploration, approach, and completion phases, (2) a dual-memory integration system, combining short-term working memory with selective episodic memory, (3) an uncertainty-modulated prediction regulator that adjusts exploration behavior based on prediction error. The method is evaluated on MineDojo, a benchmark built on Minecraft, and shows performance improvements over baselines including DreamerV3, LS-Imagine, STEVE-1, and VPT.", "questions": "Could you clarify the goals of the paper with respect to human-likeliness further?\n\nIn the limitation section, you mention real-world applications, and suggest investigating the applicability to \"medical domains where adaptive exploration strategies could benefit procedures such as minimally invasive blood clot removal in constrained vascular spaces\". This example seems to me surprising and potentially problematic, as such applications would require rigorous safety guarantees before any RL-based exploration could be considered. I would have expected domains with lower risk, such as assistive robotics, as plausible deployment scenarios. Could you please elaborate why you chose this particular example?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\nThis paper addresses an important problem, the inflexibility of fixed exploration strategies in prior open-world RL methods. The authors propose a new adaptive approach and position it well within existing literature. The proposed approach is motivated by human cognitive mechanisms, and its evaluation shows strong performance in exploration efficiency compared to previous methods. The paper includes ablation studies that dissect the individual contributions of each component.\n\n\nWeakness:\nAs a cognitive scientist, I found the framing of the paper in terms of human-likeness somewhat unclear: is the goal to model human behavior or to leverage insights from cognitive science for performance improvements?\nGiven that the evaluation is strictly performance-based, measuring performance and efficiency, I believe the latter appears to be the focus.\nHowever, the paper uses language such as \"human-like\" that may overstate the behavioral similarity without supporting evidence.\nIn particular, no experiments are presented to assess whether the system's behavior aligns with human planning, nor is there any behavioral validation (e.g., human judgments of how natural or human-like the agent appears).\nThe authors also mention, to some extent, this limitation in the conclusion: \"Despite these advances, several limitations remain. Our approach implements simplified cognitive approximations rather than neurologically precise models\" (line 300).\nHowever, mentioning this limitation implies that a cognitive model capturing human exploration would be more desired instead of one with high performance and efficiency.\nI would suggest replacing \"human-like\" with \"human-inspired\" throughout the paper, including the title, to avoid confusion, make the scope clearer, and reformulate the limitations.\n\nThere is a minor typo in line 58 (\"We\" should be lowercase).", "was_revised": true, "final_justification": "The authors clarified that their work does not aim to provide a cognitive model of human exploration, but rather draws inspiration from human behavior. With that clarification, the proposed method appears to be a promising and novel approach to exploration that leverages insights from cognitive science. While I am not in a position to thoroughly compare this approach to the most recent exploration algorithms, the overall idea is interesting and may contribute a useful perspective to the field.", "rebuttal": "We are deeply grateful for your thoughtful assessment of our work. We are particularly honored to receive feedback from a cognitive scientist, as this interdisciplinary perspective is invaluable for our research bridging reinforcement learning and cognitive mechanisms. Your recognition of our paper's contribution to addressing \"an important problem, the inflexibility of fixed exploration strategies\" and our \"new adaptive approach\" is greatly appreciated.\n___\n-\tW1: human-likeness\n- A1: You have identified a crucial clarity issue in our framing that we completely agree with. You are absolutely right that our primary goal is to leverage insights from cognitive science to improve reinforcement learning performance, rather than to precisely model human behavior. We sincerely appreciate your suggestion to replace \"human-like\" with \"human-inspired\" throughout the paper, including the title. We commit to implementing this change throughout the manuscript to better reflect our research goals and avoid overstating behavioral similarity. This change will help readers better understand our contribution without implying claims about precise cognitive modeling that we do not substantiate with behavioral validation.\n___\n-\tW2: \"We\" should be lowercase\n- A2: We are genuinely touched by your attentiveness to such details, it reflects the thoroughness with which you reviewed our work. We will correct this in the revised manuscript.\n___\n-\tQ1: our goals and the medical application example\n- A3: We are impressed by your perceptiveness in questioning our choice of medical example. Our initial motivation for this research was indeed to design intelligent agents capable of goal-directed exploration in dynamic environments. The medical application of nanorobots for tasks like blood clot removal represents our long-term vision for this research direction, where adaptive exploration could have profound human impact in healthcare. However, we fully acknowledge, as you rightly pointed out, that such applications require rigorous safety guarantees beyond our current capabilities. Our plan following this paper is to develop simulated vascular environments with synthetic targets resembling blood clots to test our algorithms in controlled settings. We recognize this is an ambitious direction requiring collaboration with both cognitive science experts and medical professionals. We appreciate your suggestion of safer alternatives like assistive robotics, which represent more appropriate near-term applications, and will revise our examples accordingly.\n___\nWe are truly grateful for your cognitive science insights and for identifying both the strengths and limitations of our approach. Your feedback has substantially improved our paper's positioning and clarity. We will incorporate all your suggestions in our revised manuscript to ensure our research contribution is presented accurately within its proper scope and context."}, {"review_id": "KJjS6LrfzP", "reviewer": "Reviewer_UBqv", "summary": "The authors propose a cognitively inspired method to enhance the learning and usage of a world model in reinforcement learning. Their approach integrates three key components: \n(1) a dynamically updated classifier that adaptively determines the agent's current behavioural phase—exploration, approach, or completion (2) a dual-memory system that balances long-term storage of interesting transitions with short-term recall of recent, task-relevant experiences\n(3) an uncertainty prediction module to guide exploration more effectively. \n\nThe proposed system is evaluated within the MineDojo environment and compare to several classical baselines.", "questions": "The reviewer does not understand why Dreamer-V3 performs better than  CPP in the \"harvest sand\" task but not in \"harvest water\". Based on the explanation provided in lines 268–269, the results seem to suggest Dreamer should have the advantage in both cases. Could the authors clarify what factors may have contributed to this discrepancy?\n\nThe proposed method appears to be sensitive to initial visual conditions. How does the system behave when the target object is not immediately visible—for example, if a sheep is occluded by a forest at the beginning of the episode and the task requires exploratory behavior? Has this been tested or analyzed?\n\nHave the authors experimented with more complex, multi-stage goals that require achieving several sub-goals—such as collecting different materials before crafting a final object? If so, how well does the dual-memory system scale in such scenarios?\n\nHow do the authors envision adapting their approach to goal-conditioned settings, where the agent is required to generalize across different target objectives? Would the current phase classification and memory mechanism still be applicable or require modifications?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper clearly explains a complex, multi-stage architecture, making the proposed method easy to follow despite its intricacy.\n\n- The ablation study is well conducted, effectively isolating the contribution of each component.\n\n- The appendix provides a detailed analysis of the complementary mechanisms, offering valuable insights beyond the main text.\n\n**Weaknesses**:\n\n- The authors report only the mean performance in their experiments; including variance would provide a more complete understanding of the method's stability.\n- The experiments focus primarily on short-horizon tasks, where the agent needs to interact with a nearby object (e.g., a sheep or a tree not far from starting position). The evaluation would be more convincing if it included more complex goals, such as \"Combat Zombie,\" to test the method’s robustness in long-horizon scenarios.", "was_revised": true, "final_justification": "The reviewer is fully satisfied with the answers provided to his numerous questions. The paper is well-reasoned, and experiments on more complicated or lengthy tasks would only strengthen it. The reviewer maintains the total score and increases the clarity rating to 4 in light of the answers provided.", "rebuttal": "Thank you for your thoughtful and detailed review. We greatly appreciate your recognition that our paper \"clearly explains a complex, multi-stage architecture\" despite its intricacy, and that our ablation study is \"well conducted, effectively isolating the contribution of each component.\" Your insightful questions have helped us identify important areas for clarification and improvement in our manuscript.\n___\n- W1: Regarding performance variance\n- A1: You raised an excellent point about reporting only mean performance. We agree this limit understanding of our method's stability. We will incorporate these statistics into Tables 1 and 2 in the revised paper, following your valuable suggestion.\n___\n- W2: On short-horizon tasks\n- A2: We appreciate your concern about evaluating primarily on short-horizon tasks. Our task selection aimed to balance comprehensive assessment with computational resource constraints. While most tasks involve nearby resources, the \"Mine iron ore\" task represents a more complex scenario requiring navigation through underground cave systems with limited visibility, serving as our long-horizon test case. We agree that additional complex tasks would strengthen evaluation, and we had considered \"Combat Zombie\" as you suggested. Due to time constraints, we couldn't complete these experiments for this submission, but we commit to expanding our evaluation to include more complex, long-horizon tasks in future work, as we believe CPP's adaptive exploration would excel in such scenarios.\n___\n- Q1: Regarding the Dreamer-V3 performance discrepancy\n- A3: You identified an insightful question about why Dreamer-V3 outperforms CPP in \"harvest sand\" but not \"harvest water.\" This apparent inconsistency stems from environmental characteristics that interact differently with our memory system. Water bodies have distinctive visual features (blue coloration, reflective properties) and dynamic behavior (flowing patterns), creating strong perceptual signals that our dual-memory system can effectively encode and retrieve. In contrast, sand blocks have more subtle visual characteristics that can occasionally be confused with similar terrain elements. Dreamer-V3's consistent exploration strategy handles visually ambiguous but static resources better in some cases. We will add this clarification to lines 268-275 in the revised paper.\n___\n- Q2: On initial visual conditions\n- A4: Your question about occlusion is highly relevant. We have indeed tested scenarios where targets are initially occluded. As shown in Figure 3 (particularly in the middle row for the exploration phase), our system handles non-visible targets through its phase-adaptive exploration. When targets are occluded, the stagnation detection mechanism (Equations 17-18) identifies lack of progress and maintains the exploration phase until visual detection occurs. In Appendix C.1, we show how affordance maps evolve from diffuse patterns during exploration to concentrated activation when targets become visible. We will enhance this analysis in Appendix C.1.\n___\n- Q3: Regarding multi-stage goals\n- A5: Your question about complex, multi-stage goals touches on an important direction. We conducted preliminary investigations on crafting tasks requiring multiple resources (e.g., crafting an iron sword requiring wood collection → crafting tools → mining ore → smelting → crafting). Our dual-memory system theoretically supports such multi-stage goals by organizing memories according to cognitive phases and resource types, allowing retrieval of relevant past experiences for each subtask. The phase-adaptive controller naturally transitions between exploration, approach, and completion for each subtask sequentially. However, comprehensive evaluation of complete crafting sequences would require substantially more computational resources than our current budget allowed. We believe this represents an exciting direction for future work and will add this discussion to Section 5.\n___\n- Q4: On goal-conditioned settings\n- A6：You raise an excellent question about adapting to goal-conditioned settings. Based on our framework (Section 3), we envision two promising approaches: (1) Encoding goal representations as additional inputs to the phase classifier (Equation 3), allowing phase definitions to dynamically adapt based on goal embeddings, and (2) Structuring the episodic memory retrieval (Equation 13) to prioritize experiences relevant to the current goal. The key challenge would be learning generalizable representations of goals that inform appropriate exploration parameters. Our current phase classification and memory mechanisms would require extension rather than fundamental redesign, specifically by conditioning the functions $f_\\theta$ and similarity metrics on goal embeddings. This represents an exciting direction we're actively exploring, and we will add discussion of this potential extension to Section 5 of the revised paper.\n___\nWe sincerely thank you for your thoughtful questions that have helped us articulate both the strengths and limitations of our approach more clearly. Your feedback on addressing performance variance, occlusion handling, and potential extensions to multi-stage and goal-conditioned settings has been invaluable for improving the paper. We will incorporate all these clarifications and additional analyses in our revised manuscript."}]

Author rebuttal:

Meta-review: This paper proposes a human-inspired architecture for exploration, demonstrating improvements over current methods. Reviewers agreed on the significance of the results and the potential contribution of the work. The main limitations lie in the presentation, and I also found that the choice of parameters in Table 5 requires further justification beyond simply being based on cognitive science principles, especially since no supporting references are provided. Nevertheless, the work is interesting and believe it would make a valuable contribution to the literature.
Decision: Accept (poster)