Goal Reduction with Loop-Removal Accelerates RL and Models Human Brain Activity in Goal-Directed Learning
Authors: Huzi Cheng
Joshua W Brown
Venue: NeurIPS 2024 spotlight (2024)
Keywords: goal-conditioned RL
planning
multi-task RL
vmPFC
goal-directed behavior
cognitive control
spatial navigation
Primary area: neuroscience_and_cognitive_science
TL;DR: We introduced a new goal reduction mechanism that outperforms RL algorithms in multi-goal tasks and models brain activity.

Abstract: Goal-directed planning presents a challenge for classical RL algorithms due to the vastness of the combinatorial state and goal spaces, while humans and animals adapt to complex environments, especially with diverse, non-stationary objectives, often employing intermediate goals for long-horizon tasks.
Here, we propose a goal reduction mechanism for effectively deriving subgoals from arbitrary and distant original goals, using a novel loop-removal technique.
The product of the method, called goal-reducer, distills high-quality subgoals from a replay buffer, all without the need for prior global environmental knowledge.
Simulations show that the goal-reducer can be integrated into RL frameworks like Deep Q-learning and Soft Actor-Critic.
It accelerates performance in both discrete and continuous action space tasks, such as grid world navigation and robotic arm manipulation, relative to the corresponding standard RL models.
Moreover, the goal-reducer, when combined with a local policy, without iterative training, outperforms its integrated deep RL counterparts in solving a navigation task.
This goal reduction mechanism also models human problem-solving.
Comparing the model's performance and activation with human behavior and fMRI data in a treasure hunting task, we found matching representational patterns between an goal-reducer agent's components and corresponding human brain areas, particularly the vmPFC and basal ganglia. The results suggest that humans may use a similar computational framework for goal-directed behaviors.

Goal Reduction with Loop-Removal Accelerates RL and Models Human Brain Activity in Goal-Directed Learning
Huzi Cheng
Department of Psychological and Brain Sciences Indiana University Bloomington
hzcheng150gmail. com
Joshua W. Brown
Department of Psychological and Brain Sciences Indiana University Bloomington jumbrown@iu.c edu
Abstract
Goal-directed planning presents a challenge for classical RL algorithms due to the vastness of the combinatorial state and goal spaces, while humans and animals adapt to complex environments, especially with diverse, non-stationary objectives, often employing intermediate goals for long-horizon tasks. Here, we propose a goal reduction mechanism for effectively deriving subgoals from arbitrary and distant original goals, using novel loop-removal technique The product of the method, called goal-reducer, distills high-quality subgoals from replay buffer, all without the need for prior global environmental knowledge. Simulations show that the goal-reducer can be integrated into RL frameworks like Deep Q-learning and Soft Actor-Critic. It accelerates performance in both discrete and continuous action space tasks, such as grid world navigation and robotic arm manipulation, relative the corresponding standard RL models. Moreover, the goal-reducer, when combined with local policy, without iterative training, outperforms its integrated deep RL counterparts in solving a navigation task. This goal reduction mechanism also models human problem-solving Comparing the model's 's performance and activation with human behavior and IMRI data in a treasure hunting task, we found matching representational patterns between a goal-reducer agent's 'ss components and corresponding human brain areas, particularly the vmPFC and basal ganglia. The results suggest that humans may use similar computational framework for goal-directed behaviors.
1 Introduction
Humans and animals must develop capabilities to pursue time varying goals in continuously changing environments. The pressure for survival prohibits slow, linear adaptation to different goals, ie.. learning value functions from scratch for each new objective. quick and versatile paradigm is necessary for such goal-directed learning scenarios. However, traditional Reinforcement Learning (RL) algorithms are not specifically designed for this, encountering challenges in the goal-directed learning context. They highly optimized for scenarios with relatively fixed goals, e.g., winning in Go, or reducing building energy consumption [5I]. In these situations, iterative methods like the
Our code is available at aigiihubssoncchooghzrggggreeeeee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Bellman equation are effective for approximating value/adwanttggeennaatvee cost functions across various states, maintaining stability over time. Nonetheless, if the goal changes during training, classical RI. becomes highly inefficient due to: 1) The significant increase in state space caused by the introduction of the goal set; 2) The inability to reuse experiences across different goals.
On the other hand, traditional heuristic algorithms like Dijkstra's shortest path algorithm [12] and the A path-finding algorithm (24) excel in these problems. These algorithms leverage goal- independent environment knowledge to construct real-time goal-conditions
maps during navigation. They use selective intermediate goal states to simplify the problem, thus reducing pathfinding time. By breaking down a distant goal into nearer ones, these algorithms utilize local knowledge, independent of goal changes, to solve the problem. More importantly, this principle resembles ways humans use to deal with complex problems. Recently, a number of studies have shown that, besides stimuli and association representation, human and animal brains leverage goal and reduced subgoal representations to solve tasks in various settings 44 11 45 40]. But a common drawback of these algorithms is that they require predefined representations of all states in the task, prohibiting them from scaling to large-scale realistic tasks.
In this paper, we try to bridge such goal reduction mechanisms with neural models. The resulting algorithm is an effective goal-reducer that can accelerate and beat standard RL in different tasks through recursively reducing complex goals into simpler subgoals. The main contributions of this paper are two-fold: 1) Computationally, we propose novel methods to train an effective goal-reducer. After just random walking, it extracts nearly optimal subgoals when the state-goal combination space is large without prior knowledge about the cost/effort between them. This is based on the effective representation structure learned during training. This mechanism, as shown below, can not only be used to accelerate standard RL algorithms in various settings but also to guide navigation in a multi-goal setting without iterative Bellman equation convergence processes and outperforms RL accelerated with it, let alone standard RL itself. 2) Biologically, by comparing the model's s activity with brain IMRI data in the same cognitive task, we show the similarity between the model's component, e.g., the goal- reuuer, and brain regions like vmPFC and putamen, highlighting the potential of using this model to explain how brains solve goal goal-directed problems.
2 Related work
2.1 Goal-directed RL in deep learning
Developing an effective goal-directed algorithm has been a long-lasting open question in the RL. community (2). Various methods have been proposed to mitigate the experience sparsity issue in goal-directed learning. For an extensive review, we refer the reader [35]. Here, we introduce some representative solutions, where some focus on the modification of reward functions, using tricks like
average action cost [25] and sibling trajectory average goals [49] to alleviate the sharpness in reward distribution in the early stage of training. Another line of work uses planning to solve the problem. For example, an explicit graph can be constructed from sensory inputs for traditional path-finding
algorithms [16, 55 34 26]. However, we see these approaches as not completely neural- based
models, and the pathfinding algorithm part may prohibit them from smoothly scaling to tasks with larger state spaces.
Another family of algorithms uses subgoal generation as a core mechanism to resolve the same issue. [8] trained a GAN to predict intermediate goals. There are also attempts to construct subgoals with different heuristics: (56) uses the uncertainty of Q-functions to help train subgoals. [9] treats the midpoint of value functions as optimal subgoals during training. 3 optimizes subgoal generation by minimizing integrated segment costs produced by "local" subgoals. These subgoal- related approaches are intriguing as they match the "divide and conquer" principle in neural way, but a common limitation of these methods is that they work in a bootstrapping manner, i.c., the quality of the Q-function indirectly determines the quality of subgoals generated, it is involved in the subgoal sampling process.
2.2 Goal-directed learning and subgoal generation in neuroscience
There exists a duality of interest in neuroscience on goal-directed learning that can be dated back to the 1940s, when [48) famously showed that rats can take shortcuts that hadn't 't been experienced
2

---

before to reach goals in a maze. Later research showed that there are representations for different goals in the brain that are tailored for planning optimal paths to the ultimate goal [T] Models have been built for these processes to reveal possible mechanisms of how the brain may make use of these goal representations to calculate subgoals (or called "landmarks") for navigation [46 47]. Recently, more empirical and modeling work has come out showing that the subgoal navigation hypothesis, and the underlying cognitive map theory that supports it [3], could be implemented by animals [23 45 and humans (50, 15 54 Altogether, these studies imply the existence of a subgoal generation mechanism in the brain to support effective planning in complex environments.
However, many models for subgoal generation in neuroscience still either rely on manual/one-hot coding of states [15] [54] or focus on revealing the existence rather than the potential development of subgoals during training 45 We therefore think that, in these two fields, the community of deep RL and neuroscience, there is space for neurally plausible algorithm that can provide a biological model of how subgoals are naturally generated from the brain during training and can show its computational efficiency over plain RL. algorithms.
Our resulting work presents a trained goal-reducer neural network that generates nearly optimal subgoals without requiring additional value information from the environment. This approach distinguishes itself from other subgoal generators that depend on Q-functions. It can integrate seamlessly with standard RL frameworks and independently operate with local policy that just learns associations in neighboring states. Using the latter approach, we also demonstrate its capability in solving cognitive tasks in human-like manner and its correspondence with various brain regions. indicating its potential for modeling human problem-solving processes.
3 Methods
3.1
Problem formulation
For a goal-directed Markov decision process, we characterize it with (S. A. 9). where S and AA are the observation and action spaces, respectively, and G is the goal space. One interaction step in this environment can be written as (st, (t, Tt, Gt, 1), where 8t is the current observation, is the assigned goal, dt is the action executed, rt is the reward, 9t is the achieved goal, and 81+1 is the next state. In some cases, G = S occurs when a possible goal is among one of all states, e.g., spatial navigation when the input and goal space is all plausible locations. In other words, the goal and state space are the same. In such settings, we may use G and S interchangeably, and the interaction can be reduced to (St. 9: Ge, T'a, 8t+1). But in some real-world tasks, G may be in a different space from S: G could be a target coordinate in allocentric space when S is the pose of a multi-joint robot arm.
With this formulation, we first consider the simple case when G S as the general case can be easily extended from it. In goal-directed learning, E 9 may change substantially, making conventional RI. algorithms inadequate due to the enormous size of S G. Additionally, in many scenarios, the reward is sparse, and the agent only receives a positive reward or avoids punishment when it reaches the goal state, i.e., when g S and r(st, ag. g) = 1 (s, = g). This, combined with the complexity of the state space, further complicates goal- -iieeddeeeeed learning.
We propose that, given specific g. if an agent can effectively reduce goal y into subgoal 8n E G that is "closer" to its current state st, it may alleviate the task's 's difficulty. Furthermore, this can be applied recursively to find subgoals that are arbitrarily close to the current state 8t.
3.2 Effective goal-reducer through Loop-removal sampling
The most straightforward solution the problem above is train a function & that generates sye which we refer to as a goal-reducee, 4(st, g) S x G + G. The goal-reducer can reduce the computational burden of policy [[ass, g) as can now generate a subgoal sy 2(s, (5,g) for a hard task, under basic assumption that harder problems can be decomposed into simpler problems as long as these problems are in the optimal path towards the final solution (Fig. TA). Training such a goal-reducer, however, can be challenging. There are two intuitive strategies. First, one can sample 8g uniformly from S, as done by 19. This approach referred to as Random sampling (Fig. 1 left). An alternative is to sample 8g from past experiences, a strategy known as Trujectory sumpling. Assuming a sequence of interactions (st. (, Pt, 8841) for L = 1,,,... T stored in memory, Sy can be sampled from the 8t in this sequence Fig. middle). A common technique in goal-directed
3

---

RL, Hindsight Experience Replay (HER), proposed by (2), conceptually resembles this approach, as it encourages the agent to learn associations between current states and some its future states in the same trajectory. Other studies, such as [41], also find trajectory-based sampling effective in goal-directed learning, though expert experience is not required in our case [41]. Furthermore, neuroscience research indicates that episodic memories, i.e., sequences of experienced states, are
vital for learning 200
However, this strategy does not guarantee the sampled 8y to be effective: an agent with limited environmental knowledge may simply engage in random walk within the state transition graph. rendering the states experienced in the episodic memory ineffective for connecting the trajectory's s start and end points. To address this, we introduce third strategy, termed Loop-removal sampling (Fig. BB right), which may mitigate this issue. The underlying rationale is that when the agent has minimal knowledge of the environment, the trajectories creates will likely random and involve numerous "loops". A "loop" occurs when the agent revisits the same state at least twice within trajectory. The Loop-removal sampling posits that by eliminating these "loops" from the episodic memory, the 5g sampled from the remaining trajectories will be more advantageous, ineffective experiences are excluded, potentially resulting trajectory that more closely resembles a linear or shortest path in the best-case scenario.
While Loop-remone sampling effective when S is discrete, i.e., observations are either the same or different, it faces challenges in environments where E S can be infinitely close but not the same to each other, c.g., when the observation an image input for an agent's 's current location. This issue is also encountered in earlier concepts like [28). To overcome this, we adopt an idea from persistent homology [4), defining filtration process determine the existence of "loops."
A
a(als,9 g) sy)
B Randon Sampling
Trajectory Sampling Loop- Leno-ven Sampling
(a.g)
Original Loop renoval
C
Sparse
Proper
Cluttered
D
Filtration radius increases
Fig. I: A: A schematic view of how goal-reducer can be integrated with policy network to generate actions. B: 3 types of subgoal sampling strategies. Red triangles; current states, green squares: goals, light green squares: subgoals sampled. C: An diagram of filtration radius changes trajectory connectivity. D: An example of proper filtered trajectory (black) compared with original random walk (gray) in 3D space.
This process is schematically depicted in Fig. CC: In continuous space, trajectories are sparse, as distances between states are always greater than zero, making exact state overlaps unlikely. However, by assigning filtration radius p to each state and incrementally increasing the connectivity of the episodic memory trajectory changes. At a certain point, the algorithm detects a "loop" (proper case in Fig. IC), where Loop-removal sampling most effective. If the filtration radius continues to increase, the trajectory becomes fully interconnected, leading to Loop-removal sampling failure. We demonstrate that with an appropriate filtration parameter, a random walk path in 3D space can be effectively simplified (Fig. IP) with p set to 0.8. The optimal radius is determined through grid search between and the longest distance between any pair points in the space, with the objective of maximizing the rate at which task performance improves with learning. This search process is also used in later experiments to maximize learning efficiency in terms two efficiency metries, Optimality and Equidex (described in detail below).

---

This approach may seem akin to the Search on the Replay Buffer by [17], but there is a key distinction: Search on the Replay Buffer depends on value function to estimate distances among states, whereas Loop-removal sampling operates independently of any prior environmental knowledge. Instead, it relies on a minimal assumption applicable to all state spaces: loops indicate redundancy. Following this, we developed an online mechanism ( (Allgorihmm in appendix) to train the goal-reducer, @ using loop-removed trajectories stored in D', which is refined from the replay buffer D.
To test if goal-reducer trained with Loop-removal sampling generates better subgoals compared to Trajectory sampling and Random sampling, we developed two metrics to quantify the quality of generated subgoals: Optimality and Equidex. First, we represent the effort required for an agent to reach goal g from state s as ||s, gll. the distance across legal transitions of the state graph. This concept mirrors the idea of shortest distances, though it may not be symmetrical. i.c., ||s * llg, s|l, due to potentially irreversible state transitions. Also, the state space graph of legal transitions may not be fully connected because barriers may exist in the state space so that some transitions are not possible, 8g is generally not simply the linear midpoint between s and g E S. Building on this, Optimality is defined as:
Optimalit((s, g, 8n)) = gll.gl/(( (Ils, 8,ul + 11sg.91)).
(I)
When Optimality + 1, the total effort expended by agent to reach g via sg is near the optimal effort. However, a &y with an Optimality close to may not be informative if 8g is set to either g or 8, where no new information is provided. To address this, Equidex is introduced:
Equidex(s, (s.,assssgg. _g) (1lsg: gll |s, syyl) ([Is, syll + [[sg.91])
When Equidex + O, it indicates that the effort from to subgoal 8,, is similar to the effort from 8g to the final goal g- This suggests that 8g is situated on the hyperplane formed by midpoints in the state space between s and g. Accordingly, as Equidex -> 1, the subgoal becomes closer to the current state, while a value nearing -1 suggests the subgoal is closer to the ultimate goal. In summary, the quality of a single subgoal can be characterized by both Optimality and Equider. The ideal subgoal is one where Optimality equals I and Equidex equals
C
5.0
Optimality
1.0
Equidex
Oplimality
1.0
Equidex
08
0.5
Bandom Tajoctory Looperenooa
0.0
5.0
Optimality
1.0
Equidex
Optimality
1.0
Equidex
2.5
0.5
0,5
Rardon Trjectory
0.0
Loopeemoool
0.0
0.0
2000
2000
2000
1
3
Runs
Runs
Runs
Planning steps
Planning steps
Fig. 2: goal-reducer training results of geometric random graph (top) and the four-room gridworld task (bottom) with different strategies. A: Environment examples. B: Left: training loss, middle: training Optimality, right: training Equidex. C: Left: Optimality change when applying a trained goal-reducer recursively, right: same, but for Equidex.
Using these indices, we assess the performance of the goal-reducer on two datasets. The first dataset derives from a constructed state graph, characterized by random connections among states and allowing self-comnections (Fig. 2A top). We execute undirected random walks on this graph. The second is based on a four-room gridworld task, commonly used in multi-goal RL benchmark tests (Fig. 2A bottom). In this dataset, we simulate agent exploring the environment without any prior knowledge of its structure.
For both datasets, we use the goal-reducer architecture to learn the s, - Sy association, employing an VAE (1.. This network accepts concatenated representations of s and as inputs, initially producing latent probabilistic representation = through the encoder Enc,. A prediction of 8g is then generated by Decs. To circumvent sparse and discrete encoding in both datasets, and also to avoid introducing prior knowledge, random embeddings are applied for each. Consequently,
5

---

and 8g are represented as high- sigh-dimensional random vectors. All random embeddings remain fixed during training. The adopted loss function mirrors that of classical VAE, comprising subgoal reconstruction loss and a weighted (a)) KI. divergence penalty with prior latent distribution p(z) T):
= -a-aDx (Encc(e ())) (sla.g) log Deca(18122)
Training results Fig. 2B left) indicate that in both datasets Loop-removal sampling surpasses and is more stable than Trajectory sampling, which in turn outperforms Random sampling in terms of loss. The Optimality values follow similar trend (Fig. 2B middle):
Loop-removal sampling > Trajectory sampling >> Random sampling.
Given that all subgoals are uniformly sampled from the dataset, the Equidex for all strategies approximates 0 (Fig. 2 right). To further examine the capability and stability of the trained goal- reducer, assess its proficiency in recursive subgoal reduction. In this process, input the predicted subgoal embedding into the goal-reducer, treating it as to generate a subsequent subgoal <g. By iterating this procedure for several steps, postulate that the goal-reducer, particularly when trained with Loop-removal sampling, will produce subgoals that are both effective and sufficiently close to the current states facilitate easier navigation. The experimental results reveal that, after three iterations of recursive goal reduction (t = 3, in FFi.. 2C), the goal-reducer trained with Loop-removal sampling achieves the most favorable optimality distribution Fig.. 2C left).
In terms of Equidex, as illustrated in the right column Fig. 2F. recursive goal reductions enhance the goal-reducer's ability to predict subgoals that nearer to the current states. This improvement evidenced by the shift in the equidex distribution from = 1 to t = 3. Together, the results in Fig demonstrate the superiority of Loop-removal sampling training strategy for goal-reducer to produce proper subgoals.
2
3.3 goal-reducer integrated with RL
The above experiments have shown the superiority of Loop-removal sampling other strategies in training goal-reducer from environments without external knowledge about the "distance" between any states in them. Next we integrate this process with RL algorithms into tasks with discrete and continuous action spaces using Deep Q-learning (DQL) 338 and Soft Actor-Critic (SAC) [22].
Discrete case In the discrete setting, DQL used. The algorithm is trained optimize a value function Q through Bellman equation convergence:
Qulti arg min (s (s 10 QΦ (st. g. a @
(4)
O = (st, 9. uz) + B. Oo (8t+ 9, 0z+1).
(5)
Since we assume goal-reducer, (s,, (ss, g). can learn generate subgoals without a mature Q, we use to accelerate the convergence of Eq. 4 by updating Q's parameter using an extra regularization loss
Σ Wan. DeL TT (aaesse. s) (arls,, ((sss),)sss))))))))))))))))))) ((s. #1.9
Lag
(6)
where the policy т (acest. St: 9) is a categorical distribution softmaxQ(s:, g) and the loss is weighted by the entropy of policy, H J (a st, g)]:
1, if (TT g)] 2 ([TT(a (ss
W Wa.g
0. otherwise.
Continuous case In continuous action space, SAC is used. Since π (ax (8..)) g) cannot be calculated explicitly, we approximate it with an online sampling process of actions executed af:
Tus = Σ Woig T (aglst, g) loog m (a,lss,, ((*,g))
(8)

---

where the ws..g if T (o)
< <a]]ss, D(s, 9)) and otherwise O. In both cases, W 1 means Q more uncertain about the ultimate goal when compared with a trustworthy goal generated by the roal-reducer.
For both cases, for baseline RL methods (plain DQL and SAC) and their goal-reducer augmented version, to accelerate learning, we used Hindsight Experience Replay (HER) during training, a standard technique used improve RL algorithm's performance in goal-directed learning (2].
3.4 Standalone goal-reducer
As shown in previous Optimality and Equidex experiments, goal-reducer can gradually reduce the "distance" between the agent and the goal through recursive goal reduction. We thus test if this mechanism alone can solve some tasks that are usually handled with model-free RL using Bellman equation iteration. To do this, we take an unsupervised approach: When an agent is initialized, explores the environment randomly. We train the goal-reducer using such exploration trajectories. At the same time, we train a local policy Mocal (a: (t, g) that only learns goals that are one step away from and generates a uniform distribution otherwise. When these two components are trained, we exccute the planning process by detecting reachable goals using the entropy Tical (for details, see the appendix).
4 Results
4.1 goal-reducer accelerates standard RI.
Fig. 3/4 left), an Four-room maze navigation task In a modified mini-grid environment 10
agent receives two images inputs and outputs action indicating which direction to go among four possible directions. One image the partial observation a four-room grid world maze, St, while the other image is similar "picture" from the "goal" location, g. No other visual cue beyond the "picture" of the goal location is given, preventing the agent from cheating using easy visual landmarks like the green dots used in the original version of this task. In each episode within this environment, g and Su are uniformly sampled from all possible locations. The agent receives a constant negative reward every step until it reaches g. In this experiment, the DQL algorithm is represented as DRL, while goal-reducer redueer augmented DQL is denoted DRL+GR. The results clearly show that DRL+GR outperforms DRL in terms of convergence time Fig. 3 right).
Robot arm reach task We adopted panda-gyn gym 19 to implement an environment where a robot arm with 7 degrees freedom is trained to reach an arbitrary location sampled uniformly in the space (Fig. 3B left). Like the navigation task, the agent receives a constant negative reward every interaction before reaching within a close region centered on the specified goal location. In this experiment, for plain DQL we used SAC (denoted as DRI)) and goal-reducer augmented SAC (DRL+GR), The results FFg.. 3 right) are consistent with the navigation task.
Correct Rate
Correct fate
Invas
GR-DRL
1.0
GRABRE
1.0
GR-ORL
Correct
(
8.5
Vija ofaaa
6.0
0.8
0K
50K
188K
18K
23K
0K
5BK
168K
Transition
Steps
Transition
Steps
Transition Steps
Fig. 3: goal-reducer accelerates standard RL. A: An example input in the four-room navigation task (left) and performance comparison (right). B: Robot arm reach task (left) and performance comparison (right). C: How goal-reducer agent works with only local policy (left) and performance comparison of 3 algorithms (right).
4.2 Standalone goal-reducer surpasses goul-redduce--accelerrtte RT.
The next question we ask is related to the results in recursive goal reduction (Fig. 2 2C): Can one use just the goal-reducer to perform task as seems to reduce Equidex recursively to neighbor goals? This requires standalone goal- goal-reducer and "local" policy Tlocal that can learn how to associate st
7

---

and 81+1 with a, The Four-room navigation task naturally fits this need, as Tocal in it can be easily defined as a policy that learns to associate two connecting grids. Under this setting, a goal-reducer can recursively generate subgoals using existing goals/subgoals until Tocal finds proper subgoals that are close enough to make a proper decision.
This time a 19x19 maze is used to make the task harder, as goal-reducer's training effect may not be obvious smaller environments. In this environment, we dropped the Bellman equation (for details, see appendix) and compared its performance (denoted as GR) with the previous winner, goal-reducer augmented DQL (GR+DRL) and the baseline plain DQL (DRL). Results in Fig. 3C right column clearly show that GR outperforms both GR+DRL and DRL, while the latter two's s performance relationship consistent with the right column in Fig. 3/A.
4.3 goal-reducer in the Brain
The efficiency the goal-reducer has shown in previous experiments, when compared to plain RLs, naturally leads us to wonder if the brain adopts similar strategy to solve goal-directed behaviors. To address this, we used cognitive task, treasure-hunting (Fig. 4A),, that necessitates flexible goal representation changes. the task, subjects were placed in one four possible starting states on one of two maps (Fig. 4B) and were required reach states designated But having a "key" is necessary when reaching the "chest" to obtain a reward. The locations of the "key" and "chest" are presented to the subject at the start of each episode and change randomly across episodes. The goal-reducer, ((s, g). when paired with a Tlocals forms an agent and is also trained on the same task, using the same strategy adopted in Fig.
"chest" (the ultimate goal).
A
Ave Stump
Farm House
Scarecrow
Cow Field
Scarecrow
0-46
2s
Farm House
Axe Stump
1-6s
C
D
(/s.g)
Inorma pormalized Lized)
6
@
(s, g)
E
Region
2 (normalized)
4
Pu(L)
Pu(R)
Region
Fig. goal-reducer in the treasure hunting task. A: The treasure hunting task description. B: Two configurations of maps used in the task. C: Population z-maps of S, G, and /b in vmPFC. D: The relative representation z- value distribution for the three centers marked as S, G. and & vmPFC. E: Population z-maps of S, Int. S, and & in bilateral putamen. F: Same D, but for bilateral putamen.
After training, we analyzed the goal-reducer agent's 's neural activation and compared it with human subjects' brain activity measured via fMRI. The fMRI data, including human subjects' participation details, were reported previously, and human subjects were compensated $25/hr for fMRI participation [54). particular, subjects were apprised of the mild risks of fMRI including boredom, fatigue, and loss of confidentiality. All research was approved by the institution's IRB, and all subjects provided full informed consent. Our fMRI analysis used representational similarity analysis (RSA) 32). RSA considers all conditions that occurred in the task and compares the activity similarity of the
8

---

model/brain between each pair of conditions via Pearson correlation, thereby forming a symmetric representational dissimilarity matrix (RDM). The entries in the RDM range from 0 to 2 (1 minus the possible correlation ranging from 11 to 1). where a lower value indicates higher similarity. RDMs are calculated for all voxels in human subjects' brains and for different components in the goal- goal-reducer agent, including the input representation S, the goal representation G, and the goal-reducer, Φ(8, g).
The results show that the activity in the ventromedial prefrontal cortex (vmPFC) corresponds to E G in the goal-reducer agent. Next, we evaluated the activity matching the internal neurons of the goal-reducer ((s, g). As depicted in Fig. 4C, there are several regions whose activities correlate with P. including parts of the vmPFC (indicated by the right circle in Fig. 4F) and the left accumbens. This finding is particularly interesting for two reasons: 1) numerous studies have established that the vmPFC is related to value and goal representations (29 43 : 2) the close spatial relationship between regions matching 9 and + (Fig. 4C middle and right columns) reflects the goal- goal-reducer's organization (Fig. 4 right), wherein + and G are interconnected through the generation and input of different goals. This recurrent connectivity suggests that the widespread recurrent connections in the vmPFC (37 may fulfill the role of goal reduction or perform reverse future simulation (6). thereby simplifying the task. The finding is consistent with a posterior-to-anterio hierarchy of goal planning in the prefrontal cortex (7]. and our results may suggest the function of such a hierarchy namely that anterior regions provide goal reduction functionality for more posterior regions.
Following this, we next investigated the Z map of state representations and indeed found region above & and G in the vmPFC that is correlated with S (as denoted by the top circle in Fig. 4C left). The average z-values in these regions show differential representational loading on the goal- agent components (Fig. 4D), although the ROI z-value comparison a descriptive statistic only given that the regions were selected for strong loading on the various model layers 33
Aside from the goal reduction, we also compared S in the goal-reducer agent with brain data. The RSA results indicate significantly elevated representational loading for these two layers in the bilateral putamen (Fig. 4E), while there is lower level of loading for the goal representation layer and an even lesser extent for the goal reduction layer. The putamen, a component of the larger basal ganglia system, is recognized for its involvement in habitual behaviors (53) and goal-directed actions 4 27]. This suggests that such an area should engage in both local policy enactment (the habitual aspect) and goal reduction (goal-directed component). The congruence between the state representation and its intermediate layer with this role may illuminate why activation related to the goal and its reduction is lower in this region (Fig. 4F), since they are not as essential for the habitual component of the local policy.
5 Discussion
Limitations and future work In this work, the trained goal reduction mechanism has shown its capability in terms of its computational advantage and as a biological model of human goal-directed behaviors. This suggests a possible bridge between efficient human problem-solving in multi-goal settings and machine learning. However, key part of the training of the goal-reducer, the loop removal sampling process, does not have clear mechanistic biological process correspondence. Two ways exist to address this issue in the future. Computationally, it may be possible to derive a purely neural model to perform the loop removal process, making the sampling process also biologically plausible. A potential solution will be to leverage a lateral-inhibitiin ieceallinhibiiion--ik [8] mechanism to inhibit all associative synaptic connections between neurons when the same group of neurons are activated more than once in a time range, simulating canceling the "loop" in a memory trajectory. Empirically, it may be possible collect brain activity data during training of the same or similar tasks, or during offline replay, which may incorporate the loop removal as part of memory consolidation.
Conclusion We developed novel general goal reduction mechanism using the loop removal trick and trained a network goal-reducer that can learn to predict nearly optimal subgoals from distant ultimate goals. We show that this approach does not rely on prior knowledge about the global structure/distance of the environment and uses just random explorations. We further demonstrate that it can be integrated into various existing RL frameworks and outperforms them. Besides, after removing the Bellman equation part, when applied recursively, this framework can perform goal- directed learning and even outperform goal-reducer augmented RL methods. This goal-reducer agent was next applied to cognitive task and compared with human fMRI data. Our analyses show

---

that various regions in the brain, including the vmPFC and basal ganglia, can be mapped to the goal reduction representation and state representations in the goal-reducer agent, implying that the brain may instantiate similar computational circuit to perform goal-directed learning.
References
[1] A. Abraham, Pedregosa. M. Eickenberg, Gervais, A. Mueller, J. Kossaifi, A. Gramfort, B. Thirion, and G. Varoquaux. Machine learning for neuroimaging with scikit-learn Frontiers in newroinformatics, 8:14, 2014.
[2] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances neural information processing systems, 30, 2017.
[3] S. Ao, T. Zhou, G. Long, Q. Lu, L. Zhu, and Jiang. Co-pilot: Collaborative planning and reinforcement learning sub-task curriculum. Advances in Neural Information Processing Systems, 34: 1100444445550 2021.
[4] B. W. Balleine and J. P. O'doherty. Human and rodent homologies in action control: corticostri- atal determinants of goal-directed and habitual action. Neunppycchophurmacooggy 35(1): 4586)), 2010.
[5] T. E. Behrens, T. H. Muller, J. C. Whittington, S. Mark, A. B. Baram, K. L. Stachenfeld, and Z.. Kurth-NNlson. What a cognitive map? organizing knowledge for flexible behavior. Neuron, 100(2)4990-50) 2018.
[6] R. G. Benoit, J. Davies, and M. C. Anderson. Reducing future fears by suppressing the brain mechanisms underlying episodic simulation. Proceedings of the National Academy of Sciences, (1362)):8842-EE8500 2016.
[7] I. K. Brunec and I. Momennejad. Predictive representations in hippocampal and prefrontal hierarchies. J. Neurosci 42(2)2299-312, Jan. 2022.
[8] G. Buzsáki. Feed-forward inhibition in the hippocampal formation. Progress in neurobiology. 22(2):131-153, 1984.
[9] E. Chane-Sane, C. Schmid, and Laptev. Goal-conditioned reinforcement learning with imagined subgoals. ArXiv, abs/2107.00541, 2021.
[10] M. Shevalier-Boisverd B. Dai, M. Towers, R. de Lazcano, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306. 0188066.18311 13831, 2023.
[11] Crivelli-Decker, A. Clarke, S. Park, D. Huffman, E. Boorman, and C. Ranganath. Goal-centered representations in the human hippocampus. bioRxiv, pages 2021-08, 2021.
[12] E. W. Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik, 1:269-271, 1959.
[13] D. Dupret, J. neill, B. Reyyell-Bouverii and J. Csicsvari. The reorganization and reactivation of hippocampal maps predict spatial memory performance. Nature neuroscience, 13(8):995- 1002, 2010.
[14] H. Edelsbrunner and J. L. Harer. Computational topology: an introduction. American Mathe- matical Society, 2022.
[15] R. A. Epstein, E. Z. Patai, B. Julian, and H. J. Spiers. The cognitive map in humans: spatial navigation and beyond. Nature neuroscience, 20 20(11):104-1113, 2017.
[16] B. Eysenbach, R. Salakhutdinov, and Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Neural Information Processing Systems, 2019.
10

---

[17] B. Eysenbach, R. R. Salakhutdinov, and S. Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 32. 2019.
[18] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528, PMLR, 2018.
[19] Q. Gallouédec, N. Cazin, E. Dellandréa, and L. Chen. panda-gym: Open-source goal- conditioned environments for robotic learning. arXiv preprint arXiv:2106. /3687, 2021.
[20] J. Gershman and N. D. Daw. Reinforcement learning and episodic memory in humans and animals: an integrative framework. Annual review of psychology, 68: 2017.
[21] D. Ha and Schmidhuber. World models. arXiv preprint arXiv: 10122, 2018.
[22] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[23] D. A. Hamilton, C. S. Rosenfelt, and I. Q Whishaw. Sequential control of navigation by locale and taxon cues in the morris water task. Behavioural brain research, 154(2):385-397, 2004.
[24] P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.
[25] K. Hartikainen, X. Geng, Haarnoja, and S. Levine. Dynamical distance learning for semi- supervised and unsupervised skill discovery. arXiv preprint 1XXi::9977008235 2019.
[26] C. Hoang, S. Sohn, Choi, W. Carvalho, and Lee. Successor feature landmarks for long- horizon conl-conditioned reinforcement learning. In Neural Information Processing Systems, 2021.
[27] M. Jahanshahi, I. Obeso, J. C. Rothwell, and J. A. Obeso. A ffonoo-striico-ubbblllaii----pii network for goal-directed and habitual inhibition. Nature Reviews Neuroscience, 16(12):719- 732, 2015.
[28] Z. Jin, Jin, and W. Liu. Autonomous discovery of subgoals using acyclic state trajectories. In Information Computing and Applications: First International Conference, ICICA 2010, Tangshan, China, October 15-18, 2010. Proceedings 1, pages 49-56. Springer, 2010.
[29] J. W. Kable and W. Glimcher. The neural correlates of subjective value during intertemporal choice. Nature neuroscience, 10(12):1625- 1633, 2007.
[30] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.690 6980, 2014.
[31] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114 2013.
[32] N. Kriegeskorte, M. Mur, and P. Bandettini. Representational similarity anolysis-comnecting the branches of systems neuroscience, Frontiers in systems neuroscience, page 4, 2008.
[33] N. Kriegeskorte, W. K. Simmons, S. F. Bellgowan, and C. I. Baker. Circular analysis in systems neuroscience: the dangers of double dipping. Nat. Neurosci., 12(5):535-540, May 2009.
[34] H. Lai, J. Shen, W. Zhang, and Y. Yu. Bidirectional model-based policy optimization. ArXiv, abs/2007.01999 2020.
[35] M. Liu, M. Zhu, and W. Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arxiv preprint
2022.

---

[36] N. Makris, J. M. Goldstein, D. Kennedy, S. M. Hodge, V. S. Caviness, S. V. Faraone, M. T. Tsuang. and L. J. Seidman. Decreased volume of left and total anterior insular lobule in schizophrenia. Schizophrenia research, 83(2-3):155-171 2006.
[37] E. Miller. The prefontral cortex and cognitive control. Nature reviews neuroscience, 1(1)599- 65, 2000.
[38] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 518(7540):529-533 2015.
[39] H. Nili, C. Wingfield, A. Walther, L. Su, W. Marslen- Wilson, and N. Kriegeskorte. A toolbox for representational similarity analysis. PLoS computational biology, e0(4)=e 2014.
[40] N. Nyberg, É. Duvelle, C. Barry, and H. J. Spiers. Spatial goal coding in the hippocampal formation. Neuron, 1 110(3):394 422, 2022.
[41] S. Paul, J. Vanbaar, and A. Joy-Chowdhury. Learning from trajectories via subgoal discovery. Advances Neural Information Processing Systems, 32, 2019.
[42] W. D. Penny, J. J. Friston, J. T. Ashburner, S. J. Kiebel, and T. E. Nichols. Statistical parametric mapping: the analysis of functional bruin images. Elsevier, 2011.
[43] M. F. Rushworth, M. P. Noonan, E. D. Boorman, M. E. Walton, and T. E. Behrens. Frontal cortex and reward-guided learning and decision-making Neuron, 70(6):1054-1066 2011.
[44] A. B. Satpute, K. N. Ochsner, and D. Badre. The neuroscience of goal-directed behavior. In Goal- behavior, pages 49-84. Psychology Press, 2012.
[45] Shamash, F. Olesen, P. Iordanidou, D. Campagner, N. Banerjee, and T. Branco. Mice learn multi-step routes by memorizing subgoal locations. Nature neuroscience, 24(9):170--1279, 2021.
[46] H. J. Spiers and S. J. Gilbert. Solving the detour problem in navigation: model of prefrontal and hippocampal interactions. Frontiers in human neuroscience, 9:125, 2015.
[47] K. Stachenfeld, M. M. Botvinick, and S. Gershman. The hippocampus as a predictive map. Nature neuroscience, 20(11) 20(11):1643-1633 2017.
[48] E. C. Tolman. Cognitive maps in rats and men. Psychological review, 55(4))189, 1948.
[49] A. Trott, S. Zheng, C. Xiong, and R. Socher. Keeping your distance: Solving sparse reward tasks using self- tif-bbllancing shaped rewards. Advances in Neural Information Processing Systems, 32, 2019.
[50] A. Verma and Mettler. Investigating human learning and eeceisionmmaiing in navigation of unknown environments. IFAC-PupersOnLiin 19332):113-11 2016.
[51] T. Wei, Y. Wang, and Q. 7hu. Deep reinforcement learning for building hvac control. In Proceedings of the 54th annual design automation conference 2017, pages 1-6, 2017.
[52] J. Weng, H. Chen, D. Yan, K. You, A. Dubureq, M. Zhang, Su, H. Su, and J. Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):166, 6, 2022.
[53] H. H. Yin and B. J. Knowlton. The role of the basal ganglia in habit formation. Nature Reviews Neuroscience, 7(6):464 2006.
[54] N. Zarr and J. W. Brown. Foundations of human spatial problem solving. Scientific Reports, 13, 2023.
[55] L. Zhang, G. Yang, and B. C. Stadie. World model as a graph: Learning latent landmarks for planning, ArXiv, abs/201 1. 12491, 2020.
[56] Y. Zhang, P. Abbeel, and L. Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33:7648-7659 2020.
12

---

A Appendix
The appendix is divided into four sections. Section A. introduces the computational resources used for all experiments. Section A.2 describes the training algorithm used for Loop-removal sampling. Section A.3 details the experiment setup and extra results of how goal-reducer is used accelerate RL. Section A.4 is similar to Section [A.3] but focuses on the standalone goal-reducer experiment. The last section, A.5 includes training and analysis details of the goul-reducer-brain fMRI data comparison experiment.
A.1 Computation resources
We evaluated all simulations on server with an 11 GB NVIDIA 2080Ti GPU and computational node with a 32 GB Tesla V100 GPU. The fMRI data processing was executed on server with 128 GB RAM and an Intel(R) Xeon(R) X7560 CPU.
A.2 Loop-removal sampling
In this experiment, three types of sampling strategies used. For Random sampling, uniformly sample subgoals from the set of all possible state representations. For Trajectory sampling, a single trajectory, subgoals are uniformly sampled from states that are between the first (start state) and the last state (goal). For Loop-removal sampling, we implemented a filtering algorithm to remove loops from trajectories. Below (Algorithm 1) is the pseudocode used implement the Loop-removal sampling in all experiments:
Algorithm 1 Train goal-reducer with Loop-removal sampling
Input: Replay buffer D, filtration threshold p, and loop-removed set D',
for each iteration
Sample trajectory d D. Initialize an empty d'.
for each (st, 9, ae, Tr: S1+1) in d do for each (s), g. a4, r/. 81441) in do
if ||s x111 < p then
Remove all transitions from t' to the tail d'.
break end if end for Append (st, ag, St+1) to d'.
end for
Sample to, t from d' where to S t1.
Append (sta 9t,19) D'. end for
Training For the two environments in Fig. A, we used the same goal-reducer architecture all sampling: a VAE with two architecturally identical 3-layer MLPs for encoding and decoding. The Adam optimizer 30 is used for training. For some important hyperparameters, see Table
Table I: Hyperparameters used in A.2
Name
Description
Value
lr bsz
Learning rate for goal-reducer optimizer
x 10-3
Batch size
256
epochs
Max number of epochs
4096
A.3 goal-reducer accelerates DRL
Task the four-room navigation task, we created new environment based on Minigrid [0] Specifically, we:
13

---

uniformly sampled initial locations and goal locations among all plausible locations.
changed the action space from composition of turning angle and moving forward to plain four directions (up. down, left, right).
made the environment partially observable to the agent by allowing squared image of its surroundings (a 13x13 image with the agent in the center).
see only limited
made the goal (the green square) invisible to the agent, i.e., the agent will only receive a picture taken if it in the goal location its goal input.
In the robot arm reach task, we adopted the panda-gy gym library [19] Initial joint angles and goal coordinates are also uniformly sampled from plausible value ranges.
Scalability One reason we used the four-room navigation task test the capability of goal-reducer that we can easily adjust the size of the maze. This gives us sense about how goal-reducer with Loop-removal sampling scales. As shown in Fig. 3A, four-room environment is composed of two rooms and three walls (left border, middle, right border) if one looks from a single side, so the legal border size takes the form of + 2k, where k is the size of the room. We initially tested sizes 15 and 19. as shown in the main text. Next, we examined other sizes and formed results for 13, 15, 17, 19, trend that DRL+GR outperforms DRL in all cases. Though the gap in a finite step becomes smaller as the sizes increase, given the increased task difficulty, the performance difference is positive and consistent across different conditions.
and 21. From Fig.S. T
A
13x13
15x15
17x17
19x19
Correct Rate
1.0
GReBRL
GR+DRL
GR-DRL
GR+DRL
DRL
DRL
DRL
0.5
0.0
0K
50K
100K 0K
50K
100K 0K
50K
100K 0K
50K
100K
Transition Steps
Transition Steps
Transition Steps
Transition Steps
B
21x21
Correct Rate
1.0
GR+DRL
DRL
0.5
0.0
0K
50K
100K
150K
Transition Steps
S. I: Performance comparison between DRL+GR and DRL in the four-room navigation task with different sizes.
Training For the DRL (DQL) and DRL+GR (DQL+goal-reducer) agent implementation, we used and modified Tianshou (52 to implement the baseline algorithm and the trainer. To handle pixel-like inputs the four-room navigation task, an extra single CNN (ObsEnc) used to preprocess the 8 and g (this is not used in the robot arm reach task). Following that, an MLP is used as the Q-net in DQL. Two Adam optimizers are used during training. One is used to train the DQL part, while another is turned on only when the goal-reducer is ON. In both DRL and DRL+GR, we applied the same Hindsight Experience Replay [2] buffer to accelerate training.
For some important hyperparameters used, see Table 2.
A.4 goal-reducer surpasses DRL+GR training details
Task In this experiment, we compared standalone local policy, GR, with DRL+GR and DRL in a four-room gridworld navigation task.
Planning process In the GR case, standalone goal-reducer and local policy are used to generate actions for every possible s, g combination. The pseudocode for this process is listed in Algorithm
14

---

Table 2: Hyperparameters used in A.3
Name
Description
Value
Ir lrgR bsz
learning rate for DQL optimizer
10
learning rate for goal-reducer optimizer
5 10-1
batch size
256
epochs
max number of epochs
2
Using this algorithm, the goal-reducer agent can navigate in the environment by recursively producing batch of subgoals over time, until least one of them close enough to the current location. The local policy's entropy plays the role of epistemic uncertainty [56] to help goal-reduce
decide whether goal/subgoal is close enough. An example shown in Fig.S. 2
Algorithm 2 Action generation of goal-reducer with a local policy
Input: Goal reduction step & = 0, max allowed goal reduction steps K, state representation 8t. goal representation 9. parallel goal numbers M, entropy threshold r)
04 Tlual (St, g)
if Entropy(a:) < 1) then
return
end
Initialize subgoal list G.M [g] x M
while < K do
for i = 1 to M do
GM = (ss. GM(l))
end for if min(Entroyy < 1) then
g" arg min (Entropy(G.M)
ag = (St,
return G₁ end if
=
end while return at
Fig.S. 2: An example of the goal-reducer planning process. Red dots show the agent's location, dark green dots (upper left) show the goal, and shadowed green circles show subgoals generated by goal-reducer over time. Darker green indicates more subgoals the same location.
The training and agent settings the same above for DRL+GR and DRL. For the GR case, besides the optimizers for the DQL part and the goal-reducer part, we introduced an extra Adam optimizer for the CNN preprocessing part in the pipeline. Without the constraint from Q function to learn effective state representations, an extra self-supervised learning approach must be used. Similar 211, we introduced "world model" (WorldModel) and decoder (ObsDec) for the agent. Specifically, decoder and an abstract next state representation are introduced:
Training
he+ MorldMoodel(bbsEnn(s ue)
(1)
6 = ObsDec(h,+1).
(2)
15

---

where $141 is the prediction of 81+1- To train this model, BCE loss between $1+1 and 8t+1 is used, paired with a weighted (0.001) n2-morm penalty of h441 avoid exploding gradients. For some important hyperparameters used, see Table 3.
Table 3: Hyperparameters used in A.4
Name
Description
Value
1r
learning rate for DQL optimizer
X 10
lrGR
learning rate for goal-reducer optimizer
10
Worrldaade
learning rate for "world model"
5 X 10
bsz
batch
256
epochs
max number of epochs
40
max goal generation number
entropy threshold max goal reduction steps
0.8
A.5 goal-reducer fMRI data analysis
Task For human subjects, the treasure hunting task is presented with a series of images and verbal instructions on the screen (see Fig. 4A). To train the GR agent to perform this task, we turned all presentations into verbal instructions (see Fig.S. 3) and then encoded them with OpenAl's ada-002 text embedding model to transform them into vectors.
Goal: Key: Scarecrow, Chest: Axe Stump
Obs:
Field
Fig.S. 3: An example text input in the treasure hunting task for the goal-reducer agent
This unifies the representation of this task with all previous tasks and allows future generalization to novel tasks without modifying the agent itself.
Training The network architecture is the same as the GR agent used in previous experiments. For some important hyperparameters used, see Table 4.
Table 4: Hyperparameters used in A.5
Name
Description
Value
1r
Learning rate for DQL optimizer
1 X 10-3
IrGR
Learning rate for goal-reducer optimizer
1 x 10-3
11WerldNodel
Learning rate for WorldModel Batch size
1 10-3
bsz
256
steps
Max number of transition steps
15000
M
Max goal generation number
12
n
Entropy threshold
0.8
K
Max goal reduction steps
3
EMRI data analysis We included 24 human subjects' fMRI data for analysis All research involving human subjects was approved by the institution's IRB. Subjects provided informed consent, and the procedures adhered to applicable guidelines and the Declaration of Helsinki. The data were re-used from [54). which also provides the full methods.
Data is available osf.io: part part 2 and part
16

---

In the RDM analysis, each RDM for a brain voxel or model component is 64 x 64 matrix, where = 8 x 2. Here, 4 represents the four possible current subject/model locations on the map. represents all possible episode configurations (comprising four two-step key-chest configurations when the subject/model is at the starting location, two one-step "key-chest" configurations when the subject/model is at the starting location, and two two-step key-chest configurations when the subject/model is at the middle location). The factor accounts for the two phases of each movement (planning and action phase).
For the human fMRI data, such RDMs are calculated for each voxel in the brain for each subject. with neighborhood radius set to 10 mm. For the model, RDMs are first calculated for learned S and G representations and the intermediate representations in the corresponding neural networks. Moreover, we calculated RDMs for the goal-reducer's hidden representations as well when the goal reduction performed. The action layer, i.e., the output distribution at each step is also calculated. Together, these form the following model RDMs: S RDM for state representations. Int.d S RDM for state representations in hidden layers, 9 RDM for goal representations, Int. 9 RDM for goal representations in hidden layers, RDM for action representation, and & RDM for goal-reducer's hidden layers.
Once RDMs are calculated, Pearson correlation between each voxel and a model RDM's upper triangular part, given their symmetrical nature, is calculated. This forms series of fMRI RSA maps for each subject, showing whether certain brain regions are positively correlated with the aforementioned model components. Second-level analysis is then performed to extract the population effect. To do so, we first transform the individual correlations into 7-scores using the Fisher transform. Then, all 7-score maps are projected into a shared space, and a 1-sample T-test is performed for each voxel in the shared brain space to examine whether the corresponding correlation is significant across subjects. After that, cluster-based correction performed to filter out statistically significant regions. We used nilearn [1 rsatoolbox 39 Python version, and SPM5 42 perform this process.
With the above analysis pipeline, found that different brain regions have high match with distinct components in the goal-reducer agent. For a complete list of all clusters, see Table
17

---

Table Significant similarity clusters derived from SPM5 42 for RSA analysis. Anatomical region labels are derived from [36]. The Poorreted and cluster size (clusterg,) values are both corrected using SPM5. Note that some layers may involve more than one region due the probabilistic nature of atlases in [36].
Component
X
Z
Peorrected
clusterk
Region(s)
Int.S
32
-9
21
0
15,323
Right Cerebral White Matter
Int.5
29
11
15,323
Right Putamen
Int.S
15
-12
-2
0
15,323
Right Cerebral White Matter
Int.S
43
38
0.02
82
Frontal Pole, Right Cerebral Cor-
tex
Int.S
-50
-64
28
0.01
96
Lateral Occipital Cortex, supe- rior division, Left Cerebral Cor-
tex
Int.S
-44
-67
21
0.01
96
Lateral Occipital Cortex, supe- rior division, Left Cerebral Cor- tex
Int.S
-50
-47
35
0.01
96
Supramarginal Gyrus, posterior division, Left Cerebral White Matter
Int.g
-37
-12
-26
18,578
Parahippocampal Gyrus, ante- rior division, Left Cerebral Cor-
Int.
-26
-23
1
0
18,578
Left Cerebral White Matter
Int.g
-26
-16
32
18,578
Left Cerebral White Matter
-30
-9
15
0
10,362
Left Cerebral White Matter
-23
-26
32
10,362
Left Cerebral White Matter
+
25
-40
21
0
10,362
Right Cerebral White Matter
-13
32
52
183
Superior Frontal Gyrus, Left
Cerebral Cortex
-16
43
45
183
Frontal Pole, Left Cerebral Cor-
tex
-13
50
35
183
Frontal Pole, Left Cerebral Cor- tex
29
18
14,266
Right Cerebral White Matter
S
25
-23
35
0
14,266
Right Cerebral White Matter Right Cerebral White Matter Supramarginal Gyrus, posterior division, Left Cerebral Cortex
18
-12
14,266
S
-50
-50
38
0.05
70
S
-47
-54
28
0.05
70
Angular Gyrus, Left Cerebral White Matter
a
29
-13
1,148
Right Cerebral White Matter
a
-2
-26
-40
1,148
Brain-Stem
a
22
-19
32
1,148
Right Cerebral White Matter
a
-16
36
-2
0.01
104
Left Cerebral White Matter
a
-13
32
-9
0.01
104
Frontal Medial Cortex, Left Cerebral White Matter Subcallosal Cortex, Left Cere-
a
22
-13
0.01
104
bral White Matter
g
-16
8
.9
16,195
Left Putamen
G
29
-12
28
0
16,195
Right Cerebral White Matter
G
16,195
Right Putamen
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
18

---

Answer: [Yes]
Justification: We developed a goal reduction model and test it in both simulations and cognitive tasks and compared with brain fMRI data. All experiments results match the clasim we made in the abstract and intro.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the work are discussed in the last part, the discussion section, of the main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper is mostly about empirical observations and has theorems.
Guidelines:
19

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Code is released along the manuscript.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
20

---

For the simulation part, people can reproduce the results with provided code and without any extra data. For the brain imaging analysis part, data will be available upon reasonable request.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Sublic/guides httpe/uuuiddee/eeeeuabiieeeeee
CCodeSSbmissioonooll
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurlPS code and data submission guidelines (https Sotpe///oooeeeeesiiiiieiii
//nips.c/ppil htte//ubliiggguuuue hwwbbli///giddde/dddSduuiiissis
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiment settings are revealed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For experiments we' 've included errorbar (STE). But notice that Fig. there seem to be error bars, because the STE is too small.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We've disclosed resources in A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://arrrp..ccppoaaaacccttcaaa
Answer: [Yes]
Justification: Research in this paper conform the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This paper's central results are not about social impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We've credited 3rd party code bases used in the research in the appendix.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
The human subjects data were reused from previous paper, which is cited in the text and reports all the details of the human subjects participation.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
The human subjects data were reused from previous paper, which is cited in the text and reports all the details of the human subjects participation. In particular, also enumerate in the main text the risks, IRB approval, and full informed consent obtained from all subjects.
Justification:
24


Reviews:
[{"review_id": "vRAdVDDOqj", "reviewer": "Reviewer_ww9y", "summary": "This paper introduces a new automated method to identify subgoals in reinforcement learning. The method is based on a replay buffer of agent trajectories and proceeds by identifying and eliminating “loops” (points where the agents revisit almost the same state), in effect attempting to identify efficient shortcuts in the replays and set those shortcuts as subgoals. This method for subgoal extraction can be applied to a variety of reinforcement learning algorithm, and is successfully demonstrated with two popular approaches (deep Q learning and soft actor critic), leading to performance gains in two environments (maze exploration and robotic arm control). In addition, a human neuroimaging experiment shows that the latent space of the “goal-reducer” components of an agent shares similarity with brain activity in key regions, suggesting that the proposed goal reduction mechanism may be at play in biological networks. In summary, the paper proposed a new broadly applicable approach to identify subgoals for reinforcement learning, demonstrates its effectiveness on established RL benchmarks and also offers evidence in support of a biological implementation of this process in human brains The paper thus presents a very strong and multidisciplinary set of evidence.", "questions": "The authors extend the grid exploration task to 19 x 19 cells in order to test the scalability of the method. Why 19? And not 20? Have the authors found a “breaking point” in the scalability of the approach? Also, insights on the computational scalability of the approach would be useful. \n\nThe authors identified similarity of representation with the goal reducer in the white matter (which appears significant in the provided putamen slices). How do the authors explain that? Could it suggest some motion or physiological artifacts correlated with the artificial representations? \n\nAlthough fMRI results are presented, this neuroimaging modality has very limited temporal resolution. Could this affect the representational similarity analysis? It looks like the steps of the games are slow by design, but this could be discussed briefly.", "limitations": "There is no clear neuronal mechanism proposed in support of the goal reduction process, beyond association with an indirect brain measure (blood oxygenation). But bringing any kind of multidisciplinary evidence is already very commendable.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is well written. I am not an expert in RL and found it hard to follow at times, with heavy use of mathematical notations without qualitative explanations (the blocks of equations 4 to 7 for DQN for example). Still, I found the algorithm well motivated and the analogy with Dikjstra’s shortest path algorithm helped me understand the motivation of local optimization well I think. \n\nThe fact that the method can be applied across a range of RL approaches, or stand-alone, is appealing. \n\nCombining a novel method, a rigorous RL benchmark with a complex set of human neuroimaging experiments in one paper is impressive.\n\nThe methods, algorithms and experiments are very carefully described, including the version of software used and details on the hardware used to run the experiments, aiding reproducibility.", "weaknesses": "Although the authors claim that the code was shared with the manuscript, I was not able to find it. I either did not look in the right place (I was expecting a supplementary material link on the submission page, or a link in the manuscript), or the authors forgot to upload it. In any case, access to the code for review would address this weakness, but in the current state I am not able to assess the quality of the code and whether the code seems to cover all the parts in the paper. Also, it is stated that the fMRI data is available “upon reasonable request”, it is not possible for me to assess to what extent the authors will actually be responsive, and whether the data is reusable. I strongly encourage the authors to use an established open data sharing platform such as openneuro and standard data structure such as BIDS to increase reproducibility. Statistical maps could be shared on neurovault for increased reproducibility. \n\nIt would have been beneficial to contrast different models in the neuroimaging experiment, and incorporate some baseline models and an ablation study. The origins of the similarity in representations may be trivial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive feedback. Regarding equation 4-5, they represent the Bellman part in RL, equation 6 and 7, together, shows how a goal reducer can accelerate RL by using the generated subgoal to regularize (KL divergence is used) the learning of value of the original goal. Specifically, when the entropy of a Q net towards an ultimate goal state is higher than that of a subgoal generated by the goal-reducer, we use the Q value of the subgoal to teach the ultimate goal’s Q value distribution. This relies on two assumptions: 1) “closer” goals are easier to learn than distant goals in RL and 2) the goal-reducer can generate fine subgoals.\n\nRegarding the code, previously we put the code in a github repo, but due to the NeurIPS requirement of anonymity, we removed the corresponding URL in the final submission as it would have identified us. The same applied to the fMRI data, which we published on osf.io. These changes will be reflected in the final version.\n\nRegarding your questions, \n\n1. Size issue: 19 is used because we adopted a 4 room setting in the navigation task, where in the center we need a “wall”, so to make all rooms have the same size, i.e, 1 (left wall)+ 8 (left room) + 1 (center wall) + 8 (right room) + 1 (right wall). Due to limited computation resources, we didn’t try environments with larger sizes. However, we observed consistent trends for sizes like 11, and 13, 15, 17, and 19, though as the sizes increase, all models converge more slowly. After we received your feedback, we’ve run experiments with size 21 and the same trend persists, namely that performance with the goal reducer is superior. In the future, one direction to go will be to scale the model on larger environments with more powerful computational resources. The latest size 21 result will also be updated in the supplementary material.\n1. White matter issue: In our fMRI analysis, we found very widespread state representation across the brain. We see white matter due to the putamen's close proximity to white matter tracts. The 3D smoothing process may contribute to the spillover effects in the white matter.\n1. Temporal resolution issue: Yes, in order to capture the existence of the potential subgoal reduction process, the steps of the games are in fact slow by design. Moreover, there are several events within each trial, and we included jitter in the timing between events in order to estimate the BOLD response to each event within a trial separately, as is common practice. These individual event beta weights are in turn individually present in the RSA matrices, so that the activity of each event within a trial is accounted for. These events are included for each of the trial conditions which include various combinations of state and goal pairs. Thus the RSA results account for events within trials as well as the various task conditions tested.\n\nLastly, we thank the reviewer for inquiring about potential neuronal mechanisms of goal reduction in the brain. The goal reduction process involves two key components: subgoal generation and loop removal. The subgoal generation is already \"neural\" as we used a network to generate subgoals rather than other heuristics. Regarding loop removal, our hypothesis is that feedforward inhibition in the hippocampus may implement it. Consider a replay event in which the animal has already traversed a point twice (a loop); the average activation of that place cell (PC) pattern will be stronger than other PCs. If a threshold is set between them, an inhibition mechanism may inhibit all PC patterns between the two activations, thus strengthening non-overlapping PC associations while suppressing \"loops\". Of course, more refined simulations and experiments are necessary to validate this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mrqz3jqDRH", "reviewer": "Reviewer_SziM", "summary": "The paper proposes a goal reduction mechanism utilizing a novel loop-removal technique to derive subgoals from arbitrary and distant original goals. This method, called goal-reducer, distills high-quality subgoals from a replay buffer without prior global environmental knowledge. It accelerates performance in various RL frameworks and matches human brain activity in goal-directed tasks.", "questions": "- In algorithm 1, how to calculate $||s_t - s_t'||$? Does this distance depend on Euclidean distance? If yes, the Euclidean distance is a strong prior knowledge of the environment to the agent.\n- I agree that SoRB relies on strong prior knowledge. However, there are some related work that trains a sub-goal generator along with the RL learning process, e.g., [1]. \n- Can this method be extended to any goal state? If there are multiple goals or no specific goal state is specified, then removing the historical state traversed by the agent will weaken the agent's ability to reach certain goals.\n- Why didn't the authors compare the performance of GR+DRL  with SoRB? \n\n[1] Shuang Ao, Tianyi Zhou, Guodong Long, Qinghua Lu, Liming Zhu, Jing Jiang, CO-PILOT: COllaborative Planning and reInforcement Learning On sub-Task curriculum", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This study presents an innovative method for eliminating loops in order to reduce goals, hence condensing intricate goals into more manageable subgoals. This breakthrough tackles the issue of extensive state and goal spaces in conventional RL algorithms, enabling faster and more effective learning.\n- The goal-reducer can be easily integrated with existing RL frameworks like DQN/SAC, enhancing their performance without requiring major modifications to the underlying algorithms", "weaknesses": "Refer to questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback as well as pointing out pieces we missed in the original writing. Regarding your questions,\n\n1. State distance estimate issue: Yes we used Euclidean distance to measure the similarity between different states representations, however, this does not necessarily impose prior knowledge during training. As we’ve shown in line 201, the state representations are random vectors without any knowledge about the actual cost between different states. But definitely, if we use a trained observation encoder, the performance will be better.\n1. Missing references issue: We apologize for the unintentional omission of Shuang et al., which we now cite and discuss as prior work in the 2nd paragraph of section 2.1 and this will be reflected in the final version.\n1. Arbitrary goal issue: We apologize that the model’s strengths were not described more clearly. The model method can be extended to any goal state as well as underconstrained goal sets. It must be noted that the goal space is not necessarily the state space. For example, if a task requires an agent to collect $K$ gemstones to get a reward, the goal state is defined as having obtained all $K$ gemstones and correspondingly a subgoal might be having obtained  $K-1$ gemstones. With this formulation, the loop removal trick still works.\n1. SoRB issue: We agree comparing SoRB with goal reducer will be an extremely important and intriguing  question to explore in our future work. Because SoRB has one huge advantage and also a potential drawback: it explicitly builds a graph and thus the search will be mathematically optimal. However, the quality of its state graph relies on the quality of the value function, this may make the process ineffective in the early stage of training, when the Q net is still not mature enough. In contrast, our model can work with less prior knowledge in estimating the subgoal (or waypoint in SoRB) while we suffer from a trained goal reducer, who may not always generate the optimal subgoal. So this could result in a crossing effect: Our model may converge faster initially and have an overall a bit lower performance when compared with SoRB after long enough training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zwdy0QYcfU", "reviewer": "Reviewer_i4LG", "summary": "The paper provides a novel way to find subgoals just from past experiences (without any prior knowledge), named goal-reducer. Goal-reducer works by finding “loops” in trajectories and remove them. The authors extend the concept of loops in the continuous cases by defining a filtration radius (distances of two states lower than the radius means they are the same state). Through experiments, the authors show goal-reducer can 1) help RL by providing subgoals; 2) can work on its own to find immediate next subgoals and outperform RL; 3) has a corresponding human brain activity in tasks involving subgoals. In general, the paper is sound, makes meaningful contributions, and is written well.", "questions": "See weakness", "limitations": "While the paper discusses their approach's limitations, they do not discuss the potential negative societal impact of their work. I hope the authors consider including them in the rebuttal.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The writing is very easy to follow and describes contributions very clearly w.r.t. previous methods. \n2. The paper looks into an important problem in both computer science and neuroscience, finding connections between the two mechanisms. The problem is motivated well in introduction. \n3. The proposed method is intuitive yet very effective. It makes sense both theoretically and empirically via experiments. The proposed method also removes the need for prior knowledge about the environment and the task, which are required by most previous methods. \n4. The experiments support the paper’s claims well. I appreciate the authors’ effort in removing domain knowledge by random embedding for raw states on Line 200.", "weaknesses": "1. I am not sure about the applicability of goal-reducer beyond just goal-reaching behavior. Take the fMRI study task as an example, how do we make sure the goal-reducer does not condense these two states together: both states are in the same place, but one with the “key” and one without the “key”. If the mapping to embedding space is random, we cannot guarantee the distance of having/not-having the key is larger than the distance of being in two different places. Essentially, I am saying that what if the loop detour is necessary to get some treasure. I wonder about the authors’ thoughts of the applicability of goal-reducer in harder tasks than just navigation. \n2. I am not sure about the motivation/justification for Equation 6-7: why should the loss be present only when the entropy decreases when the subgoal is introduced? Wouldn’t that encourage the policy to be less certain about what to do when given the subgoal? \n3. Line 297 mentioned Indiana University IRB – while it is great to show the study was IRB-approved, I wonder if this is in violation to anonymity requirements in reviewing stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the feedback and valuable insights provided by you. For your questions, here are some of our thoughts:\n\n1. Applicability issue: We agree that when a detour is necessary, things could be confusing. However, to our knowledge this usually depends on the problem formulation. For example, when a certain detour, e.g., capturing a distant key is a prerequisite to unlock the reward in the box, the goal space will no longer be the location space, and it will be occupancy of both key and box. With such transformations, all tasks can be formulated in the form that is solvable by this approach.\n2. Eq 6 and 7 issue: Eq. 6 and 7 help to train the Q-net. The process relies on two crucial assumptions: First, a Q net will have larger entropy when it’s not sure about its own prediction. Second, “closer” goals are easier to learn (and thus have lower entropy) for a Q-net. Third, a goal reducer may work independent of the Bellman equation. With these assumptions, Eq. 7 says that when a Q-net is not very sure about its prediction (larger entropy) of the ultimate goal, it should trust a closer subgoal more (lower entropy), as the subgoal is generated by an independent goal reducer (assumption 1 and 3). And since a closer subgoal’s Q-value is easier to learn and it’s informative about the ultimate goal due to the recursive nature, we can use the Q-value distribution to regularize the ultimate goal’s Q-value distribution.\n1. IRB issue: We apologize for identifying the  IRB  of record. This will be changed to “institution’s IRB” in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' thoughtful feedback and the opportunity to clarify several key aspects of our work. Individual rebuttals are made below to avoid confusion. In this global rebuttal, we aim to address the concerns raised by reviewer [ww9y](https://openreview.net/forum?id=Y0EfJJeb4V&noteId=vRAdVDDOqj), which pertain to the scalability of the goal reduction mechanism. In short, we show that the trend of goal reduction outperforming its deep RL counterparts is consistent across different environment sizes and persists even beyond the range we tested in the original submission. However, the computation resources have limited us from running even larger tests. This will be a direction to explore in our future work. For details, see the uploaded PDF.

Meta-review: This paper proposes a reinforcement learning method (snipping out loops in collected data), presents results showing that this method improves performance, and shows evidence that this method may be similar to biological networks. Reviewers found the paper well written, commending the paper for including both RL experiments and human neuroimaging experiments. Reviewers also appreciated the high degree of reproducibility, experiments demonstrating that the proposed method works for multiple RL backbones, and the discussion about removing the need for domain knowledge. The reviewers asked some conceptual questions that the authors clarified during the review process. The authors shared code with the AC, per NeurIPS guidelines. During the reviewer discussion, Reviewer i4LG (score=9) reviewed the concerns of Reviewer Szim (score=4), and agrees with the authors that those concerns may stem from a misunderstanding. Taking this all into account, I recommend that the paper be accepted.
Decision: Accept (spotlight)