Goal Reduction with Loop-Removal Accelerates RL and Models Human Brain Activity in Goal-Directed Learning

Authors: huzi  cheng
joshua w brown
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:48:02

Abstract: Goal-directed planning presents a challenge for classical RL algorithms due to the vastness of the combinatorial state and goal spaces, while humans and animals adapt to complex environments, especially with diverse, non-stationary objectives, often employing intermediate goals for long-horizon tasks.
Here, we propose a goal reduction mechanism for effectively deriving subgoals from arbitrary and distant original goals, using a novel loop-removal technique.
The product of the method, called goal-reducer, distills high-quality subgoals from a replay buffer, all without the need for prior global environmental knowledge.
Simulations show that the goal-reducer can be integrated into RL frameworks like Deep Q-learning and Soft Actor-Critic.
It accelerates performance in both discrete and continuous action space tasks, such as grid world navigation and robotic arm manipulation, relative to the corresponding standard RL models.
Moreover, the goal-reducer, when combined with a local policy, without iterative training, outperforms its integrated deep RL counterparts in solving a navigation task.
This goal reduction mechanism also models human problem-solving.
Comparing the model's performance and activation with human behavior and fMRI data in a treasure hunting task, we found matching representational patterns between an goal-reducer agent's components and corresponding human brain areas, particularly the vmPFC and basal ganglia. The results suggest that humans may use a similar computational framework for goal-directed behaviors.

Main paper: 
 introduction: 
 Humans and animals must develop capabilities to pursue time-varying goals in continuously changing environments. The pressure for survival prohibits slow, linear adaptation to different goals, i.e., learning value functions from scratch for each new objective. A quick and versatile paradigm is necessary for such goal-directed learning scenarios. However, traditional Reinforcement Learning (RL) algorithms are not specifically designed for this, encountering challenges in the goal-directed learning context. They are highly optimized for scenarios with relatively fixed goals, e.g., winning in Go, or reducing building energy consumption [51] . In these situations, iterative methods like the Bellman equation are effective for approximating value/advantage/negative cost functions across various states, maintaining stability over time. Nonetheless, if the goal changes during training, classical RL becomes highly inefficient due to: 1) The significant increase in state space caused by the introduction of the goal set; 2) The inability to reuse experiences across different goals.On the other hand, traditional heuristic algorithms like Dijkstra's shortest path algorithm [12] and the A* path-finding algorithm [24] excel in these problems. These algorithms leverage goalindependent environment knowledge to construct real-time goal-conditioned maps during navigation. They use selective intermediate goal states to simplify the problem, thus reducing pathfinding time. By breaking down a distant goal into nearer ones, these algorithms utilize local knowledge, independent of goal changes, to solve the problem. More importantly, this principle resembles ways humans use to deal with complex problems. Recently, a number of studies have shown that, besides stimuli and association representation, human and animal brains leverage goal and reduced subgoal representations to solve tasks in various settings [44, 11, 45, 40] . But a common drawback of these algorithms is that they require predefined representations of all states in the task, prohibiting them from scaling to large-scale realistic tasks.In this paper, we try to bridge such goal reduction mechanisms with neural models. The resulting algorithm is an effective goal-reducer that can accelerate and beat standard RL in different tasks through recursively reducing complex goals into simpler subgoals. The main contributions of this paper are two-fold: 1) Computationally, we propose novel methods to train an effective goal-reducer. After just random walking, it extracts nearly optimal subgoals when the state-goal combination space is large without prior knowledge about the cost/effort between them. This is based on the effective representation structure learned during training. This mechanism, as shown below, can not only be used to accelerate standard RL algorithms in various settings but also to guide navigation in a multi-goal setting without iterative Bellman equation convergence processes and outperforms RL accelerated with it, let alone standard RL itself. 2) Biologically, by comparing the model's activity with brain fMRI data in the same cognitive task, we show the similarity between the model's component, e.g., the goal-reducer, and brain regions like vmPFC and putamen, highlighting the potential of using this model to explain how brains solve goal-directed problems. 
related work 2.1 goal-directed rl in deep learning: 
 Developing an effective goal-directed algorithm has been a long-lasting open question in the RL community [2] . Various methods have been proposed to mitigate the experience sparsity issue in goal-directed learning. For an extensive review, we refer the reader to [35] . Here, we introduce some representative solutions, where some focus on the modification of reward functions, using tricks like average action cost [25] and sibling trajectory average goals [49] to alleviate the sharpness in reward distribution in the early stage of training. Another line of work uses planning to solve the problem. For example, an explicit graph can be constructed from sensory inputs for traditional path-finding algorithms [16, 55, 34, 26] . However, we see these approaches as not completely neural-based models, and the pathfinding algorithm part may prohibit them from smoothly scaling to tasks with larger state spaces.Another family of algorithms uses subgoal generation as a core mechanism to resolve the same issue. [18] trained a GAN to predict intermediate goals. There are also attempts to construct subgoals with different heuristics: [56] uses the uncertainty of Q-functions to help train subgoals. [9] treats the midpoint of value functions as optimal subgoals during training. [3] optimizes subgoal generation by minimizing integrated segment costs produced by "local" subgoals. These subgoalrelated approaches are intriguing as they match the "divide and conquer" principle in a neural way, but a common limitation of these methods is that they work in a bootstrapping manner, i.e., the quality of the Q-function indirectly determines the quality of subgoals generated, as it is involved in the subgoal sampling process.There exists a duality of interest in neuroscience on goal-directed learning that can be dated back to the 1940s, when [48] famously showed that rats can take shortcuts that hadn't been experienced before to reach goals in a maze. Later research showed that there are representations for different goals in the brain that are tailored for planning optimal paths to the ultimate goal [13] . Models have been built for these processes to reveal possible mechanisms of how the brain may make use of these goal representations to calculate subgoals (or called "landmarks") for navigation [46, 47] . Recently, more empirical and modeling work has come out showing that the subgoal navigation hypothesis, and the underlying cognitive map theory that supports it [5] , could be implemented by animals [23, 45] and humans [50, 15, 54] . Altogether, these studies imply the existence of a subgoal generation mechanism in the brain to support effective planning in complex environments. However, many models for subgoal generation in neuroscience still either rely on manual/one-hot coding of states [15, 54] or focus on revealing the existence rather than the potential development of subgoals during training [45] . We therefore think that, in these two fields, the community of deep RL and neuroscience, there is space for a neurally plausible algorithm that can provide a biological model of how subgoals are naturally generated from the brain during training and can show its computational efficiency over plain RL algorithms.Our resulting work presents a trained goal-reducer neural network that generates nearly optimal subgoals without requiring additional value information from the environment. This approach distinguishes itself from other subgoal generators that depend on Q-functions. It can integrate seamlessly with standard RL frameworks and independently operate with a local policy that just learns associations in neighboring states. Using the latter approach, we also demonstrate its capability in solving cognitive tasks in a human-like manner and its correspondence with various brain regions, indicating its potential for modeling human problem-solving processes.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations of the work are discussed in the last part, the discussion section, of the main text.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 For a goal-directed Markov decision process, we characterize it with (S, A, G), where S and A are the observation and action spaces, respectively, and G is the goal space. One interaction step in this environment can be written as (s t , g, a t , r t , ĝt , s t+1 ), where s t is the current observation, g is the assigned goal, a t is the action executed, r t is the reward, ĝt is the achieved goal, and s t+1 is the next state. In some cases, G = S occurs when a possible goal is among one of all states, e.g., spatial navigation when the input and goal space is all plausible locations. In other words, the goal and state space are the same. In such settings, we may use G and S interchangeably, and the interaction can be reduced to (s t , g, a t , r t , s t+1 ). But in some real-world tasks, G may be in a different space from S: G could be a target coordinate in allocentric space when S is the pose of a multi-joint robot arm.With this formulation, we first consider the simple case when G = S as the general case can be easily extended from it. In goal-directed learning, g ∈ G may change substantially, making conventional RL algorithms inadequate due to the enormous size of S × G. Additionally, in many scenarios, the reward is sparse, and the agent only receives a positive reward or avoids punishment when it reaches the goal state, i.e., when G = S and r(s t , a t , g) = 1(s t = g). This, combined with the complexity of the state space, further complicates goal-directed learning.We propose that, given a specific g, if an agent can effectively reduce a goal g into a subgoal s g ∈ G that is "closer" to its current state s t , it may alleviate the task's difficulty. Furthermore, this can be applied recursively to find subgoals that are arbitrarily close to the current state s t .The most straightforward solution to the problem above is to train a function Φ that generates s g , which we refer to as a goal-reducer: Φ(s t , g) : S × G → G. The goal-reducer can reduce the computational burden of a policy π(a|s, g) as it can now generate a subgoal s g = Φ(s, g) for a hard task, under a basic assumption that harder problems can be decomposed into simpler problems as long as these problems are in the optimal path towards the final solution (Fig. 1A ). Training such a goal-reducer, however, can be challenging. There are two intuitive strategies. First, one can sample s g uniformly from S, as done by [9] . This approach is referred to as Random sampling (Fig. 1B  left ). An alternative is to sample s g from past experiences, a strategy known as Trajectory sampling. Assuming a sequence of interactions (s t , g, a t , r t , s t+1 ) for t = 1, 2, • • • , T is stored in memory, s g can be sampled from the s t in this sequence (Fig. 1B middle ). A common technique in goal-directed RL, Hindsight Experience Replay (HER), proposed by [2] , conceptually resembles this approach, as it encourages the agent to learn associations between current states and some of its future states in the same trajectory. Other studies, such as [41] , also find trajectory-based sampling effective in goal-directed learning, though expert experience is not required in our case [41] . Furthermore, neuroscience research indicates that episodic memories, i.e., sequences of experienced states, are vital for learning [20] .However, this strategy does not guarantee the sampled s g to be effective: an agent with limited environmental knowledge may simply engage in a random walk within the state transition graph, rendering the states experienced in the episodic memory ineffective for connecting the trajectory's start and end points. To address this, we introduce a third strategy, termed Loop-removal sampling (Fig. 1B right), which may mitigate this issue. The underlying rationale is that when the agent has minimal knowledge of the environment, the trajectories it creates will likely be random and involve numerous "loops". A "loop" occurs when the agent revisits the same state at least twice within a trajectory. The Loop-removal sampling posits that by eliminating these "loops" from the episodic memory, the s g sampled from the remaining trajectories will be more advantageous, as ineffective experiences are excluded, potentially resulting in a trajectory that more closely resembles a linear or shortest path in the best-case scenario.While Loop-removal sampling is effective when S is discrete, i.e., observations are either the same or different, it faces challenges in environments where s i ∈ S can be infinitely close but not the same to each other, e.g., when the observation is an image input for an agent's current location. This issue is also encountered in earlier concepts like [28] . To overcome this, we adopt an idea from persistent homology [14] , defining a filtration process to determine the existence of "loops." This process is schematically depicted in Fig. 1C : In a continuous space, trajectories are sparse, as distances between states are always greater than zero, making exact state overlaps unlikely. However, by assigning a filtration radius ρ to each state and incrementally increasing it, the connectivity of the episodic memory trajectory changes. At a certain point, the algorithm detects a "loop" (proper case in Fig. 1C ), where Loop-removal sampling is most effective. If the filtration radius continues to increase, the trajectory becomes fully interconnected, leading to Loop-removal sampling failure. We demonstrate that with an appropriate filtration parameter, a random walk path in 3D space can be effectively simplified (Fig. 1D ) with ρ set to 0.8. The optimal radius is determined through a grid search between 0 and the longest distance between any pair of points in the space, with the objective of maximizing the rate at which task performance improves with learning. This search process is also used in later experiments to maximize learning efficiency in terms of two efficiency metrics, Optimality and Equidex (described in detail below). This approach may seem akin to the Search on the Replay Buffer by [17] , but there is a key distinction: Search on the Replay Buffer depends on a value function to estimate distances among states, whereas Loop-removal sampling operates independently of any prior environmental knowledge. Instead, it relies on a minimal assumption applicable to all state spaces: loops indicate redundancy. Following this, we developed an online mechanism ( Algorithm 1 in appendix) to train the goal-reducer, Φ, using loop-removed trajectories stored in D ′ , which is refined from the replay buffer D.To test if a goal-reducer trained with Loop-removal sampling generates better subgoals compared to Trajectory sampling and Random sampling, we developed two metrics to quantify the quality of generated subgoals: Optimality and Equidex. First, we represent the effort required for an agent to reach goal g from state s as ||s, g||, the distance across legal transitions of the state graph. This concept mirrors the idea of shortest distances, though it may not be symmetrical, i.e., ||s, g|| ̸ = ||g, s||, due to potentially irreversible state transitions. Also, the state space graph of legal transitions may not be fully connected -because barriers may exist in the state space so that some transitions are not possible, s g is generally not simply the linear midpoint between s and g ∈ S. Building on this, Optimality is defined as:Optimality(s, g, s g ) = ||s, g||/ ||s, s g || + ||s g , g|| .(When Optimality → 1, the total effort expended by an agent to reach g via s g is near the optimal effort. However, a s g with an Optimality close to 1 may not be informative if s g is set to either g or s, where no new information is provided. To address this, Equidex is introduced:EQUATIONWhen Equidex → 0, it indicates that the effort from s to subgoal s g is similar to the effort from s g to the final goal g. This suggests that s g is situated on the hyperplane formed by midpoints in the state space between s and g. Accordingly, as Equidex → 1, the subgoal becomes closer to the current state, while a value nearing -1 suggests the subgoal is closer to the ultimate goal. In summary, the quality of a single subgoal can be characterized by both Optimality and Equidex. The ideal subgoal is one where Optimality equals 1 and Equidex equals 0. Using these indices, we assess the performance of the goal-reducer on two datasets. The first dataset derives from a constructed state graph, characterized by random connections among states and allowing self-connections (Fig. 2A top). We execute undirected random walks on this graph. The second is based on a four-room gridworld task, commonly used in multi-goal RL benchmark tests (Fig. 2A bottom). In this dataset, we simulate an agent exploring the environment without any prior knowledge of its structure.For both datasets, we use the goal-reducer architecture to learn the s, g → s g association, employing an VAE [31] . This network accepts concatenated representations of s and g as inputs, initially producing a latent probabilistic representation z through the encoder Enc Φ . A prediction of s g is then generated by Dec Φ . To circumvent sparse and discrete encoding in both datasets, and also to avoid introducing prior knowledge, random embeddings are applied for each. Consequently, s, g, and s g are represented as high-dimensional random vectors. All random embeddings remain fixed during training. The adopted loss function mirrors that of a classical VAE, comprising a subgoal reconstruction loss and a weighted (α) KL divergence penalty with a prior latent distribution p(z) ∼ N (0, I):EQUATIONTraining results (Fig. 2B left) indicate that in both datasets Loop-removal sampling surpasses and is more stable than Trajectory sampling, which in turn outperforms Random sampling in terms of loss.The Optimality values follow a similar trend (Fig. 2B middle):Loop-removal sampling > Trajectory sampling ≫ Random sampling.Given that all subgoals are uniformly sampled from the dataset, the Equidex for all strategies approximates 0 (Fig. 2B right). To further examine the capability and stability of the trained goalreducer, we assess its proficiency in recursive subgoal reduction. In this process, we input the predicted subgoal embedding into the goal-reducer, treating it as g to generate a subsequent subgoal s ′ g . By iterating this procedure for several steps, we postulate that the goal-reducer, particularly when trained with Loop-removal sampling, will produce subgoals that are both effective and sufficiently close to the current states to facilitate easier navigation. The experimental results reveal that, after three iterations of recursive goal reduction (t = 3, in Fig. 2C ), the goal-reducer trained with Loop-removal sampling achieves the most favorable optimality distribution (Fig. 2C left ).In terms of Equidex, as illustrated in the right column of Fig. 2C , recursive goal reductions enhance the goal-reducer's ability to predict subgoals that are nearer to the current states. This improvement is evidenced by the shift in the equidex distribution from t = 1 to t = 3. Together, the results in Fig. 2 demonstrate the superiority of Loop-removal sampling as a training strategy for goal-reducer to produce proper subgoals.The above experiments have shown the superiority of Loop-removal sampling over other strategies in training goal-reducer from environments without external knowledge about the "distance" between any two states in them. Next we integrate this process with RL algorithms into tasks with discrete and continuous action spaces using Deep Q-learning (DQL) [38] and Soft Actor-Critic (SAC) [22] .Discrete case In the discrete setting, DQL is used. The algorithm is trained to optimize a value function Q through Bellman equation convergence:EQUATIONEQUATIONSince we assume goal-reducer, Φ(s, g), can learn to generate subgoals without a mature Q, we use it to accelerate the convergence of Eq. 4 by updating Q's parameter using an extra regularization lossEQUATIONwhere the policy π a t |s t , g is a categorical distribution softmaxQ(s t , g) and the loss is weighted by the entropy of policy, H[π(a t |s t , g)]:EQUATIONContinuous case In continuous action space, SAC is used. Since π a t |s t , g cannot be calculated explicitly, we approximate it with an online sampling process of actions executed a ′ t :EQUATIONwhere the w st,g = 1 if π a ′ t |s t , g < π a ′ t |s t , Φ(s, g) and otherwise 0. In both cases, w st,g = 1 means Q is more uncertain about the ultimate goal when compared with a trustworthy goal generated by the goal-reducer.For both cases, for baseline RL methods (plain DQL and SAC) and their goal-reducer augmented version, to accelerate learning, we used Hindsight Experience Replay (HER) during training, a standard technique used to improve RL algorithm's performance in goal-directed learning [2] .As shown in previous Optimality and Equidex experiments, goal-reducer can gradually reduce the "distance" between the agent and the goal through recursive goal reduction. We thus test if this mechanism alone can solve some tasks that are usually handled with model-free RL using Bellman equation iteration. To do this, we take an unsupervised approach: When an agent is initialized, it explores the environment randomly. We train the goal-reducer using such exploration trajectories. At the same time, we train a local policy π local a t |s t , g that only learns goals that are one step away from s t and generates a uniform distribution otherwise. When these two components are trained, we execute the planning process by detecting reachable goals using the entropy of π local (for details, see the appendix).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This paper is mostly about empirical observations and has no theorems.Justification: For the simulation part, people can reproduce the results with provided code and without any extra data. For the brain imaging analysis part, data will be available upon reasonable request. Guidelines: Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
results: 
 4.1 goal-reducer accelerates standard RL Four-room maze navigation task In a modified mini-grid environment [10] (Fig. 3A left), an agent receives two images as inputs and outputs an action indicating which direction to go among four possible directions. One image is the partial observation in a four-room grid world maze, s t , while the other image is a similar "picture" from the "goal" location, g. No other visual cue beyond the "picture" of the goal location is given, preventing the agent from cheating using easy visual landmarks like the green dots used in the original version of this task. In each episode within this environment, g and s 0 are uniformly sampled from all possible locations. The agent receives a constant negative reward every step until it reaches g. In this experiment, the DQL algorithm is represented as DRL, while goal-reducer augmented DQL is denoted DRL+GR. The results clearly show that DRL+GR outperforms DRL in terms of convergence time (Fig. 3A right ).Robot arm reach task We adopted panda-gym [19] to implement an environment where a robot arm with 7 degrees of freedom is trained to reach an arbitrary location sampled uniformly in the space (Fig. 3B left ). Like the navigation task, the agent receives a constant negative reward every interaction before reaching within a close region centered on the specified goal location. In this experiment, for plain DQL we used SAC (denoted as DRL) and goal-reducer augmented SAC (DRL+GR). The results (Fig. 3B right) are consistent with the navigation task. The next question we ask is related to the results in recursive goal reduction (Fig. 2C ): Can one use just the goal-reducer to perform a task as it seems to reduce Equidex recursively to neighbor goals? This requires a standalone goal-reducer and a "local" policy π local that can learn how to associate s t and s t+1 with a t . The Four-room navigation task naturally fits this need, as a π local in it can be easily defined as a policy that learns to associate two connecting grids. Under this setting, a goal-reducer can recursively generate subgoals using existing goals/subgoals until π local finds proper subgoals that are close enough to make a proper decision.This time a 19x19 maze is used to make the task harder, as goal-reducer's training effect may not be obvious in smaller environments. In this environment, we dropped the Bellman equation (for details, see appendix) and compared its performance (denoted as GR) with the previous winner, goal-reducer augmented DQL (GR+DRL), and the baseline plain DQL (DRL). Results in Fig. 3C right column clearly show that GR outperforms both GR+DRL and DRL, while the latter two's performance relationship is consistent with the right column in Fig. 3A .The efficiency the goal-reducer has shown in previous experiments, when compared to plain RLs, naturally leads us to wonder if the brain adopts a similar strategy to solve goal-directed behaviors.To address this, we used a cognitive task, treasure-hunting (Fig. 4A ), that necessitates flexible goal representation changes. In the task, subjects were placed in one of four possible starting states on one of two maps (Fig. 4B ) and were required to reach states designated as a "chest" (the ultimate goal). But having a "key" is necessary when reaching the "chest" to obtain a reward. The locations of the "key" and "chest" are presented to the subject at the start of each episode and change randomly across episodes. The goal-reducer, Φ(s, g), when paired with a π local , forms an agent and is also trained on the same task, using the same strategy adopted in Fig. 3C . After training, we analyzed the goal-reducer agent's neural activation and compared it with human subjects' brain activity measured via fMRI. The fMRI data, including human subjects' participation details, were reported previously, and human subjects were compensated $25/hr for fMRI participation [54] . In particular, subjects were apprised of the mild risks of fMRI including boredom, fatigue, and loss of confidentiality. All research was approved by the institution's IRB, and all subjects provided full informed consent. Our fMRI analysis used representational similarity analysis (RSA) [32] . RSA considers all conditions that occurred in the task and compares the activity similarity of the model/brain between each pair of conditions via Pearson correlation, thereby forming a symmetric representational dissimilarity matrix (RDM). The entries in the RDM range from 0 to 2 (1 minus the possible correlation ranging from -1 to 1), where a lower value indicates higher similarity. RDMs are calculated for all voxels in human subjects' brains and for different components in the goal-reducer agent, including the input representation S, the goal representation G, and the goal-reducer, Φ(s, g).The results show that the activity in the ventromedial prefrontal cortex (vmPFC) corresponds to g ∈ G in the goal-reducer agent. Next, we evaluated the activity matching the internal neurons of the goal-reducer Φ(s, g). As depicted in Fig. 4C , there are several regions whose activities correlate with Φ, including parts of the vmPFC (indicated by the right circle in Fig. 4C ) and the left accumbens. This finding is particularly interesting for two reasons: 1) numerous studies have established that the vmPFC is related to value and goal representations [29, 43] ; 2) the close spatial relationship between regions matching G and Φ (Fig. 4C middle and right columns) reflects the goal-reducer's organization (Fig. 4C right), wherein Φ and G are interconnected through the generation and input of different goals. This recurrent connectivity suggests that the widespread recurrent connections in the vmPFC [37] may fulfill the role of goal reduction or perform reverse future simulation [6] , thereby simplifying the task. The finding is consistent with a posterior-to-anterior hierarchy of goal planning in the prefrontal cortex [7] , and our results may suggest the function of such a hierarchy -namely that anterior regions provide goal reduction functionality for more posterior regions.Following this, we next investigated the Z map of state representations and indeed found a region above Φ and G in the vmPFC that is correlated with S (as denoted by the top circle in Fig. 4C left ).The average z-values in these regions show differential representational loading on the goal-reducer agent components (Fig. 4D ), although the ROI z-value comparison is a descriptive statistic only given that the regions were selected for strong loading on the various model layers [33] .Aside from the goal reduction, we also compared S in the goal-reducer agent with brain data. The RSA results indicate significantly elevated representational loading for these two layers in the bilateral putamen (Fig. 4E ), while there is a lower level of loading for the goal representation layer and an even lesser extent for the goal reduction layer. The putamen, a component of the larger basal ganglia system, is recognized for its involvement in habitual behaviors [53] and goal-directed actions [4, 27] . This suggests that such an area should engage in both local policy enactment (the habitual aspect) and goal reduction (goal-directed component). The congruence between the state representation and its intermediate layer with this role may illuminate why activation related to the goal and its reduction is lower in this region (Fig. 4F ), since they are not as essential for the habitual component of the local policy. 
discussion: 
 Limitations and future work In this work, the trained goal reduction mechanism has shown its capability in terms of its computational advantage and as a biological model of human goal-directed behaviors. This suggests a possible bridge between efficient human problem-solving in multi-goal settings and machine learning. However, a key part of the training of the goal-reducer, the loop removal sampling process, does not have a clear mechanistic biological process correspondence. Two ways exist to address this issue in the future. Computationally, it may be possible to derive a purely neural model to perform the loop removal process, making the sampling process also biologically plausible. A potential solution will be to leverage a lateral-inhibition-like [8] mechanism to inhibit all associative synaptic connections between neurons when the same group of neurons are activated more than once in a time range, simulating canceling the "loop" in a memory trajectory. Empirically, it may be possible to collect brain activity data during training of the same or similar tasks, or during offline replay, which may incorporate the loop removal as part of memory consolidation.We developed a novel general goal reduction mechanism using the loop removal trick and trained a network goal-reducer that can learn to predict nearly optimal subgoals from distant ultimate goals. We show that this approach does not rely on prior knowledge about the global structure/distance of the environment and uses just random explorations. We further demonstrate that it can be integrated into various existing RL frameworks and outperforms them. Besides, after removing the Bellman equation part, when applied recursively, this framework can perform goaldirected learning and even outperform goal-reducer augmented RL methods. This goal-reducer agent was next applied to a cognitive task and compared with human fMRI data. Our analyses show• uniformly sampled initial locations and goal locations among all plausible locations.• changed the action space from a composition of turning angle and moving forward to a plain four directions (up, down, left, right).• made the environment partially observable to the agent by allowing it to see only a limited squared image of its surroundings (a 13x13 image with the agent in the center).• made the goal (the green square) invisible to the agent, i.e., the agent will only receive a picture taken as if it is in the goal location as its goal input.In the robot arm reach task, we adopted the panda-gym library [19] . Initial joint angles and goal coordinates are also uniformly sampled from plausible value ranges.Scalability One reason we used the four-room navigation task to test the capability of goal-reducer is that we can easily adjust the size of the maze. This gives us a sense about how goal-reducer with Loop-removal sampling scales. As shown in Fig. 3A Training For the DRL (DQL) and DRL+GR (DQL+goal-reducer) agent implementation, we used and modified Tianshou [52] to implement the baseline algorithm and the trainer. To handle pixel-like inputs in the four-room navigation task, an extra single CNN (ObsEnc) is used to preprocess the s t and g (this is not used in the robot arm reach task). Following that, an MLP is used as the Q-net in DQL. Two Adam optimizers are used during training. One is used to train the DQL part, while another is turned on only when the goal-reducer is ON. In both DRL and DRL+GR, we applied the same Hindsight Experience Replay [2] buffer to accelerate training.For some important hyperparameters used, see Table 2 .Task In this experiment, we compared a standalone local policy, GR, with DRL+GR and DRL in a four-room gridworld navigation task.Planning process In the GR case, a standalone goal-reducer and a local policy are used to generate actions for every possible s t , g combination. The pseudocode for this process is listed in Algorithm Training The training and agent settings are the same as above for DRL+GR and DRL. For the GR case, besides the optimizers for the DQL part and the goal-reducer part, we introduced an extra Adam optimizer for the CNN preprocessing part in the pipeline. Without the constraint from a Q function to learn effective state representations, an extra self-supervised learning approach must be used. Similar to [21] , we introduced a "world model" (WorldModel) and a decoder (ObsDec) for the agent. Specifically, a decoder and an abstract next state representation are introduced:EQUATION)where ŝt+1 is the prediction of s t+1 . To train this model, BCE loss between ŝt+1 and s t+1 is used, paired with a weighted (0.001) L2-norm penalty of h t+1 to avoid exploding gradients. For some important hyperparameters used, see Table 3 . Task For human subjects, the treasure hunting task is presented with a series of images and verbal instructions on the screen (see Fig. 4A ). To train the GR agent to perform this task, we turned all presentations into verbal instructions (see Fig. Training The network architecture is the same as the GR agent used in previous experiments. For some important hyperparameters used, see Table 4 . fMRI data analysis We included 24 human subjects' fMRI data for analysis. 2 All research involving human subjects was approved by the institution's IRB. Subjects provided informed consent, and the procedures adhered to applicable guidelines and the Declaration of Helsinki. The data were re-used from [54] , which also provides the full methods.In the RDM analysis, each RDM for a brain voxel or model component is a 64 × 64 matrix, where 64 = 4 × 8 × 2. Here, 4 represents the four possible current subject/model locations on the map. 8 represents all possible episode configurations (comprising four two-step key-chest configurations when the subject/model is at the starting location, two one-step "key-chest" configurations when the subject/model is at the starting location, and two two-step key-chest configurations when the subject/model is at the middle location). The factor 2 accounts for the two phases of each movement (planning and action phase).For the human fMRI data, such RDMs are calculated for each voxel in the brain for each subject, with a neighborhood radius set to 10 mm. For the model, RDMs are first calculated for learned S and G representations and the intermediate representations in the corresponding neural networks. Moreover, we calculated RDMs for the goal-reducer's hidden representations as well when the goal reduction is performed. The action layer, i.e., the output distribution at each step is also calculated. Together, these form the following model RDMs: S RDM for state representations, Int.S RDM for state representations in hidden layers, G RDM for goal representations, Int.G RDM for goal representations in hidden layers, π RDM for action representation, and Φ RDM for goal-reducer's hidden layers.Once RDMs are calculated, Pearson correlation between each voxel and a model RDM's upper triangular part, given their symmetrical nature, is calculated. This forms a series of fMRI RSA maps for each subject, showing whether certain brain regions are positively correlated with the aforementioned model components. Second-level analysis is then performed to extract the population effect. To do so, we first transform the individual correlations into z-scores using the Fisher transform. Then, all z-score maps are projected into a shared space, and a 1-sample T-test is performed for each voxel in the shared brain space to examine whether the corresponding correlation is significant across subjects. After that, cluster-based correction is performed to filter out statistically significant regions. We used nilearn [1] , rsatoolbox [39] 's Python version, and SPM5 [42] to perform this process.With the above analysis pipeline, we found that different brain regions have a high match with distinct components in the goal-reducer agent. For a complete list of all clusters, see Table 5 .Table 5 : Significant similarity clusters derived from SPM5 [42] for RSA analysis. Anatomical region labels are derived from [36] . The p corrected and cluster size (cluster k E ) values are both corrected using SPM5. Note that some layers may involve more than one region due to the probabilistic nature of atlases in [36] . Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper's central results are not about social impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: We have no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We've credited 3rd party code bases we used in the research in the appendix.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.Our code is available at github.com/chenghuzi/goal-reducer. 38th Conference on Neural Information Processing Systems (NeurIPS2024).Data is available at osf.io: part 1,part 2 and part 
 
 Appendix:  


Decision: This paper proposes a reinforcement learning method (snipping out loops in collected data), presents results showing that this method improves performance, and shows evidence that this method may be similar to biological networks. Reviewers found the paper well written, commending the paper for including both RL experiments and human neuroimaging experiments. Reviewers also appreciated the high degree of reproducibility, experiments demonstrating that the proposed method works for multiple RL backbones, and the discussion about removing the need for domain knowledge. The reviewers asked some conceptual questions that the authors clarified during the review process. The authors shared code with the AC, per NeurIPS guidelines. During the reviewer discussion, Reviewer i4LG (score=9) reviewed the concerns of Reviewer Szim (score=4), and agrees with the authors that those concerns may stem from a misunderstanding. Taking this all into account, I recommend that the paper be accepted.

Reviews: review: paper_summary: paper_summary: This paper introduces a new automated method to identify subgoals in reinforcement learning. The method is based on a replay buffer of agent trajectories and proceeds by identifying and eliminating “loops” (points where the agents revisit almost the same state), in effect attempting to identify efficient shortcuts in the replays and set those shortcuts as subgoals. This method for subgoal extraction can be applied to a variety of reinforcement learning algorithm, and is successfully demonstrated with two popular approaches (deep Q learning and soft actor critic), leading to performance gains in two environments (maze exploration and robotic arm control). In addition, a human neuroimaging experiment shows that the latent space of the “goal-reducer” components of an agent shares similarity with brain activity in key regions, suggesting that the proposed goal reduction mechanism may be at play in biological networks. In summary, the paper proposed a new broadly applicable approach to identify subgoals for reinforcement learning, demonstrates its effectiveness on established RL benchmarks and also offers evidence in support of a biological implementation of this process in human brains The paper thus presents a very strong and multidisciplinary set of evidence.
strength_weakness: strength_weakness: Although the authors claim that the code was shared with the manuscript, I was not able to find it. I either did not look in the right place (I was expecting a supplementary material link on the submission page, or a link in the manuscript), or the authors forgot to upload it. In any case, access to the code for review would address this weakness, but in the current state I am not able to assess the quality of the code and whether the code seems to cover all the parts in the paper. Also, it is stated that the fMRI data is available “upon reasonable request”, it is not possible for me to assess to what extent the authors will actually be responsive, and whether the data is reusable. I strongly encourage the authors to use an established open data sharing platform such as openneuro and standard data structure such as BIDS to increase reproducibility. Statistical maps could be shared on neurovault for increased reproducibility. 

It would have been beneficial to contrast different models in the neuroimaging experiment, and incorporate some baseline models and an ablation study. The origins of the similarity in representations may be trivial.
questions: questions: The authors extend the grid exploration task to 19 x 19 cells in order to test the scalability of the method. Why 19? And not 20? Have the authors found a “breaking point” in the scalability of the approach? Also, insights on the computational scalability of the approach would be useful. 

The authors identified similarity of representation with the goal reducer in the white matter (which appears significant in the provided putamen slices). How do the authors explain that? Could it suggest some motion or physiological artifacts correlated with the artificial representations? 

Although fMRI results are presented, this neuroimaging modality has very limited temporal resolution. Could this affect the representational similarity analysis? It looks like the steps of the games are slow by design, but this could be discussed briefly.
limitations: limitations: There is no clear neuronal mechanism proposed in support of the goal reduction process, beyond association with an indirect brain measure (blood oxygenation). But bringing any kind of multidisciplinary evidence is already very commendable.
paper_summary: paper_summary: The paper proposes a goal reduction mechanism utilizing a novel loop-removal technique to derive subgoals from arbitrary and distant original goals. This method, called goal-reducer, distills high-quality subgoals from a replay buffer without prior global environmental knowledge. It accelerates performance in various RL frameworks and matches human brain activity in goal-directed tasks.
strength_weakness: strength_weakness: Refer to questions.
questions: questions: - In algorithm 1, how to calculate $||s_t - s_t'||$? Does this distance depend on Euclidean distance? If yes, the Euclidean distance is a strong prior knowledge of the environment to the agent.
- I agree that SoRB relies on strong prior knowledge. However, there are some related work that trains a sub-goal generator along with the RL learning process, e.g., [1]. 
- Can this method be extended to any goal state? If there are multiple goals or no specific goal state is specified, then removing the historical state traversed by the agent will weaken the agent's ability to reach certain goals.
- Why didn't the authors compare the performance of GR+DRL  with SoRB? 

[1] Shuang Ao, Tianyi Zhou, Guodong Long, Qinghua Lu, Liming Zhu, Jing Jiang, CO-PILOT: COllaborative Planning and reInforcement Learning On sub-Task curriculum
limitations: limitations: N/A
paper_summary: paper_summary: The paper provides a novel way to find subgoals just from past experiences (without any prior knowledge), named goal-reducer. Goal-reducer works by finding “loops” in trajectories and remove them. The authors extend the concept of loops in the continuous cases by defining a filtration radius (distances of two states lower than the radius means they are the same state). Through experiments, the authors show goal-reducer can 1) help RL by providing subgoals; 2) can work on its own to find immediate next subgoals and outperform RL; 3) has a corresponding human brain activity in tasks involving subgoals. In general, the paper is sound, makes meaningful contributions, and is written well.
strength_weakness: strength_weakness: 1. I am not sure about the applicability of goal-reducer beyond just goal-reaching behavior. Take the fMRI study task as an example, how do we make sure the goal-reducer does not condense these two states together: both states are in the same place, but one with the “key” and one without the “key”. If the mapping to embedding space is random, we cannot guarantee the distance of having/not-having the key is larger than the distance of being in two different places. Essentially, I am saying that what if the loop detour is necessary to get some treasure. I wonder about the authors’ thoughts of the applicability of goal-reducer in harder tasks than just navigation. 
2. I am not sure about the motivation/justification for Equation 6-7: why should the loss be present only when the entropy decreases when the subgoal is introduced? Wouldn’t that encourage the policy to be less certain about what to do when given the subgoal? 
3. Line 297 mentioned Indiana University IRB – while it is great to show the study was IRB-approved, I wonder if this is in violation to anonymity requirements in reviewing stage.
questions: questions: See weakness
limitations: limitations: While the paper discusses their approach's limitations, they do not discuss the potential negative societal impact of their work. I hope the authors consider including them in the rebuttal.

Comments: title: Response
Code
comment: We thank the reviewer for the feedback. We are concerned, however, that the use of the Euclidean distance calculation in our work may have been misunderstood. When the agent is randomly initialized, there is no prior knowledge other than that **repeatedly visited states share the same or very similar hidden representation and thus imply the existence of a loop**. This is a minimal and universal assumption across different tasks. Beyond recognizing “loops”, we do not use Euclidean distance for any other purposes, such as value/cost estimation.
Regarding data quality and representativeness, the ethics reviewer states that “The author(s) should be prepared to share the data source if required by NeurIPS in order to validate the ‘Data quality and representativeness’.” In response to a comment by the original reviewers, we have already uploaded the fMRI data used for the representational similarity analysis to a data sharing website and provided the information to the editor. We apologize that this information was not previously shared with the ethics reviewer, as we see that the “Added material” in response to Reviewer 2, where the data sharing links are provided, was not visible to the ethics reviewers.

Regarding the identification of the IRB of record, we were attempting to strike a balance between anonymity and transparency with respect to the ethical approval. Unlike with anonymized data sharing, if we did not name the IRB, there would be no way to verify that we had obtained IRB approval. In any case, we apologize for identifying the IRB of record.
Thanks for the detailed explanations on the equation. This was more of a general comment, but I still appreciate the effort. 

Re code and reproducibility, I have received a note below that the code was shared, and meets acceptable standards of quality. 

Re Q1 (size of room). Thanks for the explanations. Scaling the experiment as was done by the authors in response to reviews further strengthens the conclusions.

Re Q2, it's indeed possible that white matter activations are an effect of smoothing. This could be tested by reducing smoothing (although it's not advisable for the main analysis: smoothing is required for improving SNR and correction of multiple comparisons).. 

Re Q3: thanks for the clarifications. 

Also thanks for the speculations regarding the putative role of the hippocampal formation in loop suppression. 

I am keeping the score at 7 "accept".
Thanks for this information!
Thank you for the author's reply, which solved some of my problems. However, I still stick to my point of view that Euclidean distance is a strong prior knowledge, which is a critical ability of the agent. I agree that for the agent, the size of the Euclidean distance is not strictly equal to the cost, but imagine if we have the distance information, the goal generator is essentially a search problem. SORB does not rely on this prior knowledge, which explains why SORB might be inefficient in the early days when the Q network was immature.
Thank you to the authors for answering my questions! I will keep my score.
Dear Reviewer,

The authors have privately send me a link to their code and data for the human studies (in line with NeurIPS policies about sharing code). The data takes the form of several `.mat` files. The code seems reasonable and contains documentation for installation and for running experiments. During the discussion period, please respond about whether this addresses your concerns about the code and data.


Kind regards,

The AC