Rejecting Hallucinated State Targets during Planning
Authors: Harry Zhao
Tristan Sylvain
Romain Laroche
Doina Precup
Yoshua Bengio
Venue: ICML 2025 poster (2025)
Keywords: reinforcement learning
planning
generative models
hallucinations
delusions
deep learning
neural networks
Primary area: reinforcement_learning->planning
TL;DR: We provide a generic solution to reject hallucinated state targets during planning for decision-making agents.

Abstract: In planning processes of computational decision-making agents, generative or predictive models are often used as "generators" to propose "targets" representing sets of expected or desirable states. Unfortunately, learned models inevitably hallucinate infeasible targets that can cause delusional behaviors and safety concerns. We first investigate the kinds of infeasible targets that generators can hallucinate. Then, we devise a strategy to identify and reject infeasible targets by learning a target feasibility evaluator. To ensure that the evaluator is robust and non-delusional, we adopted a design choice combining off-policy compatible learning rule, distributional architecture, and data augmentation based on hindsight relabeling. Attaching to a planning agent, the designed evaluator learns by observing the agent's interactions with the environment and the targets produced by its generator, without the need to change the agent or its generator. Our controlled experiments show significant reductions in delusional behaviors and performance improvements for various kinds of existing agents.

Rejecting Hallucinated State Targets during Planning
Mingde "Harry" Zhao 123 Tristan Sylvain Romain Laroche
Doina Precup" 125
Yoshua Bengio 16
add-on
TAP agent
hallucinated
reject
Abstract
targets
feasibiity
EVALUATOR
In planning processes of computational decision- making agents, generative or predictive models are often used as "generators" to propose "targets" representing sets of expected or desirable states. Unfortunately, learned models inevitably halluci- nate infeasible targets that can cause delusional behaviors and safety concerns. We first investi- gate the kinds of infcasible targets that generators can hallucinate. Then, we devise a strategy to identify and reject infeasible targets by learning a target feasibility evaluator. To ensure that the eval- uator robust and non-delusional, we adopted a design choice combining off- oolicy compatible learning rule, distributional architecture, and data augmentation based hindsight relabeling, At- taching to planning agent, the designed evaluator learns by observing the agent's 's interactions with the environment and the targets produced by its generator, without the need to change the agent or its generator. Our controlled experiments show significant reductions in delusional behaviors and performance improvements for various kinds of existing agents.
GENERATOR
feasible targets
accept
paired training data
other components
reach Indicator
Earpel enpoder
actions
hindsight experience replay
ibseraations states rewards-
environment
Figure I. Target Evaluator attached a Target-Assisted Plan- ning (TAP) Agent: An abstracted framework that encompasses. but is not limited to, methods listed in Tab. (in Appendix). The generator proposes candidate target embeddings gº. Our pro- posed evaluator can be attached to learn to reject infeasible targets (related parts marked in dashed lines).
some rollout-based model-based RL methods are TAP, such as those following the classic Dyna or Monte-Carlo tree- search frameworks, utilize fixed-horizon transition models to simulate experiences (Sutton, 1991; Schrittwieser et al., 2019; Lukasz Kaiser et al., 2020); While, TAP also en- compasses methods that directly generate arbitrarily distant targets acting as candidate uub-goals to divide-and -annque the tasks into smaller, more manageable steps (Zadem et al., 2024; Zhao al., 2024; Lo et al., 2024).
1. Introduction
It is the defining characteristic of TAP the usage of gen- erators, that brings us to the topic of this work: an often unstated assumption is that generated targets are always feasible. However, the desired generalization abilities of generative models are inevitably accompanied by hallucina- tions (Xu al., 2025; Zhang et al., 2024b; Xing et al., 2024; Jesson et al., 2024; Aithal al., 2024) the "dark side" that produces infeasible targets that can never be experienced by anyhow. Hallucinations impact TAP agents differently based on their planning behaviors. In decision-time TAP methods (Alver & Precup, 2024), where models are used to make an immediate decision on what to do next, halluci- nated targets can lead to delusional plans that compromise performance and safety (Langosco et al., 2022; Bengio et al., 2024). For background TAP agents that train simulated experiences constructed with generated targets, delusional values estimated from hallucinated targets can be catastroph- ically destabilizing (Jafferjee et al., 2020; Lo et al., 2024),
The advent of computational modeling has spurred advance- ments in computational decision making agents, most no- table of which is, model-based Reinforcement Learning (RL). This work is focused those models that play the role of generators, which imagine or specify future out- comes for agents. Some generators imagine next states or observations, while others specify subgoals (sets of states) to accomplish. No matter how they are represented, we refer to these generator outputs as targets and methods that make such use of generators Tasset-Assisted Planning (TAP).
TAP is new perspective for unifying many planning/ rea- soning agents with wildly different behaviors. For instance,
Mila (Quebec AI Institute) 'MeGill University 'Wayve "RBC Borealis Gooole Deepmind Université de Montréal. Correspon- dence to: Mingde "Harry" Zhao <mingde. mail.mcgilll megill.ca>>
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Human brains reject infeasible intentions hallucinated by the belief formation system (brain's 's generator) using be-

---

Rejecting Hallucinated State Targets during Planning
lief evaluation system (Kiran& & Chaudhury, 2009). Taking inspirations, we propose to assist existing TAP agents with target evaluator, which can be used to reject infeasible targets and in turn delusional behaviors. To maximize com- patibility, the evaluator is designed as a minimally-intrusiee add-on to attach to existing TAP agents Zhao et al. (2020): learns by observing the TAP agent's interactions with the environment and the targets produced, without the need to change the agent's behaviors or the architecture. Our main contributions are as follows:
zons, we also introduce T the maximum number of time steps an agent is allowed to reach state in go.
Let D (s. represents 1s timestep t s.t. h(s;,g = 1, if т (conditional or not) is followed from state s. We define T --feaiibilitty of go from state under π as (D g)) ك 7) = ET p (x, = t). 9ô is called r-fesiible if (( S T) > 0, and r-infeasible otherwise. No- tably, r-feaiibility is an intrinsic property of state-policy- target tuple, not heuristic measure.
A target is generally evaluated as "good" if it leads to re- warding outcomes, i.e.:
1. We systematically categorized and characterized infeasi- ble targets w.r.t. time horizons
ux g,) = rx ( 9 T) + To g Va (m)i)(())((.)0)).)) (I)
2. We discussed the desiderata of learning a minimally- intrusive non-delusional target evaluator
3. We proposed a combination of a) off-policy compati- ble update rule that enables the evaluator to learn by observing, b) an evaluator architecture compatible with different time horizons that enables unified solution for most TAP agents and c) two assistive hindsight rela- beling strategies performing data augmentation to pro- vide training data beyond those collected via interactions, which ensure that the evaluator itself does not produce delusional evaluations
where min(Da g)),))))))))) denotes the timestep when the commitment to go is terminated (by h or T) (min(Dr(*g))))) is the state the agent ended up in, Ir (8,g 7) = Σ (.. (,,)))))))))))" gt- is the cumula- tive discounted reward along the way, 7x (,,,g97, = min (x.9)) is the cumulative discount, and Va(...) is the future value for following μ afterwards.
Eq. shows that g is rinfeesaseble, i.e., Smin(D..r) $ go then TAP methods blindly using g to determine Vn, will produce delusional evaluations the cause of delusional planning behaviors. For example, feasibility meth- ods, e.g., Sutton (1991); Schrittwieser et al. (2019): Hafner et al. (2025), assume that targets are always reachable as long as they can be generated. Meanwhile, planned trajec- tories involving infcasible targets are delusional; There are also some feasibility- aware methods, e.g. Nasiriany et al. (2019); Zhao al. (2024); Lo et al. (2024), in which agents estimate certain metries to decide if target is feasible. How- ever, as we will discuss later, they often produce incorrect estimates, thus may still favor infeasible targets. In later sec- tions, we propose an evaluator that simultaneously estimates the r-feasibility and D, of the proposed targets, where these estimations are used to decide if the evaluation of target should be trusted or if the target should be rejected.
4. We implemented the solution, as illustrated in Fig. 1, for several types of existing TAP agents, as discussed in Tab. 2 (in Appendix), and showed that agents can better manage generated targets, reduce delusional behaviors and significantly improve performance
2. Preliminaries
Problem Setting: For the readers' case of understanding, in this work, we use RL agents as representative of generic computational sequential decision decision-making systems. We model the interaction of an RL agent with its environment as a Markov Decision Process (MDP) M = (S,A, R. d, 7), where S and A are the sets of possible states and actions,
P S x A- - Di defines the state transitions, R: SxAxS + IR. defines the rewards, d: S > Dist(S) is the initial state distribution, and 7 E (0, 1] is a discount factor. An agent needs to improve its policy x S + Dist(A) to maximize the value, i.e., the expected discounted cumulative return E, E 7² y',,S, A4, S1+1) Su ~ d.. where Tı denotes the timestep when the episode terminates. Some environments are partially observable, which means that instead of a state, the agent receives an observation I1+1, used to infer the state (possibly with episodic history).
Source- Target Pairs & Hindsight Relabeling To learn the feasibility of a target from a given state, "source-target pairs" are needed, which are tuples involving a source state and a target embedding. The quality of these paired training data is critical for the training outcome (Dai et al., 2021; Moro et al., 2022: Davehev et al., 2022). Hindsight Expe- rience Replay (HER) can be seen as way to enhance the diversity of the pairs, by re-using targets that happened to have been achieved on existing trajectories, and pretending that they were the chosen targets during the interactions (Andrychowicz et al., 2017). HER augments a transition with an additional state s which can be passed through a target embedding function g to acquire the target embed-
Targets: for generality, we consider a target to be an embed- ding representing a set of states. Each target gD + ( is paired with an indicator function å outputting h(s' gg = if s' E { and otherwise. For the interest of time hori-
2

---

Rejecting Hallucinated State Targets during Planning
Table Categorization of Targets based on Composition, Characteristics, Risks Delusion Mitigation Strategies
Data Augmentation (Relabeling) Strategies against Feasibility Delusions
Target Composition
State Correspondence
-x-Feisiiiliity p(D+(a,g)) < 00)
Feasibility Errors Resulting
Delusional Planning Behaviors
E.0: May think G.O states infeasible, thus turn riskier alternatives, e.g., G.I or G.2 E.I: May think G.I states are favorable, thus commit to them. Impacted by ill-defined (-.)
EPISODE for G.0 (and G.2) states the same episode PERTASK for G.0 (and G.2) beyond the episode
Only or Single G.0
non-hat hon-hallucinated feasible states from
> -
hallucinated "states" not belonging to the MDP
Only Single G.1
GENERATE for G.I (and G.O G.2)
should 0
states, to be proposed by the generator
hallucinated MDP states infeasible from
PERTASK for G.2 (and G.0) heyond F.2: May think G.2 states are favorable, thus commit to them
Only or Single G.2
should
episode EPISODE for G.2 (and G.U)
states in the same episode
at least one non-hallucinated
Some G.0
(( < oc) > (Thm. 4.1)
E.0
EPISODE + PERTASK
state from s set of ONLY hallucinated states
Only G.1 & G.2
should = 0 E.1 & E.2
GENERATE GENERATE PERTASK
ding )( as the relabeled target. Relabeling strategies, corresponding to how و is obtained, are critical for HER's 's performance (Shams & Fevens. 2022: Eysenbach et al., 2021). Most existing relabeling strategies are trajectory- level, meaning that O comes from the same trajectory 8t. These include FUTURE, where 8 +- with l' > I, and EPISODE. with 0 < t' S. T1. HER greatly enhanced the sample efficiency of learning about experienced targets. Meanwhile, its limitations to learning about only experi- enced targets planted hidden risk of delusions towards hallucinated targets for TAP agents, be discussed later.
if they were evaluated from a different source state, G.2s can often be overlooked, not only because hallucinations are mostly studied in contexts that do consider the source state 8, but also because they only exist in some special MDPs.
3.4. Examples
To provide intuition about these concepts, we use the Min- iGrid platform to create a set of fully-observable environ- ments, minimizing extraneous factors to focus on the targets (Chevalier-
et al., 2023). We call this environment SwordShie Me wwwsseeeesssssiiieeeeeeioeaaoile dMonster (SSM for short); In SSM, agents navigate by moving one step at a time in one of four direc- tions across fields of randomly placed, episode-terminating lava-traps, while searching for both a sword and a shield to defeat monster with terminal reward. The lava-traps' density is controlled by a difficulty parameter 6, but there is always a feasible path to success. Approaching the monster without both the randomly placed sword and shield ends the episode. Once acquired, either of the two items cannot be dropped, leading to a state space where not all states are accessible from the others. Thus, SSM states are partitioned into 4 semantic classes, defined by 2 indicators for sword and shield possession. For example, (0, 1) denotes "sword not acquired, shield acquired"..
3. Hallucinated Targets in Planning
Categorizing targets proposed by the generator helps inform us about how to correctly learn the feasibility of targets.
Let us warm-up with singleton targets, i.e. oo has a single element 8D, and propose a characterization of singleton targets into 3 disjoint categories. Then, we will extend to the non-singleton targets composed of these 3 "ingredients".
3.1. G.O: oo-Feasible
Given source state s, a generated singleton target " is called G.0 if it maps to one state which is co-feasible from 8. with some policy æ. Note that G.O includes 7 r-infeasible states for given finite values of T.
G.1 generations in this environment may be semantically valid, e.g., an SSM "state" with the agent surrounded by lava, as in Fig. 2 (top row), or totally absurd, e.g., an SSM observation without an agent.
3.2. G.I Permanently Infeasible (Hallucinated)
G.1 includes generated "states" that do not belong to the MDP at all, i.e., a target "state" g is G.I if Vs, n, p(D,(s, 30) < oc)= = 0.
G.2 states can be once G.O but are now blocked due a past transition, e.g., after acquiring the sword in SSM, the agent transitions from class (0, 0) to (1, 0), sealing off access to (0,0) or (0, 1); G.2 can also appear due to the initial state distribution some states can only be accessed from specific initial states, e.g., an agent spawned in (1, 0) cannot reach (0, 0) or (0, 1). An example of delusional behavior caused by a G.2 target is provided in Fig. (bottom row).
3.3. G.2 - Temporarily Infeasible (Hallucinated)
This type includes those MDP states that are currently in- feasible from state s. Unlike G.I, G.2 states could be G.O
3

---

Rejecting Hallucinated State Targets during Planning
gets becomes limited to the resource cost of generating and rejecting targets, to be discussed in detail. This approach is in contrast with directly trying to address hallucinations in the generators case-by-case, which we deem to have an unbreakable glass ceiling and not versatile enough to be generalized to generic TAP methods.
agent (moves directions)
lava (episode ends)
D-1 (delusional Peasi biity
sword (can't drop picked up)
hallucinated larget (G.1)
shield (can't 't drop if picked up)
For a feasibility evaluator to be effectively differentiate the proposed targets, should correctly estimate the feasibility of targets which maps to all kinds of states (G.0, G.I & G.2). However, learning to estimate feasibility is not as trivial as it seems, because improper training could naturally lead to delusional feasibility estimations, which cannot be simply addressed by scaling up training. If the evaluator has delusions of feasibility, then its incorporation becomes futile, as hallucinated targets could still be favored.
monster
(episode ends, +1 reward wr
-inddcced delusional behavior
hallucinated target (G.2)
Del (delasional leasibility
For estimation errors, we similarly warm up with those of the singleton targets. For clarity, we use matching identifiers E.O, E. and E.2 to denote the estimation errors of feasibility towards G.O, G.I, and G.2 "states", respectively. These discussions are presented in Tab. 1.
sward& & shield both NOT asquirred
sword shield already acquired
G.2-induced delusional behavior (e in (1, 1). 9 in (0, 03)
Figure 2. Delusional Plans SSM: In both cases, TAP agent, lacking understanding about the hallucinated targets (yellow dots), mis- evaluates their feasibility, leading to delusional plans which suggest that the task goal can be achieved via the infeasible targets.
When targets correspond to general sets of states, the fol- lowing holds:
Theorem 4.1. Let g be larget embedding. Its feasibility from state satisfies:
V (t. S) =p( ST
Despite rising concerns regarding the safety of TAP agents (Bengio et al., 2024), their delusional behaviors remain under-investigated, largely due to the lack of access to ground truths needed to identify hallucinations and their resulting delusional behaviors. Thus, it is critical to analyze with clear examples and conduct rigorous controlled exper- iments where the ground truth of targets could be solved with Dynamic Programming (DP) (Howard, 1960), which why we created SSM and used it later for experiments.
where 9D is target that correspond to the set of states of g with all infeasible states (G.1 & G.2) removed.
This result indicates that a target is infeasible if and only if it consists entirely of infeasible states, allowing us to focus on learning processes that identify such cases.
4.1. Desiderata for Evaluator
We used the following important considerations to guide our design for an appropriate feasibility evaluator. [automatic] the evaluator must learn to automatically differentiate the feasibility of all kinds of targets without pre-labeling: we need exploit h
3.5. Non-Singleton Targets
For the general case, where a generated target embedding go potentially corresponds to a set of "states" (8*), the elements of the associated set may span all categories, i.e., the target can correspond to a mixture of G.O, G.I & G.2 states. Tab. summarizes the possible target compositions and their properties!.
[minimally intrusive] the evaluator should be generally applicable to existing TAP agents, without changing the agents too much to disturb the generally-sensitie RL components: we need to ensure its behavior as an add-on and it can be conditioned on the policy π of the agent, to learn alongside the agent by merely observing
4. Evaluating Targets Correctly and Robustly
Knowing that hallucinations cannot be eradicated in general, we intend to lower their risks by adopting the brain-inspired solution to reject infeasible targets post-generation. If done effectively, the negative impact of hallucinated tar-
[unified] the evaluator should have a unified behavior compatible with different rS: we can design it in way to learn the 7-feasibilitiee for many T values simultaneously
4.2. Learning Rule & Architecture for Feasibility
'There may be no explicit mapping from a target embedding to set of "states" and thus any target can always map arhitrarily many G.1 "states". Thm. 4.1 explains that this problem is henign.
Following the considerations, we propose to use the follow- ing learning rule to indirectly learn the targets' feasibility
4

---

Rejecting Hallucinated State Targets during Planning
by directly learning the distribution of (s.
ators from learning about distant targets, risking delusions when such distant targets are proposed. Short trajectories can be the product of experimental design (initial state distri- butions, maximum episode lengths (Erraqabi et al., 2022), or environment characteristics, e.g., density of terminal states).
Da g ) + 1 Dn((', .g )) with
{ D=( = (8 a, g) T g
{ (s's' = xo if s terminal and h(s', g)) (,(s' 0 if h(s 9 =1
Avoiding feasibility delusions requires learning from all kinds of targets, including those that can never be experi- enced. This is counter the exposure bias the discrepancy between (most existing) TAP agents' behaviors (involving all targets that can be generated) and training (learning from only experienced targets), identified in Talvitie (2014).
This results in off-policy compatible policy evaluation process over a parallel MDP almost-identical to the task MDP, but adapted for go, where all transitions yield "re- ward" +1 and states satisfying do are changed to terminal with state value 0. Every time an infeasible target embed- ding is sampled for training, the update rule will gradually push the estimate towards oo, for all sampled source state s.
We introduce data augmentation (relabeling) strategies to expand training source-target pair distributions.
Our design only learns Drr in way that can lead to T- feasibilities p( p )) S r). For this purpose and the consideration for unified design, we propose to use Eq. 2 in conjunction with CSt-style distributional archi- tecture (Bellemare et al., 2017), which outputs a distribu- tion represented by a histogram over pre-defined supports. When we set the support of the estimated D= ) to be 21, T] with T sufficiently large, the learned his- togram bins via Eq. 2 will correspond to the probabilities of (s, g ) = t) for all & E f1..... T - 1}. This technique of using C51 distributional learning enables the extraction of r-feasibility p (D,(s, (8,9 ) S T) from a learned T-feasibility with ((..(s.)) 0) t) over E (I....,T) T), thus learning all 7-feasibility with t < T simultaneously. Take the example of the 1-step Dyna agent we implemented for experiments (Sec. 5.2): if the estimated histogram has little probability density for p (s, ) = 1), then the tar- get (simulated next state) is likely hallucinated and should be rejected, avoiding a potential delusional value update.
4.3.1. GENERATE: EXPOSE CANDIDATES TARGETS (TO BE GENERATED)
The first strategy, named CENERATE, is to expose the targets that will be proposed during planning to the evaluator, so that it can figure out if these targets are infeasible.
We can implement this as a Just-In-Time (JIT) relabeling strategy that relabels sampled (un- (un-relabeled) transition for training with a generated target (provided by the generator). We can expect GENERATE be effective, evaluators will get exposed to hallucinated targets that the generator could offer. Note that GENERATE requires the use of the generator, thus it incurs additional computational burden, depending on the complexity of target generation. The JIT-compatibility lowers the need for storage and provides timely coverage of the generators' changing outputs, especially helpful for non non-pretrained generators. The idea behind GENERATE can be traced back to Talvitic (2014).
Note that the C51 architecture also allows us to extract the distribution of 7/x (s. (.s.go.)) which, as defined in Sec. 2, the cumulative discount with chosen target. This is done via transplanting the output histogram of (s, 2) over
new trajectory be labeled:
rellabeled transition (snurce - target palr) that helps estimaturs understand it's impoosible to go from et, Ds to <ft Is
[1,2,... T, + 1, +2.... onto the changed support of
old trajectory sampled by "pertask".
4.3. Training Data for Feasibility
Figure 3. An Example of How PERTASK Reduces E.2 by Sam- pling Across Episodes: The new trajectory acquired the sword first and the shield later, while the old trajectory acquired the shield first and then the sword. When relabeling transition in the new trajectory (in (1, 0)), target observation in the old trajectory (in (0, 1)) can be paired to make an agent realize the infeasibility of the relabeled target, thus reducing E.2 delusions.
With the proper learning rule and architecture, we now need to ensure that the evaluator have proper training data and does not become delusional. In Sec. 2, we mentioned the incompleteness of the relabeling strategies, which will be discussed in detail now: 1) Certain relabeling strategies nat- urally create exposure issues, even for G.O targets. For in- stance, FUTURE only relabels with future observations, thus only exposes a learner to future feasible targets, confusing the evaluator when "past" target is proposed during plan- ning; 2) Trajectory-level relabeling is, by design, limited. Short trajectories, common in many training procedures, cover limited portions of the state space and prevent evalu-
4.3.2. PERTASK: EXPOSE EXPERIENCED TARGETS BEYOND THE EPISODE
The second strategy, named PERTASK, is to expose the evalu- ator to all targets g)) experienced before, so that it could realize if some previously achieved targets not present in the current episode are infeasible from the current state.
5

---

Rejecting Hallucinated State Targets during Planning
We implement PERTASK by relabeling transitions with (the target embedding of) observations from the same training task, sampled across the entire replay. PERTASK can be seen as an generalization of the "random" strategy in Andrychow- icz et al. (2017) to multi-task training settings. Importantly, PERTASK exposes the evaluator to E.2 delusions and to long- distance E.0 caused by trajectory-level relabeling on short trajectories. An example is shown in Fig. 3.
of strategies pose tradeoff the learning of different kinds of source-target pairs. In the experiments, we mainly used (E+P+G), which is a mixture of 2/3 EPISODE and 1/3 PERTASK, with 1/4 chance using GENERATE JIT, result- ing in a mixture of 50% EPISODE, 25% PERTASK and 25% GENERATE. (E+P+G) exploits the fact that assisting EPISODE with GENERATE and PERTASK often results better perfor- mance in evaluator training, striking a balance between the investment of training budgets for the feasible and infeasible targets (Nasiriany et al., 2019; Yang et al., 2021).
4.3.3. APPLICABILITY
PERTASK cannot address E.1 delusions. Meanwhile, GENERATE can be used against some G.2 targets that the generator hallucinates. PERTASK is a specialized and compu- tationally efficient strategy to reduce feasibility delusions to- wards all experienced G.2 target states and importantly also the long-distance E.O errors that GENERATE cannot handle. PERTASK is expected be more effective than GENERATE in generalization-focuss scenarios, where the distribution of G.O & G.2 targets proposed by the generator during evalua- tion can go beyond those trained under GENERATE.
4.4. Computational Overhead
The portion of overhead for the evaluation of targets is straightforward, as each target will be fed into the neural networks (paired with a source state) for a forward pass at inference time. This portion of the overhead depends on the complexities of evaluator's 's architecture and of the state target representations. We can expect fast evaluations with lightweight evaluators.
It is the strategy post evaluation that determines the overall overhead, which depends on the planning behavior of the TAP agent that the evaluator is attached to. For background TAP agents that generate batches of targets, the improper ones can be rejected and the whole batch can be all rejected without problem (no Dyna update this time). For decision- time TAP agents, targets act as subgoals and when they are rejected, the agent can either re- re-generate commit to more random explorations.
Importantly, relabeling strategies such as FUTURE, EPISODE and PERTASK rely on the existence of g that maps a state into a target embedding, which is commonly found in TAP agents (Andrychowicz et al., 2017). However, if only the target set indicator function h is available. we may need to accumulate (s. g) tuples for which h(s,g) g) 1, and the use them to train a g. Or, in the cases where feasibility is only used for rejection, such as when dealing with simulated experiences and tree search, we could also rely on only GENERATE, which does not require g: Sometimes, is h that needs to be constructed. We provide detailed discussions for applying our solution Dreamer V2 in Sec. J (Appendix), with focus on how to construct a proper h.
5. Experiments
To investigate the effectiveness and generality of our pro- posed solution against delusional behaviors caused by hallu- cinated targets, we use a simple and unified implementation of our evaluator (3-layers of ReLU activated MLP with out- put bin T 16 and (E+P+G)) for 8 sets of experiments, encompassing decision-ting time V.S. background planning, TAP methods compatible with arbitrary 7S and fixed rs, single- ton and non-singleton targets, on controlled environments with respective emphases on G.I and G.2 difficulties. The implementations for these experiments can be extended to various existing TAP methods, per Tab. 2 (in Appendix). Due to page limit, we only provide brief summaries of our experimental results in the main paper and refer the readers to the Appendix for more comprehensive details.
4.3.4. MIXTURES
Both GENERATE & PERTASK bias the training data distribu- tion, making the evaluator spread out its learning efforts to the source-target pairs possibly distant from each other. Despite increasing training data diversity, distant pairs are less likely to contribute to better evaluation compared to the closer in-episode ones offered by EPISODE, as close- proximity G.O targets matter the most.
Creating a mixture of sources of training data can increase the diversity of source-target combinations. For HER specif- ically, each atomic strategy, enumerated in Tab. 3 and il- lustrated in Fig. 8 (Appendix), exhibits different estimation accuracies for different types of source-target pairs, includ- ing -hort-distance and -in-ddistncce ones involving all of G.O, G.I and G.2.
Exp.//s: Skipper (decision-time TAP with singleton tar- gets, arbitrary 7) on S5M
Exp.2/s: (Appendix, Sec. C) LEAP (decision-time TAP with singleton targets, arbitrary 7) on SSM
Exp.3/s (Appendix, Sec. D) Skipper on RDS, another con-
When the training budget is fixed, i.e., training frequency, batch sizes, etc., stay unchanged, the mixing proportions
trolled environment focusing on G.1 difficulties
Exp. LEAP on RD5
6

---

Rejecting Hallucinated State Targets during Planning
Exp. Dyna (background TAP with singleton targets, 1) on SSM
For such experimental purpose, we use distributional shifts provided in SSM simulate real-world OOD systematic generalization scenarios in evaluation tasks (Frank et al., 2009). For each seed run on SSM, we sample and preserve 50 training tasks of size 12 x 12 and difficulty = 0.4. For each episode, one of the 50 tasks is sampled for training. Agents are trained for 1.5 x 10 interactions total. To speed up training, we make the initial state distributions span all the non-terminal states in each training task, making trajectory-level relabeling even more problematic.
Exp.5/s: (Appendix, Sec. F) Dyna on RDS
Exp.7/s: (Appendix, Sec. G) Feasibility estimation of non-
singleton targets with arbitrary r on SSM
Exp.9/s: Feasibility of non-singleton targets with arbitrary I on RDS
Skipper Skipper+
E.0 ceror D.6..)
El1 crror De(-))
5.1.1. METHODS
To demonstrate the generality of our proposed solution against hallucinated targets for decision-tinn iitiiionttmmeee TAP, we apply it onto two methods utilizing targets quite differently:
3
Skipper (Zhao et al., 2024): generates candidate target states that, together with the current state, constitute the vertices of directed graph for task decomposition, while the edges are pairwise estimations of cumulative rewards and discounts, under its evolving policy. A target is chosen after applying value iteration, í.e., the values of targets are the U4 values of the planned paths.
source-farge distances
training steps
(a) Final Errors by Distance
(b) Evolation of E.I Errors
aagreesaie 000 maccest rit
Skipper
Skipper+
E2 Dafos)
2
LEAP (Nasiriany al., 2019): LEAP uses the cross-entropy method to evolve the shortest sequences of sub-goals leading to the task goal (Rubinstein, 1997). The immediate sub-goal of the elitist sequence is then used to condition a lower- level policy. Compared to Skipper, LEAP more prone to delusional behaviors, since one hallucinated sub-goal can render a whole sub-goal sequence delusional."
0.0
0.5
1.0 1.5 3i
0.0
0.5
1.0
1.5 See
training steps
training steps
(c) Evolution of E.2 Eroors
(d) Aggregoted 0OD Performance
Figure 4. Skipper on SSM: We compare the original form of Skip- per, which learns its own feasibility estimates of target states in its own way, against Skipper+,, which has our proposed evaluator injected to assist the evaluation of the feasibility of targets, pow- ered by the (E+P+G) relabeling strategy. Results more variants are presented in Fig. 10 (Appendix). a): Final E.0 errors sepa- rated across a range of ground truth distances. Both estimated and true distanees conditioned on the evolving policies; b): E. errors measured as L1 eтror in estimated (clipped) distance throughout training: G.2-counterparts of b; d): Each data point represents OOD evaluation performance aggregated over 4 x 20 newly generated tasks, with mean difficulty matching training- The decomposed results for each OOD difficulty are presented in Fig. 11 (Appendix).
For Exp. 1/8 -/s,, targets are observation-lis generations, where G.1 & G.2 can be clearly identified. See the Sec. A.2 (Appendix) for more implementation details.
5.1.2. EVALUATIVE METRICS
Feasibility Errors: At each evaluation timing, we use the average errors of [E[Dz] against the ground truths as a proxy to understand the convergence of the evaluators' estimated feasibility of targets.
Delusional Behavior Frequencies: We monitor the fre- quency of hallucinated target (made of G.1 and G.2) be- ing chosen by the agents (as the next sub-goal for Skipper, as a part of the sub-goal chain for LEAP), i.e., delusional planning behaviors. Due to the page limit, some related discussions and results are presented in Appendix.
All presented mean curves and the 95%-confidence interval bars are established over 20 independent seed runs.
5.1. Decision-Time Planning (Exp. 1/8- 4//))
For decision-time TAP agents, we are interested in under- standing how rejecting hallucinated targets influence their abilities to generalize their learned skills after learning from limited number of training tasks. This also means, the evaluator is expected to learn to generalize its identifica- tion of infeasible targets in novel situations, by identifying the patterns of the infeasible targets.
AAs shown in Tab. 2, our adaptation for Skipper can he ex- tended to methods utilizing arbitrarily distant targets, including background TAP methods such as GSP (Lo et al., 2024)
LEAP can be extended to planning methods proposing sub-goal sequences, such as PlaNet (Hafner et al., 2019)
'As shown in Tab. 2, our implementation

---

Rejecting Hallucinated State Targets during Planning
OOD Generalization Performance: We analyze the
5.1.4. A GLIMPSE ON LEAP ON RDS (EXP. 4/s)
changes in agents' 001) generalization performance. The evaluation tasks (targeting systematic generalization) are sampled from a gradient of OOD difficulties - 0.25, 0.35, 0.45 and 0.55. Because of the lack of space, we present the "aggregated" OOD performance, such as in Fig. 4 d), by sampling 20 task instances from each of the OOD difficul- ties, and combine the performance across all 80 episodes, which have mean difficulty matching the training tasks. To maximize the evaluation difficulty, the initial state is fixed in each evaluation task instance: the agents are not only
Having covered G.2-focused SSM with Skipper, we turn to another decision-time TAP agent compatible with arbitrary horizon targets
environment focused on G.I difficul- ties (more details in Sec. A.I, Appendix). As shown in Fig. 5, LEAP+ (LEAP assisted by the proposed evaluator), achieves significant fewer delusional plans and better OOD evaluation performance.
aggrogattdd OOD success rale
.8
spawned to be at the furthest edge to the monster, but also
G. schgnals plarmed
in semantic class (0, 0), i.e., with neither the sword nor the shield in hand.
LEAP
LEAP+
5.1.3. A GLIMPSE ON SKIPPER ON SSM (ExP. 1/8)
LEAP LEAP+
01
We compare the original form of Skipper with Skipper+, variant that is assisted by the proposed evaluator. Details of the variants are shown in the captions of Fig. 4.
0.5
1.0
0.0
0.5
1.0
1.5
training steps
14
training slops
(a) G. Subyoal Ratio
(b) (b)gggggatedd OOD Perfoomance
Figure 5. LEAP RDS: compared to the baseline LEAP, LEAP+
Hallucination: we first investigate generator's rates of hal- lucinations. As shown in Fig. (Appendix), the generator produces targets that correspond to G.1 and G.2 with the rate of around 3% and 5%, respectively. We leave the details of the generators there for the readers.
selects significantly fewer G. subgoals. a) Evolving ratio of G.1 subgoals among the planned subgoal chains; b): Aggregated OOD evaluation performance, same as for Fig.4 d).
valoe-ooror to GT (ka)
Feasibility Errors: Skipper relies on a built-in cumula- tive discount estimator whose estimations can be converted to feasibility estimates that our evaluator seeks to learn. Thus, we can examine the errors of the feasibility estimates corrected by the injected evaluator to understand how the proposed evaluator could reduce feasibility delusions of Subitrary-horizon TAP methods. From Fig. 4 b) and e), we can see that feasibility estimates corrected by our evalu- ator have significantly less errors compared to the origi- nal, towards both G. and G.2 targets. As a perk for Skip- per-'s utilization of PERTASK for E.2 delusions (included in (E+P+G)), its positive effect on far-away G.O targets are also shown in Fig. 4 a). It can be seen that the evaluator is generally helpful for Skipper to understand the fcasibility of all G.O, G.1 and G.2 targets.
surocesss rafte
Dyna Dyna+
12
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
training steps
training steps
(a) Comergennce to Optimal Value
(b) Traininn Performance
Figure 6. Dyna SSM: compared to the baseline Dyna, Dyna+ rejects the updates toward 1-infcasible generated states flagged by the evaluator, powered by (E+P+G). a) Evolving mean Lı distances between estimated Q & optimal values; b): task perfor- mance on the 50 training tasks & rate of Dyna+ rejecting updates.
5.1.5. SUMMARY OF EXP. l/s -//s
The purpose of identi- fying infeasible targets is to reduce delusional plans that involve them. We provide detailed results on this in Fig. 10 (Appendix). where we observed significant reduction in delusional plans involving both G. and G.2 targets.
Frequency of Delusional Plans:
For Exp. 1/s 4/s, with the proposed evaluator, we saw reduction in feasibility delusions and in delusional behav- iors, which led to better OOD generalization performance, against challenges of G. & G.2. These 4 sets of experi- ments align in terms of the effectiveness of our approach.
Generalization:
Comparing Skipper and Skipper-t, we can deduce from Fig. 4 that generally, lower E.2 errors (c) lead to less frequent delusional behaviors (shown in Fig. 10, Ap- pendix), which in turn improves the OOD performance in d). This indicates that rejecting infeasible targets can help decision-time TAP agents in systematic OOD generaliza- tion.
5.2. Background Planning: A Glimpse on Exp. 3/8 & 6/s
These experiments focus on rollout-based background TAP agent- the classical 1-step Dyna (Sutton, 1991), which uses its learned transition model to generate next states from existing states to construct simulated transitions that are used to update the value estimator, i.e. a "Dyna update". Jafferjee et al. (2020) demonstrated the benefit when the
8

---

Rejecting Hallucinated State Targets during Planning
delusional Dyna updates bootstrapped on hallucinated tar- gets rejected with oracle. We replace the oracle using our learned evaluator. With the same training setup, in Fig. 6, we present the empirical results of how target rejection can significantly improve the performance of Dyna on SSM. The rejection rate stabilizes as both the generator and the evalua- tor learns. These observations are consistent with Exp. 9/8, presented in Sec. F (Appendix).* 4
iors, such as by constraining to certain probabilistic models (Deisenroth & Rasmussen, 2011; Chua al., 2018), or training target evaluator separately on collected dataset.
Delusions in value estimates of hallucinated states are hy- pothesized to plague background planning (Jafferjee et al., 2020). Lo et al. (2024) introduced a remporall--abstrae background TAP method to limit temporal-differeet up- dates to only few trustworthy targets. Langosco al. (2022) classified goal mis-generalization, a delusional be- havior describing when an agent competently pursues problematic target. Talvitie (2017) tried to trains the model to correct itself when error is produced. Zhao et al. (2024) gave first examples of delusional behaviors caused by hallu- cinated targets decision-ting time TAP agents.
5.3. Non--ingleton Targets: A Glimpse on Exp. 7/8 & 8/s
We validate the evaluator's empirical convergence to ground truths when facing non-singleton targets. We present the results on RDS (Exp. 8/8) in Fig. and leave the SSM coun- terpart Fig. 19 in Appendix (for Exp. 7/8). The results show convergence as expected and more details are presented in Sec. G in Appendix.
Hindsight Relabeling is highlighted for its improved sam- ple efficiency towards G.0 targets, around which most follow-up works revolved as well (Andrychowicz et al., 2017; Dai et al., 2021). However, sample efficiency is not the only concern in TAP agents, as delusions toward gener- ated targets can cause delusional behaviors leading to other failure modes. Shams & Fevens (2022) studied the sample efficiency of atomic strategies, without looking into their failure modes. Deshpande et al. (2018) detailed experimen- tal techniques in sparse reward settings using FUTURE. In (Yang et al., 2021), a mixture strategy similar to GENERATE improved estimation of feasible targets, though its impact on hallucinated targets was not investigated. Note that the performance of existing HER-trained agents often limited by their reliance on FUTURE or EPISODE, whose delusions this paper intends to address.
episode
(L+P+C)
error D. (-))
orror
5
0.0
0.5
1.0
1.5
0.0
0.5
1.5
training steps
training steps
(a) Evolution of E.O Errors
(b) Evolution of E.I Errors
Figure 7. Feasibility Non-Singleton Targets on RDS: a) Evo- lution of E.0 error, b) Evolution of E.1 error; The training data is acquired with random walk.
6. Related Works
7. Conclusion, Limitations & Future Work
TAP: Most rollout-based TAP methods are oblivious to model hallucinations and utilize all generated targets with- out question. These include fixed-step background methods such as Sutton (1991); Lukasz Kaiser et al. (2020); Yun et al. (2024); Lee et al. (2024) and decision-time methods based on tree-search, such as Schritwieesr al. (2019); Hafner al. (2019); Zhao al. (2021); Zhang et al. (2024a); TAP methods compatible with arbitrarily distant targets (T oc) often struggle to produce non-delusional feasibility-like es- timates for hallucinated targets. Thus, they cannot properly reject infeasible targets despite having their own "evalua- tors". These include background methods such as Lo et al. (2024) and decision-tima methods for path planning (Nasiri- any et al., 2019; Yu et al., 2024; Duan et al., 2024), OOD generalization (Zhao al., 2024), and task decomposition (Zadem et al., 2024). Previously, there were method-specific approaches proposed against delusional planning behav-
We characterized how generator hallucinations can cause trouble for TAP agents. Then, we proposed to evaluate the feasibility of targets s.t. the infeasible hallucinations can be properly rejected during planning. We proposed a combination of learning rules, architectures and data aug- mentation strategies that leads to robust and accurate output when the proposed evaluator is applied. In experiments, we showed that the evaluator can significantly address the harm of hallucinated targets in various kinds of planning agents.
The targets we investigated in this paper are in nature, com- posed of "states". Meanwhile, some other planning agents propose "targets" that are in nature "actions", which do not directly correspond to reaching states, instead, for example, to maximize certain signals without providing h. For future work, we will investigate those agents to understand how they are impacted by hallucinations.
*The implementation here can be extended to fixed-horizon rollout agents. In the Sec. J (Appendix), we provide details how we applied our Dyna solution to DreamerV2 (Hafner et al., 2021).
9

---

Rejecting Hallucinated State Targets during Planning
Reproducibility Statement
The results presented in the experiments are fully- reproducible with the souree code published at https: //github.com/m. la- iia-iiidddddsss.ss
Acknowledgments
Mingde "Harry" is grateful for the financial support of the FRQ (Fonds recherche du Québec) and the collabora- tive efforts during his internship at RBC Borealis (formerly Borealis AI), Montréal with Tristan. We appreciate the com- putational resources allocated to from Mila (Québec Al Institute) and McGill University.
10

---

Rejecting Hallucinated State Targets during Planning
References
2018. URL U.LAttess//ppooeeddngssrrrrrpppp paper filess//aaerr/221///te/ 3ce5efBB971ssboa5116607778333333 Paper.pdf.
Aithal, S. K., Maini, P., Lipton, Z. C.. and Kolter, J. Z. Understanding hallucinations in diffusion models through mode interpolation. In Confer- ence on Neural Information Processing Systems (NeurIPS), volume 37, pp. 134614-134644, 2024. URL httpss//ppooeeding.eerri.p... paper.iieee/paperr/0//////// 22996900122211114446666222111113 Paper-Conference.ppe
Dai, T., Liu, H., Arulkumaran, K., Ren, G., and Bharath, A. A. Diversity-base -based trajectory and goal selection with hindsight experience replay. In PRICAI 2021: Trends in Artificial Intelligence: 18th Pacific Rim In- ternational Conference on Artificial Intelligence, PRI- CAI 2021, Hanoi, Vietnan, November 8-12, 2021, Pro- ceedings, Part 18, pp. 32-45. Springer, 2021, URL https: http://axxvv.orrrg /tsp//b..///////1//////
Alver, S. and Precup. D. look value-based decision-timis background planning methods across different settings. In Seventeenth European Workshop Reinforcement Learning, 2024. URL https: http//ppeeeeee.eee/ forum? d d
Davehev, T., Sushkov, O. O., Regli, J.-B., Schaal, S., Ay- tar, Y., Wulfmeier, M., and Scholz, J. Wish you were here: Hindsight goal selection for long-horizon dex- terous manipulation. In International Conference on Learning Representations (ICLR), 2022. URL https: http://wwwww.eeeeo.eeeeeee.aaaee
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Picter Abbeel, O., and Zaremba, W. Hindsight expe- rience replay. In Conference on Neural Information Processing Systems (NeurIPS), volume 30, 2017. URL. http:://proceedingg.eeeiii paper/2017/filee ASSEdbdlalaaaff0oaaaaaaaa3993333 Paper.pdf.
Deisenroth, M. and Rasmussen, C. PILCO: model- based and data-efficient approach to policy search. In International Conference Machine Learning (ICML), ICML'11, pp. 465 472. Omnipress, 2011. URL https: //ool..rgg11.117777788333333838
Bellemare. M. G., Dabney, W. and Munos, R. A distributional perspective on reinforcement learning. Machine Learning
Deshpande, A., Sarma, S., Jha, A., and Ravindran, B. Im- provements on hindsight learning, 2018. URL https: / //arxiv.org/abll /abs 1809.06719.
In International Conference (ICML), volume 70, pp. 449-458. PMLR, 06-11 Aug 2017. URL https: v70/bel 1emareeleeelllleeemmm 7a.htm1.
Duan, Y. Mao, W., and Zhu, H. Learning world models for unconstrained goal navigation. In Con-
ference on Neural Information Processing Systems (NeurIPS), volume 37, pp. 59236-59263 2024. URL http:://proceeddngs.eerrrrpp paper_Elles/ppaer//233///// (cca3388lee660079988224446600011 Paper Conf--ooereeeece.d..
Bengio, Y., Hinton, G., Yao, A., Song. D., Abbeel, P. Darrell, T., Harari, Y. N., Zhang, Y.Q.., Xue, L., Shalev-Shwartz, S., et al. Managing extreme Al risks amid rapid progress. Science, 384(6698): 842-845, 2024. URL httt:::/doi.org/0011122 science.adn0117.
Erraqabi, A., Machado., M. C., Zhao, M., Sukhbaatar, S., Lazaric, A., Ludovic, D., and Bengio, Tem- poral abstractions dbstractions-augmented temporally contrastive learning: An alternative the laplacian in rl. In Conference Uncertainty in Artificial Intelli- gence (UAI), volume 180 of Proceedings of Machine Learning Research, pp. 641-651, 01-05 Aug 2022. URL wtwptssssppppeeeeeeeeeepee wwweeediing.m........... ptg. e erraqabi22a.htm.
Chevalier-Boisvert, M., Dai, B., Towers, M., Perez-Vicente R., Willems, L., Lahlou, S., Pal, S., Castro, S., and Terry, Minigrid & miniworld: Modular customizable reinforcement learning environments for goal-oriented tasks. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, pp. 73383-73394, 2023.
URL https: http:::/pproceeiings..ee.ss paper.ftlees/ppeer/20////LLL w 6B816.18846636222222225444233399 Paper
Eysenbach, B., Salakhutdinov, R., and Levine, S. C- learning: Learning to achieve goals via recursive clas- sification. In International Conference on Learn- ing Representations (ICLR), 2021. URL https:// openreeieewveet/foouum/e.eessss
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful trials using probabilistic dynamics models. In Conference on Neural Information Processing Systems (NeurIPS), volume 31,

---

Rejecting Hallucinated State Targets during Planning
Frank, S. L.. Haselager, W. F, and Rooij, Con- nectionist semantic systematicityy. Cognition, 110(3): 358-379, 2009. URL
Langosco, L. L. D., Koch, J., Sharkey, L. D., Pfau, J., and Krueger, D. Goal misgeneraliza- tion in deep reinforcement learning. In Interna- tional Conference on Machine Learning (ICML), vol- ume 162, pp. 12004-12019, PMLR, 17-23 Jul 2022. URL htttpp://prooeddiggss...ee.ess langosco222.htmm.
J.cognitton.2008.11.1
Hafner, D., Lillicrap. T., Fischer, Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning la- tent dynamics for planning from pixels. In Interna- tional Conference
Machine Learning (ICML), vol- ume 97, pp. 2555-2565. PMLR, 09-15 Jun 2019. URL https: https://proceeddie fnerl 9a.html.
Lee, Yun, S., Yun, T., and Park, J. Gta: Generative trajec- tory augmentation with guidance for offline reinforce- ment learning. In Conference Neural Information Pro- cessing Systems (NeurIPS), volume 37, pp. 56766-56801, 2024. URL paper paper_files/paperr/22//111
Hafner, D., Lillicrap, T. P., Norouzi, M., and Ba, J. Mastering atari with discrete world models. In In- temational Conference on Learning Representations (ICLR), 2021. URL https: https////peeeeee.eeeee.. Forum?id=0oabwyal
67ea314ddddd//lbbfffa64666330 Paper.pdf.
Lo, C., Roice, K., Panahi, P. M., Jordan, S. M., White, A., Mihucz, G., Aminmansour, F., and White, M. Goal- space planning with subgoal models. Journal of Machine Learning Research, 25 2024, URL https: / //jmlr.org hwt.gg/apps////aae 20tp:////222222000.00t0t 0040.html.
Hafner, D.. Lee, K.-H., Fischer, I., and Abbeel, P. Deep hierarchical planning from pixels. In Con- ference on Neural Information Processing Systems (NeurIPS), 2022. URL hRtht:/: htt.attss//peeeeeeee..cro forum?ides (rrum??d=w226kkky99.
Moro, L., Likmeta, A., Prati, E., and Restelli, M. Goal- directed planning via hindsight experience replay. In International Conference Learning Representations (ICLR), 2022. URL https //pp///eeeeeeeet.......////ee/.e forum serum?id==on Nee.edddddeeeeeeeeeeeeee
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse control tasks through world models. Nature, 640(8059):647-633. 04 2025. URL https://wwwnaturrrt 111586-025-08744
Nasiriany, S., Pong. V., Lin, S., and Levine, S. Planning with goal-conditioned policies. In Conference on Neural Infor- mation Processing Systems (NeurIPS), volume 32, 2019. URL https::/proceeeeee paper/2019/file httess//prooeeddnggserrriso.o
Howard, R. Dynamic Programming and Markov Processes. John Wiley, 1960.
URL https://archive /etrg// Synamicprrogrami000m
Paper.pdf.
Jafferjee, T., Imani, E., Talvitie, E., White, M., and Bowling, M. Hallucinating value: A pitfall of dyna-style planning with imperfect enviromment models, 2020. URL https:
Rubinstein, R. Y. Optimization of computer simula- tion models with rare events. European Journal of Operational Research, 99(1):89-112, 1997. URL
httess//www.ccienncdiieee..... wwtpc/////ccccccc/ccccccccccccet article/p11/S00722177600
Jesson, A., Beltran- Velez, N., Chu, Q. Karlekar, S., Kossen, J., Gal, Y., Cunningham, P., and Blei, D. Estimating the hallucination rate of generative ai. In Conference Neural Information Processing Systems (NeurIPS), volume 37, pp. 31154-31201, 2024. URL https://prooeedigs..noooo..s paper filess/apprr22224/1111 (P9155f00e8-3333066aaa22222222 Paper.pdf.
Russell, L., Hu, A., Bertoni, L., Fedoseev, G., Shotton, J., Arani, E., and Corrado, G. GAIA-2: A control- lable multi-view generative world model for autonomous driving, 2025. URL https: http:////xxxvv///eeeeaa aba/ 2503.20523.
Schritwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T. Lillicrap, T. P. and Silver, D. Mastering atari, go. chess and shogi by planning with a learned model. Nature, 588, 2019. URL https / wwtp//wwwt.....omm. 4.pdf.
Kiran, C. and Chaudbury, S. Understanding delu- sions. Industrial psychiatry journal, 18(1):3-18,
2009. URL https: http:///ddd.oag/...10//////
6748.57851.
articlees/44586--
12

---

Rejecting Hallucinated State Targets during Planning
Shams, A. and Fevens, T. Addressing different goal selec- tion strategies in hindsight experience replay with actor- critic methods for robotic hand manipulation. In 2022 2nd International Conference on Robotics, Automation and Artificial Intelligence (RAAI), pp. 19-73, 2022. URL. https: /tt.aiaww...///.///////.../..ota http://////.b///......0000. 2205.05055.
98904ab124b2a246388888 Paper.pdf.
Zadem, M., Mover, S., and Nguyen, S. M. Reconciling spatial and temporal abstractions for goal representation. In International Conference Learning Representations (ICLR), 2024. URL https: /pwpneeveee.eeettt...e//// forum Beeum?id=odY380 id
Sutton, R. S. Dyna, an integrated architecture for learning. planning, and reacting. SIGART Bulletin, 2(4)::60-163, 1991. URL https:/ pttttps//d...a........ /ttp.//ccc...g//oo/////oo/////// /10.1144// 122344.122377
Zhang, D., Lv, B., Zhang, H., Yang. F., Zhao, J., Yu, H., Huang, C., Zhou, H., Ye, C., and Jiang, C. Focus on what matters: Separated models for visual-based rl generalization. ence on Neural Information Processing Systems (NeurIPS), volume 37, pp. 116960-116986, 2024a. URL attos://prooeeddnggeeee......... paper files//ppeer/2024/ffee/// d222288819ha334cd0866666888 Paper.pdf.
Talvitie, E. Model regularization for stable sample rollouts. In Conference on Uncertainty in Artificial Intelligence (UAI), UAI'14, 14, pp. 780-789, 2014. URL https:// arxiv.org/ abs/ 406.0834.
In Confer-
Talvitie, E. Self-correcting models for model-based reinforcement learning. In Proceedings of the First AAAI Conference on Artificial Intelli- gence, volume 31 of AAAI'17 AAAI Press, Feb. 2017. URL
Zhang, X., Huang, H., Zhang, D., Zhuang, S., Han, S., Lai, P., and Liu, H. Generalization vs. hallucination, 2024b. URL https::/p//. http:///xxxvx/r/////// abs/2411.02999.
www.aaa...rriiii..p...... index.php/ AMAT/artiilee/veew///oo
Xing, Y., Li, Y., Laptev, I., and Lu, S. Mitigating object hallucination via concentric causal attention. In Conference on Neural Information Processing Systems (NeurIPS), volume 37, pp. 92012-92035, 2024. URL httpss:/pproccedding..eees paper paper.filees/aaprr///2/////// 3T6ccaBBef2226223872222211ff Paper-Conference..u
Zhao, M., Luan, S., Porada, I,, Chang, X.-W., and Pre- cup, D. Meta-learning state-based eligibility traces for more sample-efficient policy evaluation. In Interna- tional Conference on Aulonomous Agents and MultiAgent Systems (AAMAS), AAMAS '20, pp. 1647-1655, Rich- land, SC, 2020, URL https: : wttp:::wwwwwwwwwa...oooooo Proceedings aama 2020 https//////ppppppp/////pp.
Zhao, M., Liu, Z., Luan, S., Zhang, S., Precup, D., and Bengio, Y. A consciousness-inpires planning agent for model-based reinforcement learning. In Conference Neural Information Processing Systems (NeurIPS), volume 34, pp. 1569-1581, 57 More- house Lane, Red Hook, NY, United States, 2021. URL
Xu, Z., Jain, S., and Kankanhalli, M. Hallucination is inevitable: An innate limitation of large language abs/
models, 2025. URL 2401.11817.
Yang, R., Fang, M., Han, L., Du, Y., Luo, F., and Li, X. MHER: Model-based hindsight experience replay, 2021. URL httph:ttp/// /ttp///axxvvvrr////////////../.t abss2207.00300
paper_files puper__Lles//pper//2D//iii 0c215571922660006666666665585555 Paper.pdf.
Yu, X., Zhang, S., Song, X., Qin, X., and Jiang, S. Trajectory diffusion for objectgoal navigation. Conference on Neural Information Processing Systems (NeurIPS), volume 37, pp. 110388-110411 2024. URL
Zhao, M., Alver, S., van Seijen, H., Laroche, R., Pre- cup, D., and Bengio, Consciousness-inpprred spatio- temporal abstractions for better generalization in rein- forcement learning. In International Conference on Learning Representattinn (ICLR), 2024. URL https: //tpeennee.eeeeetttt/orrrmoooeeo
paper files http:://procedingg.enrii..s 0736619510ff33t33661110022881111 Paper.pdf.
Lukasz Kaiser, Babaeizadeh, M., Milos, P., Osiáski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., Mohiuddin, A., Sepassi, R., Tucker, G., and Michalewski, H. Model based reinforce- ment learning for atari. In International Conference on Learning Representations (ICLR), 2020. URL https: // wwwpereeiww.eee.eeeeoeeee d=51 JHLDB.
Yun, T., Yun, S., Lee, J., and Park, J. Guided trajectory generation with diffusion models for offline model-based optimization. In Conference on Neural Information Pro- cessing Systems (NeurIPS), volume 37, pp. 83847-83876, 2024. URL htt.///tpps//pooceeds...eeerr paper enterffless files/ wwweeeflllsspape/22222222
13

---

Rejecting Hallucinated State Targets during Planning
Appendix: Part I -Reeerenced Tables & Figures
Table 2. Discussed Methods, Properties How to use the Feasibility Evaluator
Implementation Details & Challenges
Method
TAP Category
Delusional Planning Behaviors
How Our Solution Helps
The imagined transitions could contain ballucinated (next) obscrvations / states, whose delusional value estimates could destabilize bootstrapping based
Implemented (for 1-step Dynaj; output histogram of the evaluator significant density the bin corresponding
Dyna (Sutton, 1991)
Fixed-Horizon Background Planning
Cancel updates involving rejected states (not evaluated to be reuchable within timestep).
then accept generation, else, reject
TD learning
Implemented (insufticient compute for results, Sec. J): Use the deterministic state as the state representation feed evaluator (also for imagined target states). Eistablish Mahalanobis distance the representations and use discount, reward and value predictions force hehavioral realism. Truncate X-returns until infeusible imagined turget states.
Do not rejected imagined states participate construction of update targets for actor-critic system (See Soc.)..
Dreamer (Hafner et al., 2025)
Fixed-Horizun Background Planning
imagined trajectories could contain infeasible, hallucinated states
Director (Hafner
Fixed-Horizon Decision- Time
The internally sampled goals may be unreachahle
Reject unreachable goals and re se-sample reachahle ones
Similar to our implementation for Dreamer.
2022)
Planning (mainly)
MuZero
Fixed-Horizos
The predicted states in the tree search could be unreachable
Reject hallucinated state generations, regenerate node in tree search necessary
(Schrit-
Similar our implementation for 1-step Dyna
Decision- Time
twieser
Planning
hallucinations
al., 2019)
SimPLe (tukaszz Kaiser et al., 2020)
Fixed-Horizom Background Planning
The predicted next observation could be an unreachable ballucination
Reject leurning against the delusional estimates (potential)
Similar our implementation for 1-step Dyna
Use an evaluator to learn that the expected cumulative discount is 0 when aiming to reach the hallucinuted subgoals. This disconnects the hallucinated subgoals from the current state in the planning
Implemented: diversify the source-turget puirs with GENERATE and PERTASK mixtures, discrete and trivial comparison.
Skipper (Zhao et al., 2024)
Arbitrary-Hrizzon Decision- Planning
Hallucinated subgoals could lead to decision-time plunning committing to them, leading to unsafe behaviors
Use output histogram the add-on
Arbitary-Horizon Background Planning
Hallucinated subgoals could lead value estimation destabilization, like in Dyna.
(Lo al., 2024)
evaluator to correct the delusions by
Similar our implementation Skipper
GSP's own estimators. Use "support swap" technique. an evaluator learn expected cumulative distance is infinite when aiming reach
Hallucinated subgoals could help fake a sequence of subgoals that is too good to be true and committed to during planning
LEAP (Nasiriany 2019)
Arbitray-Horizzn Decision-Tine Time Planning
Implemented: pay attention to the representation space of the sub-goals.
hallucinated subgoals. This makes that subgoul sequences contuining hallucinated subgouls will not be favored
Hallucinated subgoals could help fake sequence of subgouls that is
PlaNet (Hafner
Arbitray-HHorieer Decision- Time
Reject the delusional subgoals and therefore reject delusional
Same as our implemennation for LEAP (both uses CEM planning (Rubinstein, 1997))
too good to be true and committed during planning
al., 2019)
Planning
subgoal sequences
Similar colors are used to denote similar implementations for the solution proposed in this work.
14

---

Rejecting Hallucinated State Targets during Planning
gosnder
future
episode
generate
pertask
Figure 8. Representative Atom Hindsight Relabeling Strategies & Newly Proposed Ones: The first two strategies, FUTURE and EPISODE, are widely used as they create relabeled transitions that help evaluators efficiently handle G.O targets during planning. Iwo, GENERATE and PERTASK, are effective at addressing delusions, making them useful in specific scenarios. Atomic hindsight strategies from the first group can serve as buckhones for mixture strategies, complemented by the second group to address delusions.
Strategies
Advantages
Disadvantages
Gist
When used exclusively to train evaluator, 1) cannot handle F.2 and 2) prone to F.O cannot learn well from short trajectories; Can cause G.2 targets when used to train generators
Creates training data with source-target pairs sampled from the same episodes
Efficient for evaluator leam Mlose-proxmmity relationships
EPISODE
Creates training data with temporally ordered
Can be used learn conditional generator temporal abstractions
In addition the shortcomings of epISODE (those for evaluators only), this additionally causes E.0 when used the exclusive strategy for evaluator training
source-target pairs from the episodes
Addresses E.1 with data diversity (also E.2 when generator produces G.2)
Augments training data to
Relies the generator with additional computational costs; Potentially low efficiency in reducing E.O.
GENERATE
include candidate targets proposed decision time
Addresses evaluator delusions (E.2 & E.0 for long-distance pairs)
Augments training data to include targets that were experienced
low efficiency learning close-proximity source-target relationships
PERTASK
Table Hindsight Relabeling Strategies: EPISODE and FUTURE are widely used as they increase sample efficiency towards G.O states significantly; GENERATE and PERTASK, proposed in this paper, should be applied against delusions in relevant scenarios.
15

---

Rejecting Hallucinated State Targets during Planning
Appendix: Part II Exxpeimeetss
A. More Details on Decision-Time TAP Experiments (Exp. 1/s 4/s)
A.1. RandDistShift
The second environment employed is RandD Shift abbreviated as RDS. RDS was originally proposed in Zhao et al. (2021) as a variant of the counterparts in the MiniGrid Baby-Al platform CChevalirrBBoisver et al., 2023), and then later used as the experimental backbone Zhao et al. (2024). SSM was inspired by RDS. We can view RDS as a sub-task of SSM, where everything happens semantic class (1, 1), Le., agents always spawn with the sword and the shield in hand, thus can acquire the terminal sparse reward by simply navigating to the goal. RDS instances thus have smaller state spaces than its SSM counterparts. The most important difference, in the views of this work, is that RDS removed the challenges introduced by temporary infeasibility. This means that G.2 and F.2 are no longer relevant, shifting the dominance towards G. 1. E. combination. Using RDS not only showcase the performance of the proposed strategies on a controlled environment with G.I + E. dominance, contrasting the G.2 + E.2 dominance of SSM, it also can be used to validate the performance of our adapted agents, on an environment where previous benchmarks exist.
A.2. Generator Hallucinations
We use hindsight-relabeled transitions to train the generators in the two methods (Skipper & LEAP), to demonstrate how different ways of training the generator could affect the rates of hallucinations. G.2 can appear more frequently the generator trained to imagine more diverse kinds of targets than needed. For example, conditional target generator which learns from EPISODE will be more likely produce G.2 targets (compared FUTURE). This was why we mostly used FUTURE train the generators in the related experiments.
For the HER-trained generators, Fig. 9 a), shows that different training targets for the generator could lead to different degrees of hallucinations, in terms of G.1 and G.2. Importantly, Fig. 9 b) indicates that, FUTURE generates G.2 targets significantly less frequently than EPISODE and PERTASK, as the other two wasted training budget on G.2 targets, especially PERTASK that brings in more problematic training samples from long distances. all other experiments, we only compare variants with FUTURE for the generator training.
The generator is consistently used for both Skipper and LEAP in Exp. 1/s Exp. 1/8.
future
7.5
future episode
8
episode
40
Maman
perent
G.2 generation %
G.1 generation %
perent
G.1 generation
6
30
future episode
5
4
20
pereno
2.5
2
0.0
0.5
1.0
1.5 106
0.0
0.5
1.0
1.5 166
0.0
1.0
training steps
training steps
training steps
les
G.I Condidate Ratio SSM
(b) G.2 Condidate Ratio in sSy
(c) G./ Candidate Ratio in RDS
Figure Hallucination Frequencies: a) Evolving ratio G.1 "states" among candidates each target selection, throughout training; Subfigure b) is the E.2-counterpart of a) on SSM; Subfigure c) is the RDS-counterpart of a).
16

---

Rejecting Hallucinated State Targets during Planning
B. Skipper on SSM (Exp. 1/8, continued)
B.1. Additional Results of Skipper+ with other Relabeling Strategies & Frequencies of Delusional Plans
In the main manuscript, we focused on a particular implementation of evaluator which utilizes (E+P+G) for training data. Since we have the full degrees of freedom deciding the mixture ratios of the involved relabeling strategies, i.e. EPISODE, GENERATE & PERTASK, we will provide more results here that encompass more relabeling strategies. These results could not only provide the readers with more understanding of the empirical characteristics of the relabeling strategies but also can serve an ablation test for the two alternative relabeling strategies, i.e., CENERATE & PERTASK. The variant relabeling strategies are as follows:
(E+G)- a mixture against E.I. EPISODE with 50% chance using GENERATE JIT. resulting in a half-half mixture of EPISODE & GENERATE
(E+P)- against E.2. Half EPISODE & half PERTASK
We can see that (E+P+G) is the middle ground between (E+G) and (E+P), with comprehensive coverage for both G.O, G. and G.2 cases. This is why we have chosen (E+P+G) as the default for our evaluator, since we do not wish to assume access to the state space structures of the environments. Note that for your convenience, we have used consistent colors for each variant throughout this work.
We expand the results in Fig. 4 to Fig. 10, to not only include new sub-figures on the frequencies of delusional planning behaviors but also the variants.
From Fig. 10 a) and d), we can see that the more relabeling is invested into GENERATE, the less E.1 errors the agents would have and the less frequent G.1 targets trigger delusional plans. The same can be said for G.2 & PERTASK in Fig. 10 b) and e). Because of the state space structure, on SSM, it is quite expected that (E+P) is the most efficient in terms of increasing OOD evaluation performance (Fig. 10 f)) due to the dominating challenge of G.2 targets.
B.2. Breakdown of Task Performance
In Fig. 11, we present the evolution of Skipper variants' performance on the training tasks as well as the OOD evaluation tasks throughout the training process. Note that Fig. 10 f) (and by extension Fig. 4 d)) is an aggregation of all 4 sources of OOD performance Fig. 11 b-e).
From the performance advantages of the hybrid variants (in both training and evaluation tasks), we can see that learning to address delusions during training brings better understanding for novel situations posed in OOD tasks.
C. LEAP on SSM (Exp. ?/s)
This set of experiments seeks to demonstrate that the proposed fcasibility evaluator can help reduce delusional planning behaviors in other decision-time TAP agents while facing challenges of G.2 targets. For this purpose, we study LEAP performance on SSM, and its variant LEAP+ with our evaluator injected.
Compared to Skipper, LEAP utilizes the generated targets in very different way, its decision-time planning process constructs singular sequence of subgoals leading to the task goal. Due to a lack of backup subgoals, even if one among them is problematic, the whole resulting plan would be delusional, making LEAP much more prone to failures compared to Skipper, where candidate targets can still be reused if deviation from the original plan occurred.
SSM has a relatively large state space that requires more intermediate subgoals for LEAP's plans. However, an increment of the number of subgoals also dramatically increases the frequencies of delusional plans, damaging the agents' performance, Because of this, our experimental results of LEAP on SSM with size 12 x 12 became difficult analyze because of the rampant failures. We chose instead present the results SSM with size x 8 here.
Additionally, consistent with Exp. l/s, for more comprehensive understanding of the relabeling strategies' impact on the learned evaluator, we include results beyond (E+P+G), which is the default used in the main paper.
For LEAP, we use some different metrics to analyze the effectiveness of the proposed strategies in addressing delusions. This is because, if LEAP's evaluator successfully addressed delusions and learned not to favor the problematic targets (G.1
17

---

Rejecting Hallucinated State Targets during Planning
Skipper Skippert(E+ PE+P) Skipper+(E+o
E.1 crrorr (a(.))
E.2 efrorr Da(..))
E.0 error (.(...)
2
1
3
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
2
8
10
12
14
training steps
led
training steps
166
source-target distances
(a) Evolation of E.l Errors
(b) Evolation of E.2 Errors
(c) Final E.U Errors by Distance
10-1
aggregated OOD success rate
.4
G. targets chosen %
G.2 targets chosen %
10-3
10-2
.2
10
10-4
10"4
Skipper+ (E+P+))
0.0
0.5
1.0
1.5
1.0
1.5 les
0.0
1.5 Teb
training steps
les
training steps
training steps
(d) Evolution of E Behavior Ratio
(e) Evolution of E.2 Behavior Ratio
(1) Aggregated 0OD Performance
Figure 10. Skipper and More Variants of Skipper+ SSM: In addition to subfigures that already exist in Fig. 4, i.e. a). b), c), & f), provide additional subfigures and e), demonstrate the changes of frequencies delusional behaviors throughout training, for G.1 and G.2 composed targets, respectively. The curves denote the frequencies of G.1 and G.2 targets becoming the imminent subgoals that Skipper seeks to achieve next. Each figure is augmented with results additional evaluator variants with different relabeling strategies.
Skipper Skipper((E+P) Skipper+ (F++) Skipper (E+++++E+++))) +P+G)
success rale
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
training
steps
training
training steps
(a) training. = 0.4
(b) OOD oval.. = 0.25
(c) 0OD eval., 0.35
(d) OOD eval., 5 0.45
(c) 00D eval., 5 = 0.55
Figure 11. Separated Evolution of OOD Performance of Skipper Variants on SSM
18

---

Rejecting Hallucinated State Targets during Planning
LEAP
PEAP4(E+P) LEAPYENG) LEAP+(E+P+))
C1 (11nbgool sulgoolllpaannee planned
debeimaal plan
15
1.5
aggr
0.0
0.5
2.0
1.5
0.0
0.5
1.0
1.5
0.5
2.0
1.5
10 12 entrggt dietanoe
0.0
0.5
1.0
1.5
Itaining
trainingatapp
training
traiting slops
(a) G.I Rotio Planned
(b) G.2 Ratio Pianned
(c) Defusional Plan Rotio
(d) E.0 Errors
(c) Aggregated OOD Perf.
Figure 12. LEAP on SSM: a) Ratio of G.1 subgoals among the planned sequences: b) Ratio of G.2 subgoals in the planned sequences; c) Ratio of evolved sequences containing least one G.1 or G.2 target, d) The final estimation accuracies towards G.O targets after training completed, across a spectrum of ground truth distances. In this figure, both distances (estimation and ground truth) are conditioned on the final version of the evolving policies; Each data point represents OOD evaluation performance aggregated over 4 20 newly generated tasks, with mean difficulty matching the training tasks.
and G.2), then they will not be selected the evolved elitist sequence of subgoals. This makes it inconvenient for us to use the distance error in the delusional source-target pairs during decision-ting as a metric analyze the reduction of delusional estimates, because of their growing scarcity.
As can see from Fig. 12, similar arguments about the effectiveness of the proposed hybrid strategies can be made, to those with Skipper. The hybrids with more investment in addressing E. 1, i.e., (E+G) and (E+P+G), exhibit the lowest E. errors (a)). Similarly, (E+P) and (E+P+G) achieve the lowest E.2 errors (b)). In e), we see that the 3 hybrid variants achieve better OOD performance than the baseline E. Specifically, (E+G) achieved the best performance. This is likely because that it induced the highest sample efficiency in terms of learning the estimations towards G.O subgoals, as shown in d). Assistive strategies such GENERATE and PERTASK do not only induce problematic targets, but also G.O ones that can shift the training distribution towards higher sample efficiencies in the traditional sense.
C.O.1. BREAKDOWN OF TASK PERFORMANCE
In Fig. 13, we present the evolution of LEAP variants' performance on the training tasks as well as the OOD evaluation tasks throughout the training process. Note that Fig. 12 e) is an aggregation of all 4 sources of OOD performance in Fig. 13 b-e).
LEAP+(E+P)
LEAP+(E+G)
LEAP+(E+P+G)
success
LEAP
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1,0
1.5
0.0
0.5
1.0
1.5
0.5
1.0
1.5
training steps
training steps
14
training steps
training steps
training steps
(a) training. 0.1
(b) OOD ovaluation, 8 0.25
(OD ovaluation, d 0.35
(d) OOD evaluation, 5 0.45
(e) OOD evaluation.
0.55
Figure 13. Evolution of OOD Performance of LEAP Variants SSM
D. Skipper on RDS (Exp. 3/8)
This set of experiments focus on the feasibility evaluator's abilities in the face of G.1 challenges. We present Skipper's evaluative curves in Fig. 14.
From Fig. 14 d), we can see that, probably because of the lack of dominant G.2 + E.2 cases, the OOD performance even the baseline Skipper is high (compared to the performance of LEAP baseline, because Skipper's ' planning behaviors are less prone to infeasible targets). (E+G), iee. the hybrid with the most investment GENERATE (aiming at E.1), performs the
19

---

Rejecting Hallucinated State Targets during Planning
Skipper Skipper+((3 PC+P)
Skippert (E+G) Skippert(E+PPO
aggragated OOD sucress rite
-3
E. DD(...)
targets chosen
.1
1.5
0.5
12
.001
0.0
0.5
1.0
1.5
0.0
0.5
1.0
2
6
8
10
12
0.0
0.5 training steps
1,0
1,5
training steps
training steps
source-tanget distances
(6) Estimation Errors
G.I E.I Behavior Ratio
E.0 Errors
Aggregated Perf.
Figure Skipper on RDS: delusions in terms L1 error estimated distance visualized, throughout training process. b) The curves represent the frequencies of choosing G.I "states" whenever a selection of targets is initiated; c) The final estimation accuracies towards targets after training completed, across a spectrum of ground truth distances. this figure, both distances (estimation ground truth) conditioned the final version of the evolving policies; The state structure of RDS does not permit G.2 targets and the corresponding E.2 delusions; d) Each data point represents OOD evaluation performance aggregated over 4 x 20 newly generated tasks, with mean difficulty matching the training tasks.
best both in terms of E delusion suppression (a)), and OOD generalization (d)), as expected. Also, we observe similarly that the more the evaluator is invested in GENERATE, which is appropriate for RDS without G.2 challenges, the higher the performance.
D.1. Breakdown of Task Performance
In Fig. 15, we present the evolution of Skipper variants' performance on the training tasks as well as the OOD evaluation tasks throughout the training process. Note that Fig. 14 d) is an aggregation all sources OOD performance Fig. 15 b-e).
500000 rale
Shipper+(E+ Chyppr+((++))
Skipper
Skipper+ (E (C+P+G)
1.5
training steps
training steps
training steps
training steps
training stops
training.
OOD ovaluation, 5
evaluation, 0.35
evaluation, 0.45 (e) OOD evaluation. 0.55
Figure 15. Evolution OOD Performance Skipper Variants on RDS
E. LEAP on RDS (Exp. 4/8)
The last set of experiments focus on LEAP's performance on RDS. Previously, in the main manuscript, we briefly looked the results of this part without variants other than (E+P+G).
Similarly, we present the evaluative metrics in Fig. 16.
The results in this set of experiments (Exp. 4/8) are very similar to that of Skipper on RDS (Exp. 3/s).
The conclusions are similar, despite that the OOD performance gain by addressing delusions is significantly higher than in SSM.
20

---

Rejecting Hallucinated State Targets during Planning
LEAP LEAP+(E+P) LEAP+(E+G)
LEAP+(E+R+G)
aggrogated OOD sucress rite
G1 sabgrals planned %
25
-3
delusional plan S
E. (.....)
2
1.5
5
.01
0.5 training steps
0.5
3
9 11 13
15
0.0
0.5
1,0
1.5
training steps
source-target distances
training steps
Ratio in Planned Sequence
Delusional Ratio
Estim. Erros
Aggregated 0OD Performance
Figure LEAP on RDS: Ratio of subgoals among the planned sequences; b) Ratio planned sequences containing least one G.1 target; e) The final estimation accuracies towards G.O targets after training completed, across range of ground truth distances. In figure, both distances (estimation and ground truth) are conditioned on the final version learned policies; d) Each data point represents OOD evaluation performance aggregated over 4 20 newly generated tasks, with mean difficulty matching the training tasks,
E.0.1. BREAKDOWN OF TASK PERFORMANCE
In Fig. 17, we present the evolution of LEAP variants' performance on the training tasks as well as the OOD evaluation tasks throughout the training process. Note that Fig. 16 d) is an aggregation of all 4 sources of OOD performance in Fig. 17 b-e).
5000000 2310
LEAP
LEAP+(E+P)
LEAP+(E+C)
LEAP+(E+P+G)
0.5
1.5
0.0
1.0
0.0
1.0
0.0
1.0
0.5
1.0
1.5
training steps
traniing steps
training steps
training steps
training steps
(a) training. = 0.4
(b) OOD ovaluation, 5 0.25
(c) 0OD ovaluation,
0.35
(d) OOD evaluation. 0.45
(0) OOD evaluation.
0.55
Figure 17. Evolution of OOD Performance of LEAP Variants on RDS
F. Background Planning: Dyna on RDS (Exp.%/s)
In Fig. 18, we present the empirical performance of Dyna variant with rejection enabled by (E+P+G), which is significantly better than the baseline.
G. Feasibility of Non-Singleton Targets (Exp. 7/8 & 8/8)
For this set of experiments, we want to demonstrate the capability of the learned feasibility evaluator facing non-singleton targets.
test if our implemented feasibility evaluator for Exp. 1/8 - Exp. 1/8 could withstand targets that are non-singleton. In its previous implementation, we use enforee the that the targets are singletons. In fact, each g takes the form of state representation and is only activated if state with exactly the same representation is reached. For the non-singleton experiments however, we let activate when state is within distance one to the target state, effectively expanding each target set from size to maximally size 5. Given the new termination mechanisms enforced by the new h, each target now, despite still taking the form of a state representation, has a new meaning. This setting mirrors the goal-conditioned path planning agents that seeks to reach certain neighborhoods of the planned waypoints.
With this setting, we can also intuitively analyze the composition of the target set. Specifically, if one of the member
21

---

Rejecting Hallucinated State Targets during Planning
target rejection rate (Dyna+)
.5
value error to GT (L.)
.8
-8
.4
success rale
Dyna Dyna+
.6
.6
.3
.4
.2
.2
.2
-1
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
training steps
training steps
training steps
Convergence Optimal Value
(b) Training Performance
Training Performance
Figure Dyna on RDS: a): Evolving mean I distances between estimated Q values ground truth optimals; b): evaluation performance the 50 training tasks; c): rate of rejecting Dyna updates.
state G.2, then the whole target set are fully made of G.2. all the 5 states are out of the state space, then the target fully composed of G.1. For SSM, a target in the temporarily unreachable situation, e.g., (1, 1) with target encoding @ (0, 1), could be composed of not only G.2 states but also some G.I.
We apply the new h evaluator training and to the ground truth DP solver, and then compare their differences. As could observe from Fig. 19, the proposed feasibility evaluator, with the help of the two assistive hindsight relabeling strategy, significantly reduces the feasibility errors in all categories.
episode
9
(E+P+G)
E.0 error (a(-.))
E1 error Da(t->)
E.2 error- (....))
7
61
5
1
1
0.0
0.5
1.0
1.5 1e6
0.0
0.5
1.0
1.5 166
0.0
0.5
1.0
1.5 1e6
training steps
training steps
training steps
(a) Evolution of E.O Errors
(b) Evolution of E.I Error
(c) Evolation of E.2 Emror
Figure 19. Feasibility of Non-Singleton Targets on SSM: a) Evolution of E.O error; b) Evolution of E.1 error: c) Evolution of E.2 error; The training data is acquired with random walk, since the introduced non-singleton targets do lead adequate performances.
We observe the similar results in RDS, which was presented in Fig. 7 in the main manuscript.
As extensively discussed, our evaluator is featured with three components: 1) the oof-policy compatible learning rule, 2) the unified architecture with distributional output head that can learn up to f'feaiibilytty and 3) two proper relabeling strategies against feasibility delusions. the Exp. l/s- Exp. 4/s, we demonstrated that such combination is effective against infeasible targets and used the difference compared the ground truth feasibility values quantitatively measure the convergence (the reduction in feasibility errors and delusions). In these last two sets of experiments, we tried to do the same while removing the need of control, focusing on the convergence itself under a random exploration policy, to demonstrate our evaluator's 's general applicability.
22

---

Rejecting Hallucinated State Targets during Planning
Appendix: Part III -Technical Details & Discussions
H. Discussions & More Details of GENERATE & PERTASK
H.1. Implementation of PERTASK
PERTASK takes the advantage of the fact that training is done on limited number of fixed task instances. We give each task a unique task identifier. At relabeling time, PERTASK samples observations among all the transitions marked with the same identifier as the current training task instance. This can be trivially implemented with individual auxiliary experience replays that store only the experienced states with memory-efficiet pointers to the buffered zy 's in the main HER.
H.2. Discussions
GENERATE not only creates targets with G.1 "states", but also generate valid targets that should resemble the distribution was trained on. Thus, it is not clear mixing in data augmented by GENERATE would result lower sample efficiency the estimation cases involving valid targets. Take SSM as an example, GENERATE seemed have detrimental effect E.O cases when applied to Skipper, while it greatly boosted accuracies for LEAP overall.
In some experiments, PERTASK demonstrated clear effectiveness in addressing E. as well, despite that it was not designed to. This is likely because of some generalization effects of the evaluator, which were trained with additional data that boosted data diversity.
In some environments, we expect that PERTASK could also be used (for mixtures of the generator) to learn to generate longer-distance targets from the current states if the generator has trouble doing so with FUTURE, with the accompanied risks of lower efficiency and G.2 hallucinations.
Implementation Details for Experiments
L.I. Skipper
Our adaptation of Skipper over the original implementation' Zhao et al. (2024) is minimal. We have additionally added two simple vertex pruning procedures before the vertex pruning based on k-medoids. These two procedures include: 1) prune vertices that are duplicated, and 2) prune vertices that cannot be reached from the current state with the estimated connectivity.
We implemented version of generator that can reliably handle both RDS and SSM with the same architecture. Please consult models. py in the submitted source code for its detailed architecture.
For SSM instances, since the state spaces are 4-times bigger than those of RDS, we ask that Skipper generate twice the number candidates (both before and after pruning) for the proxy problems.
All other architectures and hyperparameters are identical the original implementation.
For better adaptability during evaluation and faster training, Skipper variants in this paper keeps the constructed proxy problem for the whole episode during training and replanning only triggers re-selection, while during evaluation, the proxy problems are always erased and re-constructed.
The quality of our adaptation of the original implementation can be assured by the fact the E variant's performance matches the original on RDS.
1.2. LEAP
LEAP's training involves two pretraining stages, that are, generator pretraining and evaluator (a distance estimator) training.
We improved upon the adopted discrete-action space compatible implementation of LEAP (Nasiriany et al., 2019) from Zhao et al. (2024). We gave LEAP additional flexibility to use fewer subgoals along the way to the task goal if necessary. Also, we improved upon the Cross-Entropy Method (CEM) (Rubinstein, 1997), such that elite sequences would be kept
Shttps: twwb.com/miiii... 1a-
23

---

Rejecting Hallucinated State Targets during Planning
intact in the next population during the optimization process. We increased the base population size of each generation 512 and lengthened the number of iterations 10.
For RDS 12 x 12 and SSM 8 x 8, at most 3 subgoals are used in each planned path. We find that employing more subgoals greatly increases the burden of CEM and lower the quality of the evolved subgoal sequences, leading to bad performance that cannot be effectively analyzed.
We used the same generator architecture and hyperparameters as Skipper. All other architectures and hyperparameters remain unchanged.
Similarly for LEAP, for better adaptability during evaluation, the planned sequences of subgoals always reconstructed whenever planning is triggered. While in training, the sequence is reused and only a subgoal selection is conducted.
The quality of our adaptation of the original implementation can be assured by the fact the E variant's performance matches the original on RDS.
1.3. Dyna
The generator is a one-step model built for MiniGrid observations. For each batch update based on real, experienced transitions, an equal sized batch of simulated transitions will be generated with the help of the generator.
The threshold for 1-feasibility based rejections are set to be 0.05, i.e., if the feasibility estimator estimates that there less than 5% probability that generated target state is 1-feasible, the associated update would be rejected by setting its corresponding error to be 0 within the generated minibatch.
J. Applying the Evaluator on Dreamerv2
To demonstrate that our approach functions effectively in more generalist settings, such as those with continuous state and action spaces and partial observability, and to illustrate its application to a modern TAP agent, we integrated our proposed evaluator into Dreamerv2 (Hafner et al., 2021). The evaluator filters out potentially delusional values from infeasible states that might distort the X-returns derived from imagined trajectories. Given the technical complexity ahead, we suggest readers familiarize themselves with Dreamerv2 before continuing (Hafner et al., 2021).
Although Dreamerv2's 's stochastic states are discrete and could theoretically support similarity assessments, their design ensures they rarely repeat due to numerous possibilities, making them too random for our similarity function h. Consequently, we rely on the deterministic state representations s, which also prompt us to more thought- srogght-roooingg discussions.
Dreamerv2 operates as Dyna-like method, employing fixed-horize horizoo rollouts with autoregressively imagined states as targets. Lacking built-in similarity function h, it provides opportunity showcase how we construct l in our approach. Our method incorporates various realism aspects to assess state similarity between the next state and the target state, influencing the branching in Eq. 2 during evaluator updates (Russell et al., 2025).
J.1. How to craft h: Observational Realism
Observational realism, i... the similarity in terms of state representations is the first obvious criteria for h.
Theoretically, one might simplistically assume state equivalence by defining an c-ball around the target state. However, in practice, an e-ball based on L2 distances proves inadequate due to varying representation scales. Instead, we employ Mahalanobis distances, which better accommodate the representations' distributional variations.
To be more precise, we use an Exponential Moving Average (EMA) of the covariance of concatenated current-next deterministic state pairs [st, St+1] calculate the Mahalanobis distances between the next state pair [st, st+1] and target state pair (8,, $.+1].
J.2. How craft h: Behavioral Realism
The second focus is behavioral realism: does the agent exhibit similar behavior (e.g., in value, reward, and discount estimations) across the states (8t+1 & 1)+1)?
Here, we apply Mahalanobis distances to pairs of current and future values, rewards, and discounts, ensuring the states
24

---

Rejecting Hallucinated State Targets during Planning
appear similar from the agent's perspective.
Caution is required with action-realism. Naively applying method to one-hot encoded discrete actions could result in singular covariance matrix for the c-ball computation.
Preliminary Atari experiments suggest setting distinct e values for different components-stale -tate representations, value estimations, reward estimations, and discount estimations.
J.3. How to Relabel: Must-In-Time (JIT) Construction
Since Dreamerv2 samples sub-trajectories and computes state representations autoregressively. we forgo a separate HER for storing source-target pairs, opting instead for In-mmme (JIT) construction. Designed for single-envirommet training and evaluation, Dreamerv2 allows us to implement a (E+G) variant on agent-sampled sub-trajectories. Initial tests indicate a balanced mix of EPISODE (within sub-trajectories) and GENERATE performs effectively.
J.4. How Reject: Three Criteria for X-refurns
Dreamerv2 leverages its model imagine future states and values, using these, along with intermediate rewards and discounts, compute X-returns for each origin state.
For such strategy, implemented the following 3 criteria for rejecting the imagined states:
1. Transition-wise Rejection: If a next state seems unlikely to follow from the current state, its value is deemed untrustwor- thy. This process is repeated for all imagined transitions. Notably, state rejected as infeasible in one transition might still be reachable elsewhere, so subsequent states are not automatically discarded.
2. Point-to-Point (P2P) Rejection for Targets:
Starting from replay-sampled base state, imagined state is reachable, regardless of steps taken. This counters hallucinated targets from accumulated errors over the imagination horizon (Talvitie, 2017), excluding such states from value estimation targets.
whether each
P2P Rejection for Current States: Entirely unreachable states are excluded as current states in multi-step value updates, though subsequent states may remain viable.
A-retum over steps
A-return over steps
retumm over steps
A-relurn over 6 sleps
A-relumn over sleps
d-return over steps
A-retum over steps
A-return over steps
original targets
step return
trajectory
X
(X: rejected states)
7-teturn over steps
7-return over steps
step return
NOT UPDATING
7erelurn over sleps
step return
NOT UPDATING
dereturn over steps
step relurn
truncated targets
Figure 20. Truncated X-Returns with Rejected States: the original X-returns illustrated in the top row, while the truncated returns are illustrated in the bottom row with the differences marked in red. Our strategy ensures that the critic targets in the trajectories can be maximally preserved for updates. The states right before the rejected Starting from the last rejected state, all critic targets remain the same as the originals.
will have trustworthy critic targets and thus updated.
The first two criteria yield binary mask to truncate X-returns in sampled sub-trajectories, excluding untrustworthy values
25

---

Rejecting Hallucinated State Targets during Planning
while preserving horizons for reliable ones. Our repository offers efficient implementation, maintaining the complexity the original, un-truncated X-returs. The behavior of the (critic) target-based rejection is presented in Fig. 20.
The third criterion masks updates to wholly infeasible imagined states, By examining the rejection rate by the horizon index, the evaluator can also be used to understand how long the imagined trajectories are likely to be trustworthy and thus adjust the associated hyperparameters.
developed a standalone, user-friendly evaluator (implemented in PyTorch) that integrates seamlessly into TAP agents like Dreamerv2, employing its own optimizer and target networks for robust learning when activated. Please check eva Ivrauuaor.co py in the source code (https: http://ttt....////mm la-iqia a de luelissss).
We tuned the hyperparameters using the Atari environments and found that both the autoregressive estimations of distances and the P2P distances (towards the target states) in the sampled and imagined trajectories roughly converge to the estimated ground truth values, which are deduced from their time indices. This the best we can do for environments without ground truth access.
Regrettably, our Atari 100k preliminary results with 10s interactions show negligible performance gains over the baseline (Lukasz Kaiser et al., 2020). This is likely because the state representations of Dreamer usually takes significant portion of training to stabilize and for the evaluator to adapt to. The differences expected to show with prolonged experiments where significant number of updates will be made after the state representations stabilize. Limited computational resources prevented our extended experiments, and we invite those with greater capacity to investigate further.
Our Dreamer implementations can be found the source code repository.
26


Reviews:
[{"review_id": "WGmYuPjeYG", "reviewer": "Reviewer_m9Mj", "summary": "The paper analyzes the issue of generating invalid subgoals during planning. The authors categorize different failure modes and propose strategies for learning a classifier that can be used to estimate the distance to a proposed goal, including whether it is reachable at all. Through experimental evaluation in a grid-based task, the paper analyzes the impact of different learning strategies on the effectiveness of the evaluator.\n\n## Update after rebuttal\n\nThank you for the answers. I acknowledge the differences between the proposed strategies and those present in the literature. However, I believe these differences need to be discussed more precisely in the paper to better highlight the contribution -- as noted by other reviewers as well, this is not currently clear. I leave the particular choice of references to the authors' choice.\n\nI acknowledge the authors' focus on evaluating the benefits of a non-delusional evaluator compared to a standard one. However, completely omitting the non-evaluator aspect makes the analysis incomplete. Even if performance without the evaluator is significantly weaker, this should be demonstrated and briefly remarked. There is value in advocating for a non-delusional evaluator only if using an evaluator is beneficial in the first place, even if the overall focus of the paper is slightly different.\n\nIn summary, I believe the paper is a solid contribution, but a careful revision would considerably strengthen its impact. Specifically, I suggest:\n\n- Revising the stated contributions as discussed,\n\n- Clarifying the novelty of the proposed strategies,\n\n- Including a naive non-evaluator baseline in the comparison,\n\n- Adding experiments in widely studied environments.\n\nWith such improvements, I would consider the paper very strong. For now, I remain on the fence.\n\nI acknowledge the changes proposed by the authors, which seem to move in the right direction. I reflect this by increasing my rating. I would not oppose accepting the paper, though I believe it could still be significantly strengthened with minor effort.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We reordered the questions to streamline our response.\n\n`Discuss the relation … to [Z et al.] & [A et al.]. Are the targets for “generate” generated during sampling, or …?`\n\n[Z et al.]\n\nOur approach of rejecting infeasible targets is indeed similar to that in [Z et al.]. Yet, the approaches differ significantly. [Z et al.]’s verifier trains on a collected dataset that informs about the generated subgoals (not relabeled) & requires separate training for the verifier.\nUnlike the method-specific approach in [Z et al.], we propose a generally applicable secondary system running alongside TAP agents, using relabeling, and thus it can figure out the infeasible targets (sets of states) that were not followed, without extra interactions or supervised training.\n\nWe wanted to try our best to acknowledge the developments leading to ours. This is why we discussed MHER (Yang et al., 2021) in Sec 6 for the 1st work of model-based to relabeling transitions (even closer to “generate”) and (Jaferjee et al., 2020) for their early idea of rejecting delusional generations. It was an *honest mistake* that we missed [Z et al.] and we now added proper acknowledgements.\n\n\"generate\"& JIT relabeling\n\n\"generate\" is novel not only for its flexibility to be used just-in-time (JIT, only relabeling after batch is sampled to adhere to the generators’ changing output distributions), but also for its effectiveness to address feasibility delusions. Compared to approaches like MHER, \"generate\" lowers needed storage and provides timely coverage of the generators’ outputs, especially helpful in continual learning settings.  \n\n\"pertask\" & [A et al.]\n\n\"pertask\" is ONLY equivalent to “random” in [A et al.] when agents are trained on a single task. In settings where agents are trained on a few tasks and are expected to generalize during evaluation (where our exps were based, to force the evaluators to understand infeasible targets, instead of memorize), \"pertask\" enables relabeling beyond trajectory-level against delusions, per Sec 5.1.\n\n`the exps focus on strategies of training rather than the impact of evaluator on performance. … only indirectly argues that identifying invalid subgoals is useful. the main claims do not state any performance advantage … the impact of the evaluator on the performance of hierarchical methods?`\n\n**This is a crucial misunderstanding that we wish to clarify, so we can resolve your other concerns effectively.**\n\nOur aim was not to show methods can do better with an evaluator, but a non-delusional one.\nExps on the two HP agents focused on the fact that their basic relabeling strategies produce delusional evaluators (they have built-in evaluators). Only for methods without estimators to begin with, e.g. Dyna, we need to completely inject an evaluator. We discussed these explicitly in Sec 2, Sec 5.1.2 & Sec 6.\n\nDue to the reply limit, plz find MORE details on this, in the 1st reply to Reviewer GvFL.\n\nWe added clarifications in the revision.\n\n`The naive approach to invalid targets is to ignore, as they aren’t very common, and even invalid subgoal guidance lead somewhere, from where another subgoal can be generated.`\n\nIn Sec 2, we formulated your described scenario for the naïve approach, which motivated our contributions. The naïve approach only works when 1) infeasible targets are indeed rare and 2) the state space is simple. Also, the approach is boosted by 3) survivorship bias.\n1. There are generally no measures of how frequent infeasible targets can appear, because most existing methods are not tested on proper environments, that can be solved to produce the true frequency of infeasible targets. \n2. it is not hopeful that invalid guidance could lead agents to a recoverable region of a complex state space\n3. Empirical observations of how naïve approaches could work are influenced by survivorship bias, as methods that are more affected by delusional plans have worse performance and are thus less likely to be recognized.\n\n`computational overhead?`\n\nPlz see our reply to Reviewer bSxw (last point). \n\n`the main claims should be reformulated, as they are too optimistic. The idea … to identify infeasible subgoals was already proposed … `\n\nWe hope our previous explanations addressed your concerns on this point.\nWe summarized our work’s claims and tried our best to clarify in the revision:\n\nBuilding on the ideas of rejecting invalid goals such as in [Z et al.], our extensive study reveals the types of infeasible targets in TAP agents. Accordingly, the generality of our proposed solution leads to non-delusional feasibility estimates beyond HP methods. \n\n---\nWe appreciate your detailed review!\nThe intention of this work is to inform the research community about the associated risks and improper assumptions and save everyone’s time and effort.\nWe take your comments very seriously and tried our best to address your concerns. We hope you can increase your rating to recognize our attitude and the positive impact this work could have.", "claims_and_evidence": "The paper is generally sound, although the main claims should be reformulated, as they are too optimistic. The idea of training a model to identify infeasible subgoals was already proposed, although possibly not extensively studied (see e.g. [Zawalski et al.]). Furthermore, the proposed relabeling strategies are not \"novel\", as Generate is similar to the one used e.g. in [Zawalski et al.], and Pertask is equivalent to Random from [Andrychowicz et al.]. While itself it does not deny the contribution of the paper, the formulation of the main contributions has to be revised. I believe that is why the paper lacks focus.\n\nAdditionally, the paper only indirectly argues that identifying the invalid subgoals is useful in general. For instance, the main claims do not state any performance advantage, and the experiments focus on comparing different strategies of training the evaluator rather then the impact of evaluator on performance. The only somewhat relevant plot is Fig 3d (and counterparts in the appendix). A very good step in this direction is Section 5.2. The paper would strongly benefit from including more evaluation of this kind, i.e. demonstrating that various methods can do much better having the evaluator. Currently I see little such discussion, which is a pity.\n\nReferences:\n\n[Zawalski et al.] _Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search_ \n\n[Andrychowicz et al.] _Hindsight Experience Replay_", "methods_and_evaluation_criteria": "The presented evaluation makes sense for the formulated main claims. However, slightly changing the scope to cover the usefullness of the evaluator, possibly also in well-established benchmarks, would improve the contribution.", "theoretical_claims": "There is one theorem: Result 4.1. No proof is referenced, but since it is rather straightforward, it needs no proof.", "experimental_designs_or_analyses": "Yes, I checked the experiments in the main part.", "relation_to_broader_scientific_literature": "The problem of detecting invalid subgoals is not new and has been (at least partially) studied in previous works. Variants of the proposed strategies for learning the evaluator can be also found in related works. However, I am not aware of a systematic study of this topic, so it has the potential to be a good contribution.", "essential_references_not_discussed": "One additional reference I would like to be discussed is [Zawalski et al.], as detailed in the comments.", "other_strengths_and_weaknesses": "While the paper has potential, it should be much more focused. Too little space is reserved for experiments, and because of that many of them had to be moved to appendix. I suggest making the analysis more concise and providing broader experimental support. I suggest working on the main contributions to establish the focus. Something around [\"Systematic analysis of infeasible subgoals issue\", \"Effective training and architecture for Evaluator\", \"Experimentally validating the impact of Evaluator on performance\"] could be a good starting point.", "other_comments_or_suggestions": "Are the targets for the Generate strategy additionally generated during sampling from buffer, or are they the targets generated by the generator during collecting the experience and stored afterwards? I suppose the former, but that should be made clear in Sec 4.2.1.", "questions_for_authors": "1. Please discuss the impact of having the evaluator on the performance of hierarchical methods. The naive approach to invalid targets is to ignore them, as they are not very common, and even invalid subgoal guidance lead somewhere, from where another (hopefully valid) sobgoal can be generated. Are the methods equipped with Evaluator more effective? What is its computational overhead?\n2. Please discuss the relation of the proposed relabeling strategies (claimed to be novel) to [Zawalski et al.] and [Andrychowicz et al.]", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DBOnEGrYy4", "reviewer": "Reviewer_GvFL", "summary": "The paper proposes to augment Target-Assisted Planning (TAP) methods with an evaluator to reject  generated states that are unfeasible and improve performance. The proposed method is evaluated on two environments: SwordShieldMonster (SSM) and RandDistShift (RDS) with 3 different TAP agents: Dyna, Skipper and LEAP.  \n\nIt proposes 4 different mechanisms to create targets for the evaluator.\nFuture (F), which sample states in the future \nEpisode (E), which sample states from the same episode\nGenerate (G), which replace states by their generated version predicted by the generator\nPer task (P), which sample states from the same environment task\n\nThe experiments show that a combination of target mechanisms such as (E+P+G) helps to further reduce the evaluator prediction errors and improve final success rates on these 2 environments.\n\n## update after rebuttal\n\nI appreciate the authors clarification of the paper contributions. I understand that the algorithm tackles the rejection of hallucinated state predictions. The authors propose to learn an evaluator to reject these hallucinations. For that, the authors propose a combination of learning rules, including two novel relabelling strategies (PerTask and Generate). The technique is evaluated on two toy environments (SSM and RDS) providing game logic to prove the efficacy of the contributions.\n\nAs final decision:\n\nI am still hesitant toward the generalization of the method. I think that the experiments done on the two environments are very compelling because they offer game logic for analysis. However, I think the paper would strongly benefit from experiments on commonly used benchmarks for model-based RL in order to provide general empirical results on top SSM and RDS (Comparison with concurrent approaches, ablation study on hallucination rejection). Visualizations of rejected predictions for some of the tasks would also help to highlight the method’s contribution.\n\nI hence maintain my score to weak rejection, tending toward borderline.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`WHAT IS THE PERFORMANCE OF SKIPPER & LEAP  WHEN THE EVALUATOR IS NOT USED TO REMOVE PREDICTED INFEASIBLE STATES?`\n\nAs discussed in Sec. 2 (Line 57 right column), Sec 5.1.2 (L383 left col.) , Sec. 6 (L388 r. col.), these methods already use their own evaluators to remove the targets they think that are infeasible. Yet, they suffer from delusions due to the lack of exposure to truly infeasible targets absent from collected experience. \n\nThus, our exps with both methods \"reused\" their built-in distance / discount estimators, which were *originally trained with basic relabeling strategies*. We showed that by fixing their relabeling, these methods become robust against delusional planning behaviors. Only for methods without built-in estimators at all, e.g., Dyna, we have baselines with the evaluator removed.\n\nIn other words, exps with Skipper & LEAP focuses on the importance of training data for non-delusional estimates. While exps 5/8 - 8/8 validate the combination of architecture, learning rules and training data at the same time.\n\nHope that you can now see that your wanted baselines are already there: they are the variants with \"future\" and \"episode\" (basic relabeling that causes delusions).\n\nWe realized that this could be somewhat confusing and tried to clarify in the revised manuscript.\n\n`THE METHOD IS EVALUATED ON ONLY 2 SIMPLE ENVIRONMENTS (SSM AND RDS). WHILE THE PAPER SHOWS THAT PERFORMANCE CAN BE IMPROVED BY REMOVING DELUSIONAL PREDICTIONS. IT WOULD BE INTERESTING TO EXPERIMENT ON COMMONLY USED BENCHMARKS …`\n\nOur reasons for the selected experiments are as follows:\n1. These environments provide much more convincing quantitative validations of our claims on hallucinated targets and delusional estimates. Common benchmarks such as Atari, due to the lack of access to ground truth, cannot be properly diagnosed. As a result, we cannot prove that our method can indeed reduce delusional planning behaviors in those environments due to their nature.\n2. We aimed to show our approach’s generality by applying it on many categories of TAP methods. The compute demanded in these experiments already exceeds what our limited academic environment provides.\n3. Visual simplicity does not mean task simplicity. Due to the multi-task, generalization focused setting, agents are met with difficult combinatorial challenges that even state-of-the-art hierarchical planning methods cannot solve well, see [Zhao et al., 2024]. As an example, despite the visual simplicity, the hallucination rates remain high even with the used SOTA methods.\nFor the points above, we focused on depth rather than breath when considering the environments for our experiments.\n\n\n`FROM MY UNDERSTANDING, THE LABELING OF TARGETS AS FEASIBLE OR UNFEASIBLE REQUIRES HAVING ACCESS TO GAME INNER LOGICS (HAVE SWORD, HAVE SHIELD). SUCH THAT WE CAN CORRECTLY LABEL SAMPLED TARGETS AS FEASIBLE OR UNFEASIBLE GIVEN THE START STATE. I DO NOT SEE HOW IT COULD BE APPLIED TO ENVIRONMENTS WHERE GAME LOGIC IS NOT ACCESSIBLE.`\n\nWe respectfully note that this is a misunderstanding; our approach distinguishes itself from methods that require access to more environmental mechanics. The proposed evaluator figures out the feasibility of all targets through Eq. (2) without the need for labels: **Eq. (2) exploits the fact that infeasible targets will never be reached**.\n\nEq. (2) enables the evaluator to learn as a secondary system alongside the TAP agent to which it is attached, and figure out from data, the feasibility of the sampled targets it is trained on.\n\nGround truths are only used for experimental analyses, our solution operates without ANY.\n\n`THE PAPER TALK ABOUT POSSIBLE EXTENSIONS TO OTHER TAP METHODS SUCH AS MUZERO, SIMPLE OR DREAMER BUT DO NOT PERFORM EXPERIMENTS ON THESE KEY METHODS.`\n\nIn Table 2, we categorized the compatible methods into 3 categories of similar behaviors, and in experiments, **we implemented one representative method for each of the categories**. \n\nWe made a conscious effort to provide you with the most convincing results out of the limited resources at our disposal. We hope that you understand our difficulties in an academic setting.\n\n---\nThank you for your comments. Your biggest concern was about the applicability of the method without access to feasibility labeling. We hope that our explanations will address our miscommunication on this point. For your comments on experiments, we clarified that the certain baselines in the mentioned figures are the original performance of the agents, as both Skipper and LEAP are equipped with built-in feasibility-like estimators, which we dual-purposed as evaluators.\n\nWe are thankful that you acknowledged that “this direction deserves to be explored” and hope that our responses have addressed your concerns, and you could increase your rating of this work to recognize our contributions.", "claims_and_evidence": "The experiments show that the method can successfully reduce delusional behaviors and enhance the the performance of planning agents.", "methods_and_evaluation_criteria": "Yes, TAP and environments allows to correctly make into application the proposed method.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The contribution of this paper is related to model-based reinforcement learning agents that may suffer from hallucinations during the generation process. Early training of model-based approaches can sometimes impacted by the lower generation quality of the world model.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n- The paper tackles a very interesting problem linked to TAP, which is the generation of infeasible states by world models. TAP methods usually suppose that all generated states are feasible and reduce the agent training loss on all generated samples. This research direction deserves to be explored.\n- The paper demonstrates that learning an evaluator function can help to improve performance on the two chosen environments. The evaluator can successfully learn to progressively identify infeasible targets to remove them.\n\nWeaknesses:\n- The method is evaluated on only two simple environments (SSM and RDS). While the paper shows that performance can be improved by removing delusional predictions. It would be interesting to experiment on commonly used benchmarks such as Atari 100k or the DeepMind Control Suite.\n- From my understanding, the labeling of targets as feasible or unfeasible requires having access to game inner logics (have sword, have shield). Such that we can correctly label sampled targets as feasible or unfeasible given the start state. I do not see how it could be applied to environments where game logic is not accessible.\n- The paper talk about possible extensions to other TAP methods such as MuZero, SimPLe or Dreamer but do not perform experiments on these key methods.", "other_comments_or_suggestions": "I tend toward borderline but would be ready to increase my score to weak accept or accept if additional experiments and/or information are provided.", "questions_for_authors": "- Figure 3 (d), 10 (d) and 12 (d) compare the performance of Skipper and LEAP when using different labelling mechanisms. What is the performance of these methods when the evaluator is not used during training to remove predicted infeasible states ?\n\n- From my understanding, the labeling of targets as feasible or unfeasible requires having access to game inner logics (have sword, have shield). Could this method be used on environments where game logics is not available to label states as feasible/infeasible ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KeaSE1j1X0", "reviewer": "Reviewer_bSxw", "summary": "This paper addresses the issue of hallucinated state targets in model-based reinforcement learning (MBRL), where generative models can produce unrealistic or unreachable states, leading agents to delusional planning behaviors. Inspired by human cognition, the authors propose an evaluator that assesses the feasibility of generated targets and rejects infeasible targets before planning occurs. To ensure accurate feasibility evaluation, the authors introduce two hindsight relabeling strategies, generat and pertask, demonstrating significant performance improvements and reduced delusional planning behaviors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`HOW EVALUATOR PERFORMS IN PARTIALLY OBSERVABLE DOMAINS? WOULD THE APPROACH NATURALLY EXTEND IF THE “SOURCE STATE” IS UNCERTAIN?`\n\nWe agree with your intuition that it naturally extends.\n\nThe evaluator takes on paired inputs of the source state representation and the target representation, both are outputs from the TAP agent to which the evaluator is attached. NO change is needed when the evaluator is fed representations (that can encode uncertainty) from a POMDP-compatible encoder.\n\nWe are confident about this, yet we know it may be more convincing to conduct experiments to provide empirical evidence. Unfortunately, ICML prohibits revisions this time.\n\n`IF THE GENERATOR PRODUCES MORE ABSTRACT OR HD LATENT GOALS (E.G., LANGUAGE INSTRUCTIONS OR SUBTASK DESCRIPTORS), WHAT CHANGES ARE REQUIRED FOR THE EVALUATOR?`\n\nSince the evaluator takes in target representations, our design needs NO CHANGE. We validated this empirically in Sec. 5.3.\nWe tried to make clear that our approach currently only applies to TAP agents with a function $h$ verifying if a target is fulfilled. TAP agents’ $h$ gives us the compatibility to general targets including language instructions. \n\nInstead of changes to the evaluator, the focus should be on a proper $h$ (e.g., ask the LM if a state matches the target). We discussed this at the end of Sec. 7.\n\n`THE GRID-WORLD-BASED TASKS REMAIN SIMPLE COMPARED TO MORE COMPLEX, HD ENVIRONMENTS. UNCLEAR IF THE APPROACH WILL SCALE WITHOUT ADDITIONAL ENGINEERING. … THIS … MAY BE LESS STRAIGHTFORWARD TO REPLICATE OR VALIDATE IN CONTINUOUS OR PARTIALLY OBSERVABLE DOMAINS.`\n\nOur reasons for the selected experiments are as follows:\n1. These environments provide much more convincing quantitative validations of our claims on hallucinated targets and delusional estimates. Common benchmarks such as Atari, due to the lack of access to ground truth, cannot be properly diagnosed. As a result, we cannot prove that our method can indeed reduce delusional planning behaviors in those environments due to their nature.\n2. We aimed to show our approach’s generality by applying it on many categories of TAP methods. The compute demanded in these experiments already exceeds what our limited academic environment provides.\n3. Visual simplicity does not mean task simplicity. Due to the multi-task, generalization focused setting, agents are met with difficult combinatorial challenges that even state-of-the-art hierarchical planning methods cannot solve well, see [Zhao et al., 2024]. As an example, despite the visual simplicity, the hallucination rates remain high even with the used SOTA methods.\nFor the points above, we focused on depth rather than breath when considering the environments for our experiments. Our approach does not assume the input space to be discrete or continuous. We chose discrete input spaces because it is nearly impossible to solve the ground truths, used to provide convincing, rigorous analytics, in continuous spaces.\n\nFor POMDP, please check our prior responses.\n\n`DISCUSSIONS ON THE COMPUTATIONAL OVERHEAD WOULD BE VALUABLE.`\n\nTechnically, it is useful to view our solution as a special form of rejection sampling, where the proposal distribution and the support are provided by the generator, the target distribution is the re-normalized distribution over only feasible targets and feasibility estimates by the evaluator is used to determine the rejection. Thus, the more accurate the evaluator, the more efficient the sampling process. It is impractical to assume full access to either the target or proposal distributions because they change with the used 1) environment and 2) generator (whose output is not only different per method but also changing with learning). This means that we cannot give a blanket statement about the computational overhead, prompting us to have used only brief discussions about overhead in Sec. 4 (L189, left col.).\n\nPractically, each generated target needs only 1 evaluation. For background TAP agents that generate batches of targets, the improper ones can be rejected and the whole batch can be all rejected without problem (no \"Dyna\" update this time). For decision-time TAP agents, targets act as subgoals and when they are rejected, the agent can retry or commit to more random explorations.\n\nWith DRL, the overhead also depends on evaluator’s networks, complexity of the state / target representations. However, since the evaluator is a rather lightweight secondary end-to-end network, we can expect evaluations to be fast.\nWe added a condensed version of these comments to the revised manuscript.\n\n---\nWe appreciate your review and your concerns about the applicability of our approach to more generic cases. We explained to you our choices and wish you could perceive the difficulties we overcame in acquiring the convincing results with limited resources.\n\nWe hope our responses addressed your concerns well and please consider increasing your rating.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical section is presented in the paper.", "experimental_designs_or_analyses": "I find the experimental designs reasonable and sufficiently aligned with the paper’s objectives.", "relation_to_broader_scientific_literature": "The proposed approach builds upon prior model-based RL frameworks and opens the door to addressing the critical issue of hallucinated state targets through the explicit introduction of a feasibility evaluator trained via novel hindsight relabeling strategies.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper addresses a critical issue in model-based RL—hallucinated targets—with implications for safety and performance.\n\nThe idea of combining a feasibility evaluator with hindsight relabeling strategies has intuitive appeal. Proposed approach is a simple, plugin mechanism that could help mitigate incorrect updates or infeasible sub-goal selection in a broad class of model-based agents.\n\nThe paper provides clear taxonomy of targets (G0, G1, G2) structures the problem effectively.\n\nThe paper includes multiple experiments—on both decision-time and background planning settings—demonstrating the reduction of delusional behaviors and improved performance.\n\nWeaknesses:\n\nWhile the grid-world-based tasks are carefully designed to showcase the phenomenon of hallucinations, they remain relatively simple compared to more complex, high-dimensional environments (e.g., robotics or rich 3D worlds). It is unclear if the proposed approach will scale without additional engineering. My major concern is that the experiments are mostly on tasks where there is a clear, easy-to-obtain ground-truth for feasibility. This is ideal for demonstrating concept correctness but may be less straightforward to replicate or validate in continuous or partially observable domains.", "other_comments_or_suggestions": "Some discussion on the computational overhead would be valuable.", "questions_for_authors": "How the feasibility evaluator performs in partially observable domains? Would the proposed approach naturally extend if the “source state” is uncertain?\n\nIf the generator produces more abstract or higher-dimensional latent goals (e.g., language instructions or subtask descriptors), what changes are required for the feasibility evaluator?\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a systematic analysis of the properties and impact of different types of infeasible targets in Target-Assisted Planning (TAP) agents. The authors introduce a generic target evaluator capable of rejecting targets across a range of TAP agents. Importantly, the evaluator is designed as an add-on module that requires no modifications to the agent’s behavior or architecture, but it does require access to the same experiences. The authors analyzed the causes of evaluator delusions and proposed several methods to assess feasibility across exposed targets. Experimental results demonstrate significant reductions in delusional behavior and improved performance across various TAP agents.

This paper was reviewed by three reviewers, with broad agreement that the claims and contributions are significant and well-supported by extensive experimental evaluation.

The author response phase was intensive and very helpful in clarifying the key contributions and novelty of the work. I acknowledge the efforts by the authors in making these aspects clearer during the rebuttal. All reviewers agreed that a revised version should clarify the contributions and novelty. This was addressed in the revised version I considered for writing the meta-review.

However, all reviewers shared a concern, and I agree with them, about the limited generality of the evaluated tasks, which focus exclusively on grid-like domains. While the reasons outlined by the authors for considering this environment are appropriate and convincing for evaluating their methodology, it remains unclear whether the proposed methodology and associated computational overhead are necessary or feasible beyond these domains, given that the main purpose is simply to reject hallucinated states (as the authors state in Fig. 1 and Section 4.3 of the revised manuscript).

These considerations reduce the impact of their work, and based on them, I recommend a weak accept.
Decision: Accept (poster)