Generative Proto-Sequence: Sequence-Level Decision Making for Long-Horizon Reinforcement Learning
Authors: Netanel Fried
Liad Giladi
Gilad Katz
Venue: Accepted by TMLR ()

Abstract: Deep reinforcement learning (DRL) methods often face challenges in environments characterized by large state spaces, long action horizons, and sparse rewards, where effective exploration and credit assignment are critical. We introduce Generative Proto-Sequence (GPS), a novel generative DRL approach that produces variable-length discrete action sequences. By generating entire action sequences in a single decision rather than selecting individual actions at each timestep, GPS reduces the temporal decision bottleneck that impedes learning in long-horizon tasks. This sequence-level abstraction provides three key advantages: (1) it facilitates more effective credit assignment by directly connecting state observations with the outcomes of complete behavioral patterns; (2) by committing to coherent multi-step strategies, our approach facilitates better exploration of the state space; and (3) it promotes better generalization by learning macro-behaviors that transfer across similar situations rather than memorizing state-specific responses. Evaluations across diverse maze navigation tasks of varying sizes and complexities demonstrate that GPS outperforms leading action repetition and temporal methods in the large majority of tested configurations, where it converges faster and achieves higher success rates.

Published in Transactions on Machine Learning Research (12/2025)
Generative Proto-Sequence: Leeee Decision Mak- ing for Long-Horizon Reinforcement Learning
Netanel Fried"
Diethett@post.bgu.cc.
Deportment of Computer Science and Information
Ben- Gar-Grroon University of the Negev
Liad Giladi"
giladili@post.bgg.co
Deportment Computer Science and Information Ben- Can-Gurion University of the Negev
Gilad Katz
giladke&bgu.ac..
Department of Computer Science and Information Ben- Ben-Gurion University of the Negev
Reviewed on OpenReview: /ttps://openreeeee opwrreviee.....e...mmmom net/ per/formm? id- fSG2DRHtOg
Abstract
Deep reinforcement. learning (DRL) methods often face challenges in environments char- acterized by large state spaces, long action horizons, and sparse rewards, where effective exploration and credit assignment are critical. We introduce Generative Proto-Scquence (GPS), a novel generative DRL approach that produces variable-length discrete action se- quences. By generating entire action sequences in single decision rather than selecting individual actions each timestep, GPS reduces the temporal decision bottleneck that impedes learning in long-horizon tasks. This sequence-level abstraction provides three key advantages: (1) it facilitates more effective credit assignment. by directly connecting state observations with the outcomes of complete behavioral patterns; (2) by committing to coher- ent multi-step strategics, our approach facilitates better exploration of the state space; and (3) it promotes better generalization by learning macro-belle beaavios that transfer across similar situations rather than memorizing state-specific responses. Evaluations across diverse maze navigation tasks of varying sizes and complexities demonstrate that GPS outperforms lead- ing action repetition and temporal methods in the large majority of tested configurations, where it converges faster and achieves higher success rates.
1 Introduction
Deep reinforcement learning (DRL) has demonstrated impressive performance diverse applications (Mnih et al.] 2015 Silver et al., 2016 Levine et al., 2016). However, significant. challenges remain when dealing with environments characterized by large state spaces, long-range tasks, and sparse rewards. In such contexts, traditional DRL methods that select actions sequentially often suffer from inefficient exploration and poor credit. assignment (Mesnard et. al. 2020 Raileanu & Rocktäschel 2020: Ecoffet et al. 2021 leading to difficulties in learning effective policies for tasks that require coordinated, multi-step strategies. These challenges are further exacerbated by sparse reward signals, whose limited feedback hinders the agent's
ability discover and reinforce successful behaviors (Arjona-Medina al. 2019: Hung al. 2019).
Recent rescarch efforts have attempted to address these challenges using diverse techniques such hicrar-
chical DRL (Kulkarni et al.
2016
Xu et al. 2022), temporal abstraction (Biedenkapp el al. 2021 Zhang
et al.
2022b Saanum et al.
2023 Patel & Siegelmann, 2024), sequence modeling (Chen et al. 2021 Janner
et al.
2021 Giladi & Katz 2023), and action repetition stratogics (Srinivas et al. 2017 Sharma et al. 2017
Dabney et al. 2020).
By creating sub-tasks or capturing higher-level behavioral patterns Posste-Beaat et al.
"Equal contribution

---

Published in Transactions on Machine Learning Research (12/2025)
2023). these techniques aim to reduce decision frequency and enhance tasks. Although these methods offer promising results, they often require careful sub-task design (Ajay et al.) 2023). reward shaping (Liu et al. 2022), or complex training procedures (Soo & Abbeel 2024 Rosote-Beas ct al.) 2023 While there are temporal abstraction methods that generate multi-step action sequences; they often depend on ilerative rollouts, autoregressive decoding, or model-based simulations. These solutions introduce computational overhead and restrict the ability to efficiently generate diverse action sequences. To our knowledge, no method supports the generation of coherent, variable-length action sequences directly from state observations in single decision step.
2023
learning efficiency in Wang et al.
Vezzani et al.
2022:
In this study, we propose Generative Proto-Sequence (GPS), novel actor-critic architecture capable of producing variable-length action sequences. Instead of actions, our Actor generates a prolo-sequence em- bedding, which is then decoded into a discrete action sequence using a Decoder component. The Critic evaluates the state and the entire generated sequence jointly, with gradients flowing from the critic through the Decoder to the Actor, facilitating end-to-end learning of strategic, multi-stop action sequences. This de- sign enables the agent to generale and execute complex exploratory behaviors in single decision, enhancing both generalization and long-horiag credit assignment.
We evaluated GPS on set of challenging maze environments with varying sizes and configurations, includ- ing rooms, corridors, and randomly generated obstacles. Our results demonstrate that GPS learns more efficiently across most configuratious, generalizes better to novel maze layouts, and outperforms leading baselines in terms of success rate and convergence speed, particularly in large and complex mazes. Our contributions are as follows:
We introduce a novel architecture that enables end-to-end generation and evaluation of variable-length discrete action sequences, facilitating improved credit assignment and exploration.
We demonstrate that producing multi-step action sequences in a single shot can lead to improved generalization and faster convergence in maze navigation tasks, particularly in large and complex con- figurations.
We provide empirical results challenging maze benchmarks, showing improvements top- performing action repetition and temporal methods basclines in metrics such as convergence speed and success rate. We make our code publicly available"
2
Related Work
2.1 Temporal Abstraction Through Action Repetition
Various approaches in DRL have explored temporal abstraction by repeating primitive actions to extend decision horizons. Earlier works (Srinivas et al., 2017] (Sharma et al.) 2017) introduced frameworks for dynamic action selection and repetition, though their repetition policies operated independently from chosen actions, limiting strategic development.
DAR. (Srinivas et al. 2017) augments discrete action spaces by duplicating each base action with multiple repetition rates. While this expansion can improve learning in environments benefiting from temporal abstraction, produces an incfficient representation duplicatod actions are treated as unrelated, preventing the agent. from exploiting their shared underlying behavior and leading to slower learning and imbalanced trade-offs between coarse and fine control. FiGAR (Sharma et al. 2017) addresses this by decoupling behavior and repetition into two jointly trained policies; however, the repetition policy operates independently from the chosen action, limiting the development of muanced, action-specific repetition strategies.
The authors Dabney et al. 2020) proposed an exploration strategy repeating actions for random durations to reduce inefficient dithering. Temporl (Biedenkapp et al.] 2021)) ) advanced this by enabling agents to determine both action and repetition duration, improving learning efficiency. However, its hierarchical structure artificially decouples action selection from duration determination Metelli
Meteli et al.] (2020) instead study action persistence in batch reinforcement learning, treating repetition duration as an environmental
"ode available https///gttaaa.co////aaaaaaaaaaa
Sequence
2

---

Published in Transactions on Machine Learning Research (12/2025)
configuration parameter that modifies the effective control frequeney. They introduce Persistent Fitted Q-Tteration (PFQT), which learns value functions for different. fixed repetition factors from a single batch of experience and provides theoretical bounds on the performance loss induced by persistence as well as a hcuristic for selecting an approximately optimal repetition duration. Other recent work includes multi- Trequency control, where Lee et al. (2020) introduce Action-Persistent Actor-Critic (AP-AC) to handle cases where different action dimensions persist for different durations (e.g., repeating leg movements every two steps while repeating arm movements every four steps). Additionally, [Sabbioni et al. (2023) introduce the All-Persistence Bellman Operator to enable agents to learn from experiences collected with varying repetition durations simultaneously, by decomposing and bootstrapping across different temporal scales. They provide theoretical guarantees of convergence and empirically demonstrate improved exploration and sample efficiency through their ability to reuse experience collected at any repetition duration. Despite showing promise, these studies share a limitation: temporal abstraction is achieved solely through simple repetition of primitive actions, without generating coherent, variable-length action soquences.
2.2 Multi-Step Action Sequence Generation
Beyond single-action repetition, several methods focus on generating and partially committing to multi- step action soquences an approach often called action chunking, where multiple consocutive actions are predicted simultaneously rather than one at a time. The authors of Zhang et al.] (2022a) introduced generative planning method (GPM) that produces multi-step plans. Since GPM is trained by maximizing value, the plans generated from it can be regarded as intentional action sequences to reach high-value states and improve sample efficiency. PrAC (Coad et al. 2022) enables agents to generate n-step plans and commit to them while being predictable, balancing adaptability and control stability. The work of [Sanum et al. (2033) incentivizes compressible action sequences by integrating sequence priors, while Patel & Siegelmann (2024) introduced model-based sequence RL framework (SRL) reducing decision frequency through action chunking. Action chunking approaches have also been explored in imitation learning, where methods like
Action Chunking Transformers (ACT) (Zhao et al., 2023) and Diffusion Policy (Chi et al. 2025 generate multi-step behaviors from expert demonstrations.
Despite recent progress, most existing methods for generating multi-step action sequences still face major limitations. Many rely on computationally intensive processes such as iterative rollouts, autoregressive decoding, or model-based simulation, which can be slow and inflexible [Li et al. 2024 Li. 2023: Zhang et al. 2025). For example, methods like PrAC and SRL use learned environment models for both planning and training, adding extra model-based complexity (Kumar et al.) 2024 Luo et al., 2024). To stay adaptable, some approaches also use external switching mechanisms mid-sequence ro-planning, as seen in GPM and PrAC. This treats long-term planning as an add-on to a step-by-step framework rather than as a core design principle. As result, sequence generation and evaluation are often optimized separately, which can lead to poor credit assignment (Dai et al.) 2018). One case is the use of handcrafted regularization, such as rewarding shorter or more "compressible" sequences (Saamum et al.] 2023). However, when objectives are split in this way, it becomes unclear whether failures come from a bad plan or from breaking the secondary constraint, making end-to-end training harder and reducing stability during execution.
2.3 Temporal Abstraction Using Hierarchies and LLMs
Hierarchical methods have advanced multi-action decision-making through skill discovery and sequencing. TACO-RL (Roscte- Beas et al.) 2202)) Icarns latent skills from unstructured data for long-horizon tasks.
ASPiRe (Xu et al., (022)) accelerated RL by combining specialized skill priors. The work of Vezzani et al.)
(2022) introduced a skill scheduler sequencing pretrained skills, while SHRL (Wang et al. 2023) combined
high-level policies with low-level skills for visual navigation. These approaches improve temporal abstraction
by leveraging reusable skills rather than primitive actions.
Recent works have leveraged large language models and value- based reinforcement learning methods with action discretization for action sequence generation. CQN-AS (Sco & Abboel (024) proposod value- based algorithm learning precise value functions from noisy action sequences. AlphaMaze (Dao & Vu 2025) improved LLMs' spatial reasoning by combining supervised fine-tuning with policy optimization.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Seste-e Decoder
Critic
Sequenoe Embedding
Decoder
Buffer
Proto Sequence
Action Sequence
Action Sequence Q-value
ApM
Environment
(1)
Sequence
(2)
Inference
Training
Figure 1:
The three components of our proposed approach: (1) The Actor encodes the current state to produce a proto-sequence embedding. (2) The Decoder translates this latent embedding into a variable- length discrete action sequence. (3) The Critic evaluates the state-action-ssqqence pair and assigns it a Q-value representing the expected cunulative reward (3). During inference, only the actor and decoder components used.
Our work draws inspiration from Dulac-Arold et al. (2055): who generated embedding representations of prolo-sequences mapped to discrete actions. In GPS, we propose key improvements: our approach is fully differentiable and trainable end-to-end, unlike Dulac-Arnoll et al. (2015) whose k-nearest neighbors mapping broke the computation graph. Additionally, by using a VAE- based decoder instead of clustering, we automat- ically create sequence representations without manual embedding design. This enables efficient generation of coherent, variable-length action sequences that extend beyond simple repetition skill sequencing.
3 Method
Overview. Our proposed approach is presented in Figure 1 GPS consists of three components: Actor; Proto-Sequence Decoder (PSD), and Critic. The Actor receives the current state as input, and produces a proto-sequence - an smbedding-based reprosentation of sequence of actions. The PSD recoives the proto- sequence as imput, and translates into a discrete set of actions (e.g., at, (e,lt,,,... L.),,,, which are then executed sequentially by the agent. Finally, our Critic receives the sequence and predicts the expected cunulative reward obtained from its execution.
GPS differs from previous studies in several important aspects. First, unlike previous studies (Dulac-Arold et al. 2015), it is end-to-end differentiable and does not require training workarounds. Secondly, our VAE decoder produces more diverse and flexible action sequences than autoregressive or model-based approaches, and also does so an efficient, one-shot manner. Thirdly, sequence generation and evaluation are learned jointly, without regularization or switching mechanisms, thus improving credit assignment. Finally, by committing to the entire sequence (unlike the frequent re-ovaluation of ZZhang ct al. (2022a)) we reduce execution overhead and increase behavioral predictability by forcing GPS to learn robust policies.
Another important aspect of our proposed approach is its ability to generate action sequences that differ from those on which it was trained. By creating novel sequences, GPS does not simply "memorize" fixed set of actions, but is able lo generalize to larger action spaces. We elaborale on GPS's capacily to produce novel sequences in Section Despite using a frozen decoder pre-trained on a limited set of sequences, GPS can generate action sequences that differ from those the decoder's training data. The ability to generalize is important our method, since sequence memorization not applicable to domains with large action spaces. We elaborate GPS's capacity to produce novel sequences in Section 5.1

---

Published in Transactions on Machine Learning Research (12/2025)
3.1 The Actor
The Actor serves GPS's policy network. Given state 8t, the Actor analyzes the input and outputs proto-sequence embedding k = T&. (st), where H and are the parameters of the Actor's neural network and the current policy, respectively. The proto-sequence k is latent embedding of a sequence of actions, represented as a vector in the embedding space E R where d is the dimensionality of the embedding space. This representation provides our Actor with significant flexibility, as it can create action sequences of varying length using fixed-size representation.
The proto-sequence is next used by the PSD to produce discrcte sequence of actions, and this sequence is evaluated by the Critic (Section 3.3 The parameters 0 of the Actor are then updated using an actor-critic approach analogous to the Deep Deterministic Policy Gradient (DDPG) algorithm (Lillierap et al., 2016). leveraging the learning signal provided by the Critic. Specifically, the actor's parameters 0 are adjusted to produce proto-sequence embeddings & that maximize the expected cumulative reward estimated by the critic, Qoo a). This optimization is achieved by updating 0r to minimize the negative Q-value provided by the critic- QQpa go- ( (s))),))),, using backpropagated gradients from the output of the critic network Qoo. These gradients pass through the decoder network 90- and subsequently through the actor network TTa enabling the update of the latter's parameters 0=.
3.2 The Proto-Sequence Decoder
The goal of the PSD is to translate the latent proto-sequence & generated by the Actor into a sequence of executable actions in the original action space far, A1+1..... a++L} E A. We define the PSD as function 9o- : K + A'Lmaz, parameterized by 0", where A' extends A with an EOS token to handle pariable-lenglly sequences within a fixed-length format, padding shorter sequences as needed. This function maps from the latent proto-sqquence space K to sequences of fixed length Imaz-
We Variational Autoencoder (VAE) (Kingma & Welling 2013) as our PSD. We train the architecture a diverse set of synthetic action sequences of varying lengths. For detailed information on the generation process of these sequences, see Appendix M.1. After training, we discard the VAE's encoder and retain only the learned decoder network ge-. The decoder is integrated into our agent architecture, transforming the Actor's latent proto-sequence embeddings into sequences of discrete actions. GPS will then execute the full sequence, without changes or early stopping. We chose VAE for its efficiency, ability to generate complete sequences in a single step, and its structured latent space that enables smooth interpolation and principled probabilistic modeling.
We pre-train the PSD and keep its parameters fixed while jointly training the actor and critic. We had several reasons for this separation: a) simplicity and robustness separately training the Decoder reduces the number of moving parts in our architecture, enabling faster training. Separate training also prevents a 'moving target' scenario, where multiple components adapt at the same time, compromising training stability (Sulton et al. 1999); b) Diversily and prevention of mode collapse: by training the Decoder separately, we have full control of its training set. We use diverse training set that ensures that the Decoder can generate action sequences of all types. In addition, by freezing the Decoder after its training, we prevent scenario where this component "forgets" how to generate specific actions during the other components' training; c) Transferability- - by ensuring our Decoder can generate action sequences of all types, we can train it once and then use it in all tasks with this particular action space. It should be noted that training GPS end-to-end achieves comparable final results, with some differences in path optimality and efficiency (see Section G in the Appendix).
3.3 The Critic
The goal of our Critic is similar to the role of the critic in an actor-critic architecture. The Critic receives the current state s, and the one-hot encoded discrete actions sequence A (a,, 4.41,..., produced by the PSD. It then attempts predict Qoo (st, A), which represents the cumulative discounted reward obtained
5

---

Published in Transactions on Machine Learning Research (12/2025)
by executing A and following the policy after the end of the sequence:
T-1
Qoa (st A) Ro E.,
Σ T + V" (st+r))
where V the value function under policy म. 8 are the Critic's parameters, and L len()A denotes the effective length of the action sequence A.
The Critic's parameters 0 are updated by minimizing the Mean Squared Error (MSE) loss against Temporal Difference (TD) target Ye:
L(09) = (ss, rewaads,, [[Q[s), yc)
The target Yt is constructed from the sum of discounted rewards (A) obtained by executing sequence A, and the discounted value of the subsequent state S1+L, estimated using target Actor (Actorcarget) and target Critic (Qtarget) networks:
y/ (A) + Qtarget (sL+ (SD(ootoraagee ( (8++1));;
This update mechanism, which relies on TD errors and target networks, is characteristic of many actor-critic algorithms, and shares similarities with methods such DDPG (Lillierap et al., 2016). While the Critic learns to accurately predict Q(st, A; A9) the Actor is trained to produce proto-sqquences that, when decoded by the PSD, maximize this predicted Q-value.
3.4 Training Set Augmentation Using Sequence Subsets and Inference
To enhance learning efficiency and improve credit assignment, our training procedure leverages reward in- formation from subsequences of each executed action sequence. For each sequence A = (at,..., ax+L) of length L, we extract transitions corresponding to multiple contiguous subsequences (a....., ay) where S < < < + For each such subsequence starting from an intermediate state 8,, we calculate the accumulated discounted reward obtained during its execution. This process effectively generates multiple learning samples of varying lemporal lengths from a single interaction sequence, enriching the training data.
The subsequence extraction strategies for these state-subsequence-rrwaad tuples, which we add the replay buffer, include two primary approaches: (1) prefix extraction, which fixes the starting state while varying the end point, and (2) sullix extraction, which fixes the goal state while varying the starting point. This bidirectional approach diversifies the replay buffer with different time scales and enables the Critic to learn .j)) for sequences of different lengths concurrently. As shown in our analysis
value estimates Qao (&;, in Section 5.2, these extraction strategies significantly accelerate learning and improve overall performance. It is important to note that during inference (test time), our architecture does not utilize the Critic component, since no training takes place. Instead, the Actor and PSD produce the action sequence, and the latter is executed in full.
4 Experiments and Results
4.1 Evaluation Environment
Mazes foundational benchmark in DRL research, commonly used to evaluate an agent's ability to per- form complex sequential decision-making and navigation tasks. Their structured yet variable environments provide a controlled setting for evaluating generalization, exploration, and memory, which are central to DRL performance (Pašukonis et. al. 2023). We use four types of mazes in our evaluation:
Empty. These mazes have walls obstacles, except for their boundaries.
Sparse Obstacles. This setup has randomly placed obstacles in K% of the cells of each maze (c.g..
15%).

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: The setup and properties of the mazes used in the evaluation.
Dist. from
Train Optimal
Val Set
Optimal
Set
Optimal
Environment
start to goal
Set Size
Avg. Path
Size
Avg. Path
Size
Avg. Path
8x8
- 14]
100
5.14
100
5.49
1000
5.31
16x16
16
100
18.01
18.0
1000
17.98
16x16 obstacles 15%
20 30
100
21.02
100
21.35
210
21.31
16x16 obstacles
16x16
ronms
20 30
100
20.93
100
21.01
585
21.02
16x16 corridors
12.84
12.76
545
13.14
24x24
20 30
100
23.39
100
23.26
1000
23.56
24x24
obstacles 15%
10 20
100
15.04
100
14.58
1000
14.73
24x21
obstacles 25%
110
15.05
1000
15.11
Rooms. This setup consists four large rooms with small doors between them. randomly placed obstacles in 5% of open cells.
We also add
Corridors. These mazes have only narrow corridors for the agent to navigate.
Our evaluation uses fully observable MDPs where the agent has complete visibility of the entire maze grid, including its position, goal location, and all obstacles.
Similarly to Dao & Vu 2022), we use an LLM to produce the code used in our maze generation. Our code, as well as the mazes generated for our evaluation, are available in the appendix. All information on our generated mazes is presented in Table [ For each maze size and type, the table presents: a) the sizes of our training, validation, and test sets, b) the range for the distance between the start and goal positions, and c) the average length of the optimal path.
4.2 Baselines & Evaluated Methods
We evaluate two versions of GPS and three discrete-action baselines: DQN, TempoRL, and DAR. Full implementation details of our approach are included in the Appendix.
GPS: Our primary approach generates action sequences using VAE-based decoder with Gumbel-Softman sampling. This stochastic mechanism applies a lemperature-connrolleed softmax to produce action distribu- tions that maintain differentiability while approximating discrete samples. The Gumbel-Softmar technique creates a relaxation of categorical distributions that preserves gradients for backpropagation, facilitating end-to-end training of our actor-critic architecture.
GPS-D: A deterministie variant of our approach that uses argmax operations with straight-through esti- mator in the decoder instead of Cumbel-Softtax sampling. This version produces consistent, deterministic action sequences for each proto-sequence embedding.
DQN: Deep Q-Network (Mnih et al. 2013) is a foundational model-free DRL algorithm that learns state- action values. DQN utilizes experience replay and a target network to stabilize its learning.
DAR: Dynamic Action Repetition (Srinivas al 2017) extends discrete action spaces by repeating original actions at varying rates. DAR enables the agent to select different levels of temporal control, allowing for some action abstraction.
Temporal Reinforcement Learning [Beddenkapp et al. 2021 introduces a proactive approach, where the agent selects both an action and its duration. TempoRL employs a hierarchical structure with a behavior policy for action selection and a skip policy for duration, enabling more fine- ane-grained temporal abstraction and efficient exploration.
TempoRL:
4.3 Experimental Setup
State and action representations. We represent the state using a tensor of shape (N, M,3),,, where and M are the height and width of the maze grid. The channels use similar encoding to that of MiniGridLabrary (Chevalier-Boiste Boisvert et al. 2003):: a) Object type: identifies all environmental elements including walls, empty spaces, agent position, goal location, and starting position; b) Object color: provides distinguishing colors
7

---

Published in Transactions on Machine Learning Research (12/2025)
Empty
Obstalces
Rooms
Corridors
Figure 2: Examples of our generated mazes (16x16). We use four maze environments (left to right): EMPTY open space; 15% Obstacles random obstacle placement; ROOMS structured rooms with doorways; CORRIDORS narrow paths requiring precise navigation. Red squares mark start positions, green squares mark goals, and yellow circles show the current agent location.
for the start position, goal location, and current agent position; c) Placcholder channel: consistently set to 0, maintaining compatibility with the MiniGrid format. Our discrete environment supports the four basic actions up, down, left, right represented as four-entry one-hot vector
Reward function. We define the reward function identically across all methods as follows:
if goal is reached
R
1/1maz if valid action is executed
3/lmaz if an invalid action is executed
where max is the maximal start-goal distance (see Table acting regularizer. When GPS executes sequence of length L, sum these Markovian step rewards: Reotal (rr)) for = I.... This reward structure encourages goal achievement while penalizing excessive steps and invalid actions, ensuring fair comparison across all methods.
Evaluation metrics. We use three evaluation metrics:
Average success rate (ASR): the percentage of episodes evaluated where the agent navigates successfully from start to goal position within predefined number of steps.
Path efficiency ratio (PER): for successfully completed episodes, we calculate the ratio between the episode length and the optimal (minimal) length:
Topt
PER
Lepisode
Sequence Generation Frequency (SGF): This metric reflects how often the agent generates new action sequence. calculated as the average number of times the Actor is invoked per evaluation episode. Lower values suggest. the agent relies on longer-term proto-sequences before needing to generate new sequence. This metric relevant to GPS, DAR, and TempoRL, baselines.
These metrics are complementary, as they allow us to evaluate the policy's effectiueness, efficiency, and decision frequency, under identical reward optimization across all approaches.
Neural architecture sctup. All models use a shared CNN feature extractor followed by method-specific linear layers. DQN outputs Q-values for cardinal directions, DAR expands this for multiple repetition rates, and TempoRT. implements a branching architecture for action and skip duration. In GPS, actor and

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: ASR Performance at Different Training Steps. Values represent mean + standard deviation over five sceds.
DP50
140
Txy
67%
CPSD
140
14y
12%
100
DP5D
675 -15
800
110-E00
1000
334
Bock
038
040 61
108-5:0
18
100.
Mink
196
04-40
05--115
38
03
00
03.55
652
600833 811
48
08:00
02--0
007-42
056-117
185-11
0:0..00
02.56
Note: The ASR each algorithm at specific training step intervals. gray background indicates the highest ASR achieved that environment across all steps and algorithms. A yellow background indicates the highest ASR within that specific step interval (excluding any cell already marked gray). 1 indicates unavailable data.
critic networks use separate but identical CNN architectures. The actor produces a 16-dimensional proto- sequence embedding, which the decoder converts into action sequences through a multi-layer network with normalization. Full details are in the appendix.
Hyperparameters & Hardware. Unless otherwise noted for specific ablation studies, experiments were conducted using a common set of key hyperparameters, summarized in Tables 14 and 22 in the appendix. We selected the values based on preliminary experiments and common practices. All experiments were conducted on a system running Red Hat 5.14 with x86 64 architecture. We used an NVIDIA RIX 2080 GPU with 8GB of VRAM.
Training Protocol & Model Selection. We used different training setups based on maze size and type. Detailed step counts are in Table 2 Model selection for final testing used the checkpoint from each run yielding the highest average success rate on a held-out set of validation enviromments during training. Exploration employed an e-greedy strategy, with random sequences being sampled from the same pool of 400 synthetic sequences used to train the PSD (see Appendix M. for generation details). This ensures exploratory sequences follow similar structural patterns to those the decoder was trained to generate. While GPS incurs computational overhead per decision, our analysis shows this is often offset by reduced decision frequency and faster convergence (see Appendix for detailed trade-offs).
4.4
Evaluation Results
4.4.1
Evaluating the Average Success Rate (ASR).
The results of our evaluation are presented in Table 2 GPS putperosm the baselines in the majority of tested maze configurations. Below we present several key observations:
Ability to learn, converge quickly, and generalize. GPS demonstrates high sample efficiency and rapid convergence, consistently achieving the highest ASR 100k steps across all environments. GPS achieves an ASR of 0.74 on the 16x16 empty maze with only 100K training steps, compared to DQN (0.19) and TempoRT. (0.36). This early advantage is even more pronounced in larger mazes: in 24x24 with 15% obstacles, GPS reaches 0.25 at 100k steps while all basclines remain below 0.04. GPS also substantially outperforms the baselines with an ASR of 0.77 for 24x24 empty maze at 500k steps, while the top baseline TempoRL only reached 0.28. These results support our hypothesis that modeling action sequences rather than individual actions enables more strategic exploration and faster discovery of goal-reaching strategies. GPS primarily learns to generate sequences that move the agent in the correct general direction toward goals, allowing progress in unseen environments even without. perfectly optimized paths. The deterministic variant. GPS-D, also shows strong performance, supporting the robustness of the proto-sequence concept.
Scalability and superior ability to solve complex environments. The performance gap widens in larger environments. In the empty 24x24 maze, GPS significantly outperforms baselines, achieving an almost perfect ASR of 0.94 at 1.5M steps compared to DQN (0.28) and TempoRI. (0.52). In the complex 24x24 environment with 15% obstacles, GPS achieves an ASR of 0.83 after 1.5M steps, significantly outperforming the top performing baselines. Even in the most difficult. 24x24 environment. with 25% obstacles, GPS
9

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Comparative Performance Analysis: Convergence Speed and Efficiency Metrics. Values represent
mean + standard deviation over five seeds.
Evvinonnci
JSHI Geneergeo... Step
PER
SG2
1828
147%
Terrpell.
DAI
Teerpeltt.
Terspottl.
DAR
51.04
-41.33
0.00 0.12
0.51
0.00
0.00
0.00
5.26
7.68
12.70
7.50 0444
tacth_.tee 18%
8008
-33.38
-4.35
-3.333
0.38
6.05
0.78 1110
1136 0.00
1:50 -0111
0/81 0.1
16.06 RL5
11:01 11.41
18:34
10.35
14.251
035
038 16.00
2/15
0.38 0.82
17:77 6.87
24/04
30.18 2:
N/M
16c1o more
51.5M
12.33M
23.05
080 0.19
016 0.00
0.80 0.01
0.36 0.81
N/R
15 0.51
12:45
2.22
12:77 -3000
N/R
1006
5006
2-13
213
0.0
0.00
0.78 1L06
0.07
0.01
[.90 0.00
6,00 000
5/02 0:0
7:25
0.14
= 0.31
6,90 1.09
2te21
21.50 -1.804
13M
23.30
54.55
51.30
8/11
0.77
1 0.00
0.00
0.82
0J00
1/11
1200 4 1:79
11,80 L 4:48
16.57
à: 1.36
N/36
Lite 15%
551.001
51.35
53.390
54.33
51.30
8/11
0.59 1 6.03
2/14
3/11
2/11
14.47 1 6.30
2/31
2/16
N/3
olte 15% (-1
534
21.55M
1:1.
21.5M
N/R
X/R
X/R
3/R
N/R
N/R
N/R
X/R
N/R
Note: Wo present four key performance metrics. The first column shows training steps required to achieve a 90% success rate (lower is better), with highlighted values indicating the fastest convergence. Path Efficiency Ratio (PER) measures trajectory optimality (higher is better, max=1.0), with bold values showing best performance. Sequence Generation Frequency (SGF) indicates the average number of decision points needed per episode (lower generally indicates better temporal abstraction). '.' indicates N/A and 'N/R' for ASR < 0.5, indicating insufficient success rate for meaningful PER/SGT evaluation.
maintains a relative advantage, significantly outperforming DQN, the top bascline, with an ASR of 0.49 versus 0.10.
Performance on medium-sized and structured environments. In the structured "rooms" environ- ment, GPS achieves an ASR of 0.85 at 1.5M steps, significantly outperforming DQN (0.64) and TempoRL. (0.67), while DAR shows a significantly lower score (0.13). This demonstrates GPS's effectiveness in navi- gating complex structured layouts that require coordinated multi-step strategies. In the challenging 16x16 16 maze with 15% obstacles, GPS achieves strong performance with an ASR of 0.94, showing significant. im- provements over DQN (0.84), TempoRL (0.8), and DAR (0.58). In the most challenging evaluation setups with 25% obstacles, GPS achieves an ASR of 0.57 in the 16x16 maze, which is lower than TempoRL (0.73) and DQN (0.72). These results, along with the relatively slower convergence and lower final ASR observed in rooms environment (reaching 0.85 ASR at 1.5M steps compared to near-perfect performance achieved in 100-900k steps in simpler configurations), suggest that extremely dense obstacle distributions and highly constrained spatial layouts can limit the advantages of sequence-level decision making when precise naviga- tion is required. However, our approach significantly outperforms these baselines in the larger 24x24 setup with the same obstacle density, achieving an ASR of 0.49 compared to TempoRL (0.08) and DQN (0.10). These results suggest that the relative performance in dense obstacle configurations is influenced by the interplay between maze scale and obstacle density rather than obstacle density alone.
Performance characteristics of the basclines. The baseline methods demonstrate varying strengths across different enviromments. DQN achieves competitive performance several cases, particularly in medium-complexxtt environments like 16x16 with 15% obstacles (0.84), 16x16 with 25% obstacles (0.72) and structured rooms (0.64). TempoRL shows strong performance in specific configurations, excelling in corridors (ASR of 0.89) and achieves solid results in 16x16 obstacle environments (ASR of 0.73-0.80). DAR consistently underperforms, with ASR declining from 0.74 in 8x8 mazes to 0.66 in 16x16 and 0.29 in 24 empty mazes. Performance degrades further with obstacles, typically ranging between 0.05 0.16, though it reaches 0.58 in the 16x16 maze with 15% obstacles. The structured 'rooms' environment presents simi- lar challenges, with performance at 0.13, suggesting its action repetition strategy is ineffective for complex navigation tasks.
GPS's strong performance stems from operating in the space of action sequences rather than individual actions, enabling more strategic exploration and the discovery of horizoo rewards that would be difficult to find using single actions or simple repetition methods. Note that for TempoRL and DAR, which might require more extensive training to converge optimally, we observed improved performance with larger training datasets (see Table 4 in the appendix), though computational constraints limited further exploration. While DAR also showed improvements under these conditions, it still achieves relatively lower results in complex structured environments, suggesting that simple action repetition may have inherent limitations for certain maze types.
10

---

Published in Transactions on Machine Learning Research (12/2025)
4.4.2 Evaluating the Path Efficiency Ratio (PER).
The results of our evaluation are presented in Table 3 PER is calculated at the final training checkpoint using the total time steps per environment detailed in Table I We report PER for GPS, GPS-D, and two baselines introduced in Section 4.2 allowing for direct comparison across methods. A key GPS characteristic is Self-Correction Through Sequential Decision Points. GPS can adjust its course at subsequent decision points without requiring an initially perfect action sequence. This sequence-level closed--oop contro) enables course corrections while retaining the benefits of temporal abstraction (empirical validation of this self-correction capability is provided in Appendix [E] demonstrating GPS's ability to generate correction sequences with high success rates (62.4-99.6%) immediate correction) across diverse maze environments). Leveraging this capability, GPS adopts strategy of Trading Path Efficiency for Robust Navigation, prioritizing directional correctness over strict path optimality. This approach develops more transferable navigation skills - particularly evident in larger or more obstaclo-dense mazes explaining cases where PER is lower despite higher ASR and faster convergence (see Tables 2 and 3
GPS-D consistently yields higher PER than GPS across all environments. For example, in the 16x 16 empty maze, GPS-D achieves PER of 0.99 versus GPS's 0.85; in the 24x24 empty maze, GPS-D reaches 0.99 compared to 0.78 for GPS. GPS's Gumbel--Sftmax sampling introduces stochasticity that enables broader exploration but can cause path deviations, while GPS-D's deterministic argmax decoder produces more consistent trajectories, trading exploration advantages for improved exploitation. DQN and TempoRL achieve higher PER than GPS in most environments (e.g., DQN reaches 0.98 and TempoRL 0.96 in 16x 16 with 15% obstacles, compared to GPS's 0.74), and comparable or slightly lower PER than GPS-D. However, PER only meaningful when the agent successfully reaches the goal. In many environments where basclines achieve lower ASR, their high path efficiency applies to only small fraction of episodes. Overall, DAR generally shows lower PER across all environments.
These findings reveal a conceptual trade-off in our approach: GPS's stochasticity boosts exploration and rapid convergence to high ASR, while GPS-D's determinism excels in path efficiency once a good policy is learned. This sequence-level exploration-eeplooiation trade-off offers practitioners a choice between priori- lizing solution discovery (GPS) or execution efficiency (GPS-D) based on their specific requirements.
4.4.3 Evaluating Sequence Generation Frequency (SGF).
The results are presented in Table 3 where lower values generally indicate superior temporal abstraction due to fewer policy invocations per episode. For methods reporting SGF, our approaches GPS and GPS-D demonstrate competitive performance across environments. In the 16x16 empty maze, GPS-D achieves an SGF of 7.68, comparable to DAR (7.99) and substantially lower than TempoRL (12.73), while GPS records 8.76. However, in the 16 x 16 corridors environment, DAR (6.30) outperforms GPS-D (7.23) and GPS (9.02). Despite this mnvironment-deeeddeer GPS-D's 11.80 SGF in 24 x 24 empty mazes versus TempoRL (16.57), demonstrating effective generation of extended proto-sequences.
variation, our methods often operate with limited interventions-such as
GPS-D's generally low SGF combined with its high PER indicates capability for efficient, strategic trajectory generation through robust behavioral patterns, making it ideal for sccnarios requiring predictable exccution or constrained resources. GPS offers compelling trade-off with competitive PER and favorable SGF compared to TempoRL (e.g., 8.76 vs. 12.73 in 16 16 empty; 15.96 vs. 17.77 in 16 x 16 rooms), alongside faster convergence and higher Average Success Rate as discussed in Section 4.4.] It balances path efficiency, sequence compactness, and learning speed effectively.
In conclusion, SGF analysis supports our sequence-generation paradigm's effectiveness for temporal ab- straction. GPS-D provides efficient, long-term utility with fewer, optimal decisions, while GPS balances competitive SGF, good PER, and rapid ASR. The choice between them depends on application priorities: efficiency vs. predictability adaptation vs. broader performance.
11

---

Published in Transactions on Machine Learning Research (12/2025)
16x16
16x16
Actor: [512.32]
Actor: [512.128,32]
1.0
1.0
0.6
0.6
ASR
Prefixes_Sep TRUE, Suffixes_So TRUE
Prefixes_Segp TRUE, Suffixes_See S_eq: TRUE
0.2
Prefixes_Sede Seq: FALSE, Suffixes_ Seq: TRUE
0.2
Prefixes_ Seq: FALSE, Suffixes, Seq: TRUE
prefixesSSqq TRUE, Suffixes_Seqp FALSE
Prefixes_Sepp TRUE, Suffixes_sse Seq: FALSE
Prefixes_Seds Seq: FALSE, Suffixes_Sers Seq: FALSE
Prefixes_See Seq: FALSE, Sullixes_Soi _Sq; FALSE
16x16 obstacles 15% Actor: [512,32]
16x16_ obstacles [6x16_obstacles Actor: [512, 128,32)
15%
0.8
0.8
ASR
0.4
0.4
Prefixes, Seq: Suffixes_Serr Seq:
Prefixes Seq: TRUE, Suffixes Seq:
Profixes, Seq: FALSE, Suffixes, TRUE
Profixes, FALSE, Suffixes, Seq: TRUE
Prefixes_Sede Seq: TRUE, Suffixes_Sece FALSE
Prefixes_Seti Seq: TRUE, Suffixes_Seet Seq: FALSE
Pretixes Seq; FALSE, Suftixes Seq: FALSE
Profixes Seq: FALSE, Suftikes Seq: FALSE
0.0
0.0
100k
200k
300k
400k
SOuk
600k
700k
BOOK
900k
IM
100k
200k
300k
400k
500k
600k
700k
Book.
900K
IM
Training Steps
Training Steps
Figure 3: Impact of Subsequence Buffering Strategy on Average Success Rate.
5 Analysis and Discussion
5.1 Analyzing GPS's Ability Generate Novel Sequences
While provided our Decoder with diverse training sct, the latter did not include all possible trajectories. Our reasons were twofold. First, while including all possible action combinations was feasible in our (relatively small) action space, doing the same for larger, more complex action spaces would be infeasible or very costly. secondly, we wanted to evaluate GPS's ability to generalize and produce trajectories that were not in the training set. We consider the ability to generalize important, because the lack it may limit the usefulness of our approach large action spaces.
As described in Appendix M.I the PSD was pre-trained on a set of 400 synthetic sequences generated according to simple, common-sense heuristics for navigation tasks: a) each sequence contained most two distinct action types. b) Actions of the same type appeared in contiguous blocks (e.g., "up, up, left" allowed; "up, left, up" disallowed). e) No immediately contrasting actions were allowed (c.g., "up, down" prohibited). d) Maximum sequence length was capped al Lmax (shorter sequences permilled). e) Avoidance of loops. After full training, we gathered 15 action sequences by sampling states from the GPS replay buffer and generating the corresponding action sequences through the actor and PSD. Eleven of these did not appear in the PSD's training set and were not fully aligned with least one of the navigation patterns described above. Using the encoding up-+0, down-+1, left->2, right-3,, the novel sequences were:
12

---

Published in Transactions on Machine Learning Research (12/2025)
16x16 Mazes
24x24 Mazes
0.6
16x16 obstacles_24 16x16_obstacles,
25%%
24X24_ wwwpaclee_2saa555 25%%
16x16 obstacles_ 15%
24x24 Sisslatles_15%
16x16 rooms
24x24
0.4
16x16
16x16 _corridors
0.2
ASR
ASR
0.0
-0.2
-0.4
100k
200k
300k
200%
Sook
Gnok
7ook
anok
gook
1.1M
1.2M
1.3M
15M
100k 200k
20ok
sook
soot
600%,
100%
gook
gook
1.M 12M
1.M
IAM
1.5M
Training Steps
Figure 4: Impact of Actor Network Scaling Average Success Rate (A ASR) in Mazes.
11,,,,,1], [1,, 1,,,0, (0,0,3, 1], [3,,,2, 3, 0, 11,,2,3). [2,0,
[0, 3,,0,3,3,0.3] (,,3,12 1,
These results show that GPS can create new action sequences seen during training because works in a structured embedding space. In this space, sequences with similar structures grouped together, making it possible to blend known patterns and generate new ones, as shown in Figure t in the appendix.
5.2 Sequence Subsets Augmentation
We investigate our subsequence buffering approach (Section 3.4 varying end point) and suffizes (fixed goal state, varying starting point). Figure shows that the baseline without subsequence buffering (red) consistently learns most slowly and often converges sub-optimally, while all subsequence buffering variants substantially improve learning efficiency. Using prefixes and suffixes simul- tancously (blue) generally produces the most rapid learning, though the suffix-only configuration (orange) performs nearly well, suggesting backward sampling provides particularly valuable learning signals. The prefix-only approach (green) typically shows slower convergence than other subsequence methods. These performance patterns remain consistent across different maze structures and actor networks.
implemented through prefizes (fixed start,
5.3 Performance Under Stochastic Dynamics
Up to this point, we conducted our evaluation deterministic setting. We now evaluate GPS's robustness in two stochastic settings, which are more challenging our approach because of its "commitment" to fully implementing its generated trajectories:
"Sticky" actions. We used the well established "sticky actions" mechanism (Dabney al, 2020 with 25% probability of repeating the previous action instead of executing the planned
Random actions. In this setup, which is also common in the literature (Luu al. 2024 each action has 25% probability of being replaces with random action.
13

---

Published in Transactions on Machine Learning Research (12/2025)
These two setups introduce temporal correlations and execution uncertainty that challenge sequence- based methods, as errors can potentially cascade throughout multi-step sequences.
Results for the sticky actions setup show that GPS maintains strong performance under the sticky actions setup, achieving near-perfect or perfect success rates across most tested environments while preserving com- petitive convergence times and reasonable path efficiency. In the random actions setup, which proved to be more challenging to all evaluated algorithms (see Appendix [ for details), GPS is again the top performer. Moreover, while all evaluated algorithms suffer from degradation in their performance in this setup, GPS's relative degradation is the smallest. This result is noteworthy, since we would expect DQN, with its single actions, to be the most robust.
The robust performance under stochastic conditions provides evidence that GPS's sequence generation ap- proach may confer resilience beyond deterministic settings. Operating at the sequence level appears to offer some natural buffering against action exccution uncortainties, though comprehensive evaluation across diverse stochastic environments would be needed to fully establish this robustness. Detailed experimental details and quantitative results supporting this analysis are presented in Appendix
5.4 Performance Under Partial Observability
Beyond stochastic dynamics, we evaluated GPS's robustness under partial observability constraints, where agents only perceive a 7 x 7 local view window centered on their current position rather than the full maze structure. This setting is particularly challenging for sequence- based methods as it requires generating multi-stop action sequences without complete environmental information, testing whether our approach can maintain effective navigation strategies when operating with limited sensory input.
The results reveal corridor environments, GPS demonstrates strong performance, maintaining advantages over DQN in empty 16x16 maze at convergence (0.71 vs 0.60 ASR at 1M steps). Corridor environment show more pronounced advantages, with GPS achieving 3x improvement over DQN (0.79 vs 0.26 ASR at 1M steps), suggesting that multi-step sequences provide inherent benefits for maintaining directional consistency despite limited visibil- ity. However, moderate path efficiency ratios (PER 0.57-0.61) indicate suboptimal navigation compared to DQN's near-perfect efficiency, suggesting that while GPS reaches goals successfully, the generated sequences include unnocessary detours when operating with incomplete information. Notably, in larger 24x24 empty maze, performance converges more closely between methods (0.31 vs 0.30 ASR). This convergence reflects the compounding challenge of limited visibility over longer navigation distances while the 7 x observation window covers reasonable fraction of 16 x 16 mazes, it represents much smaller proportional view in 24 x 24 environments, diminishing GPS's dircctional consistency advantages as both methods struggle with the fundamentally limited information available for loug-distance navigation.
peooomenn-deeeddenn performance patterns under partial observability. In empty and
In obstacle-dense and rooms environments, GPS faces increased challenges despite consistently outperform- ing DQN. In 16 x 16 maze with 15% obstacles, both methods struggle to achieve high success rates (GPS: 0.34 ASR, DQN: 0.29 ASR at 1.5M steps), while performance gaps widen in denser configurations (25% obstacles: 0.34 vs 0.21; rooms: 0.45 vs 0.28). These results represent large degradation compared to GPS's full observability performance in most configurations (typically achieving ASR 20.90), indicating that par- tial observability fundamentally challenges requence-based planning. We hypothesize these difficulties stem from GPS generating multi-step sequences without seeing upcoming barriers or doorways, committing to trajectories that encounter unseen obstacles. While DQN's step-by-step approach theoretically allows im- mediate incorporation of newly revealed features, its poor performance suggests that reactive single-action decisions may be insufficient for effective exploration under limited visibility. The elevated Sequence Gener- ation Frequency under partial observability reflects GPS's adaptive response, with the proportional increase growing substant larger in 24 x 24 enviromments, demonstrating that the impact of partial observability on sequence effectiveness scales disproportionately with environment size. No configuration reaches 0.9 ASR within 1.5M steps, indicating substantial sample complexity increases compared to full observability settings, where convergence typically occurs within up to 1M steps. Complete experimental analysis is provided in A Appendix [
14

---

Published in Transactions on Machine Learning Research (12/2025)
5.5 Impact of Actor Network Scaling
We examine the impact of actor network size (small: two-layer (512, 32); large: three-layer (512, 128, 32))
maze navigation performance, measured by AASR, (Large Small) (sce Figure 4 In simpler 16x16 mazes (empty or corridor), both architectures performed similarly. With 15% obstacles, the smaller network initially outperformed (AASR ~ -0.3 at 100K steps) before convergence at ~300K steps. In denser 25% obstacles, the smaller network outperformed from ~800K steps, peaking at AASR R -0.5 at 1.1M steps. In larger 24x24 mazes with 15% obstacles, the small network generally led, despite the large network's brief advantage (~500K steps). However, in the most complex 25% obstacles maze, the large network outperformed throughout training, maintaining AASR between 0.1-0.15. The 24x24 empty maze showed fluctuating performance with occasional spikes for the larger network around 200K and 500K steps.
These results suggest a trade-ofl: smaller networks sulfice or excel in smaller or moderately complex envi- ronments (possibly due to better regularization or more stable sequence generation learning), while larger networks demonstrate clear benefits in more complex environments.
5.6 Computational Complexity and Runtime
GPS introduces additional runtime overhead compared to baselines, as its Actor- pipeline makes each decision 1.6-3.3x slower in wall-clock time. However, because GPS operates at a sequence level and makes decisions less frequently, the effective runtime per episode is comparable to baselines in moderately complex settings (o.g., 16x16 mazes). In larger, more difficult environments, GPS shows higher normalized costs, but this is partly because it continues to solve tasks where baselines fail. Despite the per-decision slowdown, GPS converges to high success rates much faster overall (90% in 3.3 hours vs. DQN's 5.83 hours), demonstrating that its superior sample efficiency compensates for runtime overhead in practice. This makes GPS more useful in challenging tasks where training effectiveness outweighs raw inference speed. A comprchensive analysis is presented in Section in the Appendix.
6 Conclusions, Limitations, and Future Work
GPS is a novel actor-critic method that generates variable-length action sequences in single step. GPS maps state observations to proto-sequences, which are decoded into discrete action sequences. This approach enhances credit assignment and exploration in long-horizon tasks by moving beyond sequential single-action selection. Our evaluation on maze navigation lasks shows GPS surpasses leading action repetition and temporal methods across the majority of tested configurations, achieving higher success rates and faster convergence in environments ranging from simple open spaces to large complex mazes. Porformance patterns across different maze scales reveal that GPS's sequence-level approach is particularly advantageous in larger environments, where it achieves considerably higher success rates than baselines even with dense obstacles.
Although our approach shows benefits, particularly in complex environments, several limitations should be acknowledged. First, new PSD needs to be trained for each unique action space, which adds to the complexity of our approach. GPS has not been evaluated on large action spaces, so adaptations to the decoder component may be needed. The results reveal that performance is influenced by the interaction between environment scale and obstacle density. High-density random obstacles require fine- ine-grained navigation that can limit the ad advantages of sequence-level decision making. In medium-sized environments with very dense obstacles, this precision requirement offscts GPS's other advantages. In larger environments, however, GPS's strengths weigh this difficully, maintaining significant performance advanlages over the baselines. Finally, we have not yet adapted GPS to continuous action spaces.
Another important aspect of our approach is its suitability for domains with varying requirements. While GPS outperforms the baselines in the large majority of lested configurations and is able to generate solutions rapidly, the resulting solutions are not always strictly optimal. This trade-off between solution quality and computational efficiency is a well-documented phenomenon in deep reinforcement learning, where methods that prioritize rapid inference may forgo perfect optimality in favor of operational practicality and responsive-
ness. Similar observations have been reported in recent DRL research (Sohaib et al. 2025 Wu et al. 2023
15

---

Published in Transactions on Machine Learning Research (12/2025)
highlighting how haster-converging methods frequently produce solutions that, while feasible and effective in real-world applications, may not match the optimality of simpler, slower approaches such as DQN.
Future work will focus on extending to more complex domains, such as those with larger action spaces or continuous control. As part of this research direction, we plan to explore advanced initialization strategies for our Decoder, so that our approach can more efficiently explore large action spaces. Additionally, we will aim enhance GPS's effectiveness in partial information and stochastic settings, as well as in environments requiring precise local navigation. To this end, and mechanisms that will adapt the maximal length of the generated sequences based on both available information and local environment complexity. Finally, we will explore modifications to our approach that will enable us to define desired trade-ofls between path optimality and success. By doing so, our aim is to automatically adapt GPS's strategy to match the requirements of various domains.
considering the creation of a re-planning component,
References
Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. Advances in Neural Information Processing Systems, 11-23304-22255, 2023.
Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019.
André Biedenkapp, Raghu Rajan, Frank Hutter, and Marius Lindauer. Temporl: Learning when to act. In International Conference on Machine Learning, pp. 914 924. PMLR, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbecl, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097, 2021.
Maxime -heivelierBooisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular customizable reinforcement learning environments for goal-oriented tasks. CoRR, abbs/2306.1382) 2023.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Jowrnal of Robotics Research, 44110-11:10844-7044 2025. doi: 10.1177 10.1177 02789799777779422227996 URL httttttt:://do 0tg//00.117//07778649441777783
Josiah D Coad, James Ault, Jeff Hykin, and Guni Sharon. A framework for predictable actor-critic control.
In Deep Reinforcement. Learning Workshop NewrIPS 2022, 2022.
Will Dabney, Georg Ostrovski, and André Barreto. Temporally (epsilon)-greede exploration.
arXiv preprint arxi:2006.00 01782, 2020.
Zihang Dai, Qizhe Xie, and Eduard Hovy. From credit assignment to entropy regularization: Two new algorithms for neural sequence prediction. arxiv preprint 1xXi::1804.1997 2018.
Alan Dao and Dinh Bach Vu. Alphamaze: Enhancing large language models' spatial intelligence via grpo.
arXiv preprinl Xim:25002. 14669, 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXic:2010. 11929, 2020.
Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunchag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. Xiv preprint 1512: 2015.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590((847):5800-55) 586, 2021.
Liad Giladi and Gilad Katz. Feedback decision transformer: Offine reinforcement learning with feedback. In IEEE International Conference on Data Mining (ICDM). TEEE, 2023.
Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. Nature commmuiiationss 10(1):5223, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modcling problem. Advances in neural information processing systems, 34:1273 1286, 2021.
Diederik P Kingma and Max Welling. Auto-cncoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforce- ment learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural in information processing systems, 29, 2016.
Nishanth Kumar, Tom Silver, Willie McClinton, Linfeng Zhao, Stephen Proulx, Tomás Dozano-Péres Leslie Pack Kaelbling, and Jemnifer Barry. Practice makes perfect: Planning to learn skill parameter policies. arXiv preprint arXiv:2402. 150::2202.1022,,, 2024.
Jongmin Lee, Byung-Jun Lee, and Kee-E Kim. Reinforcement learning for control with multiple frequencies. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3254-3264. Curran Associates, Inc., 2020. URL 2164442222ddee7771149999eass Paper. pdf
netpsps.cppp.p.ppppprrrrr. ftles//papr/2222//ftel
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-44) 2016.
Jinghan Li, Zhicheng Sun, and Yadong Mu. Closed-loop long-horizon robotic planning via equilibrium sequence modeling. arXiv preprint arXhv:2410.01400 2024.
Wenhao Li. Efficient planning with latent diffusion. arXiv preprinl Xrx::22310.000 00311,, 2023.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations (ICLR), 2016.
Guanlin Liu, Zhihan Zhou, Han Liu, and Lifeng Tai. Efficient action robust reinforcement learning with probabilistic policy execution uncertainty. Transactions on Machine Learning Research, 2024. ISSN 2835- 8856. URL httts://opeerreview wttp://oooomiiddde2eeeiii
Jinxin Liu, Donglin Wang, Qiangxing Tian, and Zhengyu Chen. Learn goal-conditioned policy with intrinsic motivation for deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp.
2022.
Fan-Ming Luo, Tian Xu, Hang Lai, Xiong- Hui Chen, Weinan Zhang, and Yang Yu. A survey on model-based reinforcement learning. Science China Information Sciences, 67(2)1121101, 2024.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowl- ing. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Thomas Mcsnard, Théophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, et al. Counterfactual credit assignment in model-free reinforcement learning. Xix preprint or Xiv: 21w:2011100944, 2020.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Alberto Maria Metelli, Flavio Mazzolini, and Marcello Restelli. Control frequency adaptation via action persistence in batch reinforcement learning. In International Conference on Machine Learning, pp. 6862- 6873. PMLR, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXic preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, and Georg Ostrovski. Human-level control through deep reinforcement learning. nature, 518(754)):529 533, 2015.
Jurgis Pašukonis, Timothy P Lillicrap, and Danijar Hafner. Exaluating long-term memory in 3d mazes. Tn The Eleventh International Conference Learning Representations, 2023. URL httpe://openrevies:
pot/forummiddyyHLIIIII
Devdhar Patel and Hava Siegelmann. Overcoming slow decision frequencies in continuous control: Model- based sequence reinforcement learning for model-free control. arXic preprint orXiv:2410.08900 2024.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. arXiv preprint arXiv:2002. 12992, 2020.
Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task-agnostic offline reinforcement learning. In Conference on Robot Learning, pp. 1838 1849, PMLR, 2023.
Tankred Saanum, Noémi Éltetö, Peter Dayan, Marcel Binz, and Eric Schulz. Reinforcement learning with simple sequence priors. Advances in Neural Information. Processing Systems, 36:61985-62005, 2023.
Luca Sabbioni, Luca Al Daire, Lorenzo Bisi, Alberto Maria Metelli, and Marcello Restelli. Simultaneously updating all persistence values in reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 9668 9676, 2023.
Younggyo Seo and Pieter Abbeel. Coarse-to-fine q-network with action sequence for data-efficient robot learning. aix preprint 111 2024.
Sahil Sharma, Aravind Srinivas, and Balaraman Ravindran. Learning repeat: Fine grained action repe- tition for deep reinforcement learning. arXiv preprint arxi:: 702.060544 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwicsor, Ioannis Antonoglou, Veda Pannocrshelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nalure, 529(758):4484 489, 2016.
Rana Muhammad Sohaib, Syed Tariq Shah, Muhammad Ali Jamshed, Oluwakayode Onireti, and Poonam Yadav. Optimizing urlle in open ran: A deep reinforcement learning-based trade-off analysis. IEEE Communications Standards Magazine, 2025.
Aravind Srinivas, Sahil Sharma, and Balaraman Ravindran. Dynamic action repetition for deep reinforcement learning. In Proc. AAAI, 2017.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradicnt methods for reinforcement learning with function approximation. Advances in neural information processing systems,, 12, 1999.
Giulia Vezzani, Dhruva. Tirumala, Markus Wulfmeier, Dushyant Rao, Abbas Abdolmaleki, Ben Moran, Tuomas Haarnoja, Jan Humplik, Roland Hafner, Michael Neunert, et al. Skills: Adaptive skill sequencing for efficient lemporally-extendee exploration. arXiv preprinl 2211 2022.
Shuo Wang, Zhihao Wu, Xiaobo Hu, Youfang Lin, and Kai Lv. based hierarchical reinforcement learning for target visual navigation. IEFF. Transactions on Multimedia, 25:8920 8932, 2023.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Pengfei Wu, Chen Chen, Dexiang Lai, and Jian Zhong. A safe drl method for fast solution of real-time optimal power flow. Xiv preprint (rXiv2308.034220 2023.
Mengda Xu, Manuela Veloso, and Shuran Song. Aspire: Adaptive skill priors for reinforcement learning.
Aduances in Newral Information Processing Systems, 18886000386114 2022.
Dongkun Zhang, Jiaming Liang, Ke Guo, Sha Lu, Qi Wang, Rong Xiong, Zhenwei Miao, and Yue Wang. Carplanner: Consistent ruto-segressive trajectory planning for large-scale reinforcement learning in au- tonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17239- 17248, 2025.
Haichao Zhang, Wei Xu, and Haonan Yu. Generative planning for temporally coordinated exploration in reinforcement learning. In International Conference on Learning Representations, 2022a.
Haichao Zhang, Wei Xu, and Haonan Yu. Generative planning for temporally coordinated exploration in reinforcement learning, arXiv preprint X200:2201009765,, 2022b.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-gre ine-grained bimanual manipulation with low-cost hardware. Xiv preprint arXiv:2304. 13705, 2023.
19

---

Published in Transactions on Machine Learning Research (12/2025)
A Analysis of Reward Strategy Impact on ASR
This appendix details the comparison of two step penalty strategies, illustrated in Figure 5 The strategics are the default Max-Level-based penalty (-1/lmax; where mmx is normalization factor related to lask depth, e.g., (max R 30) and an alternative Map-Size- based penalty (-1/maze size, e.g.. - for a 16 x 16 maze). Their relative efficacy measured by AASR = Max- - ASRMap-Size where positive values indicate superior performance for the Max-Level strategy.
Figure reveals distinct performance patterns across the tested environments.
In simple 16 16 mazes, the Max-Level strategy provides substantial initial learning speedup (AASR = +0.8 at 200k steps), although its final Average Success Rate (ASR) is matched by the Map-Size strategy after approximately 400k training steps.
When 15% obstacles are introduced in the 16 x 16 maze, increasing its complexity, the Max-Level strategy maintains a consistent performance advantage throughout the training. AASR peaks at approximately +0.44 and remains positive (settling around +0.1).
In larger 24 24 mazes, the Max-Tevel strategy's superiority becomes more pronounced. AASR dramatically increases after 500k steps, reaching and sustaining value of approximately +0.9. This highlights the diminishing effectiveness of the Map-Size penalty (e.g.. -1/576 for 24 x 24) as it becomes increasingly diluted in larger state spaces.
The superior performance of the Max-Level strategy, particularly in more complex or larger environments, can attributed to several factors. Firstly, it provides a more impactful and relevant penalty signal. The normalizdl penalty (c.g., 1/330 provides stronger and more consistent learning foedback than the Map-Size penalty, which drops off the maze size increases. Secondly, liax factor that likely correlates better with the intrinsic task difficulty and typical solution length than the raw cell count of the maze, which does not inherently capture navigational complexity. Consequently, the Max-Level penalty structure appears to offer more effective exploration guidance and promotes greater learning efficiency.
normalization
In summary, normalizing step penalties by mmax (Max-Level-
strategy) leads to a more robust. and effective reward scheme for the navigation tasks studied. This approach fosters more efficient learning and achieves higher success rates by aligning the penalty signal more accurately with the inherent challenges of the environment, proving especially advantageous task complexity and scale increase.
B Evaluating the Average Success Rate (ASR) With Larger Train dataset
To assess performance on a larger dataset, we trained the agent on 2000 mazes. Table presents the Average Success Rates (ASR) across various maze configurations.
Table 4: Average Success Rates (ASR) with 2000 mazes in train datasel across maze lypes.
Max
Train
Dist. from
Maze Type
DQN
GPS
GPS-D
TempoRL
DAR
Steps
Size
Start to Goal
8x8
1M
2000
[] 14]
1.00
1.00
1.00
1.00
0.81
16x16
1M
2000
[16 26
1.00
1.00
1.00
1.00
0.89
16x16
obst 15%
1.5M
2000
[20 30
0.94
0.99
0.94
0.91
0.78
16x16
obst__ 25%
1.5M
2000
[20 30
0.98
0.98
0.94
0.39
0.16
16x16
1.5M
2000
(20 30)
1.00
0.95
0.82
0.98
0.13
16x16 corr
1M
2000
[10 30
1.00
1.00
1.00
1.00
0.67
24x24
1.5M
2000
20 30
0.98
1.00
0.98
1.00
0.75
24x24 obst.
15%
1.5M
2000
10 30)
0.05
0.09
0.05
0.03
0.13
20

---

Published in Transactions on Machine Learning Research (12/2025)
Reward Strategy Impact on ASR
0.6
ASR
16x16_en 16x16_obst_ 15%
16x16
24x24
0.4
0.0
100k
200k 300k
400k
500k
600k
700k 800k
900k
1M
Training Steps
Comparison of average success rate differences (AASR) between two reward stralegies: Max- Level (the default strategy) and Map-Size, which is identical except that it replaces mmax with the total number of cells the maze (o.g., 256 for a 16x16 maze). The values are evaluated across training steps. A positive AASR indicates that the Max-Level reward strategy yields better performance.
Figure 5:
C Evaluation on Stochastic Environments
Real-world environments feature inherent stochasticity through action execution noise, sensor uncertainty, and environmental dynamics. To evaluate GPS's robustness under such conditions, we extended our evalu- ation framework to include two types of stochastic dynamics:
"Sticky" actions. We implemented setup where there a 25% probability that the agent executes the previous action in the sequence instead of the current planned action. This "sticky actions" mechanism follows established protocols in reinforcement learning evaluation (Machado et 2018 Dabney al.) (020). introducing temporal correlations and systematic execution biases that create realistic uncertainty and test agent's 's ability to adapt to imperfect action execution.
Random actions. In this setup each action has 25% probability of being replaces with random action. These sctups are also common in the literature, where they are used to simulate adversarial environments
[Liu et al. 2024). or environments with noisy signals (Sutton et al. 1999).
These two setups present unique challenges for bequence-based methods like GPS, Since our approach com- mits to entire action sequences, exccution errors can potentially cascade throughout the sequence, leading significant. deviations from intended trajectories. This makes stochastic environments a critical test. for the robustness of our sequence generation paradigm.
C.1 Sticky Actions Results
Table presents GPS performance results under the "sticky" stochastic conditions across our maze config- urations, comparing GPS under deterministic conditions versus GPS with sticky actions (GPS-SA) under stochastic conditions. The results demonstrate that GPS exhibits remarkable resilience to stochastic dynam- ics. Despite the 25% action exccution uncortainty, GPS-SA achieves ncar-perfect or perfect success in most environments, reaching ASR 1.00 in empty mazes (8x8, 16x16, and 24x24) and maintaining strong performance (ASR 0.95) even in the challenging 16x 16 maze with 15% obstacles, and comparable
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: GPS Performance on Stochastic Maze Environments (25% Sticky Actions). GPS-SA denotes GPS with sticky actions. Both GPS and GPS-SA results are based on single-sced experiment.
ASR Converge
ASR Training Slepe
20.9 Step
PER
SGI
Faviroomess
IM
GPS
GPS-SA
GPS
GPS-SA
GPS
GPS-SA
GPS
GPS-SA
GPS
GPS-SA
GPS
GPS-SA
GPS
GPS-SA
8x8
1.00
1,00
1.00
100k
0.80
0.71
6.45
16216
0.71
0.77
1.00
1.00
1.00
1.000
1.00
1.000
3XIK
300k
0.85
0.80
8.99
8.83
16x16 obstacles 15%
0.43
0.33
0:92
0.90
0.92
0.94
0.92
0.95
3X0k
600k
0.75
0.09
14.35
16.26
10c16 olstacles 25%
0.04
0.37
0.22
0.47
>116M
>1.5M
0.68
0.61
18.47
19.41
16216 rooms
0.21
0.06
0.72
0.34
0.91
0.54
0.91
0.81
700k
>1MM
0.68
0.00
16.27
19.84
10c16 corridors
0.99
0.99
24x21
0.25
0.13
1.00
1.00
1.00
1.00
1.00
1.00
200k
300k
0.81
0.78
11.39
12:42
24<24 pistantes 15%
0.66
0.74
24x21 obetacler 25%
0.14
0.13
0.36
0.29
0.40
0.38
0.51
0:40
2-15M
>1.5M
0.55
0.48
17.22
18.72
Table 6: GPS Performance on Stochastic Maze Environments (25% Uniform Random Actions). Both GPS and DQN results are based on single-seed experiment.
ASR Couverge >009 Step
ASR at Training Stepe
PER
SCF
Envirommeut
100K
500K
1.5M
GPS
DQN
GPS
DQN
GPS
DQN
GPS
DQN
GPS
DQN
GPS
DQN
GPS
DQN
Sx8
0.98
0.74
0.99
0.36
1.0
0.86
100k
>1M
0.51
0.66
7.42
16x16
0.39
0.10
0.79
0.53
0.91
0.59
0.91
0.60
900k
>1.5M
0.54
0.71
11.14
16x16 obstacles 15%
0.11
0.05
0.35
0.38
0.63
0.66
0.83
0.69
>1.5
>1.5M
0.52
0.72
20.85
16x16 obstacles 25%
0.04
0.02
0.11
0.23
0.13
0.59
0.16
0.62
>1.M
>1.5M
0.49
0.68
20.96
moms
0.06
0.03
0.13
0.52
0.32
0.52
>15M
>1.5
0.47
0.66
24.29
corridors 24x24
16x16
0.71
0.34
0.98
0.98
0.81
200k
>1M
0.55
0.72
13.34
0.08
0.07
0.35
0.09
0.72
0.19
0.72
0.20
>1.5
>1.M
0.54
0.71
19.33
21x21 obstacles 15%
0.19
0.01
0.28
0.12
0.33
0.16
0.17
0.16
$1.M
>1.5M
0.13
0.56
20.05
21x24
obstacles
25%
0.09
0.03
0.17
0.08
0.21
0.14
0.21
0.16
>1.M
>1.5M
0.42
0.57
21.97
results (ASR = 0.61, ASR = 0.40) in 24x24 obstacles mazes. Comparing GPS performance across condi- tions reveals execution uncertainty's impact: while GPS achieves rapid convergence in deterministic settings, sticky actions extend training requirements. The 24x24 empty maze exemplifies this trade-off: convergence time triples from 200K to 300K steps under sticky actions, yet final ASR remains perfect (1.00) in both conditions. Path Efficiency Ratio shows moderate degradation from deterministic to stochastic conditions (0.84 to 0.78), indicating reduced oplimality while maintaining elfective navigation. Obstacle-dense environ- ments face more pronounced challenges, with the 16x 16 maze requiring doubled convergence time (300K to 600K steps). Although the PER reduction is comparable to that in the 24x24 empty maze, the geometry exposes and amplifies the impact of stochasticity where the same level degradation leads to much slower convergence and higher SGF as execution errors constr spaces increase collision likelihood.
The Path Efficiency Ratio (PER) analysis reveals muanced navigation behavior under stochastic conditions. GPS-SA maintains PER values ranging from 0.48 to 0.80 across environments, demonstrating reasonably efficient pathfinding despite 25% action execution uncertainty. While PER degradation from deterministic to stochastic conditions is relatively uniform across environment types (approximately 0.05-0.10 reduction: e.g., 0.84+0078 in 24x24 empty, 0.75-+0.69 16x16 with 15% obstacles, 0.68 in 16x16 rooms), these mod- erate declines indicate GPS successfully maintains goal-directed behavior while accommodating stochastic perturbations, avoiding catastrophic navigation failures even when exccution reliability compromised.
The Sequence Generation Frequency (SGF) reveals how GPS responds to execution uncertainty. Under sticky actions, SGF increases across most environments, indicating more frequent sequence generation to handle execution errors. The absolute increase varies by environment without clear pattern: some simple mazes show notable increases (8x8: +2.15 from 3.30 to 5.45), while others show minimal change (24x24 empty: +1.03 from 11.39 to 12.42), and complex environments similarly show variable increases (16x 16 with 15% obstacles: +1.91 from 14.35 to 16.26; 16x16 rooms: +3.57 from 16.27 to 19.84). This variability suggests that the impact of execution uncertainty on decision frequency depends on multiple factors beyond simple environment complexity.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Notably, GPS-SA's performance under stochastic dynamics surpasses baseline methods operating in deter- ministic environments. GPS-SA achieves perfect ASR (1.00) in the 24 maze at 500K training steps, while DQN reaches only 0.26 ASR even at 1.5M steps under deterministic conditions (see Table 2 Similarly, in the 16x16 maze with 15% obstacles, GPS-SA maintains ASR = 0.95 at 1.5M steps despite sticky actions, exceeding DQN's deterministic performance of 0.85, TempoRL'ss 0.80, and DAR's 0.84. The exception is the 16x16 maze with 25% obstacles, where both GPS (0.48) and GPS-SA (0.47) achieve lower final ASR than TempoRL (0.74) and DQN (0.73) under deterministic conditions, indicating that extremely dense ob- stacle configurations challenge the sequence-level decision-making paradigm under both deterministic and stochastic conditions. This cross-condition superiority - where GPS under adversarial stochasticity outper- forms baselines under ideal deterministic execution obviates the need for baseline evaluation under sticky actions, as these methods already struggle in the simpler setting.
C.2 Random Actions Results
The stochastic setup of random actions presents potential larger challenge to our proposed approach. Although this sctup has the same probability as the previous one of injocting an unintended action into the sequence, these actions may be more diverse (and polentially disruptive) than those of the previous setup. For example, an optimal trajectory for a given maze may include only "up" and "right" actions; in the sticky setup, the injected actions may only include these two actions. In the random sctup, every action - including those not included in the optimal trajectory is equally possible.
Table t presents our evaluation results comparing GPS to DQN, the baseline most likely to be robust against random action injection due to its singlo-action-at---tii docision-making. The results show that both algorithms achieve reduced performance in this stochastic setup: in all evaluated environments except 8x8 (the smallest and easiest), both algorithms underperform compared to their non-stochastic results. GPS maintains advantages across environments of varying complexity. In simple mazes, GPS achieves higher final ASR: 8x8 (1.00 vs 0.86), 16x16 empty (0.91 vs 0.60), and 16x corridors (0.98 vs 0.81). In larger mazes, GPS shows pronounced advantages: 24x24 emply (0.72 vs 0.20) and 24x24 with 15% obstacles (0.47 vs 0.16). The exceptions are highly constrained environments with dense obstacles, where DQN's reactive single-action approach proves more resilient: 16x16 with 25% obstacles (0.16 vs 0.62) and 16x 16 rooms (0.32 vs 0.52). GPS maintains moderate advantages in obstacle environments with lower density (16x16 16 with 15% obstacles: 0.83 vs 0.69).
The Path Efficiency Ratio (PER) analysis reveals that uniform random action injection causes more se- vere and uniform degradation compared to sticky actions. PER values under uniform random conditions cluster tightly between 0.42-0.55 across all environments, representing drops of 0.06-0.26 from sticky action performance. While PER under sticky actions varied by environment (0.48-0.80 range), uniform random perturbations compress nearly all enviromnents to approximately 0.50 PER. When both algorithms succeed, DQN achieves consistently higher PER (0.56-0.72) than GPS (0.42-0.55), indicating more optimal paths but lower success rates overall. This uniform degradation pattern reflects how unpredictable disruptions systematically undermine temporal abstraction of sequence coherence: sticky perturbations preserve some directional consistency by repeating previous actions, whereas uniform random perturbations completely dis- rupt planned multi-step trajectories, eliminating the path optimality advantages that temporal abstraction provides under more predictable conditions.
Sequence Generation Frequency (SGF) increases under uniform random conditions compared to sticky ac- tions, though the magnitude varies by environment. Simple mazes show large relative increases (8x8: +36% from 5.45 to 7.42; 16x 16 empty: +60% from 8.83 to 14.14; 24x24 empty: +36% from 12.41 to 19.33). while obstacle-dense environments show more modest increases (16x 16 with 15% obstacles: +28% from 16.26 to 20.85; 16x16 with 25% obstacles: +8% from 19.41 to 20.96). The elevated SGF values demonstrate that uniform random perturbations erode temporal abstraction benefits, as GPS must generate new action se- quences more frequently when unpredictable execution errors disrupt the execution of planned trajectories. Nevertheless, GPS's sustained ASR advantage over DQN in most environments indicates that multi-stop sequence generation retains value over purely reactive single-action decisions, even under high exccution uncertainty.
23

---

Published in Transactions on Machine Learning Research (12/2025)
C.3 Summary
These findings suggest that GPS's sequence-level decision making provides inherent robustness against ac- tion exccution noise. By generating coherent multi-step action sequences, the approach crcates a natural buller against individual action failures. The sequence generation paradigm appears to offer resilience to stochastic perturbations, the method can adapt its behavioral patterns over time while maintaining overall navigational effectivenoss despite local exccution uncertainties.
The robust performance under stochastic conditions provides initial evidence that temporal abstraction through sequence generation may offer advantages beyond sample efficiency. The ability to operate at the soquence lovel appears to confer resilience to environmental uncertaintics, suggesting potential applicability to scenarios where perfect action execution cannot be guaranteed, though further evaluation across diverse stochastic environments would be needed to establish broader generalizability of these findings.
Evaluation on Partial Observability Environments
Real-world tasks often involve partial observability, where agents must make decisions based on limited local information rather than complete environmental knowledge. To evaluate GPS's robustness under such constraints, we tested our approach with agents restricted to 7 x 7 observation window centered their current position, obscuring the majority of the 16 x 16/24 x 24 maze structure.
We implemented partial observability by modifying the observatioe ion space to show only a 7 7 window around the agent's current position. Cells outside this window are marked as unobserved, preventing the agent from accessing global maze structure or distant goal locations. This constraint fundamentally challenges sequence- based methods like GPS, which must generate entire action sequences without visibility of the complete path to the goal. The partial observability setting introduces limited planming horizons (visibility of only 3 cells in any direction), implicit memory requirements for previously observed areas, and increased exploration complexity without global visibility.
Table presents GPS's performance under partial observability constraints compared to the DQN baseline across all maze configurations except x 8. The results reveal distinct performance patterns across envi- ronment types that illuminate both the strengths and limitations of sequence- decision making under uncertainty.
Performance in Open Spaces and Corridor Environments. In empty and corridor enviromments, GPS demonstrates strong early-stage performance under partial observability. In the empty 16x 16 maze, GPS shows substantial initial advantage (0.54 vs 0.27 ASR at 100K steps), though this gap narrows con- siderably by the time of convergence (0.71 vs 0.60 at 1M steps). In the corridor environment, GPS maintains more pronounced advantages, achieving 0.79 ASR at. 1M steps compared to DQN's 0.26, representing a 3x improvement. These patterns suggest that generating action sequences provides inherent advantages in these structurally simple settings: multi-step soquences naturally maintain movement direction and exploration consistency, helping the agent navigate effectively even when it cannot its destination. The narrow pathways of corridor mazes may provide implicit guidance that reduces the disadvantage of partial observ- ability; once an agent commits to moving down a corridor, the constrained enviromment naturally channels movement toward valid destinations.
Scaling Effects and Path Efficiency Trade-offs. However, GPS's advantages diminish with increasing environment size. In the larger 24x24 empty maze, performance converges more closely between the methods (0.31 vs 0.30 ASR at 1.5M steps), contrasting with the substantial gaps observed in smaller mazes. This con- vergence likely reflects the compouningg challenge of limited visibility over longer navigation distances; while the 7x7 window covers reasonable fraction of the 16x16 maze, it represents a much smaller proportional view in 24x24 environments, diminishing GPS's advantage from directional consistency as both met struggle with the fundamentally limited information available for further supports this interpretation: GPS requires substantially more frequent sequence generation in 24x24 environments compared to 16x16, and the proportional increase in decision frequency from full to partial observability grows considerably larger environment size increases. Additionally, the moderate path ef-
navigation. The SGF metric
24

---

Published in Transactions on Machine Learning Research (12/2025)
Table 7: GPS and DQN Performance Under Partial Observability (7 x view window). Both GPS and DQN
results are based on singlo-soed experiment.
ASR Converge
ASR at Training Steps
20.9 Step
PER
SGF
Environment
100K
500K
1M
1.5M
GPS
DQN
GPS
DQN
GPS
DQN
GPS
DQN
DQN
GPS
DQN
GPS
DQN
16x16
0.54
0.27
0.67
0.71
0.60
0.71
0.60
215M
>1.5M
0.61
10.88
16x16_ obstacles__ss.......
16x16_obstacleg,
15%
0.30
0.15
0.34
0.25
0.34
0.29
0.34
0.29
>1.5
>1.5
0.70
1.00
12.74
16x16_obstaclees obstacles 25%
0.21
0.10
0.25
0.20
0.34
0.21
0.3-1
0.21
>1.5
>1.6
0.59
1.00
18.02
16x16 rooms
0.11
0.10
0.39
0.27
0.15
0.28
0.15
0.28
>1.5
>1.5M
0.66
0.97
14.08
16x16 corridors
0.48
0.19
0.79
0.26
0.79
0.26
>1M
>1M
0.57
0.97
11.79
0.22
0.14
0.31
0.30
0.31
0.30
>15M
>1.M
0.61
0.96
19.26
24v24 obstaccles 15%
0.17
0.05
0.32
0.05
0.33
0.05
0.33
0.05
>155
>1.M
0.47
1.00
16.65
24x24_obstacler wosttacless.......
25%%
0.12
0.04
0.25
0.04
0.27
0.04
0.27
0.04
1155
>1.M
0.48
1.00
18.49
ficiency ratios (PER. 0.61 in empty, 0.57 in corridors) indicate suboptimal navigation compared to DQN's near-perfect efficiency (PER 0.96-0.97), suggesting that while GPS reaches goals successfully, the generated sequences include unnecessary detours when operating with incomplete information. Importantly, DQN's higher PER values are calculated only its successful episodes, which represent much smaller and po- tentially easier subset of test instances compared to GPS's broader success coverage, potentially overstating DQN's true path optimality advantage.
Complex Spatial Navigation. GPS faces increased challenges in obstacle-dense and rooms environments under partial observability, though consistently outperforms DQN across these configurations. In the 16x16 maze with 15% obstacles, while GPS maintains carly advantages (0.30 vs 0.15 at 100K steps), both methods struggle to achieve high success rates, with GPS plateauing at 0.34 ASR and DQN at 0.29 ASR by 1.5M steps. The performance gap widens in denser obstacle configuratonss: in the 16x 16 maze with 25% obstacles, GPS achieves 0.34 ASR compared to DQN's 0.21, while in rooms enviromments, GPS reaches 0.45 ASR versus DQN's 0.28. These results represent substantial degradation compared to GPS's performance under full observability (where it typically achieves 0.90+ ASR in similar environments), indicating that partial observability fundamentally challenges sequence-basee based planning. In larger mazes, the challenges in- tensify: 24x24 environments show GPS achieving 0.27-0.33 ASR across obstacle configurations, compared DQN's 0.04-0.05, with performance substantially below what either method achieves under full observability.
Sequence Commitment vs. Reactive Adaptation. We hypothesize these difficulties stem from the compounded uncertainly of partial observability and complex spatial navigation. In obstacle envirouments, GPS must generate multi-stpp sequences without seeing upcoming barriers, leading to sequences that com- mit to trajectorics encountering unseen walls. Rooms environments require precise doorway navigation and chamber transitions that are particularly challenging when global layout information unavailable GPS may generate sequences directing movement toward distant goal without knowledge of intervening walls or doorways, resulting in collision-heavy paths. While DQN's step-by-step approach theoretically allows immediate incorporation of newly revealed environmental features, its poor performance suggests that reac- tive single-action decisions prove insufficient for effective exploration and navigation under limited visibility. GPS's relative advantage despite its commitment to multi-step sequences indicates that maintaining direc- tional consistency through sequences provides more value than reactive flexibility when visibility is severely constrained, though neither approach fully solves the challenge of navigating complex partially-observed spaces.
Training Efficiency and Sample Complexity. None of the tested configurations achieved the 0.9 ASR convergence threshold within 1.5M training steps, indicating substantially increased sample complexity under limited visibility compared to full observability settings where convergence typically occurs within up to 1M steps. This represents a significant challenge for sequence-based methods under partial observability, as the compound effect. of incomplete information and multi-step commitment makes effective policy learning considerably more sample-intnnsive.
Behavioral Adaptation Through Sequence Frequency. The Sequence Generation Frequency (SGF)
metric further illuminate GPS's adaptive response to partial observability. The elevated SGF under partial
25

---

Published in Transactions on Machine Learning Research (12/2025)
observability (10.88-19.26 across environments vs 8.76-17.77 with full visibility) reflects the increased chal- lenges of operating with limited environmental information more frequent sequence generation becomes necessary to maintain navigation progress when initial sequences prove less effective. The variation in SGF across environment types reveals two distinct dimensions of challenge: simple environments like the empty 16x16 maze shows SGF of 10.88, while more complex obstacle environments reach 10.65518.455 and the proportional increase from full to partial observability grows substantially larger 24x24 environments, in- dicating that both environmental complexity and scale compound the limitations of sequence- based planning under limited visibility. This pattern indicates that GPS generates new sequences more frequently when nav- igating uncertain terrain, likely because sequences planned with incomplete knowledge of obstacle locations prove less effective during execution, requiring more decision points to navigate around newly discovered barriers. This supports our hypothesis about the fundamental challenges of sequence planning in complex partially-observed spaces, where the commitment to multi-step plans conflits with the need to respond to newly revealed information.
Trade-offs and Future Directions. These findings highlight. both the potential and limitations of sequence-based decision making under partial observability. GPS's sustained advantages in simple envi- ronments (particularly the 3x improvement in corridors) demonstrate that multi-step action sequences can provide useful navigational structure even with limited sensory input, though performance degrades sub- stantially compared to full observability, particularly in obstacle-dense environments where the increased generation frequency (higher SGF) still proves insufficient to overcome the challenges of planning with in- complete spatial information. These results suggest several promising directions for future work. Confidence- conditioned sequence horizons could allow the decoder to learn generate shorter sequences when obser- vation uncertainty is high, balancing temporal abstraction benefits with the need for frequent replanning. Alternatively, integrating memory mechanismss that explicitly track previously observed regions could help GPS build a more complete spatial representation over time, reducing the disadvantage of limited immediate visibility and enabling more effective sequence planning in complex environments.
E
Self-Correen Colf-Correction Analysis
To substantiate our claim that GPS can self-correct through sequential sequences (Section 4.4.2) we con- ducted a comprehensive analysis of GPS's ability to recover from deviations in its trajectory.
E.1 Methodology
We developed a quantitative framework to measure GPS's self-correction behavior by tracking progress toward the goal using Breadth-First Search (BFS) distance. Our analysis focuses on identifying when GPS deviates from optimal progress and measuring its ability to generate correction sequences in subsequent decisions.
E.1.1 Deviation Detection
For each action sequence executed during an episode, we calculate:
Initial BFS distance (dbefore): The shortest path distance to the goal before sequence execution
2. Final BFS distance (datter): The shortest path distance to the goal after sequence execution
A deviation is detected when the sequence does not bring the agent closer to the goal:
dafter 2 dhefore
(1)
This indicates the agent either maintained the same distance or moved further from the goal, representing sequence that requires correction.
26

---

Published in Transactions on Machine Learning Research (12/2025)
E.1.2 Correction Sequence Classification
When a deviation is identified, we track whether GPS generates a correction sequence in subsequent decisions. We measure correction at multiple time horizons:
Immediate Correction: The very next sequence reduces the optimal path length from its own starting position to its ending position, demonstrating GPS's ability to generate locally optimal moves that recover from the previous suboptimal decision
Near-term Correction (within 2 sequences): Recovery occurs within the next two decision points
Medium-term Correction (within 3-4 sequences): Recovery within 3-4 subsequent decisions
Long-term Correction (5+ sequences): Recovery before episode termination
A correction sequence is considered successful when the agent's BFS distance falls below what would have been expected from optimal execution of the original deviating sequence.
E.2 Results
Table presents the self-correction analysis results across representative maze environments. The analysis was performed only on successful episodes to focus on GPS's recovery capabilities when ultimately reaches the goal.
Table & GPS Self- Colf-Correction Success Rates Across Maze Environments
Environment
Immediate Correction
Near-term
Medium-term
Long-term
(s2 seq)
(<4 seq)
(5+ seq)
8x8
0.885 + 0.095
0.889 = 0.092
0.896 = 0.086
1.000 = 0.000
16x16
0.996 + 0.006
1.000 + 0.000
1.000 + 0.000
1.000 + 0.000
16x16
obstacles
15%
0.753 + 0.311
0.779 = 0.313
0.794 + 0.292
1.000 + 0.000
16x16
1ox_1__ssstaclss 25%
0.765 = 0.324
0.789 = 0.298
0.808 = 0.272
1.000 = 0.000
16x16
rooms
0.624 0.114
0.773 0.088
0.805
0.079
1.000
0.000
16x16 corridors
0.965 0.046
1.000
0.000
1.000
0.000
1.000
0.000
24x24
0.899 0.143
1.000 + 0.000
1.000 0.000
1.000 + 0.000
Note: Values represent mean standard deviation across representative models from three different training seeds per maze type. Analysis conducted only on successful episodes.
E.3
Analysis and Discussion
The results provide strong empirical evidence for GPS's self-correction capability:
High Immediate Correction Rates: Across all environments, GPS demonstrates substantial immediate correction rates, ranging from 62.4% in complex room environments to 99.6% in open spaces. This indicates that GPS frequently generates correction sequences at the very next decision point after detecting a deviation.
Environment--peecfic Patterns:
Open spaces (16x16, 16, 24x24): Nearly perfect immediate correction (>89%), reflecting GPS's ability to quickly identify and generate correction sequences for directional errors
Structured environments (rooms, obstacles): Lower immediate correction rates (62-77%) but strong recovery within 2-4 sequences, suggesting GPS adapts its correction strategy to environmental complexity
27

---

Published in Transactions on Machine Learning Research (12/2025)
Corridors: High immediate correction (96.5%), likely due to the constrained nature limiting devi- ation possibilities
Guaranteed Long-term Recovery: All environments show 100% long-term correction rates in successful episodes, confirming that GPS consistently generates effective correction sequences even after significant trajectory deviations.
Higher standard deviations in obstacle-rich environments (+0.3) compared to open spaces (=0.006) indicate that correction difficulty varies with local maze structure, yet GPS maintains robust recovery capabilities.
Variance Patterns:
The evidence presented here directly supports claim that GPS employs sequenco-lveel closed-loop con- trol, generating correction sequences at subsequent decision points without requiring initially perfect action sequences. This self-correction mechanism helps explain GPS's performance characteristics, where the ability to recover from deviations enables reliable goal-reaching (high ASR) even when individual sequences may not follow strictly optimal paths (resulting in lower PER).
F Computational Cost Analysis
While GPS demonstrates superior sample efficiency, we analyze its computational trade-offs compared to baseline methods to provide a complete picture of the method's practicality.
Pro-training Costs. The VAE decoder requires approximately 10-15 minutes of pre-training on an Apple M1 Max with 64 GB RAM. This is a one-time cost per action space, and the trained decoder can be reused across different environments with the same action space, amortizing this cost over multiple experiments.
Inference Overhead. During inference, GPS requires 1.6-3.3x the wall-clock time per decision compared to baselines due to the Actor-Decope pipeline. In the 16x16 maze with 15% obstacles, GPS takes 2.7 2.72ms + 0.25ms per decision versus DAR's 1.39ms = 0.31ms (1.96x overhead) and DQN's 1.67ms + 0.05ms (1.63x overhead), while being 1.17x faster than TempoRL's 3.18mss + 0.76ms. In the larger 24x24 environment, the overhead pattern persists: GPS requires 3.82ms + 0.81ms per decision compared DAR's 1.16ms + 0.09ms (3.29x overhead), DQN's 1.68ms + 0.05ms (2.27x overhead), and TempoRL's 2.89ms + 0.77ms (1.32x overhead). However, since GPS makes decisions less frequently due to its scquence-level abstraction, also measure normalized episode efficiency the total episode wall-clock time divided by the optimal path length to normalize for task difficulty and enable fair comparison across algorithms with different success rates. In the 16x16 with 15% obstacles environment, GPS (4.6ms + 0.19ms per optimal step) performs comparably to basclines (DQN: 4.03ms, TempoRL: 4.55ms, DAR: 4.65ms), indicating that the per-decision overhead is largely offset. by reduced decision frequency. However, the 24x24 environment, GPS shows higher normalized costs (7.24ms +/ 1.51ms per optimal step) compared to baselines (DQN: 5.83ms, TempoRL: 5.83ms, DAR: 4.92ms). Note that this metric is computed only successful episodes, which may bias results toward baselines that solve only easier test instances (GPS: 90% ASR vs DQN: 24%, TempoRL: 46%, DAR: 23%), suggesting the true computational cost difference for solving challenging instances may be smaller than indicated.
Training Time to Convergence. Despite higher per-step costs, GPS achieves faster wall-clock convergence in medium complexity environments like 16x16 mazes with 15% obstacles due to its superior sample efficiency. In 16x16 mazes with 15% obstacles, GPS reaches 90% ASR in 3.3 hours (500K steps), while other methods fail to reach 90% ASR within the same training budget: DQN achieves only 85% ASR after 5.83 hours (1.5M steps), TempoRL reaches 82% ASR after 9.2 hours (1.5M steps), and DAR. achieves 64% ASR after 4.08 hours (1.5M steps). For the larger 24x24 maze, GPS's computational overhead becomes more apparent, with training times becoming comparable to the fastest baseline despite superior convergence and final performance. GPS achieves 90% ASR in 11.5 hours (IM steps), while all baselines fail to reach 90% ASR: DQN reaches only 24% ASR after 10.5 hours (1.5M steps), TempoRL. achieves 46% ASR after 13.02 hours (1.5M steps), and DAR reaches 23% ASR after 6.26 hours (1.5M steps), illustrating the trade-off between computational efficiency and learning effectiveness.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Computational Trade-offs and Practical Considerations. While GPS incurs additional computational overhead per decision compared to most baselines, its reduced decision frequency often compensates for this cost. The computational trade-offs of GPS vary with environment complexity: in medium complexity environments like 16x16 with 15% obstacles maze, GPS provides clear training time advantages, reaching larget performance levels considerably faster than baselines. larger environments such as 24x24 maze, training times become comparable the fastest basclines, though GPS achieves substantially superior final performance (90% 24% ASR for DQN). This pattern suggests that GPS becomes most practical when learning effectiveness is prioritized over computational speed, particularly in scenarios where baseline methods struggle to reach acceptable performance levels rather than purely on training efficiency grounds.
Table 9: Computational cost analysis across environments and methods. Values represent mean standard
deviation over runs.
Environment
Metric
GPS
DQN
TempoRL
DAR
Inference time per decision (ms)
2.72 0.25
1.67 0.05
3.18 + 0.76
1.39 0.31
Normalized episode efficiency (ms/opt. step)
+ 0.19
4.03 0.08
4.55 + 0.75
4.65 0.56
16x16 obst 15%
Training wall-clock to 90% ASR
3.3h
5.83h
9.2h
4.08h
Total training wall-clock (h)
9.97 + 0.31
5.83 + 0.35
9.2 + 1.72
4.08 0.65
Inference time per decision (ms)
3.82 0.81
1.68 0.05
1 0.77
1.16
0.09
Normalized episode efficiency (ms//ptt step)
7.24 1.51
5.83 4 0.17
5.83 1: 1.18
4.92 0.54
24x24
Training wall-clock to 90% ASR.
11.5h
10.5h
13.02h
6.26h
Total training wall-clock (h)
17.35 + 2.11
10.5 + 0.39
13.02
1.67
6.26 0.12
VAE Pre-training (one-time)
10-15 minutes on Apple MI Max
In summary, GPS represents a favorable computational trade-off for challenging navigation tasks where base- line methods fail to achieve acceptable performance, offering superior learning effectiveness at competitive training costs.
G Comparison: Pre-trained Decoder End-to-End Training
When training GPS, one needs choose whether to pre-train the Proto-Soquence Decoder (PSD) or train it jointly end-to-end with the actor and critic networks. To assess the impact of this choice, we conducted comprehensive comparison across that decoder pre-training could potentially encode implicit biases about feasible action sequences, raising questions about whether GPS's advantages stem from the sequencc-level decision-making paradigm itself or from initialization artifacts.
environments. This analysis was motivated by the observation
G.1 Experimental Setup
We evaluated two GPS variants:
GPS: Our default implementation with a pre-trained VAE decoder (frozen during actor-critic train- ing)
GPS-E2E: End-to-end training where all three components (actor, decoder, critic) are jointly opti- mized from random initialization
Both variants were evaluated on the same set of maze environments using identical hyperparameters for the actor and critic networks. The decoder architecture remained unchanged; only the training procedure differed.
G.2 Results and Analysis
Table 10 presents the comparative results across maze environments. The findings reveal several important insights:
29

---

Published in Transactions on Machine Learning Research (12/2025)
Table 10: Performance comparison between GPS with pre-trained decoder and end-to-end training (GPS-
E2E). Both GPS and GPS-E2E results are based on singlo-sced experiment.
ASR Conseage
ASR Treining Stepe
2-09 Step
PER
SGF
Fovirommnt
100K
GooK
1.5M
GPS
GPS-EaE
GFS-E2E
GPS-EZE
GFS-E2E
8.8
1.00
0.588
1.000
1.00
1.30
1.00
100A
1004.
0.80
0.97
3.30
6.28
18x16
0.71
0..8
1.000
1,100
1.30
1,00
1,00
1,000
2034
0185
0.80
3.90
11.10
16-16_ obstacles
16%
14.35
16x16 obstacles 2555
0.05
0.07
0.50
0.67
0.89
0.53
>1.5M
14M
0.68
0.79
18.
27.03
16x16 rooms
0:31
0.13
0.72
0.89
0.01
0.90
0.91
0.90
700k
1M
0.68
0.70
16.27
30.37
18x16. corridors
0.90
0.56
0.99
0.99
0.99
100k
1002
0.79
9.11
14.95
24x24
0.228
11.39
25x24 obstacles 1856
0.201
0.34
0.06
1.67
0.74
01588
0.81
0.77
51.M
>1.MM
0.38
0.52
14.64
29.08
24x24 obstncles 1036
0.14
0.14
0.30
0.44
0.46
0.50
0.51
0.53
21.5
21.6
0.55
0.52
17.22
29.28
G.2.1 Performance Parity in Success Rate and Convergence
The most significant finding is that GPS-E2E achieves comparable final performance GPS with pre-trained decoder across most tested environments. Both variants converge to similarly high success rales, (e.g., 16x16: GPS-E2E reaches ASR >> 0.9 at 200k steps vs GPS at 300k steps; 24x24: GPS converges at 200k steps vs GPS-E2E at 300k steps). The final ASR values are nearly identical across environments (e.g., 16x16: both reach 1.00; 16x 16 obs 15%: 0.92 vs 0.94; 16x16 16 rooms: 0.91 vs 0.90). This demonstrates that:
with convergence speeds being environment-depentee
The performance advantage of GPS stems primarily from the sequence-level decision-making paradigm rather than from implicit biases encoded decoder pre-training.
GPS can successfully learn effective action sequence representations from scratch during joint train- ing.
Decoder pre-training is not a fundamental requirement. However, as we show below, there are trade-off's with regard to path optimality and the number of sequences that need to be generated per trajectory.
G.2.2 Path Efficiency vs. Temporal Abstraction Trade-off
An interesting pattern emerges when examining the Path Efficiency Ratio (PER) and Sequence Generation Frequency (SGF) metrics:
Higher PER for GPS-E2E: End-to-end training produces slightly higher path efficiency (e.g.. 16x16: 0.89 vs 0.85; 16x16 obs 15%: 0.82 vs 0.75). This indicates that GPS-E2E learns to generate action sequences that more closcly follow optimal paths.
Lower SGF for GPS: The pre-trained decoder variant requires fewer sequence generations per episode (c.g., 16x16: 8.99 vs 11.10; 24x24_ obstacles 15%: 14.64 29.68; 16x16 16x16_rooms: 30.37), suggesting produces longer, _4x2__osstaclees_____
rooms: 16.27 lemporal abstraction action sequences.
This trade-off reveals an important behavioral difference: GPS with pre-trained decoder appears to develop more aggressive temporal abstraction strategies generating longer sequences that reduce decision frequency at the cost of some path optimality. In contrast, GPS-E2E learns to balance temporal abstraction with path efficiency, generating somewhat shorter sequences that more closely track optimal trajectories.
We hypothesize this difference arises because:
The pre-trained decoder's latent space structure, learned from diverse synthetic sequences, encour- ages exploration of longer, more varied action patterns
End-to-end training jointly optimizes sequence generation with the specific task objectives, poten- tially leading to more task-specific (and thus more efficient but less exploratory) sequence patterns
30

---

Published in Transactions on Machine Learning Research (12/2025)
This suggests that decoder pre-training may provide form of implicit exploration regularization, encouraging the agent to commit to longer temporal abstractions even when shorter sequences might be locally optimal.
G.3 Decoder Transferability Across Tasks
An important practical advantage of the pre-trained decoder approach is its reusability, In our experiments, the same pre-trained decoder was successfully used across:
All maze configurations (emply, obstacles with varying densities, rooms, corridors)
All maze sizes (8x8, 16x16, 24x24)
Multiple experimental conditions (deterministic dynamics, stochastic environments and partial ob- servability)
This transferability demonstrates that the decoder learns general-purpose sequence generation capabilities that apply across diverse navigation scenarios within the same action space. For applications involving multiple related tasks with shared action spaces, pre-training the decoder once and reusing it can provide practical benefits. However, our end-to-end results confirm that decoder reusability is an added benefit rather than a necessity for GPS's effectiveness. The choice between variants ultimately reflects the PER- SGF trade-off: pre-training encourages more aggressive temporal abstraction (lower SGF) while end-to-end training optimizes for path efficiency (higher PER).
G.4 Conclusion
Our analysis demonstrates that GPS achioves strong performance with both pre-trained and end-to-end training, confirming that advantages stem from sequence-level decision-making rather than initialization artifacts. The comparable performance of GPS-E2E validates that GPS's superior performance compated to the baselines (Section 4.4h is derived by its architectural approach to temporal abstraction.
The pre-trained variant offers decoder reusability and more aggressive temporal abstraction (lower SGF), while the end-to-end variant provides superior path efficiency (higher PER) with comparable success rates. This PER-SGF trade-off suggests decoder initialization influences the exploration-eexloittaioo balance. Prac- titioners can select either variant based on specific requirements without sacrificing fundamental performance.
Future work could explore combining both approaches' strengths through adaptive docoder training schodules (starting pre-trained for exploration, transitioning lask-specific optimization) or curriculum learning that leverages the complementary benefits of both paradigms.
H ASR Statistical Significance Testing
To assoss the statistical significance of the differences in Average Success Rates (ASR) between proposed method (GPS) and the baselines (DQN, TempoRL, DAR), employed McNemar's test. This section details the methodology and presents the results of these tests.
Methodology
McNemar's test is non- non-parametric test suitable for paired nominal data. It is used to determine whether there is a significant difference in the proportions of two related samples, such when two algorithms are evaluated on the same set of test instances. In our context, each maze evaluation episode serves as a paired instance, and the outcome for each algorithm (GPS or DQN for example) is categorized as either a success or failure.
An episode was deemed a success if the agent reached the goal in an episodic length of less than 75 steps. Otherwise, it considered failure.
31

---

Published in Transactions on Machine Learning Research (12/2025)
For each pair of algorithms (GPS vs. DQN) on given maze type, constructed 2 x contingency table based on the outcomes of common evaluation episodes:
Algorithm B (DQN)
Success
Failure
Algorithm (GPS)
Success
a
Failure
Where:
a: Number of episodes where both GPS and DQN succeeded.
b: Number of episodes where GPS succeeded and DQN failed.
Number of episodes where GPS failed and DQN succeeded.
d: Number of episodes where both GPS and DQN failed.
McNemar's test focuses on the discordant pairs (b and c). The null hypothesis (Ho) is that the two algorithms have the same ASR. The test statistic is calculated as:
(b c)
b+c
This statistic follows chi-squared distribution with degree of freedom. We used the version of the test without continuity correction, implemented in itsasmmmeelssttts.connnngeeeee satpocy___bbleseeeeee tables.menemar"
The significance level was established at = 0.05. However, in all comparisons, the calculated p-values were substantially lower (p < 0.0001), providing strong evidence to reject the null hypothesis and confirming statistically significant differences ASR performance.
Results: GPS vs. DQN
The results of McNemar's test comparing GPS (Algorithm A) to DQN (Algorithm B) across various maze configurations are summarized in Table 11 The Average Success Rate (ASR) reported in the table for cach algorithm based on Table 2
ASR (GPS) = + b/+bbbbbbct
ASR (DQN) (a (a+ (otc)/aabb+++
All comparisons in Table 11 yielded p-values below 0.0001, indicating highly statistically significant differ- ences between GPS and DQN across all environments. GPS outperforms DQN in the majority of setups. The sole exception is the 16x16 maze with 25% obstacles, where DQN achieves significantly better per- formance. As discussed in Scction 4.4.1. this result reflects the interplay between maze scale and obstacle densily rather than general limitation in dense environments, given that GPS significantly outperforms DQN in the 24x24 setup with the same obstacle density.
The following sections provide the dotailed per-run summarics logged and the specific contingency tables used for McNemar's lest for each maze configuration.
Maze: 8x8
GPS (Algorithm A) Summary: Total episodes: 4978, Successes: 4978, Failures: 0, Errors:
-DQN (Algorithm B) Summary: Total episodes: 4978, Successes: 4211, Failures: 767, Errors:
Contingency Table:
32

---

Published in Transactions on Machine Learning Research (12/2025)
Table 11: McNemar's Test Results for GPS vs. DQN. All differences are statistically significant
< 0.0001). Results favor GPS in all cases except where marked with which favors DQN.
Environment
ASR (GPS)
ASR (DQN)
McNemar Stat.
8x8
1.00
0.85
767.00
16x16
1.00
0.74
1350.00
16x16 w/ 15% obs.
0.94
0.84
75.98
16x 16 w/ 25% obs.
0.57
0.72
114.57t
16x16 rooms
0.85
0.64
414.42
16x16 corridors
0.98
0.80
448.63
24x24
0.94
0.28
3244.29
24x24 w/ 15% obs.
0.83
0.13
3385.79 79
24x24 w/ 25% obs.
0.49
0.10
1674.61
DQN (Alg. B)
Success
Failure
GPS (Alg, A)
Success
4211 (a)
767 (b)
Failure
0 (c)
0 (d)
Common episodes for comparison: 4978
McNemar's Statistic: 767.0000, p-value: 0.0001
Maze: 16x16
GPS (Algorithm A) Summary: Total episodes: 4994, Successes: 4994, Failures: 0, Errors: 0
DQN (Algorithm B) Summary: Total episodes: 4994, Successes: 3644, Failures: 1350, Errors: 0
Contingency Table:
DQN (Alg. B)
Success
Failure
GPS (Alg. A)
Success
3644 (a)
1350 (b)
Failure
0 (c)
0 (d)
Common episodes for comparison: 4994
McNemar's Statistic: 1350.0000, p-value: < 0.0001
Maze: 16x16 Obstacles 15%
GPS (Algorithm A) Summary: Total episodes: 1050, Successes: 981, Failures: 69, Errors:
DQN (Algorithm B) Summary: Total episodes: 1050, Successes: 882, Failures: 168, Errors: 0
Contingency Table:
DQN (Alg. B)
Success
Failure
GPS (Alg. A)
Success
867 (a) 15 (c)
114 (b)
Foilure
54 (d)
Common episodes for comparison: 1050
McNemar's Statistic: 75.9767, p-value: < 0.0001
33

---

Published in Transactions on Machine Learning Research (12/2025)
Maze: 16x16 Obstacles 25%
GPS (Algorithm A) Summary: Total episodes: 2010, Successes: 1143, Failures: 867, Errors: 0
DQN (Algorilhm B) Summary: Total episodes: 2010, Successes: 1451, Failures: 559, Errors:
Contingency Table:
DQN (Alg. B)
Success
Failure
GPS (Alg. A)
Success
883 (a)
260 (b)
Fuilure
568 (c)
299 (d)
Common episodes for comparison: 2010
McNemar's Statistic: 114.5700, p-value: 0.0001
Maze: 16x16 Rooms
GPS (Algorithm A) Summary: Total episodes: 2930, Successes: 2496, Failures: 434, Errors:
DQN (Algorithm B) Summary: Total episodes: 2930, Successes: 1888, Failures: 1042, Errors: 0
Contingency Table:
DQN (Alg. B)
Success
Failure
GPS (Alg. A)
Success
1746 (a)
750 (b)
Failure
142 (c)
292 (d)
Common episodes for comparison: 2930
McNemar's Statistic: 414.4215, p-value: 0.0001
Maze: 16x16 Corridors
GPS (Algorithom A) Summary: Total episodes: 2725, Successes: 2671, Failures: 54, Errors:
DQN (Algorithm B) Summary: Total episodes: 2725, Successes: 2182, Failures: 543, Errors:
Contingency Table:
DQN (Alg. B)
Success
Failure
GPS (Alg. A)
Success
2160 (a)
511 (b)
Fuilure
22 (c)
32 (d)
Common episodes for comparison: 2725
McNemar's Statistic: 448.6323, p-value: 0.0001
Maze: 24x24
GPS (Algorithm A) Summary: Total episodes: 4999, Successes: 4675, Failures: 324, Errors:
DQN (Algorithm B) Summary: Total episodes: 4999, Succossos: 1366, Failuros: 3633, Errors: 0
Contingency Table:
34

---

Published in Transactions on Machine Learning Research (12/2025)
DQN (Alg. B)
Success
Failure
GPS (Alg, A)
Success
1333 (a)
3342 (b)
Failure
33 (c)
291 (d)
Common episodes comparison: 4999
McNemar's Statistic: 3244.2907, p-value: 0.0001
Maze: 24x24 Obstacles 15%
GPS (Algorithm A) Summary: Total episodes: 4997, Successes: 4098, Failures: 899, Errors: 0
DQN (Algorithm B) Summary: Total episodes: 4997, Successes; 615, Failures: 4382, Errors:
Conlingency Toble:
DQN (Alg. B)
Success
Failure
GPS (Alg. A)
Success
565 (a)
3533 (b)
Failure
50 (c)
849 (d)
Common episodes comparison: 4997
McNemar's Statistic: 3385.7910, p-value: < 0.0001
Maze: 24x24 Obstacles 25%
GPS (Algorithm A) Summary: Total episodes: 5000, Successes: 2443, Failures: 2557, Errors:
DQN (Algorithm B) Summary: Total episodes: 5000, Successes: 489, Failures: 4511, Errors:
Contingency Table:
DQN (Alg. B)
Success
Failure
GPS (Alg. A)
Success
326 (a)
2117 (b)
Failure
163 (c)
2394 (d)
Common episodes for comparison: 5000
McNemar's Statistic: 1674.6123, p-value: < 0.0001
This detailed breakdown for each environment shows the specific data underlying the McNemar's tests.
Results: GPS vs. TempoRL
Table 12 summarizes the Average Success Rates (ASR) for GPS and TempoRL, along with the McNemar test statistics and p-values derived from common paired evaluation episodes.
All comparisons in Table 12 yielded p-values below 0.0001, indicating highly statistically significant differ- ences between GPS and TempoRL across all environments. GPS outperforms TempoRL in the majority of setups. The sole exception is the 16x16 maze with 25% obstacles, where TempoRL achieves significantly better performance. As discussed in Section 4.4.1 this result reflects the interplay between maze scale and obstacle density rather than a general limitation in dense environments, given that GPS significantly outperforms TempoRL in the 24x24 setup with the same obstacle density.
The following sections provide the detailed per-run summarics and the specific contingency tables used for McNemar's test. for each maze configuration when comparing GPS with TempoRT.
35

---

Published in Transactions on Machine Learning Research (12/2025)
Table 12: McNemar's Test Results for GPS vs. TempoRL. All differences are statistically significant
< 0.0001). Results favor GPS in all cases except where marked with which favors TempoRL.
Environment
ASR (GPS)
ASR (TempoRL)
McNemar Stat.
8x8
1.00
0.94
311.00
16x16
1.00
0.88
609.00
16x16 w/ 15% obs.
0.94
0.80
86.78
16x16 w/ 25% obs.
0.57
0.73
105.01
16x 16 rooms
0.85
0.67
268.69
16x16 corridors
0.98
0.89
176.14
24x24
0.94
0.52
1782.46
24x24 w/ 13% obs.
0.83
0.17
2956.75
24x24 w/ 25% obs.
0.49
0.08
1746.57
Maze: 8x8
GPS (Algorithm A) Summary: Total episodes: 4978, Successes: 4978, Failures: 0, Errors:
TempoRL (Algorithm B) Summary: Total episodes: 4978, Successes: 4667, Failures: 311, Errors: 0
Contingency Table:
TempoRL (Alg. B)
Success
Failure
GPS (Alg. A)
Success
4667 (a)
311 (b)
Failure
0 (c)
0 (d)
Common episodes for comparison: 4978
McNemar's Statistic: 311.0000, p-value: 0.0001
Maze: 16x16
GPS (Algorithm A) Summary: Total episodes: 4994, Successes: 4994, Failures: 0, Errors:
TempoRL (Algorithm B) Summary: Total episodes: 4994, Successes: 4385, Failures: 609, Errors:
Contingency Table:
TempoRL (Alg. B)
Success
Failure
GPS (Alg. A)
Success
4385 (a)
609 (b)
Failure
0 (c)
0 (d)
Common episodes for comparison: 4994
McNemar's Statistic: 609.0000, p-value 0.0001
Maze: 16x16 Obstacles 15%
GPS (Algorithm A) Summary: Total episodes: 1050, Successes: 981, Failures: 69, Errors:
TempoRL (Algorithm B) Summary: Total episodes: 1050, Successes: 834, Failures: 216, Errors:
Contingency Table:
36

---

Published in Transactions on Machine Learning Research (12/2025)
TempoRL (Alg. B)
Success
Failure
GPS (Alg. A)
Success
783 (a)
198 (b)
Failure
51 (c)
18 (d)
Common episodes comparison: 1050
McNemar's Statistic: 86.7831, p-value: 0.0001
Maze: 16x16 Obstacles 25%
GPS (Algorithm A) Summary: Total episodes: 2010, Successes: 1143, Failures: 867, Errors: 0
Te (Algorithm B) Summary: Total episodes: 2010, Succosses: 1462, Failures: 548, Errors: 0
Contingency Table:
TempoRL (Alg. B)
Success
Failure
GPS (Alg. A)
Success
818 (a)
325 (b)
Failure
644 (c)
223 (d)
Common episodes for comparison: 2010
McNemar's Statistic: 105.0165, p-value: 0.0001
Maze: 16x16 Rooms
GPS (Algorithm A) Summary: Total episodes: 2930, Successes: 2496, Failures: 434, Errors: 0
TempoRL (Algorithm B) Summaar: Total episodes: 2930, Successes: 1944, Failures: 986, Errors: 0
Contingency Table:
TempoRL (Alg. B)
Success
Failure
GPS (Alg. A)
Success
1653 (a)
843 (b)
Failure
291 (c)
143 (d)
Common episodes comparison: 2930
McNemar's Statistic: 268.6984, p-value 0.0001
Maze: 16x16 Corridors
GPS (Algorithm A) Summary: Total episodes: 2724, Succosses: 2670, Failures: 54, Errors: 0
TempoRL. (Algorithm B) Summary: Total episodes: 2724, Successes: 2426, Failures: 298, Errors: 0
Contingency Table:
TempoRL (Alg. B)
Success
Failure
GPS (Alg. A)
Success
2379 (a)
291 (b)
Failure
47 (c)
7 (d)
Common episodes for comparison: 2724
McNemar's Statistic: 176.1420, p-value: < 0.0001
37

---

Published in Transactions on Machine Learning Research (12/2025)
Maze: 24x24
GPS (Algoritim A) Summary: Total episodes: 4999, Successes: 4675, Failures: 324, Errors: 0
TempoRL (Algorithm B) Summary: Total episodes: 4999, Successes: 2611, Failures: 2388, Errors:
Contingency Table:
TempoRL (Alg. B)
Success
Failure
GPS (Alg. A)
Success
2448 (a)
2227 (b)
Failure
163 (c)
161 (d)
Common episodes for comparison: 4999
McNemar's Statistic: 1782.4669, p-value: < 0.0001
Maze: 24x24 Obstacles 15%
GPS (Algorithm A) Summary: Total episodes: 4997, Successes: 4098, Failures: 899, Errors: 0
TempoRL (Algorithm B) Summary: Total episodes: 4997, Successes: 860, Failures: 4137, Errors: 0
Contingency Table:
TempoRL (Alg. B)
Success
Failure
GPS (Alg. A)
Success
706 (a)
3392 (b)
Failure
154 (c)
745 (d)
Common episodes for comparison: 4997
McNemar's Statistic: 2956.7524, p-value: < 0.0001
Maze: 24x24 Obstacles 25%
GPS (Algorithm A) Summary: Total episodes: 5000, Successes: 2443, Failures: 2557, Errors:
TempoRLL (Algorithom B) Summarry: Total episodes: 5000, Successes: 358, Failures: 4642, Errors: 0
Contingency Table:
TempoRL (Alg. B)
Success
Failure
GPS (Alg. A)
Success
156 (a)
2287 (b)
Failure
202 (c)
2355 (d)
Common apisodes comparison: 5000
McNemar's Statistic: 1746.5749, p-value: 0.0001
This detailed breakdown for each environment when comparing GPS to TempoRL. shows the specific data underlying McNemar's tests.
Results:
GPS vs. DAR
Table summarizes the Average Success Rates (ASR) for GPS and DAR, along with the McNemar test statistics and p-values derived from common evaluation episodes.
38

---

Published in Transactions on Machine Learning Research (12/2025)
All comparisons in Table 13 yield p-values below 0.0001, demonstrating highly statistically significant im- provements of GPS over the DAR baseline across all tested maze environments. These consistent outcomes and extremely low p-values robustly support the conclusion that GPS offers superior performance compared to the DAR. baseline under these experimental conditions.
Table 13: McNemar's Test Results for GPS vs. (p < 0.0001). Results favor GPS in all listed cases.
DAR. All differences are statistically significant
Environment
ASR (GPS)
ASR (DAR)
McNemar Stat.
8x8
1.00
0.74
1385.00
16x16
1.00
0.66
1749.00
16x16 w/ 15% obs.
0.94
0.58
322.84
16x16 w/ 25% obs.
0.57
0.16
584.31
16x16 rooms
0.85
0.13
2066.76
16x16 corridors
0.98
0.61
1020.16
24x24
0.94
0.29
3029.20
24x24 w/ 15% obs.
0.83
0.12
3357.5r .55
24x24 w/ 25% obs.
0.49
0.05
1953.79
The following sections provide the detailed per-run summaries logged by the script and the specific contin- gency tables used for McNemar's test for each maze configuration when comparing GPS with DAR.
Maze: 8x8
GPS (Algorithu A) Summary: Total episodes: 4978, Successes: 4978, Failures: 0, Errors:
DAR (Algorithm B) Summary: Total episodes: 4978, Successes: 3593, Failures: 1385, Errors: 0
Contingency Table:
DAR (Alg. B)
Success
Failure
GPS (Alg. A)
Success
3593 (a)
1385 (b)
Failure
0 (c)
0 (d)
Common cpisodes for comparison: 4978
McNemar's Statistic: 1385.0000, p-value: < 0.0001
Maze: 16x16
GPS (Algorithm A) Summary: Total episodes: 4994, Successes: 4994, Failures: 0, Errors:
DAR (Algorithm B) Summary: Total episodes: 4994, Successes: 3245, Failures: 1749, Errors: 0
Contingency Table:
DAR (Alg. B)
Success
Failure
GPS (Alg. A)
Success
3245 (a)
1749 (b)
Foilure
0 (c)
0 (d)
Common episodes for comparison: 4994
McNemar's Statistic: 1749.0000, p-value: < 0.0001
39

---

Published in Transactions on Machine Learning Research (12/2025)
Maze: 16x16 Obstacles 15%
GPS (Algorithm A) Summary: Total episodes: 1050, Successes: 981, Failures: 69, Errors:
DAR (Algorithm B) Summary: Total episodes: 1050, Successes: 599, Failures: 451, Errors: 0
Contingency Table:
DAR (Alg. B)
Success
Failure
GPS (Alg. A)
Success
564 (a)
417 (b)
Fuilure
35 (c)
34 (d)
Common episodes for comparison: 1050
McNemar's Statistic: 322.8407, p-value: 0.0001
Maze: 16x16 Obstacles 25%
GPS (Algorithm A) Summary: Total episodes: 2010, Successes: 1143, Failures: 867, Errors: 0
DAR (Algorithm B) Summary: Total episodes: 2010, Successes: 319, Failures: 1691, Errors: 0
Contingency Table:
DAR (Alg. B)
Success
Failure
GPS (Alg. A)
Success
150 (a)
993 (b)
Failure
169 (c)
698 (d)
Common episodes for comparison: 2010
McNemar's Statistic: 584.3167, p-value: 0.0001
Maze: 16x16 Rooms
GPS (Algorithu A) Summary: Total episodes: 2930, Successes: 2496, Failures: 434, Errors: 0
DAR (Algorithm B) Summary: Total episodes: 2930, Successes: 330, Failures: 2600, Errors: 0
Contingency Table:
DAR (Alg. B)
Success
Failure
GPS (Alg. A)
Success
278 (a)
2218 (b)
Fuilure
52 (c)
382 (d)
Common episodes for comparison: 2930
McNemar's Statistic: 2066.7648, p-value: < 0.0001
Maze: 16x16 Corridors
GPS (Algoritium A) Summary: Total episodes: 2725, Successes: 2671, Failures: 54, Errors:
DAR (Algorithm B) Summary: Total episodes: 2725, Successos: 1594, Failures: 1131, Errors: 0
Contingency Table:
40

---

Published in Transactions on Machine Learning Research (12/2025)
DAR (Alg. B)
Success
Failure
GPS (Alg, A)
Success
1564 (a)
1107 (b)
Failure
30 (c)
24 (d)
Common episodes comparison: 2725
McNemar's Statistic: 1020.1662, p-value: 0.0001
Maze: 24x24
GPS (Algorithm A) Summary: Total episodes: 4999, Successes: 4675, Failures: 324, Errors: 0
DAR (Algorithm B) Summary: Total episodes: 4999, Successes: 1470, Failures: 3529, Errors: 0
Contingency Table:
DAR (Alg. B)
Success
Failure
GPS (Alg. A)
Success
1377 (a)
3298 (b)
Failure
93 (c)
231 (d)
Common episodes for comparison: 4999
McNemar's Statistic: 3029.2023, p-value: < 0.0001
Maze: 24x24 Obstacles 15%
GPS (Algorithm A) Summary: Total episodes: 4997, Successes: 4098, Failures: 899, Errors: 0
DAR (Algorithm B) Summary: Total episodes: 4997, Successes: 571, Failures: 4426, Errors:
Contingency Table:
DAR (Alg. B)
Success
Failure
GPS (Alg. A)
Success
482 (a)
3616 (b)
Failure
89 (c)
810 (d)
Common episodes for comparison: 4997
McNemar's Statistic: 3357.5517, p-value: < 0.0001
Maze: 24x24 Obstacles 25%
GPS (Algorithm A) Summary: Total episodes: 5000, Successes: 2443, Failures: 2557, Errors:
DAR (Algorithm B) Summary: Total opisodes: 5000, Successes: 252, Failures: 4748, Errors:
Contingency Table:
DAR (Alg. B)
Success
Failure
GPS (Alg. A)
Success
119 (a)
2324 (b)
Failure
133 (c)
2424 (d)
Common episodes comparison:
5000
McNemar's Statistic: 1953.7977, p-value: < 0.0001
This detailed breakdown for each environment when comparing GPS DAR shows the specific data under- lying McNemar's tests.
41

---

Published in Transactions on Machine Learning Research (12/2025)
H.1 Paired t-test Analysis
To complement the episode-level McNemar's test analysis, we conducted paired (-lests to assess the statistical significance of the differences in Average Success Rates (ASR) between our proposed method (GPS) and baselines. This analysis treats cach random seed as an independent experimental unit.
Methodology
The paired t-test is parametric test used to determine whether the mean difference between two sets of paired observations is significantly different from zero. In our context, each of the five random seeds provides a pairod observation: the ASR achieved by GPS and the ASR achieved by a bascline algorithm, when trained and evaluated under identical conditions.
For each maze environment, we computed:
The ASR for each algorithm across five independent seeds
The difference in ASR between GPS and the baseline algorithm for each seed
The mean and standard deviation of these differences
The null hypothesis (Ho) is that the mean difference in ASR bet ween GPS and baseline algorithm is zero. The test statistic is calculated as:
d
sa/Vi V
where is the mean the paired differences, $4 the standard deviation of the differences, and n is the number of pairs (seeds).
The significance level was set 0.05. We also report Cohen's d a measure of effect size, calculated
8a
Effect sizes are interpreted as small (d z 0.2), medium (d m 0.5), or large (d2 0.8).
Results: GPS DQN
The paired t-test results corroborate the McNemar's test findings, demonstrating statistically significant improvements of GPS over DQN in the majority of tested environments.
The following sections provide detailed results for each maze configuration.
Environment: 8x8:
GPS (%)
DQN (%)
Difference (%)
100.00
94.60
+5.40
100.00
84.80
+15.20
100.00
84.60
+15.40
100.00
81.30
+18.70
100.00
77.80
+22.20
Summary Statistics
100.00 +
0.00 0.00
84.62
+15.38 (mean)
Paired t-test Results
t-statistic 5.4864, p-value 0.0054 df = 4, 0.05 (significant)
Cohen's 's d = 2.4536 (large effect)
42

---

Published in Transactions on Machine Learning Research (12/2025)
Environment: 16x16:
GPS (%)
DQN (%)
Difference (%)
100.00
69.90
+30.10
100.00
67.00
+33.00
100.00
81.30
+18.70
100.00
74.00
+22.00
100.00
75.80
+-24.20
Summary Statistics
100.00 + 0.00 73.60 = 5.51
+26.40 (mean)
Paired t-test. Results
t-statistic 10. p-value 0.0004 df = 4, 0.05 (significant))
Cohen's d = 4.7893 (large effect)
Environment: 16x16 Obstacles 15%:
GPS (%)
DQN (%)
Difference (%)
91.90
84.80
+7.10
88.60
87.60
+1.00
97.10
81.40
+15.70
93.30
80.00
+13.30
97.10
86.20
+10.90
Summary Statistics
93.60 +: 3.62
84.00 +: 3.21
+9.60 (mean)
Paired t-test Results
t-statistic 3. 3.7255, p-value- = 0.0204 df = 4, a = 0.05 (significant)) Cohen's = 1.6661 (large effect)
Environment: 16x 16 Obstacles 25%:
GPS (%)
DQN (%)
Difference (%)
48.30
79.60
-31.30
50.50
72.10
-21.60
81.60
63.90
+17.70
41.80
71.40
-29.60
62.70
73.90
-11.20
Summary Statistics
56.98
15.71 72.18 + 5.64
-15.20 (mean)
Paired t-test Results
f-statistic = -1.6963, p-value = 0.1651
= 4, a = 0.05 (not significant))
Cohen's d -0.7586 (medium effect)
Environment: 16x16 Rooms:
43

---

Published in Transactions on Machine Learning Research (12/2025)
GPS (%)
DQN (%)
Difference (%)
91.00
64.70
+26.30
80.70
61.10
+19.60
80.40
66.20
+14.20
79.70
64.00
+15.70
94.20
66.20
+28.00
Summary Statistics
85.20 + 6.86
64.44 + 2.10
+20.76 (mean)
Paired t-test Results
I-statistic 7.5035, p-value 0.0017 df 4, a 0.05 (significant))
Cohen's d 3.3557 (large effect)
Environment: 16x16 16 Corridors:
GPS (%)
DQN (%)
Difference (%)
99.10
80.40
+18.70
93.80
80.40
+13.40
99.60
84.20
+15.40
100.00
77.60
+22.40
99.40
79.80
+19.60
Summary Statistics
98.38 + 2.58
80.48 2.38
+17.90 (mean)
Paired t-test Results
i-statistic 11.2894, p-value 0.0004 df = 4, = 0.05 (significant))
Cohen's d = 5.0488 (large effect)
Environment: 24x24:
GPS (%)
DQN (%)
Difference (%)
100.00
24.20
+75.80
77.40
27.00
+50.40
100.00
31.90
+68.10
99.90
31.30
+68.60
90.30
25.10
+65.20
Summary Statistics
93.52 + 9.94 27.90 + 3.53
+65.62 (mean)
Paired t-test Results
t-statistic 15.6753, p-value 0.0001 df 4, a 0.05 (significant)) Cohen's d 7.0102 (large effect)
Environment: 24x24 Obstacles 15%:
44

---

Published in Transactions on Machine Learning Research (12/2025)
GPS (%)
DQN (%)
Difference (%)
81.40
15.30
+66.10
82.90
10.80
+72.10
82.50
12.70
+69.80
83.80
10.70
+73.10
83.70
13.30
+70.40
Summary Statistics
82.86 + 0.98
12.56 + 1.91
+70.30 (mean)
Paired t-test Results
I-statistic 58.4011, p-value 0.0000 df = 4, a = 0.05 (significant))
Cohen's d 2671178 (large effect)
Environment: 24x24 Obstacles 25%:
GPS (%)
DQN (%)
Difference (%)
50.50
11.10
+39.40
22.90
10.10
+12.80
63.10
9.20
+53.90
58.70
11.60
+47.10
51.40
9.20
+42.20
Summary Statistics
49.32 15.67 10.24 = 1.09
+-3908 (mean)
Paired t-test Results
1-statistic 5.5703, p-value = 0.0051 df = 4, a 0.05 (significant)) Cohen's d 2.4911 (large effect.
Results: GPS TempoRL
The paired t-test results corroborate the McNemar's test findings, demonstrating statistically significant improvements of GPS over TempoRL in the majority of tested environments.
The following sections provide detailed results for each maze configuration.
Environment: 8x8:
45

---

Published in Transactions on Machine Learning Research (12/2025)
GPS (%)
TEMPORL (%)
Difference (%)
100.00
98.80
+1.20
100.00
91.00
+9.00
100.00
92.60
+7.40
100.00
95.20
+4.80
100.00
92.80
+7.20
Summary Statistics
100.00 0.00
94.08 = 3.04
+5.92 (mean)
Paired t-test Results
I-statistic = 4.3614, p-value 0.0120 df 4, 0.05 (significant))
Cohen's 1.9505 (large effect)
Environment: 16x16:
GPS (%)
TEMPORL (%)
Difference (%)
100.00
84.80
+15.20
100.00
86.80
+13.20
100.00
93.60
+6.40
100.00
91.60
+8.40
100.00
82.20
+17.80
Summary Statistics
100.00 + 0.00
87.80 = 4.73
+12.20 (mean)
Paired t-test Results
I-statistic 5.7691, p-value 0.0045 df = 4, 0.05 (significant))
Cohen's = 2.5800 (large effect.
Environment: 16x 16 Obstacles 15%:
GPS (%)
TEMPORL (%)
Difference (%)
91.90
81.90
+10.00
88.60
79.00
+9.60
97.10
79.00
+18.10
93.30
80.00
+13.30
97.10
80.00
+17.10
Summary Statistics
93.60 3.62
79.98 + 1.18
+13.62 (mean)
Paired t-test Results
1-statistic = 7.7640, p-value 0.0015 df 4, a 0.05 (significant))
Cohen's 3.4722 (large effect)
Environment: 16x16 16 Obstacles 25%:
46

---

Published in Transactions on Machine Learning Research (12/2025)
GPS (%)
TEMPORL (%)
Difference (%)
48.30
79.10
-30.80
50.50
74.10
-23.60
81.60
63.70
+17.90
41.80
67.20
-25,40
62.70
79.60
-16.90
Summary Statistics
56.98 = 15.71
72.74 7.10
-15766 (mean)
Paired t-test Results
I-statistic = -1.8108, p-value = 0.1444 df 4, 0.05 (not significant))
Cohen's 's -0.8098 (large effect)
Environment: 16x16 Rooms:
GPS (%)
TEMPORL (%)
Difference (%)
91.00
62.50
+28.50
80.70
55.60
+25.10
80.40
74.10
+6.30
79.70
72.70
+7.00
94.20
68.60
+25.60
Summary Statistics
85.20 + 6.86
8.670
+18.50 (mean)
Paired t-test Results
t-statistic 3. 3.7959, p-value 0.0192 dr = 4, 0.05 (significant))
Cohen's 's d 1.6976 (large effect)
Environment: 16x 16 Corridors:
GPS (%)
TEMPORL (%)
Difference (%)
99.10
89.70
+9.40
93.80
89.40
+4.40
99.60
93.90
+5.70
100.00
85.10
+14.90
99.40
88.30
+11.0
Summary Statistics
98.38 2.58
89.28 3.16
+9.10 (mean)
Paired t-test Results
1-statistic = 4.8169, p-value = 0.0085 df 4, a = 0.05 (significant))
Cohen's 2.1542 (large effect)
Environment: 24x24:
47

---

Published in Transactions on Machine Learning Research (12/2025)
GPS (%)
TEMPORL (%)
Difference (%)
100.00
50.60
+49.40
77.40
52.20
+25.20
100.00
60.80
+39.20
99.90
46,40
+53.50
90.30
51.20
+39.10
Summary Statistics
93.52 9.94
5.22
+41.28 (mean)
Paired t-test Results
I-statistic = 8.4008, p-value 0.0011 dr 4, a 0.05 (significant))
Cohen's 3.7570 (large effect)
Environment: 24x24 Obstacles 15%:
GPS (%)
TEMPORL (%)
Difference (%)
81.40
20.30
+61.10
82.90
12.80
+70.10
82.50
18.20
+64.30
83.80
17.60
+66.20
83.70
17.60
+66.10
Summary Statistics
82.86 0.98
17.30 2.75
+-6.56 (mean)
Paired t-test Results
i-statistic 44.8201, p-value 0.0000 df 4, 0.05 (significant))
Cohen's 's d 20.0442 (large effect)
Environment: 24x24 Obstacles 25%:
GPS (%)
TEMPORL (%)
Difference (%)
50.50
7.80
+42.70
22.90
8.00
+14.90
63.10
6.00
+57.10
58.70
7.80
+50.90
51.40
10.40
+41.00
Summary Statistics
49.32 = 15.67
8.00 + 1.57
+41.32 (mean)
Paired t-test Results
t-statistic = 5.7278, p-value = 0.0046
dr= 4, = 0.05 (significant))
Cohen's = 2.5616 (large effect)
Results:
GPS vs. DAR
The paired t-test results corroborate the McNemar's test findings demonstrating statistically significant improvements of GPS over DAR across all of the tested environments.
48

---

Published in Transactions on Machine Learning Research (12/2025)
The following sections provide detailed results for each maze configuration.
Environment: 8x8:
GPS (%)
DAR (%)
Difference (%)
100.00
76.60
+23.40
100.00
72.20
+27.80
100.00
78.60
+21.40
100.00
70.20
+29.80
100.00
71.20
+28.80
Summary Statistics
100.00 + 0.00 73.76 3.65
+-26.4 (mean)
Paired t-test Results
I-statistic = 16.0960, p-value- = 0.0001
= 4, a 0.05 (significant))
Cohen's d 7.1984 (large effect)
Environment: 16x16:
GPS (%)
DAR (%)
Difference (%)
100.00
66.40
+33.60
100.00
60.20
+39.80
100.00
68.60
+31.40
100.00
70.40
+29.60
100.00
65.20
+34.80
Summary Statistics
100.00 + 0.00
66. = 3.89
+33.84 (mean)
Paired t-test Results
I-statistic = 19.4676, p-value = 0.0000
= 4, a = 0.05 (significant))
Cohen's d = 8.7062 (large effect)
Environment: 16x16 Obstacles 15%:
GPS (%)
DAR (%)
Difference (%)
91.90
63.80
+28.10
88.60
55.70
+32.90
97.10
55.20
+41.90
93.30
52.40
+40.90
97.10
61.40
+35.70
Summary Statistics
93.60 + 3.62
57.70 + 4.72
+35.90 (mean)
Paired t-test Results
f-statistic 14.0337, p-value 0.0001 df = 4, a = 0.05 (significant))
Cohen's 's = 6.2761 (large elfect)
Environment: 16x16 16 Obstacles 25%:
49

---

Published in Transactions on Machine Learning Research (12/2025)
GPS (%)
DAR. (%)
Difference (%)
48.30
13.90
+34.40
50.50
19.40
+31.10
81.60
11.90
+69.70
41.80
19.40
+22.40
62.70
13.90
+48.80
Summary Statistics
56.98 + 15.71 15.70 3.47
+41.28 (mean)
Paired t-test Results
I-statistic 4.9846, p-value = 0.0076 df 4, a 0.05 (significant)) Cohen's d 2.2292 (large effect)
Environment: 16x16 16 Rooms:
GPS (%)
DAR (%)
Difference (%)
91.00
14.70
+76.30
80.70
12.60
+68.10
80.40
19.80
+60.60
79.70
8.50
+71.20
94.20
8.90
+85.30
Summary Statistics
85.20 + 6.86
12.90 + 4.64
+72.30 (mean)
Paired t-test Results
t-statistic 17.5163, p-value 0.0001 df = 4, 0.05 (significant))
Cohen's d = 7.8335 (large effect)
Environment: 16x 16 Corridors:
GPS (%)
DAR (%)
Difference (%)
99.10
62.80
+36.30
93.80
67.20
+26.60
99.60
59.40
+40.20
100.00
57.40
+42.60
99.40
59.10
+40.30
Summary Statistics
98.38 +: 2.58
61.18 +: 3.89
+37.20 (mean)
Paired t-test Results
t-statistic 13.1137, p-value = 0.0002 df = 4, a = 0.05 (significant))
Cohen's d 5.8646 (large effect)
Environment: 24x24:
50

---

Published in Transactions on Machine Learning Research (12/2025)
GPS (%)
DAR (%)
Difference (%)
100.00
30.20
+69.80
77.40
32.00
+45.40
100.00
27.80
+72.20
99.90
31.60
+68.30
90.30
25.80
+64.50
Summary Statistics
93.52 + 9.94
29.48 + 2.63
+64.04 (mean)
Paired t-test Results
I-statistic 13.2731, p-value 0.0002 df = 4, a = 0.05 (significant))
Cohen's d 5.9359 (large effect)
Environment: 24x24 Obstacles 15%:
GPS (%)
DAR (%)
Difference (%)
83.50
11.60
+71.90
82.90
11.60
+71.30
82.50
9.60
+72.90
83.80
10.80
+73.00
83.70
14.20
+69.50
Summary Statistics
83.28 + 0.56
11.56 + 1.69
+71.72 (mean)
Paired t-test Results
t-statistic 112. 112.2270, p-value 0.0000 df = 4, 0.05 (significant)
Cohen's d 50.1894 (large effect)
Environment: 24x24 Obstacles 25%:
GPS (%)
DAR (%)
Difference (%)
50.50
6.80
+43.70
23.60
5.20
+18.40
10
4.20
+58.90
58.70
4.60
+54.10
51.40
5.00
+446.40
Summary Statistics
49.46 + 15.37 5.16 + 0.99
+44.30 (mean)
Paired t-test Results
t-statistic 6.3132, p-value 0.0032 dr = 4, 0.05 (significant))
Cohen's 's d = 2.8233 (large effect.)
Baselines and Architecture
Each baseline is evaluated using grid search over multiple hyperparameter configurations; Tables 14 17
18
19 and 22 detail the specific value ranges for these parameters.
51

---

Published in Transactions on Machine Learning Research (12/2025)
All baseline models employ the same CNN feature extractor architecture followed by similarly sized linear layers, differing only in the final output layer size. For example, DQN outputs 4 Q-values (one per action); while DAR outputs 12 (4 actions x 3 repetition heads). TempoRL requires an additional network head to implement the skip policy, adding architectural complexity but gaining ficxibility in temporal decision- making. In our GPS method, the actor and critie networks each have their own separate CNN state feature extractors. future work, we plan explore a shared CNN feature extraction architccture as implemented in TempoRL, which could potentially improve computational efficiency and state representation learning.
For TempoRL, we configured the model with maximum skip length between 1..10 to allow variable sequence lengths of action repetition. For DAR, we evaluated possible coarse control values of 1,5,10 to allow the same maximum sequence length and mid-sequence capability, with the fine control value fixed at 1 to allow for actions at every time step. We based our implementations on the publicly available code at https: //github. wwwccomoctuttc htt///attmmmmmppaaee but reimplemented from scratch to enrich with more metrics and employ our ovaluation methodology. Detailed architectures, hyperparameter configurations, and implementation
specifics be found in Appendix K and N
While GPS, like hierarchical methods, targets long-horizon tasks, its approach is fundamentally different. Hicrarchical RL learns temporal abstractions via skill discovery and multi-level policies, whereas GPS gen- erates full action sequences directly from state observations without skills or sub-goal decomposition. The proto-sequence decoder is pre- trained and fixed, providing a structured output space rather than a learned controller. Because of these differences and GPS focus on discrete action spaces, we compare against DAR and TempoRL, which similarly extend temporal horizons through action repetition and sequence commit- ment.
J DQN Baseline Implementation Details
This scction outlines the architocture and configuration of the Deep Q-Network (DQN) agent used baseline. It details the neural network structure, hyperparameter settings, exploration strategy, optimization method, and other relevant training aspects.
J.1 Model Architecture (QNetwork)
The Q-Network is neural network designed to approximate the action-value function Q(s, It consists of convolutional part for feature extraction from the input observation and a linear part for producing Q-values for each action.
In our maze environments, as depicted Subsection 4.3 the input observation has shape (C, H, W), where the number input channels is The number of output channels, n_oupput channels, corresponds the number of available actions, which is 4 (right, left, up, down).
J.1.1 Convolutional Neural Network (CNN) Part
The CNN component processes the input observation through a sequence of convolutional layers:
Conv2D Layer 1:
Input channels:
Output channels: 16
Kernel size: 2
Stride:
2. Activation: ReLU
3. Conv2D Layer 2:
Input channels: 16
Output channels: 32
52

---

Published in Transactions on Machine Learning Research (12/2025)
Kernel size: 2
Stride:
4. Activation: ReLU
5. Conv2D Layer 3:
Input channels:
Output channels: 64
Kernel size: 2
Stride:
6. Activation: ReLU
7.
The output of the convolutional layers is flattened into 1D vector. The size of this vector, flatten, is computed automatically.
Flatten Layer:
J.1.2 Linear Part
The flattened output from the CNN (n_flatten) is fed into a sequence fully connected linear layers:
The hidden layer sizes are configurable via grid search (see Table 14 for details). The activation function for these hidden layers is Leaky ReLU (negative slope 0.1).
The final linear layer maps the last hidden layer's output to n outpul. channels (4 actions).
J.2
Hyperparameters
The agent's behavior and training process are governed by set of hyperparameters, detailed in Tables 14 and 16
Table 14: General Experiment Hyperparameters for DQN Bascline
Parameter seed derch_ddeermiiittic
Default Value
123
True
Lave_model. strategy stt_tdee_strttegy
SUCCESS_ RAAEE
bal_eval_ffee
5000
train_eval_ eval_test, wasaaaasttttaannnnnne
freq
5000
100000
Table 15: Environment-Spocife Hyperparamoters for DQN Bascline
Parameter
Default Value
max episodeeipee__eeeee steps
75
reward_straaee
NEGATIVE BASED ON MAX LEVEL WITH PENALTIES
obscrvation crctiioo__cooiin__ttteee
DEFAULT
Max Path Length (maz level)
Varies (see Env. Def. in Table
Min Path Length (start level)
Varies (see Env. Def. in Table
J.3 Epsilon-Greedy Exploration
The agent uses epsilon-greedy strategy for action selection. The value of epsilon (c) is linearly nealed from start_e (1.0) to end_e (0.1) over duration. This duration is calculated [exp frac where frac is the selected exploration_fracee is the current global timestep. The epsilon at timestep f is:
(from options in Table 16 and
f = max
start e)/duration) x start end e)
With probability 6, random action is chosen; otherwise, the action with the highest Q-value is selected.
53

---

Published in Transactions on Machine Learning Research (12/2025)
Table 16: Algorithm Specific Hyperparameters for DQN Baseline
Parameter
Default Value /Options
total timesteps
Environment specific
learning rate _rate
[] x 10-3, 3, x 40
size
[10000, 50000]
? (discount factor)
T (target update rate)
[0.01, 0.005]
target network frequency
[10, 100] (soft-target update freq.)
batch size
256
start_e
1.0 (initial e)
end_e exploration fraction
0.1 (final s)
[0.1, 0.3, 0.5]
learning starts
1000 (timestep to begin learning)
train frequeery linear layers activation_ function
(Q-network update freq.)
["512,128,321, "512,32"]
Leaky ReLU (slope 0.1)
J.4 Optimizer
The Q-Network is trained using the Adam optimizer (torch. by the learning_rate hyperparameter (sec Table 16..
The learning rate is controlled
J.5 Replay Buffer
A replay buffer
common. wuffers..feeegeeeeeee RepllyrreRR))))uuffer)
stores
experiences
(St, re, $141, de).
The buffer size is specified in Table 16
seciiiiie_mmmrry.ooom Key configurations include optimize, memory. usage False and
handle handle_timeout timeout_termmnationn False.
J.6 Training Details
Loss Function. The Q-Network paramcters (f) are updated by minimizing the Mean Squared Error (MSE) loss:
L( = [Es.aa,s [(y a()
where the TD target:
yt = rt + max ( 0(1 -d)
Here, is the reward, 7 is the discount factor, Qtarget is the target network with parameters 0=, and d, indicates if 8141 is terminal. This is implemented via torch. mn. functionnal mss_ loss.
Target Network. A separate target network Qtarget with parameters A- stabilizes training. Its weights are updated using Polyak averaging: 9 <- 78 + (1 T)0-. The soft update rate 7 and update frequency target_neet network _tt_eeookfffreuuecccc are specified in Table 16
Training Procedure.
Learning Starts: Training begins after learning starts timesteps (sec Table 16
Training Frequency:
The Q-network is updated every train_frequent global steps (see Table
16].
Batch Size: Number of experiences sampled per training slep is batch_size size (see Table 16
J.7 Evaluation
The agent's performance is evaluated periodically validation and test datasets.
54

---

Published in Transactions on Machine Learning Research (12/2025)
Evaluation the validation dataset occurs every val_eval _eeel_rree steps.
Evaluation on the test dataset can occur during training every aval_test, @eal_tes__daaaett.ee steps.
freq
During evaluation, actions are chosen greedily (or with a small fixed epsilon, e.g., 0.05 or 0.0).
Metrics logged include mean episodic return, success rate, and agent step ratio.
Model saving is based performance metrics (c.g., highest success rate reward validation) per save_mo geve_model_stratee
DAR Baseline Implementation Details
This section outlines the architecture and configuration of the Dyanmic Action Repetition (DAR) agent used as baseline. The DAR agent builds upon the Deep Q-Network (DQN) architecture and training methodol- ogy. Therefore, for aspects not explicitly mentioned here, such as the general experiment configuration (Table
14 environment--peciicc arguments (Table 15 epsilon-greedy exploration strategy (Section J.3), optimizer (Section J.4h. replay buffer (Section J.5 general training procedure (Section J.6). and evaluation methodol- ogy (Section J.7 please refer to the corresponding descriptions in the DQN baseline implementation details (Section J
The primary distinctions of the DAR baseline are its modified network architecture to support an expanded action space and an additional algorithm-specific hyperparameter, dar aat_r__ r_l,, related to action repetition.
K.1 Model Architecture
The DAR network for the DAR. agent, similar to DQN, approximates the action-value function Q(s, a). comprises a convolutional part for feature extraction and a linear part for producing Q-values.
The input observation from the maze environments has shape H,, W), where C 3, identical to the DQN baseline (Section J.1
K.1.1 Convolutional Neural Network (CNN) Part
The CNN component is identical to the used in the DQN baseline. For details the architecture (number of layers, channels, kernel sizes, strides, and activations), please refer to Section J.1 this part is a flattened ID vector of size n flatten.
The output of
K.1.2 Linear Part
The flattened output (m_flatten) from the CNN is processed by sequence of fully connected lincar layers:
The hidden layer sizes are configurable via grid search, with the same options as the DQN basclino (see Table 17 for iinear_layere, The activation function for these hidden layers is Leaky ReLU (negative slope 0.1).
The final linear layer maps the last hidden layer's output to n output channels. For the DAR agent, n wwtoouutt_choheeee chonnels = 12, corresponding to base actions (right, left, up, down) each associated with repetition heads/levels.
K.2 Hyperparameters
The general experimental configuration and swvioonment-ppecife hyperparameters for the DAR baseline are the same those for the DQN baseline, detailed in Table 14 and Table 15 respectively.
55

---

Published in Transactions on Machine Learning Research (12/2025)
K.2.1 Algorithm Specific Arguments
The algorichm-specific hyperparameters for the DAR baseline, including the newly introduced dar dar_r_] pa- rameter, are listed in Table 17 These parameters are subject to grid search to find the optimal configuration for each environment.
Table 17: Algorithm Specific Hyperparameters for DAR Basclinc
Parameter
Default Value / Options
lotal limesteps
Environment specific
learning
_rat rate
[1 x 10-3, x 10-
buffer size
[10000, 50000]
(discount factor)
0.99
(target update rate)
[0.01, 0.005]
newwork frequeec__eeeyy
[10, 100] (soft-target update freq.))
batch size
256
start_e
1.0 (initial 6)
end_e exploration fraction
0.1 (final e)
[0.1, 0.3, 0.5]
learning starts
1000 (timestep begin learning)
train train_ffrquue frequency
(Q-network update freq.)
linear layers activation detivation_fuunctinn function
[2512,128,32*, *512,32"]
Leaky ReLU (slope 0.1)
dar
[1, 5, 10) (repetition level parameter)
K.3 Training Details
Loss Function. For DAR, the Q-Network parameters (0) are updated by minimizing Hubber loss. This is implemented via forch..n.SmoothilL..
L TempoRL Baseline Implementation Details
This section describes the architecture and configuration of the TempoRL agent, a baseline designed for temporal abstraction by learning how long to repeat actions. TempoRL shares several components and pro- cedures with the DQN baseline. For details on the general experiment configuration (Table 14p environment-
specific arguments (Table 15). replay buller (Section J.5) and evaluation methodology (Section J.7. please
refer the corresponding descriptions in the DQN baseline implementation details (Section
Key distinctions of the TempoRL agent include its specialized network architecture with separate heads for action selection and skip duration, unique hyperparameters related to these mechanisms (skip dim, weight_ shggt_sharing), and the use of Huber loss for training.
L.1 Model Architecture
The TempoRL network processes input observations to produce Q-values for primitive actions and Q-values for skip durations. The input observation from the maze environments has shape (C, H, W), where C 3, identical to the DQN baseline (Section J.1
L.1.1 Convolutional Neural Network (CNN) Part
The CNN component used for initial feature extraction is identical to the one in the DQN bascline. For details on its architecture (number of layers, channels, kernel sizes, strides, and activations), please refer Section J.1 The output of this CNN part is flattened 1D vector of size flatten.
56

---

Published in Transactions on Machine Learning Research (12/2025)
L.1.2 Linear Heads for Action and Skip Policies
Following the CNN, the network processes features through structure that leads to two distinct output heads: one for action selection and one for determining the skip duration. The MLP for each pathway (from CNN output to pre-output layer) consists of layers with output units [512, 128, 32].
Feature Processing and Weight Sharing:
If weight_sharing- True (default configuration): The n p_ffatten vector is first processed by shared linear layer producing 512 output unils, followed by Leaky ReLU activation (negative slope 0.1). This 512-umit feature vector serves as the common input to the subsequent differing layers of the action and skip heads.
Tf weight_sharing False: The n flatten vector is independently fed into the first linear layer (512 output units, Leaky ReLU) of both the action and skip processing streams. Each stream then continues with its own [128, 32] layers.
Action Head:
Starting from the 512-unit ura vector (either shared or head-specific), it is processed through two subsequent linear layers with 128 and 32 output units, respectively. Each of these hidden layers uses Leaky ReLU activation (negative slope 0.1).
The final linear layer of the action head maps the 32-unit feature vector to __outppt actions Q-values, where n outpput octions = (corresponding to right, left, up, down).
Skip Head:
Similarly starting from the 512-unit feature vector, it is processed through two subsequent lincar layers with 128 and 32 output units, each followed by Leaky ReLU activation (uegative slope 0.1).
The final linear layer of the skip head maps the 32-unit feature vector to skip_dim. Each corresponds to the utility of repeating the chosen primitive action for a specific number of steps, from up to skip_dim.
L.2
Hyperparameters
General experimental configuration (Table 14) and environment-spocffic arguments (Table 15 are consistent with the DQN baseline. Algorithm-specific hyperparameters for TempoRL, including those unique to its architecture, detailed in Table 18
L.3 Action Selection and Exploration
TempoRL employs two-step e-greedy strategy for exploration and action selection:
Primitive Action Selection: Given the current state 8t, a primitive action or (e.g., right, left, up, down) is chosen. With probability e, a, is selected randomly from the set of n output _ctionss. Otherwise (with probability 1 c), = argmax, a(; where Qs,,, x are the Q-values produced by the action head of the online network.
2. Skip Duration Selection: Conditioned on the current state s; and the chosen primitive action de a skip duration k, (number of times to repeat a,, from to skip_dim) is selected. With probability baki__dim], Otherwise, k argmax Qekip az, k': Oswip).
k2 is chosen randomly from where (st at, (skip) are the Q-values for different skip durations produced by the skip head (which might use shared parameters weight_sharing = True).
The selected primitive action a, then exccuted in the environment for k, consecutive timesteps. The value of e is typically linearly annealed from start_ e end_e e over exploration fraction detailed for the DQN baseline (see Section J.3 and Table 18
fractio of total timesteps,
57

---

Published in Transactions on Machine Learning Research (12/2025)
Table 18: Algorithm Specific Hyperparameters for TempoRL Baseline
Parameter
Default Value Options
total timesteps
Environment specific
learming rate
[ x 10 1 x 10-*)
buffer size
[10000, 50000]
gamma (7)
0.99 (discount factor)
lau (7)
[0.01, 0.005] (larget network update rate)
target__nttwaatt network frequency
[10, 100] (frequency of applying soft target network update)
batch_sii
256
slart
1.0 (starting epsilon for exploration)
end_ exploration_ fraction
0.1 (ending epsilon for exploration)
[0.1, 0.3, 0.5]
learning starts
1000 (timestep to start learning)
train frequency
2 (frequency of training the Q-network)
activation ffnctiin skip dim weight sharing
Leaky ReLU (negative slope 0.1 for hidden layers)
10 (maximum skip size)
True (whether to share the first. 512-unit layer)
L.4 Optimizer
Separate Adam optimizers (torch. owthh..pmmmmm..m..... Adam) are used for the action Q-network parameters and the skip Q-network parameters. The learning rate for both optimizers is controlled by the learning. rate hyperparameter (see Table 18 Gradients for both networks are clipped ggad_cclip clip_vaill val 40.0).
L.5 Replay Buffers
TempoRL utilizes two distinct replay buffers with capacity buffer_siee size (sec Table 18 to store experiences for training its action and skip policies:
Action Replay Buffer: This is a slandard replay buller RRelaaBuffeer from Stable Baselines3) that stores transitions corresponding to individual primitive actions. Each experience tuple is of the form (St, Oe, Tt, 8t41,d)) where:
The state at time
ae: The primitive action taken at time t.
Pr: The reward received at time 1.
8t+1: The state at time t ++1. 1.
d:: A boolean flag indicating if 8141 terminal state.
Experiences sampled from this buffer are used to train the action Q-network (the action head).
Skip Replay Buffer: This a custom replay buffer (referred to as NoneConcaatStiipRelllayyfferrr in the implementation) specifically designed to store experiences related to the execution of multi-step skip actions. Each experience tuple is of the form (8); kij, 83tkys R,, djt Iden, a abhhar), representing:
8y: The state from which the skip action (repeating heehav commenced.
ky: The selected skip duration (i.e., the 'action' taken by the skip policy).
8j-k,: The state reached after the primitive action a ahekav was executed kjen times.
Ry: The accumulated (and potentially discounted, depending on exact calculation before stor-
age) reward received over the course of the k/en steps of the skip.
dytk;: A boolean flag indicating if sytk, (the state after the skip) is terminal state, ky (length): The actual number of steps the primitive action heehan was repeated (this is equivalent to ky).
58

---

Published in Transactions on Machine Learning Research (12/2025)
b The underlying primitive action that was chosen to be repeated for knm steps. 0j
Experiences sampled from this buffer are used to train the skip Q-network (the skip head).
L.6 Training Details
TempoRL involves separate training updates for the action Q-network and the skip Q-network, both utilizing the Huber loss function.
L.6.1
Loss Function and Updates
The network parameters are updated by minimizing the Huber Borch.m.Smmothh.ILss) for both action and skip prodictions.
loss
(This
implemented via
Action Q-Network Update: Experiences (sj, (j, rj, 85+1, dj) are sampled from a standard replay buffer. The target value paction computed using a Double DQN-style approach:
sgetion =r, + - dj)Qtarget (8)+1, arg max (';))=)
where Q is the online action Q-network with parameters 0, and Qtarget is its target network with parameters 0- The loss is then:
Laction (0) (e (13 HuberLoss(y]] gy - ((ss,;;;)))
az;
Skip Q-Network Update: Experiences (8is aaekav ký, R2 85thy dj+t,) are sampled from a sepa- rate replay buffer for skips. Here, d; abchav is the primitive action executed, ky is the skip duration (number of times a was repeated), R2 is the accumulated discounted reward during these ky steps, and 8itk, is the state after ky steps. The target value vi S is calculated as:
5 pkipp = Ry + gt= ditk, Qtarrget (8j4ky arg max d';;
Note that the future value component Qtarget ( (8ytky....) uses the main action Q-network and its larget, reflecting the value of the optimal next primitive action after the skip concludes. The current prediction is Qakip (8y, a some parameters with 0 if triggt_shaiinggg True). The loss is:
abehav ky; Oskip) where Qskip has parameters Oskip (which may share
L0ksp(0 = Eq... BBB.
HuberLLossyy, tkip Qokip a
behav kyj 0kkss))
Both loss functions are optimized using their respective Adam optimizers, and gradients are clipped to prevent large updates.
L.6.2 Target Network Updates
To stabilize training, target networks are employed.
target network Qtarget (with parameters 0-) is maintained for the primary action Q-network Q (parameters 0). This Qtorget is always updated using Polyak averaging: 0 +-00 + 71-
weight_sharing 'False', the distinct skip Q-network Qskip (parameters Oskip) has its own separate target network, Qskip_targer (parameters 0 0uss))- This Qekip_target is similarly updated using Polyak averaging with Oakip and 0 "kipp"
weight_ reight_sharing is 'Truc', the parameters of the skip mochanism are part of the overall network structure whose online parameters are 0 (which includes the shared trunk and potentially specific skip head layers not part of the action head). In this scenario, a separate Polyak update for a distinct Qskip target is not performed; target values for the skip component's loss are derived using Qtarget for estimating future state-action values, as shown in the skip larget formula.
The soft update rate and the update frequency target_networkd network saaeee__ettrr_ffeeeeeeyee are specified in Table 18
59

---

Published in Transactions on Machine Learning Research (12/2025)
M Proto Sequence Decoder (PSD) Implementation Details
This section outlines the architecture and configuration of the Proto-Sequence Decoder (PSD) module used in our model. It describes the decoder network structure, training objectives, latent space regularization, sequence reconstruction process, and other relevant design choices that enable effective decoding of proto- sequence emboddings into action-sequence.
M.1 Training Dataset Setup
To ensure that the model learns from meaningful and structured data rather than arbitrary noise, constructed the training dataset according to the following constraints:
Sequence Length Constraint: All action sequences have lengths between and Iymax = 10 steps.
Action Diversity Constraint: Each sequence includes most two distinct action types. For example, valid sequences include [up, up] or [up, left], whereas a sequence like [up, leftl down] is considered invalid.
3. Switch Constraint:
Each sequence may contain at most one switch between action types. For instance, (up, left] is allowed, but (up, left, up) is not.
Directional Conflict Constraint: Sequences cannot include opposite directions, such as both up and down, or both left and right.
5. Avoidance of loops.
Following these criteria, we generated a total of 400 unique action sequences. These sequences were one-hot encoded and padded with EOS tokens to a fixed length of Lmaz to suit VAE input requirements. These sequences form the basis of the training data for the Proto-Sequence Decoder (PSD).
M.2 Model Architecture
The Proto-Sequence Decoder (PSD) is implemented as Variational Autoencoder (VAE) designed to map prolo-sequence embeddings into action sequences of varying length. Tt is trained using reconstruction loss combined with a Kullback-Leibler divergence (KLD) regularization toward a standard Gaussian prior. The decoder operates on flattened onc-hot sequence representations of actions and outputs reconstructed se- quences over predefined action vocabulary.
M.2.1 Input Representation
Each action in the sequence is represented as onc-hot vector over vocabulary of size Towords = 5 correspond- ing to {up, down, right, left, eos token). The decoder models sequences of up to input_ length = 10 actions, resulting in an input vector of dimension 10 5 = 50.
M.2.2 Encoder Network
The encoder receives flattened 50-dimensional imput vector and passes it through series of fully connected layers:
Linear Layer
Input size: 50
Output size: 32
Normalization: InstanceNorm
Activation: LeakyRel.U (slope 0.2)
60

---

Published in Transactions on Machine Learning Research (12/2025)
2. Linear Layer 2:
Output size: 16
Normalization: InstanceNormid
Activation: LeakyReLU
3. Linear Layer 3:
Output size: 16
Normalization: InstanceNormid
Activation: Tanh
The output is then projected into two parallel linear layers to produce the latent mean RR and log- variance log o² E IR 16 latent sample 2 is drawn using the reparamctcrization trick: 2 = f + a where ~~ 1).
M.2.3 Decoder Network
The sampled latent vector R16 is decoded through symmetric feedforward network:
Lincar Layer 1:
Output size: 16
Normalization: InstanceNormid
Activation: LeakyReLU
Linear Layer
Output size: 32
Normalization: InstanceNormid
Activation: LeakyReLU
Linear Layer
x$
Output size: 50 (reconstructed sequence)
Normalization: InstanceNormid
Final Activation: Sigmoid (applied element-wise)
M.3
Training Objective
The PSD is optimized using a combination of:
Reconstruction Loss:
Binary cross-entropy loss between the inpul sequence and its reconstruction. normalized by sequence length.
KL Divergence Loss: Encourages the latent distribution to match a unit Gaussian prior.
All input sequences are EOS-pddded to the maximum length of 10 to ensure uniform input dimensionality across batches.
M.4 Hyperparameters
The training of the Proto-Sequence Decoder (PSD) is governed by of fixed hyperparameters, detailed in Table 19 These parameters control aspects such optimization, batch processing, and reproducibility.
61

---

Published in Transactions on Machine Learning Research (12/2025)
Table 19: Proto-Sequence Decoder (PSD) Training Hyperparameters
Parameter
Value
train Secin_on_entti antire_datase
True
seed
42
optimizer
Adam
@ooimizer_leariing
x 10
oprinizer_weigjt__deeea
10-3
batch size
32
M.5
Optimizer
The PSD is trained using the Adam optimizer ftorch.optin Adam). The learning rate is controlled by the learning_rate rate hyperparameter (see Table 19
M.6
Training Details
M.6.1
Loss Function
The Proto-Sequence Decoder (PSD) parameters are updated by minimizing a combined loss:
L = Lree + LKL.
where:
"Troe is the label-smoothed binary cross-entropy over the reconstructed sequence:
7 Σ yt log Dr (1 31) log(1 b.))
Lroc
with yt replaced by 9. = yc(1 - + 2. e 0.1, and = 10 the sequence length.
Lkt. the Kullback-Leii divergence between the approximate posterior and a unit Gaussian:
LKL. DRL. (M ()).
M.6.2 Training Procedure
Maximum Steps: Train for up to 20,000 epochs.
Batch Composition: Split sequences into
Short (< 5 actions) and
Long (> actions),
and sample each batch with a 50/50 ratio of short and long sequences.
M.7 Evaluation
The Proto-Sqquence Decoder's (PSD) performance was assessed on the entire training set using two key metrics. Evaluations were conducted every 50 epochs, and the checkpoint yielding the highest exact match accuracy was retained. After 20,000 epochs, the following results were achieved:
Exact Match Accuracy: This metric measures the proportion of sequences reconstructed with errors. The PSD achieved Fxact Match Accuracy of 0.978.
This metric calculates the fraction of correctly reconstructed actions across all positions within the sequences. The PSD achieved Per-Step Accuracy of 0.99.
Per-Step Accuracy:
62

---

Published in Transactions on Machine Learning Research (12/2025)
M.8 Visualization of the Learned Embedding Space
Figure 6 illustrates the two-dimensional t-SNE projection of the learned proto-aciion-ssqquenee embeddings. Each point corresponds to one sequence from the dataset, with colors representing the effective sequence length. As can be seen, sequences with similar structural propertics tend to form dense clustors, indicating that the embedding space preserves meaningful relationships between sequences.
A particularly noteworthy observation is the position of the red star, which represents a previously unseen sequence included during training. This sequence located within the cluster of its closest structural neighbors, suggesting that the learned representation generalizes effectively to new data. In other words, the embedding model is able to position novel sequences near the most similar examples from the training set, supporting its potential for robust retrieval, similarity search, and downstream predictive tasks.
T-SNE 2D representation of entire sequence dataset coloured by sequence length
20
10
8,3,0,0,0,0,0,00000
Second Dimension
-10
3,3,3,0,000000000000
Action Sequenice
Length
-20
3,3,3.0,0,000,33333
-20
-10
20
First Dimension
Figure 6: T-SNE 2-D projection of the proto-action-sequenc embedding space. The map contains embed- dings for the 400 original sequences the dataset together with one previously unseen sequence (red star). Points are colored by their effective sequence length, and sequences with similar structure form tight neighborhoods. The unseen sequence falls naturally inside the cluster of its closest structural neighbors, showing that the learned representation places new, nut-of-training sequences adjacent to the most similar known examples.
N GPS (Generative Proto Sequence) Implementation Details
This section outlines the architecture and configuration of our GPS (Generative Proto Sequence) method. GPS is actor-critic based algorithm where the actor network generates latent representation, termed "proto-sqquence" This proto-sequence is then processed by a pre-trained generative decoder model (PSD) to produce a sequence of discrete actions. A critic network evaluates this action sequence to guide learning. The subsequent subsections detail neural network structures, hyperparameter settings, exploration strategy, optimization methods, and other relevant training aspects.
N.1 Model Architecture and State Representation
The GPS agent consists of three primary neural network components: an Actor, a Critic, and a pre-trained Decoder. It processes observations from the environment. For details on the specific state representation, input shape (c.g., height, width, channels), and preprocessing, please refer the "Stte Representation' within Section 4.3
63

---

Published in Transactions on Machine Learning Research (12/2025)
N.1.1 Convolutional Neural Network (CNN) Part
Both the Actor and Critic networks use separate but identical Convolutional Neural Network (CNN) ar- chitecture to extract Teatures from the input observation. The CNN component used for initial feature extraction identical the one in the DQN baseline. For details on its architecture (number of layers, channels, kernel sizes, strides, and activations), please refer to Appendix J.1 The output of this CNN part is flattened 1D vector of size n n_ftten..
Note: in our current implementation, the actor and critic networks each have their own separate CNN state feature extractors. In future work, we plan to explore a shared CNN feature extraction architecture as implemented in TempoRL, which could potentially improve computational efficiency and state representation learning.
N.1.2
Actor Network
The Actor network takes the extracted features from the CNN and produces proto-sequence embedding.
Input:
The flattened feature vector Rifatures from the CNN concatenated with a positional en- coding. The embedding dimension this positional encoding is specified by pe_embedding_
Architecture:
The combined features are processed through a series of fully connected linear layers, defined by the actor aiterrlieerr___ layers parameter (e.g., [512, 128, 32)). The activation function for these hidden layers is specified by actor 1inear lawers_activattieee stntrrattttttttofffnccinnnn function which is "leaky_ relu" with negative slope 0.1.
layer
The actor generates a single proto-soquence embedding. This embedding is a vector of size actor_n_ channels and serves as input to the Decoder PSD network.
Output:
N.1.3 Position Encoding in Proto-Seet Seat--e Generation
To enhance the expressiveness of our action sequence generation, we incorporate positional encoding within the Actor network. This technique, inspired by transformer architectures (Dosovitskiy al. 2020), helps the Actor generate more contextually aware proto-sequence embeddings by providing explicit spatial information about the agent and goal positions.
Implementation Details. Our positional encoding implementation combines both row and column infor- mation for cach grid cell in the observation space:
We create sinusoidal encodings for both dimensions (height and width) separately:
pe_row[;, 0::2] torch.sin
div_term)
pe_rou[:, 1::2] torch. cos cos
(position_rov div_term
2. These encodings are combined into a unified representation where the first half of each cell's embed- ding encodes its row position and the second half encodes its column position.
During forward passes, we extract the agent and goal positions from the observation and retrieve their respective positional encodings:
agent_pe self iearitiiooeenoiinnniiin goal_pe pe self, politiossiionnoooo wrw.iiionnenodiinggegeee (goal_rox goal_col]
agent_coll
4. These position-specific features are concatenated with the CNN-cxtracted features before being processed by the linear layers of the Actor
Motivation and Benefits. Integrating positional encoding within the Actor network provides several advantages:
64

---

Published in Transactions on Machine Learning Research (12/2025)
Table 20: GPS with vs. without Positional Encoding
ASR at Training Steps
ASR Couverge
Positional Encoding
Environment
>0.9 Step
PER
SGF
100K
500K
1M
1.5M
With
0.22
0.96
0.99
0.96
300k
0.72
10.8
[6x16__bstadel 15%
Without
0.15
0.38
0.51
0.52
>155M
0.62
17.67
With
0.12
0.45
0.80
0.91
1.5M
0.48
6.34
24x24 obstacles
15%
Without
0.07
0.12
0.13
0.13
>1.5M
0.54
16.91
Enhanced Spatial Reasoning: By explicitly encoding agent and goal positions, the Actor can better understand spatial relationships, which crucial for navigation tasks.
2. Improved Exploration Early in Training: The position encodings enable the generation of more diverse proto-sequence embeddings in the initial training phases, facilitating better exploration before the CNN features become well-trained.
3. Direction-Aware Sequence Generation: The relative positions of agent and goal inform the Actor about the general direction of movement required, allowing it to generate more purposeful action sequences even with limited experience.
Invariance to Visual Feature Quality: be unreliable, position encodings provide a stable signal that enables meaningful action sequence generation.
Espocially early in training when the CNN features may
Our manual investigations and targeted experiments suggest that incorporating position encodings enhances the model's capabilities in several ways. We observed that the position-enriched Actor generates proto- sequence embeddings with greater contextual awareness of spatial relationships, which in turn produces more diverse and situation-approoriate action sequences. Without position encoding, the proto-sequence embeddings tended to cluster more closely in the latent space, resulting in less differentiated action patterns. This difference was particularly evident in larger and more complex maze environments, where the position- encoded model demonstrated an improved ability generate directionally coherent sequences that efficiently navigated toward goals. The positional information appears to provide structural prior that helps the Actor formulate meaningful navigation strategics even before the CNN fcatures are fully refined through training.
Quantitative Ablation Study. To provide empirical evidence for the benefits of positional encoding, we conducted controlled ablation study comparing GPS with and without positional encoding on two challenging environments. Table 20 presents the comparative results.
The ablation results provide strong quantitative evidence for positional encoding's critical importance. GPS with positional encoding achieves convergence (ASR 0.9) in 300K-L.5M steps, while the variant without positional encoding fails to converge after 1.5M steps. Performance gaps substantial: 44 percentage in the 16x16 16 with 15% obstacles environment (0.96 vs 0.52 final ASR) and 78 percentage in the 24x24 with 15% obstacles enviromment (0.91 vs 0.13 final ASR). Path efficiency results show mixed patterns, with positional encoding improving PER by 0.10 in the 16x16 with 15% obstacles environment (0.72 vs 0.62) but showing lower PER in the 24x24 with 15% obstacles environment (0.48 vs 0.54). However, this apparent PER advantage for the variant without positional encoding misleading, as its extremely low ASR (0.13) indicates fundamental task learning failure likely only succeeds on the casiest instances, artificially inflating the path efficiency metric. Both environments show substantial reductions in sequence generation frequency with positional encoding: 39% reduction in 16x16 (10.8 vs 17.67 SGF) and 63% reduction in 24x24 (6.34 vs 16.91 SGF), indicating more coherent action sequences requiring fewer decision points.
N.1.4 Critic Network
The Critic work estimates the Q-value of state and decoded action sequence.
65

---

Published in Transactions on Machine Learning Research (12/2025)
Input:
The flattened feature vector Mettures from the CNN, representing the current state.
The action sequence generated by the Decoder from the actor's proto-sequence. The represen- tation used for this action sequence is SPIION_SEQ_AS_OR _OOT Namely, the input is tensor where cach action in the sequence of length n a_actions_ _in seq is one-hot encoded. Shorter sequences are padded to this length using an End-of-Sequence (EOS) token. Each one-hot vector has dimen- sion equal to the srrioonsppce_eeee plus one (for the EOS token). Consequently, the total input dimension for the action sequence part, action saiioo_see_din is n. n_actions, in in_seq x (action_space _ize + 1).
The inputs are processed through series of fully connected linear layers, de- fined by critic_limear layers (e.g., [512, 128, 32]). The activation function is specified by Iritac Latyerrsaactiratioootttttttaaaoe which is 'leaky_ relu' with negative slope 0.1.
Architecture:
Output: A single scalar Q-value. The output Q-value be optionally clipped between siin_gf_vee value and max_af_vallluu value.
N.1.5 Decoder Network
A pre-trained generative model, specifically a Variational Autoencoder, acts as the Decoder (PSD).
Loading:
The Decoder is loaded from a pre-trained model specified by
decoder_modee _aaedeer_nde__pthh__
Input: The proto-sequence embedding (sizc actor a_output, o_ooupput_chameeee generated by the Actor.
Output: A sequence of i_aatiios_inn_aee discrete actions. Each action is selected from vocabulary token.
of action_space_ntt size primitive actions plus mnd_of_sequence_
Generation: The Decoder can use Gumbel-Softmax for differentiable sampling if wtt_g_llee__deeceeeee true, or deterministic argmax with Straight- Straight-Through Through Estimator otherwise. See explanation in Section 4.2
N.2
Hyperparameters
The GPS method configured by wide range of hyperparamctens. General experiment settings and enviromment-ppecffee configurations are typically managed as detailed for the DQN bascline (sce Tables 14
and 15 Key algorithm-speeific hyperparameters for GPS listed in Tables 21 22
N.3 Reward Function and Empty Sequence Handling
The GPS agent utilizes a structured reward function designed to encourage efficient navigation while penal- izing inefficient invalid behaviors. The reward function is defined as:
3
R = Tgoal
Devald
X Pinvalid
Imax
Imax
where Fgoa is if the agent reached the goal (0 otherwise), mmax is the maximal start-goal distance acting regularizer, Avalid is the number of valid actions taken, and Niavalid is the number of invalid actions (e.g.. bumping into wall).
To handle cases during training where the actor generates prolo-sequences that are decoded as emply se- quences (i.e., where the PSD outputs the EOS token as the first action), we apply a harsh penalty -20. For these instances, we also hard-code the action be (DOWN) to ensure the agent always takes some action. This direct negative reinforcement was found be highly effective in guiding the actor to produce valid proto-sequence embeddings that decode into meaningful action sequences. Without this penalty, the
66

---

Published in Transactions on Machine Learning Research (12/2025)
Table 21: Algorithm Specific Hyperparameters for GPS
Parameter
Value / Options / Description
total _timesteps
Env. specific
actor learningg_ate
Actor LR (e.g. 10 10"*).
critic_ Baattic_lerrningraae
Critic LR (e.g., 1 10 +).
buffer_sine _siee
[10000, 50000] Discount factor.
gamma (7)
tau (T)
Target net. soft update rate (e.g., 0.005).
batch_sizz
Experiences per train step.
learning_starts
Timestep train begins.
actor_policy_frequen
Actor net. update freq. rel. to Critic (e.g., 2). Target Actor net. update freq- (e.g,, 10 steps). Target Critic net. update freq. (c.g., 10 steps). Initial for e-greedy (e.g., 1.0).
actor_target.nettor__fff
rrtticttargee.nexxxxxrrrggu start_e
end_e
Final e value (e.g,, 0.1).
total_ster _otal_steps_e
Timesteps for annealing (e.g., 15000). Boolean; varying sub-seq. start points. Boolcan; varying sub-seq. end points. Min. jump for start-moved sub-seq. gen. Min. jump for end-moved sub-seq. gen. Boolean; all single-step trans. stored. Proto-action-seg, embed dim. (e.g., 16). Actor MLP hidden layer sizes (e.g., [512, 128, 32))
sub mulb_sequenee__ate
loov_starr__ponn
btbo_sequeacas__oveeeeed_ddd
sub cab_seqquences_min__mmee ____e_ove_ed_ _nin_t
hrwry__on_stee_traneeeeee to buffer buffff
lrtoo_n_output_chanie
actor_ lattr_linear__ayen.
actor__inear__ayrrs_aaiiiii
Actor MLP activation (e.g. "caaky_rclu))
actor_ fatto_weight_decaa
Actor L2 reg. strength (e-g,, x 10 3).
_e__mp_aeee j___mbeddiing_dd _iim
Positional encoding dim. in Actor (e.g.. 16). Critic MLP hidden layer sizes (e.g., [512, 128, 32)).
"ritic_linear_layerr
onncttooon
Critic MLP activation (e.g., leakk_reluu))
critic_weight_deee.
Critic L2 reg. strength (e.g. x 3).
decoder posoder_model_path n_ aactionssinn... iin_seq action wwwsssaccaaaaeeeeeee
Path to pre-trained VAE Decoder model. Decoder action seq. length (e.g., 10).
Unique primitive actions env. (e.g., 4).
end_ of wwf.oooeeeeeeeeeeeeeeeeeeeeeeeee token
Decoder integer token for end of seq. (e.g., 4).
use_gumm hie_gumble__ in decoder
Boolean; Decoder uses Gumbel-Softmax.
actor might frequently produce embeddings that map to empty,, significantly hampering exploration and learning progress. This approach provides a clear signal to the actor network about the importance of gen- erating proto-sequences that translate to substantive action sequences, accelerating the learning process and improving the overall stability of training, Empirically, observed that this simple yet effective mechanism substantially reduced the occurrence of empty sequences.
N.4 Action Selection and Exploration
At each decision step, the Actor network generates a proto-sequence embedding. The Decoder then translates this embedding into corresponding action-sequence of primitive actions. The agent employs e-greedy exploration strategy:
With probability & (where 6, anneals from start_e e to end_e total_steps, steps): An exploratory action sequence is selected. This sequence is typically chosen randomly from pre-defined set of valid action sequences see Section 4.3 for more details.
With probability 1-c (exploitation): The action sequence generated by the pipeline used for execution.
67

---

Published in Transactions on Machine Learning Research (12/2025)
Table 22: GPS Hyperparameters Settings
Parameter
Value
total_ _timesteps
Environment specific
buffer_size
[10000, 50000]
ganma
0.99
tau
[0.01, 0.005]
batch_size
256
start_e
1
end_e
0.1
gearning_starte
1000
actor actor_learming_ratt
[1e-3,
1e-4, 1e-5]
critic_leaaring_rrate
1e-04
actor_policy_frreqeeny
2
sub_ cabb_seqeencc___ove_ssaaaat____
TRUE
sub gub_ssqquence__mvv
TRUE
sub_ besauunnces_iin_jupppoooo
start_poiine
1
sub jttp:uueence_mm__uuppppppp move__end_point
1
TRUE
occoo__arrget_netoor__eeeeee
[10, 100]
critic_target_neeook.Ferrr
10
total_steps_e
15000
actor_n_output__Cae
16
actor_linear__ayen
["512, 32", "512, 128, 32"] leaky_red relu (negative slope 0.1) 1e-04
actor_weiggtt_decay
actor
pe_ pe_embedding_dim
128
critic_lineat _lyeers
[512, 128, 32]
leaky_res melu (negative slope 0.1)
writic_weight_der
1e-04
max_ max_level
Environment specific
start_ start_level
Environment specific
TRUE
The value of e is linearly annealed:
c = max (ndd e, start e (start end e) (current step/total steps e)
The chosen action sequence is subsequently trimmed using the end_ of sequuecce_ooke before execution.
N.5 Optimizer
The Actor and Critic networks trained using separate Adam optimizers (torch. Cefrchh.pptim.Adee Adam).
The Actor's optimizer is configured with a learning rate actor lew.ning rate and applics L2 weight decay with coelficient of actor_ved bttor_weight_decayy
The Critic's optimizer uses a learning rate leitic__earning.rer and L2 weight decay with a coef- ficient of critic_ beteie_neight_decae
N.6 Replay Buffer
A replay buffer (BeplayHemory) with capacity buffer_aize stores past experiences. Each stored transition typically includes: the current state obscrvation (s:), the next state obscrvation (8:41); the selected action se- quence (act_sequ) the received reward (rx), terminal flag (d:), and the actor's proto-action-sequene embed- ding that generated act. seq, (emb:). The system may also store sub-sequences derived from executed plans
68

---

Published in Transactions on Machine Learning Research (12/2025)
if parameters such as wwwpey.oneseeep.rreeee to_bufer, puet_aub__euuencee_ aequences_ ttteeesttt_buufftteeeeeeeeeeeeee _ove___oee_____eeeee wwveeettttt_pppeeeeeeeee start_point, and push_aut sub_ wub_ssquences_ce buffer_ sorieeer_rve__eeeeeeeeeee weweeeoeeeeee_eeeeeeeeee peteeennd__odtttt_ee are enabled, potentially enriching the diversity of experiences in the buffer.
N.7 Training Details
Network training commences after learning_stann starts timesteps have been collected. Updates are performed using batches of batch_size experiences sampled from the replay buffer.
N.7.1 Critic Network Update
The Critic network parameters (0c) are updated by minimizing the Mean Squared Error (MSE) loss:
L(Oc)= [s, [ts.act_seq.n.s seq,r, d,emb)--- [[Q(s, emb, act seq; 0c) )+
The target Q-value V/ is computed using the target Actor (target_ actor actor_netooo) and target Critic ttarget_critic_netooiii networks to ensure stability:
- ddQrarged (s', Actortanget (s'), Decoder((Actorrarged (s))) 0c)
where (ctortarget (8') is the proto-sequence embedding from the target actor for state 8', Decoder(-) converts it action sequence, and 0 are the parameters of the target critic. The Q-values from the target critic can be clipped using min_qf value and lox__ffvvluue
N.7.2 Actor Network Update
The Actor network parameters (0A) are updated with frequency of actor aittr_policy_frqueeee (delayed policy update). The goal is to adjust the actor's paramcters to produce proto-sequence that leads to a higher Q- value as estimated by the current Critic. For sampled batch of states, the Actor generates proto-sequence embedding. This is decoded into an action sequence, which then evaluated by the online Critic network Q(;0c). The actor loss is designed to maximize this Q-value:
A(OA)= = EE-B [Qss, Actor(s), (ecoolr((cctorss)):ee)
N.7.3 Target Network Updates
Separate target networks are maintained for both the Actor (Actorarget with parameters 07) and the Critic (Qtarget with parameters (g). Their parameters are updated using Polyak averaging with the parameters of their corresponding online networks (0^,0c):
0 - 70 (1 T)0
The soft update rate is r. Target network updates for the Actor occur every www.tttattttetttrrkee steps, and for the Critic every zritic_target. wttpeeeetttorreeeeeeeeeeeeeeeeee steps.
N.7.4 Training Procedure Summary
Initialization: Networks and target networks are initialized. Replay buffer is empty.
Data Collection: Agent interacts with the environment using the action selection strategy (Section N.40, storing experiences (8t, (8,,8141, act. seq, Tz: de, emb;) in the replay buffer.
Learning Phase (after learning_starts steps):
1. Sample a batch_size of experiences from the replay buffer.
2. Update Critic network parameters by minimizing the MSE loss with the computed TD targets.
3. Periodically (every actor htt_lpooollopppppllee steps), update Actor network parameters maximize the Q-value of the generated sequence as estimated by the Critic.
4. Periodically (every actor spectively); update target. Actor and target Critic networks using Polyak averaging.
and tarce_ttrtttteteeeeeeeeee ftreuuue steps re-
69

---

Published in Transactions on Machine Learning Research (12/2025)
N.8 Evaluation
The performance of the GPS agent is assessed periodically during training and/or at the end of the training process.
Frequency: Evaluations on validation datasets typically oceur every val_eval_freg steps, and on subsets of the training data every train_evall eval_freq steps. Less frequent evaluations may dedicated lest dataset (e.g., every eval_test_ eval_tent_dataset_ centaset_training.FFee steps) at the end of training.
Method:
During evaluation, the actor generates a proto-sequence, the decoder converts it to an action sequence, and this sequence executed. The Decoder may operate in a deterministic mode (leterminietic_infeeeeee inference True).
Metrics: Standard reinforcement learning metries are logged, such mean episodic return and success rate. Additional metrics might include the average number of decoder generations per episode or properties of the generated action sequences.
Model Saving: If save_model is enabled, the best performing models (actor and critic) are saved based on criteria defined by save lave_model sttrateg (c.g., best success rate mean reward the vali- dation set).
O Maze Evaluation Environments Benchmark
In our research, our evaluation environments consist of procedurally generated mazes with varying structures and complexity. We utilized synthetic maze environments created using Large Language Models (LLMs) to ensure unbiased benchmark construction. The following details our approach to maze generation for the different environment lypes used in our experiments.
0.1 Synthetic Maze Generation Process
We generated our maze environments using LLM. For each maze type (rooms, obstacles, and corridors), we provided specific prompts instructing the LLM to generate Python code that would create the maze envi- ronments according to our requirements. Importantly, all maze generation was performed programmatically without manual intervention, ensuring reproducibility and eliminating human bias. We specifically used the OpenAl o3-mini model for code generation, which was instructed to create five different variants for each maze type. To avoid experimenter bias in seed selection, we also employed an LLM generate code for choosing random seeds:
import hashlib def get_connsitent__eedd))
Selects seed number from list consistently across multiple runs.
Returns:
int: The selected seed number.
uun
seed list [42, 1234, 9999, 2024, 2025]
Create hash of the function name to ensure consistency bbbjctojjcttooo = hashlib. sha256 get_ wwt.sooonnttttseeeellle hash_valle value = int becesttbbbett.... hexdigest(), 16)
Use the hash value to select a seed from the list seed_ index hash hash_valur value % len (seed_list) return seed_list index_
70

---

Published in Transactions on Machine Learning Research (12/2025)
Get the consistent seed seed = get_ www.sosttettsseeeeeeeeeeeeeeeeee
print SSeleceed seed: {seed}")
The code generated by the LLM for each maze type is available in code repository. This approach ensured that the maze generation process fully automated and free from experimenter bias, providing consistent and fair benchmark for evaluating our GPS algorithm against the basclincs.
0.2 Maze Type Generation Prompts
For each maze type, we provided detailed prompts the LLM to guide the generation process:
0.2.1 Obstacles Maze Prompt
71

---

Published in Transactions on Machine Learning Research (12/2025)
Maze Generation with 15% Obstacles
Write Python script that programmatically generates five distinct 16 x 16 mazes with randomly placed obstacles. The grid follows these rules:
Grid & Output Format:
The maze is 16 x 16 grid.
Each cell is either open space (0) wall/obstacle (1).
The outermost border (first and last rows and columns) must remain walls (1).
15% of the inner cells (excluding the border) should be randomly chosen as obstacles
(1), while the remaining are open spaces (0).
The final maze should be output as a Python dictionary:
'maze': [
"11111111111111"
Top row (wall frame) Use for open spaces;
dots are placeholders here.
"11111111111111" Bottom row (wall frame)
}
Replace the dots with appropriate Os (open spaces) and 1s (walls/obstacles Maze Generation Method: Initialize the Maze:
Create a 16 x 16 grid where every cell is an open space (0).
The outer border (first and last rows/columns) must. always remain walls (1).
The inner 14x 14 area (excluding the border) will contain open spaces (0) and obstacles
(1).
Randomly Place Obstacles:
15% of the inner 14 14 cells should be converted into obstacles (1)-
The placement of these obstacles should be random.
Ensure that at least one path remains between any two open spaces for potential connectivity.
Generate Five Distinet Mazes:
Use different random seeds to create five unique mazes.
Ensure that each maze has exactly 15% obstacles inside the inner area.
O.2.2 Rooms Maze Prompt
You maze designer responsible for enhancing predefined base structure for navigation simulation. Your task is to decide where eutrances should be located while ensuring the maze meets the following requirements:
Maze Design Requirements Base Maze Structure:
The maze is predefined and consists of 16 16 grid.
The base structure must remain intact, but you will determine the placement of the entrances and ensure connectivity between all rooms and open spaces.
Entrance and Room Connectivity:
72

---

Published in Transactions on Machine Learning Research (12/2025)
The maze is divided into four equal-sized quadrants (rooms) separated by walls.
Each room must have only two entrances of size Use seeded random choice for entrance placement.
Entrances should be placed strategically ensure the maze is fully connected, meaning agent navigate between any two open cells (0) using up, down, left, or right movements.
Passageways between rooms must be narrow and preserve the integrity of the maze's chal- lenge.
Obstacle Coverage:
Obstacles (1) must make up 5% of the total grid (== 13 cells).
Don't consider the obstacles that are part of the maze's frame.
Obstacles may be added or removed within constraints to maintain connectivity and align- ment with the entrance placement.
Don't place obstacles in nearby squares close to any entrance. Make sure that an obstacle doesn't block any entrance.
Use a seeded random choice for obstacles placement.
Reproducibility:
Use a specific random seed to ensure the design is reproducible.
Output Format:
Generate code for creating the maze.
Generate the mase a Python dictionary with a key (e.g., 'maze") and represent each row a binary string.
Output 5 mazes using different seeds and make sure that obstacles don't block entrances
Base Maze Layout: The base structure is as follows:
base "base_maze's
1111111111111111
'10000001000000000
" 110000001000000001"""
"10000001000000000"
"10000001000000011
"10000001000000001
1100000010000000000
"111111111111111"
" "100000010000000011
1100000010000000000
100000000000000000
110000000000000000
"100000010000000
"1000000100
"10000001000000
73

---

Published in Transactions on Machine Learning Research (12/2025)
Design Task: Modily the maze by:
Placing entrances the walls separating the quadrants.
The logic should be based on seed for determining the entrances position and obstacles posi- lions.
Ensuring the maze is fully connected.
Making any minor adjustments to obstacles (1) to meet connectivity and percentage require- ments.
Verify by code that each room must have only two entrances of size 1.
Verify by code that each obstacle doesn't block any entrance.
Output the modified maze as Python dictionary with the format below:
m [
uppatedd_oo row row_1", uudaatedroo__2
"updated_rowe row 16"
General Steps:
Choose two entrances for each room by selecting a random square on each wall.
Randomly select room and place obstacles within it until the obstacle budget is reached.
O.2.3 Corridors Maze Prompt
Write Python script that programmatically generates five distinct 16 16 mazes. In each maze, start with a grid completely filled with wall cells (represented by 1), then carve out corridors by selecting one or more vertical lines and (represented by 0). The corridors will be 1-cell-wide, and they must intersect that every open cell is reachable from any other via up, down, left, and right moves. The outer border of the maze should always remain as walls.
more horizontal lines convert wall cells to open cells
Grid & Output Format:
The maze is 16 x 16 grid.
Each cell is either open (0) or a wall (1).
The outermost border (first and last rows and columns) must remain walls.
The final maze should be output as a Python dictionary:
'maze': [
"1111111111111111
Top row (wall frame) Use open spaces; dots are placeholders here.
Bottom row (wall frame)
74

---

Published in Transactions on Machine Learning Research (12/2025)
]
}
Replace the dots with the appropriate Os and is as per the carved corridors.
Corridor Carving Method: Initialize the Maze:
Create a 16 16 grid where every cell is a wall (1), with the outer border fixed as walls.
Select Corridor Lines:
Vertical Corridors: Choose up to vertical columns (not including the outer borders) that will serve as corridors.
Restriction: Ensure that no consecutive vertical columns are selected there must be at least one wall column between any two chosen corridor columns.
Horizontal Corridors: Choose 2 up to 4 horizontal rows (again, not including the outer borders) that will serve corridors.
Restriction: Ensure that no consecutive horizontal rows are selected there must be at least one wall row between any two chosen corridor rows.
These lines will form a network of corridors that cross each other.
Carve the Corridors:
For each selected vertical column, change all cells in that column (except the outer border) from (wall) to (open space).
Similarly, for each selected horizontal row, change all cells in that row (except the outer border) from 1 to 0.
The intersections of these corridors (where selected vertical column crosses selected hori- zontal row) will naturally be open, ensuring connectivity.
Ensure Full Connectivity:
The chosen vertical and horizontal corridors should intersect, guaranteeing that every open cell (in the corridors) is reachable from any other open cell.
Optionally, you add additional corridor "branches" (by clearing cells adjacent to the main corridors) to create more interesting maze layout, as long all open cells remain interconnected.
Randomness:
Generate five distinct mazes by using different random seeds and varying the selected vertical and horizontal corridor positions.
75


Reviews:
[{"review_id": "7s3Rz3L4qK", "reviewer": "Reviewer_kSJ6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a new method to generate action sequences for reinforcement learning rather than predicting individual actions.\n\nStrengths:\n- clear writing\n- the model is compared to strong baselines\n\nWeaknesses:\n- The environment used is increadibly simple. Specifically, it does not feature any stochasticity. This means that the agent can always predict the maximal number of steps and does not need to trade off prediction horizon against accuracy. I recommend trying this work on more standard control tasks from e.g. gymnasium.\n- The split of pretraining the decoder and training the policy is poorly motivated: pretraining the decoder only works if one has very similar structure and action spaces since otherwise it cannot even be guaranteed that a specific action sequence is generatable. In fact, one cannot guarantee that the space reachable by the decoder is spanning the complete set of all A^K action-sequences.\n- The reward function used looks very \"reward shaped\", and - more importantly - is not markov since the reward at state $s_k$ depends on all previous steps $s_0,\\dots,s_k$.\n- The paper, to our knowledge, does not list the reward and only displays surrogate metrics. Ultimately the reward would be the interesting part since this is the only thing directly selected for by the RL agent, while everything else depends on tenous connections between reward and surrogate and the implicit biases encoded in the metrics. This means even if this method works well in this setting it just might be due to implicit biases favouring the GPS module which would not generalize to other domains\n- Multi-step prediction is poorly motivated: Of course receding horizon methods used in MPC have advantages due to being able to anticipate the future, but if one has to commit to a sequence _without_ being able to update it later this actively reduces the amount of information available to the agent. In some sense, this is the opposite of MPC since the agent has to be able to perfectly anticipate the future states K steps into the future without being able to re-ground itself. This only works in simple, completely determinstic settings: If the environment is not deterministic, one might get into an unanticipated state and continue from there without knowing it. If the environment is too complex, the model will be unable to capture it (it will become stochastic from the model's POV). Any sort of partial observability is also expected to break this. In general, since we do not have a receding horizon the problem you end up modelling is over an augmented MDP with size state S and action space $A^K$ and transition function $T^K=T\\circ T\\circ\\dots\\circ T$ (with $K$ being the stepsize). It is unclear why this problem would every be easier than the original MDP. I recommend improving the motivation.", "claims_and_evidence_yn": "No", "claims_explanation": "Only deterministic environments are being tested. No reward statistics are disclosed. Lack of usage of standardized benchmarks.", "audience_yn": "Yes", "audience_explanation": "The composition of actions as part of the network output is theoretically interesting.", "requested_changes": "- benchmark other environments (particularly stochastic ones)\n- adress the motivation problem discussed in the weaknesses\n- discuss why the decoder has to be pretrained\n- disclose the reward (behaviour over time and at the end of training)\n- use a markov reward formulation", "broader_impact_concerns": "We do not have broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lerdIrd0uQ", "reviewer": "Reviewer_Vp97", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces **Generative Proto-Sequence (GPS)**, a novel actor-critic reinforcement learning architecture designed to tackle long-horizon, sparse-reward tasks. The core contribution is a method that generates **variable-length, discrete action sequences in a single, atomic decision step**. Instead of a traditional policy that outputs one action per timestep, the GPS Actor generates a latent \"proto-sequence\" embedding. This embedding is then translated into an executable action sequence by a pre-trained VAE-based Proto-Sequence Decoder (PSD). The Critic evaluates the value of the entire state-sequence pair, allowing for end-to-end training.\n\n---\n\n**Key Strengths:**\n* **Novel Architecture:** The proposed method of generating a complete, variable-length action sequence via a latent embedding in a single forward pass is a novel and elegant approach to temporal abstraction. The end-to-end differentiability, achieved by passing gradients from the Critic through the Decoder to the Actor, is a significant technical strength.\n* **Strong Empirical Performance:** The paper provides extensive evidence across a variety of procedurally generated maze environments, demonstrating that GPS consistently and significantly outperforms strong baselines (DQN, DAR, TempoRL) in terms of sample efficiency (faster convergence) and final performance (higher success rates), especially in larger and more complex mazes.\n* **Effective Exploration and Credit Assignment:** By making decisions at the sequence level, GPS effectively reduces the decision horizon, which appears to directly address the challenges of temporal credit assignment and strategic exploration in sparse-reward settings. The analysis of subsequence buffering provides direct support for this claim.\n* **Generalization:** The paper shows that GPS can generate novel action sequences not seen during the decoder's training, suggesting it learns a structured representation of behaviors rather than simply memorizing effective trajectories. This is a strong argument for the model's ability to generalize.\n\n---\n\n**Key Weaknesses:**\n* **Limited Evaluation Domain:** While the maze environments are well-controlled and systematically test navigation, they represent a single domain. The claims of general applicability to long-horizon tasks would be stronger if tested on other benchmark domains.\n* **Decoder Pre-training Overhead:** The method requires a separate pre-training phase for the Proto-Sequence Decoder (PSD) on a dataset of synthetic action sequences. A new PSD must be trained for every unique action space, which adds complexity and may limit transferability.\n* **Path Efficiency Trade-off:** The main (stochastic) GPS model often achieves a higher success rate at the cost of path efficiency (PER) compared to its deterministic variant (GPS-D) and some baselines. While the authors frame this as a reasonable trade-off for robustness, it is a limitation in scenarios where optimality is critical.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1.  **Claim of Superior Performance:** The primary claim that GPS \"consistently outperforms the baselines in most cases\" is convincingly demonstrated in **Tables 2 and 3**. In nearly all environments, GPS and its deterministic variant, GPS-D, achieve higher Average Success Rates (ASR) and converge in significantly fewer training steps than DQN, TempoRL, and DAR. The authors validate these results with **McNemar's tests (Appendix J)**, showing that the improvements over all baselines are statistically significant ($p < 0.0001$), which adds a strong layer of confidence to the results.\n\n2.  **Claim of Improved Credit Assignment & Exploration:** This is supported by multiple pieces of evidence.\n    * **Indirectly:** The dramatic speed-up in learning (Table 3, \"ASR Converge>0.9 Step\") is strong evidence that the learning signal is being propagated more effectively.\n    * **Directly:** The ablation study on subsequence buffering (**Figure 3**) clearly shows that training the Critic on prefixes and suffixes of action sequences significantly accelerates learning. This technique directly leverages the structure of the sequence-based decisions to provide denser, more effective learning signals, supporting the credit assignment claim.\n    * **Exploration:** The lower Sequence Generation Frequency (SGF) in **Table 3** indicates that GPS commits to longer, more coherent action sequences, which prevents dithering and promotes more structured exploration compared to step-by-step methods.\n\n3.  **Claim of Better Generalization:** The ability of the trained agent to achieve high success rates on a held-out test set of unseen mazes is the primary evidence for generalization. Furthermore, the analysis in **Section 5**, which shows the model generating plausible and novel action sequences that were not in the decoder's training set, is a powerful demonstration that GPS learns a flexible, compositional representation in its latent space rather than merely memorizing training trajectories.", "audience_yn": "Yes", "audience_explanation": "1.  **Addresses a Fundamental RL Problem:** The paper tackles the long-standing challenges of exploration and credit assignment in long-horizon, sparse-reward environments. This is a core problem in reinforcement learning, and novel methods that show significant progress are of high interest to the research community.\n2.  **Novel and Elegant Method:** The concept of generating entire action sequences as single decisions through a latent space is a conceptually clean and powerful idea. It moves beyond simple action repetition or pre-defined skills, offering a more flexible form of temporal abstraction.\n3.  **Strong and Clear Results:** The paper presents unambiguous and statistically significant improvements over relevant and well-established baselines on a challenging benchmark. The clarity and strength of the results make the paper's contribution easy to understand and appreciate.\n4.  **Sparks Future Research:** The proto-sequence idea opens up several interesting avenues for future work, such as applying it to continuous control, using more powerful generative models for the decoder, or exploring ways to learn the decoder's action vocabulary online. This potential to inspire follow-up research makes it a valuable contribution to the field.", "requested_changes": "The paper is well-written and the research is solid. The following changes would strengthen it further.\n\n**Critical to Acceptance:**\n* None. The paper in its current form meets the bar for acceptance. The claims are well-supported by the evidence provided.\n\n**Strengthening the Work (Recommended):**\n* **Discuss Computational Cost:** Please add a brief discussion of the computational trade-offs. While GPS is more sample-efficient, what is the wall-clock time cost? This includes the one-time cost of pre-training the VAE decoder and the per-step inference time of the Actor-Decoder pipeline compared to the baselines. This would provide a more complete picture of the method's practicality.\n* **Expand on the Path Efficiency Trade-off:** The results clearly show a trade-off between the stochastic GPS (better ASR, worse PER) and the deterministic GPS-D (often slightly lower ASR, much better PER). This is an interesting finding. I recommend moving a sentence or two from the results section into the main discussion or conclusion to explicitly highlight this \"exploration-exploitation\" trade-off at the sequence level and provide guidance on when one variant might be preferred over the other.\n* **Clarify the Exploration Mechanism:** In Section 4.3, it is stated that exploration uses an $\\epsilon$-greedy strategy with \"random sequences being sampled from a predefined pool\". For clarity, please explicitly state that this pool is the same set of 400 synthetic sequences used to train the PSD, as described in Appendix E.1. This closes the loop for the reader on where these exploratory sequences originate.\n* **Position Encoding Ablation:** The use of positional encoding is well-motivated in Appendix F.1.3. However, the authors state that \"manual investigations and targeted experiments suggest that incorporating position encodings enhances the model's capabilities\". This is a strong claim that would be more convincing with a quantitative ablation study. Adding a small table or plot comparing the ASR of GPS with and without positional encoding on a challenging environment (e.g., 24x24 with obstacles) would provide direct evidence and significantly strengthen this architectural design choice.", "broader_impact_concerns": "I have no broader impact concerns with this work. The research is foundational and focuses on algorithmic improvements for decision-making in simulated environments (mazes). It does not engage with sensitive data, and the potential applications are not directly tied to societal or ethical issues that would require a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ByXWB3BLzP", "reviewer": "Reviewer_yP6u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method that generates a sequence of actions by VAE (encoder is fixed) along with an actor-critic architecture. Authors compared their methods with Deep Q Network (DQN), Dynamic Action Repetition (DAR), and TempoRL (Temporal Reinforcement Learning), and showed that the proposed methods achieve better performance and learns faster than the baselines in the grid environment.", "claims_and_evidence_yn": "No", "claims_explanation": "* A detailed explanation of the implementation is provided in the appendix.\n\n* The authors briefly mention the rich literature on hierarchical reinforcement learning but do not sufficiently compare their method to existing approaches in either the related works or the experiments section.\n\n* On page 4, the authors claim that generating unseen action sequences is an important aspect of their work. However, any non-rule-based model is expected to generate novel action sequences when presented with novel observations. Therefore, it is unclear why this should be considered a significant contribution.\n\n* The proposed methods, GPS and GPS-D, do not demonstrate improved path efficiency, although they achieve a better success rate. Furthermore, in Section 4.4.2, the authors claim that GPS can self-correct via sequential decision points but provide no supporting evidence for this claim.\n\n* According to the supplementary materials, the authors used only a single, different seed for each experimental method. Since reinforcement learning results are known for their high variance, reporting the mean and standard deviation over multiple seeds is necessary for a robust evaluation.\n\n* The experiments were only conducted in simple, 2D grid-world environments. I recommend that the authors evaluate their method in more complex environments. It is also worth noting that the environment used appears to be a fully observable Markov Decision Process (MDP), not a Partially Observable MDP (POMDP).\n\n\nBacon, Pierre-Luc, Jean Harb, and Doina Precup. \"The option-critic architecture.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 31. No. 1. 2017.\n\n\nBotvinick, Matthew M., Yael Niv, and Andew G. Barto. \"Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective.\" cognition 113.3 (2009): 262-280.\n\n\nMcClinton, Willie, Andrew Levy, and George Konidaris. \"Hac explore: Accelerating exploration with hierarchical reinforcement learning.\" arXiv preprint arXiv:2108.05872 (2021)\n.\nOsa, Takayuki, Voot Tangkaratt, and Masashi Sugiyama. \"Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization.\" International Conference on Learning Representations.", "audience_yn": "Yes", "audience_explanation": "The topic of long-horizon reinforcement learning has gained increasing attention along with embodied intelligence. Although the authors did not conduct any experiments on robot simulation rather only focused on 2D bird-eye view grid world maze environments, I think it can still gain attention from TMLR’s audience.", "requested_changes": "I recommend moving Appendices A-G to the end of the appendix since these are implementation details, while starting from Appendix H, which supplements the experimental results mentioned in the main paper.\n\n\nPlease differentiate \\citet and \\citep. Also, please change the authors of \\citep to just \\citet.\n\n\nIn Section 3.1, embedding space $k \\in K$: $K$ is not defined, and $k$ is the embedding, not the embedding space mentioned in the previous sentence.\n\n\nIn Figure 2 caption, please add the meaning of each symbol in the figure.", "broader_impact_concerns": "No broader impact concern that requires a Broader Impact Statement", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)