Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Authors: Yuhui Wang
Qingyuan Wu
Dylan R. Ashley
Francesco Faccio
Weida Li
Chao Huang
Jürgen Schmidhuber
Venue: ICML 2025 poster (2025)
Keywords: Value Iteration Networks
Long-term Planning
Reinforcement Learning
Deep Neural Network
Primary area: reinforcement_learning
TL;DR: A novel 5000-layer Dynamic Transition Value Iteration Network performs well in extremely long-term large-scale planning tasks

Abstract: The Value Iteration Network (VIN) is an end-to-end differentiable neural network architecture for planning. It exhibits strong generalization to unseen domains by incorporating a differentiable planning module that operates on a latent Markov Decision Process (MDP). However, VINs struggle to scale to long-term and large-scale planning tasks, such as navigating a $100\times 100$ maze---a task that typically requires thousands of planning steps to solve. We observe that this deficiency is due to two issues: the representation capacity of the latent MDP and the planning module's depth. We address these by augmenting the latent MDP with a dynamic transition kernel, dramatically improving its representational capacity, and, to mitigate the vanishing gradient problem, introduce an "adaptive highway loss" that constructs skip connections to improve gradient flow. We evaluate our method on 2D/3D maze navigation environments, continuous control, and the real-world Lunar rover navigation task. We find that our new method, named Dynamic Transition VIN (DT-VIN), scales to 5000 layers and solves challenging versions of the above tasks. Altogether, we believe that DT-VIN represents a concrete step forward in performing long-term large-scale planning in complex environments.

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Yuhui Wang" 1 Qingyuan Wu 22 Dylan R. Ashley 13 Francesco Faccio 13 Weida Li*
Chao Huang² Jürgen Schmidhuber
Abstract
some older algorithms (e.g., Dyna (Sutton, 1991). A (Hart et al., 1968), and others (Schmidhuber, 1990a;b)) and many recent ones (c.g., the Predictron (Silver et al., 2017), the Dreamer family of algorithms (Hafner et al., 2020; 2021; 2023). SoRB (Eysenbach et al., 2019). SA-CADRL (Chen et al., 2017), and the LLM-planner (Song al., 2023)), ef- fective planning is long-standing and important challenge in artificial intelligence (AI).
The Value Iteration Network (VIN) is an end-to- end differentiable neural network architecture for planning. It exhibits strong generalization to un- seen domains by incorporating a differentiable planning module that operates on latent Markov Decision Process (MDP). However, VINs strug- gle to scale to long-term and large-scale planning tasks, such as navigating 100 x 100 maze-a task that typically requires thousands of planning steps to solve. We observe that this deficiency is due to two issues: the representation capac- ity of the latent MDP and the planning module's s depth. We address these by augmenting the latent MDP with dynamic transition kernel, dramati- cally improving its representational capacity, and, to mitigate the vanishing gradient problem, intro- duce an "adaptive highway loss" that constructs skip connections to improve gradient flow. We evaluate our method on 2D/3D maze navigation environments, continuous control, and the real- world Lunar rover navigation task. We find that our method, named Dynamic Transition VIN (DT-VIN), scales to 5000 layers and solves chal- lenging versions of the above tasks. Altogether, we believe that DT-VIN represents a concrete step forward in performing long-term large-scale plan- ning in complex environments.
Traditional search-based planning algorithms like A# re- quire an accurate environmental model. Thus, these algo- rithms are less effective in scenarios with unknown envi- ronmental models or when the state and action spaces are large or continuous. In such scenarios, policy can be learned either through imitation learning (IL), which lever- ages expert demonstrations, or through trial and error with reinforcement learning (RL). Within RL and II, the Value Iteration Network (VIN) (Tamar et al., 2016) stands out as quite unique due to its distinctive architecture that integrates differentiable latent "planning module" into the deep neu- ral network, rather than maintaining an explicit learned environment model like Dreamer (Hafner et al., 2020) or MuZero (Schrittwieser et al., 2020). This integrated plan- ning structure of VINs endows them with powerful gener- alization capabilities for unseen planning tasks. VINs have been shown to perform well in some small-scale short-term planning situations, like path planning (Pflueger et al., 2019; Jin et al., 2021), autonomous navigation (Wöhlke al., 2021), and complex decision- making in dynamic environ- ments (Li et al., 2021). However, they still struggle to solve larger-scale and longer-term planning problems. We refer to large-scale planning tasks as those with high-dimensional observation space (c.g.. .. the maze size), and long-term plan- ning tasks those necessitating extended planning horizons to achieve the goal. For example, in a 100 100 maze nav- igation task, the success rate of VINs in reaching the goal drops to well below 40% (see Figure 1(b)). Even in smaller 35 x 35 mazes, the success rate of VINs drops to 0% when the required planning steps exceed 60 (see Figure 1(c)).
1. Introduction
Planning is the problem of finding a sequence of actions that achieves a specific pre-defined goal. As the aim of both
'Equal contribution 'Center of Excellence for Generative AI, King Abdullah University of Science and Technology "The Uni- versity of Southampton 'The Swiss AI Lab IDSIA/USI/SUSII, Switzerland "National University Singapore. Correspondence to: Yuhui Wang <yuhui. wang @ kau.s..dd..s>>.
Our work identifies that the principal deficiency causing this is the mismatch between the complexity of planning and the comparatively weak representational capacity of
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Tern Planning
Maze with Various Sizes
35 x 35 Maze
Optimal Path
100
100
Goal
Success Rate (9)
80
Success Rate (%)
80-
DT-VIN (N=600)
Start
DT-VIN (N= 300)
60
60
DT-VIN (N=100)
VIN (N 600)
40
VIN (N=300)
VIN (N=100)
20
DT-VIN (ours)
20
VIN
0
15
25
35
100
100
200
300
Maze Size
Shortest Path Length
100
Perfommance by Domain Scales
(c) Perfomancc by Planning Steps
Figure I: (a) shows an example of 100 x 100 maze navigation task, where the green line shows the optimal path from the start position (blue) to the goal position (red). See Appendix Figure for more examples of mazes with other sizes. (b) shows the success rate of VIN and DT-VIN on the maze navigation tasks computed in expectation over different shortest path lengths for each maze size. (c) shows the success rate of VIN (Tamar et al., 2016) and our DT-VII VT-VIN as a function of planning steps the 35 x 35 maze benchmark.
function of maze size. The reported results are
the relatively shallow networks that it uses. And while there has been moderate success in learning more compli- cated networks (c.g., GPPN (Lee et al., 2018) and Highway VINs (Wang et al., 2024a)), until now, VINs of a scale ca- pable of long-term or large-scale planning have not been computationally tractable due to persistent issues with van- ishing and exploding gradients-a fundamental problem of deep learning (Hochreiter, 1991).
creasingly complex planning tasks alongside the increasing availability of computing power.
2. Preliminaries
Reinforcement Learning (RL) and Imitation Learning (IL). The most common formalism used for RL is that of the Markov Decision Process (MDP) (Bellman, 1957). We consider an MDP- as per Puterman (2014)-to to be the 6- tuple A, R, 7s p)), where S is a countable state space, A is a finite action space, T(s'|s, a) represents the tran- sition probability to state s' E S from state E S and taking action E A, R(s, a, s') is the reward function, 7 E 1) is discount factor, and f is distribution over initial states. The behaviour of an artificial agent in an MDP is defined by its policy T(als), which speci- fies the probability of taking action a in state s. The state value function (s) is the expected discounted sum of rewards from state and following policy æ, i.e., V (s) A E M>> R's ae, sc+1)|so = s; <]. The goal of RL is usually to find an optimal policy x² that achieves the high- est expected discounted sum of rewards. The value function of an optimal policy is denoted by (s) = max, V*(,,, and satisfies V* (8) = Vs(V)s.. The Value Iteration (VI) algorithm iteratively applies the following update to all states to obtain the optimal value function: v(n+1(ss) = max Σ. T (s'1ss, a) | (s, s') ( >V(mm)) (s'], where is the iteration number. In scenarios where designing a com- prehensive reward function is difficult, IL offers a practical alternative. IL enables agents to learn from human or al- gorithmic demonstrations, with approaches like Behavioral Cloning directly mimicking expert actions in similar states (Bain & Sammut, 1995; Schaal, 1996; Ross et al., 2011).
In this work, we aim to surgically correct deficiencies in VIN-based architectures to enable large-scale long-term planning. Specifically, we first identify the limitations of the latent MDP in the planning module of VIN and pro- pose dynamic transition kernel to dramatically increase the representational capacity of the network. We then build on existing work that identifies the connection between net- work depth and long-term planning (Wang et al., 2024a) and propose an "adaptive highway loss" that selectively constructs skip connections to the final loss according to the actual number of planning steps. This approach helps mitigate the vanishing gradient problem and enables the training of very deep networks. With these changes, we find that our new Dynamic Transition Value Iteration Network (DT-VIN), is able to be trained with 5000 layers and scale to 1800 planning steps in a 100x x 100 maze navigation task (compared to the original VIN, which only scaled to 120 planning steps in 25 x 25 maze). We apply our method to various challenging tasks, including 2D/3D maze navigation tasks (Wydmuch et al., 2019), continuous control (Founda- tion, 2022; Fu et al., 2020), and real-world Lunar rover navigation tasks (Berlin, 2018). We find that DT-VINS can solve both despite these problems requiring hundreds to thousands of planning steps. Together, these demonstrate the practical utility of our method on based tasks that previous methods are simply unable to solve. This also serves to highlight the potential of our method to scale to in-
Value Iteration Networks (VINs). VIN is an end-to-end differentiable architecture that conducts planning on a latent
2

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Transition Mapping
Loss
Policy Mapping
Reward Mapping
kri
A
(b1) VI module with Invariant Transition
Value Iteration Module
B
G
scrvodioe sponing
Value Iteration Modula
Start
Goall
(a) Mapping
(b2) module Dynamic Transilion
Planning Module, Policy Mapping Loss
Figure 2: The architecture of VIN and DT-VIN in the maze navigation task. (a) shows the observation of the maze, which is mapped to the latent reward/transition matrix of the latent MDP through the reward/transition mapping module. (c) shows the "planning module", the policy mapping module and the loss. The "planning module" contains numerous stacked Value Iteration (VT) modules. The green and orange connections show an example of adaptive highway loss for planning tasks starting from and B, respectively. (bl) shows the VI module of the original VIN with invariant transition - E RIX (b2) shows the VI module of DT-VIN with dynamic transition kemel Iyyn E Mx|xxx(AAxxxxF
MDP M (Tamar et al., 2016). Below, we use to denote all the terms associated with the latent MDP M. For each decision, VIN first maps an observation x, e.g., an image of maze and the agent's position, to M. M described by the latent state space S (JEEMM) where M denotes its size; fixed discrete latent action space A; a latent reward matrix R = f*(()) E RMM xM, where f is learnable NN called a reward mapping module; and latent transition kernel ilav x FxF with F representing the dimension of the kernel. The latent transition kernel is learnable parameter matrix that is invariant of both the latent state and the observation x. Next, VIN conducts VI on the latent MDP M to approximate the latent optimal value func- tion V To ensure the differentiabilityy a differentiable module is proposed, simulating VI computation using CNN operations, i.e., convolutional and max-pooling operations:
a given observation x. Finally, the model can be trained by standard RL and IL algorithms with the following gen- eral loss: (0) b EeeweD (fr ( ..))
where D {(z, v)} is the training data, x is the observa- tion, is the label, and e is the sample-wise loss function. The specific meaning of these items varies depending on the task. For example, in imitation learning, where the ex- pert data provided, the label is the expert action and
is the cross-entropy loss, i.c., e {(1T (1 (T .y)
E.EA {a=y] log f (v))(()))) where 1 is the indi- cator function.
3. Method
In this section, we discuss how to train scalable VINs for long-term large-scale planning tasks. Our method addresses the two key issues with VIN that identified as hampering its scalability: the capacity of the latent MDP representation and the depth of the planning module.
max Σ Thyy
R 1ー (1)
J.
This equation sums over a matrix patch centered around position (i.j). After the above, by stacking the VI mod- ule for N layers, the latent value function is then fed to policy mapping module by fo represent a pol- icy that is applicable the actual MDP M. Here, yar (x)))
3.1. Improving Latent MDP's Representation Capacity
VIN utilizes CNNs simulate the VI computation process, where the latent transition kernel is implemented as a learn-
(x)), represents the probability of taking action
3

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
able parameter T E R as described in Section 2.
Table I: Types of latent transition kernels, categorized by whether they are latent state-dynant (i.e., vary with the latent state) and observation-yynamic (i.e., vary with the
However, there is a discrepancy between the computation process of CNNs and the general VI.
First, the latent transition kernel of VIN is invariant for each latent state = (i, This severely limits the latent MDP's capacity to represent the complex, statedeependen transitions of the real MDP-an ability that is essential for its effectiveness. For example, in the maze navigation problem shown in Figure 2 (a), the transition probabilities are quite different if the adjacent cell is wall versus an empty cell.
observation).
Types
Latent state-
Observation- dy syamiie?
Implementation
dynamic?
learnablc parameter T BlAsses learnable parameter T e M (Zpers
fully invariant
X
Iatent state- dynamic only
learnable model (T) which cusput T eE B Wlleee
observation- dynamic only
Therefore, propose use latent state-dynamic transi-
learnable model f (()) which oclput T & R Aparter
Fully dynamic
tion kernel T E RM x FxX and the augmented
VI module is computed as follows:
for each latent state, where F' is the convolutional kernel size of the transition mapping module, and F is the size of the latent transition kernel. This architecture only requires |AA F F² additional parameters. Note that in practice, set- ting both F" and to the small value 3 suffices to achieve strong performance. Thus, this alternative module greatly improves the representation capacity of VIN, but typically does not introduce significant change in training cost.
Tavn
i (Ri-aa- ( ( (2)
max)
V
Different from VIN's VI module in Eq. (1), which uses
invariant kemne Tlay E R [xF that is same across all latent states (i, here Eq. (2) utilizes a dynamic kernel dyn 1. that adjusts dynamically to each latent state x = (i,j)-
Lastly, we further enforce a softmax operation on the values of the latent transition kernel for each latent state 3 to avoid the gradient exploding problem. This not only provides a statistical semantic meaning to the kernels but also plays crucial role in ensuring the training stability, as will be demonstrated in Section 4.4.
Second, although the original VIN paper proposes a general framework where the latent transition kernel depends the observation, i.e., T' = fT (z), with f as a learnable transition mapping module, the latent transition kernel in all VIN's experiments is implemented as learnable parameter independent of the observation. Intuitively, ignoring obser- vations prevents VIN from exploiting any information in the observation to model varying transition dynamics across different environments. While this limitation does not im- pact VIN's performance in small-scale, short-term planning tasks (as were tested on in the original work) where the state space is limited and only a few steps are needed to reach the goal, we found it be a major barrier to VIN's effectiveness when employed to plan on large-scale, long-term planning tasks with completely different observations. Therefore, transition kernels,
Altogether, these above components form our proposed method, Dynamic Transition VINs (DT-VINs), which em- ploys latent transition kernels that dynamically adapt based on both the latent state and observation, along with a soft- max normalization operation.
3.2. Increasing Depth of Planning Module
Recent work on Highway VIN has demonstrated the rela- tionship between the depth of VIN's planning module and its planning ability (Wang et al., 2024a). A deeper planning module implies more iterations of the value iterations pro- cess, which is proven to yield more accurate estimation of the optimal value function (see Theorem 1.12) (Agarwal et al., 2019). However, training very deep neural networks is challenging due to the vanishing or exploding gradient problem (Hochreiter, 1991). Highway VINs address this issue by incorporating skip connections within the context of reinforcement learning, showing similarities to existing works for classification tasks (Srivastava et al., 2015b; He et al., 2016). Although Highway VINs can be trained with up to 300 layers, they still fail to achieve perfect scores in larger-scale and longer-term planning tasks and necessitate a more intricate implementation. Here, we present a simpler, easy-to-implement method for training very deep VINs.
we propose using observation-dynane which employ a learnable transition mapping module ST to dynamically generate the latent transition kernel for each observation.
The notions of latent state-dynamic and observation- dynamic features are orthogonal, each capturing an inde- pendent aspect of variation in the latent transition kernel. Their combination yields four distinct configurations, along with corresponding implementations, as summarized in Ta- ble 1. Our experiment in Section 4.4 shows that the fully dynamic which varies with both the latent state and the inteevaatio-achivvee the best performance. To imple- ment this kernel, we design a CaNebased architecture for the transition mapping module fT, which maps each local F' F' maze patch to a [A] F x F latent transition kernel
4

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
To facilitate the training of very deep VINs, we also adopt the skip connections structure but implement it differently. Our central insight is that short-term planning tasks gen- erally require fewer iterations of value iteration compared to long-term planning tasks. This is because the informa- tion from the goal position propagates to the start position in fewer steps when their distance is short. Therefore, we propose adding additional loss to shallower layers directly when the task requires only few steps during the train- ing process. We achieve this by introducing the following adaplive highway loss:
encompassing both 2D and 3D environments (Wydmuch et al., 2019), detailed in Section 4.1. To demonstrate the potential for complex action spaces, we further evalu- ate the generality of DT-VIN on continuous control tasks (Section 4.2), which present more challenges than the con- tinuous control tasks described in the original VIN paper due to increased complexity of the maze and the physics (Fu et al., 2020). Additionally, we assess DT VIN in rover nav- igation tasks (Section 4.3), where the agent must perform planning based on orthomosaic images (Berlin, 2018).
Each task includes a start position and a goal position. We say that an agent has succeeded in a task if it can navigate from the start position to the goal position within a prede- termined number of steps (M2 in our paper). To assess planning capabilities across tasks of varying complexities, our experiments evaluate various tasks distinguished by their shortest path lengths (SPLs). The SPL represents the length of the expert path, derived from either human or algorithmic experts. We further define path as relatively optimal if it has the shortest length among all solutions provided by vari- ous models, including the expert solution. We follow GPPN and use these for the success rate (SR), which is the rate at which the algorithm succeeds in the task, and the optimality rate (OR), which is the rate at which the algorithm provides relatively optimal path.
1
(0)
K
(3)
Σ Σ (R.Y.)) 1S
1,22 In mod e e fr (v(m)())))) .u)
Here, K- Σ E (m21) 1 (n mod J=0}, 1 is the IENSSE
indicator function, and I is the length of the planning path or trajectory, which can be computed from the training data. For example, in the imitation learning of the maze naviga- tion task, the length / of the provided expert path from start to goal is inherently known for each maze in the dataset. Here, Eq. (3) presents a general form of L with an arbitrary single-term loss function l (e.g., cross-entropy loss); see Appendix B.1 for its specific form in different cases.
On the above tasks, we compare our DT-VIN method with several advanced neural networks designed for planning tasks, including the original VIN (Tamar et al., 2016), GPPNs (Lee et al., 2018), and Highway VIN (Wang et al., 2024a). The models are trained through imitation learn- ing using labeled dataset. We then identify the best- performing model based on its results on a validation dataset and evaluate it on a separate test dataset. Following the methodology from the GPPN paper, we conduct evaluations using three different random seeds for each algorithm. This is sufficient to provide a reliable performance estimate here due to the low standard deviation observe in the tasks. All figures that show learning curves report the mean and standard deviation on the test set.
As Eq. (3) implies, it constructs skip connections for the hid- den layers to improve information flow, similar to existing works such as Highway Nets and subsequently, Residual Nets (Srivastava et al., 2015b; He et al., 2016). However, we connect hidden layers directly to the final loss, while existing works typically connect skip connections between the intermediate layers. Besides, we construct skip connec- tions for each layer n 2 | rather than at the specific layer = 1, as the value iteration process should converge to the correct solutions even with additional iterations. Note that this additional loss will not alter the inherent structure of the value iteration process and will be removed during the execution phase. Moreover, to decrease computational com- plexity, we only apply adaptive highway loss to the layers that satisfy the condition mod J = O, where J > 1 is hyperparameter.
4.1. Maze Navigation
Setting. We first evaluate 2D maze navigation tasks with sizes M set to 15, 35, and 100, where the agent moves one step at a time to any of the four adjacent cells. Each task is defined by a starting position, a visual representation of the M x M map design matrix (with 0 and representing obstacles and roads, respectively). and an M x M goal matrix indicating the position of the goal. These mazes require hundreds to thousands of planning steps, as shown in Figure 1(a) and further detailed in Appendix Figure 8. For each maze size, we generate a dataset following the methodology in GPPN (Lee et al., 2018) and ensure no
4. Experiments
We perform several experiments to test if our modifications to VIN's planning module allow training very deep DT- VINs for large-scale long-term planning tasks. Following previous work (Tamar et al., 2016; Lee et al., 2018), we focus on the imitation learning scenario, where we leverage expert demonstrations to evaluate planning capabilities.
In line with previous works (c.g.. (Lee et al., 2018)), we assess our planning algorithms on maze navigation tasks,
5

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
overlap between training and test datasets. Specifically, each unique obstacle arrangement-with its varying start positions and goals-is exclusive to either the training or the test dataset, preventing information leakage. To assess the performance of each algorithm, we test various neural network depths N. For the largest mazes (M = 100), we specifically examine N = 600 and N 5000, while for M 35, we test multiple depths: N = 30, 100, 300, and 600. For more details, see Appendix C.I.
Table 2: The success rates of the methods on Point Maze and Ant Maze tasks.
Point Maze
Ant Maze
100 100
35
100 x 100
VIN
67.42
18.191s.7
31.7 61.851s:
GPPN Highway DT-VIN (ours)
91.21+1.94
09:37
86. 36.00
61.7840.12
89.6322.71
66.45:448
82.3444119
42.91
99. +9.87
98.04-1.08
96.57=22.1
98. 98.2628.14
Table 3: The success rates of the algorithms rover navi- gation tasks with various image sizes.
Results and Discussion. Figure 3(a) shows the success rates (SRs) of our method and the baseline methods, as function of the SPLs. For each algorithm and environment configuration, we report the performance of the NN with the best depth N across the ranges specified in the previous paragraph (see Figure 9 in Appendix C.2 for additional results concerning different values of N).
Image Sizes
270x270
450x450
630x6330
VIN
85.3210.14
71.71 72.4310.74
GPPN
85.79
81.72+0.22
76.31 20.75
Highway VIN
85.81+0.6
81.8820.90
73.21
DT-VIN (ours)
86.5440.5
82.78+o.a
77.4+0.98
CNN+A'
84.4220.7
82.11
76.1921.23
4.2. Continuous Control
Here, DT-VIN outperforms all the other methods on all the maze navigation tasks under all the various sizes M and SPLs. Notably, on small-scale mazes with size in M = 15, 35, DT-VIN achieves approximately 100% SRs on all the tasks. For the most challenging environment with M = 100, DT-VIN performs best with the full 5000 layers, and it maintains an SR of approximately 100% on short- term planning tasks with SPL ranging in [1, 200] and an SR of approximately 88% on tasks with SPLs over 1200. Com- paratively, VIN performs well on small-scale and short-term planning tasks. However, even on a small-scale maze with size M = 15, VIN's SRs drop to 0% when the SPL exceeds 30. Morcover, when the maze size increases to 100, VIN only achieves an SR of less than 40% even on short-term planning tasks with SPL within [1, 100). GPPN performs well short-term planning tasks, but it fails to generalize well on long-term planning tasks, which also decreases to an SR of 0% as the SPL increases. Highway VIN performs well across tasks with various SPLs on a small-scale maze with M = 15. However, it shows a performance decrease on larger-scale maze tasks with M = 35, 100. Figure 3(b) shows the optimality rates (ORs) of the algorithms, which measure the rate at which the model outputs the optimal path. Our DT-VIN maintains consistent ORs compared to SRs. However, some other methods-esppecialy Highway exhbith a clear decrease in ORs, indicating that the paths generated by these methods is often sub-optimal.
We evaluate the algorithms on two continuous control tasks: Point Maze and Ant Maze (Foundation, 2022; Fu al., 2020). These tasks are more challenging than simple maze navigation: the agent must not only conduct (explicit or implicit) planning over the complex maze but also output the torques to direct the controlled agent (a 2-DOF ball or a 3D quadruped robot) toward the correct direction, as shown in Figure 4. The input includes the maze's top-down view, agent and goal locations, positional values, and velocities of the agent's body parts. The network architecture integrates planning module structurally similar to that used in 2D maze navigation. The outputs of this module, combined with additional state information, are aggregated and passed to the policy mapping module to produce the control action. We train the model using the D4RL offline Point/Ant Maze dataset, which features a fixed x 12 maze layout across all tasks, varying only in start and goal positions. However, during evaluation, the model is tested on substantially larger and more complex mazes (35 x 35 and 100 100) with previously unseen layouts and start/goal configurations, sig- nificantly increasing task complexity and demanding strong generalization capabilities. To address this challenge, we pretrain the planning module on a diverse and readily avail- able 2D maze navigation dataset endow the model with transferable planning skills. Please refer to Appendix D for more details. Table 2 shows the results of this experiment. DT-VIN solves the mazes at a much higher rate than all the baseline methods. Specifically, in the challenging 100 x 100 Ant Maze, DT-VIN achieves a notable 51% higher success rate than the second-best baseline.
Addtional Experiments. Due to space constraints, we eval- uate challenging cases with noisy maze observations in Ap- pendix C.4 and different transition types (agents moving to any of the eight Moore neighborhood cells) in Appendix C.3. We also evaluate 3D ViZDoom navigation with first-person inputs (Wydmuch et al., 2019), where the model plans based on noisy maze matrix predictions (Appendix C.5). DT-VIN outperforms all compared methods these cases.
4.3. Rover Navigation
We further evaluate the algorithms on the rover navigation task, where the algorithm conduct planning based on a ter-
6

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long- Term Planning
DT-VIN (ours) --- VIN
---
GPPN ...... High-ay VIN
15 x 15 Mase
35 25 Maue
300 x 120 Moze
15 15 Maze
35 x 25 Muse
100 x 100 Muse
Succoss Itate CM1
2000
Hale CR.)
100
Bale (K1
300
Optmailty Rate: 130
Optimaltt Rate (M)
50-
60-
50-
50-
16n
19h6
250
1950 Shortest Fath Length
Shortest Path Length
Shortest Path Longth
Shortest Path Longth
Shortest Path Length
Shortest Path Length
(a) Success Rate
(b) Optimality Rate
Figure 3: SRs and ORs for different algorithms function of the shortest path length on 2D maze navigation tasks with various sizes. For each algorithm, we select the best result across various depths. Specifically, for our DT-VIN, the optimal depth consistently corresponds to the maximum value in the range: 200 for mazes of size 15, 600 for size 35, and 5000 for size 100. For other methods, the optimal depth differs per task. In the maze of size 100, the optimal depth for all the baselines is 600. See Figure 9 and Figure 10 Appendix C.2 for additional results at other depths.
the task setting and the models' architecture.
As shown in Table 3, our DT-VIN outperforms all compared methods across various terrain image sizes. Notably. with larger image sizes (particularly 630 x 630), our DT-VIN outperforms VIN by more than 5%. We also compare with special baseline, CNN+A", which combines classification and classical planning algorithms. This method trains a CNN classify whether an 18 18 image patch is an obsta- cle using elevation data, then uses A to conduct planning based on this prediction. While this unsurprisingly is able to outperform VIN, it is still outperformed by DT-VIN.
(a) Point Maze
(b) Ant Maze
Figure 4: The Point Maze and Ant Maze environment in the training dataset from D4RL (Foundation, 2022; Fu et al., 2020), where map size 9 12. Please refer to Appendix D for larger (c.g.. 100 100) mazes in our testing dataset.
140
4.4. Ablation Study
120
-2200
We perform multiple ablation studies with M = 35 maze and an NN with depth N = 600 to assess the impact of DT- VIN components, including the dynamic latent transition kernel and its associated softmax operation, the adaptive highway loss and the network depth. Unless otherwise indicated, all these elements are included.
2220
63
2250
(a) Terrain image
(b) Elevation data
Figure 5: (a) and (b) show patch of terrain image and elevation data from Apollo 17 landing tasks.
Dynamic Latent Transition Kernel. We evaluate four vari- ants of dynamic latent transition kernels: fully invariant, observationn-yyamme only, latent state-dynamic only, fully dynamic (incorporating both observation and latent state- dynamic). Figure 6(a) and (b) show the SRs of all these variants w.r.t. the SPLs and obstacle counts respectively, The variant with invariant kernels performs the worst. Both ineevvationdynamiiee and latent state-ynaaii components contribute to the performance of DT- VIN removing either results in a substantial performance drop. It's worth noting that the performance gap between the dynamic kernel and invariant kernel grows with increasing the SPL or obstacle counts. This improvement is due to the improved model's representational capacity by the dynamic kemnl,, reducing compounded errors over extended planning horizons and complex environments with increased obstacle counts, as shown in Figure 6(c) and (d). Figure 6(e) gives an illustra-
rain image (e.g., Figure 5(a)) rather than the elevation data (e.g., Figure 5(b)). While terrain images, visual represen- tations from aerial photographs, typically lack elevation information, they are generally more readily available or less expensive to obtain. We evaluate the Apollo 17 landing tasks, featuring images with a resolution 0.5 meters per pixel (Berlin, 2018). We crop the terrain image into various- sized patches, each 18 18 patch defining cell. The cell is considered obstacle if the associated area exhibits ele- vation angle exceeding 10 degrees. Analogous to supervised learning, the training phase uses external elevation data to generate the expert path, while in the testing phase, this data is inaccessible to the models and only used to assess performance metrics. Please refer to Appendix for details

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Term Planning
Invariant
latent state-dynamic only -.-
--bberrvtoondyyyami only
fully dynamic
35
Success Rate (15).
1000
Success (96)
Cross Entropy Locs
Cross Entropy Loss
75-
75
1.
50
50
2.0
3.0
25-
100
200
300
100
200 300 400
150
200
300
300 200 300 400 Obstacle Counts
Shortstt Path Lar gth
Obstadi Counts
Shortest
Length
(a) SRs w.r.t. SPI.s
(b) SRs w.r. Obstacle Counts
(c) Loss w.r.l. SPLs
(d) Loss W.C. Obstrele Counts
(c) Learned Kernels
Figure 6: Ablation studies on the dynamic latent transition kernel. (a) and (b) show the success rates (SRs) w.r.t. to
various shortest path lengths (SPLs) and obstacle counts of the tasks, respectively. (c) and (d) show the losses. (e) shows the learned latent transition kernels of DT-VIN.
Success Rata (9%)
100
35 x 35 Made
Softmax w/ Softmax
Norm of Gradient
35 35 Maze
Success Rate (16)
100
35 x 35 Mace
L Norm or Gradlent
1.00
35. Ja Mace
Success Rate (%)
1.00
35 35 Maze
75-
1.4
0.75
Stngle rightay Law
July Highway Loan
D DVVN (N06001
8
9,50
Highway Connuction
DIANN (N--100)
25-
0.5
0.25
04
DTVIN IN-
0.00
100
300
200
300
10
100
300
Shortest Path Length
Training Epochs
Shorlest Path Length
Training Epachs
Shortest Path Length
(a) Softmax Operation: Success Rate(Left), Gradient Norm(Right)
(b) Adaptive Highway Loss: Success Rate(Left), Gradient NommRRight)
(c) Depths Model
Figure 7: (a) and (b) show ablation on softmax operation and adaptive highway loss, respectively. The left and right sides of each sub-figure show the success rate, and the averaged LI norm of the gradient over the first 10 layers, respectively. (c) show the the success rate across various depths of model.
tion of DT-VIN's dynamic transition kernels.
worst because the skip connections over intermediate layers can disrupt the structure of value iteration. These results are consistent with those in existing work (Wang et al., 2024a).
Softmax Operation on Latent Transition Kernel. As shown in Figure 7(a) (left). the variant without the softmax operation on the latent transition kernel fails on all the tasks. This failure is due to exploding gradients, wherein the gra- dient becomes extremely large, eventually resulting in the model's parameters overflowing and becoming a NaN (Not a Number) value, as depicted in Figure 7(a) (right).
Depth of Planning Module. Figure 7(c) shows the SRs of VIN with various depths. Here, increasing the depth dra- matically improves the long-term planning ability. For ex- ample, for tasks with an SPL of 200, the variant with depth N = 300 outperforms the variant with depth N = 100. Moreover, for tasks with an SPL of 300, the deeper variant with depth N 600 excels greatly. Other methods like VIN and GPPN do not show a clear performance improve- ment when the depth increases, please refer to Figure 9 in Appendix C.2 for more details.
Adaptive Highway Loss. We evaluate several additional variants verify the effectiveness of adaptive highway loss. Figure 7(b) (left) shows the success rate of all variants. (a) The variant without highway loss suffer decrease in performance, as shown in Figure 7(b). (b) We also evaluate variant fully highway loss, which enforces a highway loss on each hidden layer without adaptive adjustment by removing the filter L [n21] in Eq. (3). This variant performs worse than the one with adaptive highway loss, highlighting the necessity of adaptive adjustment. (c) We assess a variant called single highway loss, which only apply highway loss to a specific layer n where n = 1 by substituting 11 (n221) with ((m=1) in Eq. (3). This variant performs worse than the adaptive highway loss, demonstrating the critical role of the component n > I in performance. (d) Finally, we evaluate a variant named inter highway connection, which constructs skip connections across the intermediate layers of the planning module, akin to techniques used in Highway Nets (Srivastava et al., 2015a). This variant performs the
Additional Ablation Studies. Due to space constraints, please refer to Appendix F and G. which show the robust- ness of DT-VIN with 50% of the training dataset, the choice of the length of the expert path I, impact of jumping hyper- parameter J, and the scaling of computation complexity, dataset requirements, and model size with task scale.
5. Related Work
Variants of Value Iteration Networks (VINs). Several variants of VIN (Tamar et al., 2016) have been proposed in recent years. Gated Path Planning Networks employ gating recurrent mechanisms to reduce the training instability and hyperparameter sensitivity seen in VINs (Lee et al., 2018).
8

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
To mitigate overestimation bias (which is detrimental to learning here), dVINs were proposed and use weighted double estimator as alternative to the maximum opera- tor (Jin et al., 2021). For addressing challenges in irregular spatial graphs, Generalized VINs adopt a graph convolution operator, extending the traditional convolution operator used in VINs (Niu et al., 2018). To improve scalability, AVINS introduce an abstraction module that extracts higher-level information from the enviromment and the goal (Schleich et al., 2019). To improve scalability, Zhao et al. proposed implicit differentiable planners that decouple forward and backward passes (Zhao et al., 2023). In contrast, our work takes a complementary architectural approach, redesigning VINs for deeper and more expressive planning over larger- scale tasks. For transfer learning, Transfer VINs address the generalization of VINs target domains where the action space or the environment's 's features differ from those of the training environments (Shen et al., 2020). More recently, VIRN was proposed and employs larger convolutional ker- nels to plan using fewer iterations as well as self- atfeettention to propagate information from each layer to the final output of the network (Cai et al., 2022). Similarly, GS- VIN also uses larger convolutional kernels but to stabilize training and also incorporates gated summarization module that reduces the accumulated errors during value iteration (Cai et al., 2023). Most related to DT-VIN is other recent work that focused on developing very deep VINs for long-term planning. Specifi- cally, Highway VIN (Wang et al., 2024a) incorporates the theory of Highway Reinforcement Learning (Wang et al., 2024b) to create deep planning networks with up to 300 layers for long-term planning tasks. Highway VIN modifies the planning module of VIN by introducing an exploration module that injects stochasticity in the forward pass and uses gating mechanisms to allow selective information flow through the network layers. Our method, however, achieves even deeper planning by incorporating a dynamic transition matrix in the latent MDP and adaptively weighting each layer's connection to the final output.
proach is more elosely aligned with the computation of the true VI algorithm. Similar kernels, dependent image (Chen et al., 2020) or the coordinates of an image (Liu et al., 2018), have been previously used in Computer Vision.
input
6. Conclusions
Previous work introduced VIN as differentiable neural net- work for planning in artificial intelligence and reinforcement learning. While VINs have been successful at short-term small-scale planning, they start to fail quite rapidly as the planning horizon and the scale of the tasks grows. We ob- served that this decay in performance is principally due to limitations in the (1) representational capacity of their network and (2) its depth. To alleviate these problems, we propose several modifications to the architecture, including dynamic transition kernel to increase the representation capacity and an adaptive highway loss function to ease the training of very deep models. Altogether, these modifica- tions have allowed us to train networks with 5000 layers. In line with previous work, we evaluate the efficacy of our proposed Dynamic Transition VINs (DT+VINS) on 2D/3D maze navigation, continuous control, and rover navigation. We find that DT-- scale longer-term and larger-scale planning problems than previous attempts. To the best of our knowledge, DT-VINs is, at the time of publication, the current state-of-the-arr planning solution for these specific environments. We note that the upper bound for this ap- proach (i.e., the scale of the network and, consequently, the scale of the planning ability) remains unknown. As our experiments were limited mostly by computational cost and did not observe instability, we expect that with the growth of available computational power, our method will scale to even longer-term and larger-scale planning.
Neural Networks with Deep Architectures. There is long history of developing very deep neural networks (NNs). For sequential data, this prominently includes the LSTM architecture and its gated residual connections, which help alleviate the "vanishing gradient problem" (Hochreiter & Schmidhuber, 1997; Hochreiter, 1991). For feedforward NNs, a similar gated residual connection architecture was used in Highway Networks (Srivastava et al., 2015b) and later in the ResNet architecture (He et al., 2016), where the gates were kept open. Such residual connections are still ubiquitous in modern language architectures, such as the Generative Pre-trained Transformer (GPT) (Achiam et al., 2023). Our method dynamically employs skip connections from select hidden layers to the final loss, utilizing a state and observation map-dependent transition kernel. This ap-
9

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Acknowledgements
Cai, J., Li, J., Zhang, M., and Tei, K. Value iteration net- works with gated summarization module. IEEE Access, 2023.
We gratefully acknowledge the insightful comments from the ICML 2025 reviewers, as well as the constructive feed- back received during earlier stages of peer review. We especially thankful to Yanning Dai for her valuable support with the continuous control experiments. The re- search reported in this publication was supported by funding from King Abdullah University of Science and Technol- ogy (KAUST) - Center of Excellence for Generative AI, under award number 5940. We also acknowledge the sup- port by the grant EP/Y002644/1 under the EPSRC ECR International Collaboration Grants program, funded by the International Science Partnerships Fund (ISPF) and the UK Research and Innovation. This work was additionally sup- ported by the European Research Council (ERC, Advanced Grant Number 742870). For computer time, this research used Ibex managed by the Supercomputing Core Laboratory at King Abdullah University of Science and Technology (KAUST) in Saudi Arabia.
Chen, Dai, X.,, Liu, M., Chen, D., Yuan, L., and Liu, Z. Dynamic convolution: Attention over convolution kernels, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11030-1 1039, 2020.
Chen, Y. F., Everett, M., Liu, M., and How, J. P. Socially aware motion planning with deep reinforcement learning. In 2017 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pp. 1343-1350 IEEE, 2017.
Eysenbach, B., Salakhutdinov, R. R., and Levine, S. Search on the replay buffer: Bridging planning and reinforce- ment learning. Advances in neural information process- ing systems, 32, 2019.
Foundation, F. Gymnasium: A toolkit for developing and comparing reinforcement learning environments. https://git /ttl:://gitttii com Sotm//arram-FFundaaaaaaa Foundation/ im,
Impact Statement
Gymnas i um, 2022.
This paper aims to advance the field of Machine Learning. While acknowledging there are many potential societal con- sequences of our work, we believe that none need be specially highlighted here.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Hafner, D., Lillicrap, T. P., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. Pro- ceedings of the 8th International Conference on Learning Representations, 2020. URL https: /tappp:://ppeeeee /oppnrevie..eeeeeeee net/ / forumr 1d=51 1iie-$11occC 10TC4tDS.
References
Achiam, J., Adler, S., Agarwal. S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Hafner, D., Lillicrap. T. P., Norouzi, M., and Ba, J. Mas- tering atari with discrete world models. Proceedings of the 9th International Conference on Learning Represen- tations, 2021. URL https: / opwnreviewee..ee/////p net/ forum? 1dd
Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W. Rein- forcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32, 2019.
Hafner, D., Pasukonis, J.,
Ba, J., and Lillicrap, T. P. Master- ing Diverse Domains through World Models. arXiv, 2023. URL https: / arxiv. www.eiwwwiw.....e..ee
Bain, M. and Sammut, C. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
orgg abs///////// / 2/0//3311110 04104.
Bellman, R. E. A markovian decision process. Journal of Mathematics and Mechanics, 6(5))679-68)) 1957. URL http:/ /tww//wwww/t. http://wwww......o//o/.o wwtp:orrrggg//abbeeeeboeeeeee 9000000 900506.
Hart, F.,, Nilsson, N. J., and Raphael, B. formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100-177 1968. doi:
Berlin, T. U. Moon apollo 17 Iroc nac land- URL. usgs. . search/ map/moon_ apwllloooonaaolooooaaoaaaallllla 17 lroo_rrro____eeeeeeeeeeeee__eele pac_aa___aaa laa__aa__aandnnnggggaa landing site_orthomosaa c c_50em. Accessed: 2024-10-02.
ing site orthomosaic 50cm, 2018.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
https://astrogeolog
Cai, J., Li, Mao, Z., and Tei, K. Value iteration residual network with self-attention. In International Conference on Intelligent Systems Design and Applications, pp. 16- 24. Springer, 2022.
Hochreiter, S. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München, 1991. Ad- visor: J. Schmidhuber.
10

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Hochreiter, S. and Schmidhuber, J. Long Short-Term Mem- ory. Neural Computation, 9()):1335-178) 1997.
Schaal, S. Learning from demonstration. Advances in neural
information processing systems, 9, 1996.
Janner, M., Du, Y., Tenenbaum, J., and Levine, S. Planning with diffusion for flexible behavior synthesis. In Interna- tional Conference on Machine Learning, pp. 9902-9915. PMLR, 2022.
Schleich, D., Klamt, T., and Behnke, S. Value iteration
networks on multiple levels of abstraction, arXiv preprint arXiv: 1905. 1068, 2019.
Schmidhuber, J. Making the world differentiable: On us- ing fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non- stationary environments. Technical Report FKI-126-90 (revised), Institut für Informatik, Technische Universität München, November 1990a. (Revised and extended ver- sion of an earlier report from February.).
Jin, X., Lan, W. Wang. T., and Yu, Value iteration networks with double estimator for planetary rover path planning. Sensors, 21(24):8418, 2021.
Lample. G. and Chaplot, D. S. Playing fps games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Schmidhuber, J. An on-line algorithm for dynamic rein- forcement learning and planning in reactive environments. In Proc. IEEE/INNS International Joint Conference Neural Networks, San Diego, volume 2, pp- 253-258- 1990b.
Lee, L., Parisotto, E., Chaplot, D. S., Xing. E., and Salakhut- dinov, R. Gated path planning networks. In Interna- tional Conference on Machine Learning, pp. 2947-2955. PMLR, 2018.
Li, W., Yang. B., Song, G., and Jiang, X. Dynamic value iteration networks for the planning of rapidly changing uav swarms. Frontiers of Information Technology & Elec- tronic Engineering. 22(5):687-666 666, 2021.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, I.,, Schmitt, S., Guez, A., Lockhart, F.,, Hassabis, D.. Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839): 604-609 2020.
Liu, R., Lehman, J., Molino, P., Petroski Such, F., Frank, E., Sergeev, A., and Yosinski, J. An intriguing failing of convolutional neural networks and the coordconv solution. Advances in neural information processing systems, 31, 2018.
Shen, J., Zhuo, H., Xu, Zhong, B., and Pan, S. Transfer value iteration networks. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 34, pp. 5676- 5683, 2020.
Mishra, U. A., Xue, S., Chen, Y., and Xu, Generative skill chaining: Long-horizon skill planning with diffussin models. In Conference Robot Learning, pp. 2905- 2925. PMLR, 2023.
Silver, D., van Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley, T., Dulac-Arnold, G., Reichert, D. P., Rabi- nowitz, N. C., Barreto, A., and Degris, T. The predic- tron: End-to-end learning and planning. Proceedings of the 34th International Conference on Machine Leurning, 70:3191-3199, 2017. URL mir. wwwpss/vvvv.e...... wttpee//t///iivvee1eeee///e///e/ sileeeeellliiiialaa.a.a.a pdf.
Niu, S., Chen, S., Guo, H., Targonski, C., Smith, M., and Kovačević, J. Generalized value iteration networks: Life beyond lattices. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L.,, and Su, Y. LIm-planner: Few-shot grounded plan- ning for embodied agents with large language models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2998-3009, 2023.
Pflueger, M., Agha, A., and Sukhatme, G. S. Rover-irl: In- verse reinforcement learning with soft value iteration net- works for planetary rover path planning. IEEE Robotics and Automation Letters, 1(2):138711394, 2019.
Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Srivastava, R. K., Greff, K., and Schmidhuber, J. Highway networks, arXiv preprint
2015a.
Srivastava, R. K., Greff, K., and Schmidhuber, J. Training very deep networks. Advances in neural information processing systems, 28, 2015b.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita- tion learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 627-635. JMLR Workshop and Conference Proceedings, 2011.
Sutton, R. S. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4): 160-163, 1991. doi: 10.1145/122344.12277

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Sutton, R. S. The Bitler Lesson.
2019. URL
etei deas net/ IncIdeas/ BitterLesson html.
Tamar, A., Levine, S., Abbeel, P., Wu, Y., and Thomas, G. Value iteration networks. Advances in Neural Information 2016.
Processing Systems,
Wang, Y., Li, W., Faccio, F., Wu, Q. and Schmidhu- ber, J. Highway value iteration networks. Proceed- ings of the 41st International Conference on Machine Learning, 2024a. URL 2406. 03485.
org/ abs/
Wang, Y., Liu, H., Strupl, M., Faccio, F., Wu, Q. Tan, X., and Schmidhuber, Highway Reinforcement Learn- ing. arXiv, 2024b.
org/ abs/
2405. 18289.
Wöhlke, J., Schmitt, F., and van Hoof, H. Hierarchies of planning and reinforcement learning for robot navigation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 10682-10688. IEEE, 2021.
Wydmuch, M., Kempka, M., and Jaskowski,
ViZ- Doom Competitions: Playing Doom from Pixels. JEEE Transactions on Games, 11(3):44--59, 2019. doi: 10.1
Zhao, L., Xu, H., and Wong, L. L. Scaling up and stabiliz- ing differentiable planning with implicit differentiation. In The Eleventh International Conference on Learning Representations, 2023. URL https: / /pwnrreiee......eee net wtt////rummii///// wiw.umidddddxbbeeooo30o3
12

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
A. Limitations and Future Works
The principal limitation of our work compared to VIN and Highway VIN is the increased computational cost (see Appendix G). This a consequence of the scale of the network. The past decades have seen AI dominated by the trend of scaling up systems (Sutton, 2019), so this is not likely long-term issue.
Future work will explore the impact of a more sophisticated transition mapping module (this work uses a single CNN layer for this purpose) in more challenging real-world applications, such as real-time robotics navigation in dynamic and unpredictable environments, Additionally, recent diffusion-basee planners (Janner et al., 2022; Mishra et al., 2023) offer a complementary approach to long-horizon reasoning, which may be integrated with VIN-style architectures to improve planning capacity and temporal abstraction.
B. Method: Additional Details
B.1. Loss Function
In imitation learning, the loss function in Eq. can can be written as
(m2g (n mod J=0) - (w))
log (
(0)
Σ
Σ
K
(2,8.1) a
In RL, where the policies are learned through policy gradient, the loss function can be rewritten
(0) D (Z.D,R ISHSN
Σ
Σ
1 (m21) (n mod J-0] (-Rlog Rlog fr (v (z).
where y the excuted action, and R the cumulative future reward.
B.2. Translation Equivariance
Although our dynamic latent transition kernel is input-dependent and breaks the conventional weight-sharing scheme, it still preserves translation equivariance- desirable property in spatial planning tasks. other words, shifting the input (e.g., the maze image) leads corresponding shift in the output. While this may seem counterintuitive at first, it is in fact direct consequence of our design: the kernel T f (xx) is produced by CNN, which is itself translation equivariant. As result, the downstream value iteration process retains this property (see Eq. 2).
C. Maze Navigation: Additional Experimental Details and Results
C.1. 2D Maze Navigation: Experimental Details
Figure shows some visualizations of some of the different 2D maze navigation tasks we experiment with. Our experimental setup follows the guidelines established in the GPPN paper (Lee al., 2018). For these tasks, the datasets for training, validation, and testing comprise 25000, 5000, and 5000 mazes, respectively. Each maze features goal position, with all reachable positions selected as potential starting points. Note that this setting, as done by GPPN, produces a distribution of mazes with non-uniform SPLs, which is skewed towards shorter SPLs. Table 4 shows the hyperparameters used by our method. Note that, while DT-VIN consistently uses 3 for the size of the latent transition kernel F and 4 for the size of the latent action space [A,, other methods instead used their best rest-performing sizes from between 3 and 5, and between 4 and 150, respectively.
C.2. 2D Maze Navigation with Different Depths of Models
Figure shows the success rate of all the algorithms on the 15 x 15, 35 x 35, 100 100 mazes as a function of the shortest path length and the depth of the network. Similarly, Figure 10 shows the corresponding optimality rates.
C.3. 2D Maze Navigation with Difffrrnt Transition Types
Following the GPPN paper (Lee et al., 2018), we run additional experiments using different transition type: the Differential Drive transition, where the agent can move forward along its orientation or rotate 90 degrees left or right, and the MOORE
13

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long- Term Planning
Optimal Path
Optimal Path
Goal
Goal
Start
Start
Optimal Path
Goal Start
(a) 15 X 15 Maze
(b) 35 X 35 Maze
(c) 100 x 100 Maze
Figure 8: Examples of the 2D maze navigation tasks.
Table 4: 2D Maze Navigation Hyperparameters
Hyperparameter
Value
Transition Mapping Module
A 1 1-layer CNN with 3 x kernel
Reward Mapping Module
A 1 I-layer CNN with 1 x kernel
Policy Mapping Module
A I-layer FCN
Latent Transition Kernel Size (F)
Latent Action Space Size ([AA)
Optimizer Learning Rate
RMSprop le-3
Batch Size
32
15 X 15 maze: 200
Depth of Planning Module
35 x 35 maze: 600
100 x 100 maze: 5000
transition, where the agent can relocate to any of the eight adjacent cells that comprise its Moore neighborhood. As shown in Table and 9, DT-VIN consistently outperforms all the compared methods regardless of the transition type used.
C.4. 2D Maze Navigation with Controlled Noisy Maze
We evaluated the methods under controlled noise. To emulate the prediction noise, we add Gaussian noise to the original maze and enforce the value within (0, 1) by clip(maze- + c, 1), where c a noise sampled from Gaussian distribution. The results of this are below. DT-VIN seems more robust handling noise than Highway VIN and VIN. GPPN shows robustness in the short term setting, but is not able to solve the task when long term planning required.
C.5. 3D ViZDoom Navigation
Following the methodology of the GPPN paper, we test our method on 3D ViZDoom (Wydmuch et al., 2019) environments. Here, instead of directly using the top-down 2D maze as in the previous experiments, we use the observation consists of RGB images capturing the first-person perspective of the environment, as illustrated Figure la). Then, CNN is trained to predict the maze map from the firs-person observation. The map is then given input the planning model, using the same architecture and hyperparameters as the 2D maze environments. For each algorithm, we select the best result across the various network depths N = 30, 100, 300, 600. We find that the optimal depth for DT-VIN is 600, for GPPN is 300, for VIN is 300, and for Highway VIN is 300. We evaluated the algorithm on 3D ViZDoom mazes with grid 35 x 35, where each cell in the grid corresponds to a 64 x 64 map unit area, the standard spatial measurement in the game engine. Figure 11(b) shows the SRs. Predictably, the performance of all the baselines decreases compared to the 2D maze environments due to
14

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Table 5: Rover Navigation Hyperparameters
Hyperparameter
Value
Transition Mapping Module
A 10-layer CNN 10-layer CNN (sharing the first layers with Transition Mapping Module)
Reward Mapping Module
Policy Mapping Module
A l-layer FCN
Latent Transition Kernel Size (F)
Latent Action Space Size (lA)
Optimizer Learning Rate
RMSprop 1e-3
Batch Size
32
270 x 270 50
Depth of Planning Module
450 x 450 100
630 x 630: 200
Table 6: 3D ViZDoom Preprocessing Network
Hyperparameter
Value
Batch Size (B)
32
Image Directions (D)
4
Image Channels (C)
3
Image Width (W)
24
Image Height (H)
32
Input Size
(B, M, M, D, W,H,C
Layer 1 (Convolution)
(3, 8, 1)
Layer 2 (Convolution)
(32, 64, 2, (384, 256)
Layer 3 (Linear)
Layer 4 (Convolution)
(1024, 64, 1,1)
Layer 5 (Convolution)
(64, 1, 3, 1)
Output Size
(B, M, M)
Optimizer
Adam
Learning Rate
le-3 (0.9, 0.999)
Betas
15

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Table 7: The success rates for each method under tasks with different ranges of shortest path length. For each algorithm, we choose the best result from a range of depths. Specifically, for our VI-, the optimal depth consistently corresponds to the maximum value in the range: 600 for size 35, and 5000 for size 100. For other compared methods, the optimal depth differs depending on the task. In the maze of size 100, the optimal depth for all the baselines is 600. For additional results, see Figure in Appendix C.2.
Maze Size
35 x 36
100 x 100
Ranges of Shortost Path Lengths
[1.100]
[100, 200)
[200, 300]
[1,600]
[600, 1200]
[1200, 1800]
VIN (Tamar
2016)
68. 68.41
0.0g0.00
0,00g..0 oo
40.000g0
0.00 00
0:000.0000
GPPN (Lee et al., 2018)
95.7 14..5
0.30ga
0.00g0.00
75.7220.04
0.00
0.00g0000
Highway (Wang al., 2024aa
90.6743.02
66 20.000
34.40gm02
0.00ga.00
0.000
DT-VIN (ours)
100.00
99.99.0.01
99.77 +
90.9840.00
99.56
88.65-4477
Table 8: The success rate (%) for each method in 35 x 35 2D maze navigation with Differential Drive transition type, where the agent can move forward along its orientation or rotate 90º left or right.
Shortest Path Length
[1,150]
[150,300]
[300,500]
VIN
68. 18.12
0.03
0.00
GPPN
83. 1:223
0.31
0.0
Highway VIN
87. #3.73
57.1
49.1
DT-VIN (ours)
100.00
100.00.0.00
99.9940.00
the additional noise introduced by the predictions. Here, DT-VIN outperforms all the methods compared to the task over all the various SPLs.
To be in line with previous work, to that used in the GPPN paper and others (Lee et al., 201 8; Lample & Chaplot, 2017). In 3D ViZDoom, maze is designed on grid of M x M cells. Each cell in this grid corresponds to an area of 64 x 64 map units within the 3D ViZDoom environment. The map unit is the basic measure of space used in the ViZDoom game engine to define distances and sizes. Specifically, for each cell in the M x M 3D maze, the RGB first-person views for each of the four cardinal directions are given as state to a preprocessing network (see Figure 1a))). This network then encodes this state and produces an M x M binary maze matrix. The hyperparameters and exact specification of the network are given in Table 6.
state representation preprocessing stage for the 3D ViZDoom environment similar
D. Continuous Control: Additional Experimental Details and Results
In this section, we present experimental details of continuous control, including Point Maze and Ant Maze. Figure 12 demonstrates Point Maze with larger sizes 35 x 35 and 100 x 100. For continuous control tasks, the input includes a 4M x 4M top-down view of the maze, agent and goal locations, positional values, and velocities of the agent's body parts. Network architecture comprises 2-layer CNNs for reward and transition mapping. and 3-layer fully connected network for policy mapping, which takes outputs from the planning module and additional state information to produce controlled actions. The planning module in this architecture overlaps with that in the 2D maze navigation task. Therefore, pre-trained parameters from 2D maze navigation tasks. Table 12 lists the hyperparameters of DT-VIN for the continuous control tasks. We train using PointMaze-Large and Largeeze-arge-Paay datasets from D4RL (Fu et al., 2020) and pre-train on the 100x x 100 2D maze.
E. Rover Navigation: Additional Experimental Details and Results
Table 5 shows the hyperparameters of DT-VIN for the rover navigation tasks. For the transition and reward mapping modules, we employ 10-layer CNNs, with the first 8 layers shared between them.
F. Ablation Studies: Additional Experimental Details and Results
F.I. Ablation on the Jumping Hyperparameter J
To reduce the computational complexity of highway loss, we apply adaptive highway loss only to layers n satisfying the condition n mod J = 0, where J is a hyperparameter set to 10 our experiments. Here, the main idea is to build the
16

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
100
15 x 15 Maze
100
35 x 35 Maze
100
100 x 100 Maze
DT-VIN(N 200)
Success Rate (%)
80-
DT-VIN(N= =00))
Success Rate (%)
80
Success Rate (%)
80-
DT-VIN(N=30)
60
60
3T-VIN(N=300)
DT-VIN(N=600)
60
500V
DT-VINNN=100)
DT-VIN(N=600)
40
40
30-
40-
20
20-
20
0-
0-
0
20
40
60
0
100
200
300
500
1000
1500
Shortest Path Length
Shortest Path Length
Shortest Path Length
(a) DT-VIN (ours)
15 x 15 Maze
35 x 35 Maze
100 x 100 Maze
100
100
100
VIN (N=200)
VIN (N= 600)
5INON=
Success Rate (%)
80
VIN (N=100)
Success Rate (96
80
VIN (N=300)
Success Rate (%)
66N(N=
VIN (N=30)
VIN (N=100)
60
VIN (N=30)
60
40
40
20
20
0
20
40
100
200
300
500
1000
1500
Shortest Path Length
Shortest Path Length
Shortest Path Length
(b) VIN
15 x 15 Maze
35 Maze
100
100 Maze
100
100
100
GPPN (N=600)
GPPN(N=50000
Success Rate (%)
80
Success Rate (%)
80
GPPN (N=300)
Success Rate (%)
GPPN(N=600)
GPPN (N=100)
60
GPPN (N-30)
60
40
40
GPPN (N=
20
GPPN (N=100)
20
20
GPPN (N=30)
0
20
40
60
100
200
300
500
1000
1500
Shortest Path Length
Shortest Path Length
Shortest Path Length
(c) GPPN
Figure 9: The success rate of each method as a function of shortest path length and network depth. The green and red curves overlap in the first plot of (a).
highway connections an interval J for example, every 10 neural network layers, as in our setting where J 10. Using this, the number of the loss terms will reduce only 1/J of the original one. Table 13 shows the magnitude of the computational speedup as consequence of this implementation detail.
F22 Ablation the Choice of l
The knowledge of the length I of the expert path naturally exists in the imitation learning case. However, for the case where such information is unknown, one can use either the length of non-expert data or some heuristic methods to estimate I when the actual / is completely unknown, e.g., using the distance between the start and the goal position.
To measure the effect of overesimatiioruuderssimma
we experiment with various estimated values of the length of the shortest path 7, which are 0, 0/1,2,1, 1, 22, N (where / is the actual length of the shortest path, N is the depth of the planning module). Second, to evaluate the case when the estimation of / has variance, we use 7. max(c, 0) as the estimation, with e sampled from a Gaussian distribution (11, 1). Third, we also assess two additional variants for estimating I: (a) One variant that utilizes the length of non-expert trajectories for 1; (b) Another variant that estimates the shortest path length
17

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
15 x 15 Maze
35 x 35 Maze
100 x 100 Maze
100
100
100
Optimality Rate (%)
DN 200)
Optimality Rate (%)
Optimality Rate (%)
80
DT-VIN(N=100
80
33))
60
DT-VIN(N 300)
DT-VININ= = 600)
60
DT-VIN(N=5000)
DT-VININ=100)
DT-VIN(NN-00)
40
40
DT-VIN(N
40
20
20
0
20
40
60
100
200
300
500
1000
1500
Shortest Path Length
Shortest Path Length
Shortest Path Length
(a) DT-VIN (ours)
15 x 15 Maze
35 x 35 Maze
100 x 100 Maze
100
100
100
Optimality Rate (%)
VIN (N=200)
Optimality Rate (%)
VIN (N= 600)
Optimality Rate (%)
50N0N=
80
VIN (N=100)
80
VIN (N=300)
60N(N=
VIN (N=30)
VIN (N=100)
60
VIN (N=30)
60
40
40
20
20
0
20
40
100
200
300
500
1000
1500
Shortest Path Length
Shortest Path Length
Shortest Path Length
(b) VIN
15 x 15 Maze
35 Maze
100 x 100 Maze
100
100
100
Optimality Rate (%)
Optimality Rate (9%)
GPPN (N=600)
Optimality Rate (%)
GPPN(N=50000
80
80
GPPN (N=300)
GPPN(N=600)
GPPN (N=100)
60
GPPN (N-30)
60
40
40
GPPN (N=
20
GPPN (N=100)
20
20
GPPN (N=30)
20
40
60
100
200
300
500
1000
1500
Shortest Path Length
Shortest Path Length
Shortest Path Length
(c) GPPN
Figure 10: The optimality rate of each method as function of shortest path length and network depth. The green and red curves overlap in the top-left plot.
heuristically using the 1.1 distance between the start Us) and the goal (Igs i.e., = |ar, - 37gl + 1y+ yg
As indicated in Table 14, both overestimation and underestimation lead a performance degradation of no more than 7%. Additionally, we find that leveraging non-expert data or the heuristic L1 distance only yields a nearly 3% degradation in performance, and performs better than the case when the optimal length is extremely overssiinaeduundeesiineee. These results imply that employing the information from non-expert data or heuristic estimation could be taken as an alternative when the optimal length is not available.
F.3. A Ablation on the Size of Training Dataset
Even in situations where data is rare, DT-VIN still outperforms compared methods. As shown in Table 15, with only 50% of the original dataset, DT-VIN greatly outperforms existing methods. We also highlight the changes compared to the performance with a full-sized dataset in Table 15, where DT-VIN results in less than a 0.2% degradation for tasks within the range [1, 100], while the performing comparison method, GPPN, incurs a degradation of nearly 12%.
18

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Table 9; The success rate for each method in 35 x 35 2D maze navigation with Moore transition type, where the agent can relocate to any of the eight adjacent cells that comprise its Moore neighborhood.
Shortest Path Length
[1,100]
[100,200]
[200,250]
VIN GPPN
66.44
0.00
0.00=0000
1.31
0.04
0.00=0.00
Highway VIN DT-VIN (ours)
83. 83.14
37.1
25. 1 23.28
100.040.00
98.9
96.7 41.22
Table 10: The success rate for each method in 35 x 35 2D maze navigation, with maze noise ~ (0,00.01)
Shortest Path Length
[1,100]
[100,200]
[200,300]
VIN
66.41 27.25
0.00
0.00
GPPN
91.71 11.33
0.21 10.27
0.00
Highway VIN DT-VIN (ours)
$6.67=4.90
57.50
46.40_11.2
99.21
98.1710.01
97. 97.77 -
G. Scaling Experiments
In this section, we examine how the methods scale in terms of computational complexity, dataset requirements, and model size relative to task scale.
G.1. Required Computational Resources
As we have discussed in Section 3.1, our approaches only require |A| x F parameters, where we set |A| = 4 and F = 3 in our experiments. Table 17 shows the memory consumption and training time on NVIDIA A100 GPUs for DT-VIN and the compared methods when using 5000 layers and training for 90 epochs on 100 x 100 maze. As shown in the table below, DT-VIN incurs only slightly higher GPU memory and time costs compared VIN, while being far more memory- than GPPN and Highway VIN. These results are generally consistent with those observed in the 35 x 35 2D maze in Table 16.
G.2. Required Size of Model
In our experiments, the depth of the network required to solve the problem is close to línear with the number of planning steps required by the problem. For maze size M 15, 25, 35, test DT-VIN models at increasing depths in increments of 100 until the optimal performance is achieved. For instance, for mazes of size 25 x 25, we assess depths of 100, 200,300, 400. For maze size M = 100, we assess depths of 4000, 5000, 6000. As Table 18 illustrates, the depth of the smallest network that can solve the task increases slightly more than linearly with the required planning steps. Therefore, it might be feasible to continue increasing the network depth as the problems become more complex.
G.3. Required Size of Training Data
We evaluate the models on the mazes of size 15 x 15, 25 x 25, and 35 x 35, on increasingly larger numbers of expert steps (a step here is one transition from a cell to another cell). Specifically, for each maze size, we look for the smallest n such that, with 5000 x n different mazes (with each maze having a number of expert trajectories and larger mazes having more expert trajectories). DT-VIN can achieve a 2 98% success rate on the longest length planning problem for each maze. The results of this experiment are below (and we will add this to camera-ready version of the paper). While this is bit of coarse result, comparing the length column to the expert data column, we see growth rate that looks more than linear, but still indicates a manageable degree of complexity increase. We would like to note that using more mazes in the training set with fewer expert trajectories for each maze might further increase the sample efficiency of our method and the baselines. Designing an appropriate curriculum for this could be an exciting area of future work.
The scale of the dataset needs to scale up with the complexity of the problem rather than the model depth. Under the same
19

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Table 11: The success rate for each method in 35 35 2D maze navigation, with maze noise (00,0.04).
Shortest Path Length
[1,100]
[100,200]
[200,300]
VIN
59.11
0.00+0.00
0.00
GPPN
87.43
0.15
0.00
Highway VIN DT-VIN (ours)
81.76_4.74
49.5
35. 11..3
95.17 20.00
92.29 10.57
91.99 10.27
3D VIZDoom
100
Success Rate (%)
80
60
DT-VIN (ours)
40
VIN
GPPN
20
Highway VIN
0
100
200
Shortest Path Length
(a) 3D ViZDoom
(b) Performance
Figure (a) an example of ViZDoom 3D maze and the first-person view of the environment with each of the corresponding four orientations. the success rates of the algorithms over various SPLs.
scale of the problem, didn't 't find that increasing model depth requires additional data. As shown in Table 20, without expanding the dataset, increasing the model depth does not reduce the performance.
20

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Table 12: Continuous Control Hyperparameters
Hyperparameter
Value
Transition Mapping Module
A 2-layer CNN with 3 x kernel
A 2-layer CNN (sharing the first layers with Transition Mapping Module)
Reward Mapping Module
Policy Mapping Module
A 3-layer FCN
Latent Transition Kernel Size (F)
Latent Action Space Size (IA)
Optimizer Learning Rate
RMSprop le-3
Batch Size
32
35 x 35 maze: 600
Depth of Planning Module
100 x 100 maze: 5000
Table 13: Training time and success rate (%) across different ranges of SPL.s for DT-VIN with different J values.
Wall-Clock Time (hours)
[1,100]
[100,200]
[200,300]
J =
37
100.00+o.on
99.9940.01
99.78+0.21
= 10
12.1
100.00+0.00
99.9940.00 01
70.233
7.1
100.00_..o0
99.9810.02
99.69 10.27
Table 14: Ablation study for using various estimated lengths of optimal paths for adaptive highway loss, under 35 x 35 ViZDoom navigation. The best results are highlighted.
Shortest Path Length
[1,100]
[100, 200]
[200,300]
? 0 (connected to all hidden layers)
99.49.0.33
94.51 10.77
89.9
[ 1/2
99.62440.9
96.21
91.2441.08
[ =
99.67 00.67 22
97.92+0.11
96. 96.41
Î = 2 * I
99.61
96. 96.29
93.12
[ N (connected to only last layer)
99.52++029
95.52+0.86
91.12+1164
[ = max(e, 0), 1),
99.62 99.62+0.50
06.1990.115
93.21 +0.92
i len (nonneeexx path)
99.62.40.12
97.01 40.69
93.31 20.31
i D (I.1 distance)
99.6440.19
96.92
93.52+0.89
Table 15: The success rates and the changes in success rates for each method, using a dataset reduced to 50% of the original size, are presented. These changes are compared the results from the full-sized dataset, with more negative values indicating worse performance.
Success Rate
Changes [100,200]
Shortest Path Length
[1,100]
[100,200]
[200,300]
[1,100]
[200,300]
VIN GPPN
32.41 32.2114.25 14.25
0.00.1o.0
0.00_1o.00
-36.0013.12
0.00
0.00
83.1 83.11
0.01
0.00
-12.60+1.29
-0.38
0.00+
Highway VIN
45.4114.15
37.41 13.25
21.41 36.98
-45.2613.48
-28.0912.98
-32.994..11
DT-VIN (ours)
99.9640.01
99.8+0.12
96.01 +0.33
00.440..1
-0.19-0.04
-3.76-0.31
21

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Jong-Term Planning
(a) 35 x 35 Maze
(b) 100 x 100 Maze
Figure 12: Some examples of the Point Maze tasks.
Table 16: Computational complexity during training of each method, employing 600 layers and trained over epochs, evaluated in 35 35 2D maze navigation.
Method
GPU Memory (GB)
Wall-Clock Time (h)
GPU Hours (h)
VIN
4.2
8.4
8.4
GPPN
182
4.2
12.6
Highway VIN
41.3
14.3
14.3
DT-VIN
7.2
12.3
12.3
Table 17: Computational complexity during training of each method using 5000 layers and training for 90 epochs, evaluated
100 100 2D maze navigation.
Method
GPU Memory (GB)
Wall-Clock Time (h)
GPU Hours (h)
VIN
36
GPPN
710
310
Highway VIN DT-VIN (ours)
111
1 112
224
61.2
97
97
Table 18: Minimal depths of DT-VIN model across various maze sizes.
Maze Size
Longest Length of Optimal Path
Minimal Depth of DT-VIN
15
80
100
25
200
300
35
300
500
100
1800
5000
22

---

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning
Table 19: Minimal required size of training data for DT-VIN across various maze sizes.
Maze Size
Longest Length of Optimal Path
Required Number of Mazes
Required Amount of Expert Steps
15
80
15K
4M
25
200
15K
24M
35
300
10K
45M
Table 20: The success rate DT-VIN across various model depths N, maintaining the same size as the original dataset.
Shortest Path Length
[1,100]
[100,200]
[200,300]
N 300
99.99+0.01
99.81
92. 11.31
N 600
100.00
99.994..01
99.77
+0.23
N 1200
100.00
99.99
99.81 10.11
23


Reviews:
[{"review_id": "LmDsBNF1Rx", "reviewer": "Reviewer_HD2P", "summary": "This paper tackles the problem of extending value iteration networks (VINs) to handle very long-horizon planning. To achieve this, the authors propose two main modifications:\n\n- A dynamic transition kernel that relaxes the standard weight sharing of convolution layers (i.e., removing strict translation equivariance), and\n- An adaptive highway loss designed to enable the training of much deeper planning modules.\n\nExperimental results are presented primarily on 2D path planning tasks, demonstrating the network’s ability to scale to thousands of planning steps. Overall, the paper is very empirical and focuses on a niche domain related to path planning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer HD2P's valuable time and constructive comments.\n\n**To improve clarity and conciseness, we have reorganized the reviewers’ comments by grouping similar points.**\n\n---\n ### Q1: Experiments are limited to 2D path planning. Evaluating more realistic tasks, integrating visual perception, and discussing broader applications.\n\n### Answer:\n\nWe thank the reviewer for their thoughtful suggestions and for pointing out [r1]; we will cite and discuss it in the final version. However, we believe there may be a misunderstanding—our submission already includes tasks aligned with [r1] and extends them in several ways:\n\n- **3D Visual Maze Navigation** (Sec. 4.1, Additional Experiments, Appendix C.3): Our setup is identical to that of [r1], with both following established protocols from prior work [r2]. As the reviewer noted, this involves “visual navigation with perception input using additional network heads.” Agents must infer maze layouts from noisy, ambiguous first-person visual views, making the task especially challenging.\n\n- **Continuous Control Tasks** (Sec. 4.2): These involve long-horizon planning with low-level torque control to actuate movement of the agent. While [r1] uses 2-DOF manipulation tasks, we evaluate on a 2-DOF ball and an 8-DOF quadruped robot—sharing the same fundamental task but introducing greater control complexity due to higher dimensionality.\n\n- **Rover Navigation** (Sec. 4.3): This more realistic task, inspired by the original VIN paper but absent in [r1], involves planning over noisy, incomplete aerial terrain images—adding complexity beyond synthetic 2D gridworlds.\n\n- **Scale**: Our benchmarks feature much larger mazes (up to 100×100 vs. up to 49×49 in [r1]), posing harder long-horizon challenges.\n\nThese tasks—spanning visual navigation, control, and realistic planning—are already described throughout the original submission (Abstract, Introduction, Experiments). Nonetheless, we will revise the final version and cite [r1] to more clearly highlight their generality and realism.\n\nExtending the VIN architecture to broader applications is a compelling direction for future work—previously barred by the limited planning capacity of VIN-based architectures. However, this work aims to focus on the first step in this of demonstrating the performance gains of DT-VIN over related methods, which we view as a necessary precursor to deployment in more applied domains. Indeed, while we are looking into more applied domains currently, such an investigation is of the scale that would necessitate a separate paper.\n\n[r1] Zhao et al., Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation, ICLR 2023\n\n[r2] Lee et al., Gated Path Planning Networks, ICML 2018\n\n---\n ### Q2: Concern over the impact of relaxed dynamic kernel weight sharing on translation equivariance and performance.\n\n### Answer:\n\nThank you for the thoughtful and constructive feedback.\n\nWhile our dynamic transition kernel relaxes the weight sharing, it preserves translation equivariance—i.e., shifting the input (e.g., maze image) results in a corresponding shift in the output. This may seem counterintuitive but is straightforward upon closer inspection. This holds because the kernel $\\overline{\\mathsf{T}}^{\\rm dyn} = f^{\\overline{\\mathsf{T}}}(x)$ is generated by a CNN, which is itself translation equivariant. Consequently, the downstream value iteration process retains this property (see Eq. 2).\n\nTo verify this, we conducted a small experiment comparing DT-VIN, a standard CNN, and VIN on 5,000 maze images with translations of 1 to 10 pixels. All models use stride and pooling of (1,1). We measured the L2 error between the output on a translated input and the translated output of the original. Results in Table R1 show perfect translation equivariance across all models.\n\nAdditionally, ablation studies in the original submission (Figs. 6a and 6b) compare dynamic vs. invariant kernels. Dynamic kernels consistently outperform their invariant versions, particularly in long-horizon and obstacle-rich settings—highlighting both practical advantages and robustness.\n\nTable R1: Translation Equivariance — L2 Error\n\n|Model|Avg. L2 Error|\n|---|---|\n|CNN|0.0|\n|VIN|0.0|\n|DT-VIN (ours)|0.0|\n\n---\n ### Q3: Benefits and limitations of adaptive highway loss.\n\n### Answer:\n\nThank you for the suggestion. The adaptive highway loss avoids the vanishing gradient problem in very deep networks by adding skip connections from intermediate layers to the final loss, guided by the planning trajectory length (computed from training data). This improves gradient flow and stabilizes training (Sec. 4.4, adaptive highway loss).\n\nA potential limitation is computational overhead, which we mitigate by applying the loss every $J$ layers (Eq. 3). This balances efficiency and performance; e.g., $J=10$ significantly accelerating training with minimal performance loss (Appendix F.1).\n\nWe will clarify this in the revised manuscript.", "claims_and_evidence": "- Claims:\n    - The proposed modifications enable VINs to plan over very long horizons (up to thousands of steps) by increasing network capacity and facilitating deep credit assignment.\n    - Relaxing the invariant kernel (i.e., weight sharing) boosts performance on these tasks.\n- Evidence:\n    - Empirical results support improved planning in extended 2D path planning scenarios.\n    - However, the paper does not convincingly analyze the impact of removing translation equivariance (i.e., the dynamic kernel is simply a relaxation of the weight-sharing property) and lacks detailed insights or ablations on this point.", "methods_and_evaluation_criteria": "- Methods:\n    - The paper proposes modifications to the standard VIN architecture by replacing invariant (shared-weight) convolutions with dynamic kernels and enforcing an adaptive highway loss to help train very deep networks.\n    - While the adaptive loss is an interesting idea, the paper provides limited discussion on its theoretical justification.\n- Evaluation:\n    - Experiments are conducted on standard 2D path planning benchmarks, which reflect a narrow application domain, although showing extended horizon capability. I would suggest to follow the mentioned work below for more tasks, e.g., for visual navigation with perception input using additional network heads for perception networks.\n    - The evaluation criteria remain focused on path planning success rates; broader or more realistic planning tasks would strengthen the work.\n\nScaling up and Stabilizing Differentiable Planning with Implicit Differentiation. ICLR 2023.", "theoretical_claims": "paper’s contributions are mainly empirical, and no detailed theoretical proofs were provided regarding the benefits of the dynamic kernel or the adaptive loss.", "experimental_designs_or_analyses": "- Design:\n    - The experimental setup largely mirrors that of the original VIN paper (from eight years ago) but with extended planning horizons.\n- Analysis:\n    - While the experiments demonstrate that the approach scales to very long horizons, they remain confined to a niche 2D planning domain.\n    - The paper would benefit from further analysis or ablation studies that clarify how the removal of weight sharing affects network performance and equivariance properties.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "This work also studies scaling up the training of value iteration networks, which is potentially related. It also has some more interesting domains to study.\n\n- Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation. ICLR 2023.", "other_strengths_and_weaknesses": "- Strengths:\n    - Tackles an interesting and challenging problem of extending VINs to very long-horizon planning.\n    - Introduces conceptually appealing modifications (dynamic kernel and adaptive highway loss) that are intuitively motivated.\n- Weaknesses:\n    - The approach is limited to a niche domain (2D path planning) and may not generalize well to more realistic problems.\n    - The relaxation of the invariant (weight-sharing) property is not thoroughly analyzed; the paper lacks convincing insights on how this change impacts translation equivariance.\n    - Experimental validation remains confined to benchmarks from older literature, rather than exploring potential applications unlocked by long-horizon planning capabilities (e.g., integration with latent perception modules).", "other_comments_or_suggestions": "- Consider broadening the experimental evaluation to include more realistic planning scenarios.\n- Provide additional ablation studies or analysis that clarifies the impact of using a dynamic (non-shared) kernel on equivariance properties.\n- Elaborate further on the adaptive highway loss to help the reader understand its benefits and limitations.\n- Discuss potential applications of long-horizon planning beyond pure path planning to motivate the broader relevance of the approach.", "questions_for_authors": "1. Dynamic Kernel Analysis:\n    \n    Could you provide further analysis or ablation studies on how the dynamic transition kernel (i.e., relaxing weight sharing) affects the network’s translation equivariance and overall performance?\n    \n2. Generalization Beyond 2D Planning:\n    \n    Have you considered applying your approach to planning problems outside of 2D path planning? What challenges do you foresee, and how might your method generalize to more realistic or complex environments?\n    \n3. Broader Implications:\n    \n    Can you elaborate on what new capabilities or applications might be enabled by scaling VINs to such long horizons? For instance, could this be integrated with perception modules to tackle more challenging planning tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LQGEpbOYzk", "reviewer": "Reviewer_TBA2", "summary": "Value Iteration Networks (VIN) struggle to scale with large scale planning problems, typically in problems involving a higher number of steps to reach to goal. The paper provides two observations that explain the poor performance (1) low representation capacity (2) lack of depth in VINs - due to vanishing gradient issue for higher depths. The authors propose to use dynamic transition kernels to improve representational capacity (solve (1)) and propose ‘adaptive highway loss’ to solve issues with vanishing gradients allowing higher depths. Their method, named DT-VIN, shows state-of-the-art performance with respect to other VIN frameworks, especially for large scale planning problems.\n\n\n## Update after Rebuttal\nThanks for a detailed response. That is, I still think the paper lacks clarity in proposed method section (aligns closely with reviewer vY56). I do not agree with authors' final comment on \"minor\" writing issues. Although I understand minor writing issues are fine - however if they obstruct understanding of the paper (and thus critical review of the contributions) then in my opinion those issues are not \"minor\".\nGiven my concern I will be retaining my original score as of now.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer TBA2 for the valuable comments.\n\n---\n ### Concern 1: Clarification on state/observation-dynamic in Sec 3.1.\n\n### Answer:\n\nTo clarify: the *observation* refers to the maze image input, which is mapped by the model to a latent MDP. Each *latent state* corresponds to a specific node within this latent MDP.\n\nWe distinguish two types of dynamics:\n\n- **Latent state-dynamic**: As the reviewer correctly noted, in this setup each latent state $(i,j)$ has its own transition kernel $\\overline{\\mathsf{T}}_{i,j} \\in \\mathbb{R}^{|\\overline{\\mathcal{A}}| \\times F \\times F}$.\n\n- **Observation-dynamic**: The transition kernel is generated from the input observation $x$ via a learned mapping function: $\\overline{\\mathsf{T}} = f^{\\overline{\\mathsf{T}}}(x)$.\n\nThese two notions are orthogonal, defining independent aspects of variation in the transition kernel. They give rise to four configurations: latent state-dynamic/invariant and observation-dynamic/invariant (see Table R1). Our ablation study (Sec. 4.4) shows that both components are essential—removing either significantly degrades DT-VIN performance.\n\nWe will clarify this distinction in the final version.\n\n\n\nTable R1: Four configurations of latent transition kernels based on whether they are dynamic with respect to latent state and/or observation.\n\n| Type of latent tranisiton kernels | latent state-dynamic? | observation-dynamic? | represenation of latent tranisiton kernels                   |\n|---|---|---|---|\n| **fully invariant**               | $\\times$              | $\\times$             | parameter $\\overline{\\mathsf{T}} \\in \\mathbb{R}^{\\lvert \\overline{\\mathcal{A}}\\rvert\\times F \\times F} $ |\n| **latent state-dynamic only**     | $\\checkmark$          | $\\times$             | parameter $\\overline{\\mathsf{T}} \\in \\mathbb{R}^{M \\times M \\times \\lvert \\overline{\\mathcal{A}}\\rvert \\times F \\times F} $ |\n| **observation-dynamic only**      | $\\times$              | $\\checkmark$         | output of the mapping function, $\\overline{\\mathsf{T}} = f^{\\overline{\\mathsf{T}} }(x) \\in \\mathbb{R}^{ \\lvert \\overline{ \\mathcal{A} } \\rvert \\times F \\times F} $ |\n| **fully dynamic**                 | $\\checkmark$          | $\\checkmark$         | output of the mapping function, $\\overline{\\mathsf{T}} = f^{\\overline{\\mathsf{T}} }(x) \\in \\mathbb{R}^{ M \\times M \\times \\lvert \\overline{ \\mathcal{A} } \\rvert \\times F \\times F} $ |\n\n---\n ### Q1:  How is suboptimality measured when the success rate is less than 100%?\n\n### Answer:\n\nAs defined in the paper (Line 220), suboptimality is measured relative to the best solution across all models, including the expert. This is applicable in the 2D Maze Navigation task, where the expert solution can be computed using Dijkstra’s algorithm with access to an underlying binary representation of the maze. For tasks without expert access (e.g., continuous control), only the success rate is reported.\n\n---\n ### Q2: Why does highway VIN get a better optimality rate with increasing shortest path lengths?\n\n### Answer:\n\nThank you for the thoughtful question. One possible explanation is that longer planning tasks offer a more structured search space, allowing Highway VIN to better propagate informative signals over multi-steps. In contrast, shorter mazes may contain more local optima or distractors, leading to suboptimal choices despite a shorter horizon. As Highway VIN is not our focus, we leave deeper analysis to future work.\n\n---\n ### Q3: line 260-261 - we examine depths of $N = 600, 5000$ what do you mean? Do you mean in range $[600, 5000]$?\n### Answer:\nThank you for pointing this out. We meant that we examine depths at two specific values, $N = 600$ and $N = 5000$, rather than a continuous range. We will clarify this in the final version.\n\n---\n ### Q4: Why is pretraining required in Sec 4.2?\n### Answer:\n\nPretraining is necessary because the training dataset for the continuous control task includes only a single $9\\times12$ maze layout, differing solely in start and goal positions. However, during evaluation, the model encounters significantly larger ($35\\times35$ and $100\\times100$) and previously unseen maze layouts, substantially increasing task complexity and requiring strong generalization abilities.\n\nTo address this challenge, we leverage pretraining on the diverse and easily accessible 2D maze navigation dataset. This approach helps the model acquire generalizable planning skills, resulting in strong performance with success rates of 98% on the Point Maze and 93% on the Ant Maze tasks (refer to Table 1 in Sec. 4.2 for further details).\n\nWhile it is feasibly possible for it to solve the task without pretraining, the sample complexity would be high—a property shared with VIN (and VIN variants) here.\n\n---\n ### Suggestion 1: Minor comments on figure readability and typos.\n\n### Answer:\nThank you for bringing this to our attention. We will correct this for the camera-ready version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, to my knowledge.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments sound solid for most part. See questions section for questions.", "relation_to_broader_scientific_literature": "They advance a widely used framework for large scale planning problems. I think the contribution is sufficient.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n- The paper provides rigorous experiments and strong ablation studies providing insightful conclusions.\n- The results are impressive on large scale planning problems. \n- The paper is well structured\n\nWeakness\n- Section 3.1 is not clear to me. Specifically, justification for 'state’-dynamic and 'observation’ dynamic kernels is not clear to me. What is the difference between them in this context? How does a kernel of |A| \\times F \\times F make it `state’/’observation’ dependent? In this case, should we use |A| \\times F \\times F parameters for each state?", "other_comments_or_suggestions": "Minor\n- Text in images in too small (e.g., Fig. 1 Fig. 2)\n- Line 244 right ('Additional’ experiments). Due to space constraints we `report’ (or evaluate and report) Additional is also not spelled correctly.", "questions_for_authors": "1. How is suboptimality measured when success rate is less than 100%? \n2. Why does highway VIN get a better optimality rate with increasing shortest path lengths?\n3. line 260-261 - we examine depths of N = 600, 5000 what do you mean? Do you mean in range [600, 5000]?\n4. Why is pretraining required in section 4.2?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XG4NxfN7Nw", "reviewer": "Reviewer_vY56", "summary": "In this paper, the authors investigated to extend Value Iteration Network (VIN) to address longer-term and larger-scaled planning tasks. It is not feasible just by applying VIN, and they found the reasons from invariant transition in the network and inefficient loss design for long-term planning tasks. To address them, the authors proposed a new module, Dynamic Transition kernel which transits under the consideration of observation knowledge, and adaptive highway loss to give the learning signals to the each layer of network when the planning horizon is over the oracle length. It not just leads the network can find the shorten path but also improves the information flow. They called it as Dynamic Transition VINs (DT-VINs) and they showed this planning modules can be scaled up to 5000 layers empirically. They empirically evaluated their model on large-scaled discrete/continuous action conditioned maze environments and Rover Navigation environment. In the evaluations, their model outperformed baselines. They also studied ablations when different kernels and losses are applied and the performance change through different number of layers applications.\n\n## update after rebuttal\n\nThe authors properly addressed our concerns, we keep our score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful and helpful comments from Reviewer vY56. \nPlease find our responses to your concerns and questions below.\n\n---\n ### Suggestion 1: The tasks are related to diffusion models for long-term planning.\n\n### Answer:\n\nThank you for noting the connection to diffusion-based planning. As references [1,2] were missing, we reviewed related works [r3, r4]. Though differing in approach—trajectory generation vs. step-wise planning—both address long-horizon dependencies in complex tasks. We will revise the manuscript to reflect this connection and consider future integration.\n\nAdditionally, if the reviewer can provide [1,2], we will be happy to cite and discuss them in our final version.\n\n[r3] Janner M. et al. Planning with diffusion for flexible behavior synthesis. ICML 2022.\n\n[r4] Mishra U.A. et al. Generative skill chaining: Long-horizon skill planning with diffusion models. CORL 2023.\n\n---\n ### Weakness 1: More discussion on methodology.\n\n### Answer:\n\nWe agree that the methodology section could be clearer and will revise it for the final version. In particular, we agree with your proposal to lean more on illustrative figures in the paper. Currently, Figure 6(e) visualizes the learned dynamic latent transition kernel of DT-VIN across several states. We plan to integrate this into Figure 2 and include a side-by-side comparison with an invariant transition kernel to better highlight the advantages of the dynamic approach.\n\n---\n ### Weakness 2: More discussion on trade-offs between performance gains and the additional computational costs introduced by using deeper networks.\n\n### Answer: \n\nWe already partially address this in Appendix G, where we found that:\n\n- At the same depth, DT-VIN achieves better performance with a similar or lower GPU memory and training time (Appendix G.1, Tables 14–15).\n\n- DT-VIN’s depth scales almost linearly with planning steps, ensuring manageable complexity as tasks grow harder (Appendix G.2).\n\nTo more specifically address your concern, we have produced Tables R1 and R2 (shown below), which explicitly analyze the trade-off between computational cost and performance across different network depths $N$.\nWhile deeper networks naturally incur higher computational costs, DT-VIN demonstrates substantial performance gains as depth increases. Specifically, it achieves +81.23% and +99.77% success rate improvements at $N=300$ and $N=600$ over $N=100$, respectively. In contrast, VIN, GPPN, and Highway VIN show little to no improvement despite incurring similar or even higher computational costs.\n\nWe will incorporate these new tables into the revised manuscript to clarify the performance–efficiency trade-offs of DT-VIN.\n\nTable R1: *GPU memory (GB)* and *GPU hours (h)* during training,  on $35\\times 35$ maze tasks for varying network depths $N$.\n\nGPU memory (GB):\n\n||N=100|N=300|N=600|\n|---|---|---|---|\n| VIN |1.3 |2.7|4.2|\n| GPPN |42.1|135.2|182|\n| Highway VIN|15.2|31.5|41.3|\n| DT-VIN (ours)|18.8|35.8|53.3|\n\nGPU hours (h):\n\n||N=100|N=300|N=600|\n|---|---|---|---|\n|VIN |2.8|5.2|8.4|\n|GPPN |4.2|9.1|12.6|\n|Highway VIN |4.9|9.8|14.3|\n|DT-VIN (ours) |4.8|8.7|12.1|\n\nTable R2: Success rates of different models at varying depths on $35\\times 35$ mazes with shortest paths in [200, 300]. See Appendix Fig. 9 for plots.\n\n||N=100|N=300|N=600|\n|---|---|---|---|\n| VIN | $0.0_{\\pm 0.0}$   | $0.0_{\\pm 0.0}$ | $0.0_{\\pm 0.0}$|\n| GPPN| $0.0_{\\pm 0.0}$   | $0.0_{\\pm 0.0}$| $0.0_{\\pm 0.0}$|\n| Highway VIN | $0.0_{\\pm 0.0}$| $34.23_{\\pm 11.27}$| $54.41_{\\pm 10.2}$|\n| DT-VIN (ours)| $0.0_{\\pm 0.0}$ | $\\mathbf{81.23_{\\pm 1.34}}$ | $\\mathbf{99.77_{\\pm 0.23}}$ |\n\n---\n ### Q1:  Whether planning in continuous action mazes is essentially the same as in discrete ones. May the main difference is the controller.\n\n### Answer:\n\nYou're generally right—the high-level planning in continuous control tasks is conceptually similar to that in discrete mazes (all planning in reinforcement learning is essentially about producing a sequence of actions—discrete or continuous—that will lead to a desired state or sequence of states). \n\nHowever, DT-VIN is not simply a stack of a high-level planner and a low-level controller. Instead, it is trained end-to-end to map observations directly to actions, learning high-level planning implicitly from expert demonstrations without requiring high-level planning supervision. This simplifies training and improves robustness.\n\nFor example, in D4RL tasks, we supervise using expert control actions (e.g., from a 2-DOF ball or a 8-DOF quadruped robot). While D4RL uses the same $9\\times 12$ maze layout for all training samples (varying only the start and goal positions), we evaluate on much larger, unseen mazes ($35\\times35$, $100\\times100$), making the task significantly more challenging.", "claims_and_evidence": "This paper claims that by augmenting the standard VIN architecture with the dynamic transition kernel and an adaptive highway loss, one can overcome the limitations in long-term planning present in conventional VINs. DT-VINs achieve much higher success and optimal rates in tasks where traditional methods fail, and the ablation studies clearly show the contribution of each component. While the improvements are well supported, a deeper discussion on the trade-offs between performance gains and increased computational complexity would further strengthen the evidence.", "methods_and_evaluation_criteria": "The methodology is well motivated. The use of a dynamic transition kernel addresses the representational bottleneck in traditional VINs, and the adaptive highway loss provides an elegant solution to train extremely deep networks. The evaluation criteria—primarily success rate and optimality rate across varied maze sizes—are appropriate for assessing long-term planning performance. The extensive experiments on multiple benchmarks and controlled ablation studies make a strong case for the proposed modifications.", "theoretical_claims": "The paper references theoretical work linking network depth with value estimation accuracy (e.g., via Theorem 1.12 from prior studies) but does not provide new proofs.", "experimental_designs_or_analyses": "The experimental design is reasonable and their experiments are comprehensive. The authors evaluated DT-VINs across a variety of settings—from small-scale discrete action spaced mazes to complex continuous control and real-world rover navigation tasks. The soundness of the experimental analyses is supported by thorough ablation studies, comparisons with multiple baselines, and assessments under different noise conditions.", "relation_to_broader_scientific_literature": "In the perspective of the targeting tasks, long-term planning, it can be related to the recent works applying the Diffusion model to the long-term planning tasks [1,2]. Their methods approach to generate the long-term planning through the powerful generative modeling with the strengths of the diffusion model, holistically generating data while DT-VIN can make it possible through very deep VIN model learning.", "essential_references_not_discussed": "The essential references are properly addressed in this paper.", "other_strengths_and_weaknesses": "Strengths:\n- A good motivation and well-designed methodology: One of the weaknesses of VIN for long-term planning is well addressed with properly designed methodology.\n- Extensive empirical results: They empirically evaluated their methodology with several environments and studied diverse ablations.\n\nWeaknesses:\n- Too compressed discussion for their methodology: Their discussion for proposed methodology is relatively short, and it was hard to understand at first glance. More detailed figures and explanations are given, it would be better. For instance, why the dynamic transition kernel is required is shown through figure that shows the strength of it by comparing the region with walls and empty areas are encoded invariantly and observation conditionally.\n- Computational overhead discussion: Deeper network requires more computational overhead, but it is not discussed in their paper.", "other_comments_or_suggestions": "We do not have any other comments.", "questions_for_authors": "- For continuous action maze environments, you applied the pre-trained VIN model for planning. Then, in the aspect of the planning, can we think the task is similar to discrete action spaced maze? May the difference is the controller I guess.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a significant advancement in the field of reinforcement learning by extending the capabilities of Value Iteration Networks (VINs) to handle extremely long-term planning tasks. The authors introduce a novel architecture, Dynamic Transition VIN (DT-VIN), which incorporates a dynamic transition kernel and an adaptive highway loss to address the limitations of traditional VINs in terms of representation capacity and depth. The proposed method demonstrates impressive performance across a variety of challenging tasks, including 2D/3D maze navigation, continuous control, and applications like the Lunar rover navigation.

+ The introduction of a dynamic transition kernel and adaptive highway loss is a well-motivated and innovative solution to the challenges faced by traditional VINs.
+ The paper provides extensive empirical evidence supporting the effectiveness of DT-VIN, with experiments conducted on a diverse set of tasks and environments, and ablations supporting each component.
- The paper could benefit from a more detailed discussion of the trade-offs between performance gains and the increased computational complexity associated with deeper networks. (Partially addressed in rebuttal.)
+ Clear paper

Overall, I recommend this submission for acceptance at ICML 2025. The paper presents significant contributions to improving VINs for long-term planning tasks through innovative architectural modifications. While there are some concerns regarding clarity and experimental breadth, the authors have adequately addressed these in their rebuttal.
Decision: Accept (poster)