Information-based Value Iteration Networks for Decision Making Under Uncertainty
Authors: Cynthia Chen
Samantha Johnson
Cindy Poo
Michael A Buice
Koosha Khalvati
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
value iteration networks
planning under uncertainty
Primary area: reinforcement learning
TL;DR: We proposed a novel deep architecture for decision making under uncertainty based on planning for reward maximization and information gathering.

Abstract: Deep neural networks that incorporate classic reinforcement learning methods, such as value iteration, into their structure significantly outperform randomly structured networks in learning and generalization. These networks, however, are mostly limited to environments with no or very low uncertainty and do not extend well to partially observable environments. In this paper, we propose a new planning module architecture, the VI$^2$N (Value Iteration with Value of Information Network), that learns to act in novel environments with high perceptual ambiguity. This architecture over-emphasizes reducing uncertainty before exploiting the reward. VI$^2$N can also utilize factorization in environments with mixed observability to decrease the computational complexity of calculating the policy and to facilitate learning. Tested on a range of grid-based navigation tasks, each containing various types of environments with different degrees of observability, our network outperforms other deep architectures. Moreover, VI$^2$N generates interpretable cognitive maps highlighting both rewarding and informative locations. These maps highlight the key states the agent must visit to achieve its goal.

Published as a conference paper at ICLR 2026
INFORMATION-BASED VALUE ITERATION NETWORKS FOR DECISION MAKING UNDER UNCERTAINTY
Cynthia Chen Allen Institute Seattle, WA 98109
Samantha N. Johnson University of Chicago Chicago, II. 60637
Cindy Poo Allen Institute Seattle, WA 98109
Michael A. Buice Allen Institute Seattle, WA 98109
Koosha Khalvati ? Allen Institute Seattle, WA 98109
ABSTRACT
Deep neural networks that incorporate classic reinforcement learning methods, such as value iteration, into their structure significantly outperform randomly struc- tured networks learning and generalization. These networks, however, are mostly limited to environments with no or very low uncertainty and do not extend well to partially observable environments. module architecture, the VI (Value Iteration with Value of Information Network), that learns to act in novel environments with high perceptual ambiguity. This archi- tecture over-emphasizes reducing uncertainty before exploiting the reward. VIEN can also utilize factorization in environments with mixed observability to decrease the computational complexity of calculating the policy and to facilitate learning. Tested on a range grid grid-based navigation tasks, each containing various types of environments with different degrees of observability, our network outperforms other deep architectures. Moreover, VI generates interpretable cognitive maps highlighting both rewarding and informative locations. These maps highlight the key states the agent must visit to achieve its goal.
this paper, we propose new planning
INTRODUCTION
Deep neural networks have provided powerful end-to-end solutions Reinforcement Learning (RL) problems that map perception to action (François-L Lavet et al. 2018). One can approach this end-to-end learning in classic supervised fashion, especially when provided an expert policy to imitate. However, several studies have shown that incorporating cognitive/classie RL mechanisms, such simulation of future events and experience replay, improve the learning process significantly. For example, Value Iteration Networks (VINs) incorporate long-term planning (the simulation of future events) by implementing the value iteration algorithm (i.e., sequence of Bellman updates) via convolutional layers (Bellman 1957 Tamar et al. 2016 Niu et al. 2018 Zhang et al. 2020: Ishida Henriques] 2022). Trained either by reward or through imitation an expert's 's actions, VINs can learn to navigate in fully observable novel environments significantly better than fully connected and untied convolutional networks (Tamar et al. 2016). Furthermore, their generated model of the environment correctly identifies the rewarding areas (e.g., the goal state).
While VINs and deep reinforcement learning architectures in general have been very successful in many applications, they face tremendous challenge in many real-world scenarios due to perceptual ambiguity (Ni et al., 2022). Perceptual ambiguity. often called partial observability, introduces uncertainty about the current state of the environment. Therefore, the agent must form a probability distribution, or "belief", over its current state and choose its action based on this belief. Even simple networks with a probabilistic belief representation outperform networks with more sophisticated
"Co-first author; Work conducted during an internship at the Allen Institute. I FEmails:(cmtthia.hhe cindy.poo, michaelb, koosha khalvati @alleninstitute.org snjoonso snjjonns@@uchicago..c.t

---

Published as a conference paper at ICLR 2026
encoding and RL modules that perform well in challenging obsly-bbee environments (Ni et al. 2022, Karkus et al.] 2017). However, the main challenge in uncertain environments is that action selection is based the current belief, not the belief representation. Therefore, more advanced policy/planning modules are required to perform well in more complex uncertain environments.
Formally defined by the Partially Observable Markov Decision Process (POMDP) framework (Thrun ett al.] 2005), optimal decision making under uncertainty is not achievable in polynomial time (Sondikk
[Sondik][978)) 1978). Additionally, powerful sub-optimal approximations involve sampling and tree search techniques with no differentiable implementation, hindering their use in neural network implementations (Ross al. 2008). Because of these limitations, the current state-of-the-a value iteration network for decision making under partial observability is founded upon a very simple POMDD-solver, QMDP, which assumes that all uncertainty disappears after the first step (Karkus) et al. 2017). While this allows for differential heuristic, this assumption causes the solver to fail in highly uncertain environments.
This paper proposes new network architecture, the V (Value Iteration with Value of Information Network), that can learn to plan in unseen environments with high uncertainty. VI is based on the Pairwise Heuristic (Khalvati & Mackworth 2013), which calculates the solution of sub-problems where the belief state is limited to only a pair of states. Since the Pairwise Heuristic can be calculated by the Bellman equation, it can be implemented with neural network similar to the VIN. Moreover, VIeN can utilize factorization in mixed observable environments, formally modeled with Mixed Observable Markov Decision Processes (MOMDPs) (Ong et al. 2010 Araya--López al.) 2010). This factorization leads to a huge reduction in computational and memory complexity of the network. We demonstrate the power of our approach by testing it on navigation problems in the presence of different types of uncertainty in various environments. In addition to superior performance in challenging environments with high degree of ambiguity, VI generates interpretable maps highlighting key states with high reward and information.
2
BACKGROUND
2.1 MARKOV DECISION PROCESS (MDP)
Sequential decision making is usually expressed as a Markov Decision Process (MDP) (Thrun et al. 2005). Formally, an MDP is (8, T, R, 7) where S is the set of states in the environment, A is the set of available actions to the agent, the transition function T |S| x |A| x |S| + [0, 1] defines T(s, a, s') = P(s'|s, |s. a), the probability of ending up in state by performing action a in state s, R: [S| A| + R is bounded function determining the reward gained in state s, shown as R(s), and 7 E (0, 1] is the discount factor for the reward (Thrun et al. 2005). Starting from an initial state, 8g, the goal of the agent is to come up with policy for action selection, that maximizes the total discounted reward. Since the system is Markovian, the policy can be expressed as a mapping from states to actions, i.e. क : |S| x |A| + [0, 1]. The optimal policy π* is x² arg max, Et 7 "EE [[+))(()) |TTT so] where the horizon H defines the length of this sequence. In classic Reinforcement learning, the value iteration algorithm computes the optimal policy through series of Bellman updates (Bellman 1957). i.e. (s(s) max, Qe a) where Qi(s. a) = [[(s, a) + 7 Ev T(s,, a, V
(t S H).
In deep reinforcement learning. this optimal policy/mapping is learned with a network with the state S (or representation of it, ((s)) as the input and the action as the output of the network
( (François-Lane Lavet et al. 2018).
2.2 VALUE ITERATION NETWORK (VIN)
When the transition function is spatially invariant, neural network can learn T and R by imple- menting the Bellman equation with convolutional layers Tamar et al., 2016)). More specifically, given the map of the environment and the current state of the agent (e.g., its position on the map) as the inputs and an expert's action or reward as the output, VIN learns convolutional kernels of fR and fP representing reward and transition functions. Such a network with the integration of value iteration as an explicit planning module, generally known as a Value Iteration Network (VINs), significantly outperforms networks with similar computational power (e.g.. layers) in learning to plan in unseen environments ( (Tamar et al. 2016). Originally built for simple lattice worlds with spatially invariant transition functions, value iteration networks have been significantly improved
2

---

Published as a conference paper at ICLR 2026
in terms of applicability to domains with more complex structures over the past years. All of these improvements. however, are still mainly limited to fully observable environments (Niu et al. 2018
Zhang et al. 2020; Ishida & Henriques 2022).
2.3
PARTIALLY OBSERVABLE MARKOV DECISION PROCESS (POMDP)
Existence of uncertainty in the real world, especially in the form of perceptual ambiguity, has made MDPs impractical in many situations (Thrun et al. 2005). Similarly, state-of-the-arr networks reaching extraordinary performance in very complicated yet fully observable tasks often fail to handle seemingly small amounts of ambiguity in the environment (Ni et al.) (022). Partially Observable MDPs (POMDPs) represent the closest approach to MDPs that deals with the uncertainty by adding an observation set and observation function to its framework. Formally, a POMDP is a tuple (S, A, Z, T, O, 7) where S, A, T, R, and 7 are defined very similar to their definition in MDP. 2 is the set of observations and O: [S] x [Z| + [0, 1] is the observation function determining probability of observation 2 in state s. i... ((s, 2) = P(z]s). In POMDP, the agent is not fully aware of its current state. Therefore, it has to maintain a probability distribution over states, often called its belief b(s). Starting from prior probability distribution over states of the environment, called the initial belief (bo), the goal is to maximize the expected discounted reward (Ross et al. 2008). For a POMDP, the optimal decision policy n² can be expressed as mapping from belief states (probability distributions over states) to distribution of actions that maximizes the total expected reward (Sondik) 7 E
1978), i.e. Λ' = arg max, at
x]. The uncertainty about the state makes the agent navigate the belief state space instead of the state space. At time step t. the belief state by is updated based on the previous belief state bt1 after action a, and observation 2, as follows: (s)) x P(ze|s, an-1) E) P(s]s', -;-1)b(-1(s) Partial observability also makes the problem of finding the optimal policy exponentially more complex than the MDP. While the optimal policy of an MDP can found in polynomial time, finding the optimal policy of a POMDP is NP-hard (Thrun et al.) 2005). As result, the optimal policy can only be approximated by methods such as heuristies, sampling, and search trees (Ross et al. 2008 Khalvati & Mackworth, 2013).
R((s at #t+1: St-1 1b,,
2.4 MIXED OBSERVABLE MARKOV DECISION PROCESS (MOMDP)
Although most environments in the natural world incorporate elements of uncertainty, not all aspects of one's current state are unknown. By factorizing the state space into different dimensions, one can separate the visible factors, which contain full observability, from the hidden factors, which are uncertain. An environment with a factorized state space can be referred to as a Mixed Observable Markov Decision Process (MOMDP) (Ong et al. 2010; Araya-López MOMDPs still involve the computation of a belief probability distribution over states, the number of states this belief distribution needs to be computed over can be reduced significantly by only computing solutions for belief distributions on the hidden, or uncertain, factors. Notably, there is not much difference in MOMDP notations other than factorizing the state space into visible/fully observable (S,) and hidden/pe hidden/partially observable (S,), where S = Sn, x S.. Transition, reward, and observation functions could still depend on the whole state. However, in many cases, the factorization could be applied to these functions as well. Notably, not all POMDP solvers can benefit from factorization, but the ones that do can compute equally accurate solutions with less computations and memory.
al. 2010). Although
2.5 DEEP NETWORKS FOR SOLVING POMDPS AND MOMDPS
Since POMDPs and MOMDPs have an additional observation function compared to MDPs, deep architectures for decision making under partial observability represent observation kernel fz in addition to transition function kernel fr and reward kernel fn- Notably, the structure of these kernels depend on the problem, but the general methodology and algorithm of learning is the same. For example, in a navigation problem, the agent might observe only its surrounding instead of the whole world, which makes the observation kernel 3 x 3 convolutional kernel. In different domain, observation kernel might consist of only a 1 x 1 convolution over the agent's 's current position. Similar to classic POMDP solvers, POMDP solver network consists of two modules of belief update and policy, forming a recurrent architecture together. The belief update can be casily implemented in a network, as exemplified in the QMDP-Net architecture (Karkus al. 2017) (Also
3

---

Published as a conference paper at ICLR 2026
demonstrated in figure A-1). However, designing powerful policy module is very challenging due to the differentiability requirement. As a result, current networks for solving POMDPs have a very simple policy module, c.g., model-free frre RL module (Ni et al. 2022) and QMDP. MOMDP solver network has the same structure, but with a smaller-sized belief state space covering only the unobservable factor of the state space. Like classic POMDP solvers, how the policy module takes
advantage of mixed observability is significantly more important than the belief update module.
MODEL
Our main goal is to provide a better policy module for the value iteration networks in partially observable environments. Our architecture is founded upon "Pairwise Heuristic". Originating from Bayesian active learning in which the heuristic is used to find the correct hypothesis with set of noisy tests (Golovin et al.] (2110), the Pairwise Hcuristic has also been used in general-purpose POMDP-solver when the environment model is fully known (Khalvati & Mackworth, (013)) Here we present this version, slightly modified for our framework.
3.1 THE PAIRWISE HEURISTIC FOR SOLVING POMDPS
The main idea of the pairwise heuristic to use solutions of the smallest sub-problems that still consider the uncertainty about the true hypothesis/state, which would be pairs (sets of 2) of hy- potheses/states (Golovin et al. 2010). In a POMDP, this would be the set of [S(( - 1)/2 optimal policies in each of which the belief is .5 for two states. The expected total reward of each of these policies the value of the pair, shown by V(8, 8'), for 8, E S. This is differentiated from V(s), which corresponds to the value of a state based on the optimal policy computed through Value Iteration on fully observable representation of the state space. Calculating pairwise optimal policies is still computationally very expensive. Therefore, the pairwise heuristic for POMDPs applies an additional heuristic to calculate (ss, s') (Khalvati & Mackworth 2013). For each pair, it resolves the uncertainty first and then exploits the reward. Given the observation function, the uncertainty is already resolved for some pairs of states. To be more precise, it is highly unlikely to have a notable probability/belieff for states with different observations. These pairs are "distinguishable.. For other pairs of states, i.c. indistinguishable ones, the Pairwise Heuristic resolves the uncertainty by going to distinguishable pairs. Two states are distinguishable if there is a high probability that different observations are recorded in the two states. Formally, and s' are distinguishable if and only if:
Ep(ols) ((p(oss) + p(ols')
p(ols)) 2 2A
A is constant that is specified by domain expert. If there is noise in observations, this value is 1. Otherwise, this value is close to but less than 1. The pairwise value (V(s, s')) of distinguishable pairs is simply the average of the value function of each of the states in the underlying MDP model of the environment (assuming full observability in the environment), i.e., (5(V(s) + (ss)).. To find the value function of the indistinguishable pairs, we use a value iteration algorithm an MDP where the states are pairs of states of our original problem. The transition function of this MDP is determined by the joint transition probability distribution of the original environment:
T(s,, s'), a, (s", 8")) = p((",, (s)) 8'), a)) = p(s"|s, (s,
The reward of each pair is simply the average reward of the two states in the original problem:
R(s, 8') 0.5(R(s) + +(('))
(3)
Therefore, the Bellman equation of our pairwise value iteration algorithm is as follows:
Vx(,,8') = mar., ( +
Σ T((,, 8'),,, (6", (6",,"")" V") "") 8 8
Initial pairwise values, i.c.. Vo(s,, s'), in the above equation is .5(((s) + V(s')) for distinguished pairs and the minimum possible reward for indistinguished ones. To select action, the Pairwise
4

---

Published as a conference paper at ICLR 2026
Heuristic POMDP-solver maximizes the expected value of pairs using the joint belief state, i..., b(s, 5') = b(s)b(s'). This essentially represents Value Iteration over the set of pairs of beliefs.
Σ Ms, (( (5,5*)
arg max
If the probabilities of all states, except the most likely one, become negligible, the selected action would be the optimal action of the underlying MDP for that most likely state.
Notably, in MOMDP, any pair of states with different visible factors are distinguishable. Therefore, the heavy computation of value iterations on pairs would be reduced to | -1//2, which a significant decrease in computational complexity when [S% << | More specifically these pairwise values can be represented as (s Sh, 5%)) where s, E S. and S/, 8%. E Sn.
3.2 VI ARCHITECTURE
All of the pairwise heuristic POMDP solver processes have a straightforward differentiable imple- mentation. The central part of this solver is the pairwise value iteration (Eq. 4.. which uses the pairwise transition (Eq. 2) and reward functions (Eq, 3 Moreover, the initial pairwise values are determined by the value of states (V() in the underlying MDP and the distinguishability of each pair of states (Eq. D The network implementation of these components is demonstrated in figure Ia
Starting from the value iteration algorithm implemented by a VI module, the network learns fr and fR. determining (s, a, s') and R(s) of the environment. They are combined as the value iteration equation dictates, to yield (s). From this point, the objective becomes converting elements of the environment to a pair-space representation to allow for the VI² module implementation. Specifically, we must convert T(s, a) and R(s) into T(x,, 8¹), a) and R(s, s') for all s, 8' E S S. We can convert R(s) using an averaging layer across all 8. As mentioned before, while the principles of value iteration networks remain the same, the details of kernels depend on the problem. Here, following the previous works focus 2D goal-based navigation with different types of uncertainty (More details in section 4. where the agent moves to one of the adjacent cells. Therefore, the transition function is implemented by 3 x kernel. This kernel must be transformed into transition kernel for the pairwise state space, which involves increasing the size of the kernel from (3, 3) to + 1) (2(VS+ grid world is a VS x VS square). This kernel is constructed using the learned transition probabilities from the VI Module and has number of channels equal to the number of actions available in the environment similar to fP. All nine values of each channel of fp would be mapped to the main diagonal of the pairwise transition kernel in the corresponding channel as demonstrated figure lb
+,)++122 1, 1,2((S + +1) +)++] 1) to allow for row and column transitions between pairs (assuming the
We must also determine which set of pairs (s,8') s') are distinguishable. We implement this by applying the convolutional kernel for the observation function, fz, to all states (the gridworld map) to get matrix Z. Then use the outer product of Z and implement Eq. 1 We express the distinguishability by a binary |S] x [S] matrix, D. The pairwise value initialization (Vo(s, s')) is done using matrix multiplication of D and (s) + V(s') (for distinguished pairs) in addition to multiplication of (1-D) D) and min in the shape of an |S] x |S] matrix (for indistinguishable pairs). With the pairwise reward and transition function calculated, pairwise value iteration (Eq. 4 is just another VI module, which we call the VI² module. Finally, the action selection (Eq. 5) is done by multiplying the pairwise belief state (outer product of belief by itself) with pairwise Q values and applying the max pooling layer (figure
Z and compare it with threshold to
The size of VI decreases significantly if the state space can be factorized into visible and hidden factors: = (sc, .8)) where S = Sy x Sr. In this case, the pairwise modules of R, V. and D would be |,S,| x |5%] [Sp] (with size [S.] (2) instead of |S] x |S| (with size = =S., 2 2. as
demonstrated in figure Dd. This is significant reduction when [Sh] <<< < |So).
4 RESULTS
We compared VI with the QMDP-Net on two types of grid-world navigation problems with differ- ent types of uncertainty and challenges. Since QMDP-Net has been shown to perform significantly better than unconstrained networks (Karkus et al. 2017,, we did not include them in analysis.
5

---

Published as a conference paper at ICLR 2026
Vise faradice Mooile
Ortwerdd
Transition Keme!
Pairwise Transition Kernel
1000
CA
0.
02
TU
Thu
Dittggishaad
(a)
(b)
Viluo
Viee
(g(ki)
Muxi
for oit
0, Au
V/A S
5%,
%
(d)
Figure 1: VI Architecture. a) Required preprocessing to prepare for Pairwise Value Iteration b) Example conversion of one channel of the transition kernel fp to the corresponding channel of pairwise transition kernel. Example assumes 4x4 environment of 16 states, where S = number of states, kernel size 2(v5 1) + 1, and modules, which uses outputs from part (a) to determine the selected action. d) Factorized Pairwise Value Iteration modules (Factorized version of part (c)).
kernell size/2 v8 + 1. Pairwise Value Iteration

---

Published as a conference paper at ICLR 2026
Additionally, we tested state-of- oftthe-rrttttttt transformer-basee architecture for solving fully observable MDPs on our task (Chen et al., (021). That model failed to succeed on over 10% of our test cases, so we excluded it from our analysis. We used the same belief update mechanism for all agents to have fair comparison between the policy modules. Moreover, we set the number of recurrences in the VI module of QMDP-net equal to the total number of recurrences in VI and VI modules of our network. Importantly, the transition kernel did not get updated in the pairwise module (VI²) (we did not pass gradients in this module). Therefore, Q-MDP net did actually have a computational advantage over VI in terms of the number of free parameters. Although QMDP is computationally advantageous to VIºN, the increase in success rate compensates for that increased complexity.
Our first task/problem set was the standard single-goal navigation where the position of the goal visible the input map, but the agent does not know its own position. Instead, it observes its surroundings. This problem set included several environments with various amounts of perceptual ambiguity. In the second navigation task/problem set, the agent knows its position but is unaware of the exact position of the goal. Instead, few possible locations for the goal are given to the agent. The agent should go to a "landmark" spot to find the goal. Multiple potential goal positions make the state space significantly larger. However, the state space be represented as (agent's position, goal's s position) in which the first factor is fully observable. Each of the two tasks/problem sets includes different structural versions, 4 in the first task and 3 in the second. Networks are trained on each structure (15000 to 30000 instances), and tested on novel environments of the same structure (3000 to 6000 instances). Details of the process can be found in appendix A
4.1 TASK OBSERVABLE GOAL, UNKNOWN POSITION
For the first task, we designed four environments with different levels of perceptual ambiguity. Each type of environment was tested at various levels of obstacle density. The observation and action functions were consistent among the environments to have a systematic comparison in terms of uncertainty and complexity of the decision-making, making. The actions were "right", "up". "down", and "left", moving the agent one cell in the direction specified by the name, and also action "stay". The agent was able to observe the cell it was on and also the neighboring cells in each of the cardinal directions. Since our focus was on perceptual ambiguity, not handling noise in sensors and actuators, both of our sensors and actuators were noise-free. Notably, we tested the networks on environments with noise, and both of them maintained their performance.
We started with a "random" environment. In the random environment. obstacles are randomly placed within the arena at both 5% and 10% density/sparsity levels (figure 2al top left). With an average of 20 or 40 obstacles in this environment, uncertainty would be resolved in a few steps. As a result, both networks had very high success rate (table[[ top row). We increased the ambiguity by adding the constraint of minimal continuity in each axis to the random environment, which produces very few blocks with side size of 4. This type of environment called "blocks", was generated to model environments where obstacles are randomly placed as independent clusters ((figure 2a bottom left). With the increase in perceptual ambiguity compared to the "random" environment, both networks' performance dropped. However, the drop was lower for VI (table ! the second row from the top). Our third environment called "walls", contained long walls parallel to the border in an empty arena, resembling long hallways for robots with sonar sensors ((figure 2a top right). The superiority of VIEN became appreciable in this challenging environment, where the middle walls and borders are not easily distinguishable (table the second row from the bottom).
Our most challenging environment was called "symmetric". In this environment, four copies of smaller random environment are placed in each corner of a larger grid-world (figure 2a bottom right). The density of each of the four repeated environment blocks was 5%, 10%, or 15%. This environment requires more long-term planning and information gathering, as it has more indistinguishable states that could lead to incorrect assumptions about belief. In this environment, the VI drastically outperformed the QMDP-Net, which demonstrates its ability to use long-term planning to generate effective policies. Moreover, VI was more robust to changes in sparsity in complex environments, whereas the QMDP-Net performance dropped at a higher rate as each type of environment became more challenging with increased sparsity. We also tested the robustness of networks against minor changes in the new environment by corupptnng the symmetry. We corrupted the full symmetry of our 10% density test set by changing five random empty cells to obstacles in one of the rooms (5% of that 10x x 10 room). The median success rate dropped to 68% for VI2N, and 48% for QMDP-net.
7

---

Published as a conference paper at ICLR 2026
Random
Walls
2 goals
Blocks
Symmetric
goals
(a)
(b)
Figure 2: Example testing environments. For the observable goal, unknown position task, there
four types of random, blocks, walls, and symmetric environments, with various density rates. The goal is not marked in those maps. b) For the unknown goal, observable position task, there are different numbers of potential places for the goal in the environment, and displayed are 2-goal and 4-goal examples. The potential goals are shown in green, and the landmark shown in orange. The agent finds out where the goal is actually placed if and only if goes to the landmark spot.
Table 1: Success rate of network solvers in task (unknown agent's position) in different environ- ments.
Model
Environment
Random
Random( 10%)
Walls(1)
Walls(2)
Walls(3)
VI
93 + 1%
95 + 1%
77 + 1%
83 + 1% 78 + 2%
82 = 1%
QMDP-Net
93 + 1%
96 + 1%
69 + 1%
80 = 2%
Blocks(5%)
Blocks(10%)
Symm(5%)
Symm (10%)
Symm(15%%
VI QMDP-Net
91% =
91 = 1%
76 = 3% 61 + 3%
74 # 3% 51 + 5%
65 4%
88 + 1%
89 + 1%
41 + 4%
4.2 TASK 2 UNKNOWN GOAL, OBSERVABLE POSITION
Our gridworlds for the second navigation task/problem set were generated with a similar script as the "random" worlds used the first problem set, with the side of each world, N, being 10, and obstacle density of 20%. However, we manipulated the number of possible goal locations displayed on the map, GG), for these worlds, increasing the uncertainty of the environment by adding more possible goal locations. Each location added increases the state space by N x N, yielding [S] N x N x |G| states. Our worlds all included "landmark" location, which produced an observation that revealed the hidden location of the goal the agent (number of observations O equals | + 1, representing each goal and a "neutrall" observation). Other locations had absolutely no information about the position of the goal. Example environments with 2 and possible goal locations are shown in figure 2b
The potential goal locations either yield large reward large penalty, disincentivizing the agent from going to potential goal places one by one, skipping the "landmark". This is similar to a natural task where an agent knows where it is. but does not know where the reward is. those cases, the agent may navigate the enviromment in order to gain information about the goal before proceeding to exploit the reward. This task also explores the concept of a MOMDP, incorporating visible and hidden component. In this task the visible factor of the state, S., is the agent's location. The hidden factor, Sr. the goal's position, which is one the potential locations (SS, [G]. Using this factorization, our pairwise modules were 100 times smaller (N x N x [ instead of (N x N)² x |G|2 where N -= =110). As seen in table our network significantly outperformed QMDP-net in all conditions. found that V v visited the landmark state more frequently than QMDt-net, visiting the landmark an average of 96% of the time, whereas the QMDP-net only visited the landmark average of 34% of the time.
Upon further examination of each solver's 's trajectory,
8

---

Published as a conference paper at ICLR 2026
Table Success rate of network solvers in task 2 (unknown goal. observable position) in different conditions (number of potential goal locations and stochasticity of the environment).
Number of goals
2
4
2
Transition Function
Deterministic
Stochastic
VIaN QMDP-net
98 = 2%
96 + 2%
95 - 2%
91 = 2%
91 = 3%
91 + 2%
57 + 4%
53 + 2%
2%
27 + 2%
+ 2%
30 + 17%
Table 4: Effect number of recurrence on VI²N success rate (task 2, G 4).
Table 3: Success Rate of network solvers
in task 2 (unknown goal, observable po- sition) in 20 20 environment with 4 goals.
lv1
ky
Success %
5
58 + 16%
Model
Success
20
1
41 + 2%
1
60
45 + 9%
V QMDP-net
79 + 3%
40
20
95 + 2%
27 2%
60
40
+ 2%
We also tested the models' robustness to noise by implementing a stochastic transition function. Specifically, we created environments in which the probability of taking the intended action was 60%, whereas the probability of moving left or right relative to the intended action was 10%, and the probability not moving was 20%. The entire evaluation process remained unchanged. Table 2 the gap between the performance of N and QMDP-net increased. Finally, we tested model scalability a 20 x 20 deterministic environment with goals, where V found the goal 79% in average, while QMDP-net's average success rate was 27% (Table 3
shown in
4.3 EFFECT OF RECURRENCE
V has two planning modules, each with recurrence parameter & (kvz and (2) specifying the planning horizon. To assess the importance of these modules and their associated parameters, we trained and tested VI with different recurrence parameters in the task 2 with 4 potential places for the goal. As demonstrated in table short horizons hurt the performance, while horizons above certain values do not lead to any improvement, similar classic value iteration. Moreover, the better performance of network with kv = kv = 5 compared to networks with k = 1 in one of the modules well QMDP-Net highlights the importance of both planning modules in V V22,, one for reward exploitation, and the other for resolving uncertainty.
4.4 INTERPRETABILITY
Besides the superiority of VI measured by an objective measure of success rate, our method produces interpretable maps that highlights both informative and rewarding areas. Specifically, in addition to producing value maps representing the space in terms of the value function of single states (V))), VI specifies informative areas via marginal pairwise values, i.e., E V (s, 8'), as demonstrated in figure This map significantly contributes to the interpretability of our method, explaining why specific actions were performed and why certain areas were visited more, especially when they
not directly related to the source of reward (goal). Informative states are not represented in the QMDP-net value function, even in the environments where the QMDP-Net performs well. This is expected, the algorithm behind policy generation of QMDP-Net does not take resolving uncertainty into account.
5 DISCUSSION
We have introduced the VI deep learning architecture for decision making under uncertainty, modeled after the fully differentiable Pairwise Heuristic. The VI'N architecture demonstrates the ability for long-term planning for resolving the uncertainty which exceeds the capacity of previously

---

Published as a conference paper at ICLR 2026
Map
VI3)
T
L
T
(a)
(b)
Figure 3: V represents rewarding areas through the value function of single states and informative areas through the value function of pairs. QMDP-Net focuses only the reward. a) Example generated maps in the first problem set. The leftmost maps represent the environment where the gray areas obstacles and the black cell represents the goal. b) Example generated maps in the second data set with for goal spots. Rewarding goal is shown in green, the mislcading "goals" shown red, the landmark state orange and the obstacles are shown in black
proposed network architectures seen in the VIN and the QMDP-Net, especially in challenging environments with high perceptual ambiguity. Moreover, it's the only network that can take advantage of mixed observability of the environment and factorization. Furthermore, addition to reward value maps, V12 generates information value maps, highlighting the informative areas in respect to the reward (goal). Since the main focus of our work is on the planning/policy module of the network, environments were simple 2D binary grid worlds similar to VIN and QMDP-No -ett papers. We expect improvements of classic VIN over the past years (Niu et al. 2018 Zhang et al. 2020 Ishida & Henriques] 2022) to be easily applicable
network as the main component of our network is still a VI module. In fact, applying these improvements an exciting future research direction to extend the applicability of our network.
Similar to other networks in the VIN family, the most significant limitation of VI²N is the requirement of spatial invariance for transition and observation functions. Moreover, all of the tasks in this paper were 2D navigation. This allowed us to have a systematic comparison based on different types and levels of uncertainty. Testing other tasks, such as grasping, would definitely contribute the reliability of our results. However, designing scalable, challenging, and intuitive setups for other tasks is unfortunately complicated. For example, as shown in the QMDP-Net paper, the available grasping environment is not even challenging for the classic QMDP algorithm with more than 98 percent success rate (Karkus et al. 2017), also worth mentioning that pairwise modules are easily scalable for environments with higher dimensions. For example, the only major structural change for a 3D navigation task is using a larger kernel for pairwise transition function. Finally, our results are limited to learning from expert (and not from reward reinforcement).
6 REPRODUCIBILITY STATEMENT
Details of the experiments, including network parameters, the training and testing processes, and environment generation, are available the appendix(A] The code for implementing the networks and generating environments is available at http:///ghhhb.commcyyyaaaaeeemii
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Mauricio Araya-López, Vincent Thomas, Olivier Buffet, and Francois Charpillet. A Closer Look at MOMDPs. 2010 22nd IEEE International Conference on Tools with Artificial
Intelligence, volume 2, pp. 197-204, October 2010. doi: 10.110//CTTA.2210.1000 URL
https://ieeexploree ieee wwt.eee..rr/ooommett//5557777770
ISSN: 2375-0197.
Richard Bellman. A Markovian Decision Process. Journal of Mathematics and Mechanics, (5):679 684, 1957. ISSN 00959057, 19435274. URL httlhhtt:////t www....stt.r.................... wwt.:rsttoooooooooooooooo stablel 24900506 Publisher: Indiana University Mathematics Department.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 14:15084-15000 11087, 2021.
Vincent François-Laved Peter Henderson, Riashat Islam, Mare G. Bellemare, and Joelle Pineau. An Introduction Deep Reinforcement Learning. Foundations and Trends@ Machine Learning, 11 (3-4):219-355) 354,, December 2018. ISSN 1935-8237, 9245- doi: 10. 001156/2200000071 URL
https: wwww www.nnwwwwwbss.....oomoo www.//rrcee///eeeeaaaaaaooaoa Publishers, Inc.
071
Publisher: Now
Daniel Golovin, Andreas Krause, and Debajyoti Ray. Near-optimal Bayesian active learning with noisy observations. In Proceedings of the 23rd International Conference on Neural Information Processing Systems Volume 1, NIPS' 10, pp. 766-774, Red Hook, NY, USA. December 2010.
Curran Associates Inc.
Shu Ishida and João F. Henriques. Towards real-world navigation with deep differentiable planners. In 2022 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2022.
Peter Karkus, David Hsu, and Wee Sun Lee. QMDP-Net: Deep Learning for Planning under Partial
Observability. In Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https: /tttpss//proceeeeii ngs neurips http:/pappe/20111///00 e9412ee 38 4b w0w.8 780844488888888886668688888688 02004400000000000000606666666060
Abst Abst html
Koosha Khalvati and Alan Mackworth. A Fast Pairwise Heuristic for Planning under Uncertainty. Proceedings of the AAAI Conference on Artificial Intelligence, 27(1):503 5095000559,,000 June 2013. ISSN 2374-3468, 2159-5399.
doi: 10. 10.1160/aaii.v22 v27il 22711.8672. URL https://ojs .wa.s.aaaa...... org/
Innex.phh/AAA///ttiice//////s// 86
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent Model-Free RI Can Be Strong Baseline for Many POMDPs, June 2022. URL htt.:tttp//axii..... http://axxi....rg/b//////// Number: 1rXiv:2110.05033 arXiv:21 [es].
05038
Sufeng NIu, Siheng Chen, Hanyu Guo, Colin Targonski, Melissa Smith, and Jelena Kovačević. Generalized Value Iteration Networks:L.ff Beyond Lattices. Proceedings the AAAI Conference on Artificial Intelligence, 32(1), April 2018. ISSN 2374-3468. doi: v0.1609/aaai.v32i 12211.12081. URL htttss://oss aaai org/ indei.nee.. 1.
AAAI http://pp/p/.aaaaaattaaleeea//ea
Number:
Sylvie C. W. Ong, Shao Wei Png, David Hsu, and Wee Sun Lee. Planning under Uncertainty for Robotic Tasks with Mixed Observability. The International Journal of Robotics Research, 29 (8):1033-1068 July 2010. ISSN 0278-3649. doi: (0.1177/027834491036666 Publisher: SAGE Publications Ltd STM.
URL https
S. Ross, J. Pineau, S. Paquet, and B. Chaib-draa Online Planning Algorithms for POMDPs. Journal of Artificial Intelligence Research, 32:663-704, July 2008. ISSN 1076-9757, doi: 10.1613/jair.257..
URL https:
www.. jair htt.rrorg//iieeeeppppaaaaaaaaaaa
10559
Edward J. Sondik. The Optimal Control Partially Observable Markov Processes over the In-
finite Horizom: Discounted Costs. Operations Research, 26(2):282-300 1978. ISSN 0030-
http://noooroorooeeee 364X. URL https: Sttts::/ecoopappers repec wwtp://ioggggaaaaeaaaaoeaeeo.oae wwtp///rriilee//ooooorreee//////
3ay 3a1 978 3a 3a2 3ap 3a 304. htm
Publisher: INFORMS.

---

Published as a conference paper at ICLR 2026
Aviv Tamar, YI WU, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value Itera- tion Networks. Advances in Neural Information Processing Systems, volume 29. Cur- Associates, Inc., 2016. URL
www.errs.iiii.omm www.niipp...oo...mm wwwppppcppppppp.cppptpppppppp cctpppperr/..//aaa..... c21 22202 Abstract ot htmml
Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics (Intelligent Robotics and Autonomous Agents), 2005.
Li Zhang, Xin Li, Sen Chen, Hongyu Zang. Jie Huang, and Mingzhong Wang. Univer- sal Value Iteration Networks: When Spatially-Invariant Is Not Universal. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6778-677) April 2020. ISSN 2374- 3468. doi: htt.//////aa..344..66.. axticle/view61151 Number: 04.
aaai org/ http:iidexxppppee Aapp/AAAI
12

---

Published as a conference paper at ICLR 2026
A NETWORK ARCHITECTURE AND DETAILS OF EXPERIMENT
A value iteration network for decision-making in partially observable environments is recur- rent network consisting of two main modules: The belief update (Bayes Filter) module and the policy/planning module (figure A-1 left). The input to the network is mainly the map of the environ- ment. After selecting each action (output), the network also receives an observation which used along with the chosen action update the belief for the actions. Belief update operations, such as update after action and normalization, are easily implementable in a network, as exemplified the QMDP-Net paper Karkus et al. (2017 (figure A-1 right). The policy module selects the action given the map and the belief, as demonstrated in figure of the main text.
GridWorkd
Map
by(s)
Di(s,
norm
Belief
0r0
Updete
Module
De-1
Palley Module
Bt-1
Gridworld
lz
Zis,
by.165)
Figure A-1: Value iteration networks for decision making under partial observability. a) The network consists of two main modules: the belief update module and the policy module. b) Belief update module: This module implements the step of belief update in the POMDP as explained in the main text and other works such QMDP-Net Karkus
Karkus et al. (2017).
For the first task, the input map is two binary matrices indicating obstacles v.s. empty space, and goal v.s. no goal, respectively. This means that, given the size of the map in all of our environments (20 20), this map was a binary x 20 x tensor. As mentioned in the main text, the actions were "right', 'up', 'down', and 'left', moving the agent one cell in the direction specified by the name and also action 'stay'. The observation range was also represented by able to observe the neighboring cells in each of the cardinal directions. Moreover, the goal state was observable when the agent was in Therefore, there were observations total. As mentioned in the main part of the paper, the action and observation functions were noise-freet foise-free/ / deterministic. Therefore, and for simplicity, we represented the observation function as a 5 x 3 tensor, using five channels instead of 17, each of which is representative of one of the (out of 9) observable cells.
3 kernel. The agent was
As our goal was to compare the policy modules, we assumed that the agent knows the observation function and updates the belief perfectly in all tested architectures (including QMDP nets). Therefore, each data point consisted of the grid world map, the belief as the input, and the selected action as the output. Networks for all environment types were trained and tested within the same type except "random" and "blocks". The training set for these two was simply the aggregate of data points of both types (we basically combined data points of these environments during training).
Each training data set for the first problem consisted of around 12000 environments, each of which with two or three different start states. For each type, density rates were distributed uniformly, e.g., 6000 environments with 5% density for the "random" and "blocks". The validaaaiotionnn ratio was 95% 5%. Moreover, the test success rate was calculated by running the generated model on 1,000 novel environments of the same type, each of which with 20 different start states for each density. The batch size was 100 for all methods and environments. The maximum number of epochs was 1000 for QMDP-Net and 300 for VIºN as it computationally more expensive. Both networks reached a stable point in the last quarter of training, suggesting that more epochs were unnecessary. All implementations N and QMDP- -Net) were in Pytorch, and the experiments were done with GPU boxes with NVIDIA GeForce RTX 2080 Ti graphic cards. Each epoch took at most hour (for VI
For our second task, the input map consists of obstacle, goal, and landmark channels. The obstacle channel is binary matrix like the obstacle matrix for our first problem set, representing obstacles vs.
13

---

Published as a conference paper at ICLR 2026
empty space. Each possible goal location in the environment represented in its own layer, the existence of one goal means the other goals are not valid in that state. Finally, the landmark, which returns observation revealing the true goal, was represented by another binary matrix indicating landmark state vs non-landmark state in the last layer. All of the grid worlds were 10 x 10 matrices, with varying number of possible goals, yielding input maps of size 10 10 (2 + [G])- In this case, the agent always knew its (x, y) dimension location, so the observation function was represented by well
|G] + 1 vector, with "null" observation, each state. As with the first problem, we first assume the agent knows the observation and transition functions and updates the belief perfectly all tested functions.
individual observation corresponding to
Our training set for the second task is slightly smaller, consisting of around 3000 environments with start states in each environment, totaling about 15000 unique action labels. We used a smaller dataset because of the smaller belief distribution in these tasks. Our test set was also slightly smaller, consisting of 2000 environments with 5 start states for each environment. Like with the first task, the train-validation ratio was 95 5%, and used batch size of 100 and allowed our models to learn for 300 epochs.
14


Reviews:
[{"review_id": "QkxtFCaSOb", "reviewer": "Reviewer_wGbv", "summary": "This paper introduces new a differentiable planning architecture ($VI^2N$) for decision making under uncertainty in partially observable environments. The method extends Value Iteration Networks (VINs) by integrating a pairwise heuristic mechanism (from prior works) that aims to explicitly model and reduce uncertainty before exploiting rewards. The authors also proposed how $VI^2N$ can leverage Mixed Observable MDP (MOMDP) factorisation to reduce computational complexity and improve scalability. They then present various experiments in several simple grid-world domains with different degrees of uncertainty (covering cases where the agent’s position or the goal’s position is unknown) and shows that $VI^2N$ outperforms QMDP-Net (the most directly related prior work) in success rate. An ablation study on the recurrence depth of $VI^2N$ further highlights the importance of the model’s planning horizon.", "questions": "Please see the weaknesses above. Mainly:\n\n1. **Baselines:** Why were no more recent POMDP or offline RL baselines included? Could you compare against transformer-based or uncertainty-aware architectures/algorithms (like CQL) to contextualise performance?\n2. **Scalability:** How does $VI^2N$ scale computationally and in performance for larger or continuous environments? Have you tested on larger grid-worlds or 3D navigation tasks as suggested in discussion, since you claim it is easily doable?\n3. **Statistical significance:** How many training seeds were used? Could you report mean ± std over multiple runs to assess (or at least illustrate) the significance of the improvements?\n4. **Robustness to noise:** How would $VI^2N$ perform under increased stochasticity (e.g., transition noise or noisy observations)? Does the network maintain its advantage over QMDP-Net in such cases? Does the plots of informative areas through the value function of pairs change as one would expect? \n5. **Ablation depth:** Beyond recurrence count, could you ablate other design elements (e.g., factorisation, pairwise distinguishability threshold λ) to clarify what drives improvements and the corresponding tradeoffs? Are there failure cases as a result of the way the rewards and Bellman equations are modified (or can the authors prove they maintain optimality)? In general could the authors analyse failure cases?\n6. **Interpretability consistency:** Are the “informative area” maps consistent across runs and different environment structures, or do they vary significantly depending on training initialisation?\n7. **Generalisation to non-expert settings:** Given that experiments rely on expert trajectories, how would $VI^2N$ perform under non-expert trajectories?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "* **Significance and originality:** The paper tackles an important limitation of previous value iteration-based networks (VINs, QMDP-Nets) by explicitly integrating information-gathering behavior into planning under uncertainty. This is highly relevant for domains such as robotic navigation and grid-world planning tasks.\n* **Experimental coverage:** The study provides experiments in several partially observable gridworld settings, including tasks with unknown agent position / known goal and known agent position / unknown goal, capturing a different types of uncertainty.\n* **Comparative results:** The paper demonstrates consistent and often substantial improvement over QMDP-Net, the closest relevant baseline, showing that $VI^2N$ can perform better in environments with significant perceptual ambiguity.\n* **Insightful ablation:** The analysis on the number of recurrences in the VI and $VI^2$ modules reveals how performance depends on planning depth, providing useful interpretability and model understanding.\n* **Clarity and theoretical grounding:** The exposition of the pairwise heuristic is conceptually clear and builds logically on established literature in POMDPs.", "weaknesses": "* **Limited generality:** Like QMDP-Net and other VIN-based models, $VI^2N$ assumes a discrete environment with spatially invariant transition kernels and discrete actions, which restricts applicability to continuous or large-scale real-world domains.\n* **Figure clarity:** The main architecture diagram (Figure 1) is dense and difficult to interpret, lacking a proper legend and a lot of missing arrow direction indicators, which makes following data flow challenging.\n* **Restricted evaluation scale:** Experiments are limited to small binary grid-worlds, which makes unclear how it will scale to larger maps or continuous domains (e.g., those used in the prior VIN or QMDP-Net works).\n* **Outdated baselines:** The authors only compare against QMDP-Net, which is a fairly old baseline (Karkus et al., 2017). They justify this by saying that the QMDP-Net paper showed that they perform significantly better than unconstrained networks. However there are several more recent POMDP baselines other than RNNs/LSTMs and behavior cloning (which is what Karkus et al. compared against), like transformers (e.g Decision transformers, TrXL, etc). Additionally, given that the experiments are all trained offline using expert trajectories, there are several more recent offline RL baselines (like CQL, IQL, etc).\n* **Weak evidence for claims:** It looks like the authors did not average their results across several training runs (or the number of seeds used is not reported). Hence it is unclear if the results are significant (let alone statistically significant). There are also no plots/results to validate the claim that $VI^2N$ focuses on reducing uncertainty *before* exploiting rewards. Finally, it is unclear if QMDP-Net also got/used the same factorised representation in Task 2 with the fully observable agent’s location (so the belief should similarly be only over unobserved variables).\n* **Little ablations and analysis:** Only one ablation (recurrence depth) is provided. No analysis of other hyperparameters and model failure modes is given. For example it is unclear how performance varies with $\\lambda$, and what tasks/situations are problematic for $VI^2N$ due to the changed bellman equation (Equation 4).\n* **Lack of robustness tests:** Although the architecture is designed for high uncertainty, the paper does not test performance under systematic increased stochasticity. E.g.\n  - Increasing action slip probabilities\n  - Noisy observation models like the classical “noisy TV” scenarios where observations become uninformative (e.g. where transitions into a specific grid position gives uniformly random observations).\n* **Scope of generalisation:** All experiments use offline expert demonstrations. Hence it is unclear how the method will be affected by non-expert demonstrations or the online RL settings. \n\nSome of these limitations are acknowledged in the last section (Discussion), including a couple additional ones. However given the severe lack of analysis as mentioned above (and no theory), the authors really should have included some of the experiments they leave to future works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oI1Z6Rbn7k", "reviewer": "Reviewer_6LxJ", "summary": "The paper \"Information-based Value Iteration Networks for Decision Making Under Uncertainty\" proposes to combine Value-Iteration networks with Partially Observable MDPs (POMDPs) to account for uncertainty in the environment. To overcome the computational complexity of obtaining the optimal policy for POMDPs they adapt a solver that uses the pairwise heuristic that estimates the value $V(s,s')$ for states $s,s'\\in S$. Notably, the authors argue that this heuristic is only necessary for the features that are uncertain, which can drastically decrease the amount of required computations. They provide empirical evaluation on two original gridworld datasets and show that their approach notably outperforms QMDP in environments with high uncertainty. Finally, they present \"information maps\" for each state that appear to make model decisions more interpretable than the value function.", "questions": "I am unsure how the threshhold $\\lambda$ has to be selected by a domain expert and why it should be close to 1. The reasoning behind this choice would be an interesting note to fully grasp the presented approach.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The combination of the solid theoretical foundation of value iteration with MOMDPs to account for uncertainty in the environment is inspiring. The theory developed in this paper was very clearly stated and easy to follow, even without significant knowledge on partial observability or pairwise heuristics for such problems. Furthermore, the observable goal, unknown position environments were motivated by real-world applications (like robots with sonar sensors), justifying their relevance. The emperical results show the benefits of this approach for difficult navigation environments. On top of that, the analysis highlights that both reward exploitation and resolving uncertainty are essential for the model's success. Finally, the obtained information maps provide an impressive insight into the agent's capabilities.", "weaknesses": "As mentioned in the discussion, the lack of evaluation beyond 2D navigation tasks is unfortunate, since it would help to understand the algorithms capabilities in other reinforcement learning domains. Specifically, you state that you successfully tested noisy environments; it would therefore have been interesting to see quantitative results for these experiments, since they are especially relevant for real-world applications. Furthermore, though QMDP-Net performs better than unconstrained networks, a comparison to at least one state-of-the-art baseline not specifically built for partial observability would have been beneficial.\n\nSo, the main issues are:\n- Limited experiments on relatively toy 2d environments \n- Lack of comparison against other established methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wzZw1SlbQT", "reviewer": "Reviewer_SnV9", "summary": "The paper seems to extend partially observable VIN, in particular the method QMDP-Net, with a pairwise heuristic for solving POMDP. Experiments on simple gridworld tasks show their approach solves more randomly generated tasks than QMDP-Net.", "questions": "- Why should the actions selected in Equation 5 solve a given POMDP?\n- What are the standard deviations in the tables over? Did you train the kernels multiple times and this is the STD over the random seeds? Is this about multiple runs of one set of kernels with noisy observations?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The pairwise approach seems to be especially well suited for tasks where the state space factors in observable and unobservable variables.", "weaknesses": "I recommend to reject this paper, because I simply fail to understand it. I am familiar enough with the original VIN approach, but without reading the QMDP-Net paper (which is quite old by now) I doubt many will be able to understand this paper. What is needed here is a complete rewrite that explains (in equations) how the kernels are learned, how the belief is updated, and how the resulting VIN is actually solving the POMDP. In particular the main innovation, the pairwise approach, must be explained much more for the presented equations to make sense. I tried to read Section 3 a couple of times, but I am still unsure what is computed here, why it is computed, and how this is supposed to solve a POMDP.\n\nIf other reviewers disagree with this statement, I am happy to be convinced otherwise, but I believe an ICLR paper should be accessible even to non-expert of a field like this. \n\n**Detailed Comments**\n- To solve the induced belief-MDP, one needs to do value iteration over the space of all possible belief distributions. I do not understand how this can be achieved in VIN, which only seems to work over discrete state spaces (beliefs are continuous).\n- The term uncertainty is ambiguously used: I believe you mean partial observability, and sometimes noisy observations, but uncertainty is more often associated with stochastic environments (aleatoric), or incomplete sampling (epistemic).\n- Some symbols are never or insufficiently defined, like $Q$ in Equation 5. This extends to fairly important concepts as the belief distribution $b(s,s')=b(s)b(s')$, where it is never defined how these $b(s)$ are updated or why they are independent from each other. \n- The notation often changes during the text. For example, the reward function is defined (and first used) as $R(s,a)$, but then later used as $R(s)$ or $R(s,a,z)$ without defining these terms formally.\n- Equation 1 defines whether $s$ and $s'$ are *distinguishable*, but contains a sum over $s$ and $s'$, which does not seem to make any sense.\n- It is unclear to me how the values $V(s, s')$ and $V(s_v, s_h, s'_h)$ are actually represented in the architecture.\n- The experiments are missing baselines, i.e., other approaches to solve POMDPs. Just comparing to QMDP-Net is not enough for a top-tier conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VTSWt0lwqe", "reviewer": "Reviewer_bA3z", "summary": "The paper introduces a new DNN architecture for RL. This architecture builds on the idea of Value Iteration Networks, extending it to better contend with the partial observability of POMDPs using the idea of Pairwise Heuristic.", "questions": "I'm open to changing the review score/s. Specifically:\n1. If the authors could motivate well that: the evaluation is sufficient for the method to convincingly dominate the baseline in a major set of tasks. Alternatively, increasing the number and types of tasks in the evaluation.\n2. If the authors could motivate well that the combination of VIN and PH is not straightforward.\n3. If the authors will add and motivate the HP tuning proces and stat. signif. evaluation.\n\nAdditional comments:\n1. Uncertainty can refer to different things, that are traditionally dealt with very differently, in RL: the partial observability of POMDPs, the epistemic-uncertainty that drives exploration in sparse-reward domains, and the stochasticity of reward / transitions (\"aleatoric\" uncertainty) that makes everything more challenging. Although I think the presentation is rather clear in its focus on POMDPs exclusively, I would have liked it to be even more explicit - preferably as early as possibly (i.e. intro), on the types of uncertainty that will / will not be addressed in this work.\n2. Lines 019-020: \"Tested on a diverse set of domains\". Since the evaluation is limited to navigation tasks in grid envs. with goal / position known / unknown, I do not think that this can be considered \"a diverse set of domains\". I would be more comfortable with a claim along the lines of \"a range of grid-based navigation tasks\".\n3. I would have liked the QMDP baseline to be presented in more detail (perhaps in a related work section?), so that the reader can better understand and contrast the method and the (only, previously SOTA?) baseline.\n4. In Section 3 there are multiple references to \"the value V\". It's not clear to me whether this refers specifically the value of the optimal policy $V^*$, the value of a general policy $V^\\pi$, or specifically the value of some expert policy $V^{\\pi_e}$. Could the authors specify (And add to the paper)?\n5. Table 2 and Table 3 seem to measure the same thing (success rate) in different units (% and out of 1). Have I misunderstood? If not - is there a reason they are not presented in the same units?\n6. Non-VI-based baselines for POMDP solving (i.e. standard popular algorithms with an LSTM) would put the results in better context.\n7. Can the authors include additional results / discussion of compute cost contrast between QMDP and their method (comparable / one is significantly more expensive than the other?)\n\nAdditional  minor comments\n1. Line 011: In my opinion (/understanding), value iteration is a dynamic programming method, not an RL method (it relies on knowing the full model of the env., not on learning from a dataset of interactions). I do not mean to nitpick, it is more that it would have been easier for me to follow the narrative had DP rather than RL been the term used.\n2. Line 017: \"This architecture over-emphasizes..\". Over means \"too much\" -> bad. Is that the intention? Perhaps simply \"emphasizes\"?\n3. Equation 1 sums across s' s', probably should be across s' s. I'd also denote the set $o \\in Z$ for the first sum, to improve clarity.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors are very clear and direct about the limitations of the paper, which I greatly appreciate.\n2. I found the presentation for the most part very clear.\n3. The idea seems very natural for the purpose of applying VI nets to POMDPs.", "weaknesses": "1. Limited evaluation: only navigation tasks. No RL experiments - only learning from an expert. \n2. Limited novelty: the paper takes two developed ideas (Value Iteration Networks (VIN) and Pairwise Heuristic (PH)) and combines them in an apparently straight forward manner.\n3. HP tuning: The HP tuning process for the method and the baseline QMDP are (I believe) unspecified. I.e. was there any? Was it for both methods? Was it under equivalent conditions (i.e. same amount of compute dedicated to the tuning of each?) and motivation for the decisions are missing.\n4. Statistical significance: There is a measure of stat. signif. in the tables. Is it standard div.? SEM? Other? I believe it is unspecified (unless I've missed it). Is it over different seeds (how many)? Is there a reason seeds are not necessary here? (random init of the DNN seems to justify different seeds, to me).\n5. Introduction to an RL audience could be expanded (see questions / comments). Specifically, I would have liked a brief overview of the training process (dataset with expert actions? learning through interaction with the environment? other?). If the authors could please include a brief explanation in the rebuttal, and add a description into the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)