Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces two alternatives to value iteration network (VIN) proposed by Tamar et al. VIN was proposed to tackle the task of learning to plan using as inputs a position and an image of the map of the environment. The authors propose two new updates value propagation (VProp) and max propagation (MVProp), which are roughly speaking additive and multiplicative versions of the update used in the Bellman-Ford algorithm for shortest path. The approaches are evaluated in grid worlds with and without other agents.

I had some difficulty to understand the paper because of its presentation and writing (see below). 

In Tamar's work, a mapping from observation to reward is learned. It seems this is not the case for VProp and MVProp, given the gradient updates provided in p.5. As a consequence, those two methods need to take as input a new reward function for every new map. Is that correct?
I think this could explain the better experimental results

In the experimental part, the results for VIN are worse than those reported in Tamar et al.'s paper. Why did you use your own implementation of VIN and not Tamar et al.'s, which is publicly shared as far as I know?

I think the writing needs to be improved on the following points:
- The abstract doesn't fit well the content of the paper. For instance, "its variants" is confusing because there is only other variant to VProp. "Adversarial agents" is also misleading because those agents act like automata.

- The authors should recall more thoroughly and precisely the work of Tamar et al., on which their work is based to make the paper more self-contained, e.g., (1) is hardly understandable.

- The writing should be careful, e.g., 
value iteration is presented as a learning algorithm (which in my opinion is not) 
\pi^* is defined as a distribution over state-action space and then \pi is defined as a function; ...

- The mathematical writing should be more rigorous, e.g., 
p.2:
T: s \to a \to s', \pi : s \to a
A denotes a set and its cardinal
In (1), shouldn't it be \Phi(o)? all the new terms should be explained
p. 3:
definition of T and R 
shouldn't V_{ij}^k depend on Q_{aij}^k?
T_{::aij} should be defined
In the definition of h_{aij}, should \Phi and b be indexed by a?

- The typos and other issues should be fixed:
p. 3:
K iteration
with capable
p.4:
close 0
p.5:
our our
s^{t+1} should be defined like the other terms
"The state is represented by the coordinates of the agent and 2D environment observation" should appear much earlier in the paper. 
"\pi_\theta described in the previous sections", notation \pi_\theta appears the first time here...
3x3 -> 3 \times 3
ofB
V_{\theta^t w^t}
p.6:
the the
Fig.2's caption:
What does "both cases" refer to? They are three models.
References:
et al.
YI WU

review_title: An extension of Value Iteration Network; the writing needs to be greatly improved 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The original value-iteration network paper assumed that it was trained on near-expert trajectories and used that information to learn a convolutional transition model that could be used to solve new problem instances effectively without further training.

This paper extends that work by
- training from reinforcement signals only, rather than near-expert trajectories
- making the transition model more state-depdendent
- scaling to larger problem domains by propagating reward values for navigational goals in a special way

The paper is fairly clear and these extensions are reasonable.  However, I just don't think the focus on 2D grid-based navigation has sufficient interest and impact.  It's true that the original VIN paper worked in a grid-navigation domain, but they also had a domain with a fairly different structure;  I believe they used the gridworld because it was a convenient initial test case, but not because of its inherent value.   So, making improvements to help solve grid-worlds better is not so motivating.  It may be possible to motivate and demonstrate the methods of this paper in other domains, however.  The work on dynamic environments was an interesting step:  it would have been interesting to see how the "models" learned for the dynamic environments differed from those for static environments.


review_title: An extension to value-iteration networks that improves performance on grid-worlds

final_score: rating: 7: Good paper, accept
review_content: ORIGINALITY & SIGNIFICANCE

The authors build upon value iteration networks: the idea that the value function can be computed efficiently from rewards and transitions using a dedicated convolutional network. The authors point out that the original "value iteration network” (Tamar 2016) did not handle non-stationary dynamics models or variable size problems well and propose a new formulation to extend the model to this case which they call a value propagation network.  It seems useful and practical to compute value iteration explicitly as this will propagate values for us without having to learn the propagated form through extensive gradient update steps. Extending to the scenario of non-stationary dynamics is important to make the idea applicable to common problems. The work is therefore original and significant.

The algorithm is evaluated on the original obstacle grids from Tamar 2016 and larger grids generated to test scalability. The authors Prop and MVProp are able to solve the grids with much higher reliability at the end of training and converge much faster.  The M in MVProp in particular seems to be very useful in scaling up to the large grids. The authors also show that the algorithm handles non-stationary dynamics in an avalanche task where obstacles can fall over time.


QUALITY

The symbol d_{rew} is never defined — what does “new” stand for? It appears to be the number of latent convolutional filters or channels generated by the state embedding network. 

Section 2.2 Sentence 2: The final layer representing the encoding is given as ( R^{d_rew  x d_x x d_y }.
Based on the description  in the first paragraph of section 2, it sounds like d_rew might be the number of channels or filters in the last convolutional layer. 

In equation 1, it wasn’t obvious to me that the expression max_a q_{ij}^{k-1} q^{k} corresponds to an actual operation?
The h( \Phi( x ), v^{k-1} ) sort of makes sense …  value is only calculated with respect to only the observation of the maze obstacles but the policy \pi is calculated with respect to the joint  observation and agent state. 

The expression 

   h_{aid} ( \phi(0), v )   =   <  Wa,   [ \phi(o) ; v ]   >   +   b

makes sense and reminds me of the Value Iteration network work where we take the previous value function, combine it with the reward function and use convolution to compute the expectation (the weights Wa encode the effect of transitions). I gather the tensor Wa = R^{|A| x (d_{rew} x d_x x d_y } both converts the feature embedding \phi{o} to rewards and represents the transition / propagation of reward across states due to transitions and discounts at the same time? 

I didn’t understand the r^in, r&out representation in section 4.1. These are given by the domain?

I did get the overall idea of efficiently creating a local value function in the neighborhood of the current state and passing this to the policy so that it can make a local decision.

A bit more detail defining terms, explaining their intuitive role and how the output of one module feeds into the next would be helpful.


POST REVISION COMMENTS:

- I didn't reread the whole thing -  just used the diff tool.  
- It looks like the typos in the equations got fixed
- The new phrase "enables to learn to plan" seems pretty awkward


review_title: Useful extension to Value Iteration Networks to extend value of explicitly incorporate VI into problems with dynamic state

Meta-review: This paper and reviews makes for a difficult call.  The reviewers appear to be in agreement that Value Propagation provides an interesting algorithmic advance over earlier work on Value Iteration networks.  AnonReviewer1 gives a strong rationale why the advance is both original and significant.  Their experiments also show very nice results with VProp and MVProp in 2-D grid-worlds.

However, I also fully agree with AnonReviewer2 that testing in other domains beyond 2-D grid-world is necessary.  Earlier work on VIN was also tested on a Mars Rover / continuous control domain, as well as graph-based web navigation task.  The authors' rebuttal on this point comes across as weak.  In their view, they can't tackle real-world domains until VProp has been proven effective in large, complex grid-worlds.  I don't buy this at all -- they could start initial experiments right away, which would perhaps yield some surprising results. Given this analysis, the committee recomments this paper for workshop.

Pros: significant algorithmic advance, good technical quality and writeup, nice results in 2-D grid world.

Con: Validation is only in 2-D grid-world domains. 
Decision: Invite to Workshop Track