Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: # Update after the rebuttal
Thank you for the rebuttal.
The authors claim that the source of objective mismatch comes from n-step Q-learning, and their method is well-justified in 1-step Q-learning. However, there is still a mismatch even with 1-step Q-learning because the bootstrapped target is also computed from the TreeQN. More specifically, there can be a mismatch between the optimal action sequences computed from TreeQN at time t and t+1 if the depth of TreeQN is equal or greater than 2. Thus, the author's response is still not convincing to me.
I like the overall idea of using a tree-structured neural network which internally performs planning as an abstraction of Q-function, which makes implementation simpler compared to VPN. However, the particular method (TreeQN) proposed in this paper introduces a mismatch in the model learning as mentioned above. One could argue that TreeQN is learning an "abstract" planning rather than "grounded" planning. However, the fact that reward prediction loss is used to train TreeQN significantly weakens this claim, and there is no such an evidence in the paper. 
In conclusion, I think the research direction is worth pursuing, but the proposed modification from VPN is not well-justified.

# Summary
This paper proposes TreeQN and ATreeC which perform look-ahead planning using neural networks. TreeQN simulates the future by predicting rewards/values of the future states and performs tree backup to construct Q-values. ATreeC is an actor-critic architecture that uses a softmax over TreeQN. The architecture is trained through n-step Q-learning with reward prediction loss. The proposed methods outperform DQN baseline on 2D Box Pushing domain and outperforms VPN on Atari games.

[Pros]
- The paper is easy to follow.
- The application to actor-critic setting (ATreeC) is novel, though the underlying idea was proposed by [O'Donoghue et al., Schulman et al.].

[Cons]
- The proposed method has a technical issue.
- The proposed idea (TreeQN) and underlying motivation are almost same as those of VPN [Oh et al.], but there is no in-depth discussion that shows why TreeQN is potentially better than VPN. 
- Comparison to VPN on Atari is not much convincing. 

# Novelty and Significance
- The underlying motivation (planning without predicting observations), the architecture (transition/reward/value functions applied to the latent state space), and the algorithm (n-step Q-learning with reward prediction loss) are same as those of VPN. But, the paper does not provide in-depth discussion on this. The following is the differences that I found from this paper, so it would be important to discuss why such differences are important.

1) The paper emphasizes the "fully-differentiable tree planning" aspect in contrast to VPN that back-propagates only through "non-branching" trajectories during training. However, differentiating TreeQN also amounts to back-propagating through a "single" trajectory in the tree that gives the maximum Q-value. Thus, the only difference between TreeQN and VPN is that TreeQN follows the best (estimated) action sequence, while VPN follows the chosen action sequence in retrospect during back-propagation. Can you justify why following the best estimated action sequence is better than following the chosen action sequence during back-propagation (see Technical Soundness section for discussion)?

2) TreeQN only sets targets for the final Q-value after tree backup, whereas VPN sets targets for all intermediate value predictions in the tree. Why is TreeQN's approach better than VPN's approach? 

- The application to actor-critic setting (ATreeC) is novel, though the underlying idea of combining Q-learning with policy gradient was proposed by [O'Donoghue et al.] and [Schulman et al.].

# Technical Soundness
- The proposed idea of setting targets for the final Q-value after tree backup can potentially make the temporal credit assignment difficult, because the best estimated actions during tree planning does not necessarily match with the chosen actions. Suppose that TreeQN estimated "up-right-right" as the best future action sequence the during 3-step tree planning, while the agent actually ended up with choosing "up-left-left" (this is possible because the agent re-plans at every step and follows epsilon-greedy policy). Following n-step Q-learning procedure, we end up with setting target Q-value based on on-policy action sequence "up-left-left", while back-propagating through "up-right-right" action sequence in the TreeQN's plan. This causes a wrong temporal credit assignment, because TreeQN can potentially increase/decrease value estimates in the wrong direction due to the mismatch between the planned actions and chosen actions. So, it is unclear why the proposed algorithm is technically correct or better than VPN's approach (i.e., back-propagating through the chosen actions in the search tree).
 
# Quality
- Comparison to VPN on Atari is not convincing because TreeQN-1 is actually (almost) equivalent to VPN-1, but the results show that TreeQN-1 performs much better than VPN on many games. Since the authors took the numbers from [Oh et al.] rather than replicating VPN, it is possible that the gap comes from implementation details (e.g., hyperparameter). 

# Clarity
- The paper is overall easy to follow and the description of the proposed method is clear.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
 This was an interesting read.  

I feel that there is a mismatch between intuition of what a model could do (based on the structure of the architecture) versus what a model does. Just because the transition function is shared and the model could learn to construct a tree, when trained end-to-end the system is not sufficiently constrained to learn this specific behaviour. More to a point. I think the search tree perspective is interesting, but isn’t this just a deeper model with shared weights? And a max operation? It seems no loss is used to force the embeddings produced by the transition model to match the embeddings that you would get if you take a particular action in a particular state, right? Is there any specific attempt to visualize or understand the embeddings inside the tree? The same regarding the rewards. If there is no auxiliary loss attempting to force the intermediary prediction to be valid rewards, why would the model use those free latent variables to encode rewards? I think this is a pitfall that many deep network papers fall, where by laying out a particular structure it is directly inferred that the model discovers or follows a particular solution (where the latent have prescribed semantics). I would argue that is rarely the case. When the system is learned end-to-end, the structure does not impose the behaviour of the model, and is up to the authors of the paper to prove that the trained model does anything similar to expanding a tree. And this is not by showing final performance on a game. If indeed the model does anything similar to search, than all intermediary representations should correspond to what semantically they should. 
Ignoring my verbose comment, another view is that the baseline are disadvantaged to the treeQN, because they have less parameters (and are less deep which has a huge impact on the learnability and expressivity of the deep network). 

review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose a new network architecture for RL that contains some relevant inductive biases about planning. This fits into the recent line of work on implicit planning where forms of models are learned to be useful for a prediction/planning task. The proposed architecture performs something analogous to a full-width tree search using an abstract model (learned end-to-end). This is done by expanding all possible transitions to a fixed depth before performing a max backup on all expanded nodes. The final backup value is the Q-value prediction for a given state, or can represent a policy through a softmax.

I thought the paper was clear and well-motivated. The architecture (and various associated tricks like state vector normalization) are well-described for reproducibility. 

Experimental results seem promising but I wasn’t fully convinced of its conclusions. In both domains, TreeQN and AtreeC are compared to a DQN architecture, but it wasn’t clear to me that this is the right baseline. Indeed TreeQN and AtreeC share the same conv stack in the encoder (I think?), but also have the extra capacity of the tree on top. Can the performance gain we see in the Push task as a function of tree depth be explained by the added network capacity? Same comment in Atari, but there it’s not really obvious that the proposed architecture is helping. Baselines could include unsharing the weights in the tree, removing the max backup, having a regular MLP with similar capacity, etc.

Page 5, the auxiliary loss on reward prediction seems appropriate, but it’s not clear from the text and experiments whether it actually was necessary. Is it that makes interpretability of the model easier (like we see in Fig 5c)? Or does it actually lead to better performance?  

Despite some shortcomings in the result section, I believe this is good work and worth communicating as is.
review_title: Interesting proposal, well-written, some short-comings in the results.

Meta-review: This is a nicely written paper proposing a reasonably interesting extension to existing work (e.g.
VPN). While the Atari results are not particular convincing, they do show promise. I encourage
the authors to carefully take the reviewers' comment into consideration and incorporate them
to the final version.
Decision: Accept (Poster)