Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper presents a nearest-neighbor based continuous control policy.  Two algorithms are presented: NN-1 runs open-loop trajectories from the beginning state, and NN-2 runs a state-condition policy that retrieves nearest state-action tuples for each state.  

The overall algorithm is very simple to implement and can do reasonably well on some simple control tasks, but quickly gets overwhelmed by higher-dimensional and stochastic environments.  It is very similar to "Learning to Steer on Winding Tracks Using Semi-Parametric Control Policies" and is effectively an indirect form of tile coding (each could be seen as a fixed voronoi cell).  I am sure this idea has been tried before in the 90s but I am not familiar enough with all the literature to find it (A quick google search brings this up: Reinforcement Learning of Active Recognition Behaviors, with a chapter on nearest-neighbor lookup for policies: https://people.eecs.berkeley.edu/~trevor/papers/1997-045/node3.html).

Although I believe there is work to be done in the current round of RL research using nearest neighbor policies, I don't believe this paper delves very far into pushing new ideas (even a simple adaptive distance metric could have provided some interesting results, nevermind doing a learned metric in a latent space to allow for rapid retrainig of a policy on new domains....), and for that reason I don't think it has a place as a conference paper at ICLR.  I would suggest its submission to a workshop where it might have more use triggering discussion of further work in this area.
review_title: Well-written but nothing particularly new.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work shows that a simple non-parametric approach of storing state embeddings with the associated Monte Carlo returns is sufficient to solve several benchmark continuous control problems with sparse rewards (reacher, half-cheetah, double pendulum, cart pole) (due to the need to threshold a return the algorithms work less well with dense rewards, but with the introduction of a hyper-parameter is capable of solving several tasks there). The authors argue that the success of these simple approaches on these tasks suggest that more changing problems need to be used to assess new RL algorithms.

This paper is clearly written and it is important to compare simple approaches on benchmark problems. There are a number of interesting and intriguing side-notes and pieces of future work mentioned.

However, the originality and significance of this work is a significant drawback. The use non-parametric approaches to the action-value function go back to at least [1] (and probably much further). So the algorithms themselves are not particularly novel, and are limited to nearly-deterministic domains with either single sparse rewards (success or failure rewards) or introducing extra hyper-parameters per task.

The significance of this work would still be quite strong if, as the author's suggest, these benchmarks were being widely used to assess more sophisticated algorithms and yet these tasks were mastered by such simple algorithms with no learnable parameters. Yet, the results do not support the claim. Even if we ignore that for most tasks only the sparse reward (which favors this algorithm) version was examined, these author's only demonstrate success on 4, relatively simple tasks.

While these simple tasks are useful for diagnostics, it is well-known that these tasks are simple and, as the author's suggest "more challenging tasks  .... are necessary to properly assess advances made by sophisticated, optimization-based policy algorithms." Lillicrap et al. (2015) benchmarked against 27 tasks, Houtfout et al. (2016) compared in the paper also used Walker2D and Swimmer (not used in this paper) as did [2], OpenAI Gym contains many more control environments than the 4 solved here and significant research is pursing complex manipulation and grasping tasks (e.g. [3]). This suggests the author's claim has already been widely heeded and this work will be of limited interest.

[1] Juan, C., Sutton, R. S., & Ram, A. Experiments with Reinforcement Learning in Problems with Continuous State and Action Spaces.

[2] Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2017). Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560.

[3] Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2017). Overcoming exploration in reinforcement learning with demonstrations. arXiv preprint arXiv:1709.10089.
review_title: Limited impact

final_score: rating: 4: Ok but not good enough - rejection
review_content: SUMMARY
The paper deal with the problem of RL.  It proposes a non-parametric approach that maps trajectories to the optimal policy.  It avoids learning parameterized policies.  The fundamental idea is to store passed trajectories.  When a policy is to be executed, it does nearest neighbor search to find then closest trajectory and executes it.

COMMENTS

What happens if the agent finds it self  in a state that while is close to a state in the similar trajectory the action required to could be completely different.

Not certain about the claim that standard RL policy learning algorithms make it difficult to assess the difficulty of a problem. 

How do you execute a trajectory? Actions in RL are by definition stochastic, and this would make it unlikely that a same trajectory can be reproduced exactly.

review_title: The algorithm presented despite its simplicity cannot compete with classic RL algorithms.  The claim about assessing problem difficulty with this approach is not well justified.

Meta-review: Evaluating simple baselines for continuous control is important and nearest neighbor search methods are interesting. However, the reviewers think that the paper lacks citation and comparison to some prior work and evaluation on more challenging benchmarks.
Decision: Reject