Learning  agents with prioritization and parameter noise in continuous state and action space

Abstract: Reinforcement  Learning  (RL) problem can be solved in two different ways - the Value function-based approach and the policy optimization-based approach - to eventually arrive at an optimal policy for the given environment. One of the recent breakthroughs in reinforcement learning is the use of deep neural networks as function approximators to approximate the value function or q-function in a reinforcement learning scheme. This has led to results with agents automatically learning how to play games like alpha-go showing better-than-human performance. Deep Q-learning networks (DQN) and  Deep Deterministic Policy Gradient (DDPG) are two such methods that have shown state-of-the-art results in recent times. Among the many variants of RL, an important class of problems is where the state and action spaces are continuous --- autonomous robots, autonomous vehicles, optimal control are all examples of such problems that can lend themselves naturally to reinforcement based algorithms, and have continuous state and action spaces. In this paper, we adapt and combine approaches such as DQN and DDPG in novel ways to outperform the earlier results for continuous state and action space problems.  We believe these results are a valuable addition to the fast-growing body of results on Reinforcement Learning, more so for continuous state and action space problems.

Reviews and author responses:

The paper proposes an augmentation of the DDPG algorithm with prioritized experience replay plus parameter noise. Empirical evaluations of the proposed algorithm are conducted on Mujoco benchmarks while the results are mixed.

As far as I can see, the paper contains almost no novelty as it crudely puts together three existing algorithms without presenting enough motivation. This can be clearly seen even from the structuring of the paper, since before the experimental section, only a short two-paragraph subsection (4.1) and an algorithm chart are devoted to the description of the main ideas. Furthermore, the algorithm itself is a just simple addition of well-known techniques (DDPG + prioritized experience replay + parameter noise) none of which is proposed in the current paper. Finally, as shown in the experimental sections, I don't see a evidence that the proposed algorithm consistently outperform the baseline.

To sum up, I believe the submission is below the novelty threshold for a publication at ICLR.

This paper combines elements of two existing reinforcement learning approaches, namely, Deep Q-learning Networks (DQN) with Prioritised Experience Replay (PER) and Deep Deterministic Policy Gradient (DDPG) to propose the Prioritized Deep Deterministic Policy Gradient (PDDPG) algorithm. The problem is interesting and there is a nice review of relevant work. The algorithm has a limited novelty with a simple modification of the DDPG algorithm to add the PER component. Experiment results show improvements in certain simulation environments. However, the paper lacks insight on how and why results are improved on some settings while performing worse than the others. Detailed comments are as follows:

1. Algorithm 1 is not self-contained. Yes, I understand that it is a slight modification to DDPG with changes being Line 11 and 16. But p_i^alpha is not defined anywhere in Algorithm 1. How the transition probabilties are updated on Line 16 is also not clear to me.

2. It would be better if multiple simulation runs on the same experiment can be performed to have a more reliable display of performance.

3. Section 6 is on Parameter Space Noise for Exploration. This is not the authors' proposed work so it is strange to have a separate section here. In the end of Section 1, the authors wrote that "We then use the concept of parameter space noise for exploration and show that this further improves the rewards achieved." This seems to be a bold claim from the varying performance displayed in Figure 2-4. Similar to Comment 2, more simulation runs and statistical tests need to be conducted to support this claim.

The paper proposes PDDPG, a combination of prioritized experience replay, parameter noise exploration, and DDPG. Different combinations are then evaluated on MuJoCo domains, and the results are mixed. 

The novelty of the work is limited, and the results are hard to interpret: sometimes PDDPG performs better, sometimes worse, and the training curves are only obtained with a single random seed. Also presented results are substantially worse than current state of the art (e.g., TD3, SAC).


Meta-review: The authors take two algorithmic components that were proposed in the context of discrete-action RL - priority replay and parameter noise - and evaluate them with DDPG on continuous control tasks. The different approaches are nicely summarized by the authors, however the contribution of the paper is extremely limited. There is no novelty in the proposed approaches, the empirical evaluation is inconclusive and limited, and there is no analysis or additional insights or results. The AC and the reviewers agree that this paper is not strong enough for ICLR.

Decision: rejected-papers