Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a first step in the direction of 'real-world relevant RL approaches' in the sense of considering environments that don't halt their execution until an agent has finished its optimal action computation and execution but actually just go on being an environment. For this, the notion of a concurrent action is introduced.

The paper focuses on value-based RL approaches. It introduces modifications to the classical MDP formulation such that concurrent actions can be handled. From a theoretical perspective the resulting Bellman operators (for both continuous and discrete time) remain contractions and thus maintain q-learning convergence guarantees. Qlearning models are adopted to support concurrent actions and the experiments  demonstrate that the suggested enhancements are working well.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper considers the theoretically interesting and practically important problem of concurrent deep reinforcement learning (DRL), i.e., DRL in which the agent has to decide the next action while performing the previous one. This introduces several significant challenges, including delays/latency and interruption of an on-going action. To address this issue, this paper proposes to consider the continuous time formulation of the concurrent control problem, derive a continuous-time Bellman equation for the concurrent control scenarios, and then derive its discrete-time counterpart. Contraction properties are shown for both the continuous-time and discrete-time concurrent Bellman equations, and a value-based DRL algorithm based on the concurrent Bellman equations is proposed and tested on a few tasks. 

The high level idea of this paper is very interesting and attractive, and in particular, the introduction of continuous-time reinforcement learning is novel. In addition, the numerical experiments do show that a consistently improved performance of the proposed approach on both synthetic and more real-world robotic control tasks. However, there are several significant issues about technical clarity or even correctness in this paper, which I elaborate below:

1. The settings in sections 3.1 and 3.2 are not clear. In particular, for 3.1, the author may want to specify the policy clearly, including whether it is stationary or non-stationary. And in addition, Q and V functions should either come with a \pi superscript, indicating which policy they use, or a \star superscript to indicate optimality. Section 3.2 does not make sense to me in general. It is not clear what the index i and the state value s_i(t) are. And it is not clear why we need to differentiate between values of states/actions and the functions themselves. The trajectory \tau is also not clearly defined. The authors need to make these much more clear, and should clearly state the main setting/model that the paper is considering (which seems to be the concurrent discrete-time case, but also not very clear to me).

2. The explanation of concurrent actions in continuous and discrete time is not clear. In particular, Section 3.3 only speaks of the settings on a high level, and only brief explanations are given in Figure 1b and the beginning of Section 3.4. Since the concurrent action setting is the central theme in this paper, I think a much more formal explanation should be given about how the system proceeds, instead of just a graphical example illustration. In addition, the concurrent actions in discrete-time setting part is not even clearly mentioned (but is stated in the title of Section 3.3 and discussed subsequently). The authors may also want to explain clearly what the episode is at the beginning of Section 3.4.

3. The concurrent Bellman equation does not make much sense to me. In particular, I think to define the optimal Q function, the bellman equation (7) and (9) should have a \max operator included. Otherwise, it is only for policy evaluation. Since the authors didn't clearly specify what the exact algorithm they are using (apart from a brief explanation by words in Section 3.5), I'm not sure whether I'm missing anything or not. But the authors should definitely include a algorithm frame at least in the appendix, to clearly specify which of and how the concurrent Bellman equations are applied in their algorithm.

So in sum, although I think the paper is interesting and novel on the high level, I don't think it's ready for publishing.

############ post rebuttal comment ############
After reading the authors' rebuttal and the modified version of the paper, I think most of my concerns have been correctly addressed. So I decide to improve my score to weak accept.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper tackles the problem of making decisions for the next action while still engaged in doing the previous actions. Such a delay could either be part of the design (like a robot deciding the next action before its actors and motors have come to full rest after the current action) or an artefact of the delays inherent in the system (i.e. latency induced by calculations or latency of sensors). The paper shows how to model such delays within the Q-learning framework, show that their modelling preserves desirable contraction property of the Bellman update operator, and put their model into practice by an extensive set of experiments: learning policies for several simulated and a real-world setting.

The authors claim that that addition of this "delay" does not hinder the performance much of the RL method is given sufficient "context" about the delay, i.e., given extra features as input in order to learn to compensate for it. The writing of the paper is lucid and sufficient background is provided to make the paper self-sufficient in its explanations.

However, there are some reasons which do not allow me to fully support the paper's acceptance.

The changes made to the basic Q-learning setup, albeit novel and with desirable properties, in my opinion, are (i) theoretically relatively straight forward, (ii) are not expressive enough to capture the problem in its full generality (explained later), and (iii) need more empirical justification with problems where their modification is indeed indispensable. The authors touch on several different research areas cursorily (viz. continuous reinforcement learning, Bellman contractions, feature engineering) while providing grounds for their idea, but in the end return to the familiar domain of discrete Q-learning with semi-hand-crafted (though theoretically motivated) features where the latency of actions can take a set of fixed values and the state is sampled at fixed intervals. 

If the actions are continuous, then could method from Doya (2000) directly be used to solve these problems? Can the value-based models which he describes be augmented and extensions developed which build on Lemma 3.1 instead of the well-trodden ground of Lemma 3.2? Especially, if one of the objectives which the authors claim their policies are better is "policy duration", then the absence of purely continuous policies is particularly egregious. Further, reducing the policy duration seems like an independent objective which perhaps can be used for reward shaping for the traditional policy methods, which will also lead to different baselines.

The authors explicitly say that their method focuses on "optimizing for a specific latency regime as opposed to being robust to all of them;" and that they explicitly avoid learning forward models by including additional features. However, the advantages of placing such restrictions on the design space are unclear at best. Would it be the case that the high-dimensional methods will fail in this setting? Are there theoretical advantages to working on limiting the attention to known latency regimes? I suspect that the authors have concrete reasons for making these design decisions, but these do not come across in the paper in the writing, or by means of additional baselines.

As an example of a different approach towards the problem, which the authors overlook in their related work section, is that of learning with spiking neurons and point processes. These areas of research have also been interested in problems of the "thinking while moving" nature: that of reinforcement learning in the context of neurons where the neurons act by means of spikes in response to the environment and other "spikes" [1, 2]. More recently, with point processes, methods have been developed to attain truly asynchronous action and state updates [3, 4]. A differently motivated work which ends up dealing with similar problems is in the direction of adaptive skip intervals [5], where the network also chooses the "latency" in the discrete sense. Adding such related work would help better contextualize this paper.

Some other ways the authors can improve the paper are (in no particular order):

 - The description of the Vector-to-go is insufficient; some concrete examples will help.
 - The results of the simulated experiments are given in the form of distributions and it is very difficult to discern the effect of individual features in Figure 1. Additionally, due to missing error bars, or other measures of uncertainty, the claim that the performance of models with and without the delayed-actions is comparable to the blocking setting seems tenuous at best, just looking at the rewards.
 - In particular, for the real experiments, we need more details about the experiment runs to determine why the performance of the policies in the real world is so vastly different. Could the authors describe why the gap can be completely covered through simulations but not in the real world?

[1]: Vasilaki, Eleni, et al. "Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail." PLoS computational biology 5.12 (2009): e1000586.
[2]: Frémaux, Nicolas, Henning Sprekeler, and Wulfram Gerstner. "Reinforcement learning using a continuous time actor-critic framework with spiking neurons." PLoS computational biology 9.4 (2013): e1003024.
[3]: Upadhyay, Utkarsh, Abir De, and Manuel Gomez Rodriguez. "Deep reinforcement learning of marked temporal point processes." Advances in Neural Information Processing Systems. 2018.
[4]: Li, Shuang, et al. "Learning temporal point processes via reinforcement learning." Advances in Neural Information Processing Systems. 2018.
[5]: Neitz, Alexander, et al. "Adaptive skip intervals: Temporal abstraction for recurrent dynamical models." Advances in Neural Information Processing Systems. 2018.
review_title: Official Blind Review #3

Meta-review: This paper studies the setting in reinforcement learning where the next action must be sampled while the current action is still executing. This refers to continuous time problems that are discretised to make them delay-aware in terms of the time taken for action execution. The paper presents adaptions of the Bellman operator and Q-learning to deal with this scenario.

This is a problem that is of theoretical interest and also has practical value in many real-world problems. The reviewers found both the problem setting and the proposed solution to be valuable, particularly after the greatly improved technical clarity in the rebuttals. As a result, this paper should be accepted.
Decision: Accept (Poster)