Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a Locally Linear Q-Learning (LLQL) method for continuous action control. It uses a short-term prediction model and a long-term prediction model to generate actions that achieve short-term and long-term goals simultaneously. The problem the paper seeks to solve is important. However, this paper has several issues.

First, there seems to be an over-claim of the contribution. The proposed method is more like hybrid of model-based and model-free RL method. Specifically, the short-term prediction model is in fact the linearized dynamic system with system parameters modeled by deep neural networks, while the long-term prediction model is in fact different (state- and action-) value functions. For this reason, it is probably unnecessary to name them as “short-term network” or “long-term network”, since they are simply the system model (or the model-based part) and the value functions (the model-free part).

Second, the proposed method is not sufficiently evaluated. It is only evaluated on the toy Mountain Car task (and the Crane system in supplementary material). In order to justify the performance of an RL algorithm for continuous action space, it should be at least evaluated on the set of MuJoCo tasks. 

Third, the proposed method is not sufficiently compared with different baselines. In Figures 3-5, the proposed LLQL algorithm is never compared to any baseline method, leaving it open whether it is actually better than earlier methods like DDPG. In Table 1, LLQL is compared to DDPG (a model-free method), and is shown to achieve better performance. However, this seems to be unfair because the proposed method is in fact a model-based RL algorithm. Therefore, it should at least compare to other model-based algorithms (and also other riche set of safe-exploration RL methods).

Other comments:
•	In eqn. (6), \gamma should be \gamma^{i-k}?
•	In the paragraph after (8), “Q-learning algorithms (16)…” is referring to a wrong equation (16) for Q-learning. Or probably the authors are not using the correct format to cite the reference. (This seems to happen repeatedly in later part of the paper such as as in the paragraph between (14) and (15).) It confuses the equation number and the reference number.
•	More explanation should be given about d(x_k|\theta^d) and h(x_k|\theta^h) after (15). The meaning of them has never to defined before.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Thank the authors for the response. I will keep my score.
----------------------------------------
Summary
In this paper, the author proposes LLQL (Locally Linear Q-Learning), which separates the action design from prediction models, such that the model can have short-term and long-term goals. The short-term network takes the current state and the action, and predicts the next state (to be specific, the status change) under the assumption that the output is linear to the action. The long-term network takes the current state as input, controls the controller to take action, and also optimizes a Q-function. I lean to vote for accepting this paper, though the experiment part can be further improved.
Strengths
- The proposed model is novel, which can take a trajectory or a constraint as a short-term goal. As a result, we can control the behavior of the model easily. Otherwise, we have to design a new reward function to guide our model indirectly. As the authors showed in Tables 1 & 2, LLQL outperforms this approach.
- The paper is clearly written. The author provides clean formulas throughout the paper, which makes the paper easy to understand.
Weaknesses
More experiments can be conducted to demonstrate the effectiveness of LLQL.
- Only tested in one scenario. In the main text, though it is already very long, there are still only experiments in one scenario (i.e., Mountain Car).
- Only compared to one baseline. In order to demonstrate the effectiveness of LLQL taking a short-term trajectory or constraint, the authors compared it to only one method (i.e., DDPG) with a few manually designed reward function.
Minor
Equation 6 looks incorrect to me.
Possible Improvements
As mentioned before, it would be great to include more experiments in the main text, comparing LLQL to more baselines, testing LLQL in more scenarios, or doing some ablation studies (removing one part of LLQL to demonstrates the effectiveness of that part).
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes a method for combining continuous Q learning and linear MPC. They propose learning a continuous control policy for long term behavior trained with standard model free reinforcement learning and a short term linear control model implemented as a linear dynamical system. They propose learning them separately each with its own loss function. At test time a combined controller can be used to infer actions by solving small optimization problem. The linearity makes inference very efficient and even when additional constraints are added only at test time. The model should therefore me more flexible and allows for adapting 0 shot when the constraints are known. The paper is well written and mostly clear. 

The equations seem correct but I am not an expert in continuous control. The examples considered seem a bit too simple to be insightful. 

* the algorithm seems to be be setup to with the dynamics model being learnt on the transitions coming from an epsilon greedy with respect to the optimal policy. Does this mean that generalization may be affected if the model strays off the optimal trajectories ?
* There would be some benefit in trying the model in some other domain maybe reacher with some constraints ?


review_title: Official Blind Review #2

Meta-review: This paper proposes a reinforcement learning algorithm for continuous action domains that combines a short-horizon model-based objective and a long-horizon value estimate.  The stated benefits to this approach are the ability to modify the model-based objective without extensive retraining.  This model-based objective can also capture custom constraints and implements a linear dynamics model that is used in conventional control theory.  The proposed method was evaluated on two domains (the mountain car domain and a custom crane domain) and compared to a continuous action space method (DDPG).  

This discussion of this paper highlighted both strengths and weaknesses.  The reviewers said the presentation was clear.  The reviewers also appreciated the relevance of the problem.  The primary weakness was the evaluation of the method. One repeated concern from the reviewers was having only one standard domain for evaluation (mountain car).  Another concern was the absence of other model-based algorithms, which was addressed by the author response.   

This paper is not yet ready to be published, despite its possible benefits,  due to the lack of evidence for this method on more continuous action problems.   
Decision: Reject