Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Thank the authors for the response. The major novelty of this paper is encoding the objective as a logical expression and the experiment part is limited. I will keep my score.
----------------------------------------
Summary
This paper presents a new approach for MORL (Multi-Objective Reinforcement Learning), which handles multi-objective as a pre-defined logical language, and estimates Q-function by using UVFA. This approach can handle the case when the final objective is not a linear combination of the individual objectives. It feeds the specification into a GRU (Gated Recurrent Units) as an encoder, concatenate the encoding with the state, and then send it to the Q-function, estimated by a UVFA. I am kind of on the borderline, but still lean to reject this paper. I am happy to change my score based on the reviews from other reviewers.
Strengths
- The idea of modeling multi-objective as a logical language is novel. By describing the final objective in a logical language, many more cases that are not linear can be covered.
Weaknesses
- Lack of baselines / experiments. It seems the only baseline in this paper is agents trained on a single policy, i.e., no baseline from previous works. It is possible to compare the performance with previous works when the final objective is linear to each individual objective.
- Only tested on one simple scenario. More scenarios can be included to justify the effectiveness of the proposed approach (e.g., Deep Sea Treasure, SuperMario, etc.).
Possible Improvements
As mentioned before, more baselines and scenarios can be included.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors propose to tackle the problem of multi-objective reinforcement learning (MORL) by considering a logical function as reward signal. In their proposed solution, the logical function is also encoded and concatenated with the state. They argue, via simulation and toy examples, that the proposed model is able to generalize to logical formulas of the multidimensional rewards that has not observed during training.

The combination of logic and machine learning is certainly an interesting direction. However, the current contribution is rather limited in terms of methodology and shallow in terms of experimental evaluation. As a result, I do not support acceptance. More specifically:

1. The only methodological novelty of proposed contribution is the idea of encoding the multi objective reward as a logical function. As a result, the experimental evaluation should be much more thorough.

2. The general claims made by the authors are not really supported by the experimental evaluation. In particular, many details of their experimental setup are missing (e.g., the experiments use 50,000 specifications, however, no stats about these specifications are given), the experiments are mainly about performance rather than exploring the encodings, and thus it is difficult to judge whether the proposed solution is actually meaningfully encoding the logic and achieve generalization.

3. In the experiments, the objectives stay on the road, avoid hazards and move right are not clearly specify mathematically. What is the range of o1, o2, o3? o1, o2, o3 are sometimes used in inequalities, sometimes they are used as Boolean variables. The authors should more clearly explain this.

Minor comments: "agent for use with" -> "agent to use" (Just a couple of paragraphs before Background)
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes using logical specifications to facilitate Q-learning in multi-objective reinforcement learning (MORL). Empirically the proposed method can generalize to unseen reward specifications with performance competitive to agents being fully trained in the new environment. The proposed setting employs a more expressive objectives space induced by propositional logic. The proposed method uses a recurrent encoder to embed specifications into vectors and uses them to parametrize the Q-function. 

Overall, I weakly recommend accepting the submission for the following reasons: (+) it proposes using propositional logic to specify reward functions, which broadens the objective space in an interpretable way, (+) the learned objective embedding demonstrates the ability to generalize to unseen environments(rewards). However, there is still room for improvement. I will raise my score if the following problems are addressed: (-) Needs more diverse experiments (instead of grid worlds) to support the paper (-) It might be hard to express objectives using logic formulas in real-world applications.

More specifically, the problems of the paper are,

(-) The scalability issue. The most complicated problem in the experiments has a 20x20 state space, which is pretty small for a typical RL problem. I wonder whether the model is still generalizable for larger problems. Even in the case of 20x20 grids, we can notice the gap between the baseline and the proposed method. 

(-) The assumption. This paper assumes the logical specifications are given by human. However, we usually don't know how to describe the true objective with logic formulas. Sometimes, finding the specification(reward) itself is as difficult as finding a good policy. Is it possible that we can relax this assumption?

Minor comments:
Figure 6: The word "baseline" here is misleading. A better choice would be "upper-bound"?
review_title: Official Blind Review #2

Meta-review: The reviewers generally agreed that the technical novelty of the work was limited, and the experimental evaluation was insufficient to make up for this, evaluating the method only on relatively simple toy tasks. As much, I do not think that the paper is ready for publication at this time.
Decision: Reject