Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper discusses the old problem of mismatch between the ultimate reward obtained after optimizing a  decision (planning or control) over a probabilistic model (of dynamics) and  the training  objective for the model (log-likelihood). Experiments highlight that the NLL and reward can be very poorly correlated, that improvements in NLL initially improve reward but can later degrade it, and that models with similar NLLs can lead to very different rewards. A  reweighting trick is proposed and summarily evaluated.

I like the topic of this paper but there are several aspects which I see as making it weaker than my acceptance threshold.

First, the paper overclaims in originality. This mismatch problem is not new, it is an instance of a more general issue that end-to-end training and meta-learning try to address, and has been already studied in the context of MBRL by many authors, who actually proposed more substantial solutions. When I read the abstract I had the impression that the paper actually had a theoretical analysis showing the correlation problem, but there is no such thing, only experiments. Section 3 does not actually provide a new insight. Still, the experiments are interesting in that they reveal that the magnitude of the mismatch is probably more serious than most RL researchers believed.

Second, the 'fix' proposed is not well justified nor well tested (e.g. no quantiative comparisons, no comparisons against existing alternative methods to address the same problem, etc). This seriously weakens conclusions like "shows improvements in sample efficiency".

One concern I have about the experiments of fig 3 is that NLL can be really bad, thus distorting rho, which is not a robust measure. So I would only look at NLLs of models with good NLLs, to obtain a more interesting analysis.

Another concern about experiments is that I am not convinced that they were performed with SOTA MBRL methods and hyper-parameters (as demonstrated by SOTA performance on known benchmarks). Otherwise I could easily imagine how the mismatch could be much more severe than in the actual scenarios of interest.

Minor points:


Bottom of page 6 refers to visualizations but I did not see if or where they were shown.

Why the e in the numerator of eq 2e? Seems useless to put any constant there.

The section on 'Shaping the cost or reward' was not clear enough to me (please expand).

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper claims that it identifies a fundamental issue in model-based reinforcement learning methods. The issue is called objective mismatch, which arises when one objective is optimized (for example, model learning objective) without taking into consideration of another objective (for example policy optimization). The author shows several experiments to illustrate the issue and proposes a method to mitigate it by assigning priorities to samples when training the model. 

The issue of objective mismatch is not being noticed the first time. The paper "Reinforcement learning with misspecified model classes" has mentioned similar phenomenon. And other work such as https://arxiv.org/abs/1710.08005 also discussed similar issue. 

I disliked the way how the paper is motivated. The paper says “in standard MBRL framework”, what is the standard MBRL framework? I think there is no such standard so far. The claim saying that the mismatch is a crucial flaw in current MBRL framework is too strong. The paper at least completely ignored two broad classes of MBRL methods. The first is value-aware MBRL (which attempts to take into account decision making when learning a model), there are actually many MBRL methods are in this category. Some examples: value prediction network, Predictron, Value-Aware Model Learning. The second class of MBRL approach is Dyna. Several works (continuous deep q-learning with model based acceleration, Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domains, Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation, Recall Traces: Backtracking Models for Efficient Reinforcement Learning, Hill Climbing on Value Estimates for Search-control in Dyna) show that even with the same model (hence the same model error), using different simulated experiences play a significant role of improving sample efficiency. It is unclear whether model-error plays a decisive role in improving sample efficiency. 

The proposed method in section 5 lacks of justification, even a regular ER buffer is asymptotically on-policy, and hence it is indirectly linked with the control performance. It is unclear why introducing the weights based on expert trajectory can be helpful — it can be worse because it should be far away from on-policy distribution. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper "OBJECTIVE MISMATCH IN MODEL-BASED REINFORCEMENT LEARNING" explores the relationships between model optimization and control improvement in model-based reinforcement learning. While it is an interesting problem, the paper fails at demonstrating really useful effects, and the writting needs to be greatly improved to help reader to focus on salient points. 

From my point of view, the main problem of this paper is that it is too messy and it is very difficult to understand what authors want to show, as i) there is a very important lack of experimental details (e.g., main aspects of models and controllers should be clearly stated) and ii) analysis is to wordy, authors should emphasize the message in each part. From the experiments in 4.1, the only thing that I got is from the last sentence "noisy trend of higher reward with better model loss". are these results from LL computed on a validation set ? If not, this is not reallly meaningfull since high LL may only indicate overfitting. If yes, how was the validation data collected ? If the collection is not inline with training it is difficult to understand what we observe since we only need LL to be good on the path from the current to the opitmal policy, not everywhere. Even if the validation data is inline with training, there remains the difficulty of over-fitting in the policy area (for the on-policy experiments at least). Is there something else ?  From 4.2 we observe that it is unsurprisingly better to learn the model from the policy trajectories. From 4.3, we observe that an adversarial is able to reduce rewards without losing in LL. Ok, the adversarial is able to lock the controler in a sub-optimal area while still being good to model the dynamics elsewhere, but what does it show ?  Finally, proposal to cope with the identified mismatch are not clearly explained and not very convincing. Is re-weighting helping in collecting higher rewards ? 

From my point of view, this work is in a too preliminary state to be published at ICLR 
review_title: Official Blind Review #1

Meta-review: As the reviewers point out, this paper has potentially interesting ideas but it is in too preliminary state for publication at ICLR.  
Decision: Reject