Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper considers the problem of learning a policy for an MDP with unspecified reward, given user-provided goal states. To this end, a reward model and a policy are jointly learned: the reward model is the conditional normalized maximum likelihood (CNML) learned from a training set consisting of the example goal states as positive examples, and the policy trajectories as negative examples; the policy is trained to optimize the MDP using the learned reward. Meta-learning is applied to reduce the cost of learning the CNML models.

Pros
+ The idea of using CNML to obtain a smoother reward as compared to a single model (together with the efficient meta-learning approximation) is interesting.
+ The algorithm is compared with several baselines and seem to perform well. Ablation study suggests that goal examples and meta-learning are important for the proposed approach.

Cons
- The paper is unnecessarily hard to read. A high-level description of the approach early in the paper will be helpful. This is also related to the comments below: it is not clear why the algorithm should work.
- A claimed contribution of the paper is to "produce more tractable RL problems and solve more challenging classes of tasks". The limited feedback provided by the example goal states perhaps make the RL problem more tractable, but how is it possible to solve more challenging classes of problems when less information is available?
- To learn a useful reward model from the example goal states, the CNML approach alone seems insufficient, and it seems necessary to require a good reward to be a smooth function of feature vectors. For example, if we work in a grid world with random rewards, does the approach still work?
- Both the paragraph before Sec 3.2 and Alg 1 mention that the set of negative examples keeps growing. This implies that the reward model will become more and more sparser (values closer to zero), even for the goal states? How is such a reward model still useful?
- Another question about the reward model is that when the policy becomes better, it is more likely to reach the goal states, thus the goal states are more likely to be labeled as both positive and negative. Thus the reward model is more likely to assign lower reward to goal states when more training is done?
- Fig. 1 seems to be overstating the problem with MLE. What are the features used and what is the classifier model? If the feature is the real-valued position, and a regularized logistic regression model is used, then MLE will not produce such a sparse reward as in (b)?
- The experiments section should provide more details about the experimental setup: the choice of candidate classifier models, explanation of the baselines (e.g. Sparse Reward seems not mentioned in the text at all), detailed description of the performance evaluation metric. Is Manhattan distance to goal a sensible performance metric for maze navigation?

Minor comments
- "OpenAI et al.": wrong citation format
- Define L in Eq. (2)

Post-rebuttal
After reading the rebuttal and other reviewers' comments, my score remained the same. The rebuttal helped to clarify some issues, but it is still not clear to me why the algorithm should work. I agree with other reviewers that a more careful revision of the paper, and a further analysis on the algorithm will be beneficial.
review_title: some interesting ideas but require deeper analysis

final_score: rating: 4: Ok but not good enough - rejection
review_content: This manuscript aims to solve reinforcement learning problems where the reward is unknown but a set of successful states are available. Iteratively, it trains a classifier using provided successful states as positive and on-policy samples as negative and use its predictions as the reward function to learn RL policy. 

It may be worth clearly explaining the connection and improvement on the work of Fu et al.(2018b, Variational inverse control with events: A general framework for data-driven reward definition) in introduction since both mainly solve the same problems. Fu’s paper first introduced the event framework to generalize inverse RL that also solves the sparse RL problems by iteratively training a classifier to predict the probability on successful states.

Compared to previous work, it seems a major difference in this paper is to change the event classification model, replacing the neural network classifier with CNML classifier. It could enhance the contribution if  good theoretical analysis can be provided. E.g. why the manuscript concludes that CNML performs better than standard neural network clasisifers in terms of uncertainty aware on reward estimation, and how CNML model connects with better exploration and goal-oriented reward shaping.

Other comments are written below.

Theorem 4.1 is defined, but it seems not used in the following sections. It would be helpful if the following can explain whether it is used  to develop the algorithm or explain empirical findings.

It may need more evidence in Section 4.3 to support the conclusion that the CNML is doing better at reward shaping. It is not easy to make a conclusion based on some specific example in Figure 1. Similarly Section 4.1 also try to show CNML gives reward that can improve exploration and becomes goal oriented using the same example.

Section 5.1 describes how to use meta learning to approximately solve CNML problem so it can reduce computation complexity. It is an interesting idea, but it may need more analysis. Some questions are listed below.

- The distribution of tasks usually has different samples, while the CNML problem has similar data sets for its tasks as each task only adds one sample to the original dataset. Does this difference have influence on learning the model parameter in meta learning?

- It would be interesting to analyze the computation complexity as it is the main reason for applying meta learning in solving CNML.

In experiments, It may be interesting to test the CNML estimation with full NN convergence on entire augmented dataset every time, to show the trade-off between accuracy and computation complexity when compared with meta learning.

Based on the number of epochs, it seems BayCRL converges slower in some problems, such as zigzag, spiral, sawyer. It may be interesting to give some insights, as good reward may speed the learning process. Moreover, it may be more accurate to consider the time complexity in each epoch when validating the time complexity.

On page 7, it may need explanation on how to find that BayCRL outputperforms in terms of sample comcplexity as shown in Figure 4.

Some minor comments:
- On page 5, Appendix Appendix A.5 -> Appendix A.5 
- It seems Appendix A.2 duplicates Section 5.1, particularly the Eq. 6 and the equations for p_meta-NML and \theta_y.
- Is there any reason for line 7 in Algorithm 1 that skips meta learning.

review_title: This paper aims to better address a known RL problem that learns reward from a set of successful events

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
-------

This paper addresses a reinforcement learning problem where the reward
function is learned through a classifier that decides whether states are
successful or not based on previous examples (i.e. RL after inverse RL).
The authors show that this requires uncertainty-aware predictions, which
are difficult with neural networks. An algorithm, BayCLR, is proposed
that uses MAML to meta-learn the conditional normalized maximum
likelihood, i.e. the "maximum likelihood distribution". Connections to
the proposed algorithm and exploration methods are discussed before
using the algorithm to solve various robotics tasks.

Decision
--------

Although I liked this paper overall, I am rating it tentatively as
marginal below the acceptance threshold. The paper is very well written
and addresses a relatively clear problem (inverse RL with classifier)
with an interesting method (meta-learning CNML). I have some issues with
unclear statements in the motivation and method that should be
addressed. While the experiments provide some insight, I think the
conclusions the authors draw from them are far stronger than the results
imply.

Originality
-----------

I am not very familiar with CNML, but this paper seems very original. In
particular, the application of meta-learning to conditional normalized
seems novel, as well as its application to inverse RL.

Quality and Clarity
-------------------

The paper is well-written, most statements are clear and easy to follow.

Strengths
---------

-   The approach of meta-learning CNML is interesting, and if anything
    deserves further analysis outside of inverse RL / RL.
-   Although I have some issues with the motivation (see below), I think
    the authors do a good job of explaining their rationale for using
    CNML. In particular, quantifying neural network uncertainty through
    posterior analysis is quite difficult.
-   The experiments seem comprehensive. However, I am not very familiar
    with the environment suite used. Due to the lack of work in this
    area, there are not many possible baselines, and so the VICE
    baseline seems like the best choice. The baseline is further kept
    fair by adding exploration heuristics. I especially like Section 6.2
    that analyzes BayCRL on the zigzag maze task, which I assume is a
    tabular environment. In addition, there is a simple ablation but I
    would prefer more work on this.

Weaknesses
----------

-   In Section 4, some motivating statements are unclear to me (see
    Detailed Comments).

-   A single data-point being added to the dataset may not change the
    distribution much, and this crucial point is only addressed (in an
    ad-hoc way) in the appendix. It would be good to see an ablation
    study on this, or perhaps a plot of the average difference between
    different query points. Perhaps Figure 6 may indirectly explain
    this, but it should be explicitly addressed.

-   In Section 6, many statements about BayCLR seem stronger than the
    results imply. Many confidence intervals overlap, and the results
    themselves are hard to parse with so many lines in each plot.
    Perhaps some of these baselines which are not competitive can be
    excluded, or perhaps different linestyles should be used.

Detailed comments
-----------------

-   Early mentions of exploration (i.e. in the abstract) seem out of
    place. While you elaborate on the connection to exploration methods,
    it does not seem like the main point of this paper is to address
    exploration. It seems to me that you are tackling a novel RL problem
    where the task is specified through goal states. Specifically, you
    learn classifier in place of a reward function, and this is
    exploited to shape an otherwise sparse reward.

-   Section 4.1, "To create effective shaping, we need to impose a prior
    on our classifier so that it provides a more informative reward when
    evaluated at rarely vis- ited states that lie on the path to
    successful outcomes."

    Why is a prior strictly necessary for reward shaping? Unless you
    mean prior in a very general sense, not a Bayesian prior, I don't
    see why a prior is strictly necessary.

-   Section 4.1, "\[CNML\].. is essentially imposing a uniform prior
    over the space of possible outcomes". This is not obvious to me, and
    perhaps further explanation is needed.

-   Section 4.2, Theorem 4.1: Perhaps I misunderstand but if $G(s) > 0$,
    shouldn't $p(e = 1 | s) = 1$? Further, why is that when the agent
    visit a successful state, i.e. $N(s)$ increases, $p(e = 1 | s)$
    decreases after each visit.

-   Section 5.1, "This algorithm, which we call meta-NML, allows us to
    obtain normalized likelihood estimates without having to retrain
    maximum likelihood to convergence at every single query point, since
    the model can now solve maximum likelihood problems of this form
    very quickly. "

    Is it correct to say that meta-NML does not need to retrain to
    convergence at every query point? The second part of the sentence
    elaborates that metatraining allows you to solve the problem very
    quickly, but it seems that you still need to solve it at every query
    point.

-   Section 6.2, Figure 4: I don't think its fair to say that BayCLR
    performs substantially better. The confidence intervals overlap in
    all but Spiral Maze and Sawyer 3d Pick-and-Place. Other subtleties
    are not addressed, such as why RND/count-bonus actually hurt VICE in
    sawyer 2d push. Other statements such as "significantly more
    efficiently" need explanation as well.

-   Section 6.3, Figure 5: for reproducibility, you should include
    exactly how many gradient steps are used in the model without
    meta-learning.

-   Section 6.4, Figure 6: I'm unfamiliar with the environment being
    used, so some additional details explaining what z means would be
    helpful.

    "Furthermore, meta-NML is able to reasonably approximate the
    idealized NML rewards with just one gradient step…"

    How is this shown in Figure 6? I don't see anything showing the
    idealized NML rewards.

Minor Comments
--------------

-   Section 4, Line 3: missing space: "ples.For example,"

Post Rebuttal
--------------

After reading the comments by the other reviewers, I have decided to keep my score at a 5. The authors reply, and the updated manuscript, helped my understanding of the paper. I was considering raising my score, however, the reviewers were nearly unanimous in their confusion regarding the framework or application of CNML. For future iterations of the paper, I suggest that the authors describe CNML, event-based control and their connection more explicitly. If the main contribution is using CNML as the classifier in event-based control, then it would also help to conduct experiments on meta-learning CNML in a supervised learning setting to further elucidate its effectiveness in the reinforcement learning application. I think your paper is very interesting, and I hope that the authors are able to use this feedback to improve their paper.
review_title: An interesting paper, but some concerns regarding motivation and experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies how to solve RL problems with a set of success states instead of a standard reward function. The central idea is to firstly train a Bayesian classifier from both the input success examples and the on-policy sampling using the conditional normalized maximum likelihood (CNML) and then use the learned classifier as a reward function to guide exploration. It is proved that in a tabular case, the success classifier trained with CNML is equivalent to a version of count-based exploration and it is claimed that with function approximation, the classifier attains non-negligible generalization. Empirically, it is claimed that this approach outperforms existing algorithms on a number of navigation and robotic manipulation domains.

The novelty of this work lies in the use of CNML to train a more regularized success classifier and further use meta-learning to implement CNML in practice. 

There are several concerns I have:
1. Can the authors indicate the following information in the experiment:
- Which algorithms are provided with success examples as prior knowledge in all testing domains? The descriptions in Sec. 6.1 and 6.2 are a little confusing.
- In Figure 4, I didn’t see the lines for VICE+count-bonus.


2. The claim that BayCRL outperforms uninformed, task-agnostic exploration (VICE+count-bonus and VICE + RND ?) is not surprising since the former has prior knowledge.


3. The authors claimed that the proposed approach can achieve both effective reward-shaping and exploration. I agree with the first point by comparing it with other IRL methods. But how is the latter true? I think this needs to be further demonstrated in a different setting such as the one in the next point.


4. All the tested domains have only one success state. Thus, the example set is informational complete. I wonder about the robustness of the algorithm if the example set does not contain all success states while a reward function (which is surely super sparse) is provided. For example, if there are 10 success ground-truth states and only 5 are provided in the example set and the uncovered states are quite remote from the provided ones (but a reward function is available, i.e., if we reach these hidden success states, a positive reward will be received), then how would this degenerate the performance of the proposed approach comparing with other methods? I think this also testifies the generalization/exploration ability of the algorithm from another perspective.

I vote for weak reject since both CNML and MAML including the reformulation of the problem follow the prior works, which kind of limits the novelty of this paper as applying known algorithms to a defined problem. But I am open to adjusting the score if the rebuttal can address my concerns.

review_title: Review

Meta-review: Summary:
This paper introduces a method to try to learn in environments where a person specifies successful outcomes  but there is no environmental reward signal.

I'd personally be interested in knowing where people were able to easily provide such successful outcomes instead of, for instance, providing demonstrations or reward feedback. Similarly, I'd be interested in how other methods of providing human prior knowledge compared.

Discussion:
Reviewers agreed the paper was interesting, but none of the 4 thought the paper should be accepted.

Recommendation:
While I do not think this paper should be accepted in its current form, I hope the authors will find the comments and constructive criticism useful.
Decision: Reject