Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This manuscript investigates the setting where there exists a mismatch between the training MDP and the testing MDP (known as robust RL). While robust RL is quite conservative in that the adversary could minimize the protagonist's utility over every MDP in the uncertainty set, this manuscript believes that it is too conservative. Instead, the manuscript defined a feasible set of MDPs that the adversarial could achieve, by asserting that there must exist a policy in that adversarial MDP to achieve at least $\lambda$ in terms of the return. It subsequently lifts the constraint by replacing it with a substantially large penalty if the support of the random MDP of the adversarial has a positive density on the non-feasible area. With this setting, the robust RL policy and the adversarial could be trained by PSRO. Simulations are conducted on gridworld and 3 MuJoCo tasks.
strength_and_weaknesses: I agree that robust RL is indeed too conservative and the new feasible uncertainty set is relevant. The new setting proposed by the manuscript makes sense to me. It is also interesting that the setting could be solved by PSRO in quite an intuitive way. Some detailed reviews follow.

Pros:

1. The new setting is an interesting extension towards robust RL
2. The proposed algorithm through PSRO is intuitive and sensible.
3. Presentation of this manuscript is easy to follow.

Cons:

The major concern of mine is how useful this method could be, in practice. In the experiments, the agents that are trained against the *feasible adversary* demonstrates some decent performance in only 3 MuJoCo tasks (I have to assume that it doesn't pan out on the rest of the tasks). Notice that this is when the algorithm trained against the feasible adversary is tested against the feasible adversary. What if the agent trained against the feasible adversary is tested against a general adversary (potentially described by a real application task)? The manuscript does not provide such an answer, but I'm not very confidence on this.

Misc:

In the abstract "In real-world environments, choosing the set of possible values for robust RL can be a difficult task", the term "value" could be confused with the value function of RL. Use a different term.
clarity,_quality,_novelty_and_reproducibility: The manuscript has a great presentation and is enjoyable to read. The clarity and quality are excellent and the novelty is significant due to the introduction of the new robust RL setting. I did not check the reproducibility.
summary_of_the_review: I'm positive on this manuscript for its novelty in the new setting and its quality in presentation. I have reservation, though, on its empirical performance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on generalization performance in reinforcement learning: when applying the Robust Reinforcement Learning approach, it is important to know how to design the set of environmental parameters to be considered. The authors propose a method to approximate the Nash equilibrium solution in a zero-sum game, called Policy Space Response Oracles (PSRO), which is based on the Nash equilibrium solution in a zero-sum game. By applying a method for approximating Nash equilibrium solutions, called Policy Space Response Oracles (PSRO), a method is proposed for finding policies that maximize worst-case performance for a restricted set of environmental parameters. Experimental results show that higher utility can be achieved in a simple Grid World-like task and in a task using MuJoCo, compared to the case where Domain Randomization, Maximin, and Regret are used as measures.
strength_and_weaknesses: # Strength: 

Approach: How to design the set of uncertainty parameters to be considered is a very important issue in robust reinforcement learning. Introducing a threshold for the utility and limiting the set of uncertainty to the parameters where the best possible performance is better than or equal to the threshold is a simple and natural approach. This paper proposes an approach to achieve this goal and show its usefulness in experiments. 

Test Task Design: The test problem designed in this paper is, I think (not claimed by the authors), a good task revealing the issue of domain randomization, maximin approach, and regret minimization approach. It is very simple and easy to understand. 

Evaluation: Numerical experiment on the test problem reveals the advantage of the proposed framework over domain randomization, worst-case maximization, and regret minimization. 

 # Weaknesses
Min-Max vs Max-Min: In the proposed framework, the environment parameters where the optimal policy can reach the utility above the given threshold are considered as feasible parameters and the worst-case maximization is performed on such a restricted environment parameter set. This does not guarantee that the optimal policy for the worst-case in the restricted environment parameter set is above the threshold in general, because min-max is not equal to max-min. This limitation is not evaluated nor discussed in this paper. 

Computational Time: In the experimental evaluation, the performance of the approach is displayed against PSRO iterations. Therefore, it is unclear how many interactions have been performed in the approach. 

Evaluation (in particular on MuJoCo): The usefulness of the proposed approach is evaluated only on the basis of PSRO. However, because domain randomization, worst-case maximization, and regret minimization are proposed in conjunction with other baseline RL approaches, the usefulness of the proposed approaches over existing robust reinforcement learning approaches are not evaluated.

Regret minimization vs proposed approach: In Section 5.1, the authors say "The regret objective produces a distribution of both feasible and infeasible goals where non-lava goals are the majority, however this distribution is not an adversarial robust NE with respect to the set of feasible goals, so worst-case performance with the regret objective is still not optimal." It is understandable that the authors want to say that the optimal min-max regret solution is not the optimal min-max in the feasible environment. However, it is not clear why this is the case in this task. Please provide more explanation in relation with Theorem 1 or 2 of PAIRED paper. I feel that it is the matter of the magnitude of the reward. I guess min-max regret and the proposed formulation is the same if the penalty of lava is greater. At the same time, I understand that it is not always the case that there is a clear notion of success and we would like to have a good generalization ability in the set of environments where the optimal utility is sufficiently large. More explanation would be nice.

clarity,_quality,_novelty_and_reproducibility: This paper is well-motivated and clearly written. The idea of using the threshold for the utility to define the uncertainty set is a natural approach, yet the approach is, to my knowledge, novel. However, it is largely based on the existing approach, PSRO, and its novelty is limited. The details of the experiments are given in the main part and in the appendix. Unfortunately, its code is not publicly available. 
summary_of_the_review: Based on the above mentioned comment, I suggest weak accept. Although there are weak points, it has a potential to be usefulness in practice.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work focuses on the robust reinforcement learning paradigm, whereby an agent is trained to be robust to worst case variations in an environment. The paper proposes an approach called FARR, whereby the adversary (proposing variations) is penalized if the environment proposed is not solvable. The paper sets out with great promise, working in an interesting area, but falls short with implementation details and experiments.
strength_and_weaknesses: ### Strengths

* The idea is sensible and well motivated, this is clearly an important area for real-world RL and has received increased interest recently with methods such as PAIRED, PLR and ACCEL. 
* The nice thing with this in principle is it allows the adversary to propose the hardest solvable environments, although that was not really demonstrated anywhere. 


### Weaknesses

* The entire paper relies on an assumption that we have access to the feasibility of an environment, which is almost never the case in any of the practical settings where this may be used. The authors attempt to sweep this under the rug by saying "we note that, in order to evaluate our approach in this paper, we focus on environments where we can, in fact, calculate (the feasible region)". This is remarkable to me. The entire paper is focused on the idea of being able to learn a good policy with unfeasible regions where the feasibility is unknown, but then in the experiments we assume access to feasibility. This completely invalidates the entire point of the work, and in fact, makes me even more impressed by the performance of regret in the experiments which comes close despite not having extremely privileged information. 
* The paper also implies that the reward penalty is fed back into a learned adversary which can effectively infer feasibility. Sadly this is also not true as the authors instead use random sampling. 
* Given that the current method relies on random sampling, it should also compare against Robust PLR (Jiang et al, Replay Guided Adversarial Environment Design, NeurIPS 2021) which is a far stronger method empirically than PAIRED. 
* The experiments are very toy, using HalfCheetah and Walker and then a tabular MDP. The field has moved on to much larger settings since the original RARL paper in 2017. At minimum it should be considering more environments, something like the RWRL suite from pixels would make sense. This is not even super expensive to run. 
* There is no analysis on the different types of emergent environments proposed by the different methods. This is crucial given toy experiments!
* What makes all of this even worse is that there is no limitations section where it is acknowledged that the paper doesn't do what it states out to do in the intro. If someone new to the field read this they may be led to believe FARR could infer the feasibility and that it was superior to regret based approaches. Please revise this before putting the paper on arxiv etc, even if unable to address the other issues. 
clarity,_quality,_novelty_and_reproducibility: The novelty is that this applies a reward penalty for infeasibility rather than using regret which uses the performance of another agent (approximating the optimal policy).
summary_of_the_review: The idea makes sense and is well motivated. I was assuming from the intro that the authors had figured out how to learn to predict the feasibility of an environment, which seems like the hardest part of getting this to work. Unfortunately instead the method relies on knowledge of the feasibility which is not possible in any large scale environment we would care about. Further, baselines like regret perform well and don't need this. 

Not only that, but FARR has no theoretical results (vs. regret-based approaches which do), and the experiments are low dimensional and toy, with no analysis on the different environments proposed by the adversaries. This paper makes sense as v0 of an idea for a workshop but is far below the bar for ICLR. 

I would definitely provide a higher score if the authors figure out how to learn feasibility, provide greater analysis of the emergent environments proposed and also a greater breadth of tasks. This seems beyond the scope for a two week discussion phase but hopefully is possible for the next conference cycle.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the robust reinforcement learning problem where the set of feasible environmental parameters are restricted. The authors propose the Feasible Adversarial Robust RL (FARR) formulation to this problem which specifies that the feasible parameter set is the set where the expected return of the protagonist agent is not less than a certain threshold. Essentially being a min-max game, the authors then solve the FARR problem using the Policy Search Response Oracle (PSRO) algorithm. 
strength_and_weaknesses: Strengths: 
1. The problem of robust optimization with feasible parameter set is interesting and practical. The setting is well-motivated.
2. The line of related work seems to be well-discussed. 

Weaknesses:
1. Although the problem is well-motivated, the formulation is somewhat trivial since it requires prior-knowledge of the feasible set of parameters. This is either directly fed as prior-knowledge, or using the penalty coefficeint C (which itself is unknown and questionable) as well as the threshold $\lambda$ for determining the feasible set. This makes the practicality of the algorithm very limited. In some sence, if one treats it as prior knowledge, the problem reduces to the same standard robust RL problem where the adversary's feasible set of parameters is specified. 
2. Because the essense of the formulation is still aligned with a typical robust RL setting, the algorithms for robust RL, like PSRO, are still applicable to the FARR formulation. And the methodological/technical contribution is therefore very limited. 
3. In evaluation, the solution based on the FARR formulation is better than the baselines. This is not surprising since the baselines are not fed with the feasibility information that the FARR formulation has.
4. The feasiblity definition is a bit contrived in the sense that it essentially removes all the environments where the expected return for the protagonis agent is small (than a threshold). First of all, the threshold is not always known, or sometimes unclear. Second, it is not equivalent to "feasible". In practice, many feasible parameters may yield very low return, but you cannot just ignore these parameter. 
clarity,_quality,_novelty_and_reproducibility: Clarity: good. 
Quality: fair. 
Novelty: low. 
Reproducibilty: high.
summary_of_the_review: My main concerns are summarized at the weaknesses part of the above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers agreed that the paper considers an interesting setting for robust reinforcement learning and proposes a novel approach for defining uncertainty sets. However, the reviewers pointed out several weaknesses in the paper and shared common concerns. We want to thank the authors for their detailed responses. Based on the raised concerns and follow-up discussions, unfortunately, the final decision is a rejection. Nevertheless, the reviewers have provided detailed and constructive feedback. We hope the authors can incorporate this feedback when preparing future revisions of the paper.
justification_for_why_not_higher_score: The reviewers pointed out several weaknesses in the paper and reached a rejection decision.
Decision: Reject