Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents the issue of task underspecification in evaluating reinforcement learning algorithms. This issues emerges when an RL algorithm is evaluated on a single instance of an underlying (often implicit) task family, and then that performance is implicitly assumed to be robust to the choice of instance in the task family. The paper shows that this robustness is often not the case, and in some cases the ordering between different algorithms can change when different task instances from the task family are chosen. Recommendations for ensuring a more robust evaluation of algorithms are provided, although the core problem of the computational burden of robust evaluations still somewhat remains (and is seemingly unavoidable).
strengths_and_weaknesses: I think the issue of task underspecification, especially in the traffic control domain used, is a real one. The demonstration of this phenomena is valuable and significant. The paper is generally well-written and clear. I think the recommendations for more robust evaluation are beneficial, and the results in section 4 are interesting and beneficial to the research community, especially in the RL traffic control literature (which I'm less familiar with).

To me, the key issue arising from task underspecification is when the relative performance or ordering of different methods changes when making choices that were implicitly assumed not to change the relative performance or ordering. This is what is demonstrated very clearly in the traffic control domain, where the experiments demonstrate that when performing a more robust evaluation, the non-RL methods are preferred. Further, I think the results in observation 2 are unsurprising (different point MDPs are difficult); and we don't really care about absolute performance of algorithms on these toy benchmarks, as it's not that useful as information about how well they'll do on some other down-stream task. Performance relative to other algorithms (to enable an informed choice of which algorithm to use for a downstream task), or to a human expert on the task (so as to have a notion of whether the algorithm can outperform humans on some narrow domain), are more important.

Given this, it would be beneficial to highlight this precise phenomena in the other experiments. Figure 1 doesn't make it easy to see if choosing a different point MDP changes the preferred algorithm, or if it would also change if averaging over the performance across all point MDPs. More broadly, I'd prefer to see more of an emphasis and investigation of the possibility of this phenomena of reordering algorithms, for example another problem settings and with other algorithms. I imagine a key place where this could occur is in more complicated continuous control settings such as MuJoCo/DMControl/Brax. Investigating whether changing the physical parameters or agent morphology parameters (in a similar way to the experiments in section 2) also changes the SOTA algorithm would provide more evidence of the conclusions drawn in the paper.

While this paper focused entirely on different point MDPs within an MDP family, a common method of evaluation is to consider a task family, such as a variety of MuJoCo, Atari or Procgen tasks, to ensure that evaluation of the algorithm is robust. A comparison of the discussion in this paper to works with that evaluation protocol would be beneficial; crucially, demonstrating that the problem of non-robust evaluation demonstrated in the toy settings and the traffic control setting aren't just specific to those settings, and that evaluation on a range of (point MDPs from) different tasks solves the problem that's caused by task underspecification in terms of robust evaluation.

Overall, I'm currently recommending a weak reject, due to the fact that I don't think the scientific contribution of the paper is sufficient. I think the most interesting results are those presented in section 4, and if similar results could be demonstrated in other domains (as mentioned above) and with more algorithms, then I would raise my score to recommending acceptance. I realise this requires a high computational workload. If these results were demonstrated, and the recommendations for improving the robustness of evaluations were applied to them successfully, then I think the paper would be a robust and worthwhile contribution worthy of acceptance.

EDIT: I thank the authors for their diligent efforts in responding to my comments. Given the new experimental results seem to validate this phenomena across a sufficiently wide variety of environments, I'll raise my score from 4 to 6.

questions: What do you mean when you say that "not all parallel runs succeed" in footnote 1? Do you mean they don't converge? If so, I think it's more representative to also plot the failing (i.e. non-converging) runs also, so as not to give a skewed picture of the performance of different algorithms.

In figure 1, is each algorithm trained once on each MDP, and then tested on all the MDPs labelled on the x axis? Also, it would be good to see which random MDP is chosen in each case, as that would explain the performance of each algorithm trained on this MDP.

limitations: I think a more explicit discussion of the fact that the experiments are limited to two specific domains (traffic control and small-scale continuous control) would be beneficial: either discussing this limitation, or addressing it through further experiments.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper identifies a largely overlooked issue of task underspecification in the deep reinforcement learning (DRL) community: the performance of DRL approaches has been typically assessed on point MDPs, which are only special instances of the corresponding MDP families in many real-world problems. The paper empirically demonstrates that this selection bias can result in significant discrepancies in the relative performance of DRL approaches across different point MDPs in control benchmarks and a real-world traffic signal control problem, and suggests a series of recommendations for designing more accurate and efficient performance evaluation protocols.
strengths_and_weaknesses: ### Strengths
1. This paper contributes to the well-known and important problem in the DRL community of performance evaluations of DRL algorithms.
2. The identified task underspecification issue is interesting and not covered by prior work on performance evaluations of RL as far as I can tell.
3. The paper is well-organized and well-written.

### Weaknesses
1. I am a little doubtful about whether the suggested three techniques (random sampling with or without replacements & $k$-means clustering) for evaluating DRL methods on a family of MDPs are generally practical. While they would clearly be better than the traditional protocol of evaluating DRL algorithms only on point MDPs, balancing the efficiency and the reliability of these evaluations (e.g., deciding the sampling number or the number of clusters) still seems to be a non-trivial task without sufficient domain knowledge.
2. Compared with prior work on the evaluation protocols of RL algorithms, the evaluation in this paper is less extensive. I agree that using a real-world problem where RL algorithms are actually employed is beneficial for highlighting the pitfalls of the current protocol in realistic scenarios. However, given that the authors have argued that the task underspecification issue is prevalent in common DRL benchmarks, it would be great if more empirical results (e.g., on some of the environments listed in Table 1 in the appendix) are reported to show the generality of such pitfalls.

questions: 1. How are the parameters of different point MDPs selected for each MDP family in Table 2 in the appendix? Are they selected by some sort of randomization or according to the prior knowledge of the tasks?
2. In general, what is the main advantage of random sampling from a family of MDPs with replacements over random sampling without replacements? My current intuition is that if different point MDPs are of sufficiently different probabilities when being sampled, then random sampling with replacements may be better than random sampling without replacements since the latter may excessively alter the inherent distribution of point MDPs. Are there other explanations?

### Minor
- Line 72: pifalls $\rightarrow$ pitfalls
- Table 2, last row: MPDs $\rightarrow$ MDPs
- Line 88 in the appendix: missing reference

limitations: No concern.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper argues that RL algorithms should be evaluated on a suite of families of similar tasks (e.g., Mujoco benchmark that includes several parameterizations of Hopper with different limb lengths / weights / rewards, etc.) as opposed to a suite of individual tasks (e.g., Mujoco benchmark that includes only one particular parameterization of Hopper). It shows that the same algorithm can have different performance under different parameterizations of individual environments and argues that good evaluation in deep RL should therefore measure performance based on families of tasks. It provides recommendations for evaluating deep RL algorithms going forward, which include the creation of benchmarks that depict families of tasks.
strengths_and_weaknesses: Originality: This is definitely an original take. The authors seem to adopt a single MDP approach (rather than a robust RL / domain randomization approach / OOD generalization approach), but assert that algorithms should be evaluated on several similar single MDPs. 

Quality/Significance: I was not persuaded by the authors that the problem they are addressing affects deep RL as a whole or that it can accurately be termed “task underspecification”, or that their proposed approach is necessary or helpful in the evaluation of deep RL algorithms (as opposed to the suites of tasks that they recommend modifying at line 209). I buy the Section 4 case example, as one instance where, *if we seek to make conclusions about the performance of RL algorithms on the family of tasks, then we should evaluate the RL algorithms on the family of tasks*, but this is an argument against relying on a single task to evaluate an RL algorithm, rather than an argument for using a family of tasks vs a suite of tasks. For this reason, it is difficult for me to give this a high score on quality and significance. My reasoning is as follows; responses appreciated.

- Benchmarks already include a variety of tasks/MDPs. Not only do these tasks/MDPs differ, but they are quite diverse, with entirely different setups. It is unclear why evaluating on a family of similar MDPs (the “family of MDPs”) is superior to evaluating on a suite of tasks  (Atari has 57 games; do we need to add noise to all 57 to get a suite of 5700 games, where there are very similar sets of 100?). Of course, you can say it is because there are more tasks, but that would change the scope/thesis of the paper.
- The tasks are not underspecified… they are precisely specified. Pendulum is a specific pendulum, with a particular mass and length. Of course they are chosen somewhat arbitrarily, but that’s the point… that a good RL algorithm should perform well on any reasonable/solvable MDP. Choosing a family of tasks would be similarly arbitrary, and may involve creating unreasonable/unsolvable MDPs.
- There is insufficient evidence to support the hypothesis at lines 41-43. I do not see anywhere in Sections 2 where there is error in the evaluation due to the use of a point MDP rather than a familiar of MDPs. It is known that DRL methods are noisy, and that’s why we test them on a benchmark suite of several envs. On the other hand, Section 4 is a nice example, and while it is indeed “evidence of pitfalls in point MDP-based evaluations” (Footnote 4), I don’t see as providing sufficient support for the bolded hypothesis at lines 41-43, or the comment/recommendation at line 209. 
- Re: lines 99-106 about Figure 1... The observations (1)-(2) are obvious. In my view, the observation (3) simply lacks evidence. How many seeds were tested, where are the confidence bounds? In any case, noise in one task that shows up when evaluating it on the task family, would also show up when evaluating the algorithm on the full suite. Also it is unclear why TD3 is used in one figure, but TRPO in the others.
- Section 2 Observation 2 would also apply to across tasks in a suite (they don’t need to be part of the same task family). 
- Line 193: Is anyone claiming that these RL algorithms generalize to the model family? I would think not, because that is what multi-task RL / robust RL are about. 

I recognize that a large part of my criticism has to do with the framing of the paper. I think if the thesis of this paper were modified to be less "all encompassing", and more along the lines of the more specific thesis of robust RL (that when seeking to transfer RL algorithms to similar scenarios, we should change our approach to training/evaluation), that might fix the issues. But in that case, the recommendations may need to be modified. To maintain the current thesis, I would want to see the paper demonstrate an advantage of using families of similar tasks, over suites of different tasks. I suspect such an advantage would be hard to demonstrate. Alternatively, the authors could demonstrate that the recommendation is easy to implement, by modifying one of the suites themselves (Mujoco control suite is probably easiest since there are only a few environments --- would it change the usual empirical results; e.g., re: DDPG, TD3, SAC, or model-based RL?) 

Clarity: Overall the paper is well written / clear. 
questions: Why is a suite of different tasks insufficient, as opposed to a family of tasks? 
limitations: Yes
review_title: null

Meta-review: metareview: This paper proposes to evaluate deep RL algorithms on a family of MDPs, rather than single 'point' MDPs.  This seems a reasonable way to reduce performance variance, though I found the paper to be quite lacking in terms of depth, realism, and real impact.

In particular, the construction of the family of MDPs seems like a hard problem in general, which does not get a deep treatment in this paper. For instance, for many 'point' MDPs that might actually be of real-world interested it might be quite easy to construct a family of MDPs in a seemingly reasonable way, but for which change the conclusions in such a way that they do not generalise to problems of actual interest anymore.  Furthermore, as also raised by reviewers, if the goal is to evaluate generality of an algorithm, it would seem much better to evaluate on a benchmark of carefully-chosen (e.g., actually-interesting) diverse point MDPs than on a family of similar MDPs constructed around a single point.  Finally, many 'point' MDPs might contain substantial randomness themselves, and the distinction and similarlities between sampling multiple random (task) seeds on a single point MDP versus sampling from a family of MDPs is not sufficiently discussed.

Despite these limitations, I think it is good for the community to engage into discussions on how best to evaluate its methods.  I will therefore take the recommendation of the reviewers, and accept this paper.  I hope the comments on limitations above will resonate with the authors, or might inspire reasonable pushback about parts on which I'm perhaps being naive, given the authors will likely have thought about these issues a great deal.
confidence: Less certain
Decision: Accept