The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning

Abstract: Evaluations of Deep Reinforcement Learning (DRL) methods are an integral part of scientific progress of the field. Beyond designing DRL methods for general intelligence, designing task-specific methods is becoming increasingly prominent for real-world applications. In these settings, the standard evaluation practice involves using a few instances of Markov Decision Processes (MDPs) to represent the task. However, many tasks induce a large family of MDPs owing to variations in the underlying environment, particularly in real-world contexts. For example, in traffic signal control, variations may stem from intersection geometries and traffic flow levels. The select MDP instances may thus inadvertently cause overfitting, lacking the statistical power to draw conclusions about the method's true performance across the family. In this article, we augment DRL evaluations to consider parameterized families of MDPs. We show that in comparison to evaluating DRL methods on select MDP instances, evaluating the MDP family often yields a substantially different relative ranking of methods, casting doubt on what methods should be considered state-of-the-art. We validate this phenomenon in standard control benchmarks and the real-world application of traffic signal control. At the same time, we show that accurately evaluating on an MDP family is nontrivial. Overall, this work identifies new challenges for empirical rigor in reinforcement learning, especially as the outcomes of DRL trickle into downstream decision-making.

Reviews and author responses:

 We are thankful to the reviewer for taking a careful look at our rebuttal and taking the time to revise their assessment! We are glad that the reviewer found our new experiments interesting. We also would like to thank the reviewer for pointing out the debate between the family of MDPs from the same task vs. a suite of tasks. It indeed helped us better position our work. We will certainly use colors when showing changes in the future. Thanks again for all the suggestions and feedback.

 The rebuttal was responsive on all major points, and I like the new experiments (Section 5). I land somewhere between a 6 and 7 right now, and will update my scores accordingly. For future revisions on openreview, I'd recommend manually showing changes in color (no need to do it now!), just because the compare revisions is finnicky and doesn't work sometimes (like now for some reason). 

 Thank you very much for taking a careful look at our rebuttal. We're happy that our rebuttal alleviated the reviewer's main concerns. Thank you again for your detailed feedback and suggestions throughout the whole process!

 We're very thankful that the reviewer took the time to revise their assessment! The reviewer's feedback led to insightful experiments, which certainly strengthened our work. We appreciate the time that the reviewer has taken throughout the process to thoroughly review our work!

 Thank you for the response which addressed many of my concerns. I think the revised contributions indeed provide a more accurate location of the paper, and also appreciate the efforts that the authors have made during the rebuttal. At this moment I will keep my score towards acceptance.

 I thank the authors for their diligent efforts in responding to my comments. Given the new experimental results seem to validate this phenomena across a sufficiently wide variety of environments, I'll raise my score from 4 to 6.

 We are pleased that the reviewer found work to be interesting, novel and contributes to the well-known and important problem in the DRL community. We want to thank the reviewer for highlighting the value of our work, specially when DRL is applied to real world problems. We highly value the reviewer’s feedback on practical aspects of the some of the recommendations and also raising the limited evaluation aspect of the work. Taking the reviewer’s suggestion, we have conducted more experiments to further validate the pitfalls of point MDP based evaluations and to confirm the validity of our recommendations. 

- Compared with prior work on the evaluation protocols of RL algorithms, the evaluation in this paper is less extensive. I agree that using a real-world problem where RL algorithms are actually employed is beneficial for highlighting the pitfalls of the current protocol in realistic scenarios. However, given that the authors have argued that the task underspecification issue is prevalent in common DRL benchmarks, it would be great if more empirical results (e.g., on some of the environments listed in Table 1 in the appendix) are reported to show the generality of such pitfalls.

We certainly agree and thank the reviewer for pointing this out. We have added more experiments and demonstrated the pitfalls of point MDP based evaluations in 3 more control tasks including tasks from the Table 1 in the Appendix as the reviewer suggested. We have provided details of the results in the general section titled “Additional experiments to demonstrate pitfalls of point MDP based evaluations”. We hope the reviewer will find the results interesting and of greater value to the DRL community at large. 

- I am a little doubtful about whether the suggested three techniques (random sampling with or without replacements & k-means clustering) for evaluating DRL methods on a family of MDPs are generally practical. While they would clearly be better than the traditional protocol of evaluating DRL algorithms only on point MDPs, balancing the efficiency and the reliability of these evaluations (e.g., deciding the sampling number or the number of clusters) still seems to be a non-trivial task without sufficient domain knowledge.

We agree that balancing the efficiency and the reliability of these evaluations is a non-trivial task without sufficient domain knowledge. We thank the reviewer for highlighting  this challenge which led to some interesting discussions among the authors as potential future work. In the scope of this work, we assume a computational budget is given in terms of number of models that can be trained (line 146 and the footnote in the main paper). Therefore, sampling number or the number of cluster are defined to be equal to the total number of models that can be trained (the budget). Usually, the computational budget is less controllable and comes as a hard constraint. Therefore, we framed our work as attempting to achieve best possible performance evaluation reliability under the given budget. However, we do think there is an opportunity for future work to address this challenge in more details with possible theoretical bounds on the reliability of the performance evaluations. 

 - How are the parameters of different point MDPs selected for each MDP family in Table 2 in the appendix? Are they selected by some sort of randomization or according to the prior knowledge of the tasks?

Good point! Yes, we mostly selected them based on general understanding on which parameters and their realizations would be most meaningful for contextualizing the underlying task. We took inspirations from the generalizable RL/ robust RL community where such environment modifications are generally used [1, 2]. We do not have a general technique for parameter selection and believe that an appropriate design will be application-dependent. For example, in the context of the traffic signal control case study, a reasonable way to select parameters is to consider what parameters describes a typical signalized intersection. 

- In general, what is the main advantage of random sampling from a family of MDPs with replacements over random sampling without replacements? My current intuition is that if different point MDPs are of sufficiently different probabilities when being sampled, then random sampling with replacements may be better than random sampling without replacements since the latter may excessively alter the inherent distribution of point MDPs. Are there other explanations?

Exactly! We definitely agree with the reviewer’s intuition. Although not particularly designed to validate this intuition, we further provide a sensitivity analysis of the proposed three performance approximation methods in Appendix Section E.2 which shed some lights on the same intuition experimentally. We hope the reviewer may find it useful. 

Lastly, we are very thankful to the reviewer for recommending acceptance of our work. We would like to request the reviewer to consider increasing our score if our additional experiments and analysis are further satisfying to the reviewer. 

References

[1] Benjamins, Carolin, Theresa Eimer, Frederik Schubert, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. "Carl: A benchmark for contextual and adaptive reinforcement learning." arXiv preprint arXiv:2110.02102 (2021).

[2] Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, and Armando Solar-Lezama. Synthesizing programmatic policies that inductively generalize. In ICLR, 2020.


 We thank the reviewer for the very thought provoking review. We agree that the debate between a family of MDPs from the same task vs a suite of tasks (one MDP from each task) indeed deserves greater attention. We are thankful to the reviewer for pointing this out and calling for a change in framing of the work, which helped us a lot to review our contributions and revise them accordingly. 

- I recognize that a large part of my criticism has to do with the framing of the paper. I think if the thesis of this paper were modified to be less "all encompassing", and more along the lines of the more specific thesis of robust RL (that when seeking to transfer RL algorithms to similar scenarios, we should change our approach to training/evaluation), that might fix the issues.


We took the reviewer’s advice and revised our contributions accordingly. We refer the reviewer to the general section titled “Revised contributions” for more details on the revised contributions. We hope our revisions and clarifications would be helpful to address many of the comments the reviewer has provided. 

- [if the thesis of the paper were to be modified] But in that case, the recommendations may need to be modified. 


We appreciate the Reviewer’s thoughtfulness in bringing up this possibility. We believe that our recommendations remain valid and do not require modifications — even for the reframed scope of evaluating algorithmic generalization within a task.

 
- the authors could demonstrate that the recommendation is easy to implement, by modifying one of the suites themselves (Mujoco control suite is probably easiest since there are only a few environments — would it change the usual empirical results; e.g., re: DDPG, TD3, SAC, or model-based RL?)

We thank the reviewer for this interesting suggestion. We found that generalizable/robust RL community has already explored and created such modified benchmarks [1] or has shown such modifications [2], although their motivation for the use of them are different. One could easily use the same benchmarks just by re-purposing them without major modifications. These benchmarks include task suites such as classic control, box2D, Brax, Mario, RNA Design. 

For the question “would it change the usual empirical results?”, our additional experiments suggest it would. For example, in Brax half cheetah environment, we see a clear ranking change among the DRL methods (TD3, SAC, PPO, TRPO). We refer the reviewer to the section titled “Additional experiments to demonstrate pitfalls of point MDP based evaluations” in the general comments section for more details. 

To better clarify our response, we provide below a grouped response to the comments and questions from the reviewer which we believe to arise as a result of the debate on “Considering a suite of different tasks (one point MDP from each task) as opposed to a family of point MDPs within a task, for evaluating DRL methods”. We first list all the relevant comments and questions from the reviewer and provide our grouped response subsequently.

- Benchmarks already include a variety of tasks/MDPs. Not only do these tasks/MDPs differ, but they are quite diverse, with entirely different setups. It is unclear why evaluating on a family of similar MDPs (the “family of MDPs”) is superior to evaluating on a suite of tasks (Atari has 57 games; do we need to add noise to all 57 to get a suite of 5700 games, where there are very similar sets of 100?). Of course, you can say it is because there are more tasks, but that would change the scope/thesis of the paper.

- The tasks are not underspecified… they are precisely specified. Pendulum is a specific pendulum, with a particular mass and length. Of course they are chosen somewhat arbitrarily, but that’s the point… that a good RL algorithm should perform well on any reasonable/solvable MDP. Choosing a family of tasks would be similarly arbitrary, and may involve creating unreasonable/unsolvable MDPs.

- [About lines 99-106 on Figure 1] In any case, noise in one task that shows up when evaluating it on the task family, would also show up when evaluating the algorithm on the full suite. 

- Why is a suite of different tasks insufficient, as opposed to a family of task

These comments from the reviewer prompted interesting discussion among the authors, which we believe could lead to impactful future research, and for that we are grateful. The comment has inspired a new (preliminary) set of experiments in the section titled “Considering a suite of different tasks (one point MDP from each task) as opposed to a family of point MDPs within a task, for evaluating DRL methods.” (under general comments). In short, we consider whether the “somewhat arbitrary” selection of point MDPs (e.g., pendulum with a specific mass and length) impacts what we consider to be a “good RL algorithm.” The preliminary experiments suggest that choosing different “default” point MDPs across tasks would change the ranking of DRL algorithms.

We hope our revised contributions and previously stated experiments have addressed the reviewer’s comment on “why evaluating on a family of similar MDPs (the “family of MDPs”) is superior to evaluating on a suite of tasks”. In short, we do not make such a claim. As the reviewer also pointed out at the end, such a claim would be hard to demonstrate. As stated above, our contributions in this work focus on reliable performance evaluations when evaluating algorithmic generalization of DRL methods within a task. Therefore, we do not focus on algorithmic generalization of DRL methods across tasks. But we indeed recognize it as an interesting future research direction that requires attention. 

As for the selection of a family of MDPs, we agree that the selection would need to be treated with care. We do not have a general solution for this, and believe that an appropriate design will be application-dependent. For example, in the context of the traffic signal control case study, a reasonable way to select a family of MDPs is based on a city’s distribution of MDPs, because traffic signals are often operated by cities.

 
- The authors seem to adopt a single MDP approach (rather than a robust RL / domain randomization approach / OOD generalization approach), but assert that algorithms should be evaluated on several similar single MDPs.

Thanks in large part due to the Reviewer’s thoughtful comment, we would like to refer the Reviewer to our revised contributions (in the general section titled “Revised contributions”). The section clarifies that we are not taking robust RL / domain randomization / OOD generalization approach. We would like to highlight our focus on the emerging issue where real-world use cases of DRL often induces a family of similar MDPs. In these circumstances, we may not know in advance the specific MDP of most interest to the engineer (ex: which scenario(s) within traffic signal control, autonomous driving, chemical reaction optimizations under varying instrument errors). We reason that assessing the algorithmic generalization of DRL methods on a family of MDPs of the task should give a stronger indication of how well the algorithm would perform in a target MDP of interest to the engineer.

The callout to multi-task learning, however, is interesting and we believe would be important future research; here, we adopt the single-MDP training regime mostly to simplify the starting point to the identifying issues pertaining to evaluation. When considering a multi-task extension to the work, it will be important to consider additional factors such as the number of training tasks and how they are selected. We believe these extensions to be valuable, but that a single-task training regime provides an initial but meaningful avenue to identifying the core issues.

- There is insufficient evidence to support the hypothesis at lines 41-43. I do not see anywhere in Sections 2 where there is error in the evaluation due to the use of a point MDP rather than a familiar of MDPs. It is known that DRL methods are noisy, and that’s why we test them on a benchmark suite of several envs. On the other hand, Section 4 is a nice example, and while it is indeed “evidence of pitfalls in point MDP-based evaluations” (Footnote 4), I don’t see as providing sufficient support for the bolded hypothesis at lines 41-43, or the comment/recommendation at line 209.

We are thankful to the reviewer for identifying a discrepancy in our stated hypothesis and the paper’s organization. Our intention in Section 2 was to lay out supporting observations, and then provide empirical evidence in Section 4 (as noted by the Reviewer) and in newly added Section 5. However, it is easy to interpret Section 2 as already providing evidence. To remedy this, we have clarified this in the main paper. 

As the reviewer noted, Section 4 is indeed designed to provide evidence of pitfalls in point MDP based evaluations. In addition, we have also added 3 more experiments as presented in general Section titled “Additional experiments to demonstrate pitfalls of point MDP based evaluations” to validate our claims (more details in Section 5 of the paper). We hope the reviewer will find them insightful. 

 - Re: lines 99-106 about Figure 1... The observations (1)-(2) are obvious. In my view, the observation (3) simply lacks evidence. How many seeds were tested

Regarding the observations (1) and (2): We agree that this observation is unsurprising to anyone who has worked with DRL. We felt it was important to substantiate the “common knowledge” with empirical experiments, especially because the observation is foundational to our investigation of task families. However, we are open to removing these observations on the toy experiments (or shifting it to the Appendix). 

Regarding the observations (3), first, we are referring to the relative performance gaps that can be seen in the sub figures of figure 1. For example, for Quad, point MDP (0,6) indicates both PPO and TRPO are equally well-performing (score difference is 0). However, under point MDP (0.5,3), we see that PPO outperforms TRPO with a ~0.75 point difference. This indicates that comparing DRL methods based on point MDPs can provide conflicting conclusions. We also added a sentence to the paper giving more details to help readers better understand our claims (line 114). Second, we tested 20 seeds for each DRL method on each point MDP when training. For the illustrative example purpose, we picked the run with the best inference time performance and plot the results for each point MDP and DRL method combination. 

- [About lines 99-106 on Figure 1] Also it is unclear why TD3 is used in one figure, but TRPO in the others.

We used TD3 for the Pendulum task because it performed better (based on point MDP evaluations), and vice versa for TRPO on the other tasks. We chose the best performing methods according to the standard evaluation procedure to illustrate our point that such procedures can be misleading when considering a family of MDPs within a task.

- Section 2 Observation 2 would also apply to across tasks in a suite (they don’t need to be part of the same task family).

Totally agree. They indeed would apply to across tasks in a suite. However, our aim of Section 2 is to provide an illustrative example of pitfalls of point MDP based evaluation when evaluating algorithmic generalization within a task and not across tasks. Therefore, in the scope of this work, we do not intend to discuss observation 2 as it can be seen across tasks in a suite. However, as we have mentioned earlier in our response, we hope the reviewer may find our preliminary experiments on “Considering a suite of different tasks (one point MDP from each task) as opposed to a family of point MDPs within a task, for evaluating DRL methods” in the general comments section as useful for observing the overall effects of point MDP based evaluations in evaluating algorithmic generalization across tasks which implicitly includes this phenomenon as well. 

- Line 193: Is anyone claiming that these RL algorithms generalize to the model family? I would think not, because that is what multi-task RL / robust RL are about.

We are grateful to the reviewer for this comment; it helped us to clarify our contributions by contrasting our work with multi-task / robust RL. In particular, multi-task RL and robust RL are about policy generalization whereas our focus is on algorithmic generalization (both considered within a task). Given this new clarity, we rephrased our contributions as focusing on the emerging problem of point MDP based performance evaluations when evaluating algorithmic generalization of DRL methods within a task, rather than as an already prevalent problem within DRL performance evaluation.

Lastly, we would like to request the reviewer to consider increasing our score to recommend acceptance if our rebuttal is satisfying and sufficiently covers the reviewer’s concerns and suggestions. 


References

[1] Benjamins, Carolin, Theresa Eimer, Frederik Schubert, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. "Carl: A benchmark for contextual and adaptive reinforcement learning." arXiv preprint arXiv:2110.02102 (2021).

[2] Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, and Armando Solar-Lezama. Synthesizing programmatic policies that inductively generalize. In ICLR, 2020.

 We’re extremely grateful for the numerous, detailed feedbacks and suggestions that Reviewer **Ybmr** has made. We agree with the reviewer’s assessment of our main contributions, and here we provide additional experimental results addressing Reviewer Ybmr’s suggestions.

- More broadly, I'd prefer to see more of an emphasis and investigation of the possibility of this phenomena of reordering algorithms, for example another problem settings and with other algorithms. I imagine a key place where this could occur is in more complicated continuous control settings such as MuJoCo/DMControl/Brax. 

We definitely agree and thank the reviewer for pointing this out. We have added more experiments and demonstrated the phenomena of reordering algorithms and changes to relative performances in 3 more control tasks including a task from Brax and more DRL methods for comparison, as the reviewer suggested. We have provided details of the results in the general section titled “Additional experiments to demonstrate pitfalls of point MDP based evaluations”. Interestingly, we see the phenomena of reordering algorithms even in simple tasks like pendulum (continuous control), which further signifies the pitfalls of point MDP-based evaluations. We hope the reviewer will find the results interesting. 

- While this paper focused entirely on different point MDPs within an MDP family, a common method of evaluation is to consider a task family, such as a variety of MuJoCo, Atari or Procgen tasks, to ensure that evaluation of the algorithm is robust. A comparison of the discussion in this paper to works with that evaluation protocol would be beneficial.

This is a good point. We address this suggestion in detail with experiments in the general section titled “Considering a suite of different tasks (one point MDP from each task) as opposed to a family of point MDPs within a task, for evaluating DRL methods”. 

 
- I think the results in observation 2 are unsurprising (different point MDPs are difficult); 

We agree that this observation is unsurprising to anyone who has worked with DRL. We felt it was important to substantiate the “common knowledge” with empirical experiments, especially because the observation is foundational to our investigation of MDP families. However, we are open to removing the observation 2 on the toy experiments (or shifting it to the Appendix).

- we don't really care about absolute performance of algorithms on these toy benchmarks [Section 2], as it's not that useful as information about how well they'll do on some other down-stream task. Performance relative to other algorithms (to enable an informed choice of which algorithm to use for a downstream task), or to a human expert on the task (so as to have a notion of whether the algorithm can outperform humans on some narrow domain), are more important.

In large, we agree with this assessment. Although, we want to clarify that we only used the toy examples as illustrative examples for pointing out pitfalls in point MDP based evaluations. That is to say that these toy tasks (e.g. Quad) should be interpreted as down-stream tasks in this section.

- What do you mean when you say that "not all parallel runs succeed" in footnote 1? Do you mean they don't converge? If so, I think it's more representative to also plot the failing (i.e. non-converging) runs also, so as not to give a skewed picture of the performance of different algorithms.

Yes, we meant they did not converge. Having said that, we note in Figure 2 (the main content that footnote 1 refers to), we only intend to provide an illustrative example of how DRL training can get impacted by the choice of the point MDP. We want to clarify to the reviewer that we do not attempt to compare algorithms in this example. In fact, we only use PPO for the analysis and do not compare DRL methods. Our focus is solely on pointing out how the choice of point MDPs can affect the DRL training. We achieve this by showing how training on different point MDPs converge to different scores (either to different local optima or global optimum). Thus we did not plot the failing runs because we wanted to isolate the impact on DRL training caused by the choice of the point MDP from other factors such as choosing a bad seed. That is, if all parallel runs fail, that is sufficient evidence that even if a good seed is chosen, the characteristics of the point MDP can cause the training to fail (which supports the main point we are making). 

- In figure 1, is each algorithm trained once on each MDP, and then tested on all the MDPs labelled on the x axis? 

To clarify, each algorithm is individually trained and tested on each MDP (as explained in line 105). We do not do any zero-shot transfer evaluations. 

- [In figure 1] Also, it would be good to see which random MDP is chosen in each case, as that would explain the performance of each algorithm trained on this MDP.

For exactly the reason pointed out by the Reviewer, we consider all MDPs as potential random MDPs, as explained in line 106. Red dots indicate the performance score of generic MDPs and blue dots indicate the same for simplified MDPs. However for a given task, generic MDP and the simplified MDP could also be candidates for the random MDP. 

- I think the most interesting results are those presented in section 4, and if similar results could be demonstrated in other domains and with more algorithms, then I would raise my score to recommending acceptance. If these results were demonstrated, and the recommendations for improving the robustness of evaluations were applied to them successfully then I think the paper would be a robust and worthwhile contribution worthy of acceptance.

We are grateful to the reviewer for these suggestions and willingness to increase our score and recommend acceptance. As stated in our previous comments, we have performed 3 more experiments with different control tasks as the reviewer suggested. We have also updated the recommendations (specially the recommendation in Section 6.2: performance approximation with a budget) with additional experiments stemming from the extra 3 control tasks to further validate our recommendations (please refer to Section 6.2 of the main paper and Section E in the appendix). Therefore, we request the reviewer to consider increasing our score to recommend acceptance if these presented results are satisfying and address the reviewer’s concerns and suggestions. 


 We thank the reviewers for your insightful and thoughtful feedback. We are encouraged that you found our work to be original and novel (reviewers **Ybmr, QUf6, bTHb**), has a real-world value (reviewer **QUf6**), and is well-written and clear (reviewers **Ybmr, QUf6, bTHb**). We are pleased that reviewers **Ybmr** and **QUf6** acknowledge that demonstrating point MDP based evaluation pitfalls is valuable and significant for the broader benefit of the research community. We are glad that all reviewers unanimously found the traffic signal control case study insightful and conveyed significant evidence of pitfalls in point MDP-based evaluations and that it is beneficial to the research community (reviewers **Ybmr, QUf6, bTHb**). We are pleased that reviewer **Ybmr** found interest in our demonstrated insights on the relative performance or ordering of different methods changes when making choices that were implicitly assumed not to change the relative performance or ordering. Overall, our recommendations for a more robust evaluation were recognized as beneficial (reviewers **Ybmr, QUf6**). 

On the constructive side, all reviewers noted that more empirical evidence of point MDP pitfalls on other tasks such as continuous control could benefit the paper. We definitely agree. We were initially constrained by the significant computational requirements needed for further evaluations. Another concern that was primarily raised by reviewer **bTHb** is why a suite of different tasks is insufficient, as opposed to a family of point MDPs in DRL evaluations. In this rebuttal, we provide more empirical evidence of point MDP based evaluations using 3 more control tasks in both continuous and discrete control. In response to the helpful reviewer comments, we further clarify and revise our contributions regarding evaluating a DRL algorithm on a suite of different tasks vs a family of point MDPs. 

 
## Revised contributions

Based on the highly constructive reviewer feedback, we have rephrased our overall contributions to improve clarity and to refine the scope of our work:


1. Our main contribution is identifying and substantiating an emerging problem in DRL performance evaluation, where a single or small set of arbitrarily selected point MDPs is used to **evaluate algorithmic generalization of DRL methods within a task**. To put this into context of related DRL practices, in this work we neither address the evaluation of algorithmic generalization **across task families** (standard DRL evaluations using task suites), nor the evaluation of **policy generalization** within a task family (as addressed in multi-task, robust RL).
    
2. Our second contribution is proposing a series of recommendations to alleviate identified pitfalls and conduct more reliable performance evaluations to assess algorithmic generalization of DRL methods within a task. We discuss techniques for standardizing performance evaluations, performing performance evaluations under a computational budget, and the use of performance profiles to cope with large quantities of MDPs. 


3. Lastly, through extensive experiments in multiple standard control tasks and a case study of a real-world application, we demonstrate the pitfalls of performance evaluations when point MDPs are used to evaluate algorithmic generalization of DRL methods within a task. Extensive experiments are also conducted to substantiate the proposed recommendations. 

In the following sections (unless otherwise stated) by “performance evaluation,” we mean point MDP-based performance evaluations for algorithmic generalization of DRL methods within a task. 

 
## Additional experiments to demonstrate pitfalls of point MDP based evaluations

We appreciate the reviewers for pointing out the need for more experiments to substantiate the prevalence of the pitfalls of point MDP based evaluations. In addition to the traffic signal control case study presented in Section 4 of the paper, we performed 3 more experiments using three control tasks as suggested by reviewer **Ybmr** and **bTHb**. In particular we conducted experiments on brax half-cheetah (continuous control), pendulum (continuous control) and cartpole (discrete control). 

We report our results in details in the newly added Section 5 of the revised paper and Section D of the appendix. We summarize the insights here. We encourage and appreciate if the reviewers can check the revised paper for more details. 

As noted by reviewer **Ybmr**, a key issue arising from task underspecification occurs when the ranking of methods changes when the underlying task is modified. In our additional experiments, we further validate this issue by showing how ranking and/or relative performance of DRL methods change as a result of evaluating on a family of point MDPs. To evaluate the reported performance, we use the default point MDP given in OpenAI gym [1] for cartpole and pendulum and the default point MDP given in Brax [2] for half cheetah. A uniform point MDP distribution was used to evaluate overall performance (see Section 3 Definition 1 for the definition of overall performance).

**Cartpole**: We use four DRL methods for training: ARS, PPO, TRPO and DQN. We use a MDP family of 576 point MDPs, corresponding to changes in pole length, mass of the cart, mass of the pole, force magnifier and gravity. For reported performances, all methods solved the cartpole default point MDP optimally indicating equal performances. However, we found that when a family of MDPs are used, the results significantly change. We summarize the overall performance in the below table (higher the scores the better). Although ARS obtained a perfect score for the default cartpole point MDP, we see that it in fact has significantly low overall performance — as low as 0.46 — and none of the methods can solve all 576 point-MDPs optimally. This also results in a relative performance change of methods. The new ranks are 1. PPO, 2. TRPO, 3. DQN and 4. ARS. As a side effect, although Cartpole is often used as a sanity check rather than to rank DRL methods, this experiment indicates that a Cartpole MDP family could still be informative for ranking DRL methods.

|                      | ARS  | PPO  | TRPO | DQN  |
| -------------------- | ---- | ---- | ---- | ---- |
| Reported Score       | 1.0  | 1.0  | 1.0  | 1.0  |
| Overall Score (ours) | 0.46 | 0.80 | 0.73 | 0.66 |

**Pendulum**: We use four DRL methods for training: TD3, PPO, TRPO and SAC. We use a point MDP family of 180 point MDPs, corresponding to changes in pendulum mass, length of the pendulum and gravity. We show the reported and overall performances in the table below (lower the score the better).  We note a few interesting observations. First, performance of TRPO and SAC are clearly overestimated in reported performances. Also, there is a significant variation in performances of all four DRL methods (Appendix Section D Figure 6). Although one would tend to pick TRPO as the best performing method based on reported performance, our analysis shows that SAC is in fact the highest performing method. Thus, once more, the use of a MDP family changes the DRL method ranking.

|                      | TD3  | PPO  | TRPO | SAC  |
| -------------------- | ---- | ---- | ---- | ---- |
| Reported Score       | 1.49 | 0.87 | 0.16 | 0.19 |
| Overall Score (ours) | 1.13 | 0.92 | 0.84 | 0.65 |

**Half-cheetah**: We use four DRL methods for training: TD3, PPO, TRPO and SAC. We use a point MDP family of 120 point MDPs, corresponding to changes in gravity, torso mass and friction.  We show the reported and overall performances in the table below (higher the score the better). Similar to cartpole and pendulum, we see changes in rankings of the methods. In this case, the first and second performing methods (SAC and TRPO) retain their rank order; the third and fourth places are swapped. According to reported performances, PPO outperforms TD3 but we note that TD3 is in fact better than PPO when overall performance is used for ranking. Additionally, although TRPO’s rank stays the same, we see its overall performance has degraded by a significant margin. These phenomena again highlight the pitfalls of point MDP based evaluations when used in evaluating algorithmic generalization within a task.

|                      | TD3   | PPO  | TRPO | SAC  |
| -------------------- | ----- | ---- | ---- | ---- |
| Reported Score       | -0.05 | 0.15 | 0.98 | 1.84 |
| Overall Score (ours) | 0.37  | 0.17 | 0.58 | 1.87 |



 
## Considering a suite of different tasks (one point MDP from each task) as opposed to a family of point MDPs within a task, for evaluating DRL methods

We are thankful for the very thoughtful comments by reviewer **Ybmr** and **bTHb** on the subject. This is indeed a very interesting point and led to some wonderful discussions among the authors. 

While this question is definitely interesting and worth exploring, we believe it warrants to be treated as a separate contribution orthogonal to our current contributions. As suggested by reviewer **bTHb** and as explained in revised contributions section, we limit the scope of this work to pitfalls of point MDP based performance evaluations when evaluating **algorithmic generalization of DRL methods within a task**. Our proposed recommendation are also catered around this problem alone. We are not addressing the issues surrounding pitfalls of point MDP based performance evaluations when evaluating **algorithmic generalization of DRL methods across tasks** (suite of tasks). Our contribution is mainly motivated by the undesired developments in DRL where use of point MDP based performance evaluations has become the standard practice in evaluating algorithmic generalization within a task, specifically when the underlying task is a real-world task like traffic signal control [3-8]. We suggest that a sufficiently large and carefully selected family of MDPs (as opposed to a few somewhat arbitrarily selected point MDPs) should be used for reliable performance evaluations to demonstrate DRL algorithmic generalization within a task. Confirming the need for a family of point MDPs, a few studies that perform large scale performance evaluation (thus inherently using MDP families) have shown interesting insights that are hidden under the point MDP based evaluations [9]. 

However, we are quite intrigued by the questions the reviewers raised on the topic, so we added a Section G to the Appendix providing preliminary experiments on how even point MDP-based evaluations could adversely affect the standard practice in evaluating algorithmic generalization across tasks (suite of tasks), rather than within tasks (focus of our work). However, we do not discuss this in the main paper and only use the Appendix Section G as a call for future work. We summarize our insights below and request the reviewers to view Section G of the Appendix for more details.

The basis for our preliminary experiment is the hypothesis that the DRL benchmark designers may tend to select default point MDPs that are simplified/generic (because they are not thinking about or trying to solve the hardest version of cartpole, for instance). Considering an implicit family of MDPs, we can model this as a biased selection of default point MDPs. We designed a new experiment that tests whether this bias has an impact on the relative ranking of DRL methods.

We empirically validate the claim by showing depending on the choice of point MDPs from tasks, the relative performance or ranking of methods could significantly change. By using pendulum, half cheetah and mountain car continuous control tasks as a suite of tasks, we show that depending on which point MDP is used from each task, PPO is ranked first for 19% times, TRPO for 26%, SAC for 43% and TD3 for 12%. For comparison, algorithms are ranked SAC, TRPO, PPO and TD3 when trained on the suite of default point MDPs. These findings provide preliminary evidence that considering point MDP families may even benefit the performance evaluation practices for DRL across tasks, because doing so may inform better selection of “default” point MDPs. We summarize the results in table below.


|          | Ranked 1 | Ranked 2 | Ranked 3 | Ranked 4 |
| -------- | -------- | -------- | -------- | -------- |
| PPO      | 18.75%   | 29.54%   | 27.74%   | 23.97%   |
| TRPO     | 26.23%   | 24.99%   | 30.45%   | 18.33%   |
| SAC      | 42.56%   | 29.09%   | 18.50%   | 9.85%    |
| TD3      | 12.46%   | 16.38%   | 23.31%   | 47.85%   |
| Reported | SAC      | TRPO     | PPO      | TD3      |


In our eyes, this may unlock exciting avenues for research to advance empirical rigor in DRL. We believe that Reviewer **Ybmr’s** comment that “evaluation on a range of (point MDPs from) different tasks solves the problem [of non-robust evaluation]” represents the common intuition of the DRL community, but our expanded perspective (looking at MDP families) may challenge this intuition. As we showed in our experiment, even after an evaluation on a range of (point MDPs from) different tasks, the evaluations can be unreliable. However, we call for further analysis of this issue as future work as it does not fall directly under our current main contributions.


 ### References

[1] https://www.gymlibrary.ml/

[2] https://github.com/google/brax

[3] James Ault and Guni Sharon. Reinforcement learning benchmarks for traffic signal control. In
Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks
Track, 2021.

[4] Vinitsky, Eugene, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Foerster. "Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world." arXiv preprint arXiv:2206.09889 (2022).

[5] Zhenpeng Zhou, Xiaocheng Li, and Richard N. Zare. Optimizing chemical reactions with deep
reinforcement learning. ACS Central Science, 3:1337 – 1344, 2017

[6] Yufan Zhao, Michael R. Kosorok, and Donglin Zeng. Reinforcement learning design for cancer clinical trials. Statistics in Medicine, 28, 2009.481

[7] Vinitsky, Eugene, Aboudy Kreidieh, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M. Bayen. "Benchmarks for reinforcement learning in mixed-autonomy traffic." In Conference on robot learning, pp. 399-409. PMLR, 2018.

[8] Wu, Cathy, Abdul Rahman Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M. Bayen. "Flow: A modular learning framework for mixed autonomy traffic." IEEE Transactions on Robotics (2021).

[9] Chen, Chacha, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, and Zhenhui Li. “Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, pp. 3414-3421. 2020.

 This paper presents the issue of task underspecification in evaluating reinforcement learning algorithms. This issues emerges when an RL algorithm is evaluated on a single instance of an underlying (often implicit) task family, and then that performance is implicitly assumed to be robust to the choice of instance in the task family. The paper shows that this robustness is often not the case, and in some cases the ordering between different algorithms can change when different task instances from the task family are chosen. Recommendations for ensuring a more robust evaluation of algorithms are provided, although the core problem of the computational burden of robust evaluations still somewhat remains (and is seemingly unavoidable). I think the issue of task underspecification, especially in the traffic control domain used, is a real one. The demonstration of this phenomena is valuable and significant. The paper is generally well-written and clear. I think the recommendations for more robust evaluation are beneficial, and the results in section 4 are interesting and beneficial to the research community, especially in the RL traffic control literature (which I'm less familiar with).

To me, the key issue arising from task underspecification is when the relative performance or ordering of different methods changes when making choices that were implicitly assumed not to change the relative performance or ordering. This is what is demonstrated very clearly in the traffic control domain, where the experiments demonstrate that when performing a more robust evaluation, the non-RL methods are preferred. Further, I think the results in observation 2 are unsurprising (different point MDPs are difficult); and we don't really care about absolute performance of algorithms on these toy benchmarks, as it's not that useful as information about how well they'll do on some other down-stream task. Performance relative to other algorithms (to enable an informed choice of which algorithm to use for a downstream task), or to a human expert on the task (so as to have a notion of whether the algorithm can outperform humans on some narrow domain), are more important.

Given this, it would be beneficial to highlight this precise phenomena in the other experiments. Figure 1 doesn't make it easy to see if choosing a different point MDP changes the preferred algorithm, or if it would also change if averaging over the performance across all point MDPs. More broadly, I'd prefer to see more of an emphasis and investigation of the possibility of this phenomena of reordering algorithms, for example another problem settings and with other algorithms. I imagine a key place where this could occur is in more complicated continuous control settings such as MuJoCo/DMControl/Brax. Investigating whether changing the physical parameters or agent morphology parameters (in a similar way to the experiments in section 2) also changes the SOTA algorithm would provide more evidence of the conclusions drawn in the paper.

While this paper focused entirely on different point MDPs within an MDP family, a common method of evaluation is to consider a task family, such as a variety of MuJoCo, Atari or Procgen tasks, to ensure that evaluation of the algorithm is robust. A comparison of the discussion in this paper to works with that evaluation protocol would be beneficial; crucially, demonstrating that the problem of non-robust evaluation demonstrated in the toy settings and the traffic control setting aren't just specific to those settings, and that evaluation on a range of (point MDPs from) different tasks solves the problem that's caused by task underspecification in terms of robust evaluation.

Overall, I'm currently recommending a weak reject, due to the fact that I don't think the scientific contribution of the paper is sufficient. I think the most interesting results are those presented in section 4, and if similar results could be demonstrated in other domains (as mentioned above) and with more algorithms, then I would raise my score to recommending acceptance. I realise this requires a high computational workload. If these results were demonstrated, and the recommendations for improving the robustness of evaluations were applied to them successfully, then I think the paper would be a robust and worthwhile contribution worthy of acceptance.

EDIT: I thank the authors for their diligent efforts in responding to my comments. Given the new experimental results seem to validate this phenomena across a sufficiently wide variety of environments, I'll raise my score from 4 to 6.
 What do you mean when you say that "not all parallel runs succeed" in footnote 1? Do you mean they don't converge? If so, I think it's more representative to also plot the failing (i.e. non-converging) runs also, so as not to give a skewed picture of the performance of different algorithms.

In figure 1, is each algorithm trained once on each MDP, and then tested on all the MDPs labelled on the x axis? Also, it would be good to see which random MDP is chosen in each case, as that would explain the performance of each algorithm trained on this MDP.
 I think a more explicit discussion of the fact that the experiments are limited to two specific domains (traffic control and small-scale continuous control) would be beneficial: either discussing this limitation, or addressing it through further experiments.

 This paper identifies a largely overlooked issue of task underspecification in the deep reinforcement learning (DRL) community: the performance of DRL approaches has been typically assessed on point MDPs, which are only special instances of the corresponding MDP families in many real-world problems. The paper empirically demonstrates that this selection bias can result in significant discrepancies in the relative performance of DRL approaches across different point MDPs in control benchmarks and a real-world traffic signal control problem, and suggests a series of recommendations for designing more accurate and efficient performance evaluation protocols. ### Strengths
1. This paper contributes to the well-known and important problem in the DRL community of performance evaluations of DRL algorithms.
2. The identified task underspecification issue is interesting and not covered by prior work on performance evaluations of RL as far as I can tell.
3. The paper is well-organized and well-written.

### Weaknesses
1. I am a little doubtful about whether the suggested three techniques (random sampling with or without replacements & $k$-means clustering) for evaluating DRL methods on a family of MDPs are generally practical. While they would clearly be better than the traditional protocol of evaluating DRL algorithms only on point MDPs, balancing the efficiency and the reliability of these evaluations (e.g., deciding the sampling number or the number of clusters) still seems to be a non-trivial task without sufficient domain knowledge.
2. Compared with prior work on the evaluation protocols of RL algorithms, the evaluation in this paper is less extensive. I agree that using a real-world problem where RL algorithms are actually employed is beneficial for highlighting the pitfalls of the current protocol in realistic scenarios. However, given that the authors have argued that the task underspecification issue is prevalent in common DRL benchmarks, it would be great if more empirical results (e.g., on some of the environments listed in Table 1 in the appendix) are reported to show the generality of such pitfalls.
 1. How are the parameters of different point MDPs selected for each MDP family in Table 2 in the appendix? Are they selected by some sort of randomization or according to the prior knowledge of the tasks?
2. In general, what is the main advantage of random sampling from a family of MDPs with replacements over random sampling without replacements? My current intuition is that if different point MDPs are of sufficiently different probabilities when being sampled, then random sampling with replacements may be better than random sampling without replacements since the latter may excessively alter the inherent distribution of point MDPs. Are there other explanations?

### Minor
- Line 72: pifalls $\rightarrow$ pitfalls
- Table 2, last row: MPDs $\rightarrow$ MDPs
- Line 88 in the appendix: missing reference
 No concern.

 This paper argues that RL algorithms should be evaluated on a suite of families of similar tasks (e.g., Mujoco benchmark that includes several parameterizations of Hopper with different limb lengths / weights / rewards, etc.) as opposed to a suite of individual tasks (e.g., Mujoco benchmark that includes only one particular parameterization of Hopper). It shows that the same algorithm can have different performance under different parameterizations of individual environments and argues that good evaluation in deep RL should therefore measure performance based on families of tasks. It provides recommendations for evaluating deep RL algorithms going forward, which include the creation of benchmarks that depict families of tasks. Originality: This is definitely an original take. The authors seem to adopt a single MDP approach (rather than a robust RL / domain randomization approach / OOD generalization approach), but assert that algorithms should be evaluated on several similar single MDPs. 

Quality/Significance: I was not persuaded by the authors that the problem they are addressing affects deep RL as a whole or that it can accurately be termed “task underspecification”, or that their proposed approach is necessary or helpful in the evaluation of deep RL algorithms (as opposed to the suites of tasks that they recommend modifying at line 209). I buy the Section 4 case example, as one instance where, *if we seek to make conclusions about the performance of RL algorithms on the family of tasks, then we should evaluate the RL algorithms on the family of tasks*, but this is an argument against relying on a single task to evaluate an RL algorithm, rather than an argument for using a family of tasks vs a suite of tasks. For this reason, it is difficult for me to give this a high score on quality and significance. My reasoning is as follows; responses appreciated.

- Benchmarks already include a variety of tasks/MDPs. Not only do these tasks/MDPs differ, but they are quite diverse, with entirely different setups. It is unclear why evaluating on a family of similar MDPs (the “family of MDPs”) is superior to evaluating on a suite of tasks  (Atari has 57 games; do we need to add noise to all 57 to get a suite of 5700 games, where there are very similar sets of 100?). Of course, you can say it is because there are more tasks, but that would change the scope/thesis of the paper.
- The tasks are not underspecified… they are precisely specified. Pendulum is a specific pendulum, with a particular mass and length. Of course they are chosen somewhat arbitrarily, but that’s the point… that a good RL algorithm should perform well on any reasonable/solvable MDP. Choosing a family of tasks would be similarly arbitrary, and may involve creating unreasonable/unsolvable MDPs.
- There is insufficient evidence to support the hypothesis at lines 41-43. I do not see anywhere in Sections 2 where there is error in the evaluation due to the use of a point MDP rather than a familiar of MDPs. It is known that DRL methods are noisy, and that’s why we test them on a benchmark suite of several envs. On the other hand, Section 4 is a nice example, and while it is indeed “evidence of pitfalls in point MDP-based evaluations” (Footnote 4), I don’t see as providing sufficient support for the bolded hypothesis at lines 41-43, or the comment/recommendation at line 209. 
- Re: lines 99-106 about Figure 1... The observations (1)-(2) are obvious. In my view, the observation (3) simply lacks evidence. How many seeds were tested, where are the confidence bounds? In any case, noise in one task that shows up when evaluating it on the task family, would also show up when evaluating the algorithm on the full suite. Also it is unclear why TD3 is used in one figure, but TRPO in the others.
- Section 2 Observation 2 would also apply to across tasks in a suite (they don’t need to be part of the same task family). 
- Line 193: Is anyone claiming that these RL algorithms generalize to the model family? I would think not, because that is what multi-task RL / robust RL are about. 

I recognize that a large part of my criticism has to do with the framing of the paper. I think if the thesis of this paper were modified to be less "all encompassing", and more along the lines of the more specific thesis of robust RL (that when seeking to transfer RL algorithms to similar scenarios, we should change our approach to training/evaluation), that might fix the issues. But in that case, the recommendations may need to be modified. To maintain the current thesis, I would want to see the paper demonstrate an advantage of using families of similar tasks, over suites of different tasks. I suspect such an advantage would be hard to demonstrate. Alternatively, the authors could demonstrate that the recommendation is easy to implement, by modifying one of the suites themselves (Mujoco control suite is probably easiest since there are only a few environments --- would it change the usual empirical results; e.g., re: DDPG, TD3, SAC, or model-based RL?) 

Clarity: Overall the paper is well written / clear.  Why is a suite of different tasks insufficient, as opposed to a family of tasks?  Yes

Meta-review: This paper proposes to evaluate deep RL algorithms on a family of MDPs, rather than single 'point' MDPs.  This seems a reasonable way to reduce performance variance, though I found the paper to be quite lacking in terms of depth, realism, and real impact.

In particular, the construction of the family of MDPs seems like a hard problem in general, which does not get a deep treatment in this paper. For instance, for many 'point' MDPs that might actually be of real-world interested it might be quite easy to construct a family of MDPs in a seemingly reasonable way, but for which change the conclusions in such a way that they do not generalise to problems of actual interest anymore.  Furthermore, as also raised by reviewers, if the goal is to evaluate generality of an algorithm, it would seem much better to evaluate on a benchmark of carefully-chosen (e.g., actually-interesting) diverse point MDPs than on a family of similar MDPs constructed around a single point.  Finally, many 'point' MDPs might contain substantial randomness themselves, and the distinction and similarlities between sampling multiple random (task) seeds on a single point MDP versus sampling from a family of MDPs is not sufficiently discussed.

Despite these limitations, I think it is good for the community to engage into discussions on how best to evaluate its methods.  I will therefore take the recommendation of the reviewers, and accept this paper.  I hope the comments on limitations above will resonate with the authors, or might inspire reasonable pushback about parts on which I'm perhaps being naive, given the authors will likely have thought about these issues a great deal.

Decision: Accept