The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning
Authors: Vindula Jayawardana
Catherine H Tang
Sirui Li
Dajiang Suo
Cathy Wu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Evaluation
Scientific Progress
Reliability
Benchmarking

Abstract: Evaluations of Deep Reinforcement Learning (DRL) methods are an integral part of scientific progress of the field. Beyond designing DRL methods for general intelligence, designing task-specific methods is becoming increasingly prominent for real-world applications. In these settings, the standard evaluation practice involves using a few instances of Markov Decision Processes (MDPs) to represent the task. However, many tasks induce a large family of MDPs owing to variations in the underlying environment, particularly in real-world contexts. For example, in traffic signal control, variations may stem from intersection geometries and traffic flow levels. The select MDP instances may thus inadvertently cause overfitting, lacking the statistical power to draw conclusions about the method's true performance across the family. In this article, we augment DRL evaluations to consider parameterized families of MDPs. We show that in comparison to evaluating DRL methods on select MDP instances, evaluating the MDP family often yields a substantially different relative ranking of methods, casting doubt on what methods should be considered state-of-the-art. We validate this phenomenon in standard control benchmarks and the real-world application of traffic signal control. At the same time, we show that accurately evaluating on an MDP family is nontrivial. Overall, this work identifies new challenges for empirical rigor in reinforcement learning, especially as the outcomes of DRL trickle into downstream decision-making.

The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning
Vindula Jayawardana MIT vindula@mit.edu
Catherine Tang MIT cattangomit.edu
Sirui Li MIT
siruil@mit.er edu
Dajiang Suo MIT djsuoθmit. edu
Cathy Wu MIT cathywu@mit.edd edu
Abstract
Evaluations of Deep Reinforcement Learning (DRL) methods an integral part of scientific progress of the field. Beyond designing DRL methods for general in- telligence, designing task-specific methods is becoming increasingly prominent for real-workd applications. In these settings, the standard evaluation practice involves using few instances of Markov Decision Processes (MDPs) to represent the task. However, many tasks induce large family of MDPs owing to variations in the underlying environment, particularly real-world contexts. For example, in traffic signal control, variations may stem from intersection geometries and traffic flow levels. The select MDP instances may thus inadvertently cause overfitting, lacking the statistical power to draw conclusions about the method's true performance across the family, this article, we augment DRL evaluations to consider parame- terized families of MDPs. We show that in comparison to evaluating DRL methods on select MDP instances, evaluating the MDP family often yields a substantially different relative ranking of methods, casting doubt what methods should be considered state-of-the-arr We validate this phenomenon in standard control bench- marks and the real-world application of traffic signal control. At the same time, we show that accurately evaluating on an MDP family is nontrivial. Overall, this work identifies new challenges for empirical rigor in reinforcement learning, especially as the outcomes of DRL trickle into downstream decision-making.
Introduction
Deep reinforcement learning research has progressed rapidly in recent years, achieving super-human level performance in many applications. At the core of DRL research lies the need to engage in rigorous experimental design for conducting evaluations. The lack rigorous experimental design could lead to profound implications. Researchers may inadvertently mislead themselves and draw incorrect conclusions about DRL including what factors contribute to the success of method [17 what factors make the results of method reproducible [21]. whether method successfully solves one task 45] or multiple tasks 1 26 As evidenced by these findings, strong empirical rigor is crucial for research progress as it allows the research community confidently assess the current state of the field.
The real world induces many complexities for control tasks, and major complexity is the existence of multiple instances of the same task. Consider the case traffic signal control task where the goal is to design signal control strategy for intersection. To reliably claim DRL method solves the traffic signal control problem, one needs to show that the proposed method sufficiently works
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

for a considerable majority of the signalized intersection instances 14]. We see this requirement of evaluating on a family of instances as an emerging requirement in general. not just limited to traffic signal control, specifically as DRL trickles into real-world applications. We refer to such evaluations as assessing the algorithmic generalization of DRI, methods within a task.
However, many studies that evaluate algorithmic generalization of DRL methods within a task ignore this requirement. In traffic signal control, the use of few select intersection instances popular for evaluations
44]. Similar discrepancies can also be seen in other application areas such as in autonomous driving [42 251, healthcare [4 and chemistry [ Such practices may fail to guard against evaluation overfitting 45), in which the DRL method under evaluation records misleadingly high performance by overspecializing to the evaluation instance. As DRL trickles into real-world implementations and critical use cases, not having such practices can have serious implications, ranging from the opportunity cost of employing lower quality methods to hurting confidence in the field.
(a) Default MDP
(b) MDP Family
Figure 1: An illustrative example of a). a default point MDP b). family of point MDPs specified using some environment parameters ø for evaluations of traffic signal control.
To formalize, consider the traffic signal control task, which is typically represented single MDP (Figure Tap. At the same time, the task is underspecified and could easily be represented by family of MDPs (Figure lb), each specified using some environment parameters such as the number of lanes, turn contigurations, and traffic inflow levels. Standard evaluation practice for assessing algorithmic generalization of DRL methods within the task of traffic signal control is train and evaluate DRL methods on point MDP (or a subset of MDPs) chosen from an implicit family of MDPs (e.g., selecting a specific の). We refer to such evaluation approaches as point MDP-based evaluations Unfortunately. such point MDPs often appear to be selected arbitrarily or in way that inadvertently simplifies the task. It is well known that the performance of a DRL, method is sensitive to the underlying MDP 36 21 2 In this work, we hypothesize that the coupling of an arbitrary selection of point MDPs for evaluation and the general sensitivity of DRL methods can result in substantial error in the true performance of DRL methods over the implicit MDP family of a task. This, in turn, harms empirical rigor in DRL. We refer this phenomenon as the task underspecification problem in DRL evaluations.
In this work, we take a deep dive into this matter. Our main contribution identifying an emerging problem motivated by the real-world application of RL: when a point MDP is used in evaluating DRL methods within task, the point MDP does not exhibit adequate statistical power to draw conclusions about the corresponding MDP family. We demonstrate that this phenomenon exists not only in weal-world applications such as traffic signal control but also in standard control benchmarks, which indicates that it may be prevalent across the field. Our methodology consists of augmenting DRL evaluations with appropriately parameterized families of MDPs. We show that in comparison to applying DRL methods to the point MDP, evaluating the MDP family often yields substantially different relative ranking of methods, which could lead researchers to draw incorrect conclusions about the performance of some methods over others. We demonstrate this phenomenon experimentally in a case study of traffic signal control. We show that DRL methods which were originally reported as outperforming traditional traffic signal control methods, significantly underperform when family of MDPs is used for evaluation, causing a substantial change in the ranking of methods.
Unlike standard DRL benchmark suites, in which the suite designer has full control over how many MDP instances are included in the suite 27 in real-world applications, the domain dictates how
We note that point MDP- -baeed evaluations are not just limited to MDPs but applicable to all variants of MDP like partially observable MDPs. We use point MDP as general terminology to refer all such cases.
2

---

many point MDPs are in a task. Whereas benchmark suites often have dozens of MDP instances, we estimate that a task can easily have hundreds or thousands point MDPs. This thus illuminates new computational and reporting obstacles for evaluating DRL methods within task. We provide some initial studies on the statistical power of evaluating DRI. methods by sampling from the MDP family and use of performance profiles for reporting and conclude that it is a nontrivial problem owing to the sensitivity of today's DRL methods.
Important distinctions: To put our contributions into the context of standard DRL practices, in this work we neither address the evaluations of algorithmic generalization across task families (standard DRL evaluations using task suites) nor evaluations of policy generalization within a task family (as addressed in multi-task, robust RL). Second, we focus on problems where a separate individual DRL model can be trained for each MDP (i.e., to achieve overall better performance). Another parallel line is in which single agent is designed to perform well on all MDPs of the family (e.g., multi-task learning in robotics). However, in the scope of this work, we do not consider such problems.
In the following sections, unless otherwise stated, by evaluation, we mean evaluations in DRL when evaluating algorithmic generalization of DRL methods within a task.
2 Shortcomings of Point MDP-based Evaluations
We illustrate the shortcomings of point MDP-based evaluation by considering experiment derived from popular DRL benchmarks. We use three popular control tasks (Pendulum, Quad [23], and Swimmer) as example underspecified tasks. For each task, we augment the nominal task induce family of MDPs. A summary of the tasks is given in Table along with their augmentations. For each task family,
then specify five point MDPs. More details of the tasks and their corresponding modifications are given in the Appendix [.I] We consider three popular DRL algorithms (PPO [8). TRPO [37 and TD3 [9] for evaluation on the tasks. We have verified that each modified point MDP is not under-actuated and that given actuator limits, all are solvable.
Table I: Control task and the MDP families.
Task
Task description
MDP family
Quad
2D quadcopter in an obstacle course.
MDPs with varying obstacle course lengths (upper obsta- cle length and lower obstacle length).
Goal: maneuver a 2D quadcopter through an obstacle course using its vertical acceleration the control action.
Pendulum
* A pendulum that can swing.
MDPs with varying masses and lengths of the pendulum.
Goal: swing the pendulum upright.
Swimmer
MuJoCo 3-link swimming robot in a viscous fluid. Goal: make the robot swim forward as fast as
MDPs with varying capsule sizes for the segments compris-
possible by actuating the two joints.
ing the robot swimmer.
Observation 1: Reporting evaluations based on point MDPs can be misleading.
Previous works commonly use three criteria for selecting a point MDPs for evaluations. 1) rundom MDP: a random MDP is used to model a somewhat arbitrary selection of an MDPs from the family (e.g., random selection of a signalized intersection for training a traffic signal control agent 44) 2) generic MDP: an MDP that represents the key characteristics of the MDPs in the family (e.g.. the use of generic cancer progression model for training chemotherapy designing agent 46) and 3) simplified MDP: an MDP that simplifies the modeling (c.g., zero-error instrument modeling when training an agent for chemical reaction optimization 48).
In Figure 2 we report the evaluations of the DRL methods on each point MDP described in Table ! for the three control tasks. That is, each method is trained and evaluated on each individual point MDP. Given an MDP family, child MDPs are considered possible random MDPs. For each task, * denotes the generic MDP, and represents the simplified MDP. The simplified MDP is chosen as the "easiest" MDP, i.e. simplifies the transition dynamies. The generic MDP the parametric mean MDP of the other four MDPs in the family.
3

---

PPO-simphileed
PPC random
TRPO-Simplii led
TRPO
TRPO-random
PPO simplified
PRO-generic
random
TD3 random TD3-generic
TD3 simoliied
PPO random PTO-gemark
PPO simplitted
TRPO-random
mPD-gemeet TRPO siplltted
Normalized lized score
1.00
Time reach gachhgaal goal
200
Time to reach goal
10000
0.75
100
5000
0.50
0.25
(0.6)*
(0.5.3)
(0.625.4) 4)
(1.2)
(1.5)
(1.1)
(1.5,4)
1.875,5.252..
(3.10)
L057.05009)
(1.1AA)
(1/1125/3))
(1/.1511)
155.15/.15
Point MDP
Point MDP
Point MDP
(a) Quad
(h) Pendulum
(c) Swimmer
Figure 2: Performance comparison of point MDP-based evaluations of the three control tasks. The x-axis represents the five point MDPs. For Quad, performance a normalized score of the distance the quadcopter traveled before crashing or reaching the goal with respect to the total distance required to travel (higher the better). For the pendulum, it is the time to swing the pendulum upright (lower the better), and for the swimmer, is the time reach the goal (lower the better).
We observe a few interesting phenomena (1) the same method yield significant differences in performance under different point MDPs for all three tasks, (2) simplified MDPs generally achieve better performance than other point MDPs in the family, and (3) comparing DRL methods based on point MDPs can provide conflicting conclusions (e.g., conflicting relative performance benefits) based on which point MDP is used for evaluations. For example, for Quad, point MDP (0.6) indicates both PPO and TRPO are equally well-performing. However, under point MDP (0.5.3), we see that PPO outperforms TRPO with approximately 0.75 point difference. These observations highlight the variability of point MDP-based evaluations and the uncertainties involved. Such evaluations could mislead the community incorrectly conclude that one method is better than another and thereby hinder the scientific progress of the field.
Observation 2: DRL training can be sensitive to the selected point MDP properties. It is generally known that DRL training is sensitive to the underlying MDP. Therefore, selecting a point MDP from family can demonstrate a training impact that does not generalize to other MDPs in the family. Subsequently, the performance of the DRL method could be overestimated or underestimated.
PPO HOP: 425
PPO MOP: (0.61!
PPO MMP: EL 22875,2255
PRO NEP 17.6
PPD NDP (4)
mno MOI::
PPO MDD: (0.5.3)
PRO1-M9S MMP (1.,0)
ITO- MOD 12,23
(20
MDP:
mo M-37:
IPO PPO-MOP:DD0 1707-13.80 13,101
alconed
(9)
stapa allowed
Time to reach the goa)
200
Steps to reach the goal
5000
Normalized Score
0.8
300
3000
0.2
100
2000
0.0
6
5000
10000 Training Episodes
15000
5000
10C0:
15000
20000
1000
2000 2500 Training Episodes
1000
1500
Training Episades
(a) Quad
(b) Pendulum
(c) Swimmer
Figure 3: Training progress of each task for different point MDPs. For Quad, performance is normalized score that indicates the distance the quadcopter traveled before crashing or reaching the goal with respect the total distance required to travel (higher the better). In the pendulum, the training performance measured as the time to swing the pendulum upright (lower the better). For the swimmer, the training performance is measured the time reach the goal (lower the better).
We demonstrate this phenomenon using our three control tasks in Figure 3 when using the PPO algorithm for training First, in Quad (Figure Baj. reaching the goal means achieving normalized
Under some point MDPs, not all parallel runs succeed. For point MDPs where some runs succeed, we only plot the runs that succeed. For point MDPs where all runs fail, we plot them as it is. We fix the number of total training steps 5M for swimmer and 2M for quad and pendulum. Curves are truncated for better visibility.

---

score of 1. However, only under a few of the point MDPs, the agent reaches the goal during training. Specifically, out of 5 settings converge to local minimum (crashing on the same obstacle as training progress). Similarly, in Figure si for the pendulum, under one of the MDPs the agent does not achieve the goal (stuck at the 200 steps mark) while under the other four, the agent achieves the goal. Finally, we see similar behavior in Swimmer task in Figure 3c where under one point MDP the agent fails to achieve the goal (stuck at the 5000 steps) during training and succeeds under others. Further examples of training complications under other DRL algorithms are given in Appendix B.3
We hypothesize one of the root causes of this phenomenon is the complexity differences in point MDPs, For example, in Quad, making the obstacle- frree course narrower requires the DRL agent to explore some specific actions different from what would otherwise need to explore when the course is wider. This effectively makes the training process harder for point MDPs with narrower paths. Another potential cause MDP designers' over-fitting MDP design to point MDPs. For example, in the pendulum, the default reward function penalizes higher torque values while encouraging reaching the goal with a weighted composite reward. MDP designers may over-fit the weighting values to a selected point MDP which does not generalize to the entire family. This can result in point MDPs with higher pendulum mass failing if the weight on the torque limits is higher.
3 Notation and Formalism
Given sequential decision- makinn task T and a reinforcement learning method R to be evaluated, we consider the setting where there are M possible point MDPs in the family of MDPs. In general, despite the illustrative examples in the previous section, M can be quite large. Let N denote computational budget in terms of the number of models that can be trained] We assume that M >> N: that is, we cannot evaluate the given RL method on all MDPs in the family.
As we demonstrated in Section 2 the performance of DRL method R can significantly depend on the choice of the point MDP. As suggested by previous work M. it is therefore warranted to model the performance of R as a real-valued random variable Xn. This means a normalized performance score S1,i for given R and point MDP is realization of the random variable Xn. We normalize point MDP performance scores by linearly rescaling scores based on a given baseline. For example, scores in Atari games are typically normalized with respect to average human 32
Given a family of MDPs, a point MDP i may be more important or common than another point MDP j. This is a common requirement in the real world where practitioners have predefined performance requirements, such as the performance of a signalized intersection in an urban area may be more important than a signalized intersection in sub urban area in traffic signal control [Л]. This importance score PT, of a point MDP i on task T can be considered as a realization of a real- valued random variable Yr. This means depending on the importance scores of each point MDP, distribution can be generated for the MDP family. For T E I where T represents the point MDP context, therefore define the point MDP distribution as F(r) = P(Yr).
Assumption: For given task T, we assume PT. is given for each MDP
Definition 1
The overall performance of a method R. on task T is defined as ETi = [[x]] =
E4 SR,PTT where U is the set of point MDPs
However, obtaining F is not always possible because of the budget constraint M > N. Therefore, a potential solution is to select subset V of point MDPs from the MDP family to perform an evaluation. Accordingly, the estimated performance of the method R on task if not careful, selecting different subsets can greatly affect the accuracy of the evaluations.
E7 = =1
V 8R,PT, Clearly,
An intuitive approach to selecting a subset of point MDPs is to assess the contribution o; of each point MDP i to overall evaluation. Contribution depends both on the importance score, which is given, and performance score, which incurs computational cost to evaluate, and can be defined as = sP.,, Thus, we seek to find the set of point MDPs that has the highest contributions to the overall evaluation. Given that we wish to estimate the contribution of individual point MDPs without
"oost cloud service providers charge users based on the time they use the services. Therefore, approximating the average number of models that can be trained given a pricing budget can be done with rough estimates of how long it takes to train one model on single point MDP.
5

---

assessing the overall performance Egg this poses chicken-and-ege problem. In the subsequent sections, we propose approximation techniques that one can employ to identify subset V based on the approximated contributions.
4 Case Study: Traffic Signal Control
To validate the shortcomings of point MDP-hased evaluations, we consider an established henchmark that exhibits large implicit family of MDPs describing a single task and wherein there could be significant real-world implications. In particular, we consider the evaluations of DRL methods on the traffic signal control task, leveraging the RESCO benchmark
4 We use six algorithms from RESCO in our case study, namely: IDQN, IPPO B MPLighl TC MPLight", Fine-tuned Fixed time and Max pressure 4T. Further details can be found in Appendix
Naturally, traffic signal control should be considered on multiple intersection geometries and vehicle flow levels, hence family of MDPs. We base importance score Pr, of each intersection on the frequency of occurrence within a geographic region and the performance score sn as the normalized per vehicle average delay. Scores are normalized based fixed time controller baseline. The importance scores and the intersections used to build the point MDP distribution of the intersections taken from Salt Lake City in Utah. We 164 unique intersections and refer the reader to Appendix for more details on building this distribution.
untuned yet sufficiently performant
10
IDQN
Reported
10
IPPO
Reported
Number of MDPs
10
MPLight
Reported
10
MPLight* Reported
10
Fixed Time Reported
10
Max Pressure
Reported
0.5
1.0
,
2,0+
4.0+
5.0+
5.5+
6.0+
7.5+
Normalized Score (lower the better)
Figure 4: Performance vs the number of point MDPs that demonstrate the performance for all traffic signal control methods in RESCO. 164 unique point MDPs were considered for each method.
report the performance of each of six methods 164 unique point MDPs and the reported performance as per 4 First, we observe significant variations in the performance based on the point MDP used for evaluations. Specifically, all DRL methods demonstrate a significant variation while non-DRL. methods demonstrate comparatively low yet considerable variations. Second, we observe that the reported performances in related literature can be significantly biased. As example, IDQN, IPPO and MPLight performances are clearly overestimated. To quantitatively analyze the potential shortcomings, denote the performance of each method over the entire MDP family in Figure 5
In Figure T
'Reported performances based re-evaluations of the methods on Ingolstadt single intersection.

---

Overall performance
Interestingly, we see significant result change from previ- ously reported results, Although the Fixed time controller is regarded as an underperforming method as per reported performance in 4. we find that under the MDP camily-based evaluations, the non-RL Fixed time controller and Max pres- sure controller perform significantly better than all the four DRL controllers. We note that fine-tuning a non-RL Fixed time controller is simple enough that it does not pose com- putational burden and can be done easily, even on a regular computer. While reported performances ranked IPPO as well-performing model with a normalized score of 0.84, we see that, in fact, it is the powest-performe the revised normalized score is as high as 1.7 (even cannot
Normalizez Score
(ooerr the betten)
1:50
Reported performance.
1.25
1.00
0.75
0.50
IDGN
IPPO
MPLight
MPLight*
Fixed Time
Max Pressure
Method
Figure 5: Overall performance (lower the better)
method and that
outperform an untuned fixed time controller). It is thus clear that point MDP-based evaluations can be misleading and may pose performance benefits that do not generalize to the MDP family.
5 Further Evidence on Shortcomings of Point MDP-based Evaluations
To further validate the shortcomings of point MDP-based evaluations and the impact, look at three popular DRL control tasks: cartpole (discrete actions), pendulum (continuous actions), and half-cheetah (continuous actions). For each task, we devise a family of MDPs described in Table in Appendix [.I Our MDP families consist of 576, 180, and 120 point MDPs for cartpole, pendulum, and half cheetah, respectively. Due to space limitations, we provide an in-depth analysis of performance variations in Appendix [.2] and only provide a summary of the analysis in this section. In Figure g we show the significant discrepancies in overall performance and the reported performances. The reported performance of each task is measured by training and evaluating DRL methods on commonly used single point-MDP given common benchmark suites.
We see significant result changes when evaluated on the point MDP family. For example, in cartpole (Figure ba). all reported performances achieve the best score of 1.0 while we see significantly different overall performance. Although under reported performance, all four DRL methods for cartpole are ranked as equally cell-performigg we see an interesting rank change as some methods underperform when considering their overall performance. Similar insights can be seen in Pendulum (Figure
6b
and in half cheetah (Figure bc).
1.0
Overal
Blower the better)
Getral performnce Reportel perforence
Normalized Score
0.9
Normallzed Score
Elwwrr the bettert
1:25
Aepertes porormare
Normal (red Score
1.5
0.8
1.0
0.7
0.75
0.6
0.5
0.5
0.75
0.0
AR5
PPO
TRRO
DON
TD3
PRD
TRPO
SAC
TD3
PRO
TRPO
sMc
Method
Mathad
Method
(a) Cartpole (576 point MDPs)
(b) Pendulum (180 point MDPs) (c) Half Cheetah (120 point MDPs)
Figure 6: Discrepancies between reported vs. overall performance of popular DRL methods in
cartpole. pendulum and half cheetah when evaluated for algorithmic generalization within the task.
6 Reliable Evaluations Within Task
In Section and we demonstrate the shortcomings of point MDP-based evaluations. In this section, we discuss the challenges that arise as a result of conducting MDP family- based evaluations. We present three main challenges and an initial set of recommendations to the research community as summarized in Table 2
'Results reported in this work should not be illustrated as evidence against using DRL for traffic signal control and should only be used evidence of shortcomings in point MDP-based evaluations. Further studies encouraged study the benefits of DRL for traffic signal control without the point MDP-based assumptions.
7

---

Table 2: Summary of challenges and initial recommendations
Challenge
Our recommendation
Lack of benchmarks
Create benchmarks that depict MDP families.
Publish datasets of MDP families of control tasks including point-MDD distributions.
Incentivize publication of such datasets and control task leading conferences.
Large families of MDPs with limited computational budgets
Adopt performance approximations using clustering and ran- dom sampling under a computational budget. Standardize the evaluations by making the selected point- MDPs public.
Lack of emphasize on all
Use performance profiles to show detailed view of how overall performance changes with point MDPs
Point-MDP performances
6.1
Create data sets of MDP families to benchmark RL methods in evaluations
The use of benchmark datasets for evaluations well-established supervised learning. From computer vision [122 and biological data [2 to natural language processing 355] standard datasets are being used for evaluations of deep learning methods. In DRL, the practice is different. In assessing algorithmic generalization of DRL methods within a task, instead of published datasets, researchers use standardized benchmark suites like RESCO for traffic signal control and Vinitsky et al. 42 for mixed autonomy traffic.
However, we recognize that current state-of offte-of-the-arr benchmarks alone are not sufficient to standardize the evaluations in DRL. The main limitation is that they only provide select point MDPs and do not consider the family of possible MDPs. As many real-world applications inherently demonstrate a family of MDPs, the current DRL standardization of evaluations may steer the research community in vacuum, while realistically useful DRL. methods may get rejected or not even developed.
Therefore, our first recommendation is to create DRL benchmark suites which inherently demonstrate requirement to incorporate family of MDPs. However, which MDPs include in a given family is task-dependent, and the expertise of domain experts may be needed. Second, we also encourage the research community to actively create MDP families for existing control tasks (e.g., different signalized intersections as traffic signal control MDP family) and publish them publically. Finally, we also encourage main artificial intelligence conferences with datasets and benchmark tracks like NeurlPS to encourage the community to publish such datasets and tasks and to include necessary check-ins in the paper submission checklists.
6.2 Evaluate DRL methods family of MDPs instead of point MDPs
In Section next recommendation is to encourage researchers to use families of MDPs instead of point MDPs in DRL evaluations. However, evaluating performance over an entire family of MDPs can often be computationally expensive to carry out in practice due to the large cardinality of the family. A solution is to conduct performance approximations. While more sophisticated methods like active learning approaches are possible, we resort to effective yet straightforward techniques to bolster the adoptability of the techniques within a wider community.
Al and 5 we demonstrated the shortcomings of point MDP-based evaluations. Therefore,
We present three techniques: (1) M1: random sampling with replacements from the point MDP distribution, (2) M2: random sampling without replacements, and (3) M3: clustering point MDPs using k-means and assigning probability mass of all point MDPs that belong to same cluster to its centroid. More details of each method and more analysis can be found in Appendix E.
In Figure 7 we denote the approximate evaluations of IDQN, IPPO and Fixed time methods in traffic signal control in comparison to the ground-truth performance while varying the budget size. In general, k-means clustering- approximation produces better estimates than the other two techniques with smaller standard deviations. Specifically, if the budget size is half the total MDP family size, k-means clustering demonstrates reasonably accurate performance estimates. Also,
8

---

Overall performance
Overall
Overall performence
H2
-
M3
M1
M3
-
MI
N3
jormalzeed performene formence
coraalzee performanty
0.84
Approiimatoo
Approsimated
Appresimatted
0.82
cormoleed
1.5
0.80
75
125
75
200
325
lable (number models)
Available (humber
Available budget Inumber of models)
(a) IDQN
(b) IPPO
(c) Fixed Time
Figure 7: Approximated performance evaluations family of MDPs in traffic signal control with
varying budget limits. Closer to the overall performance the better.
publishing the selection of point MDPs data sets enable reproducibility and standardization of the evaluations of the tasks. A sensitivity analysis of the proposed techniques to the underlying point MDP distribution is also given in Appendix E.2
Remarks. We acknowledge that there are other factors to consider for a computational budget, including hyperparameter tuning (26) and training using multiple random seeds within each point MDP [1. Here, we focus on the computational budget allotted for the task underspecification issue.
6.3 Use performance profile of the MDP family instead of point MDP performance
Inspired by the idea of performance profiling for DRL evaluations and in optimization soft- ware [3,, our final recommendation is to report the performance of a DRL method over a family of MDPs as a performance profile. Although the overall performance of a DRL method over family of MDPs can yield more reliable evaluations than evaluating on a point MDP, it may encapsulate further insights into the method's performance. By representing the performance of method as performance profile, more detailed visualization of the performance over the point MDPs can be illustrated.
In Figure 8 we present an example performance profile for k-means clustering basterriggbbsed performance approximation with budget size 80 models in the traffic signal control task.
MDP probabiltiis Camulative probabiliiis
HPUght MPLight
A performance profile illustrates what point MDPs are most probable in the distribution and how each of the point MDP contributes to the final estimate of the performance. It enables direct comparison of methods. particular, if the cumulative performance curve of method A strictly below method B, method A is said to stochastically dom- inate method better) [I]. Furthermore, point MDPs in which a method seemingly under- performs is easily visible, giving clear overview of where the limitations and strengths are originating from.
IPPO
Max Pressure
Probability / Cumulative Probability
1.0
2.0
score (lower the b .
1.8
0.8
1.6
C (aai nor F
0.6
1.2
1.0
0.4
0.8
0.6
0.2
0.4
0.2
(lower the score the
0.0
0.0
20
40
60
80
Point MDP ID
Figure 8: Performance profile of k-means clustering based performance approximation with budget size of 80 models in traffic signal control task.
7 Related Work
Deep reinforcement learning algorithms have notoriously high variance, resulting in reliability and reproducíbility issues when applying them to real-world applications [ 9 S. Designing sound methodologies for conducting performance evaluations is, therefore, critical. To our knowledge, Falkenauer [1]] first identified overfitting design to a selected problem instance as cause for concern. Whiteson et al. 45 later used the same motivation to argue similar overfitting happen,

---

particularly in reinforcement learning. We look at the algorithmic generalization of DRL methods within a task, which poses different requirements and properties compared to previous works.
The use of a family of problem instances is not new. It has been used in combinatorial auctions 30 and in reinforcement learning 7 23
Recently, MDP families have been used to achieve better generalization in DRL. Benjamins et al. d argue that generalization in DRL is held back by factors stemming part from lack of problem formalization. Benchmarks suites such as CARL [3] are proposed to use MDP families to study generalization. Eimer et al. [ further show that such DRL problems demonstrate challenges that the DRL community has not looked at carefully.
The family of MDPs has been formally modeled in multiple ways in the literature. The most recent and general method is to model the problem where all MDPs in the family share the same MDP configuration except for the transition function and the reward function. In Hidden Parameter MDPs [4,, only the transition function changes but keep the reward function fixed over the family of MDPs. Epistemic POMDPs introduced by Ghosh et al. 20 are a special case of a cMDP where the context is assumed to be unobservable. They show that there is implicit partial observability under generalization to unseen test conditions from a limited number of training conditions. This phenomenon translates even fully observed MDP into POMDP. In comparison to these formulations, we do not restrict what components of an MDP can or should change evaluations. We let that be defined by the domain of the task.
Contextual Markov Decision Processes (cMDP) K
8 Conclusion and Future Work
In this work. we identify an important yet overlooked issue of task underspecification in DRL evaluations- the reliance of reporting outcomes on select point MDPs. We experimentally demon- strate that evaluating the MDP family often yields a substantially different relative ranking of methods compared to evaluating on select MDPs. Moreover, evaluating and is faced with multiple challenges. One exciting avenue for future work to explore if similar shortcomings occur when evaluating the algorithmic generalization of DRL methods across tasks. Furthermore, our recommendations for the related challenges when conducting reliable evaluations with family of MDPs are only starting point for more focused research. Future research can shed light on these directions, including designing efficient yet effective methods that can produce good approximate performance estimates. Overall, we intend for our findings to raise awareness of task underspecification that impacts the empirical rigor of DRL and aim to help move the needle toward a more disciplined science overall.
family of MDPs is not trivial
Broader Impact
Although there is not a single definition of responsible machine learning, the Institute for Ethical AI & Machine Learning has developed a series of eight principles to guide the responsible development of machine learning systems. Practical accuracy is one of the principles, which emphasizes that accuracy and cost metric functions are aligned to the domain-specific applications. This article contributes to bolstering the practical accuracy of RL, when employed for a complex downstream decision (e.g. whether to adopt an RL method for a societal system), by highlighting some limitations of current standard RL research practices and proposing to explicitly consider family of MDPs that constitutes the complex decision. On the other hand, even with more empirically rigorous ML practices, there are still subjective aspects of the task distribution (analogous to the family of MDPs in RL), so it remains important to not over-index on RL-based evaluations.
10 Acknowledgments
This work was supported by the MIT Amazon Science Hub, the US DOT's Federal Highway Administration and Utah Department of Transportation under project number F-ST99(783), the MIT-IBM Watson AI Lab, gift from Mathworks, and the National Science Foundation (NSF) under grant number 2149548. The authors acknowledge MIT SuperCloud and the Lincoln Laboratory Supercomputing Center for providing computational resources supporting the research results in this paper. The authors are also grateful for insightful discussions with Jiaqi Zhang and the constructive suggestions from the anonymous reviewers.
10

---

References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
[2] Marcin Andrychowicz, Anton Raichuk, Piotr Staüczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin. Marcin Michalski, et al. What matters on-policy reinforcement learning? large-scale empirical study. arXiv preprint arXiv:2006.05990 059900 2020.
[3] James Ault, Josiah P. Hanna, and Guni Sharon. Learning an interpretable traffic signal control policy. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '20, page 88- 88-96, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems.
[4] James Ault and Guni Sharon. Reinforcement learning benchmarks for traffic signal control. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2021.
[5] Carolin Benjamins, Theresa Eimer, Frederik Schubert, André Biedenkapp. Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Carl: A benchmark for contextual and adaptive reinforce- ment learning. In NeurlPS 202/ Workshop Ecological Theory of Reinforcement Learning, December 2021.
[6] Caroline Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Andr'c e Biedenkapp. Bodo Rosenhahn, Frank Hutter, and Marius Thomas Lindauer. Contextualize me the case for context in reinforcement learning. ArXiv, 0bs/2202.045000 2022.
[7] Shalabh Bhatnagar, Mohammad Ghavamzadeh, Mark Lee, and Richard S Sutton. Incremental natural actor-critic algorithms. Advances in neural information processing systems, 20, 2007.
[8] Johan Bjorek, Carla P Gomes, and Kilian Q Weinberger. Is high variance unavoidable in ri? case study continuous control. International Conference on Learning Representations, 2022.
[9] Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama. Measuring the reliability of reinforcement learning algorithms. International Conference on Learning Representations, 2020.
[10] Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang. Yuanhao Xiong, Kai Xu, and Zhenhui Li. Toward a thousand lights: Decentralized deep reinforcement learning for large- scale traffic signal control. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3414-3421 Apr. 2020.
[11] Janusz Chodur, Krzysztof Ostrowski, and Marian Tracz. Variability capacity and traffic performance at urban and rural signalised intersections. Transportation Research Procedia, 15:87-99, 2016. International Symposium on Enhancing Highway Performance (ISEHP), June 14-16, 2016, Berlin.
[12] Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé. Mot20: benchmark for multi object tracking in crowded scenes. arXiv preprint vrXiv:2003.09003 2020.
[13] Elizabeth D. Dolan and Jorge J. Moré. Benchmarking optimization software with performance profiles. Mathematical Programming, 91:201-213, 2002.
[14] Finale Deshi-Velez and George Dimitri Konidaris. Hidden parameter markov decision processes:
A semiparametric regression approach for discovering latent task parametrizations. IJCAI: proceedings of the conference, 201 6: 1:442-14400 2016.
[15] Utah DOT. Udot data portal, 2022.
[16] Theresa Fimer, Carolin Benjamins, and Marius Lindauer. Hyperparameters in contextual rl are highly situational. NewrIPS 2021 Workshop on Ecological Theory of Reinforcement Learning, 2021.

---

[17] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: case study on ppo and trpo. International Conference on Learning Representations, 2020.
[18] Emanuel Falkenauer. On method overfitting Journal of Heuristics, 4(3):211-2877 1998.
[19] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International conference on machine leurning, pages 1587-1596.
PMLR, 2018.
[20] Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P. Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. In Neural Information Processing Systems, 2021.
[21] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[22] Andrew Iyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. International Conference on Learning Representations, 2020.
[23] Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, and Armando Solar-Lezama. Synthesizing programmatic policies that inductively generalize. In International Conference on Learning Representations, 2020.
[24] Vindula Jayawardana, Anna Landler, and Cathy Wu. Mixed autonomous supervision in traffic signal control. In 202/ IEEE International Intelligent Transportation Systems Conference (ITSC), pages 1767-1773, 2021.
[25] Vindula Jayawardana and Cathy Wu. Learning eco-driving strategies signalized intersections. European Control Conference (ECC), 2022.
[26] Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the performance of reinforcement learning algorithms. In International Conference on Machine Learning, pages 4962-4973. PMLR, 2020.
[27] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, K. Czechowski, D. Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, G. Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari. Interna- tional Conference on Learning Representations, 2020.
[28] Shivaram Kalyanakrishnan and Peter Stone. An empirical analysis of value function lunetion-based and policy search reinforcement learning. Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems- Dystems-Volume 2. pages 749-756, 2009.
[29] Michael K. K. Leung, Andrew Delong, Babak Alipanahi, and Brendan Frey. Machine learning in genomic medicine: A review of computational problems and data sets. Proceedings of the IFFE, 104(1):176-197, 2016.
[30] Kevin Leyton-Brown. Mark Pearson, and Yoav Shoham. Towards a universal test suite for combinatorial auction algorithms. Proceedings of the 2nd ACM conference on Electronic commerce, pages 66-76, 2000.
[31] Open Street Map. Open street map, 2022.
[32] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 31775))):529-533 2015.
[33] Utah Department of Transportation. Automated traffic signal performance measures component details. 2020.
12

---

[34] Afshin Oroojlooy, Mohammadreza Nazari, Davood Hajinezhad, and Jorge Silva. Attendlight: Universal attention- based reinforcement learning model for traffic signal control. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[35] Yifan Peng, Shankai Yan, and Zhiyong Lu. Transfer learning in biomedical natural language processing: evaluation of bert and elmo on ten benchmarking datasets. Proceedings the 18th BioNLP Workshop and Shared Task, 2019.
[36] Daniele Reda, Tianxin Tao, and Michiel van de Panne. Learning to locomote: Understanding how environment design matters for deep reinforcement learning. In Motion, Interaction and Games, pages 1-10. 2020.
[37] John Schulman, Sergey Levine, P. Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. ArXiv, abs/1502.05477/ 2015.
[38] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1707.06333 2017.
[39] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq. Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. ArXiv, abs/1801 00//1801.0000 100690, 2018.
[40] UDOT. Aadt unrounded, 2019.
[41] Pravin Varaiya. Max pressure control of network of signalized intersections. Transportation Research Part C: Emerging Technologies, 36:177-195, 2013.
[42] Eugene Vinitsky, Aboudy Kreidieh, Lue Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre Bayen. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Conference robot learning, pages 399-409. PMLR, 2018.
[43] Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. CoLight. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. ACM, nov 2019.
[44] Hua Wei, Guanjie Zheng. Huaxiu Yao, and Zhenhui Li. Intellilight: A reinforcement learn- ing approach for intelligent traffic light control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2496-2505. ACM, 2018.
[45] Shimon Whiteson, Brian Tanner, Matthew E. Taylor, and Peter Stone. Protecting against evaluation overfitting in empirical reinforcement learning. In 20/1 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 120-127, 2011.
[46] Yufan Zhao, Michael R. Kosorok, and Donglin Zeng. Reinforcement learning design for cancer clinical trials. Statistics in Medicine, 28, 2009.
[47] Guanjie Zheng, Yuanhao Xiong, Xinshi Zang. J. Feng, Hua Wei, Huichu Zhang, Yong Li, Kai Xu, and Zhenhui Jessie Li. Learning phase competition for traffic signal control. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019.
[48] Zhenpeng Zhou, Xiaocheng Li, and Richard N. Zare, Optimizing chemical reactions with deep reinforcement learning. ACS Central Science, 3:1337 1344, 2017.
13


Reviews:
[{"review_id": "_s35tVibrQV", "reviewer": "Reviewer_Ybmr", "summary": "This paper presents the issue of task underspecification in evaluating reinforcement learning algorithms. This issues emerges when an RL algorithm is evaluated on a single instance of an underlying (often implicit) task family, and then that performance is implicitly assumed to be robust to the choice of instance in the task family. The paper shows that this robustness is often not the case, and in some cases the ordering between different algorithms can change when different task instances from the task family are chosen. Recommendations for ensuring a more robust evaluation of algorithms are provided, although the core problem of the computational burden of robust evaluations still somewhat remains (and is seemingly unavoidable).", "questions": "What do you mean when you say that \"not all parallel runs succeed\" in footnote 1? Do you mean they don't converge? If so, I think it's more representative to also plot the failing (i.e. non-converging) runs also, so as not to give a skewed picture of the performance of different algorithms.\n\nIn figure 1, is each algorithm trained once on each MDP, and then tested on all the MDPs labelled on the x axis? Also, it would be good to see which random MDP is chosen in each case, as that would explain the performance of each algorithm trained on this MDP.\n", "limitations": "I think a more explicit discussion of the fact that the experiments are limited to two specific domains (traffic control and small-scale continuous control) would be beneficial: either discussing this limitation, or addressing it through further experiments.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the issue of task underspecification, especially in the traffic control domain used, is a real one. The demonstration of this phenomena is valuable and significant. The paper is generally well-written and clear. I think the recommendations for more robust evaluation are beneficial, and the results in section 4 are interesting and beneficial to the research community, especially in the RL traffic control literature (which I'm less familiar with).\n\nTo me, the key issue arising from task underspecification is when the relative performance or ordering of different methods changes when making choices that were implicitly assumed not to change the relative performance or ordering. This is what is demonstrated very clearly in the traffic control domain, where the experiments demonstrate that when performing a more robust evaluation, the non-RL methods are preferred. Further, I think the results in observation 2 are unsurprising (different point MDPs are difficult); and we don't really care about absolute performance of algorithms on these toy benchmarks, as it's not that useful as information about how well they'll do on some other down-stream task. Performance relative to other algorithms (to enable an informed choice of which algorithm to use for a downstream task), or to a human expert on the task (so as to have a notion of whether the algorithm can outperform humans on some narrow domain), are more important.\n\nGiven this, it would be beneficial to highlight this precise phenomena in the other experiments. Figure 1 doesn't make it easy to see if choosing a different point MDP changes the preferred algorithm, or if it would also change if averaging over the performance across all point MDPs. More broadly, I'd prefer to see more of an emphasis and investigation of the possibility of this phenomena of reordering algorithms, for example another problem settings and with other algorithms. I imagine a key place where this could occur is in more complicated continuous control settings such as MuJoCo/DMControl/Brax. Investigating whether changing the physical parameters or agent morphology parameters (in a similar way to the experiments in section 2) also changes the SOTA algorithm would provide more evidence of the conclusions drawn in the paper.\n\nWhile this paper focused entirely on different point MDPs within an MDP family, a common method of evaluation is to consider a task family, such as a variety of MuJoCo, Atari or Procgen tasks, to ensure that evaluation of the algorithm is robust. A comparison of the discussion in this paper to works with that evaluation protocol would be beneficial; crucially, demonstrating that the problem of non-robust evaluation demonstrated in the toy settings and the traffic control setting aren't just specific to those settings, and that evaluation on a range of (point MDPs from) different tasks solves the problem that's caused by task underspecification in terms of robust evaluation.\n\nOverall, I'm currently recommending a weak reject, due to the fact that I don't think the scientific contribution of the paper is sufficient. I think the most interesting results are those presented in section 4, and if similar results could be demonstrated in other domains (as mentioned above) and with more algorithms, then I would raise my score to recommending acceptance. I realise this requires a high computational workload. If these results were demonstrated, and the recommendations for improving the robustness of evaluations were applied to them successfully, then I think the paper would be a robust and worthwhile contribution worthy of acceptance.\n\nEDIT: I thank the authors for their diligent efforts in responding to my comments. Given the new experimental results seem to validate this phenomena across a sufficiently wide variety of environments, I'll raise my score from 4 to 6.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EH7kMADdQoN", "reviewer": "Reviewer_bTHb", "summary": "This paper argues that RL algorithms should be evaluated on a suite of families of similar tasks (e.g., Mujoco benchmark that includes several parameterizations of Hopper with different limb lengths / weights / rewards, etc.) as opposed to a suite of individual tasks (e.g., Mujoco benchmark that includes only one particular parameterization of Hopper). It shows that the same algorithm can have different performance under different parameterizations of individual environments and argues that good evaluation in deep RL should therefore measure performance based on families of tasks. It provides recommendations for evaluating deep RL algorithms going forward, which include the creation of benchmarks that depict families of tasks.", "questions": "Why is a suite of different tasks insufficient, as opposed to a family of tasks? ", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: This is definitely an original take. The authors seem to adopt a single MDP approach (rather than a robust RL / domain randomization approach / OOD generalization approach), but assert that algorithms should be evaluated on several similar single MDPs. \n\nQuality/Significance: I was not persuaded by the authors that the problem they are addressing affects deep RL as a whole or that it can accurately be termed “task underspecification”, or that their proposed approach is necessary or helpful in the evaluation of deep RL algorithms (as opposed to the suites of tasks that they recommend modifying at line 209). I buy the Section 4 case example, as one instance where, *if we seek to make conclusions about the performance of RL algorithms on the family of tasks, then we should evaluate the RL algorithms on the family of tasks*, but this is an argument against relying on a single task to evaluate an RL algorithm, rather than an argument for using a family of tasks vs a suite of tasks. For this reason, it is difficult for me to give this a high score on quality and significance. My reasoning is as follows; responses appreciated.\n\n- Benchmarks already include a variety of tasks/MDPs. Not only do these tasks/MDPs differ, but they are quite diverse, with entirely different setups. It is unclear why evaluating on a family of similar MDPs (the “family of MDPs”) is superior to evaluating on a suite of tasks  (Atari has 57 games; do we need to add noise to all 57 to get a suite of 5700 games, where there are very similar sets of 100?). Of course, you can say it is because there are more tasks, but that would change the scope/thesis of the paper.\n- The tasks are not underspecified… they are precisely specified. Pendulum is a specific pendulum, with a particular mass and length. Of course they are chosen somewhat arbitrarily, but that’s the point… that a good RL algorithm should perform well on any reasonable/solvable MDP. Choosing a family of tasks would be similarly arbitrary, and may involve creating unreasonable/unsolvable MDPs.\n- There is insufficient evidence to support the hypothesis at lines 41-43. I do not see anywhere in Sections 2 where there is error in the evaluation due to the use of a point MDP rather than a familiar of MDPs. It is known that DRL methods are noisy, and that’s why we test them on a benchmark suite of several envs. On the other hand, Section 4 is a nice example, and while it is indeed “evidence of pitfalls in point MDP-based evaluations” (Footnote 4), I don’t see as providing sufficient support for the bolded hypothesis at lines 41-43, or the comment/recommendation at line 209. \n- Re: lines 99-106 about Figure 1... The observations (1)-(2) are obvious. In my view, the observation (3) simply lacks evidence. How many seeds were tested, where are the confidence bounds? In any case, noise in one task that shows up when evaluating it on the task family, would also show up when evaluating the algorithm on the full suite. Also it is unclear why TD3 is used in one figure, but TRPO in the others.\n- Section 2 Observation 2 would also apply to across tasks in a suite (they don’t need to be part of the same task family). \n- Line 193: Is anyone claiming that these RL algorithms generalize to the model family? I would think not, because that is what multi-task RL / robust RL are about. \n\nI recognize that a large part of my criticism has to do with the framing of the paper. I think if the thesis of this paper were modified to be less \"all encompassing\", and more along the lines of the more specific thesis of robust RL (that when seeking to transfer RL algorithms to similar scenarios, we should change our approach to training/evaluation), that might fix the issues. But in that case, the recommendations may need to be modified. To maintain the current thesis, I would want to see the paper demonstrate an advantage of using families of similar tasks, over suites of different tasks. I suspect such an advantage would be hard to demonstrate. Alternatively, the authors could demonstrate that the recommendation is easy to implement, by modifying one of the suites themselves (Mujoco control suite is probably easiest since there are only a few environments --- would it change the usual empirical results; e.g., re: DDPG, TD3, SAC, or model-based RL?) \n\nClarity: Overall the paper is well written / clear. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A42zzO5-trW", "reviewer": "Reviewer_QUf6", "summary": "This paper identifies a largely overlooked issue of task underspecification in the deep reinforcement learning (DRL) community: the performance of DRL approaches has been typically assessed on point MDPs, which are only special instances of the corresponding MDP families in many real-world problems. The paper empirically demonstrates that this selection bias can result in significant discrepancies in the relative performance of DRL approaches across different point MDPs in control benchmarks and a real-world traffic signal control problem, and suggests a series of recommendations for designing more accurate and efficient performance evaluation protocols.", "questions": "1. How are the parameters of different point MDPs selected for each MDP family in Table 2 in the appendix? Are they selected by some sort of randomization or according to the prior knowledge of the tasks?\n2. In general, what is the main advantage of random sampling from a family of MDPs with replacements over random sampling without replacements? My current intuition is that if different point MDPs are of sufficiently different probabilities when being sampled, then random sampling with replacements may be better than random sampling without replacements since the latter may excessively alter the inherent distribution of point MDPs. Are there other explanations?\n\n### Minor\n- Line 72: pifalls $\\rightarrow$ pitfalls\n- Table 2, last row: MPDs $\\rightarrow$ MDPs\n- Line 88 in the appendix: missing reference\n", "limitations": "No concern.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. This paper contributes to the well-known and important problem in the DRL community of performance evaluations of DRL algorithms.\n2. The identified task underspecification issue is interesting and not covered by prior work on performance evaluations of RL as far as I can tell.\n3. The paper is well-organized and well-written.\n\n### Weaknesses\n1. I am a little doubtful about whether the suggested three techniques (random sampling with or without replacements & $k$-means clustering) for evaluating DRL methods on a family of MDPs are generally practical. While they would clearly be better than the traditional protocol of evaluating DRL algorithms only on point MDPs, balancing the efficiency and the reliability of these evaluations (e.g., deciding the sampling number or the number of clusters) still seems to be a non-trivial task without sufficient domain knowledge.\n2. Compared with prior work on the evaluation protocols of RL algorithms, the evaluation in this paper is less extensive. I agree that using a real-world problem where RL algorithms are actually employed is beneficial for highlighting the pitfalls of the current protocol in realistic scenarios. However, given that the authors have argued that the task underspecification issue is prevalent in common DRL benchmarks, it would be great if more empirical results (e.g., on some of the environments listed in Table 1 in the appendix) are reported to show the generality of such pitfalls.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept