How to Enable Uncertainty Estimation in Proximal Policy Optimization
Authors: Eugene Bykovets
Yannick Metz
Mennatallah El-Assady
Daniel A. Keim
Joachim M. Buhmann
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Uncertainty Estimation
Out-of-distribution detection
Proximal Policy Optimizaiton

Abstract:   While deep reinforcement learning (RL) agents have showcased strong results across many domains, a major concern is their inherent opaqueness and the safety of such systems in real-world use cases. To overcome these issues, we need agents that can quantify their uncertainty and detect out-of-distribution (OOD) states. Existing uncertainty estimation techniques, like Monte-Carlo Dropout or Deep Ensembles, have not seen widespread adoption in on-policy deep RL. We posit that this is due to two reasons: concepts like uncertainty and OOD states are not well defined compared to supervised learning, especially for on-policy RL methods.
 Secondly, available implementations and comparative studies for uncertainty estimation methods in RL have been limited. To overcome the first gap, we propose definitions of uncertainty and OOD for Actor-Critic RL algorithms, namely, proximal policy optimization (PPO), and present possible applicable measures. In particular, we discuss the concepts of value and policy uncertainty. The second point is addressed by implementing different uncertainty estimation methods and comparing them across a number of environments. The OOD detection performance is evaluated via a custom evaluation benchmark of in-distribution (ID) and OOD states for various RL environments. We identify a trade-off between reward and OOD detection performance. To overcome this, we formulate a Pareto optimization problem in which we simultaneously optimize for reward and OOD detection performance. We show experimentally that the recently proposed method of Masksembles strikes a favourable balance among the survey methods, enabling high-quality uncertainty estimation and OOD detection while matching the performance of original RL agents.

Under review as a conference paper at ICLR 2023
HoW TO ENABLE UNCERTAINTY ESTIMATION IN PROXIMAL POLICY OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
While deep reinforcement learning (RL) agents have showcased strong results across many domains, a major concern is their inherent opaqueness and the safety of such systems in real-world use cases. To resolve these issues, we need agents that can quantify their uncertainty and detect out-of-de of-distributtin (OOD) states. Existing uncertainty estimation techniques, like Monte-Carlo Dropout or Deep Ensembles, have not been extended to on-policy deep RL. We posit that this is due to two reasons: concepts like uncertainty and OOD states are not well defined compared supervised learning, especially for on-policy RL methods. Secondly, available implementations and comparative studies for uncertainty estimation methods RL have been limited. To fill the first gap, we propose definitions of uncertainty and OOD for Cctor-Critic RL algorithms, namely, proximal policy optimization (PPO), and present potentially applicable measures. In particular, we discuss the concepts of value and policy uncertainty. The second problem is addressed by implementing different uncertainty estimation methods and comparing them across a number of environments. The OOD detection performance is evaluated via a custom evaluation benchmark of in-distribution RL environments. We identify a trade-off between reward and OOD detection performance. To overcome this, we formulate bi-objective meta optimization problem in which we simultaneously optimize hyperparameters for reward and OOD detection performance. Our experiments show that the recently proposed method of Masksembles (Durasov et al. (202I) strikes favourable balance among the methods, enabling high-quality uncertainty estimation and OOD detection that matches the performance of original RI. agents.
(ID) and OOD states for various
INTRODUCTION
Agents trained via deep RL have achieved several high-profile successes over the last years, e.g., in domains like game playing (Badia et al.) (2020). robotics (Gu et al. (2017)), or navigation (Kahn) et al. (2017)) However, generalization and robustness to different conditions and tasks remain a considerable challenge (Kirk et al. (202T): Lütjens et al. (2020)). Without specialized training procedures, e.g., large-scale multitask learning, RI. agents only learn very specific task in given environment conditions (OEL. Team (202T). Moreover, even slight variations in the conditions compared the training environment can lead severe failure of the agent (Lütjens et al. (2019); Dulac-Arnold et al. (2019)). This is especially relevant for potential safety critical applications. Methods uncover an agent's uncertainty can help combat some of the most severe consequences of incorrect decision brccision-making.. Agents that could indicate high uncertainty and, therefore, e.g., query for human interventions or use a robust baseline policy (e.g.. a hard-coded one) could vastly improve the reliability of systems deploying RL-trained decision-making.
In response, uncertainty estimation has been recognized as an important open issue in deep learning (Abdar et al. (202T): Malinin & Gales (2018) and has been investigated in domains such as computer Durasov vision (Lakshminarayananl et al. DDrassyy et al. (202I) ) and natural language processing (Xiao) & Wang (2018); He et al. (2020)). The topic of uncertainty estimation for deep RL has also gained traction (Clements et al. (2019)). While some work has explored OOD detection for Q-learning- based algorithms (Chen et al. (2017 2021); Lee et al. (202I)), there is almost complete lack of studies on uncertainty estimation for actor-critic algorithms, despite the fact that actor-critic algorithms are widely used in practice, e.g., in continuous control applications. Additionally, work specifically

---

Under review as a conference paper at ICLR 2023
exploring OOD detection has been mostly focused on very simple environments, and neural network architectures (Mohammed & Valdenegro- Toro (202T); [Sellmeier et al. (2019). This, therefore, limits the transferability of results to more commonly used research environments. In this paper, we bridge this research gap through a systematic study of in-distribution (ID) and out-of-distribune (OOD) detection for on-policy actor-critic RI. across set of more complex environments.
We implement and evaluate set of uncertainty estimation methods for Proximal Policy Estimation (PPO) (Schulman al. (2017), versatile on-policy actor-critic algorithm that is popular in research due to its simplicity and high performance. We compare a series of established uncertainty estimation methods in this setting. Specifically, we introduce the recently proposed method of Masksembles Durasov et al. (202I)) to the domain of RL. For multi-sample uncertainty estimation methods like Ensembles or Dropout, we identify a trade-off in on-policy RL: During training, a "closeness" the on-policy setting is required, i.e., models should train on their own recent trajectory data, However, for OOD detection, sufficiently diverse set of sub-models is required, which leads to the training data becoming off-policy. Existing methods like Monte Carlo Dropout (MC Dropout), Monte Carlo Dropconnect (MC Dropconnect), or Ensembles, therefore, struggle either to achieve good reward or OOD detection performance. The recently proposed method of Masksembles has the ability to smoothly interpolate between different degrees inter-model correlation via its hyperparameters. We propose a bi-objective meta optimization find the ideal configuration. We show that Masksembles can produce competitive results for OOD detection while maintaining good performance.
The contributions of this paper are three-fold: (1) We examine the concept of uncertainty for on-policy actor-critic RL, discussing ID and OOD settings; to tackle this, we define value- and volicy-uncertainty, as well as multiplicative measure (Sec. 3. (2) We show different methods to enable uncertainty estimation in PPO; as part of it, we establish Masksembles as a robust method in the domain of RL, particularly for on-policy actor-critic RL (Sec. D. (3) We present an OOD detection benchmark to evaluate the quality of uncertainty estimation for a series of MuJoCo and Atari environments (Sec. 5), we evaluate the presented methods and highlight key results of the comparison (Sec. 6
The full source-code of the implementations, fully compatible with StableBaseliness (Raffin et al. 1202I). will be released as open-source,
2 RELATED WORK
Defining uncertainty in RL Defining uncertainty for RL is less well studied compared to the supervised setting (Abdar et al. (2021). Existing work studied aleatoric and epistemic uncertainty in reinforcement learning (Clements et al. (2019; Charpentier et al. (2022) ) e.g., disentangling the two types of uncertainty. Here, we are interested in OOD detection during inference which is achieved by capturing epistemic uncertainty. However, as we do not directly attempt disentangle both sources, our proposed uncertainty measures can also be interpreted as estimating total uncertainty. Methods for uncertainty estimation in RL Different methods have been proposed in supervised learning settings to tackle uncertainty estimation and OOD detection. Bayesian deep learning (Kendall & Gal (2017): ; Wang & Yeung (0200) methods provide robust uncertainty estimation rooted in deep theoretical background. However, Bayesian deep learning methods often come with considerable overhead in complexity, computation, and sample inefficiency compared to non- non-Bayesian methods. This has slowed their adoption in deep learning in general and in RL in particular. Deep ensemble models have proven reliable and scalable method for uncertainty and OOD detection in supervised deep learning (Lakshminarayanan et al. (2017): Fort et al. 22019). An obvious drawback of ensemble methods is the need for a number of independently trained models, which drives up computational cost and inference time. MC Dropout has been proposed as a more resource-effective method for single-model uncertainty estimation (Gal & Ghahramani (2016)) As a disadvantage. enabling drop-out during inference can lead to lower model performance overall (Durasov et al. (2021) Although there has been work in the space of using uncertainty estimation in RL., it has mostly been limited to either simple bandit settings (Auer (2003); Lu & Van Roy| (0017), ). model-based RL (Kann et al. (2017)), or applied in training routines to improve the agent performance, e.g. by boosting exploration in uncertain states (Wu et al. (2021): Chen et al. (201)): 7): Osband et al. (2016) 2018)). Work explicitly investigating OOD detection for RL has been limited (Mohammed & Valdenegro Toro (202I); [Sedlmeier et al. (2019). Out of the existing work, most methods have been almost exclusively applied to Qllearning (Mohammed & Valdenegro- 20 Sedlmeier et al. (019))
2

---

Under review as a conference paper at ICLR 2023
Chen al. [2017] ) Lee et al. (2021)) In contrast, our work specifically targets for on on-policy and cector-citiic methods, including for environments with continuous action spaces. Evaluating uncertainty estimation in RL Work on the systematic evaluation and benchmarking of uncertainty estimation or OOD detection methods in RI has been limited. (Sedlmeier et al. (2019) propose very simple environment based gridworld setting and flipping the goal state to benchmark a set of uncertainty estimation methods. (Mohammed & Valdenegro- Poadeeegro-Tooo (2021) introduce physics-based interventions for simple control tasks like Cartpole and compare some methods. We extend these interventions to more complex simulation-based environments (Todorov al. (2012)). Additionally, the existing work has very limited discussion on reward performance, training details, or used algorithms. In this work, we keep stronger focus on implementation, the relationship between reward and OOD detection performance, as well as effect of hyperparameters.
UNCERTAINTY FOR ON-POLICY ACTOR-CRITIC REINFORCEMENT LEARNING
In this section, we provide the background on actor-critic RL. We first discuss the concept of uncertainty well as ID and OOD data for on-policy algorithms. We then introduce value uncertainty and policy uncertainty to address the architecture of actor-critic algorithms.
Actor-Critte Reinforcement Learning Markov decision process is defined as M A, P. R. Po, 7), where S is the state space, AA is the action space, P is the family of transi- tion distributions on indexed by S x A with p(s' [s, a) describing the probability of transitioning to state s' when taking action in state 8, R: S x A + IR is the reward function, Po is the initial state distribution, and y is the discount factor. A policy Te is conditional probability distribution of actions E A given states S parameterized by RL is the problem of finding the optimal policy of the agent with respect to the specified reward function. In this work, we investigate uncertainty estimation for actor-critic algorithm. Formally, the actor/policy To (o(aas)) (e) outputs a probability to take an action a E A in a state s. In actor-critic algorithms, we can define the objective of the policy and the definition of the critic/value function:
Lact (0) Eloo A. V (st) B Σ
y R(s
In line with common PPO implementations, throughout the paper, we assume the advantage function A, to be computed via generalized advantage estimation (GAE) (Schulman et al. (2166)) which internally uses the value function estimator V (s) associated with policy π and parameterized by eb. Further in the text, we will use the notation V for simplicity. Both the policy æ and value estimator V are implemented as a neural network. In the on-policy case, the trajectories to update both the policy and value function are rollouts from the most recent policy ng. For off-policy algorithms, accumulated data from current and past policy rollouts are also utilized for optimization. While this re sampling increases sample efficiency, it is traded with training instability. In on-policy RL, the target policy is directly optimized to maximize cumulative reward.
3.1 DEFINING IN-DISTRIBUTION AND OUT-OF-DISTRIBUTEI FOR ON-POLICY RL
Common implementations of actor-critic variants lack a way to output uncertainty with respect to states. We are interested in quantifying uncertainty to find OOD states for an RL agent. In supervised learning, there is a comparatively clear distinction between ID and OOD cases: data that similar to the training distribution can be generally considered as in-distribution. Importantly, the training distribution is known and fixed at training time. OOD data generally describes data that not part of the learned manifold describing the training distribution, which means, e.g., classifier is unable to interpolate to an OOD sample.
The definition of OOD for Markovian data is more complicated than the ones for the independent identically distributed (i.i.d)) case because the training data is constantly changing over the training process. Fig. shows schematic explanation of ID for different types of RL algorithms. Depending on the type of RL, offline-, off oof-pyy or on on-policy setting, we assume a different set of states to be in-distribution cut-o--disttibuttin.
3

---

Under review as a conference paper at ICLR 2023
(a) Offline/Batc-RI..
(b) Off-policy RI
(c) On-Line RL
The RL setting influences what can be considered ID and OOD. Depending the setting, a different area of the state space can be considered ID for value estimation. We use the background cones to depict the part of state space that is considered ID: f 22, 13, final notate schematic trajectories which have evolved with training time. The green dot shows the initial state of the agent, and the red one is the terminal state.
Figure 1:
Because an offline-RL RL agent is repeatedly trained on a full batch of optimal and suboptimal trajectories, it can produce consistent estimates even for these suboptimal states. In off off-policy settings, where large replay buffer is used, compare, e.g., to DQN (Mnih et al. (2015)) or SAC (Haarnoja et al. (2018). we have combination of current and outdated states, i.e., the agent is still trained to predict values for states which are not part of the trajectories produced by the most recent policies. In the on-policy case, we assume that only a narrow set of states along the trajectories produced by active policy can be considered as ID. This is because both the value estimator and the policy are only updated on data from the active policy. As part of the policy optimization procedure, due to catastrophic forgetting (French (1999)), states encountered early during training but not visited frequently due to the policy shifting to other trajectories cannot be considered II) anymore. Importantly, this also means that the set of ID states is specific the trained policy. Our goal is to enable the trained policy detect OOD states at inference time. Because of the aforementioned "closeness" of both policy and value function, we will investigate the suitability of both, the uncertainty of the value function, and the uncertainty of the policy function to quantify the agent's uncertainty.
3.2 QUANTIFYING UNCERTAINTY: POLICY UNCERTAINTY AND VALUE UNCERTAINTY
We now discuss possible uncertainty measures for actor-critic RL. As mentioned, our goal is to quantify the agent's uncertainty, i.e., the uncertainty of decision-making A direct application of this the detection of OOD states in which the agent is not able to predict an action to maximize return with high confidence. In actor-critic RL, can apply sample-based uncertainty estimation methods to both the policy and value network, which means we both get an output distribution for the action and the value estimate. This enables us investigate both policy and value uncertainty independently to probe their adequateness as uncertainty measures.
During the remainder of the paper, we use the term policy uncertainty refer to any uncertainty measure for the output distribution of the policy. In the following evaluation, we encounter both categorical and continuous output distributions. Formally they are defined as:
(a: 0))
Teal (a: [s:1s;0) 0)
Teum (als;0)
ofzz) (2)
Σラ= exp(#T(als::
where a; the i-th action discrete set of actions, and a is the j-th component of multi-variate continuous action. In the case of discrete actions, N, therefore, refers to the number of actions, and the case of continuous, N refers to the dimension of the multi-variate action. For discrete actions, categorical distribution is defined by softmax of the logits म. i... the output of the policy network. For continuous actions, multi-variate diagonal Gaussian distribution is applied, with the mean of each dimension parameterized by the logits of the policy network and state-indepent ttate-independett (reported in Appendix
In the following,
multiple predictions of different sub-models during inference, write II {I "r,,""" .,?..,, x } for & different sub-models/samples. From the perspective supervised classification, possible uncertainty measure is 21ll Leatmazz.
cat.mmaz.proe mox.prob. (8) 1 Pmaz (8) 1 maxjel,.., T. (
4

---

Under review as a conference paper at ICLR 2023
of the averaged distribution Tp (als) : which the classifier assigns the top label. However, in RL, there no clear singular correct action, and thus low probability for certain action does not necessarily correspond to high uncertainty. It could also mean that two actions are equivalent with respect to the optimal policy. Thus, instead of capturing epistemic uncertainty, which is required for OOD detection, the measure instead captures the irreducible uncertainty of the environment dynamics, i.e. aleatoric uncertainty. For the same reason, the entropy of the averaged distribution 24 (eat.entropy (s) H y (al s)) also does serve as reliable uncertainty measure. Lastly, both of these measures are influenced by the magnitude of an entropy bonus to the policy optimization objective (see (Schulman et al.) (2017) for details), which further limits general applicability. In Sec. 6 we report the performance of both uncertainty measures but found them to be unreliable for quantifying uncertainty w.r.t. ID or OOD states.
E Ex=0 ( This corresponds the confidence with
Instead, we focused on uncertainty measures that are based purely on the disagreement between the respective sub-models. We use the following formulations for policy uncertainty for continuous and categorical actions, respectively:
4 (ca)
sld(1 (d)))
(3)
N
and
N
4 stat =
1 N
Σ
std( (a:ls) [a.s))))),,
(4)
We chose this formulation, because both (a;ls) and TTa'ls) are directly parameterized by the logits of the respective policy networks. We, therefore, hypothesize that we can interpret these average standard deviations measure of the agreement/disagreement between the sub-policies x² ..,, nh. An alternative approaches to formulate uncertainty can be found in Appendix
Equivalently, we refer to uncertainty measures for the value function as value uncertainty and use the following notation LV with V {V',1'² 1/2.... f²k.. We investigate the standard deviation of the value function as potential proxy for state-visitation frequency:
U std((V (s), (3)), (s))))
(5)
Based on these measures, we define multiplicative uncertainty, which we define as the product between policy and value uncertainty for each state. We may interpret this measure as a state-action uncertainty measure:
UMult (s) = (s) ())
4 EQUIPPING PPO WITH UNCERTAINTY ESTIMATION
Based the defined uncertainty metrics, we now investigate the integration of uncertainty estimation methods into RL actor-critic architectures. We discuss how to implement uncertainty estimation for an actor-critic algorithm, specifically PPO (Schulman al. (2017)).
4.1 INTEGRATING UNCERTAINTY ESTIMATION
As pre-condition, we want the uncertainty methods to be easily integrated into existing RL setups as drop-in replacements for existing network architectures. While this limits the set of methods, ensures high practical applicability of the presented results.
Recently, Masksembles (Durasov et al. (021 were proposed as an attempt to combine the advantages of Ensembles and MC Dropout: MC Dropout randomly discards some parts of the network by applying a random binary mask at every iteration, i.e., using an infinite number overlapping models. Ensembles use a set of fixed and separate sub-models. Masksembles create only a limited and fixed set of masks but also share part of the network. Importantly, Masksembles provides parameter to control the overlap between masks, which in turn controls the correlation between the submodels. This allows us to flexibly determine the best configuration based on the problem. In terms of training speed, Masksembles performs comparably to MC Dropout and outperforms the latter in terms of
5

---

Under review as a conference paper at ICLR 2023
inference speed because Masksembles generate output distributions in single forward-pas passs While the technique was proposed in the domain of image classification, to our knowledge, this is the first use for RL.
We compare our Masksembles-basee model (Fig. 2eh against three architectures enabling uncertainty estimation and baseline. Both MC Dropout and Ensembles established uncertainty estimation methods. Additionally, we compare against the more recent MC Dropconnect. Fig. 2 is a schematic illustration of the integration of different methods in the baseline MLP architecture used for continuous control tasks. Architecture details are discussed in Appendix
Baseline: The default fully-connected networks and CNNS mance. It does not provide uncertainty estimation functionaliity (Fig. 2a
comparison for reward perfor-
Masksembles (Durasov al. (2021)): During training and inference, masks are applied across the input batch during the forward pass, which leads to l different predictions used for uncertainty estimation. We base our design choices on the respective paper (Fig. (e).
Monte-Carlo Dropout (Gal & Ghahramani (2016)): Here, train the networks with Dropout enabled. We apply Dropout to the same two layers as Masksembles to enable direct comparability. For MC Dropout, we keep Dropout enabled during inference and sample multiple times with individual Dropout masks to get a distribution predictions. Compared Masksembles, Dropout masks are unique for each prediction. To match the number of masks in Masksembles, base the uncertainty estimation on = independent samples (Fig. 2b).
Monte-Carlo Dropconnect (Mobiny et al. (2021)) Dropconnect is closely related to Dropout.
Instead of disabling neurons, i.c., the output of a layer, random weights are assigned the value O,
which achieves similar effect as Dropout (Fig. Pcp.
Ensembles: Lastly, as comparative baseline, we employ an ensemble of networks, in the case of MuJoCo 4 policy and critic networks each, in the case of Atari 4 independent CNNs with individual policy and value heads each. As mentioned, to ensure basic synchronization between the networks, all networks are trained based on the loss function computed from a shared replay buffer (Fig. 2d).
Output
Output
Output
Output
Output
Fully-Connected
Fully-Connected
Fully-Connected Prepconncct
Fully-Connected
FC
FC
biocout
Masksörbbee
ReLu
ReLu
ReLu
Rew
REW
RoLu
Rolu
Fully-Connected
Fully- Conneccon
Conle-Conedtted
Fully-Connected
FC
FC
Propconnect ieppoonecchiiil
plopout
ReLu
ReLu
Re
ReLu
ReLu
Recu
Retu
Retu
Fully- Connected
Fully-Connected
Fully-Connected
Fully-Connected
FC
PC
FC
FC
Input
Input
Input
Input
Input
Vectorized Environiment
Vectorized Environiment
Vectoriced Environiment
Vectorized Ervironment
Vectorized Envirormentt
(a)
Baseline Netw. (b) MC Dropout
(c) MC Dropconn.
(d) Ensembles
(c) Masksembles
Figure 2: Different MLP architectures:
The baseline network (a) is three-layer MLP. For MC Dropout (b), a random mask is applied to each prediction, which can be interpreted as an infinite number of sub-models.
(c) Dropconnect is similar to Dropout. For Ensembles (d) assume small number of fully independent networks. Masksembles (e) can be interpreted as an interpolation that uses fixed set of masks to create partially overlapping sub-models. For Masksembles and Ensembles, each sub-model/maask is assigned to fixed subset of environments (dashed lines).
4.2 INFERENCE WITH MULTI-SAMPLE METHODS
We apply multiple inference schemes translate the distribution of policy sub-models into a final action prediction. For categorical actions, in the games of Spacelnvaders and MSsPacman, we voting scheme. which is commonly applied in classification: Each of the sub-mu mooemmeel assigns probabilities to the available actions. We then choose the action with the highest probability from each submodel and then sample the mode of these k predicted actions. In case two more actions

---

Under review as a conference paper at ICLR 2023
are chosen in an equal amount, we randomly decide one of the actions. In preliminary experiments, we tried averaging the & categorical action distributions across actions and then sampling from the averaged distribution, and found this to have slightly better performance in Berzerk and SpaceImvaders. For each of the games, we report the best performing configuration. For continuous actions, we use the submodel predictions to parameterize diagonal Gaussian distribution. For deterministic action selection, we choose the mean predictions; for stochastic selection, we sample from the distributions.
BENCHMARKING OUT-OF-DISTRIBUTION STATE DETECTION
Environments We evaluate the presented methods on four MuJoCo Todorov et al. (2012)
) control
environments, namely, Ant-v3, HalfCheetah-v3, Swimmer-v3, and Walker2d-v3. All of these environments feature vector-valued continuous actions. Furthermore, we apply the surveyed methods to four Atari environments (Bellemare al. 2213)): Seaquest-v4, MsPacmann-v,, Berzerk-v4 and SpaceInvaders-v4, We chose these four games to represent different game-play mechanics and because recorded game-play is available as part of the Atari-Head dataset ((Zhang et al. (2019)). which we use for OOD state generation as described below.
Creating the Benchmark for Out-of-distributiin Data For MuJoCo-based environments, apply a number of random perturbations to the parameters of the physics simulation, c.g., increas- ing/decreasing gravity and/or friction or varying the dimensions of the body elements. These modifications create OOD states when the agent acts in the environment. In Appendix H. we give detailed description of the applied interventions. To create OOD states, we randomly sample from the set of available parameters and let the respective agent act for 100 steps in the perturbed environment; 50 configum ooniiuraations are sampled to create dataset of possible OOD states.
For Atari games, previous work has proposed image- bbsed perturbations (like Gaussian noise, blur, etc.) to create OOD states (Mohammed & Valdenegro- Toro (202I)). While this is a simple and generalizable method to create these states, we feel that this approach does not necessarily capture the nature of OOD states in Markovian settings. Therefore, we focus on valid game states that the agent has not yet encountered. Specifically. we sampled trajectories of advanced levels from the Atari-HEAD dataset (Zhang et al. (2019)). We ensure that the sampled states are from states of the games that the trained agents have not yet reached due to insufficient performance (e.g.. with a changed level layout). We give examples of OOD states in the Atari setting in Appendix [H] We report some additional results for additional mmage-based perturbation techniques, like adding noise or masking parts of the input image, in Appendix !|
Methodology for Evaluation We sample ID data directly by executing the trained policies in unperturbed environments. This is connected to the discussion of Sec. 3 that ID states are densely distributed around the final model trajectories. OOD states are created as described above. An effective uncertainty measure should be substantially higher for OOD states compared to ID states. We frame OOD detection as a binary classification problem: We assign ID and OOD respective labels. Then, the computed uncertainty scores, as described in Sec. B are used as threshold to classify ID and OOD states. We then compute ROC-AUC to evaluate the OOD detection quality.
Hyperparameter Optimization Most of the results that will be reported in Sec. Gare are achieved with
that were optimized for the baseline algorithm, i.e., without specifically fine- tuning the specific methods. This means that, in general, the reported results are representative of using the presented methods as full drop-in replacements without any modifications to the training process. However, we are interested in uncertainty estimation methods that are robust to the choice of hyper-parameters (like a number of models, learning rate, batch size, etc.). To enable a systematic investigation of the sensitivity to hyper-parameter choices, we run a bi-objective meta optimization procedure: We are interested in both achieving a high cumulative episode reward and a strong OOD detection performance. We sample 190 configurations per method and environment using Optuna (Akiba et al.] (2019)). During the optimization, we choose simplified OOD state generation scheme: We add uniform noise with a magnitude of 2 the element-wise standard deviation to the input observations in order to create OOD states. We chose this scheme due to its efficiency in large-scale meta optimization. We see this generation schema as supplemental to the OOD generation method described above. We showcase the results of this procedure in Fig. 4 As we have mentioned previously, this generation schema is not suitable to reliably evaluate epistemic uncertainty. it gives an approximiation of the methods capabilities.
7

---

Under review as a conference paper at ICLR 2023
Method
HallCheetah-va
Ani-y3
Walker20-v3
Swimmer-s3
Senquest-v
MoPacnan-s4
Berzerk-e4
Spucelivankrs-44
Baseline
3056.5
5189.7
2068. 6
178, 17188
3898
2405.02a002
606 34322
865. 651074
MS
26736-ccos.
4884.8.925.0
2328.2:-405
15144
3524
3
1856
G
761.3
567.3
MCDO
6
2471.3
259.7
3455.
562.3
1514
MCDC
651.8 6518815600
1010 115177
850. 1
19.3
3110.11972
1096.017009
637 111001
759.5
Ensembles
713.4a243.6
930.2
128. 9142.3
26.4g5.3
122.6
210.0g14.0
360.21277.
270.0
MS Single
2684.9_00.
$746.3
2007.1
172.69824
3242.611088
1913.62464
708.0
774.2
MCDD Single
2008 8.5m.5
2272.4 172.44995s.3 a
1
111.5
2204,3
D83 18
MCDC Single
2984.52908.8
671 F720022
58.42.50
3186.0210044
1681 Ggrier
676-3
740. Spen
Ens. Single
464.0g905.77
960.3ga44
128. 128.5g10.5
14.6g3
183.3
247
2g7.6
263.
268. Sgsa.
Table 1: Reward summary across environments and methods Masksembles (MS), Monte Carlo Dropout (MCDO), Monte Carlo Dropconnect (MCDC) and Ensembles. Rewards are measured over 10 episodes and averaged over 3 seeds. The upper results are for action sampling based on model averaging/voting. The Single scores are for action prediction with single random sub-model. For Berzerk-v4 14, we truncated episodes after 10.000 steps if case an agent gets stuck. We mark the best results for every environment in bold.
(a) HalfCheetah--3
(b) Ant-v3
(c) Walker2d-v3
(d) Swimmer-y3
(e) MsPacman
(f) Seaquest
Berzerk
(h) Spacelnvaders
Dropeneneed
Figure 3: First Row: Training Curves for MuJoCo; Second Row: Training Curves for Atari.
EVALUATION RESULTS
In this section, we focus on comparing Masksembles approach following the protocol described in Sec. 5 namely, measure training results, the performance of the agents based on the ground-truth reward by rolling out trained policics, and the ability to detect OOD states.
Reward Following the procedure defined in Sec. A. we observe the training dynamic for control- problems depicted in Fig. 3b Ba Be 3d and games depicted in Fig. Be 3c 3h During training, actions are only selected via single samples, i.e., one prediction per environment. During inference,
multiple samples can be combined into a single prediction as described in Sec. 4.2 The results are presented in Tab. 1 We also report the rewards achieved via inference with single model. From the results, we cannot conclude that combining multiple predictions via voting or averaging leads to higher reward compared to single model.
Out-of-distributinn Detection The results for the previously described OOD detection benchmark (see Sec. (5) are summarized in Tab. 2 We found that in most cases, the Masksembles approach outper- formed MC Dropout, MC Dropconnect, and Ensembles in terms of OOD detection. Interestingly, in the control environment, e.g., Ant-v3, value uncertainty also acts as a measure of physical instability, particularly detachment from the floor, e.g., it explains substantial uncertainty at the beginning of an episode when Ant spawned. Such "levitation" states mean that any action can be equivalently reasonable impaired concerning expected discounted reward.
Bi-Objective Meta Optimization The results of the hyperparameter optimization are visualized in Appendix B The results show that Masksembles has a favorable distribution of results compared to the other methods. We find that both in terms of reward and specifically terms of ROC-AUC score for OOD detection, Masksembles has larger set of well performing configurations. We also find that the best results for Masksembles are often close to the baseline. Some agents trained with MC Dropout and MC Dropconnect also show good performance, but the number of good configurations
8

---

Under review as a conference paper at ICLR 2023
Method
WC
Ami-v3
Walker2div3
Swimmer-v3
MaPacman- v4
Seaquest
Berzerk-ve
Spacelav.oed
MS V. U. (ours)
0.85 0000
0.73
0.73 0,73 00r
0.88
0.80
0.82.0.cm
1.7
0.57
MCDO V.U.
4.4440.14
2.482 0.3422.05
0.872a0ss
0,95
0.70+0.02
0.622.015
2008
MCDC V.U.
0.3200.7
0.52 a.65
0.Hig.uur
0.74
0.00
0.59-aaor
Ensemble V.U.
0.7 0.74
0.56
0.63
0.94 g0.0
0.56g000e
0.52 go.00
0.53_
0.5g001
(ours)
0.86
0.78
0.6ga.01s
g.85
0.90
2.78
=.07
2.612
MCDO P.U.
0.32 10.049
a.03
0.49 20002
0.94
1941
0.72 Lounes
0.68 .oosa
0.59 100
MCDC P.U.
0.5420.01
0.52
0.53
0.54
0.73
0.7
0.69
0.52
Rasemble P.U.
00.000
0.04222.07
0.72gau0s
0.80gauu
0.34g0000
e0.01s
0.46
0.47 2007200012
MS Mult.Ul.
0.87 0.0220
4.77
0.72
0.945.027
0.332nozs
0.85
0.78
2.62
MCDO Mult.U.
a5.00s
D.48
a.85
0,97
a.78
0.67
a.6lan0z
MCDC MultUU.
0.36
0.6429.017
0.58gu
0.4720114
0.77zues
0.73
0.69-awr7
2.50
Ensemble Mult.I.
1.83
0.61
1o.03
0.51
10.01
0.52 0.02
0.49 joon
Max. Prob. PU,. (best)
NIA
NIA
NA
N/A
0.53
0.49
0.51
0.52
Entropy P.U. (beit)
N/A
N/A
N/A
N/A
0.51
0.53
0.48
Table 2: ROC-AUC scores for OOD detection across environments and methods: MS. MC DO, MC DC and Ensembles. We mark the best methods both for value uncertainty (V.U.) and policy uncertainty (P.U.) in bold. For Max. Prob. and Entropy P.U. we present the single best results of any method. For continuous actions, the uncertainty scores U!ll moozppros. and U entropy is not applicable.
is smaller, and there is noticeable gap in OOD detection performance. Ensemble-based models show a very weak ability to learn good policies but can reach good OOD detection performance.
DISCUSSION AND CONCLUSION
In this paper, we investigated uncertainty estimation for RL. We first discussed the background of OOD detection and uncertainty for on-policy RL. We then introduced the concepts of value and policy uncertainty for actor-critic algorithms,, leading to a new multiplicative measure. Masksembles was presented as a simple, reliable uncertainty estimation method for RL. We provided implementation details and described training procedure and inference with estimated uncertainty. Finally, we proposed uncertainty evaluation benchmark and compared multiple methods.
In this work, we have focused on PPO as the algorithm of choice. We found that the value uncertainty measure can produces reliable estimates for OOD detection. While policy uncertainty does not produce notable performance by itself, the multiplicative measure outperforms value uncertainty in most cases. Value uncertainty captures the state uncertainty, while policy uncertainty could also be seen as representation of action- autiionuncertaintyy. Therefore, we might interpret the multiplicative measure as capturing state-action- unciriiinnntr..ntty... similar to using Q-value uncertainty in previous approaches.
second noticeable result of our evaluation is the weakness of ensemble- based approaches. We want to note that this is currently specific to PPO, as ensemble-based approaches have shown greater success in off-policy and DQN-baser methods (Lee et al. (202T); ) Hiraoka et al. (202T)). In Appendix C. we provide interpretation of these results. We hypothesize that samples from different models have a negative impact in on on-policy RL, because they potentially introduce off-policy states during training. This is also the case for Dropout and Dropconnect, which introduce noise during training. In Appendix (C.2 we present results for fully separate Ensembles. Because the data distribution can change significantly during training, an ensemble of independent, non-synchronized policies may diverge. This makes averaging of predictions challenging, and because the sub-models do not share a common training distribution, the definition of epistemic uncertainty becomes very challenging.
There are still limitations with the current work: We observe a certain drop-off in performance in some environments. The meta-optimization shows that particular runs can already achieve very competetive results. Specifically, we find specific benefits for Masksembles in increasing the MLP layer sizes; see Appendix(F] F This can offset the smaller layer size introduced by the fixed masks. However, we see the nessecity to further close the performance gap compared to the baseline architectures.
Future work will include the following conceptual avenues: (I) utilize uncertainty measures to improve the performance and exploration abilities of current on-policy RL algorithms, (2) use OOD detection to query for human input in high-uncertainty situations (3) discover a more elaborate way to define uncertainty and disentangle policy and value uncertainty. (4) compare our findings with results from additional algorithms and environments, for example off- policy- additional control tasks. We report some initial results for other algorithms in Appendix D
and
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Tech- niques, applications and challenges. Information Fusion, 76:243-297, 2021. ISSN 1566-2535 doi:
https://oooorg///0.000fff....fno 2021.00000...2220.0...0000 URL https: /ww.:///wwwww.. ssteenee/sttieee//////5555522202
www. www.waccaae www.::ssceceeeeiiccccocooooo com/
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuma: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '19, pp. New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781 450362001. doi: 10.1 10.1145//2225003333370.. Utt.httts:://bi..orrr
11 3330701
Peter Auer. Using confidence bounds for exploitation-expllraiion trade-offs. J. Mach. Learn. Res., 3 (null):397-422, mar 2003. ISSN 1532-4435.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Bertrand Charpentier, Ransalu Senanayake, Mykel Kochenderfer, and Stephan Günnemann. Dis- entangling epistemic and aleatoric uncertainty in reinforcement learning, 2022. URL https
waaxxiv.org/ae abw/////b//aae 6 01558
Richard Y Chen, Szymon Sidor, Picter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv: 170.00/0000./00... 2017.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2/01.05982 2021.
William R. Clements, Bastien Van Delft, Benott-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning, 2019. URL https
Gabriel Dulac-Amold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint A904 /2901, 2019.
Nikita Durasov, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Masksembles for uncertainty esti- mation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13539-13548, 2021.
Stanislav Fort, Huiyi Hu, and Balaji Lakshninarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint
2019.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-135, 1999.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050- 1059, New York, New York, USA, 20 22 Jun 2016. PMLR. URL mlr. www.preesssseee. /vs/s////////// v48//aa11151 gall6.ll11........ html
Shixiang Gu, Ethan Holly, Timothy Lillicrap. and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off policy updates. In 2017 IEEE international conference robotics and automation (ICRA), pp. 3399-3396. IEFF, 2017.
10

---

Under review as a conference paper at ICLR 2023
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor- Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), ICML, volume 80 of Proceedings of Machine Learning Research, pp. 1856- 1865. PMLR, 2018. URL. /tttt://dbbpp uni ter.trreeeee.ee.....r de/ db/ con f / / cm 1008 htm #HaHaaaaae ja a
Jianfeng He, Xuchao Zhang, Shuo Lei, Zhiqian Chen, Fanglan Chen, Abdulaziz Alhamadani, Bei Xiao, and Chang-Tien Lu. Towards more accurate uncertainty estimation in text classification. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 10 2020.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka.
Dropout q-functions for doubly efficient reinforcement learning, arXiv preprint arXiv:2110.02034, 2021.
Takahisa Imagawa, Takuya Hiraoka, and Yoshimasa Tsuruoka. Optimistic proximal policy optimiza- tion. arXiv preprint arXiv:1906. 11075, 2019.
Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine. Uncertainty-aware reinforcement learning for collision avoidance. arXiv preprini arXiv: 182, 2017.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. survey of generalisation in deep reinforcement learning, 2021. URL htttp::// org/ attpa/bs/llll./ 09794
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pp. 6131-6141. PMLR, 2021.
Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. In Proceedings of the 3 International Conference on Neural Information Processing Systems, NIPS' 17, pp. 3260-3268, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Bjöm Lütjens, Michael Everett, and Jonathan P How. Safe reinforcement learning with model uncertainty estimates. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8662-8668. IEEE, 2019.
Björ Lütjens, Michael Everett, and Jonathan How. Certified adversarial robustness for deep reinforcement learning. In Conference Robot Learning, pp. 1328-13377 PMLR, 2020.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, 18, pp. 7047-7058, Red Hook, NY, USA, 2018. Curran Associates Inc
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattic, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-333 February 2015. doi: 10. 1008//nature142...
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Aryan Mobiny, Pengyu Yuan, Supratik K Moulik, Naveen Garg, Carol C Wu, and Hien Van Nguyen. Dropconnect is effective in modeling uncertainty of bayesian deep networks. Scientific reports, 11 (1):1-14, 2021.
11

---

Under review as a conference paper at ICLR 2023
Aaqib Parvez Mohammed and Matias Valdenegro-Towe Toroo. Benchmark for out-of-distribution detection in deep reinforcement learning. arXiv preprint arXiv:2112.02694, 2021.
OEL Team. Open-ended learning leads to generally capable agents. CoRR, abs/2107.12808, 2021.
URL httph:tt/t/ttittie org/ abaaabss//111 12808
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 8626 8638, Red Hook, NY, USA, 2018. Curran Associates Inc.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-bascliness Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-33 2021. URL http::/jmmm org/papers/v22/
20 1
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMIR, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL
http://arxiv. wwtpa/v..rrabbbbb./..sa/ http://aaw...rr///////06....22.2 02438
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.0077770034777 2017.
Andreas Sedlmeier, Thomas Gabor, Thomy Phan, Lenz Belzner, and Claudia Linnhoff- Linnhoff-Popien. Uncertainty- dii-----iitiitttion detection in deep reinforcement learning. arXiv preprint UrXiv:/901.022 19, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Hao Wang and Dit- Yan Yeung. A survey on bayesian deep learning. ACM Computing Surveys (CSUR), 53(5):1-37, 2020.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning, 2021. URL
bitps://aarxia org/ abs /2105.2115. 08140
Yijun Xiao and William Yang Wang. Quantifying uncertainties in natural language processing tasks.
CoRR, abs/1 81 1311.7722 107253, 2018. URL URL.http:///axxi http:iiwwwrra/b//////ee 07253
Ruohan Zhang, Zhuode Liu, Lin Guan, Luxin Zhang, Mary M. Hayhoe, and Dana H. Ballard. Atari- head: Atari human eye-tracking and demonstration dataset. CoRR, abs/1903.0675.. 2019. URL http://arxis wwwp:/iwwwvoggaas////oooo
12

---

Under review as a conference paper at ICLR 2023
A TRAINING DETAILS
A.1 ARCHITECTURE DETAILS & TRAINING
Architecture We choose three-layer MLPs baseline architecture for continuous control, with fully independent networks for the policy and value function. We include Masksembles-Lagg drop-in after the first and second hidden layer of the MLP (see 2 for an overview). For Atari, we apply common pre-processing techniques such as no-op actions at the beginning of each episode to introduce stochasticity, frame skipping, and frame stacking the last 4 frames, as is common practice for Atari (Mnih et al. )2015). For image observations, a fully-connected layer and one-layer fully-connecter heads for the policy and value, as described in Masksembles layer after the first convolutional layer and layers
as
the shared thrce-layer CNN followed by
(Mnih et al. 2 For the CNN,
for the final fully fully-connectedd layer before the output action distribution. added in eequivalent positions inside the networks. We use sub-models for both Masksembles and Enesembles across experiments. Equivalently, for we use 4 individual samples for uncertainty estimation with MC Dropout/MC Dropconnect. reported Appendix-D we have seen a decrease in reward with increasing dropout probability, with only marginal gains in ROC-AUC Score. We ave therefore decided to keep the dropout probability 0.2 for all environments.
parallel environments and roll out each sub-model policy at a distinct set of environments, e.g., using sub-model I policy for action selection in environments 1 and 2, sub-model 2 the policy for environments 3 and 4, etc. In the case of Masksembles, the sub-models share a significant part of the network, e.g., the intermediate convolutional layers. Furthermore, the policy and value losses are computed on shared replay buffer, with transitions sampled equally from rollouts of all sub-models. This common loss function is then applied to optimize all four sub-policies. This is equivalent to the cases of the baseline and Dropout policies and, therefore, requires minor modifications to the existing architectures. For Ensemble networks, I completely independent sub-networks are used. We find that training the ensemble modles completely independently can lead to subpar results. Without any synchronization, both inference and uncertainty estimation become very difficult (for details, see Appendix (]. We, therefore, decided also use shared replay buffer for ensembles to ensure interoperability of sub-models.
Training Details During training,
We not utilize uncertainty estimates during training, although related work has shown that uncertainty can be used to improve training, e.g., by encouraging exploration (Wu et al. (2021); Chen ct al. (017);; Osband et al. (2016); [magawa et al. (2019)). In this paper, we specifically focus on uncertainty estimation during inference.
A.2 BEST HYPERPARAMETERS
We followed the hyperparameter settings the StableBaselines 3-Z00 (Raffin et al. (202T) library. These hyperparameter settings were tuned for the Bascline architectures, i.c. they favor the baselines over the adapted architectures. For the MuJoCo environments HalfCheetah-v3 and Walker2d-v3, we used larger layer sizes. We utilized masks for Masksembles, individual models for ensembles, or 4 samples in MC Dropout for each environment.
As mentioned above (in Sec. 4 we followed these settings validate the simple applicability the different methods.
In the case of Atari, layer sharing means that, e.g., for Masksembles, we apply the masks to the shared feature extractor (CNN) and, therefore, still get 4 predictions for the policy and value outputs. We utilize 4 separate feature extractors with individual policy and value heads for the ensemble.
13

---

Under review as a conference paper at ICLR 2023
Hyperparameters: HalfCheetah-v3
Algorithm
PPO with GAE
3-layer MLP, hidden size 256
Network Arch.
(384 for Masksembles)
No layer sharing,
Layer Sharing
Separate policy and value Networks
Training Timesteps
le6
Num. Parallel Environments
8
Batch Size
64
Learning Rate
2.0633e-5
Num. Epochs
20
Sampled Steps per Env.
512
GAE lambda
0.92
Gamma
0.98
Ent. Coeff
0.0004
VF Coeff.
0,581
Max. Grad Norm
0.8
Clip Range
0.1
Obs. Normalization
false
Table 3: Training hyperparameters for HalfCheetah-v3. We chose the larger network size for Masksembles but not for the other methods did not see increased performance for the baseline, Dropout, and Ensembles policies.
Hyperparameters: Ant-v3
Algorithm
PPO with GAE
Network Arch.
3-layer MLP, hidden size 64
No layer sharing,
Layer Sharing
Separate policy and value Networks
Training Timesteps
1e7
Num. Parallei Environments
8
Batch Size
32
Learning Rate
1.90e-5
Num. Epochs
10
Sampled Steps per Env.
512
GAE lambda
0.8
Gamma
0.98
Ent. Coeff
4e-7
VF Coeff.
0.677
Max. Grad Norm
0.6
Clip Range
0.1
Obs. Normalization
false
Table 4: Training hyperparameters for Ant-v3
14

---

Under review as a conference paper at ICLR 2023
Hyperparameters: Walker2d-v3
Algorithm
PPO with GAE
Network Arch.
3-layer MLP, hidden size 128
No layer sharing,
Layer Sharing
Separate policy and value Networks
Training Timesteps
le6
Num. Parallel Environments
8
Batch Size
32
Learning Rate
5.05-5
Num. Epochs
10
Sampled Steps per Env.
512
GAE lambda
0.95
Gamma
0.98
Ent. Coeff
0,00059
VF Coeff.
0.872
Max. Grad Norm
1
Clip Range
0.1
Obs. Normalization
true
Table 5: Training hyperparameters for Walker2d-v3. We decided on hidden size 128 as it slightly boosted performance Masksembles and MC Dropout while not substantially impacting baseline performance.
Hyperparameters: Swimmer-v3
Algorithm
PPO with GAE
Network Arch.
2-layer MLP, hidden size 256
No layer sharing,
Layer Sharing
Separate policy and value Networks
Training Timesteps
le6
Num. Parallel Environments
8
Batch Size
32
Learning Rate
5.49717c-05
Num. Epochs
10
Sampled Steps per Env.
512
GAE lambda
0.95
Gamma
0.9999
Ent. Coeff
0.0554757
VF Coeff.
0.38782
Max. Grad Norm
0.6
Clip Range
0.3
Obs. Normalization
false
Table 6: Training hyperparameters for Swimmer-v3.
15

---

Under review as a conference paper at ICLR 2023
Hyperparameters:
v4
Algorithm
PPO with GAE
Network Arch.
3-layer CNN, fin. hidden size 512
Shared CNN,
Layer Sharing
Separate policy and value head layers
Training Timesteps
le6
Num. Parallel Environments
Batch Size
256
Learning Rate
0,00025 (for Spacelnvaders-ve v4 linear decay)
Num. Epochs
Sampled Steps per Env.
128
GAE lambda
0.95
Gamma
0.98
Ent. Coeff
0.01 (0.2 for Seaquest)
VF Coeff.
0.5
Max. Grad Norm
Clip Range
0.2
Basis: NoFrameskip-v4
AtariWrapper with frameskip Stacking last frames, Image normalization
Env. Details
Table 7: Training hyperparameters for Atari environments.
16

---

Under review as a conference paper at ICLR 2023
B RESULTS OF THE BI-OBJECTIVE META OPTIMIZATION
Fig. 4 shows the results of the hyperparameter optimization process. The discrepancy between the ROC-AUC scores reported in Fig. and Tab. 2 can be explained by the different OOD generation processes during hyperparameter optimization and benchmarking of fully trained models. The observation space for the
approach appears be simpler, i.e., it produces more obvious OOD states compared to the thysics-/level-baae generation. However, in all cases we observe an advantage of Masksembles over the other approaches.
C RESULTS FOR ENSEMBLES
As the results for Ensembles might be surprising, especially given previous experimental evidence, we have decided to dedicate a detailed discussion in order to explain these findings.
C.I SYNCIRONNZZATONN OF ENSEMBLES
We experimented with an independent ensemble model similar to supervised settings for the method comparison. We found that a fully unsynchronized ensemble, i.e., without shared replay buffer or action output layer, produces vastly different policy and value outputs (see the following Ap- pendix C.2). This can prevent correct (1) averaging across action outputs during inference and (2) comparison of value predictions for uncertainty estimation. Policies diverge significantly due to randomness during training in the on-policy case, which explains the mismatch between predictions for non-synchronized ensembles.
To resolve this issue, we opted to use a shared rollout buffer. Each individual submodel performs predictions on distinct set of parallel environments, which is equivalent the other ensemble-lik methods (especially Masksembles). Individual experiences of the sub-models are then collected and used for common optimization. The loss is computed the shared rollout buffer. This leads to synchronization effect between models while also ensuring that cach sub-model is trained with the complete set of environment interactions.
An explanation of the shared buffer is depicted in Fig.
C.2 FULLY INDEPENDENT ENSEMBLES TRAINING
we considered an ensemble implementation with synchronized shared buffer. In this section, we consider alternative ensemble implementation: here, all sub-models are trained independently with different seeds and then combined into a common ensemble at the inference phase. This is comparable to supervised ensembles. The main difference that in the supervised case, all sub-models are trained on the same dataset, whereas, in on on-policy RL, the distribution of states agent is trained with depends on the policy itself. It is, therefore, individual to each sub-model. While significant overlap can occur, it is not guaranteed. In such a case, the sub-modek, models might have learned significantly different policies and value functions. Combining these sub-models into single ensemble can lead to unreliable predictions: both in terms of action distribution and uncertainty estimation.
In the main part, Appendix F and Appendix
Training Details For this independent ensemble training, we utilize the baseline architecture presented in Sec. 4.2 and Fig. 22] Each of the 4 sub-models is trained with 1/4 of the baseline environment interactions compared to the other methods. For the MuJoCo environments HalfCheetah- v3 and Swimmer-v3, each sub-model is trained for 250000 steps, while for MsPacman-ve and SpaceInvaders-4 we trained for 4 million/3 million steps respectively. This ensures that each method has the same amount environment interactions. To compensate for this, we increased the number of optimization epochs on the training buffer by factor of 4, meaning that the number of gradient updates for each ensemble sub-model is equivalent to the number of updates for the full training the other methods. We report average episode reward during shared inference (Tab, g) and OOD detection performance (Tab. 9 In order to create the Ensemble models, we trained five sub-models and then used each combination of four models as ensemble model, i.e. II, { .T2 x² at) II, (x', 7², 23, x²) etc. Instead of three random seeds, we report the reward performance averaged over the 5 possible combinations of sub-models without repetitions.
17

---

Under review as a conference paper at ICLR 2023
100
044
MasksemMe
Nasksentle
Essamble
Fasembte
Drapout
Dropcornect
0.
-3000
-7000
-1000
1000
2000
2002
000
Reward
(a) HalfCheetah--v
(b) Ant-v3
04
Nasksemble
Masksemble
Unsemble
Ensemtle
Dropout
Drapout
Dropcosnect
Dropcornect
Seward
Rewand
Deneline
Dereline
(c) Walker2d-v3
(d) Swimmer-v3
Figure 4: Results of the Hyperparameter Optimization Process. Each dot represents single run with corresponding hyper syper-parameter configuration (e.g. parameters like number of masks/dropout probability, learning rate, model size etc.). The configurations are sampled from wide space of possible configurations.
Results As can be expected, due to the lower environment interaction budget for sub-models, the overall performance is lower than for comparable methods, although higher than the reported (synchronized) ensemble baseline for the Atari games. For Swimmer-v3, the phenomenon missing synchronization is especially visible: Although the sub-models achieve decent performance average. the averaged probability distribution produces incoherent behavior, which in turn leads to significantly lower achieved reward. In terms of OOD detection,
drop in performance,
18

---

Under review as a conference paper at ICLR 2023
Butti
Marocnole s0o mookt
Mullsentl zub mo
Vultara 2d mNes
Yeakoendee auo modil of
Figure 5: The concept of experience synchronization via a shared rollout buffer. The vectorized environment is divided into groups so that the predicted actions of one Masksembles ensemble sub-model are executed within each group. The blue arrows mean that all of the agents write their experience the buffer irrespective the group they belong to. The red dotted arrows indicate that losses (policy loss/value loss) are computed across all samples in the shared buffer. The shared losses are used to optimize all sub-models.
Walker2d-v3
Swimmer-v3
Mspacman-y4
SpaceInvaders-v4
Independent Ensemble
435.59
0.38
1089.2
402.3
Independent Ensemble Single
445.02
170.64
1473.6
434.1
Table 8: Average Reward Performance of the independent Ensemble. The average performance lie significantly below the best single model performance.
especially for the value uncertainty measure. These observations lead us to the architectural choices made in the main paper.
Walker2d-v3
Swimmer-v3
Mspacman-v4
Spacelnvaders-v4
Ind.Ens. Value Uncertainty
0.34
0.26
0.161941
0.33
Ens. Policy Uncertainty
0.64
0.39
0.67
0.36
Ind.Ens. Max. Prob. Uncertainty
0.40
0.28
0.53
0.51
Table 9: OOD Detection ROC scores for the independent ensemble. We notice significant drop in OOD detection performance, even compared to the ensemble implementation achieving lower reward.
19

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL RESULTS: ABLATIONS & ADDITIONAL ON-POLICY ALGORITHMS
In this appendix, we want to provide more in-depth analysis of the hyperparameter sweep results. In particular, we investigate the impact of key design dimensions on reward and OOD detection performance. Secondly, give results for two additional algorithms: TRPO and A2C.
D.1 ABLATIONS OF INTER-MOR MODEL CORRELATIO//NDEEEDDENN
want to give some additional key insights into the most important hyperparameter settings.
In the context of our experiments, one of the key aspects we investigated is the effect of the number of sub-models for Masksembles and Ensembles, as well as the dropout probability for Dropout and Dropconnect. Fig. shows the reward for submodel-basee methods. Depending the environment,
4l
26
300
-10
780
(a) Avg. cumulative episode reward across all runs and environments
(b) Masksembles: Avg. cumulative (c) Ensembles: Avg. cumulative episode reward split by environment episode reward split by environment
Figure 6: Reward Performance for submodel-basse methods with varying number models: We see an stagnating or slightly negative trend for adding additional sub-models.
we might see advantage for methods with lower number of models. As hypothesized above, one reason might be that a lower number of models might be more on-policy, i.e., a larger share of steps is contributed by each individual submodel. For Masksembles, secondary effect is that the submodel size shrinks with additional masks, as the total number units in a layer stays constant, which could affect performance well. For ROC-AUC, additional models, i.e., increasing the number of models improves the quality of OOD detection.
clear improvement when adding
vl
AM Servonon
Doceetle
Seymond
(a) Avg. ROC-AUC all runs and environments
(b) Masksembles: Avg. ROC-AUC split by environment
(c) Ensembles: Avg. ROC-AUC split by environment
Figure 7: ROC-AUC for submodel- based methods with varying number of models. Here we observe a clear positive effect of adding additional models on the quality uncertainty estimation.
partially equivalent parameter for Dropout and Dropconnect is the dropout probability p. i.c., the probability of deactivating neuron/weight during single inference run. A higher p corresponds to less correlated/indiivduaa models. Other than for the submodel-basee methods, we do not see noticeable improvement in OOD detection performance when increasing the dropout probability.
D.2 OTHER ALGORRITHMS
We chose PPO as a representative on on-policy actor-critic RL algorithm due to its widespread use across research projects. However, think that the presented results representative for other on-policy
20

---

Under review as a conference paper at ICLR 2023
05
(a) Drop.-Based: Avg. cumulative episode reward across all runs and environments
(b) Dropout: Avg. cumulative
(c) Dropeonnect: Avg. cumulative episode reward split by environment episode reward split by environment
Figure 8: Reward Performance for submodel-bard methods with varying Dropout Prob.: p 0.2 can perform well, but lower is generally preferable. Still, the reward is noticeably below baseline, which corresponds to = 0.0.
to (eeooo))
33
31-
(a) Avg. ROC-AUC scores across
(b) Dropout: Avg. ROC-AUC split
(c) Dropconnect: Avg. ROC-AUC split by environment
all and environments
by environment
Figure 9: ROC-AUC for dropout-based methods with varying Dropout Prob..
RL methods. the following, we present results of the presented hyperparaaetteropiinizztooo procedure for TRPO (Schulman al. (2015)) and A2C (Mnih et al. (201)); Sutton & Barto (2018). As we can see if Fig. 10, Masksembles still outperforms the other approaches but has a significant variance. Furthermore, although some runs still perform very well. Additionally, all sample- behind the baseline. Although the results for TRPO and A2C weaker compared to the PPO, they still confirm the general trend. However, we assume that both algorithms are even more impacted by the "off- ooff-policy-ness" of the sub-model/dropout-bo weak results with high learning rates and gradient clipping ranges, which can cause complete failure of the training. However, while Masksemble produces some of the weakest results, also achieves the overall best training results, showing again the potential of Masksembles to be optimized to strong configuration. As A2C is impacted by this particular, we hypothesize that the (approximated) trust region optimization of TRPO and PPO can mitigate some of the worst effects by stabilizing the individual policies the sub-models. Investigating the particularities of other algorithms and their interplay with the presented aample-based methods is potential future line of work.
don't see a clear advantage in terms of OOD detection performance,
methods seem to lack
approaches. Especially A2C produces very
21

---

Under review as a conference paper at ICLR 2023
0.
Nasksentle
0.2
Masksemble
Ensembte
Ensentle
Rewward
(a) TRPO
A2C
Figure 10: Results of the Hyperparameter Optimization Process for the Culf-Cheetah-v3 environment for TRPO and A2C.
22

---

Under review as a conference paper at ICLR 2023
E ALTERNATIVE POLICY UNCERTAINTY MEASURES BASED ON JENSEN- DIVERGENCE
In Sec. 3.2, the standard deviation of predicted means of actions of the policy distributions in the continuous case; as well as the standard deviation of action probabilities the policy distributions in the discrete case, as policy uncertainty measure.
Here we investigate Tensen-Shannon divergence between output distributions as an alternative measure for policy uncertainty, defined follows for categorical (Eq. 7 and continuous action distributions (Eq.
u (s()) ijEl... JS(N (a]s)[[ =
max
1/2
= max sjEI....k
(7)
and
Js
max ijEI,... {1 - }
{
- (als)]la)
(8)
S.EEl....
We compute the pairwise JS- IS-divergence between the k categorical probability distributions with N actions for the discrete case. For continuous actions, we only compare the distance of mean vectors because it is proportional to the JS divergence of the action distributions.
To show that we utilize the fact that the covariance matrices are predefined, scalar and equal.
Let P = Na(mn, and Q Na(22, Σ2), and calculate Kullback-Leible divergence between two multivariate normal distributions stragthforward way:
P(x) log P(x) dx ((x)
KL(PIIQ)
f P(x)) log
1 -0 m1) TT (s mi)
(2T)
(2T)d/2) d/2
(/r)d/2/2111 1/22
exp
1
- (x 12) (x (2) do
loog
(2T) (2/)
Eg1
1/2
exp
1
Σg
(x u1) )1) Σg (x mn) + 122 Ezz (x m2)) dx
P(x)
log
2
Σ.1
1
Σz1
(x (0- (z m1)² 2T } Bp(z) (o we)""in (x p2)
Er(x) log
tr Ep(x)
2
ΣΙ
((AAB))=t((CAA)
[==((e-m))(--i) (1)
Σ2
tr [ + Er(x) (x- Σの (x [2)
log
2
tr(gi=d
Σg d +
Σ Σ E1} + (122 m1) Eタ (12 mi)
log
(9)
2
in the last transition, we used the fact:
Ep(x) (a- u22 Σ (I - v2)] = ir ( + (/2 - m1) D (N2 -m)). (10)
We can use result from Eq. to calculate lensen-Shannon divergence:
1 [S(P))Q) (KT(PIIQ) KT(Q[IP))
-
111
1
Ulog
Σ1
+ tr
(E2 En) + (H22 -1 (H2 p1)1+
Σ21
1
Σg
+
[log
d tr
tr [2) + (mn 12) ' [2)]]] (11)
Σ
23

---

Under review as a conference paper at ICLR 2023
Notice, that in case: Ej Eg oI. It implies:
(r(E-1E) (r((1) d
(12)
and
xT'a = a7 |20112 Vor IR
(13)
By substituting Eq. 12 and Eq. 13 to Eq. 11 we get:
JS(PIIQ) = 2 for u2 u1 112 + ar 1ai - p2ll8) allm - u2ll
(14)
that concludes our statement.
The results for policy uncertainty for continuous action space corresponded to Eq. 7 are depicted on Fig. 11
The results for policy uncertainty for discrete action space corresponded Eq. action space depicted on Fig. 12
We find that the alternative uncertainty measure produces very similar estimates to the one provided in Eq. 3 and Eq. 4 (compare to Fig. [23 and Fig. 28). Both policy uncertainty measures gave similar results within margin of error in all considered environments. We, therefore, decided present the more conceptually simple measure in the main part.
100
100
Gecaag FOC unn 0025
Gesagg 1830
43
02
00
02
04
18
92
04
10
(a) Masksembles
(b) MC Dropout
(c) Ensemble
Figure 11: ROC curve for alternative JS-based policy uncertainty for HaljCheetah-vv environment
64-
22
30
hoboffacc 9C Ra30
NI5 Rottrc Bdt
N:50 Histov Rutc
(a) Masksembles
(b) MC Dropout
(c) Ensemble
Figure 12: ROC curve for alternative Ja-ba policy uncertainty for Seaquest-v4 environment
F INVESTIGATION OF SCALING ABILITIES OF MASKSEMBLES, MC DROPOUT, AND ENSEMBLES
Scaling Behavior When scaling the hidden layer size, we observe that only Masksembles show significantly increased performance. We hypothesize that the scaling can partially compensate for the
24

---

Under review as a conference paper at ICLR 2023
smaller layer size due the introduction of masks. We like to note that we do not observe the same scaling properties for Dropout and Ensemble. For the baseline model, we expect the performance to be either lower equal compared to the smaller network size, the model size has been tuned as part of previous hyperparameter tuning. We evaluated the scaling properties in the context of MuloCo. Results are presented in Fig. and Fig. 14
120
500
180
Figure 13: Scaling properties of different methods in Ant-33: We find that only Masksembles show improved performance when increasing the network size. The other methods may even drop in performance when scaling the layers. Results averaged across 3 runs for 64 hidden units and 2 runs for 128 hidden units.
Mebro CA Mndrict tatnlbe fick unaa
Hales OpH Mastovende hoc mhkee sind inaiiseerae 1961 unde
Wedlerdee Cnovnbu 641 rakler Sind una
un No Oroanc hot diiid Mo Oripac bott
200
130
140
Figure 14: Scaling properties of different methods in Walker2d-v3: We evaluate the scaling across 64 (default), 96, and 128 hidden units. We find a clear improvement for Masksembles, and a slight improvement for MC Dropout, while the baseline and Ensemble policies are mainly unaffected. Although the baseline policy achieves higher reward earlier, the final reward of the baseline policy very close the original one.
G VALUE UNCERTAINTY AND POLICY UNCERTAINTY ROC CURVES
In addition ROC curve for value uncertainty for HalfCheetah-v3 in Fig. 15 we also provide ROC
curve for policy uncertainty (please see Fig. 23
Apart from that, we provide value uncertainty (corresponded to Eq. and policy uncertainty (cor- responded to Eq. [3] for continuous case and Eq. 4 for discrete action space) ROC curves for the
following environments: Ant-v3 (see Fig. 16
24
Walker-v3 (see Fig. 25 Swimmer-v3 [18 26
Pacman-v4 (see Fig. 19 27
Seaquest-v4 (see Fig. 20 Berzerk-ve v4 (see Fig. 29. and Spaceln-
vaders
It is important to note that ROC curves for ensembles could be invalid (see Fig. 16d 24d because the qualities of the Ensembles policies are quite poor in general. As a result, it cannot validly differentiate between ID and OOD states.
Another important notice regarding the "convex shape" of ROC curves for MC Dropout (see Fig. 16b 23b). In the case of simple binary classification, labels are nothing but marking the class belongings, and changing the label "0" to "1" and vice versa will not affect the classification results. This symmetry leads to symmetry of the ROC curve with respect the diagonal ROC curve (that is, the ROC curve corresponded to a random binary classifier). and it means that labels should be inverted. After the inversion of labels, the ROC curve will have normal "concave shape."
25

---

Under review as a conference paper at ICLR 2023
A convex curve, in the of binary classification, for ID and OOD states, actually reveals the inability of the model to correctly identify OOD states, which is particularly dangerous because the model is more confident in its behavior in these states than in known distribution.
(a) Masksembles
Ensemble Figure 15: Value Uncertainty: ROC Curves for the HalfCheetah--3 environment
(b) MC Dropout
MC Dropconnect
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 16: Value Uncertainty: ROC Curves for the Ant-v3 environment
06
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 17: Value Uncertainty: ROC Curves for the Walker2d-v3 environment
1000 P00
C0
Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 18: Value Uncertainty: ROC Curves for the Swimmer-v3 environment
26

---

Under review as a conference paper at ICLR 2023
SDC
64
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 19: Value Uncertainty: ROC Curves for the MsPacman v4 environment
Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 20: Value Uncertainty: ROC Curves for the Seaquest- environment
000 00-00999
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 21: Value Uncertainty: ROC Curves for the Berzerk-v4 environment
1600 noskeTsa
60
00
Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 22: Value Uncertainty: ROC Curves for the Spacelnvaders-v4 environment
27

---

Under review as a conference paper at ICLR 2023
5DC
64
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 23: Policy Uncertainty: ROC Curves for the HalfCheetah--3 environment
Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 24: Policy Uncertainty: ROC Curves for the Ant-v3 environment
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 25: Policy Uncertainty: ROC Curves for the Walker2d-v3 environment
60
Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 26: Policy Uncertainty: ROC Curves for the Swimmer-v3 environment
28

---

Under review as a conference paper at ICLR 2023
64
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 27: Policy Uncertainty: ROC Curves for the MsPacman-ves 44 environment
64
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 28: Policy Uncertainty: ROC Curves for the Seaquest-v4 environment
64
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 29: Policy Uncertainty: ROC Curves for the Berzerk-14 environment
(a) Masksembles
(b) MC Dropout
(c) MC Dropconnect
(d) Ensemble
Figure 30: Policy Uncertainty: ROC Curves for the Spacelmvaders-ve environment
29

---

Under review as a conference paper at ICLR 2023
H EXAMPLES OF OUT-OF-DISTRIBUTION STATES AND PHYSICS INTERVENTION DETAILS
To compute the ROC curves, sampled 2000 steps from the trained policy in the default environment as in-distribution-ste We then sample ca. 2000 OOD states either from the expert dataset or generated in physically modified environment.
MuJoCo To create OOD samples for Mujoco environments, we use interventions physics(See Tab [TO.. Examples of ID and OOD samples are depicted in Appendix H We independently sampled 5 configurations, trained and executed the policies in these modified environments for maximum of 500 steps, after which an episode was restarted with a modified configuration. The first 10 states of each episode were removed from the samples. Because the modified physics partially came into effect only after the agent started performing actions, the perturbed early states might be very similar to the original early states.
Atari For Atari, we sample OOD states from human gameplay, recorded for the Atari HEAD (Zhang et al. (2019) dataset. Multiple factors contribute to the states being OOD: (1) The human gameplay does generally not fully overlap with agent gameplay, which means the agent will often, e.g.. in screen positions that are not part of the trajectories produced by trained agents, (2) we sample states only from advanced levels that have significant differences to the training distribution, e.g., additional elements or changed layouts. We verify, via qualitative inspection, that trained example, the level in
agents not able to achieve high performance in presented OOD states: MsPacman-vé can be selected at initialization, and therefore trained agents can be tested in this case.
Gravity (z-direction) 5) Wind ")
Friction Cocff.
Goom. Element Scaling
elements legs individually sculed
Hal/Cheetuh-v3
[0.5 4]
J0 1]
[0.1 50]
with random factors [1.5
2.5]
Each leg is individually scaled
Ant-v3
[0.5 4]
[0-1]
[0.1 50]
with ramdom factors [1.5 2.5]
Left/rghtt leg scaled as whole with random factors [1.5
Walker2d-v3
00.5 4]
[0 1]
[0.1 50]
three body elements scaled with random factors [1.5 2.5
Swimmer-v3
[0.5 4] (also x-dir)
[0 1]
[0.1 50]
Table We sample random configurations from the presented parameter ranges. configuration, therefore, encompasses random number of different modifications to the environment. The wind parameter influences resistance/friction inside the medium.
(a) ID sample
(b) OOD sample
(c) OOD sample 2
Figure 31: ID and OOD states for MuJoCo: HalfCheetah--33 Elements the body are randomly resized, which leads vastly different effects. The two OOD samples recorded the same timestep within an episode, highlighting the different effects the modifications can have on the execution of the policy.
30

---

Under review as a conference paper at ICLR 2023
(a) ID sample
(b) OOD sample
(c) OOD sample 2
Figure 32: ID and OOD states for MuJoCo: Ant-v3. Elements of the body are randomly resized, which leads to vastly different effects.
(a) ID sample
(b) OOD sample
(c) OOD sample 2
Figure 33: ID and OOD states for MuJoCo: Walker2d-v3. Elements of the body are randomly resized, which leads vastly different effects.
(a) ID sample
(b) OOD sample
(c) OOD sample 2
Figure 34: ID and OOD states for MuloCo: Swimmer-v3. Elements of the body are randomly resized, which leads to different observation inputs.
31

---

Under review as a conference paper at ICLR 2023
3
C 3
1880
22460
25730
(a) Frame from sample
(b) Frame from OOD sample
(c) Frame from OOD sample
Figure 35: ID and OOD samples for Atari: MsPacman-v4. In later stages, the level layout is fundamentally different. Trained agents only perform very poorly in these new layouts, i.e., we find that they do not generalize to these states and therefore treat them as out-of-distributiin. Other details like the enemy ship above the water line or changing the color of fish also occur at later stages of the game.
20
30820
45190
トトト
AITI-SiON
AETI-SION
AETI-SiON
(a) Frame from ID sample
(b) Frame from OOD sample
(c) Frame from OOD sample 2
Figure 36: ID and OOD samples for Atari: Seaquest-v4. While core parts are shared (i.e., the core gameplay), important on- screen elements are not visible in ID states, e.g., none of the trained RL agents manages collect large number of divers or extra lives (yellow submarines top screen).
32

---

Under review as a conference paper at ICLR 2023
ft
ft
6GD
7430
7500
(a) Frame from sample
(b) Frame from OOD sample
(c) Frame from OOD sample
Figure 37: ID and OOD samples for Atari: Berzerk-v4. Layouts can be different but also match in later stages. In the second sample, the positions of agents and enemies, as well as the score, are different from the earlier ID samples.
I ADDITIONAL OUT-OF-DISTRIBUTTON DETECTION RESULTS
In this section, we provide results regarding additional pixel perturbations. An average uncertainty score for ID states is sampled via rolling out the policy in the enviromment. This averaged uncertainty score is taken as a baseline for each method (score 1.0). We then compute the uncertainty score for: (1) Predicting based on white frame, (2) based on a black frame, based on the original with increasing levels of added Gaussian noise: (3) +500 (4) 1000, and (5) 1500. Lastly, we experimented with applying a custom mask to the input image (e.g.. masking out areas of the game frame). As basis, we chose SeannextNoFrrueeskip.
environment. We report both the effect on value uncertainty (see
Fig. 38) and policy uncertainty (see Fig. 39). Compared to the average ID uncertainty score (reported as score 1.0 on per-method basis), the uncertainty scores are noticeably higher than the baseline score (averaged across non-corrupted ID states). The results on the extreme cases (black/white frame), as well as predicting on states with very high noise levels) show that the predicted uncertainty is not necessarily reliable. We based our decision to move to advanced game-state attack these initial results, as we found, e.g., that adding Gaussian noise is not a good representation for OOD states.
Copprrmen
Nowe NAPAAED
Figure 38: The effect of different pixel perturbations value uncertainty. From left-to-right: no perturbation, replacement to black frame, replacement to white frame, adding Uniform Noise [-50; 50), adding Uniform Noise [-100; 100), adding Uniform Noise [-150; 150), hiding parts of the images with the static mask.
Uncertainty Measure Curves For the benchmark runs, we can look at the temporal structure of the value estimates. For HalfCheetah-v3 and Seaquest-v3, present exemplary plots that show the magnitudes the uncertainty measures presented in the main part. In each plot, the first 2000 steps correspond to ID states, whereas the following steps correspond OOD steps.
In the Fig. 40 we provide policy uncertainty and value uncertainty plots for HalfCheetah-v3.
In the Fig. provide policy uncertainty and value uncertainty plots for Swimmer-v3.
33

---

Under review as a conference paper at ICLR 2023
Preer 2413
Figure 39: The effect of different pixel perturbations policy uncertainty. From leighto-right: no perturbation, replacement to black frame, replacement to white frame, adding Uniform Noise [-50; 50], adding Uniform Noise [-100; 100), adding Uniform Noise [-150; 150), hiding parts of the images with the static mask.
In the Fig. 42 we provide policy uncertainty and value uncertainty plots for MsPacmanNoFrameskip v4.
the Fig. 43 we provide policy uncertainty and value uncertainty plots for SeapeenNoFramsskip-v.
We observe that Masksembles are better for detecting OOD samples compared to MC Dropout and Ensemble. We see that the uncertainty level for OOD samples is much higher than for in the case of Masksembles.
Apart from that, we observe that value uncertainty and policy uncertainty are much better synchronized compared to Dropout, Dropconnect and Ensemble.
34

---

Under review as a conference paper at ICLR 2023
(a) Masksembles: Value Uncertainty
(b) Masksembles: Policy Uncertainty
300
(c) MC Dropout: Value Uneertainty
(d) MC Dropout: Policy Uncertainty
(c) MC Dropconnect: Value Uncertainty
(f) MC Dropconnect: Policy Uncertainty
(g) Ensembles: Value Uncertainty
(h) Ensembles: Policy Uncertainty
Figure 40: Value and policy uncertainty plots over time for different methods in HalfCheetah-v3. The first 5000 steps correspond to ID samples, the following 5000 samples to OOD samples, separated by the vertical red line.
35

---

Under review as a conference paper at ICLR 2023
(a) Masksembles: Value Uncertainty
(b) Masksembles: Policy Uncertainty
(c) MC Dropout: Value Uneertainty
(d) MC Dropout: Policy Uneertainty
(c) MC Dropconnect: Value Uncertainty
(f) MC Dropconnect: Policy Uncertainty
g Ensembles: Value Uncertainty
(h) Ensembles: Policy Uncertainty
Figure 41: Value and policy uncertainty plots time for different methods in Swimmer-y3. The first 5000 steps correspond to ID samples, the following 5000 samples - to OOD samples, separated by the vertical red line.
36

---

Under review as a conference paper at ICLR 2023
1.0
(a) Masksembles: Value Uncertainty
Masksembles: Policy Uncertainty
(c) MC Dropout: Value
Uncertainty
(d) MC Dropout: Policy Uncertainty
(c) MC Dropconnect:
Value Uncertainty
(f) MC Dropconnect: Policy Uncertainty
5.om
(g)
Ensembles: Value Uncertainty
(h) Ensembles: Policy Uncertainty
Figure 42: Value and policy uncertainty time for different methods in MsPacmanNoFrrmesskp v4. The first 5000 steps correspond to ID samples, the following 5000 samples to OOD samples, separated by the vertical red line.
37

---

Under review as a conference paper at ICLR 2023
1.00
0.0
0200
(a) Masksembles: Value Uncertainty
(b) Masksembles: Policy Uncertainty
0.50
0.0
(c) MC Dropout: Value Uncertainty
(d) MC Dropout: Policy Uncertainty
0.00
030
(c) MC Dropconnect:
Value Uncertainty
(f) MC Dropconnect: Policy Uncertainty
0000
(g)
Ensembles: Value Uncertainty
(h) Ensembles: Policy Uncertainty
Figure 43: Value and policy uncertainty time for different methods in SeaquestNoFrameskip..t The first 5000 steps correspond to ID samples, the following 5000 samples to OOD samples, separated by the vertical red line.
38


Reviews:
[{"review_id": "yxcV8nunA1h", "reviewer": "Reviewer_5Wtm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the approach of incorporating an uncertainty estimator into the PPO model. The authors define the uncertainty of RL models by the standard deviation of logits from neural models and propose capturing these uncertainties by utilizing a previously proposed model named Masksembles. To evaluate the model, this paper introduces a benchmark that includes OoD datasets in the previously studied RL environment, such MojoCo, and Atari. The empirical evaluation shows the RL model with Masksembles layers can outperform previous work in terms of collecting more rewards and detecting OoD samples.", "summary_of_the_review": "This paper has some critical issues in writing. The contributions are marginal, and I have listed some concerns about their major contributions, especially about the concept of uncertainty. It seems the paper is composed in a rush. I recommend revising the paper for a reader-friendly format. I vote for a rejection based on the current format.", "strength_and_weaknesses": "### Strength\n\n1. This paper explores an intriguing research topic about how to enable uncertainty in RL models.\n2. The empirical studies demonstrate the leading performance of proposed models under multiple environments.\n\n### Weaknesses\n\n1. **Writing.** The writing style is cumbersome, including many incompatible expressions. I list some problems in the abstract, but there are more issues in the main paper. Reviewing based on the current version of this paper is difficult. It seems the paper introduces a fairly easy-to-follow model but the writing makes it hard to comprehend. Many details require further clarification. I strongly recommend using some help from the writing service from your school and other institution or companies in order to improve your writing. \n\n\nTo overcome these issues -> To resolve these issues\n\nhave not seen widespread adoption -> have not been extended to\n\nTo overcome the first gap -> To fill this gap\n\npossible applicable -> potentially applicable\n\nThe second point -> The second issue or problem\n\nWe show experimentally -> Our experiment shows\n\namong the survey methods -> among the previously-proposed method or in a survey paper?\n\ndetection while matching -> that matches\n\nvia a custom evaluation benchmark  -> via a customized evaluation benchmark\n\n2. **Uncertainty Estimation.** It seems the authors misunderstand the concept of uncertainty in RL. An RL task commonly has two kinds of uncertainties, including epistemic uncertainty and aleatoric uncertainty. At the beginning of this paper, the authors emphasize the problem of the OoD dataset, which is a typical source of epistemic uncertainty, but then in the benchmark, the authors mention \"perturbations to the parameters of the physics simulation\", which, on the other hand, defines aleatoric uncertainty. I recommend looking into the definition of uncertainties in a common machine learning class. The works from Yarin Gal's group could serve as nice examples, for instance \n\nJishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H. S. Torr, and Yarin Gal.\nDeterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. CoRR, abs/2102.11582, 2021.\n\n3. **Comparison Methods.** The paper argues \"In contrast, our work specifically targets for on-policy and actor-critic methods.\" I believe the uncertainty of Q values can define policy uncertainty since both Q-learning methods and policy gradient methods rely on the Q values for determining or updating policy. The uncertainty of cumulative rewards can directly capture the uncertainty in an RL task since the cumulative rewards term is the objective of RL agents. Compared to policy, it directly reflects the agent's preference under different states and is more compatible with the goal of MDP. More importantly, **I do not understand why we can not use the uncertainty of Q values to detect OoD samples.** What's the problem and why not compare with previous methods?\n\n4. **Additional Issues**\n\n- In Formula (3). The logits have not been normalized. Their ranges can significantly deviate from one to another. Their stds do not have a consistent meaning.\n\n- In Figure 2. What does the dashed line mean in this plot?\n\n- In section 5. What's the problem with using a D4RL dataset? We can create an OoD dataset by simply excluding the optimal trajectories or sub-optimal from the D4RL dataset.\n\n- \"Two-Objective Optimization\" in Section 6. It seems the Maskembles method does not require an objective to optimize. It is in fact a neural architecture that can be integrated into any of the deep models. The deep model does not have to be for solving the RL problem. The only objective the paper optimized is the traditional RL objective.\n\n- In Figure 3. The legends and labels are impossible to read. Please resize them.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ubCOV2_cuP", "reviewer": "Reviewer_dCQj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focues on quantify the uncertianty of online RL and detect out-of-distribution (OOD) states. Specifically, the definitions of uncertainty and OOD was presented for proximal policy optimization. The cocepts of value and policy uncertainty were discussed. Moreover,  this paper formulates a Pareto optimization problem to overcome the trade-off between reward and OOD detection. Empirically, they found Masksembles enables high-quality uncertainty estimiation and OOD detection while matching the performance of original RL agents. ", "summary_of_the_review": "Overall, I found this work is a intreresting work and novel in providing the definition of policy and value uncertainty. ", "strength_and_weaknesses": "## Strength\n\n+ Introduce the concept of uncertainty fir on-policy actor-critic RL, specifically, introduce policy and value uncertainty in section 3. There is limited work study uncertainty for OOD detection in RL. Unlike iid samples in supervised learning, defining indistribution and out-of-distribution data for RL is challeging. Also the definition of uncertainty for RL should be different than that in supervised learning. Previous uncertainty definition in supervised learning does not scale to RL domains. This work smartly present the uncertainty of RL policy and value as the disagreement between submodels. The disagreement was performed as the standard deviation between submodels and alternatively as Jensen-Shannon divergence between output distributions (in Appendix D). With the definition of policy and value uncertainty, existing uncertainty estimation methods can be intergrated for uncertainty estimation in RL setups. Then, the OOD detection was framed as a binary classification problem, using the computed uncertainty scores as a threshold to classifiy ID and OOD states. \n\n+ The empirical study was conducted in both discrete action space and continuous action space. The limitations of this work has been discussed.  \n\n## Weakness\n\n+ The main contribution of this work is providing definitions for policy and value uncertainties in RL. However, the practical uncertainty estimation implementation directly uses existing uncertainty estimation methods. If a novel uncertainty estimation method can be proposed  for RL would further increase the quality of this work. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZIlSwRKYtA", "reviewer": "Reviewer_rBt3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates OOD detection, with different ensembles amd dropout methods, of policy and value networks in PPO. Experiments report RL-performance and OOD-ROC curves on 4 Mujoco and 4 Atari benchmarks. All methods decreased perrformance in (almost all) environments, but Masksembles have been reported to work somewhat better for OOD.", "summary_of_the_review": "The paper does, in the eyes of this reviewer, not have enough novelty to warrant publication. While well written, the paper misrepresents some key aspects and does not yield a particularly actionable result. The reviewer therefore strongly argues for rejection.\n\nTo change this recommendation, the authors would have to convince the reviewer that there is significant novelty in the paper that has been overlooked in this review.", "strength_and_weaknesses": "**Stength**\n\n- OOD detection, as a first step to react somehow to it, is an important topic in RL research.\n\n**Weaknesses**\n\n- Unless the reviewer misunderstood something fundamental, there seems to be almost no novelty here. The authors empirically compare known uncertainty estimators, most of them known to work well for OOD in RL (see missing references below), and come to the conclusion that the newest method (Masksembles) works the best. The only other novelty is how OOD samples are generated, which did not convince this reviewer (see below).\n- The evaluation is in this reviewers eyes insufficient and conceptually flawed. First, every uncertainty method seems to reduce the performance. Why is that? Second, none of the tables report standard deviations over the 3 seeds, which are very few to make a strong argument anyway. And last, the way the OOD data is generated is very random and does not allow to say how similar evaluated samples are to the training data. Some samples will therefore be very similar and others dissimilar, without a way to correct for this confounding factor.\n- The authors also seem to misunderstand epistemic and aleatoric uncertainty. Most of Section 3.2 concerns methods that estimate the *aleatoric uncertainty* of $\\pi$, which does not affect OOD! For example, the confidence or entropy of the policy network can be extrapolated to be extremely certain, even if the state is OOD. Only the ensemble methods (including dropout) estimate actually estimate the required epistemic uncertainty.\n- The definition of policy uncertainty remains questionable: in RL multiple policies can have the same (or near identical) values. The same problem can therefore have multiple optimal policies that assign different probabilities to the same actions. Due to the training procedure this will probably not happen very often, but it still makes the metrics somewhat questionable.\n- The introduction of the evaluated methods omits many (possibly important) details. Furthermore, MC Dropconnect is neither defined nor cited in the main paper.\n- The paper does not cite a large body of works on exploration in RL that uses OOD detection with ensembles (and other methods) to produce intrinsic rewards (see below).\n- Finally, the reviewer is not certain what the takeaway from the paper is. The authors present two OOD detectors based on value and policy, but how should those be used and what is the meaning if they should disagree? To warrant publication, the authors should introduce some novel idea or analysis and demonstrate them on tasks that are of practical interest, like e.g. exploration.\n\n**Missing references**\n\nMarc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. \"Unifying count-based exploration and intrinsic motivation.\" In Advances in Neural Information Processing Systems (NIPS) 29, pages 1471-1479, 2016. URL https://arxiv.org/abs/1606.01868\n\nYuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. \"Exploration by random network distillation.\" In 7th International Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id=H1lJJnR5Ym\n\nMeire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. \"Noisy networks for exploration.\" In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1706.10295\n\nXiuyuan Lu and Benjamin Van Roy. \"Ensemble sampling.\" In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017. URL https://papers.nips.cc/paper/2017/hash/49ad23d1ec9fa4bd8d77d02681df5cfa-Abstract.html\n\nBrendan O’Donoghue, Ian Osband, Rémi Munos, and Volodymyr Mnih. \"The uncertainty Bellman equation and exploration.\" In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 3836–3845, 2018. URL https://arxiv.org/abs/1709.05380\n\nIan Osband, Benjamin Van Roy, and Zheng Wen. \"Generalization and exploration via randomized value functions.\" In Proceedings of the 33rd International Conference on International Conference on Machine Learning (ICML), pages 2377–2386, 2016. URL https://arxiv.org/abs/1402.0635\n\nIan Osband, John Aslanides, and Albin Cassirer. \"Randomized prior functions for deep reinforcement learning.\" In Advances in Neural Information Processing Systems (NeurIPS) 31, pages 8617–8629. 2018. URL https://arxiv.org/abs/1806.03335\n\nGeorg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos. \"Count-based exploration with neural density models.\" In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 2721–2730, 2017. URL https://arxiv.org/abs/1703.01310\n\nDeepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. \"Curiosity-driven exploration by self-supervised prediction.\" In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017. URL https://arxiv.org/abs/1705.05363\n\nHaoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. \"#Exploration: A study of count-based exploration for deep reinforcement learning.\" In Advances in Neural Information Processing Systems (NIPS) 30, pages 2753–2762. 2017. URL https://arxiv.org/abs/1611.04717\n\nM. Sensoy, L. Kaplan, and M. Kandemir, \"Evidential deep learning to quantify classification\nuncertainty.\" arXiv preprint arXiv:1806.01768, 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FDwey-Kolq", "reviewer": "Reviewer_vngt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a study into epistemic uncertainty estimation and OOD detection for reinforcement learning (RL), focusing on proximal policy optimization (PPO). The paper frames the OOD detection problem for RL, the differences and challenges encountered for RL as compared to supervised learning, and proposes value and policy based uncertainty measures. Several methods were considered for uncertainty estimation: Masksembles, MC Dropout, MC Dropconnect, and ensembles across many simulation settings. Masksembles was found to maintain the performance of RL (in terms of reward) while outperforming other methods in uncertainty estimation. ", "summary_of_the_review": "Overall, this is an interesting, clearly written paper that considers a timely problem of epistemic uncertainty estimation in on-policy RL. The authors do their due diligence in considering several baselines across many datasets. The paper frames important considerations for OOD detection in the RL setting. However, further insight gleaned from the experiments in terms of statistic significance and policy versus value uncertainty, as well as baselining against a one-shot uncertainty estimation method (e.g., Charpentier et al., 2022) would strengthen the paper.", "strength_and_weaknesses": "Strengths:\n* I really enjoyed reading this paper. It is well written and the ideas are easy to follow. \n* The topic is timely in terms of OOD detection in RL for safety-critical applications.\n* The problem is well-motivated and the literature review does a good job of contextualizing the paper in prior work.\n* The methods section introduced concepts eloquently, clearly, and sequentially - excellent exposition!\n* The figures are informative and effectively illustrate the benefits of the proposed approach.\n* The experiments consider a good number of baselines and many simulation settings, as well as include a thorough ablation study.\n* The future work and limitations sections were interesting and pertinent to the research community. Specifically, some important insights and open problems to address are: \n    * The OOD detection and reward trade-off, which includes robust uncertainty estimation methods potentially not being performant for on-policy RL;\n    * Training data is constantly changing for on-policy RL causing the OOD states to evolve and catastrophic forgetting causing ID states to become OOD;\n    * Since there is not necessarily a singular correct action, a low probability for a certain action does not necessarily correspond to high epistemic uncertainty;\n    * Framing of OOD makes sense for RL as valid game states that the agent has not yet encountered rather than, for example, random perturbations to images.\n\nWeaknesses:\n* The Pareto optimization problem is only explicitly mentioned in the abstract. I am assuming this is referring to the Fig. 4 discussion?\n* The paper lacks polish as there are a number of formatting and referencing inconsistencies (see below).\n* A more in-depth discussion of how the proposed approach differs from Clements et al. (2019) and Charpentier et al. (2022) would be prudent.\n* The authors highlight that since there is not necessarily a best correct action in on-policy RL inducing high aleatoric uncertainty, this does not necessarily correspond to high epistemic uncertainty. Does this not mean that methods that disentangle aleatoric and epistemic uncertainty (e.g., Charpentier et al. 2022) would work better in the RL setting? Similarly, the discussion that disagreement between sub-models causes a decrease in reward appears to lead to the benefit of one-shot uncertainty estimation models, such as Charpentier's evidential deep learning architecture. It seems that baselining against such an approach would make the paper stronger.\n* Since Masksembles are a critical part of the paper, it would be helpful to include a more detailed explanation of the method in Sec. 4.1.\n* The performances in Tables 1 and 2 were not particularly decisive, although the Masksemble did seem to outperform in the majority of the settings. It would be helpful if standard errors across the 3 seeds were reported.\n* I was missing a takeaway conclusion for the experiment in the rewards paragraph in Sec. 6. Why is it important to note that multiple predictions does not necessarily lead to better rewards? \n* As the authors state, it is very surprising that ensembles performed so poorly in the PPO setting given many prior successes across domains. Do the authors know of any other papers that support poor performance of ensembles in on-policy settings? Perhaps 4 models in the ensemble is not sufficient?\n* The experiments section would benefit from additional qualitative and quantitative analysis. Qualitatively, it would be interesting to look at some example states that were marked as OOD by Masksembles. Quantitatively, it would be helpful to look at perturbation magnitude (in continuous perturbation settings) versus estimated epistemic uncertainty for different methods to examine if an increase in perturbation leads to higher estimated uncertainty.\n* There was no clearly better approach in terms of value versus policy uncertainty estimation and no associated discussion.\n\nSome typos and points of confusion are listed below:\n1. Sec. 1, paragraph 1: 'safety-critic' -> 'safety critical'.\n2. The reference citation format is inconsistent throughout the paper, often with citations being placed with or without parentheses arbitrarily (e.g., first example is second line on page 2).\n3. OOD and ID acronyms are not defined in the body of the paper (should be defined in paragraph 1 of page 2).\n4. Sec. 2, paragraph 1: 'aleatoric' -> 'epistemic'.\n5. 'Bayesian' should be capitalized.\n6. Sec. 3, second paragraph: 'Markov decision process is set $\\mathcal{M}$ is incorrect grammar.\n7. Sec. 3, third paragraph: should all the $a$ and $s$ in the sentence 'Formally...' have $t$ subscripts?\n8. Eqs. 1, 4, 5: missing periods at the end of sentences.\n9. Eq. 2, I do not believe $\\tilde{\\pi}$ was defined formally.\n10. Sec. 4.1, second paragraph: 'determine flexibly determine'.\n11. Sec. 4.1, third paragraph: missing space in 'Fig.2'.\n12. Fig. 2 caption seems like the semi-colon should be a period?\n13. Sec. 5, second paragraph: 'varying the dimensions of body bodies'.\n14. Sec. 5, second paragraph: 'These modifications create OOD states when the agent acts in it.' is a bit grammatically awkward.\n15. Sec. 5: referencing 'section 6' instead of 'Sec. 6'.\n16. Sec. 6, reward paragraph, line 2: extra parentheses added in two places.\n17. Sec. 6, reward paragraph: 'in 4.2' -> 'in Sec. 4.2'.\n18. Sec. 6, reward paragraph: 'in the Tab. 1' -> 'in Tab. 1'.\n19. Tables 1 and 2 were not properly formatted. Try using the booktabs package for formatting tables.\n20. MS, MCDO, MCDC acronyms were not defined in Table 1, but were defined in the caption of Table 2.\n21: Sec. 6, OOD detection paragraph: 'These results' -> 'The results', \n22: Fig. 4 axis labels are too close to the sub-figure labels.\n23. Fig. 4 caption: 'The configuration are sampled' -> 'The configurations are sampled'.\n24. Table 2: V. U./P. U./Value U. were not defined. 'MC DO', 'MC DC' have a space in this table, but not in Table 1.\n25. Appendix should start on a new page after references list.\n26. Appendix sections should consistently either start on a new page or not.\n27. Appendix F, paragraph 2: extra space in 'Eq.  5'.\n28. The references should be proofread (e.g., to ensure the year is not entered twice in a citation, the conference venue is listed instead of ArXiv when available, the conference name formatting is consistent, etc.).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject