Learning to Be Cautious
Authors: Montaser Mohammedalamen
Dustin Morrill
Alexander Sieusahai
yash satsangi
Michael Bowling
Venue: Accepted by TMLR ()

Abstract: A key challenge in the field of reinforcement learning is to develop agents that behave cautiously in novel situations. It is generally impossible to anticipate all situations that an autonomous system may face or what behavior would best avoid bad outcomes. An agent that could learn to be cautious would overcome this challenge by discovering for itself when and how to behave cautiously. In contrast, current approaches typically embed task-specific safety information or explicitly cautious behaviors into the system, which is error-prone and imposes extra burdens on practitioners. In this paper, we present both a sequence of tasks where cautious behavior becomes increasingly non-obvious, as well as an algorithm to demonstrate that it is possible for a system to learn to be cautious. The essential features of our algorithm are that it characterizes reward function uncertainty without task-specific safety information and uses this uncertainty to construct a robust policy. Specifically, we construct robust policies with a $k$-of-$N$ counterfactual regret minimization (CFR) subroutine given a learned reward function uncertainty represented by a neural network ensemble belief. These policies exhibit caution in each of our tasks without any task-specific safety tuning. Our code is available at https://github.com/montaserFath/Learning-to-be-Cautious

Published in Transactions on Machine Learning Research (10/2025)
Learning to Be Cautious
Montaser Mohammodalamen
mohmmeeia@@ullerrl..cc
University of Alberta; Alberta Machine Intelligence Institute (Amäi)
Dustin Morrill
dustin.morrillBsonyy com
SonyAl"
Alexander Sieusahai
asieusah@uulbertt.cco
University of Alberta
Yash Satsangi
yash.s. satsangi@gmaillcco com
Independent Researcher'
Michael Bowling
mbowling@ualberto.com
University of Alberta; Alberta Machine Intelligence Institute (Amit)
Reviewed OpenReview: https: //
https://ppenrevieu openreview. net/ ftt//f id= NXxGGaYEnG
Abstract
A key challenge in the field of reinforcement learning is to develop agents that behave cautiously in novel situations. It is generally impossible to anticipate all situations that autonomous system may face or what behavior would best avoid bad outcomes. An agent that could learn to be cautious would overcome this challenge by discovering for itself when and how to behave cautiously, In contrast, current approaches typically embed task-specific safety information or explicitly cautious behaviors into the system, which is error-prone and imposes extra burdens on practitioners, In this paper, we present both a sequence of tasks where cautious behavior becomes increasingly non-obvious, as well as algorithm to demonstrate that it is possible for system to learn to be cautious. The essential features of our algorithm are that it characterizes reward function uncertainty without task-specific safety information and uses this uncertainty to construct a robust policy. Specifically, we construct robust policics with a k-of-N counterfactual regret minimization (CFR) subroutinc given a learned reward function uncertainty represented by a neural nelwork ensemble belief. These policies exhibit caution in each of our tasks without any task-specific safety tuning. Cautiioos Our code is available at ottp
1 Introduction
A key challenge in Artificial Intelligence (AI) is developing agents that can operate safely, avoiding harm humans, other agents, property, themselves. This is of particular concern in complex, dynamic environ- ments that change over time, where exhaustive training and testing are infeasible. For example, consider a self-driving that learns to drive through experience. The first lime that the car encounters roads covered in snow, two problems arise and compound. Snow makes the environment more hazardous and gives the road a new appearance. A good response is to react by behaving cautiously, e.g., driving more slowly, but current learning algorithms do not develop such an intuition. Generally, there are two common approaches for training Al agents, first, anddeland-eppoy where an agent is trained in an environment while receiving the reward, then deployed (evaluated) without receiving reward or adapting its policy further. Second, contin- ual learning involves contimuuus interactions between the agent and the environment while always receiving the reward. In this paper, we larget safety in the train-and-deploy scenario since it is the most common approach for current Al systems. Our primary concern is not whether the agent might learn quickly in novel situations during deployment or how it adapts after experiencing novel states, but rather understanding its zero-shot behavior when facing unfamiliar situations during deployment. Specifically, whether exhibits cautious behavior in such circumstances.
Work done when the author the University of Alberta

---

Published in Transactions on Machine Learning Research (10/2025)
Existing approaches for safety in reinforcement learning (RL) often specify safe behavior via constraints that
agent must not violate (García & Fernández, 2015 Berkenkamp et. al.
2017 Chow et al. 2018 Broadly,
this amounts to formulating tasks as constrained Markov decision processes (CMDPs) (Altman 1999 A
CMDP can be solved using RL in a model-based (Aswani et al., 2013
Berkenkamp et al., 2017] or model-
Tree (Achiam et al.] 2017 Chow et al. 2018 Srinivasan et al. 2020) way. However, this approach requires pro-defining safe states that the agent is allowed to visit or safe actions the agent can take. Alternatively,
some approaches design "safety functions" that incentivize pre-defined safe behaviors (Turchetta et al,
2016 Wachi & Sui 2020 Turchetta et al.] 20200. Approaches that require an priori description of safety information about specific scenarios present scaling problem, as it is generally infeasible to enumerate all potentially hazardous situations in realistic or open-ended applications. Our goals for this work are: (i) to illustrate why it would be useful to design agents that can automatically learn cautious behavior, (ii) to describe a series of simple tasks where success requires learned caution, and (ii) to provide an example system that learns to be cautious while showing its caution increases with training.
We summarize our contributions as follows:
1. We describe a sequence of tasks where cautious behavior is increasingly non-obvious, starting from contextual bandit environment. with obvious cautious action, leading to environments where cautious actions depend on context and cautious behavior must be lcarned, and concluding with gridworld driving enviromment for which caution requires multi-step planning.
2. We show that the k-of-N counterfactual regret minimization (CFR) (Chen & Bowling 2012) robust optimization algorithm, using an ensemble of neural networks capturing epistemic uncertainty, can generate policies that adopt cautious behavior in all of these tasks.
3. We extend K-of-N CFR from the finite horizon setting to continuing MDPs, proving that it is efficient and sound for continuing MDPs under reward uncertainty.
This paper structured as follows: Section - presents a simple motivating example that explains the importance of learning to be cautious, followed by a discussion of related work. Section 3 introduces our algorithm. Experiments and results are shown in Section and our theoretical contribution extending k-
of-N CFR to continuing MDPs is provided in Section 5 Section 6 outlines the approach's limitations and
potential directions for future rescarch. Finally, Section 7 reflects on the broader impact.
2 A Motivating Example
digits (familiar)
fashion (novel)
letter (novel)
Consider a decision-making task where you are shown an image and must choose one of eleven actions. The images are hand-drawn digits from MNIST (LeCun et al. 1998), (e.g., Figure [a) and you observe a reward of +1 for se- lecting the action with the index matching the portrayed digit and zero otherwise, except for the eleventh action, which always yields a small reward, +0.25. Now, what do you do when the image is not of famillar digit but is instead a nove a shoe image from MNIST fashion (Xiao) et al. 2017), (e.g., Figure [])) or a letter from EMNIST (Cohen et al. 2017). (e.g., Figure [e)? natural choice
(a)
(b)
(c)
Figure 1: (a) 5 from MNIST, (b) a boot from MNIST fashion, (c) "M" from EMNIST letters.
is action eleven which has always given a reward regardless of the image, while every other action often gave no reward at all. But is this choice common for current Al algorithms?
One obvious approach for choosing the next action is to guess what reward each action will yield with the new image and choose the action with the largest estimate. For example, nearest- approach would select previous image that resembles the new image in some way and use the rewards from the previous image as the reward estimates, effectively extrapolating from familiar to novel images. Considering that the
" here refers to the relative novelty due lack of its presence in the past experiences in Barto al. 2013
2

---

Published in Transactions on Machine Learning Research (10/2025)
new image looks very different from all the previous ones, this extrapolative approach relies a questionable premise. Algorithms like this are unlikely to choose action eleven, since its reward was always small and only one of the ten other extrapolations need to look promising for action eleven to be overlooked. A conventional RL approach like Q-lcarning (Watkins 1989) or policy gradient (Williams/ 1992 Sutton et al.) 2000) with lunction approximation also employs extrapolative guessing and fails to behave cautiously this task. In fact, after training on MNIST digits, a greedy policy with respect to a single neural network model of the reward function (effectively Q-learning) chooses action eleven less than 2% of the time when presented images from MNIST fashion (see Section 40.
A common approach to ensuring caution is to incorporate prior knowledge about safe behaviors. For example, we could designate action eleven as a "safe action" and encourage the agent to choose it when encountering a non-digit image or when it has no strong preference for any other action. Thomas et al. (2019) outlines a general methodology for such algorithms, while Kahn et al. (2017) provides a more sophisticated example. Embodding prior knowledge about safety into an algorithm would be easy and effective in this particular lask, but it is not very general as salety is highly task-specific and the design burden becomes worse for complicated tasks. In this vein, we present variations on our MNIST task such that cautious behavior becomes increasingly non-obvious.
An alternative to explicilly specifying cautious behavior or safety incentives is risk-sensitive RL. Broadly, these methods characterize an agent's uncertainty about future rewards of different behaviors, and then choose robust behaviors, (.e., those that maximize the agent's 's reward assuming unfavorable conditions (often with a formal risk measure). Two types of uncertainty might be present in a decision makion task, (i) aleatoric uncertainty that is stochasticity inherent in the environment, 6.g., the agent may be uncertain about the nunber that a die will show before it is rolled, and (ii) epistemic uncertainty that stems from the agent's 's lack of certainty about the specific environment, e.g., the agent may be uncertain about a die's probability distribution, not just its outcome,
Various methods for learning policies are robust to aleatoric uncertainty (Chow et al., 2017 Tang et al. 2020 Clements et al. 2019). Still since the mapping from images and actions to rewards is deterministic our MNIST example task, there is no aleatoric uncertainty to be robust to. Consequently, these methods could consider
do not behave differently from extrapolative systems in tasks like this. Alternatively, robustness to epistemic uncertainly. If the agent is inherently certain about action eleven's expected reward and inherently less certain about the expected rewards of the other actions, then a robust policy would choose action eleven, provided the level of uncertainty is great enough. In this case, the agent's beliefs are crafted with the domain in mind to achieve the desired behavior in much the same way as the previonsly discussed prior knowledge approaches. There are many more sophisticated variations on this idea (Petrik & Subramanian 2014 Chow et al. 2015 Ghavamzadeh et al.) 2016 Zahavy et al., 2021 Rigter al. 20021). However, they share similar downsides prior knowledge approaches. See Appendix for more related work, including POMDPs (Aström 1965). distributional RL (Bellemare et al. 2017). and CMDPs (Altman 1999
Our approach uses robust optimization with a learned (i.e., posterior) belief without imposing any task- specific safety information into either component to automatically construct cautious policies. This algorithm learns autonomously to identify and choose cautious behavior that is unique to cach task. We evaluate our approach in a sequence of tasks where cautious behavior is increasingly complex. This sequence begins with the previously described MNIST example and advances to gridworld driving task that requires sequential decision-making
3 Learning to Be Cautious
We have used the term caution colloquially to refer to common human behavior in unfamiliar settings, such as: (i) seeking additional oversight when available; (ii) when considering similarly promising courses of actions, preferring those with certainty about outcomes; or (iii) preferring situations where mistakes are less costly. In order to operationalize this notion into agent behavior, we propose that human notions of cautious behavior can be well thought of as risk oversion in the face of epistemic uncertainty remaining after repeated interactions with the environment. While notions of safety are often tied to risk aversion generally, here we
3

---

Published in Transactions on Machine Learning Research (10/2025)
are concerned with an aversion to risk due to lack of familiarity with novel situations (i.e. uncertainty that the agent has complete model of the world) rather than risk aversion due to stochastic outcomes.
Markov Decision Processes. In order to formalize the problem of learning to be cautious, we will separate the agent's "world" into the familiar and the novel, each represented as a Markov decision process (MDP). A finite, discounted MDP, (S, p,,p,, de. 7), is a finite set of states, S, a finite set of actions, A, a Markovian state transition probability distribution, p(- s,,a) A(S) for all states and actions a (where A(S) is the probability simplex over set S), an initial state distribution dæ A(S), and discount factor, Y [0, 1 Feedback evaluating the agent's behavior within an MDP is typically encoded
scalar reward function, r: S x Ax - [-U, U), where the magnitude of cach reward is bounded by U E R. This allows us to evaluate a (stationary) policy, π i.e., an assignment of probability distributions, 7 8) E (AA, to each state 8 according to its y-discounted expected return. If S. ~ dg, A, ~ T(-|S;-1), and S. ~ ((-IS)--.AA)) then the Furthermore, S441 1 S = s, A, = a].
normalized -ddicouuted expected return is U2 (T; = (1 - 7) [> (r, A441,4411
the algorithms we discuss will make use of q, (a, TT;)) = (1 - (11-)])(22)) >){ A4,,
which is the (normalized) y-discounted expected return for taking action a in state s under reward function r and discount factor y, before following policy æ thereafter.
Extrapolation. Since we are primarily interested in examining the agent's behavior in novel situations about which they have never received feedback, do not define reward function for the novel MDP. Our primary concern is not whether the agent might learn quickly in novel situations or how it adapts after experiencing novel states and then rewards, but rather understanding its zero-shot behavior when facing the unfamiliar. Therefore, the agent does not perform any adaptation after seeing novel states, like other works (Zinggraf et al. 2020; Zhang et al. 2020: Filos et al., 2020 Rigter et al. 2021). Instead, we assess the agent's 's behavior in the novel MDP qualitatively, looking at the degree to which its actions in novel settings represent cautious behavior. Furthermore, we will focus only reward uncertainty; investigating caution (and risk-aversion more generally) with transition uncertainty needs further investigation both theoretically and practically.
A straightforward approach for the agent to formulate goals for the novel MDP is to extrapolate the familiar reward function. Ordinary RL planning algorithms can then be applied to generate a policy that will perform well if the novel and familiar MDPs are very similar. Extrapolation can be done with conventional regression methods, e.g., we can model the reward function as a neural network and train its parameters by applying optimization algorithm like stochastic gradient descent to minimize the network's mean squared error. A natural approach, given such an extrapolated reward function model, f, is then to behave according to an optimal policy with respect to that reward, Optim(f). This approach will represent a simple non-cautious baseline in our experiments.
Inference. A fundamental problem with extrapolation is that there are typically multiple reward models that match the familiar reward function but differ in novel situations from the novel MDP (k.e. state, action, next state triples not present in the familiar MDP), even within a restricted model class, To address this issue, we can infer posterior belief (a probability distribution) about which reward models are more reasonable, given a prior belief that describes what it means for a reward model to be "reasonable". Fxact Bayesian inference is typically intractable for high-dimensional data, but a convenient approximation is to train an ensemble of neural networks, each with unique initialization parameters, and trained using stochastic gradient descent on independently shuffted familiar examples. Each neural network acts like a sample from a posterior with an implicit prior so that the entire ensemble implicitly characterizes a posterior-like belief, resulting in a larger variance in reward models output when presented with novel states compared to familiar states. Various previous works e.g., Tibshirani (1996); Heskes et. al. (1997); Lakshminarayanan al. (2017): Lu & Van Roy| y2017); Pearce et al. (2018) Osband et al. (2019) have used neural networks in similar ways to characterize uncertainty with connections to Bayesian inference. Alternatively, we could use methods
designed to encode such a posterior compactly, such as noisy networks (Fortunato et al. 2018 or epistemic
neural networks (Osann et al. 2023)
We use the 7 ===00 0 case address the contextual bandit sctting, such as example in Scction 3 As in Kakade (2003), we use return functions that are normalized by the effective horizon, 7, so that returns have the same scale rewards. This makes optimality approximation emrors easier interpret. See Section 2.2.3 of Kakade (2003).
4

---

Published in Transactions on Machine Learning Research (10/2025)
Robust Optimization. An inference approach characterizes the agent's uncertainty about what reward functions are reasonable in the novel MDP given the experience from the familiar MDP, but ordinary RT al- gorithms can not make use of this information beyond optimizing for a single reward function generated from the belief (e.g., a sample, the expected posterior, or the maximum a posteriori reward function). Howover, robust policy optimization algorithms are designed to learn policies that are robust to such uncertainty. The k-of-N CFR algorithm computes an approximate W--of-N-robus policy, which is a policy that approximately minimizes the k-of-N risk measure, Ji-of-NN (Chen & Bowling 2012). This Bayesian risk measure is a smooth generalization of CVaR. measure. By tuning the k >> 0 and N 2 k parameters, the algorithm designer set a desired robustness level between worst-case (N > k) and average-case (k = N). As N increases, Pk-of-N approximates the CVaRR measure at the k/N percentile. k-of-N CFR works by iteratively sampling N reward functions from a belief and updating the current policy to improve its value under the average of the k-worst rewards. Importantly, the computational cost of using ensembles in this framework scales with the number of reward models N and the number of k-of-N itcrations, not with the size of the state space.
As originally presented, k-of-N CFR assumes that the uncertainly distribution is given. Another limitation of k-of-N CFR that we overcome in this work is that it has only been described for fixed horizon MDPs, i.e., those that terminate after a fixed number of decisions. We show how it can be applied in any continuing MDP, but we defer these details to Section in favor of experimental results that first illustrate the utility of learning to be cautious.
Complete Algorithm. First, we train multiple reward models with different initializations, constructing an ensemble on the familiar MDP. Each reward model takes state as an input and outputs the reward for each action. The ensemble of reward models produces a learned distribution (e-g., posterior). Second, optimize CVaR using k-of-N CFR. At every iteration and given a state (from familiar novel MDP), k-of-N CFR selects the average of the least k rewards from N randomly selected rewards and calculates the immediate regret given the current policy until convergence. Then it updates the policy using regret matching (Hart) & Mas-Colell, 2000) with the accumulated regrets. Optimizing CVaR lcads to learned cantious behavior. A practitioner should choose the values for k, N parameters, as some applications may require a higher or lower degree of caution. A description of the complete algorithm is shown in Algorithm
The experiments in the next section show that k-of-N CFR under neural network ensemble belief can effectively learn to be cautious in various lasks, investigating Contribution and 2
Algorithm Learning to Be Cautious
Initialize N, ls & number of iterations T.
Train ensemble of N * ' reward models R(rlX) the familiar MDP dataset of images X
Input: An image Initialize: Total Regret pT =0 0 & random policy x)(a
for 1= to T do
Sample N reward models from the ensemble & get reward predictions R. Estimated state value: Vx(X) Em Rn(r(a.) (ala).)
Calculate the mean of k least reward models W,, = 1 R
Calculate immediate regret: p W,, E/4I T (a X) W, Total Regret: pT = pT max(0, Pt)
Updale policy: (a/x Σ. pT
pr
end for
4 Experiments
We introduce a sequence of tasks designed to encourage agents to autonomously learn cautious behavior. Tasks vary in difficulty from one that requires no sequential reasoning and includes a universal cautious action, to one that requires sequential reasoning and where the return from each action is context-dependent, with a natural progression in-between. Additionally, we demonstrate that k-of-N policies exhibit increasing
5

---

Published in Transactions on Machine Learning Research (10/2025)
cantion the k/N ratio decreases, (i.e., with more risk aversion We compare our approach to the Optim(f) (f)
4
baseline, which represents the average performance of an ensemble, in which each neural network is trained to
fit the return (like traditional RL). While POMDPs (Aström 1965), Bayesian RL (Dearden et al. 1998.. or
distributional RL (Bellemare et al. 2017) may be considered as alternative basclines, these are not applicable
in our experiments, as our deterministic environments lack aleatoric uncertainty and are fully observable. Experimental design details and hyperparameter tuning for the algorithms are provided in Appendix B
Learning to Ask for Help. Our first task is the previously described decision-making task with MNIST images in Section 2 The familiar states are the 60K training images in the MNIST digit dataset, where the initial state and each next state are sampled uniformly at random. Ten actions correspond to a digit label, and a reward of +1 is given when the label matches the image and zero otherwise. The eleventh action can be thought of as an "ask for help" option that always receives a reward of +0.2 All action labels are solely to aid our discussion, whereas the agent only observes action indicos. As 7 = 0, the agent's return is simply the reward, making this a contextual bandit task. We represent each k-of-N CFR instance with the last policy generated after 100 iterations.
We construct novel MDPs from the MNIST fashion test set (Xiao et al. 2017 and EMNIST letters test set |Cohen et al., 2017) (lower and uppercase). Using the images as states, we construct two novel MDPs with two different state distribution schemes representing two evaluation scenarios. The first scenario replicates the dynamics of the familiar MDP in that each image is sampled uniformly. This describes a task where the agent must come up with a policy that works well on all novel images, without emphasizing the performance on any particular one, which we call the all-images regime. Our second scenario uses a point-mas initial state distribution and identity transition distribution. This scenario corresponds to decision-making task where a single crucial novel state is given instead of a distribution over multiple possible novel states, which we call the single-image regime. In this scenario, the impact of robustness is exaggerated because the k-of-N CFR policy trains on the k-worst reward functions specifically targeted to a single state rather than the k-worst averaged across many states. Figure 2a shows the results of both experiments, where the individual points within each k-of-N bar show the value of ten individual runs, while the bar height marks the average value. The solid horizontal line for Optim(f) represents the average across all 2K neural network reward functions in the ensemble belief, and the violin shows the variance.
In both state distribution regimes, the classification accuracy of all policies, including the most robust k-of-N policies, on the MNIST digit test set, ranges from 96.9% to 99.4%, so even the most robust policy does not sacrifice much in accuracy. Furthermore, even the two most robust policies, 1-of-20 and 1-of-10, choose the help action less than 2% of the time when presented with familiar images. The rest of the policies, on the other hand, almost never choose the help action.
In novel settings, the "all fashion images" scenario replicates our motivating example and shows that the help action is utilized more on the fashion images by k-of-N policies A/N is decreased (ée., with more risk aversion), up to 29% for 1-of-20. The Optim(f) baseline, which represents the traditional approach that ignores caution, is the lcast likely to use the help action on cach novel dataset. We see similar but more muted effect on the letter images, where the help action frequency ranges from 3% to 6%. Nevertheless, in the imngee-imageea regime, 1-of-20 selects the help action 89% of the time on the fashion images and 68% on the letter images 46 and 69 times more often, respectively, than the Optim(f) bascline. And when 1-of-20 does not select the help action with the letter images, it does so for letters that resemble digits, e.g., O, s, i, I, i, and resemble 0, 5, 1, and 2, respectively. See Appendix B for more details, including confusion matrices of selected actions.
Cautious Actions. In this task, an image is presented as before but there are only ten actions, and the reward for action indexed as a E (0....,9) 9) is (1) if a is the correct label for a given digit image or -(a+2)/9 otherwise. The reward for a correct classification increases with the action index, but so does the cost of misclassifecation, while keeping the mean constant under a uniform digit distribution. As a result, policies that choose lower index actions can be thought to be more cautious, avoiding the possibly large negative rewards of misclassifecation. Hand-crafted methods explicitly designed
Discovering
'Our code available at this GitHub repository https://githu: httpe////ttuub.coo//oottttt wwwm///onttsserrssttttsannnnnnno to- Caut
We also conducted a sensitivily study of the eleventh action's 's reward ranges in 00.1,0.55, 0.5) as shown B.
6

---

Published in Transactions on Machine Learning Research (10/2025)
1-of-20
1-of-10
5-of-10
10-of-10
Optim(()
1.0
frequency
0.8
0.6
0.4
0.2
0.0
f
α
f
α
(a)
Average frequency of the help action in (left) the all-images and (right) the single-image regimes.
action index
9
6
3
0
f
α
f
a
(b) Average action index chosen in (left) the all all-images and (right) the single-image regimes.
Figure 2: Results for the "learning to ask for help" and "discovering non-obvious cautious actions" tasks in novel environments ('f" for fashion and "a" for letters). Each star represents a single run, and the violin represents the variance across 10 runs.
to detect novel or out-of-distribution states, like (Vyas et al. 2018), cannot be employed in this and the following tasks because there is designated cautious action. Again, evaluate our approach in two regimes, one where the set of novel states is test set and another where evaluation is done on each of these images individually. Figure 25 shows the average action index chosen by each algorithm in the novel environments.
Again, while not shown, all policies correctly label nearly all familiar test digits from MNIST images. both regimes, on fashion images, we see that 1-of-20 and 1-of-10 systematically choose smaller indices on average than non-robust algorithms, and this increases with the k/n degree of robustness. The differences are smaller on the letter images in the all-images regime, likely due to many similarities between letter and digit images, but the ordering of methods according to robustness is preserved in both regimes.
Ask for Help Only When it is Available. In previous tasks, cautious actions could be identified without considering input features. However, in this (ask, we modify the *discovering non-obvious cautious actions" task by adding the "help" action, whose value depends on an additional input feature. This feature is a binary signal that signals the availability of help. When the signal is true, the "help" action receives a reward of +/220 otherwise receives reward of 11/9. The "help" action is better than any incorrect classification and worse than correctly classifying even the least valuable digit (zero) if help is available. If help is unavailable, the "help" action is the worst action it always receives -11/9. Figure 3a shows the results for each policy in the all-images (left) and single-image regimes (right). With both fashion and letter images, we see that the robust methods with k < N select the help action much more than the non-robust. methods when help available. When help is unavailable, these methods never select the help action and instead choose actions with smaller indices. The average action index docrcases much more when help is available because policies switch from choosing actions with high indices choosing the help action.
How Caution Depends on the Extent of Training Data? Do k-of-N policies really learn to be cautious? Here, we investigate how our cautious algorithms behave with more or less training data. We repeat
We also conducted sensitivity study of the "help" action's reward ranges in (+1/50, + 1/20, +1/5) as shown in B.1.3

---

Published in Transactions on Machine Learning Research (10/2025)
1-of-20
1-of-10
5-of-10
10-of-10
Optim
A.5
Erequent
0.0
100%
1005
10%
1007
1.5
(a) Average action index and help action frequency chosen by each method in each novel enviroument in the "ask help only when it is available"
(b) Average frequency of the help action in each novel environment the "learning to ask for help" task with perturbed rewards where reward models are trained on [1%, 10%, 100%] of the digit dataset
Figure 3: (Left) Results for the "ask for help only when it is available" task. (Right) Results for the "learning to ask for help" task. "f" for fashion and "a for letters. Each star represents a single run.
0.42
1-o8-20
2-of-20
0.28
4-of-20
5-of-20
0.14
10-of-20
20-of-20
collision frequency
Optim!?)
speed
collision speed
Figure 4: Left: frame from the driving gridworld environment. The car is the cyan square, the obstacles are red, and the speedometer is green. Right: normalized 7-discounted safety statistics for each algorithm in the driving gridworld, where each star represents single run.
the "learning to ask for help" task except that rewards perturbed by white noise (standard deviation 0.1) before training, and the training data are [1%, 10%, 100%] of the full digit datased"] In Figure 3b when reward models are trained with 1% of the digit images, observe that decreasing & to increase robustness does not induce caution. Effectively, the neural network ensemble belief has not seen enough data to infer that the "ask for help" action yields a small but consistent reward. Increasing the training set size to 10%, the correlation between robustness and caution returns even stronger than when training with the full datasot. This shows that caution requires enough training data for the agent accurately infer the training reward function, and once achieved, the robusi agents can identify cautious behavior.
Driving Gridworld. For more complex sequential decision-making task, to thoroughly evaluate Con- tribution and 2 we introduce a gridworld driving environment (sce Figure 4h, inspired by AI safety gridworlds Leike et al.) (207)). The state is five-column image: two center columns for a two-lane road, two outer columns for road shoulders, and the last column as a speedometer displaying values from to 3. The agent's car is at the bottom, with the world shifting downward as moves forward. The image height determines the agent's vision range. To limit state complexity, only one obstacle can exist on each half of the vision range plus one,
the gridworld a time, and the vision range is two. The speed limit allowing the agent to "overdríve" its vision by unit. The agent has five actions: change lane left, change lane right, accelerate, brake, and cruise. Acceleration and braking adjust speed by one unit, affecting future movement. Lane changes require momentun. Cruising maintains the current lane and speed. The goal is to maximize distance traveled, as the task is contimuing MDP. Rewards are +1 per forward movement, -2 per off-road space, and -2 times the car's speed for hitting an obstacle.
We investigate how our algorithm reacts to novel situations by restricting obstacles to off-road columns in the familiar MDP and allowing them appear on the road in the novel MDP. Figure shows that more robust policies drive slower, and drive over obstacles both less frequently and at slower speeds in the novel
7 Noise ensures that multiple training samples are needed learn that the help action's expected reward constant.

---

Published in Transactions on Machine Learning Research (10/2025)
MDP, reflecting intuitively cautious driving behavior. Meanwhile, the non-robust policies almost always drive at full speed, matching their behavior with the familiar MDP.
Why do we see this difference? Since obstacles are never observed on the road in the familiar MDP, there is no clear signal that driving over these obstacles will cause a bad outcome. There is a clear signal, however, that driving fast on the road yields larger rewards, so the non-robust policies optimize their behavior around this signal. The robust policies instead take the belief's uncertainty about what could happen when the car drives over an obstacle the road into account. Since there are some reward functions in the ensemble belief that generalize collisions off-road with collisions on the road, the agent learns to avoid collisions in the novel MDP, and prefers slower speeds as there are lower worst-case consoquences.
Our experiments showed a set of simple tasks where it is easy to observe cautious behavior, but still, traditional methods fail to exhibit such behavior. We then showed a proof-of-concept algorithm based on ensembles and robust policy optimization to demonstrate lcarned cautious behavior. Consequently, the findings support Contribution 2
5 N-ff-N CFR for Continuing MDPs
The Driving Gridworld is a continuing MDP, yet k-of-N CFR was first introduced only for the finite horizon setting. This section provides the necessary theory to extend k-of-N to the continuing setting. CFR (Zinke- vich et al. 2008) is an iterative policy improvement algorithm that uses no-regret learning. The regrets in CFR are crounterfactual because, for each abstract decision poinl (something like an MDP state), 8, we imagine modifying the CFR agent's policy in two ways: modifying to maximize its probability of reaching and further modifying it to play a given action at s. We compute the expected return of choosing cach action at under the policy that plays to reach s and apply the CFR agent's policy afterward (counterfac- tual values). The counterfactaal regret of action a at decision point is the immediate counterfactuual-aalee advantage of compared to the policy that plays to reach and applies the CFR agent's policy there. CFR alternates between computing counterfactual regrets (policy evaluation) and applying no-counteffactual regret learning rule at. each decision point (policy improvement). CFR's theoretical properties were originally proven in the extensive-form game framework, where decision points are analogous to state histories in a fized horizon MDP, exponentially expanding the computation and storage required for CFR policies compared to state-based policies. Chen & Bowling (002)) shows that as long as there is no transition uncertainty, is safe to instead implement CFR for fixed horizon MDPs where decision points state state-time-step pairs, and it can be used to compute approximately optimal policy under the k-of-N robustness measure.
We extend Chen & Bowling 20112) 's results to show that, when there is only reward uncertainty, it is actually safe and convenient to define decision points as states alone. We achieve this result by implementing k-of-N CFR in terms of expected returns rather than counterfactual values. For the rest of this section, assume that all MDPs have reward uncerlainly and transition certainly.
denotes the (normalized) 7-discounted expected return of policy π from Let (T;T) [A [A (A, r))
state and Pa(a, r:) (a T:r) (x;r) is the (normalized) expected advantage of choosing action (S: = s') S. = s]. where A; T(-(S,-1) and S: ~ S( 1. A:)
over x in 8. d, : 8'; - (1 7) [>: (S; for i 2 1, is the 7-discounted future state distribution induced by π from initial state s. Kakade 2003) 's
performance difference lemma for this setting is:
Lemma 1. The full regret for using stationary policy π instead of stationary competitor policy x' from state 8 in MDP A,P, de; 7) under reward function r is
(T(TT) (T(T) 1 1-7 E(ps(A, x;r),, where S ~ d'(;3)) and A~ ~ T'(-15).
From Lemma 1 we derive a new regret and optimality bound for CFR and k-of-N CFR, respectively, in continuing MDPs. Given a sequence of reward functions, (r")_ CFR produces sequence of poli- cies, (Tt))= ensures the cumulative advantage of each action at each state grows sublinearly, i.e., ΣT Pt=iPola,tt x'rr) < CT E o(T) for bound CT depends on the state-local learning algorithm used. For exam- ple, it may use regret matching (Hart& & MaseCo 2000) instances at each state to learn from x(r²)
9

---

Published in Transactions on Machine Learning Research (10/2025)
and generate x²(-|s) getting bound of C = 2U /AAT. Combining with Lemma 1 arriving at CFR's cumulative full regret bound (proofs in Appendix [
Theorem 1. CFR bounds cumulative full regret with respect to any stationary policy as
Σ vo(())))) - (=( s C² c01(11-)))
Taking into account Monte Carlo reward function sampling, k-of-N CFR thus inherits the following regret bound:
Theorem 2. With probubility - p > O, the full regret of k-of-N CFR with respect to any stalionary policy, n, is upper bounded by
C
1 1-y + +U 2T
Finally, our theoretical inquiry culminates in the following optimality approximation bound for k-of-N CFR policies:
Theorem 3. With probability 1- p, p O, the best policy in the sequence of policies generated by k-of-N CFR, (T') is an = eTpaproozimation to ( -rrobust policy where
C
2 22og 1/D
T
(1- AU
T
and with probability at least (1 (1 - 9) 0, randomly sampled policy from this sequence is //paapprocimaaton to pa-of---robbusl policy.
Thus, as long as a no-regret algorithm is deployed at each state in k-of-N CFR so that CJ grows sublinearly with T, the sequence of policies generated by k-of-N CFR converges to Mi-ot-N-robust policy with high probability, as stated in Contribution 3
The proofs are in Appendix
6 Conclusions and Future Work
Our algorithm based on a neural network ensemble and k-of-N CFR, shows that agents can learn to be cautious. Our testbeds are simple, they capture key aspects of AI safety, and they facilitate experimental comparisons. We hope that algorithms that learn be cautious can improve the safety of, and our confidence in, deployed Al systems. There are several useful directions for future work. First, critical limitation of our k-of-N CFR. implementation is that it is tabular and requires exact policy evaluation to determine the worst-k reward functions. CFR has been used with function approximation (Waugh et al. 2015 Morrill 2016 Brown et al., 2019; D'Orazio et al. 2020 Steinberger et al. 20200: D'Orazio 2020) and approximate worst-case policy evaluation (Davis) 2015) applying these enhancements could scale our approach to more complicated environments. Second, transition certainty is a strong assumption that will need to be relaxed for additional applications. The increased difficulty of computing robust policics or even minimizing regret with transition uncertainty discussed by Chen & Bowling (2012) and Even-Dar al. (2005). It appears an algorithm must search through policies that condition the entire state history to be sound, which makes policies complex in typical environments. Both theoretical and experimental work, such as weakening the notion of regret as in Morrill et al. 2021 is required to overcome this hurdle.
7 Broader Impact
We hope that lou-eounlgg algorithms will enhance the safety and reliability of deployed Al systems. However, automated safety measures are meant to complement, not replace, human judgment and safety planning. Ultimately, human oversight remains essential ensuring safety. Morcover, while adversarial manipulation of uncertainty or concealment of reward-relevant features could induce excessive conservatism, our framework is designed to drive the agent to act cautiously.
10

---

Published in Transactions on Machine Learning Research (10/2025)
References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Interna- tional Conference on Machine Learning, pp. 22-31, 2017.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Karl Johan Aström. Optimal control of Markov processes with incomplete state information Journal of Mathematical Analysis and Applications, 10:174 205, 1965.
Anil Aswani, Humberto Gonzalez, S Shankar Sastry, and Claire Tomlin. Provably safe and robust learning- based model predictive control. Aulomatica, 49(5):1216 1226, 2013.
Andrew Barto, Marco Mirolli, and Gianluca Baldassarre. Novelty or surprise Frontiers in Psychology. 4: 907, 2013.
Mare Bellemare, Will Dabuey, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pp. 449 458, 2017.
Felix Berkenkamp, Matteo Turchetta, Angela P Schoellig, and Andreas Krause. Safe model-based reinforce- ment learning with stability guarantees. In Advances in Neural Information Processing Systems, 2017.
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret minimization. In International Conference on Machine Learning, pp. 793 802, 2019.
Anthony R Cassandra, Leslie Pack Kaelbling, and Michael I. Littman. Acting optimally in partially ob- servable stochastic domains. In AAAI Conference on Artificial Intelligence, volume 94, pp. 1023-1028. 1994.
Katherinc Chen and Michael Bowling. Tractable objectives for robust policy optimization. In Advances in Newral Information Processing Systems, pp. 2069 2077, 2012.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision-making: a ovar optimization approach. Tn Advances Neural Information Processing Systems, volume 28, pp. 1522 1530, 2015.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-612) 2017.
Yinlam Chow, Ofir Nachum, Edgar Ouenez-Guzmann and Mohammad Ghavamzadeh. A Lyapumov-assed approach to safe reinforcement lcarning. In Advances in Neural Information Processing Systems, pp. 8103 8112, 2018.
William R Clements, Bastien Van Delft, Benoît-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Es- timating risk and uncertainty in deep reinforcement learning. In Workshop Uncerlainly and Robustness in Deep Learning at International Conference on Machine Learning, 2019.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST handwritten letters. In International Joint Conference on Newral Networks, pp. 2921-2926, 2017.
Trevor Davis. Using Response Functions for Strategy Training and Evaluation. Master's thesis, University of Alberta, 2015.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Newral Computation, 5(4):613-6%) 624, 1993.
Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian Q-learning, In AAAI Conference on Artificial Intelligence, volume 1998, pp. 761 -61, 1998.
Ryan D'Orazio. Regret minimization with function approzimation in extensive-form games. Master's thesis, University of Alberta, 2020.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Ryan D'Orazio, Dustin Morrill, James R. Wright, and Michael Bowling. Alternative function approximation parameterizations for solving games: An analysis of f-regression counterfactual regret minimization. In International Conference on Autonomous Agents and Multi- Agent Systems, 5 2020.
Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Experts in a Markov decision process. Advances in Neural Information Processing Systems, pp. 401-408, 2005.
Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Stochastic regret minimization in extensive-Torm games, In International Conference on Machine Learning, pp. 3018 3028, 2020.
Angelos Filos, Panagiotis Tigas, Rowan McAllister, Nicholas Rhinchart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In International Conference on Machine Learning, 2020.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, lan Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pictquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learning Representations, 2018.
Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-144) 1480, 2015.
Mohammad Ghavamzadeh, Marek Petrik, and Yinlam Chow. Safe policy improvement by minimizing robust baseline regret. In Advances in Neural Information Processing Systems, volume 29, pp. 2298-2306, 2016.
S. Hart and A. Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica, 6865)11127- 2000.
TM Heskes, WAJJ Wiegerinck, and HJ Kappen. Practical confidence and prediction intervals for prediction tasks. Progress in Neural Processing, pp. 128-135, 1997.
Zaynah Javed, Daniel Brown, Satvik Sharma, Jerry Zhu, Ashwin Balakrishna, Marek Petrik, Anca Dragan, and Ken Goldberg. Policy gradient bayesian robust optimization for imitation learning. International Conference of Machine Learning, 2021.
Michael Johanson, Nolan Bard, Neil Burch, and Michael Bowling. Finding optimal abstract strategies in extensive-form games. In AAAI Conference Artificial Intelligence, number 1, 2012.
Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine. Uncertainty-aware rein- forcement learning for collision avoidance. orXiv preprint 1xxi::1022.0116 2017.
Sham Machandranath Kakade. On the sample complerity of reinforcement learning. PhD thesis, UCL (University College London), 2003.
Diederik Kingma and Jimmy Ba. Adam: mothod for stochastic optimization. In International Conference for Learning Representations, 2015.
Balaji Takshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive certainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6405-6416, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Cradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 234222,227-- 1998.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. AI safety gridworlds. arXie preprint arXis:1711.09888, 2017.
Lihong Li, Michael L Littman, and Thomas Walsh. Knows what it knows: a framework for self-aware learning. In International Conference on Machine Learning, pp. 568-575, 2008.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pp. 13644 13668, 2022.
Edward Lockhart, Marc Lanctot, Julien Pérolat, Jean-Baptiste Lespiau, Dustin Morrill, Finbarr Timbers, and Karl Tuyls. Computing approximate equilibria. in sequential adversarial games by exploitability descent. International Joint Conference on Artificial Intelligence, 2019a.
Edward Lockhart, Marc Lanctot, Julien Pérolat, Jean-Baptiste Lespiau, Dustin Morrill, Finbarr Timbers, and Karl Tuyls. Computing approximate equilibria. in sequential adversarial games by exploitability descent. International Joint Conference on Artificial Intelligence, 2019b.
Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. In Advances in Neural Information Processing Systems, pp. 2260-3268, 2017.
Colin McDiarmid. Concentration. In Probabilistic methods for algorithmic discrete mathematics, pp. 198- Springer, 1998.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Non- parametric return distribution approximation for reinforcement learning. In International Conference on Machine Learning, pp. 799 806, 2010.
Dustin Morrill. Using regret estimation to solve games compactly. Master's thesis, University of Alberta, 2016.
Dustin Morrill, Ryan D'Orazio, Reca Sarfati, Mare Lanctot, James R. Wright, Amy R. Greenwald, and Michael Bowling. Hindsight and sequential rationality of correlated play. In AA Conference on Artificial Intelligence, 2021.
Tan Osband, Benjamin Van Roy, Daniel J Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1 62, 2019.
Ian Osband, Zheng Wen, Soyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. In Advances in Neural Information Processing Systems, volume 36, pp. 2795-2823, 2023.
Adamn Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performanee deep learning library. In Advances in Newral Information Processing Systems, pp. 8024-8035, 2019.
Tim Pearce, Mohamed Zaki, and Andy Necly. Bayesian neural network ensembles. In Workshop on Boyesion Deep Learning, Neural Information Processing Systems, 2018.
Marek Petrik and Dharmashankar Subramanian. Raam:
The benefits of robustness in approximating aggre- gated MDPs reinforcement learning. In Advances Neural Information Processing Systems, volume 27, pp. 1979 1987, 2014.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Leurning Representations, 2018.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse Bayes-adaptive reinforcement learning. In Ad- vances in Neurul Information Processing Systems, volume 34, pp. 1142 1154, 2021.
Krishnan Srinivasan, Benjamin Eysenbach, Schoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep RL with safety critic. arxiv preprinl or arxiv:2010. 16::2010.14003, 2020.
Eric Steinberger, Adam Lerer, and Noam Brown. DREAM: Deep regret minimization with advantage baselines and model-free learning. arXiv preprint ar Xim:2066. 10410, 10, 2020.
13

---

Published in Transactions on Machine Learning Research (10/2025)
R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Newral Information Processing Systems, pp- 1057-1063, 2000.
Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. In Conference on Robot Learning, pp. 1078-1093, 2020.
Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill. Preventing undesirable behavior of intelligent machines. Science, 366(6466):999-10044 2019.
Robert Tibshirani. A comparison of some error estimates for neural network models. Neural Compulation, 8(1):152 163, 1996.
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite Markov decision processes with Gaussian processes. In Advances in Newral Information Processing Systems, 2016.
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. In Advances in Neural Information Processing Systems, volume 33, pp. 12151-12162, 2020.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L Willke. Out- of-distribution detection using an ensemble of self supervised leave-out classifiers. In Europeon Conference on Computer Vision, pp. 550-564, 2018.
Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained Markov decision processes. In International Conference on Machine Learning, pp.
2020.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, King's College, Cambridge, 1989.
Kevin Waugh, Dustin Morrill, J. Andrew Bagnell, and Michael Bowling. Solving games with functional regret estimation. In AAAT Conference Artificial Intelligence, 2015.
Ronald J. Williams. Simple statistical gradient-foloowing algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229 256, May 1992.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint NNN::100.0072 2017. MIT license.
Qisong Yang, Thiago D Simão, Simon Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft actor critic for safoty-constrained reinforcement learning. In AAAT Conference on Artificial Intelligence, number 12, pp. 10639-10646. 2021.
ChengYang Ying, Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu. Towards safe reinforcement lcarning via constraining conditional valuo-at-risk. International Joint Conference on Artificial Intelli- gence, 2022.
Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with a safety editor policy. In Advances in Neural Information Processing Systems, volume 35, pp. 2608 26212 2022.
Tom Zahavy, Andre Barreto, Daniel Mankowitz, Shaobo Hou, Brendan O'Donoghue, lurii Kemaev, and Satinder Singh. Discovering a set of policies for the worst case reward. In International Conference on Learning Representations, 2021.
Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, and Dinesh Jayaraman. Cautious adaptation for reinforcement learning in safety-critical settings. In International Conference on Machine Learning, pp. 11055 11065, 2020.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization games with incomplete information. In Advances in Neural Information Processing Systems, volume 20, pp. 1229-173, 2008.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shi- mon Whiteson. Varibad: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. Tn International Conference on Learning Representations, 2020.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Supplementary Materials
A
Related Work
Our work builds on a growing body of research in robust and safe RL, particularly methods leveraging CVaR to mitigate undesirable outcomes under uncertainty. While we share with prior work the motivation to develop policies that are robust to risk and uncertainty, our problem formulation introduces a distinct setting based on epistemic, not aleatory, uncertainty. Furthermore, in our setting, the agent learn to act cantiously without prior knowledge or constraints.
In partially observable MDPs (POMDPs), the agent does not observe the environment state, instead the agent should learn to infer the state from its observations (Aström, 1965). In contrast, our approach assumes full state observability during both training and deployment. Since learning does not occur after deployment, rewards can be considered unobserved. Cautious behavior during deployment. would then seek to avoid states and actions whose value has epistemic uncertainty, as this uncertainty cannot be resolved post-deployment. This is in contrast with POMDPs, where all uncertainty is aleatoric and can be resolved through further observation. For instance, in the classic Tiger POMDP (Cassandra et al., 1994), tiger is randomly selected to be behind one of two doors, and cautious behavior might be to listen for the tiger behind the door before opening, as the listening observation can resolve the uncertainty. However, this paper focuses on fully-observable state settings, where uncertainly comes from facing novel situations and caution is about epistemic uncertainty.
In distributional RL instead of estimating the expected return, a full distribution over returns estimated, aimed to capture stochasticity in rewards and transitions (Dearden et al., 1998). This distribution can be used in policy evaluation and control settings (Bellemare et al. 2017). but maybe most relatedly, it has been used to achieve risk-sensitive robustness by optimizing conditional value at risk (CVaR) in non-stationary environments where the reward is stochastic (Morimura el al. 2010). The distribution over the expected return can quantify both aleatoric and epistemic uncertainties; however, it does not differentiate between them, and so such risk sensitivity would avoid all stochastic outcomes, not just those that are unfamiliar. Similarly, Risk-averse RL characterizes an agent's 's uncertainty about future return and then chooses robust behaviors, f.e., those that maximize the agent's 's reward assuming unfavorable conditions (often with formal risk measure). There are various methods for learning policios that are robust to alcatoric uncortainty (Chow et al. 2017 Tang et al. 2020: Clements et al. 2019), which again applies risk sensitivity to the wrong calegory of uncertainty. Furthermore, since the mapping from images and actions to rewards is deterministic in our MNIST example task, there is, in fact, no alcatoric uncertainty to be robust to.
Constrained MDPs (CMDPs) (Altman) 1999) specifies safe behavior via constraints that an agent must not violate (Garcia & Fernández 2015 [erkenkamp al., 2017 Chow al. 2018). CMDP can be solved using RL in a model-based (Aswani et al. 2013 Berkenkamp et al.] 2017] model-free (Achiam)
et al. 2017 Chow et al.) 2018 Srinivasan et al.) 2020) way. However, these approaches require pre-defining safe states that the agent is allowed to visit or safe actions the agent can take. Some approaches design "safety functions" that incentivize pre-defined safe behaviors (Turchetta et al. 2016 Wachi & Sui 2020: Turchetta et al.] 2020). or a prior over the reward in imitation learning (Javed et al. 2021). Similarly, Yuu et al. 2022 Liu et al. 2022) operate under CMDPs framework, assuming known constraints and observable rewards during training and evaluation. These methods aim to minimize constraint violations or optimize under constraint satisfaction. Compared to CMDPs, our work aims to make the agent learn to behave cautiously when facing novel situations without relying explicit constraints. Approaches requiring priori description of safcty information about specific scenarios present a scaling problem, as it is generally infeasible to enumerate all potentially hazardous situations in a realistic application. CMDPs can be used as an approach for achieving a cautious policy with expert domain knowledge, but in no sense is it learning it.
In Knows What It Knows (KWIK), given state-action-nest state tuple, the agent should predict the probability of the transition if the agent has observed it sufficiently (known) 1. if it has not observed
16

---

Published in Transactions on Machine Learning Research (10/2025)
it sufficiently (unknown) Li et al. 2008). Therefore, KWIK does not know what to do when facing the "unknown". KWTK can be viewed as classifier to distinguish between familiar and novel states, which can then be used to hand-craft a cautious policy e.9., "action eleven" in the MNIST example. However, as with CMDPs, KWIK does not learn this cautious behavior.
Several prior methods incorporate the CVaR risk measure into policy optimization, such as Ying et al. 2022). However, their formulation defines risk solely over returns within a single fixed training environment, capturing only alcatoric uncertainty. Their subsequent evaluation explores whether optimizing for risk in aleatoric uncertainty helps with environment perturbations (transition observation changes), which be thought of as adding epistemic uncertainty. Their results largely showed little induction of caution under perturbation scenarios due to optimizing for only aleatoric uncertainty. Similarly, (Yang et al. 2021 uses CVaR. to temper pessimism in worst-case optimization, but assumes that rewards remain defined and learnable during adaptation. In contrast, we focus on epistemic uncertainty stemming from lack of knowledge deterministic environments, where the agent encounters novel MDPs (o.g., MNIST vs. Fashion- MNIST) in which rewards are entirely undefined at test time.
Overall, while these works focus robustness under aleatoric uncertainty or predefined constraints, our approach addresses epistemic-safe reinforcement learning by enabling agents to act cautiously in novel envi- ronments without access to reward or constraint supervision. We emphasize zero-shot generalization, where agents must navigate novel tasks with entirely undefined rewards.
B Experiments
In all experiments, k-of-N CFR is implemented with regret matching (Hart & Mas- 2000 which is deterministic and hyperparameter-free free. However, since k-of-N CFR requires sampling N reward functions, its output policy random. Fach of the 2, 000 trained neural network reward function models represents a single sample from an implicit belief, so sampling N of them consists of pulling N of these reward function models out of a queue. To account for the random variation caused by the ordering of the reward function models in the queue, we run multiple repetitions of k-of-N CFR by shuffling the order of the reward function models in the queue before the start of cach run. We run 10 repetitions in cach MNIST experiment and 5 in the driving gridworld experiment. PyTorch Paszke et al. 2019) is used to build and train all neural networks.
B.1 MNIST Experiments.
For MNIST experiments, we tested three neural network architectures. One used four fully connected layers soparated by rectified lincar unit (ReLU) activations and the second used two convolutional layers each with one output channel, followed by two fully connected layers. These architectures were outperformed by one that begins with two convolutional layers and ends with three fully connected layers, all separated by ReLU activations. The first convolutional layer has single input channel and 64 output channels with x 4 kernel followed by 2 x 2 max-pooling. The second convolutional layer is the same except it has only 16 output channels. The fully connected layers have 50, 15, and 10 outputs, respectively. All results use this architecture.
Networks are trained to minimize the mean-squared error (MSE) between reward predictions and target rewards with the Adam optimizer (Kingma & Ba 2015) using a learning rate of 0.0016 (we also try 0.01, and 0.001). The remaining parameters for Adam in PyTorch (8,, fg, G, and weight decay) are set to their defaults (0.9, 0.999, 10-8, 0) without the AMSGrad (Reddi et al., 2018)) modification. In the "discovering non-obvious cautious actions" and "ask for help only when it is available" experiment, we weight the loss each output index a E (0....,9) 9} according to 1/(a + 1)2 and weight the help action by one. See Table 1 for the batch sizes and the number of epochs run in each MNIST experiment.
For all MNIST experiments, we used an NVIDIA Tesla V100 GPU and a 2.2 GHz Intelø Xeon@ CPU with 100 GB memory. Since we use a neural network ensemble with 2, 000 models for each experiment, takes about 50 GPU hours for each experiment, which makes total of 300 GPU hours for all of our MNIST experiments.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 2 Learning to Be Cautious
Initialize N, k, and number of iterations T
Train N * T reward functions R(r(X) using Deep ensemble models on familiar MDP Input: dataset X containing images (e.g., MNIST, Fashion-MNIST, EMNIST) Initialize: Total Regret pf 0, a random policy xo(a[X)
for 0 to I do
Sample reward functions R()lX) from the ensemble Inference rewards from reward functions R, Estimated state value:
m
(x( Σ (r(r, m((alz;)
==0
Select least & estimated state value (X( and corresponding least reward functions. and Calculato mean of least reward functions
Wp Σ (r(
j=0
Immediate regret:
|
p = W - Σ
x) W
a=0
Total Regret: p = p + max pt)
Update policy: Tal1( end for
p"
E.50 pr
Table 1: The batch size and number of epochs used to train the neural network reward models for each setting in the "how caution depends on the extent of training data" experiment. All other MNIST experiments use the same settings as the 100% case.
training data fraction
batch size
# of epochs
1%
64
10, 100
10%
128
0,00
100%
512
100
The code for each MNIST experiment is available in zip file The actions directory contains k-of-N policies and expected values across each ileration and run in the MNIST experiments. The MNIST experiment code can be found in mode/mnist__ mamiis_experimenteee code split into a single shared file k_of n. py and two Jupyter notebooks for each experiment, one to train neural networks and run k-of-N CFR, and another to generate plots. The driving gridworld code can be found in http//dvvvinngridec aerieirreeeeepermeeetttt experiment_codee
B.1.1 K-of-N convergence
The progress of cach k-of-N CFR policy, measured in terms of expected return on the N reward functions used on each CFR iteration, is given for each MNIST experiment in Figures 7 progress k-of-N CFR in the driving gridworld is similar. The value always plateaus relatively quickly with more repetitions would not change
10 13 and 17 to 19 The
little variation between runs, indicating that rumning more iterations the results substantially. Because each run uses different sets of N reward functions on each iteration by design, the value would still show variation even if the policies for different were identical.
18

---

Published in Transactions on Machine Learning Research (10/2025)
B.1.2 Confusion matrix
We analyzed the behavior of k-of-N CFR. policies on the E-MNIST dataset by examining the confusion matrix of the final action probabilities for each letter class. The matrix reveals that, in the 'learning to ask for help' experiment, for most letters, the policies confidently select the classification action rather than the "help" action, indicating generalization from previously monitored digit classes. However, a distinct pattern emerges for specific letters that visually resemble digits. For instance, the policies are more likely to select the "help' action when shown the letters o, 8, i, 1, J, and 2, which closely resemble the digits 0, 5, 1, and 2, respectively, as shown in Figure g This behavior suggests that the agent recognizes higher epistemic uncertainty in such ambiguous cases and opts for monitoring to avoid potential misclassification.
We analyzed the behavior of k-of-N CFR policies on the E-MNIST dataset by examining the confusion matrix of the final action probabilities for each letter class. The heatmaps for these confusion matrices are presented Figure 12 for the 'discovering non-obvious cautious actions' task, and in Figure 16 for the "ask for Help Only When It Available" task. Meanwhile, the confusion matrices for the "how caution depends the extent of training data" task are shown in Figures 24 and 25
B.1.3 "last action" reward sensitivity
We conducted a sensitivity analysis on the reward associated with the "help' action in both the 'learning to ask for help" and 'ask for help only when it is available' tasks.
In the 'learning to ask for help' experiment, we varied the 'help' reward across 0.1, 0.25, 0.5 in both the Fashion and E-MNIST environments. The results show that when the reward is low (e-g., 0.1), cautious agents tend to request help less frequently. In contrast, with higher rewards (e.g., 0.5), help is selected almost 100% of the time. Importantly, the relative performance ranking among cautious policies remains consistent across these values, as shown in Figure 5
We evaluated accuracy and the frequency of selecting the "help' action across varying levels of the 'help" reward (E 0.1,0.25 0.5). Across all reward values, the "all-images" regime maintained consistently high classification accuracy (> 99%), with a negligible reliance on help, even at the highest reward level (0.5); help was selected less than 1% of the time. In contrast, the "single-image" regime showed greater variability. At low help rewards (0.1), agents rarely used the help action, while at higher rewards (e.g., 0.5), the frequency of requesting help increased substantially (up to 7.63% in the 1-of-20 case), particularly under more cautious settings. Importantly, as the 'help' reward increased, "help" usage rose and accuracy improved, suggesting that agents Icarned to request help when beneficial, especially under epistomic uncertainty. These findings validate that our cautious policies adapt help-seeking behavior based on its utility, while maintaining strong performance across regimes, as shown in Tables to
Table 2: Frequency of the correct label action index and the help action across the 10,000 MNIST test images in the "Learning to Ask for Help" task, when the 'help' reward 0.1. Values reported as 0.00 are not necessarily zero but have been rounded to two decimal places.
1-of-20
1-of-10
5-of-10
10-of-10
Correct 99.38:10.03
90.08+0.02
99.39-10.01
99.40-10.01
all-images regime
Help 0.00+0.00
0.00
0.00
0.00
Correct
98.83++0.11 99.16=#0.06
99.37+0.04
99.39-10.01
single-image regime
Help 0.00+10.00
0.00+00.00
0.00-10.00
0.00+10.00
In the "ask for help only when it is available" experiment, we evaluated rewards in 0.02, 0.05, 0.2 to understand behavior when help availability is intermittent. As expected, higher 'help' rewards increased the frequency of selecting the help action when it was available, as illustrated in Figure
The most cautious policy (1-of-20) in the all-images regime selects the help action upon observing most letters except for those similar to digits (e.g., T/i, I,/1, O/o, S/s and 7.22) as shown in Figure g The effect
19

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Frequency of the correct label action index and the help action across the 10,000 MNIST test images in the "Learning Ask for Help" task, when the "help' reward 0.25
1-of-20
1-of-10
5-of-10
10-of-10
Correct
99.35=0.02
99.36=0.03
99.38=0.01
99.3911.01
all all-images regime
Help 0.03++0.01
0.03+0001
0.02++0.00
0.02+00.00
Correct
96.87:20.05
97.95
99.07100002 0.41#00001
99.39-10.01 0.02+0.00
single-image regime
Help 2.83#0.01
1.72#0.04
Table 4: Frequency of the correct label action index and the help action across the 10,000 MNIST test images in the "Learning to Ask for Help" task, when the "help" reward = 0.5
1-of-20
1-of-10
5-of-10
10-of-10
Correct
98.82:40.04
98.92:1.04
99.05=1.001
99.10=10.01
all-images regime
Help 0.89+0.04
0.76+0.05
0.57#0.01
0.52+0.01
Correct
92.37:t0.12 94.94+0.08
99.08=0.02
99.10:1.001
single-image regime
Help 7.63+0.12
5.05+
1.80++0.22
0.52++0.02
is exaggerated in the single-image regime where 1-of-20 selects the help action with very high probability except for letters similar to digits. The bascline Optim(f) does not sclect the help action at all.
The most cautious policy (1-of-20) in the all-images regime picks the help action when help is available and otherwise selects less risky actions with small indices upon observing most letters except for those similar to digits, as shown Figure I6 The effect is exaggerated in the single-image regime.
20

---

Published in Transactions on Machine Learning Research (10/2025)
1-of-20
1-of-10
5-of-10
10-of-10
Optim(F)
1.0
frequency
0.8
0.6
0.44
0.2
0.0
f
f
a
(a) "help" arm 0.1
1.0
frequency
0.8
0.6
0.4
0.2
0.0
f
α
α
(b) "help" amm 0.25
1.0
Trequency
0.8
0.6
0.4
0.2
0.0
f
α
f
α
(c) "help" arm 0.5
Figure "Help" arm reward sensitivity the "learning to ask for help" task, where the 'help" reward E (..1, 0.5), ("f" for fashion and "a" for letters). Each star represents a single run, and the violin represents the variance across 10 runs.
Table 5: Frequency of the correct label action index and the help action across the 10,000 MNIST test. images in the "Learning to Ask for Help" task
1-of-20
1-of-10
5-of-10
10-of- 10
Correct
99.35-10.02
99.36-10.03
99.3811.01
99.39-10.01
all-images regime
Help 0.03+0.01
0.03+0.01
0.02+00.00
0.02+0.00
Correct Help
96.87+0.05
97.9510.07
99.07+0.02
99.39-10.01
single-image regime
2.83+0.01
1.7240.04
0.41+00.01
0.02+0..00
21

---

Published in Transactions on Machine Learning Research (10/2025)
1-of-20
1-of-10
5-of-10
10-of-10
1.0
action index action index help
0.5
3
6
a
(a) "help" arm 0.02
action index action index help
0.5
0.0
f
(b) "help" amm 0.05
1.0
action index action index help
0.5
6
6
3
a
a
(c) "help" 0.2
Figure 6: Average action index and frequency of selecting the 'help' action by each method in various novel environments, under different rewards for the "help" arm in the "learning to ask for help only when available" task. The "help' reward takes values E {0.02, 0.05, 0.2}, ("f" for fashion and "a" for letters). Each star represents a single run, and the violin represents the variance across 10 runs.
22

---

Published in Transactions on Machine Learning Research (10/2025)
1-ot-20
1-0f-10
5-0f-10
10-of-10
0.8
0.8
0.8
0.5
0.5
Velmi:
0.2
0.8
0.8
Veln
0.8
0.2
0.2
0.0
20
40
60
100
0.0
20
40
80
100
0.0
20
40
60
80
100
iteration
iteration
iteration
The expected return of each k-of-N policy on each iteration & given the sampled k-of-N reward function, rt, in the "learning to ask for help" task. A single bold line shows the average across all ten runs while the values from individual runs are given by thinner lines. (top row) All-images regime, (bottom row) single-image regime, (left column) digits, (middle column) fashion, (right column) letter.
Figure 7:
I-oF-20
1-of-10
5-of-10
10-of-10
Optim(A)
frequency
0.4
0.2
frequency
0.6
0.4
0.2
help
help
action
action
Figure 8: Action distribution of each k-of-N policy and bascline in the "learning to ask for help" task. (top
row) All-images regime, (bottom row) single-image regime, (left column) fashion, (right column)
letter.
9help
9help
8 9hepp
01010
R/
R/b 0
64
88
8h 01
82
0c
Ck
83
0
01000000111
R8
CK
05
81
0103
010
0id 01000
00000000000000000000000000000000 Did
Be 0000000 01000100000000000000000001100001
Si 1
010302
8270
64
61
8161
4 1617
3 8183
1UN
16 0
IM 00
00000000000000000000000000000000 00000000000000000000000000000000
KK S
02010
letter
LI
6T 6181816133 L1 X
Mmm
Km
0100000010 Kmn 011
Nm
Nm
D 001090 hp D U 00000
PP
On D D 100000000000
ON
03 03 DARS
Ra
RA
0
RA
0102010107
56
51
0
01
201 0303
Ul
54
Ui 0 0101 0 RF
UN
05
84
01 5
VIV
12
WV 00000000 0
WV
2012
m3
My 5.1
Wy
87
22
0
22
D.S 0 0
action
action
action
Figure 9: The average frequency of each action each letter, averaged over lowercase and uppercase images, in the "learning to ask for help" task. (left) 1-of-20 all-images regime, (middle) 1-of-20 singlc-image regime, (right) Optim(f).
23

---

Published in Transactions on Machine Learning Research (10/2025)
1-of-20
1-of-10
5-of-10
10-of-10
0.5
0.5
0.4
0.4
Vebr:
0.3
0.3
0.3
0.1
0.5
0.5
0.4
0.4
Ve
0.4
0.1
20
40
60
80
100
0.0
20
40
60
80
100
0.0
20
40
60
100
iteration
iteration
iteration
Figure 10: The expected return of each k-of-N policy each iteration given the sampled k-of-N reward function, Ft, in the "discovering non-obvious cautious actions" task. (top row) All-images regime, (bottom row) single-image regime, (left column) digits, (middle column) fashion, (right column) letter.
1-of-20
1-oF:10
5.of-10
10.of-10
Optim(F)
0.8
frequency
0.6
0.4
0.2
0.0
0.8
frequency
0.6
0.4
0.2
0.0
0
1
2
6
7
8
9
0
1
2
6
7
8
9
action
action
Figure 11: Action distribution of each k-of-N policy and baseline in the "discovering non-obvious cautious actions" task, dotted lines represent average action taken by each policy. (top row) All-images regime, (bottom row) single-image regime, (left column) fashion, (right column) letter.
A/a
81
AW
70
67
82
6
AW 01
JA
10
D
03
0
01
010
63
B0
01
5C
0-15100111
01010
BC
01
63
FY
RA
010701
M
03
0/9
6T
5
0
6
P S
00030
W
03103
0110
8.01
0261111
0.
010110
KI
100
020110
62033
letter
81
01
03
Mm
03
00
Km
37
01
01
Nh
03
0
010011
0/0
0
ON
SJOT
MT 11
RT
8 STRTKT
STAT
03
SN
B
SA
3
TA
0203
0.000
15
03000330:0000
8101
TM
6203
0000000000000
02
0
01
-02
WWW
87
6
Www
50
61
WM
03
5.1
Xo
XX
07
XX
0
vY
action
action
action
Figure 12:
The average frequency of each action on each letter, averaged over lowercase and uppercase images, in the "discovering non-obvious cautious actions" task. (left) 1-of-20 all-images regime, (middle) 1-of-20 ingle-image regime, (right) Optim(f).
24

---

Published in Transactions on Machine Learning Research (10/2025)
o-
1-of-10
5-o5-10
0.3
0.3
0.3
0.2
0.2
0.1
0.0
0.0
0.0
0.4
0.4
0.4
0.3
0.2
0.1
0.1
0.0
20
100
0.0
40
100
0.0
20
00
100
iteration
iteration
iteration
Figure 13: The expected return of each k-of-N policy on each iteration given the sampled k-of-N reward function, Ft, in the "ask for help only when is available" task. (top row) All-images rogime, (bottom row) single-image regime, (left column) digits, (middle column) fashion, (right column) letter.
1-06-20
10-af-10
Optimiii
frequency
3.4
0.0
1.0
frequency
8
6
belp
6
help
action
action
Figure 14: Actions distribution for each k-of-N policy and baseline the "ask for help only when is available" task in case that help is available. dotted lines represent the average action taken by each policy. (top row) All-images regime, (bottom row) single-image regime, (left column) fashion, (right column) letter.
1-u620
1-of-10
5-of-10
10-aF-10
Optiza7)
frequency
9.6
0.4
0.2
0.8
frequency
86
922
0.0
3
6
8
belp
1
2
3
4
6
5
9 help
action
action
Figure 15: Actions distribution for each k-of-N policy and baseline in the "ask for help only when it is available" task in case that help is unavailable. dotted lines represent the average action taken by each policy. (top row) All-images regime, (bottom row) single-image regime, (left column) fashion, (right column) letter.
Table 6: Frequency of the correct label action index and the help action across the 10,000 MNIST test images in the "ask for help only when is available" all-images regime.
1-of-20
1-of-10 10
5-of-10
10-of-10
Optim())
Correct
97.50+3..11
98.57+1.00
99.27+0.01
99.27+4.01
89.43:+4.16
help is available
Help 0.39-10.48
0.14+0.02
0.09+10.00
0.07-10.00
1.16+10.00
Correct
97.644+3.05
98.67+00.99
99.32++0.01
99.31++0.01
94.96++.00
help is unavailable
Help 0.00+0.00
0.0010.00
0.00+00.00
0.00+00.00
0.2343.87
25

---

Published in Transactions on Machine Learning Research (10/2025)
Table 7: Frequency of the correct label action index and the help action across the 10,000 MNIST test images in the "ask for help only when it is available" single-image regime.
1-of-20
1-of-10
5-of-10
10-of-10
Optim(F)
Correct
84.13+1.00
90.68+1.54
98.80=0.001
99.28+1001
89.43+44.16
help is available
Help
10.42:40.51
3.25+0.47
0.05-+0.01
0.07++0.01
1.16:t0.00
Correct
88.54+1.14 92.18+1.59
99.06+0002
99.31+0.01
94.96+4.01
help is unavailable
Help
0.00+10.00
0.00
0.00+10.00
0.00::0.00
0.23
thepp
0
9heto
0
Shelp
0
Ala
00
0
00
8
63
Ck
0
0
0
0
CK
02
03
13
83
5
Dd 02
3
67
Be
0
Be
0
0
0.0
0
Eie 01
03
00
M
M 0
030000
RV9
69 0
02 0
Gg
0303000
N6
Mh
02
KK
KX
66
letter
LI
0
5
Mm 0
0
DM
Mim MA
Kn 0
Nmn C
On 03
ON 09
PP
Pi 13
0/
S
56
04
1
14
0T
67
UV
Ub 0
U
U
09
DNI
D6
WV
09
WV 010T
0300000 060
Xx
XX
Tiy
MV
Ala
Al
07 0
Gk
11A
10000
CK
0
00000 0
Did
85
04
0
0100
Ew
01
3
FY
RY
F# 4
03
6/
00000
Grg
Mhh
Nh
F10.
0
Hhh
letter
0
0
87
Nn 2110.
0
3
0
Nn UA
03
do
030000
DV
09
Pp
F10
Pip
-04
01000000
Qig
03
13000
54
Wu
01 11
87
000000
Unu 87
VN
VV
0.52
WV 0101 6
05
0
Wh
WW
0
New
XXX
XX
00
XX
Tyy
TV
0000000
000000000
action
action
action
The average frequency of each action on each letter, averaged over lowercase and uppercase images, in the "ask for help only when is available" task. (top row) Help available (bottom row), help is unavailable. (Left column) 1-of-20 all-images regime, (middle column) 1-of-20 singlo-image regime, (right column) Oftim(f).
Figure 16:
26

---

Published in Transactions on Machine Learning Research (10/2025)
Table 8: Frequency of the correct label action index and the help action across the 10,000 MNIST test images in the "learning to ask for help" task with perturbed rewards in the all-images regime, where reward models are trained on 1%, 10,, or 100% of the digit dataset.
1-of-20
1-of-10
5-of-10
10-of-10
Optim(f)
Correct
95.81+0.14
95.82++0.18
95.97=10.08
96.10+0.006
89.43:t4.16
1%
Help
0.39-10.03
0.41
0.34
0.30:10.03
1.16:10.93
Correct
98.63++0.07
98.62+00.08
98.67+00.04
98.67+00.03
94.96+4..01
10%
Help 0.1310.01
0.1210.01
0.12+0.01
0.10:10.01
0.23+3.87
Correct
99.34:10004
99.3611.04
99.3811.002
99. 99.3811.002
98.23
100%
Help
0.04+0.00
0.03#0.01
0.03+0000
0.02+0.00
0.08
Table 9: Frequency of the correct label action index and the help action across the 10,000 MNIST test images in the "learning ask for help" lask with perturbed rewards the single-image regime, where reward models are trained on 1%, 10%, or 100% of the digit dataset.
1-of-20
1-of-10
5-of-10
10-of-10
Optim(f)
Correct
86.96:10.41
89.75-10.34
94.56::0.12
96.11-0..06
89.43114116
1%
Help 5.80#1026
4.49
1.61++0.06
0.29+0000
1.16=#0.3
Correct
88.78++0.16
92.51
97.50++0.06
98 98.68-+0.03
94.96++4.00
10%
Help 9.08+1.10
6.61
1.16::0.02
0.11
0.23
Correct
96.6110.00
97.67 97.6710.06
99.07
99.37110.02
33.53
100%
Help 2.47+0.02
1.52+0.03
0.36+0.01
0.02+0.00
0.08
1-06220
1-of-10
5-of-10
10-of-10
0.6
Ve(ai-
0.0
0.0
0.0
0.6
Voln?))
0.4
0.4
0.4
0.2
0.2
0.2
0.0
20
60
100
0.0
40
00
100
0.0
20
60
100
iteration
iteration
iteration
Figure 17: The expected return of each k-of-N policy each iteration given the sampled k-of-N reward function, ff, in the "learning to ask for help" task with perturbed rewards, where reward modols are trained on 1% of the digit dataset. (top row) All-images regime, (bottom row) single-image regime, (left column) digits, (middle column) fashion, (right column) letter.
27

---

Published in Transactions on Machine Learning Research (10/2025)
o-
1-0f:10
5-of-10
10.of:10
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.8
0.8
Ve
0.6
0.6
0.6
0.4
0.4
0.2
40 iteration
0.0
iteration
iteration
The expected return of cach k-of-N policy each itcration given the sampled k-of-N reward function, Ft; in the "learning to ask for help" lask with perturbed rewards, where reward models are trained on 10% of the digit dataset. (top row) All-images regime, (bottom row) single-image regime, (Ieft column) digits, (middle column) fashion, (right column) letter.
Figure 18:
1-ot-20
1-of-10
5-of-10
0.8
0.8
0.8
0.5
0.6
0.4
0.2
0.8
Vele:
0.5
0.2
0.2
0.2
0.0
20
40
60
80
100
0.0
20
40
60
60
100
0.0
20
40
60
80
100
iteration
iteration
iteration
The expected return of each k-of-N policy on each iteration given the sampled k-of-N reward function, rf, in the "learning to ask for help" task with perturbed rewards, where reward models are trained 100% of the digit dataset. (top row) All-images regime, (bottom row) single-image regime, (left column) digits, (middle colunn) fashion, (right column) letter.
Figure 19:
Gholp
9holp
Ghopp
Ara
Ala
Ala
02
82
62
01
8/
Bib
B6
D1
0101
08
01
BT
CX
GK
CK
Fi B7
61
61866111111 116116
Fix 03
5 8101810111
3 816618
T8
0303333000
R
0303
010383
PM 2 01011 3
Gi 010100111100000000000000
01111101111111000000
Gig 00100000100000000011
40h
000000
M0
0
D10 413 413 0
81018197 487 4.61 3
KV
Kk D R7C
KK
B1 070105
63
letter
LA
D3 1000000
Nm 05
0 0 03
Nm DE
D1
02
0301
03
000
0 0000
010
03 0 0 0
ON DM 5 6
5
P1
00100
Og
Pp
63
K3 00 61STON
636163
63
Gi
Pg 01
XX DH St 2 01
01010101
02
0101010101
020163
-04
S/s
SS 0 D
4 VTV
TA
TR D
CEETMOORION 0010 BORIIE
TR BIRTHU 81018100 02
ET
mM F111 m1 BT
81
A1 13
3
8. 3 8 BT
Xo
0203
XXx
9203
0 03
XX 2 02011
0113
03 02 03
Yiy
Vy
Vy
00003
action
action
action
The average frequency of each action on each letter, averaged over lowercase and uppercase images, in the "learning to ask for help" task with perturbed rewards, where reward models are trained on 1% of the digit dataset. (left) 1-of-20 all-images regime, (middle) 1-of-20 single-image regime, (right) Oftim(f).
Figure 20:
28

---

Published in Transactions on Machine Learning Research (10/2025)
1-9E-20
1-of-10 5-oé-10
10-af-10
Optim/hi
frequence
0.8
0.2
1.0
frequency
0.0
0.6
help
action
action
Figure 21: Action distribution of each k-of-N policy and baseline in the "learning to ask for help" task with perturbed rewards, where reward models trained on 1% of the digit dataset. (top row) All-images regime, (bottom row) single-image regime, (left column) fashion, (right column) letter.
1-oF-20
1-of-10
5-06-10
10-of-10
Optint?)
1.0
frequency
0.0
0.4
1.0
frequency:
0.8
0.6
34
0.0
3
6
8
help
2
3 4
6
8
help
action
action
Figure 22: Action distribution of each k-of-N policy and baseline in the "learning to ask for help" task with perturbed rewards, where reward models are trained on 10% of the digit dataset. (top row) All-images regime, (bottom row) single-image regime, (left column) fashion, (right column) letter.
1-uE-20
1-of-10
5-o6-10
10-aF-10
Optimt7)
1.0
frequency
0.01
0.6
0.4
0.
0.0
frequency
0.0
help
help
action
action
Figure 23: Action distribution of each k-of-N policy and bascline the "learning ask for help" task with perturbed rewards, where reward models are trained 100% of the digit dataset. (top row) All-images regime, (bottom row) single-image regime, (left column) fashion, (right column) letter.
29

---

Published in Transactions on Machine Learning Research (10/2025)
9help
9help
9help
00
Bb
62 0
69
8h 01
04
0
Crc
02
03
83991
CR 83
00000000000000000000000000000001
0
GC 06
01
01
014
01 83
00000000000000000000000000000000
Did 030103
61
61
56
01010111110000000000000011011011
56
64
G1g
Gig
01
01 01000000000000000000000000000001
Gg
C 0100
110011001101
01010200
01000000100
01010
10h
10h
0
01101000000001000000000000111011 01
1th 01
12 853
4
1000000
01010101100000
3
Kk 1
0.01 01001110000 0
KK
020103
0
64
letter
L1
U1
1OT 000000000000
U1
5
0113 5
Mm
Nim
01000001
Nim 01
OTS 010T3
KTS KIS KTD
PI 1M
Pi LI DISIO OIT 3 1
PIP
X1
BIT BIT BEST
QR D
D.
U.E 8 0 D
QNQ UJL
DAS D3 BAOE
Ba
PA
RA 91
011000104 3
56
SI
TA
010101
0201010702
R4
0010
Uh0 03
08
DAI
83
5 01
03
VIV
0
WV
D8
WV 0101
3013
2 7M
BM
71v
10
action
7V
action
action
213
-00
The average frequency of each action each letter, averaged lowercase and uppercase images, in the "learning to ask for help" task with perturbed rewards, where reward models are trained on 10% of the digit dataset. (left) 1-of-20 all-images regime, (middle) 1-of-20 single-image regime, (right) Optim(f).
Figure 24:
Shelp
Shelp
9help
010
B6
8b
62
07
630
GX 03
0
165 0000000011001
CK
5
81
0101
3d 0
0000000 Ddd
0
01
513
6113
01000001110
bN
04
0 0
0800
01
01001
0103
0100
B10
10000
NA 0
RA 81 5
83
000000
00000000000000000000000000000000
0
00110000000000000000000000000000
02010
66
letter
L1
LI 54 O
01000010000000000000000000000000
31000
LI
Mmm
B MOET
Ki 0 0
Kim TAL
B33 C03010T
10h
Nm
Nm
1
000000
5
K15 8
PI
Pp
PuP
0101
C USDD
0
Qq 0
D.S 0 63
QHq
D31S
2
03 0-05
Ri
RJ
0
RA
TA
51
TA 0
0 DF
TR
81
02 00503 3
U0
54
081001111
Uu
83
UN
83
88 0313 3
VI
WV 0
0 0
WV 0101
00
6363 4
1000000000
00
D1 B
My
10
0
10000
7/
10
action
7/
action
7/
action
00
The average frequency of each action each letter, averaged lowercase and uppercase images, in the "learning to ask for help" task with perturbed rewards, where reward models are trained on 100% of the digit dataset. (left) 1-of-20 all-images regime, (middle) 1-of-20 single-image regime, (right) Oftim(f).
Figure 25:
30

---

Published in Transactions on Machine Learning Research (10/2025)
B.2 The Driving Gridworld Experiment
For the driving gridworld experiment, the training data for our neural nelworks are generated by enumeraling all of the (state, next state, reward)-tuples in the familiar driving gridworld where obstacles can only appear in either of the two ditch lanes on the far left or far right column. The action is omittod because the familiar driving gridworld's reward function depends only on the imitial and next state. Each driving gridworld state image is pre-processed into a four-channel image where each channel encodes the positions of different aspects of the environment: pavement, ditch, car, and obstacle.
Our networks have two parallel convolutional layers with four output channels and xx 2 filters, each followed by ReLU transformation. The outputs from these layers are flattened and concatenated together, and that result is concatenated with a one-hot encoding of the car's speed in the next state. Next, we apply two fully connected layers separated by a ReLU function, the first with 32 outputs and the second with a single output, respectively. For each possible speed the car could have in the initial state, we manage a different pair of fully connected layers with the same shapes. To compute the expected reward for a given action given stale, we query the neural network with every possible state that could follow from the given action and use the transition probabilities to combine these state- next state reward estimates.
Networks are trained over 51, 200 epochs using a batch size of 800 to minimize the MSE using Adam with a learning rate of 0.0001 and weight decay of 10-5 The remaining parameters for Adam in PyTorch (31, $2, and e) are set to their defaults (0.9, 0.999, 10 10-$) 8) without the AMSGrad (Reddi et al. 2018 modification.
Policics are evaluated by itcrative dynamic programming according to the Bellman operator. An approx- imation of the 7-discounted expected relurn function is updated simultaneously for each state and action until the maximum absolute change is smaller than 10-6, We use discount factor of y 0.99. On every k-of-N CFR iteration, this process is run N times given the current CFR policy to evaluate it under each of the N reward functions sampled at the start of the iteration [
This experiment was run on a 3.60GHz Intel CMreTM 19-9900K CPU with 7.7 GB of memory without a GPU. Model training took roughly 25 minutes per random initialization, so 833 CPU hours for all 2, 000 models. 100 iterations of k-of-N CFR took roughly three minutes, so for all six settings of and N, and all five random repetitions, took about 100 minutes. In total, our experiment used about 835 CPU hours of computation.
8 AAternatively, the successor representation [Dayan esscntially the cost single policy evaluation. Civen this information, computing the expected return given reward function can computed with single pass over each state and action, thereby reducing the amount of computation that scales with N.
1993 could be used to fully characterize the current CFR policy for
31

---

Published in Transactions on Machine Learning Research (10/2025)
C k-of-N CFR for Continuing MDPs
Here restate and prove thcoretical results for CFR and k-of-N CFR in continuing MDPs with reward uncertainly.
Define
d.:8'3m + (1 7) Σ (s =
where A, (-1S(-1) and S. p(-1S--1, A,) for i 2 1, be the 7-discounted future state distribution
induced by from initial state Kakade (2003) 's performance difference lemma for this setting is:
Lemma 1. The full regret for using stationary policy π instead of stationary competitor policy x' from state in MDP (S,A,,, de, 7) under reward function is
(.(:rr)) (T(T)) - [(,s(A where S ~ da(;zt) and A ~ T'(1))).
Theorem 1. CFR bounds cumulative full regret with respect to ony stationary policy as
Σ ()(();")))))) - u S (1-11)
Proof. Let S ~ (z and (-1S). By Lemma 1 the lincarity of expectation, and CFR's definition,
T Σ (
1
E
Σ ps ps(A,Tt:r))
(1)
Y Y
- 1
E|O"1
(2)
0 ((1-))
(3)
for each state s. Since this bound holds for all states simultaneonsly, it holds for the y-discounted expected return as long as the initial state distribution is constant across iterations, i.e., assuming that the initial state is S ~ dæ on each iteration,
- -va(n';"")
Elus(<;r*) Elos(T;r"'-u((() -
(4)
t=1
E
us( r') -s(
(5)
< E|CT /(1 [1-
(6)
/C1((17)),
(7)
which completes the argument.
We make use of the Azuma-Hoeflding incquality in the k-of-N CFR regret bound it is restated here for completeness:
Proposition (Azuma-Hoeffdhng Hoeffding inequality). For constants (cT) martingale difference sequence (Y
where [Y*| et for each t, and r 2 0,
T
-72
P
Σ 1=1
yt
27
<
2exp
2
2
32

---

Published in Transactions on Machine Learning Research (10/2025)
For proof, see that of Theorem 3.14 by McDiarmid (1998).
We construct the following regret bound for k-of-N CFR based on Theorem !
Theorem 2. With probability 1 - p 0, the full regret of k-of-N CFR with respect to any stationary policy, n, is upper bounded by
C 1-7 + 40 AUU/TTlog/
Proof. Let the reward function distribution be R. On each iteration, i, of the k-of-N CPR algorithm, N reward functions, (r) R) are sampled and the worst l reward functions for the algorithm's 's current policy, n², are mixed into 8² : s, + _ - 7(g (s, a, s') where r'g is the w reward function. The k-clement average reward function Ff is sample from R according to the k-of-N probability mcasure, PR-of-N (see Proposition of & Bowling (2012) for formal description of this measure's ' density) where the reward functions are ranked with respect to n².
Let Pk-of-N (") denote the k-of-N reward function distribution with respect to policy x!. The expected return of n' under the k-of-N robustness objective is then just the expectation of its return under f Pk-of-N (1; xt &e., ( (T'; Thus, we seek bound the regret under this payoff function,
Σ Pe (T; x =
Σ Elve (T: [Tu
(8)
Elvo (T: F'] ((
(9)
Elvo (T; (*) - Ue (T': r*)),
(10)
(m:ar)))
where F (+:ot-N(-;TT))) and where the inequality follows from the fact that ranking the reward functions according to nO instead of can only improve r's return. The (a m²) terms in the sum and expectation are the random instantaneous regrets that the k-of-N CFR algorithm computes each iteration.
The rest of the proof largely follows the proof of Parina et al. (2020) 's Proposition 1. Since ((pa(T;)')) is the expectation of pa (T;TT)) the sequence of differences,
(Ejpe(m;m)) (T;; n )
is a martingale difference sequence. Furthermore, Elpo (T;: (1;; twice as large as the largest return and returns are bounded by the largest reward (thanks to normalization).
- Pa (T:T²)) < 40 since regret can only be
The probability that the cumulative regret, Σ Pa r"), is bounded by the cumulative sampled regret plus slack F 2 is bounded according to the Azuma- Hoofna-Hofffingg incquality (Proposition D
T PolT x0) + P
Σ pe T;; x
Σ
E (m; -
(11)
T
1 P
Elpo(T; - - 27
(12)
2T²
$ 1 exp
(13)
4AU
Setting T = 4U 22V ensures that
T
Σ ps (T; n < Σ
pz (T; + 4U 2T 1/p
1=1
33

---

Published in Transactions on Machine Learning Research (10/2025)
with probability p. Since Po Po & (1-(1))))
Σ Po (T: S C* /(1 - 77) + 4U / 2/2llogg 1/p
with probability p, as required.
The k-of-N CFR optimality bound makes use of another elementary result, Markov's inequality: Proposition 2 (Markov's inequality). The probability that non-negalive random variable X is at least is upper bounded by X 's expectation divided by a, i.e., P[X a) E(XI/a.
Proof. By the law of total expectation,
E[X] = E[[[X|]]]] X & al]
(14)
= P[X S [[X X S + x|E[X|X a].
(15)
Since X is non-negative and that F[ X > a] conditions on being no-smaller than a
> P[X 2 <a.0 P[X ala
(16)
= P[X > a/a.
(17)
Dividing both sides by yields the desired statement.
Theorem 3. With probability 1- p, p 0, the best policy in the sequence of policies generated by k-of-N CFR, (n'){=>> is ef -pparroocimation to (k-off--Tron --robust policy where
T= CT
2 2 1/2
(1 4)T 4U
and with probability at least (1 - p1 - 4), > O, a randomly sampled policy from this sequence is an s" /ppapprozimttion to a pk-of-N- policy.
Proof. The first half of the proof is essentially that of Lockhart et al. 2019a 's Lemma
As before, let 7 (+=nf- Define oi = max,* Er v(-:=))()))))))))>))))>))))>>))- [2 (T F)] be the return of P- robust policy, .² Since the competitor term of the regret does depend on the iteration number, we can rewrite the average regret
T
T
2
0z -
(T'3
(18)
1=1
1/
(19)
T
2 1/
E tr #E(#*)E
max
(20)
where the last inequality holds with probability according to Theorem that
Rearranging terms, we see
(= (oo 2 Ug =T.
max *)("!)? T
Thus, the best policy in the sequence, â, achieves the optimal k-of-N value minus s" with probability 1 1-28 p. i.e., A is an e" ap-appooximation
Mt-of-N-robus policy.
The last half of this proof is essentially that of Johanson et al. 2012 's Theorem 4.
34

---

Published in Transactions on Machine Learning Research (10/2025)
Let A ~ Unif({=')[=) )[=1) be the random variable representing uniformly sampled policy. Then,
1 2 0g T
(21)
El 10 E Ua)
(22)
with probability - according to Theorem
Let X U/ xs [+; 2 By Markov's inequality (Proposition
ET ET |E|X|
N E[}]
P
X
S ET)
q
q
Since E" >>> 0,
8 [[[]] S TT
q 92 P
X
(23)
q
8²
x
[[x]
S 5")
(24)
and thus P x < - [[X] S $²)
21 11-9 Finally,
P P SET x ET
ET q
P
X
[[X] S S P[E[X] P[F(X SE
(25)
(1 1) p)),
(26)
proving that an s" 9 (1 (1-p) p)(1 9).
approximation to Pb-of-N robust policy with probability
35


Reviews:
[{"review_id": "nr9yp04gbQ", "reviewer": "Reviewer_1hwv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__:\n\n- The focus on zero-shot cautious behavior in novel environments is an important aspect for AI safety, especially in train-and-deploy settings.\n- The combination of neural network ensembles (to capture uncertainty) and k-of-N CFR (to optimize for risk-sensitive policies) is motivated and well-justified with intuitive reasoning and experiments.\n- The progression of tasks is thoughtfully constructed to show that cautious behavior emerges naturally under reward uncertainty. The use of contextual bandit tasks and sequential decision-making environments like gridworld provides strong qualitative evidence.\n- The extension of k-of-N CFR to continuing MDPs bridges theoretical robustness guarantees with practical deployment settings.\n\n\n__Weaknesses__:\n\n- The current implementation of k-of-N CFR is tabular and uses exact policy evaluation, limiting applicability to small or discretized environments. The paper acknowledges this limitation but does not explore approaches to overcome such constraints, particularly in the experimental setup.\n- The proposed algorithm assumes known and deterministic transitions, which may not hold in many real-world applications. This limits generalizability.\n- This paper does not compare against alternative risk-sensitive or uncertainty-aware baselines (for example, Bayesian RL, POMDPs, distributional RL) in the more complex tasks.\n- While the visualizations and action frequency plots demonstrate the emergence of caution, quantitative evaluation of task performance (for example, safety violations, reward tradeoffs) under uncertainty is limited. In addition, training and querying large ensembles approximating the MDP reward functions has a high computational cost.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of inducing zero-shot cautious behavior in reinforcement learning agents, focusing specifically on the train-and-deploy setting where agents must act safely in novel, previously unseen environments. The authors propose a method that models epistemic uncertainty over the reward function using deep neural network ensembles and then optimizes policies via k-of-N Counterfactual Regret Minimization (CFR) to be robust to the worst k reward scenarios. This combination leads to policies that learn caution without task-specific safety engineering, as demonstrated across a well-designed progression of tasks - from contextual bandits to a grid-world environment. The paper is further strengthened by a theoretical extension of k-of-N CFR to continuing MDPs, including regret and convergence guarantees. Overall, the method is well-motivated, clearly presented, and supported by insightful experimental results and theoretical analysis. However, the current implementation is limited to tabular settings and assumes transition certainty, which restricts scalability and generality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have outlined my concerns in the weaknesses subsection above. It would further strengthen the paper if authors could include quantitative comparisons with some prior baseline algorithms.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DkwU5kg9F5", "reviewer": "Reviewer_eKFH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The problem motivation is clear. Figure 1 and Section 2 vividly illustrate how extrapolative policies collapse when the input distribution shifts. The staged task suite allows the reader to witness caution emerge as the scenarios become progressively more demanding. The overall exposition also shows a clear narrative.\n\n- The optimization algorithm, built on counterfactual regret minimization and robust optimization, rests on a solid theoretical foundation. Section 5 provides regret bounds for continuing MDPs and states the precise assumptions where the authors are concerned with uncertain rewards only, with transitions assumed to be known (no aleatoric uncertainty).\n\n- The ablations on data size show interesting insights. The numerical study on perturbed training data in Section 4 shows that caution surfaces only after the reward ensemble has learned enough to place trust in the “help” action, thereby clarifying how the method’s behaviour hinges on the quality of its epistemic-uncertainty estimates.\n\n\n**Weaknesses**\n\n- The first point concerns studies on hyperparameter sensitivity. Both k and N are manually selected, as are the ground-truth reward functions in the experiments. Although Figure 2 (p. 6) shows reassuring monotonic trends, practitioners would benefit from an adaptive strategy, such as temporarily reducing $k$ when model uncertainty spikes.\n\n- The comparison is limited to a risk-neutral ensemble average. Omitting stronger safe-RL references—such as distributional RL with CVaR targets, CMDP methods like CPO, or uncertainty-aware MPC—makes it harder to gauge the real significance of the gains. Including at least one modern safety-oriented baseline would ground the empirical claims.\n\n- Combining an ensemble with CVaR is not entirely new, and the main advance here is operationalising the idea via k-of-N CFR and visualising emergent caution. A head-to-head evaluation on quantitative risk metrics (such as expected worst-case loss) would better highlight the paper’s distinct contribution. I note, however, that TMLR’s acceptance criteria does not concern novelty, so this point is largely advisory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Learning to Be Cautious, a framework that combines an ensemble of neural‐network reward models to quantify epistemic uncertainty and a robust optimization algorithm based on k‑of‑N Counterfactual Regret Minimisation (CFR). By optimising the worst‑k-of-N sampled reward functions, the agent effectively maximises a smooth CVaR‑like risk measure and therefore behaves conservatively in states that are novel with respect to the training data. The authors present three image-based decision-making tasks that progressively remove “obvious” safe actions, a grid-world driving domain that requires sequential planning, and a proof that k-of-N CFR extends from episodic to continuing MDPs. Across all tasks, more risk‑averse settings (small $k/N$) demonstrably yield safer, slower or “help‑seeking” policies without injecting task‑specific safety rules.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Figure clarity – In Figure 2, the bars overlay individual run markers; consider separating them (e.g., violin plots) so the variance is visually clearer.\n\n- The appendix is a blend of figures and proofs, with one section overlapping with another. The description and discussion of the figures also blurred with the figures themselves. Could you consider separating these two sections more clearly to help the reader locate the main results within each section?\n\n- A brief sensitivity test (e.g., 0.1 vs. 0.5) would indicate whether caution is robust to the choice of +0.25 “help” reward. It also helps if the author can include another sensitivity test on the specific reward setting in the \"Ask for help only when it is available\" paragraph.\n\n- In the broader‑impact section, the authors might also mention potential misuse that an adversary could hide reward features to trick a cautious agent into excessive conservatism.", "broader_impact_concerns": "I am aware of no ethical concerns regarding the paper's results. On the other hand, I suggested to the authors in the requested change section that they further improve the broader impact section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wzUO0gvwA5", "reviewer": "Reviewer_r1oU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think the strengths of this work beat the weaknesses.\n\nStrengths:\n\n- The considered setting, learning to Be Cautious when facing novel situations, is indeed important and practical.\n\n- I appreciate the idea to generalize settings ranging from contextual bandits to environments that depend on contexts.\n\n- The theoretical analyses of extending k-of-N CFR for Continuing MDPs in Sec. 5 are clear and solid.\n\nWeaknesses:\n\n- It is better to add a pseudo-code of the proposed method in the paper to help the reader better understand this work.\n\n- Why does Fig.4 mainly consider N=20, and what about other choices of N in these experiments? \n\n- It is required to discuss more works about safe RL [1-2] and contained RL [3-4] that are closely related to this work.\n\nRef:\n\n[1] Towards Safe Reinforcement Learning via Constraining Conditional Value at Risk\n\n[2] Towards safe reinforcement learning with a safety editor policy\n\n[3] WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning\n\n[4] Constrained variational policy optimization for safe reinforcement learning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work considers how to adapt agents when facing new situations, where the agents need to be cautious. Especially, this work proposes a sequence of tasks in this setting, ranging from contextual bandits to environments that depend on contexts. For all these works, this work shows that the k-of-N counterfactual regret minimization (CFR) with an ensemble of networks can generate policies that adopt cautious behavior.  Extending k-of-N CFR to continuing MDPs, the proposed method is efficient and sound for remaining cautious under uncertainty.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)