A Black Swan Hypothesis: The Role of Human Irrationality in AI Safety
Authors: Hyunin Lee
Chanwoo Park
David Abel
Ming Jin
Venue: ICLR 2025 Poster (2025)
Keywords: AI Safety
Risk
Reinforcement Learning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We hypothesize that Black Swan events (rare but high-risk events) can originate from human misperceptions of an event’s reward and likelihood, even within static environments.

Abstract: Black swan events are statistically rare occurrences that carry extremely high risks. A typical view of defining black swan events is heavily assumed to originate from an unpredictable time-varying environments; however, the community lacks a comprehensive definition of black swan events. To this end, this paper challenges that the standard view is incomplete and claims that high-risk, statistically rare events can also occur in unchanging environments due to human misperception of their value and likelihood, which we call as spatial black swan event. We first carefully categorize black swan events, focusing on spatial black swan events, and mathematically formalize the definition of black swan events. We hope these definitions can pave the way for the development of algorithms to prevent such events by rationally correcting human perception.

Published as conference paper ICLR 2025
A BLACK SWAN HYPOTHESIS: THE ROLE OF HUMAN IR- RATIONALITY IN AI SAFETY
Hyunin Lee Chanwoo Park² David Abel UC Berkeley, PMIT, *Google DeepMind, 'Virgina Tech
Ming Jin
ABSTRACT
Black swan events are statistically rare occurrences that carry extremely high risks. A stan- dard view of black swans assumes that they originate from unpredictable and changing environment: however, the community lacks a comprehensive definition of black swan events. To this end, this paper challenges that the standard view is incomplete and claims that high-risk, statistically rare events can also occur in unchanging environments due to human misperception of events' values and likelihoods, which we refer as S-BLACK SWAN We first carefully categorize black swan events, focusing on S-BLACK SWAN and mathematically formalize the definition of black swan events. We hope these definitions can pave the way for the development of algorithms to prevent such events by rationally correcting limitations perception.
INTRODUCTION
To successfully deploy machine learning (ML) systems open-ended environments, these systems must exhibit robustness against rare and high-risk events, often referred to as black swans [Taleb] 2010). Achiev- ing this robustmess requires a deep and precise understanding of the origins of such events, which has been increasingly recognized critical factor for enabling ML algorithms to attain full control and make op- timal decisions (Chollet 2019 Silva & Najafirad, 2020 He et al., 2021 Li et al. 2023 Yang et al. 2024). Nevertheless, many contemporary MI. systems remain vulnerable to black swans real-world scenarios,
evidenced by automated trading systems that overreact to market anomalies (Kirilenko et al.) 2017 2017: Phillips
2021 Stafford 2022). unexpected bankruptcies (Wiggins al. 2014 Akhtaruzzaman
(023), the
Covid pandemic (Antipova, 2020), and autonomous vehicles encountering unforeseen road or weather con-
ditions (Tesla 202 Witman al. 2023 Nordhoff al. 2023).
In this paper, we argue that ML systems remain susceptible to black swan events, regardless of an algorithm's 's representation capacity or scalability. due to an AI community's 's incomplete understanding of the origins of these events. The prevailing belief in most algorithmic approaches to preventing black swan events (Prest- al.] [202T] Wabartha et al. [002]] Bhanja & Das 2024, Jin (2224) is that such events primarily arise from dynamic, time-varying environments. We contend, how- ever, that black swans can also emerge from static, stationary environments. To this end, we propose a new hypothesis their origins:
wich 20199] Artemenko al.) [2200] Devarajan
Hypothesis 1. Black swans can originate from misperceptions of an event's 's reward and likelihood, even within static environments.
"Corresponding author: hyunin @berkeley.ddu

---

Published as a conference paper at ICLR 2025
To warmly introduce our new hypothesis, consider the bankruptcy of Lehman Brothers, widely recognized as the most significant black swan event in the financial industry (Wiggins et al.) 2014). A strong explanation points to the investors making rational decisions on the false market perception which appeared rational at the time but proved irrational by correcting their perception in hindsight. The firm declared bankruptcy within 72 hours without any precursor (McDonald Robinson 2009), and the only factor that changed during those three days was investors' perception of the company (Housel) 2023; [Mawutor 2014] Fleming & Sarkar 2014 [ Investors made optimal decisions based on this perception, which turned out to be suboptimal once the perception evidence of our hypothesis.).
revealed to be false during those 72 hours (See Appendix B for additional supporting
Contribution. We refer to black swan events in stationary environments as S-BLACK SWAN and define them in the context of a Markov Decision Process (MDP) as follows:
(Informal) An S-BLACK SWAN event a state-action pair where humans misperceive both its like- lihood and reward. It is perceived as impossible, despite occurring with small probability, while its reward is overestimated relative its true value in a stationary environment.
Our work begins with a case study on how S-BLACC SWAN emerge and cause suboptimality gaps in various MDP settings, such bandit (Theorem I small state spaces (Theorem 2 and large state spaces (Theorem Bb. We introduced three MDPs to define S- S-BLACK SWAN: the ground truth MDP (GMDP), the Human MDP (HMDP), and the Human-Estimation MDP (HEMDP). The GMDP represents the real world, while the S-BLACK SWAN (Definitions 4 and (5) are state-action pairs perceived as impossible the HMDP but occur with small probability and higher rewards in the GMDP. Our main finding (Theorem 4 shows that while the HEMDP value function asymptotically converges to that of the HMDP over longer horizons, the gap between HMDP and GMDP has a lower bound, influenced by reward distortion, the size of the S-BLACK SWAN set, and their minimum probability of occurrence. Finally, 3xrcoremm)xxamingg S-BLACK SWAN hitting time, showing that larger reward distortion and higher S-BLACK SWAN probability necessitate more frequent updates to human perception functions.
HMDP reflects humans' biased perceptions (Definitions I and 2
2 PRELIMINARY
Notations. The sets of natural, real, nonnegative, and nonpositive real numbers are denoted by N, R, Rsu- and Rgo respectively. For finite set Z, the notation |Z] represents its cardinality, and A(Z) denotes the probability simplex on Z. Given X, Y E N with X < Y, we define [X] := (1,2,..., X), the closed interval [X,Y]= = (X, + +1. I...., J. For x E Rzo, the floor function [x] is defined as max(n (n e N u {0} n S x)
Markoy Decision Process. We consider finite-horizon MDP denoted as M (S. P (P.)To and R [Ro]T- for E N. Here, S represents the state space, A denotes the action space, Pr: S x A A(S) is the transition probability function at time t, RG S A + IR is the reward function at time i, 7 is the discount factor, and E N the horizon length. We define M as a stationary MDP Re-1(s, a) for all (s',
where
(s s,a) and (s, te [T - 1]. Otherwise, we define M as a non stationary MDP. In the stationary case, we denote P and R the single transition probability function and reward function, respectively. A policy is denoted as e II, where IT : S + (AA) is the set of policies. We denote a T-length trajectory from A under policy л as 8T-1, aT- 8T), where 8,- P.( 9--1,0+1) and rt R& (st, at). Assume that all rewards are bounded, i.c., Ti & [- Rmaxx Rmax] for all t. The agent's 's goal is to compute the optimal
if Ps(s' s,
S S x A and for all
[T'-1]
(so, 0o, To, 81,
'The bank's loss endurance, evaluated at 11 11.7% by the U.S. government, stayed stationary over the 72 hours. FFor clarity and readability, all notations used throughout the entire paper are elaborated in Appendix A
2

---

Published as a conference paper at ICLR 2025
policy nt' 1I that maximizes the value function: (Z( Ex 2 Σ
s]. We further define the normalized visitation probability as (T. == Σ y'P((s,, (s.a))so, n, P), where
P(s, also: n, P) is the probability of visiting (8,4) at time under policy π and transition probability starting from 80
The following three theorems, drawn from existing work, lay the groundwork for mathematically formulat- ing misperception of the Hypothesis
Expected Utility Theory. Given an outcome space O= = .o...... oK} J. we define a utility function g O + TR that quantifies the gain or loss associated with each outcome oz. An individual agent is faced with choices, where each choice represents a scenario in which the outcomes o; occur with given probabilities, ps, summing one. The set of all choices is denoted by C. Each choice e e returns O with a probability distribution Pe
Under a given choice Expected Utility Theory (EUT) evaluates the riskiness of that choice (v) .(x)) )
as ((c) Σ p(o))p (e) (von Neumann 1944, Rabin 2013). To illustrate, consider a stock market invest- ment scenario where O [Economic Boom (EB), Economic Recession (ER)). Here, g (EB) represents a gain, while (R) represents a loss. The set of choices C = (invest in stocks, invest in bonds, keep cash)
corresponds to different probability distributions P. (p19) P2
(c) (c) of outcomes.
Prospect Theory. However, Expected Utility Theory (EUT) fails to account for empirical observations from psychological experiments (Drakopoulos & Theodossiou, 2016. Pandit et al.] 2019: Wahlberg & Sjoberg 2000: Vasterman et al. [2005] van der Meer et al. 2022) and economic cases (Rogers 1998 Wheeler & Wheeler | Bptttrrpp] (022) that demonstrate human irrationality. Specifically, humans tend to ex- hibit internal distortions when perceiving event probabilities P. and evaluating outcome values (O) for any choice e (Opaluch & Segerson| 1989). To address these discrepancies, Prospect Theory (PT) introduces a probability distortion function w: [0,
[0, 1] and value distortion function u. R - R. which modify
the expected utility calculation to (c) Σx= (p)()) (Kahneman & Tversky 2013 Fennema & Wakker [997). The motivation for introducing PT is not only to acknowledge human irrationality but also to provide a more accurate mathematical framework for how people actually perceive probabilities and outcomes. PT describes the characteristics of the functions and w based on empirical case studies. The function represents value distortion, capturing how individuals assess gains and losses (x-axis of Figure la represents the true value, and the y-axis represents the perceived value). The function w represents probabil- ity distortion, reflecting how individuals tend to overestimate the likelihood of rare events and underestimate the likelihood more probable events. (I-axis of Figure Tb represents the true probability, and the y-axis represents the perceived probability.)
Cumulative Prospect Theory. To enhance mathematical rigor- specifficllll to ensure that distorted prob- abilities still sum to one Prospect Theory (PT) was further revised into Cumulative Prospect Theory (CPT). In CPT, the expected value is defined as V(c) Σx u(g(o.)) (w (E)=);"") - (Σ (()) where the function w distorts the cumulative probability of event or. The following insurance example illustrates CPT in action.
Example 1 (Insurance policies). Consider an example where the probability of an insured risk is 1,, the potential loss is 1, 000, and the insurance premium is 15. According to CPT, most would opt to pay the 15 premium to avoid the larger loss.
Example shows how a simple decision can be modeled as a two-step Markov Decision Process with states S { (base, Spremium, Srisk) } representing utility value of 0, -15, and 10000 and actions (or choice set C) A = (ap, anp) for paying or not paying the premium. At = O. humans choose between ap (leading to Spremium) and Oap- which could result in Stase with 99% probability Srisk with 1 % probability. Expected
3

---

Published as a conference paper ICLR 2025
M
MT
(a) Value distortion
(b) Probability distortion (c) with black swans
with black swans
Figure 1: Value distortion function and probability distortion function The gray line in Figures la and
1b
represents y x.
utility theory suggests app is optimal since its expected value (V(anp) 1 0.01 = -10) is lower than that of ap (Vpap) = 15 1 15), but real-worid decisions often favor ap, highlighting a divergence from theoretical rationality.
Therefore, begin by formalizing the key empirical observations from CPT into the following definitions. Definition (Value Distortion Function). The value distortion function u is defined as:
u(x) S (x) if O, (z)z)
<f
where x" Rg Ra is non-decreasing, concave with lim (u" Y(h) s 1, and u Rggo Reso non-decreasing, convex with lima-o (1))())> 1.
Definition (Probability Distortion Function). The probability distortion functtooo w defined as:
w(p:) = (w:) if gf;(;; 2,
w (p:) if if < O,
where w², w [00 [0.1] satisfy: wt (0) w (0) wt (1) w (1) 1; w² (t) a and w (b) for some a. b E (0, 1): (w+) ((z) (z) is decreasing on [0, a) and increasing on (a. 1]: (w (x)) is increasing on [0, and decreasing on (b, 1].
The derivative constraints encapsulate the core observations of CPT. Specifically, the conditions on (u)) and (ut) y in Definition I formalize the tendency for individuals value losses more heavily than equivalent gains (see Figure Lal. The constraints on (w and (w*) in Definition the tendency to overweight (or underweight) the prohabilities of rare events and underweight (or overweight) those of average events where the outcome results in gain (or a loss) (see Figure 1b
3 BLACK SWAN IN STATIONARY AND NON-STATIONARY ENVIRONMENTS
Hypothesis concerns the feasibility of black swan events existing in stationary environments. We next illustrate how black swans originate from both stationary and non-stationary environments. We begin by defining the black swan event dimension as follows.
Definition 3 (Black Swan Event Dimension). For given MDP M, define the dimension of a black swan event as the set A x [T].
Then, informally refer (s, tbs) E S x Ax []] as black swan event if it represents rare, high- risk occurrence that significantly deviates from expected outcomes based on prior experience in the real world M. This could involve unexpected transition a classification rule that distinguishes black swan events based on whether they occur non-stationary environments arise within stationary environments, follows.
anomalous reward signal. We then introduce
4

---

Published as a conference paper ICLR 2025
Algorithm (Black Swan Classification: S-BLACK SWAN ). For given (possibly non-stationary) M, suppose (s, a, thx) is black swan event. If (s, a,t) is black swan event for Vt E [T], then we classify (s,a,ts)) as a black swan that originates from environment's 's stationarity (S-BLACK SWAN ).
Based on Algorithm D one can always identify a unit time interval that classifies any black swan event as an S-BLACK SWAN, as stated in the following proposition.
Proposition 1. If (s, a, lbe) is a black swan event, then there exists a time interval [41,42] [T) such that for every t & fi,tt the (s, ss,,t) t) is classified as S-BLACK SWAN
We provide an intuitive interpretation of Proposition | through the following example.
Example 2. Suppose (s, a, las) is a black swan event.
Case 1. Consider M as a non-stationary MDP where Pt and Rs change at each time step, ie.. Pt + Pi-1 and Re Ret1- If t1 12 tbss then (s, a, ths) is classified However, if b1 + l2 and lha [2, then (8, O, lis) cannot be definitively classified as an S-BLACK SWAN.
S-BLACK SWAN
Case 2. Consider M as piecewise non-stationary MDP where Pt and Re change every [T time steps, i.e., Pt- Pi+1 and R₂ = RRy+ for é [kj.kj (I- (k-1)) 1)] where j 0, 0,I,.... [T/kk. If t = kijs and l2 = kibss + (k- 1), then (8, a, lbs) is classified Jos satisfies lbs [kjbs, kjbs + (k-1)),
an S-BLACK SWAN where
Case 3. Consider M as stationary MDP where P-= Pt=Pur P+1 and R, = Re+1 for all t [T 1]. In this case, (8, a, tbs) is always classified as an S-BLACK SWAN, regardless of the interval [21.
We then present Case of Example as the following main remark:
Remark 1. If M is stationary, then any black swan event (s. a, t) is classified as an S-BLACK SWAN In this case, we omit and denote the s- BLACK SWAN simply as (s,a).
Our main goal for the remainder of the paper is to explore Remark [ with a focus on mathematically defining S-BLACK SWAN within stationary MDP M. We will retain the notation for stationary transition probabilities and reward functions as P and R, respectively, omitting the subscript
4 THE EMERGENCE OF S-BLACK SWAN IN SEQUENTIAL DECISION MAKING
We next present case study to substantiate Hypothesis before formally defining S-BLACK SWAN We begin by examining how S- S-BLACK SWANS emerge in sequential decision within a stationary envi- ronment, starting with the bandit case. For given (s E S A, let assume that the function distorts the reward R(s, a). and the function w distorts the transition probabilities {P(s'Is, [s, a))veess where s' is the next state. In this Section, refer the MDP distorted by functions and as the distorted MDP Ma := (S,A,
A, w P), u ()),)), with this notation being used exclusively within this section.
4.1 CASE 1. CONTEXTUAL BANDIT (T (T=1)
We begin with simple case where the horizon length is T 1, commonly referred to as contextual bandit (Lattimore & Szepesvári] 2020). Surprisingly, in this setting, the optimal policy of a distorted world coincides with the real world optimal policy as a following Theorem.
Theorem (One-Ster Step Optimality Deviation), IfT = 1, then the optimal policy in the MDP M identical the optimal policy in the distorted MDP Ma-
Theorem may seem counterintuitive, Example T illustraates that human decision-making often exhibits irrationality. In single-step decision-making. distortions in perception do not significantly affect the opti- mal policy. For clarification, as shown in Example the perceived reward order remains u" (r(Stoas))
5

---

Published as a conference paper ICLR 2025
(r(Spremium) )) uu (r (r((base)) because u is non-decreasing convex function. This further implies that a short decision horizon may reduce the influence of human irrationality.
4.2 CASE 2. |S| = 2 WHEN T> 1
Now, let us consider the simplest case where T and [S] = Surprisingly, the result that optimality does not deviate still holds similarly to Theorem 1
Theorem 2 (Multi-step Optimality Deviation with [] 2). If|S] =2, then the optimal policy from the MDP M is also identical to the optimal policy of the distorted MDP fa all t [T].
Theorem 2 may initially seem counterintuitive, given that model errors propagate through distorted transition probabilities and rewards as time progresses (Janner et al. (019). However, a straightforward explanation is that for any state-action pair (s, a) x,A, the function w preserves the order of probabilities. Specifically, if P(si|s, a) p( a), then w (1ss a)) > w (2sss) a)) still holds, where S = {$1, $2). This suggests that when the state space |S] is small, the informational complexity required to determine the real-world optimal action remains relatively low.
4.3 CASE 3. |S| = 3 WITH UNBIASED REWARD PERCEPTION
We now consider a general setting with arbitrary S, A, and T, but under the assumption that u ((R(s,a) R(s, a) for all (s, a). indicating that humans have an unbiased perception of their rewards.
Theorem (Two-step Optimality Deviation with |S| 3). If and T there exists a transition probability function P and a reward functionn R such that the optimal policy of the MDP M from that of the distorted MDP Ma-
The optimality deviation in Theorem now aligns with the empirical observation in model-based rein- forcement learning; increasing suboptimality is caused by model error propagation (Janner et al.) 2019). In summary. Theorems [ μ and 3 demonstrate that the discrepancy between the optimal policy derived from human perception and the real-world optimal policy increases as the complexity of the environment (S) grows or as the horizon length (T) extends, regardless of the w function.
5 AGENT- ENVIRONMENT FRAMEWORK PEEREPTION AS INTERSECTION
To explore Hypothesis [ we propose a novel agent-cnvironment framework that treats misperception as information loss in an agent's 's understanding of the real world (See Figure 2 This framework introduces two stationary MDPs: the Human MDP and the ary ground MDP (GMDP) M as abstraction of real-world environments without information loss. The following subsections detail the Human MDP (HMDP) and the -stimattttitation MDP (HEMDP).
MDP. We begin by defining the station-
5.1 HUMAN MDP
We define the Human MDP M (S, T),, where the human (agent) misperceives the visita- tion probability (" a) through the function denoted as pt. (s, a), and the reward function R(s, a) through the function denoted as R (s, a). An internal assumption in the HMDP that its state and action spaces are identical to those of the GMDP M, i.e., S S and AAT = A. Although this assumption may seem unrealistic, especially given that insufficient exploration in large discrete state and action spaces may violate it, the following method shows how the human (agent) can approximate S and A to S and A, thus supporting this assumption.
We detail how misperception reflects information loss from the agent's perspective in Appendix C
6

---

Published as a conference paper ICLR 2025
Remark 2. If the human (agent) cannot perceive a state s E S, the state space S can be updated to S - (8), then set R' (s.a) = R(s,a) and (' I s. ) P(s'1 | s,a) while ensuring P(s s'.a) for all s S' and a A'. As a result, the new state does not influence decision-makin in the HMDP, since the probability of the trajectory visiting remains zero.
For discrete S and A, the order statistics of p can be defined over the sequence [[S][[[]] with each (s, a) corresponding to an order index enablig For brevity, we denote the cumulative distribution of p" (x.a) f P" (s,a) The distortions are then defined by the following relationships:
enabling the subsequent definition of the cumulative distribution.
f Pt (s,a)
af wt (' Pd (s
w (f (f <0 ,V(5,a)ESxA
(1)
u (H((s))) R(s,)) 20 R(s.a) <0
(s,
((ss,))SxxA
(2)
10 (R(s,a)) if
We introduce the concept of the perception gap: if (ax)) [R(s, R (s.a) < cr, then (i.s a) is referred as an ey-perceived reward. Similarly, if max(s.o) |PM (s.a) pm (s, a)l < cd, then pte (s,a) is called an ea-perceived visitation probability, where e,: E RR. The case where
= Ed = 0 represents an unbiased perception. Once the agent perceives M as M., it executes the policy и in M' and collects a trajectory. Finally, the value function of M is given by V (+) == E. [T' R² (se, az Pt so s].
key challenge in understanding MM is why distortions occur in visitation probability rather than transition probability, discussed in Section 5 This distinction arises because (s, a) is the fundamental event unit (see Remark [], and a distortion in transition probability implies a distortion in the state itself. The central question, then, is how distortions in visitation probability relate directly to data collection. The following lemma partially addresses this question.
Lemma 1. For given M, there always exists J Pa (P((s)) a) holds for any function w.
a function h : S - S such that w (ʃ P" (s,a))
Our perspective is that distortions in the probability distribution, state space, other factors lead distortions in visitation probabilities. With unbiased perception, the agent collects a trajectory 7 However, when the agent perceives M as M observes per- where function h distorts the states. Lemma demonstrates that visitation probability distortion arises from state distortion via h.
{so, 00, To, $1, $7-1, ceived trajectory 7 (h(so), ao, ((so)),(ss)) .a....,,........................ ,),...,,((TTI)) (r,1,,((T)]
5.2 HUMAN-ESTIMATION MDP
After the agent have perceived the ground truth world as M, it estimates the perceived reward R' (s.a) a) as R' (s, a) and visitation probability p (s, as Pi. (s,a) a) from its perceived trajectory -!. We define a Cuiman-Estimation MDP as M (S,A, Pi F r, Note that this estimation process is the same estimation of generative model in -bael- reinforcement learning (Gheshlaghi Azar et al.
Environment
2013 Sidford et al. 2018 Agarwal et al. 2020 Kakade 2003). We also introduce estimation gap, that if max(s,a) (s, R' (s,a)) Kr
M
perception
M
estimation
M
holds, then R' (s, a) is k, esttmmttee reward, and if max (PT (s,aa) (stt(s,a) Ka holds, then P (s, is ra-esmmmated visitation proba- bility for constant Kr, Kid IR.,. Finally, the value function of M is given VE M (s) = E, [>* Rss,s a.) Pt,
tried
My.
Agent
Figure 2: The agent and environ- ment intersect with perception.
7

---

Published as a conference paper ICLR 2025
We use the perception and estimation gaps illustrate the novel agent- environment framework in Figure 2
6 S-BLACK SWAN
Finally, Section 6 provides a definition of S-BLACK SWAN and presents theoretical analysis aimed guiding the design of safer ML algorithms in the future.
6.1 A DEFINITION OF S-BLACK SWAN
Assume that the rewards for state-action pairs are ordered as Rl $ ... E Rug & 0 < R & & R[SsLA]]. and the visitation probabilities are ordered as [1] & [2] 1 .. P [[] We denote the order index of R(s, a) (,(sa)) [[S][A[]] and the order index of p" (s.a) as (2(s,)) [[S][[[] such that
(s.a) R(s, a) and Pit [1,(x.a)] P" as, We first provide the definition of S-BLACK SWAN in case of discrete state and action space.
Definition 4 (S-BLACC SWAN Discrete State and Action Space), Given distortion functions u, w and constants Cbs >> O and 6bs > 0, (s) satisfies:
1. risgh-risk): R[I,(s.a)] u (R(,((,a))) < -Ss--
P 2. (Rare): w (E// PTi) = w EEj= E/2(sa)=1 pT1). yet Up(s,a)] Cbs-
then we define (8 as S-BLACK SWAN
Definition 4 finally formalizes the informal concept of black events introduced in Section The D first property Definition identifies a high-risk event through value distortion. Specifically, if the agent perceives R optimistically, such that R u (R) 0, classified as a high-risk event (see Figure Ic.. The second property characterizes rare event through probability distortion, describing an S-BLACK SWAN event that occurs with a small probability in the real world (o < PT [1,(s.a)]
Ebs but is perceived by the agent as infeasible (w- ()/((00) (G)) w (M/=(s0)) (G) ) (See Figure Id).
The constants C.. and cbs in Definition 4 quantify the extent of distortion in the functions u and w, spectively. Intuitively, Cos and Ebs are directly related to the magnitude of the misperception gap between M and M denoted by e, and Ep. This relationship will be further formalized in Theorem We now ex- tend the definition of S-BLACK SWAN to continuous state and action spaces. Suppose the reward function R - R is bijective. Then, the probability R- p R + [0. 1] denotes the probability of a feasible reward induced by policy т, denoted as P.. We then have the following definition.
Definition (S-BLACK SWAN Continuous State and Action Space). Given distortion functions u, w and constants Cbe >> O and cbs > 0, if (s,a) satisfies:
1. R(s.)) u (R(s. a)) < <-Obe-
2. du (x) de =F((((),))) Pr(r R(s, = yer 0 R(s,a)) < Cbs.
where F(r): == f- dP,, is the cumulative distribution of Pr. then we define (s, as S-BLACK SWAN
min := min(s.a) Pr(rr We then define the minimum probability of S-BLACK SWAN as ebs min denoted as tbs
R(s, a)). Let B denote the collection of all S-BLACK SWAN For given constants Cb. and Ebs, we define the distortion functions w and u that result in B u, represent safe perception, meaning that if an agent perceives the world through those, then B Ø. However, it is important to note that wr and u, are not unique functions (see Figure
w,, and u,, respectively. Intuitively, w. and
8

---

Published as a conference paper ICLR 2025
6.2 THEORETICAL ANALYSIS OF S-BLACK SWAN
Subsection 6.2
explores the properties of S-BLACK SWAN focusing how their presence establishes a lower bound on policy performance (Theorem 4 and the timing of their occurrences (Theorem [}. laying the groundwork for future algorithm design. For further analysis, assume the following.
Assumption (Relative convexity). Assume u. (r) u (r) holds for r < O.
Assumption 1 ensures that a human (agent) with v perceives rewards more optimistically than one with v.
across all (8,a) pairs. This concept is well illustrated in Figure Tcl where the function u (r) r represents an unbiased perception, and deviations from this line indicate increasing reward distortion. In conjunction with Assumption
! we introduce proposition regarding S-BLACK SWAN enabling interpretation within the reward space | Rmaxx R.
Proposition 2 (S-BLACK SWAN ). Let the intersection of the functions r + Chs and ui" (r) occur at -Ross (see Figure| lc). Under Assumption 7 rf) Rmax, - Ros] satisfies:
1. ur (r) -Obs- 2. w (F(r) = O, with 0 < F(r) Eha,
then the (s,a) is S-BLACK SWAN
key insight from Proposition 2 is that as u (r) approaches u., (r). the approximation -Ros -Rmas occurs, finally leading to [B] 0 since ||- [l-Rmax: RRs O (see Figures Ic]. In other words, Proposition 2 demonstrates that reducing the perception gap directly correlates with a decrease in [B].
Now, to provide an guideline for designing safe learning algorithms to prevent S-BLACK SWAN it crucial to quantify how the existence S-BLACK SWAN leads to an inevitable deviation from the real-world optimal policy. We address this by analyzing how the misperception gap establishes lower bound on the value function gap between the HMDP M and the GMDP M, as presented in the following theorem.
Theorem 4 (Convergence of value estimation gap but lower bound value perception gap). Under As- sumption the asymptotic convergence of the value function estimation holds as follows.
V (s) (8) a.s.
T co, Vs, ग S II.
However, under specific conditions on Ebs: Eb emnn Ros. the lower bound of value perception gap as follows.
- VV ((Rmax Ros) enin RogEbs) (Rmax Rio Cbs
(4)
R2
There are two key consequences of Theorem 4 First, Equation 3 demonstrates that the value estimation еπrо converges to zero as the agent rolls out longer trajectories. However, Equation (4) reveals that the value perception gap has a non-zero lower bound, regardless of the horizon length. Equation (4) further indicates that if u (x) u, (x) and w (z) + w. (z), then Rs, + Rmax and Eb.- + 0 (see Figures and [d]. leading the convergence of this lower bound zero. Second, Equation 44) aligns with the intuition that greater distortion in reward perception (i.e., larger Cha) and an increased number of S-BLACK SWAN (i.e., larger min (Rmax- Ro.)) coupled with a higher minimum probability of S-BLACK SWAN occurrence (i.e., larger Ehs result in a higher lower bound. Therefore, Theorem concludes that even with zero estimation error, a lower bound on approximating the true value function remains, and this lower bound increases as Co. and min become more pronounced.
Then, the next natural question how to decrease that lower bound, specifically, how can an agent can learn self-correct toward safe perception, i.e., u u, and w" w._ This question can be further refined to: What is the probability of encountering S-BLACK SWAN if the agent takes t steps? We address this under the assumption of non-zen non-zero one-step reachability, as follows.
9

---

Published as a conference paper ICLR 2025
Theorem (S-BLACK SWAN hitting time). Assume Pg- (5' 5) > 0 for any s, s S. indicating that the
one-step state reachability equipped with optimal policy is non-zero, and consider that one step corresponds unit time. Then, if the agent takes t steps such that 2 log
/log(1 Pmax) 1, where Pmin Rmax- Roe Ebs it will encounter S-BLACK SWAN with at least probability 5 e (0, 1].
Panin
Rmax-Roo Ros min and Pmax 2R2as bs
2Hmss
A key takeaway of Theorem 5 is determining how often human should correct their internal perception.
large perception gap (Rmax - Ros) and frequent occurrence of black swan events (chg require more frequent execution of the self-perception correction algorithm.
7 RELATED WORKS: NECESSITY OF S-BLACK SWAN
This section discusses safe reinforcement learning (RI.) algorithms, emphasizing the limitations of existing approaches in addressing black swan events and highlighting the need for new perspective"
Safe RL algorithms are generally classified into three approaches: worst-case criterion, risk-sensitive cri- terion, and constrained criterion (Garcıa & Fernández] 2015). However, these approaches face significant limitations when dealing with black swan events. The worst-case criterion, which optimizes policy perfor- mance under the least favorable scenarios by maximizing the minimum return, becomes overly conservative when black swan events are considered, as they expand the uncertainty set W. leading to impractical de- cisions such as avoiding all risky activities or adopting extreme safety measures (Heger 1994, Coraluppi
1997 Coraluppi & Marcus 1999, 2000). Similarly, risk-sensitive algorithms, which incorporate a sensitivity factor to balance return maximization and risk management (Howard Matheson
1972; Chung Sobel, 1987 Patek 2001), are inadequate for handling black swan events because return variance, a commonly used
risk measure, fails to account for the fat tails in distributions (Huisman et al. 1998; Bradley & Taqqu 2003 Bubeck et al. 2013) Agrawal et al. 202T). Additionally, log-exponential utility functions, often associated with robust MDPs, do not effectively address the risks posed by black swans (Osogami 2012 Moldovan &
Abbeel 2012 Leqi et al. 2019). The constrained criterion, which maximizes expected returns while meeting
multiple utility constraints such as return variance or minimum thresholds (Geibell 2006 Delage & Mannor Ponda al., 2013 Di Castro al. 2012), also faces challenges with black swan events. These events complicate threshold selection, often necessitating more conservative policies, and suggest that constraints should be redefined focus on state and action-specific risks rather than overall returns (Bagnell al.,
2010
2001 Iyengar| (2005) Nillim & El Ghaoui 2005 Wiesemann et al.) 2013; Xu & Mannor 2010). Furthermore, distributional RL vulnerable to black swans, as extreme outliers the reward distribution slow the con- vergence of the Bellman operator and provide a large suboptimality gap due to biased return expectations
(Bellemare al. 2017).
In summary, traditional risk criteria in RL are insufficient for managing the unique risks associated with black swan events, highlighting the need for novel approaches.
8 CONCLUSION
In conclusion, this paper redefines black swan events by introducing S-BLACK SWAN highlighting that such high-risk, rare events can occur even in unchanging environments due to human misperception. We categorized and mathematically formalized these events, aiming guide the development of algorithms that correct human perception to prevent such occurrences. This work opens the door for future research to enhance prcision-making systems and reduce the impact of black swan events.
*Further details in Appendix D along with discussion of CPT's application in risk analysis in Appendix
10

---

Published as a conference paper ICLR 2025
ACKNOWLEDGMENTS
We would like to express our heartfelt gratitude to Jason Jangho Choi for his thoughtful and instructive insights on the early draft of this paper. Special thanks to Donghao Ying for the enriching discussions on mathematical notation. Our sincere appreciation goes to Theophane Weber, Csaba Szepesvari, and the rest of Google DeepMind team for their careful review of the first draft. We are also grateful to John D Martin from the RLC 2024 conference for the engaging discussions on algorithm design to prevent black swan events. We would also like to thank Negar Mehr, Mark Muller, Panayiotis Papadopoulos, and all the participants of the Berkeley control seminar for their invaluable and insightful contributions during our discussions of the second draft (ICLR submitted paper). Lastly, we extend our warmest thanks the BAIR researchers at UC Berkeley, including Suhong Moon for the kind invitation to speak in John Canny's 's lab, and Seohong Park for providing such thorough and constructive comments on the second draft.
REFERENCES
Alekh Agarwal, Sham Kakade, and Lin F Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory, pp. 67-83. PMLR, 2020.
Shubhada Agrawal, Sandeep K Juneja, and Wouter M Koolen, Regret minimization in heavy-tailed bandits. In Conference on Learning Theory, pp. 26-62. PMLR, 2021.
Md Akhtaruzzaman, Sabri Boubaker, and John W Goodell. Did the collapse of silicon valley bank catalyze financial contagion? Finance Research Letters, 56: 104022, 2023.
C Ambhika, S Gayathri, TP Anish, B Gracelin Sheena, et al. Enhancing predictive modeling in high di- mensional data using hybrid feature selection. In 2024 5th International Conference on Electronics and Sustainable Communication Systems (ICESC), pp. 873-879. IEEE, 2024.
Tatiana Antipova. Coronavirus pandemic as black swan event. In International conference on integrated science, pp. 356-366. Springer, 2020.
Michail Artemenko, Vladimir Budanov, and Nicoly Korenevskiy. Self-organizing algorithm for pilot mod- cling the reaction of society to the phenomenon of the black swan. In 2020 IEEE 14th International Con- ference on Application of Information and Communication Technologies (AICT), pp. 1-7. IEEF, 2020.
J Andrew Bagnell, Andrew Y Ng, and Jeff G Schneider. Solving uncertain markov decision processes, 2001.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pp. 459.458. PMLR, 2017.
BetterUp.
The availability heuristic.
https: www wttp://etteeuu.combbb..b.c 2022. Accessed: 2024-05-12.
See-availability-huue
Samit Bhanja and Abhishek Das. A black swan event-based hybrid model for indian stock markets' trends prediction. Innovations in Systems and Software Engineering, 20(2): 2024.
Michael Bowling, John D Martin, David Abel, and Will Dabney. Settling the reward hypothesis. In Interna- tional Conference on Machine Learning, pp. 1000-30200 PMLR, 2023.
Brendan O Bradley and Murad S Taqqu. Financial risk and heavy tails. In Handbook of heavy tailed distributions in finance, pp. 103. Elsevier, 2003.
Douglas Brinkley. American moonshot: John F. Kennedy and the great space race. HarperCollins, 2019.
11

---

Published as a conference paper ICLR 2025
Sébastien Bubeck, Nicolo Cesa-Bianchi, and Gábor Lugosi. Bandits with heavy tail. IEEE Transactions on Information Theory, 59(11):711-7771, 2013.
Jonathan Colaço Carr, Prakash Panangaden, and Doina Precup. Conditions preference relations that guarantee the existence of optimal policies. In International Conference on Artificial Intelligence and Statistics, pp. 3916-3924. PMLR, 2024.
François Chollet. On the measure of intelligence. arXiv preprint 1911:::99110014,,,,
Kun-Jen Chung and Matthew J. Sobel. Discounted mdp's: s:: distribution functions and exponential utility maximization. Siam Journal on Control and Optimization, 25:49-62, 1987. URL httpss//api.
Jonathan B Clark. Overview of balloon flights and their biomedical impact on human spaceflight. In
Handbook of Bioastronautics, pp. 839-856. Springer, 2021.
Stefano P Coraluppi and Steven I Marcus. Risk-sensitive and minimax control of discrete-time, finite-state markov decision processes. Automatica, 35(2):301-309 1999.
Stefano P Coraluppi and Steven I Marcus. Mixed risk-neutral/mininax control of discrete-time, finite-state markov decision processes. IEEE Transactions on Automatic Control, 45(3):528-532, 2000.
Stefano Paolo Coraluppi. Optimal control of Markov decision processes for performance and robustness. University of Maryland, College Park, 1997.
Dominic Danis, Parker Parmacek, David Dunajsky, and Bhaskar Ramasubramanian. Multi-agent reinforce- ment learning with prospect theory. 2023 Proceedings of the Conference on Control and its Applica- tions (CT), pp. 9-16. SIAM, 2023.
Erick Delage and Shie Mannor. Percentile optimization for markov decision processes with parameter uncertainty. Operations research, 58(1):203-213, 2010.
Jinil Persis Devarajan, Arunmozhi Manimuthu, and V Raja Sreedharan. Healthcare operations and black swan event for covid-19 19 pandemic: A predictive analytics. IEEE Transactions on Engineering Manage- ment, (000)3229-32233 2021.
Dotan Di Castro, Aviv Tamar, and Shie Mannor. Policy gradients with variance related risk criteria. arXiv preprint arXiv:1206.664 6404, 2012.
Stavros A Drakopoulos and Ioannis Theodossiou. Workers' risk underestimation and occupational health and safety regulation. European Journal of Law and Economics, 11:661-6566 2016.
Hein Fennema and Peter Wakker. Original and cumulative prospect theory: A discussion of empirical dif- ferences. Journal of Behavioral Decision Making, 10(1):53-64, 1997.
Michael Fleming and Asani Sarkar. The failure resolution of lehman brothers. Economic Policy Review, Forthcoming, 2014.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Peter Geibel. Reinforcement learning for mdps with constraints. In Machine Learning: ECML 2006: 17th European Conference on Machine Learning Berlin, Germany, September 18-22, 2006 Proceedings 17, pp. 646 653.. Springer, 2006.
12

---

Published as conference paper ICLR 2025
Peter Geibel and Fritz Wysotzki. Risk-sensitive reinforcement learning applied to control under constraints.
Journal of Artificial Intelligence Research, 24:81-108, 2005.
Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J Kappen. Minimax pac bounds the sample complexity of reinforcement learning with generative model. Machine learning, 91:325-349, 2013.
Thomas Gilovich, Dale Griffin, and Daniel Kahneman. Heuristics and biases: The psychology of intuitive judgment. Cambridge university press, 2002.
Abhijit Gosavi. Reinforcement learning for model building and variance-penalizae control. Proceedings of the 2009 winter simulation conference (wsc), pp. 373-379. IEEE, 2009.
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: survey of the ofate-of-the-art, Knowledge-basee systems, 212: 106622, 2021.
Matthias Heger. Consideration of risk in reinforcement learning. In Machine Learning Proceedings 1994, pp. 105-111. Elsevier, 1994.
Morgan Housel. Penguin, 2023.
Ronald A Howard and James E Matheson. Risk-sensitive markov decision processes. Management science, 18(7):356-369, 1972.
Ronald Huisman, Kees G Koedijk, and Rachel A Pownall. Var-x: Fat tails in financial risk management. Journal of risk, 1(1):47-61, 1998.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Nan Jiang. On value functions and the agent-environment boundary. arXiv preprint 1rXiv:1905.13341, 2019.
Cheng Jic, LA Prashanth, Michael Fu, Steve Marcus, and Csaba Szepesvári. Stochastic optimization in a cumulative prospect theory framework. IEEE Transactions on Automatic Control, 63())22877-2882, 2018.
Ming Jin. Preparing for black swans: The antifragility imperative for machine learning. arXiv preprint arXiv:2405. /197, 2024.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part 1, pp. 99-127. World Scientific, 2013.
Sham Machandranath Kakade. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003.
Andrei Kirilenko, Albert S Kyle, Mehrdad Samadi, and Tugkan Tuzun. The flash crash: High- figh-freeuency trading in an electronic market. The Journal of Finance, 72(3):967-998, 2017.
Tor Lattimore and Csaba Szepesvári. Bandit algorithums. Cambridge University Press, 2020.
Liu Leqi, Adarsh Prasad, and Pradeep K Ravikumar. On buman-aligned risk minimization. Advances in Neural Information Processing Systems, 32, 2019.
13

---

Published as conference paper ICLR 2025
Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy ai: From principles to practices. ACM Computing Surveys, 55(9):1-45) 2023.
Benedetto De Martino, Dharshan Kumaran, Ben Seymour, and Raymond J. Dolan. Frames, biases, and rational decision-making in the human brain. Science, 313:684 - 687, 2006.
John Kwaku Mensah Mawutor. The failure of lchman brothers: causes, preventive measures and recommen- dations. Research Journal of Finance and Accounting, 5(4), 2014.
Larry McDonald and Patrick Robinson. A colossal failure of common sense: The incredible inside story of the collapse of Lehman Brothers. Random House, 2009.
Teodor Moldovan and Pieter Abbeel. Risk aversion in markov decision processes via near optimal chernoff bounds. Advances in neural information processing systems, 25, 2012.
Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.
2005.
Sina Nordhoff, John D. Lee, Simeon C. Calvert, Siri Berge, Marjan Hagenzieker, and Riender Happee. (mis- juse of standard autopilot and full self-driving (fsd) beta: Results from interviews with of tesla's fsd beta. Frontiers in Psychology, 14, 2023. ISSN 1664-1078. URL http::://///// //www. front iersin. org/ http:///ww////ecccocooo//rttttta
2023 1101520,
James Opaluch and Kathleen Segerson. Rational roots of "irrational" behavior: new theories of economic
decision-making. Northeastern Journal of Agricultural and Resource Economics, 18(2):81-95, 1989.
Takayuki Osogami. Robustness and risk-sensitivity markov decision processes. Advances in Neurul Information Processing Systems, 25, 2012.
Debjani Panda, Ratula Ray, and Satya Ranjan Dash. Feature selection: Role in designing smart healthcare models. Smart Healthcare Analylics in loT Enabled Environment, pp. 143-162, 2020.
Bhavana Pandit, Alex Albert, Yashwardhan Patil, and Ahmed Jalil Al-Bayati. Impact of safety climate on hazard recognition and safety risk perception. Safety science, 113:44 53, 2019.
Stephen D Patek. On terminating markov decision processes with risk-averse objective function. Automat- ica, 3799)1179-13386 2001.
Matt Phillips. Gamestop's 's wild stock ride: How ai and social media drove short squeeze. The New York Times Business, 2021. URL httpp::/ www.isytimmem.....
gamestop-stock. htm.l Accessed: 2024-08-19.
Silviu Pitis. Consistent aggregation objectives with diverse time preferences requires non-markovian rewards. Advances in Neural Information Processing Systems, 36, 2024.
Sameera S Ponda, Luke B Johnson, and Jonathan P How. Risk allocation strategies for distributed chance- constrained task allocation. In 2013 American Control Conference, pp. 2230-3236. IEEE, 2013.
LA Prashanth, Cheng Jie, Michael Fu, Steve Marcus, and Csaba Szepesvári. Cumulative prospect theory meets reinforcement learning: Prediction and control. In International Conference on Machine Learning, pp. 1406-1415. PMLR, 2016.
SD Prestwich. Tuning forecasting algorithms for black swans. IPAC-PapersOOnLinee 52(13): 1500 2019.
14

---

Published as conference paper ICLR 2025
Matihew Rabin. Risk aversion and expected-utility theory: calibration theorem. In Handbook of the fundamentals of financial decision making: Part I, pp. 241-252, World Scientific, 2013.
Lillian J Ratliff and Eric Mazumdar. Inverse risk-sensitive reinforcement learning. IEEE Transactions on Automatic Control, 6563): 2019.
Paul Rogers. The cognitive psychology of lottery gambling: A theoretical review. Journal of gambling studies, 14(2):111-34, 1998.
Leonard Savage. The foundations of statistics. Courier Corporation, 1972.
Mehran Shakerinava and Siamak Ravanbakhsh. Utility theory for sequential decision making. In Interna- tional Conference on Machine Learning, pp. 19616-19625, PMLR, 2022.
Yun Shen, Michael Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning.
Neural computation, 26(7):1298 1328, 2014.
Aaron Sidford, Mengdi Wang, Xian Wu, Lin F Yang, and Yinyu Ye. Near-optimal time and sample complexities for solving discounted markov decision process with generative model. arXiv preprint wrXiv:1806.01492. 2018.
Samuel Henrique Silva and Peyman Najafirad. Opportunities and challenges deep learning adversarial robustness: A survey, arXiv preprint arXiv:2007.00753, 2020.
Herbert A Simon. Decision making: Rational, nonrational, and irrational. Educational administration quar- terly, 29(3):392-411, 1993.
Philip Stafford. Citadel securities trading algorithm triggers market volatility. Financial Times Online, 2022.
URL
ht https: wwwp///wwww... com/ http://teeet//33355---55553 https///www.....tooooocccooc//// 200--2999----2-1113333373333333- Accessed: 2024-08-19.
Robert Sugden. Rational choice: survey of contributions from economics and philosophy. The economic journal, 101(407):551-785 1991.
Peter Sunehag and Mareus Hutter. Axioms for rational reinforcement learning. In Algorithmic Learning Theory: 22nd International Conference, ALT 2011, Espoo, Finland, October 5-7, 2011. Proceedings 22, pp. 338-352. Springer, 2011.
Peter Sunehag and Marcus Hutter. Rationality, optimism and guarantees in general reinforcement learning. The Journal of Machine Learning Research, (6(1):134--1390 2015.
Richard S Sutton. The reward hypothesis, 2004. URL http://nccomplereddeas. net/rlai. cs. ualberta. hulRLA/rewarddyyottssss html.
Richard S Sutton. The quest for common model of the intelligent decision maker. arXiv preprint arXiv:2202.13252, 2022.
Hamdy A Taha. Operations research an introduction. 2007.
Nassim Nicholas Taleb. The Black Swan:: The Impact of the Highly Improbable: With a new section:" On Robustness and Fragilit", volume Random house trade paperbacks, 2010.
Tesla Tesla Al Day. hot.
Alan M Turing. Computing machinery and intelligence. Springer, 2009.
15

---

Published as conference paper ICLR 2025
Toni GLA van der Meer, Anne Kroon, and Rens Vliegenthart. Do news media kill? how biased news reality can overshadow real societal risks, the case of aviation and road traffic accidents. Social forces, 101(1):506-555 530, 2022.
Peter Vasterman, C Joris Yzermans, and Anja JE Dirkzwager. The role the media and media hypes in the aftermath of disasters. Epidemiologic reviews, 27(1):107-114, 2005.
Morgenstern von Neumann. Theory of games and economic behaviour, 1944.
Maxime Wabartha, Audrey Durand, Vincent Francois-Lavet, and Joelle Pineau. Handling black swan events in deep learning with diversely extrapolated neural networks. In Proceedings of the Twenty-Ning Inter- national Conference on International Joint Conferences on Artificial Intelligence, pp. 2140-2147, 2021.
Anders AF Wahlberg and Lennart Sjoberg, Risk perception and the media. Journal of risk research, 3(1): 31-50, 2000.
Guangyu Wang, Xiaohong Liu, Zhen Ying, Guoxing Yang. Zhiwei Chen, Zhiwen Liu, Min Zhang, Hongmei Yan, Yuxing Lu, Yuanxu Gao, et al. Optimized glycemic control of type diabetes with reinforcement learning: proof-of-coneept trial. Nature Medicine, 29310222633224 2023.
Gregory Wheeler and G Wheeler. A review of the lottery paradox. Probability and inference: Essays in honour of Henry E. Kyhurg, Jr, pp. 1-31, 2007.
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem, Robust markov decision processes. Mathematics of Operations Research, 38(1):153-183, 2013.
Rosalind Wiggins, Thomas Piontek, and Andrew Metrick. The lehman brothers bankruptcy a: overview.
Yale program on financial stability case study, 2014.
Paul D Witman, Jim Prior, Tracy Nickl, and Scott Mackelprang. Southwest airlines didn't I crash, but it nearly
fell apart... In Proceedings of the ISCAP Conference ISSN, volume 2473, pp. 4901, 2023.
Huan Xu and Shie Mannor. Distributionally robust markov decision processes. Advances in Neural Infor-
mation Processing Systems, 23, 2010.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection:
survey. International Journal of Computer Vision, pp. 1-28, 2024.
Yuanyuan Zhang, Xiang Li, and Sini Guo. Portfolio selection problems with markowitz's 's mean-variance framework: a review of literature. Fuzzy Optimization and Decision Making, 17:125-155 158, 2018.
16

---

Published as conference paper ICLR 2025
A
NOTATIONS
This section provides table summarizing all the notations and their meanings introduced in the main paper.
Notation
Meaning
Defintion
M
Ground truth MDP
Section
S
State space
Section 2
A
Action space
Section 2
T
Time horizon
Section 2
P
Transition probability function of M
Section
R
Reward function of M
Section
7
Discount factor
Section 2
Policy
Section 2
pr
Normalized visitation probability of M
Section
VM
Value function of
Section
u,
Value distortion function
Section
(Figure lap
w w
Probability distorction function
Section
(Figure lb)
Ma
Distorted MDP
Section
w( P)
Transition probability MM
Section
u(R)
Reward function of M,
Section
M
Human MDP (HMDP)
Subsection 5.1
pt
Transition probability of M
Subsection 5.1
R'
Reward function of M
Subsection 5.1
ph.
Normalized visitation probability of M
Subsection 5.1
[ p
Cumulative visitation distribution of M
Subsection 5.
p
Cumulative visitation distribution of M
Subsection 5.1
V
Value function of M
Subsection 5.1
M1
Er, ed
Perception gap
Subsection 5.1
M
Human Estimation MDP (HEMDP)
Subsection 5.2
Pt
Transition probability of M
Subsection 5.2
R
Reward function of M
Subsection 5.2
Vr
Value function of M
Subsection 5.2
M
Kr, Rid
Estimation gap
Subsection 5.2
R[j, Pr
Order statistics of reward and visitation probability
Section 6
Pr,
Probability of reward
Section 6
F(r)
Cumulative distribution function of Pr,
Section 6
B
Collection of all S-BLACK SWAN
Section
17

---

Published as a conference paper ICLR 2025
Notation
Meaning
Defintion
Cba Cbs
The extent of distortion of functions u and
Section 6
(Figures 1c
and ld)
Ehi
Minimum probability of S-BLACK SWAN
Section 6
u,
u" that satisfies B Ø, i.e. safe reward perception
Section 6
W,,
w that satisfies B = Ø, i.e. safe probability perception
Section
B SUPPORTING EVIDENCE
This section provides supporting evidence for this paper's 's main perspective; how interpreting black swans through the lens of human misperception is meaningful and applicable in the real-world scenarios beside Lehman Brothers Bankruptcy case introduced in Introduction.
Casel. Unexpected Drowning of NASA Astronauts Due to Overlooked Details (The following is summary of the case; please refer to (Clark) 2021 Brinkley 2019) for a more detailed account.) Before launching rockets, NASA conducted tests 1961, Victor Prather and another pilot ascended to 113,720 feet to evaluate the suit's performance. While the test itself was successful, an unforeseen risk led tragedy during the planned ocean landing. Prather opened his helmet faceplate to breathe fresh air, and as he slipped into the water while attaching to a rescue line, his now-exposed suit filled with water. Despite NASA's rigorous planning and preparation, the risk of opening the faceplate- perceived as an extremely minor detail was underestimated, resulting in catastrophic consequences. This highlights how rigorously meticulous planning can still fail account for overlooked events.
space suits using high-altitude hot-air balloons. On May 4,
Case2. Healthcare Diabetes patients typically experience highly chronic condition, making their state relatively predictable a few hours into the future (stationary environment). However, rare hypoglycemic events, characterized by sudden and dangerous drop in blood sugar, pose significant risks. To address this, Wang et al. (2023) developed an RL model capable of predicting changes in a patient's is condition and pro- viding optimized freatments. Furthermore, (Panda et al. 2020 Ammbiika al., 2024) emphasize the critical importance of selecting appropriate signals inputs, human misperceptions about what constitutes im- portant signals can lead to unexpected and suboptimal decisions. As a result, traditional supervised learning methods, such general Transformers paired with loss functions like MSE, may struggle to accurately predict rare events like hypoglycemic episodes when the input signals fail to align with the underlying dy- namics.
C MISPERCEPTION IS INFORMATION LOSS
Based on Hypothesis] this prompts us to investigate the concept of misperception. Initially, we must clearly define what constitutes perception. In The Quest for Common Model of the Intelligent Decision Maker, Sutton defines perception as one four principal components of agents, stating: "The perception component processes the stream of observations and actions to produce the subjective state, summary of the agent- world interaction so far that useful for selecting action (the reactive policy), for predicting future reward (the value function), and for predicting future subjective states (the transition model)" (Sutton) 2022). This definition leads us to consider misperception as the information loss occurring when processing observations into the subjective state, such that the reward and transition model are not equivalent to those from the envi- ronment. The interpretation of misperception as information loss during processing is somewhat ambiguous, depending on how the boundary between the agent and the environment is defined. Turing first proposed the
18

---

Published as a conference paper at ICLR 2025
concept of boundary between the agent and environment as a 'skin of an onion' (Turing 2009). and later, Jiang (219)) suggested that algorithms are not cuunday-iinvarian
Therefore, we propose a new agent- environment framework that incorporates the notion that misperception is the information loss from an agent's processing. This framework positions perception at the intersec- tion between the agent and the environment. We provide a detailed description of our agent-environment framework in Figure 2
D RELATED WORKS: NECESSITY OF A NEW PERSPECTIVE TO UNDERSTAND BLACK SWANS AND EVIDENCE FOR HYPOTHESIS
In this section, we focus not only on addressing the necessity of a new perspective to understand black swan events but also on providing evidence for the proposed perspective of black swan origin (Hypothesis [ This is concretized by examining the following two questions. First, in Subsection D.I we discuss the in- sufficiency of existing decision-makinn rules under risk by exploring related works, which support the need for new perspective understand black swans. Specifically, we address why existing safe reinforcement learning strategies for solving Markov Decision Processes are insufficient to handle black swan events?. If this premise is validated, then in Subsection D.2] we elaborate on the motivation and related works that sup- port our informal hypothesis of black swan origin (Hypothesis D. Specifically, we explore how irrationality relates to misperception and how irrationality could bring about black swan events.
D.1 DECISION MAKING UNDER RISK
Based on the comprehensive survey on safe reinforcement learming in Garcia & Fernández (2015). the algo- rithms can be classified into threefold: worst case criterion, risk-sensitive criterion and constraint criterion. We elaborate on why the existence of black swans in the environment renders these three approaches insuf- ficient.
Worst case criterion. Learning algorithms of the worst case criterion focus on devising a control policy that maximizes policy performance under the least favorable scenario encountered during the learning process, defined as m&X min V (s;w), where W represents the set of uncertainties. This criterion can be categorized based on whether W is defined in the environment or in the estimation of the model. The pres- ence of black swan events in the worst case, where W represents aleatoric uncertainty of the environment (Heger) 1994] Coraluppi| 1997; Coraluppi & Marcus, 1999, (000), results in overly conservative, and thus potentially ineffective, policies. This occurs because the significant impact of black swan events inflates the size of W. even though such events are rare. In practical terms, this could manifest itself as abstaining from any economic activity (T)) such as not investing in stocks or not depositing a check against future potential bankruptcies (minwew VM w))
) in order to maximize its income (max A)), or maintain- ing constant health precautions such as wearing mask or maintaining distance with groups (i) to prepare for a possible pandemic (4 = mingew VM w)) in order to maintain its health (max Similarly, when W encompasses the uncertainty of the model parameter (Bagnell et al.) 200T [yengar] 2005 Nilim & EI Ghaouil 2005; Wiesemann et al. 2013] Xu & Mannor, 2010) as seen in robust MDP or distribution- ally robust MDP this aligns closely with our black swan hypothesis, where misperception of the world model is similar to uncertainty in model estimation. However, the need to accommodate black swan events requires enlarging the possible set of models (DW), leading to extremely conservative policies. This can be likened to performing
overly pessimistic portfolio optimization (T)) where every bank is assumed to have a minimal but possible risk of bankruptcy (minwiew VT (8; w)), thus influencing asset allocation strategies (maxzen win V (s; w)) be extremely conservative in asset investing.
Risk sensitive criterion. Risk-sensitive algorithms strike a balance between maximizing reinforcement and mitigating risk events by incorporating sensitivity factor / < 0 (Howard & Matheson, 1972
19

---

Published as a conference paper at ICLR 2025
1987 Patek 2001). These algorithms optimize an alternative value function (s) 3-1 log En [exp=c S P..80 80 = $], where 3 controls the desired level of risk and G == Σi-o M( at) ) is a cumulative return. However, it is recognized that associating risk with the variance of the return is practical, in (s) 3 log [E, [cxp[x maX [r][ & var(G) O(3)). and the existence of black swan events does not significantly affect the returns of variance (var(G) G)) due to their rare nature. It should be noted that risk-sensitive approaches are not well suited for handling black swan events, as the same policy performance with small variance can entail substantial risks (Geibel & Wysotzki 22055). More generally, the objective of the exponential utility function is one example of risk-sensitive learning based on a trade-off be- tween return and risk, i.e., max (F.,,,.eCC,. [G] (ww) (Zhang et al. 2018),, where is replaced by Var(G). This approach is known in the literature as the variance-penalized criterion [Gosavi) 2009), the expected value- criterion (Geibel & Wysotzki, 2005). However, a fundamental limitation of using return variance as a risk measure is that it does not account for the fat tails of the distribution (Huisman et al. 1998 Bradley & Taqqu 2003 Bubeck et al., 2013 Agrawal et al. (021). Consequently, risk can be underestimated due to the oversight of low probability but highly severe events (black swans).
Chung & Sobel
variance criterion (Taha 2007: Heger 1994), and the sosoottd-valuemmiu---vraaaaa
Furthermore, a critical question arises regarding whether the log-exponential function belongs to appropri- ate utility function class for defining real-world risk. Risk-sensitive MDPs have been shown to be equivalent to robust MDPs that focus on maximizing the worst-case criterion, indicating that the log-exponential utility function may not be beneficial in the presence of black swans (Osogami 2012 Moldovan & Abbeel 2012] Leqi al., 2019). This issue first raised by Leqi et al. (2019) and led to the proposal of more real- istic risk definition called Human-alignedd risk', which also incorporates human misperception akin to our informal black swan hypothesis (Hypothesis] [
Constrained Criterion. The constrained criterion is applied in the literature to constrained Markov pro- cesses where the goal is to maximize the expected return while maintaining other types of expected utili- ties below certain thresholds. This can be formulated as max E, [G] subject to N multiple constraints hG( S ag. for i [N] where h4 R + R is a function of return G := Σt=o 7 *R(s st, az) (Geibel 2006). Typ- ical constraints include ensuring the expectation of return exceeds a specific minimum threshold (a). such as ER, [G] 2 Oz, or softening these hard constraints by allowing a permissible probability of violation (c), such as [ 2 a) 2 c, known as chance-constraint (Delage & Mannor (2010); Ponda et al. (2013).
Constraints might also limit the return variance, such as Var(G) S a (Di Castro et al. (2012)). However, the presence of black swans highlights one of the challenges with the Constrained Criterion, specifically the appropriate selection of a. The presence of black swans necessitates lower a, which in turn leads to more conservative policies. Furthermore, a black swan event is determined at least by the environment's 's state and its action, rather than its full return. Therefore, constraints should be redefined over more fine-grained inputs-not merely returns, but in terms of state and action-which leads to our definition of black swan dimensions (Definition 3)
D.2 How IRRATIONALITY RELATES WITH SPATIAL BLACK SWANS.
Before starting Subsection D.2 we clarify that the term irrationality is used here to denote rational behavior based on a false belief. In this subsection, we first review existing work on the four rational axioms and then claim how two of these axioms should be modified to account for irrationality in human decision-making.
Rationality in decision making. In the foundation of decision theory, rationality is understood as internal consistency (Sugden (191I>; (Savage| (1972). ). A prerequisite for achieving rationality in decision-make making is the ability to compare outcomes, denoted as set Q where |[]] = N. through a preference relation in a rational manner. In von Neumann (1944), it is demonstrated that preferences, combined with rationality axioms and probabilities for possible outcomes, denoted as p; which is a probability of outcome og Ω imply the existence of utility values for those outcomes that express a preference relation as the expectation
20

---

Published as a conference paper ICLR 2025
of scalar-valued function of outcomes. Define the choice (or lotteries) set L, which is combination of selecting total N outcomes, that is, Σin1 P:oz. The essential rationality axioms are as follows.
1. Completeness: Given two choices, either one is preferred over the other or they are considered equally preferable.
2. Transitivity: If A is preferred to B and B is preferred to C, then A must be preferred to C.
3. Independence: A preferred B. and a event probability p é [0,1]. then PA + (1-p)C' should be preferred to pB (1-p)C.
4. Continuity: such that B is considered equally preferable to pA (1 -1-17 p)C.
preferred to and B is preferred to C, there exists a event probability p E [0, 1]
Expanding on these axioms, Sunehag & Hutter (2015) extends rational choice theory to encompass the full reinforcement learning problem, further axiomatiZing the concept in [Sunehag & Hutter (2011) to establish a rational reinforcement learning framework that facilitates optimism, crucial for systematic explorative behavior. Subsequent studies focusing defining rationality in reinforcement learning, such as Shakerinava & Ravanbakhsh (2022); Bowling et al. (023), concentrate on the axioms of assigning utilities to all finite trajectories of Markov Decision Process. Specifically, [Shakerinava & Ravanbakhsh (2022); BBoliig al. (2023) clarify the reward hypothesis [Sutton that underpins the design of rational agents by introducing an additional axiom to existing rationality axioms. Furthermore, Pitis] (2024) explores the design of multi- objective rational agents, and Carr et al.] (2024) explores and defines rational feedback in Large Language Models (LLMs) by investigating the existence of optimal policies within framework of learning from rational preference feedback (LRPF).
Irrationality due to subjective probability. The definition of irrationality and its origins has been exten- sively investigated through case studies in various fields such as psychology, education, and particularly economies. Simon (993) defined irrationality as being poorly adapted to human goals, diverging from the norm of human's s object. influenced by emotional or psychological factors in decision-makinn
Subsequently, Martino al. 2006); Gilovich et al. (2002) further concretized what exactly these emotional or psycholog- ical factors entail by describing them as information loss during human perception of the real world. More specifically, Martino et al. (2006) pointed out that in a world filled with symbolic artifacts, where optimal decision-maid making often requires skills of abstraction and decontextualizztionn such mechanisms may render human choices irrational. Further studies, such as Opaluch & Segerson (1989), J scrutinize more deeply and classify the irrationality of human behavior into five factors: subjective probability, regret/disappointmen,, reference points, complexity, and ambivalence.
In this paper, focus the subjective probability factor to elucidate the relationship between irrationality and spatial black swans. Opaluch & Segerson| (1989) explores subjective probabilities as an early modi- fication the expected utility model from yvon Neumann (1944), focusing on decision-makers who rely on personal beliefs about probabilities rather than objective truths. This minor conceptual shift can lead significant behavioral changes due to the imperfect information and processing abilities of individuals. Especially, Opaauuch & Segerson| (1899) highlights the difficulty in accurately estimating the probability of rare events such as black swans which often leads critical errors in judgment. These errors occur be- cause rare events provide insufficient data for accurate probability estimation or are misunderstood due to their infrequency, leading to perceptions that such events are either less likely or virtually impossible. This misperception is exemplified in various scenarios, such as:
1. An individual working in a dangerous job who has never personally observed an accident may underestimate the probability of an accident occurring Drakopoulos & Theodossiou (2016); Pandit
et al. (2019).
2. Media coverage of events such as plane crashes may cause an overestimation of the probability of
a crash, since the public is aware of all crashes but not of all safe trips Wahlberg & Sjoberg (2000);
Vasterman et al. (2005): van der Meer al. 20022
21

---

Published as conference paper ICLR 2025
3. The popularity of purchasing lottery tickets may be explainable in terms of people's inability comprehend the true probability of winning, influenced instead by news accounts of 'real' people who win multi-million dollar prizes (Rogers (998); Wheeler & Wheeler (2007 BetterUpp(222))
E CUMULATIVE PROSPECT THEOREM AND RISK
We note that existing works on incorporating cumulattvv prospect theory (CPT) into reinforcement learn- ing, such as (Prashanth et al. (2016); Die al. (2018): : Danis et al. (2023)), primarily focus on estimating the CPT-based value function and optimizing it derive optimal policy. Specifically, (Prashanth et al. (2016); Tie et al. (2018) demonstrate how to estimate the CPT value function using the Simultaneous Perturbation Stochastic Approximation method and how to compute its gradient for policy optimization algorithms. Ad- ditionally, (Shen et al. (2014); Ratliff Mazumdar (2019) proposed a novel Q-learning algorithm that applies utility function Temporal Difference (TD) errors and demonstrated its convergence. However, these studies (Prashanth et al. (2016); Jie et al.] (2018); Danis et al. (2023): [Shen et al.J (2014); Ratliff & Mazumdar (2019)) do not focus on learning the utility and weight functions, and w, but rather assume these as simple functions and focus on how to estimate these functions.
However, this study aims elucidate the mechanisms by which black events arise from the discrep- ancies between M and M, despite the agent having perfect estimation, i.e., Rp Kp 0. As future work, concentrating on devising strategies to reweight the functions u", and to mitigate the divergence between the Human MDP M and the ground truth MDP M is suggested as way to achieve antifragility.
F PRELIMINARY FOR PROOFS
This subsection covers the preliminary concepts necessary for proving the theorems and lemmas presented the paper.
First, in discrete state and action space, the value function M could be expressed inner product of reward function R and normalized occupancy measure pa as follows,
1-7J Σ R(s,a)P=(s,aa) 1-7 (a,0)ESxd
(00
(5)
Based on Equations 5 1 and ( the CPT distorts the reward and its visitation probability as follows,
1-7T Σ u(R(s,a) 1 1-7 s.aESxA
dsda (f (Sp
VM
(6)
where f denotes the value function that was distorted due to misperception. As one property of CPT is that human perception exhibits distinct distortions of events based on whether the associated rewards are posi- tive or negative, we divide the functions u( [[ss s, a)) and w(ʃ (=(saa) into u (R(s, a)), w (ʃ (s.(s)) where R( 0, and ut (R(s,
(((s,a)) a)) w² (ʃ P (s.,a)) where R(s,a) 0. Assume that the rewards from all state-action pairs R(s R(s.a) a) are ordered as R S Pi Plss Then, the Equation (6) can be represented as follows:
Rig $ O R S ... S Ailtsasa and the visitation
probability Pij < P[2]
22

---

Published as conference paper ICLR 2025
1 1- 2² /
|SLA1
)
(x0)
Σ
u
(R(3))
w Σ
- w
Σ 11
1-7
1=1
Σ n (R(q)
w
Σ Lil
w
Σ
1-1
F1
F=1
|
/SIAA
|SIAA
Σ u² (R(z)
w"
Σ
P [s]
w"
Σ
P
#=+1
we define the reward as the random variable X, then we can regard its instance as R[g and its probability [i] where the probability is dependent on the policy Suppose that reward function R: S A - R is
one to one function. Then the probability !! p" R - [0, denotes the probability of reward and we denote Pr. Then, for a reward random variable R F expanding the how CPT- applied value function look like in Equation (4), we can rewrite the Equation based on continuous state and actions space as follows.
(xr( J w (I, (R) r)) r)) dr So w (P.(( (R) > r)) dr
(8)
We use the fact that for real-value function g. it holds that E[a(R)] fod Pr(g(R) > r)) Within the above problem setting, the agent's goal is estimate the value function under safe perception u,, w. as follows:
(00 S. w² (x) r)) Lo w, (F, r)) dr
Note that the safe perception is only defined over w" and u w, and u, However, the agent possesses its own perceptions MA, for which we assume the risk perception is represented as:
(5)( = S. "² (X)
So" w (P,(U dr
(10)
As time goes by. the agent's goal is approximating the weight functions and utility functions such as w wT and u ui,- Then, by the single trajectory data up to time i.e. (h(s;), a,, urr:), (s)+))))()))))))))))(()))))))))) where the reward value itself and its sampling distribution are distorted due to the functions u and w, respectively (see Lemmall [ for definition of function h). Since function h maps state space to state space, we just use the notation (s/, 0; u(r;), 8/1m )) to denote Let r;, i = 1,.., & denote n samples of the reward random variable X. We define the empirical distribution function (EDF) for u² (X) and u (X) as follows
- (at(s))))) and (r) - m
1 - 7 m1
-
1 (r)))))
Using the EDFs, the CPT value up to time & can be estimated as follows.
V VMT [o ut 1ct(11 (I) dr w- (1-1 dr
(11)
Again, we note that the gap between M and M is defined over gap between (u ",,w"") w) and (M, w[) that is proportional to the existence of spatial black swan events.
G PROOFS
We first like to note that the following lemma helps to quantify how much the distortion on transition prob- ability is related to the distortion the visitation probability.
23

---

Published as conference paper ICLR 2025
Lemma 2. If maxs,a ()) P*(+ss s (1-22² where Ed > 0, then the agent can guarantee ea-
perceived visitation probability.
We begin with Lemma to prove Lemma Recall that (l."(s.a) is the ed-perceived visitation proba- bility if max( (s, a) P (s, a) < Ed- This perception gap arises from factors such as transition probabilities, policy, and state space. In the following lemma, we show how the perception gap in transition probability accumulates into the visitation probability, Before, we define ep-perceived transition probability
if (s,a) [|P([|s, a) - (+)s,))) < Ep holds. We denote IP (s, a) as the probability of visiting (s, a) at time with policy
Lemma 3 (Bounding visitation probability of step i when 6p-perceived transition holds). for all (8,a) holds Ep perceived transition probability, then we have
Σ (x.0)ESXA
(s) (P7 slep
max T
that holds for all e
Proof of Lemma 3 Proof by induction. We use short notation for P(s: = St-1 = s'a a') as Pi(s s',a') and (s, = I st-1 = =s',at1 ar-1 a') as P (s| s',a') By the definition of rational transition probability the statement holds at for any policy æ. Now, suppose the statement holds for 1 for any policy æ. Then, we have
Σ aT) pp. (s,a)
(x.a)eSx
Σ (s.a)sSAA
T(u = a 81) 1
=a ({ Σ (P
Σ (s,a)sSxA
a(a=aass
sy s) Σ (Pi(s ddF E
Σ
Σ
E(((
SES
Σ
Σ
(P.
#ES dd
3'aa
Σ Σ
Σ EP((e)ddd)) - -P
Sep Σ
P (s', + (1
cpp 1+ - 1t
Step
The above inequalities hold for all æ. Therefore, the statement holds for all /
Now, prove the Lemma
24

---

Published as a conference paper ICLR 2025
Proof of Lemma 2 probability, we have
Lemma is almost a corollary that stems from Lemma
By the definition of visitation
Σ (ss.a)-PPii(
pm
Σ
Σr' (P (P. (PP
(s.0)ESAA
(s,0)ESXA
1-0
$
Σ
[PT (8,0) PP, a))
(s,a)sSxA h=0
Σ
(s, a) (s,"as))
00
(s,a)ESXA
Σ 7% (1-
Ep
Let S= = Σt 1't then y Ei
t E
E (t 1). Then by subtracting those two equations, we have
(1-7)S- Σx= 7d --x. Therefore we have 5 (1-7)2- Finally, we have the following inequality
7 Σ (P (1- ?
(1-7)²
Cp Ep
(s.0)eSxA
Proof of Lemma 7 First, note that have assumed the image of the function R is closed and dense
[-Ruax, Rmmxx] Then, in the progress of projecting all (s,a) into the reward, we define the probability of
reward P(R r) Ev(s,a)sSxA d' (s, (s
r]. we use short notation for IP rr) as PR. Now, since d= (s, a) is the visitation probability of visiting (s,a), a), then this could be converted to IP(R = r) by d' (R= T==RR-(a (s, a)) where R-1 is many to one function.
Now, since IR is the many-to-one function, we can define independent block the S as the set Z(r) ((s,a) E S *ARR(sss a) r). Note that if r1 T2, then Z(ri) n Z (r2) 0. Then, if satisfies the set Z to be connutation-invariiann Namely, if R(s1, a) = R(s2, a), then R(h(s1)) R(h(s2), a) holds then there exists a one-to-one mapping function h [-Rmaxx R + R-Rmaxx Rmaxx] such that
R(s,a) ().,))=(((((()))) h(R(h()),a))
holds. The proof can be divided into two folds. The existence of such function and its one-to-one mapping function exists. We first prove the existence of such function h. This is because for any state and action 8, a, suppose its reward value is r. Then suppose g(s) s'. Then since image of function R is closed and dense, there exists r' [-Rmax Rmax] such that R(s', a) r' holds. Then, one can say the function = h(r') exists. Now, we prove the one-to-one mapping property. suppose for two state and action pair ($1, and ($2, and let s = i(s()) and &/2 h(s)). Now, suppose R(s',a) + R(s/, a) holds. Then, due to the property of h, then it should also satisfy R(s1,a) R(s2.a). Therefore, this concludes that & is the one-to- one mapping, and the following holds
dh (R(g(s), = = (b(((((())) = h(r))
=d'(R(s, a) = (()))
=P(R= h(r))
25

---

Published as conference paper ICLR 2025
holds. we denote P (R h(r)) as P((R). Then, let's define two different functions h" and h- such that we want to claim that
w (Sr (S"imnee
-Rmaa dlP, (R)) and w" (SR dlP
S
S
dPR
-Rmaxx dl (R)+(R))
(12)
holds for any w _ Since the proof for either is similar, we prove the case for the existence of h under w distortion.
Now, recall that for 0 < < b, w" (x) holds and for < 1, w w=x)) holds and w (x) is monotically increasing function. Define гb с [- Rmax, 0] such that b:= f-Rmax dlpR holds, and for notation simplicity we
deonte F (r) - dpR. Then, one can say- - -Rmax < Tb. w (r)) < F() holds and. Then we can (r)rr always find unique ratio 0 (r) < that depends such that w (F(r)) -RRmax dP, holds where
(()) w (F((r))
This leads to set h(r) = (r)r = w (F(r) that satisfies and also one-to-one mapping. the same
manner, we can also identify h(r) (()rr = w (F(()) where < 0 holds for y(r) > 1. Then,
this completes that the function h: w (F(r) satisfies one-to-one function and Equation (12). This completes the proof.
Proof of Theorem 7 By the definition of optimal policy and the value function definition at the time T= we have the optimal policy at time 0 as follows.
ा arg max Vo(s)
arg max Q0( aeA
= arg max R(s,a) acA
arg max V/ (s) acA
arg max (s,a)
aeA
arg ((((.()))))))
aeA
for any fixed e S, let's assume a" is the argument that maximizes the R(s, a). Since u the non-decreasing convex function, a" is still the same argument that maximizes the u ([[(s s, a)). Therefore, x x .f holds.
Proof of Theorem We prove by backward induction. First by theorem . ui holds. Now suppose that के+ = T holds for all t ....., T. Now, we prove the statement holds for To prove म, = क, it is sufficient to show if QT (s,a) Q7 (s, a'). then Q (s, 2 Qt (s,a') also holds for any actions a, A. First, the gap OF (s,a) QT (s, a) could be expressed as
Rr(s, a) R a') + (P(s)]s, P(s2ls, ((ssss,,)))((s))ss)))))) (Vri(s)) =P((s1 a) P(szls, (Vi)(s)) V (82)
v (s2))
26

---

Published as conference paper ICLR 2025
and Q7"h (s,a) - Oi (s,a)an as
(s,a) - ([((s,)) -(((sa))R((((dd)) -((()ssd)(((ss)s)))))-))) (1 V ( Qi (s, (..aa
(Vti(ss)
( (Vin (s1) van (8s)
= =(w(P((s)1s,))) - w0 ((P'(ssss (vii (s1) - -Vii (s2)
the reward during t [1,7- zero by our problem formulation assumption in section ??. Now, without loss of generality, we assume (51) > (=2). Then, due to our assumption that T.. - holds for I' = 1, ..,, T, we also have V (x1) > VIT (82). Also, noticing that weight function w also increasing function, then P(s1(8,a) P(82|8, a) also guarantees wi ((1) a)) > w( (()2(8, holds. Therefore, we can claim Q7 (8, a) - (s, > holds, then Qt (8,a) Q fh (s, a) also holds. Then, this leads to claim that arg max Q a8 = arg max gaa (s, a), which implies के xt +- This completes the proof.
Assume that Theorem does hold. Given T 2. have Vi((() Ra(s) for each state 8. At time t 1, assume R2 (81) s R2 (x2) s R2 (#3). The condition Q 1. (s,01) Q (s, 02) is then expressed as:
Proof of Theorem maxaeA (2(8, 0)
(P(si (2))))
+ (1- (P (s2 (w (s2 s. s.ai)+ (s 8,0))) w (P (,,1)) (s2)
s, Rs (sa)
(P 8, R2 (51) (P 8, 8,02) P(s) ((5) (P 8, Rg (82)
+ +(1 (1-ww( 8,02) ((i 8,02)) Ra (83)
which simplifies to:
(w (P (si 8, - (s (R2) Rs (88)
+(w (s2 s, + P P( (P (s
-(w (P s, (si ss,02)
| s, (P
(P
(Rg (32) Rs (ss)) 20
For the non-distorted case, the analogous expression is:
((8118,01) P (s1 8, (2)) (Ra (81) Rs (8s)) + +P 8, 8,a1) - P Ps22 8, a2)) (Ra (82) R& (sa)) 20
For arbitrary reward functions, R2, the equality of the two cases under any weighting function w leads to:
w (P((s) 8, 88 - (P s,,2))
w (P (82 8,01) P (8) ssaa)) - w (P (s) s,a;)) (w (P (32 s, s,02) ((8 8,02)) - w (P 8, 8,a)))
P (s 8,01)- - P (s) 82)
P (x2 8,01) - P (>2 8202)
where w(p) p is the only solution, contradicting the distortion required by Definition
27

---

Published as conference paper ICLR 2025
Proof of Theorem The proof of Theorem is divided into three-fold.
1. Proof of asymptotic convergence
We first prove Equation 3 of Theorem 4 in this part 1, then we prove Equation 4 of Theorem 4 in part
3 of this proof. Note that the empirical distribution function F (r) generate Sticlgies measure which takes mass each of the sample points U (R,).
or equivalently, show that
lim Σ u' (( +10+000
n " w² n2 ()= n0000 Joae w² (P((Utt))dt w.p. 1
(13)
72
where n denotes the number of positive reward among |S||AA, Let E and f denote the t quantile of u² (X) and u (X), respectively.
For the convergence proof, first concentrate on finding the following probability,
P Σ u² (Rp)) [=' ((" [n-2 w=
(1²
'"' i
w
>E
π2
72
72
n
(14)
for any given e > O. It is easy to check that
P(
m-1
Σ
(R())=
(w+ "0== wt
11
1
n-1
(w+
12 -
w
1 1
> ()
71
SP(U
n-1
u (R(g) (w"
n1 11-
w²
11 i 1
5 (w*
i (==
w
n1
-
F=1
n
Σ p(|w" (R(q) (B(a) (w" n
(-=-)= w²
n- 1---- 1
i
n-i-1
وو
+
(wi
w
(15)
n2
n
n
n
Σ P((C'' (R(z) { ) (w* (n-
(--i-1
w²
n
n
Pe(ert(R (R(a))
(Raa) - - ( -)
n
Σ P(((u - nlla!
(16)
The right-hand side of Incquality (6) could be expressed as follows.
p P(|u* (R(a) - E n
e
= =P(u u= (R(() -  > n
P u1 (R(z)= - < <- m(laa))
28

---

Published as a conference paper ICLR 2025
We focus on the term P (u* (R(a)) - nt1aa
> ). Now, let us define an event A = 4aw (X.)=E )
where t = n. n. Since the Cumulative distribution is non-decrasing function, we have the following.
P u (R())) 5 6 = P
Σ
A₂ > n- (1 11-11111100 -m(aaaa))
0-
Ag n. [1- -= الايصميي n (F* " -
p
Σ
Using the fact that EA, = 1 F* (5 m( in conjunction with Hoeffding's 's inequality, we obtain
11=FF(F6
n P(
E Ar - n [1 [1- F (5 n )J > n [F (5 + "n((aa)) _ E -2n-6
2
(17)
1=
where 6, F ( ) i. Since F*t (x) is Lipschitz, we have that 6; s LF (1-a)
Hence, we obtain
P(u (R()) E 1 1 a
<
2n I = 2n 2n
(18)
similar fashion, one can show that
P(u" (B(g)) -
<
SE Se Ln
(19)
1-a
Combining (18) and 19 we obtain
p( (Raa) -E _ $2. 2 -2n (0,,1(
I-a
Plugging the above in (6).. we obtain
P( Σ (R(z) (w* (==²)- w² (--i-1)
Σ 5: (w"( " w² (--
-
)) >c)
n2
n
1=1
*
72
n
2n 22n Lipt Lgt
(20)
Notice that Σis 2n.e 2n Live < oo since the sequence 2n 6 2n will decrease more rapidly than the sequence vk> 1.
By applying the Borel Cantelli lemma, we have that Ve > 0
P( Σ at (Rqg) (w+ m=-1) (==²))))) w i
n-1
(w+( w w01 (----11) )) e) = 0,
Σ
Et
n
n
which implies
a² x"(RRg)- (lut(m=1) ( w -- Σ
-
n-1
ct (w* ( ( w0 ( -==in1)) n-+oo w..pl.
n
n
n
which proves 13
Also, the remaining part, conducting the proof of convergence w" and u .i..
(R(g)(W n -= w
n2 ()=1)) n-000 f(r w (P((> w.p. (21)
lim
Σ u
+00000
29

---

Published as conference paper ICLR 2025
also follows simliar manner. we omit the proof for this.
2. Proof of value function lower bound
By the definition, have the following
S wi (P,( S(o w (P.
(o0(50) (so)
(R> ((RR>))))0 dr)
w. (P,
w. (P, (R> r)))
(Lo " (P, (R> (R) [m w (Pr(" (R r)))
Lo w. (P, (R (R. S w. (Pr(u (R> r)))dr
[ w. (P, (R (R>r)))
0 w. (P (R r)))d
Term (II)
We first under bound the term (T). For notation simplicity, we let g(r) Pr (R > r))) and g.(r)
P. (r (R>r)) r))) Then we have the following
[ -Rmae
Term (I)
(r)
Now, since w. (x) is monotonically increasing in æ e [0, a] and monotonically decreasing in x e [a, 11. we could say for any y, [0, 1]. # that
w. xx) w (3)
(w))(2) min (w])(=)= min (())))) 2c(0.]]
where (x,, v). The first equality holds due to the mean value theorem. Therfore it holds that
[ Rmux
Term (I) =
-
[ Riuox
min
((v) (g. (r) ((r))((r))
f. Rmax (o. - -r)())g((r))
min (ww))(O),,
Now, recall the definition of (r) and gr)), then we have the following
|Sa (g- g()) dr |ERW,, [2,(R) u [[()]]
30

---

Published as conference paper ICLR 2025
Now, let denote the intersection of (R) and y ++ Cbe as R -Ros. We can say if the blackswan happens, then its reward is bounded between Rmax: R6s) Then we have the following,
[ Ramax
(g. - 9())
[RRP [R,(R) - u (R)]
ER-P, [1 [R Rbb.] (u, (R. u (R)))
- -ER-P, [1 [R 2 Rhs (- (R. u (R)))
ER-P, [1[ <Ro (R. vi (R)))
Term 1-1
BR-P= [1 Ras (-u1 (-u + u (R)))
Term 1-2
2 R-P,,
E
[1
[R Ros) (u, v (R)))
To lower bound the Term 1-1, let's denote the minimum reachability of blackswan events as minn # 0. Then we have
Rmaax
Roe min Chs
Term 1-1
min Re[- Reave Ro.]
Iu (R)(R -., (R)
Rmax
Rmax Rue min
2
Rmax
Ehs
|ur (- Roa) u. (RRs))
(22)
Term 1-2 5 Rés the max |u (B) u. (R)
Rmax Re (o-
Rús tbe |ui (-Rbs) u. -baa) Rmax
$
(23)
Therefore, we have the following equation,
Term 1 (Rmax
Ros) bbs RosCbs (-(-oo)) -
Rmax
Also, since the function (r) is convex, and u. (-Rmax) < - -mmax + Cbs holds. Therefore, we could say M, (r) Rmas-Caa Coe r. This leads us to come up with u, (- RB8) < Raaax- Coe Ros). Therefore, we have gap Rmav lowerbound as
Hmax
Rss
|M (-Rna) - u, (-((-b))) 2 (Rmax Cha)
(Bes Cbs)
Rmax
(Bmax Rro Che
Rmax
31

---

Published as conference paper ICLR 2025
The above inequality could be minimized as
(Rmax
Rbs)
min be
RosCba
(Rmax
Roa)
Term I
Ranax
Riax
(Rmax
Ros)
min
Rosebs)
(Bmax
Ras Cbs
R2
Ryaax
Now, let's upper bound Term 2. Before, recall that the definition of g(r)) = (P,(u r)) and note that
by the definition of black swans, we have (R) R Che holds for R RRmax; RMs). Therefore,
we can say for all [- Rmaxx - Rea),, PRa)).9e (()) holds. Therefore, for all [-Rmax: - Ros], we have
w, (g(r)) w (g(r)) (1) w (1) =1
[
W. (g(r)) - w (r(r))dr
w.()(r) w (s(r))
w w(((r)) w (g(r))
-Rmsst Che
Lr((r) gl
Remax 10p
(L -1) S
g(r)dr
Ranax Cos
Rmax
Che
<( - 1).
Ebo
2mmax
=(L - -1)
1- 1-P,(U-(R)) <r)
Rmax+Chr
=(L - 1)
P- < <r)
-Rmax+Oro
Cbaa -fo --Rmuxt
(L 1)
Pr(u (R) < r))dr)
(L 11) Cba) IRRP, [u
+ Cha < R < 0]]]
(24)
Note that if Rinax Obs Ross then
Rmax Obs Res Ros Ebs u (-Ruux + Cos) (25)
1[- [-Raaxx* + Crs < < 0]: ER-E, [u (R)
bb
2Rmax
2R
and if RRmax + Chs < - Rys: then
Rmax Cbs Ebs ui ( (-Rmax Cba)
1[-Rmax + Cbs R 0 ER--E [u (R)]
(26)
2R
Therefore, combining the Equations 24p.
25
26
we conclude that
(Rmuux
Row)
hs
min
RRscbs))
(Rmax Row Cbs
Term II s C.C
R2
where C [0, is constant. This completes the proof.
32

---

Published as conference paper ICLR 2025
3. Value function upper bound
For the proof of Equation 4 of Theorem 4 we utilized the following Lemma f which provides a concen- tration inequality on the distance between empirical distribution and true distribution.
Since ut (R) is bounded above by u² (Rmax) and wt (p) is Lipschitz with constant L (w" (w))),)), we
have the following inequality.
(P(""(X)) r)dr J.a 00 w² w111
(Bums)
w" (P(u" (X)) x)dx Soa
v (Bmana
w (1- (x))
o' (Rmux)
L* < 2) - (z))dz
ST" ut (Rmax)
sup P(u"t (X) < P. (1)
zelk
Now, plugging in the DKW inequality, we obtain
P P(1S% w w*(((( > x)dr - S,0 w F1- > ></2)
s P (L (L''((() (Rmax) (Rmax) sup (X) - 5/2 s 2e (R
(27)
zeR
Along similar manner, we have
p p(If. wi (P(( (X)) (P))((X))
2)dr w ww-
t/2 s 2e
(28)
Combining 27 and 28
we obtain
P(VV - () ( wt (P(( (X)) > x)dr [ w=111 F1 (x))dz >e12)
+ P (15= w wPP( (X)) x)d2 - Jo w w011 (1- F, (x))dx > E//2
$ de - 222
where == max([Ttu" (Rmax)l )I [T_ (- (-Rmax)
Proof of Theorem 5 For given optimal policy x., define the normalized occupancy measure dg, (1-7) EeD o ((s,, = (s, a)). Note that d,, represents the stationary distribution. Additionally, given the assumption that the reward function R S x A IR is bijection, it follows that the distribution d,, (R- a)) and P, are identical. This indicates that the occurrence of black swan events can entirely characterized by the reward values, rather than the specific state-action pairs.
33

---

Published as conference paper ICLR 2025
Now, define the event Ess {R - [-mmax- Rb-l} where Pr. The probability of event Ebs happens is bounded as follows
P(Ena) F(-Rbb)) - F(-Rmax)
F(-Réb)
(( Rmax
Rbs
Ranax
Rbs
min
max cche
bs
2max
2maax
== [Pbs min max]
.Phs
Note that have assumed the 0 (r R(s,a)) < ehe and its minimum reachable probability cbs T2,
for all reward. now, for given trajectory, the reward instance is given as , ...,,,.... where ~ Fr, the probability that the agent first visit the black swan event at step h would be defined as
P (Fis"",Th-L Ensr Th E Ers) (1- (P(()))) h- ((Ere)
(
(1 Pmmia n-1
Pmax
Therefore, its probability is bounded follows,
(1 Pmax) Puuin S (fI,-",rhee Ph-1 & Ebs: Ph Ebs) S (1 Pmii) Pmmin)~11pp Pmax
Now, to ensure that the blackswan probability be lower bounded than 6, need the following conditions,
8 & (1 Pmax) h-1 Punin
log 8 s (h 1) log (1 - Penax) + log Pruin
Therefore, we have
h log (5/Pmin) / log(1 Penar) 1.
Therefore, we can conclude that if h- 2(oog (6/Pmin)/ log(1 (1- Pmma))) then the agent's probability to meet the black swan is at least 8.
C
H HELPFUL LEMMAS
Lemma 4. (Dvoretzky-Kiefer) Wollowwit (DKW) inequality) Let (u) = Σi (((())))sw) denote the empirical distribution of a V.V. U, with (X))...)....... ((x))) being sampled from the Ev u(X). The, for any and c > 0, we have
P(sup P(sup|F. (f())) F(x) e) s 2e 2ne² 2uc
zcR
34


Reviews:
[{"review_id": "4npvrjf6t2", "reviewer": "Reviewer_2mu5", "summary": "This paper challenges the conventional understanding of black swan events, which are typically seen as arising from unpredictable and dynamic environments. The authors propose that such high-risk, statistically rare events can also occur in static environments due to human misperception of events’ values and likelihoods, introducing the concept of S-BLACK SWAN. The paper categorizes black swan events, formalizes their definitions mathematically, and provides a framework for understanding and preventing these events through improved perception.", "questions": "- Can the authors provide empirical evidence or case studies that demonstrate the occurrence of S-BLACK SWAN events in real-world scenarios and model it using the proposed algorithm?\n- How can the proposed algorithm be applied to other domains beyond finance and autonomous systems? Are there specific examples or case studies in areas like healthcare or environmental science?\n- What are the next steps for developing algorithms based on the proposed hypothesis? Are there any preliminary results or ongoing projects that the authors can share?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper presents a novel hypothesis that black swan events can occur in static environments due to human misperception, which is a significant departure from the traditional view. This new perspective could open up fresh avenues for research in risk management and machine learning.\n- The theoretical framework is well-developed, with rigorous mathematical formalizations and proofs. The use of Markov Decision Processes (MDPs) to model human perception and misperception is particularly robust.\n- The paper is well-structured, with clear definitions and logical progression of ideas.\n- By redefining the origins of black swan events, the paper has the potential to significantly impact the fields of machine learning, risk management, and decision theory. It provides a foundation for developing algorithms that can better handle rare, high-risk events.", "weaknesses": "- The paper lacks empirical validation of the proposed hypothesis. While the theoretical framework is strong, it would benefit from experimental results or real-world case studies demonstrating the occurrence of S-BLACK SWAN events.\n- The mathematical formalizations, while rigorous, are quite complex and may be difficult for practitioners to apply directly. Simplifying some of the models or providing more intuitive explanations could enhance accessibility.\n- The paper primarily focuses on financial and autonomous systems. Expanding the discussion to other domains where black swan events are critical, such as healthcare or environmental science, could broaden the impact of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CxkAj8KDWM", "reviewer": "Reviewer_DuS7", "summary": "This paper introduces the concept of \"s-Black Swan\" - statistically rare, high-risk events that can occur in unchanging environments due to human misperception of event probabilities and values. Authors present a formal framework from to define this in the context of MDPs and argue that for safety in RL systems, it's important to consider stationary MDPs that present these Black Swan events.", "questions": "- Can you explain why the focus is on visitation probabilities v/s transition function? This is something that's not very intuitive. \n\n- Why is the value distortion function and the probability distortion function modeled in such a piece-wise way? Is this just for simplicity? How do we decide how to model them? \n\n- Example 2, case 3 - what does it mean for an MDP to be always a black swan? \n\n- What is the significance of Lemma 1? Can you describe how should one intuitively understand it? \n\n- Can you describe some applications where this will be useful? And how should one think about modeling such scenario - my understanding is that HEMDPs would be most appropriate for such situations.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper presents a new view point on how to look at Black Swan events. Specifically, it points to the case of stationary MDPs where agents have distorted perspective on reward signals and visitation probabilities which are likely to be overlooked by researchers. \n\n- The mathematical rigor is strong - the authors have done a great job at defining s-Black Swan using the existing concepts of MDP and its special cases. It gives a good framework for future researchers to build upon while trying to model such s-Black Swan events. Specifically, theorem 5 seems to be the most useful aspect of this paper which provides an analytical bound of encountering the rare event with event probability. \n\nParticularly, I liked the formulation of HEMDPs - seems to be a particularly useful modeling strategy.", "weaknesses": "- While the paper is very rigorous, the details might be very hard to follow for non-specialists. Some of the aspects are not very intuitive. \n\n- This paper lacks a practical application demonstration - it'll be great if the authors can describe how a practitioner can use the definitions that the paper provides for a practical applications. \n\n- Building upon the previous point - it'll be useful for us to understand how frequent are such MDPs where the users have a distorted view of the reward signals and the visitation probabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pROUXeeVAk", "reviewer": "Reviewer_WW6X", "summary": "This paper is a theory paper that challenges the view that black swan events only originate from changing (non-stationary) environments. Instead, the paper focuses on defining S-Black Swan events, which occur in unchanging environments due to human misperception of events’ values and probabilities. The paper is focused on formalizing the definition of S-Black-Swan events, by starting from Hypothesis 1 in the introduction.", "questions": "1. Are there any examples where a machine learning study/problem/algorithm would have benefited from the definition of a S-Black-Swan event?\n2. The probability distortion function seems like it would be an easier function to measure in practice compared to the value distortion function. Each individual might legitimately have different reasons to value outcomes differently. Going with Example 1, a loss of -1000 might be significantly worse for a poorer person than a richer person. It looks like the theory still holds when $\\epsilon_r = 0$, but $\\epsilon_d > 0$, but could the authors comment on that?\n3. If possible, could the authors provide additional context to the novelty of the work and how previous works have not considered an S-Black-Swan event, and attributed rare events to changing environments.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The main arguments of the paper are well-structured and well-communicated. The flow of the paper is helpful to the reader in communicating both the preliminary materials as well as leading to the mathematical formulation of the S-Black-Swan definition. For a theory paper, which could have the tendency to overcomplicate results, it feels like the authors have made significant effort to make the paper readable and therefore potentially meaningful to those who might use it as a future reference.\n* The definitions of the stationary ground MDP, the Human MDP, and the Human Estimation MDP form a clear picture of the potential Agent-Environment Framework that could lead to a S-Black-Swan event.\n* The presentation showing that black-swan events can arise in stationary environments is interesting.", "weaknesses": "* One weakness of the paper is in its ability to build a strong link between the results of the paper and how it might affect the wider machine learning community. This weakness can be broken down into a combination of the following:\n\t*  The related works section is left to the end and reads a bit like a list of works at the intersection of expected utility theory and reinforcement learning. The reader gets to the end of the section and is then told that this literature does not cover black swan events, but this statement lacks enough motivation.\n\t* The contribution of the paper is highlighted as defining an S-Black Swan event, but this contribution does not appear to be motivated by issues the current RL algorithms struggle within the literature. Every so often the paper includes comments like “aimed at guiding the design of safer ML algorithms in the future”; “laying the groundwork for future algorithmic design”; and “traditional risk criteria in RL are insufficient for managing the unique risks associated with black swan events”. While these comments might be the motivation of the paper, a more concrete motivation could be showing (or referencing) a specific RL (or ML) scenario which could benefit from this new definition and formalization of a S-Black-Swan event. This update seems like it would be important for an ICLR conference venue.\n\nTypos:\n* Line 108 $\\mathbf{p}_c = …$ should be 3 prob choices.\n\nComment:\n* Figures 1c and 1d could benefit from being moved further down in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rOPUXQyN6Z", "reviewer": "Reviewer_234c", "summary": "This paper proposes that rare and high-risk events, namely black swans, can originate from misperceptions of an event’s reward and likelihood even in static environments. The paper formalizes this concept in the context of MDP-based decision tasks using the machineary of cumulative prospect theory, where the misperceptions of rewards and transition functions in MDPs are characterized by distortion functions, resulting in a gap between the ground truth MDP and the MDP perceived by humans. The paper then theoretically analyzes the impact of black swans on the value function and the hitting time of black swan events.", "questions": "- Is the proof of Lemma 1 in the appendix correct? The lemma in the main text is stated for visitation probabilities, while the proof in the appendix seems to only deal with rewards. The authors could clarify if there is a connection between the reward-based proof and the visitation probability statement that is not immediately apparent.\n\n- Theorem 3 seems a quite general (and somewhat trivial) result to me: of course if we consider a sufficiently general case, the difference in MDP transition probabilities would result in different optimal policies. What is the concrete relation between this result and the s-black swan defined by the paper?\n\n- What is the role of Proposition 1？And isn't it contradictory to Algorithm 1 (Algorithm 1 defines an s-black swan for arbitrary $t$, while Proposition 1 considers $t$ in specific time intervals)?\n\n- How the definitions and theoretical results in the paper may be used to aid algorithm design?\n\n- Could the hitting time analysis in Theorem 5 be used to inform when to trigger perception updates?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main proposal of the paper, namely the black swan events can emerge in stationary environments, is interesting and makes sense to me. I also found the formalization using distortion functions on transition probabilities and rewards natural and captures the intuition of human irrationality.", "weaknesses": "In my opinion, the most important weakness of the paper is that apart from the main claim (black swans in unchanging environments), it is not clear to me what the main takeaways of the paper are. The authors wrote in the abstract \"We hope these definitions can pave the way for the development of algorithms to prevent such events by rationally correcting limitations in perception\", but it is not clear to me how the definitions and theoretical results presented in the paper can really benefit this: under the formalization of the paper, black swans in stationary environments essentially stem from the distorted transition and reward functions, which naturally create a gap between the ground truth MDP and the human MDP. The main theoretical result of the paper (Theorem 4) focuses on proving this gap in the context of value function estimation, yet such a result is quite straightforward and the techniques used in proving it (bounding the gap in value estimation given the gap in transition and reward functions) is also somewhat common in the RL literature. Perhaps more importantly, I feel that such a result (and also the result in Theorem 5) is not really useful to inspire algorithm design since it does not tell us _how_ to correct the misperceptions in the human MDP. Other results in the paper are also quite natural to me and seem not really tied to the specific black swan problem considered by the paper (see Questions for details). While the authors have extensively discussed in the appendix that existing solutions in safe RL may also fall short, only _defining_ such a problem is not enough to me for acceptance.\n\nI also have some concerns about the overall presentation of the paper. For example, the formal definition of s-black swan is deferred to Section 6 but heavily referred to in earlier sections, and I think it may be better to move this definition to earlier sections in the paper. I also do not understand the role of Section 3 beyond it re-emphasizes the main claim of the paper in Remark 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)