A Black Swan Hypothesis: The Role of Human Irrationality in AI Safety

Authors: Hyunin Lee, Chanwoo Park, David Abel, Ming Jin
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: AI Safety, Risk, Reinforcement Learning

Abstract:
Black swan events are statistically rare occurrences that carry extremely high risks. A typical view of defining black swan events is heavily assumed to originate from an unpredictable time-varying environments; however, the community lacks a comprehensive definition of black swan events. To this end, this paper challenges that the standard view is incomplete and claims that high-risk, statistically rare events can also occur in unchanging environments due to human misperception of their value and likelihood, which we call as spatial black swan event. We first carefully categorize black swan events, focusing on spatial black swan events, and mathematically formalize the definition of black swan events. We hope these definitions can pave the way for the development of algorithms to prevent such events by rationally correcting human perception.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729602459651

thread_type:
official_review

<thread_object>
creator: reviewer_234c

<content>
summary:
This paper proposes that rare and high-risk events, namely black swans, can originate from misperceptions of an event’s reward and likelihood even in static environments. The paper formalizes this concept in the context of MDP-based decision tasks using the machineary of cumulative prospect theory, where the misperceptions of rewards and transition functions in MDPs are characterized by distortion functions, resulting in a gap between the ground truth MDP and the MDP perceived by humans. The paper then theoretically analyzes the impact of black swans on the value function and the hitting time of black swan events.

strengths:
The main proposal of the paper, namely the black swan events can emerge in stationary environments, is interesting and makes sense to me. I also found the formalization using distortion functions on transition probabilities and rewards natural and captures the intuition of human irrationality.

weaknesses:
In my opinion, the most important weakness of the paper is that apart from the main claim (black swans in unchanging environments), it is not clear to me what the main takeaways of the paper are. The authors wrote in the abstract "We hope these definitions can pave the way for the development of algorithms to prevent such events by rationally correcting limitations in perception", but it is not clear to me how the definitions and theoretical results presented in the paper can really benefit this: under the formalization of the paper, black swans in stationary environments essentially stem from the distorted transition and reward functions, which naturally create a gap between the ground truth MDP and the human MDP. The main theoretical result of the paper (Theorem 4) focuses on proving this gap in the context of value function estimation, yet such a result is quite straightforward and the techniques used in proving it (bounding the gap in value estimation given the gap in transition and reward functions) is also somewhat common in the RL literature. Perhaps more importantly, I feel that such a result (and also the result in Theorem 5) is not really useful to inspire algorithm design since it does not tell us _how_ to correct the misperceptions in the human MDP. Other results in the paper are also quite natural to me and seem not really tied to the specific black swan problem considered by the paper (see Questions for details). While the authors have extensively discussed in the appendix that existing solutions in safe RL may also fall short, only _defining_ such a problem is not enough to me for acceptance.

I also have some concerns about the overall presentation of the paper. For example, the formal definition of s-black swan is deferred to Section 6 but heavily referred to in earlier sections, and I think it may be better to move this definition to earlier sections in the paper. I also do not understand the role of Section 3 beyond it re-emphasizes the main claim of the paper in Remark 1.

questions:
- Is the proof of Lemma 1 in the appendix correct? The lemma in the main text is stated for visitation probabilities, while the proof in the appendix seems to only deal with rewards. The authors could clarify if there is a connection between the reward-based proof and the visitation probability statement that is not immediately apparent.

- Theorem 3 seems a quite general (and somewhat trivial) result to me: of course if we consider a sufficiently general case, the difference in MDP transition probabilities would result in different optimal policies. What is the concrete relation between this result and the s-black swan defined by the paper?

- What is the role of Proposition 1？And isn't it contradictory to Algorithm 1 (Algorithm 1 defines an s-black swan for arbitrary $t$, while Proposition 1 considers $t$ in specific time intervals)?

- How the definitions and theoretical results in the paper may be used to aid algorithm design?

- Could the hitting time analysis in Theorem 5 be used to inform when to trigger perception updates?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730466503349

thread_type:
official_review

<thread_object>
creator: reviewer_ww6x

<content>
summary:
This paper is a theory paper that challenges the view that black swan events only originate from changing (non-stationary) environments. Instead, the paper focuses on defining S-Black Swan events, which occur in unchanging environments due to human misperception of events’ values and probabilities. The paper is focused on formalizing the definition of S-Black-Swan events, by starting from Hypothesis 1 in the introduction.

strengths:
* The main arguments of the paper are well-structured and well-communicated. The flow of the paper is helpful to the reader in communicating both the preliminary materials as well as leading to the mathematical formulation of the S-Black-Swan definition. For a theory paper, which could have the tendency to overcomplicate results, it feels like the authors have made significant effort to make the paper readable and therefore potentially meaningful to those who might use it as a future reference.
* The definitions of the stationary ground MDP, the Human MDP, and the Human Estimation MDP form a clear picture of the potential Agent-Environment Framework that could lead to a S-Black-Swan event.
* The presentation showing that black-swan events can arise in stationary environments is interesting.

weaknesses:
* One weakness of the paper is in its ability to build a strong link between the results of the paper and how it might affect the wider machine learning community. This weakness can be broken down into a combination of the following:
	*  The related works section is left to the end and reads a bit like a list of works at the intersection of expected utility theory and reinforcement learning. The reader gets to the end of the section and is then told that this literature does not cover black swan events, but this statement lacks enough motivation.
	* The contribution of the paper is highlighted as defining an S-Black Swan event, but this contribution does not appear to be motivated by issues the current RL algorithms struggle within the literature. Every so often the paper includes comments like “aimed at guiding the design of safer ML algorithms in the future”; “laying the groundwork for future algorithmic design”; and “traditional risk criteria in RL are insufficient for managing the unique risks associated with black swan events”. While these comments might be the motivation of the paper, a more concrete motivation could be showing (or referencing) a specific RL (or ML) scenario which could benefit from this new definition and formalization of a S-Black-Swan event. This update seems like it would be important for an ICLR conference venue.

Typos:
* Line 108 $\mathbf{p}_c = …$ should be 3 prob choices.

Comment:
* Figures 1c and 1d could benefit from being moved further down in the paper.

questions:
1. Are there any examples where a machine learning study/problem/algorithm would have benefited from the definition of a S-Black-Swan event?
2. The probability distortion function seems like it would be an easier function to measure in practice compared to the value distortion function. Each individual might legitimately have different reasons to value outcomes differently. Going with Example 1, a loss of -1000 might be significantly worse for a poorer person than a richer person. It looks like the theory still holds when $\epsilon_r = 0$, but $\epsilon_d > 0$, but could the authors comment on that?
3. If possible, could the authors provide additional context to the novelty of the work and how previous works have not considered an S-Black-Swan event, and attributed rare events to changing environments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731034731297

thread_type:
official_review

<thread_object>
creator: reviewer_dus7

<content>
summary:
This paper introduces the concept of "s-Black Swan" - statistically rare, high-risk events that can occur in unchanging environments due to human misperception of event probabilities and values. Authors present a formal framework from to define this in the context of MDPs and argue that for safety in RL systems, it's important to consider stationary MDPs that present these Black Swan events.

strengths:
- This paper presents a new view point on how to look at Black Swan events. Specifically, it points to the case of stationary MDPs where agents have distorted perspective on reward signals and visitation probabilities which are likely to be overlooked by researchers. 

- The mathematical rigor is strong - the authors have done a great job at defining s-Black Swan using the existing concepts of MDP and its special cases. It gives a good framework for future researchers to build upon while trying to model such s-Black Swan events. Specifically, theorem 5 seems to be the most useful aspect of this paper which provides an analytical bound of encountering the rare event with event probability. 

Particularly, I liked the formulation of HEMDPs - seems to be a particularly useful modeling strategy.

weaknesses:
- While the paper is very rigorous, the details might be very hard to follow for non-specialists. Some of the aspects are not very intuitive. 

- This paper lacks a practical application demonstration - it'll be great if the authors can describe how a practitioner can use the definitions that the paper provides for a practical applications. 

- Building upon the previous point - it'll be useful for us to understand how frequent are such MDPs where the users have a distorted view of the reward signals and the visitation probabilities.

questions:
- Can you explain why the focus is on visitation probabilities v/s transition function? This is something that's not very intuitive. 

- Why is the value distortion function and the probability distortion function modeled in such a piece-wise way? Is this just for simplicity? How do we decide how to model them? 

- Example 2, case 3 - what does it mean for an MDP to be always a black swan? 

- What is the significance of Lemma 1? Can you describe how should one intuitively understand it? 

- Can you describe some applications where this will be useful? And how should one think about modeling such scenario - my understanding is that HEMDPs would be most appropriate for such situations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731120777186

thread_type:
official_review

<thread_object>
creator: reviewer_2mu5

<content>
summary:
This paper challenges the conventional understanding of black swan events, which are typically seen as arising from unpredictable and dynamic environments. The authors propose that such high-risk, statistically rare events can also occur in static environments due to human misperception of events’ values and likelihoods, introducing the concept of S-BLACK SWAN. The paper categorizes black swan events, formalizes their definitions mathematically, and provides a framework for understanding and preventing these events through improved perception.

strengths:
- The paper presents a novel hypothesis that black swan events can occur in static environments due to human misperception, which is a significant departure from the traditional view. This new perspective could open up fresh avenues for research in risk management and machine learning.
- The theoretical framework is well-developed, with rigorous mathematical formalizations and proofs. The use of Markov Decision Processes (MDPs) to model human perception and misperception is particularly robust.
- The paper is well-structured, with clear definitions and logical progression of ideas.
- By redefining the origins of black swan events, the paper has the potential to significantly impact the fields of machine learning, risk management, and decision theory. It provides a foundation for developing algorithms that can better handle rare, high-risk events.

weaknesses:
- The paper lacks empirical validation of the proposed hypothesis. While the theoretical framework is strong, it would benefit from experimental results or real-world case studies demonstrating the occurrence of S-BLACK SWAN events.
- The mathematical formalizations, while rigorous, are quite complex and may be difficult for practitioners to apply directly. Simplifying some of the models or providing more intuitive explanations could enhance accessibility.
- The paper primarily focuses on financial and autonomous systems. Expanding the discussion to other domains where black swan events are critical, such as healthcare or environmental science, could broaden the impact of the work.

questions:
- Can the authors provide empirical evidence or case studies that demonstrate the occurrence of S-BLACK SWAN events in real-world scenarios and model it using the proposed algorithm?
- How can the proposed algorithm be applied to other domains beyond finance and autonomous systems? Are there specific examples or case studies in areas like healthcare or environmental science?
- What are the next steps for developing algorithms based on the proposed hypothesis? Are there any preliminary results or ongoing projects that the authors can share?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731985631034

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Main Message

comment:
We would like to sincerely thank all the reviewers for their thorough and insightful feedback! We hope that this response effectively delivers the main message of our draft and contributes to a clearer and more comprehensive understanding of our work.

**Our central argument highlights that if black swan events can also arise from human misperception, AI safety algorithms should broaden their focus beyond solely developing accurate predictive models for rare events.** Specifically, they should also aim to address and correct biases in data generated by human behavior, which are shaped by such misperceptions.  This perspective emphasizes the importance of prioritizing **not only algorithmic design**, which is the current standard approach in AI safety, but also the mitigation of human-induced biases in the **data**. We believe that introducing a new direction for AI safety algorithms - one that has not been widely explored - is vital for the AI community. A deeper understanding of the origins of black swans paves the way for achieving greater control and making more informed, optimal decisions.


We appreciate that this paper primarily focuses on definitions and offers a viewpoint that we believe is essential for guiding the direction of future AI safety algorithm design, rather than proposing specific algorithms to correct human misperception. ***Its central goal is to establish a foundational understanding of how black swans may be interpreted as a result of human misperception.*** While we acknowledge that the paper may not meet the standard for acceptance if it does not provide sufficient evidence to support this interpretation in the context of human perception, we also see significant potential in its contributions.

If the paper presents a well-reasoned perspective on the possibility of black swans occurring in stationary environments due to human misperception, supported by convincing evidence, we believe it could positively influence the AI community. Specifically, it could inspire efforts to develop AI safety algorithms that address human-biased data, moving beyond a sole focus on creating accurate forecasting algorithms for black swans. Exploring methods for bias correction or algorithm development would be a natural and meaningful extension of this work, although we recognize that such efforts fall outside the immediate scope of this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732232651499

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Experiments to Support Our Main Message

comment:
Dear Reviewers 2mu5, DuS7, WW6X, and 234c,  

We hope our comments address the reviewers' concerns! During the active discussions with Reviewer 234c, who raised *a very insightful question* to support the clarity and significance of this paper’s contribution, we focused on the following key question:  

**"How does CPT-characterized distortion provide an additional meaning to the suboptimal gap?"**  

To address this question, we conducted **simulation experiments in Gridworld.** We kindly recommend the reviewers refer to the section titled:  "[Weakness 1] S-black swan is just an emergence from a non-zero model error from true P and R, so the theoretical analysis is trivial, making it indistinguishable from prior RL theory literature focusing on PAC bound analysis via model estimation - (Additional Answer 2): Experiment to support our answer for [Weakness 1]"

This discussion is included in the **discussion panel with Reviewer 234c.**  

Through the experiment, we concluded that:  

**"CPT-distortion provides an intrinsic motivation (or helps shape a beneficial intrinsic reward) for the agent to discover the optimal policy, potentially outperforming that of the real-world scenario."**  

These experimental results strongly support our main message: Interpreting the risk by human misperception reveals novel phenomena, underscoring the need for in-depth scrutiny in future algorithm design.

We hope our experiment results address the concerns of reviewers who were interested in seeing simulation results.  Thank you again for your thoughtful feedback and active engagement. We welcome any further questions or discussions!  

Best regards,  
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
Thank you for your submission to ICLR. This paper introduces the S-Black Swan (i.e., black swan events in stationary environments), which provides formalism for the setting where black swans occur in unchanging environments due to a misperception of events’ probabilities and values. This is in contrast to the typical assumption that a black swan event occurs in an unpredictable and dynamic environment. The authors define S-Black Swan in the context of a Markov Decision Process (MDP).

Reviewers agreed that the paper was clearly written, provides a novel contribution, and has strong mathematical formalism (which captures intuition well). However, there were concerns from multiple reviewers about whether this paper was well-enough motivated from problems in machine learning, and in general whether it has enough connection to the broader machine learning community—and wanted a clear explanation of how results from this paper can benefit this community. Also, multiple reviewers requested an empirical/practical application demonstration to make this connection more clear. In their rebuttal, the authors put a lot of effort into justifying this connection, and giving concrete demonstrations of applications, along with an experimental result. On balance, a majority of reviewers updated their scores by the end of the rebuttal period, and felt that their concerns were addressed. I therefore recommend this paper for acceptance, and encourage the authors to keep this feedback in mind when putting together the camera-ready version of their paper.