A Risk-Averse Equilibrium for Multi-Agent Systems
Authors: Oliver Slumbers
David Henry Mguni
Stephen Marcus McAleer
Jun Wang
Yaodong Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: game theory
safe game theory
risk averse game theory
safe equilibrium
population learning
game theory equilibrium

Abstract: In multi-agent systems, intelligent agents are tasked with making decisions that lead to optimal outcomes when actions of the other agents are as expected, whilst also being prepared for their unexpected behaviour. In this work, we introduce a novel risk-averse solution concept that allows the learner to accommodate low probability actions by finding the strategy with minimum variance, given any level of expected utility. We first prove the existence of such a risk-averse equilibrium, and propose one fictitious-play type learning algorithm for smaller games that enjoys provable convergence guarantees in games classes including zero-sum and potential. Furthermore, we propose an approximation method for larger games based on iterative population-based training that generates a population of risk- averse agents. Empirically, our equilibrium is shown to be able to reduce the utility variance, specifically in the sense that other agents’ low probability behaviour is better accounted for by our equilibrium in comparison to playing other solutions. Importantly, we show that our population of agents that approximate a risk-averse equilibrium is particularly effective against unseen opposing populations, especially in the case of guaranteeing a minimum level of performance, which is critical to safety-aware multi-agent systems.

Under review as a conference paper at ICLR 2023
A RISK-AVERSE EQUILIBRIUM FOR MULTI-AGENT SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
In multi-agent systems, intelligent agents are tasked with making decisions that lead optimal outcomes when actions the other agents are as expected, whilst also being prepared for their unexpected behaviour. In this work, we introduce a novel risk-averse solution concept that allows the learner to accommodate low probability actions by finding the strategy with minimum variance, given any level expected utility. We first prove the existence of such a risk-averse equilibrium, and propose one fictitionus-lay type learning algorithm for smaller games that enjoys provable convergence guarantees in games classes including zero-sum and potential. Furthermore, we propose approximation method for larger games based on iterative population-base averse agents. Empirically, our equilibrium shown be able reduce the utility variance, specifically in the sense that other agents' low probability behaviour is better accounted for by our equilibrium in comparison to playing other solutions. Importantly, we show that our population of agents that approximate risk-avers equilibrium particularly effective against unseen opposing populations, especially in the case of guaranteeing minimum level of performance, which is critical to safety-aware multi-agent systems.
training that generates a population of risk-
INTRODUCTION
Game Theory (GT) has become an important analytical tool in solving Machine Learning (ML) problems; the idea of "gamification" has become popular in recent years (Wellman) 2006 Lanctot
(et al. 2017) particularly
multi-agent systems research. The importance of risk- aversion in the
single-agent decision making literature ęZhang et al. 20200) Mihatsch & Neuneier 2002 Chow et al.)
2017) is obvious, whilst there still exist many open questions in the current game theory research domain. This paper aims to add to the current research in the multi-agent strategic decision-making literature based on the notion of risr-aversion through the lens of a new equilibrium concept.
One reason that avversin is important is that multi-agent interaction is rife with strategic uncer- tainty; this is because performance doesn't solely depend on ones own action. It is rarely the case that will have certainty over the execution and the strategy of the opponent in situations ranging from board games to economic negotiations (Calford, 2020). This presents a dilemma for autonomous decision-makers in human-Al interaction as one can no longer rely on perfect execution or complete strategy knowledge. Therefore, an important issue is what happens when actors take dangerous low probability actions such that could be considered as mistakes. These issues in play can arise in an array of circumstances, from misunderstandingg of reward structures to execution fatigue, leading to the execution of unexpected pure strategy. Hedging against unexpected play important for the agents as otherwise it can lead to large costs. As demonstrated in Fig. (I) a mistake in the execution of the pure-strategy Nash equilibrium (NE) could lead to both cars overtaking and crashing into each other, negative yet critical outcome in multi-agent system.
Traditional equilibrium solutions in GT (e.g. NE, Trembling Hand Perfect Equilibrium (THPE) (Bielefeld 1988)) lack the ability to handle this style of risk as either: 1) they assume strategies are executed perfectly, and/or, 2) large costs may be undervalued if there is a low probability attached to them. We address these by introducing new framework for studying risk in muuli--ageni systems through mean- variance analysis. In our framework, strategies are evaluated both in terms of expected utility against the opponent, but also the potential utility variance if the opponent played

---

Under review as a conference paper at ICLR 2023
Lane
Overtake
Stay Lane
5, 5
0 20
Streayy Equiibruum
Overtake
20,
0
50,
-50
Rok Avema Equilbruum
Figure I: Cars are rewarded for reaching their destination. They are behind slow tractors but can stay in their lanes and arrive safely, but slowly. They can overtake to arrive quickly, but if the other also overtakes they will crash, leading to large negative payoffs. The Overtake strategy is high-risk, high-reward and susceptible to errors, and is selected under a Nash equilibrium. The Stay in Lane strategy is low-risk, low-reward with low variance and selected by our mean-variance RAE approach.
low probability pure strategies. For example, the driving example in Fig. I describes a simple scenario where, due to the critical nature of wanting to avoid crashing, the benefits of overtaking may be entirely redundant with the possibility of low probability play leading to crashes. We summarise the contributions of our paper here:
We introduce a novel risk-averse equilibrium (RAE) based on mean-variane, components of the available strategies. Our framework generalises the single-agent mean-variance decision framework to multi-agent settings.
2. We show that the RAE always exists in finite games, and that it is solvable the class of games with the fictitious-plat property. This, as we later show, unlocks powerful array of computational methods for solving games.
3. We demonstrate that: 1) RAE is able to locate a minimum variance solution for any given level of utility 2) A by-product of RAE is that it can be used as a Nash equilibrium selection tool in the presence of a "risk-dominant" equilibrium 3) RAE is able find a low risk strategy in a safety-sensitiin autonomous driving environment.
2 RELATED WORK
There exists three relevant bodies of work, those works that empirically study the presence of risk- aversion in humans, those that aim develop new equilibrium frameworks and those that study how risk-averse agents alters classical game-theoretic results.
On the empirical side, the first paper to show that humans prefer to bet on known probability devices, rather than on other human choices, suggesting strategic uncertainty avoidance (Camerer Karjalainen] 1994). |Bohnet & Zeckhauser (2004) similarly found that subjects are more trusting in objective randomisation device rather than other humans. Eichberger et al. (2008) found that more trust placed in game theorists than "grannies" as the latter is source strategic ambiguity. Similar practices especially in the form of ambiguity aversion, such as those games outside of 3-color Ellsberg Urn tasks (Kelsey & Le Roux 2015), public goods and weakest link games (Kelsey & Le Roux 2017), in the presence of strategic complements and strategic substitutes (Kelsey & le Roux 2018).
noted in the game setting which more closely model multi-agent interactions,
11. For an
extensive survey of the experimental evidence, we refer readers to (Harrison & Rutström
2008).
The equilibrium literature can be divided into three distinct sections. Harsanyi et al. 1988) introduced
risk-dominant Nash equilibria (NE) (Nash] 1951). which suggests that increasing levels of strategic ambiguity will lead to the equilibrium with the lowest deviation losses. Risk dominance is limited as is restricted to the set of NE strategies. and therefore may be risk-dominant in comparison to other NE but not particularly risk-averse at all. [Bellceld (1988) set out the THPE which deals with strategic risk by accounting for off effilquuli play. However, this is sensitive to strictly dominated strategies and, because all trembles happen with marginal probability large downside risk can be masked, (In Fig.1 an error probability of 0.01 would only impact the utility function by 0.5, whereas we later show that our variance solution values the error at 47.6%, where 7 0 and non-marginal), which is problematic for safety-sensitin systems (e.g., autonomous driving). McKelvey & Palfrey (1995) utilises the Quantal Response Equilibrium (QRE) introduce errors into strategy selection, but with lower percentages on big mistakes which also discounts the impact of large downside risk. We argue that QRE undervalues big costs which particularly damaging in real-world settings, whereas
2

---

Under review as a conference paper at ICLR 2023
our mean-variance approach hedges away from high cost risk even in the presence of marginal probabilities. Yekkehkhany et al. (2020) utilises a similar mean-variance equilibrium concept based on risk derived from one-shot play in the probabilistic setting, rather than the expectation setting of this work. This does not apply more generally to the model-free machine learning setting where utility probabilities are not known and is therefore more difficult apply practically.
In terms of risk-aversion outside of equilibrium concepts, competitive network games Wardrop 1952) and the non expected utility maximising setting Fiat & Papadimitriou (2010) have been studied the most. Risk aversion in the network setting is based on generalisation of the classical selfish routing model [Beckmann et al. (1956) to incorporate uncertain delays. Nikolova & Stier- (2014) consider a mean-vanee -vrraacce framework for Wardrop equilibria in this setting,
Liancas ct al. 2019) extend this research to looking at how risk aversion degrades the performance of a routing system. Whilst the mean-variance approach is the same underlying notion as our work, we instead propose a solution for general games rather than routing games. In general games, FFat & Papadimitrioul 22010) remove the assumption of expectation maximisation and show that under risk averse utility functions there may exist no Nash equilibria. Further works have generally focused on Price of Anarchy Piliouras et al. 2016); Kesselhemm & Kodric (2018), which study how removing the assumptions of risk neutrality in general games impacts the difference between the achieved worst equilibrium and the maximum possible welfare of the system. Our work follows a similar strand in looking at general games, but focuses on defining a new equilibrium concept rather than establishing how risk-averse agents change the convergence properties classical equilibrium concepts. In addition, we frame our work such that it is more scalable for usage alongside RL techniques.
Our framework fits broadly into the areas of risk-sensitive RL ((Chow & Ghavamzadehi 2014
Keramati et al. 2020; Zhhag et al., (02Ib) and risk measures, such as mean-variance, value at risk (VaR) and conditional value at risk (CVaR), in RL ((Garcıa & Fernández 2015; Tang et al.) 2019]
Hiraoka et al.) 2019 Ma et al. 2020). The focus of risk-sensitive RL has remained predominately in single-agent settings where risk is due to the environment, rather than from other inhabitants of the environment. Multi-agent solutions include: RMIX (Qiu et al. (021) which optimises decentralised CVaR policies in cooperative risk-sensitive settings, RAM-Q and RA3-Q (Gao et al., 202T) which tackles algorithmic trading by utilising an adversarial approach to promote variance reduction, or risk-sensitive DAPG (Eriksson et al. 2022) which approaches risk in Bayesian games in terms of the CVaR induced by the possible combinations of types in the game. However, as we are specifically concerned with game-theue theoretic equilibrium concepts we will not directly compare to these methods.
Historically, the key challenge of computational GT is how solve for NE. For example, in two- player zero-sum games, it is theoretically possible to solve for an NE directly via linear programming (L.) (Morgenstern & Von Neumann 1953). Another approach to finding an equilibrium is the iterative method Fictitous Play (FP) (Brown, 1951). where players make best-responses to the time- average action of the opponent. However, in practice both the above approaches can be strictly intractable. Limitations due to action space size led to a general wave of methods that focus on starting with a "restricted" action space and iteratively expanding said space in order to approximate an equilibrium with the best possible strategies. Notably, Double Oracle (DO) (McMahan et al.) 2003; Dinh et al. M McAleer et al., (02T)) and Policy-Space Response Oracles (PSRO) (Lanctot et al. 2017 McAler et | 2020; Perez-Niever et al., 2021: Feng et al.) 2021) methods are the two major frameworks in this area. In this work we face a similar challenge in terms of the difficulty of solving for our own equilibrium. In this paper we demonstrate how FP and PSRO can be applied as a solver for our new equilibrium concept. In doing so, we provide a concrete methodology for obtaining solutions in setting. However, we must adapt them as they are generally designed for risk-neutral equilibria which is not the case for this work.
3 PRELIMINARIES & NOTATIONS
In this section, we introduce the preliminaries and notation required to understand our formulation.
A normal-form game (NFG) is the standard representation of strategic interaction in GT. A finite n- n-person NEG is a tuple (N, u). where N is a finite set of n players, A = A x, ..., AA" is the joint action profile, with A' being the strategies available to player i, and u (u ...., a²)) where u' : A- + RR is the valued expected utility function for each player. A player plays a mixed- strategy, a¹ E AA which is a probability distribution over their possible actions. In Sec. 6 we
3

---

Under review as a conference paper at ICLR 2023
replace our atomic pure strategies with neural network based strategies and therefore re-define notation to keep clarity between the two game schemes.
The central equilibrium concept in game-theory is the Nash equilibrium (NE), which is strategy profile where no players have an incentive to deviate. Let a-i E A- , be the pure strategy sets for all players other than i. Let (a,, a-;) be the expected utility function for player versus all players other than i, the strategy profile a" = (a, is a if,
Ue uz(a a 2 uz(as,
Va, E Ea
4 MEAN- VARIANCE EQUILIBRIUM
In the following section, we introduce our mean-variance based total utility function and then show how it can be used as an equilibrium concept. Our proposed variance method aims to deal with the main downside of QRE and THPE, that they both undervalue large downsides. For example, QRE designed that action probabilities are proportional to expected utility and THPE assigns 'error' probability to all zero- croo-prbbbbility actions. In both of these cases, variance from the average expected utility will provide a more pronounced effect risk than using the raw values.
4.1 UTILITY FUNCTION
Here we propose total utility function that measures both the expected utility, but also the potential variance of utility dependent on the opponent's strategy.
For simplicity we provide definitions based on playing symmetric game, such that two players share an action set A, utility function u. We extend this to the non-symmetric case in Appendix H. Define the expected utility of action O; E A against action a; E A as u(a,, a,) and the full expected utility table as M. where the entry M.j refers to u(a;, az) and M, refers to u(a,, aj) Vj. i.e. the vector of expected utilities that action ag receives against all other actions. We now define the expected utility of the mixed- mixed-strategy for player I versus the mixed strategy for player
u( a S, M) =
a(ass(aa))u(aaa) az) oT M. GLEAOEEA
ΣΣ
The weighted co-variance matrix for is a |A| x |A| matrix EM.s = [cty] with entries
Σ ((a;)(u(as;a)) ay) M,) (u(a;, - MR), 8.EA
Cjk=
(3)
where M: =
E (a is the weighted average expected utility for action i. This is standard co-variance matrix where the values for each action are weighted by the likelihood of them being selected by the opponent. A uniform weighting could be used, however we believe that in terms of utility variability avoidance is more intuitive to hedge against the variance caused by high likelihood actions. However, as will be discussed later, all actions will still receive positive probability under our framework and therefore will always provide some weight in the variance calculation, leading to low likelihood high- variance actions still having a large impact on the final result. This accounts for the idea that mistakes may happen such that all actions can be played with at least low probability. This allows us to define the mixed-strategy or variance utility as follows:
|AI [A]
IAI
|AI
[A]
Var(o, s. M)
Σ
(an)))
Σ
(aa))
Cu +
Σ
Σ
(an))n
(4)
n=1
n=b=
=oT EM.
The final total utility function r which considers expected and variance utility for strategy is,
r(o, s. M) = ula,5 M) Var(o, s, M),
(5)
where y E IR is the risk-aversion parameter. Applying Eq. (5) to Fig. we show how we arrive strategy profile that has desired properties. Consider two joint strategy profiles,
4

---

Under review as a conference paper at ICLR 2023
S = (1=
and Nash equilibrium S (()
where
(1-c,0+e) represents playing Stay in Lane with probability (1 - c). é is arbitrarily small and used to ensure fully mixed strategies, for the example we
0.01. Profile S receives u(S) 5 and
the Nash profile receives u(S2) 20. However, Var(S) = 0.32 and Var(S²) = 47.6, i.e. the Nash strategy has huge variance for Player 1. Therefore, r(Si) 5 0.32y and r(S2) 20 47.6y and
we have for any risk-aversion parameter > 0.32 it is optimal to play St.
4.2 EQUILIBRIUM CONCEPT
We now define our new equilibrium concept based on the total utility function 5 defining the best-response map:
First start by
(s) E arg max r(a, s,
s.t. (a) 2 0 Va A o² 1-
where due the quadratic term of E.Ms- and the constraints, have Quadratic Programme (QP), The above programme finds a such that the total utility is maximised, whilst ensuring no strategies assigned negative action probability, and that the action probabilities sum one. We now propose the following:
PROPOSITION 1. For any given expected utility ps: there exists y such that the solution to [ is the minimum variance solution.
We defer the proof to Appendix (A). This proposition implies that when using 6 given any expected utility value yas, there exists y that achieves μo, with the minimum possible variance. Therefore, y can be tuned to provide desired expected utility whilst giving the user the minimum viable variance solution. Therefore, based on Eq. (6). we define the equilibrium for the strategy profile o,
DEFINITION (Risk-Averse Equilibrium (RAE)). A strategy profile {a,s) s} is risk-averse equi- librium if both a and & are risk-averse best responses to each other.
Finally, a property of most game-ther theoreetc equilibria is that solution exists, at least in the finite game setting. For our equilibrium, we note the following result in mixed-strategies:
THEOREM 3. For any finite N-player game where each player i has a finite k number of pure strategies, A {af...., ..., d RAE exists.
We defer the proof of the result to Appendix (A). Importantly, Theorem B establisshe the existence of
solutions providing practical relevance for our equilibrium concept.
EQUILIBRIUM LEARNING VIA STOCHASTIC FICTITIOUS PLAY
We start by showing that our total utility function can be used a form of stochastic fictitious play (SFP) (Fudenberg & Kreps 1993) for finding an RAE small NFGs. SFP has convergence guarantees in selection of games, most notably potential games (Monderer & Shapley 1996a b) and finite two-player zero-sum games (Robinson) 195T). Furthermore, SFP is robust to games outside of the above game classes /Ganzfried/2000) 2020),, and we extend these observations in Appendix (B).
SFP describes a learning process where each player chooses a best response to their opponents' time-average strategies. In SFP, group of n 2 players repeatedly play n n--layer NFG. The state variable is Z, E As, whose components 2; describe the time averages of each player's behaviour,
a
where oj E A represents the observed strategy of player at time-step f. SFP process one where each player best responds to the time averaverage strategy of their opponent, Z, such that,
d' E arg max u'(a, Z M) (v
(7)
5

---

Under review as a conference paper at ICLR 2023
PSRO Outer Loop 43 (d) 03)
4
PSRO Inner Loop
Meta-Game ++
Ø
BR
M
M
RAE Solver
Best Response Oracle
a/()) argmax, náo Vario.g.M)
S BB
Covariance Matrix
Covariance Matrix
Figure 2: Iterative agent generation process. Note that u(.) and Var(-) are overloaded to represent utlity/variianc between distributions over a population or utility/variance between two policies.
vo)) A + R is strictly convex function, and the gradient of (arr becomes arbitrarily large near the boundary of the strategy simplex, i.e. -8 [v² = co. We propose the following with regards to our total utility function (Proofs are deferred to Appendix (A))
THEOREM 4. Given the total utility function Eq. there exist RAE convergence guarantees in the category of games that are solved by SFP.
SFP does not necessarily converge in all game classes (but is robust empirically). Therefore, show that if the SFP process does converge to a strategy then that strategy is guaranteed to be an RAE,
PROPOSITION 5. Suppose the SFP sequence {Ze} converges to a in observed strategies then o a risk-averse equilibrium.
Note that for SFP we require stronger notion of convergence in observed strategies o rather than in beliefs Zi, but the usage of a converged final of guarantees a risk-averse equilibrium.
6 EQUILIBRIUM LEARNING VIA ITERATIVE AGENT GENERATION
For games that can't be tractably displayed in the normal-form, which make use of reinforcement learning (RL) policies as proxies for actions. This approach aims approximate equilibria in large games by finding small representative collection of risk-averse policies which can instead be selected over by RAE. We provide a visualisation of the following iterative agent generation process Fig. 2 and provide algorithm in Appendix D.
iterative solution frameworks,
Consider two-player stochastic games G defined by the tuple {S, A, P, RJ. where S is the set of states, A A4' 2 is the joint action space. P S A - [o, 1] is the state-transition function and R 5 x A > IR is the reward function for player i. An agent is a policy Φ, where policy is mapping Φ S x A + [0, which can be described in both tabular form as a neural network. The expected utility between two agents is defined to be M( Φb) (i.e., in the same manner defined for NFGs in Sec. 4.1). and represents the expected utility to agent di against opponent dj.
Our iterative framework does T E N+ iterative updates a meta-game M (an NFG made up of RL agents as actions) following the framework of PSRO (Lanctot et al. 2017). At every iteration t t T. player is defined by population of fixed agents 2, Do Φ1. 22 .... $.), where Po is case where
the initial random agent. For notation convenience, we consider the dingle-populate players share the same Φ, As such, the population will generate meta game M,, an expected utility matrix between all the agents in the population, with individual entries M(Ø,, Φ3) Vo,, Φ5 E +
To make use of population & require way to select which agents di D will be utilised for training. This function f is a mapping f M. + [0, 17a which takes as input a meta meta-gams game M, and outputs meta-distribution a. ((M)). The output ar is probability assignment
each agent in the population 4 and, as we are in the single-pon -popullaptttiionn setting (i.e., symmetric play), we do not distinguish between populations. This the equivalent of mixed-strategy in a NFG, except now the
'Convergence in the time-average Z, does not imply convergence in the actual strategy taken each f, hut may for example imply cyclic actual hehaviour that results in average hchaviour converging.

---

Under review as a conference paper at ICLR 2023
Fictitious Play
PSRO
Expected Utility vs. Variance Utility -7
Expected Utility Variance Utility
Nusaa
*SdE Play
11
14
0.22
7
u.l
Expected Utility
Experted Utility
Risk De Deminant
12
12
"QR
- 0.23
10
0.3
.THPE
10
8
*Rask Deminore Eecs
7-003
--07
77=05
QRE
5
15
20
5
10 15
25 Varience Utility
20
30
Variance Utility
Figure 3: a) SFP on NFGs with 100 actions, b) PSRO on NFGs with 500 actions. Both compare final expected utility vs. variance utility results. We plot RAE values for multiple 7 to form 'efficient frontier' and show that, whilst baselines achieve similar expected utilities they are always finding solutions that are too high in variance utility. In Fig. a) we exclude the Payoff Dominant result provided huge final variance utility, whilst in Fig. b) we exclude the THPE result for the same reason.
actions are RL policies. We apply our risk-averse equilibrium concept (Def. 2 the meta-solver. As Φ are RI. policies then the policies are sampled by their respective probability in dz.
At each epoch the population . is augmented with new agent that is best-response to the meta-distribution dx. Generally, this will be purely in terms of the expected reward, and can be found with any optimisation process such Reinforcement Learning. In our setting there are two properties that we are concerned with when adding a new agent to the population. Notably, how it impacts the expected return but also how it impacts the variance utility of the population a EM.-I.a--. 0++1
To do this, we follow the PPO approach of (Zhang et al. 2020) that optimises both performance and per-step RL--reward variance. It is shown by (Bisi et al. 2019) that the variance of the per-step RL-reward bounds the variance of the total-RL. reward from above. Notably, the variance utility of population is measured in terms of the variance of the total RL-reward, and therefore shrinking the variance of the per-step RL-reward will also shrink the variance of the total RL-reward. To achieve this, an augmented MDP is used where the MeP-reward, gi is replaced as follows:
g = o >(gi)² + (2Ag(us)
where yi - - of is the average of the RL-rewards during the data collection phase. Notably, as this variance is also with respect to the sampling probability defined by o, this optimises the correct co-variance matrix which is similarly weighted by oz.
7 EXPERIMENTS
We validate the effectiveness of RAE on three environments that all display some risk component.
1. Randomly generated coordination games where some actions provide high expected utility if the other player selects the same action, but have large costs not. There also exist actions that have lower coordinated expected utility but lower costs. We conduct experiments testing SFP on games with 100 actions, and utilising our iterative approach on games with 500 actions. Vanilla policy gradient RL agents are used for the iterative approach.
2. A generalised grid- world stag- hagghunt (Peysakhovich & Lerer 2017) game that has a payoff- dominant and risk-dominant equilibrium. In this game it is not possible tractably to list out all actions and therefore our iterative approach is applied. PPO RL agents are used.
3. An autonomous driving environment (Leurent 2018) based on Fig. (I) for testing that our RAE Fig. T is attainable in an RL setting. PPO RL agents are used.
For SFP we select the baselines to be NE (including risk/dominant payoff NE), THPE (Bielefeld 1988) and QRE (McKelvey & Palfrey 1995). For our iterative experiments, select the baselines to be PSRO- (Nosh, Uniform, Self-Play, THPE, QRE) where the brackets refer to the meta-solver used. In the nopulation-basee setting believe it is fair to restrict baselines to algorithms that operate
7

---

Under review as a conference paper at ICLR 2023
Catch
Num. Plant Gather
1.00
0.09
PSRO-Unilom
1.24
PSRO-THPE 1.22
0.23
(Dun))
0.19
5.65 0.72
0.86
PSRO-QRE
0.51
0.11
28.11 2.13
4.42 0.98
1.09 0.21
Stag
On On-Equl Performance
<f RAF Populetion vs. Nesh Populltiio
Numiter of Stage Conglt
Namber of trow Cooggt Stog
Nambbrr of Plants Galienal
130
Tooal Vi00
T Valle
Tocal Vile
0.00
Tod Ejicals
Tea.
RAR
York
Figure 4: Stag-hunt environment results. a) Visualisation of the environment b) Results for intra- population play c) Results for RAE population vs. Nash population.
within this framework, and to not consider non-population risk-aversion algorithms and opponent modelling frameworks. In addition, our goal is introduce and evaluate new rame-theorete concept and therefore we believe the most natural comparisons are those from GT. We present our results in the form of answering three critical questions w.r.t the effectiveness of RAE.
Question 1. Do RAE solutions have similar expected utility whilst lowering variance?
We start by investigating performance randomly generated coordination NFGs. These NFGs are designed so that there are actions in the games that perform well if your opponent follows the same strategy but have large negative payofts otherwise. There also exist actions that perform worse (but still positively) when coordinated on, but also maintain better performance (albeit worse than coordination) when the opponent plays a different action. We provide an example Appendix F.
These games draw out potential pitfalls in current GT solution concepts (e.g. Nash) that focus on appealing coordination utility, without considering the variability of expected utility. We present our results in Fig. (3) where (a) represents games with 100 actions solved using SFP, and (b) represents games with 500 actions solved using our itcrative framework. We plot our RAE results across multiple values of in order to generate theoretical 'efficient frontier'. An efficient frontier shows for values of expected utility what are the minimum possible variance solutions that you can find to attain said expected utility. Our figures show that, whilst our baselines are able to achieve diverse range of expected utility values, they are unable to find the minimum variance solution which our RAE is able to find. We believe this shows the strong flexibility of our approach, in that it is able attain any utility reward that the baselines can achieve, whilst finding better solution in terms of variance.
Question 2. Can RAE act NE selection method?
A by-product of RAE is that it can be used as NE selection tool. We evaluate this in a generalised stag-hunt grid world (Peysakhovich Lerer 2017) where there exist both 'payoff-dominnnt and risk-ddmmnantt NEs. We provide full environment details in Appendix (F) and provide a visualisation in Fig. (4). There are two differing goals in the environment: 1) Collect plants on your own and receive low expected utility or 2) Hunt the stag and receive a large positive expected utility if the agents catch the stag together, and large negative expected utility only one agent hunts the stag. These are the tisk-ddminant" safe strategy and the 'payoff-dominan' risky strategy.
In Fig. ( we demonstrate how RAE can effectively act as a NE selection method. In Fig. (4p) we present the expected utility where each meta-solver population is trained against itself. Notably, the final strategy of all the baselines focus on capturing the stag, which is the risky payoff-domin dominntt strategy. However, our RAE instead finds the risk-dominan strategy in which focuses on gathering plants and not going after the stag. The impact of this is particularly noticeable when we place an RAE population and Nash population into the environment together co-players, shown in Fig.
8

---

Under review as a conference paper at ICLR 2023
Eqm Reward
Egyn Variance
Werst-Case
Nam. Grashes
Num. Arrivats
PSRO-Nush
0.86 + 2.84
1.51 0.24
.4,84 s 5.76
39.5 = 2.12
10.5 2.12
PSRO-Unifom
-0.69 1 0.87
1.70 0.09
-7.00 2:01
42 2.81
8.00 2,83
PSROO-THP
0.34 + 1.29
1.80 0.14
+5.32 + 3.40
41.5 2:16
8.50 2:12
1.60 + 0.97
1.44 0.13
-2.84 0.94
43 2.85
7,00 2.85
0.97 a 2.14
1.53 0.12
-4,80 a 5.92
38.5 4.96
11.50 = 4.96
PSRO
0.33 0.004
0.
5.5 0.71
46.00
Naan Heettrap
Heatmas
00
6
6
60
5.0
Coortiaate
X Ceorcmnuee
65
Figure 5: Results on autonomous driving environment. a) Results on 100 episodes over seeds b) Position heat-map for RAE solution e) Position heat-map for Nash solution.
(4p). The Nash population still attempts to hunt the stag, but in this case the RAE population is still gathering plants therefore leading to the Nash population being caught by the stag a numerous amount of times leading to very negative expected utility. This is an overall desirable property of RAE as it suggests, the case that RAE and NE overlap, RAE will find the risk-dominant strategy.
Question 3. In safety-sensitive environments what sort of strategy does RAF learn to follow?
Finally, how does RAE act in an environment where avoiding any large downside possibility is critical, for example autonomous driving. Our environment is modelled on the example in Fig. (T) where there exists two-way traffic with slow-moving vehicles and faster moving agents behind that may be interested in overtaking. From game-theoretic standpoint this is surprisingly difficult problem. NE prescribes that one agent overtakes and the other waits, which is a strategy that is exposed to errors and low probability play. We provide full environment details in Appendix (F).
In Fig. (5) we provide our results. In Table (a) we provide metrics where the average value is based off of 100 episodes in the environment and the standard deviation is based over 5 different training seeds. Firstly, we note that in terms of expected utility and variance-utio utility RAE outperforms the baselines, whilst also maintaining strong worst-case performance. Notably, RAE arrives a strategy that very rarely crashes, and nearly always arrives at the final destination. The same conclusion can not be drawn for any of the provided baselines.
To see why this is happening, in Fig. (4b) and (4c) we provide position heat-maps of the cars utilising the RAE strategy and the Nash strategy respectively. In the RAE heat-map one can see that the strategy taken is the safe strategy, i.e. follow behind until all vehicles in the on-coming lane have passed and then proceed to overtake. This strategy provides little expected utility for the vast majority of the episode, but remains sensitive to the risk-element of the environment which is our desired outcome. On the other hand, the Nash heat-map shows that the strategy is to overtake straight away and nearly always ends up in crash due to car congestion in the middle of the episode.
CONCLUSION
We introduce a new risk-averse equilibrium concept, RAE, based on mean-variance analysis. Theo- retically, we prove the existence and solvability of RAE and provide methods for arriving at an RAE in both small and large scale game settings. Empirically, we show that our RAE is able to locate minimum variance solutions for any expected utility, act as a NE selection method in the presence of risk-dominant equilibria and is effective at finding safe equilibrium in safety-sensitive autonomous driving environment. Avenues for future work should focus on the limitations of the current RAE approach, namely non-convergence guarantees in certain classes of games and the fact that RAE minimises upside and downside variance, where minimising downside variance only would be a desirable property.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Beckmann, Charles B McGuire, and Christopher B Winsten. Studies in the economics of transportation. Technical report, 1956.
Selten Bielefeld. Reexamination of the perfectness concept for equilibrium points in extensive games. In Models of Strategic Rationality, pp. 1-31. Springer, 1988.
Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, and Marcello Restelli. Risk-averse trust region optimization for reward-volatilty reduction. arXiv preprint arXiv: 1eXiv:1912.001 1912.0319,, 2019.
Iris Bohnet and Richard Zeckhauser. Trust, risk and betrayal. Journal of Economic Behavior & Organization, 35(4))467-4844 2004.
George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374-333 1951.
Evan M. Calford. Uncertainty aversion in game theory: Experimental evidence. Journal of Economic Behavior and Organization, 176:720 734, 2020. ISSN 0167-2681. doi: https://doi.org1100011111
.jeebo.2020.066 06.011 URL https: http:p:://www.co.. ott.:///..aaaoooaaooooo/ooaoaaaa http::/wwww.ceenceeeeeeeet..o
011/801672681200 980
Colin F Camerer and Risto Karjalainen. Ambiguity-aversie and non-additive beliefs in non- cooperative games: experimental evidence. In Models and experiments in risk and rationality, pp. 325-358. Springer, 1994.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. Advances in neural information processing systems, 27, 2014.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1))6070-6120, 2017.
Le Cong Dinh, Yaodong Yang. Zheng Tian, Nicolas Perez Nieves, Oliver Slumbers, David Henry Mguni, and Jun Wang. Online double oracle. arXiv preprint arXiv:2103.07788, 2021.
Jürgen Eichberger, David Kelsey, and Burkhard C Schipper. Granny versus game theorist: Ambiguity in experimental games. Theory and Decision, 64(2-3):333--333 2008.
Hannes Eriksson, Debabrota Basu, Mina Alibeigi, and Christos Dimitrakakis. Risk-sensitive bayesian games for multi-agent reinforcement learning under policy uncertainty, arXiv preprint 10045 2022.
Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. Neural auto-curricula in two-player zero-sum games. Advances in Neural Information Processing Systems, 34, 2021.
Amos Fiat and Christos Papadimitriou. When the players are not expectation maximizers. In International Symposium on Algorithmic Game Theory. pp. 1-14. Springer, 2010.
Drew Fudenberg and David M Kreps. Learning mixed equilibria. Games and economic behavior, (3):320-367, 1993.
Sam Ganzfried. Fictitious play outperforms counterfactual regret minimization. arXiv preprint arXiv:2001.11165, 2020.
Yue Gao, Kry Yik Chau Lui, and Pablo Hernandez-Leal. Robust risk-sensitive reinforcement learning agents for trading markets. arXiv preprint arXiv:2107.08083 2021.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-148) 2015.
Glenn W Harrisnn and Elisabet Rutström. Risk aversion in the laboratory. In Risk aversion in experiments. Emerald Group Publishing Limited, 2008
10

---

Under review as a conference paper at ICLR 2023
John C Harsanyi. Reinhard Selten, et al. A general theory of equilibrium selection in games. MIT Press Books, 1, 1988.
Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learn- ing robust options by conditional value at risk optimization. Advances in Neural Information Processing Systems, 32, 2019.
David Kelsey and Sara Le Roux. An experimental study on the effect of ambiguity in a coordination game. Theory and Decision, 19(4):667-688 2015.
David Kelsey and Sara Le Roux. Dragon slaying with ambiguity: theory and experiments. Journal of Public Economic Theory, 19(1):17--197, 2017.
David Kelsey and Sara le Roux. Strategic ambiguity and decision-making experimental study. Theory and Decision, 84(3):387-405) 2018.
Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning a evar policy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4436 4443, 2020.
Thomas Kesselheim and Bojana Kodric. Price of anarchy for mechanisms with risk-averse agents. arXiv preprint arXiv: 804.09468, 2018.
Mare Lanctot, Vinicius Zambaldi, Audrūnas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp.
2017.
Edouard Leurent. An enviromment for autonomous driving decision-making. https://github
com/ wew.euuenttiigggeeee env 2018.
Thanasis Lianeas, Evdokia Nikolova, and Nicolas E Stier-Moses. Risk-averse selfish routing. Mathematics Operations Research, 44(1):38 57,88-572700 2019.
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004. 2020.
Stephen McAleer, John Lanier, Roy Fox, and Pierre Baldi. Pipeline PSRO: A scalable approach for finding approximate nash equilibria in large games. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
Stephen McAleer, John Lanier, Kevin Wang, Pierre Baldi, and Roy Fox. XDO: double oracle algo- rithm for extensive-form games. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior, 10(1):6-38, 1995.
H Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by adversary. In Proceedings of the 20th International Conference Machine Learning (ICML-03), pp. 536-543, 2003.
Robert C. Merton. An analytic derivation of the efficient portfolio frontier. Journal of Financial and Quantitative Analysis, 744):1851-188 1972. doi: 10.2307/2329221.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49 (2):267-290, 2002.
Dov Monderer and Lloyd S Shapley. Fictitious play property for games with identical interests. Journal of economic theory, 68(1):258-265, 1996a.
Dov Monderer and Lloyd S Shapley. Potential games. Games and economic behavior, 14(1):124-14) 143, 1996b.
11

---

Under review as a conference paper at ICLR 2023
Oskar Morgenstern and John Von Neumann. Theory of games and economic behavior. Princeton university press, 1953.
John Nash. Non-cooperative games. Annals of mathematics, pp. 286-295, 1951.
Evdokia Nikolova and Nicolás E Stier-Moses. A mean-risk model for the traffic assignment problem with stochastic travel times. Operations Research, 62(2):366-388 2014.
Nicolas Perez-Nieves, Yaodong Yang, Oliver Slumbers, David H Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8514 PMLR, 18 24 Jul 2021. URL htttps://proceedinn ngs mlr.mmr...... press v pttpp//ess...e//eeeeeeeeeeeeeaaa html
Alexander Peysakhovich and Adam Lerer. Prosocial Iearning agents solve generalized stag hunts better than selfish ones, 2017. URL htttps://aaxi...tott wwwp:/www...aab/////aaaaa
02865
Georgios Piliouras, Evdokia Nikolova, and Jeff Shamma. Risk sensitivity of price anarchy under uncertainty. ACM Transactions on Economics and Computation (TEAC), 5(1):1-27, 2016.
Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang. Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich, Rmix: Learning risk-sensitive policies for cooperative reinforcement learning agents. Advances in Neural Information Processing Systems, 34:23049- 2021.
Antonin Raflin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 8, 2021. URL http: jmlr. http://iw...ro/ppppp////////-210
html
Julia Robinson. An iterative method of solving game. Annals of Mathematics, 54(2)229--311, 1951. ISSN 0003486X. (tL::tt:://wet jstor.toooooooooooo wtw.eorrrrrr/saabbee99933555
Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. arXiv preprint wwt.19911.0311.. 2019.
John Glen Wardrop. Road paper. some theoretical aspects of road traffic research. Proceedings of the institution of civil engineers, 1(3):325-62,, 1952.
Michael P Wellman. Methods for empirical game-theoretic analysis. In AAAI, pp. 1552-1556, 2006.
Ali Yekkehkhany, Timothy Murray, and Rakesh Nagi. Risk-averse equilibrium for games, 2020.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning. arXiv preprint arXiv:2004.10833 2020.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean- Mean-variance policy iteration for rvsk-averse rein- forcement learning. Proceedings the AAAI Conference on Artificial Intelligence, volume 35, pp. 10905-10913, 2021.
12

---

Under review as a conference paper at ICLR 2023
A FULL PROOFS
A.1 PROPOSITION [MINIMUM VARIANCE SOLUTION]
PROPOSITION The solution to the optimisation (6) provides the same solutions to the following:
0² E arg min T. E S
s.t. σ M o 2
(8)
(a) Va E A a'
where Hbb E R the lowest level of expected return that the actor is willing to accept.
Proof.
(Merton 1972) shows by a Lagrange multiplier argument that the optimisation problem,
a' E arg min o² Σm
s.t. o" M 2 1b
(a) 2 0 E
d'1=1
can be rewritten as
o" E arg min TT Em a - -T(OT M. M.s)
s.. ((a) 2 Va E A
(10)
0'1-1
which can be equivalently expressed as,
o² E arg min o M 5 No 2n o)
(11)
s.t. (()) 2 0 Va E A
d11=
where À =
A.2 THEOREM [RAE EXISTENCE]
THEOREM 3. For any finite N-player game where each player i has a finite l number of pure strategies, A {af....,a) ".). RAE exists
Proof. We base proof on Kakutani's Fixed Point Theorem
Lemma (Kakutani Fixed Point Theorem). Let be a non-empty subset of a finite
dimensional Euclidean space. Let f = A be correspondence, with A f(x) C A, satisfying the following conditions:
a compact and convex set.
2. ((x is non-empty for all x E A.
3. f(x) is convex-valued correspondence: for all A, f(x) is convex set.
13

---

Under review as a conference paper at ICLR 2023
4. (ax has closed graph: that is, if {x", y² + fx, y) with y" E f(="), then y E f(z).
Then, has a fixed point, that is, there exists some a E A. such that f(x).
We define our best-response function as Bi(o-:) arg ma: (a, o=:) where u, defined as in Eq. (5) and by definition must satisfy all of the properties of proper mixed-strategy and the best-response correspondence is B A A such that for all a E A, we have:
B = [Bi(a-i]kEN
(12)
We show that ((r) satisfies the conditions of Kakutani's Fixed Point Theorem
A compact, convex and non-empty.
By definition
(13)
where each A; = {al E; = 11 a simplex of dimension |A - 1, thus each A, is closed and bounded, and thus compact. Their product set also compact.
2. (o) is non-empty.
By the definition of (o where A; is non-empty and compact, and y² is quadratic and hence a polynomial function a. It is known that all polynomial functions are continuous, invoke Weirstrass's 's Extreme Value Theorem which states
Lemma. If a real valued-function f is continuous on the closed interval (a, b). then f must attain a maximum and minimum, each at least once. That is, there exist numbers and in Ja, b) such that:
f(c) 2 f(x) 2 f(d) Vz E la,
Therefore, A, is non-empty and compact and p² is continuous in a, Bi(o-i) is non-empty, and therefore B(or) is also non-empty.
3. B(ar) is a convex-valued correspondence.
Equivalently, B(a) C A is convex if and only B,(a-:) convex for
In order to show that B;(o-;) is convex for all i, we instead show that the Quadratic Programme defined by Eq. (6) special case of convex optimisation under certain conditions, and therefore by definition has a feasible set which is a convex set.
A convex optimisation problem is one of the form,
minimize
fo(æ)
s.t. f((x) 0, i == 1,..., af x b i i= 1,.....
(14)
where fo...., fm are convex functions. The requirements for a problem to be a convex optimisation problem are:
(a) the objective function must be convex
(b) the inequality constraint functions must be convex
(c) the equality constraint functions h₂(x) a/ x b; must be affine
We note that a quadratic form x" Ax is convex if A is positive semi-definite, and strictly convex A is positive definite (we can guarantee strict convexity by adding a small constant the diagonal of A without impacting the variance values). In our constrained optimisation, the quadratic term o' T Eo is always guaranteed to at least convex as Σ, the covariance matrix, is always at least PSD. Therefore, objective function is convex. Additionally, it
14

---

Under review as a conference paper at ICLR 2023
is easy to see that our inequality constraint functions are also convex and that equality constraint function affine. Therefore, our Quadratic Programme is an instance of a convex optimisation problem.
Importantly, the feasible set of a convex optimisation problem is convex, since it is the intersection of the domain of the problem
D
domfi-
(15)
which itself is a convex set.
Therefore, for all members of the feasible set æ; E (:(o-:) and all A [0, have that Or + - O)y E S and we have a convex-valued correspondence.
4. Bo has a closed graph.
Suppose obtain contradiction, that B( does not have closed graph. Then, there exists a sequence (a" 6²) + (o,, ④) with o" E B(o"), but ó & B( i.e. there exists some such that o; & Bi(o This implies that there exists some o E A and some e 0 such that
r: a-)) ri 3c.
(16)
By the continuity of r, and the fact that o², - aii have for sufficiently large n,
ri(o', 2 ri(o
(17)
and combining the preceding two relations we obtain
r. (o o",;"",)) > (,( + 2e 2 r.(o,,,, o",) +e
(18)
where the second relation follows from the continuity of r.. This contradicts the assumption that 6 E B(o",,) and completes the proof.
Therefore, (oo) satisfies the conditions of Kakutani's Fixed Point Theorem, and therefore if o B(o") then o" is an equilibrium.
A.3 THEOREM 4 [SFP CONVERGENCE]
THEOREM 4. Given the total utility function Eq. there exist RAE convergence guarantees the category of games that are solved by SFP.
Proof. We show that our utility measure can be embedded as a version of stochastic fictitious play and therefore be used find equilibrium in two-player zero-sum games and potential games.
smooth fictitious play procedure is one in which the best-response, B(ar), is derived from maximis- ing function of the form ro()) (u;(o;)) where,
1. v,(o,, A; + R is strictly convex function.
2. The gradient of (,()) becomes arbitrarily large near the boundary of the strategy simplex, i.e. lim,,--a,, (ox)) oo
which ensures that there exists unique solution to the best-response, and that all pure strategies receive strictly positive probability in the best-response.
We have shown that variance measure is strictly convex objective under the assumption that Σ, is positive-definite Therefore, we need to show that the gradient satisfies the boundary condition.
We start by showing that lim ||r,1 lim x if lim Fn
x,
15

---

Under review as a conference paper at ICLR 2023
Theorem. Let X and Y be normed spaces. If lim = x X and - Y is
continuous, then
lim (xxn) T(lim Fn)
Proof. Let > 0. As T' is continuous, by the epsilon-delta definition of continuous functions, there exists 8 > 0 such that,
| - yll < 8 +
<e
As lim Ty. 2, there exists no E N such that,
no 1 2|1 <
and it follows that,
n > no - [[T(sn)) - T(x))l <e
and thus
lim limT(rn)) = T(x) = T lim (z,)
Since,
T: X + x + |||||
continuous, we have lim ||,,1|| [immima x
Next, we show that our gradient has a lower bound that satisfies the boundary condition. Note for this proof we replace with x and E with Cov the proof relies upon the singular value decomposition and notation may become confusing.
Due to the symmetry of Cov, Vx Covx Cov 0A, limx-8A |||||| > +00.
W, and show that
lim |Xx|
lim |UEVTx]
x-00
x-0A
lim
||V x||
as U orthogonal
x-0A
lim
||E(V* x)||
x-0A
lim Σ 0: (V x);
where os is the i-th singular value
x-0A
> lim Jmin Σ 1(V x)d
x-
lim Mmin /minl|/Txxl x||
x-0A
lim mmin |x||
V is orthogonal
x-0A
Jmin lim [x|]
x-
Gmin
lim x due to Theorem 4
x5
At the boundary of the simplex, ie. utilising a pure strategy, this is the specific case of a mixed- strategy where only Dirac probability distributions can be used. Therefore, in the limit there is infinite
16

---

Under review as a conference paper at ICLR 2023
density upon the pure strategy at the edge of the simplex and have that limx-004 +00. We can replace this the above,
lim
[[
min
lim
x
x x-8
x X
min
(+x)
Cov is restricted positive-deffniteness, all singular values are strictly positive and we have the desired result
lim 00
|[
2
+0c
(19)
Therefore, our variance function is admissible as the perturbation function va(o;) in stochastic fictitious play, and retains convergence guarantees.
A.4 PROPOSITION [SFP IS RAE]
PROPOSITION 5. Suppose the SFP sequence {Z,) converges to in the observed stralegy sense then a is Risk-Averse equilibrium.
Proof. Assume the observed strategy has converged (a 2²) and that the strategy an RAE. This implies there exists some o.! such that:
(''
(20)
However, because a has converged then the SFP sequence { will also converge such that lim; ooe 7: and because we are in an SFP process it must be the case that:
o r'""", o- Yo e A
(21)
and therefore o¹ can not be best response
PCoveegennce in the time-average Z, does not imply convergence in the actual strategy taken each f, hut may for example imply cyclic actual hchaviour that results in average hchaviour converging.
17

---

Under review as a conference paper at ICLR 2023
B
SEP ROBUSTNESS
SFP Robustness Anti Cocrdination Games
Eucliccon Diteince betticon OEserves Actions
0.1.5
Gamma: 0.3
Gamma: 0.5
0.150
Gamma:
0.125
0.100
3.075
0.030
0.025
3.000
100
SFP Ittratioo
Figure 6: Euclidean distance between observed actions after each iteration on randomly generated anti-coordination games. A distance of ( implies that the process has converged.
Coordination
Eucticcan octance between Obeervr Actios
0.20
Gamma: 0.1
Gamma: 0.7
0.15
Gamma: 0.9
0.10
0.05
20
40
100
SFP Reration
Figure 7: Euclidean distance between observed actions after each iteration on randomly generated coordination games. A distance of ( implies that the process has converged.
18

---

Under review as a conference paper at ICLR 2023
Eudiidcan Disaancs between Oczerved Actens
Gamma: 0.1
0.150
0.125
Gammaa 1.0
Gammaa: 0.9
0.050
0.025
100
Figure 8: Euclidean distance between observed actions after each iteration on randomly generated games. distance of implies that the process has converged.
19

---

Under review as a conference paper at ICLR 2023
C
FIGURE 3 TRAINING CURVES
Squilleizn
Seatogy Mezas
Jecedferntann oodieno cooeooo your
Eggtladun Sowand
Stratogy Vienoor
PS30
Xinerrars Benanl
Prcbolalry Selacting Toilcy Action
Misiumn Rowad
Prodeieter of Scictiex Bake Strosge
Beoug
30
20
130
38
80
20
PSRO
Nad
BAX- 113
QRI
Fond
Jen Bamn
RAB
61
RAB 0.2
THPE
RiA Does Eyn
(b)
Figure 9: Training curves over multiple seeds for Figure 3.
20

---

Under review as a conference paper at ICLR 2023
D PSEUDO-CODE
Algorithm 1 Iterative RAE Solver
1: Initialise: the "high-level" policy set Ili $
2: for iteration {1, 2.... do:
3: for each player i E A do:
4: Compute meta-policy गH by SFP (Eq)
5: Find new policy by Oracle: of O' (
6: Expand & $ U {
7: Update meta-payoff M,+1.
8: Return: and +
21

---

Under review as a conference paper at ICLR 2023
E HYPERPARAMETER SETTINGS
Table 1: Hyper-parameter settings for our experiments.
SETTINGS
VALUE
DESCRIPTION
SEP COORDINATION GAMES
ACTION DIMENSION
100
NUMBER
STRATEGIES AVAILABLE
FP ITERATIONS
NUMBER OF FP BELIEF UPDATES
TREMBLE PROBABILITY
0.001
PROBABILITY OP TREMBLING ANOTHER STRATEGY
QUANTAL TYPE
SOFTMAX
TYPE OF QUANTAL RESPONSE EQUILIBRIUM
# OF SEEDS
50
# TRIALS
PSRO NFG COORDINATION GAMES
ORACLE METHOD
REINFORCE
SUBROUTINE GETTING ORACLES
ITERATIONS
NUMBER
ITERATIONS
ACTION DIMENSION
500
NUMBER OF PURE STRATEGIES AVAILARLE
LEARNING RATE
ORACLE LEARNING RATE
ORACLE EPOCHS
2000
ORACLE TOTAL EPOCHS
RAF GAMMA
0.1.0.5
VARIANCE AVERSION PARAMETER
METASOLVER METHOD
METASOLVER ITERATIONS
100
METASOLVER ITERATIONS
MV-PPO ZHANG
2020
PSRO ITERATIONS
10
NUMBER OF PSRO ITERATIONS
PPO HYPERPPRAMS
DEPAULT SB3
RATTIN ET AL.
2021
PPO HYPERPARANEE RAATEE VALUES
MV-PPO
RAE GAMMA
0.15
VARIANCE AVERSION PARAMETER
MFTASOLVER
SAP
METASOLVER
METASOLVER ITERATIONS
100
METASOLVER ITERATIONS
#
#
Wo-WAY ENVIRONMENT
MV-PPO
2020
PSRO ITERATIONS
NUMBER OF PSRO ITERATIONS
PPO HYPERPARAMS
DEFAULT SB3
RAFFIN
2021
PPO HYPERPARAMETES VALUES
PV.PPO VARIANCE AVERSION
U.S
PPO VARIANCE AVERSION PARAMETER
RAE GAMMA
0.5
VARIANCE
PARAMETER
METASOLVER
RAE SEP 100
METASOLVER METHOD
METASOLVER ITERATIONS
METASOLVER ITERATIONS
# OF SEEDS
# OF TRIALS
22

---

Under review as a conference paper at ICLR 2023
F ENVIRONMENTS
F.1 RANDOMLY GENERATED NFGS
We randomly generate coordination games with N actions in the following way:
Algorithm 2 Iterative RAE Solver
1: Initialise: Empty N x N payoff matrix P
2: for each action do:
3: Sample coordination element, Pú; ~ U(5, 15)
4: Set Payoff matrix element P.: lpal
5: if P(X I Pa) > 0.9 do
6: for all other actions j do
7:
Sample anti-coordination element Pij ~ 101, 15) Set Payoff matrix element P.j Ps+ = Pij
8:
9:
else do
10:
for all other actions j do
11:
Sample anticcoordinttion element Pij ((, 10) Set Payoff matrix element P.j = Pfs = Pij Return: P.
12:
A simple action example of a NFG generated following the above would be:
S
S2
S3
S
7,7
3,3
1,1
S2
4,4
9,9
2,2
S3
0,0
-4,-4
15,15
Low Risk, Low Reward
High Risk, High Reward
Figure 10: Game where one strategy (dotted outline) provides a high return assuming successful coordination but high variance in case the opponent does not coordinate correctly.
F.2 STAG HUNT GRID WORLD
Our stag-hunt environment is taken from (Peysakhovich & Lerer 2017) where slightly alter the parameters of the game. A 5 x 5 grid used with 2 players, I stag and plants randomly spawned in. The action set of the players is A (left, right, up, down). The stag every time-step will move one grid space closer to the closest player on the grid, the plants do not move.
There 3 different rewards signals in the game:
23

---

Under review as a conference paper at ICLR 2023
1. If player moves over plant they get 2 and the plant respawns elsewhere the grid.
2. If both players move over the stag at the same time both receive 5 and the stag respawns elsewhere on the grid.
3. If player moves over the stag on their own, the stag moves them on their own the player receives r = - and the stag respawns elsewhere on the grid.
4. Otherwise r 0.
F.3 AUTONOMOUS DRIVING ENVIRONMENT
Our driving environment is based on the two-way environment from (Leurent 2018) where we make modifications to the reward function to introduce a larger factor of risk aversion into the game. The goal of the controlled drivers is to reach the end of the road (the destination) whilst avoiding crashing and coming into too close contact with other vehicles. Slow moving drivers populate the roads moving at a constant speed 20.
There are four reward signals in the environment:
1. If the car crashes 22
2. If the car arrives at the destination = 2.
3. If the car travelling at a good speed ([25,30)), r = 0.2.
4. If the car comes very close to another car -0.1
5. Otherwise 0.
24

---

Under review as a conference paper at ICLR 2023
G COMPUTE ARCHITECTURE
All experiments run on one machine with:
AMD Ryzen Threadripper 3960X 24 Core
x NVIDIA GeForce RTX 3090
25

---

Under review as a conference paper at ICLR 2023
H ASYMMETRIC FORMULATION
In the following section we will show the formulation of Sec. 4 but for the asymmetric case.
H.1 UTILITY FUNCTION
Player i has an action set A., and utility function u, We define the on-equilibrium utility of action aF E Ar against action af E Aj as u(a a and the full utility table for player as M,, where the entry M refers to u(af, a and M refers to u(af, . Vk', i.e. the vector utilities that action a receives against all other actions. Take the 2-player case, we now define the utility of the mixed-strategy for player versus the mixed strategy for player 2
u(or)= Σ Σ
ouswu(af a = M
(22)
of EA
The weighted corvaniance matrix for the utility matrix M, is a |A, x [Aj] matrix Em. = [Ckke] with entries
|4
Ckkk'
Σ
S (u(af, ag)
(u(a ag) -M")
(23)
where M = MA
EtAal ouu(at az), i.e. the weighted average utility for action i. As we are trying to minimise variance with respect to the opponent strategy we used weighted covariance matrix such that potential variance caused by each action is weighted by its probability selection under the opponent strategy. As will be discussed later, all actions will receive positive probability under our framework and therefore will always provide some weight in the variance calculation. This allows us define the mixed mixed-strategy utility variance based as follows:
[4l Σ
|1 M1
Var(o, (o, M) = ΣΣ
Σ
TkOnCkn
of
Σ
Σ
JkTaCkn = =T Em, a.
(24)
n=1==
1=1
k=1n=kt
The final utility function which considers both on- and off-equilibrium utility for strategy or is,
r(or, 5) = o -M, -M.--- EM. -o).
(25)
where y E IR is the risk-aversion parameter.
H.2 EQUILIBRIUM CONCEPT
We now define our new equilibrium concept based on the utility function the best-response map:
First start by defining
(5) E arg max TT M 15-7700 Em, σ
S.t. o(a) 2 0, Va
(26)
1=
26

---

Under review as a conference paper at ICLR 2023
I QRE FAILURE CASE
In the following section we present results on the two-action driving game described in Sec. of the main article and displayed in Fig. 5.
Stay in Lane
Overtake
Stay
5
Pure Nach Equilbruum
Overtakk
20,0
-50,
Equibriuum
Figure 11: Two-action driving risk game.
We specifically utilise this game show failure case of QRE risk- sensisvnsitiee solution. Ideally, risk-sensitive solution concept would only play the Stay in Lane strategy as the Overtake strategy has far too high potential downside risk.
Equilibrium Reward
Strategy Variance
Minimum Reward
km
Reward
Reward
090
FP Iteratton
FP Iteration
FP Iteration
Figure 12: QRE and RAE results on two-action driving game.
As can be seen from the results in Fig. 6, for large sample of QRE hyperparameters the equilibrium found is high variance with potential poor downside performance. We believe this is because the very large costs the errors are easily picked up by variance analysis, but not so easily by the setup of QRE
27


Reviews:
[{"review_id": "uXfB3_N6VFx", "reviewer": "Reviewer_WyXC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper claims to be amongst the first paper's to consider the interplay of games and risk aversion in decision making and examines existence of a solution/equilibrium concept as well as methods for computing of approximating them.\n\n\"Despite the importance of risk-aversion in the single-agent decision making literature (Zhang et al., 2020; Mihatsch & Neuneier, 2002; Chow et al., 2017), it is surprising that this research idea is not lively in the current game theory research domain.\"\n\nThis is a patently false statement. Below, I put forward a number of recent papers in top game theory conferences and journals which are directly working on the intersection of game theory and risk averse agents. None of these papers are cited.\n\nThanasis Lianeas, Evdokia Nikolova, Nicolas E. Stier Moses. \n\"Risk-averse selfish routing\" \nMathematics of Operations Research, vol. 44(1), pp. 38-57, 2019.\n\nGeorgios Piliouras, Evdokia Nikolova and Jeff S. Shamma.\n\"Risk Sensitivity of Price of Anarchy under Uncertainty.\"  \nACM Transactions on Economics and Computation (TEAC), Volume 5, Issue 1, November 2016, Article No. 5.\n\nEvdokia Nikolova, Nicolas E. Stier Moses. \n\"The Burden of Risk Aversion in Mean-Risk Selfish Routing\" \nIn Proceedings of the Sixteenth ACM Conference on Economics and Computation (EC'15). Portland, OR, June 15-19, 2015.\n\nHota, Ashish R., Siddharth Garg, and Shreyas Sundaram. \n\"Fragility of the commons under prospect-theoretic risk attitudes.\" \nGames and Economic Behavior 98 (2016): 135-164.\n\nAngelidakis, Haris, Dimitris Fotakis, and Thanasis Lianeas. \n\"Stochastic congestion games with risk-averse players.\" \nInternational Symposium on Algorithmic Game Theory. Springer, Berlin, Heidelberg, 2013.\n\nFiat, Amos, and Christos Papadimitriou. \n\"When the players are not expectation maximizers.\" \nInternational Symposium on Algorithmic Game Theory. Springer, Berlin, Heidelberg, 2010.\n\nThis is a far from exhaustive list with several of these papers having dozens of citations. I believe the authors need to carefully examine the related literature and put their work in the right context.", "summary_of_the_review": "Interesting subject matter but the authors are clearly unaware of a lot of relevant work. The paper is not ready for publication without comparing/contrasting with these prior approaches.", "strength_and_weaknesses": "Strengths:\n1) The paper examines a very interesting setting that combines many agents (decision-making) as well as risk-averse agents (i.e. non-standard expected utility maximizers)\n\nWeaknesses:\n1) The authors are not aware of a lengthly line of research that examines answers to exactly these type of questions on the intersection of economics and computation. I do not believe that the paper should be considered for publication before its finding are put in their correct context.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sjZ3B7ojC6G", "reviewer": "Reviewer_PRwZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the new notion of risk-averse equilibrium (RAE) for multi-player games, which generalizes the single agent risk-averse mean-variance framework. RAE consists of an equilibrium point of the modified game where a trade-off term representing the variance of the rewards is added to the usual expected reward. Authors prove the existence of RAEs, as well as that RAEs retain the convergent guarantees of standard stochastic fictitious play. Moreover, for large games, a population based RL approach by (Lanctot et al., 2017) is adapted to compute RAEs. Finally, several experiments are performed showing that RAEs: 1) identify minimum variance solutions for any given level of utility 2) can be used as a Nash equilibrium selection tool in the presence of a \"risk-dominant\" equilibrium 3) are able to find a low risk strategies in a safety-sensitive environments such as autonomous driving.", "summary_of_the_review": "I find the paper well-motivated and the proposed notion of equilibrium relevant to the community (to the best of my knowledge, it is a novel notion). It is a very simple, yet powerful notion of risk-aversion in multi-agent scenarios, which comes with theoretical as well as practical advantages. Hence, I tend towards acceptance. However, I would be happy if the authors address the questions above. Moreover, I believe the paper would benefit from proofreading. ", "strength_and_weaknesses": "Strengths:\n- The notion of RAE, to the best of my knowledge, is novel and can be interesting to the community.\n- Conceptual benefits of RAEs are not only discussed but also supported by several experiments.\n\nWeaknesses/Questions: \n- The authors do not relate the proposed RAEs with its single-agent risk-averse counterparts. In which sense is this a generalization?\n- Why that particular choice of augmented MDP was used to approximate rewards variance? I guess this is the main difference w.r.t. the existing methods so I believe it deserves further explanations. \n- The experiments are performed with a sweep over the trade-off parameter $\\gamma$. However, in practice, one does not know a reasonable value of $\\gamma$ upfront and thus could reach low expected utilities as a consequence. How do you foresee $\\gamma$ to be chosen? Perhaps, an adaptive approach could be used throughout training? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ktMteZ_rp4", "reviewer": "Reviewer_1b8K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a risk-averse mean-variance equilibrium to multi-agent games, and the authors also showed the existence of such a risk-averse equilibrium. A fictitious play type of learning algorithm is proposed to solve for the equilibrium and a corresponding RL-based approximation algorithm enjoys good empirical performance on a set of real-world learning scenarios.", "summary_of_the_review": "The paper studies an important and relevant topic of multi-agent game learning with risk-averse equilibrium, and the authors proposed their algorithms for learning such equilibrium on real-world data. However, some of the theoretical guarantees for both the equilibrium and the algorithm are still lacking.\n\n===========================\n\nI tend to agree with the reviewers WyXC and UpJk that the paper requires more improvement at the current stage. Therefore, I lower my score to 5.", "strength_and_weaknesses": "**Strength:**\n- The authors considered an interesting and common risk-averse learning scenario where safety-aware strategies are desired. To the best of my knowledge, the case where the utility function measures the potential variance of the opponents' strategy has not been well-studied in the existing literature. Traditional methods, such as quantal response equilibrium, are not able to resolve such tasks perfectly.\n- A learning method based on the stochastic fictitious play is proposed for the equilibrium, and a theoretical guarantee is provided as long as the SFP sequence converges.\n- An RL-based learning algorithm is available to handle large games, where the empirical performance of the algorithm is promising. It offers a way to find min variance strategies for a given expected utility and select Nash equilibrium in the risk-dominant equilibria.\n\n**Weakness:**\n- The authors showed the existence of the mean-variance equilibrium, but the uniqueness of the risk-averse equilibrium is still lacking. Which equilibrium those proposed algorithms converge to is not clear to me.\n- Proposition 1 shows that there exists $\\gamma$ such that the solution is the min variance solution. My question is: how do we choose the parameter $\\gamma$ if we would like to prevent some extreme behavior to happen? This does not seem to be obvious for large games.\n- There are no theoretical guarantees for equilibrium learning with iterative agent generation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1kJo2eGCXm", "reviewer": "Reviewer_UpJk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper defines a notion of risk-averse equilibrium, develops some theory surrounding the notion, and runs some experiments illustrating that their notion achieves a better trade-off between utility and variance than other notions.", "summary_of_the_review": "An interesting premise, but lack of clarity and seemingly numerous critical typos made the paper impossible to parse for me. I would be willing to read a revised version that more clearly illustrates the main idea. ", "strength_and_weaknesses": "Broadly speaking, risk aversion is usually implemented by passing the utility function of the risk-averse player through a concave function, after which the player will become more risk-averse. That would be insufficient to handle the chicken game though-- a chicken game with utilities passed through a concave function is still a chicken game and will still have the \"undesirable\" equilibria. As such, I find the premise and motivating example of this paper--that there is some notion of equilibrium under which (Stay, Stay) is actually an equilibrium in a chicken game--interesting. \n\nUnfortunately, I cannot seem to understand the technical portion of the paper, including in the motivating example, due to what appear to be errors in the writing. Here are some concrete questions, in order of appearance in the paper.\n\n\n1. Eq (3-4), which seems to be absolutely critical to the paper, doesn't really make sense to me. It is also inconsistent with Eq (23-24) in Appendix H, and I'm not sure which one is the one intended by the authors.\n\n    **a)** The denominator $1 - \\sum_a \\varsigma(a)$ is always zero. I'm assuming that this should be $1 - \\sum_a \\varsigma(a)^2$, matching (23). \n\n    **b)** In the main body, I am assuming that the utilities $u(a_i, a_j)$ and $u(a_i, a_k)$ should instead be $u(a_j, a_i)$ and $u(a_k, a_i)$, matching (23).\n\n    **c)** I am assuming that, in the appendix, the exclusion of the subscript $\\boldsymbol\\varsigma$ is a mere notational oversight and not an indication that $\\mathbf{\\Sigma}_{\\mathbf{M}_1}$ does not actually depend on $\\boldsymbol\\varsigma$.\n\n    **d)** $\\mathbf{\\bar M}\\_i = \\frac{1}{|A|} \\sum_{k=1}^{|A|} \\sigma_k u_i(a_i, a_k)$ (immediately after Eq (3)): assuming $\\sigma_k$ should be $\\varsigma_k$\n    \n    **e)** In (4) and (24), the $\\sum_k \\sum_{n > k}$ term should be multiplied by 2.\n\n    With the above fixes, it seems that $\\boldsymbol \\sigma^\\top \\mathbf{\\Sigma}_{\\mathbf{M}, \\boldsymbol\\varsigma} \\boldsymbol\\sigma$ should be, roughly speaking, the variance that player 2 contributes to player 1's utility under profile $(\\boldsymbol\\sigma, \\boldsymbol\\varsigma)$, scaled by $1 / (1 - \\lVert \\boldsymbol\\varsigma \\rVert_2^2)$. Also, it is unclear to me what the intuition is behind the scaling factor. I would appreciate the authors explicitly including extra discussion surrounding the equilibrium notion.\n\n1. Even the fixed formulation would divide by zero when $\\lVert \\boldsymbol\\varsigma \\rVert_2^2 = 1$, i.e., when $\\boldsymbol\\varsigma$ is a pure strategy. In particular, that seems to happen in the example calculation at the end of Sec 4.1. The authors do write \"all actions will still receive positive probability in our framework\", but the example is clearly is not fully mixed. What is the calculation used in the example to conclude that (Stay, Stay) is an equilibrium? In any case, for the rest of my review I will assume that $\\boldsymbol\\varsigma$ is actually fully mixed and therefore there is no division by zero.\n\n1. The use of the variance in Eq (5), in particular instead of standard deviation, seems rather strange to me, as it implies that the resulting notion of equilibrium will not be scale-invariant: a player whose utilities are halved will suddenly find herself essentially less risk-averse, because the variance will be cut by 1/4. This feels like it requires further motivation.\n\n1. In Proposition 1, what is $u_b$? Eq (6) contains no $u_b$ nor an obvious place to insert one.\n\n1. Theorem 4's proof sketch in the paper body seems wrong: $\\text{Var}(\\boldsymbol\\sigma, \\boldsymbol\\varsigma, \\mathbf M)$ is quadratic in $\\boldsymbol\\sigma$, and therefore cannot possibly have arbitrarily large gradient at the boundary of a compact set (namely, a simplex). Also, I have no a priori reason to believe that the variance is *strictly* convex (as requried by SFP), merely convex. Am I missing something? \n\n\nRelatively minor things not affecting my score:\n\n1. Related work, second paragraph, \"suggesting strategic uncertainty is by\" - seems like an unfinished sentence.\n1. $\\sigma_k$ and $\\sigma(a_k)$ are both used to mean the probability that a mixed strategy $\\sigma$ assigns to action $k$. In the interest of cleanliness, I'd advise picking one of the two and sticking with it.\n1. Indices $i, j, k$ are used to index both players and actions at various points in the paper. This is unnecessarily confusing; I'd advise again using indices to mean only one thing. For example, $i, j$ can index players, and $a, a'$ can index actions.\n1. The theorem numbering in the appendix is inconsistent with the numbering in the main body. This is unnecessarily confusing. For example, Theorems 3 and 4 in the main body are proven as Theorems 4 and 5 in the appendix, respectively.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject