Kati8/ICLR_2023_2025 huggingface dataset:
authors: Eric Mazumdar
authors: Kishan Panaganti
authors: Laixi Shi
reviews: 
Comments: The authors propose an algorithm for multi-agent game and reinforcement learning inspired by behavioral economics, with risk aversion and bounded rationality. They study a new class of equilibria called risk-averse quantal response equilibria (RQE), which is computationally tractable and with more human characteristics. Moreover, it provides an analysis of sample complexity for computing RQE in finite-horizon Markov games using generative models

Especially, these equilibria are independent of specific game structures and instead hinge on the degree of agents' risk aversion and bounded rationality. The authors also validate the effectiveness of RQE by showing its alignment with human behavior in experimental game studies.
Strengths: - The authors introduces games with risk-averse agents and bounded rationality, allowing for the computation of risk-averse quantal response equilibrium (RQE), which is realistic and computationally feasible.
- RQE can be computed in polynomial time in all $n$-player matrix games under specific risk aversion and bounded rationality conditions, irrespective of the underlying payoffs.
- The authors present a sampling-based algorithm with finite-sample guarantees for learning approximate RQE in unknown environments.
Weaknesses: - While the paper applies duality and risk aversion concepts, these techniques have been previously explored in the context of game theory and multi-agent reinforcement learning, raising questions about the technical novelty of the contributions.
- Extending the findings to finite-horizon Markov Decision Processes (MDPs) may be somewhat straightforward, as the process involves relatively conventional methods. While this extension adds scope, it may not significantly deepen the theoretical insights or complexity of the approach.
- The proposed algorithm requires access to a simulator, which is a strong assumption and may limit the practical applicability of the method, especially in real-world scenarios where such simulators are unavailable or costly to obtain.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper discusses multi-agent learning in a setting where the agents are risk-averse. By introducing risk aversion and quantal response into the agents' behaviour, the authors simplify computations to achieve a risk-averse quantal response equilibrium (RQE). This is accomplished via some clever tooling, applying a dual representation from [1] to convert the problem into a convex optimization problem under certain values of risk aversion. The trade-off is that it introduces n additional players into the game. The theory from the repeated game setting is then extended to the Markov game setting, where it establishes equivalence to the repeated game setting, quantifies the learning error, and implies convergence to RQE under specific parameter settings. A toy example is presented to convince the reader that the technology is functional.

[1] Föllmer, Hans, and Alexander Schied. "Convex measures of risk and trading constraints." Finance and stochastics 6 (2002): 429-447.
Strengths: - The authors perform a rigorous analysis and provide convincing arguments for a well-defined system.
- The work could have real economic applications.
- There is good novelty in applying the dual formulation of [1] to this multi-agent setting and constructing the loss functions in Eq. (6) and (7).
Weaknesses: - The notation could be a bit confusing at times; the difference between $\pi$ and $p_i$ is not clearly defined and can be hard for the reader to follow.
- Typo in line 160: "we make u a general class..."
- The experiments lack breadth in demonstrating the algorithm’s efficacy across diverse problem settings. Only one cliff walk example is demonstrated, and very standard simple benchmarks are used for the repeated game.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 4
reviews: 
Comments: The paper investigates incorporating risk aversion and bounded rationality in order to tractably compute risk-averse quantal response equilibria (RQE) for n-player matrix and finite-horizon Markov games. The paper proposes computing the RQE is advantageous because it is tractable under certain conditions independent of the game structure and it models human behavior better. The conditions for tractability of RQE is stated with respect to degree of risk aversion and degree of bounded rationality. The connection between using no-regret learning algorithms and computing the coarse correlated equilibrium (CCE) of the 2n-player game and computing the RQE of the n-player game is provided. The method and the definitions are extended to Markov games and the experiment results on a toy Cliff Walk grid-world game is given which support the claims of the paper.
Strengths: * Novel contribution while bringing a theoretically grounded view on risk aversion and bounded rationality to Markov games which can eventually be applied to more complex MARL scenarios. Adjusting the games so that RQE is achieved tractably by no-regret learning and having the tractability not depend on the underlying game structure is useful.
* Claims are supported extensively by proofs.
* The experiment on Cliff Walk demonstrates the risk averseness well.
* Paper is well written in general.
Weaknesses: * There is only one example game for Markov game results and it is very toy.
* Bounded rationality is less obvious than risk averseness on Cliff Walk experiments. Interpreting bounded rationality from the experiments is difficult.
* Paper has some minor typos (example line 160).
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 4