Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning

1 INTRODUCTION . Multi-agent reinforcement learning ( MARL ) has shown great utility in complex agent-based model ( ABM ) simulations , e.g. , in economics , games , and other fields ( Zheng et al. , 2020 ; Baker et al. , 2020 ; Leibo et al. , 2017 ; Yang et al. , 2018 ) . ABM is a field of research which has been found important and useful for quantifying the potential impacts of various policies in diverse real world applications ( eg . market ( Bozanta & Nasır , 2014 ) or environmental impact ( Raihanian Mashhadi & Behdad , 2018 ) simulations ) by examining real-world systems through simulation . In many of such simulations , the behavioral rules of agents may be too difficult for designers to specify . With MARL , designers instead specify the agents ’ objective functions and the reinforcement learning ( RL ) agents autonomously learn behaviors to optimize their objectives . However , this approach may be problematic when simulating systems of human agents , because in contrast to established models of human decision-making , it assumes that agents behave rationally and execute the objective-maximizing behavior . Prior literature demonstrates that using established models of human-irrationality during decision making yields results and implications that are significantly different from those obtained using rationality assumptions ( Sims , 2003 ; Maćkowiak & Wiederholt , 2011 ; Jiang et al. , 2019 ) . Therefore , it is important to account for this irrationality when simulating systems involving human ( -like ) agents . To this end , we introduce Rational Inattention Reinforcement Learning ( RIRL ) , a MARL framework with agents that can be rationally inattentive . Rational Inattention ( RI ) is a model of bounded rationality ( Sims , 2003 ) that is well established in behavioral economics ( Mackowiak et al. , 2021 ) . It attributes human irrationality to the costliness of mental effort ( e.g . attention ) required to identify the optimal action . Mathematically , the RI framework measures these costs as the mutual information ( MI ) between the variables considered by the decision process and the decisions ultimately made . This captures the intuition that a policy has a higher cognitive cost if its execution requires more information about the state of the world and thus more attention . MI-based rewards have been used in RL , ( e.g . Leibfried & Braun ( 2016 ) ; Leibfried et al . ( 2018 ) ; Grau-Moya et al . ( 2018 ) ; Leibfried & Grau-Moya ( 2020 ) use RI-style MI costs to regularize learning ) but not to model sub-optimal behavior . When used to model sub-optimal behavior , RIRL can rationalize seemingly sub-optimal behavior by including this cognitive cost in the reward function , i.e. , by adding the MI cost ( s ) . That is , the “ rational ” behaviors of an RIRL-actor can mimic human-like bounded rationality . Contributions . The main contributions of this work are : ( 1 ) we propose RIRL , a framework for modeling bounded-rationality which can analyze complex scenarios not previously possible ; and ( 2 ) we validate RIRL ’ s modeling capabilities in classical economic settings intractable using the analytical methods of prior work . To our knowledge , we are the first to study the effects of Rational Inattention in MARL-based simulations . RIRL extends the single-timestep framework proposed by Peng et al . ( 2017 ) , which decomposes decision-making into stochastic perception followed by stochastic action , each subject to its own MI cost . RIRL provides a novel boundedly-rational policy class that generalizes this to multiple information channels with heterogeneous costs , hidden-state policy models , and sequential environments . To achieve this increased modeling power , our architecture uses methods significantly different from prior work , such as using deep mutual information estimation modules to allow for multiple channels and using an LSTM to account for multiple timesteps . This allows RIRL to analyze settings with rich cognitive cost structures , e.g. , when information about state variables have different observation difficulty . For example , in hiring , a candidate ’ s past job performance may be more relevant but harder to evaluate than her employment history . We evaluate RIRL in two Principal-Agent ( PA ) problems , specifically manager-employee relations , where the Principal is boundedly rational . We use RIRL to model asymmetric information ( i.e . hard to obtain information ) between the Principal and the Agent they are incentivizing . Real-world PA experiments have shown that bounded rationality is key to explaining marked deviations between equilibria reached by human participants and theoretical predictions ( Erlei & Schenk-Mathes , 2017 ) . Prior work considered bounded rationality assumptions but found it to be analytically difficult ( Mirrlees , 1976 ) and further research in this direction is sparse We show that RIRL allows us to analyze generalized PA problems that are analytically intractable , such as a sequential PA problem with multiple Agents and heterogeneous information channels . Across all settings , we observe that equilibrium implications depends strongly on the cost ( s ) of attention and differs from that under rational assumptions sometimes in ways that are hard to intuitively predict . We observe that , depending on the channel , increasing Principal inattention can either increase Agent welfare due to increased compensation or decrease Agent welfare due to encouraging additional work . Compared to under rational assumptions , we additionally observe the emergence of different strategies , such as Agents choosing to misrepresent their ability , classically referred to as signaling ( Spence , 1973 ) . Our results show that RIRL can be a powerful tool to model boundedly rational behavior and analyze its emergent consequences in multi-agent systems . 2 RELATED WORKS . Multi-Agent Reinforcement Learning for Agent-Based Simulation . Agent-Based Models ( ABM ) are a popular tool use to study real-world systems ( e.g. , organizations or market systems ) and discover emergent behavior through simulation with fixed or manually-specified agent behaviors ( Bonabeau , 2002 ) . Instead , Multi-Agent Reinforcement Learning ( MARL ) -based simulations use RL agents which autonomously learn utility-maximizing behavior , so designers do not need to specify behavioral rules . MARL has been used to study tax policy ( Zheng et al. , 2020 ; 2021 ; Trott et al. , 2021 ) , games ( Baker et al. , 2020 ) , and social dilemmas ( Leibo et al. , 2017 ) among others ( Yang et al. , 2018 ) . However , RL agents are mostly assumed rational which contradicts how humans make decisions . Some recent work considers bounded rationality in MARL by accounting for cognitive limitations when reasoning about the behaviors of other agents ( Evans & Prokopenko , 2021 ; Wen et al. , 2019 ; Łatek et al. , 2009 ) . We model a complementary source of bounded rationality : the cognitive costs of attending to all available information when forming a decision . Models of Irrationality . Behavioral economics has shown extensively that human decision-making is not fully rational , but instead features many cognitive biases ( Caverni et al. , 1990 ) . Bounded rationality ( Simon , 1957 ) attributes these human irrationalities to resource limitations , e.g. , bounded cognitive capabilities or costliness of using ( more ) time to make decisions . Rational Inattention ( RI ) is a well-established model of bounded rationality which models the cognitive cost of decisions as the mutual information between variables relevant to the decision and the decision itself ( Sims , 2003 ; Mackowiak et al. , 2021 ) . RI has been tested in real world experiments ( Dean & Neligh , 2017 ) and used to model human behavior in a wide variety of domains ( Hoiles et al. , 2020 ; Mackowiak et al. , 2021 ) . Closest to our work , Jiang et al . ( 2019 ) study a multi-agent system for traffic route choice under RI but where the agents do not react to each other ’ s actions . Another line of work uses human behavioral data with deep learning to model human cognition ( Kubilius et al. , 2019 ; Battleday et al. , 2017 ; Ma & Peters , 2020 ) or predict human behavior ( Bourgin et al. , 2019 ; Kolumbus & Noti , 2019 ; Hartford , 2016 ) . In contrast , our ABM approach examines the implications of RI in complex settings where experimental data are presently unavailable . Mutual Information in Reinforcement Learning . MI has been extensively studied in the intrinsically motivated RL literature for curiosity-driven exploration and unsupervised skill and option discovery ( Still & Precup , 2012 ; Campos et al. , 2020 ; Mohamed & Rezende , 2015 ; Gregor et al. , 2016 ; Eysenbach et al. , 2019 ) , often with differing techniques used to measure MI . Recently MI-based rewards have been used to regularize exploration ( Grau-Moya et al. , 2018 ; Leibfried & Grau-Moya , 2020 ) . Comparatively , very little work has considered MI-based rewards for modeling boundedly rational behavior ( Ortega et al. , 2015 ; Peng et al. , 2017 ) , with no prior work , to our knowledge , considering the domain of multi-agent simulations . Principal-Agent Problems . In addition to our guiding example , Principal-Agent problems have been used to describe various settings , including politics and insurance ( Miller , 2005 ; Grossman & Hart , 1992 ) . Prior economics literature mainly use analytical methods and narrow modeling assumptions , e.g. , that all stochasticity follows Brownian motion ( Sannikov , 2008 ) or certain separability conditions are satisfied ( Grossman & Hart , 1992 ) . Generally , Principal-Agent settings fall under the category of Stackelberg games where it has been shown computing optimal strategies can be NP-hard ( Korzhyk et al. , 2010 ) . Instead , MARL can study complex setups that are analytically intractable . Shu & Tian ( 2019 ) ; Shi et al . ( 2019 ) ; Ahilan & Dayan ( 2019 ) studied coordinating cooperation in a Principal-Agent model , with an RL Principal learning to incentivize Agents to achieve an overall goal . However , they do not consider bounded rationality . 3 PRELIMINARIES . We consider multi-agent simulations in the form of partially-observable Markov Games ( MGs ) , formally defined by ( S , A , r , T , γ , O , I ) ( Sutton & Barto , 2018 ) . Here S is the state space of the game , A is the combined action spaces of the actors , and I are actor indices . We use oi = O ( s , i ) to denote the portion of the game state s that actor i can observe . In addition , oi may include a ( possibly learnable ) encoding of the observation history . Each game episode has a horizon of T ≥ 1 timestep ( s ) . Each timestep t , actor i selects action ai , t , sampling from its policy πi ( ai , t|oi , t ) . Given the sampled actions , the transition function T determines how the state evolves . Each actor ’ s objective is encoded in its reward function ri ( s , a ) , ( boldface denotes concatenation across actors ) . When modeling economic behavior , the reward is the ( marginal ) utility Ui ( st , at ) , an abstract measure of happiness . Each actor optimizes its policy to maximize its γ-discounted sum of future rewards . While RL can be used to discover ( approximately ) utility-maximizing policies , such rational behavior fails to account for characteristic irrational human behavior . Behavioral economic models often model irrationality as consequences of inattention ( Gabaix , 2017 ) . Rational Inattention formalizes this intuition using a modified objective that includes a cost to the mutual information I ( ai ; oi ) between the ( observable ) state of the world oi and the actions ai ∼ πi ( ·|oi ) . This definition captures the intuition that if the agent puts in more effort to pay attention to oi , its action ai likely becomes more correlated with the observation oi , and thus high MI . π†i = argmax πi ( Eπ [ Ui ( s , a ) ] − λI ( ai ; oi ) ) . ( 1 ) Note that this is equivalent to learning the optimal policy for an adjusted reward function : r†i ( st , at ) = Ui ( st , at ) − λĨ ( ai , t ; oi , t ) , where Ĩ ( ai , t ; oi , t ) = log p ( ai , t , oi , t ) − log p ( ai , t ) p ( oi , t ) is a Monte Carlo estimate of I ( ai ; oi ) , and λ is the utility cost per bit of information . Here p ( ai , oi ) , p ( ai ) , and p ( oi ) are the joint and marginal distributions over ai and oi induced by the environment and policies π . Below , we describe our method for estimating these quantities during training .

Summary: Standard Bayesian-type decision making models have that given some signal X, our decision makers choose an action A to maximize utility given the signal X. This gives a conditional distribution A|X. Rational inattention models say that the there is limted processing capacity in the agent and model this by requiring that A | X has a capacity constraint (i.e. which under some extra assumptions essentially becomes maximum amount of mutual information).   The authors put this into RL by, basically, writing a policy P(a|s) into an observation component (O(observation | state)) and an action conditional on the realized observation with an information penalty (so the higher capacity the channels, the more penalty).   The authors apply this RIRL model to some canonical game theory games.