Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning

1 INTRODUCTION . Multi-agent reinforcement learning ( MARL ) has shown great utility in complex agent-based model ( ABM ) simulations , e.g. , in economics , games , and other fields ( Zheng et al. , 2020 ; Baker et al. , 2020 ; Leibo et al. , 2017 ; Yang et al. , 2018 ) . ABM is a field of research which has been found important and useful for quantifying the potential impacts of various policies in diverse real world applications ( eg . market ( Bozanta & Nasır , 2014 ) or environmental impact ( Raihanian Mashhadi & Behdad , 2018 ) simulations ) by examining real-world systems through simulation . In many of such simulations , the behavioral rules of agents may be too difficult for designers to specify . With MARL , designers instead specify the agents ’ objective functions and the reinforcement learning ( RL ) agents autonomously learn behaviors to optimize their objectives . However , this approach may be problematic when simulating systems of human agents , because in contrast to established models of human decision-making , it assumes that agents behave rationally and execute the objective-maximizing behavior . Prior literature demonstrates that using established models of human-irrationality during decision making yields results and implications that are significantly different from those obtained using rationality assumptions ( Sims , 2003 ; Maćkowiak & Wiederholt , 2011 ; Jiang et al. , 2019 ) . Therefore , it is important to account for this irrationality when simulating systems involving human ( -like ) agents . To this end , we introduce Rational Inattention Reinforcement Learning ( RIRL ) , a MARL framework with agents that can be rationally inattentive . Rational Inattention ( RI ) is a model of bounded rationality ( Sims , 2003 ) that is well established in behavioral economics ( Mackowiak et al. , 2021 ) . It attributes human irrationality to the costliness of mental effort ( e.g . attention ) required to identify the optimal action . Mathematically , the RI framework measures these costs as the mutual information ( MI ) between the variables considered by the decision process and the decisions ultimately made . This captures the intuition that a policy has a higher cognitive cost if its execution requires more information about the state of the world and thus more attention . MI-based rewards have been used in RL , ( e.g . Leibfried & Braun ( 2016 ) ; Leibfried et al . ( 2018 ) ; Grau-Moya et al . ( 2018 ) ; Leibfried & Grau-Moya ( 2020 ) use RI-style MI costs to regularize learning ) but not to model sub-optimal behavior . When used to model sub-optimal behavior , RIRL can rationalize seemingly sub-optimal behavior by including this cognitive cost in the reward function , i.e. , by adding the MI cost ( s ) . That is , the “ rational ” behaviors of an RIRL-actor can mimic human-like bounded rationality . Contributions . The main contributions of this work are : ( 1 ) we propose RIRL , a framework for modeling bounded-rationality which can analyze complex scenarios not previously possible ; and ( 2 ) we validate RIRL ’ s modeling capabilities in classical economic settings intractable using the analytical methods of prior work . To our knowledge , we are the first to study the effects of Rational Inattention in MARL-based simulations . RIRL extends the single-timestep framework proposed by Peng et al . ( 2017 ) , which decomposes decision-making into stochastic perception followed by stochastic action , each subject to its own MI cost . RIRL provides a novel boundedly-rational policy class that generalizes this to multiple information channels with heterogeneous costs , hidden-state policy models , and sequential environments . To achieve this increased modeling power , our architecture uses methods significantly different from prior work , such as using deep mutual information estimation modules to allow for multiple channels and using an LSTM to account for multiple timesteps . This allows RIRL to analyze settings with rich cognitive cost structures , e.g. , when information about state variables have different observation difficulty . For example , in hiring , a candidate ’ s past job performance may be more relevant but harder to evaluate than her employment history . We evaluate RIRL in two Principal-Agent ( PA ) problems , specifically manager-employee relations , where the Principal is boundedly rational . We use RIRL to model asymmetric information ( i.e . hard to obtain information ) between the Principal and the Agent they are incentivizing . Real-world PA experiments have shown that bounded rationality is key to explaining marked deviations between equilibria reached by human participants and theoretical predictions ( Erlei & Schenk-Mathes , 2017 ) . Prior work considered bounded rationality assumptions but found it to be analytically difficult ( Mirrlees , 1976 ) and further research in this direction is sparse We show that RIRL allows us to analyze generalized PA problems that are analytically intractable , such as a sequential PA problem with multiple Agents and heterogeneous information channels . Across all settings , we observe that equilibrium implications depends strongly on the cost ( s ) of attention and differs from that under rational assumptions sometimes in ways that are hard to intuitively predict . We observe that , depending on the channel , increasing Principal inattention can either increase Agent welfare due to increased compensation or decrease Agent welfare due to encouraging additional work . Compared to under rational assumptions , we additionally observe the emergence of different strategies , such as Agents choosing to misrepresent their ability , classically referred to as signaling ( Spence , 1973 ) . Our results show that RIRL can be a powerful tool to model boundedly rational behavior and analyze its emergent consequences in multi-agent systems . 2 RELATED WORKS . Multi-Agent Reinforcement Learning for Agent-Based Simulation . Agent-Based Models ( ABM ) are a popular tool use to study real-world systems ( e.g. , organizations or market systems ) and discover emergent behavior through simulation with fixed or manually-specified agent behaviors ( Bonabeau , 2002 ) . Instead , Multi-Agent Reinforcement Learning ( MARL ) -based simulations use RL agents which autonomously learn utility-maximizing behavior , so designers do not need to specify behavioral rules . MARL has been used to study tax policy ( Zheng et al. , 2020 ; 2021 ; Trott et al. , 2021 ) , games ( Baker et al. , 2020 ) , and social dilemmas ( Leibo et al. , 2017 ) among others ( Yang et al. , 2018 ) . However , RL agents are mostly assumed rational which contradicts how humans make decisions . Some recent work considers bounded rationality in MARL by accounting for cognitive limitations when reasoning about the behaviors of other agents ( Evans & Prokopenko , 2021 ; Wen et al. , 2019 ; Łatek et al. , 2009 ) . We model a complementary source of bounded rationality : the cognitive costs of attending to all available information when forming a decision . Models of Irrationality . Behavioral economics has shown extensively that human decision-making is not fully rational , but instead features many cognitive biases ( Caverni et al. , 1990 ) . Bounded rationality ( Simon , 1957 ) attributes these human irrationalities to resource limitations , e.g. , bounded cognitive capabilities or costliness of using ( more ) time to make decisions . Rational Inattention ( RI ) is a well-established model of bounded rationality which models the cognitive cost of decisions as the mutual information between variables relevant to the decision and the decision itself ( Sims , 2003 ; Mackowiak et al. , 2021 ) . RI has been tested in real world experiments ( Dean & Neligh , 2017 ) and used to model human behavior in a wide variety of domains ( Hoiles et al. , 2020 ; Mackowiak et al. , 2021 ) . Closest to our work , Jiang et al . ( 2019 ) study a multi-agent system for traffic route choice under RI but where the agents do not react to each other ’ s actions . Another line of work uses human behavioral data with deep learning to model human cognition ( Kubilius et al. , 2019 ; Battleday et al. , 2017 ; Ma & Peters , 2020 ) or predict human behavior ( Bourgin et al. , 2019 ; Kolumbus & Noti , 2019 ; Hartford , 2016 ) . In contrast , our ABM approach examines the implications of RI in complex settings where experimental data are presently unavailable . Mutual Information in Reinforcement Learning . MI has been extensively studied in the intrinsically motivated RL literature for curiosity-driven exploration and unsupervised skill and option discovery ( Still & Precup , 2012 ; Campos et al. , 2020 ; Mohamed & Rezende , 2015 ; Gregor et al. , 2016 ; Eysenbach et al. , 2019 ) , often with differing techniques used to measure MI . Recently MI-based rewards have been used to regularize exploration ( Grau-Moya et al. , 2018 ; Leibfried & Grau-Moya , 2020 ) . Comparatively , very little work has considered MI-based rewards for modeling boundedly rational behavior ( Ortega et al. , 2015 ; Peng et al. , 2017 ) , with no prior work , to our knowledge , considering the domain of multi-agent simulations . Principal-Agent Problems . In addition to our guiding example , Principal-Agent problems have been used to describe various settings , including politics and insurance ( Miller , 2005 ; Grossman & Hart , 1992 ) . Prior economics literature mainly use analytical methods and narrow modeling assumptions , e.g. , that all stochasticity follows Brownian motion ( Sannikov , 2008 ) or certain separability conditions are satisfied ( Grossman & Hart , 1992 ) . Generally , Principal-Agent settings fall under the category of Stackelberg games where it has been shown computing optimal strategies can be NP-hard ( Korzhyk et al. , 2010 ) . Instead , MARL can study complex setups that are analytically intractable . Shu & Tian ( 2019 ) ; Shi et al . ( 2019 ) ; Ahilan & Dayan ( 2019 ) studied coordinating cooperation in a Principal-Agent model , with an RL Principal learning to incentivize Agents to achieve an overall goal . However , they do not consider bounded rationality . 3 PRELIMINARIES . We consider multi-agent simulations in the form of partially-observable Markov Games ( MGs ) , formally defined by ( S , A , r , T , γ , O , I ) ( Sutton & Barto , 2018 ) . Here S is the state space of the game , A is the combined action spaces of the actors , and I are actor indices . We use oi = O ( s , i ) to denote the portion of the game state s that actor i can observe . In addition , oi may include a ( possibly learnable ) encoding of the observation history . Each game episode has a horizon of T ≥ 1 timestep ( s ) . Each timestep t , actor i selects action ai , t , sampling from its policy πi ( ai , t|oi , t ) . Given the sampled actions , the transition function T determines how the state evolves . Each actor ’ s objective is encoded in its reward function ri ( s , a ) , ( boldface denotes concatenation across actors ) . When modeling economic behavior , the reward is the ( marginal ) utility Ui ( st , at ) , an abstract measure of happiness . Each actor optimizes its policy to maximize its γ-discounted sum of future rewards . While RL can be used to discover ( approximately ) utility-maximizing policies , such rational behavior fails to account for characteristic irrational human behavior . Behavioral economic models often model irrationality as consequences of inattention ( Gabaix , 2017 ) . Rational Inattention formalizes this intuition using a modified objective that includes a cost to the mutual information I ( ai ; oi ) between the ( observable ) state of the world oi and the actions ai ∼ πi ( ·|oi ) . This definition captures the intuition that if the agent puts in more effort to pay attention to oi , its action ai likely becomes more correlated with the observation oi , and thus high MI . π†i = argmax πi ( Eπ [ Ui ( s , a ) ] − λI ( ai ; oi ) ) . ( 1 ) Note that this is equivalent to learning the optimal policy for an adjusted reward function : r†i ( st , at ) = Ui ( st , at ) − λĨ ( ai , t ; oi , t ) , where Ĩ ( ai , t ; oi , t ) = log p ( ai , t , oi , t ) − log p ( ai , t ) p ( oi , t ) is a Monte Carlo estimate of I ( ai ; oi ) , and λ is the utility cost per bit of information . Here p ( ai , oi ) , p ( ai ) , and p ( oi ) are the joint and marginal distributions over ai and oi induced by the environment and policies π . Below , we describe our method for estimating these quantities during training .

Summary: The paper proposes a RIRL, which incoporate rational inattention suboptimality into the optimal RL model. The authors simulate RIRL in a single agent and a multi-agent setting. They compare RIRL behavior with the rational model in those settings and discuss the emergent behavior of RIRL.