Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning
Authors: Tong Mu
Stephan Zheng
Alexander R Trott
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Multi-Agent Reinforcement Learning
Bounded Rationality
Rational Inattention
Simulations

Abstract: Multi-agent reinforcement learning (MARL) is a powerful framework for studying emergent behavior in complex agent-based simulations. However, RL agents are often assumed to be rational and behave optimally, which does not fully reflect human behavior. Here, we study more human-like RL agents which incorporate an established model of human-irrationality, the Rational Inattention (RI) model. RI models the cost of cognitive information processing using mutual information. Our RIRL framework generalizes and is more flexible than prior work by allowing for multi-timestep dynamics and information channels with heterogeneous processing costs. We evaluate RIRL in Principal-Agent (specifically manager-employee relations) problem settings of varying complexity where RI models information asymmetry (e.g. it may be costly for the manager to observe certain information about the employees). We show that using RIRL yields a rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions. For instance, some forms of a Principal's inattention can increase Agent welfare due to increased compensation, while other forms of inattention can decrease Agent welfare by encouraging extra work effort. Additionally, new strategies emerge compared to those under rationality assumptions, e.g., Agents are incentivized to misrepresent their ability. These results suggest RIRL is a powerful tool towards building AI agents that can mimic real human behavior. 

Under review as a conference paper at ICLR 2022
MODELING BOUNDED RATIONALITY IN MULTI- AGENT SIMULATIONS USING RATIONALLY INATTEN- TIVE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-agent reinforcement learning (MARL) is a powerful framework for study- ing emergent behavior in complex agent-based simulations. However, RL agents are often assumed to be rational and behave optimally, which does not fully re- flect human behavior. Here, we study more human-like RL agents which incor- porate established model of numan-irrationallee model. RI models the cost of cognitive information processing using mutual in- formation. Our RIRL framework generalizes and is flexible than prior work by allowing for multi-timestep dynamics and information channels with hetero- geneous processing costs. We evaluate RIRL in Principal-Ageni (specifically manager-employee relations) problem settings of varying complexity where RI models information asymmetry (e.g. may be costly for the manager to observe certain information about the employees). We show that using RIRL yields rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions. For instance, some forms of Principal's 's inattention can increase Agent welfare due to increased compensation, while other forms of inattention can decrease Agent welfare by encouraging extra work effort. Additionally, new strategics emerge compared those under rationality assumptions, c.g., Agents are incentivized to misrepresent their ability. These results suggest RIRL is powerful tool towards building Al agents that can mimic real human behavior.
the Rational Inattention (RI)
INTRODUCTION
Multi-agent reinforcement learning (MARL) has shown great utility in complex agent-based model (ABM) simulations, e.g.. economics, games, and other fields (Zheng et al., 2020; [Baker et al. Leibo et al.] 2017 7 Yang et al. 2018). ABM is field of research which has been found important and useful for quantifying the potential impacts of various policies in diverse real world applications (eg. market (Bozanta & Nasir 2014) environmental impact (Raihanian Mashhadi & Behdadd 2018) simulations) by examining real-world systems through simulation. In many of such simulations, the behavioral rules of agents may be too difficult for designers specify. With MARL,, designers instead specify the agents' objective functions and the reinforcement learning (RI.) agents autonomously learn behaviors to optimize their objectives. However, this approach may be problem- atic when simulating systems of human agents, because in contrast to established models of human decision-making, it assumes that agents behave rationally and execute the objective- maxiiiiii behavior. Prior literature demonstrates that using established models of ing decision making yields results and implications that are significantly different from those ob- tained using rationality assumptions (Sims) 2003) Mačkowiak & Wiederholt, [011] Jiang et al.) 2019). Therefore, it important to account for this irrationality when simulating systems involving human(-like) agents.
2020
dur-
To this end, we introduce Rational Inattention Reinforcement Learning (RIRL), a MARL framework with agents that can be rationally inattentive. Rational Inattention (RI) is model of bounded ra- tionality (Sims, 2003) that is well established in behavioral economics (Mackowiak et al. 2021). attributes human irrationality the costliness of mental effort (e.g. attention) required to identify the optimal action. Mathematically, the RI framework measures these costs as the mutual information (MI) between the variables considered by the decision process and the decisions ultimately made.

---

Under review as a conference paper at ICLR 2022
This captures the intuition that policy has higher cognitive cost if its execution requires more information about the state of the world and thus more attention. MI-based rewards have been used in RL, (e.g. Leibfried & Braun (2016); Leibfried et al. (2018); Grau-Moya et al. (2018); Leibfried] & Grru-Moya (2020) use RI-style MI costs to regularize learning) but not to model sub-optimal behavior. When used model sub-optimal behavior, RIRL can rationalize seemingly sub-optimal
behavior by including this cognitive cost in the reward function, i.e., by adding the MI cost(s). That
is, the "rational" behaviors of an RIRL-actor can mimic human-like bounded rationality.
Contributions. The main contributions of this work are: (I) we propose RIRL, a framework for modeling nounded-rationalle which can analyze complex scenarios not previously possible; and (2) we validate RIRL's 's modeling capabilities in classical economic settings intractable using the analytical methods of prior work. To our knowledge, we are the first to study the effects of Rational Inattention in MARI-bbased simulations.
RIRL extends the single-timestep framework proposed by Peng et al. (2017), which decomposes decision-making into stochastic perception followed by stochastic action, each subject to its own MI cost. RIRL provides a novel rounoedly-rational policy class that generalizes this to multiple information channels with heterogeneous costs, hidden-state policy models, and sequential envi- ronments. To achieve this increased modeling power, our architecture uses methods significantly different from prior work, such using deep mutual information estimation modules to allow for multiple channels and using an LSTM to account for multiple timesteps. This allows RIRI. to an- alyze settings with rich cognitive cost structures, e.g., when information about state variables have different observation difficulty. For example, in hiring, candidate's 's past job performance may be more relevant but harder to evaluate than her employment history.
We evaluate RIRL in two Principal-Agent (PA) problems, specifically manager-employee relations, where the Principal is boundedly rational. We use RIRL to model asymmetric information (i.e. hard to obtain information) between the Principal and the Agent they are incentivizing. Real-world PA experiments have shown that bounded rationality is key to explaining marked deviations between equilibria reached by human participants and theoretical predictions (Erlei & Schenk-Mather 2017)- Prior work considered bounded rationality assumptions but found it to be analytically difficult (Mir- rlees 1976) and further research in this direction is sparse
We show that RIRL allows us to analyze generalized PA problems that are analytically intractable, such as a sequential PA problem with multiple Agents and heterogeneous information channels. Across all settings, we observe that equilibrium implications depends strongly on the cost(s) of attention and differs from that under rational assumptions sometimes in ways that are hard to in- tuitively predict. We observe that, depending on the channel, increasing Principal inattention can either increase Agent welfare due increased compensation decrease Agent welfare due to couraging additional work. Compared to under rational assumptions, we additionally observe the emergence of different strategies, such as Agents choosing to misrepresent their ability, classically referred to as signaling (Spence) 1973). Our results show that RIRL can be a powerful tool to model boundedly rational behavior and analyze its emergent consequences in multi-agent systems.
RELATED WORKS
Multi-Agent Reinforcement Learning for Agent-Based Simulation. Agent-Based Models (ABM) are a popular tool use to study real-world systems (e.g., organizations or market systems) and discover emergent behavior through simulation with fixed or manually-speeiffcc agent behav- iors (Bonabcau) 2002). Instead, Multi-Agent Reinforcement Learning (MARL)-based simulations use RL agents which autonomously learn millitzimaimizzng behavior, so designers do not need to specify behavioral rules. MARL has been used to study tax policy (Zheng et al. 2020; 2021 Troti al. 2021), games (Baker et al. 2020), and social dilemmas (Leibo et al. (017) among others (Yang et al. 2018). However, RL agents are mostly assumed rational which contradicts how humans make decisions. Some recent work considers bounded rationality in MARL by accounting for cognitive limitations when reasoning about the behaviors of other agents (Evans & Prokopenko 202 Wen jet al., 2019 Latek et al., 2009). We model a complementary source of bounded rationality: the cognitive costs of attending to all available information when forming a decision.
2

---

Under review as a conference paper at ICLR 2022
Models of Irrationality. Behavioral economics has shown extensively that human decision- is not fully rational, but instead features many cognitive biases (Caverni et al. 1990). Bounded rationality (Simon 1957p attributes these human irrationalities to resource limitations, e.g., bounded cognitive capabilities or costliness of using (more) time to make decisions. Rational Inattention (RT) is well-established model of bounded rationality which models the cognitive cost of decisions as the mutual information between variables relevant to the decision and the decision itself (Sims) 2003 Mackowiak al. 2021). RI has been tested in real world experiments (Dean & Neligh 2017) and used to model human behavior in wide variety of domains (Hoiles et al. 2020] Mackowiak al.] (02T). Closest to our work, Diang et al. (2019) study a multi-agent system for traffic route choice under RI but where the agents do not react to each other's actions.
Another line of work uses human behavioral data with deep learning to model human cognition (Ku- bilius et al. 2019; Battleday et al. 2017 Ma & Peters 2020) or predict human behavior (Bourgin jt al.] 2019 Kolumbus Noti] 2019 | In contrast, our ABM approach examines the implications of RI in complex settings where experimental data are presently unavailable.
Mutual Information in Reinforcement Learning. MI has been extensively studied in the intrinsi- cally motivated RL literature for curiosity-driven exploration and unsupervised skill and option dis-
covery (Still& & Precup 2012 Campos et al.) 2020 Mohamed & Rezende 2013 Gregor et al., 2016
Eysenbach et al., 2019), j. often with differing techniques used to measure MI. Recently MI-based
rewards have been used to regularize exploration (Grau-Moya et al., 2018 Leibtried & Grau-Moya 2020). Comparatively, very little work has considered MI-based rewards for modeling boundedly rational behavior (Ortega et al. 2015 Peng et al., 2017). with no prior work, to our knowledge, considering the domain of multi-agent simulations.
Principal-Agent Problems. In addition to our guiding example, Principal-Agent problems have been used describe various settings, including politics and insurance (Miller) 2005 Grossman & Hart 1992). Prior economics literature mainly use analytical methods and narrow modeling assump- tions, e.g., that all stochasticity follows Brownian motion (Sannikov 2008) or certain separability conditions are satisfied (Grossman & Hart 1992). Generally, Principal-Agen settings fall under the category of Stackelberg games where it has been shown computing optimal strategies can be NP-hard (Korzhyk et al.) 2010). Instead, MARL can study complex setups that are analytically intractable. [Shu & Tian (2019); Shi et al. (2019); AAhilan & Dayan 2019) studied coordinating co- operation in a Principal- Agent model, with an RL Principal learning to incentivize Agents to achieve an overall goal. However, they do not consider bounded rationality.
3 PRELIMINARIES
We consider multi- simulations in the form of partially-observable Markov Games (MGs), for- mally defined by (S, A, r, T, 7, O,I) (Sutton& & Barto 2018). Here S is the state space of the game, A is the combined action spaces of the actors, and Z are actor indices. We use o, = O(s i) to denote the portion of the game state s that actor can observe. sibly learnable) encoding of the observation history. Each game episode has a horizon of T 2 timestep(s). Each timestep f, actor i selects action ai,s, sampling from its policy n. (a,,, (o..).. Given the sampled actions, the transition function T determines how the state evolves. Each actor's objec- tive is encoded in its reward function rs(8, a). (boldface denotes concatenation across actors). When modeling economic behavior, the reward is the (marginal) utility Us( az), an abstract measure of happiness. Each actor optimizes its policy to maximize its 7-discounted sum of future rewards.
addition, o, may include a (pos-
While RL can be used to discover (approximately) miximiima policies, such rational behav- ior fails to account for characteristic irrational human behavior. Behavioral economic models often model irrationality as consequences of inattention (Gabaix) (017). Rational Inattention formalizes this intuition using modified objective that includes a cost to the mutual information I I(a;; o,) be- tween the (observable) state of the world o and the actions o; ~ nr(-)o;). This definition captures the intuition that if the agent puts in more effort to pay attention to og, its action a, likely becomes more correlated with the observation o,, and thus high MI.
/ = arg max (R [U, a)) AT(a;)))).
(1)
3

---

Under review as a conference paper at ICLR 2022
Note that this is equivalent to learning the optimal policy for adjusted reward function: r! (stt,aa) at) Us(s; a+) (T(a,, Oc,t), where [(,r,; 05,)) log p(as,;; is a Monte Carlo estimate of [a;; o;), and À is the utility cost per bit of information. Here p(a;, oi), p(a:), and p(o;) are the joint and marginal distributions over a, and o, induced by the environment and policies π. Below, we describe our method for estimating these quantities during training.
log p(a),.)p((
4 MODELING AND TRAINING BOUNDEDLY RATIONAL ACTORS WITH RIRL
MI Estimation. RIRL uses pereros-purpose module for estimating In, (a:; ie., the single- sample MC estimate of the mutual information between or and 0z ~ Ti (T((o))))) Given the pair (a),;;), we estimate Ie, (a:; o;) from the ratio between log p(a;, 0;) (the log-odds under the joint distribution) and log p(pp)pppo (the log-odds under the factorized distributions). This ratio can be estimated using discriminator dx, (a,, 0₂) that learns to classify which of the two distributions the sample (a,, o:) came from. Samples from p(a,, o.) are generated during on-policy rollouts, while samples from p(a:)p(o;; can be generated by shuffling a batch of samples from the joint distribution. As such, on-policy rollout data can be used both to optimize ir and to train dz,, and compute rl,,- - ri, - AIa (4i,1) 01,)).
While we favor the above approach for its simplicity, other techniques for estimating MI have also been proposed, e.g. [Belghazi et al.] (2018). Our framework is not restricted to the choice of MI estimator used here, but comparison of these techniques is outside the scope of the current work.
Action- Perrep-eennn Decoupling with Multiple Information Channels. Penalizing (a;; o;) models the intuition that is costly to obtain or use information about the world, e.g., reduce uncertainty about o, or to identify the dullity-mxiimize extend our framework with mulriple channels of information with heterogeneous cognitive costs. For example, when buying a used car it is easier to see car prices than their actual condition.
action given og,s- To support richer modeling, we also
To that end, we extend the faction-perceplin models π (a) s) as stochastic perception module ((yls)) followed by an action module trained to optimize an RI-style reward r(s, a) >yI (y: s) class that can flexibly model M information channels (i.e., partitions of o) with different processing costs A We also use recurrent policies which may allocate processing costs strategically over time.
decoupling" strategy of Peng et al. (2017) which jointly
Ap Jp(a; y). We extend this using a policy
More formally, we assume that an observation or can be decomposed as a set of M 2 observations Or = {o...... of }. with of" being the observation from information channel m. In addition, we assume that each channel has an associated information cost: À {x ...., AM) J. For each channel, we learn an encoder f (yf" of"" th)), which takes of and recurrent state eh (see below) as inputs and outputs the parameters (means and standard deviations) of a stochastic encoding v!" In practice, we implement each fm as a residual encoder, with samples given by:
(2) where et"h is a random sample from a spherical Gaussian with dimensionality equal to that of y and Tt. For each fr" we also train a discriminator d (vi"' ( used to estimate (u"" [o]" ../a). The full encoding Yt- ples. Before sampling an action, we use an LSTM (Hochreiter & Schmidhuber 1997) to maintain a
1 o m 1m th),
yh" = + P. + o. ' m7
yy- y...... yr of o concatenates all M encoder sam-
history of (encoded) information: t/++1 = LS LSM ).). The encoding y/+ and updated LSTM state 1 are used as inputs to the stochastic action module w warllt, v1-1) which outputs a distribution over actions.
Training. We train this architecture with policy gradients (Williams 1992x
AT x
E. (Vlog (vb. ..... y.M aelse, the, Mett)rr).
(3)
log (yt. M arlor, the, 12) = log 12+1)
Σ log lr (y""lo"" (4)),
(4)
M
r = -U(ss, a.) ( Iy,, ve+l))
n I pm (21 ,o,l)
(5)
m=
4

---

Under review as a conference paper at ICLR 2022
5 VALIDATING RIRL IN PRINCIPAL-AGENT PROBLEMS
We demonstrate RIRL's 's modeling capabilities in two Principal-Agent (PA) problems of varying complexity. In both settings, a boundedly rational manager (the Principal) decides how to compen- sate its workers (the Agents). In general, labor is costly to the Agents but beneficial to the Principal, while the Agents benefit from income that is costly for the Principal to provide. Therefore, the Principal aims to find a wage schedule W that maximizes its profits. PA problems often consider how information asymmetry between the Principal and Agent(s) influences equilibrium schedules. RIRL can analyze a meaningful extension in which we model how the cost of information influences cquilibrium schedules, i.e., where the Principal can spend effort reduce information asymmetry.
We first study a bandit-style PA setting with a Principal that optimizes a wage schedule for each level of Agent output, subject to cognitive cost for observing output. Because some aspects of the solution are tractable with prior analytical methods, this simple setting provides useful validation of RIRL multi-Agent game. However note, even in this simple setting, aspects of the solution (such the constants) are not tractable analytically. Secondly, we apply RIRL to analyze a complex sequential PA problem with multiple Agents and heterogeneous information channels that outside the scope of analytical methods of prior work.
Our experimental results were able to uncover several non-trivial insights when bounded rationality is assumed that are hard to intuitively predict, demonstrating the importance of modeling bounded rationality in such simulations. Insights include (1) Agent utility can increase with principal inatten- tion and more rational Agent benefits more from Principal inattention due to the Principal setting higher pay schedules. To our knowledge, no prior work has identified the consequences of bounded rationality in both the Principal and Agent. (2) In the case of where there are multiple Agents, boundedly rational Principal can induce Prisoner's Dilemma like incentives for the Agents, reduc- ing the welfare of all agents. We discuss these insights along with others in detail below.
5.1 PRINCIPAL-AGENT WITH A SINGLE AGENT, SINGLE TIMESTEP (BANDIT SETTING)
We first explore RIRL in PA setting with single Agent, following classic work Mirrlees 1976 Holmstrom Milgrom 1987; Spremann 1987: Haubrich 1994). The Agent's labor output Me) is a stochastic function of its effort action e. The Agent chooses how much to work based on its payment schedule W. which controlled by the Principal and yields payment w as a stochastic function of output i.e., W(z) (M2, og). Here, the stochasticity in the pay schedule aims to model the Principal's I's uncertainty over z, e.g., due to limited attention. By assumption, the Agent accurately anticipates the shape of the payment schedule W when making its decisions, e.g., based on prior experience or reputation. As such, the RIRL-Principal sets the payment schedule:
W = arg max [ [Up (w, 2 (w: b(l)
(6)
W
Here, Alw 2) is the attention cost for W. For the Agent, (u,(())W)) /Z with normalization Z, where (elW) denotes the expected utility of effort given schedule W. Note that _ can be calculated directly. Importantly, the soft-Q policy models a form of bounded rationality, i.e., the log-odds of effort is proportional to its expected utility, with higher 3 making "better" actions e more likely. Hence, low 3 corresponds to less-than hhan-rational behavior.
soft-Q policy T
The utility of the Principal (Up) and Agent (Ua) follow standard economic utility functions:
Uu(w,e) CRRA(W; p)
and
Up Up(w,2)
Udility
Work Disutility
Prodit
Revenue
Amount Paid
Here CRRA is the concave Constant Relative Risk Adverse function (Prat] 1978); the risk aversion
parameter p sets its concavity (we use p = 2). More environment details are the Appendix. We use N N-sample policy gradients estimates of Equation t to learn the parameters μa and og of W
r
Σ Up 24) - xiw(w; wi W(z), 24 ~ h(e)). (e))))
(8)
Results. Figure [ shows the effects of the Principal's inattention on simulated outcomes. We also compare RIRT. outcomes against those found including entropy-based rewards. MI-based and
5

---

Under review as a conference paper at ICLR 2022
Principal Utility
Payments under MI Regularization
Payments under Ent. Regularization
0
0:
A
AVg W12
R:3
Avg W(2
A
X:5
:
MI
-2
Ercropy
Constant
MI/Ent. (A)
Agent Output (2)
Agent Output (z)
Figure 1: Bandit Experiment Results. All results are averaged over random seeds and plotted with 95% confidence intervals. (a) Comparing MI and Entropy regularization on Principal utility, The constant O-pay schedule provides a meaningful lower bound. (b, e) Pay schedule means under (b) and Entropy (c) regularization (pay schedule standard deviations are plotted in the Appendix).
(Leest deteminnitte)
Bodd
fo=7
(Most determinietie)
Agent Utility
b
Principal Utility
Payment Mutual Information
1.5
-3
I(w:
0.0
Mutual Information Cost (A)
Figure 2: Additional Bandit Experiment Results. All results are averaged over random seeds and plotted with 95% confidence intervals. Results are shown for each level of MI cost (A, x-axis) across multiple levels of Agent policy temperatures (3, color). (a, b) Agent and principal utility. (c) Mutual Information between output = and payment w.
entropy-based regularization are closely related (Grau-Moya et al.) 2018 Leibfried & Grau-Moya 2020), with subtle but important differences Nevertheless, they yield markedly different outcomes.
To highlight an important difference, consider the constant O-pay schedule. This provides no incen- tive for the Agent work and costs the Principal nothing, and therefore provides reasonable lower bound on Up. This lower bound should hold under bounded rationality, since no attention is required when W treats all outputs identically. Indeed, under RIRL, Up approaches this lower bound as increasing À leads the Principal to trade more profitable W's for ones with smaller demands on attention (Figs. Th. b. ( contrast, under entropy regularization, increasing quickly yields W's that violate the Up lower bound (Fig. [Ta c).
Figure Souure2shone shows how Agent and Principal utility and [w: 2) change as a function of X, across a range of Agent I's. Note that higher 3 increases the odds that the Agent selects the optimal action. This reveals an interesting multi-Agent interaction. As before, we observe that the Principal's utility Up (without the attention cost) decreases with increasing X. Furthermore, Up is generally lower when is lower (Fig. 2p), the Principal must pay more to influence T towards different actions.
Interestingly, we observe that the Agent's utility U. can actually increase when the Principal is inattentive, and that higher 3 tends to increase the beneficial range of inattention (Fig. 2 Lastly, B determines how the Principal responds to increasing attention costs. Agents that behave optimally more often (higher 3) incentivize the Principal more to pay attention (Fig. 2
Comparison With Prior Literature. For this setting, Mirrlees (996) found the theoretically opti- mal pay schedule should be of the form W(=) max Az + B, C) While they assume slightly
Under some conditions. they are identical. However, the key difference is that entropy regularization penalizes the policy for deviating from fixed, uniform prior, whereas RI-style MI regularization penalizes deviations from optimal prior.

---

Under review as a conference paper at ICLR 2022
Wege for each agent (w)
Principal:
Agents:
Each Agent:
Camuletivn Gurgul (2)
Hours
indil Hours Worked (h)
Elfort
Output
Individual Exfort (e)
Abilyy
Incividual Outpur Contribution (2)
A depiction of single timestep in an episode of our Sequential Multi-Agent Setting. Green variables (such as cumulative output) are not costly while blue variables (such as individual output) are costly for the principal to observe. The red variable (Ability) cannot be observed.
Figure 3:
different model of costly attention the emergent pay schedules match closely with this work: when fitting A, B, and p to data, measure r² 0.99 for all 3. The best fit p close to the theoretical value for some values of 3 (true p: 2. best fit p. 2.19 for 3 = 5). Interestingly, we do observe that the best fitting & tends to increase with 33 in model. Notably, the influence of Agent sub-optimality was outside the scope of this prior work. We are able to observe this relationship between p and 3 through the modelling complexity enabled by our RIRL framework.
5.2 PRINCIPAL-AGENT
THE SEQUENTIAL MULTI- AGENT SETTING
We show that RIRL enables modeling sequential Principal-Agent problems with multiple Agents with T timesteps. We consider horizons [2, 10] and teams of n. 4 Agents. We assume there are K = possible Agent abilities k. The Principal cannot see the Agents abilities. Each Agent's ability sampled randomly at the start of each episode. At each timestep. Agent i's output is:
Zi,t higt (V, ei,),
where it works h, hours and exerts effort e,e. The Principal moves first and sets a wage W1,t- Each Agent moves second: it knows Wi,t before choosing h4. and ei, and earning income Wie hag- As before, Principal utility Up measures profit. We define Ua following standard utility functions, where the optimal h increases with w. As consequence this configuration (demonstrated below), the mroii-maximizing wage for Agent i increases with its ability v, The Agent utility is:
(w, h,,, h,,e) CRRA(W h)
(1+(1ee) Up(w,
Σ wehis iE(mal
(10)
[mall
Income Utility
Work Disutility
Protit
Revenne
Wages Paid
where ct and a are constants governing the shape of U..
Attention Costs. Because strategic Principal must infer private Agent features, e.g., ability, its equilibrium behavior depends on any inference costs it experiences, e.g., attention costs. Note that, unlike in the above bandit setting, here we train both the Principal and the Agent policies, each of which is modeled using the RIRL-actor architecture (Section 4. However to isolate and explore the effects of distinct Principal attention costs, we do not impose any attention costs on the Agents. Therefore, the Agents reward is only their utility Ti, Un(wi,r. (i,)).
For the Principal, we model M = 3 information channels: one is "easy" and low-cost to observe and two are "hard" and high-cost. The low-cost channel of includes information that we regard as freely available (> 0), e.g., the time f, the hours worked h (workers often fill out timesheets which makes h; easy to see), and the total output, Z = Eienn.] ži (managers can see the final result). However, it is high-cost to see individual contributions: we use a high-cost channel of for efforts e, and high-cost channel of for outputs z. This models Principal who can spend time and attention to observe individual Agents to reduce uncertainty about their true ability, e.g., their working styles and productivity (Figure B. Modeling the attention cost of output and effort separately lets us study the effects and interactions of unequal observation costs. The Principal's 's reward is
p.. Up (w,, f4, =1)
x x=I (ivi:an 21)
X (((viee) es)
(1I)
Individual Outpur and
Effort Perception Cost
Their Principal pays cost to reduce the noise added to its observed output, opposed to our MI cost; the Appendix for full details.
7

---

Under review as a conference paper at ICLR 2022
Apen: Utilty (All Types)
d
Agemt Usility (lypar k-0)
Agent U-iiity (Type: k::4)
Principal Unilty
110
75
A A
Octpat Mi Cont (h2
Principal UN (16)
Avg. Wage (3/he)
Octput Mit Cant (hx)
(lk)
Agent Utiliyy (Ub)
Agent Type
(A+)
Figure 4: Sequential, multi-agent experiment results. All results are averaged over 20 runs and generated with = 5. (a, b) Principal and Agent utility heatmaps (a) and scatter plot (b), for each X and X (c) Average wage for each Agent type, under rational (A= 0, yellow) and boundedly rational (X= = 6, blue) Principal. Error bars denote to 95% confidence intervals. (d) Utility for the lowest (A: 0) and highest (k- 4) ability Agent types, for each X and X
a
Acent Skill "Signaling" Behaviar
Agent across Equilibia
Principal's Attenticn Cutput over Time
on First Timestop (cwi)
Equality of Unilyy scrous Types
0.00
Encodingg Kyees
1.0
TTpee
058
2.
25
(1ghee))
330
0.25
335
6.8
6775
2:0
2.25
7.20
Effort MI Cost (1a)
Average Agent (Ub)
Timrstap (1)
Figure 5: Additional sequential, multi-agent experiment results. All results are averaged over 20 runs. Shaded regions denote 95% confidence. All results except (c) were generated with T 5. (a) Average effort at t = for each Agent type. (b) Equality across Agent types vs. Agent utility, for each X and X (c) Output information y (vi: 2+) encoded over time (when A= x==30 3).
Thus, the Principal's 's bounded rationality is modeled through the cost to get information about effort and individual outputs. The Principal's RIRL-actor architecture would also let us use attention costs [lyl o and [(w; ).). but we omit those here Additional training details are in the Appendix.
Results. We highlight several noteworthy observations revealed by our RIRL framework: (1) Prin- cipal inattention to individual outputs z and inattention to efforts e have distinct, nearly opposite, consequences actor utilities; (2) Agents respond to the incentive structures created by Principal inattention to effort, with parallels to a prisoner's ' dilemma; (3) The temporal patterns of Principal attention reflect the value of information over time.
Welfare Implications of Modeling Bounded Rationality. We analyze welfare related to actors' utilities. At equilibrium, the Principal and Agent utilities negatively correlated (Figure 4bb). comparing across varying levels of X= and X Note that the "rational" model has X = X O. This indicates that the Principal's bounded rationality has opposing implications for the Principal and Agents. From Figure 4.. we see that Agents' average utility increases and the Principal's 's utility decreases when the Principal's 's attention cost for individual outputs X increases. Conversely, Agent utility decreases with increasing Principal attention cost on effort X5.
The cause for this is as follows. The Principal has a different optimal wage for each ability wtt as shown in Figure Lt. When the Principal is fully rational it can use output and effort to accurately infer ability. Similar to what was seen in the bandit setting (above), increasing attention costs (in this case, on output X²) lead the Principal to set wages in manner that is less profitable but also less attentionally demanding. At the resulting equilibria, the Principal has more uncertainty over each Agent's type and adopts "better safe than sorry approach" and increases the average wage to ensure output. In sum, the same force that creates a lower-utility equilibrium for the Principal has higher average utility for the Agents.
We do add entropy regularization over (aly) for both the Principal and Agent to encourage exploration.
8

---

Under review as a conference paper at ICLR 2022
However, while the average Agent utility increases with X², it does not increase for all Agent types. Specifically, the utility of the (highest) lowest-ability Agent's (decreases) increases. Hence, the Principal's uncertainty over individual outputs decreases the wage (and utility) differences between Agents of different ability (Fig. Fd.d. This is particularly relevant when considering the equality of utility. A common inequality metrie is the Gini coefficient (Gini 1971), , computed sum of income differences; equality can be defined as eq
normalized
N N-Igini gini. Figure 5 shows that Agents' average utility and equality across types both increase for higher output attention cost X=
Bounded Rationality induces Social Dilemma Dynamics. This setup gives rise interesting temporal strategic behaviors akin to signalling (Spence 1973). Note that, while effort e increases an Agent's work disutility without increasing its income, Agents may still choose to exert non-zero effort > 0. To provide an intuition for this, first, note that spending effort increases output rate (z per hour f), resembling higher ability v², Second, the profit-maximizing wage w; increases with Agent ability [3) v The Principal can't 't see each v but can see the output 2; and hours h;. Hence, output rate is a "signal" of ability and Agents can misrepresent their ability by spending effort.
When X increases, it becomes costly for the Principal to distinguish between the Agent's ability and effort-action, Interestingly, this leads to signaling equilibria in which Agents choose to use more effort (Fig. Eb), resulting in lower average Agent utility and higher Principal utility (Fig. (5)). In effect, this Agent behavior has parallels to the "defect-defect" equilibrium in the prisoner's dilemma (Shoham& & Leyton-Brown 2008). When higher-ability Agents are not using effort, lower- ability Agents are incentivized to use effort to misrepresent themselves as having higher ability. This incentivizes the higher-ability Agents to also use effort distinguish themselves from the lower-ability Agents. Figure [5 shows this effect: effort increases with X at all ability levels. In effect, increasing X creates equilibria where the Principal enjoys higher Agent effort essentially for free. Interestingly, the effects of effort being costly to observe are nullified if the individual outputs are also hard to observe (Fig. 5D). showing these two cost parameters interact.
RIRL-actors Learn the Time- Value of Information. We also show that RIRL can discover the time-value of information-acquuiiiton and how it depends on the time horizon. Figure 5 shows output information that the Principal encodes (wF; zt) over time. There is initial spike in the amount of information at the beginning of the episode and this decreases over time. Intuitively, it most efficient for the Principal to pay attention to outputs at the beginning of the episode, as this information can be used throughout the episode. Additionally, the initial spike is larger for longer horizons, as initial information has more value for longer episodes.
Limitations. While RI is a general model of boundedly rationality, it does not cover all models of human irrationality. For simulations that wish to use more specialized models of bounded rationality, RIRL may not be sufficient. Examples include hyperbolic di discounting (Kirby & Herrsttein] 1995) to model amme-incossistett delay discounting (humans tend to prefer rewards in the near future much more than rewards farther in the future) or prospect theory (Tversky & Kahneman, (992) to model loss-aversion. Future research may extend RIRI. to include such notions of bounded rationality.
6 CONCLUSION
We propose a novel framework, RIRL, and associated policy class for modeling bounded-rationalle in MARL simulations with complexity beyond the scope of prior techniques. Our method incorpo- rates Rational Inatlention, an established model of human bounded rationality which uses mutual information to model the cognitive cost of processing information. Mutual information has been used extensively in RL for exploration and skill discovery, but, to knowledge, we the first to incorporate it for modeling human-like behavior in multi-agent simulations. We evaluate our method in two Principal-Agent problem settings, including complex multi-agent setting, with multiple in- formation channels with heterogeneous costs. Incorporating bounded rationality leads to different actor strategies and welfare outcomes as compared to under rational assumptions. These results establish RIRL as a promising framework for using MARL to analyze systems of human agents.
*This is a simple consequence of how utility functions are defined. See Fig 4 for confirmation.
9

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY AND ETHICS STATEMENTS
Ethics Statement. Our work proposes a framework to model bounded rationality in Multi-Agent Reinforcement Learning based simulations. Thus our framework may be used to draw implications in simulations modeled after real-world systems, While addressing the rationality gap between hu- man actors and RL actors is an important step towards this goal, it is hardly sufficient for achieving the realism required for real-world decision making based on Al simulations. We anticipate much longer path forward before this technology matures to such degree and note that any premature usage of our framework towards real-world decision making would contradict its intended research purpose. Furthermore, our framework should not used to explore methods increase discrimina- tion or unfairness in real-world systems, instead it should be use to investigate how to decrease such biases. We acknowledge that our PA settings identify a tension between employer (Principal) profit and employee (Agent) utility/equality. Our results are not meant to provide any actionable insight into how this tension could be manipulated for the benefit of one party. Any such application of our results or proposed framework would constitute ethical violation.
Reproducibility Statement. We discuss specific training details for all experiments such as hy- perparameters, random seeds used, and our process for setting the random seed in the Appendix. Upon acceptance, all code for this project will be made open source and publicly available for re- producibility purposes and further research. We will share code with reviewers and ACs for review upon the opening of the discussion forums.
REFERENCES
Sanjeevan Ahilan and Peter Dayan. Feudal multi-agent hierarchies for cooperative reinforcement learning. Workshop on "Structure Priors in Reinforcement Learning" at ICLR, 2019.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. ICLR, 2020.
Ruairidh M Battleday, Joshua C Peterson, and Thomas L Griffiths. Modeling human categorization of natural images using deep feature representations, arXiv preprint arXiv:1711.0485,, 2017.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. ICML, 2018.
Eric Bonabeau. Agent-basee modeling: Methods and techniques for simulating human systems.
PNAS, 2002.
David Bourgin, Joshua Peterson, Daniel Reichman, Stuart J Russell, and Thomas L Griffiths. Cognitive model priors for predicting human decisions. In International conference on machine learning, pp. 5133-5141. PMLR, 2019.
Aysun Bozanta and Aslthan Nasir. Usage of agent-based modeling and simulation in marketing. Journal of Advanced Management Science Vol, 2(3), 2014.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. Interna- tional Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
J-P Caverni, J-M Fabre, and Michel Gonzalez. Cognitive biases. Elsevier, 1990.
Mark Dean and Nate Leigh Neligh. Experimental tests of rational inattention. 2017.
Mathias Erlei and Heike Schenk-Mather Bounded rationality in principal-agent relationships. Ger- man Economic Review, 18(4):411 443, 2017.
Benjamin Patrick Evans and Mikhail Prokopenko. Bounded rationality for relaxing best response and mutual consistency: The quantal hierarchy model of decision-making, arXiv preprint arXiv:2/06. 15844, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. ICI.R 2019.
10

---

Under review as a conference paper at ICLR 2022
Xavier Gabaix. Behavioral Inattention. In Handbook of Behavioral Economics. 2017.
Corrado W Gini. Variability and mutability, contribution to the study of statistical distributions and relations, studi cconomico-giurión della r. universita de cagliari (1912). reviewed in: Light, rj, margolin, bh: An analysis of variance for categorical data. J. American Statistical Association, 06:534-544, 1971.
Jordi Grau-Moya, Felix Leibfried, and Peter Vrancx. Soft q-learning with mutual-information reg- ularization. In ICML, 2018
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv: 1.75077..,. 2016.
Sanford J Grossman and Oliver Hart. An analysis the principal-agem problem. In Foundations of insurance economics, pp. 302-340. Springer, 1992.
Jason Siyanda Hartford. Deep learning for predicting human strategic behavior. PhD thesis, Uni- versity of British Columbia, 2016.
Joseph G Haubrich. Risk aversion, performance pay, and the principal-agent problem. Journal of Political Economy, 102(2):258-2766 1994.
Sepp Hochreiter and Jtirgen Schmidhuber. Long Short-Term Memory. Neural Computation, 1997.
William Hoiles, Vikram Krishnamurthy, and Kunal Pattanayak. Rationally inattentive inverse rein- forcement learning explains youtube commenting behavior. Mach. Learn. Res., 21(170):1-39 2020.
Bengt Holmstrom and Paul Milgrom. Aggregation and Linearity in the Provision of Intertemporal Incentives. Econometrica, 1987.
Gege Jiang, Mogens Fosgerau, and Hong K Lo. Route choice, travel time variability, and rational inattention. Transportation Research Procedia, 38:482-502, 2019.
Kris N Kirby and Richard Hermstein. Preference reversals due to myopic discounting of delayed reward. Psychological science, 6(2):83-89, 1995.
Yoav Kolumbus and Gali Noti. Neural networks for predicting human interactions in repeated games. arXiv preprint arXiv:191.00323.. 2019.
Dmytro Korzhyk, Vincent Conitzer, and Ronald Parr. Complexity of computing optimal stackelberg strategies in security resource allocation games. In Twenty-fourth aaai conference on artificial intelligence, 2010.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Ha Hong, Najib Majaj, Rishi Rajalingham, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy. Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. arXiv preprint arXiv: www.it999909. 06161, 2019.
Maciej Latek, RL Axtell, and Bogumil Kaminski. Bounded rationality via recursion. In Proceedings of Eighth International Conference Autonomous Agents and Multi-Agent Systems (AAMAS 2009), pp. 457-464, 2009.
Felix Leibfried and Daniel A. Braun. Bounded rational decision-making in feedforward neural networks. In UAI, 2016.
Felix Leibfried and Jordi Grau-Moya. Mutual-information regularization in markov decision pro- cesses and actor-critic learning. In CoRL., pp. 360-373. PMLR, 2020.
Theoretii Optimality Felix Leibfried, Jordi Grau-Moya, and Haitham Bou-Ammar, An Information Theoretic Principle for Deep Reinforcement Learning. NeurlPS Workshop on Deep Reinforcement Learn- ing, 2018.
Joel Z. Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In AAMAS, 2017.

---

Under review as a conference paper at ICLR 2022
Wei Ji Ma and Benjamin Peters. A neural network walks into lab: towards using deep nets as models for human behavior. arXiv preprint arXiv:2005.02/81, 2020.
Bartosz Maćkowiak and Mirko Wiederholt. Busines Cycle Dynamics Under Rational Inattention. European Central Bank Working Paper Series, 2011.
Bartosz Mackowiak, Filip Matejka, and Mirko Wiederholt. Rational inattention: A review. 2021.
Gary J Miller. The political evolution of principal-agen models. Annu. Rev. Polit. Sci., 8:203-200 222-225, 2005.
James Mirrlees. The optimal structure of incentives and authority within an organization. The Bell Journal of Economics, pp. 105- 131, 1976.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning. NeurlPS, 2015.
Pedro A. Ortega, Daniel A. Braun, Justin Dyer, Kee-Eung Kim, and Naftali Tishby. Information- Theoretic Bounded Rationality. arXiv preprint arXiv: 11.5512.67778 2015.
Zhen Peng. Tim Genewein, Felix Leibfried, and Daniel A Braun. An information-theoreti on-line update principle for perception-action coupling. In 2017 IEEE/RSJ International Conference on intelligent robots and systems (IROS), pp. 789-796. IEEE, 2017.
John W Pratt. Risk aversion in the small and in the large. Uncertainty in economics, pp. 59-79.
Elsevier, 1978.
Ardeshir Raihanian Mashhadi and Sara Behdad. Environmental impact assessment of the hetero- geneity in consumers' usage behavior: An agent-based modeling approach. Journal of Industrial Ecology, 22(4):706-719, 2018.
Yuliy Sannikov. A continuous time version of the principal-agee aggett problem. The Review of Economic Suudies, 75(3):957-984, 2008.
Zhenyu Shi, Runsheng Yu, Xinrun Wang, Rundong Wang, Youzhi Zhang, Hanjiang Lai, and Bo An. Learning expensive coordination: An event-based deep rl approach. In ICLR, 2019.
Yoav Shoham and Kevin Leyton-Brown Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008
Tianmin and Yuandong Tian, M3rl: Mind-aware multi-agent management reinforcement learn- ing. ICLR, 2019.
Herbert A Simon. Models of man; social and rational. 1957.
Christopher A Sims. Implications of rational inattention. Journal of monetary Economics, 50(3): 665-690, 2003.
Michael Spence. Job Market Signaling. The Quarterly Journal of Economics, 1973.
Klaus Spremann. Agent and Principal. Agency Theory, Information, and Incentives, 1987.
Susanne Still and Doina Precup. An information-thoomm theooretc approach to curiosity-driven reinforce- ment learning. Theory in Biosciences, 131(3): 2012.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
Alexander Trott, Sunil Srinivasa. Douwe van der Wal, Sebastien Haneuse, and Stephan Zheng. Building a Foundation for Data-Driven, Interpretable, and Robust Policy Design using the AI Economist. arXiv preprint arXiv:2108.02900 2021.
Amos Tversky and Daniel Kahneman. Advances in Prospect Theory: Cumulative Representation of Uncertainty. Journal of Risk and Uncertainty, 5:297-323, 1992.
Ying Wen, Yaodong Yang, Rui Luo, and Jun Wang. Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. arXiv preprint arXiv: 19t1/99911...6..1 2019.
12

---

Under review as a conference paper at ICLR 2022
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 1992.
Yaodong Yang, Lantao Yu, Yiwei Bai, Jun Wang, Weinan Zhang, Ying Wen, and Yong Yu. A Study of Al Population Dynamics with Million-agent Reinforcement Learning. In AAMAS, 2018.
Stephan Zheng, Alexander Trott, Sunil Srinivasa, Nikhil Naik, Melvin Gruesbeck, David C Parkes, and Richard Socher. The ai economist: Improving equality and productivity with ai-driven tax policies. arXiv preprint arXiv:2004. /3332, 2020.
Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C. Parkes, and Richard Socher. The AI Economist: Optimal Economic Policy Design via Two-level Deep Reinforcement Learning. arXiv preprint arXiv:2108.0275:0 2021.
A ADDITIONAL BANDIT EXPERIMENT DETAILS AND RESULTS
A.1 ADDITIONAL DETAILS
Noise Structure. Recall in the bandit experiments, the output 2 is noisy function of the hours action h chosen by agent. Specifically we use the following output distribution:
0.7
if 2 = h
p(=lh)=
(12)
0.7la-h1
otherwise
0.3-c
Where i numerically calculated factor such that p(z h)) is proper probability distribution satis- rying Exe 2 p(z/h) = 1. This noise distribution assures high correlation between / and z with the probability of z levels decreasing exponentially as the difference between h and grows. We plot the probability heatmap in Figure 6a
Constant Relative Risk Aversion (CRRA) Function The constant relative risk aversion (erra function (Pratt) 1978) commonly used in economics has constant risk aversion which means de- cisions are invariant to scale.
concave function where the risk aversion parameter, p 2 0, determines the concavity. This models diminishing returns with higher values. The function is given by:
11--P-1
1,20 p=1
CRRA(U, p)
1-p
(13)
In(u)
Inattention Model From Theory. The model studied in classical theoretical Principal-Agent lit- crature we compare with (Mirrlees) 1976) utilizes the same utility functions for the principal and the agent. It has a slightly different model of principal inattention where it models costly principal inattention with a factor 0 that controls the magnitude of the observation noise in the output signal. Specifically, for true output level 2², the principal observes output 2 2" j.c where is noise. that is function of 0² and is subtracted from their utility
The principal incurs attention function. While this attention model is different from our mutual information based model, it shares many similar properties. In both our models the inattention introduces noise which makes acting optimally difficult. Additionally in both models, an attention parameter can reduce this noise at cost the principal and the principal can choose the optimal, muxiit--aaimizing value of this atten- tion parameter. Such similarities explain why, even under different models, our results are similar to those from theoretical analysis.
Training Hyperparameters. We used learning rates of 1e-3 for the training the principal policy parameters and 5e-3 for the mutual information classifier. We used a batch size of 128 and trained the principal for total of 100000 batches. During training we gradually annealed λa, from 0 to the desired value at a rate of 4/10000 per batch. We average all results across random seeds and we set the random seed for pytorch, numpy, and python's 's internal random module for each run. We used seeds of [0, 4]. All experiments were run on 16CPU cloud compute machines with 54GB of memory. Given a pay schedule which consists of mean and standard deviation parameters (M2, ox) for each output level z e Z, to calculate the principal policy we first sample 100 pay schedules and calculate the agent utility per output for each output level for each pay schedule. We then average to
13

---

Under review as a conference paper at ICLR 2022
Noise Distribution: P(z,h)
Pay Schedule Std. Dev MI Regularization
Pay Schedule Std. Dev Entropy Regularization
0.6
Hours (n)
0.4
10
-0.
6'
0.2
5
-0.2
8-
A:1
5:5
10
2
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
Output Level (2)
Output (z)
Output (z)
(a)
(b)
Pay Schedule Means
0.75
3
0.75
0.75
5
0.50
0.50
0.50
A. 7
0.25
0.00
0.00
0.00
Output (z)
(c)
Figure 6: Additional bandit experiment results. All results were averaged across 5 random seeds and have 95% confidence regions shaded. (a) The noise distribution between hours and output (P(=lh)). (b) The pay schedule standard deviation plots for MI and Entropy Regularization. (c) comparison of pay schedule means across different Agent policy 3 values.
calculate the average agent utility for each output level. We use the noise structure to calculate the average utility for each action and use the soft-q formulation over the utilities given in the main text obtain the agent's stochastic policy.
A.2 ADDITIONAL RESULTS
We provide few additional figures for the bandit experiment. Recall the principal's policy was the pay schedule parameterized by mean and standard deviation values for each output level (#z oz). Figure shows the standard deviation parameters for different cost factors Increasing entropy regularization results in very large increases to standard deviation parameters while MI regularization leads to much smaller standard deviation increases. The pay schedule means for Entropy regularization become sharper to overcome this increase in standard deviation.
learned under MI and Entropy regularization
Figure 6c compares pay schedule means (μ2)) across different Agent policy temperature (5) values. For low inattention costs, the pay schedules are similar. However the pay schedules for higher (more deterministic agents) decrease slower with increasing principal attention cost. Given the pay schedule parameters, the agent has optimal h action. More deterministic agents take this optimal action with higher probability so it is more valuable for the principal to incentivize them to higher optimal actions, which in turn lead higher outputs. This explains why Agent and Principal utility are higher for higher B values.
B ADDITIONAL MULTI-AGENT MULTI-TIMESTER EXPERIMENT DETAILS AND RESULTS
B. ADDITIONAL DETAILS
Training Hy perepaarseesrss. We used learning rates of le-4 for the Principal and Agent's policy parameters and le-3 for all the mutual information classifiers. We used a batch size of 512 episodes and train the principal and agent through 60000 batches. We train single RIRL-actor for the
14

---

Under review as a conference paper at ICLR 2022
Princ pal palttaitt (U))
25
100
snec palpallt UnI
110
20
100
80
%
0.5
3.0
0.0
0.5
Output cost (A,)
Effort Cost (7g]
Effort Mi Cost (ua)
Effrrt MI Cost Oel
(a)
(b)
(c)
U. U.
U. Ua
U₂ U.
U. U. Agent Type: 3
U. Ü. Agent Type:
Agent Type:
Agent Type:
Agent Type: 2
Output MI Cost (c))
0
4
-4
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Effort MI Cost (Aa)
(d)
Agent Utility (MA)
Hours (h)
AB-3
Cutput
-5
-1.0
-10
-15
-15
Output Cos: (h2)
Outout Cost (12)
Outgut Cost (2g)
Output Cost 2al
(e)
Figure 7: Additional sequential, Multi-Agent experiment results. All results are averaged over 20 random seeds and all confidence regions depict 95% confidence intervals. (a, b) Principal Utility with Output (X=) and Effort (A*) MI Costs. (c) The unscaled Output and Effort MI with X and X (d) The different between each agent type's utility and the mean utility across all types with X and X (e) How agent utility, effort, hours, and output changes with respect to Output MI Cost A for each agent type. All values are plotted as the change amount from the X value.
Agents and concatenate the experiences of all na Agents when updating the policy. The Agent policy therefore effectively has batch size of 512n.- a To avoid vastly different total episode returns, we scaled the rewards by the horizon during training. We run all experiments on SCPU cloud computing machines with 26GB of memory. We average all results across 20 random seeds and we set the random seed for pytorch, numpy, and python's 's internal random module for each run. We used seeds of [0, 19).
B.2 ADDITIONAL RESULTS
We present additional results for the sequential multi-agent experiments. Figures 7b show Principal utility with output MI cost (A*) and effort MI cost (X") for different horizons. We see the trend of Principal utility decreasing with X= and increasing with X occurs across all horizons.
7 unscaled MI cost for output and effort with x and X We see that I (yale) decreases with both A and x Tz decreases with X but affected greatly by X=.
Figure 7d] the agent utility for all agent types. expands the plot of Figure 41. The plots show the difference between the utility for each agent type and the average utility.
Figure 7e demonstrates how different values for different agent types change with output MI cost (A*). We plot each result the change from the x values. As mentioned previously, utility
15

---

Under review as a conference paper at ICLR 2022
increases with X for lower ability agents while it decreases for the highest ability agent. We observe similar trends for hours h. We additionally notice effort e generally decreases which leads to staying constant or decreasing for all agent types, even though some agent types have increased hours. This lower output along with higher average wages explains the decrease in Principal utility.
C RIRL IMPLEMENTATION DETAILS
We use this section of the appendix to cover important implementation details and tips. This is intended for practical guidance.
Stochastic Encoder Module Configuration and Initialization. Section 4 describes the architec- ture of our RIRL-actor policy class. As described, we learn an encoder ft (yt"" of (4) for each observation channel m. Encoder f takes observation of and recurrent state the as inputs and outputs the parameters (means and standard deviations) of a stochastic encoding y!"
We find that, in practice, learning does not progress if each y* contains very little information about TTT at the start of training. To address this, we recommend two implementation choices. First, implement n residual-style module:
pr, oo = [o]"
yi"h = p2 + -e,"
(14)
This simply requires setting the output y" to have the same size as observation ot and adding o," to the mean μız". Second, initialize the output layer of fr such that omh is consistently very small. We perform this by adding a constant negative offset to the bias units associated with the log T outputs, which we exponentiate to get a² As result of this strategy, yf" closely follows of the start of training.
Hidden State as an Encoder Input. We emphasize that the inputs to encoder fr is the con- catenation of the observation of and the hidden state th. Similarly, when using the discriminator dth d""",, [oo" viel) to estimate pm and when training the discriminator, we also apply this concate- nation. In other words, d regards [ot" vhs] as the observation, such that the Tym captures the MI between "!" and lo,", e/e]-
This is an important detail for ensuring that MI regularization works as expected in the multi-step setting. For instance, we observed that, if the discriminator does not see the hidden state th fo learns encode otn such that "y""; o²)) is minimal but where of can still be easily recovered given yith and th..
Optimization During training, we found that learning was most stable if separate learning rates were used for the policy modules {f] {fI...., fM, LSTM, w/ and for the discriminator modules (d'...., dd dd Importantly, the discriminator modules use a 10x higher learning rate. Con- cretely, we learning rates of 0.0001 and 0.001 for the policy and discriminator modules, spectively. Configuring learning rates this way helps to ensure that discriminator d'd can adjust to changes in encoder n faster than the encoder can adapt changes in the discriminator. Intuitively, this improves the quality of the MI estimates during training.
Another important optimization detail concerns gradient flow. Gradients from V log (arl)) need to backpropagate through the encodings [..... meaningful gradients. In Pytorch, which we use for this implementation, ensuring that this gra- dient flow occurs requires some attention. Given the (learndd) mean and standard deviation pa- rameters (which are functions of the encoder input), Pytorch constructs the sampling distribution as Normal (1,,a). The output of the encoder module yf" must be sampled via the repa- rameterization trick: = m. r zsapple (), which allows gradients to flow through y," Finally, this output should be detached from the backpropagation graph when calculating its log probabil- ity: encoder log -rrpbb-obbboooo = m. log prob y detach () ), which is needed for computing the policy gradients. Similarly, care must be taken to ensure that inputs to the discriminators are detached in the same way.
in order for the encoder modules to receive
16


Reviews:
[{"review_id": "mLF9uuX8mEM", "reviewer": "Reviewer_HuB7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written. I find the model interesting and the authors discuss some emergent behavior from the model. The model parameters are also relatively interpretable. Coming from a computational social science background, I can see this paper as a nice methodology paper for economics/psychology fields. The authors also promise the sharing of the code, which is a plus.\n\nAt the same time, as ICLR is still a more ML venue, the lack of more empirical results can be a big minus. The experiments are all simulations based. They do serve their purpose of illustrating the model and showing the various behavior the model can capture, but it is hard to conclude how well this model can capture actual human behavior or how it might compare to other models. The paper can benefit from some empirical results, though this might not be reasonable to ask given the short rebuttal period.  ", "summary_of_the_paper": "The paper proposes a RIRL, which incoporate rational inattention suboptimality into the optimal RL model. The authors simulate RIRL in a single agent and a multi-agent setting. They compare RIRL behavior with the rational model in those settings and discuss the emergent behavior of RIRL. ", "summary_of_the_review": "The paper can be a nice methodology paper but lacks empirical results for an ML venue. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "bjB2ZBu8Hzj", "reviewer": "Reviewer_FsBe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nInjecting more “psychologically plausible” models of human behavior into RL is very important for many applications\n\nAdding the information channel is a simple way to add rational inattention into RL\n\nCons:\nIt is unclear to me in which cases rational inattention is the right framework to use. Certainly if you are designing an agent, you probably don’t want to give them rational inattention unless you actually have a channel capacity constraint, so I guess the point here is to model humans? \n\nIn that case, I’m not sure what the gain of using RL is. The authors state: “Prior economics literature mainly use analytical methods and narrow modeling assumptions…Instead, MARL can study complex setups that are analytically intractable”. However, the simple model presented in experiment 1 is absolutely tractable. The second experiment may not be, but it is too simple to be a “real world example” and though it is more complex than the first experiment, it’s not clear what new insights we are supposed to gain.\n\nFinally, there doesn’t seem to be much theoretical guarantees here, and some of the patterns seem to be specific to the details of the game setup/RL setup so I am not sure what exactly we are supposed to take away from the experiments.\n\nOverall, I think this paper needs more convincing results that tell us something general either about PA+RI problems or just RI itself.\n", "summary_of_the_paper": "Standard Bayesian-type decision making models have that given some signal X, our decision makers choose an action A to maximize utility given the signal X. This gives a conditional distribution A|X. Rational inattention models say that the there is limted processing capacity in the agent and model this by requiring that A | X has a capacity constraint (i.e. which under some extra assumptions essentially becomes maximum amount of mutual information). \n\nThe authors put this into RL by, basically, writing a policy P(a|s) into an observation component (O(observation | state)) and an action conditional on the realized observation with an information penalty (so the higher capacity the channels, the more penalty). \n\nThe authors apply this RIRL model to some canonical game theory games.", "summary_of_the_review": "Overall, I think this paper needs more convincing results that tell us something general either about PA+RI problems or just RI itself.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QhLtEHk180c", "reviewer": "Reviewer_tabk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of bringing rational inattention into multi-agent reinforcement learning setups is very conceptually interesting. While the model for rational inattention seems to be standard in behavioral economics, the incorporation of this model into RL settings appears to be novel. Moreover, the formalization of irrationality as the mutual information between the action and the observed state seems natural in RL settings.\n\nI found the most interesting part of the paper to be the empirical evaluation section for principal-agent problems. In particular, the simulations in Section 5.2 demonstrate how the level of inattention of the principal can significantly impact the equilibrium reached relative to the typical case of fully rational behavior. For example, they are able to elucidate the effects of inattention on individual outputs and efforts, and they demonstrate that the Principal's behavior reflects understanding of how value of information over time changes. \n\nSome comments/questions:\n1. How much do the insights on this paper rely on the particular form of inattention considered? For example, if other metrics are considered besides mutual information, would similar results be observed? \n2. The experiments seem to focus on a single actor following rational inattention. What would happen, for example in Section 5.2, if some (or all) of the agents also faced inattention and experienced attention costs? ", "summary_of_the_paper": "The paper proposes Rational Inattention Reinforcement Learning (RIRL), a multi-agent reinforcement learning framework that incorporates the behavioral model of rational inattention. Building on the classical formalization of rational inattention, the objective penalizes for the mutual information between the action selected and the observable state of the world. They also extend their framework to support multiple channels of information with heterogeneous cognitive costs. \n\nThey empirically evaluate RIRL in two principal-agent problems, building on classical models from the economics literature. The first simulation consists of a single agent, while the second simulation explores interactions between multiple agents following the RIRL-actor architecture (although only the principal faces inattention). ", "summary_of_the_review": "Due to the conceptual novelty of incorporating rational inattention into multi-agent RL, I recommend weak acceptance. \n\n---\nUpdate after author response: Thanks to the authors for their detailed responses, and for clarifying the results in Section 5.1. I would encourage the authors to provide a more thorough discussion of these comparisons in a future version of the paper. Moreover, although it would be useful to consider multiple agents facing inattention (e.g. in Section 5.2), I found the existing results to be sufficiently interesting for acceptance to ICLR. Thus, I keep my assessment the same. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OJHN6QwlwLj", "reviewer": "Reviewer_6aq5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:  Using MARL to model human bounded rationality is interesting. It is meaningful to generalize the rational inattention model to allow dynamics of multiple time steps and multiple information channels with different cognitive costs. Moreover, extensive experimental results are presented to show that the RIRL framework can reproduce rich information about bounded rational behaviors. The paper is generally well-written and easy to follow.\n\nWeakness: The extension of the existing rational inattention model is straightforward and the technical contributions of this work seem marginal. The literature review is insufficient. For example, Wen et al. [1] also model bounded rationality in multi-agent interactions using MARL, however, this work has not been discussed. Moreover, the huge literature (e.g., [2,3]) on modeling/predicting human behaviors in games (i.e. multi-agent interactions) has not been mentioned; the works in this line of research not only present new models, but also validate their models through comparisons with real human decision results. \n\nFor the experiments, while I appreciate that the paper presents a quite extensive analysis on the principal-agent problem, I need to say that I can't evaluate the significance/novelty of the results as I am not familiar with the literature on the principal-agent problem.\n\n\n[1] Wen, Y., Yang, Y., Luo, R., & Wang, J. (2019). Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. IJCAI.\n\n[2] Hartford, J.S., Wright, J. R., & Leyton-Brown, K. (2016). Deep Learning for Predicting Human Strategic Behavior. NIPS.\n\n[3] Bourgin, D. D., Peterson, J. C., Reichman, D., Russell, S. J., & Griffiths, T. L. (2019). Cognitive model priors for predicting human decisions. ICML.\n", "summary_of_the_paper": "This paper studies human bounded rationality using multi-agent reinforcement learning. It particularly focuses on the rational inattention model and proposes the RIRL framework which allows multi-timestep dynamics and information channels with heterogeneous processing costs. The proposed RIRL framework is evaluated in two specific Principal-Agent problems. ", "summary_of_the_review": "The paper presents a new model that accounts for human rational inattention and evaluates the model in a specific problem, however, the technical contributions seem marginal and the literature review is insufficient. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a multi-agent RL framework that make decisions in a more human-like manner by incorporating rational inattention. The approach is evaluated on two game theoretic problems. The reviewers agree that the topic of the paper is interesting. However, there are concerns about the significance of the proposed approach. As the method incorporates human-inspired limitations, it's aim is not to outperform SOTA RL methods on regularly considered domains; at the same time, as the approach is only evaluated on two simulation-based tasks, it is unclear how it would perform in more realistic scenarios that may benefit from human-like decision making. For these reasons, I recommend rejection.
Decision: Reject