Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning
Authors: Tong Mu
Stephan Zheng
Alexander R Trott
Venue: Accepted by TMLR ()

Abstract: Multi-agent reinforcement learning (MARL) is a powerful framework for studying emergent behavior in complex agent-based simulations. However, RL agents are often assumed to be rational and behave optimally, which does not fully reflect human behavior. In this work, we propose a new, more human-like RL agent, which incorporates an established model of human-irrationality, the Rational Inattention (RI) model. RI models the cost of cognitive information processing using mutual information. Our RIRL framework generalizes and is more flexible than prior work by allowing for multi-timestep dynamics and information channels with heterogeneous processing costs. We demonstrate the flexibility of RIRL in versions of a classic economic setting (Principal-Agent setting) with varying complexity. In simple settings, we show using RIRL can lead to optimal agent behavior policy with approximately the same functional form as what is expected from the analysis of prior work, which utilizes theoretical methods. We additionally demonstrate that using RIRL to analyze complex, theoretically intractable settings, yields a rich spectrum of new equilibrium behaviors that differ from those found under rationality assumptions. For example, increasing the cognitive cost experienced by a manager agent results in the other agents increasing the magnitude of their action to compensate. These results suggest RIRL is a powerful tool towards building AI agents that can mimic real human behavior. 

Published in Transactions on Machine Learning Research (12/2022)
Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning
Tong Mu"
tongm www.:escssssssccccooo edu
Salesforce Research, Polo Alto, CA, USA
Stephan Zheng
stephan. Segphannzhengg@@seeeffoorre
Salesforce Research, Polo Alto, CA, USA
Alexander Trott'
Salesforce Research, Polo Alto, CA, USA
Reviewed on OpenReview: https: // opwnreviee.....e...mmme net/ formm? id= DV1p//rmDmm
Abstract
Multi-agent reinforcement learning (MARL) is powerful framework for studying emergent behavior in complex agent-based simulations. However, RL agents are often assumed to be rational and behave optimally, which does not fully reflect human behavior. In this work, we propose a new, more human-like RL agent, which incorporates an established model of cuman-irrationality the Rational Inattention (RI) model. RI models the cost of cognitive information processing using mutual information. Our RIRL framework generalizes and is more flexible than prior work by allowing for multi-timestep dynamics and information chan- nels with heterogeneous processing costs. We demonstrate the flexibility of RIRL in versions of a classic economic setting (Principal-Agent setting) with varying complexity. In simple settings, we show using RIRL can lead to optimal agent behavior policy with approximately the same functional form of what is expected from the analysis of prior work, which utilizes theoretical methods. We additionally demonstrate that using RIRL analyze complex, theoretically intractable settings, yields a rich spectrum of new equilibrium behaviors that differ from those found under rationality assumptions. For example, increasing the cognitive cost experienced by manager agent results in the other agents increasing the magnilude of their action to compensate. These results suggest RIRL is a powerful tool towards building AI agents that can mimic real human behavior.
1
Introduction
Multi-agent reinforcement learning (MART.) has shown great utility in complex agent-based model (ABM) simulations (c.g., in economics, games, and other fields (Zheng et al.) 2020: Baker et al.) 2020 Leibo et al. 2017 Yang el al.] 2018). ABM is influential field of research which examines various real-world systems through simulation to quantify the potential impacts of various policies (eg. market (Bozanta & Nasir 2014]
or environmental impact (Raihanian Mashhadi & Behdad 2018)) simulations). It has the promising potential to allow for better, more informed policies in critical applications areas, such as federal government economic policy (Farmer& & Foley 2009). Classically, in ABM, designers programmatically specify the behavioral rules of agents. This may be difficult in complex simulations, so in the MARL approach, designers instead specify the agents' objective functions and the reinforcement learning (RL) agents autonomously learn behaviors to optimize their objectives.
However, using rational RL agents is problematic when simulating systems with human agents. Rational agents execute the objective-mxximiiing behavior, which can differ greatly from established models of hu- Work done while at Salesforce.

---

Published in Transactions on Machine Learning Research (12/2022)
man decision-making. But humans often act non-rationally: they might not optimize perfectly for their objectives, and it is unclear to what degree human behavior can be described using a single (composite) scalar optimization objective. Such non-rational behavior is referred to as bounded rationality.
Prior economics literature has shown that modeling bounded rationality during decision decision-making gives results and implications that are significantly different from those obtained using rationality assumptions (Sims 2003 Maćkowiak & Wiederholt, 2011 Jiang et al., 2019). Therefore, it is important to account for bounded rationality when simulating human(-like) agents. To address this issue, there has been body of rescarch in the economic literature that analyzes systems when human decision making follows established models of human-irrationalii (Barazza & Strachan, 2020; Ww & Chen 2014 Mullainathan, 2002 Huang et al.
2013 Ren & Huang, 2018), such as under the Rational Inattention model (Sims 2003 Hoiles et al. 2020
Mackowiak et al. 2021 Jiang et al., 2019).
In the direction of this prior work, we study incorporating an explicit model of bounded rationality into MARL simulations. A key issue in prior work is that explicitly finding agent behaviors, given an oper- ational definition of bounded rationality, can be challenging learning and optimization problem. Here, reinforcement learning provides a powerful conceptual and algorithmic framework to model bounded ratio- nal behaviors. To this end, we introduce Rational Inattention Reinforcement Learning (RIRL), a MARL framework with agents that follow the well-established Rational Inattention (RI) model of bounded ratio- nality (Sims) 2003 Mackowiak et al., 2021). The RI model attributes human irrationality to the costliness of mental effort, e.g,, attention, required to identify the optimal action. Mathematically, the RI framework measures these costs as the mutual information (MI) between the variables considered by the decision pro- cess and the decisions ultimately made. This captures the intuition that policy has higher cognitive cost if its execution requires more information about the state of the world and thus more attention.
MI-based rewards have been used in RL, (e.g. Leibfried & Braun (2016); Leibfried et al. (2018): Grau- Moya et al. (2018); Leibfried & Grau-Moya (2020) use RI-style MI costs to regularize learning) but not to model sub-optimal behavior. When used to model sub-optimal behavior, RIRL can rationalize seemingly sub-optimal behavior by including this cognitive cost in the reward function, i.e., by adding the MT cost(s). That is, the "rational" behaviors of an RIRL-actor can mimic human-like bounded rationality.
1.1 Contributions
In this paper, we introduce a novel and flexible policy class for analyzing multi-agent systems with agents that are boundedly rational. The main contributions of this work are:
We propose RIRL, a framework for modeling founded-rationalit which can analyze complex sce- narios, not previously possible. Our method allows for the analysis of settings where different information has different observation or processing costs and multi-timestep settings.
2.
We validate RIRL's modeling capabilities in classical economic settings, where the majority of prior work focused on theoretically solving for the agent's behavior. We demonstrate in a simpler, previ- ously studied setting that our results match the expected functional form. Additionally in a more complex, theoretically intractable setting, our method is able to find equilibria of agent behavior that are different from rational equilibria and can be used to provide detailed analysis of these phenomena.
Modeling bounded rationality in complex environments. RIRL extends the single-timester frame- work proposed by Peng et al. (2017), which decomposes decision-making into stochastic perception followed by stochastic action, each subject to its own MI cost. RIRL provides novel boundedly-rational policy class that greatly expands this framework lo multiple information channels with heterogeneous costs, hidden-state policy models, and sequential environments. To achieve this increased modeling power, our architecture uses mothods significantly different from prior work, such as using deep mutual information estimation modules to allow for multiple channels and using an LSTM to account for multiple timesteps. This allows RIRL to analyze settings with rich cognitive cost structures, e.g., when information about state variables have
2

---

Published in Transactions on Machine Learning Research (12/2022)
different observation difficulty. For example, in hiring, a candidate's past job performance may be more relevant but harder to evaluate than her employment history.
Evaluation in the Principal-Agent setting. To illustrate our framework's modeling abilities, we evalu- ate RIRL in classical oconomic settings, specifically two Principal-Agent (PA) problems (Grossman & Hart, 1992: Sannikov 2008: Haubrichi 1994) and we consider when the Principal is boundedly rational. Typical economics research rely on theoretical analysis to analyze these settings. We use RIRL to model information asymmetry between the Principal and the Agents they are incentivizing, i.c., it is hard for the Principal to obtain information about the Agents. Real-world PA experiments have shown that bounded rationality is key to explaining marked deviations between equilibria reached by human participants and theoretical pre- dictions (Erlei & Schenk-Mathes 2017). Prior work considered bounded rationality assumptions but found it to be difficult to analyze theoretically (Mirrlecs 1976;; research in this direction is sparse.
We show that RIRL allows us to analyze generalized PA problems that are analytically intractable, such as a sequential PA problem with multiple Agents and heterogeneous information channels. Across all settings, we observe that equilibrium implications depends strongly the cost(s) of attention and differs from that under rational assumptions sometimes in ways that are hard to intuitively predict. We observe that increasing Principal inattention can either increase Agent welfare due to increased compensation or decrcase Agent welfare due to encouraging additional work. Compared with rational agents, we additionally observe the emergence of different strategies, such as Agents choosing to misrepresent their ability, classically referred to as signaling (Spence 1973). Our results show that RIRL can be powerful tool to model boundedly rational behavior and analyze its emergent consequences in multi-agent systems.
2 Related Works
Multi-Agent Reinforcement Learning for Agent-Based Simulation. Agent-Based Models (ABM) are a popular tool used to study real-world systems (e.g., organizations or market systems) and discover emergent behavior through simulation with fixed or manually-specified agent behaviors (Bonabeau 2002). Instead, Multi-Agent Reinforcement Learning (MARL)-baseed simulations use RL agents which autonomously learn hallity-mxxmiiiing behavior, so designers do not need to specify behavioral rules. MARL has been used to study tax policy (Zheng et al. 2020 2021 Trott et al. 20021) games (Baker et al.. 2020). and social dilemmas (Liboo et al. 2017) among others (Yang et al.) 2018).. However, RL agents are mostly assumed rational which contradicts how humans make decisions. Some recent work considers bounded rationality in MARL by accounting for cognitive limitations when reasoning about the behaviors of other agents (Evans & Prokopenko| 2021: Wen et al.) 2019 Latek et al. 2009). We model a complementary source of bounded ralionality: the cognitive costs of attending to all available information when forming a decision.
Models of Irrationality. Behavioral economics has shown extensively that human decision-making is not fully rational but instead features many cognitive biases (Caverni et al.) 1990). Bounded rationality (Simon 1957) attributes these human irrationalities resource limitations, e.g., bounded cognitive capabilities costliness of using (more) time to make decisions. Simon (2000) gives high-level overview of different ways to implement bounded rationality, but did not yet construct explicit solutions for (optimal) agent policies (under bounded rationality). Our work relates to this by learning explicit solutions in a setting with multiple strategic agents. As another example, Hernandez & Ortega (2019) studied bounded rationality in organizations and its impact on growth.
Rational Inattention (RI) is well-established model of bounded rationality which models the cognitive cost of decisions as the mutual information between variables relevant to the decision and the decision itself (Sims, 2003 Mackowiak et al. 2021). RI has been tested in real world experiments (Dean & Neligh 2017) and used to model human behavior in a wide variely of domains (Hoiles et al.) 2020 Mackowiak el. al.) 2021 Clasest to our work, Jiang et al.) (2019) study a multi-agent system for traffic route choice under RI but where the agents do not react to each other's actions. Another line of work uses human behavioral data with deep learning to model human cognition (Kubilius et al., [2019] Battleday et al., 2017 Ma & Peters 2020)
predict human behavior (Bourgin et al. 2019 Kolumbus & Noti 2019 Hartford 2016). In contrast, our
3

---

Published in Transactions on Machine Learning Research (12/2022)
ABM approach examines the implications of RI in complex settings where experimental data are presently unavailable.
Mutual information and rational inattention. MI has been extensively studied in the intrinsically motivated RL literature for curiosity-driven exploration and unsupervised skill and option discovery (Still & Precup 2012 Campos et al.) 2020 Mohamed & Rezende 2015 Gregor et al.] 2016 Eysenbach et al, 2019), often with differing techniques used measure MI. Recently MI-based rewards have been used to regularize exploration (Grau-Moya et al.]
2018 Leibfried & Grau-Moya, 2020 They have also been used to regularize policy learning (Malloy el al.] 2021 to learn better policies. However this work differs as it does not consider modeling bounded rationality for more realistic MARL simulations. Comparatively, very little work has considered MI-based rewards for modoling boundedly rational behavior (Ortega et al. 2015;] Peng et al., 2017). Genewein et al. (2015) introduced a mutual information between action and world state. However, it focuses on the single-agent setting. In our paper, we consider multiple (strategic agents), which significantly complicates finding the optimal policy for the bounded rational agent, and the corresponding best-response of the other agent(s). To our knowledge, there is with prior work considering the domain of multi-agent simulations.
For example, our policy class considers allowing a rich set of behavior to be modeled, such as different attention costs on different parts of the observation space. Additionally, due to the difference in settings, we also draw different conclusions, focusing on the implications of modeling human irrationality such as which agents benefit and how metrics, like equity, evolve.
Principal-Agent problems. In addition to our guiding example, Principal-Agent problems have been used to describe various settings, including politics and insurance (Miller 2005 Grossman & Harl, 1992)
Prior economics literature mainly use analytical methods and narrow modeling assumptions, e.g., that all stochasticity follows Brownian motion (Sannikov) 2008) or certain separability conditions are satisfied Grossman & Hart 1992.. Generally, Principal-Agent settings fall under the category of Stackelberg games where has been shown computing optimal strategies can be NP-hard (Korzhyk et al., 2010). Instead, MARL can study complex setups that are analytically intractable. [Shu & Tian (2019); Shi et al. (2019)
Ahilan & Dayan (0019) studied coordinating cooperation in Principal-Agent model, with an RL Principal learning to incentivize Agents to achieve overall goal. However, they do not consider bounded rationality.
Bounded rationality and reinforcement learning. Kokolakis et al. (2021) models bounded rationality using level-k policies instead of perfect rationality, which converge to the Nash equilibrium as & increases. Erevl& & Roth (2002) use RL to model bounded rationality agents in iterated prisoners dilemmas,, extending prior analytical models in game theory. Duéñeez-Gzznae et al. (202I) studies how statistical discrimination may arise in agents due to (limitations on) information processing using a theoretical model and RL agents.
3 Preliminaries
We consider multi-agent simulations in the form of partially-observable Markov Games (MGs), formally defined by ",,,Ar,TT7, O,I) (Sutton & Barto 2018). Here S is the state space of the game, A is the combined action spaces of the actors, and are actor indicos. Generally, the full state is not observable to each actor (for example, actors may not be able to observe the full policy of other actors, and only the history of actions the other actors took). We use o, = is,,)) to denote the portion of the game state that actor i can observe and use for their policy. In addition, O may include a (possibly learnable) encoding of the observation history. Each game cpisode has a horizon of T 2 1 timestep(s). Each timestep f, actor i selects action ai,t, sampling from its policy n; (ac,lor,)) Given the sampled actions, the transition function T determines how the state evolves. Each actor's objective is encoded in its reward function r, a), where boldface denotes concatenation across actors
Rewardd utility: In the economics literature agent's reward is often modeled by its utility (its mcasure of happiness) and denoted by u. Similarly, rewards and utilities model incentives that motivate agent's behavior. In this work, we will use "reward", denoted by r, throughout refer these concepts.
4

---

Published in Transactions on Machine Learning Research (12/2022)
Traditionally, prior work considers the case where the actor optimizes its policy ns to maximize its 7 discounted sum of future rewards:
Ext(ssaa
T arg max B
(1)
Using the subscript notation where subscript (eg. aus denotes the vector element relating to actor i and -i (e.g. at;-)) indicates vector with elements that not pertinent to actor i.
E ars oi oi.1))
m3 arg max E T T
(2)
s.t. oig ((()),,, x3(100) arg max 18, x
En'ry (Sv, augi an--i)
aes
Vj vj i.
(3)
u2t
Equation 2 states that the actor optimizes its return assuming the other actors play a best response, i.e., use a policy zj that maximizes their return.
3.1 Modeling Bounded Rationality using Rational Inattention
While RL can be used to discover (approximately) reward-maximizing policies, such rational behavior fails to account for characteristic irrational human behavior. Rational Inattention models bounded rationality using a modified objective that includes a cost to the mutual information [(;;; between the (observable) state of the world o and the actions a; ~ TT(-)))). This definition captures the intuition that if the agent puts in more effort to pay attention to og, its action a, likely becomes more correlated with the observation os and thus Mutual Information is high.
We optimize for the policy of the rationally- iniateentiii objective:
E Σn' (r(ss, as) - - )((anion)))
9: arg max
(4)
Note that this is equivalent to learning the optimal policy for an adjusted reward function: (s,, az)
=
rr(5,, a.) - Ai(a,, where
I(ar,ti = log (ar1, p((r)))(orr))
(5)
is a Monte Carlo estimate of I(a;; o,), and À the cost per bit of information. Here pla, o,), p(a;), and p(o:) are the joint and marginal distributions over a, and o, induced by the environment and policics æ.
To build intuition for the use of I, imagine the extreme the actor selects actions uniformly at random). The probability of given action being selected wouldn't depend on the observation and the MI cost I(a;; o,) would be O. On the other hand, if the action correlates with the observation, the actor is assumed to have paid attention to O in order to select a; and 1 >> 0. From a learning perspective, the MI cost acts as a regularizer which encourages simpler policies, i.e., where actions are less dependent on the procise state of the world and/or history of observations. Rationally inattentive actors therefore are incentivized source of information asymmetry. From economic perspective, the attention cost may model limits information processing and hence encourage different types of actor strategies. Alternatively, attention costs could reflect actual expenditures, i.e. monitoring costs.
where the actor pays no attention to o, (e.g.
select which information they use, making inattention an endogenous
3.2 Multiple Information Channels
To model more general forms of rational inattention, we extend our framework to support:
5

---

Published in Transactions on Machine Learning Research (12/2022)
1. multiple channels of information with heterogeneous cognitive costs, and
2. modeling which observation channel an actor pays attention to.
For example, when buying used car it is easier to see car prices (which might still take time) than ascer- taining the actual condition of the car.
As such, we first extend the action-pereeption decoupling strategy of Peng et al. (2017) to model :<alo) as a stochastic perception module qyylo) (encoder) followed by an action module w(aly) (decoder). The RI reward then becomes:
(6) where we omitted subscripts for clarity. the above, recall 8 denotes the true state the world and o denotes the portion observable to the agents. Parameters Ay and A control the amount of inattention simulated. This decomposes the attention cost into: (1) the cost of paying attention to the observation o and (2) the cost of using that information to select action.
r r(s, - X110 (a)
Second, we extend the encoder to capture attention costs that depend on M parallel observations fo'...., giving an RI reward function:
rl rl=r(s,a) [ Nw M),
where brackets denote concatenation, and each channel has its own attention cost A of observing o"".
4 Modeling and Training Boundedly Rational Actors with RIRL
We now describe practical approach to estimating the attention cost in Equation and solving the opti-
mization problem in Equations 2 and 3
These optimization problems present several technical challenges: (i) Finding the optimal policy is challenging in sequential sctting and with multiple actors. (ii) Computing the true mutual information cost is computationally expensive.
Our framework RIRL solves the policy optimization problem using reinforcement learning (RL), while we propose sampling-base based estimator for I that works well in practice.
4.1 Mutual Information Estimation
The key challenge in computing I is that computing p(o) requires full knowledge of the dynamics of the Markov Game, while computing p(a) requires marginalizing over all observations (which can be intractable when, e.g., the observation space is high-dimensional). Hence, of T(a; o). using data {(a, o) a ~ T} collected while exccuting policy Note an unbiased estimator has I= Ex [7] The reward for rationally inaltentive actor is then computed as re = T XI., (at,s 0r.))
use Monte Carlo estimate (a oz)
Estimating Mutual Information using Discriminators. Given a pair (a, o) for a single agent (omitting subscripts for clarity), we estimate [x from the ratio between log blg o) (the log-odds under the joint distribution) and logr(a)p(o) (the log-odds under the factorized distributions). This ratio can be estimated using a discriminator model (a that Icarns classify whether the sample (a, o) came from the joint or factorized distribution. We generate samples from p(a,o) by sampling trajectories using म, while samples from p(a)p(o) can be generated by creating random pairs (a, from batch of sampled trajectories. This approach is inspired by the structure of Generative Adversarial Networks Goodfellow et al. (2014), which use discriminators to learn to synthesize images that are almost indistinguishable from real images. This technique for MI estimation can be applied for any pair of variables within some generative process. We leverage this generality throughout RIRL.
Our framework compatible with any MI estimator. Our approach is simple and effective, although other MT estimation techniques have been proposed, e.g., Belghazi et. al. (2018).

---

Published in Transactions on Machine Learning Research (12/2022)
Limited variance of our mutual information estimator. In principle, this sample- based estimator could have high variance, but in practice, even using a single sample to compute i is effective. In particular, estimating 1 using data sampled from the RL policy works well.
pla)ploj. not the individual probabilitics itself. Second, the RL policy typically changes slowly during training. Hence, there is only a small amount of distribution shift in the rollout samples across training sleps. So using a single example could be high variance, but training still works when training the discriminator slowly enough. Third, the pair (a,0) of a randomly sampled a and a randomly sampled can be taken from a batch of trajectories that were collected under different states 8, because such randomly sampled pairs are more likely to be independent across trajectories as the policy changes.
There are multiple reasons for this. First, we train discriminator to estimate the ratio pla,0)
As such, if we sampled many trajectories from the same policy म, and given discriminator with enough model capacity, it should have the capacity to learn the true odds-ratios. With smaller number of samples, training the discriminator needs to be done slowly enough, but in practice, our approach is sufficient to learn meaningful MI estimates, ie., that sufficiently distinguish independent vs dependent samples.
4.2 Recurrent Policies for Sequential Settings and Heterogeneous Attention Costs
In addition, we aim to apply our framework in the sequential setting and with heterogeneous at tention costs. To do so, we use a recurrent model that can summarize historical observations. In particular, such models can learn to strategically allocate attention over time.
Before describing details, we provide the following rough sketch of our policy architecture: (i) The observation is first separated into distinct "channels". (ii) For each channel, there is an encoding module which converts its portion of the observation into noisy encoding. (iii) The encodings from each channel are combined and then fed into a recurrent neural network model to update its recurrent state. (iv) The combined encodings and the updated recurrent stale are fed inlo a stochastic action module to produce a probability distribution over actions and sample accordingly.
4.2.1 Encoder Modules
For each channel, we use the reparameterization trick Kingma & Welling (2013) to compute the mean and standard-deviation of Gaussian distribution over yt", which the noisy encoding of of":
m n = (o," ht), Pe ot
(8)
Vi" = of m
+ He + of c,; m 1(),
(9)
Here hy is the recurrent state of the recurrent model (see below), and e,th is a random sample from a spherical Gaussian with dimensionality equal to that of y and om
This decomposition allows us to separate out attention costs for distinct channels of information. In other words, for each channel we want. to add attention cost proportional to the amount of information about 6t encoded in y" To that end, analogous to Section 4.1 for each channel m also train discriminator Drr [om", hel) used to estimate I. (y7": [ot he]), where brackets denote concatenation.
4.2.2 Recurrent Stochastic Action Module
The full encoding y, [....... T of or concatenates all M encoder samples. We introduce recurrence by layering recurrent neural network model that takes and hy as inputs and outputs an updated recurrent state M++1. In our experiments, we use LSTMs (Hochreiter & Schmidhuber] 1997). although many other variations exist in the machine learning literature. The encoding yt and recurrent state h441 are used as inputs to the decoder w(a (y. h441) which outputs a distribution over actions, from which a, is sampled. As with the encoders, we can train discriminator (ay, (y. h(t1)) used to estimate Is (at, [yr het1l), introducing additional attention cost capturing how much of the encoded information is used to select. an action.
7

---

Published in Transactions on Machine Learning Research (12/2022)
4.3 Optimizing with Reinforcement Learning
Given the model structure discussed previously, we now solve the optimization problem (Equation 2 using reinforcement learning (RI.) Sutton & Barto 2018). Denoting the weights of a policy by 0, we update v (alo; 0) using policy gradients (Williams
1992):
A T
Σ Vo log raa,, yolo,, hi :
R
Σ k=
(10)
log π (a,, Ve yelor, hy 1et1) layle h141) +
Σ
log "oe""""""""""" |o he).
(11)
m=1
M
= rr-(s, - Au (atss [ve, het1l) - Σ
Am I' (vt"; [o, hel)-
(12)
m=1
For further details on training and model implementation, see the Appendix.
5 Validating RIRL in Principal-Agent Problems
We demonstrate RIRL's modeling capabilities in two Principal-Agent (PA) problems of varying complexity. We specifically consider manager-workers relations, and in both settings, a boundedly rational manager (the Principal) decides a compensation ion structure for the workers (the Agents). In general, labor is costly to the Agents but beneficial to the Principal, while the Agents benefit from income that costly for the Principal to provide. Therefore, the Principal aims find a wage schedule W that maximizes its profits. PA problems often consider how information asymmetry between the Principal and Agent(s) influences equilibrium sched- ules. RIRL can analyze meaningful extension in which we model how the cost of information influences equilibrium schedules, i.e., where the Principal can spend effort to reduce information asymmetry. We focus on studying bounded rationality in the principal as is common in prior work (Mirrlees) 1976..
We first study a single-timestop PA setting with a Principal that optimizes a wage schedule for each level of Agent output, subject to a coguilive cost for observing output. Because some aspects the solution are tractable with prior analytical methods, this simple setting allows us to draw comparisons and provides useful validation of RIRL in a multi-Agent game. However note, even in this simple setting, aspects of the solution (such as the constants) are not tractable analytically. Secondly, apply RIRL to analyze complex sequential PA problem outside the scope of analytical methods of prior work that involves multiple Agents and heterogeneous information channels.
Demonstrating the importance of modeling bounded rationality, our experimental results were able to un- cover several non-trivial insights when bounded rationality assumed that are hard to intuitively predict. Specifically, in our PA settings, insights include (1) Agent reward can increase with principal inattention and a more rational Agent benefits more from Principal inattention due to the Principal setting higher pay schedules. To our knowledge, no prior work has identified the consequences of bounded rationality in both the Principal and Agent. (2) In the case of multiple Agents, boundedly rational Principal can induce Pris- oner's Dilemuna like incentives for the Agents, reducing the welfare of all agents. We discuss these insights along with others in detail below.
5.1 Principal-Agemt with Single Agent, Single Timestep
We first evaluate in PA setting with two actors (the Principal and a single Agent), following classic work (Mirlless 1976: Holmstrom & Milgrom 1987 Spremann 1987 Haubrich 1994). When modeling economic behavior, the reward is the (marginal) reward r, an abstract measure of happiness. We use such terminology throughout the sections. The Agent's labor output 2 ~ h(e) is a stochastic function of its effort action e. The Agent chooses how much to work based on its wage schodule W, which yields wage w as a stochastic function of output 2, i.e., w W(z) W(z.,o). Here, the stochasticity the pay schodule aims to model the Principal's uncertainty over 2, e.g., due to limited attention. By assumption, the Agent accurately
8

---

Published in Transactions on Machine Learning Research (12/2022)
a
Principal Utility
b
Payments under MI Regularization
C
Payments under Ent. Regularization
A:
0:
X:
A:1
Avg. W(2)
A:
Avg. W(2)
3
À:
5
S
MI
-2
Entropy
Constant Pay
MI/Ent. Cost (A)
Agent Output (z)
Agent Output (z)
Figure Bandit. Experiment Results. All results are averaged over 5 random seeds and plotted with 95% confidence intervals. (a) Comparing MI and Eutropy regularization on Principal reward. The constant 0-pay schedule provides a meaningful lower bound. (b, c) Pay schedule means under MI (b) and Entropy (c) regularization (pay schedule standard deviations are plotted in the Appendix).
anticipates the shape of the wage schedule W when making its decisions, e.g., based on prior experience or reputation. As such, the RIRI.Principal sets the wage schedule:
W = arg max [E [rp 2)] Aww(w; 2) Mer
(13)
Here, (w;(w;2) is the attention cost for W.
For the Agent, we use a soft-Q policy Th (elw) (rlw)))))) /z with normalization Z, where Ta (c/w) denotes the expected reward of effort e given wage w, sampled from the schedule W. Note that _ can be calculated directly, Importantly, the soft-Q policy models form of bounded rationality, i.e. the log-odds of effort is proportional to its expected reward, with higher 3 making "better" actions more likely. Hence, low B corresponds to ass-thaa-rattona behavior.
The reward of the Principal (rp) and Agent (ra) follow standard economic reward functions:
Ta ra(w,e) e) CRRA (w
rp(w, and
(14)
Income reward
Work Disreward
Profi
Revenue
Amount Paid
Here CRRA is the concave Constant Relative Risk Adverse function (Pratt 1978); the risk aversion parameter p sets its concavity (we use 2). This models diminishing returns with income increases. More environment details are in the Appendix. We use N-sample policy gradients on estimates of Equation parameters H, and 02 of W
to learn the
1 N
[rp(w, 24) Xiw(wa2)), wy W(z), ~ h(ei),e (((())
(15)
Results. Figure shows the effects of the Principal's inattention on simulated outcomes. We also compare RIRL outcomes against those found including entropy-based rewards. MI-based and entropy-based regular- ization are closely related (Grau-Moya et al. 2018 Leibfried Grau-Moya 2020), with subtle but important differencesP] Nevertheless, they yield markedly different outcomes.
To highlight important difference, consider the constant O-pay schedule. This provides incentive for the Agent to work and costs the Principal nothing, and it therefore provides reasonable lower bound Ty. This lower bound should hold under bounded rationality, since no attention is required when W treats all outputs identically. Indeed, under RIRL, Tр approaches this lower bound as increasing X leads the Principal to trade more profitable W's for ones with smaller demands on attention (Figs. 1h, b. 2c). . In contrast, under entr regularization, increasing À quickly yields W's that violate the Tp lower bound [T
Under some conditions, they are identical. However, the key difference is that entropy regularization penalizes the policy deviating from a fized, uniform prior, whereas RI-style MI regularization penalizes deviations from an optimal prior.

---

Published in Transactions on Machine Learning Research (12/2022)
(Least deterministic)
Baw3
Band
5
Ba*7
(Most deterministic)
a
Agent Utility
Principal Utility
Payment Mutual Information
0.2
1.5
3.5
I(w: 2)
-3
3
2.5
0.0
0.0
2
2
Mutual Information Cost (A)
Figure 2: Additional Bandit Experiment Results. All results averaged over 5 random seeds and plotted with 95% confidence intervals. Results are shown for each level of MI cost (X, x-axis) across multiple levels of Agent policy temperatures (3, color). (a, b) Agent and principal reward. (c) Mutual Information between output z and wage w.
Figure 2 shows how Agent and Principal reward and T(u;)) change as a function of X, across range of Agent 3's. Note that higher 8 increases the odds that the Agent selects the optimal action. This reveals an interesting multi-Agent interaction. As before, we observe that the Principal's reward rp (without the attention cost) decreases with increasing X. Furthermore, Tp generally lower when / is lower (Fig. 2p), as the Principal must pay more to influence Thf towards different actions.
Interestingly, we observe that the Agent's reward ra can actually increase when the Principal is inatlentive, and that higher & tends to increase the beneficial range of inattention (Fig, 2b). Lastly, B determines how the Principal responds to increasing attention costs. Agents that behave optimally more often (higher B) incentivize the Principal more to pay atlention (Fig.
Comparison With Prior Literature. For this setting, Mirrlees (9766) found the theoretically optimal pay schedule should be of the form W(=) = max(Az + B, C) While they assume a slightly different model of costly attention +
the emergent pay schedules match closely with this work: when fitting A, B, and p to our data, we measure ,2 0.99 for all B. The best fit p was close to the theoretical value for some values of 3 (true p: 2, best fit p: 2.19 for 3 = 5). Interestingly, with B in our model. Notably, the influence of Agent sub-optimality was outside the scope of this prior work. We are able to observe this relationship between P and 3 through the modelling complexity enabled by our RIRL framework.
do observe that the best fitting P tends to increase
5.2 Siincipal-Agent in the Sequential Multi-Agent Setting
We show that RIRL enables modeling sequential Principal-Agent problems with multiple Agents with T' > 1 timesteps. We consider horizons [2, 10] and teams of n. Agents. We assume there are K = 5 possible Agent abilities k. The Principal cannot see the Agents' abilities. Each Agent's ability is sampled randomly at the start of each episode. At each timestep, Agent i's output is:
24, = hig (v, + cx,))
(16)
where it works hig hours and exerts effort e The Principal moves first and sets wage Wi.t. Each Agent moves second: it knows wi, before choosing his and ei,r, and earning income Wi, hig As before, Principal reward Tp measures profit. We define ra following standard reward functions, where the optimal h increases with w. As a consequence of this configuration (demonstrated below), the profit-maximizing wage w; for
Theeir Principal pays cost reduce the noise added to observed output, as opposed our cost; see the Appendix for full details.
10

---

Published in Transactions on Machine Learning Research (12/2022)
Wage agent (w)
Principal:
Agents:
Each Agent:
Hours
Cumulative Output (2))
Individual Hours Worked (h)
Effort
Output
Individual Effort (e)
Abiliyy
vidual Output Contribution (x)
Figure A depiction of a single timestep an episode of our Soquential Multi-Agent Sctting. Green variables (such as cumulative output) are not costly while blue variables (such as individual output) are costly for the principal to observe. The red variable (Ability) cannot be observed.
Agent i increases with its ability v. The Agent reward is:
Ua Ua(w, h, e) = CRRA(W - h) Income reward
a
Up(w, h
Σ
Σ
wahii
(17)
vE(nal
iE
Work Disreward
Profit
Revenue
Wages Paid
where p, and a are constants governing the shape of Ta-
Attention Costs. Because a strategic Principal must infer private Agent features, c.g., ability, its equi- librium behavior depends on any inference costs it experiences, e.g, allention costs. Note that, unlike in the above bandit setting, here we train both the Principal and the Agent policies, each of which is modeled using the RIRL-actor architecture (Section [ However, isolate and explore the effects of distinet Principal attention costs, do not impose any attention costs the Agents. Therefore, the Agents' reward is only their reward ri, = U. (Wi,t bi.e: e.,r).
For the Principal, we model M = 3 information channels: one is "easy" and low-cost to observe and two are "hard" and high-cost. The low-cost channel of includes information that we regard as freely available (A = 0), c.g., the time t, the hours worked h (workers often fill out timeshects which makes h; casy to see), and the lotal output, Z = EiE(na) 2j (managers can see the final result). However, it is high-cost to see individual contributions: we use a high-cost channel of for efforts e, and a high-cost channel of for outputs z. This models a Principal who can spend time and attention to observe individual Agents to reduce uncertainty about their true ability, c.g., their working styles and productivity (Figure 3). Modeling the attention cost of output and effort separately lets study the effects and interactions of unequal observation costs. The Principal's reward is
p.. Tpp(w, Zt)
xi(ri Individual Output and
Xi(yfie)) et)
(18)
Effort Perception Cost
Thus, the Principal's bounded rationality is modeled through the cost to get information about effort and individual outputs. The Principal's RIRI-actor architecture would also let us use attention costs i(u! of) and [(wry)) but we omit those here Additional training details are in the Appendix.
Results. We highlight several noteworthy observations revealed by our RIRL framework: (1) Principal inattention to individual outputs z and inattention to efforts e have distinct, nearly opposite, consequences on actor utilities; (2) Agents respond to the incentive structures created by Principal inattention to effort, with parallels to a prisoner's dilemma; (3) The temporal patterns of Principal attention reflect the value of information over time.
Implications of Inattention Differ by Agent Ability We analyze welfare related actors' utilities. At cquilibrium, the Principal and Agent utilitics are negatively correlated (Figure 4b,b), comparing across
We do add entropy regularization over w(ay)) for both the Principal and Agent encourage exploration.
11

---

Published in Transactions on Machine Learning Research (12/2022)
Agent Utility (All Types)
b
Ar Ap
Agent Utility (Type: k=O)
Agent Utility (Type: km4)
Principal Utllity
A==
A
Cutput MI Cost (A.)
Principal Uiiliyy (U6)
Avg. Wage (3/hr)
Cutput MI Cost (k-)
100
60
6.5
0.3
Effort MI Cost (Aa)
AgentUtillty (Ua)
Agent Type (K)
Effort MI Cost (Av)
Figure 4: Sequential, multi-agent experiment results. All results averaged over 20 runs and generated
with T = 5. (a, b) Principal and Agent reward heatmaps (a) and scatter plot (b), for each A and X (c) Average wage for each Agent type, under rational (X* = O, yellow) and boundedly rational (X= = 6, blue) Principal. Error bars denote to 95% confidence intervals. (d) reward for the lowest (k: = 0) and highest (A = 4) ability Agent types, for each X and X
a
Agent Skill "Signaling" Behavia at
b
Agent Wilfare across Equilibria
Principal's 's Altention to Outpul over Time
Effort on Filst "imestep (ex-i)
Equaliity of Liility across Types
0.90
MI of Ourput Crcoding, (()e)))))
5/2
T:8
1.0
0.55
10
4.5
0.8
()ooess))
4.
0.6
075
3.5
3.25
7.00 Average Agent Utility (Ub)
Effort MI Cost (As)
Timestep (1)
Figure 5: Additional sequential, multi-agent experiment results. All results are averaged over 20 runs. Shaded regions denote to 95% confidence. All results except (c) were generated with = 5. (a) Average effort at f for each Agent type. (b) Equality across Agent types vs. Agent reward, for each x and X (c) Output information (zz 2,) encoded over time (when x==3)
varying levels of A and Ra Note that the "rational" model has X= == x 0. This indicates that the Principal's bounded rationality has opposing implications for the Principal and Agents. From Figure Eh, we see that Agents' average reward increases and the Principal's reward decreases when the Principal's attention cost for individual outputs X² increases. Conversely, Agent reward docreases with increasing Principal attention cost on effort X The cause for this as follows. The Principal has a different optimal wage for each ability u as shown in Figure D. When the Principal is fully rational it can use output and effort to accurately infer ability. Similar to what was seen in the bandit setting (above), increasing attention costs (in this case, on output X+) lead the Principal to set wages in manner that is less profitable but also less atlentionally demanding. At the resulting equilibria, the Principal has more uncerlainly over each Agent 's type and adopts a "better safe than sorry approach" " and increases the average wage to ensure output.
However, while the average Agent reward increases with X², it does not increase for all Agent types. Specif- ically, the reward of the (highest) lowest-ability Agent's (decreases) increases. Hence, the Principal's un- certainty over individual outputs decreases the wage (and reward) differences between Agents of different ability (Fig. dd). This is particularly relevant when considering the equality of reward. A common inequal- ity metric is the Gini coefficient (Gini 1971), computed as a normalized sum of income differences; equality can be defined as eq = - N gigii. Figure 50 shows that Agents' average reward and equality across types both increase for higher output attention cost. X².
Bounded Rationality induces Social Dilemma Dynamics. This setup gives rise to interesting tem- poral strategic behaviors akin to signaling Spence 1973). Note that, while effort e increases Agent's
12

---

Published in Transactions on Machine Learning Research (12/2022)
work disreward without increasing its income, Agents may still choose to exert non-zero effort 0. To provide an intuition for this, first, note that spending effort increases output rate (z per hour h), resembling higher ability v. Second, the profit-maximziing wage wi increases with Agent ability v4. The Principal can't see each V/ but can see the output 2, and hours hę. Hence, output rate is a "signal" of ability and Agents can misrepresent their ability by spending elfort.
When ' increases, it becomes costly for the Principal to distinguish between the Agent's ability and effort- action. Interestingly, this leads to signaling equilibria in which Agents choose to more effort (Fig. 4b). resulting in lower average Agent reward and higher Principal reward (Fig, 55). In effect, this Agent behavior has parallels to the "defect-defect" equilibrium in the prisoner's 's dilemma (Shoham & Lcyton-Brown 2008). When higher-ability Agents are not using effort, lower-ability Agents are incentivized to use effort to mis- represent themselves as having higher ability. This incentivizes the higher-ability Agents to also use effort to distinguish themselves from the lower-ability Agents. Figure F shows this effect: effort increases with A at all ability lovels. In effect, increasing X creates equilibria where the Principal enjoys higher Agent effort essentially for free. Interestingly, the effects of effort being costly to observe are nullified if the individual outputs are also hard to observe (Fig. 5p), showing these two cost parameters interact.
RIRL-actors Learn the Time- Value of Information. We also show that RIRL can discover the time- value of intormatioon-cquisitiio and how it depends on the time horizon. Figure 5 shows output information that the Principal encodes Fy. (y{: 21) over time. There is an initial spike in the amount of information at the beginning of the episode and this decreases over time. Intuitively, it is most efficient for the Principal to pay attention to outputs at the beginning of the episode, as this information can be used throughout the episode. Additionally, the initial spike is larger for longer horizons, initial information has more value.
Limitations. While RI is a general model of boundedly rationality, it does not cover all models of human irrationality. For simulations that wish to use more specialized models of bounded rationality, RIRL may not be sufficient. Examples include hyperbolic discounting (Kirby & Herrnstein 1995) to model inno-inconsistent delay discounting (humans tend to prefer rewards in the near future much more than rewards farther in the future) or prospect theory (Tversky & Kahneman 1992) to model boss-aversion. Future research may extend RIRL to include such notions of bounded rationality.
6 Conclusion
We propose a novel framework, RIRL, for modeling Nounded-rationallis in MARL simulations with com- plexity beyond the scope of prior techniques. Our method incorporates Rational Inattention, an established model of human bounded rationality which uses mutual information to model the cognitive cost of processing information. Mutual information has been used extensively in RL for exploration and skill discovery, but, to our knowledge, are the first to incorporate it for modeling human-like behavior in multi-agent simulations. We evaluate our method in two Principal-Agent problem settings, including a complex multi-agent setting, with multiple information channels with heterogeneous costs. Incorporating bounded rationality leads to different actor strategies and welfare outcomes as compared to under rational assumptions. These results establish RIRL as a promising framework for using MARL to analyze systerss of human agents.
7 Ethics and Societal Impact Statements
Our work proposes a framework to model bounded rationality in Multi-Agemt Reinforcement Learning based simulations. Thus our framework may be used to draw implications in simulations modeled after real- world systems. While addressing the rationality gap between human actors and RL actors is an important step towards this goal, there is still much more development needed for achieving the realism required for real-world decision making based on AT simulations. We do not intend our framework to be used to explore mothods to increase discrimination or unfairness in rcal-world systems, instcad for to be used to investigate how to decrease such biases. For our work, and in general for agent based simulations works, clearly stating the current limitations of the models used helps mitigate this issue.
TTiis is simple consequence of how reward functions are defined. See Fig 4 for confirmation.
13

---

Published in Transactions on Machine Learning Research (12/2022)
References
Sanjeevan Ahilan and Peter Dayan. Feudal multi-agent hierarchies for cooperative reinforcement learning. Workshop on "Structure & Priors in Reinforcement Learning" at ICLR, 2019.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. ICLR, 2020.
Elsa Barazza and Neil Strachan. The impact of heterogeneous market players with bounded--ationaaity on the electricity sector low-carbon transition. Eneryy Policy, 138:111274, 2020.
Ruairidh M Battleday, Joshua C Peterson, and Thomas L Griffiths. Modeling human categorization of natural images using deep feature representations. arXiv preprint arXiv:1711.04855, 2017.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. TCML, 2018.
Eric Bonabeau. Agent- besnt-based modeling: Methods and techniques for simulating human systems. PNAS, 2002.
David D Bourgin, Joshua C Peterson, Daniel Reichman, Stuart Russell, and Thomas L Griffiths. Cognitive model priors for prodicting human dccisions. In International conference on machine learning, pp. 5133- 5141. PMLR, 2019.
Aysun Bozanta and Ashhan Nasi. Usage of agent-based modeling and simulation in marketing. Journal of Advanced Management Science Vol, 2(3), 2014.
Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
J-P Caverni, J-M Fabre, and Michel Gonzalez. Cognitive blases. Elsevier, 1990.
Mark Dean and Nate Leigh Neligh. Experimental lests of rational inattention. 2017.
Edgar A Duénez-Guzmán, Kevin R McKee, Yiran Mao, Ben Coppin, Silvia Chiappa, Alexander Sasha Vezhnevets, Michiel A Bakker, Yoram Bachrach, Suzanne Sadedin, William Isaac, et al. Statistical dis- crimination lcarning agents. arXiv preprint arXiv:2110.1140, 2021.
Ido Erevl and Alvin E Roth. Simple reinforcement learning models and reciprocation in the. Bounded rationality: The adaptive toolbor, pp. 215, 2002.
Mathias Erlei and Heike Schenk-Mathes. Bounded rationality in principal-agent relationships. German Economic Review, 1844:441,,443, 2017.
Benjamin Patrick Evans and Mikhail Prokopenko. Bounded rationality for relaxing best response and mutual consistency: The quantal hierarchy model of decision-makinng making. arXiv preprint arXi::2166.158: 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. ICLR, 2019.
Doyne Farmer and Duncan Foley. The economy needs based modelling. Nature, 460(7256).685-66) 6866 2009.
Tim Genewein, Felix Leibfried, Jordi Grau-Moya, and Daniel Alexander Braun. Bounded rationality, ab- straction, and hierarchical decision-making: An information-theoreiic optimality principle. Frontiers in Robotics and AI, 2:27, 2015.
Corrado W Gini. Variability and mutability, contribution to the study of statistical distributions and rela- tions. studi coonomico-giue giuridici della universita de cagliari (1912). reviewed in: Light, rj, margolin, bh: An analysis of variance for categorical data. J. American Statistioal Association, 66:534-544 1971.
14

---

Published in Transactions on Machine Learning Research (12/2022)
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural informalion processing systems, 27, 2014.
Jordi Grau-Moya, Felix Leibfried, and Peter Vranex. Soft q-learning with mutual-information regularization. In ICML, 2018.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:661.00750, 2016.
Sanford J Grossman and Oliver D Hart. An analysis of the principal-agent problem. In Foundations of insurance economics, pp. 292-340. Springer, 1992.
Jason Siyanda Hartford. Deep learning for predicting human stralegic behavior. PhD thesis, University of British Columbia, 2016.
Joseph G Haubrich. Risk aversion, performance pay, and the principal-agent problem. Journal of Political Economy, 102(():258-2))) 276, 1994.
Jose G Vargas Hernandez and Ricardo Perez Ortega. Bounded rationality in decision-making MOJ Research Review, 2(1):1 8, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short- Term Memory. Neural Computation, 1997.
William Hoiles, Vikram Krishnamurthy, and Kunal Pattanayak. Rationally inattentive inverse reinforcement learning explains youtube commenting behavior. J. Mach. Learn. Res., 21(170):1 39, 2020.
Bengt Holmstrom and Paul Milgrom. Aggregation and Linearity in the Provision of Intertemporal Incentives. Econometrica, 1987.
Tingliang Huang, Gad Allon, and Achal Bassamboo. Bounded rationality in service systems. Manafacturing & Service Operations Management, (5(2):263-274) 2013.
Gege Jiang, Mogens Fosgerau, and Hong K To. Route choice, travel time variability, and rational inattention. Transportation Research Procedia, 38:482-503 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Kris N Kirby and Richard J Herrustein. Preference reversals due to myopic discounting of delayed reward. Psychological science, 6(2):83-89, 1995.
Nick-Marios T Kokolakis, Aris Kanellopoulos, and Kyriakos G Vamvoudakis. Bounded rationality in dif- ferential games: A reinforcement learning-based approach. In Handbook of Reinforcement Learning and Springer, 2021.
Control, pp.
Yoav Kolumbus and Gali Noti. Neural networks for predicting human interactions in repeated games. arXiv preprint arxiv:1911. 032333, 2019.
Dmytro Korzhyk, Vincent Conitzer, and Ronald Parr. Complexity of computing optimal stackelberg strate- gies in security resource allocation games. In Twenty-fourth caai conference on artificial intelligence, 2010.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, et al. Brain- like object recognition with high- performing shallow recurrent anns. arXiv preprint arXi (ele:1009..,,, 2019.
Macicj Latek, RL Axtell, and Bogumil Kaminski. Bounded rationality via recursion. In Proceedings of Eighth International Conference on Autonomous Agents and Multi fulti-Agent Systems (AAMAS 2009), pp. 457-464, 2009.
15

---

Published in Transactions on Machine Learning Research (12/2022)
Felix Leibfried and Daniel A. Braun. Bounded rational decision-making in feedforward neural networks. In UAI, 2016.
Felix Leibfried and Jordi Grau-Moya. Mutual-information regularization markov decision processes and actor-critic learning. In CoRL, pp. 360-373. PMLR, 2020.
Felix Leibfried, Jordi Grau-Moya, and Haitham Bou-Ammar. An citommation-Thhoriiie Optimality Principle for Deep Reinforcement Learning. NewrIPS Workshop on Deep Reinforcement Learning, 2018.
Joel Z. Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marocki, and Thore Gracpel. Multi-agent reinforce- ment learning in sequential social dilemmas. In AAMAS, 2017.
Wei Ji Ma and Benjamin Peters. A neural network walks into a lab: towards using deep nets as models for human behavior. arXiv preprint orXix:2055.021 2020.
Bartosz Mačkowiak and Mirko Wiederholt. Busines Cycle Dynamics Under Rational Inattention. European Central Bonk Working Paper Series, 2011.
Bartosz Mackowiak, Filip Matejka, and Mirko Wiederholt. Rational inattention: A review. 2021.
Tyler Malloy, Chris R Sims, Tim Klinger, Miao Liu, Matthew Riemer, and Gerald Tesauro. Capacity- limited decentralized actor-critic for multi-agent games. In 2021 IEEE Conference on Games (CoG), pp. 1 8. IEEE, 2021.
Gary J Miller. The political evolution of principal-agent models. Annu. Rev. Polit. Sci., 8:203-225, 2005.
James A Mirrlces. The optimal structure of incentives and authority within organization. The Bell Journal of Economics, pp. 105-131, 1976.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically moti- vated reinforcement learning. NewrIPS, 2015.
Sendhil Mullainathan. A memoryybaas based model of bounded rationality. The Quarterly Journal of Economics, 117(3):735-774 2002.
Pedro A. Ortega, Daniel A. Braun, Justin Dyer, Kee-Eung Kim, and Naftali Tishby. Information- Theoretic Theootrttii Bounded Rationality. arXiv preprint arXiv:1512.06783 2015.
Zhen Peng, Tim Genewein, Felix Leibfried, and Daniel A Braun. An informationn-theoreie on-line update principle for perception-action coupling. In 2017 IEEE/RS International Conference on intelligent robots and systems (IROS), pp. 789-796. IEFE, 2017.
John W Pratt. Risk aversion in the small and in the large. In Uncertainty in economics, pp. 59 79, Elsevier, 1978.
Ardeshir Raihanian Mashhadi and Sara Behdad. Environmental impact asscssment of the heterogeneity in consumers' usage behavior: An agent-based modeling approach. Journal of Industrial Ecology, 22(4): 706 719, 2018.
Hang Ren and Tingliang Huang. Modeling customer bounded rationality in operations management: A review and research opportunitics. Computers & Operations Research, 91:48-58, 2018.
Yuliy Sannikov. A contimuous-time version of the principal-agent problem. The Review of Economic Studies, 75(3):957 984, 2008
Zhenyu Shi, Runsheng Yu, Xinrun Wang, Rundong Wang. Youzhi Zhang, Hanjiang Lai, and Bo An. Learning expensive coordination: An event-based deep rl approach. In ICLR, 2019.
Yoav Shoham and Kevin Leyton-Browa. Multiagent systems: Algorithmic, game-theorete and logical foun- dations. Cambridge University Press, 2008.
16

---

Published in Transactions on Machine Learning Research (12/2022)
Tianmin Shu and Yuandong Tian. M3rl: Mind-aware multi-agent management reinforcement learning. TCLR, 2019.
Herbert A Simon. Models of man; social and rational. 1957.
Herbert A Simon. Bounded rationality in social science: Today and tomorrow. Mind & Society, 1(1):25-33 39, 2000.
Christopher A Sims. Implications of rational inattention. Journal of monetary Economics, 50(3):665 6600, 2003.
Michael Spence. Job Market Signaling. The Quarterly Journal of Economics, 1973.
Klaus Spremann. Agent and Principal. Agency Theory, Information, and Incentives, 1987.
Susanne Still and Doina Precup. An conommmtiio-thhoretie approach to curiosity-driven reinforcement learn- ing. Theory in Biosciences, 131(3):139-143 2012.
Richard S. Sutton and Andrew G. Barto.
Reinforcement learning: An introduction. MIT press, 2018.
Alexander Trott, Sunil Srinivasa, Douwe van der Wal, Sebastion Haneuse, and Stephan Zheng. Building Foundation for Data-Driven, Interpretable, and Robust Policy Design using the Al Economist. arXiv preprint Xrx::22108. 02904, 2021.
Amos Tversky and Daniel Kahneman. Advances in Prospect Theory: Cumulative Representation of Uncer- tainty. Journal of Risk and Uncertainly, 5:297-323, 1992.
Ying Wen, Yaodong Yang, Rui Luo, and Jun Wang. Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. arXiv preprint wrXiv:1901.99211 2019.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 1992.
Diana Yan Wu and Kay-Yut Chen. Supply chain contract design: Impact of bounded rationality and individual heterogeneity. Production and Operalions Munagement, 23(2):253 268, 2014.
Yaodong Yang, Lantao Yu, Yiwei Bai, Jun Wang, Weinan Zhang, Ying Wen, and Yong Yu. A Study of Al Population Dynamics with Million-agent Reinforcement Learning. In AAMAS, 2018.
Stephan Zheng, Alexander Trott, Sunil Srinivasa, Nikhil Naik, Melvin Gruesbeck, David C Parkes, and Richard Socher. The ai economist: Improving equality and productivity with ai-driven tax policics. arXiv preprint arXix:2004.130 13332, 2020.
Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C. Parkes, and Richard Socher. The Al Economist: Optimal Economic Policy Design via Two-level Deep Reinforcement Learning, arXiv preprint arXiv:2108.02755, 2021.
A Additional Bandit Experiment Details and Results
A.1 Additional Details
Noise Structure. Recall in the bandit experiments, the output is noisy function of the hours action h chosen by agent. Specifically we use the following output distribution:
0.7 =
p(=[h)- =
(19)
0-7
otherwise
0.3
Where e is a numerically calculated factor such that p(2[h) a proper probability distribution satisfying Evez p=2[h) = This noise distribution assures high correlation between and 2 with the probability of levels decreasing exponentially as the difference between and grows. We plot the probability heatmap in Figure 6al
17

---

Published in Transactions on Machine Learning Research (12/2022)
Constant Relative Risk Aversion (CRRA) Function The constant relative risk aversion (crra fune- tion (Pratt] 1978) commonly used in economics has constant risk aversion which means decisions are in- variant to scale. It is concave function where the risk aversion parameter, p 2 0, determines the concavity, This models diminishing returns with higher values. The function is given by:
pf 1020 CRRA(u, p) = In(u) p=
(20)
Inattention Model From Theory. The model studied in classical thcoretical Principal-Agent literature we compare with MMrrlless 1976) utilizes the same reward functions for the principal and the agent. It has slightly different model of principal inattention where it models costly principal inattention with factor 0 that controls the magnitude of the observation noisc in the output signal. Specifically, for a true output level =", the principal observes output = = z" + je where is noise. The principal incurs an altention cost that is a function of 8 and is subtracted from their reward function. While this attention model is different from our mutual information based model, it shares many similar properties. In both our models the inattention introduces noise which makes acting optimally difficult. Additionally in both models, an attention parameter can reduce this noise at cost to the principal and the principal can choose the optimal, reward-maximizing value of this attention parameter. Such similarities explain why, even under different models, our results are similar to those from theoretical analysis.
Training Hyperparamcters. We used learning rates of 1c-3 for the training the principal policy param- eters and 5e-3 for the mutual information classifier. We used a batch size of 128 and trained the principal for a total of 100000 batches. During training we gradually annealed Ao, from 0 to the desired value at a rate of 4/10000 per batch. We average all results across 5 random seeds and we set the random seed for pytorch, numpy, and python's internal random module for each run. We used seeds of [0. 4]. All Axperimenn were run on 16CPU cloud compute machines with 54GB of memory. Given pay schedule which consists of mean and standard deviation parameters (M2, Ox) for each output level E Z, to calculate the principal policy we first sample 100 pay schedules and calculate the agent reward per output for each output level for each pay schedule. We then average to calculate the average agent reward for each output level. We use the noise structure to calculate the average reward for each action and use the soft-q formulation over the utilities given in the main text to obtain the agent's 's stochastic policy.
experiments
A.2 Additional Results
We provide a few additional figures for the bandit experiment. Recall the principal's policy was the pay schedule parameterized by mean and standard deviation values for each output level (12, oz). Figure Gb shows the standard deviation parameters aN learned under MI and Entropy regularization for different cost factors A. Inereasing entropy regularization results in very large increascs to the standard deviation parameters while MI regularization leads to much smaller standard deviation increases. The pay schedule means for Entropy regularization become sharper to overcome this increase in standard deviation.
Figure 6c compares pay schedule means (H²) across different Agent policy temperature (5) values. For low inattention costs, the pay schedules are similar. However the pay schedules for higher 3 (more deterministic agents) docrease slower with increasing principal attention cost. Given the pay schedule paramcters, the agent has an optimal h action. More deterministic agents take this optimal action with higher probability it is more valuable for the principal to incentivize them to higher optimal actions, which in turn lead to higher outputs. This explains why Agent and Principal reward are higher for higher / values.
B Additional Multi- Multi- Timessep Experiment Details and Results
B.1
Additional Details
Training Hyperparameters. We used learning rates of 1c-4 for the Principal and Agent's policy param- eters and 1e-3 for all the mutual information classifiers. We used batch size of 512 episodes and train the
18

---

Published in Transactions on Machine Learning Research (12/2022)
Noise Distribution: P(z,h)
Pay Schedule Std. Dev MI Regularization
Pay Schedule Std. Dev Entropy Regularization
2
Hours (h)
0.4
4-
0.4
8'
0.2
0.2
A:
3:
8-
A:1
-
A:5
0.0
10
2
4
6
10
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
Output Level (z)
Output (z)
Output (z)
(b)
Pay Schedule Means
0.75
A, -
Nr. 0 (S)
0.75
3 (S)
B.
0.50
B 7
0.25
Uz.
0.25
Hz.
0.25
0.00
0.00
0.00
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
Output (z)
(c)
Figure 6: Additional bandit experiment results. All results were averaged across 5 random seeds and have 95% confidence regions shaded. (a) The noise distribution between hours and output (P(zlh)). (b) The pay schedule standard deviation plots for MT and Entropy Regularization. (c) A comparison of pay schedule means across different Agent policy 3 values.
principal and agent through 60000 batches. We train a single RIRL-actor for the Agents and concatenate the experiences of all na Agents when updating the policy. The Agent policy therefore effectively has batch size of 512n,. To avoid vastly different total episode returns, we scaled the rewards by the horizon during training. We run all experiments on 8CPU cloud computing machines with 26GB of memory. We average all results across 20 random seeds and we set the random seed for pytorch, numpy, and python's internal random module for each run. We used seeds of [0, 19].
B.2 Additional Results
We present some additional results for the sequential multi-agent experiments. Figures 7a 7b show Principal reward with output MI cost (X*) and effort MI cost (X+) for different horizons. We see the trend of Principal reward decreasing with X= and increasing with X occurs across all horizons.
Figure 7c shows unscaled MI cost for output and effort with and X We see that [yylee) decreases with both X and x T(y=|2) decreases with X but not affected greatly by X
Figure 7d plots the agent reward for all agent types. It expands the plot of Figure 41. The plots show the difference between the reward for each agent type and the average reward.
Figure 7c demonstrates how different values for different agent types change with output MI cost (A+). We plot each result. as the change from the x= = 0 values. As mentioned previously, reward increases with )= for lower ability agents while it decreases for the highest ability agent. We observe similar trends for hours h. We additionally notice effort generally decrcases which leads to staying constant decreasing for all agent types, even though some agent types have increased hours. This lower output along with higher average wages explains the decrease in Principal reward.
19

---

Published in Transactions on Machine Learning Research (12/2022)
Ry
Ny6
110
Principa Utility (Up)
Principal Utllity (U.)
Output MI Cost (A,)
15
110
70
15
90
105
Honzzn:
Honzen:
10
Hertzon:
100
Horzon:
Hortzon: 30
Horizon:
10
2
4
6
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
Output Cost (Ag)
(Ae)
Effort MI Cost (Aa)
Effort MI Cost (A,)
(a)
(b)
(c)
U2 U.
U2 - Ü,
U3-Ú, Ü,
U₂ U.
U3 Ü,
Agent Type: 0
Agent Type: 1
Agent Type:
Agent Type: 3
Agent Type:
Output MI Cost (Az)
2
0
4
-2
8.0
0.5
1.0
0.0
0.5
1.0
0.0
1.0
0.0
0.5
1.0
0.0
1.0
Effort MI Cost (Ap)
(d)
Agent (U,)
AT n
0.0
4E
Hoars (h)
-0.5
Output
Effort
-1.0
-1.5
-15
Output MI Cost (A))
Output Cost Vol
Output MI Cost (.))
Output MI Cost (A))
Figure Additional sequential, Multi- Multi-Agent experiment results. All results are averagod over random seeds and all confidence regions depict 95% confidence intervals. (a, b) Prineipal reward with Output (X*) and Effort (X*) MI Costs. (c) The unscaled Output and Effort MI with X= and A5, (d) The different between each agent type's 's reward and the mean reward across all types with X and X (c) How agent reward, effort, hours, and output changes with respect to Output MI Cost X= for each agent type. All values are plotted as the change amount. from the X 0 value.
C RIRL Implementation Details
We use this section of the appendix to cover important implementation details and tips. This intended for practical guidance.
Stochastic Encoder Module Configuration and Initialization. Scction 4 describes the architecture of our RIRL-actor policy class. As described, we learn an encoder f ("/"" 1h) for each observation channel m. Encoder fm takes observation of and recurrent state h as inputs and outputs the parameters (means stochastic encoding y/"
and standard deviations)
We find that, in practice learning does not progress if each yd contains very little information about Tt the start of training. To address this, recommend two implementation choices. First, implement fm
20

---

Published in Transactions on Machine Learning Research (12/2022)
residual-style module:
,0"00"" fo"" v/==="
= of + p. o% et
(21)
This simply requires setting the output ym to have the same size observation o and adding of to the mean 14. Second, initialize the output layer of f such that 6th is consistently very small. We perform this by adding constant negalive offset to the bias units associated with the log 6 outputs, which we exponentiate to get omh As a result of this strategy, y4" closely follows of" at the start of training.
Hidden State as an Encoder Input. We emphasize that the inputs to encoder f is the concatenation of the observation of and the hidden state ht. Similarly, when using the discriminator d' (""", ,oo", ht]) to estimate f and when training the discriminator, we also apply this concatenation. In other words, dmd regards 1ot" hh] as the observation, such that the f captures the MI between y/" and [o]".....
This is an important detail for ensuring that MI regularization works as expected in the multi-step setting, For instance, we observed that, if the discriminator does not see the hidden state h, f learns to encode om such that ; , is minimal but where of" can still be easily recovered given y²" and h.
Optimization During training, we found that learning was most stable separate learning rates were used for the policy modules {f'...,, SMM LSTM, w) and for the discriminator modules {d',... "ef.....de dM d''). Im- portantly, the discriminator modules use a 10x higher learning rate. Concretely, we use learning rates of 0.0001 and 0.001 for the policy and discriminator modules, respectively. Configuring learning rates this way helps to ensure that discriminator dm can adjust to changes in encoder fm faster than the encoder can adapt to changes in the discriminator. Intuitively, this improves the quality of the MI estimates during training.
Another important optimization detail concerns gradient flow. Gradients from V log ((rl)) need to back- propagate through the encodings "......,," in order for the encoder modules to receive meaningful gra- dients. In Pytorch, which we use for this implementation, ensuring that this gradient flow occurs requires some attention. Given the (lcarned) mean and standard deviation paramcters (which are functions of the en- coder input), Pytorch constructs the sampling distribution as (), The oulpul of the encoder module y!" must be sampled via the reparameterization trick: y m. rsample((), which allows gradients to flow through y," Finally, this output should be detached from the backpropagation graph when calculating its log probability: encoder_ Log_proo m. log__prob( y. ()tach() ) which is needed for computing the policy gradients. Similarly, care must be taken to ensure that inputs to the discriminators are detached in the same way.
21


Reviews:
[{"review_id": "pICGArMtbM", "reviewer": "Reviewer_6GV3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n- Incorporating human-like rationality into RL is an interesting problem.\n- The paper is generally well-written and easy to follow.\n- The emergent behavior experiments are interesting, especially the multi-agent experiments.\n\nCons\n- The idea of modeling bounded rationality with mutual information is not novel. In [2], as Eq.8 shows, mutual information has been used.\n- Related work is not sufficient. Maybe the authors should discuss [1,2]\n- The single-agent experiment is a toy example. Because the paper mainly discusses MARL, while the single agent experiment degrades to a bandit situation.\n- The empirical results, though interesting, are not complex enough. \n- Human-like behavior modeling is important to the MARL community, but I am not sure whether Principal-Agent problems can be the main part of human-like behavior modeling. Maybe the authors should consider more scenarios.\n \nMinor\n- In Eq.13, e_i is sampled from \\pi(e|W), however W is a function. The authors should clarify how this soft-Q policy takes a function as one parameter.\n- Why stochasticity can be modeled as a Gaussian distribution for the wage distribution?\n\n[1] Herbert A Simon. Theories of bounded rationality. Decision and organization, 1(1):161–176, 1972\n\n[2] Tim Genewein, Felix Leibfried, Jordi Grau-Moya, and Daniel Alexander Braun. Bounded rationality, abstraction, and hierarchical decision-making: An information-theoretic optimality principle. Frontiers in Robotics and AI, 2:27, 2015.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is about modeling bounded rationality in multi-agent reinforcement learning (MARL). RL is often modeled as an MDP problem. So, the objective of RL, to maximize the cumulated rewards, requires agents to be rational, which is not human-like. To this end, this paper tries to build human-like agents with bounded rationality. \n\nSpecifically, the authors propose Rational Inattention Reinforcement Learning (RIRL), where bounded rationality in this paper is modeled by mutual information. The intuition is that the agent should take actions that are irrelevant to current observations. Such bounded rationality is used as an additional reward/cost in the RL formulation. The authors propose several channels, i.e., I(o;a), I(y;o)+I(a;y), I(y;o)+I(a;[y1,...yM]), where y is obtained by VAE. Besides, due to the challenges to compute mutual information, the authors use a discriminator to estimate mutual information.\n\nThe experiments are based on Principal-Agent problems with a bounded rational Principal.  RIRL is evaluated in two scenarios. One is involved with only one Principal and one Agent. Another is involved with one Principal and a team of Agents. The results show the emergent behavior of RIRL.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Discussion about the related work, especially the difference with [2].\n- More experiments in more scenarios except for Principal-Agent problems.", "broader_impact_concerns": "No further concerns about the ethical impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vXMRxAq1MS", "reviewer": "Reviewer_pDCi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper incorporated human irrationality into reinforcement learning models. The authors motivate the need for such incorporation well, and propose a corresponding rationally-inattentive objective with extensions to more practical setups. The authors also propose a practical implementation of the objective following the literature.\n\n- Strengths\n    - Originality: The setup seems quite original - incorporating rationality concerns to multi-agent RL, although the way to estimate mutual information follows previous literature. \n\n    - Quality and clarity: The paper is well-written - the authors motivate the need to incorporate rationality to RL agents, and propose a rationally-inattentive objective with a practical implementation of the method. \n\n- Weakness\n    - Significance: My main concern for the paper is in its experimental evaluation part. I acknowledge the experiments the authors conducted. However, the bandit setting is not practical enough. In addition, there are a number of well-established MARL benchmarks (multi-agent particle, StarCraft II micromanagement tasks). Can the method scale to the more practical and complex tasks (which may need to revise the environment)? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the incorporation of human irrationality into reinforcement learning models (Rational Inattention model), which has the potential to improve traditional RL models that assume agents behave rationally. The authors propose a rationally-inattentive objective and extend it to allow for multi-step dynamics and information channels with heterogeneous costs. The authors also follow previous literature for practical estimation of mutual information. Experiments are conducted based on a bandit setup and a multi-agent scenario.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Experiments: I think the paper can be improved by evaluating the method on more complex tasks (e.g., StarCraft II micromanagement tasks) besides the domains considered in the paper. \n\n- Writing: I think it would improve the paper by elaborating on more details for Equation 6 and Equation 7 (extension of the method to support multiple channels, etc). It is not very straightforward to understand these equations.\n\n- Discussion about entropy-regularized RL objectives: It would be interesting to also discuss the connection of the rationally-inattentive objectives and entropy-regularized objectives (which can be considered as motivating the policy to be uniform, quite similar to what is done here-ignoring part of the observations).\n", "broader_impact_concerns": "I do not have significant concerns on the broader impact statement section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TSJEgs4WYm", "reviewer": "Reviewer_H3aW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe article is reasonably well written and provides an important extension to the modelling of bounded rationality using reinforcement learning agents. The use of an explicit term to model the difficulty of incorporating observed information into a policy is novel to my knowledge, and provides an interesting way to study bounded rationality explicitly rather than implicitly.\n\nThe generalisation to multiple possible observation channels, each with their own intrinsic attention cost makes this even more flexible to model human cognitive processes.\n\nThe results of the Principal-Agent models, particularly when compared against analytical models, and then extended to more complex and analytically intractable variants is well presented and explained. I found it interesting to see the responses of costs of rationality on the decisions made, and hence on the utilities received.\n\n# Weaknesses\n\nReinforcement learning has been used to model bounded rationality in a wide variety of situations. The assumption that RL agents are perfect optimisers doesn't hold in many realistic circumstances. It is a disservice to the article to frame it like this, because the present work provides a good way to _explicitly_ model the bounds of rationality, rather than having those bounds be _implicit_ like it has been done in prior literature.\n\nThe connection to an entropy cost regulariser should be brought up earlier. It is strange to have it be the basic baseline for comparison when it has not been discussed before section 3.1.\n\nThe mixed use of reward and utility can be confusing. It would be better to stick with reward / return, and just mention that in the economics literature it tends to be referred to as the utility (or the other way around).\n\nThe discussion on mutual information estimation should provide more details. For instance, the authors define an unbiased estimator, but then don't address whether their montecarlo estimation is unbiased or nor. In fact, I would suspect that the estimation is highly biased as it comes from rollouts of the current policy, where the $\\log p(o)p(a)$ samples are simply computed from across the batch, and $\\log p(a, o)$ comes from along a single rollout. How does this estimation compare to the real mutual information? The authors claim that even a single sample is good in practice, but no analysis is provided.\n\nThe authors suggested that the mutual information should take into account the hidden information in section 4.2.1. However, later on, in their actual examples, the authors explicitly state that in sequential decision making it is particularly advantageous for the Principal to expend attention early in the episode and have that information available for the rest of the episode. This seems to be in conflict,  am I missing something?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article proposes a new multiagent reinforcement learning algorithm to model bounded rationality by means of a Lagrangian term to the reward function. The particular form of the term is a mutual information between the relevant features (observation) and the actions taken by the agent. This term has an interpretation of modelling rational inattentiveness from economic theory. Authors briefly contrast this to a traditional entropy cost regularisation. The article presents a comparison with analytical models in a simple Principal-Agent setting where exact solution are known, and extend the setting to sequential decision making and multiple observational channels, each with its own attention cost. The authors draw some interesting conclusions and show how the extensions provide a more realistic sets of behaviours for both principals and agents.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Although the authors cite a good number of bounded-rationality articles, there are some important omissions that would enhance the framing of the article:\n\n* Nick-Marios T. Kokolakis, Aris Kanellopoulos & Kyriakos G. Vamvoudakis. \"Bounded Rationality in Differential Games: A Reinforcement Learning-Based Approach\": This article models bounded rationality as a level-k policy instead of full Nash equilibria.\n* Erevl, Ido, and Alvin E. Roth. \"Simple Reinforcement Learning Models and Reciprocation.\" Bounded rationality: The adaptive toolbox (2002): 215. This article uses RL to model bounded rationality agents in iterated prisoners dilemmas, extending prior analytical models in game theory.\n* Hernandez, Jose G. Vargas, and Ricardo Perez Ortega. \"Bounded rationality in decision–making.\" MOJ Research Review 2.1 (2019): 1-8.\n* Duéñez-Guzmán, Edgar A., et al. \"Statistical discrimination in learning agents.\" arXiv preprint arXiv:2110.11404 (2021). This article uses RL agents to model bounded rationality, and explicitly addresses information processing of agents to explain their behaviour.\n\nPlease add more detail to the estimation of mutual information, and how biased is it.", "broader_impact_concerns": "Whenever a technology can be used to model humans, there might be a risk that it can be abused. In this case, the risk is minimal as the tools can be better used for pure understanding than manipulation. No salient ethical concerns exist.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a rationally-inattentive RL learning that uses mutual information as an objective function. The paper compares the approach to the entropy regularization typically used by RL practitioners during training. There are applications of the idea to several different settings (within Principal-Agent problems); the results are interesting and relevant to the community. 

It remains to be seen how easily scalable these ideas are, and several reviewers pointed the lack of evidence on larger domains as one drawback of the paper. However, after some discussion, reviewers agreed that this does not significantly affect the claims substantiated by the paper, and that the paper ultimately satisfies the review criteria described in TMLR's guidelines.

The main problems with the paper have been identified by reviewers, partly due to the vast prior work on these topics: 
- There is only modest novelty. Any clarifications that can be made to highlight the novelty from past work is encouraged.
- There are also missing citations and discussion of related work. 

Please take the review feedback into consideration when finalizing the paper. 
Decision: Accept (poster)