Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning

Abstract: Multi-agent reinforcement learning (MARL) is a powerful framework for studying emergent behavior in complex agent-based simulations. However, RL agents are often assumed to be rational and behave optimally, which does not fully reflect human behavior. Here, we study more human-like RL agents which incorporate an established model of human-irrationality, the Rational Inattention (RI) model. RI models the cost of cognitive information processing using mutual information. Our RIRL framework generalizes and is more flexible than prior work by allowing for multi-timestep dynamics and information channels with heterogeneous processing costs. We evaluate RIRL in Principal-Agent (specifically manager-employee relations) problem settings of varying complexity where RI models information asymmetry (e.g. it may be costly for the manager to observe certain information about the employees). We show that using RIRL yields a rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions. For instance, some forms of a Principal's inattention can increase Agent welfare due to increased compensation, while other forms of inattention can decrease Agent welfare by encouraging extra work effort. Additionally, new strategies emerge compared to those under rationality assumptions, e.g., Agents are incentivized to misrepresent their ability. These results suggest RIRL is a powerful tool towards building AI agents that can mimic real human behavior. 

Reviews and author responses:

The paper proposes Rational Inattention Reinforcement Learning (RIRL), a multi-agent reinforcement learning framework that incorporates the behavioral model of rational inattention. Building on the classical formalization of rational inattention, the objective penalizes for the mutual information between the action selected and the observable state of the world. They also extend their framework to support multiple channels of information with heterogeneous cognitive costs. 

They empirically evaluate RIRL in two principal-agent problems, building on classical models from the economics literature. The first simulation consists of a single agent, while the second simulation explores interactions between multiple agents following the RIRL-actor architecture (although only the principal faces inattention).  The idea of bringing rational inattention into multi-agent reinforcement learning setups is very conceptually interesting. While the model for rational inattention seems to be standard in behavioral economics, the incorporation of this model into RL settings appears to be novel. Moreover, the formalization of irrationality as the mutual information between the action and the observed state seems natural in RL settings.

I found the most interesting part of the paper to be the empirical evaluation section for principal-agent problems. In particular, the simulations in Section 5.2 demonstrate how the level of inattention of the principal can significantly impact the equilibrium reached relative to the typical case of fully rational behavior. For example, they are able to elucidate the effects of inattention on individual outputs and efforts, and they demonstrate that the Principal's behavior reflects understanding of how value of information over time changes. 

Some comments/questions:
1. How much do the insights on this paper rely on the particular form of inattention considered? For example, if other metrics are considered besides mutual information, would similar results be observed? 
2. The experiments seem to focus on a single actor following rational inattention. What would happen, for example in Section 5.2, if some (or all) of the agents also faced inattention and experienced attention costs?  Due to the conceptual novelty of incorporating rational inattention into multi-agent RL, I recommend weak acceptance. 

---
Update after author response: Thanks to the authors for their detailed responses, and for clarifying the results in Section 5.1. I would encourage the authors to provide a more thorough discussion of these comparisons in a future version of the paper. Moreover, although it would be useful to consider multiple agents facing inattention (e.g. in Section 5.2), I found the existing results to be sufficiently interesting for acceptance to ICLR. Thus, I keep my assessment the same. 

 To respond to the reviewer’s comments about the novelty of insights, we copy below the section of our general response about novel insights.

**Insights/Novelty of findings.** In Principal-Agent problems, modeling information-cost based bounded rationality is a direction that was studied but found to be analytically difficult/intractable (Mirrlees, 1976) and further research in this direction is sparse. Therefore we utilize Principal-Agent problems as a direct illustration of the main objective of our paper: to incorporate bounded rationality in agent-based modeling and study emergent behavior under bounded-rationality assumptions not previously possible. As we clarify at the beginning of section 5 in our main text, our results add several non-trivial insights on the effects of bounded rationality that are hard to intuitively predict. We unpack several of these insights below:

- Building on the optimal payment model analyzed in Mirrlees (1976), we use our framework to model a Principal with limited attention to output. RIRL payment schedules agree with the predictions from the theoretical analysis, but we observe a systematic deviation. This deviation stems from a novel aspect of our model: that the Agent is also boundedly rational, such that the probability of pursuing a given output level depends on its expected utility. We see that Agents can actually benefit from Principal inattention, with the range of beneficial inattention expanding with more optimal Agent behavior. On the other hand, when the Agent behaves less optimally, the Principal has less incentive to pay attention, causing payments to more quickly approach 0 with rising attention costs. To our knowledge, no prior work has identified these consequences of bounded rationality in both Principal and Agent in this classic Principal-Agent model.
- Our second setting introduces concepts related to signaling (Spence, 1973), where the Principal must infer qualities of the Agent based on a signal that the Agent can distort at some cost to itself. In our setting, this quality is skill and the associated signal is output per hour. The Principal’s inference requires attention across heterogeneously costly information channels (individual output and effort). We identify distinct emergent consequences of these two attention costs. Inattention to effort creates Agent incentives akin to a Prisoner’s Dilemma. Interestingly, inattention to output mitigates this effect.  
A useful analogy here is a technical interview, where an employer (Principal) uses a relatively standardized test to evaluate a prospective employee (Agent). The test is easy enough to evaluate (“output” attention is not costly), but accounting for how much the applicant practiced for the test would be much harder (“effort” attention is costly). In this case, our results predict an equilibrium where every applicant practices thoroughly. The test remains informative for the employer, but the applicants’ utility suffers.

References:  
James A Mirrlees. The optimal structure of incentives and authority within an organization. 1976.  
Michael Spence. Job Market Signaling. 1973.  
Dmytro Korzhyk, Vincent Conitzer, and Ronald Parr. Complexity of computing optimal Stackelberg strategies in security resource allocation games. 2010.

 Thank you for your valuable feedback.

**Related Work:** Thank you for highlighting areas of related work that we overlooked! We have updated our Related Works to include the suggested papers as well as others. Our work considers different settings, such as agent-based simulations, where prior human behavior data are not yet available. In addition, we consider bounded rationality due to cognitive limits of information processing (e.g., limits on the ability to understand tax implications when deciding whether to pursue higher levels of income) rather than due to incomplete modeling of other agents. We view the works highlighted by the reviewer as complementary to our own, in terms of both the behavioral models considered and the methodological approach.

**Technical Contributions:** To respond to the reviewer’s comments on technical Novelty, we copy the “Technical Novelty” section of the General Response here for convenience to the Reviewer:   
Our method proposes a novel architecture that can be trained end-to-end; it is highly flexible, allowing for multi-timestep settings and for heterogeneous information channels with different processing costs. To do this, we needed to use significantly different methods than prior work. Whereas the single information channel case studied in prior work (Peng, 2017) was able to solve mutual information regularization by enforcing the expected form of the solution, we instead utilize deep mutual information estimation. Additionally, this prior work did not consider the multi-timestep setting, which we consider and account for by utilizing an LSTM to model information processed in previous timesteps. Additionally ensuring correct training for all interconnect components of such an agent required non-trivial technical engineering. We have made these difference more clear, including an appendix section touching upon:
- How to configure and initialize the output head in the stochastic encoder module
- Relative learning rates between policy and discriminator parameters
- Ensuring proper gradient flow
- Using the LSTM hidden state as an input to the stochastic encoder module

References:   
Zhen Peng, et al.  An information-theoretic online update principle for perception-action coupling. 2017


 Thank you for your valuable feedback.

**The importance of agent-based simulation:** We thank the reviewer for highlighting the need to better motivate agent-based modeling (ABM) in our text. ABM studies real-world systems through simulation (Bonabeau 2002). In ABM, simulated agents each execute individual behaviors and the emergent phenomena are observed. Agents can use hardcoded rule-based behaviors or use RL to autonomously learn utility-maximizing behaviors. Recently ABM has increased in popularity and use in diverse real-world applications such as market or environmental impact simulations among others (Bonabeau 2002, Bozanta 2014, Raihanian 2018).  They are useful for exploring and quantifying the potential benefits of certain strategies before making large changes in the real world (e.g., would including an extra lane reduce freeway traffic jams?). We have modified the beginning of the introduction to emphasize the importance of simulation.

**Validation with real world data:** We agree that validation against real-world data is important. We choose to use the Rational Inattention framework due to the wide variety of prior work validating the model using real-world data from diverse domains (traffic, YouTube commenting behavior, macroeconomics, etc). We propose a framework for incorporating this well-established model into MARL and demonstrate the new capabilities it can provide for modeling emergent behavior in agent-based simulations. It is beyond the scope of this paper to collect and evaluate on a relevant dataset. Nevertheless, we are excited by the possibility that this model could be used to develop and evaluate new hypotheses that could be tested in laboratory experiments of human decision-making. We consider this another compelling avenue of possible future work.

References:
Eric Bonabeau.   Agent-based modeling:  Methods and techniques for simulating human systems. 2002   
Aysun Bozanta and Aslıhan Nasır. Usage of agent-based modeling and simulation in marketing. 2014   
Ardeshir Raihanian Mashhadi and Sara Behdad. Environmental impact assessment of the heterogeneity in consumers’ usage behavior: An agent-based modeling approach. 2018

 Thank you for your valuable feedback.

**When should our framework be used?:** The reviewer is correct that our RIRL framework is intended to simulate more human-like decision-making behavior. As rational inattention has been shown to be applicable to many fields (e.g., macroeconomics, youtube commenting behavior), our RIRL-actor is suitable for use whenever the simulated agent is a model of a human agent (e.g., drivers in a traffic light simulation), but may not be suitable for non-human agents where the goal is to maximize some task reward (e.g., the computer system which controls the traffic lights).

**Tractability of the models considered experiment 1:** The reviewer is correct that certain aspects of the solution in experiment 1 are tractable analytically, such as the general form the solution takes. This is by design. We present this experiment to draw comparisons with prior work that studies the problem analytically. However, even in this simpler setting, there are aspects of the solution (such as the constants that define the exact equation of the solution) that are not tractable to analytically obtain (Mirrlees, 1976). In addition, our setting relaxes the assumption of Agent rationality, which further reduces tractability. We have modified the beginning of Section 5 to better clarify how tractable our experiments are with the analytical methods used in prior work.

**The takeaways from the second experiment:** We agree that the second experiment is not analytically tractable and provides a clear case where using RL is beneficial. This experiment (1) illustrates the ability of our RIRL framework to discover reasonable results, and (2) demonstrates how behaviors under a bounded-rationality assumption differ from those under a rationality assumption. This is, to our knowledge, the first study of rational inattention in Principal-Agent systems with multiple agents. Accordingly, we chose to validate these points using a smaller scale experiment to better interpret and isolate the effects of bounded-rationality. Our method can readily extend to larger-scale systems, which we agree is interesting. Indeed, this is very much the goal of future work.

**Theoretical Results:** We thank the reviewer for bringing up an interesting point of discussion. Prior work has shown computing optimal strategies in Stackelberg games, where one player (e.g. the Principal) chooses an action first and the second player (e.g. the Agent) observes the chosen action before choosing its own response, can be NP-hard for the first player (Korzhyk, 2010). Therefore we do not focus on theoretical results in our work. We modified the “Principal-Agent Problems” section of our related work to mention this.

 Thank you for your valuable feedback.

**Comparison to other models:** We agree this is an interesting point and we explore it in our first experiment. We compare it to classical work in Principal-Agent problems (Mirrlees, 1976) which uses a different formalization of costly attention, i.e. that the Principal can incur an information cost to reduce the amount of noise in the observed output. As we describe in Section 5.1, our framework discovers similar solutions to those predicted by Mirrlees, but we observe novel interactions between Principal inattention and Agent stochasticity.   
Additionally we compare the RIRL MI cost with entropy regularization. Maximizing entropy is a common principle in RL and shares formal similarities with our MI objective. We show entropy regularization results in very different equilibria that may not be reasonable under an attention model (such as the Principal setting payments that would result in worse profits than if it had not set payments at all, e.g. always paying 0).

**Agents with inattention costs:** We agree that adding attention costs for *all* actors is an interesting and natural future direction. We focus on Principal inattention for the paper to better isolate and explore the effects of modeling inattention without needing to disentangle various effects. We will clarify this scoping decision in the main text.

References:  
James A Mirrlees.  The optimal structure of incentives and authority within an organization. 1976


 We thank all the reviewers for their thorough and helpful feedback. We are encouraged that all reviewers seem to agree that incorporating rational inattention into MARL is an interesting and important direction of research. We respond here to common concerns and separately respond to each individual reviewer. 

**Insights/Novelty of findings.** A few reviewers were unclear about the insights revealed by the Principal-Agent experiments. We thank the reviewers for giving us an opportunity to clarify this. In Principal-Agent problems, modeling information-cost based bounded rationality is a direction that was studied but found to be analytically difficult/intractable (Mirrlees, 1976) and further research in this direction is sparse. Therefore we utilize Principal-Agent problems as a direct illustration of the main objective of our paper: to incorporate bounded rationality in agent-based modeling and study emergent behavior under bounded-rationality assumptions not previously possible. As we clarify at the beginning of section 5 in our main text, our results add several non-trivial insights on the effects of bounded rationality that are hard to intuitively predict. We unpack several of these insights below:
- Building on the optimal payment model analyzed in Mirrlees (1976), we use our framework to model a Principal with limited attention to output. RIRL payment schedules agree with the predictions from the theoretical analysis, but we observe a systematic deviation. This deviation stems from a novel aspect of our model: that the Agent is also boundedly rational, such that the probability of pursuing a given output level depends on its expected utility. We see that Agents can actually benefit from Principal inattention, with the range of beneficial inattention expanding with more optimal Agent behavior. On the other hand, when the Agent behaves less optimally, the Principal has less incentive to pay attention, causing payments to more quickly approach 0 with rising attention costs. *To our knowledge, no prior work has identified these consequences of bounded rationality in both Principal and Agent in this classic Principal-Agent model.*
- Our second setting introduces concepts related to signaling (Spence, 1973), where the Principal must infer qualities of the Agent based on a signal that the Agent can distort at some cost to itself. In our setting, this quality is skill and the associated signal is output per hour. The Principal’s inference requires attention across heterogeneously costly information channels (individual output and effort). We identify distinct emergent consequences of these two attention costs. Inattention to effort creates Agent incentives akin to a Prisoner’s Dilemma. Interestingly, inattention to output mitigates this effect.  
A useful analogy here is a technical interview, where an employer (Principal) uses a relatively standardized test to evaluate a prospective employee (Agent). The test is easy enough to evaluate (“output” attention is not costly), but accounting for how much the applicant practiced for the test would be much harder (“effort” attention is costly). In this case, our results predict an equilibrium where every applicant practices thoroughly. The test remains informative for the employer, but the applicants’ utility suffers.


**Technical Contributions:** Some of the reviewers have helpfully pointed out our technical contributions may not be clearly distinguished from prior work. Our method proposes a novel architecture that can be trained end-to-end; it is highly flexible, allowing for multi-timestep settings and for heterogeneous information channels with different processing costs. To do this, we needed to use significantly different methods than prior work. Whereas the single information channel case studied in prior work (Peng, 2017) was able to solve mutual information regularization by enforcing the expected form of the solution, we instead utilize deep mutual information estimation. Additionally, this prior work did not consider the multi-timestep setting, which we consider and account for by utilizing an LSTM to model information processed in previous timesteps. Additionally ensuring correct training for all interconnect components of such an agent required non-trivial technical engineering. We have made these difference more clear, including an appendix section touching upon:
- How to configure and initialize the output head in the stochastic encoder module
- Relative learning rates between policy and discriminator parameters
- Ensuring proper gradient flow
- Using the LSTM hidden state as an input to the stochastic encoder module

References:  
James A Mirrlees.  The optimal structure of incentives and authority within an organization. 1976  
Michael Spence. Job Market Signaling. 1973  
Zhen Peng, et al.  An information-theoretic online update principle for perception-action coupling.  2017


Standard Bayesian-type decision making models have that given some signal X, our decision makers choose an action A to maximize utility given the signal X. This gives a conditional distribution A|X. Rational inattention models say that the there is limted processing capacity in the agent and model this by requiring that A | X has a capacity constraint (i.e. which under some extra assumptions essentially becomes maximum amount of mutual information). 

The authors put this into RL by, basically, writing a policy P(a|s) into an observation component (O(observation | state)) and an action conditional on the realized observation with an information penalty (so the higher capacity the channels, the more penalty). 

The authors apply this RIRL model to some canonical game theory games. Pros:
Injecting more “psychologically plausible” models of human behavior into RL is very important for many applications

Adding the information channel is a simple way to add rational inattention into RL

Cons:
It is unclear to me in which cases rational inattention is the right framework to use. Certainly if you are designing an agent, you probably don’t want to give them rational inattention unless you actually have a channel capacity constraint, so I guess the point here is to model humans? 

In that case, I’m not sure what the gain of using RL is. The authors state: “Prior economics literature mainly use analytical methods and narrow modeling assumptions…Instead, MARL can study complex setups that are analytically intractable”. However, the simple model presented in experiment 1 is absolutely tractable. The second experiment may not be, but it is too simple to be a “real world example” and though it is more complex than the first experiment, it’s not clear what new insights we are supposed to gain.

Finally, there doesn’t seem to be much theoretical guarantees here, and some of the patterns seem to be specific to the details of the game setup/RL setup so I am not sure what exactly we are supposed to take away from the experiments.

Overall, I think this paper needs more convincing results that tell us something general either about PA+RI problems or just RI itself.
 Overall, I think this paper needs more convincing results that tell us something general either about PA+RI problems or just RI itself.


The paper proposes a RIRL, which incoporate rational inattention suboptimality into the optimal RL model. The authors simulate RIRL in a single agent and a multi-agent setting. They compare RIRL behavior with the rational model in those settings and discuss the emergent behavior of RIRL.  The paper is clearly written. I find the model interesting and the authors discuss some emergent behavior from the model. The model parameters are also relatively interpretable. Coming from a computational social science background, I can see this paper as a nice methodology paper for economics/psychology fields. The authors also promise the sharing of the code, which is a plus.

At the same time, as ICLR is still a more ML venue, the lack of more empirical results can be a big minus. The experiments are all simulations based. They do serve their purpose of illustrating the model and showing the various behavior the model can capture, but it is hard to conclude how well this model can capture actual human behavior or how it might compare to other models. The paper can benefit from some empirical results, though this might not be reasonable to ask given the short rebuttal period.   The paper can be a nice methodology paper but lacks empirical results for an ML venue. 

This paper studies human bounded rationality using multi-agent reinforcement learning. It particularly focuses on the rational inattention model and proposes the RIRL framework which allows multi-timestep dynamics and information channels with heterogeneous processing costs. The proposed RIRL framework is evaluated in two specific Principal-Agent problems.  Strength:  Using MARL to model human bounded rationality is interesting. It is meaningful to generalize the rational inattention model to allow dynamics of multiple time steps and multiple information channels with different cognitive costs. Moreover, extensive experimental results are presented to show that the RIRL framework can reproduce rich information about bounded rational behaviors. The paper is generally well-written and easy to follow.

Weakness: The extension of the existing rational inattention model is straightforward and the technical contributions of this work seem marginal. The literature review is insufficient. For example, Wen et al. [1] also model bounded rationality in multi-agent interactions using MARL, however, this work has not been discussed. Moreover, the huge literature (e.g., [2,3]) on modeling/predicting human behaviors in games (i.e. multi-agent interactions) has not been mentioned; the works in this line of research not only present new models, but also validate their models through comparisons with real human decision results. 

For the experiments, while I appreciate that the paper presents a quite extensive analysis on the principal-agent problem, I need to say that I can't evaluate the significance/novelty of the results as I am not familiar with the literature on the principal-agent problem.


[1] Wen, Y., Yang, Y., Luo, R., & Wang, J. (2019). Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. IJCAI.

[2] Hartford, J.S., Wright, J. R., & Leyton-Brown, K. (2016). Deep Learning for Predicting Human Strategic Behavior. NIPS.

[3] Bourgin, D. D., Peterson, J. C., Reichman, D., Russell, S. J., & Griffiths, T. L. (2019). Cognitive model priors for predicting human decisions. ICML.
 The paper presents a new model that accounts for human rational inattention and evaluates the model in a specific problem, however, the technical contributions seem marginal and the literature review is insufficient. 

Meta-review: The paper proposes a multi-agent RL framework that make decisions in a more human-like manner by incorporating rational inattention. The approach is evaluated on two game theoretic problems. The reviewers agree that the topic of the paper is interesting. However, there are concerns about the significance of the proposed approach. As the method incorporates human-inspired limitations, it's aim is not to outperform SOTA RL methods on regularly considered domains; at the same time, as the approach is only evaluated on two simulation-based tasks, it is unclear how it would perform in more realistic scenarios that may benefit from human-like decision making. For these reasons, I recommend rejection.

Decision: Reject