Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Rational Inattention Reinforcement Learning (RIRL), a multi-agent reinforcement learning framework that incorporates the behavioral model of rational inattention. Building on the classical formalization of rational inattention, the objective penalizes for the mutual information between the action selected and the observable state of the world. They also extend their framework to support multiple channels of information with heterogeneous cognitive costs. 

They empirically evaluate RIRL in two principal-agent problems, building on classical models from the economics literature. The first simulation consists of a single agent, while the second simulation explores interactions between multiple agents following the RIRL-actor architecture (although only the principal faces inattention). 
main_review: The idea of bringing rational inattention into multi-agent reinforcement learning setups is very conceptually interesting. While the model for rational inattention seems to be standard in behavioral economics, the incorporation of this model into RL settings appears to be novel. Moreover, the formalization of irrationality as the mutual information between the action and the observed state seems natural in RL settings.

I found the most interesting part of the paper to be the empirical evaluation section for principal-agent problems. In particular, the simulations in Section 5.2 demonstrate how the level of inattention of the principal can significantly impact the equilibrium reached relative to the typical case of fully rational behavior. For example, they are able to elucidate the effects of inattention on individual outputs and efforts, and they demonstrate that the Principal's behavior reflects understanding of how value of information over time changes. 

Some comments/questions:
1. How much do the insights on this paper rely on the particular form of inattention considered? For example, if other metrics are considered besides mutual information, would similar results be observed? 
2. The experiments seem to focus on a single actor following rational inattention. What would happen, for example in Section 5.2, if some (or all) of the agents also faced inattention and experienced attention costs? 
summary_of_the_review: Due to the conceptual novelty of incorporating rational inattention into multi-agent RL, I recommend weak acceptance. 

---
Update after author response: Thanks to the authors for their detailed responses, and for clarifying the results in Section 5.1. I would encourage the authors to provide a more thorough discussion of these comparisons in a future version of the paper. Moreover, although it would be useful to consider multiple agents facing inattention (e.g. in Section 5.2), I found the existing results to be sufficiently interesting for acceptance to ICLR. Thus, I keep my assessment the same.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Standard Bayesian-type decision making models have that given some signal X, our decision makers choose an action A to maximize utility given the signal X. This gives a conditional distribution A|X. Rational inattention models say that the there is limted processing capacity in the agent and model this by requiring that A | X has a capacity constraint (i.e. which under some extra assumptions essentially becomes maximum amount of mutual information). 

The authors put this into RL by, basically, writing a policy P(a|s) into an observation component (O(observation | state)) and an action conditional on the realized observation with an information penalty (so the higher capacity the channels, the more penalty). 

The authors apply this RIRL model to some canonical game theory games.
main_review: Pros:
Injecting more “psychologically plausible” models of human behavior into RL is very important for many applications

Adding the information channel is a simple way to add rational inattention into RL

Cons:
It is unclear to me in which cases rational inattention is the right framework to use. Certainly if you are designing an agent, you probably don’t want to give them rational inattention unless you actually have a channel capacity constraint, so I guess the point here is to model humans? 

In that case, I’m not sure what the gain of using RL is. The authors state: “Prior economics literature mainly use analytical methods and narrow modeling assumptions…Instead, MARL can study complex setups that are analytically intractable”. However, the simple model presented in experiment 1 is absolutely tractable. The second experiment may not be, but it is too simple to be a “real world example” and though it is more complex than the first experiment, it’s not clear what new insights we are supposed to gain.

Finally, there doesn’t seem to be much theoretical guarantees here, and some of the patterns seem to be specific to the details of the game setup/RL setup so I am not sure what exactly we are supposed to take away from the experiments.

Overall, I think this paper needs more convincing results that tell us something general either about PA+RI problems or just RI itself.

summary_of_the_review: Overall, I think this paper needs more convincing results that tell us something general either about PA+RI problems or just RI itself.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a RIRL, which incoporate rational inattention suboptimality into the optimal RL model. The authors simulate RIRL in a single agent and a multi-agent setting. They compare RIRL behavior with the rational model in those settings and discuss the emergent behavior of RIRL. 
main_review: The paper is clearly written. I find the model interesting and the authors discuss some emergent behavior from the model. The model parameters are also relatively interpretable. Coming from a computational social science background, I can see this paper as a nice methodology paper for economics/psychology fields. The authors also promise the sharing of the code, which is a plus.

At the same time, as ICLR is still a more ML venue, the lack of more empirical results can be a big minus. The experiments are all simulations based. They do serve their purpose of illustrating the model and showing the various behavior the model can capture, but it is hard to conclude how well this model can capture actual human behavior or how it might compare to other models. The paper can benefit from some empirical results, though this might not be reasonable to ask given the short rebuttal period.  
summary_of_the_review: The paper can be a nice methodology paper but lacks empirical results for an ML venue.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies human bounded rationality using multi-agent reinforcement learning. It particularly focuses on the rational inattention model and proposes the RIRL framework which allows multi-timestep dynamics and information channels with heterogeneous processing costs. The proposed RIRL framework is evaluated in two specific Principal-Agent problems. 
main_review: Strength:  Using MARL to model human bounded rationality is interesting. It is meaningful to generalize the rational inattention model to allow dynamics of multiple time steps and multiple information channels with different cognitive costs. Moreover, extensive experimental results are presented to show that the RIRL framework can reproduce rich information about bounded rational behaviors. The paper is generally well-written and easy to follow.

Weakness: The extension of the existing rational inattention model is straightforward and the technical contributions of this work seem marginal. The literature review is insufficient. For example, Wen et al. [1] also model bounded rationality in multi-agent interactions using MARL, however, this work has not been discussed. Moreover, the huge literature (e.g., [2,3]) on modeling/predicting human behaviors in games (i.e. multi-agent interactions) has not been mentioned; the works in this line of research not only present new models, but also validate their models through comparisons with real human decision results. 

For the experiments, while I appreciate that the paper presents a quite extensive analysis on the principal-agent problem, I need to say that I can't evaluate the significance/novelty of the results as I am not familiar with the literature on the principal-agent problem.


[1] Wen, Y., Yang, Y., Luo, R., & Wang, J. (2019). Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. IJCAI.

[2] Hartford, J.S., Wright, J. R., & Leyton-Brown, K. (2016). Deep Learning for Predicting Human Strategic Behavior. NIPS.

[3] Bourgin, D. D., Peterson, J. C., Reichman, D., Russell, S. J., & Griffiths, T. L. (2019). Cognitive model priors for predicting human decisions. ICML.

summary_of_the_review: The paper presents a new model that accounts for human rational inattention and evaluates the model in a specific problem, however, the technical contributions seem marginal and the literature review is insufficient.
review_title: null

Meta-review: The paper proposes a multi-agent RL framework that make decisions in a more human-like manner by incorporating rational inattention. The approach is evaluated on two game theoretic problems. The reviewers agree that the topic of the paper is interesting. However, there are concerns about the significance of the proposed approach. As the method incorporates human-inspired limitations, it's aim is not to outperform SOTA RL methods on regularly considered domains; at the same time, as the approach is only evaluated on two simulation-based tasks, it is unclear how it would perform in more realistic scenarios that may benefit from human-like decision making. For these reasons, I recommend rejection.
Decision: Reject