SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The probability of an outcome  $o_1$  and an action  $a_2$  for the agent of the strength  $s_1 = i$  was derived from the outcome-action tensor  $[OA]_{ijkl}^{a_1a_1a_2}$ :
$$
P r \left(\left\{o _ {1}, a _ {2} \right\} \mid s _ {1} = i\right) = \sum_ {j k l} \left[ O A \right] _ {i j k l} ^ {o _ {1} a _ {1} a _ {2}}. \tag {16}
$$
The probability for the agent to be of strength  $s_1 = i$  was taken from the belief tensor  $B_{11}^{i}$ :
$$
P r \left(s _ {1} = i\right) = B _ {1 1} ^ {i}. \tag {17}
$$
The marginal probability of the observation  $\{o_1, a_2\}$  was defined by the outcome-action tensor  $[OA]_{ijkl}^{o_1a_1a_2}$  scaled by the beliefs about strength  $B_{11}^i, B_{12}^j$ , etc. to scale the optimal actions with the probabilities of their underlying strengths:
$$
P r \left(\left\{o _ {1}, a _ {2} \right\}\right) = P r \left(o _ {1} \mid a _ {2}\right) P r \left(a _ {2}\right) = \sum_ {i j k l} \left[ O A \right] _ {i j k l} ^ {o _ {1} a _ {1} a _ {2}} B _ {1 1} ^ {i} B _ {1 2} ^ {j} B _ {2 1} ^ {k} B _ {2 2} ^ {l}. \tag {18}
$$
Together, the four equations above formed the update rule for agent beliefs based on their observations:
$$
P r \left(s _ {1} = i \mid \left\{o _ {1}, a _ {2} \right\}\right) = \frac {\sum_ {j k l} \left[ O A \right] _ {i j k l} ^ {o _ {1} a _ {1} a _ {2}} B _ {1 1} ^ {i} B _ {1 2} ^ {j} B _ {2 1} ^ {k} B _ {2 2} ^ {l}}{\sum_ {i j k l} \left[ O A \right] _ {i j k l} ^ {o _ {1} a _ {1} a _ {2}} B _ {1 1} ^ {i} B _ {1 2} ^ {j} B _ {2 1} ^ {k} B _ {2 2} ^ {l}}. \tag {19}
$$
The update could proceed at an arbitrary learning rate  $\varepsilon$ :
$$
B _ {1 1} ^ {i} \leftarrow \varepsilon P r \left(s _ {1} = i \mid \left\{o _ {1}, a _ {2} \right\}\right) + (1 - \varepsilon) B _ {1 1} ^ {i} \tag {20}
$$
We describe additional details regarding the update procedure for the beliefs in Appendix B.4 and summarize them in Supplementary Algorithm 3.
4 Results: inference of animal choice mechanisms in social conflict
4.1 Model fit and comparison
We used the framework defined above to test hypotheses about mouse social conflict-related choices. To test hypotheses against their alternatives, we optimized pairs of models on the training data (88 mice participating for 22 days). To this end, we specified a negative log-likelihood (NLL) function  $\mathcal{L}$  comparing the action probabilities predicted in our model to the mouse actions logged in the experiment:
$$
\mathcal {L} = - \sum_ {m t} \left(\log P r \left(a _ {m} ^ {t}\right) + \log P r \left(o _ {m} ^ {t}\right)\right). \tag {21}
$$
To fit the model parameters, we minimized the NLL regularized with the  $l_{2}$  norm of the model arguments. We chose the regularization coefficient such that it resulted in the best fits in a simulated experiment. For the real mice in the experiment, we used the data spanning all 22 days to propagate the beliefs but the NLL was only computed for the data from days 1-3 and 21-22 to avoid the impact of the repeated actions on days 4-20. We report the implementation details of the model fit in Appendix B.5 and summarize them in Supplementary Algorithm 4.
We then proceeded to compare how well different models explain the behaviors observed in our experiment. Here we follow the standard way to compare models in behavioral research that is using formal comparison criteria, such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC). Both criteria are computed as a function of the number of parameters in the model less the log-likelihood of this model to generate the original data. Most of the models that we considered have the same number of parameters, therefore, as we only evaluated the difference of AICs (or BICs) between the models, the term reflecting the number of parameters cancelled out, resulting in the difference in log-likelihoods that we present.
To compare the models, we computed the changes in the NLLs based on the predictions of the models for the testing data (114 mice participating for 3/10/20 days). We performed the t-test on the changes in NLLs for individual mice followed by the false discovery rate (FDR) correction. Our comparisons were iterative: every time a model under consideration outperformed the previous best model, it became the new best model, starting a new round of comparisons. This procedure lasted until a model under consideration could not be outperformed by any other model. We used the q-value of  $q = 0.05$  as a cutoff. Below, we report the results for the final round of the comparisons. We report additional details in Appendix B.6; the results of simulated experiments are described in Appendix C.2.
4.1.1 Baseline models
Here we use the conventional approach and compare our model with the standard baselines (Devaine et al., 2014; Khalvati et al., 2019). The comparison results are displayed in Figure 4. We provide detailed numerical results in Supplementary Table 7 in Appendix C.3.
First, we determined the depth of reasoning involved in social conflict. We found that the Bayesian model that, along with "primary" beliefs, uses "secondary" beliefs to anticipate opponents' actions (a first-order theory of mind, 1-ToM) is more consistent with the data than a model where decisions are only based on "primary" beliefs about the strengths of the contestants (0-ToM).
Second, we analyzed the availability of information during conflict behavior. We established that the model that endows each animal with individual beliefs explains the animals' actions in the experiment better than the model that uses "shared" beliefs available to both animals in the pair.
Third, we assessed the flexibility of policy during a single agonistic interaction. We show that the model where animals decide on their actions before a fight ("fixed policy") explains our data better than the model where animals adapt their actions to the opponent's actions during a fight thus converging to a different Nash equilibrium ("flexible policy"; Appendix C.1.3).
Finally, to probe the class of algorithms used by animals in social conflict, we compared the Bayesian belief-based model with the Rescorla-Wagner reinforcement learning model where each action ("attack", "defend") was associated with a value updated based on the rewards. Our behavioral data was insufficient to distinguish between the two algorithms; we present additional considerations based on neural data in Section 4.2.
Figure 4: Model comparison on (A) training data and (B) testing data. Here  $(^{*})$  indicates a significant difference between the models (FDR  $q \leq 0.05$ ), (ns) indicates a non-significant difference (FDR  $q > 0.05$ ), and the whiskers show the mean  $\pm$  the standard error of the mean.
4.1.2 Ablations
To infer the dynamics of beliefs that best describe the animals' actions, we compared the Bayesian belief-based model ("dynamic beliefs") with two models based on fixed beliefs ("static beliefs"; equivalent to the zero learning rate) (Baker et al., 2011). We found that the "dynamic beliefs" explained the animal data better than two types of "static beliefs": "prior beliefs" identical to those used to initialize the "dynamic beliefs" model and "posterior beliefs" identical to the output of the "dynamic beliefs" model reflecting the most complete knowledge about the animals.
To test the role of body weight in the initial beliefs about animals' strengths, we compared the model where the beliefs were initialized using the animals' body weights with the model where the body weights were shuffled and with the model where the correct body weights were used but their prior distribution was wider than the true distribution. Our comparison shows that using the true animals' body weights and their distribution to initialize beliefs has positively affected the predictions.
4.1.3 Optimal parameters
To obtain further insights into the aggressive behavior of mice, we evaluated the optimal parameters for the first-order Bayesian belief-based model. We observed a local minimum of the NLL at the parameter values  $\sigma_{1} = 3 \pm 1\mathrm{g}$ ,  $\sigma_{2} = 10 \pm 1\mathrm{g}$ ,  $\beta_{o} = 6 \pm 1$ ,  $\beta_{a} = 3 \pm 1$ ,  $\alpha = 3 \pm 1$ ,  $\mathcal{A} = 10 \pm 1$ , and  $\varepsilon = 0.8 \pm 0.2$ . The identified parameters  $\beta_{o}$  and  $\alpha$  correspond to the policy with  $\delta = 0$  where mice attack any opponents of equal or lower strength. The uncertainty  $\sigma_{1}$  in initial estimates of own strength has a relatively low value suggesting that mouse body weight is an informative proxy for its strength. The uncertainty  $\sigma_{2}$  in initial estimates of the opponent's strength is relatively high suggesting that the opponent's body weight carries no significant information about its estimated strength and that such strength is rather estimated based on their actions. We summarize these findings in a normative theory of social conflict in Appendix C.4.
4.2 Model correlates in the brain
To analyze the neural correlates of the model variables, based on the behavioral data, we estimated the beliefs for each mouse (Figure 5A-D). We then computed their correlations, voxel by voxel, with the c-Fos activity in the entire brain (Figure 5E). We evaluated the beliefs at the end of the experiments because the c-Fos imaging only allows collecting one brain activity snapshot per animal. To study brain activity at different stages of social conflict, we used mice with varied participation in the experiment, i.e. "winners"  $(W)$  and "losers"  $(L)$  on days 3/10/20 (subscripted in the group names in Figure 5) and also inverts (e.g. "winners" who become "losers",  $WL$ ) on the day 22. We describe the details of this comparison in Appendix B.7 and summarize them in Supplementary Algorithm 5.
Figure 5: Reconstructed beliefs for individual mice: (A) about oneself; (B) about last opponent; (C) about last opponent's belief about oneself; (D) about last opponent's belief about themselves. Group key:  $L_{3}$  stands for "losers" observed on day 3 etc.  $WL$  stands for "winners" who become "losers" observed on day 22 etc. (E) Belief regression to the c-Fos activity in whole-brain samples.
In the correlation analysis, we use the "primary" and "secondary" beliefs reconstructed with the 0-ToM and 1-ToM models. We found significant (cumulative FDR  $q \leq 0.1$ ) neural correlates for both types of these variables (Figure 6A-F). To analyze the representations of 0-ToM and 1-ToM beliefs, we examined the set of voxels whose activity was correlated with either 0-ToM or 1-ToM beliefs (Figure 6G), i.e. the union of the voxels correlated with the two models. We found that  $77\%$  of these voxels were correlated with the 1-ToM beliefs only, while  $8\%$  of the voxels were uniquely correlated with the 0-ToM beliefs. The remaining  $15\%$  of voxels were correlated with both 0-ToM and 1-ToM beliefs. Thus, brain activity appears to contain signatures of both 1-ToM and 0-ToM beliefs, suggesting that both models may be relevant to the animals' behavior. We describe the results of additional tests, further supporting this conclusion, at the end of this section.
We evaluated the brain regions hosting significant neural correlates of 1-ToM beliefs. We found that the "primary" beliefs about oneself and the opponents were correlated with clusters of neural activity in the median preoptic nucleus (MEPO), the periventricular hypothalamic nucleus (PV), and the parabrachial nucleus (PB). The "secondary" beliefs were correlated with neural activity in the brain regions including the medial septal nucleus (MS), the medial and lateral preoptic areas (MPO, LPO), the lateral septal nucleus (LS), the superior and inferior colliculi (SC, IC), the dentate gyrus (DG), the parabrachial nucleus (PB), the anterodorsal and median preoptic nuclei (ADP, MEPO), the periventricular hypothalamic nucleus and the dorsomedial nucleus of the hypothalamus (PV, DMH), the pallidum (PAL), the zona incerta (ZI), the tuberomammillary and tuberal nuclei (TM, TU), the pontine reticular nucleus (PRN), and the lateral hypothalamic area (LHA). Some of these regions are known for their involvement in conflict behaviors (Aleyasin et al., 2018; Diaz and Lin, 2020).
To control for alternative explanations of the observed neural activity, we repeated the analysis for model-unrelated variables. These variables included the body weights of the animals (used to initialize the beliefs), the outcomes of agonistic interactions, and the binary winner/loser variable. As a control, we also considered the shuffled beliefs of each type, such that we shuffled the animal identities within all groups of "winners" or "losers". Among these, we only found substantial correlates of the winner/loser variable whose presence in the brain is unsurprising. The correlates of this variable only partially overlapped with the correlates of the beliefs (Figure 6G). We further examined the neural correlates of variables reconstructed with the Rescorla-Wagner model. We found that such correlates are mostly explained by the winner/loser variable, whereas a larger portion of
Figure 6: Correlates of the reconstructed beliefs in the c-Fos activity in the brain. (A-B) 0-ToM; (C-F) 1-ToM. Red: positive correlation; blue: negative correlation. (G) Voxels correlated with the models' latent variables.
the 1-ToM correlates does not have an alternative explanation (Figure 6G). This result supports the relevance of the proposed ToM model for conflict-related decision-making.
5 Broader impact
In our model, we applied evidence accumulation approaches to the domain of game theory which studies optimal interactions between agents. This allowed us to build a theory of chronic social conflict that can be used in future works for building quantitative models of social interactions. We then used Inverse Rational Control (IRC) to model the beliefs of animals based on their behavior. Combining the IRC with evidence accumulation models limits its degrees of freedom, increases robustness, and offers an interpretation for its predictions. Finally, we used the whole-brain c-Fos data (a proxy of neuronal activity) in combination with the IRC for an unbiased search for neural correlates of reconstructed beliefs. Overall, we combine game theory, evidence accumulation models, inverse rational control, and whole-brain imaging to propose a framework for building normative models of social behaviors and grounding them in neural circuitry in the brain. All animal procedures in this work were approved by the Stony Brook University Institutional Animal Care and Use Committee in accordance with the National Institutes of Health regulations. A detailed discussion of our results including the strengths and limitations of our approach is provided in Appendix A.
We thank Pavel Osten and Kannan Umadevi Venkataraju for collecting and processing the imaging data; Nikhil Bhattasali and Kchristina Samoilova for helpful discussions. This work was supported by grants from the National Institutes of Health (R01 DA050374; R01 AG076937), RSF (19-15-00026 to NK), the Swartz Foundation for Computational Neuroscience, and the Mathers Foundation.
Alefantis, P., Lakshminarasimhan, K. J., Avilla, E., Pitkow, X., and Angelaki, D. (2021). Sensory evidence accumulation using optic flow in a naturalistic navigation task. bioRxiv.
Aleyasin, H., Flanigan, M. E., and Russo, S. J. (2018). Neurocircuitry of aggression and aggression seeking behavior: nose poking into brain circuitry controlling aggression. Current opinion in neurobiology, 49:184-191.
Allefeld, C. and Haynes, J.-D. (2014). Searchlight-based multi-voxel pattern analysis of fmri by cross-validated manova. Neuroimage, 89:345-357.
Andersson, M. and Iwasa, Y. (1996). Sexual selection. Trends in ecology & evolution, 11(2):53-58.
Archer, J. (1988). The behavioural biology of aggression, volume 1. Cup Archive.
A Discussion
A.1 Normative theory of social conflict
In this work, we formulated a theory of social conflict based on game theory and Bayesian inference of agents' beliefs. We refined our model using mouse behavioral data and further validated it using neuronal data.
Our results suggest that animals' behavior during conflict is consistent with maintaining and updating beliefs about their strength and the strengths of their opponents. This observation is similar to prior work on Bayesian Theories of Mind proposing that in a variety of tasks humans and animals make decisions based on estimates of environmental variables updated with Bayes' rule (Baker et al., 2011). This finding also supports prior work showing the lack of naive (without evolving beliefs) game-theory-optimal behaviors in humans (Stahl and Wilson, 1995).
Our data suggest that, along with "primary" beliefs about their own and opponent's strengths, animals maintain "secondary" beliefs estimating the "primary" beliefs of opponents. This result is consistent with previous work in humans reporting that choices in the volunteer's dilemma are better explained with the 1-ToM model compared to 0-ToM and 2-ToM models (Khalvati et al., 2019). Our results are also consistent with behavioral observations in mice suggesting that animals facing an aggressive/defeated opponent anticipate their actions (Kudryavtseva et al., 2014).
We show that initial estimates of animals' own strength were correlated with their body weight. This finding is consistent with the proposals that body weight is a strong predictor for fighting performance in mice (Andersson and Iwasa, 1996; Cooper et al., 2020) and social status in rats (Nagy et al., 2023). At the same time, our data suggest that animals may not use correlates of opponents' body weights as surrogates for strength, rather relying on opponents' actions.
Our fits to behavioral data suggest that the decisions about animals' actions in the chronic conflict task were made before each agonistic interaction with other mice and were not optimized within a single interaction. These fits are consistent with our observations of animal behavior in individual interactions. This result also aligns with prior work in humans reporting bounded rationality in a series of games (Stahl and Wilson, 1995). Further experiments may be designed to validate this result.
Using behavioral data, we have arrived at a payoff matrix describing the animals' decisions in social conflict. We found that the cost of defense (via escaping confrontation) is small compared to the cost of defeat (via attack), in agreement with minimizing the physical damage reported by Crowcroft (1966). At the same time, the cost of defeat after an attack exceeds the reward for victory. This may be due to the immediate effect of the cost of defeat (e.g. physical damage) compared to a discounted delayed reward (e.g. mating, access to food, etc.) associated with winning in a conflict (Kable and Glimcher, 2007; Kobayashi and Schultz, 2008).
Finally, we identified the neural correlates of the beliefs computed in our model. Previous studies have characterized multiple brain regions whose activity modulates aggression. Our approach offers an interpretation of the computations performed in several brain regions. We find correlates of beliefs in hypothalamic regions, consistent with the previously reported role of the hypothalamus in aggression (Aleyasin et al., 2018; Diaz and Lin, 2020). Within the hypothalamus, a recent study describes the short-term aggression ramping encoded in the ventromedial hypothalamus (VMHvl) but not in the medial preoptic area (MPO) (Nair et al., 2023). Here, we observe the correlates of animals' beliefs in preoptic areas (MPO, LPO, MEPO) but not in VMHvl in the conditions of chronic social conflict. This finding may indicate differences in the coding schemes for short-term and chronic conflict. We find that the neural activity in MPO is correlated with the "secondary" beliefs about opponents' strengths, in agreement with a recent study on the role of cMPO neurons (Wei et al., 2023). It is likely that other brain regions, such as VMHvl encode variables different from beliefs. This possibility may explain why we did not find significant correlates of beliefs in other brain regions related to aggression including some limbic areas, the brain stem, and the reward system (Aleyasin et al., 2018).
A.2 Limitations and strengths of our approach
In our setting, animals made their decisions in the conditions of incomplete information and refined their policies based on the observed outcomes of their encounters. Agents in such a setting face the problem of a partially observable Markov decision process (POMDP). Here, we use biological
decision-making to propose an approach to solving the POMDP problem in multi-agent settings. Our work extends machine-learning methods for analyzing behavioral and neural data in these conditions. Combining inverse rational control (Dvijotham and Todorov, 2010; Kwon et al., 2020) with game theory (Cressman et al., 2003) offers an interpretation of model parameters. Specifically, the variables in our models either represent the rewards for various outcomes or describe the confidence in taking various actions. A low number of fitting parameters makes our approach data-efficient. This property may be useful in neuroscience where data are typically small (Stahl and Wilson, 1995). Game theory makes the transitions between beliefs deterministic, reducing the computations compared to probabilistic Baum-Welch algorithms (Baum et al., 1970), conventionally used in POMDPs (Wu et al., 2018). The benefits of our approach can be quantified in follow-up studies.
A limitation of our game-theory-based approach is that it involves model comparisons between ad hoc models (Stahl and Wilson, 1995). To mitigate this limitation, we based our models on the results of previous studies, using standard ToM/ML baselines (Stahl and Wilson, 1995; Khalvati et al., 2019). In future studies, we may consider additional models including mixtures of ToMs (Khalvati et al., 2019), naive Nash, and rational expectation strategies (Stahl and Wilson, 1995).
Our behavior-based model reconstructs relevant variables for individual animals, allowing us to search for their correlates in neural activity. As a proxy of neural activity, we used the c-Fos expression, captured with whole-brain 3D microscopy (Renier et al., 2016). Using c-Fos comes with several limitations: i) c-Fos lacks temporal resolution, allowing one snapshot of brain activity per mouse; ii) the c-Fos signal is nonspecific, potentially encoding factors other than neuronal activities (Herrera and Robertson, 1996). Among the benefits, c-Fos does not perturb animals' social behavior, making it the standard choice for studies of aggression (Aleyasin et al., 2018; Diaz and Lin, 2020; Wei et al., 2021). Ex-vivo 3D imaging allowed us to reconstruct the whole-brain neural activity. While previous studies focused on tissue sections of preselected brain regions, our whole-brain data has enabled an unbiased study of conflict circuitry.
The neural activation analysis in this work is performed with the correlation analysis followed by the FDR correction (Benjamini and Hochberg, 1995). This is a conservative analysis establishing whether the model variables are directly represented in neural activity. Alternative approaches could include GLMs for detecting linear combinations of model parameters (Lindquist, 2008), Gaussian random fields for modeling spatial correlations of the signal (Rue and Held, 2005), and MANOVA for distinguishing representations of correlated model variables (Allefeld and Haynes, 2014). Using these and other fMRI approaches (Skup, 2010; Ashburner et al., 2014) may allow identifying the correlates of individual model variables.