A normative theory of social conflict
Authors: Sergey A. Shuvaev
Evgeny M Amelchenko
Dmitry Smagin
Natalia Kudryavtseva
Grigori Enikolopov
Alexei A. Koulakov
Venue: NeurIPS 2023 poster (2023)
Keywords: neuroscience
decision-making
normative modeling
game theory
Bayesian methods
POMDP
inverse rational control
belief
theory of mind
TL;DR: We combine our behavioral and neuronal data to formulate a theory of chronic social conflict based on game theory and Bayesian inference of agents’ beliefs.

Abstract: Social conflict is a survival mechanism yielding both normal and pathological behaviors. To understand its underlying principles, we collected behavioral and whole-brain neural data from mice advancing through stages of social conflict. We modeled the animals’ interactions as a normal-form game using Bayesian inference to account for the partial observability of animals’ strengths. We find that our behavioral and neural data are consistent with the first-level Theory of Mind (1-ToM) model where mice form “primary” beliefs about the strengths of all mice involved and “secondary” beliefs that estimate the beliefs of their opponents. Our model identifies the brain regions that carry the information about these beliefs and offers a framework for studies of social behaviors in partially observable settings.

A normative theory of social conflict
Sergey Shuvaev', Evgeny Amelchenko", Dmitry Smagin
Natalia Kudryavtseva", Grigori Enikolopov?, and Alexei Koulakov' Cold Spring Harbor Laboratory, Cold Spring Harbor, NY, USA: "Department of Anesthesiology and Center for Developmental Genetics, Stony Brook University, Stony Brook, NY, USA; Institute of Cytology and Genetics, Siberian Branch of Russian Academy of Sciences, Novosibirsk, Russia. koulakov8cshl.
Abstract
Social conflict a survival mechanism yielding both normal and pathological behaviors. To understand its underlying principles, we collected behavioral and whole-brain neural data from mice advancing through stages of social conflict. We modeled the animals' interactions inference to account for the partial observability of animals' strengths. We find that behavioral and neural data consistent with the first-level Theory of Mind (1-ToM) model where mice form "primary" beliefs about the strengths all mice involved and "secondary" belief's that estimate the beliefs of their opponents. Our model identifies the brain regions that carry the information about these belief's and offers a framework for studies of social behaviors in partially observable settings.
normal-form game using Bayesian
Introduction
Research into conflict behavior is an established domain within cognitive science and neuroscience with focus on exploring the origins of pathological aggression and defeat (Archer 1988 Rosell and Siever 2015). The broad interest in this topic is driven by the pervasive occurrence of pathologically aggressive behaviors such as school bullying and prison hostility that remain significant societal concern. Gaining insight into the neural and behavioral foundations of conflict behavior is a crucial step toward mitigating the adverse consequences of aggression and preventing instances of hostility.
Toward this goal, the field of conflict studies has adopted an experimental approach. Researchers have monitored rodent behavior and recorded corresponding brain activity, as rodents naturally exhibit aggressive behaviors allowing experimenters to minimize additional stress for animals during data collection. This approach has yielded valuable insights into the factors that promote aggression (Archer 1988, Wang and Anderson] Lorenz Lorenz (005) and led to identifying brain regions affected by conflict (Aleyasin et al. [2018] Diaz and Lin 2020 Wei et al., 202Tp. The studies in this field, however, rarely focused on the dynamic nature of aggressiveness in individuals, thus constraining our ability to understand and predict harmful behaviors. Here, we use behavioral data modeling to explore the strategies of aggressive behavior in individual mice over prolonged periods of time.
Our study contributes to the field follows. We collected large-scale data behavior and whole- brain neural activity that follows the emergence of aggression and defeat in individual mice. We also introduced family of models to characterize the learning rules, decision- and cost functions underlying aggression. These models were applied to the observed behaviors of individual mice to identify the best-fitting model. Using that model, we described the decision-mail making mechanism consistent with the emergence of aggression and submissiveness. We then conducted an analysis of the neural correlates of both explicit and hidden model variables across the entire brain, thus validating the hidden dynamics of the model and linking specific brain regions to individual model variables. While similar approaches were used theoretical and computational neuroscience, they have not been previously applied in studies of conflict.
processes.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Our findings show that behavioral and neural data describing chronic conflict aligns with the first- level Theory of Mind (1-ToM) model. According to this model. mice form "primary" beliefs about their own and their opponents" strengths along with "secondary" beliefs estimating their opponents" corresponding beliefs. These belief systems are updated with Bayes' rule based on the actions of opponents and the outcomes of aggressive encounters. We propose that, by representing Bayesian beliefs, animals in social conflict situations address the challenge partial observability. Our model thus holds the potential for generalization to other social interactions characterized by uncertainty.
The proposed model not only provides insights into the development of pathological aggression and defeat but also offers a path toward their mitigation. The next steps will involve applying the model to newly available data and identifying the neural correlates of aggression-relates variables within the brain. This will be followed by the exploration of connectivity changes between the identified brain regions, which may unveil the neural circuit governing decision-makinn in aggression. Ultimately, the shift from a normative to a circuit model of conflict may enable the design of policies geared toward reducing aggression in a variety of social settings.
2 Related work
Behavioral biology of aggression. Social conflict has been studied in humans, non-human primates, and mice- a model organism whose behavioral states can be manipulated. Mouse studies have investigated how social conflict leads to the formation, maintenance. and plasticity of aggressive and subordinate behavioral states in animals (Wong et al. 2016; Hashikawa et al., 2017). 7) Such behavioral states can be modeled in the chronic social conflict paradigm where mice are allowed limited interactions over extended periods of time (Kudryavtseval 2000, Miczek et al.) 200T). Here,
we use the chronic social conflict paradigm to induce various behavioral states in mice and monitor their related behaviors. As the previous studies of social conflict have shown the evolutionary preservation of its basic mechanisms (Wang and Anderson] 2010, Watanabe et al. 2017). we expect our results to be relevant to human behavior.
Game theory. Optimal behaviors of interacting agents are conventionally described in terms of game theory. Game theory considers rational agents developing their strategies to maximize rewards. The rewards received by the agents depend on their actions and the actions of their opponents. The acquisition of optimal strategies in games can be described by probabilities of available actions (Smith, 1982, Cressman et al.) 2003). Such strategies of agents co-evolve to reinforce higher-reward actions until the rewards can't grow any further (Nash equilibrium). Game-theoretic approaches have been used in models of human and animal behaviors in multi-agent settings including agonistic interactions (Smith) 1974 Hofbauer et al., 1998| Wilson 2000, Lorenz] (000).. Here, we use game theory to model agonistic interactions in the condition of chronic social conflict in mice.
Beliefs and Theories of Mind. Humans and animals in social settings have limited access to environmental variables, which can be modeled with the partially observable Markov decision process (POMDP) framework. To gain evidence about hidden variables, real-world agents may maintain probabilistic internal models of environment - the "beliefs" based on which their actions can be considered rational, maximizing a reward function (Fahlman et al. 1983 Alefantis al.) 2021). The agents' rewards and beliefs can be inferred from their behavior using inverse control techniques that maximize the likelihood of the observed behavior based on a hidden dynamics model (Russell) 1998) Choi and Kim| 2 Dvijotham and Todorov 2010 Kwon et al. 2020). In biologically relevant multi-agent settings, beliefs are studied in the Theories of Mind (ToM) framework, proposing that humans and animals maintain beliefs about the beliefs of their adversaries (Baker et 2011 or
aides (Khalvat et al. 2019). As previous studics were successful in inferring beliefs (Schmitt et al., 2021 and regressing them to neural activity in simulations (Wu et al.) 2022) or (Koter-Hale and Saxe 2013), here we introduce a framework for the inference of belief's in social setting and regress them to high-resolution whole-brain neural activity in mice.
2017 Alefantis et al.
low-resolution IMRT
C-Fos as a whole-brain marker of neural activity. The search for the brain regions accumulating evidence about the environment requires large-scale neural activity data. Such data can be obtained by monitoring the levels of c-Fos, an immediate early gene whose activation reflects neuronal activity (Sagar et al. 1988 Herrera and Robertson 1996). C-Fos data lacks temporal resolution, yet it allows observing brole-brain activity at a high spatial resolution without using equipment that may affect animals' choices. Local expression of e- c-Fos has implicated several brain regions in agonistic

---

interactions (Hashikawa et al. 2017 Aleyasin et al. 2018 Diaz and Lin 2020 Wei et al. 2021 Here, use 3D sheet microscopy the c-Fos signal whole-brain samples (Renier al. 2016) to identify the brain-wide neural activity in animals with varying exposure to social conflict. We compare our c-Fos data to the beliefs identified based on behavior individual mice and report the regions that may be involved in the computation of conflict-related variables in the brain.
3 Methods: a normative POMDP framework for social behavior modeling
The goal of this work is build a quantitative theory for the formation of social conflict-related behavioral states in mice. In Section 3.1 we introduce mouse behavioral paradigm where we recorded the actions leading to different behavioral states and the brain activities corresponding to these states. In Sections t.2 3.4 we define normative POMDP- based framework for studying real-world social behaviors. this framework, the game-theory optimal actions agents rely on their beliefs, defined as distributional estimates of the hidden variables of the environment. In Section 4 we examine hypotheses about the reward schedule, information availability, and evidence accumulation related to social conflict. We compare results to behavioral data in Section 4. and to neural data in Section 4.2] We discuss our findings in Appendix A detailed description of the methods is provided in Appendix B Derivations and supplementary results provided in Appendix C
Explore (1 day)
Contact (10 min)
D Initial
Various winners vs. losers
Inver-
pairs
sion
Losers Neutral Winners
Mice
"attack" "defend"
61
Initial pairs
Various winners vs. losers
Inversion
(3 days)
(17 days)
(2 days)
88
22
Days
22
Figure 1: The chronic social conflict paradigm. (A-B) Stages of single sensory contact event. (C) Stages of multi-day experiment. (D) Recorded actions of mice in the experiment (white: no encounter scheduled).
3.1 Running example: the chronic social conffict paradigm
Our goal was come up with unified behavioral model explaining various stages of social conflict. As previous experiments have mostly focused on isolated aspects social conflict (e.g. its pathological form only), the data needed for building unified model was readily available. necessitating the conduct of new experiment. To this end, we implemented the chronic social conflict paradigm (Kudryavtseval 2000; Kudryavtseva et al. [2014), well-established experimental paradigm for studying aggression and conflict. Within the paradigm, we focused on conventionally studied stages of social conflict, the adaptive (3 days), intermediate (10 days), and pathological (20 days) aggression/defeal (Kudryavtseva et 2014). For completeness, we additionally considered the reversal of pathological behaviors (after 20 days) the standard experimental perturbation of behavior. We used the conventional experimental parameters (Kudryavtseva et al. 2014).
We conducted our experiments follows. Pairs of meight-matched (as proxy for being strength- matched) mice were placed in cages separated by a perforated partition (Figure MTA). Once daily, the partition was removed for 10 minutes to enable agonistic interactions between mice (Figure [ After 3 days, mice were classified into "winning". "losing", and "neutral" (non-fighting) (Figure [C.D). The participation of "neutral" mice the experiment was discontinued. A Afterward, each winning mouse remained in its cage, while each losing mouse was daily relocated to an unfamiliar cage with an unfamiliar winning mouse (Figure [TC). Regardless of longer being weight- matched, mice have retained their winning or losing behavioral states, thus transitioning to the pathological regime of After 20 days of interactions, mice were exposed to opponents of equal behavioral state (Figure[[p) The newly formed pairs underwent two more days of agonistic interactions throughout which new win/lose relationships were established (Figure TD). We provide more details in Appendix B.I
social conflict-related decision-making (Figure After
3

---

3.2 Game-theory optimal actions
We build normative POMDP model for social behaviors using the chronic social conflict paradigm. In this section, we start with an approximation where each agent has all information about itself and its opponents. In that case, we use game theory to define the optimal actions for each agent.
We formalize our behavioral paradigm as a normal form game where, on each iteration, two agents have decide simultaneously on what action to take. We defined the possible actions "attack" or "defend". Depending on the actions d1 and 02 selected by the two agents respectively, they received rewards defined as follows. If both agents chose to "defend", no fight happened, leading to zero reward = r2 O assigned to each agent. If both agents "attacked", the outcome of the game was defined by their strengths 8, an additional parameter assigned each agent in the model. The outcome probability puamn was defined by the softmax rule over the strengths parameterized with the "outcome confidence" Bg:
(I) Here and below Z denotes the normalization coefficient. Once the outcome was determined, the winning agent received a reward of r = +1, and the losing agent expended a cost r = -A. The reward expectation was equal to:
Pi p Z exp(3
K2 Pi + (-A) (1-p. = (1 + A)p, win A.
(2)
The cost of loss was reduced if one of the agents chose to "defend" while the other agent "attacked". In that case, the "attacking" agent always won and received the reward of = +1 while the losing agent expended the cost of r -a such that a < A. The reward expectations were described in the payoff matrix it, whose rows correspond to the actions of the agent ("attack" and "defend") and columns correspond the actions of its opponent:
R2
16g
To determine the optimal strategies in this game, we used game theory. In this approach, the goal of every participant was to maximize its expected reward E[r,]:
Blri] = PT R Pz.
Here the vectors P; define the probabilities to "attack" p; and to "defend" 1 - p; for an agent i:
P. = (1 Pt
1-
(5)
similar expression can be written for the expected reward of the second agent E[[r]] To maximize the rewards E[ra], computed their gradients with respect to the probabilities to "attack" pa (an agent could only update its own policy, but not that of the opponent). We used these gradients to update the policies leading to joint maximization of the expected rewards (Figure(A).
08[r11
(K1 + a - 1)p2 + 1;
dpl
88 (Ng + a 1)p1 +
Ups
We provide the derivation for the equation above in Appendix C.I.I The probabilities of actions can
be only defined in the range 0 < {PI. < 1. Their evolution P: x /EEr://öpp, governed by the equations above (red arrows Figure 2), converges to zeros or ones (blue arrows in Figure ( forming pure strategies (we show the lack of mixed strategies in Appendix C.1.2). To determine optimal pure strategies, we chose the strategies whose reward gradients (red arrows in Figure 2 pointed outwards the [0- 1] interval for both agents. We represented the optimal policies via the tensor of probabilities for each possible action a depending on the agent's strength 81 and their opponent's 's strength 82, averaging over all optimal pure strategies (Figure 2C):
Aai Pr(a; = a|s1 = i,82 =))
(7)
The resulting optimal policies in the model depended on the relative strengths of the agents (Figure
We parameterized the optimal policies with the maximum relative strength of the agents 8 82 $1
at which it was still optimal to "attack". We then explored how depends on the task parameters and H, and found that most of these parameters' values correspond to the optimal strategy with 5 = ("xx" in Figure 2D) i.e. to "attacking" any opponent who is weaker or equal. We describe additional details regarding the acquisition of optimal policies Appendix B.2 and summarize the procedure in Supplementary Algorithm

---

A
Policy gradient
B Pure Nash equilibrium
C
Optimal policy
D
Policy phase diagram
100
(0:1)
(1:1)
10.00
6
20
20
Outcome confidence
no
no
Príd; = =atttckk"
"attack
Own strength
15
"attack"
2.50
16
0.5
0.64
12
0.16
10:0)
yes
(1:0)
ddfeend
0.04
4
0.0
0.0
0.5 Pr(a;= "'attakk))"k"))
1.0
0.0
Pria, = atttcck")
1,0
Opponent's strength
10 15 20
0.01
0.0
0.2
0.4 0.6
1.0
0
"Defense" cost
Figure 2: Game-theoretic model of the chronic social conflict paradigm. (A) Policy gradients. (B) Gradient
orientations yield a pure strategy as a Nash equilibrium. (C) Optimal policy and dependence on the relative strength. (D) Dependence optimal policy on the model parameters.
3.3 Partial observability: belief's about hidden variables
Game theory our setting predicts a static policy, unchanging over time. Conversely, weight-matched mice in the experiment who initially attacked each other later split into always-attacking "winners" and ever-defending "losers" (Figure (D). To account for this dynamic, we expand our model to a scenario where agents do not possess the full information about their strengths but accumulate this information over the task. In this section, we model the agents' information about strengths via beliefs the probability distributions for agent to belong to certain strength category.
We used body weights w of the animals proxy of their strengths (Andersson and Iwasa 1996 Cooper et al. 2020):
Wi X 8i-
We modeled the animals' initial estimates of their body weights w as normally distributed around true values w. Under this assumption, we reconstructed the probability distributions for animals' strengths (their initial beliefs) B' = Pr(s i|ı0) using their body weights i in Bayes' rule:
B Pr(s (i) PN((,,) (w) Pr(s-P(())
Pr(w)
where PN((..)) (w) normal distribution probability density function representing the noise in the estimate of the animal's body weight; Pr(w) is the distribution of the animals' estimated body weights {w and Pr(s) is the distribution of their strengths {& To denoise the strength distribution Pr(s). we approximated the experimentally observed body weight distribution Pr(w) xc Pr(s) with a normal distribution.
We consider up four types of beliefs. The two "primary" beliefs describe the animals' estimates of their own strength and of the strength of the opponent. The two "secondary" beliefs estimate expected to better estimate their
the "primary" beliefs of the opponent. The animals strength compared to that of their opponents. To this end, we used separate uncertainty parameters for estimating one's own strength = a1) and that of an opponent (a σ2). For the "secondary" belief', the uncertainties were combined initialization of beliefs in Appendix B.3 and summarize them in Supplementary Algorithm
V oz). We provide the details regarding the
3.4 Evidence accumulation: Bayesian update of beliefs
In this section, we expand our model with an update mechanism for the agents' beliefs. We update the beliefs B using Bayes' rule and the information about the actions a and outcomes o of agonistic interactions (Figure 3A).
We define the outcome tensor Opab tensor describing the probabilities of each possible outcome for an agent of strength = i after choosing an action a provided that the opponent has strength 82 = j and chose an action 02 = b:
(10) The reward tensor Rob describes the expected reward E(r1) for an agent of a strength 8 i after choosing action provided that the opponent has strength $2 and chose an action
O
Pr(oi = 0lai = a, = b, 81 i, = j).
5

---

02 b. The reward expectation was based on probabilities of possible outcomes:
R Σ r(o, a, b)
(11)
The outcome-action and reward-action tensors predicted the probability of an outcome or and expectation of reward assuming that the opponent's 's actions d2 are game-theory-opimal!
[RA]iSh
Re Ca /o
(12)
[OA]iih = Oig conv (A, NA.
(13)
Here the opponent's action is estimated based the "secondary" belief's and the estimated outcome based on the "primary" beliefs. The convolution reflects that the opponent's s action is based on the distributional belief rather than point estimate of the strengths. The standard deviation
v Vot + oB applies the estimated opponent's estimates of both its own strength and "our" strength.
B
Bellefs loser
Beliefs B winner
B2
B,,
8,
beliefs
8
B
By
time
Time (days)
prob
1=1,2
latent observed
10
actions
20
outcomes
O₁
5
20 5
10 15
20
Self-estimated strength Self-estimated strength
Figure 3: Bayesian update beliefs the model. (A) Belief update diagram two mice. (B,C) Reconstructed helief dynamics for representative loser and winner mice.
To decide action d1 agent maximized its reward r1- We computed this reward by summing the reward-action tensor [RAA] iikz multiplied by the belief tensors Bi B Bh BLg reflecting the probability distributions for strengths:
Q1 softmax
CRA SIRE Bi B12 B BB2
(14)
Ejke
Here the belief tensors Bu and B12 describe the "primary" beliefs of an agent about its own strength and the strength of its opponent respectively. Likewise, the belief tensors B21 and B22 describe the "secondary" beliefs, i.e. the agent's estimate its opponent's "primary" beliefs. The indices j. k iterate over all possible strengths, e.g. B 0.2 would mean that, according the agent's belief, the probability of its own strength to be equal to constitutes 20%. To update the agents' beliefs using the observed actions and outcomes {01, a2), used Bayes' rule individually for every element of each belief tensor B.::
Pr(si = (()))1))))) = Pr(fon, a2H|s Pr) =)
(15)
Pr({oi,u2)
The probability of an outcome 0₁ and an action 02 for the agent of the strength 81 = was derived from the outcome-action tensor | 10laiaz
Pr(for, agHls = E
(16)
jkl
The probability for the agent to be of strength = i was taken from the belief tensor Bi Pr(s1
Bi
(17)
The marginal probability of the observation {01, 02) was defined by the nntcome-action tensor [OA] scaled by the belief's about strength Big Big etc. to scale the optimal actions with the probabilities of their underlying strengths:
Pr(tor, a2)) Pr(oilas)PP(o)))
IIAA Bi Bi B
(18)
ijkl

---

Together, the four equations above formed the update rule for agent beliefs based on their observations:
Pr(s1 = ilfor, Ejk10 020ia.02 B Aliht
B/2
(19)
AA "B Bi B
The update could proceed at an arbitrary learning rate e:
Bi + Pr(#1 = (()0,,))) (2 + - Bi
(20)
We describe additional details regarding the update procedure for the belief's in Appendix B.4 and summarize them in Supplementary Algorithm 3
4 Results: inference of animal choice mechanisms in social conflict
4.1 Model fit and comparison
We used the framework defined above to test hypotheses about mouse social conflict-related choices. To test hypotheses against their alternatives, optimized pairs of models the training data (88 mice participating for 22 days). To this end, we specified a negative log- log-likeldhood (NLL) function C comparing the action probabilities predicted in our model the mouse actions logged in the experiment:
L Elog Pr(a()) + log Pr(o))))
(21)
To fit the model parameters, we minimized the NLL regularized with the la norm of the model arguments. We chose the regularization coefficient such that it resulted in the best fits simulated experiment. For the real mice in the experiment, we used the data spanning all 22 days to propagate the beliefs but the NLL was only computed for the data from days 1-3 and 21-22 to avoid the impact of the repeated actions on days 4-20. We report the implementation details of the model fit in Appendix B.5 and summarize them in Supplementary Algorithm 4
We then proceeded to compare how well different models explain the behaviors observed in our experiment. Here we follow the standard way compare models in behavioral research that is using formal comparison criteria, such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC). Both criteria are computed the model less the log-liken likelioel of this model to generate the original data. Most of the models that we considered have the same number of parameters, therefore, as we only evaluated the difference of AICs (or BICs) between the models, the term reflecting the number of parameters cancelled out, resulting in the difference log- lielliiiihoodss that we present.
function of the number of parameters in
To compare the models, we computed the changes in the NLLs based on the predictions of the models for the testing data (114 mice participating for 3/10/20 days). We performed the t-test the changes in NLLs for individual mice followed by the false discovery rate (FDR) correction. Our comparisons were iterative: every time model under consideration outperformed the previous best model, became the new best model, starting new round of comparisons. This procedure lasted until a model under consideration could not outperformed by any other model. We used the q-value of q = 0.05 as cutoff. Below, we report the results for the final round of the comparisons. We report additional details in Appendix [B.6] the results of simulated experiments are described in Appendix C.2
4.1.1 Baseline models
Here we use the conventional approach and compare our model with the standard baselines (Devaine al. 2014: Khalvati et al. 2019). The comparison results are displayed in Figure We provide
detailed numerical results in Supplementary Table 7 in Appendix C.3
First, we determined the depth of reasoning involved in social conflict. We found that the Bayesian model that, along with "primary" beliefs, uses "secondary" beliefs to anticipate opponents' actions (a first-order theory of mind, 1-ToM) is more consistent with the data than a model where decisions only based on "primary" beliefs about the strengths of the contestants (O-ToM).
Second, we analyzed the availability of information during conflict behavior. We established that the model that endows each animal with individual beliefs explains the animals' actions in the experiment better than the model that uses "shared" belief' available to both animals in the pair.
7

---

Third, we assessed the flexibility of policy during a single agonistic interaction. We show that the model where animals decide on their actions before fight ("fixed policy") explains our data better than the model where animals adapt their actions to the opponent's actions during a fight thus converging to different Nash equilibrium ("flexible policy"; Appendix C.I.3).
Finally, to probe the class of algorithms used by animals in social conflict, we compared the Bayesian belief-based model with the Rescorla- wagnerr reinforcement learning model where each action ("attack", "defend") was associated with value updated based on the rewards. Our behavioral data was insufficient distinguish between the two algorithms; we present additional considerations based on neural data in Section
Training data
Testing data
0- M Shared
of Manry
of M
of Manar
Individual belleffs
Shared
Individual beliets
Flexible in-fight policy
Fixed
Flexible in-fight policy
Fixed in-fight policy
in-fight policy
53
Rescorla- Wagner
Rescoria- Wagner
ns
Bayesian bellef-based
belter-based
Fixed prior beliefs
Fixed prior beliefs
Bavesian belierf
Bavesian update of
of
Fixed pos- terior beliefs
Fixed pos- terior
Shuffled body weights
Shuffled body weights
True body weights for initill beliefs
True body weichts for initial beliefs
Wide body weight prior
null model
model
Wide body weight
null modek
model
-2 log likelihodd
2
0
A log likelihood
Figure 4: Model comparison on (A) training data and (B) testing data. Here (*) indicates a significant difference between the models (FDR S 0.05), (ns) indicates a non-significant difference (FDR > 0.03), and the whiskers show the mcan the standard error of the mean.
4.1.2 Ablations
To infer the dynamics beliefs that best describe the animals' actions, we compared the Bayesian belief-based model ("dynamic beliefs") with two models based on fixed beliefs ("static beliefs"; equivalent to the zero learning rate) (Baker et al. 201T).
found that the "dynamic beliefs" explained the animal data better than two types of "static beliefs": "prior beliefs" identical those used to initialize the "dynamic beliefs" model and "posterior beliefs" identical to the output of the "dynamic beliefs" model reflecting the most complete knowledge about the animals.
To test the role of body weight in the initial beliefs about animals' strengths, we compared the model where the beliefs were initialized using the animals' body weights with the model where the body weights were shuffled and with the model where the correct body weights were used but their prior distribution was wider than the true distribution. Our comparison shows that using the true animals' body weights and their distribution initialize beliefs has positively affected the predictions.
4.1.3 Optimal parameters
To obtain further insights into the aggressive behavior of mice, we evaluated the optimal parameters for the first-order Bayesian belief-based model. We observed a local minimum of the NLL at the parameter values σ + 1g. σ₂ 10 + 1g, 8, 6 + 1, 3_ 3 + 1, a and = 0.8 + 0.2. The identified parameters 3, and a correspond to the policy with 5 = 0 where mice attack any opponents of equal or lower strength. The uncertainty d1 in initial estimates of own strength has a relatively low value suggesting that mouse body weight is an informative proxy for its strength. The uncertainty 02 in initial estimates of the opponent's strength relatively high suggesting that the opponent's s body weight carries no significant information about its estimated strength and that such strength rather estimated based on their actions. We summarize these findings in normative theory of social conffict in Appendix C.4
1, A 10 +
8

---

4.2 Model correlates in the brain
To analyze the neural correlates of the model variables, based on the behavioral data, we estimated the beliefs for each mouse (A-g We then computed their correlations, voxel by voxel, with the c-Fos activity the entire brain (Figure F)) We evaluated the beliefs at the end of the experiments because the c-Fos imaging only allows collecting one brain activity snapshot per animal. To study brain activity at different stages social conflict, we used mice with varied participation in the experiment, i.c. "winners" (W) and "losers" (L) days 3/10/20 (subscripted in the group names in Figure (5) and also inverts (e.g. "winners" who become "losers", W L) on the day 22. We describe the details of this comparison in Appendix B.7 and summarize them in Supplementary Algorithm
5A-D).
A
Beliefs By
B
Beliefs B
Beliefs B21
Beliefs B22
E c-Fos
L5
Behavioral condition
W
prob
Lip
0.4
54 mice
W
Lop
W
0.2
WL
0.1
WW
LL
0.0
10 Strength
20
10 Strength
20
nn Voxels
Strength
Strength
Figure 5: Reconstructed beliefs for individual mice: (A) about oneself; (B) about last opponent; (C) about last opponent's belief about oneself; (D) about last opponent's 's belief about themselves. Group key: Lg stands for "losers" " observed on day 3 etc. WL stands for "winners" who become "losers" observed on day 22 etc. (E) Belief regression to the c-Fos activity in whole-brain samples.
In the correlation analysis, we use the "primary" and "secondary" beliefs reconstructed with the O-ToM and 1-ToM models. We found significant (cumulative FDR S 0.1) neural correlates for both types of these variables (Figure 6A-F). To analyze the representations of 0-ToM and 1-ToM beliefs, we examined the set of voxels whose activity was correlated with either O-ToM or 1-ToM beliefs Trigure@G). i.c. the union of the voxels correlated with the two models. We found that 77% of these voxels were correlated with the 1-ToM beliefs only, while 8% of the voxels were uniquely correlated with the 0-ToM beliefs. The remaining 15% of voxels were correlated with both T- and 1-ToM beliefs. Thus, brain activity appears to contain signatures of both 1-ToM and 0-ToM beliefs, suggesting that both models may be relevant the animals' behavior. We describe the results of additional tests, further supporting this conclusion, at the end of this section.
We evaluated the brain regions hosting significant neural correlates of 1-ToM beliefs. We found that the "primary" belief's about oneself and the opponents were correlated with clusters of neural activity in the median preoptic nucleus (MEPO), the periventricular hypothalamic nucleus (PV), and the parabrachial nucleus (PB). The "secondary" beliefs were correlated with neural activity in the brain regions including the medial septal nucleus (MS), the medial and lateral preoptic areas (MPO, LPO), the lateral septal nucleus (LS), the superior and inferior colliculi (SC, IC), the dentate gyrus (DG), the parabrachial nucleus (PB), the anterodorsal and median preoptic nuclei (ADP, MEPO), the periventricular hypothalamic nucleus and the dorsomedial nucleus of the hypothalamus (PV, DMH), the pallidum (PAL), the zona incerta (ZI), the tuberomammillary and tuberal nuclei (TM, TU), pontine reticular nucleus (PRN), and the lateral hypothalamic area (IHH).. Some of these regions are known for their involvement in conflict behaviors (Aleyasin et al. 2018 Diaz and Lin 2020).
To control for alternative explanations of the observed neural activity, we repeated the analysis for model-unrelated variables. These variables included the body weights of the animals (used to initialize the beliefs), the outcomes of agonistic interactions, and the binary winner/loser variable. As control, we also considered the shuffled beliefs of each type, such that we shuffled the animal identities within all groups of "winners"" or "losers". Among these, we only found substantial correlates the winner/loser variable whose presence in the brain unsurprising. The correlates of this variable only partially overlapped with the correlates of the beliefs (Figure We further examined the neural correlates of variables reconstructed with the Rescorla- Wagner model. We found that such correlates are mostly explained by the winner/loser variable, whereas larger portion of

---

Correlates of
Correlates of B1
E
Correlates of Bzn
Correlates of
model variables
PB
Pypo MEPO
PB
PAL PVpo
PB PRN
MEPO
ZI ZI-MRN
MRN
MPO
SN
1-ToM
O-ToM
Correlates of
D
Correlates of B12
F
Correlates of B22
RW
IC
SC IC
isWinner
MS
PAL MS ADP
PAL
PVp
MEPO
MPO DMH JU LHA TM
LHA
TM
Figure 6: Correlates of the reconstructed beliefss in the c-Fos activity in the brain. (A-B) O-ToM; (C-F) 1-ToM. Red: positive correlation; blue: negative correlation. (G) Voxels correlated with the models' latent variables.
the 1-ToM correlates does not have alternative explanation (Figure 6G). This result supports the relevance of the proposed ToM model for conflict-related decision-making.
5 Broader impact
In our model, we applied evidence accumulation approaches to the domain of game theory which studies optimal interactions between agents. This allowed us to build a theory chronic social conflict that can be used in future works for building quantitative models of social interactions. We then used Inverse Rational Control (IRC) to model the beliefs of animals based on their behavior. Combining the IRC with evidence accumulation models limits its degrees of freedom, increases robustness, and offers an interpretation for its predictions. Finally, we used the whole-brain c-Fos data (a proxy of neuronal activity) in combination with the IRC for unbiased search for neural correlates of reconstructed beliefs. Overall, we combine game theory, evidence accumulation models, inverse rational control, and whole-bran imaging to propose framework for building normative models of social behaviors and grounding them in neural circuitry in the brain. All animal procedures this work approved by the Stony Brook University Institutional Animal Care and Use Committee in accordance with the National Institutes of Health regulations. A detailed discussion of our results including the strengths and limitations of approach is provided in Appendix A
Acknowledgements
We thank Pavel Osten and Kannan Umadevi Venkataraju for collecting and processing the imaging data; Nikhil Bhattasali and Khristina Samoilova for helpful discussions. This work was supported by grants from the National Institutes of Health (R01 DA050374: R01 AG076937), RSF (19-15-00026 to NK), the Swartz Foundation for Computational Neuroscience, and the Mathers Foundation.
References
Alefantis, P., Lakshminarasimman. K. J., Avilla, E., Pitkow, X., and Angelaki, D. (2021). Sensory evidence accumulation using optic flow in naturalistic navigation task. hioRxiv.
Aleyasin, H. Flanigan, M. E.. and Russo, J. (2018). Neurocireuitry of aggression and aggression seeking behavior: nose poking into brain circuitry controlling aggression. Current opinion neurobiology, 49:184- 191.
Allefeld, C. and Haynes, J.-D. (2014). Searchlight-based multi-voxel pattern analysis of fmri by cross cross- validated manova. Neuroimage, 89:345-357.
Andersson, M. and Iwasa, Y. (1996). Sexual selection. Trends in ecology & evolution, 11(2):53-4 58.
Archer, J. (1988). The behavioural biology of aggression,, volume Cup Archive.
10

---

Ashburner, J., Barnes, G.. Chen. C.-C., Daunizeau. J., Flandin, G., Friston, K., Kiebel, S., Kilner, J., Litvak, V., Moran, R., et al. (2014). Spm12 manual. Wellcome Trust Centre for Neuroimaging, London, UK. 2464(4).
Baarslag. T. Hendrikx, M. J., Hindriks, K. V., and Jonker, C. M. (2016). Learning about the opponent in automated bilateral negotiation: comprehensive survey of opponent modeling techniques. Autonomous Agents and Multi-Agent Systems, 30:849-896
Baker, C., Saxe, R., and Tenenbaum, (2011). Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of annual meeting of cognitive science society, volume 33.
Baum, L. E., Petrie, T., Soules, G., and Weiss, N. (1970). A maximization technique occurrigg in the statistical
analysis of probabilistic functions of markov chains. The annals of mathematical statistics, 41(1):164-1711
Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: practical and powerful approach multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300.
Choi, J. and Kim, K.-E. (2011). Inverse reinforcement learning in partially observable environments. Journal of Machine Learning Research, 12:691-730.
Cooper. A. N.. Cunningham. C. B., Morris. J. S., Ruff. J. S., Potts. W. K., and Carrier, D. R. (2020). Muscu- loskeletal mass and shape correlated with competitive ability in male house mice (mus musculus). Journal of Experimental Biology, 223())jeb213389.
Cressman, R., Ansell, C., and Binmore, K. (2003). Evolutionary dynamics and extensive form games, volume 5. MIT Press.
Crowcroft, P. (1966). Mice all over.
Devaine, M., Hollard, G., and Daunizeau, J. (2014). Theory of mind: did evolution fool us? PloS One, 9(2)=88761..
Diaz, V. and Lin, D. (2020). Neural circuits for coping with social defeat. Current opinion in neurbbioloyy, 60:99-107.
Dvijotham, K. and Todorov, E. (2010). Inverse optimal control with linearly-solvable mdps. In JCML.
Fahlman, S. F., Hinton, G. F., and Sejnowski, J. (1983). Massively parallel architectures for al: Netl, thistle, and boltzmann machines. In National Conference on Artificial Intelligence, AAAI.
Hashikawa, K., Hashikawa, Y., Tremblay, R., Zhang, J., Feng, E., Sabol, A., Piper, W. T., Lee, H., Rudy, B., and Lin, D. (2017). Esrl+ cells the ventromedial hypothalamus control female aggression. Nature neuroscience, 20(1 1011158019900
Herrera, D. G. and Robertson, H. A. (1996). Activation of c-fos in the brain. Progress in neurobiology. 50(2-3):83-107.
Hofbauer, J., Sigmund, K., et al. (1998). Evolutionary games and population dynamics. Cambridge university press.
Kable, J. W. and Glimcher, W. (2007). The neural correlates of subjective value during intertemporal choice. Nature neuroscience, 10(12):622) 1625-1633.
Khalvati, K., Mirbagheri, S., Park, S. A., Dreher, J.-C., and Rao, P. (2019). A bayesian theory of conformity in collective decision making. Advances in Neural Information Processing Systems, 32.
Kohayashi, S. and Schultz, (2008). Influence reward delays on responses of dopamine neurons. Journal of neuroscience, (8(31):78777785
Koster-Hale, Saxe, R. (2013). Functional neuroimaging theory of mind.
Kudryavtseva, N. (2000). Agonistic behavior: model, experimental studies, and perspectives. Neuroscience and behavioral physiology. 30(3):293-305
Kudryavtseva, N. N.. Smagin, D. A., Kovalenko, L. L., and Vishnivetskaya, G. B. (2014). Repeated positive fighting experience male inbred mice. Nature protocols, 9(1 (1:2055 27177. 2717.
Kwon, M., Daptardar, S., Schrater, P., and Pitkow, X. (2020). Inverse rational control with partially observable continuous nonlinear dynamics, arXiv preprint arXiv:2009.1257
Lindguist, M. A. (2008). The statistical analysis of fmri data.

---

Lorenz, K. (2005). On aggression. Routledge.
Miczek, K. A., Maxson, S. C., Fish, E. W., and Faccidomo, S. (2001). Aggressive behavioral phenotypes in mice. Behaviourd brain research, 1 121(1-1):16--111.
Nagy, M., Davidson, J. D., Vásárhelyi, G., Abel, D., Kubinyi, E., El Hady, A., and Viesek, T. (2023). Social Structure and individuals' behavior shaped by group composition in long-term tracking of rats. bioRxiv, pages 2023-03.
Nair, A., Karigo, T,, Yang, B., Ganguli, S., Schnitzer, M. J., Linderman, S. W., Anderson, J. and Kennedy, A. (2023). An approximate line attractor in the hypothalamus encodes aggressive state. Cell, 186(1):188-193.
Renier, N., Adams, E, L., Kirst, C., Wu, Z., Azevedo, R., Ko,l, Autry, A. E., Kadiri, L.,, Venkataraju, K. U., Zhou, Y., et (2016). Mapping of brain activity by automated volume analysis of immediate early genes. Cell, 165(():1789-1188)
Rosell, D. R. and Siever, L. J. (2015). The neurobiology of aggression and violence. CNS spectrums, 20(3):254- 279.
Rue, H. and Held, I.. (2005). Gaussian Markov random fields: theory and applications. CRC press.
Russell, S. (1998). Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pages 101-103.
Sagar, S., Sharp, F. R., and Curran, T. ( (1988). Expression of c-fos protein in brain: metabolic mapping at the cellular level. Science, 240(4857)11328-1333
Schmitt, F., Bieg. H.-J., Herman, M., and Rothkopf, C. A. (2017). I see what you see: Inferring sensor and policy models of human real-world motor behavior. In Thirty-First AAAI Conference Artificial Intelligence.
Schroeder, H. and Desor, D. (2005). The behavioural differentiation between the carrier and the non-carrier profiles groups subjected to the diving-for-foot situation: complex social model to study anxicty rodents. In Proceedings of Measuring Behavior 2005, 5th International Conference on Methods and Technics in Behavioural Research (Wageningen, August 30-September 02, 2005), 390, volume 393.
Skup, M. (2010). Longitudinal fmri analysis: A review methods. Statistics and interface, 3(2):332.
Smith, M. (1974). The theory games and the evolution animal conflicts. Journal of theoretical biology. 47(1):209-221.
Smith, M. (1982). Evolution and the Theory of Games. Cambridge university press.
Stahl, D. O. and Wilson, P. W. (1995). On players' models of other players: Theory and experimental evidence. Games and Economic Behavior, 10(1):218 254.
Wang, L. and Anderson, J. (2010). Identification of aggression-promotin pheromone and its receptor neurons drosophila. Nature, 463(7278):27--31.
Watanabe, K., Chiu, H., Pfeiffer. B. D.. Wong, A. M., Hoopfer, E. D., Rubin, G. M., and Anderson, D. J. (2017). circuit node that integrates convergent input from neuromodulatory and social rehavior-promodtie neurons to control aggression in drosophila. Neuron, 95(5):1112-11
Wei, D., Osakada, T., Guo, Z., Yamaguchi, T., Varshneya, A., Yan. Jiang. Y., and Lin. D. (2023). hypothalamic pathway that suppresses aggression toward superior opponents. Nature Neuroscience, pages 1-14.
Wei, D., Talwar, V., and Lin, D. (2021), Neural circuits social behaviors: innate yet flcxible. Neuron.
Wilson, O. (2000). Sociobiology: new synthesis. Harvard University Press.
Wong. L. C., Wang, L., D' D'amour, J. A.. Yumita, T., Chen, G., Yamaguchi, T., Chang. C., Bernstein, H., You, X., Feng, J. E., al. (2016). Effective modulation male aggression through lateral septum to medial hypothalamus projection. Current biology, 26(5):593 604.
Wu. Z. Kwon, M., Daptardar, S. Schrater. and Pitkow, (2020). Rational thoughts in neural codes. Proceedings of the National Academy of Sciences, 117(47)29311-299
Wu, Z., Schrater, P. and Pitkow, X. (2018). Inverse pomdp: Inferring what you think from what you do. arXiv preprint 1805 09864.
12

---

A Discussion
A.1 Normative theory of social conflict
In this work, we formulated a theory of social conflict based on game theory and Bayesian inference of agents" beliefs. We refined our model using mouse behavioral data and further validated it using neuronal data.
Our results suggest that animals' behavior during conflict is consistent with maintaining and updating beliefs about their strength and the strengths of their opponents. This observation is similar to prior work on Bayesian Theories of Mind proposing that in a variety of tasks humans and animals make decisions based on estimates of environmental variables updated with Bayes' rule (Baker al. 2011). This finding also supports prior work showing the lack of naive (without evolving beliefs) same-theory-oplee
behaviors in humans (Stahl and Wilson| 1995).
Our data suggest that, along with "primary" beliefs about their own and opponent's strengths, animals maintain "secondary" beliefs estimating the "primary" beliefs of opponents. This result is consistent with previous work in humans reporting that choices in the volunteer's 's dilemma are better explained with the 1-ToM model compared O-ToM and 2- 2-ToM models (Khalvati et al.) 2019). Our results are also consistent with behavioral observations in mice suggesting that animals facing an aggressive/defeated opponent anticipate their actions (Kudryavtseva et al.) 2014).
We show that initial estimates of animals' own strength were correlated with their body weight. This finding is consistent with the proposals that body weight is strong predictor for fighting performance in mice (Andersson and Iwasa [996] Cooper et al.] (020) and social status in rats (Nagy et al.) (023). At the same time, our data suggest that animals may not use correlates of opponents" body weights as surrogates for strength, rather relying on opponents' actions.
Our fits to behavioral data suggest that the decisions about animals' actions in the chronic conflict task were made before each agonistic interaction with other mice and were not optimized within a single interaction. These fits consistent with our observations of animal behavior in individual interactions. This result also aligns with prior work in humans reporting bounded rationality in a series of games (Stahl and Wilson 1995). Further experiments may be designed to validate this result.
Using behavioral data, we have arrived at a payoff matrix describing the animals' decisions in social conflict. We found that the cost of defense (via escaping confrontation) is small compared to the cost of defeat (via attack), in agreement with minimizing the physical damage reported by Crowcroft (1966). At the same time, the cost of defeat after an attack exceeds the reward for victory. This may be due to the immediate effect of the cost of defeat (e.g. physical damage) compared to a discounted delayed reward (c.g. mating, access to food, etc.) associated with winning in a conflict (Kable and Glimcher 2007 Kobayashi and Schultz 2008).
Finally, we identified the neural correlates of the beliefs computed in our model. Previous studies have characterized multiple brain regions whose activity modulates aggression. Our approach offers an interpretation of the computations performed in several brain regions. We find correlates of beliefs in hypothalamic regions, consistent with the previously reported role of the hypothalamus in aggression (Aleyasin et al., 2018 Diaz and Lin 2020). Within the hypothalamus, recent study describes the short-term aggression ramping encoded in the ventromedial hypothalamus (VMHvl) but not the medial preoptic area (MPO) (Nair et al. 2023). Here, we observe the correlates of animals' beliefs in preoptic areas (MPO, LPO, MEPO) but not in VMHvv in the conditions of chronic social conflict. This finding may indicate differences in the coding schemes for short-term and chronic conflict. We find that the neural activity in MPO is correlated with the "secondary" belief's about opponents' strengths, in agreement with recent study the role of cMPO neurons (Wei et al. 2023), It is likely that other brain regions, such as VMHvl encode variables different from beliefs. This possibility may explain why we did not find significant correlates of beliefs in other brain regions related to aggression including some limbic areas, the brain stem, and the reward system (Aleyasin et al. 2018).
A.2 Limitations and strengths of our approach
In our setting, animals made their decisions in the conditions of incomplete information and refined their policies based on the observed outcomes of their encounters. Agents such setting face the problem of partially observable Markov decision process (POMDP). Here, use biological
13

---

decision- decision-making to propose an approach to solving the POMDP problem in multi settings. Our work extends machine-leanaee learning methods for analyzing behavioral and neural data these conditions.
Combining inverse rational control (Dvijotham and Todorov 2010 K won et al., (020) with game theory (Cressman et al., 2003) offers an interpretation of model parameters. Specifically, the variables in our models either represent the rewards for various outcomes or describe the confidence in taking various actions. A low number of fitting parameters makes our approach data-eflicient. This property may be useful in neuroscience where data are typically small (Stahl and Wilson 1995). Game theory makes the transitions between beliefs deterministic, reducing the computations compared to probabilistic Baum-Weled algorithms (Baum et al., 1970), conventionally used in POMDPs (Wu et al. 2018). The benefits of our approach can be quantified in follow-up studies.
A limitation of our game-theory-eee theory-based approach is that it involves model comparisons between ad hoc models (Stahl and Wilson 1995). To mitigate this limitation, we based our models on the results of previous studies, using standard ToM/MI. baselines (Stahl and Wilson 1995| Khalvati et al.) 2019). In future studies, we may consider additional models including mixtures of ToMs (Khalvati et al.. 2019) naive Nash, and rational expectation strategies (Stahl and Wilson 1995).
Our behavior-based model reconstructs relevant variables for individual animals, allowing us search for their correlates in neural activity. As a proxy of neural activity, we used the c-Fos expression, captured with whole-brain 3D microscopy (Renier et al., 2016). Using c-Fos comes with several limitations: i) c-Fos lacks temporal resolution, allowing one snapshot of brain activity per mouse; ii) the c-Fos signal is nonspecific. potentially encoding factors other than neuronal activities (Herrera and Robertson, 1996). Among the benefits, c-Fos does not perturb animals' social behavior, making the standard choice for studies of aggression (Aleyasin et al., [018] DDaz and Lin 2020, Wei et al., 2021). Ex-vivo 3D imaging allowed us to reconstruct the whole- neural activity. While previous studies focused on tissue sections of preselected brain regions, our whole-brain data has enabled an unbiased study of conflict circuitry.
The neural activation analysis in this work is performed with the correlation analysis followed by the FDR correction (Benjamini and Hochberg 1995). This is conservative analysis establishing whether the model variables are directly represented in neural activity. Alternttve approaches could include GLMs for detecting linear combinations of model parameters (Lindquist (2008), Gaussian random fields for modeling spatial correlations of the signal (Rue and Held| 2005), and MANOVA for distinguishing representations of correlated model variables (Allefeld and Haynes 2014). Using these and other fMRI approaches (Skup/ 2010 Ashburner et al. 2014) may allow identifying the correlates of individual model variables.
Our results are based on data comprising "attack"""defend" and "win"/"lose" readouts but our analyses can be extended to different metrics and social behaviors. Long-term tracking of social structure in rats (Nagy et al., 2023) may allow for testing our conclusions in a different species in a complex social environment. The data on carrier/non currier/ooncarrier phenotypes (Schroeder and Desor (2005) would allow us to see if our results generalize a cooperative setting. The model may also be applied to other domains, such as automated negotiation (Baarslag et al. 2016). Overall, our framework may be used for studying competitive or cooperative behaviors in real-world settings using limited data.
B Methods
B.1 Mouse chronic social conflict paradigm
To induce varied behavioral states in mice, applied the chronic social conflict paradigm. Pairs of weight-matched mice were separated by perforated partition in cages. Once daily, the partition was removed for 10 minutes to enable agonistic interactions between mice. To study the adaptive properties of the aggressive/defeated states in mice, we kept the opponents unchanged for 3 days. Then, to reveal the pathological properties of behavioral states, each winning mouse was kept in its cage, while each losing mouse was daily relocated to an unfamiliar cage with an unfamiliar winner. Non-fighting mice were discontinued from participating in the experiment. After a total of 20 days of interactions, to reveal the flexibility of behavioral states, winning and losing mice were reorganized to face opponents of the same state. The newly formed pairs underwent 2 more days of interactions. During all interactions, we logged the mouse actions ("attack", "defend") and the outcomes of interactions ("win", "lose", "draw"). We varied the participation of mice in the experiment:
14

---

Table 1: Datasets
DATASET
II
III
TOTAL
DAYS
3/ 10
20
22
MOUSE BEHAVIOR
EXP: 56; CTL: 6
EXP: 58; CTL:
ExP: 88; CTL: 6
EXP: 202; CTL: 18
BRAIN 3D
EXP: CTL:
EXP: 11; CTL:
EXp: 19; CTL:
EXp: 54; CTL: 18
B.2 Game theory optimal actions
To define a class of models to test against data, described the mouse task a normal-form game. Here, the agents simultaneously chose their actions; the available actions were to "attack" and "defend", The outcome of the game depended on the actions and strengths of participating agents. The strengths s; were defined as constants between 20, specific to an agent and unchanging.
Table 2: Game outcomes
ACTION
#2 "ATTACKS"
#2 "DEFENDS"
#1 "ATTACKS"
Pirin
Gibbs(S1. 82)
#1 WINS DRAW
"DEFENDS"
WINS
both agents in the game chose to "attack", the probability of "winning" the game was defined by the softmax rule over their strengths parameterized with the "outcome confidence" 3.. The reward assigned to each agent depended on the outcome of the game and on their action as follows:
Table 3: Rewards
ACTION
"WIN"
"LOSE"
"DRAw"
"ATTACK"
-A
N/A
"DEFEND"
N/A
To derive the optimal actions for agents in fully observable setting, parametrized the agents' actions by the probability to "attack" and derived the gradients of the expected reward w.r.t. these probabilities. We found pure-strategy Nash equilibria by choosing the pure strategies whose gradients were directed outside the support of action probabilities. We averaged the action probabilities over Nash equilibria. Finding the optimal policies is summarized in Algorithm T
Algorithm Game theory optimal actions
Input: strength range Smaz, costs a, confidence 8. Initialize actions A = zeros (2, Smaz, Smaz).
for strengths $1; 82 1 to mmar do expectation 61 = (1 + AA) softmax, (B081, (,82) gradient أ (k| + 0 1 1)p2 + 1; similarly find p
for policies P1. = 0 or do
Find pure strategy: (1) (P1(2) 0.5) > 0 end for Average the strategies: A(1, 82) mean ((p1)) A A(2, 81, $2) A(1, 82)
end for
B.3 Bayesian belief initialization
To account for the partial observability of information the task, we defined the probability distri- butions (beliefs) describing the strengths of the agents. We initialized the beliefs with the normal distributions whose mean values corresponded to the body weight of the animals (shifted by -15g to fit the range). To account for the prior information about the body weight distribution, we applied
15

---

Bayes' rule to the initial beliefs. To reflect possible depths of reasoning, we considered four types of beliefs parameterized by different standard deviations of the normal distribution:
Table 4: Initial helief standard deviations
BELIEF
"MINE"
"OPPONENT'S S"
"ABOUT MYSELF"
O1
Vo V 0 +
"AROUT OPPONENT"
02
The "opponnnnt's beliefs here reflect the agent's beliefs about the opponent's beliefs that may differ from the opponent's beliefs in the model. Overall, the acquisition of initial beliefs about the animals' strengths is summarized Algorithm
Here B Bimm is the belief of mouse mi about mouse m2.
Algorithm 2 Bayesian belief initialization
Input: body weights w;, uncertainties 01,02
Initialize beliefs B zeros(8mar, (8man, max(m), max(m)).
strengths Pr(s) (M(3]w)
for mice 1, m2
max( (m) do
Set using Table 4
for strength = to max {+}) do
belief B Bmmm, Z (wmm))
end for end for
B.4 Bayesian belief update
To enable evidence accumulation in our model, we used Bayes' rule to update the beliefs based on the observations. The agonistic interaction outcomes used to update the primary beliefs about the agent's own and the opponent's s strengths. The opponents' actions were used to update the beliefs about the opponent's beliefs. The update procedure for beliefs is summarized in Algorithm 3 below.
Algorithm 3 Bayesian belief update
Input: pairs p'm- actions a'n- outcomes of Compute actions using Algorithm precompute [RA]ijR = E, Rgg sonvga(A, P "rii, +o k
[OA]ehh = O conv (a P. (' Vottoo
Initialize beliefs B using Algorithm
for time t 1 to max((t)) do for mouse m, to max (fm}) do opponent m2 PLm, for mouse m m2 do
a = softmax, Eijual RAA iigt Bi B/, Bh BBn end for for strength i to max((s)) do
for mice m,, m2 do
AB Z
Ejkd [A [
Bi, Bi,
update Big eA (1-
Repeat for other belief types B.,
end for
end for
end for
end for
16

---

B.5
Model fit
Besides the belief update, the above algorithm allows for predicting the animals' actions based on their beliefs. We consider the opponent's 's predicted action to be game-theon thwooh- based on the opponent's 's estimated beliefs. To predict the agent's action, we maximize reward expectations based on its own "primary" beliefs and the opponent's predicted action. To optimize the model predictions, we update the model's s parameters to minimize the negative log-likelihood (NLL) for the actions in our behavior data to be sampled from the model predictions. While we use the entire data to propagate the beliefs, we only use days 1-3 and 21-22 for the NLL evaluation to avoid the impact of repeated actions on days 4-20. We use bootstrap compute the error quantiles for the identified parameters. Overall, the inference model parameters summarized in Algorithm below.
Algorithm 4 Model parameter inference
Input: pairs piu- actions a'n- outcomes on Initialize parameters x = [0,, 02 Sa Sor Ot A,
Initialize Zopt zeros 100, (0000)
for repeat i to 100 do
Define subset т E [1, T] with repetitions Define actions Pr(a using Algorithm| Eme log (r(a
Define likelihood L
Regularize L +- C X / 11,
Optimize Eopt (1) = fminsearch(() end for Compute E [[Fopp) , ID [Tpp)
To optimize the parameter inference, we applied it to simulated data with known parameters. We added noise Algorithm to generate the initial beliefs about strengths in the model and used Algorithm 3 simulate the actions. Once the model parameters were reconstructed with Algorithm used Gaussian processes estimate the means and the error quantiles for the predictions. We varied the regularizer À E (0; 0.1; 10] and chose the one that led to the best reconstruction (A = 1).
B.6 Model comparison
To test hypotheses about mouse decision- maaking we performed the model comparison. To this end, we optimized pairs of models on the training data (88 mice participating for 22 days) and computed the changes in the NLL based on the predictions for the testing data (104 mice participating for 3/10/20 days). We performed the t-tests on the changes in the NLLs for individual mice followed by the FDR correction. For pairwise comparisons, we picked initial model and considered one-step deviations from it (e.g. for the 0-ToM model, we separately considered "frozen" prior beliefs and, alternatively, shuffled body weights, but not their combination). We fitted each model in the same way, using a numerical optimizer until convergence. We considered new models until one of the models outperformed the initial one; then used this model round of comparisons using one-step deviations from this new model. We stopped after arriving at a model that was outperformed by any of the other models. Here list the considered models and specify their differences in comparison with the "default" (1-ToM) model described above:
initial model for the next
Table 5: Baseline models
TEST
BASELINE MODEL.
DEPTH OF REASONING
O-THEORY OF MIND
AVAILABILITY OF INFORMATION
SHARED BELIEFS
FLEXIBILITY OF POLICY CLASS ALGORITHMS
FLEXIBLE IN-FIGHT POLICY
RESCORLA- WAGNER
In Saecolla-Wagnee model, each action ("attack", "defend"") was associated with a value representing the expected future reward. This value was independent of the mouse's strength or the identity of an opponent. The values for the "attack" and "defend" actions were initialized using the expectations
17

---

over the distribution of game theory optimal opponents, then updated at learning rate e using the reward as a teaching signal. In the 0-Theory of Mind model, we only considered the "primary" beliefs of an agent (about its own and the opponent's s strength) assuming that the opponent acts in accordance with the same beliefs (even though the actual beliefs of the two agents were independent and may have differed). In the shared beliefs model, the 0-ToM beliefs were identical between the two opponents. In the flexible in-fight policy model, the policy gradients used in determining the Nash equilibria were computed for the expected rewards integrated with the "primary" beliefs of each mouse reflecting the in-fight adaptation to the opponent's actions. As the opponent's 's actions were observable in this setting, the model did not include "secondary" beliefs.
Table 6: Ablation studies
TEST
ABLATION
DYNAMICS OF RELIEFS
FIXED PRIOR BELIEFS
FIXED POSTERIOR BELIEFS
ROLE OF BODY WEIGHT
SHUFFLED BODY WEIGHTS WIDE BODY WEIGHT PRIOR
In the fixed prior beliefs model we used the zero learning rate for the belief update, retaining the beliefs unchanged from their body weight-bases initial values. In the fixed posterior beliefs model, we used the zero learning rate but the initial beliefs were substituted with the final beliefs produced with our default model. the shuffled body weights model, we shuffled the animals' body weights before using them initialize the beliefs. In the wide body weight prior model used the double standard deviation for the prior distribution of the animals' body weights at the belief initialization.
B.7 Neural correlates of model variables
To find neural correlates of the model variables, we normalized the registered c-Fos activity in each brain sample by dividing by the average c-Fos activity of all control samples from the same dataset. We converted the relative c-Fos activities to the log scale to obtain roughly normal distribution of relative activities. We normalized the model variables by their average estimated values for the control samples. Then, we computed the voxel-wise correlations between the processed c-Fos activity and model variables. To select the voxels significantly correlated with model variables, we performed the FDR correction. Finding the belief correlates in the brain is described in Algorithm bbllo.
Algorithm 5 Belief correlates in the brain Input: 3D brain images aligned to atlas A; beliefs B
for sample s in experimental samples do Normalize I3 log 42 - log (mean(eert) Normalize by = mean (b.) - mean(beert)
end for
for belief b in (B11, B ) do
for voxel i in 3D brain atlas volume do Compute pa + corr( b)rr(()),,))
end for
Compute {q:} FDR(U: (p.))
Find corT: = (j S i) 12 sort(qs) < 0.1}
Find brain regions r in atlas A: corr r
end for
Considered variables included: "primary" and "secondary" beliefs about the mouse's 's own and the opponent's 's strength for O-ToM and 1-ToM models; the estimated advantages (i.e. the differences between the same-order beliefs within a model); the estimated outcomes (i.e. the binarized estimated advantages); the predicted actions for oneself and the opponent; the Rescorla-Wagne values of the two actions and their difference. The control variables included: the body weights of mice the observed actions, the outcomes of the agonistic interactions, and the shuffled beliefs of each type.
18

---

B.8 Software and time budget
All computations were performed with Matlab R2022b. The mouse data fits were performed with GHz Intel Core i7-based Dell laptop computer. The simulated data fits were performed with 3 GHz Intel Xeon-based Supermicro computer server in 40 parallel threads. 0-ToM models converged within 2 minutes (per model per thread); 1-ToM models converged within 3 hours (per model per thread).
C Supplementary results
C.1 Game theory optimal actions
C.1.1
Pure strategies
In this section, we provide simple derivation for the game theory optimal actions in our model. The expected reward E[r1] for the first agent can be computed as its payoff matrix R multiplied by the action probability vectors Pi,2 describing the policies of the first and the second agents:
( -a 1) 1 PIP2(K) + (
E(ri] - PT R.P2 (p1 1 - pı)
+ pl app- (22)
Here P1,2 are the probabilities to "attack" for the first and the second agents, fi the expected reward for the first agent in the case when both agents "attack", and -a is the cost for the first agent in the case when only the second agent "attacks". The first agent may increase its expected reward by changing its policy Pi that depends on pı (the first agent can't 't change P2 P2 is controlled by the opponent). To optimize the expected reward [r]r..
we compute gradient w.r.t. P1:
[E[r1] = (R1 + 1)p2 1.
(23)
0p1
The agents may follow pure strategy with binary probabilities P1,2 to "attack". Such strategies correspond to Nash equilibria if the reward gradients point outwards the 10-11 interval for both agents so that the policy won't t have room to evolve. This requirement can be formalized as:
DEim (pi 1) > 0; DE
(24)
(p2 - ()) > 0.
Up2
Using the values of the derivatives from Equation 23 we arrive at the criterion for Nash equilibria:
{ (12 pure (Ki a a 1)+ + (p pure
( 0;
(o (urer (52 a- 1) + 1) gure
0.
(25)
An arbitrary task setting (defined by the parameters and a) may correspond to multiple Nash equilibria. To this end, we approximate the expected policy by averaging the action probabilities p1,2 over the identified equilibria. A better way to account for multiple Nash equilibria is integrate the action probabilities over their basins of attraction; this analysis can be added in future studies.
C.1.2 Lack of mixed strategies
To consider mixed strategies with nonbinary probabilities to take either of the available actions, assume that both agents in the game follow the gradients of their respective rewards to update their action probabilities. Following Equation 23 for these gradients, we arrive at the dynamical system:
dp p2
+ a 1) +1;
(26)
dp2 (R2 1
Any mixed strategy {pirised pmiized) } corresponds to attractor of this dynamical system, i.e. point (Pr, P2] where the time derivatives of both variables Pr, P2 are equal to zero and the phase portrait of the system is shaped as stable node a stable spiral that the agents' action probabilities could converge to this point. To analyze the dynamical system around the point of zero gradient, we linearize its dynamics via computing the Jacobian:
Pi P5
+ a K1
(27)
di
Ki
+
a
0
19

---

To describe the phase portrait of this system, we compute the eigenvalues of the Jacobian:
11.2 = =v + 1) + a - 1).
(28)
In the case of real eigenvalues 11, their product negative implying that the system's 's dynamics form saddle point. In the case of complex cigenvalues, the zero trace of the Jacobian implies that the system's 's dynamics form limit cycle, In both cases, the system's 's dynamics do not converge to the point of zero gradient, rendering mixed strategies infeasible.
C.1.3 Flexible in-fight policy
In the case where agents do not have perfect information about each other's strengths, their estimates of the Nash equilibria may be based on their beliefs about the strengths of the agents in the game. As these beliefs, generally speaking, are different between the contestants, their estimates of the Nash equilibria may also be mismatched. As such, these estimates cannot be considered as true Nash equilibria, requiring further evolution of the contestants' policies.
One way to account for this discrepancy is to update the policies during the fight. Although our tests suggest that such a mechanism inconsistent with animals' behavior in the experiment and, as such, is not used in our final model, we provide its description for completeness.
Updating the policies during the fight is supposed lead to joint Nash equilibrium. This involves two simple additions to our previously described model: i) the reward expectations in the payoff matrices 21,2 are integrated over the "primary" beliefs about the participants' strengths, and ii) the optimization is performed jointly for both participating agents as their policies are observable each other under the assumptions about this case.
Amending the payoff matrix Rr can be reduced to redefining its entry fig, the only one depending on the agents' strengths. An fighttht Nash equilibrium is only defined under the 0-ToM model: the "primary" beliefs are used to predict the odds of winning while the "secondary" beliefs, normally used for predicting the opponent's actions, are unnecessary actions directly observed:
Rig Σ K2 i, B2
(29)
That is, the equilibrium policy P1 for the first agent depends on the second agent's estimate of its expected reward 23 for the case when both agents "attack". The expression for the optimal pure strategies is then rewritten based on Equation 25 follows:
(pparee (N] + 1) + 1)
1 0;
(30)
( (pruret( (N2 a 1) + 1) pure-
>
The equations above describe the in-fight Nash equilibrium which is not a part of our model. This analysis, however, may be useful in future studies of different social behaviors.
C.2
Model fit
In data fits, we used the entire data propagate the beliefs while we only considered days 1-3 and 21-22 for the NLL evaluation. This is because, on days 4-20, "winners" were always matched with "losers" predetermining the results of the interactions and reducing the NLL sensitivity the model parameters. As result, the model using days 1-3 and 21-22 has a local minimum (Figure 73) at the identified parameters, while the model using days 1-22 has a plateau instead (Figure (7)).
To assess the reliability of our data fits we performed additional fits on simulated data. To this end, replicated the mouse experiment in simulated environment and generated 500 choice patterns using various predefined parameters. We sampled parameters uniformly from the ranges previously used for parameter fits in mouse data, selecting only the policies that resulted in 30%-70% fraction of either action. We then evaluated the accuracy of reconstructed beliefs in both 0-ToM (Figure 7 and 1-ToM (Figure (-H models.
C.3 Model comparison
Below, we report the numerical results for the final round of model comparisons on our behavioral data. The differences in NLLs were computed for each mouse individually; here show the mean
20

---

Belief
BeliefBr
Belief
15
15
T15
10a
Reconstructed
Full data
B
Initial pairs inversion
10a
10
10
10
Negative Log- -i-klllhhoood
0=8.7
5
0=.73
5
r=0.64
10
15
5
10
15
5
10
15
D
Bellef
Bellef
Bellef
Reconstructed
3
3
10
10
10
10
10
10
Uncertainty d;
Uncertainty
0.66
00034
0=0.6
10 Simulated
15
5
10
15
10
15
Simulated
Simulated
Figure 7: Parameter inference. (A-B) NLL for reconstructing the animal data: (C-H) reconstructed belief's in simulation (color code: density beliefs; r-values: Pearson correlation between simulated, reconstructed beliefs.
ANLL and its standard error of the mean for the training and testing datasets. Positive numbers correspond to our model explaining the data better than baselines/ablations. To establish which of the results are significant, we performed the t-test A followed by the FDR correction. We highlight in italies the results that did not pass the threshold q-value of 0.05.
Table Model comparison
MODEL
LN
Perain
Itrain
LN
Ptest
Oteat
O-THEORY MIND
0.31 1 0.06
10-6 8
10-6
0.44
0.08
10-7
10 6
SHARED BELIEFS
0.56 1 0.12
10-5
10-5
0.65 + 0.11
10-7
10-7
FLEXIBLE IN-FIGHT POLICY
0.46 0.17
0.01
0.01
0.29 0.13
0.03
0.04
RESCORLA- WAGNER
0.13
0.29
0.29
0.14
0.21
0.21
FIXED PRIOR BELIEFS
0.11
10-6
10
0.68 1 0.12
10-7
10-7
FIXED POSTERIOR BELIEFS
0.52 0.100
10
10-6
0.51
10-5
10-5
SHUFFLED BODY WEIGHTS
0.16
0.02
0.02
0.18
0.07
0.08
WIDE BODY WEIGHT PRIOR
0.12 0.02
10-7
10
0.12
10-6
10
C.4 Normative theory of social conflict
Lastly, we summarize our behavioral and neural findings to propose reward- model of decision-makee makiin during social conflict.
1. Mouse decisions in chronic social conflict can be described with a game theory model where agents choose actions maximize the perceived reward of winning minus the of losing, and the probability of winning depends on the animals' strengths;
2. Animals' behavior is consistent with the perceived cost of losing while attacking being larger than the cost of losing while defending and also larger than the reward for winning;
3. Animals' actions and neural activity are consistent with maintaining the first-order Theory of Mind (1-ToM) endowing animals with primary beliefs about their strengths and strengths their opponents, and secondary beliefs about the beliefs of their opponents;
4. Initial beliefs, best describing the animals' actions the model, are based on the body weight for oneself and reflect the population average for opponents;
5. Belief updates, consistent with animals' behavior, are described with Bayes' rule based on observed actions and interaction outcomes;
6. Animals' actions are likely decided before each agonistic interaction and do not adjust to opponents' actions during agonistic interaction.
These conclusions suggest that pathological aggression and social defeat may emerge from beliefs about own strength, updated over interactions.
21


Reviews:
[{"review_id": "b6Y8j8oTbs", "reviewer": "Reviewer_hLmb", "summary": "This work provides a normative framework for reasoning about the strength and, ultimately, the chance of winning in a social conflict in mice. The analysis supporting the practicality of the framework includes both behavioral and neural data.", "questions": "Mostly reflection of mentioned limitations, but to be more precise: \n1- Are there any more human interpretable measures that you could use for your comparisons?\n2- Sorry if I missed it, but it looks like that the models are not penalized for number of parameters or depth of computaions? Could you present your results considering these penalties?\n3- What is the performance of the model that assumes a shared model between opponents? The authors mentioned that they don't have to be, but did they compare with the fixed version (+ considering it has less parameters)? I think if this shared model works it is still valuable in terms of social models (something like joint attention), and also more convincing to be existed in mice than 1-ToM (and maybe 1-Tom do exists, but as I mentioned above it is very controversial)   \n\nUpdate after rebuttal: from 5 to 6 as mentioned in the comments", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The topic of the paper, i.e., using AI models for the analysis of behavioral and neural data in animals during social interactions, is really interesting. It also covers a variety of topics and models relevant to the work. Furthermore, it used different types of data (behavioral and neural) in the analysis.", "weaknesses": "My main concern about the paper is their claim about the existence of Theory of Mind (ToM) in mice. This is highly controversial in psychology, even for non-human primates, and such a claim needs a lot of control experiments.  Moreover, reasoning about strength is different from mind. In addition, I am not fully convinced that level-1 performs significantly better than level-0 as the cost of parameters and deeper reasoning is not considered. Presentation of the results in likelihood, as opposed to sth more human interpretable such as accuracy, makes it hard to evaluate the fits. Even if the results hold, I think the authors should use another term such as joint reasoning about the opponent, or \"1-ToM like\" framework, with a detailed explanation about the model mimics 1-ToM models in humans and do not necessarily mean the existence of ToM in mice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and comments. As we respond to specific points below, please also see the general response to all Reviewers.\n\n**Existence of ToMs in mice.** We agree that the _existence_ of ToMs in mice is controversial and, therefore, we limit our claims in the text to mouse behavior being _consistent_ with running ToMs, which is a lesser claim. We reflect it in the paper’s title, throughout the text, and in the Discussion. We’ll be happy to stress this distinction more.\n\n**0-ToM vs 1-ToM.** The exciting thing about the game-theoretic model that we used is that the 0-ToM and 1-ToM models have the same number of parameters, so they can be compared directly using formal model comparison methods. As the behavioral data is noisy, we also considered the neural correlates of hidden variables in 0-ToM and 1-ToM. We found unique correlates of 1-ToM in the brain that are not explained by 0-ToM or other task-related variables; this result is consistent with the 1-ToM model. Notably, we also registered a (smaller) number of the unique 0-ToM correlates in the brain, so we do not refute that model either.\n\n**Log-likelihood vs. prediction accuracy.** The standard way to compare models in behavioral research is using formal comparison criteria, such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC); both are computed as a (function of) the number of parameters in the model minus the log-likelihood of this model to generate the original data. Most of our models have the same number of parameters. As we only evaluate the difference of AICs (or BICs) between the models, the term reflecting the number of parameters cancels out, resulting in the difference in log-likelihood that we present. We thus follow the standard approach in behavior modeling.\n\n**Human-interpretable measures.** In principle, we could use the prediction accuracy as another readout for models’ performance, although we don’t think that it would be informative. The AIC/BIC, represented by the log-likelihood in our case, is a standard metric for behavior models and is also more sensitive to the models’ performance compared to (binarized) classification accuracy. Therefore, here we present the standard AIC/log-likelihood metric.\n\n**Penalty for the number of parameters.** Most of our models (counterintuitively) have the same number of parameters, so the penalty is not needed (e.g. 0-ToM vs. 1-ToM). The ablation models have 1 less parameter each, and the Rescorla-Wagner has three less parameters. Here we describe why we chose not to put this distinction into a penalty. Consider the AIC criterion: AIC = 2(k – log L) where k is the number of the parameters. Delta k between the models is no larger than 3 (between RW and ToMs), which has to be divided by ~60 mice in a dataset (since we fit the parameters jointly for all mice but evaluate each mouse separately to enable the t-test on Delta AIC). Thus, in our plot, the correction for the number of parameters would be smaller than 3/60 = 0.05, which is negligible compared to the typical Delta log-likelihood. As for the penalty for the model complexity, we’ll be happy to hear the Reviewer’s suggestions.\n\n**Shared model / fixed model.** We are not sure what the Reviewer means by the shared model but, when it comes to the ‘fixed’ model, we performed this analysis (see the ‘fixed prior beliefs’ analysis). It has one less parameter than the ToM models (the AIC correction for this penalty would be 1/60) so, even accounting for the number of parameters, it is still outperformed by the ToM model.\n\nWe are happy to answer any further questions and/or to provide further clarifications."}, {"review_id": "HKieazI5g4", "reviewer": "Reviewer_QTqc", "summary": "This paper uses behavioral and neural data in mice to develop a game-theoretic and Bayesian theory-of-mind based model of social conflict. Mice are assumed to choose whether to attack or defend optimally (the game-theoretic component) based on their beliefs (and beliefs about opponent's beliefs; the BToM component). The authors show that mice behavior is best explained by the BToM model, rather than simpler models (e.g. pure RL over attack/defend actions) or ablated versions of their model. Finally, neural correlates of the model are identified.", "questions": "- I found Fig. 1D-F very confusing. Why are there gaps in the data, both on the X-axis (\"skipped\" days) and Y-axis (\"skipped\" mice)? Substantially more explication would be helpful here.  \n- Could you clarify the justification for exposing mice to varying schedules of conflict (i.e. 3 / 10 / 20 / 22 days)? It wasn't clear to me how this related back to the theory, or why it was necessary. Was this solely for the purpose of analyzing neural correlates at different stages of conflict? Providing a diagram or a table illustrating the different conditions and analyses would help clarify this.\n- Were nuisance variables (e.g. overall win-loss record) regressed out prior to performing the neural analysis? It seems like you might want to control for this.\n\n**Note**: original score `5`, post-rebuttal `6`", "limitations": "No concerns. Review board approval was obtained for animal experiments. ", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is ambitious, covering both a theoretical model and extensive empirical validation. Based on my limited understanding, it proposes and provides evidence for the role of ToM in a fundamental social conflict paradigm. If the general novelty of the approach is in fact high, I agree this could form an interesting basis for future work.\n- The model itself is interesting and seems well-justified (although it took a few reads to understand the structure, see weaknesses). \n- The science (experimental paradigm, model comparisons and ablations) seems sound. I appreciated the comparisons to, e.g., simpler R-W reinforcement, which justify the ToM-based approach.", "weaknesses": "- Presentation. I found the paper difficult to understand: explanations of the experimental paradigm, model, and neural analysis were all compact and somewhat cryptic. Relationship to prior literature-- especially to contextualize the findings, such as neural correlates-- was terse, and the discussion itself was relegated to the appendix. This made it difficult to understand the contribution.\n- The discussion of neural findings (both in the main text and appendix) was particularly brief. I'm vaguely aware of neural imaging of ToM elsewhere (e.g. in human gameplay [1,2] or pedagogy [3]) but these connections aren't discussed. Obviously these studies are done in people not mice, but given the authors' stated ambition to be relevant to human behavior, I think it'd be helpful to sketch out how these findings relate to imaging work there. \n- This is more of a note to the authors than a weakness of the scientific contribution itself, but I'm not sure NeurIPS is the best venue for this work. There's a lot of ground to cover here  for 9 pages.  Similarly, I'm not sure how relevant the findings will be for the majority of the NeurIPS audience. I was personally unfamiliar with the experimental paradigms and background literature, so I had trouble judging its significance. A neuro journal with a more expansive format and familiar audience might be a better venue.\n\n[1] https://www.pnas.org/doi/10.1073/pnas.0807721106\n[2] https://www.pnas.org/doi/10.1073/pnas.0711099105\n[3] https://www.pnas.org/doi/abs/10.1073/pnas.2215015120", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and comments. As we respond to specific points below, please also see the general response to all Reviewers.\n\n**The general novelty of the approach and relevance to NeurIPS.** Please see the general response to all Reviewers.\n\n**Presentation.** We are happy to further work on the text with the coauthors and broader colleagues to make the text more understandable. The current unfortunate terseness was imposed by the NeurIPS page limit and numerous considerations we had to discuss regarding our analysis. Specifically, we are happy to extend the Introduction / related work section along the lines discussed in the general response to all Reviewers here to better contextualize our work for the NeurIPS readers.\n\n**Neural imaging of ToM.** We thank the Reviewer for this suggestion and agree that we should discuss more related fMRI literature in the paper. Sadly, the techniques used in fMRI do not exactly apply to our analysis, as they mostly seek the _differences_ caused by a certain experimental condition and/or the _regression_ of the signal, while we focus on the _correlations_ between different levels of c-Fos with different amplitudes of the beliefs. We have a separate project that uses the same data where the fMRI-style analyses would apply, so we appreciate the literature recommendation.\n\n**Figure 1D.** We would appreciate comments on how to improve this figure! Below, please find our initial thoughts:\n- the three panels were meant to highlight the three phases of aggressiveness: ‘normal’, the formation of ‘pathological’, and the ‘reversal’ (from ‘winner’ to ‘loser’ and the other way around);\n\n- what looks like the ‘skipped days’ meant to separate the data of different days. Additionally, different mice had different numbers of interactions on the same day (e.g., known ‘aggressors’ would be matched with ‘undecided’ mice to facilitate them forming a ‘winner’ or a ‘loser’ phenotype);\n\n- the ‘skipped mice’ on the first three days are actually (mostly) a plotting error, thanks for catching that one! We will fix that in the revised figure.\n\n**Varying participation of mice.** The Reviewer is correct in suggesting that the varying participation of mice was done to enable brain imaging on different stages of conflict: ‘normal’, ‘pathological’, and ‘reversal’). Please find additional details in Supplementary Table 1.\n\n**Nuisance variables.** Unlike in fMRI, neural correlates of these variables are something that is directly looked at in conventional studies of aggression. We did not regress the nuisance variables out in the presented analysis as what _we_ looked for were the _correlates_ of _individual_ model variables and not the brain regions whose activity could be regressed to the _set_ of the model variables. Instead, to make sure that nuisance variables do not affect our results, we looked at the voxels whose activity is directly correlated with various task-related variables. We established that a substantial number of voxels, correlated with 1-ToM beliefs, are not explained by any other considered variables. This design choice was loosely inspired by the body of literature (circa 2000-2015) on the effects of normalizing the fMRI signal, which hasn’t converged to a universally accepted way of processing data, which has convinced us to present what we saw as the most direct and conservative analysis. With that said, before submission, we performed additional analyses where we _normalized_ the c-Fos signal by some nuisance variables (data not shown); the result was similar to the one we presented.\n\nWe are happy to answer any further questions and/or to provide further clarifications."}, {"review_id": "ZULQkNlRD1", "reviewer": "Reviewer_UypE", "summary": "The paper models social conflict in mice using a game theoretic Bayesian theory-of-mind model. They find that the mice's actions are most consistent with the ToM model where agents (mice) maintain primary beliefs and secondary beliefs about the beliefs of their opponents. ", "questions": "See above", "limitations": "The strengths and limitations are relegated to the appendix. As described in \"weaknesses\", I don't think there is enough contextualizing for the motivation and takeaways of the paper. I think the authors should move the limitations/strengths to the main text and move some of the ablation studies/method description to the appendix instead.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper models an interesting dataset of mice in social conflict which includes whole-brain neural data. They do many comparisons between different models and ablation studies in their experiments.", "weaknesses": "I saw two main weaknesses in the paper. The first is that the paper needs more explanation of the significance of the results. The introduction is combined with the related work, and in particular, the \"introduction\" part is only one paragraph while there are four paragraphs describing related work. I couldn't understand what the specific contributions of this paper were compared to previous results. I also wasn't sure about the \"so what?\" of the paper. I think the authors could move the ablation studies in Sec 3.1.1 to a appendix and expand more about the contributions of their results. The description of the model and how it is fit also takes up a lot of space, some of this could probably be moved to an appendix.\n\nThe second weakness is in the empirical evaluation of experiments. I'm confused why the models are evaluated by the change in negative log likelihood on the test set. If I'm understanding currectly the actions and outcomes are discrete (either attack/defend or win/lose).. meaning the ground-truth probability of an action/outcome at a given time is either 1 or 0. So, evaluating the models on log likelihood means that models that make more extreme predictions are preferred. Log likelihood is used to fit the model, but in testing, we just care about the predictive accuracy. Why not just evaluate on predictive accuracy? It would be more interpretable too.\n\nThe authors evaluate many models, compute many significance tests, and control for FDR because of the multiple corrections. However, they only seem to report the results for the *best* models: \"Our comparisons were iterative: they lasted until a model under consideration outperformed all the other models. Below, we report the results for the final round of these comparisons.\" There aren't enough details in the paper to understand how exactly they controlled for FDR. Did they only control for FDR for the best model comparisons shown in Fig 4? It seems that there may be a _selective inference_ problem going on here. Generally, it makes sense to report a table in the appendix with all hypothesis tests that were run, the resulting p-values, and their corresopnding q-values. I would ask that the authors add this in their revision for clarity.\n\nSince the main result seems to be that the 1-ToM does better than 0-ToM (at least that's what is highlighted in the abstract), why not also test 2-ToM? Is it possible to recover a 0-ToM model using the 1-ToM model class they've defined? In that case, one would expect that 1-ToM fits the data better (but perhaps doesn't generalize as well). If the authors compared to 2-ToM, it would provide stronger evidence that 1-ToM is the correct model (as opposed to simply having lower log likelihood because of having the flexibility to use more parameters.\n\nI may have misunderstood some of the experimental evaluation, and if so, I am willing to change my mind on the above points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and comments. As we respond to specific points below, please also see the general response to all Reviewers.\n\n**Significance of the results and contribution of the paper.** Please see the general response to all Reviewers.\n\n**Log-likelihood vs. prediction accuracy.** The standard way to compare models in behavioral research is using formal comparison criteria, such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC); both are computed as (a function of) the number of parameters in the model minus the log-likelihood of this model to generate the original data. Most of our models have the same number of parameters; others have a similar number of parameters. As we only evaluate the difference in AICs (or BICs) between the models, the term reflecting the number of parameters cancels out, resulting in the difference in log-likelihood that we present. We thus follow the standard approach in behavior modeling.\n\n**Log-likelihood and the extreme predictions.** While this is true that the log-likelihood favors extreme predictions as long as they are _correct_, the _incorrect_ extreme predictions are heavily penalized. As a result, the log-likelihood cost function overall favors balanced, non-extreme predictions.\n\n**FDR for non-final comparisons.** Thank you for this comment, this is something that we overlooked. In the revised version, we’ll be happy to present the FDR score computed across all final and intermediate results to avoid selection bias. We ran this analysis and the results seem to hold, although we’ll doublecheck it to make sure that we included all interim results in the study.\n\n**2-ToM.** This sounds like a good idea, thanks. Initially, we only compared 0-ToM and 1-ToM because we thought that the result would favor 0-ToM (consistent with the Reviewer's proposal). Time permitting, we will consider doing the 2-ToM analysis (it may take a long time to script it and ensure that all equations are correct; it also takes an exponentially more extended time to run).\n\n**Recovering a 0-ToM model with the 1-ToM model.** If we understood the question correctly, it is not possible to recover a 0-ToM model using the 1-ToM class under our definition. Our 0-ToM model assumes a game-theory-optimal opponent under the same beliefs, whereas the 1-ToM model separately estimates the opponent’s beliefs and assumes that the opponent assumes _our_ action to be game-theory optimal. We agree that this is an important consideration that would be well-placed in the Discussion. We would like to highlight that the 0-ToM and the 1-ToM model have the same number of parameters, so the latter model shouldn’t be overfitting compared to the former model.\n\nWe are happy to answer any further questions and/or to provide further clarifications.\n"}, {"review_id": "8Rxh1cbQVc", "reviewer": "Reviewer_RYok", "summary": "This paper aims at explaining the underlying principles of social conflict with the Theory of Mind modeling. The authors collected the behavioral data of 100+ mice in agnostic contact events, and fit the parameters of the Bayesian inference model. The experiments show that animals' actions are consistent with modeling both the first-level belief of strengths and the second-level belief of the opponents. ", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The model comparison considers a comprehensive suite of contributing factors. The model’s effectiveness is clearly demonstrated by contrasting different control groups. The success regression of the Bayesian model to the mice data shows that both the first and second-level beliefs are employed for decision-making.  \n", "weaknesses": "This paper is difficult for me to make a judgment on. The modeling and the ablation studies are sound. However, the Bayesian belief update framework is already well-studied and there is no new algorithm-wise contribution. The model fitting showing both the first and second-level ToM are used in multi-agent decision-making is not new discovery as well. I’m open to discussion with other reviewers and authors about how to posit this paper and its contribution to Neurips. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and comments. For the details on the novel contributions of the paper, please see the general response to all Reviewers. We look forward to editing the paper’s Introduction based on the Reviewers’ suggestions and the upcoming discussion, along the lines outlined in our response to all Reviewers.\n\nWe are happy to answer any further questions and/or to provide further clarifications.\n"}, {"review_id": "uRHHFtEMOG", "reviewer": "Reviewer_uQtV", "summary": "The paper presents experiments conducted with real mice, that were put in varying conditions of social stress. In particular, pairs of mice where put together and their behaviour was observed. The attacking mouse was classified as dominant and winner. Initially, loosing mice were move, whereas winning mice stayed. Mice were euthanised and their brains were analysed for brain activity, based on some marker which has been shown to be an indicator.\n\nThe results were recoded and various models where fitted to the data in order to explain when mice attacked or defended, based on their believe of the opposing mouse. The data is based on more than 100 mice.", "questions": "- Are there any documentation of ethical standards, consideration, etc. which are required for mice experiments?\n- What are the conclusiosn from fitting the models to the data?\n- Please explain the underlying assumption of the data, why is attacking = winning? What effect does moving mice have as compared to remaining? Are the environments different? Is moving more stressful for the mouse that is moved, if so, why?\n\n", "limitations": "The authors state \"A detailed discussion of our results including the strengths and limitations of our approach is provided in Appendix A\". I expect the discussion of the limitations in the main paper.\n\nThe title of the paper is \"A normative theory of social conflict\". I don't see various models fitted but I can't see a normative theory that is derived from the experiments.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of the paper is appealing. Using data of real animal behaviour and try to model it with a game theoretic approach. The sample size is large, compared to other animal studies. The math and results are presented well.", "weaknesses": "There are several issues with this paper.\n\nThe biggest issue that I have with this paper is that more than 100 mice where ethanised without any discussion of documentation of ethical standards and considerations.\n\nFurthermore, the mice are placed purposely in stressful conditions and it is not clear how the collected data is a good reflection of their beliefs. As an example. Winning mice remain in the environment, while loosing mice are moved to another environment. Yet, attacking is considered winning. As a mouse, not moved to another enviroment, defending could be the more natural behaviour if a new animal is placed in the environment. A male lion defends it's place while the youg male attacks to take it's place. The classification of attacking = winning is not well argumented, which questions the underlying assumption.\n\nThe discussion of the broader impact is mainly restating the experiemnts and methods. There is no real discussion what can be learned from fitting the models to the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "While we appreciate the Reviewer taking possible ethics concerns seriously, we would like to point out that they’ve been (i) addressed in the original text via the ethics statement in the Methods section [line 95, supplement] and via referencing the experimental paradigm in the Introduction [line 23, main text], (ii) not raised by 4 other Reviewers, and (iii) not confirmed as problematic by 2 Ethics reviews based on the very same text. Please see the details below, alongside our responses to other specific questions.\n\n**As the rating and confidence mean that they’re based on the ethics concern, we ask that the Reviewer re-evaluates the paper to reflect the cleared ethics concern.**\n\n**Ethics concern.** Our experiments match or exceed the requirements equivalent to those of the NIH. Before the start of the project, the experiments were approved by a qualified board specialized in animal research ethics. We will name the exact board and the exact guidelines in the non-anonymous version of this paper. Our experimental design is standard for the field of aggression, which uses mouse/rat behavioral experiments with subsequent c-Fos imaging. Based on a paradigm used for over 25 years (e.g. see Kudryavtseva et al, 2000), similar experimental designs have been approved in a large number of experiments by boards in Europe and the US.\n\n**Number of mice.** To track the progression of aggressiveness/defeat and related neural activity, we needed to evaluate the stages of aggressiveness including ‘normal’ aggression (3 days), ‘pathological’ aggression (20 days), the transition between the two (10 days), and the reversal of aggression (22 days) separately for ‘winners’ and ‘losers’. Each experiment has to come with a control group. Mice could not be shared among experiments because the c-Fos imaging can only be performed once per animal.\n\n**For the other details of the experiment and their justification**, including the moving of the mice, please refer to Kudryavtseva et al 2000, cited in our submission.\n\n**The conclusion from fitting the data / the normative theory that’s derived from the experiments.** Thank you for this question. In the draft version of the paper, we had a formal summary of our findings, framed as a theory of social conflict. While we removed it from the text to save space, it may make sense to re-introduce it to the Appendix as a whole. Please find it below:\n\n1. Mouse decisions recorded in the chronic social conflict paradigm are consistent with a game theory model where agents choose actions to maximize the perceived reward of winning minus the cost of losing, the probability to win depends on the animals’ strengths, and the uncertainty about strengths is parameterized as probabilistic beliefs;\n2. The perceived reward of winning, consistent with animals’ behavior, is smaller than the cost of losing while attacking but larger than the cost of losing while defending;\n3. Animals’ actions and neural activity are consistent with maintaining the first-order Theory of Mind (1-ToM) including primary beliefs about the strengths of all involved and secondary beliefs about the beliefs of their opponents;\n4. Initial beliefs, best describing the animals’ actions in the model, are based on the body weight for oneself and reflect the population average for others;\n5. Belief updates, consistent with animals’ behavior, are described with the Bayes’ rule based on observed actions and interaction outcomes, progressing at the unit learning rate;\n6. Animals’ actions are likely decided in advance and do not adjust to opponents' actions.\n\n**For the discussion of these statements,** please refer to the Supplementary Discussion section of the original text.\n\n**For the broader implications of our results,** please see the general response to all Reviewers.\n\n**On attacking and winning.** We do not equate the attacking and the winning. For example, early in the experiment, both mice would attack but the assigned outcome would be a ‘draw’. Overall, we suggest interpreting the ‘attack’, ‘defend’, ‘win’, ‘lose’, and ‘draw’ terms within the game theory in the way we defined it in the original text.\n\nWe are happy to answer any further questions and/or to provide further clarifications.\n"}]

Author rebuttal:
We thank the Reviewers for their time and comments. To our understanding, the main request put forward by the Reviewers is to expand the paper’s introduction, describing the state of the art in aggression/conflict research and specifying our contribution to it. We address this request here in our general response and respond to other questions individually for each Reviewer.

Research of conflict is an established field in cognitive science and neuroscience, asking what processes lead individuals to become pathologically aggressive or pathologically submissive. Among numerous other researchers, we find this question to be of high societal importance, as aggressive/submissive behaviors emerge in various social settings, notably including bullying in schools and hostility in prisons. Understanding the neural and behavioral basis of conflict should ultimately allow us as a society to mitigate the negative consequences of aggression and prevent cases of hostility.

The field of aggression/conflict studies has traditionally been qualitative/experimental. The established approach in the field is to ask a binary question (e.g., whether a certain environmental factor increases aggressiveness, or whether a certain brain region changes its activity upon social defeat). To ask these questions, researchers conventionally observe rat/mouse behaviors and record from their brains. Notably, rodents exhibit aggressive behaviors naturally when co-housed in the same cage (which is standard in animal research), quickly forming hierarchies, so no additional actions/stress is put on animals during these experiments. The state-of-the-art results in the field comprise the lists of environmental factors facilitating aggression and the lists of brain regions whose activity is changed as a result of aggression.

While asking binary questions has revealed many facts about aggressive behaviors, this approach doesn’t tell us _how_ aggressiveness develops in an individual, limiting our ability to predict and mitigate harmful behaviors. Here, we set to address this question using behavioral data modeling (which, to the best of our knowledge, hasn’t been applied to aggressive behaviors) for which we needed expressive enough data (as a result, we had to collect it). Compared to prior research in the field, we:

- collected data that covers the _progression_ of aggressiveness/defeat in individual mice;

- proposed a family of models to describe the learning rule, decision rule, and cost function in aggression;

- fitted the family of models to the data on _dynamics_ of aggression to find a model matching the data;

- based on model fits, described the decision-making related to the aggression/submissiveness;

- analyzed the neural correlates of explicit and _hidden_ variables across the _entire_ brain;

- provided validation for our hidden-dynamics model, relating brain regions to _individual_ variables.

To the best of our knowledge, none of the above has been done in aggression literature. Why is it important? A model of this kind can be extremely useful in providing a clear path to addressing the major question of aggression research, that is, how does pathological aggression/defeat develop and what could be the steps to mitigate it. Straightforward follow-up research, enabled by our approach and results, includes:

- testing the model on newly available data (see Discussion);

- identifying correlates of all aggression-related variables (observed and hidden) in the brain and tracing the connectivity of corresponding brain regions. This may reveal the decision-making neural circuit for aggression, highlight the roles of individual brain regions, and enable the shift from a normative to a circuit model of conflict;

- either model then may be used in modeling social interactions to design policies reducing overall aggression. The result may be useful for the school system to reduce bullying.

While the future directions (i.e. the result of our research) drew attention in neuroscience, the machine-learning framework enabling it (i.e. the way these results were obtained) was among the secondary considerations for this experiment-heavy field. Nonetheless, it’s this novel combination of standard machine-learning tools that has enabled these results and that can be highly useful for modeling other social interactions. We find NeurIPS most appropriate for this goal for the following reasons:

- NeurIPS conventionally encompasses neuroscience, machine learning, and their useful interaction;

- we are aware of a relatively large community within NeurIPS who would be interested in these results (perhaps, the fact that we are aware of them, has excluded them from our Reviewers);

- the conference attendees may benefit from our work being exposed to them as, apart from neuroscience, our novel approach combining game theory and Bayesian beliefs can be useful in designing human-robot interactions, e.g. assisted manipulation or autonomous driving, and recommender systems, e.g. informing a person of the robot’s/service’s capabilities.

Finally, our work belongs to the field of NeuroAI, one of whose aims is to use machine-learning approaches to enable better data analysis in neuroscience. While there is a huge potential for addressing questions of societal importance using ML (as we hope is exemplified by this paper), the NeuroAI approach, being on the interface of the two fields, is somewhat alien to either of them. The only way we can showcase the usefulness of this approach is through representation and we are confident that NeurIPS is the best venue for such representation, featuring the growing NeuroAI community and participants from adjacent fields. Delivering long-awaited results regarding the brain-wide involvement in conflict that would not be possible without machine learning, our paper fits the NeurIPS agenda.

We are happy to participate in the discussion and to answer any further questions.

Thanks for your consideration.

Meta-review: This paper links experiments with computational modeling and crosses levels of analysis. The results highlight the role of particular brain regions in relation to a computational model of social decision making. Reviewers and myself appreciated the thorough analysis and the way it linked abstract variables of the computational model to specific brain regions, measured with c-Fos. This is exactly as computational neuroscience is meant to work. It is clearly in scope for NeurIPS.
Decision: Accept (poster)