SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sergey Shuvaev<sup>1</sup>, Evgeny Amelchenko<sup>2</sup>, Dmitry Smagin<sup>3</sup>, Natalia Kudryavtseva<sup>3</sup>, Grigori Enikolopov<sup>2</sup>, and Alexei Koulakov<sup>1</sup>
$^{1}$ Cold Spring Harbor Laboratory, Cold Spring Harbor, NY, USA;  $^{2}$ Department of Anesthesiology and Center for Developmental Genetics, Stony Brook University, Stony Brook, NY, USA;  $^{3}$ Institute of Cytology and Genetics, Siberian Branch of Russian Academy of Sciences, Novosibirsk, Russia. koulakov@cshl.edu
Social conflict is a survival mechanism yielding both normal and pathological behaviors. To understand its underlying principles, we collected behavioral and whole-brain neural data from mice advancing through stages of social conflict. We modeled the animals' interactions as a normal-form game using Bayesian inference to account for the partial observability of animals' strengths. We find that our behavioral and neural data are consistent with the first-level Theory of Mind (1-ToM) model where mice form "primary" beliefs about the strengths of all mice involved and "secondary" beliefs that estimate the beliefs of their opponents. Our model identifies the brain regions that carry the information about these beliefs and offers a framework for studies of social behaviors in partially observable settings.
Research into conflict behavior is an established domain within cognitive science and neuroscience with a focus on exploring the origins of pathological aggression and defeat (Archer, 1988; Rosell and Siever, 2015). The broad interest in this topic is driven by the pervasive occurrence of pathologically aggressive behaviors such as school bullying and prison hostility that remain a significant societal concern. Gaining insight into the neural and behavioral foundations of conflict behavior is a crucial step toward mitigating the adverse consequences of aggression and preventing instances of hostility.
Toward this goal, the field of conflict studies has adopted an experimental approach. Researchers have monitored rodent behavior and recorded corresponding brain activity, as rodents naturally exhibit aggressive behaviors allowing experimenters to minimize additional stress for animals during data collection. This approach has yielded valuable insights into the factors that promote aggression (Archer, 1988; Wang and Anderson, 2010; Lorenz, 2005) and led to identifying brain regions affected by conflict (Aleyasin et al., 2018; Diaz and Lin, 2020; Wei et al., 2021). The studies in this field, however, rarely focused on the dynamic nature of aggressiveness in individuals, thus constraining our ability to understand and predict harmful behaviors. Here, we use behavioral data modeling to explore the strategies of aggressive behavior in individual mice over prolonged periods of time.
Our study contributes to the field as follows. We collected large-scale data on behavior and whole-brain neural activity that follows the emergence of aggression and defeat in individual mice. We also introduced a family of models to characterize the learning rules, decision-making processes, and cost functions underlying aggression. These models were applied to the observed behaviors of individual mice to identify the best-fitting model. Using that model, we described the decision-making mechanism consistent with the emergence of aggression and submissiveness. We then conducted an analysis of the neural correlates of both explicit and hidden model variables across the entire brain, thus validating the hidden dynamics of the model and linking specific brain regions to individual model variables. While similar approaches were used in theoretical and computational neuroscience, they have not been previously applied in studies of conflict.
Our findings show that behavioral and neural data describing chronic conflict aligns with the first-level Theory of Mind (1-ToM) model. According to this model, mice form "primary" beliefs about their own and their opponents' strengths along with "secondary" beliefs estimating their opponents' corresponding beliefs. These belief systems are updated with Bayes' rule based on the actions of opponents and the outcomes of aggressive encounters. We propose that, by representing Bayesian beliefs, animals in social conflict situations address the challenge of partial observability. Our model thus holds the potential for generalization to other social interactions characterized by uncertainty.
The proposed model not only provides insights into the development of pathological aggression and defeat but also offers a path toward their mitigation. The next steps will involve applying the model to newly available data and identifying the neural correlates of aggression-related variables within the brain. This will be followed by the exploration of connectivity changes between the identified brain regions, which may unveil the neural circuit governing decision-making in aggression. Ultimately, the shift from a normative to a circuit model of conflict may enable the design of policies geared toward reducing aggression in a variety of social settings.
Behavioral biology of aggression. Social conflict has been studied in humans, non-human primates, and mice – a model organism whose behavioral states can be manipulated. Mouse studies have investigated how social conflict leads to the formation, maintenance, and plasticity of aggressive and subordinate behavioral states in animals (Wong et al., 2016; Hashikawa et al., 2017). Such behavioral states can be modeled in the chronic social conflict paradigm where mice are allowed limited interactions over extended periods of time (Kudryavtseva, 2000; Miczek et al., 2001). Here, we use the chronic social conflict paradigm to induce various behavioral states in mice and monitor their conflict-related behaviors. As the previous studies of social conflict have shown the evolutionary preservation of its basic mechanisms (Wang and Anderson, 2010; Watanabe et al., 2017), we expect our results to be relevant to human behavior.
Game theory. Optimal behaviors of interacting agents are conventionally described in terms of game theory. Game theory considers rational agents developing their strategies to maximize rewards. The rewards received by the agents depend on their actions and the actions of their opponents. The acquisition of optimal strategies in games can be described by probabilities of available actions (Smith, 1982; Cressman et al., 2003). Such strategies of agents co-evolve to reinforce higher-reward actions until the rewards can't grow any further (Nash equilibrium). Game-theoretic approaches have been used in models of human and animal behaviors in multi-agent settings including agonistic interactions (Smith, 1974; Hofbauer et al., 1998; Wilson, 2000; Lorenz, 2005). Here, we use game theory to model agonistic interactions in the condition of chronic social conflict in mice.
Beliefs and Theories of Mind. Humans and animals in social settings have limited access to environmental variables, which can be modeled with the partially observable Markov decision process (POMDP) framework. To gain evidence about hidden variables, real-world agents may maintain probabilistic internal models of environment – the “beliefs” – based on which their actions can be considered rational, maximizing a reward function (Fahlman et al., 1983; Alefantis et al., 2021). The agents’ rewards and beliefs can be inferred from their behavior using inverse control techniques that maximize the likelihood of the observed behavior based on a hidden dynamics model (Russell, 1998; Choi and Kim, 2011; Dvijotham and Todorov, 2010; Kwon et al., 2020). In biologically relevant multi-agent settings, beliefs are studied in the Theories of Mind (ToM) framework, proposing that humans and animals maintain beliefs about the beliefs of their adversaries (Baker et al., 2011) or aides (Khalvati et al., 2019). As previous studies were successful in inferring beliefs (Schmitt et al., 2017; Alefantis et al., 2021) and regressing them to neural activity in simulations (Wu et al., 2020) or low-resolution fMRI (Koster-Hale and Saxe, 2013), here we introduce a framework for the inference of beliefs in social setting and regress them to high-resolution whole-brain neural activity in mice.
C-Fos as a whole-brain marker of neural activity. The search for the brain regions accumulating evidence about the environment requires large-scale neural activity data. Such data can be obtained by monitoring the levels of c-Fos, an immediate early gene whose activation reflects neuronal activity (Sagar et al., 1988; Herrera and Robertson, 1996). C-Fos data lacks temporal resolution, yet it allows observing whole-brain activity at a high spatial resolution without using equipment that may affect animals' choices. Local expression of c-Fos has implicated several brain regions in agonistic
interactions (Hashikawa et al., 2017; Aleyasin et al., 2018; Diaz and Lin, 2020; Wei et al., 2021). Here, we use 3D light-sheet microscopy of the c-Fos signal in whole-brain samples (Renier et al., 2016) to identify the brain-wide neural activity in animals with varying exposure to social conflict. We compare our c-Fos data to the beliefs identified based on behavior in individual mice and report the regions that may be involved in the computation of conflict-related variables in the brain.
3 Methods: a normative POMDP framework for social behavior modeling
The goal of this work is to build a quantitative theory for the formation of social conflict-related behavioral states in mice. In Section 3.1 we introduce a mouse behavioral paradigm where we recorded the actions leading to different behavioral states and the brain activities corresponding to these states. In Sections 3.2 to 3.4 we define a normative POMDP-based framework for studying real-world social behaviors. In this framework, the game-theory optimal actions of agents rely on their beliefs, defined as distributional estimates of the hidden variables of the environment. In Section 4 we examine hypotheses about the reward schedule, information availability, and evidence accumulation related to social conflict. We compare our results to behavioral data in Section 4.1 and to neural data in Section 4.2. We discuss our findings in Appendix A. A detailed description of the methods is provided in Appendix B. Derivations and supplementary results are provided in Appendix C.
Figure 1: The chronic social conflict paradigm. (A-B) Stages of a single sensory contact event. (C) Stages of a multi-day experiment. (D) Recorded actions of mice in the experiment (white: no encounter scheduled).
3.1 Running example: the chronic social conflict paradigm
Our goal was to come up with a unified behavioral model explaining various stages of social conflict. As previous experiments have mostly focused on isolated aspects of social conflict (e.g. its pathological form only), the data needed for building a unified model was not readily available, necessitating the conduct of a new experiment. To this end, we implemented the chronic social conflict paradigm (Kudryavtseva, 2000; Kudryavtseva et al., 2014), a well-established experimental paradigm for studying aggression and conflict. Within the paradigm, we focused on conventionally studied stages of social conflict, i.e. the adaptive (3 days), intermediate (10 days), and pathological (20 days) aggression/defeat (Kudryavtseva et al., 2014). For completeness, we additionally considered the reversal of pathological behaviors (after 20 days) – the standard experimental perturbation of behavior. We used the conventional experimental parameters (Kudryavtseva et al., 2014).
We conducted our experiments as follows. Pairs of weight-matched (as a proxy for being strength-matched) mice were placed in cages separated by a perforated partition (Figure 1A). Once daily, the partition was removed for 10 minutes to enable agonistic interactions between mice (Figure 1B). After 3 days, mice were classified into "winning", "losing", and "neutral" (non-fighting) (Figure 1C,D). The participation of "neutral" mice in the experiment was discontinued. Afterward, each winning mouse remained in its cage, while each losing mouse was daily relocated to an unfamiliar cage with an unfamiliar winning mouse (Figure 1C). Regardless of no longer being weight-matched, mice have retained their winning or losing behavioral states, thus transitioning to the pathological regime of social conflict-related decision-making (Figure 1D). After 20 days of interactions, mice were exposed to opponents of equal behavioral state (Figure 1C). The newly formed pairs underwent two more days of agonistic interactions throughout which new win/lose relationships were established (Figure 1D). We provide more details in Appendix B.1.
3.2 Game-theory optimal actions
We build a normative POMDP model for social behaviors using the chronic social conflict paradigm. In this section, we start with an approximation where each agent has all information about itself and its opponents. In that case, we use game theory to define the optimal actions for each agent.
We formalize our behavioral paradigm as a normal form game where, on each iteration, two agents have to decide simultaneously on what action  $a$  to take. We defined the possible actions  $a$  as "attack" or "defend". Depending on the actions  $a_1$  and  $a_2$  selected by the two agents respectively, they received rewards  $r$  defined as follows. If both agents chose to "defend", no fight happened, leading to a zero reward  $r_1 = r_2 = 0$  assigned to each agent. If both agents "attacked", the outcome of the game was defined by their strengths  $s$ , an additional parameter assigned to each agent in the model. The outcome probability  $p^{win}$  was defined by the softmax rule over the strengths parameterized with the "outcome confidence"  $\beta_0$ :
$$
p _ {i} ^ {\text {w i n}} = Z ^ {- 1} \exp \left(\beta_ {o} s _ {i}\right). \tag {1}
$$
Here and below  $Z$  denotes the normalization coefficient. Once the outcome was determined, the winning agent received a reward of  $r = +1$ , and the losing agent expended a cost of  $r = -\mathcal{A}$ . The reward expectation was equal to:
$$
\kappa_ {i} = 1 \cdot p _ {i} ^ {w i n} + (- \mathcal {A}) \cdot (1 - p _ {i} ^ {w i n}) = (1 + \mathcal {A}) p _ {i} ^ {w i n} - \mathcal {A}. \tag {2}
$$
The cost of loss was reduced if one of the agents chose to "defend" while the other agent "attacked". In that case, the "attacking" agent always won and received the reward of  $r = +1$  while the losing agent expended the cost of  $r = -\alpha$  such that  $\alpha < \mathcal{A}$ . The reward expectations were described in the payoff matrix  $\hat{R}_i$  whose rows correspond to the actions of the agent ("attack" and "defend") and columns correspond to the actions of its opponent:
$$
\hat {R} _ {i} = \left( \begin{array}{c c} \kappa_ {i} & 1 \\ - \alpha & 0 \end{array} \right). \tag {3}
$$
To determine the optimal strategies in this game, we used game theory. In this approach, the goal of every participant was to maximize its expected reward  $\mathbb{E}[r_i]$ :
$$
\mathbb {E} \left[ r _ {1} \right] = P _ {1} ^ {T} \hat {R} _ {1} P _ {2}. \tag {4}
$$
Here the vectors  $P_{i}$  define the probabilities to "attack"  $p_i$  and to "defend"  $1 - p_{i}$  for an agent  $i$ :
$$
P _ {i} \equiv \left( \begin{array}{c} p _ {i} \\ 1 - p _ {i} \end{array} \right). \tag {5}
$$
A similar expression can be written for the expected reward of the second agent  $\mathbb{E}[r_2]$ . To maximize the rewards  $\mathbb{E}[r_i]$ , we computed their gradients with respect to the probabilities to "attack"  $p_i$  (an agent could only update its own policy, but not that of the opponent). We used these gradients to update the policies leading to joint maximization of the expected rewards (Figure 2A):
$$
\left\{ \begin{array}{l} \frac {\partial \mathbb {E} [ r _ {1} ]}{\partial p _ {1}} = (\kappa_ {1} + \alpha - 1) p _ {2} + 1; \\ \frac {\partial \mathbb {E} [ r _ {2} ]}{\partial p _ {2}} = (\kappa_ {2} + \alpha - 1) p _ {1} + 1. \end{array} \right. \tag {6}
$$
We provide the derivation for the equation above in Appendix C.1.1. The probabilities of actions can be only defined in the range  $0 \leq \{p_1, p_2\} \leq 1$ . Their evolution  $\dot{p}_i \propto \partial \mathbb{E}[r_i] / \partial p_i$ , governed by the equations above (red arrows in Figure 2A), converges to zeros or ones (blue arrows in Figure 2A) forming pure strategies (we show the lack of mixed strategies in Appendix C.1.2). To determine optimal pure strategies, we chose the strategies whose reward gradients (red arrows in Figure 2B) pointed outwards the  $[0 - 1]$  interval for both agents. We represented the optimal policies via the tensor  $A$  of probabilities for each possible action  $a$  depending on the agent's strength  $s_1$  and their opponent's strength  $s_2$ , averaging over all optimal pure strategies (Figure 2C):
$$
A _ {a i j} = \Pr \left(a _ {1} = a \mid s _ {1} = i, s _ {2} = j\right). \tag {7}
$$
The resulting optimal policies in the model depended on the relative strengths of the agents (Figure 2C). We parameterized the optimal policies with the maximum relative strength of the agents  $\delta = s_2 - s_1$  at which it was still optimal to "attack". We then explored how  $\delta$  depends on the task parameters  $\alpha$  and  $\beta_o$  and found that most of these parameters' values correspond to the optimal strategy with  $\delta = 0$  ("x" in Figure 2D), i.e. to "attacking" any opponent who is weaker or equal. We describe additional details regarding the acquisition of optimal policies in Appendix B.2 and summarize the procedure in Supplementary Algorithm 1.
Figure 2: Game-theoretic model of the chronic social conflict paradigm. (A) Policy gradients. (B) Gradient orientations yield a pure strategy as a Nash equilibrium. (C) Optimal policy and its dependence on the relative strength. (D) Dependence of optimal policy on the model parameters.
3.3 Partial observability: beliefs about hidden variables
Game theory in our setting predicts a static policy, unchanging over time. Conversely, weight-matched mice in the experiment who initially attacked each other later split into always-attacking "winners" and ever-defending "losers" (Figure 1D). To account for this dynamic, we expand our model to a scenario where agents do not possess the full information about their strengths but accumulate this information over the task. In this section, we model the agents' information about strengths via beliefs – the probability distributions for an agent to belong to a certain strength category.
We used body weights  $w$  of the animals as a proxy of their strengths  $s$  (Andersson and Iwasa, 1996; Cooper et al., 2020):
$$
w _ {i} \propto s _ {i}. \tag {8}
$$
We modeled the animals' initial estimates of their body weights  $\tilde{w}$  as normally distributed around true values  $w$ . Under this assumption, we reconstructed the probability distributions for animals' strengths (their initial beliefs)  $B^{i} \equiv Pr(s = i|\tilde{w})$  using their body weights  $\tilde{w}$  in Bayes' rule:
$$
B ^ {i} \equiv P r (s = i | \tilde {w}) = \frac {P _ {\mathcal {N} (i , \sigma)} (\tilde {w}) P r (s = i)}{P r (\tilde {w})}, \tag {9}
$$
where  $P_{\mathcal{N}(i,\sigma)}(\tilde{w})$  is a normal distribution probability density function representing the noise in the estimate of the animal's body weight;  $Pr(\tilde{w})$  is the distribution of the animals' estimated body weights  $\{\tilde{w}_i\}$ , and  $Pr(s)$  is the distribution of their strengths  $\{s_i\}$ . To denoise the strength distribution  $Pr(s)$ , we approximated the experimentally observed body weight distribution  $Pr(w) \propto Pr(s)$  with a normal distribution.
We consider up to four types of beliefs. The two "primary" beliefs describe the animals' estimates of their own strength and of the strength of the opponent. The two "secondary" beliefs estimate the "primary" beliefs of the opponent. The animals were expected to better estimate their own strength compared to that of their opponents. To this end, we used separate uncertainty parameters for estimating one's own strength  $(\sigma = \sigma_{1})$  and that of an opponent  $(\sigma = \sigma_{2})$ . For the "secondary" beliefs, the uncertainties were combined  $(\sigma = \sqrt{\sigma_{1}^{2} + \sigma_{2}^{2}})$ . We provide the details regarding the initialization of beliefs in Appendix B.3 and summarize them in Supplementary Algorithm 2.
3.4 Evidence accumulation: Bayesian update of beliefs
In this section, we expand our model with an update mechanism for the agents' beliefs. We update the beliefs  $B$  using Bayes' rule and the information about the actions  $a$  and outcomes  $o$  of agonistic interactions (Figure 3A).
We define the outcome tensor  $O_{ij}^{oab}$  as a tensor describing the probabilities of each possible outcome  $o_1 = o$  for an agent of a strength  $s_1 = i$  after choosing an action  $a_1 = a$  provided that the opponent has a strength  $s_2 = j$  and chose an action  $a_2 = b$ :
$$
O _ {i j} ^ {o a b} = \Pr \left(o _ {1} = o \mid a _ {1} = a, a _ {2} = b, s _ {1} = i, s _ {2} = j\right). \tag {10}
$$
The reward tensor  $R_{ij}^{ab}$  describes the expected reward  $\mathbb{E}[r_1]$  for an agent of a strength  $s_1 = i$  after choosing an action  $a_1 = a$  provided that the opponent has a strength  $s_2 = j$  and chose an action
$a_2 = b$ . The reward expectation was based on probabilities of possible outcomes:
$$
R _ {i j} ^ {a b} = \sum_ {o} r (o, a, b) O _ {i j} ^ {o a b}. \tag {11}
$$
The outcome-action and reward-action tensors predicted the probability of an outcome  $o_1$  and expectation of a reward  $r_1$  assuming that the opponent's actions  $a_2$  are game-theory-optimal:
$$
\left[ R A \right] _ {i j k l} ^ {a} \equiv \sum_ {b} R _ {i j} ^ {a b} \operatorname {c o n v} _ {k l} \left(A, P _ {\mathcal {N} \left(0, \sqrt {\sigma_ {1} ^ {2} + \sigma_ {2} ^ {2}}\right)}\right) _ {k l} ^ {b}; \tag {12}
$$
$$
[ O A ] _ {i j k l} ^ {o a b} \equiv O _ {i j} ^ {o a b} \operatorname {c o n v} _ {k l} \left(A, P _ {\mathcal {N} \left(0, \sqrt {\sigma_ {1} ^ {2} + \sigma_ {2} ^ {2}}\right)}\right) _ {k l} ^ {b}. \tag {13}
$$
Here the opponent's action is estimated based on the "secondary" beliefs and the estimated outcome is based on the "primary" beliefs. The convolution reflects that the opponent's action is based on the distributional belief rather than on a point estimate of the strengths. The standard deviation  $\sqrt{\sigma_1^2 + \sigma_2^2}$  applies to the estimated opponent's estimates of both its own strength and "our" strength.
Figure 3: Bayesian update of beliefs in the model. (A) Belief update diagram for two mice. (B,C) Reconstructed belief dynamics for representative loser and winner mice.
To decide on an action  $a_1$ , an agent maximized its reward  $r_1$ . We computed this reward by summing the reward-action tensor  $[RA]_{ijkl}^a$  multiplied by the belief tensors  $B_{11}^i, B_{12}^j, B_{21}^k, B_{22}^l$  reflecting the probability distributions for strengths:
$$
a _ {1} = \operatorname {s o f t m a x} _ {a} \sum_ {i j k l} [ R A ] _ {i j k l} ^ {a} B _ {1 1} ^ {i} B _ {1 2} ^ {j} B _ {2 1} ^ {k} B _ {2 2} ^ {l}. \tag {14}
$$
Here the belief tensors  $B_{11}$  and  $B_{12}$  describe the "primary" beliefs of an agent about its own strength and the strength of its opponent respectively. Likewise, the belief tensors  $B_{21}$  and  $B_{22}$  describe the "secondary" beliefs, i.e. the agent's estimate of its opponent's "primary" beliefs. The indices  $i,j,k,l$  iterate over all possible strengths, e.g.  $B_{11}^{5} = 0.2$  would mean that, according to the agent's belief, the probability of its own strength to be equal to 5 constitutes  $20\%$ . To update the agents' beliefs using the observed actions and outcomes  $\{o_1,o_2\}$ , we used Bayes' rule individually for every element of each belief tensor  $B_{**}^{i}$ :
$$
P r \left(s _ {1} = i \mid \left\{o _ {1}, a _ {2} \right\}\right) = \frac {P r \left(\left\{o _ {1} , a _ {2} \right\} \mid s _ {1} = i\right) P r \left(s _ {1} = i\right)}{P r \left(\left\{o _ {1} , a _ {2} \right\}\right)}. \tag {15}
$$