SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our results are based on data comprising "attack"/"defend" and "win"/"lose" readouts but our analyses can be extended to different metrics and social behaviors. Long-term tracking of social structure in rats (Nagy et al., 2023) may allow for testing our conclusions in a different species in a complex social environment. The data on carrier/non-carrier phenotypes (Schroeder and Desor, 2005) would allow us to see if our results generalize to a cooperative setting. The model may also be applied to other domains, such as automated negotiation (Baarslag et al., 2016). Overall, our framework may be used for studying competitive or cooperative behaviors in real-world settings using limited data.
B Methods
B.1 Mouse chronic social conflict paradigm
To induce varied behavioral states in mice, we applied the chronic social conflict paradigm. Pairs of weight-matched mice were separated by a perforated partition in cages. Once daily, the partition was removed for 10 minutes to enable agonistic interactions between mice. To study the adaptive properties of the aggressive/defeated states in mice, we kept the opponents unchanged for 3 days. Then, to reveal the pathological properties of behavioral states, each winning mouse was kept in its cage, while each losing mouse was daily relocated to an unfamiliar cage with an unfamiliar winner. Non-fighting mice were discontinued from participating in the experiment. After a total of 20 days of interactions, to reveal the flexibility of behavioral states, winning and losing mice were reorganized to face opponents of the same state. The newly formed pairs underwent 2 more days of interactions. During all interactions, we logged the mouse actions ("attack", "defend") and the outcomes of interactions ("win", "lose", "draw"). We varied the participation of mice in the experiment:
Table 1: Datasets
DATASET	I	II	III	TOTAL
DAYS	3 / 10	20	22
MOUSE BEHAVIOR	EXP: 56; CTL: 6	EXP: 58; CTL: 6	EXP: 88; CTL: 6	EXP: 202; CTL: 18
BRAIN 3D DATA	EXP: 24; CTL: 6	EXP: 11; CTL: 6	EXP: 19; CTL: 6	EXP: 54; CTL: 18
B.2 Game theory optimal actions
To define a class of models to test against our data, we described the mouse task as a normal-form game. Here, the agents simultaneously chose their actions; the available actions were to "attack" and to "defend". The outcome of the game depended on the actions and strengths of participating agents. The strengths  $s_i$  were defined as constants between 1 - 20, specific to an agent and unchanging.
Table 2: Game outcomes
ACTION	#2 “ATTACKS”	#2 “DEFENDS”
#1 “ATTACKS”	p1win ∼ Gibbs(s1, s2)	#1 WINS
#1 “DEFENDS”	#2 WINS	DRAW
If both agents in the game chose to "attack", the probability of "winning" the game was defined by the softmax rule over their strengths parameterized with the "outcome confidence"  $\beta_{o}$ . The reward assigned to each agent depended on the outcome of the game and on their action as follows:
Table 3: Rewards
ACTION	“WIN”	“LOSE”	“DRAW”
“ATTACK”	1	-A	N/A
“DEFEND”	N/A	-α	0
To derive the optimal actions for agents in a fully observable setting, we parametrized the agents' actions by the probability to "attack" and derived the gradients of the expected reward w.r.t. these probabilities. We found pure-strategy Nash equilibria by choosing the pure strategies whose gradients were directed outside the support of action probabilities. We averaged the action probabilities over Nash equilibria. Finding the optimal policies is summarized in Algorithm 1:
Algorithm 1 Game theory optimal actions
Input: strength range  $s_{max}$  costs  $\alpha$  A, confidence  $\beta_{o}$  Initialize actions  $A = \mathrm{zeros}(2,s_{max},s_{max})$    
for strengths  $s_1,s_2 = 1$  to  $s_{max}$  do expectation  $\kappa_{1} = (1 + \mathcal{A})$  softmaxs  $(\beta_o s_1,\beta_o s_2) - \mathcal{A}$  gradient  $\dot{p}_1 = (\kappa_1 + \alpha -1)p_2 + 1$  ; similarly find  $\dot{p}_2$  for policies  $p_1,p_2 = 0$  or 1 do Find pure strategy:  $\dot{p}_{1(2)}(p_{1(2)} - 0.5) > 0$  end for Average the strategies:  $A(1,s_1,s_2) = \mathrm{mean}(\{p_1\})$ $A(2,s_1,s_2) = 1 - A(1,s_1,s_2)$    
end for
B.3 Bayesian belief initialization
To account for the partial observability of information in the task, we defined the probability distributions (beliefs) describing the strengths of the agents. We initialized the beliefs with the normal distributions whose mean values corresponded to the body weight of the animals (shifted by  $-15\mathrm{g}$  to fit the range). To account for the prior information about the body weight distribution, we applied
Bayes' rule to the initial beliefs. To reflect possible depths of reasoning, we considered four types of beliefs parameterized by different standard deviations of the normal distribution:
Table 4: Initial belief standard deviations
BELIEF	“MINE”	“OPPONENT’S”
“ABOUT MYSELF”	σ1	√σ12+σ22
“ABOUT OPPONENT”	σ2	√σ22+σ12
The "opponent's" beliefs here reflect the agent's beliefs about the opponent's beliefs that may differ from the opponent's beliefs in the model. Overall, the acquisition of initial beliefs about the animals' strengths is summarized in Algorithm 2. Here  $B_{m_1m_2}^i$  is the belief of mouse  $m_1$  about mouse  $m_2$ .
Algorithm 2 Bayesian belief initialization
Input: body weights  $w_{i}$  ,uncertainties  $\sigma_1,\sigma_2$    
Initialize beliefs  $B = \mathrm{zeros}(s_{max},\mathrm{max}(m),\mathrm{max}(m))$    
strengths  $Pr(s) = P_{\mathcal{N}(\mathbb{E}[w],\mathbb{D}[w])}$    
for mice  $m_{1},m_{2} = 1$  to max  $\{\{m\}\}$  do Set  $\sigma$  using Table 4 for strength  $i = 1$  to max  $\{\{s\}\}$  do belief  $B_{m_1m_2}^i = Z^{-1}P_{\mathcal{N}(i,\sigma)}(w_{m_2})Pr(s)(i)$  end for   
end for
B.4 Bayesian belief update
To enable evidence accumulation in our model, we used Bayes' rule to update the beliefs based on the observations. The agonistic interaction outcomes were used to update the primary beliefs about the agent's own and the opponent's strengths. The opponents' actions were used to update the beliefs about the opponent's beliefs. The update procedure for beliefs is summarized in Algorithm 3 below.
Algorithm 3 Bayesian belief update
Input: pairs  $p_m^t$ , actions  $a_m^t$ , outcomes  $o_m^t$   
Compute actions  $A$  using Algorithm 1; precompute  $[RA]_{ijkl}^a \equiv \sum_b R_{ij}^{ab} \mathrm{conv}_{kl}(A, P_{\mathcal{N}(0, \sqrt{\sigma_1^2 + \sigma_2^2})})_{kl}^b$ $[OA]_{ijkl}^{aab} \equiv O_{ij}^{aab} \mathrm{conv}_{kl}(A, P_{\mathcal{N}(0, \sqrt{\sigma_1^2 + \sigma_2^2})})_{kl}^b$   
Initialize beliefs  $B$  using Algorithm 2  
for time  $t = 1$  to max(\{t\}) do  
for mouse  $m_1 = 1$  to max(\{m\}) do  
opponent  $m_2 = p_{m_1}^t$   
for mouse in  $m_1, m_2$  do  
 $\tilde{a}_k^t = \text{softmax}_a [RA]_{ijkl}^a B_{11}^i B_{12}^j B_{21}^k B_{22}^l$   
end for  
for strength  $i = 1$  to max(\{s\}) do  
for mice in  $m_1, m_2$  do  
 $\Delta B_{12}^i = Z^{-1} \sum_{jkl} [OA]_{ijkl}^{o_1 a_1 a_2} B_{11}^i B_{12}^j B_{21}^k B_{22}^l$   
update  $B_{12}^i \gets \varepsilon \Delta B_{12}^i + (1 - \varepsilon) B_{12}^i$   
Repeat for other belief types  $B_{**}$   
end for  
end for  
end for  
end for
B.5 Model fit
Besides the belief update, the above algorithm allows for predicting the animals' actions based on their beliefs. We consider the opponent's predicted action to be game-theory-optimal based on the opponent's estimated beliefs. To predict the agent's action, we maximize its reward expectations based on its own "primary" beliefs and the opponent's predicted action. To optimize the model predictions, we update the model's parameters to minimize the negative log-likelihood (NLL) for the actions in our behavior data to be sampled from the model predictions. While we use the entire data to propagate the beliefs, we only use days 1-3 and 21-22 for the NLL evaluation to avoid the impact of repeated actions on days 4-20. We use bootstrap to compute the error quantiles for the identified parameters. Overall, the inference of model parameters is summarized in Algorithm 4 below.
Algorithm 4 Model parameter inference
Input: pairs  $p_m^t$ , actions  $a_m^t$ , outcomes  $o_m^t$   
Initialize parameters  $x \equiv [\sigma_1, \sigma_2, \beta_a, \beta_o, \alpha, \mathcal{A}, \varepsilon]$   
Initialize  $x_{opt} = \text{zeros}(100, 7)$
for repeat  $i = 1$  to 100 do Define subset  $\tau \in [1,T]$  with repetitions Define actions  $Pr(a_{m}^{\tau})$  using Algorithm 3 Define likelihood  $\mathcal{L} = -\sum_{mt}\log Pr(a_m^\tau)$  Regularize  $\mathcal{L}\gets \mathcal{L} + \lambda \| x / x_{\mathrm{max}}\| _2$  Optimize  $x_{opt}(i) = fminsearch(\mathcal{L})$    
end for
Compute  $\mathbb{E}[x_{opt}],\mathbb{D}[x_{opt}]$
To optimize the parameter inference, we applied it to simulated data with known parameters. We added noise to Algorithm 2 to generate the initial beliefs about strengths in the model and used Algorithm 3 to simulate the actions. Once the model parameters were reconstructed with Algorithm 4, we used Gaussian processes to estimate the means and the error quantiles for the predictions. We varied the regularizer  $\lambda \in \{0;0.1;1;10\}$  and chose the one that led to the best reconstruction  $(\lambda = 1)$ .
B.6 Model comparison
To test hypotheses about mouse decision-making, we performed the model comparison. To this end, we optimized pairs of models on the training data (88 mice participating for 22 days) and computed the changes in the NLL based on the predictions for the testing data (104 mice participating for 3/10/20 days). We performed the t-tests on the changes in the NLLs for individual mice followed by the FDR correction. For pairwise comparisons, we picked an initial model and considered one-step deviations from it (e.g. for the 0-ToM model, we separately considered "frozen" prior beliefs and, alternatively, shuffled body weights, but not their combination). We fitted each model in the same way, using a numerical optimizer until convergence. We considered new models until one of the models outperformed the initial one; we then used this new model as an initial model for the next round of comparisons using one-step deviations from this new model. We stopped after arriving at a model that was not outperformed by any of the other models. Here we list the considered models and specify their differences in comparison with the "default" (1-ToM) model described above:
Table 5: Baseline models
TEST	BASELINE MODEL
DEPTH OF REASONING	0-THEORY OF MIND
AVAILABILITY OF INFORMATION	SHARED BELIEFS
FLEXIBILITY OF POLICY	FLEXIBLE IN-FIGHT POLICY
CLASS OF ALGORITHMS	RESCORLA-WAGNER
In Rescorla-Wagner model, each action ("attack", "defend") was associated with a value representing the expected future reward. This value was independent of the mouse's strength or the identity of an opponent. The values for the "attack" and "defend" actions were initialized using the expectations
over the distribution of game theory optimal opponents, then updated at a learning rate  $\varepsilon$  using the reward as a teaching signal. In the 0-Theory of Mind model, we only considered the "primary" beliefs of an agent (about its own and the opponent's strength) assuming that the opponent acts in accordance with the same beliefs (even though the actual beliefs of the two agents were independent and may have differed). In the shared beliefs model, the 0-ToM beliefs were identical between the two opponents. In the flexible in-fight policy model, the policy gradients used in determining the Nash equilibria were computed for the expected rewards integrated with the "primary" beliefs of each mouse reflecting the in-fight adaptation to the opponent's actions. As the opponent's actions were observable in this setting, the model did not include "secondary" beliefs.
Table 6: Ablation studies
TEST	ABLATION
DYNAMICS OF BELIEFS	FIXED PRIOR BELIEFS
FIXED POSTERIOR BELIEFS
ROLE OF BODY WEIGHT	SHUFFLED BODY WEIGHTS
WIDE BODY WEIGHT PRIOR
In the fixed prior beliefs model we used the zero learning rate for the belief update, retaining the beliefs unchanged from their body weight-based initial values. In the fixed posterior beliefs model, we used the zero learning rate but the initial beliefs were substituted with the final beliefs produced with our default model. In the shuffled body weights model, we shuffled the animals' body weights before using them to initialize the beliefs. In the wide body weight prior model we used the double standard deviation for the prior distribution of the animals' body weights at the belief initialization.
B.7 Neural correlates of model variables
To find neural correlates of the model variables, we normalized the registered c-Fos activity in each brain sample by dividing it by the average c-Fos activity of all control samples from the same dataset. We converted the relative c-Fos activities to the log scale to obtain a roughly normal distribution of relative activities. We normalized the model variables by their average estimated values for the control samples. Then, we computed the voxel-wise correlations between the processed c-Fos activity and model variables. To select the voxels significantly correlated with model variables, we performed the FDR correction. Finding the belief correlates in the brain is described in Algorithm 5 below.
Algorithm 5 Belief correlates in the brain
Input: 3D brain images  $I$  aligned to atlas  $A$ ; belief  
for sample  $s$  in experimental samples do  
Normalize  $I_s = \log I_s - \log (\mathrm{mean}(I_{ctrl}))$   
Normalize  $b_{s} = \mathrm{mean}(b_{s}) - \mathrm{mean}(b_{ctrl})$   
end for  
for belief  $b$  in  $\{B_{11}, B_{12}$ , etc.} do  
for voxel  $i$  in 3D brain atlas volume do  
Compute  $p_i \gets \mathrm{corr}(\{I\}_i, b)$   
end for  
Compute  $\{q_i\} = \mathrm{FDR}(\cup_i\{p_i\})$   
Find  $corr_b = \{j \leq i\} : \{\sum_{k=0}^i \mathrm{sort}(q_k) < 0.1\}$   
Find brain regions  $r$  in atlas  $A$ :  $corr_b \in r$   
end for
Considered variables included: "primary" and "secondary" beliefs about the mouse's own and the opponent's strength for 0-ToM and 1-ToM models; the estimated advantages (i.e. the differences between the same-order beliefs within a model); the estimated outcomes (i.e. the binarized estimated advantages); the predicted actions for oneself and the opponent; the Rescorla-Wagner values of the two actions and their difference. The control variables included: the body weights of mice, the observed actions, the outcomes of the agonistic interactions, and the shuffled beliefs of each type.
B.8 Software and time budget
All computations were performed with Matlab R2022b. The mouse data fits were performed with a 3 GHz Intel Core i7-based Dell laptop computer. The simulated data fits were performed with a 3 GHz Intel Xeon-based Supermicro computer server in 40 parallel threads. 0-ToM models converged within 2 minutes (per model per thread); 1-ToM models converged within 3 hours (per model per thread).
C Supplementary results
C.1 Game theory optimal actions
C.1.1 Pure strategies
In this section, we provide a simple derivation for the game theory optimal actions in our model. The expected reward  $\mathbb{E}[r_1]$  for the first agent can be computed as its payoff matrix  $\hat{R}_1$  multiplied by the action probability vectors  $P_{1,2}$  describing the policies of the first and the second agents:
$$
\mathbb {E} \left[ r _ {1} \right] = P _ {1} ^ {T} \hat {R} _ {1} P _ {2} = \left(p _ {1} \quad 1 - p _ {1}\right) \left( \begin{array}{l l} \kappa_ {1} & 1 \\ - \alpha & 0 \end{array} \right) \left( \begin{array}{l} p _ {2} \\ 1 - p _ {2} \end{array} \right) = p _ {1} p _ {2} (\kappa_ {1} + \alpha - 1) + p _ {1} - \alpha p _ {2}. \tag {22}
$$
Here  $p_{1,2}$  are the probabilities to "attack" for the first and the second agents,  $\kappa_{1}$  is the expected reward for the first agent in the case when both agents "attack", and  $-\alpha$  is the cost for the first agent in the case when only the second agent "attacks". The first agent may increase its expected reward by changing its policy  $P_{1}$  that depends on  $p_{1}$  (the first agent can't change  $p_{2}$  as  $p_{2}$  is controlled by the opponent). To optimize the expected reward  $\mathbb{E}[r_1]$ , we compute its gradient w.r.t.  $p_{1}$ :
$$
\frac {\partial \mathbb {E} [ r _ {1} ]}{\partial p _ {1}} = (\kappa_ {1} + \alpha - 1) p _ {2} + 1. \tag {23}
$$
The agents may follow a pure strategy with binary probabilities  $p_{1,2}$  to "attack". Such strategies correspond to Nash equilibria if the reward gradients point outwards the  $[0 - 1]$  interval for both agents so that the policy won't have room to evolve. This requirement can be formalized as:
$$
\left\{ \begin{array}{l} \frac {\partial \mathbb {E} [ r _ {1} ]}{\partial p _ {1}} \left(p _ {1} - \frac {1}{2}\right) > 0; \\ \frac {\partial \mathbb {E} [ r _ {2} ]}{\partial p _ {2}} \left(p _ {2} - \frac {1}{2}\right) > 0. \end{array} \right. \tag {24}
$$
Using the values of the derivatives from Equation 23, we arrive at the criterion for Nash equilibria:
$$
\left\{ \begin{array}{l} \left(p _ {2} ^ {\text {p u r e}} \left(\kappa_ {1} + \alpha - 1\right) + 1\right) \left(p _ {1} ^ {\text {p u r e}} - \frac {1}{2}\right) > 0; \\ \text {(p u r e)} (1) = 1) (\text {p u r e}) \end{array} \right. \tag {25}
$$
$$
\left\{\left(p _ {1} ^ {p u r e} \left(\kappa_ {2} + \alpha - 1\right) + 1\right) \left(p _ {2} ^ {p u r e} - \frac {1}{2}\right) > 0. \right.
$$
An arbitrary task setting (defined by the parameters  $\kappa$  and  $\alpha$ ) may correspond to multiple Nash equilibria. To this end, we approximate the expected policy by averaging the action probabilities  $p_{1,2}$  over the identified equilibria. A better way to account for multiple Nash equilibria is to integrate the action probabilities over their basins of attraction; this analysis can be added in future studies.
C.1.2 Lack of mixed strategies
To consider mixed strategies with nonbinary probabilities to take either of the available actions, assume that both agents in the game follow the gradients of their respective rewards to update their action probabilities. Following Equation 23 for these gradients, we arrive at the dynamical system:
$$
\left\{ \begin{array}{l} \frac {d p _ {1}}{d t} = p _ {2} \left(\kappa_ {1} + \alpha - 1\right) + 1; \\ \frac {d p _ {2}}{d t} = p _ {1} \left(\kappa_ {2} + \alpha - 1\right) + 1. \end{array} \right. \tag {26}
$$
Any mixed strategy  $\{p_1^{mixed}, p_2^{mixed}\}$  corresponds to an attractor of this dynamical system, i.e. a point  $\{p_1, p_2\}$  where the time derivatives of both variables  $p_1, p_2$  are equal to zero and the phase portrait of the system is shaped as a stable node or a stable spiral so that the agents' action probabilities could converge to this point. To analyze the dynamical system around the point of zero gradient, we linearize its dynamics via computing the Jacobian:
$$
\frac {d}{d t} \left( \begin{array}{c} p _ {1} ^ {*} \\ p _ {2} ^ {*} \end{array} \right) = \left( \begin{array}{c c} 0 & \kappa_ {1} + \alpha - 1 \\ \kappa_ {2} + \alpha - 1 & 0 \end{array} \right) \left( \begin{array}{c} p _ {1} ^ {*} \\ p _ {2} ^ {*} \end{array} \right). \tag {27}
$$
To describe the phase portrait of this system, we compute the eigenvalues of the Jacobian:
$$
\lambda_ {1, 2} = \pm \sqrt {\left(\kappa_ {1} + \alpha - 1\right) \left(\kappa_ {2} + \alpha - 1\right)}. \tag {28}
$$
In the case of real eigenvalues  $\lambda_{1,2}$ , their product is negative implying that the system's dynamics form a saddle point. In the case of complex eigenvalues, the zero trace of the Jacobian implies that the system's dynamics form a limit cycle. In both cases, the system's dynamics do not converge to the point of zero gradient, rendering mixed strategies infeasible.
C.1.3 Flexible in-fight policy
In the case where agents do not have perfect information about each other's strengths, their estimates of the Nash equilibria may be based on their beliefs about the strengths of the agents in the game. As these beliefs, generally speaking, are different between the contestants, their estimates of the Nash equilibria may also be mismatched. As such, these estimates cannot be considered as true Nash equilibria, requiring further evolution of the contestants' policies.
One way to account for this discrepancy is to update the policies during the fight. Although our tests suggest that such a mechanism is inconsistent with animals' behavior in the experiment and, as such, is not used in our final model, we provide its description for completeness.
Updating the policies during the fight is supposed to lead to a joint Nash equilibrium. This involves two simple additions to our previously described model: i) the reward expectations in the payoff matrices  $\hat{R}_{1,2}$  are integrated over the "primary" beliefs about the participants' strengths, and ii) the optimization is performed jointly for both participating agents as their policies are observable to each other under the assumptions about this case.
Amending the payoff matrix  $\hat{R}_1$  can be reduced to redefining its entry  $\kappa_{2}$ , the only one depending on the agents' strengths. An in-fight Nash equilibrium is only defined under the 0-ToM model: the "primary" beliefs are used to predict the odds of winning while the "secondary" beliefs, normally used for predicting the opponent's actions, are unnecessary as actions are directly observed:
$$
\kappa_ {2} ^ {2} = \sum_ {i j} \kappa_ {2} \left(s _ {2} = i, s _ {1} = j\right) B _ {2 2} ^ {i} B _ {2 1} ^ {j}. \tag {29}
$$
That is, the equilibrium policy  $p_1$  for the first agent depends on the second agent's estimate of its expected reward  $\kappa_2^2$  for the case when both agents "attack". The expression for the optimal pure strategies is then rewritten based on Equation 25 as follows:
$$
\left\{ \begin{array}{l} \left(p _ {2} ^ {\text {p u r e} *} \left(\kappa_ {1} ^ {1} + \alpha - 1\right) + 1\right) \left(p _ {1} ^ {\text {p u r e} *} - \frac {1}{2}\right) > 0; \\ \left(p _ {1} ^ {\text {p u r e} *} \left(\kappa_ {2} ^ {2} + \alpha - 1\right) + 1\right) \left(p _ {2} ^ {\text {p u r e} *} - \frac {1}{2}\right) > 0. \end{array} \right. \tag {30}
$$
The equations above describe the in-fight Nash equilibrium which is not a part of our model. This analysis, however, may be useful in future studies of different social behaviors.
C.2 Model fit
In data fits, we used the entire data to propagate the beliefs while we only considered days 1-3 and 21-22 for the NLL evaluation. This is because, on days 4-20, "winners" were always matched with "losers" predetermining the results of the interactions and reducing the NLL sensitivity to the model parameters. As a result, the model using days 1-3 and 21-22 has a local minimum (Figure 7B) at the identified parameters, while the model using days 1-22 has a plateau instead (Figure 7A).
To assess the reliability of our data fits we performed additional fits on simulated data. To this end, we replicated the mouse experiment in a simulated environment and generated 500 choice patterns using various predefined parameters. We sampled parameters uniformly from the ranges previously used for parameter fits in mouse data, selecting only the policies that resulted in a  $30\% -70\%$  fraction of either action. We then evaluated the accuracy of reconstructed beliefs in both 0-ToM (Figure 7C,D) and 1-ToM (Figure 7E-H) models.
C.3 Model comparison
Below, we report the numerical results for the final round of model comparisons on our behavioral data. The differences in NLLs were computed for each mouse individually; here we show the mean
Figure 7: Parameter inference. (A-B) NLL for reconstructing the animal data; (C-H) reconstructed beliefs in simulation (color code: density of beliefs; r-values: Pearson correlation between simulated, reconstructed beliefs.
$\Delta NLL$  and its standard error of the mean for the training and testing datasets. Positive numbers correspond to our model explaining the data better than baselines/ablations. To establish which of the results are significant, we performed the t-test on  $\Delta NLLs$  followed by the FDR correction. We highlight in italics the results that did not pass the threshold q-value of  $q = 0.05$ .