Using Large Language Models to Simulate Human Behavioural Experiments: Port of Mars
Oliver Slumbers Joel Z. Leibo Marco A. Janssen
Abstract
Collective risk social dilemmas (CRSD) highlight a trade-off between individual preferences and the need for all to contribute toward achieving a group objective. Problems such as climate change are in this category, and so it is critical to understand their social underpinnings. However, rigorous CRSD methodology often demands large-scale human experiments but it is difficult to guarantee sufficient power and heterogeneity over sociodemographic factors. Generative AI offers a potential complementary approach to address this problem. By replacing human participants with large language models (LLM), it allows for a scalable empirical framework. This paper focuses on the validity of this approach and whether it is feasible to represent a large-scale human-like experiment with sufficient diversity using LLM. In particular, where previous literature has focused on political surveys, virtual towns and classical game-theoretic examples, we focus on a complex CRSD used in the institutional economics and sustainability literature known as Port of Mars. 1. Introduction
A Collective Risk Social Dilemma (CRSD) is a situation where a group must reach a collective target through individual contributions where everybody suffers if they fail to achieve the target. CRSD encompass many of the challenges facing society today. Some examples include emission reductions to reduce risks of climate change, and the vaccination for infectious diseases to create herd
immunity. Societal challenges related to the need for collective action are well-studied, in particular in trying to understand the governance of shared resources. Hardin introduced the term ‘tragedy of the commons’, encapsulating the notion that communities fail to govern shared resources without a central government regulating them, or private property systems. Ostrom later demonstrated that communities can self-govern under certain conditions. Still, more understanding is needed about the conditions that encourage collective action and this research span multiple disciplines. Experimental social science, rather than observational social science, provides us with a method to approach this question. Artificial experiments in lab settings provide controlled variation, for example helping social scientists rule out confounding effects when analysing the factors contributing to collective action. Ascertaining causality can prove difficult with the kinds of data available to observational social scientists. This is why experimental CRSD research is needed. However, this is afflicted by low power due to the difficulty, and cost, of large-scale human experiments. It is difficult to create large enough samples to validate results, and insufficient heterogeneity amongst participants can limit generalizability. Our hypothesis, along with others, is that large language models (LLMs) can be used to mitigate some obstacles that would otherwise make experimental CRSD research difficult. LLMs will not replace experimental research but can provide a useful complement, for example by identifying relevant configurations of experiments with human participants to implement and to provide predictions for those experiments. LLMs have the potential to model knowledge, beliefs and patterns of social behaviour, facilitating in silico research on human social behaviour. Due to being trained on a large corpus of human data, it is plausible to expect that LLMs may contain substantial latent social information. LLMs could be cost-effective / efficient tools to aid empirical studies by narrowing the pool of hypotheses and gaining guiding insights. However, the conditions that must hold for LLM-“participants” to be viable as a tool of in silico research remain an equally open-question.
Argyle et al. introduced Algorithmic Fidelity : ‘the
Round Summary:
The players spent < Time-Blocks > on System Health, an average of < Time-Blocks > each. The following trades were proposed and accepted / rejected: < Trades >
Discarding Prompt:
Current game state and rules: < Game Information > Summary of discussion: < Summary > Current player information: < Information > You can discard the following Accomplishment Cards: < Accomplishment Cards >, would you like to do so? Trading Prompt:
Current game state and rules: < Game Information > Summary of discussion: < Summary > Current player information: < Information > You require the following resources to complete your planned Accomplishment: < Resources > Based on the information, would you like to make a trade request? Accomplishment Card Prompt:
``` Summarise
Discarding
Trading
```
``` Events
Discussion
Planning
```
Event Prompt:
Current game state and rules: < Game Information > You are currently experiencing < Event Name > You must make the following decisions: < Event Decisions >
Discussion Prompt:
Current game state and rules: < Game Information > Summary of previous round: < Summary > Current player information: < Information > Generate a conversation between all of the players
Planning Prompt:
Current game state and rules: < Game Information > Summary of discussion: < Summary > Current player information: < Information > Based on the information, create two plans: One for how much to invest in System Health, one for which Accomplishment Card to prioritise. System Health Prompt:
Current game state and rules: < Game Information > Your plan with regards to System Health: < System Health Plan > How many time-blocks would you like to spend on System Health? ``` Accomplishment Cards System Health
```
Current game state and rules: < Game Information > You are working towards the following Accomplishment Card: < Accomplishment Card > You have the required resources to complete this card, would you like to complete it? Figure 1. Visualisation of a single round. We show the general format of the prompts that are passed to the players, these are simplified and detailed fully in Appendix B. Items in green refer to Port of Mars specific terminology. Items in orange refer to public information.
Items in blue refer to private player information. degree to which the complex patterns of relationships between ideas, attitudes, and sociocultural contexts within a model accurately mirror those within a range of human subpopulations’. The main idea is that LLM responses are coherent with those of distinct human subpopulations can be elicited by prompting the model to continue a response from the perspective of a member of the group in question. They suggested four minimum conditions to check for the validity of a given piece of LLM-based in silico research:
1. Social Science Turing Test —LLM responses are indistinguishable from parallel human texts. 2. Backward Continuity —LLM responses are consistent with the attitudes and sociodemographic information of the conditioning context, such that humans viewing responses can infer key elements of that input. 3. Forward Continuity —LLM responses proceed naturally from the conditioning context provided, reliably inflecting the form, tone and content of the context. 4. Pattern Correspondence —LLM responses reflect underlying patterns of relationships between ideas, demographics, and behaviour that would be observed in comparable human-produced data. When all four criteria hold then we can say there is algorithmic fidelity, and the implication is that it would be valid to conduct in silico research on LLM agents in such a setting . Algorithmic fidelity has predominantly received interest in domains where the goal is to characterize a distribution of static human responses and structured interview data Amirova et al.). These studies did not put multiple LLM agents together in the same environment to study their social interactions or how they solve problems together. Seperately, another line of work has focused on socially interacting LLM agents. Generative agents, such as those studied in, have been used to simulate groups of interacting individuals. These methods have potential in agent-based modeling of collective action dilemmas. They enable the creation of anew kind of agent-based model where it is possible to study the emergence of group-level social phenomena from the interaction of individual agents. These individuals can now talk to each other and utilize much more complex kinds of cognition than was possible with simpler agents. 1 There are a great many caveats, and the full methodological significance of algorithmic fidelity is far from settled. For further discussion see. In this work, we explore LLM agents in a novel CRSD. We initially validate the algorithmic fidelity of LLM agent responses within the CRSD by directly comparing against human responses and metrics from the original works. Then, based on an underlying assumption of algorithmic fidelity, we demonstrate the flexibility and power to model and study the interactions of differing social personas within CRSDs. In particular, we focus on Social Value Orientation (SVO) as the primary way of defining differences between our agents and study how this impacts the outcomes in the CRSD. Furthermore, we study the effects of leadership on top of the SVO outcomes, demonstrating the flexibility of traits and problem variants that can be effectively studied under an LLM agent approach.
2. Port of Mars
Collective action problems are widely studied, but they are generally abstracted as simple matrix games. While such abstractions have been used for decades, many have argued that they are of limited utility for predicting human behaviour in more complex (and realistic) settings. In particular, matrix games generally ignore critical features of real-world social dilemmas. Firstly, they miss the temporal extendedness of real-world dilemmas. Secondly, they do not require decisions with only partial information about the state of the world, the activities of others and true uncertainty. For example, the true impact of the climate crisis and human action towards it is an unknown, and unexpected events will occur regularly. In traditional public goods / common pool resources experiments, deep uncertainty is often elided since the probabilities of events are known (e.g. the probabilities of stochastic events in a poker game).
Therefore, it is important to ensure any new framework is capable of introducing ‘unknown unknowns’ to represent the “sometimes we just have to cross that bridge when we come to it” feature of real-world dilemmas. We must also consider how LLMs may impact methodological decisions, such as player motivation. Often monetary rewards are attached to performance, however, it is debated whether this an effective motivator. Gneezy & Rustichini found that appealing to moral incentives may be more effective. In contrast to AI research methods, like reinforcement learning, LLMs have no natural analog of reward. Therefore it is unnatural to motivate LLMs with monetary incentives. Instead, LLM agents can be motivated in other ways, e.g. by appealing to their social identity, or by asking them to role play as a certain character. We must choose a CRSD that appeals to such incentives in order to motivate behaviour. However, a critical prob
lem is the extensive literature that exists about CRSD. This contaminates the LLM training data, making it difficult to separate the LLMs social capabilities from simply recreating strategies discussed in the literature. We require a near novel CRSD that is not prominent in the LLM training
corpus. We have chosen the Port of Mars (PoM) game for the following reasons: 1) PoM meets the criteria of Hughes et al. and thus is a sequential social dilemma, this may render it full of potential for emergent social complexity according to conjectures such as those of Leibo et al. 2) PoM focuses on building a narrative for the players to engage with. This naturally fits the LLM framework. 3) PoM is specifically designed to include ‘unknown unknowns’ (it is a game of incomplete information in the game-theoretic sense), 4) PoM is a relatively novel game. While it has been previously described, it has a limited presence in the wider literature. This suggests that training data contamination is unlikely to prove overly problematic. PoM is a resource allocation game where five players balance individual goals against the conflicting needs of maintaining shared infrastructure called System Health, in the face of ongoing environmental, social, and technical challenges. The narrative identifies the players as members of Generation Zero: the first group of long-term residents to arrive on the Red Planet, and the Players experience the challenges of life as early citizens of a Martian settlement. Full game details are provided in Appendix A. Each player has a unique role: the Curator, Pioneer, Researcher, Politician, and Entrepreneur. Although the roles have a different narrative, the underlying payoff structure is the same.
This is not known by the players. Shared infrastructure degrades due to wear and tear. Players must maintain a minimum level of shared infrastructure for the game to continue. Players spend personal resources on shared infrastructure or on acquiring points to win. ’Dirty cards’ exist which players can claim for points at the expense of the shared infrastructure. As such, the game experience provides a fundamental CRSD between individual and collective goals. Each round features random event cards. These can lead to reductions of shared infrastructure
(e.g. crop failures) or other positive / negative consequences. Normally, players draw one event card per round, however this increases as the shared infrastructure degrades. These events are a source of incomplete information, as the deck of event cards is unknown, and the LLM players do not retain memory of previous games to memorise event cards. 2 See Du et al.
3. Framework Overview
All of the players in our framework are controlled by a single LLM, and they are differentiated through their prompting. All of the prompts, at minimum, contain the following information:
1. Game Description - A minimal description of the rules and narrative such that the LLM can play the game fully without requiring additional game information. 2. Round Information - Description of event cards that have been drawn in this round. Information about level of system health. 3. Player Identity - Provides the player role (e.g. Pioneer), and a description of their personality / traits. 4. Phase Information - Description of what currently needs to be done by the LLM player. Full prompts for each phase are provided in Appendix B. Our version of PoM consists of 9 rounds, however the LLM agents are not aware of the number of rounds. In each round the players make the following decisions:
1. Handle specific decisions relating to events. 2. Have a planning meeting with all of the other players. 3. Create or update longer-term plans. 4. Spend time-blocks on either the system health or the
resources. 5. Propose and accept/reject trade requests with other players for resources. 6. Decide whether to complete / discard available accomplishment cards. An early problem we encountered was that providing full histories of the game did not lead to strong gameplay (e.g. forgetting the rules). However, without full history we often found that round-by-round behaviour was inconsistent, for example with players acquiring different resources across round that were not consistent with the accomplishment card they were aiming for in the earlier rounds. To overcome this, we carry three pieces of information through the rounds. First, we construct a summary of the round for the players that tracks information on trades that happened, system health spending in the previous the round, and an LLM generated summary of the planning meeting. Secondly, in order to generate more consistent decision-making, we get the LLM players to generate two plans that are used to condition future behaviour: 1) Health Plan - The players decide how much of their time-blocks they want to invest in the system health, with their planning being contingent on the current rounds events and the player planning meeting, alongside the general information, and 2) Accomplishment Plan - The players decide which of their available accomplishment cards they want to complete next. At the start of each round, players can either decide to continue working towards the accomplishment card they worked towards in the previous round, or update to anew one. We found empirically that without these elements it was difficult to get strong, consistent behaviour from the LLM players. 4. Experiments
All experiments used gemini-1.5-flash-002 for the LLM players. The purpose of this work is not to test the differences between specific LLMs (e.g. OpenAI vs. Gemini vs. Claude vs. etc...) behaviour in these social science experiments, and due to lack of resources we chose the gemini model as it provided free-tiered API requests. 4.1. Algorithmic Fidelity
We begin our experiments by partially validating algorithmic fidelity, adapting the concept from to our LLM framework. In this work, we do not claim that we fully validate algorithmic fidelity, as were were only able to perform 2/4 of the tests due to the lack of corresponding human data. Specifically, we were unable to perform the social science Turing test here since we did not have parallel human-generated gameplay text, for example communication logs, (though it would be possible to add in the future), and also backward continuity required large-scale human evaluation of the LLM outputs, which we defer to future work. Therefore, our focus is on forward continuity and pattern correspondence as we have accessible human data to check these conditions. We provide these results as an initial proof of concept, and in combination with results in later sections which are well validated by the wider social science literature, we believe we can make reasonable claims about the replication of human behaviour in our experiments. Forward Continuity states that the LLM agent actions should proceed naturally from the conditioning context pro vided . We choose to initially demonstrate this by evaluating the performance of agents with traits that are likely to elicit obvious responses. For example, we prompted two types of players: Cooperative players with Altruis tic, Cooperative, Empathetic, Generous, Selfless traits and Selfish players with Self-Centred, Selfish, Uncooperative, Machiavellian traits.
We do this because the goal of forward continuity is to probe the features of the distribution, P (behaviour | backstory) . This distribution is naturally summarised by the performance numbers and other features of
Points % Dirty Cards Health Spend Personality Group
LLM Score Human Rank LLM Score Human Rank LLM Score Human Rank
Forward Continuity Selfish 8 . 8 ± 0 . 8 N/A 25% N/A 8 . 3 ± 1 . 9 N/A Cooperative 3 . 7 ± 0 . 4 N/A 11% N/A 40 . 8 ± 1 . 9 N/A
Pattern Correspondence Egalitarian Individualist 7 . 2 ± 0 . 6 1 13% 2 17 ± 1 . 7 52 . 5 ± 8 . 5 Egalitarian Communitarian 5 . 0 ± 0 . 3 4 4% 4 32 ± 2 . 1 47 . 5 ± 8 . 4 Hierarchical Individualist 6 . 2 ± 0 . 5 2 15% 1 13 ± 1 . 4 51 . 4 ± 4 . 2 Hierarchical Communitarian 3 . 7 ± 0 . 7 2 7% 3 34 ± 2 . 2 51 .
prompting them as a Egalitarian Individualist). The prompts for the four cultural groups are:
- Hierarchical Individualist - You value a well-defined social order and respect authority, yet emphasise personal freedom and personal achievement within that
structure. - Hierarchical Communitarian - You value a struc tured society where roles are clearly defined. You prioritise community welfare and collective responsibilities. - Egalitarian Individualist - You prioritise personal achievements. You see interactions as negotiations and value free competition among individuals.
- Egalitarian Communitarian - You place high value on the collective and prioritise making decisions that do not negatively impact anyone. Our goal is to demonstrate that LLM agents prompted as these groups reflect similar relationships between the groups as observed in the human data. We demonstrate our results in Table 1. For the Points (final game score) and the % Dirty Cards (percentage of time a player picks a Dirty card when it was available to them) we provide the raw values for the LLM agents, and the ranking observed in the human data. For the points, we have access to the average finishing position of each group and use this to provide rankings for each group. Note that we do not use the exact same metric for the LLM
players, instead providing the average number of points in successful games for the experiments with leadership spread over the SVO angles. For each subplot, there are three bars which represent the three different leadership variants proposed in Sec 4.4. position). We do not use the exact average finishing position for the LLM results as group composition is important towards this value, and this was not a controllable feature in the human data. In terms of the points, we can that both of the Individualist groups match up well between LLM and human ranks, whilst the Communitarian groups do not much up well. Overall, whilst this is not a fully positive result we do note that without group compositions matching up exactly it is difficult to exactly replicate the human results. Therefore, we find the overall strong performance of the individualist groups in both the human and the LLM experiments to be an interesting validation of pattern correspondence. The biggest discrepancy is the Communitarian groups which receive substantially fewer points in the LLM setting, in particular with the Hierarchical Communitarian LLM player performing very poorly in comparison to the human results. One potential explanation is that the descriptions of the cultural groups provided by may be less accurate for the Communitarian players, possibly over-emphasising the pro-social aspect in comparison to how this represents itself in humans. However, in terms of the % Dirty Cards metric, our LLM agent results match, in terms of ranking, exactly with those of the human data. Furthermore, the percentages also show similar inter-group differences to the human data, for example with the difference between the Hierarchical Individualists and the Egalitarian Individualists being small, and the subsequent gaps between the other groups being larger. We would argue that this is more indicative of algorithmic fidelity in comparison to points / final rank comparisons, as these as dirty card usage is less dependent on the group composition. Finally, in terms of system health spend we note a discrepancy between our results and those of the human experiments. However, the human results suggest no significant difference in the system health spend between the cultural groups, which we find surprising and goes strongly against the results of the LLM experiments.
the Politician role affecting decisions). Results are shown in Fig. 2 with three visualisations. An individual point of note is that 81% of games ended in survival, significantly higher than the survival rates in recent human PoM studies. The recent experiments were performed in an online environment in contrast to which was a card game. Possible explanations include: 1) LLM players may exhibit inherently prosocial behaviour that SVO prompting cannot fully override, or 2) our framework ensures consistent communication, which corresponds with findings in that groups that survived communicated more, a factor we explore in Sec.
4.3. Figs. 2A and 2B detail how players’ SVO angles affect gameplay. Points are generally inversely related to SVO angles, with the -15 [◦] player averaging 10 points compared to the 60 [◦] player’s 4. As shown in Fig. 2A, this is due to dif
Figure 5. We breakdown the results of Fig. 4, showing for which SVO angle player was the leader (x-axis), which SVO angle player won (y-axis). Values are displayed as percentages of total runs, and therefore may not sum to 100 as some runs end in no-one winning (due to port collapse). ferences in system health investment, the -15 [◦] player spends only 3 time-blocks per round, allowing more resource investment, while the 60 [◦] player spends 7 time-blocks, limiting their resources. In Fig. 2B, the -15 [◦] player’s total health spend rarely exceeds 40 blocks, whilst the 60 [◦] player’s spend rarely drops below 40. Interestingly, ports failed to survive when high-SVO players invested less, indicating group reliance on their contributions. We would also argue that this is a demonstration of forward continuity, as the natural behavioural change as a players SVO angle increases is to move towards the more altruistic actions, increasing system health spend. The percentage of dirty cards used shows similar trends to system health spending. This metric tracks how often a player uses a dirty card (gaining points at the expense of port health) relative to opportunities. The -15 [◦] player used dirty cards 60% of the time, compared to 20% for the 60 [◦] player. However, the trend across SVO angles is less consistent, with no clear linear decrease. For low SVO angles, behaviour aligns with competitive, individual point maximisation. For higher SVO angles, the trade-off between social decision-making and personal gain is less clear, as dirty card impacts can be offset by investing more in system health. Fig. 2B shows that players with the highest SVO angle use noticeably fewer dirty cards overall.
Fig. 2C visualises metrics from the trading phase. Trades are categorised as ’Fair’, ’Generous’ (offering more than requested), or ’Selfish’ (offering less than requested). While these definitions may oversimplify contextdependent decisions, clear patterns emerge. Pro-self players (lower SVO angles) reject more trades compared to the prosocial (higher SVO angles) players. The trend is especially evident in ’Selfish’ trades, with pro-self players requesting more resources than they offer far more frequently than pro-social players. 4.3. Communication
Fig. 3 presents the results of experiments examining the role of communication on player behaviour. These experiments used the same group composition as described in Sec. 4.2, consisting of five players with varying SVO values. The key difference between the two result sets lies in the presence or absence of any communication, controlled by having the round-by-round meeting or not. Communication is a well-studied solution to cooperation in social dilemmas, with studies like showing significant positive effects. In our PoM framework, communication modestly enhances cooperative behaviour. Without communication, system health spending decreases moderately across all SVO values, making teams more vulnerable to damaging events and less able to mitigate dirty card use. We find this to be statistically significant (p-values shown on the figure) for all SVO values other than the 15 [◦] and 30 [◦]
players. Notably, this analysis would validate that of the wider literature that communication is a good way to alleviate the negative group effects of particularly ego-centric players within the group. The effect for the percentage of dirty cards used, the other main ego-centric behaviour, is more muted at the lower SVO angles, especially being statistically insignificant at the lowest SVO angle. This could be explained by the fact that where system health spend is a major topic during communication (alongside trades), actions like dirty card selection are very rarely discussed by the players. Finally, we also note that group survival drops from 81% when communication is allowed to 71% without, most likely explained by the across board increase in system health spend. 4.4. Leadership
One of the key advantages of an LLM-based approach, as highlighted earlier in the paper, is the ability to easily investigate the effects of varying social variable compositions, a challenge in human-based studies. In this work, we focus on the concept of leadership and its impact on outcomes.
Leadership is a well-research topic in social dilemmas, with findings such as the self-moderating effects of SVO on leadership influencing outcomes. Our results are presented in Fig. 4. In these experiments, we modified prompts to designate one player as the leader and tested three variations of how this leadership role was communicated. In the ‘Vanilla’ setting, only the leader is informed of their role, with the option to disclose it to others. In the ‘Announce’ setting, the leader is openly identified to all player.
In the ’Unaware’ setting, the leader is not told their role, but the other players are informed of who the leader is–without knowing the leader is unaware. Each leadership variant was tested over 50 games, with win rates and detailed breakdowns shown in Fig. 4. In Fig. 4, we present survival rates dependent on both which SVO angle player was the leader, and which leadership variant was employed. A trend emerges: survival rates decrease as the leader’s SVO angle increases (no matter the variant), with the group surviving the least when the 30 [◦] and 60 [◦] players acted as the leaders. This result may align with findings from, which suggest that pro-self (low SVO angle) leaders are more likely to act in self-interest. In this context, self-interested play involves encouraging high system health spending from others while maximising individual points, leveraging their leadership role to ensure the port’s survival. Another way to view this is that pro-self leaders may engage in “cheap talk”, influencing others’ behaviour through verbal framing rather than altering their own resource allocation.
As leaders, they may influence the group’s decision-making without necessarily increasing their pro-social contributions. We give two examples of this for the -15 [◦] SVO player when they were acting as the leader, demonstrating how their comments during communication do not necessarily match the actions that they take:
Example 1: Communication Comments:
”Remember, the Port’s health comes first. We can’t focus solely on our individual goals, we need to be mindful of the overall health of the Port.”
Personal Planning: ”I believe it’s crucial to maintain a stable environment while focusing on my own goal. The port health will fall to 27 after the deduction, placing it in the zone for three events. While I acknowledge the need for collaboration, I prioritize my own goals this round. Therefore, I will allocate a minimal amount to the port’s health.”
Example 2 Communication Comments
”Hold on, Entrepreneur. We just lost all of our resources. Let’s try to conserve our coins and resources this round, we need to get a handle on our health first.” ”We all have individual goals, yes, but we need to prioritize the Port’s survival first. We can worry about our personal goals later”
Personal Planning: ”The port health is currently stable, and I see an op portunity to focus on acquiring resources for my goals. While I recognize the importance of the port’s health, I am willing to take a risk for my own benefit. I will contribute minimally to the port’s health to ensure it remains above the critical threshold.”
On the other hand, pro-social (high SVO angle) leaders might underperform because their contributions to the group are already near their maximum, leaving little room to improve outcomes further. This limitation could explain the poorer survival rates observed when high SVO angle players assume leadership roles.
In Fig. 4 we present a breakdown of the wins across the variants. On the x-axis we have the SVO angle of the leader, and on the y-axis we have the corresponding SVO angle of the player that one an experiment. For example, in the first heat map, the box 18 in the top-left corner captures the fact the -15 [◦] SVO player won 18% of the experiments when the -15 [◦] player itself was the leader. We highlight along the diagonal the outcomes for the player that is the leader. For each of the leadership variants, we note a specific finding. Firstly, for the announced leader variant we can see that the overall survival rate is comfortably the highest in comparison to the two other leadership variants. If we
Figure 6. Results of the optimal group experiments. Total system health spend is the sum over all of the players contribution throughout a full run. A lower Gini Inequality score implies more equality across the players. The dots are scaled by the average score of all the players in the experiments, with larger dots implying the average score across the players was higher. add up all of the percentages (note the total runs for each variant is the same), we find the total for the announced variant to be 414 vs. 336 (unaware) vs. 334 (vanilla), which suggests the announced variant leads to the highest survival rate overall. This aligns with findings suggesting that the mere presence of a recognised leader can be crucial for maintaining a social-ecological system. We note slightly different effects for both the unaware and vanilla leadership variants. For the unaware variant, the impact is probably as expected - a much lower overall survival rate than the announced variant, probably due to the lack of a leadership effect, and also little noticeable impact on the leaders winning performance themselves (not deviating far from their mean performance under any other leader). The vanilla variant, on the other hand, shows an interesting impact on the leaders own performance. Whereas the overall survival rate under the vanilla variant is much lower than the announced variant, the leaders individual win-rates are greatly reduced for all but one of the SVO values (consider the highlighted green boxes in comparison to the other values in the rows). This may be explained by a mixture of cheap talk which causes the leaders to take pro-self actions, whilst not gaining the recognised leader effect of. 4.5. The Optimal Group
In our final experiment, shown in Fig. 6, we aim to identify the optimal group composition based on system health, average player points and the Gini inequality (a measure of the equality across the five players). While simplistic, one could argue that optimal play balances high system health investment (to prevent port collapse), high individual points (ensuring no player’s success is compromised) and a low Gini inequality score (equality of points across players). Fig. 6 summarises the average outcomes from all our experiments. Results for SVO angles ranging from -15 [◦] to 60 [◦]
correspond to those discussed in the main figures, while an additional set of experiments with SVO angles from -30 [◦] to 60 [◦] explores scenarios with lower SVO values within the group composition. Full details of these experiments, along with those categorised as ’Other’, are provided in Appendix F. The first observation is that groups composed of lower SVO angles consistently perform worse across all of the categories than those with higher SVO angles. This aligns with earlier experiments, where lower SVO groups invested less in system health. However, their average player points are also often lower, likely due to frequent port collapses, which reduces all players points to 0. Next, consider the best runs (those in the top right corner of the plot) in terms of inequality and health spend. These are exclusively from the SVO -15 [◦] to 60 [◦] experiment set, with the majority of runs all occurring when the other players were aware a leader was present. Interestingly, there is a noticeable difference between the best runs in terms of Gini and health spend being in the ’Unaware’ leader variant and those in terms of average points and health spend being in the ’Announced’ leader variant. One explanation for this is that, in the ’Unaware’ variant, there is the general impact of the presence of a leader without the potential cheap talk aspects of a player being aware that they are, in fact, the leader. 5. Conclusion & Future Work
This work is an initial attempt to understand the behaviour of LLM agents in CRSDs. Specifically, we began by exploring two of the conditions of algorithmic fidelity within a novel CRSD, Port of Mars. We hypothesise that PoM is an ideal candidate for LLM experimentation on CRSDs, in particular due to lack of strategy contamination in the LLM training data, unlike other CRSDs. We demonstrate initial signs of algorithmic fidelity, in terms of two of the four conditions over a range of personality variables. We leverage this validation to further investigate a selection of variables that have been shown to impact social behaviour in social dilemmas, demonstrating that LLMs can simulate the results of human behaviour experiments. Future work can take two avenues for advancing our understanding of human behaviour in CRSDs - first by further validating the algorithmic fidelity of an LLM framework through more comparisons to human results, and second by exploring the impacts of other personal and social variables at a scale that is difficult to match in human experiments.