# Alternative cliques of coexisting species in complex ecosystems

Guim Aguadé-Gorgorió, Sonia Kéfi

<figure><img src="image_1.png" /></figure>

## To cite this version:

Guim Aguadé-Gorgorió, Sonia Kéfi. Alternative cliques of coexisting species in complex ecosystems. Journal of Physics: Complexity, 2024, 5 (2), pp.025022. 10.1088/2632-072X/ad506a. hal-04668222

HAL Id: hal-04668222

https://hal.umontpellier.fr/hal-04668222v1

Submitted on 11 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

PAPER • OPEN ACCESS

# Alternative cliques of coexisting species in complex ecosystems

To cite this article: Guim Aguadé-Gorgorió and Sonia Kéfi 2024 *J. Phys. Complex*. **5** 025022

View the [article online](#) for updates and enhancements.

## You may also like

- Comparing three generations of D-Wave quantum annealers for minor embedded combinatorial optimization problems
Elijah Pelofske

- Explicit and probabilistic constructions of distance graphs with small clique numbers and large chromatic numbers
A. B. Kupavskii

- General clique percolation in random networks
Jingfang Fan and Xiaosong Chen
<figure><img src="image_4.png" /></figure>

Original Content from
this work may be used
under the terms of the
Creative Commons
Attribution 4.0 licence.

Any further distribution
of this work must
maintain attribution to
the author(s) and the title
of the work, journal
citation and DOI.

<figure><img src="image_5.png" /></figure>

## Abstract

The possibility that some ecosystems can exist in alternative stable states has profound implications for ecosystem conservation and restoration. Current ecological theory on multistability mostly relies on few-species dynamical models, in which alternative states are intrinsically related to specific non-linear dynamics. Recent theoretical advances, however, have shown that multiple stable ‘cliques’—small subsets of coexisting species—can be present in species-rich models even under linear interactions. Yet, the mechanisms governing the appearence and characteristics of these cliques remain largely unexplored. In the present work, we investigate cliques in the generalized Lotka–Volterra model with mathematical and computational techniques. Our findings reveal that simple probabilistic and dynamical constraints can explain the appearence, properties and stability of cliques. Our work contributes to the understanding of alternative stable states in complex ecological communities.

## 1. Introduction

Multistability, the process by which a dynamical system can reach different stable states under the same environmental conditions, has key implications in ecology. The most dramatic one, albeit not completely understood, is that of ‘catastrophic shifts’: the phenomena by which ecosystems, from shallow lakes to microbial communities, can undergo unexpected and abrupt transitions between healthy and degraded states [1–3]. Beyond examples of catastrophic shifts, the possibility of alternative stable states has long been known to play a role in ecological succession and community assembly. There, it provides a framework to understand historical community patterns and the contingency of species invasions [4, 5].

Most mathematical theories explaining ecological multistability have been grounded on simple, few-species dynamical models [6, 7]. In these, alternative stable states can arise by imposing specific non-linear dynamics. A single dynamical equation can then stabilize at one of multiple stable abundances depending on the initial conditions [8, 9]. However, this oversimplified picture fails at explaining the mechanisms by which multistability can appear in complex communities where many different species interact [6, 10].

Recent theoretical developments are starting to help us understand what multistability looks like in complex models [7, 11–13]. More particularly, we now know that multiple stable states can appear in species-rich models without the need to impose specific non-linear dynamics [7, 12]. The most commonly used framework to investigate these systems is the generalized Lotka–Volterra (GLV) model, where the growth of each species is affected by self-regulation as well as positive and negative linear impacts by other species [14]. Theoretical and empirical research on the species-rich GLV model has revealed a rich landscape of dynamical behaviors, from species coexistence to chaotic fluctuations and multistability (see e.g. [12, 14–20] and figure 1(A)). Depending on the strength and heterogeneity of interactions, the GLV model can exhibit four well-defined phases. These are (i) population divergence at high cooperation, (ii) a single state of species coexistence, (iii) a regime of competitive exclusion states, each with only one surviving species, and
(iv) a much more complex and less understood phase, where chaotic fluctuations and multiple fixed points can arise (figure 1(B)).

In the present work we focus on the fourth phase. Recent developments have helped us understand the properties of chaotic dynamics in this phase [17, 20]. High heterogeneity in species interactions can drive the system towards a pinball-like scenario, where species approach a stable community state that is then invaded by other species from the pool [17]. This rich behavior is potentially related to empirical evidence of chaotic ecological dynamics [21] and has been recently tested in microbial community experiments [22].

On the contrary, the properties of multiple stable states in this phase remain an uncharted territory [20]. We know that these so-called *clique* states can be found across mathematical models and usually support small and excluding subgroups of coexisting species [7, 23, 24]. Theoretical advancements have predicted the number and stability of cliques in the infinite species limit, which depend on the symmetry of the interaction matrix (see e.g. [16, 18, 24]). However, other key ecological properties of cliques remain to be explained. We do not totally understand the mechanisms by which these small cliques appear in the first place. Therefore, a fundamental question is uncovering the rules that determine which and how many species can coexist in a clique.

In the present work, we investigate cliques in the GLV model. Our goal is to uncover the mechanisms that dictate their presence and constrain their properties. Our results show that simple rules can predict how many species can coexist inside a clique, the patterns constraining their interactions and the impact of species migrations. Each of these processes depends predictably on the properties of the studied system, providing potentially testable signatures to detect cliques in experimental data. Our work provides a first step towards detecting the presence of alternative cliques of coexisting species in complex empirical systems.

## 2. Methods

In the present methods, we describe the GLV model (section 2.1), the rationale behind sampling interaction strengths at random (section 2.2) and how we parametrize them (section 2.3). The central aspect of the work is to systematically confront two mathematical approaches. First, we explain how we numerically simulate the dynamics of the model to find and characterize stable cliques (section 2.4). Second, we define a mathematical test to study if randomly selected subsets of species can be identified as a clique without simulating the dynamics (section 2.5). Finally, we describe how species migration and invasions are implemented (section 2.6).

### 2.1. The GLV model

The central point of our work is to study the GLV model of S interacting species [12]. In this model, each species is labeled $i = 1, 2, \dots, S$ and the abundance $N_i$ of species $i$ follows

$$ \frac{dN_i}{dt} = r_i N_i \left( 1 - \frac{N_i}{K_i} + \sum_{j \neq i}^{S} a_{ij} N_j \right). \qquad (1) $$

In the absence of species interactions ($a_{ij} = 0 \forall i \neq j$), the model does not show multistability. Our main goal is to analyze how interactions impact the presence of multiple stable states. A typical procedure is to set intra-species parameters $r_i$ and $K_i$ to have low heterogeneity, so that we can write $r_i = r$ and $K_i = K \forall i$. We subsequently rescale time as $t \to rt$, divide species abundances by carrying capacities ($x_i \equiv N_i/K_i$) and rescale interaction strengths ($A_{ij} \equiv a_{ij}K_j$) [12, 20]. This leaves us with a simpler set of differential equations to study:

$$ \frac{dx_i}{dt} = x_i \left( 1 - x_i + \sum_{j \neq i}^{S} A_{ij} x_j \right). \qquad (2) $$

A fundamental question in theoretical community ecology is then to understand how the dynamical properties of this model depend on the properties of the interaction matrix $A$. In our notation, a positive interaction strength ($A_{ij} > 0$) indicates cooperation and a negative strength ($A_{ij} < 0$) indicates competition. While other researchers have used the opposite sign convention (see e.g. [12]), we believe that keeping the same sign for interaction strengths and their ecological meaning make the results easier to interpret.

### 2.2. Disordered systems and random interactions

The number of species ($S$) in natural ecosystems is often very large. Attempts at empirically characterizing the complete interaction matrix $A$ become extremely complex. Inspired by the framework of disordered systems in statistical mechanics [25], another approach is to sample the elements of $A$ from probability
<figure><img src="image_6.png" /></figure>

distributions. Since the influential work by Robert May [26], modeling species-rich communities with random interaction strengths has provided a unified lens to understand many ecological scenarios. When many species and interaction types become intertwined, the species-specific details cease to matter and community-level patterns emerge (see SI I.C and [26, 27]).

In this context, parametrizing our model is simplified to tuning the statistical properties of A. Following the standard approach of the GLV model [12], we generate $A_{ij}$ from a normal distribution without imposing correlations between matrix elements [12, 18]. The interaction matrix A is built as an Erdős–Rényi graph, where $A_{ij}$ is normally distributed $N(\mu, \sigma)$ with probability $p$ or $A_{ij} = 0$ with probability $1-p$. Here we focus in the $p=1$ scenario, so that $\sigma$ is the only parameter that controls interaction heterogeneity, and discuss the impact of $p < 1$ in the supplementary material (SM) II.D. Each system is therefore characterized by $S_c$, a unique $(\mu, \sigma)$ pair and the interaction matrix $A \sim N(\mu, \sigma)$. We investigate how the dynamics of (2) change with the mean $\mu$ and standard deviation $\sigma$ of A (figure 1(A)).

## 2.3. Parametrizing $(\mu, \sigma)$ to find cliques

We first plot a complete phase space of (2) to visualize the parameter domain where cliques are found [12, 20]. We do so by setting $S = 50$, $\mu \in [-1.8, 0.5]$ and $\sigma \in [0, 1.6]$, and generate a grid of $40 \times 40 = 1600$ systems. We integrate each system and count the fraction of simulations that reach a stable state. This allows us to have a first picture of the dynamical regimes of the model (figure 1(A)). The specific simulation procedure used to integrate each system is explained in the next section.

Our main task is to evaluate the properties of clique states. To explore the properties of cliques across a variety of systems, we set $\mu$ and $\sigma$ values chosen from random uniform distributions falling inside the *Cliques, cycles and chaos* phase of figure 1(A). For simplicity, we choose the $(\mu, \sigma)$ area under study to be a rectangle that does not span the whole phase. This is done to simplify simulations and avoid contaminating results of clique properties with those of states in the neighbouring phases, as boundaries are not necessarily sharp for low $S$ (figure 1(A), [12]). The choice of the shape might alter which specific cliques are found, but not their general properties (see e.g. figures S8 and S9 in [20]).

For sections 3.1 and 3.3, we sample $\mu$ and $\sigma$ from uniform distributions $(\mu \sim U(-1.3, 0.5), \sigma \sim U(0.3, 0.6))$ and study the properties of cliques appearing within this domain. We generate 1000 systems inside the rectangle, each with a unique $(\mu, \sigma)$ pair. Interaction strengths $(\mu)$ in this range are consistent with empirical estimates of GLV dynamics in bacterial communities with $S \approx 50$ [22]. Conversely, estimating empirical $\sigma$ values for GLV-like dynamics remains an open task [28].
In section 3.2 we present the properties of cliques in a single system with $\mu = -0.5$ and $\sigma = 0.5$. This is done to present our results in the simplest possible format for readability. The equivalent results for systems spanning the whole rectangle are presented in the SM (section II.C).

In section 3.2 we also explore the properties of cliques for systems with increasing $S$ and close to the critical line separating stable coexistence from the cliques regime. This line begins at the point $(\mu, \sigma) = (-1, 0)$ and ends at $(\mu, \sigma) = (0, \sqrt{2/S})$ [12, 20, 29] (figure 1(A)). The equation for the line is $\sigma_c = \sqrt{2/S}(1 + \mu)$. The steepness of this line then decreases with $S$. At higher $S$, the coexistence phase shrinks and the line lowers [12, 20]. We study the diversity of cliques as $S$ grows by simulating systems with $\mu = -0.5$ and displacing $\sigma$ slightly above the critical line ($\sigma = 0.5\sqrt{2/S} + 10^{-2}$).

## 2.4. Simulating the dynamical equations

Once a system is fully defined by $\mu$, $\sigma$ and $S$, we generate 100 different random initial conditions for the initial population abundances. For each initial condition, we integrate the system during $\Delta t_1 = 3 \cdot 10^3$ timesteps. Most clique states become stable much faster, but we choose a longer timeframe to allow for potentially long transients (see e.g. figure 1(B), where relaxation to clique states is of the order of $\Delta t \approx 10^2$). To check if a state is stable, we further integrate the system during $\Delta t_2 = 10^2$ additional timesteps and check if all species abundances have remained the same, as previously done in [7] (see SM I.A for additional details on the numerical implementation).

For those states that are stable, we define as $x'^*$ the vector containing only the abundances of surviving species, and $A'$ is the reduced interaction matrix containing only the interaction strengths between these surviving species. A clique state is characterized by the properties of the system in which it appeared ($S, \mu, \sigma$) and the diversity, abundances and interactions of the final surviving species ($S', x'^*, (\mu', \sigma')$). This variables are the central object of study of the present work.

The interaction matrices of systems with $S = 50$ species already contain $S(S-1) = 2450$ elements, so that the statistics of the finite-size sample $A \sim N(\mu, \sigma)$ are likely to be very similar to those of the original distribution $N(\mu, \sigma)$. In systems with fewer species, smaller matrices might have statistics that deviate from those of the original distribution. Given that cliques support a small number of species $S' < S$, we try to find and explain the patterns underlying these deviations. First, we measure the difference in terms of mean and standard deviation between the initial system and each clique ($\Delta\mu = \mu - \mu'$, $\Delta\sigma = \sigma - \sigma'$). Second, we study the presence of correlations between abundances $x'^*$ and interaction strengths $A'$ (see [30] and Results section). Third, we measure how the statistics of interactions in a clique $(\mu', \sigma')$ are related to the number of species $S'$.

## 2.5. An analytical framework to obtain cliques from A

We define a mathematical test to check if randomly selected subsets of species can be stable cliques of (2). Our framework is based on a two-step process: we first select a random set of $S' < S$ species, and latter test if these selected species can coexist in a clique state.

Species that coexist in a clique are not expected to be a completely random subset of the original system. Yet, identifying the specific coexisting sets from (2) demands very advanced mathematical machinery [18]. Instead, here we test four different methods to randomly select subsets of species from the original system. We either select species at random with probability $\gamma$ (method 1) or proportional to their interactions (methods 2–4). For the latter method, we select a species with a probability proportional to the perceived impacts by other species (method 2), the exerted impacts towards other species (method 3) and the sum of both (method 4). For example, in the last method a species is chosen to participate in a potential clique with probability

$$p_i = \gamma \frac{\sum_j (A_{ij} + A_{ji})}{\max_{k \in \{1, \dots, S\}} (\sum_j (A_{kj} + A_{jk}))} \quad (3)$$

Here $\gamma \in [0, 1]$ is constant across all species. By choosing larger or smaller values of $\gamma$ we can control the average amount of species ($S'$) that are selected. The $\max_k$ term in the denominator indicates that, across all possible species $k \in \{1, \dots, S\}$, the filter selects the one for which the sum between parenthesis is largest.

For each of the methods, we select $S'$ species and obtain their interaction matrix $A'$ from the original values of A. Now we set a test for the conditions by which it can be a feasible and stable state. First, the abundance of species can be calculated from $x'^* = (-A')^{-1}$. Clique states will need to be feasible, so that only subsets with positive species abundances are considered ($x'^* > 0 \forall i$). Second, we compute the Jacobian matrix, with elements $J'_{ij} = x_i'^* A_{ij}$. Clique states need to be stable, so that only subsets with negative eigenvalues of the Jacobian are considered. With this process, we filter random subsets of the original system
that are feasible and stable. We next evaluate if the properties of these subsets are consistent with the properties of cliques found by numerically solving the system (see above). The details of the mathematical implementation can be found in the SM, section I.B.

## 2.6. Species migration

Albeit unstable, the species-extinction state $x_i = 0$ is an absorbing state of the dynamics of (2). The relation between chaotic fluctuations and species extinctions has been long studied in theoretical ecology (see e.g. [31–33]) and has recently been analysed in the GLV model [17, 20]. Heterogeneous interactions in A can drive the GLV model to a state with chaotic abundance fluctuations that could drive species to extinction (figure 1(B)). In this section we study how the properties of cliques change if extinct species can reinvade.

In the previous sections, we have presented the conditions by which cliques are stable, assuming that extinct species cannot reinvade. However, if extinct species were able to reinvade, they could destabilize the stable community towards another fixed point or even a chaotic regime. This is commonly studied by introducing a minimal migration rate $m \gtrsim 0$:

$$ \frac{dx_i}{dt} = x_i \left( 1 - x_i + \sum_{j \neq i}^{S} A_{ij}x_j \right) + m. \quad (4) $$

We implement this in the simulations (section 2.3) by admitting a very small population threshold below which the abundance of a species cannot keep decreasing (see [17] and SM I.A for details). In the mathematical tests (section 2.4), we study the effects of migration by asking if cliques are invadable. We test if at least one species not present in a chosen subset has a positive growth rate when invading at low abundance:

$$ r_{i \in I} = \frac{1}{x_i} \frac{dx_i}{dt} = 1 + \sum_{j \in R} A_{ij} x_j^* > 0. \quad (5) $$

Here, subscripts I and R indicate species belonging to the Invader (extinct) and Resident (clique) sets, respectively (see SM I.B.3).

In both the simulations and mathematical tests, we study the likelihood of finding uninvadable cliques as the size of the original system S grows and species can reinvade. In the simulations, by adding a small population threshold forbidding extinctions. In the mathematical tests, by testing if species not selected in a subset can invade it. The impact of migrations is analyzed because cliques are likely to become unstable in very large systems [18, 20]. The hypothesis we test is that the likelihood of finding a destabilizing invader increases rapidly as the size of the system S grows.

# 3. Results

The results of this paper are organized as follows. In section 3.1 and figure 2 we show that cliques display predictable patterns in species interactions. In section 3.2 and figure 3 we show how competition bounds the number of coexisting species in cliques. In section 3.3 and figure 4, we show how species migration and a larger system size can make cliques unstable.

## 3.1. Predictable interaction patterns in cliques

The GLV model harbors a phase with multiple stable states of coexisting species (figure 1(B)). We study how the interaction strengths in cliques ($A'$) compare with the interactions in the original system (A). We start with strictly positive initial abundances ($x_i(t = 0) > 0 \forall i$) and suppose that species which go extinct due to the dynamics are not able to reinvade (we explore the impact of invasions in section 3.3).

We first study if the interaction statistics of cliques ($\mu', \sigma'$) deviate from those of their system of origin ($\mu, \sigma$). We plot in figure 2(A) the difference in mean interaction ($\Delta\mu = \mu - \mu'$) and standard deviation ($\Delta\sigma = \sigma - \sigma'$) between each original system and the stable cliques it generates. As a general rule of thumb, in the species-rich GLV model, we know that species will stably coexist if interactions are geared towards weak and homogeneous competition [12, 26] (figure 1(A)).

We find that cliques systematically fall into a region where they are less competitive and less heterogeneous than in their system of origin $A(\mu, \sigma)$ (figure 2(A)). The ‘less competitive, less heterogeneous’ signature is found by numerically integrating (2) (see section 2.3). The same signature also appears by mathematically testing which randomly chosen species can coexist stably (see sections 2.4 and SM II.A). In the mathematical test, however, the condition is not as strict. This is because random species choices permit some competitive cliques with $\mu' \lesssim \mu$ that are hardly reached by the dynamics (see SM II.A for a detailed description and comparisons across methods).
<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /><figcaption>**Figure 2.** Interaction patterns in cliques. In (A) we show that mean interaction strength in cliques is less competitive than in the original system ($\mu' > \mu$) and heterogeneity is lower ($\sigma' < \sigma$). Each dot represents the difference $\mu - \mu'$, $\sigma - \sigma'$ between an original system A with $S=50$ and a stable clique $A'$. The equivalent comparison with mathematical tests can be found in SM II.A. Cliques appear as subsets that harbor less competitive and less heterogeneous interactions than their system of origin. In (B) we test the theoretical feasibility condition from [30] see (6)). We find that the interaction strengths in cliques ($A'_{ij}$) correlate with species abundances ($x_i^{*}$), creating the predicted diffuse pattern. Roughly put, abundant species in a clique (larger circles in the community sketch) need to compete weakly (thinner interaction lines). Instead, less abundant species (smaller circles) can compete strongly (thicker lines) without impairing coexistence [30]. Each dot represents an $A'_{ij}$ value and its mathematical expectation, while the dark line indicates the 1 : 1 relation expected from the theory. For both panels we plot in red the linear regression of the data to help visualize its qualitative trend.</figcaption></figure>

We also study if there are deeper patterns at the level of species abundances and their impacts on other species. Barbier and colleagues described a species-level interaction pattern pervading coexistence in large communities [30]. One of the mathematical results is that species that coexist at high abundance are found to compete weakly, while species with lower abundance are allowed to impact each other at higher strengths. This allows us to predict the expected value of an interaction between two species in the clique, $A'_{ij}$, as a function of their abundances $x_i', x_j'$ (see [30] for the complete development):

$$E(A'_{ij}|x_i', x_j') = \mu + (1 + \mu) \frac{(x_i' - \bar{x}') x_j'}{\sum_{k \neq i} x_k'^2} \quad (6)$$

where $\bar{x}'$ is a measure for the expected abundances that species would hold if all interactions had the same value $A'_{ij} = \mu$:

$$\bar{x}' = \frac{1 + \mu \sum_i x_i}{1 + \mu}. \qquad (7)$$

Species-level interactions in cliques follow the same diffuse pattern (figure 2(B)). Interactions in a clique state ($A'$) deviate from the randomness of A in a precise way, by which high-abundance species do not impact each other strongly. Species then coexist inside a clique following the same interaction patterns previously tested in large empirical communities [30]. Cliques are then those interacting subsets ($A'$) of the original system (A) that fulfill two constraints. First, they contain the most homogeneous and weakly competing interactions of the pool (figure 2(A)). Second, they ensure that abundant species in the clique are in turn weak competitors (figure 2(B)). Next we study what are the possible constraints behind these patterns.

## 3.2. The diversity of cliques is bounded by ($\mu, \sigma$)

We have seen that cliques represent a subset of species of the original interaction matrix A that tend to hold weaker competition ($\mu' > \mu$) and lower heterogeneity ($\sigma' < \sigma$). We now study if there are explicit relations between the properties of cliques ($S', \mu', \sigma'$) and those of the system of origin ($S, \mu, \sigma$).

We define a single system ($S, \mu, \sigma$), and study the diversity of cliques that appear from it ($S'$) and the lowest and highest interaction strength values these cliques support (min($\mu'$), max($\mu'$)). We show that the relations between $S'$ and $\mu'$ explain the maximum number of species that can survive in a clique given $S, \mu$ and $\sigma$. An equivalent analysis for a range of ($\mu, \sigma$) values instead of a single one is shown in SM II.C.
<figure><img src="image_9.png" /><figcaption>Figure 3. Species diversity in cliques is bounded by $\mu$ and $\sigma$. In (A) we plot the observed mean interaction strength of cliques ($\mu'$) in terms of their diversity ($S'$). Light gray dots are the statistics of randomly sampled elements of A. Colored dots indicate the observed $\mu'$ values of cliques appearing in a system with $\mu = -0.5$ (dashed gray lines) and $\sigma = 0.5$. The maximum and minimum competition strengths in cliques are predictable and depend only on the parameters of the original distribution $\mu, \sigma$ and the clique size $S'$. Equating the two expressions allows us to compute the maximum diversity ($S'_M$) that a clique could ideally attain given the statistics of the original pool. In (B) we show that, even if we increase system size, the maximum observed diversity of cliques (max($S$)) remains close to the predicted $S'_M(\mu = -0.5, \sigma = 0.5) \approx 12$. In (C) we show the maximum size of a clique for a system close to the coexistence-to-cliques transition (fixed $\mu = -0.5$, decreasing $\sigma = 0.5\sqrt{2/S + 10^{-2}}$). The maximum clique diversity scales as half of the total diversity, consistent with the diversity of the coexistence state before the critical transition [12].</figcaption></figure>

### 3.2.1. Probabilistic constraint on highest $\mu'$

We first study the constraints of the random sampling process. Interactions in a clique, encoded in $A'$, are sampled from the original system A. As clique diversity $S'$ increases and we sample more elements from A, the statistical properties of the clique ($\mu', \sigma'$) will inevitably resemble those of the original system ($\mu, \sigma$) [34]. In figure 3(A) we plot in light gray the mean of randomly chosen subsets of A as their size increases. We see that a clique will have $\mu' \to \mu, \sigma' \to \sigma$ as $S'$ increases. The highest values of $\mu'$ that a clique can harbor follow the expected trend for the statistics of $S'(S' - 1)$ elements sampled from a normal distribution [34]. Because of the tendency to minimize competition (figure 2(A)), some cliques can fall as far as a $4\sigma$ distance from $\mu$. Then, we can approximate that the clique with highest mean interaction strength is bounded by

$$ \max(\mu') \approx \mu + 4 \frac{\sigma}{\sqrt{S'(S' - 1)}}. \qquad (8) $$

The most cooperative clique in the studied range can have a mean interaction as high as the original system permits ($\mu = -0.5$), plus an added deviation proportional to the heterogeneity of the system and the number of species in the clique $S'$. The factor $S'(S' - 1)$ is the number of elements sampled from A, as each of the $S'$ species in the cliques interacts with $S' - 1$ other species. Equation (8) imposes a probabilistic constraint on $\mu'$: cliques cannot be more cooperative than what the original system ($\mu, \sigma$) and their size $S'$ allow (dashed black line, figure 3(A)). The probability of finding cooperative cliques decreases very rapidly with their diversity $S'$.

### 3.2.2. Stability constraint on lowest $\mu'$

We next ask what is the smallest, most competitive $\mu'$ value that cliques can support. Observed $\mu'$ values (colored dots, figure 3(A)) do not appear as low as they could by sampling (light gray dots). The most competitive cliques observed (min($\mu'$)) are not as competitive as probability could allow (figure 3(A)). There is a stronger dynamical constraint limiting competition in stable cliques.

We know that there is a limit to how competitive and heterogeneous species can be to coexist stably [12, 20, 26, 35]. This ($\mu_c, \sigma_c$)-limit induces a transition which separates stable community coexistence from the chaos and cliques regime (figure 1(A)). We hypothesize that species surviving in a stable clique fulfill the same coexistence condition: once other species have gone extinct, the remaining species must obey the GLV criteria for stable coexistence.

In the original parameter space, the condition is a line crossing from points ($\mu = -1, \sigma = 0$) to ($\mu = 0, \sigma = \sqrt{2/S}$) (figure 1(A)), [12, 20, 29]). The first point does not depend on system size: neutral coexistence ($\sigma = 0$) is always disrupted when competition overcomes self-regulation, no matter the number of species [15, 36]. The second point decreases with system size: a system with more species will have a smaller coexistence phase [12, 26]. For a given clique with mean competition $\mu'$, heterogeneity $\sigma'$ and number of species $S'$, the condition for stability is to ensure that average interaction strength is above
$$ \min(\mu') \approx \sqrt{\frac{S'}{2}\sigma' - 1}. \qquad (9) $$

A larger ($S'$) or more heterogeneous ($\sigma'$) clique requires more positive interactions to maintain stable coexistence [12, 26]. This imposes a lower bound to the competitive strengths that cliques can support given their size and interaction heterogeneity.

The mathematics behind (9) are valid for large systems: the critical transition is not sharp for $S' \lesssim 100$, so that smaller systems could hold $\mu' \lesssim \min(\mu')$ and still be stable. (see figure 5(b) in [12]). Because cliques are particularly small communities, it is likely that stable cliques of size $S'$ can still exist beyond the threshold.

To estimate a correction of (9) for low $S$ values, we perform a separate numerical test of destabilizing small GLV communities (see SM II.B for a complete description). In it, we generate systems with $\mu = 0$ and find the value of $\sigma_c^{90\%}$ for which 90% of states are unstable as $S$ increases. As predicted [12], $\sigma_c^{90\%} = \sqrt{2/S}$ beyond $S = 50$. However, at lower $S$ values, we find that $\sigma_c^{90\%} > \sigma$: smaller communities can support higher interaction heterogeneity. We hypothesize that this is consistent with evidence of deviations in the most positive eigenvalue of a random finite-size matrix [37] (see SM II.B). Because cliques are particularly small, their stability threshold is better approximated by a line from $(\mu' = -1, \sigma' = 0)$ to $(\mu' = 0, \sigma_c^{90\%} = \gamma S'^{-1.08})$, with $\gamma = 14.118$ (see SM II.B). Under this correction, the most competitive clique ($\min(\mu')$) sampled from $A$ that can be stable then sits at

$$ \min(\mu') \approx \frac{(S')^{1.08}}{\gamma} \min(\sigma') - 1. \qquad (10) $$

This value is lowest when the heterogeneity of clique interactions ($\sigma'$) is also lowest. Given the original system with $(\mu, \sigma)$, the lowest standard deviation $\min(\sigma')$ that can be obtained when sampling $S'(S' - 1)$ elements from $A$ has an analytical expression depending only on $\sigma$ and $S'$ [34] (see SI II.B). Applying it, we show that the lowest interaction strength of cliques only depends on clique size ($S'$) and the statistics of the original system:

$$ \min(\mu') \approx \frac{(S')^{1.08}}{\gamma} \sigma \sqrt{1 - \frac{2}{n-1} \left( \frac{\Gamma(n/2)}{\Gamma((n-1)/2)} \right)^2} - 1. \qquad (11) $$

where $\sigma$ is the standard deviation of the system of origin and $n = S'(S' - 1)$. As $S'$ increases, larger cliques need higher $\mu'$ values to maintain stable coexistence (figure 3(A), dashed red line).

Conditions (8) and (11), corresponding to the black and red dashed lines in figure 3(A), explain the interaction constraints of cliques in the GLV model. Cliques cannot be more cooperative than what sampling from $A$ allows, nor more competitive than what stable coexistence allows.

### 3.2.3. Competition bounds clique diversity

Our results explain the maximum diversity ($S'_M$) that clique states can attain given a system with fixed $(\mu, \sigma)$. This can be estimated by the crossing of the two constraints (black and red lines, figure 3(A)). Interestingly, a system with more species ($S$) but equal $(\mu, \sigma)$ will not allow for larger cliques: ($S'_M$) only depends on $(\mu, \sigma)$ (figure 3(B)).

Finally, we study the highest diversity that cliques can attain if $(\mu, \sigma)$ can vary. The highest diversity is found when $\mu$ is highest and $\sigma$ is lowest, so that $(\mu, \sigma)$ is closest to the coexistence regime (figure 1(A)). Interestingly, the location of this transition depends on $S$. As $S$ increases, the line decreases and the coexistence phase shrinks (figure 1(A)). Given a fixed $\mu$, cliques appear at lower $\sigma$ in larger systems ($\sigma_c = \sqrt{2/S(1 + \mu)}$). A clique generated from a system with lower $\sigma$ is likely to hold more species (higher $S'$).

The prediction at the coexistence phase is that the number of surviving species decays as $\sigma$ increases [12]. At the critical line, the number of surviving species is about half of the system size. Right above the critical line, results indicate that cliques should maintain a similar diversity of $S' = S/2$ (see figure 3(a) in [12]). In figure 3(C) we plot the largest observed clique for a single system poised right above the critical line. We can see that the most diverse clique that each system harbors scales approximately as $S/2$. Smaller systems slightly overcome this threshold (see SM II.B), while in larger systems it becomes computationally harder to find the largest possible clique (figure 3(C)).

## 3.3. Species migration and clique instability

Theoretical results indicate that many clique states can exist in the GLV model (see e.g. [16, 18] and [7] for a review of the literature). Yet, models estimate that most cliques are unstable in the many-species limit ($S \to \infty$) [18], often leading to chaotic dynamics [20]. In our analysis, however, we find many stable cliques.
<figure><img src="image_10.png" /><figcaption>**Figure 4.** Species migration decreases clique stability. We plot the fraction of stable simulations against system size with (blue dots) and without (orange dots) species migration. We compare these with two theoretical predictions: the presence of successful invaders on cliques generated from the dynamics of the GLV model (dark line) and on cliques filtered by the mathematical procedure (gray line). Larger system sizes destabilize cliques due to adding invaders (blue dots, dark line). Randomly selected stable subsets do not accurately mimic the composition and resistance to invasions of cliques (gray line).</figcaption></figure>

The main reason is that our studied system has a moderate number of species ($S \in [10, 100]$) and no species migration ($m = 0, (4)$). If a species goes extinct, it remains so and cannot reinvade [33]. Our hypothesis is that some of the observed cliques would become unstable if extinct species can reinvade. Moreover, if we increase system size, we would not observe larger cliques (see sections above). Instead, it would only increase the likelihood of finding potential invaders. In the many species limit, there would always be an invader to any clique state, reaching the predicted theoretical scenario of unstable cliques [16–18].

We test this hypothesis with two coherent methods (see SM I.B.3). First, we simulate (4) with $m > 0$ and count the fraction of initial conditions that reach a stable state. Second, we simulate (4) with $m = 0$ and test if there are species not present in each clique that fulfill the invasion criteria (5). Our results can be summarized into two main outcomes.

First, stable cliques become increasingly rare as system size increases and species do not get extinct ($m > 0$, figure 4, blue dots). The theoretical tests of successful invasions ($m = 0, (5)$) follow the same trend (figure 4, dark line). This is consistent with the hypothesis that larger systems harbor fewer stable cliques: system size $S$ increases the likelihood of finding a destabilizing invader [16–18]. On the contrary, a larger fraction of cliques is stable if species can go extinct or the system size is moderate (figure 4, yellow dots).

Second, the mathematical test provides a fast method to estimate aggregate clique properties (figures 2, 3 and SM II.A) but does not accurately capture the effects of invasions (figure 4). These errors become more likely as system size increases. This is because, in the mathematical tests, we select species at random and test if they can coexist stably without accounting for invasions by other non-selected species. The selected species set accurately recovers typical clique properties (see above and SM II.A), but it is likely that it is invadable when other species can migrate from the pool (figure 4, gray dashed line). To find uninvadable cliques, one would require a much slower test that checks each species subset first for stability and then against all potential invaders [38].

# 4. Discussion

Cliques have been identified in a variety of complex biological models [7], making their analysis relevant across life science disciplines. Previous research has uncovered the phase in which cliques emerge [12] or the number of cliques [18] in the species-rich GLV model and across complex biological systems [7]. Here we have shown that simple rules can explain the basic ecological properties of these small cliques of coexisting species.

## 4.1. How do cliques arise in random interaction models, and why are they so small?

Multistability in community models appears when the interaction network (A) is too heterogeneous for species to coexist stably [12, 26, 35]. Our results indicate that the specific subsets of stable coexisting species beyond this threshold fulfill three main constraints.

First, cliques are those subsets of species of the original pool that have the most positive and homogeneous interactions possible. This is consistent with the notion that strong competition [36] and interaction heterogeneity [26] destabilize community coexistence. At the species-level, coexistence also
selects for a subtle non-random pattern, whereby pairs of species with high abundance are in turn weak
competitors [30].

Second, the number of species that a clique can support is bounded by competition. Cliques with more species increase their average competition while at the same time reduce the competitive strength they can support. This two-fold constraint explains the small diversity of most clique states: too many coexisting species rapidly overcome the stability threshold. At most, clique states emerging very close to the stable coexistence regime can harbor about half of the species of the original system.

Third, species migration from the original pool can destabilize cliques. Larger systems cannot generate more diverse cliques, but in fact only provide destabilizing invaders. This explains mathematical predictions by which most cliques become unstable at very large community sizes [18]. As S grows, the likelihood of finding a potential invader increases and stable cliques become rarer.

**4.2. Empirical evidence of cliques**

We outline potential links between the above results and field and laboratory experiments. In the field, community shifts have been observed across environmental and altitudinal gradients [39–41]. Sharp shifts between different bacterial compositions have also been observed in experiments and clinical analyses of microbial communities [42–44]. These two avenues hint towards the possibility that experiments with plant communities [45–47] or bacterial ecosystems [3, 48, 49] could hold key information about ecosystem multistability.

Confronting dynamical observations with theoretical predictions is a much more nuanced problem. Using the GLV model, recent work has observed the mathematical signatures of coexistence and chaotic dynamics in field and laboratory experiments [22, 30]. The GLV model has also been recently used to test multistability signatures in microbial communities at a much lower diversity of two to six bacterial strains [48, 49]. Connecting theoretical explanations of multistability and empirical species-rich observations is a promising avenue [50].

Towards this goal, our results provide a set of empirical fingerprints to assess the presence of cliques in complex empirical communities. These are (i) the selection for weaker and homogeneous competition (figure 2(A)), (ii) the correlation between abundances and competition strengths (figure 2(B)), (iii) how competition bounds species richness (figure 3(A)) and (iv) how community stability decays with species invasions (figure 4).

**4.3. Limitations and next steps: the role of non-random interactions**

Empirical parametrization of complex species-rich communities is particularly difficult [28]. To overcome this barrier, we have followed the common convention of sampling interactions at random [26]. This theoretical framework has recently showed its potential in capturing the behavior of experimental systems of many interacting species [22, 30].

Ecological networks, however, typically hold non-random structures [51–53]. Non-random interaction patterns have been suggested to play an essential role in the occurrence of multiple stable states [11, 13], but the question of how the properties of the GLV model are affected by non-random interactions remains open. We have shown that the properties of cliques are governed by the statistical properties of A. If the interaction matrices *A* were non-random, we hypothesize that the properties of cliques could change dramatically.

**5. Conclusion**

Multistability has been observed in natural species-rich communities, both in the form of large catastrophic transitions as well as sharp shifts between similar community states. Because of the importance of these shifts, an important body of literature has been devoted to understand the mechanisms underlying multistability. While simple dynamical models can provide an explanation for multistable behavior, the mechanisms explaining multistability in complex ecosystems remain mostly unknown. In species-rich models, a specific type of multiple stable states has been found to appear when heterogeneous competition is at play [7, 12, 16, 18]. While simple models typically require specific non-linearities for multistability to appear, these so-called cliques can arise even under linear interactions in species-rich models.

In the present work, we have investigated cliques in the GLV model. Our results provide both an explanation for their occurrence and a prediction for their ecological properties. Cliques are subsets of the original ecosystem that fulfill a set of simple dynamical and statistical constraints. These predictable signatures provide potential benchmark tests to confront experimental evidence of multistability. Our work, therefore, brings forward an explanation and a testable framework of multistability in empirical species-rich systems.
## Data availability statement

The data that support the findings of this study are openly available at the following URL/DOI: https://github.com/GuimAguade/Alternative-cliques-of-coexisting-species-in-complex-ecosystems.

## Acknowledgments

The authors thank B Pichon, I Lajaaiti, N Humbert, M Barbier, L Arola, J-F Arnoldi and two anonymous reviewers for insight and feedback on the manuscript and the mathematical methods involved. G A-G thanks the hospitality of the team at Espai Món where most of this work was done. Special thanks to J Goldberg for inspiration. G A-G was supported by a 2022 postdoctoral fellowship of the Fundación Ramón Areces. S K was supported by the Alexander von Humboldt foundation and by the Grant ANR-18-CE02-0010-01 of the French National Research Agency ANR.

## ORCID iDs

Guim Aguadé-Gorgorió https://orcid.org/0000-0001-9467-3027

Sonia Kéfi https://orcid.org/0000-0002-9678-7770

## References

[1] Scheffer M, Carpenter S, Foley J A, Folke C and Walker B 2001 *Nature* **413** 591

[2] Petraitis P 2013 *Multiple Stable States in Natural Ecosystems* (Oxford University Press)

[3] Lahti L, Salojärvi J, Salonen A, Scheffer M and De Vos W M 2014 *Nat. Commun.* **5** 4344

[4] Kadowaki K, Nishijima S, Kéfi S, Kameda K O and Sasaki T 2018 *Ecol. Indic.* **85** 991

[5] Bunin G 2021 *Oikos* **130** 489

[6] Kéfi S, Saade C, Berlow E L, Cabral J S and Fronhofer E A 2022 *Phil. Trans. R. Soc. B* **377** 20210386

[7] Aguadé-Gorgorió G, Arnoldi J-F, Barbier M and Kéfi S 2024 *Ecol. Lett.* **27** e14413

[8] Noy-Meir I 1975 *J. Ecol.* **63** 459

[9] May R M 1977 *Nature* **269** 471

[10] Kéfi S, Holmgren M and Scheffer M 2016 *Funct. Ecol.* **30** 88

[11] Lever J J, van Nes E H, Scheffer M and Bascompte J 2014 *Ecol. Lett.* **17** 350

[12] Bunin G 2017 *Phys. Rev. E* **95** 042414

[13] Karatayev V A, Baskett M L and van Nes E H 2023 *Am. Nat.* **202** 260

[14] Galla T 2018 *Europhys. Lett.* **123** 48004

[15] Kessler D A and Shnerb N M 2015 *Phys. Rev. E* **91** 042705

[16] Biroli G, Bunin G and Cammarota C 2018 *New J. Phys.* **20** 083051

[17] Roy F, Barbier M, Biroli G and Bunin G 2020 *PLoS Comput. Biol.* **16** e1007827

[18] Ros V, Roy F, Biroli G, Bunin G and Turner A M 2022 arXiv:2212.01837

[19] Lorenzana G G, Altieri A and Biroli G 2023 arXiv:2309.09900

[20] Mallmin E, Traulsen A and De Monte S 2024 *Proc. Natl Acad. Sci.* **121** e2312822121

[21] Benincà E, Huisman J, Heerkloss R, Jöhnk K D, Branco P, Van Nes E H, Scheffer M and Ellner S P 2008 *Nature* **451** 822

[22] Hu J, Amor D R, Barbier M, Bunin G and Gore J 2022 *Science* **378** 85

[23] Fried Y, Kessler D A and Shnerb N M 2016 *Sci. Rep.* **6** 1

[24] Fried Y, Shnerb N M and Kessler D A 2017 *Phys. Rev. E* **96** 012412

[25] Villain J 1977 *J. Phys. C: Solid State Phys.* **10** 1717

[26] May R M 1972 *Nature* **238** 413

[27] Barbier M, Arnoldi J-F, Bunin G and Loreau M 2018 *Proc. Natl Acad. Sci.* **115** 2156

[28] Carrara F, Giometto A, Seymour M, Rinaldo A and Altermatt F 2015 *Methods Ecol. Evol.* **6** 895

[29] Akjouj J, Barbier M, Clenet M, Hachem W, Maïda M, Massol F, Najim J and Tran V C 2024 *Proc. R. Soc. A* **480** 20230284

[30] Barbier M, De Mazancourt C, Loreau M and Bunin G 2021 *Phys. Rev. X* **11** 011009

[31] Berryman A and Millstein J 1989 *Trends Ecol. Evol.* **4** 26

[32] Allen J, Schaffer W M and Rosko D 1993 *Nature* **364** 229

[33] Solé R and Bascompte J 2012 *Self-Organization in Complex Ecosystems*. (MPB-42) (Princeton University Press)

[34] Johnson R A et al 2002 *Applied Multivariate Statistical Analysis* (Prentice Hall)

[35] Marcus S, Turner A M and Bunin G 2024 Extinctions as a vestige of instability: the geometry of stability and feasibility (arXiv:2405.11360)

[36] Armstrong R A and McGehee R 1980 *Am. Nat.* **115** 151

[37] Dean D S and Majumdar S N 2006 *Phys. Rev. Lett.* **97** 160201

[38] Arnoldi J-F, Barbier M, Kelly R, Barabás G and Jackson A L 2022 *Methods Ecol. Evol.* **13** 167

[39] Kitayama K 1992 *Vegetatio* **102** 149

[40] Hemp A 2006 *Plant Ecol.* **184** 27

[41] Cornwell W K and Ackerly D D 2009 *Ecol. Monogr.* **79** 109

[42] Gajer P et al 2012 *Sci. Trans. Med.* **4** 132ra52

[43] Gibbons S M, Kearney S M, Smillie C S and Alm E J 2017 *PLoS Comput. Biol.* **13** e1005364

[44] Fujita H et al 2023 *Microbiome* **11** 1

[45] Van Ruijven J and Berendse F 2009 *Oikos* **118** 101

[46] Reich P B, Tilman D, Naeem S, Ellsworth D S, Knops J, Craine J, Wedin D and Trost J 2004 *Proc. Natl Acad. Sci.* **101** 10101

[47] Tilman D, Reich P B and Knops J M 2006 *Nature* **441** 629
[48] Wright E S, Gupta R and Vetsigian K H 2021 *FEMS Microbiol. Ecol.* 97 *fiab073*

[49] Lopes W, Amor D and Gore J 2023 Multistability driven by cooperative growth in microbial communities *bioRxiv* Preprint

[50] Long C, Deng J, Nguyen J, Liu Y-Y, Alm E J, Solé R and Saavedra S 2024 *Proc. Natl Acad. Sci.* 121 e2312521121

[51] De Ruiter P C, Neutel A-M and Moore J C 1995 *Science* 269 1257

[52] Kéfi S, Miele V, Wieters E A, Navarrete S A and Berlow E L 2016 *PLoS Biol.* 14 e1002527

[53] Guimaraes P R Jr 2020 *Annu. Rev. Ecol. Evol. Syst.* 51 433