# Ancestral complexity and constrained diversification of the ant olfactory system

Simon Marty, Antoine Couto, Erika H Dawson, Neven Brard, Patrizia d'Ettorre, Stephen H Montgomery, Jean-Christophe Sandoz

► To cite this version:

Simon Marty, Antoine Couto, Erika H Dawson, Neven Brard, Patrizia d'Ettorre, et al. Ancestral complexity and constrained diversification of the ant olfactory system. Proceedings of the Royal Society B: Biological Sciences, 2025, 292 (2045), pp.20250662. 10.1098/rspb.2025.0662. hal-05399089

HAL Id: hal-05399089

https://hal.science/hal-05399089v1

Submitted on 4 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
PROCEEDINGS B

royalsocietypublishing.org/journal/rspb

<figure><img src="image_2.png" /></figure>

Research

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

Check for
updates

**Cite this article:** Marty S, Couto A, Dawson EH, Brard N, d'Ettorre P, Montgomery SH, Sandoz J-C. 2025 Ancestral complexity and constrained diversification of the ant olfactory system. *Proc. R. Soc. B* **292**: 20250662.

https://doi.org/10.1098/rspb.2025.0662

Received: 7 March 2025
Accepted: 27 March 2025

**Subject Category:**
Evolution

**Subject Areas:**
evolution, neuroscience, behaviour

**Keywords:**
evolution, social insects, nestmate recognition, cuticular hydrocarbon, Formicidae, olfaction

**Authors for correspondence:**
Antoine Couto
e-mail: antoine.couto06@gmail.com
Jean-Christophe Sandoz
e-mail: jean-christophe.sandoz@universite-paris-saclay.fr

Electronic supplementary material is available online at https://doi.org/10.6084/m9 figshare.c.7750306.

Ancestral complexity and constrained diversification of the ant olfactory system

Simon Marty<fnref n="1" />,†, Antoine Couto<fnref n="1" />,†, Erika H. Dawson<fnref n="2" />, Neven Brard<fnref n="2" />, Patrizia d'Ettorre<fnref n="2" />, Stephen H. Montgomery<fnref n="3" />,‡ and Jean-Christophe Sandoz<fnref n="1" />,‡

<figure><img src="image_5.png" /></figure>

Communication is a cornerstone of social living, allowing the exchange of information to align goals and synchronize behaviour. Ants, a group of highly successful social insects, have heightened olfactory abilities that are integral to their evolutionary success. Essential for colony cohesion and cooperation, a female-specific olfactory subsystem processes information about nestmate recognition cues (cuticular hydrocarbons), including basiconic sensilla on the antenna and a cluster of specific glomeruli in the antennal lobe. While it has often been linked to ants' social lifestyle, the evolutionary origins and phylogenetic distribution of this system remain unknown. We conducted a comparative exploration of the ant olfactory system across eight major subfamilies, integrating neuroanatomical, chemical and behavioural analyses. Our findings reveal that sophistication of the ant olfactory system has deep evolutionary roots. Moreover, antennal lobe investment is not associated with social traits such as colony size, polygyny or foraging strategies, but correlates with cuticular hydrocarbon profile complexity. Despite neuroanatomical differences, different ant species consistently excel in nestmate discrimination, indicating adaptation to chemical diversity while maintaining reliable social recognition. This suggests that cuticular hydrocarbon profile and neuronal investment in olfactory neuropil have co-evolved to sustain discrimination performance.

1. Introduction

Effective communication is fundamental to the success of social living. It enables group members to exchange information, resolve conflicts and coordinate their actions, allowing successful cooperation and group cohesion. Consequently, whether in mammalian societies or vast colonies of social insects, social evolution is believed to influence the development of recognition and communication systems, thereby supporting reliable social interactions [1–4]. Despite significant progress in understanding the neural components of communication systems, particularly in social insects [5], a notable knowledge gap remains regarding the evolution of the sensory systems that support this critical function. Therefore, exploring the roles of novel neuronal populations and other adaptations within the sensory pathways that support communication is key to understanding the mechanisms underlying the evolution of social behaviour.

Ants stand as prominent models of group living and cooperation, owing to their remarkably large and complex colony organizations, diverse kin structures and extensive interspecific variation in morphological traits, dietary preferences, foraging behaviours and life history strategies [6,7]. At the

© 2025 The Author(s). Published by the Royal Society under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/4.0/, which permits unrestricted use, provided the original author and source are credited.

<footnote>†These authors contributed equally to the study.</footnote>

<footnote>‡These authors contributed equally to the study.</footnote>

<footnote n="1">¹IDEEV, Université Paris-Saclay, CNRS, IRD, Evolution Genomes Behaviour and Ecology, 91190 Gif-sur-Yvette, France</footnote>

<footnote n="2">²Laboratory of Experimental and Comparative Ethology, Université Sorbonne Paris Nord, 93430 Villetaneuse, France</footnote>

<footnote n="3">³School of Biological Sciences, University of Bristol, Bristol BS8 1TQ, UK</footnote>
core of their cooperative behaviours lies a sophisticated communication system that supports pheromonal signalling and the perception of recognition cues, primarily odourant compounds detected by the olfactory system [8,9]. Accordingly, ants possess one of the most complex olfactory systems among insects [10–12], as evidenced by studies on selected species which highlighted its pivotal role in social interactions and scent-guided behaviours [13,14]. Nevertheless, a significant gap remains regarding the evolutionary trajectory of ants’ olfactory system. Understanding these trajectories could illuminate the adaptive significance of olfaction within the context of sociality and the remarkable taxonomic radiation of ants.

Insect antennae are typically covered with different types of sensory hairs known as sensilla, which enclose the dendrites of olfactory sensory neurons (OSNs). The axons of these neurons project to the antennal lobe (AL) in the brain, where they form glomeruli, discrete spheroidal structures that serve as processing units. Each OSN generally expresses a single olfactory receptor (OR)—together with the ubiquitous co-receptor—which defines its response profile to odourant stimuli [15,16]. The OR expression also dictates the specific glomerular target of each neuron, establishing a nearly one-to-one correspondence between OR numbers and glomerular count [17]. These olfactory glomeruli serve as central hubs where local interneurons and neuromodulatory neurons refine olfactory information before it is relayed to higher brain centres by projection neurons [18].

The complexity of ant olfactory systems is supported by two key observations. First, ants possess a high number of AL glomeruli and a greater abundance of associated OR genes compared with most other insect clades [10, 11, 15, 19]. Second, ants exhibit an olfactory specialization that has been associated with the detection of recognition cues [5]. This specialized pathway is characterized by a unique type of antennal sensilla, the basiconic sensilla, wherein OSNs express a specific clade of 9-exon ORs [11] and project exclusively to a distinct cluster of glomeruli [20,21], which lack serotonergic innervation [22–25]. Notably, electrophysiological recordings have revealed that cuticular hydrocarbons (CHCs), which are identity-signalling compounds, are sensed by the 9-exon ORs [26–28] and basiconic sensilla OSNs [29–33]. These molecules are pivotal in insect communication [34], conveying information about species, colony affiliation, life stages and reproductive status [35], thereby facilitating the recognition of nestmates over non-nestmates and the maintenance of colony cohesion [36]. Thus, this olfactory subsystem is believed to have evolved and expanded significantly in ants, driven by the need to meet sophisticated communication demands within complex social colonies [11, 19, 28, 37].

At this stage, our knowledge of ants' olfactory system is biased towards a few specific clades, significantly limiting insight into how its sophistication relates to ants' social behaviour and ecology. For example, it is not yet known whether the increased complexity of the AL and the extensive repertoire of ORs translate into enhanced accuracy in recognizing social identities, such as distinguishing between nestmates and non-nestmates. Additionally, the diversification of recognition cues, particularly the chemical composition and complexity of CHC profiles [38,39], likely interplays with the evolution of the ant olfactory subsystem. This co-evolution suggests a dynamic feedback mechanism in which increasingly complex chemical signals drive adaptations within the neural and sensory apparatus. However, to date, little is known about the influence of diverse communication demands and varied social structures on the evolutionary trajectory of the olfactory system across the ant phylogeny [40]. Therefore, it remains uncertain whether the olfactory system was already complex at the onset of ants' remarkable taxonomic radiation or if its complexity evolved during their diversification.

Here, we employed a comparative approach to explore the evolutionary history of key olfactory neuroanatomical traits in ants, examining whether variation in ecology and social structure have influenced differential investment in their olfactory system. Overall, we investigated 14 species from the eight most species-rich Formicidae subfamilies [41], differing in feeding behaviour, ecological niche and colony kin structure. We compared the distribution of basiconic sensilla across antennal segments and examined the AL glomeruli that receive serotonergic innervation, aiming to identify those involved in social recognition. Additionally, we analysed neuropil volumes and glomerular counts, integrating these measurements to provide a comprehensive assessment of the olfactory system in ants. We conducted ancestral state reconstructions and evolutionary rate analyses on the number of glomeruli to identify derived traits and periods of rapid evolution. Using this evolutionary framework, we finally tested whether the anatomical variation observed in the olfactory system correlates with behavioural differences between three exemplary species, and is linked to ecological or chemical factors. By integrating these diverse approaches, we aimed to elucidate the role of ecological and/or social factors as selection pressures shaping the evolution of the olfactory system across the ant phylogeny.

## 2. Results

### (a) Diversity and distribution of antennal sensilla

To explore the evolution of sensory structures in ants, we conducted scanning electron microscopy on the antennae of 13 ant species spanning a broad phylogenetic range. Among the various morphological types of sensilla, basiconic sensilla, identified by their peg-in-socket shape and porous tip, contrast with other sensory hairs [42]. This sensillum type was consistently present across all studied species, exhibiting only slight morphological differences (figure 1a–c).

In ants, the distribution of basiconic sensilla is strongly biased towards the distalmost segments of the antennae (figure 1d,e; two-way ANOVA, *p* < 0.001 across flagellomeres), with their proportion decreasing towards proximal segments. Consequently, there is a progressive decline in both the number and density of sensilla along the antenna (electronic supplementary material, figure S1d,e), albeit with significant variability in attenuation patterns across species (figure 1e; two-way ANOVA, species × segment interaction, *p* < 0.001; electronic supplementary material, figure S1d,e). The decline is particularly pronounced in Myrmicinae species, where basiconic sensilla are absent beyond the third segment in *Cephalotes atratus*, the fourth segment in
<figure><img src="image_6.png" /><figcaption>**Figure 1.** Distribution of basiconic sensilla on ant antennae. (a–c) Scanning electron micrographs of basiconic sensilla across ant species. These sensilla show slight morphological differences, including a reduction in base thickness from basal taxa (Neoponera apicalis; a) to Formicinae (Formica fusca; b) and Myrmicinae clades (Cataglyphis cursor; c). (d) Scanning electron micrographs of Paraponera clavata, Eciton burchellii and Cephalotes atratus antennae, illustrating the distribution of basiconic sensilla (red dots) across antennal segments. These sensilla are more densely packed on the distal segments of the antenna. Scale bars, 0.5 mm. (e) Relative proportion of basiconic sensilla across antennal segments in various ant species (coloured dots). Basiconic sensilla are notably concentrated in the distal part of the antennae, prominently in Myrmicinae (varying green shades). (f) Number of basiconic sensilla on the antenna plotted against antenna surface area, showing species means (large coloured dots) and individual data points (small dots). The number of basiconic sensilla correlates strongly with the size of the antenna (Pearson test: *t* = 4.97, d.f. = 11, ***p* < 0.001, *R*² = 0.72).</figcaption></figure>

*Atta sexdens*, and the fifth segment in *Messor barbarus*. By contrast, *Eciton burchellii* (Dorylinae) exhibits a more even distribution of basiconic sensilla across the first four segments, with each containing approximately 18% of the total count (figure 1d,e; post hoc Tukey, *p* > 0.05 between segments 1, 2, 3 and 4). Across species, the total number of basiconic sensilla varies significantly (electronic supplementary material, table S1 and figure S1e; two-way ANOVA, *p* < 0.001), ranging from 52.2 ± 5.6 in *M. barbarus* to 382 ± 80 in *Paraponera clavata*. These differences primarily reflect variation in antenna size, as indicated by the strong correlation between basiconic sensilla counts and the measured antennal surface area (figure 1f; Pearson test, *t* = 4.9664, d.f. = 11, *p* < 0.001, *R*² = 0.72).

## (b) Characterization of anatomical regionalization in the antennal lobe

Given the marked variation in sensilla numbers, despite a consistent organizational pattern, we investigated how these
differences affect the structure of the antennal lobe (AL). Using immunohistochemistry and confocal microscopy, we character-
ized the AL across 14 species of Formicidae. In all examined species, the AL exhibits two distinct regions resembling segregated
glomerular clusters (figure 2a–c). The dorso-caudal region typically features more compact glomeruli compared with the rostral
region of the AL (referred to as the main-AL, figure 2a–c). This organization was consistently observed across species, along
with additional shared characteristics that define the T<sub>B</sub> cluster [20,21]. A detailed description of the criteria used to delineate
the T<sub>B</sub> cluster is provided in the supplementary results (electronic supplementary material, figure S5 and §S3.2, Definition of the
TB cluster), along with movies of AL scans and three-dimensional (3D) reconstructions [43]. This cluster, previously termed T6
in ants owing to the numbering of its input tract [25,44], was renamed T<sub>B</sub> to ensure consistency across Hymenoptera [5,45], as
its relative position varies among taxa (e.g. T3 in bees [46], T8 in hornets [45,47]). In ants and hornets, basiconic sensilla OSNs
project exclusively into this cluster [20,21,47], which is reported to lack serotonin innervation in the *Camponotus* clade, unlike
the rest of the AL, which is typically innervated by serotonergic neurons [22–24]. Lack of serotonergic innervation has been
<figure><img src="image_7.png" /></figure>

**Figure 2.** Anatomical characterization of the antennal lobe (AL). (a,b) Confocal optical sections of the antennal lobe in an ant species lacking serotonin innervation in the T<sub>B</sub> cluster (*a*; *Messor barbarus*) and a species with serotonin immunoreactivity in this region (*b*; *Ectatomma tuberculatum*). The glomeruli are stained with hydrazide-conjugated dye displayed in magenta, and immunolabelled serotonergic projections are displayed in green. The AL subregions are outlined with dashed lines. (c) 3D reconstruction of glomerular volumes and antennal lobe regionalization in *Neoponera apicalis*. The main-AL, which receives serotonergic innervation, is shown in green, while the T<sub>B</sub> cluster, which lacks such innervation in some species, is in magenta. (d) Variation in serotonergic innervation across ant species summarized on the phylogeny. Green dots indicate serotonergic innervation, while empty circles in the T<sub>B</sub> cluster column represent the absence of such projections. Losses of serotonergic projections, predicted by the most parsimonious scenario, are marked with empty circles on the tree branches. (e) Diagram summarizing serotonergic projections in the AL of ants with serotonergic T<sub>B</sub> clusters. In all species, the giant neuron innervating the AL (GAL, in blue) innervates the main-AL, while the deutocerebral projection neuron (DPN, in red) targets the dorsal cluster of glomeruli (black dashed line). In species with immunoreactive T<sub>B</sub> clusters, DPN extends an additional branch to innervate the T<sub>B</sub> glomeruli. Dashed lines represent structures that are dorsal relative to the AL. All scale bars represent 50 µm (r, rostral; c, caudal; m, medial; l, lateral).

proposed as a marker for the T<sub>B</sub> cluster [47]. Therefore, we investigated the innervation patterns of serotonergic neurons across the Formicidae, as a marker of functional and anatomical regionalization within the AL of ants.

We found that the T<sub>B</sub> cluster lacks serotonergic neurons in *A. sexdens* (electronic supplementary material, figure S2a), *M. barbarus* (figure 2a), *C. atratus*, *Aphaenogaster senilis* and *Camponotus aethiops* (electronic supplementary material, figure S2b), while the rest of their ALs exhibit clear innervation (summarized in figure 2d). Similarly, the T<sub>B</sub> cluster in *P. clavata* and *E. burchellii* appears to lack serotonergic fibres, although this observation is less certain owing to lower quality of our immunostaining replicates. By contrast, *Ectatomma tuberculatum* (figure 2b), *Neoponera apicalis* (electronic supplementary material, figure S2c), *Pseudomyrmex boopis* (electronic supplementary material, figure S2d), *Formica fusca*, *Lasius niger*, *Cataglyphis cursor* and *Azteca trigona* all exhibit clear serotonergic innervation in the T<sub>B</sub> cluster (as summarized in figure 2d), with varying densities across both the main-AL and the T<sub>B</sub> glomeruli.

We further traced and examined the innervation pattern of serotonin-immunoreactive neurites within the two subregions of the AL (figure 2e). In all species, the glomeruli of the main-AL primarily receive projections from a neuron known as the giant neuron (GAL), which originates in the suboesophageal zone and innervates the AL [23]. Additionally, the soma of another neuron, known as the deutocerebral projection neuron (DPN), is consistently observed within the lateral cell cluster near the rostral edge of the AL [23]. In all species, this DPN innervates a few glomeruli in the dorsal region of the main-AL and sends ipsilateral projections towards the mushroom bodies (figure 2e). However, in T<sub>B</sub> immunoreactive species, we observed that the DPN exhibits an additional branch that specifically innervates the T<sub>B</sub> cluster. Therefore, the absence of serotonergic innervation in T<sub>B</sub> clusters correlates with the loss of this DPN extension, which can serve, along with morphological characteristics, as a key anatomical feature for characterizing T<sub>B</sub> across species.

### (c) Volumetric relationships in the ant antennal lobes

Using 3D models reconstructed from confocal image stacks (figure 2c), we measured the volume of the AL as the total glomerular area and found strong differences across species (electronic supplementary material, table S1 and figure S2e; Kruskal–Wallis test, $\chi^2 = 38.9$, d.f. = 14, *p* < 0.001). This is illustrated by the 20-fold difference in AL size between *L. niger* (0.69 × 10<sup>6</sup> µm<sup>3</sup>) and *P. clavata* (12.1 × 10<sup>6</sup> µm<sup>3</sup>), likely reflecting their substantial difference in body size. We also investigated neural investment
<figure><img src="image_8.png" /><figcaption>**Figure 3.** Evolutionary dynamics in the ant antennal lobe. (a) Volume of the T<sub>B</sub> cluster plotted against the volume of the main-AL across different ant species. There is a significant correlation between the volumes of the T<sub>B</sub> cluster and of the main-AL (t = 15.8, ***p < 0.001). (b) Mean volume of glomeruli in the T<sub>B</sub> cluster plotted against the mean volume of glomeruli in the main-AL across different species. There is a correlation between the mean glomerular volumes in the two subregions of the AL (t = 33.9, ***p < 0.001), with T<sub>B</sub> glomeruli being consistently smaller than those of the main-AL. (c) Number of glomeruli in the T<sub>B</sub> cluster plotted against the number of glomeruli in the main-AL. These counts are not correlated (t = 0.70, p = 0.48). However, when excluding outlier species (squares), the counts significantly correlate (t = 6.61, **p < 0.01). The red star on the graph represents the putative position of ant's most recent common ancestor (see text and electronic supplementary material, figure S3a-c). (d) Residual evolutionary rates of the number of glomeruli in the T<sub>B</sub> cluster, relative to the main-AL, mapped onto the branches of the phylogenetic tree of the sampled species. The branches leading to *Eciton burchellii* and, to a lesser extent, *Pseudomyrmex boopis*, display high evolutionary rates, suggesting two independent expansions of the T<sub>B</sub> cluster. Numbers indicated at the nodes represent the inferred ancestral state estimates for the number of glomeruli in both the main-AL and the T<sub>B</sub> cluster. (e) Plot showing the number of glomeruli, similar to (c), with species categorized according to the number of compound classes (alkanes, alkenes, mono-, di- or tri-methyl alkanes) within their CHC profiles. There is a significant effect of the number of CHC classes on the proportion of glomeruli between the T<sub>B</sub> cluster and the main-AL (*p*<sub>MCMC</sub> < 0.01).</figcaption></figure>

in the two AL subdivisions, which are believed to serve different functions, by examining the scaling relationship between the volume of the main-AL and that of the T<sub>B</sub> cluster. Despite considerable total volumetric variation, we found a significant correlation between the volumes of the main-AL and that of the T<sub>B</sub> cluster (figure 3a; Pearson test, t = 15.8, *p* < 0.001, R<sup>2</sup> = 0.86). This indicates that neuronal investment in the T<sub>B</sub> cluster follows a relatively stable allometric relationship across ant species (electronic supplementary material, table S1).

We examined the relationship between the mean glomerular size within both subregions, by dividing each subregion's volume by its number of glomeruli. Our analysis revealed a significant correlation between the volumes of glomeruli in the T<sub>B</sub> cluster and those in the main-AL (figure 3b; Pearson test, t = 33.9, *p* < 0.001, R<sup>2</sup> = 0.97). The slope (0.65 [0.607, 0.684] with 95% CI) is significantly lower than 1 (t-test, t = -32.3, *p* < 0.001), with a negative y-intercept. This indicates that, across species, glomeruli of the T<sub>B</sub> cluster are consistently smaller than those of the main-AL.

### (d) Glomerular counts and evolutionary dynamics in the ant antennal lobes

The glomerular count offers a measure of the AL's computational power, with more glomeruli suggesting a greater capacity to integrate and discriminate diverse olfactory information through combinatorial processing [48]. Across our sample set, the number of glomeruli in the AL of worker ants varied significantly across species (electronic supplementary material, table S1 and figure S3a; Kruskal-Wallis test, $\chi^2$ = 38.5, d.f. = 14, *p* < 0.001), ranging from 227 ± 10 (mean ± s.d.) glomeruli in *C. cursor* to 543 ± 13 glomeruli in *N. apicalis*. These differences are evident within each AL subdivision, with significant variation in the number of glomeruli in both the T<sub>B</sub> cluster (electronic supplementary material, table S1 and figure S3b,c; Kruskal-Wallis test, $\chi^2$ = 38.9, d.f. = 14, *p* < 0.001) and the main-AL (electronic supplementary material, table S1 and figure S3b,c; Kruskal-Wallis test, $\chi^2$ = 39.4, d.f. = 14, *p* < 0.001).
We further examined whether ants invest differently in the two AL subdivisions by studying the scaling relationship between the number of glomeruli in the T<sub>B</sub> cluster and the main-AL. These counts are not correlated across all ants (Pearson test, t = 0.70, *p* = 0.48), owing to notable deviations of a few species (figure 3c). We therefore analysed the evolutionary rate of glomerular numbers using a variable-rate Brownian motion model across the ant phylogeny [49]. The tree was rooted using additional glomerular counts from the paper wasp *Polistes dominula* (electronic supplementary material, figure S2f), the emerald cockroach wasp, *Ampulex compressa* (electronic supplementary material, figure S2g), and the honeybee *Apis mellifera* ([50], see electronic supplementary material, Methods). We generated separate models for the two subregions, and displayed the T<sub>B</sub> cluster residuals regressed against the main-AL along the tree to identify deviations in the allometric relationship between these subregions (figure 3d). Our analysis revealed a higher evolutionary rate on the branches leading to *E. burchellii* and, to a lesser extent, *P. boopis*, compared with the rest of the tree. These two species are notable outliers, with their T<sub>B</sub> clusters comprising 63.3 and 50.8% of the total AL glomeruli, respectively. Excluding these species, the glomeruli numbers between the two AL subdivisions show a significant correlation (figure 3c; Pearson test, t = 6.61, *p* < 0.01, R<sup>2</sup> = 0.56). This suggests that the variation in glomerular numbers is allometrically consistent between AL compartments, except in *E. burchellii* and *P. boopis*, which exhibit a disproportionately high number of T<sub>B</sub> glomeruli. Incorporating literature data on *Ooceraea biroi* [11] and *Dolichoderus* sp. [51], we corroborate this trend and provide support for an expanded T<sub>B</sub> cluster in the Dorylinae subfamily (electronic supplementary material, figure S3d,e).

We finally used maximum likelihood ancestral state estimation to reconstruct the evolutionary history of the AL across ants. The global model predicted approximately 367 AL glomeruli in the most recent common ancestor (MRCA) of ants (electronic supplementary material, figure S3a). Specifically, the analysis predicted 137 glomeruli for the T<sub>B</sub> cluster (figure 3d and electronic supplementary material, figure S3b) and 242 for the main-AL (figure 3d and electronic supplementary material, figure S3c) in independent models. Contrary to previous conclusions [11,37], our findings suggest that the T<sub>B</sub> subsystem was already elaborated before the extensive taxonomic radiation of ants. Furthermore, the model estimated 142 T<sub>B</sub> glomeruli in the MRCA of Dorylinae and other formicoids, and 133 T<sub>B</sub> glomeruli in the MRCA of *P. boopis* and the Dolichoderinae, indicating distinct expansions of T<sub>B</sub> glomeruli relative to the main-AL in these lineages (figure 3d and electronic supplementary material, figure S3b,d). These expansions were accompanied by a reduction in the number of other glomeruli (electronic supplementary material, figure S3c), distinguishing these species from the general trend.

(e) Social and chemical predictors of glomerular variation

We next aimed to elucidate the social and ecological correlates of the number of glomeruli in the T<sub>B</sub> cluster and the main-AL using phylogenetically controlled mixed models, which incorporate social and chemical predictors (electronic supplementary material, table S2). First, we analysed social predictors, including types of polygyny (strictly monogynous versus facultatively or obligatorily polygynous), foraging strategies (solitary versus collective) and colony sizes (categorized as fewer than 1000, between 1000 and 10 000, and more than 10 000). None of these factors had any significant effect on the number of glomeruli (total, T<sub>B</sub> or main-AL), the number of basiciconic sensilla, or volumetric measures of the AL (total, T<sub>B</sub> or main-AL) (see electronic supplementary material, Data file, tab mcmcGLMM). Likewise, these social factors did not affect the scaling of glomeruli between the T<sub>B</sub> cluster and the main-AL (electronic supplementary material, figure S4; polygyny: *p*<sub>MCMC</sub> = 0.388; foraging: *p*<sub>MCMC</sub> = 0.827; colony size: *p*<sub>MCMC</sub> = 0.386 and 0.387).

Next, we investigated whether variation in species-specific cuticular chemical profiles is related to these anatomical traits, using chemical analyses by gas chromatography coupled with mass spectrometry (see electronic supplementary material, §S1.7 Chemical analysis of CHC profiles). From each species' chemical profile, we extracted three key variables: the number of individual CHCs (CHC number), the number of distinct CHC classes (ranging from 1 to 5, including alkanes, alkenes, mono-, di- or tri-methyl alkanes) and a measure of profile complexity, the Shannon index (SI; see §4).

All models that included the SI showed it as a significant predictor of the number of glomeruli in the T<sub>B</sub> cluster (full model: posterior mean = -2.014, *p*<sub>MCMC</sub> = 0.005). Specifically, CHC profile complexity was negatively associated with the number of T<sub>B</sub> glomeruli, suggesting that species with less complex CHC profiles have more glomeruli in the T<sub>B</sub> cluster. However, the inclusion of the SI only marginally improved the model fit compared with the model in which it was excluded (ΔDIC = 0.25, where DIC is the deviance information criterion).

Since the SI accounts for both the number of CHC compounds and the number of classes, we explicitly tested these parameters in the model. While the number of compounds did not significantly influence the number of T<sub>B</sub> glomeruli (posterior mean = 0.0049, *p*<sub>MCMC</sub> = 0.595), the number of CHC classes was negatively associated with this count (posterior mean = -0.827, *p*<sub>MCMC</sub> = 0.045). This indicates that a lower diversity of CHC classes is associated with a higher number of T<sub>B</sub> glomeruli, although the improvement in DIC was minimal when comparing the full model with the one without CHC class (ΔDIC = 0.05). However, none of the chemical variables (SI, CHC number, CHC classes) was a significant predictor of glomeruli numbers in the main-AL or of the total glomerular count (for SI, main-AL: posterior mean = -0.373, *p*<sub>MCMC</sub> = 0.591; total: posterior mean = -0.873, *p*<sub>MCMC</sub> = 0.0976). These chemical factors also did not significantly predict the volume of the AL (total, T<sub>B</sub>, or main-AL) or the number and density of basiciconic sensilla. Investigating the scaling of glomeruli numbers between the T<sub>B</sub> cluster and the main-AL, we again found SI to be significantly associated with investment in both olfactory regions (posterior mean = -1.924, *p*<sub>MCMC</sub> = 0.0100).

In conclusion, models including chemical predictors were the most parsimonious and best-fitting, though they showed only a slight DIC improvement (ΔDIC < 1). Specifically, the chemical variables, SI and the number of CHC classes emerged as significant predictors of ants' investment in glomeruli numbers in the T<sub>B</sub> cluster.
<figure><img src="image_9.png" /><figcaption>**Figure 4.** Comparative analysis of nestmate discrimination abilities. (a) Boxplots show median, quartiles and minimum/maximum values (whiskers) for aggression index. All investigated species discriminate nestmate from non-nestmate odours (***p < 0.001, *p < 0.01). (b) Boxplots represent median, quartiles and minimum/maximum values (whiskers) for delta aggression scores. Delta scores were calculated by subtracting the aggression index towards nestmates from that towards non-nestmates. Positive scores indicate more aggression towards non-nestmates, while negative scores indicate more aggression towards nestmates. The species were equally efficient at discriminating between nestmate and non-nestmate stimuli (NS: *p* > 0.05). In each boxplot, black circles indicate outliers. *n* = 30 individual ants for each species.</figcaption></figure>

### (f) Neuroanatomy's impact on ants' discrimination performance

Building on the interspecific differences we found in serotonergic innervation and glomerular number, we tested whether this variation reflects differences at the behavioural level. We conducted a nestmate discrimination assay on a selected set of species: *M. barbarus* and *F. fusca*, which share approximately the same number of glomeruli in both subregions, and *L. niger*, which has a lower number of glomeruli in the T<sub>B</sub> cluster compared with the other two species. Additionally, the T<sub>B</sub> clusters of *F. fusca* and *L. niger* receive serotonergic innervation, while it is absent in *M. barbarus*. Workers of all these three species were able to perform a discrimination task, consistently showing higher aggression levels towards non-nestmates than towards nestmates (figure 4a; *F. fusca*: χ<sup>2</sup> = 18.2, d.f. = 1, *p* < 0.001; *L. niger*: χ<sup>2</sup> = 5.52, d.f. = 1, *p* = 0.0188; *M. barbarus*: χ<sup>2</sup> = 13.0, d.f. = 1, *p* < 0.001). Overall, we found that the three species were similarly efficient at discriminating between nestmate and non-nestmate stimuli, with differentiation scores that did not significantly differ among species (figure 4b; χ<sup>2</sup> = 0.098, d.f. = 2, *p* = 0.95).

## 3. Discussion

In this study, we investigated anatomical variation within the olfactory system in a subset of ant species, encompassing all major taxonomic lineages of the Formicidae, to explore the evolution and adaptation of their CHC-related subsystem. We observed significant variation in olfactory traits, including the number of basiconic sensilla, as well as the volume and number of glomeruli in the antennal lobes (ALs). Despite this variation, the olfactory systems of these ant species exhibit a common ground plan characterized by a higher proportion of basiconic sensilla on the distalmost segments of the antennae, and consistent proportions of neuropil volume and glomeruli number between AL subregions. However, we also identified remarkable outliers, demonstrating a high evolutionary rate of T<sub>B</sub> glomeruli within specific subfamilies, resulting in the expansion of the T<sub>B</sub> cluster relative to the main-AL. Nonetheless, contrary to previous assumptions [11,19,37], ancestral traits reconstruction suggests that the olfactory system was already sophisticated in the most recent common ancestor (MRCA) of ants, with possibly 367 glomeruli, including 142 within the CHC-related subsystem. In considering ecological and chemical factors that might influence this evolution, the complexity of ants' CHC profiles, particularly CHC class diversity, emerged as a potential predictor of variation in glomerular investment, albeit with modest explanatory power. Nonetheless, neither the relative nor absolute glomerular investment appears to influence nestmate discrimination performance across species. Thus, our results suggest that ants possessed a sophisticated olfactory system prior to their taxonomic radiation, and subsequent evolution was likely shaped by the necessity to maintain high olfactory performance across diverse ecological conditions and life history strategies.

### (a) An olfactory subsystem for hydrocarbon sensing in ants

Ants universally bear a wide diversity of CHCs on their body surface, which serve as efficient recognition cues [52]. These compounds play a pivotal role in social communication, especially in nestmate recognition, which is crucial for preventing exploitation by competitors and parasites, and maintaining colony cohesion. Given that this task is typically carried out by
female workers, the absence of basiconic sensilla and T<sub>B</sub> glomeruli in males hints at their specialized function in detecting recognition cues [25,53–55]. This idea is further supported by electrophysiological studies demonstrating the role of basiconic sensilla OSNs [29,31,33], as well as expression-biased ORs of the 9-exon clade [11,56], in responding to CHCs [26,27]. While this subsystem has been described in a handful of species spanning only three Formicidae subfamilies (*Camponotus* spp. [24,55], *A. vollenweideri* [20], *O. biroi* [11], *E. burchellii* [12]), the present study provides an evolutionary perspective across the ant phylogeny, showing the presence of this subsystem in all eight major Formicidae clades. Beyond ants, a similar pattern of CHC-sensitive basiconic sensilla exclusively projecting into the equivalent T<sub>B</sub> cluster has been described in hornets [47]. In honeybees, studies suggest that the putative equivalent of the T<sub>B</sub> cluster (the T3 cluster) is predominantly innervated by basiconic sensilla OSNs but also receives inputs from sensory neurons housed in other sensilla types [46]. Whether this pattern of multi-sensilla input to T<sub>B</sub> glomeruli applies to ants and wasps remains unknown, and further investigation is needed to elucidate its functional role in CHC perception.

Overall, we found the presence of basiconic sensilla on the antennae of all species, with higher density of basiconic sensilla on the distal segments, despite varying quantities across species. This observation is consistent with previous findings [11,54,57,58], supporting the role of basiconic sensilla as close-range chemoreceptors, as the distal segments of the antennae come into close proximity to other individuals during antennation [59].

Remarkably, all species also exhibited a distinct separation between the main-AL and the T<sub>B</sub> cluster, forming two glomerular rings on optical sections, with the T<sub>B</sub> cluster characterized by a congregation of small, uniformly sized glomeruli. The relative size of glomeruli typically reflects the quantity of incoming OSNs, which in turn correlates with detection sensitivity to associated odourants [60–62]. As such, the smaller size of T<sub>B</sub> glomeruli, indicating fewer OSNs inputs, likely reflects reduced sensitivity compared with the glomeruli of the main-AL. Consequently, T<sub>B</sub> glomeruli might be well adapted for close-range detection but less effective in detecting more dispersed environmental odours. Selection pressures on the specific ability to perceive CHC profiles may therefore shape the number of T<sub>B</sub> glomeruli rather than their size, emphasizing discrimination over sensitivity.

## (b) Evolutionary dynamics of ant olfactory systems

The number of OR genes and the corresponding glomerular count in the AL are believed to reflect a species' olfactory discrimination power, as odours are processed through the combinatorial activity of OSN populations [48]. Ants demonstrate extensive variation in glomeruli numbers, from 198 in *Cataglyphis fortis* [63] to 630 in *Apterostigma* cf. *mayri* [10], which is at least 3–4 times higher than in most holometabolous insects whose olfactory pathway has been described, such as *Drosophila melanogaster* with 52 [64], *Aedes aegypti* with 50 [65] or *Manduca sexta* with 65 [66]. This high number is believed to heighten ants' olfactory discrimination abilities, supporting their prominent collective behaviours [40]. Nonetheless, such substantial glomerular counts might have either expanded in response to increasing social complexity or, conversely, been a pre-existing feature that facilitated the radiation of social lineages [2,67]. Our ancestral state reconstructions of glomerular number suggest that an estimated 383 glomeruli were already present in the MRCA of ants, supporting the latter hypothesis. From this ancestral state, our analyses suggest that the number of glomeruli evolved dynamically to adapt to the ecological conditions of different clades, particularly the variety of chemical cues present in their environment, rather than social factors such as polygyny, colony size or foraging strategy.

The specific family of 9-exon ORs has been highlighted as particularly expanded in ants, and showing signatures of positive selection [11,19,37], suggesting that dynamic gene family evolution has accompanied the evolution of ant sociality. Additionally, several pieces of evidence hint that the size of the 9-exon OR repertoire and the associated number of glomeruli within the T<sub>B</sub> cluster may adapt to varying social traits. For example, species that independently evolved social parasitism, resulting in the loss of important social traits, showed a convergent loss of ORs, particularly within the 9-exon subfamily [68,69]. As such, it was anticipated that glomeruli number (especially in the T<sub>B</sub> cluster) would also show correlated evolution with social traits.

hints of this association within the AL have also been reported, with closely related species of *Dolichoderus* ants exhibiting a higher number of T<sub>B</sub> glomeruli with larger colony size [51]. However, at a broader taxonomic scale, variation in the T<sub>B</sub> cluster is not significantly associated with any of the social traits we measured, including colony size. Moreover, our ancestral trait reconstruction predicts a high number of T<sub>B</sub> glomeruli in the MRCA of ants, contradicting previous suggestions of a specific expansion of the CHC-related subsystem in ants [11,37]. These results rather align with later studies showing a very high number of 9-exon ORs in non-social apoid wasps outside the Formicidae family [70], suggesting that a well developed CHC-sensitive subsystem was conserved across these lineages and existed before the diversification of ants. In solitary parasitic wasps, particularly those with host-specific behaviours, females must navigate complex chemical cues during interactions with both their hosts [71] and their kleptoparasites (e.g. cuckoo wasps [72]). Similar to ants, these wasps may rely on a sophisticated CHC-sensing system, shaped by their parasitic lifestyle and competitive interactions.

Recent studies have highlighted the crucial role of serotonin in modulating olfactory behaviour across various species. In *D. melanogaster*, serotonin signalling is essential for social attraction, influencing social behaviour through the integration of sensory cues [73]. In moths, serotonin modulates sensitivity to sex pheromones, with its levels fluctuating according to the day-night cycle [74,75]. In *A. mellifera*, serotonin inhibits appetitive olfactory learning, affecting the processing of olfactory memories [76–78]. In ants, we suggest that serotonergic innervation of the entire AL is ancestral to Formicidae, with distinct serotonergic neurons innervating the main-AL and T<sub>B</sub> cluster. Notably, the branch innervation of the T<sub>B</sub> cluster has been repeatedly lost in different ant subfamilies. This functional specialization may reflect the differentiation of the subregions, but the exact significance of this loss for olfactory modulation remains to be elucidated.
(c) Ecological correlates of olfactory adaptation in ants

When assessing the scaling relationship of AL subsystems as an indicator of relative investment in perceiving distinct sets of
olfactory stimuli, we observed a general covariation suggesting that ants generally maintain a consistent scaling relationship
between the T<sub>B</sub> cluster and the main-AL. However, *P. boopis* and Dorylinae species exhibit a notable deviation from this trend,
characterized by a disproportionately high number of T<sub>B</sub> glomeruli. This drastic expansion has been linked with predatory
myrmecophagy, nomadism and cyclical reproduction within the Dorylinae [12]. By contrast, *Pseudomyrmex* are herbivorous and
omnivorous ants, residing in moderately sized colonies [79], predominantly in arboreal environments, which does not align
with these interpretations. In addition to prey-related chemical cues, long-chain hydrocarbons from food sources such as pollen,
and seed elaiosomes may also be detected by the T<sub>B</sub> subsystem. Consequently, foraging strategies that involve detecting these
lipophilic odours could also have influenced the evolution of the T<sub>B</sub> cluster in non-predatory species.

Testing ecological factors in phylogeny-controlled generalized linear mixed models (GLMMs), we found evidence of a potential role of CHC complexity in influencing the relationship between the T<sub>B</sub> cluster and the main-AL. Species with a lower diversity of CHC classes in their chemical profiles exhibit a higher number of glomeruli in the T<sub>B</sub> cluster. This correlation implies that the sophistication of the olfactory system may compensate for a reduced discriminative value in chemical signatures, thereby enabling accurate recognition among a smaller set of informative compounds within the CHC profile. Our behavioural experiments further support this hypothesis by demonstrating that despite variation in the number of glomeruli in the T<sub>B</sub> cluster, different species perform similarly in simple nestmate discrimination tasks. Therefore, the complexity of CHC profiles likely interacts with the neuronal investment in the T<sub>B</sub> cluster to maintain consistent behavioural responses in nestmate recognition, despite interspecific variation in colony-specific CHC profiles. However, it is important to note that CHC complexity exhibited a relatively low explanatory power in our analyses. Hence, in addition to other ecological factors, future research should explore how the complexity of sensory systems and the chemical environment (CHC profile, but also nest and food odours) interact to drive adaptations within communication systems and influence social evolution.

Thus, ants' remarkable evolutionary radiation, which led to a wide array of social structures and life history traits, was supported by an already complex olfactory system. In most cases, this evolution maintained a strict balance between the main-AL, thought to be responsible for detecting general odourants, and the T<sub>B</sub> cluster, assumed to be specialized in recognizing social cues. However, recent deorphanization of certain 9-exon ORs challenges this strict functional separation, as these receptors respond not only to CHCs but also to more general odourants [26,27]. Despite these insights, how these two AL regions process olfactory information at the neuronal level remains largely unexplored. Future central electrophysiological and imaging studies will be essential to further investigate potential functional specialization in odour processing between the two clusters and to clarify the significance of the T<sub>B</sub> cluster adaptations.

**4. Material and methods**

Detailed methods for sample preparation, imaging, behavioural assays, chemical analysis and statistical tests are provided in electronic supplementary material, Methods. In brief, 14 ant species from eight subfamilies were studied, including nine species from lab rearing and five collected in the wild. In all cases, ant workers were anaesthetized on ice before heads and antennae were separated, and bodies were stored in 95% ethanol. To investigate phenotypic variation of the olfactory system, antennae were preserved in 2.5% glutaraldehyde at 4°C, and brains were dissected and preserved in methanol, following procedures in electronic supplementary material, Methods. In all but one species, antennal sensilla were analysed via scanning electron microscopy, with standard sample preparation and scanning procedures. Antennal lobe neuroanatomy was investigated using immunostaining and confocal laser scanning microscopy. Behavioural assays for nestmate recognition in *L. niger*, *M. barbarus* and *F. fusca* were recorded and scored for aggression. Chemical analyses of CHC profiles were performed via gas chromatography coupled with mass spectrometry (GC-MS). Phylogenetic analyses used branch lengths inferred from phylogenomic studies [80–82]. Data analysis was performed using R and established statistical packages, with phylogenetic comparisons and evolutionary analyses exploring relationships between antennal morphology, neuroanatomy, chemical complexity and behaviour. All specific details of materials used, such as reagents, animal models or software are detailed in electronic supplementary material, Methods.

**Ethics.** The study was conducted following ethical practices for insect research, ensuring humane handling and minimizing harm. For wild species, the samples were exported under permit PA-01ARB-083-2022.

**Data accessibility.** All datasets supporting this study are publicly available in the online electronic supplementary material [83] and the Dryad Digital Repository [43].

**Declaration of AI use.** We have used AI-assisted technologies in creating this article.

**Authors' contributions.** M.S.: formal analysis, investigation, visualization, writing—original draft, writing—review and editing; A.C.: conceptualization, formal analysis, funding acquisition, investigation, methodology, supervision, visualization, writing—original draft, writing—review and editing; E.H.D.: formal analysis, investigation, methodology, visualization, writing—review and editing; N.B.: investigation; P.d'E.: conceptualization, funding acquisition, methodology, project administration, resources, supervision, writing—review and editing; S.H.M.: conceptualization, funding acquisition, methodology, project administration, resources, supervision, validation, writing—review and editing; J.C.S.: conceptualization, data curation, funding acquisition, methodology, project administration, resources, supervision, validation, writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed herein.
**Conflict of interest declaration.** We declare we have no competing interests.

**Funding.** This research was supported by the Leverhulme Trust (RPG-2019-287) and the French National Research Agency (ANR-20-CE02-0012).

**Acknowledgements.** We gratefully acknowledge Fabrice Savarit for providing access to *Ectatomma tuberculatum* and Jonathan Romiguier for supplying the time-calibrated phylogeny of ants. We also thank Chloé Leroy for assistance with the chemical analysis (GC-MS). We are grateful to the Wolfson Bioimaging Centre, University of Bristol, and Judith Mantell for their expertise in SEM imaging. We thank the Smithsonian Tropical Research Institute of Barro Colorado (Panama) for granting access to their facilities and permitting the collection of wild species.

## References

1. Freeberg TM, Dunbar RIM, Ord TJ. 2012 Social complexity as a proximate and ultimate factor in communicative complexity. *Phil. Trans. R. Soc. B* **367**, 1785–1801. (doi:10.1098/rstb.2011.0213)

2. Nehring V, Steiger S. 2018 Sociality and communicative complexity: insights from the other insect societies. *Curr. Opin. Insect Sci.* **28**, 19–25. (doi:10.1016/j.cois.2018.04.002)

3. Peckre L, Kappeler PM, Fichtel C. 2019 Clarifying and expanding the social complexity hypothesis for communicative complexity. *Behav. Ecol. Sociobiol.* **73**, 11. (doi:10.1007/s00265-018-2605-4)

4. Fichtel C, Kappeler PM. 2022 Coevolution of social and communicative complexity in lemurs. *Phil. Trans. R. Soc. B* **377**, 20210297. (doi:10.1098/rstb.2021.0297)

5. Couto A, Marty S, Dawson EH, d'Ettorre P, Sandoz J, Montgomery SH. 2023 Evolution of the neuronal substrate for kin recognition in social Hymenoptera. *Biol. Rev.* **98**, 2226–2242. (doi:10.1111/brv.13003)

6. Hölldobler B, Wilson EO. 1990 *The ants*. Cambridge, MA: Harvard University Press.

7. Ward PS. 2014 The phylogeny and evolution of ants. *Annu. Rev. Ecol. Evol. Syst.* **45**, 23–43. (doi:10.1146/annurev-ecolsys-120213-091824)

8. Wyatt TD. 2014 *Pheromones and animal behavior: chemical signals and signatures*. Cambridge, UK: Cambridge University Press.

9. Leonhardt SD, Menzel F, Nehring V, Schmitt T. 2016 Ecology and evolution of communication in social insects. *Cell* **164**, 1277–1287. (doi:10.1016/j.cell.2016.01.035)

10. Kelber C, Rössler W, Roces F, Kleineidam CJ. 2009 The antennal lobes of fungus-growing ants (Attini): neuroanatomical traits and evolutionary trends. *Brain Behav. Evol.* **73**, 273–284. (doi:10.1159/000230672)

11. McKenzie SK, Fetter-Pruneda I, Ruta V, Kronauer DJC. 2016 Transcriptomics and neuroanatomy of the clonal raider ant implicate an expanded clade of odorant receptors in chemical communication. *Proc. Natl Acad. Sci. USA* **113**, 14091–14096. (doi:10.1073/pnas.16160800113)

12. McKenzie SK et al. 2021 The genomic basis of army ant chemosensory adaptations. *Mol. Ecol.* **30**, 6627–6641. (doi:10.1111/mec.16198)

13. Trible W, Olivos-Cisneros L, McKenzie SK, Saragosti J, Chang NC, Matthews BJ, Oxley PR, Kronauer DJC. 2017 *orco* mutagenesis causes loss of antennal lobe glomeruli and impaired social behavior in ants. *Cell* **170**, 727–735. (doi:10.1016/j.cell.2017.07.001)

14. Yan H et al. 2017 An engineered *orco* mutation produces aberrant social behavior and defective neural development in ants. *Cell* **170**, 736–747. (doi:10.1016/j.cell.2017.06.051)

15. Yan H, Jafari S, Pask G, Zhou X, Reinberg D, Desplan C. 2020 Evolution, developmental expression and function of odorant receptors in insects. *J. Exp. Biol.* **223**, jeb208215. (doi:10.1242/jeb.208215)

16. Mika K, Benton R. 2021 Olfactory receptor gene regulation in insects: multiple mechanisms for singular expression. *Front. Neurosci.* **15**, 738088. (doi:10.3389/fnins.2021.738088)

17. Couto A, Alenius M, Dickson BJ. 2005 Molecular, anatomical, and functional organization of the *Drosophila* olfactory system. *Curr. Biol.* **15**, 1535–1547. (doi:10.1016/j.cub.2005.07.034)

18. Zhao Z, McBride CS. 2020 Evolution of olfactory circuits in insects. *J. Comp. Physiol.* **206**, 353–367. (doi:10.1007/s00359-020-01399-6)

19. Zhou X, Rokas A, Berger SL, Liebig J, Ray A, Zwiebel LJ. 2015 Chemoreceptor evolution in Hymenoptera and its implications for the evolution of eusociality. *Genome Biol. Evol.* **7**, 2407–2416. (doi:10.1093/gbe/evv149)

20. Kelber C, Rössler W, Kleineidam CJ. 2010 Phenotypic plasticity in number of glomeruli and sensory innervation of the antennal lobe in leaf-cutting ant workers (*A. vollenweideri*). *Dev. Neurobiol.* **70**, 222–234. (doi:10.1002/dneu.20782)

21. Uebi T et al. 2022 Chemical identification of an active component and putative neural mechanism for repellent effect of a native ant's odor on invasive species. *Front. Physiol.* **13**, 844084. (doi:10.3389/fphys.2022.844084)

22. Dacks AM, Christensen TA, Hildebrand JG. 2006 Phylogeny of a serotonin-immunoreactive neuron in the primary olfactory center of the insect brain. *J. Comp. Neurol.* **498**, 727–746. (doi:10.1002/cne.21076)

23. Tsuji E, Aonuma H, Yokohari F, Nishikawa M. 2007 Serotonin-immunoreactive neurons in the antennal sensory system of the brain in the carpenter ant, *Camponotus japonicus*. *Zool. Sci.* **24**, 836–849. (doi:10.2108/zsj.24.836)

24. Zube C, Rössler W. 2008 Caste- and sex-specific adaptations within the olfactory pathway in the brain of the ant *Camponotus floridanus*. *Arthropod Struct. Dev.* **37**, 469–479. (doi:10.1016/j.asd.2008.05.004)

25. Nakanishi A, Nishino H, Watanabe H, Yokohari F, Nishikawa M. 2010 Sex-specific antennal sensory system in the ant *Camponotus japonicus*: glomerular organizations of antennal lobes. *J. Comp. Neurol.* **518**, 2186–2201. (doi:10.1002/cne.22326)

26. Pask GM et al. 2017 Specialized odorant receptors in social insects that detect cuticular hydrocarbon cues and candidate pheromones. *Nat. Commun.* **8**, 297. (doi:10.1038/s41467-017-00099-1)

27. Slone JD, Pask GM, Ferguson ST, Millar JG, Berger SL, Reinberg D, Liebig J, Ray A, Zwiebel LJ. 2017 Functional characterization of odorant receptors in the ponerine ant, *Harpegnathos saltator*. *Proc. Natl Acad. Sci. USA* **114**, 8586–8591. (doi:10.1073/pnas.1704647114)

28. d'Ettorre P, Deisig N, Sandoz JC. 2017 Decoding ants' olfactory system sheds light on the evolution of social communication. *Proc. Natl Acad. Sci. USA* **114**, 8911–8913. (doi:10.1073/pnas.1711075114)

29. Ozaki M, Wada-Katsumata A, Fujikawa K, Iwasaki M, Yokohari F, Satoji Y, Nisimura T, Yamaoka R. 2005 Ant nestmate and non-nestmate discrimination by a chemosensory sensillum. *Science* **309**, 311–314. (doi:10.1126/science.1105244)

30. Kidokoro-Kobayashi M, Iwakura M, Fujiwara-Tsuji N, Fujiwara S, Sakura M, Sakamoto H, Higashi S, Hefetz A, Ozaki M. 2012 Chemical discrimination and aggressiveness via cuticular hydrocarbons in a supercolony-forming ant, *Formica yessensis*. *PLoS One* **7**, e46840. (doi:10.1371/journal.pone.0046840)

31. Sharma KR et al. 2015 Cuticular hydrocarbon pheromones for social behavior and their coding in the ant antenna. *Cell Rep.* **12**, 1261–1271. (doi:10.1016/j.celrep.2015.07.031)

32. Ghaninia M, Haight K, Berger SL, Reinberg D, Zwiebel LJ, Ray A, Liebig J. 2017 Chemosensory sensitivity reflects reproductive status in the ant *Harpegnathos saltator*. *Scient. Rep.* **7**, 3732. (doi:10.1038/s41598-017-03964-7)
33. Watanabe H, Ogata S, Nodomi N, Tateishi K, Nishino H, Matsubara R, Ozaki M, Yokohari F. 2023 Cuticular hydrocarbon reception by sensory neurons in basiconic sensilla of the Japanese carpenter ant. *Front. Cell. Neurosci.* **17**, 1084803. (doi:10.3389/fncel.2023.1084803)

34. Blomquist GJ, Bagnères AG. 2010 *Insect hydrocarbons: biology, biochemistry, and chemical ecology*. Cambridge, UK: Cambridge University Press.

35. Oi CA, van Zweden JS, Oliveira RC, Van Oystaeyen A, Nascimento FS, Wenseleers T. 2015 The origin and evolution of social insect queen pheromones: novel hypotheses and outstanding problems. *BioEssays* **37**, 808–821. (doi:10.1002/bies.201400180)

36. d'Ettorre P, Lenoir A. 2009 Nestmate recognition. In *Ant ecology* (eds L Lori, P Catherine, A Kirsti), pp. 194–209. Oxford, UK: Oxford University Press. (doi:10.1093/acprof:oso/9780199544639.003.0011)

37. Engsontia P, Sangket U, Robertson HM, Sasatook C. 2015 Diversification of the ant odorant receptor gene family and positive selection on candidate cuticular hydrocarbon receptors. *BMC Res. Notes* **8**, 380. (doi:10.1186/s13104-015-1371-x)

38. van Wilgenburg E, Symonds MRE, Elgar MA. 2011 Evolution of cuticular hydrocarbon diversity in ants. *J. Evol. Biol.* **24**, 1188–1198. (doi:10.1111/j.1420-9101.2011.02248.x)

39. Sprenger PP, Menzel F. 2020 Cuticular hydrocarbons in ants (Hymenoptera: Formicidae) and other insects: how and why they differ among individuals, colonies, and species. *Myrmecol. News* **30**, 1–26. (doi:10.25849/myrmecol.news_030:001)

40. Ferguson ST, Bakis I, Zwiebel LJ. 2021 Advances in the study of olfaction in eusocial ants. *Insects* **12**, 252. (doi:10.3390/insects12030252)

41. Borowiec ML, Moreau CS, Rabeling C. 2020 Ants: phylogeny and classification. In *Encyclopedia of social insects* (ed. CK Starr), pp. 1–18. Cham, Switzerland: Springer International Publishing. (doi:10.1007/978-3-319-90306-4_155-1)

42. Keil TA. 1999 Morphology and development of the peripheral olfactory organs. In *Insect olfaction* (ed. BS Hansson), pp. 5–47. Berlin, Germany: Springer. (doi:10.1007/978-3-662-07911-9_2)

43. Marty S, Couto A, Dawson EH, Brard N, Montgomery SH, Sandoz JC. 2025 Data from: Ancestral complexity and constrained diversification of the ant olfactory system – antennal lobe imaging and 3D reconstructions dataset. Dryad Digital Repository. ()

44. Zube C, Kleineidam CJ, Kirschner S, Neef J, Rössler W. 2008 Organization of the olfactory pathway and odor processing in the antennal lobe of the ant *Camponotus floridanus*. *J. Comp. Neurol.* **506**, 425–441. (doi:10.1002/cne.21548)

45. Couto A, Lapeyre B, Thiéry D, Sandoz J. 2016 Olfactory pathway of the hornet *Vespa velutina*: new insights into the evolution of the Hymenopteran antennal lobe. *J. Comp. Neurol.* **524**, 2335–2359. (doi:10.1002/cne.23975)

46. Kropf J, Kelber C, Bieringer K, Rössler W. 2014 Olfactory subsystems in the honeybee: sensory supply and sex specificity. *Cell Tissue Res.* **357**, 583–595. (doi:10.1007/s00441-014-1892-y)

47. Couto A, Mitra A, Thiéry D, Marion-Poll F, Sandoz JC. 2017 Hornets have it: a conserved olfactory subsystem for social recognition in Hymenoptera? *Front. Neuroanat.* **11**, 48. (doi:10.3389/fnana.2017.00048)

48. Su CY, Menuz K, Carlson JR. 2009 Olfactory perception: receptors, cells, and circuits. *Cell* **139**, 45–59. (doi:10.1016/j.cell.2009.09.015)

49. Revell LJ, Collar DC. 2009 Phylogenetic analysis of the evolutionary correlation using likelihood. *Evolution* **63**, 1090–1100. (doi:10.1111/j.1558-5646.2009.00616.x)

50. Brand P, Ramírez SR. 2017 The evolutionary dynamics of the odorant receptor gene family in corbicate bees. *Genome Biol. Evol.* **9**, 2023–2036. (doi:10.1093/gbe/evx149)

51. Godfrey RK, Oberski JT, Allmark T, Givens C, Hernandez-Rivera J, Gronenberg W. 2021 Olfactory system morphology suggests colony size drives trait evolution in odorous ants (Formicidae: Dolichoderinae). *Front. Ecol. Evol.* **9**, 733023. (doi:10.3389/fevo.2021.733023)

52. Martin S, Drijfhout F. 2009 A review of ant cuticular hydrocarbons. *J. Chem. Ecol.* **35**, 1151–1161. (doi:10.1007/s10886-009-9695-4)

53. Nishikawa M, Nishino H, Misaka Y, Kubota M, Tsuji E, Satoji Y, Ozaki M, Yokohari F. 2008 Sexual dimorphism in the antennal lobe of the ant *Camponotus japonicus*. *Zool. Sci.* **25**, 195–204. (doi:10.2108/zzsj.25.195)

54. Nakanishi A, Nishino H, Watanabe H, Yokohari F, Nishikawa M. 2009 Sex-specific antennal sensory system in the ant *Camponotus japonicus*: structure and distribution of sensilla on the flagellum. *Cell Tissue Res.* **338**, 79–97. (doi:10.1007/s00441-009-0863-1)

55. Nishikawa M, Watanabe H, Yokohari F. 2012 Higher brain centers for social tasks in worker ants, *Camponotus japonicus*. *J. Comp. Neurol.* **520**, 1584–1598. (doi:10.1002/cne.23001)

56. Zhou X, Slone JD, Rokas A, Berger SL, Liebig J, Ray A, Reinberg D, Zwiebel LJ. 2012 Phylogenetic and transcriptomic analysis of chemosensory receptors in a pair of divergent ant species reveals sex-specific signatures of odor coding. *PLoS Genet.* **8**, e1002930. (doi:10.1371/journal.pgen.1002930)

57. Renthal R, Velasquez D, Olmos D, Hampton J, Wergin WP. 2003 Structure and distribution of antennal sensilla of the red imported fire ant. *Micron* **34**, 405–413. (doi:10.1016/S0968-4328(03)00050-7)

58. Gellert HR, Halley DC, Sieb ZJ, Smith JC, Pask GM. 2022 Microstructures at the distal tip of ant chemosensory sensilla. *Scient. Rep.* **12**, 19328. (doi:10.1038/s41598-022-21507-7)

59. Brandstaetter AS, Endler A, Kleineidam CJ. 2008 Nestmate recognition in ants is possible without tactile interaction. *Naturwissenschaften* **95**, 601–608. (doi:10.1007/s00114-008-0360-5)

60. Hansson BS, Christensen TA, Hildebrand JG. 1991 Functionally distinct subdivisions of the macroglomerular complex in the antennal lobe of the male sphinx moth *Manduca sexta*. *J. Comp. Neurol.* **312**, 264–278. (doi:10.1002/cne.903120209)

61. Wanner KW, Nichols AS, Walden KKO, Brockmann A, Luetje CW, Robertson HM. 2007 A honey bee odorant receptor for the queen substance 9-oxo-2-decenoic acid. *Proc. Natl Acad. Sci. USA* **104**, 14383–14388. (doi:10.1073/pnas.0705459104)

62. Kuebler LS, Kelber C, Kleineidam CJ. 2010 Distinct antennal lobe phenotypes in the leaf-cutting ant (*Atta vollenweideri*). *J. Comp. Neurol.* **518**, 352–365. (doi:10.1002/cne.22217)

63. Stieb SM, Kelber C, Wehner R, Rössler W. 2011 Antennal-lobe organization in desert ants of the genus *Cataglyphis*. *Brain Behav. Evol.* **77**, 136–146. (doi:10.1159/000326211)

64. Grabe V, Strutz A, Baschwitz A, Hansson BS, Sachse S. 2015 Digital *in vivo* 3D atlas of the antennal lobe of *Drosophila melanogaster*. *J. Comp. Neurol.* **523**, 530–544. (doi:10.1002/cne.23697)

65. Ignell R, Dekker T, Ghaninia M, Hansson BS. 2005 Neuronal architecture of the mosquito deutocerebrum. *J. Comp. Neurol.* **493**, 207–240. (doi:10.1002/cne.20800)

66. Huetteroth W, Schachtner J. 2005 Standard three-dimensional glomeruli of the *Manduca sexta* antennal lobe: a tool to study both developmental and adult neuronal plasticity. *Cell Tissue Res.* **319**, 513–524. (doi:10.1007/s00441-004-1016-1)

67. Taniguchi R, Grimaldi DA, Watanabe H, Iba Y. 2024 Sensory evidence for complex communication and advanced sociality in early ants. *Sci. Adv.* **10**, eadp3623. (doi:10.1126/sciadv.adp3623)

68. Jongepier E, Séguret A, Labutin A, Feldmeyer B, Gstöttl C, Foitzik S, Heinze J, Bornberg-Bauer E. 2022 Convergent loss of chemoreceptors across independent origins of slave-making in ants. *Mol. Biol. Evol.* **39**, msab305. (doi:10.1093/molbev/msab305)

69. Schrader L et al. 2021 Relaxed selection underlies genome erosion in socially parasitic ant species. *Nat. Commun.* **12**, 2918. (doi:10.1038/s41467-021-23178-w)

70. Obiero GF, Pauli T, Geuverink E, Veenendaal R, Niehuis O, Große-Wilde E. 2021 Chemoreceptor diversity in apoid wasps and its reduction during the evolution of the pollen-collecting lifestyle of bees (Hymenoptera: Apoidea). *Genome Biol. Evol.* **13**, a269. (doi:10.1093/gbe/evaa269)
71. Mouratidis A, Vacas S, Herrero J, Navarro-Llopis V, Dicke M, Tena A. 2021 Parasitic wasps avoid ant-protected hemipteran hosts via the detection of ant cuticular hydrocarbons. *Proc. R. Soc. B288*, 20201684. (doi:10.1098/rspb.2020.1684)

72. Castillo R, Wurdack M, Pauli T, Keller A, Feldhaar H, Polidori C, Niehuis O, Schmitt T. 2022 Evidence for a chemical arms race between cuckoo wasps of the genus *Hedychrum* and their distantly related host apoid wasps. *BMC Ecol. Evol.* **22**, 138. (doi:10.1186/s12862-022-02093-8)

73. Sun Y, Qiu R, Li X, Cheng Y, Gao S, Kong F, Liu L, Zhu Y. 2020 Social attraction in *Drosophila* is regulated by the mushroom body and serotonergic system. *Nat. Commun.* **11**, 5350. (doi:10.1038/s41467-020-19102-3)

74. Kloppenburg P, Mercer AR. 2008 Serotonin modulation of moth central olfactory neurons. *Annu. Rev. Entomol.* **53**, 179–190. (doi:10.1146/annurev.ento.53.103106.093408)

75. Gatellier L, Nagao T, Kanzaki R. 2004 Serotonin modifies the sensitivity of the male silkmoth to pheromone. *J. Exp. Biol.* **207**, 2487–2496. (doi:10.1242/jeb.01035)

76. Menzel R, Heyne A, Kinzel C, Gerber B, Fiala A. 1999 Pharmacological dissociation between the reinforcing, sensitizing, and response-releasing functions of reward in honeybee classical conditioning. *Behav. Neurosci.* **113**, 744–754. (doi:10.1037//0735-7044.113.4.744)

77. Mercer AR, Menzel R. 1982 The effects of biogenic amines on conditioned and unconditioned responses to olfactory stimuli in the honeybee *Apis mellifera*. *J. Comp. Physiol.* **145**, 363–368. (doi:10.1007/BF00619340)

78. Lai Y, Despouy E, Sandoz JC, Su S, de Brito Sanchez MG, Giurfa M. 2020 Degradation of an appetitive olfactory memory via devaluation of sugar reward is mediated by 5-HT signaling in the honey bee. *Neurobiol. Learn. Mem.* **173**, 107278. (doi:10.1016/j.nlm.2020.107278)

79. Greer JA, Moreau CS. 2021 Phylogenetic analysis and trait evolution of ant cocoons. *Insect Syst. Evol.* **53**, 60–77. (doi:10.1163/1876312x-bja10008)

80. Romiguier J *et al*. 2022 Ant phylogenomics reveals a natural selection hotspot preceding the origin of complex eusociality. *Curr. Biol.* **32**, 2942–2947. (doi:10.1016/j.cub.2022.05.001)

81. Blaimer BB, Brady SG, Schultz TR, Lloyd MW, Fisher BL, Ward PS. 2015 Phylogenomic methods outperform traditional multi-locus approaches in resolving deep evolutionary history: a case study of formicine ants. *BMC Evol. Biol.* **15**, 271. (doi:10.1186/s12862-015-0552-5)

82. Branstetter MG, Longino JT, Reyes-López JL, Brady SG, Schultz TR. 2022 Out of the temperate zone: a phylogenomic test of the biogeographical conservatism hypothesis in a contrarian clade of ants. *J. Biogeogr.* **49**, 1640–1653. (doi:10.1111/jbi.14462)

83. Simon M, Couto A, Dawson EHH, Brard N, d'Ettorre P, Montgomery SH *et al*. 2025 Supplementary material from: Ancestral complexity and constrained diversification of the ant olfactory system. Figshare. (doi:10.6084/m9.fishshare.c.7750306)