# The effect of experimental hybridization on cognition and brain anatomy: limited phenotypic variation and transgression in Poeciliidae

Catarina Vila-Pouca, Hannah de Waele, Alexander Kotrschal

► To cite this version:

Catarina Vila-Pouca, Hannah de Waele, Alexander Kotrschal. The effect of experimental hybridization on cognition and brain anatomy: limited phenotypic variation and transgression in Poeciliidae. 2022. [hal-04709498](https://hal.science/hal-04709498v1)

HAL Id: hal-04709498

https://hal.science/hal-04709498v1

Preprint submitted on 8 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY-SA 4.0 - Attribution - ShareAlike - International
License
# The effect of experimental hybridization on cognition and brain anatomy: limited phenotypic variation and transgression in Poeciliidae

Catarina Vila-Pouca*, Hannah De Waele, Alexander Kotrschal

Behavioural Ecology Group, Wageningen University & Research, Wageningen, The
Netherlands

<footnote>* corresponding author: catarinavilapouca@gmail.com</footnote>
**Abstract**

Hybridization can promote phenotypic variation and often produces trait combinations distinct from the parental species. This increase in available variation can lead to the manifestation of functional novelty when new phenotypes bear adaptive value under the environmental conditions in which they occur. While the role of hybridization as a driver of variation and novelty in traits linked to fitness is well recognized, it remains largely unknown whether hybridization can fuel behavioural novelty by promoting phenotypic variation in brain morphology and/or cognitive traits. To address this question, we investigated the effect of hybridization on brain anatomy, learning ability, and cognitive flexibility in first- and second-generation hybrids of two closely related fish species (*Poecilia reticulata* and *P. wingei*). Overall, we found that F1 and F2 hybrids showed intermediate brain morphology and cognitive traits compared to parental groups. Moreover, as phenotypic dispersion and transgression was low for both brain and cognitive traits, we suggest that hybridization is not a strong driver of brain anatomical and cognitive diversification in these Poeciliidae. To determine the generality of this conclusion, hybridization experiments with cognitive tests need to be repeated in other families.

**Keywords:**

Transgressive segregation; morphology; brain morphology; learning; cognitive flexibility; guppy
Introduction

In the last decade, studies on phenotypic diversification have established the role of hybridization as a
driver of trait variation that often leads to ecological and evolutionary innovations (Rieseberg et al.
2003a; Mallet 2007; Stelkens et al. 2009; Seehausen 2013). Through recombination of parental alleles,
hybridization can generate high levels of heritable variation in hybrids, including individuals that
express new trait combinations that exceed the range of both parental species (Lewontin and Birch
1966; Dittrich-Reed and Fitzpatrick 2013). When novel phenotypes bear adaptive value, hybrids may
persist and influence the evolutionary trajectories of parental species, for example through novel
adaptations or colonisation of underused niches (Mallet 2007; Seehausen 2013). Novel or transgressive
phenotypes are commonly defined as traits or combinations of traits that fall outside the range of
variation of parental species (Rieseberg et al. 2003b; Bell and Travis 2005; Stelkens et al. 2009).
Transgressive phenotypes are common in both plants and animals and have so far been demonstrated
for several fitness-related traits, including morphology (skull morphology in fish, Stelkens et al. 2009;
wing morphology in butterflies, Mérot et al. 2020), physiology (temperature tolerance in copepods,
Pereira et al. 2014), life-history (number and size of offspring in snails, Façon et al. 2008), and
behavioural traits (mating behaviour in fruit flies, Ranganath and Aruna 2003; foraging behaviour in
fish, Selz and Seehausen 2019; Feller et al. 2020). Different mechanisms have been proposed to explain
how recombination of parental genomes can result in novel traits (Rieseberg et al. 2003b; Bell and
Travis 2005; Stelkens et al. 2009; Thompson et al. 2021). Extreme hybrid phenotypes can arise in first-
generation (F1) hybrids, a phenomenon usually referred to as heterosis, through dominance of some loci
contributing to variation in the same trait or epistatic interactions of alleles at different loci, among
other mechanisms (Stelkens et al. 2009; Mérot et al. 2020; Thompson et al. 2021). More often,
however, increased phenotypic variation and extreme phenotypes occur in subsequent hybrid
generations. Possible sources of transgression include dominance/overdominance of some loci and
epistasis, as in F1 hybrids, but it may also arise from the action of complementary genes between
quantitative trait loci that carry alleles of opposing effects. When alleles have complementary additive
effects, F1 hybrids can only express intermediate phenotypes but effects may sum in the F2 and later
generations and lead to increased phenotypic variance and transgression, even if the mean hybrid
phenotype is intermediate between the parentals (Rieseberg et al. 2003b; Thompson et al. 2021). While
not classically considered a transgressive effect from a genetic point of view, hybrid novelty may also
be observed when hybrid traits fall between disjunct parental ranges or when individual traits do not
segregate outside the parental range but their combination results in functional transgression (Dittrich-
Reed and Fitzpatrick 2013; Holzman and Hulsey 2017; Thompson et al. 2021). Since these novel
phenotypes may perform new functions or facilitate expansion into new areas of functional space, they
also represent important variation with potential impacts on hybrid fitness and on the evolutionary
trajectory of the species. In addition to increasing heritable phenotypic variation, hybridization can also
relax the genetic constraints and covariance structures between certain traits, generating novel trait
combinations which may facilitate expansion into new areas of functional space (Selz et al. 2014).
Quantifying general patterns of phenotype expression in hybrids is therefore important to help clarify if,
and by which mechanism, phenotypic diversification may emerge from hybridization.

One way in which hybridization can influence the trajectories of species is through effects on
behavioural and cognitive phenotypes. Although hybrids may show maladaptive behaviours, such as
impaired foraging, compromised reproductive behaviours, or poor learning and memory (Linn et al.
2004; Bridle et al. 2006; Turissini et al. 2017; McQuillan et al. 2018; Pärssinen et al. 2020),
hybridization can also drive behavioural novelty and promote evolvability. For example, african cichlid
hybrids show higher feeding efficiency on novel food types, even though they have lower efficiency on
parental food types, indicating they may have an advantage in ecological contexts outside the parental
species' niches (Selz and Seehausen 2019). In addition, behavioural novelty can also arise indirectly from transgression in related traits, most notably in morphological traits. Hybrids of two Galapagos finches showed transgressive segregation of bill size and shape, which allowed them to forage on novel food items and become ecologically successful, ultimately leading to the formation of a new species (Lamichhaney et al. 2018). Similarly, transgression in jaw morphology of hybrids of two African cichlids is linked to a novel sand-sifting foraging behaviour, a specialized mode of feeding on a resource that neither parental species typically exploits (Feller et al. 2020). Similar to head morphology and associated foraging behaviours, one morphological trait that is tightly linked to behavioural phenotypes and that can have important fitness consequences is brain anatomy. Variation in overall brain size and in the relative size and function of different brain regions is ubiquitous in nature, and this variation often correlates with behavioural and cognitive traits. For example, larger overall brains or larger sizes of certain brain regions tend to positively correlate with spatial learning and memory, self-control, cognitive flexibility, and innovation and problem solving (Lefebvre et al. 2004; Deaner et al. 2007; Herculano-Houzel 2017; Buechel et al. 2018; Triki et al. 2022). In addition, both brain morphology and cognitive traits are associated with survival, reproduction, or colonisation success, among others (Dukas and Bernays 2000; Sol et al. 2005; Cole et al. 2012; Kotrschal et al. 2013; Madden et al. 2018). Therefore, any effects of hybridization on brain morphology and/or cognitive traits are likely to be important for hybrid ecological success and fitness. While the role of hybridization as a driver of variation and novelty in several morphological traits linked to fitness is well recognized, it remains largely unknown whether hybridization can fuel behavioural novelty by promoting phenotypic variation in brain morphology and/or cognitive traits.

To address this empirical gap, here we investigated the effect of hybridization on brain anatomy, learning ability, and cognitive flexibility in F1 and F2 hybrids of two closely related fish species (*Poecilia reticulata* and *P. wingei*). Our previous study of learning ability and cognitive flexibility in F1 females of this cross showed that female hybrids had slightly higher phenotypic dispersion, some hybrid individuals had transgressive trait scores, and the mean phenotype of one hybrid group deviated away from the axis of variation of the parentals, suggesting that hybridization may promote cognitive variation and generate new trait combinations (Vila Pouca et al. 2022). These results on F1 female hybrids provided an important first test of whether hybridization can promote cognitive variation, a prerequisite for hybrids to have cognitive innovation potential (Seehausen 2013; Selz and Seehausen 2019). However, a few questions were left unanswered. Since only F1 individuals were tested, we could not assess if the observed increased phenotypic variation was due to heterosis, and is therefore transient, or if hybrid phenotypic variation in cognitive abilities is observed in subsequent hybrid generations and may therefore be heritable. Furthermore, the observed asymmetry between the reciprocal crossings of F1 hybrids (only one hybrid group was mismatched with the parentals) suggested the contribution of uniparentally inherited genetic factors to the traits analysed. Yet, only females were tested, which are the homogametic sex in *Poecilia* guppies; examining if greater reciprocal asymmetries are observed in males compared to females is a vital test to assess whether asymmetries derive from uniparentally inherited factors. Finally, a few studies have demonstrated that behavioural novelty can emerge from transgression in morphological traits linked to fitness (Lamichhaney et al. 2018; Feller et al. 2020). Since brain morphology is tightly linked to behaviour and cognition, and both brain and cognitive traits have important fitness consequences, it is essential to include brain morphology analyses if we are to gain a complete understanding on whether hybridization may fuel behavioural novelty through effects on the brain and/or cognitive abilities. The current study therefore aimed to address these questions by testing both males and females of F1 and F2 fish and by examining not only learning and cognitive flexibility but also overall brain size and brain region anatomy. Using probability density estimation methods combined with an analytical geometry approach, we quantify patterns of phenotype expression
in hybrids and test if hybrid phenotypes are over-dispersed relative to parentals and/or tend to deviate
from parental mean phenotypes. If hybrids show poorer or intermediate brain and/or cognitive traits,
then these traits may represent a postzygotic isolating barrier against hybridization (Rice and McQuillan
2018; Rice 2020). It is possible that hybrids with intermediate traits are at a competitive disadvantage
with parentals since they have invested significant energy into trait development but without obtaining
the benefits they should provide, at least compared to the parental group with the most developed traits
(Niven and Laughlin 2008; Dunlap and Stephens 2016). However, if hybrid phenotypes are over-
dispersed relative to parents or deviate from parental mean phenotypes, then those hybrids may have
cognitive innovation potential (Seehausen 2013; Thompson et al. 2021). Additionally, we may expect
differences between F1 and F2 hybrids depending on the traits' genetic architecture, namely if heterosis
is the mechanism underlying phenotypic variation in F1 fish. Furthermore, we may expect brain
anatomy traits to be particularly affected by hybridization if covariation between traits is relaxed (Selz
et al. 2014; Johnson et al. 2015) and leads to a releasing of the energy trade-offs that constrains
variation in brain region sizes (Niven and Laughlin 2008; Kotrschal et al. 2013).

**Methods**

Parental species and hybrid breeding

Parental fish used for breeding derived from laboratory populations that were kept in identical
conditions at Wageningen University & Research. Common guppy (*Poecilia reticulata*) populations
descended from wild individuals from the upper Aripo river, Trinidad, and Endler's guppy (*P. wingei*)
populations descended from wild individuals from Cumaná, Venezuela in 2006. Experimental fish were
produced according to standard hybridization methods (Stelkens et al. 2009; Vila Pouca et al. 2022). In
brief, non-hybrid (parental) crosses and reciprocal first-generation (F1) hybrids (i.e., *P. reticulata*
females crossed with *P. wingei* males, and *P. wingei* females crossed with *P. reticulata* males) were
bred simultaneously, in January and August 2020. Second-generation (F2) hybrids were then obtained
by crossing F1 females and F1 males from stock populations. To ensure that focal F2 hybrids were
compared to parental individuals of the same age, F2 fish were bred simultaneously with a group of
common and Endler's guppies, in January 2021. All groups of fish were bred and raised in identical
conditions in a common garden environment and examined for brain morphology or cognitive traits at
approximately one year of age. In the F1 cohort, F1 hybrids and parental offspring were obtained each
from 12-16 families (Vila Pouca et al. 2022). In the F2 cohort, F2 hybrids were obtained each from 16
families and parental offspring each from 10-12 families. All aquaria were part of a recirculation
system, in which water chemistry, temperature, light conditions, and feeding regime were kept constant.
For further details on breeding and rearing, see Suppl. 1.1 and Vila Pouca et al. (2022). We will refer to
the parental species as R for *P. reticulata* and W for *P. wingei* and use these abbreviations for the
hybrid crosses as mother species × father species (i.e., R×W and W×R).

Ethics

The care and use of experimental animals complied with animal welfare laws, guidelines, and policies
as approved by the Animal Welfare Body (Instantie voor Dierenwelwijn, IvD) at Wageningen
University and Research and the Centrale Commissie Dierproeven (AVD10400202010625),
Netherlands.
## Brain morphology measurements

Fish from each sex and crossing group (n = 232; detailed sample size in Suppl.2) were euthanised in a water bath containing an overdose of 2-Phenoxyethanol. Lateral view photographs were taken with a Nikon DSLR camera immediately after euthanasia. Fish were then fixed with 4% paraformaldehyde in phosphate-buffered saline (PBS) for two weeks. The samples were then washed twice in PBS and stored at 4 °C pending dissections. Standard length of each fish (tip of the snout to the end of the caudal peduncle) was measured from photographs using *Fiji* (Schindelin et al. 2012). Whole brains were dissected out from each fish and photographed from a dorsal, right lateral, left lateral, and ventral view using a ZEISS SteREO Stemi 305cam microscope with integrated camera. Brains were stored in PBS and later weighed to the nearest 0.01 mg using a Mettler Toledo XPE105 analytical scale. The length, width, and height of six major brain regions (telencephalon, optic tectum, cerebellum, dorsal medulla, hypothalamus, and olfactory bulbs) were measured using *Fiji* (Schindelin et al. 2012). The volume of each brain region was then estimated using: V = (L × W × H)π/6 (White and Brown 2015). All dissections were performed by the same researcher and body size and brain measurements were taken blind to species group.

## Learning performance

Fish from each sex and crossing group (n = 278; detailed sample size in Suppl.2) were tested for associative learning and cognitive flexibility at approximately one year of age. Females were tested in a colour discrimination assay that tests associative learning and in a reversal learning assay that tests cognitive flexibility, using red and yellow as stimulus colours (Fig. 1a,b; Buechel et al. 2018; Fuss and Witte 2019; Vila Pouca et al. 2022). F1 and parental females were tested with this assay in a previous study; detailed methods and results are given in Vila Pouca et al. (2022), and those data are included here as part of this larger study. For the present study, F2 and parental females were tested in Sep-Oct. 2021. Since males have proven difficult to train with this protocol (unpublished pilot study; Fuss and Witte 2019), parental, F1, and F2 males were tested in this study in a different colour discrimination task (Fig1c,d) adapted from Laland and Williams (1997), in Nov-Dec. 2021. For both female and male assays, experimental fish were individually housed in the experimental tank. Each tank included a home compartment (45×20 cm) and an experimental compartment (15×20 cm) at the front of the tank, only accessible during training sessions through a guillotine door (Fig. 1a,c). Fish were confined to the home compartment outside of training sessions. The experimental compartment was visually isolated from neighbours to avoid social learning effects. To control for potential colour biases, fish were randomly assigned to either red or yellow as the correct stimulus, balanced across the crossing groups. Trials were run blind to species group.

### Pre-training – females

The goal of the pre-training was to train females (n=194) to dislodge a green disc to access a food reward (one defrosted *Artemia*) hidden in a hole underneath. During the first trials, the disc only partially covered the hole, leaving the reward exposed. We then gradually trained the fish to dislodge the disc by moving it from partially to fully covering the hole (over an average of 30 trials; additional details in Vila Pouca et al. (2022) and Suppl. 1.2). A total of 185 females (R, n=40; W, n = 40; F1 R×W, n=27; F1 W×R, n=29; F2 R×W, n=25; F2 W×R, n=24) successfully retrieved the food reward by dislodging the disc in all trials of the last two sessions and continued the experiment.
<figure><img src="image_2.png" /><figcaption>**Figure 1.** Schematic representation of the experimental tank and associative and reversal learning tasks for females (A,B) and males (C,D). For both experiments, the experimental tank consisted of a home compartment and a training compartment accessible through guillotine doors (A,C). Females were pre-trained to dislodge a green plastic disc to access a food reward and tested for associative and reversal learning using red and yellow discs (B). Males were pre-trained to access the experimental chamber and obtain a free food reward by swimming through a hole with a green edge and tested for associative and reversal learning using holes with red and yellow edges (D).</figcaption></figure>

## Associative learning – females

After being trained to retrieve a reward underneath a movable disc, females (n = 185) were tested in their ability to learn a colour association. In this task, females were given a choice between a red and a yellow disc, both concealing a food reward (to ensure fish could not be learning through olfactory cues). The disc assigned as the correct colour for that fish could be dislodged to reveal a reward; the incorrect colour disc was stuck in the hole with a plastic knob and could not be moved, preventing the fish from retrieving the food underneath. To control for potential side biases, we randomised the position (left/right) of the correct colour in each trial. Choice was recorded as the first disc the fish touched. The fish was given 1 min to dislodge the correct coloured disc and eat the reward. If fish made an incorrect choice, correction was allowed within 3 min. If the fish failed to make any choice within 1 min or to correct its choice within 3 min, we moved the rewarded disc 5 mm to the side to allow easy access to the food and ensure that all fish experienced the same number of reinforced trials throughout the experiment. Fish received daily sessions of three or four trials, excluding weekends. The learning criterion consisted of seven consecutive correct choices (significant according to a binomial probability). As soon as a female reached the learning criterion, with the constraint that it ran a minimum of 12 trials, the next phase commenced. If a female did not reach the learning criterion within 40 trials, it was excluded from further training (R, n=9; F1 R×W, n=1; F1 W×R, n=1; F2 R×W, n=3; F2 W×R, n=5).

## Reversal learning – females

After succeeding in the colour association task, the reversal learning task started. The procedure was the same except the reward contingency was reversed: fish previously trained on yellow were now trained
on red and vice-versa. Each female ran a minimum of 24 training trials and continued the reversal task until it reached 7 consecutive correct choices, up to a maximum of 60 trials.

## Pre-training – males

The goal of pre-training was to train males (n=84) to swim through a hole in a partition to access the experimental chamber and obtain a food reward. It is worth noting that due to laboratory space constraints, the sample size of males initially included in the assays (n=14 per group) is smaller than that of females. To familiarise the fish with obtaining food in the experimental chamber, we started by giving males three trials per day over two days where a guillotine door was lifted to give access to the experimental chamber and to a free food reward (a small portion of flake placed on the white tank floor). From the third day, when the guillotine door was lifted, the fish encountered a partition with a circular hole that gave them access to the experimental chamber. The edge of the hole was painted green to make the fish familiar with swimming through a coloured hole. We gradually reduced the size of the circular hole (from 5.5mm ø to 4.5mm ø to 3mm ø) over training trials (Fig. 1d). We performed 4 daily trials over eight days (with rest days on Thursday and Sunday; additional details in Suppl. 1.3). On the last day of pre-training, a total of 74 males (R, n=13; W, n = 14; F1 R×W, n=11; F1 W×R, n=11; F2 R×W, n=13; F2 W×R, n=12) successfully swam through the 3mm ø green hole and ate the reward within 3min in all trials of the last session, and therefore continued the experiment.

## Associative learning – males

After being trained to swim through a hole in a partition to access a food reward, males (n = 74) were tested in their ability to learn a colour association. In this task, when the guillotine door was lifted, males were given a choice between a hole with a red edge and a hole with a yellow edge to access the experimental chamber and obtain the food reward (Fig. 1d). Fish with red assigned as the correct colour were allowed to swim through the red hole, but not the yellow hole, which was blocked with a transparent acrylic sheet; the opposite was true for fish trained to yellow. To control for potential side biases, we randomised the position (left/right) of the correct coloured hole in each trial. Choice was recorded as the first hole the fish attempted to swim through. The fish was given 3 min to swim through the correct coloured hole and eat the reward. For incorrect trials, where fish kept pushing against the transparent sheet, correction was allowed within 5 min. If the fish failed to make any choice within 3 min or to correct its choice within 5 min, we covered the incorrect option with an opaque sheet and only the correct hole was visible; this ensured that all fish experienced the same number of reinforced trials throughout the experiment. Fish received daily sessions of four trials, excluding Thursdays and Sundays. The learning criterion consisted of seven consecutive correct choices, with the constraint that the fish ran a minimum of 16 trials. As a control for the presence of the transparent sheet blocking the incorrect option, we ran two daily test sessions as soon as each fish reached the learning criterion. Test sessions consisted of 5 unrewarded trials where both red and yellow holes were unblocked interspersed with 5 regular trials. Fish progressed into the reversal learning task after the test. If a male did not reach the learning criterion within 40 trials, it was excluded from further training (R, n=7; F1 R×W, n=6; F2 R×W, n=4; F1 W×R, n=4; F2 R×W, n=3; F2 W×R, n=4).

## Reversal learning – males

After succeeding in the colour association task, the reversal learning task started. The procedure was the same except the reward contingency was reversed: fish previously trained to swim through the red hole were now trained to the yellow hole and vice-versa. Each male ran the reversal task until it reached 7
consecutive correct choices, up to a maximum of 60 trials. One male (F1 W×R) was suddenly found dead in the housing compartment on day 5 of the reversal task; since it only ran 16 trials, it was excluded from the learning criterion analyses. At the end of the experiment, 80 males (12-14 individuals of each of the six groups) were euthanised and used as part of the brain morphology dataset. To keep exposure to the learning task similar across all individuals, fish continued running one trial per day after they succeeded the task. It is unlikely that brain morphology comparisons between species groups are influenced by learning effects since our sample is balanced over groups, and a previous study in guppies found that learning does not impact coarse brain plasticity (Fong et al. 2019).

## Statistical analysis

Statistical analyses were performed in R v.3.6.3 (R Core Team 2020). Code and model results are available in the Supplementary Material. Linear models were generated using ‘lme4’ (Bates et al. 2015). Model terms were tested for significance using the ‘Anova’ function in ‘car’ (Fox and Weisberg 2019). Comparisons with null models without interaction terms were done using the ‘anova’ function from ‘stats’ (R Core Team 2020). If species group was a significant predictor in the model, we assessed Tukey corrected multiple comparisons between species levels using the ‘glht’ function in ‘multcomp’ (Hothorn et al. 2008).

### Brain morphology

To achieve normality and linearize the allometries between response variables and predictors, we log-transformed body size (mm), brain weight (mg), and brain region volume (mm³). To test for species differences in relative brain size, brain weight was fit to a linear model with the predictors of species group, body size, sex, and all two-way interactions. Body size was fitted as a covariate to account for allometry and focus on species differences in relative brain size. To test for species differences in relative brain region volumes, each brain region volume was fit to a linear model with the predictors of species group, brain remainder volume, sex, and all two-way interactions. Brain remainder volume (total brain volume minus each region volume) was used instead of total brain volume to avoid statistical confounds resulting from the inclusion of the region being analysed in the covariate. For all models, two-way interactions were removed if they were not significant, not relevant for our study question, and decreased the explanatory power of the final model (AIC comparisons).

### Learning performance

To test for species differences in learning performance, we compared for each learning task (i) number of trials to reach learning criterion using a generalised linear model (Poisson distribution) with the predictors of species group and colour; and (ii) learning rate, i.e., probability of success per trial (correct = 1; incorrect = 0) using a generalised linear mixed-effect model (binomial distribution) with the predictors of trial number, species group, colour, and the interaction of trial number × species group and trial number × colour, as well as a random intercept and slope for fish identity to account for repeated observations of individual fish. We tested the significance of the random effects in models (ii) with likelihood ratio tests, by comparing models which culled the intercept or slope term to our final model. To examine if performance in the associative learning task had carryover effects on performance in the reversal task, we tested the inclusion of the predictor ‘trials to learn the associative learning task’ and its interaction with species group in models (i, ii) for the reversal task. For model (ii) in the reversal task, trial number was log-transformed to meet the assumption of linearity on the logit-scale. These analyses were run separately for females and males. Due to the low sample size of males that succeeded the
reversal learning task, we could not run model (i) to compare the number of trials to reach learning criterion. Additionally, for the test trials of male fish, a non-parametric one-sample Wilcoxon rank sum test was used to compare the observed proportion of correct choices per individual against a chance value of 0.5 (null hypothesis of learning the task using cues from the transparent sheet blocking incorrect option).

Phenotypic trait variation and transgression

To quantify phenotypic dispersion, identify transgressive hybrids, and compare the mean phenotypes of
species groups, we used a novel methodology developed to quantify functional transgression in
multidimensional phenotypes (Mérot et al. 2020; Thompson et al. 2021; Vila Pouca et al. 2022). For
brain morphology analyses, we ran an exploratory analysis to combine relative whole brain and brain
region traits in order to define the brain morphospace of each species; however, since brain traits are
highly correlated, we found it best to use raw data of a few traits. We chose to focus the analyses on
brain weight relative to body size and on telencephalon and optic tectum volume since these are the
brain regions most likely to be important neural correlates of cognitive abilities such as perception and
learning (Broglio et al. 2011; Triki et al. 2022). For cognitive performance analyses, we used individual
scores in the associative and reversal learning tasks (number of trials to learn), following Vila Pouca et
al. (2022). The number of trials to learn was log-transformed to approximate a continuous variable with
Gaussian distribution.

Kernel Density Estimation (KDE) clusters of trait space (in 3D for brain morphology and 2D for cognitive performance) were computed with the ‘kde’ function in ‘ks’ (Duong 2021), with the bandwidth estimated using a grid-search estimation. Phenotypic dispersion was estimated as the hypervolume of the KDE cluster that contains 95% of the individuals in the group (Mérot et al. 2020; Vila Pouca et al. 2022) with the ‘contourSizes’ function in ‘ks’. We considered hybrid phenotypes to be transgressive if their trait values fell outside the range of both parental species, i.e., outside the 95% KDE of both parents (Stelkens et al. 2009; Dittrich-Reed and Fitzpatrick 2013). For each hybrid group, we quantified the frequency and proportion of transgressive individuals using a custom-made function (attached as supplement for revisions; will be deposited on OSF on publication) based on the packages ‘ks’, ‘misc3d’, and ‘akima’ (Feng and Tierney 2008; Duong 2021; Akima and Gebhardt 2022). To test if the mean phenotypes of our hybrid groups deviate from a linear combination of the parental phenotypes (Thompson et al. 2021), we simulated hybrid phenotypes by randomly sampling a pair of individuals from each parental species and calculating the mean trait values of the parental pair. To approximate the sample size of our experimental groups, we generated 35 simulated hybrids for brain morphology and 25 simulated hybrids for cognitive performance and repeated this process 100 times. We then compared our experimental hybrids’ mean phenotype with that of simulated hybrids in ‘parental bias’, which captures deviation from the arithmetic mean of the parental phenotypes in the direction of either parent, and ‘phenotypic mismatch’, which captures deviation away from the line connecting parental mean phenotypes (Mérot et al. 2020; Thompson et al. 2021; Vila Pouca et al. 2022). The two metrics were estimated by analytical geometry calculations using a custom-made function (attached as supplement for revisions; will be deposited on OSF on publication).
## Results

Parental species and hybrids differ in some brain morphology traits

When comparing parentals and F1 hybrids, we found a strong allometry between brain weight and body length (n = 107, estimate = 1.17, s.e. = 0.11, F₁,₁₀₁ = 117.34, p < 0.0001) and no differences in allometry between species groups or sex (both interactions p > 0.05; Suppl. 3.1.1). After accounting for these effects, we found differences among parentals and F1 hybrids in relative brain size (F₃,₁₀₁ = 5.07, p = 0.003) with Endler's guppies (W) showing smaller relative brain size compared to common guppies, R (est. = -0.050, s.e. = 0.014, p = 0.003) and to F1 R×W (est. = -0.034, s.e. = 0.012, p = 0.025). The analysis of brain region volumes, after accounting for the allometry associated with brain remainder volume (all p < 0.0001, Suppl. 3.1.2), revealed no differences in allometry between parentals and F1 hybrids or by sex (interactions all p > 0.05; Fig. 2a,c) and no differences in relative brain region volume between species groups except for olfactory bulb volume (F₃,₁₀₁ = 2.80, p = 0.044), with Endler's guppies (W) showing larger relative olfactory bulbs compared to F1 R×W (F1 R×W – W: est. = -0.15, s.e. = 0.051, p = 0.026). This indicates that species differences in brain size were overall equally expressed over the different brain regions. We also found an effect of sex on some brain region volumes, with males having a larger relative optic tectum but smaller cerebellum, dorsal medulla, and hypothalamus than females (Suppl. 3.1.2; Fig. 2a,c).

When comparing parentals and F2 hybrids, we found a strong allometry between brain weight and body length that varied by sex (n = 125, est. = 0.88, s.e. = 0.24, F₁,₁₁₈ = 12.89, p < 0.001) but not between species groups (F₃,₁₁₂ = 0.77, p = 0.51; Suppl. 3.2.1). After accounting for these effects, we found no differences among parentals and F2 hybrids in relative brain size (F₃,₁₁₈ = 1.45, p = 0.23). The analysis of brain region volumes, after accounting for the allometry associated with brain remainder volume (all p < 0.01; Suppl. 3.2.2), revealed no differences in allometry between parentals and F2 hybrids (interactions all p > 0.05; Fig. 2b,d) or differences in allometry by sex except for telencephalon volume (sex × brain remainder volume, est. = 0.40, s.e. = 0.15, F₁,₁₁₈ = 7.23, p = 0.008; Suppl. 3.2.2). We found differences between species groups in telencephalon, optic tectum, and hypothalamus volume, with F2 W×R showing a larger relative telencephalon compared to parental common guppies, R (F2 W×R – R: est. = 0.033, s.e. = 0.011, p = 0.017) and smaller relative optic tectum compared to parentals R and W and to F2 R×W (F2 W×R – R: est. = 0.030, s.e. = 0.010, p = 0.006; F2 W×R – W: est. = 0.026, s.e. = 0.010, p = 0.035; F2 W×R – F2 R×W: est. = 0.021, s.e. = 0.010, p = 0.043), and with both F2 hybrids showing a larger relative hypothalamus compared to parental Endler's guppies, W (F2 R×W – W: est. = 0.049, s.e. = 0.015, p = 0.008; F2 W×R – W: est. = 0.047, s.e. = 0.015, p = 0.014; Suppl. 3.2.2; Fig. 2b,d).
<figure><img src="image_3.png" /><figcaption>**Figure 2.** Differences in brain region volume relative to brain remainder volume (total brain volume minus each region volume) between parentals and F1 females (A), parentals and F2 females (B), parentals and F1 males (C), and parentals and F2 males (D). Remainder brain volume (x axis) and brain region volume (y axis) are shown in log10 scales. Model predictions are plotted as the best fit line with 95% CI. Parental groups are given in shades of grey; F1 hybrids are given in shades of red; F2 hybrids are given in shades of blue.</figcaption></figure>

Parental species and hybrids do not differ in learning performance

Over 90 % of females could be successfully pre-trained to dislodge the disc (R, 40/43; W, 40/41; F1 R×W, 27/30; F1 W×R, 29/30; F2 R×W, 25/25; F2 W×R, 24/25), with 185 females progressing to the learning tasks.

Detailed results on F1 hybrid females' performance in both associative and reversal learning is published elsewhere (Vila Pouca et al. 2022). In brief, we found that F1 hybrids showed overall intermediate learning ability between the two parental groups in the two tasks, as seen in Fig 3a,b (Vila Pouca et al. 2022).

When comparing F2 hybrid and parental females in the associative learning task, we found that F2 W×R hybrids had the lowest success rate with only 79% (19/24) females succeeding in the task compared to 88% (22/25) in F2 R×W and to 85% (11/13) and 100% (10/10) in parental common (R) and Endler's (W) guppies, respectively. When comparing the number of trials needed to learn the colour association, we find that F2 R×W performed tendentially worse than parentals, taking significantly longer to succeed compared to common guppies (Fig. 3a; F2 R×W – R: est. = 0.25, s.e. = 0.084, p = 0.014; Suppl. 4.2.1). Comparing learning rates, all four groups improved their success at a similar rate (species × trial number: $\chi^2$ = 2.98, df = 3, p = 0.40; Suppl. 4.2.1).

In the reversal learning task, success rates of F2 hybrid and parental females were very high with 100% (22/22) of F2 R×W, 90% (17/19) of F2 W×R and 100% of both R (11/11) and W (10/10) females reaching learning criterion. Contrary to the associative learning task, we found that common guppies
performed worse than F2 R×W hybrids, needing more trials to succeed in the reversal learning task (Fig. 3b; F2 R×W – R: est. = 0.19, s.e. = 0.072, p = 0.041; Suppl. 4.2.2). All four groups improved their success at a similar rate as the task progressed (species × log trial number: $\chi^2$ = 0.13, df = 3, p = 0.99; Suppl. 4.2.2). Importantly, similar to our previous results with F1 females (Vila Pouca et al. 2022), we found a weak effect of the associative learning task on reversal learning performance with only W females showing a negative correlation between trials needed to learn the association and trials needed to learn the reversal task (species W × trials to success in A.L.: est. = -0.029, SE = 0.010, z = -2.80, P = 0.005; Suppl. 4.2.2).

<figure><img src="image_4.png" /><figcaption>**Figure 3.** Number of trials needed to reach learning criterion in the associative learning (A,C) and reversal learning (B,D) tasks, for females (top) and males (bottom) of each species group. Note that data for parental groups in female plots (A,B) are pooled from the two experiments for ease of visualisation.</figcaption></figure>

In males, success in pre-training the fish to swim through a hole to access a food reward was above 90% for parentals and F1 hybrids but of 79% for both F2 hybrid groups (R, 13/14; W, 14/14; F1 R×W, 11/14; F1 W×R, 11/14; F2 R×W, 13/14; F2 W×R, 12/13). At the end of pre-training, 74 males progressed to the learning tasks.

In the associative learning task, only 46 out of 74 males succeeded in learning to access the reward through the correctly coloured hole. Comparing species groups, the success rate of parental males was slightly lower than F1 and F2 hybrid groups, and F2 R×W showed the highest success rate (R, 46%; W, 57% F1 R×W, 64%; F1 W×R, 64%; F2 R×W, 77%; F2 W×R, 67%). Of the males that succeeded, we found that males from the F2 groups were tendentially faster in learning the colour association (Fig. 3c), but the differences were only significant between F2 R×W and F1 W×R (est. = -0.33, s.e. = 0.10, p = 0.021; Suppl. 4.3.1), likely due to our small sample size. Comparing learning rates, all groups improved their success rate as the task progressed indicating learning of the task contingencies ($\chi^2$ = 44.57, df = 1, p < 0.001; Suppl. 4.3.1); no differences were detected in learning rate between parental and hybrid groups (species × trial number: $\chi^2$ = 4.84, df = 5, p = 0.44; Suppl. 4.3.1). As a control for the presence of the transparent sheet blocking the incorrect option during the associative learning trials, we ran unrewarded test trials where both correct and incorrect options were unblocked. Males showed a significant preference for the correct coloured option during the test (mean % correct = 79%, 95% CI =
75-83%; Wilcoxon rank sum test: V = 986.5, P < 0.001; Suppl. 4.3.2), showing that they learnt the task using the colour cues surrounding the hole.

In the reversal learning task, success rates of male fish were very low, with only 16 of 46 fish reaching learning criterion. F2 R×W were the most successful group with 6/10 fish (60%) learning the task, followed by 4/7 fish (57%) of F2 W×R and 3/7 fish (43%) of F1 R×W. Parental groups and F1 W×R had the lowest success rates with a single individual reaching learning criterion (R, 17%, W, 13%, F1 W×R, 17%). Due to the low sample size, we could not statistically compare the number of trials taken to learn per species group (Fig. 3d; Suppl. 4.3.3). Regarding the probability of choosing the correct option over trials for the 46 males that participated in the task, all species groups improved their success rate as the task progressed (χ² = 67.71, df = 1, p < 0.001; Suppl. 4.3.3) and the rate of learning was similar between parental and hybrid groups (species × log trial number: χ² = 3.01, df = 5, p = 0.70; Suppl. 4.3.3).

## Phenotypic dispersion in brain morphology in F1 but not F2 hybrids

Comparing both females and males of F1 hybrids and parentals, the hypervolume of brain morphospace describing phenotypic dispersion of F1 hybrid groups was over three times larger compared to parental groups and simulated hybrids, and larger for F1 R×W compared to F1 W×R (Fig. 4a,c; Suppl. 5.1). The frequency of F1 hybrid individuals found outside parental ranges, considered as transgressive, was of 9% for F1 R×W (n=3) and 6% for F1 W×R (n=2) individuals, thus at levels close to the 5% expected by chance (Fig. 4a; Suppl. 5.1). Regarding a deviation of the mean hybrid phenotypes, we found that both parental bias and phenotypic mismatch were low since the trait values of F1 R×W and F1 W×R largely overlapped with simulated hybrid populations, although there was a tendency for some phenotypic mismatch of F1 R×W (Fig. 4b,d,e; Suppl. 5.1).

When comparing both females and males of F2 hybrids and parentals, the phenotypic dispersion of brain morphospace of F2 hybrid groups was similar to one parental group (Endler's guppies, W). Common guppies (R) and simulated hybrids showed lower dispersion (Fig. 4f,h; Suppl. 5.2). The frequency of transgressive F2 hybrids was of 5% for F2 R×W (n=2) and 8% for F2 W×R (n=3) individuals, thus also at levels close to the 5% expected by chance (Fig. 4f; Suppl. 5.2). Regarding a deviation of the mean hybrid phenotypes, we found that parental bias of F2 R×W was low but F2 W×R hybrids were slightly biased toward Endler's guppies, W (Fig. 4i; Suppl. 5.2). Similarly, the phenotypic mismatch of F2 R×W was low but F2 W×R showed high mismatch with parentals, since the mean phenotype of F2 W×R deviated away from the line connecting parental mean phenotypes towards a smaller relative optic tectum (Fig. 4g,j; Suppl. 5.2).
<figure><img src="image_5.png" /><figcaption>**Figure 4.** Summary of phenotypic variation and transgression in brain morphology for parentals and F1 hybrids (A-E) and for parentals and F2 hybrids (F-J). Distribution of individual hybrids and parentals in the brain morphospace, with shaded 95% KDEs (A,F). Position of the mean phenotype of hybrid and parental groups and of simulated hybrid populations (B,G). Hypervolume of observed and simulated 95% KDEs as an estimate of phenotypic dispersion (C,H). Distance of observed and simulated mean hybrid phenotypes from the midpoint between the parentals as an estimate of parental bias (D,I). Deviation of observed and simulated mean hybrid phenotypes from the line connecting parental mean phenotypes as an estimate of phenotypic mismatch (E,J).</figcaption></figure>

No hybrid transgression in cognitive traits in F2 hybrids

Due to the small number of males that succeeded in the reversal learning task, here we could only run phenotypic dispersion and transgression analyses on female cognitive traits. Measures of ‘cognitive space’ describing female F1 hybrid groups are published elsewhere (Vila Pouca et al. 2022); in brief, we found that F1 phenotypes had slightly higher dispersion relative to parents, that some hybrid individuals were transgressive, and that F1 R×W had high phenotypic mismatch with parentals (Vila Pouca et al. 2022). Comparing female F2 hybrids and parentals, we found no evidence of phenotypic dispersion or transgression in F2 hybrids. The hypervolume of ‘cognitive space’ describing phenotypic dispersion was similar between the four groups (Fig. 5a-c; Suppl. 5.4) and none of the F2 hybrid females had trait combinations outside the phenotypic space of the parentals (Fig. 5a,b; Suppl. 5.4). Regarding a deviation of the mean hybrid phenotypes, we found that both parental bias and phenotypic mismatch were low since the trait values of F2 R×W and F2 W×R largely overlapped with simulated hybrid populations, although there was a tendency towards phenotypic mismatch of F2 R×W (Fig. 5d,e; Suppl. 5.4).
<figure><img src="image_6.png" /><figcaption>**Figure 5.** Summary of phenotypic variation and transgression in cognitive traits for female F2 hybrids. Distribution of individual hybrids and parentals in the two-dimensional ‘cognitive space’, with shaded 95% parental KDEs (A,B). Hypervolume of observed and simulated 95% KDEs as an estimate of phenotypic dispersion (C). Distance of observed and simulated mean hybrid phenotypes from the midpoint between the parentals as an estimate of parental bias (D). Deviation of observed and simulated mean hybrid phenotypes from the line connecting parental mean phenotypes as an estimate of phenotypic mismatch (E).</figcaption></figure>

Discussion

We investigated brain anatomy, learning ability, and cognitive flexibility in F1 and F2 hybrids of two
fish species (*Poecilia reticulata* and *P. wingei*) to characterize patterns of hybrid trait expression and to
test if hybridization may promote phenotypic variation in brain morphology and/or cognitive traits.
Overall, we found that F1 and F2 hybrids showed intermediate brain anatomy traits, learning ability,
and cognitive flexibility compared to the parental groups. Moreover, in both brain and cognitive traits,
phenotypic dispersion and transgression were low and hybrid mean phenotypes did not deviate from the
axis of variation of the parentals. These results therefore suggest that hybridization is not a strong driver
of brain anatomical and cognitive diversification in these Poeciliidae.

Interspecific hybridization is generally assumed to be an important source of heritable phenotypic
variation (Seehausen 2004; Mallet 2007; Dittrich-Reed and Fitzpatrick 2013). Such variation may
enlarge the working surface for natural selection and promote the evolutionary potential of hybrid
populations (Rieseberg et al. 1999; Seehausen 2013). For example, some hybrids of African cichlids or
Galapagos finches show greater variation in morphological and/or behavioural phenotypes and express
transgressive traits that allow them to explore ecological opportunities unavailable to the parentals
(Lamichhaney et al. 2018; Selz and Seehausen 2019; Feller et al. 2020). In a previous study, we found
that cognitive phenotypes of F1 hybrid females were slightly over-dispersed compared to parentals, that
some hybrid individuals had transgressive combinations of learning scores, and that F1 R×W female
hybrids showed a significant deviation away from the axis of variation of the parentals (Vila Pouca et
al. 2022). Here, we aimed to expand on these findings by testing both males and females of F1 and F2
hybrid groups and by assessing not only learning and cognitive flexibility but also relative brain size
and brain region anatomy. In total, we assayed brain morphology of 232 animals and cognitive abilities
of 259 animals. Despite this very large sample size we found no evidence of phenotypic dispersion or
transgression in cognitive phenotypes of F1 males and in F2 females and males, as well as little to no
dispersion and transgression in brain anatomy traits of F1 and F2 fish. Therefore, hybridization does not
seem to be a prominent promotor of diversification in brain anatomy and cognitive traits, at least in
these Poeciliidae species. What may explain the apparent discrepancy in cognitive results between F1
and F2 fish? We suggest that heterosis, also called hybrid vigour, might explain our previous results on cognitive traits of F1 females since it is only present in first-generation hybrids (Lippman and Zamir 2007; Proops et al. 2009; Vila Pouca et al. 2022). Any fitness advantage from heterosis is generally short lived because recombination in subsequent generations causes hybrid breakdown (Lippman and Zamir 2007), indicating that the phenotypic variation we observed in cognitive traits of F1 females might have no adaptive potential.

Hybridization can alter trait correlations by relaxing phenotypic trade-offs and genetic correlations (Selz et al. 2014; Johnson et al. 2015). In this study, we expected that brain anatomy traits might be particularly affected by relaxed covariation and a release of energetic trade-offs constraining brain region development (Niven and Laughlin 2008; Kotrschal et al. 2013), leading to increased phenotypic variation in hybrid groups. However, we found that all brain traits were strongly correlated both in parental and hybrid groups, that species differences in brain size were overall equally expressed over the different brain regions, and that hybrids showed limited phenotypic dispersion compared to parentals. Furthermore, our main result was that hybrid phenotypes were overall intermediate between parentals, both in relative brain size, brain region volume, learning ability, and cognitive flexibility. Intermediate phenotypes are often at a competitive disadvantage in the presence of parental individuals (Mallet 2007; Rice and McQuillan 2018; Rice 2020). Regarding the traits under consideration in this study, although the functional and fitness consequences of intermediate brain size and cognitive abilities are unclear, it is possible that hybrids with intermediate traits have invested significant energy into their development but without obtaining the benefits they should provide, at least compared to the parental group with the most developed traits (Niven and Laughlin 2008; Dunlap and Stephens 2016). If hybrids with intermediate phenotypes are indeed disadvantaged compared to parentals, then brain anatomy and/or cognitive traits may represent a postzygotic isolating barrier against hybridization (McQuillan et al. 2018).

Two non-mutually exclusive hypotheses may explain the lack of transgression and phenotypic dispersion we found in these traits. One possibility is that *P. reticulata* and *P. wingei*, the two parental species used in this study, are too phylogenetically close, as the amount of transgression has been suggested to increase as a function of the genetic distance between the parental lines (Rieseberg et al. 1999; Stelkens et al. 2009). However, genetic distance may not necessarily predict transgression or any other aspect of trait expression in hybrids (Thompson et al. 2021). Another possibility is that the traits we chose to study are not impacted by hybridization, for example if they are under consistent directional selection. A previous study on morphological transgression in hybrids between two cichlid fish species found that phenotypic diversity and transgression was reduced in traits with consistent directional selection, in contrast to traits that evolved in response to stabilizing selection (Albertson and Kocher 2005). These results indicate that the effects of hybridization can be limited to specific traits, i.e., those that have not diverged in response to strong directional selection (Albertson and Kocher 2005). The basic organization of the brain and associated brain functions are highly conserved in vertebrates and under strong energetic constraints and selection pressures (Jerison 1973; Broglio et al. 2005; Niven and Laughlin 2008; Yopak et al. 2010). It is possible that in our parental species, which share a generally similar ecology (Alexander and Breden 2004; Poeser et al. 2005), brain anatomy and associated cognitive abilities are under consistent directional selection, thereby limiting the necessary genetic variation for transgressive segregation to occur. Additionally, our fish have been in captive conditions in the laboratory for several generations. They were always kept in several large aquaria to keep as much genetic variation in the populations, yet effects from domestication (Marchetti and Nevitt 2003; Burns et al. 2009) cannot be excluded. In the laboratory, natural selection pressures are relaxed, and any other pressures are likely to be in the same direction in both species, potentially eroding or limiting any potential effects of hybridization in these traits.
In our previous study on learning ability and cognitive flexibility in F1 females, we found that F1 R×W hybrids showed a phenotypic mismatch with the parentals, while the reciprocal W×R group did not (Vila Pouca et al. 2022). This observed asymmetry between reciprocal crossings suggested the contribution of uniparentally inherited genetic factors (Turelli and Moyle 2007). Hybrid asymmetries typically arise from incompatibilities involving uniparentally inherited genetic factors (from mitochondria, chloroplasts, maternal transcripts, or sex chromosomes) and tend to be more pronounced in the heterogametic sex (Turelli and Moyle 2007; Bolnick et al. 2008). In fruit flies, hybrid males (the heterogametic sex) show greater impairment of foraging behaviour (Turissini et al. 2017), and in wild chickadees the females (heterogametic sex in birds) show poorer cognitive skills (McQuillan et al. 2018). In the present study, we expanded cognitive testing and brain analyses to males, which are the heterogametic sex in *Poecilia* fish. While we are unable to draw strong conclusions regarding male cognitive flexibility due to the small number of males that succeeded in the reversal task, our results do not show evidence in favour of stronger asymmetries or incompatibilities in male *Poecilia* fish. Instead, we found that males needed a similar number of trials to learn the colour association and the reversal task compared to females, that reciprocal male hybrid crossings showed similar cognitive performance, and that males and females differed little in how hybridization impacted brain anatomy traits.

Taken together, our results suggest that hybridization is not a strong driver of brain anatomical and cognitive diversification in the two *Poeciliidae* species studied here. As the first test of the impact of hybridization in brain morphology and cognitive traits, our results contribute to further elucidate the role of hybridization as a driver of variation and novelty in important traits linked to fitness. To determine the generality of our conclusions, hybridization experiments with further cognitive tests need to be repeated in other families.

**Author contributions**

C.V.P. developed ideas and designed the study with H.D.W. and A.K.. C.V.P collected the data with H.D.W. C.V.P. analysed the data and wrote the first draft. All authors edited and contributed to finalizing the manuscript.

**Acknowledgements**

We thank Bart Pollux for providing stock fish to start our experimental populations at WUR and the staff from the BHE group and Carus at WUR for technical and husbandry assistance. Further, we are grateful to our undergraduate students who helped maintain the fish stocks and assisted with data collection, namely Sijmen Vedder, Thirza Weenink, Emma van Veenendaal, Emma Scheltens, and Sander Endert. We also thank Maria Vila Pouca for her help with analytical geometry calculations and Regina Vega-Trejo for helpful discussions. A.K. was funded by Vetenskapsrådet (2017-04957) and Carl Tryggers (CTS18:205). C.V.P. was funded by an ASAB Research Grant 2020.

**Data accessibility statement:**

Raw data and analysis code will be deposited in a freely accessible repository upon acceptance.

**Conflict of interest**

We declare we have no conflict of interest.
References

Akima, H. and A. Gebhardt. 2022. akima: Interpolation of irregularly and regularly spaced data. R package version 0.6-3.4.

Albertson, R. C. and T. D. Kocher. 2005. Genetic architecture sets limits on transgressive segregation in hybrid cichlid fishes. Evolution 59:686-690.

Alexander, H. J. and F. Breden. 2004. Sexual isolation and extreme morphological divergence in the Cumaná guppy: a possible case of incipient speciation. J. Evol. Biol. 17:1238-1254.

Bates, D., M. Mächler, B. Bolker, and S. Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. J. Stat. Softw. 1.

Bell, M. A. and M. P. Travis. 2005. Hybridization, transgressive segregation, genetic covariation, and adaptive radiation. Trends Ecol. Evol. 20:358-361.

Bolnick, D. I., M. Turelli, H. n. López-Fernández, P. C. Wainwright, and T. J. Near. 2008. Accelerated Mitochondrial Evolution and “Darwin's Corollary”: Asymmetric Viability of Reciprocal F1 Hybrids in Centrarchid Fishes. Genetics 178:1037-1048.

Bridle, J., C. Saldamando, W. Koning, and R. Butlin. 2006. Assortative preferences and discrimination by females against hybrid male song in the grasshoppers *Chorthippus brunneus* and *Chorthippus jacobsi* (Orthoptera: Acrididae). J. Evol. Biol. 19:1248-1256.

Broglio, C., A. Gómez, E. Durán, F. M. Ocaña, F. Jiménez-Moya, F. Rodríguez, and C. Salas. 2005. Hallmarks of a common forebrain vertebrate plan: Specialized pallial areas for spatial, temporal and emotional memory in actinopterygian fish. Brain Res. Bull. 66:277-281.

Broglio, C., A. Gómez, E. Durán, C. Salas, and F. Rodríguez. 2011. Brain and cognition in teleost fish *in* C. Brown, K. Laland, and J. Krause, eds. Fish cognition and behavior. Wiley-Blackwell, Oxford.

Buechel, S. D., A. Boussard, A. Kotrschal, W. van der Bijl, and N. Kolm. 2018. Brain size affects performance in a reversal-learning test. Proc. R. Soc. B 285:20172031.

Burns, J. G., A. Saravanan, and F. Helen Rodd. 2009. Rearing environment affects the brain size of guppies: Lab-reared guppies have smaller brains than wild-caught guppies. Ethology 115:122-133.

Cole, Ella F., J. Morand-Ferron, Amy E. Hinks, and John L. Quinn. 2012. Cognitive ability influences reproductive life history variation in the wild. Curr. Biol. 22:1808-1812.

Deaner, R. O., K. Isler, J. Burkart, and C. Van Schaik. 2007. Overall brain size, and not encephalization quotient, best predicts cognitive ability across non-human primates. Brain Behav. Evol. 70:115-124.

Dittrich-Reed, D. R. and B. M. Fitzpatrick. 2013. Transgressive hybrids as hopeful monsters. Evolutionary biology 40:310-315.

Dukas, R. and E. A. Bernays. 2000. Learning improves growth rate in grasshoppers. PNAS 97:2637-2640.

Dunlap, A. S. and D. W. Stephens. 2016. Reliability, uncertainty, and costs in the evolution of animal learning. Curr. Opin. Behav. Sci. 12:73-79.

Duong, T. 2021. ks: Kernel Smoothing.

Facon, B., J.-P. Pointier, P. Jarne, V. Sarda, and P. David. 2008. High genetic variance in life-history strategies within invasive populations by way of multiple introductions. Curr. Biol. 18:363-367.

Feller, A. F., O. M. Selz, M. D. McGee, J. I. Meier, S. Mwaiko, and O. Seehausen. 2020. Rapid generation of ecologically relevant behavioral novelty in experimental cichlid hybrids. Ecology and evolution 10:7445-7462.

Feng, D. and L. Tierney. 2008. Computing and displaying isosurfaces in R. J. Stat. Softw. 28:1-24.

Fong, S., S. D. Buechel, A. Boussard, A. Kotrschal, and N. Kolm. 2019. Plastic changes in brain morphology in relation to learning and environmental enrichment in the guppy (*Poecilia reticulata*). The Journal of Experimental Biology 222:jeb200402.

Fox, J. and S. Weisberg. 2019. An R companion to applied regression. Sage publications, Thousand Oaks, CA.

Fuss, T. and K. Witte. 2019. Sex differences in color discrimination and serial reversal learning in mollies and guppies. Curr. Zool. 65:323-332.
Herculano-Houzel, S. 2017. Numbers of neurons as biological correlates of cognitive capability. Curr. Opin. Behav. Sci. 16:1-7.

Holzman, R. and C. D. Hulsey. 2017. Mechanical transgressive segregation and the rapid origin of trophic novelty. Sci. Rep. 7:40306.

Hothorn, T., F. Bretz, and P. Westfall. 2008. Simultaneous inference in general parametric models. Biometrical Journal: Journal of Mathematical Methods in Biosciences 50:346-363.

Jerison, H. 1973. Evolution of the brain and intelligence. Academic Press, New York.

Johnson, J. B., Z. W. Culumber, R. Easterling, and G. G. Rosenthal. 2015. Boldness and predator evasion in naturally hybridizing swordtails (Teleostei: Xiphophorus). Curr. Zool. 61:596-603.

Kotrschal, A., B. Rogell, A. Bundsen, B. Svensson, S. Zajitschek, I. Brännström, S. Immler, A. A. Maklakov, and N. Kolm. 2013. Artificial selection on relative brain size in the guppy reveals costs and benefits of evolving a larger brain. Curr. Biol. 23:168-171.

Laland, K. N. and K. Williams. 1997. Shoaling generates social learning of foraging information in guppies. Anim. Behav. 53:1161-1169.

Lamichhaney, S., F. Han, M. T. Webster, L. Andersson, B. R. Grant, and P. R. Grant. 2018. Rapid hybrid speciation in Darwin's finches. Science 359:224-228.

Lefebvre, L., S. M. Reader, and D. Sol. 2004. Brains, innovations and evolution in birds and primates. Brain Behav. Evol. 63:233-246.

Lewontin, R. C. and L. C. Birch. 1966. Hybridization as a source of variation for adaptation to new environments. Evolution 20:315-336.

Linn, C. E., H. R. Dambroski, J. L. Feder, S. H. Berlocher, S. Nojima, and W. L. Roelofs. 2004. Postzygotic isolating factor in sympatric speciation in Rhagoletis flies: reduced response of hybrids to parental host-fruit odors. PNAS 101:17753-17758.

Lippman, Z. B. and D. Zamir. 2007. Heterosis: revisiting the magic. Trends Genet. 23:60-66.

Madden, J. R., E. J. Langley, M. A. Whiteside, C. E. Beardsworth, and J. O. van Horik. 2018. The quick are the dead: pheasants that are slow to reverse a learned association survive for longer in the wild. Philos. Trans. R. Soc. B 373:20170297.

Mallet, J. 2007. Hybrid speciation. Nature 446:279-283.

Marchetti, M. P. and G. A. Nevitt. 2003. Effects of hatchery rearing on brain structures of rainbow trout, Oncorhynchus mykiss. Environ. Biol. Fishes 66:9-14.

McQuillan, M. A., T. C. Roth II, A. V. Huynh, and A. M. Rice. 2018. Hybrid chickadees are deficient in learning and memory. Evolution 72:1155-1164.

Mérot, C., V. Debat, Y. Le Poul, R. M. Merrill, R. E. Naisbit, A. Tholance, C. D. Jiggins, and M. Joron. 2020. Hybridization and transgressive exploration of colour pattern and wing morphology in *Heliconius* butterflies. J. Evol. Biol. 33:942-956.

Niven, J. E. and S. B. Laughlin. 2008. Energy limitation as a selective pressure on the evolution of sensory systems. J. Exp. Biol. 211:1792-1804.

Pärssinen, V., K. Hulthén, C. Brönmark, C. Skov, J. Brodersen, H. Baktoft, B. B. Chapman, L. A. Hansson, and P. A. Nilsson. 2020. Maladaptive migration behaviour in hybrids links to predator-mediated ecological selection. J. Anim. Ecol. 89:2596-2604.

Pereira, R. J., F. S. Barreto, and R. S. Burton. 2014. Ecological novelty by hybridization: experimental evidence for increased thermal tolerance by transgressive segregation in Tigriopus californicus. Evolution 68:204-215.

Poeser, F. N., M. Kempkes, and I. J. Isbrücker. 2005. Description of *Poecilia* (*Acanthophacelus*) *wingei* n. sp. from the Paría Peninsula, Venezuela, including notes on *Acanthophacelus* Eigenmann, 1907 and other subgenera of *Poecilia* Bloch and Schneider, 1801 (Teleostei, Cyprinodontiformes, Poeciliidae). Contributions to Zoology 74:97-115.

Proops, L., F. Burden, and B. Osthaus. 2009. Mule cognition: a case of hybrid vigour? Anim. Cogn. 12:75-84.

R Core Team. 2020. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.

Ranganath, H. and S. Aruna. 2003. Hybridization, transgressive segregation and evolution of new genetic systems in *Drosophila*. Journal of Genetics 82:163-177.

Rice, A. M. 2020. The Overlooked Influence of Hybridization on Cognition. Frontiers in Ecology and Evolution 8:39.
Rice, A. M. and M. A. McQuillan. 2018. Maladaptive learning and memory in hybrids as a reproductive isolating barrier. Proc. R. Soc. B 285:20180542.

Rieseberg, L. H., M. A. Archer, and R. K. Wayne. 1999. Transgressive segregation, adaptation and speciation. Heredity 83:363-372.

Rieseberg, L. H., O. Raymond, D. M. Rosenthal, Z. Lai, K. Livingstone, T. Nakazato, J. L. Durphy, A. E. Schwarzbach, L. A. Donovan, and C. lexer. 2003a. Major ecological transitions in wild sunflowers facilitated by hybridization. Science 301:1211-1216.

Rieseberg, L. H., A. Widmer, A. M. Arntz, and B. Burke. 2003b. The genetic architecture necessary for transgressive segregation is common in both natural and domesticated populations. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences 358:1141-1147.

Schindelin, J., I. Arganda-Carreras, E. Frise, V. Kaynig, M. Longair, T. Pietzsch, S. Preibisch, C. Rueden, S. Saalfeld, B. Schmid, J.-Y. Tinevez, D. J. White, V. Hartenstein, K. Eliceiri, P. Tomancak, and A. Cardona. 2012. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9:676-682.

Seehausen, O. 2004. Hybridization and adaptive radiation. Trends Ecol. Evol. 19:198-207.

Seehausen, O. 2013. Conditions when hybridization might predispose populations for adaptive radiation. J. Evol. Biol. 26:279-281.

Selz, O. M., K. Lucek, K. A. Young, and O. Seehausen. 2014. Relaxed trait covariance in interspecific cichlid hybrids predicts morphological diversity in adaptive radiations. J. Evol. Biol. 27:11-24.

Selz, O. M. and O. Seehausen. 2019. Interspecific hybridization can generate functional novelty in cichlid fish. Proc. R. Soc. B 286:20191621.

Sol, D., R. P. Duncan, T. M. Blackburn, P. Cassey, and L. Lefebvre. 2005. Big brains, enhanced cognition, and response of birds to novel environments. Proc. Natl. Acad. Sci. USA 102:5460-5465.

Stelkens, R. B., C. Schmid, O. Selz, and O. Seehausen. 2009. Phenotypic novelty in experimental hybrids is predicted by the genetic distance between species of cichlid fish. BMC Evol. Biol. 9:283.

Thompson, K. A., M. Urquhart-Cronish, K. D. Whitney, L. H. Rieseberg, and D. Schluter. 2021. Patterns, predictors, and consequences of dominance in hybrids. Am. Nat. 197:E72-E88.

Triki, Z., S. Fong, M. Amcoff, and N. Kolm. 2022. Artificial mosaic brain evolution of relative telencephalon size improves inhibitory control abilities in the guppy (*Poecilia reticulata*). Evolution 76:128-138.

Turelli, M. and L. C. Moyle. 2007. Asymmetric Postmating Isolation: Darwin's Corollary to Haldane's Rule. Genetics 176:1059-1088.

Turissini, D. A., A. A. Comeault, G. Liu, Y. C. G. Lee, and D. R. Matute. 2017. The ability of *Drosophila* hybrids to locate food declines with parental divergence. Evolution 71:960-973.

Vila Pouca, C., S. Vedder, and A. Kotrschal. 2022. Hybridization may promote variation in cognitive phenotypes in experimental guppy hybrids. Am. Nat. online early

White, G. E. and C. Brown. 2015. Variation in brain morphology of intertidal gobies: a comparison of methodologies used to quantitatively assess brain volumes in fish. Brain Behav. Evol. 85:245-256.

Yopak, K. E., T. J. Lisney, R. B. Darlington, S. P. Collin, J. C. Montgomery, and B. L. Finlay. 2010. A conserved pattern of brain scaling from sharks to primates. PNAS 107:12946-12951.