Roles for serotonin and vasotocin in regulating social behavior in Astatotilapia burtoni [electronic resource]

Serotonin (5-HT) and vasotocin (AVT) have become well known from many species for their role(s) in regulating the frequency of aggressive and sex-related behaviors. Studies show that one common effect of serotonin is to inhibit aggression, but whether these roles require serotonin signaling in discrete brain nuclei is unknown. Vasotocin effects on social behavior on the other hand, cannot be generalized because they vary across species and behavioral context. However, the V1a receptor has been shown to mediate behavioral effects in many species. I chose Astatotilapia burtoni as a model to study the serotonergic system because of its well-described subordinate and dominant social phenotypes that have non-overlapping aggression profiles. In this species, subordinate males are less aggressive than dominants, and this low aggression phenotype is associated with increased serotonin turnover in the brain. I predicted that if serotonin has mainly an inhibitory effect on sites of release, subordinates would have higher 5-HT turnover (i.e. higher serotonin activity) in the preoptic area (POA) and ventral tuberal nucleus (VTn), given these nuclei have been implicated in sexual behavior and aggression in other species. I found that indeed, in the POA, but not VTn (a putative partial homolog to the mammalian anterior hypothalamus), subordinates had higher serotonin turnover. The ability of serotonin to exert either an inhibitory or excitatory effect on a post-synaptic neuron depends on which receptors that cell expresses. Among serotonin receptors (5-HTRs), the subtype 5-HTR1A is known to mediate at least partially, inhibitory effects of serotonin on cell firing and subsequent 5-HT release. Importantly, activation of the 5-HTR1A receptor is known to decrease aggression in the fighting fish Betta splendens, but in contrast, solely elevating levels of serotonin uniformly throughout the brain does not have the same effect. I show that among the six known 5-HTRs in A. burtoni, expression of the genes encoding 5-HTR1A and 5-HTR2A is higher in subordinates compared to dominants, but only in the telencephalon and anterior preoptic area. This result suggests brain nuclei in these regions are involved in some aspect of behavior and/or physiological regulation of the subordinate phenotype. I report for the first time, evidence that subordinates have greater 5-HT depletion in dorsal, but not medial, 5-HT cells compared to dominants. So far, detailed studies on the neuroanatomy of the serotonergic system in zebrafish have shown that dorsal and medial 5-HT cells differ in the regions they project to, but whether and how this is associated with social behavior is unknown. Dorsal and medial 5-HT cells in rodents also differ in their projection areas and how dorsal cells respond to social defeat determines whether social avoidance will develop in cases of chronic social defeat. Therefore, confirmation of 5-HT depletion in dorsal, but not medial, raphe cells, suggests a functional role for dorsal 5-HT cells in processing outcomes from interactions that involve social defeat, is conserved between rodents and teleosts. I used a non-biased approach to identify areas of the brain that differ in their activity during fighting and courting behaviors. From this work, there was a salient difference in neural activity in the POA between males that had recently fought, versus those that has recently courted females. I chose to test directly whether AVT cells, which are located in the POA and are known to vary in AVT mRNA abundance by social status, were differentially activated between courters and fighters. AVT cells in A. burtoni are divided into 3 subpopulations: parvocellular, magnocellular and gigantocellular. A previous model for A. burtoni, proposed that gigantocellular AVT cells were involved in activating a neural circuit controlling territorial behaviors and that magnocellular cells did not play an important role in mediating behavioral differences between subordinates and dominants. In contrast, I found that magnocellular, but not gigantocellular, AVT cells had greater neural activity during fighting, but not courting. Furthermore, I characterized the distribution of AVT receptors V1a1 and V1a2 using in situ hybridization and found that V1a2 is most highly expressed in the dorsal nucleus of the ventral telencephalon (Vd), as well as VTn, subdivisions of the lateral tuberal nucleus (NLT) and POA, suggesting AVT axon terminals may form synapses with neurons in these brain regions. I confirmed that the 5-HTR1A receptor, which showed greater forebrain mRNA abundance in subordinates relative to dominant males, is expressed in AVT cells. The two most salient lines of evidence that have been used to argue for a direct relationship between 5-HT and AVT in the modulation of aggression are 1) that SSRIs and 5-HTR agonists, if administered prior to delivery of AVT, block AVT-induced behaviors and 2) high density of 5-HT varicosities surround AVP neurons in the hamster. Because 5-HT and AVP have opposite effects on aggression in rodents and stimulation of 5-HT receptors of the 1A subtype leads to inhibition of cell firing, several studies have suggested serotonin may modulate aggression by regulating the firing patterns of AVP/AVT cells and their subsequent hormone release. Based on these results, I speculate that 5-HT cells, perhaps exclusively originating from the dorsal raphe, may regulate at least partially, the firing patterns of AVT cells in the POA. This form of regulation by serotonin would be expected to decrease aggression by changing AVT release volumes in the pituitary and/or brain. Future studies should explore these possibilities, including differences in projection areas and release patterns across AVT subpopulation.

Keywords: Biology.
Advisor: Fernald Russell D. primary advisor. Thesis advisor
Buckmaster Paul advisor. Thesis advisor
Parker Karen J.