Serotonin modulates nucleus accumbens circuits to suppress aggression in mice

Domain: neuroscience
Published: 2026-02-14
Editorial outcome: major_revision

Abstract
Serotonin (5-hydroxytryptamine; 5HT) has long been considered anti-aggressive, but the mechanisms by which 5HT regulates downstream circuits to control aggression remain unclear. Combining fiber photometry, optogenetics, and miniaturized microscope recordings in double-transgenic male mice, we find that 5HT levels ramp up in the nucleus accumbens during aggression, inhibiting a subset of D1 medium spiny neurons to suppress attacks. Our results reveal a novel 5HT-mediated neuromodulatory mechanism for limiting aggressive behavior.

Introduction
Low levels of brain serotonin (5HT) have been associated with heightened aggression across the animal kingdom1,2. Drugs boosting the 5HT system, however, produce variable effects on aggression3, forcing clinicians to prescribe agents that sedate patients rather than specifically reduce aggression4,5. The complex behavioral effects of brain-wide 5HT modulation may be due to heterogeneity in the 5HT system, with distinct 5HT neuron populations projecting to different targets to mediate diverse behaviors6–8. While recent studies have begun to map specific 5HT targets onto their associated behaviors8–11, little is known about the target regions that control aggression, or how 5HT modulates neural activity in these targets12,13.

The nucleus accumbens (NAc) has long been identified as a key node controlling motivated behaviors14,15. Although most research has focused on the role of dopamine (DA) in this region16, recent studies have implicated NAc 5HT release in promoting a range of prosocial behaviors, including behavioral antecedents of empathy17–20. In particular, the medial shell subregion of the NAc, the site with the most abundant aggression-activated neurons21, is a major target of 5HT axons projecting from the dorsal raphe (DR)10. We therefore hypothesized that 5HT release in the NAc medial shell may also influence aggression, and that it would do so via distinct mechanisms from those engaged by DA.

Results
5HT and DA levels in the NAc increased during aggression with distinct temporal profiles
To measure real-time changes in 5HT and DA levels in the NAc during aggression, we used fiber photometry to record either GRAB-5HT22 or GRAB-DA23 signals in the NAc during a resident-intruder assay24, where a weak intruder mouse was introduced to the home cage of the test mouse (Fig. 1a, b; Supplementary Fig. 1a, b). We found that 5HT levels were low at aggressive approach, ramped up during the attack itself, and peaked around aggression offset (Fig. 1c-e). While the peak and area under the curve (AUC) positively correlated with aggression duration (Supplementary Fig. 2), the slope of the 5HT ramp showed a strong negative correlation (Fig. 1f): the faster the rise in 5HT, the shorter the aggressive bout. In contrast, DA levels rose during both the pre-attack approach period and at attack onset (Supplementary Fig. 1c-e), and only weakly correlated with aggression duration (Supplementary Fig. 1f). These data suggest that DA and 5HT may signal different phases of aggression, with 5HT selectively engaged in mediating the termination of attacks.

5HT but not DA release in the NAc suppresses aggression
We next tested the causal roles of 5HT or DA release in aggressive behavior. We initially examined the effects of (±) 3,4-methylenedioxymethamphetamine (MDMA), a translationally-relevant compound that enhances both 5HT and DA release but promotes prosocial behaviors primarily by increasing 5HT levels in the NAc18,25. Consistent with prior studies26, administration of MDMA (7.5 mg/kg, i.p.) to the resident mouse 15 min before the resident-intruder test completely abolished aggression, compared to a saline control (Fig. 1g, h). Although this finding suggests that increased 5HT release may reduce aggression, the drug has numerous effects throughout the brain, and cannot be interpreted as selective to either 5HT or the NAc.

To more specifically manipulate either 5HT or DA release in the NAc, we injected the Flp-dependent excitatory opsin, FLPX-ChRimsonR or fDIO-rsChRmine, in either the DR of SERT-Flp mice (Fig. 1i), or the ventral tegmental area of DAT-Flp mice, and implanted optical fibers in the NAc (Supplementary Fig. 1g). Optogenetic activation of DA inputs in the NAc during the resident-intruder test did not affect either the frequency of attacks or the duration of each attack bout (Supplementary Fig. 1h, i), although we validated that photostimulation did trigger DA release (Supplementary Fig. 1j-l). In contrast, 5HT input activation significantly reduced the duration of attacks, without changing attack frequency (Fig. 1j, k, Supplementary Fig. 3) or basic locomotor, social, or valence-related behaviors (Supplementary Fig. 4).

To begin to determine the 5HT receptors mediating the anti-aggressive properties of NAc 5HT release, we infused a variety of 5HT receptor-specific agonists into the NAc immediately prior to the resident-intruder assay. Consistent with previous results examining the prosocial effects of NAc 5HT release17, infusion of a 5HT1b receptor agonist (CP93129) significantly reduced aggression, while 5HT1f and 5HT2c receptor agonists did not (Supplementary Fig. 5). These results suggest that in the medial NAc shell, 5HT release, but not DA release, is sufficient to reduce aggressive behavior, with a particular role in terminating attack bouts.

D1 but not D2 MSNs in the NAc promote aggression
What downstream cells mediate the effects of NAc 5HT on aggression? The vast majority of NAc neurons are medium spiny neurons (MSNs) expressing one of two DA receptors: the D1 receptor, which is thought to augment neural activity in response to DA; or the D2 receptor, which does the opposite27. In the context of aggressive behavior, D1 MSNs in the NAc have been suggested to signal the rewarding aspect of aggression21,28. However, optogenetic stimulation of DA axons in the NAc, which presumably activates D1 MSNs, failed to affect aggression in prior studies29,30.

To determine how D1 and D2 MSNs are engaged during aggression, we injected the Cre-dependent calcium indicator, FLEX-GCaMP8m, in either D1-Cre or A2A-Cre mice, and recorded the activity of D1 or D2 MSNs during the resident-intruder test using a miniature microscope (Fig. 2a, b). On average, both MSN cell types exhibited increased activity during aggression (Supplementary Fig. 6a-d), but significantly more D1 MSNs than D2 MSNs showed robust activation across attacks (Fig. 2c, d). Examining individual trials, we found that the activity of D1 MSNs, but not D2 MSNs, positively correlated with the duration of aggressive episodes (Fig. 2e, f). This suggests that D1 MSN activity, like 5HT release, may influence the length of aggressive episodes. To test this hypothesis, we injected the Cre-dependent inhibitory opsin, NpHR3.0 in the NAc of either D1-Cre or A2A-Cre mice to optogenetically inhibit either D1 or D2 MSNs during the resident-intruder test (Fig. 2g). Similar to the effect of 5HT input activation, D1 MSN (but not D2 MSN) inhibition reduced the duration of attack bouts without influencing attack frequency (Fig. 2h, i; Supplementary Fig. 3c-f) or other locomotor, social, or valence-related behaviors (Supplementary Fig. 6e-j).

5HT release in the NAc differentially modulates D1 and D2 MSNs
Given that 5HT stimulation and D1 MSN inhibition in the NAc produce similar anti-aggressive behavioral effects, we hypothesized that 5HT may specifically reduce the activity of D1 MSNs to suppress attack duration. To test this prediction, we first measured D1 or D2 MSN GCaMP8m activity using a miniscope when the mice were administered the same dose of MDMA (7.5 mg/kg) that abolished aggression in the resident-intruder tests (Fig. 3a). We found that MDMA administration led to a significant decrease in D1, but not D2 MSN firing (Fig. 3b-d). This result is consistent with our hypothesis that 5HT inhibits D1 MSN activity, but again does not control for other possible network effects of MDMA administration.

To more specifically assess the effects of endogenous 5HT release on MSN activity, we generated double-transgenic mice by crossing D1-Cre or A2A-Cre mice with SERT-Flp mice, and injected them with Cre-dependent GCaMP8m virus in the NAc and Flp-dependent ChRimsonR virus in the DR (Fig. 3e, f). Placement of a GRIN lens over the NAc injection site allowed us to simultaneously stimulate 5HT release and record genetically-identified downstream neurons (Fig. 3g). Consistent with the MDMA results, optogenetic stimulation of 5HT inputs in the NAc (Fig. 3h) overall inhibited the activity of D1 MSNs, but not D2 MSNs (Fig. 3i-l).

5HT preferentially inhibits MSNs that signal aggression in the NAc
Next, we took advantage of our cellular-resolution recording technique to determine which D1 MSNs were particularly sensitive to 5HT-mediated inhibition. In two sessions on the same day, we first recorded D1 MSNs during a resident-intruder task, and later recorded the same neurons during optogenetic stimulation of 5HT release. We sorted aggression-modulated D1 MSNs (~37% of all D1 MSNs) by the magnitude of their GCaMP8m activity at aggression onset (Fig. 3m), and then plotted their response to later stimulation of 5HT inputs (Fig. 3n). We found that the D1 MSNs that were more active during aggressive behavior were also more inhibited by 5HT release (Fig. 3o). In contrast, when we repeated the same experiments with D1-Cre × DAT-Flp mice (Supplementary Fig. 7a-c), we found that DA release predominantly excited D1 MSNs, with no preferential effects on aggression-activated neurons (Supplementary Fig. 7d-f). Taken together, our results imply that 5HT, but not DA, selectively inhibits a subset of aggression-active D1 MSNs to suppress attacks.

So far we have measured the activity of the same D1 MSNs in two separate sessions: first in a resident-intruder paradigm, and later with optogenetic stimulation of 5HT inputs. However, it is possible that the effect of 5HT on D1 MSN activity differs depending on context. Thus, we ran animals through a third session that combined optogenetic stimulation with the resident-intruder paradigm, allowing us to examine the effect of optogenetically-triggered 5HT release on D1 MSNs during aggression itself (Supplementary Fig. 8). Although 5HT release did not change the fraction of D1 MSNs that were activated at aggressive approaches (Supplementary Fig. 8d), it significantly suppressed their activity after attack onset (Supplementary Fig. 8e, f).

Methods
Mice
Wild-type C57BL6/J (Jackson Laboratory #000664, stock number: 017260-UCD), SERT-Flp (Jackson Laboratory #034050), DAT-Flp (Jackson Laboratory #035436), D1-Cre (MMRRC, #029178-UCD) and A2A-Cre (MMRRC, #036158-UCD) mice were used for breeding SERT-Flp+/−, D1-Cre+/−, A2A-Cre+/−, SERT-Flp × D1-Cre, SERT-Flp × A2A-Cre and DAT-Flp × D1-Cre mice for neural recording and manipulation. Adult male BALB/c mice (>8 weeks old, group-housed, 3–5 per cage) were used as intruders into the home cages of male resident mice (>8 weeks old, singly housed). All experimental mice were housed on a 12 h light/dark cycle in ambient temperature and humidity with food and water ad libitum. Behavioral tests were run in the dark phase. All procedures complied with the animal care standards set forth by the National Institute of Health and were approved by Stanford University’s Administrative Panel on Laboratory Animal Care.

Surgeries
General surgery procedure: mice (8-12 weeks old, male) were anaesthetized with isoflurane (5% induction, 1.5% maintenance) for surgeries. A stereotaxic frame (Kopf instruments) was used to target injections and implantations (AP and ML are relative to bregma and DV relative to the skull surface). The coordinates used for the NAc were the following in all experiments: AP: +1, ML: 0.7, DV: −4.4 mm. Viruses were infused with a syringe pump (Harvard Apparatus) at a rate of 150 nl/min and allowed to diffuse for at least 5 min. Implants were secured to the skull using screws (Antrin Miniature Specialties) with dental cement (Super-Bond C&B, Sun-Medica and Dual-Cure Resin-ionomer, Geristore). Optical implant placements were histologically verified post-hoc, and mice with mistargeted fibers or lenses were excluded from further analysis. Below lists the virus, target coordinates and implant details specific for different experiments.
For optogenetic stimulation of DR or ventral tegmental area (VTA) terminals, 1 μl AAV8-Syn-FLPX-rc [ChRimsonR-tdTomato] (Addgene) or AAV8-Ef1a-fDIO-rsChRmine-oScarlet-WPRE (Stanford viral core) was injected in the DR of SERT-Flp mice, or 0.5 ul of the same virus was injected in bilateral ventral tegmental area of DAT-Flp mice. rsChRmine was used in the optogenetic experiments for its high sensitivity46.
For optogenetic inhibition of D1 or D2 MSNs, 700 nl of AAVdj-EF1a-DIO-NpHR3.0-mcherry or AAVdj-Ef1a-DIO-mcherry (Stanford viral core) was injected bilaterally in the NAc of D1-Cre or A2A-Cre mice. Bilateral optical fibers (200–400 µm diameter, Doric) were implanted over the NAc.
For fiber photometry recordings of 5HT or DA, 700 nl of either AAV9-hSyn-5HT3.0 or AAV9-hSyn-GRABDA2m (WZ Biosciences) was injected in the NAc. An optical fiber (400 µm diameter, Doric) was implanted over the NAc for photometry recording.
For ‘Miniscope’ experiments, SERT-Flp × D1-Cre, SERT-Flp × A2A-Cre or DAT-Flp × D1-Cre mice were injected with 1 μl of AAV9-syn-FLEX-jGCaMP8m-WPRE (1:2 dilution with phosphate-buffered saline (PBS), Addgene) in the NAc, together with 1 μl of AAV8-Syn-FLPX-rc [ChRimsonR-tdTomato] (Addgene) injected in either the DR (AP: −4.6, ML: 0, DV: −3 mm) of SERT-Flp mice or the VTA (AP: −3.4, ML: 0.2, DV: −4.25 mm) of DAT-Flp mice. We chose ChRimsonR for the miniscope experiments to minimize potential crosstalk between the blue LED (435-460 nm wavelength for calcium imaging) and optogenetic stimulation47. A gradient index (‘GRIN’) lens (0.6 mm in diameter, 7.3 mm long, ProViewTM Integrated Lens, Inscopix) was implanted at the injection site in the NAc.
For drug microinjection, bilateral cannulae (P1 technologies) were implanted in the NAc of mice of wild-type background.

Fiber photometry
Mice with fiber implants were connected through low-autofluorescence patch cords and a dichroic cube (Doric #FMC6) to a photodetector (Newport #2151). GRAB sensors were excited by frequency-modulated LED light of 405 and 465 nm for near-isosbestic (used as control) and activity-dependent recordings, respectively. The resulting fluorescence signals were demodulated and recorded using a signal processor (RZ5P from Tucker-Davis Technologies running Synapse software, version 92-41665P) sampled at 1.0173 KHz. For experiments combining fiber photometry with optogenetic stimulation, 9–10 mW, 20 Hz pulses of red LED light (5 ms per pulse) were delivered in either 30 s trains (10 trains, each separated by 90 s; Fig. 3h, Supplementary Fig. 1k) or one 3 min train (Supplementary Fig. 1l). Raw data was converted into .mat files using software API from Tucker-Davis Technologies (TDT). For the aggression sessions, raw traces were down-sampled by 3x and de-bleached48 to get session-wise zscored fluorescence (Fzsc) which was then cropped and aligned to behavioral events (aggression approaches, attacks etc.).

Miniscope imaging and optogenetic stimulation
Calcium imaging and photostimulation were performed using a miniature microscope (nVoke 2.0, Inscopix)47. Blue LED light (0.5–1 mW, 435–460 nm) was used for imaging at 10 Hz per imaging plane (512 × 512 pixels per plane). Two planes separated by ~150 µm were imaged near-simultaneously.
For imaging sessions with MDMA/saline injections, mice were habituated to intraperitoneal injections a day before experiments. On the day of experiments, we recorded baseline activity for 5 min, administered either MDMA (7.5 mg/kg, Sigma-Aldrich) or saline intraperitoneally, waited 5 min, and then recorded for another 20 min. The sequence of MDMA or saline injection was counter-balanced across mice. Injections in the same mice were separated by at least 48 h.
For imaging sessions during resident-intruder tests and/or optogenetic stimulation, three recordings were completed on the same day with the same field-of-view. The first recording was performed during the resident-intruder test, without any photostimulation. In the second recording, 3 min of photostimulation (12 mW, 20 Hz) was delivered in the middle of a 9 min resident-intruder assay (with a different intruder). In the third recording, 10 bouts of photostimulation were delivered while the animal was alone in its home cage; these bouts consisted of 30 s, 12 mW, 20 Hz pulses of red LED light (590–650 nm, 5 ms per pulse) separated by 90 s. The order of the first and second recordings was counterbalanced across days, and each mouse experienced two days of experiments (Supplementary Fig. 8a).

Miniscope imaging data analysis
Raw imaging data were converted into H5 data files using API from Inscopix (Bruker). Sessions recorded on the same day from the same field-of-view were concatenated before motion correction and neural signal extraction. Piecewise rigid motion correction was applied to each imaging plane49. Neuronal shapes, denoised calcium activity traces, and inferred spikes were extracted by a constrained nonnegative matrix factorization (CNMF) method developed for endoscopic datasets50. To avoid double counting the neurons that were captured in both imaging planes, the detected ROIs across planes that were spatially close (centroids < 10 pixels apart) and also had high activity correlation (> 0.5) were merged into one. The calcium traces were z-scored for each session and aligned to either behavior timestamps (aggressive approach, attack, etc.) or photostimulation onsets. These z-scores were then baseline-corrected to produce ΔFzsc, with the baseline defined as either the 1 s window starting 2 s before the behavior, or the 10 s window before photostimulation onset. Timestamps within 3 s of a prior behavioral event were ignored to avoid confounding baseline subtraction with behavior-related activity.
To identify aggression-modulated neurons, the aggression-aligned activity (averaged in a 2 s window following each aggressive approach) was compared to a distribution of activity values aligned to 1000 random timestamps. A neuron was considered to be aggression-modulated if its aggression-aligned activity was higher or lower than 90% of the random-timestamp-aligned activity values. For sessions with drug or saline injections, the inferred spike rate from each neuron was normalized by that during the baseline period.

Optogenetic manipulations during resident-intruder
For optogenetic manipulations, mice with optical implants were connected to a 625 nm LED light source (Prizmatix) via a plastic fiber (1 mm diameter, NA 0.63) and a fiber optic rotary joint (Doric). For photostimulation of DA or 5HT inputs to the NAc, mice underwent two days of optogenetics experiments. On each day, they performed two 9-minute sessions separated by >1 h, one with photostimulation and one without. The order of photostimulation was counterbalanced between days. The photostimulation (3 min at 20 Hz, 12 mW, 5 ms pulse width) was initiated 3 min after intruder entry. For photoinhibition of D1 or D2 MSNs in the NAc, mice likewise underwent two sessions, separated by >1 h. In the second session, we delivered a 9-minute train of 0.4 Hz, 8 mW, 2 s light pulses. The duration of optogenetic terminal stimulation was chosen to balance two factors: it had to be long enough to capture enough aggressive bouts to test the behavioral effect, but short enough to avoid opsin desensitization. Similar durations of 5HT terminal stimulation (5 min for three-chamber sociability and 2 min for juvenile interaction) produced pro-social effects17, so we hypothesized that 3 min elevations of 5HT levels in the NAc would similarly influence attacking behavior. Video recordings and optogenetic stimulations were synchronized by TTL triggers from a data acquisition card (USB6259, National Instrument) controlled by PackIO51.

Drug injections
For intracranial drug microinjection, two 9-minute sessions were performed, separated by >1 h. 15 min before the second session, mice received a bilateral intra-NAc infusion of either saline with 1% DMSO, or one of the selective 5HT receptor agonists: 0.5 µg CP93129 diluted in saline targeting 5HT1b receptors; 1 µg LY344864 diluted in 1% DMSO targeting 5HT1f receptors; or 0.5 µg WAY163909 diluted in saline targeting 5HT2c receptors. Drugs were infused through an injector cannula using a microinfusion pump (Harvard Apparatus) at a continuous rate of 0.1 µl per min to a total volume of 0.3 µl per hemisphere. Injector cannulae were removed 2 min after infusions were complete. Injections in the same mice were separated by at least 48 hours.
For systemic MDMA/saline injections, mice were habituated to intraperitoneal injections a day before experiments. On one day, either MDMA (7.5 mg/kg, Sigma-Aldrich) or saline was administered intraperitoneally 15 min before the behavioral tests. The sequence of MDMA or saline injection was counter-balanced across mice. Injections in the same mice were separated by at least 48 h.

Resident-intruder tests
Mice were habituated to the recording arena (with fiber connection for optogenetic experiments) by performing a daily resident-intruder test until they attacked the intruder within 5 min after the intruder was introduced, and more than 2 times in 9 min on two consecutive days. Mice for microinjection experiments underwent a sham injection handling before these habituation sessions. 32 mice that did not reach the aggression criteria after 1 week were excluded from further experiments. On rare occasions when an intruder was aggressive towards the resident, the experiment was immediately aborted, and the intruder was not used for further experiments. For each test mouse, a different intruder was used for control (saline or no-light) and test (drug-treated or light-on) conditions. All resident-intruder sessions were 9 min long. We did not observe any injuries to the intruder that would require capping the number of bites or ending the session early. For detailed descriptions of how we combined resident-intruder tests with imaging or manipulations, please see the relevant sections above.

Resident-intruder behavior analysis
Behavioral chambers were illuminated with dim red light and videos were taken by an overhead camera (FLIR Blackfly) at 20 Hz frame rate in SpinView software (version 2.6.0.157, Teledyne Vision Solutions). Mouse postures were extracted from behavior video recordings using ‘DeepLabCut’52. For sessions with neural activity recordings, to get the exact onset time of attacks, videos were manually labelled by experimenters who were blinded to experimental condition or simultaneously-collected imaging data. To obtain more objective timestamps of behaviorally-relevant epochs in our neural data, we defined the onsets/offsets of aggression episodes (i.e. the ‘aggressive approaches’ and ‘aggression offsets’) as the timepoints when the two mice became close to (<5 cm) or far away from (>5 cm) each other before/after attacks. To reduce confounds in baselining, we only analyzed bouts that lasted at least 2 s and were separated by at least 2 s from a prior bout. For sessions without neural activity recordings, we used a machine learning based software, ‘SIMBA’53, to identify attacking behavior (Supplementary Notes, Supplementary Video 1). The detected attacking bouts that were <0.25 s apart were merged into one, and those that were shorter than 0.5 s were filtered out. The behavioral metrics (duration per attack, sniffs per minute etc.) under LightOff and LightOn conditions were divided by the corresponding value in the LightOff condition to get a percentage change (%).

Open field test
Mice were placed into a 40 × 40 cm behavioral arena and allowed to explore freely for 12 min. Optogenetic manipulations were delivered in four alternating light-on (0.4 Hz, 8 mW, 2 s light pulses for NpHR3.0 stimulation) and light-off epochs that lasted 3 min each. Epoch order was counterbalanced across mice. Biobserve Viewer behavioral tracking program (version 3.0.1.442) was used to extract distance traveled during each epoch.

Real-time place preference
Real-time place preference (RTPP) tests were performed in an arena with three chambers as previously described10: a neutral center chamber with clear floors, and two identical side chambers with white walls and floors. Mice were confined to the center chamber for a ~2 min habituation period. Then, the barriers were removed to begin the first half of the test during which mice were free to explore all three chambers for 15 min and one side chamber (left or right, counterbalanced) was paired with photostimulation (20 Hz, 12 mW, 5 ms pulses for terminal ChRimsonR or rsChRmine stimulation, or 0.4 Hz, 8 mW, 2 s light pulses for cell-body NpHR3.0 stimulation) that began when the mouse entered the light-paired chamber and continued for the entire duration that the mouse occupied the chamber. After 15 min, the side that was paired with photostimulation was swapped and mice were free to explore all three chambers for an additional 15 min reversal phase.

Three-chamber sociability assay
A three-chamber sociability assay was performed in an arena with three separate chambers as previously described17. On day one, the test mice were habituated to the arena, with two empty wire mesh cups placed in the two outer chambers for 5 min. On day two, the test mouse was placed in the center chamber and a conspecific juvenile (3–5-week-old males) was placed under one of the wire mesh cups. The test mice were first placed in the center chamber for 2 min. The barriers were then raised and the test mouse was allowed to explore freely for a 20 min session with 5 min epochs of photostimulation (0.4 Hz, 8 mW, 2 s light pulses for cell-body NpHR3.0 stimulation). The order of these epochs (LightOn or LightOff) was counterbalanced across mice. The placement of juvenile mice in the chamber was also counterbalanced across sessions.

Histology
Mice were transcardially perfused with 4% (w/v) paraformaldehyde (PFA) in phosphate-buffered saline (PBS) and postfixed in PFA overnight. Coronal sections 70 µm thick were immunostained for GFP (primary: chicken anti-GFP, Aves 608 #GFP-1020; secondary: goat anti-chicken 488, Invitrogen #A-11039), and/or mCherry (primary: rat anti-mCherry, Invitrogen #M11217; secondary: goat anti-rat 594, Invitrogen #11007). Primary antibodies were used at a concentration of 1:1000 and incubated on a shaker overnight at room temperature. Secondary antibodies were used at a concentration of 1:750 and incubated on a shaker for 2 hours at room temperature. Anatomical microscopy images were collected using BZ-X800 Viewer software (Keyence).

Statistics and reproducibility
Statistical analyses were performed in MATLAB (R2022b and R2021a, MathWorks) and Python (3.8 and 3.9). No statistical methods were used to predetermine sample sizes, which were based on previous experience with the variance of the assays. Statistical parameters such as the types of statistical tests, sample numbers (n) and statistical significance are reported in the figures and figure legends. T-tests were used to test if a sample mean was different from zero. Two-sided Wilcoxon rank-sum tests were performed to compare unpaired measurements between two groups, and two-sided Wilcoxon signed-rank tests were conducted to compare paired measurements. For multi-group comparisons, we conducted one- or two-way analyses of variance followed by Tukey’s honestly significant difference procedure. All statistical tests were two-sided. All error bars in the figures are s.e.m. unless otherwise noted.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The forebrain-projecting serotonin system has been found to be heterogeneous, with projection-defined serotonin subsystems responsible for distinct behavioral functions. It is also well established that serotonin exerts an inhibitory effect on aggression. However, the circuit-level mechanisms through which serotonin modulates aggression remain unclear. In this study, Zhang et al. propose that serotonin suppresses attacks by inhibiting a subset of D1 medium spiny neurons (MSNs). The authors combine fiber photometry and optogenetics to demonstrate a correlation between 5-HT levels in the nucleus accumbens (NAc) and aggressive behavior in mice. They further show that increasing 5-HT levels in the NAc is sufficient to reduce the duration of aggressive episodes. Using miniature microscope recordings, they identify a subset of D1 MSNs whose activity correlates with aggression duration; notably, the D1 MSNs that were more active during aggressive behavior were also more inhibited by 5-HT release. Overall, the findings are compelling, but several concerns need to be addressed.\nMajor Concerns:\nThe narrative progression of the study appears rushed, and several key perspectives are not properly discussed.\n1. In Figure 2, the authors report that MDMA injection selectively affects D1 MSNs without influencing D2 MSN activity. However, a 2019 study [1] using whole-cell voltage-clamp recordings found that MDMA induces long-term depression (LTD) of excitatory postsynaptic currents (EPSCs) in both D1 and D2 MSNs. This discrepancy raises concerns and suggests that MDMA may impact both neuron types, contrary to the authors’ interpretation.\n2. Previous studies have shown that inhibition of D1-, but not D2-expressing neurons in the NAc, reduces aggression-seeking and self-administration behaviors [2]. The current study builds upon this by linking 5-HT release to selective modulation of D1 MSNs. The manuscript would be strengthened if the authors provided insight into the underlying mechanism. For instance, is there evidence that D1 MSNs express higher levels of 5-HT receptors—such as 5-HT1B receptors—which have been implicated in the regulation of aggression [3]?\n3. Is there existing literature that distinguishes serotonin receptor expression between D1 and D2 MSNs? Several studies have reported single-cell sequencing analyses in the NAc that may be relevant.\n4. If dopamine inhibits all D1 MSNs, why is there no observable effect on aggressive behavior? Could the authors propose a hypothesis to address this apparent discrepancy?\nMinor Concerns:\n1. Previous studies have suggested a role for dopamine (DA) signaling in the NAc in modulating aggression [4,5]. However, in this manuscript, the authors do not observe significant behavioral effects following optogenetic stimulation of DA release in the NAc (Extended Figure 1), nor do they detect clear activation of D1 MSNs after stimulating VTA DA neurons (Extended Figure 3). This raises questions about the effectiveness of their optogenetic approach. It is recommended that the authors provide validation demonstrating that their stimulation protocol successfully induces DA release in the NAc.\n2.\n3. There is a typo in line 116. The sentence currently reads: “…ramp up during aggressive episodes, peaking at aggression onset…”. Based on the study’s conceptual framework, this should likely read: “…peaking at aggression offset…”.\nReferences\n[1]. Heifets, Boris D et al. “Distinct neural mechanisms for the prosocial and rewarding properties of MDMA.” Science translational medicine vol. 11,522 (2019): eaaw6435. doi:10.1126/scitranslmed.aaw6435\n[2]. Golden, Sam A et al. “Nucleus Accumbens Drd1-Expressing Neurons Control Aggression Self-Administration and Aggression Seeking in Mice.” The Journal of neuroscience : the official journal of the Society for Neuroscience vol. 39,13 (2019): 2482-2496. doi:10.1523/JNEUROSCI.2409-18.2019\n[3]. de Almeida, R., Miczek, K. Aggression Escalated by Social Instigation or by Discontinuation of Reinforcement (“Frustration”) in Mice: Inhibition by Anpirtoline: A 5-HT1B Receptor Agonist. Neuropsychopharmacol 27, 171–181 (2002).\nhttps://doi.org/10.1016/S0893-133X(02)00291-9\n[4]. van Erp, A. M. & Miczek, K. A. Aggressive behavior, increased accumbal dopamine, and decreased cortical serotonin in rats. J. Neurosci. 20, 9320–9325 (2000).\n5. Rodriguiz, R. M., Chu, R., Caron, M. G. & Wetsel, W. C. Aberrant responses in social interaction of dopamine transporter knockout mice. Behav. Brain Res. 148, 185–198 (2004).", "reviewer_2": "The present manuscript from Zhang et al. details the effects of monoaminergic signaling manipulations on DRN-NAc 5HT projections and NAc D1- vs. D2-MSNs. The strength of this manuscript is in the use of biosensors to uncover unique attack-relative signatures of 5HT vs. DA release in the NAc. Namely, DA ramps up during attacks whereas 5HT ramps up during the conclusion of aggressive interactions. While these findings are extremely interesting, there are numerous issues with subsequent experimental designs and their corresponding interpretations. These issues are noted below along with several minor queries and recommendations:\nMajor:\n1.Experiments depicted in Figure 1: To consider MDMA a specific drug that serves as a “5HT manipulation” would be overlooking its non-specific but notable influence on DA signaling. It is unclear why the authors chose MDMA rather than a serotonin-specific compound, e.g., d-fenfluramine. This is particularly confounding, considering the authors make claims regarding specific effects of serotonin on D1 vs. D2 MSN activity.\n2.Experiments depicted in Figure 2: Line 85 – first sentence of this paragraph states that the following experiments test if 5HT reduces D1-MSN activity to suppress attacks. Yet, neither aggression nor direct 5HT inputs from DRN NAc(D1-MSN) are evaluated. From the text and corresponding figure (Fig. 2a-d), it seems miniscope recordings were collected without simultaneously aggression evaluations. Moreover, MDMA was administered systemically and while MDMA may increase NAc 5HT as the authors point out, it also increases 5HT in the numerous other projection targets of DRN 5HT neurons. Finally, similar to its effects on serotonin signaling, MDMA also increases release and blocks reuptake of DA, so it is quite likely that the MDMA’s effects on D1 vs. D2 MSN cell activity is the result of combined action via 5HT and DA (and likely NE as well). In sum, the changes in D1 MSN calcium-dependent fluorescence (GCaMP8m signal) reported after systemic MDMA administration could be the outcome of any number of the other monoaminergic pathways that either directly or indirectly influence D1-MSNs.\n3.Based on the design of experiments depicted in Fig. 2m and 2n, the authors appear to assume that network-level brain activity is irrelevant to the effects of their local optogenetic manipulations. This contradicts increasing evidence to suggest that real-time network-level brain state mediates the influence of neuromodulators like 5HT. As such, it is unclear how to interpret the effects of optogenetic stimulations when they are introduced outside of an aggressive context. Simply because a cell ensemble is dynamic during aggression doesn’t mean that the cells comprising this ensemble serve a single aggression-associated purpose. When animals are not engaged in aggression, it is most likely that these same cells have alternative functions associated with the current (non-aggressive) network state. In sum, optogenetic manipulations must be introduced during aggression to understand their impact on aggression and on the cells that are dynamic during a fight.\n4.It is unclear why behavioral data are portrayed as % change from control, e.g., in Fig 1j, 1k, 1s, 1t. It is ideal to provide readers with the raw data values (even if only in the supplement) for these behaviors so they may understand the findings in the broader context of past literature and to interpret the practical significance of the manipulation outcome.\n5.Figure 2h is confusing as there is no explanation of an associated surgical or experimental approach in the text or figure caption – are these transgenic mice and if so, are they D1-Cre:SERT-Flp or A2A-Cre:SERT-Flp mice? Did they receive the virally-encoded 5HT biosensor instead of FLEX-GCaMP8m? Where - intra-NAc? Which red-shifted opsin construct and where - DRN? Were DRN cell bodies or DRN  NAc terminals photostimulated?\nMinor:\n1.In referencing the resident-intruder aggression test in mice, the authors should cite Miczek and O’Donnell 1978 (https://doi.org/10.1007/BF00426957). Similarly, the authors should acknowledge existing literature demonstrating anti-aggressive effects of systemically administered MDMA (e.g., Miczek and Haney 1994).\n2.LED wavelengths should be specified in nm rather than as “blue” or “red” in the miniscope methods section.\n3.Traditionally, control conditions are depicted as the left-most bar in histograms rather than the right-most as in Fig. 1s and 1t – might be worth considering changing unless there’s a reason to portray the data as-is.\n4. For the blown-up trace showing alignment with attack onset and aggression offset (Fig 1a/b), the authors should consider including a scale bar for time and labeling the scale bar for Z scores.\n5. Main text, line 56, please add something along the lines of “and implanted optic fibers terminating in the NAc.”", "reviewer_3": "Review Zhang et al., 2025\nIn their manuscript “Serotonin modulates nucleus accumbens circuits to suppress aggression”, Zhang et al describe a novel mechanism for serotonin to modulate aggression, whereby serotonin input into the nucleus accumbens inhibits a subpopulation of D1 medium spiny neurons to suppress aggressive behavior after fighting has initiated. Experiments are cleverly structured and the inclusion of MDMA, albeit mechanistically dirty is translationally interesting.\nSpecifically, the authors report that serotonin release (GRAB-5HT) in the NAc ramps up during an attack and ramps down with aggression offset. The ramp-up slope inversely correlates with attack duration.\n1   what about the peak, ramp-down slope, and AUC?\nAugmenting serotonin release (MDMA or ChRimimsonR or rsChRmine) reduces attack duration, demonstrating that serotonin release in the NAc is sufficient to inhibit attack behavior. Photostimulation of DA release in the NAc had no effect on aggressive behavior.\n2   please explain the use of ChRimsonR versus rsChRmine.\n3   an open field control would be great here (see MSN inhibition and also Walsh et al). is the effect of serotonin specific to aggression behavior? Is NAc serotonin release aversive? Are social or goal-directed behaviors affected more generally? (see MSN inhibition. See also Walsh et al and Rein et al).\n4   photostimulation for behavior was initiated 3 minutes after intruder entry and lasted for 3 minutes. This temporal dynamic of experimentally induced serotonin release is very different from the identified tight temporal dynamic of endogenous serotonin release during aggression behavior. Why did the authors choose such different parameters? A rationale should be provided in the main text.\nUsing a GRIN lens GCaMP8m approach, the authors find that the activity of D1 MSNs (but not D2 MSNs) in the NAc positively correlates with aggression episode duration. Inhibiting D1 MSNs in the NAc (NpHR3.0) reduces attack duration, while inhibiting D2 MSNs in the NAc (NpHR3.0) has no effect on attack duration, indicating a necessary role for NAc D1 MSN activity in attack duration.\n7   the inhibition protocol lasts the entire 9 minutes of the encounter. Is the effect of inhibition towards the beginning different than towards the end (effect of time? Correlation with time?)\nThe authors finish their manuscript with a beautiful in vivo assessment of the effect serotonin has on MSNs. They find that MDMA or optogenetic stimulation of serotonin release in the NAc inhibitsD1 (but not D2) MSNs. Moreover they found that D1 MSNs that were more active during attack were also more sensitive to the inhibitory effect of serotonin release.\nTaken together this is a really nice manuscript with elegant techniques and novel findings, but I feel that addressing a couple of open questions will allow for much tighter conclusions."}, "author_response": "Nature Communications Manuscript NCOMMS-25-26724-T\nResponse to Referees\nWe are pleased to submit a revised version of our paper with the title, “Serotonin modulates\nnucleus accumbens circuits to suppress aggression.” We thank the reviewers for their helpful\nfeedback. In particular we are pleased that Reviewer #1 found that our ‘findings are\ncompelling’; Reviewer #2 considered that our ‘findings are extremely interesting’; and\nReviewer #3 stated that our findings are ‘translationally interesting’ and that our paper is ‘a\nreally nice manuscript with elegant techniques and novel findings.’\nWe have taken their feedback seriously and have attempted to address each point. In our\nrevision we have performed a large number of new experiments and analyses, provided more\ntechnical details, rewritten experimental rationales and interpretations, added extensive\ndiscussion, and corrected small errors in the text and figures. The most important revisions\ninclude:\n1. New experiments showing that serotonin terminal stimulation during miniscope\nrecordings inhibited aggression-activated D1 MSNs (Supplementary Figure 8).\n2. New experiments showing that local infusion of a 5HT1b receptor agonist in bilateral\nNAc significantly reduced aggression (Supplementary Figure 5)\n3. New experiments validating the effectiveness of ChRimsonR in triggering DA release\n(Supplementary Figure 1j-l)\n4. New analyses and experiments showing that 5HT terminal stimulation in the NAc did\nnot modulate running speed or non-aggressive social behaviors during resident-\nintruder tests; nor did it elicit real-time place preference (Supplementary Figure 4).\n5. New experiments showing that D1 and D2 MSN inhibition in the medial shell of the\nNAc did not affect performance on RTPP and sociability tests (Supplementary Figure\n6f,g,i,j).\n6. New analyses showing the correlation between 5HT signals and aggression duration\n(Supplementary Figure 2).\n7. Inclusion of raw, non-normalized behavioral data (Supplementary Figure 3).\n8. Substantially expanded discussion to more accurately reflect our findings and their\nplace in the greater literature\nIn total, there are five entirely new supplementary figures, new data added to all existing\nsupplementary figures, as well as numerous other changes to the text. Below, in blue font, are\nour point-by-point responses to the reviewers’ comments.\nReviewer #1\nThe forebrain-projecting serotonin system has been found to be heterogeneous, with\nprojection-defined serotonin subsystems responsible for distinct behavioral functions. It is also\nwell established that serotonin exerts an inhibitory effect on aggression. However, the circuit-\nlevel mechanisms through which serotonin modulates aggression remain unclear. In this study,\nZhang et al. propose that serotonin suppresses attacks by inhibiting a subset of D1 medium\nspiny neurons (MSNs). The authors combine fiber photometry and optogenetics to\ndemonstrate a correlation between 5-HT levels in the nucleus accumbens (NAc) and\naggressive behavior in mice. They further show that increasing 5-HT levels in the NAc is\nsufficient to reduce the duration of aggressive episodes. Using miniature microscope\nrecordings, they identify a subset of D1 MSNs whose activity correlates with aggression\nduration; notably, the D1 MSNs that were more active during aggressive behavior were also\nmore inhibited by 5-HT release. Overall, the findings are compelling, but several concerns\nneed to be addressed.\nWe are grateful that the reviewer thought our findings are compelling. We thank the reviewer\nfor the thoughtful comments, which we have addressed point-by-point below.\nMajor Concerns:\nThe narrative progression of the study appears rushed, and several key perspectives are not\nproperly discussed.\nWe have substantially expanded the narrative to discuss these key perspectives, as described\nbelow.\n1. In Figure 2, the authors report that MDMA injection selectively affects D1 MSNs without\ninfluencing D2 MSN activity. However, a 2019 study [1] using whole-cell voltage-clamp\nrecordings found that MDMA induces long-term depression (LTD) of excitatory postsynaptic\ncurrents (EPSCs) in both D1 and D2 MSNs. This discrepancy raises concerns and suggests\nthat MDMA may impact both neuron types, contrary to the authors’ interpretation.\nWe thank the reviewer for pointing this out. Several factors may contribute to the different\nobservations between the previous and current study. First, the MSNs in ex vivo slices are\nlikely to be in a different physiological state than in vivo conditions, which may result in different\nresponses to neuromodulators. Second, since long-range inputs are largely severed in the\nslice preparation, the effect of DA and 5HT on glutamatergic inputs, and the amount of 5HT\nand DA release induced by MDMA, may all be different from our in vivo measurements,\nresulting in different downstream activity. Third, subthreshold EPSCs are different than the\ncalcium activity we measure here. Fourth, even if MDMA reduces activity in a subset of D2\nMSNs, the population activity may remain unchanged due to the GABAergic inter-connections\nbetween D2 MSNs. Finally, the prior study recorded from a different region in the NAc (anterior\ncore rather than posterior medial shell). In summary, both experimental conditions and\nmeasurements are different in the two studies, which could explain any discrepancies. We\nhave added discussions in the main text that cite this prior study and temper the strength of\nour conclusions.\n2. Previous studies have shown that inhibition of D1-, but not D2-expressing neurons in the\nNAc, reduces aggression-seeking and self-administration behaviors [2]. The current study\nbuilds upon this by linking 5-HT release to selective modulation of D1 MSNs. The manuscript\nwould be strengthened if the authors provided insight into the underlying mechanism. For\ninstance, is there evidence that D1 MSNs express higher levels of 5-HT receptors—such as\n5-HT1B receptors—which have been implicated in the regulation of aggression [3]?\nWe thank the reviewer for the insightful comment. Our lab has indeed investigated the\nexpression of 5HT receptors on D1 and D2 MSNs (published in a preprint: Pinto et al, 2025,\nhttps://doi.org/10.1101/2025.04.17.649431). From both single-cell RNA sequencing and\nfluorescence in situ hybridization datasets, we found that D1 MSNs in the medial NAc express\nhigher levels of Gi-coupled receptors (including 5-HT1b) than D2 MSNs, whereas D2 MSNs\nexpress more Gq-coupled receptors. We have extended our discussion to include this putative\nmechanism.\nTo further examine which 5HT receptor might be involved in regulating aggression in the NAc,\nwe have performed a new set of experiments where either a 5HT receptor agonist or vehicle\nsolution was directly infused into the NAc before the resident-intruder test. Specifically, we\ntested 5HT1b and 5HT1f receptors, which are preferentially expressed on D1 MSNs and\npresynaptic terminals, and 5HT2c receptors, which are preferentially expressed on D2 MSNs\n(Figure 1k in the Pinto et al. preprint). We found that local infusion of a 5HT1B receptor agonist\nsignificantly reduced aggression behavior while infusion of the two other agonists did not. We\nhave added this new result in a new Supplementary Figure 5, copied below:\n3. Is there existing literature that distinguishes serotonin receptor expression between D1 and\nD2 MSNs? Several studies have reported single-cell sequencing analyses in the NAc that may\nbe relevant.\nThere is indeed differential expression of 5HT receptors in D1 and D2 MSNs. Our lab recently\nsubmitted a separate study (Pinto et al, bioRxiv, 2025. doi: 10.1101/2025.04.17.649431) that\nmined a single-cell RNA sequencing dataset and found biased expression of Gi-coupled\nreceptors in the D1 MSNs (also see response to point 2 above). We have now cited this other\nstudy in an extended discussion of putative mechanisms.\n4. If dopamine inhibits all D1 MSNs, why is there no observable effect on aggressive behavior?\nCould the authors propose a hypothesis to address this apparent discrepancy?\nWe found that DA excites D1 MSNs, not inhibits them, regardless of their activity during\naggression. As the reviewer correctly notes, we found that stimulation of VTA DA terminals in\nthe NAc had no observable effect on aggression. This is consistent with all previous attempts\nat this manipulation that we could find in the literature (Dai et al, Nature, 2025, DOI:\n10.1038/s41586-024-08459-w, Mahadevia et al, Nature Communication, 2021, DOI:\n10.1038/s41467-021-27092-z). As noted above, at the same time, DA release did excite D1\nMSNs. This is an interesting discrepancy, and we do not have a full explanation for it. One\npossibility has to do with which particular D1 MSNs respond to DA vs 5HT. Only a subset of\nD1 MSNs are active during aggression, and this is the same subset that is inhibited by 5HT.\nPerhaps this preferential inhibition of aggression-relevant D1 MSNs is particularly effective at\nreducing aggressive behavior. In contrast, DA activates essentially all D1 MSNs, regardless\nof their activity during aggression. Perhaps this non-selective activation produces a global\nincrease in D1 MSN excitability, leaving the relative strength of the aggression signal\nunchanged. We have now added this speculation to the discussion.\nMinor Concerns:\n1. Previous studies have suggested a role for dopamine (DA) signaling in the NAc in\nmodulating aggression [4,5]. However, in this manuscript, the authors do not observe\nsignificant behavioral effects following optogenetic stimulation of DA release in the NAc\n(Extended Figure 1), nor do they detect clear activation of D1 MSNs after stimulating VTA\nDA neurons (Extended Figure 3). This raises questions about the effectiveness of their\noptogenetic approach. It is recommended that the authors provide validation demonstrating\nthat their stimulation protocol successfully induces DA release in the NAc.\nWe have performed validation experiments to test if our stimulation protocol was able to induce\nDA release in the NAc. Specifically, we recorded GRAB-DA signal in the NAc during\noptogenetic stimulation of DA terminals (new Supplementary Figure 1j-l). We found that\nboth 30-second stimulations (as used in the miniscope experiments in Supplementary Figure\n7) and 3-minute stimulations (as used in optogenetic behavioral experiments in\nSupplementary Figure 1g-j) were effective in increasing DA levels in the NAc.\nThe fact that DA axon stimulation in the NAc did not affect aggression is consistent with prior\nstudies that have performed the same manipulation (Dai et al, Nature, 2025, DOI:\n10.1038/s41586-024-08459-w, Mahadevia et al, Nature Communication, 2021, DOI:\n10.1038/s41467-021-27092-z). We also did find an activation of D1 MSNs after stimulating\nVTA DA neurons (Supplementary Figure 7c).\n2. There is a typo in line 116. The sentence currently reads: “…ramp up during aggressive\nepisodes, peaking at aggression onset…”. Based on the study’s conceptual framework, this\nshould likely read: “…peaking at aggression offset…”.\nWe thank the reviewer for pointing this out. We have now corrected the text.\nReviewer #2\nThe present manuscript from Zhang et al. details the effects of monoaminergic signaling\nmanipulations on DRN-NAc 5HT projections and NAc D1- vs. D2-MSNs. The strength of this\nmanuscript is in the use of biosensors to uncover unique attack-relative signatures of 5HT vs.\nDA release in the NAc. Namely, DA ramps up during attacks whereas 5HT ramps up during\nthe conclusion of aggressive interactions. While these findings are extremely interesting, there\nare numerous issues with subsequent experimental designs and their corresponding\ninterpretations. These issues are noted below along with several minor queries and\nrecommendations:\nWe thank the reviewer for this very positive review and respond in detail to all the specific\nissues below.\nMajor:\n1.Experiments depicted in Figure 1: To consider MDMA a specific drug that serves as a “5HT\nmanipulation” would be overlooking its non-specific but notable influence on DA signaling. It\nis unclear why the authors chose MDMA rather than a serotonin-specific compound, e.g., d-\nfenfluramine. This is particularly confounding, considering the authors make claims regarding\nspecific effects of serotonin on D1 vs. D2 MSN activity.\nWe apologize for our lack of clarity here. We used MDMA for its known prosocial effect and\ntranslational relevance. The reviewer is absolutely right that due to its influence on DA, we\ncannot use MDMA for testing 5HT-specific effects. This is why we created double-transgenic\nmouse lines, SERT Flp x D1 Cre and SERT Flp x A2A Cre to gain genetic access to 5HT cells\nand D1 or D2 MSNs (Figure 3e). We then expressed Flp-dependent opsin in the DR and Cre-\ndependent GCaMP in the NAc to test specific effects of serotonin on D1 vs D2 MSNs (Figure\n3f-o).\nTo avoid confusion, we have rewritten the paragraph describing the MDMA experiments, and\nmade it clear that all subsequent experiments used serotonin-specific manipulations.\n2.Experiments depicted in Figure 2: Line 85 – first sentence of this paragraph states that the\nfollowing experiments test if 5HT reduces D1-MSN activity to suppress attacks. Yet, neither\naggression nor direct 5HT inputs from DRNNAc(D1-MSN) are evaluated. From the text and\ncorresponding figure (Fig. 2a-d), it seems miniscope recordings were collected without\nsimultaneously aggression evaluations. Moreover, MDMA was administered systemically and\nwhile MDMA may increase NAc 5HT as the authors point out, it also increases 5HT in the\nnumerous other projection targets of DRN 5HT neurons. Finally, similar to its effects on\nserotonin signaling, MDMA also increases release and blocks reuptake of DA, so it is quite\nlikely that the MDMA’s effects on D1 vs. D2 MSN cell activity is the result of combined action\nvia 5HT and DA (and likely NE as well). In sum, the changes in D1 MSN calcium-dependent\nfluorescence (GCaMP8m signal) reported after systemic MDMA administration could be the\noutcome of any number of the other monoaminergic pathways that either directly or indirectly\ninfluence D1-MSNs.\nWe would like to clarify that although the top row of the old Figure 2 (now Figure 3, panels a-\nd) refers to experiments with systemic MDMA, the remainder of the figure (panels e-o) all\ninvolve direct optogenetic stimulation of 5HT inputs to the NAc. We apologize that this was\nnot sufficiently clear in the original submission, and we have attempted to clarify the text in\nthis resubmission by changing the first sentence of the paragraph and separating the\nconclusions from MDMA vs optogenetic manipulations. Because the optogenetic\nmanipulations were in transgenic mice, we selectively targeted 5HT (and not DA) inputs to the\nNAc.\nIn terms of the timing of the experiments, our miniscope recordings were completed both\nbefore and during aggression sessions. We used the pre-aggression recordings to measure\nthe effects of optogenetically-evoked 5HT release on MSN activity at ‘baseline’, when animals\nwere alone in their cage. During aggression sessions, we then recorded the same MSNs, with\nand without optogenetic stimulation. In the original submission, we reported that the same D1\nMSNs that were inhibited by optogenetic stimulation at ‘baseline’ were activated during\naggressive bouts. In this revision, we have added new analyses of the effects of optogenetic\nstimulation on MSN activity during aggression sessions (new Supplementary Figure 8).\nConsistent with our prior findings, we show that serotonin terminal stimulation during\nagression reduces the activity of aggression-active D1 MSNs.\n3.Based on the design of experiments depicted in Fig. 2m and 2n, the authors appear to\nassume that network-level brain activity is irrelevant to the effects of their local optogenetic\nmanipulations. This contradicts increasing evidence to suggest that real-time network-level\nbrain state mediates the influence of neuromodulators like 5HT. As such, it is unclear how to\ninterpret the effects of optogenetic stimulations when they are introduced outside of an\naggressive context. Simply because a cell ensemble is dynamic during aggression doesn’t\nmean that the cells comprising this ensemble serve a single aggression-associated purpose.\nWhen animals are not engaged in aggression, it is most likely that these same cells have\nalternative functions associated with the current (non-aggressive) network state. In sum,\noptogenetic manipulations must be introduced during aggression to understand their impact\non aggression and on the cells that are dynamic during a fight.\nWe thank the reviewer for the constructive comment. We have indeed performed MSN activity\nrecordings with and without optogenetic stimulation when mice were engaged in aggression.\nThe combined miniscope/optogenetic experiments were all performed unilaterally, so we did\nnot expect a robust behavioral effect, like we found in separate experiments with bilateral\noptogenetic manipulations. But in new analyses for this revision, we did find that the D1 MSNs\nthat were activated during aggressive approach were suppressed during stimulation of\nserotonin terminals (new Supplementary Figure 8, copied above in response to referee\ncomment #2).\nWe have added this new data and extended our discussion to include the reviewer’s point\nabout the relevance of ongoing behavior and brain activity to the effects of optogenetic\nmanipulations.\n4.It is unclear why behavioral data are portrayed as % change from control, e.g., in Fig 1j, 1k,\n1s, 1t. It is ideal to provide readers with the raw data values (even if only in the supplement)\nfor these behaviors so they may understand the findings in the broader context of past\nliterature and to interpret the practical significance of the manipulation outcome.\nWe have added raw behavioral data to new Supplementary Figure 3, copied below:\n5.Figure 2h is confusing as there is no explanation of an associated surgical or experimental\napproach in the text or figure caption – are these transgenic mice and if so, are they D1-\nCre:SERT-Flp or A2A-Cre:SERT-Flp mice? Did they receive the virally-encoded 5HT\nbiosensor instead of FLEX-GCaMP8m? Where - intra-NAc? Which red-shifted opsin construct\nand where - DRN? Were DRN cell bodies or DRN NAc terminals photostimulated?\nWe apologize for the lack of clarity. The figure shows intra-NAc GRAB-5HT signals in D1-\nCre:SERT-Flp or A2A-Cre:SERT-Flp mice that were injected with GRAB-5HT virus in the NAc\nand ChRimsonR virus in the DR. Photostimulation was delivered in the NAc to activate the\nDR 5HT terminals in the NAc.\nWe have added the details in the figure legend.\nMinor:\n1.In referencing the resident-intruder aggression test in mice, the authors should cite Miczek\nand O’Donnell 1978 (https://doi.org/10.1007/BF00426957). Similarly, the authors should\nacknowledge existing literature demonstrating anti-aggressive effects of systemically\nadministered MDMA (e.g., Miczek and Haney 1994).\nWe thank the reviewer for the references. We have now cited these papers in the main text.\n2.LED wavelengths should be specified in nm rather than as “blue” or “red” in the miniscope\nmethods section.\nWe have added the wavelengths.\n3.Traditionally, control conditions are depicted as the left-most bar in histograms rather than\nthe right-most as in Fig. 1s and 1t – might be worth considering changing unless there’s a\nreason to portray the data as-is.\nWe have moved the control conditions to the left.\n4. For the blown-up trace showing alignment with attack onset and aggression offset (Fig 1a/b),\nthe authors should consider including a scale bar for time and labeling the scale bar for Z\nscores.\nWe have added scale bars under the blown-up trace.\n5. Main text, line 56, please add something along the lines of “and implanted optic fibers\nterminating in the NAc.”\nWe thank the reviewer for pointing this out. We have added the detail.\nReviewer #3\nIn their manuscript “Serotonin modulates nucleus accumbens circuits to suppress aggression”,\nZhang et al describe a novel mechanism for serotonin to modulate aggression, whereby\nserotonin input into the nucleus accumbens inhibits a subpopulation of D1 medium spiny\nneurons to suppress aggressive behavior after fighting has initiated. Experiments are cleverly\nstructured and the inclusion of MDMA, albeit mechanistically dirty is translationally interesting.\nWe are glad that the reviewer thinks our ‘experiments are cleverly structured’ and the MDMA\nexperiments are ‘translationally interesting’.\nSpecifically, the authors report that serotonin release (GRAB-5HT) in the NAc ramps up during\nan attack and ramps down with aggression offset. The ramp-up slope inversely correlates with\nattack duration.\n1 what about the peak, ramp-down slope, and AUC?\nWe have analyzed how the peak, ramp-down slope, and AUC of GRAB-5HT signal correlate\nwith aggression duration: both peak and AUC positively correlate with aggression duration,\nwith the AUC having a strong correlation. We think the ramp-down slope is dominated by the\nslow off-kinetics of the GRAB sensors in most trials, rather than reflecting biologically\ninteresting events and therefore it does not correlate with attack duration.\nWe have added a new Supplementary Figure 2 to show the peak and AUC correlation with\naggression duration.\nAugmenting serotonin release (MDMA or ChRimimsonR or rsChRmine) reduces attack\nduration, demonstrating that serotonin release in the NAc is sufficient to inhibit attack behavior.\nPhotostimulation of DA release in the NAc had no effect on aggressive behavior.\n2 please explain the use of ChRimsonR versus rsChRmine.\nWe used ChRimsonR for the miniscope experiments to minimize potential crosstalk between\nthe blue LED (for calcium imaging) and optogenetic stimulation: ChRimsonR is much less\nsensitive to blue light compared to rsChRmine, and it has been shown that blue LED light at\n2 mW (we used 0.5-1mW in our experiments) results in minimal activation of terminals\nexpressing the opsin (Stamatakis et al, Front Neurosci., 2018, DOI:\n10.3389/fnins.2018.00496).\nWe used both opsins in the optogenetic experiments where cross-talk was not a concern, and\nwe can take advantage of the higher sensitivity of rsChRmine (Kishi et al, Cell, 2022, DOI:\n10.1016/j.cell.2022.01.007).\nWe have expanded the Methods section to add this rationale.\n3 an open field control would be great here (see MSN inhibition and also Walsh et al). is the\neffect of serotonin specific to aggression behavior? Is NAc serotonin release aversive? Are\nsocial or goal-directed behaviors affected more generally? (see MSN inhibition. See also\nWalsh et al and Rein et al).\nWe thank the reviewer for these important questions. The effect of serotonin release in the\nNAc is likely to be context-dependent. Previous papers from our lab have shown that NAc\nserotonin release is pro-social in juvenile interaction and three-chamber sociability tests\n(Walsh et al, 2018, DOI: 10.1038/s41586-018-0416-4, Heifets et al, 2019, DOI:\n10.1126/scitranslmed.aaw6435), and plays an opponent role against DA in reinforcement\nlearning (Pinto et al, 2025, DOI: 10.1038/s41586-024-08412-x). In contrast, 5HT release in\nthe NAc has no effect on locomotor activity in the open-field test or the real-time place\npreference test (Walsh et al, 2018, DOI: 10.1038/s41586-018-0416-4, Rein et al, 2024, DOI:\n10.1126/sciadv.adl6554, Pinto et al, 2025, DOI: 10.1038/s41586-024-08412-x). Before this\npaper, the role of NAc 5HT release in aggression has never been tested.\nIn addition to expanding our discussion of the literature cited above, we have also conducted\nnew experiments and analyses to address this comment. In our resident-intruder tests, we\nfound no significant change in running speed or non-aggressive social behaviors with 5HT\nterminal stimulation in the NAc (new Supplementary Figure 4a-e).\nWe have also performed RTPP tests, and, consistent with previous findings, we found that the\ntest mice showed no preference for the side associated with 5HT terminal stimulation (new\nSupplementary Figure 4f). We have added these data and associated discussion to the\nmanuscript.\n4 photostimulation for behavior was initiated 3 minutes after intruder entry and lasted for 3\nminutes. This temporal dynamic of experimentally induced serotonin release is very different\nfrom the identified tight temporal dynamic of endogenous serotonin release during aggression\nbehavior. Why did the authors choose such different parameters? A rationale should be\nprovided in the main text.\nThe duration of optogenetic stimulation was chosen to balance two factors: it had to be long\nenough to capture enough aggressive bouts to test the behavioral effect, but short enough to\navoid opsin desensitization. In our prior paper (Walsh et al., Nature, 2018, DOI:\n10.1038/s41586-018-0416-4), we used similar durations (5-min for three-chamber sociability\nand 2-min for juvenile interaction) and found pro-social effects even though endogenous\nserotonin release during social interactions had different temporal dynamics. We therefore\nhypothesized that longer-term elevations of 5HT levels in the NAc would similarly influence\nattacking behavior.\nThe timescales by which serotonin acts on downstream circuits to change behavior remains\na fascinating and important problem. We are currently working on closed-loop stimulation\nparadigms, where optogenetic stimulation of serotonin release in multiple regions is paired in\nreal time with aggression behavior, and we hope to publish the results of these experiments\nin future studies. We have added these important points to the discussion.\nserotonin inhibition experiment would be great. (see Walsh et al)\nWe appreciate the reviewer’s suggestion. We have attempted a set of experiments to inhibit\nserotonin terminals in the NAc during the resident-intruder paradigm. Since the first author\ngave birth to a baby boy shortly before the reviews returned, this required adding another co-\nauthor (a graduate student), who bred the animals and performed the experiments. To stay\nconsistent with our original experiments, we used SERT-Flp mice, but since no simple AAV-\nfDIO-NpHR3.3 exists, we were forced to inject AAV8-nEF-Coff/Fon-NpHR3.3-EYFP in the DR.\nWe then performed the experiment, and found no behavioral effect of light whatsoever. But\nupon histology, despite the prediction that this viral construct should work fine in any Flp driver\nmouse line, we were unable to detect adequate expression of NpHR in serotonin neurons. We\ndo not know exactly why this virus did not express in our mice, but perhaps the Cre-related\nsequences reduced the efficacy. Considering that this is the only commercially available Flp-\ndependent NpHR, we think it is outside the scope of this current study to: 1) further validate\nand determine proper parameters for using this virus or 2) generate our own Flp-dependent\nNpHR without Cre-related sequences. Therefore, we have tempered our conclusions to clarify\nthat although additional serotonin release in the NAc is sufficient to suppression aggression,\nwe cannot conclude that endogenous release is necessary to limit aggression.\nUsing a GRIN lens GCaMP8m approach, the authors find that the activity of D1 MSNs (but\nnot D2 MSNs) in the NAc positively correlates with aggression episode duration. Inhibiting D1\nMSNs in the NAc (NpHR3.0) reduces attack duration, while inhibiting D2 MSNs in the NAc\n(NpHR3.0) has no effect on attack duration, indicating a necessary role for NAc D1 MSN\nactivity in attack duration.\nWhat is “D duration per attack (%)”? is this normalized twice?\nThe data are plotted differently in 1j,k vs 1s,t (now 2h,i) because in the former panels, we are\nusing within-animal comparisons (LightOn vs LightOff), whereas in the latter panels we are\ncomparing different cohorts of mice (mCherry controls vs D1 inhibition vs D2 inhibition). The\nsame normalization was applied throughout, and nothing was normalized twice. In all cases,\nthe data was normalized to the LightOff condition. For example, in the case of mCherry\ncontrols, we are plotting the difference between LightOn vs LightOff in these animals. Ditto for\nthe D1 and D2 inhibition animals. In all cases, ‘Duration per attack (%)’ is the average duration\nof each attack (in seconds) divided by the average duration of each attack in the LightOff\nperiod. We have added these details in Methods and tried to clarify the description in the text\nand caption.\n7 the inhibition protocol lasts the entire 9 minutes of the encounter. Is the effect of inhibition\ntowards the beginning different than towards the end (effect of time? Correlation with time?)\nThis is an interesting idea that we did not initially consider. We have analyzed the effect of\ninhibition in three time bins (0-3 min, 3-6 min, and 6-9 min). We did not find any relationship\nbetween the time bin and the effect of inhibition. In the future we plan to inhibit only in the first\nfew minutes of the encounter to test if D1 inhibition has a prolonged effect on aggression\nbehavior, even after the stimulation is turned off.\nentire period has specific effects on aggression or if it interferes with other behaviors in a more\ngeneral way. The open field test is a great control, but are social or goal-directed behaviors\naffected? The conclusions drawn from this set of experiments are too specific given the limited\ndata provided and the broader known role for NAc D1 MSNs.\nTo address the reviewer’s question, we have performed three-chamber sociability tests and\nRTPP tests to examine if D1 or D2 MSN inhibition in the medial NAc shell affects social or\ngoal-directed behavior, respectively (new Supplementary Figure 6). We saw no significant\neffect in either assay. This does not exclude the role of D1 or D2 MSNs in other behaviors,\nbut implies some level of specialization. Furthermore, negative results when performing\noptogenetic inhibition manipulations are difficult to interpret because it is impossible to know\nthat the degree of inhibition was sufficient to generate measurable behavioral results.\nAlthough NAc D1 MSNs, especially in the core region, have been hypothesized to signal\nreward, previous studies have shown that inhibiting D1 MSNs in the medial shell region does\nnot always modulate food-seeking behavior (Golden et al, 2019. DOI:\n10.1523/JNEUROSCI.2409-18.2019). Instead, chemogenetic inhibition of medial shell D1\nMSNs reduced aggression-seeking (Golden et al, 2019), which is consistent with our results\nhere. In the present study, we targeted the medial shell of the NAc because it has the densest\nserotonin terminals (Pinto et al, 2025, DOI: 10.1101/2025.04.17.649431). Future studies may\nexamine different sub-regions of the NAc, or target genetically distinct sub-groups of D1 MSNs,\nto dissect the involvement of these neural populations in aggression and in other contexts.\nWe have expanded our discussion to describe the specific NAc area and how our results fit\ninto the literature. We have also tempered our conclusions about the specificity of these\nneurons for aggression; they certainly play other roles as well.\nThe authors finish their manuscript with a beautiful in vivo assessment of the effect serotonin\nhas on MSNs. They find that MDMA or optogenetic stimulation of serotonin release in the NAc\ninhibitsD1 (but not D2) MSNs. Moreover they found that D1 MSNs that were more active\nduring attack were also more sensitive to the inhibitory effect of serotonin release.\neffective photostimulation of serotonin release in the NAc is in inhibiting D1MSNs during\naggression/attack behavior.\nWe have added a new Supplementary Figure 8 to address this point. For the miniscope\nexperiments, we recorded three sessions on the same day, all with the same field-of-view:\none with an intruder mouse (but no optogenetic stimulation), one with an intruder mouse and\noptogenetic stimulation, and one with optogenetic stimulation while the mouse was alone.\nIn the original submission, we only presented data from the intruder (no stimulation) and\nstimulation (no intruder) sessions, showing that the same neurons that respond to optogenetic\nstimulation also respond to aggression. Now we analyze the third session, showing that\nserotonin stimulation did in fact reduce D1 MSN activity during aggression (Supplementary\nFig. 8e-f).\nTaken together this is a really nice manuscript with elegant techniques and novel findings, but\nI feel that addressing a couple of open questions will allow for much tighter conclusions.\nWe thank the reviewer for the positive comments. We believe the revised version fully\naddresses the questions and tightens the conclusions.\nResponse to the second round of reviews, NCOMMS-25-26724A\nWe are grateful to the reviewers for their continued support and detailed reading of our\nmanuscript. Reviewer 1 says that the revision has “addressed most of our concerns”;\nReviewer 2 says that the revised version is “improved”; and Reviewer 3 says that the paper now\nprovides “rigorous data analyses, valid conclusions, and impactful novel insight”, with “no\nremaining concerns or suggestions”. In response to the suggestions from Reviewer 1 and 2, we\nhave made further changes to the text, especially with regards to the Methods section. We\nbelieve the paper is substantially improved as a result and will now be ready for publication in\nNature Communications. We hope that the Reviewers and Editor agree.\nReviewer #1:\nThe authors has addressed most of our concerns.\nWe thank the Reviewer for their continued support.\nOne point that we believe requires further clarification concerns the serotonin effect on D1\nMSNs. According to a recent study (Pinto et al., bioRxiv, 2025; doi:10.1101/2025.04.17.649431),\ndirect application of serotonin to D1 MSNs did not alter their neuronal activity. Given that the\nserotonin system can co-release multiple neurotransmitters in addition to serotonin itself,\nwhy the authors believe that the inhibitory effect observed in Figure 3i is specifically mediated\nby serotonin?\nWe thank the reviewer for highlighting a related study from our labs. In that work, serotonin was\ndirectly applied to MSNs in brain slices, a preparation that differs substantially from the in vivo\nconditions examined here. In the in vitro setting, we did not observe a major effect of serotonin\non D1 MSN activity. However, in that study we also performed c-fos experiments, which we\nhave replicated and extended since the preprint. In these c-fos experiments, we found that\ninfusing escitalopram (a serotonin reuptake inhibitor) directly into the NAc caused a dramatic\nreduction in the activity of D1 MSNs, compared to saline infusion. The c-fos results in that\nmanuscript are consistent with the miniscope results we report in this manuscript, and\nemphasize that the effects of neuromodulators differ between in vivo and in vitro settings. Our\nother manuscript is also in the midst of revisions and the preprint will be updated soon with\nthese new findings.\nNevertheless, the reviewer raises an important point. Dorsal raphe serotonin neurons can co-\nexpress gene markers associated with glutamatergic or GABAergic transmission, or both (Ren\net al., eLife, 2019, DOI: 10.7554/eLife.49424). While glutamate appears to be the most\nprevalent co-transmitter, there is some evidence that DR 5-HT neurons can release GABA in\nhighly specialized conditions (Li et al, Science, 2024, DOI: 10.1126/science.adj5996). It is\nunclear whether those co-transmitting neurons project to the NAc to contribute to the inhibitory\ndownstream effect we observed here. But we cannot fully rule out the involvement of GABA in\nour experiment. In addition, NAc-projecting serotonin neurons likely co-express several\nneuropeptides, which may also contribute to the inhibitory effects we observe here. These\nconcerns are partially mitigated by the fact that infusing serotonin agonists into the NAc mimics\nour behavioral result (Supplementary Fig. 5) and that MDMA administration mimics our D1\ninhibition result (Fig. 3b). Future work should more directly test whether neurotransmitter or\nneuropeptide co-release contributes to the effects observed in our study. We appreciate the\nReviewer’s question and we have now expanded our discussion to address these points.\nReviewer #2:\nThe present revised version of the manuscript from Zhang et al. is improved but several\nconcerning details are lacking in the Methods section, making it challenging to understand\nprecisely how the authors conducted their behavioral assessments and how behavioral data\nwere quantified using machine learning approaches. These major issues are outlined below:\n1. Key information is missing from the “Mice” subsection of the Methods, e.g., sex of\naggressors, housing conditions of aggressors vs. intruders (single-housing vs. group-housing\nand if group-housed, how many to a cage?), what time of day animals were tested (i.e., in the\nlight phase or dark phase), etc.\nWe thank the reviewer for pointing out these details, which we have now added to the Methods,\n‘Mice’ subsection.\n2. There is no dedicated subsection of the Methods describing the aggression protocol – how\nwere intruders assigned to residents (different intruder for every resident every test day?), how\nlong were resident-intruder tests?, were attack bites capped after a certain number or was the\nconfrontation capped depending on the number of bites?, were intruders ever aggressive\ntoward the resident and if so, what steps were taken to prevent defeating the resident?\nWe thank the reviewer for the helpful comment. We have reorganized the subsections related to\nresident-intruder tests, and re-wrote a subsection titled ‘Resident-intruder tests’ in Methods with\nthese details.\n3. The methodological details of the machine learning (ML) pipeline are cryptic – also, as\ndescribed, the video capture parameters raise concerns regarding the validity of downstream\nML approaches. Typically, vigorous behaviors like aggression need to be captured at a frame\nrate of ≥30Hz to train a DLC model for multi-animal pose estimation – it is concerning that the\nauthors captured video at 20Hz under dim illumination to train their DLC model. These video\ncapture parameters would produce blurry training frames especially during vigorous and highly\nphysical attacks, making it impossible to accurately place body points on DLC training frames.\nIn their supplement, the authors need to provide a test video (i.e., a video that was NOT used to\ntrain any pose estimation or classification models) that was generated using their DLC-SimBA\npipeline– providing such a video will be minimal effort since SimBA automatically generates\nvideos that integrate DLC pose estimations and SimBA classification. In addition, it is also\nnecessary for the authors to provide standard benchmarking details of their DLC pose\nestimation model and for each SimBA classification model – the authors should refer to Table 1\nin the maDLC paper (Lauer et al. 2022 Nature Methods) and to the table in Figure 2b in the\nSimBA paper (Goodwin et al. 2024) for guidance on expected benchmarking information that\nthey need to include. The authors also need to operationally define ML-ID’d behaviors, so\nreaders know how each SimBA classification model was trained (see Goodwin et al. 2022\nsupplement, Figure S1). It is also unclear at present how many SimBA classifiers were trained –\nattack, approach, sniff? – and why the ML pipeline was only applied to videos that did not\ninclude neural recordings since it is well-established that these models can be trained to track\nand classify behaviors accurately even in tethered animals.\nWe thank the reviewer for pointing out these gaps in the methods description, and we have now\naddressed all the points. First, we have added a test video generated using our DLC-SimBA\npipeline in the Supplement. Second, we have added the required modeling details in a new\nSupplementary Note, as discussed below.\nWe have been using 20Hz routinely for recording mouse social behavior in our labs and DLC\nmodels have achieved satisfactory performance across multiple behavioral setups. Specifically\nin our experiments here, the test mice and the intruders are of distinct colors (dark gray vs\nwhite, Supplementary Video 1), so we used the classic DLC method (Mathis et al, Nature\nNeuroscience, 2018) without multi-animal tracking functionality. The training dataset details are\nsummarized in Table 1. 8 body parts (‘keypoints’) were used for each animal: Ear_left,\nEar_right, Nose, Center, Lateral_left, Leteral_right, Tail_base, Tail end. The model achieved\ntrain error of 2.13 pixels and test error of 2.18 pixels after 675000 iterations. We found that the\n‘Tail end’ point was often off in test videos, which presumably contributed most to the errors, and\nso these keypoints were not used for further behavioral identifications.\nFeature Value\nKeypoints (total) 16\nIndividuals 2\nLabeled frames 522\nLabeled videos 20\nTotal duration (s) >10800\nTable 1: Training dataset characteristics for generating a DeepLabCut model\nWe used three previously trained SIMBA classifiers (published in Goodwin et al. 2024) for\ndetecting attacks, approaches and sniffs (face to body). Approaching and face-body sniffing\nbehaviors were used as less-aggressive controls. The information regarding the training sets is\nsummarized in Table 2. The behavioral definitions are summarized in Table 3.\nNo. No. of\nNo. of Optimal\nClassifier annotated positive F1 Precision Recall\nvideos threshold\nframes frames\nAttack 126604 25214 27 0.7665 0.868 0.798 0.951\nApproach 126604 5881 27 0.7689 0.836 0.747 0.948\nFaceBody sniff 126604 26958 27 0.8269 0.839 0.771 0.912\nTable 2: Information for the training sets for the SIMBA classifiers\nClassifier Description Start frame Duration of End frame\nbehavior\nAttack Clear physical Resident mouse Short breaks may Resident mouse\nantagonistic makes physical be present in attack orients away from\ninteraction initiated antagonistic behavior, if resident Intruder. Typically this\nby the Resident contact with the is still oriented is a slight turning of\nmouse. Brief pauses Intruder. Typically toward Intruder. the head to look in a\ncan occur if the this is Attacks can include different direction,\nResident mouse characterized by tussling, biting, followed by a\nremains oriented outstretched boxing, and relaxation of the body\ntoward the Intruder. Resident corralling as part of and moving away from\nforepaw(s) the attack bout. the Intruder.\ncontacting the\nIntruder, while the\nResident has an\nopen mouth to\ninitiate a bite. Can\nalso be\ncharacterized by\nthe first frame of a\nslap or quick bite\nwithout the\nforepaws being\noutstretched.\nApproach Resident mouse Resident mouse is Resident is oriented Resident mouse stops\norients self to point moving directly at and moving moving toward anon-\nat Intruder and towards a non- directly toward escaping intruder\nmoves directly escaping intruder. Intruder. Intruder is mouse or when\ntoward Intruder. not moving directly intruder begins moving\nIntruder is not away directly away from\nactively moving from resident and resident.\naway from Resident may be engaging in\nand may be either non-escape\nstationary or behaviors. The\nengaging another Intruder may still\nnon-escape traverse the\nbehavior, including operant chamber\ngrooming and so long as it does\noperant chamber not begin\nexploration. specifically moving\naway from an\napproaching\nresident.\nFaceBody Resident mouse is Resident mouse is Uninterrupted Resident mouse\nsniff sniffing the Intruder, clearly sniffing the sniffing of facial or moves head away\nbut not in the intruder in a facial bodily region from Intruder, sniffs\nanogenital region or bodily region the anogenital/tail\n(base of tail). region, or engages in a\nseparate behavior.\nTable 3: Operational behavioral definitions for SIMBA classifiers\nTo re-use the classifiers for new videos captured by a different camera lens, we scaled our DLC\noutputs by changing the SIMBA input parameter, ‘pixels/mm’, from 1.51 (used for the training\nset) to 0.9. We used threshold = 0.8 for all three behaviors. To avoid continuous behavioral\nbouts being incorrectly separated into multiple bouts, or false positive detections in single\nframes, the detected bouts that were < 0.25 s apart were merged into one, and those that were\nshorter than 0.5 s were filtered out.\nFinally, to validate the performance of the full DLC-SIMBA pipeline, we annotated five videos\nrecorded under our experimental conditions, which were not used for training either the DLC or\nthe SIMBA models. Using the manual labels as ‘ground truth’, the performance of the full\npipeline is summarized in Table 4.\nClassifier F1 Precision Recall\nAttack 0.712 0.617 0.842\nApproach 0.647 0.719 0.517\nFaceBody sniff 0.602 0.640 0.654\nTable 4: Performance of classifiers on test videos\nPlotting hand annotation against DLC-SIMBA detection results (Figure R1), we found that\nalthough there are offsets between the values, as reflected in the shifts or tilts of the linear fits\n(red) from the diagonal lines (grey), the hand annotation and SIMBA detection results, in\nparticular for attack and approach, are significantly and linearly correlated, with R’s>0.8. The\noffsets may in part reflect annotator’s bias (as she was not involved in annotating the training\ndata).\nFigure R1: Hand annotation versus DLC-SIMBA detections for each behavior. (****P<0.0001, *P\n= 0.036, n = 5 videos, 5 resident-intruder pairs; each 9-min video was cut into 3 x 3-min sections\nand each dot corresponds to the duration of a specific behavior in a section)\nWe used the ML pipeline to analyze videos in optogenetic/drug manipulation experiments where\nthe relevant outcome was behavioral changes between test and control conditions. We\npreferred the ML pipeline in these analyses to avoid human bias in comparing behavioral results\nacross conditions. In contrast, for the neural recording experiments, we wanted to maximize\ntiming accuracy to align the neural activity to behavioral timestamps on a frame-by-frame basis.\nWe avoided bias by blinding the experimenter from the neural data while they labeled frames.\nWe felt that manual labeling, although time-consuming, would maximize the accuracy of\ntimestamps for neural data analyses.\nReviewer #3:\nThe authors have done a great job addressing comments and have significantly improved their\nmanuscript. With plenty new data allowing for tighter conclusions I have no remaining concerns\nor suggestions. I appreciate the personal tone regarding the failed inhibition experiment (we can\nall relate), and I think the paper provides rigorous data analyses, valid conclusions, and\nimpactful novel insight even without it.\nWe are glad that our revision has left the reviewer with ‘no remaining concerns or suggestions’.\nWe are grateful that the reviewer thinks our paper ‘provides rigorous data analyses, valid\nconclusions, and impactful novel insight’."}, {"round": 2, "reviewer_comments": {"reviewer_1": "The authors has addressed most of our concerns.\nOne point that we believe requires further clarification concerns the serotonin effect on D1 MSNs. According to a recent study (Pinto et al., bioRxiv, 2025; doi:10.1101/2025.04.17.649431), direct application of serotonin to D1 MSNs did not alter their neuronal activity. Given that the serotonin system can co-release multiple neurotransmitters in addition to serotonin itself, why the authors believe that the inhibitory effect observed in Figure 3i is specifically mediated by serotonin?", "reviewer_2": "The present revised version of the manuscript from Zhang et al. is improved but several concerning details are lacking in the Methods section, making it challenging to understand precisely how the authors conducted their behavioral assessments and how behavioral data were quantified using machine learning approaches. These major issues are outlined below:\n1. Key information is missing from the “Mice” subsection of the Methods, e.g., sex of aggressors, housing conditions of aggressors vs. intruders (single-housing vs. group-housing and if group-housed, how many to a cage?), what time of day animals were tested (i.e., in the light phase or dark phase), etc.\n2. There is no dedicated subsection of the Methods describing the aggression protocol – how were intruders assigned to residents (different intruder for every resident every test day?), how long were resident-intruder tests?, were attack bites capped after a certain number or was the confrontation capped depending on the number of bites?, were intruders ever aggressive toward the resident and if so, what steps were taken to prevent defeating the resident?\n3. The methodological details of the machine learning (ML) pipeline are cryptic – also, as described, the video capture parameters raise concerns regarding the validity of downstream ML approaches. Typically, vigorous behaviors like aggression need to be captured at a frame rate of ≥30Hz to train a DLC model for multi-animal pose estimation – it is concerning that the authors captured video at 20Hz under dim illumination to train their DLC model. These video capture parameters would produce blurry training frames especially during vigorous and highly physical attacks, making it impossible to accurately place body points on DLC training frames. In their supplement, the authors need to provide a test video (i.e., a video that was NOT used to train any pose estimation or classification models) that was generated using their DLC-SimBA pipeline– providing such a video will be minimal effort since SimBA automatically generates videos that integrate DLC pose estimations and SimBA classification. In addition, it is also necessary for the authors to provide standard benchmarking details of their DLC pose estimation model and for each SimBA classification model – the authors should refer to Table 1 in the maDLC paper (Lauer et al. 2022 Nature Methods) and to the table in Figure 2b in the SimBA paper (Goodwin et al. 2024) for guidance on expected benchmarking information that they need to include. The authors also need to operationally define ML-ID’d behaviors, so readers know how each SimBA classification model was trained (see Goodwin et al. 2022 supplement, Figure S1). It is also unclear at present how many SimBA classifiers were trained – attack, approach, sniff? – and why the ML pipeline was only applied to videos that did not include neural recordings since it is well-established that these models can be trained to track and classify behaviors accurately even in tethered animals.", "reviewer_3": "The authors have done a great job addressing comments and have significantly improved their manuscript. With plenty new data allowing for tighter conclusions I have no remaining concerns or suggestions. I appreciate the personal tone regarding the failed inhibition experiment (we can all relate), and I think the paper provides rigorous data analyses, valid conclusions, and impactful novel insight even without it."}, "author_response": "Response to the second round of reviews, NCOMMS-25-26724A\nWe are grateful to the reviewers for their continued support and detailed reading of our\nmanuscript. Reviewer 1 says that the revision has “addressed most of our concerns”; Reviewer\n2 says that the revised version is “improved”; and Reviewer 3 says that the paper now\nprovides “rigorous data analyses, valid conclusions, and impactful novel insight”, with “no\nremaining concerns or suggestions”. In response to the suggestions from Reviewer 1 and 2, we\nhave made further changes to the text, especially with regards to the Methods section. We\nbelieve the paper is substantially improved as a result and will now be ready for publication in\nNature Communications. We hope that the Reviewers and Editor agree.\nReviewer #1:\nThe authors has addressed most of our concerns.\nWe thank the Reviewer for their continued support.\nOne point that we believe requires further clarification concerns the serotonin effect on D1\nMSNs. According to a recent study (Pinto et al., bioRxiv, 2025; doi:10.1101/2025.04.17.649431),\ndirect application of serotonin to D1 MSNs did not alter their neuronal activity. Given that the\nserotonin system can co-release multiple neurotransmitters in addition to serotonin itself,\nwhy the authors believe that the inhibitory effect observed in Figure 3i is specifically mediated\nby serotonin?\nWe thank the reviewer for highlighting a related study from our labs. In that work, serotonin was\ndirectly applied to MSNs in brain slices, a preparation that differs substantially from the in vivo\nconditions examined here. In the in vitro setting, we did not observe a major effect of serotonin\non D1 MSN activity. However, in that study we also performed c-fos experiments, which we\nhave replicated and extended since the preprint. In these c-fos experiments, we found that\ninfusing escitalopram (a serotonin reuptake inhibitor) directly into the NAc caused a dramatic\nreduction in the activity of D1 MSNs, compared to saline infusion. The c-fos results in that\nmanuscript are consistent with the miniscope results we report in this manuscript, and\nemphasize that the effects of neuromodulators differ between in vivo and in vitro settings. Our\nother manuscript is also in the midst of revisions and the preprint will be updated soon with\nthese new findings.\nNevertheless, the reviewer raises an important point. Dorsal raphe serotonin neurons can co-\nexpress gene markers associated with glutamatergic or GABAergic transmission, or both (Ren\net al., eLife, 2019, DOI: 10.7554/eLife.49424). While glutamate appears to be the most\nprevalent co-transmitter, there is some evidence that DR 5-HT neurons can release GABA in\nhighly specialized conditions (Li et al, Science, 2024, DOI: 10.1126/science.adj5996). It is\nunclear whether those co-transmitting neurons project to the NAc to contribute to the inhibitory\ndownstream effect we observed here. But we cannot fully rule out the involvement of GABA in\nour experiment. In addition, NAc-projecting serotonin neurons likely co-express several\nneuropeptides, which may also contribute to the inhibitory effects we observe here. These\nconcerns are partially mitigated by the fact that infusing serotonin agonists into the NAc mimics\nour behavioral result (Supplementary Fig. 5) and that MDMA administration mimics our D1\ninhibition result (Fig. 3b). Future work should more directly test whether neurotransmitter or\nneuropeptide co-release contributes to the effects observed in our study. We appreciate the\nReviewer’s question and we have now expanded our discussion to address these points.\nReviewer #2:\nThe present revised version of the manuscript from Zhang et al. is improved but several\nconcerning details are lacking in the Methods section, making it challenging to understand\nprecisely how the authors conducted their behavioral assessments and how behavioral data\nwere quantified using machine learning approaches. These major issues are outlined below:\n1. Key information is missing from the “Mice” subsection of the Methods, e.g., sex of\naggressors, housing conditions of aggressors vs. intruders (single-housing vs. group-housing\nand if group-housed, how many to a cage?), what time of day animals were tested (i.e., in the\nlight phase or dark phase), etc.\nWe thank the reviewer for pointing out these details, which we have now added to the Methods,\n‘Mice’ subsection.\n2. There is no dedicated subsection of the Methods describing the aggression protocol – how\nwere intruders assigned to residents (different intruder for every resident every test day?), how\nlong were resident-intruder tests?, were attack bites capped after a certain number or was the\nconfrontation capped depending on the number of bites?, were intruders ever aggressive\ntoward the resident and if so, what steps were taken to prevent defeating the resident?\nWe thank the reviewer for the helpful comment. We have reorganized the subsections related to\nresident-intruder tests, and re-wrote a subsection titled ‘Resident-intruder tests’ in Methods with\nthese details.\n3. The methodological details of the machine learning (ML) pipeline are cryptic – also, as\ndescribed, the video capture parameters raise concerns regarding the validity of downstream\nML approaches. Typically, vigorous behaviors like aggression need to be captured at a frame\nrate of ≥30Hz to train a DLC model for multi-animal pose estimation – it is concerning that the\nauthors captured video at 20Hz under dim illumination to train their DLC model. These video\ncapture parameters would produce blurry training frames especially during vigorous and highly\nphysical attacks, making it impossible to accurately place body points on DLC training frames.\nIn their supplement, the authors need to provide a test video (i.e., a video that was NOT used to\ntrain any pose estimation or classification models) that was generated using their DLC-SimBA\npipeline– providing such a video will be minimal effort since SimBA automatically generates\nvideos that integrate DLC pose estimations and SimBA classification. In addition, it is also\nnecessary for the authors to provide standard benchmarking details of their DLC pose\nestimation model and for each SimBA classification model – the authors should refer to Table 1\nin the maDLC paper (Lauer et al. 2022 Nature Methods) and to the table in Figure 2b in the\nSimBA paper (Goodwin et al. 2024) for guidance on expected benchmarking information that\nthey need to include. The authors also need to operationally define ML-ID’d behaviors, so\nreaders know how each SimBA classification model was trained (see Goodwin et al. 2022\nsupplement, Figure S1). It is also unclear at present how many SimBA classifiers were trained –\nattack, approach, sniff? – and why the ML pipeline was only applied to videos that did not\ninclude neural recordings since it is well-established that these models can be trained to track\nand classify behaviors accurately even in tethered animals.\nWe thank the reviewer for pointing out these gaps in the methods description, and we have now\naddressed all the points. First, we have added a test video generated using our DLC-SimBA\npipeline in the Supplement. Second, we have added the required modeling details in a new\nSupplementary Note, as discussed below.\nWe have been using 20Hz routinely for recording mouse social behavior in our labs and DLC\nmodels have achieved satisfactory performance across multiple behavioral setups. Specifically\nin our experiments here, the test mice and the intruders are of distinct colors (dark gray vs\nwhite, Supplementary Video 1), so we used the classic DLC method (Mathis et al, Nature\nNeuroscience, 2018) without multi-animal tracking functionality. The training dataset details are\nsummarized in Table 1. 8 body parts (‘keypoints’) were used for each animal: Ear_left,\nEar_right, Nose, Center, Lateral_left, Leteral_right, Tail_base, Tail end. The model achieved\ntrain error of 2.13 pixels and test error of 2.18 pixels after 675000 iterations. We found that the\n‘Tail end’ point was often off in test videos, which presumably contributed most to the errors, and\nso these keypoints were not used for further behavioral identifications.\nFeature Value\nKeypoints (total) 16\nIndividuals 2\nLabeled frames 522\nLabeled videos 20\nTotal duration (s) >10800\nTable 1: Training dataset characteristics for generating a DeepLabCut model\nWe used three previously trained SIMBA classifiers (published in Goodwin et al. 2024) for\ndetecting attacks, approaches and sniffs (face to body). Approaching and face-body sniffing\nbehaviors were used as less-aggressive controls. The information regarding the training sets is\nsummarized in Table 2. The behavioral definitions are summarized in Table 3.\nNo. No. of\nNo. of Optimal\nClassifier annotated positive F1 Precision Recall\nvideos threshold\nframes frames\nAttack 126604 25214 27 0.7665 0.868 0.798 0.951\nApproach 126604 5881 27 0.7689 0.836 0.747 0.948\nFaceBody sniff 126604 26958 27 0.8269 0.839 0.771 0.912\nTable 2: Information for the training sets for the SIMBA classifiers\nClassifier Description Start frame Duration of End frame\nbehavior\nAttack Clear physical Resident mouse Short breaks may Resident mouse\nantagonistic makes physical be present in attack orients away from\ninteraction initiated antagonistic behavior, if resident Intruder. Typically this\nby the Resident contact with the is still oriented is a slight turning of\nmouse. Brief pauses Intruder. Typically toward Intruder. the head to look in a\ncan occur if the this is Attacks can include different direction,\nResident mouse characterized by tussling, biting, followed by a\nremains oriented outstretched boxing, and relaxation of the body\ntoward the Intruder. Resident corralling as part of and moving away from\nforepaw(s) the attack bout. the Intruder.\ncontacting the\nIntruder, while the\nResident has an\nopen mouth to\ninitiate a bite. Can\nalso be\ncharacterized by\nthe first frame of a\nslap or quick bite\nwithout the\nforepaws being\noutstretched.\nApproach Resident mouse Resident mouse is Resident is oriented Resident mouse stops\norients self to point moving directly at and moving moving toward anon-\nat Intruder and towards a non- directly toward escaping intruder\nmoves directly escaping intruder. Intruder. Intruder is mouse or when\ntoward Intruder. not moving directly intruder begins moving\nIntruder is not away directly away from\nactively moving from resident and resident.\naway from Resident may be engaging in\nand may be either non-escape\nstationary or behaviors. The\nengaging another Intruder may still\nnon-escape traverse the\nbehavior, including operant chamber\ngrooming and so long as it does\noperant chamber not begin\nexploration. specifically moving\naway from an\napproaching\nresident.\nFaceBody Resident mouse is Resident mouse is Uninterrupted Resident mouse\nsniff sniffing the Intruder, clearly sniffing the sniffing of facial or moves head away\nbut not in the intruder in a facial bodily region from Intruder, sniffs\nanogenital region or bodily region the anogenital/tail\n(base of tail). region, or engages in a\nseparate behavior.\nTable 3: Operational behavioral definitions for SIMBA classifiers\nTo re-use the classifiers for new videos captured by a different camera lens, we scaled our DLC\noutputs by changing the SIMBA input parameter, ‘pixels/mm’, from 1.51 (used for the training\nset) to 0.9. We used threshold = 0.8 for all three behaviors. To avoid continuous behavioral\nbouts being incorrectly separated into multiple bouts, or false positive detections in single\nframes, the detected bouts that were < 0.25 s apart were merged into one, and those that were\nshorter than 0.5 s were filtered out.\nFinally, to validate the performance of the full DLC-SIMBA pipeline, we annotated five videos\nrecorded under our experimental conditions, which were not used for training either the DLC or\nthe SIMBA models. Using the manual labels as ‘ground truth’, the performance of the full\npipeline is summarized in Table 4.\nClassifier F1 Precision Recall\nAttack 0.712 0.617 0.842\nApproach 0.647 0.719 0.517\nFaceBody sniff 0.602 0.640 0.654\nTable 4: Performance of classifiers on test videos\nPlotting hand annotation against DLC-SIMBA detection results (Figure R1), we found that\nalthough there are offsets between the values, as reflected in the shifts or tilts of the linear fits\n(red) from the diagonal lines (grey), the hand annotation and SIMBA detection results, in\nparticular for attack and approach, are significantly and linearly correlated, with R’s>0.8. The\noffsets may in part reflect annotator’s bias (as she was not involved in annotating the training\ndata).\nFigure R1: Hand annotation versus DLC-SIMBA detections for each behavior. (****P<0.0001, *P\n= 0.036, n = 5 videos, 5 resident-intruder pairs; each 9-min video was cut into 3 x 3-min sections\nand each dot corresponds to the duration of a specific behavior in a section)\nWe used the ML pipeline to analyze videos in optogenetic/drug manipulation experiments where\nthe relevant outcome was behavioral changes between test and control conditions. We\npreferred the ML pipeline in these analyses to avoid human bias in comparing behavioral results\nacross conditions. In contrast, for the neural recording experiments, we wanted to maximize\ntiming accuracy to align the neural activity to behavioral timestamps on a frame-by-frame basis.\nWe avoided bias by blinding the experimenter from the neural data while they labeled frames.\nWe felt that manual labeling, although time-consuming, would maximize the accuracy of\ntimestamps for neural data analyses.\nReviewer #3:\nThe authors have done a great job addressing comments and have significantly improved their\nmanuscript. With plenty new data allowing for tighter conclusions I have no remaining concerns\nor suggestions. I appreciate the personal tone regarding the failed inhibition experiment (we can\nall relate), and I think the paper provides rigorous data analyses, valid conclusions, and\nimpactful novel insight even without it.\nWe are glad that our revision has left the reviewer with ‘no remaining concerns or suggestions’.\nWe are grateful that the reviewer thinks our paper ‘provides rigorous data analyses, valid\nconclusions, and impactful novel insight’."}]

Decision letter