Editor's evaluation: Nucleus accumbens dopamine tracks aversive stimulus duration and prediction but not value or prediction error

Field: Neuroscience (Life Sciences)

Article Figures and data Abstract Editor's evaluation Introduction Results Discussion Materials and methods Data availability References Decision letter Author response Article and author information Metrics Abstract There is active debate on the role of dopamine in processing aversive stimuli, where inferred roles range from no involvement at all, to signaling an aversive prediction error (APE). Here, we systematically investigate dopamine release in the nucleus accumbens core (NAC), which is closely linked to reward prediction errors, in rats exposed to white noise (WN, a versatile, underutilized, aversive stimulus) and its predictive cues. Both induced a negative dopamine ramp, followed by slow signal recovery upon stimulus cessation. In contrast to reward conditioning, this dopamine signal was unaffected by WN value, context valence, or probabilistic contingencies, and the WN dopamine response shifted only partially toward its predictive cue. However, unpredicted WN provoked slower post-stimulus signal recovery than predicted WN. Despite differing signal qualities, dopamine responses to simultaneous presentation of rewarding and aversive stimuli were additive. Together, our findings demonstrate that instead of an APE, NAC dopamine primarily tracks prediction and duration of aversive events. Editor's evaluation The article by Goedhoop et al. provides an important analysis of the role of terminal dopamine release in the nucleus accumbens in processing aversive events that will be of value to researchers interested in the neural mechanisms of reinforcement learning and computational modeling of dopamine function. Using a variety of conditions, the authors provide convincing data in support of the role of accumbal dopamine release in processing aversive events that situate the current report among growing interest and mounting investigations into the role of dopamine in aversion. https://doi.org/10.7554/eLife.82711.sa0 Decision letter eLife's review process Introduction The midbrain dopamine system plays critical roles in motivation, learning, and movement; specifically for learning about rewards and creating motivational states that promote reward-seeking (Berridge and Robinson, 1998; Bromberg-Martin et al., 2010; Berke, 2018; Schultz, 2019). One of the most prominent functions of dopamine is the encoding of a so-called reward prediction error (RPE) signal (Schultz et al., 1997): when a reward is fully predicted by a cue, the increase in dopamine cell firing and terminal release of dopamine shifts 'backward' in time from the moment of reward delivery, to that of cue presentation (Schultz et al., 1997; Flagel et al., 2011). Furthermore, dopamine neurons pause their firing when a predicted reward is omitted and increase their firing in response to the delivery of an unpredicted reward. Thus, dopamine neurons encode the difference between predicted and obtained reward, which is corroborated by the fact that dopamine neuron activity scales with the relative value of reward and unexpected deviations from this value (Bromberg-Martin and Hikosaka, 2009). Although the vast majority of studies focus on the relationship between dopamine and stimuli with a positive valence (rewards), the relevance of the dopamine system in processing stimuli with the opposite valence (aversive) has also generated great interest. In contrast to the primarily stimulatory response of rewards on dopamine activity, the reports on the effect of aversive events on the dopamine system are less consistent. For example, on the level of dopamine neuron cell bodies, aversive stimuli were demonstrated to result in inhibition of neuronal activity (Ungless et al., 2004; Mileykovskiy and Morales, 2011), excitation thereof (Anstrom et al., 2009; Valenti et al., 2011), or no effect at all (Mirenowicz and Schultz, 1996; Fiorillo, 2013). The widely accepted explanation for these varying results is that subpopulations of dopamine neurons exhibit different response profiles to aversive stimuli (Schultz and Romo, 1987; Guarraci and Kapp, 1999; Coizet et al., 2006; Bromberg-Martin and Hikosaka, 2009; Zweifel et al., 2011; Cohen et al., 2012; Lammel et al., 2011), whereby variance is presumably introduced by different types of aversive stimuli, by the fact that some studies were performed in awake and others in anesthetized animals, and by the location and projection targets of the recorded dopamine neurons (Brischoux et al., 2009; Matsumoto and Hikosaka, 2009; Lammel et al., 2011). However, activity at the level of dopamine neuron cell bodies does not necessarily always translate to their projection targets (Mohebi et al., 2019) as axonal terminal release of dopamine may be capable of operating independently from cell body activity (Threlfell et al., 2012). Therefore, in interrogating the entire spectrum of functions of the dopamine system, it is imperative to include measurements of extracellular dopamine concentrations in the projection target. Midbrain dopamine neurons modulate their targets via population signals: dopamine release from a large number of extrasynaptic terminals, combined, constitutes a diffusion-based signal that is perpetuated by volume transmission (Rice and Cragg, 2008). The vast majority of projections from dopaminergic neurons target the striatum and its subregions. Inconsistent with the classic hypothesis positing that the dopamine system broadcasts a uniform signal across the striatum, it has been reported multiple times in recent years that dopamine signals display regional heterogeneity (Willuhn et al., 2012; Willuhn et al., 2014a, Lammel et al., 2011; de Jong et al., 2019; Menegas et al., 2017; Klanker et al., 2017; van Elzelingen et al., 2022a; van Elzelingen et al., 2022b). This heterogeneity is reflected in dopamine responses to aversive events throughout the striatum: whereas microdialysis studies report an increase in dopamine release in the nucleus accumbens in response to aversive events (Young et al., 1993; Young, 2004; Wilkinson et al., 1998; Bassareo et al., 2002; Pascucci et al., 2007; Ventura et al., 2007; Martinez et al., 2008 but see Mark et al., 1991; Liu et al., 2008), studies employing techniques with a higher, subsecond temporal resolution (e.g., fast-scan cyclic voltammetry [FSCV] or fluorescence fiber photometry) arrive at less consistent conclusions. For example, aversive stimuli produced an increase in dopaminergic activity in the nucleus accumbens shell (NAS) in some studies (Badrinarayan et al., 2012; de Jong et al., 2019), but a decrease in others (Roitman et al., 2008; Wheeler et al., 2011; McCutcheon et al., 2012; Twining et al., 2015). Similarly, contradictory findings are also reported in the neighboring nucleus accumbens core (NAC), where studies found both increased (Budygin et al., 2012; Mikhailova et al., 2019; Kutlu et al., 2021; Kutlu et al., 2022) and decreased dopamine activity (Badrinarayan et al., 2012; Oleson et al., 2012; de Jong et al., 2019; Stelly et al., 2019). In contrast, in the tail of the striatum, aversive events exclusively result in increased dopaminergic activity (Menegas et al., 2017; Menegas et al., 2018). Overall, it can be concluded that most studies observe a change in dopaminergic activity in response to aversive stimuli, that there are substantial differences between striatal regions in this response, and that it remains unclear what determines whether aversive events provoke an increase or a decrease in dopaminergic activity within striatal regions. Delineating the role of dopamine in processing aversive events crucially requires understanding what the above-described changes in dopamine signaling encode specifically; or, in other words, whether these changes reflect aversive prediction errors (APEs, in which the dopamine response would reflect the discrepancy between expected and received aversive events) or merely individual aspects of aversive conditioning (such as the presence of aversive stimuli, and/or their prediction). A thorough analysis by Fiorillo, 2013 concluded that dopaminergic midbrain neurons do not encode aversive stimuli, but other studies have observed aspects of a dopamine APE, such as the predictive cue adopting the dopamine response of an aversive stimulus (Guarraci and Kapp, 1999; Oleson et al., 2012; Badrinarayan et al., 2012), or an APE-like response when the aversive stimulus was unpredicted or omitted (Matsumoto and Hikosaka, 2009; Matsumoto et al., 2016; Menegas et al., 2017; Salinas-Hernández et al., 2018; de Jong et al., 2019). However, it should be kept in mind that in case of omission or early termination of an expected aversive event, rewarding aspects of a milder-than-expected aversive event (RPE) may be mixed with an APE (Oleson et al., 2012; Salinas-Hernández et al., 2018; Stelly et al., 2019). To consolidate these contradictory findings, we systematically evaluated whether dopamine truly signals an APE through a series of behavioral experiments in rats, in which we varied the value of the aversive stimulus, context valence, and probabilistic contingencies, and compared aversive and appetitive conditioning. Using FSCV, we measured the NAC dopamine response to these conditions since the NAC is a hotspot for RPE-like signals (Flagel et al., 2011; Papageorgiou et al., 2016) and is also tightly linked to motivational processes related to aversion avoidance (Badrinarayan et al., 2012; Oleson et al., 2012; Stelly et al., 2019). We employed loud white noise (WN) as the aversive stimulus. WN possesses several merits as it is well-tolerated by rats and is not painful, precisely controllable (intensity and duration can be effortlessly titrated), aversive without inducing freezing (most pertinent to this study as this might interfere with the dopamine signal), does not jeopardize the recording equipment, does not introduce artifacts to the recordings, and can be administered reliably (see 'Discussion'). Based on the above-described findings, we hypothesized that NAC dopamine would exhibit an APE. However, we found that NAC dopamine concentration ramps down in response to both WN exposures and its predicting cue, ramps back upward upon stimulus cessation, that these ramps were qualitatively different from appetitive conditioning, and were inconsistent with a full APE signal. Results WN is aversive We established the aversiveness of loud WN in a series of experiments. First, we used a real-time place aversion test in an open field, in which rats were exposed to 90 dB WN upon entry into one of the four quadrants of an open field (the WN quadrant was assigned randomly; Figure 1A). An example path traversed by a rat over the course of 30 min is depicted in Figure 1A (left panel), where the dark shaded area represents the quadrant paired with WN. On average, rats (n = 10) spent a significantly smaller percentage of time in the WN quadrant (9.71 ± 1.65) compared to chance level (t(9) = 9.585, p<0.0001; Figure 1A, right panel). Second, we exposed a new cohort of rats (n = 15) to the same open field without WN exposure to assess their preferred quadrant (Figure 1B, left panel). After this session, rats were placed in the open field in a second session, where now the entry into their preferred quadrant led to 90 dB WN exposure (Figure 1B, middle panel), which caused the rats to spend significantly less time in it (no WN: 48.5 ± 0.2474, 90 dB: 7.983 ± 0.3875, t(14) = 13.43, p<0.0001; Figure 1B, right panel). Next, a third cohort of rats (n = 12) was exposed to an open-field foraging task in which food pellets were placed between grid-floor bars in one of the quadrants. Entry into this grid-floor quadrant led to exposure of 0, 70, 80, 90, or 96 dB of WN, or a tone in separate sessions (Figure 1C, left panel). Without WN, rats spent 65.2% of the time in the maze foraging in the grid-floor quadrant. The greater the WN intensity, the shorter the amount of time rats spent foraging for food pellets (χ2(5) = 27.6, p<0.0001), whereas the tone did not reduce foraging (Figure 1C, middle panel), indicating that WN is more aversive than a pure tone (t(10) = 2.389, p=0.0381). The average number of entries into the grid-floor quadrant per session was 25.9 ± 2.5. The number of entries did not differ significantly across auditory stimuli. Rats were exposed to each auditory stimulus during foraging in two rounds of sessions, where they underwent exposure to all WN intensities and the tone in a first round of sessions, before exposure to each in a second round of sessions. The rats' response to the different WN intensities and the tone was reliable across first and second sessions (Figure 1C, right panel), demonstrating the absence of sensitization or habituation of the behavioral response to WN across sessions and days. Interestingly, exposure to WN stimulated locomotor activity: rats in an operant box significantly increased locomotion speed in response to semi-random presentation of 6 s WN bouts (t(13) = 7.059, p<0.0001; Figure 1D, middle panel). This locomotor response was WN-intensity dependent as we found a main effect of intensity on baseline-subtracted locomotion speed (χ2(3) = 13.80, p=0.0032) in another cohort of rats (n = 13), and significant differences between 70 and 90 dB (p=0.005) and 70 and 96 dB (p=0.0143) (Figure 1D, right panel), which demonstrates that rats were able to discriminate between the different WN intensities. Rats responded with a comparable increase in locomotion speed to WN on days 1 and 6 of behavioral training, underlining the stability of the WN effect across sessions and days (t(16) = 0.9111, p=0.3757; Figure 1E). Next, we validated that rats can discern between different magnitudes of WN in an operant choice task. Here, rats could choose between pressing one of two extended levers, both of which prompted immediate delivery of a food pellet, but one of which additionally presented 5 s of 0, 70, 90, or 96 dB WN. Unsurprisingly, rats (n = 6) preferred the non-WN lever as indicated by a significant main effect of WN intensity using a Friedman test (χ2(3) = 11.57, p=0.0003). Importantly, when comparing the WN-paired lever presses, rats significantly preferred the 70 dB WN to the 96 dB WN (post-hoc Dunn's tests, p=0.0027; Figure 1F). Figure 1 Download asset Open asset White noise (WN) is an aversive stimulus that lowers dopamine concentration in the nucleus accumbens core (NAC). (A) Left: example trajectory of a rat in the real-time place aversion test (30 min), in which entry into one quadrant (shaded), which was randomly determined prior to the session, led to 90 dB WN exposure. Right: aversiveness of WN in the place aversion test quantified as significantly decreased time spent in the WN quadrant (n = 10, 9.71 ± 1.65, t(9) = 9.585, p<0.0001). (B) Left: a second cohort of rats was placed in the open field to assess their preferred quadrant in a first session. In a second session, the entry into the preferred quadrant led to 90 dB WN exposure. Right: when WN was administered in this quadrant, rats spent significantly less time in it (n = 15, No WN: 48.5 ± 0.2474, 90 dB: 7.983 ± 0.3875, t(14) = 13.43, p<0.0001). (C) Left: a third cohort of rats (n = 12) was exposed to an open-field foraging task in which food pellets were placed between grid-floor bars in one of the quadrants. Entry into the grid-floor quadrant (shaded) led to exposure of 0, 70, 80, 90, or 96 dB of WN, or a 3 kHz tone in separate sessions. Middle: increasing WN intensity dose-dependently decreased the amount of time rats spent foraging for food pellets in the grid-floor quadrant (χ2(5) = 27.6, p<0.0001), whereas the tone did not reduce foraging (70 dB WN vs. 70 dB tone (#); t(10) = 2.389, p=0.0381). Right: the rats' response to the different WN intensities and the tone was reliable across first and second sessions of exposure, where rats underwent exposure to all WN intensities and the tone in a first session, before exposure to each in a second session. (D) Left: all other experiments took place in operant boxes equipped with a food magazine, a multiple-tone generator (cue speaker), and a WN generator (WN speaker). Middle: semi-random presentations of 6 s WN bouts increased the locomotion speed of rats (cohort 4; n = 14) in an operant box during the WN epoch compared to pre-WN baseline (post-hoc Dunn's test, t(13) = 7.059, p<0.0001). Right: in another cohort of rats (cohort 5; n = 13), we tested different WN intensities and found a main effect of intensity on locomotion speed (χ2(3) = 13.80, p=0.0032) and significant differences between 70 and 90 dB (p=0.005) and 70 and 96 dB (p=0.0143) (n=13). (E) Rats (n = 17) responded reliably with increased locomotion speed to WN across days. (F) Rats (cohort 6; n = 9) discern between different WN magnitudes in an operant choice task, where they had to choose between pressing a lever that resulted in a food-pellet delivery and a lever that resulted in a food-pellet delivery plus simultaneous 5 s of 0, 70, 90, or 96 dB of WN (pellet + WN; Friedman test, χ2(3) = 11.57, p=0.0003). (G) Left: example cresyl violet-stained brain slice depicting an electrolytic lesion in the NAC (outlined) at the tip of the fast-scan cyclic voltammetry (FSCV) electrode (vertical black line). Right: schematic overview of FSCV recording locations (blue dots) in the NAC (gray) of all animals. (H) Single-trial pseudocolor plot (top panel), dopamine trace (bottom panel), and cyclic voltammograms (inset in bottom panel) for representative, dopamine-specific current fluctuations recorded in NAC, 5 s before WN (dashed line), during 6 s of WN (gray bar), and 14 s after WN. Except for panel (H), data are mean ± SEM. *p<0.05, **p<0.01, ***p<0.001. WN suppresses dopamine release in the NAC All FSCV recordings were conducted in operant boxes equipped with a food magazine, a multiple-tone generator (cue speaker), and WN generators (WN speaker) (Med-Associates; Figure 1D, left panel). Electrolytic lesions at the FSCV electrode tip were used to histologically verify electrode placement (Figure 1G, left panel). Histological analysis confirmed for all animals included in this study that the sensing end of their FSCV electrodes was consistently placed in the NAC (Figure 1G, right panel). Unexpected exposure to 6 s of 90 dB WN strongly and reliably decreased extracellular concentrations of dopamine in the NAC (Figure 1H). Different temporal NAC dopamine dynamics during aversive and appetitive Pavlovian conditioning Dopamine release in the NAC is often consistent with a temporal difference RPE, where an increase in dopamine activity, initially time-locked to the delivery of a reward, shifts backward in time to its predicting cue. It is assumed that this phenomenon reflects the learned association between predictive cue and reward, where the reward becomes fully predicted by the cue and, therefore, no prediction error occurs at the time of reward delivery after sufficiently repeated cue-reward pairings (e.g., Schultz et al., 1997; Flagel et al., 2011). Our first experiment investigated whether a similar phenomenon also applies to aversive stimuli and their predictors, and in what time frame such a shift may occur. Rats (n = 16) were exposed to 30 pairings of cue (5 s) and 90 dB WN (6 s) that were separated by a variable intertrial interval (Figure 2A). In order to visualize the rapid changes in dopamine response presumably reflecting learning, the first five are depicted and on their and were Using in which we compared the average dopamine concentration during cue, and WN we found significant main in all = = = = = p<0.0001; = = p<0.0001). using with a significantly dopamine concentrations during the WN epoch compared to baseline during 1 = = 3 = 5 = = and = p<0.0001), indicating that the decrease in dopamine during WN is an response, since it is observed during the first In contrast, we only observed significantly dopamine in the cue epoch compared to baseline during 5 = = and = p<0.0001), indicating that this decrease over time and, therefore, is a In these first 30 the decrease in extracellular dopamine during the WN epoch did not or Thus, WN does not provoke a substantial temporal shift of NAC dopamine signaling from to cue within the first session of Figure Download asset Open asset accumbens core dopamine signaling and rat during Pavlovian white noise conditioning and varying WN intensities. (A) extracellular concentrations of dopamine in in the NAC is shaded during the first 30 pairings of cue (5 s and WN (6 90 To the of WN, the first five are The dopamine release for cue, and WN WN decreased dopamine significantly in all The WN-paired cue to decrease dopamine significantly at (B) of dopamine release during aversive n = and appetitive n = 10) Pavlovian conditioning. subsecond changes in dopamine concentration on 1 and 6 of the between the and on 1 and and 6 during aversive and appetitive conditioning. For aversive conditioning, dopamine between 1 and 1 and between 1 and 6 For appetitive conditioning, a significant difference was found between early and conditioning on 1 and dopamine between 1 and 1 compared to (C) behavioral response to Left: during aversive conditioning, locomotion speed during cue presentation increased from 1 to 6 = and also during WN = (n = Right: during appetitive conditioning, time spent in of the reward increased between days 1 and 6 both during cue presentation (t(9) = and after delivery (t(9) = (n = (D) WN and its paired cue decreased dopamine concentration reliably across days. decrease was between days 5 and and decrease was between days and (E) In an session, for WN was after cue presentation (n = and dopamine significantly between and (F) In contrast, we no differences in dopamine release between exposure to varying WN intensities 80, 90, or 96 = n = (G) We observed no significant differences in dopamine during cue = or WN = when two separate were used as for dB or 96 dB WN (n = (H) between locomotion speed and dopamine concentration during cue and WN for dB or 96 dB WN were not represents one from all animals (n = 9) were no during dB WN = or its cue = no during 96 dB WN = or its cue = Right: for each individual rat that there is no significant between locomotion speed and Data are mean ± + or SEM. *p<0.05, **p<0.01, ***p<0.001. One explanation for an shift of dopamine signaling from WN to cue is that 30 pairings are to fully the Therefore, we a of rats (n = for five days. of rats (n = 10) received food-pellet rewards paired with a predictive cue to the temporal dynamics of aversive dB WN; Figure and appetitive conditioning (Figure in dopamine are across time (Figure and in order to the shift of the dopamine response from the to the we the between the the between during the first of conditioning the of the of 1 and on the of conditioning (see Figure aversive conditioning, we using a a significant main effect of the of conditioning = and using test significant differences between the of 1 early and 1 as as between 1 early and 6 However, no difference was observed between 1 and 6 appetitive conditioning, we also found a main effect of the of conditioning on the between the and the = and, in contrast to aversive conditioning, the on 6 is significantly different from both 1 early and 1 In we found a significant difference between 1 early and 1 The of behavioral responses to 90 dB WN between 1 and 6 using and a for multiple an increase in locomotion speed (Figure compared to on was to the WN epoch = WN: = p<0.0001). On we observe an increase in locomotion speed during both the cue = and WN = epoch compared to which are both also significantly compared to 1 = WN: = appetitive conditioning, we see the same temporal in the response, where rats the reward more during the cue epoch on 6 compared to 1 (t(9) = p<0.0001). However, during appetitive conditioning, but not during aversive conditioning, an shift of the dopamine response from the to the Together, these results demonstrate differences in the temporal dynamics of dopamine signaling during aversive and appetitive conditioning. and of dopamine signaling To test the with which WN and its paired cue decrease dopamine concentration across we compared behavioral sessions on days and The decrease in dopamine concentration more between days 1 and which is by the cue during 1 (Figure 2A). decrease was between days 5 and and decrease was between days and (n = = 1 vs. 1 vs. 5 vs. WN: = Figure In to the of the (Figure to verify that these were a learned response, we tested the association between cue and WN could be Rats with were for more than 6 were exposed to in which WN was Using a Friedman test, we found a significant effect of = analysis using a Dunn's test a significant difference between the decrease in extracellular dopamine concentration of and Figure with the no a decrease in Thus, over the course of the cue its dopamine NAC dopamine does not reflect WN intensity and For the we used WN with an intensity of 90 We whether WN of different intensities would extracellular dopamine in the NAC since we observed increased avoidance of intensities of WN (Figure and greater locomotion in response to intensities of WN (Figure 1D, right panel). First, we tested whether there is a relationship between different WN intensities and We exposed rats (n = 6) to four different intensities of WN 80, 90, and 96 which were in a semi-random order (Figure Although all WN intensities decreased dopamine we found no significant effect of intensity