DOI: 10.5061/dryad.mkkwh71f6
title: Prospective contingency explains behavior and dopamine signals during associative learning

description: Associative learning depends on contingency, the degree to which a
 stimulus predicts an outcome. Despite its importance, the neural
 mechanisms linking contingency to behavior remain elusive. In the present
 study, we examined the dopamine activity in the ventral striatum—a signal
 implicated in associative learning—in a Pavlovian contingency degradation
 task in mice. We show that both anticipatory licking and dopamine
 responses to a conditioned stimulus decreased when additional rewards were
 delivered uncued, but remained unchanged if additional rewards were cued.
 These results conflict with contingency-based accounts using a traditional
 definition of contingency or a new causal learning model (ANCCR), but can
 be explained by temporal difference (TD) learning models equipped with an
 appropriate intertrial interval state representation. Recurrent neural
 networks trained within a TD framework develop state representations akin
 to our best ‘handcrafted’ model. Our findings suggest that the TD error
 can be a measure that describes both contingency and dopaminergic
 activity.
description: # Prospective contingency explains behavior and dopamine signals during
 associative learning ## Directory Structure ``` . ├── README.md ├── Code/
 │ └── Code/ │ ├── checked_figure_code/ # R code for generating figures │
 ├── matlab_simulation_code/ # MATLAB code for TD and ANCCR simulations │
 ├── photometry_data_analysis/# Python code for photometry analysis │ ├──
 python_simulation_code/ # Python code for TD and RNN simulations │ ├──
 r_analysis_code/ # R code for statistical analysis │ └── value_rnn/ #
 Python package for value-RNN models └── Data/ ├── photo_data.parquet #
 Processed photometry data ├── lick_data.parquet # Licking behavior data
 ├── trial_data.parquet # Trial-level behavioral data ├── mouse_table.csv #
 Mouse metadata (group, sex, photometry) ├── Photometry Mice/ # Photometry
 and behavioral data │ ├── Raw_data/ # Raw photometry data (zipped) │ └──
 Preprocessing/ # Code for preprocessing of raw data └──
 Pre_processing_code/ # Data preprocessing scripts ``` ## Data Files ####
 File: Data.zip ### Main Data Files | File | Description | Format | |
 -------------------- |
 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | | `mouse_table.csv` | Mouse metadata including group assignment (conditioning/degradation/cuedrew), sex, and photometry status | CSV | | `lick_data.parquet` | Trial-by-trial licking timestamps for all mice - variables are: lick (timestamp in seconds, relative to start of trial of lick), trialtype,    trialnumber, day (session), mouse (name) | Apache Parquet | | `trial_data.parquet` | Trial-level information including trial type, timing, and outcome | Apache Parquet | | `photo_data.parquet` | Processed fiber photometry data from ventral striatum: variables are: trial,    time (in seconds, relative to start of trial), day (session), data (z-scored photometry signal), mouse (name), site (recording location, see abbreviations below),        trial\_type | Apache Parquet | ### Raw Data Raw photometry data files are provided as zip archives in `Data/Photometry Mice/Raw_data/`: * `CuedRew.zip` - Raw data from the cued reward group * `Deg.zip` - Raw data from degradation group The organization of these files is as follows: Photometry Mice\Raw_data\\[Group: CuedRew or Deg].zip\\[Group: CuedRew or Deg].zip\\[animal_name]\Selina_C5D5R3E5R3 Within this, there is one folder called "Session Settings" containing a mat file with behavioral parameters include odor identity. In the folder called "Session Data" are three folders: Bpod - containing bpod raw data (files named Animal_Task_Date_session.mat). See bpod wiki for further information about this format: [https://sanworks.github.io/Bpod_Wiki/user-guide/](https://sanworks.github.io/Bpod_Wiki/user-guide/) Doric - containing the raw output of the photometry recordings as csv files, following the same naming convention as the bpod folder. See the Doric website for further details on the system: [https://neuro.doriclenses.com/collections/software/products/doric-neuroscience-studio](https://neuro.doriclenses.com/collections/software/products/doric-neuroscience-studio) Processed - Mat files, processed using the code provided in Code\photometry_data_analysis\preprocessing. Naming convention as above. Note that this data is compiled into the .parquet files provide as main data files for ease of use and legibility.  ### Mouse Groups Mice were assigned to three experimental groups: * **Conditioning (Cond):** Continued standard conditioning (n=6 behavioral only) * **Degradation (Deg):** Received uncued background rewards (n=11 total, n=8 with photometry) * **Cued Reward (CuedRew):** Received cued background rewards (n=13 total, n=5 with photometry) ### Photometry Recording Sites Fiber photometry recordings were made from six locations in the ventral striatum: * Lateral nucleus accumbens (lNAc) - primary analysis site * Medial nucleus accumbens (mNAc) * Anterior lateral olfactory tubercle (alOT) * Anterior medial olfactory tubercle (amOT) * Posterior lateral olfactory tubercle (plOT) * Posterior medial olfactory tubercle (pmOT) ## Code Files #### File: Code.zip ### Figure Generation (`Code/Code/checked_figure_code/`) R scripts for reproducing manuscript figures: | File | Description | | ------------------------------- | ---------------------------------------------------------------------------------- | | `00_data_import.R` | Data loading and preprocessing | | `01_figure1.R` - `07_figure7.R` | Main text figure generation | | `S1_figure1.R` - `S6_figure6.R` | Extended data figure generation | | `flm_effect_plotter.R` | Functional linear model plotting utilities, called by various figure plotting code | ### Figure Descriptions | Figure | Brief Description of Associated Figure Code | | :--------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Figure 1 | Summarizes the behavioral (lick timing) data in all conditions | | Figure 2 | Plots example data and summarizes the fiber photometry data, in particular the response to odor A across all conditions | | Figure 3 | Plots Temporal Difference (TD) models explaining the pattern of results seen. See the associated MATLAB and Python code for model data generation. | | Figure 4 | Summarizes the behavioral and photometry data that supports the Belief-State TD model | | Figure 5 | Summarizes the non-odor A responses in the photometry data and compares against the Belief-State TD model | | Figure 6 | Summarizes and compares the Value-Recurrent Neural Network (Value RNN) model of the photometry data and visualizes the state space. See the value RNN python code description below for generation of data. Associated files (dataprep, run\_cca.py) perform data loading and Canonical Correlation Analysis (CCA), respectively. | | Figure 7 | Summarizes the ANCCR model data. See the Matlab code section below for details on running the ANCCR models. | | Supplementary Figure 1 | Summarizes other behavioral data across all groups | | Supplementary Figure 2 | Summarizes photometry data across all areas recorded and performs pairwise correlation | | Supplementary Figure 3 | Explores the effect of lick timing on reward response | | Supplementary Figure 4 | Summarizes the effect of discounting (gamma) on the model predictions of Figure 4 | | Supplementary Figure 5 | Summarizes the predictions of a microstimuli TD model (see python code below) | | Supplementary Figure 6 | Summarizes other predictions of the value RNN and Belief-State TD model and compares to actual photometry data | ### Computational Modeling #### MATLAB Simulation Code (`Code/Code/matlab_simulation_code/`) * `generate_simulated_experiments.m` - Generate simulated behavioral experiments * `run_belief_state_model.m` - Belief-State TD model simulations (Figure 3) * Variations are `run_belief_state_model_extinction.m` and `run_belief_state_model_changetransition. m`, which models the Belief-State model of extinction (Figure 5) and the effect of changing transition probabilities on model predictions (Extended Data Fig 4D and E) * `run_anccr_model.m` - ANCCR model simulations * `compute_belief_r2.m` - computes the R2 values used to compare the Belief-State model to the value RNN data used in Figure 6 * `td_belief.m` - TD learning with belief state representation - code called by run_belief_state_model to do the TD belief-state modelling. Variation `td_belief_garr.m` considers outcome-specific value (Extended Data Figure 8) * `useful_functions/` - Helper functions called by simulation code * `anccr_simulation_cluster_code/` - Versions of ANCCR simulation (see above) designed to be run on Cluster Computing #### Python Simulation Code (`Code/Code/python_simulation_code/`) * `train_rnns.py` - Train value-RNN models (Figure 6) * `generate_rnn_test_data.py` - Generates simulated data from trained value-RNN models (Figure 6) * `main_garr.py` - Simulations for outcome-specific degradation (Extended Data Figure 8) * `do_td_sim.py` - TD model simulations (Figure 3) * `do_td_sim_ms_fullrun.py` - TD model microstimuli simulations (Extended Data Figure 5) * `td_sim/` - TD simulation module - imported by `do_td_sim.py`. * `rnn_analysis/` - helper functions imported by `train_rnns.py`. #### Value-RNN Package (`Code/Code/value_rnn/`) Python package for training recurrent neural networks to predict value: * `valuernn/model.py` - RNN model architecture * `valuernn/train.py` - Training routines * `valuernn/tasks/` - Task implementations (contingency, blocking, etc.) See install instructions below - code is called by train_rnns.py (see above) ### Photometry Analysis (`Code/Code/photometry_data_analysis/`) * `preprocessing/` - Raw data processing scripts involving the following steps * a: process bpod and doric files to align behavior and photometry (Matlab code) * b: combine sessions into a single file per mouse (Python code) ## Software Requirements ### R (version 4.4+) Required packages: * tidyverse * arrow * lme4 * lmerTest ### Python (version 3.9+) Required packages: * numpy * scipy * matplotlib * pytorch * scikit-learn * pyarrow * pyrcca (for CCA analysis) Install value-RNN package: ```bash conda create --name valuernn python=3.9 pytorch matplotlib numpy scipy scikit-learn conda activate valuernn cd Code/Code/value_rnn pip install -e . ``` ### MATLAB (version 2022b+) Required toolboxes: * Statistics and Machine Learning Toolbox * Parallel Computing Toolbox (optional, for faster simulations) ## Reproducing Results ### Behavioral and Photometry Analysis 1. Set working directory to `Code/Code/checked_figure_code/` 2. Ensure data files are in `./data/` subdirectory (or adjust paths in `00_data_import.R`) 3. Run figure scripts in order: ```r source("00_data_import.R") source("01_figure1.R") # etc. ``` ### TD Model Simulations 1. Navigate to `Code/Code/matlab_simulation_code/` 2. Run `generate_simulated_experiments.m` to create simulated experiments 3. Run `run_belief_state_model.m` for Belief-State TD simulations 4. Run `run_anccr_model.m` for ANCCR comparisons ### RNN Model Training 1. Navigate to `Code/Code/python_simulation_code/` 2. Run: ```bash python train_rnns.py ``` ## Experimental Methods Summary ### Animals * 31 mice total (18 wild-type, 13 DAT-Cre x Ai148 transgenic) * Age: 3-6 months * Both sexes (balanced across groups) ### Phases 1. **Conditioning (Phase 1):** 5 sessions, standard odor-reward pairing 2. **Manipulation (Phase 2):** 5 sessions * Cond: continued conditioning * Deg: uncued background rewards added * CuedRew: cued background rewards added 3. **Recovery/Extinction (Phases 3-5):** Deg group only ### Photometry * GCaMP6f expressed in dopamine neurons (DAT-Cre x Ai148) * Multi-fiber recording from 6 ventral striatum locations * 473nm excitation, 405nm isosbestic control * 20 Hz effective sampling rate