Title: Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dynamics of neural representations

probabilistic neural predictive models

primary visual cortex of mice

variational encoding models

stimulus-independent variability
Abstract: The neural activity in the visual processing is influenced by both external stimuli and internal brain states. 
Ideally, a neural predictive  model should account for both of them.
Currently, there are no dynamic encoding models that explicitly model a latent state and the entire neuronal response distribution.
We address this gap by proposing a probabilistic model that predicts the joint distribution of the neuronal responses from video stimuli and stimulus-independent latent factors. 
After training and testing our model on mouse V1 neuronal responses, we find that it outperforms video-only models in terms of log-likelihood and achieves improvements in likelihood and correlation when conditioned on responses from other neurons. 
Furthermore, we find that the learned latent factors strongly correlate with mouse behavior and that they exhibit patterns related to the neurons’ position on the visual cortex, although the model was trained without behavior and cortical coordinates.  
Our findings demonstrate that unsupervised learning of latent factors from population responses can reveal biologically meaningful structure that bridges sensory processing and behavior, without requiring explicit behavioral annotations during training.
The code is attached to the submission.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel algorithm for predicting neural responses to dynamic naturalistic stimuli while modeling stimulus-independent variability in the neuronal population (internal brain states). \n\nMost previous approaches tend to either predict neural activity from stimuli without modeling a latent state or focus on modeling a latent state from neural responses. Some previous work predicts neural activity based on sensory and neuronal input. However, the only previous work combining visual input with a latent state is restricted to static stimuli. \n\nTo bridge this gap, the authors propose a model to predict neural activity combining video visual input with a latent state. Technically, it consists of a 3D convolutional neural network “core” that processes the video frames and a latent variable that could be conditioned upon neural activity and captures stimulus-independent fluctuations. The response of each neuron is modeled with a Zero-Inflated Gamma (ZIG) distribution whose parameters depend on both the video-driven features and the latent state. ZIG is chosen because it aligns better with calcium imaging data.\n\nThe model is evaluated on neural recordings from mouse primary visual cortex (V1) in response to grayscale natural video stimuli (from the SENSORIUM dataset). The proposed video+latent model improves the log-likelihood of the neural responses over baselines and also achieves comparable Pearson correlation. In addition, the proposed method seems to handle out-of-distribution stimuli better. \n\nMore importantly, the latent states highly correlate with behavioral variables (pupil dilation and locomotion speed), although they are not used in training. This suggests the latent state successfully captured intrinsic neural dynamics. The latent states also seem to exhibit topographic organization, although the cortical positions of the neurons are not used in the training. Those conclusions show that the latent features uncovered by the model have a meaningful biological basis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper proposes a novel integration of dynamic visual stimuli and latent brain states in order to predict neural activity, addressing a gap in the literature.\n2. The latent states (learned in an unsupervised fashion during training) have a meaningful biological basis. They highly correlate with behavioral variables, and they reflect an anatomical organization.\n3. The model handles OOD samples better\n3. The paper conducted a thorough experiment and analysis\n\nWeaknesses:\n1. The proposed model (more complex than the baseline) does not substantially improve (and in fact slightly underperforms) the baseline in terms of predicting the mean neural response as measured by correlation. The authors have provided an explanation for this, but I am not sure if the method's added complexity and parameters.\n2. To remedy the above weakness, maybe the authors can show that the latent states (the other important part learned in the model) are somehow \"better\" than a baseline. But such a baseline is missing from the paper.\n\nQUESTIONS:\n1. Regarding weakness #2: Would you be able to evaluate the quality of the latent factors against a baseline? I am impressed that they have a solid biological basis, but I am wondering if they are somehow better than a baseline because they are learned together in a neural activity prediction task.\n2. Mouse v1 neurons should encode other features such as orientation. Your analysis in table 3 shows that the latents encode contrast minimally. Do you think it'd be meaningful to extend your analysis to other features?\n3. For the neurons that are conditioned upon - how are they chosen? I would assume they are chosen randomly from reading the paper. Does using a different subset of neurons matter? Does restricting the choice to purely layer 2/3/4/5 matter? I am wondering if the different choice here can lead to \"better\" latent states or higher accuracy.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission introduces a probabilistic latent-variable model that predicts the full distribution of mouse V1 calcium responses using a video-encoding core and an inference model capturing a low-dimensional, stimulus-independent brain state. Compared with video-only baselines, the model (i) raises test log-likelihood, (ii) generalises better to OOD inputs when conditioned on partial population activity; (iii) the learned latents correlate with behaviour and reveal a cortical topography without supervision.\n\nSTRENGTHS AND WEAKNESSES:\n1. Quality: \n- (+) The paper is technically sound with a well-formulated probabilistic model. \n- (-) It would be important to include errors in Table 4. The claims cannot be fully substantiated without them.\n\n2. Clarity:\n- (+) Clearly written and well-organised\n- (-) Minor typos (e.g. “non-condititoned”, “percise”). \n- (-) Readers unfamiliar with ZIG would definitely benefit from a short paragraph before Eq. 1.\n\n3. Significance:\n- (+) First probabilistic model that jointly handles natural-movie input and models latent brain state for neuronal response prediction, filling a gap in dynamic encoding models;\n- (+) The latent factors capture behaviour and cortical topography without supervision;\n- (-) Comparisons stop at the Sensorium CNN/GRU baseline; the authors acknowledge more recent transformer cores [3, 27] but do not test on them, so the incremental benefit on modern architectures is unknown.\n- (-) Impact seems questionable due to the modest gain in correlation against baselines and due to the fact that conditioning requires simultaneous recording of many neurons. This would limit immediate applicability to smaller-scale datasets.\n\t    \n4. Originality:\n- (+) Novel combination of (a) ZIG, (b) latent-state inference from a subset of neurons, and (c) dynamic video core. Prior latent models either ignore stimulus or are image-based; the present work unifies both for video. The discovery of topographic maps in the weights that connect latents to neurons is interesting.\n- Methodologically the pieces (variational auto-encoding, ZIG, 3D CNN core) are known, but the novelty lies in their integration.\n\nQUESTIONS:\n1. Have you tried replacing the 3D CNN core with a recent ViT-based core such as V1T? Even a small-scale experiment would show whether the latent state provides performance gains, which would be quite important, in my opinion. \n\n2. Could you please add standard errors or statistical significance results where they are missing?\n\n3. On the significance of the paper: could the authors please clarify whether the inference of latent variables is performed conditioned on the neuronal responses given the same stimuli and from the same timesteps? If so, I struggle to justify the significance, or rather the claimed improvement, in the model's ability to better predict the responses of other neurons. If one assumes that a subset of neurons at timestep $t$ can inform the prediction of (or can contain information about) other neurons, then it is only natural that a model that conditions its prediction on a subset of neurons can theoretically perform better than a model that does not. As such: Is a comparison to models that do not condition on the subsets of neurons justified / fair / meaningful? I would be happy to hear your clarifications on this matter.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a probabilistic latent-variable framework that models neural data to videos by combining stimulus-driven features with a low-dimensional, stimulus-independent brain state, addressing variability that cannot be explained by the visual input alone. Building on a 3D CNN core and a Gaussian readout, the model predicts a Zero-Inflated-Gamma (ZIG) response distributions whose parameters are jointly modulated by the video and the inferred latent variables. Compared with Poisson or video-only ZIG baselines, it improves log-likelihood and single-trial correlation. These results demonstrate that unsupervised inference of latent variables can enhance predictive accuracy and discover biologically meaningful structure without requiring behavioural supervision.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. The authors proposed a 3D video encoder with a Zero-Inflated-Gamma (ZIG) likelihood whose parameters are jointly shaped by stimulus features and a latent state. This combination goes beyond the common Poisson-mean setting, making the model capture both firing-rate modulations and trial-to-trial variability in a mathematical way. \n\n**Weaknesses**:\n\n1. A minor concern is that this paper spends several dense paragraphs summarizing prior methods, where many only loosely connected to the proposed latent-ZIG model, which distracts from the main ideas and makes the paper feel unfocused.  A brief overview in the main text with further details moved to the supplementary material would improve clarity and readability.\n\nQUESTIONS:\n1. This paper compared a video-only ZIG-model without latent to disentangle the impact of the ZIG. Could you provide a latent-Poisson baseline using the same video core? One suggestion is to keep the CNN core and readout but predict Poisson rates, then compare likelihood and correlation with the latent-ZIG model. This will help us understand whether the observed performance gains come from the ZIG likelihood itself or from the latent design.\n\nLIMITATIONS:\nThe authors have discussed the limitations in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a probabilistic model for predicting neural activity in the mouse primary visual cortex (V1) in response to video stimuli. The model aims to account for both the stimulus and unobserved internal brain states by incorporating a latent variable inferred from a portion of the recorded neural population. The authors evaluate this model on a public dataset, reporting improvements in log-likelihood over a video-only model. The study also analyzes the learned latent variables, showing they correlate with the mouse's behavior and have a spatial organization related to the cortical layout of the neurons.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n **A clear contribution:** \n\nThe paper addresses the important problem of modeling stimulus-independent variability in neuroscience. The proposed model, which combines a dynamic video encoder with a latent variable framework for predicting full response distributions, is a logical step forward in the field. \n\n**Sufficient experimental validation:** \n\nThe authors perform a number of experiments to validate their approach. They compare against reasonable baselines, test on out-of-distribution stimuli, and perform control analyses to investigate the properties of the learned latent variables.\n\n\n**Biologically relevant findings:** \n\nA key result is that the latent variables, learned without supervision from behavior or location data, correlate with external variables like mouse behavior and cortical topography. This suggests the model is capturing meaningful structure in the neural data. \n\n\n## Weaknesses:\n\n**1. Performance on key metrics:** \n\nA notable weakness is that the proposed model performs worse on the single-trial correlation metric compared to a simpler Poisson baseline. While the authors report an improvement in log-likelihood, the trade-off with the correlation metric, which is a standard measure of predictive accuracy in this domain, is a significant concern. \n\n**2. Model and training complexity:** \n\nThe latent variable model introduces considerable complexity over the baselines. The appendices show that the best performance is achieved only when initializing from a pre-trained model, which suggests that training from scratch is challenging and may lead to poor solutions. This could be a practical barrier to adoption. \n\n**3. Purity of latent states:** \n\nThe central claim is that the latents capture stimulus-independent variability. However, the authors' own analysis shows that for one of the five mice, the latents contain residual stimulus information. This finding, while reported transparently, somewhat undermines the model's core premise.\n\nQUESTIONS:\n1. The justification for using a Zero-Inflated Gamma (ZIG) distribution over a Poisson one is clear, given that ZIG is better suited for the continuous and sparse nature of calcium imaging data. However, the choice of the Gamma component itself is worth probing. Have you considered other heavy-tailed distributions to model the positive responses? Do you have any intuition on whether this might help close the approximation gap?\n\n2. Here is a suggestion for a baseline that could further contextualize your model's performance. Instead of a direct ZIG observation model, an alternative would be a more biophysically-inspired model. One could first predict underlying discrete “spikes” with a Poisson model and then apply a deterministic temporal convolution to model the slow dynamics of the calcium transients. A comparison against such a benchmark would be very interesting to see and would significantly strengthen the paper's claims about the effectiveness of your end-to-end framework.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}