Title: Modeling Neural Activity with Conditionally Linear Dynamical Systems
Venue: NeurIPS 2025
Accepted: yes
Keywords: Linear Dynamical Systems

Gaussian Processes

Neural data analysis

Expectation-Maximization
Abstract: Neural population activity exhibits complex, nonlinear dynamics, varying in time, over trials, and across experimental conditions. Here, we develop *Conditionally Linear Dynamical System* (CLDS) models as a general-purpose method to characterize these dynamics. These models use Gaussian Process priors to capture the nonlinear dependence of circuit dynamics on task and behavioral variables. Conditioned on these covariates, the data is modeled with linear dynamics. This allows for transparent interpretation and tractable Bayesian inference. We find that CLDS models can perform well even in severely data-limited regimes (e.g. one trial per condition) due to their Bayesian formulation and ability to share statistical power across nearby task conditions. In example applications, we apply CLDS to model thalamic neurons that nonlinearly encode heading direction and to model motor cortical neurons during a cued-reaching task.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce conditionally linear dynamical systems (CLDS), where system parameters are modulated nonlinearly as a function of observed covariates $u_t$ (e.g., via a kernel mapping). This allows the latent dynamics to flexibly vary across contexts or conditions, while retaining the interpretability and tractability of linear state-space models. The model is fit using an approximate GP prior over parameters, enabling closed-form EM updates under Gaussian noise assumptions.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*:\nOverall, I like this paper. It’s clearly written, easy to follow, and communicates its ideas well. The motivation for combining input-dependent parameter modulation with linear dynamics is compelling, especially in neuroscience where interpretability matters. The figures are informative, and the structure of the paper makes it straightforward to trace the reasoning from model definition to inference to experiments. The theoretical results in Appendix A.2, especially the bounds on the approximation error for composite dynamics (A.2.2), are a valuable addition. Overall, it’s a useful contribution that sits in an important design space between black-box flexibility and mechanistic transparency.\n\n*Weaknesses*:\n\n*Overall Weakness*:\n While the paper presents a clean and interpretable framework, its empirical scope feels slightly narrow, and the evaluation does not seem to probe the limits or broader applicability of the method. All experiments are tailored to idealized conditions that align well with the model’s assumptions (Gaussian observations, well-structured modulators, modest data scale, and low latent dimensionality). Many of the paper’s selling points (e.g., “easy inference,” “simple extensions,” interpretability) are stated as general advantages, but it feels like they are not properly stress-tested or fully explored in more challenging settings. This leaves open the question of whether the performance gains and interpretability benefits are worth the constraints introduced by the modeling assumptions.\n\nThat said, if the authors are able to convincingly address these concerns, e.g. through additional ablations, clarifications of model limitations, and a deeper discussion of scalability and generalizability, I would be happy to support acceptance.\n\n*Spike Data*:\nThe authors emphasize that one of the key advantages of CLDS models is ease of inference, particularly under a Gaussian noise model where Kalman smoothing and closed-form EM enable efficient learning (§2.3). They also note that the extension to non-Gaussian observations (e.g., Poisson spike counts) is straightforward in principle, since the posterior remains log-concave and tractable. However, all real data applications, including the macaque spike recordings, still use the Gaussian observation model. While this is mentioned in the limitations, this feels like a missed opportunity, particularly since the whole paper is tailored to \"modeling neural activity\": if extending to count-based models is practical and aligns better with the data-generating process, it would have been useful to demonstrate this empirically. At minimum, a concrete discussion of what would be required to implement this extension (and what challenges, if any, prevented it here) would help clarify how straightforward this path really is. As it stands, the claim of “easy extensibility” feels somewhat contradicted by its absence in the main experiments, where it would fit perfectly.\n\n*EM Scaling*:\nAs a related point (given EM is highlighted as a major strength of the CLDS framework) while EM is indeed attractive for small- to medium-scale models, it is well known to scale badly with increasing latent dimensionality or large datasets (which would become relevant in many modern neural recording settings). The authors mention that “in principle” the model could be extended to use variational inference or MCMC methods, but do not test these alternatives or provide a discussion of where EM might become a bottleneck. An analysis of scalability (e.g., in terms of latent dimensionality, trial count, or observation length) would strengthen the paper’s claims about practical applicability and robustness. Even a partial exploration of variational alternatives would also help contextualize how “easy to fit” this model remains.\n\n*Kernel function*:\nThe use of an approximate GP prior over system parameters is presented as the paper’s main technical contribution. However, the empirical evaluation does not really explore how the structure or complexity of the kernel, or the form of the input u, affects performance. In the experiments, the modulators are closely linked to the observed neural dynamics (e.g., categorical or smooth latent variables), making it difficult to assess the generality or necessity of the kernel. I could image that there are forms of inputs that would benefit from different kernels. Since the GP prior induces a specific form of smoothness in the system parameters, it could strengthen the paper to include ablations comparing different modulator structures and kernel variants (e.g., alternative basis functions or GP approximations), especially given the centrality of this mechanism to the paper's novelty. For instance, what prior would you recommend if smoothness assumptions over system parameters are violated (e.g. task switches or abrupt transitions)? How do find an optimal $\\kappa$ depending on smoothness?\n\n*Input-driven DS*:\nThe paper briefly mentions an important conceptual parallel to time-varying or input-driven LDSs, where external inputs u_t​ enter the system via a factor-loading matrix B. This structure is known to induce time-dependent fixed points or more complex attractors (e.g., ring attractors) if the input signal is sufficiently expressive. While not explicitly framed this way, the proposed model can be viewed as implicitly implementing a snapshot attractor. Given the close mathematical similarity to classical input-driven LDSs, it would be helpful to more explicitly articulate what is novel about the kernel-based parameterization (particularly e.g. compared to Input Switched Affine Network that are cited by the authors). At least to my understanding, similar expressive dynamics could potentially be captured by using known u_t​ in combination with a flexible encoder. A systematic empirical comparison to such baselines could help clarify this distinction and further strengthen the case for the proposed approach.\n\nQUESTIONS:\n*Comparisons/Related Work*:\nThe proposed approach shares a lot of conceptual similarities linear/structured SSMs, particularly to Mamba, where the recurrence kernel is conditioned on the input sequence, yielding a similarly dynamically reparameterized linear system. While the technical motivations differ (e.g., sequence modeling vs. spike dynamics), it could be beneficial to include this in the related work discussion, also with respect to training algorithms, benefits of linearity etc. Another recent line of work (Almost-Linear RNNs, Brenner et al., NeurIPS 2024) bears close conceptual similarity to recurrent switching linear DS, and could be mentioned.\n\nLIMITATIONS:\nClear limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a class of latent dynamics models for modeling neural activity. In particular, they propose time-varying linear dynamical systems, where the dynamics and readout parameters vary smoothly as a function of external inputs at each point in time. They demonstrate how these models can be efficiently trained, and show that they are performant on both synthetic and real data tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors convincingly show that both training and inference of CLDS models can be efficiently done via classical techniques such as Kalman smoothing and EM. \n- There are many avenues towards interpretibility of CLDS models (e.g. linearizing around a particular input, or marginalizing out the inputs to directly yield possibly nonlinear composite dynamics)\n\nWeaknesses:\n- While the authors do a great job comparing CLDS to models traditionally used in neuroscience (such as (r)SLDS), the class of models introduced here---linear RNNs with parameters being time-varying solely through input dependence---is not at all new in the ML literature, and is a popular formulation of modern selective state space models (e.g. Mamba). The authors should comment on this, and perhaps discuss how the Bayesian perspective (which is indeed distinct from that setting), or the distinct parameterization strategy for achieving input dependence (truncated approximate GP prior as opposed to an MLP), yield unique advantages.\n\nQUESTIONS:\n- Can the authors comment on the tradeoff between latent dimension $D$  and approximate GP prior truncation length $L$ in terms of expressivity and risk of overfitting?\n- One interpretation of the parameterization $\\textbf{b}(\\textbf{u}_t)$ is that the external inputs undergo a nonlinear featurization before being linearly read into the state updates (setting aside the dynamics matrix also depending on $\\textbf{u}_t$ for now). How well would other models do with this nonlinear input featurization step? I would imagine that even a standard LDS model could improve greatly on nonlinear tasks with this modification.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework named the CLDS: Conditionally Linear Dynamical Systems, a novel probabilistic modeling framework for the neural population activity along with the task-related or the behavioral-related covariates. CLDS first have the interpretability and tractability of classical linear dynamical systems. And also, it has the flexibility of modern non-linear powerful models by allowing system parameters (e.g., transition and emission matrices) to vary as a function of observed covariates via the pre-defined Gaussian Process (GP) priors. Like most works in this probabilistic modeling genre, the CLDS derives an efficient MAP inference method using approximate GP basis expansions and EM optimization. As for the experiment part, this work has done extensive results on both synthetic and real neural datasets of mouse and macaque. The results manifest that CLDS achieves superior performance aganist the baseline methods, especially in data-limited situations.\n\nSTRENGTHS AND WEAKNESSES:\nI think that the strengths of this work lie in its new probabilistic modeling framework with certain extent of locally interpretation. The idea of adding some conditional covairates to the traditional LDS framework is interesting, which lacks some expressive power. With some new derivations, the work also presents the framework with closed-form EM updates. As for the empirical experimental results, the model demonstrates good performance on both synthetic and multiple real-world neural datasets. On the other hand, I think the concerns of this method lies in what's the background neuroscience intuition behind this proposed elegant framework, what part of the method is brain-inspired. How could you prove the soundness of CLDS besides the experimental results.\n\nQUESTIONS:\n1. What's the most difference and novelty of your probabilistic modeling framework compared to existing works? If there is a table that compare between these methods, that would be apparent to the audience. Also, there is a recent paper [1] whose method that can be compared with the proposed CLDS.\n2. How does u_t enables non-linearity and how much could it introduce?\n3. Could this framework extend to non-Gaussian likelihoods?\n\n[1] Multi-Region Markovian Gaussian Process: An Efficient Method to Discover Directional Communications Across Multiple Brain Regions. Li, et, al. ICML 2024.\n\nLIMITATIONS:\nPlease refer to my 'Questions' section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel approach that they term “Conditionally Linear Dynamical system” (CLDS) to model the neural population dynamics underlying activity of large groups of simultaneously recorded neurons. Their method complements a growing set of related methods to fit and interpret neural population dynamics. With CLDS, the authors aim to strike a balance between explanatory power and interpretability. Neural activity is modeled with a linear dynamical system (good interpretability) whose parameters are conditioned on task and/or behavioral variables that are defined by the experimenter, which makes the resulting dynamics overall non-linear (good explanatory power). The authors show analytically how CLDS relates to and differs from several previously proposed methods to fit neural population dynamics. They apply their method to three example datasets. One example consists of simulated responses based on a model consistent with CLDS (a test of model/parameter recovery), the remaining two examples are fits to open experimental datasets from mice and monkeys.\n\nSTRENGTHS AND WEAKNESSES:\nThe presented methods (including analytical derivations of expectation-maximization steps to fit the model in several settings) are described in detail and could be an interesting, original addition to current methods. But while the proposed method seems promising, the results appear quite preliminary. The three example applications are limited in scope and are somewhat insufficient to fully assess the value of the method. As I explain in more detail below, several important controls are missing and in general the limits of the models and fitting procedures are not sufficiently explored and discussed.\n\nQUESTIONS:\n(1) For a given choice of conditional variables u_k, the definition of the models includes several meta parameters like the dimensionality of the latent space and the parameters of the GP kernel. The authors mention that some of these parameters are optimized with cross-validation, while others appear to be fixed ad-hoc (e.g. the parameter L). Fig. 6 only provides limited insights into this cross-validation procedure for a single dataset. The latent dimensionality D employed in the examples if quite low, 2D (?) in the mice dataset and 5D in the primate dataset. These values seem low compared to past estimates of the dimensionality of neural activity, which in similar primate datasets was often found to be in the range of 10-15 dimensions. One worry is that such a discrepancy may imply that CLDS is more data-hungry than the authors suggest, at least for the employed parameterization of the Gaussian Processes in those examples. More generally, the demonstration of a robust, systematic approach for determining all meta-parameters would seem to be required to make this method broadly usable in practice. \n\n(2) The linear models that are a component of the proposed methods are known to be ill-posed, in that “dynamics” and “input” components (A and b in Eq 1a) can be non-identifiable, i.e. different combinations of inputs and dynamics can explain the observed activity equally well. This issue of non-identifiability is separate to the one between the latent and observation stages discussed by the authors. What is the implication of this non-identifiability for the application and interpretation of CLDS fits? I would guess that the addition of the conditional dependency of the linear dynamics on u_k could create an additional “level” of non-identifiability, in that it is not clear what variables u_k should be included in a model, and how exactly they should be parameterized (see also my comments below). Does this second type of non-identifiability occur in the model? If so, do the authors have a strategy on how to deal with it?\n\n(3) Fig. 2 show the main validation of the method on simulated data. These results are encouraging, but the scope of this validation is quite limited. The authors consider one specific type of 2D latent dynamics that was produced with the CLDS model and ask if the model can retrieve the underlying parameters. However, it is not clear how well this example matches the experimental realities. For example, what happens to the fits when the observation noise is increased? It is currently not clear how if the simulated observation noise matches those required to explain neural data. Also, what happens if there is a mismatch between the structure of the ground truth model (inputs, u_k) and the structure of the fitted model? Can the model reliably retrieve other types of dynamics, including higher dimensional dynamics, rotational dynamics, or non-normal dynamics? \n\n(4) The authors say that they expect CLDS to provide a good estimate of the “composite” non-linear dynamical system when u_k and x_t “tightly” co-determine each other. But how would one decide if the later condition applies? In a setting that is more akin to a switching LDS, with switches determined by some latent variable that is not obviously linked to a specific location in state-space, CLDS would presumably fail to account for the resulting dynamics. Is there some diagnostic feature of the data or the model that could be used to identify such failures? \n\n(5) I was confused by the non-linear dynamics in Fig. 2b. The ground-truth model appears to be a CLDS model, so the dynamics in Fig 2b does not seem to “appear” anywhere in the definition of the ground truth model. In fact, it would be interesting to see how well CLSD can fit data that was generated from a ground-truth model that exactly matches that in Fig. 2b. \n\n(6) The fits of CLDS to the example experimental datasets are also promising, but again the model validation is quite limited. Fig. 3b shows that a few neurons’ dependency on theta is captured, and that for many neurons the mean activity is captured. But what else can be gained from that plot? Likewise, Fig. 3c shows that for a few neurons the tuning curve for theta is well captured. But what about all the other neurons? Dynamics as in Fig 4 are typically shown as condition-averaged trajectories (see for example the original LFADS paper). How well does the model capture those, e.g. compared to LFADS? Also, how exactly was the 3d-space in Fig. 4b defined?\n\n(7) One peculiar choice in the fits of Fig. 4 is the definition of the condition variable z_t, which switches from 0 to 1 100ms after reach onset. Notably, this variable has no immediate parallel in the task, but rather seems to have been chosen by the authors to match the time at which firing rates across the population consistently increase. I find this choice problematic, because it involves a degree of “hand-engineering” the solution found by CLDS, in that it assumes that if a change in dynamics is happening during a trial, it must happen at that (human-chosen) time. I would have found this example more convincing if the authors had chosen time within trial as one of the conditional variables and had then shown that CLDS recovers a change in dynamics (and potentially inputs) sometime after the go cue.\n\nLIMITATIONS:\nSee questions above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}