Modeling state-dependent communication between brain regions with switching nonlinear dynamical systems
Authors: Orren Karniol-Tambour
David M. Zoltowski
E. Mika Diamanti
Lucas Pinto
Carlos D Brody
David W. Tank
Jonathan W. Pillow
Venue: ICLR 2024 poster (2024)
Keywords: neuroscience
neural dynamics
dynamical systems
decision making
Primary area: applications to neuroscience & cognitive science
TL;DR: We propose a multiregion, switching nonlinear state space model to model neural dynamics and communication

Abstract: Understanding how multiple brain regions interact to produce behavior is a major challenge in systems neuroscience, with many regions causally implicated in common tasks such as sensory processing and decision making. A precise description of interactions between regions remains an open problem. Moreover, neural dynamics are nonlinear and non-stationary. Here, we propose MR-SDS, a multiregion, switching nonlinear state space model that decomposes global dynamics into local and cross-communication components in the latent space. MR-SDS includes directed interactions between brain regions, allowing for estimation of state-dependent communication signals, and accounts for sensory inputs effects. We show that our model accurately recovers latent trajectories, vector fields underlying switching nonlinear dynamics, and cross-region communication profiles in three simulations. We then apply our method to two large-scale, multi-region neural datasets involving mouse decision making. The first includes hundreds of neurons per region, recorded simultaneously at single-cell-resolution across 3 distant cortical regions. The second is a mesoscale widefield dataset of 8 adjacent cortical regions imaged across both hemispheres. On these multi-region datasets, our model outperforms existing piece-wise linear multi-region models and reveals multiple distinct dynamical states and a rich set of cross-region communication profiles.

Published as a conference paper at ICLR 2024
MODELING STATE-DEPENDENS COMMUNICATION BE- TWEEN BRAIN REGIONS WITH SWITCHING NONLINEAR
DYNAMICAL SYSTEMS
Orren Karniol-Tambont David M. Zoltowski, Mika Diamanti, Lucas Pinto!,
Carlos D. Brody, David W. Tank, Jonathan W. Pillow
Princeton Neuroscience Institute, 'Northwestern University forrenkt, dzol dowltowski addaaaam cbrody, dwtank, hpt../ooopoo/o/o/etettooooo lucas, lucas.pinto@nn wwwtoooorttweseeeeeeeooooo
mdiamanti,
ABSTRACT
Understanding how multiple brain regions interact to produce behavior is major challenge in systems neuroscience, with many regions causally implicated in common tasks such as sensory processing and rrcision-making Moreover, neural dynamics are nonlinear and non-stationary, exhibiting switches both within and across trials. Here propose multi-region switch- ing dynamical systems (MR-SDS), a switching nonlincar state space model that decomposes multi-region neural dynamics into local and cross-region components. MR-SDS includes directed interactions between brain regions, allowing for estimation of state-dependent communication signals and sen- sory inputs effects. We show that our model accurately recovers latent trajectories, vector fields underlying switching nonlinear dynamics, and cross-region communication profiles in three simulations. We then apply method to two large-scale, multi-region neural datasets involving mouse decision making. The first includes hundreds of neurons per region, recorded simultaneously at single-cell-resolution across 3 distal cortical regions. The second is a mesoscale widefield dataset of 8 adjacent cortical regions imaged across both hemispheres. On these multi-region datasets, MR-SDS out- performs existing models, including multi-region recurrent switching linear models, and reveals multiple distinct dynamical states and a rich set cross-region communication profiles.
INTRODUCTION
Advances in neural recording techniques and large-scale electrophysiology (Sofroniew et al., 2016 Song et al. 2017 Liu et al. 2021) have transformed systems neuroscience over the past decade, providing simultaneous measurements of neural activity across regions at high temporal and spatial resolutions. Experiments using these technologies have shown that noural computation is highly distributed (Stcinmetz et al. 2019 Cowley et al. 2020
Musall al. 2018 Allen ct al. 2019 Makino et al. 2017 Allen et al. 2017 Gilad et al.
2018). The need for complementary multiregion mods has led to a flurry of activity
in this area (Semedoo al.) 2014 Kohn al.] 2020 Semedo al. 2019 Glaser al.
2020 Perich & Rajan 2020 Keeley el al. 2020a by Gokcen el al. 2024j. In This work, we build upon previous contributions in multiregion models, switching nonlinear time-series models, and transformer based models for neural data. We develop a probabilistic model that accounts for multiregion activity through latent, nonlinear dynamical systems that evolve and communicate in directed fashion at each timestep. The approach models high dimensional multi-region observations as emissions from coupled, low dimensional dynamical systems with explicit local dynamics and communication. We use Transformer encoder with per-region embeddings for posterior inference. We additionally introduce a measure of the volume of communications between brain regions in the model, allowing to quantify the directional 'messages' communicated between regions at each timepoint. We show information about stimuli and internal cognitive variables be decoded from messages,

---

Published as a conference paper at ICLR 2024
providing description of information flow across brain regions. Finally, we apply model multiregion neural datasets.
to three simulations and two
BACKGROUND
2.1 NONLINEAR NEURAL DYNAMICS
A wide variety of nonlinear dynamics models for sequential neural data have been introduced
in recent years (Lndderman al. 2017 Pandarinath et al., 2018 Duncker al.) 2019 Kim
ct al.] (021). One popular approach is to model nonlincar dynamics using switching lincar
dynamical systems (Linderman et al. 2017] Nassar et al. 2019 Zoltowski et al. 2020p. While
switching linear models can accurately approximate nonlinear dynamics, they require multiple discrele stales to instantiale a single nonlinear vector field, which reduces interpretabiliy, Similarly, the expressivity of each linear regime depends the number of latent. dimensions, which can make visualization challenging. A separate approach is LFADS, which uses a sequential autoencoder with a generative recurrent neural network (RNN) Pandarinath et al. (2018). While this approach is powerful, the RNN typically must be high-dimensional to learn complex dynamics, which makes visualization of the full dynamics challenging. Additionally, LFADS instantiates a single set of dynamics, whereas our aim is to learn multiple nonlinear dynamical regimes. Kim al. (2021) introduced PLNDE, a powerful approach which uses neural-ODE to model nonlinear, low-dimensional dynamics. Our proposed method extends this approach in discrete time to allow for multiple regions and switching between multiple nonlinear dynamics flow fields. Finally, Ye & Pandarinath (2021) used a Transformer to fit highly expressive nonlinear model neural activity without explicit dynamics. Our approach similarly exploits the expressivity and trainability of Transformers, but constrains their use to an encoder, allowing us to efficiently perform inference.
2.2 SWITCHING NONLINEAR DYNAMICS
Two recent papers introduced switching nonlinear dynamics models for time-scries data. Dong el al. (2020) proposed a model with switching nonlinear dynamics that uses an RNN inference network to infer continuous latents, with collapsed variational inference approach for the discrete latent variables. AAsari al. (202T) extended this approach to explicitly model discrete state duration through. We borrow Dong al's approach to discrete state inference in the mean-field inference method proposed for MR-SDS.
2.3 MULTI-RECION MODELING AND COMMUNICATION
Various approaches to modeling multi-region neural data have been proposed (Semedo et al., 2014; Kohn et al. 2020 Semedoo et al. 2019 Glaser et al. 2020 Perich et al.) 2020; Perich
& Rajan 2020 Keeley et al. 2020a b Gokcen et al. 2024 Our model directly relates to
mp-rSLDS, the multiregion recurrent switching LDS model introduced in Glaser et al. (2020). In this setup, the observation process is constrained such that continuous latent states map uniquely to observations originating in a particular brain region. MR-SDS can be seen as a generalization of mp-rSTDS with nonlinear dynamics, communication, and emissions.
2.4 DYNAMICS OF MEKINGG
Many tasks studied by neuroscicntists involve decision-making. common paradigm is sensory evidence accumulation, in which an animal accumulates competing sensory cues towards a decision (Brunton et al. 2013 Mazurek et al. 2003). Previous work has focused on
modeling dynamics of neural activily on single decision making trials, Lypically in individual
brain regions (Latimer et al., 2015 Zoltowski el al.) 2020 DePasquale et. al. 2021 Luo
et al. 2023). Two emerging experimental results from the decision making literature add complexity to modeling brain-widd neural dynamics. The first is that the neural basis of decision making appears to be highly distributed. Recent work shows evidence and decision related information is present in many brain regions (Pinto et al.] 2020 2019), and precise inactivation studies provide evidence that multiple regions are causally involved in accumulation and decision (Pinto et al. 2020). However, the precise role of different regions remains an open problem, underscoring the need for novel multiregion analysis methods that can address questions about cross-region communication. The second challenge is that decision-making behavior appears to exhibit persistent states in which different decision
strategies employed ( (Ashwood et al. 2020 Stone et al. 2020). This indicates neural
2

---

Published as a conference paper at ICLR 2024
dynamics may be non-stationary within and across trials, motivating use of switching nonlinear dynamics. Finally, cue-locked responses in early visual and intermediate brain regions are modulated by recent sensory stimuli and evidence history (Koay et al. 2019). These findings motivate our use nonlinear interactions between brain regions, history dependence, and stimuli effects.
3 MULTIREGION, SWITCHING DYNAMICAL MODEL
3.1 SWITCHING NONLINEAR DYNAMICS AND COMMUNICATION WITH MULTIPLE REGIONS
Here describe the MR-SDS model in detail (Figure 1). The model is designed to capture the following key features. First, our goal is to learn low-dimensional, nonlinear dynamics and corresponding latent trajectories underlying neural data. We expect that neural dynamics are nonstationary, and that different nonlincar vector fields may better describe neural dynamics different. timepoints. We therefore incorporate discrete state that ena bles switching between different sets of nonlinear dynamiis.. Second, we are interested in modeling multiregion neural data, and thus our model is designed to separate the activity of each brain region into different continuous latent dimensions, such that within-region and across-region dynamics and communication are accessible for analysis. Finally, we include a nonlinear emission mapping between continuous latent and observed neural data, to reflect the nonlinear observation process of calcium imaging.
3.2 DISCRETE SWTTCHES BETWEEN DIFFERENT DYNAMICAL REGIMES
The MR-SDS model has both continuous and discrete latent variables, respectively denoted by x and The generative process
zt Xt-1, 2 ~ Cat(T:), Te = (/0=taax(((((()1111-11))1)
(1)
1, Ze. NN (x), Q*1)
(2)
Yt Xx ((g(x)),,
Above, У1.т and u1. are the observation and input sequences. The discrete state 2 switches probabilistically at each timestep history (similar to a Hidden Markov Model or HMM). It is also possible for the inputs ut to effect transitions directly, as they do in our first simulation (4.1). The discrete state 2 indexes the dynamical regime active at time t, with different dynamics f governing the contimuous latent variable x, in each discrete state. Hence, changes in the discrete state over time cause switches in the nonlinear dynamics. Notably, the model does not explicilly specify a condition or distribution on switching locations or times induced by the discrete transition dynamics priori; transitions are probabilistic and not known in advance. Thus, the model is free to learn transitions driven by a combination of continuous and discrete states, and possibly external inputs.
function of the continuous latent x, well as its own
3.3 MULTIREGION DYNAMICS AND EMISSIONS STRUCTURE
Importantly, we constrain the model to have multi-region structure (Figure 1). We consider decomposition of the global continuous state into K variables private to cach region x; { J=ik and define the overall global dynamics function f using additive components acting on each region's 's latents a.. Thus, the dynamics function for the k'th region is
a = ( Et((())) füu (ur) + ~ QE)
(4)
j
We similarly decompose the observations as Yt (vt b=IIK. Thus, the corresponding emissions function g for the k'th region is
J gu (x + 8 of ~ NO, RA)
Above, far fij. fit f and gb nonlinear functions parameterized by neural networks. The generative model accounts for each region's observations y² as nonlinear 'private' emissions from a corresponding continuous latent variable z The discrete state 2 remains a global switch that selects which of M local dynamics and communication functions are active at each timepoint. Thus, f indicates the nonlinear local dynamics in region k active
3

---

Published as a conference paper at ICLR 2024
at time t in discrete state 2;- In practice, single conditional dynamics subnetwork can be used for each fak: fkj. fku across discrete states, with the state index provided to the network at each layer. We note that emissions in the model constrained such that latents from one region only map to that region's observations. MR-SDS can thus be seen as generalization of the LDS based multiregion model, mp-rST.DS (Glaser et al. 2020 For spiking data, it's also possible to use the emissions mapping uf ~ (oo)(
3.4 ESTIMATING CROSS-RRGION COMMUNICATION, OR 'MESSAGES' AND INPUT EFFECTS
By explicitly inferring each region's local dynamics, communccttion and input processing functions, the model exposes estimates of communication streams between regions. We define messages and regional input effects in the model as follows:
kk my
F (o
( =t-1]
(6)
m
kj E 2 (v)
(ij(C(-1)
(7)
m, km Eg(salo) Sha (us)
(8)
where above the expectations are taken with respect the approximate posterior over the latent states, z and æ. Notably, these messages combine additively to the estimated overall 'flow' of the k'th region's trajectory al each time point, thus decomposing the relative contribution of each region and of inputs. Thus, the model allows us to produce estimates and quantify communication and input effects for further downstream analysis. Furthermore, we can perform decoding analyses on messages to track how information about stimuli or internal cognitive variables flow across regions over time.
generative model
messages
inference
impolt
mean field
structured
communuu
Tranlemeeennde
mmelly
cosernetions
u - ب! 2- xl
e
e
y/
++1
--1
1
Figure 1: Generative model, messages and inference network for MR-SDS.
3.5 AMORTIZED VI WITH MULTIREGION STRUCTURED TRANSFORMER NETWORK
We propose two variational inference (VI) schemes for MR-SDS: mcan-field and structured. Both rely on a Transformer encoder with multiregion embeddings. In mean-field, the encoder directly produces independent. means and covariances of the approximate posterior on x:
y = ( u1:T)
(9)
PT. DE Trans wwr.remeertttt... (1D he-k U1r))
(10)
go(xi:lyy:: (1r))
II N 1 Σ3
(11)
Above, d is food-foward neural network embedding region I's activity vi. along with observed stimulus inputs u1.7 We use an encoder-only, non-causal Transformer architecture, motivated by the need to approximate the smoothing distribution of the continuous latents. Following (Dong et al., 2020). we treat samples &1.r from the approximate posterior as assuud-observattons for the subgraph p(11r/&1: U1rr), which forms an HMM with as discrete states and & as emissions. We perform conditionally exact inference for the discrete states with the wackard- algorithm for HMMs, which differentiate through
4

---

Published as a conference paper at ICLR 2024
Dong et al. 2020 With o, 0 the parameters of the inference network and generative model, this results in a factorized variational approximation to the full posterior:
Rabiner 1989
(x11;, 21:2 MI.r, u1:1) 90 (x1: Sy:r, (1:r)po(2g) (21:1 Y1:r, X1:7, u1:r)
(12)
A vantage of this approach is that the encoder must implicitly learn dynamics on without using the generative model. In the structured approach, the Transformer encoder instead directly outputs approximate posterior 2, well smoothing potentials vh:
Pi.r. Hir Eig Transformert ((V b=i:K, u1.2)
(13)
In this case the factorized variational approximation is:
T
(x1.T, 2irly,er,
II
po (x, Xx- =i, the (olt, belyier wir)
(14)
t=1
The forward generative dynamics are conditioned on the discrete state & and smoothing potentials vé to form the posterior x, thuss directly reusing the generative model, as in structured VAFs (Archer et al., 2015 Johnson et al. 2016: Zhao & Linderman, 2023)
While both schemes lead to good inference, We find the structured approach performs better generative prediction on high-dimensional neural datasets. In both inference approaches, the
parameters of the generative model and inference network are learned jointly by maximizing the evidence lower bound (ELBO):
CELBO Ego (lnr))))))) log Po (po
(15)
go (oxx.lyy:T u1:)
To avoid posterior collapse (Bowman et al. 2015), in which training produces good inference and reconstruction but poor generative dynamics, utilize beta. upweighting and 'over- shooting' (Hafner et al. 2019). Briefly, in overshooting, we evaluate the likelihood of the latents and emissions by marginalizing over multi-step predictive distribution:
x
(16)
T
30(1:r/uur1 =
II
w po (f (rr))
(17)
Po Xx:r) = II wh (y x(.r)
(18)
T=0
Above, xi-r is the trajectory produced by applying dynamics fx to the inferred latents times. This modified objective, similar to (Lusch et al. 2018), forces the emission distribution to pass through multiple timesteps of the dynamics, thus coupling the generative dynamics and emissions networks during training. It also encourages alignment of inferred latents and generative dynamics over multiple timesteps. The resulting multi-step predictive or "overshooting" ELBO can be seen as a lower bound on the standard ELBO, which recovered by setting T = (Hafner al., 2019). wF, wo weights that trade off inference (7 0) and short vs. long term generative predictive accuracy. Finally, 3 is weighting factor, analagous the 3-VAE (Higgins et al. 2017), annealed over training. Setting 3 pushes the model to prioritize accurate dynamics over reconstruction.
4 EXPERIMENTS
4.1 MULTI-REGION SWITCHING LOTKA- VOLTERRA SIMULATION
We first demonstrate the MR-SDS modeling approach in multi-region simulation that switches between two sets of latent nonlinear Lotka-Volters (LV) dynamics. Conceptually, our intention is to emphasize how learning switching nonlinear dynamics vector fields can uncover interpretable changes in dynamics over time. LV dynamics are 2d model of interacting predator-pon prett populations (Lotka 1925 ; LV models with regime switching have been used to model nonstationary environments and studied in the control theory literature
(Li et al. 2009: Liu et al. 2013). These dynamics provide an example of non-stationary oscillatory coupling, potentially important mode of neural communication (Kohn et al. 2020 Kastner et al. 2020 Modi et al. 20023] Khodagholy et al. 2017 Tal et al.) 2020). We model each latent dimension separate region, with half of observations function of x1
5

---

Published as a conference paper at ICLR 2024
and half a function of Zg. The simulation switches between two sets of LV dynamics, with the discrete switch driven by an external input Ug representing a stimulus or upstream brain region: p(z Re, u++x) Cat(softmax( (R2, Ut+h)). The dynamics are:
x az (1) 32. (1) x
(19)
#/ = 52
(1) x/ a (2)
(20)
Above, each parameter {ar 3, 5, depends on the current discrete state 2e: each state corre- sponds to different set of dynamics. We used the values {0.67, 1.33, 1,3,, 1.,, 0.8).
Figure 2 shows dynamics for two states on both dimensions of the joint state space. de is an external input and has value I when switch is signaled and 0 otherwise, with switch probabilities controlled by the matrix R. (Figure D.. The model was simulated in discrete time using the Euler approximation. To ensure stability, step size of 0.001 and then downsampled trials to 150 time steps. For each trial, between 1-3 discrete state switching times were sampled uniformly between 10 and 140. The model
simulated 75,000 steps with
had linear observations, a multi-region embedding network, and was fit with M = 2 discrete states, with global set of nonlinear dynamics in each state 2t = (x(-(( + ct, with
3 = [x (1) st and 6 ~ (O, Q*+). Learned dynamics under the model matched the true dynamics, as shown by similarity the dynamics vector fields (Figure 33). Additionally, generated latent states on simulated test trials closely matched the known ground truth (Figure 2). Next, we computed communication profiles. We computed the within region dynamics as f(x,"), 0.00) and across region dynamics as f(x ot (=()))- f(z!") 0.0). The profiles are marginalized over the discrete state posteriors. The generated communccation profiles matched the true profiles (Figure 2C).
state
state 2
messages
discreto continuous latents
true N:
true
MR-50S
true
MR-SDS
MALDS mr-SLDS
true
301 Strweiep
mR-SLDS
DhAp
SAN MOO
timestep 125
dmn1
Figure 2: Multi-region Lotka- Volterra. (A) Example trial with comparison of true latents and latents generated by MR-SDS, mr-ISLDS mr-LDS Continuous latents were mapped via a linear transformation to account for non-identifability. (B) True dynamics fields across two discrete states vs. dynamics fields learned by MR-SDS, mr-rSLDS, (C). Overlaid true and generated messages between regions (corT. coef. 0.99). (D) Distribution of generated continuous latent 22 values per trial for MR-SDS, mr-rSL.S, mr-LDS.
4.2 MULTIREGION EVIDENCE ACCUMULATION AND RESET WITH DOUBLE WELL DYNAMICS
To further molivate the use of our model for neural data we simulated a multiregion system performing a decision decision-making task with switching nonlinear dynamics (Figure 3). Animals performing evidence accumulation in sequential trials appear to display robust and structured sequences of behavior, including long periods of lapsos, bias and strategy switches (Ashwood et al. 2020: Stone et al. 2020 Koay et al. 2019). Thus, our aim was to show how intrinsic dynamics of a hierachical, auultregion, switching nonlinear model can account for accumulation dynamics across consecutive trials in way that produces history effects and bias; and to show how these dynamics could be recovered and analyzed with our approach. The simulated system was composed of two regions, an 'accumulator' and 'controller', each with two distinct dynamical phases: accumulation and return. The accumulation dynamics of the accumulator were inspired by a classic model of perceptual decision making with
6

---

Published as a conference paper at ICLR 2024
bi-stable attractor dynamics (Wong & Wang 2006). During accumulation, the accumulator presented with competing left and right evidence stimuli, and follows nonlinear dynamics leading to one of two attractor wells. The controller receives imput from single dimension of the accumulator, providing information about the progression of the accumulation process, but not the winning side. This drives the controller to its own attractor well; when it reaches this attractor, the return phase on both regions is triggered. During the return phase, the controller provides feedback to the accumulator, which follows different of nonlinear dynamics, returning it (roughly) to its initial condition. This final location serves as the initial condition for the next trial. The variability resulting initial conditions leads to biased Irial sequences, similar to those seen in animals (Ashwood et al. 2020 Stone el al. 2020). We provide further details in the appendix. MR-SDS recovered the correct discrete and continuous states, as well as the dynamics gradient fields for both regions and states. As an additional check on the quality learned dynamics, we used the dynamics model to simulate new trajectories from the true initial condition and inputs, and found these were similar to true and inferred trajectories.
True
MR-SDS
mI-SLLDS
accumulation
retur
accumulation
return
eccumulation
retum
accumulator
accuracy
MS-505
controller
true
gen
gen
Figure 3: (A) True and generated latent trajectorics and gradient fields for each region and 2 states for MR-SDS and mr-rSLDS. (B) Distribution of r2 across trials
4.3 INTERACTING LIGII-IIEENSIONLL RNNs DRIVEN MULTIPLE INPUTS
potential concern with state-space models amongst neuroscientists is that there no physically instantiated low dimensional latent state in the brain neural data comes from a high dimensional dynamical system with dov-dimensional intrinsic dynamics. Thus, our intention in this simulation is to show that MR-SDS is able to recover important qualitative aspects of a multiregion system with high dimensional, nonlinear dynamics, despite model mismatch. Furthermore, than simply using PCA to visualize the true high-dimensional dynamics of the model. Following (Perich et al. 2020 Perich & Rajan 2020), we simulated multiregion system with 3 interacting, high-D D IN=1000) RNNs, with sparse connectivity structure (Figure 4). Two RNNs were driven by separate inputs: discrete step and smooth sequence; the third RNN was driven solely by the other RNNs and exhibited chaotic dynamics. We emphasize that this simulation, there mismatch between the true generative model and MR-SDS. We thus compared dynamics and latents recovered by MR-SDS to the true dynamics and trajectories projected onto their first 2 PCs. We found MR-SDS embeds richer representation of the high dimensional dynamics into 2d than PCA, as well recovers important features of the system dynamics and communication.
wish to show that approach provides better interpretability
4.4 APPLICATION: REGION, SINGLE-CELL RESOLUTION MESOSCOPE DATA
We applied our method calcium imaging data recorded mice performing a sensory evidence accumulation task (Figure 4 In the task, headfixed mouse runs a lincar track in virtual reality while columns ('towers") are presented on both sides of the track (Pinto) et al. 2019). The mouse must decide which side more towers were presented and turn correctly to get a reward following short memory delay section the track. We analyzed a single day of mesoscope calcium imaging data of 3 distal brain regions in a single hemisphere, consisting of 178 correct trials. The regions were: AM (a visual area), retrosplenial cortex (RSC), and M2 (a higher level motor planning area). Preliminary results showed distinct
7

---

Published as a conference paper at ICLR 2024
trial averaged communication profiles for left and right trials, as well strong stimuli effect for visual area A.M
model
K
L
D
regions
mesoscope
Widefield
mr-PCA
2/2
3/8
0.0045469
9.083e-05
mr-LDS
1
2/2
3/8
0.0043695
0.000133
mr-ar-LDS
1
2
3/2
3/8
0.0043951
0.0001337
mr-SLDS
2
2/2
3/8
0.0043723
0.0001506
mr-rSLDS
2
2/2
3/8
0.0043632
0.000157
mr-ar-rSLDS
2
2
3/2
3/8
0.0043988
0.0002
mr-rSLDS
2
5/-
3/-
0.0042142
mr-rSLDS
2
10/-
3/-
0.0040676
MR-SDS (biRNN)
2/2
3/8
0.0040761
7.444e-05
MR-SDS
2
2/2
3/8
0.0037734
7.491e-05
lds rsids
1
9/24
1/1
0.0040886
0.0001144
2
9/24
1/1
0.0040877
0.00011744
LFADS
200/200
1/1
0.004344633
9.240c-05
Table 1: Held-out trial and neuron co-smoothing test MSE, across models on mesoscope and widefield datasets. Top:
multiregion. Bottom: single region models. K: discrete states. L: time-lags. D: latent dimensions per region.
RSC
AM
messages
Accumulattin towers
(1 m)
Gues
Adl felco''e wiw
from tatents
decoding evidence from
dacoding
local incoming P*
N2
100
trendep
Figure 4: Mesoscope imaging data. (A) Towers task and multiregion imaging at single cell resolution. (B) Inferred mean latent trajectorics and gradient fields for each region and states; mean occupied state in blue. (C) Inferred message norms from sending (columns) to receiving (rows). Decoding evidence. Left: from latents. Right: relative information about evidence present in incoming messages compared to local dynamics. Incoming stimuli effect in red. Note big incoming spike in M2, explaining higher decodability from latents on left plot.
4.5 APPLICATION: REGION, MESOSCALE WIDEFIELD DATA ACROSS HEMISPHERES We also applied our method to wide-field calcium imaging data from the same experimental task. 16 regions (8 bilaterally) were imaged across the cortical surface (Figure We analyzed single day of recordings consisting of 63 correct trials. To analyze communication between regions, we examined estimated messages under the inferred model parameters and approximate posterior over contimuoss latents, To quantify the messages, we computed their Frobenius norm at each timepoint. Figure 55 shows a 'macro' picture of communication streams between all regions. Highlighted are streams with notable feedforward, feedback and stimuli components. We draw attention to a few important features. First, profiles showed strong feedforward drive from VI to mV2 and PPC. Second, PPC showed strong feedback drive back to V1 and mV2, well as feedforward drive to upstream regions RSC and SS. These results are potentially consistent with a hypothesized central role of PPC in evidence accumulation, suggesting PPC may act a hub between early visual areas and upstream regions (Pinto et al.) 2019). Notably, PPC also showed messages incoming with high information about evidence compared to local dynamics (Figure br). Further analysis
8

---

Published as a conference paper at ICLR 2024
may seek to probe the timing of single trial commmications, well as their relationship to evidence levels in the trial.
massages
V
slate
decoding evidence
local incoming meg
10 1 60
Figure 5: Mesoscale widefield imaging data. (A) Inferred mean latent trajectories and gradient fields for each region and 2 states; mean occupied state in blue. (B) Top: Imaging performed over 8 regions across both hemispheres. Bottom: decoding evidence from MR-SDS inferred latents. (C). Relative information about evidence present in incoming messages vs. local dynamics. (D) Macro view of communication streams. Inferred message norms from sending (columns)) to receiving (rows). We highlight upstream communication from mV2 and PPC to higher order areas, and M1 receiving broad inputs from all regions.
5 DISCUSSION
We propose a switching, nonlinear dynamics approach for modeling, multi-region neural data, comparing it with piece-wise linear models, and show that our model compares favorably with these in simulations, We emphasize that our simulations were designed to capture important aspects of multiregion communication, rather than performance gains and expressivity provided by nonlinear dynamics and emissions. We therefore also included comparisons on calcium imaging neural data, showing MR-SDS more accurately predicts held out neurons' activity, reflecting higher expressivity and performance benefits. A key benefit of the model's nonlinear dynamics and emissions the ability to represent rich nonlinear dynamics and account for nonlinear emissions process. Pieco-wise linear multiregion models, by contrast, require higher latent dimensions and additional discrete switches to reach similar expressivity; argue this is a disadvantage because it requires use of dimensionality reduction methods to visualize the resulting dynamics. Similarly, while RNN based models such as LFADS are extremely powerful, they generate dynamics in very high dimensional space, making dynamics challenging to visualize. While our approach uses Transformer, which is challenging to interpret, we constrain its use to our inference network, which does not need to be interpreted for scientific analysis, and use simple feed-forward networks to paramcterize generative dynamics in our model. An interpretational limitation of MR-SDS, as in any switching or clustering model, is the relative advantage of lumping vs splitting, given that switching nonlinear dynamics can also be described by a single set of nonlinear dynamics in transformed space. There remain important aspects of future work. In particular, our method does not account for unobserved regions or neurons, and learned interactions in MR-SDS are not causal. It may also be desirable to learn the number of regions, or find clusters of neurons across regions with coherent functional dynamics. Additionally, future work should explore alternative types of multiregion commuicattion, and demonstrate whether our approach succeeds or fails in each case.
6 ACKNOWLEDGEMENTS
We thank Brian DePasquale, Stephen Keeley, Josh Glaser and Matt Whiteway for helpful discussions.

---

Published as a conference paper at ICLR 2024
REFERENCES
William E Allen, Isaac V Kauvar, Michael Z Chen, Ethan B Richman, Samuel J Yang, Ken Chan, Viviana Gradinaru, Benjamin E Deverman, Liqun Luo, and Karl Deisscroth. Global representations of goal-directed behavior in distinct cell types of mouse ncocortex. Newron, 94(4):891 907, 2017.
William E Allen, Michael Z Chen, Nandini Pichamoorthy, Rebecca H Tien, Marius Pachitariu, Liqun Luo, and Karl Deisseroth. Thirst regulates motivated behavior through modulation of brainwide neural population dynamics. Science, 364(6437), 2019.
Abdul Fatir Ansari, Konstantinos Benidis, Richard Kurle, Ali Caner Turkmen, Harold Soh, Alexander Smola, Bernie Wang, and Tim Januschowski. Deep explicil duration switching models for time series. Advances in Neural Information Processing Systems, 34, 2021.
Evan Archer, Memming Park, Lars Buesing, John Cunningham, and Liam Paninski. Black box variational inference for state space models. arXiv preprint orXiw:1511.0330 2015.
Zoe Ashwood, Anne Urai, Valeria Aguillon, International Brain Laboratory, Anne Churchland, Alexandre Pouget, and Jonathan Pillow. Discrete latent states underlie sensory decision- making behavior in mice. In Cosyne Abstracts, 2020.
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. ar preprint arXic:1707.00 05776, 2017.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a contimuous space. arXiv preprint ar urXic:1511.003 2015.
Bingni W Brunton, Matthew M Botvinick, and Carlos D Brody. Rats and humans can optimally accumulate evidence for decision-making. Science, 340(6128):95 98, 2013.
Benjamin R Cowley, Adam C Snyder, Katerina Acar, Ryan C Williamson, M Yu Byron, and Matthew Smith. Slow drift of neural activity as a signature impulsivity in macaque visual and prefrontal cortex. Neuron, 108(3):551158)) 567, 2020.
Brian DePasquale, Carlos D Brody, and Jonathan W Pillow. Neural population dynamics underlying evidence accumulation in multiple rat brain regions. BioRwiv, pp. 2021 10, 2021.
Zhe Dong, Bryan Seybold, Kevin Murphy, and Hung Bui. Collapsed amortized variational inference for switching nonlinear dynamical systems, In International Conference on Machine Learning, pp. 2638-2647 PMLR, 2020.
Lea Duncker, Gergo Bohner, Julien Boussard, and Maneesh Sahani. Learning interpretable contimuous-time models of latent stochastic dynamical systems. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1726-1734 Long Beach, California, USA, 09-15 Jun 2019. PMTR. URL httpp http:::/pproceeeinggs mlr https//97/duncckr1113 html
Ariel Gilad, Yasir Gallero-Salas, Dominik Groos, and Fritjof Helmchen. Behavioral strategy determines frontal or posterior location of short-term memory in neocortex. Neuron, 99 (4):814 828, 2018.
Joshua Glaser, Matthew Whiteway, John P Cunningham, Liam Paninski, and Scott Lin- derman. Recurrent switching dynamical systems models for multiple interacting neural populations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14867-14878. Cur- ran Associates, Inc., 2020. URL https: //poceedd///oooeiigggg neurips wwwturps.ccppppeeee wwtpppppee//2/fiii...e. 2a1157773777444777eccceel33
Paper pdf
10

---

Published as a conference paper at ICLR 2024
Evren Gokcen, Anna Jasper, Alison Xu, Adam Kohn, Christian K Machens, and Byron M Yu. Uncovering motifs of concurrent signaling across multiple neuronal populations. Advances in Neural Information Processing Systems, 36, 2024.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555 2565. PMLR, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vac: Learning basic visual concepts with a constrained variational framework. JCLR (Poster), 3, 2017.
Matthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. Advunces in neural information processing syslemss, 29, 2016.
Sabine Kastner, Ian C Fiebelkorn, and Manoj K Eradath. Dynamic pulvino-cortical in- teractions in the primate attention network. Current opinion in newrobiology, 65:10-19, 2020.
Stephen L Keeley, Mikio C Aoi, Yiyi Yu, Spencer LaVere Smith, and Jonathan W Pillow. Identifying signal and noise structure in neural population activity with gaussian process factor models. BioRaiv, 2020a.
Stephen L Keeley, David M Zoltowski, Mikio C Aoi, and Jonathan W Pillow. Modeling statistical dependencies in multi-region spike train data. Current Opinion in Neurobiology, 2020b.
Mohammad Reza. Keshtkaran and Chethan Pandarinath. Enabling hyperparameter optimiza- tion in sequential autoencoders for spiking neural data. Advances in Neural Information Processing Systems, 32, 2019.
Dion Khodagholy, Jennifer N Gelinas, and György Buzsáki. Learning-enhanced coupling between ripple oscillations in association cortices and hippocampus. Science, 358(6361): 369 372, 2017.
Timothy D) Kim, Thomas Luo, Jonathan W Pillow, and Carlos Brody. Inferring latent dynamics underlying neural population activity via neural differential equations. In International Conference on Machine Learning, pp. 5551 5561. PMLR, 2021.
Sue Ann Koay, Stephan Y Thiberge, Carlos D Brody, and David W Tank. Neural correlates of cognition in primary visual versus neighboring posterior cortices during visual evidence- accumulation- based navigation. bioRric, pp. 568766, 2019.
Adam Kohn, Anna I Jasper, João D Semedo, Evren Gokcen, Christian K Machens, and M Yu Byron. Principles of corticocortical communication: proposed schemes and design considerations. Trends in Newrosciences, 2020.
Kenneth W Latimer, Jacob L Yates, Miriam LR Meister, Alexander C Huk, and Jonathan W Pillow. Single-trial spike trains in parietal cortex reveal discrete steps during decision- making. Science, 31846444)118441877 2015.
Xiaoyue Li, Daqing Jiang, and Xuerong Mao. Population dynamical behavior of lotka- volterra system under regime switching. Journal of Computational and Applied Mathematics, 232 (2):427-448, 2009.
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial Intelligence and Statistics, pp. 914-922. PMTR, 2017.
Hong Liu, Xiaoxia Li, and Qingshan Yang. The ergodic property and positive recurrence of a multi-group Sotka-vottera mutualistic system with regime switching. Systems & Control Letters, 62(10):805 8810 2013.
11

---

Published as a conference paper at ICLR 2024
Xin Liu, Chi Ren, Yichen Lu, Yixiu Liu, Jeong-Hoon Kim, Stefan Leutgeb, Takaki Komiyama, and Duygu Kuzum. Multimodal neural recordings with neuro-fitm uncover diverse patterns of portica-hhipoccnn
interactions. Technical report, Nature Publishing Group, 2021.
Alfred James Lotka. Elements of physical biology. Williams & Wilkins, 1925.
Thomas Zhihao Luo, Timothy Doyeon Kim, Diksha Gupta, Adrian G Bondy, Charles D Kopec, Verity A Elliot, Brian DePasquale, and Carlos D Brody. Transitions in dynamical regime and neural mode underlie perceptual decision-making, bioRaiv, pp. 2023 10, 2023.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature commumications, 9(1):4950, 2018.
Jakob H Macke, Lars Buesing, John P Cunningham, M Yu Byron, Krishna V Shenoy, and Maneesh Sahani. Empirical models of spiking in neural populations. In Advances in Neural Information Processing Systems, pp. 1350-1330 1358, 2011.
Hiroshi Makino, Chi Ren, Haixin Liu, An Na Kim, Neehar Kondapaneni, Xin Liu, Duygu Kuzumn, and Takaki Komiyama. Transformation of cortex-wide emergent properties during motor learning. Newron, 94(4):880 890, 2017.
Mark E Mazurek, Jamie D Roitman, Jochen Ditterich, and Michael N Shadlen. A role for neural intogrators in perceptual docision making. Cerebral Cortez, 13(11):1257-1226, 2003.
Brijesh Modi, Matteo Guardamagna, Federico Stella, Marilena Griguoli, Enrico Cherubini, and Francesco P Battaglia. State-dependent coupling of hippocampal oscillations. Elife, 12:c80263, 2023.
Simon Musall, Matthew Kaufman, Steven Gluf, and Anne K Churchland. Movement- related activity dominates cortex during sensory-guided decision making. BioRriv, pp. 308288, 2018.
Josue Nassar, Scott Linderman, Monica Bugallo, and II Memming Park. Tree-structured recurrent switching linear dynamical systems for multi-scale modeling. In International Conference on Learning Representations (ICLR), 2019. URL httph:tpp/:///t httpps://operreview nnt/ forum?id=HkzAAhhR9YY
Chethan Pandarinath, Daniel O'Sheea Jasmine Collins, Rafal Jozefowicz, Sergey Stavisky, Jonathan C Kao, Eric M mrautmann, Malthew T Kaufman, Stephen I Ryu, Leigh R Hochberg, et al. Inferring single-trial neural population dynamics using sequential auto- encoders. Nature Methods, pp. 1, 2018.
Felix Pei, Joel Ye, David Zoltowski, Anqi Wu, Raced H Chowdhury, Hansem Sohn, Joseph E O'Doherty, Krishna V Shenoy, Matthew T Kaufman, Mark Churchland, et al. Neural latents benchmark'21. Evaluating latent variable models of neural population activity. Xix preprint or N1N:22009004400 2021.
Matthew G Perich and Kanaka Rajan. Rethinking brain-wide interactions through multi- region 'network of networks' models. Current Opinion in Neurobiology, 65:146-151, 2020.
Matthew G Perich, Charlotte Arlt, Sofia Soares, Megan E Young, Clayton P Mosher, Juri Minxha, Eugene Carter, Ucli Rutishauser, Peter H Rudebeck, Christopher D Harvey, et al. Inferring brain-wide interactions using data-constrained recurrent neural network models. BioRriv, pp. 2020 12, 2020.
Lucas Pinto, Kanaka Rajan, Brian DePasquale, Stephan Y Thiberge, David W Tank, and Carlos D Brody. Task-dependent changes the large-scale dynamics and necessity of cortical regions. Neuron, 104(4):810 824, 2019.
Lucas Pinto, David W Tank, and Carlos Brody. Multiple timescales of sensory-evidence accumulation across the dorsal cortex. bioRnin, 2020.
Lawrence Rabiner. tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEFF, (7(2))257 286, 1989.
12

---

Published as a conference paper at ICLR 2024
Joao Semedo, Amin Zandvakili, Adam Kohn, Christian K Machens, and M Yu Byron. Extracting latent structure from multiple interacting neural populations. In Advances in neural information processing systems, pp. 2942 29500. Citeseer, 2014.
João D Semedo, Amin Zandvakili, Christian K Machens, M Yu Byron, and Adam Kohn. Cortical areas interact through communication subspace. Neuron, 102(1):249922) 259, 2019.
Nicholas James Sofroniew, Daniel Flickinger, Jonathan King, and Karel Svoboda. A large field of view two-photon mesoscope with subcellular resolution for in vivo imaging. Ekffe 5:e14472, 2016.
Alexander Song, Adam S Charles, Sue Ann Koay, Jeff L Gauthier, Stephan Y Thiberge, Jonathan W Pillow, and David W Tank. Volumetric two-photon imaging of neurons using stereoscopy (vtwins). Nature methods, 14(4):420 426, 2017.
Nicholas A Steinmetz, Peter Zatka-Haas, Mattco Carandini, and Kenneth Harris. Dis- tributed coding of choice, action and engagement across the mouse brain. Nature, 576 (7786):266 273, 2019.
Iris Stone, Scott Bolkan, Ilana Witten, and Jonathan Pillow. Latent-state models reveal a state-dependent contribution of the striatum to docision-making. In Cosyne Abstracts, 2020.
Idan Tal, Samuel Neymotin, Stephan Bickel, Peter Lakatos, and Charles E Schroeder. Oscil- latory bursting as mechanism for temporal coupling and information coding. Frontiers in Compulational Neuroscience, 14:82, 2020.
Kong-Fatt Wong and Xiao-Jing Wang. A recurrent network mechanism time integration in perceptual decisions. Journal of Newroscience, 26(4):1314-3328, 2006.
Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers. arXiv preprint Ntf::2108.012100 2021.
Yixiu Zhao and Scott Linderman. Revisiting structured variational autocncoders. In International Conference on Machine Learning, pp. 42046 42057. PMLR, 2023.
David Zoltowski, Jonathan Pillow, and Scott Linderman. A general recurrent state space framework for modeling neural dynamics during decision-making. International Confer- ence Machine Learning, pp. 11680 11691. PMLR, 2020.
A APPENDIX
A.1 INTERACTING HIGH-DIMENSIONAL RNNs DRIVEN BY MULTIPLE INPUTS ADDITIONAL DETAIL
Additional: The first RNN was externally driven by sequence input representing sequen- tially active neural population, and the second RNN was externally driven by a discrete step signal, representing a population generating fixed points. The third region was driven by both the first and second RNNs, and exhibited chaotic dynamics. The simulation in (Perich) al. 2020) was modified make the determinics RNN dynamics stochastic and additive across nonlinear components.
13

---

Published as a conference paper at ICLR 2024
region
region
12
PC1
PCI
x1
Figure 6: Interacting high-D RNNs driven by inputs.
Placeholder fig
A.2 HYPERPARAMETER SWEEPS.
model
K
L
D
regions
mesoscope
widefield
MR-SDS
1
2/2
3/8
0.00405374
8.335e-05
MR-SDS
2
2/2
3/8
0.00392798
8.267e-05
MR-SDS
3
2/2
3/8
0.00387180 80
MR-SDS
2
3/3
3/8
0.00390648
7.743e-05
MR-SDS
2
4/4
3/8
0.00387056
7.739e-05
MR-SDS
2
5/5
3/8
0.00389255
7.982c-05
MR-SDS
2
-/2
16
8.304c-05
Table Held-out trial and neuron 1-step generative co-smoothing test mean square ertor,
across hyperparameter sweeps of number of states and number of dimensions per region. Final row includes a 16 region result. for the widefield dataset. K: number of discrete states. D: dimensions of latent dynamics used per region.
M2
AM
RSC
k1-d2
k2-d2
k2-d3
AM
k2-d4
k2-d5
0
75
150
75 150
0 75 150
timestep
Figure 7: We compare the mean message norms across trials for each of the model hyper parameters on the mesoscope dataset.
All experiments and analysis were run on a 28 CPU, GPU (GeForce RTX 2080 Ti) server. A.3 MULLIREGION RNN SIMULATION, ADDITIONAL, DETAIL.
A.4 EVALUATING PERFORMANCE ON SIMULATION.
We note that inferred messages in MR-SDS are produced by combining inferred latents at each timestep with single generated step through the dynamics model. Thus, when
14

---

Published as a conference paper at ICLR 2024
Accuracy
mesescope
widefie
0.002
Ids models MR-SDS
P-
0.0040
Figure 8: We compare the square the co-smoothing target for LDS based models (green) and MR-SDS (red). We compute p-values based on Gaussian fits to the other scores, which are highly significant for both datasets
A
region
region
region
0.50
PC1
PC1
PC1
0.8
0.9
0.9250.950
x1
Figure 9: (A) Top: network activity projected onto first PCs. Bottom: inferred latents using MR-SDS. (B) x² across trials per region, on reconstruction of observations from first 2 PCs vs under model.
validating MR-SDS on simulations, we took special care to evaluate the accuracy of single- step latent generation under the generative model, rather than inference alone. In single-step latent generation, we first infer the latents ( under the inference model, and then evaluate (f(4)hr-1 where f is the generative dynamics.
A.5 EVALUATING PERFORMANCE ON NEURAL DATASETS
To evaluate the performance of neural data, we compared held out model predictions for a family of related multi-region latent variable models (Table 2 These include PCA, LDS, SLDS, and rSLDS. Importantly, these models can be seen as special cases of MR-SDS, instantiated with less expressive components: PCA has no dynamics and linear emissions; iLLDS has linear transitions, dynamics and emissions. We used the 'co-smoothing" metric (Macke et al. 2011 Pei et al. (021). which evaluates reconstruction loss on 25% held-out neurons in held-out trials. Cosmoothing requires infering latents at test time based on subset of neurons, and predicting activity of held out set from those latents. The latent representation thus has to be robust to input removal, a more challenging test than simple reconstruction. MR-SDS model better performance than comparable models on both datasets. We additionally compared to rSDLS models with higher latent dimensionality, observing that mr-ISLDS models required 10 latent dimensions per region (vs to achieve comparable
15

---

Published as a conference paper at ICLR 2024
performance on the mesoscope dataset. Finally, we found MR-SDS was comparable with rSLDS models fit to the entire dataset as a single region providing a baseline for performance without multiregion description.
A.6 COSMOOTHING TIME EVALUATION
A key question in evaluating unsupervised models of neural data is regarding the appropriate metric use assess model fits. A limitation of looking only at reconstruction error on held out data is that the model may simply learn to compress and decompress the observalions, effectively learning an identily mapping. This can be seen in the relative strength of this in MR-PCA section below) on test reconstruction alone. An alternative suggested by (Macke et al. 2011h and adopted in the recent Neural Latents Benchmark (Pei ct al.) 2021), is coossmotthng in which model is evaluated reconstruction loss of held-out neurons on held-out trials (see Figure A1). This tests the ability of the model to learn a latent representation of the data that is robust to removal of inputs, and is thus a more challenging test of a model's performance. A key result in the benchmarks is that models that perform reasonably well on test reconstruction may perform poorly on cosmoothing. The Neural Latents Benchmark uses a 25% neuron drop out rate, and we apply this rate, with one modification we evaluate all models against 25% drop out rate per region. This ensures that results are not dominated by regions with more neurons.
the PCA benchmark (more
A.7 COSMOOTHING MULTIREGION DROPOUT TRAINING
To improve model gencralization and allow it to accurately infer latent variables given only subset of neural responses, we trained the model on real neural data by dropping out a subset. of inputs to the inference network in each batch. In particular, we dropped out inputs from individual neurons over time. Notably, it was important that we drop out equivalent fractions of neurons from each brain region.
To make MR-SDS and the latent represention it learns robust to missing inputs inherent in the co-smoothing test, we modify the training procedure as presented in algorithm 1.
There, the monte carlo samples used to evaluate the expectation are taken with respect to the approximate posterior evaluated on the dropped inputs. We dropped out 25% neurons per region. We found that dropout of 50-80% of trials works well, depending Empirically, datasets with higher correlation between observations, such as the widefield dataset used in the experiments, require lower trial dropout rate. We note that this is similar to the 'coordinated dropout', or 'speckled' holdout strategy used by (Keshtkaran & Pandarinath 2011)), but extended here multiregion data, and we drop out all of the timepoints for random neurons on each trial.
the dataset.
We used a dropout rate P1 (0., for the mesoscope and widefield datasets respectively in the experiments presented in the paper.
16

---

Published as a conference paper at ICLR 2024
Algorithm Cosmoothing mulliregion drop out training
Inputs: batch size region sizes {ri,...; FR) trial dropout rate P1, neuron dropout rate
for batch = (Yi 111:m E Dtrain do
Create batch mask b 1+1 Sample [m| Pim trials from i for region r E 1: R do
Sample [n, = P2 neurons for each trial m
Set mask 8m
end for
==Y (ie input 'dropout')
Evaluate the elbo as: 102. Take training step
end for
co-smoothing test
co-smoothing dropout training
Input
eval
input
eval
Figure 10: Multiregion cosmoothing and multiregion cosmoothing dropout training. Left: When cosmoothing, we drop a percent of neurons from each region on each test. trial and present this inputs to the model; we then evaluate the reconstruction error on those neurons alone. Right: In multiregion cosmoothing dropout training, we drop a percent of neurons from each region some percent of training trials, and evaluate reconstruction error on all neurons.
dataset mesoscope
P1
Lest.
cosmooot
0.8
0.0040009
0.0040761
mesoscope
0
0.0040569
0.0045115
Table 3: Held-out test crror with and without multiregion dropout training.
A.8
TRAINING DETAILS
A.9 DOUBLE WELL SIMULATION DETAILS
The simulation switches between two sets of dynamics, accumulation dynamics, and return dynamics. The system switches from accumulating to return at fixed time following the stimulus presentation and memory periods, mimicking the visual cue marking the end of the maze and the beginning of the maze choice arms for the animal in the real experiment.
17

---

Published as a conference paper at ICLR 2024
(1)
33)3
4 a.ccmm
a
accum
(1)
14,0
(1)
+ U2 + oz
(1),0c
oi
(),a
~ N0, E
(21)
-1111
d Laccum decum
2)
1) (2) + (1- (,, + oi ace
o
(2),
*N(0, mcccum)
(22)
d itret aret
(1)
(1)
+ b1) + 2 (1)
1,, + 0 re)
o
())ret
NN(0, 2(d))
(23)
Ft,ret (A
1)
(2) + + 2r, () + oi r
o
r2)
NO 2(re)
(24)
The following values are used for the variables:
(1) accur
ret 0.1
(25)
6(1) [0, 0.5]
(26)
c1)) = [0.005, 0.08]
(27)
(1) accum
0.00057
(28)
(1) ret
0.035
0
(29)
0
A (2) 0.5I
(30)
( 0.11
(31)
(2)
(2)
ret = 0.055/
(32)
accam
(33)
A.10 BENCHMARKS
We provide details on the models used in the benchmarks presented across experiments.
A.10.1 MR-PCA
PCA is surprisingly powerful baseline on many unsupervised machine learning tasks, e.g. [Bojanowski al. (017). We include in the benchmarks a comparison to multi-region PCA (MR-PCA), by which we mean fitting PCA to each region's training data separately. Test data for each region is then reconstructed by first projecting it onto the top principal components for that region. We note that while MR-PCA is strong baseline for test, can only compress data lincarly, and this performs poorly on co-smoothing on both datasets, emphasizing the importance of this metric. Additionally, note that MR-PCA is a much stronger baseline on the widefield dataset, which consistent with the low intrinsic dimensionality of the data (see Figure A3 of cumulative variance explained for both datasets).
18

---

Published as a conference paper at ICLR 2024
mesoscope
widefield
0.4 10
10
15
25
30
PC
PC
Figure 11: Variance explained by PC dimension per region, for both real datasets. Intrinsic (linear) dimensionality of the mesoscope data is higher than widefield.
A.10.2
MULTIREGION LDS, SLDS, AND RSLDS MODELS
Briefly, Glaser et al. 2020 introduced multiregion rSLDS model of the following form:
Cat(TT)) T.
softmax Σ
Rh - + tr -
(34)
at41 = A k +
Σ jeb
Anjat + B + Et ex ~ N(0,00
(35)
v = s(Crat dd)
(36)
Where above A, B, R, Q, are matrices and r, d are bias vectors. We refer to the model above an MR-RSLDS. Similarly, refer to versions of this model with discrete state as an MR-LDS, and versions with no recurrence in the switching dynamics as MR-SLDS.
All LDS variant models were run in parallel on 28 CPUs using modified version of the SSM package [Linderman et al https: /ttghtb..tbtom///gmg glinderman/ssm). using variational Laplace
EM (vLEM) (Zoltowski et al. 2020
A.10.3 HIGHER-ORDER AUTO-REGRESSIVE OR AR(P) MR-RSLDS MODELS
We include in the benchmarks comparison LDS and rSLDS models with higher order auto- regressive continuous latent dynamics. These models (AR(p) MR-RSLDS) have the following modified latent dynamies:
41
Auci xt + Σ A2ei
EAps 1 Ba " + e ~ ~NO, Q
(37)
jzk
Above, 7 the order the AR process, or the total number of lags, with i indexing the т lagged dynamics, communication, and inputs matrices, A A and B Similarly, It-t and u4-4 represent the i'th lagged state and input at time t. The emissions and discrete latent transition dynamics remain in the single lag models.
Adding higher order autoregressive dynamics adds to the expressivity of this class of models. While dynamics remain piecewise lincar, they no longer lincar with respoct to the previous timepoint alonc. We note that these models performod better than single lag models (sce results in Tables 1,A1).
In order to fit these models, we extended vLEM (Zoltowski et al. 2020) to higher order lags. Briefly, this involved modifying the Hessian to account for higher order terms.
19

---

Published as a conference paper at ICLR 2024
A.10.4 SINGLE REGION MODELS
In the benchmarks, include a comparison with single region LDS and rSLDS models that have a latent dimension equal to the sum of the latent dimensions used across regions in the multiregion models. Fitting a single region model multiregion data results in meaningful loss of interpretability, because there are no per-region latents, and no estimates of communication. However, these models in gencral can achieve lower test and cosmooth errors, since they are free to use latent dimensions to explain any part of the data across regions. As such, they give a baseline for performance on these datasets. Table 1 shows that MR-SDS achieves lower cosmoothing test error than both single region LDS and rSLDS models on both datasets, thus providing an increase in both performance and interpretability, Table 2 shows that better calibrated model improves in test region LDS and rSLDS models on mesoscope data, but not widefield data. We think this is likely due to the higher intrinsic dimensionality of the mesoscope data (see earlier comments on dimensionality in the MR-PCA section). x
well relative to single
A.11 ARCHITECTURE AND TRAINING DETAILS
We provide additional details on the network architecture and hyperparameters used to fit: the model for each experiment:
K
R
d
fuk.kj 32x2
fku
Ok
Gemb
Obirnn
Oran
g
P1
Ir
steps
LV
2
L
3;3
64,32
1e-3
2.5e3
DW
32x2
16
3:3
32x2
1c-3
5c3
RNN
2
3
32x2
L
L
16
3:3
32x2
0
lc-3
5c3
Meso
32x2
4x2
128x2
32x2
32:32
256
0.8
1e-3
15e3
Wide
8
2
32x2
4,2
32x3
32,
12x16x2
32:32
64
128
0.5
2e-4
15e3
Table 4: Parameters used for each of the experiments presented in the paper. LV: Switching Lotka- Voltera. DW: Double well. Meso: mesoscope. Wide: widefield. learning rate. p2: trial dropout rate used for cosmoothing training. I: Identity mapping. L: lincar layer. d: dimensionality of latents for cach region. For Bi-RNN, semicolon (:) indicates a single layer with two concatenated RNNs of that size, one forward and one backward.
A.12 EXPLICIT DURATION
In the explicit duration case, the full generative model is:
Pm( = CaaCCadmii .... dmax)
(38)
Pm((c
Um = 1
(39)
Edmer Pm((d
Us (c--1)
if 4-1+1
Ca 2t, G-1
(40)
Use
(((-1)
ifq=1
Cat(TT)) if=
2 X1-1,
(41)
if
In words, each state m has a learned categorical distribution (m(c) over durations. Counts at time are then sampled from normalized version of this distribution, according to the discrete state m active at time and denoted by 24. Conditioning on the counts allows the state through the transition distribution
discrete state to persist without sampling и. The choice hyper parameters dmin dmaz confers an inductive bias on the augmented model, that impacts the persistence of inferred discrete states.
20

---

Published as a conference paper at ICLR 2024
p(state)
frame
Figure 12: Explicit duration results in longer state persistence.
Above Inferred discrete state for single mesoscope trial with no learned explicit duration variabl. Bottom: Same, with explicit duration. Note the inferred discrete state persists for longer.
21


Reviews:
[{"review_id": "22BCIBoIQv", "reviewer": "Reviewer_iayK", "summary": "This paper presents a novel approach, Multi-Region Switching Dynamical Systems (MR-SDS), for modeling neural interactions across multiple brain regions. The proposed method models low dimensional nonlinear neural dynamics. With transformer encoders inferring the states, the model provides a more precise representation compared to existing models such as rSLDS and RNNs. The model is validated by three simulations and is applied to two multi-region neural datasets.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors extend existing models to make them nonlinear, while keeping key aspects of modeling such as low dimensional dynamics in the neural space. Switching dynamics are able to describe neural activity in a variety of behaviors. The model is able to perform well on held-out and co-smoothing tests.", "weaknesses": "1. The authors state that RNNs have higher dimensionality while solving such problems, and thus, it is difficult to interpret the inferred dynamics. However, MR-SDS has a transformer included which may be even more difficult to interpret. Indeed, while the authors show a higher accuracy, they do not show the interpretability at all; it is not clear how interpretable this model is, more so since the communication between different brain regions cannot be accurately recovered (e.g., Fig. 2D).\n2. In Figure 2, what is rho? Additionally, the reconstruction of states and the reconstruction of states’ communication are actually not good, thus, it is hard to say the model reconstructs the latent states successfully.\n3. Could the authors detail the parameters, such as alpha, beta, and so on, in section 4.1.\n4. In section 4.3, it would be better to link the paragraph to the figure with the corresponding result, referring to which figure the authors are talking about. Additionally, I assume that the figure is Figure 6, but how do the authors find that MR-SDS embeds a richer representation than PCA? And why is a richer representation better here? Since the resulting PCs are orthogonal, it is unfair to compare them with the richness of the data representation. \n\nMoreover, clarity is not as high as it should be. The authors would need to proofread the submission carefully and generate the figures professionally. Here are some examples that may need to be revised:\nSection 4.1, last sentence: “MAKE SURE to say here that we also include in appendix A1 a result without the external input and show that we can still do inference but can’t generate” is not appropriate to show here.\nFigure 2B needs a legend.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IMaNWhzFI5", "reviewer": "Reviewer_zWh9", "summary": "This paper presents a novel state-space model for analyzing multi-region neural recordings. Briefly, the multi-region switching NLDS architecture assumes multiple, switching nonlinear latent dynamical systems per region, governed by a discrete state variable and continuous latent dynamics, whose transitions are all functions of the system input and previous states parameterized by neural nets. The emission model is likewise parameterized by an NN, while the latent state inference is performed by a Transformer that pools across regions (after local embeddings). Training is done via maximizing ELBO. The authors apply their method to 3 simulated systems, finding good performance compared to ground-truth latents and interpretable flow fields and inter-areal messages. Furthermore, the proposed method is benchmarked against a variety of previous single/multi-region switching/non-switching (non-)linear methods on two real datasets, with best performance on the co-smoothing metric. Lastly, they demonstrate how the method can be used to gain dataset-specific neuroscientific insights.", "questions": "- a major question is whether the existing results are robust to different choices of the number of switching states, latent dims, etc., especially in the simulation experiments but using an incorrect number, as well as how do results change when applied to experimental data. I leave up to the authors to decide how best to demonstrate this, but in general, for a practicing neuroscientist using such a method, I would expect somewhat robust/consistent results independent of hyperparam choices.\n- The inferred systems from the real data (Figures 3 & 4) don’t look that great, and I’m generally not sure what I’m suppose to get from them in general. First, in Figure 3B, the flow fields in state all look like linear 1D flows. Is this meaningful? Could this not have been combined with the first state? Or what happens if one uses a larger number of states, are all the ones after state 1 simple linear-ish flows? And how would one interpret this either way?\n- In general, the figures could use a bit more explanation, e.g., what are the main occupied states? Why is the same trajectory plotted in both states but different portions are highlighted in blue in 3B? How exactly should someone interpret 3C / 4D? Various plots don’t have labeled axes (e.g., 3D, 4B/C), I guess that’s time in ms?\n- one assumption the model makes is that communication across regions are in the form of the latent variable, while in the brain this happens concretely and literally via spikes, which is more akin to the observed signal in this model (though here it’s calcium signals). How is the original assumption valid given this?\n- in practice, how does one decide on the various hyperparameter values, such as the number of latent dimensions, switching states, etc.?\n- Some more discussion regarding whether there is an advantage to using multiple discrete nonlinear DS instead of one bigger / flexible one is warranted. In particular, is the switching timescale realistic considering neurobiology, and what would the switching states represent?\n- I’m not sure what to make of the comparison to PCA in the high-D RNN experiment: the inferred flow field look quite similar, and if anything the PCA one looks cleaner. Maybe some further expansion on what we should be looking for would be helpful? (Also what are the markers?)\n- some small typos and such: page 6 line 2, “Figure 2a” should be 2c; line directly above figure 2 is a runaway latex comment; page 8 section 4.6 “Figure 5 shows a macro…” should be Figure 3; Supplemental text refers to Figure AX while the numbers continue from the main figures; Figure 3D missing caption label, and what is the dark brown suppose to be? Fixing these (and the figure issues above) would lift the overall quality of the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is a technical tour-de-force, and represents a generalization of many existing algorithms (e.g., MR-SLDS). The 3 simulation experiments are quite nice, and demonstrate the variety of systems for which this method may be applicable, even in the case of a mismatched model (i.e., the multiregion RNN model). The benchmark experiments against existing methods is also appreciated. The paper is clearly and concisely written for the most part, especially the introductory pages and explanation of the method, though one gets the feeling that the authors ran out of time (or space) for the results half of the paper, and there are some “rough edges” throughout. Overall, it seems like a promising and well-developed method with somewhat natural assumptions for modeling multi-region interactions in the brain.", "weaknesses": "I have two main concerns, which I will outline in brief here, with specific comments/questions in the next section:\n\nFirst, the method is powerful by design, leveraging multiple nonlinear DS per region. However, the flexibility comes with a series of issues, most prominently the choice of hyperparameter values, such as the dimensionality of the system and, critically, the number of switching states. In the simulation experiments, if I understand correctly, the model was given the ground-truth number of states, but how would the inferred systems look like when starting with a different number (e.g., for the switching Lotka-Volterra system)? This is further an issue with the real neural data, where one does not have access to the ground-truth, and it’s arguable whether there is such a number.\n\nSecond, is that the applications to real data is somewhat unconvincing for me, both in terms of the assumptions one makes regarding model architecture and hyperparameter values (related to above) and what actual insight was gained. Given the technical contribution in the method, I don’t think the authors should necessarily be penalized for attempting to apply it on real data. Nevertheless, I feel that the main claims regarding neuroscientific insight was not sufficiently evidenced.\n\nTherefore, my recommendation is borderline reject, and I would be willing to update my score if the authors can conduct a few robustness experiments, as well as argue why some of the hyperparam decisions and model assumptions are not problematic in a neuroscience setting, and/or are acknowledged as limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vihjKVFmCG", "reviewer": "Reviewer_rCyL", "summary": "This paper presents a method for modeling interactions between brain regions in the context of a collection of nonlinear state space models. It aims to describe the neural dynamics both internal to a brain region and between regions. The authors provide plentiful comparisons to other similar methods, applied mostly to decision-making paradigms in neuroscience.", "questions": "The authors appear to have left in a note: 'MAKE SURE to say here that we also include in appendix A1 a result without the external input and show that we can still do inference but can’t generate'. (Why can't it generate?)\n\nCould the authors comment on 'mr-rSLDS models required 10 latent dimensions per region (vs 3) to achieve comparable performance on the mesoscope dataset' and why more latent dimensions per region is a downside? Is it just about the computational load? Or is it important to model the latent dimensions with e.g. 3 dimensions because the brain itself 'uses' fewer than e.g. 10 dimensions? There is some trade off between summarization (fewer dimensions) and models that are structured more similarly to the higher-dimensional neural dynamics (more biological comparisons?). \n\nWould this method be able to discern the number of interacting brain regions? Rather than take the regions are proscribed (3 regions, 16/8 regions in the experimental datasets).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The intuition underlying the method is well described, and the math is clearly laid out. Quantifying the messages between regions in a nice addition beyond existing methods. \n\nThe method does outperform other (very similar) methods in the datasets tested here.", "weaknesses": "Some clarifications of the use cases / benefits of the method would be helpful. For instance why is 'switching dynamics enables modeling the dynamics in low-dimensional space using more persistent discrete states' a benefit? A clearer description of the goal is needed.\nOverall the discussion was weak and read as a series of unjustified statements. \n\nSome of the figures (e.g. Figure 5 B) were very low resolution, unlike other areas of the same figure. The caption would also be improved by defining terms ('accum' is different than 'accumulator'?, 'msgs'). Figure 5 has no C label. \nFigures don't appear to be consistently formatted and the text is often too small to read. (except Figure 1, which is well done and easily understood even without a caption).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NftoYv9s7r", "reviewer": "Reviewer_yiV5", "summary": "This paper proposes a switching nonlinear state space model to capture the nonlinear pattern in multi-region neural dynamics. The authors use a deep neural network-based approach, where they amortize the cost of inference by learning an inference neural network (Transformer in this paper), which predicts the mean and variance of the latent dynamics. In the experiments, they compare the proposed method with PCA and some widely used state space models in the neuroscience community by evaluating the prediction performances. They also apply the proposed method to multi-region calcium imaging data to explore the communication streams between regions.", "questions": "* Could SLDS / rSLDS give similar latent dynamics and message communications in Lokta-Voterra simulated data, mesoscope, and widefield calcium imaging data? Although SLDS is a linear model, it could capture the nonlinearity in latent space.\n\n* Are there any insights as to why you choose Transformer instead of some simpler and faster deep neural networks like RNNs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Propose a switching nonlinear dynamics system by using the powerful expression of deep neural networks.", "weaknesses": "* The experiment part needs clarification. Section 4.2 simulates a multi-region decision-making task, but there is no comparison of MR-SDS with other methods such as LDS, SLDS, and rSLDS. Besides, the author mentions their method, MR-SDS, is able to learn the nonlinear dynamics in this complex case, but there needs to be a figure to visualize such latent dynamics. Similarly, section 4.3 simulates a multi-region system with three interacting and high-dimensional RNNs, but there needs to be a figure to support their claim.\n\n* The comparisons with LDS, SLDS, and rSLDS are not enough. This paper only compares their prediction/reconstruction performance on calcium imaging data. It would be better to show the proposed method, MR-SDS, could capture more meaningful latent dynamics than other commonly used models, which may be more important than prediction performances in terms of neuroscience research. For example, could you provide some cases where MR-SDS captures some nonlinearity in the neural recordings while rSLDS cannot?\n\n* No standard deviation (variance) in Table 1.\n\n*  Many typos and wrong figure index/reference. E.g., in section 4.1 and the second line of page 6, it should be \"Figure 2(c) shows ....\". In section 4.1, there is no reference to Figure 2(d), etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4aDfv7IHrO", "reviewer": "Reviewer_LTdS", "summary": "This paper proposed the multi-region switching nonlinear state space model (MR-SDS) for learning state switching and nonlinear evolution of continuous latent with multi-region interactions setup. The inference algorithm uses the forward-backward algorithm to learn the latent state, and a transformer encoder as the variational distribution to infer the continuous latent. Experiments show that the proposed MR-SDS is able to learn the model parameter and then infer the nonlinear latent and predict future data. Besides, the volume of messages passing between different regions can be understood by such a model.", "questions": "* Why ‘posterior collapse’ is enclosed single quotes, and also others.\n* What about other methods for section 4.1 and 4.2? Is that other methods in Table 1 not applicable to experiment 4.1 and 4.2? For, example, the generative model of the Lokta-Voterra model is nonlinear. Authors show that MR-SDS is able to capture the nonlinear evolution of the then latent process $x_t$. But for SLDS, for example, $x_{t+1} \\sim \\mathcal N(Ax_t,\\Sigma)$. It seems like $Ax_t$ is a linear process, but there is a Gaussian distribution, so it is actually a nonlinear model. No matter how we define \"linear/nonlinear\", why not test whether SLDS can learn the Lokta-Voterra latent?\n* Is Table 1 the held-out likelihood? Where is the variance? Are the likelihood of MR-SDS significantly better than others? Is it likelihood, or log-likelihood?\n* I cannot see the significant benefits of the proposed model than others, in terms of both prediction on future data and latent estimation.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The maths explaining the model is clear to me, with an intuitive schematic for the generative process.\n* The model is able to capture the nonlinear evolution of the latent process $x_t$.", "weaknesses": "* The -2th line of page 3, $f_z^{kk}$ should be $f_{kk}^z$\n* Then main weaknesses is the mismatches between texts and figures:\n    * We know Fig. 1 is a summary of the whole generative model, but it is informal to not mention it in the main text.\n    * Page 6, 2nd line should be Figure 2(c).\n    * Page 6, 3-5 lines and Figure 2. Where is the $R$ and $u$ in Figure 2.\n    * Figure 2(b), what is true, what is the dashed curve? There is no clear legend.\n    * Other mismatches in Figure 2, and etc.\n* No \".\" in the last sentence of section 4.1.\n* I think forward Euler is not appropriate for generating the data. Although I don't understand different curves in Figure 2(b), I can see the numerical error amplifies significantly because of forward Euler.\n* Why only one state switch is allowed in experiment 4.1 and has to be within 40-60? This generating process is too simple and not complicated enough to validate the model\n* Overall, too many errors in this paper. I cannot understand most of the experiments since wrong text-figure assignments, no legend, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)