Alternators For Sequence Modeling
Authors: Mohammad Reza Rezaei
Adji Bousso Dieng
Venue: Accepted by TMLR ()

Abstract: This paper introduces alternators, a novel family of non-Markovian dynamical models for sequences. An alternator features two neural networks: the observation trajectory network (OTN) and the feature trajectory network (FTN). The OTN and the FTN work in conjunction, alternating between outputting samples in the observation space and some feature space, respectively. The parameters of the OTN and the FTN are not time-dependent and are learned via a minimum cross-entropy criterion over the trajectories. Alternators are versatile. They can be used as dynamical latent-variable generative models or as sequence-to-sequence predictors. Alternators can uncover the latent dynamics underlying complex sequential data, accurately forecast and impute missing data, and sample new trajectories. We showcase the capabilities of alternators in three applications. We first used alternators to model the Lorenz equations, often used to describe chaotic behavior. We then applied alternators to Neuroscience to map brain activity to physical activity. Finally, we applied alternators to Climate Science, focusing on sea-surface temperature forecasting. In all our experiments, we found alternators are stable to train, fast to sample from, yield high-quality generated samples and latent variables, and often outperform strong baselines such as Mambas, neural ODEs, and diffusion models in the domains we studied.

Published in Transactions on Machine Learning Research (06/2025)
Alternators For Sequence Modeling
Mohammad R. Rezaci
mw.rezacii@maillulooo uloronto..r
University of Toronto
Vertoia
Adji Bousso Dieng
adji@princeton.com edu
Department of Computer Science, Princeton University
Vertaia
Reviewed on OpenReview: Atips: // openreview. net/ met/ forum? id- 070C1HQOV0
Abstract
This paper introduces alternators, a novel family of non-Markovian dynamical models for sequences. An alternator features two neural networks: the observation trajectory network (OTN) and the feature trajectory network (FTN). The OTN and the FTN work in con- junction, alternating between outputting samples in the observation space and some feature space, respectively. The parameters of the OTN and the FTN are not time-dependent and are learned via a minimun cross-enlropy criterion over the trajectories. Allernators are versatile. They can be used as dynamical to-sequence predictors. Alternators can uncover the latent dynamics underlying complex sequential data, accurately forecast and impute missing data, and sample new trajectories. We showcase the capabilities of alternators in three applications. We first used alternators to model the Lorenz equations, often used to describe chaotic behavior. We then applied alternators to Neuroscience, to map brain activity to physical activity. Finally, we applied alternators to Climate Science, locusing on sea-surface temperature forecasting. In all our experiments, we found alternators are stable to train, fast to sample from, yield high-quality generated samples and latent variables, and often outperform strong baselines such as Mam- bas, neural ODEs, and diffusion models in the domains we studied.
variaale generative models or as sequence-
1 Introduction
Time underpins many scientific processes and phenomena. These are often modeled using differential equa- tions (Schrödinger 1926 Lorenz 1963 McLean 2012). Developing these equations requires significant domain knowledge. Over the years, scientists have developed various familics of differential equations for modeling specific classes of problems. The interpretability of these equations makes them appealing. How- ever, differential equations are often intractable. Numerical solvers have been developed to find approximate solutions, often with significant computation overhead (Wanner & Hairer 1996 Hopkins & Furber 2015)) Several works have leveraged neural networks to speed up or replace numerical solvers. For example, neural operators have been developed approximately solve differential equations (Kovachki et al. 2023). Neural operators extend traditional neural networks to operate on functions instcad of fixed-size vectors. They can
approximate solutions complex functional relationships described partial differential equations. How-
ever, neural operators still require data from numerical solvers to train their neural networks. They may
face challenges in generalizing to unseen data and are sensitive to hyperparamcters (Li et al. 2021 Kontolati
et al. 2023
Beyond their intractability, differential equations as a framework may not be amenable to all time-dependent problems. For example, not clear how to model language, which inherently soquential, using differential equations. For such general problems that are inherently time-dependent, fully data-driven methods become appealing. These methods are faced with the complexities that time-dependent data often exhibit, including

---

Published in Transactions on Machine Learning Research (06/2025)
t= 1
t ==2
t= 3
X2
X3
Observation space
F
N
Feature space
Zo
21
Z-
Figure/ 1: Generative process of an alternator with a cycle of length = 3. An initial random feature 2o is generated from fixed distribution, e.g. standard Gaussian. The rest of the observations æ1.7 and features #1:T are generated by alternating between sampling from the OTN and the FTN, respectively.
high stochasticity, high dimensionality, and nontrivial temporal dependencies. Generative modeling is a data-driven framework that has been widely used model soquences. Several dynamical generative models
have been proposed over the years (Gregor et. al. 2014 Fraccaro et. al.
2016 Du et al.
2016 Dieng et al.
2016 2019
Kobyzev et al. 2020 Ho et al. 2020
Kobyzev et al. 2020 Rasul al.
2021
Yan et al 2021
Unlike differential equations, generative models can account for the stochasticity in observations and can be easy to generate data from. However, they are less interpretable than differential equations, may require significant training data, and often fail to produce predictions and samples that are faithful to the underlying dynamics.
Dutordoir et al. 2022 Li et al. 2022b Neklyudov et al. 2022 Lin et al.
2023 Li et al. 2024
This paper introduces olternators, a new framework for modeling time-dependent data. Alternators model dynamics using two neural networks called the observation trajectory network (OTN) and the feature tra- jectory network (FTN), that alternate between generating observations and features over time, respectively. These two neural networks are fit. by minimizing the cross entropy of two joint distributions defined over the observation and feature trajectories. This framework offers great ftexibility. Alternators can be used as gencrative models, in which case the fcatures correspond to interpretable low-dimensional latent variables that capture the hidden dynamics governing the observed sequences. Alternators can also be used to map an observed sequence to an associated observed sequence, for supervised learning. In this case, the features represent low-dimensional representations of the input sequences. These features are then used predict the output sequences. Alternators be used to efficiently impute missing data, forecast, sample new trajectories, and encode sequences. Figure illustrates the generative process of alternator over three time steps.
Section ssoowasees the capabilities of alternators in three different applications: the Lorenz attractor, neural decoding of brain activity, and sea-surface temperature forecasting. In all these applications, we found that alternators tend to outperform other sequence models, including dynamical VAEs, neural ODEs, diffusion models, and Mambas, on different sequence modeling lasks.
Alternators present significant opportunities and considerations across multiple domains. In scientific ap- plications, the framework's 's ability to model complex temporal dynamics with interpretable low-dimensional latent variables could accelerate discovery in fields ranging from climate science to neuroscience, where understanding underlying dynamical systems crucial. Our contributions are threefold: (1) introducing Alternators, novel framework using alternating OTN/FTN networks trained via cross-cntropy minimiza- tion, (2) demonstrating versatility across generative modeling, prediction, imputation, and forecasting with minimal modifications, and (3) showing consistent, superior performance across dynamical systems.
2

---

Published in Transactions on Machine Learning Research (06/2025)
2 Alternators
We are interested in modeling time-dependent data in a general and flexible way. We seek to be able to sample new plausible sequences fast, impute missing data, forecast the future, learn the dynamics underlying observed sequences, learn good low-dimensional representations of observed sequences, and accurately predict sequences. We now describe alternators, a framework for modeling sequences that offers the capabilities described above.
Generative Modeling. We assume the data are from an unknown sequence distribution, which we denote by p(ær:)) with T being a pro-specified sequence length. Here cach æ E RD= We approximate p(æ1:1) with a model with the following generative process:
Sample 20 ~ p(zo).
For t I:
(a) Sample xt po( 21-1).
(b) Sample pol 21,1,
Here Z0.T is a sequence of low-dimensional latent variables that govern the observation dynamiis. Each Ze E R., with D. < Dz. The distribution p(zo) is a prior over the initial latent variable ø0- It is fixed. The distributions (xr 1t-1) and pa(z; a4) relate the observations and the latent variables at each time step. They are parameterized by and o, which are unknown. The latent variable z;-1 acts a dynamic memory used to predict the next observation æ, at time and to update its state using the newly observed xt.
The generative process described above induces a valid joint distribution over the data trajectory æı.r and the latent trajectory z0:T,
T
(TI.T ZO.T) = p(zo) II pa |2: 12=11)po(z 21-1,
(1)
This joint yields valid marginals over the latent trajectory and data trajectory,
(x.0 / p(zo) II
po po | 21-1)p((21 21-1, 24))
dzpr
(2)
(0.0(ZuT) / p(zu) II po (x) =111)p |
x1)
dx1.T
(3)
These two marginals describe flexible models over the data and latent trajectories. Even though the model is amenable to any distribution, here we describe distributions for modeling continuous data. We define
p(zo)= = I)
(4)
po(a 21-1) = =N (V(1-033 . fo((z-1), Dao?)
(5)
(z 20.1, = N
(2 (Var- (x1) + v (1-a--033 21-1, D2o?)
(6)
where fo(.) and (.(+) are two neural networks, called the observation trajectory network (OTN) and the feature trajectory network (FTN), respectively. Here o and o² are hyperparameters such that o < < The sequence 01rr is also fixed and pre-specified. Each or is such that 0 < az S -1-03 o3.
Learning. Traditionally, latent-variable models such the one described above are learned using variational inference (Blei et al. 2017). Here we proceed differently and fit alternators by minimizing the cross-entropy between the joint distribution defining the model p0.((m1:7) Zo:T) and the joint distribution defined as the
3

---

Published in Transactions on Machine Learning Research (06/2025)
product of the marginal distribution over the latent trajectories (o.)(zo:)) and the data distribution p(æ1:1). That is, we learn the model parameters O and by minimizing the following objective:
Φ) -Ep(arr) po [log Pa (21.T, zorT]]
(7)
To gain more intuition on why minimizing ((0,0) is a good thing to do, let's expand it using Bayes' rule,
[(0, Φ) - po) [log pe (zo.r) + log pa ( [0:r)]
(8)
Hpe (Zo.T)) + Bpp ( Ilpe (X1.T) (0:r)]
(9)
Here H(po.o is the entropy of the marginal over the latent trajectory and the second term is the expected Kullback-Leibler (KL) divergence between the data distribution p(æ1:)) and pe.s(<1r1) (o:r), the conditional distribution of the data trajectory given the latent trajectory.
Eq. 9 is illuminating. Indeed, it says that. minimizing () with respect to 0 and e minimizes the entropy of the marginal over the latent trajectory, which maximizes the information gain the latent trajectories. This leads to good latent representations. On the other hand, minimizing [(0, Φ) also minimizes the expected KL between the data distribution and the conditional distribution of the observed sequence given the latent trajectory. This forces the OTN to learn parameter settings that generate plausible sequences and forces the FTN to generate latent trajectories that yield good data trajectories.
It may be tempting to view Eq. 9 as the evidence lower bound (ELBO) objective function optimized by variational autoencoder (VAE) Kingma et al., 2019). That would be incorrect for two reasons. First, interpreting Eq. gas an ELBO would require interpreting po.o (Fo.T) as approximate posterior distribution, or a variational distribution, which we can't do since (zoo(Zo:r) depends explicitly on model parameters. Second, Eq. is the sum of the entropy of (Zo:)) and the expected log-likelihood of the observed sequence, whereas an ELBO would have been the sum of the entropy of the variational distribution and the expected log-joint of the observed sequence and the latent trajectory.
To minimize Φ) we expand it further using the specific distributions we defined in Eq. 5 and 6
T
Doo2 D
6) = Ep(arr) (iliesrr)ppppooCo.. (Foor))
Σ
|| 1=.11
||,, - Mzz 12
(10)
Pr. = V(1 of)) fo(z1-1) and pz, = var go(x)) v(1 Or - - -2-1
(11)
Although Eq. 10 is intractable still depends on expectations we can approximate it using Monte Carlo,
1
B
T
20000
1
2 Doo D
2
() ~
x
(12)
B
b=1
where x ii:T..... 21. are data trajectories sampled from the data distribution D and 2.7: trajectories sampled from the marginal (=,0(= using ancestral sampling on Eq.
(1)
(1)
(B)
2or are latent
Algorithm 1 summarizes the procedure for dynamical generative modeling with alternators. At each time step t, the OTN tries to produce its best guess for the observation H; using the current memory 2;-1- The output from the OTN is then passed as input to the FTN to update the dynamic memory from 2,-1 to zt- This update is modulated by a,, which determines how much we rely on the memory 2t-1 compared to the new observation Zy. When dealing with data sequences for which we know certain time steps correspond to more noisy observations than others, we can use or rely more on the memory žt-1 than the noisy observation Σz. When the noise in the observed sequences is not known, which is often the case, set a fixed across time. The ability to change a across time steps provides alternators with an enhanced ability to handle noisy observations compared to other generative modeling approaches to sequence modeling.
Although the true data distribution p(Z1:T) unknown, we have some samples from which are the observed sequences, which we can use approximate the expectation.
4

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm 1: Dynamical Generative Modeling with Alternators
Inputs: Samples from p(æ1:r), batch size B, variances of and o. schedule a1.7 Initialize model parameters 8 and Φ
while not converged do for b= 1,..., B do
Draw initial latent 20 for t= , do
(b)
NN(,
Draw noise variables Ext ~0, ID.) and Ezt (O,
(b)
(b)
Draw xt (b) = /(1 - 2 fo(2)-11 (b)
(b)
o, Est
Draw (b) voi go(a) ( + v(1 ar o (b) (b)
(b)
Zt
22-1 02 Es1
end
end
Compute loss [(,, Φ) in Eq. 12 using 1:B and data samples from p(æ1:))
Backpropagate to get VoL(e, and Vol(0,
Updale parameters 0 and o using stochastic optimization, e.g. Adam.
end
To-Seuunneee Prediction. When given paired sequences æ1. and yıe, we can use alternators to predict f1.r given #1:7 and vice-versa. We simply replace p(æ1.1 (F,o)) with the product of the joint data distribution, p(11.T M1.T) and p(zo). The objective remains the cross entropy,
[( Φ) = -
[log po, (T1.1 VI.T, x0)]
(13)
This leads to the same tractable objective Eq.
12, replacing ø1.7 with Vi.T,
Dyo
0(
Me
(b)
c (0)
2
D20
(b)
p (0)
(14)
where x 21. ... 2 and V1.7
B V1r are sequence pairs sampled from the data distribution, Pr. V(1- d)) 10 and Pv. Va: (x(x)) + (1-aa as - of)) y:-1- Algorithm summarizes the pro- (B)
cedure for sequence-to-soquunec prediction with alternators.
Imputation and forecasting. Imputing missing values and forecasting future events are simple using alternators. We simply follow the generative process of an alternator, each time using æ, when it is observed or sampling it from pa( z when it is missing.
Encoding sequences. It is easy to get a low-dimensional sequential representation of a new sequence Ni.ri we simply plug of at each time step t in the mean of the distribution p 2;-1, v.) in Eq.
z Vai (x( v a, - -²) 2
(15)
The sequence zi.r is the low-dimensional representation of vं.r given by the alternator. To uncover the dynamics underlying collection of B sequences - ................... (BB) instead, we can simply use Eq. 15 for each sequence .1.7 and take the mean for each time step. The resulting sequence is compact representation of the dynamics governing the imput sequences.
(B)
3 Related Work
Alternators are a new family of models for time-dependent data. As such, they are related to many existing dynamical models.
Autoregressive models (ARs) define probability distribution for the next element in a sequence based on the previous elements, making them effective for modeling high-dimensional structured data (Gregor et. al.
5

---

Published in Transactions on Machine Learning Research (06/2025)
2014). They have been widely used in applications such speech recognition Chung et al. (0119) language
modeling Black et al. (2022), and image generative modeling Chen et al. (201 8b) However, ARs don't 't have latent variables, which may limit their applicability.
Temporal point processes (TPPs) were introduced to model event data (Du et al. 2016 TPPs model both
event timings and associated markers by defining an intensity function that is nonlinear function of history using recurrent neural networks (RNNSs). However, TPPs lack latent variables and are only amenable to discrete data, which limits their applicability.
Dynamical VA such as VRNNS (Chung et al. 2015 and SRNNs (Fraccaro et al. 2016) model sequences by parameterizing VA with RNNs and bidirectional RNNs, respectively. This enables these methods to learn good representations of time-dependent data by maximizing the ELBO. However, they fail to generalize and struggle with generating good observations due to their parameterizations of the sampling process.
Differential equations are the traditional way dynamics are modeled in the sciences. However, they may be slow to resolve. Recently, neural operators have been developed to extend traditional neural networks to operate on functions instead of fixed-size vectors (Kovachki et al.] 2023). They can approximate solutions to complex functional relationships modeled as partial diffcrential equations. However, neural operators rely on numerical solvers to train their neural networks. They may struggle to generalize to unseen data and are sensitive to hyperparameters (Li et al. 2021 Kontolati et al. 2023
Neural ordinary differential equations (NODEs) model time-dependent data using a neural network to predict an initial latent state which is then used to initialize a numerical solver that produces trajectories (Chen et al. 2018ab. NODEs enables continuous-time modeling of complex temporal patterns. They provide flexible framework than traditional ODE solvers for modeling time series data. However, NODEs are still computationally costly and can be challenging to train since they require careful tuning of hyperparameters and still rely on numerical solvers to ensure stability and convergence (Finlay et al., 2020). Furthermore, NODEs are deterministic; stochasticity in NODEs is only modeled in the initial state. This makes NODEs not ideal for modeling noisy observations. Alternators also differ fundamentally from neural stochastic differential equations (SDEs) introduced by Liu et al.) (2019) in several key ways. While neural SDEs incorporate stochastic terms to model randomness, but rely computationally expensive numerical solvers and maintain high-dimensional state representations, Alternators use direct neural network mappings and explicitly model low-dimensional latent variables (D2 < Da), making them significantly faster as demonstrated in our sea- surface temperature forecasting experiments while maintaining stochasticity through noise models for both latent and observation spaces.
Probability flows are generative models that utilize invertible transformations to convert simple base dis- tributions into complex, multimodal distributions (Kobyzev et al. (020)). They employ contimuus--time stochastic processes to model dynamics. These models explicitly represent probability distributions using normalizing flow (Papamakarios et al. 2021). While normalizing flows offer advantages such tractable computation of log-likclhhooss they have high-dimensional latent variables and require invertibility, which hinders flexibility.
Recently, diffusion models have been used to model sequences (Lin et al. 2023 For example DDPMs can be used to denoisc a sequence of noise-perturbed data by itcratively removing the noise from the sequence (Rasul et al., 2021 Yan et al. 2021 Bilos et al., 2022 Lim et al. 2023). This iterative refinement enables DDPMs to generate high-quality samples. TimeGrad is diffusion-based approach that introduces noise at each time step and gradually denoises it through a backward transition kernel conditioned on historical time series (Rasul et al. 2021). ScoreGrad follows a similar strategy but extends the diffusion process to a continuous domain, replacing discrete steps with interval-based integration (Yan et al. 2021 Neural diffusion processes (NDPs) are another type of diffusion process that extend diffusion models to Gaussian processes, describing distributions over functions with observable inputs and outputs (Dutordoir et al. 2022). Discrete stochastic diffusion processes (DSDPs) view multivariate time series data as values from a continuous underlying function (Biloš et al. 2022 Unlike traditional diffusion models, which operate vector observations at each time point, DSDPs inject and remove noise using continuous function. D3VAE is yet another ceatusion-based model for sequences (Li et al. 2022a).. It starts by employing a coupled diffusion process for data augmentation, which aids in creating additional data points and reducing noise, The model
6

---

Published in Transactions on Machine Learning Research (06/2025)
then utilizes BVAE alongside denoising score matching to further enhance the quality of the generated samples. Finally, TSGM is a diffusion-basel approach to sequence modeling that uses three neural networks to generate sequences (Lim et al., 2023). An encoder is trained to map the underlying time series data into a latent space. Subsequently, a conditional score-matching network samples the hidden states, which a decoder then maps back to the sequence. This methodology enables TSGM to generate good sequences. All these diffusion-based methods lack low-dimensional dynamical latent variables and are slow to sample from as they often rely on Langevin dynamics.
Action Matching (AM) is method that learns the continuous dynamics of system from snapshots of its temporal marginals, using cross-scctional samples that are not correlated over time (Ncklyudov et al., 2022). AM allows sampling from a system's time evolution without relying on explicit assumptions about the underlying dynamics or requiring complex computations such as backpropagation through differential equations. However, AM does not have low-dimensional dynamical latent variables, which can limit. its applicability.
The current widely used class of dynamical models for modeling sequences state-space models, particu- larly Mambas (Gu & Dao 2023). A Mamba uses a latent dynamical model to capture temporal dependencies and an observation process driven by the latent variables to generate data. Mambas are able to model com- plex and diverse sequences iveey However, Mambas have limitations. First, the latent variables in Mambas have the same dimensionality as the data, just as for flows, which leads to big models and increases computational complexity. Second, this same high-diinensoonaliy of the latents reduces interpretability, making extracting meaningful insights about the underlying dynamics challenging.
In contrast to the approaches above, Alternators explicitly alternate between generating obsorvations and latent features over time using two neural networks, the OTN and the FTN, jointly optimized to minimize cross-entropy over the observation and feature trajectories. Alternators have low-dimensional latent variables, which enhances their interpretability and makes them more robust to noise in data. Unlike Mambas, which prioritize expressivity using high-dimenniooal latent variables, Alternators balance computational efficiency, interpretability, and flexibility, excelling scenarios where understanding a sequence's low-dimensional dynamics is critical.
4 Experiments
We now showcase the capabilities of alternators in three different domains. We first studied the Lorenz attractor, which exhibits complex chaotic dynamics. We found alternators are better at capturing these dynamics than baselines such as VRNN, SRNN, NODE, and Mamba. We also used alternators for neural decoding on three datasets map brain activity to movements. We found that alternators tend to outperform VRNN, SRNN, NODE, and Mamba. Finally, we show that. alternators can produce reasonably accurate sea- surface temperature forecasts while only taking a fraction of the time required by diffusion models and Mambas. For comprehensive details regarding implementation specifics and hyperparameter configurations across each experiment, we refer the reader to the Appendix
B [code available at: https://githu: comtt..cooom
pertaix/Alternattnn
4.1
Model System: The Lorenz Attractor
The Lorenz attractor is a chaotic system with nonlinear dynamics described by set of differential equations [orenz 1963). We use the attractor to simulate features 21,7, with 2; E R for all {1...., T) and T 400. We simulate from the Lorenz equations by adding noise variables 62, <a to the coordinates,
(t) = a. (zz(t) 21) +
6 N(011
(t)) 21 2t) (p- 25(1)) - 22(() c2.
62 N((1))
=a()) = 2) (t) (2) B =3(() 63,
63 ~~~N N(0,1).
The parameters o, p. 3 control the dynamics. Here we set o dynamics which we hope to capture well with alternators. Given the features 21:T, we simulated æur;
10, 28. 8/3 to define complex
7

---

Published in Transactions on Machine Learning Research (06/2025)
Ground Truth
Alternator
VRNN
10
10.
0.8
0.8
0.6 Z
0.6
0.6 Z
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
1.00
1.00
1.00
0.75
0.75
0.75
0.00 0.25
5.50
0.50
Y
0.00
0. 25
0.50
0.25
0.50
0.00
0.25
0.50
0.25 Y
0.50
X
0.75
1.000.00
X
0.75
1.000.00
X
0.75
1.000.00
SRNN
Mamba
NODE
11.0
1.0
1.0
0.8
0.8
0.8
0.6 Z
0.6 Z
0 Z
0.4
10.4
0.4
0.2
0.2
0.2
0.0
0.0
0
1.00
1.00
1.00
0.75
0.00 0.25
0.25 X 0.75
0.50
0.25
0.50
Y
0.00
0.25
2.500
0.50
Y
0.00
0.50
0.25
0.50
1.000.00
X
0.75
1.000.00
X
0.75
1.000.00
Figure 2: Alternators better tracking the chaotic dynamics defined by Lorenz attractor, especially during transitions between attraction points, than baselines such VRNN, SRNN, NODE, and Mamba.
Table 1: Alternators outperform several dynamical models in predicting the dynamics defined by the Lorenz equations terms of MAE, MSE, and CC.
Method
MAET
MSE 4
CC
SRNN
0.052 0.017
0.148 +:2 0.007
0.955 + 0.001
VRNN
0.074 0.003
0.173 + 0.002
0.963 = 0.001
NODE
0.044 0.013
0.220 + 0.012
0.888 + 0.012
Mamba
0.045 = 0.003
0.135
+ 0.001
0.958 + 0.001
Alternator
0.030 0.005
0.076 + 0.003
0.977 + 0.001
with each xt E {0, 13 resolution small enough to ensure 3, E {0, 1} for We use the Poisson process to mimic spiking activity data. Fmpiricall studies have shown that spike counts within fixed intervals often align well with the Poisson distribution, making it a practical and widely used model neuroscience (Rezaei al. 2021 Truccolo al. 2005). The intensity the point process nonlinear function of the features, \2 1) )₂(2) (I), where we define
1]100, 100
by sampling from a time-dependent Poisson point process. We selected the time
(2t P
(t - sn 20
(z exp
and (t)
Σ S,ES;
20
2
for j E (1,..., 1003. Here Hj.z and of.2, are the center and width of the receptive field model of æt, aj is the maximum firing rate, and Sy is the collection of all the spike times of the jth channel. They are drawn from
Mja ~ U(((z)) - 2 ((2x), + 2 (x())
(16)
jj., ~ (m 1/100), Umminn 1/100), and aj ~ (Srmmnn (rmaz).
(17)
We frmin 0, frmaz 10, and Unin 0.001. We then used the paired data (#1.7- #1:7) in sequence- to-sequence prediction task to train alternator well NODE, an SRNN, VRNN, and Mamba.

---

Published in Transactions on Machine Learning Research (06/2025)
Mean Absolute Error
4
Mean Squared Error 4
Correlation Coefficient
0.14
0.8
0.4
Motor Cortex
0.10
0.3
0.08
0.06
0.2
0.04
0.1
0.02
0.00
0.4
Tomatosensory
0.3
0.100
0.075
0.2
0.4
0.050
0.1
0.2
0.025
0.000
0.0
0.0
0.20
0.6
Hippocampus
0.15
0.5
0.5
0.4
0.4
0.10
0.3
0.3
0.2
0.05
0.1
0.1
0.00
0.0
0,0
Alternator
Mamba
NODE
VRNN
SRNN
Alterator
Mamba
NODE
VRNN
SANN
Alternator
Mamba
NODE
VRNN
SRNN
Figure 3: Alternators tend to outperform VRNN, SRNN, NODE, and Mamba on trajectory prediction in the neural decoding task on three different datasets in terms of MAE, MSE, and CC.
We didn't include diffusion model a baseline here since it lacks a dynamical latent process that can be inferred from the spiking activities for sqquenco-to-sequence prediction.
We evaluate each model by simulating 100 new paired sequences following the same simulation procedure. We used the new observations to predict the associated simulated features. We assess feature trajectory prediction performance using three metrics that compare predictions from each model with the ground truth features: Mean Absolute Error (MAF), Mean Squared Error (MSE), and Correlation Coefficient (CC). We used 2-layer attention models, each followed by hidden layer containing 10 units for both the OTN and the FTN. We set a2 0.1, 0a 0.3, and a. 0.3 is fixed for all The models were trained for 500 epochs using the Adam optimizer with initial learning rate of 0.01. We applied cosine annealing learning rate scheduler with a minimum learning rate of 1c-4 and 10 warm-up epochs.
Figure 2 shows the simulated features, along with fits from an alternator, an SRNN, a VRNN, a NODE, and a Mamba. The alternator is better at predicting the true latent trajectory compared to the baselines. Specifically, alternators accurately capture the chaotic dynamics characterized by the Lorenz attractor, especially during transitions between attraction points. The results presented in Table I quantiff this, with alternators achieving better MAF, MSE, and CC than the baselines.

---

Published in Transactions on Machine Learning Research (06/2025)
Mean Absolute Error
Mean Squared Error
Correlation Coefficient
0.7
0.35
Motor Cortex
0.30
0.25
0.5
0.20
0.4
0.15
0.3
0.10
0.2
0.05
0.1
0.00
0.35
0.7
0.8
Somatosensory
0.30
0.6
0.25
0.5
0.20
0.4
0.4
0.15
0.3
0.2
0.05
0.00
0.0
0.4
0.6
Hippocampos
0.3
0.6
0.2
0.4
0.3
0.1
0.2
0.0
0.0
0.0
Atternator
Mamba
NODE
VRNN
SRNN
Alternatrr
Mamba
NODE
VRNN
SRNN
Alternator
Mamba
NODE
VRNN
SANN
Figure 4: Alternators tend to outperform VRNN, SRNN, NODE, and Mamba on missing value imputation in the neural decoding lask on three datasets in terms of MAE, MSE, and CC. The results are averaged across several imputation settings, where we varied the missing value rate from 10% to 95%. The standard errors are shown as vertical bars.
4.2 Neural Decoding: Mapping Brain Activity To Movement
Neural decoding is fundamental challenge in neuroscience that helps increase our understanding of the mechanisms linking brain function and behavior. In neural decoding, neural data are translated into in- formation about variables such as movement, decision-making, perception, or cognitive functions (Donner
ct al. 2009 Lin al. 2022 Rezaci et al., 2018 2023).
We use alternators to decode neural activities from three experiments. In the first experiment, the data recorded are the 2D velocity of monkey that controlled cursor screen along with 21-minute recording of the motor cortex, containing 164 neurons. In the second experiment, the data are the 2D velocity of the same monkey paired with recording from the somalosensory cortex, instead of the molor cortex. The recording was 51 minutes long and contained 52 neurons. Finally, the third experiment yielded data on the 2D positions of rat chasing rewards on platform paired with recordings from the hippocampus. This recording 75 minutes long and has 46 neurons. We refer the reader to Glaser al.) (2020) 2018) for more details how these data collected. For these experiments, the time horizons were divided into
10

---

Published in Transactions on Machine Learning Research (06/2025)
(A)
Motor Cortex
(B)
Somatosensory
(C) Hippocampus
1.0
Alternator
0.0
-1.0
1.0
bod Maniaa
0.0
-1.0
1.0
wmy WM
0.0
SRNN
-1.0
1.0
hyg
0.0
-1.0
1.0
X
0.0
VRNN
-1.0
1.0
M
Y
0.0
-1.0
1.0
M
0.0
Mamba
-1.0
Wo MMy yn mmn
-1.0
1.0
0.0
NODE
-1.0
1.0
-1.0
0.25
0.50
0.75
1.0
0.25
0.50
0.75
1.0
0.25
0.50
0.75
1.0
Time (Sec)
Time (Sec)
Time (Sec)
Ground Truth Trajectory.
Decoded Trajectory.
Figure 5: set of 20 trajectories sampled from different models conditional on spiking activities from datasets: Motor cortex, Somatosensory, and Hippocampus. The alternator produces samples that are closer to the ground truth dynamics.
1-second windows for decoding, with time resolution of ms. We use the first. 70% of each recording for training and the remaining 30% as the test set.
Similarly to the Lorenz experiment, we used attention models comprising two layers, cach followed by hidden layer containing 10 units for both the OTN and the FTN. We set a₂ = 0.1, a₂ 0.2, and a, = 0.4 was fixed for all t. The model underwent training for 1500, 1500, and 1000 epochs for Motor Cortex, Somatosensory, and Hippocampus datascts; respectively. We used the Adam optimizer with an initial learning rate of 0.01. We also used cosine anncaling learning rate scheduler with a minimum learning rate of 1e-4 and warm-up epochs.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Performance of different models on sea-surface temperature forecasting 1 to 7 days ahead. Numbers are averaged over the evaluation horizon. For SSR, a value closer to 1 is better. The time column represents the time needed to forecast all 7 timesteps for single batch. Alternators perform reasonably well in terms of CRPS and MSE, and are fast. However, they achieve worse SSR than MCVD and Dyffusion.
Method
CRPS +
MSEL
SSR (= 1)
Time [s]
DDPM-P
0.281 + 0.004
0.180 + 0.011
0.411 + 0.046
0.4241
DDPM-D
0.267 + 0.003
0.164 + 0.004
0.406 + 0.042
0.4241
DDPM
0.246 + 0.005
0.177 = 0.005
0.674 + 0.011
0.3054
Alternator
0.221 + 0.031
0.144 + 0.045
1.325 + 0.314
0.7524
Dyffusion
0.224 +: 0.001
0.173 = 0.001
1.033 + 0.005
4.6722
Mamba
0.219 + 0.002
0.134 + 0.003
0.753 0.009
0.6452
MCVD
0.216
0.161
0.926
79.167
In this experiment, we define the features as the velocity and the observations as the neural activity data. We benchmarked alternators against state-of-the-ar models, including VRNN, SRNN, NODE, and Mamba on their ability to accurately predict velocity/position /position given neural activity. We didn't include a diffusion model bascline for the same reason as in the Lorenz experiment, which was also a supervisod learning task. We used the same metrics as for the Lorenz experiment. The results are shown in Figure B Figure 4 and Figure] 5 Alternators are better at decoding neural activity than the baselines on all three datasets.
4.3 Sea-Surface Temperature Forecasting
Accurate Sca-Surface Temperature (SST) dynamics prediction is indispensable for weather and climate fore- casting and coastal activity planning. Expressivity is important here since prediction performance matters a lot more than interpretability. However, we tested alternators on this task to gauge how they would fare against models such as Mambas and diffusion models on this task. The SST dataset we consider here is the NOAA OISSTv2 dataset, which comprises daily weather images with high-resolution SST data from 1982 to 2021 (Huang et al., 2021]. We used data from 1982 to 2019 (15,048 data points) for training, data from the year 2020 (396 data points) for validation, and data. from 2021 (396 data points) for testing. We further turned the training data into regional image patches, selecting 11 boxes with resolution of 60 x 60 (latitudex longitude) in the eastern tropical Pacific Ocean. Specifically, we partitioned the globe into a grid, creating 60 x 60 (latitude x longitude) tiles (Cachay et al.) 2023). Eleven grid tiles are strategically subsampled, with a focus on the eastern tropical Pacific region, establishing refined and consistent dataset for subsequent SST forecasting
7 days into the future.
We used an ADM (Dhariwal & Nichol 2021 to jointly model the OTN and the FTN. The ADM is specific U-Net architecture that incorporates attention layers after each intermediate CNN unit in the U-Net. We selected 128 base channels, 2 ResNet blocks, and channel multipliers of (1,2 We trained the model with a batch size of 10 for 800 epochs, setting a₂ = 0.2, 0a = 0.3, and fixed 0₁ = 0.6 for all t. We used the Adam optimizer with an initial learning rate of 0.001 and applied a cosine anncaling learning rate scheduler with a minimum learning rate of 1e and warm-up epochs.
We compared the alternator against several baselines: DDPM (Ho et al., 2020). MCVD Voleti et al. 2022), DDPM with dropout enabled at inference time (Gal & Ghahramani 2016) (DDPM-D), DDPM with
random perturbations of the initial conditions coaditions/inputss /inputs with a fixed variance (DDPM-P) Pathak et all 2022 dyffussin (Cachay et al. 2023), and Mamba (Gu & Dao 2023). We used several performance metries. One such metric is the Continuous Ranked Probability Score (CRPS) (Matheson & Winkler 1976.. a proper scoring rule widely used in the probabilistic forecasting literature (Gneiting & Katzfuss 2014 de Bézenac
let al.] (2020). In addition to CRPS, we also used MSE and Spread-Skill Ratio (SSR). SSR asscsses the reliability of the ensemble and is defined as the ratio of the square root of the ensemble variance to the corresponding ensemble RMSF. It serves as measure of the dispersion characteristics, with values less
12

---

Published in Transactions on Machine Learning Research (06/2025)
than 1 indicating underdispersion (i.e., overconfidence in probabilistic forecasts) and larger values indicating overdispersion (Fortin et al., 2014). We used 50-member ensemble for each method and compute MSF based on the ensemble mean prediction.
Table 2 shows the results. The alternator achieves reasonable performance in terms of CRPS and MSE, even out purperrmmnng Dyffusion and MCVD while being significantly faster. However, the alternator performing worst in terms of SSR. We attribute this to the alternator's stochasticity, which introduces greater variability into the ensemble predictions.
5
Conclusion
We introduced alternators, a new flexible family of non-Markovian dynamical models for sequences. Alter- nators admit two neural networks that work in conjunction to produce observation and feature trajectorios. These neural networks are fit by minimizing the cross-entropy between two joint distributtions over the trajectories the joint distribution defining the model and the joint distribution defined as the product of the marginal distribution of the features and the marginal distribution of the observations, i.e. the data dis- tribution. We showcased the capabilities of alternators in three different applications: the Lorenz attractor, neural decoding, and sea-surface temperature prediction. We found alternators to be stable to train, fast to sample from, and reasonably accurate, often outperforming several strong baselines in the domains we studied.
While alternators demonstrate strong performance across various tasks, several limitations warrant consider- ation. For imputation tasks, our current approach uses missing-at-randonn sampling and relies on the latent state z dynamic memory to generate missing observations in forward manner. However, bidirectiona architectures could enhance performance when both the beginning and end of sequences are available by incorporating backward information flow. Future work could explore integrating bidirectional structures into both the OTN and FTN to smooth predictions through bidirectional processing, polentially improving imputation accuracy for sequences with complex temporal dependencies.
References
Marin Bilos, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Stephan Günnemann. Modeling temporal data continnous functions with process diffusion. arXiv preprint arXiv:221 1.221....2900 2022.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-206l An open-source autoregressive language model. arXiv preprint ar 2tff::20240074444 2022.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112((18))859 8779 2017.
Salva Rühling Cachay, Bo Zhao, Hailey James, and Rose Yu. Dyffusion: A dynamics-informed diffusion model for spatiotemporal forecasting, arXiv preprint ar Xiv:22330 01984, 2023.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018a.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Picter Abbeel. Pixelsnail: An improved autoregressive generative model. In International conference on machine learning, pp. 864 872. PMLR, 2018b.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. Advances in neural information processing systems, 28, 2015.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass, An unsupervised autoregressive model for speech representation learning. Xix preprint NNi::1944003400 2019.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke-Schncider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschosski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Information Processing Systems, 33:2995-3007, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Aduances in neural information processing systems, 188780-8794 2021.
Adji B Dieng, Chong Wang, Jianfeng Gao, and John Paisley, Topiernn: A recurrent neural network with long-range semantic dependency. arXiv preprint arXw::6111.01002 2016.
Adji B Dieng, Francisco JR. Ruiz, and David M Blei. The dynamic embedded topic model. orXiv preprint ar Xiv:v:1907.00 2019.
Tobias H Donner, Markus Siegel, Pascal Fries, and Andreas K Engel. Buildup of choice-predictive activity in human motor cortex during perceptual decision making. Current Biology, 19(18):1581-1588 2009.
Nan Du, Hanjun Dai, Rakshit Trivodi, Utkarsh Upadhyay, Manuel HomezzRoodrgueez and Le Song. Recur- rent marked temporal point processes: Embedding event history to vector. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1555 1564, 2016.
Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes. arXiv
preprint arXit:2206.080) 089992, 2022.
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In International conference on machine learning, pp. 3154-3164. PMLR, 2020.
Vincent Fortin, Mabrouk Abaza, Francois Anctil, and Raphael Turcotte. Why should ensemble spread match the rmsse of the ensemble mean? Journal of Hydrromeleovooooy, 15(4):1708 1713, 2014.
Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. Advances in neural information processing systems, 29, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In indernational conference machine learningg pp. 1050 1059. PMLR, 2016.
Joshua I Glaser, Matthew G Perich, Pavan Ramkumar, Lee E Miller, and Konrad P Kording. Population coding of conditional probability distributions in dorsal premotor cortex. Nature communications, 9(1): 1788, 2018.
Joshua Glaser, Ari Benjamin, Raeed Chowdhury, Matthew Perich, Lee Miller, and Konrad P Kording. Machine learning for neural decoding. Eneuro, 7(4), 2020.
Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting. Annual Review of Statistics and Its Application, 1:125-151, 2014.
Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. Deep autoregressive networks. In International Conference on Machine Learning, pp. 1242-1250. PMLR, 2014.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851,, 2020.
Michael Hopkins and Steve Furber. Accuracy and efficiency in fixed-point neural ode solvers. Neural com- pulation, 27(10):2148 2182, 2015.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Boyin Huang, Chunying Liu, Viva Banzon, Eric Freeman, Garrett Graham, Bill Hankins, Tom Smith, and Huai-Min Zhang. Improvements of the daily optimum interpolation sea surface temperature (doisst) version 2.1. Journal of Climate, 34(8))29233 2939, 2021.
Diederik P Kingma, Max Welling, et al. An introduction to variational autocncodors. Foundations and Trends@ in Machine Learning, 12(4):307-39) 2019.
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An rntroduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43((1):3364 -979, 2020.
Katiana Kontolati, Somdatta Goswami, Michael D Shields, and George Em Karniadakis. On the influence of over-parameeleriztion in manifold based surrogates and deep neural operators. Journal of Compulational Physics, 479:112008, 2023.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizmdonesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1-99 977, 2023.
Anjian Li, Zihan Ding, Adji Bousso Dieng, and Ryne Beeson. Efficient and guaranteed-safe non-convex trajectory optimization with constrained diffusion model. arXiv preprint arXiv:2403.066 03.07711. 2024.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Im improves controllable text generation. arXie preprinl 2xXv:2205.14217, 2022a.
Yan Li, Xinjiang Lu, Yaqing Wang. and Dejing Dou. Generative time series forecasting with diffusion, de-
noise, and disentanglement. Advances in Neural Information Processing Systems, 35:23009- 25-23009-23022, 2022b.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzade- nesheli, and Anima Anandkunar. Plysics-informed neural operator for learning partial differential equa- tions. ACM/JMS Journal of Data Science, 2021.
Haksoo Lim, Minjung Kim, Sewon Park, and Noseong Park. Regular time-series generation using sgm. arXiv preprint crXi::2301.08518, 2023.
Baihan Lin, Djallel Bouneffouf, and Guillermo Cecchi. Predicting human decision making in psychological tasks with recurrent neural networks. PloS one, 17(5):00267907, 2022.
Lequan Lin, Zhengkun Li, Ruikun Li, Xuliang Li, and Junbin Gao. Diffusion models for time series applica- tions: survey. arXiv preprint wrXiv:2355.0662, 2023.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural sde: Stabilizing neural ode networks with stochastic noise, 2019. URL hatps:///arxw..or//b//////// 02355.
Edward N Lorenz. Deterministic nonperiodic flow. Journal of atmospheric sciences, 20(2):130- 141, 1963.
James F. Matheson and Robert I. Winkler. Scoring rules for continuous probability distributions. Management science, 22(10))1087 1096, 1976.
Doug McLean. Continuum fluid mechanics and the navicr-stokos equations. Understanding Aerodynamics: Arguing from the Real Physics, pp. 13-78, 2012.
Kirill Neklyudov, Daniel Severo, and Alireza Makhzani. Action matching: A variational method for learning stochastic dynamics from samples. arXiv preprint 2:Xic:2210006622 2022.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1 1-67):1664, 64, 2021.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzaenesheli, et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint wrXhu2222..1124, 2022.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning, pp. 8857 8868. PMLR, 2021.
Mohammad R Rezaei, Anna K Gillespie, Jennifer A Guidera, Behzad Nazari, Saeid Sadrí, Loren M Frank, Uri T Eden, and Ali Yousefi. A comparison study of point-process filter and deep learning performance in estimating rat position using an ensemble of place cells. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Sociely (EMBC), pp. 4732 4735. IEEE, 2018.
Mohammad R Rezaci, Haseul Jeoung, Ayda Gharamani, Utpal Saha, Venkat Bhat, Milos R Popovic, Ali Yousefi, Robert Chen, and Milad Lankarany. Inferring cognitive state underlying conflict choices in verbal stroop task using heterogeneous input eeeriminative-eeneaatyy decoder model. Journal of Neural Engineering, 2055)00 2023.
Mohammad Reza Rezaci, Kensuke Arai, Loren M Frank, Uri T Eden, and Ali Youscfi. Real-time point process filter for multidinensional decoding problems using mixture models. Journal of neuroscience methods, 348:109006, 2021.
Erwin Schrödinger. An undulatory theory of the mechanics of atoms and molecules. Physical review, 28(6): 1049, 1926.
Wilson Truccolo, Uri T Eden, Matthew R Fellows, John P Donoghue, and Emery N Brown. point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. Journal of newrophysiologı, 93(2):1074 1089, 2005.
Vikram Voleti, Alexia Jolicoeur-Martinenn and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in Neural Information Processing Systems, 35:23371 23385, 2022.
Gerhard Wanner and Ernst Hairer. Solving ordinary differential equations II, volume 375. Springer Berlin Heidelberg New York, 1996.
Tijin Yan, Hongwei Zhang, Tong Zhou, Yufeng Zhan, and Yuanqing Xia. Scoregrad: Multivariato probabilistic time series forecasting with continuous energy-based generative models. arXiv preprinl arXiv:2106.1011 10121, 2021.
A
Appendix
Sequence-to-scqueenee prediction algorithm. Given paired sequences æı. and fı.r, employ alter- nators to predict f1.r from æ1.r and vice versa. Am 2 outlines the procedure for sequeecceto-ssqquenee prediction using alternators.
Estimating the log-likelihood of a new sequence. Somctimes, scientists may be interested in scoring given sequence using a model fit on data to study how the new input sequence deviates from the data. Alternators provide way to do this using the log-likelihood. Assume given input sequence कं.-- We
16

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm 2: Sequence-To--eqqeeee Prediction with Alternators
Inputs: Samples from p(œ1:7, Mi.r), batch size B, o² and oy. schedule a1: Initialize model parameters 8 and while not conweryed do for b= L,..., B do
Draw initial latent 20 for t= ...... do
(6) ND.,
Compute pave
(11 -²2) So fo
Compute Pir /ou go(a;; (b)) + (1 oz)) -
end end
Compute loss C(O, Φ) in Eq. 14 using samples from p(ær., yı.))
Backpropagate to get Vo(C(O, Φ)) and Vo(C(O, Φ))
Update parameters 0 and ø using stochastic optimization, e.g. Adam.
end
can estimate its likelihood under the Alternator as follows:
log (e log / po.o ZOiT) dzo.T
(18)
log / po,o(Zor)) po( 20)) II
po(2; z(-1))
(19)
1=2
log (= exp log 20) +
log pa 21-1)
(20)
K K
T
ss log
exp
log
12
+
log pa (x) (
(21)
where 20.T:-. 2oT are K samples from the marginal pe. (zor). Eq. 21 sequence scoring function and it be computed in numerically stable way using the function logsumexp()).
(1) (K)
Neural activity forecasting. We applied alternators to forecast neural activity across Motor Cortex, Somatosensory Cortex, and Hippocampus datasets, with forecasting rates from 10% to 50%. Evaluated using Mean Absolute Error (MAE), Mean Squared Error (MSE), and Correlation Coefficient (CC), alternators consistently outperform VRNN, SRNN, NODE, and Mamba (Figure 6b, demonstrating superior robustness and accuracy across varying forecasting horizons.
B Implementation Details
This section provides comprehensive implementation details for experimental tasks, including hyperparame- ter optimization strategies, architectural configurations, and training procedures used across the three main experimental domains.
B.1 Lorenz Attractor Modeling
For the Lorenz attractor experiments, we employed a 2-layer iatention-based architecture for both the Obscrvation Transition Network (OTN) and Feature Transition Network (FTN). Each attention layer was followed by hidden layer containing 10 units, providing sufficient capacity to capture the complex chaotic dynamics while maintaining computational efficiency.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Mean Absolute Error
Mean Squared Error
Correlation Coefficient
0.30
0.6
0.25
Motor Cortex
0.25
0.20
0.20
0.4
0.15
0.15
0.10
0.10
0.2
0.05
0.1
0.05
0.00
0.7
0.4
0.35
Somatosensory
0.6
0.30
0.3
0.5
0.25
0.4
0.20
0.2
0.3
0.15
0.10
0.1
0.0
0.0
0.00
0.6
0.8
0.4
Hippocampus
0.5
0.6
0.4
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.0
0.0
0.0
Alternator
Mamba
NODE
VRNN
SRNN
Alternator
Mamba
NODE
VRNN
SRNN
Alternator
Mamba
NODE
VRNN
SRNN
Figure 6: Alternators outperform VRNN, SRNN, NODE, and Mamba on forecasting in the neural decoding lask on all three datasets in terms of MAE, MSE, and CC. The results are averaged across several forecasting settings, where we varied the forecasting rate from 10% to 50%. The standard errors are shown as vertical bars.
The noise variance parameters were carefully tuned through grid search optimization 0 0g E [0.01, 0.8). We find the latent noise variance a, 0.1 and observation noise variance d, 0.3 as the best choices. The alternation parameter a; 0.3 was kept fixed across all time steps maintain consistent switching dynamics between the forward and backward processes. In this experiment, models were trained for 500 epochs using the Adam optimizer with initial learning rate of 0.01. We applied cosine annealing learning rate scheduler that reduced the learning rate to a minimum of 1 x 10-4 over the training period, with 10 warm-up epochs to stabilize initial training dynamics. The training data consisted of 400 time steps simulated from the Loreuz equations with added Gaussian noise. Model performance was evaluated on 100 newly simulated paired soquences following the same simulation procodure as the training data. We assessed feature trajectory prediction performance using Mean Absolute Error (MAE), Mean Squared Error (MSE), and Correlation Coefficient. (CC) metrics, comparing predictions against ground truth Lorenz attractor features.
18

---

Published in Transactions on Machine Learning Research (06/2025)
B.2 Neural Decoding Experiments
We conducted experiments three distinct neural datasets. The motor cortex dataset contained 164 neurons recorded over 21 minutes, paired with 2D cursor velocity data. The somatosensory cortex dataset included 52 neurons recorded over 51 minutcs, also paired with 2D velocity measurements. The hippocampus dataset comprised 46 neurons recorded over 75 mimutes, paired with 2D position data of a rat navigating a reward platform. Time horizons were segmented into 1-second windows for decoding analysis, with a temporal resolution of 5 ms. We allocated the first 70% of each recording for training purposes and reserved the remaining 30% as the test set to ensure robust performance evaluation.
Similar to the Lorenz experiments, we utilized attention models comprising two layers for both OTN and FTN components. Each attention layer was followed by a hidden layer containing 10 units, maintaining architectural consistency across experimental domains while adapting to the specific characteristics of neural data.
Through systematic hyperparameter search, we determined optimal settings for each dataset from range of 02 Ua [0.01, 0.8). The latent noise variance was set to a= 0.1, while the observation noise variance was configured as ox = 0.2. The alternation parameter ar 0.4 was maintained constant across all time steps.
Training epochs were customized for each dataset based on convergence characteristics. The motor cortex dataset required 1500 epochs, the somalosensory dataset also needed 1500 epochs, while the hippocampus dataset converged after 1000 epochs. We employed the Adam optimizer with an initial learning rate of 0.01, coupled with cosine anncaling scheduler that reduced the lcarning rate to 1 10 with warm-up epochs. Model performance was evaluated using the same metrics the Lorenz experiments, specifically MAE, MSE, and CC, to assess the accuracy of velocity and position predictions from neural activity patterns.
B.3 Sea-Surface Temperature Forecasting
We utilized the NOAA O1SSTv2 dataset, encompassing daily high-resolution SST data from 1982 to 2021. The dataset was partitioned into training data from 1982 2019 (15,048 data points), validation data from 2020 (396 data points), and test data from 2021 (396 data points). We focused the eastern tropical Pacific Ocean region, extracting 11 boxes with 60 x 60 (latitude x longitude) resolution for detailed analysis.
We employed an ADM network structure to jointly model the OTN and FTN components. The ADM utilized a specialized U-Net architecture incorporating attention layers after each intermediate CNN unit. The configuration included 128 base channels, 2 ResNet blocks per resolution, and channel multipliers of 21, to capture multi-scale spatial-temporal patterns effectively.
Models were trained with batch size of 10 over 800 epochs. The noise variance parameters were set to = 0.2 and 0₂ 0.3, with a fixed alternation paramcter a 0.6 across all time steps. We used the Adam optimizer with an initial learning rate of 0.001, applying cosine annealing scheduler that reduced the learning rate to 1 x 10-1 with 5 warm-up epochs.
The models were designed to predict SST dynamics to 7 days into the future, providing short to medium- term forecasting capabilities essential for weather and climate applications. Performance assessment utilized multiple complementary metrics. The Continuous Ranked Probability Score (CRPS) served as the primary probabilistic forecasting metric, supplemented by Mean Squared Error (MSE) for deterministic accuracy, The Spread-Skill Ratio (SSR) was employed to evaluate ensemble reliability, with values closer to 1 in- dicating optimal calibration between forecast uncertainty and actual forecast skill. All SST experiments were conducted on NVIDIA A6000 GPUs with 48GB of memory, enabling efficient processing of the high- dimensional spatial-temporal inputs essential for accurate SST forecasting. All experiments were conducted using appropriate computational resources ensure reprodueible results and fair comparison with base- line methods. Timing measurements recorded to assess computational efficiency alongside prediction accuracy.
19


Reviews:
[{"review_id": "7o1q2m39Ly", "reviewer": "Reviewer_2iWE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* The paper is well written and has a clear story\n* Related works and baselines covers diverse set of the literature, which I appreciate\n* The approach is an interesting twist on the recurrent modeling architecture and has competitive performance.\n\n## Weaknesses\n\n**[W1]** Some derivation would benefit from showing some derivation, notably eq. (9). Note: in eq. (8), is the Bayes rule really used or is it just the conditional probability definition?\n\n**[W2]** The manuscript lacks a bit of discussion on the limitation of the approach. A concrete example is in the 'Imputation and forecasting' use case: because the alternator is a sequential model, it might be limited in settings where data is missing in the middle of a sequence, but the start and the end of the sequence is available. A model with an internal hidden state, e.g., LSTM, or an attention-type model would be able to leverage signal from the end of the sequence for inferring the missing data. Other limitation I might not thinking of now, but the authors might have came across during experimentation would be worth some discussion also.\n\n**[W3]** It seems attention-type architecture (Transformers, Conformer) is missing from related works and baselines. Is there a reason for this? I hate to ask you to add another baseline to your already extensive set of baseline, however they have proven to be very useful in a wide range of task, I would expect them to be competitive in this setting too. See [1,2].\n\n**[W4]** In the related works section authors mentions that a weakness of the Neural ordinary differential equation (NODEs) is that NODEs suffers from modeling noisy observations because they are deterministic. See [3,4].\n\n**[W5]** In the evaluation settings, the authors list the hyperparameters, e.g., $\\sigma_z$, $\\sigma_x$, $\\alpha_t$, lr, ..., used in the experiments. However, it is not clear how the authors arrived at these values. I would be good for a bit of details about what was explored, even is very high level. Hyperparameters obtained by manual exploration of the authors is fine also, lets just mention it here.\n\n**[W6]** Again regarding the hyperparameters: were all the baselines ran with the same set of hyperparameters (lr, warmup period, etc.)? If so, were they the optimal setting for each?\n\n## References\n[1] Vaswani, Ashish, et al. \"Attention is all you need.\" Advances in neural information processing systems 30 (2017).\n\n[2] Gulati, Anmol, et al. \"Conformer: Convolution-augmented transformer for speech recognition.\" arXiv preprint arXiv:2005.08100 (2020).\n\n[3] Oganesyan, Viktor, Alexandra Volokhova, and Dmitry Vetrov. \"Stochasticity in neural odes: An empirical study.\" arXiv preprint arXiv:2002.09779 (2020).\n\n[4] Oh, YongKyung, Dong-Young Lim, and Sungil Kim. \"Stable neural stochastic differential equations in analyzing irregular time series data.\" arXiv preprint arXiv:2402.14989 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents a novel family of models for sequential data. It comprises a pair of learnable neural networks, one (OTN) of which maps from observation space to abstract feature space and the other (FTN) does the opposite. These are trained jointly by minimizing the a cross-entropy type objective over the observation space and the feature space.\n\nAuthors use this framework in a couple settings. First, they learn an artificial nonlinear mapping of Lorentz equation trajectories. Second, they learn a real-world mapping between movement and brain activity. Finally, they evaluate alternators on a sea-surface temperature. Overall the performance is satisfactory compared to baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical\n\n**[RC1]** Attention-type baselines (Transformer, or conformer). See [W3]\n\n**[RC2]** Description of how the hyperparameter were obtained (Could be in the appendix section). See [W5,W6]\n\n## Recommendation\n\n**[RC3]** Derivation for eq (9) in the appendix. See [W1]\n\n**[RC4]** Either a limitation section or limitation discussion in the capabilities sub-sections of Section 2. See [W2]\n\n**[RC5]** Amend related works section to include neural stochastic differential equations. See [W4]\n\n**Minor**\n* Some acronyms are sometimes not defined (e.g., SSR and SST in Section 4.3, MAE, MSE, and CC in Section 4.1). I do agree that these are either standard or easily inferable, but it would improve the reading experience.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RU0opms2XE", "reviewer": "Reviewer_XceT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The alternator framework introduces a unique alternating mechanism between observation and feature spaces, balancing flexibility and interpretability.\n- Validated on diverse tasks (chaotic dynamics, neuroscience, climate) with consistent improvements over strong baselines.\n- Faster sampling than diffusion models and competitive runtime compared to Mambas.\n\nWeaknesses:\n- The justification for the cross-entropy loss (Eq. 7) lacks rigorous connection to established objectives (e.g., ELBO).\n- The Lorenz experiment uses simulated data, raising questions about generalizability to real-world latent dynamics.\n- Poor SSR in SST forecasting (Table 2) is attributed to stochasticity but not analyzed or mitigated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces alternators, a novel framework for sequence modeling featuring two neural networks (OTN and FTN) that alternate between generating observations and low-dimensional latent features. Key contributions include:\n- A flexible architecture for both generative and sequence-to-sequence tasks, enabling efficient sampling, imputation, and forecasting.\n- A training objective based on cross-entropy minimization over joint distributions of observations and latent trajectories, promoting interpretable latent dynamics.\n- Empirical validation across three domains: chaotic systems (Lorenz attractor), neural decoding, and climate forecasting, demonstrating superior performance over baselines like Mambas, neural ODEs, and diffusion models in accuracy and speed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical for acceptance:\n1. Clarify the theoretical motivation for the loss function (Eq. 7) and its relationship to variational inference or other frameworks.\n2. Address the SSR issue in SST forecasting with analysis or adjustments to the model’s stochasticity.\n\nNon-critical but recommended:\n1. Provide details on architecture/hyperparameter selection (e.g., why fixed αₜ values, attention layers).\n2. Test alternators on non-synthetic latent dynamics or larger-scale datasets.", "broader_impact_concerns": "The paper lacks a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c1Mt1AMhw5", "reviewer": "Reviewer_eog1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method was introduced in detail and authors also provided necessary mathematical proofs.\n2. Authors conduct extensive experiments on different domains along with sufficient analysis, which made the results more convincible. \n3. The paper is well organized, and the writing is relatively clear. \n\nWeaknesses:\n1. The motivation of the model design is not fully convincible. According to authors description, $x_{t}$ is derived from $z_{t-1}$ ($x_{t} \\sim F(z_{t-1})$), and then $z_{t}$ is obtained from both $x_{t}$ and $z_{t-1}$ ($z_{t} \\sim G(z_{t-1}, F(z_{t-1}))$), why the path should be $z_{t-1}$ - $x_{t}$ - $z_{t}$ instead of directly deriving $z_{t}$ from $z_{t-1}$ and decoding the latent $z_{t}$ to $x_{t}$?  In some previous works, the observation $x$ participates into the propagation of latent $z$ to correct the latent feature with given ground-truth, but in authors setting, $x$ doesn’t bring any additional ground-true information. \n2. In related works, authors claimed that there is no stochasticity in neural ODE. But there are already some works related to neural SDE, which is a step further beyond ODE by taking stochastic terms into the propagation. Moreover, besides those numerical solvers, there are also simple implementations such as using Euler method, which is in an auto-regressive manner, just like authors model does.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, authors proposed a new family of dynamical sequential model called Alternators. By using two separate networks to alternatively generate observations and latent features, authors claimed that the model can better capture the latent dynamics underlying complex data. Through experiments on different datasets, the proposed method has proven to be effective.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think authors should spend more efforts on demonstrating their design reasonability and distinguishing with those stochastic differential models.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that  the claims made in the submission supported by accurate, convincing and clear evidence, and that at least some individuals in TMLR's audience be interested in knowing the findings of this paper. 
Two reviewers suggest acceptance, while Reviewer eog1 suggests rejection.

While I agree with Reviewer eog1 that neural SDEs could have been an interesting baseline, there was no explicit request from the reviewer to run further experiments with this baseline, neither in the original review, nor between the authors' reply (12 Apr 2025) and the reviewer's recommendation (15 May 2025).

Authors, please update your manuscript with all promised changes before publication.
Decision: Accept (poster)