Neurons as Detectors of Coherent Sets in Sensory Dynamics
Authors: Joshua L. Pughe-Sanford
Xuehao Ding
Jason J Moore
Anirvan M. Sengupta
Charles Epstein
Philip Greengard
Dmitri Chklovskii
Venue: NeurIPS 2025 poster ()
Keywords: Neuron
self-supervised learning
dynamics
spectral clustering
Primary area: neuroscience_and_cognitive_science
TL;DR: Neurons can cluster sensory dynamics into coherent sets by projecting inputs onto a singular vector and rectifying the result

Abstract: We model sensory streams as observations from high-dimensional stochastic dynamical systems and conceptualize sensory neurons as self-supervised learners of compact representations of such dynamics. From prior experience, neurons learn {\it coherent sets}—regions of stimulus state space whose trajectories evolve cohesively over finite times—and assign membership indices to new stimuli. Coherent sets are identified via spectral clustering of the {\it stochastic Koopman operator (SKO)}, where the sign pattern of a subdominant singular function partitions the state space into minimally coupled regions. For multivariate Ornstein–Uhlenbeck processes, this singular function reduces to a linear projection onto the dominant singular vector of the whitened state-transition matrix. Encoding this singular vector as a receptive field enables neurons to compute membership indices via the projection sign in a biologically plausible manner. Each neuron detects either a {\it predictive} coherent set (stimuli with common futures) or a {\it retrospective} coherent set (stimuli with common pasts), suggesting a functional dichotomy among neurons. Since neurons lack access to explicit dynamical equations, the requisite singular vectors must be estimated directly from data, for example, via past–future canonical correlation analysis on lag-vector representations—an approach that naturally extends to nonlinear dynamics. This framework provides a novel account of neuronal temporal filtering, the ubiquity of rectification in neural responses, and known functional dichotomies. Coherent-set clustering thus emerges as a fundamental computation underlying sensory processing and transferable to bio-inspired artificial systems.

Neurons as Detectors of Coherent Sets in Sensory Dynamics
Joshua L. Pughe-Sanfordd" Xuehao Ding!." Jason Moore
Anirvan M. Sengupta
Charles Epstein' Philip Greengard' Dmitri B. Chklovskii
Flatiron Institute, Simons Foundation, New York, NY, USA
Neuroscieence Institute, NYU Langone Medical Center, New York, NY, USA
PPhysics Department, Rutgers University, New Brunswick, NJ, USA
[jpughesanford, xding, cepstein, pgreengard, mityaleflatirooisstiion
org,
j ason. mwwreenyylaanooee. org, anirvans waw.aassppyyiismmmaal.... com 'qqual contribution
Abstract
We model sensory streams as observations from high-dimensional stochastic dy- namical systems and conceptualize sensory neurons as self-supervised learners of compact representations of such dynamics. From prior experience, neurons learn coherent sets regions of stimulus state space whose trajectories evolve cohesively over finite times and assign membership indices to new stimuli. Coherent sets are identified via spectral clustering of the stochastic Koopman operator (SKO), where the sign pattern of a subdominant singular function partitions the state space into Coistein-Uhlenbben
minimally coupled regions. For multivariate singular function reduces to a linear projection onto the dominant singular vector of the whitened state-transition matrix. Encoding this singular vector as receptive field enables neurons to compute membership indices via the projection sign in a biologically plausible manner. Each neuron detects either a predictive coherent set (stimuli with common futures) or a retrospective coherent set (stimuli with common pasts), suggesting a functional dichotomy among neurons. Since neurons lack access to explicit dynamical equations, the requisite singular vectors must be estimated directly from data, for example, via past-future canonical correlation analysis on lag-vector representations an approach that naturally extends to non- linear dynamics. This framework provides a novel account of neuronal temporal filtering, the ubiquity of rectification in neural responses, and known functional dichotomies. Coherent-set clustering thus emerges as a fundamental computation underlying sensory processing and transferable to bio-inspired artificial systems.
processes, this
Neurons in early sensory areas are traditionally thought to extract from recent inputs low-dimensional latent variables that are maximally informative about the near future 2 Such extraction exploits statistical regularities acquired over evolutionary, developmental, and behavioral timescales from previously encountered natural stimuli [ 7 To formalize this intuition for temporally correlated sensory stimuli, we postulate that they are generated by high-dimensional, potentially nonlinear, stochastic dynamical processes, and conceptualize neurons as self-supervised learners of coherent sets retsons of the stimulus state space that evolve cohesively over finite time intervals [8] 9 thus enabling compact representations of sensory dynamics.
Coherent sets can be uncovered via spectral clustering of the stochastic Koopman operator (SKO) linear, albeit infiite-dimmesional operator that evolves observables over time interval [0 13). The sign of the first non-trivial (subdominant) singular function of the SKO partitions state space into two minimally interacting coherent sets (Fig. la). Accordingly, a neuron can compute membership index of input by evaluating the sign of subdominant singular function. Because singular values and functions remain real even for irreversible dynamies, this approach generalizes metastable set detection beyond the reversible cases that eigenfunction methods require [14].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We demonstrate that for multivariate OrasteinUUhlenbeck (OU) process 15 a canonical example of linear stochastic dynamics and a reasonable model of summed imput to a neuron [6
subdominant singular function of the SKO corresponds to projection of the input onto singular vector of the whitened finite-time transition matrix, Fig. 1b. neuron that stores this singular vector in its synaptic weights and temporal filter can compute the corresponding membership index via the sign of the weighted sum of the inputs. Although the conventional spectral clustering framework assumes stability, our singular function results extend to unstable systems by focusing singular values closest to one. In addition to projecting on the right singular vector predicting near-future inputs (Fig. lb), projecting
the left one is also possible possible-rrtrosppeeeee retrospecting the recent past (Fig. 1c). Because such projections require different synaptic weights and temporal filters they must be implemented by distinct neurons.
Recognizing that the underlying dynamical equations are not available to neurons, a biologically plausible detection of coherent sets requires data-driven algorithm that can infer them directly from observations. Coherent sets can be learned and membership sets inferred locally using canonical correlation analysis (CCA) of past-future data pairs [88 I3 [9]. This algorithm has the additional advantage of being applicable to nonlinear dynamics: relies on estimating a Galerkin projection of the SKO onto functional basis via Monte Carlo integration over observed data Io 20 21 12]. If the upstream neurons implement such projection, a post-synaptic neuron could learn/represent a requisite singular vector in synaptic weights indicating set membership by the sign of the inputs sum.
Viewing neurons as coherent set detectors sheds light onto several longstanding neurophysiological observations. First, temporal receptive fields of neurons emerge naturally as subdominant singular vectors projecting input lag-vector representations of dynamical states. Second, the ubiquity of response rectification in neurons, exemplified by the well-known ON/OFF segregation in early visual circuits, is interpreted as a principled clustering mechanism. Finally, the theory predicts complementary neuronal classes that predict near future using predictive coherent sets retrospect recent past using retrospective ones, consistent with known neuronal functional dichotomies such as tufted versus mitral cells in the olfactory bulb non-lagged versus lagged cells in the lateral geniculate nucleus (LGN). Thus, the detection of coherent sets can serve as a powerful algorithmic primitive for neural computation supporting prediction and retrospection. This offers insights into biological processes and could inspire future artificial neural networks.
The remainder of the paper is organized as follows. Section 1 discusses related work. Section 2 reviews the definition of transfer operators, coherent sets and their connection to spectral clustering through the singular functions of the SKO. In Section 3, we derive the central result: under OU dynamics, the subdominant singular functions of the SKO correspond to dot products between the state and the singular vectors of the whitened transition matrix. Section 4 reviews data-driven algorithm for identifying subdominant singular functions, which extends naturally to nonlinear dynamics and defines neuronal units rectifying positive negative parts of the subdominant singular function. Section analyzes and reviews several experimental datasets, interpreting temporal receptive fields through the hypothesis that biological neurons cluster coherent sets.
Ug
Goordinate
coordinate
coordinate
coordinate 1
coordinate 1
Figure 1: Subdominant SKO singular functions partition states into coherent sets (blue vs red). (a) For one-dimensional double-well potential (black) the dominant singular function corresponds to a stationary distribution (yellow) and the subdominant singular function (red-blue) partitions the state space to minimize the leakage between the coherent sets. Left (b) and right (e) subdominant singular functions partition the state space near a 2D saddle point based on shared future and past, respectively. Black lines indicate attractive and repulsive invariant manifolds approximated by the linear subspaces in the vicinity of the saddle point (circle). Left (green) and right (black) singular vectors of the finite-time whitened transfer matrix, [u ug A (vi v2] As the forecast horizon grows, V1 becomes orthogonal to the stable subspace and U2 becomes orthogonal to the unstable subspace.

---

Related Work
Viewing early sensory processing as efficient or predictive coding of natural stimuli has long tradition S
6 7
22 2] Closest to our work is the information bottleneck (IB) framework [ 3 4
which for Gaussian variables and stable dynamical systems, reduces to past-future CCA 23 In this setting, optimal one-bit encoding corresponds to the sign of the subdominant singular function of the SKO; we generalize this result to wnstable dynamics. Detecting coherent sets can also be viewed as an extension of slow feature analysis [24] [25] However, our explicit formulation in terms of unstable dynamical systems yields a principled nonlinearity that supports non-trivial feature generation and naturally extends to deep architectures [26). Predictive and retrospective neuron classes have also appeared in a lattice filter model of the visual pathway [27]. though that framework was restricted to linear processing. Relative to predictive coding theories 228 29 30 27 which posit that neurons transmit prediction errors, we instead emphasize how predictions can be computed-via coherent-set memberships learned from past-future correlations. Finally, while ON/OFF pathway segregation has been attributed to metabolic efficiency [1], we propose a complementary computational explanation. Our account of lagged and non-lagged LGN cells differs from 32 in that it does not require an explicit nonlinearity, suggesting that these types can emerge alongside ON/OFF segregation.
Prediction and retrospection in saddle-point dynamics have been discussed in the context of unstable periodic orbits of chaotic attractors using the dominant mode of the local SKO (and its adjoint) 33 34). whereas we focus on a subdominant singular function defining a coherent set pair K5
Saddle point analysis based on eigenfunctions instead of singular functions predicted neuronal filters orthogonal to decaying exponentials, yielding predictive, but not retrospective, neurons (36).
Clustering has been previously proposed as a model of static neuronal computation on temporally uncorrelated inputs (37), capturing rectification and sparsity but not temporal receptive fields or sensory dynamics. Probabilistic coding frameworks such as the Bayesian brain hypothesis [B8] have been widely studied although we are not aware of the suggestion that single neurons represent eigenfunctions or singular functions of the SKO. Self-supervised learning has been applied to visual networks [39]. but these models typically omit analysis on the neuronal level. Koopman spectral analysis and data-driven Koopman approximations (e.g. DMD) have long tradition in nonlinear dynamics [00 II 40 20
22. 13 and have recently been incorporated into deep architectures to extract predictive features [41] In contrast, we use SKO singular functions to define coherent sets 33 9 providing direct explanation for rectification and a principled division into predictive and retrospective neurons.
21
Estimation of singular functions from data originated in molecular dynamics as the VAMP frame- work [4]. VAMPnets [42) learn such functions with deep ReLU networks trained by backpropagation. In our model, features arise from rectified projections onto singular vectors within each neuron and can be hierarchically composed without backpropagation.
2 Discovering coherent sets via spectral clustering of transfer operators
We begin with the concept of coherent sets, which are regions of state space that tend to move as whole under the stochastic dynamics over finite time intervals 4 35 I. Formally, a pair of sets (A, B3) is coherent if a state that begins in A at time t is very likely to be found in B at time t +
P|X(t + 7) E ((t) E Al 1,
where P[X|Y] is a probability of X given Y. X(() E X' denotes the random state of the system at time t. If in addition the probabilities of visiting A at time t and B at time t + 7 are equal, then this relation works in both directions: observing the system in B at time + makes it likely that it was in A at time t. Thus, membership in coherent sets, .A and B, can be used to predict the future and retrospect the past, respectively, of stochastic process.
To compute coherent sets we utilize the stochastic Koopman operator (SKO), which encodes how observables evolve in expectation [10, 44 11 35
9 13 [2], see the Supplement, Section for concise introduction. An observable is any scalar function X with a finite expectation value over the stochastic process. For instance, from the neuron's 's perspective, each synaptic input can be thought of as such an observable of stimuli. The SKO acts on an observable f
K, f() = El +))x((+++))((() (t) x].
(I)
3

---

Le. it maps the present observable to its expected value units of time later. Linearity of the expectation makes K linear operator.
Let (x() and v(x) denote the probability densities of the system's state at times and & + т,
respectively 35 13 For any measurable region A E X,
[ ((x) dx,
P (t) E A| =
E] = v(x) dx.
(2)
These densities define inner products between functions f and g as
(f.9)P Sx fxx)* 9(x) p(x) dx,
(f.g)v f f(xx) g(x) v(x) dx.
When the state space is divided into two pairs of regions, identifying the most coherent ones amounts maximizing the following objective [ 13):
max A,
IP
X(t x( E
X(t) E A) +P + x((+T) X(1) E Ar}
max A,B B
(14, 1A)p
(1A- 1Ar)
( [xxxxx=== E A] = -[ E = u(x) dx,
subject to
where (A, B) are measurable subsets of the state space X, and (A4,, B³) denote their complements. The indicator function (x) equals 1 if x E A and 0 otherwise. Large values of the objective correspond to pairs of regions that remain coherent under the dynamics-thon that is, regions that are least dispersive over the time interval
In practice, the maximizers of this quotient are -ell-approximated by certain singular functions of To see this, note that Kr, has an adjoint operator 'CT, defined so that
the SKO 35 9 13
(f.K-g) (K.
(5)
The finite-time forward-backwart and backward-foree operators
=- K,K
B, K.t
are then self-adjoint and, under mild assumptions, compact spectral decomposition in terms of orthonormal basis (10)
13
18 As such, each admits a countable
D
=
Σ X us (vey 2)
B,
Σ
X us (U:
where may be infinite. The functions (x(x) and us (x) are the singular functions of Kr,, satisfying
[x)u:[xx= = (x);
[K]xs](x) = Axux(x).
(8)
Pairs of singular functions with large X; are approximate maximizers V fx (x) 0}:
Moreover, the sets
4 {x -u (x) 0),
approximate coherent sets, where 4 is approximately the image of Di under the dynamics.
The principal singular functions of the SKO are trivial (x() = up xx) = 1: they define the trivial coherent set, 2á Vo- X. The subdominant singular functions define the least dispersive non-trivial coherent set pair of the dynamics, ,V;=,,,**) We propose that the neurons compute membership indices by evaluating the signs of such singular functions. Left singular functions look forward in time predicting the future. Right singular functions look backward in time allowing to retrospect past events. Both types of measurements are important ascertaining the state of partially observed dynamical systems, and we predict the existence of both predictive and retrospective neurons.
3 Koopman singular functions for Ornstein-Uhlenber processes
Identifying coherent sets through the subdominant singular functions of the SKO provides a rigorous theoretical framework, whose application requires computing these singular functions. In this section,

---

we attempt to provide an intuitive picture by considering linear stochastic dynamical system - OU process for which subdominant SKO singular functions can be found in the closed form providing much needed intuition. The temporal dynamics of neuronal input currents elicited by sensory stimuli
has long been approximated by the OU process [16 17
dx(() (x(() + E(1) dt
(10)
where (E())<(t)T)= D8(t (t),--)). t'). Here, we assume that A is real matrix with real eigenvalues and eigenvectors. We are particularly interested in saddle-point OU for the following reasons. If critical point is purely repulsive, it will not be visited by the autonomous dynamics and, hence, will be physically irrelevant. Applying this approach to nearly isotropic attractive critical points gives partitions that are not "distinguished' [9], i.e. not represent genuine clustering of states which could describe qualitatively different parts of the phase space.
The probability density of the state variable x. p(x, t), evolves according to the forward Kolmogorov equation (aka the Fokker-Plann Planck equation) utilizing the forward Kolmogorov operator & 45
0p(x. t) = Cp(x, t) V (Axp(x,t) t)) + J - (DVp(x, t)).
(11)
aL
The dynamics of measurement expectation g(x,t) (l)(X = x] are given by the backward Kolmogorov equation:
0g(x,t) t) cl (x), (Ax) v 9xx 1) + v (DVg(xx 1)), 01
(12)
2
where Lt is the adjoint of the forward Kolmogorov operator with respect to the standard Euclidean inner product, serving as the generator of the SKO [T0]:
= exp
(13)
The stationary distribution of probability density under L, satisfying Cpo 0, is derived in the Supplement, Section 2:
Po(x) ~ exp
exp
2 xxTE>>x)
(14)
where x E R and Σ is solution of the Lyapunov equation,
AE + ΣA DD
(15)
For attractive dynamics, where all eigenvalues of A have negative real parts, Eq. (5) has a unique positive-definite solution corresponding to the covariance matrix, Σ E xx For repulsive or saddle-point dynamics, Eq. (15) is a Sylvester equation, which has a unique solution and only if A has no eigenvalues related by sign reversal (46). Thus, for generic choice of A, Eq. (15) has a unique solution that is symmetric, real, and invertible. For repulsive directions, corresponding to eigenvalues of A with positive real parts, Eq. (14) characterizes how fast the density grows away from the fixed point 47].
Under stationarity, (xx) vxx) (x))., the adjoint of K, with respect to po(x) is given by
c diag(po)~ exp (Cr) (Cr)diag(Po),
(16)
which is termed the reweighted Perron- FroronnFronbenuus operator and can also be interpreted as the SKO of the reverse-time dynamics. We then search for the eigenfunctions of the operators F,, and B,, Eq. (6). which are well defined on short time scales.
We find (see the Supplement, Section 3) family of eigenfunctions of Fr, Br that can be expressed in terms of the linear projection of the state vector, x:
(V Tx x) = (v, xx),
(u x) (u; Tx).
(17)
where and v, satisfy the following matrix eigenvector equations (see the Supplement, Section 3): E-IeArE T =
A A X
(18)
Eigenvalues of transfer operators for expanding maps could be greater than one 48). Since ideal coherent sets with zero leakage correspond to unity eigenvalue, in this paper subdominant eigenfunction refers to the non-trivial eigenfunction associated with eigenvalue closest to one.
5

---

That is, the subdominant eigenfunction for attractive (repulsive) dynamics is associated with the second largest (smallest) eigenvalue. We proved that the subdominant eigenfunctions belong to the linear family Eq. (17) for attractive and repulsive dynamics. Other eigenfunctions of Fr. B, are either a constant, associated with the eigenvalue one, that does not change sign, or represent higher-order polynomials in x, associated with less dominant eigenvalues (see the Supplement, Section 3, which, however, does not consider saddle points). Therefore, to identify the least dispersive coherent sets, is sufficient to focus on the eigenfunctions in Eqs. (17). The pair of minimally leaking coherent sets is given by the solution of Eqs. (T8) with X closest to unity, To obtain the coherent set membership indices, we recover indicator functions, Eq. by taking the sign of these singular functions.
Eigenvectors Vi and u, play complementary role in prediction and retrospection. As the operator F, propagates observables forward then backward in time, neurons projecting inputs onto v, are predictive (Fig. 1b). As the operator B, propagates the observable backward then forward in time, neurons projecting inputs onto I1 are retrospective (Fig. 1c). In the case of 2DD saddle-poin QU (Fig. Ib,c), when the shared past and future refer to other fixed points, the coherent sets of interest are the expanding coherent set for prediction (v1) and the contracting coherent set for retrospection (u2). In Section the Supplement, we analytically prove that, as the forecast horizon goes to infinity, singular vectors 1 and U2 converge to the top and bottom left eigenvector of A which are orthogonal to the stable and unstable invariant subspaces, respectively.
So far we considered fully observable OU processes. However, in reality, the OU process is often only partially observed for example via linear projection of the state onto scalar variable,
y(t) C'((t).
(19)
Neurally, such projection can be an input to single-synapen neuron or a total synaptic current. Assuming observability, the state can be represented by an n-dimensional lag vector 49 50 51
x() (v((), y(t 1), ..... rlt n + =)]",
(20)
evolving via a companion matrix, equivalent to the following auto-regressive model: y(t + 1) = a (t) + azy(t 1) + + any(t 1) ((t))
(21)
4 Coherent sets from data
The infiniteddimensional stochastic Koopman operator (SKO) can be approximated by a Galerkin
projection onto finite set of basis functions (or features), { o (x) 1d 13 In the data-driven formulation, the SKO matrix representation under this basis is 18 (see Supplement, Section 5):
K, = EE
(22)
where Eg and Σx are the covariance matrices:
Eoly Σ (x())()))
E ~ S Σ (X(()))) +r)),
(23)
and describe the correlation between measurement functions and j measured instantancouslly or with delay т, respectively.
Such matrices are well defined if the process is stationary which may not be true for neurons. A common workaround is to assume local stationarity, that is, the distribution of inputs changes slowly compared to the timescale of the dynamics. In this case, the neuron estimate of the dynamics using exponential forgetting (52). Our method can accommodate this type of non-stationarity by computing Eqs. (23) locally in time, using temporal filter that discounts older observations. This allows the estimated covariances to track gradual changes in the underlying distribution without assuming global stationarity.
continuously update the
En K, Ep. The forward and As a direct consequence of Eq. ( the adjoint of K is KJ- backward operators have matrix representations [8
F, = K, K Σ.
B, En K, Σ,K, = E (24)
which reduce to Eq. 18) for attractive OU processes (see the Supplement, Section 6). The singular functions of the SKO within the Galerkin projection are given by
(x) = U, -(())
(x(x) = -.: xx())
(25)

---

where i7 and üi are eigenvectors of the matrices F, and B,, respectively. Notice that i7 and dī are precisely the solution of past-future CCA.
By representing these eigenvectors in synaptic weights and temporal filters, a neuron may compute a leading singular function of the SKO restricted to the linear span of its input features. By indicating the sign of the singular function, the neuron then computes the membership index for each input. For the sake of simplicity, below we only consider the temporal component of the singular vector. Even if the neuron has access to only a single synapse, it can retain a history of that synapse's ' activity over consecutive time steps. This construction known as delay embedding or delay coordinates also yields a Galerkin approximation [49 [5]]. We take x E IR to be a lag vector of the input current to the neuron over the previous d intervals of time, such that (t) (I(t), (t--tt).
[[t ((t-(d- (d-1)At)). where I (t) is the observed current at time t. In this regime, the basis functions are given by
(x(x) = Zi
(26)
i.e. ((x) is the identity function. Singular vectors act as temporal filters over the delayed signal, to
d-1
pre (1) = H (- (x(())) = H
Tt
(t) = (=u, H
[T.le/(t [tlle(tt-Aa kLt))
(27)
where ūi and 07 are singular vectors (cf. Eq. 25 ) and the Heaviside function, (x) 1 for x and (z) = 0 for x < O.
Therefore, sensory streams can be clustered into coherent sets via datt--riven Galerkin approximations of the SKO. By learning the singular vectors from the features represented by the activity of the upstream neurons and encoding them in the synaptic weight and temporal filters, biological neurons can cluster inputs using integrate-and-fing dynamics.
5 Coherent set clustering perspective on neurophysiology
Here we apply the coherent set clustering framework to biological neurons focusing on three ob- servations: temporal receptive fields, neuronal rectification and predictive/retroseettie properties. We restrict our consideration to the processing of a scalar time series viewed as a one-dimensional projection of a multidimensional state-space dynamics, Eq. (27). Such a scalar time series could represent an input to single-input neuron or the total synaptic current into a multi-input neuron. Here, inspired by our result for OU processes that the relevant singular functions lie in the span of lag vector basis, we choose Eq. (26) even in the data-driven, non-linear setting. Even such simple basis choice produces tangible results.
5.1 Temporal receptive fields
We consider early visual processing where a natural stimulus can be generated by emulating the movement of the retinal image due to self-motion or saccades by scanning a natural image or its model. In turn, natural images are commonly modeled by the "dead leaves" model partitioning the space into patches of different but uniform luminance with sharp transitions between them [53 (4). Therefore, the resulting input time series a set of plateaus at different levels with sharp transitions between them, Fig. 2a. Such scalar time series models input to post-photoreceptor neurons for invertebrates such as Drosophila, where initial processing is segregated between adjacent "pixels" or total current in a vertebrate bipolar cell.
We view this scalar time series as linear projection of a high-dimensional dynamical system. Thus, we perform fast-future CCA on the lag- vectors formed from the scalar time series. The canonical correlations reveal a spectral gap following the top two. The lag- lag-vector space partitioned into a pair of coherent sets in a canonical direction. The top right singular vector amounts to low-pass filter (Fig. 2b) similar to sustained bipolar cells of the vertebrate retina or L3 neuron in Drosophila (55). Intuitively, this reflects the slow variation of luminance the plateaus, so the near future is expected to resemble the recent past. The second canonical direction acts as a smoothed temporal derivative (Fig. 2b) in general agreement with experimentally reported filters of transient bipolars in vertebrates,
7

---

Fig. 2c or LI and L2 cells in Drosophila 28|55 Because the stimulus is symmetric with respect to time-reversal, the complementary left singular vectors are obtained by simply inverting the time axis (and inverting the sign for the second singular vector). Intuitively, the second canonical direction reflects the inertia of transitions- an edge traversing the photoreceptor aperture is unlikely to reverse direction-ane and predicts the future by extrapolating via the temporal derivative. For results of this algorithm applied to different stimuli, see [26).
We further characterize the experimentally measured temporal responses by interpreting them from the perspective of predictive and retrospective coherent sets. To make this connection, we take advantage of the observation that practical predictive filter must be significantly aligned with an unstable cigendirection and a practical retrospective filter must be significantly aligned with stable eigendirection, Fig. lb,c. Moreover, for a two- woo-dimensional (hyperbolic) saddle point, the linear temporal filter of the predictive neuron must be orthogonal to the attractive subspace, while the linear temporal filter of the retrospective neuron must be orthogonal to the repulsive subspace, Fig. lb,c, see Section and the Supplement, Section Repulsive modes in the lag- lag-vector space are expanding exponentials in real time and attractive modes are contracting exponentials, suggesting simple way to interpret linear temporal receptive fields by computing the cosine similarity, S, with exponentials or, equivalently, computing normalized Laplace transforms, Fig. 2c,d. The experimentally measured temporal filter is orthogonal to a contracting exponential and aligned with the expanding exponential (Fig. 2c,d), indicating a predictive neuron. For details, see the Supplement, Section 7.
(d) 100
5 10
10
10
-0.5
0.0
-0.4
-0.2
-20 0
20
Time
Time lag
Time
Figure 2: Data-driven temporal filters learned the scalar time series from the natural image model compared with that of a bipolar cell. (a) Stimulus produced by the "dead leaves" model of natural images
3 (green), stimulus filtered through first (dashed black) and second (solid black) right singular vectors. (b) First (dashed) and second (solid) right singular vectors from Eq. (24) corresponding to predictive neurons. (c) Black: experimentally measured flash response of the salamander retinal bipolar cell (D.B. Kastner & S.A. Baccus, personal communication) on the inverted time axis approximates its linear filter. Compare this with with the second singular vector (solid) in b. Note that the decay of the filter at zero time delay (absent in the theoretical result without additional constraints) is consequence of causality and continuity of the filter implementation by biological system. The maximally aligned (red) and the orthogonal (blue) exponentials are overlaid. (d) Cosine similarities of the temporal filters with a battery of exponentials. Blue dot indicates orthogonality and red triangle indicates maximum alignment. Closer alignment with expanding exponentials and orthogonality to a contracting exponential indicates the neuron's predictive nature.
5.2 Neuronal rectification exemplified by ON and OFF cells
Our coherent set clustering perspective suggests that, in the vicinity of dynamical saddles, neurons determine membership indices by applying a Heaviside step-function to the SKO singular function (or its sign-inverse), 27) Such non-linearity could be naturally implemented by the spiking mechanism and correspond to the ON and OFF ganglion cells of the vertebrate retina, named this way because they respond to luminance increments and decrements, respectively. However, neurons post-synaptic to photoreceptors, bipolar cells in vertebrates and large monopolar cells in flies, are non- spiking. Neuronal activity is represented by continuously varying graded potentials that determine a non-negative synaptic vesicle release rate. Many such neurons thus rectify their input and are also classified as either ON or OFF cell. Such rectified response to luminance variation is graded and could be viewed as soft clustering by using not only the sign but also the magnitude of the singular function, Fig. 1, resulting in a rectified linear unit (ReLU)-like operation (26).
5.3 Predictive and retrospective properties of biological neurons
In this Subsection, we analyze and review large datasets of experimentally measured temporal receptive fields from the perspective of predictive and retrospective coherent sets.
8

---

Tufted and mitral cells of the mammalian olfactory bulb. In mammals, information from olfactory sensory neurons relayed to the rest of the brain by two neuron classes: tufted cells (TCs) and mitral cells (MCs). We analyzed a dataset of the temporal receptive fields of 204 TCs and MCs recorded blindly from the rat olfactory bulb [56] from the coherent set clustering perspective. As described above (5.1, Temporal Receptive Fields), we computed the cosine similarity of experimentally mea- sured temporal filters with a battery of growing and decaying exponentials. In addition to identifying the orthogonal exponential, we also identified the sign of the exponent which has the highest cosine similarity with the temporal filter: positive exponents correspond to predictive neurons, and negative to retrospective. As result, we found mixture of predictive and retrospective properties, Fig. 3. Approximately half of the recorded cells had temporal filters orthogonal to decaying exponentials and, therefore, were likely predictive. About 4% were orthogonal to growing exponentials and, hence, were likely retrospective. About one-third were orthogonal to both growing and decaying exponentials suggesting that they analyze higher- than 2-dimensional dynamics and could be either retrospective or predictive. See the Supplement, Section 7 for details.
We speculate that TCs and MCs are mostly predictive and retrospective, respectively, based on the existing literature. First, they respond at different phases of the sniff cycle as monitored by air flow [57 TCs are preferentially active during the exponential growth phase and silent during the exponential decay phase. Conversely, MCs are preferentially active during the exponential decay phase and silent during the exponential growth phase. Second. MC responses lag relative to TCs suggesting they play a different role [57] 58 59 60), but see [6] Finally. TCs and MCs differ in their responses to synaptic inputs from the clectrophysiologicaly stimulated olfactory sensory neurons [3]] TCs receive direct inputs and respond with an immediate biphasic profile typical of prediction, while MCs are activated indirectly with delay characteristic of retrospection.
a
Tufted/Mitral Cells
Fiber Vatue (AU)
Filer of Rea Vilae (AU)
TU
6
10
10
of 0r0 Vigu
10
2.
413
40.5
4 Time (8)
(5)
TuftediMitral
Retinal Ganglion
100%
3115
Algned
Neurrne
10%5
Nourtons
10%
A
89L
10
Balt
00 of 1 span
Orthogonellc
Orthogonaa
Figure 3: Temporal receptive fields of olfactory bulb and retinal neurons interpreted through the lens of coherent sets exhibit predictive and retrospective properties. (a) Linear temporal filters (black) from three rat olfactory bulb tufted/mitral neurons [56). Solid colored lines are orthogonal exponentials; dotted colored lines are aligned exponentials. Top insets show the cosine similarity, S, of the temporal filter with exponentials. Orthogonal and aligned values are marked by dots and triangles, respectively. Bottom insets show convolution of the filter with a step pulse. The first neuron orthogonal to a decaying and aligned with expanding exponentials and interpreted as predictive. The second neuron orthogonal to an expanding and aligned with decaying exponentials and interpreted as retrospective. The third neuron is orthogonal to both a decaying and expanding exponential but aligned to an expanding one. (b) Left, for each tufted/mmitall cell, the a value yielding an orthogonal exponential is plotted against the that is maximally aligned. Filled dots indicate neurons orthogonal to single Open dots indicate neurons orthogonal to two a values. Right, The proportion of cells orthogonal to negative exponential only, positive exponential only, both, or neither is shown in the bar graph. 95% confidence intervals of the mean obtained through a binomial fit are plotted as black lines. (c) Same as b but for retinal ganglion cells (RGCs). See Fig. SI for example temporal receptive fields. Note that very few RGCs are orthogonal to expanding exponentials.
Retinal ganglion cells (RGCs). To explore whether the distributions of predictive and retrospective neurons vary across sensory modalities, we analyzed a large dataset of temporal receptive fields of 1345 RGCs from dissociated vertebrate retina [64). Figs. S1, 3C, In vertebrates, signals from
9

---

differentially stimulated photoreceptors are combined in the retina and, therefore, an RGC's receptive field has a spatial component. Here, we only focus on the temporal component obtained by a rank-1 decomposition of the spatio-temporal receptive field. The majority of RGCs' receptive fields were orthogonal to decaying exponentials and aligned with expanding exponentials, hence predictive, though 1% of cells were retrospective, Fig. 3C. See the Supplement, Section 7 for details.
Lagged and non-lagged cells in the Lateral Geniculate Nucleus. RGCs project to the Lateral Geniculate Nucleus (LGN) [55, in which two classes of relay cells have been described in cat monkey 669 and mouse [70]: one with the temporal receptive fields of predictive cells and one with retrospective properties, Fig. S2. Because the firing of the latter class of neurons substantially lags a step stimulus onset, they were termed "lagged cells". Comparing the response profiles of our dataset of tufted/mitral cells to step stimulus provides insight into how lagged cells may manifest, Fig. 3A. Note that the middle, retrospective, neuron is initially inhibited by the positive step response, but overcomes this inhibition due to the delayed positive filter lobe. Thus, we identify non-lagged and lagged neurons with predictive and retrospective units, respectively.
666 67
6 Discussion
We propose that neurons act as coherent-sel detectors: each unit computes a linear projection of its input onto singular vector of a whitened finite-time transition matrix and then takes the sign or rectifies its positive or negative part. We derive this mechanism analytically for saddle-poin Ornstein-Ullente
(OU) dynamics and suggest that it extends to nonlinear systems via Galerkin projection on a chosen feature basis. In practice, leading singular function can be learned directly from data, providing a path toward biologically plausible implementations (e.g., local weight updates). This perspective captures several physiological regularities, including rectification and selective sensi- tivity to temporally extended neuron types.
structure in stimuli as well as predictive and retrospective
Open issues:
(i) Unstable modes and reference measures. Section 3 characterizes SKO singular functions for the OU model, but linking them to the spectral clustering framework of Section under instability requires reference measure that decays away from the saddle neighborhood. Practically, this sug- gests finite-time localization (windowing), exponential discounting, or commitor/Doobb-taassfoon weights so that the inner product prioritizes trajectories that remain near the saddle. Interpreting the OU model as local description then would render the spectral objects well posed on that neighborhood.
(ūi) Data-driven estimation along unstable directions. The Galerkin procedure of Section 4 matches the OU analysis for stable modes; extending it to unstable modes likely requires modified for- ward-backward compositions over short horizons so that finite-time expansion (singular values (>1)) is preserved rather than suppressed by whitening. Establishing a rigorous equivalence in this regime remains open.
(iti) Beyond 2D saddles. The correspondence between finite-time singular vectors and eigenvectors shown for 2D saddles may not extend to higher dimensions, especially with multiple unstable direc- tions. Systematic analysis in dimensionnll settings is needed.
(iv) Circuit feedback loops. While many early sensory pathways are predominantly feedforward and fall within framework, feedback is ubiquitous in deeper circuits. Incorporating explicit closed-loop interactions-for example, via the controller perspective of could unify coher- ent-set detection with action selection and behavior generation.
(v) Intra-neuronal jetre--aurrnaleeeddbc.. feedback. In our current account, rectification follows linear projection and does not influence learning of synaptic weights or temporal filters. In reality, synaptic plasticity may depend on rectified output via dendritic backpropagation. Incorporating such output into projection learning could provide the needed windowing for local learning mechanisms.
Societal impact. By linking biological neural computation to finite-tima dynamical structure, this work advances our understanding of brain function and may inform approaches to mental health and neurological disorders. It could also inspire biologically grounded, sulf--uper architectures for artificial neural networks operating in dynamical settings.
Acknovledgments. We are grateful to T. Ahamed, R. Cosentino, A. Genkin, B. Gutkin, J. M. Heninger, P. Kidd, A. Koulakov, I. Nemenman, S.E. Palmer, S. Qin, D. Rinberg, E. Schneidman, D.
10

---

Schwab, E. Simoncelli, S. Solla, P. Thomas and, especially, P. Cvitanovic for insightful discussions, P. Gupta, D.F. Albeanu, D.B. Kastner, and S.A. Baccus for sharing their datasets. This research was supported in part by the National Science Foundation (NSF) EFRI BRAID: Scalable-L Learning Neuromorphies, award No. 2318152. Some work was initiated and performed at the the Kavli Institute for Theoretical Physics (KITP) supported by grant NSF PHY-2309133 and at the Aspen Center for Physics, which supported by NSF grant PHY-2210452.
References
[1] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[2] William Bialek, Rob R De Ruyter Van Steveninck, and Naftali Tishby. Efficient representation as design principle for neural coding and computation. In 2006 IEEE international symposium on information theory, pages 659-663. IEEE, 2006.
[3] Stephanie Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in sensory population. Proceedings of the National Academy of Sciences, (12(22):6908-6911)
2015.
[4] Matthew Chalk, Olivier Marre, and Gašper Tkačik. Toward unified theory of efficient, predictive, and sparse coding. Proceedings of the National Academy of Sciences, 115(1):186- 191, 2018.
[5] Fred Attneave. Some informational aspects of visual perception. Psychological review, 61(3):183, 1954.
[6] Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01):217-233, 1961.
[7] Eero Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual review of neuroscience, 24(1):1933-1216 2001.
[8] Michael Dellnitz and Oliver Junge. On the approximation of complicated dynamical behavior. SIAM Journal on Numerical Analysis, 36(2):491-515, 1999.
[9] Gary Froyland. An analytic framework for identifying finite-time coherent sets in time- dependent dynamical systems. Physica D: Nonlinear Phenomena, 250:1-19 2013.
[10] Igor Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41(1-3):309-325, August 2005.
[11] Clarence Rowley, Igor Mezić, Shervin Bagheri, Philipp Schlatter, and Dan Henningson. Spectral analysis of nonlinear flows. Journal of fluid mechanics, 641:115-127, 2009.
[12] Steven L Brunton and J Nathan Kutz. Data-driven science and engineering: Machine learning, dynamicul systems, and control. Cambridge University Press, 2022.
[13] Stefan Klus and Nataša Djurdjevae Conrad. Dynamical systems and complex networks: A koopman operator perspective. Journal of Physics: Complexity, 5(4)004101,, 2024.
[14] Hao Wu and Frank Noé. Variational approach for learning markov processes from time series data. Journal of Nonlinear Science, 30(1):23-66, 2020.
[15] Wikipedia contributors. Ornstein-ullenbed process - Wikipedia, the free encyclopedia, 2025. [Online; accessed 15-May-2025).
[16] Henry C Tuckwwel.. Stochastic partial differential equations in neurobiology: linear and nonlinear models for spiking neurons. In Stochastic biomathematical models: with applications to neuronal modeling, pages 149-173, Springer, 2012.
[17] Alain Destexhe, M Rudolph, J Fellous, and J Sejnowski. Fluctuating synaptic conductances recreate in vivo-like activity in neocortical neurons. Neuroscience, 107(1):13-24, 2001.

---

[18] Péter Koltai, Hao Wu, Frank Noé, and Christof Schütte. Optimal data-driven estimation of generalized markov state models for non-equiiibrum dynamics. Computation, 611):2,, February 2018.
[19] David Lipshutz, Yanis Bahroun, Siavash Golkar, Anirvan M Sengupta, Dmitri B Chklovskii, and others many. A biologically plausible neural network for multichannel canonical correlation analysis. Neural Computation, (3(9))2309-23355 2021.
[20] Jonathan Tu. Dynamic mode decomposition: Theory and applications. PhD thesis, Princeton University, 2013.
[21] Matthew O. Williams, Clarence W. Rowley, and loamnis G. Kevrekidis. A kesel--based method for data-driven koopman spectral analysis. Journal of Computational Dynamics, 2(2):247-265. 2015.
[22] Byron H Price and Jeffrey P Gavornik Efficient temporal coding in the early visual system: existing evidence and future directions. Frontiers in Computational Neuroscience, 16:929348, 2022.
[23] Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for gaussian variables. Advances in Neural Information Processing Systems, 16, 2003.
[24] Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4):715-770, 2002.
[25] David Lipshutz, Charles Windolf, Siavash Golkar, and Dmitri B Chklovskii. A biologically plausible neural network for slow feature analysis. Advances in Neural Information Processing Systems, 33:14986-14999 2020.
[26] Shanshan Qin, Joshua I. Bighe-Sanford, Alexander Genkin, Pembe Gizem Ozdil, Philip Green- gard, Anirvan Sengupta, and Dmitri B Chklovskii. A network of biologically inspired rectified spectral units (resus) learns hierarchical features without error backpropagation. arXiv
preprint arXiv:2512.23316. 2025.
[27] Karol Gregor and Dmitri Chklovskii. A lattice filter model of the visual pathway. Advances in Neural Information Processing Systems, 25, 2012.
[28] Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: fresh view of inhibition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences, 216(1205):427-444
1982.
[29] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.
[30] Shaul Druckmann, Tao Hu, and Dmitri Chklovskii. A mechanistic model of early sensory processing based on subtracting sparse representations. Advances in Neural Information Processing Systems, 25, 2012.
[31] Julijana Gjorgjieva, Haim Sompolinsky, and Markus Meister. Benefits of pathway splitting in sensory coding. Journal of Neuroscience, 34(36): 2014.
[32] Dawei W Dong and Joseph J Atick. Temporal decorrelation: theory of lagged and nonlagged responses in the lateral geniculate nucleus. Network: Computation in Neural Systems, 6(2):159- 178. 1995.
[33] Predrag Cvitanović and Domenico Lippolis. Knowing when to stop: How noise frees us from determinism. In AIP Conference Proceedings, volume 1468, pages 82-126. American Institute of Physics, 2012.
[34] Jeffrey M Heninger, Domenico Lippolis, and Predrag Cvitanović. Neighborhoods of periodic orbits and the stationary distribution of noisy chaotic system. Physical Review E, 9266)222222, 2015.
12

---

[35] Gary Froyland, Naratip Santitissadeekorn and Adam Monahan. Transport in time- deneddeendent dynamical systems: Finite-time coherent sets. Chaos: An Interdisciplinary Journal of Nonlinear Science, 20(4), 2010.
[36] Siavash Golkar, Jules Berman, David Lipshutz, Robert Mihai Haret, Tim Gollisch, and Dmitri B Chklovskii. Neuronal temporal filters as normal mode extractors. Physical Review Research, 6(1):01311,, 2024.
[37] Cengiz Pehlevan, Alexander Genkin, and Dmitri B Chklovskii. A clustering neural network model of insect olfaction. In 2017 51 Asilomar Conference on Signals, Systems, and Comput- ers, page 593 600. IEEE, IEEE, 2017.
[38] Wei Ma, Jeffrey M Beck, Peter Latham, and Alexandre Pouget. Bayesian inference with probabilistic population codes. Nature neuroscience, 911)):133-1445 2006.
[39] Yosef Singer, Yayoi Teramoto, Ben DB Willmore, Jan WH Schnupp, Andrew J King, and Nicol S Harper. Sensory cortex is optimized for prediction of future input. Elife, 7:e31557, 2018.
[40] Marko Budišić, Ryan Mohr, and Igor Mezić. Applied koopmanism. Chaos: An Interdisciplinary Journal of Nonlinear Science, 22((), 2012.
[41] Jinho Choi. Sivaram Krishnan, and Jihong Park. Koopman autoencoder via singular value decomposition for data-driven long-term prediction. In 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6. IEEE, 2024.
[42] Andreas Mardt, Luca Pasquali, Hao Wu, and Frank Noć. Vampnets for deep learning of molecular kinetics. Nature communications, 9(1):5, 2018.
[43] Igor Mezić and Stephen Wiggins. A method for visualization of invariant sets of dynamical systems based on the ergodic partition. Chaos: An Interdiscipliuary Journal of Nonlinear Science, (1):2133-211 218, 1999.
[44] Samuel E Otto and Clarence W Rowley. Koopman operators for estimation and control of dynamical systems. Annual Review of Control, Robotics, and Autonomous Systems, 4(1):59-87, 2021.
[45] Grigorios Pavliotis. Stochastic processes and applications. Texts in applied mathematics, 60, 2014.
[46] Rajendra Bhatia and Peter Rosenthal. How and why solve the operator equation xb= y. Bulletin of the London Mathematical Society, 29(1):1-21, 1997.
[47] Chulan Kwon, Ping Ao, and David Thouless. Structure of stochastic dynamics near fixed points. Proceedings of the National Academy of Sciences, 102(37):13029-1333 2005.
[48] Oscar Bandtlow, Wolfram Just, and Julia Slipantschuk. Spectral structure of transfer operators for expanding circle maps. In Annales de I'Institut Henri Poincaré C, Analyse non linéaire. volume 34, pages 31 31-43. Elsevier, 2017.
[49] Igor Mezić and Andrzej Banaszuk. Comparison of systems with complex behavior. Physica D: Nonlinear Phenomena, 1011133) 2004.
[50] Steven L Brunton, Bingni W Brunton, Joshua L Proctor, Eurika Kaiser, and Nathan Kutz. Chaos an intermittently forced linear system. Nature communications, 8(1):19, 2017.
[51] Tohru Katayama. Subspace methods for system identification. Springer, 2005.
[52] Bernard Widrow and Samuel D Stearns, Adaptive Signal Processing. Prentice-Hall Englewood Cliffs, NJ, 1985.
[53] D Jeulin. Dead leaves models: from space tesselation to random functions. In Symposium on the Advances the Theory and Applications of Random Sets, pages 137-156. World Scientific, 1996.
13

---

[54] Daniel L Ruderman. Origins of scaling in natural images. Vision research, 37(23):3385-3399 1997.
[55] Madhura D Ketkar, Burak Gür, Sebastian Molina-Obando, Maria Ioannidou, Carlotta Martelli, and Marion Silies. First-order visual interneurons distribute distinct contrast and luminance information across on and off pathways to achieve stable behavior. Elife, 11:e74937, 2022.
[56] Priyanka Gupta, Dinu F Albeanu, and Upinder S Bhalla. Olfactory bulb coding of odors, mixtures and sniffs is linear sum of odor time profiles. Nature neuroscience, 18(2):272-28) 281, 2015.
[57] Izumi Fukunaga, Manuel Berning, Mihaly Kollo, Anja Schmaltz, and Andreas T Schaefer. Two distinct channels of olfactory bulb output. Neuron, 75(2):320-329, 2012.
[58] Nixon M Abraham, Hartwig Spors, Alan Carleton, Troy Margrie, Thomas Kuner, and Andreas T Schaefer. Maintaining accuracy at the expense of speed: stimulus similarity defines odor discrimination time in mice. neuron, 44(5):865 865-876, 2004.
[59] Naoshige Uchida and Zachary Mainen. Speed and accuracy of olfactory discrimination in the rat. Nature neuroscience, 6(11):1224-122) 2003.
[60] Dmitry Rinberg, Alexei Koulakov, and Alan Gelperin. Speed-accuracy tradeoff in olfaction. Neuron, 51())3551-358 2006.
[61] Marta Díaz-Quesaan Isaac A Youngstrom, Yusuke Tsuno, Kyle R Hansen, Michael N Economo, and Matt Wachowiak. Inhalation frequency controls reformatting of mitral/tufted cell odor representations in the olfactory bulb. Journal of Neuroscience, 38(9):2189-222) 2018.
[62] Shaina M Short and Matt Wachowiak. Temporal dynamics of inhalation- linked activity across defined subpopulations of mouse olfactory bulb neurons imaged in vivo. eneuro, 6(3), 2019.
[63] David H Gire, Kevin M Franks, Joseph D Zak, Kenji F Tanaka, Jennifer D Whitesell, Abigail A Mulligan, René Hen, and Nathan E Schoppa. Mitral cells in the olfactory bulb mainly excited through a multistep signaling path. Journal of Neuroscience, 32(9):2964-299 2012.
[64] David B Kastner and Stephen A Baccus. Coordinated dynamic encoding in the retina using opposing forms of plasticity. Nature Neuroscience, 14(10)1311-11322 2011.
[65] Daqing Cai, Gregory C Deangelis, and Ralph Freeman. Spatiotemporal receptive field organization in the lateral geniculate nucleus of cats and kittens. Journal of neurophysiology. 78(2):1045-1011, 1997.
[66] David N Mastronarde. Two classes of single-input x-cells cat lateral geniculate nucleus. i. receptive-field properties and classification of cells. Journal of neurophysiology 57(2):357-380, 1987.
[67] Jonathan Wolfe and Larry A Palmer. Temporal diversity in the lateral geniculate nucleus of cat. Visual neuroscience, 15(4):653-66) 675, 1998.
[68] Alan B Saul and AL Humphrey. Spatial and temporal response properties of lagged and nonlagged cells in cat lateral geniculate nucleus. Journal of neurophysiology. 64(1):206-224, 1990.
[69] Alan B Saul. Lagged cells in alert monkey lateral geniculate nucleus. Visual neuroscience, 25(5-6):647-655 2008.
[70] Denise M Piscopo, Rana N El-Danaf, Andrew D Huberman, and Cristopher M Niell. Di- verse visual features encoded in mouse lateral geniculate nucleus. Journal of Neuroscience, 33(11):4642-400 2013.
[71] Jason J Moore, Alexander Genkin, Magnus Tournoy, Joshua Pughe-Sanford, Rob R de Ruyter van Steveninck, and Dmitri Chklovskii. The neuron as a direct data-driven controller. Proceedings of the National Academy of Sciences, 121(27):e3111893122 2024.
14

---

NeurIPS Paper Checklist
422
423
1.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
425
428
Answer: [Yes]
Justification: The abstract and introduction closely and accurately describe the content of the paper.
428
429
Guidelines:
430
The answer NA means that the abstract and introduction do not include the claims made the paper.
431
432
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
433
434
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
436
437
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
438
2.
Limitations
440
Question:
Does the paper discuss the limitations of the work performed by the authors?
441
Answer: [Yes]
442
Justification: While conceptualizing neurons through coherent set clustering offers a novel perspective on neural computation, this framework does not yet account for the role of feedback. In photoreceptor neurons, rectification output appears not to influence synaptic learning, but in other neuron types, learning may depend critically on dendritic backpropa- gation. Furthermore, the widespread presence of feedback loops in brain circuits highlights the need to incorporate such mechanisms into the broader dynamical systems framework.
443
444
446
447
448
Guidelines:
449
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
451
The authors are encouraged to create separate "Limitations" section in their paper.
452
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
453
454
455
456
457
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
459
480
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
461
462
463
464
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
466
467
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
468
469
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
470
471
473
13

---

475
3.
Theory assumptions and proofs
478
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
477
478
Answer: [Yes]
Justification: Proofs are provided in Section 3 and the methods.
Guidelines:
481
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
483
484
All assumptions should be clearly stated or referenced in the statement of any theorems.
485
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
486
488
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
489
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
492
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
483
484
495
Answer: [Yes]
496
Justification: Simulation parameters and pseudocode for Figure are provided in the supplemental material. The procedure to arrive at results in Figure 3 is detailed in the supplemental material.
498
499
Guidelines:
500
The answer NA means that the paper does not include experiments.
501
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
503
504
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
505
506
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
508
509
510
512
513
514
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
516
518
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
519
520
(c) If the contribution is new model (c.g., s. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
823
625
526
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility.
14

---

528
In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
529
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
533
Answer: [NA]
Justification: The simulations and analyses performed can be described with words, and are described in the supplemental material.
537
538
Guidelines:
539
The answer NA means that paper does not include experiments requiring code.
540
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
541
for more details.
542
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
544
545
546
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
548
ctipss.c//ublic//uuii guwpe//deeeeeeee CooeeSumiiai issiiissioon onyolicy) for more details.
519
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
550
551
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
553
554
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
555
556
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
$59
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
580
582
Answer: [NA]
583
Justification: No training or testing was performed.
584
Guidelines:
585
The answer NA means that the paper does not include experiments.
566
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
568
The full details can be provided either with the code, in appendix, or as supplemental material.
589
570
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
572
573
Answer: [Yes]
574
Justification: Error bars Figure 3C indicate 95% confidence intervals of the mean, obtained through binomial fit. No statistical tests were performed.
575
576
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

578
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
579
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
582
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
585
S86
The assumptions made should be given (e.g.. Normally distributed errors).
587
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
586
589
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
591
592
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
584
595
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
598
8. Experiments compute resources
598
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
599
600
601
Answer: [Yes]
602
Justification:
The modest compute requirements are described in the supplemental material.
Guidelines:
604
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
606
607
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
609
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
610
612
9. Code of ethics
613
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
616
Justification: Our work conforms in every respect with the NeurIPS Code of Ethics
617
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
619 If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
620
621
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
622
10.
Broader impacts
624
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
625
626
Answer: [Yes]
Justification: Discussed at the end of the paper. The work is foundational research that can facilitate our understanding of the brain and facilitate designing biologically inspired neural networks.
628
16

---

630
Guidelines:
631
The answer NA means that there no societal impact of the work performed.
632
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
633
634
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
636
638
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
639
640
642
643
644
645
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
647
649
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
650
651
652
653
11. Safeguards
654
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
656
657
Answer: [NA]
Justification: Our paper poses such risks.
659
Guidelines:
660
The answer NA means that the paper poses no such risks.
661
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
662
683
664
665
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
688
689
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
672
673
674
Answer: [Yes]
875
Justification: Datasets from two previously published studies were used to create Figure 3. These papers appropriately cited, but the datasets were obtained directly from their creators, and are not publicly licensed assets.
677
678
Guidelines:
The answer NA means that the paper does not existing assets.
680
The authors should cite the original paper that produced the code package or dataset.
681
The authors should state which version of the asset used and, possible, include URL.
17

---

683
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
685
688
If assets are released, the license, copyright information, and terms of use in the bataasess.paerssttt paperswi thcode. cow condsatases package should be provided. For popular datasets,
687
688
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
689
690
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
692
If this information is available online, the authors are encouraged to reach out to the asset's ss creators.
693
694
13. New assets
695
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
687
Answer: [NA]
698
Justification: This paper does not release new assets.
899
Guidelines:
700
The answer NA means that the paper does not release new assets.
701
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
703
704
The paper should discuss whether and how consent was obtained from people whose asset used.
705
706
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14.
Crowdsourcing and research with human subjects
709
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
710
711
712
Answer: [NA]
713
Justification:
We do not crowdsource nor use human subjects.
714
Guidelines:
715
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
716
717
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
718
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
721
722
15.
Institutional review board (IRB) approvals equivalent for research with human subjects
724
725
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
727
729
Answer: [NA]
730
Justification:
We do not crowdsource nor use human subjects.
731
Guidelines:
732
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

734
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
735
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
738
740
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
742
16.
Declaration of LLM usage
743
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
745
747
Answer: [NA]
748
Justification:
The core development in this research does not involve LLMs in any aspect.
749
Guidelines:
750
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
753
19


Reviews:
[{"review_id": "V2lWWBWZSR", "reviewer": "Reviewer_AD8g", "summary": "The authors propose a novel perspective on the functional role of sensory neurons, as \"coherent set detectors\" that identify regions of the stimulus space that evolve cohesively over time. Through modelling neurons as latents underlying stochastic dynamical systems governing the dynamics of sensory stimuli, the authors show that neurons can detect coherent sets through spectral clustering of the stochastic Koopman operator underlying the dynamic system. For OU processes, the authors show that the subdominant singular basis functions reduce to linear projections. It is additionally possible to extend the framework to nonlinear dynamics with Galerkin projections and CCA for interpreting real neural data.", "questions": "- How could neurons biologically learn the required singular vectors? The whitening step requires global covariance - how is this reconciled with local learning rules?", "limitations": "See above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths.\n- The authors propose a novel theoretical framework connecting dynamical systems theory to neural computation through coherent set analysis, which is an under-explored question in understanding functions of sensory neurons.\n- The derivation for OU processes is technically sound, with clear progression from the Koopman operator to implementable neural operations.\n- The paper is well-written, with clear demonstrations and easy-to-follow derivations, despite the mathematical complexity.\n\nWeaknesses.\n- The extension to non-linear dynamics is largely data-driven, and largely base on the stationarity assumption, whereas real neural dynamics almost always violates such assumption.\n- Neuron classification based on orthogonality to exponentials lacks rigorous statistical testing, with no comparison to null hypothesis/model.", "was_revised": true, "final_justification": "Raised scores post-rebuttal as the authors have provided additional clarifications.", "rebuttal": "We thank the reviewer for taking the time to provide thoughtful feedback on our manuscript. Below, we address each of the reviewer’s comments in turn.\n\n*1. The extension to non-linear dynamics is largely data-driven, and largely based on the stationarity assumption, whereas real neural dynamics almost always violates such assumption.*\n\nWe agree with the reviewer’s point: non-stationarity presents a significant challenge for algorithmic models of neural activity. A common workaround is to assume local stationarity, that is, the distribution of inputs changes slowly compared to the timescale of the dynamics. In this case, we can continuously update the estimate of the dynamics using exponential forgetting (see, e.g., B. Widrow and S. D. Stearns, Adaptive Signal Processing, Prentice Hall, 1985). For example, visual input from ambient light may remain relatively stable while an agent is indoors, only shifting to a different distribution once the agent steps outside. Our method can accommodate this type of non-stationarity by computing Eqs. 19-20 locally in time, using a temporal filter that discounts older observations. This allows the estimated covariances to track gradual changes in the underlying distribution without assuming global stationarity. We will add this clarification to the manuscript. \n\n*2. Neuron classification based on orthogonality to exponentials lacks rigorous statistical testing, with no comparison to null hypothesis/model.*\n\nWe agree that more rigorous statistical validation is important and aim to incorporate such testing in future work. However, our analyses were restricted due to using previously published data. Neural data was supplied in the form of temporal receptive fields for Mitral/Tufted cells and Spatiotemporal receptive fields for RGCs, without a record of individual stimulus/response trials. RGC receptive fields were estimated by the original authors using reverse correlation of white noise. Receptive fields for Mitral/Tufted cells were estimated by the original authors by describing the best response to brief odor pulses. Given the structure of our current dataset, it is unclear what form of statistical testing would be both valid and informative. In particular, we lack measures of variability and cannot construct a meaningful null model. If the reviewer has specific suggestions for how to proceed despite these limitations, we would be happy to apply the proposed tests to our data. \n\nTo better validate this hypothesis, we plan to run the same analysis on filters generated using a lag-vector based Galerkin expansion of a simple OU process. We expect to see similar classes of filters that are orthogonal to exponentials at specific timescales set by the dynamics of the OU process. The insights gained from this analysis will be added to the revised main text.\n\n*3. How could neurons biologically learn the required singular vectors? The whitening step requires global covariance - how is this reconciled with local learning rules?*\n\nAn online CCA algorithm implemented by neural networks with biologically plausible local learning rules was previously derived by optimizing a similarity matching objective (Lipshutz et al., 2021). This approach jointly determines both the neuronal activity dynamics and synaptic update rules, while implicitly performing whitening. Future work will aim to extend this framework—originally developed for static CCA between two concurrent data streams, where sufficient statistics are stored in synaptic weights—to the setting of past-future CCA, in which the sufficient statistics take the form of spatiotemporal filters. We anticipate that incorporating temporal structure will not pose significant challenges, as it can be implemented locally within each neuron through the use of distinct ion channels with different time constants. Another difference from the original CCA formulation is that the neuronal output corresponds to a projection of only past inputs onto the canonical direction, with future inputs used exclusively for learning. However, a similar configuration has been addressed in the static case (Golkar et al., 2020), and we therefore do not expect it to present major difficulties.\n\nD Lipshutz, Y Bahroun, S Golkar, AM Sengupta, DB Chklovskii (2021) Neural Computation 33 (9), 2309-2352\n\nS Golkar, D Lipshutz, Y Bahroun, A Sengupta, D Chklovskii (2020) Advances in neural information processing systems 33, 7283-7295"}, {"review_id": "7aUT43JWLZ", "reviewer": "Reviewer_EbEi", "summary": "Stochastic processes are typically described by stochastic differential equations (SDEs) or (via the Fokker-Planck equation) a deterministic PDE for the probability density function.  However, they can also be characterized in terms of \"transfer operators\" (Koopman, Perron-Frobenius, etc.) for some fixed amount of time evolution tau (as opposed to the infinitesimal evolution given by the PDEs).  The advantage of these transfer operators is that they can be estimated directly from data: pick a finite set of basis functions; evaluate them at a set of samples X as well as their time-evolved (by tau) counterparts Y; and then compute all second-order statistics.  The approximate Koopman operator, e.g., corresponds to the normal equations (the optimal linear description of the evolution of x in this basis); the forward-backward operator amounts to applying the normal equations once in the forward and once in the backward directions.  Furthermore, the discrete-time, discrete-state counterpart to the SDE is a random walk on a graph; and consequently, spectral clustering methods have counterparts for SDEs, which are  derivable from the transfer operators.  Intuitively, the eigenfunction with the largest eigenvalue corresponds to the most coherent set of points under the trajectory.  Of interest is the *second* eigenfunction, which corresponds to a partition of the state space into the two most coherent regions (the first eigenfunction includes the entire state space, I think).\n\nThe present MS shows that, for Ornstein-Uhlenbeck (OU) processes (\"a reasonable model of summed input to a neuron\"), the second eigenfunctions of the forward-backward (F) and backward-forward (B) operators are linear projections of the state, which can be derived explicitly from the equations for the dynamics.  The F eigenfunction corresponds to expanding coherent sets, and therefore tells us about low-dimensional structure in the past (\"retrospective\"), whereas the B eigenfunction corresponds to contracting coherent sets and tells us about low-dimensional structure in the near future (\"predictive\").\n\nFor neurons to extract this structure, they must estimate the dynamics directly from the data, along the lines sketched above.  The authors propose that neurons project incoming data onto the second singular vector (as shown to be optimal for OU processes) via their vector of synaptic weights, and then rectify the result (or its additive inverse).  The magnitude of this scalar output indicates how strongly the input belongs to one (or the other) of the two coherent sets.  Neurons that project onto the F eigenvector are predictive; onto the B eigenvector, retrospective.\n\nFinally, the authors consider neural data, letting the basis functions be simply a 1D variable at a set of discrete delays.  A standard model of luminance input yields singular vectors that resemble the temporal filters of retinal cells in drosophila.  ON and OFF cells in the early visual system are explained as the rectified responses to the positive and negative projections of luminance.  Tufted and mitral cells in the olfactory bulb are interpreted as predictive and retrospective, respectively.  Consistent with this claim, temporal receptive fields for such cells are shown to be (in many cases) orthogonal to either growing or decaying exponentials (or, somewhat confusingly, both).  Something similar can be said of retinal ganglion cells, although almost all of these are predictive.", "questions": "- How are we to interpret cells with filters orthogonal to both a decaying and a growing exponential?\n - The purpose of the analytical result is (I take it) to show that the procedure is optimal for at least one plausible kind of stochastic dynamics.  But in the data-driven case, wouldn't one use the linear projection onto the second singular vector even without this proof?\n - The basis used in section 5 is sensible, but in general how does one choose the basis?  It seems that it would be difficult to compare against electrophysiological data without having first nailed this down.\n - Why use CCA in Section 5.2?  (It seems that would be slightly different from finding the eigenvectors of the matrices in Eqn. 22.)", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "This is a refreshingly different take on neural coding (at least to this reviewer's knowledge), and provides a powerful framework for interpreting the temporal filters of neurons that fits nicely with the electrophysiology of early sensory areas.  The explanation of rectification is also compelling (although perhaps more of a precising of existing intuition than a brand new idea).  The approach has the potential to be a very general explanatory principle for temporal receptive fields.  Nevertheless:\n\nWeaknesses:\n - I am not wholly convinced that the sets of points that cohere in state space are the right low-dimensional signal.  There is not much intuition provided about why (e.g.) the smoothed derivatives of Fig. 2B are good for extracting what is presumably the feature of interest, the luminance itself.  (How would the filters change for a different model of luminance?)\n - It would be nice to make a stronger connection to the historical theoretical work on this topic, especially the literature on efficient coding (as well as the information bottle in refs 1-4), particularly in the retina where the authors provide their own interpretation.  (See e.g. the section on \"Efficient Coding in Retina and Thalamus\" in https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2022.929348/full).  I realize the authors are short on space. \n - The MS is not exactly self-contained (this reviewer had to read ref. 10 first, but perhaps that's my fault).", "was_revised": true, "final_justification": "The authors propose a refreshingly different take on neural coding (at least to this reviewer's knowledge), and a powerful framework for interpreting the temporal filters of neurons that fits nicely with the electrophysiology of early sensory areas.  The explanation of rectification is also compelling (although perhaps more of a precising of existing intuition than a brand new idea).  The approach has the potential to be a very general explanatory principle for temporal receptive fields.  My remaining reservation is that the manuscript may not reach all of its intended audience (computational neuroscientists) in part because of its novelty; but the authors have proposed to rectify some of this in the supplement with a \"short pedagogical introduction to transfer operators.\"", "rebuttal": "We thank the reviewer for taking the time to provide thoughtful feedback on our manuscript. Below, we address each of the reviewer’s comments in turn.\n\n*1. I am not wholly convinced that the sets of points that cohere in state space are the right low-dimensional signal. There is not much intuition provided about why (e.g.) the smoothed derivatives of Fig. 2B are good for extracting what is presumably the feature of interest, the luminance itself. (How would the filters change for a different model of luminance?)*\n\nWhat we show with the example OU process is that for a locally linear dynamical system, this algorithm extracts the coherent sets of the full state of the system. The instantaneous luminance itself is not the full state of the system. Rather, it is the local dynamics of the luminance (encoded in the correlation of the lag vectors) that the neuron has to cluster. For instance, the same luminance value can initiate an ON edge if the luminance is locally increasing or an OFF edge if the luminance is locally decreasing. In fact, the two coherent sets in our example correspond to the ON and OFF edges which are important latent variables as they indicate the passing of an object boundary over the photoreceptors. The smoothed derivatives of Fig 2B are optimal for extracting that signal, when restricted to the linear setting, their exact shape adapts to the statistics of whatever (locally linear) model of luminance is provided. We will add a paragraph with this clarification to the text and plot the response of the model ON and OFF neurons to the luminance trace in Figure 2 to illustrate this point.\n\n*2. It would be nice to make a stronger connection to the historical theoretical work on this topic, especially the literature on efficient coding (as well as the information bottle in refs 1-4), particularly in the retina where the authors provide their own interpretation. (See e.g. the section on \"Efficient Coding in Retina and Thalamus\" in https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2022.929348/full). I realize the authors are short on space.*\n\nThanks for the reference! We will cite it in the revised manuscript. Efficient coding is a general principle that aims to maximize the mutual information between stimuli and neural responses. In contrast, the aim of our work and the information bottleneck method is to infer the information from past stimuli about future stimuli most effectively. We can show in Supplementary Material that the linear projection (but not rectification) in our method is equivalent to the information bottleneck method for Gaussian distributions and linear systems. They may not be equivalent for more complex situations. As suggested by the reviewers, we will re-write, expand and rename the Relationship to Other Work Section and place it after the Introduction.  In particular, we will address the work of Dong and Atick (1995) in the re-write of the Relationship to Other Work Section. Briefly, while their explanation of the lagged and non-lagged cells relies on non-linearity, ours does not.\n\n*3. The MS is not exactly self-contained (this reviewer had to read ref. 10 first, but perhaps that's my fault).*\n\nWe will add a section to the Supplementary Material providing a short pedagogical introduction to transfer operators in the context of drift-diffusion models. \n\n*4. How are we to interpret cells with filters orthogonal to both a decaying and a growing exponential?*\n\nSuch filters could correspond to a saddle point in a >2-dimensional space with 2 unstable directions. These filters would have to be orthogonal to the other unstable direction and a stable direction. \n\nTo better explicate this analysis in the manuscript, we plan to run an analysis of a known OU process in the lag vector coordinates, compute the CCA filters, and repeat the orthogonality analysis. The insights gained from this analysis will be added to the revised main text.\n\n*5. The purpose of the analytical result is (I take it) to show that the procedure is optimal for at least one plausible kind of stochastic dynamics. But in the data-driven case, wouldn't one use the linear projection onto the second singular vector even without this proof?*\n\nSolving OU analytically relates the subdominant singular function of the SKO (the sign of which partitions the phase space into coherent sets, so-called spectral clustering) and the subdominant singular vector which can be extracted from the data. Specifically, we show that the subdominant singular function is an inner product of the subdominant singular vector and the state. The Galerkin projection onto the feature basis, such as lag-vectors, generalizes this relation to nonlinear dynamics. \n \n*6. The basis used in section 5 is sensible, but in general how does one choose the basis? It seems that it would be difficult to compare against electrophysiological data without having first nailed this down.*\n\nFor a partially observed OU system, time-delay embedding with the lag-vector length greater or equal to the order of the system fully captures its dynamics, De Persis & Tesi (2019). Furthermore, we analytically proved that the subdominant singular function of an OU process lies in the space spanned by linear functions. When applying our method to early sensory processing, we compute such a basis from sensory stimuli and find it satisfactory. If the reviewer is asking about choosing a correct basis of stimulus features for more central neurons we agree that this is a difficult problem. Instead, we anticipate that the lag-vector constructed from the activity of upstream neurons would provide an appropriate basis.\n\n*7. Why use CCA in Section 5.2? (It seems that would be slightly different from finding the eigenvectors of the matrices in Eqn. 22.)*\n\nEigenvectors of the matrices in Eqn. 22 are equivalent to the canonical directions just like eigenvectors of the gramian of a matrix being equal to the singular vector of the matrix. We will clarify this connection in the revised manuscript.\n\nDong, D. W., & Atick, J. J. (1995). Network: Computation in neural systems, 6(2), 159.\n\nC. De Persis, P. Tesi, IEEE Trans. Autom. Control 65, 909–924 (2019)."}, {"review_id": "JnkMKSH6p4", "reviewer": "Reviewer_LTSZ", "summary": "This paper proposes that sensory neurons function as detectors of coherent sets, which are defined here as maximally stable binary partitions of the input state space. The paper first formalizes coherent sets in terms of the subdominant singular vectors of the stochastic Koopman operator. It then shows that for OU dynamics the optimal detector is a linear projection onto the dominant singular vector, followed by positive or negative rectifying (for detecting one set or its complement). For unknown and possibly nonlinear dynamics it then proposes a data-driven method based on past-future cross-covariance. Coherent sets for prediction are defined by thresholding in subspaces with contracting dynamics while coherent sets for retrospection are defined by thresholding in subspaces with expanding dynamics. Analysis of temporal receptive fields of biological neurons shows how this framework can classify them into predictive versus retrospective cells.", "questions": "I'm confused about the association of expanding regions (i.e., directions in state space) with retrospection and contracting directions with prediction. For example take the 2d OU process with A = [[1,0],[0,-1]] and D = [[1,0],[0,1]]. The dynamics are expanding along x_1 and contracting along x_2. IIUC, the coherent sets for prediction are {x: x_2 < 0} and {x: x_2 ≥ 0} while the coherent sets for retrospection are {x: x_1 < 0} and {x: x_1 ≥ 0}. This seems backwards. Isn't it much easier to predict sign(x_1) than sign(x_2)? For example if x_1(0) = 2 then there's a high probability that sign(x_1(t)) > 0 for any t > 0, because of the repulsive dynamics. In contrast, if x_2(0) = 2 then we can't reliably predict x_2(t) for t > 0, because the attractive dynamics will erase the initial conditions and the future trajectory will bounce back and forth around x_2 = 0.\n\nWhat happens when the dynamics are repulsive (or attractive) in all directions? I believe B_tau (F_tau) is still well-defined. If a neuron projects to its top singular vector does the neuron just fail to make meaningful predictions (retrospections)?\n\nI was convinced by the proof on Wikipedia but an academic citation would be more appropriate. Maybe the Bhatia & Rosenthal (1997) reference there would work.\n\nI may not have been the ideal reviewer for this paper but I had to do a lot of secondary reading to understand the mathematical properties of several of the constructions (Perron-Frobenius operator, forward-backward and backward-forward operators, Galerkin projections). Some intuitive guidance in secs 3-4 would probably help other readers.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nSophisticated mathematical approach\nClever and novel bridge between inference over stochastic processes and temporal characteristics of real neurons.\n\nWeaknesses:\n\nEvidence is largely circumstantial. The theoretical framework offers an organization of existing data based on qualitative measures (i.e., the sign of the time constant of the orthogonal exponential function), but does it make quantitative new predictions?", "was_revised": true, "final_justification": "Sophisticated and well-conceived theory offering a novel perspective on neural data", "rebuttal": "We thank the reviewer for taking the time to provide thoughtful feedback on our manuscript. Below, we address each of the reviewer’s comments in turn.\n\n*1. Evidence is largely circumstantial. The theoretical framework offers an organization of existing data based on qualitative measures (i.e., the sign of the time constant of the orthogonal exponential function), but does it make quantitative new predictions?*\n\nBased on our OU analysis in >3 dimensional space, we expect to see neuronal temporal filters with multiple (>2) phases. We think that such observations are difficult because they are often obscured by noise and limited amounts of data. We predict that, as the recording techniques improve and the amount of data grows,  many more such multiphasic (>2) filters will be seen.    \n\n*2. I'm confused about the association of expanding regions (i.e., directions in state space) with retrospection and contracting directions with prediction. For example take the 2d OU process with A = [[1,0],[0,-1]] and D = [[1,0],[0,1]]. The dynamics are expanding along x_1 and contracting along x_2. IIUC, the coherent sets for prediction are {x: x_2 < 0} and {x: x_2 ≥ 0} while the coherent sets for retrospection are {x: x_1 < 0} and {x: x_1 ≥ 0}. This seems backwards. Isn't it much easier to predict sign(x_1) than sign(x_2)? For example if x_1(0) = 2 then there's a high probability that sign(x_1(t)) > 0 for any t > 0, because of the repulsive dynamics. In contrast, if x_2(0) = 2 then we can't reliably predict x_2(t) for t > 0, because the attractive dynamics will erase the initial conditions and the future trajectory will bounce back and forth around x_2 = 0.*\n\nWe are afraid that the provided example, as described, is backwards: {x: x_2 < 0} and {x: x_2 ≥ 0} should be expanding coherent sets for retrospection, {x: x_1 < 0} and {x: x_1 ≥ 0} should be contracting coherent sets for prediction. When you focus on the set where x_2 has a fixed sign, the dynamics will expand along positive and negative directions of x_1. When you focus on the set where x_1 has a fixed sign, the dynamics will contract to the x_1 axis on one side of the x_2 axis. \n\n*3. What happens when the dynamics are repulsive (or attractive) in all directions? I believe B_tau (F_tau) is still well-defined. If a neuron projects to its top singular vector does the neuron just fail to make meaningful predictions (retrospections)?*\n\nIf a critical point is purely unstable, it will not be visited by the autonomous dynamics. Hence it will not be physically relevant. Moreover, applying this approach to nearly isotropic attractive fixed points gives partitions that are not ‘distinguished’, to use Froyland’s (2013) terminology. These partitions do not represent a genuine clustering of states which could describe qualitatively different parts of the phase space. We will add this paragraph to the manuscript.\n\n*4. I was convinced by the proof on Wikipedia but an academic citation would be more appropriate. Maybe the Bhatia & Rosenthal (1997) reference there would work.*\n\nThanks! We will adjust the reference.\n  \n*5. I may not have been the ideal reviewer for this paper but I had to do a lot of secondary reading to understand the mathematical properties of several of the constructions (Perron-Frobenius operator, forward-backward and backward-forward operators, Galerkin projections). Some intuitive guidance in secs 3-4 would probably help other readers.*\n\nWe will add a section to the Supplementary Material providing a short pedagogical introduction to transfer operators in the context of drift-diffusion models, as well as the Galerkin projections. \n\nFroyland, G. (2013). Physica D: Nonlinear Phenomena, 250, 1-19."}, {"review_id": "7g3HRSfUw0", "reviewer": "Reviewer_qZtM", "summary": "This paper proposes a theoretical framework conceptualizing sensory neurons as detectors of coherent sets in high-dimensional stochastic dynamical systems. For OU processes, they analytically derive that the subdominant singular function of the stochastic Koopman operator reduces to a linear projection followed by rectification, naturally explaining neuronal temporal receptive fields and the ubiquity of rectification in neural responses. The framework predicts complementary neuronal classes specialized for prediction (contracting coherent sets) versus retrospection (expanding coherent sets), which the authors connect to known functional dichotomies like tufted/mitral cells and lagged/non-lagged cells.", "questions": "1. The analytical results are limited to OU processes, but the authors claim the framework extends to nonlinear dynamics. Could authors provide concrete examples with analytical or numerical solutions for specific nonlinear systems relevant to sensory processing?\n2. How does the proposed framework account for the extensive feedback connections in sensory systems? Would feedback fundamentally alter the coherent set structure or the neuronal computations?\n3. What are the computational requirements for neurons to perform the proposed coherent set detection? How does this scale with the dimensionality of the sensory input space?\n4. How do authors distinguish the proposed coherent set framework from other established theories like efficient coding, sparse coding, or predictive coding that also explain temporal receptive fields and rectification?", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n1. The connection between coherent set detection and neural computation is original and mathematically rigorous.\n2. Analysis of real neuronal data from multiple sensory systems supports theoretical predictions.\n\n### Weaknesses:\n1. Oversimplified dynamics: The analytical results are restricted to linear OU processes; real sensory dynamics are likely far more complex.\n2. Incomplete nonlinear treatment: The extension to nonlinear dynamics via Galerkin projections is mentioned but not thoroughly developed or validated.\n3. Limited scope of data analysis: The empirical analysis focuses on relatively simple temporal receptive field patterns and doesn't address more complex neuronal response properties.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns.", "rebuttal": "We thank the reviewer for taking the time to provide thoughtful feedback on our manuscript. Below, we address each of the reviewer’s comments in turn.\n\n*1. Oversimplified dynamics: The analytical results are restricted to linear OU processes; real sensory dynamics are likely far more complex.*\n\nExactly! 1. This is why we propose the data-driven approach that can be applied to almost any dynamics. 2. Any local neighborhood of a nonlinear system can be approximated as a linear system through Taylor expansion. In many cases, the behavior of a nonlinear system can be largely captured by the local linear behaviors around its critical points. 3. Critical points and invariant manifolds linking them can serve as a skeleton of the global nonlinear system. In potential future works, we are going to propose a neural network such that each critical point corresponds to a group of neurons, and the information from different groups of neurons are aggregated through the next layer of the neural network. But this is beyond the scope of this manuscript.\n\n*2. Incomplete nonlinear treatment: The extension to nonlinear dynamics via Galerkin projections is mentioned but not thoroughly developed or validated.*\n\nWe will add a section to the Supplementary Material providing a short pedagogical introduction to the Galerkin method using a time-delay embedding basis.\n\n*3. Limited scope of data analysis: The empirical analysis focuses on relatively simple temporal receptive field patterns and doesn't address more complex neuronal response properties*\n\nWe restricted our analysis to neurons close to the sensory periphery where we could closely link the neural responses to the stimuli. We suspect that more complex neuronal response properties to sensory stimuli, for instance in cortical areas, may be described by similarly simple receptive fields with respect to intermediate (upstream) neural activity. If such a dataset were available we would be able to apply our approach to it. \n\nOur analyses were restricted due to using previously published data. As noted elsewhere, neural data was supplied in the form of temporal receptive fields for Mitral/Tufted cells and Spatiotemporal receptive fields for RGCs, without a record of individual stimulus/response trials. RGC receptive fields were estimated by the original authors using reverse correlation of white noise. Receptive fields for Mitral/Tufted cells were estimated by the original authors by describing the best response to brief odor pulses. More complex neuronal response properties would require purposefully defined stimuli and experimental conditions.\n\n*4. The analytical results are limited to OU processes, but the authors claim the framework extends to nonlinear dynamics. Could authors provide concrete examples with analytical or numerical solutions for specific nonlinear systems relevant to sensory processing?*\n\nEarlier work by Froyland (2013) and others show the efficacy of this method for nonlinear systems. We are not sure if there is an obvious explicit dynamical system model ‘relevant to sensory processing’ that we can analytically or numerically solve. Perhaps, the results in our Fig. 2 come closest to answering this question. \n\n*5. How does the proposed framework account for the extensive feedback connections in sensory systems? Would feedback fundamentally alter the coherent set structure or the neuronal computations?*\n\nWe focus on the feedback free setting, noting that certain latent features can be extracted from stimulus streams without feedback. While there is feedback in the vertebrate retina and invertebrate lamina, the  processing is primarily feedforward. Previously many feedforward network models were proposed to capture retinal computation accurately [Maheswaranathan & Baccus, 2023]. We can clarify these points in the future version of the manuscript. The potential impacts of feedback would be interesting to explore in future work.\n\n*6. What are the computational requirements for neurons to perform the proposed coherent set detection? How does this scale with the dimensionality of the sensory input space?*\n\nWhat makes our approach so enticing is its biological plausibility. Currently, we are focusing on neurons processing one-dimensional stimuli, such as Drosophila lamina neurons and sensory and relay neurons of the olfactory bulb. So long as one uses a lag vector basis, the computational requirement of the proposed approach from each neuron is simply a linear temporal filter followed by a spiking nonlinearity, both could be implemented by ion channels with different time constants. In future works, we will consider spatio-temporal stimuli where the spatial component of the spatio-temporal filter could be implemented by synaptic weight vectors.\n\n*7. How do authors distinguish the proposed coherent set framework from other established theories like efficient coding, sparse coding, or predictive coding that also explain temporal receptive fields and rectification?*\n\nEfficient coding is a general principle that aims to maximize the mutual information between stimuli and neural responses. In contrast, the aim of our work and the information bottleneck method is to infer the information from past stimuli about future stimuli most effectively. We can show in Supplementary Material that the linear projection (but not rectification) in our method is equivalent to the information bottleneck method for Gaussian distributions and linear systems. They may not be equivalent for more complex situations.  In predictive coding, neurons output prediction error, i.e. the difference between optimal prediction and actual signal. The above methods do not give a principled derivation of rectification. Finally, sparse coding is typically static and discards the dynamical properties of the stimuli. As suggested by the reviewers, we will re-write, expand and rename the Relationship to Other Work Section and place it after the Introduction.  \n\nFroyland, G. (2013). Physica D: Nonlinear Phenomena, 250, 1-19.\n\nMaheswaranathan, N., McIntosh, L. T., Tanaka, H., Grant, S., Kastner, D. B., Melander, J. B., ... & Baccus, S. A. (2023). Neuron, 111(17), 2742-2755."}]

Author rebuttal:

Meta-review: This paper proposes a novel method for understanding early sensory encoding in terms of transfer operators in nonlinear dynamics. In contrast with efficient or predictive coding, which focus on information compression or future prediction of stimuli, this work views populations of early sensory neurons as detectors for snippets of _dyanmics._ By assuming that summed sensory inputs to neurons follow a multivariate Ornstein-Uhlenbeck process, the authors establish that the optimal solution to the general problem -- projection onto the subdominant singular vector of the stochastic Koopman operator -- can be implemented by linear projection and threshold rectification. They then apply this method to luminance and olfactory data, finding close matches to the structure of coding in the retina and olfactory bulb, respectively.

Reviewers found the approach highly novel and clearly written, though the material may be quite technical for many neuroscientists. The method presents some problems for a biologically plausible implementation, though the authors provide some suggestions by which the required operations (learning dynamics from data, learning the subdominant eigenfunction) could be carried out. Reviewers also suggested that authors clarify the relationship between their proposed approach and others like efficient coding that also explain many aspects of early sensory coding.

During discussion, the authors made several clarifications on the method and its potential implementation, largely satisfying reviewer concerns from the first round.

Overall, this is a well-executed and highly novel contribution to the understanding of early stage sensory neurons as encoders of complex dynamical stimuli.
Decision: Accept (poster)