Title: Neurons as Detectors of Coherent Sets in Sensory Dynamics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neuron

self-supervised learning

dynamics

spectral clustering
Abstract: We model sensory streams as observations from high-dimensional stochastic dynamical systems and conceptualize sensory neurons as self-supervised learners of compact representations of such dynamics. From prior experience, neurons learn {\it coherent sets}—regions of stimulus state space whose trajectories evolve cohesively over finite times—and assign membership indices to new stimuli. Coherent sets are identified via spectral clustering of the {\it stochastic Koopman operator (SKO)}, where the sign pattern of a subdominant singular function partitions the state space into minimally coupled regions. For multivariate Ornstein–Uhlenbeck processes, this singular function reduces to a linear projection onto the dominant singular vector of the whitened state-transition matrix. Encoding this singular vector as a receptive field enables neurons to compute membership indices via the projection sign in a biologically plausible manner. Each neuron detects either a {\it predictive} coherent set (stimuli with common futures) or a {\it retrospective} coherent set (stimuli with common pasts), suggesting a functional dichotomy among neurons. Since neurons lack access to explicit dynamical equations, the requisite singular vectors must be estimated directly from data, for example, via past–future canonical correlation analysis on lag-vector representations—an approach that naturally extends to nonlinear dynamics. This framework provides a novel account of neuronal temporal filtering, the ubiquity of rectification in neural responses, and known functional dichotomies. Coherent-set clustering thus emerges as a fundamental computation underlying sensory processing and transferable to bio-inspired artificial systems.

Neurons as Detectors of Coherent Sets in Sensory Dynamics

Abstract

We model sensory streams as observations from high-dimensional stochastic dynamical systems and conceptualize sensory neurons as self-supervised learners of compact representations of such dynamics. From prior experience, neurons learn coherent sets -regions of stimulus state space whose trajectories evolve cohesively over finite times-and assign membership indices to new stimuli. Coherent sets are identified via spectral clustering of the stochastic Koopman operator (SKO) , where the sign pattern of a subdominant singular function partitions the state space into minimally coupled regions. For multivariate Ornstein-Uhlenbeck processes, this singular function reduces to a linear projection onto the dominant singular vector of the whitened state-transition matrix. Encoding this singular vector as a receptive field enables neurons to compute membership indices via the projection sign in a biologically plausible manner. Each neuron detects either a predictive coherent set (stimuli with common futures) or a retrospective coherent set (stimuli with common pasts), suggesting a functional dichotomy among neurons. Since neurons lack access to explicit dynamical equations, the requisite singular vectors must be estimated directly from data, for example, via past-future canonical correlation analysis on lag-vector representations-an approach that naturally extends to nonlinear dynamics. This framework provides a novel account of neuronal temporal filtering, the ubiquity of rectification in neural responses, and known functional dichotomies. Coherent-set clustering thus emerges as a fundamental computation underlying sensory processing and transferable to bio-inspired artificial systems.

Neurons in early sensory areas are traditionally thought to extract from recent inputs low-dimensional latent variables that are maximally informative about the near future [1, 2, 3, 4]. Such extraction exploits statistical regularities acquired over evolutionary, developmental, and behavioral timescales from previously encountered natural stimuli [5, 6, 7]. To formalize this intuition for temporally correlated sensory stimuli, we postulate that they are generated by high-dimensional, potentially nonlinear, stochastic dynamical processes, and conceptualize neurons as self-supervised learners of coherent sets -regions of the stimulus state space that evolve cohesively over finite time intervals [8, 9]-thus enabling compact representations of sensory dynamics.

Coherent sets can be uncovered via spectral clustering of the stochastic Koopman operator (SKO) -a linear, albeit infinite-dimensional, operator that evolves observables over a time interval [10, 11, 12, 13]. The sign of the first non-trivial (subdominant) singular function of the SKO partitions state space into two minimally interacting coherent sets (Fig. 1a). Accordingly, a neuron can compute a membership index of an input by evaluating the sign of a subdominant singular function. Because singular values and functions remain real even for irreversible dynamics, this approach generalizes metastable set detection beyond the reversible cases that eigenfunction methods require [14].

We demonstrate that for multivariate Ornstein-Uhlenbeck (OU) process [15]-a canonical example of linear stochastic dynamics and a reasonable model of summed input to a neuron [16, 17]-a subdominant singular function of the SKO corresponds to a projection of the input onto a singular vector of the whitened finite-time transition matrix, Fig. 1b. A neuron that stores this singular vector in its synaptic weights and temporal filter can compute the corresponding membership index via the sign of the weighted sum of the inputs. Although the conventional spectral clustering framework assumes stability, our singular function results extend to unstable systems by focusing on singular values closest to one. In addition to projecting on the right singular vector predicting near-future inputs (Fig. 1b), projecting on the left one is also possibleretrospecting the recent past (Fig. 1c). Because such projections require different synaptic weights and temporal filters they must be implemented by distinct neurons.

Recognizing that the underlying dynamical equations are not available to neurons, a biologically plausible detection of coherent sets requires a data-driven algorithm that can infer them directly from observations. Coherent sets can be learned and membership sets inferred locally using canonical correlation analysis (CCA) of past-future data pairs [18, 13, 19]. This algorithm has the additional advantage of being applicable to nonlinear dynamics: it relies on estimating a Galerkin projection of the SKO onto a functional basis via Monte Carlo integration over observed data [10, 20, 21, 12]. If the upstream neurons implement such projection, a post-synaptic neuron could learn/represent a requisite singular vector in synaptic weights indicating set membership by the sign of the inputs sum.

Viewing neurons as coherent set detectors sheds light onto several longstanding neurophysiological observations. First, temporal receptive fields of neurons emerge naturally as subdominant singular vectors projecting input lag-vector representations of dynamical states. Second, the ubiquity of response rectification in neurons, exemplified by the well-known ON/OFF segregation in early visual circuits, is interpreted as a principled clustering mechanism. Finally, the theory predicts complementary neuronal classes that predict near future using predictive coherent sets or retrospect recent past using retrospective ones, consistent with known neuronal functional dichotomies such as tufted versus mitral cells in the olfactory bulb or non-lagged versus lagged cells in the lateral geniculate nucleus (LGN). Thus, the detection of coherent sets can serve as a powerful algorithmic primitive for neural computation supporting prediction and retrospection. This offers insights into biological processes and could inspire future artificial neural networks.

The remainder of the paper is organized as follows. Section 1 discusses related work. Section 2 reviews the definition of transfer operators, coherent sets and their connection to spectral clustering through the singular functions of the SKO. In Section 3, we derive the central result: under OU dynamics, the subdominant singular functions of the SKO correspond to dot products between the state and the singular vectors of the whitened transition matrix. Section 4 reviews a data-driven algorithm for identifying subdominant singular functions, which extends naturally to nonlinear dynamics and defines neuronal units rectifying positive or negative parts of the subdominant singular function. Section 5 analyzes and reviews several experimental datasets, interpreting temporal receptive fields through the hypothesis that biological neurons cluster coherent sets.

a

(b)

u12

u2

V1

1

coordinate

V2

coordinate

V2

ul1

u1

0

-1

coordinate

coordinate

1

coordinate

1

1 Related Work

Viewing early sensory processing as efficient or predictive coding of natural stimuli has a long tradition [5, 6, 7, 22, 2]. Closest to our work is the information bottleneck (IB) framework [1, 2, 3, 4], which for Gaussian variables and stable dynamical systems, reduces to past-future CCA [23]. In this setting, optimal one-bit encoding corresponds to the sign of the subdominant singular function of the SKO; we generalize this result to unstable dynamics. Detecting coherent sets can also be viewed as an extension of slow feature analysis [24, 25]. However, our explicit formulation in terms of unstable dynamical systems yields a principled nonlinearity that supports non-trivial feature generation and naturally extends to deep architectures [26]. Predictive and retrospective neuron classes have also appeared in a lattice filter model of the visual pathway [27], though that framework was restricted to linear processing. Relative to predictive coding theories [28, 29, 30, 27] which posit that neurons transmit prediction errors, we instead emphasize how predictions can be computed-via coherent-set memberships learned from past-future correlations. Finally, while ON/OFF pathway segregation has been attributed to metabolic efficiency [31], we propose a complementary computational explanation. Our account of lagged and non-lagged LGN cells differs from [32] in that it does not require an explicit nonlinearity, suggesting that these types can emerge alongside ON/OFF segregation.

Prediction and retrospection in saddle-point dynamics have been discussed in the context of unstable periodic orbits of chaotic attractors using the dominant mode of the local SKO (and its adjoint) [33, 34], whereas we focus on a subdominant singular function defining a coherent set pair [35, 9]. Saddle point analysis based on eigenfunctions instead of singular functions predicted neuronal filters orthogonal to decaying exponentials, yielding predictive, but not retrospective, neurons [36].

Clustering has been previously proposed as a model of static neuronal computation on temporally uncorrelated inputs [37], capturing rectification and sparsity but not temporal receptive fields or sensory dynamics. Probabilistic coding frameworks such as the Bayesian brain hypothesis [38] have been widely studied although we are not aware of the suggestion that single neurons represent eigenfunctions or singular functions of the SKO. Self-supervised learning has been applied to visual networks [39], but these models typically omit analysis on the neuronal level. Koopman spectral analysis and data-driven Koopman approximations (e.g. DMD) have a long tradition in nonlinear dynamics [10, 11, 40, 20, 21, 12, 13] and have recently been incorporated into deep architectures to extract predictive features [41]. In contrast, we use SKO singular functions to define coherent sets [35, 9], providing a direct explanation for rectification and a principled division into predictive and retrospective neurons.

Estimation of singular functions from data originated in molecular dynamics as the VAMP framework [14]. VAMPnets [42] learn such functions with deep ReLU networks trained by backpropagation. In our model, features arise from rectified projections onto singular vectors within each neuron and can be hierarchically composed without backpropagation.

2 Discovering coherent sets via spectral clustering of transfer operators

We begin with the concept of coherent sets, which are regions of state space that tend to move as a whole under the stochastic dynamics over finite time intervals [43, 35, 9]. Formally, a pair of sets ( A , B ) is coherent if a state that begins in A at time t is very likely to be found in B at time t + τ :

where P [ X | Y ] is a probability of X given Y , X ( t ) ∈ X denotes the random state of the system at time t . If in addition the probabilities of visiting A at time t and B at time t + τ are equal, then this relation works in both directions: observing the system in B at time t + τ makes it likely that it was in A at time t . Thus, membership in coherent sets, A and B , can be used to predict the future and retrospect the past, respectively, of a stochastic process.

To compute coherent sets we utilize the stochastic Koopman operator (SKO), which encodes how observables evolve in expectation [10, 44, 11, 35, 9, 13, 12], see the Supplement, Section 1 for a concise introduction. An observable is any scalar function of X with a finite expectation value over the stochastic process. For instance, from the neuron's perspective, each synaptic input can be thought of as such an observable of stimuli. The SKO acts on an observable f

i.e. it maps the present observable to its expected value τ units of time later. Linearity of the expectation makes K τ a linear operator.

Let µ ( x ) and ν ( x ) denote the probability densities of the system's state at times t and t + τ , respectively [35, 9, 13]. For any measurable region A ⊆ X ,

These densities define inner products between functions f and g as

When the state space is divided into two pairs of regions, identifying the most coherent ones amounts to maximizing the following objective [9, 13]:

where ( A , B ) are measurable subsets of the state space X , and ( A c , B c ) denote their complements. The indicator function 1 A ( x ) equals 1 if x ∈ A and 0 otherwise. Large values of the objective correspond to pairs of regions that remain coherent under the dynamics-that is, regions that are least dispersive over the time interval τ .

In practice, the maximizers of this quotient are well-approximated by certain singular functions of the SKO [35, 9, 13]. To see this, note that K τ has an adjoint operator K † τ , defined so that

The finite-time forward-backward and backward-forward operators

are then self-adjoint and, under mild assumptions, compact [13, 18]. As such, each admits a countable spectral decomposition in terms of orthonormal basis [10]

where D may be infinite. The functions u i ( x ) and v i ( x ) are the singular functions of K τ , satisfying

Pairs of singular functions with large λ i are approximate maximizers of (4). Moreover, the sets

approximate coherent sets, where U ± i is approximately the image of V ± i under the dynamics.

The principal singular functions of the SKO are trivial, u 0 ( x ) = v 0 ( x ) = 1 : they define the trivial coherent set, U 0 = V 0 = X . The subdominant singular functions define the least dispersive non-trivial coherent set pair of the dynamics, ( V ± i , U ± i ) . We propose that the neurons compute membership indices by evaluating the signs of such singular functions. Left singular functions look forward in time predicting the future. Right singular functions look backward in time allowing to retrospect past events. Both types of measurements are important in ascertaining the state of partially observed dynamical systems, and we predict the existence of both predictive and retrospective neurons.

3 Koopman singular functions for Ornstein-Uhlenbeck processes

Identifying coherent sets through the subdominant singular functions of the SKO provides a rigorous theoretical framework, whose application requires computing these singular functions. In this section,

we attempt to provide an intuitive picture by considering a linear stochastic dynamical system - OU process - for which subdominant SKO singular functions can be found in the closed form providing much needed intuition. The temporal dynamics of neuronal input currents elicited by sensory stimuli has long been approximated by the OU process [16, 17]:

where ⟨ ξ ( t ) ξ ( t ′ ) ⊤ ⟩ = D δ ( t -t ′ ) . Here, we assume that A is a real matrix with real eigenvalues and eigenvectors. We are particularly interested in saddle-point OU for the following reasons. If a critical point is purely repulsive, it will not be visited by the autonomous dynamics and, hence, will be physically irrelevant. Applying this approach to nearly isotropic attractive critical points gives partitions that are not 'distinguished' [9], i.e. do not represent a genuine clustering of states which could describe qualitatively different parts of the phase space.

The probability density of the state variable x , p ( x , t ) , evolves according to the forward Kolmogorov equation (aka the Fokker-Planck equation) utilizing the forward Kolmogorov operator L [45]:

The dynamics of measurement expectation g ( x , t ) = E [ g ( X ( t )) | X (0) = x ] are given by the backward Kolmogorov equation:

where L † is the adjoint of the forward Kolmogorov operator with respect to the standard Euclidean inner product, serving as the generator of the SKO [10]:

The stationary distribution of probability density under L , satisfying L ρ 0 = 0 , is derived in the Supplement, Section 2:

where x ∈ R n and Σ is a solution of the Lyapunov equation,

For attractive dynamics, where all eigenvalues of A have negative real parts, Eq. (15) has a unique positive-definite solution corresponding to the covariance matrix, Σ = E [ xx ⊤ ] . For repulsive or saddle-point dynamics, Eq. (15) is a Sylvester equation, which has a unique solution if and only if A has no eigenvalues related by sign reversal [46]. Thus, for a generic choice of A , Eq. (15) has a unique solution that is symmetric, real, and invertible. For repulsive directions, corresponding to eigenvalues of A with positive real parts, Eq. (14) characterizes how fast the density grows away from the fixed point [47].

Under stationarity, µ ( x ) = ν ( x ) = ρ 0 ( x ) , the adjoint of K τ with respect to ρ 0 ( x ) is given by

which is termed the reweighted Perron-Fronbenius operator and can also be interpreted as the SKO of the reverse-time dynamics. We then search for the eigenfunctions of the operators F τ and B τ , Eq. (6), which are well defined on short time scales.

We find (see the Supplement, Section 3) a family of eigenfunctions of F τ , B τ that can be expressed in terms of the linear projection of the state vector, x :

where u i and v i satisfy the following matrix eigenvector equations (see the Supplement, Section 3):

Eigenvalues of transfer operators for expanding maps could be greater than one [48]. Since ideal coherent sets with zero leakage correspond to a unity eigenvalue, in this paper a subdominant eigenfunction refers to the non-trivial eigenfunction associated with an eigenvalue closest to one.

That is, the subdominant eigenfunction for attractive (repulsive) dynamics is associated with the second largest (smallest) eigenvalue. We proved that the subdominant eigenfunctions belong to the linear family Eq. (17) for attractive and repulsive dynamics. Other eigenfunctions of F τ , B τ are either a constant, associated with the eigenvalue one, that does not change sign, or represent higher-order polynomials in x , associated with less dominant eigenvalues (see the Supplement, Section 3, which, however, does not consider saddle points). Therefore, to identify the least dispersive coherent sets, it is sufficient to focus on the eigenfunctions in Eqs. (17). The pair of minimally leaking coherent sets is given by the solution of Eqs. (18) with λ 2 i closest to unity. To obtain the coherent set membership indices, we recover indicator functions, Eq. (4), by taking the sign of these singular functions.

Eigenvectors v i and u i play a complementary role in prediction and retrospection. As the operator F τ propagates observables forward then backward in time, neurons projecting inputs onto v i are predictive (Fig. 1b). As the operator B τ propagates the observable backward then forward in time, neurons projecting inputs onto u i are retrospective (Fig. 1c). In the case of 2D saddle-point OU (Fig. 1b,c), when the shared past and future refer to other fixed points, the coherent sets of interest are the expanding coherent set for prediction ( v 1 ) and the contracting coherent set for retrospection ( u 2 ). In Section 4 of the Supplement, we analytically prove that, as the forecast horizon goes to infinity, singular vectors v 1 and u 2 converge to the top and bottom left eigenvector of A which are orthogonal to the stable and unstable invariant subspaces, respectively.

So far we considered fully observable OU processes. However, in reality, the OU process is often only partially observed for example via a linear projection of the state onto a scalar variable,

Neurally, such projection can be an input to a single-synapse neuron or a total synaptic current. Assuming observability, the state can be represented by an n -dimensional lag vector [49, 50, 51]:

evolving via a companion matrix, equivalent to the following auto-regressive model:

4 Coherent sets from data

The infinite-dimensional stochastic Koopman operator (SKO) can be approximated by a Galerkin projection onto a finite set of basis functions (or features), { ϕ i ( x ) } d i =1 [21, 13]. In the data-driven formulation, the SKO matrix representation under this basis is [18, 13] (see Supplement, Section 5):

where Σ 0 and Σ τ are the covariance matrices:

and describe the correlation between measurement functions i and j measured instantaneously, or with delay τ , respectively.

Such matrices are well defined if the process is stationary which may not be true for neurons. A common workaround is to assume local stationarity, that is, the distribution of inputs changes slowly compared to the timescale of the dynamics. In this case, the neuron can continuously update the estimate of the dynamics using exponential forgetting [52]. Our method can accommodate this type of non-stationarity by computing Eqs. (23) locally in time, using a temporal filter that discounts older observations. This allows the estimated covariances to track gradual changes in the underlying distribution without assuming global stationarity.

As a direct consequence of Eq. (5), the adjoint of K τ is K † τ = Σ -1 0 K ⊤ τ Σ 0 . The forward and backward operators have matrix representations [18, 13]:

which reduce to Eq. (18) for attractive OU processes (see the Supplement, Section 6). The singular functions of the SKO within the Galerkin projection are given by

where ⃗ v i and ⃗ u i are eigenvectors of the matrices F τ and B τ , respectively. Notice that ⃗ v i and ⃗ u i are precisely the solution of past-future CCA.

By representing these eigenvectors in synaptic weights and temporal filters, a neuron may compute a leading singular function of the SKO restricted to the linear span of its input features. By indicating the sign of the singular function, the neuron then computes the membership index for each input. For the sake of simplicity, below we only consider the temporal component of the singular vector. Even if the neuron has access to only a single synapse, it can retain a history of that synapse's activity over d consecutive time steps. This construction-known as delay embedding or delay coordinates-also yields a Galerkin approximation [49, 50]. We take x ∈ R d to be a lag vector of the input current to the neuron over the previous d intervals of time, such that X ( t ) = ( I ( t ) , I ( t -∆ t ) , . . . , I ( t -( d -1)∆ t )) , where I ( t ) is the observed current at time t . In this regime, the basis functions are given by

i.e. ⃗ ϕ ( x ) is the identity function. Singular vectors act as temporal filters over the delayed signal, to

where ⃗ u i and ⃗ v i are singular vectors (cf. Eq. (25)), and the Heaviside function, H ( x ) = 1 for x ≥ 0 , and H ( x ) = 0 for x < 0 .

Therefore, sensory streams can be clustered into coherent sets via data-driven Galerkin approximations of the SKO. By learning the singular vectors from the features represented by the activity of the upstream neurons and encoding them in the synaptic weight and temporal filters, biological neurons can cluster inputs using integrate-and-fire dynamics.

5 Coherent set clustering perspective on neurophysiology

Here we apply the coherent set clustering framework to biological neurons focusing on three observations: temporal receptive fields, neuronal rectification and predictive/retrospective properties. We restrict our consideration to the processing of a scalar time series viewed as a one-dimensional projection of a multidimensional state-space dynamics, Eq. (27). Such a scalar time series could represent an input to a single-input neuron or the total synaptic current into a multi-input neuron. Here, inspired by our result for OU processes that the relevant singular functions lie in the span of a lag vector basis, we choose Eq. (26) even in the data-driven, non-linear setting. Even such simple basis choice produces tangible results.

5.1 Temporal receptive fields

We consider early visual processing where a natural stimulus can be generated by emulating the movement of the retinal image due to self-motion or saccades by scanning a natural image or its model. In turn, natural images are commonly modeled by the "dead leaves" model partitioning the space into patches of different but uniform luminance with sharp transitions between them [53, 54]. Therefore, the resulting input time series is a set of plateaus at different levels with sharp transitions between them, Fig. 2a. Such scalar time series models input to post-photoreceptor neurons for invertebrates such as Drosophila , where initial processing is segregated between adjacent "pixels" or total current in a vertebrate bipolar cell.

We view this scalar time series as a linear projection of a high-dimensional dynamical system. Thus, we perform past-future CCA on the lag-vectors formed from the scalar time series. The canonical correlations reveal a spectral gap following the top two. The lag-vector space is partitioned into a pair of coherent sets in a canonical direction. The top right singular vector amounts to a low-pass filter (Fig. 2b) similar to sustained bipolar cells of the vertebrate retina or L3 neuron in Drosophila [55]. Intuitively, this reflects the slow variation of luminance on the plateaus, so the near future is expected to resemble the recent past. The second canonical direction acts as a smoothed temporal derivative (Fig. 2b) in general agreement with experimentally reported filters of transient bipolars in vertebrates,

Fig. 2c or L1 and L2 cells in Drosophila [28, 55]. Because the stimulus is symmetric with respect to time-reversal, the complementary left singular vectors are obtained by simply inverting the time axis (and inverting the sign for the second singular vector). Intuitively, the second canonical direction reflects the inertia of transitions-an edge traversing the photoreceptor aperture is unlikely to reverse direction-and predicts the future by extrapolating via the temporal derivative. For results of this algorithm applied to different stimuli, see [26].

We further characterize the experimentally measured temporal responses by interpreting them from the perspective of predictive and retrospective coherent sets. To make this connection, we take advantage of the observation that a practical predictive filter must be significantly aligned with an unstable eigendirection and a practical retrospective filter must be significantly aligned with a stable eigendirection, Fig. 1b,c. Moreover, for a two-dimensional (hyperbolic) saddle point, the linear temporal filter of the predictive neuron must be orthogonal to the attractive subspace, while the linear temporal filter of the retrospective neuron must be orthogonal to the repulsive subspace, Fig. 1b,c, see Section 3 and the Supplement, Section 4. Repulsive modes in the lag-vector space are expanding exponentials in real time and attractive modes are contracting exponentials, suggesting a simple way to interpret linear temporal receptive fields by computing the cosine similarity, S , with exponentials or, equivalently, computing normalized Laplace transforms, Fig. 2c,d. The experimentally measured temporal filter is orthogonal to a contracting exponential and aligned with the expanding exponential (Fig. 2c,d), indicating a predictive neuron. For details, see the Supplement, Section 7.

a

(b)

(c)

(d)10

10-2

10

4

0

5

10

-0.5

0.0

-0.4

-0.2

0

-20

0

20

Time

Time lag

Time

5.2 Neuronal rectification exemplified by ON and OFF cells

Our coherent set clustering perspective suggests that, in the vicinity of dynamical saddles, neurons determine membership indices by applying a Heaviside step-function to the SKO singular function (or its sign-inverse), Eq.(27). Such non-linearity could be naturally implemented by the spiking mechanism and correspond to the ON and OFF ganglion cells of the vertebrate retina, named this way because they respond to luminance increments and decrements, respectively. However, neurons post-synaptic to photoreceptors, bipolar cells in vertebrates and large monopolar cells in flies, are nonspiking. Neuronal activity is represented by continuously varying graded potentials that determine a non-negative synaptic vesicle release rate. Many such neurons thus rectify their input and are also classified as either ON or OFF cell. Such rectified response to luminance variation is graded and could be viewed as soft clustering by using not only the sign but also the magnitude of the singular function, Fig. 1, resulting in a rectified linear unit (ReLU)-like operation [26].

5.3 Predictive and retrospective properties of biological neurons

In this Subsection, we analyze and review large datasets of experimentally measured temporal receptive fields from the perspective of predictive and retrospective coherent sets.

Tufted and mitral cells of the mammalian olfactory bulb. In mammals, information from olfactory sensory neurons is relayed to the rest of the brain by two neuron classes: tufted cells (TCs) and mitral cells (MCs). We analyzed a dataset of the temporal receptive fields of 204 TCs and MCs recorded blindly from the rat olfactory bulb [56] from the coherent set clustering perspective. As described above (5.1, Temporal Receptive Fields), we computed the cosine similarity of experimentally measured temporal filters with a battery of growing and decaying exponentials. In addition to identifying the orthogonal exponential, we also identified the sign of the exponent which has the highest cosine similarity with the temporal filter: positive exponents correspond to predictive neurons, and negative to retrospective. As a result, we found a mixture of predictive and retrospective properties, Fig. 3. Approximately half of the recorded cells had temporal filters orthogonal to decaying exponentials and, therefore, were likely predictive. About 4% were orthogonal to growing exponentials and, hence, were likely retrospective. About one-third were orthogonal to both growing and decaying exponentials suggesting that they analyze a higher- than 2-dimensional dynamics and could be either retrospective or predictive. See the Supplement, Section 7 for details.

We speculate that TCs and MCs are mostly predictive and retrospective, respectively, based on the existing literature. First, they respond at different phases of the sniff cycle as monitored by air flow [57]: TCs are preferentially active during the exponential growth phase and silent during the exponential decay phase. Conversely, MCs are preferentially active during the exponential decay phase and silent during the exponential growth phase. Second, MC responses lag relative to TCs suggesting they play a different role [57, 58, 59, 60], but see [61, 62]. Finally, TCs and MCs differ in their responses to synaptic inputs from the electrophysiologically stimulated olfactory sensory neurons [63]: TCs receive direct inputs and respond with an immediate biphasic profile typical of prediction, while MCs are activated indirectly with a delay characteristic of retrospection.

(a)

Tufted/MitralCells

Filter or exp Value (AU)

10°

Filter or exp Value (AU)

10°

Filter or exp Value (AU)

100

0.5

0.5

0.5

10-2

10-2

10-2

......

-10

0

10

10~4

-10

0

10

104

-10

0

10

-0.5

2s

O

-0.5

-0.5

y

~e

at

-0.5

0

-0.5

0

-0.5

0

Time (s)

Time (s)

Time (s)

(b)

(c)

Tufted/Mitral

RetinalGanglion

10

5

8

%Neurons

100%

10%

55%

4%

31%

10%

中

20

10

8品

% Neurons

100%

10%

92%

6%

中

0

1% 1%

中

D

1%

1%

中

中

-10

-5

Orthogonalα

0

5

nly

None

-20

Orthogonalα

0

20

O

None

Retinal ganglion cells (RGCs). To explore whether the distributions of predictive and retrospective neurons vary across sensory modalities, we analyzed a large dataset of temporal receptive fields of 1345 RGCs from dissociated vertebrate retina [64], Figs. S1, 3C. In vertebrates, signals from

differentially stimulated photoreceptors are combined in the retina and, therefore, an RGC's receptive field has a spatial component. Here, we only focus on the temporal component obtained by a rank-1 decomposition of the spatio-temporal receptive field. The majority of RGCs' receptive fields were orthogonal to decaying exponentials and aligned with expanding exponentials, hence predictive, though 1% of cells were retrospective, Fig. 3C. See the Supplement, Section 7 for details.

Lagged and non-lagged cells in the Lateral Geniculate Nucleus. RGCs project to the Lateral Geniculate Nucleus (LGN) [65], in which two classes of relay cells have been described in cat [66, 67, 68], monkey [69], and mouse [70]: one with the temporal receptive fields of predictive cells and one with retrospective properties, Fig. S2. Because the firing of the latter class of neurons substantially lags a step stimulus onset, they were termed "lagged cells". Comparing the response profiles of our dataset of tufted/mitral cells to a step stimulus provides insight into how lagged cells may manifest, Fig. 3A. Note that the middle, retrospective, neuron is initially inhibited by the positive step response, but overcomes this inhibition due to the delayed positive filter lobe. Thus, we identify non-lagged and lagged neurons with predictive and retrospective units, respectively.

6 Discussion

We propose that neurons act as coherent-set detectors : each unit computes a linear projection of its input onto a singular vector of a whitened finite-time transition matrix and then takes the sign or rectifies its positive or negative part. We derive this mechanism analytically for saddle-point Ornstein-Uhlenbeck (OU) dynamics and suggest that it extends to nonlinear systems via Galerkin projection on a chosen feature basis. In practice, a leading singular function can be learned directly from data, providing a path toward biologically plausible implementations (e.g., local weight updates). This perspective captures several physiological regularities, including rectification and selective sensitivity to temporally extended (finite-time) structure in stimuli as well as predictive and retrospective neuron types.

Open issues:

(i) Unstable modes and reference measures. Section 3 characterizes SKO singular functions for the OU model, but linking them to the spectral clustering framework of Section 2 under instability requires a reference measure that decays away from the saddle neighborhood. Practically, this suggests finite-time localization (windowing), exponential discounting, or committor/Doob-transformed weights so that the inner product prioritizes trajectories that remain near the saddle. Interpreting the OU model as a local description then would render the spectral objects well posed on that neighborhood.

(ii) Data-driven estimation along unstable directions. The Galerkin procedure of Section 4 matches the OU analysis for stable modes; extending it to unstable modes likely requires modified forward-backward compositions over short horizons so that finite-time expansion (singular values (>1)) is preserved rather than suppressed by whitening. Establishing a rigorous equivalence in this regime remains open.

(iii) Beyond 2D saddles. The correspondence between finite-time singular vectors and eigenvectors shown for 2D saddles may not extend to higher dimensions, especially with multiple unstable directions. Systematic analysis in higher-dimensional settings is needed.

(iv) Circuit feedback loops. While many early sensory pathways are predominantly feedforward and fall within our framework, feedback is ubiquitous in deeper circuits. Incorporating explicit closed-loop interactions-for example, via the controller perspective of [71]-could unify coherent-set detection with action selection and behavior generation.

(v) Intra-neuronal feedback. In our current account, rectification follows linear projection and does not influence learning of synaptic weights or temporal filters. In reality, synaptic plasticity may depend on rectified output via dendritic backpropagation. Incorporating such output into projection learning could provide the needed windowing for local learning mechanisms.

Societal impact. By linking biological neural computation to finite-time dynamical structure, this work advances our understanding of brain function and may inform approaches to mental health and neurological disorders. It could also inspire biologically grounded, self-supervised architectures for artificial neural networks operating in dynamical settings.

Acknowledgments. We are grateful to T. Ahamed, R. Cosentino, A. Genkin, B. Gutkin, J. M. Heninger, P. Kidd, A. Koulakov, I. Nemenman, S.E. Palmer, S. Qin, D. Rinberg, E. Schneidman, D.

Schwab, E. Simoncelli, S. Solla, P. Thomas and, especially, P. Cvitanovic for insightful discussions, to P. Gupta, D.F. Albeanu, D.B. Kastner, and S.A. Baccus for sharing their datasets. This research was supported in part by the National Science Foundation (NSF) EFRI BRAID: Scalable-Learning Neuromorphics, award No. 2318152. Some work was initiated and performed at the the Kavli Institute for Theoretical Physics (KITP) supported by grant NSF PHY-2309135 and at the Aspen Center for Physics, which is supported by NSF grant PHY-2210452.

References

Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057 , 2000.

William Bialek, Rob R De Ruyter Van Steveninck, and Naftali Tishby. Efficient representation as a design principle for neural coding and computation. In 2006 IEEE international symposium on information theory , pages 659-663. IEEE, 2006.

Stephanie E Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in a sensory population. Proceedings of the National Academy of Sciences , 112(22):6908-6913, 2015.

Matthew Chalk, Olivier Marre, and Gašper Tkaˇ cik. Toward a unified theory of efficient, predictive, and sparse coding. Proceedings of the National Academy of Sciences , 115(1):186191, 2018.

Fred Attneave. Some informational aspects of visual perception. Psychological review , 61(3):183, 1954.

Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication , 1(01):217-233, 1961.

Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual review of neuroscience , 24(1):1193-1216, 2001.

Michael Dellnitz and Oliver Junge. On the approximation of complicated dynamical behavior. SIAM Journal on Numerical Analysis , 36(2):491-515, 1999.

Gary Froyland. An analytic framework for identifying finite-time coherent sets in timedependent dynamical systems. Physica D: Nonlinear Phenomena , 250:1-19, 2013.

Igor Mezi´ c. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics , 41(1-3):309-325, August 2005.

Clarence W Rowley, Igor Mezi´ c, Shervin Bagheri, Philipp Schlatter, and Dan S Henningson. Spectral analysis of nonlinear flows. Journal of fluid mechanics , 641:115-127, 2009.

Steven L Brunton and J Nathan Kutz. Data-driven science and engineering: Machine learning, dynamical systems, and control . Cambridge University Press, 2022.

Stefan Klus and Nataša Djurdjevac Conrad. Dynamical systems and complex networks: A koopman operator perspective. Journal of Physics: Complexity , 5(4):041001, 2024.

Hao Wu and Frank Noé. Variational approach for learning markov processes from time series data. Journal of Nonlinear Science , 30(1):23-66, 2020.

Wikipedia contributors. Ornstein-uhlenbeck process - Wikipedia, the free encyclopedia, 2025. [Online; accessed 15-May-2025].

Henry C Tuckwell. Stochastic partial differential equations in neurobiology: linear and nonlinear models for spiking neurons. In Stochastic biomathematical models: with applications to neuronal modeling , pages 149-173. Springer, 2012.

Alain Destexhe, M Rudolph, J M Fellous, and T J Sejnowski. Fluctuating synaptic conductances recreate in vivo-like activity in neocortical neurons. Neuroscience , 107(1):13-24, 2001.

Péter Koltai, Hao Wu, Frank Noé, and Christof Schütte. Optimal data-driven estimation of generalized markov state models for non-equilibrium dynamics. Computation , 6(1):22, February 2018.

David Lipshutz, Yanis Bahroun, Siavash Golkar, Anirvan M Sengupta, Dmitri B Chklovskii, and others many. A biologically plausible neural network for multichannel canonical correlation analysis. Neural Computation , 33(9):2309-2352, 2021.

Jonathan H Tu. Dynamic mode decomposition: Theory and applications . PhD thesis, Princeton University, 2013.

Matthew O. Williams, Clarence W. Rowley, and Ioannis G. Kevrekidis. A kernel-based method for data-driven koopman spectral analysis. Journal of Computational Dynamics , 2(2):247-265, 2015.

Byron H Price and Jeffrey P Gavornik. Efficient temporal coding in the early visual system: existing evidence and future directions. Frontiers in Computational Neuroscience , 16:929348, 2022.

Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for gaussian variables. Advances in Neural Information Processing Systems , 16, 2003.

Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural Computation , 14(4):715-770, 2002.

David Lipshutz, Charles Windolf, Siavash Golkar, and Dmitri B Chklovskii. A biologically plausible neural network for slow feature analysis. Advances in Neural Information Processing Systems , 33:14986-14996, 2020.

Shanshan Qin, Joshua L Pughe-Sanford, Alexander Genkin, Pembe Gizem Ozdil, Philip Greengard, Anirvan M Sengupta, and Dmitri B Chklovskii. A network of biologically inspired rectified spectral units (resus) learns hierarchical features without error backpropagation. arXiv preprint arXiv:2512.23146 , 2025.

Karol Gregor and Dmitri Chklovskii. A lattice filter model of the visual pathway. Advances in Neural Information Processing Systems , 25, 2012.

Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: a fresh view of inhibition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences , 216(1205):427-459, 1982.

Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience , 2(1):79-87, 1999.

Shaul Druckmann, Tao Hu, and Dmitri Chklovskii. A mechanistic model of early sensory processing based on subtracting sparse representations. Advances in Neural Information Processing Systems , 25, 2012.

Julijana Gjorgjieva, Haim Sompolinsky, and Markus Meister. Benefits of pathway splitting in sensory coding. Journal of Neuroscience , 34(36):12127-12144, 2014.

Dawei W Dong and Joseph J Atick. Temporal decorrelation: a theory of lagged and nonlagged responses in the lateral geniculate nucleus. Network: Computation in Neural Systems , 6(2):159178, 1995.

Predrag Cvitanovi´ c and Domenico Lippolis. Knowing when to stop: How noise frees us from determinism. In AIP Conference Proceedings , volume 1468, pages 82-126. American Institute of Physics, 2012.

Jeffrey M Heninger, Domenico Lippolis, and Predrag Cvitanovi´ c. Neighborhoods of periodic orbits and the stationary distribution of a noisy chaotic system. Physical Review E , 92(6):062922, 2015.

Gary Froyland, Naratip Santitissadeekorn, and Adam Monahan. Transport in time-dependent dynamical systems: Finite-time coherent sets. Chaos: An Interdisciplinary Journal of Nonlinear Science , 20(4), 2010.

Siavash Golkar, Jules Berman, David Lipshutz, Robert Mihai Haret, Tim Gollisch, and Dmitri B Chklovskii. Neuronal temporal filters as normal mode extractors. Physical Review Research , 6(1):013111, 2024.

Cengiz Pehlevan, Alexander Genkin, and Dmitri B Chklovskii. A clustering neural network model of insect olfaction. In 2017 51st Asilomar Conference on Signals, Systems, and Computers , page 593-600. IEEE, IEEE, 2017.

Wei Ji Ma, Jeffrey M Beck, Peter E Latham, and Alexandre Pouget. Bayesian inference with probabilistic population codes. Nature neuroscience , 9(11):1432-1438, 2006.

Yosef Singer, Yayoi Teramoto, Ben DB Willmore, Jan WH Schnupp, Andrew J King, and Nicol S Harper. Sensory cortex is optimized for prediction of future input. Elife , 7:e31557, 2018.

Marko Budiši´ c, Ryan Mohr, and Igor Mezi´ c. Applied koopmanism. Chaos: An Interdisciplinary Journal of Nonlinear Science , 22(4), 2012.

Jinho Choi, Sivaram Krishnan, and Jihong Park. Koopman autoencoder via singular value decomposition for data-driven long-term prediction. In 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP) , pages 1-6. IEEE, 2024.

Andreas Mardt, Luca Pasquali, Hao Wu, and Frank Noé. Vampnets for deep learning of molecular kinetics. Nature communications , 9(1):5, 2018.

Igor Mezi´ c and Stephen Wiggins. A method for visualization of invariant sets of dynamical systems based on the ergodic partition. Chaos: An Interdisciplinary Journal of Nonlinear Science , 9(1):213-218, 1999.

Samuel E Otto and Clarence W Rowley. Koopman operators for estimation and control of dynamical systems. Annual Review of Control, Robotics, and Autonomous Systems , 4(1):59-87, 2021.

Grigorios A Pavliotis. Stochastic processes and applications. Texts in applied mathematics , 60, 2014.

Rajendra Bhatia and Peter Rosenthal. How and why to solve the operator equation ax- xb= y. Bulletin of the London Mathematical Society , 29(1):1-21, 1997.

Chulan Kwon, Ping Ao, and David J Thouless. Structure of stochastic dynamics near fixed points. Proceedings of the National Academy of Sciences , 102(37):13029-13033, 2005.

Oscar F Bandtlow, Wolfram Just, and Julia Slipantschuk. Spectral structure of transfer operators for expanding circle maps. In Annales de l'Institut Henri Poincaré C, Analyse non linéaire , volume 34, pages 31-43. Elsevier, 2017.

Igor Mezi´ c and Andrzej Banaszuk. Comparison of systems with complex behavior. Physica D: Nonlinear Phenomena , 197(1-2):101-133, 2004.

Steven L Brunton, Bingni W Brunton, Joshua L Proctor, Eurika Kaiser, and J Nathan Kutz. Chaos as an intermittently forced linear system. Nature communications , 8(1):19, 2017.

Tohru Katayama. Subspace methods for system identification . Springer, 2005.

Bernard Widrow and Samuel D Stearns. Adaptive Signal Processing . Prentice-Hall, Englewood Cliffs, NJ, 1985.

D Jeulin. Dead leaves models: from space tesselation to random functions. In Symposium on the Advances in the Theory and Applications of Random Sets , pages 137-156. World Scientific, 1996.

Daniel L Ruderman. Origins of scaling in natural images. Vision research , 37(23):3385-3398, 1997.

Madhura D Ketkar, Burak Gür, Sebastian Molina-Obando, Maria Ioannidou, Carlotta Martelli, and Marion Silies. First-order visual interneurons distribute distinct contrast and luminance information across on and off pathways to achieve stable behavior. Elife , 11:e74937, 2022.

Priyanka Gupta, Dinu F Albeanu, and Upinder S Bhalla. Olfactory bulb coding of odors, mixtures and sniffs is a linear sum of odor time profiles. Nature neuroscience , 18(2):272-281, 2015.

Izumi Fukunaga, Manuel Berning, Mihaly Kollo, Anja Schmaltz, and Andreas T Schaefer. Two distinct channels of olfactory bulb output. Neuron , 75(2):320-329, 2012.

Nixon M Abraham, Hartwig Spors, Alan Carleton, Troy W Margrie, Thomas Kuner, and Andreas T Schaefer. Maintaining accuracy at the expense of speed: stimulus similarity defines odor discrimination time in mice. neuron , 44(5):865-876, 2004.

Naoshige Uchida and Zachary F Mainen. Speed and accuracy of olfactory discrimination in the rat. Nature neuroscience , 6(11):1224-1229, 2003.

Dmitry Rinberg, Alexei Koulakov, and Alan Gelperin. Speed-accuracy tradeoff in olfaction. Neuron , 51(3):351-358, 2006.

Marta Díaz-Quesada, Isaac A Youngstrom, Yusuke Tsuno, Kyle R Hansen, Michael N Economo, and Matt Wachowiak. Inhalation frequency controls reformatting of mitral/tufted cell odor representations in the olfactory bulb. Journal of Neuroscience , 38(9):2189-2206, 2018.

Shaina M Short and Matt Wachowiak. Temporal dynamics of inhalation-linked activity across defined subpopulations of mouse olfactory bulb neurons imaged in vivo. eneuro , 6(3), 2019.

David H Gire, Kevin M Franks, Joseph D Zak, Kenji F Tanaka, Jennifer D Whitesell, Abigail A Mulligan, René Hen, and Nathan E Schoppa. Mitral cells in the olfactory bulb are mainly excited through a multistep signaling path. Journal of Neuroscience , 32(9):2964-2975, 2012.

David B Kastner and Stephen A Baccus. Coordinated dynamic encoding in the retina using opposing forms of plasticity. Nature Neuroscience , 14(10):1317-1322, 2011.

Daqing Cai, Gregory C Deangelis, and Ralph D Freeman. Spatiotemporal receptive field organization in the lateral geniculate nucleus of cats and kittens. Journal of neurophysiology , 78(2):1045-1061, 1997.

David N Mastronarde. Two classes of single-input x-cells in cat lateral geniculate nucleus. i. receptive-field properties and classification of cells. Journal of neurophysiology , 57(2):357-380, 1987.

Jonathan Wolfe and Larry A Palmer. Temporal diversity in the lateral geniculate nucleus of cat. Visual neuroscience , 15(4):653-675, 1998.

Alan B Saul and AL Humphrey. Spatial and temporal response properties of lagged and nonlagged cells in cat lateral geniculate nucleus. Journal of neurophysiology , 64(1):206-224, 1990.

Alan B Saul. Lagged cells in alert monkey lateral geniculate nucleus. Visual neuroscience , 25(5-6):647-659, 2008.

Denise M Piscopo, Rana N El-Danaf, Andrew D Huberman, and Cristopher M Niell. Diverse visual features encoded in mouse lateral geniculate nucleus. Journal of Neuroscience , 33(11):4642-4656, 2013.

Jason J Moore, Alexander Genkin, Magnus Tournoy, Joshua L Pughe-Sanford, Rob R de Ruyter van Steveninck, and Dmitri B Chklovskii. The neuron as a direct data-driven controller. Proceedings of the National Academy of Sciences , 121(27):e2311893121, 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction closely and accurately describe the content of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: While conceptualizing neurons through coherent set clustering offers a novel perspective on neural computation, this framework does not yet account for the role of feedback. In photoreceptor neurons, rectification output appears not to influence synaptic learning, but in other neuron types, learning may depend critically on dendritic backpropagation. Furthermore, the widespread presence of feedback loops in brain circuits highlights the need to incorporate such mechanisms into the broader dynamical systems framework.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs are provided in Section 3 and the methods.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Simulation parameters and pseudocode for Figure 2 are provided in the supplemental material. The procedure to arrive at results in Figure 3 is detailed in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The simulations and analyses performed can be described with words, and are described in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: No training or testing was performed.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars in Figure 3C indicate 95% confidence intervals of the mean, obtained through a binomial fit. No statistical tests were performed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The modest compute requirements are described in the supplemental material. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work conforms in every respect with the NeurIPS Code of Ethics Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Discussed at the end of the paper. The work is foundational research that can facilitate our understanding of the brain and facilitate designing biologically inspired neural networks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Datasets from two previously published studies were used to create Figure 3. These papers are appropriately cited, but the datasets were obtained directly from their creators, and are not publicly licensed assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not crowdsource nor use human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not crowdsource nor use human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core development in this research does not involve LLMs in any aspect.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a theoretical framework conceptualizing sensory neurons as detectors of coherent sets in high-dimensional stochastic dynamical systems. For OU processes, they analytically derive that the subdominant singular function of the stochastic Koopman operator reduces to a linear projection followed by rectification, naturally explaining neuronal temporal receptive fields and the ubiquity of rectification in neural responses. The framework predicts complementary neuronal classes specialized for prediction (contracting coherent sets) versus retrospection (expanding coherent sets), which the authors connect to known functional dichotomies like tufted/mitral cells and lagged/non-lagged cells.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n1. The connection between coherent set detection and neural computation is original and mathematically rigorous.\n2. Analysis of real neuronal data from multiple sensory systems supports theoretical predictions.\n\n### Weaknesses:\n1. Oversimplified dynamics: The analytical results are restricted to linear OU processes; real sensory dynamics are likely far more complex.\n2. Incomplete nonlinear treatment: The extension to nonlinear dynamics via Galerkin projections is mentioned but not thoroughly developed or validated.\n3. Limited scope of data analysis: The empirical analysis focuses on relatively simple temporal receptive field patterns and doesn't address more complex neuronal response properties.\n\nQUESTIONS:\n1. The analytical results are limited to OU processes, but the authors claim the framework extends to nonlinear dynamics. Could authors provide concrete examples with analytical or numerical solutions for specific nonlinear systems relevant to sensory processing?\n2. How does the proposed framework account for the extensive feedback connections in sensory systems? Would feedback fundamentally alter the coherent set structure or the neuronal computations?\n3. What are the computational requirements for neurons to perform the proposed coherent set detection? How does this scale with the dimensionality of the sensory input space?\n4. How do authors distinguish the proposed coherent set framework from other established theories like efficient coding, sparse coding, or predictive coding that also explain temporal receptive fields and rectification?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes that sensory neurons function as detectors of coherent sets, which are defined here as maximally stable binary partitions of the input state space. The paper first formalizes coherent sets in terms of the subdominant singular vectors of the stochastic Koopman operator. It then shows that for OU dynamics the optimal detector is a linear projection onto the dominant singular vector, followed by positive or negative rectifying (for detecting one set or its complement). For unknown and possibly nonlinear dynamics it then proposes a data-driven method based on past-future cross-covariance. Coherent sets for prediction are defined by thresholding in subspaces with contracting dynamics while coherent sets for retrospection are defined by thresholding in subspaces with expanding dynamics. Analysis of temporal receptive fields of biological neurons shows how this framework can classify them into predictive versus retrospective cells.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nSophisticated mathematical approach\nClever and novel bridge between inference over stochastic processes and temporal characteristics of real neurons.\n\nWeaknesses:\n\nEvidence is largely circumstantial. The theoretical framework offers an organization of existing data based on qualitative measures (i.e., the sign of the time constant of the orthogonal exponential function), but does it make quantitative new predictions?\n\nQUESTIONS:\nI'm confused about the association of expanding regions (i.e., directions in state space) with retrospection and contracting directions with prediction. For example take the 2d OU process with A = [[1,0],[0,-1]] and D = [[1,0],[0,1]]. The dynamics are expanding along x_1 and contracting along x_2. IIUC, the coherent sets for prediction are {x: x_2 < 0} and {x: x_2 ≥ 0} while the coherent sets for retrospection are {x: x_1 < 0} and {x: x_1 ≥ 0}. This seems backwards. Isn't it much easier to predict sign(x_1) than sign(x_2)? For example if x_1(0) = 2 then there's a high probability that sign(x_1(t)) > 0 for any t > 0, because of the repulsive dynamics. In contrast, if x_2(0) = 2 then we can't reliably predict x_2(t) for t > 0, because the attractive dynamics will erase the initial conditions and the future trajectory will bounce back and forth around x_2 = 0.\n\nWhat happens when the dynamics are repulsive (or attractive) in all directions? I believe B_tau (F_tau) is still well-defined. If a neuron projects to its top singular vector does the neuron just fail to make meaningful predictions (retrospections)?\n\nI was convinced by the proof on Wikipedia but an academic citation would be more appropriate. Maybe the Bhatia & Rosenthal (1997) reference there would work.\n\nI may not have been the ideal reviewer for this paper but I had to do a lot of secondary reading to understand the mathematical properties of several of the constructions (Perron-Frobenius operator, forward-backward and backward-forward operators, Galerkin projections). Some intuitive guidance in secs 3-4 would probably help other readers.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStochastic processes are typically described by stochastic differential equations (SDEs) or (via the Fokker-Planck equation) a deterministic PDE for the probability density function.  However, they can also be characterized in terms of \"transfer operators\" (Koopman, Perron-Frobenius, etc.) for some fixed amount of time evolution tau (as opposed to the infinitesimal evolution given by the PDEs).  The advantage of these transfer operators is that they can be estimated directly from data: pick a finite set of basis functions; evaluate them at a set of samples X as well as their time-evolved (by tau) counterparts Y; and then compute all second-order statistics.  The approximate Koopman operator, e.g., corresponds to the normal equations (the optimal linear description of the evolution of x in this basis); the forward-backward operator amounts to applying the normal equations once in the forward and once in the backward directions.  Furthermore, the discrete-time, discrete-state counterpart to the SDE is a random walk on a graph; and consequently, spectral clustering methods have counterparts for SDEs, which are  derivable from the transfer operators.  Intuitively, the eigenfunction with the largest eigenvalue corresponds to the most coherent set of points under the trajectory.  Of interest is the *second* eigenfunction, which corresponds to a partition of the state space into the two most coherent regions (the first eigenfunction includes the entire state space, I think).\n\nThe present MS shows that, for Ornstein-Uhlenbeck (OU) processes (\"a reasonable model of summed input to a neuron\"), the second eigenfunctions of the forward-backward (F) and backward-forward (B) operators are linear projections of the state, which can be derived explicitly from the equations for the dynamics.  The F eigenfunction corresponds to expanding coherent sets, and therefore tells us about low-dimensional structure in the past (\"retrospective\"), whereas the B eigenfunction corresponds to contracting coherent sets and tells us about low-dimensional structure in the near future (\"predictive\").\n\nFor neurons to extract this structure, they must estimate the dynamics directly from the data, along the lines sketched above.  The authors propose that neurons project incoming data onto the second singular vector (as shown to be optimal for OU processes) via their vector of synaptic weights, and then rectify the result (or its additive inverse).  The magnitude of this scalar output indicates how strongly the input belongs to one (or the other) of the two coherent sets.  Neurons that project onto the F eigenvector are predictive; onto the B eigenvector, retrospective.\n\nFinally, the authors consider neural data, letting the basis functions be simply a 1D variable at a set of discrete delays.  A standard model of luminance input yields singular vectors that resemble the temporal filters of retinal cells in drosophila.  ON and OFF cells in the early visual system are explained as the rectified responses to the positive and negative projections of luminance.  Tufted and mitral cells in the olfactory bulb are interpreted as predictive and retrospective, respectively.  Consistent with this claim, temporal receptive fields for such cells are shown to be (in many cases) orthogonal to either growing or decaying exponentials (or, somewhat confusingly, both).  Something similar can be said of retinal ganglion cells, although almost all of these are predictive.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a refreshingly different take on neural coding (at least to this reviewer's knowledge), and provides a powerful framework for interpreting the temporal filters of neurons that fits nicely with the electrophysiology of early sensory areas.  The explanation of rectification is also compelling (although perhaps more of a precising of existing intuition than a brand new idea).  The approach has the potential to be a very general explanatory principle for temporal receptive fields.  Nevertheless:\n\nWeaknesses:\n - I am not wholly convinced that the sets of points that cohere in state space are the right low-dimensional signal.  There is not much intuition provided about why (e.g.) the smoothed derivatives of Fig. 2B are good for extracting what is presumably the feature of interest, the luminance itself.  (How would the filters change for a different model of luminance?)\n - It would be nice to make a stronger connection to the historical theoretical work on this topic, especially the literature on efficient coding (as well as the information bottle in refs 1-4), particularly in the retina where the authors provide their own interpretation.  (See e.g. the section on \"Efficient Coding in Retina and Thalamus\" in https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2022.929348/full).  I realize the authors are short on space. \n - The MS is not exactly self-contained (this reviewer had to read ref. 10 first, but perhaps that's my fault).\n\nQUESTIONS:\n- How are we to interpret cells with filters orthogonal to both a decaying and a growing exponential?\n - The purpose of the analytical result is (I take it) to show that the procedure is optimal for at least one plausible kind of stochastic dynamics.  But in the data-driven case, wouldn't one use the linear projection onto the second singular vector even without this proof?\n - The basis used in section 5 is sensible, but in general how does one choose the basis?  It seems that it would be difficult to compare against electrophysiological data without having first nailed this down.\n - Why use CCA in Section 5.2?  (It seems that would be slightly different from finding the eigenvectors of the matrices in Eqn. 22.)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
The authors propose a novel perspective on the functional role of sensory neurons, as "coherent set detectors" that identify regions of the stimulus space that evolve cohesively over time. Through modelling neurons as latents underlying stochastic dynamical systems governing the dynamics of sensory stimuli, the authors show that neurons can detect coherent sets through spectral clustering of the stochastic Koopman operator underlying the dynamic system. For OU processes, the authors show that the subdominant singular basis functions reduce to linear projections. It is additionally possible to extend the framework to nonlinear dynamics with Galerkin projections and CCA for interpreting real neural data.

STRENGTHS AND WEAKNESSES:
Strengths.
- The authors propose a novel theoretical framework connecting dynamical systems theory to neural computation through coherent set analysis, which is an under-explored question in understanding functions of sensory neurons.
- The derivation for OU processes is technically sound, with clear progression from the Koopman operator to implementable neural operations.
- The paper is well-written, with clear demonstrations and easy-to-follow derivations, despite the mathematical complexity.

Weaknesses.
- The extension to non-linear dynamics is largely data-driven, and largely base on the stationarity assumption, whereas real neural dynamics almost always violates such assumption.
- Neuron classification based on orthogonality to exponentials lacks rigorous statistical testing, with no comparison to null hypothesis/model.

QUESTIONS:
- How could neurons biologically learn the required singular vectors? The whitening step requires global covariance - how is this reconciled with local learning rules?

LIMITATIONS:
See above.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-05