Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Time
Authors: Duc Anh Nguyen
Ernesto Araya
Adalbert Fono
Gitta Kutyniok
Venue: ICML 2025 poster (2025)
Keywords: Spiking neural networks
Linear regions
Approximation theory
Primary area: theory->deep_learning

Abstract: Recent years have seen significant progress in developing spiking neural networks (SNNs) as a potential solution to the energy challenges posed by conventional artificial neural networks (ANNs). However, our theoretical understanding of SNNs remains relatively limited compared to the ever-growing body of literature on ANNs. In this paper, we study a discrete-time model of SNNs based on leaky integrate-and-fire (LIF) neurons, referred to as discrete-time LIF-SNNs, a widely used framework that still lacks solid theoretical foundations. We demonstrate that discrete-time LIF-SNNs realize piecewise constant functions defined on polyhedral regions, and more importantly, we quantify the network size required to approximate continuous functions. Moreover, we investigate the impact of latency (number of time steps) and depth (number of layers) on the complexity of the input space partitioning induced by discrete-time LIF-SNNs. Our analysis highlights the importance of latency and contrasts these networks with ANNs that use piecewise linear activation functions. Finally, we present numerical experiments to support our theoretical findings.

Time to Spike?
Understanding the Representational Power of Spiking Neural Networks in Discrete Time
Duc Anh Nguyen Ernesto Araya
Adalbert Fono
Gitta Kutyniok 1234
Abstract
2022). However, it is well-established that the implemen- tation of ANNs in modern applications often necessitates a considerable allocation of computational resources, with hardware requirements exhibiting an unsustainable rate of growth (Thompson et al., 2021). Spiking neural networks (SNNs), often referred as the third generation of ANNs (Maass, 1997a), have been developed as a potential energy- efficient alternative thanks to their event-driven nature in- spired by biological neuronal dynamics and compatibility with neuromorphic hardware (Mehonic et al.. 2024).
Recent years have seen significant progress in developing spiking neural networks (SNNs) as a potential solution to the energy challenges posed by conventional artificial neural networks (ANNs). However, our theoretical understanding of SNNs remains relatively limited compared to the ever- growing body of literature on ANNs. In this paper, we study a discrete-time model of SNNs based on leaky integrate-and-fire (LIF) neurons, referred to as discrete-time LIF-SNNs, a widely used framework that still lacks solid theoretical foundations. We demonstrate that discrete-time LIF-SNNs with static inputs and outputs realize piecewise constant functions defined on polyhe- dral regions, and more importantly. we quantify the network size required to approximate continu- ous functions. Moreover, we investigate the im- pact of latency (number of time steps) and depth (number of layers) on the complexity of the input space partitioning induced by discrete-time LIF. SNNs. Our analysis highlights the importance of latency and contrasts these networks with ANNs employing piecewise linear activation functions. Finally, we present numerical experiments to sup- port our theoretical findings.
Despite the rapid advancements in neuromorphic computing and SNNs (Guo et al., 2023; Yuan et al., 2022; Fang et al., 2023b; Lv al., 2024), our foundational understanding of SNNs remains incomplete. A fundamental characteristic of any model class is its expressivity- both in terms of the functions it can accurately represent or approximate, and the computational efficiency with which these representations can be obtained. Analyzing the representational power of ANNs has been a central concern in deep learning that show- cased their (universal) capabilities (Cybenko, 1989; Hornik et al., 1989; Leshno et al., 1993; Hanin, 2019; Shen et al., 2020; Petersen & Voigtlaender, 2018: Gühring et al., 2020; Bölcskei et al., 2019). The expressivity of SNNs has been discussed by several papers (Maass, 1994; 1997a; Zhang & Zhou, 2022; Singh et al., 2023; Neuman et al., 2024), which derived results comparable to those established for ANNs. However, these works focus on continuous-timg models of SNNs, mostly based on the spike response model (Gerst- ner & van Hemmen, 1992), combined with specific coding schemes such as time-to-fire time-to-frrst-piik first-spike coding (Singh et al., 2023; Neuman et al., 2024) or instantaneous rate coding (Zhang & Zhou, 2022). While there exist implementations of the continuous- morphic hardware (Göltz al., 2021), the setting differs crucially from the commonly applied SNN framework based time discretization (Eshraghian al., 2023; Fang et al., 2023a), which is currently more prevalent due to its straightforward applicability on (commercially) available digital neuromorphic hardware like Loihi 2 or Spinnaker 2 (Orchard et al., 2021; Gonzalez et al., 2023).
1. Introduction
Artificial neural networks (ANNs) have emerged as a fun- damental component of artificial intelligence, exhibiting remarkable achievements across a wide range of applica- tions (Abiodun al., 2018; Sarker, 2021; Choudhary al.,
models on specific analog neuro-
Depparment of Mathematics, Mataiiig-Mximmiianss--- Universittt München, Germany "Munich Center for Machine Learning (MCML), Munich, Germany 'Institute of Robotics and Mechatron- ics, DLR-German Aerospace Center, Germany 'Department of Physics and Technology, University of Tromsø, Tromse, Norway, Correspondence to: Duc Anh Nguyen <danguyen @
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Therefore, this paper directly focuses on the computational power and expressivity of SNNs designed with discrete

---

Time to Spike? Understanding the Representational Power of SNNS
time dynamics. Based on the simple and computationally efficient leaky-integrate-and-fff (LIF) neuron model (Ger- stner et al., 2014), we mathematically formalize and in- vestigate the discrete-time LIF-SNN, a framework formally introduced in Section 2, that accommodates a broad range of practically employed model classes. Due to their prevalence, our focus is on LIF neurons but principle, our framework can be easily extended different types of integrate-and -nd-fire models and beyond.
In low-latency SNNs with a narrow first hidden layer, adding neurons deeper layers yields limited accuracy gains compared to ANNs. In contrast, high-latency SNNs benefit from richer input partitioning, leading to significant improvements. We also experimentally demonstrate how the temporal dimension shapes paral- lel hyperplanes as the parameters of our discrete SNN model vary.
In SNN literature (Gerstner et al., 2014; 4; Maass, 1997a), the LIF model is usually referred to as special case of the spike response model, while the discrete-time setting is obtained by discretizing the continuous-time framework. However, discrete-time LIF-SNNs are shown to preserve expressive power that is comparable to the more general continuous-time models. This is true although discrete-time LIF-SNNs inherently realize discrete functions, meaning that the input space is partitioned into polyhedral regions, i.e. regions with linear boundaries, associated with constant outputs.
Related work. While ANN expressivity has been exten- sively studied, research on this topic in SNNs remains lim- ited. Exceptions include (Maass, 1994; 1995; 1996a;;; 1997a;b; Comsa al., 2020; Mostafa, 2018; Singh et al., 2023; Neuman et al., 2024) that analyze approximation properties of continuous-timg SNNs based on temporal coding and the spike response model. However, these ap- proaches differ conceptually from our model, which aligns with the widely used discretized implementation framework (Eshraghian et al., 2023; Fang et al., 2023a). In case of single time step, our model simplifies to Heaviside (or threshold) ANNs, for which approximation results have been established (Leshno et al., 1993; Anthony, 2001). How- ever, our results emphasize the rates of approximation in terms of the number of neurons.
Our goal to elucidate the internal mechanisms of discrete- time SNNs, highlighting their capacity in a learning frame- work. We analyze their expressivity through approximation properties and input partitioning into linear regions (Mon- túfar et al., 2014), focusing on the case of static data common benchmark in SNN research (Eshraghian et al., 2023).
The expression of constant functions on polyhedra by dis- cretized LIF networks and the generation of parallel hyper- planes by single neurons with increased latency are high- lighted in (Kim et al., 2022), motivating their neural archi- tecture search algorithm. but without offering thcoretical insights. In Heaviside ANNs, bounds on linear regions have been established (Khalife et al., 2024), though they do not address the role of time as our results do. A more detailed discussion of related works on the expressive power and properties of ANNs and SNNs is provided in Appendix D.
Contributions. We demonstrate that discrete-time LIF- SNNs partition the input space differently from ReLU- ANNs, with the temporal aspect-the the third dimension along- side width and depth- pllyipglynnggg crucial role. Our main contributions are:
We show that discrete-time LIF-SNNs are universal approximators of continuous functions on compact do- mains by demonstrating that they realize piecewise con- stant functions with polyhedral regions and, conversely, can express any such function in a single time step. Ad- ditionally, we establish upper and lower bounds on the number of neurons required, yielding order-optimal approximation rates.
2. The network model and its neuronal dynamics
SNNs in discrete time are computational models that pro- cess time series data (a((()))(T) and produce binary spike outputs (s(t))EE(T) where s(t) E (0, 1}. Similar to con- ventional ANNs, SNNs consist of neurons organized in graph, with dynamics inspired by biological neurons. Each neuron maintains a membrane potential u(t), evolving over time based on input contributions. When the membrane potential exceeds threshold, the neuron fires, producing a spike encoded as 1, followed by a reset mechanism.
We formalize constant (polyhedral) regions in discrete- time LIF-SNNs and establish a tight upper bound their number. Although they could theoretically grow exponentially with time, we surprisingly show that they scale quadratically. Crucially, we show that the temporal dimension introduces parallel hyperplanes forming the boundaries of the polyhedral regions for each neuron in the first hidden layer, while subsequent layers do not increase the number of regions.
This work focuses on discrete-time formulation of leaky integrate-and and-fire (LIF) neurons, a widely used model that approximates membrane potential dynamics with a resistor- capacitor analogy. Our formulation builds on and formalizes existing frameworks, providing a structured representation that incorporates encoding and decoding schemes, facili-
Our empirical results support the theoretical analysis.
2

---

Time to Spike? Understanding the Representational Power of SNNS
tating both analysis and implementation. Details on the biological motivations and connections to continuous-time dynamics are provided in Appendix E.
Remark (Learnable parameters). For an SNN satisfy- ing Definition 2.1, the learnable parameters include ((Wé, b¹), (0), ))) sELL) along with the decoder D, which is typically parameterized (as specified in Defini- tion 2.2). In fact, the learnability of temporal parameters, namely (u' (0). 3 )), is flexible and varies across imple- meniations. Earlier works often keep these parameters fixed, while more recent studies propose making them learnable (Fang et al., 2021; Shen et al., 2024).
The key properties required for the definition of the discrete- time LIF-SNN model considered in this paper are as fol- lows:
1. Network (spatial) architecture (L, n) E E x N Neurons are arranged in layers with information being propagated from input to output layer feed-forwardly, where L is the number of hidden layers (referred to as the depth) and n (no..... RL++1 is the number of neurons in each layer (referred to as the width) with input dimension rig := Tlin and output dimension n1+1 == out
Remark (Extensions). Our definition of discrele-time LIF- SNNs decomposes the model into modular components (1-4 above), each of which can be flexibly modified or gener- alized. In particular, the neuronal dynamics can be easily adapted to incorporate alternative integrate-and and-fire or even more advanced models, balancing biological plausibility and computational efficiency. Additionally, the model can accommodate recurrent network architectures in place of the feed-forware architecture, and the encoding schemes can be extended to handle time-series data in RR XT, rather than 'static' inputs in T.
2. Neuronal (temporal) dynamics. T E N denotes the number of time steps, also referred as the network's latency. For each hidden layer E [Tt], the spike (ac- tivation) vector 8f (t) E 1)"r and the membrane potential vector u (t) E R at time step t E [T] are given by
Coding schemes. In this paper, we assume a discrete-time LIF-SNN with direct encoding and membrane potential out- puts, which are general enough to capture a wide range of applications including image classification, while also al- lowing us demonstrate our theoretical insights. In direct encoding, the core idea is to directly input analog signals without converting them into binary spike trains. This ap- proach, appearing under different names (Rueckauer et al., 2017; Wu et al., 2019; Fang et al., 2021), turned out as a practical alternative to more biologically plausible schemes (Rathi & Roy, 2023). The idea behind membrane potential outputs is often explained as adding one more layer after the last spike layer without firing and threshold mechanism (Henkes et al., 2024; Eshraghian et al., 2023). This is equiv- alent to adding an affine layer to convert the binary spike activations into real-valued outputs, a technique also com- monly applied in non-spiking ANNs. Please see Appendix E. for more details.
(1) = (1 - 1) + W W' (1) b 0²1 1mm))) u (t) = =0=0//n
W (t) b² st))
(1)
where H is the Heaviside function (applied entry-wise) and (t))see)))) are the initial spike activations. The remaining parameters including W b' are defined below.
3. Coding schemes E R R and D (0, 10,1)mhxT - R where the input encoding E maps an input vector æ E R to a time se- ries (a(()) re:yy representing the initial spike activa- tions and the output decoding D maps any time se- ries (s(1) cem {0, 10,1)" to an output vector D ((s(()) (E(2)) E RRM.
4. (Hyper)parameter (WWe, bO), (u' (0), )))) rE(r.] where the weight matrices W' E - and bias vectors b' E T represent the parameters commonly used in ANNs, whereas the initial membrane poten- tial vector u (0) E Rac,, the leaky term f E 1] and the threshold of (0, oo) represent SNN-specific temporal (hyper)parameters.
Definition 2.2 (Direct input encoding, membrane potential decoding). Let E RT + RT.xTT and D : {0, 10,1)% 1) + F be an encoder and decoder for a discrete-time LIF- SNN, respectively. The encoder E employs a direct encod- ing scheme if
E(x)(()= = x ET).
Definition 2.1 (Discrete-time LIF-SNN), An SNN in the discrete-time LIF model is given by the tuple
The decoder D relies on membrane potential outputs if
= ((W", bb)) rE(D) )eEE)))))))) (u' (0), 8 8t) re(p T, (F. and realizes the mapping R( Th + Tr according
((((()) (a()) = Σ (. v((()= +
to (I):
(x)) = ((stt()) ) with 8 = E(x)).
for a = (aI,..., @T) E ERT, c Ba,, and V E R
3

---

Time to Spike? Understanding the Representational Power of SNNS
3. Structure of computations in SNNS
side ANNs of arbitrary depth. We leverage the insights from (Khalife et al., 2024) enhance the universal approxima- tion property for discrete-time LIF-SNNs (which for T = 1 are equivalent to Heaviside ANNs) and derive approxima- tion rates for Lipschitz continuous functions (on compact domains). We recall that a function f : C "h + R is r-Lipschitz with respect to the Il>o norm if
Our goal is to deepen the understanding of the computa- tional framework underlying the discrete-time LIF model. We begin by examining fundamental properties that arise directly from its definition, followed by more in-depth analysis of its computational structure, focusing on (func- tional) approximation capabilities and input partitioning.
[f(z) - f(w) - F| ylloo for all x, y E Ω
3.1. Elementary properties
where the -nom is defined as || = (x) lmal-
One immediately observes an intrinsic recursiveness of the model even without explicit recursive connections in the spatial feedforward architecture (Neftei et al., 2019). This inherent statefulness property indicates the ability of this model to process dynamical (neuromorphic) data effectively (Rathi et al., 2023). Nevertheless, we study SNNs in a simpler setting with static data carve out the underlying properties.
Theorem 3.2. Let f be a continuous function on compact set S C IR=.. For all & >> O, there exists a discrete-time LIF- SNN & with direct encoding, membrane potential output, I. = 2 and = such that
[R( - f1lo0
(2)
Moreover, if is I-Lipschitz, then & can be chosen with width parameter n = (n1, given by
In the considered setting, discrete-time LIF-SNNs are closely linked to ANNs with Heaviside activation func- tion; see Appendix E.6 for more details. Crucially, for T discrete-time LIF-SNNs are equivalent to Heaviside ANNs meaning that discrete-time LIF-SNNs can realize any Boolean function and approximate continuous functions to arbitrary degree inheriting these properties from Heav- iside ANNs (Leshno et al., 1993; Anthony, 2001). More exactly, we conclude that one-layer (i.e., L = 1) discrete- time LIF-SNNs possess the universal approximator property for continuous functions, which similarly as in the case of ANNs can be extended to arbitrary T.L N by the following result.
diam (92
n1 = max
1 +1
Rins
diimmo (12)
122 = max
r
1
(3)
6
where diam (92) = yu |
ylloo.
Proof sketch. The proof consists of showing the following two statements: (1) continuous functions can be approx- imated by step functions, and (2) step functions can be realized by discrete-time LIF-SNNS. More precisely, as- suming w.l.o.g. that Q c [-K, K]nd " for some K E IR we prove that I can be approximated to arbitrary precision by function that is constant on partition of [-K, KJ"> into hypercubes {C, ]m i..., f Σ filc,- Next, show that there exists discrete-time LIF-SNN & that realizes J. i.e., R(क)= = f, using exactly the number of neurons given in equation 3. The complete proof is given in Appendix B.
Proposition 3.1 (Identity mapping). Let & be discrete- time LIF-SNN with latency T TEN, N and I. E N layers with constant width n E N in each layer. Then, there exists a configuration of parameters such that (s](()) = for any 8 E 1
Proof sketch. It suffices to consider the case L = 1. is straightforward to verify that
Remark (Lipschitz condition). The Lipschitz assumption primarily serves to make the result more explicit but is not essential. Since f is continuous on a compact set, it is uniformly continuous, allowing us to replace diam (50) r
+ = (W, b),
with W- = (1 + =) In for 0- < F. b 0, (0) = 0,
1, and 1 satisfies (s [E[2] 0 for any
with diam (12) where wi (s) = inf w(t > s is the wl (e)
80 E (0, 13 n Details are provided in Appendix B.I. 0
generalized inverse of the modulus of continuity
3.2. A Approximation properties
Notably, when r = 0, i.e., J is constant, equation 3 yields a width parameter of n = (2min, 1). Here, 2nmn corresponds to the minimum number of hyperplanes required to define a hypercube covering St
(Khalife et al., 2024) extended the observation that Heavi- side ANNs compute piecewise constant functions on poly- hedra, analogous to how ReLU ANNs compute continuous piecewise linear functions, by proving that two-layer Heavi- side ANNs can represent any function realizable by Heavi-
Remark (Approximation rates). Theorem 3.2 provides ex- plicit convergence rates terms of both the width parameter
4

---

Time to Spike? Understanding the Representational Power of SNNS
n and input dimension Dig as shown in equation 3. This represents improvement over traditional universal ap- proximation results, which typically lack quantitative guar- antees. While (Khalife et al., 2024) recently established bounds on neural network size, our result differs in two crucial aspects. First, their analysis focuses on represent- ing functions that ure constant on polyhedrul pieces, rather than approximating general continuous finnctios. Second, Mhie
While discrete-time LIF-SNNs and Heaviside ANNs are not equivalent for T > 1, they maintain structural simi- larities: discrete-time LIF-SNNs can be reformulated as Heaviside ANNs with block-diagone weight matrices (re- peated T times) and time-varying bias terms that depend on initial spike activation (see equation 18 in Appendix E.6), This raises a fundamental question about how the trade-off between constrained weights and time-dependent biases af- fects the computational capabilities of discretized LIF SNNs relative to Heaviside ANNs-a a question we address through input partitioning.
their bound exhibits quadratic scaling in diam, (53)
whereas our bound achieves linear scaling in this term.
Our next results tell that there exist (Lipschitz) continu- ous functions for which reducing significantly the width of the first hidden layer provokes significant increase in the approximation error. This demonstrates that our bound on the required number of neurons is optimal up to constant factors in the worst case. Specifically, we can construct continuous functions for which this number of neurons is necessary for achieving the desired approximation accuracy, establishing a matching lower bound. We prove it in the special case of Ω2 = [0, I]. the proof is deferred to Appendix B.I.
4. Input space partitioning complexity of discrete-time LIF-SNNS
The seminal works (Pascanu et al., 2014; Montúfar et al., 2014) pioneered research on the complexity of input par- titioning in ANNS to address model expressivity in deep learning (see Appendix for details). The direct corre- spondence between a discrete-time LIF-SNN's realizable values and its space partitions suggests that partition com- plexity even better characterizes SNN computation than the corresponding results in ANNs. While the partitioning mechanisms of ANNs are well understood, this question re- mains largely unexplored for SNNs, with only experimental insights from (Kim et al., 2022).
Proposition 3.3. For any I > define f: [0, 1] + I by f(z) = Tr. For any E (0, 1], there exists discrete-time LIF-SNN 4 with T 1, L = 2 and n1 [T/=] + 1, such that
[ fllo0 S,
but any discrete-time LIF-SNN 4' with T = 1, and first layer width n < TT/E" 1, with a < 1, will satisfy
| |R( (') - fl 2 20a
Remark (Multidimensional case). Although we have not formally proven it, we believe our result exlends the malti- dimensional case. A weaker version of Proposition 3.3 holds for any nij E N, as the construction in Theorem 3.2 remains order-optimal for certain continuous functions, specifically Lipschitz functions funnciins depending on a single coordinate. Refin- ing this condition in the multivariate setting is left for future work.
(a) = 1,8=1
(b) = 1,8=2 2
Our insights into the computational structure so far still lack in describing two important aspects: What exactly is the contribution of the depth L and the latency T to the com- putational capacity, respectively? When T 1. our model reduces to standard Heaviside ANNs, for which the role of depth has been thoroughly analyzed (see (Khalife al., 2024))Theorem IJ). While these networks, regardless of depth. can only realize functions that are piecewise constant on polyhedra, a fundamental question remains: Can deeper architectures achieve faster convergence rates or provide more precise approximation guarantees?
(0)==2,(=1
(d) 2,6
Figure I. Each color represents distinct constant value of R(+) in the corresponding region. At ( = 2, neurons define parallel hyperplanes, while e 2 some regions merge.
This section establishes a theoretical framework for under- standing input patterns in discrete-tima LIF-SNNs. We extend the concepts of activation patterns and regions from
5

---

Time to Spike? Understanding the Representational Power of SNNS
(Hanin & Rolnick, 2019b) to the discrete-time LIF-SNN setting.
input space. Indeed, the spike activation o ()) of an arbitrary hidden neuron & can be shown to be equal to
Definition 4.1 (Regions). Let o be discrete-time LIF- SNN and denote by 8² (t;x)) the spike activations 8 (t) E {0.1) in the first hidden layer i E [T] given an input vector x E 1.... The activation patterns of 4 for a are defined as
H w x + by ('u - (1+ > Bi -1)
Σ1=1 1==0 3
(4) =: (t.
A(x) = (8 (t:a))) (ETT E [0,11"MXX,
where wy and by are the k-th row and element of W and b. respectively, with the layer indices omitted for convenience (see Appendix B.2.2 for the detailed derivation). Two key observations follow: (1) Since the spatial transformation (W,b) b) is shared across all time steps, all induced hyper- planes are parallel, whether they originate from different time steps or distinct spike activations of previous steps, as highlighted by the shift term Jt in equation 4. (2) With 2 possible spike sequences in {0, (0,1),-1, neuron at time t may correspond to up to 2 hyperplanes. This results in + ++E== 2 = 2 input space regions, each mapped to a unique spike pattern (su(()) E 1)", matching the cardinality of 10, However, we show that the actual number grows only quadratically with T.
The activation regions of i are defined as the (non-enug con-emppy) sets of input vectors that lead to an activation pattern A:
R(A) = {x A(xx) = -A--
Finally, the constant regions of & are defined as the sets of imput vectors that lead to the same output vector y E R
Cly) = fxr E TM (x)) =y)
We study the cardinality of the sets
R = (R(A): : AE {0, 1 is activation pattern of }).
Lemma 4.2 (Temporal input separation). Consider an arbi- trary neuron l in the first hidden layer of a discrete-time LIF- SNN with T time steps. Then k partitions the input space TS via parallel hyperplanes into at most T²+T+2 regions, on each of which the time series (s)(t) LE(T takes differ- ent value in {0, 10 This upper bound on the maximum number of regions is tight in the sense that there exist values of the spatial parameters ug (0), 0 IR [0, R.+ such that the bound is attained.
and
C: (==(C((()(E))) : E }
referee as the number of activation regions and the num- ber of constant regions, respectively. Here, activation re- gions are maximal input subsets yielding identical spike patterns in the first hidden layer, while constant regions are maximal subsets yielding identical outputs.
2
Remark (Activation versus constant regions). For any ac- rivation region (A), all inputs x E (R) yield the same spike pattern (s ())z)) ()=r)) [T[T] = A in the first hidden layer, resulting in identical inputs to subsequent layers. Therefore, each activation region maps to a unique constant outpul, implying C _ R. However, multiple activation regions may map to the same output, potentially reducing the number of constant regions.
The proof of the statement is provided in Appendix B.2.2. Next, we aim to refine the estimation of activation and con- stant regions in discrete-ting time LIF-SNNs by incorporating (1) the spatial arrangement of hyperplanes from different neurons in the first hidden layer and (2) the effect of deeper layers. The first requires reconsidering Zaslavsky's theorem, as the hyperplanes are not in general position. The second follows naturally from the observation that increasing depth does not add activation regions; see also Figure 1. The proof of the next statement can be found in Appendix B.2.4.
For shallow ANNs with piecewise linear activations (e.g., ReLU or Heaviside), the input partition can be characterized geometrically via hyperplane arrangements (Pascanu et al., 2014). Each hidden neuron corresponds to a hyperplane in forming an arrangement whose number of regions is determined by Zaslavsky's 's theorem (Zaslavsky, 1975; Stan- ley, 2011), a key result in enumerative combinatorics. The maximum number of regions created by an ANN with n hid- den neurons is 2n for I 7m and En (?) for n Tin- achieved only when the hyperplanes are in general posi- tion. We refer to Appendix B.2 for formal discussion of hyperplane arrangements and the notion of general position.
Theorem 4.3 (Maximum number of regions). Consider the set V of discrete-time LIF-SNNs with T time steps, imput dimension Thm and n neurons in the first hidden layer. Then the number of constant and activation regions generated by any & E V is upper bounded by
EE
T 2
(n:)
if 2 Puns
I S R
T2+T+2
otherwise.
In the case of discrete-tin time LIF-SNNs, each neuron in the hidden layer corresponds to multiple hyperplanes in the
Moreover, the upper bound is tight: There exist certain & e Vi that attain the bound with appropriate configurations of
6

---

Time to Spike? Understanding the Representational Power of SNNs
the network parameters. In particular, for the first inequality become equality & must have at least To neurons in each layer.
Remark (Comparison with ReLU). The maximum num- ber of activation regions for shallow ReLU or Heaviside ANNs scales as either (22*),, i.e., exponential in ny when n1 < Tiie- or (n)"), i.e., polynomial in nı when nı >> with nman treated constant ( (Pascanu et al., 2014). For shallow discrete-ting LIF-SNNs, we instead obtain the scal- ing behaviour ( and O (T²) respectively. The multiplicatiive factor T indicates that shallow discrete- time LIF-SNNs with high latency T generate signifi- cantly more activation regions than shallow ReLU ANNs.
80
300
(a) Training accuracy achieved an ANN and SNNs with different numbers of time steps but identical spatial architectures
T-1
However, the maximum number of activation regions in discrete-time LIF-SNNs does not scale with depth L as does in ReLU ANNs. While the number of activation regions in ReLU ANNs increases exponentially with depth (Montúffar et al., 2014; Serra al., 2018; Arora et al., 2018), the exponential growth is based n,, typically a value even smaller than Therefore, the growth of activation regions in discrete-time LIF-SNNs with latency is much faster than the growth in depth deep ReLU ANNs.
0:50
Twa
3.20
3.15
linear dependence
800
Wim
(h) The improvement of training accuracy when increasing the width of the subsequent hidden layers.
This observation highlights the distinct data-fitting ap- proaches of discrete-time LIF-SNNs and ReLU ANNs. In ReLU ANNs, increasing depth and width refines the input partition, improving training data interpolation. In contrast, the input partition discrete- time LIF-SNN is fixed with depth but detailed from the first hidden layer, given suffi- cient latency T. While adding more layers doesn't 't refine the input partition, it can improve the accuracy of the func- tion realized by these layers, which combine regions and map to the output. Although one additional Heaviside layer could suffice, its neuron count would be large compared to deeper architectures, similar to expressing a boolean func- tion with negation and OR operations, achievable through linear combinations of neuron outputs.
Figure 2. Comparison of train accuracies achieved by ANN and SNNs with different numbers of time steps. Both types of networks share the same spatial architectures: the first hidden layer is bottleneck with only 20 neurons and the subsequent layers are progressively widened in each experiment. We consider 4 layers in both cases.
neurons in subsequent layers ranges from 20 to 200. Due to the bottleneck layer, all networks are limited in their ability to fit the training data, even after extensive training, pre- venting full interpolation. Additionally, increasing latency in SNNs consistently improves training accuracy, aligning with Theorem 4.3, which shows that the representational complexity of SNNs, measured by the maximum number of generated regions, increases with T.
5. Experimental results
5.1. First hidden layer as a bottleneck layer
While increasing T in SNNs quite consistently enhances the training accuracy, the particular gain differs in each setting. For very small values of T, growing the width of subsequent layers from 20 neurons to 200 neurons only improves the training accuracy by approximately 7%. For
In this subsection, we conduct experiments to assess the contribution of subsequent hidden layers to network expres- sivity. Specifically, we compare ANNs and SNNs (with varying time steps). For each model, we have four layers and we first set the first hidden layer as bottleneck, with small number of neurons that limits expressivity. We then progressively increase the width of subsequent layers to evaluate how this affects network expressivity.
T 4 and T 8, the gain in training accuracy becomes much higher, almost reaching 20%. Nevertheless, this still lags behind the gain of ANN (approximately 21%) and of SNN with T = 16 (approximately 30%) on average.
Fig. 2 presents results for CIFARIO classification, where the first hidden layer is set to 20 neurons, and the number of
This phenomenon can be explained by the fact that discrete- time LIF-SNNs generate regions only in the first hidden

---

Time to Spike? Understanding the Representational Power of SNNs
layer, with subsequent layers merging regions rather than augmenting their number. In the low-latency regime, the first hidden layer may not generate sufficiently rich input partition, limiting the ability of later layers, even with high capacity, to form effective decision boundaries.
Figure 4. Table I compares the number of regions per layer across five random initializations (see Appendix C) against
the theoretical upper bound. As expected, the number of
pre training
post-training
theory
T
R4
f=
2
{=
1
f = 2
f f=
5.2. Shifting of parallel hyperplanes
2
3
4
3
4
3
4
7
3
7
We analyze the hyperplane shifts 9t-1 defined in equation 4 for a single neuron in the first hidden layer. Fixing the initial membrane potential (0),,, threshold 0, and bias by, we compute (9t-1 (1)((),.... sk(1 1))) t=1 for given values of (Wx, æ) bb and B. When B 1, the shifts appear to converge to (w,, a) + by with some oscillation at large t,
4
10
4
6
4
11
2
16
14
4
3
16
3
14
10
5
37
14
12
67
Table 1. Maximum number of regions pre versus post training
as shown in Figure 3 for (w), x) + by 0.7 (this behavior persists across different values in [0, 1). In contrast, with 0.8, the shifts exhibit periodic pattern up to numerical precision.
For certain values of (e.g., = 0.8), the shift term It-1 takes only few distinct values due to periodic behavior, pattern not observed for 3 = 1, where the values do not repeat, but their differences diminish over time. Thus, al- though the number of regions grows with T, their widths appear to vanish with T, raising questions about the model's effective capacity. In the bottleneck experiments, see Fig. 2a, latency appears to influence final accuracy, possibly because the small T value maintains relatively large oscilla- tions in 9t-1 and thus wider regions. This opens the question about the utility of considering larger values for T. How- ever, accuracy also depends on factors not analyzed here, such as training process and data complexity, presenting an intriguing direction for future research.
-1
-1
Figure 4. Illustration of the toy dataset consisting of points in [-1, from two classes separated by the yellow line vi- sualized by color.
regions often decreases after training, the model is trying to fit the very simple decision boundary of this toy dataset. A natural extension of our theory will be to prove whether randomly initialized discrete-time LIF-SNNs achieve or approximate the theoretical bound in Theorem 4.3.
8-1
0.8
0.7
6. Conclusion
As core component of neuromorphic computing, SNNs have made rapid practical strides in recent years. We complement this progress with theoretical insights into their computational power, particularly for discretized-time implementations. While the low-latency regime's s equiv- alence to Hcaviside-activan the model's distinctive features emerge the high-latency regime. Here, SNNs differ not only from Heaviside ANNs but also from ANNs with piecewise linear activations due to the impact of depth. An important question is whether and when these computational distinctions can be exploited in practical applications. Increasing latency for static tasks may improve SNN performance, which however not
20
20
10
70
60
Time
Figure Value of 9 3 and and (w, 2) + 0.7.
ANNs is well understood,
5.3. Counting regions: pre-training versus post-training
We consider a discrete-time LIF SNN with 2 and 2, and count the number of linear regions before and after training on a linearly separable dataset depicted in
8

---

Time to Spike? Understanding the Representational Power of SNNS
given in light of our numerical experiments, but at the cost of higher energy consumption, which depends on spike den- sity (Dampthoffer et al., 2023; Lemaire et al., 2023). Thus, the true potential of SNNs may lie in dynamic, event-driven tasks, an area still lacking rigorous theoretical analysis and requiring further exploration.
Recent advances and applications of deep learning meth- ods in materials science. npj Computational Materials, 8 (1):59, Apr 2022.
Comsa, I. M., Potempa, K., Versari, L.,, Fischbacher, T., Gesmundo, A., and Alakuijala, J. Temporal coding in Spiking neural networks with alpha synaptic function. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1229-8533, 2020.
Acknowledgements
The authors acknowledge support by the project "Next Gen- eration AI Computing (gAln)", which is funded by the Bavarian Ministry of Science and the Arts (SMMWK Bayern) and the Saxon Ministry for Science, Culture and Tourism (SMWK Sachsen).
Cybenko, G. V. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2:303-314 1989.
Dampfhoffer, M., Mesquida, T., Valentian, A., and Anghel, L. Are SNNs really more energy-efficient than ANNs? An in-depth hardware-aware study. IEEE Transactions Emerging Topics in Computational Intelligence IEEE, 7(3):731 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Eshraghian, J. K., Ward, M., Neftei, E. O., Wang, X., Lenz, G., Dwivedi, G., Bennamoun, M., Jeong, D. S., and Lu, W. D. Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE, 111(9): 1016-1054, 2023.
References
Abiodun, O. I., Jantan, A., Omolara, A. E., Dada, K. V., Mohamed, N. A., and Arshad, H. State-of-the-art in artificial neural network applications: survey. Heliyon, 4(11):e00938, 2018.
Fang, W., Yu, Z., Chen, Y.. Masquelier, T., Huang, T., and Tian, Y. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2641-2651, 2021.
Anthony, M. Discrete Mathematics of Neural Networks. Society for Industrial and Applied Mathematics, 2001.
Fang, W., Chen, Y., Ding, J., Yu, Z., Masquelier, T., Chen, D., Huang, L., Zhou, H., Li, G., and Tian, Y. Spikingjelly; An open-source machine learning infrastructure platform for spike-based based intelligence. Science Advances, 9(4)): eadi Milll- 1480, 2023a.
Arora, R., Basu, A., Mianjy, P., and Mukherjee, A. Under- standing deep neural networks with rectified linear units. In International Conference on Learning Representations, (ICLR), 2018
Balestriero, R. and Baraniuk, R. A spline theory of deep learning. In Proceedings of the International Conference on Machine Learning (ICML), volume 80, pp. 374-383, 2018.
Fang, W. Yu, Z., Zhou, Z., Chen, D., Chen, Y., Ma, Z., Masquelier, T., and Tian, Y. Parallel spiking neurons with high efficiency and ability to learn long-tern dependen- cies. In Conference on Neural Information Processing Systems (NeurIPS), 2023b.
Balestriero, R., Cosentino, R., Aazhang, B., and Baraniuk, R. G. The geometry of deep networks: power diagram subdivision. Advances on Neural Information Process- ing Systems (NeurIPS), 2019.
Gerstner, W. and van Hemmen, J. L. Associative memory in network of 'spiking' neurons. Network: Computation in Neural Systems, (22::139-164 1992.
Bölcskei, H., Grohs, P., Kutyniok, G., and Petersen, P. Opti- mal approximation with sparsely connected deep neural networks. SIAM Journal on Mathematics of Data Science, 1(1):8-45, 2019.
Gerstner, W., Kistler, W. M.. Naud, R., and Paninski, L. Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition. Cambridge University Press, USA, 2014.
Choudhary, K., DeCost, B., Chen, C., Jain, A., Tavazza, F, Cohn, R., Park, C. W., Choudhary, A., Agrawal, A., Billinge, J. L.,, Holm, E., Ong, P., and Wolverton, C.
Gonzalez, H. A., Huang, J., Kelber, F., Nazeer, K. K., Langer, T. H., Liu, C., Lohrmann, M. A., Rostami, A., Schöne, M., Vogginger, B., Wunderlich, T., Yan, Y., Akl,

---

Time to Spike? Understanding the Representational Power of SNNS
M., and Mayr, C. SpiNNaker2: A large-scale neuro- morphic system for event-based and asynchronous ma- chine learning. In Machine Learning with New Compute Paradigms, 2023.
Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neu- ral Networks, 2(5):359-36, 1989.
Huchette, J., Muñoz, G., Serra, T., and Tsay, C. When deep learning meets polyhedral theory: A survey. arXiv: 230500200 2023.
Grigsby, J. E. and Lindsey, K.
On transversality of bent hyperplane arrangements and the topologiall expressive- ness of relu neural networks. SIAM Journal on Applied Algebra and Geometry, 6(2):216-242, 2022.
Humayun, A. I., Balestriero, R., and Baraniuk, R. Deep net- works always grok and here is why. In High digensiimelt Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024.
Gühring, I., Kutyniok, G., and Petersen, P. Error bounds for approximations with deep ReLU neural networks in WaP norms. Analysis and Applications, 18(05):803-89) 2020.
Jentzen, A., Kuckuck, B., and von Wurstemberger, P. Math- ematical introduction to deep learning: Methods, imple- mentations, and theory. arXiv:2310. 20.::233000000 20360, 2023.
Guo, Y., Huang, X., and Ma, Z. Direct learning-based deep spiking neural networks: a review. Frontiers in Neuroscience, Volume 17 2023, 2023.
Khalife, S., Cheng, H., and Basu, A. Neural networks with linear threshold activations: structure and algorithms. Mathematical Programming, 206(1):333-356 2024.
Göltz, J., Kriener, L., Baumbach, A., Billaudelle, S., Bre- itwieser, O., Cramer, B., Dold, D., Kungl, A., Senn, W., Schemmel, J., Meier, K., and Petrovici, M. Fast and energy-efficient neuromorphic deep learning with first- spike times. Nature Machine Intelligence, 3:823-835, 2021.
Kim, Y., Li, Y., Park, H., Venkatesha, Y., and Panda, P. Neural architecture search for spiking neural networks. In Proceedings of the European Conference on Computer Vision( ECCV), pp. 36-56, 2022.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, ICLR 2015,, 2015.
Hanin, B. Universal function approximation by deep neural nets with bounded width and ReLU activations. Mathe- matics, 7(10), 2019.
Kutyniok, G., Petersen, P., Raslan, M., and Schneider, R. A theoretical analysis of deep neural networks and para- metric pdes. Constructive Approximation, 55(1):73-125, 2022.
Hanin, B. and Rolnick, D. Complexity of linear regions in deep networks. In Proceedings of the International Conference on Machine Learning (ICML), volume 97, pp. 2596-2604, 2019a.
Lee, J. H., Delbruck, T., and Pfeiffer, M. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10, 2016.
Hanin, B. and Rolnick, D. Deep relu networks have sur- prisingly few activation patterns. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019b.
Lemaire, E.,, Cordone, L.,, Castagnetti, A., Novac, P.-E., Courtois, J., and Miramond, B. An analytical estimation of spiking neural networks energy efficiency. In Neural Information Processing: ICONIP 2022, pp. 574-587. Springer International Publishing, 2023.
Har-Peled, S. and Jones, M. On separating points by lines. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Sym- posium on Discrete Algorithms, SODA '18 pp. 918-932. Society for Industrial and Applied Mathematics, 2018.
ISBN 9781611977 1975031.
Leshno, M., Lin, V. Y., Pinkus,
A., and Schocken, S. Mul- tilayer feedforward networks with nonpolynomial ac- tivation function can approximate any function. Neural Networks, 6(6):861 1667, 1993.
He, K., Zhang. X.,
Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance im- agenet classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV "15, pp. 1026-1034, USA, 2015. IEEE Computer Society. ISBN 9781467383912 doi: 10. 10/1 123. URL /ttp::///oi..org/10.10000 11t.//1....1000000 2015. 123.
Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. The expres- sive power of neural networks: a view from the width. In Proceedings of the International Conference on Neural In- formation Processing Systems (NeurIPS), pp. 6232-6240 6240, 2017.
Henkes, A., Eshraghian, J. K., and Wessels, H. Spiking neural networks for nonlinear regression. Royal Society Open Science, 11(5):231600 2024.
Lv, C., Wang. Y., Han, D., Zheng, X., Huang, X., and Li, D. Efficient and effective time-series forecasting with spiking neural networks. 4300:2240..01333, 2024.
10

---

Time to Spike? Understanding the Representational Power of SNNS
Maass, W. On the computational complexity of networks of spiking neurons. In Advances in Neural Information Processing Systems (NeurIPS), volume 7, 1994.
Neuman, A. M., Dold, D., and Petersen, C. Stable learn- ing using spiking neural networks equipped with affine encoders and decoders, arXiv:2404.0455 04549, 2024.
Maass, W.
On the computational power of noisy spiking neurons. In Advances in Neural Information Processing Systems, volume 8, 1995.
Orchard, G., Frady, E. P., Rubin, D. B. D., Sanborn, S., Shrestha, S. B., Sommer, F. TT., and Davies, M. Efficient neuromorphie signal processing with loihi 2. In 2021 IEEE Workshop Signal Processing Systems (SiPS), pP. 254-259, 2021.
Maass, W. Lower bounds for the computational power of networks of spiking neurons. Neural Computation, 8(1): 1-40, 1996a.
Pascanu, R., Montúfar, G., and Bengio, Y. On the number of inference regions of deep feed forward networks with piece-wise linear activations. In International Conference on Learning Representations, (ICLR) 2014, 2014.
Maass, W. Noisy spiking neurons with temporal coding have more computational power than sigmoidal neurons. In Advances in Neural Information Processing Systems, volume 9, 1996b.
Patel, N. and Montúfar, G. On the local complexity of linear regions in deep relu networks. arXiv:2412.1828, 2024.
Maass, W. Networks of spiking neurons: The third genera- tion of neural network models. Neural Networks, 10(9): 1659-1671, 1997a.
Petersen, P. and Voigtlaender, F. Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Networks, 108:296-330, 2018.
Maass, W. Fast sigmoidal networks via spiking neurons. Neural Computation, 9(2:2799-304, 02 1997b.
Petersen, and Zech, Mathematical theory of deep learning, arXiv:2407.18384. 2024.
Mchonic, A., Ielmini, D., Roy, K., Mutlu, O., Kvatinsky, S., Gotarredon,,,,, T., Barrrnco, B., Spiga, S., Savel'ew ev, S., Balanov, A. G., Chawla, N., Desoli, G., Malavena, G., Monzio Compagnoni, C., Wang, Z.,, Yang. J. J., Sarwat, S. G., Sebastian, A., Mikolajick, T., Slesazeck, S., Noheda, B., Dieny, B., Hou, T.-H. A., Varri, F., Pernice, W., Zhang, X., Pazos, S., Lanza, M., Wiefels, S., Dittmann, R., Ng, W. H., Buckwell, M., Cox, H. R. J., Mannion, D. J., Kenyon, A. J., Lu, Y., Yang, Y., Querlioz, D., Hutin, L., Vianello, E.,, Chowdhury, S. S., Mannocci, P., Cai, Y., Sun, Z., Pedretti, G., Strachan, J. P., Strukov, D., Le Gallo, M., Ambrogio, S., Valov, I., and Waser, R. Roadmap to neuromorphic computing with emerging technologies. APL Materials, 12(10):00201, 2024.
Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl- Dickstein, J. On the expressive power of deep neural networks. In Proceedings of the International Conference on Machine Learning (ICML), volume 70, pp. 2847-28544 2017.
A.,
Rathi, N. and Roy, K. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 34(6):3174-3182 2023.
Rathi, N., Chakraborty, I., Kosta, A., Sengupta, A., Ankit, A., Panda, P., and Roy, K. Exploring neuromorphic com- puting based on spiking neural networks: Algorithms to hardware. ACM Comput. Surv. 55(12), 2023.
Montúfar, G. F., Pascanu, R., Cho, K., and Bengio, Y. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems (NeuRIPS), volume 27, 2014.
Rueckauer, B., Lungu, I.A.., Hu, Y., Pfeiffer, M., and Liu, S.-C. Conversion of continuous-valued deep networks to efficient drent-driven networks for image classification. Frontiers in Neuroscience, 11, 2017
Mostafa, H. Supervised learning based on temporal coding in spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 29(7):322733233 2018 88
Sarker, H. Deep learning: A comprehensive overview on techniques, taxonomy, applications and research direc- tions. SN Computer Science, 2(6):420, Aug 2021.
Neftci, E. O., Mostafa, H., and Zenke, F. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-63, 2019.
Serra, T., Tjandraatmadja, C., and Ramalingam, S. Bound- ing and counting linear regions of deep neural networks. In Proceedings of the International Conference on Ma- chine Learning (ICML), volume 80, pp. 4558 4566, 2018.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsu- pervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Shen, H., Zheng, Q., Wang, H., and Pan, G. Rethinking the membrane dynamics and optimization objectives of

---

Time to Spike? Understanding the Representational Power of SNNs
spiking neural networks. In The Annual Conference on Neural Information Processing Systems (NeuRIPS), 2024.
Zhang, S.-Q., Chen, J.-Y., Wu, J.-H., Zhang, G., Xiong, H., Gu, B., and Zhou, Z.-H. On the intrinsic structures of spiking neural networks. Journal of Machine Learning Research, 25(194):1-74, 2024.
Shen, Z., Yang. H., and Zhang. S. Deep network approx- imation characterized by number of neurons. Commu- nications in Computational Physics, 28()):168--181,, 2020.
Zhou, D.-X. Universality of deep convolutional neural works. Applied and Computational Harmonic Analysis, 48(2):787-794, 2020. 2020.
Singh, M., Fono, A., and Kutyniok, G. Expressivity of spik- ing neural networks through the spike response model. In UniReps: the First Workshop on Unifying Representa- tions in Neural Models, 2023.
Stanley, R. P. Enumerative Combinatorics: Volume I. Cam- bridge University Press, USA, 2nd edition, 2011.
Stanojevic, A., Wožniak, S., Bellec, G., Cherubini, G., Pan- tazi, A., and Gerstner, W.
deep spiking neural networks with 0.3 spikes per neuron. Nature Com- munications, 15(1):6793, 2024.
Telgarsky, M. Benefits of depth in neural networks. In Annual Conference on Learning Theory, volume 49. pp. 1517-1539, 2016.
Thompson, N. C., Greenewald, K., Lee, K., and Manso, F. Deep learning's 's diminishing returns: The cost of improvement is becoming unsustainable. IEEE Spectrum, 58(10):50-55, 2021.
Wu, Y., Deng, I.., Li, G., Zhu, J., Xie, Y., and Shi, L. Direct training for spiking neural networks: Faster, larger, bet- ter. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1315 2019.
Yamazaki, K., Vo-Ho, V.-K., Bulsara, D., and Le. N. Spiking neural networks and their applications: A review. Brain Sciences, 12(7), 2022.
Yarotsky, D. Error bounds for approximations with deep relu networks. Neural Networks, 14:103-1 2017.
2017.
Yuan, R., Duan, Q., Tiw, P. J., Li, G., Xiao, Z., Jing, Z., Yang. K., Liu, C., Ge. C., Huang, R., and Yang, Y. A calibratable sensory neuron based epitaxial VO2 for spike-based neuromorphic multisensory system. Nature Communications, 13, 2022.
Zaslavsky, T. Facing up to arrangements: Face-count for- mulas for partitions of space by hyperplanes. Number 154 Memoirs of the American Mathematical Society. American Mathematical Society, Providence, RI, 1975.
Zhang, S.-Q. and Zhou, Z.-H. Theoretically provable spik- ing neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2022.
12

---

Time to Spike? Understanding the Representational Power of SNNs
A. Roadmap to the Appendix
The appendix is organized as follows
In Section B, we provide the paper's main proofs. Section B.1 and Section B.2 contain proof's for the results in Sections and 4, respectively.
Section presents details of our experimental methodology.
Section provides an expanded review of related literature and an in-depth discussion of existing research.
Section E motivates the discrete-time LIF-SNN model, reviews alternative SNN models, and discusses practical encoding schemes. Furthermore, we derive elementary properties and suggest potential extensions of our theory. While not essential, this section is included for self-containment and aid readers unfamiliar with SNNs.
Table 2 summarizes the notation used throughout the paper and appendix aid reader comprehension.
Type
Notion
Notation
Domain
Number of spike layers
Network sizes
Layer width
12g
Latency (number of time steps)
T
N
Membrane potential vector
u u'(()
Neuron states
Spike activation vector
8 s1(1)
{0, F
Weight matrix
W
TR
Bias vector
6'
Re
(Hraee::Parrmmetrrs
Initial membrane potential vector
u (0)
The
Leaky parameter
0. 1
Threshold value
g
Input/output//arget vectors
Input vector
x
Output vector
2
Label or target vector
y
7lour
Table Table of SNN notions
B. Proofs
B.1. Proofs of Section
1.1.1. EXPRESSING THE IDENTITY
We start with the proof of Proposition 3.1 in Section 3, which we repeat for convenience
Proposition B.1. Let be a discrete-time LIF-SNN with latency N and N luyers with constant width n E N in each layer. Then, there exists configuration of parameters such that (s)( sd for any s (0,
Proof. It suffices to consider the case L = 1 since the presented argument can be repeated for any subsequent layer. Fix the parameter of
(WW b), (1),
as W = (1 In with 0 < < $. b 0, u0) = 0, B 1, and 9 1. Then, equation implies for each neuron i E [r]] the dynamic
st()) H - (1 (1+ -1) (t(t) u, 1) + (1+ st)
where 8 (t)(t)) (0, 13
'The leaky parameters and threshold values be either hyperparameters trainable parameters depending on the implementation (Fang et al., 2021).
13

---

Time to Spike? Understanding the Representational Power of SNNS
We show by induction over / [T] that u.( E & (ke lj. For 1 O. it trivially holds true uy u²(() Assuming that (t 1) E 1,, consider the induction step from 1 1 to for some fixed >> O. If the input at time t is zero, i.e.,
(t) = 0, then we have
(t(t) H + -1) H
U4(t 100-11 1) -)
0 by induction
and thus
(t) = (t - + s;()) =u(t - 1) E E C
On the other hand, if (t()) = 1, we have
(t) = H - 1) (1 11 = H. (4(t-11) 1) =
20
and
(1) (t 1) 1) (+()==)((()))((++)()))))) st( -t +E Ex.
E6
Note that the induction also showed that st) (t) for all [T] which implies that (s) 1E S for any
8 E {0, 13" p as required.
1.2. UNIVERSAL APPROXIMATION
We now present the proof of the main result of Section 3, which addresses the approximation of continuous functions by discrete-time LIF-SNNs. Throughout this section, we consider the constant encoder as fixed. We remind the reader of the result we aim to prove, which corresponds to Theorem 3.2. For simplicity, we will henceforth replace fh with n in the notation.
Theorem B.2. Let E EN, e 0, RD be a compact domain and f E C(2) be an arbitrary continuous function on Ω. Then there exists discrete-ting time LIF-SNN with T time step, spike layers, constant encoding, and membrane potential outputs, which satisfies
[R($) - fll := : sup (x) f(x) <8
In words, discrete-tin time LIF-SNNs are universal approximators for continuous finciintossinnctio on Additionally, if f is F --iiscchitz then T can be chosen with width parameter n (n1, n2) given by
diam (9)
71 =
max
+ n
c
ddiamoo (S2
n22 = max
(5)
where diamno (l) = yEn | yl1
The proof involves two main steps: (1) demonstrating that such SNNs can represent any constant function defined on hyperrectangles (step functions), and (2) establishing that step functions can approximate any continuous function on a compact set with arbitrary precision. For clarity, we formally define relevant geometric notions below.
Definition B.3 (Polyhedra, hyperrectangles, indicator functions, step functions). Let n E N.
A subset P C is called polyhedron if there exist A E RRXn b IP, p N such that
P I Ar < b).
where "S" understood entry-wise. If is bounded, we also refer to it as a polytope.
14

---

Time to Spike? Understanding the Representational Power of SNNS
subset R C IR" called hyperrectangle hyperbox in IR" it is the Cartesian product of n one-dimensional (non-degenerate) intervals I; C IR, ....... A hypercube is particular of hyperrectangle where all the intervals I; are of equal length. In the sequel, all hypercubes and hyperrectangles in R are assumed to be non degenereee (with positive Lebesgue measure).
For some subset Q E IR", the indicator function on R is defined by 1q(x) if Q 1 (Q(x) = 0 E (QQ
Step functions (on hyperrectangles) are defined as linear combinations of indicator functions on hoperrectangular pieces. More precisely, function IR" R. is called a step function if there exist N, f fm E R and
m polyhedra Ri...., Rm C R such that -==== 1 f.l
In summary, polyhedra are subsets defined by finitely many linear boundaries, with hyperrectangles special case where all boundaries are parallel to the coordinate axes. Although constant functions on polyhedra are not strictly necessary to prove Theorem B.2, as can be seen in the proof of Lemma B.7, they naturally arise in this context as showcased in the intermediate Lemmas B.4 and B.6. Additionally, this concept important in the study of linear regions Section 4.
The following two results demonstrate that step functions can be expressed by a discrete-time LIF-SNN with T 1 and 2 using suitable weights and biases. Furthermore, they establish that the depth L 2 optimal, there exist step functions that cannot be approximated to arbitrary precision by discrete-ting LIF-SNN with L =
Lemma B.4. Let m, E N, fi...., fm E R, PI,..., Pm S I be arbitrary polyhedra and Cl...., Cm be disjoint (up to Lebesgue measure zero hypercubes in R" with the same volume, such that U C₁ = C, where C is a hypercube in R". Then the following holds:
(i) The function f == >m Jiln p, can be expressed by a single time step discrete-time LIF-SNN with two hidden layers. Moreover, Pf = fI E R² Ayæ s by}. with E RP5x*, xn, then the width of the SNN that realizes f is described by
Σ 1=1
Ph:
m
(ii) The (step) function f = Di i= filc, can be expressed by a two-layer discrete-time LIF-SNN with first and second layer widths given by
n1 ( 1/n + +1) n, n22 m
Proof. We begin by proving part (i). Observe that a general polyhedron P for E R" Ax S b). with A =xx and b E TRP, contains point IR" if and only if (a,, x)) < b, for every i [p) (here a, is the i-th of A), equivalently
H (a, x)) 1, Vi E [p].
Since the Heaviside function takes only values 0 or 1, this is in turn equivalent
Σ H(b (a., x)) = p.
or, cquivalently,
p
H
Hb: - -a;; -p 1.
=1J H(b ((b-AA))- Am)
This means that the indicator function 1p can be written as a discrete-time LIF-SNN with T = 1, L, 2, and neurons in the first hidden layer. Note that discrete-time LIF-SNN corresponding to the previous network can constructed with parameters (here we consider 3 1)
W = A, bl = = b 1
-1= u (0) = 0
2 = 1 T,
b2 -
0=
n (0) 0.
15

---

Time to Spike? Understanding the Representational Power of SNNS
For the desired function f. one simply stack (the hidden layers) of each of the LIF-SNN expressing 1 E [p] together and set the weight matrix of the output layer be [f [Ji....;fml Sm]. Thus, the size of the hidden layers is
m
11
Pis
n2 = m.
1=1
We now prove part (ii). Consider f Ex=1 Sil.c. Given that C - C, is a hypercube, we can write = xi-
where 1j is an interval, for all j E [n]. Defining N := m E N, we divide each of intervals in {4,}²=1 as follows
1 = J 1 k +h
where of and +1 are the upper and lower limits of the interval Ij. respectively, and rf rx 1/N, for k E [N]. Thus, given that the hypercubes CI...., C. have the same volume and their union is C, we have that for each O with i= l,..., there exists a sequence (km)) C [N] such that
C = x x Riy kiyt
With this, for any input x = (f1,,...,Tmh) 1 1c, (1) is a boolean function that can be written as
1a (x) II 1 Iri -hy (zz).
j=1
On the other hand, for any sequence of boolean variables X1...., Xn we have
II x = ==1
H
Σ X -
(6)
From the above, follows that we know how express Xj 1 1r tky +1 (2;) using the first layer of a discrete-
LIF-SNN network, then the second layer can be used to implement equation straightforwardly, with m neurons. Indeed, the expression inside the Heaviside in equation 6 an affine transformation of x = (x....., .x). From this, it is clear that
X H(z, - T H(z rby+t)-
Hence, X can be written affine transformation of the variables in {H(zz - Tkey (e( Consequently, built a discrete-time LIF-SNN with the first hidden layer having (N + 1)n neurons, each of which will implement one of the elements of the set (H, rk,,, [be To finish the proof, we can set the weights in the output layer to be [fi...., fm], as in the proof of part (i).
Minimality with respect to the number of neurons. Our proof of Theorem B.2, will use elements on the span (1l,, } for some m and hypercubes { that partition a larger cube C. In part (ii) of Lemma B.4, we demonstrated that any function in this space can be realized by a discrete-time LIF-SNN. We now prove that the construction provided in Lemma B.4 is minimal, meaning uses the smallest possible number of neurons necessary to construct a discrete-time LIF-SNN with two hidden layers, capable of expressing any function in span( 1
12 1). For that, note that considering the Lz-inner product (f.g) fg. [ (x)g(xx)dz the functions {1c,, Je are orthogonal (because C, n C2 0, for i + j. modulo a set of measure zero). So, as subspace of (RR"), span ((1c. J. )m,) has dimension m.
Lemma B.5. Let Cı,..., Cm be hypercubes in R satisfying the conditions of Lemma B.4. If F is family of discrete-time LIF-SNN networks with T' = 1, L- = 2, such that for any f E spann( {1c, ) there exist network in F that realizes f. then there exist an element F. with spatial architecture parameters (L = 2, = n2)), with n. = (m l/ + 1)m and 722 = m.
Proof. To prove that n2 = m is necessary, it suffices to note that in the output layer, the boolean functions obtained in the second hidden layer are multiplied by constants and added. This operation cannot increase the linear dimension of the space
16

---

Time to Spike? Understanding the Representational Power of SNNS
of functions realized up to the second hidden layer. Given that span ((1 c; J has dimension m, and that each neuron in the second hidden layer implements one boolean function, it is clear that express all elements in span( ) least in neurons in the second hidden layer are necessary, i.e., f22 = m. On the other hand, given that up to the second layer we can express a boolean function with each neuron, each neuron should express a function of the form lu,c, We now consider a function f E span ((1 c, } such that, f has different values for each cube (Cr [C- In particular, if e; is the center of the cube C, then f(;;) * f(z) for i i j. Given that - i; = C, the points { form a regular grid in R" The minimum number of hyperplanes needed to separate a regular grid with m points in R" is (m11/ 1)m. Indeed, fixing d [n|] and considering æi.d. the d-th coordinate of o,, we have that ( mm m n//n, and the number of hyperplanes needed to separate m collinear points is m 1. To separate the points 3; need to be able to separate each of the sets { for d E [n]. Therefore, at least (m - 1)x hyperplanes are required to perform the separation of the points in { 1= Note that in the previous argument for each coordinate d there where two unbounded regions (corresponding to min; a and max, T.,d). Thus, to obtain closed regions, we need two extra hyperplanes per coordinate. (m//n
Then the total number of hyperplanes needed is (m 1/m + 1)n. Since each neuron in the first hidden layer generates one of these hyperplanes, we deduce that ni has to be at least 1/n + 1)n.
Remark (Point separability). In the proof, we repeatedly use the fact that a grid point set of size m can be separated by (m!/ + 1) hyperplanes. A similar observation is made in -Hlr- Jones, 2018][aage 4], where the more general problem of point separability (not necessarily on a grid) is discussed (mainly in R²).
Minimality with respect to the depth. We now prove that expressing an indicator function over an arbitrary polyhedron (excluding half-spaces and stripes between two half-spaces) fundamentally requires least two layers. A comparable result for threshold Artificial Neural Networks was independently established in (Khalife al., 2024), although with a different proof.
Lemma B.6. Lern 2
Rmx2 and (= : 2 0} be a non-empty polyhedron with two non-parallel linear boundaries, i.e., A has full rank. Then the indicator function 4p cannot be expressed by any (neither with spike output nor membrane potential ousput) single time step discrete-time LIF-SNN with only one hidden (spike) layer.
Proof. Toward a contradiction, we assume there exists such polyhedron P C R" that can be expressed by discrete-time LIF-SNN with T = 1 and -= 1. We first consider the case of spike outputs. Since the output vectors have only one dimension, the spike layer is allowed to have only one neuron. It follows that the network has the form x + H(a x - 6) for some a E R" and b E R. However, this only realizes the indicator function of a half-space, which is not P (by our assumption on ā1 and ā²).
Now we consider the more general case where we use membrane potential outputs, i.e., the decoder D : RT R is an affine map of the form Rm y [y1,.... ymm + we + > ways E R. where m is the number of neurons in the last spike layer while w = [Wi...., Wm] and wy denote the weights and bias of the decoder. Then the network realizes the following mapping
x + we +
wi H (a., x)) + b.) =: f(x),
i=1
where A == [aT ...., IT E RR and b [b1,
bm]] T E Rm the weight matrix and bias vector of the spike layer.
To simplify notations, we set
H4 = (I R" (a,, x) + +b; = 0}
to be the hyperplane defined by the i-th neuron the spike layer, i i= I,.... m, Without loss of generality, we assume that H. are distinct hyperplanes and w; # 0 for all i. We will show that the union of the hyperplanes H4 must be equal the union of the hyperplanes A that define P, i...
A = x EER::
R" 3i E (1,2) such that (ā., x) + b, = =]]
To see that U H4 C A, let x E H4 for some arbitrary E [m]. Observe that if x & A, then there exists some neighborhood of x that lies completely in R" \AA. Then H4 divides this neighborhood into two regions that correspond to the same value when evaluated by 1r but different values when evaluated by f (as wi * 0). This means that f + 1p.
17

---

Time to Spike? Understanding the Representational Power of SNNS
On the other hand, to see A A 1 let a E A. If does belong any hyperplane Hi, then similarly there exists some neighborhood of it that possesses only one constant value of f but two different values of r, so that f + 1p.
Next, we proceed with the case m 2, i.e., there are only two neurons in the hidden layer. Observe that R is divided into 4 different regions by H.1 and H2, and inserting some arbitrary element of each of these regions subsequently gives (via (7))
Wp + w1 + 1/.
Wo + 1 = 4/2
Wp + u22 ys+
Wo y4.
whereas among y1, 1/2 V/, E (0, 1), there is exactly one 1 and three O's (so that indeed the indicator function of P is realized in the end). Here, the exact values the y 's depend on (the signs of) the equations corresponding to H4 and H2; however, for any configuration of (y:) >_ this always leads the contradiction that y/ Y/3 = yı + 3/4 Finally, the case that we have more than two neurons in the hidden layer can be reduced to the case of two neurons by restricting the functions and 1 p to the interior of the regions created by Hi and Hz (since showed that - H4 A). This completes the proof.
Approximation with step functions. We now prove that step functions can be used to approximate continuous functions. More precisely, we prove that given any continuous functions f defined on compact set Ω one can select a finite set of hypercubes [C for some m, such that ʃ is close to span((1c. }i.)).
Lemma B.7. The of step functions hypercubes can approximate any continuous function on compact subset S C R N, arbitrarily well. In particular, for all E C())) and for every 0 there exists E N such that the following holds: there exists hypercubes Cj..... C. of same volume with their union forming again a hypercube and fi...., fm E R
such that [[f flloe S. with f= E filc.
Proof. Let f E C(22) be a continuous function and & > 0 be arbitrary. Since is compact, there exists constant K >> O such that Q C [-K, K]" and, moreover, f uniformly continuous on Q. Hence, there exists 8 > such that for any a, y E Ω with [ yl = max;e]j] J. - yıl
holds I/(x)) - f(y)] < e.
Now we partition the domain 52 [-K,, KJ" into hypereubes C, 1, I,..., m by dividing the interval [-K, K] into & sub-intervals of size at most 5. We define step function f f= Σm fil., on Ω by specifying its value fü on each hypercube Cs to be the mean value of f on the intersection of C; N. For any arbitrary cube C, it can be observed that for any x E C2 n it holds
[
/
[f(z) f(x) = f(x) IC n Ja,n fly)dy S
C. c Ja,e If(z) f(y) dy <E
(8)
where for any Lebesgue measurable set A C R", |A| represents its Lebesgue measure. Since both and C2 are arbitrary, this means that
| - Al.. = -sup [f()) - f(x)) S s.
B. 1.3. APPROXIMATION RATES
In this section provide approximation rates for discrete-time LIF-SNN in term of its size. More specifically, we combine Lemmas B.4 and B.7 for the special case of Lipschitz functions (chosen for analytical tractability) to give an estimate on the number of neurons needed to approximate a Lipschitz function using a discrete-time LIF-SNN. We recall that a function f: + R is r-Lipschitz (with respect to the |loc norm) if for all æ, E Ω we have
(f(x) f(y) & F| y11
18

---

Time to Spike? Understanding the Representational Power of SNNs
Ω
2K
Figure 5. Illustration of the grid and cube partition used in the proof of Lemma B.7.
Corollary B.8. Let be I'-Lipschitz continuous functions on the hypercube [- K, K\". Then for all e > 0 there exists discrete-time LIF-SM JF--NN with T = 1, L = 2 and
2K
22
01
max
l
.1 + n,
112 = max
--11
such that [[R(Φ) fl E.
Proof. Similar the proof of B.7, will construct function of the form f= 1= f11., where m = [[2 ó to be determined. In particular, å corresponds the size of the hypercubes O. with , K, Kln By applying equation 8 and Lipschitz continuity we have, for E C. n [- K. Kj".
[= 1m for some
|f flbx |nn I c,na
[f(z) f(y)ldy S T8,
given that || y/1x0 S for Kn| Thus, choosing 6 min 2K3 get
| - 01 SE,
and, by Lemma B.4 part (fi), that discrete-time LIF-SNN with 2, n1 (m//" 1/m + 1)n, n2 = m can be constructed 2K 1)n. With our choice of 8, this becomes
express J. Replacing the value of m, have
71
max
2K
r
.1 1
1
+1
n
8
Remark, In Theorem B.2, we use diam. (I) instead of K, but it is easy to see that can be chosen as diam (S2) given that a compact set.
We now finish the proof of Theorem B.2 by simply applying Lemma B.4, B.7 and Corollary B.8.
Proof of Theorem B.2. By Lemma B.4, we can express any linear combination of indicator functions of hypercubes of the same volume forming a larger hypercube by discrete-time LIF-SNNs with 7' = 1 and L = 2. By Lemma B.7, these step functions can uniformly approximate any continuous function on compact subsets R to arbitrary precision. Therefore, the class of SNNs described above forms a universal approximator for continuous functions. Finally, the last statement in Theorem B.2, for Lipschitz, f. follows from Corollary B.8.
19

---

Time to Spike? Understanding the Representational Power of SNNS
Lower bound on neuronal requirements for SNN function approximation. The construction in the proof of Lemma B.7 reveals that two-layer discrete-time LIF-SNNs approximate continuous functions by discretizing them a grid of hypercubes. We aim to prove that for certain classes of functions, this grid-based approximation strategy is optimal up to constant factors, leading to lower bound on the required number of neurons. The next proposition, which is a restatement of Proposition 3.3 in the main paper, establishes the order-optimal approximation bound for one-dimensional functions defined on the unit interval [0, 1].
Proposition B.9. For any I' > O, we define f:
f(x) = Iz. For any E E (0, 1J. there exists a discrete- discrete-LIF-SNN
with two layers and the width of the first hidden layer n, [T/E] + 1, such that [[ flloo I E. On the other hand, any discrete-ting timeeteeemmeetee LIF-SNN 4 with first layer hidden width n² s T/ for a 1, will satisfy [R((') fllx 2 5
Proof. The first statement is a direct consequence of Corollary B.8. In particular, by construction in Corollary B.8, respectively Lemma B.4, R(D) is constant regular grid with nı points defined by ( )== (corresponding ni intervals on [0, IJ..
Now consider a discrete-time LIF-SNN 4' with n/j neurons in the first layer. In that case, R('") is constant on (at most)
B1 n + intervals. Then, (via the pigeonhole principle) there exists at least one interval that contains q = +1
points of
the set (,,- 1e This means that this interval contains segment of length g/ 1). Given that the function R((') is constant on this interval and that f is linear, the error satisfies
1 || [R(*)= نلال= 2 Tg/(m 1) 2
r
11
1 r
0
11
+
A1
2
11
+
In the last inequality, we used our assumption on n'1.
Remark (Extensions). The proof remains valid when n1 takes the more general form h(?), where h is any function 1 with a < 1 used above. While we focused on the unit interval for clarity, the results naturally extend to arbitrary compact sets. More interestingly, we believe that extensions to the n-dimensional case are possible, but this general case is left for future work However, a weaker version of Proposition B.9 holds for arbitrary dimension N, as can be shown by considering Lipschitz continuous functions that depend single coordinate.
satisfying limg-+so z
hiz) = 0, rather than the specific form go
B.2. Proofs of Section 4
In this section, we present detailed proof of Theorem 4.3, which states a tight upper bound on the number of regions created by discrete-time LIF-SNNs. The process is organized as follows.
We start by introducing in Subsection 1 the basic notions of hyperplane arrangements and general position as well as analyzing a special case of hyperplane arrangements, where the arrangement is a collection of families of parallel hyperplanes. particular, we introduce an upper bound on the number of regions created in this case and prove that the bound is attained in order of magnitude if the hyperplane arrangements satisfy a certain refined general position condition.
Next, in Subsection B.2.2, consider the mapping from the input to the first hidden layer (or equivalently shallow SNN) and show that each neuron in the first hidden layer corresponds to number of parallel hyperplanes in the input space which scales exponentially in the number of time steps T. This allows us to apply the previous result and characterize an upper bound for the maximal number of regions created by a shallow discrete-time LIF-SNN. To see the tightness of this bound, one has to show the existence of a shallow discrete-time LIF-SNN that realizes the mentioned general position condition, which is discussed in Subsection B.2.3.
Finally, in Subsection B.2.4, we analyze deeper spatial architectures and derive the main result for the number of regions created by discrete-time LIF-SNNs which is stated in Theorem 4.3.
B.2.1. NUMBER OF REGIONS CREATED BY FAMILIES OF PARALLEL HYPPERPLANES
First, we give a formal definition for hyperplane arrangements and general position, which will be used below to characterize the input space partitioning of discrete-tima LIF-SNNs. In Def. 1. the notions of hyperplane arrangements and their general position are taken from (Stanley, 2011), while the notion of general position for families of parallel hyperplanes focuses on the special case of hyperplane arrangements that are created by discrete-time LIF-SNNs on the input space.
20

---

Time to Spike? Understanding the Representational Power of SNNs
Definition B.10 (Hyperplane arrangements, general position). Let E.
hyperplane arrangement A C B& is defined is (respectively the hyperplanes in A are) in general position if the intersection of any p distinct elements from A is either (1) the empty set if > d or (2) (d -
finite collection of hyperplanes in R Furthermore, say that A
d p)-dimensional affine subspace if S d.
Lel Ui C Rdd be a hyperplane arrangement thut consists of different families i [n], of finitely many hyperplanes, where the hyperplanes within each family A, are parallel. We say that the families A, are in general position if the set {aI...., ] with any choices Og E A,, i E [n], a hyperplane arrangement in general position. We call such a subset (a1...., an a representative subset of Ai.
Remark. Informally, a hyperplane arrangement in Rd is in general position if and only if no pair of hyperplanes is parallel and no set of d + 1 hyperplanes coincides. The notion of general position for families of parallel hyperplanes requires the same conditions for every representative subset. obvious that whenever the non-parallelism is satisfied by any representative subset, it is satisfied by all such subsets. However, it is in general more difficult to control whether the second condition is satisfied.
While the fundamental theorem of Zaslavsky (Zaslavsky, 1975; Stanley, 2011) establishes an abstract way to compute the number of regions in the general case via the so-called characteristic polynomials, we only focus on the special case of hyperplanes created by discrete-time LIF-SNNs, which consist of collection of families of (the same number of) parallel hyperplanes. The following lemma demonstrates a tight upper bound on the number of regions created in hyperplanes in those special configurations.
Lemma B.11. Let n. k. d N. Consider hyperplane arrangement A - in Fd consisting of n. different families n. Assume that each family Ai consists of at most k distinct parallel hyperplanes. Then the hyperplane
A. i = 1, ,,...,,, arrangement A partitions RR into a number of regions, which is upper bounded by
{ EE= 1) if d.
(()
ifn 2 d,
Furthermore, the upper bound is attained if the families A,, i == I,.... . n, are in general position.
The proof is based on the "deletion-restriction principle (Stanley, 2011) 1) (or simply recursion), but simplified to our special case.
Proof. We denote the maximal number of regions created by an arrangement of n families of & parallel hyperplanes Rd by p n.dd In the following, we will compute " recursively over n and d.
First, we consider removing a family from the hyperplane arrangement A. say An. To that end, let h E An be an arbitrary hyperplane removed from A. Note that in the cases h is parallel to or coincides with a hyperplane in Ai, the number of regions created by An is strietly smaller than rft n. thus exclude those cases in the computation of r a Then, each pair of hyperplanes formed by h and hyperplane from A. i S n 1 may have one intersection, which is a (d 2) di-dinnssi affine subspace.
Observe that these (d - 2) dimmensiona affine subspaces are in turn hyperplanes if restricted to h, which is of dimension Furthermore, among all such (d - -)dmmensio hyperplanes of h, the ones coming from the same families A;, i [n- 1]. are parallel. Hence, h is divided into most r 11d-1 regions.
On the other hand, observe that when adding h back to A \ An, the number of regions added is exactly equal to the number of regions that A \ An creates on h. Also, recall that h is only an arbitrary representative from i elements of A,, Therefore, we arrive at the following recursive formula
rk In- + krk 1-d-1-
n.d
With the convention (7) = 0 for any i >> n, we can prove by induction over + d that
b 11
p n
im0
21

---

Time to Spike? Understanding the Representational Power of SNNS
Indeed, the claim holds trivially for n 0 as an empty hyperplane arrangement would leave the space I& not partitioned. If the claim holds for any pair (n',, d'') with n' d' S n + d for some fixed pair (n, then by the above recursive formula and induction hypothesis, we get
7 n n- + kr
1,d-1
d
n2
1
+
k
d-1
1
i
i=0
1
+
d
k.
11 1
+
d
h
n 1
=
1
+
d
k
11
i
i-1
iml
0=
In case n S d, follows that r = u r n.d if and only if AA attains 7n-1,d and the (d 2)-dimensional intersections of clements of A,, and
k2 (?) = (k: + 1)a. Finally, notice that the hyperplane arrangement attains the
upper bound of elements of other families all exist and also attain In-1. This condition recursively recovers the property of general position and completes the proof.
B.2.2. CHARACTERIZATION OF THE HYPERPLANE ARRANGEMENTS CREATED BY THE FIRST HIDDEN LAYER
In this subsection, we only consider the mapping from the input to the first hidden layer, or equivalently, shallow architecture. Hence, the layer indices can be dropped for convenience. In particular, we will write W, b, B, 0, s(t)),(t)) for W ,,,, 3), 01, u² (1), s()) respectively. With the introduced reduced notation, we obtain the following dynamics
{ s(() = -H(Bu(t - 1) + W x + b 21) ru(t) Buu(t 1) + W b ds(t), t E [T].
For convenience, we repeat the statement of Lemma 4.2 in the main paper in the introduced notation. Note that Lemma B. is stated for shallow discrete-time LIF-SNNs only because of notational convenience. It is straightforward to extend this result to deeper architectures stated in Lemma 4.2.
Lemma B.12. Consider shallow discrete-ting time LIF-SNN with T time steps and an arbitrary neuron l: in the hidden layer. Then the input space Th is partitioned by k via number of parallel hyperplanes into most T²=T+2 E O(T2) regions, on each of which the time series (&x (4) cetry takes a different value in 10, This upper bound on the maximum number of regions is fight in the sense that there exist values of the spatial parameters uy (0), bound is atlained.
B, 3,0- R [0, 1] x R4 such that the
Proof.
1. Introducing notations.
The dynamics with the introduced simplified notations imply
g'u(t 1 L - 2 i) + (W -33'sst 1- -i).
Taking the sum for i 0. ..... 2, we obtain
1-2
1-2
8 (Wx + b) - Σ u(t - 1) = -0 + Σ
(t 1-1) 1).
1=0
==0
Therefore, the spike activation st) is determined by
s(t) H (3u(t - 1) Wx + b -01)
1-1
1-1
H ('u(()) Σ
3d 3(Wx x + b) - -0(1
Σ
s( - -i)
22

---

Time to Spike? Understanding the Representational Power of SNNs
For the given neuron & E [ma] in the first hidden layer, this becomes
1-1
(t) H
3 ( (wk,, x) + bb) + +/ 32ux(0) - 0 (1+ + Σ
8 ss(t - 11
0=
1=1
Blua (0) - 0 Ei=l Sk - -i)
H (wz; x) + by
(9)
1-1 Σ gi
where w; E "h denotes the i-th row vector of W. This means that at time step L E [T], the value of sn(()) E {0, gives information about the half-space the input vector æ E R" lies in with respect to the hyperplane
h41 (sx (t 1)) {xr R (wk: x)) + bx -9--1 ................................ su 1)) = 0} R
where the function ---1 defined by
(0.1) + R, It-1 (a1,,a),..., a-11) 3r (0) + (1 B'aris)
9--1
(10)
Σ0 3
Furthermore, for each binary code (d:) {0, 1)" define the corresponding region
Re-1(01,...- at-1) (a R sx a; Vi E [t 1]} = - far E RT sx(1) =a,)-
Note that such a region can be empty (see helow) and we denote by ((1) the number of non-empty such regions (which is also the total number of regions created at time step t). Our starting point is the step = 1, i.e., t- 0, where the whole space R which corresponds the empty code (a, ) is divided by exactly 2 hyperplane, namely (according to (10)) the one given by the shift go - Bug (0) + v, into different regions (depending on whether Q1 0 or d1 1).
2. Not every binary code corresponds to a non-empty region.
In principle, after time step 1, or equivalently, before time step i, there can be 2 possible binary codes (a:))e(t-1]
13 and accordingly the same number of hyperplanes /4- (ai...., a1))). Each of these hyperplanes may separate (at most) one region into two sub-regions, thus increasing the total number of regions by onc. This means that the number of regions might be doubled in each time step, i.e., N(t) N(t - 1) might reach 2t-1, which possibly leads to + E 2 = 2 regions in total at time step T.
However, in reality, hyperplane can divide region into two sub-regions only if it intersects (in our case, as the hyperplanes are parallel, if lies inside) that region, because otherwise the region remains one whole region. More specifically in our case, a region R (rr-((a,, (a1,..,, az-1) corresponding to the code (aI,..... at-1) defined before time / is separated into two sub-regions at time t and only if it contains the hyperplane ha a(-1) (created at time step f), i.e.,
1 .t-1) R (a..(a)....,t...).
(11)
According to our previous notion of (non-)empt) regions, this means that if (ai, at-1) falls outside of (h-1(a1,..... (t-1), i.e., the condition (1I) is violated, then the whole region R (01, ... at-1) must lie on one side of the hyperplane h (a1, ...,, (1- and therefore either Ra(a,,...,,te (1-1, R2 Ra(a),.... 0t-1, 1) is empty, while the other set is the same as Ra-1(a1,... R.-.(as.....4.))) a.-1).
The requirement (11) significantly reduces the number of separated regions, equivalently, reduces the increase N(t) N(t - 1) in the number of regions from time step t 1 to f.
3. Showing N(L) Ntt 1) < / and deriving the bound on N (T)).
We fix time step & [7] and consider the transition from to f. Moreover, let E (0....,t 1) be arbitrary and consider the set
Am = (
10,
Σ
a m and R a-11) #1
1=1
23

---

Time to Spike? Understanding the Representational Power of SNNs
00
01
10
000
001
01
10
110
111
(a) The separation after time step f. 2: there are most regions {s(1) = d2, s(2) = (2) for (a1; (2) E 1)". The regions are sorted in the lexicographic order of their corre- sponding code. A region corresponding to the code (a1,02) (a) divided into 2 sub-regions in the next time step, i.c., t == 3, and only it contains the hyperplane ha (2101, 02).
At 3. the hyperplane (a 02) induces one more region if lies in the "correct" region R2(as, 02). Since the hyperplane ha(0,, 1) lies on the left of hea 0), while the region (0. 1) lies on the right of (1, 0), no more than one of them lie in the correct region.
Figure 6. Illustration of the proof. From time step 1 to time step 4, most 1 additional region is created in the regions with E 0; = m for any fixed m E (O...., L- 1). Therefore, at most regions are additionally created in this time step.
of all binary codes of length t - 1 that have m ones in their representation and correspond to a non-empty region created before time f. Observe that if we arrange the codes (a1,...,,;... in Am in increasing lexicographic order, then the corresponding values Σ , are in decreasing order (since 3 decreases with increasing t). This means that while the regions R at-1) are arranged in increasing lexicographic order of (a1..... a--1) ? the position of their corresponding hyperplanes h41, 04-1) are arranged in the reversed order, i.e., in lexicographic order of (d.-1,..... a1): ): see Figure 6 for an illustration. Since the regions are all disjoint, it follows that there is at most one hyperplane belonging the 'correct' region, i.e., the region that corresponds to the same binary code. Since most f hyperplanes belonging the 'correct' regions
(0, 1} was arbitrary, deduce that there time step Hence, at the transition from time step t to we obtain
N( < N(1 - 1) +
Taking the sum over E [T], we get
T(T + 1)
N(T) 1 Σ
E O(T)).
4. Tightness of the bound.
Now it is left to show that the number of O(?²) parallel hyperplanes can indeed be achieved. For this, we simply let
B = 1 and i = 1. Then the shifts at the transition from time step - 1 to t, as given in (10), become
9t- (t-(Qq....,a..)) a4-1) - (0) 1 + Σ at-1
Therefore, for any fixed m E {O....,t t 1). any binary code (ai...., a-11) 1) E Am corresponds to the same hyperplane
Ri (Wk, + by 1 1 (4x(0) 0)
One observes the followings:
(a) For any m (1,..., 2). the set U(a......-))E. AEA.. R Re-i(a,, a: i.e., the union of all non-empty regions
with Σ Or- is exactly the region between the hyperplanes corresponding to the shifts m m- (0) and m nn (0) In the transition from time step t to f, if holds that
up (0)
1 1+ m up (0)
1 m uu (0)
(12)
t
L
then the hyperplane corresponding to the shift 1+m un (0)
must belong the set
f+1
I)EAA R R ...1)).......)). Thus, it must lie in one of such regions and separates it into two sub-regions.
Intuuiiivlly, the new sub-region at any time step lies on the left of the hyperplane h4-1(11,-.. a)11) if a, 0 and on the right if a; = 1, and this process is performed from = on. The process actually reflects the ordering binary codes in lexicographic order.
24

---

Time to Spike? Understanding the Representational Power of SNNs
(b) For the outer-most regions, namely R R ,., 0) and R 1 (1, ...., 1), observe that if (O,....
++ - 24 (0) t -- 24 (0) Uy. (0) 1 - (0) and
(13)
+
then those two regions are divided each into two sub-regions respectively by the hyperplanes corresponding the shift -=ax(0) and (+]=az(0))
1+1 (c) we can, in addition to the two above conditions, ensure that for any m E {o..... tt and m' E {o..... t'} as
1+1
well as for any 1. > it holds
U/ (0) m - (0)
(14)
t'
(i.e. there do not exist two time steps such that some shift of the current time step is equal to some shift of a step far away in the past), then at each time step f. the number of regions is increased by exactly / and thus the maximum number of regions is achieved.
Note that condition (12) holds for (0) 0 and condition (13) holds for any un (x( E (0, 1). For an arbitrary but fixed T, it is simple to see that for us (0) sufficiently small both these conditions are satisfied (since the involved terms are all continuous in un (0) and so is condition (14). Another simple way to guarantee (14) is to choose an irrational value for ux(0), This shows the desired existence statement.
B.2.3. EXISTENCE OF GENERAL POSITIONED FAMILIES OF HYPERPLANES CREATED BY DISCRETE-TIME LIF-SNNS
In the previous subsection, we have seen that each neuron in the first hidden layer of a discrete-time LIF-SNN corresponds to family of at most T'+T' parallel hyperplanes, a situation where Lemma B. 1 can be applied obtain an upper bound on the number of regions. Our only concern left is the tightness of the bound in higher dimensions, which in turn leads to the question of whether the families of hyperplanes are in general position according to Lemma B.11. Here, we prove that for appropriate spatial parameters (W, b), our shallow discrete-timg LIF-SNNs will satisfy the condition about general position.
Lemma B.13. Consider a shallow discrete-time LIF-SNN with T time steps with inputs from The and ny neurons in the hidden layer. We denote by As, k [m.], the families of parallel hyperplanes corresponding to the k-th neuron of the first hidden layer. Then one can construct weight matrices W E 2mn and bias vector b E I. such that the families Ak. k E [m1]. of parallel hyperplanes are in general position.
Proof. Observe that n S Tii, the notion of general position reduces to the condition that at least one of the representative subsets of the hyperplane arrangement 1 is in general position. In fact, if there exists a representative subset which does not have any parallel pair of hyperplanes, then the same holds for any other representative subset, while the condition of non-parallelism is sufficient for general position when nı S Nin- This means that in this case, one just needs to simply choose W to be full-ranked so that the corresponding families of hyperplanes are non-parallel. It only remains to consider the case n1 2 Tin-
Now, we will show the existence of the desired discrete-time LIF-SNN by iteratively constructing (w,, bi) for i I.... .... R4.
First, we choose (w: b:) for i S Rin such that the families A,, i E [Mim). are in general position (which is straightforward due to the above observation). Thus, it suffices to show how to choose (W bn+1) properly given (w, 6:)
for S n for some fixed n 2 nin such that the corresponding families A, E [n]. are in general position.
Obviously, the families A., i [m], have only finitely many intersection points, in particular, each intersection point the intersection of Tin hyperplanes from Rin different families. Let (w,+1, bb41) define a hyperplane
Hntt {ar E R (Wa+1, x) + b = 0}
that is not parallel to any family A,, i E [n], and all the intersection points lie on the same side of Hn It is not difficult to see that the shifts (as defined in the proof of Lemma B1) are bounded, say by some constant B > 0. Thus, by translating Ha-1 towards the half-space that does not contain the intersection points of A., i E [n)] we obtain a hyperplane
Hnt1 1 = {xr E R (ur,+1, x²) + buti = 0}
25

---

Time to Spike? Understanding the Representational Power of SNNS
so that all the intersection points of A < [n] lie on the same side of all of its shifted versions with the shifts bounded by B. This way gain collection of families As, E (n + 1] where Tinn hyperplanes coincide.
Finally, we obtain a hyperplane arrangement , which satisfies the following property by construction: (1) no pairs of hyperplanes from different families are parallel. (2) no sets of nin hyperplanes coincide. Thus, the hyperplane arrangement in general position (according to Remark B.2.1) and the proof is complete.
Remark, While Lemma B. 13 is proven in a constructive manner, the choice of the spatial parameters is quite flexible. We believe that for "almost all' choices of (W, b). i.e., up to a set of zero measure on the set of all spatial parameters, the resulting shallow discrete-time LIF-SNN creates families of parallel hyperplanes, which are in general position.
B.2.4. COMPLETING THE PROOF OF THEOREM 4.3
In this subsection, we combine the auxiliary lemmata discussed in previous subsections to finalize the proof of Theorem 4.3, repeated here in slightly revised version.
Theorem B.14. Consider a discrete-time LIF-SNN & with T time steps, input dimension Tme and n neurons in the first hidden layer. Then the maximum number of constant and activation regions taken over all choices of spatial and temporal parameters 0- ((W b)); (0), 36, of) re(r) is upper bounded by
Ex=o [
T2+T
(?))
if 2 Tiins
max |C| S max (R]
S
2
12+T++
otherwise,
2
The first inequality becomes equality if each spike layer has at least ny neurons. The second inequality becomes equality for appropriate choices of the network parameters O.
Proof. By Lemma 1.12, each neuron & [ni] in the first hidden layer corresponds to family A of most T'+T parallel hyperplanes. Lemma B. shows the desired upper bound on maxe (R]. To see the tightness of this bound, choose (u(0), B, i) according to Lemma B. B212 and (W1, b²) according to Lemma B.13 so that the families Ar are in general position while having the maximum number of hyperplanes.
For the first inequality to become an equality, we apply Proposition 3.1 to construct an identity mapping from the first to (the first n1 neurons) of the last spike layer, (s). */, (0) [T +> (t) sh (t)) Note that this construction only involves the subsequent layers f 2 2 and does not involve the first hidden layer. One can observe that any two distinct activation regions are mapped to two distinct output time series, thus |C| = [R]
C. Experimental details
C.1. Additional experiment results
In this section, we complement the experimental results shown in Section 5.1 with (1) similar experiments the SVHN dataset (Netzer et al., 2011) (in place of CIFAR10) and (2) comparison of test accuracies. In particular, for SVHN, we conduct experiments with the same models as well as technical settings as for CIFAR10 (see Section C.2 below). The corresponding results are presented in Fig. 7, which shows a similar trend as before for CIFAR10, with a few minor exceptions (possibly due to training instability in the cases of low spatial and temporal model sizes). While our experiments aim to verify our theoretical results on the expressivity of discrete-time LIF-SNNs, for completeness, we include the comparison of test accuracies achieved by the considered SNN models, see Fig. 8. Note that in both datasets the generalization performance is quite limited for both ANNs and SNNs (of any latency). The reason behind this is that did not apply any techniques to avoid overfitting, since justification of our theoretical expressivity results, the networks should have the same architectures as in the theoretical expressivity findings and aim merely to fit the training data (and not to generalize well to test data). In such a setting, ANNs are observed to generalize better than SNNs (of any presented latency) and SNNs, even with low latency such as T = 4 or T = S, show certain overfitting phenomena when the width of the subsequent becomes large enough.
26

---

Time to Spike? Understanding the Representational Power of SNNs
28
040
05
2014
100
120
140
200
120
340
200
with abeeruoee lidter lpers
with wessebogga baddes
(a) Train accuracy achieved by an ANN and SNNs with different (b) The improvement training accuracy when increasing numbers of time steps but identical spatial architectures
width of the subsequent hidden layers.
Figure Comparison train accuracies achieved by ANN and SNNs on SVHN dataset with different numbers of time steps. Both types of networks share the same spatial architectures: the first hidden layer is bottleneck with only 20 neurons and the subsequent layers are progressively widened each experiment. We consider layers both cases.
030
075
0.10
3.4
032
Twl
0.5
330
050
200
170
Wolthal sutsesuund loye
Wadti thaaaannggg bakben
(a) Test accuracy on CIFAR 10 dataset
(b) Test accuracy on SVHN dataset
Figure Test accuracies achieved by ANN and SNNs on CIFARIO SVHN dataset with different numbers of time steps. Both types of networks share the same spatial architectures: the first hidden layer is hottleneck with only 20 neurons and the subsequent layers are progressively widened in each experiment. We consider 4 layers in both cases.
C.2. Experimental setting
We employ the snntorch package (Eshraghian et al., 2023) for implementing discrete-time LIF-SNNs, where all Icaky parameters f and thresholds S are set to be learnable (Fang al., 2021) while the initial membrane potential vectors u' (0) are set to 0 as default. Furthermore, we deploy the cross-entropy loss applied to spike count outputs (see Example E.1 in Supplementary Material E.5). For the training, apply backpropagation through time (Lee al., 2016; Neftci et al., 2019; Eshraghian et al., 2023) with the arctan surrogate function (Fang al., 2021).
The numerical experiments described in Section 5.1 were conducted under the following set-up: Both ANNs and SNNs are trained in 200 epochs using the Adam optimizer (Kingma & Ba, 2015) with 31 = 0.9 and 3g = 0.999. The learning rate initialized at 10-3 and decays by factor of after 100 epochs. The batch size set to 512. We used the pytorch default Kaiming initialization (He al., 2015) all our experiments.
The numerical experiments described in Section 5.3 were performed in the same set-up only the learning rate, number of epochs, and batch size were adjusted to fixed value 5 x 10-4, 1000 and 256, respectively.
D. Related works
Here, we provide a more detailed discussion of the relevant literature going beyond the discussion in Section 1. We contrast the extensive work on the expressive power of ANNs with the currently still (in comparison) limited insights for ANNs.
27

---

Time to Spike? Understanding the Representational Power of SNNs
D.I. The expressive power of ANNS
Understanding the expressive power of ANNs is a central concern in the field of deep learning, with long and rich history of research. Investigations in this topic can be broadly divided into the following two key areas.
Approximation capabilities This field focuses the ability of ANNS to approximate certain function classes well as the complexity of the network to attain prescribed accuracy of the approximation. A starting point are the Universal Approximation Theorems (Cybenko, 1989; Hornik et al., 1989 Leshno et al., 1993), which established that shallow ANNs with sufficient neurons and appropriate activation functions can approximate any continuous function on a compact domain arbitrarily well. Building on these foundational results, subsequent research has focused on various extensions and refinements, ranging from investigations on the effect of diverse architectural aspects such as depth, width, number of neurons, convolutional layers, etc., to approximation certain function classes (Telgarsky, 2016; Yarotsky, 2017; Lu et al., 2017; Hamin, 2019; Shen et al., 2020; Zhou, 2020) to deriving (optimal) approximation rates (Yarotsky, 2017; Petersen & Voigtlaender, 2018; Gühring et al., 2020; Böleskei et al., 2019; Kutyniok et al., 2022).
Input partitioning The focus in this field lies on understanding the internal processing in ANNs more deeply, specifically their ability to represent complex patterns and hierarchical features. Notably, the pioneering works (Pascanu et al., 2014; Montúfar et al., 2014) have sparked research work on the piecewise linearity of ANNS with piecewise linear, including ReLU, activation functions (Raghu et al., 2017; Serra et al., 2018; Arora et al., 2018; Hanin & Rolnick, 2019a:b). Hence, a natural metric to theoretically quantify the expressivity of an ANN in this case is the maximum number of linear regions into which it can separate its input space. By reducing the problem to counting the number of regions created by hyperplane arrangement and directly applying Zaslavsky's s Theorem (Zaslavsky, 1975; Stanley, 2011), (Pascanu et al., 2014) proved a simple yet tight upper bound for this number for shallow ReLU networks. In the follow-up works (Montúfar et al., 2014; Raghu et al., 2017: Serra et al., 2018; Arora et al.. 2018), various theoretical upper and lower bounds on the maximum number of regions have been derived and improved. The study of input partitioning has subsequently broadened, leading to connections with diverse impactful aspects of neural network theory, such as practical expressivity (Hanin & Rolnick, 2019a;b), functionality and geometry of decision boundaries (Balestriero & Baraniuk, 2018; Balestriero et al., 2019; Grigshy & Lindsey, 2022) and local complexity, generalization and robustness (Humayun et al., 2024; Patel & Montúfar, 2024) of ANNs. Finally, for a more comprehensive survey on the relation of deep learning and polyhedra theory, please refer to (Huchette et al., 2023).
For comprehensive overview on the mathematical theory of ANN expressivity, we also want to highlight the dedicated sections in the recent books (and references therein) (Jentzen et al., 2023; Petersen & Zech, 2024).
D.2. The expressive power of SNNS
Approximation capabilities While the literature on the expressivity of ANNs is extensive, the theory for SNNs still remains quite limited. A valuable foundation for the theory of SNNs lies in the research conducted by Maass in the 90s. These early works introduced several expressivity results for both discrete and contimuous-time models with temporal coding often focusing on the spike response model (Maass, 1994; 1996a; 1997a) or its stochastic extensions (Maass, 1995; 1996;; 1997a;b) for describing the neuronal dynamics. The continuous-time spike response model has been revisited in several recent works (Comsa et al., 2020; Stanojevic et al., 2024; Singh et al., 2023; Neuman et al., 2024). which provide more quantitative approximation results as well as discussions on related theoretical aspects of SNNs. However, as mentioned in (Neuman et al., 2024) the analyzed continuous-time models have not yet found broad applications on dedicated neuromorphic platforms, thus limiting the adaptability of these results to practical settings. Meanwhile, the works (Zhang et al., 2024; Zhang& & Zhou, 2022) examine various SNN models that are closely related to each other and establish several approximation properties. However, these works often include ceff-connections and rate coding in the continuous-time framework in smoothed form, which diverges significantly from common practical SNN implementations.
In contrast to the previously mentioned approaches, our research directly targets straightforward and widely adopted SNN model frequently used in practical SNN implementation frameworks (Eshraghian et al., 2023; Fang et al., 2023a; Gonzalez et al., 2023). Given that SNNs are still evolving with new models being proposed progressively, we do not claim that our paper introduces the most appropriate model of SNNs, but rather that it focuses on the one that is widely used today.
Input partitioning With regard to the input partitioning of SNNs, the piecewise functionality has been observed in (Singh et al., 2023; Mostafa, 2018), which discuss the continuous-time framework (see above discussion). More precisely, these
28

---

Time to Spike? Understanding the Representational Power of SNNS
works consider continuous-time SNNs based on the spike response model combined with the fime-lo-frrst-ppike coding and narrow the focus to the single spike scenario, i.e., context where the (unique) continuous firing time of each neuron can be seen as its activation. Informally speaking, since the input current depends on the (differences of) firing times in certain linear manner (possibly after change of variables as in (Mostafa, 2018)), the activation of post-synaptic neuron also depends linearly on its pre-synaptic neurons' activations. In our case, the linearity dependence between pre- and post-synaptic activations is in a stricter sense, namely because the activations take only binary values.
Kim et al. (2022) investigate an SNN model that is most closely related ours, differing mainly in the membrane potential reset mechanism, and also points out the piecewise linearity of the realization. Notably, the authors introduce time- dependent input partitioning concept the motivation for the proposed neural architecture search, where regions each time step are defined based on neuron transitions as outlined in (Raghu et al., 2017). Compared to (Kim et al., 2022), our work provides a deeper theoretical analysis of the piecewise constant functionality of discrete-time LIF-SNNs, characterizing the complexity of their input partition.
E. The discrete-time LIF-SNN model
In this section we provide further background information for reader less familiar with SNNs. We start with the description of the ceaky-integrate-and-ff model and short derivation to obtain the discretized LIF model employed in this work. Subsequently, we discuss further models of neuronal dynamics and embed the LIF dynamic in this vast space. Moreover, we introduce practically applied coding schemes and show that our coding framework encompasses them. Finally, we derive elementary properties of discrete- time LIF-SNNs.
E.1. The leaky-integrate-and-t neuronal model
The leaky-integratt-and-iii (LIF) neuron is one of the simplest models of neuronal dynamics to study information processing (in biological neural networks) (Gerstner et al., 2014). A linear differential equation, which also underlies a basic capacitor-resistor electrical circuit, describes the LIF dynamics
du(t) dt
u(t) + RI((),
(15)
where u(!) and T()) represent the membrane potential and input current at time I, respectively, while r and R denote the membrane time and impedance constant. The second component, a thresholding operation with parameter 9, translates the dynamics of the potential into spike emission: whenever u(t) reaches the threshold i (from below) a spike is generated and u is immediately reset to a new value below the threshold mimicking the observed biological patterns in a severely simplified fashion.
Solving the differential equation via a forward Euler discretization with equidistant time steps t,, well including the thresholding with the so-called reset-by-subee -usbtraiunttiiiio method, i.e., subtracting the value of the threshold & from the potential after a spike, yields
{ stan) (Bu((tn-)) + (1 (1- (1))) 19
(16)
u((x)) = 1u((nn)) + (1- ()) - Os(l,))
where H = 1 (o) denotes the Heaviside step function and 3 >> O is a coefficient depending on the step size; see for instance (Eshraghian et al., 2023). As a computational model, it is natural to consider networks of spiking neurons and introduce learnable parameters analogously to the structure of moon-spiking) ANNs (independent from biological plausibility). The latter can be achieved by assuming that incoming weighted spikes from pre-synaptic neurons, the so-called spike trains, generate the input current of neuron i
[tt,,) =
Σ
(t.) + b;,
j:d presymattic neuron of
where Wi.j E R denotes synaptic weight replacing/absorbing the coefficient (1 B)R in equation 16) and by E IR is a synaptic bias, whereas s, (tx) E {0,1] specified via equation 16 indicates whether neuron j emitted spike at time ty. This leads to the computational model of a network of spiking LIF neurons, which we refer to as the discrete-time LIF-SNN model, formally introduced in Definition 2. The discrete-time LIF-SNN model is a fairly general and flexible framework to study SNNs encompassing a wide range of practically applied models (see below). Its advantage is that it
29

---

Time to Spike? Understanding the Representational Power of SNNS
incorporates spike-based processing into artificial neural networks while still maintaining advantageous properties from ANNS such the sequential optimization process with gradient methods although some new obstacles arise (Eshraghian et al., 2023). Nevertheless, the number of SNN models employed in practice is vast and therefore we summarize and relate other approaches to the discrete-ting saccettetimme/a LIF-SNN framework. We approach this task from two perspectives. First, we embed the LIF neuron in the landscape proposed by neuroscience to model realistic behaviour of biological neurons. Second, we discuss adaptations of the discrete-time LIF-SNN as a computational model to obtain better performance in practice, i.c., by making it easier to train or exploiting task-dependent properties.
E.2. Neuronal models in biology
For a detail-oriented biological motivation of various neuronal models we refer to (Gerstner et al., 2014; Yamazaki et al., 2022), instead we only provide a short summary. One take-away is that integrate-and-fin (IF) models, thereof LIF is a prominent example, are one of the main classes to describe neuronal dynamics. As the name suggests, these models capture the integration of incoming action potential/spikes and the resulting generation of new spikes, i.e., firing, via differential equations. Single exponential, double exponential, more general non-linear (leaky) IF models extend the basic LIF model, which is based on linear differential equations, to achieve more biological plausibility. The spike response model, a superset of IF models, goes step further by emphasizing various effects occurring in a neuron after emitting a spike, i.c., the spike response, such as refractoriness. Besides, there exists a vast amount of models focusing on various biophysical and biochemical aspects of biological neurons, among them the famous Bodgkinn-Huxlen model. These models are more apt to study physiological processes but (currently) less amenable as computational models employed in computer science since the increased biological plausibility of the models often comes at the cost of higher computational complexity. Hence, the LIF model provides reasonable balance between complexity and efficiency to study the effectiveness of more biologically inspired neurons (in comparison with classical artificial neurons) as a computing framework.
E.3. LIF SNNs computational model
Having established the goal of computational power and efficiency, one is bound by biological plausibility to lesser degree and is inclined to adjust the model to achieve the desired goal. First, employing networks of (LIF) neurons as computing model requires the implementation and subsequent optimization of the network on a computing platform to solve a given task. One key distinction in implementations is the time-discrete versus time-continuous approach. Both are viable from a theoretical perspective but currently, time-discrete solutions are favored in practice, mainly due to two reasons. First, the discretization framework aligns well with the typically employed digital hardware platforms, which however may change with the development of (analog) neuromorphic hardware (Mehonic al., 2024). Second, after discretization, the obtained model such as the discrete- time LIF-SNN can be mostly treated via the established and geghpperformnnc optimization pipeline already developed for ANNs (Eshraghian et al., 2023). Moreover, the time-continuous implementation needs to overcome specific obstacles, which makes the networks currently difficult to scale, although progress has been made in this regard (Göltz et al., 2021; Mostafa, 2018; Comsa et al., 2020).
In the discretized setting, various options are still available. First, depending on the assumptions the discretization via the Euler method might lead to slightly different dynamics (Eshraghian et al., 2023; Neftci et al., 2019). Moreover, the reset mechanism, i.e., the rule how to reset the potential of a neuron once a spike is emitted (meaning the potential crossed the threshold), does not need to follow the introduced susbrbyy- method but can be implemented for instance as reset-to-zero approach. Additionally, the choice of learnable parameters and fixed (hyper)parameters also influences the model capacity. Less learnable parameters ease the training process whereas more learnable parameters obviously extend the computational capacity of the model. Therefore, research focused on obtaining efficient training pipelines incorporating many learnable parameters and/or improving the spiking neuron model to exhibit better learning characteristics; see (Fang et al., 2023b) and the references therein. We skip the specific details but we would like to emphasize that research is still progressing, e.g., by modifying the model to allow for more parallelization in the computing process and thereby more efficiency. Finally, the coding scheme is highly relevant for the performance of the full computational pipeline, independent of the specific spiking model employed. Next, we will deepen this discussion with an emphasis on the output decoding scheme since the input encoding already mostly converged towards direct and learnable schemes.
30

---

Time to Spike? Understanding the Representational Power of SNNs
E.4. Input encoding schemes
Classical encoding schemes such as rate or temporal coding often exhibit several inherent drawbacks that may negatively impact model performance and efficiency (Rueckauer et al., 2017; Wu al., 2019). First, each input neuron can usually represent only grid of T + different values, thus requiring high latency T to achieve high precision. Second, the analog-spike transformation based on rate coding is likely to introduce variability into the firing of the network. In other words, one analog input signal, e.g., an image, may be transformed into a number of rate-coded spike trains and thus lead to totally different output predictions, which not only impairs the model performance but also raises concerns regarding the well-definedness and stability of the model. To prevent such shortcomings as well as to seek for low-latency computations, direct coding has nowadays become a standard input encoding regime for SNNs (Rueckauer et al., 2017; Wu et al., 2019; Fang et al., 2021).
E.5. Output decoding schemes
An important component of any SNN model is how the information encoded in the spikes is decoded into final output. In the discrete LIF-SNN model defined in Section 2, this appears as output decoding mapping D from the space of spike trains to the actual output space, c.g., a real vector space in the considered case of static data. This abstraction covers many common decoding schemes as will be demonstrated in this section. To this end, we categorize them into two main approaches based on whether the output codes directly act on the (binary) spike activations of the final layer ('spike output') or an additional spatial transform is interconnected ("membrane potential output').
B. SPIKE OUTPUTS
To apply the concept of spike outputs, one would certainly require that the size of the last layer of the network is designed to be equal the dimension of the target vectors, i.e., n2 Pour- In our treatment of spike outputs, will always assume that this condition is satisfied. Two of the simplest decoding schemes are rate coding and count coding, where the output signal is understood as the average and spike count of the output spike trains over time, respectively
Example E.1 (Spike rate coding and spike count coding). The output rate decoding mapping is given by
T
D {0, 1 xT
Mar (s(1))
Σ
s().
T
1=1
The oulput count decoding mapping can be written as
T
Droms f0, 13 x
X - Re (s(t) (E(T
H > s(t)
Note that these decoding schemes restrict the output values to be on the grid {0, I)..., possibly rescaled by the factor 1/T. However, these approaches have shown effectiveness in applications where the target set is simple and small such as classification tasks (Eshraghian et al., 2023; Fang et al., 2021). For instance, for nou different classes with one-hot encoded labels E {0, 1 i.e., for class e E [Reat] the corresponding label satisfies y/ = 1 if i e and zero otherwise, the training objective given a discrete-time LIF-SNN & is to minimize C(R(() (=))) y²) averaged over all samples in the dataset (a', y²) ). for some loss function L : TH X R + IR.. Minimizing the objective corresponds to aligning the output vector R( ( (z=) with the one-hot encoded label y with respect to some measure. In the case of rate-coded outputs, this is equivalent to requiring that the neuron in the last spike layer corresponding the correct class fires as many times as possible, while all other neurons in the last layer stay silent often as possible. The idea of forcing many spikes at the neuron associated with the correct class and few spikes at the other neurons can be also applied without an explicit decoding scheme by incorporating the decoding step in the training pipeline, i.e., finding the optimal decoding scheme is part of the learning objective. However, it is not a priori clear how to embed this setting into our framework.
A completely different approach for the output decoding is to rely on the spike times instead of the spike rates/counts via temporal coding. The first spike time of neuron i in the last layer is simply
for min (1 E (T): (f(()) = =],,
if sf = 0 Vt E [7] otherwise,
f
31

---

Time to Spike? Understanding the Representational Power of SNNS
where Ju E IR is value specified beforehand in case neuron i does not spike. The spike time vector f (Ji) [meea E R then used to define the final output vector, possibly after some transformation.
Example E.2 (Spike time/temporal coding). The output spike time mapping can be written as
Diiee: : {0.1)
mo XT + RaN, (s(t)) + h(f)
for some function h : U ( + R applied entry-wise.
For the previously considered classification task with one-hot encoded label, setting So T and h(x)) 1/2 encourages the neuron associated with the correct class to spike early and all other neurons to spike late not at all.
E.5.2. MEMBRANE POTENTIAL OUTPUTS
The decoding regimes that are based on the spike outputs often lead to outputs restricted to the grid {o..... by a simple transformation such as rescaling, inverting, etc. This might be too restrictive in certain tasks where finer accuracy required, e-g. regression tasks (Henkes al., 2024). However, this problem can be addressed by transforming the output spike train of the last layer of (t), t [T] via an affine mapping first. Informally, this can be thought of as adding another layer to the network but without any firing or reset mechanism (which is comparable the last affine layer in ANNs typically employed in regression tasks). More precisely, let WL+1 X x be the weight matrix and d the bias vector of the affine transformation from layer J. to the newly introduced layer I. + 1. Furthermore, let BL+1 be the leaky parameter of that layer and define the membrane potential vector by
T) accompanied
u ++ (t) = 3 u L+ - 1) + W (1)+ b4t1, t [T],
with some initial membrane potential vector u (0) R given beforehand. The explicit formula can now be expressed as
u +1) (8l+1) (1) Σ CBLt1 s i) Σ (BL+T1y844)
R can be treated analogously to spike outputs (s (s"(t) tem E {0,1] Pou x For instance, similarly to the spike rate coding, we can also define the output vector to be the average of the membrane potential time series (u (+1 (f))
Now, the time series of membrane potential vectors (u (t)+
[E(M]]
eetry over time. To embed, this decoding scheme in our framework, first aim express the time series (I) as function of the spike time series (s* (1)) (4)) (E in more compact way. For this, we stack the quantities of different time steps together as a matrix (with each column representing a time step):
U = [u ++ (I),..., ...., 1+1 (T)] RR XT and : [1) st (()) E R
Furthermore, we define the temporal weight vectors
(tt) = [(3*+1))=1 (all))__..BBtt)))) (1 (3 (8 1o0ttt
0. , o RD
and stack them as well as the bias vectors to matrices
A = [111)....... a()] E R Tx and
B = b-+ 8 1+1 ( E R
bb+l]
For clarity, we assume that u 4+1 (0) (the general case a simple extension) and obtain
u(S)= = = (w L+1 S+B)A
for u: {0, 10
R XT
T u
T Σ u(t)= Ulr. T
Example E,3 (Membrane potential rate coding). The membrane potential rate coding mapping can be writlen as
1 T
D Poot-rate R XT - R. (s(t)) ( + D. uS((
( 1++1 S A)
32

---

Time to Spike? Understanding the Representational Power of SNNs
with
T D Ru X RRw, (a(()) + T u())=UU
Hence, in comparison to spike rate coding, membrane potential rate coding incorporates an affine spatial transformation 1R XT + x h(z)= = Wx + B and a linear temporal transformation given by hemmp R XT
given by hspat
Rem homp xx) x =A11 Tr, which assigns a non-uniform weight distribution to the time steps.
In combination with commonly used loss functions, membrane potential rate coding drives the membrane potential of the neuron corresponding the correct class be large, while diminishing the other ones.
E.5.3. SUMMARY
The notion of membrane potential outputs is generic decoding scheme without fixing any specific output decoding regime. It simply assumes that the last spike activations are further passed to subsequent layer via an affine mapping over all time steps similar to spikes being transmitted in the internal layers. This analogy motivates the notion of membrane potential outputs, which are averaged over all time steps given some (temporal) weights. However, note that by substituting the affine mapping with the identity function one recovers a 'spike output approach" that directly relies on the final (binary) spike activations and consequently allows for less flexibility. In practice, the parameters of the decoder, particularly the affine mapping, are considered trainable parameters of the model, which is comparable to the fact that the last layer in conventional ANNs usually consists of trainable affine transformation.
E.6. Elementary properties of discretized LIF SNNs
We want to highlight that discrete-time LIF-SNNs can be linked to feedforward ANNs with Heaviside activation function. The realization of an ANN V can be expressed as
RANN V) = A
oo
where IR + R is activation function that acts entry-wise and '' - + R A'(x) W' + are affine trans- formations between layers - 1 and of size n and nıg respectively, encompassing the trainable parameters (W .T) E Racxa x R. In contrast, given discrete-ting time LIF-SNN & (wW", (a) LE[U) (u' S 0r) DEEE T, (E, D)) its realization can be written as
R( = D (-) W oz 2 of ()) W o E,
where the weight matrices W' represent the affine, in fact linear, mappings and the activation (F((( RRT + RRT operates on the sets of (spatially one-dimensional) time series length
THe (b) ((z + b)
for some b E RT.
For an actual initial spike activation " = (s(t)),e(T) E RT the composition o.r(.)) o W simple represents the (time-enrolled) computation in the 8-th layer according to equation 1
(((()))))))))))
H(W )Em + (b(t))(E(m)
H(W" )eerr
+ "
(.3 u (t - + + 1001-111)))0 (n))))))
(b(t))eETT) is chosen as (0' 1) +b 0f 1n )ee))- Hence, the activation (Fr(.) is not fixed throughout the network but depends via the membrane potential on the initial spike activation. more insightful option to express the realization of
S is given by
R(2) Do 1l e (+ 0f ..... HH A'))
(17)
with A²(:) : XXT + RR xT specified by
(A) (W's b(t)) W + b for s (s(t))e(T) E RRLLIXT b= (b(t))eel R
Note that now the specific form the affine mapping (-) depends on the variable b (which represents the dynamical aspects including the membrane potential of the neuronal model), whereas the activation function H is kept fixed. For the
33

---

Time to Spike? Understanding the Representational Power of SNNs
same choice of b before, immediately verifies that the expression in equation 17 equivalent the one provided in Definition 2.1.
The benefit of equation 17 is the previously mentioned link to feedforward ANNs. In particular, for T = 1 (and appropriate choices for E and D) the model is equivalent to ANNs with Heaviside activation function. Indeed, since there are no temporal dynamics that need to be taken into account the layer-wise affine mapping is simply (((s) W 8 + for some E RV as in case ANNs.
For T > the equivalence between ANNs and discrete-time LIF-SNNs entirely valid anymore, however, structural similarities remain and can potentially be exploited. In partieular, vectorizing the spike activations s (s* ................. s (T))² R (instead of stacking them in matrix) and using that the weight matrix is shared over all time steps in layer, the affine mappings in the realization can be rewritten as
W
A((:) : RT--IT + R (s)
8 s+b for b ... ((()))...(T))" e RTT
(18)
W
Hence, observe that (+) represents special case of the affine mapping in Heaviside ANN with the weight matrix taking block-diagonal structure (where the same block is repeated T times), i.e., the time dimension is expressed in higher-dimensional spatial structure. In contrast to ANNs, the bias term is not fixed but varies (based on the neuronal dynamics) and is therefore dependent on the initial spike activation.
34


Reviews:
[{"review_id": "eWiaEZivZf", "reviewer": "Reviewer_QUDk", "summary": "Authors theoretically analyse a specific LIF neuron model in discrete time. They observe that they realize piecewise constant functions and quantify the network size required to approximate continuous functions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer QUDk for the feedback. Below, we address the mentioned concerns individually:\n\n## 1. Relationship to the work [1]. \nFrom a high level, this work discusses the capacity of neurons under a specific model with time encodings. The capacity in their work is related to the ability of the neuron to learn time-encoded spatio-temporal patterns. This has several key differences with our work, as we list below:\n\n **(1) Model:** The model considered in [1] is a spike-response model with time encoding, which differs from ours in several key aspects. Their approach primarily focuses on a specific time-based encoding for spatio-temporal patterns. In contrast, in the model we consider, time plays a key role in the neuronal dynamics but not in the encoding scheme. Our framework does not rely on a specific encoding strategy tailored to particular data. Instead, it adopts a more general representation of spike patterns, consistent with the use of SNNs as computational models—a perspective reinforced by the widespread adoption of the discrete LIF model in diverse practical applications. Moreover, the methodology in [1] is limited to a single neuron, and therefore does not incorporate the concept of a neural network. \n\n**(2) Capacity measure:** Their measure of capacity appears to be linked to the neuron's ability to learn specific patterns and is therefore closely tied to the learning rule. In contrast, our work does not theoretically address the training process. Instead, our capacity measures are based on function approximation and the study of linear regions. These notions seem largely unrelated to those developed in [1].\n        \n**(3) Motivation:** The primary motivation in [1] is to understand the learning process in biological neurons through a spiking neuron model with biologically plausible learning. In contrast, our work focuses on the discrete LIF model, treating it as a computational framework. Although this model is widely used, many of its theoretical properties remain insufficiently explored. Our work aims to contribute in that direction.\n\nIn summary, our work differs from [1] in its motivation, modeling framework, and capacity measures.\n    \n## 2. Generalization to other neuron models. \n\nFirst, it should be mentioned that the elementary LIF neuron model discussed in our paper is covered by many more complex (and often more biologically plausible) neuron models in the literature as a special case. For instance, the current-based model (see e.g. [Eshraghian et al., 2023]) can be reduced to the LIF model by setting the synaptic current decay rate to $1$. In this case, our universality result can be directly extended to these neuron models. Certainly, SNNs may gain some benefit in terms of expressive power when being constructed based on a more complex neuron model, which may lead to better approximation rates for certain function classes. \n    \nSecond, concerning the input partitioning result, the relevant proofs rely mainly on the affine linearity of the synaptic response (given by the weight matrices and biases) as well as the Heaviside spike activation, which are the common trends in SNNs. Thus, we think that our technique might also apply to other neuron models, yet with several technical modifications. As an example, one may consider the current-based model mentioned above. By taking the sum over time steps, one again obtains a time-dependent affine relation between the membrane potential over the first hidden layer and the input vector, which would lead to an analogous input partitioning as in the LIF case. \n\nWe would also like to clarify that our focus on the basic LIF model is not a limitation but rather because of its widespread use, simplicity, and computational efficiency. While extending our results to other neuron models, as discussed above, is realistic and beneficial, a more rigorous exploration would require careful consideration, and we leave this for future work. \n    \nFinally, our discussion above relies on the time discretization as an essential high-level aspect in the model comparison. We are not aware if the reviewer counts this into the neuron model (i.e. if you are asking for generalization of our results to continuous-time models). A continuous-time model would likely differ crucially from our discrete-time model (please refer to our related works section and also our response to Reviewer V5so), and hence it is not clear to us how to meaningfully generalize our results to continuous time.\n\n\n[1]: *The tempotron: a neuron that learns spike timing–based decisions.* R. Güter and H. Sompolinsky, Nature Neuroscience 9, 420–428, 2006", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes, seem correct.", "relation_to_broader_scientific_literature": "What is relationship to https://pubmed.ncbi.nlm.nih.gov/16474393/.", "essential_references_not_discussed": "Overall sufficient.", "other_strengths_and_weaknesses": "I haven't seen this analysis. I am not surprised by the main observation. I haven't seen but also not checked the detailed theoretical analysis.", "other_comments_or_suggestions": "None.", "questions_for_authors": "How does this generalize to other Neuron models.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gdr4mqiuim", "reviewer": "Reviewer_V5so", "summary": "The manuscript investigates the theoretical expressivity of discrete-time leaky integrate-and-fire (LIF) spiking neural networks (SNNs) and compares them to conventional analog neural networks (ANNs). The manuscript establish that LIF-SNNs realize piecewise constant functions defined on polyhedral regions and derive upper and lower bounds on the network size required to approximate continuous functions. Furthermore, the paper explores the influence of latency (number of time steps) and depth (number of layers) on the complexity of input space partitioning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer V5so for the feedback. Below, we address the concerns individually:\n\n## 1. Training and architectural design \n\nThese features of ANNs may benefit from expressivity insights [1,2, Appendix D.1], while today's SNN architectures and training strategies are typically based on those of ANNs. Given that SNNs function differently, one should reconsider the common practices of architectures and training (e.g., increasing depth and latency, applying convolution, batch normalization, pooling, or applying more complex neuron dynamics). Our analysis questions the possible benefits of certain architectural design techniques and relates them to the increase in computation and complexity expense in training and inference. Suggestions for improving the spatial and temporal components of SNN architectures include e.g. NAS [Kim et al, 2022] or DCLS [3]. Starting with our result, one could study how input partitioning influences the training dynamics/performance, e.g., is it advantageous to initialize networks with a dense and complex baseline input partitioning as in ANNs [2]? Convergence speed and generalization in SNNs could also benefit from this strategy. \n    \n[1]: *Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective*, Chen, Gong, Wang, ICLR 2021\n\n[2] *Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training*, Milkert, Hyde, Laine, arxiv 2025\n\n[3] *Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings*, Hammouamri, Khalfaoui-Hassani, Masquelier, ICLR 2024\n        \n## 2. Continuous- vs discrete-time SNN expressivity\n\n- [Neuman et al 2024] shows that shallow continuous-time SRM-SNNs with temporal coding and linear response function are universal approximators for compactly supported continuous functions. They restrict to single-spike neurons and does not quantify network size.\n\n- [Singh et al 2024; Stanojevic et al 2024] show the ability of similar models in emulating ReLU networks, possibly requiring larger architectures. Due to known results for ReLU networks, one can deduce expressivity results for these SNNs. \n\n- [Comsa et al 2020] shows universality of similar SNNs but with exponential response function. Notably, target function outputs must be bounded below. For $\\epsilon$-accuracy, they require $\\Theta(n(\\Gamma\\sqrt{n}/\\epsilon)^n)$ neurons, compared to our $\\Theta((\\Gamma/\\epsilon)^n)$.\n    \n- For input partitioning, [Singh et al 2024] show that single-output single-layer continuous-time SNNs with linear response realize piecewise linear functions with at most $2^n-1$ pieces. In our case, this bound becomes $(T^2+T+2)/2$ constant regions, independent of input dimension. \n\nHowever, such direct comparisons of complexity across models may be not entirely fair, as different models realize distinct function classes.\n\n## 3. Partitioning complexity with temporal data \n    \nExtending our result to temporal data is challenging, as defining input partitioning is non-trivial. A natural approach is to stack temporal elements and repeat our analysis. While feasible, it is unclear if this approach is meaningful, as the simple stacking process may overlook temporal dependencies. Also, we need to characterize what functions are realized to get a reasonable complexity proxy. Since the output for dynamic data is not necessarily constant, we must reconsider the notion of constant regions.\n\nFor now, we analyze static data as a first step toward that goal. As our experiments aim to validate our theoretical results, which do not cover dynamic data, experiments with dynamic data are beyond this paper’s scope.\n\n## 4. Quadratic growth of partitioning complexity \n\nWith growing latency, both expressivity and computational complexity increase. This is reflected by practical observations that increasing $T$ leads to performance gains, but inference and training become more expensive in terms of energy and time consumption. Therefore, investigating the trade-off between computational and model complexity is important. The quadratic growth of the input partitioning complexity in latency suggests that expanding SNNs in temporal domain may increase the model complexity, but not at an exponential pace. Moreover, Section 5.2 suggests that the constant regions might be very thin with growing latency. Hence, one could prefer a smarter way to scale up and improve performance, rather than simply increasing the latency. This also aligns with the current trend of designing low-latency SNNs. \n    \nCounting regions is just the first step, as it does not provide sufficient practical insights. In classical DL, follow-up studies have been published on various aspects such as shape, size, and evolution over time, which offer valuable practical implications (see Related Works). Similarly, for spiking neural networks (SNNs), it is important to explore these aspects, along with their temporal dependencies.", "claims_and_evidence": "The manuscript claims that LIF-SNNs are universal approximators of continuous functions, that their input space partitioning grows quadratically with latency, and that deeper layers contribute less to expressivity than latency. These claims are well-supported by theoretical derivations. However, while the theoretical framework is compelling, the practical significance of these findings remains uncertain.", "methods_and_evaluation_criteria": "The manuscript mathematically formalizes the expressivity of LIF-SNNs. The evaluation criteria focus on input space partitioning complexity and approximation capabilities. While these are relevant to understanding the computational properties of SNNs, the experimental validation is limited to static datasets (CIFAR-10), which do not fully exploit the temporal nature of SNNs. It would strengthen the paper to test the framework on temporal tasks (sequential decision-making or event-driven processing) to assess its broader applicability.", "theoretical_claims": "The theoretical claims in the manuscript appear to be correct and well-supported. The proofs establish clear upper and lower bounds on the number of neurons required for function approximation and provide a quantitative analysis of input space partitioning complexity. The results extend prior work on Heaviside ANNs and SNNs by explicitly incorporating latency as a factor in expressivity.", "experimental_designs_or_analyses": "The experiments effectively illustrate the theoretical findings but are limited in scope. The choice of static classification benchmarks does not fully demonstrate the implications of latency in real-world neuromorphic computing tasks. Additionally, while the paper provides quantitative comparisons between different architectures (width) and time steps, it lacks a direct comparison to continuous-time SNN models studied in previous works, which would be valuable for positioning discrete-time LIF-SNNs within the broader landscape of neuromorphic computing.", "relation_to_broader_scientific_literature": "The manuscript builds on prior research on ANN expressivity and SNN computational power. It extends results on piecewise constant functions from Heaviside ANNs to discrete-time LIF-SNNs, demonstrating their approximation properties and partitioning behavior. The work aligns with recent studies on neuromorphic computing and the role of time dynamics in SNN learning. However, additional discussion on how these results relate to continuous-time SNN models would further contextualize the findings.", "essential_references_not_discussed": "The paper does not appear to omit any critical references necessary for understanding its key contributions.", "other_strengths_and_weaknesses": "One of the strengths of this work is its theoretical analysis, which provides insights into how discrete-time LIF-SNNs process information. However, the practical applicability of these results remains uncertain. The paper would benefit from additional discussion on how these theoretical insights could inform SNN training, architecture design, or real-world applications.", "other_comments_or_suggestions": "-", "questions_for_authors": "1.\tCould the authors discuss how these theoretical insights could guide training strategies or architectural choices for practical SNN implementations?\n2.\tHow do the expressivity results compare to continuous-time SNN models that employ spike response dynamics rather than discrete-time formulations?\n3.\tCould the authors evaluate the partitioning complexity on tasks with actual temporal dependencies (sequential processing or event-driven tasks)?\n4.\tAre there practical implications of the observed quadratic growth in partition complexity with latency, particularly in terms of computational efficiency or network scalability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AGypQkfVhQ", "reviewer": "Reviewer_9JUu", "summary": "This paper analyzes the representational power of discrete-time leaky integrate-and-fire spiking neural networks (LIF-SNNs) as function approximators. It demonstrates that LIF-SNNs realize piecewise constant functions over polyhedral regions, establishes their universal approximation capabilities, quantifies the required network size, and provides supporting numerical experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 9JUu for the feedback. Below we address each mentioned point:\n\n## 1. Distinction from activation quantization studies\n    \nFirst, we confirm that our analysis in Section 3.2 indeed builds on existing results on Heaviside ANNs, as we point out at line 165 (but should be stated more clearly in a revised version). Moreover, we also discussed the relation to existing results for Heaviside ANNs from [Khalife et al., 2024]. Overall, the proof idea is adopted from the consideration of Heaviside ANNs in this paper, with necessary adjustments due to the considered function class and the specific type of result.\n\nSecond, concerning the ANN-SNN conversion and quantization perspective, we think that this is an interesting and valid viewpoint about SNNs. In fact, this perspective has been discussed in several previous works, e.g. [Eshraghian et al., 2023] or [1]. However, such discussions often do not offer a rigorous mathematical verification or are restricted to special cases. For instance, the work [1] shows the equivalence between ANNs with the quantization clip-floor activation and SNNs based on non-leaky IF neurons with rate coding. In this case, one could study the input partitioning of SNNs through the specific quantized ANNs. However, such an equivalence does not hold in more general cases, in particular in the case of LIF neuron model that we study in our paper. In other words, given a discrete-time SNN model, it is, in general, not clear how to get an exactly equivalent quantized ANN, through which one could better understand the original SNN. Nevertheless, we appreciate the reviewer's perspective about the link between quantized ANNs and SNNs and believe that describing this connection in a rigorous mathematical way would indeed provide meaningful insights and is an important future work.\n    \n## 2. Modeling of multi-step SNNs and its implications\n    \nFirst, the connection between Section 3 and 4 can be explained as follows. Given that discrete-time LIF-SNNs are universal, even in the 'trivial' case of $T=1$, one natural question is what the different hyperparameters, such as depth, width, and especially latency contribute specifically to the network expressivity (and beyond). This question is addressed in Section 4 by studying the model complexity measured by a reasonable metric, which is the richness of the input partitioning. In this sense, Section 4 can be seen as a natural continuation of Section 3, albeit now the latency by design plays a more crucial role compared to the previous analysis. \n\nSecond, we want to clarify the meaning of the last paragraph of Section 3. In general, this paragraph was intended to introduce the similarity (linear partitioning) between multi-step SNNs and Heaviside-ANNs as motivation for the upcoming Section 4. With \"varying\" or \"flexible\" biases, we indeed mean that the biases are input- and time-dependent, as pointed out by the reviewer. This connection is rigorously established Equation 13 (in the appendix) that we referred to in the paragraph. Moreover, we would like to emphasize that we do not assume any simplification of the temporal dependencies. In fact, our analysis accounts for such temporal dependencies and involves computing the exact bias in each time step in relation to inputs and weights. Dealing with the mentioned temporal dependencies is the main point in Lemma 4.2 (or repeated as Lemma B.13 in the appendix) and poses the main challenge in analyzing the actual number of linear regions. \n\nFinally, we will try to clarify these issues in a revised version of the paper to avoid such confusion by modifying the mentioned paragraph. \n    \n## 3. Assumptions on temporal constraints and their impact\n\nAs stressed in the previous point, we do not assume any simplification on temporal constraints, in particular, we do not assume that each time step involves an independent network. In fact, since we took into account the temporal dynamics, the firing pattern of each neuron is not arbitrary (which would lead to $2^T$ different binary patterns from $\\set{0,1}^T$ ), but is restricted to only $T^2$ possible binary codes. This shows that the temporal dynamic has a noticeable limiting effect on the number of regions created by the SNN, as discussed by the reviewer. \n\nAs a side note, we agree with the reviewer that the temporal constraints are helpful to understand the shift behavior observed in Section 5.2. In fact, we were able to prove (after the submission) that in the non-leaky case, the shift converges to the input current. In the leaky case, it seems that the shift still 'tries' to approximate the input current, but the approximation cannot be accurate. This leads to a periodic output spike pattern and thus the observed periodic shift behavior.\n\n[1]: *Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks*, T. Bu, W. Fang, J. Ding, P. Dai, Z. Yu and T. Huang, ICLR 2022.", "claims_and_evidence": "I appreciate the theoretical focus of this work, which is relatively rare in the SNN literature. However, given the complexity of the proofs, even after reviewing parts of the appendix, I am not fully certain that I have grasped all the details. I outline a few key concerns below.\n\n1. Distinction from activation quantization studies: In Section 3.1, the paper acknowledges that for $T=1$, LIF-SNNs are equivalent to Heaviside ANNs, effectively performing 0-1 binarization. This raises the question of whether Section 3.2 primarily builds upon existing results on Heaviside ANNs. Similarly, could the theoretical results in Section 4 be interpreted as an extension to $T$-level quantization? From an ANN-to-SNN conversion perspective, an SNN can be viewed as a higher-order quantized ANN, which seems to suggest potential overlaps with prior work.\n\n2. Modeling of multi-step SNNs and its implications: The theoretical framework in Section 4 appears largely independent of Section 3. Moreover, the modeling of $T$-step SNNs raises concerns. In line 271, the paper claims that an LIF-SNN with $T>1$ can be represented as a binary ANN with a block-diagonal weight matrix and varying bias terms. However, this claim is not entirely rigorous, as the bias terms are not arbitrary but rather determined by the input and weight parameters. This simplification significantly reduces the temporal dependencies that characterize multi-step SNN dynamics, potentially underestimating the complexity introduced by temporal correlations.\n\n3. Assumptions on temporal constraints and their impact: Due to the aforementioned simplification, the subsequent analysis in Section 4 seems to model each time step as an independent network. This implicitly assumes a weak constraint: that at step $t−1$, the firing patterns of neurons are arbitrary and can freely modify the hyperplane structure at step $t$. The combinatorial arguments then proceed based on this assumption. However, in reality, there exist strong temporal constraints between steps $t$ and $t−1$ (e.g., for the first layer, each step $i$ contributes a bias term given by $input \\times i \\mod Threshold$. These constraints could limit the network's expressiveness and might explain certain periodic behaviors observed in Section 5.2 that were not explicitly addressed in the theoretical discussion.", "methods_and_evaluation_criteria": "Given the theoretical nature of the paper, the use of toy experiments for numerical validation is reasonable.", "theoretical_claims": "The theoretical results are complex. While I grasp the general ideas, I have not thoroughly verified the appendix proofs.", "experimental_designs_or_analyses": "The experiments are well-structured and systematically explore low-latency effects in SNNs. The findings in Section 5.3 align directly with the theoretical results on quantization, whereas certain observations in Section 5.2 do not appear to have been anticipated in the theory. This might be related to the assumptions discussed in Question 3.", "relation_to_broader_scientific_literature": "This paper adopts an ANN-to-SNN conversion perspective, interpreting SNNs as a form of quantized ANNs and extending this view beyond binary quantization. This is an interesting and relatively novel approach in the SNN literature.", "essential_references_not_discussed": "No additional references need to be suggested.", "other_strengths_and_weaknesses": "No additional comments.", "other_comments_or_suggestions": "No additional comments.", "questions_for_authors": "My primary questions correspond to the points 1-3 discussed above. Overall, I find this work valuable and insightful, but I would appreciate further clarification on these theoretical assumptions in the rebuttal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NtD0AcJHEh", "reviewer": "Reviewer_Wwin", "summary": "The authors have innovatively proposed a discrete-time Leaky Integrate-and-Fire (LIF) neuron model for Spiking Neural Networks (SNNs), which represents a fundamental and cutting-edge contribution with significant implications for the entire field of neuromorphic computing. The authors have dedicated substantial effort to theoretically show how discrete-time models can approximate continuous functions. Furthermore, they have extensively validated the impact of simulation time steps and the number of layers on the performance of the discrete-time LIF model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank to Reviewer Wwin for the feedback.\n\nWe would like to start our response by clarifying—although this is stated multiple times in the paper (e.g. lines 21, 55, 100) as well as discussed explicitly in the related works section (e.g. lines 1466-1469)—that we are not proposing a new SNN or neuronal model. Rather, we are analyzing a well-known model and providing a rigorous mathematical framework for it. The reviewer may have been confused by our use of the term 'our model' on a few occasions. However, we used this term only after clarifying that the model is already widely used and was intended merely as a shorthand for the one analyzed in our paper-a stylistic choice in writing. We believe this is a common practice, but if it caused confusion, we are open to revising it in the final version of the paper. \n\nOur motivation for formalizing the discussed SNN model in a rigorous framework is to ensure that our results are both mathematically sound, general, and understandable. It is important to note that several variants of the studied model exist (which are captured by our framework), differing in aspects such as which parameters are trainable. Comparisons with other neuron models have already been well established in numerous prior works e.g. [Stanojevic et al., 2024], [Fang et al., 2023], etc. In this regard, we consider the comparative experiments and experimental validation (in the sense suggested by the reviewer) to be beyond the scope of our paper. \n\nOn the other hand, this model is already widely used; for instance, it is essentially the one implemented in many neuromorphic software and hardware platforms such as [Eshraghian et al., 2023], [Fang et al., 2023], [Gonzalez et al., 2023]. Therefore, we find the concern about the feasibility of deployment on neuromorphic chips not valid. \n\nConcerning the reviewer's comment on writing, the related work section is expanded in Appendix D, and we highlighted only the most relevant literature in the related work of the main part. Additionally, we could not resolve the formatting issues yet due to the fixed ICML template, however, this should not be an issue for a camera-ready version.", "claims_and_evidence": "Yes, the claims are very clear.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, I have checked the correctness of all proofs.", "experimental_designs_or_analyses": "The effectiveness of the proposed method has been validated through some experiments. However, the experimental scope of the paper remains relatively limited. For instance, the discrete-time LIF model should be integrated and tested as a replacement for the standard LIF in existing deep SNNs to comprehensively evaluate its impact on performance, inference time, power consumption, and other critical metrics. Such experiments would provide a more thorough and convincing validation of the proposed method's practical applicability and advantages.", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. Designing a novel model is highly significant, as it represents a foundational contribution to the broader fields of SNNs and neuromorphic computing. This work has strong potential for widespread adoption and application.\n\n2. The writing is very clear, easy to understand, and straightforward, making the content accessible to readers.\n\n3. The authors provide rigorous theoretical proofs for the discrete-time LIF model and validate the effectiveness of the proposed neuron model through a series of experiments.\n\n\nWeaknesses:\n\n1. Limited Experimental Validation. The authors have validated the impact of simulation time steps and the number of layers on the performance of the discrete-time LIF model. However, these experiments are insufficient to fully demonstrate the effectiveness of the proposed novel neuron. It would be more convincing if the authors could replace standard LIF or other neurons with the proposed model in various mainstream deep SNNs and compare key metrics such as accuracy, inference time, and power consumption. Additionally, validating the proposed neuron across multiple tasks (e.g., classification, object detection, and even optical flow estimation) would make the work more comprehensive and solid.\n\n2. Lack of Comparative Experiments. The authors did not compare the proposed neuron with other existing neuron models, which is crucial for highlighting the novelty and contributions of their work. Including such comparisons would help demonstrate the advantages of the proposed neuron over existing alternatives.\n\n3. Feasibility Analysis for Neuromorphic Chips Deployment. The paper lacks an analysis of the feasibility of deploying discrete-time LIF-SNNs on neuromorphic hardware. One of the most significant advantages of SNNs is their potential for low-power operation, and the ultimate goal is to validate their performance on neuromorphic computing chips. It is essential for the authors to provide a theoretical feasibility analysis of whether the proposed neuron can be effectively deployed in neuromorphic systems, as this would greatly enhance the practical relevance of their work.\n\n4. Writing Needs Improvements. The paper could benefit from several writing improvements. For instance, the related work section is too brief and should be expanded to provide a more thorough background. Additionally, there are formatting issues, such as excessive spacing on page 8, which should be corrected to improve the overall readability and professionalism of the manuscript.", "other_comments_or_suggestions": "The authors' work is still in its early stages, and with further refinement, it has the potential to become a highly impactful contribution. We look forward to seeing this research evolve into a more comprehensive study, with the possibility of being published in top-tier conferences such as NeurIPS or ICLR in the future. This work holds great promise for advancing the field of neuromorphic computing.", "questions_for_authors": "Please see the weaknesses and response each comment. Besides, two questions are listed below:\n\n1. Why are the proposed discrete-time LIF-SNNs more advantageous than quantized ANNs' activation functions? Although there are some differences in their mechanisms, what are the distinctions in terms of performance, inference time, and power consumption? Which approach do you think is more likely to represent the future trend?\n\n2. Is the deployment of the proposed discrete-time LIF-SNNs on neuromorphic computing chips feasible? For instance, could they be implemented on platforms like Intel Loihi2?\n\nBased on the above comments, I am currently leaning toward a weak reject, or perhaps borderline. However, I will also consider the other reviewers' opinions and the authors' responses. If the authors provide satisfactory answers, I will likely raise my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZwTfva8zTt", "reviewer": "Reviewer_MuQ3", "summary": "The paper presents bounds on the capacity of spiking neural networks to approximate functions on static inputs for discrete-time spiking neural networks. The theory includes a universal approximation theorem (which is simple, as noted by the authors), and d bound on the number of regions. The experiments cover the effect of layer width on the capacity of the network, some effects of the decay term, and a counting of the number of regions in very small networks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer MuQ3 for the feedback. Below we individually address the (not discussed) concerns:\n\n## 1. Abstract/title\n\nWe agree that the focus on static data should be highlighted in the abstract (instead of only in the introduction as of now). We will acknowledge this in the abstract of the revised version. Concerning the title, with 'time' we actually referred to the temporal aspects in the neuronal dynamics itself, i.e. the importance of time in the propagation of information via binary spikes.\n\n## 2. Mismatch between LIF models\n\nAs of today, reset-by-subtraction seems to be the more common choice for the reset mechanism (it is the standard setting in many neuromorphic software platforms such as snntorch or SpikingJelly) in comparison to reset-to-zero (or to a fixed value). Experimental evidence and heuristic reasons (often to avoid information loss) for this choice can be found e.g. in [1], [Eshraghian et al., 2023], [Rueckauer et al., 2017]. Also, we believe that extending our results to the other mechanism is feasible and possibly not too difficult to achieve.\n\n[1]: RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network, Han, Srinivasan, Roy, CVPR 2020\n\n## 3. Proof sketch issue\n\nWe believe that the reviewer is referring to Thm 3.2 rather than Prop 3.1. In this case, the main points of the sketch are (1) continuous functions can be approximated by step functions, and (2) step functions can be realized by discrete-time LIF-SNNs. Such an argument is quite common in deep learning literature on expressivity. In the revised version, we will take into account the reviewer's high-level suggestion to improve the clarity of this proof sketch.\n\n## 4. Significance of Prop 3.3\n\nIn Prop 3.3, the first statement is indeed a direct consequence of Thm 3.2, while the second statement requires a different construction. Basically, the result answers the question of whether the constructive approximation derived in Thm 3.2 is optimal by showing in the worst-case scenario that the number of required neurons is minimal up to constant factors. In the formulation of Prop 3.3, we repeated the result from Thm 3.2 in the first part to facilitate the comparison between upper and lower bounds. As the reviewer noted, this proposition is not applied anywhere else. It is intended as an extension of Theorem 3.2, showing the optimality of the approximation.\n\n## 5. Possible misleading comparison with ReLU case\nIn our analysis, we do not distinguish between the leaky and non-leaky case. Thus, in both cases, the maximum number of regions could achieve the quadratic growth with respect to the latency. The mentioned remark compares the scaling of the maximum number of regions with respect to different hyperparameters, highlighting the role of depth and latency in the model complexity. \n    \nHowever, we think that just directly considering the maximum number of regions is not sufficient, especially when comparing the expressivity of SNNs to ReLU-ANNs. The main reason is while SNNs realize piecewise constant functions, ReLU-ANNs realize piecewise linear functions. Furthermore, it is worthwhile to clarify that other aspects besides the maximum number of regions are also relevant to the model expressivity. This is also the motivation for our ablation study in Section 5.2 and 5.3.\n\n## 6. Experiments in Section 5.1    \n\n- Lack of test accuracy plots: As the reviewer also pointed out, we found these experimental results not relevant to the theoretical expressivity results and therefore excluded them from the paper. However we may add test accuracies to the revised paper for completeness. \n    \n- Choice of datasets: We conducted experiments on MNIST, but our observation was that MNIST is too simple to gain insights into expressivity as the training data is almost interpolated even with very small networks, which makes it difficult to reflect the improvement of the model expressivity along with the subsequent hidden layers' width. Therefore, we think that a more complex dataset like CIFAR10 is more appropriate. We also conducted similar experiments on SVHN, leading to results with the same trend as seen for CIFAR10 and will include these results in the revised version. \n\n## 7. Experiments in Section 5.3\n\n- Table 1: We agree that it would be useful to know how the number of regions scales. However, for now, our naive counting algorithm is based on grid search and may overlook several small regions when $T$ gets larger (given the observations in Section 5.2). In the revised version, we will incorporate the reviewer's suggestion by complementing the case $T=1$ as well as a few more different (small) values for $T$. \n\n- Choice of toy example: Our goal is to show that the model may significantly reduce the number of regions during training. Choosing more complex data may be misaligned with this goal, while requiring larger networks and making region counting prohibitive.", "claims_and_evidence": "I think the claims on the abstract are omitting an important fact: their results are only applicable to static inputs. Even the title refers to time, but the rest of the paper does not really address timing or temporal patterns. The authors should really mention this.", "methods_and_evaluation_criteria": "It is not entirely clear to me why did the authors chose CIFAR for small, feed-forward networks, where something simpler (MNIST) would be more appropriate given the scale used.\nThe use of very small networks in 5.3 is ok for illustration, but it would be useful to know how this scales (if possible, if not explain).\nTable 1 should be complemented by the growth of number of regions with T. ALso, I would imagine that for T=1 this result is the same as for heavyside networks, so it would be useful to know how much the difference is.", "theoretical_claims": "The proofs look reasonable.\n\nHowever, there are a couple of issues:\n- The standard LIF model is not strictly what is used in Eq. 1. There the autors reduce the membrane potential by a fixed amount, whereas the classical LIF model sets the membrane potential to a specific value. The authors should mention this distinction.\n- The proof sketch for 3.1 is rather hard to follow for something that is meant to give an intuition. I think the authors should state something along the lines of: partition the input state space in hypercubes of maximum radius r (bounded by the lipschitz constant I assume), this is the first layer. then partition the ouptut space in a similar fashion, every output is one neuron. Connect every input to every output. Although I might have misunderstood the proof.\n- The first part of Proposition 3.3 seems to be the same as Thm. 3.2. If that is the case, then the proposition should also mention the lower bound. But then again, is not clear to me when do you used the lower bound. So I don't really know why you need this propositoin.\n- The remark on the comparison with RELUS (expressivity growth) could be misleading. If I understand this correctly, the growth might be faster but it would saturate if beta < 1 (periodic case, there is a maximum latency). For beta =1, I am not 100% that the growth will continue (because this might be similar to a quantized network, which is definitelly less expressive than ReLu).\n- I think a comparison with the continuous-time LIF should be discussed.", "experimental_designs_or_analyses": "The experiments are alright, but there are a couple of problems:\n- They only report training accuracies, not testing accuracies. While I get that for the expressivity it does not matter, I think it would be useful to at least know if the trends translate to testing.\n- Why did the authors pick a linear separation toy example, when their theory refers to networks that could compute any function? I think it would be useful to try some more complex logic functions (XOR or parity checks, for example). This would require more regions than currently specified by the linear case.", "relation_to_broader_scientific_literature": "I think that the authors could discuss the relationship of their work with the expessivity of quantized neural networks, specially in the case of beta=1. The rationale is that if the neurons are simply integrating binary inputs, then it can be considered as a form of counting discrete inputs. It would also help to note that quantized networks might in principle be a better solution for low-power ANNs dealing with static inputs.", "essential_references_not_discussed": "Quantized neural networks, if the connection is relevant (which I think it is).\nAlso, whether T=1 recovers the existing bounds with heavyside activation functions on ANNs should be mentionned.", "other_strengths_and_weaknesses": "I think it's good to have such theorems for SNNs. Also, the relevance of the first layer is an interesting angle.\nThe main weakness is that the title/abstract does not mention the key limitation (temporal structure is missing).", "other_comments_or_suggestions": "- Typo in Proposition 3.3, the infinity should be on the low.\n When writing back, please do not be overly polite. I would prefer to have just the answers to the points raised without mentioning how nice of me to do this at every answer.", "questions_for_authors": "I don't see the point of Proposition 3.3. Looks very similar to the previous theorem, just with a lower bound as an add-on that is not really used anywhere else. Did I miss something?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates a theoretical analysis of the discrete-time Leaky Integrate-and-Fire (LIF) neuron model in Spiking Neural Networks (SNNs), approximating continuous functions via piecewise constant functions. The reviewers acknowledge the pape'r's solid theoretical contribution to understanding the expressivity of discrete-time LIF-SNNs. Reviewers V5so and 9JUu highlighted the analysis of delay and depth in input space partitioning. Reviewers Wwin and MuQ3 recognized the significance of this paper to the field of neuromorphic computing.

The reviewers also noted several limitations. Reviewer QUDk raised questions on the generalizability of the results. Reviewers V5so and 9JUu raised concerns about validation. Wwin also pointed out limitations on evaluations and writing. The authors addressed concerns during the rebuttal phase. After the discussion phase, the paper received scores of (4, 4, 3, 3, 3), and none of the reviewers raised further concerns. Therefore, the paper is recommended for acceptance.
Decision: Accept (poster)