PPLNs: Parametric Piecewise Linear Networks for Event-Based Temporal Modeling and Beyond
Authors: Chen Song
Zhenxiao Liang
Bo Sun
Qixing Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Spiking mechanism
event camera
event vision
Primary area: deep_learning_architectures
TL;DR: We present Parametric Piecewise Linear Networks (PPLNs) for temporal vision inference.

Abstract: We present Parametric Piecewise Linear Networks (PPLNs) for temporal vision inference. Motivated by the neuromorphic principles that regulate biological neural behaviors, PPLNs are ideal for processing data captured by event cameras, which are built to simulate neural activities in the human retina. We discuss how to represent the membrane potential of an artificial neuron by a parametric piecewise linear function with learnable coefficients. This design echoes the idea of building deep models from learnable parametric functions recently popularized by Kolmogorov–Arnold Networks (KANs). Experiments demonstrate the state-of-the-art performance of PPLNs in event-based and image-based vision applications, including steering prediction, human pose estimation, and motion deblurring.

PPLNs: Parametric Piecewise Linear Networks for Event-Based Temporal Modeling and Beyond
Chen Song
Zhenxiao Liang Bo Sun Qixing Huang
Department of Computer Science
The University of Texas Austin
Austin, TX 78712
{song, liangzx, bosun, huangax) (aangg))@cs.utexas www.uuteee......... edu
Abstract
We present Parametric Piecewise Linear Networks (PPLNs) for temporal vision inference. Motivated by the neuromorphic principles that regulate biological neural behaviors, PPLNs are ideal for processing data captured by event cameras, which are built to simulate neural activities in the human retina. We discuss how to represent the membrane potential of an artificial neuron by parametric piecewise linear function with learnable coefficients. This design echoes the idea of building deep models from learnable parametric functions recently popularized by Kol- mogorov-Arnold Networks (KANs). Experiments demonstrate the state-of-the-art performance of PPLNS in event- beent-based and image-based vision applications, includ- ing steering prediction, human pose estimation, and motion deblurring. The source code of our implementation is available at htttp::/giiuub..o//chenent http://cccennoogg9/PPPLL 1995/PPLN
1 Introduction
Event cameras are neuromorphic sensors that summarize the evolving world stream of events. Each event describes the pixel coordinates, time, and polarity of intensity change. Thanks to the simplicity of this data representation, event cameras enjoy multiple advantages over conventional cameras, including but not limited to fast data rate and high dynamic range (Gallego al.) 2020).
Over the past, researchers have developed based algorithms to solve various computer vision problems, such motion deblurring (Pan al.) 2019 2020 Wang al. 2020 Song al. 2022
2024). human pose estimation (Calabrese et al.) 2019), 3, and autonomous driving (Bmas et al. 2017
Hu al. ) A recent survey (Zheng al.
2023) indicates a rapidly increasing community
interest in event-based research, with motion deblurring being the most popular task. Extensive experiments demonstrate that by utilizing events as auxiliary input, algorithms perceive fine motion details that are absent from conventional image captures, leading to substantial performance gain over methods that make inferences from conventional images alone.
The success event cameras demonstrates the power of imitating biological neuromorphic principles. The event camera comprises rectangular array of dynamic vision sensors, each of which is analogous to a visual receptor neuron on the human retina dedicated to perceiving one specific location. The neuron experiences excitement and produces spike when the environmental intensity varies significantly, corresponding to an event generated as a response to brightness changes.
This paper presents Parametric Piecewise Linear Networks (PPLNs)) to understand the bio-inspired event data with a bio-inspired deep learning model. As opposed believe and verify in this paper that it is highly beneficial to build a processing network that caters to the principles of the data source (event cameras). As illustrated by [ (Middle), the key idea to explicitly approximate the membrane potential of a neuron as piecewise linear mapping from time to electric voltage. Figure] (Right) presents the sketch of a PPLN node, whose internal mechanism
generic network design, we
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

dendrites
dendrite another neuron
Vi
V logtneembrane potential)
T1
axon
5
3/2
coefficient prediction
(t)
y
VO
Th
soma
synapse
Figure 1: (Left): A biological neuron has three main components: the dendrites (blue), the axon (orange, pink), and the soma (green). The dendrites are responsible for receiving external inputs. The axon transmits signals to the dendrites of other neurons through the synapses. The soma is the body of the cell and connects the dendrites to the axon. (Middle): The membrane potential, defined the voltage difference between the interior and the exterior of the cell, regulates the neuron's 's behavior and can be approximately modeled by a piecewise linear function. When the neuron is at rest, the potential stays at a constant level Vo. @ An external input is received by the dendrites, causing an instantancous perturbation to the membrane potential. @ The perturbation is not significant enough to excite the neuron, and the potential leaks over time exponentially (i.e., linearly in the logarithmic space). Another external input happens. @ The input fails to excite the neuron. @ A third input causes the membrane potential to exceed the threshold voltage Van The neuron becomes excited and generates spike. @ The excitement opens ion channels, and the ion flow causes reset the membrane potential. @ After the excitement, the ion channels close again, and the potential continues to decay. @ The neuron returns to the resting state, waiting for new inputs. (Right): A PPLN node. Given inputs {zi)/= we predict the linear coefficients @ for the membrane potential function, including the slope, intercept, and endpoints of each line segment. The resulting parametric function Ve is then used to evaluate the neuron output at the timestamp of interest (f,,,..., Tk. 1) = a (t)), where a-) is the integral normalization defined in Section 3.4
is explained thoroughly in Section 3 While the event camera imitates how a single layer of visual receptor neurons react external intensity changes, PPLNs are motivated by observations of how layers of biological neurons communicate. Inspired by the Leaky Integrate-and-Fri 1999). we propose to use a piecewise linear parameterization to approximate its temporal evolution in the logarithmic space. The key difference between PPLNs and the bio- inspiredddd Spiking Neural Networks (SNNs) (Eshraghian et al., 202T) existing literature that PPLNs are an alternative to general GPU-based temporal inference models, whereas SNNs aim at the deployment on hardware neuromorphic chips (Davies et al., 2018). Instead of real-valued membrane potentials, SNNs also propagate binary spikes, leading to reduced energy consumption and training instabilities.
model (Abbott
PPLNs conceptually similar the emerging Kolmogonov-Arnol Networks (KANs) (Liu et al. 2024). Both KANs and PPLNs leverage learnable parametric functions to build deep network. Different from KANs, which use input-independent B-splines, PPLNs exploits input-dependent piecewise linear functions. Using piccewise linear functions allows our brain-inspired design to mimic biological neural principles and the event generation model. Predicting function coefficients at inference time allows the network to better handle input heterogeneity.
We modify the network architecture in state-of-the -htt-att event-based vision algorithms by replacing multi-layer perceptions and convolution operators with PPLN nodes and evaluate PPLNs on three applications: steering prediction, 3D human pose estimation, and motion deblurring. With fewer or similar number of trainable parameters, PPL.N improve baselines by 30.8% in steering prediction, 11.1% in human pose estimation, and 5.6% in motion deblurring. To demonstrate the potential of PPLNs, we experiment on the conventional frame-based version of the same applications without the event input and observe consistent improvements. Additionally, we present mathematical analysis of the convergence properties to showcase the robustness.
In summary, make the following contributions:
We propose Parametric Piecewise Linear Networks (PPLNs), mimicking biological principles by approximating membrane potentials as parametric mappings. We show how to predict a set of parametric coefficients from the input of the PPLN node and evaluate the membrane potential at any timestamp of interest.

---

We present a mathematical analysis of the convergence properties of PPLNs.
We apply PPLNs to vent-based and frame-based applications and achieve sate-of-the-as results.
2 Related work
Biological neurons, As shown in Figure (Left) if the combined effect of small perturbations over time causes the membrane potential exceed the threshold potential, the neuron will become excited. During the excitement, the neuron produces an output spike through the synapses (Lacinová) 2005
Maeda et al. 2009). In Figure (Middle), the Leaky Integrate-and-Fire model (Abbott) 1999) suggests that the membrane potential can be approximated by piecewise linear function as parametric mapping from time the logarithm of the voltage difference.
Spiking neural networks. Spiking Neural Networks (SNNs) are Recurrent Neural Networks (RNNs) that simulate biological neurons (Eshraghian al.. 2021). Each SNN node carries internal variable corresponding to the membrane potential. The output of the SNN node is binary signal that is zero by default and becomes one if excited (Abboti [999). The key advantage of SNNs is low power consumption, making them the ideal model to be deployed on hardware neuromorphic chips (Davies) et al. 2018). By contrast, PPLNS designed to be alternative general GPU-based temporal inference models, even though SNNs and PP are motivated by similar biological principles.
Event cameras. Event cameras neuromorphic devices that summarize the evolving environment as a stream of events (Lichtsteiner et al. 2008). Each event is analogous to a significant intensity change that exceeds the hardware threshold and excites biological light-sensing neuron. An event is represented as a 4-tuple (æ, y, t, p), where (x. y) are the pixel coordinates, is the timestamp, and p f-1. +1} is the polarity of the intensity change. Event cameras have fast data rate, high dynamic range, low power consumption, and minimal motion blur compared to conventional cameras (Gallego al. 2020). A recent trend is utilize hardware that simultaneously captures event and conventional streams, where the event stream is used as an auxiliary input to enhance regular computer vision algorithms. For example, while image-to-image deblurring
well-studied
problem in the research community (Richardson] 1972. Fish et al. 1995 Krishnan & Fergus] 2009
Joshi et al. 2009 Levin al. 20077; Kim et al. [998] (haa al. [2008] Fergus al. [2006] Xu et al.
2013; Xu & Jia 2010 Perrone & Favaro [014]] Babacan et al. 2012 Kupyn ct al. 2018 2019))
several works in wvent-based vision demonstrate the possibility of converting a blurry image into a
sharp video that explains the motion during the exposure interval (Pan et al. 2019 2020 Wang et al.
2020
Song et al.] 2022] 2024).
Learning activation functions. The Rectified Linear Unit (ReLU) (Nair & Hinton 2010 represents two-piece linear activation function, f=x) = x if > O, and f(x) 0f æ and has several variants. Our design is conceptually similar to the Piecewise Linear Unit (PWI.U) (Zhou et al. [022 where the activation is defined as an n-piece linear function with Icarnable slopes and intercepts. Kolmogorov-Amold Networks (KANs) have also received wide attention, which build a deep model by stacking layers of parametric B-spline activation functions (Liu al. 2024). In addition the conceptual similarities, PPLNs are fundamentally different from PWLUs and KANs since PPLNs incorporate temporal modeling. Additionally, PPLNs allow discontinuities at segment endpoints and the learning of endpoint locations, neither of which is supported by PWLUs or KANs. Furthermore, the ReLUs are activation functions to be appended after prediction layers (e.g., linear and convolution layers), whereas the PPLNs are designed to replace the prediction layers in temporal learning models.
3 Method
3.1 Overview
As shown in Figure 2 (a, b), a PPLN node implements the following mapping:
(1) where x E R is the k-dimensional non-temporal component of the input, and E [o, 1] is the normalized input scalar timestamp. The mapping converts the input (x, () to a scalar in IR.
f: 1 x [0, 1] RR
The first step the calculation of f to predict the linear coefficients, θ = {m, b, s). using the trainable parameters, W Wi, W.. and wy. Section 3.2 explains this process in detail. The
3

---

predicted coefficients allow us to assemble the piecewise linear membrane potential function, as illustrated by the blue plot in the bottom-left corner of Figure (a. b). To better handle the numerical instability, Section 3.3 discusses the procedure to smooth the boundaries of the predicted linear pieces. The smoothed function is then normalized by another predicted value V, as explained in Section 3.4 Finally, the output of the PPLN node is given as f(x,() (Ve(()) (i.e., the smoothed, and normalized potential function evaluated the timestamp of interest). While it is straightforward to construct fully-com cunnecco network by stacking PPLN nodes, Section 3.5 discusses how to support convolution operations.
3.2 Coefficient prediction
Let be a hyper-paraa parameer denoting the number of line segments in the piecewise linear modeling. The parametric coefficients = (m, b, s} are given by:
m tanh(Wm. x)
(2)
b W,
(3)
softmax(W, x)
(4)
where Wm. We, and W,
& dimensional matrices containing trainable weights.
(m1...., ,, and b = (bi.... "it.....,!?? "n are the slopes and intercepts of n different line segments, respectively. s (81,.... 8n ) defines the temporal interval size for each line segment. Let to and t; 4-1 + st. With the softmax function, the temporal space [0, 1) is divided into
non-overlapping intervals by We predict the aforementioned coefficients and approximate the membrane potential as:
mit + bi mat be
to S
ti S t 12
(o())= =
(5)
mat+by bn fn-1 St S tn
Here. the hyperbolic tangent function restricts the slope, m, preventing the exploding gradient
problem commonly observed in temporal models (Pascanu et al. 2013 Bengio et al. 1994).
3.3 Smoothing
While we can build network by stacking layers of the vanilla PPLN nodes described above, training
presents a challenge for numerical stability. Equation (5) suggests that @Va(t) is an all-zero vector. In 2t
other words, gradient-based optimizers (Bottou et al. 1991 Kingma & Ba 2014) cannot update the
values of 4ss (the location of segment endpoints).
To address this issue, we propose smooth the segment boundaries. The key idea is to blend the linear pieces across adjacent intervals. Let (zx (x)) be the point on the 7th predicted segment, that is, (2) = m4x + b,
Assuming 1 I < to (i.e., belongs to the tt predicted segment), the smoothed potential is:
Vid (t) ( n. (t) + (1 - w/O w ( )=. - w (9 ++)((t)
(6)
where the weights w and w (i) are defined through the temperature hyper- parareraratttrrree T::
w
-1
i 2...., 1 exp (T(4: - 11) -1
exp (T(I (T(I)))-)))
w i
i
0
i == n
Importantly,
1ol (t) (t).. T T-++0
lim
The difference between the smoothed potential, (t), and the unsmoothed potential, (t), is that the smoothed gradients avg (e) do not vanish. We present theorem with proof in the appendix
(t),

---

Wm
We W.
Wm.
W.
W.
Wy
(c)
image
steering
Conv
Linear
angle
events
time
multipyy
convolute
(d)
image x10
pool
m.
(per-pixel
m. b.
steering
nor
Coav
PPLN
angle
events X10
smooth
smooth
image x10
pool
(b)
steering angle x10
Coav
Linear
events X10
Figure 2: (a) A linear PPLN node, which maps the input (x, t) to output f. The trainable parameters Wi,, m. Ws, Ws. and wy. (b) A similarly structured 2D convolutional PPLN node. (c) The baseline architecture for steering angle prediction (Hu). (d) Our model. (e) The modified baseline (HuMod).
showing the local convergence properties of the piecewise linear model after smoothing. Assuming the underlying function to fit is indeed an n-piece piecewise linear function, the theorem states that of noisy samples, provided that the noises are
the coefficients can be accurately learned from reasonably small, the coefficients are adequately initialized, and the temperature I' is sufficiently large. For ease of discussion, the theorem uses segment endpoints {tz] instead of interval lengths s for the parameterization. Their relation is given in Section 3.2
Theorem 3.1. (Informal) Consider underlying n-segment piecewise linear function parameterized by O" = {m", by, } as defined in (5). Let (Tj, Uz). j = I,...: m be m point samples, where "y Ve (T₃) + 1b3 in which 1/ is small random noise.
The 1.2 loss for the smoothed curve is defined by:
c (0)== (T)
(7)
Denote by Oj the weights at which the minimum of ( attained at temperature T. Then we show that starting from some initial Ou close to O", by applying vanilla gradient descent with appropriate temperature increase strategy and learning rate rj ( θ is guaranteed converge to O at linear convergence rate.
Specifically, the error of recovered segments is bounded by:
sup
IVez (T) Ve- (r)l < O(max (ty))s
(8)
Tuin
where Tmius Tmax the smallest and largest values among Tj. j = 1, . respectively.
In addition to the theorem above, the supplementary material uses ablation studies discuss the practical implication of incorporating the smoothing operation.
3.4 Integral normalization
While the smoothing operator introduced above enriches temporal gradients, 8Va(t) at the integral
normalization operator addresses the issue that @Va(t) 8m and (Ve(t) are both very sparse vectors with only one non-zero entry out of all elements. From Equation Sp. we have:
ab
[ Vo (t)dt
Σ mi(tt - t2-1) + Σ bi(tt - f(-1)
(9)
Let V be parameter that controls the mean of (1) when 51511 1. The integral normalization operator (-)) is defined as:
o(V((t) Vo(() f. Vo(t)dt+
(10)
5

---

After the normalization, do(Ve(()) information in every element, encouraging smooth and swift convergence. A side effect of the normalization that the temporal derivative, (o((a(t)) also becomes non-zero, allowing segment endpoints to be learned even without smoothing.
dm and do(Ve(t)) are both dense vectors containing rich gradient
ät
Notably, the ground-truth parameter V is observable in certain applications. In motion deblurring, the task is to generate sharp video from blurry image. Mathematicallly the mean of all the frames in the output must be equal to the input, restricting the temporal average Jo o(Ve (t)) be equal to the input pixel values. When cannot be easily observed, V is regressed from the input x:
V (wV,x)
(11)
where wy is k-dimensional vector of trainable weights, and (..)) stands for the inner product.
Section 4.4 uses ablation studies to demonstrate the effectiveness of integral normalization. In the appendix, we additionally use two-piece toy example to analyze the normalization.
3.5 Supporting the convolution operation
The above modeling naturally extends to convolutions, which are the fundamental building blocks of contemporary deep learning. In convolution layer, each input pixel only affects the output in small spatial neighborhood rather than across the entire grid. To support convolution, the trainable parameters, Wm: W. W., and wy. become sparse matrices and vectors with non-zero entries only in locations within the spatial perceptive field. In practice, we predict the coefficients as:
m
(12)
b conv(W,,
(13)
softmax(conv(W.. x))
(14)
V conv(wv.x)
(15)
where, after reshaping, Wm. Wo. W., and Wy are kernels in the convolution operation. The channel- wise softmax operator ensures the temporal interval sizes of each pixel add up to one, which is requirement posed by the valid range of input timestamps our code release for how the above design is implemented under PyTorch.
[0, 1)). We refer interested readers to
4 Evaluation
Section 4.] starts by showing how PPLNs outperform various methods in motion deblurring, the most popular event-based application as indicated by recent survey (Zheng et al., 2023). In Sections 4.2 and 4.3 we proceed with two other tasks where the goals are to predict the vehicle's steering angle from the dashcam footage and to estimate the 3D human pose from binocular 2D event camera captures. These are two 'mainstream" applications in wvent-based vision, ranking immediately after deblurring, as reported by the survey. Finally, Section 44 and the appendix demonstrate the importance of the integral normalization operator, the effect of changing the number of line segments in the parameterization n, as well the practical implication of the smoothing operator.
ablation studies to
We emphasize event-based applications because event cameras and PPLNs are both designed to mimic biological neural principles. Due to the limited availability of high-quality data, event-based vision is an emerging field where modeling plays more important role than data and the effects of PPLNs can be best demonstrated. However, we also present an evaluation on conventional Trame-based tasks to demonstrate the generalizability. Mcanwhile, we do not include Spiking Neural Network (SNN) basclines. While SNNs focus on the deployment onto neuromorphic chips, PPLNs are designed to be an alternative to general GPU-based temporal inference models.
4.1 Task I: motion deblurring
Task deseription. Event-enhanced motion deblurring is a popular research domain. Variants of the task include image-to-image and image-to-vide video deblurring, Our experiments focus on the highly challenging mnge-tt--vidoo problem, establishing a thorough competition against various of the-art approaches. Given blurry image and its associated events during the exposure interval, our

---

Input
GT
EDI
cSt-Net
E-CIR
DeblurSR
Ours
Figure 3: Motion deblurring visualizations. More are available in the supplementary material.
goal is to reverse the exposure process and reconstruct sharp video describing the relative motion between the camera and the environment. The following experiments utilize the High Quality Frames (HQF) (Stoftregen et al., 2020) dataset and the preprocessing procedure documented by [Song et al. We construct PPLN based on the U-Net architecture (Ronneberger et al., 2015) by replacing all 2D convolution layers with PPLN nodes (Figure 2 (b)). The imput to the PPLN is the concatenation the blurry image and the event histograms, and the histograms are constructed following Zhu al. The output of the PPLN is sharp frames at 14 uniformly spaced timestamps. We utilize ADAM (Kingma) & Ba 2014) to train the network for 50 epochs using the L1 loss. We set the learning rate 10 and reduce the rate by half after 20 and 40 epochs, respectively. The number of line segments is n = 3.
Evaluation metric. We use the Mean Squared Error, the Peak Signal-to-Noise Ratio, and the Structural Similarity Index Measure.
Results and discussions. As shown in Table T (Left), PPLN has a very strong performance in motion deblurring. PPLN improves DeblurSR (Song al. 2024). the state-of-the-ar event-based motion deblurring model at the time of paper submission, by 5.6% in MSE, 0.372 dB in PSNR, and 4.9% in SSIM. Importantly, we underscore that the PPLN is generic network architecture and does not require task-specife modeling, whereas the baseline approaches utilize techniques that cater to the deblurring problem, such as dictionary learning (eSL-Nett (Wang et al. 2020), per-pixel polynomial approximation (E-CIR) (Song et al.) 2022). and implicit neural representation (DeblurSR) (Song (ct al. 2024). The success of the PPLN reveals the strength of mimicking biological neural behaviors.
In Table 1 (Left), we also use regular convolution layers to construct the U-Net. We increase the number of convolutional channels in the regular U-Net approximately match the number of trainable parameters in the PPLN (1 (173M versus 192M). The last two rows suggest that the PPLN improves the U-Net by 54.5% in MSE, 6.45 dB in PSNR, and 43.3% SSIM. Qualitatively, as shown in Figure 3 our method generates sharper and more realistic frames than the baseline approaches. The proposed PPLN offers vivid details around salient features. the first row, our method reconstructs the dark patterns on the white background with sharp edges. In the second row, our method gives the best contrast between the white board and the black letters.
4.2 Task II: steering angle prediction
Task description. The DAVIS Driving Dataset released in 2020 (DDD200 (Hu et al. 2020) contains 51 hours dashcam recordings with both neuromorphic events and conventional frames. Following Hu et al.) we select 15 recordings during the day and 15 at night across the western United States, and train a deep network to regress steering angles from the dual-modal input.
Table 1: (Left): Motion deblurring quality. (Right): Steering prediction errors.
RMSE
MSF image-to-vides
PSNR 1
SSIM t
night
day
baselines
Hu
3.05 = 0.104
5.71 4 0.334
4,35
= 0.236
EDI
17.822
HuMod
2.64
3.94
6S1
0.452
14.988
0.282
Ours
2,53
0.040
0,150 3,68
3.15
0.081
SSL-Net++
0.385
16.870
0.363
EVA
E-CIR
0.207
21.713
0.609
might
day
DeblurSR
0.161
23,912
0.694
N-Net
0.334
17.834
0.508
Hu
0.940 = 0.004
0.713 = 0.033
0.845
= 0.016
HuMod
0.954 4: 0.001
0.864 +: 0.005
0.977 + 0.003
Ours
0.152
24.284
0,728
0.001
7

---

Table 2: (Left): Human pose estimation errors. (Right): Frame-based steering prediction errors.
20 MPJPE_..
Cam 2
Cam
3D MPIPE &
all RMSE 1
all EVA T
Hu HuMod
4: 0.110
0.771 1 0.009
CaldbreseMod
Calabrese
11.90
7.49
11.78
7.29
130.24
82.17
3.55 4: 0.154
0.907 1: 0.007
Ours
6.76
6.51
73.05
Ours
3.16 +: 0.130
0.927 : 0,005
As shown in 2 we build PPLN upon the existing baseline, consisting of convolutional head, a pooling layer, and linear component. The convolutional head reduces the spatial dimension of the data and increases the number of channels. The pooling layer eliminates both spatial dimensions by averaging all the pixels on the reduced spatial grid. Finally, the linear component of the network maps the 64-dimensional feature to the scalar steering angle output. The network contains a total number of 463,425 (463K) parameters. The imput frame and events contain 50 ms of historical data.
Our model (Figure 2 (d)) lowers the prediction frequency to every 500 ms. The network takes ten conventional frames and ten times as many events input and predicts the steering angle at ten uniformly distributed timestamps. We replace the linear component with PPLN nodes and slightly increase the number of layers. To match the number of parameters in the original architecture, we shrink the convolutional head. This results in a network with 455,338 (455K) parameters. We utilize ADAM (Kingma & Ba 2014) to train the network for 200 epochs using the L2 loss. The learning rate 10-8 with a weight decay of 10-4, The number of line segments in the parameterization is n =
Evaluation metric. We use the Root Mean Square Error and the Explained VAriance. We train with five random seeds and report the mean and standard deviation.
Results and discussion. As shown in Table T (Right), our approach outperforms the baseline model in RMSE, with 17.0% improvement night, 35.6% improvement during the day, and 30.8% improvement overall. Similar enhancement is observed in EVA, with 1.9% improvement at night, 23.6% improvement during the day, and 9.6% improvement overall.
The input to our model contains ten times as much information as the baseline approach (Hu et al. 2020). To investigate whether the performance gain is simply result of enriched input information, we present an additional comparison where the baseline is modified to have the same input and output dimensions our model (Figure (e)). From the second and the third rows in Table T (Right), we observe that PPLN improves the modified baseline by 5.7% in RMSE and 1.0% in EVA.
4.3 Task III: human pose estimation
Task description. The dynamic Vision Sensor Human Pose (DHP19) dataset (Calabrese et al. 2019) is collected by inviting human subjects into a cubic space and using event cameras in the four ceiling corners to record various body movements, such as walking, jumping, and walking. DHP19 contains recordings of 17 human subjects performing 33 different body movements. In addition to the events, DHP19 includes the 3D coordinates of 13 body joints. The goal is to predict 3D joint coordinates.
Calabrese et al. use two frontal cameras ("Cam 2" and "Cam 3")) in their experiments. The overall pipeline has two stages. First, they utilize deep network to predict the 2D joint coordinates from the events in each view. After that, they project the 2D predictions into 3D using the calibration matrices.
The deep network used by Calabrese et al. is fully convolutional network (Long al. 2015) that contains 17 layers and 21 218,592 5.59 (219K) trainable parameters. The input events are represented as histograms (Zhu al. 2019), and the output 2D joint coordinates represented as heatmaps. After collecting 25,000 events from all four views, the algorithm assembles event histograms from two frontal cameras and discards the events from the other two cameras. The model updates the 3D joint coordinates if the 2D prediction confidence exceeds threshold r = 0.3 in both views.
In our experiment, the model makes an inference every 250,000 events. The network takes ten times as many events as input and predicts 2D joint coordinates at ten different timestamps. We modify the prediction network by introducing PPLN layers. The modified network contains 215,648 (216K) trainable parameters. We utilize RMSProp (Kingma & Ba 2014) to train the network for 20 epochs using the 1.2 loss. The learning rate is 10 in the first 10 epochs, 10-1 from epochs 10 to 15, and 10-5 from epochs 15 to 20. The number of line segments in the parameterization is n = 3.
Evaluation metric. We use Mean Per Joint Position Error and report in 2D pixels and 3D millimeters.
8

---

Table 3: Ablation studies justifying normalization. The appendix discusses the number of segments.
Motion Deblurring
Steering Prediction
Human Pose
Norm?
MSE
PSNR 1
SSIM
RMSE J
EVA
2D-24
20:3 1
3D
0.264
19.721
0.554
332 = 0.339
0.897 0.018
6.81
6.74
74.88
0.152
24.284
0.728
3 0.081
0.926
0.004
6.76
6.51
73.05
Results and discussion. As shown in Table PPLN estimations have a 2D MPJPE 6.76 pixels in Cam 2, 2D MPJPE of 6.51 pixels in Cam 3, and a 3D MPJPE of 73.05 mm. Compared to the original network used by Calabrese el al., we achieve 9.7% improvement Cam 2, 10.7 improvement in Cam and 11.1% improvement in 3D. Similar to Section 4.2 simply enlarging the temporal horizon confuses the network and leads to performance degradation. Our method enhances the modified baseline by 43.2% in Cam 2, 44.7% in Cam 3, and 44.0% in 3D.
4.4 Ablation studies
In Bection[3.4] we introduce the integral normalization operator, which theoretically stabilizes training by enriching the gradient information. We now ablation studies examine the effectiveness of integral normalization in practice. As shown in Table 3, the introduction of this operator improves motion deblurring quality by 42.4%, 4.563 dB, and 11.4% in MSE, PSNR, and SSIM respectively. For steering prediction, integral normalization improves the accuracy by 17.5% in MSE and 3.2% in EVA. The p-values from one-tailed t-tests are 0.007 and 0.014 for MSE and EVA, giving us reasonable confidence that integral normalization has enhanced accuracy. For human pose estimation, integral normalization improves the 2D MPJPE by approximately 0.2 pixels and the 3D MPJPE by 1.8 mm. The improvement suggests that integral normalization can effectively regularize the training in scenarios such as deblurring where the normalization target has a clear semantic meaning. We refer readers to the appendix for the impact of line segment number n and smoothing on performance.
4.5 Conventional frame-based vision
To demonstrate the generalizability of PPLNs, we remove the event input from the steering angle prediction model. Table 2 (Right) shows that PSNNs still outperform the baselines in the conventional frame-based only setting. Importantly, we observe a significant performance drop by taking out the event input from both baseline approaches. However, the frame-based PPLN has surprisingly similar prediction quality to the dual-modal PPLN. This result demonstrates that by simulating the biological behaviors, our model effectively overcome the imperfections in the input data. Note that among the three tasks discussed above, only steering prediction allows inference from conventional frames alone. Human pose estimation takes events the single-modal input, and image-to-vide video deblurring requires events to address the motion ambiguity.
5 Conclusion and Future Work
This paper presents Parametric Piecewise Linear Networks (PPLNs), novel temporal learning architecture inspired by biological neural principles. The key idea is to represent the membrane potential as parametric piecewise lincar function with predictable coefficients. Experiments on various event-basee based vision applications, including steering prediction, human pose estimation, and motion deblurring, demonstrate that PPLNs outperform state-of-the-an models. In the future, plan to use recurrent prediction model to support a dynamic number of line segments. Another direction adopt more accurate modeling for the membrane potential function, including mechanisms such as the refractory period after each spike.
6 Acknowledgement
Q.H. would like to acknowledge NSF IIS 2047677 and NSF IIS 2413161.

---

References
Abbott, L. F. Lapicque's introduction of the integrate-and-fiee model neuron (1907). Brain research bulletin, 50(5-6):303-30 1999.
Babacan, D., Molina, R., Do, M. N., and Katsaggelos, A. K. Bayesian blind deconvolution with general sparse image priors. In European conference computer vision, pp. 341-355. Springer, 2012.
Bengio, Y., Simard, P., and Frasconi, P. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Binas, J. Neil, D., Liu, S.-C., and Delbruck, T. Ddd17: End-to-end davis driving dataset. arXiv preprint arXiv:1711.01 1.04458, 2017.
Bottou, L. et al. Stochastic gradient learning in neural networks. Proceedings of Newro-Numex, 91))):
12, 1991.
Calabrese, E., Taverni, G., Awai Easthope. C., Skriabine, S., Corradi, F., Longinotti, L., Eng. K., and Delbruck, T. Dhp19: Dynamic vision sensor 3d human pose dataset. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Davies, M., Srinivasa, N., Lin, T.-H., Chinya, G., Cao, Y., Choday, S. H., Dimou, Joshi, P., Imam, N., Jain, S., al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38(1):82 2018.
Eshraghian, J. K., Ward, M.. Neftei, E., Wang. X., Lenz, G., Dwivedi, G., Bennamoun, M., Jeong, D. S., and Lu, W. D. Training spiking neural networks using lessons from deep learning. arXiv preprint 1009, /2894, 2021.
Fergus, R.. Singh, B., Hertzmann, A., Roweis, S. T., and Freeman, W. T. Removing camera shake from single photograph. In ACM SIGGRAPH 2006 Papers, pp. 787-794, 2006.
Fish, D., Brinicombe, A., Pike, E., and Walker, J. Blind deconvolution by means of the richardson- lucy algorithm. JOSA A, 12(1):58-65, 1995.
Gallego, G., Delbruck, T., Orchard, M., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S.,
Davison, A., Conradt, J., Daniilidis, K., and Scaramuzza, D. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intellience, pp. 1-1, 2020. doi: 10.1109/TPAMI. 2020.3008413.
Hu, Y., Binas, J., Neil, D., Liu, S.-C., and Delbruck, Ddd20 end-to-end event camera driving dataset: Fusing frames and events with deep learning for improved steering prediction. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1-6. IEEE, 2020.
Joshi, N., Zitnick, C. L.,, Szeliski, R., and Kriegman, Image deblurring and denoising using color priors. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 1550-1557. IEEE, 2009.
Kim. S. K., Park. S. R., and Paik, J. K. Simultaneous out-of-focus blur estimation and restoration for digital auto-focusing system. IEEE Transactions on Consumer Electronics, 1471-107 1998. doi: 10993003 132336.
Kingma, D. and Ba, J. Adam: method for stochastic optimization. arXiv: 1412.6991)6 2014.
arXiv preprint
Krishnan, D. and Fergus, R. Fast image deconvolution using hyper- priors. Advances in neural information processing systems, 22:1033-1041, 2009.
Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., and Matas, J. Deblurgan: Blind motion deblurring using conditional adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8183-8192, 2018.
10

---

Kupyn, O., Martyniuk, T., J., and Wang. Z. Deblurgan-v2. Deblurring (orders-op of magnitude) faster and better. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2019.
Lacinová, Voltage-dependent calcium channels. General physiology and biophysics, 24:1-78, 2005.
Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910009700, 2019.
Lapicque, L. Recherches quantitatives sur l'excitation electrique des nerfs traitee comme une polarization. Journal de physiologie de pathologie générale, 9:620-635, 635, 1907.
Levin, A., Fergus, R., Durand, F., and Freeman, W. T. Image and depth from conventional camera with coded aperture. ACM transactions on graphics (TOG), 26(3):70-es 2007.
Lichtsteiner, P., Posch, C., and Delbruck, T. 128x 128 120 db 15 jus latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, 43(2):566-577 2008. doi: 10.1 10.1109/SSSC.2007.999 91433377.9.
Liu, Z., Wang. Y., Vaidya, S., Ruehle, F., Halverson, J., Soljačié, M., Hou, T. Y., and Tegmark, M. Kan: Kolmogorov-arnold networks, 2024.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Maass, W. Networks of spiking neurons: The third generation of neural network models.
Neural Networks, 10(9):669--16) 1997. ISSN 0893-6080. doi: https://doi.org/10..00 1016/ (0893-6080(97)00011 URL htthttts::// www. www.ssceeeeedrret.ccooom http://ccccccccccccccctaiieeeeee $0893608097000117
Maeda, S., Nakagawa, S., Suga, M., Yamashita, E., Oshima, A., Fujiyoshi, Y., and Tsukihara, T. Structure the connexin 26 gap junction channel at 3.5 resolution. Nature, 458(7238):597-660. 2009.
Nair, V. and Hinton, G. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., and Dai, Y. Bringing blurry frame alive at high frame-rate with an event camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 6820-6829, 2019.
Pan, L., Hartley, R., Scheerlinck, C., Liu, M., Yu, X., and Dai, Y. High frame rate video reconstruction based event camera. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020.
Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. PMLR, 2013.
Perrone, D. and Favaro, P. Total variation blind deconvolution: The devil is in the details. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2909-2916, 2014. doi: 10. 10.1100//VP.222014.332
Richardson, W. H. Bayesian- based iterative method image restoration. JoSA. 62(1):55-59) 1972.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-usiisted intervention, pp. 234-241. Springer, 2015.
Shan, Q., Jia J., and Agarwala, A. High-quality motion deblurring from a single image. Acm transactions graphics (tog), 27, 2008.

---

Song. C., Huang, Q. and Bajaj, C. E-cir: Event-enhanced continuous intensity recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7803-7812, 2022.
Song, C., Bajaj, C., and Huang, Q Deblursr: Event-based motion deblurring under the spiking representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4900-4908, 2024.
Stoffregen, T., Scheerlinck, C.,, Scaramuzza, D., Drummond, T., Barne,, N., Kleeman, L., and Mahony, R. Reducing the sim-to-real gap for event cameras. In European Conference on Computer
Vision, pp. 534-549. Springer, 2020.
Wang, B., He, J., Yu, L., Xia, G.-S,, and Yang. W. Event enhanced high-quality image recovery, In European Conference on Computer Vision. Springer, 2020.
Xu, L. and Jia, J. Two-phase kernel estimation for robust motion deblurring. In European conference on computer vision, pp. 157-170. Springer, 2010.
Xu, L., Zheng, S., and Jia, J. Unnatural 10 sparse representation for natural image deblurring. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1107-1114, 2013.
Zheng, X., Liu, Y., Lu, Y., Hua, T., Pan, T., Zhang, W., Tao, D., and Wang. L. Deep learning for event-based vision: A comprehensive survey and benchmarks, 2023.
Zhou, Y., Zhu, Z., and Zhong, Z. Learning specialized activation functions with the piecewise linear unit. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12104, 2021.
Zhu, A. Z., Yuan, L., Chaney, K., and Danilidiss, K. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), June 2019.
12

---

A Appendix supplemental material
A.1 Detailed results for task III
We report the 3D MPJPE in millimeters for all 33 different types of body movements evaluated on testing subjects. As shown Table and Table simply increasing the input horizon leads an overall performance degradation. One possible explanation is that the network becomes confused about temporal relationship among the input events. The modified baseline is not a time series model and can only infer the temporal relation from the input stacking order (the first temporal event slice the first channel, the second temporal event slice in the second channel, etc.). the other hand, PPLN keeps track of the timestamp of interest in every model layer. This allows PPLN to outperform the baseline in almost all cases by a significant margin. We point out that Table and Table the asterisk for testing subject in movement 29 indicates missing data in the DHP19 dataset (Calabrese et al. 2019).
Table 4: Baseline 3D MPJPE (mm) for all 33 movements (Mov) across testing subjects (S1-S5).
Calabrese
Calabrese
Mov
ST
S2
S3
S4
S5
S1
S2
S3
S4
S5
129.98
106.31
76.50
208.78
150.00
93.10
131.08
80.60
126.23
141.42
55.99
102.24
114.81
209.18
185.36
112.76
143.94
182.98
184.06
150.85
62.37
101.59
87.35
73.37
91.80
119.69
181.39
127.62
117.21
104.18
4
80.61
97.78
77.96
79.43
75.53
134.74
196.88
114.39
144.19
162.52
218.86
110.42
131.30
184.56
107.39
93.58
85.00
108.69
112.07
115.57
129.51
127.40
110.65
284.49
180.67
134.96
121.44
199.31
149.55
123.78
65.64
77.11
67.41
88.31
78.75
96.42
100.02
83.96
106.65
86.91
8 55.38
80.06
68.16
72.23
73.87
73.71
100.67
88.75
95.25
95.17
27.02
58.79
44.59
67.95
57.86
50.73
100.72
75.09
97.14
89.23
10 44.20
41.18
44.48
74.10
133.35
143.43
86.08
155.72
134.12
221.21
69.24
53.57
54.84
80.80
112.31
176.06
117.72
136.26
142.60
212.07
12 27.05
47.68
43.23
75.01
55.49
63.81
74.67
98.47
118.29
84.77
13 43.24
47.00
51.16
70.75
55.70
83.29
91.36
98.98
114.56
74.45
14
29.65
47.56
44.99
70.29
58.73
69.81
89.07
90.21
113.56
76.72
78.41
152.67
145.61
176.31
115.96
73.83
236.78
119.71
138.51
110.47
16
163.25
171.15
121.49
171.82
115.15
89,90
333.42
126.44
180.29
131.83
17
76.99
157.57
103.21
122.52
104.77
125.37
160.45
100,94
159.16
100.92
18
93.29
134.34
121.31
120.69
118.20
100.01
222.67
176.49
172.93
133.98
19
74.57
131.33
102.23
114.75
101.88
66.02
268.41
90.25
111.69
94.62
116.45
102.90
78.70
112.17
114.54
88.80
171.14
75.72
155.22
96.05
21 34.04
54,99
57.06
68.36
56.26
77.92
97.53
121.81
113.11
102.91
22 59.59
74.76
62.84
100.41
77.84
210.92
168.00
174.50
242.17
219.59
23
99.88
152.25
120.68
103.85
99.92
96.98
172.55
168.81
112.61
113.61
105.15
131.17
125.13
96.49
93.74
112.49
145.38
167.80
119.84
94.63
99.21
124.79
173.60
82.97
128.60
153.60
192.03
235.51
123.17
168.88
26
96.56
109.12
155.25
87.75
134.26
126.87
184.11
185.12
112.14
168.00
27
61.77
159.26
105.29
139.66
123.18
68.55
96.35
64.91
132.84
94.20
28
75.63
116.32
168.53
133.53
110.60
71.64
123.16
83.89
140.78
118.35
68.34
101.05
220.31
159.51
109.18
93.69
107.75
94.93
30
80.43
111.21
107.29
157.46
134.06
119.36
117.88
80.42
166.10
120.45
31
72.11
97.21
107.93
121.62
115.68
88.70
73.28
84.35
92.15
152.39
32
78.22
159.89
114.66
153.90
124.55
78.31
129.95
80.68
95.26
112.57
77.70
101.20
173.90
136.04
135.47
81.68
130.34
157.63
142.18
110.33
Mean
62,23
82,71
79.56
94.13
82.17
I16.63
129.81
130.46
132.89
137.60
A.2
Additional motion deblurring visualizations
Figure presents seven additional examples for the motion deblurring results. We also encourage readers to watch the supplementary animations in the gif format, which demonstrate the temporal smoothness of our results. Overall, the visual quality of PPLN reconstructions is on par with the sfate-of-the-ar method (DeblurSR (Song al. 2024)). Quantitatively, PPLN outperforms DeblurSR by small margin (see paper body). We point out that while all baseline approaches are specifically
13

---

Table 5: PSNN's 3D MPJPE (mm) for all 33 movements (Mov) across 5 testing subjects (S1-S5).
Ours
Mov
S1
S2
S3
S4
S5
45.21
53.92
50.51
97.34
100.01
38.14
59.32
72.31
114.74
104.61
3
58.61
98.07
89.94
70.99
85.94
59.11
107.57
68.32
72.22
81.14
5
51.39
88.91
101.32
120.83
80.06
76.07
88.30
76.61
114.44
109.65
7
48.38
65.89
62.70
80.92
72.89
47.00
69.42
63.82
68.31
68.48
26.96
59.52
43.89
61.84
58.18
10
44.16
44.46
48.24
71.71
130.15
11
66.40
51.82
53.39
78.02
113.78
12
27.17
47.39
44.84
74.25
54.05
13
42.66
47.47
48.26
67.90
52.92
32.54
45.69
42.11
65.85
56.23
15 63.60
141.86
88.23
120.45
101.50
16
60.67
162.74
83.70
121.69
108.83
17
62.47
152.04
77.33
111.53
88.25
18
82.96
113.02
78.43
114.36
93.18
19
58.50
135.82
78.28
95.76
82.82
20
58.75
117.22
65.94
100.34
76,95
21
32.30
62.41
59.28
68.59
55.58
22
52.19
72.31
62.70
99.06
73.88
23
80.53
117.88
111.23
92.39
83.53
24
85.01
119.82
124.99
89.69
69.63
25
94.64
106.63
168.56
78.97
113.24
26
96.09
90.18
133.45
80.44
104.53
42.82
87.68
54.94
84.40
77.92
28
51.48
78.21
56.26
88.34
79.03
51.48
80.17
68.89
78.50
30
68.61
70.03
68.89
107.84
85.72
31
52.56
88.16
86.70
87.64
83.46
32
54.93
79.87
73.45
95.04
88.60
33
63.91
80.45
101.89
111.04
73.66
Mean
52.93
75.11
71.48
80.88
79.54
designed for motion deblurring, the proposed PPLN a general framework that can be applied in various event-based vision tasks.
A.3
The neuromorphic mechanism
Some properties of the proposed model are slightly different from the standard practice in existing research in computer vision and machine learning (i.e., Spiking Neural Networks, SNNs (Maass 1997)), even though both of them are inspired by biological neural principles (Lapicque) 1907).
First, cxisting SNNs focus the interconnection between artificial neurons, with different layers communicating through binary signals. On the other hand, PPLN focuses on representing the membrane potentials, defined as the voltage differences between the interior and the exterior of the cell. The membrane potentials change with time according to the piecewise linear representation, whose parameters predicted from the input.
Second, the signal transmitted between SNN layers carries a one if the neuron is excited and a zero the neuron is not excited. In contrast, our work models the membrane potential using real value. When spike occurs, neuron becomes excited, and the vallvvllud membrane potential function experiences a discontinuous gap.
14

---

Input
Ground Truth
ED1
eSL-Net
E-CIR
DeblurSR
Ours
Figure 4: Motion deblurring visualizations on the HQF dataset.
Third, existing SNNs utilize the linear decay mechanism. In other words, the membrane potential function only decreases in the absence of excitement. This contrasts our parametric mechanism where the slope of each line segment can be either positive or negative. We point out that in biological neurons, the resting potential is not the "absolute" zero. When biological neuron is at rest, there naturally a voltage difference between the interior and the exterior of the cell. If the membrane potential exceeds the resting potential, the neuron will decrease the potential an approximately linear rate. Similarly, the membrane potential drops below the resting potential, the neuron will gradually increase the potential. The linear potential increment corresponds to positive slope in the proposed mechanism.
Fourth, we do not explicitly enforce any line segment to be flat. Instead, we expect the network to implicitly learn when line segment should have a zero slope from the training dataset. This allows more flexibility in the design.
A.4 Coefficient visualization
In figure(3] we plot a few randomly sampled piecewise linear functions predicted by the network. We observe uneven segment lengths and discontinuities (shown in orange) boundaries. The piece-wise linear function has varying slopes, suggesting the model does not collapse to somewhere far away from the design. We also observe that with 3, the network has the capability to represent functions with less than linear segments.
A.5 Baseline implementation
Despite our best efforts, we fail to exactly reproduce the baseline performance reported by Hu et al. for steering angle prediction and Calabrese et al. for human pose estimation. According to Hu

---

Figure 5: Randomly sampled piecewise linear predictions.
al. the baseline model has RMSE of 4.13 + 0.24 and EVA of 0.881 + 0.009. Our baseline implementation gives an RMSE 4.55 + 0.24 and an EVA of 0.845 + 0.016. According to Calabbeese al., the baseline 3D MPJPE is 79.63 mm. Our implementation gives 82.17 mm. Similar confusions have been reported by various teams. We refer readers to the issue pages on their GitHub repositories for detailed discussions.
Compared to the relative improvement brought by PPLNs, the differences above are insignificant. We believe the inconsistency in steering angle prediction is the result of ambiguities in the data pruning procedure discussed in Section III-A the original paper by Hu et al., We provide step-by-step instructions for the data pruning procedure used when generating release. We believe training randomness and differences in optimizer hypermeters cause inconsistency in human pose estimation. Our code release includes the random seed, optimizer hypermeters, and the pre-trained weights for both the baseline model and the PPLN.
reported figures in the code
A.6 Data
In steering angle prediction, we use the DDD20 dataset (Hu et al. 2020) released under GNU Lesser General Public License v3.0. In human pose estimation, we use DHP19 dataset (Calabrese et al.) 2019). The DHP19 dataset is released under the Creative Commons Stribution-ShareAiiie 4.0 International License, and the dataset utility scripts are released under MIT License. motion deblurring, the HQF dataset (Stoffregen al., 2020). At the time of paper writing, the HQF dataset available for public download, but the licensing details are unclear. To the best of our knowledge, the datasets do not contain personally identifiable information or offensive content.
A.7 Computation resources
We train the model for steering angle prediction using NVIDIA TITAN V GPU. We train the model for human pose estimation using one NVIDIA Tesla V100 SXM2 GPU. We train the model for motion deblurring using NVIDIA Tesla V100 SXM2 GPU.
A.8 Limitations
In this paper, we define the number line segments in the piecewise linear approximation the membrane potential function as a hyperparameter n. The choice of this hyperparameter affects the representation capacity the PPLN node, as well as the run-time complexity and convergence rate. At the moment, we are unable to provide theoretical guideline that allows developers choose its value based on the input properties and output requirements. Hyperparameter tuning is needed to balance the prediction quality and computational cost.
16

---

A.9 Proof of Theorem 3.1
Theorem 3.1 Consider an underlying n segment piecewise linear function parameterized by 0 {m" by, t*). Let (T), vy), j L,..., m be m point samples, where vy = Ve- (T,) + t'y. and By is small random noise.
Recall that the ground-truth membrane potential was defined by:
mst bi
S t < ti fi < tz
mēt bz
Ve- (t) =
(16)
m2 + by 1-1 S L7 i
Without losing generality, assume the following constraints:
l min Tj =
15msm
t max Tj 1. I
The following sampling assumption will be put in order to prevent uncontrollable error due to noise. Uniform Sampling Assumption: We assume there exists some constant 0 such that the n, samples (Tu, va)), - -* (Tiy, Vin.) on the 1h segment satisfy
Epsy Tip Tegl
c/ max P.
Tap
Tagl-
Epeol TTp Tig)2
Then under the objective:
(T( = Σ (T)
(17)
and by applying vanilla gradient descent, O* can be recovered up to an error:
max OSE1
V-T e__.
(t)
Ve- (t) < 0 max (85))
(18)
(), the learning rate 17 17 ((4) to the ground truth:
where parameter >> measures the distance from the initialization
(O.) + S |t f/-1 Vi
The uniform sampling assumption is necessary. Imagine a scenario where a dispropor- tionately large number of samples are clustered around (r, (r, (0, 0), with only single sample (T, b) (1, 0). In such an extreme case, the linear regression model might yield a significantly steep slope due to noise, leading to substantial errors when predicting for 1. This example highlights the potential pitfalls of non-uniform sampling in affecting the reliability of regression outcomes. In addition, it can be shown that of uniformly random samples would provide constant with high probability.
Comment:
Proof. Let O be the coefficients in the th iteration of the optimization. We first consider the case where all samples Uy)s are classified into the correct intervals (i.e., (4-((,,))) S Tj < (0,)) implies f Tj S t,). With sufficiently large temperature T, the effect of smoothing becomes minimal, and the estimated potential is:
V (t) (m,t b.) o(cxp(-T))
for all 4:-1 St 1.- On the other hand, if 1 S Tj < t the noisy sample is:
し m T, + 6j w/y
17

---

This gives the objective as:
c² (0) Σ uy)²
/=1
Σ
(miTT + + o(exp(-T)))
15Tyct3
(mfry + b + v; 2
Σ (m= -m;m; + (b. 6U)
イo by + (eepp(-
(19)
To analyze Equation Proposition A.1. The solution for real-valued variables a, b the optimization problem:
19 we present a proposition to simplify notations.
min
ams + +b+
(20)
in which u, 's and o₂'s 's are known constants with the constraint that u, 's are not all the same, satisfies that:
k
k
a S ((z) max max b= k
Ci
Up
h
1=1
where ((u) Eies l14 uyl/ Σies (1t 1144-10 un) u2)
Proof of Proposition A.1 Let L(a, b) = =E((uu; + b+ + c;)² and enforce
al
OL
=0 we have:
de
06
a Σ u + b Σ U + Σ cuu = 0
(21)
and
Σ
Ui + bk: + Σ G 0.
(22)
This gives:
Σ
2 k-1 Σ U1 /))
u
Σ Σ
C Σ cui
Σ 15
(us- -u,)3) 71 (- Σ
-))))² -u/((cal + lc,l)
(wi - uuy(ee -c))
icj
Eiey u2
Eiey 124 uzl
2 max |ci Ciey(u: (w:: uz)²
Therefore, a ((u) max; |c:| and the remaining part regarding b can be obtained directly from Eq. 22 The following corollary applies Proposition A.l to the learning of the piecewise linear coefficients.
18

---

2.0
Tranning Samples
Training Samples
Real Signal
2.0
Real Signal
1.5
Value
1.0
0.5
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
1.0
time
Uime
2.0-
Training Samples
Real Signal
Model
0.5
0.0
0.2
0.4
0.6
0.0
1.0
time
Figure (Left) Initial model. (Middle) Without integral normalization, the model cannot fit piecewise linear signals with unequal segment lengths. (Right) After integral normalization, the model successfully fits the piecewise linear signal with unequal segment lengths.
Corollary A.2. The solution O of the optimization problem:
m
minimize (T) Σ
(Vg (T;) -1)) m2)
(23)
θ
1=1
satisfies:
max [ (T₁) Vol (Ty) 50(
max (v5l) 15/5m
(24)
Proof of Corollary A.2
Comparing Equation
19 and Equation 20). by setting Tj and
Cj -W + olexp] -T)
we have:
mi m O (mmm((()) (b))) + o(exp(-T)))
(25)
b. by + 0 (max( by o(exp(-T)))
(26)
given that the sampling points Ty's are fixed (i.e. ((u) is a constant). When + co, we obtain m, m + O( (max, (1); and b, = b, + O( max, (1bb)) )), which implies:
Vi (T) - Ver(r)l = (0,) m²)r + (l.(O b7)
< o (mxx((1))))
since 0 5 L.
The of the proof shows that under our assumptions, the sample points will eventually be classified into correct intervals.
In the following, objective L (e). Due to noisy samples and smoothing process, the optimal solution is different from O*
(m,(T)), (b:(T)), and (t,(T)} } to denote the optimal solution for the
Consider (Ty: vz)) with 4-1 S < 4. Since (4()))) - -1 s - 44-11, must fall into one of the intervals (to))(00), (-,()) (t.. and [t.)))), (O,)
19

---

In the following, we discuss the stationary point at specific temperature T. In practice, we can stay the temperature until the gradient vanishes and then we move on higher temperature.
ouf (8)
(1) T wi exp (T(T - 4-1))
01,-1
Ow)) (1)
- (0) exp (T(t: 7)),
-T.
dt,
Setting
ac at,
0 we have:
Σ (Arufith (xp((((y-tti)
0= T
((miT) + b,)) - (mi+1 + +b+1)))
-T. Σ (Annf(?? -exp(T(4: - Ty)))
11-1ST <t
((m4+17y + bs+1) - (miTT + b.))
where Ay is defined by:
A = VT((T)) - V2
Thus, we have:
Σ (A exp(TT(T) - 4:))
((miTy + b,) - (mi+IT) + b(+1))
= Σ (Aruf exx((T(4 eexpT(t) -()))
((m4417) b(+1) (m,,, b,)),
or equivalently,
exp(- Σ
(Anufttit) eexp(TT,))
4STyKtH11
((meTy + (,) - (Mi+1Tj + +bst1))
= exp((tt)
Σ
(A)ro(i²² exp((
4 ST1STT
((m4417 bi41) ((iTT+b:i)
(27)
Consider the case where 4 is not currently at the correct location. Without loss of generality, we assume & is too far to the right. This implies that there exists Tj satisfying 4-11 S t S Tj 4,. Suppose:
11-1 S Tjo < T10+1
T3-1 <
and
t S Ty < t S TJ+1
Tt t++1-
Since ........: Tj all fall inside the correct interval, we can fully recover the corresponding segment and have A, O(e 'T) for q j + 1.... J1. The left-hand side of Equation (27) is, therefore. ((exp(-T))) which enforces the right-hand side of the equation to keep getting smaller. Since (TJ, consistent with Tj..... ... J--1 and A's are the eπrors of linear regression, we claim that Ar,r= = jo.....j would never be close to zero. To make Equation (27) hold true, f, has to be decreased. This process will last until f; < T (i.e, the sample point (Tj, Ury) excluded from the wrong interval t.).
In this way, prove that eventually all sample points will be classified into correct intervals, thereby reducing to the base case that has been proved earlier.
20

---

Table 6: Steering prediction accuracy when PPLNs have different numbers of line segments.
Segment Count
6
RMSE
3.15 = 0.081
0.200 0.923 + 0.010
3.23
EVA 1
0.926 + 0.004
Segment Count
12
RMSE +
3.44 = 0.141
3.34 + 0.148
EVA 1
0.913 + 0.007
0.917 + 0.008
Table 7: We use ablation studies to demonstrate the practical implication of the smoothing operator:
Motion Deblurring
Steering Prediction
Human Pose Estimation
Smoothing?
MSE 1
PSNR f
SSIM f
RMSE
EVA f
2D-2
2D-3 L
3D
0.157
23.994
0.718
3.07 + 0.100
0.930 + 0.004
6,58
6.45
71.64
0.152
24.284
0.728
3.15 0.081
0.926 + 0.004
6.76
6.51
73.05
A.10
Justifying integral normalization
Recall that when we construct PPLN node without integral normalization or smoothing, the output
(t) is given
mit + bi m2t + by
to S t t S < l2
(t) =
(28)
mnt+ bn
1- t S tn
With m (m.,..., mn) _ b (b1 (b1.....,b) b,)) and t (to: ................................ ).
have:
(ta) (0, 0, fg. 0. ... 0)
(29)
dm
(Vx(t)) (0.... 0.....01,11 0,1,0 ...
(30)
(ta(t))
(31)
Øt
at the input timestamp 4g S la < ln-
We make two observations here. First, both (Va(ta) 3m and (t4) very sparse vectors with only
one non-zero entry corresponding to the specific segment l. belongs to. At training time, it is natural assume the training timestamps are uniformly distributed across [lo, (n). This means the amount of training data is proportional to the segment length. Long segments receive intensive training because there lot of available training examples. Short segments receive little training because there only a limited number of training examples falling inside their ranges. The imbalance presents an
instability risk because different segments are learning different rates. Additionally, @Va.(t))
øt an
all-zero vector. A -baseed optimizer is therefore unable to adjust the segment lengths. This means the segment endpoints are hard-coded positions instead of trainable parameters.
To illustrate the second observation, consider the toy example of a 2-segment linear signal, as shown in Figure 6 The green real signal generates set of orange training samples with measurement noises. Assuming no prior knowledge of the real signal besides that it has two segments, we initialize 2-segment parametric model with zero slopes, zero intercepts, and equal segment lengths (Figure (Left)). After training, the node converges to the model shown in Figure 6 (Middle). The second segment deviates from the real signal significantly because attempts to accommodate the last several training samples in the first segment.
As discussed in the body, we the integral normalization operator a(-) to address the above challenges:
(vo(t)) = Vo(() S Vo(t)dt+
(32)
where V is a parameter that controls the mean of (t( when 0 1.
21

---

This gives:
@o(Ve(tt)) Vee(ta)
dm
(1t tg. , k-1 ft -
(33)
(o(Ve(ta))
(va(t))
0b
- (11 to: ... 11 1 .nn (n-1)
(34)
do @o((Vett)) (..,, m bk+1 myte- bk.
(35)
Ōt
where all three gradient vectors contain rich gradient information in every element, encouraging smooth and swift convergence. Figure (Right) illustrates how the model accurately approximates both linear segments in the toy example after incorporating integral normalization.
A.11 Ablation studies: number of line segments
As shown in Table 6, we fail to observe noticeable performance improvement when increasing the number of line segments in the formulation. Therefore, decide to 3 in all the experiments discussed in the body of this paper for better efficiency.
A.12 Ablation studies: smoothing
As shown in Table 7 the smoothing operator has an insignificant impact the prediction quality, However, we point out several facts related to the smoothing operator as guideline for potential future applications. First, the smoothing operator allows us to have simple model (Theorem 1) with analytical analyzable properties. Second, the smoothing operator does not introduce any additional trainable parameters. Third, when the size of the network is relatively small (i.e., steering prediction and human pose estimation), our results suggest that using PPLNs without smoothing slightly better. Finally, when the size of the network is large (i.e., motion deblurring), smoothing introduces small performance improvement.
A.13 Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential socictal consequences of our work, none which we feel must be specifically highlighted here. Regarding the environmental impact,
experiments conducted an internal GPU cluster, and the total emission is estimated be 8 81.12 kgCOzeq. equivalent to 328 km driven by an average This emission estimation is conducted using the Machine Learning Impact calculator presented by Lacoste et al. To mitigate repetitive labor and negative environmental impact in future research, we have released open-source implementation together with trained network weights after the anonymous period.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's main contribution is the proposal and evaluation of the parametric piecewise linear networks. In Section 4, our experiment emphasizes event-based vision applications because event cameras and PPLNs are both designed to mimic biological neural principles. Due to the limited availability of high-qualit data, event-based vision is an emerging field where modeling plays more important role than data and the effects of PPLNs can be best demonstrated. However, we also present an evaluation on conventional frame-based tasks demonstrate the generalizability.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed in Section A.8.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
23

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification: Discussed Section A.9.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We have released our source code on GitHub.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
24

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We have released our source code GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/esoc/uuuuie 7/nips cc/ wtc/publicc/ g/guiiggss////eeeeui Codeeobmiiiie SanSSuubissss sorbbbisssiooi ssisssiicccyyyi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Discussed Section 4 with full details in the source code on GitHub. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the mean and standard deviation of evaluation metrics for the steering prediction task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Discussed Section A.7.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully read the NeurlPS Code of Ethics and confirm that our research conforms with the Code.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed in Section A.13.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Discussed in Section A.6.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have released our source code GitHub.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "7bX81Tqn11", "reviewer": "Reviewer_NNdg", "summary": "This paper introduces Parametric Piecewise Linear Networks (PPLNs), a novel approach to temporal modeling inspired by biological neural principles. PPLNs represent neuron membrane potentials as piecewise linear functions with learnable coefficients, aiming to allow for explicit temporal modeling. The authors evaluate PPLNs on three event-based vision tasks: motion deblurring, steering prediction, and human pose estimation, claiming significant improvements over baseline models and some state-of-the-art methods. The paper also attempts to show that PPLNs can generalize to conventional frame-based versions of these tasks.", "questions": "1.\tWhat is the exact computational overhead of PPLNs compared to traditional neural network layers?\n2.\tCan you provide a detailed sensitivity analysis for the key hyperparameters of PPLNs?\n3.\tCan you provide more evidence to support the biological plausibility of PPLNs beyond the initial inspiration?", "limitations": "The authors have addressed some limitations of their work, particularly in the supplementary material. However, a more comprehensive discussion of potential negative societal impacts and failure cases would be beneficial. The paper could also benefit from a more detailed analysis of the computational requirements and scalability of PPLNs.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe PPLN approach presents an interesting attempt to model temporal dynamics in neural networks, drawing inspiration from biological neural systems.\n2.\tThe paper evaluates the proposed method on multiple event-based vision tasks, providing a reasonably broad assessment of its performance.", "weaknesses": "1.\tThe paper lacks a thorough analysis of the computational costs associated with PPLNs. Without this information, it's difficult to assess the practical viability of the approach, especially in comparison to existing methods.\n2.\tWhile the authors claim biological inspiration, the paper does not adequately explore how closely PPLNs actually mimic neuronal behavior. The connection to biological systems seems superficial and not well-substantiated.\n3.\tThe paper's claims about PPLNs' applicability to conventional frame-based tasks are not adequately supported. The limited experiments in this domain do not provide convincing evidence of the method's broader applicability beyond event-based vision.\n4.\tThe paper fails to provide a detailed analysis of how sensitive PPLNs are to hyperparameter choices, such as the number of line segments in the parameterization. This omission raises questions about the robustness and reproducibility of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nHere are the responses to the concerns in your review. We look forward to your additional input during the discussion period. Meanwhile, we encourage you to consider raising the rating if you deem it appropriate.\n\n**Comment:** The paper lacks a thorough analysis of the computational costs associated with PPLNs. Without this information, it's difficult to assess the practical viability of the approach, especially in comparison to existing methods.\n\n**Question:** What is the exact computational overhead of PPLNs compared to traditional neural network layers?\n\n**Response:** Thank you for your comment and question! We analyze the computational cost of PPLNs and the baseline approaches in Section A.8 of the supplementary material. For your convenience, below is a table comparing various performance indicators between our PPLN approach and baseline methods for different tasks. All timing measurements were conducted on the same Titan V server in our institution. For motion deblurring, the baseline approach is U-Net, which has an identical network architecture to PPLNs constructed from regular convolutional layers. For steering prediction and human pose estimation, the baseline approaches are \"Calabrese\" and \"Hu\", respectively.\n\nImportantly, we emphasize that in steering prediction and human pose estimation, PPLNs output ten times as much information as the baseline, which contributes to their slower performance in these two tasks.\n\n| Task | PPLN | Baseline |\n|---------------------|-----------------------------------|-----------------------------------| \n| Motion Deblurring | Training: 30 minutes for 50 epochs | Training: 24 minutes for 50 epochs |\n| | Inference: 16.75 blurry frames/s | Inference: 19.16 blurry frames/s |\n| | Parameters: 192,095,064 | Parameters: 172,622,332 |\n| Steering Prediction | Training: 16.8 hours for 200 epochs | Training: 8.2 hours for 200 epochs |\n| | Inference: 0.27s/iteration | Inference: 0.11s/iteration | \n| | Parameters: 455,338 | Parameters: 463,425 | \n| Human Pose Estimation| Training: 3.1 hours for 20 epochs | Training: 1.3 hours for 20 epochs | \n| | Inference: 0.32s/iteration | Inference: 0.18s/iteration | \n| | Parameters: 215,648 | Parameters: 218,592 |\n\n**Comment:** While the authors claim biological inspiration, the paper does not adequately explore how closely PPLNs actually mimic neuronal behavior. The connection to biological systems seems superficial and not well-substantiated.\n\n**Question:** Can you provide more evidence to support the biological plausibility of PPLNs beyond the initial inspiration?\n\n**Response:** Thank you for your comment and question! In Section A.4 of the supplementary, we discuss how PPLNs are connected with the biological neuromorphic mechanism in detail. In particular, we highlight that PPLNs focus on an explicit parameterization of the membrane potential, which is underexplored in existing work. While they do not follow all neural principles in biology, we hope you agree that PPLNs are closer to real neural systems than conventional artificial neural networks. \n\n**Comment:** The paper's claims about PPLNs' applicability to conventional frame-based tasks are not adequately supported. The limited experiments in this domain do not provide convincing evidence of the method's broader applicability beyond event-based vision.\n\n**Response:** Thank you for your comment! We agree with the reviewer that the experiments in the paper do not justify the use of PPLNs in conventional frame-based tasks. In fact, for most conventional frame-based tasks, we would advocate against the use of PPLNs. As discussed in the paragraph starting at line 195, most conventional frame-based tasks, such as object detection and segmentation, are associated with sufficient high-quality training data. PPLNs are most applicable to scenarios when the training data is limited, causing modeling to play a more important role. Such scenarios include most tasks in event-based vision, as well as very niche situations in conventional frame-based vision.\n\n**Comment:** The paper fails to provide a detailed analysis of how sensitive PPLNs are to hyperparameter choices, such as the number of line segments in the parameterization. This omission raises questions about the robustness and reproducibility of the results.\n\n**Question:** Can you provide a detailed sensitivity analysis for the key hyperparameters of PPLNs?\n\n**Response:** Thank you for your comment and question! We appreciate the opportunity to clarify. Our analysis indicates that the integral normalization constant has the greatest influence on PPLNs, as demonstrated in Table 3. Particularly in motion deblurring tasks, where the ground-truth constant value is available, the MSE improvement attributable to integral normalization can be as high as 42.4%. In tasks such as steering prediction and human pose estimation, where the constant value must be predicted, integral normalization still enhances performance but to a lesser extent. This result demonstrates the importance of using a good value as the integral normalization constant.\n\nFurthermore, our experiments reveal that the number of line segments per layer also impacts prediction quality, as evidenced by Table 7 in the supplementary material. We recommend utilizing three segments per layer as it strikes a balance between quality and parameter count, resulting in optimal performance.\n\n**Comment**: However, a more comprehensive discussion of potential negative societal impacts and failure cases would be beneficial. \n\n**Response**: Thanks for your comment! We fail to observe a noticeable improvement when having sufficient high-quality training data, which is rarely the case in event-based applications. However, this impedes PPLNs from being applied widely to conventional vision tasks. A promising direction is to integrate PPLNs into Parameter-Efficient Fine-Tuning (PEFT) techniques to fine-tune model parameters on small datasets.\n\nSincerely,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kM4ge73NnK", "reviewer": "Reviewer_4tcq", "summary": "1. The paper presents Parametric Piecewise Linear Networks (PPLNs), a novel neural network architecture inspired by neuromorphic principles for temporal vision inference.\n\n2. The innovative approach of PPLNs lies in modeling the membrane potential of artificial neurons as parametric piecewise linear functions with learnable coefficients, echoing the design of Kolmogorov-Arnold Networks (KANs) but with input-dependent adaptability.\n\n3. Experimental results demonstrate PPLNs' state-of-the-art performance in event-based vision applications such as steering prediction, human pose estimation, and motion deblurring, outperforming existing methods with improved accuracy and efficiency.", "questions": "1. The piecewise linear nature of this paper is akin to an input-dependent activation function. An SNN using multi-layer LIF neurons with learnable parameters also seems to achieve a similar level of biological plausibility. Could the authors provide further explanation from this perspective?\n\n2. Based on Figure 2d, I am still unclear whether the input t for PPLN is an external query (you can choose your desire timestamp t) or an inherent timestamp of the event. If it is the former, does this mean that all neurons query with the same input t? If it is the latter, how is this implemented? Given that there are many input events with numerous timestamps, how does the method handle this?\n\nA good response to the Weaknesses and Questions will improve my initial rating.", "limitations": "The authors discussed the environmental impact of the training phase for their proposed solution.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. PPLNs are inspired by neuromorphic principles, mimicking the behavior of biological neurons, which adds a layer of biological plausibility to their computational model.\n\n2. The authors provide a thorough evaluation, including ablation studies, which helps in understanding the contribution of different components of PPLNs to the overall performance. The experiments showcase PPLNs achieving state-of-the-art results across various vision tasks, highlighting the effectiveness of the model in processing event-based data. The paper also evaluates PPLNs on conventional frame-based tasks, showing the model's generalizability beyond just event-based applications.", "weaknesses": "1. The authors did not compare with some of the latest event-based deblurring algorithms, such as REFID[1]. \n\n2. In addition, the authors should provide a direct comparative experiment using a KAN-based backbone to further demonstrate the superiority of the proposed solution.\n\n[1] Sun, Lei, et al. \"Event-based frame interpolation with ad-hoc deblurring.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nHere are the responses to the concerns in your review. We look forward to your additional input during the reviewer-author discussion period. While we are grateful for the weak accept vote, we encourage you to consider raising the rating if you deem it appropriate.\n\n**Comment:** The authors did not compare with some of the latest event-based deblurring algorithms, such as REFID[1].\n\n**Response:** Thank you for your comment! We recognize that REFID is an important work in event vision. However, REFID takes two blurry frames as input and performs frame interpolation. The design of REFID relies on the fact that there are two conventional frames available as input, which deviates from our setting that assumes only one input frame. We will make sure to discuss REFID and why it is not included as a baseline approach in our revision.\n\n**Comment:** In addition, the authors should provide a direct comparative experiment using a KAN-based backbone to further demonstrate the superiority of the proposed solution.\n\n**Response:** Thank you for your comment! Unfortunately, there are two reasons why we are unable to provide such a direct comparative experiment. First, KANs do not have temporal modeling, and there is not yet consensus on how KANs can be modified to support temporal modeling. Additionally, KANs are very slow (c.f., page 33 of 2404.19756). This is why KANs are experimented on very simple toy datasets. Sadly, we do not have access to the computational resources to evaluate KANs on larger-scale applications discussed in our submission.\n\n**Comment:** The piecewise linear nature of this paper is akin to an input-dependent activation function. An SNN using multi-layer LIF neurons with learnable parameters also seems to achieve a similar level of biological plausibility. Could the authors provide further explanation from this perspective?\n\n**Response:** Thank you for your comment! We agree that the proposed PPLN is conceptually very similar to the SNN. However, we highlight four unique characteristics of PPLNs. First, PPLN focuses on representing the membrane potentials instead of the interconnection between artificial neurons. Second, PPLN models the membrane potential using a real value instead of explicit binary spikes. Third, PPLN does not restrict the sign of the slope. Lastly, PPLN does not explicitly enforce any line segment to be flat. For more details, we encourage you to check out the discussion in Section A.4 of the supplementary material.\n\n**Comment:** Based on Figure 2d, I am still unclear whether the input t for PPLN is an external query (you can choose your desire timestamp t) or an inherent timestamp of the event. If it is the former, does this mean that all neurons query with the same input t? If it is the latter, how is this implemented? Given that there are many input events with numerous timestamps, how does the method handle this?\n\n**Response:** Thank you for your comment! We appreciate the opportunity to clarify. The model input consists of two parts, the non-temporal component $\\mathbf{x}$, and the temporal component $t$. For each $\\mathbf{x}$, we typically hope to make inferences at multiple timestamp $t$'s. For example, we are interested in the sharp frames at different timestamps $t \\in [0, 1]$ that correspond to the input blurry image and events ($\\mathbf{x}$). From this perspective, the input $t$ is an external query, and all the neurons receive the same $t$ value.\n\n**Comment:** A good response to the Weaknesses and Questions will improve my initial rating.\n\n**Response:** We sincerely appreciate your willingness to improve the rating! Please let us know if our response has addressed your concerns. We look forward to discussing the submission more with you.\n\nSincerely,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7nC47uYDpW", "reviewer": "Reviewer_oEKu", "summary": "This paper proposes Parametric Piecewise Linear Networks (PPLNs) for event-based temporal modeling, which emulate biological principles by representing membrane potentials as parametric mappings. The authors demonstrate how a straightforward modification enables standard multi-layer perceptrons and convolution operators to accommodate this parameterization of membrane potential. Experimental results showcase that the proposed PPLNs attain state-of-the-art performance in typical event-based vision tasks, including steering prediction, human pose estimation, and motion deblurring.", "questions": "Please see the weakness and response each comment.", "limitations": "There is a lack of analysis of the computational complexity of PPLNs, including experimental tests on the inference time of three downstream event-based vision tasks.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "i) The topic of bio-inspired parametric piecewise linear networks for temporal modeling is very novel and attractive.\n\nii) The authors sufficient experiments in the main paper and the supplemental material to help reader better understand the main contributions of this work.\n\niii) The writing is straightforward, clear, and easy to understand.", "weaknesses": "i) Could you give the time complexity of PPLNs, or the inference time on the three downstream tasks, such as steering prediction, human pose estimation, and motion deblurring?\n\nii) Which hyperparameters have the greatest influence on LLPNs? Please explain the analysis from an experimental perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe appreciate the discussion with you previously in our submission to another venue. We have answered the questions you listed earlier but did not receive your response. May we ask whether you think our rebuttal has clarified your concerns? If so, we kindly encourage you to consider raising the rating. Thanks!\n\n**Comment:** Could you give the time complexity of PPLNs, or the inference time on the three downstream tasks, such as steering prediction, human pose estimation, and motion deblurring?\n\n**Response:** Thank you for your comment! We appreciate the opportunity to provide additional information. Below is a table comparing various performance indicators between our PPLN approach and baseline methods for different tasks. All timing measurements were conducted on the same Titan V server in our institution. For motion deblurring, the baseline approach is U-Net, which has an identical network architecture to PPLNs constructed from regular convolutional layers. For steering prediction and human pose estimation, the baseline approaches are \"Calabrese\" and \"Hu\", respectively.\n\nImportantly, we emphasize that in steering prediction and human pose estimation, PPLNs output ten times as much information as the baseline, which contributes to their slower performance in these two tasks.\n\n| Task | PPLN | Baseline |\n|---------------------|-----------------------------------|-----------------------------------| \n| Motion Deblurring | Training: 30 minutes for 50 epochs | Training: 24 minutes for 50 epochs |\n| | Inference: 16.75 blurry frames/s | Inference: 19.16 blurry frames/s |\n| | Parameters: 192,095,064 | Parameters: 172,622,332 |\n| Steering Prediction | Training: 16.8 hours for 200 epochs | Training: 8.2 hours for 200 epochs |\n| | Inference: 0.27s/iteration | Inference: 0.11s/iteration | \n| | Parameters: 455,338 | Parameters: 463,425 | \n| Human Pose Estimation| Training: 3.1 hours for 20 epochs | Training: 1.3 hours for 20 epochs | \n| | Inference: 0.32s/iteration | Inference: 0.18s/iteration | \n| | Parameters: 215,648 | Parameters: 218,592 |\n\n**Comment:** Which hyperparameters have the greatest influence on LLPNs? Please explain the analysis from an experimental perspective.\n\n**Response:** Thank you for your comment! We appreciate the opportunity to address this question from an experimental perspective. Our analysis indicates that the integral normalization constant has the greatest influence on PPLNs, as demonstrated in Table 5. Particularly in motion deblurring tasks, where the ground-truth constant value is available, the MSE improvement attributable to integral normalization can be as high as 42.4%. In tasks such as steering prediction and human pose estimation, where the constant value must be predicted, integral normalization still enhances performance but to a lesser extent. This result demonstrates the importance of using a good value as the integral normalization constant.\n\nFurthermore, our experiments reveal that the number of line segments per layer also impacts prediction quality, as evidenced by Table 7 in the paper. We recommend utilizing three segments per layer as it strikes a balance between quality and parameter count, resulting in optimal performance.\n\nSincerely,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to the reviewers for their thoughtful evaluation of our submission. While we appreciate recognition from all reviewers, we believe there may be additional aspects of our research that warrant further consideration. Specifically, we feel that Theorem 3.1, which theoretically analyzes the convergence properties of PPLNs, has not received sufficient attention. We also encourage the reviewers to check out our supplementary material, which discusses the biological principles and the hyperparameter choices. In the responses below, we aim to address any remaining concerns and highlight the strengths of our contribution. We welcome further discussion and clarification and are committed to addressing any lingering questions to the best of our ability.

Meta-review: The reviewers recognize the proposed approach as a novel and interesting approach to event-based vision processing, with a convincing demonstration of its performance, including a number of ablation studies. There were some discussions about inference time, hyperparameters and biological plausibility, which the authors could address for the most part during the discussion period. Overall, there is concensus that the paper makes a solid contribution and should be accepted.
Decision: Accept (poster)