Title: PPLNs: Parametric Piecewise Linear Networks for Event-Based Temporal Modeling and Beyond
Venue: NeurIPS 2024
Accepted: yes
Keywords: Spiking mechanism

event camera

event vision
Abstract: We present Parametric Piecewise Linear Networks (PPLNs) for temporal vision inference. Motivated by the neuromorphic principles that regulate biological neural behaviors, PPLNs are ideal for processing data captured by event cameras, which are built to simulate neural activities in the human retina. We discuss how to represent the membrane potential of an artificial neuron by a parametric piecewise linear function with learnable coefficients. This design echoes the idea of building deep models from learnable parametric functions recently popularized by Kolmogorov–Arnold Networks (KANs). Experiments demonstrate the state-of-the-art performance of PPLNs in event-based and image-based vision applications, including steering prediction, human pose estimation, and motion deblurring.

PPLNs: Parametric Piecewise Linear Networks for Event-Based Temporal Modeling and Beyond

Abstract

We present Parametric Piecewise Linear Networks (PPLNs) for temporal vision inference. Motivated by the neuromorphic principles that regulate biological neural behaviors, PPLNs are ideal for processing data captured by event cameras, which are built to simulate neural activities in the human retina. We discuss how to represent the membrane potential of an artificial neuron by a parametric piecewise linear function with learnable coefficients. This design echoes the idea of building deep models from learnable parametric functions recently popularized by Kolmogorov-Arnold Networks (KANs). Experiments demonstrate the state-of-the-art performance of PPLNs in event-based and image-based vision applications, including steering prediction, human pose estimation, and motion deblurring. The source code of our implementation is available at https://github.com/chensong1995/PPLN.

1 Introduction

Event cameras are neuromorphic sensors that summarize the evolving world as a stream of events . Each event describes the pixel coordinates, time, and polarity of an intensity change. Thanks to the simplicity of this data representation, event cameras enjoy multiple advantages over conventional cameras, including but not limited to fast data rate and high dynamic range (Gallego et al., 2020). Over the past, researchers have developed event-based algorithms to solve various computer vision problems, such as motion deblurring (Pan et al., 2019, 2020; Wang et al., 2020; Song et al., 2022, 2024), human pose estimation (Calabrese et al., 2019), and autonomous driving (Binas et al., 2017; Hu et al., 2020). A recent survey (Zheng et al., 2023) indicates a rapidly increasing community interest in event-based research, with motion deblurring being the most popular task. Extensive experiments demonstrate that by utilizing events as an auxiliary input, algorithms perceive fine motion details that are absent from conventional image captures, leading to substantial performance gain over methods that make inferences from conventional images alone.

The success of event cameras demonstrates the power of imitating biological neuromorphic principles. The event camera comprises a rectangular array of dynamic vision sensors , each of which is analogous to a visual receptor neuron on the human retina dedicated to perceiving one specific location. The neuron experiences excitement and produces a spike when the environmental intensity varies significantly, corresponding to an event generated as a response to brightness changes.

This paper presents Parametric Piecewise Linear Networks (PPLNs) to understand the bio-inspired event data with a bio-inspired deep learning model. As opposed to a generic network design, we believe and verify in this paper that it is highly beneficial to build a processing network that caters to the principles of the data source (event cameras). As illustrated by Figure 1 (Middle), the key idea is to explicitly approximate the membrane potential of a neuron as a piecewise linear mapping from time to electric voltage. Figure 1 (Right) presents the sketch of a PPLN node, whose internal mechanism

dendrites

axon

soma

synapse

dendrite of

another neuron

time

log(membrane potential)

①

② ③

④ ⑤

⑥

⑦

⑧ ⑨

coefficient

prediction

V

0

V

th

x

1

x

2

. . .

x

k

t

Θ

˜

V

Θ

(

t

)

σ

(

·

)

y

is explained thoroughly in Section 3. While the event camera imitates how a single layer of visual receptor neurons react to external intensity changes, PPLNs are motivated by observations of how layers of biological neurons communicate. Inspired by the Leaky Integrate-and-Fire model (Abbott, 1999), we propose to use a piecewise linear parameterization to approximate its temporal evolution in the logarithmic space. The key difference between PPLNs and the bio-inspired Spiking Neural Networks (SNNs) (Eshraghian et al., 2021) in existing literature is that PPLNs are an alternative to general GPU-based temporal inference models, whereas SNNs aim at the deployment on hardware neuromorphic chips (Davies et al., 2018). Instead of real-valued membrane potentials, SNNs also propagate binary spikes, leading to reduced energy consumption and training instabilities.

PPLNs are conceptually similar to the emerging Kolmogorov-Arnold Networks (KANs) (Liu et al., 2024). Both KANs and PPLNs leverage learnable parametric functions to build a deep network. Different from KANs, which use input-independent B-splines, PPLNs exploits input-dependent piecewise linear functions. Using piecewise linear functions allows our brain-inspired design to mimic biological neural principles and the event generation model. Predicting function coefficients at inference time allows the network to better handle input heterogeneity.

We modify the network architecture in state-of-the-art event-based vision algorithms by replacing multi-layer perceptions and convolution operators with PPLN nodes and evaluate PPLNs on three applications: steering prediction, 3D human pose estimation, and motion deblurring. With fewer or a similar number of trainable parameters, PPLNs improve baselines by 30.8% in steering prediction, 11.1% in human pose estimation, and 5.6% in motion deblurring. To demonstrate the potential of PPLNs, we experiment on the conventional frame-based version of the same applications without the event input and observe consistent improvements. Additionally, we present a mathematical analysis of the convergence properties to showcase the robustness.

In summary, we make the following contributions:

We propose Parametric Piecewise Linear Networks (PPLNs), mimicking biological principles by approximating membrane potentials as parametric mappings.

We show how to predict a set of parametric coefficients from the input of the PPLN node and evaluate the membrane potential at any timestamp of interest.

We present a mathematical analysis of the convergence properties of PPLNs.

We apply PPLNs to event-based and frame-based applications and achieve state-of-the-art results.

2 Related work

Biological neurons . As shown in Figure 1 (Left), if the combined effect of small perturbations over time causes the membrane potential to exceed the threshold potential, the neuron will become excited. During the excitement, the neuron produces an output spike through the synapses (Lacinová, 2005; Maeda et al., 2009). In Figure 1 (Middle), the Leaky Integrate-and-Fire model (Abbott, 1999) suggests that the membrane potential can be approximated by a piecewise linear function as a parametric mapping from time to the logarithm of the voltage difference.

Spiking neural networks . Spiking Neural Networks (SNNs) are Recurrent Neural Networks (RNNs) that simulate biological neurons (Eshraghian et al., 2021). Each SNN node carries an internal variable corresponding to the membrane potential. The output of the SNN node is a binary signal that is zero by default and becomes one if excited (Abbott, 1999). The key advantage of SNNs is low power consumption, making them the ideal model to be deployed on hardware neuromorphic chips (Davies et al., 2018). By contrast, PPLNs are designed to be an alternative to general GPU-based temporal inference models, even though SNNs and PPLNs are motivated by similar biological principles.

Event cameras . Event cameras are neuromorphic devices that summarize the evolving environment as a stream of events (Lichtsteiner et al., 2008). Each event is analogous to a significant intensity change that exceeds the hardware threshold and excites a biological light-sensing neuron. An event is represented as a 4-tuple ( x, y, t, p ) , where ( x, y ) are the pixel coordinates, t is the timestamp, and p ∈ {-1 , +1 } is the polarity of the intensity change. Event cameras have a fast data rate, high dynamic range, low power consumption, and minimal motion blur compared to conventional cameras (Gallego et al., 2020). A recent trend is to utilize hardware that simultaneously captures event and conventional streams, where the event stream is used as an auxiliary input to enhance regular computer vision algorithms. For example, while image-to-image deblurring is a well-studied problem in the research community (Richardson, 1972; Fish et al., 1995; Krishnan & Fergus, 2009; Joshi et al., 2009; Levin et al., 2007; Kim et al., 1998; Shan et al., 2008; Fergus et al., 2006; Xu et al., 2013; Xu & Jia, 2010; Perrone & Favaro, 2014; Babacan et al., 2012; Kupyn et al., 2018, 2019) , several works in event-based vision demonstrate the possibility of converting a blurry image into a sharp video that explains the motion during the exposure interval (Pan et al., 2019, 2020; Wang et al., 2020; Song et al., 2022, 2024).

Learning activation functions . The Rectified Linear Unit (ReLU) (Nair & Hinton, 2010) represents a two-piece linear activation function, f ( x ) = x if x > 0 , and f ( x ) = 0 if x ≤ 0 and has several variants. Our design is conceptually similar to the Piecewise Linear Unit (PWLU) (Zhou et al., 2021), where the activation is defined as an n -piece linear function with learnable slopes and intercepts. Kolmogorov-Arnold Networks (KANs) have also received wide attention, which build a deep model by stacking layers of parametric B-spline activation functions (Liu et al., 2024). In addition to the conceptual similarities, PPLNs are fundamentally different from PWLUs and KANs since PPLNs incorporate temporal modeling. Additionally, PPLNs allow discontinuities at segment endpoints and the learning of endpoint locations, neither of which is supported by PWLUs or KANs. Furthermore, the ReLUs are activation functions to be appended after prediction layers ( e.g. , linear and convolution layers), whereas the PPLNs are designed to replace the prediction layers in temporal learning models.

3 Method

3.1 Overview

As shown in Figure 2 (a, b), a PPLN node implements the following mapping:

where x ∈ R k is the k -dimensional non-temporal component of the input, and t ∈ [0 , 1] is the normalized input scalar timestamp. The mapping f converts the input ( x , t ) to a scalar in R .

The first step in the calculation of f is to predict the linear coefficients, Θ = { m , b , s } , using the trainable parameters, W m , W b , W s , and w V . Section 3.2 explains this process in detail. The

predicted coefficients allow us to assemble the piecewise linear membrane potential function, as illustrated by the blue plot in the bottom-left corner of Figure 2 (a, b). To better handle the numerical instability, Section 3.3 discusses the procedure to smooth the boundaries of the predicted linear pieces. The smoothed function is then normalized by another predicted value V , as explained in Section 3.4. Finally, the output of the PPLN node is given as f ( x , t ) = σ ( ˜ V Θ ( t )) ( i.e. , the smoothed, and normalized potential function evaluated at the timestamp of interest). While it is straightforward to construct a fully-connected network by stacking PPLN nodes, Section 3.5 discusses how to support convolution operations.

3.2 Coefficient prediction

Let n be a hyper-parameter denoting the number of line segments in the piecewise linear modeling. The parametric coefficients Θ = { m , b , s } are given by:

where W m , W b , and W s are n × k dimensional matrices containing trainable weights. m = ( m 1 , . . . , m n ) T and b = ( b 1 , . . . , b n ) T are the slopes and intercepts of n different line segments, respectively. s = ( s 1 , . . . , s n ) T defines the temporal interval size for each line segment. Let t 0 = 0 and t i = t i -1 + s i . With the softmax function, the temporal space [0 , 1] is divided into n non-overlapping intervals by s . We predict the aforementioned coefficients and approximate the membrane potential as:

Here, the hyperbolic tangent function restricts the slope, m , preventing the exploding gradient problem commonly observed in temporal models (Pascanu et al., 2013; Bengio et al., 1994).

3.3 Smoothing

While we can build a network by stacking layers of the vanilla PPLN nodes described above, training presents a challenge for numerical stability. Equation (5) suggests that ∂ ˜ V Θ ( t ) ∂ t is an all-zero vector. In other words, gradient-based optimizers (Bottou et al., 1991; Kingma & Ba, 2014) cannot update the values of t i 's (the location of segment endpoints).

To address this issue, we propose to smooth the segment boundaries. The key idea is to blend the linear pieces across adjacent intervals. Let ( x, π i ( x )) be the point on the i th predicted segment, that is, π i ( x ) := m i x + b i .

Assuming t i -1 ≤ t < t i ( i.e. , t belongs to the i th predicted segment), the smoothed potential is:

where the weights w ( i ) ← and w ( i ) → are defined through the temperature hyper-parameter T :

Importantly,

The difference between the smoothed potential, ˜ V T Θ ( t ) , and the unsmoothed potential, ˜ V Θ ( t ) , is that the smoothed gradients ∂ ˜ V T Θ ( t ) ∂ t do not vanish. We present a theorem with proof in the appendix

multiply

smooth

norm

convolute

smooth

norm

(per-pixel)

image

events

Conv

pool

steering

angle

time

(c)

Linear

image x10

events x10

Conv

pool

steering

angle

PPLN

image x10

events x10

Conv

pool

steering

angle x10

Linear

...

(a)

(b)

(d)

(e)

x

x

W

m

W

b

W

s

w

V

W

m

W

b

W

s

w

V

m

,

b

,

s

m

,

b

,

s

V

V

t

f

t

f

showing the local convergence properties of the piecewise linear model after smoothing. Assuming the underlying function to fit is indeed an n -piece piecewise linear function, the theorem states that the coefficients can be accurately learned from a set of noisy samples, provided that the noises are reasonably small, the coefficients are adequately initialized, and the temperature T is sufficiently large. For ease of discussion, the theorem uses segment endpoints t = { t i } instead of interval lengths s for the parameterization. Their relation is given in Section 3.2.

Theorem 3.1. (Informal) Consider an underlying n-segment piecewise linear function parameterized by Θ ⋆ = { m ⋆ , b ⋆ , t ⋆ } as defined in (5). Let ( τ j , v j ) , j = 1 , . . . , m be m point samples, where v j = ˜ V Θ ⋆ ( τ j ) + ψ j in which ψ j is a small random noise.

The L2 loss for the smoothed curve is defined by:

Denote by Θ ⋆ T the weights at which the minimum of (7) is attained at temperature T . Then we show that starting from some initial Θ 0 close to Θ ⋆ , by applying vanilla gradient descent with appropriate temperature increase strategy and a learning rate η = O ( 1 T ) , Θ is guaranteed to converge to Θ ⋆ ∞ at a linear convergence rate.

Specifically, the error of recovered segments is bounded by:

where τ min , τ max are the smallest and largest values among τ j , for j = 1 , . . . , m , respectively.

In addition to the theorem above, the supplementary material uses ablation studies to discuss the practical implication of incorporating the smoothing operation.

3.4 Integral normalization

While the smoothing operator introduced above enriches temporal gradients, ∂ ˜ V Θ ( t ) ∂ t , the integral normalization operator addresses the issue that ∂ ˜ V Θ ( t ) ∂ m and ∂ ˜ V Θ ( t ) ∂ b are both very sparse vectors with only one non-zero entry out of all n elements. From Equation (5), we have:

Let V be a parameter that controls the mean of ˜ V Θ ( t ) when 0 ≤ t ≤ 1 . The integral normalization operator σ ( · ) is defined as:

After the normalization, ∂σ ( ˜ V Θ ( t )) ∂ m and ∂σ ( ˜ V Θ ( t )) ∂ b are both dense vectors containing rich gradient information in every element, encouraging a smooth and swift convergence. A side effect of the normalization is that the temporal derivative, ∂σ ( ˜ V Θ ( t )) ∂ t , also becomes non-zero, allowing segment endpoints to be learned even without smoothing.

Notably, the ground-truth parameter V is observable in certain applications. In motion deblurring, the task is to generate a sharp video from a blurry image. Mathematically, the mean of all the frames in the output must be equal to the input, restricting the temporal average ∫ 1 0 σ ( ˜ V Θ xy ( t )) to be equal to the input pixel values. When it cannot be easily observed, V is regressed from the input x :

where w V is a k -dimensional vector of trainable weights, and ⟨· , ·⟩ stands for the inner product.

Section 4.4 uses ablation studies to demonstrate the effectiveness of integral normalization. In the appendix, we additionally use a two-piece toy example to analyze the normalization.

3.5 Supporting the convolution operation

The above modeling naturally extends to convolutions, which are the fundamental building blocks of contemporary deep learning. In a convolution layer, each input pixel only affects the output in a small spatial neighborhood rather than across the entire grid. To support convolution, the trainable parameters, W m , W b , W s , and w V , become sparse matrices and vectors with non-zero entries only in locations within the spatial perceptive field. In practice, we predict the coefficients as:

where, after reshaping, W m , W b , W s , and w V are kernels in the convolution operation. The channelwise softmax operator ensures the temporal interval sizes of each pixel add up to one, which is a requirement posed by the valid range of input timestamps ( t ∈ [0 , 1] ). We refer interested readers to our code release for how the above design is implemented under PyTorch.

4 Evaluation

Section 4.1 starts by showing how PPLNs outperform various methods in motion deblurring, the most popular event-based application as indicated by a recent survey (Zheng et al., 2023). In Sections 4.2 and 4.3, we proceed with two other tasks where the goals are to predict the vehicle's steering angle from the dashcam footage and to estimate the 3D human pose from binocular 2D event camera captures. These are two 'mainstream' applications in event-based vision, ranking immediately after deblurring, as reported by the survey. Finally, Section 4.4 and the appendix use ablation studies to demonstrate the importance of the integral normalization operator, the effect of changing the number of line segments in the parameterization n , as well as the practical implication of the smoothing operator.

Weemphasize event-based applications because event cameras and PPLNs are both designed to mimic biological neural principles. Due to the limited availability of high-quality data, event-based vision is an emerging field where modeling plays a more important role than data and the effects of PPLNs can be best demonstrated. However, we also present an evaluation on conventional frame-based tasks to demonstrate the generalizability. Meanwhile, we do not include Spiking Neural Network (SNN) baselines. While SNNs focus on the deployment onto neuromorphic chips, PPLNs are designed to be an alternative to general GPU-based temporal inference models.

4.1 Task I: motion deblurring

Task description . Event-enhanced motion deblurring is a popular research domain. Variants of the task include image-to-image and image-to-video deblurring. Our experiments focus on the highly challenging image-to-video problem, establishing a thorough competition against various state-ofthe-art approaches. Given a blurry image and its associated events during the exposure interval, our

Input

GT

EDI

eSL-Net

E-CIR

DeblurSR

Ours

KEDP TIS

AREA CLEAN

goal is to reverse the exposure process and reconstruct a sharp video describing the relative motion between the camera and the environment. The following experiments utilize the High Quality Frames (HQF) (Stoffregen et al., 2020) dataset and the preprocessing procedure documented by Song et al. We construct a PPLN based on the U-Net architecture (Ronneberger et al., 2015) by replacing all 2D convolution layers with PPLN nodes (Figure 2 (b)). The input to the PPLN is the concatenation of the blurry image and the event histograms, and the histograms are constructed following Zhu et al.. The output of the PPLN is sharp frames at 14 uniformly spaced timestamps. We utilize ADAM (Kingma &Ba, 2014) to train the network for 50 epochs using the L1 loss. We set the learning rate to 10 -3 and reduce the rate by half after 20 and 40 epochs, respectively. The number of line segments is n = 3 .

Evaluation metric . We use the Mean Squared Error, the Peak Signal-to-Noise Ratio, and the Structural Similarity Index Measure.

Results and discussions . As shown in Table 1 (Left), PPLN has a very strong performance in motion deblurring. PPLN improves DeblurSR (Song et al., 2024), the state-of-the-art event-based motion deblurring model at the time of paper submission, by 5.6% in MSE, 0.372 dB in PSNR, and 4.9% in SSIM. Importantly, we underscore that the PPLN is a generic network architecture and does not require task-specific modeling, whereas the baseline approaches utilize techniques that cater to the deblurring problem, such as dictionary learning (eSL-Net) (Wang et al., 2020), per-pixel polynomial approximation (E-CIR) (Song et al., 2022), and implicit neural representation (DeblurSR) (Song et al., 2024). The success of the PPLN reveals the strength of mimicking biological neural behaviors.

In Table 1 (Left), we also use regular convolution layers to construct the U-Net. We increase the number of convolutional channels in the regular U-Net to approximately match the number of trainable parameters in the PPLN (173M versus 192M). The last two rows suggest that the PPLN improves the U-Net by 54.5% in MSE, 6.45 dB in PSNR, and 43.3% in SSIM. Qualitatively, as shown in Figure 3, our method generates sharper and more realistic frames than the baseline approaches. The proposed PPLN offers vivid details around salient features. In the first row, our method reconstructs the dark patterns on the white background with sharp edges. In the second row, our method gives the best contrast between the white board and the black letters.

4.2 Task II: steering angle prediction

Task description . The DAVIS Driving Dataset released in 2020 (DDD20) (Hu et al., 2020) contains 51 hours of dashcam recordings with both neuromorphic events and conventional frames. Following Hu et al., we select 15 recordings during the day and 15 at night across the western United States, and train a deep network to regress steering angles from the dual-modal input.

MSE

↓

PSNR

↑

image-to-video baselines

EDI

0.336

17.822

eSL-Net eSL-Net++

E-CIR

DeblurSR

U-Net

Ours

0.452

0.385

0.207

0.161

0.334

0.152

14.938

16.870

21.713

23.912

17.834

24.284

SSIM

↑

0.515

0.282

0.363

0.609

0.694

0.508

0.728

As shown in Figure 2 (c), we build a PPLN upon the existing baseline, consisting of a convolutional head, a pooling layer, and a linear component. The convolutional head reduces the spatial dimension of the data and increases the number of channels. The pooling layer eliminates both spatial dimensions by averaging all the pixels on the reduced spatial grid. Finally, the linear component of the network maps the 64-dimensional feature to the scalar steering angle output. The network contains a total number of 463,425 (463K) parameters. The input frame and events contain 50 ms of historical data.

Our model (Figure 2 (d)) lowers the prediction frequency to every 500 ms. The network takes ten conventional frames and ten times as many events as input and predicts the steering angle at ten uniformly distributed timestamps. We replace the linear component with PPLN nodes and slightly increase the number of layers. To match the number of parameters in the original architecture, we shrink the convolutional head. This results in a network with 455,338 (455K) parameters. We utilize ADAM(Kingma & Ba, 2014) to train the network for 200 epochs using the L2 loss. The learning rate 10 -3 with a weight decay of 10 -4 . The number of line segments in the parameterization is n = 3 .

Evaluation metric . We use the Root Mean Square Error and the Explained VAriance. We train with five random seeds and report the mean and standard deviation.

Results and discussion . As shown in Table 1 (Right), our approach outperforms the baseline model in RMSE, with a 17.0% improvement at night, 35.6% improvement during the day, and 30.8% improvement overall. Similar enhancement is observed in EVA, with 1.9% improvement at night, 23.6% improvement during the day, and 9.6% improvement overall.

The input to our model contains ten times as much information as the baseline approach (Hu et al., 2020). To investigate whether the performance gain is simply a result of enriched input information, we present an additional comparison where the baseline is modified to have the same input and output dimensions as our model (Figure 2 (e)). From the second and the third rows in Table 1 (Right), we observe that PPLN improves the modified baseline by 5.7% in RMSE and 1.0% in EVA.

4.3 Task III: human pose estimation

Task description . The dynamic Vision Sensor Human Pose (DHP19) dataset (Calabrese et al., 2019) is collected by inviting human subjects into a cubic space and using event cameras in the four ceiling corners to record various body movements, such as walking, jumping, and walking. DHP19 contains recordings of 17 human subjects performing 33 different body movements. In addition to the events, DHP19 includes the 3D coordinates of 13 body joints. The goal is to predict 3D joint coordinates.

Calabrese et al. use two frontal cameras ('Cam 2' and 'Cam 3') in their experiments. The overall pipeline has two stages. First, they utilize a deep network to predict the 2D joint coordinates from the events in each view. After that, they project the 2D predictions into 3D using the calibration matrices.

The deep network used by Calabrese et al. is a fully convolutional network (Long et al., 2015) that contains 17 layers and 218,592 (219K) trainable parameters. The input events are represented as histograms (Zhu et al., 2019), and the output 2D joint coordinates are represented as heatmaps. After collecting 25,000 events from all four views, the algorithm assembles event histograms from two frontal cameras and discards the events from the other two cameras. The model updates the 3D joint coordinates if the 2D prediction confidence exceeds a threshold τ = 0 . 3 in both views.

In our experiment, the model makes an inference every 250,000 events. The network takes ten times as many events as input and predicts 2D joint coordinates at ten different timestamps. We modify the prediction network by introducing PPLN layers. The modified network contains 215,648 (216K) trainable parameters. We utilize RMSProp (Kingma & Ba, 2014) to train the network for 20 epochs using the L2 loss. The learning rate is 10 -3 in the first 10 epochs, 10 -4 from epochs 10 to 15, and 10 -5 from epochs 15 to 20. The number of line segments in the parameterization is n = 3 .

Evaluation metric . We use Mean Per Joint Position Error and report in 2D pixels and 3D millimeters.

Results and discussion . As shown in Table 2 (Left), PPLN estimations have a 2D MPJPE of 6.76 pixels in Cam 2, a 2D MPJPE of 6.51 pixels in Cam 3, and a 3D MPJPE of 73.05 mm. Compared to the original network used by Calabrese et al. , we achieve 9.7% improvement in Cam 2, 10.7% improvement in Cam 3, and 11.1% improvement in 3D. Similar to Section 4.2, simply enlarging the temporal horizon confuses the network and leads to performance degradation. Our method enhances the modified baseline by 43.2% in Cam 2, 44.7% in Cam 3, and 44.0% in 3D.

4.4 Ablation studies

In Section 3.4, we introduce the integral normalization operator, which theoretically stabilizes training by enriching the gradient information. We now use ablation studies to examine the effectiveness of integral normalization in practice. As shown in Table 3, the introduction of this operator improves motion deblurring quality by 42.4%, 4.563 dB, and 31.4% in MSE, PSNR, and SSIM respectively. For steering prediction, integral normalization improves the accuracy by 17.5% in MSE and 3.2% in EVA. The p-values from one-tailed t-tests are 0.007 and 0.014 for MSE and EVA, giving us reasonable confidence that integral normalization has enhanced accuracy. For human pose estimation, integral normalization improves the 2D MPJPE by approximately 0.2 pixels and the 3D MPJPE by 1.8 mm. The improvement suggests that integral normalization can effectively regularize the training in scenarios such as deblurring where the normalization target has a clear semantic meaning. We refer readers to the appendix for the impact of line segment number n and smoothing on performance.

4.5 Conventional frame-based vision

To demonstrate the generalizability of PPLNs, we remove the event input from the steering angle prediction model. Table 2 (Right) shows that PSNNs can still outperform the baselines in the conventional frame-based only setting. Importantly, we observe a significant performance drop by taking out the event input from both baseline approaches. However, the frame-based PPLN has a surprisingly similar prediction quality to the dual-modal PPLN. This result demonstrates that by simulating the biological behaviors, our model can effectively overcome the imperfections in the input data. Note that among the three tasks discussed above, only steering prediction allows inference from conventional frames alone. Human pose estimation takes events as the single-modal input, and image-to-video deblurring requires events to address the motion ambiguity.

5 Conclusion and Future Work

This paper presents Parametric Piecewise Linear Networks (PPLNs), a novel temporal learning architecture inspired by biological neural principles. The key idea is to represent the membrane potential as a parametric piecewise linear function with predictable coefficients. Experiments on various event-based vision applications, including steering prediction, human pose estimation, and motion deblurring, demonstrate that PPLNs outperform state-of-the-art models. In the future, we plan to use a recurrent prediction model to support a dynamic number of line segments. Another direction is to adopt more accurate modeling for the membrane potential function, including mechanisms such as the refractory period after each spike.

References

Abbott, L. F. Lapicque's introduction of the integrate-and-fire model neuron (1907). Brain research bulletin , 50(5-6):303-304, 1999.

Babacan, S. D., Molina, R., Do, M. N., and Katsaggelos, A. K. Bayesian blind deconvolution with general sparse image priors. In European conference on computer vision , pp. 341-355. Springer, 2012.

Bengio, Y., Simard, P., and Frasconi, P. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks , 5(2):157-166, 1994.

Binas, J., Neil, D., Liu, S.-C., and Delbruck, T. Ddd17: End-to-end davis driving dataset. arXiv preprint arXiv:1711.01458 , 2017.

Bottou, L. et al. Stochastic gradient learning in neural networks. Proceedings of Neuro-Nımes , 91(8): 12, 1991.

Calabrese, E., Taverni, G., Awai Easthope, C., Skriabine, S., Corradi, F., Longinotti, L., Eng, K., and Delbruck, T. Dhp19: Dynamic vision sensor 3d human pose dataset. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2019.

Davies, M., Srinivasa, N., Lin, T.-H., Chinya, G., Cao, Y., Choday, S. H., Dimou, G., Joshi, P., Imam, N., Jain, S., et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro , 38(1):82-99, 2018.

Eshraghian, J. K., Ward, M., Neftci, E., Wang, X., Lenz, G., Dwivedi, G., Bennamoun, M., Jeong, D. S., and Lu, W. D. Training spiking neural networks using lessons from deep learning. arXiv preprint arXiv:2109.12894 , 2021.

Fergus, R., Singh, B., Hertzmann, A., Roweis, S. T., and Freeman, W. T. Removing camera shake from a single photograph. In ACM SIGGRAPH 2006 Papers , pp. 787-794, 2006.

Fish, D., Brinicombe, A., Pike, E., and Walker, J. Blind deconvolution by means of the richardsonlucy algorithm. JOSA A , 12(1):58-65, 1995.

Gallego, G., Delbruck, T., Orchard, G. M., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A., Conradt, J., Daniilidis, K., and Scaramuzza, D. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , pp. 1-1, 2020. doi: 10.1109/TPAMI. 2020.3008413.

Hu, Y., Binas, J., Neil, D., Liu, S.-C., and Delbruck, T. Ddd20 end-to-end event camera driving dataset: Fusing frames and events with deep learning for improved steering prediction. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) , pp. 1-6. IEEE, 2020.

Joshi, N., Zitnick, C. L., Szeliski, R., and Kriegman, D. J. Image deblurring and denoising using color priors. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pp. 1550-1557. IEEE, 2009.

Kim, S. K., Park, S. R., and Paik, J. K. Simultaneous out-of-focus blur estimation and restoration for digital auto-focusing system. IEEE Transactions on Consumer Electronics , 44(3):1071-1075, 1998. doi: 10.1109/30.713236.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Krishnan, D. and Fergus, R. Fast image deconvolution using hyper-laplacian priors. Advances in neural information processing systems , 22:1033-1041, 2009.

Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., and Matas, J. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 8183-8192, 2018.

Kupyn, O., Martyniuk, T., Wu, J., and Wang, Z. Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 8878-8887, 2019.

Lacinová, L. Voltage-dependent calcium channels. General physiology and biophysics , 24:1-78, 2005.

Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 , 2019.

Lapicque, L. Recherches quantitatives sur l'excitation electrique des nerfs traitee comme une polarization. Journal de physiologie et de pathologie générale , 9:620-635, 1907.

Levin, A., Fergus, R., Durand, F., and Freeman, W. T. Image and depth from a conventional camera with a coded aperture. ACM transactions on graphics (TOG) , 26(3):70-es, 2007.

Lichtsteiner, P., Posch, C., and Delbruck, T. A 128 × 128 120 db 15 µ s latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits , 43(2):566-576, 2008. doi: 10.1109/JSSC.2007.914337.

Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljaˇ ci´ c, M., Hou, T. Y., and Tegmark, M. Kan: Kolmogorov-arnold networks, 2024.

Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 3431-3440, 2015.

Maass, W. Networks of spiking neurons: The third generation of neural network models. Neural Networks , 10(9):1659-1671, 1997. ISSN 0893-6080. doi: https://doi.org/10.1016/ S0893-6080(97)00011-7. URL https://www.sciencedirect.com/science/article/pii/ S0893608097000117 .

Maeda, S., Nakagawa, S., Suga, M., Yamashita, E., Oshima, A., Fujiyoshi, Y., and Tsukihara, T. Structure of the connexin 26 gap junction channel at 3.5 å resolution. Nature , 458(7238):597-602, 2009.

Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) , pp. 807-814, 2010.

Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., and Dai, Y. Bringing a blurry frame alive at high frame-rate with an event camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 6820-6829, 2019.

Pan, L., Hartley, R., Scheerlinck, C., Liu, M., Yu, X., and Dai, Y . High frame rate video reconstruction based on an event camera. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020.

Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In International conference on machine learning , pp. 1310-1318. PMLR, 2013.

Perrone, D. and Favaro, P. Total variation blind deconvolution: The devil is in the details. In 2014 IEEE Conference on Computer Vision and Pattern Recognition , pp. 2909-2916, 2014. doi: 10.1109/CVPR.2014.372.

Richardson, W. H. Bayesian-based iterative method of image restoration. JoSA , 62(1):55-59, 1972.

Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pp. 234-241. Springer, 2015.

Shan, Q., Jia, J., and Agarwala, A. High-quality motion deblurring from a single image. Acm transactions on graphics (tog) , 27(3):1-10, 2008.

Song, C., Huang, Q., and Bajaj, C. E-cir: Event-enhanced continuous intensity recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7803-7812, 2022.

Song, C., Bajaj, C., and Huang, Q. Deblursr: Event-based motion deblurring under the spiking representation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pp. 4900-4908, 2024.

Stoffregen, T., Scheerlinck, C., Scaramuzza, D., Drummond, T., Barnes, N., Kleeman, L., and Mahony, R. Reducing the sim-to-real gap for event cameras. In European Conference on Computer Vision , pp. 534-549. Springer, 2020.

Wang, B., He, J., Yu, L., Xia, G.-S., and Yang, W. Event enhanced high-quality image recovery. In European Conference on Computer Vision . Springer, 2020.

Xu, L. and Jia, J. Two-phase kernel estimation for robust motion deblurring. In European conference on computer vision , pp. 157-170. Springer, 2010.

Xu, L., Zheng, S., and Jia, J. Unnatural l0 sparse representation for natural image deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1107-1114, 2013.

Zheng, X., Liu, Y., Lu, Y., Hua, T., Pan, T., Zhang, W., Tao, D., and Wang, L. Deep learning for event-based vision: A comprehensive survey and benchmarks, 2023.

Zhou, Y., Zhu, Z., and Zhong, Z. Learning specialized activation functions with the piecewise linear unit. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 12095-12104, 2021.

Zhu, A. Z., Yuan, L., Chaney, K., and Daniilidis, K. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

A Appendix / supplemental material

A.1 Detailed results for task III

We report the 3D MPJPE in millimeters for all 33 different types of body movements evaluated on 5 testing subjects. As shown in Table 4 and Table 5, simply increasing the input horizon leads to an overall performance degradation. One possible explanation is that the network becomes confused about the temporal relationship among the input events. The modified baseline is not a time series model and can only infer the temporal relation from the input stacking order (the first temporal event slice is in the first channel, the second temporal event slice is in the second channel, etc.). On the other hand, PPLN keeps track of the timestamp of interest in every model layer. This allows PPLN to outperform the baseline in almost all cases by a significant margin. We point out that in Table 4 and Table 5, the asterisk for testing subject 2 in movement 29 indicates missing data in the DHP19 dataset (Calabrese et al., 2019).

A.2 Additional motion deblurring visualizations

Figure 4 presents seven additional examples for the motion deblurring results. We also encourage readers to watch the supplementary animations in the .gif format, which demonstrate the temporal smoothness of our results. Overall, the visual quality of PPLN reconstructions is on par with the state-of-the-art method (DeblurSR (Song et al., 2024)). Quantitatively, PPLN outperforms DeblurSR by a small margin (see paper body). We point out that while all baseline approaches are specifically

designed for motion deblurring, the proposed PPLN is a general framework that can be applied in various event-based vision tasks.

A.3 The neuromorphic mechanism

Some properties of the proposed model are slightly different from the standard practice in existing research in computer vision and machine learning (i.e., Spiking Neural Networks, SNNs (Maass, 1997)), even though both of them are inspired by biological neural principles (Lapicque, 1907).

First, existing SNNs focus on the interconnection between artificial neurons, with different layers communicating through binary signals. On the other hand, PPLN focuses on representing the membrane potentials, defined as the voltage differences between the interior and the exterior of the cell. The membrane potentials change with time according to the piecewise linear representation, whose parameters are predicted from the input.

Second, the signal transmitted between SNN layers carries a one if the neuron is excited and a zero if the neuron is not excited. In contrast, our work models the membrane potential using a real value. When a spike occurs, the neuron becomes excited, and the real-valued membrane potential function experiences a discontinuous gap.

Input

Ground

EDI

eSL-Net

E-CIR

DeblurSR

Ours

Truth

Third, existing SNNs utilize the linear decay mechanism. In other words, the membrane potential function only decreases in the absence of excitement. This contrasts our parametric mechanism where the slope of each line segment can be either positive or negative. We point out that in biological neurons, the resting potential is not the 'absolute' zero. When a biological neuron is at rest, there is naturally a voltage difference between the interior and the exterior of the cell. If the membrane potential exceeds the resting potential, the neuron will decrease the potential at an approximately linear rate. Similarly, if the membrane potential drops below the resting potential, the neuron will gradually increase the potential. The linear potential increment corresponds to a positive slope in the proposed mechanism.

Fourth, we do not explicitly enforce any line segment to be flat. Instead, we expect the network to implicitly learn when a line segment should have a zero slope from the training dataset. This allows more flexibility in the design.

A.4 Coefficient visualization

In figure 5, we plot a few randomly sampled piecewise linear functions predicted by the network. We observe uneven segment lengths and discontinuities (shown in orange) at boundaries. The piece-wise linear function has varying slopes, suggesting the model does not collapse to somewhere far away from the design. We also observe that with n = 3 , the network has the capability to represent functions with less than 3 linear segments.

A.5 Baseline implementation

Despite our best efforts, we fail to exactly reproduce the baseline performance reported by Hu et al. for steering angle prediction and Calabrese et al. for human pose estimation. According to Hu

1

0

1

1

0

1

1

0

1

1

0

1

1

0

1

1

0

1

1

0

1

1

0

1

et al., the baseline model has an RMSE of 4.13 ± 0.24 and an EVA of 0.881 ± 0.009. Our baseline implementation gives an RMSE of 4.55 ± 0.24 and an EVA of 0.845 ± 0.016. According to Calabrese et al., the baseline 3D MPJPE is 79.63 mm. Our implementation gives 82.17 mm. Similar confusions have been reported by various teams. We refer readers to the issue pages on their GitHub repositories for detailed discussions.

Compared to the relative improvement brought by PPLNs, the differences above are insignificant. We believe the inconsistency in steering angle prediction is the result of ambiguities in the data pruning procedure discussed in Section III-A of the original paper by Hu et al.. We provide step-by-step instructions for the data pruning procedure used when generating our reported figures in the code release. We believe training randomness and differences in optimizer hypermeters cause inconsistency in human pose estimation. Our code release includes the random seed, optimizer hypermeters, and the pre-trained weights for both the baseline model and the PPLN.

A.6 Data

In steering angle prediction, we use the DDD20 dataset (Hu et al., 2020) released under GNU Lesser General Public License v3.0. In human pose estimation, we use the DHP19 dataset (Calabrese et al., 2019). The DHP19 dataset is released under the Creative Commons Attribution-ShareAlike 4.0 International License, and the dataset utility scripts are released under MIT License. In motion deblurring, we use the HQF dataset (Stoffregen et al., 2020). At the time of paper writing, the HQF dataset is available for public download, but the licensing details are unclear. To the best of our knowledge, the datasets do not contain personally identifiable information or offensive content.

A.7 Computation resources

We train the model for steering angle prediction using one NVIDIA TITAN V GPU. We train the model for human pose estimation using one NVIDIA Tesla V100 SXM2 GPU. We train the model for motion deblurring using one NVIDIA Tesla V100 SXM2 GPU.

A.8 Limitations

In this paper, we define the number of line segments in the piecewise linear approximation to the membrane potential function as a hyperparameter n . The choice of this hyperparameter affects the representation capacity of the PPLN node, as well as the run-time complexity and convergence rate. At the moment, we are unable to provide a theoretical guideline that allows developers to choose its value based on the input properties and output requirements. Hyperparameter tuning is needed to balance the prediction quality and computational cost.

A.9 Proof of Theorem 3.1

Theorem 3.1 Consider an underlying n segment piecewise linear function parameterized by Θ ⋆ = { m ⋆ , b ⋆ , t ⋆ } . Let ( τ j , v j ) , j = 1 , . . . , m be m point samples, where v j = ˜ V Θ ⋆ ( τ j ) + ψ j , and ψ j is a small random noise.

Recall that the ground-truth membrane potential was defined by:

Without losing generality, we assume the following constraints:

The following sampling assumption will be put in order to prevent uncontrollable error due to noise.

Uniform Sampling Assumption: We assume there exists some constant c > 0 such that the n i samples ( τ i 1 , v i 1 ) , · · · , ( τ in i , v in i ) on the i th segment satisfy

and by applying vanilla gradient descent, Θ ⋆ can be recovered up to an error:

if the learning rate η = O ( h n ) , where parameter h > 0 measures the distance from the initialization to the ground truth:

Comment: The uniform sampling assumption is necessary. Imagine a scenario where a disproportionately large number of samples are clustered around ( τ, v ) = (0 , 0) , with only a single sample at ( τ, v ) = (1 , 0) . In such an extreme case, the linear regression model might yield a significantly steep slope due to noise, leading to substantial errors when predicting for τ = 1 . This example highlights the potential pitfalls of non-uniform sampling in affecting the reliability of regression outcomes. In addition, it can be shown that a set of uniformly random samples would provide constant c with high probability.

Proof. Let Θ r be the coefficients in the r th iteration of the optimization. We first consider the case where all samples ( τ j , v j ) 's are classified into the correct intervals ( i.e. , t i -1 (Θ r ) ≤ τ j < t i (Θ r ) implies t ⋆ i -1 ≤ τ j ≤ t ⋆ i ). With a sufficiently large temperature T , the effect of smoothing becomes minimal, and the estimated potential is:

Then under the objective:

for all t i -1 ≤ t < t i .

On the other hand, if t ⋆ i -1 ≤ τ j < t ⋆ i , the noisy sample is:

This gives the objective as:

To analyze Equation (19), we present a proposition to simplify notations.

Proposition A.1. The solution for real-valued variables a, b in the optimization problem:

in which u i 's and c i 's are known constants with the constraint that u i 's are not all the same, satisfies that:

where ζ ( u ) = 2 ∑ i<j | u i -u j | / ∑ i<j ( u i -u j ) 2

Proof of Proposition A.1 . Let L ( a, b ) = ∑ i ( au i + b + c i ) 2 and enforce ∂L ∂a = ∂L ∂b = 0 , we have:

Therefore, a ≤ ζ ( u ) max i | c i | and the remaining part regarding b can be obtained directly from Eq. 22.

The following corollary applies Proposition A.1 to the learning of the piecewise linear coefficients.

and

This gives:

satisfies:

0.0

0.2

0.4

0.6

0.8

1.0

time

0.0

0.5

1.0

1.5

2.0

value

Training Samples

Real Signal

Initial Model

0.0

0.2

0.4

0.6

0.8

1.0

time

0.5

1.0

1.5

2.0

value

Training Samples

Real Signal

Fitted Model

0.0

0.2

0.4

0.6

0.8

1.0

time

0.5

1.0

1.5

2.0

value

Training Samples

Real Signal

Fitted Model

Corollary A.2. The solution Θ ∞ of the optimization problem:

Proof of Corollary A.2 . Comparing Equation (19) and Equation (20), by setting u i = τ j and c j = -ψ j + o (exp( -T )) we have:

given that the sampling points τ j 's are fixed ( i.e. , ζ ( u ) is a constant). When T → ∞ , we obtain m i = m ⋆ i + O ( max j ( | ψ j | ) ) and b i = b ⋆ i + O ( max j ( | ψ j | ) ) , which implies:

since 0 ≤ τ ≤ 1 .

The rest of the proof shows that under our assumptions, the sample points will eventually be classified into correct intervals.

In the following, we use { m i ( T ) } , { b i ( T ) } , and { t i ( T ) } to denote the optimal solution for the objective L T (Θ) . Due to noisy samples and smoothing process, the optimal solution is different from Θ ⋆ .

Consider ( τ j , v j ) with t ∗ i -1 ≤ τ j < t ∗ i . Since | t i (Θ 0 ) -t ⋆ i | ≤ h ≤ | t ⋆ i -t ⋆ i -1 | , τ j must fall into one of the intervals [ t i -2 (Θ 0 ) , t i -1 (Θ 0 )] , [ t i -1 (Θ 0 ) , t i (Θ 0 )] , and [ t i (Θ 0 ) , t i +1 (Θ 0 )] .

In the following, we discuss the stationary point at a specific temperature T . In practice, we can stay at the temperature T until the gradient vanishes and then we move on to a higher temperature.

Setting ∂ L T ∂t i = 0 we have:

where ∆ j is defined by:

Thus, we have:

or equivalently,

Consider the case where t i is not currently at the correct location. Without loss of generality, we assume t i is too far to the right. This implies that there exists τ j satisfying t i -1 ≤ t ⋆ i ≤ τ j < t i . Suppose:

and

Since τ j +1 , . . . , τ j 1 all fall inside the correct interval, we can fully recover the corresponding segment and have ∆ q = O ( e -T ) for q = j +1 , . . . , j 1 . The left-hand side of Equation (27) is, therefore, O (exp( -T )) , which enforces the right-hand side of the equation to keep getting smaller. Since ( τ j , v j ) is not consistent with τ j 0 , . . . τ j -1 and ∆ 's are the errors of linear regression, we claim that ∆ r , r = j 0 , . . . , j -1 would never be close to zero. To make Equation (27) hold true, t i has to be decreased. This process will last until t i < τ j ( i.e. , the sample point ( τ j , v j ) is excluded from the wrong interval [ t i -1 , t i ] ).

In this way, we prove that eventually all sample points will be classified into correct intervals, thereby reducing to the base case that has been proved earlier.

A.10 Justifying integral normalization

Recall that when we construct a PPLN node without integral normalization or smoothing, the output ˜ V Θ ( t ) is given as:

With m = ( m 1 , . . . , m n ) T , b = ( b 1 , . . . , b n ) T , and t = ( t 0 , . . . , t n ) T , we have:

at the input timestamp t 0 ≤ t k ≤ t n .

We make two observations here. First, both ∂ ˜ V Θ ( t k ) ∂ m and ∂ ˜ V Θ ( t k ) ∂ b are very sparse vectors with only one non-zero entry corresponding to the specific segment t k belongs to. At training time, it is natural to assume the training timestamps are uniformly distributed across [ t 0 , t n ] . This means the amount of training data is proportional to the segment length. Long segments receive intensive training because there are a lot of available training examples. Short segments receive little training because there are only a limited number of training examples falling inside their ranges. The imbalance presents an instability risk because different segments are learning at different rates. Additionally, ∂ ˜ V Θ ( t k ) ∂ t is an all-zero vector. A gradient-based optimizer is therefore unable to adjust the segment lengths. This means the segment endpoints are hard-coded positions instead of trainable parameters.

To illustrate the second observation, consider the toy example of a 2-segment linear signal, as shown in Figure 6. The green real signal generates a set of orange training samples with measurement noises. Assuming no prior knowledge of the real signal besides that it has two segments, we initialize a 2-segment parametric model with zero slopes, zero intercepts, and equal segment lengths (Figure 6 (Left)). After training, the node converges to the model shown in Figure 6 (Middle). The second segment deviates from the real signal significantly because it attempts to accommodate the last several training samples in the first segment.

As discussed in the body, we use the integral normalization operator σ ( · ) to address the above challenges:

where V is a parameter that controls the mean of ˜ V Θ ( t ) when 0 ≤ t ≤ 1 .

This gives:

where all three gradient vectors contain rich gradient information in every element, encouraging a smooth and swift convergence. Figure 6 (Right) illustrates how the model accurately approximates both linear segments in the toy example after incorporating integral normalization.

A.11 Ablation studies: number of line segments

As shown in Table 6, we fail to observe noticeable performance improvement when increasing the number of line segments n in the formulation. Therefore, we decide to use n = 3 in all the experiments discussed in the body of this paper for better efficiency.

A.12 Ablation studies: smoothing

As shown in Table 7, the smoothing operator has an insignificant impact on the prediction quality. However, we point out several facts related to the smoothing operator as a guideline for potential future applications. First, the smoothing operator allows us to have a simple model (Theorem 1) with analytical analyzable properties. Second, the smoothing operator does not introduce any additional trainable parameters. Third, when the size of the network is relatively small ( i.e. , steering prediction and human pose estimation), our results suggest that using PPLNs without smoothing is slightly better. Finally, when the size of the network is large ( i.e. , motion deblurring), smoothing introduces a small performance improvement.

A.13 Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Regarding the environmental impact, our experiments are conducted on an internal GPU cluster, and the total emission is estimated to be 81.12 kgCO 2 eq, equivalent to 328 km driven by an average car. This emission estimation is conducted using the Machine Learning Impact calculator presented by Lacoste et al.. To mitigate repetitive labor and negative environmental impact in future research, we have released our open-source implementation together with trained network weights after the anonymous period.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper's main contribution is the proposal and evaluation of the parametric piecewise linear networks. In Section 4, our experiment emphasizes event-based vision applications because event cameras and PPLNs are both designed to mimic biological neural principles. Due to the limited availability of high-quality data, event-based vision is an emerging field where modeling plays a more important role than data and the effects of PPLNs can be best demonstrated. However, we also present an evaluation on conventional frame-based tasks to demonstrate the generalizability.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Discussed in Section A.8.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Discussed in Section A.9.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have released our source code on GitHub.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have released our source code on GitHub.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Discussed in Section 4 with full details in the source code on GitHub.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the mean and standard deviation of evaluation metrics for the steering prediction task.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Discussed in Section A.7.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully read the NeurIPS Code of Ethics and confirm that our research conforms with the Code.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Discussed in Section A.13.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Discussed in Section A.6.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We have released our source code on GitHub.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes Parametric Piecewise Linear Networks (PPLNs) for event-based temporal modeling, which emulate biological principles by representing membrane potentials as parametric mappings. The authors demonstrate how a straightforward modification enables standard multi-layer perceptrons and convolution operators to accommodate this parameterization of membrane potential. Experimental results showcase that the proposed PPLNs attain state-of-the-art performance in typical event-based vision tasks, including steering prediction, human pose estimation, and motion deblurring.

STRENGTHS:
i) The topic of bio-inspired parametric piecewise linear networks for temporal modeling is very novel and attractive.

ii) The authors sufficient experiments in the main paper and the supplemental material to help reader better understand the main contributions of this work.

iii) The writing is straightforward, clear, and easy to understand.

WEAKNESSES:
i) Could you give the time complexity of PPLNs, or the inference time on the three downstream tasks, such as steering prediction, human pose estimation, and motion deblurring?

ii) Which hyperparameters have the greatest influence on LLPNs? Please explain the analysis from an experimental perspective.

QUESTIONS:
Please see the weakness and response each comment.

LIMITATIONS:
There is a lack of analysis of the computational complexity of PPLNs, including experimental tests on the inference time of three downstream event-based vision tasks.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. The paper presents Parametric Piecewise Linear Networks (PPLNs), a novel neural network architecture inspired by neuromorphic principles for temporal vision inference.\n\n2. The innovative approach of PPLNs lies in modeling the membrane potential of artificial neurons as parametric piecewise linear functions with learnable coefficients, echoing the design of Kolmogorov-Arnold Networks (KANs) but with input-dependent adaptability.\n\n3. Experimental results demonstrate PPLNs' state-of-the-art performance in event-based vision applications such as steering prediction, human pose estimation, and motion deblurring, outperforming existing methods with improved accuracy and efficiency.\n\nSTRENGTHS:\n1. PPLNs are inspired by neuromorphic principles, mimicking the behavior of biological neurons, which adds a layer of biological plausibility to their computational model.\n\n2. The authors provide a thorough evaluation, including ablation studies, which helps in understanding the contribution of different components of PPLNs to the overall performance. The experiments showcase PPLNs achieving state-of-the-art results across various vision tasks, highlighting the effectiveness of the model in processing event-based data. The paper also evaluates PPLNs on conventional frame-based tasks, showing the model's generalizability beyond just event-based applications.\n\nWEAKNESSES:\n1. The authors did not compare with some of the latest event-based deblurring algorithms, such as REFID[1]. \n\n2. In addition, the authors should provide a direct comparative experiment using a KAN-based backbone to further demonstrate the superiority of the proposed solution.\n\n[1] Sun, Lei, et al. \"Event-based frame interpolation with ad-hoc deblurring.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nQUESTIONS:\n1. The piecewise linear nature of this paper is akin to an input-dependent activation function. An SNN using multi-layer LIF neurons with learnable parameters also seems to achieve a similar level of biological plausibility. Could the authors provide further explanation from this perspective?\n\n2. Based on Figure 2d, I am still unclear whether the input t for PPLN is an external query (you can choose your desire timestamp t) or an inherent timestamp of the event. If it is the former, does this mean that all neurons query with the same input t? If it is the latter, how is this implemented? Given that there are many input events with numerous timestamps, how does the method handle this?\n\nA good response to the Weaknesses and Questions will improve my initial rating.\n\nLIMITATIONS:\nThe authors discussed the environmental impact of the training phase for their proposed solution.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Parametric Piecewise Linear Networks (PPLNs), a novel approach to temporal modeling inspired by biological neural principles. PPLNs represent neuron membrane potentials as piecewise linear functions with learnable coefficients, aiming to allow for explicit temporal modeling. The authors evaluate PPLNs on three event-based vision tasks: motion deblurring, steering prediction, and human pose estimation, claiming significant improvements over baseline models and some state-of-the-art methods. The paper also attempts to show that PPLNs can generalize to conventional frame-based versions of these tasks.\n\nSTRENGTHS:\n1.\tThe PPLN approach presents an interesting attempt to model temporal dynamics in neural networks, drawing inspiration from biological neural systems.\n2.\tThe paper evaluates the proposed method on multiple event-based vision tasks, providing a reasonably broad assessment of its performance.\n\nWEAKNESSES:\n1.\tThe paper lacks a thorough analysis of the computational costs associated with PPLNs. Without this information, it's difficult to assess the practical viability of the approach, especially in comparison to existing methods.\n2.\tWhile the authors claim biological inspiration, the paper does not adequately explore how closely PPLNs actually mimic neuronal behavior. The connection to biological systems seems superficial and not well-substantiated.\n3.\tThe paper's claims about PPLNs' applicability to conventional frame-based tasks are not adequately supported. The limited experiments in this domain do not provide convincing evidence of the method's broader applicability beyond event-based vision.\n4.\tThe paper fails to provide a detailed analysis of how sensitive PPLNs are to hyperparameter choices, such as the number of line segments in the parameterization. This omission raises questions about the robustness and reproducibility of the results.\n\nQUESTIONS:\n1.\tWhat is the exact computational overhead of PPLNs compared to traditional neural network layers?\n2.\tCan you provide a detailed sensitivity analysis for the key hyperparameters of PPLNs?\n3.\tCan you provide more evidence to support the biological plausibility of PPLNs beyond the initial inspiration?\n\nLIMITATIONS:\nThe authors have addressed some limitations of their work, particularly in the supplementary material. However, a more comprehensive discussion of potential negative societal impacts and failure cases would be beneficial. The paper could also benefit from a more detailed analysis of the computational requirements and scalability of PPLNs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}