# An inverse problem resolution for 3D particle tracking using event-based vision

Charlotte Vienney, Cheng Wang, Pierre Borgnat, Mickaël Bourgoin, Nelly Pustelnik

► To cite this version:

Charlotte Vienney, Cheng Wang, Pierre Borgnat, Mickaël Bourgoin, Nelly Pustelnik. An inverse problem resolution for 3D particle tracking using event-based vision. 2026. hal-05525842

HAL Id: hal-05525842

https://hal.science/hal-05525842v1

Preprint submitted on 24 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# An inverse problem resolution for 3D particle tracking using event-based vision

Charlotte Vienney, Cheng Wang, Pierre Borgnat, Mickaël Bourgoin, Nelly Pustelnik

CNRS, ENS de Lyon, Laboratoire de Physique, UMR 5672, Lyon, France

{charlotte.vienney, cheng.wang, pierre.borgnat, mickael.bourgoin, nelly.pustelnik}@ens-lyon.fr

***Abstract***—Particle tracking in fluid systems is fundamental for the characterization of complex flow dynamics, but current methods are constrained by temporal and spatial resolution of conventional frame-based cameras. This paper proposes the application of the emerging event-based technology to reconstruct a 3D trajectory from 2D projections acquired by event-based cameras. Building on the pinhole camera model, we reformulate the well-established relation between the 2D projections and 3D estimate. This allows us to define a convex data-fidelity term, and a reconstruction procedure based on the minimization of a penalized least-squares functional. Experimental results show that this approach enables the reconstruction of a 3D trajectory with high accuracy. This method opens new perspectives for the analysis of complex flows in fluid systems.

***Index Terms***—Event-based camera, particle tracking, 3D reconstruction, inverse problem, gradient descent, fluid mechanics

## I. INTRODUCTION

Event-based cameras, also called neuromorphic cameras, inspired by the human visual system, have recently emerged as a paradigm shift in computer vision by capturing asynchronous, event-driven data instead of traditional frame-based images. These sensors respond only to local changes in brightness, offering an increasing temporal and spatial resolution compared to conventional cameras. However, the nature of event data presents unique challenges for signal processing, including noise sensitivity, spatiotemporal sparsity, and the absence of standardized processing pipelines.

The emergence of event-based cameras, see [1], reflects a growing interest in this technology, which offers significant advantages over conventional frame-based imaging systems. The numerous advantages and remaining challenges are reviewed in the survey [2]. Among the key advantages is the high temporal resolution, as events are detected with microsecond precision, enabling the capture of extremely fast motions and effectively eliminating motion blur typically observed in frame-based cameras. Event-based cameras have demonstrated strong potential across a broad spectrum of applications, including high-speed motion analysis [3], object detection and tracking [4], image and video reconstruction [5], with recent advances extending their use to depth estimation and 3D reconstruction [6].

For 3D reconstruction, most studies focus on recovering the full 3D scene from 2D views [7]. On the one hand, this

process can be solved considering model-based procedure. For instance, an optimization framework has been introduced in [8], where the authors introduced a variational method to estimate the 3D fluid velocity vector field by combining data fidelity with physical and geometric constraints. On the other hand, data-based methods have been considered for processing event-based data (see for instance the survey on deep learning methods for event-based vision [9]), emphasizing that event representation is the central challenge. In [10], hybrid strategies, derived from the ISTA algorithm with temporal constraints, jointly exploit event-based measurements and intensity image information in order to perform image recovery.

**Particle tracking with event-based cameras** – For the characterization of complex flow dynamics, it is necessary to estimate 3D particle velocity and acceleration statistics. This requires long and high-quality trajectories. It motivates the use of event-based cameras, as they offer high temporal resolution and precise motion information, while significantly reducing hardware and data storage costs compared to conventional high-speed cameras. In this context, existing algorithms also rely on 3D grid-based image recovery, whether from traditional [11] or event-based acquisition [3]. However, in the context of event-based cameras, a key question arises: How can we design an algorithm that directly processes raw event data, bypassing the need for 3D grid-based reconstruction, to fully benefit the potential of this event-based camera?

**Contribution** – In this paper, we address the reconstruction of a 3D trajectory of a particle in a turbulent flow from 2D projections captured by multiple event-based cameras. Therefore, unlike approaches relying on reconstructed image frames, our study is conducted directly on raw event-based data, in order to only recover the quantity of interest. This paradigm requires rethinking and redesigning conventional algorithms to effectively exploit the sparse and asynchronous event stream.

The remainder of this paper is organized as follows. Section II presents the event-based camera encoding and the pinhole camera model. Section III reformulates the forward model and the associated convex data-fidelity term. Section IV motivates the introduction of a regularization term, presents two regularization matrices, and details the algorithm. Experimental results on both simulated and real raw event-based data are reported in Section V. Section VI concludes the paper.

<footnote>This work was supported by state funding managed by the National Research Agency under the France 2030 program, with reference ANR-24-RRII-0001.</footnote>
## II. PHYSICAL MODEL

### A. Encoding characteristics of event-based cameras

Event-based cameras respond to changes in brightness intensity. When the brightness at a pixel exceeds or falls below a threshold, an event is triggered [2]. This differs from frame-based cameras whose pixels are all read according to a fixed sampling period $\Delta t$, as illustrated in Fig. 1. This enables higher effective event acquisition rates while generating lower data throughput.

Each event, acquired by a camera c, is defined by 2D pixel coordinates $[x_c \ y_c]^\top \in \mathbb{R}^2$, a polarity $p_c \in \{-1, 1\}$ indicating the sign of the brightness change, and a timestamp $t_c \in \mathbb{R}^+$.

<figure><img src="image_2.png" /><figcaption>Fig. 1. The cameras observe a red ball following the green dashed trajectory. (a) Frame-based camera outputs, with fixed sampling period $\Delta t$, corresponding to the frame acquisition frequency of the camera. (b) Event-based cameras outputs, with events of polarity $p = 1$ in blue and $p = -1$ in red.</figcaption></figure>

### B. The pinhole camera model

For 3D reconstruction, general geometric methods have been exploited [6]. Triangulation is a fundamental technique in stereovision for recovering a 3D point from its projections in two or more camera views [12]. The spatial location of the point is determined by intersecting rays extending from each camera through the corresponding image points. Epipolar geometry simplifies this process by restricting the search for a corresponding point to the epipolar line in the second image [13].

The pinhole camera model [14] is a fundamental geometric framework that describes how a camera c forms a 2D image of a 3D scene through perspective projections.

Let $u = [u \ v \ w]^\top$ be a 3D point in the world reference system. u is related to a point $u_c = [u_c \ v_c \ w_c]^\top$ in the 3D camera c reference system by the linear transformation:

$$u_c = R_c^\top (u - \tau_c), \qquad (1)$$

where $R_c \in \mathbb{R}^{3 \times 3}$ and $\tau_c \in \mathbb{R}^{3 \times 1}$ are, respectively, the rotation matrix and the translation vector, obtained through the calibration process of camera c.

The point $u_c$ is projected onto the 2D camera c reference system at point $x_c = [x_c \ y_c]^\top$ according to the geometry of similar triangles (in blue in Fig. 2), leading to the following projection equations:

$$\begin{cases} w_c x_c - f u_c = 0, \\ w_c y_c - f v_c = 0, \end{cases} \qquad (2)$$

<figure><img src="image_3.png" /><figcaption>Fig. 2. Pinhole camera model for a 3D point u, transformed into a 3D point $u_c$ in the camera c reference system, and projected onto its corresponding 2D point $x_c$ on the image plane of camera c of center O.</figcaption></figure>

where f denotes the focal length; without loss of generality, it is set to $f = 1$ in the following.

### III. REFORMULATION OF THE FORWARD MODEL

Starting from the well-established relation between the 2D projection and 3D estimate, a convex data-fidelity term is built by reformulating the original forward model.

For each camera c and event location $x_c$, equation (2) can be reformulated as:

$$x_c \odot T_1 u_c - T_2 u_c = 0, \qquad (3)$$

where

$$T_1 = \begin{bmatrix} 0 & 0 & 1 \\ 0 & 0 & 1 \end{bmatrix} \quad \text{and} \quad T_2 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}$$

and $\odot$ denotes the Hadamard product.

The 3D position of a point in the world reference system can be estimated from its 2D projections recorded by C event-based cameras by solving:

$$\hat{u} \in \operatorname{Argmin}_{u \in \mathbb{R}^3} F(u) := \sum_{c=1}^{C} F_c(u; x_c), \qquad (4)$$

where

$$F_c(u; x_c) = \frac{1}{2} \|x_c \odot T_1 R_c^\top (u - \tau_c) - T_2 R_c^\top (u - \tau_c)\|^2. \qquad (5)$$

This can be simply reformulated as

$$\hat{u} \in \operatorname{Argmin}_{u \in \mathbb{R}^3} F(u) := \frac{1}{2} \|\mathbf{A}u - \mathbf{z}\|_2^2, \qquad (6)$$

where

$$\mathbf{A} = [\mathbf{A}_1^\top \quad \dots \quad \mathbf{A}_C^\top]^\top \in \mathbb{R}^{2C \times 3},$$

and

$$\mathbf{z} = [\mathbf{z}_1^\top \quad \dots \quad \mathbf{z}_C^\top]^\top \in \mathbb{R}^{2C},$$

and for all cameras $c \in \{1, \dots, C\}$:

$$\begin{cases} A_c = x_c \odot T_1 R_c^\top - T_2 R_c^\top, \\ z_c = x_c \odot T_1 R_c^\top \tau_c - T_2 R_c^\top \tau_c. \end{cases}$$

In the previous equation, there is a slight abuse of notation regarding the Hadamard product, which is expressed as $x_c \odot$
$T_1 R_c^\top$ while the correct notation would be $[x_c \ x_c \ x_c] \odot T_1 R_c^\top$.

By standard computation, being in a over-determined configuration, the closed-form expression of (6) is obtained by

$$ \hat{\mathbf{u}} = (\mathbf{A}^\top \mathbf{A})^{-1} \mathbf{A}^\top \mathbf{z}. \qquad (7) $$

## IV. REGULARIZATION AND ALGORITHMIC STRATEGY

### A. Need of a regularization term

The optimal estimation (7) is obtained under the ideal assumption of noise-free measurements. In pratice, event-based cameras are affected by various noise sources: photon noise produces spurious events unrelated to motion [15], thermal noise generates random scene-independent events [16], pixel mismatch arises from variations in brightness-change thresholds [17], and temporal noise pertubs event timestamps, reducing timing accuracy [16].

This motivates the addition of a penalization term in (4) in order to regularize the solution.

Equation (4) is formulated for the recovery of a single point. We thus extend it to the estimation of a trajectory of N points, necessitating the introduction of extended notations. 3D coordinates are noted as $u_{n,\bullet} = (u_{n,m})_{m=1,2,3}$ in the world reference system (i.e., $u_{n,1} = u_n$, $u_{n,2} = v_n$, and $u_{n,3} = w_n$). These N points will be a uniformly sampled trajectory at times $t_n$ (ordered increasingly). The concatenation of these N points are the trajectory: $\mathbf{u} = (u_{n,m})_{1\le n\le N, 1\le m\le 3}$. Finally, $u_{\bullet,m}$ denotes the concatenation of the m-th coordinates of the N points.

Henceforth, the penalized problem is written as:

$$ \hat{\mathbf{u}} \in \underset{\mathbf{u} \in \mathbb{R}^{N \times 3}}{\operatorname{Argmin}} \Psi(\mathbf{u}), \qquad (8) $$

with

$$ \Psi(\mathbf{u}) := \sum_{c=1}^{C} \sum_{n=1}^{N} F_c(u_{n,\bullet}; x_{c,n}) + \frac{\lambda}{2} \sum_{c=1}^{C} \sum_{m=1}^{3} \|P_c u_{\bullet,m}\|^2, \quad (9) $$

where $P_c \in \mathbb{R}^{\tilde{N} \times N}$ is a penalization operator (that will be discussed in the next section) with $\tilde{N} \le N$, and $\lambda \in \mathbb{R}^+$ is a penalization parameter. For each camera, the $x_{c,n}$ in the data terms $F_c(u_{n,\bullet}; x_{c,n})$ corresponds to the event that is the closest to the time $t_n$.

In this preliminary study, standard smooth penalization is considered to simplify the optimization process and prevent irregular behaviour in the resulting solution.

### B. Choice of the penalization operator $P_c$

**Temporal finite-difference penalization matrix** – A first possible penalization enforces a spatial linkage between each point of the 3D trajectory and its nearest neighbor in the previous time step. It results in the following penalization matrix:

$$ P_c = \begin{bmatrix} -1 & 1 & 0 & \cdots & 0 \\ 0 & -1 & 1 & \cdots & \vdots \\ \vdots & \vdots & \vdots & \ddots & 0 \\ 0 & \cdots & 0 & -1 & 1 \end{bmatrix}. \qquad (10) $$

Here $\tilde{N} = N-1$. For this penalization, the same matrix is used for all event-based cameras and therefore the camera index c could be omitted.

**Graph-based Laplacian penalization matrix** – An alternative penalization term is defined using graphs constructed from the 2D projections. An assumption is that, when events are close in their 2D+t space, the corresponding $u_{n,\bullet}$ should also be close along the 3D trajectory.

For camera c, a graph $G_c$ is built with nodes representing events, and edges connecting each node to its $\kappa$ nearest neighbors. Edges exceeding a distance threshold are removed to maintain compactness, and disconnected components are reconnected via shortest inter-component distances. The resulting graph is then used to compute the penalization matrix, i.e., the graph Laplacian $P_c = D_c - W_c$, where $W_c \in \mathbb{R}^{N \times N}$ denotes the adjacency matrix of $G_c$, with $W_{c,ij} = 1$ if nodes i and j are connected and $W_{c,ij} = 0$ otherwise. $D_c \in \mathbb{R}^{N \times N}$ is the degree matrix, with $D_{c,ii} = \sum_{j=1}^{N} W_{c,ij}$.

### C. Algorithm

In order to solve (8), the Nesterov accelerated gradient algorithm is considered [18], with iterations indexed by k. The update equations are then given by:

$$ \begin{gathered} \mathbf{u}^{[k+1]} = \mathbf{y}^{[k]} - \tau \nabla \Psi(\mathbf{y}^{[k]}), \\ \alpha_{k+1} = \frac{1 + \sqrt{1 + 4\alpha_k^2}}{2}, \end{gathered} \qquad (11) $$

$$ \mathbf{y}^{[k+1]} = \mathbf{u}^{[k+1]} + \frac{\alpha_k - 1}{\alpha_{k+1}} (\mathbf{u}^{[k+1]} - \mathbf{u}^{[k]}). $$

## V. EXPERIMENTAL RESULTS USING TWO EVENT-BASED CAMERAS

### A. Experimental setup

Experiments were conducted with Prophesee EVK4 event-based cameras, equipped with the IMX636 sensor (1280 × 720 pixels, pixel size of 4.86 × 4.86 µm).

A turbulent flow (with Reynolds number $Re \approx 330$) was generated in a square von Kármán setup (15 cm × 15 cm × 20 cm), made of acrylic glass, with two counter-rotating disks (diameter 14.2 cm) [19]. Dilute polyethylene particles (1.0 g cm⁻³) were illuminated by LED blacklights. Eight event-based cameras observe the scene, with centers 35 cm from the setup center: four ($c_1 - c_4$) from one direction and four ($c_5 - c_8$), approximately perpendicular.

### B. Results on simulated data

A 3D trajectory of N = 100 points is generated, and its 2D projections from cameras $c_1$ and $c_8$ are obtained using the pinhole camera model, with additive white Gaussian noise $\varepsilon \sim N(0, \sigma^2)$ with varying $\sigma$.

Reconstruction performance is evaluated using the Root Mean Squared Error (RMSE), which quantitatively measures the discrepancy between the estimated trajectory $\hat{\mathbf{u}}$ and the ground-truth trajectory $\bar{\mathbf{u}}$.

**Closed-form solution** – In the ideal scenario where $\sigma = 0$, see Fig. 3, with noise-free projections, the estimated trajectory
<figure><img src="image_4.png" /><figcaption>Fig. 3. Estimation of a 3D particle trajectory, from 2D simulated projections from cameras $c_1$ and $c_8$, for different estimation strategies, and for different values of $\sigma$ and $\lambda$.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4. RMSE versus $\lambda$ for temporal finite-difference penalization, and Laplacian penalization with $\sigma = 0.01$ (left) and $\sigma = 0.05$ (right). In both plots, the dashed line indicates the RMSE of the closed-form estimation in a noise-free scenario ($10^{-14}$).</figcaption></figure>

almost perfectly matches the ground-truth trajectory. Fig. 4 reports a very low RMSE value in such case.

When the projection noise increases, $\sigma > 0$, see Fig. 3 for $\sigma = 0.01$ and $\sigma = 0.05$, the estimated trajectory deviates further from the ground-truth trajectory. As a result, the closed-form solution no longer proposes a satisfying solution. In such cases, the problem (8) must be solved with a non-zero $\lambda$.

**Penalized least-square solution** – The estimated 3D trajectory is computed using the Nesterov accelerated gradient algorithm for $K = 10^5$ iterations and different values of $\lambda$. When dealing with noisy projections, adding a penalization term improves the reconstruction, see Fig. 3 for $\sigma = 0.01$ and $\sigma = 0.05$, and Fig. 4. The results are presented for values of $\lambda$ larger than the optimal value in terms of RMSE (cf. Fig. 3), as they provide visually better recovery. It can be observed that the temporal finite-difference penalization leads to better recovery performance in this context.

### C. Results on real data

The asynchronous nature of event-based cameras is handled by temporally aligning all 2D projections using interpola-

tion, such that the same number of events is considered at a common timestamp. Each projection, having 345,524 events, includes events from both polarities. For computational efficiency, one event every hundred is retained. Nesterov algorithm is run for $K = 400$ iterations.

Results of the 3D estimation for the temporal finite-difference, and the Laplacian penalizations are shown in Fig. 5. This figure illustrates that increasing the parameter $\lambda$ smooths the reconstructed trajectory for both regularizations, with a stronger effect observed for the temporal finite-difference case.

## VI. CONCLUSIONS AND PERSPECTIVES

We proposed an inverse problem framework for recovering a 3D trajectory from 2D event-based projections. Using the pinhole camera model with known rotation matrices and translation vectors, we provided an interpretable recovery framework for multi-view 3D trajectory recovery in fluid mechanics.

In a future work, we aim to deeper investigate the choice of the regularization term, possibly considering a Kalman filter framework, in order to perform both 3D recovery and estimation of the dynamic (velocity and acceleration). Future work will also address the challenge of handling the simultaneous observations of multiple particles, which further complicates data pairing and state estimation.

While the pinhole model is adequate when both event-based cameras and scenes are in air, when the scene is underwater and the cameras remain in air, optical phenomena such as refraction and diffraction introduce distortions that are not described by this model. Extending the inverse formulation to incorporate more accurate underwater imaging models, while preserving known extrinsic calibration, is another key direction for future work.
<figure><img src="image_6.png" /><figcaption>Fig. 5. Estimation of a 3D particle trajectory, from 2D raw event-based projections from cameras $c_1$ and $c_8$, using the Nesterov accelerated gradient algorithm for the two penalization strategies and for increasing values of $\lambda$.</figcaption></figure>

## REFERENCES

[1] B. Chakravarthi, A. A. Verma, K. Daniilidis, C. Fermuller, and Y. Yang, "Recent event camera innovations: A survey," in *European Conference on Computer Vision*, Milano, Italy, 2024, pp. 342–376.

[2] G. Gallego et al., "Event-based vision: A survey," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 42, no. 6, pp. 154–180, Jan. 2020.

[3] J. Howell, T. C. Hammarton, Y. Altmann, and M. Jimenez, "High-speed particle detection and tracking in microfluidic devices using event-based sensing," *Lab Chip*, vol. 20, pp. 3024–3035, 2020.

[4] Z. El Shair and S. A. Rawashdeh, "High-temporal-resolution object detection and tracking using images and events," *Journal of Imaging*, vol. 8, no. 8, 2022.

[5] L. Pan, R. Hartley, C. Scheerlinck, M. Liu, X. Yu, and Y. Dai, "High frame rate video reconstruction based on an event camera," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 44, no. 5, pp. 2519–2533, Nov. 2020.

[6] H. Rebecq, G. Gallego, E. Mueggler, and D. Scaramuzza, "EMVS: Event-based multi-view stereo - 3d reconstruction with an event camera in real-time," *Int. J. Comput. Vis.*, vol. 126, pp. 1394–1414, 2018.

[7] C. e. a. Xu, "A survey of 3d reconstruction with event cameras," 2025. arXiv: 2505.08438.

[8] Y. Wang, R. Idoughi, and W. Heidrich, "Stereo event-based particle tracking velocimetry for 3d fluid flow reconstruction," in *Springer, Champ*. 2020.

[9] X. Zheng et al., "Deep learning for event-based vision: A comprehensive survey and benchmarks," arXiv:2302.08890, 2023.

[10] S. Liu, R. Alexandru, and P. L. Dragotti, "Convolutional ista network with temporal consistency constraints for video reconstruction from event cameras," in *IEEE Int. Conf. Acoust., Speech Signal Process.*, 2022, pp. 1935–1939.

[11] M. Bourgoin and S. Huisman, "Using ray-traversal for 3D particle matching in the context of particle tracking velocimetry in fluid mechanics," *Review of scientific instruments*, vol. 91, 2020.

[12] H. G. Maas, A. Gruen, and D. Papantoniou, "Particle tracking velocimetry in three-dimensional flows," *Experiments in Fluids*, vol. 15, pp. 133–146, 1993.

[13] J. Carneiro, S.-H. Ieng, C. Posch, and R. Benosman, "Event-based 3d reconstruction from neuromorphic retinas," *Neural Networks*, vol. 45, pp. 27–38, 2013.

[14] R. Tsai, "A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses," *IEEE Trans. Robot. Autom.*, vol. 3, no. 4, pp. 323–344, Aug. 1987.

[15] R. Cao, D. Galor, A. Kohli, J. L. Yates, and L. Waller, "Noise2image: Noise-enabled static scene recovery for event cameras," *Optica*, vol. 12, pp. 46–58, Jan. 2025.

[16] Y. Nozaki and T. Delbruck, "Temperature and parasitic photocurrent effects in dynamic vision sensors," *IEEE Trans. Educ.*, vol. 64, no. 8, pp. 3239–3245, Aug. 2017.

[17] Z. Wang, Y. Ng, P. van Goor, and R. Mahony, "Event camera calibration of per-pixel biased contrast threshold," arXiv:2012.09378, 2020.

[18] Y. Nesterov, "A method of solving a convex programming problem with convergence rate $O(1/k^2)$," Dokl. Akad. Nauk SSSR, vol. 269, pp. 543–547, 1983.

[19] P. D. Huck, N. Machicoane, and R. Volk, "Lagrangian acceleration timescales in anisotropic turbulence," *Phys. Rev. Fluids*, vol. 4, 2019.