IncEventGS: Pose-Free Gaussian Splatting from a Single Event Camera

## Abstract

Abstract Implicit neural representation and explicit 3D Gaussian Splatting (3D-GS) for novel view synthesis have achieved remarkable progress with frame-based camera ( e.g . RGB and RGB-D cameras) recently. Compared to frame-based camera, a novel type of bio-inspired visual sensor, i.e . event camera, has demonstrated advantages in high temporal resolution, high dynamic range, low power consumption and low latency. Due to its unique asynchronous and irregular data capturing process, limited work has been proposed to apply neural representation or 3D Gaussian splatting for an event camera.
In this work, we present IncEventGS , an incremental 3D Gaussian Splatting reconstruction algorithm with a single event camera.
To recover the 3D scene representation incrementally, we exploit the tracking and mapping paradigm of conventional SLAM pipelines for IncEventGS .
Given the incoming event stream, the tracker firstly estimates an initial camera motion based on prior reconstructed 3D-GS scene representation. The mapper then jointly refines both the 3D scene representation and camera motion based on the previously estimated motion trajectory from the tracker.
The experimental results demonstrate that IncEventGS delivers superior performance
compared to prior NeRF-based methods and other related baselines, even we do not have the ground-truth camera poses. Furthermore, our method can also deliver better performance compared to state-of-the-art event visual odometry methods in terms of camera motion estimation. Code is publicly available at: https://github.com/wu-cvgl/IncEventGS .

## 1 Introduction

Acquiring accurate 3D scene representations from 2D images has long been a challenging problem in computer vision. Serving as a fundamental component in various applications such as virtual/augmented reality and robotics navigation *etc*., substantial efforts have been dedicated to addressing this challenge over the last few decades. Among those pioneering works, Neural Radiance Fields (NeRF) Mildenhall et al. () and 3D Gaussian Splatting (3D-GS) Kerbl et al. (), stand out for its utilization of differentiable rendering technique, and have garnered significant attention due to its capability to recover high-quality 3D scene representation from 2D images.
Commonly used sensors for 3D scene reconstruction are usually frame-based cameras, such as the RGB and RGB-D cameras. They usually capture full-brightness intensity images within a short exposure time at a regular frequency. Due to the characteristic of the data capturing process, they often suffer from motion blur or fail to capture accurate and informative intensity information under extreme brightness or darkness in the environment, which would further affect the performance of downstream applications.

The event camera, a bio-inspired sensor, has gained significant attention in recent years for its potential to address the limitations of frame-based cameras under challenging conditions. Unlike conventional cameras, event cameras record brightness changes asynchronously at each pixel, emitting events when a predefined threshold is surpassed. This unique operation offers several advantages over conventional cameras, in terms of high temporal resolution, high dynamic range, low latency and power consumption. Although event cameras have attractive characteristics for challenging environments, they cannot be directly integrated into existing frame-based 3D reconstruction algorithms that rely on processing dense 2D brightness intensity images, due to its time continuous, sparse, asynchronous and irregular data capturing characteristics.

Several pioneering works have been proposed to exploit event stream Kim et al. (); Rebecq et al. (); Gallego et al. () to recover the motion trajectory and scene representation. While existing methods deliver impressive performance, they usually exploit 2.5D semi-dense depth maps to represent the 3D scene, and bundle adjustment (BA) is hardly being performed, due to the asynchronous and sparse characteristics of event data stream. Klenk et al. () recently proposes to convert event stream into event voxel grids, and then adapt a previous frame-based deep visual odometry pipeline Teed et al. () for accurate camera motion estimation. As neural radiance fields (NeRF) exhibits impressive scene representation capability recently, Klenk et al. (), Hwang et al. (), Rudnev et al. (), Low & Lee () and Low & Lee () explore to recover the underlying dense 3D scene NeRF representation from event stream, by assuming ground truth poses are available.

In contrast to those works, we propose IncEventGS, an incremental dense 3D scene reconstruction algorithm from a single event camera, by exploiting Gaussian Splatting as the underlying scene representation. Different from prior event-based NeRF reconstruction methods, IncEventGS does not require any ground truth camera poses, which is more challenging and provides more flexibility for real-world application scenarios. To overcome the challenges brought by unknown poses, IncEventGS adopts the tracking and mapping paradigm of conventional SLAM pipelines Mur-Artal & Tardós (). In particular, IncEventGS exploits prior explored and reconstructed 3D scene for camera motion estimation of incoming event stream during the tracking stage. Both the 3D-GS scene representation and camera motions are then jointly optimized (*i.e*. event-based bundle adjustment) during the mapping stage, for more accurate scene representation and motion estimation. The 3D scene is progressively expanded and densified.
Both synthetic and real datasets are used to evaluate our method. The experimental results demonstrate that IncEventGS is able to recover the underlying 3D scene representation and camera motion trajectory accurately. In particular, IncEventGS outperforms prior NeRF-based methods and other related baselines in terms of scene representation recovery, even IncEventGS does not have the ground-truth poses. Furthermore, our method also delivers better camera motion estimation accuracy than a most recent state-of-the-art visual odometry algorithm, in terms of both the Absolute Trajectory Error (ATE) metric. The recovered 3D scene representation can be further used to render novel brightness images. Our main contributions can be summarized as follows:

- •
We present an incremental 3D Gaussian Splatting reconstruction algorithm from a single event camera, without requiring the ground truth camera poses;
- •
The experimental results on both the synthetic and real datasets demonstrate superior performance of our method over prior NeRF based methods and related baselines in terms of novel view synthesis, and better performance over state-of-the-art event-based visual odometry algorithm in terms of camera motion estimation;
- •
Compared to prior methods, we are able to efficiently render high-quality brightness images thanks to the powerful representation capability of 3D Gaussian Splatting.

## 2 Related Works

We review two main areas of prior works: event-based neural radiance fields and 3D Gaussian Splatting, which are the most related to our work.

Event-based neural radiance fields. Prior works from Klenk et al. (), Hwang et al. () and
Rudnev et al. () propose to exploit event stream to recover the neural radiance fields with known camera motion trajectory.
Low & Lee () further improves the reconstruction algorithm to handle the situation with sparse and noisy events under non-uniform motion.
The recovered neural radiance fields can then be used to render novel view brightness images. The ground truth poses are usually computed from corresponding brightness images via COLMAP Schonberger & Frahm () or provided by indoor motion capturing system.
Recently, Qu et al. () proposed to integrate event measurements into an RGB-D implicit neural SLAM framework and achieve robust performance under the situation with motion blur.
Li et al. () also propose to exploit event measurements and a single blurry image to recover the underlying neural 3D scene representation. In contrast to those works, IncEventGS conduct incremental 3D scene reconstruction without requiring any prior ground truth poses, which is more challenging and provides more flexibility for practical application scenarios. The method further exploits 3D Gaussian Splatting as the underlying scene representation, which demonstrates better image rendering quality and efficiency, compared to NeRF-based representation.

3D Gaussian Splatting. With the recent success of NeRF Mildenhall et al. () and its further developments Fridovich-Keil et al. (); Müller et al. (), novel view synthesis utilizing 3D representations like MLPs, voxel grids, or hash tables has advanced significantly. While these NeRF-inspired models perform admirably, they frequently require long training and rendering times for individual scenes. The introduction of 3D Gaussian Splatting Kerbl et al. () proposes a novel explicit 3D representation to further improve both the training and rendering efficiency. Due to its impressive efficient scene representation capability, several pioneering work have been proposed to exploit 3D-GS for incremental 3D reconstruction. For example, Keetha *et al*. propose an RGBD-based 3D-GS SLAM Keetha et al. (), employing an online tracking and mapping system tailored to the underlying Gaussian representation. Yan *et al*. implement a coarse-to-fine camera tracking approach based on the sparse selection of Gaussians Yan et al. (). Matsuki *et al*. propose to apply 3D Gaussian Splatting to do incremental 3D reconstruction using a single moving monocular or RGB-D camera Matsuki et al. (). Huang *et al*. exploits ORB-SLAM3 to compute accurate camera poses and feeds it into a 3D-GS algorithm for dense mapping Huang et al. (). Fu *et al*. uses monocular depth estimation with 3D-GS Fu et al. (). Yugay *et al*. combine DROID-SLAM Teed & Deng () based camera tracking with active and inactive 3D-GS sub-maps Yugay et al. (). Hu *et al*. propose a novel depth uncertainty model to ensure the selection of valuable Gaussian primitives during optimization Hu et al. (). While those methods deliver impressive performance in terms of 3D scene recovery and motion estimation, they usually assume the usage of frame-based images (*i.e*. either RGB or RGB-D date). In the contrary, we propose to exploit pure event measurements for incremental 3D-GS reconstruction.
Two concurrent work have also tried to exploit 3D Gaussians for event-based reconstruction recently, *i.e*. EvGGS Wang et al. () and Event3DGS Xiong et al. (). However, both of them rely on ground-truth poses for training, and EvGGS further constrained to object reconstruction from 360-degree surrounding views, while ours assumes the camera poses are not available and is more challenging.

## 3 Method

The overview of our IncEventGS is shown in Fig. 1. Given only a single event camera, IncEventGS incrementally performs tracking and dense mapping under the framework of 3D Gaussian Splatting, to recover both the camera motion trajectory and 3D scene representation simultaneously. The main insight of IncEventGS is to accumulate incoming event data into chunks and treat each chunk as a special “image”. We associate each chunk with a continuous time trajectory parameterization in the $\mathfrak{se}3$ space. Two close consecutive timestamps (i.e., $t_{k}$ and $t_{k+\Delta t}$, where $\Delta t$ is a small time interval) can be randomly sampled within the chunk and the corresponding event stream can then be integrated into an image ${\mathbf{E}}(x)$. Based on the parameterized trajectory, the corresponding camera poses (*i.e*., $\mathbf{T}_{k}$, $\mathbf{T}_{k+\Delta t}$) can be computed and the images (*i.e*., $\hat{\mathbf{I}}_{k}$, $\hat{\mathbf{I}}_{k+\Delta t}$) can be further rendered from the 3D Gaussian Splatting. The synthesized image $\hat{\mathbf{E}}(x)$ can be computed for event loss computation. During tracking, we only optimize for the camera motion trajectory of the newly accumulated event chunk and exploit the recovered trajectory to initialize the dense bundle adjustment (BA) algorithm for the mapping stage. During mapping stage, we continuously densify 3D Gaussians for newly explored areas and prune transparent 3D Gaussians. For computational efficiency consideration, we exploit a sliding window of the latest chunks and perform BA only within this window for both 3D-GS reconstruction and motion trajectory estimation. We will detail each component as follows.

Figure: Figure 1: The pipeline of IncEventGS. IncEventGS processes incoming event stream by dividing it into chunks and representing the camera trajectory as a continuous model. It randomly samples two close consecutive timestamps to integrate the corresponding event streams. Two brightness images are rendered from 3D Gaussian distributions at the corresponding poses, and we minimize the log difference between the rendered images and the accumulated event images. During initialization, a pre-trained depth estimation model estimates depth from the rendered images to bootstrap the system.
Refer to caption: x1.png

### 3.1 3D Scene Representation

Following 3D-GS Kerbl et al. (), the scene is represented by a set of 3D Gaussian primitives, each of which contains mean position $\boldsymbol{\mu}\in\mathbb{R}^{3}$ in the world coordinate, 3D covariance $\boldsymbol{\Sigma}\in\mathbb{R}^{3\times 3}$, opacity $\mathbf{o}\in\mathbb{R}$, and color $\mathbf{c}\in\mathbb{R}^{3}$. To ensure that the covariance matrix remains positive semi-definite throughout the gradient descent, the covariance $\boldsymbol{\Sigma}$ is parameterized using a scale vector $\mathbf{s}\in\mathbb{R}^{3}$ and rotation matrix $\mathbf{R}\in\mathbb{R}^{3\times 3}$:

$$ $\boldsymbol{\Sigma}=\mathbf{RSS}^{T}\mathbf{R}^{T},$ (1) $$

where scale matrix $\mathbf{S}=diag([s])$ is derived from the scale vector $\mathbf{s}\in\mathbb{R}^{3}$.

In order to enable rendering, 3D-GS projects 3D Gaussian primitives to the 2D image plane from a given camera pose $\mathbf{T}_{c}=\left\{\mathbf{R}_{c}\in\mathbb{R}^{3\times 3},\mathbf{t}_{c}
\in\mathbb{R}^{3}\right\}$ using following equation:

$$ $\boldsymbol{\Sigma}^{\prime}=\mathbf{JR}_{c}\boldsymbol{\Sigma}\mathbf{R}_{c}^ {T}\mathbf{J}^{T},$ (2) $$

where $\boldsymbol{\Sigma}^{\prime}\in\mathbb{R}^{3\times 3}$ is the 2D covariance matrix, $\boldsymbol{J}\in\mathbb{R}^{2\times 3}$ is the Jacobian of the affine approximation of the projective transformation. After projecting 3D Gaussians onto the image plane, the color of each pixel is determined by sorting the Gaussians according to their depth and then applying near-to-far ${\alpha}$-blending rendering via the following equation:

$$ $\mathbf{I}=\sum_{i}^{N}\mathbf{c}_{i}\alpha_{i}\prod_{j}^{i-1}\left(1-\alpha_{ j}\right),$ (3) $$

where $\mathbf{c}_{i}$ is the learnable color of each Gaussian, and $\alpha_{i}$ is the alpha value computed by evaluating the 2D covariance $\boldsymbol{\Sigma}^{\prime}$ multiplied with the learned Gaussian opacity $\mathbf{o}$:

$$ $\alpha_{i}=\mathbf{o}_{i}\cdot\exp\left(-\sigma_{i}\right),\quad\sigma_{i}= \frac{1}{2}\Delta_{i}^{T}\boldsymbol{\Sigma}^{\prime-1}\Delta_{i},$ (4) $$

where $\Delta_{i}\in\mathbb{R}^{2}$ is the offset between the pixel center and the 2D Gaussian center. Depth is rendered by:

$$ $\mathbf{D}=\sum_{i}^{N}\mathbf{d}_{i}\alpha_{i}\prod_{j}^{i-1}\left(1-\alpha_{ j}\right),$ (5) $$

where ${d}_{i}$ denotes the z-depth of the center of the i-th 3D Gaussian to the camera. We also render alpha map to determine visibility:

$$ $\mathbf{V}=\sum_{i}^{N}\alpha_{i}\prod_{j}^{i-1}\left(1-\alpha_{j}\right),$ (6) $$

The derivations presented above demonstrate that the rendered pixel color, denoted as I in (3), is a function that is differentiable with respect to the learnable attributes of all Gaussians, and the camera poses $\mathbf{T}_{c}$. This facilitates our bundle adjustment formulation, accommodating a set of event chunks and inaccurate camera motion trajectories within the framework of 3D-GS.

### 3.2 Event Data Formation Model

An event camera records changes of the brightness as a stream of events asynchronously. Every time a pixel brightness change reaches a contrast threshold (*i.e*. $|L(\mathbf{x},t_{i}+\delta t)-L(\mathbf{x},t_{i})|\geq C$), the camera will trigger an event $e_{i}=(\mathbf{x},t_{i},p_{i})$, where $p_{i}\in(-1,+1)$ is the polarity of the event, $L(\mathbf{x},t_{i})=\log(\mathbf{I}(\mathbf{x},t_{i}))$ is the brightness logarithm of pixel $\mathbf{x}$ at timestamp $t_{i}$, $C$ is the fixed contrast threshold.

To relate 3D-GS representation with the event stream, we sample two close consecutive timestamps (i.e., $t_{k}$ and $t_{k+\Delta t}$, where $\Delta t$ is a small time interval), and accumulate the real measured events within $\Delta t$ to an image $\mathbf{E}(\mathbf{x})$. The accumulation is defined as:

$$ $\mathbf{E}(\mathbf{x})=C\{e_{i}(\mathbf{x},t_{i},p_{i})\}_{t_{k}<t_{i}<t_{k}+ \Delta t},$ (7) $$

where $e(\mathbf{x},t_{i},p_{i})$ is the $i^{th}$ event within the defined time interval corresponding to pixel $\mathbf{x}$.
The corresponding camera poses $T_{k}$ and $T_{k+\Delta t}$ can be interpolated from the camera motion trajectory parameterization, allowing us to render two grayscale images (*i.e*. $\hat{\mathbf{I}}_{k}$ and $\hat{\mathbf{I}}_{k+\Delta t}$) from the previously recovered 3D-GS. The synthesized accumulated event image $\hat{\mathbf{E}}$ can then be computed as:

$$ $\hat{\mathbf{E}}(\mathbf{x})=\log(\hat{\mathbf{I}}_{k+\Delta t}(\mathbf{x}))- \log(\hat{\mathbf{I}}_{k}(\mathbf{x})),$ (8) $$

where $\hat{\mathbf{E}}(\mathbf{x})$ depends on the parameters of both the motion trajectory parameters and 3D-GS, and is differentiable with respect to them.

Both in tracking and mapping, inspired by the work of Rudnev et al. (), we segment the current event chunks into $n_{seg}$ equal segments according to the number of events, obtaining $n_{seg}$ timestamps that correspond to the end of each segment. We then randomly select one timestamp from these $n_{seg}$ timestamps to serve as $t_{k+\Delta t}$, and we randomly sample an integer $n_{win}$ between the integer bounds $n_{low}$ and $n_{up}$. The index of $t_{k}$ is equal to the index of $t_{k+\Delta t}$ subtract $n_{win}$. $n_{seg}$, $n_{low}$ and $n_{up}$ are hyperparameters. This sampling strategy enables the model to capture both local and global information.

### 3.3 Camera Motion Trajectory Modeling

Since each event chunk usually contains too many events, we sample a portion of them according to the total number of events during optimization. We formulate the corresponding poses (*i.e*. $\mathbf{T}_{k}$ and $\mathbf{T}_{k+\Delta t}$) at the beginning and end of the sampled event portion within each chunk, by employing a camera motion trajectory. The trajectory is represented through linear interpolation between two camera poses, one at the beginning of the chunk $\mathbf{T}_{\mathrm{start}}\in\mathbf{SE}(3)$ and the other at the end $\mathbf{T}_{\mathrm{end}}\in\mathbf{SE}(3)$. The camera pose at time $t_{k}$ can thus be expressed as follows:

$$ $\mathbf{T}_{k}=\mathbf{T}_{\mathrm{start}}\cdot\mathrm{exp}(\frac{t_{k}-t_{ start}}{t_{end}-t_{start}}\cdot\mathrm{log}(\mathbf{T}_{\mathrm{start}}^{-1} \cdot\mathbf{T}_{\mathrm{end}})),$ (9) $$

where $t_{start}$ and $t_{end}$ represent the timestamps corresponding to the boundary of the event chunk. It follows that $\mathbf{T}_{k}$ is differentiable with respect to both $\mathbf{T}_{\mathrm{start}}$ and $\mathbf{T}_{\mathrm{end}}$. The objective of IncEventGS is thus to estimate both $\mathbf{T}_{\mathrm{start}}$ and $\mathbf{T}_{\mathrm{end}}$ for each event chunk, along with the learnable parameters of 3D Gaussians $\mathbf{G}_{\theta}$.

### 3.4 Incremental Tracking and Mapping

For both tracking and mapping, we exploit the previously introduced event data formation model to compute the loss from the synthesized and real accumulated event images. In particular, we compute the loss of the latest event chunk only for the tracking stage and minimize the following energy function:

$$ $\mathbf{T}_{start}^{*},\mathbf{T}_{end}^{*}=\operatorname*{argmin~{}}_{\mathbf {T}_{start},\mathbf{T}_{end}}\left\lVert\mathbf{E}(\mathbf{x})-\hat{\mathbf{E} }(\mathbf{x})\right\rVert_{2},$ (10) $$

where both $\mathbf{E}(\mathbf{x})$ and $\hat{\mathbf{E}}(\mathbf{x})$ are the accumulated real and synthesized event images respectively, corresponding to a randomly sampled event portion within the latest event chunk.

Once the tracking is done, we insert the latest event chunk to mapper and exploit the estimated $\mathbf{T}_{start}^{*}$ and $\mathbf{T}_{end}^{*}$ as the initial value of the chunk to perform dense bundle adjustment. For computational consideration, we exploit a sliding window BA of the latest $n_{w}$ chunks and $n_{w}$ is a hyperparameter. In particular, we optimize both the motion trajectories and the 3D-GS jointly by minimizing the following loss functions:

$$ $\displaystyle\mathcal{L}$ $\displaystyle=(1-\lambda)\mathcal{L}_{event}+\lambda\mathcal{L}_{ssim},$ (11) $\displaystyle\mathcal{L}_{event}$ $\displaystyle=\frac{1}{n_{w}}\sum_{i=0}^{n_{w}}\left\lVert\mathbf{E}_{i}( \mathbf{x})-\hat{\mathbf{E}}_{i}(\mathbf{x})\right\rVert_{2},$ (12) $\displaystyle\mathcal{L}_{ssim}$ $\displaystyle=\frac{1}{n_{w}}\sum_{i=0}^{n_{w}}SSIM(\mathbf{E}_{i}(\mathbf{x}) ,\hat{\mathbf{E}}_{i}(\mathbf{x}))$ (13) $$

where $\lambda$ is a hyperparameter, SSIM is the structural dissimilarity loss Wang et al. (), both $\mathbf{E}_{i}(\mathbf{x})$ and $\hat{\mathbf{E}}_{i}(\mathbf{x})$ are the corresponding accumulated real and synthesized event images of the latest $i^{th}$ event portion respectively. As the event data streams in, we alternatively perform tracking and mapping.

3D-GS Initialization and System Boot-strapping. We initialize the 3D-GS by sampling point cloud randomly within a bounding box. The first $m$ event chunks (where $m$ is a hyperparameter) are selected for initialization, and all corresponding camera poses (e.g., $\mathbf{T}_{*}$) are randomly initialized to be near the identity matrix. We then minimize the loss computed by Eq. (11) with respect to the attributes of 3D-GS and the parameters of camera motion trajectories jointly.

Through experiments, we found that the above initialization procedure consistently produces high-quality brightness images. However, the 3D structure remains of low quality due to the short baselines of the event chunks. We further find that it could potentially affect the performance of the whole pipeline as more event data is received. Therefore, we utilize a monocular depth estimation network Ke et al. () to predict a dense depth map from the rendered brightness image. This depth map is then used to re-initialize the centers of the 3D Gaussians by unprojecting the pixel depths, after which we repeat the minimization of Eq. (11) for system bootstrapping.

3D-GS Incrementally Growing. As the camera moves, new Gaussians is periodically introduced to cover newly explored regions. After tracking, we obtain an accurate camera pose estimate for each new event chunk. The center of new Gaussians are determined by:

$$ $p=T\cdot\pi^{-1}(u,d_{u})$ (14) $$

where $p\in\mathbb{R}^{3}$ is a 3D point, $u\in\mathbb{R}^{2}$ is a point in the image plane, $d_{u}$ is depth of the 3D point $p$ projecting on pixel u, which is rendered by equation 5, $\pi^{-1}$ denotes camera inverse projection, $T$ is the camera pose from tracking. To ensure that new Gaussians are only added in previously unmapped areas, a visibility mask is generated to guide the expansion of the Gaussian splatting process, as following:

$$ $M(p)=V<\lambda_{V}$ (15) $$

where $V$ is the rendered alpha map, $\lambda_{V}$ is the hyperparameter.

## 4 Experiments

### 4.1 Experimental Setups.

#### Implementation Details.

All experiments were conducted on a desktop PC equipped with a 5.73GHz AMD Ryzen 9 7900x CPU and an NVIDIA RTX 3090 GPU. The first $m=3$ event chunks were used for initialization. During the mapping stage, a sliding window size of $n_{w}=20$ was employed for the bundle adjustment algorithm. The hyperparameters were set as follows: $\lambda=0.05$, $\lambda_{V}=0.8$, and $n_{seg}=100$. For the synthetic dataset, $n_{low}=400k$ and $n_{up}=500k$, while for the real dataset, $n_{low}=60k$ and $n_{up}=80k$. Each event chunk had a time interval of 50 ms. The learning rate of the camera poses is set to 1e-4 and that for the attributes of 3D-GS are set the same as the original 3D-GS work. The number of optimization steps for initialization is 4500, and that for tracking and mapping are set to 200 and 1500 respectively. The contrast threshold $C$ of the event camera is set to 0.1 for synthetic datasets and 0.2 for real datasets empirically.

Baselines and Evaluation Metrics.
IncEventGS performs incremental dense 3D-GS reconstruction and motion estimation using only a single event camera. To the best of our knowledge, there are no existing event-only NeRF or 3D-GS SLAM methods that do not rely on ground-truth poses, making direct comparisons challenging. Therefore, we conduct a thorough comparison of our method with several event-based NeRF approaches, including E-NeRF Klenk et al. (), EventNeRF Rudnev et al. (), and Robust e-NeRF Low & Lee (), as well as two-stage methods such as E2VID Rebecq et al. () + COLMAP Schönberger & Frahm () + 3DGS Kerbl et al. (), and E2VID Rebecq et al. () + DEVO Klenk et al. () + 3DGS Kerbl et al. (). E-NeRF, EventNeRF, and Robust e-NeRF leverage implicit neural radiance fields for 3D scene representation, requiring ground-truth camera poses for accurate NeRF reconstruction. The two-stage methods we examine include E2VID + COLMAP + 3DGS and E2VID + DEVO + 3DGS. In those approaches, event data is first converted into grayscale images using E2VID. The subsequent steps differ: in the E2VID + COLMAP + 3DGS method, camera poses are estimated from these images using COLMAP, while in the E2VID + DEVO + 3DGS method, poses are estimated using DEVO. Finally, 3D-GS is trained with the generated images and poses in both methods. Both the quantitative and qualitative comparisons are performed on the synthetic dataset. Since there are no paired ground truth images for the real dataset, we only perform qualitative comparisons on the real dataset. In terms of motion trajectory evaluations, we use the publicly available state-of-the-art event-only visual odometry method, *i.e*. DEVO Klenk et al. (), for comparison, both quantitatively and qualitatively.

The metrics used for novel view synthesis (NVS) include the commonly employed PSNR, SSIM, and LPIPS. For motion trajectory evaluations, we utilize Absolute Trajectory Error (ATE). To ensure fair comparisons, we employ the evaluation code provided by EventNeRF to compute the NVS metrics, which applies a linear color transformation between predictions and ground truth. Additionally, we use the public EVO toolbox Grupp () to compute the trajectory metrics.

#### Benchmark Datasets.

To properly evaluate the performance of NVS and motion trajectory estimation, we synthesized event data using the 3D scene models from the Replica dataset Straub et al. (). In particular, we exploit the room0, room2, office0, office2, and office3 scenes. We rendered high frame rate RGB images at 1000 Hz with a resolution of 768x480 pixels. Those images are then converted to grayscale and the event data is generated via the events simulator Gehrig et al. (). The contrast threshold is set to 0.1. To simulate real-world camera motions, we exploit the same motion trajectories as that of NICE-SLAM Zhu et al. () for data generation.

We used the event dataset provided by TUM-VIE Klenk et al. () for real data evaluations, which is also used by E-NeRF and Robust e-NeRF. TUM-VIE Klenk et al. () captured the event datasets by a pair of Prophesee Gen4 HD event cameras with a resolution of 1280x720 pixels. We only use the left-event camera data for our experiment.

### 4.2 Quantitative Evaluations.

We conduct quantitative evaluations against event NeRF methods(E-NeRF, EventNeRF, and Robust e-NeRF) and two-stage methods (E2VID + COLMAP + 3DGS and E2VID + DEVO + 3DGS) in terms of the quality of NVS and pose estimation performance.

The NVS performance is evaluated on Replica-dataset and the results are presented in Table 1. It is important to note that the metrics are lower than those typically observed in standard NeRF/3D-GS methods for RGB images, primarily due to the absence of adequate RGB image supervision. Even though nerf-based methods use ground truth poses for training, IncEventGS still significantly outperforms them, highlighting the advantages of our approach utilizing a 3D Gaussian representation. Additionally, our method greatly surpasses two-stage methods that also employ 3D Gaussian representation, demonstrating superior pose estimation and the effectiveness of our bundle adjustment technique.

**Table 1: NVS performance comparison on Replica dataset. The result demonstrates that our method outperforms NeRF-based and two-stage methods.**
|  | room0 | room2 | office0 | office2 | office3 |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |  |
| E-NeRF | 13.99 | 0.58 | 0.51 | 15.56 | 0.47 | 0.58 | 18.91 | 0.51 | 0.57 | 13.05 | 0.65 | 0.44 | 14.01 | 0.62 | 0.48 |
| EventNeRF | 17.29 | 0.62 | 0.39 | 16.02 | 0.54 | 0.64 | 18.90 | 0.43 | 0.62 | 15.18 | 0.66 | 0.45 | 16.77 | 0.73 | 0.33 |
| Robust e-NeRF | 17.26 | 0.84 | 0.18 | 16.43 | 0.50 | 0.52 | 18.93 | 0.52 | 0.56 | 16.81 | 0.81 | 0.25 | 19.22 | 0.84 | 0.18 |
| E2VID+<br>COLMAP+3DGS | 14.45 | 0.44 | 0.52 | 15.74 | 0.51 | 0.55 | 18.91 | 0.31 | 0.68 | 14.03 | 0.57 | 0.48 | 13.25 | 0.47 | 0.53 |
| E2VID+<br>DEVO+3DGS | 14.35 | 0.42 | 0.56 | 15.73 | 0.49 | 0.59 | 18.90 | 0.28 | 0.72 | 13.84 | 0.56 | 0.55 | 13.28 | 0.46 | 0.57 |
| Ours | 24.31 | 0.85 | 0.17 | 23.75 | 0.79 | 0.23 | 25.64 | 0.54 | 0.30 | 21.74 | 0.82 | 0.23 | 21.18 | 0.88 | 0.13 |

**Table 2: Pose accuracy (ATE, cm) on Replica and TUM-VIE datasets. The results demonstrate that our method delivers better performance in terms of camera motion estimation.**
|  | room0 | room2 | office0 | office2 | office3 | 1d | 3d | 6dof | desk | desk2 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DEVO | 0.289 | 0.266 | 0.138 | 0.281 | 0.156 | 0.147 | 0.303 | 2.93 | 0.732 | 0.201 |
| E2VID+COLMAP | 17.93 | 59.96 | 105.19 | 18.414 | 17.28 | 4.268 | 16.90 | 9.88 | 21.57 | 10.13 |
| Ours | 0.046 | 0.067 | 0.045 | 0.046 | 0.054 | 0.115 | 0.298 | 0.251 | 0.231 | 0.129 |

We evaluate pose estimation performance using the $ATE$ metric on both synthetic and real datasets, comparing our method with EVO and E2VID + COLMAP. The results, presented in Table 2, show that our method outperforms both baselines, validating the effectiveness of our incremental tracking and mapping technique.

**Table 3: Ablation Study about Depth Initialization. The unit of ATE is cm. The experimental results demonstrate the effectiveness of the initialization strategy. It not only improves the quality of rendered images, but also improves the accuracy of the camera motion estimation significantly.**
| Setting | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | ATE |
| --- | --- | --- | --- | --- |
| full | 21.74 | 0.82 | 0.23 | 0.046 |
| w/o | 17.80 | 0.76 | 0.26 | 1.534 |

### 4.3 Qualitative Evaluations.

We evaluate our method against event NeRF methods and two-stage methods qualitatively in terms of novel view image synthesis, both on synthetic and real data. The results are presented in both Fig. 2 and Fig. 3. It demonstrates that our method can deliver better novel view images, while event NeRF methods and two-stage methods render images with additional artifacts. Compared to NeRF-based methods, our approach demonstrates the advantage of IncEventGS by leveraging 3D Gaussian Splatting as the underlying scene representation. In contrast to two-stage methods, our dense bundle adjustment optimizes both 3D Gaussian Splatting and camera pose using event data, whereas two-stage approaches tend to accumulate errors over time, as confirmed by the experimental results. We also provide representative visualization of ATE error mapped onto trajectories in Fig. 4, both on synthetic and real dataset. It demonstrates that IncEventGS is able to recover more accurate motion trajectories.

### 4.4 Ablation Study

We conduct ablation studies to confirm our design choices. In particular, we study the effect of a monocular depth estimation network for system bootstrapping and event slicing hyperparameters $n_{low}$, $n_{up}$. The experiments are conducted with the Replica dataset and the results are shown in Table 4 and Table 4 respectively.

We found that depth initialization significantly impacts pose estimation, reducing the Average Trajectory Error (ATE) from 1.534 cm to 0.064 cm. Additionally, this improvement in pose estimation leads to a slight enhancement in Novel View Synthesis (NVS) performance. These results verify the importance of using depth initialization.

We compare several combinations of hyperparameters $n_{\text{low}}$ and $n_{\text{up}}$, which refer to the range of event slicing window sizes. Table 4 demonstrates that both too small and too large window sizes negatively impact the performance of Novel View Synthesis (NVS) and pose estimation. Consequently, we select $n_{\text{low}}=400k$ and $n_{\text{up}}=500k$ for our experiments on the Replica dataset.

## 5 Conclusion

We present the first incremental 3D dense reconstruction algorithm, *i.e*. IncEventGS, with a single event camera under the framework of 3D Gaussian Splatting. We adopt the tracking and mapping paradigm in conventional SLAM pipeline to do incremental motion estimation and 3D scene reconstruction simultaneously. To handle the continuous and asynchronous characteristics of event stream, we exploit a continuous trajectory model to model the event data formation process. The experimental results on both synthetic and real datasets demonstrate the superior performance of IncEventGS over prior state-of-the-art methods in terms of high-quality novel image synthesis and camera pose estimation.

Figure: Figure 2: Qualitative evaluation of novel view image synthesis on the Replica dataset. The experimental results demonstrate that our method renders higher-quality images with fewer artifacts compared to event-based NeRF and two-stage approaches.
Refer to caption: extracted/5938137/imgs/enerf/room0_long_modesk2_enerf_ep3920_0040.png

Figure: Figure 3: Qualitative evaluation for novel view image synthesis on real dataset. It demonstrates that our method is able to render better images with fewer artifacts than event NeRF methods and two-stage methods. Note there are no GT images aligned with the event camera, we choose closest images of RGB camera and crop it to the the size with rendered images for visual comparisons.
Refer to caption: extracted/5938137/imgs/e2vid+colmap+3dgs/desk_18.jpg

Figure: Figure 4: Representative visualization of ATE error mapped onto trajectories for the synthetic (office0) and real (6dof) datasets, generated by the EVO toolbox using the same ground truth poses, demonstrating the superior performance of our method in pose estimation.
Refer to caption: extracted/5938137/imgs/fig4_traj/office0_devo_map.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses. I have assigned a preliminary score based on the initial manuscript, but I may adjust this score based on the authors' responses and feedback from other reviewers.
soundness: 2
strengths: 1. The topic of event-based 3D reconstruction without camera pose is very interesting topic.

2. The authors conducted extensive experiments demonstrating that IncEventGS outperforms previous NeRF-based methods and other baselines, even without ground-truth camera poses.

3. The writing is clear and easy to understand.
summary: This paper proposes IncEventGS, an incremental dense 3D reconstruction method using a single event camera. To incrementally recover the 3D scene, IncEventGS leverages the tracking and mapping approach of traditional SLAM. The tracker first estimates initial camera motion from prior 3DGS reconstructions, while the mapper refines both the 3D scene and camera motion using the tracker’s motion trajectory estimates. The advantage of IncEventGS does not require any ground truth camera poses. The results show that IncEventGS outperforms prior NeRF-based methods and related baselines, even without ground-truth camera poses. Additionally, it surpasses SOTA event-based VO methods in camera motion estimation.
weaknesses: 1. While I acknowledge that this paper is the first to explore 3D reconstruction using a single event camera combined with 3D ground segmentation without camera poses, its novelty appears to be limited. There are existing works using traditional RGB cameras for 3D reconstruction without relying on camera poses, and the approach of directly accumulating events into event images does not clearly highlight significant contributions to the field, whether from the image-based 3D ground segmentation community or the event-based community. I encourage the authors to articulate the specific technical contributions of this work.

2. I recommend that the authors include more examples of extreme scenarios, such as high-speed motion and low-light conditions, alongside comparisons with RGB images. This could better demonstrate the advantages of using event cameras for 3D reconstruction.

3. Regarding the possibility of achieving colored 3D reconstruction, can this method be applied? Since there are existing color event cameras, could the authors obtain data from such cameras to create an example of colored reconstruction?

4. The writing could be further improved in several ways: a) The title in line 97 should be bolded and capitalized. b) Section 3.2 does not require an extensive explanation of event camera principles and image accumulation. c) The font sizes in Tables 1 and 2 should be made consistent.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1) Although this type of work is new in the area of event cameras, why is there no mention of other pose-free camera work in the field of frame-based cameras?

2) Since the document only expresses high-level ideas, are there any plans to make the code publicly available in the future?

3) Why are there no supplementary videos supporting the results shown in the manuscript?
soundness: 2
strengths: 1) An original concept in which 3D-GS and camera poses are optimized simultaneously.

2) Results that surpass the current state-of-the-art.
summary: 1) This manuscript presents a method in which a 3D scene is reconstructed using a single event camera and 3D-GS. The authors describe a process where the 3D scene reconstruction does not require provided camera poses. The 3D-GS parameters and camera poses are simultaneously calculated, using a concept similar to SLAM, but generating a dense 3D scene.

2) The presented method produces results that outperform the current state-of-the-art by a significant margin.
weaknesses: The manuscript is clearly written but does not explain in a precise and in-depth manner how it is carried out. In other words, the concepts expressed are only shown at a high level, without delving into small key details, such as the “continuous time trajectory parameterization” or how “the camera poses (T_k) and (T_{k+\Delta t}) can be interpolated,” and how exactly to “render two grayscale images (i.e., (\hat{I}k) and (\hat{I}{k+\Delta t})) from the previously recovered 3D-GS,” which makes it very difficult to reproduce the results.

Although the manuscript mentions some studies related to 3D-GS and event cameras, it does not mention 3D-GS works that perform 3D reconstruction or novel view synthesis with pose-free cameras and frame-based cameras.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Why did IncEventGS stop using the ground-truth pose after adopting Gaussian Splatting representations compared to Nerf-based representations?

2. As we know, 3DGS hardware friendliness is superior to Nerf-based representations, and I'm curious about the overall runtime of the system compared to Nerf-based.

3. More experiments need to be done, such as  Tanks and Temples.
soundness: 4
strengths: 1.  The results of IncEventGS shown in Tabs 1 and 2  are amazing and effective.

2.  This paper is written in an easy-to-understand manner.
summary: This paper proposes IncEventGS, which is an incremental 3D Gaussian Splatting reconstruction algorithm with a single event camera. IncEventGS employed the tracking and mapping paradigm of conventional SLAM pipelines.
weaknesses: 1. The motivation of this paper is weak, as the paper claims, "Due to its unique asynchronous and irregular data capturing process, limited work has been proposed to apply neural representation or 3D Gaussian splatting for an event camera". I think the authors should discover the reasons behind it rather than the superficial phenomenon.

2. The title is "Pose-free." Why the author did this is not explained.  I think that although no pose ground truth is provided, using conventional slam pipelines actually provides this variable implicitly. Conventional slam pipelines will be more robust than pose estimators, which use deep learning methods.

3. This paper mentions several times “due to its time continuous, sparse, asynchronous and irregular data capturing characteristics.” I don't think the authors have solved this problem; they are still taking the approach of stacking events into the frame.

4. In line 62, citation duplication.

5. The contribution needs to be rewritten, which is just like changing the representation from Nerf to GS. However, this work has already been done.

6. "The main insight of IncEventGS is to accumulate incoming event data into chunks and treat each chunk as a special "image".  This is not a contribution and does not need to be emphasized.

7. In line 216 and 307, C in (3) and equation 6.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. The re-initialization using the pre-trained depth model for regularization with the proposed Gaussian model is not clear. Can the authors provide more details about it? Especially regarding the visualization of the intermediate process.

2. For SLAM or VIO, the accuracy of the trajectory is crucial. However, for NVS (Novel View Synthesis) tasks, the proposed method merely reconstructing a gray map of the scene can diminish the significance of the task to some extent. It is not enough to work only on the gray map. Could we perform the NVS task on the RGB event dataset? For example, the dataset from [1] or [2] or the event-based color synthetic Replica dataset.

3. What's more, I noticed that the authors did not provide any supplementary materials. Could the authors provide some visual demos to better observe the overall effect of this method?

[1] Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion

[2] EventNeRF: Neural Radiance Fields from a Single Color Event Camera
soundness: 3
strengths: Overall, this paper presents an incremental 3D Gaussian Splatting reconstruction algorithm from a single event camera, without requiring the ground truth camera poses. It has a motivation and is adequate for the audience and also solid on the technical side and adds the event-based VO tricks and off-the-shelf depth model for re-initialization to 3DGS. Thus, this work is interesting for readers working at the intersection of novel view synthesis and neuromorphic vision.
summary: The present paper proposes a novel view synthesis approach that reconstructs 3D scenes from event camera streams without precise camera poses. The main goal is to handle the unique asynchronous and irregular data characteristics of event cameras, which pose significant challenges for traditional 3D reconstruction methods. By utilizing a tracking and mapping approach inspired by SLAM pipelines, the method estimates camera motion based on prior reconstructed 3D-GS scenes and incrementally refines both the camera motion and scene representation. It's capable of handling real-world scenarios with no gt. poses, offering improved performance compared to NeRF-based methods and event-based visual odometry. It efficiently renders high-quality brightness images, outperforming baseline methods in terms of novel view synthesis and motion estimation.
weaknesses: 1. The setup of the real-world experiments lacks validity. Specifically, for the evaluation on the TUM-VIE dataset, the qualitative results alone are insufficient. The authors should also include quantitative analysis using no-reference image quality assessment metrics to provide a more comprehensive evaluation.

2. Although the proposed method can operate using an event stream as input to reconstruct 3D Gaussians, it still relies on uniform event stream as input. The proposed method is, therefore, limited by the density of event data streams, which restricts its practical applications. 

3. Despite the detailed comparison of the quality of rendered images, the efficiency of the training and rendering process is not included, which is an important metric of NVS methods. Extra comparisons with other methods on training time and inference FPS would help better evaluate the proposed method.

4. This method is valuable for addressing event-based visual odometry. However, the authors focus more on the NVS task, and using Gaussian functions to reconstruct grayscale scenes seems less relevant, as they are mainly suited for head-mounted devices, which reduces the method’s rationale.

Beyond this I have mainly minor comments and nitpicks:

l.117, the sentence contains a grammatical error and should be revised. Specifically, "IncEventGS conduct..." should be corrected to "IncEventGS conducts...".

l.142, the expression should be standardized by changing "se3" to "se(3)" for clarity and consistency.

l.162~186, I think the re-initialization process is vital to the method, but the main figure of the pipeline does not reflect this which may generate some confusion with readers not familiar with the method.