3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt
Authors: name: Lukas Höllein, name: Aljaž Božič, name: Michael Zollhöfer, name: Matthias Nießner

We present 3DGS-LM, a new method that accelerates the reconstruction of 3D
Gaussian Splatting (3DGS) by replacing its ADAM optimizer with a tailored
Levenberg-Marquardt (LM). Existing methods reduce the optimization time by
decreasing the number of Gaussians or by improving the implementation of the
differentiable rasterizer. However, they still rely on the ADAM optimizer to
fit Gaussian parameters of a scene in thousands of iterations, which can take
up to an hour. To this end, we change the optimizer to LM that runs in
conjunction with the 3DGS differentiable rasterizer. For efficient GPU
parallization, we propose a caching data structure for intermediate gradients
that allows us to efficiently calculate Jacobian-vector products in custom CUDA
kernels. In every LM iteration, we calculate update directions from multiple
image subsets using these kernels and combine them in a weighted mean. Overall,
our method is 30% faster than the original 3DGS while obtaining the same
reconstruction quality. Our optimization is also agnostic to other methods that
acclerate 3DGS, thus enabling even faster speedups compared to vanilla 3DGS.