Package: taichi-splatting
Taichi SplattingRasterizer for Guassian Splatting using Taichi and PyTorch - embedded in python library. Currently very usable but in active development, so likely will break with new versions!This work is originally derived offTaichi 3D Gaussian Splatting, with significant re-organisation and changes.Key differences are the rendering algorithm is decomposed into separate operations (projection, shading functions, tile mapping and rasterization) which can be combined in different ways in order to facilitate a more flexible use, and gradients can be enabled on "all the things" as required for the application (and not when disabled, to save performance).Using the Taichi autodiff for a simpler implementation where possible (e.g. for projection, but not for the rasterization).Examples:Projecting features for lifting 2D to 3DColours via. spherical harmonicsDepth covariance without needing to build it into the renderer and remaining differentiable.Fully differentiable camera parameters (and ability to swap in new camera models)PerformanceA document describing some performance benchmarks of taichi-splattinghere. Through various optimizations, in particular optimizing the summation of gradients in the backward gradient kernel. Taichi-splatting achieves a very large speedup (often an order of magnitude) over the original taichi_3d_gaussian_splatting, and is faster than the reference diff_guassian_rasterization for a complete optimization pass (forward+backward), in particular much faster at higher resolutions.Major dependenciestaichi-nightlySome bug fixes which have occurred after the 1.7.0 release:pip install --upgrade -i https://pypi.taichi.graphics/simple/ taichi-nightlytorch >= 1.8 (probably works with earlier versions, too)InstallingInstall taichi-nightly as above (note that the external repo cannot be listed in the pyproject.toml)Clone down withgit cloneand install withpip install ./taichi-spattingpip install taichi-splattingExecutablesfit_image_gaussiansThere exists a toy optimizer for fitting a set of randomly initialized gaussians to some 2D imagesfit_image_gaussians- useful for testing rasterization without the rest of the dependencies.Fitting an image (fixed points):fit_image_gaussians <image file> --show  --n 20000Fitting an image (split and prune to target):fit_image_gaussians <image file> --show --n 1000 --target 20000See--helpfor other options.benchmarksThere exist benchmarks to evaluate performance on individual components in isolation undertaichi_splatting/benchmarks/testsTests (gradient tests and tests comparing to torch-based reference implementations) can be run with pytest, or individually undertaichi_splatting/tests/splat-viewerA viewer for reconstructions created with the original gaussian-splatting repository can be foundhereor installed with pip. Has dependencies on open3d and Qt.splat-benchmarkA benchmark for a full rendererer (in the same repository as above) with real reconstructions (rendering the original camera viewpoints).  Options exist for tweaking all the renderer parameters, benchmarking backward pass etc.ProgressDoneBenchmarks with original + taichi_3dgs rasterizerSimple view cullingProjection with autogradTile mapping (optimized and improved culling)Rasterizer forward pass and optimized backward passSpherical harmonics with autogradGradient tests for most parts (float64) - including rasterizer!Fit to image training example/testDepth and depth-covariance renderingCompute point visibility in backward pass (useful for model pruning)Example training on images with split/prune operationsNovel heuristics for split and prune operations computed optionally in backward passTodoDepth covariance example3D training code (likely different repository)Backward projection autograd takes a while to compile and is not cached properlyImprovementsExposed all internal constants as parametersSwitched to matrices as inputs instead of quaternionsTile mapping tighter culling for tile overlaps (~30% less rendered splats!)All configuration parameters exposed (e.g. tile_size, saturation threshold etc.)Warp reduction based backward pass for rasterizer, a decent boost in performanceConventionsTransformation matricesTransformations are notatedT_x_y, for exampleT_camera_worldcan be used to transform points in the world to points in the local camera bypoints_camera = T_camera_world @ points_world